NSNQuant: A Double Normalization Approach for Calibration-Free Low-Bit Vector Quantization of KV Cache
Authors: Donghyun Son
Euntae Choi
Sungjoo Yoo
Venue: NeurIPS 2025 poster ()
Keywords: KV cache quantization
vector quantization
large language model
inference optimization
Primary area: deep_learning

Abstract: Large Language Model (LLM) inference is typically memory-intensive, especially when processing large batch sizes and long sequences, due to the large size of key-value (KV) cache.
Vector Quantization (VQ) is recently adopted to alleviate this issue, but we find that the existing approach is susceptible to distribution shift due to its reliance on calibration datasets.
To address this limitation, we introduce $\textbf{NSNQuant}$, a calibration-free Vector Quantization (VQ) technique designed for low-bit compression of the KV cache. 
By applying a three-step transformation—$\textbf{1)}$ a token-wise normalization ($\textbf{N}$ormalize), $\textbf{2)}$ a channel-wise centering ($\textbf{S}$hift), and $\textbf{3)}$ a second token-wise normalization ($\textbf{N}$ormalize)—with Hadamard transform, NSNQuant effectively aligns the token distribution with the standard normal distribution. 
This alignment enables robust, calibration-free vector quantization using a single reusable codebook.
Extensive experiments show that NSNQuant consistently outperforms prior methods in both 1-bit and 2-bit settings, offering strong generalization and up to 3$\times$ throughput gains over full-precision baselines.

NSNQuant: A Double Normalization Approach for Calibration-Fres Low-Bit Vector Quantization of KV Cache
Donghyun Son Seoul National University Sappydh1@snu ac kr
Euntae Choi Seoul National University euntae. cho.ccohee maloi175@gmai.com
Sungjoo Yoo" Seoul National University sungjoo oo yoo@gmail..... com
Abstract
Large Language Model (LLM) inference is typically memory-intensive especially when processing large batch sizes and long sequences, due to the large size of key-value (KV) cache. Vector Quantization (VQ) is recently adopted to alleviate this issue, but we find that the existing approach is susceptible to distribution shift due to its reliance on calibration datasets. To address this limitation, we introduce NSNQuant, calibration-fres Vector Quantization (VQ) technique designed for low-bit compression of the KV cache. By applying a three-step tansformation-11) token-wise normalization (Normalize), 2) channel-wise centering (Shift), and 3) second token-wise normalization (Normalize)-wint Hadamard transform, NSNQuant effectively aligns the token distribution with the standard normal distribution. This alignment enables robust, calibration vector quantization using single reusable codebook. Extensive experiments show that NSNQuant consistently outperforms prior methods in both 1-bit and 2-bit settings, offering strong generalization and up 3x throughput gain over full-precision baselines.
Introduction
Large language models (LLMs) have been widely adopted across various domains due to their strong
generalization capabilities (I]. Recently, with the emerging trend of using LLMs solve complex problems, their use has expanded into long-conteds scenarios such as long-contex reasoning [40] [5]
and augmented generation (RAG) [24]. However, when processing long sequences, LLM
inference requires a significant amount of memory and is memory-bound [22 19].
One of the major causes is the large size of key-value (KV) cache, which linearly increases with the sequence length. To alleviate this issue, many studies have proposed approaches to compressing
KV cache effectively, such eviction 44 25 32 and low-rank approximation [5] 26 Among
them, quantization has been one the most widely adopted approaches. Previous studies 30 [9
analyze how outliers emerge in the KV cache and propose to quantize key and value along the channel dimension and token dimension, respectively. Recently, Coupled Quantization (CQ) [4 proposed to quantize multiple channels together, using centroids obtained from the calibration set. The idea of CQ is identical to the concept of vector quantization (VQ), where groups values are jointly quantized into codebook indices. Utilizing the centroids codebooks, CQ achieves tfate-of-the-a performance in diverse tasks, proving the effectiveness of VQ in KV cache quantization.
In our study, we observe that CQ fails to generalize in out-of-distributin (OOD) scenarios where the input distribution different from that of the calibration set (in-distribution) 42 For example, Figure La shows that while CQ excels in WikiText-2, it performs much worse in C4, since it is
*Corresooning author.
39th Conference on Neural Information Processing Systems (NcurIPS 2025).

---

LLaMA3.1-8B KV (Layer 10, Head
50
49.00
20
1.2
12.50
6.25
Keys
Values
NENQvart-26
c0-8:10b NSKQuant-1a
C4
WikfText2
MultiNews
LCC
SAMSum
(a)
(b)
Figure (a) PPL evaluation results with LLaMA3.1-8B. Although CQ achieves lower PPL in WikiText-2 (in-distribution), it performs worse on C4 (out-of-distribution)) (b) t-SNE visualization of LLaMA3.1 1-8B key and value. The clustering pattern shows that key and value distributions strongly depend on the input data. More visualization can be found in Figure
calibrated using only a few samples from WikiText-2 dataset. Moreover, Figure Ib demonstrates the distribution of key and value varies greatly with the input distribution. This implies that learning centroids from the small set of data is very risky when applying them to other datasets. To this end, we propose NSNQuant, a calibration-free vector quantization (VQ) method that generalizes well to wide range of datasets. Our contributions are summarized as follows:
We empirically show that the strong dependence of key-value distributions on the input dataset can lead to severe errors in the existing VQ method. In particular, we observe that the centroids learned by CQ-4c9b fail to accurately quantize important punctuation tokens in LLaMA3-8B and 1.....11-8
C4, resulting degraded performance.
To address this limitation, we propose NSNQuant, a calibration- free VQ method for KV cache. NSNQuant effectively matches the key and value distribution with the standard nor- mal distribution through the Jormalize-Shift-Normall transform, shown in Figure Since the approximate distribution of key-value is known prior to inference, an effective codebook for VQ can be built without any external data.
(NSN) process and the Hadamard
We conduct comprehensive experiments and analysis show the effectiveness of NSNQuant across different tasks and models. The results with the LLaMA [3 36 14] and Mistral models (2I) clearly show that NSNQuant outperforms other baselines in 1-bit and 2-bit quantization. Moreover, we implement efficient CUDA kernels for the fow-bit computation, improving throughput and reducing memory usage.
2 Preliminary
LLM inference and KV cache LLM inference has two main stages: prefilling and decoding. In the prefilling stage, all prompt tokens are processed simultaneously by the transformer decoder layers (38. In the decoding stage, new tokens are generated one by one in an autoregressive manner. In both stages, each token only attends to previous tokens due to the causal nature of masked self- attention. To avoid redundant computation, KV cache stores key-value pairs from previous tokens. It is initialized during prefilling and updated at each decoding step by appending the latest key-value pair. It becomes primary bottleneck when processing long sequences, as its size linearly increases with the sequence length.
Vector quantization Unlike scalar quantization (SQ) where each scalar value is quantized individ- ually, vector quantization (VQ) compresses a group of values jointly using codebook. In VQ, d-dimensional vector matched the closest entry codebook, and quantized as follows: VQ(v) argmin D(e, Cli),
where C denotes the codebook and D(a,) 6) is distance function between vectors and 6.

---

Rynd
st 4maaa
Wg
RoPE
soltmax
W= o
Riind
W
(e(KOOP<(">))
NSN
RoPE
Rgtad
S
@ 15kin) Charmal wase Centering
Read
NSN
G. Permatbo
Vector
Scale Adjustment
Hadamarr
Normolize- Shift-
Quantization
Transtorm
NSN
Fused
Custom Karmita
(Matrix)
Normalize
Weights
Key Varkum rokene
Figure 2: Overall structure of attention under NSNQuant. Residuals are omitted for simplicity. We use superscript & and v to mark values associated with key and value, respectively. Since NSN is applied to keys before RoPE, two branches are needed to correctly compute attention scores. Details of the attention computation shown in the figure are provided in Appendix B
Original distribution unpredictatdee
Hadamard transform Gaussian--hhpped.
NSN Hadamard transform slighed distrbbttion!
Figure 3: A visual illustration of the effect of NSN on the per-channel value distribution. Our Normalize-Shinn
(NSN) process aligns the distribution with the standard normal distribu- tion, when used with the Hadamard transform.
Hadamard transform Hadamard matrix is an orthogonal matrix whose entries all have the same magnitude. Its size can be doubled recursively through Sylvester's construction, which builds larger Hadamard matrix by combining copies of a smaller one in specific pattern. This recursive definition forms the basis of the -alsh-Hadamard transform, allowing matrix-vector multiplication in (d log d) time. QuIP# (37] adopts the randomized Hadamard transform (RHT) [[6), where the sign of each row and column is flipped independently with probability 1/2. We also employ RHT to compute the theoretical bounds presented in Lemma
Method
3.1 Motivation
As shown in Figure Ta we observe several cases where the existing VQ method, coupled quantization (CQ) [43], suffers from severe performance degradation when tested on OOD datasets. We attribute this to the distribution mismatch between datasets, as visualized in Figure [b] We observe that this mismatch can cause severe errors in certain datasets. A notable example of such errors is the large quantization error observed in punctuation tokens in LLaMA3-8B and LLaMA3.1-8B.
WikiText-2, calibration dataset for CQ, only contains dataset includes "." (without whitespace) tokens. This mismatch leads to large errors for the keys corresponding to the punctuation token " in the first layer of LLaMA3 models, since the centroids of CQ are obtained only from WikiText-2. This leads to significant distortion in attention weights because certain heads, the "." token accounts for over 90% of the attention weights. By preserving the keys corresponding to these tokens in the first layer, the perplexity of CQ-4c9b on C4 is improved from 13.97 to 9.15 LLaMA3-8P and from 12.24 to 9.16 L. I.LMA3.. 1-8B, closely matching the results of NSNQuant.
(with whitespace) tokens, while the C4

---

Table Average chan KL divergence with the standard normal distribution measured on key and value of LLaMA2-7B on WikiText-2 with sequence length of 4096. KL divergence is computed between the binned empirical distribution and the standard normal CDF. We used 4096 random samples for Oracle, to match the sequence length.
Key
Value
Method
w/ Had.
w/o Had.
w/ Had.
w/o Had.
N
0.2476
0,5405
0.0817
0.1109
NS
0.3068
0,5261
0.0356
0.0666
NSN
0.0197
0.2230
0.0252
0.0537
Oracle
(torch.randa)
0.0096
To avoid these calibration-induced errors, we propose a calibration-fres vector quantization (VQ) method, NSNQuant, which does not rely on any external data. While CQ tries to match the codebook the key-value (KV) distribution, we instead propose to match the KV distribution to a well-known prior. Motivated by the recent success of Hadamard-based methods in producing normal-like output distributions regardless of the input, we introduce a novel transformation Normalize Normaliee (NSN) that aligns key and value channels to standard normal dis- tribution. This enables us to construct single reusable codebook tailored for the standard normal distribution, making our method, NSNQuant, both calibration-free and robust across diverse inputs.
The overall structure of NSNQuant is depicted in Figure 2 Key-value tokens first processed through the NSN and Hadamard transform. They are then compressed by vector quantization (VQ) using a codebook, and scaling parameter $2 adjusted adaptively the VQ result. The detailed process of NSN is described in Section 3.2 The post-scaling technique and the recipe for building a codebook are introduced in Section 3.3 and respectively.
3.2
(NSN)
NSN consists of three steps: 1) token-wise normalization (Normalize), 2) a channel-wise centering (Shift), 3) second token-wise normalization (Normalize).
Let E Ipl be the tensor where is sequence length and is hidden dimension per head. In the first Normalize step, each token normalized to have norm of vd, where dl a token dimension per head. This prevents outlier tokens 7 from dominating the next steps and incurring large errors in tokens with small magnitude. In the Shift step, channel-wise mean is calculated and subtracted, so that the resulting distribution is vero-centered. Finally, in the second Normalize step, each token is again normalized to have norm vd. The entire process is formulated as follows:
1. Normalize: $1 < norm(u, dim=token)/ /d, on 0/51
2. Shift: 4- dim=ccanee). Uns f- Un
3. Normalize: $2 + norm(uns (us. =orm(()e.dimm-tkee))) /d.
Unsn +- Uns Uns/$2
Each step produces byproduct-denottee tensor by v = 81 (82 + o). Although the last step can deviate the channel-wise mean from zero, we find that its effect is negligible, as further discussed in Appendix D.I
81, o, and w2-which is used to restore the original
When used together with the subsequent Hadamard transform, our NSN process effectively aligns the channel distribution with the standard normal distribution, as shown in Table[T ① As identified in previous studies 337
20), the Hadamard transform results in a normal-like distribution, which is supported by the central limit theorem. 2 NSN process roughly standardizes the distribution of each channel when used with the following Hadamard transform. Putting ! and 2 together, the
resulting channel distribution is aligned with the standard normal distribution. ② can be justified through the following lemma, which gives theoretical bounds for variances:
Let X (XI,... Xa) Idd be the random vector from the joint distribution of the channels after the NSN process, which satisfies
Lemma 1.
(I) Nearly centered: (E(.))) S.. (2) Normalized: C d.

---

I. For the randomized Hadamard transform RHT(X), 0 < a and E .1.,........
(3) Covariance bound: || Cov(X) -
P ((Va((Y)) (1- Ba: 1+ 1+1 2 -
1 (n(2/a)) and e is a constant from the Hanson- inequality B4y. where Ba = d
The proof for the lemma is provided in Appendix A.1 Here, adopt randomized Hadamard transform (RHT) to make the probabilistic claim, but using the naive Hadamard transform works well in practice, as presented in Table Since NSN tightens the bound e, and T is small for most of the layers, we observe that the resulting variances are generally close to 1. However, we find that does not hold in certain heads in the early layers, presented in Figures 12 and [3] This is due to the presence of outlier channels with huge variances in the first layer, which enlarges the bound Despite this limitation, Figure demonstrates that the quantization error remains low in these layers as well. We leave it to future work to explicitly model and account for these exceptions.
Residual To implement the second step (Shift) the decoding stage, we bring the idea of residual from KIVI [3). Following KIVI, we split KV cache into two parts: one part with quantized KV cache, and the other part with full -ree-iiin KV cache (residual). the size of the residual reaches its maximum capacity, the KV cache in the residual is flushed, quantized, and appended to the quantized part. We introduce hyperparameter called residual size control its size. To ensure consistency, we also apply NSN in residual-size chunks during the prefilling stage. In our experiments, we set the residual size to 64.
NSN applied to key and value As illustrated in Figure NSN is applied slightly differently the key and value. For keys, NSN is applied immediately after the projection layer, and Unan is quantized following the RoPE and Hadamard transform. While applying NSN after RoPE may seem more straightforward since RoPE might affect the channel-wise mean, we find that this ordering yields better quantization quality. Since RoPE is applied kernel when computing attention scores, as shown in the figure. For values, the Hadamard transform fused into the projection layers, and NSN is applied right afterward. Note that since the Hadamard transform is equivalent to multiplication by a rotation matrix, its order with the adjacent NSN does not change the output.
yet, instead apply it within
custom
3.3 Scale adjustment
Let u E Rd be token vector that is processed by the NSN and Hadamard transform. It is then divided into 8-dimensional sub- vectors and quantized using the codebook, following the objective in Equation | Let uo be the vector restored by looking up the codebook. i.e., eq C[VQ(u)]. We find that rather than using ti as-is, scaling tiQ adaptively improves performance. Specifically, we find that it is beneficial to scale uo as follows, which is identical to scaling 82 since vo is multiplied by 82 when restoring:
||u1B V- uQ
||013
UQ
(i.e., 82 + U UQ
82).
This is a scaling strategy which makes Uo - orthogonal to v. In other words, it preserves components local context and
parallel v, while allowing some orthogonal errors. Interpreting distinctive token feature, this strategy can be interpreted as making each token distinctive, which essential for KV cache considering its selective property. We provide a comparison of different scaling strategies in Appendix C.3
3.4 Codebook tuning
We construct single global codebook for compressing 8-dimensional vectors using integer indices, following QulP# 37]. NSNQuant-2b uses 8 bits for signs and 8 bits for codebook indices, while NSNQuant- lb uses only bits for indices. simple baseline can be built via K-Means on standard normal data, but its local optimality limits performance. We improve this by fine-tuning synthetic
5

---

standard normal data (torch.randn) to minimize cosine distance between original and quantized vectors, since the error of scale adjustment depends only on the angle between them. As the lookup is non-differentiable, gradients are propagated only through post-lookup operations. The PyTorch implementation for this process runs in less than 5 minutes on an RTX 3090, unlike the calibration process of CQ or KVQuant which requires backpropagation through model weights. While E8P [37 is also a competitive 2-bit baseline, it is hard to apply to the 1-bit scenario, and requires expensive comparisons with over 2000 entries. We provide comparison results in Table I0
3.5 Double quantization
To further reduce the memory overhead, we employ double quantization (DQ) proposed in QLoRA [T], which quantizes parameters used for the quantization. Specifically, we quantize and 81 with group size of 32 and residual size, respectively, using 4-bit round-to-nearest (RTN) quantization. DQ significantly reduces the average bit width. As a result, NSNQuant costs only additional 0.23 bits on average for the NSN process, when residual size is set to 64. Moreover, to reduce the amount of shared memory required for the codebook, we also apply 4-bit quantization to codebook entries. As shown in Table T2 DQ barely affects the effectiveness of NSNQuant.
3.6 Efficient kernel implementation
We implement the CUDA kernels for efficient execution of NSNQuant-2b and NSNQuant- 1b. For codebook matching, compute and manage distances on-the-fly in streaming multiprocessors (SMs), while loading codebook tiles into shared memory. For dequantization and matrix-vector multiplications, our kernel loads the codebook into shared memory minimize access to DRAM. We implement two different matrix-vector multiplication kernels for qk" and We (W: attention weights) since they use different axes for reduction. We also fuse the computation of both the quantized and the residual parts to maximize GPU utilization even in the small-batch scenarios.
4 Experiments
4.1 Experimental setup
Datasets
First, we evaluate perplexity (PPL) on WikiText-2 and C4 dataset to show quantization
error in language modeling. Second, we report evaluation results on LongBench 3 We choose the
task subset from LongBench following KIVI [30]: Qasper for a single-document QA task; QMSum and MultiNews for summarization tasks; TREC, TriviaQA and SAMSum for few-shot learning tasks; LCC and RepoBench-P for code completion tasks. Lastly, we run evaluations on GSMSK [8], HumanEval (6), CoQA (33, and MMLU [18] using LM-Eval framework [3] to test generation capability in more diverse generation scenarios.
Baselines We compare NSNQuant against four baselines: KIVI 30
KIVI Hadamard,
KVQuant 19), and CQ 43 While some of these methods provide multiple versions with similar average bits (e.g., CQ-4c8b and CQ-4c9b), we adopt the stronger variants to better demonstrate the superiority of our approach. Simpler baselines like INT2 are excluded, as prior studies [19 43 have shown them to be ineffective. For NSNQuant, we fix the residual size to 64. For KIVI, we use group size of 64 for keys and 128 for tokens, to maintain consistency with our residual policy. Detailed explanations for each baseline are provided in Appendix E.3 Note that since CQ does not provide their official implementation, we reproduce it based the official implementation of KVQuant. The result of CQ is slightly worse than the one reported in the original paper.
Policy for full-precision eache Since maintaining full-preeision cache largely affects generation quality (30 43) we unify several settings to focus only on quantization quality: all methods adopt NSNQuant's 's residual policy, which buffers full precision caches until the residual is full. We fix the residual size to 64 in all experiments. Attention sink-aware quantization is removed from KVQuant, as can be easily applied to other methods as well. Since some methods (e.g.. KIVI) use full-precision cache in the prefilling stage, while others (e.g.. CQ) use quantized cache, we standardize this by using perplexity evaluations, where key and value quantization is necessary to measure forward-pass quality.
full full-precision cache in the prefilling stage for all methods-

---

4.2 PPL evaluation
Table 2: Perplexity (PPL) evaluation results on WikiText-2 and C4 with a context length of 4096.
The results of CQ reported in the original paper marked with f-
Method
Avg. hit width
Dataset
HLaMA2-13B HaMMA27B
LLaMA3-8B
LLMMA3.1-B
Mistral- -B-
C4
6.63
6,04
8.32
8.43
7.48
FP16
WikITex-22
5.12
4.57
5.75
5.84
4.95
16.43
15.80
8.83
KIVI-2
2.38
WikiText-2
6,14
5.30
10.93
10.55
6.03
7.57
6.68
12.67
12.54
8.43
KIVI-2 Had
2.38
WikiTexi-2
5.79
5.05
8.69
8.86
5.65
KVQuant-2h
C4
7.09
6.37
9.75
9.60
7.93
Wikilext-2
5.52
4.88
6.74
6.71
5.32
CQ-4cbb
C4
7.12 (7.02l)
6.45 (6.36)
13.97
12.24
7.86
WikiTex--2
5,36 (3.32))
4,76 (4,741)
6.16
6.28
5.16
C4
6.86
6.21
9.08
9.15
7.69
NSNQuant-26
2.23
WikiText-2
5.29
4,71
6.23
6.34
5.12
KVQuant-Ib
30.79
14.27
33.17
37.37
12.45
WikiTexi-2
13.5
9.91
27.57
33.96
9.06
C4
9.25
(9.12))
8.17 (8.01))
43.78
49.50
9.60
WikiTex-22
6.33 (6.251)
5.53 (5.471)
7.69
7.87
6.01
NSNQuant-1b
C4
8.70
7.55
16.69
17.20
9.67
WikiText-2
6.69
5.70
11.70
12.58
6.66
Table presents the perplexity (PPL) of various models evaluated on WikiText-2 and C4. Although our framework uses full-precision key and value during the prefilling stage, we follow the evaluation protocol of CQ and KVQuant by quantizing them all in this experiment. Despite its difference from the generation scenario, we find PPL evaluation useful because measures quantization quality for sequence with a single forward pass. Furthermore, the results are aligned with those in Table 17] where we report the PPL evaluation results in the generation scenario with residuals. Therefore, report PPL results in the main table, and use them in ablation studies.
Across both datasets, CQ and NSNQuant outperform the other methods. On Wiki'Text-2, the two methods exhibit comparable performance in 2-bit quantization, with CQ outperforming NSNQuant in the 1-bit setting. However, C4, NSNQuant consistently outperforms CQ in both 2-bit and 1-bit quantization. Notably, CQ suffers from severe performance degradation on C4 when using LLaMA3-SB and LLaMA3. whereas NSNQuant maintains strong performance. These results suggest that NSNQuant generalizes better across datasets, while CQ struggles under distribution shift from the calibration dataset.
4.3 LongBench evaluation
Table 3: Evaluation results on LongBench. The task subset selected following KIVI results with different models can be found in Table T8
30
More
Qumper
QMSum
MolaNews
TREC
TrivaQA
SAMSum
LCC
RepoBenshh
Model
Method
Bits
Avg.
FP16
16
13.11
23.53
26.74
72.50
91.65
43.78
63.04
56.17
48.82
KIVI-2
KIVI-2 Had
2.38
11.57
24.28
26.51
72.50
92.09
43.21
62.90
55,20
48.53
LLaMA3.1 1-BB-BBBBBBB In-snsttttt
OQ-4c%
KVQuant-25 2b + 1%
2.26
2.32
12.25
13.15
23,80
23.45
26.24
25.74
71.50
72.00
91.63
91.53
41.39
60.80
18
54.41
$4,46
47.88
47 87
NSNQuunt-26
2.23
12:44
23.74
26.95
72.50
91.75
44.01
62.05
55.09
48.56
KVQuant-Ib 1%
1.32
9.91
22.19
22.27
47.50
88. 92
35.76
$0.27
43.79
40.0%
Sc
1.27
8.84
21.18
22.40
47.50
87.94
38.86
53.81
45.73
40.78
NSNQumm-1h
27.16
71.50
60.08
FP16
41.13
25.75
27.78
76.00
88.59
47.47
59.52
60.64
$3.36
KIVI-2
2.38
37.86
24.62
26.85
76.00
88.51
45.93
58.72
57.87
$2.05
KIV1-2 + Had
2.58
39.99
25.42
27.50
76.00
88. 12..22
46.52
59. 54.54
66.11
$2.94
Mistral- Instruct-v0.3
KVOuant-20 + 1%
2.33
38.98
25.10
27.22
76.00
89.02
45.27
58.57
61.59
52.72
NSNQuamt226
2,23
39.96
24.91
27.54
76.00
88.96
46.47
$8.70
59.45
$2.75
K KVQuant-Ib 1%
1.32
28.58
21,89
22.76
50.50
87.
39.62
54.73
$4,46
45.04
CQ-8c106
31.21
$3,82
NBNQuan-16
1.23
37.94
25.03
26.81
76.00
89.39
46.37
56.75
55.57
51.73

---

ggives performance comparison on LongBench. In 1-bit quantization, NSNQuant- 1b outper- forms other baselines by large margin. However, 2-bit quantization, all methods exhibit similar performances. We attribute this trend to the noisy nature of certain tasks. For example, in code generation tasks like LCC and RepoBench-P, models tend to generate additional descriptions of the codes. These descriptions do not hurt the code quality, but eventually degrade the metric. To this end, for tasks with potentially noisy metrics, we also report the ROUGE-L score with the FP16 output to measure how well each method preserves the original model output. Table 19 clearll shows that NSNQuant best preserves original model outputs.
4.4 Evaluation on more datasets
Table Evaluation results on GSM8K, HumanEval, CoQA, and MMLU. Accuracy is reported for all tasks. Results with LLaMA2-13B-CHL and -aaMa3-BB-Instruet can be found in Table 20
MMLU (4-shot, CoT)
Model
Method
Biis
GSMSK (8-shot, CoT)
Humanlival
CoQA
Humunitie
STEM
Social
Other
76.65
57.93
63.78
71.47
57.96
72.52
KIVI-2
2.38
64.59
48.17
63.60
64.44
50.09
66.84
66.13
Had
68.14
LLaMA3.1 1-8B-Instract
CQ-4c9h
KVQuant-20 1%
2.32
2,26
72.93
70.05
48.78
53.05
62.93
62.37
67.07
68.18
52.06
34.78
71.31
70.22
09.33
69.61
NSNQuart-2b
75.89
70.74
KVQuant + 1%
1.32
21.53
23.17
53.55
23.04
11.23
37.59
33.02
CQ-8ct0b
1,27
44.88
25.61
56.58
28.21
21.34
31.97
41,10
NSNQuant-h
1,23
62,70
$9.82
FP16
16
53.15
31,10
65.58
65.98
50.46
71.06
68.26
KIVI-2
2.38
43.75
28.66
64,45
60.96
39.93
63.52
59.05
KIVI-2 Had
2.38
46.10
28.05
65.48
63.28
45.11
66.99
63.07
Mistral-788 10irmll-7--nsttme--x0...
CQ-49b
KVQuant-bb 18
2.26
2.32
46.63
27.44
64.228
63.00
62,48
45.47
67.39
63.96
66.27
NSNQuant-20
2.23
51.02
31.10
65.62
64.92
47.65
69.11
67.56
KVQuant-Bi
18
1.32
16.
19.51
55.95
16.48
9.88
17.18
14.21
CQ-Sc10b
1,27
59.07
NSNQuam-11 1.23
38.89
27.44
63.60
58.52
40.34
02.34
58.43
In addition to the LongBench results, we further evaluate the models on GSM8K (mathematical reasoning). HumanEval (code generation), CoQA (conversational question answering), and MMLU (multi-task language understanding) to provide a more comprehensive assessment of generation quality. As shown in Table NSNQuant outperforms other baselines in most of the settings. In particular, NSNQuant excels in GSM8K and MMLU a few-shot CoTT (Chain-of- Thought) setting, which requires models generate exact and striet reasoning path.
3000
70
Throughput (tokensis)
2500
60
Memory (GB)
2000
90
1500
40
30
1000
FP16
FP16
20
NSNQuant-2b
500
NS-NQaant-26
NSNQuuntt11
NSNQuant-1b
10
200
400
600
1000
Batch Size
Batch Size
Figure 4: Peak memory usage (left) and throughput (right) measured with varying batch sizes. The residual size is set to 64. Results with varying residual sizes are available in Figure 8
4.5 Memory usage and throughput
To evaluate the efficiency of NSNQuant in terms of both memory and runtime, we measure its memory consumption and throughput. Following previous works 30 23 we use the synthetic data to simulate the scenario of ShareGPT (35) where the average input length is 161 and the average generation length is 338. The evaluation is performed on a Linux server with a single A100-80GB GPU with LLaMA2-7B.

---

Table 5: Latency (ms) breakdown of NSNQuant-2b compared with the FP16 baseline. HT and VQ refer to Hadamard transform and vector quantization, respectively. More results with different configurations can be found in Table 21
Stage
Method
Total
NSN
VQ
FP16
1225.90
Prefill
NSNQuant-2b
1707.22
155.22
15.04
326.85
Decode
FP16
50.80
(per step)
NSNQuant-26
36.49
0.34
1.88
0.66
Table 6: PPL measured on WikiText-2 with LLaMA2-7B without each component of NSNQuant-2b.
Method
PPL.
NSNQuant-2b
5.285
w/o first token- wise normalization
6.293
w/o channel-wise centering
5.842
w/o second token-wa wise normalization
5,456
w/o Hadamard transform
5.730
The result is illustrated in Figure NSNQuant-2b and NSNQuant- 1b scale efficiently to larger batch sizes, achieving 4x larger batch sizes and 3x speedup in throughput. Note that the memory increases are not proportional to average bits since the residual is also included in the memory usage.
4
While FP16 baseline suffers from OOM for large batch sizes,
4.6 Latency breakdown
Although NSNQuant enables the of larger batch sizes, it introduces additional computations for the Hadamard transform, vector quantization, and NSN. To better understand their impact on efficiency, measure the latency of each operation during the forward pass. We set the batch size to 32 and the input length to 512, and generate 64 tokens to match the residual size. The latency is measured with LLaMA2-7B averaged over 100 runs.
Linux server with A100-80GB, and the reported values
The result is given in Table 5 Due to the additional overhead, NSNQuant has higher latency in the prefilling stage. On the other hand, in the decoding stage, achieves lower latency since NSNQuant alleviates the memory-boum bound nature of the attention computation by compressing the KV cache into lower bits. This suggests that NSNQuant is particularly advantageous for decode-heavy tasks such as reasoning code generation.
4.7 Ablation study
Effects of the NSN and Hadamard To assess the contribution of each NSN component and the Hadamard transform, we measure perplexity on WikiTeext-2 while removing them individually. As shown in Table 6 skipping any step results higher perplexity. This trend aligns with Table [ which shows using NSN and the Hadamard transform together leads to the best alignment. The first token-wise normalization has the greatest impact, as it suppresses outlier tokens before centering. In contrast, the second token-wise normalization has minimal effect, suggesting that channel-wise centering does not significantly alter token scale.
Effects of codebook tuning Figure 5) shows the cosine similarity between the original vector v and the quantized vector U/ == C[)Q(v)) with different codebooks. The fine-tuned version gives higher cosine similarities compared to the K-Means codebook. Furthermore, the cosine similarities are consistently high the early layers, although the standardization of NSN does not hold these layers. Notably, the cosine similarities measured using the key-value tokens (lines with markers) are very similar to those measured using synthetic standard normal data (dotted lines). This suggests that NSN successfully aligns the output distribution with the standard normal distribution, and the codebook trained only on synthetic data effectively quantizes such outcomes. As result, codebook tuning improves perplexity from 5.294 to 5.285 with NSNQuant-2b and 6.910 to 6.703 with NSNQuant-Il lb.

---

Key
Value
2-bit FT
2-bit KM
Cosine similarity
1-bit FT
0.950
0.950
1-bit KM
2-bit FT (std)
0.83
0.83
2-bit KM (std)
1-bit FT (std)
1-bit KM (std)
0.62
0.62
10
15
20
25
30
5
20
15
20 25
30
Layer
Layer
Figure 5: Cosine similarity between original and reconstructed vectors when applying VQ to the NSN-processed keys and values LLaMAA-7B. Similarities are measured on WikiText-2 dataset with different codehooks. KM denotes the K-Means codebook, and FT denotes the fine-tune codebook. The lines with markers show cosine similarity measured using the key-value from the model, whereas the dotted lines show measurements using synthetic standard normal data.
Related work
Recent efforts reduce the memory and latency bottlenecks of LLMs have largely focused weight quantization and KV cache compression. Weight quantization methods such as GPTQ [12] and AWQ (27] significantly reduce the memory footprint model weights while preserving accuracy. In particular, vector quantization (VQ) methods like QulP# 37 AQLM and VPTQ [28] enable extremely low-bit quantization of model weights. However, these approaches do not mitigate the growing memory consumption of the KV cache, which scales linearly with sequence length during long-context inference.
To address this, recent studies have explored compressing the KV cache. KVQuant 19 and KIVI 30 observe the presence of outlier patterns in KV representations and propose apply channel-wise quantization for key and wise quantization for value. CQ 43 extends non-uniform quantization (nuq) multiple channels, achieving effective 1-bit KV cache compression. Orthogonal approaches, such as H2O 44 and SnapKV (25), reduce memory through cache eviction of uninformative tokens. Hybrid strategies like ZipCache [17 and MiKV 4T selectively maintain high-precision cache for important tokens while applying low-bit quantization elsewhere. Our method uses the same precision for all tokens, but future work could incorporate token importance in hybrid approaches.
Conclusion
In this work, we propose NSNQuant, calibration vector quantization (VQ) method for com- pressing cache of LLMs. NSNQuant effectively aligns the token distribution with the standard normal distribution through the NSN (Normalize-Shift-Noooe Noriit--No process, enabling the use of
the specialized codebook for the standard normal distribution. Through extensive experiments, we demonstrate that unlike calibration baased VQ methods, NSNQuant generalizes well across different tasks and datasets. In particular, NSNQuant excels in 1-bit quantization, outperforming the previous saate-of-the-add method by huge gap. We also implement efficient CUDA kernels for NSNQuant, and verify that NSNQuant achieves 3 speedup compared to the FP16 baseline.
Acknowledgements
This work was supported by SK Hynix, Samsung Advanced Institute of Technology (SAIT), IITP/RRF grants funded by the Korean government (MSIT, 2021 10-00005 Development of Model Compression Framework for Scalable On-Devic AI Computing on Edge Applications, 0021-0-01343 Artificial Intelligence Graduate School Program (Seoul National University). 2021-0-02068 Artificial Intel- ligence Innovation Hub), and Inter-university Semiconductor Research Center (ISRC), SNU. We are also grateful to VESSI. AT for providing the compute resources used for the experiments in this work.
10

---

References
[1] Josh Achiam, Steven Adler, Sandhini Agarwal, Lama Ahmad, Tlge Akkaya, Florencia Leoni Aleman, Diogo Almeida, Janko Altenschmidt, Sam Altman, Shyamal Anadkat, et al. Gpt-4 technical report. arXiv preprint arXiv:2303.08774 2023.
[2] Salch Ashkboos, Amirkeivan Mohtashami, Maximilian Croci, Bo Li. Pashmina Cameron, Martin Jaggi, Dan Alistarh, Torsten Hoefler, and James Hensman. Quarot: Outlier-free 4-bit inference in rotated Ilms, Advances in Neural Information Processing Systems, 37:100213: 100240, 2025.
[3] Yushi Bai, Xin Lv, Jiajie Zhang, Hongchang Lyu, Jiankai Tang, Zhidian Huang, Zhengxiao Du, Xiao Liu, Aohan Zeng. Lei Hou, Yuxiao Dong, Jie Tang, and Juanzi Li. LongBench: A bilingual, multitask benchmark for long context understanding. In Lun-Wei Ku, Andre Martins, and Vivek Srikumar, editors, Proceedings of the 62nd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Pupers), pages 3119-3137, Bangkok, Thailand, August 2024. Association for Computational Linguistics.
[4] Zefan Cai, Yichi Zhang, Bofei Gao, Yuliang Liu, Tianyu Liu, Keming Lu, Wayne Xiong. Yue Dong, Baobao Chang, Junjie Hu, et al. Pyramidkv: Dynamic kv cache compression based on pyramidal information funneling. arXiv preprint arXiv:2406. 02069, 2024.
[5] Chi-Chih Chang. Wei-Cheng Lin, Chien- Yu Lin, Chong- Man Chen, Yu-Fang Hu, Pei-Shuo Wang, Ning-Chi Huang, Luis Ceze, Mohamed S Abdelfattah, and Kai-Chiang Wu. Palu: Kv- cache compression with low-rank projection. In The Thirteenth International Conference on Learning Representations.
[6] Mark Chen, Jerry Tworek, Heewoo Jun, Qiming Yuan. Henrique Ponde de Oliveira Pinto, Jared Kaplan, Harri Edwards, Yuri Burda, Nicholas Joseph, Greg Brockman, Alex Ray, Raul Puri, Gretchen Krueger, Michael Petrov, Heidy Khlaaf, Girish Sastry, Pamela Mishkin, Brooke Chan, Scott Gray, Nick Ryder, Mikhail Pavlov, Alethea Power, Lukasz Kaiser, Mohammad Bavarian, Clemens Winter, Philippe Tillet, Felipe Petroski Such, Dave Cummings, Matthias Plappert, Fotios Chantzis, Elizabeth Barnes, Ariel Herbert-Voss, William Hebgen Guss, Alex Nichol, Alex Paino, Nikolas Tezak, Jie Tang, Igor Babuschkin, Suchir Balaji, Shantanu Jain, William Saunders, Christopher Hesse, Andrew N. Carr, Jan Leike, Josh Achiam, Vedant Misra, Evan Morikawa, Alec Radford, Matthew Knight, Miles Brundage, Mira Murati, Katic Mayer, Peter Welinder, Bob McGrew, Dario Amodei, Sam McCandlish, llya Sutskever, and Wojeiech Zaremba. Evaluating large language models trained on code. 2021.
[7] Mengzhao Chen, Yi Liu, Jiahao Wang, Yi Bin, Wenqi Shao, and Ping Luo. Prefixquant: Static quantization beats dynamic through prefixed outliers in llms. arXiv preprint arXiv:2410.05265. 2024.
[8] Karl Cobbe, Vineet Kosaraju, Mohammad Bavarian, Mark Chen, Heewoo Jun, Lukasz Kaiser, Matthias Plappert, Jerry Tworek, Jacob Hilton, Reiichiro Nakano, et al. Training verifiers to solve math word problems. arXiv preprint arXiv:21 10, 14/68, 2021.
[9] Nachshon Cohen, Amit Portnoy, Besnik Fetahu, and Amir Ingber. Sdr: Efficient neural re- ranking using succinet document representation. arXiv preprint arXiv:2110002065, 2021.
[10] Tim Dettmers, Artidoro Pagnoni, Ari Holtzman, and Luke Zettlemoyer. Qlora: Efficient finetuning of quantized Ilms, 2023, URL https://arxiv. 2rg/abw/2305.14314, 2, 2023.
[11] Vage Egiazarian, Andrei Panferov, Denis Kuznedelev, Elias Frantar, Artem Babenko, and Dan Alistarh. Extreme compression of large language models via additive quantization. arXiv preprint arXiv:2401.06// 2024.
[12] Elias Frantar, Salch Ashkboos, Torsten Hoefler, and Dan Alistarh. Gptq: Accurate post-training quantization for generative pre-trained transformers. arXiv preprint arXiv:2210.17323, 2022.
[13] Leo Gao, Jonathan Tow, Baber Abbasi, Stella Biderman, Sid Black, Anthony DiPofi, Charles Foster, Laurence Golding, Jeffrey Hsu, Alain Le Noac'h, Haonan Li, Kyle McDonell, Niklas Muennighoff, Chris Ociepa, Jason Phang, Laria Reynolds, Hailey Schoelkopf, Aviya Skowron,
11

---

Lintang Sutawika, Eric Tang, Anish Thite, Ben Wang, Kevin Wang, and Andy Zou. A framework for few-shot language model evaluation, 07 2024.
[14] Aaron Grattafiori, Abhimanyu Dubey, Abhinav Jauhri, Abhinav Pandey, Abhishek Kadian, Ahmad Al-Dahle, Aiesha Letman, Akhil Mathur, Alan Schelten, Alex Vaughan, et al. The llama
3 herd of models. arXiv preprint arXiv:240..21783, 2024.
[15] Daya Guo, Dejian Yang, Haowei Zhang, Junxiao Song, Ruoyu Zhang, Runxin Xu, Qihao Zhu, Shirong Ma, Peiyi Wang. Xiao Bi, et al. Deepseek-rl: Incentivizing reasoning capability in Ilms via reinforcement learning. arXiv preprint arXiv:2501.12945 2025.
[16] Nathan Halko, Per-Gunnar Martinsson, and Joel A Tropp. Finding structure with randomness: Probabilistic algorithms for constructing approximate matrix decompositions. SIAM review, 53(2):217-288, 2011.
[17] Yefei He, Luoming Zhang, Weijia Wu, Jing Liu, Hong Zhou, and Bohan Zhuang. Zipcache: Accurate and efficient kv cache quantization with salient token identification. arXiv preprint arXiv:2405. 14256, 2024.
[18] Dan Hendrycks, Collin Burns, Steven Basart, Andy Zou, Mantas Mazeika, Dawn Song, and Jacob Steinhardt. Measuring massive multitask language understanding. arXiv preprint wrXiv:2009.03300 2020.
[19] Coleman Hooper, Sehoon Kim, Hiva Mohammadzadeh, Michael W Mahoney, Sophia Shao, Kurt Keutzer, and Amir Gholami. Kvquant: Towards 10 million context length Ilm inference with kv cache quantization. Advances in Neural Information Processing Systems, 37:1270-1300 2024.
[20] Xing Hu, Yuan Cheng, Dawei Yang, Zukang Xu, Zhihang Yuan, Jiangyong Yu, Chen Xu, Zhe Jiang, and Sifan Zhou. Ostquant: Refining large language model quantization with orthogonal and scaling transformations for better distribution fitting. arXiv preprint arXiv:2501. 13987, 2025.
[21] Albert Q. Jiang, Alexandre Sablayrolles, Arthur Mensch, Chris Bamford, Devendra Singh Chaplot, Diego de las Casas, Florian Bressand, Gianna Lengyel, Guillaume Lample, Lucile Saulnier, Lélio Renard Lavaud, Marie-Anne Lachaux, Pierre Stock, Teven Le Scao, Thibaut Lavril, Thomas Wang, Timothée Lacroix, and William El Sayed. Mistral 7b, 2023.
[22] Sehoon Kim, Coleman Hooper, Amir Gholami, Zhen Dong, Xiuyu Li, Sheng Shen, Michael W Mahoney, and Kurt Keutzer. Squeezellm: Dense-and-see sparse quantization. arXiv preprint 1.Xiv:2306.07629, 2023.
[23] Woosuk Kwon, Zhuohan Li, Siyuan Zhuang, Ying Sheng, Lianmin Zheng, Cody Hao Yu, Joseph Gonzalez, Hao Zhang, and Ion Stoica. Efficient memory management for large language model serving with pagedattention. In Proceedings of the 29th Symposium on Operating Systems Principles, pages 61 62666, 2023.
[24] Patrick Lewis, Ethan Perez, Aleksandra Piktus, Fabio Petroni, Vladimir Karpukhin, Naman Goyal, Heinrich Küttler, Mike Lewis, Wen-tau Yih, Tim Rocktäschel, et al. auurenedl-augmented generation for knowledge-intensvee intensive nip tasks. Advances in neural information processing systems, 33:9459- 9474, 2020.
[25] Yuhong Li, Yingbing Huang, Bowen Yang, Bharat Venkitesh, Acyr Locatelli, Hanchen Ye, Tianle Cai, Patrick Lewis, and Deming Chen. Snapky: Llm knows what you are looking for before generation. Advances Neural Information Processing Systems, 37:22947-22970, 2024.
[26) Bokai Lin, Zihao Zeng, Zipeng Xiao, Siqi Kou, Tianqi Hou, Xiaofeng Gao, Hao Zhang, and Zhijie Deng. Matryoshkakv: Adaptive kv compression via trainable orthogonal projection. arXiv preprint M.XX::2410.143311 2024.
[27] Lin, Jiaming Tang, Haotian Tang, Shang Yang, Wei-Ming Chen, Wei-Chen Wang, Guangxuan Xiao, Xingyu Dang, Chuang Gan, and Song Han. Awq: Activation-aware weight quantization for on-device IIm compression and acceleration. Proceedings of Machine Learning and Systems, 6:87-100, 2024.
12

---

[28] Yifei Liu, Jicheng Wen, Yang Wang, Shengyu Ye, Li Lyna Zhang, Ting Cao, Cheng Li, and Mao Yang. Vptq: Extreme low-bit vector post-training quantization for large language models. arXiv preprini arXiv:2409. 0000997770000 /7066, 2024.
[29] Zechun Liu, Changsheng Zhao, Igor Fedorov, Bilge Soran, Dhruv Choudhary, Raghuraman Krishnamoorthi, Vikas Chandra, Yuandong Tian, and Tijmen Blankevoort. Spinquant: Llm quantization with learned rotations. arXiv preprint arXiv:2405. 16406, 2024.
[30] Zirui Liu, Jiayi Yuan, Hongye Jin, Shaochen Zhong, Zhaozhuo Xu, Vladimir Braverman, Beidi Chen, and Xia Hu. Kivi: A tuning-free asymmetric 2bit quantization for kv cache. arXiv preprint arXiv:2402.02750, 2024.
[31] Meta. ott. -aaaamtttaaaaaaa11aaa1-- 2024.
meta.
[32] Ziran Qin, Yuchen Cao, Mingbao Lin, Wen Hu, Shixuan Fan, Ke Cheng, Weiyao Lin, and Jianguo Li. Cake: Cascading and adaptive kv cache eviction with layer preferences. arXiv preprint arXiv:2503. 2025.
[33] Siva Reddy, Danqi Chen, and Christopher D) Manning. Coqa: A conversational question answering challenge. Transactions of the Association for Computational Linguistics, 7:249-266, 2019.
[34] Mark Rudelson and Roman Vershynin. Hamson-wright inequality and sub- sub-gaussian concentra- tion. 2013.
[35] ShareGPT Team. https://sharegpt.com
2023.
[36] Hugo Touvron, Louis Martin, Kevin Stone, Peter Albert, Amjad Almahairi, Yasmine Babaci, Nikolay Bashlykov, Soumya Batra, Prajjwal Bhargava, Shruti Bhosale, et al. Llama 2: Open foundation and fine-tuned chat models. arXiv preprint arXiv:2307.00 092888, 2023.
[37] Albert Tseng, Jerry Chee, Qingyao Sun, Volodymyr Kuleshov, and Christopher De Sa. QulP#: Even better LLM quantization with hadamard incoherence and lattice codebooks. In Ruslan Salakhutdinov, Zico Kolter, Katherine Heller, Adrian Weller, Nuria Oliver, Jonathan Scarlett, and Felix Berkenkamp, editors, Proceedings of the 41st International Conference on Machine Learning, volume 235 of Proceedings of Machine Learning Research, pages 48630-48656. PMLR, 21-27 Jul 2024.
[38] Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Lukasz Kaiser, and Illia Polosukhin. Attention all you need. Advances in neural information processing systems, 30, 2017.
[39] Hongyu Wang, Shuming Ma, and Furu Wei. Bitnet v2: Native 4-bit activations with hadamard transformation for 1-bit Ilms. arXiv preprint urXiv:2504.18415 2025.
[40] Jason Wei, Xuezhi Wang, Dale Schuurmans, Maarten Bosma, Fei Xia, Ed Chi, Quoe V Le, Denny Zhou, et al. Chain-of-thought prompting elicits reasoning in large language models. Advances in neural information processing systems, 15:24824-24482 2022.
[41] June Yong Yang, Byeongwook Kim, Jeongin Bae, Beomseok Kwon, Gunho Park, Eunho Yang, Se Jung Kwon, and Dongsoo Lee. No token left behind: Reliable kv cache compression importance-awant mixed precision quantization. arXiv preprint arXiv:2402. 2024.
[42] Zhihang Yuan, Jiawei Liu, Jiaxiang Wu, Dawei Yang, Qiang Wu, Guangyu Sun, Wenyu Liu, Xinggang Wang, and Bingzhe Wu. Benchmarking the reliability of gost-training quantization: particular focus on worst-case performance. arXiv preprint arXiv:2303. 13003, 2023.
[43] Tianyi Zhang, Jonah Yi, Zhaozhuo Xu, and Anshumali Shrivastava. Kv cache is 1 bit per channel: Efficient large language model inference with coupled quantization. Advances in Neural Information Processing Systems, 37:3304-3331, 2024.
[44] Zhenyu Zhang, Ying Sheng, Tianyi Zhou, Tianlong Chen, Lianmin Zheng, Ruisi Cai, Zhao Song, Yuandong Tian, Christopher Ré, Clark Barrett, et al. H2o: Heavy-hitter oracle for efficient generative inference of large language models. Advances in Neural Information Processing Systems, 36:34661-3366666 -347100, 2023.
13

---

NeurIPS Paper Checklist
1. Claims
Question: Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?
Answer: [Yes]
Justification: Our observations and technical contributions are clearly introduced in the
"Method" section. "Experiment" and "Ablation studies" sections support major claims effectively.
Guidelines:
The answer NA means that the abstract and introduction do not include the claims made the paper.
The abstract and/or introduction should clearly state the claims made, including the contributions made in the paper and important assumptions and limitations. A No or
NA answer to this question will not be perceived well by the reviewers.
The claims made should match theoretical and experimental results, and reflect how much the results be expected to generalize other settings.
is fine include aspirational goals as motivation long is clear that these goals are not attained by the paper.
2. Limitations
Question:
Does the paper discuss the limitations of the work performed by the authors?
Answer: [Yes]
We disclose our limitations "Method" and "Limitation and broader impacts"" sections.
Justification:
Guidelines:
The answer NA means that the paper has limitation while the answer No means that the paper has limitations, but those are not discussed in the paper.
The authors are encouraged to create separate "Limitations" section in their paper.
The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (e.g., independence assumptions, noiseless settings, model well-specification. asymptotic approximations only holding locally). The authors should reflect on how these assumptions might be violated in practice and what the implications would be.
The authors should reflect the scope of the claims made, e.g., if the approach was only tested on a few datasets or with few runs. In general, empirical results often depend on implicit assumptions, which should be articulated.
The authors should reflect the factors that influence the performance of the approach. For example, facial recognition algorithm may perform poorly when image resolution low or images are taken in low lighting. Or speech-to-text system might not be used reliably to provide closed captions for online lectures because it fails to handle technical jargon.
The authors should discuss the computational efficiency the proposed algorithms and how they scale with dataset size.
If applicable, the authors should discuss possible limitations of their approach to address problems of privacy and fairness.
While the authors might fear that complete honesty about limitations might used by reviewers as grounds for rejection, worse outcome might be that reviewers discover limitations that aren't 't acknowledged in the paper. The authors should use their best judgment and recognize that individual actions in favor of transparency play impor- tant role in developing norms that preserve the integrity of the community. Reviewers will be specifically instructed to not penalize honesty concerning limitations.
3. Theory assumptions and prool's
Question: For each theoretical result, does the paper provide the full of assumptions and complete (and correct) proof?
14

---

Answer: [Yes]
Justification: We clearly state assumptions in "Lemma I" and provide proofs in "Proof for variance bounds of NSN" section.
Guidelines:
The answer NA means that the paper does not include theoretical results.
All the theorems, formulas, and proofs in the paper should be numbered and cross- referenced.
All assumptions should be clearly stated or referenced in the statement of any theorems.
The proofs can either appear in the main paper or the supplemental material, but if they appear in the supplemental material, the authors are encouraged to provide a short proof sketch to provide intuition.
Inversely, any informal proof provided in the core of the paper should be complemented by formal proofs provided in appendix or supplemental material.
Theorems and Lemmas that the proof relies upon should be properly referenced.
4. Experimental result reproducibility
Question: Does the paper fully disclose all the information needed to reproduce the main ex- perimental results of the paper to the extent that affects the main claims and/or conclusions of the paper (regardless whether the code and data are provided or not)?
Answer: [Yes]
Justification: We describe all the components included in our method in "Method" section. We provide exact formulas to compute attention scores and outputs in the main figure and "Attention computation NSN" section.
Guidelines:
The answer NA means that the paper does not include experiments.
If the paper includes experiments, No answer this question will not be perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data are provided or not.
If the contribution is dataset and/or model, the authors should describe the steps taken to make their results reproducible or verifiable.
Depending on the contribution, reproducibility can be accomplished in various ways. For example, if the contribution is a novel architecture, describing the architecture fully might suffice, or if the contribution is specific model and empirical evaluation, it may be necessary to either make it possible for others to replicate the model with the same dataset, or provide access the model. In general. releasing code and data is often one good way to accomplish this, but reproducibility can also be provided via detailed instructions for how replicate the results, access a hosted model (e.g.. in the of large language model), releasing of a model checkpoint, or other means that are appropriate to the research performed.
While NeurIPS does not require releasing code, the conference does require submis- sions to provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution. For example
(a) the contribution is primarily a new algorithm, the paper should make clear how reproduce that algorithm. (b) If the contribution is primarily
model architecture, the paper should describe the architecture clearly and fully.
(c) the contribution is a new model (e.g., a large language model), then there should either be way to access this model for reproducing the results or way to reproduce the model (e.g., with an open-source dataset or instructions for how to construct the dataset).
(d) We recognize that reproducibility may be tricky in some cases, in which case authors are welcome describe the particular way they provide for reproducibility. In the case of closed-source models, it may be that access to the model is limited in some way (e.g., to registered users), but it should possible for other researchers to have some path to reproducing verifying the results.
15

---

5. Open access to data and code
Does the paper provide open access to the data and code, with sufficient instruc- tions to faithfully reproduce the main experimental results, as described in supplemental material?
Question:
Answer: [Yes]
Justification: Our implementation is included in the Supplementary material in an anonymized form. We plan to release it publicly upon acceptance.
Guidelines:
The answer NA means that paper does not include experiments requiring code.
Please see the NeurlPS code and data submission guidelines (https://nips.co cc/
public//guiee//ooeeSutiin www.aaaboiiiiioooooooooooooo. for more details.
While we encourage the release of code and data, we understand that this might not be possible, "No" an acceptable answer. Papers cannot be rejected simply for including code, unless this is central to the contribution (e.g., for new open-source benchmark).
The instructions should contain the exact command and environment needed to run to reproduce the results. See the NeurlPS code and data submission guidelines (https for more details.
//nips //iiss.c/ppul
guides/ ProddeSubmissiooool....
The authors should provide instructions on data access and preparation, including how access the raw data, preprocessed data, intermediate data, and generated data, etc.
The authors should provide scripts to reproduce all experimental results for the new proposed method and baselines. only subset of experiments are reproducible, they should state which ones are omitted from the script and why.
At submission time, to preserve anonymity, the authors should release anonymized versions (if applicable).
Providing as much information as possible in supplemental material (appended to the paper) recommended, but including URLs to data and code permitted.
6. Experimental setting/details
Does the paper specify all the training and test details (e.g., data splits, hyper- parameters, how they were chosen, type of optimizer, etc.) necessary to understand the results?
Question:
Answer: [Yes]
Justification: We provide the detailed experimental settings in the "Experiments" and "Experiment details" sections. We clearly specify all the datasets used for visualization.
Guidelines:
The answer NA means that the paper does not include experiments.
The experimental setting should be presented in the core the paper to a level of detail that is necessary to appreciate the results and make sense of them.
The full details can be provided either with the code, appendix, as supplemental material.
7. Experiment statistical significance
Question: Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?
Answer: [No]
Justification:
We only provide the metrics without error bars, following the conventions.
Guidelines:
The answer NA means that the paper does not include experiments.
The authors should answer "Yes" if the results are accompanied by emoor bars, confi- dence intervals, or statistical significance tests, least for the experiments that support the main claims of the paper.
16

---

The factors of variability that the error bars are capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, or overall run with given experimental conditions).
The method for calculating the error bars should be explained (closed form formula, call to a library function, bootstrap, etc.)
The assumptions made should be given (e.g., Normally distributed errors).
It should be clear whether the error bar is the standard deviation the standard error of the mean.
is OK to report 1-sigma error bars, but one should state it. The authors should preferably report a 2-sigma error bar than state that they have 96% CI, if the hypothesis of Normality of errors is not verified.
For asymmetric distributions, the authors should be careful not to show in tables or figures symmetric error bars that would yield results that are out range (e.g. negative error rates).
If error bars reported in tables or plots, The authors should explain in the text how they were calculated and reference the corresponding figures or tables in the text.
8. Experiments compute resources
For each experiment, does the paper provide sufficient information on the com- puter resources (type of compute workers, memory, time of execution) needed to reproduce the experiments?
Question:
Answer: [Yes]
Justification: We provide information regarding GPU resources in the "Experiment" section, and report latency in the "Latency breakdown" section.
Guidelines:
The answer NA means that the paper does include experiments.
The paper should indicate the type of compute workers CPU or GPU, internal cluster, cloud provider, including relevant memory and storage.
The paper should provide the amount of compute required for each of the individual experimental rums as well as estimate the total compute.
The paper should disclose whether the full research project required more compute than the experiments reported in the paper (e.g., preliminary or failed experiments that didn't fidn't make it into the paper).
9. Code of ethics
Question: Does the research conducted in the paper conform, in every respect, with the NeurIPS Code of Ethics httpp/////eerrp...ccppbbbbll//// (etb
Answer: [Yes]
Justification: We have carefully reviewed the NeurIPS Code of Ethics and confirm that our research fully adheres to its principles.
Guidelines:
The answer NA means that the authors have reviewed the NeurIPS Code of Ethics.
the authors answer No, they should explain the special circumstances that require deviation from the Code of Ethics.
The authors should make to preserve anonymity (e.g., if there is special consid- eration due to laws or regulations their jurisdiction).
10. Broader impacts
Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?
Answer: [Yes]
Justification:
We discuss potential socictal impacts in the "Limitation and broader impacts"
section.
Guidelines:
17

---

The answer NA means that there is no societal impact of the work performed.
If the authors answer NA or No. they should explain why their work has no societal impact or why the paper does not address societal impact.
Examples of negative societal impacts include potential malicious or unintended uses (e.g., disinformation, generating fake profiles, surveillance), fairness considerations (e.g., deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations.
The conference expects that many papers will be foundational research and not tied to particular applications, let alone deployments. However, if there is direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that an improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point out that a generic algorithm for optimizing neural networks could enable people to train models that generate Deepfakes faster.
The authors should consider possible harms that could arise when the technology is being used intended and functioning correctly, harms that could arise when the technology is being used as intended but gives incorrect results, and harms following from (intentional unintentional) misuse of the technology.
there are negative societal impacts, the authors could also discuss possible mitigation strategies (e.g.. gated release of models, providing defenses in addition to attacks, mechanisms for monitoring misuse, mechanisms to monitor how a system learns from feedback over time, improving the efficiency and accessibility of ML).
11. Safeguards
Does the paper describe safeguards that have been put in place for responsible release of data models that have high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)?
Question:
Answer: [NA]
Justification:
We do release new models datasets.
Guidelines:
The answer NA means that the paper poses no such risks.
Released models that have high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere usage guidelines or restrictions to access the model or implementing safety filters.
Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images.
We recognize that providing effective safeguards challenging, and many papers do not require this, but we encourage authors to take this into account and make best faith effort.
12. Licenses for existing assets
Question: Are the creators or original owners of assets (e.g., code, data, models), used in the paper, properly credited and are the license and terms of use explicitly mentioned and properly respected?
Answer: [Yes]
Justification:
We include proper citations of the models and datasets we used.
Guidelines:
The answer NA means that the paper does not use existing assets.
The authors should cite the original paper that produced the code package or dataset.
The authors should state which version of the asset used and, possible, include a URL
The name of the license (e.g., CC-BY 4.0) should be included for each asset.
For scraped data from a particular source (e.g., website), the copyright and terms of service of that source should be provided.
18

---

If assets are released, the license, copyright information, and terms of use in the thcode has curated licenses for some datasets. Their licensing guide can help determine the license of dataset.
package should be provided. For popular datasets, paperswi
latasets.paperssit
For existing datasets that are re-packaged, both the original license and the license of the derived asset (if it has changed) should be provided.
If this information not available online, the authors are encouraged to reach out to the asset's s creators.
13. New assets
Question: Are new assets introduced in the paper well documented and is the documentation provided alongside the assets?
Answer: [NA]
Justification:
We do not release new assets.
Guidelines:
The answer NA means that the paper does not release new assets.
Researchers should communicate the details of the dataset/code/moode as part of their submissions via structured templates. This includes details about training, license, limitations, etc.
The paper should discuss whether and how consent obtained from people whose asset is used.
At submission time, remember to anonymize your assets (if applicable). You can either create an anonymized URL or include an anonymized zip file.
14. Crowdsoureing and research with human subjects
Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given to participants and screenshots, if applicable, as well details about compensation (if any)?
Answer: [NA]
Justification: This paper does not involve human subjects in the research.
Guidelines:
The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.
Including this information the supplemental material is fine, but if the main contribu- tion of the paper involves human subjects, then as much detail as possible should be included in the main paper.
According to the NeurIPS Code of Ethics, workers involved in data collection, curation, or other labor should be paid at least the minimum wage in the country of the data collector.
15. Institutional review board (IRB) approvals or equivalent for research with human subjects
Question: Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or equivalent approval/review based the requirements of your country institution) were obtained?
Answer: [NA]
Justification: This paper does not involve human subjects in the research.
Guidelines:
The answer NA means that the paper does not involve crowdsourcing research with human subjects.
Depending the country in which research is conducted, IRB approval (or equivalent) may be required for any human subjects research. If you obtained IRB approval, you should clearly state this in the paper.
19

---

We recognize that the procedures for this may vary significantly between institutions and locations, and we expect authors adhere the NeurIPS Code of Ethics and the guidelines for their institution.
For initial submissions, do not include any information that would break anonymity (if applicable), such as the institution conducting the review.
16. Declaration of LLM usage
Does the paper describe the usage of LLMs if it is an important, original, or non-standard component of the core methods in this research? Note that if the LLM is used only for writing, editing, formatting purposes and does not impact the core methodology. scientific rigorousness, or originality of the research, declaration is not required.
Question:
Answer: [NA]
Justification: LLMs are not used for the core method development of this research.
Guidelines:
The answer NA means that the method development in this research does not involve LLMs as any important, original, or non-standard components.
Please refer to our LLM policy (https: /ttprii///eurps... http:///rrrrrrec/ooooe/reree//// Cc/Comferrecces/2255//1. for what should or should be described.
20

---

A Proof for variance bounds of NSN
A.1 Proof of Lemma
1 d
Notation. Write b [[X;] and
Σ H Σ
Decompose Cov(X) D A.
D diag(E), An O, |||l S.
Expected diagonal term. Because 2. d,
d Tr(D) 1 Σ
(Var(X.)) = -1-8. =1-6
==1
3. Quadratic form. Choose Hadamard row h = 1 va SE {{t]dd Then 1
h Dh 1- f(h) == h Ah 7
and Var(Y.) (1- B[s(h)] 0 since in randomized with the equal probability of 1/2.
4. Hanson-Wright Since s is Rademacher vector, it is a sub-gaussian vector. Applying the Hanson- Wright Haihhnn-rright inequality [34], for any u
Pr(1s As| > u) s 2 exp( /
where e is a universal constant. Put dt; then
Pr([((h) t) s 2 exp -cd
Tail parameter: Choose t = 13.. 8n In(2/a)/c. Exponent becomes -In(2/a); hence Pr(]f(h)) > t) Sa.
Combine. Since 0 SESE E, with probability at least 1 - a
Var(Y.) [1 - PBn, + 13a.
A.2 Off-diagonal Frobenius norms of covariance
To obtain insights regarding the bounds from Lemma 1 we measure the layer-wise average off- diagonal Frobenius norms of covariance matrixs. For key, the covariance computed right before the Hadamard transform. For value, we remove fused Hadamard transform the value projection matrix and compute covariance right after the NSN. Note again that the order between adjacent NSN and Hadamard transform is interchangable. The result is shown in Figure 6 The Frobenius norm is large in the first few layers for both key and value, and remains low in the later layers. This finding is consistent with the pattern observed in Figure 12 and 13, where standardization fails in the early layers. The reason for the large Frobenius norm the first layers is visualized in Figure 7 For both key and value, there exist outlier values in covariance matrices, leading to the large Frobenius norm.
B Attention computation in NSN
NSNQuant computes the attention weights and outputs using the byproducts from the NSN process. First, the dot product between the query and key computed follows:
qK- (ARRoPE(KproRoraet
(RoPE (sf (sf + +
(sfa RoPE + RoPE(o*))
s (HTT(OO(()))))) + ((((ROP(()'))"
S + (R
21

---

Off-diagonal Frobenius Norm of Covariance per Layer
Key
35
Value
[Cov -dlag(Cov(())
30
25-
20
15
10:
Layer
Figure 6: Average off-diagonal Frobenius norm of covariance matrix. The results are measured with LLaMA2-7B on WikiText-2.
(Key) Covoriance Range
Covariroe
21
13
15
(Value) Covariance Range per Head
Cousrance
Figure 7: The min-max range of off-diagonal covariances of keys (top) and values (bottom) in the first layer of LLaMA2-7B, For both key and value, some channels suffer from outliers covariance matrices. We use the first sample from the WikiText-2 for visualization.
Here, HT(-) is the Hadamard transform, and v == CIVQ(HT
Mad HT(g).
Since ot has a different length from K, we expand within a kernel to match the shape. We obtain attention weights through W softmax(g and then compute the outputs follows:
We W s us + + ~ W Ws + +²),, va == C[VQ(e,
C
Additional ablation studies
C.1 Effects of using randomized Hadamard transform (RHT)
From Lemma we find that adopting randomized Hadamard transform (RHT) after the NSN process gives theoretical bounds to the variance of each channel. However, since the covariances between channels tend to have uniform signs, we find that using the naive Hadamard transform works well enough. As shown in T both transforms give similar results. Since RHT needs more parameters and computations, we use the naive Hadamard transform in the final design.
Table 7: Perplexity of LLaMA2-7B on WikiText-2 and C4 with NSNQuant-2b using different types of Hadamard transform. We follow the settings from the main experiments for WikiText-2 and C4. For LCC and SAMSum, we truncate each data sample to 4096 tokens if needed.
Method
WikiText-2
C4
LCC
SAMSum
Hadamard transform
5.285
6,856
2.128
6.466
RHT
5.290
6.854
2.128
6.461
22

---

C.2
Effects of residual size
70
3000
Throughput (takensjis)
60
2500
Memory (GB)
50
2000
FP16
40
NSNQQant-20 (AS=32)
1500
NSNQuant-2b (R5=32)
NSNQuant-2b (R5-64)
NSNGuant-2b (RS-64)
30
NSNQuant-25 (RS-128)
1000
NSMQuant-2b (RS-128)
NSNQuant-10 (RS-32)
NSNQuant-1b (RS-32)
20
NSNQuant-1c (RS-64)
500
NSNQUant-1b IRS-64)
NSNQuant-1b (RS=128)
NSNQuant-1e (RS=128)
10
1000
400
1400
Batch Size
Batch Size
Figure 8: Peak memory usage (left) and throughput (right) measured with varying batch sizes and
residual sizes
Table 8: Evaluation results on GSM8K, HumanEval, CoQA, and MMLU with LLaMA3.1-8B Instruct..
refers to residual size.
MMLU (4-shot, CoT)
Method
Bits
GSM8K (8-hot, CoT)
HumanEval
CoQA
Humanities
STEM
Social
Other
NSNQuant-20 (RS=32)
2.30
74.90
53.05
70.24
56.56
74.22
70.98
NSNQuant-2b (RS=64)
2.23
75.89
56.10
63.83
71.04
55.64
73.42
70.74
NSNQuant-2b (RS=128)
2.19
75.66
56.71
62.93
69.29
$7.38
74.49
70.80
NSNQuant-1b lb (RS=32)
1,30
42.08
42.07
63.67
59.47
44.88
65.95
63,42
NSNQuant-16 (RS=64)
1.23
53.45
44.51
62,70
59.82
45.83
65.34
63.77
NSNQuant-16 (RS=128) 128)
1.19
67.63
We fix the residual size to in our main experiments, for fair evaluation across different methods. However, the residual size is important hyperparameter that determines the number of full precision caches. Therefore, we evaluate NSNQuant with different residual sizes (32, 64, 128) with CAMMA3.--B-IIntruuee Since LongBench is not effective show performance differences in 2-bit regime, we use GSM8K, HumanEval, and CoQA. The result is shown in Table 8 For NSNQuant- lb, observe that performance is highly sensitive to the choice of residual size. A residual size of 128 yields the best results across most benchmarks, while a residual size 32 performs the worst. In contrast, NSNQuant-26 shows comparable performance with different residual sizes. These results suggest that NSNQuant-2b produces higher-quality quantizations, making it more robust to variations in residual size.
also report the memory usage and throughput with different residual sizes in Figure S The result shows that the methods with smaller residual sizes require less memory and achieve higher throughput. Note that the impact of residual size on memory and throughput will decrease as the sequence length gets longer.
C.3 Effects of scale adjustment
Motivating Example Suppose we are quantizing a 2-dimensional vector (1, 2) using a codebook with entries: (0.8, 1.6) and (2, 3). The vector would be approximated by (0.8, 1.6), incurring a non-negligible error. However, this error could be significantly reduced by scaling the quantized vector by a factor of 1.25.
As this example illustrates, using the quantized vector eq without any scaling leads to suboptimal approximation. To identify more effective scaling strategy, we evaluate three probable approaches, as visualized in Figure The first approach to scale vn to minimize the 1.2 error between e and vQ. the second approach is to scale eq match the size of v. and the third approach is to scale no to preserve components parallel v. The result of applying three strategies is presented in Table 9 For both key and value, strategy 3 shows the best quantization quality. By applying it to both key and
23

---

81Vo
Or Vo
03Vo
V
scaling: suboptimal
(1) (v a:Vo) Vo
(2) ||ay'l |v]l
(3) -:va)
Figure 9: Visualization of the three tested scale adjustment strategies.
Table 9: Perplexity of LLaMA2-7B WikiText-2 under different scaling strategies for key and value. NSNQuant- -b is used here for ablation.
Strategy
PPL
No scaling
5.395
Key Strategy 1
5.378
Key Strategy 2
5,329
Key Strategy 3
5,317
Value Strategy 1
5,394
Value Strategy 2
5.355
Value Strategy 3
5.335
Key Strategy 3 Value Strategy 5.285
value in NSNQuant-2b, the perplexity of LLLMMA-7B on WikiTexx-2 is reduced from 5.395 to 5.285.
C.4 Effects of codebook tuning
Table 10: Perplexity of LLaMA2-7B on WikiText-2 using different codebooks.
Method
Codebook
PPL
K-Means
5.294
NSNQuant-2b
E8P
5.284
K-Means + Fine-tuning
5.285
K- M-Mans
6.910
NSNQuant-Ib
K-Means + Fine-tuning
6.703
As mentioned in section 3.4 algorithm. Since the error of scale adjustment only depends on the angle between v and uq. we set the objective to minimize the cosine distance between them. The perplexity result with LLaMA2-7B on WikiText-2 presented Table T0 The result shows that fine-tuning improves quantization quality in both NSNQuant- 1b and NSNQuant-22 2b, and matches the performance of ESP 37 in 2-bit quantization.
we propose to fine-tune a codebook obtained from the K-Means
C.5 Effects of pre-RoPE NSN
As visualized in Figure 2 NSNQuant applies NSN to keys before the RoPE. However, it seems natural to apply NSN after RoPE since RoPE may induce noises the channel -iise mean. The PPL evaluation results in both settings are presented in Table II] The Pre-RoPE NSN achieves slightly lower PPL than RoPE NSN. We attribute this to two reasons. First, the effect RoPE on the channel-wise mean is minimal, as the tokens within the residual share similar rotation angles. Second, computing full-precision RoPE for o is beneficial, since the RoPE rotation matrix contains important position information.

---

Table 11: Perplexity of LLaMA2-74 and LLaMA2-13B on WikiText-2 with NSNQuant-22 using different locations of NSN.
Normalization
LLaMA2-7B
LLaMA2-13B
pre-RoPE NSN
5.29
4.71
post- NSN
5.33
4.73
C.6 Effeets of double quantization
Table T2 shows the performance change when applying double quantization. The performance degradation minimal, while the memory savings substantial.
Table 12: Impact of double quantization the perplexity of LLaMA2-7B WikiText-2, using NSNQuant-2b for ablation.
Method
Avg. bit width
PPL
Baseline
2.5
5.278
Codebook Double Quantization
2.5
5.280
+ Mean Double Quantization
2.33
5.287
Scale Double Quantization
2.23
5.285
C.7 Scaling calibration set for CQ
Since CQ uses only very small dataset of 16 samples, natural question arises: CQ be further improved to outperform NSNQuant by scaling its calibration set? To answer this question, we test variants of CQ where the size of the calibration set doubled 32. For the first variant, use 32 samples from WikiText-2 (W32), enlarging the size of the calibration set. For the other variant, we use 16 samples from WikiText-2 and another 16 samples from C4 (W16C16). This variant not only increases the size but diversifies the calibration set, covering wider range of input distribution. We use LLaMA3-8B-Inse
model test performance on the three downstream tasks: HumanEval, GSM8K, and CoQA. We denote two variants as W32 and W W16C16, respectively.
The result shown in Table 13 While W32 shows only marginal improvement from the baseline, W16C16 provides clearer benefits. It suggests that to improve the robustness of CQ, is important to diversify the calibration set, rather than just scaling it up. In addition, despite improved performance, NSNQuant still shows better results. For example, on HumanEval, CQ-8c1 10b still suffers from significant performance degradation since its input distribution (i.e. codes) is not covered by either WikiText-2 or C4.
We believe that further scaling of the calibration set of CQ will improve its generalization ability. However, since CQ needs gradient computation, the calibration process incurs comparable cost to fine-tuning, the calibration set grows. Moreover, it's challenging to prepare the calibration set that adequately covers the full diversity of real-world input distributions (e.g. different languages). Therefore, we believe that NSNQuant remains highly practical and robust option in general scenarios, especially when broad generalization is required
Table 13: Performance comparison of NSNQuant and CQ baselines on HumanEval, GSM8K (8-shot, CoTT), and CoQA.
Method
HumanEval GSM8K (8-shot, CoT)
CoQA
FP16
28.05
76.62
61.47
NSNQuant-2h
32,93
74,53
61.62
CQ-4c9b
23.78
72.55
60.60
CQ-4c9b (W32)
25.00
72.93
60.38
CQ-4c9b (W16C16)
29.27
73.16
60.98
NSNQuant-1b
29.27
61.64
60.28
CQ-8c10h
13.41
24.26
53.38
CQ-8c10b (W32)
15.24
21.76
53.37
CQ-8c10b (W16C16)
12.20
46.47
54.07
25

---

D Variants of NSN
D.I Replacing the Shift step with Weiszfeld algorithm
As noted in Section 3.2 the final normalization step of NSN can introduce a small bias, leaving the channel-wise mean slightly off zero. To verify that it barely affects the quantization quality, we provide mathematical explanation and an experiment where we totally remove such bias by adopting Weiszfeld algorithm.
First, here is an intuitive explanation of why the small bias introduced by the last step of NSN does not significantly affect the quantization error. Let Un = {U1, 2/2
( ..., 2/ E I denote the output tokens after the first normalization step. Since the next step, Shift, subtracts the mean,
have
Uns = {or - E Elus], 222 - Elv;],-.. ..l. f2 - E[us]]- (i) If Var[us] is small, then E((u; - E[ux]) []] is also small. Since save o E[ls]] in full-prrc,s,,n, the fact that the norm of the leftover part (wy- Ele:)) is small implies that the final error will also be small. (ii) If Var [ys] is large, then [[vs] will be small because EE[r,]2= [[ - Var[u,]
Var[u;]. Small E[u;] means that the token-wise norm will not change much, and the last step will not significantly affect the channel-wise mean. To conclude,
regardless of the magnitude of Var[u:]. the final quantization error would remain small because either the alignment is well preserved, or the leftover magnitude is small.
Second, to empirically explore the impact of removing this drift, we reformulate the second step (Shift) as the search for the geometric median. Let {4)]21 C Rd denote the tokens in group of residual size after the first normalization step. We seek O₂ such that
((0)
|1 oll 0.
Setting = oll gives (F))) = L(): hence any satisfying the geometric median of the set (f,).
2 minimizes F and
We compute o, with the Weiszfeld algorithm-the standard iterative solver for geometric medi- ans-and subtract it instead of the arithmetic mean in NSN's second step. The resulting variant keeps the post-normalizztion mean essentially at zero. Table 14 presents the perplexity evaluation results on WikiText-2 and C4 with ALaMA2-7B. The change yields only marginal negative effects despite the additional cost from the iterative updates. Hence, we keep the original mean- meeansubbraattio step in the final NSN design.
Table 14: Perplexity of LLaMA2-7B on WikiText-2 and C4 with NSNQuant-2b using different normalization strategies.
Normalization
WikiText-2
C4
NSN
5.285
6.856
N- N-Weiszfeld-N
5.289
6.863
D.2 Replacing the second Normalization step with channel-wise scaling
straightforward way to standardize the output distribution is to simply shift and scale in a channel dimension. Therefore, we try replacing our third step-token-wise normalization-viin channel-wise scaling, where sample standard deviation is divided from each channel. We also move the location of normalizations next the Hadamard transform so that it does not affect channel-wise statistics. The result is shown in Table 15 Using channel-wist scaling instead of token- wise normalization is not as effective as token-wise normalization. This is because channel-wist scaling does not suppress outlier tokens, especially the attention sink token. For example, using the first sample from WikiText-2 dataset, the average norm of the first token in the first layer is 19.1, which nearly twice as large as the overall average (11.3). Considering a ball-shaped property of our codebook, quantization error would be large for these tokens since their magnitudes are far from zero. On the other hand, our token-wiss wisc normalization effectively regulates the scale, making our codebook work effectively.
26

---

Table 15: Perplexity of LLaMA2-7B on WikiText-2 and C4 with NSNQuant-2b using different normalization strategies
Normalization
WikiText-2
C4
NSN NS wise Scaling
5,285
6,856
6.251
8.265
E Experiment details
E.1 Experiment environments
The PPL evaluation is performed on a Linux server with 2 RTX Titan GPUs. The evaluations on LongBench, GSM8K, HumanEval, CoQA, MMLU are conducted on Linux server with RTX 3090 GPUs. Efficiency analysis is conducted on Linux server with single A100-80GB GPU. All methods are implemented based on the HuggingFace Transformers library using PyTorch framework.
E.2
LongBench evaluation metrics
Table 16 shows the evaluation metrics used in LongBench. Qasper, TREC, and TriviaQA use exact matching- metrics, while the other tasks use heuristic metrics.
Table 16: Evaluation metries used in LongBench evaluation
Task
Evaluation Metric
Qasper
F1
QMSum
ROUGE-L ROUGE-L
MultiNews
TREC
Accuracy
TriviaQA
F1
SAMSum
ROUGE-L
LCC
Edit Sim
RepoBench-P
Edit Sim
F,3
Additional explanation on baselines
KIVI 30 is one of the pioneering works in low-bit quantization of the KV cache of LLMs. KIVI quantizes key cache along the channel dimension and value cache along the token dimension, considering its outlier patterns. To maintain consistency with our residual policy, we use group size of 64 for keys. Also, since smaller group sizes incur large additional bits which is critical for low-bit scenario, use 128 for values.
KIVI Had is variant of KIVI, where the Hadamard transform for key and value is added the existing KIVI framework. We add this variant because Hadamard transform is known to reduce errors in token-wise quantization a lot, especially
round-to-neare (RTN)-based method like KIVI.
KVQuant 19 another pioneering work in KV cache quantization. Similar to KIVI, KVQuant quantizes key and value cache along the channel dimension and token dimension, respectively. K KVQuant employs non-uninee con--nfform quantization (nuq) and dense-and-sparse quantization to improve performance further. We apply Q-Norm and 1% cense-and-spaan quantization to both KVQuant- lb and KVQuant-22 to obtain the best result.
CQ is our main competitor, which shares the core idea of VQ. CQ quantizes grouped channels using learned centroids, which serve as a codebook. We evaluate the performance of CQ-4ccb for 2-bit quantization, and CQ-8c 8cb0b for 1-bit quantization.
27

---

E.4 LM-Eval tasks
For ing tasks provided in Im-evaluatioon-arnees
evaluation on GSM8K, HumanEval, CoQA, and MMLU, we use the follow-
H. gsmak_cot, humaneval,
htwwsshhttuumhateeeeeee... ffll__laa___e cot frtt__ewwhot__eeeeee coqa, imlu_flamd laalu_flaa_ooo_fffaaeeeeeeeeeeee www_otootootttteeeeeeeeoeoteoeee mmlu_ lltfallnanotnoeeeeeeeeee
cot sow.aaooooeeeoooooolaassiiiiaaae
mmlu_flan, cot htwaaaaooooeewhoottttttteeee
F
Additional results
F1
PPL evaluation in generation setting
Since the perplexity (PPL.) evaluation in Table iis is different from our generation setting, we provide PPL. evaluation results in the generation setting. We obtain the output logits by processing tokens one-by-one, just like in a generation scenario. We also adopt residuals to maintain recent tokens in full-precision. The result is presented in Table 17 All methods achieve lower PPLs compared the results in Table because of the residual, but their relative order exhibits similar trends. Since the PPL evaluation setting the main table much more efficient measure and closer to convention in the previous studies, we use the same setting in the ablation studies.
Table 17: Perplexity measured in the generation setting with residual. We evaluate A2-M and LLMM33.1-80 on WikiText-2 and C4.
Method
Avg. bit width
Dataset
LLaMA2-7B
LLaMA3.1-8B
C4
7.04
10.36
WikiText-2
5.42
7.04
C4
6.63
8.43
KIVI-2
2.38
WikiText-2
5.12
5.84
C4
6.88
9.56
KIVI-2 + Had
2.38
WikiTeext-2
5.30
6.68
C4
6.84
8.88
KVQuant-2b 1%
2.32
WikiText-2
5.32
6.20
C4
6.84
9.48
CQ-4c9b
2.26
WikiText-2
5.22
6.00
C4
6.69
8.69
NSNQuant-2b
2,23
WikiText-2
5.16
6.03
C4
9.67
16.05
KVQuant- 1b 1%
1.32
WikiText-2
6.70
12.02
CQ-8c10b
1.27
C4
7.61
20.33
WikiTest-2
5.56
6,53
C4
7.07
10.32
NSNQuant-Ib
1.23
WikiText-2
5.48
7.55
F.2
Additional results on LongBench
Table 18 presents evaluation results on LongBench with LLaMA2-7B-CA and LLaMA3-8B-Instrru Similar to the main table, NSNQuant-Ib 1b outperforms other 1 1bit quantization methods by large margin. However, 2-bit results are quite noisy, without any clear trend. For example, K KVQuant-2b struggles in I.LAMA--BB 2---A-----hh,, but achieves the best score in example that shows the noisiness metrics, As shown in Table I8 KVQuant-2b achieves 56.51 in RepoBench-P, which is much higher than that of FP16. This result is unreliable since KVQuant suffers from performance degradation in other tasks.
This is another
To reduce the impact of noisiness, we evaluate each method from different perspective: how well each method preserves the original outputs. To achieve this, we measure the ROUGE-L score of each method by comparing their outputs with FP16 outputs. Since Qasper, TREC, and TriviaQA are evaluated using exact matching-base
metrics, we exclude them from the task list. The result shown in Table 19 It clearly shows that NSNQuant is the most effective method which preserves the original output faithfully. On the other hand, CQ is even worse than KIVI Had, suggesting that calibration basod VQ suffers from significant performance degradation when applied to diverse tasks.
28

---

Table 18: Additional results on LongBench with -LaMA2-13B--- LLaMA2-7B- and
LLaMA3-8B --sttucctt
Qaopeet
QMSum
MoltiNews
TREC
TrvisQA
SAMSum
LCC
RepoBenchhP
Model
Method
Bits
Avg.
FP16
16
17.06
20:95
26.55
68.50
87.75
42.59
48
49.80
45.18
KIVI-2
2.38
17.44
20.53
26.03
67.00
57.39
41.79
46.54
47.33
44.26
KIV1-2 + Had
2.38
15.44
19.59
26. 19
68,00
86.55
41.93
48.22 22
50.10
44,50
LLaMA2-13B-Chan
CQ-4ch
VVQunt--2
2.26
18.42
19.72
25.
67.00
87.69
40.72
32
47.18
48.32
44,50
44.35
NSNQuant-20
2.23
17.28
20.41
26. 16
68.50
87.51
42.48
47.882
49.69
44,98
KVQuant-Ih
CQ-RcT0b
1.27
16.08
18.67
20.87
49.00
87.12
37 /4
43:17
42.34
39.
NSNQuant-10
1.23
17.98
20:56
25.92
67.50
87.17
41.08
48.19
50,10
44.81
FP16
16
21.95
20.71
26.21
64,00
83.09
41.39
58.31
52,16
45.98
KIVI-2
56.69
KIVI-2 + Had
2.38
20.62
21.03
25.98
64.00
83.45
41.11
56.79
$1.10
45.51
LLMMA--7B-CMa
CQ-4%%
KVQuant-20 2h + 19
2.33
20.84
21 22
24.89
24.92
62,50
83.82
39.87
555 55.48
49.73
44,79
NS NSNQuant-2h
2,23
22.01
20.87
26.42
64.00
83.67
40.51
57.71
51.47
45.83
KVQuant- lb 1%
1.32
13.10
20.27
20.82
30.00
61.43
34.14
43.16
39.31
32,78
CQ.8c10b
42,00
NSNQuunt-h
1.23
17,70
20.74
25.49
64.00
82.06
40.31
55.14
50.57
44.50
FP16
31.25
23.54
26.69
74,00
90.31
42.65
57.23
51.69
49.67
KIVI-2
2.38
20.92
23.82
26.34
74,00
90.08
40.87
46.64
46.87
46.
KIVI-2 Had
35
41.38
46.92
LLaMA3- MD-A--D--AAAA Instruct
CQ-4chh
KVQuant-20 2b + 1%
2.32
2.26
27.91
26.26
23.16
22.60
25.10
25.26
74.00
73.50
90.78
90.33
39.95
40.48
57.89
56.50
53,28
56.51
48.74
49.20
NSNQuant-26
26.28
74.00
41.73
50.31
KV VuQnn---l lb + 1%
1.32
13.01
21 48
21.39
61.50
86.12
34.23
47.89
45.96
41.44
CQ-&c10b
1.27
15.25
14
21.63
44.50
87.26
36.63
51.24
46.36
40.49
NSNQuant-B
21.57
26.36
73.50
F.3 Additional results on GSM8K, HumanEval, CoQA, and MMLU
Table 20 presents the additional results for GSM8K, HumanEval, CoQA, and MMLU. The result shows a similar trend to Table where NSNQuant generally outperforms other methods in both 1-bit and 2-bit quantization.
F.4 Additional latency breakdown results
Table 21 presents additional latency breakdown results under different configurations, measured with LLaMA2-7B. We fix the number of generated tokens to 64 and vary the batch size and input prompt length to observe performance trends. The results exhibit a similar pattern to Table where NSNQuant incurs additional overhead during the prefill stage but demonstrates lower latency during the decode stage.
F55 Results on AIME-2024
To evaluate the long-context reasoning ability, evaluate our methods AIME-2024, with DeepSeck-RI-Dise -il--isiill--Lama-81 As suggested in DeepSeek [5), we set temperature to 0.6, top-p to 0.95, and maximum generated tokens 32768. Since AIME-202 only contains 30 problems, run the evaluation with 5 different seeds.
The result is shown in Table 22 In 2-bit quantization, NSNQuant and CQ both show little accuracy drop with small differences. On the other hand, under 1bit quantization, both methods experience significant performance degradation, but NSNQuant-11 1b achieves over 2x higher accuracy than CQ- 8c10b. While CQ-4c9b is generally strong in math reasoning tasks such as GSM8K and AIME-2024, CQ-8c 10b fails severely in such datasets.
F.6 Comparison with QuaRot and SpinQuant
QuaRot 2 and SpinQuant (29 are the recent quantization methods for KV cache which also leverage the Hadamard transform. However, we don't don't include them our basclines because they
29

---

Table 19: Average ROUGE-L score measured with FPI6 output. Qasper, TREC and TriviaQA are excluded since these tasks provide the objective metrics based on exact matching.
Model
Method
QMSum
MultiNews
SAMSum
RepaHench..P
Avg.
KIVI
2.38
52.34
$2.18
73.17
52.51
49.06
55.85
KIVI Had
2.38
55.24
55.39
76.53
60.55
57.24
60.99
KVQuant-2b + 196
2.32
55.85
53.02
76.55
57.07
53.91
59.28
CQ-4c9b
2.26
56.07
49.19
LLaMA2-13B-CHA
NSNQuant-26
2.23
62.73
59.69
85.02
70.93
68.23
69.32
KVQuant-Ib- + 1%
1,32
40.88
32.00
42.86
22.95
18.32
31.40
CQ-8c10b
1.27
43.39
34,49
39.21
30.59
26.41
38.82
NSNQuant-1
1.23
51.63
71.77
48.44
KIVI
2.38
46.95
48.04
67.22
47.32
46.96
$1.30
KIVI Had
2.38
49.12
50.13
73.87
55.30
56.36
56.95
KVQuant-2h 2b + 19h
2.32
50.77
47.71
74.73
$5.20
55.24
56.73
CQ-4c9b
2.26
52.04
69.22
48.80
LLaMA2-7B-Chat
NSNQuant-26
2.23
58.19
55,42
80.14
69.93
67.36
66.21
KVQuant-Ib 1%
1.32
37,66
33.14
44, 14.16
22.34
23.54
32.17
CQ-8c10b
1.27
40.01
31.07
56.00
25.52
27.02
35.92
NSNQuant-II
1.23
46.42
47.91
66.08
47.65
46.92
51,00
KIVI KIVI Had
2.38
49.38
48.54
44.29
44.45
40.33
45,40
2,38
49.35
51.77
46.83
49.65
46.67
48.85
KVQuant-22 2b + 1%
2.32
50.60
50.00
45.66
53.18
46.14
49.12
CQ-4c9b
2.26
50.06
47.76
49.25
49.65
41.63
47.67
LLaMA3-8B-Instmn
NSNQuant-26
2.23
57.24
57.02
55.29
65.43
60.09
59.01
KVQuant-Ib 1%
1.32
37.22
31.72
29.58
25.71
22.70
29.39
CQ-8c10b
1.27
37.92
32.27
34,20
28.07
22.25
30.94
NSNQuant-IB
1.23
44.00
48.31
41.04
42.99
36.17
42.50
KIVI KIVI Had
2.38
45.23
47,48
42.19
$2.31
43.98
46.24
2.38
47.75
50.65
44.11
$7.71
$1.76
50.40
KVQuant-22 + 1%
2.32
46.33
49.32
45.85
$7.20
$2.07
50.15
CQ-4c9b
2.26
45.45
48.54
47.76
$4.13
45.55
48.29
LLaMA3.1-88 8is-rnstttct
NSNQuant-26
2.23
51.72
$6.69
52.27
68.32
64.37
58.67
KVQuant- 1b 1%
1.32
34,62
33,02
32.44
26.67
22.68
29.89
CQ-8c10b
1.27
33.70
32.47
29.32
29.36
22.30
29.43
NSNQuant-Ib
1.23
41.92
46.91
40.68
46.85
37.56
42.78
KIVI KIVI + Had
2.38
50.56
49,99
78.84
61.02
49.35
$7.95
2,38
$6.40
55.62
79.54
65.26
$7.83
62.93
KVQuant-20 2b+ + 1%
2.32
53.09
51.12
77.38
64.04
56.96
60.52
CQ-4c9b
2.26
52.58
51.55
80.26
60.15
52,80
$9,46
Mistral-7B- 7inn----n-ntttOO....
NSNQuant-22 2h
2.23
63.19
60.06
84.42
75.32
69.85
70.57
KVQuant- -b 1%
1.32
36.65
32,30
60.32
34.81
28.97
38.61
CQ-8c10b
1.27
39.17
33.26
64.46
34.86
27.29
39.81
NSNQuant-1b
1.23
49.96
50.01
74.29
55.03
45.03
54.86
Table 20: Additional results on GSM8K, HumanEval, CoQA, and MMLU with LLaMA2-13B-Char and
(8-shot,
HumanEival
CoQA
MMLLU CoT)
Humanities
STEM
Social
Other
FP16
37.30
17.07
64.08
60.89
41.5 11.54
62.54
60.95
KIVI
2.38
30.63
15.24
63.70
57.48
38.66
57.42
$6.21
KIVI Had
2.38
31.69
14.02
63.03
39.91
40.21
15.05
59.25
KVQuant-2h
39.31
$7.96
LLaMA2-138-CM
4cbb
2.26
34.04
14,63
64.60
56.26
35.70
57.5
$5.47
N5NQuan-22h
2.23
35.48
18.29
63.67
59.16
41.47
62.34
60.26
KVQuant
1%
1.32
8.64
9.15
$7.08
29.89
23.14
35.
21 14.44
10.
1.27
19.71
12,20
60.88
21.95
11.25
30.84
19.33
N5NQuant-Ib
63.10
38.15
57.71
FP16
16
76.72
28.05
61.47
70.51
$3.00
70.89
70.66
KIVI + Had
2.38
69.75
32,93
60.72
63..56
47.36
63.005
65.06
LLLMA3388-
KVQuant-20 2h 1%
2.32
70.74
23.78
60.30
64.944
46.34
64
66.11
4c9b
2.26
72.55
23.78
60.60
63.86
47.94
64.20
65.26
2unQuant-2b
2.23
74.53
32.93
61.62
68.95
52.44
69.36
68.91
KVQuant-
13.73
CO 10b
1.27
24.26
13.41
$3.38
22.05
14.44
26.42
16.
Ib
61.64
29.27
60.28
$8.73
42.70
59.76
60.94
30

---

Table 21: Latency (ms) breakdown of NSNQuant-22 compared with the FPI6 baseline. BS, L. HT and VQ refer to batch size, imput prompt length, Hadamard transform and vector quantization, respectively. The number of generated tokens is fixed 64.
I.
Stage
Method
Total
NSN
VQ
FP16
1294.84
Prefill
NSNQuant-2b
1761.45
104.13
14.69
324.16
4096
Decode
FP16
49.34
(per step)
NSNQuant-2b
33.17
0.29
1.70
0.14
FP16
I 1225.90
Prefill
NSNQuant-2h
1707.22
155.22
15.04
326.85
512
Decode
FP16
50.80
(per step)
NSNQuant-2b
36.49
0.34
1,88
0.66
FP16
1223.20
Prefill
NSNQuant-2b
1698.50
105.76
15.02
329.62
128
Decode
PP16
63,45
(per step)
NSNQuant-2b
45.25
0.83
1.87
2.43
Table 22: Evaluation results on AIME-2024 with DeepSeek-RI--istill---
Method
pass
FP16
43.3
CQ-4c9b
40.7
NSNQuant-2b
40.0
CQ-8c10b
6.7
NSNQuant-11 16.7
are not suitable for low-bit quantization. Their evaluation results on GSM8K, HumanEval, CoQA, and MMLU with LLaMA3.1
are shown in Table 23 We use clipping ratio of 0.95, following QuaRot, and use group size of 128 for both keys and values. We only train R2 rotation matrix in SpinQuant, since we don't apply quantization weights or activations. We also apply the residual policy of NSNQuant, following the setting in the main experiments. The result shows that QuaRot and SpinQuant underperform every dataset, and they are worse than our weakest baseline, KIVI-2.
Table 23: Evaluation results GSM8K, HumanEval, CoQA, and MMLU with LLaMA3.1-8B. Instruct.
MMLU (4-shot, CoT)
Method
Bits
GSM8K (8-shot, CUi))
HumanEval
CoQA
Humanities
STEM
Social
Other
66.84 84
QuaRot (W16A16KV2)
2.25
48.67
47.56
64.05
57.32
41.91
62.64
60.51
SpinQuant
2.25
41.09
40.24
62.52
56.30
39.53
59.85
59.39
NSNQuant-2b
75.89
56.10
63.83
71.04
55.64
73.42
70.74
G Visualizations
G.1
Inter-channel correlation after the NSN and Hadamard
We verify that the NSN and Hadamard transform align each channel distribution with the standard normal distribution. However, large inter-channel dependency can change its multi-dimensiona shapes, affecting the efficiency of our tuned codebooks. Tigure[OO]showe the layer-wise mean absolute correlation (MAC) between channels after the NSN and Hadamard transform. Using WikiText-2 and I.L.MA--BB model, we measure the correlations between different channels which are grouped and quantized together. The results show that the correlation is generally small except for the early
31

---

layers. This indicates our codebook is close to optimal in the later layers, while there is still room for improvement the early layers.
Layer-wise Mean Absolute Correlation (MAC) Key
Mean Absolute Correlation (MAC)
0.30
0.25
0.20
0.10
5
10
15
20
25
30
Layer
Figure 10: Layer-wise mean absolute correlation (MAC) between different channels. The results are measured on WikiText-2 with LLaMA2-7B.
G.2 t-SNE visualization of keys and values
To visualize the divergent token distribution of different datasets, plot 2D t-SNE results in Figure [I] For MultiNews, LCC, and SAMSum, the "context" column is used for visualization. We sample first 10 samples in the test split for each datasets. The input length is limited 2048 tokens, resulting in ~20000 tokens per dataset. We run t-SNE for all tokens, and randomly select 2000 tokens from each dataset for plotting. None of the normalization methods rotations applied keys and values in this visualization and pre-RoPE keys are used for visualization.
G.3 Channel-wise mean and standard deviation
We visualize the channel-wise Hadamard transform in Figure 12 and Although mean and standard deviation are generally close and expected, there are non-negligible errors some of the heads in the early layers.
and standard deviation of keys and values after the NSN and
H Comparison with CQ
Motivation Although CQ and NSNQuant both quantize multiple channels jointly, their motivations behind adopting vector quantization (VQ) differ. CQ employs VQ to capture correlations across channels, whereas NSNQuant is inspired by the observation from QuIP# [37] that VQ performs particularly well when quantizing ball-shaped vectors. Since the channel distribution is aligned with the standard normal distribution regardless of model and data, we can build an optimized codebook for compressing the normal distribution data in advance.
Calibration CQ requires calibration data and backward passes through model weights to compute Fisher information. In contrast, NSNQuant does not require any external data, and its codebook tuning can be completed within few minutes single GPU. Moreover, the tuned codebook is model-ag;ostic; can be reused across models long as the hidden dimension per head remains unchanged. This property makes NSNQuant much easier to integrate and significantly improves its generalization ability.
Codebook CQ requires separate codebook for each group of coupled channels. For example, in CQ-8c10b, the total size of all codebooks matches the KV cache for 1024 tokens in fp16, which equivalent to 16384 tokens in 1-bit representation. contrast, NSNQuant uses single shared codebook for the entire model, resulting in almost additional memory overhead.
32

---

I Limitation and broader impacts
1.1 Limitation
Although proposed Normalize-Shin Sei---Shhfttttiiii (NSN) is empirically proven to align the output distribution with the standard normal distribution, find that it does not work well in the early layers due to outlier channels with huge variances. Our analysis shows that the quantization errors in these layers still remain low, but handling these channels properly will further improve performance. Also, NSNQuant does not consider inter-channel correlations or dependencies, while CQ induces emrors by relying too much on them. Exploring middle ground between two methods will be beneficial: exploiting inter-channel relations, while avoiding overfitting to calibration datasets.
1.2 Broader impacts
Our work improves the scalability and efficiency of LLM inference by reducing memory usage and increasing throughput. This enables broader adoption of LLMs across diverse applications and hardware setups. By significantly reducing memory consumption for long-context inference, our method makes advanced use cases like long-context reasoning feasible even on devices with constrained memory resources.
That said, our work does not provide any empirical analysis regarding safety aspects. As a result, applying quantization may introduce unintended effects such hallucinations degraded reliability, especially in sensitive scenarios. We therefore advise caution when deploying quantized models in safety-critical applications.
33

---

Layer 1, Head
Layer 10, Head
Layer 20,
Layer
Keys
Values
C4
Wikflext2
MultiNews
LCC
SAMSum
Figure 11: t-SNE visualization LLaMA3.1-8B key and value.
34

---

Channel-wise mean / stddev WikiText-2
meen, Layer
mear / Layer 10
mean / Layer 20
mean / Layer 30
0.2
Hoad
Head
-0.2
Channel
Charnel
Channel
Channel
stedev
snddew 10
stodev
stridew Layer
144
Hoad
Heed
Hoad
Heed
1.0
0.8
Charnel
Charnel
Channel
Channel
Channel-wise mean / stddev on C4
layer
Layer
0.4
NTN WW
Hosd
Head
moon
0.0
-044
Charnel
Charnel
Layer
saddov Layer
stodev Layer
stddey Layer
12
Head
Head
10
0.5
Visualizations of channel-wise and Hadamard transform to LLaMA2-7B on WikiText-2 (top) and C4 (bottom). The first sample from the test split of each dataset is used for visualization. While NSN overall performs standardization fairly well, it struggles in certain heads of the early layers.
Figure 12:
and standard deviation of keys after applying NSN
35

---

Channel-wise mean / stddev WikiText-2
mean, Layer
mear Layer 10
mean / Layer 20
mean / Layer 30
0.2
Hedd
Hoad
Head
-0.2
Channel
Charnel
Channel
Channel
stedev
snddew 1.3
stodev Layer
stridew Layer
144
Head
Flaati me War
Head
Head
Heed
1.0
0.8
Channel
Charnel
Channel
Channel
Channel-wise mean / stddev on C4
Layer
layer
0.4
Hosd
Head
moon
0.0
-044
Charnel
Charnel
Channel
Layer
stodev Layer
stdday Layer
12
Head
10
0.5
and standard deviation of values after applying NSN and Hadamard transform to LLaMA2-7B on WikiText-2 (top) and C4 (bottom). The first sample from the test split of each dataset is used for visualization. While NSN overall performs standardization fairly well, it struggles in certain heads of the early layers.
Figure 13: Visualizations of channel-wid wiine
36


Reviews:
[{"review_id": "yIh3BMbAgO", "reviewer": "Reviewer_etjS", "summary": "The authors propose NSNQuant, a calibration-free vector quantization (VQ) method aimed at addressing the distribution-shift sensitivity observed in previous KV cache quantization methods such as CQ (Zhang et al., 2024). NSNQuant utilizes a three-step Normalize-Shift-Normalize (NSN) transformation combined with a Hadamard transform to align key-value (KV) vectors to a standard normal distribution. Consequently, this allows the codebooks to be trained solely on synthetic Gaussian data, bypassing the need for calibration datasets. Extensive experiments demonstrate that NSNQuant outperforms existing calibration-based approaches across various benchmarks.", "questions": "- As noted in weaknesses, how much additional latency do the proposed transformations (NSN and Hadamard) introduce during inference compared to CQ? Is the overhead negligible in practice? \n- The authors mention that early layers exhibit higher variance after transformation. What might be the possible causes for this phenomenon? Is there a risk that quantization errors in these early layers propagate and compound across subsequent layers? \n- In the case of CQ, would scaling up the calibration set help reduce out-of-distribution errors? While NSNQuant benefits from faster training of the centroids, KV cache applications prioritize their inference efficiency. In such scenarios, how does NSNQuant compare to CQ if calibration can be scaled effectively?\n- VQ is powerful for compression due to its strong representation capacity. While enforcing a prior enables calibration-free codebook training, could this constraint limit VQ’s expressiveness? For example, for the early layers, would finetuning the codebook with calibration sets further mitigate the quantization error?", "limitations": "The authors have discussed limitations of their study properly in the appendix.", "rating": 4, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 2, "significance": 2, "originality": 3, "strengths_and_weaknesses": "Strengths: \n- The proposed idea of aligning KV states to a known distribution prior is novel and can effectively removes the calibration requirement.\n- The authors show a variance bound on the transformed key/value states in Lemma 1.\n- Experiments are comprehensive, clearly illustrating NSNQuant’s superiority over existing quantization baselines, particularly its robustness to distribution shifts which notably improves upon CQ.  \n\nWeaknesses: \n- Efficiency evaluation is inadequate. Since NSNQuant involves additional computational steps (NSN and Hadamard transformations) during inference, it is essential to measure and clearly report the latency overhead. The absence of detailed latency comparison with calibration-based methods, such as CQ, weakens the practical appeal of the proposed approach. \n- The overall presentation of the paper needs improvement, e.g., Table 1 is never explicitly referenced or discussed in the main text, which is somewhat confusing.", "was_revised": true, "final_justification": "The authors have provided a justification for the method’s latency, which addressed my concerns. Given its unique advantages over the baseline, I will maintain my score of acceptance.", "rebuttal": "We sincerely appreciate the time and effort you devoted to reviewing our submission.\n\n**[W1, Q1] Latency of NSN, Hadamard and VQ**\n\nTo investigate the overhead of each component of NSNQuant, we measure the latency caused by each component: NSN, Hadamard transform and VQ.\nHere, the double quantization is included as part of NSN, and the scale adjustment is grouped under VQ.\nWe evaluate latency under two different settings with LLaMA2-7B, each averaged over 100 runs to ensure stable measurements.\nWe report per-token latency for the decoding stage.\nAll experiments are done on a linux server with a single A100-80GB GPU.\n\n**[1]** Setting 1: batch size = 1, prompt length = 2048, generated tokens = 128 \n\nThis setting is for comparison with CQ. In the CQ paper, authors provide latency measurements on Appendix K. They used the batch size of 1, prompt length of 2000 and generate 100 tokens.\nWe try to match this setting as closely as possible, and set the closest numbers which are divisible by residual size (64).\nSince the experiments are conducted with different GPUs (CQ: A100-40GB, ours: A100-80GB), it's hard to directly compare these results.\nWe instead compare the **relative overhead** added over the FP16 baseline, as the CQ inference kernel is not publicly available.\n\n**[1-1]** Prefill\n| Method         | Total | NSN | Hadamard transform | VQ |\n|----------------|-----------|--------|-------|-----|\n| FP16           | 169.85ms | N/A | N/A | N/A |\n| NSNQuant-2b    | 244.73ms | 22.51ms | 2.97ms | 42.22ms |\n\n**[1-2]** Decode (per token)\n| Method         | Total | NSN | Hadamard transform | VQ |\n|----------------|-----------|--------|-------|-----|\n| FP16           | 28.21ms | N/A | N/A | N/A |\n| NSNQuant-2b    | 33.95ms | 0.30ms | 1.83ms | 0.08ms |\n\nThe result shows that NSNQuant-2b requires 44% and 20% of additional overhead in prefill and decode, respectively, compared to the full precision baseline. \nDespite the additional latency from NSN and Hadamard transforms, the overhead remains lower than CQ (see Table 14 in the CQ paper).\nCQ-4c8b requires nearly twice the prefilling time and incurs a 26% overhead in the decoding stage.\n\nWe attribute this result to the optimized custom kernel implementation.\nIn fact, our VQ implementation can be used in K-Means algorithm, and this can boost the K-Means algorithm needed for CQ.\nSurprisingly, using our implementation, the centroid for CQ-8c10b can be found in 15 minutes with a single RTX-3090, whereas the original CQ paper reports that it requires more than 100 minutes.\n\n\n**[2]** Setting 2: batch Size = 32, prompt length = 512, generated tokens = 64 \n\nThis setting reflects a more realistic deployment scenario, using a moderate batch size and prompt length.\n\n**[2-1]** Prefill\n| Method         | Total | NSN | Hadamard transform | VQ |\n|----------------|-----------|--------|-------|-----|\n| FP16           | 1225.90ms | N/A | N/A | N/A |\n| NSNQuant-2b    | 1707.22ms | 155.22ms | 15.04ms | 326.85ms |\n\n**[2-2]** Decode (per token)\n| Method         | Total | NSN | Hadamard transform | VQ |\n|----------------|-----------|--------|-------|-----|\n| FP16           | 50.80ms | N/A | N/A | N/A |\n| NSNQuant-2b    | 36.49ms | 0.34ms | 1.88ms | 0.66ms |\n\n\nWhile NSNQuant-2b has higher prefilling latency, it achieves lower decoding latency due to reduced DRAM access during attention computation.\n\nInterestingly, both settings show that while VQ has the largest overhead in the prefill stage, Hadamard transform has the largest overhead in the decoding stage. This is similar to MLP layers becoming the main bottleneck in the decoding stage where it becomes memory-bound.\n\n---\n\n**[W2] Presentation of the paper**\n\nWe will polish references to tables and figures, and make sure they are properly used. For example, Table 1 will be referenced in Section 3.2 to demonstrate the effectiveness of NSN. Also, we will fix typos and add missing descriptions for notations (e.g. W_q, W_k, W_v in Figure 2).\nIn addition, our paper has multiple references to Appendix, which makes it hard to read the main text. This was due to the page limit, so will be improved in the final version.\n\n---\n\n**[Q2] High covariances observed in the early layers**\n\nAs mentioned in the paper, certain channels in the early layers exhibit high variance or covariance after the NSN transformation. This occurs because these channels originally have larger variances compared to the other channels, which become more pronounced through the normalization process of NSN.\n\nHowever, this phenomenon does not degrade quantization quality. Because of the *isotropic nature of the codebook*, NSNQuant remains robust even when such outlier channels exist: (Recall that after applying NSN and Hadamard transform, NSNQuant splits the vector into 8-dimensional subvectors and selects the closest match from the codebook.)\n\n- The codebook is trained on synthetic standard normal data (`torch.randn`), which are uniformly distributed in direction. Therefore, the quantization quality depends on the **magnitude** of the subvector rather than its **direction**.\n- Importantly, the following Hadamard transform spreads the contribution of each channel across all dimensions, making it unlikely for any single subvector to be dominated by an outlier channel. While the magnitude distribution may deviate from that of standard normal vectors, it remains within a range well-handled by the codebook.\n\nAs a result, the quantization quality remains high even in the early layers as shown in Figure 5. However, the codebook is far from optimal in these layers since it does not consider correlations caused by the outlier channels. We believe that further improvements are possible by explicitly accounting for these outlier channels, and leave this for future work.\n\n---\n\n**[Q3] Effect of calibration set size on the performance of CQ**\n\nTo investigate the effect of scaling the calibration set, we test 2 variants of CQ where the size of the calibration set is doubled (From 16 to 32).\nFirst, we use 32 samples from WikiText-2 (W32).\nThis variant enlarges the size of the calibration set.\nSecond, we use 16 samples from WikiText-2 and another 16 samples from C4 (W16C16).\nThis variant not only increases the size but diversifies the calibration set, covering a wider range of input distribution.\nWe use LLaMA3-8B-Instruct model to test performance on the three downstream tasks: HumanEval, GSM8K, and CoQA.\nWe denote two variants as W32 and W16C16, respectively.\n\n| Method         | HumanEval | GSM8K (8-shot, CoT) | CoQA  |\n|----------------|----------------|--------------------------|----------|\n| FP16           | 28.05          | 76.62                    | 61.47    |\n| NSNQuant-2b    | **32.93**          | **74.53**                    | **61.62**    |\n| CQ-4c9b        | 23.78          | 72.55                    | 60.60    |\n| CQ-4c9b (W32)       | 25.00          | 72.93                    | 60.38    |\n| CQ-4c9b (W16C16)       | 29.27          | 73.16                    | 60.98    |\n| NSNQuant-1b    | **29.27**          | **61.64**                    | **60.28**    |\n| CQ-8c10b       | 13.41          | 24.26                    | 53.38    |\n| CQ-8c10b (W32)      | 15.24          | 21.76                    | 53.37    |\n| CQ-8c10b (W16C16)      | 12.20          | 46.47                    | 54.07    |\n\nWhile W32 shows only marginal improvement from the baseline, W16C16 provides clearer benefits. This result shows that to improve the robustness of CQ, it is important to diversify the calibration set, rather than just scaling it up. In addition, despite improved performance, NSNQuant still shows better results. For example, on HumanEval, CQ-8c10b still suffers from significant performance degradation since its input distribution (codes) is not covered by either WikiText-2 or C4.\n\nWe believe further scaling calibration set of CQ will further improve its generalization ability. However, since CQ needs gradient computation, the calibration process incurs comparable cost to fine-tuning, as the calibration set grows. Moreover, it's challenging to prepare the calibration set that adequately covers the full diversity of real-world input distributions (e.g. different languages).\nTherefore, we believe that NSNQuant remains a highly practical and robust option in general scenarios, especially when broad generalization is required.\n\n---\n\n**[Q4] Impacts of NSN on the expressiveness of VQ**\n\nAs you pointed out, applying the NSN may suppress certain channel-specific trends, potentially limiting the expressiveness of vector quantization (VQ). However, as shown in Figure 10, channel-wise correlations still remain after NSN, particularly in the early layers. This suggests that further fine-tuning the codebook using a small calibration set could indeed help reduce quantization error, especially in these layers. Moreover, NSNQuant does not consider any channel-wise  importance during codebook construction. Incorporating this information could offer an additional opportunity to enhance quantization quality.\nWe believe future work could explore a promising middle ground between normalization and calibration, potentially yielding better trade-offs between generalization and expressiveness.\n\nIn addition to this, we would like to emphasize that per-head codebooks or fine-tuned variants introduce additional memory overhead. For example, CQ-8c10b requires ~500MB of additional memory space with LLaMA2-7B. In contrast, NSNQuant maintains a single shared codebook across the layers, achieving strong compression and generalization without calibration."}, {"review_id": "USl77nnpU0", "reviewer": "Reviewer_Ynez", "summary": "This paper proposes NSNQuant, a novel calibration-free vector quantization method for KV Cache. Its key innovation is the Normalization-Shift-Normalization module, which aligns channel distributions with the standard normal distribution. This approach addresses a key limitation of prior methods, the reliance on specific calibration datasets, which limits generalizability. The method demonstrates superior accuracy and efficiency compared to existing techniques through comprehensive evaluation.", "questions": "1. Why is the order of applying NSN different for keys and values? Is there a specific reason or theoretical analysis supporting this design choice?\n2. Which dataset is used for codebook fine-tuning? How does the choice of dataset impact performance and generalization ability?\n3. Since CQ was calibrated on WikiText-2, why does it still underperform compared to NSNQuant?", "limitations": "Yes", "rating": 4, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 3, "significance": 3, "originality": 3, "strengths_and_weaknesses": "Strengths:\n\n1. The paper introduces a novel calibration-free vector quantization method. It incorporates the NSN module to align channel distributions with the standard normal distribution. This is a significant innovation addressing a core limitation of existing methods that depend on calibration datasets.\n\n2. The paper provides a comprehensive evaluation of the proposed method across various datasets, demonstrating its superior performance over previous methods. It also includes a well-structured ablation study validating the effectiveness of its components.\n\n3. The paper proves the mathematical effectiveness of the proposed module. It also provides a CUDA implementation to enhance inference efficiency.\n\nWeaknesses:\n\n1. This method is sensitivite to the choice of hyperparameters, e.g., residual size when quantization bit is 1.\n2. This method incur extra latency due to the additional normalization steps especially in prefilling stage and the paper does not provide a detailed analysis of its latency impact.", "was_revised": true, "final_justification": "Most of my concerns have been addressed by authors' response. \nI do not raise the score because the incurred overhead is relatively high, which may have a negative impact on practical usage.\nHowever, this paper still deserves an acceptance due to its contributions and evaluation results compared to related work.\nSo, we choose to maintain the original score -- borderline accept.", "rebuttal": "We sincerely appreciate the time and effort you devoted to reviewing our submission.\n\n**[W1] Sensitivity to hyperparameter selection**\n\nWhile NSNQuant-1b appears sensitive to the choice of residual size, we emphasize that this sensitivity is not due to random fluctuations but reflects a controllable trade-off between performance and memory cost. As residual size increases, performance improves due to the inclusion of more full-precision tokens, at the cost of additional memory usage.\nWe believe that this cost-reliability tradeoff will become less pronounced as future work improves the quantization quality further.\nPlease refer to Appendix C.2 for detailed results.\n\nFurthermore, we would like to clarify that NSNQuant is not broadly sensitive to hyperparameter selection. In practice, residual size is the only major hyperparameter that significantly affects performance. Other components, including the NSN process and the codebook, are fixed and do not require tuning across different models or tasks.\n\n---\n\n**[W2] Latency of NSN, Hadamard transform and VQ**\n\n\nTo investigate the overhead of each component of NSNQuant, we measure the latency caused by each component: NSN, Hadamard transform and VQ.\nHere, the double quantization is included as part of NSN, and the scale adjustment is grouped under VQ.\nWe evaluate latency under two different settings with LLaMA2-7B, each averaged over 100 runs to ensure stable measurements.\nWe report per-token latency for the decoding stage.\nAll experiments are done on a linux server with a single A100-80GB GPU.\n\n**[1]** Setting 1: batch size = 1, prompt length = 2048, generated tokens = 128 \n\nThis setting is for comparison with CQ. In the CQ paper, authors provide latency measurements on Appendix K. They used the batch size of 1, prompt length of 2000 and generate 100 tokens.\nWe try to match this setting as closely as possible, and set the closest numbers which are divisible by residual size (64).\nSince the experiments are conducted with different GPUs (CQ: A100-40GB, ours: A100-80GB), it's hard to directly compare these results.\nWe instead compare the **relative overhead** added over the FP16 baseline, as the CQ inference kernel is not publicly available.\n\n**[1-1]** Prefill\n| Method         | Total | NSN | Hadamard transform | VQ |\n|----------------|-----------|--------|-------|-----|\n| FP16           | 169.85ms | N/A | N/A | N/A |\n| NSNQuant-2b    | 244.73ms | 22.51ms | 2.97ms | 42.22ms |\n\n**[1-2]** Decode (per token)\n| Method         | Total | NSN | Hadamard transform | VQ |\n|----------------|-----------|--------|-------|-----|\n| FP16           | 28.21ms | N/A | N/A | N/A |\n| NSNQuant-2b    | 33.95ms | 0.30ms | 1.83ms | 0.08ms |\n\nThe result shows that NSNQuant-2b requires 44% and 20% of additional overhead in prefill and decode, respectively, compared to the full precision baseline. \nDespite the additional latency from NSN and Hadamard transforms, the overhead remains lower than CQ (see Table 14 in the CQ paper).\nCQ-4c8b requires nearly twice the prefilling time and incurs a 26% overhead in the decoding stage.\n\nWe attribute this result to the optimized custom kernel implementation.\nIn fact, our VQ implementation can be used in K-Means algorithm, and this can boost the K-Means algorithm needed for CQ.\nSurprisingly, using our implementation, the centroid for CQ-8c10b can be found in 15 minutes with a single RTX-3090, whereas the original CQ paper reports that it requires more than 100 minutes.\n\n\n**[2]** Setting 2: batch Size = 32, prompt length = 512, generated tokens = 64 \n\nThis setting reflects a more realistic deployment scenario, using a moderate batch size and prompt length.\n\n**[2-1]** Prefill\n| Method         | Total | NSN | Hadamard transform | VQ |\n|----------------|-----------|--------|-------|-----|\n| FP16           | 1225.90ms | N/A | N/A | N/A |\n| NSNQuant-2b    | 1707.22ms | 155.22ms | 15.04ms | 326.85ms |\n\n**[2-2]** Decode (per token)\n| Method         | Total | NSN | Hadamard transform | VQ |\n|----------------|-----------|--------|-------|-----|\n| FP16           | 50.80ms | N/A | N/A | N/A |\n| NSNQuant-2b    | 36.49ms | 0.34ms | 1.88ms | 0.66ms |\n\n\nWhile NSNQuant-2b has higher prefilling latency, it achieves lower decoding latency due to reduced DRAM access during attention computation.\n\nInterestingly, both settings show that while VQ has the largest overhead in the prefill stage, Hadamard transform has the largest overhead in the decoding stage. This is similar to MLP layers becoming the main bottleneck in the decoding stage where it becomes memory-bound.\n\n---\n\n**[Q1] Order for applying NSN to keys**\n\nApplying NSN to values is straightforward, as they are not affected by position embedding.\nHowever, for keys, the presence of the RoPE (Rotary Position Embedding) requires additional consideration. Motivated by prior studies such as KVQuant and CQ, which observed that key vectors are easier to quantize before RoPE, we empirically compare applying pre-RoPE NSN and post-RoPE NSN.\nAs detailed in Appendix C.5, the pre-RoPE variant yields slightly better quantization quality, and we adopt this setting as our final design.\n\n---\n\n**[Q2] Datasets used for codebook tuning**\n\nAs stated in Section 3.4, we fine-tune the codebook on **synthetic standard normal data** generated by `torch.randn`.\nWe do not need any external data because NSN and Hadamard transform align the KV distribution with the standard normal distribution, and it is effective across different models and datasets.\n\nMoreover, the ablation study on codebook tuning (Section 4.6) shows that the codebook tuned solely on synthetic data works well when applied to actual keys and values.\nInterestingly, the cosine similarity measured using the keys and values of LLaMA2 is very close to that measured using the synthetic data.\nThis indicates that NSN successfully aligns the key-value distribution with the standard normal distribution.\n\n---\n\n**[Q3] Lower performance of CQ on WikiText-2**\n\nAlthough CQ is calibrated on WikiText-2, it may still underperform compared to NSNQuant for the following reasons:\n\n- **Limited calibration set size**: CQ uses only 16 samples from WikiText-2 for calibration. This small set may fail to capture the full diversity of input distribution within the dataset, leading to suboptimal codebook generalization even on in-distribution data.\n\n- **Additional techniques in NSNQuant**: NSNQuant applies additional techniques like scale adjustment to further improve performance. Please refer to Appendix C for ablation studies on these techniques."}, {"review_id": "9Q2asZEU69", "reviewer": "Reviewer_C5ZW", "summary": "It is a well-known problem that memory issue is the major bottleneck of LLM (Large Language Model) inference because LLM inference is memory-bounded. Recently, a lot of work has emerged to resolve this issue, and one of the representative solutions is KV cache quantization.\n\nThis work points out that the distribution of the KV cache differs according to inputs, thus the distributions of the KV cache shift depending on the calibration set. It leads to performance degradation in general areas. To resolve this distribution shift problem, this work proposes token-wise normalization, following channel-wise centering and token-wise normalization again. With the above process, the distributions of tokens become standard normal distributions, and it prevents distribution shifts of the KV cache after quantization. Also, by applying Hadamard transform and scale adjustment, the paper achieves state-of-the-art results among other KV cache compression works.", "questions": "- How much is the overhead of NSNQuant and Hadamard transform? The paper compares memory consumption and throughput with 16-bit results. Is the overhead negligible compared to previous KV cache quantization works? Or, those proposals are harmful to the overall memory consumption or throughput?", "limitations": "Yes", "rating": 4, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 3, "significance": 2, "originality": 2, "strengths_and_weaknesses": "Strengths\n- This work shows the distribution shift problem that occurred while applying vector quantization to the KV cache.\n- This work resolves the above problem in a simple yet effective way; add token-wise normalization, channel-wise shift, and token-wise normalization to the KV cache, along with Hadamard transform.\n- This work provides optimized kernels to execute vector quantization, scale adjustment, and attention with quantized KV caches.\n\nWeaknesses\n- Along with the proposed method, NSNQuant and Hadamard transform, this paper applies other techniques such as scale adjustment and double quantization. However, the effect of those suggestions is not addressed in the experiments.\n- Llama and Mistral have similar architectural features (e.g., attention architecture, activation function, and normalization function). The paper should have examined other LLM models that have different architectural features to verify the general effectiveness of the proposal.", "was_revised": true, "final_justification": "Under the assumption that the reviewers incorporates the reviewers’ comments in the paper, I think that the paper is worth accepting.", "rebuttal": "We sincerely appreciate the time and effort you devoted to reviewing our submission.\n\n**[W1] Missing analysis for scale adjustment and double quantization**\n\nWe've already included the analysis for different components of our method in Appendix C.\nPlease refer to Appendices C.3 and C.6 for the ablation studies on scale adjustment and double quantization.\n\n---\n\n**[W2] Experiments on more architectures**\n\nAs you pointed out, LLaMA and Mistral share similar architectural features. Therefore, we add experiments with Qwen1.5-MoE-A2.7B and Qwen1.5-MoE-A2.7B-Chat models to verify that our method is effective even with the modern MoE (Mixture-of-Experts) architecture. \n\n[1] PPL evaluation result on WikiText-2 and C4 (Qwen1.5-MoE-A2.7B)\n\n| Method         | WikiText-2 | C4     |\n|----------------|-----------|--------|\n| FP16           | 6.79      | 8.88   |\n| NSNQuant-2b    | **7.07**      | **9.31**   |\n| CQ-4c9b        | 7.11      | 9.52   |\n| NSNQuant-1b    | 9.93      | 13.05  |\n| CQ-8c10b       | **8.46**      | **12.41**  |\n\n[2] Evaluation results on HumanEval, GSM8K-CoT, and CoQA (Qwen1.5-MoE-A2.7B-Chat)\n| Method         | HumanEval | GSM8K (8-shot, CoT) | CoQA  |\n|----------------|----------------|--------------------------|----------|\n| FP16           | 29.27          | 52.01                    | 59.62    |\n| NSNQuant-2b    | **26.22**          | **46.78**                    | **59.03**    |\n| CQ-4c9b        | 24.39          | 43.97                    | 58.62    |\n| NSNQuant-1b    | **17.68**          | **23.88**                    | **55.57**    |\n| CQ-8c10b       | 12.20          | 19.26                    | 50.32    |\n\n\nAlthough NSNQuant-1b achieves slightly worse PPL compared to CQ-8c10b, it achieves much better scores on lm-eval tasks. These results reinforce the claim that NSNQuant is the most effective method across different datasets and models.\n\n---\n\n**[Q1] Latency of NSN, Hadamard, and VQ**\n\n\nTo investigate the overhead of each component of NSNQuant, we measure the latency caused by each component: NSN, Hadamard transform and VQ.\nHere, the double quantization is included as part of NSN, and the scale adjustment is grouped under VQ.\nWe evaluate latency under two different settings with LLaMA2-7B, each averaged over 100 runs to ensure stable measurements.\nWe report per-token latency for the decoding stage.\nAll experiments are done on a linux server with a single A100-80GB GPU.\n\n**[1]** Setting 1: batch size = 1, prompt length = 2048, generated tokens = 128 \n\nThis setting is for comparison with CQ. In the CQ paper, authors provide latency measurements on Appendix K. They used the batch size of 1, prompt length of 2000 and generate 100 tokens.\nWe try to match this setting as closely as possible, and set the closest numbers which are divisible by residual size (64).\nSince the experiments are conducted with different GPUs (CQ: A100-40GB, ours: A100-80GB), it's hard to directly compare these results.\nWe instead compare the **relative overhead** added over the FP16 baseline, as the CQ inference kernel is not publicly available.\n\n**[1-1]** Prefill\n| Method         | Total | NSN | Hadamard transform | VQ |\n|----------------|-----------|--------|-------|-----|\n| FP16           | 169.85ms | N/A | N/A | N/A |\n| NSNQuant-2b    | 244.73ms | 22.51ms | 2.97ms | 42.22ms |\n\n**[1-2]** Decode (per token)\n| Method         | Total | NSN | Hadamard transform | VQ |\n|----------------|-----------|--------|-------|-----|\n| FP16           | 28.21ms | N/A | N/A | N/A |\n| NSNQuant-2b    | 33.95ms | 0.30ms | 1.83ms | 0.08ms |\n\nThe result shows that NSNQuant-2b requires 44% and 20% of additional overhead in prefill and decode, respectively, compared to the full precision baseline. \nDespite the additional latency from NSN and Hadamard transforms, the overhead remains lower than CQ (see Table 14 in the CQ paper).\nCQ-4c8b requires nearly twice the prefilling time and incurs a 26% overhead in the decoding stage.\n\nWe attribute this result to the optimized custom kernel implementation.\nIn fact, our VQ implementation can be used in K-Means algorithm, and this can boost the K-Means algorithm needed for CQ.\nSurprisingly, using our implementation, the centroid for CQ-8c10b can be found in 15 minutes with a single RTX-3090, whereas the original CQ paper reports that it requires more than 100 minutes.\n\n\n**[2]** Setting 2: batch Size = 32, prompt length = 512, generated tokens = 64 \n\nThis setting reflects a more realistic deployment scenario, using a moderate batch size and prompt length.\n\n**[2-1]** Prefill\n| Method         | Total | NSN | Hadamard transform | VQ |\n|----------------|-----------|--------|-------|-----|\n| FP16           | 1225.90ms | N/A | N/A | N/A |\n| NSNQuant-2b    | 1707.22ms | 155.22ms | 15.04ms | 326.85ms |\n\n**[2-2]** Decode (per token)\n| Method         | Total | NSN | Hadamard transform | VQ |\n|----------------|-----------|--------|-------|-----|\n| FP16           | 50.80ms | N/A | N/A | N/A |\n| NSNQuant-2b    | 36.49ms | 0.34ms | 1.88ms | 0.66ms |\n\n\nWhile NSNQuant-2b has higher prefilling latency, it achieves lower decoding latency due to reduced DRAM access during attention computation.\n\nInterestingly, both settings show that while VQ has the largest overhead in the prefill stage, Hadamard transform has the largest overhead in the decoding stage. This is similar to MLP layers becoming the main bottleneck in the decoding stage where it becomes memory-bound."}, {"review_id": "gc7ZQ1Dyfc", "reviewer": "Reviewer_K1rM", "summary": "In this study, we first identify the problem of distribution shift in current KV cache vector quantization, and then propose a new vector quantization method called “NSNQuant” that is robust to distribution shift in datasets: normalization and mean shift followed by Hadamard Transformation to approximate standard normal distribution. Experiments have demonstrated that NSNQuant consistently outperforms prior methods in both 1-bit and 2-bit settings.", "questions": "I would like to point out some readability issues which directly affects our understanding of the proposed method.\n- Figure 2: I could not understand the pipeline of the overall structure. Firstly, W_q, W_k, and W_v are model parameters related with attention? or is it a latent representation? Secondly, what is R_Had? Thirdly, what is the mathematical formula before and after softmax? Please explain the detail in natural language.\n- Normalization alongside token dimension (in NSN procedure): I think it is NOT possible when we start inference because in the beginning of the inference, we cannot foresee the future tokens. Please explain the detailed procedure.\n- \"1-bit and 2-bit settings\": I'm not quite sure which part is quantized in the proposal. Is it that each vector element in the codebook is quantized to 1-bit or 2-bit?\n- Figure 3: What is channel distribution? Does it refer to the distribution of values in a single selected channel?", "limitations": "Yes.", "rating": 4, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 2, "significance": 3, "originality": 3, "strengths_and_weaknesses": "Strength:\n- Based on experimental results, the author points out the problems with existing research and argues for the need for new methods.\n- The novelty of the proposed method is directly linked to solving the problem of distributional shift in KV cache vector: normalization and mean shift followed by Hadamard Transformation to approximate standard normal distribution.\n- The effectiveness of the proposed method is demonstrated by conducting comparative experiments with representative quantization baselines.\n- Source code is available for reproducibility.\n\nWeakness:\n- Readability: some of the proposed method is hard to understand. See the details for \"Questions\".", "was_revised": true, "final_justification": "Our remaining concerns are clarified by the author's detailed explanations.\n\nSo, I have raised my score from borderline reject to borderline accept.\n\nI hope the authors will incorporate the information in the revised manuscript version.", "rebuttal": "We sincerely appreciate the time and effort you devoted to reviewing our submission.\nWe acknowledge that some notations of our method section can be misleading and make it hard to understand the paper.\nWe will polish the method section and fix informal notations and expressions based on your thoughtful reviews.\n\n**[Q1] Clarification on Figure 2**\n- **W_q, W_k, and W_v** are the weights of the linear projection layers for query, key, and value, respectively. In HuggingFace Transformers, these correspond to the weights of `q_proj`, `k_proj`, and `v_proj` in the `LlamaAttention` module. We omitted detailed explanation of these notations since they are widely adopted in prior works such as LoRA [1] and SpinQuant [2].\n- **R_Had** refers to the **Hadamard matrix**, and it is used in Figure 2 to visualize the Hadamard transformation. Since applying a Hadamard transform is equivalent to multiplying by a Hadamard matrix, many previous works (e.g., SpinQuant [2], RotateKV [3]) describe this as applying a rotation, using the notation **R** and calling it \"online rotation.\" In our case, we use this notation to indicate that a Hadamard transform is applied in the pipeline. This transformation is linear and can be merged into the projection weights (e.g., `W_v`, `W_o`) to minimize computational overhead, as shown in the figure. We also use the notation **q_Had** to denote the Hadamard-transformed query vector. We acknowledge that these notations might be confusing and will add a clarifying note in the figure caption in the final version.\n- The mathematical formula before and after softmax is explained in Appendix B, as stated in the caption. We explain how each formula is derived using the restoration formula v = s_1(s_2 v_nsn + o). Due to the page limit, we placed this derivation in the appendix, but we will consider including it in the main text in the camera-ready version to improve clarity.\n\n---\n\n**[Q2] Feability of normalization in NSN**\n\nNSN procedure incorporates two types of transformation:\n1. Token-wise normalization: normalize each key or value token to have a scale of sqrt(d)\n2. Channel-wise centering: make each channel dimension have a mean of 0\n\nThe token-wise normalization is feasible because norm of each token can be easily computed online.\nHowever, as you pointed out, channel-wise centering is infeasible because it requires statistics of future tokens.\n\nTo resolve this issue, we adopt a concept of residual from KIVI [3], as explained in the Section 3.2.\nWe split KV cache into two parts: one part with quantized cache and the other part (residual) with full-precision cache for recent tokens.\nOnce the size of residual (which is set to 64 in this paper) reaches its maximum capacity, the tokens within the residual are transformed together through NSN and Hadamard transform, and then quantized using a codebook.\nDuring these steps, channel-wise mean is now calculable since multiple tokens are quantized at once.\nKIVI also adopts residual for a similar reason: they quantize key cache along the channel dimension.\n\nWe omitted residual from Figure 2 for simplicity of the figure.\nHowever, we acknowledge that it can be misleading and will try to visualize the residual in the Figure 2 as well.\n\n---\n\n**[Q3] Regarding the term \"1-bit setting\" and \"2-bit setting\"**\n\nWe used the terms **\"1-bit setting\"** and **\"2-bit setting\"** to refer to the setting where KV cache is quantized into approximately 1 and 2 bits per each value, respectively. In \"k-bit setting\", \"k-bit quantization methods\" are adopted for quantizing KV cache. This categorization is important because average bit width (average bit per value) decides the memory consumption.\nSpecifically:\n\n- Methods with an average bit width around 1 (e.g., **NSNQuant-1b**, **CQ-8c10b**, **KVQuant-1b**) are categorized as **1-bit quantization methods**.\n- Likewise, methods with an average bit width around 2 (e.g., **NSNQuant-2b**, **CQ-4c9b**, **KVQuant-2b**) are categorized as **2-bit quantization methods**.\n\nThese methods compress both **key** and **value** caches into the corresponding bitwidth.\nFor example, in **NSNQuant-1b**, each key or value token is divided into multiple **8-dimensional sub-vectors**, and each subvector is quantized using **8-bit indices** that point to entries in a **256-entry codebook** (each entry being an 8D vector).\nSince **8 bits are used to represent each 8D vector**, this results in an **average of 1 bit per dimension** (i.e., 8 bits / 8 dimensions = 1).\nNote that the average bit width for NSNQuant-1b is slightly larger than 1 because of additional parameters incurred by NSN.\n\n---\n\n**[Q4] Regarding the term \"channel distribution\" (Figure 3)**\n\nYes, we use the expression \"channel distribution\" to denote the distribution of values in a single selected channel.\nWe acknowledge that the term may be ambiguous, and we will revise it to a clearer phrase like \"per-channel value distribution\" in the final version.\n\n---\n\nWe sincerely thank the reviewer for pointing out the readability issues.\nYour feedback is valuable and will help us significantly improve the clarity of the paper.\nWe truly appreciate the opportunity to further refine our work based on your suggestions.\nWe hope these answers help you understand the paper in depth.\nIf you have any additional questions or need further clarification, we would be glad to provide more details.\n\n---\n**References**  \n[1] Lora: Low-rank adaptation of large language models, Hu, Edward J., et al., ICLR 2022  \n[2] Spinquant: Llm quantization with learned rotations, Liu, Zechun, et al., ICLR 2025.  \n[3] KIVI: A Tuning-Free Asymmetric 2bit Quantization for KV Cache, Liu, Zirui, et al., ICML 2024"}]

Author rebuttal:

Meta-review: The paper studies the problem of KV cache compression using vector quantization (VQ) techniques. It first makes an important observation that the VQ techniques that rely on calibration data to learn the codebook don't generalize well to OOD data. To address this, the paper first transforms the keys, and values to an approximately Gaussian distribution. It then relies on a codebook learned for Gaussian data to compress the transformed keys, and values. This technique is calibration data free and doesn't face the OOD issues faced by traditional VQ techniques that  rely on calibration data.  Experimental results show that the proposed technique generalizes well to diverse datasets.

All the reviewers appreciated the novelty in the work, and the thorough experimental evaluation, and are inclined towards accepting it. However, there are some key limitations of the proposed technique which need to be addressed in the paper.
 - We ideally want a single Hadamard matrix to transform all our data into approximate Gaussian distribution. It's not clear if this is even feasible. (Lemma 1 doesn't address this because it considers a random Hadamard transform.)  Intuitively, one can always construct data points where a given Hadamard transformation can fail to yield gaussian distribution. If that happens quite frequently, the proposed technique can break too. Unfortunately, the experiments in the paper don't address this. They are conduced only on two datasets (wikitext and C4). It would be good if the authors show how the performance varies on diverse datasets. 
 - As pointed out by multiple reviewers, the additional transformations performed on keys, values increase the latency of the proposed technique. While the authors addressed this to some extent in their rebuttal, I would prefer having a thorough discussion on this in the main paper (including long context latency evals). 

I urge the authors to incorporate this feedback into the final camera ready version.
Decision: Accept (poster)