Title: NSNQuant: A Double Normalization Approach for Calibration-Free Low-Bit Vector Quantization of KV Cache
Venue: NeurIPS 2025
Accepted: yes
Keywords: KV cache quantization

vector quantization

large language model

inference optimization
Abstract: Large Language Model (LLM) inference is typically memory-intensive, especially when processing large batch sizes and long sequences, due to the large size of key-value (KV) cache.
Vector Quantization (VQ) is recently adopted to alleviate this issue, but we find that the existing approach is susceptible to distribution shift due to its reliance on calibration datasets.
To address this limitation, we introduce $\textbf{NSNQuant}$, a calibration-free Vector Quantization (VQ) technique designed for low-bit compression of the KV cache. 
By applying a three-step transformation—$\textbf{1)}$ a token-wise normalization ($\textbf{N}$ormalize), $\textbf{2)}$ a channel-wise centering ($\textbf{S}$hift), and $\textbf{3)}$ a second token-wise normalization ($\textbf{N}$ormalize)—with Hadamard transform, NSNQuant effectively aligns the token distribution with the standard normal distribution. 
This alignment enables robust, calibration-free vector quantization using a single reusable codebook.
Extensive experiments show that NSNQuant consistently outperforms prior methods in both 1-bit and 2-bit settings, offering strong generalization and up to 3$\times$ throughput gains over full-precision baselines.


Peer reviews:
{"IS_META_REVIEW": false, "comments": "SUMMARY:\nIn this study, we first identify the problem of distribution shift in current KV cache vector quantization, and then propose a new vector quantization method called “NSNQuant” that is robust to distribution shift in datasets: normalization and mean shift followed by Hadamard Transformation to approximate standard normal distribution. Experiments have demonstrated that NSNQuant consistently outperforms prior methods in both 1-bit and 2-bit settings.\n\nSTRENGTHS AND WEAKNESSES:\nStrength:\n- Based on experimental results, the author points out the problems with existing research and argues for the need for new methods.\n- The novelty of the proposed method is directly linked to solving the problem of distributional shift in KV cache vector: normalization and mean shift followed by Hadamard Transformation to approximate standard normal distribution.\n- The effectiveness of the proposed method is demonstrated by conducting comparative experiments with representative quantization baselines.\n- Source code is available for reproducibility.\n\nWeakness:\n- Readability: some of the proposed method is hard to understand. See the details for \"Questions\".\n\nQUESTIONS:\nI would like to point out some readability issues which directly affects our understanding of the proposed method.\n- Figure 2: I could not understand the pipeline of the overall structure. Firstly, W_q, W_k, and W_v are model parameters related with attention? or is it a latent representation? Secondly, what is R_Had? Thirdly, what is the mathematical formula before and after softmax? Please explain the detail in natural language.\n- Normalization alongside token dimension (in NSN procedure): I think it is NOT possible when we start inference because in the beginning of the inference, we cannot foresee the future tokens. Please explain the detailed procedure.\n- \"1-bit and 2-bit settings\": I'm not quite sure which part is quantized in the proposal. Is it that each vector element in the codebook is quantized to 1-bit or 2-bit?\n- Figure 3: What is channel distribution? Does it refer to the distribution of values in a single selected channel?\n\nLIMITATIONS:\nYes.", "TITLE": "Official Review", "RECOMMENDATION": 4, "REVIEWER_CONFIDENCE": 3, "CLARITY": 2, "IMPACT": 3, "ORIGINALITY": 3, "DATE": "2025-06-17"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nIt is a well-known problem that memory issue is the major bottleneck of LLM (Large Language Model) inference because LLM inference is memory-bounded. Recently, a lot of work has emerged to resolve this issue, and one of the representative solutions is KV cache quantization.\n\nThis work points out that the distribution of the KV cache differs according to inputs, thus the distributions of the KV cache shift depending on the calibration set. It leads to performance degradation in general areas. To resolve this distribution shift problem, this work proposes token-wise normalization, following channel-wise centering and token-wise normalization again. With the above process, the distributions of tokens become standard normal distributions, and it prevents distribution shifts of the KV cache after quantization. Also, by applying Hadamard transform and scale adjustment, the paper achieves state-of-the-art results among other KV cache compression works.\n\nSTRENGTHS AND WEAKNESSES:\nStrengths\n- This work shows the distribution shift problem that occurred while applying vector quantization to the KV cache.\n- This work resolves the above problem in a simple yet effective way; add token-wise normalization, channel-wise shift, and token-wise normalization to the KV cache, along with Hadamard transform.\n- This work provides optimized kernels to execute vector quantization, scale adjustment, and attention with quantized KV caches.\n\nWeaknesses\n- Along with the proposed method, NSNQuant and Hadamard transform, this paper applies other techniques such as scale adjustment and double quantization. However, the effect of those suggestions is not addressed in the experiments.\n- Llama and Mistral have similar architectural features (e.g., attention architecture, activation function, and normalization function). The paper should have examined other LLM models that have different architectural features to verify the general effectiveness of the proposal.\n\nQUESTIONS:\n- How much is the overhead of NSNQuant and Hadamard transform? The paper compares memory consumption and throughput with 16-bit results. Is the overhead negligible compared to previous KV cache quantization works? Or, those proposals are harmful to the overall memory consumption or throughput?\n\nLIMITATIONS:\nYes", "TITLE": "Official Review", "RECOMMENDATION": 4, "REVIEWER_CONFIDENCE": 4, "CLARITY": 3, "IMPACT": 2, "ORIGINALITY": 2, "DATE": "2025-06-30"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper proposes NSNQuant, a novel calibration-free vector quantization method for KV Cache. Its key innovation is the Normalization-Shift-Normalization module, which aligns channel distributions with the standard normal distribution. This approach addresses a key limitation of prior methods, the reliance on specific calibration datasets, which limits generalizability. The method demonstrates superior accuracy and efficiency compared to existing techniques through comprehensive evaluation.\n\nSTRENGTHS AND WEAKNESSES:\nStrengths:\n\n1. The paper introduces a novel calibration-free vector quantization method. It incorporates the NSN module to align channel distributions with the standard normal distribution. This is a significant innovation addressing a core limitation of existing methods that depend on calibration datasets.\n\n2. The paper provides a comprehensive evaluation of the proposed method across various datasets, demonstrating its superior performance over previous methods. It also includes a well-structured ablation study validating the effectiveness of its components.\n\n3. The paper proves the mathematical effectiveness of the proposed module. It also provides a CUDA implementation to enhance inference efficiency.\n\nWeaknesses:\n\n1. This method is sensitivite to the choice of hyperparameters, e.g., residual size when quantization bit is 1.\n2. This method incur extra latency due to the additional normalization steps especially in prefilling stage and the paper does not provide a detailed analysis of its latency impact.\n\nQUESTIONS:\n1. Why is the order of applying NSN different for keys and values? Is there a specific reason or theoretical analysis supporting this design choice?\n2. Which dataset is used for codebook fine-tuning? How does the choice of dataset impact performance and generalization ability?\n3. Since CQ was calibrated on WikiText-2, why does it still underperform compared to NSNQuant?\n\nLIMITATIONS:\nYes", "TITLE": "Official Review", "RECOMMENDATION": 4, "REVIEWER_CONFIDENCE": 3, "CLARITY": 3, "IMPACT": 3, "ORIGINALITY": 3, "DATE": "2025-06-30"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe authors propose NSNQuant, a calibration-free vector quantization (VQ) method aimed at addressing the distribution-shift sensitivity observed in previous KV cache quantization methods such as CQ (Zhang et al., 2024). NSNQuant utilizes a three-step Normalize-Shift-Normalize (NSN) transformation combined with a Hadamard transform to align key-value (KV) vectors to a standard normal distribution. Consequently, this allows the codebooks to be trained solely on synthetic Gaussian data, bypassing the need for calibration datasets. Extensive experiments demonstrate that NSNQuant outperforms existing calibration-based approaches across various benchmarks.\n\nSTRENGTHS AND WEAKNESSES:\nStrengths: \n- The proposed idea of aligning KV states to a known distribution prior is novel and can effectively removes the calibration requirement.\n- The authors show a variance bound on the transformed key/value states in Lemma 1.\n- Experiments are comprehensive, clearly illustrating NSNQuant’s superiority over existing quantization baselines, particularly its robustness to distribution shifts which notably improves upon CQ.  \n\nWeaknesses: \n- Efficiency evaluation is inadequate. Since NSNQuant involves additional computational steps (NSN and Hadamard transformations) during inference, it is essential to measure and clearly report the latency overhead. The absence of detailed latency comparison with calibration-based methods, such as CQ, weakens the practical appeal of the proposed approach. \n- The overall presentation of the paper needs improvement, e.g., Table 1 is never explicitly referenced or discussed in the main text, which is somewhat confusing.\n\nQUESTIONS:\n- As noted in weaknesses, how much additional latency do the proposed transformations (NSN and Hadamard) introduce during inference compared to CQ? Is the overhead negligible in practice? \n- The authors mention that early layers exhibit higher variance after transformation. What might be the possible causes for this phenomenon? Is there a risk that quantization errors in these early layers propagate and compound across subsequent layers? \n- In the case of CQ, would scaling up the calibration set help reduce out-of-distribution errors? While NSNQuant benefits from faster training of the centroids, KV cache applications prioritize their inference efficiency. In such scenarios, how does NSNQuant compare to CQ if calibration can be scaled effectively?\n- VQ is powerful for compression due to its strong representation capacity. While enforcing a prior enables calibration-free codebook training, could this constraint limit VQ’s expressiveness? For example, for the early layers, would finetuning the codebook with calibration sets further mitigate the quantization error?\n\nLIMITATIONS:\nThe authors have discussed limitations of their study properly in the appendix.", "TITLE": "Official Review", "RECOMMENDATION": 4, "REVIEWER_CONFIDENCE": 4, "CLARITY": 2, "IMPACT": 2, "ORIGINALITY": 3, "DATE": "2025-07-02"}