Title: SALS: Sparse Attention in Latent Space for KV Cache Compression
Venue: NeurIPS 2025
Accepted: yes
Keywords: KV Cache Compression; Sparse attention; Low-rank projection
Abstract: Large Language Models (LLMs) capable of handling extended contexts are in high demand, yet their inference remains challenging due to substantial Key-Value (KV) cache size and high memory bandwidth requirements. Previous research has demonstrated that KV cache exhibits low-rank characteristics within the hidden dimension, suggesting the potential for effective compression. However, due to the widely adopted Rotary Position Embedding (RoPE) mechanism in modern LLMs, naive low‑-rank compression suffers severe accuracy degradation or creates a new speed bottleneck, as the low-rank cache must first be reconstructed in order to apply RoPE. In this paper, we introduce two key insights: first, the application of RoPE to the key vectors increases their variance, which in turn results in a higher rank; second, after the key vectors are transformed into the latent space, they largely maintain their representation across most layers. Based on these insights, we propose the Sparse Attention in Latent Space (SALS) framework. SALS projects the KV cache into a compact latent space via low-rank projection, and performs sparse token selection using RoPE-free query--key interactions in this space. By reconstructing only a small subset of important tokens, it avoids the overhead of full KV cache reconstruction. We comprehensively evaluate SALS on various tasks using two large-scale models: LLaMA2-7b-chat and Mistral-7b, and additionally verify its scalability on the RULER-128k benchmark with LLaMA3.1-8B-Instruct. Experimental results demonstrate that SALS achieves SOTA performance by maintaining competitive accuracy. Under different settings, SALS achieves 6.4-fold KV cache compression and 5.7-fold speed-up in the attention operator compared to FlashAttention2 on the 4K sequence. For the end-to-end throughput performance, we achieves 1.4-fold and 4.5-fold improvement compared to GPT-fast on 4k and 32K sequences, respectively. The source code will be publicly available in the future.


Peer reviews:
{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe authors introduce Sparse Attention in Latent Space (SALS), a lightweight framework that reduces both storage and computation during inference by compressing the Key–Value (KV) cache. Rather than applying Rotary Position Embeddings (RoPE) before compression — which inflates the effective rank of key matrices — SALS first projects all multi-head key vectors into a shared, low-dimensional latent space via a rank-r eigen-projection. In this latent domain, approximate attention scores are computed using only the top coordinates, enabling selection of a small set of the most salient tokens. Only the corresponding top-k key–value pairs are then reconstructed back into their original space and attended using the standard mechanism, yielding a sparse attention over a highly compact cache representation. Empirical evaluation on LLaMA2-7B-Chat and Mistral-7B across benchmarks such as GSM8K, CoQA, and LongBench shows that SALS matches or surpasses the accuracy of FlashAttention2 and GPT-Fast baselines while achieving up to a 6.4× reduction in KV-cache size, a 5.7× speed-up in the attention operator at 4K context length, and overall 1.4–4.5× gains in end-to-end throughput for contexts ranging from 4K to 32K tokens. These results demonstrate that SALS can be seamlessly integrated into existing large-language-model pipelines to extend context lengths with minimal performance trade-offs.\n\nSTRENGTHS AND WEAKNESSES:\nStrengths:\n\n+ The authors present a rigorously derived three-stage pipeline, including latent-space projection, critical-token selection, and selective reconstruction, that is mathematically sound and clearly justified. Their use of eigen-projection to compress pre-RoPE key vectors is underpinned by solid linear-algebraic reasoning, and the paper carefully measures the approximation error at each stage, demonstrating high fidelity to full attention.\n+ By achieving up to 6.4× KV-cache compression and 1.4 - 4.5× end-to-end throughput gains on LLaMA2-7B-Chat and Mistral-7B, SALS addresses one of the most pressing bottlenecks in scaling transformer contexts.\n+ The core idea of performing latent-space attention scoring prior to RoPE - thus sidestepping rank inflation - is a novel twist on existing compression and sparsity techniques. While latent projections and sparse attention have each seen prior study, their careful integration here constitutes a fresh contribution.\n\nWeaknesses:\n\n- Although the pipeline is robust, the overall methodological novelty is somewhat incremental: SALS combines two well-explored ideas (latent compression and sparse token selection) without introducing fundamentally new theory.\n\n- The paper does not deeply investigate how SALS interacts with grouped-query attention (GQA) and multi-query attention (MQA) variants. Although a GQA-based model is included in the experimental suite, the efficiency analysis remains focused on the multi-head-attention (MHA) case and fails to articulate how the latent-space selection and sparse reconstruction behave under GQA/MQA. This omission leaves open the possibility that the reported performance gains may not generalize beyond the standard MHA setting.\n\n- While throughput and memory gains are impressive, the experimental evaluation lacks comparisons to several leading sparse-attention frameworks, such as Quest, H2O and ArkVale that also target long contexts.\n\nQUESTIONS:\nQ1: It would strengthen the empirical foundation to report SALS’s performance alongside contemporary sparse-attention methods, especially dynamic selection method like Quest or ArkVale.\n\nQ2: Although you include a GQA-based model, the efficiency analysis may remain tailored to standard multi-head attention. Providing benchmarks on grouped-query attention layers — reporting memory access and throughput — would clarify SALS’s behavior under these widely used architectures.\n\nQ3: Table 3 appears without narrative discussion, leaving its per-task metrics under-interpreted. A brief in-text analysis defining each column, highlighting key trends across Single-QA/Multi-QA/Summarization, and relating memory-access reductions to latency savings could improve clarity.\n\nLIMITATIONS:\nYes", "TITLE": "Official Review", "RECOMMENDATION": 4, "REVIEWER_CONFIDENCE": 4, "CLARITY": 3, "IMPACT": 2, "ORIGINALITY": 2, "DATE": "2025-06-27"}

comments: SUMMARY:
The paper introduces a framework named SALS to compress the KV cache and accelerate the generation of LLMs. It first analyzes the impact of RoPE on QK low rank and adopts a pre-RoPE low rank strategy. To build the framework, it transforms queries and keys into latent space, selects critical tokens in latent space and reconstructs them for sparse attention. Evaluations demonstrates the effectiveness and efficiency of SALS.

STRENGTHS AND WEAKNESSES:
Strengths:
1) The paper is clearly written and easy to follow, with sufficient empirical observations demonstrating the motivation for each design.

2) The proposed framework is carefully designed and demonstrates strong performance.


Weaknesses:
1) The value tensors remain in full rank. Despite the paper adopts quantization to compress value cache, it complicates the overall framework. Experimental results with low rank value caches should be provided.
2) A discussion between SALS and NSA [1] should be provided.
3) Incorporating comparisons of multiple value cache quantization methods (e.g. [2-3]) can further enhance the robustness of the SALS.


[1] Native Sparse Attention: Hardware-Aligned and Natively Trainable Sparse Attention. Arxiv 2502.11089.

[2] ZipCache: Accurate and Efficient KV Cache Quantization with Salient Token Identification. NeurIPS 2024.

[3] GEAR: An Efficient KV Cache Compression Recipe for Near-Lossless Generative Inference of LLM. Arxiv 2403.05527.

QUESTIONS:
See weaknesses.

LIMITATIONS:
See weaknesses.
TITLE: Official Review
CLARITY: 3
IMPACT: 3
ORIGINALITY: 3
DATE: 2025-07-01

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe paper proposes SALS, a method to compress the KV cache in large language models by combining low-rank projection and token sparsity. Instead of reconstructing the full KV cache, SALS projects pre-RoPE keys into a compact latent space, selects top-k important tokens using lightweight scoring, and reconstructs only those tokens for final attention. This approach significantly reduces memory and compute without sacrificing accuracy. Experiments on Llama-2 and Mistral showcase that SALS achieve better accuracy and efficiency than existing baselines.\n\nSTRENGTHS AND WEAKNESSES:\n### Strengths\n\n1. This work provides good system support, showcasing great speedups relative to strong baselines such as FlashAttention.\n2. The proposed approach is well-motivated through careful analysis.\n3. Theoretical justifications are provided for using multi–head joint projection matrix.\n\n### Weaknesses:\n\n1. The paper reports memory access as a primary metric, but this does not directly capture the efficiency of the proposed method. It would be more informative to report latency or throughput for SALS and compare it with baseline methods.\n\n2. Most experiments use relatively short context lengths (1K, 2K, 4K as noted on line 301). It would strengthen the evaluation to include results on longer contexts, such as 32K tokens or more, which are more representative of long-context use cases.\n\n3. The contributions appear limited. The core idea of using low-rank projections to identify top-k tokens resembles existing methods like Loki [1]. The paper would benefit from a clearer explanation of what distinguishes SALS from these prior approaches.\n\n[1] Loki: Low-rank Keys for Efficient Sparse Attention\n\nQUESTIONS:\n- You project all multi-head keys into a shared latent space using a joint low-rank projection. Have you considered whether using per-head projections might improve token selection accuracy, despite the increased overhead?\n- SALS is conceptually similar to Loki, which also performs low-rank key compression and token selection. Could you clarify the key differences in token scoring or reconstruction strategies between SALS and Loki?\n\nLIMITATIONS:\nYes.", "TITLE": "Official Review", "RECOMMENDATION": 3, "REVIEWER_CONFIDENCE": 3, "CLARITY": 2, "IMPACT": 2, "ORIGINALITY": 2, "DATE": "2025-07-02"}

comments: SUMMARY:
SALS is a framework designed to improve inference efficiency for Large Language Models (LLMs) with long contexts by compressing the large Key-Value (KV) cache. It leverages two insights: RoPE increases key vector variance (raising rank), and key representations remain stable across layers. SALS projects the KV cache into a compact latent space using low-rank projection and performs sparse token selection via RoPE-free interactions, reconstructing only key tokens. This avoids full KV reconstruction overhead. Evaluated on LLaMA2-7b-chat and Mistral-7b, SALS achieves up to 6.4× KV compression, 5.7× faster attention, and up to 4.5× end-to-end speedup, while maintaining competitive accuracy.

STRENGTHS AND WEAKNESSES:
Strengths:
1. The paper is easy to follow.
2. The baseline implementations (GPT-Fast) are valid.
3. This paper discusses an important problem of low rank KV compression suffering severe accuracy degradation. 

Weaknesses:

1. The evaluation models (Llama2-7b and mistral) are even not long-context models (Llama3.1 is more prefered).
2. The evaluation dataset is not standard. RULER, InfiniBench, and GSM-Infinite are more convincing.

QUESTIONS:
1. Can you provide numbers to justify Table 1?
2. Additional experiments as stated in the weaknesses.
3. InfiniGen, ShadowKV also employs low-rank compression for KV cache. Could you provide any insights about the core differences?.

LIMITATIONS:
There are many limitations generally for sparse attention. For example, the speed up is limited by how much memory taken by KV cache. I think this paper does not discuss tlimitations in the work.
TITLE: Official Review
CLARITY: 3
IMPACT: 2
ORIGINALITY: 3
DATE: 2025-07-04