SALS: Sparse Attention in Latent Space for KV Cache Compression
Authors: Junlin Mu
Hantao Huang
Jihang Zhang
Minghui Yu
Tao Wang
Yidong Li
Venue: NeurIPS 2025 poster ()
Keywords: KV Cache Compression; Sparse attention; Low-rank projection
Primary area: deep_learning
TL;DR: We propose SALS, a sparse attention framework in latent space that enables low-rank KV cache compression with minimal reconstruction overhead, achieving up to 6.4× compression and 5.7× attention speed-up without sacrificing accuracy.

Abstract: Large Language Models (LLMs) capable of handling extended contexts are in high demand, yet their inference remains challenging due to substantial Key-Value (KV) cache size and high memory bandwidth requirements. Previous research has demonstrated that KV cache exhibits low-rank characteristics within the hidden dimension, suggesting the potential for effective compression. However, due to the widely adopted Rotary Position Embedding (RoPE) mechanism in modern LLMs, naive low‑-rank compression suffers severe accuracy degradation or creates a new speed bottleneck, as the low-rank cache must first be reconstructed in order to apply RoPE. In this paper, we introduce two key insights: first, the application of RoPE to the key vectors increases their variance, which in turn results in a higher rank; second, after the key vectors are transformed into the latent space, they largely maintain their representation across most layers. Based on these insights, we propose the Sparse Attention in Latent Space (SALS) framework. SALS projects the KV cache into a compact latent space via low-rank projection, and performs sparse token selection using RoPE-free query--key interactions in this space. By reconstructing only a small subset of important tokens, it avoids the overhead of full KV cache reconstruction. We comprehensively evaluate SALS on various tasks using two large-scale models: LLaMA2-7b-chat and Mistral-7b, and additionally verify its scalability on the RULER-128k benchmark with LLaMA3.1-8B-Instruct. Experimental results demonstrate that SALS achieves SOTA performance by maintaining competitive accuracy. Under different settings, SALS achieves 6.4-fold KV cache compression and 5.7-fold speed-up in the attention operator compared to FlashAttention2 on the 4K sequence. For the end-to-end throughput performance, we achieves 1.4-fold and 4.5-fold improvement compared to GPT-fast on 4k and 32K sequences, respectively. The source code will be publicly available in the future.

SALS: Sparse Attention in Latent Space for KV cache Compression
Junlin Mu 1233; Hantao Huang"t Jihang Zhang', Minghui Yu Tao Wang Yidong Li Key Laboratory of Big Data & Artificial Intelligence in Transportation, Ministry of Education, China Scchoo of Computer Science & Technology, Beijing Jiaotong University, China ByyteDance Seed, China
Abstract
Large Language Models (LLMs) capable of handling extended contexts are in high demand, yet their inference remains challenging due to substantial Key-Value (KV) cache size and high memory bandwidth requirements. Previous research has demonstrated that KV cache exhibits low-rank characteristics within the hid- den dimension, suggesting the potential for effective compression. However, due to the widely adopted Rotary Position Embedding (RoPE) mechanism in modern LLMs, naive low-rank compression suffers severe accuracy degradation ates a new speed bottleneck, as the low-rank cache must first be reconstructed in order to apply RoPE. In this paper, introduce two key insights: first, the application of RoPE to the key vectors increases their variance, which in turn re- sults in higher rank; second, after the key vectors transformed into the latent space, they largely maintain their representation across most layers. Based on these insights, we propose the Sparse Attention in Latent Space (SALS) frame- work. SALS projects the KV cache into a compact latent space via low-rank projection, and performs sparse token selection using RoPE-free query-key in- teractions in this space. By reconstructing only a small subset of important to- kens, avoids the overhead of full KV cache reconstruction. We comprehensively evaluate SALS on various tasks using two large-scale models: LLaMA2-7b-oo and Mistral-7b, 7b, and additionally verify its scalability on the RULER- 128K bench- mark with I.a.MA3.1 1-88- Experimental results demonstrate that SALS achieves SOTA performance by maintaining competitive accuracy. Under differ- ent settings, SALS achieves 6.4-fold KV cache compression and 5.7-fold speed-up in the attention operator compared to FlashAttention2 on the 4K sequence. For the end-to-end throughput performance, ment compared to GPT-fast on 4k and 32K sequences, respectively. The source code will be publicly available in the future.
achieves 1.4-fold and 4.5-fold improve-
Introduction
The groundbreaking success Large Language Models (LLMs), such as ChatGPT [05]] and Claude [II], is transforming information retrieval in many service requests is creating an unprecedented demand for inference optimization algorithms, espe- cially for long-context applications. In particular, many research efforts [[4] 20, 28, [33] show that the Key-Valur cache (KV cache) acts as a major performance bottleneck in LLM serving. As the sequence length increases, the KV cache consumes large portion of GPU device memory. This
The exponential growth in LLM
"Work done ByteDance
Conresoonding authors. authors.twang @ bjtu.edu.cn, biuuu..d....buunggaaaaaay...
@eytedance.com. waw.aamcee.oo.cliieieee. wwt..yydd@@@ju..d....
39th Conference on Neural Information Processing Systems (NcurIPS 2025).

---

Table 1: KV data movement, memory cost and complexity comparison of quantization, low rank
and token sparse method that is with fixed, dynamic and local token selection strategy.
Name
Methods
KV data movement
Memory size
Computation Complexity
Accuracy
Palu [3]
Low Rank
Median
Low
High
Low
Loki A0
Dynamic
Rank
Low
Median
Median
Median
StreamingLLMM [53
Fixed pattern
Low
Median
Low
Low
Quest [[2]]
Dynumic
Low
High
Median
Median
DS (29
Dynamic
Low
Median
Median
High
Habare [6)
Dynamic
Low
Median
Median
High
SALS (Ours)
Dynamic+Low Rank
Low
Low
Midian
High
high resouree requirement underscores the urgent need to mitigate the KV cache memory overhead and further improve attention efficiency.
To tackle the KV cache overhead challenge, previous works [L 20J show that the KV cache exhibits low-rank characteristics within the hidden dimension, and thus can be effectively compressed in the latent space. Palu [H] further shows that there exists clear trade-off between accuracy and reconstruction overhead for multi-head attention. Compressing each KV head separately reduces the overhead but the cost of losing accuracy. However, compressing all heads together by low- rank projection improves accuracy by preserving the global information but results in significantly increased reconstruction cost. In this work, show that sparsely selecting small subset of tokens from the KV cache significantly reduces reconstruction error while preserving model accuracy.
Recently, Rotary Position Embedding (RoPE) [20]] has been widely adopted in LLMs; it introduces sinusoidal positional information by multiplying query and key states with rotation matrices. This not only prevents the fusion of low rank weights into the query state, but also requires the reconstruc- tion of key states from the latent space. In this work, we observe that the application of RoPE to the key vectors increases their variance, which in turn results in higher rank. Therefore, the KV cache must be compressed before applying RoPE, which leads to high reconstruction complexity. We fur- ther observe that the key vectors in the latent space largely preserve their representation across most layers. Inspired by Double Sparse [29], we use these compressed key vectors as token selection guidance, so that only the selected tokens need to be reconstructed, which significantly reduces the reconstruction complexity.
In this paper, we propose the Sparse Attention in Latent Space (SALS) framework. SALS projects all attention heads into a shared single-head latent space via low-rank projection, for both Pre-RoPE queries and pre-Ro tre-RoPE keys. The approximation attention scores computed in the latent space and the top-k tokens are selected accordingly. These selected tokens are then reconstructed and applied with RoPE for the final attention computation. By reconstructing only small subset of all tokens, it avoids the overhead of full KV cache reconstruction.
In summary this paper makes the contributions as follow:
We observe that applying RoPE to key vectors leads to increased variance, resulting in higher rank and lower compression rates. In key vectors before RoPE can still maintain the represen- tation for the critical token selection.
We propose the Sparse Attention in Latent Space (SALS) framework to utilize the low-rank Pre-ROPE KV cache compression and further to adopt the KV cache in latent space to select critical tokens. The sparse attention is then proposed with reduced KV data movement and computational complexity.
We comprehensively evaluate SALS on various tasks using two large-scale models: LLaMA2- 7b-chat and Mistral-7b, and additionally verify its scalability on the RULER- 128k benchmark with
Experimental results show that we can achieves much higher ac- curacy comparing to the low-rank based KV cache compression. Comparing to sparse attention works, under the similar benchmark accuracy, SALS achieves 6.4-fold KV cache compression and 5.7-fold speed-up in attention operator compared to FlashAttention2 on 4K sequence. For the end-to-end throughput performance, SALS achieves 1.4-fold and 4.5-fold improvement com- pared to GPT-fast on 4k and 32K sequences respectively.

---

Inference Time
Module Across Methods
Effect RoPE PCA Directions and Variance
25
Recover
bask (only SALS)
Sobmax
y-pe
20
Refore Appaing RoFE After Applying RoP
pwak
26-
Ovethads
Lowrnnk mecoar uttd
attrtton
SALS 01aeto0
apprtion
10k
32k 64k 128k
(8)
Figure I: (a) Increasing inference time due to low-rank matrix reconstruction. Low-rank KV cache with overhead leads to longer inference time than the standard attention due to the reconstruction overhead; (b) A simplifed key vector example with changed PCA direction after applying RoPE
Related work
2.1
Attention in Latent Space
The low rank matrix decomposition such singular value decomposition (SVD) compress LLMs has been actively studied. Existing works such as ASVD [3I]], SVD-LLL [24]] and CALDERA [[S]] are proposed compress LLM weights. However, as the context sequence length and batch size increase, the KV cache size is getting larger than LLM weights, urging the need for KV cache com- pression. Eigen Attention [19] has been proposed compress the KV cache after applying RoPE but suffers relatively large accuracy loss. To improve the accuracy, the most naive way is to com- press the pre-RoPE KV cache. However, as Palu [[]] points out, this will greatly introduce additional computation for recovering the key vectors. Palu further proposes the grouped-head low-rank de- composition optimization to reduce the computation overhead. We agree with Palu's observation but tackle the computation from the sparse perspective. By only selecting a subset of all the to- kens (the critical tokens), low-rank key vectors reconstruction complexity is greatly reduced. By integrating sparse attention, we observe that the KV cache size, data movement and computation complexity are all reduced as summarized Table
2.2 Sparse Attention
Early efforts to reduce attention complexity in transformers, such Sparse Transformer [S]], Re- former [122), and Longformer [3], primarily focused on training- basedd approaches to achieve sparse attention. In contrast, recent studies [[77 22 63) have explored post- post-training sparse attention, lever- aging the inherent sparsity of attention scores to identify the most salient tokens without retraining. For instance, StreamingLLM [26] identifies initial and recent tokens critical, while others [52] 33] accumulate attention scores to locate important tokens in the KV cache. Quest [22] assesses KV cache page importance via min/max metries, and Double-sparse [ selects key KV tokens across important channels. However, most of these post-training approaches mainly reduce bandwidth and computation cost, leaving the KV cache uncompressed and thus potential bottleneck [20, 28).
Natively Sparse Attention (NSA) [BI] introduces a dynamic hierarchical sparse strategy combining coarse-grained token compression with fine-grained token selection. Unlike SALS, NSA requires training from scratch to learn a compressed attention module. In contrast, SALS can be calibrated post-training, performing top-k token selection based low-rank latent approximations rather than a trained sparse module. This allows SALS to achieve comparable sparsity behavior without retrain- ing and with precise control over the KV-cache compression process.
Overall, our work addresses the limitation of prior sparse attention methods by storing and operat- ing on low-rank KV cache, then selectively reconstructing high-dimensional representations for exact sparse attention. This design preserves accuracy while substantially reducing memory and computation overhead.
3

---

3 Challenges on Latent Space Transformation
In this section, we first review the RoPE mechanism preventing the fusion of query and key states in latent space. We also show that after RoPE, the key vector variance increases as well as the rank. We then analyze the pre-RoPE representational capability of tokens in the latent space, focusing on their ability to preserve key semantic features.
3.1 Latent Space Transformation with RoPE
The introduction of the RoPE mechanism complicates the low-rank transformation into the latent space. We begin by considering the case without RoPE. Let K E IR*xd denote the key matrix, where is the number of tokens and d is the dimensionality of each head. By applying an orthonormal projection matrix U E RRd with r & d, we can transforme origin Key matrix into latent space:
==
so that each original Key vector K; E Ed is mapped latent space k, I' leading kv cache compression rate d/r. This method works well on the basis of that
UUTZL QK QUU'KT
(2)
However, once RoPE is introduced, the relative order between the RoPE rotation matrix R. and the lowrank projector UU becomes crucial. In Equation B. show both design choices in single expression: (i) pre-RoPE transformation, where keys are first transformed and then rotated, and (ii) post-RoPE transformation, where keys rotated first and transformed afterwards. For query position and a key at position j. the attention score he approximated in two lowrank forms:
Q, KJ ~ QR. U KT or QR.(UU K
pre-RoPE
post-RoPE
where R; and R, are the RoPE rotation matrices for positions i and j, respectively.
Ideally, we can adopt the post-RoPE transformation to cache K = KR, U and transform the rotated query into latent space Q = QR U to avoid the reconstruction complexity. However, discuss later, applying RoPE will rotate the key vectors with larger variance, making them difficult approximate using a low-rank matrix. On the other hand, in the pre-RoPE transformation, caching K and reconstructing the full-rank keys as KUT will result in substantial overhead.
will
Variance amplification for post-RoPE rotation. Previous works [L 20J observe that the key cache exhibits low-rank characteristics with relatively small rank retain approximately 90% of the en- ergy. As such, we can use single projection weight U as shown in Equation I to compress the key cache. However, we find that this phenomenon only holds for the key cache prior to applying RoPE. From Figure Dibb,, we observe that applying RoPE to a set of key vectors pushes the data points outward and rotates their principal axes. As shown in Figure [I(b), the principal component is rotates away from its original direction and the points become more scattered with two main prin- cipal components. This indicates the increased rank due to applying RoPE. Moreover, the principal components rotate based on the token position, suggesting that single fixed projection matrix may no longer approximate transformation is preferable for maintaining accuracy.
these rotated subspaces. Therefore, a Pre-RoPE key for latent space
Compute overhead during RoP- reconstruction From Figure I(a), we compare full atten- tion with Pre-RoPE low-rank compression across sequence lengths ranging from 1 K to 32 K to- kens. While low-rank compression can reduce memory size, the time spent reconstructing low-rank keys/values from their rank-r form and applying the RoPE rotation grows linearly with sequence length and soon dominates the total attention runtime 32 tokens. This aligns with our analysis of Equation & which intruoduces a off of memory savings and computational cost.
3.2 Token Representation in Latent Space

---

Appma
Query
Alteniion Soure
Selected
Fused Recover Kernel
Post-rope Query
Dim.
Recundract & Rope
Pre-mpe Key
Topsk looes [.7]
Offline Calibration
Lak,
Key
Cache
Chitical Past-rope
Peonge
Transpose
Suge
aodon
Sage Tokordaza
Spare viuatoo
Figure 3: Overall architecture of SALS. Three stages are introduced with stage I for multi-head KV Cache compression, stage 2 for token selection in latent space and stage 3 for sparse attention.
Overlap Score Acrosn Layers
To quantify the token representations the query and key the latent space, we first compute the latent-space attention vec- tor QK, where in the pre-RoPE setting, Q QU. Let = {ir, (2..2...... Iy. _ {1..... (1.....s), N < s, denote the in-
Overbo Scor (40
the top-N, entries of p. The overlap score (OS) measures the token representation in the latent space, which is fraction of the full attention mass that these indices cap- ture. OS = Exec P4/ Ei=1 P;, where p; denotes the i-th entry of the full attention distribution p E R*
dex corresponding
LEMMD
Luyer
Figure presents the overlap score for LLaMA-7B-chat, LLaMA- Figure 2: High overalp rate of 8B and Mistral-7B-v0.2. We observe that the average overlap
pre-RoPE across layers mostly remains above 90% for layers 2-29 but drops below 50% in layers 0, 1. A similar observation has been reported for hybrid RoPE attention design [27]]. This finding indicates that,
when RoPE is completely omitted, latent space tokens can still preserve almost all of the full attention
scores across the majority of layers.
4 Sparse Attention in Latent Space Framework
4.1 SALS Framework
In this section, introduce our proposed SALS, which leverages low-rank projection matrix to transform multi-head pre-RoPE keys into compact single-head latent space, thereby markedly shrinking the memory footprint. To avoid the prohibitive cost of reconstructing the entire key cache, SALS estimates attention scores directly in the latent space, selects the top-Ne most critical tokens, and reconstructs only their corresponding keys. RoPE is only applied to the reconstructed keys, which are then used to compute the exact attention weights. Figure L provides high-level summary of the SALS pipeline. Here, we use the top-k index [2, to indicate the selected tokens. The selected tokens are then reconstructed and reshaped to multi-head keys for sparse attention. The SALS framework is pipelined with three stages: KV cache compression latent space, critical token selection in latent space and selective reconstruction from latent space for the later sparse attention.
4.2 Multi-head KV cache Compression to Latent Space
In the offline calibration process, we select a small calibration dataset from the pre-training corpus and collect its Rre-RoPE key tensors. Let K E Rxxmd denote the stacked multihead keys of the s calibration sequences. We begin by computing the empirical covariance of the calibration keys: C K K. Applying an eigenvalue decomposition, eigenvectors U, yields the optimal rank-r projection matrix for the latent space. Note that the key vectors K E R are reshaped by merging the head number with head dimension. We provide the rational using the following lemma.
UTU and selecting the leading
Lemma For any column-orthonon orthonormal matrix E Rhxr satisfying UTU = I,, we define the captured variance by U as EUU) For the multi-head joint projection, all possible projection matrix set is defined as U, == {U 1 Rdd
UTU = I, and B, {diag( U...., U,) U,
5

---

Algorithm Sparse Attention in Latent Space
Input:
Pre-RoPE query q E Rnd key k E Rnd, value Rnd Projection matrix U, E Rudxr
Low-rank key cache K (R-51))r,, quantised value cache V R 1)x=-))xnd Sparsity budget
Approx latent rank r**
Output: Attended output y E IRnd
1: Function SPARSEATENTIONN U, K, V,)
2: q - quqUr: KU,
Project new token to r-dim. latent space
3:
K concat(K, k); V t concat((.v
4:
p' qr.
Cheap similarity on top-r" dims # Selectr top-k critical token index
5:
C + TopK(p',
6:
Kc + K
# Reconstruct critical latent key cache # Standard flash attention (FA) computation
7:
q RoPE(qq: K + RoPE(Ke)
8:
P softmax a K /Va)
9: y y-
return
11: End Function
Rdxr' U,U,= where r' =y=r// r/n, is all possible projection matrix set for per-head projection. For all situations, the optimal U from multi-head joint projection matrix can capture more enery then per-head projection matrix:
E(U) UEu,
max E(U) UEB.
Proof. Pick any U = diag( UI..... U.) E B, with U.U, = I,s. Because the blocks occupy disjoint coordinate ranges, their product is blockdiagonal:
U U = diag U,, U, = diag(I, Dirag[I,...,,II...... I,,
Hence U is column-orthonormal and thus U E Ur: consequently B, C Up. Since every candidate in B, is feasible for the joint search U, maximizing over the larger set cannot yield smaller value. C
Critical Token Selection Latent Space
As discussed in previous works |2S 29). tokenwise attention is inherently sparse. However, identi- fying the relevant (critical) tokens fast enough remains main bottleneck of most sparse-attention schemes. Guided by the observation in Sec [ that Pre-RoPE latent keys already reveal the sparsity pattern of the middle layers, we forgo the expensive full-rank kee multiplication and instead work in the latent space.
More specifically, let U, E Radxr be the joint projector from Equation DI and let r* for scoring. We project the query once, q U.q RT, and keep only its leading coordinates q,- E R Each cached key already stores its full latent vector k, = U. from which we extract the first dimensions, kj.rr= E R²". The approximate attention score of token in position j is a cheap inner product
Sy q.,- ky.r- Thus, critical tokens are identified directly from the existing key cache, without extra storage and at fraction of the original compute.
4.4 Selective Reconstruction from Latent Space for Sparse Attention
Sparse attention is defined by restricting the standard attention computation to selected subset of keys and values. Given a query matrix Q xa and a full set key and value matrices of length identify subset of salient indices
= (i,, ...., iN.) (L....,)), s). No <s.
(4)

---

where No denotes the number of selected positions on the token sequence. The corresponding sub- matrices Ke E xxxx and Ve E xxnx are extracted based on the selected indicesC. Here, the salient indices C are obtained via critical token selection in latent space. The sparse attention output then computed by restricting the standard attention to the selected tokens:
QK?
Pc = softmax
y= Pe
This method works well since the softmax operation normalizes the input distribution so that the output sums to I and only a few positions (or tokens) in dominate with large values. The rest scores after softmax closes to O. The whole SALS algorithm is summarized in Algorithm M.
4.5 Performance Discussion
As the roofline model [8] suggests, the performance of a GPU is bounded by either computation or memory bandwidth. In attention computation, performance commonly bounded by memory bandwidth [Z]], which is primarily consumed by KV cache data movement. For a context of tokens and feature dimension d, full attention needs transfer 2sd elements of keys and values. On the other hand, for the SALS framework, critical token selection and sparse attention both involve data movement. Critical token selection requires r < latent dimensions, so the query-key dot products read sr* elements. After selecting the top-k: tokens, SALS only needs to access low-rank key and value caches, each size kr. The second pass therefore moves 2kr elements.
Combining this two phases, the sparse variant moves ST" + 2kr scalars, so its memory-bound 2nd
speed-up is: 4rr d,-/2 d. Here d,- = -r"/d, d, = r/d, k, = k/s denote the ratio, the low-rank ratio, and the token sparsity, respectively. We develop Triton kernel that fuses token selection, reconstruction and RoPE rotation into a single pass. This fused econstruct-RoPE kernel reduces memory traffic by 7.69x to 14.28x, depending on the chosen sparsity level and low-rank compression ratio, compared the standard FlashAttention implementation.
latent-an seeent-aproximmtee--eee
5 Experiment
5.1 Setup
Models and tasks. We evaluate our SALS framework on two main- stream 7B chat models: LLaMA2-
Table 2: Evaluation on GSM8K and CoQA datasets for LLaMA2-7B-chat
7B-Chat [23], that employs multi- head attention (MHA), and Mistral-7B-
Memory Access.
Comp. ratio].
Method
GSM8K (strict/flexible))
CoQA1
v0.2 [L]]], that uses grouped-quest at- tention (GQA). To assess accuracy, we report results on the reasoning bench- mark GSM8K [6]], the conversational benchmark CoQA [I6]], and the 16 En- glish subsets of LongBench [2] that probe long context understanding. All experiments are conducted on a machine with Xeon(R) Platinum 8336C CPU, one GPU (ampere architecture), and 128G RAM.
baseline
0.2335 0.2335
0.5997
1.00
1.00
0.2297 0.2305
0.5992
KIVI-2
0.2047 0,2047
0.6010
0.19
0.19
(ali-30%%(3bit
0.1713/0.1759
0.5938
0.14
0.14
(alu-50%(3bi)
0.061440.00879
0,5560
0.09
0.09
SALS-255
0.2312 0,2343
0.5975
0.13
0.28
SA1S-12.5%
0.2176 0.2229
0.6070
0.07
0.15
Baselines. We compare SALS with several state the art bascline in two directions. For KV-cache
compression select Palu (low-rank projection) [ and KIVI (quantization) [[_]] For token- sparse decoding we include Double Sparse [29], Hshare [2S], and Loki [20]. For the KV-eache compression baselines, we evaluate Palu at every reported compression ratio, including its variants that apply the optional quantisation step. For KIVI we evaluate both official settings: 4 bit and bit. For the loken-sparse baselines, sparsity is introduced only during the decoding phase, and enforce the same overall sparsity ratio as Hshare to ensure a fair comparison.
Compression setup and calibration. To obtain the latent projection matrix, we randomly sample 512 sequences length 4096 from the C4 corpus [X] and compute offline. We apply multi-head
joint compression to the key cache at two ratios: d, = 25% and d, 12.5%. For latent scoring,

---

Table Evaluation on LongBench datasets
Single- QA
Multi- QA
Summari- zution
Memory
Model
Method
Few
Synthetic
Code
Avg
Access]
buscline
24.50
22.18
23.64
62.80
6.50
56.29
32.65
1.00
KIVI-4bit
24.76
22.22
23.40
62.67
6.75
56.42
32.70
0.31
Palu-30% (3bit)
22.36
22.01
22.66
60.39
7,50
50.53
30.91
0.13
LLaMA2-7B-char
SALS-25%
24.37
22.50
23.40
63.01
6.50
53.80
32.26
0.11
KIVI-2bit
23.11
21.40
22.67
62.81
5.75
55.84
31.93
0.19
Palu-50% (3bit)
20.48
22.18
21.44
54.62
5.75
37.07
26.92
0.09
SALS-12.5%
24.81
22.52
23,08
62.84
5.00
53,58
31.97
0.06
baseline
36.43
29.65
28.06
66.72
44.87
52,96
43.12
1.00
KIVI-4bit
36.34
29.85
28.09
66.81
44.67
52,90
43.11
0.31
Palu-30% (3bit)
29.48
36.40
27.20
65.73
53.19
40.77
34,74
0.13
Mistral-7B-v0.2
SALS-255 25%
36.61
29.92
27.57
66.63
43.54
52,49
42.79
0.11
KIVI-2bit
35.34
28 28.63
27.72
66.77
39.68
52.63
41.80
0.19
Palu-50% (3bit)
26.73
32.72
25.73
63.25
18.57
44,43
35.71
0.09
12.5%
35.18
29.88
27.18
66.98
35,32
51.39
40.99
0.06
we simply set r* 0.5r for all models. Since value tensors are almost full rank and play a pivotal role in attention, forgo low-rank projection for them and instead perform channel-wise group quantisation that mirrors the key-cache setting (4 bit at 25% and 2 bit at 12.5%).
Following KIVI, we adopt mixed high- low- low-precision scheme: tokens in the most recent window are compressed by only 50%, whereas all preceding tokens are compressed according to the target ratio of the experiment. The high-precision window is aligned with the sparsity window: when the sparse mechanism always selects the most recent w tokens, the compression stage likewise keeps those same tokens in high precision. Based on the observations in Figure D, we skip the sparsification for layers 0, 1, and 31 across all models to ensure more accurate compression and sparsity results.
5.2 Vcache Compression Comparison
Implementation details. For the dataset GSM8K and CoQA, we always keep the most recent 128 tokens and decode the remaining context fixed sparsity of 1/4. For the dataset LongBench, LLaMA2-7B-Cha supports 4 k context window, whereas Mistral-7B-v0.2 supports 32 k. To equalise the average sparsity 1/8 on this benchmark, we retain N₂ 512 tokens for LLaMA2 1AMA--7B--- - and N: = 1024 tokens for Mistral-7B-vO.. For LLaMA2-77 1122-7B-hhar we follow the Hshare [25]] configuration (r 16 sink tokens, y = 432 critical tokens, 2 64 recent tokens) and simply double each count for Mistral-7B-v0.2
Results. Tables A shows the evaluation the benchmark (GSM8K, CoQA) ILLMM2-7BB-dR model. We can see that although Palu achieves the highest compression rate, the acurracy drop is non-negligible, especillay for the GSM8K dataset. SALS provides two setting results with 25% KV cache compression rate (SALS-25%) and 12.5% compression rate (SALS-12.5%). SALS- 25% achieves the best accuracy compared KIVI and Palu, with neligible loss compared the baseline. Tables B compares task-level performance across six LongBench categories, alongside normalized memory access costs. The first three columns reflect core reasoning tasks, where SALS achieves consistently strong performance. Notably, SALS-12.55 retains competitive accuracy while reducing memory access to just 6% of the baseline. This indicates that SALS preserves informative tokens more effectively. The memory-acces, savings translate into practical latency improvements, as fewer KV memory lookups reduce attention computation overhead.
5.3 Token-Sparse Comparison
Since SALS incorporates sparsification stage, also compare it with state of the art sparse de- coders, such as Double Sparse, HShare and Loki in LongBench datasets.
Implementation details. We use the same sparse setting as Sec 5.2 for all methods, which have I = 16 sink tokens, = 432 critical tokens and z 64 recent tokens, achieving sparsity on LLaMA2-7b-chan at 1/8.
8

---

Table Comparison of Token Sparse Methods on LLaMA2-7B-chat Using LongBench Tasks
Single- QA
Multi- QA
Summari- zation
Memory Access,
Method
Few-shot
Synthetic
Code
Avg
baseline
24.50
22.18
23.64
62.80
6.50
56.29
32.65
1.00
Double Sparse
24.78
22.72
24.70
61.84
4.17
51.66
31.64
0.16
HShare
24.59
22.18
24.54
61.69
4.74
53.26
31.83
0.14
Loki
24.57
18.09
24.37
63.43
4.75
56.49
31.95
0.19
SALS-25%
24.37
22.50
63.01
6.50
53.80
32.26
0.11
SAL5-12.5%
24.81
22.52
23.08
62.84
5.00
53.58
31.97
0.06
Table 5: Performance comparison of baseline and SALS methods on the RULER dataset with -8B-tttutttt
Llama3.1-88 Instruct (4k sequence length).
Method avg
S2
MK1
MK2
MQ
FEW
QAI
QA2
Baseline
81.60
99.6
96.0
94.6
73.6
93.85
96.9
68.47
69.6
4L.
SALS-25%
80.81
99.6
95.2
94.2
65.8
96.4
71.53
70.2
41.2
SA.5-12.5%
75.86
97.4
93.8
92.8
42.2
84.05
93.05
72.53
67.8
39.14
Result, Table @ shows that, on LongBench, SALS with 25% key compression matches the average accuracy of the baseline while issuing only about 11% of the baselines memory traffic. When the compression is tightened to 12.5%, the Memory- Accers Ratio drops to just 5.8%. yet SALS still outperforms all competing sparse methods in accuracy. These results suggest that the latent-space scoring mechanism selects critical tokens more accurately than existing heuristics, enabling KV- cache compression and token sparsity to coexist without loss of accuracy.
5.4 RULER Benchmark
Dataset. RULER [100] is a recently proposed long-context benchmark designed to evaluate the rea- soning, retrieval, and compositional understanding capabilities of large language models across wide range of imput lengths and retrieval patterns. It decomposes the evaluation into fine-graines re- trieval types, including single-key, multi-key, multi-value, and quely tasks, along with few-shot and
(QA) subtasks. Compared with LongBench, RULER provides more de- tailed breakdown of retrieval behaviors, making well- for assessing token selection accuracy and KV-cache compression performance.
Implementation details. The experimental setup follows the same configuration in Sec. 5.2 We evaluate all methods on 128k-token context, applying an 8x sparsity ratio, i.e., selecting 16k active tokens during inference. The retained-to-prumee token ratio is identical that used in the LongBench experiments, ensuring consistent compression levels and comparable evaluation metrics. All experiments are conducted on the LLaMA InstSu-Instruct model with 4k sequence length.
Result. Table E summarizes the performance SALS on the RULER dataset. At a 25% com- pression ratio, SALSS achieves nearly identical average accuracy to the baseline (80.81 81.60), demonstrating its high fidelity in token selection and K V--cache reconstruction. Performance remains consistent across most retrieval subtasks, including NIAH-Single- 1/2, Multi-Value, and Multi futti--uery, indicating that latent-space sparsification preserves key contextual semantics even under substantial cache reduction.
When the compression is further tightened 12.5%, accuracy degradation mainly occurs in NIAH- Single-1 and Ke where the retrieval dependency is strongest. Nevertheless, the model still achieves competitive overall performance (75.86 average) and stable scores on Few-shot and QA tasks. These results suggest that the proposed latent-space attention mechanism can maintain accurate token importance estimation even under extreme sparsity, enabling effective KV-cache com- pression with minimal accuracy loss across retrieval-oriented benchmarks.
5.5 Efficiency Evaluation
Inference details. Following Hshare [25], all speed tests are performed on the PyTorch back- end, with the Triton-base fused kernel discussed in Sec I. We evaluate two aspects: (i)

---

Table Attention Operator Latency (ms) across Methods and Input Configurations
Config (ms)
Flash-attn [+]
Loki [20]]
Double-sparse [20]]
Hshare [25]]
SAL.5-25%
SA1.S-12.5%
bs=8, 1k
0.230
0.248
0.138
0.134
0.409 + 0.093
0.357
0.051
bs=8,
0.830
0.464
0.237
0.430
0.430 0.018
0.359
0.012
hs=8,
1.630
1.102
0.724
0.576
0.530 0.008
0.439
0.016
bs=16, 1k
0.440
0.452
0.223
0.134
0.415 0.008
0.360 0.011
bs=16. 2k
1.630
0.434
0.246
0.552 0.040
0.444 0.019
bs=16, 3.230
2.101
1.319
1.067
0.737 0.026
0.565 0.013
self-attention latency and (ii) end-to-end generation speed-up. Self-attention latency is com- pared against FlashAttention-v2 [[]], whereas end-to-end throughput benchmarked against GPT- Fast [[_)]. Experiments use batch sizes of and 16 and sequence lengths of k, 2 k, and k tokens; the sparsity ratio is fixed at 1/8 for all methods.
Result. Tables R reports our attention latency comparisons. Mean and varaince are reported using 1000 repetions. Thanks to the added token sparsity, SALS accelerates the standalone attention opera- tor by 7, 464 speedup for batch size (bs) =8 and 4k sequence length (4k). Similar performance speed also observed when batch size is 16. SALS will introduce some overhead for short sequences (e.g. 1k) but for longer sequences the speed-up is significantly better than the state-of-the-an sparse algo- rithm. As discussed in Section A reducing memory access will improve the attention performance.
Table shows the end-to-end through- put speed-up of SALS comparing to the GPT-Fast results. Note that batch size is chosen 4 for sequence 64k to avoid GPU out of memory. Mean and varaince are reported using 50 repetions. We observe that when sequence is 8k, the speed-up of SALS-12.5 5% is 2. 23x although the sparsity ratio is 1/8. This due to the overhead of reconstructing the selected token. When the sequence is set 32k, the speed-up of SALS- 12.5%% increasing and reaches 4.57x. Coupled with the strong accuracy of SALS, these results confirm that combin- ing latentspace KVcache compression with token sparsification offers effective tradeoff between speed and accuracy.
Table 7: Throughput Comparison (token/second)
Long Context End-to-End Performance
Bsz
Seq (k)
GPT-Fast
SALS-25%
SALS-
8
4
118
154.1 = 7.1
163.5 5.1
8
8
70.6
128.6 = 5.1
150.1 3.3
8
16
38.3
102.5 0.3
122.7 7.7
32
19.8
67.97 =
89.47 1.4
4
64
9.2
32.92
0.3
42.96 0.9
6 Conclusion
This paper investigates the KV cache compression based the low-rank characteristics in the hid- den dimension. We further discuss the increasing variance of keys due to applying rotary position embedding and propose use the Pre-RoPE keys select the critical tokens. We observe that after the key vectors are transformed into the latent space, they largely maintain their representa- tion across most layers and thereby the salient tokens are selected with high accuracy. Based on these insights, we propose the Sparse Attention in Latent Space (SALS) framework. The SALS framwork achieves compressed KV cache to the latent space, selects the critical token in the latent space with significantly less computation load and performs sparse attention on the selected tokens. By reconstructing only a small subset of important tokens, it avoids the overhead of full KV cache reconstruction. Experimental results demonstrate that SALS achieves SOTA performance by main- taining competitive accuracy under different settings and achieving 6.4-fold KV cache compression and 5.7-fold speed-up in attention compared to FlashAttention2 4K sequences. For the end-to- end throughput performance, we achieve 1.4-fold and 4.5-fold improvement compared to GPT-fast on 4k and 32K sequences respectively.
Acknowledgment
This work is supported by the National Nature Science Foundation China (No. 62376020) and the Fundamental Research Funds for the Central Universities (No. 2025JBZY01I).
10

---

References
[1] Anthropic. Claude. Large language model, May 2023. URL https://ww..attrror
Clande Accessed May 5, 2025.
[2] Y. Bai, X. Lv, J. Zhang, H. Lyu, J. Tang, Z. Huang, Z. Du, X. Liu, A. Zeng, L. Hou, al. Longbench: A bilingual, multitask benchmark for long context understanding. arXiv preprint arXiv:2308. 14508, 2023.
[3] I. Beltagy, M. E. Peters, and A. Cohan. Longformer: The long-document transformer. arXiv preprint arXiv:2004.05// 5000 2020.
[4] C.-C. Chang, W.-C. Lin, C.-Y. Lin, C.-Y. Chen, Y.-. Hu, P.-S. Wang, N.-C, Huang, L. Ceze, M. Abdelfattah, and K.-C Wu. Palu: Compressing kv-cache with low-rank projection. arXiv preprint arXiv:2407.21118, 2024.
[5] R. Child, S. Gray, A. Radford, and I. Sutskever. Generating long sequences with sparse trans- formers. arXiv preprint arXiv: 1904. 10509, 2019.
[6] K. Cobbe, Kosaraju, M. Bavarian, M. Chen, Jun, Kaiser, M. Plappert, J. Tworek,
J. Hilton, R. Nakano, et al. Training verifiers to solve math word problems. arXiv preprint arXiv:2110.14166 14168, 2021.
[7] T. Dao. Flashattention-2: Faster attention with better parallelism and work partitioning, arXiv preprint arXivv2307.08661 2023.
[8] N. Ding and S. Williams. An instruction rooftine model for gpus. IEEE, 2019.
[9] I. Habernal, O. Zayed, and I. Gurevych. C4corpus: Multilingual web-size corpus with free license. In Proceedings of the Tenth International Conference on Language Resources and Evaluation (LREC' 16), pages 314-922,11 2016.
[10] C.-P. Hsieh, S. Sun, S. Kriman, S. Acharya, D. Rekesh, F. Jia, Y. Zhang, and B. Ginsburg. Ruler: What's 's the real context size of your long-context language models?, 2024. URL https:
7/arxiv. owgxivvoogg/baa www.////g.///.....//////// 06654.
[11] A. Q. Jiang, A. Sablayrolles, A. Roux, A. Mensch, B. Savary, C. Bamford, D.S. Chaplot, D. d. Casas, E. B. Hanna, F. Bressand, et al. Mixtral of experts. arXiv preprint arXiv:240..04888, 2024.
[12] N. Kitaev, E. Kaiser, and A. Levskaya. Reformer: The efficient transformer. arXiv preprint arXiv:2001.04451, 2020.
[13] P. Labs. gpt-fast: Simple and efficient pytorch-native transformer text generation in <1000 loc of python, 2023. URL 2025-05-14.
fast. Accessed:
[14] Z. Liu, J. Yuan, H. Jin, S. Zhong, Z. Xu, V. Braverman, B. Chen, and X. Hu. Kivi: A tuning- free asymmetric 2bit quantization for kv cache. arXiv preprint arXiv:2402.002550 2024.
[15] OpenAl. Chatgpt. Large language model, May 2023. URL. Accessed May 5, 2025.
com/
[16] S. Reddy, D. Chen, and C. D. Manning. Coqa: A conversational question answering challenge. Transactions of the Association for Computational Linguistics, 7:249-266, 2019.
[17] L. Ribar, I. Chelombiey, L. Hudlass-Galley, C. Blake, C. Luschi, and D. Orr. Sparq attention: Bandwidth-effiche Ilm inference. arXiv preprint arXiv:23/2.04988, 2023.
[18] R. Saha, N. Sagan, V. Srivastava, Goldsmith, and M. Pilanci. Compressing large language models using low rank and low precision decomposition. Advances in Neural Information Processing Systems, 37:88981-899 890188 2024.
[19] U. Saxena, G. Saha, Choudhary, and K. Roy. Figen attention: Attention low-rank space for ky cache compression. arXiv preprint arXiv:2408.05666, 2024. 2024.

---

[20] P. Singhania, S. Singh, S. He, S. Feizi, and A. Bhatele. Loki: Low-rank keys for efficient sparse attention. arXiv preprint 222.22406.0254. 2024.
[21] Su, M. Ahmed, Y. Lu, S. Pan, W. Bo, and Y. Liu. Roformer: Enhanced transformer with rotary position embedding. Neurocomputing. 568:127063, 2024.
[22] J. Tang, Y. Zhao, K. Zhu, G. Xiao, B. Kasikei, and S. Han. Quest: Query-aware sparsity for efficient long-context IIm inference, 2024.
[23] H. Touvron, T. Lavril, G. Izacard, X. Martinet, M.-A. Lachaux, T. Lacroix, B. Rozière, N. Goyal, E. Hambro. F. Azhar, al. Llama: Open and efficient foundation language models. arXiv preprint arXiv:2302. 13971, 2023.
[24] X. Wang. Y. Zheng, Z. Wan, and M. Zhang. Svd-llm: Truncation-wware singular value decom- position for large language model compression. arXiv preprint arXiv:2403.0737% 2024.
[25] H. Wu, L. Li, H. Huang, Yi, J. Zhang, M. Yu, and J. Yan. Hshare: Fast Ilm decoding by hierarchical key-value sharing. In The Thirteenth International Conference on Learning Representations, 2025.
[26] G. Xiao, Y. Tian, B. Chen, S. Han, and M. Lewis. Efficient streaming language models with attention sinks. arXiv preprint arXiv:2309. 17453, 2023.
[27] B. Yang, B. Venkitesh, D. Talupuru, H. Lin, D. Cairuz, P. Blunsom, and A. Locatelli. Rope to nope and back again: A new bybrid attention strategy, arXiv preprint arXiv:2501. 18795, 2025.
(28) D. Yang, X. Han, Y. Gao, Y. Hu, S. Zhang, and H. Zhao. Pyramidinfer: Pyramid kv cache compression for high-throughput IIm inference. arXiv preprint arXiv:2405. 12532, 2024.
[29] Yang, Sheng, J. Gonzalez, Stoica, and L. Zheng. Post-training sparse attention with double sparsity. arXiv preprint ar Www::2008.0009000000 2024.
[30] J. Yuan, H. Gao, D. Dai, J. Luo, L. Zhao, Z. Zhang, Z. Xie, Y. X. Wei, L. Wang, Z. Xiao, Y. Wang, C. Ruan, M. Zhang, W. Liang, and W. Zeng. Native sparse attention: Hardware- aligned and natively trainable sparse attention, 2025. URL htt.t:t//aaaivv.oottt harrg//bb/5/2. 2502. 11003
[31] Z. Yuan, Y. Shang, Y. Song, Q. Wu, Y. Yan, and G. Sun. Asvd: Activation-aware singular value decomposition for compressing large language models. arXiv preprint arXiv:2312.05821, 2023.
[32] T. Zhang, J. Yi, Z. Xu, and A. Shrivastava. Kv cache is bit per channel: Efficient large language model inference with coupled quantization. arXiv preprint trXiv:2405.03917 2024.
[33] Z. Zhang, Y. Sheng, T. Zhou, T. Chen, L. Zheng, R. Cai, Z. Song, Y. Tian, C. Ré, C. Barrett, al. H2o: Heavy-hitter oracle for efficient generative inference of large language models. Advances in Neural Information Processing Systems, 36, 2024.
12

---

NeurIPS Paper Checklist
1. Claims
Question: Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?
Answer: [Yes]
Justification: The abstract and introduction in this paper accurately reflect the papers con- tributions and scope.
Guidelines:
The answer NA means that the abstract and introduction do not include the claims made in the paper.
The abstract and/or introduction should clearly state the claims made, including the contributions made the paper and important assumptions and limitations. A No or NA answer to this question will not be perceived well by the reviewers.
The claims made should match theoretical and experimental results, and reflect how much the results can be expected to generalize to other settings.
It is fine to include aspirational goals motivation as long as it is clear that these goals are not attained by the paper.
2. Limitations
Question: Does the paper discuss the limitations of the work performed by the authors?
Answer: [Yes]
Justification: We discuss the limitation our method in the experiment section. Result is shown in Table and discuss Line 303.
Guidelines:
The answer NA means that the paper has limitation while the answer No means that the paper has limitations, but those are not discussed in the paper.
The authors are encouraged to create separate "Limitations" section in their paper.
The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (c.g., independence assumptions, noiseless settings, model well-specification, asymptotic approximations only holding locally). The au- thors should reflect on how these assumptions might be violated practice and what the implications would be.
The authors should reflect the scope of the claims made, e.g., if the approach was only tested on few datasets or with few runs. In general, empirical results often depend on implicit assumptions, which should be articulated.
The authors should reflect on the factors that influence the performance of the ap- proach. For example, facial recognition algorithm may perform poorly when image resolution is low or images are taken in low lighting. Or a speech-to-tex system might not be used reliably provide closed captions for online lectures because fails to handle technical jargon.
The authors should discuss the computational efficiency of the proposed algorithms and how they scale with dataset size.
If applicable, the authors should discuss possible limitations of their approach ad- dress problems of privacy and faimesss.
While the authors might fear that complete honesty about limitations might be used by reviewers as grounds for rejection, a worse outcome might be that reviewers discover limitations that aren't acknowledged in the paper. The authors should use their best judgment and recognize that individual actions in favor of transparency play an impor- tant role in developing norms that preserve the integrity of the community, Reviewers will be specifically instructed to not penalize honesty concerning limitations.
3. Theory assumptions and proofs
Question: For each theoretical result, does the paper provide the full set of assumptions and complete (and correct) proof?
13

---

Answer: [Yes],
Justification: In this paper, all proofs of theorems are provided and all assumptions are clearly stated referenced in the statement of any theorems
Guidelines:
The answer NA means that the paper does not include theoretical results.
All the theorems, formulas, and proofs in the paper should be numbered and cross- referenced.
All assumptions should be clearly stated or referenced in the statement of any theo- rems.
The proof' can either appear in the main paper or the supplemental material, but they appear in the supplemental material, the authors are encouraged to provide short proof sketch to provide intuition.
Inversely, any informal proof provided in the core of the paper should be comple- mented by formal proofs provided in appendix supplemental material.
Theorems and Lemmas that the proof relies upon should be properly referenced.
4. Experimental result reproducibility
Question: Does the paper fully disclose all the information needed to reproduce the main experimental results of the paper to the extent that it affects the main claims and/or conclu- sions of the paper (regardless of whether the code and data provided not)?
Answer: [Yes]
Justification: All the settings are clearly discussed and cited. The results in this paper can be reproduced.
Guidelines:
The answer NA means that the paper does not include experiments.
If the paper includes experiments, a No answer this question will not be perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data are provided or not.
If the contribution is a dataset and/or model, the authors should describe the steps taken make their results reproducible or verifiable.
Depending on the contribution, reproducibility can be accomplished in various ways. For example, if the contribution is a novel architecture, describing the architecture fully might suffice, if the contribution is specific model and empirical evaluation, may be necessary to either make it possible for others to replicate the model with the same dataset, or provide access to the model. In general. releasing code and data is often one good way to accomplish this, but reproducibility can also be provided via detailed instructions for how replicate the results, access a hosted model (e.g., in the case of a large language model), releasing of a model checkpoint, other means that are appropriate the research performed.
While NeurIPS does not require releasing code, the conference does require all sub- missions to provide some reasonable avenue for reproducibility, which may depend on the nature the contribution. For example
(a) the contribution primarily new algorithm, the paper should make it clear how to reproduce that algorithm.
(b) If the contribution is primarily new model architecture, the paper should describe the architecture clearly and fully.
(c) the contribution is a new model (e.g., a large language model), then there should either be way to access this model for reproducing the results way to re- produce the model (e.g., with an open-source dataset or instructions for how to construct the dataset).
(d) We recognize that reproducibility may be tricky in some cases, in which case au- thors are welcome to describe the particular way they provide for reproducibility. In the case of closed-source models, it may be that access to the model is limited in some way (e.g., to registered users), but it should possible for other researchers to have some path to reproducing or verifying the results.
14

---

5. Open access to data and code
Question: Does the paper provide open access to the data and code, with sufficient instruc- tions to faithfully reproduce the main experimental results, described in supplemental material?
Answer: [Yes]
Justification: We provide open access the data and code, with detailed instructions to reproduce all the experimental results.
Guidelines:
The answer NA means that paper does not include experiments requiring code.
Please see the NeurlPS code and data submission guidelines (https://nips.cc.
for more details.
While we encourage the release of code and data, we understand that this might not be possible, so No an acceptable answer. Papers cannot be rejected simply for not including code, unless this is central to the contribution (e.g., for new open-source benchmark).
The instructions should contain the exact command and environment needed to run to reproduce the results. See the NeurIPS code and data submission guidelines (https:)
7 //ipp/.ccppblic/guuiiee Coteee//doeSSSmmmieeee otg.uaeeeeeeeeeetbaaaaaaaooooooo ssionoolicy for more details.
The authors should provide instructions on data access and preparation, including how access the raw data, preprocessed data, intermediate data, and generated data, etc.
The authors should provide scripts to reproduce all experimental results for the new proposed method and baselines. If only subset of experiments are reproducible, they should state which ones are omitted from the script and why.
At submission time, to preserve anonymity, the authors should release anonymized versions (if applicable).
Providing as much information possible in supplemental material (appended to the paper) recommended, but including URLs to data and code permitted.
6. Experimental setting/details
Question: Does the paper specifyy all the training and test details (e.g., data splits, hyper- parameters, how they were chosen, type of optimizer, etc.) necessary to understand the results?
Answer: [Yes]
Justification: The paper details all the calibration and evaluation details necessary to under- stand the results.
Guidelines:
The answer NA means that the paper does not include experiments.
The experimental setting should be presented in the core of the paper to level of detail that is necessary appreciate the results and make sense of them.
The full details can be provided either with the code, in appendix, as supplemental material.
7. Experiment statistical significance
Question: Does the paper report error bars suitably and correctly defined or other appropri- ate information about the statistical significance of the experiments?
Answer: [Yes]
Justification: For the speed-up experiment in our paper, we repeat the same setting for 10 times and report the average and varaince, For our benchmark accuracy, we do not repeat run the same benchmark multiple times since we notice very little variance in the accuracy.
Guidelines:
The answer NA means that the paper does not include experiments.
The authors should answer "Yes" if the results are accompanied by emor bars, confi- dence intervals, or statistical significance tests, least for the experiments that support the main claims of the paper.
15

---

The factors of variability that the error bars are capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, or overall run with given experimental conditions).
The method for calculating the error bars should be explained (closed form formula, call to a library function, bootstrap, etc.)
The assumptions made should be given (e.g.. Normally distributed errors).
It should be clear whether the error bar is the standard deviation the standard error of the mean.
It is OK to report I-sigma error bars, but one should state it. The authors should prefer- ably report a 2-sigma error bar than state that they have 96% CI, if the hypothesis of Normality of errors is not verified.
For asymmetric distributions, the authors should be careful not show in tables figures symmetric error bars that would yield results that are out of range (e.g. negative error rates).
If error bars are reported in tables or plots. The authors should explain in the text how they were calculated and reference the corresponding figures or tables in the text.
8. Experiments compute resources
For each experiment, does the paper provide sufficient information on the com- puter resources (type of compute workers, memory, time of execution) needed reproduce the experiments?
Question:
Answer: [Yes]
Justification: For each experiment, the paper provide sufficient information on the computer resources needed reproduce the experiments
Guidelines:
The answer NA means that the paper does not include experiments.
The paper should indicate the type of compute workers CPU GPU, internal cluster, or cloud provider, including relevant memory and storage.
The paper should provide the amount of compute required for each of the individual experimental runs as well estimate the total compute.
The paper should disclose whether the full research project required more compute than the experiments reported in the paper (e.g.. preliminary failed experiments that didn't 't make it into the paper).
9. Code of ethics
Question: Does the research conducted in the paper conform, in every respect, with the NeurlPS Code of Ethics http:///wwaaaoccccccaa/ac/ccccaa
Answer: [Yes]
Justification: The research conducted in the paper conform, in every respect, with the NeurIPS Code of Ethics.
Guidelines:
The answer NA means that the authors have reviewed the NeurlPS Code of Ethics.
If the authors answer No, they should explain the special circumstances that require deviation from the Code of Ethics.
The authors should make to preserve anonymity (e.g.. if there is special consid- eration due to laws or regulations in their jurisdiction).
10. Broader impacts
Question: Does the paper discuss both potential positive societal impacts and negative socictal impacts of the work performed?
Answer: [Yes]
Justification: The paper discusses the challenging problem of LLM long-context inference in the Introduction and the benefits to adopt our method. There is no negative societal impacts for the work.
16

---

Guidelines:
The answer NA means that there no societal impact of the work performed.
If the authors answer NA or No, they should explain why their work has no societal impact why the paper does not address societal impact.
Examples of negative societal impacts include potential malicious or unintended uses (e.g., disinformation, generating fake profiles, surveillance), fairness considerations (e.g., deployment of technologies that could make decisions that unfairly impact spe- cific groups), privacy considerations, and security considerations.
The conference expects that many papers will be foundational research and tied to particular applications, let alone deployments. However, if there is a direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that an improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, is not needed to point out that a generic algorithm for optimizing neural networks could enable people to train models that generate Deepfakes faster.
The authors should consider possible harms that could arise when the technology is being used intended and functioning correctly, harms that could arise when the technology is being used as intended but gives incorrect results, and harms following from (intentional unintentional) misuse of the technology.
If there are negative societal impacts, the authors could also discuss possible mitiga- tion strategies (e.g., gated release of models, providing defenses in addition to attacks, mechanisms for monitoring misuse, mechanisms to monitor how a system learns from feedback over time, improving the efficiency and accessibility of ML).
11. Safeguards
Does the paper deseribe safeguards that have been put in place for responsible release of data models that have high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)?
Question:
Answer: [NA]
Justification:
The paper has such risks.
Guidelines:
The answer NA means that the paper poses no such risks.
Released models that have high risk for misuse or dual-use should be released with necessary safeguards allow for controlled use of the model, for example by re- quiring that users adhere to usage guidelines or restrictions to access the model or implementing safety filters.
Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images.
We recognize that providing effective safeguards is challenging, and many papers do not require this, but we encourage authors to take this into account and make best faith effort.
12. Licenses for existing assets
Are the creators or original owners of assets (e.g., code, data, models), used in the paper, properly credited and are the license and terms of use explicitly mentioned and properly respected?
Question:
Answer: [Yes]
The creators or original owners of assets used in the paper are properly credited and cited. The licenses and terms of use have been clearly mentioned and properly respected.
Justification:
Guidelines:
The answer NA means that the paper does existing assets.
The authors should cite the original paper that produced the code package dataset.
The authors should state which version of the asset used and, possible, include URL.
17

---

The name of the license (e.g., CC-BY 4.0) should be included for each asset. For scraped data from particular source (e.g., website), the copyright and terms of service of that source should be provided.
If assets are released, the license. copyright information, and terms of use in the pack- age should be provided. For popular datasets, paperswi thcode cow /ataattsstts has curated licenses for some datascts. Their licensing guide can help determine the li- cense of dataset.
For existing datasets that are re-packaged, both the original license and the license of the derived asset (if it has changed) should be provided.
If this information is available online, the authors are encouraged reach out to the asset's s creators.
13. New assets
Question: Are new assets introduced in the paper well documented and is the documenta- tion provided alongside the assets?
Answer: [Yes]
Justification:
The paper does not release new assets.
Guidelines:
The answer NA means that the paper does not release new assets.
Researchers should communicate the details the dataset/code/moddl as part of their submissions via structured templates. This includes details about training, license, limitations, etc.
The paper should discuss whether and how consent was obtained from people whose asset used.
At submission time, remember to anonymize your assets (if applicable). You can either create anonymized URL or include an anonymized zip file.
14. Crowdsourcing and research with human subjects
Question: For crowdsourcing experiments and research with human subjects, does the pa- per include the full text of instructions given to participants and screenshots, if applicable, as well as details about compensation (if any)?
Answer: [NA].
Justification:
The paper does not involve crowdsourcing nor research with human subjects.
Guidelines:
The answer NA means that the paper does involve crowdsoureing nor research with human subjects.
Including this information in the supplemental material is fine, but if the main contri- bution of the paper involves human subjects, then as much detail as possible should be included in the main paper.
According the NeurIPS Code of Ethics, workers involved in data collection, tion, or other labor should be paid at least the minimum wage the country of the data collector.
15. Institutional review board (IRB) approvals equivalent for research with human subjects
Question: Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or an equivalent approval/review based on the requirements of your country or institution) were obtained?
Answer: [NA]
Justification: The paper does not involve crowdsourcing nor research with human subjects.
Guidelines:
The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.
18

---

Depending on the country in which research is conducted, IRB approval (or equiva- lent) may be required for any human subjects research. If you obtained IRB approval, you should clearly state this in the paper.
We recognize that the procedures for this may vary significantly between institutions and locations, and we expect authors to adhere to the NeurIPS Code of Ethics and the guidelines for their institution.
For initial submissions, do include any information that would break anonymity (if applicable), such as the institution conducting the review.
16. Declaration of LLM usage
Does the paper describe the usage of LLMss if it is an important, original, or non-standard component of the core methods in this research? Note that if the LLM is used only for writing, editing, or formatting purposes and does not impact the core methodology, scientific rigorousness, originality of the research, declaration is not required.
Question:
Answer: [No]
Justification: This paper is written by the authors and LLM is only used for grammar checking.
Guidelines:
The answer NA means that the core method development in this research does not involve LLMs as any important, original, non-standard components.
Please refer to our LLM for what should or should not be described.
19

---

A Key Dimensianlity Analysis with RoPE
Ejewakees Dietrbuution
Egenialues Distibutionn ofleess
Dered Utea Demenare Tipenoal Songe 21.2]
Detal
Rommivet Eiger Egerasiie Range (1-2)
39
12
15"
3.0
0.
10
14
147
Prgmeddy
(a)
(b)
Shat
Mistia-FB ncbruct-6.2
1000
laper
(c)
(d)
Figure 4: (b)- Eigenvalue distributions of key covariance matrices in LLaMA-2-7B-Chaa and Mistral-7B--nnsruuee-ve before and after applying Rotary Position Embedding (RoPE). (c)- Number of principal components required retain 90% of the total variance across transformer layers, indicating changes in effective rank after RoPE.
In this section, we conduct numerical analysis to quantify how rotary position embedding (RoPE) alters the principal-component structure of the key states. We perform principal component analysis (PCA) on the key tensors before and after applying RoPE, denoted as Pre-RoPE and post-RoPE, respectively. Concretely, we first compute the covariance matrix of the key states and then carry out an eigenvalue decomposition. The magnitude of each eigenvalue reflects the contribution of its associated eigenvector: the presence of many small eigenvalues indicates that the corresponding representation is effectively low rank. To relate rank to the spectrum more systematically, we adopt the metrix introduced in Loki[20]]
(a) min { d E 2
100
where 2,(0)
denotes the i-th largest eigenvalue of the covariance matrix for the keys in layer This definition specifies the smallest number of principal components needed to explain at least v% of the total variance, enabling a direct comparison of the intrinsic dimensionality before and after RoPE.
Figure A ab) illustrate the cigen-value spectra of the first layer (1 = 0) for -lama-2-7B-Chi and -n-sruttttt under the Pre-RoPE and RoPE conditions. The pre-RoPE spectra exhibit a markedly larger number of small eigenvalues, whereas the post- post-RoPE spectra are shifted upward, corroborating our earlier finding that RoPE increases the overall variance. Figure (d)))(d) present the layer-wise (ank;(90) values, i.e. the minimal dimensionality needed to retain 90 % of the variance. For both models, the RoPt condition consistently requires a higher rank, reflect- ing the broader spectra observed in panels (a)-(b). In addition, the required rank varies substantially across layers, indicating that ldyer-ad rank selection scheme could further enhance compres- sion efficiency.
20


Reviews:
[{"review_id": "cG49kO9M90", "reviewer": "Reviewer_YyV8", "summary": "SALS is a framework designed to improve inference efficiency for Large Language Models (LLMs) with long contexts by compressing the large Key-Value (KV) cache. It leverages two insights: RoPE increases key vector variance (raising rank), and key representations remain stable across layers. SALS projects the KV cache into a compact latent space using low-rank projection and performs sparse token selection via RoPE-free interactions, reconstructing only key tokens. This avoids full KV reconstruction overhead. Evaluated on LLaMA2-7b-chat and Mistral-7b, SALS achieves up to 6.4× KV compression, 5.7× faster attention, and up to 4.5× end-to-end speedup, while maintaining competitive accuracy.", "questions": "1. Can you provide numbers to justify Table 1?\n2. Additional experiments as stated in the weaknesses.\n3. InfiniGen, ShadowKV also employs low-rank compression for KV cache. Could you provide any insights about the core differences?.", "limitations": "There are many limitations generally for sparse attention. For example, the speed up is limited by how much memory taken by KV cache. I think this paper does not discuss tlimitations in the work.", "rating": 4, "confidence": 5, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 2, "clarity": 3, "significance": 2, "originality": 3, "strengths_and_weaknesses": "Strengths:\n1. The paper is easy to follow.\n2. The baseline implementations (GPT-Fast) are valid.\n3. This paper discusses an important problem of low rank KV compression suffering severe accuracy degradation. \n\nWeaknesses:\n\n1. The evaluation models (Llama2-7b and mistral) are even not long-context models (Llama3.1 is more prefered).\n2. The evaluation dataset is not standard. RULER, InfiniBench, and GSM-Infinite are more convincing.", "was_revised": true, "final_justification": "This paper explores an interesting method for KV sparsity+low rank, which is promising. However, I feel it is similar to other KV low rank algorithms, although this work shows some advantages. In addition, this paper does not thoroughly compare with competitive baselines, such as Quest and ShadowKV, in benchmarks like RULER in a longer context (128K).", "rebuttal": "> Q1 Can you provide numbers to justify Table 1?\n> \n\n**A:** Thank you for you suggestions. We have provide the KV data movment, memory size and computation complexity comparision equation. These equations are used to support the claim low/median/high marks.\n\n\n**Table R1: KV data movement, memory cost and complexity comparison of quantization, low rank and token sparse methods with fixed, dynamic and local token selection strategies. We use $b$, $n$, $n_q$, $s$, $d$ to represent the batch size, KV head number, Q head number, sequence length and head dimension, respectively. We further use $s'$ and $s_{fixed}$ for top-k selected token length and fixed pattern length, respectively, for sparse attention. $r$ represents the compressed rank of KV cache.**\n\n| **Name** | **Methods** | **KV data movement** | **Memory size** | **Computation Complexity** | **Accuracy (gap)** |\n| --- | --- | --- | --- | --- | --- |\n| Palu | Low Rank | Median-$O(bsnr)$ | Low-$O(bsnr)$ | High-$O(bsnrd)+O(bns^2d)$ | Low--(1%-5%) |\n| Loki | Dynamic + Low Rank | Low-$O(bs'nd)$ | Median-$O(bsnd)$ | Median-$O(bns'^2d)$ | High--(0%-1%) |\n| StreamingLLM | Fixed pattern | Low-$O(bs_{fixed}nd)$ | Median-$O(bsnd)$ | Low-$O(bns_{fixed}^2d)$ | Low--(3%-5%) |\n| Quest | Dynamic | High-$O(bs'n_qd)$ | Median-$O(bsnd)$ | Median-$O(bns'^2d)$ | Median--(1%-2%) |\n| DS | Dynamic | Low-$O(bs'nd)$ | Median-$O(bsnd)$ | Median-$O(bns'^2d)$ | High--(0%-1%) |\n| Hshare | Dynamic | Low-$O(bs'nd)$ | Median-$O(bsnd)$ | Median-$O(bns'^2d)$ | High--(0%-1%) |\n| **SALS (Ours)** | Dynamic + Low Rank | Low-$O(bs'nr)$ | Low-$O(bsnr)$ | Median-$O(bns'^2d)$ | High--(0%-1%) |\n\nThe accuracy gap is difficult to compared. For the Palu' accuracy, we take the experiments from Table 3 in our paper. StreammingLLM is based on Tabel 3 in Hshare [1] paper on the longbench dataset. For the rest of methods , the accuracy gap is taken from Table 4 in our paper.\n\n[1] Wu, Huaijin, et al. \"HShare: Fast LLM decoding by hierarchical key-value sharing.\" The Thirteenth International Conference on Learning Representations. 2025.\n\n--- \n\n>\n> Q2 Additional experiments as stated in the weaknesses.\n> \n\n> W1 The evaluation models (Llama2-7b and mistral) are even not long-context models (Llama3.1 is more preferred).\n> \n\n> W2 The evaluation dataset is not standard. RULER, InfiniBench, and GSM-Infinite are more convincing.\n> \n\n**A:** Thank you for your suggestions. We have added benchmark results on the **RULER** dataset using **LLaMA3.1-8B-Instruct**.  **Table R2-a** and **Table R2-b** present our main results on 4k and 32k sequence lengths. Following the LongBench setting, we use an overall sparsity of 1/8. The results demonstrate that SALS retains high performance under long-context conditions, confirming its robustness.\n\nDue to the multimodal and video-based nature of **InfiniBench** and the lack of standardized implementations for **GSM-Infinite**, we focused our benchmark extension on **RULER**, which is fully text-based and compatible with our current evaluation pipeline. We plan to include GSM-Infinite and other long-context datasets in future evaluations as implementations mature.\n\nWe selected **LLaMA3.1-8B-Instruct** to better reflect practical long-context use. We report results at a sequence length of **32k**, as even optimized attention implementations (e.g., FlashAttention) face memory constraints beyond this point on single-GPU setups. The 32k setting thus offers a meaningful and representative context length for evaluation.\n\nWe will include these extended results in the revised version of the paper.\n\n\n**Table R2-a: Performance comparison of baseline and SALS methods on the RULER dataset with Llama3.1-8B-Instruct (4k sequence length).**\n\n| Method        | avg        | niah_single_1 | niah_single_2 | niah_single_3 | niah_multikey_1 | niah_multikey_2 | niah_multikey_3 | niah_multivalue | niah_multiquery | fwe    | qa_1  | qa_2 |\n|---------------|------------|----------------|----------------|----------------|------------------|------------------|------------------|------------------|------------------|--------|-------|------|\n| baseline | 92.8845    | 100            | 99.8           | 100            | 100              | 99.8             | 99.8             | 99.9             | 99.5             | 85.73  | 82.2  | 55   |\n| sals-25%      | 92.1045    | 100            | 99.6           | 99.8           | 100              | 100              | 99.8             | 99.7             | 99.45            | 78.8   | 81.2  | 54.8 |\n| sals-12.5%    | 90.6109    | 99.6           | 95.6           | 97.6           | 97.4             | 99.2             | 97.0             | 98.65            | 98.2             | 77.27  | 80.6  | 55.6 |\n\n**Table R2-b: Performance comparison of baseline and SALS methods on the RULER dataset with Llama3.1-8B-Instruct (32k sequence length).**\n\n| Method        | avg        | niah_single_1 | niah_single_2 | niah_single_3 | niah_multikey_1 | niah_multikey_2 | niah_multikey_3 | niah_multivalue | niah_multiquery | fwe    | qa_1  | qa_2 |\n|---------------|------------|----------------|----------------|----------------|------------------|------------------|------------------|------------------|------------------|--------|-------|------|\n| baseline_fp16 | 89.5618    | 91             | 98             | 100            | 98.8             | 98.4             | 98.8             | 98.75            | 98.1             | 80.33  | 76.4  | 46.6 |\n| sals-25%      | 89.2482    | 91             | 97.8           | 100            | 98.6             | 98.0             | 98.4             | 98.35            | 97.85            | 80.13  | 75.6  | 46.0 |\n| sals-12.5%    | 88.7755    | 93.4           | 97.6           | 99.2           | 98.2             | 97.2             | 93.0             | 95.3             | 98.3             | 83.13  | 75.2  | 46.0 |\n\n---\n\n> Q3 InfiniGen, ShadowKV also employs low-rank compression for KV cache. Could you provide any insights about the core differences?\n> \n\n**A:** Thanks for your comments. Although our SALS share some similarities with InfiniGen and ShadowKV in employing low-rank compression for KV cache, there exist several core differences:\n\n**(1) The difference from InfiniGen.**  InfiniGen is built based on two observations: the attention input similarity and the skewed partial weights. Skewed partial weights is the most related to our work, however, skewed weight is designed to identify the large-value channels (outliers) in the queries and keys without considering the RoPE position embedding. As discussed in our work, RoPE leads to increasing rank. In the InfiniGen work, the skewing (low-rank) matrix is fused with weights, and the outliers in the queries and keys are element-wise added to select the top-30% channels for the approximated attention scores computations.\n\nIn our work, we directly adopt the low rank keys and queries to compute the approximated attention scores. We do not fuse the orthogonal matrix into weights as discussed in our work, the RoPE operation introduces new variances and increases rank. Another key difference is that we compress the keys and only store the low-rank key cache, whereas InfiniGen stores the full rank keys in the CPU memory and performs prefetching of keys to avoid long communication latency.\n\n**(2) The differences fromShadowKV.** our method differs in two key aspects: the usage of low-rank compression and the strategy for critical token selection.\n\na) shadowKV performs SVD-based low-rank decomposition for each token during decoding, producing token-specific low-rank matrices (A, B). While this may offer finer-grained reconstruction, it introduces additional online compression overhead and increases memory usage. In contrast, our method applies **offline PCA-based calibration** to derive a **shared low-rank projection matrix** across all tokens, which significantly reduces runtime cost and simplifies the memory layout during inference.\n\nb) ShadowKV selects a chunk of tokens (e.g. 8 tokens) and adopts chunk-level approximation (mean of each trunk) for selecting important tokens. However, such approximation is not accurate enough, therefore, a small set of outlier chunks also has to be stored. In our case, we perform token level approximation by utilizing a smaller rank than stored KV cache. Our selection is much simpler without the need to perform complex outlier selection and also achieve comparable accuracy.\n\nThe discussion above will be added in our revised manuscript.\n\n---\n\n> **L1** There are many limitations generally for sparse attention. For example, the speed up is limited by how much memory taken by KV cache. I think this paper does not discuss limitations in the work.\n> \n\n**A:** Thanks for your comments.  We will add the following discussion about the limitations in the revised manuscript. \n\nIn this paper, we introduce the Sparse Attention in Latent Space (SALS) framework, which projects the KV cache into a compact latent space via low-rank projection and performs sparse token selection in this space for sparse attention. The benefits of adopting sparse attention is greatly depending on the sequence length. As shown in Table 5, when the sequence length is not long enough such as 1k, the sparse attention (latency 0.409ms) performs worse than the standard flash attention (latency 0.23ms). On the other hand, when the sequence length is 4k, sparse attention achieves 3.08X speed-up.  \n\nThe current paper has not explored the trade-off for the KV cache compression rate and the accuracy of token selection.  With consideration of the sequence length, there will be a sweet point to improve the compression rate without losing the accuracy.  Moreover,  the current paper considers a fixed compression rate and sparse ratio for all the layers. In the future work, we are interested to explore memory-performance-accuracy trade-offs on different layers."}, {"review_id": "CayWc01CQk", "reviewer": "Reviewer_CDzv", "summary": "The paper proposes SALS, a method to compress the KV cache in large language models by combining low-rank projection and token sparsity. Instead of reconstructing the full KV cache, SALS projects pre-RoPE keys into a compact latent space, selects top-k important tokens using lightweight scoring, and reconstructs only those tokens for final attention. This approach significantly reduces memory and compute without sacrificing accuracy. Experiments on Llama-2 and Mistral showcase that SALS achieve better accuracy and efficiency than existing baselines.", "questions": "- You project all multi-head keys into a shared latent space using a joint low-rank projection. Have you considered whether using per-head projections might improve token selection accuracy, despite the increased overhead?\n- SALS is conceptually similar to Loki, which also performs low-rank key compression and token selection. Could you clarify the key differences in token scoring or reconstruction strategies between SALS and Loki?", "limitations": "Yes.", "rating": 3, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 2, "clarity": 2, "significance": 2, "originality": 2, "strengths_and_weaknesses": "### Strengths\n\n1. This work provides good system support, showcasing great speedups relative to strong baselines such as FlashAttention.\n2. The proposed approach is well-motivated through careful analysis.\n3. Theoretical justifications are provided for using multi–head joint projection matrix.\n\n### Weaknesses:\n\n1. The paper reports memory access as a primary metric, but this does not directly capture the efficiency of the proposed method. It would be more informative to report latency or throughput for SALS and compare it with baseline methods.\n\n2. Most experiments use relatively short context lengths (1K, 2K, 4K as noted on line 301). It would strengthen the evaluation to include results on longer contexts, such as 32K tokens or more, which are more representative of long-context use cases.\n\n3. The contributions appear limited. The core idea of using low-rank projections to identify top-k tokens resembles existing methods like Loki [1]. The paper would benefit from a clearer explanation of what distinguishes SALS from these prior approaches.\n\n[1] Loki: Low-rank Keys for Efficient Sparse Attention", "was_revised": true, "final_justification": "I thank the authors for their detailed responses. Some of my concerns have been addressed and I have revised my score to 3. But I still have reservations about the paper's overall contribution. The technique of pre-RoPE compression has already been explored in prior works such as KIVI and KVQuant, and low-rank decomposition of the KV cache has been proposed in Loki. The proposed method appears to be a combination of existing techniques rather than a fundamental advancement in compute- and memory-efficient attention.", "rebuttal": "> W1 The paper reports memory access as a primary metric, but this does not directly capture the efficiency of the proposed method. It would be more informative to report latency or throughput for SALS and compare it with baseline methods.\n> \n**A:** Thank you for the valuable suggestions. Table R1 reports attention‑operator latency and the corresponding relative memory‑access ratios at **batch = 16, sequence length = 4 k**. \n\nAs you pointed out, the latency of SALS does not reach its theoretical optimum for short sequences. Nevertheless, the measured latency mirrors the trend in the memory‑access ratio, and as the sequence length grows the fixed overhead becomes negligible; the latency ratio therefore converges toward the relative memory‑access ratio, as demonstrated in Table R2.\n\n**Table R1: Latency and relative memory‑access ratios (Flash‑Attention = 1.0) at batch = 16, sequence length = 4 k**\n\n| **Method** | **Latency (ms)** | **Memory‑Access Ratio** |\n| --- | --- | --- |\n| Flash‑Attn | 3.230 | 1.00 |\n| Loki | 2.101 | 0.19 |\n| Double Sparse | 1.319 | 0.16 |\n| H‑Share | 1.067 | 0.14 |\n| SALS 25% | 0.757 | 0.11 |\n| SALS 12.5% | 0.565 | 0.06 |\n\n---\n\n> W2 Most experiments use relatively short context lengths (1K, 2K, 4K as noted on line 301). It would strengthen the evaluation to include results on longer contexts, such as 32K tokens or more, which are more representative of long-context use cases.\n> \n\n**A:** Thank you for pointing out the importance of long sequences. We have extended our experiments from 8 k up to **128 k tokens**; the new measurements are summarized in **Table R2** below (batch = 16, A100 80 GB, FP16). \n\n**Table R2: Attention operator latency and relative latency ratios of SALS‑25 % / SALS‑12.5 % versus Flash‑Attention at batch = 16, A100 80GB, FP16.**\n\n| **Seq Len (tokens)** | **Flash -attn(ms)** | **SALS‑25 % (ms)** | **SALS‑12.5 % (ms)** |\n| --- | --- | --- | --- |\n| 8 k | 6.072 | 1.247 | 0.843 |\n| 16 k | 11.850 | 2.282 | 1.469 |\n| 32 k | 23.922 | 4.212 | 2.600 |\n| 64 k | 51.056 | 8.294 | 5.082 |\n| 128 k | 102.628 | 16.240 | 9.830 |\n\nAt a 4 k sequence length, SALS delivers a **5.7 ×** speed-up in attention operator latency. However, when the sequence length increases to 128 k, the fixed-overhead fraction becomes negligible, and the attention operator under **SALS-12.5%** achieves a **10.44 × acceleration** over FlashAttention.\n\nThis speed-up aligns with the **theoretical latency reduction** discussed in Section 4.5 of our paper: since the dominant cost is memory access, the acceleration factor approximates the reciprocal of the memory-access ratio. For **SALS-12.5%**, the ratio is **0.06**, yielding a theoretical upper bound of **~16.6×**, which closely matches the trend observed in practice.\n\n---\n\n> W3 The contributions appear limited. The core idea of using low-rank projections to identify top-k tokens resembles existing methods like Loki [1]. The paper would benefit from a clearer explanation of what distinguishes SALS from these prior approaches.\n> \n\n**A:** Thank you for your comments. Our work significantly differs from Loki in three manifolds:\n\n**1) Latent Space with RoPE**\n\nLoki obtains its low-rank projection matrix by performing PCA on either the pre-RoPE or post-RoPE keys. Regardless of how the projection is computed, Loki ultimately applies the transformation to **post-RoPE** keys and queries, and retains **full-rank** representations. This enables approximate attention score computation for token selection, but **does not perform actual compression**.\n\nAs discussed in Section 3.1 of our paper, performing PCA after RoPE results in a significantly higher intrinsic rank, which leads to poor **low-rank approximation accuracy**. To address this, SALS applies low-rank projection **before** RoPE, enabling both accurate token selection and **key compression**. We also introduce a novel token selection mechanism in the pre-RoPE latent space, achieving better accuracy–latency trade-offs compared to Loki.\n\n\n**2) KV cache compression**\n\n As mentioned in the limitation part of Loki paper, Loki has not achieved reduced memory usage of the KV cache. The KV cache is not compressed and stored in the low-rank fashion. Loki only used the low-rank queries and keys to perform the approximate attention. In the appendix E of Loki paper, Loki also proposed PCA-Attn, which directly used the reduced-dimensional scores. In this way, the KV cache is compressed but PCAAttn performs poorly comparing to all the baselines as summarized in Table 5 (Loki paper).\n\nIn constrast, our work achieves KV cache **3.57× (1/0.28)** compression rate as shown in Table 2 (SALS paper). Moreover, because of the KV cache compression, the memory access is also reduced compared to Loki.\n\n\n\n**3) Joint-head compression vs. per-head compression**\n\nLoki's strategy is to perform per-head low-rank compression. As discussed in Fig. 2 from the Loki paper, the compression rate is around 50% for most models (e.g., for Llama2-7B, head dimension is compressed from 128 to 80 for most layers).\n\nDifferently, our strategy is to perform joint-head compression, leading to more compression ratio. As shown in Table R2 of our paper, we can achieve **4× and 8× compression** while maintaining the accuracy.\n\n**Summary:** To achieve KV cache compression using the low-rank strategy brings new challenges to the existing method. Our work tackles this challenge by merging heads for compression and carefully integrating rotary embedding in the SALS architecture. We achieve **6.4× KV cache compression** while maintaining the accuracy on 4K sequences compared to FlashAttention 2.\n\nReference: Singhania, Prajwal, et al. *\"Loki: Low-rank keys for efficient sparse attention.\"* Advances in Neural Information Processing Systems 37 (2024): 16692-16723. https://arxiv.org/pdf/2406.02542\n\n---\n\n> Q1: You project all multi-head keys into a shared latent space using a joint low-rank projection. Have you considered whether using per-head projections might improve token selection accuracy, despite the increased overhead?\n> \n\n**A:** Thank you for the insightful question. We explored this alternative and implemented a **per‑head** variant where each attention head is equipped with its own rank‑*r* projection matrix. To ensure a fair comparison, we kept the total projection rank constant: for *h* heads the per‑head rank is set to *r / h*, so both variants introduce the same number of projected dimensions. To ensure robustness, our results are averaged over 500 input samples and cover all attention layers.\n\n**Table R3:  Token‑selection alignment of joint vs per‑head low‑rank projections**\n\n| Metric ↑ | Joint (mean) | Per‑head (mean) |\n| --- | --- | --- |\n| **Overlap Score** | **0.931** | 0.812 |\n| **Overlap Rate** | **0.624** | 0.469 |\n\nTo evaluate the token selection module, we report two metrics in Table R3: **Overlap Score**, which reflects the **effectiveness** of selection in capturing high-attention tokens (i.e., whether the selected tokens are valuable under full attention), and **Overlap Rate**, which measures the **accuracy** of selection by comparing against the top-*k* tokens in the full attention.\n\nAs shown in Table R3, joint-head projections outperform per-head projections. While joint projections may sacrifice per-head specificity, they better preserve the globally important directions in the low-rank space, as discussed in Lemma 1. This results in higher overall token selection accuracy.\n\nIn addition, as also observed in prior work (Palu), **joint-head projections have the advantage of preserving the common principal components shared across heads**, thereby achieving better compression efficiency under a constrained projection budget.\n\n---\n\n> Q2: SALS is conceptually similar to Loki, which also performs low-rank key compression and token selection. Could you clarify the key differences in token scoring or reconstruction strategies between SALS and Loki?\n> \n\n**A:** Thank you for your comments. Besides the difference in the Latent space with RoPE,  KV cache compression, and joint-head compression mentioned in the previous reply, the token scoring and reconstruction strategies are also different:\n\n(1) Since Loki performs the compression on each head, the token scores are for each head and tokens are selected differently for each head. For our case, we perform the joint-head compression, where each head selects the same tokens.\n\n(2) For the reconstruction strategies, Loki performs no KV cache compression, and thereby it can directly select the chosen KV cache. However, for our SALS, the selected token is stored in the compressed fashion and only reconstructed when it is selected."}, {"review_id": "UTIRm8Cb7F", "reviewer": "Reviewer_v3oz", "summary": "The paper introduces a framework named SALS to compress the KV cache and accelerate the generation of LLMs. It first analyzes the impact of RoPE on QK low rank and adopts a pre-RoPE low rank strategy. To build the framework, it transforms queries and keys into latent space, selects critical tokens in latent space and reconstructs them for sparse attention. Evaluations demonstrates the effectiveness and efficiency of SALS.", "questions": "See weaknesses.", "limitations": "See weaknesses.", "rating": 4, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 3, "significance": 3, "originality": 3, "strengths_and_weaknesses": "Strengths:\n1) The paper is clearly written and easy to follow, with sufficient empirical observations demonstrating the motivation for each design.\n\n2) The proposed framework is carefully designed and demonstrates strong performance.\n\n\nWeaknesses:\n1) The value tensors remain in full rank. Despite the paper adopts quantization to compress value cache, it complicates the overall framework. Experimental results with low rank value caches should be provided.\n2) A discussion between SALS and NSA [1] should be provided.\n3) Incorporating comparisons of multiple value cache quantization methods (e.g. [2-3]) can further enhance the robustness of the SALS.\n\n\n[1] Native Sparse Attention: Hardware-Aligned and Natively Trainable Sparse Attention. Arxiv 2502.11089.\n\n[2] ZipCache: Accurate and Efficient KV Cache Quantization with Salient Token Identification. NeurIPS 2024.\n\n[3] GEAR: An Efficient KV Cache Compression Recipe for Near-Lossless Generative Inference of LLM. Arxiv 2403.05527.", "was_revised": true, "final_justification": "The authors provide detailed clarifications about low rank value caches and a comparison with the NSA method. I have no further questions to the authors. Therefore, I will keep my rating.", "rebuttal": "> W1. The value tensors remain in full rank. Despite the paper adopts quantization to compress value cache, it complicates the overall framework. Experimental results with low rank value caches should be provided.\n> \n\n**A:** Thank you for your suggestions. We have added the value compression results in the Table R1 below. After the PCA compression, 50% and 25% values are stored. We find that the accuracy drop when performing value compression is larger than the quantization of SALS. The high sensitivity of value cache is also observed in [1].\n\n**Table R1: Performance on LongBench under different value compression settings.**\n\n| Method     | Single-Document QA | Single-Document QA | Summarization | Few-shot Learning | Synthetic | Code Completion | avg     |\n|------------|---------------------|---------------------|----------------|--------------------|-----------|------------------|----------|\n| baseline   | 24.50               | 22.18               | 23.64          | 62.80              | 6.50      | 56.29            | 32.65   |\n| SALS-25%   | 24.37               | 22.50               | 23.40          | 63.01              | 6.50      | 53.80            | **32.26** |\n| v-50%      | 21.64               | 22.06               | 22.32          | 61.59              | 6.00      | 49.07            | 30.45   |\n| v-25%      | 19.64               | 20.28               | 18.95          | 58.59              | 5.25      | 39.30            | 27.00   |\n\n[1] Yixin Ji, Yang Xiang, Juntao Li, Qingrong Xia, Zi Ye, Xinyu Duan, Zhefeng Wang, Kehai Chen, and Min Zhang. 2024. *Adaptive Feature-based Low-Rank Compression of Large Language Models via Bayesian Optimization.* In Findings of the Association for Computational Linguistics: EMNLP 2024, pages 4152–4168, Miami, Florida, USA. Association for Computational Linguistics.\n\n---\n\n> W2. A discussion between SALS and NSA [1] should be provided.\n> \n\n**A:** Thank you for your suggestions. We will add the following discussion in our revised paper.\n\nNatively sparse attention (NSA) employs a dynamic hierarchical sparse strategy with coarse-grained token compression and fine-grained token selection. The major difference between NSA and SALS is that adopting NSA requires training from scratch, whereas SALS can be calibrated and achieved during the post-training phase. SALS shares the similar idea to perform the top-k token selection. However, NSA performs this token selection based on the trained compressed attention module. SALS performs the token selection based on the low-rank approximation.\n\n---\n\n> W3. Incorporating comparisons of multiple value cache quantization methods (e.g. [2-3]) can further enhance the robustness of the SALS.\n> \n\n**A:** Thank you for your suggestions. We have added the ZipCache and GEAR quantization methods for comparisons using llama-7b-chat on the longbench dataset. As reported, GEAR, ZipCache and kivi quantization achieve similar accuracy under 4× (25%) compression rate, which illustrates the robustness of our SALS algorithms in incorporating different value cache quantization methods.\n\n**Table R2: Performance on LongBench under different value compression methods.**\n\n| Method | Single-Document QA | Single-Document QA | Summarization | Few-shot Learning | Synthetic | Code Completion | avg |\n| --- | --- | --- | --- | --- | --- | --- | --- |\n| baseline | 24.50 | 22.18 | 23.64 | 62.80 | 6.50 | 56.29 | 32.65 |\n| SALS-25%-kivi | 24.37 | 22.50 | 23.40 | 63.01 | 6.50 | 53.80 | 32.26 |\n| SALS-25%-gear | 23.15 | 22.56 | 22.85 | 63.63 | 6.25 | 55.22 | 32.28 |\n| SALS-25%-zipcache | 24.33 | 22.24 | 23.29 | 62.87 | 6.50 | 56.05 | **32.55** |"}, {"review_id": "Wsgnf0yoJg", "reviewer": "Reviewer_BamQ", "summary": "The authors introduce Sparse Attention in Latent Space (SALS), a lightweight framework that reduces both storage and computation during inference by compressing the Key–Value (KV) cache. Rather than applying Rotary Position Embeddings (RoPE) before compression — which inflates the effective rank of key matrices — SALS first projects all multi-head key vectors into a shared, low-dimensional latent space via a rank-r eigen-projection. In this latent domain, approximate attention scores are computed using only the top coordinates, enabling selection of a small set of the most salient tokens. Only the corresponding top-k key–value pairs are then reconstructed back into their original space and attended using the standard mechanism, yielding a sparse attention over a highly compact cache representation. Empirical evaluation on LLaMA2-7B-Chat and Mistral-7B across benchmarks such as GSM8K, CoQA, and LongBench shows that SALS matches or surpasses the accuracy of FlashAttention2 and GPT-Fast baselines while achieving up to a 6.4× reduction in KV-cache size, a 5.7× speed-up in the attention operator at 4K context length, and overall 1.4–4.5× gains in end-to-end throughput for contexts ranging from 4K to 32K tokens. These results demonstrate that SALS can be seamlessly integrated into existing large-language-model pipelines to extend context lengths with minimal performance trade-offs.", "questions": "Q1: It would strengthen the empirical foundation to report SALS’s performance alongside contemporary sparse-attention methods, especially dynamic selection method like Quest or ArkVale.\n\nQ2: Although you include a GQA-based model, the efficiency analysis may remain tailored to standard multi-head attention. Providing benchmarks on grouped-query attention layers — reporting memory access and throughput — would clarify SALS’s behavior under these widely used architectures.\n\nQ3: Table 3 appears without narrative discussion, leaving its per-task metrics under-interpreted. A brief in-text analysis defining each column, highlighting key trends across Single-QA/Multi-QA/Summarization, and relating memory-access reductions to latency savings could improve clarity.", "limitations": "Yes", "rating": 4, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 3, "significance": 2, "originality": 2, "strengths_and_weaknesses": "Strengths:\n\n+ The authors present a rigorously derived three-stage pipeline, including latent-space projection, critical-token selection, and selective reconstruction, that is mathematically sound and clearly justified. Their use of eigen-projection to compress pre-RoPE key vectors is underpinned by solid linear-algebraic reasoning, and the paper carefully measures the approximation error at each stage, demonstrating high fidelity to full attention.\n+ By achieving up to 6.4× KV-cache compression and 1.4 - 4.5× end-to-end throughput gains on LLaMA2-7B-Chat and Mistral-7B, SALS addresses one of the most pressing bottlenecks in scaling transformer contexts.\n+ The core idea of performing latent-space attention scoring prior to RoPE - thus sidestepping rank inflation - is a novel twist on existing compression and sparsity techniques. While latent projections and sparse attention have each seen prior study, their careful integration here constitutes a fresh contribution.\n\nWeaknesses:\n\n- Although the pipeline is robust, the overall methodological novelty is somewhat incremental: SALS combines two well-explored ideas (latent compression and sparse token selection) without introducing fundamentally new theory.\n\n- The paper does not deeply investigate how SALS interacts with grouped-query attention (GQA) and multi-query attention (MQA) variants. Although a GQA-based model is included in the experimental suite, the efficiency analysis remains focused on the multi-head-attention (MHA) case and fails to articulate how the latent-space selection and sparse reconstruction behave under GQA/MQA. This omission leaves open the possibility that the reported performance gains may not generalize beyond the standard MHA setting.\n\n- While throughput and memory gains are impressive, the experimental evaluation lacks comparisons to several leading sparse-attention frameworks, such as Quest, H2O and ArkVale that also target long contexts.", "was_revised": true, "final_justification": "In the paper, the authors propose SALS that compresses the KV cache to reduce both storage and computation. The paper demonstrates promising speedup. The authors have addressed my concerns on the compatibility with GQA, and on more thorough comparison with Quest, Arkvale, etc, which is the main reason for me to raise the score to 4. However, the concern on limited novelty remains as SALS combines two well-explored ideas (latent compression and sparse token selection) without introducing fundamentally new theory.", "rebuttal": "> W1:  Although the pipeline is robust, the overall methodological novelty is somewhat incremental: SALS combines two well-explored ideas (latent compression and sparse token selection) without introducing fundamentally new theory.\n> \n\n**A:** Thank you for your comments.  \n\nThe latent compression is studied by many recent works such as Loki and Palu. The KV cache compression strategy and compression rate are not fully discussed, while the accuracy is hard to maintain.  To achieve higher accuracy yet better KV cache compression rate, our work proposes the joint-head compression comparing to the existing works of per-head compression (Loki) and group-head compression (Palu).  Joint-head has the advantage of preserving of the common principal components shared among different heads and thereby achieves better compression rate. Loki has not achieved any compression on KV cache whereas Palu achieves high compression rate (50%) but suffers from the accuracy loss (>5%).   On the other hand, our work achieves high compression rate (25%）by adopting joint-heads compression. Furthermore, with the sparse attention, the reconstruction overhead is significantly reduced with maintained accuracy (loss gap <1%).  \n\n---\n\n> W2: The paper does not deeply investigate how SALS interacts with grouped-query attention (GQA) and multi-query attention (MQA) variants. Although a GQA-based model is included in the experimental suite, the efficiency analysis remains focused on the multi-head-attention (MHA) case and fails to articulate how the latent-space selection and sparse reconstruction behave under GQA/MQA. This omission leaves open the possibility that the reported performance gains may not generalize beyond the standard MHA setting.\n> \n\n**A:** We are sorry for the confusion caused.  As shown in Figure 3, the token is selected based on the joint-head low-rank compressed latent space.  For the token selection stage, the latent keys and quires both only have one head. As such, there is only one head to select the token index from latent key cache and these indexes are later shared  by all the heads in the sparse attention.  Therefore, for GQA/MQA,  SALS works fine by sharing the critical tokens among the KV heads. \n\nOur experiment results in Table 5 are also performed based on the GQA setting (32 query heads and 8 KV heads).  We have shown clear speed-up (4.27X) comparing to the Flash-attention under the setting of batch size 16 and 16k sequence length. \n\n---\n\n> Q1 It would strengthen the empirical foundation to report SALS’s performance alongside contemporary sparse-attention methods, especially dynamic selection methods like Quest or ArkVale.\n> \n\n> W3 While throughput and memory gains are impressive, the experimental evaluation lacks comparisons to several leading sparse-attention frameworks, such as Quest, H2O, and ArkVale that also target long contexts.\n> \n\n**A:** Thank you for highlighting recent sparse attention baselines. We have included experimental comparisons to **Quest** and **H2O**, which are leading dynamic selection methods. The results are summarized in **Table R1** below.\n\n**Table R1: Accuracy, latency, and memory access comparison with recent sparse attention baselines on LongBench.**\n\n| Method | Accuracy (avg / LongBench) | Latency (ms) | Memory Access ↓ |\n| --- | --- | --- | --- |\n| Flash‑Attn | 32.6519 | 3.23 | 1.00 |\n| H2O | 31.0919 | 0.85 | 0.12 |\n| Quest | 30.9719 | 2.21 | 0.25 |\n| SALS‑25% | 32.2628 | 0.757 | 0.11 |\n| SALS‑12.5% | 31.9706 | 0.565 | 0.06 |\n\nWe compare SALS with FlashAttention, Quest, and H2O in terms of average accuracy (on LongBench), attention operator latency (ms), and normalized memory access. SALS achieves competitive accuracy with significantly lower memory cost and latency\n\nWe also attempted to run **ArkVale**, but unfortunately its official implementation produced incorrect outputs without modification, and significant internal changes would be required to integrate it into our evaluation pipeline. Given the risk of introducing discrepancies, we chose not to include ArkVale results in this version.\n\nThat said, SALS differs from ArkVale in several key aspects (e.g., latent-space selection, fully shared selection heads), and we plan to revisit this comparison in future work with a stabilized version of ArkVale.\n\n---\n\n> Q2 Although you include a GQA-based model, the efficiency analysis may remain tailored to standard multi-head attention. Providing benchmarks on grouped-query attention layers — reporting memory access and throughput — would clarify SALS’s behavior under these widely used architectures.\n> \n\n**A:** Thank you for the question regarding SALS’s behavior under Grouped-Query Attention (GQA) and Multi-Query Attention (MQA).\n\nOur method is fully compatible with GQA/MQA architectures due to the **shared token selection mechanism**: token selection is performed in a joint latent space and is shared across all query heads. As a result, **the number of memory accesses does not increase with the number of query heads**, and **the selection cost remains constant regardless of the GQA/MQA configuration**.\n\nFurthermore, as discussed in Section 4.5 of our paper, the memory access formula is **independent of the number of attention heads**, ensuring that the efficiency gains of SALS naturally extend to GQA and MQA architectures.\n\nWe provide additional runtime benchmarks in **Table R2** below, which reports the attention operator latency and normalized memory access cost (FlashAttention = 1.0) under a **GQA configuration** (32 query heads, 8 KV heads, batch size = 16, sequence length = 4k). These results are consistent with the end-to-end throughput evaluation presented in Table 6 of our main paper and further demonstrate that SALS achieves strong efficiency improvements under GQA settings.\n\n**Table R2: Latency and relative memory‑access ratios (Flash‑Attention = 1.0) at batch = 16, sequence length = 4 k**\n\n| **Method** | **Latency (ms)** | **Memory‑Access Ratio** |\n| --- | --- | --- |\n| Flash‑Attn | 3.230 | 1.00 |\n| Loki | 2.101 | 0.19 |\n| Double Sparse | 1.319 | 0.16 |\n| H‑Share | 1.067 | 0.14 |\n| SALS 25% | 0.757 | 0.11 |\n| SALS 12.5% | 0.565 | 0.06 |\n\n---\n\n> Q3 Table 3 appears without narrative discussion, leaving its per-task metrics under-interpreted. A brief in-text analysis defining each column, highlighting key trends across Single-QA/Multi-QA/Summarization, and relating memory-access reductions to latency savings could improve clarity.\n> \n\n**A:** Thank you for the insightful suggestion. We agree that Table 3 would benefit from clearer narrative guidance. In the revised version, we will include a brief in-text analysis defining each column, highlighting trends across key task types (e.g., Single-QA, Multi-QA, Summarization), and explaining how reduced memory access contributes to improved efficiency. Specifically, we will add the following narrative:\n\nTable 3 compares task-level performance across six LongBench categories, alongside normalized memory access costs. The first three columns reflect core reasoning tasks, where SALS achieves consistently strong performance. Notably, SALS-12.5% retains competitive accuracy while reducing memory access to just 6% of the baseline. This indicates that SALS preserves informative tokens more effectively. The memory-access savings translate into practical latency improvements, as fewer KV memory lookups reduce attention computation overhead.\n\n\nTo further support this, we will also include the Table R2 in the final version, which shows a clear correlation between memory-access reduction and latency savings across multiple methods:"}]

Author rebuttal:

Meta-review: This paper introduces SALS, a method for KV cache compression in LLMs that combines pre-RoPE low-rank projection with sparse token selection in a latent space. The key insight is that applying RoPE inflates the effective rank of key vectors, making compression less effective; by compressing in the pre-RoPE space and using joint-head PCA projections, SALS achieves substantial KV-cache compression and attention operator speedups while maintaining accuracy. The approach is carefully engineered, and the authors provide system-level implementation and evaluations showing up to 6.4× compression, 5.7× operator speedup, and 1.4–4.5× throughput gains on contexts up to 128k tokens.

The main strengths are the clarity of exposition, strong empirical validation (including extensions during rebuttal to RULER at 128k tokens, and comparisons to baselines such as Quest, H2O, and ShadowKV), and theoretical justification for joint-head projection. Reviewers appreciated the careful design and the robustness of results across models.

The weaknesses lie in the perceived incremental novelty. Several reviewers noted that SALS builds upon existing low-rank and sparsity ideas (e.g., Loki, ShadowKV, InfiniGen), and while the pre-RoPE latent-space integration is a meaningful refinement, it may not represent a fundamental theoretical breakthrough. Evaluation breadth was also initially limited, though the rebuttal strengthened this with additional long-context and baseline comparisons. Some questions about interaction with GQA/MQA and value-cache compression were raised, and the authors provided clarifications and ablations showing compatibility and robustness.

Overall, the rebuttal and discussion addressed most concerns to a reasonable degree. While the contribution is more incremental than transformative, the system-level execution, thorough analysis, and empirical results establish SALS as a useful and practical advancement for efficient long-context inference.
Decision: Accept (poster)