MUSTAFAR: Promoting Unstructured Sparsity for KV Cache Pruning in LLM Inference
Authors: Donghyeon Joo
Helya Hosseini
Ramyad Hadidi
Bahar Asgari
Venue: NeurIPS 2025 poster ()
Keywords: KV cache compression
Unstructured Pruning
Efficient LLM Inference
Primary area: infrastructure
TL;DR: Our work, Mustafar, unlocks 70% sparsity in KV cache pruning by leveraging unstructured sparsity pattern, supported by a custom attention kernel, and boosts the inference efficiency of LLMs.

Abstract: We demonstrate that unstructured sparsity significantly improves KV cache compression for LLMs, enabling sparsity levels up to 70\% without compromising accuracy or requiring fine-tuning. We conduct a systematic exploration of pruning strategies and find per-token magnitude-based pruning as highly effective for both Key and Value caches under unstructured sparsity, surpassing prior structured pruning schemes. The Key cache benefits from prominent outlier elements, while the Value cache surprisingly benefits from a simple magnitude-based pruning despite its uniform distribution. KV cache size is the major bottleneck in decode performance due to high memory overhead for large context lengths. To address this, we use a bitmap-based sparse format and a custom attention kernel capable of compressing and directly computing over compressed caches pruned to arbitrary sparsity patterns, significantly accelerating memory-bound operations in decode computations and thereby compensating for the overhead of runtime pruning and compression. Our custom attention kernel coupled with the bitmap-based format delivers substantial compression of KV cache up to 45\% of dense inference and thereby enables longer context lengths and increased tokens/sec throughput of up to 2.23$\times$ compared to dense inference. Our pruning mechanism and sparse attention kernel is available at https://github.com/dhjoo98/mustafar.

MUSTAFAR: Promoting Unstructured Sparsity for KV Cache Pruning in LLM Inference
Donghyeon Joo', Hella Hosscini', Ramyad Hadidi Bahar Asgari' Department of Computer Scienee, University of Maryland, 'd-Matrix {dhj 0098, hel ia, bahar) edud detidiidddmaiiiii.mmmmm .i
Abstract
We demonstrate that unstructured sparsity significantly improves KV cache com- pression for LLMs, enabling sparsity levels up to 70% without compromising accuracy or requiring fine-tuning. We conduct systematic exploration of pruning strategies and find per-token magnitude- magnitude-based pruning highly effective for both Key and Value caches under unstructured sparsity, surpassing prior structured pruning schemes. The Key cache benefits from prominent outlier elements, while the Value cache surprisingly benefits from a simple magnitude-based pruning de- spite its uniform distribution. KV cache size the major bottleneck in decode performance due to high memory overhead for large context lengths. To address this, we use bitmap-based sparse format and custom attention kernel capable of compressing and directly computing over compressed caches pruned to arbitrary sparsity patterns, significantly accelerating memory-bowe bounn operations in decode computations and thereby compensating for the overhead of runtime pruning and compression. Our custom attention kernel coupled with the bitmap- based format delivers substantial compression of KV cache up to 45% of dense inference and thereby enables longer context lengths and increased tokens/sec throughput up to 2.23x compared to dense inference. Our pruning mechanism and sparse attention kernel is available at hnttps://githube hotpa///ocoo/oooooo///aaaaaaa
Introduction
In the age of Large Language Models (L.LMs), advances in the machine learning domain [41 2 6 and
the fast and efficient computing systems [21 35] have led to the emergence of highly capable LLMs that can summarize a book [22). write compelling story [18), code a library (53), and generally reason over longer contexts than ever before 77). As LLMs are increasingly tasked with processing
longer sequences, memory overhead associated with key-value (K caching has emerged as critical bottleneck to scaling context length.
Prior work has approached the challenge of KV cache memory overhead through techniques such as quantization [0 48 52], low-rank approximation [7
37
50
26
token-wise eviction [5T]
29
and structured pruning (e.g., channel- removal 44 31 The need improve
individual compression techniques has become increasingly important, especially as joint applications of multiple methods, such as pruning combined with token eviction (44) quantization with token-wiin wise eviction [52). , and low-rank approximation with quantization (4), gain popularity. However, previous work on K cache pruning have been limited to structured pruning, primarily due to the difficulty of efficiently leveraging finer-grained (i.e., unstructured) sparsity during execution. Effective pruning of the KV cache entails two core challenges: (1) achieving substantial reduction in KV cache size while preserving model accuracy, and (2) ensuring that the runtime pruning and compression processes sufficiently efficient (i.c., the associated overhead must not outweigh the latency gains introduced by the resulting sparsity).
39th Conference on Neural Information Processing Systems (NcurIPS 2025).

---

In this paper, we find that removing any con- straint on the sparsity pattern, effectively un- structured sparsity can ensure that compressed KV cache perform with minimal model accu- racy degradation while being pruned to higher sparsity. In Section 2 (green region of Figure [], we first present journey to find the optimal pruning algorithm for the key and value cache, based the element magnitude distributions of the KV cache. We explore the feasibility of various pruning algorithms both KV cache to conclude that applying a simple per-token magnitude-base based pruning both Key and Value caches is capable of preserving the model accu- racy at high sparsity, while also demonstrating strong compatibility with orthogonal compres- sion techniques.
LLH inference (Pretil and Decada)
Kay cachs outler chammsts.
Attendon by dattn
Chernel
poaning
?
Vakue cookte per-tolas prosing le ougurt amara
Cronnet
Compresston
Seation: 3, Pruring Algurtti
Sectiond: Sparse Attention
Figure I: High-level overview of Mustafar. Green region describes the pruning algorithm of Sec- tion 2
pink region describes the custom sparse attention kernel of Section
Section 3 (pink region of Figure I discusses the next step: having induced sparsity in the KV cache, the challenge becomes leveraging the unstructured sparsity to reduce memory footprint and accelerate computation. To this end, we adopt a bitmap-based based sparse format that serves two purposes. First, the bitmap enables maximal compression matrices with arbitrary sparsity patterns. Second, this maximal compression of matrix operands translates into computational speedup of the attention operation, which is severely memory-bound on GPUs. Alongside the sparse format, we introduce the custom attention kernel tailored to operate the bitmap-based sparse format. We that the speedup of our attention kernel overshadows the latency introduced by runtime pruning and compression, meanwhile effectively compressing the KV cache to high sparsity with minimal accuracy degradation.
In summary, we demonstrate that adopting unstructured sparsity in the KV cache without imposing constraints the pruning pattern enables higher degrees of sparsity while preserving model accuracy. Furthermore, we introduce the necessary computational tools to support unstructured sparsity effi- ciently, ensuring that the derived high sparsity leads to gains in memory compression and end-to-end inference throughput.
Pruning Algorithm for Unstructured Sparsity
Question: Does removing structural constraints in KV cache pruning allow for higher sparsity while preserving model accuracy more effectively than structured pruning methods?
We explore the potential unstructured sparsity KV cache pruning by considering the two factors for Key and Value eache pruning: pruning direction and output- awareness.. Pruning Direction refers the axis along which sparsity is induced when selecting elements for removal. Since both the Key and Value caches are represented as matrices with dimensions [lokens x channels) we consider two primary pruning directions: per-channel pruning, which determines target sparsity across each channel (i.e., across tokens for each channel), and per-token pruning, which determines target sparsity across each token's cache (i.e., across model dimensions for each token). Output-Awareness refers to the use of scoring metric that serves as a proxy for estimating each element's 's contribution to the operation's 's output. Commonly employed in LLM weight pruning [38] and structured KV cache pruning [44], this technique involves computing a score for each pruning unit such as channel or an element by taking the product of the corresponding element with its associated input. This approach effectively captures the element's 's influence on the final output, guiding more informed pruning decisions. For a fair and effective comparison between pruning strategies, we uniformly employ local dense window, where the recent 32 tokens remain untouched during the decode phase. Previous works (5T) (44 have shown that this effective in preserving model accuracy, meanwhile being small enough in size to not severely impact the compression.

---

2.1 Pruning Key Cache
In deciding the pruning direction, we build on top of the observation of KIVI [30]. that Key cache exhibits distinct channel-win wwie outliers, where "channel" refers to the head dimension (Figure 2a). This leads us to focus on per-token pruning for key cache, as can effectively capture the elements the outlier channel.
Layer 6 Keys
Layer 8 Keys
Layer Values
Layer 8 Values
2
2000 Taken
2000
2000 Taken
2000
1000
1000
Taken
1000
0
1000
Taken
50 Head Dim
50
Head Dim
Head Dim
100
Head Dim
100
0
(a) Magnitude distribution of Key cache
(b) Magnitude distribution of Value cache
Figure 2: Visualization of the KV cache in LLaMA-2 7B. Color intensity indicates element magnitude. The figure was generated using the visualization code from KIVI 30
Based on the same observation to perform structured pruning of individual channels, ThinK [44 incorporates output-awareness by using per-channel score of the accumulation of last 32 query. multiplied by each channel. To this end we compare the accuracy of ThinK (44), per-token magnitude- based unstructured pruning, and output-aware unstructured pruning of our design.
Figure elaborates the per-token output-aware of Key cache. The element-wise Lı accumulation of the current and next 31 Query vector (blue) is multiplied element-wise across each token's key vector (pink) derive the pruning score (green). The absolute value of the score element in the corre- sponding position of each Key cache element is used to decide the elements to be pruned within token's Key vector. In other words, we for- mulate the per-token output-aware unstructured pruning score S of Key cache K to be:
unstructured pruning
Chermet
Conebrution Ourns
Figure 3: Per-token, output-aware pruning of Key cache
S = |K| C broadcast
Σ
where Qr is the query at time t
where multiple queries correspond to KV cache pair, we sum the pruning score of all queries mapped each KV cache.
For Group Query Attention (GQA)
Table 1: Comparison of ThinK 44 structured pruning, per-token magnitude-based unstructured pruning, and per-token output-aware unstructured pruning on LongBench with Llama-3-8B- Instruct Key cache. K, denotes Key cache sparsity.
K, = 0.5 Unstructured Output-aware
K, = 0.7
Task
Dense
ThinK (Structured)
Unstructured
Think (Structured)
Unstructured Output-aware
Unstructured Magnitudo
Magnitude
Average
43.19
38.53
43.23
42.84
26.55
42.13
41.55
Dong QA
36.66
35.61
36.57
36.90
25.26
35.78
35.53
MuetiDoc QA
36.09
34.99
35.92
35.77
29.75
35.55
35.40
Summarization
26.75
24.96
26.87
26.45
17.70
25.16
25.18
Few-shot
68.96
66.54
68.82
68.75
44.88
67.22
67.84
Symthetic
37.25
35.50
37.00
36.75
16.86
35.25
35.00
Code
55.58
29.56
56.61
54.14
19.15
56.19
51.47
3

---

In Table we compare Llama-3-8P B-n-n-rrtett accuracy of different pruning methods on LongBench [3]. For structured pruning, we see that even moderate sparsity, model accuracy retention is dismal compared to pruning to unstructured sparsity pattern. Notably, unstructured pruning is capable of outperforming structured pruning even without the memory footprint of pruning scores involved with output-wwareess. Applying output-awareness to unstructured pruning results in slight improvement the LongBench total average score, while individual task performance is mixed with each method outperforming the other different tasks.
Key Cache Verdiet: While the existence of outlier channels with exceptionally high magni- tudes show promise for per-channel structured pruning, unstructured sparsity achieves higher accuracy at greater sparsity levels, even without output-awareness.
2.2 Pruning Value Cache
As shown Figure 26] Value cache exhibits more uniform distribution of activations, making challenging to apply the same channel-wise pruning without incurring substantial degradation in model accuracy. This difficulty has led recent Value cache pruning approaches to be more susceptible to accuracy degradation.
With no discernible outliers in certain direction, we explore all possible combinations of (pruning direction, magnitude/outtut--a aware) pairs. How- ever, we are able to rule out per-token output- aware pruning, as the attention formulation Allention.Score V Value involves multiply- and-accumulate operation along the token di- mension. As seen in Figure 4 every element of a token's 's Value cache is multiplied by the same element of the attention score, with each element's impact on the output proportionate to the magnitude of each value. That is, for Value cache pruning, per-token magnitude- based pruning is already output-aware. For per-channel pruning, we prune each channel to the target sparsity in groups of 32 tokens, for compatibility with the local window size. For per-channel output-aware pruning, we accumulate the current and subsequent 31 attention score a of each token, which is then element-wise multiplied to the corresponding Value Cache (V) element. The following formula describes the pruning score of per-channel output-aware pruning:
Taber
stutt beme
Figure 4: Output-aware per-channel (red) and magnitude-based per-token (pink) pruning of Value cache. Magnitude-basee per-token pruning is equal to output-aware per-token pruning (yellow).
T++31
V broadcast
Σ
[mrl)
where an is the attention score time /
44 structured pruning, per-channel magnitude based unstructured pruning, per-channel output-aware unstructured pruning, and per-token magnitude based pruning on LongBench (3] with 1aana----B-Instteee Value Cache. V. denotes Value cache sparsity.
Table 2: Comparison of ThinK
=.5
=
Task
Dense
ThinK (Structured)
Magnirude (Per-channel)
Ourput-aware (Per-chunnel)
Magnitude (Per-token)
ThinK (Structured)
Magnitude (Per-chanel)
Outpur- aware
Magnitude (Per-token)
(Per-chanel)
Average
$8.45
42.50
42.84
30.00
41.69
42.67
42.78
SinglaDoo QA
36.66
34.92
36.56
36.24
36.75
15:05
36.11
36.03
36.96
MultiDoe
36,09
31.74
35.15
36.07
36.22
23.90
35.11
36.20
35.82
Summarization
24.74
20.41
22.72
Few shot 68.96
67.18
67.66
68.65
68.91
60.16
67.39
68.23
68.08
Syntheric 37.25
35.43
38.31
37.00
36.25
29.63
38.75
37.25
35.50
Code 55.58
31.97
35.07
55,57
$5.77
20.85
52.65
56.17
57,62
As shown in the Table we first see that applying structured pattern to Value cache pruning incurs significant accuracy degradation even in 50% sparsity. This is concurrent with ThinK 44 findings, which points to 30% sparsity the upper-bound acceptable accuracy. In contrast, per-token

---

magnitude pruning is capable of preserving model accuracy even at 70% sparsity. For per-channel pruning, we see that incorporating output-awareness boasts model accuracy retention almost the level of per-token pruning. However, we prefer per-token magnitude-basee pruning for the following two reasons. First, output-aware per-channel value cache pruning requires access to the attention score which requires additional recomputation when used alongside FlashAttention [6]. where the full attention score matrix does not materialize in the global memory. Second, per-token magnitude- pruning allows smooth compatibility with orthogonal compression method token-wise eviction [24] 5I]. where the retained token's ' KV cache can be pruned individually. We examine the accuracy of joint application in Section 4.2
Value Cache Verdiet: All unstructured pruning methods explored outperform structured pruning. Among unstructured pruning methods, token-wise pruning, which is inherently output-aware by matrix multiplication formulation, best preserves model accuracy even at high sparsity levels. While channel-wise pruning with output-awarenes, can achieve comparable accuracy, token-wise pruning offers advantages both efficiency and modularity.
With the two verdicts in Key and Value caches, on Table 3 we finally validate the model accuracy retention of per-token magnitude- based pruning with both Key and Value caches pruned. Not only can Value cache be pruned to high sparsity with unstructured sparsity, but both KV cache can be pruned to 70% sparsity while showing similar better accuracy than Key-only 50% structured pruning of Think (44) In Appendix A. methodology of this section is applied on Llama-2 7B to reinforce the effectiveness of per-token magnitude-bast based KV cache pruning.
Table 3: Longbench Score of Llama-3-88 ---B--Insstcct and Per-Token Magnitude-based Pruning.
with KV Cache
--ama-3-8B-Insse
Mistral Mistral-7B-Insstnen In-
Task
K, 0.5
K,=0.
0.5
K, 0.7
Dense
=.5
V, =.7
Dense
=.5
V. = 0.7
Average
43.19
42.65
40.96
42.65
42.30
40.95
SingleDoc QA
36,66
36.67
35.28
36.21
36.22
36.08
MultiDoc QA
36.09
36.23
35.11
29.93
30.42
29.40
S Summarization
26.75
26.05
23.57
28.
27.77
26.72
Few-shot
68.96
68.18
66.10
66.68
66.70
66.24
Synthetic
37.25
36.00
34.13
41.92
36.13
Code
55.58
54.50
53.49
54,98
$4.83
53.84
3 Sparse Attention Kernel
Our findings establish that unstructured sparsity offers superior sparsity ratios over structured sparsity while preserving accuracy. In turn, a crucial contribution of Mustafar is to leverage this advantage to enable high compression efficiency while minimizing the latency overhead of runtime pruning and compression. Prior compression methods such as quantization, structured pruning, and token eviction reduce matrix dimensions or element bitwidths. In terms of efficiency, speedup from the reduced size of dense matrix operands compensates for the additional latency introduced by compression (i.e. pruning score computation, quantization). contrast, unstructured sparsity with no regular reduction in dimensions or element bitwidth demands different approach.
Mustafar is motivated by the observation that attention operations in the autoregressive decode stage, the Query x Key"" and Attention Score x Value computations are batch (different heads) of matrix-vector products (MVs) that are significantly memory-bound on GPUs compared to the prefill stage. To exploit this property, we extend the bitmap-based sparse format of Coruscant (20) as shown in Figure Sa to maximally compress the pruned KV cache. consists of compressed tiles corresponding to the position of non-zeros, and tile offset used address the correct position of each tile's starting non-zero. Pruning and compression are performed on-the-fly. with compression accelerated on GPU with a Triton kernel, and attention is computed directly on the compressed representation with custom CUDA kernel that performs batch SpMV on the bitmap-based sparse format. Memory-bound decode-phase phose attention is accelerated by reducing the data movement from global memory to GPU Streaming Multiprocessors.
64 column of the pruned cache. Per-tile bitmap of 64 bits is used to represent
5

---

Prafil Rilsen
Decade Plaze
Column
OA"
02
Tie Offrre
Amantion
OUT
Spev
NU
Comppesse
0.5
.01
Tie-aa Bitnap (54 per
KV cache
OUT
(b) Mustafar attention kernel formulation Figure 5: Overview of Mustafar sparse attention kernel. In (b), multi-head, softmax, and normalization omitted for simplicity.
(a) Coruscant 20 bitmap based sparse format
Figure 5b and Algorithm| presents the Mustafar sparse attention kernel. KV cache generated in prefill stage is pruned and compressed before the start of decode stage, therefore compatible with prefill FlashAttention [6]. KV cache generated in decode stage is kept as-is (dense) while within the local window, then pruned and compressed afterwards. This entails the attention computations in the decode stage to be reformulated into two parts: SpMV for compressed KV cache (line and 5 of Algorithm T and dense MV for the KV cache within the local window (line and of Algorithm
Algorithm Decode Phase Attention with Dense Local and Compressed KV Caches
Input: Query Q gd; Local KV cache K,, V IdxxNa, where Na is size of local window in tokens; Compressed KV cache Kc, Va E RdxN,, where N, is number of compressed tokens.
Attention Score Computation
1: S, E R Na
Q,Kr
Dense local window attention score Sparse atlention score over compressed KV cuche Full attention score
2: So E BIXN,
Q.Kc
3: S, E (M.((N+NNa)) < softmax(concal(Se.
Output Computation
4: [Sc, Sy] + split(S,;
Partition attention score Final output vector
5: O, E IRd VoST + V.S
Return O
Mustafar SpMV kernel follows the Soad-as-compressed compute-as-dent paradigm adopted by FlashLLM (3). SpInfer (9), and Coruscant (20), which target sparse matrix-dens matrix multiplica- tion in LLM weight projection layers. The compressed KV cache is loaded from GPU global memory into registers in compressed form, decompressed into shared memory, and then used for tile-wise dense computation. We evaluate the performance of the Mustafar attention kernel and quantify the runtime overhead of pruning and compression Section 4.3) We further detail the formulation the SpMV kernel, as well the management of the compressed KV cache in Appendix C
Evaluation
We evaluate Mustafar on two aspects: Accuracy and Efficiency. For accuracy evalua- tion, we use tasks from LongBench (3] to test the accuracy retention of per-token magnitude-base
Methodology:
pruning of KV cache. We evaluate on three models: Llama-2-7B [40], Llama-3-8B-Ing Instruuutt [12]. and Disstrl-7B-IIsttuutt--222 [19]. We also explore the impact of Mustafar when jointly used with orthogonal compression techniques, KV cache quantization KIVI (30) and token-wiss wise eviction H2O [51). For efficiency evaluation, we evaluate the impact on KV cache compression and com- putational latency with Llama-2-7B and Llama-3-8B, Innstruuutt Efficiency evaluation is tested on NVIDIA RTX 6000ADA GPU and measured with NVIDIA Nsight Profiling Tool. Additionally, we provide accuracy evaluation on RULER (17) benchmark in Appendix A.3 accuracy comparison of
Mustafar's unstructured sparsity with 2:4 semi-structured sparsity Appendix and additional
kernel throughput evaluation in Appendix C.3

---

4.1 LongBench Results
Table Ihows the extended LongBench evaluation of Mustafar per-token magnitude- pruning with comparison to dense model and ThinK [44]. Under the same Key cache sparsity, unstructured nature of Mustafar constantly achieves higher accuracy than structured sparsity on ThinK (44) across all tasks. A key advantage of unstructured pruning is its ability to effectively prune the Value cache with minimal accuracy degradation, which structured pruning has struggled to achieve. Even under high sparsity 70% for both the Key and Value caches, unstructured pruning (yellow) consistently outperforms ThinK's Key-only 50% structured pruning (pink) on LLaMA-3 8B and Mistral 7B, and achieves comparable accuracy on LLaMA-2-7B.
Table Mustafar accuracy with Llama and Mistral on LongBench
Singk-Docmmeot& Q4
Multi-Ducummen
Summariration
Feu-shot Leurning
Synthetic
KV Sparsity
Nevoa
Quaper
MF44
Hmspe
Masipue
Gow GowRopuart
MallNows
TREE
S4MSio
PCouns
Fite
RBF
Avg.
Ilama-1100
Demsc
46.39
38.66
29.91
27.77
74.50
42.11
ThinK0.5
22:38
40.96
43:48
44.01
38:37
22.59
26.61
22.20
25.08
24.00
63.81
36.79
6.00
65.00
27.95
31.17
v0.0
23:40
43.68
43:63
46.00
22.72
29:39
22.33
27.64
74.50
90.66
41.09
5.00
68. 50
55.89
42.84
ThinK0
17:58
27.40
30.50
50
19.15
18.13
17.28
17.20
34.00
83.09
4.71
17.88
30:42
Ko
22.91
45.51
18.50
23.16
25.63
21.90
73.00
90.81
68
4.50
51.04
50.99
K0.0 v0.5
23:80
43.14
45:32
46.23
39.42
22.97
29.18
22.70
27.13
74.50
90.50
41.74
5.00
67.50
57.25
54.30
43.04
K0.0 V0.7
42.78
43.92
45.83
23.53
26.92
23.52
25.0 12
24.00
90.35
39.88
5,50
65.3 50
59.18
56.05
42:77
K0.5 V0.5
23:40
46.83
42.98
46. 35
19:27
23.13
26.
22.78
27.07
24.00
90.55
39.97
5.00
67.00
55.56
53:46
42:65
KO.7 V8.7
24.10
40.85
40.88
44.91
38.03
27.36
24.02
21.90
24.78
70.50
90.04
37:37
5.25
63.00
54.12
53.56
40.96
Mistral-7BB-muae InS--nstaattt
Dense
26.76
49.36
43.49
27.48
32:05
24.35
27.00
21.00
86.23
42.80
2.89
$6.81
54.07
42.68
ThinK0
24.03
45.42
34.03
32:73
24.65
71.00
2.70
73.67
47.09
K0.5 V0.0
26.38
33.08
49.20
43.90
29.57
18.65
32.47
24.21
27.05
71.00
$6.25
42.66
3,00
84.23
55.72
54.16
42.56
ThinK0.7
19:25
21.33
36.48
27.96
20.34
14.08
29.32
22.25
25.64
70.50
78.99
29 6m
2.92
54.43
34.28
31.68
33:44
Ko.7 V0.0
27.02
34.37
49.26
43.71
26:37
17.45
32.05
27.43
71.00
$7.19
42.3 3033
4.65
77.24
$4.26
53.06
K0.0 Vo.5
26.29
30.54
49.01
43.99
18.00
19:28
32.07
96
71.00
56
42:79
2.71
$1:77
16
K0.0 V0.7
26.85
31.66
49.24
44.15
27.40
15.35
30.5%
23,80
25.63
71.00
56.81
42.02
3.77
76.33
55.58
54.16
41.77
K0.5 V0.5
26.90
32.99
48.76
43.90
28.90
18,45
32.24
24.09
25.99
71.00
$6.65
42.41
3,20
30.64
55.51
54.15
42.30
K0.7 V0.5
27.11
$0.23
48.90
43.83
27.12
17.43
29.38
23.99
25.79
71.00
$6.59
41.14
4.58
67.57
54.85
52.82
$0.95
1.aama-2 7B
Dense
15.04
9:66
21.38
7.89
9.95
3.65
17:26
21.29
3.5
66.00
87.72
41.66
1.70
6.64
66.66
59.52
27.51
ThinK0.5
15:57
9:95
23:31
6.50
9.60
2.77
1.34
20.15
0.38
65.00
85.53
41.48
2.04
2.79
64.77
25.09
K0.5
14.79
9:65
21:67
7.48
10.00 10
17:34
20.84
3.64
65.00
K7.73
41.26
1.38
6.42
65.15
21.46
ThinK0.
13.76
4.5%
6.24
12.96
14.88
66.00
26.05
1.77
6.91
40.5
21.57
K0.7 V0.0
14.37
3.15
20.55
6.64
9.95
3.28
13.80
20.25
0.88
65.00
$6.64
38.32
2.32
4.04
61.36
58.59
26.17
K0.0 V0.5
15.71
10.02
21.12
7.38
9.64
3.75
18.86
21.37
2.38
65.00
87.72
41.04
1.55
6.75
65.79
60.09
27.40
K00 V0.5
15:57
20.97
7.33
10.1] 14
15:40
20.77
188
65.00 00
87.72
40.69
1.40
6.50
27.05
K0.5 Vo.5
15:49
9:17
20.97
7.51
10:04
178
15 15.46
21.02
3.36
66.00
73
40.8
1.22
5.85
68.78
27.23
K0.7 V0.7
13.76
7,83
19.27
0.57
10.26
3.51
8.70
20.04
0.47
64.50
$6.89
36.37
1.54
3.62
63.95
36.75
25.26
4.2 Joint Application with Orthogonal KV Cache Compression Techniques
Mustafar's per-token pruning enables seamless integration with orthogonal KV cache compression techniques. We evaluate its effectiveness when combined with token eviction from H2O (51) and KV cache quantization from KIVI (30), using representative subset of LongBench tasks from each category. H2O application is conducted with Llama-2 7B and KIVI application is conducted with LLaMA-3-8B-Instruee
4.2.1 Joint Application with Token Eviction
H2O [51] retains fixed budget of recent tokens and critical heavy-hitter tokens. Applying Mustafar to H2O, we retain the same scheme of pruning the KV cache of tokens that exit the local dense window. We configure 10% of KV cache budget each to recent tokens and heavy-hitter tokens. Jointly applied, all heavy-hitter tokens and part of recent tokens is kept as pruned and compressed. In Table 55) we validate the efficacy of Mustafar's 's accuracy retention when jointly applied with token eviction, as we see that 50% sparsity in both KV cache retains the dense accuracy with some degradation when pruned to 70% sparsity.
7

---

Table LongBench evaluation of Mustafar- Harffar--H22 joint application on Llama-2-7B
Single-Doe QA
Multi-Doc QA HotpotQA
Summarization GovReport
Few-shot Learning
Synthetic Peount
Code
NirVAA
TREC
Lec
cache
7.69
17:26
[120
20%
KV Budget
Dense
8.35
7.76
64.00
64.20
K0.5 V0.0
11.83
8.47
7.65
64.00
2.08
64.72
KO.7 V0.0
11.39
8.46
6.34
64,00
1.69
6..92
K0.0
64.00
64.83
K0.0 VO.7
12.39
7.79
5.81
64.00
0.76
64.88
K0.5 V0.5
12.07
8.16
7.61
64,00
2.05
65.15
K0.7
12.20
5.22
64.00
63.73
4.2.2 Joint Application with Quantization
KIVI 30) applies per-channel quantization of Key cache and per-token quantization of Value cache. Following findings of Harma et al. [14), first prune each token's KV cache before quantization is performed. However, we note that current Mustafar sparse attention kernel implementation does performed on sparse
not support low-bit precision. Therefore, the accuracy measurement quantized KV cache. 6 the performance of Mustafar and KIVI applied together. Similar to joint application with H2O, Key cache, Value cache, as well as both Key and Value caches. We observe a decrease in accuracy at 70% pruning, with Summarization task seeing the most significant drop. However, other tasks, such as Single-Document QA maintain the same performance as naive 16-bit model, suggesting the potential for applying varying degrees of compression tailored to specific tasks.
that model accuracy is retained across the tasks for 50%
Table 6: LongBench evaluation of Mustafar- KIVI joint application on Llama-3-a Maama-3-BB-Inttuute -n-tuutt
Single-Doc QA NtrvQA
Multi-Doc QA HolpotQA
Summarization GovReport
Few-shot Learning
Synthetic
Code
Lec
Naive 16-bit
23.39
46.39
29.91
74.50
4.50
57.11
KIV1
Dease
23.60
46.39
29.84
74.50
5.00
57.35
KO.5 Vo.0
23.46
46.21
28.90
74.50
5.50
56.05
K0.0 Vo.5
23.68
46.39
29. 10
74.50
5.50
58.30
K0.0 VO.7
24.10
45.66
27.21
74,00
5.50
59.30
KO.5 Vo.5
23.22
46.06
28.18
74.00
6.00
56.04
KO.7 VO.7
23.74
45,50
23.57
70.50
6.25
54.12
KIVI 2-bit
Dense
23.33
45.47
29. 89
74.50
6.50
50.38
KO.5 V0.0
22.86
74,00
49.92
K0.7 V0.0
22,88
44.60
26.91
73.00
4.50
43.84
K0.0 V0.5
23.65
45.67
29.05
74,00
5.50
51.94
KO.5 V0.5
22.46
45.47
28.61
74.00
4.50
48.76
KO.7 VO.7
22.72
45.18
23,84
71.00
5.12
45.68
4.3 Efficiency Evaluation
crucial aspect of Mustafar is to ensure that the exploitation of sparsity for compressing the KV cache does not deter the inference latency. Mustafar compensates the overhead of runtime pruning and compression by achieving speedup in the memory-bound SpMV. Figure (6a compares the normalized lateney of dense batched MV of cuBLAS with the components Mustafar sparse attention kernel (Figure 5b): batched SpMV, dense batched MV of local window, runtime pruning, and compression, for input sequence length 2048 for Llama-2 and 4096 for Llama-3 and generation length 1024. In the multi-head attention of Llama-2-77 pruning introduces 1.84%, compression introduces 6.25%, and MV of local window introduces 0.62% of the cuBLAS execution time dense inference. In both 50% and 70%, the speedup gained from SpMV kernel more than compensates for the introduced overheads. In 50% sparsity, SpMV takes 81.07% of cuBLAS execution time and for 70% sparsity, SpMV takes 61. 11.87% of cuBLAS execution time. In Grouped-Quer Attention of Llama-3-8B, where there is reduced set KV cache, compression and pruning overhead reduce down to 1.47% and 0.47% of cuBLAS execution time respectively.

---

100
No Normatzed Latency (%)
Thinkk-
Mua tafatr--aaa------- Lww.m----ma-----. 3-mtrtnttttttttt Instruct
Muza ta--aa-----ii -al.-a---------P
Average Lverage LangBench Socre
40
sinvetins
20
Inference
CuBLAS m SpMV Local MV Prune II Compress
40%
50%
60%
70%
80%
90%
Compression Ratio
(a) Normalized kernel latency breakdown.
(b) Compression ratio-accuracy comparison of Mustafar and ThinK.
Figure Efficiency evaluation of Mustafar. In (b), compression ratio refers percentage of compressed size compared to dense KV cache.
Figure 6b compares the KV cache compression ratio (% of size in memory compared to dense KV cache) of Mustafar and ThinK along with the LongBench average score achieved with Llama-2-78 and Llama-3-8B Instrunt In this plot, the red arrow points to the optimal direction, where model achieves higher LongBench score while achieving high compression of the KV cache. For ThinK [44] which prune only Key cache, 50% sparsity leads to 75% compression ratio to dense KV cache, and 70% Key cache sparsity leads to 65% compression ratio. the case of Mustafar where both Key and Value Cache can be pruned, KV cache 50% sparsity leads to 65% compression ratio. The reason behind 15% additional memory footprint due to the tile offset overhead as shown in Figure 5a and the multiples-of-8 padding enforced to coalesce memory access in GPU. KV cache 70% sparsity leads to 45% compression ratio, 50% sparsity to either Key or Value cache leads to 83% compression ratio, and single-cache 70% sparsity leads to 72.5% compression ratio. Overall, we see that Mustafar able to achieve better accuracy given the compression ratio, with the compression aacio-cccracy curve closer to the optimal direction than ThinK.
Figure shows the throughput comparison inference with dense models. For Llama-2 7B, we used input sequence length of 2048 and generated 2048 tokens. For Llama-3 8B, we use input sequence length of 4096 and generated 4096 tokens. For dense baseline, FlashAttention (6) was used on prefill and decode phase. Overall, we see that Mustafar is able to achieve higher throughput support larger batch size owing to the reduced memory footprint KV cache. Llama-3, we see that enabling batch size of 8 leads to 2.23x tokens/sec throughput compared the dense inference of
well as
350
350
Throughout (Tokens/Seconds)
300
Throughput (Tokens/Seconds
300
250
250
200
200
150
150
100
100
50
2
2
Batch Size
Batch Size
Dense
Mustafar-KV-50%
Mustafar-KV-70%
Dense
Mustafar-KV-50f
Musstafar-K-779
(a) Llama-2 Throughput.
Llama-3 8B Instruct Throughput.
Figure 7: Throughput comparison of Mustafar to dense inference.

---

batch size 6. Even within the same batch size, we see an increased throughput upto 1.89x. This is due to the pruning and compression overhead amortized by the speedup of Mustafar sparse attention kernel, leading to faster inference latency. However in batch size 1, we see that throughput is lower than dense inference. This due to the underutilization of GPU in Mustafar sparse attention kernel with small batch size, where the number of threadblocks is smaller than the number of SMs. We provide additional throughput comparison with different input:output token ratios on Appendix C.3
Related Work
KV cache compression Alongside aforementioned work in KV cache pruning 44 31 quantiza-
tion 30
15
48
52). token-wise eviction [51]
29
25
8
and low-rank approximation [47]
37
50
26 KV cache offloading (24 28
B evicts KV cache CPU memory and speculatively
prefetchs critical tokens' KV cache. Layer-centric compression [49 27 applies different level of
compression different layers, adhering layer-wise importance. Head-level compression [TO] 39]
applies different level of compression to each heads, from the observation that not all heads contribute equally. Phase-specific compression [42] applies different strategy for prefill and decode phase, with information retention prioritized in prefill and heavy hitter selection applied on decode phase.
System/Kernel for Attention While Mustafar attention kernel focuses operating directly on the comppe-e sparse KV cache, there exists various contributions from the system and kernel-levels to optimize for attention. PagedAttention 23 introduces paging-inspinn inspired attention algorithm that partitions KV cache into memory blocks to reduce memory fragmentation and efficient sharing across sequences. FlashDecoding [6) introduces double-buffering to accelerate memory- bound GeMM of decode phase. FlashInfer (45) unifies KV cache format using a block-sparse representation for an efficient management of KV cache that leads to increased throughput. Loki (36) uses sparse attention method that leverages the low-dimensionality of key vectors to perform an approximate attention reduced PCA space.
Conclusion and Limitations
In this work, we demonstrate that unstructured sparsity presents a powerful and novel solution for KV cache pruning- By removing constraints on the pruning pattern, we show that per-token magnitude- based pruning achieves high sparsity while maintaining model accuracy. To unlock the practical benefits of unstructured sparsity, we introduce a bitmap-based sparse format and custom attention kernel that directly operates compressed KV cache. Together, our pruning strategy, sparse format, and custom kernel form an end-to-end system that substantially reduces KV cache memory usage and improves throughput, making it possible support longer contexts and more efficient inference. Mustafar establishes a foundation for future efforts to integrate unstructured sparsity into practical LLM deployment pipelines and opens directions for memory- -fffeiiett LLM inference at scale. In future work, we plan to explore the joint effect of leveraging KV sparsity of Mustafar with sparsity in weights derived by works such output-aware weight pruning (38) , pruning with low-rank adapters for accuracy retention [34 33). and activation-aware calibration and efficiency enhancement [32] 46). Additionally, this paper focuses on showing that unstructure sparsity can prune both Key and Value caches to a higher sparsity with better accuracy than structured sparsity, leaving our method's 's ability to map arbitrary sparsity degree untouched. While we explore higher sparsity uniformly applied to the entire KV cache in Appendix A.4, a future work involves deriving the optimal target sparsity to a smaller granularity (e.g. per-head or par-layer) maximize sparsity and accuracy retention.
Acknowledgments and Disclosure of Funding
We gratefully acknowledge the support of National Science Foundation (NSF) under program PPoSS, Award Number 2316177.
10

---

References
[1] Muhammad Adnan et al. "Keyformer: KV Cache reduction through key tokens selection for Ef- ficient Generative Inference". In: Proceedings of Machine Learning and Systems. Ed. by P. Gib- bons, G. Pekhimenko, and C. De Sa. Vol. 6. 2024, pp. 114-127. httpp///////pppppprr/rreiiiiaa/a
mlsys. hrgg:s..rrrroo www.orgg/aaaaee llapppprfilleaee Paper- Sopprr-ooffeeence.. pdf
[2] Joshua Ainslic et al. "GQA: Training Generalized Multi-Query Transformer Models from Multi-Head Checkpoints". In: The 2023 Conference on Empirical Methods in Natural Lan-
guage Processing. 2023. URL: htttts:///ppprrreee
[3] Yushi Bai et al. "LongBench: A Bilingual, Multitask Benchmark for Long Context Under- standing". In: Proceedings of the 62nd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers). Ed. by Lun-Wei Ku, Andre Martins, and Vivek Srikumar. Bangkok, Thailand: Association for Computational Linguistics, Aug. 2024, pp. 3113-3137. URL: httpss://ccaaaait haap://:aaaaatthloogooo 202/2024
DO1: 10. 186538633/11 / / acl acn-long. 172
ac1 ac1-long.11 172/
[4] Chi-Chih Chang et al. "Palu: KV-Cache Compression with Low-Rank Projection". In: The Thirteenth International Conference on Learning Representations. 2025. URL: https:
openreview.oo net net/forte
[5]
Zhuoming Chen et al. "MagicPIG: LSH Sampling for Efficient LLM Generation". In: Proceed- ings the 13th International Conference on Learning Representations (ICLR). 2025. URL:
https://arxiv> org/ /b/////////// aab///1bb/200/.///////////////// 16179
[6] Tri Dao. "FlashAAttention222 2: Faster Attention with Better Parallelism and Work Partitioning". In: Proceedings of the 12th International Conference on Learning Representations. 2024.
[7] DeepSeek- AI et al. DeepSeek- RI: Incentivizing Reasoning Capability in LLMs via Reinforce-
ment Learning. 2025. URT:: ht https: /tgp://///ooooo/////// 48550/arXiv. 2501. 12948
[8] Harry Dong et al. "Get more with LESS: synthesizing recurrence with KV cache compression for efficient LLM inference". In: Proceedings of the st International Conference on Machine Learning. ICML'24. 24. Vienna, Austria: JoLR.org. 2024.
[9] Ruibo Fan et al. "SpInfer: Leveraging Low-Level Sparsity for Efficient Large Language Model Inference on GPUs". In: Proceedings of the Twentieth European Conference on Computer Systems (EuroSys '25). New York, NY, USA: Association for Computing Machinery. 2025, pp. 243 260. DOr: 10. 1145/ 3055/31681111..1 3717481
URL: https: org/ 10. 1145
3689031.
[10] Yu Fu al. "Not All Heads Matter: A Head-Level KV Cache Compression Method with Integrated Retrieval and Reasoning". In: The Thirteenth International Conference on Learning Representations. 2025. URL: htttps://ppenrevieett wttpriiwwwwweweeeee htte///oommmii-_JFeeeee.s
[11] Suyu Ge et al. "Model Tells You What to Discard: Adaptive KV Cache Compression for LLMs". In: Proceedings of the 2024 Conference on Empirical Methods in Natural Language Processing (EMNLP). 2024.
[12] Aaron Grattafiori et al. The Llama 3 Herd of Models. 2024. URL: htl::htps::/ //ax:iv.///x//////// abs/
2407.21783
[13] Jitai Hao al. "OmniKV: Dynamic Context Selection for Efficient Long-Context LLMs". In: Proceedings of the 13th International Conference on Learning Representations (ICLR). 2025.
[14] Simla Burcu Harma et al. "Effective Interplay between Sparsity and Quantization: From Theory to Practice". In: The Thirteenth International Conference on Learning Representations. 2025. URL: https: http:::///peereviine
net www.ormmiddwwwwwaa......
[15] Yefei He et al. "ZipCache: Accurate and Efficient KV Cache Quantization with Salient Token Identification". " In: The Thirty-eighth Annual Conference on Neural Information Processing
Systems. 2024. URL: /tttps::///ppt htttps://openrevierr
[16] Ke Hong et al. "Flass Faster Large I Language Model Inference with Asynchro- nization, Flat GEMM Optimization, and Heuristics". In: Proceedings of Machine Learning and Systems. Ed. by P. Gibbons, G. Pekhimenko, and C. De Sa. Vol. 6. 2024, pp. 148- 161. URL: https: : pwo.eedigs..eeeeeeeoeoe.o. mlsys org / paper_ files/ / /pppppaapp//aa// 2024 file/ 5321biddaccibeee1181146622222
eroncerence.o pdf
[17] Cheng- Hsieh et al. "RULER: What's the Real Context Size of Your Long-Context Language Models?" In: Proceedings of COLM 2024. 2024.
11

---

[18]
Fantine Huot et al. "Agents' Room: Narrative Generation through Mulp Collaboration"
In: The Thirteenth International Conference on Learning Representattins. 2025. URL: https
// /popenevvew
[19]
Albert Q. Jiang et al. "Mistral 7B". In: arXiv preprint arXiv:2310.068825. 20233 URL: https:
// //arxiv.oa org/ abs 03622
[20]
Donghyeon Joo et al. "Coruscant: Co-Designing GPU Kernel and Sparse Tensor Core to Advocate Unstructured Sparsity in Efficient LLM Inference". In: Proceedings of the 58th
IEEE/ACM International Symposium on Microurchitecture. 2025. ISBN: 0798400775730, DOI:
10. 10.1145/3725844 3756065 URL: httt::tttt//ttt http::///oooooo//////o 1145/3725844 3756065
[21] Norman P. Jouppi et al. "TPU v4: An Optically Reconfigurable Supercomputer for Machine Learning with Hardware Support for Embeddinggs.. In: Proceedings of the 50th Annual International Symposium on Computer Architecture (ISCA). 2023. U'RL: https: //axxvv org/abs/2300 01433
[22] Yekyung Kim et al. "FABLES: Evaluating faithfulness and content selection in book- length summarization". In: First Conference on Language Modeling. 2024. URL: https //openreview net Sierum??dd=EHxxSoaii
[23] Woosuk Kwon et al. "Efficient Memory Management for Large Language Model Serving with In: Proceedings of the 29th Symposium Operating Systems Principles. SOSP 23. Koblenz, Germany: Association for Computing Machinery, 2023, 61 616-626. ISBN: 9798400702297. DOI: 10. 1145/36600066 3613165
URL: httts:://dot . org/ 10. 1145/
3600006 3613165
[24] Wonbeom Lee et al. "InfiniGen: Efficient Generative Inference of Large Language Models with Dynamic KV Cache Management". In: 18th USENIX Symposium on Operating Systems Design and Implementation (OSDI 24). Santa Clara, CA: USENIX Association, July 2024,
pp. 152-172. ISBN: 978-1-939133-44-33 URL: https: // /www. usenix. org/ conferenc
[25] Yuhong Li et al. "SnapKVV LLM Knows What You are Looking for Before Generation". In: The Thirty-eighth Annual Conference on Neural Information Processing Systems. 2024, URL:
https://openrrevie: net/ peet/formmiddiie idrommidd=ooE5400g22
[26] Bokai Lin et al. "Matryoshkak Adaptive KV Compression via Trainable Orthogonal Projec- tion". In: The Thirteenth International Conference on Learning Representations, 2025, URL:
https://openrew:
[27] Akide Liu et al. "MiniCache: KV Cache Compression Depth Dimension for Large Language Models". In: The Thirty-eighth Annual Conference on Neural Information Processing Systems.
2024. URL.: httttt:://pppeeeiiiiiiiii htt.://ieeeieeoooommmoooeoeoooie 5
[28] Di Liu et al. RetrievalAttiention:: Accelerating Long-Conteds Context L.M Inference via Vector Retrieval.
2024. URL: httpss .rg/ abs/ 2409. 10516
[29] Zichang Liu al. "Seissorhands: Exploiting the Persistence of Importance Hypothesis for LLM
KV Cache Compression at Test Time". In: Seirthy Conference on Neural Information Processing Systems. 2023. URL:: fttt:::////// ptteps://ppenrrviie http:w/eeefeeee www//feuummm -d= J2
[30] Zirui Liu et al. "KIVI: A Tuming-Fiie Free Asymmetric 2bit Quantization for KV Cache".. In: Forty- first International Conference Machine Learning. 2024. URL: https: : / openreview 2d07
[31] Bo Lv et al. "KVPru Structural Pruning for Faster and Memory-Efffciint Large Language IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP). 2025, pp. 1-5. DOI: 10. 1109/CCASSP4966) 2025. 10889000
Models", In: ICASSP 2025 2025
[32] Iman Mirzadch et al. "ReLU Strikes Back: Exploiting Activation Sparsity in Large Language Models". In: Proceedings of the Twelfth International Conference on Learning Representations. 2024. URL: https: pestps://arze //axxi///xxivv/// org/påf 2310. 04564
[33] Mohammad Mozaffari, Amir Yazdanbakhsh, and Maryam Mehri Dehnavi. "SLiM: One-shot Quantization and Sparsity with Low-rank Approximation for LLM Weight Compression". In: Proceedings of the 42nd International Conference on Machine Learning (ICML 2025). 2025.
URL: https: : arxiv.v htgg/abs/200 09615
[34] Mohammad Mozaffari et al. "SLoPe: Couble-Prunne Sparse Plus Lazy Low-Rank Adapter Pretraining of LLMs". In: Proceedings of the International Conference on Learning Represen-
tations (ICLR 2025). 2025. URL: https: http///axvvvv/..///a//eee/.a.vvv .wr.vi.oorg/aaoooo abs/ 2405.16325.5
12

---

[35] Md Aamir Raihan, Negar Goli, and Tor Aamodt. Modeling Deep Learning Accelerator Enabled GPUs. 2019.
[36]
Prajwal Singhania et al. "Loki: Low-rank Keys for Efficient Sparse Attention". In: The Thirty-
eighth Annual Conference Neural Information Processing Systems. 2024. URL: https
// //openrevier
[37]
Hanshi Sun et al. "ShadowKV: KV Cache in Shadows for High-Throughpe Long- Cong-Context LLM Inference" . In: Proceedings of the Forty-Second International Conference on Machine Learning. 2025, URL: ptuttts://arx//e org/ wttpg////////////////// ab/////s//11. 21465
[38]
Mingjie Sun et al. "A Simple and Effective Pruning Approach for Large Language Models". In: Proceedings of the 12th International Conference on Learning Representations. 2024.
[39]
Hanlin Tang et al. "RazorAttention: Efficient KV Cache Compression Through Retrieval Heads". In: arXiv preprint arXiv:2407. 15891. 2024. URL: https:////// wttp///////b////
15891
[40]
Hugo Touvron et al. Llama 2: Open Foundation and Fine-Tuned Chat Models. 2023. URL:
petps://arxias org/ aa////////// abs/2307 2307 09288
[41] Ashish Vaswani et al. "Attention Is All You Need". In: Proceedings of the 31st International Conference on Neural Information Processing Systems (NeurIPS). 2017. URL: https:
00g/10.44 4/5//00..855//aaa/ http//aavvvvvvv.. 03762
[42] Jialong Wu et al. "SCOPE: Optimizing Key-Value Cache Compression in Long-context Generation", In: Proceedings of the 63rd Annual Meeting of the Association for Computational
Linguistics. 2025. URL: http::::///aaaahhhlloooooo
or g 2025. acl -ong. 529, 629.pff
[43] Haojun Xia al. "Flash- Enabling Cost-Effectivv and Highly- -ffficieff Large Generative Model Inference with Unstructured Sparsity". In: vol. 17. 2. VLDB Endowment, DOI: 10 14778/3626292.36222 3626333 URL: https: htt.:/doiiiaaa org/ 107114777/388 (1.333222.22666662
[44] Yuhui Xu al. "ThinK: Thinner Key Cache by Query-Drived Driven Pruning". In: The Thirteenth Internationaa Conference on Learning Representations. 2025. URL: http::s///pppereeee
net/forum wwwboorrmmiiiiiie iddn000G1iin
[45] Zihao et al. FlashInfer: Efficient and Customizable Attention Engine for LLM Inference
Serving. 2025.
[46] Ruokai Yin et al. DuoGPT: Training-free Dual Sparsity through Activation-awware Pruning in LLMs. 2025. Xiv: 2506. 20194 [cs LG]] URL: (tRL://tts:://rrt/t org/ abs/ 20194
[47] Zhihang Yuan et al. ASVD: Activation-Awar Singular Value Decomposition for Compressing Large Language Models. arXiv preprint arXiv:2312.0582
DOL: 10. 48550/8500000 arriv..iivvv...v. 2312
05821
URL.: https://doi org/ 10. 48550 -8550/arXiv 05821
[48] Tianyi Zhang et al. "KV Cache is 1 Bit Per Channel: Efficient Large Language Model Inference with Coupled Quantization". In: The Thirty-eighth Anmual Conference on Neural Information Processing Systems. 2024. URL: httts:://ppeneeeee
net fort/forummiddd pww.:idsdddpvvvv555
[49] Yanqi Zhang et al. Unifying KV Cache Compression for Large Language Models with LeanKV.
2024. URL: https: http://oo...gggg////
org/10. 10. 400.88550/rrra arriv. 2412. 03131
[50] Yifan Zhang et al. Tensor Product Atlention Is All You Need. 2025. URL: https://arxiv
org/abs/ 06425
[51] Zhenyu Zhang et al. "H20: Heavy-Hitter Oracle for Efficient Generative Inference of Large
Language Models". In: Thirtt-seventh Conference Neural Information Processing Systems.
2023. URL: http::ttt//// /ttttpp:///peeeeeiee..... www.://rwtteeotumueeeeemm Pd-
[52] Zhenyu Zhang et al. "Q-Hitter: A Better Token Oracle for Efficient LLM Inference via Sparse-Quantized KV Cache". In: MLSSs. 2024. URL: http:tttttss//proceeeiiii
paper httpa//ppppp..........oooooootta ott.aaaccce...tt.t.oo.e.eeeeaaaa Conference html
[53] Wenting Zhao et al. "Commit0: Library Generation from Scratch". In: The Thirteenth Inter-
national Conference Learning Representations. 2025. URL.: https://openrevie..nt
forum? Barum?id-MaaaQEVsagg
13

---

A Extended Evaluation
A.1 Section Methodology Applied LLaMA-2 7B
We follow the same methodology of exploring pruning direction and output-awarenes. on Llama-2-7B further solidify our findings on model architecture with Multi-Head Attention. In Table 7 observe similar trend to that Llama-3-8B-Instmu in Section Unstructured pruning outperforms structured pruning of ThinK 444.. with ouput-awarenes bringing small accuracy increase pure magnitude-basee pruning.
Table 7: Comparison of ThinK pruning, and per-token output-aware unstructured pruning on LongBench with Llama-2-78 Key cache.
44 structured pruning, per-token magnitude-based unstructured
K, 0.5
K, 0.7
Task
Dense
ThinK (Structured)
Unstructured Output-awarc
Unstructured Magnitude
Think
Unstructured Output-aware
Unstructured Magnitudo
(Structured)
Average
27.51
25.70
27.55
27.46
21.57
26.78
26.17
SingleDoc QA
15.53
16.28
15.52
15.37
14.17
15.82
14.43
MultiDue QA
6.30
6.90
7.23
4.33
6.62
Summarization
14.02
7,46
14.51
13.91
9.28
12.99
11.64
Few-shot
65.13
64.34
65,20
65.00
57.81
63.77
63.65
Synteticc
4.17
2,42
3.98
3.90
4.35
3.00
3.08
Code
63.24
61.57
63.22
63.52
39.85
62.67
61.73
In Table unique phenomenon is the stark contrast of model accuracy in per-channel unstructured pruning methods. Whereas per-channel magnitude-base based pruning of Table [ show good model accuracy retention for -lama-33-B-IIssteee severe. Nevertheless, concurrent to our previous finding, we once again see that per-channel pruning achieves the same level of accuracy retention per-token pruning output-awareness applied. This highlights the importance of output-awarenes: in Value cache pruning. In Table ( we see that the model accuracy of 70% unstructured sparsity on both Key and Value cache achieves similar accuracy to 50% ThinK pruning.
for Llama-2-7B see that accuracy degradation is very
Table 8: Comparison of ThinK 44 structured pruning, per-channel magnitude-based unstructured pruning, per-channel aatrut-aware unstructured pruning, and per-token magnitude- based pruning on LongBench [ with Llama-2-11 2-7B Value cache.
-
0.7
Task
Dense
Think (Stmetured)
Magniude (Per-channe))
Outpul-aware (Per-chunnel)
Magnitude (Perstoken)
ThinK (Structured)
Magnitudc (Por-chunnel)
Oulput-aware (Por-channcl)
Magniude (Perstoken)
Averuge
27.31
24.39
6.16
27.33
27.39
21.10
5.81
26.30
27,05
SingleDoe QA
15.53
12.64
1.68
15.96
15.62
10.05
1.60
15.48
15.17
MultiDee
7,10
Summarication
14.02
9,18
4.51
13.98
13.54
9.10
3.15
13.06
12.67
Few shot
65.13
61.82
9.93
64.07
64.92
57.12
8.83
60.09
64.80
Synthetic
4.17
3.86
1.82
4.45
4.20
1.65
2.45
4.69
3,95
Code
Table 9: Longbench evaluation of Llama-2 7B with KV cache per-token magnitude- pruning
Ilama
Task
K. =.3
K, = 0.7
Dense
=.5
= = 0.7
Average
27.51
SingleDoc QA
15.53
15.21
13.62
MultiDoe QA
7.10
7.11
6.78
Sunmarization
14.02
13.61
Few-shot
65.13
64.84
62.59
Synthetic
4.17
3.55
2.63
Code
63.24
63.16
60.35
14

---

A.2 Scaling to Larger Model
In Table I0] we include the accuracy evaluation of Mustafar per-token magnitude- pruning on Llama-2- 2-mB-hh [40), validating the effectiveness of Mustafar on model with larger size. While unstructured pruning constantly outperforms structured sparsity, we see that the Key cache of Llama- 2-13B-chat is more susceptible to accuracy degradation at 70% sparsity (yellow). In this case, we leverage the modularity of Mustafar, being able to apply different target sparsity to Key and Value cache to find the best combination, to use 50% sparsity for Key Cache and 70% sparsity for Value cache (pink), thereby reaching the higher overall sparsity while maintaining the model accuracy.
Table 10: Mustafar accuracy with Llama-2- 2-13B-chatt on LongBench
Deeemen Q4
Multi-Ducomen Q4
Summariration
Feu-shot Learning
St Synthetic
Code
NtrQQ
Quager
MF-a
Hoquita
SHILMM
Masiyee
GarRepurt
ONESum
MokiNews
TREE
Trivian
SAMSSon
PComnt
Pice
Sparsity
1aama-24338-CCM
Dons:
18.54
34.09
32.01
16.41
11.40
15.81
24.48
20.25
25.74
67.50
86.90
42.07
3.00
12,00
50.12
50.53
34,12
ThinKi.5
16:95
22.39
37.54
34,90
29.93
14.33
24.49
20.21
24.78
67.50
87.16
40.53
2.55
13.00
45.79
46.23
32,80
K0.5 V0.0
18:46
23.12
37.26
17.16
31.18
15.55
23.90
20.55
25.57
67.50
K7.23
41.99
3,00
11.50
50.35
48.98
33,95
ThinKn?
17.56
19.91
32:37
33.00
27.22
13.99
21.19
19:47
12.04
99.0
86.67
26
1.54
1.87
27.79
29.35
27.16
K0 V0.0
1465
30.97
34.05
34.70
10.99
13.72
13:60
20.01
7.63
61.00
81.91
37.76
1.00
1.00
45.29
33.54
28.03
K000 V0.5
18.75
23.68
17.34
16.83
11.36
15.50
23.97
20.85
25.46
67.50
87.20
41.45
2.50
10,00
49.92
49.37
33.82
V0.5
19.29
22.90
37.65
16.57
31.24
15.35
22.46
20.52
24.25
68.00
87.89
40.55
2.50
8,10
89.35
49.14
33,49
K0. V0.5
19.08
22.86
36.97
37.25
31.38
15.45
23.70
20,65
25.3 39
67.50
87.21
40.59
3,00
10,10
19.39
48.06
33.64
K05 V0.7
18.00
22.57
37.18
35,40
31.55
15.25
22.30
20.43
24.81
68.00
87.23
39.91
2.50
7,70
49.02
47.38
33.26
K07 V0.7
17.86
19.91
32.37
33.00
22
13.90
21.19
19.47
12.04
59.00
86.67
31
1.54
1.87
27.79
29.35
27.16
A.3 Evaluation RULER
For more diverse evaluation, we evaluate 1lama-3.1-88) 3.1-8B-Instruct on RULER 17 benchmark for
context length of 65,536 tokens.
Table 11: Accuracy comparison on RULER benchmark
Noodle Mutikeys
Noutle Maiikey2
Ncedle- MatiQuurry
Nocdle- Nedie-MulliVale Mollivatte
Variable Tracking
Frvq Words Extratt
Nov Novdle-iinkel Single
Noodle Singlez
Q41
QA-2
Sparsity
Method
Llama-3, 3.1-8B-Insstrue
0.979
0.973
Key 50%
0.990
0.995
0.594
0:919
Musafar
1,000
1.000
0.990
0.979
0.995
0.966
0.833
0.573
0.971
0.813
Key 70%
ThinK
0.488
0.490
0.229
0.188
0.646
0.487
0.615
0.510
0.208
0.427
Musaafar
1.000
1.000
0.900
0.969
0.992
0.903
0.833
0.594
0.966
0.823
Vilue 30%
ThinK
1.000
10000
0.990
0.96
0914
0958
0.825
0.573
0.910
0.752
Musafar
1,000
1.000
0.979
0.995
0.995
0.91
0.833
0.604
0.983
0.830
Value
ThinK
0.488
0.948
0.698
0.558
0.677
Miscafar
1.000
0.985
ValaValae 30%
0.828
Musrafar
0.990
0.997
0.862
Valae 70%
ThinK
0.000
0.073
0.000
0.000
0,000
0.000
0.219
0.250
0.000
0.035
Mustafar
1,000
1.000
0.990
0.969
0.995
0.914
0.835
0.583
0.869
0.709
As shown in Table II even in the challenging Needle-in-az H-y--sacc scenarios with multiple keys and queries, Mustafar maintains accuracy comparable to the dense model. It also outperforms the structured pruning baseline ThinK, with particularly notable gains at 70% joint Key-Value sparsity. While structured pruning does perform well in isolated cases, such the Needle-Single tasks for 70% Value sparsity, it exhibits significant accuracy drops in other tasks. contrast, Mustafar's unstructured sparsity consistently preserves accuracy across all tasks. This contrast highlights the versatility unstructured sparsity adapting to diverse task requirements.
15

---

A.4 Higher Sparsity
While the main paper primarily focused on 50% and 70% sparsity both Key and Value Cache, we present the performance of Mustafar per-token magnitude-base pruning of KV cache 80% and 90% sparsity in T2 While we see that Key cache suffers from accuracy degradation in higher sparsity, Value cache, despite the even distribution of element magnitude as in Figure 2b retains some level of the model accuracy even at 90% sparsity on selective tasks. Model accuracy retained for tasks such as 2WikiMultihopOA (pink), while degraded significantly in tasks such as GovReport (yellow).
Table 12: Mustafar accuracy with -lama-3-8B-Insteee on LongBench
Single-Drtument Document Q4
Multi-Ducument Q4
Summuriration
Fow-shot Learning
Synthetic
Code
NivQa
Qaaper
MB 4e
Heqosta
Masique
GerRayor
QNE
Molkhww
TREC
Tritaton
S4MSium
PComml
PRc
Sparsity
Avg.
Liuma
Maama-J-BBB. Austruct
Dans:
23.30
43.38
43.22
46.39
58.66
23.22
29:91
22.55
27.77
74.5
90.28
42.11
4.50
70.00
57.11
54.05
43.19
K08 V0.V
22.67
39.08
59.44
44.95
38.51
21.94
21.75
21.00
23.88
60.00
90.24
56.92
7.50
64.50
49.15
45.70
39.77
K0.9 VI.O
19.90
28.92
35.21
41.56
30.77
15.89
11.78
18.40
14.95
39.50
81.79
29.18
2.75
61.50
40.30
33.46
31,30
K0.0 VI1
24.45
42.54
43.96
45.
34.71
22.45
24.47
21.64
25.09
73,00
98.11
99.03
5.62
64.00
56.39
56.54
42.23
K0.0 VI.9
24.12
37.90
42.53
94.65
34.29
21.99
20.22
21.29
21.61
69.00
90.15
36.04
3.29
62.50
55.87
$3.59
40.19
80.8. V0.8
21.82
36.53
18.61
44.3
56.31
21.33
19:18
20.74
30. 80
59. 50
88.
32.
5.25
64.
51.08
48.20
38.05
K0.9. Va.9
17.47
24.13
30.64
38.63
29.24
17.34
13.50
19.67
15.03
35.
50
75.29
27.39
5.50
63
41.77
34.30
30.52
B Comparison with Semi-structured Sparsity
Between the structured pruning of rows and columns, and unstructured pruning of element, lies the 2:4 semi-structured sparsity where out of 4 consecutive elements are non-zero, enforcing global 50% sparsity. Supported by NVIDIA Sparse Tensor Cores, 2:4 semi-structured sparsity also pursue the same objectives of Mustafar bitmap-based sparse format (Figure (aa). maximal compression and fast computation. In Table 13 we apply 2:4 semi-structured pruning to the per-token magnitude- based scheme. Comparing semi-structured sparsity to Key, Value, and both Key and Value cache to unstructured sparsity of Mustafar, we see that unstructured sparsity constantly outperforms semi- structured pattern of the same sparsity. This emphasizes the impact of fine-grained unstructured sparsity of element-wise pruning in model accuracy retention.
Table 13: Comparison of 2:4 semi-structured and unstructured sparsity with Llama-3-8B. Instruutt LongBench
Singl- Dvcument QA
Mulii Decument QA
Summatizatio
Fow-shit Learning
Santhetic
Code
KV Spaniity
Uroger
MF-e
Masique
MallNews
TREC
Tritant
Pdce
IJama- 1.mm-----Bllsstrrtt
Desie
23.39
43.38
43.22
46.39
38.66
23.22
|
22.56
27.75
74.5
90.25
42.11
4,50
70.00
57.11
$4.05
43.19
(2:4)
VRS (2-)
23.99
43.72
43.9
45.43
59.43
22.76
38.51
22.55
26.65
13.90
90.11
40.92
4.50
68.00
58.35
$5.66
42.94
V0.5 (Usstnctsed)
23390
43.14
43.32
46.26
39.42
22.97
29.18
22.70
27.13
74.50
90.50
41.74
5.00
67.50
57.23
$4.30
43.04
30-977-4 N0.917-4)
32.53
47.64
7.5
67.5
58.07
(Unstoctured)
39.97
16

---

C Sparse Attention Kernel Details
As supplement to Section 3 offer more detail onto the Mustafar sparse attention kernel, which accelerates memory-bound batch SpMV.
C.1 Lomp-essComp,essed, Compute-as-Dent Dense Pipeline
Crucial insight of accelerating SpMV involves reducing the data movement between the GPU global memory and the local memory of each GPU Streaming Multiprocessor. First pro- posed by FLashLLM [43), load-as-compressed. compute-as-dense pipeline as shown in Figure g
Sharcd Memory
smaemate
Pegitte
gmanmvee
gnamóvve
involves sending each matrix tile in the cor- responding compressed form to the SM regis- ters ('gmem2reg' in the figure), decompressing the compressed tile into the dense from to the
Shared
HR. Laaaram
smeenth:
Momoyy
emmen2it
Time
Beration-1
heniioon-2
Innration33
Iberation--
shared memory ('extract'), ), then initializing com- putation on the next pipeline stage (" ('smem2tc') Computation is mapped to tensor core to utilize
Figure 8: Load-as-compressed compute-as- compute pipeline of FlashLLM 43
the high fp16 compute throughput. To map MV, unused N dimensions are padded to zero for compu- tation. Non-zero thread-tile of x in Figure represents repreeens the granularity of non-zeros that warp thread decompresses pipeline stage. Each warp thread decompresses 2 thread-tile per stage using corresponding bitmap to determine the correct position of each non-zero. Effectively, each warp operates 64 64 sized matrix tile at time.
C.2 KV Cache Management
Tile size of 64 x of each warp-tile (pink tiles Figure D. requires the KV cache to be compressed and appended to the existing KV cache in token groups of 64. Due to the dynamic nature of KV cache where new entries are added during generation, a compallcoe management of KV cache update is necessary. That is, (1) column tiling direction of KV cache must be orthogonal to the dimension that is being multiplied with: Key cache multiplied on the channel-dimension. thus column tiling is across token dimension (yellow arrow in Figure Daj. value cache is multiplied on the noken-dimension, thus column-tiling across the channel dimension (yellow arrow Figure 9h).
Key
64
Channel
Values
Token
64
64
Incoming takens
Token
Channel
Incoming 64 tokens
tile
Intra-tile direction
Inter-tile direction
tile
Intra-tite direction
Inter-tile direction
(a) Tile Ordering of Key Cache
(b) Tile Ordering of Value Cache
Figure 9: Tile ordering scheme of Key and Value cache
(2), the layout of warp-tile must ensure that newly compressed tokens' KV cache can appended to the existing compressed KV cache. As newly compressed KV cache are added onto the token
17

---

dimension, traversal across multiple warp- tiles is done along channel -hanoel-major dimension for both Key and and Value caches so that the compressed KV cache the new tokens can be appended at the end.
C.3 Decode Speed Evaluation
Extrapolating on Figure 7 we evaluate Mustafar decoding on various input:ouiput token ratios with batch size 4. For Llama-2-7B, we use input sequence length of 2048. For Llama-3-8B-Instrune use input sequence length of 4096. We use output sequence lengths of 512, 1024, and 2048.
Table 14: Decode speed comparison with dense inference
Decode Speed
Decode Speed (decode 1024)
Decode Speed (decode 2048)
Model
KV Format
TTFT
(decode 512)
Ilamaa2
Dense
1.396 sec
88.685 tokens sec
88.512 tokens / sec 79.185 tokens / sec
Mustafar KO.5 V0.5
2.532 sec
89.452
tokens / sec
89,514 tokens / sec 85.687 tokens / sec
Mustafar K0.7 VO.7
2.249 sec
96.386 tokens / sec
97.436 tokens / sec 95.120 tokens / sec
Llama3
Dense
2.769 sec
61.993
tokens see
61.220 tokens see 59.242 tokens / sec
Mustafar K0.5 V0.5
3.269 sec
78.434
tokens see
83.768 tokens / sec 83,303 tokens / sec
Mustafar K0.7 V0.7
3.151 sec
84.065 tokens / sec
88.293 tokens / sec 89.699 tokens / sec
While Figure 7 measured the token throughput by considering both input and output tokens processed, Table [4 we derived the average decoding speed by measuring the end-to-end duration, and dividing it to the number of tokens generated penalize Mustafar with the overhead of KV cache pruning and compression in both prefill and decode stages.
While time-to-firt foo-ff-stttokenntt is delayed due to the overhead of pruning and compressing the KV cache during the prefill stage, the delay offset by the accelerated attention computation during decoding, resulting in higher overall token generation throughput. Notably, Llama-3 exhibits a larger perfor- mance gain compared to Llama-2, its GQA architecture reduces the overhead of KV cache pruning and compression.
18

---

NeurIPS Paper Checklist
1. Claims
Question: Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?
Answer: [Yes]
Justification:
We give thorough evaluation throughout the paper to support our claims.
Guidelines:
The answer NA means that the abstract and introduction do not include the claims made in the paper.
The abstract and/or introduction should clearly state the claims made, including the contributions made in the paper and important assumptions and limitations. A No or
NA answer to this question will not be perceived well by the reviewers.
The claims made should match theoretical and experimental results, and reflect how much the results can be expected to generalize to other settings.
It is fine to include aspirational goals as motivation long is clear that these goals are not attained by the paper.
2. Limitations
Question: Does the paper discuss the limitations of the work performed by the authors? Answer: [Yes]
Justification: We give the accuracy scores of each method, as well as the extent of the kernel's support and delay in first-time-t--token.
Guidelines:
The answer NA means that the paper has no limitation while the answer No means that the paper has limitations, but those are not discussed in the paper.
The authors are encouraged to create separate "Limitations" section their paper.
The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (e.g., independence assumptions, noiseless settings, model well-specification, asymptotic approximations only holding locally). The authors should reflect how these assumptions might be violated in practice and what the implications would be.
The authors should reflect on the scope of the claims made, e.g., if the approach was only tested on few datasets or with few runs. In general, empirical results often depend on implicit assumptions, which should be articulated.
The authors should reflect the factors that influence the performance of the approach. For example, facial recognition algorithm may perform poorly when image resolution is low images are taken in low lighting. Or speech-to-text system might not be used reliably to provide closed captions for online lectures because it fails to handle technical jargon.
The authors should discuss the computational efficiency of the proposed algorithms and how they scale with dataset size.
If applicable, the authors should discuss possible limitations of their approach to address problems of privacy and fairness.
While the authors might fear that complete honesty about limitations might used by reviewers as grounds for rejection, worse outcome might be that reviewers discover limitations that aren't 't acknowledged judgment and recognize that individual actions in favor of transparency play impor- tant role in developing norms that preserve the integrity of the community. Reviewers will be specifically instructed to not penalize honesty concerning limitations.
the paper. The authors should use their best
3. Theory assumptions and proofs
Question: For each theoretical result, does the paper provide the full set of assumptions and complete (and correct) proof?
Answer: [Yes]
19

---

Justification:
We justify all theoretical claims with real measurements.
Guidelines:
The answer NA means that the paper does not include theoretical results.
All the theorems, formulas, and proofs in the paper should be numbered and cross- referenced.
All assumptions should be clearly stated referenced in the statement of any theorems.
The proofs can either appear in the main paper or the supplemental material, but they appear in the supplemental material, the authors are encouraged to provide short proof sketch provide intuition.
Inversely, any informal proof provided in the core of the paper should be complemented by formal proofs provided in appendix or supplemental material.
Theorems and Lemmas that the proof relies upon should be properly referenced.
4. Experimental result reproducibility
Question: Does the paper fully disclose all the information needed to reproduce the main ex- perimental results of the paper to the extent that affects the main claims and/or conclusions of the paper (regardless whether the code and data are provided or not)?
Answer:[Yes Yes]
Justification: Yes, and the source code is available on Github.
Guidelines:
The answer NA means that the paper does not include experiments.
If the paper includes experiments, a No answer to this question will not be perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data are provided or not.
If the contribution is dataset and/or model, the authors should describe the steps taken to make their results reproducible or verifiable.
Depending on the contribution, reproducibility can be accomplished in various ways. For example, if the contribution is a novel architecture, describing the architecture fully might suffice, or if the contribution is a specific model and empirical evaluation, it may be necessary to either make possible for others to replicate the model with the same dataset, or provide access to the model. In general. releasing code and data is often one good way accomplish this, but reproducibility can also be provided via detailed instructions for how to replicate the results, access to a hosted model (e.g.. in the case of large language model), releasing of a model checkpoint, or other means that are appropriate to the research performed.
While NeurIPS does not require releasing code, the conference does require all submis- sions to provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution. For example
(a) If the contribution is primarily a new algorithm, the paper should make clear how to reproduce that algorithm. (b) the contribution is primarily a new model architecture, the paper should describe the architccture clearly and fully.
(c) If the contribution is new model (e.g., a large language model), then there should either way to access this model for reproducing the results or way to reproduce the model (e.g., with an open-source dataset instructions for how to construct the dataset).
(d) We recognize that reproducibility may be tricky in some cases, in which case authors are welcome to describe the particular way they provide for reproducibility. the case of closed-source models, may be that access to the model is limited in some way (e.g., to registered users), but it should be possible for other researchers have some path to reproducing or verifying the results.
5. Open access to data and code
Question: Does the paper provide open access to the data and code, with sufficient instruc- tions to faithfully reproduce the main experimental results, as described in supplemental material?
20

---

Answer: [Yes]
Justification:
Source code is available on Github to reproduce results.
Guidelines:
The answer NA means that paper does not include experiments requiring code.
Please see the NeurIPS code and data submission guidelines (https://nips.cm
for more details.
While we encourage the release of code and data, we understand that this might not be possible, so "No" an acceptable answer. Papers cannot be rejected simply for not including code, unless this is central to the contribution (e.g., for new open-source benchmark).
The instructions should contain the exact command and environment needed to run to reproduce the results. See the NeurIPS code and data submission guidelines (https
/ ccpppp.ccppulli//uuuii wwwiiicgguie///eeeeeeeeic CoodeSubeiiieeee ssi wswonnolli onnolicy)))) for more details.
The authors should provide instructions on data access and preparation, including how access the raw data, preprocessed data, intermediate data, and generated data, etc.
The authors should provide scripts to reproduce all experimental results for the new proposed method and baselines. If only subset of experiments are reproducible, they should state which ones are omitted from the script and why.
At submission time, to preserve anonymity, the authors should release anonymized versions (if applicable).
Providing as much information possible in supplemental material (appended to the paper) recommended, but including URLs to data and code permitted.
6. Experimental setting/details
Question: Does the paper specify all the training and test details (e.g., data splits, hyper- parameters, how they were chosen, type of optimizer, etc.) necessary to understand the results?
Answer: [Yes]
Justification:
We specify the benchmarks, models, and platform used.
Guidelines:
The answer NA means that the paper does not include experiments.
The experimental setting should be presented in the core of the paper to level of detail that is necessary to appreciate the results and make sense of them.
The full details can be provided either with the code, in appendix, or as supplemental material.
7. Experiment statistical significance
Question: Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?
Answer: [No]
Justification: We average our measurements across multiple iterations.
Guidelines:
The answer NA means that the paper does not include experiments.
The authors should answer "Yes" if the results are accompanied by error bars, confi- dence intervals, statistical significance tests, at least for the experiments that support main claims of the paper.
The factors of variability that the error bars are capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, overall run with given experimental conditions).
The method for calculating the error bars should be explained (closed form formula, call a library function, bootstrap, etc.)
The assumptions made should be given (e.g.. Normally distributed errors).
should be clear whether the error bar is the standard deviation the standard error of the mean.
21

---

It is OK to report 1-sigma error bars, but one should state it. The authors should preferably report a 2-sigma error bar than state that they have 96% CI, if the hypothesis of Normality of errors is not verified.
For asymmetric distributions, the authors should be careful not to show in tables or figures symmetric error bars that would yield results that are out range (e.g. negative error rates).
If error bars are reported in tables or plots, The authors should explain in the text how they were calculated and reference the corresponding figures or tables in the text.
8. Experiments compute resources
For each experiment, does the paper provide sufficient information on the com- puter resources (type of compute workers, memory, time of execution) needed to reproduce the experiments?
Question:
Answer: [Yes] We provide information of our compute platform.
Justification:
Guidelines:
The answer NA means that the paper does not include experiments.
The paper should indicate the type of compute workers CPU or GPU, internal cluster, or cloud provider, including relevant memory and storage.
The paper should provide the amount of compute required for each of the individual experimental runs well estimate the total compute.
The paper should disclose whether the full research project required more compute than the experiments reported in the paper (e.g.. preliminary or failed experiments that didn't make into the paper).
9. Code of ethics
Question: Does the research conducted in the paper conform, in every respect, with the NeurIPS Code of Ethics https: httpr//ss.....bb..t./ttcccclllea
Answer:
[Yes]
Justification:
We conform with the NeurIPS Code of Ethies.
Guidelines:
The answer NA means that the authors have not reviewed the NeurIPS Code of Ethics.
If the authors answer No, they should explain the special cireumstances that require deviation from the Code of Ethics.
The authors should make sure to preserve anonymity (e.g., if there is a special consid- eration due to laws or regulations their jurisdiction).
10. Broader impacts
Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?
Answer: [Yes]
Justification:
We promote the democratization LLMs with KV cache compression.
Guidelines:
The answer NA means that there is no societal impact of the work performed.
If the authors answer NA or No. they should explain why their work has no societal impact or why the paper does not address societal impact.
Examples of negative societal impacts include potential malicious or unintended uses (e.g., disinformation, generating fake profiles, surveillance), fairness considerations (e.g., deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations.
The conference expects that many papers will be foundational research and not tied to particular applications, let alone deployments. However, if there is direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that improvement in the quality of generative models could be used to
22

---

generate deepfakes for disinformation. On the other hand, it is not needed to point out that a generic algorithm for optimizing neural networks could enable people train models that generate Deepfakes faster.
The authors should consider possible harms that could arise when the technology is being used as intended and functioning correctly, harms that could arise when the technology is being used as intended but gives incorrect results, and harms following from (intentional or unintentional) misuse of the technology.
there are negative socictal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition attacks, mechanisms for monitoring misuse, mechanisms to monitor how a system learns from feedback over time, improving the efficiency and accessibility of ML).
11. Safeguards
Does the paper describe safeguards that have been put in place for responsible release of data models that have high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)?
Question:
Answer: [NA]
Justification: Our work promotes efficient usage of existing models.
Guidelines:
The answer NA means that the paper poses no such risks.
Released models that have high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere to usage guidelines or restrictions to access the model or implementing safety filters.
Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images.
We recognize that providing effective safeguards challenging, and many papers do not require this, but we encourage authors to take this into account and make best faith effort.
12. Licenses for existing assets
Are the creators original owners of assets (e.g., code, data, models), used in the paper, properly credited and are the license and terms of use explicitly mentioned and properly respected?
Question:
Answer: [Yes]
Justification: All code, data, and model has been properly credited and referenced.
Guidelines:
The answer NA means that the paper does not use existing assets.
The authors should cite the original paper that produced the code package dataset.
The authors should state which version of the asset is used and, if possible, include URL.
The name of the license (e.g., CC-BY 4.0) should be included for each asset.
For scraped data from a particular source (c.g., website), the copyright and terms of service of that source should be provided.
If assets are released, the license, copyright information, and terms of use in the package should be provided. For popular datasets, snperswwihcoode com/ datasets has curated licenses for some datasets. Their licensing guide can help determine the license of dataset.
For existing datasets that are re-packaged, both the original license and the license of the derived asset (if has changed) should be provided.
If this information is not available online, the authors are encouraged to reach out to the asset's is creators.
13. New assets
Question: Are new assets introduced in the paper well documented and is the documentation provided alongside the assets?
23

---

Answer: [Yes]
Justification: Source code of our sparse attention kernel is provided in the supplementary material as well our project Github.
Guidelines:
The answer NA means that the paper does not release new assets.
Researchers should communicate the details of the dataset/code/model as part of their submissions via structured templates. This includes details about training, license, limitations, etc.
The paper should discuss whether and how consent was obtained from people whose asset used.
At submission time, remember to anonymize your assets (if applicable). You can either create an anonymized URL or include an anonymized zip file.
14. Crowdsourcing and research with human subjects
Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given to participants and screenshots, if applicable, as well as details about compensation (if any)?
Answer: [NA]
Justification: Paper does not involve crowdsourcing nor research with human subjects.
Guidelines:
The answer NA means that the paper does not involve crowdsoureing nor research with human subjects.
Including this information in the supplemental material is fine, but if the main contribu- tion of the paper involves human subjects, then as much detail as possible should be included in the main paper.
According the NeurIPS Code of Bthies, workers involved in data collection, curation, or other labor should be paid at least the minimum wage in the country of the data collector.
15. Institutional review board (IRB) approvals or equivalent for rescarch with human subjects
Question: Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or an equivalent approval/review based on the requirements of your country or institution) were obtained?
Answer: [NA]
Justification: Paper does not involve crowdsourcing nor research with human subjects.
Guidelines:
The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.
Depending on the country in which research is conducted, IRB approval (or equivalent) may be required for any human subjects research. If you obtained IRB approval, you should clearly state this in the paper.
We recognize that procedures for this may vary significantly between institutions and locations, and expect authors to adhere the NeurIPS Code of Ethics and the guidelines for their institution.
For initial submissions, do not include any information that would break anonymity (if applicable), such the institution conducting the review.
16. Declaration of LLM usage
Question: Does the paper describe the usage of LLMSs if it is an important, original, or non-standard component of the methods in this research? Note that if the LLM is used only for writing, editing, or formatting purposes and does not impact the core methodology, scientific rigorousness, originality the research, declaration is required.
Answer: [NA]

---

Justification:
LLMs were not used in core methods of this research.
Guidelines:
The answer NA means that the core method development in this research does not involve LLMs as any important, original, or non-standard components.
Please refer to our LLM policy hate:p/s/eeerrppoocccc//o/eeeeee for what should or should not be described.
25


Reviews:
[{"review_id": "uPxq9W6BbG", "reviewer": "Reviewer_CYDN", "summary": "This paper introduces MUSTAFAR, a method for KV cache compression in large language models using unstructured sparsity. The key contributions are (1) demonstrating that unstructured per-token magnitude-based pruning acheive better compression ratios compared with structured pruning. (2) developing a bitmap-based sparse KV sparsity. (3) custom sparse attention kernel.\nThis paper makes contribution based on ThinK with unstructured sparsity and practical system implementation to improve their drawback on V-cache sparsity.", "questions": "See above.", "limitations": "Yes", "rating": 5, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 3, "significance": 4, "originality": 3, "strengths_and_weaknesses": "Strengths and Contributions\n- The biggest contribution of the paper imo is not the algorithm design but demonstrating it is possible to be faster than dense (which sparse method similar to quantization before usually can save memory size but way slower). While the quantization community has made a great process on the usability and practicality of quantization methods, we haven't seen sparsity made similar progress. Therefore, I think this is an important step towards unleashing the potential of unstructured sparsity in general. \n- Also it is important to show how large the potential benefits of unstructured sparse. In fact, I think the authors did a good job proving it's complementary to other structured sparse and quantization methods.\n\nLimitations and Weaknesses\n- Evaluation Benchmark: The evaluation benchmarks is a bit limited (LongBench V1 only for long context evaluation). I think to convince the wider audience the power of unstructured sparse, I think we can evaluate on newer and thorough evals. We recommend using  standard ones like Ruler [1] for million-level benchmark and newer and harder ones like GSM-Infinite [3] for long-context reasoning benchmark.\n- Discussion on dynamic sparse attention: Other than static sparse and quantization methods, there's a line of work in dynamic sparse attention such as [3]. Since it requires storage format change, how does it impact dynamic sparse methods which stores the whole kv cache on GPU or CPU? Can it still be combined?   \n- The paper is built based on the observation of KIVI, and make extensive development on large language model like Llama-2-7B, Llama-3-8B-Instruct, Mistral-7B. If the observation can be applied on recent reasoning models like deepseek[7] or Qwen[8], that would greatly enhance the impact of the paper.\n\n[1] Cheng-Ping Hsieh, Simeng Sun, Samuel Kriman, Shantanu Acharya, Dima Rekesh, Fei Jia, Yang Zhang, Boris Ginsburg. RULER: What's the Real Context Size of Your Long-Context Language Models?\n\n[2] Yang Zhou, Hongyi Liu, Zhuoming Chen, Yuandong Tian, Beidi Chen. GSM-Infinite: How Do Your LLMs Behave over Infinitely Increasing Context Length and Reasoning Complexity?\n\n[3] Zhuoming Chen, Ranajoy Sadhukhan, Zihao Ye, Yang Zhou, Jianyu Zhang, Niklas Nolte, Yuandong Tian, Matthijs Douze, Leon Bottou, Zhihao Jia, Beidi Chen. MagicPIG: LSH Sampling for Efficient LLM Generation.\n\n[4] Liu A, Feng B, Xue B, et al. Deepseek-v3 technical report[J]. arXiv preprint arXiv:2412.19437, 2024.\n\n[5] Yang A, Li A, Yang B, et al. Qwen3 technical report[J]. arXiv preprint arXiv:2505.09388, 2025.\n\nAdditional Feedback:\nTypos\n- Line 206 sparisty->sparsity\n- Figure 5: Overview of Mustafar attention fernel -> Kernel\n- Line 283 FlashDecoding[15] introduces double-buffering do -> to", "was_revised": true, "final_justification": "The author addresses most of my concerns so I decide to keep the original score leaning towards acceptance.", "rebuttal": "We thank the reviewer for the constructive feedback.\n\n> **W1. Evaluation Benchmark: The evaluation benchmarks is a bit limited (LongBench V1 only for long context evaluation). I think to convince the wider audience the power of unstructured sparse, I think we can evaluate on newer and thorough evals. We recommend using standard ones like Ruler [1] for million-level benchmark and newer and harder ones like GSM-Infinite [3] for long-context reasoning benchmark.**\n\nWe had selected LongBench as it covers various use scenarios from question-answering to summarization. Also, version 1 was the primary benchmark of the previous works KIVI and ThinK. Following the recommendation of the reviewer, we evaluated Mustafar on Llama-3.1-8B-Instruct, with Ruler for context length of 65536. \n\n| Sparsity         | Method     | Needle-Single1 | Needle-Single2 | Needle-MultiKey1 | Needle-MultiKey2 | Needle-MultiQuery | Needle-MultiValue | QA-1  | QA-2  | Variable Tracking | Frequent Words Extraction |\n|------------------|------------|----------------|----------------|------------------|------------------|-------------------|-------------------|-------|-------|--------------------|----------------------------|\n| Dense            | —          | 1.00           | 1.00           | 0.990            | 0.979            | 0.990             | 0.979             | 0.844 | 0.594 | 0.973              | 0.851                      |\n| Key 50%  | ThinK  | 1.00  | 1.00  | 0.990 | 0.979   | 0.995  | 0.969   | 0.833 | 0.594 | 0.919 | 0.854  |\n|  | Mustafar   | 1.00  | 1.00  | 0.990  | 0.979 | 0.995  | 0.996  | 0.833 | 0.573 | 0.971  | 0.813  |\n| Key 70%  | ThinK | 0.448  | 0.490  | 0.229 | 0.188 | 0.646  | 0.487  | 0.615 | 0.510 | 0.208  | 0.427  |\n|  | Mustafar   | 1.00 | 1.00 | 0.990  | 0.969 | 0.992 | 0.903  | 0.833 | 0.594 | 0.966 | 0.823 |\n| Value 50%  | ThinK | 1.00 | 1.00 | 0.990  | 0.969  | 0.914   | 0.958  | 0.823 | 0.573 | 0.910   | 0.792    |\n|     | Mustafar   | 1.00   | 1.00  | 0.979    | 0.995   | 0.995   | 0.971    | 0.833 | 0.604 | 0.983   | 0.830   |\n| Value 70%  | ThinK | 0.948   | 0.927  | 0.948  | 0.510   | 0.698 | 0.688  | 0.646 | 0.500 | 0.558 | 0.677   |\n|    | Mustafar   | 1.00  | 1.00  | 1.00 | 0.979    | 0.992 | 0.969  | 0.833 | 0.594 | 0.985  | 0.826  |\n| Key&Value 50%    | ThinK      | 0.958 | 1.00 | 0.948  | 0.854 | 0.828 | 0.956  | 0.740 | 0.531 | 0.742  | 0.823   |\n|  | Mustafar   | 1.00  | 1.00 | 0.990 | 0.979 | 0.997   | 0.997  | 0.833 | 0.573 | 0.862  | 0.809 |\n| Key&Value 70%    | ThinK      | 0.000 | 0.073  | 0.000 | 0.000  | 0.000 | 0.000  | 0.219 | 0.250 | 0.000  | 0.035  |\n|    | Mustafar   | 1.00  | 1.00  | 0.990  | 0.969 | 0.995 | 0.914 | 0.833 | 0.583 | 0.869 | 0.799 |\n\nEven in challenging “Needle in a Haystack” scenarios with multiple keys and queries, Mustafar maintains accuracy comparable to dense LLMs. It also generally outperforms the structured pruning baseline ThinK, with particularly notable gains at 70% joint Key-Value sparsity. While structured pruning does perform well in isolated cases, such as the Needle-Single tasks for 70% Value sparsity, it exhibits significant accuracy drops in other tasks. In contrast, Mustafar’s unstructured sparsity consistently preserves accuracy across all tasks. This contrast highlights the versatility of unstructured sparsity in adapting to diverse task requirements.\n\nWe provide GSM-Infinite evaluation in our response for W3.\n\n> **W2. Discussion on dynamic sparse attention: Other than static sparse and quantization methods, there's a line of work in dynamic sparse attention such as [3]. Since it requires storage format change, how does it impact dynamic sparse methods which stores the whole kv cache on GPU or CPU? Can it still be combined?**\n\nWe first highlight the motivation behind Mustafar: GPU memory is a heavily contended resource due to the storage demands of both model weights and the KV cache. Mustafar addresses this by compressing the KV cache in GPU memory, thereby alleviating pressure on memory capacity. It further bridges the gap between the massively parallel compute capability of GPU microarchitecture and the memory-bound nature of attention computation during decoding. This is achieved by transferring the KV cache in its compressed form from global memory and directly computing on high-throughput Tensor Cores.\n\nIn contrast, MagicPIG leverages the larger and more flexible CPU memory to store the majority of KV cache, minus sink and local tokens. To compensate for the CPU’s limited compute throughput, it reduces the overall attention cost via a sampling-based attention scheme where selective tokens' KV cache is used for computation. \n\nRegarding support for a sampling-based attention scheme on GPU, Mustafar’s sparse format and attention kernel are readily extensible to support such sampling-based attention schemes. This would requires modest modifications to the sparse format and indexing logic within the CUDA attention kernel, enabling 64-element tiles to be created per-token (as opposed to the current across-token tiling shown in Figure 8). Since the number of tiles per token is fixed for a given LLM architecture, the attention kernel can treat this as a static stride and use it to efficiently locate and compute only the relevant tiles under the same pipeline as described in Figure 7. Similarly, this enables managing KV cache in token-granularity to offload to CPU memory.   \n\nFor CPU execution, however, the pipeline in Figure 7 is not directly applicable, as Mustafar depends on the GPU’s addressable shared memory (L1 cache) to stage decompressed tiles. In the absence of such an addressable cache on CPUs, decompression and computation must be fused and performed while operands are in CPU registers. Nevertheless, compression of KV cache will mitigate the bottleneck on CPU memory bandwidth, which is ‘usually 10 − 20% of GPU VRAM bandwidth’ (MagicPIG, Section 4.4).\n\nFinally, for hybrid CPU-GPU setups (e.g., FlexGen[1] and InfiniGen[2]) where the KV cache is stored on the CPU and transferred to the GPU for attention computation, Mustafar still offers advantages: by reducing the memory footprint of the KV cache, it proportionally reduces the data transfer time between CPU and GPU.\n\n> **W3. The paper is built based on the observation of KIVI, and make extensive development on large language model like Llama-2-7B, Llama-3-8B-Instruct, Mistral-7B. If the observation can be applied on recent reasoning models like deepseek[7] or Qwen[8], that would greatly enhance the impact of the paper.`**\n\nWe visualized the KV cache of Deepseek-R1-0528-Qwen3-8B and Deepseek-R1-Distill-Llama-8B (selected in place of the 685B parameter Deepseek-v3 due to resource constraints). Across both models, we observed a similar pattern: the Key cache exhibits an outlier-heavy distribution, whereas the Value cache displays a more uniform distribution.\n\nTo evaluate the effectiveness of Mustafar in reasoning models on reasoning tasks, we conducted experiments on Deepseek-R1-0528-Qwen3-8B using a down-sized GSM-Infinite benchmark. Due to time constraints, we limited our evaluation to the zero-noise setting, with an operation range of 1–50 for Symbolic task and 2–30 for Medium and Hard tasks, yielding 24 to 65 samples per task. We report Area Under the Curve (AUC) as the evaluation metric.\n\n| Sparsity        | Method   | Symbolic | Medium | Hard |\n|-----------------|----------|----------|--------|------|\n| Dense           | -        | 1520     | 625    | 275  |\n| Key&Value 50%   | Mustafar | 1520     | 750    | 375  |\n| Key&Value 70%   | Mustafar | 240      | 187.5  | 50   |\n| Value 70%       | ThinK    | 0        | 62.5   | 100  |\n\nWhile 70% structured pruning of Value cache performs relatively well on the Hard task, it completely fails on Symbolic and struggles on Medium. This pattern mirrors our KV cache visualization, where the Value cache exhibits a relatively uniform distribution, leading to inconsistent accuracy across task types under structured sparsity .\n\nIn contrast, at 50% unstructured sparsity applied to both Key and Value caches, Mustafar retains performance across all GSM-Infinite tasks. While accuracy degrades when sparsity is increased to 70%, Mustafar still maintains some level of accuracy across all task types, unlike structured pruning. These results demonstrate that Mustafar’s benefits extend to reasoning models. Although our current evaluation focuses on unstructured pruning with uniform sparsity, we believe that incorporating component-wise importance-based sparsity, as illustrated in our toy example response to QfM2 (Q1), offers a promising direction to increase accuracy under high sparsity. Nonetheless, the presented results affirm Mustafar's effectiveness in reasoning scenarios.\n\n\n> **Additional Feedback: Typos**\n\nThank you for pointing this out. We will correct the typos in the revised version.\n\nReferences:\n\n[1] Ying Sheng, Lianmin Zheng, Binhang Yuan, Zhuohan Li, Max Ryabinin, Daniel Y. Fu, Zhiqiang Xie, Beidi Chen, Clark Barrett, Joseph E. Gonzalez, Percy Liang, Christopher Ré, Ion Stoica, Ce Zhang. *FlexGen: High-Throughput Generative Inference of Large Language Models with a Single GPU.*\n\n[2] Wonbeom Lee, Jungi Lee, Junghwan Seo, Jaewoong Sim. *InfiniGen: Efficient Generative Inference of Large Language Models with Dynamic KV Cache Management.*"}, {"review_id": "BRV5emLKQX", "reviewer": "Reviewer_uxWr", "summary": "This paper proposes an unstructured-sparsity framework for the KV cache pruning during LLM inference, which is an area that has not been well studied so far. Two key empirical observations are provided: (1) Key cache activations contain prominent channel-wise outliers; (2) Value cache, despite its near-uniform distribution, still tolerates simple per-token magnitude pruning without compromising accuracy. Based on these findings, the proposed KV cache is stored in a bitmap-based sparse format and processed on-the-fly with a custom CUDA SpMV kernel, allowing attention to operate directly on the compressed data. Experiments on LongBench with several open-source LLMs show that unstructured pruning achieves up to 70 % sparsity while matching—or surpassing—the accuracy of a structured-pruning baseline (ThinK), all without any fine-tuning.", "questions": "Please see weaknesses.", "limitations": "N/A", "rating": 5, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 2, "clarity": 3, "significance": 3, "originality": 3, "strengths_and_weaknesses": "Strengths: \n1. The paper is the first to systematically compare unstructured KV cache pruning against structured methods, highlighting its practical superiority in accuracy–compression trade-offs.\n2. The proposed method achieves up to 70% unstructured sparsity without accuracy loss, while simultaneously reducing memory footprint and improving throughput (up to 2.2×) via a custom SpMV kernel. \n\nWeaknesses: \n1. The contribution is timely and practically valuable, but the core novelty remains limited, relying on known pruning heuristics without introducing new algorithmic or theoretical insights.\n2. The scalability of the method is not sufficiently validated, as experiments are limited to small-to-mid scale models (7B/8B), with no evidence that the approach holds for larger models like 13B or 70B.\n3. The writing can be improved further. Some notations in the equations are undefined, and there are several typos throughout the paper—for example, a typo in Line 186. Additionally, the reported average performance of Mistral-7B-Instruct-v0.2 for K0.5 V0.5 (= 54.15) in Table 4 appears to be incorrect.", "was_revised": true, "final_justification": "I support this paper to be accepted. This paper is the first to systematically compare unstructured and structured KV cache pruning, showing the superior accuracy–compression trade-off of unstructured methods. I think the contribution of this paper is timely and practically valuable.", "rebuttal": "We thank the reviewer for the constructive feedback.\n\n> **W1. The contribution is timely and practically valuable, but the core novelty remains limited, relying on known pruning heuristics without introducing new algorithmic or theoretical insights.**\n\nWe appreciate the reviewer’s recognition that the contribution is timely and practically valuable. While it is true that our pruning strategy leverages well-established magnitude-based heuristics, we believe that our key contribution lies in the novel observation, empirical analysis, and system-level integration of unstructured token-wise sparsity for KV cache compression, which is an area that prior work has left largely unexplored due to its execution inefficiency. \n\nSpecifically, our pruning strategy is rooted in critical empirical and theoretical findings:\n(i) Key cache exhibits channel-wise outliers, as originally suggested by KIVI. We extend this insight by demonstrating that per-token unstructured pruning is more effective at capturing these outliers than structured pruning.\n(ii) Value cache, on the other hand, shows no strong structural cues, but we reveal that token-wise magnitude pruning is inherently output-aware due to the attention formulation (Attention Score × Value), where every element contributes proportionally to the output.\n\nFurthermore, our work is the first to demonstrate that unstructured pruning of both Key and Value caches to 70% sparsity is feasible. This insight alone substantially pushes the boundary of what is considered achievable in KV cache sparsity, which had been limited to structured sparsity. Unstructured sparsity is then made practical and efficient with a custom CUDA attention kernel that efficiently operates on a bitmap-based compression format tailored for arbitrary sparsity patterns. We show that the speedup from sparse computation compensates for pruning overhead, enabling faster inference, KV cache compression, and longer context lengths without degrading performance.\n\nFinally, we validate that our pruning method is complementary to orthogonal techniques such as quantization and token eviction (Section 4.2), showing that Mustafar can be jointly applied in practical deployment scenarios. We believe this holistic treatment, from sparsity observations to kernel-level execution, constitutes a substantial novelty, both in terms of theoretical insight and practical system design.\n\n> **W2. The scalability of the method is not sufficiently validated, as experiments are limited to small-to-mid scale models (7B/8B), with no evidence that the approach holds for larger models like 13B or 70B.**\n\nWe provided the evaluation for LLaMA-2-13B-Chat in Appendix A.2. While we were not able to experiment with a 70B model due to resource constraints, as ThinK demonstrated the accuracy preservation of LLaMa-3-70B-Instruct with 40% and 50% sparsity structured pruning of Key cache (ThinK paper, Table 2), we expect our method to generalize similarly.\n\n> **W3. The writing can be improved further. Some notations in the equations are undefined, and there are several typos throughout the paper—for example, a typo in Line 186. Additionally, the reported average performance of Mistral-7B-Instruct-v0.2 for K0.5 V0.5 (= 54.15) in Table 4 appears to be incorrect.**\n\nThank you for pointing this out. We will correct the typos in the revised version, as well as clarify the equations. Also, the correct average entry for Mistral-7B-Instruct-v0.2 for K0.5 V0.5 on Table 4 is 42.30."}, {"review_id": "nIeNpLR2xM", "reviewer": "Reviewer_PxDY", "summary": "This paper introduces the MUSTAFAR method, which optimizes key-value cache pruning in large language model inference through unstructured sparsity to address the issue of excessive memory overhead of KV Cache in long-context scenarios. The authors claim that compared to traditional structured pruning methods, it can maintain model accuracy at higher sparsity rates. Specifically, the Key Cache employs element-magnitude-based per-token pruning combined with an output-aware strategy, while the Value Cache can be efficiently compressed through simple per-token magnitude pruning. To effectively utilize sparsity, the paper designs a bitmap-based sparse representation and customized CUDA attention kernels that directly compute on the pruned KV Cache, significantly reducing memory bandwidth requirements and accelerating attention operations.", "questions": "Please refer to Strengths And Weaknesses for details..", "limitations": "yes", "rating": 4, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 2, "clarity": 3, "significance": 2, "originality": 2, "strengths_and_weaknesses": "### Strengths\n\nS1)This paper presents a practical approach to KV cache compression through unstructured sparsity, achieving significant memory savings while maintaining model accuracy.\n\nS2)The proposed method is intuitive and well-explained, with clear writing that makes the technical content accessible.\n\n### Weaknesses\n\nMy primary concerns regarding this paper involve several missing critical experiments:\n\nW1) The baseline comparison is limited to ThinK, while omitting recent highly relevant methods in KV cache compression such as HeadKV [1]. The reported LongBench results appear inferior to HeadKV's performance, particularly since HeadKV demonstrates effectiveness even when retaining less than 5% of KV cache.\n\nW2) The evaluation lacks standard long-context benchmarks like \"Needle in a Haystack\", which have become prevalent in KV cache compression research [1-3]. LongBench's average sequence length may not adequately assess performance in true long-context scenarios.\n\nW3) For efficiency analysis, direct comparisons of first-token latency and average decoding speed against full cache or methods like HeadKV would provide more straightforward evaluation metrics.\n\nW4) In Section 3 (Sparse Attention Kernel), presenting the computation process through formal equations or algorithms would enhance clarity.\n\n[1]Fu Y, Cai Z, Asi A, et al. Not all heads matter: A head-level kv cache compression method with integrated retrieval and reasoning[J]. ICLR 2025.\n\n[2]Feng Y, Lv J, Cao Y, et al. Ada-kv: Optimizing kv cache eviction by adaptive budget allocation for efficient llm inference[J]. arXiv preprint arXiv:2407.11550, 2024.\n\n[3]Li Y, Huang Y, Yang B, et al. Snapkv: Llm knows what you are looking for before generation[J]. Advances in Neural Information Processing Systems, 2024, 37: 22947-22970.", "was_revised": true, "final_justification": "This paper presents a practical approach to KV cache compression through unstructured sparsity, achieving significant memory savings while maintaining model accuracy. The proposed method increases TTFT, and the model used is a bit outdated. Overall, I believe the paper is worthy of a borderline accept.", "rebuttal": "We thank the reviewer for the constructive feedback.\n\n> **W1) The baseline comparison is limited to ThinK, while omitting recent highly relevant methods in KV cache compression such as HeadKV [1]. The reported LongBench results appear inferior to HeadKV's performance, particularly since HeadKV demonstrates effectiveness even when retaining less than 5% of KV cache.**\n\nWe selected ThinK as our baseline because it is, to the best of our knowledge, the only prior method that performs channel-wise (per-token) pruning. This aligns with Mustafar's core contribution and is directly comparable. \n\nIn contrast, HeadKV is not a competing baseline but a complementary method, operating on a token selection axis rather than compressing the KV cache in the channel-dimension. We view Mustafar as *orthogonal* to such token-level eviction methods, and in fact, Mustafar is explicitly designed to be combined with them. \n\nTo demonstrate this, Section 4.2 includes joint application example with token eviction method of H2O. HeadKV can be seen as an advanced form of token eviction, performing fine-grained head-specific token retention. Accordingly, we present the results of combining Mustafar with HeadKV on Llama-3-8B-Instruct. For evaluation, we used one representative task from each LongBench category: single-document QA (NarrativeQA), multi-document QA (HotpotQA), summarization (GovReport), few-shot learning (TREC), synthetic (Pcount), and code (LCC).  \n\n\n| Method                        | NrtvQA | HotpotQA | GovReport | TREC | Pcount | LCC   |\n|------------------------------|-------------|----------|------------|------|--------|-------|\n| HeadKV [KV size = 128]                 | 22.33       | 43.66    | 21.67      | 72.00 | 5.67   | 59.39 |\n| HeadKV + MustafarK0.5 V0.5       | 23.24       | 42.61    | 20.56      | 71.50 | 5.92   | 58.00 |\n| HeadKV + MustafarK0.7 V0.7       | 23.45       | 39.07    | 20.18      | 64.50 | 5.94   | 54.26 |\n\n\nThe table above shows minimal accuracy loss when applying Mustafar's 50% sparsity on top of HeadKV's aggressive 128-token budget (1.5% of dense size), achieving additional compression to 0.975% of dense KV size. With 70% sparsity, Mustafar achieves further compression to 0.675%. These results reinforce our claim that Mustafar enables further compression beyond what structured or token-level techniques achieve alone, and should be seen as an enabling building block for such methods, rather than an alternative baseline.\n\n> **W2) The evaluation lacks standard long-context benchmarks like \"Needle in a Haystack\", which have become prevalent in KV cache compression research [1-3]. LongBench's average sequence length may not adequately assess performance in true long-context scenarios.**\n\nAcknowledging the need for additional benchmarks that span a longer context length, we report accuracy of Llama-3.1-8B-Instruct on RULER [1] with a context length of 65536. We note that RULER contains “Needle in a Haystack”.\n\n| Sparsity         | Method     | Needle-Single1 | Needle-Single2 | Needle-MultiKey1 | Needle-MultiKey2 | Needle-MultiQuery | Needle-MultiValue | QA-1  | QA-2  | Variable Tracking | Frequent Words Extraction |\n|------------------|------------|----------------|----------------|------------------|------------------|-------------------|-------------------|-------|-------|--------------------|----------------------------|\n| Dense            | —          | 1.00           | 1.00           | 0.990    | 0.979    | 0.990    | 0.979  | 0.844 | 0.594 | 0.973  | 0.851                      |\n| Key 50%          | ThinK      | 1.00           | 1.00           | 0.990    | 0.979  | 0.995   | 0.969  | 0.833 | 0.594 | 0.919  | 0.854     |\n|   | Mustafar   | 1.00           | 1.00           | 0.990 | 0.979  | 0.995  | 0.996   | 0.833 | 0.573 | 0.971              | 0.813        |\n| Key 70%          | ThinK      | 0.448          | 0.490          | 0.229            | 0.188            | 0.646             | 0.487             | 0.615 | 0.510 | 0.208              | 0.427                      |\n|                  | Mustafar   | 1.00  | 1.00  | 0.990  | 0.969  | 0.992   | 0.903   | 0.833 | 0.594 | 0.966              | 0.823    |\n| Value 50%        | ThinK      | 1.00           | 1.00           | 0.990  | 0.969  | 0.914   | 0.958  | 0.823 | 0.573 | 0.910   | 0.792    |\n|     | Mustafar   | 1.00   | 1.00  | 0.979    | 0.995   | 0.995   | 0.971    | 0.833 | 0.604 | 0.983   | 0.830   |\n| Value 70%  | ThinK | 0.948   | 0.927  | 0.948  | 0.510   | 0.698 | 0.688  | 0.646 | 0.500 | 0.558 | 0.677   |\n|    | Mustafar   | 1.00  | 1.00  | 1.00 | 0.979    | 0.992 | 0.969  | 0.833 | 0.594 | 0.985  | 0.826  |\n| Key&Value 50%    | ThinK      | 0.958 | 1.00 | 0.948  | 0.854 | 0.828 | 0.956  | 0.740 | 0.531 | 0.742  | 0.823   |\n|  | Mustafar   | 1.00  | 1.00 | 0.990 | 0.979 | 0.997   | 0.997  | 0.833 | 0.573 | 0.862  | 0.809 |\n| Key&Value 70%    | ThinK      | 0.000 | 0.073  | 0.000 | 0.000  | 0.000 | 0.000  | 0.219 | 0.250 | 0.000  | 0.035  |\n|    | Mustafar   | 1.00  | 1.00  | 0.990  | 0.969 | 0.995 | 0.914 | 0.833 | 0.583 | 0.869 | 0.799 |\n\nEven in challenging “Needle in a Haystack” scenarios with multiple keys and queries, Mustafar maintains accuracy comparable to dense LLM. Also, it generally outperforms the structured pruning baseline ThinK, with particularly significant gains at joint key-value 70% sparsity.\n\n> **W3) For efficiency analysis, direct comparisons of first-token latency and average decoding speed against full cache or methods like HeadKV would provide more straightforward evaluation metrics.**\n\nAs discussed in our response to W1, Mustafar is orthogonal to token-level eviction methods such as HeadKV, and is designed to be complementary rather than directly competitive. Therefore, in our efficiency analysis, we focus on comparisons against the dense KV cache, which provides a more appropriate baseline to measure the impact of Mustafar’s unstructured compression. We compare against dense decoding using FlashAttention on Llama-2-7B and Llama-3-8B-Instruct with batch size 4. For Llama-2, we used input sequence length of 2048. For Llama-3, we used input sequence length of 4096. We use output sequence length of 512, 1024, and 2048. \n\nWe derived the average decoding speed by measuring the end-to-end duration, and dividing it to the number of tokens generated to penalize Mustafar with the overhead of KV cache pruning and compression in both prefill and decode stages.\n\n|           Model            |       KV Format       |   TTFT   | Decode Speed (decode 512) | Decode Speed (decode 1024) | Decode Speed (decode 2048) |\n|---------------------------|------------------------|----------|-----------------------------|------------------------------|------------------------------|\n| Llama2               | Dense                  | 1.396 sec | 88.685 tokens / sec         | 88.512 tokens / sec          | 79.185 tokens / sec          |\n|                           | Mustafar K0.5 V0.5        | 2.532 sec | 89.452 tokens / sec         | 89.514 tokens / sec          | 85.687 tokens / sec          |\n|                           | Mustafar K0.7 V0.7        | 2.249 sec | 96.386 tokens / sec         | 97.436 tokens / sec          | 95.120 tokens / sec          |\n| Llama3               | Dense                  | 2.769 sec | 61.993 tokens / sec         | 61.220 tokens / sec           | 59.242 tokens / sec          |\n|                           | Mustafar K0.5 V0.5        | 3.269 sec | 78.434 tokens / sec         | 83.768 tokens / sec          | 83.303 tokens / sec          |\n|                           | Mustafar K0.7 V0.7%        | 3.151 sec | 84.065 tokens / sec         | 88.293 tokens / sec          | 89.699 tokens / sec          |\n\nWhile time-to-first-token is delayed due to the overhead of pruning and compressing the KV cache during the prefill stage, the delay is offset by the accelerated attention computation during decoding, resulting in higher overall token generation throughput. Notably, Llama-3 exhibits a larger performance gain compared to Llama-2, as its GQA architecture reduces the overhead of KV cache pruning and compression. \n\n> **W4) In Section 3 (Sparse Attention Kernel), presenting the computation process through formal equations or algorithms would enhance clarity.**\n\nThank you for this suggestion. As we specify the pruning algorithms in Section 2, we now include a concise sequence of formal equations that detail how our kernel operates on the compressed sparse format. For brevity, we omit the per-head decomposition and normalization step, which follow the standard transformer formulation.\n\n*Let:*\n\n Query vector at decoding step $ t $: $ \\mathbf{Q}_t \\in \\mathbb{R}^d $, \n\nKey vectors within the local window: $ \\mathbf{K}_L \\in \\mathbb{R}^{d \\times N_d}, \\text{where } N_d $ is the size of local window in tokens.\n\nCompressed Key vectors outside the local window: $ \\mathbf{K}_C \\in \\mathbb{R}^{d \\times N_s}, \\text{where } N_s $ is the number of compressed tokens.\n\nCorresponding Value caches: \n$ \\mathbf{V}_L \\in \\mathbb{R}^{d \\times N_d} , \\mathbf{V}_C \\in \\mathbb{R}^{d \\times N_s} $\n\n*Attention score computation:* \n\nDense local window attention score: $ \\mathbf{S}_L = \\mathbf{Q}_t \\cdot \\mathbf{K}_L , \\in \\mathbb{R}^{1 \\times N_d} $\n\nSparse attention score over pruned KV cache: $ \\mathbf{S}_C = \\mathbf{Q}_t \\cdot \\mathbf{K}_C , \\in \\mathbb{R}^{1 \\times N_s} $\n\nFull attention weight $ \\mathbf{S}_t = \\text{softmax}(\\text{concat}(\\mathbf{S}_C, \\mathbf{S}_L)) , \\in \\mathbb{R}^{1 \\times (N_s + N_d)} $\n\n*Final Output computation:*\n\n$ [\\mathbf{S}_C, \\mathbf{S}_L] = \\text{split}(\\mathbf{S}_t ; N_s, N_d) $\n\n$ \\mathbf{O}_t = \\mathbf{V}_C \\cdot \\mathbf{S}_C^\\top  + \\mathbf{V}_L \\cdot \\mathbf{S}_L^\\top, \\in \\mathbb{R}^{d} $\n\nReferences:  \n\n[1] Cheng-Ping Hsieh, Simeng Sun, Samuel Kriman, Shantanu Acharya, Dima Rekesh, Fei Jia, Yang Zhang, Boris Ginsburg. *RULER: What's the Real Context Size of Your Long-Context Language Models?*"}, {"review_id": "x4y0IgsSUL", "reviewer": "Reviewer_QfM2", "summary": "This paper introduces an unstructured pruning method for KV cache compression. At first, this paper compares ThinK, structured pruning, per-token magnitude-based unstructured pruning, and per-token output-aware unstructured pruning to determine the pruning strategy for the Key and Value cache, respectively. Moreover, a sparse attention kernel is implemented to show that their method is able to provide real efficiency for current LLMs. Basically, this method is faster than previous approaches when the compression rate is the same, while achieving a lower compression rate than previous approaches when the performance is the same.", "questions": "1. Does this method consider the difference in importance between layer-wise and head-wise? Could the author show some toy experiments to demonstrate this question?\n\n2. Is this method effective for models using non-standard KV cache architectures such as Mamba and RWKV?", "limitations": "Yes.", "rating": 4, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 3, "significance": 3, "originality": 3, "strengths_and_weaknesses": "Strengths \n\n1. The overall logic is clear. The proposed motivation experiments clearly support their method.\n\n2. The evaluation is sufficient on various datasets.\n\n3. The implementation of a real sparse kernel makes their method valuable in the real world. \n\n\nWeaknesses\n\n1. Is the bitmap format suitable for low sparsity scenarios? For example, at a sparsity rate <30%, the cost of bitmap+index may be greater than that of dense representation? Could the author provide some examples for this?\n\n2. This paper only introduces ThinK as a structured baseline in method comparison. Could the author add more baseline methods into the comparison? If not possible, I also hope the author can explain why.\n\n3. Typos: Miss '.' in the caption of Table 1. Line 101, 103, 110: 'Query vector(blue) ' -> 'Query vector (blue)', 'pruning score(green)' -> 'pruning score (green)', and 'Group Query Attention(GQA) ' -> 'Group Query Attention (GQA)'.", "was_revised": true, "final_justification": "I recommend accepting this paper (Borderline accept).", "rebuttal": "We thank the reviewer for the constructive feedback.\n\n> **W1. Is the bitmap format suitable for low sparsity scenarios? For example, at a sparsity rate <30%, the cost of bitmap+index may be greater than that of dense representation? Could the author provide some examples for this?**\n\nOur bitmap format compresses the KV cache using a per-element bitmap, a per-64-element tile offset (int32), and the compressed non-zero values. For a 4096 (tokens) × 4096 (channels across all attention heads) FP16 matrix, this corresponds to a fixed 2 MB bitmap, a 1 MB tile offset, and a variable-sized compressed non-zero array scaled by the sparsity (i.e., 2 × 4096 × 4096 × sparsity bytes). Given that the original dense matrix occupies 33 MB, compression yields a size benefit when sparsity exceeds 9.4%.\n\nHowever, while our format primarily targets memory compression, we also optimize for latency. Our attention kernel directly leverages the bitmap format to accelerate the memory-bound attention computation, where kernel performance scales with the compression ratio to impose a more stringent requirement on sparsity.\n\nIn terms of raw kernel throughput, excluding runtime pruning and compression overhead, our method surpasses cuBLAS at 30% sparsity and outperforms FlashAttention during decoding at 40% sparsity.\n\n> **W2. This paper only introduces ThinK as a structured baseline in method comparison. Could the author add more baseline methods into the comparison? If not possible, I also hope the author can explain why.**\n\nWe selected ThinK as the sole baseline because, to the best of our knowledge, it is the only existing work that performs channel-wise (i.e., per-token) pruning, which is compatible with and can be jointly applied alongside orthogonal KV cache compression techniques.\n\nIn our paper, what we referred to as “token-wise eviction” corresponds to structured pruning along the channel dimension. We view this form of structured pruning (H2O, SnapKV [1], AdaKV [2], and HeadKV [3]) not as a direct competitor, but rather as a complementary technique that Mustafar can be layered on top of.\n\nIn Section 4.2, we evaluated the effect of joint application with token-wise eviction of H2O to demonstrate this complementarity. Specifically, for the remaining KV cache after such token-wise pruning, Mustafar can be applied to further prune the cache at a finer, per-token granularity. We additionally demonstrate the effect of joint usage with HeadKV, a head-granular token-wise pruning method, in our response to Reviewer PxDY, W1.\n\n> **W3. Typos: Miss '.' in the caption of Table 1. Line 101, 103, 110: 'Query vector(blue)' → 'Query vector (blue)', 'pruning score(green)' → 'pruning score (green)', and 'Group Query Attention(GQA)' → 'Group Query Attention (GQA)'.**\n\nThank you for pointing this out. We will fix the typos in the revised version.\n\n> **Q1. Does this method consider the difference in importance between layer-wise and head-wise? Could the author show some toy experiments to demonstrate this question?**\n\nOur evaluation focused on the ability to achieve higher sparsity using unstructured sparsity compared to structured sparsity, using a globally uniform sparsity level of 50% and 70% applied across a model's entire KV cache. While this setup was noted in Section 6, and we explicitly pointed to layer-wise (as explored in What Matters in Transformers? [4]) and head-wise (as explored in HeadKV) sparsity as future directions, we acknowledge the reviewer’s comment and have conducted a toy experiment to explore this further.\n\nWe first combine Mustafar's 60% KV cache pruning with HeadKV, a head-level, token-wise eviction method, to reduce experiment time. This setting is denoted as HeadKV + Mustafar (uniform 60%). Next, we use the head-level importance score from HeadKV to assign 50% sparsity to the more important half of each layer's heads, and 70% sparsity to the less important half, averaging to an effective 60% sparsity overall. This setting is denoted as HeadKV + Mustafar (head-wise). We evaluate this setup on one representative task from each LongBench category: single-document QA (NarrativeQA), multi-document QA (HotpotQA), summarization (GovReport), few-shot learning (TREC), synthetic (Pcount), and code (LCC), on Llama-3-8B-Instruct. \n\n| Method                                | narrativeQA | hotpotQA | gov_report | TREC  | Pcount | LCC   |\n|---------------------------------------|-------------|----------|------------|-------|--------|-------|\n| HeadKV [KV size = 128] Dense                    | 22.33       | 43.66    | 21.67      | 72.0  | 5.67   | 59.39 |\n| HeadKV + Mustafar (uniform 60%)      | 22.14       | 41.85    | 20.57      | 70.50 | 5.87   | 56.22 |\n| HeadKV + Mustafar (head-wise)    | 23.34       | 41.99    | 20.61      | 71.0  | 5.71   | 57.46 |\n\n\nThe table above shows that, in most tasks, except for a slight deviation on Pcount, varying head-level sparsity based on head-level importance scores yields better model accuracy than using uniform sparsity. While our current setup only applies either 50% or 70% sparsity, we note that Mustafar's sparse format and attention kernel support arbitrary sparsity values. This flexibility enables more precise alignment between head-level importance and sparsity, which we expect could further improve accuracy preservation.  \n\n> **Q2. Is this method effective for models using non-standard KV cache architectures such as Mamba and RWKV?**\n\nMustafar KV cache compression method is primarily designed to address the growing memory footprint associated with conventional attention-based models, where the KV cache scales linearly with the context length. As context windows grow, so does the memory cost, making compression both necessary and impactful.\n\nIn contrast, non-standard architectures like Mamba and RWKV, not only less common in production compared to conventional attention-based models, employ constant-sized compressive memory, which decouples memory usage from context length. As such, they do not face the same scalability challenges and are not the primary target of our method.\n\nReferences:  \n[1] Yuhong Li, Yingbing Huang, Bowen Yang, Bharat Venkitesh, Acyr Locatelli, Hanchen Ye, Tianle Cai, Patrick Lewis, Deming Chen. *SnapKV: LLM Knows What You are Looking for Before Generation.*  \n\n[2] Yuan Feng, Junlin Lv, Yukun Cao, Xike Xie, S. Kevin Zhou. *Ada-KV: Optimizing KV Cache Eviction by Adaptive Budget Allocation for Efficient LLM Inference.*  \n\n[3] Yu Fu, Zefan Cai, Abedelkadir Asi, Wayne Xiong, Yue Dong, Wen Xiao. *Not All Heads Matter: A Head-Level KV Cache Compression Method with Integrated Retrieval and Reasoning.*\n\n[4] Shwai He, Guoheng Sun, Zheyu Shen, Ang Li. *What Matters in Transformers? Not All Attention is Needed.*"}]

Author rebuttal:

Meta-review: After a fruitful discussion that addressed many of the reviewers' concerns, the consensus is that this is a nice paper with results that would be of interest to the community. Thus, I recommend acceptance.
Decision: Accept (poster)