SmallKV: Small Model Assisted Compensation of KV Cache Compression for Efficient LLM Inference
Authors: Yi Zhao
Yajuan Peng
Nguyen Cam-Tu
Zuchao Li
Wang Xiaoliang
hai zhao
Xiaoming Fu
Venue: NeurIPS 2025 spotlight ()
Keywords: KV Cache Compression
Efficient LLM Inference
Primary area: deep_learning

Abstract: KV cache eviction has emerged as an effective solution to alleviate resource constraints faced by LLMs in long-context scenarios. However, existing token-level eviction methods often overlook two critical aspects: (1) their irreversible eviction strategy fails to adapt to dynamic attention patterns during decoding (the saliency shift problem), and (2) they treat both marginally important tokens and truly unimportant tokens uniformly, despite the collective significance of marginal tokens to model performance (the marginal information over-compression problem). To address these issues, we design two compensation mechanisms based on the high similarity of attention matrices between LLMs with different scales. We propose SmallKV, a small model assisted compensation method for KV cache compression. SmallKV can maintain attention matching between different-scale LLMs to: 1) assist the larger model in perceiving globally important information of attention; and 2) use the smaller model’s attention scores to approximate those of marginal tokens in the larger model. Extensive experiments on benchmarks including GSM8K, BBH, MT-Bench, and LongBench demonstrate the effectiveness of SmallKV. Moreover, efficiency evaluations show that SmallKV achieves 1.75 - 2.56 times higher throughput than baseline methods, highlighting its potential for efficient and performant LLM inference in resource constrained environments.

SmallKV: V: Small Model Assisted Compensation of KV Cache Compression for Efficient LLM Inference
Yi Zhao' Yajuan Peng"", Cam-Tu Nguyen Zuchao Li', Xiaoliang Wang Hai Zhao' 'and Xiaoming Fu AGI Institute, School of Computer Science, Shanghai Jiao Tong University
2 Shanghai Key Laboratory for Intelligent Information Processing. Fudan University State Key Laboratory for Novel Software Technology. Nanjing University School of Artificial Intelligence, Wuhan University Institute of Computer Science, University of Göttingen. Göttingen, Germany
Abstract
KV cache eviction has emerged an effective solution to alleviate resource con- straints faced by LLMs long-context scenarios. However, existing token-level eviction methods often overlook two critical aspects: (1) their irreversible evic- tion strategy fails to adapt to dynamic attention patterns during decoding (the saliency shift problem), and (2) they treat both marginally important tokens and truly unimportant tokens equally, despite the collective significance of marginal tokens to model performance (the marginal information over-compression prob- lem). To address these issues, we design two compensation mechanisms based on the high similarity of attention matrices between LLMs of different scales. We propose SmallKV, a small model assisted compensation method for KV cache compression. SmallKV can maintain attention matching between different-scale LLMs to: 1) assist the larger model in perceiving globally important information of attention; and 2) use the smaller model's attention scores to approximate those of marginal tokens in the larger model. Extensive experiments on benchmarks in- cluding GSM8K, BBH, MT MT-Bench, and LongBench demonstrate the effectiveness of SmallKV. Moreover, efficiency evaluations show that S achieves 1.75 2.56 times higher throughput than baseline methods, highlighting its potential for efficient and performant LLM inference in resource constrained environments.
Introduction
Large language models (LLMs) 22 have emerged with remarkable natural language understanding capabilities and broad application prospects. Despite the advancements, the deployment of LLMs hindered with significant computational challenges and high GPU memory consumption, particularly when processing long contexts. This issue arises from the intrinsic complexity of their self-attention mechanism, which scales quadratically with the length of the input sequence. Recent developments in reasoning models, exemplified by CharGPT-ol 28 and DeepSeek-RI have exacerbated this issue due to their lengthy reasoning process.
Equal contribution: Yi Zhao (zhao-yi @ (jt..edu.cn)) and Vajuan Peng (yjpeng24 m.fudan.edu.cn) Corresponning author: Hai Zhao (zhaohai@ca www.eeessss.c......ee and Xiaoming Fu (fuxiaoming @ ffudan.edu.cc.. This work was supported by The Major Program of Chinese National Foundation of Social Sciences under Grant "The Challenge and Governance of Smart Media on News Authenticity" (No. 237D213), the National Natural Science Foundation of China (No. 62306216), 6)), and partially supported by the EU Horizon Europe CODECO project (Grant No. 101092696).
39th Conference on Neural Information Processing Systems (NcurIPS 2025).

---

Numerous studies have demonstrated the high degree of sparsity within attention mechanism, leading the development of various Key- KeyyValue cache (KV cache) compression methods such as quantization These methods significantly reduce GPU memory usage and enhance the throughput of inference systems. Our work focuses on eviction-based methods, which identify and retain only the critical tokens to reduce KV cache consumption while minimizing performance degradation in a training-free free manner.
15 9
eviction [51 33 43 and merging 17
38 27
Current KV cache eviction methods, however, face two key challenges: (1) the saliency shift issue caused by dynamic changes token importance during decoding, where permanent token removal strategies become suboptimal when the decoding process evolves, and (2) the classification of tokens into unieoortantmttt categories fails account for marginal tokens that collectively contribute significantly to model performance despite their individually modest attention scores. Existing approaches lack mechanisms to adapt to shifting saliency patterns or to apply differentiated treatment to these three distinct token categories (critical, marginal, and unimportant), leading to either excessive memory consumption or unnecessary quality degradation.
A previous study 4 has revealed a notable similarity in attention patterns between small and large models within the BERT architecture. We further discover the similar observation in the decoder-only architecture models, leading to a novel perspective for addressing the aforementioned limitations. We then propose Small Model Assisted Compensation of KV Cache Compression for Efficient LLM Inference (SmallKV), which introduces a small language model (SLM) to perform saliency shift compensation and marginal information compensation for KV Cache Compression (of LLM). Specifically, the saliency shift compensation mechanism leverages the SLM to maintain global critical information and help identify evicted tokens that may regain significance. Meanwhile, the marginal information compensation identifies tokens with relatively lower attention scores, yet still contribute to model performance. These tokens are less sensitive to the approximation of attention, allowing us to leverage attention scores of SLM for compensation. This results in hierarchical compression strategy that differentiates and compresses tokens based on their varying levels of importance. should be noted that S is compatible with efficient attention implementation such as Flash Attention, which significantly enhances the efficiency of inference. In practical deployment, SmallKV can be used with speculative decoding (18) to speed up LLM inference further.
We conduct a comprehensive evaluation of SmallKV across several benchmarks, including GSM8K 66) BBH [3.) MT-hench [54]. and Longbench [. The experimental results indicate that SmallK V consistently delivers superior performance, especially under low KV cache budgets. Experiments on different model series (e.g., the Qwen series and LLaMA series) and model sizes (ranging from 7B to 72B) highlight the robustness and generalizability of our approach. Efficiency evaluations show that SmallKV achieves 1.75- 2.56 times higher throughput compared to previous KV cache compression methods. These experimental results collectively offer compelling evidence of SmallKV's accuracy and efficiency compensation plugin for KV cache compression.
2 Related Work
The approaches to KV cache compression can be broadly categorized into three classes: eviction, merging, and quantization. Eviction [30 31 19 25 tokens' KV caches to achieve nearly lossless inference performance. Merging leverages the high
55 49] aims to retain only small set of critical
angular similarity observed among deep KV caches to reduce both intra-layer [7 38 27
50 37 39
and cross-layer (45 23 redundancies. Quantization compresses the data by mapping the original full-precision tensor values to discrete levels and storing them at lower precision, including model
weight quantization 33 46 32 and KV-cache-only quantization [15
[6].
Numerous eviction methods have been proposed, which can be categorized into static and dynamic strategies. Static strategies I0, 21 47 perform token filtering during prefilling and maintain KV
cache of fixed size throughout the subsequent decoding steps (e.g., sliding window attention 22). A
further approach is to maintain both the initial and the recent tokens [2]. Dynamic strategies [51
43 52, [
continuously update the critical KV cache during decoding, while the KV cache of
unimportant tokens will be permanently removed or offloaded from the GPU. The core of permanent eviction lies in selecting critical tokens to minimize the damage to model accuracy. H₂O (51 observed that almost all layers exhibit sparsity exceeding 95%, indicating that maintaining just 5% of the KV cache based on the accumulative attention scores of each token is sufficient for decoding the

---

same output token at each generation step. PyramidInfer 43 processing approach, where recent tokens are assigned greater weight and the length of KV caches in deeper layers is reduced. Permanent token removal has two main limitations. Irreversible token eviction can degrade model performance multi-turn dialogue scenarios and Long-sequence tasks. Additionally, relying on attention scores prevents the model from adapting to some acceleration techniques (c.g., Flash Attention). Consequently, non-permanent eviction 440 35 48 [4 (24 has been proposed by dynamic cache offloading based on indexiig. These methods manage and access the KV cache offloaded to multi-tier cache systems at the granularity of chunks or clusters. However, achieving fast and accurate retrieval with high precision remains challenging, while the index construction for retrieval introduces significant decoding latency.
employs a syramid-shaped hierarchical
Speculative decoding 220 18 3 53 is widespread technology to accelerate inference. It employs a pair of models, where the smaller one generates candidate tokens in advanced and the larger one verifies them in parallel. By enabling parallel token generation, speculative decoding addresses the inefficiencies inherent in traditional autoregressive approaches, achieving speedups of over three times without compromising the quality of the generated output. This technique has been widely adopted in commercial LLMS 22).
3
Pilot Observation
3.1 Saliency Shift Issue
The dynamic nature of LLM decoding leads to the shift in token saliency, which is the change of the token set with high attention over time. Most existing KV cache eviction strategies, which rely on permanent token removal 43 are highly susceptible to this issue. To address this, we leverage a small language model (SLM) from the same model series to approximate the LLM's attention
scores. This approach aligns well with speculative decoding 120 18
3
53 22]
where the SLM
generates candidate tokens advance and the LLM verifies them in parallel, thereby allowing the combination of these two appealing approaches for further optimization of inference speed.
Observation 1. The saliency shift leads to the discrepancy of the important tokens between the compressed KV cache view and the uncompressed global view, and this gap causes the model to lose critical information during inference. We mimic the continual process of KV cache compression in Figure M (a). Here, in "real-drop" procedure, we perform two times compression, mimicking the continual compression in real decoding scenario. In the first compression, the important token (yellow) information in the first half of the dashed line is retained, while the remaining unimportant tokens (blue) are evicted and their KV cache are permanently removed. The important tokens, along with the new ones in the second half of the dotted line, continue to participate in subsequent decoding and the second-time compression. However, at the second time compression, the global view of an uncompressed cache shows differences in the selection of important tokens (red), i.c., the shift in token saliency between the second compression step and the global view-based compression.
Specifically, we quantified the gap caused by saliency shift on the Wikitext-v2 [26] dataset using Qwen2-7B in Figure (b). The importance of tokens is measured by accumulative attention scores, following H20[5]]. We measure the discrepancy between sets of important tokens derived from the real drop view and the global view using Jaccard similarity. The relatively low Jaccard similarity values, ranging from 0.55 to 0.77, indicate a significant difference between realistic KV cache eviction methods and the global ground truth, which means a considerable number of globally important tokens are erroneously evicted under the real drop method. In addition, the similarity maintains a clear declining trend as the KV cache budget decreases, indicating that the influence brought by saliency shift is more serious. This is particularly concerning since most current KV cache compression methods claim to operate with a low KV cache budget (c.g., 10%-30%). Therefore, an effective approach to compensate for saliency shift issue is critical and necessary.
The core issue with the saliency shift lies in the inability of existing KV cache compression methods to effectively maintain past information after compression. These methods predominantly focus on LLM itself without considering external information. We, however, observe that L.MSs of different sizes within the same series tend to exhibit highly consistent attention patterns. Therefore, we propose collaborative approach during inference, where an assisted SLM works with the LLM in parallel, using the attention matrices of the SLM for saliency shift compensation.
Insights 1.
3

---

NADA
e
Imperimal Drspact Difeceni
Overbe Acariin
Drop
Giktal View
Saliency Shift 0
47
15
42
33
10
2017
Token Indes
(a)
(b)
(c)
(d)
Figure| 1: (a) The illustration of saliency shift. (b) The quantification of saliency shift issue by measuring Jaccard similarity of important tokens between real drop and global view of critical tokens. Lower Jaccard similarity indicates that more important tokens have been wrongly evicted. (c) The visualization of highly consistent attention patterns between LLMs with different scaling. indicating that SLM can assist in capturing and preserving global attention information. (d) Comparison of 10% KV cache budget for saliency shift compensation using Qwen2-0.5B to assist Qwen2-7B.
Cittical
Sub important
Unimportant
Full
Hierarchicul Compression
Reserved
Evicted
Accumusstion dtterton
Critical
Lache
V Vache
Sub- cob--mpprttnt
Coche
V Cache
Unimportant
Top Altention Scores reserved (%)
Figure 2: Top: The distribution of attention sparsity (blue bar) along with the drop of model accuracy (red line), indicating the necessity of maintaining marginal tokens. Horizontal line (Full) represents the baseline of full KV cache. Bottom: Marginal information compensation by the hierarchical compression that differentiates tokens based on their varying levels of importance.
We measure the similarity of the attention patterns between Qwen2-0.5B and Qwen2-7B on Wikitext- v2 (26) dataset. Figure (c) illustrates one example of the trends of accumulative attention scores between SLM and LLM, as more samples and detailed discussion in Appendix B For each attention head in L.M,, we search the most similar one in the SI.M for matching. The quantitative analysis shows that average cosine similarity between Qwen2-0.51 and Qwen2-7B after similarity matching reaches 0.947. The result also visualizes high degree of consistency, particularly among tokens with high attention scores. This finding indicates promising way to effectively address the saliency shift issue without introducing significant overhead. We evaluate the effectiveness of saliency shift compensation with 10% KV cache budget across six downstream tasks, as shown in Figure [ (d). The results demonstrate that using Qwen2-0.5B as the assisted SLM for saliency shift compensation significantly outperforms using only Qwen2-7B without any compensation.
3.2 Marginal Information Overcompression
Current KV cache eviction methods divide tokens into two categories-criite and unimpor- tant oon--overoooking the presence of marginal tokens. These tokens, while less impactful than critical ones, still hold significant relevance and collectively contribute far more to model performance than their attention score suggests. Unlike unimportant tokens, marginal tokens essential for preserving output quality, yet they are currently subjected to the same aggressive compression or eviction strategies as truly negligible tokens. To address this, dedicated approach is needed to: (1) distinguish marginal tokens from unimportant ones, and (2) apply tailored compression strategies (rather than outright removal) to retain their contribution.
We measure the accumulative attention scores of each token across all heads of Qwen2.5-14B 14B on the BBH dataset [34], and statistically rank these scores in Figure 2 to reveal the necessity of maintaining marginal tokens. Consistent with previous research findings, we also observe that only small fraction of tokens account for the majority of attention scores. The sum
Observation 2.

---

of the accumulative attention score for the top 5% of tokens is approximately 7 times greater than that of the 5%-10% range, while the 5%- 10% range is only 1.34 times higher than the 10%-15% range, highlighting the dominant position of the top 5% of tokens in attention scores. However, the compression of tokens range of 5% 15% has caused the accuracy to drop from 0.482 to 0.148 under H₂O method.
Insights 2. Previous methods, however, fail recognize the necessity of maintaining marginal tokens, and conflate them with critical tokens or unimportant tokens. This leads to an underutilization of attention sparsity, inducing a mandatory trade-off between KV cache consumption and model performance degradation. Our observations suggest that maintaining marginal tokens is necessary and that they can tolerate upproximation compared to critical tokens (extended analysis can be referred to Appendix (.. Therefore, a reasonable approach is to apply different compression strategies for tokens of varying importance levels, as shown Figure Critical tokens retain the full KV cache to avoid precision loss, marginal tokens retain only the V cache and use SLM to approximate the attention mechanism, and unimportant tokens are evicted completely.
As insight indicates that LLMs of different sizes within the same series exhibit highly consistent attention patterns, we propose compensating for marginal tokens based on the attention scores from the SLM. Specifically, we approximate the attention mechanism of marginal tokens by multiplying their V cache with the corresponding attention scores derived from the SLM. This method reduces the K cache consumption for these tokens while maintaining their contribution to overall model performance. By doing so, we effectively bridge the gap between theoretical attention sparsity and practical performance, thereby enhancing the efficiency and efficacy of KV cache compression.
4 SmallKV Method
We introduce the insight of SmallKV in Section involving saliency shift compensation and marginal tokens compensation assisted by the SLM. In this section, we provide a detailed description.
Similarity Matching. In prefill stage, SmallKV establishes the similarity matching between SLM and LLM. Given an LLM M and the corresponding SLM M,, the prompt is first forward in both models to obtain all attention matrices respectively. The attention matrices of M are denoted as
A; = =Softmar [Q.KT
E I 0 S i < L D, and those in M, are denoted as 4/. S < d.d.
vds
Q E gnxd and K E i denote query matrix and key matrix in attention mechanism. L and ( represent the number of layers in the M and M,. D and denote the number of attention heads per layer in each model. represents the number of tokens in the prompt. Due to differences in model scaling, we have 1 d << L- D. Let C be the KV cache of the context, the accumulative attention score vector in a specific context be expressed
F(A,, (s), st......... ..... where s. = = A
(1)
a=1
We then can calculate pairwise similarity between A, and A according to Jaccard simularity their TopK indices as:
TopK (F(A,,)) Top (F(A),
S(A, A3)
TopK (F(A,, C)) U TopK (F)
The mapping function that maps i-th attention matrix of LLM the j-th one in SLM is obtained by:
f(i) orgmax, SAA,
Saliency Shift Compensation.
Previous eviction strategies remove tokens based on the policy:
E(A,, C) = C (v), where (v} f- (o)e C)
The saliency shift problem in the previous methods can be formally defined as:
E(A,, Cau U {t}) # E(A,, C, U (t))
(5)
where Can denotes the full KV cache, C, represents the compressed cache obtained after prefilling, and indicates the recently added token. The equation suggests that the selection of critical tokens from compressed cache (c.g., after prefilling) is different from that if we have the full cache.
5

---

Noxevel Erise
Carh
Nalinncy Alll
Prefiliing
Siitarty Voiching
Decoding
1d K Stade
GPU
Nend Tiken
Stade
Global Insorriacce
SLM
LLM Micaaiin
LLR
key mibev
Ourput
Cache
kilal
Topki
lfffced
Upante
Catkal
Driond's
Atteriia Dutpee
Figure 4: The system architecture of SmallKV. (The Figure 3: The attention illustration of the detail of SmallKV attention compensation is in Figure 3 SmallKV method in decoding.
Unlike previous methods, SmallKV maintains the full cache of SLM C.u- and performs eviction for the cache of L.LM based on he (ii)-th (full) cache of SLMM and the eviction policy ((A Cau)-
Marginal Information Compensation. In decoding stage, let the attention be A, E R xm the vanilla attention can be expressed O; A, V Rixd where V2 Rxxd denotes the value matrix. The attention mechanism for Marginal Information Compensation can be expressed as O Ai V,, where A is calculated as follows:
if E Topk (F(A'r Cail).
Ag
[K]
A' A k E Top(P K) (F(A'(0) Can)
(6)
0,
otherwise.
where K and P P K denote the numbers of critical tokens and marginal tokens, which determined by KV cache budget.
The formula shows that for marginal tokens that appear in the Top( P K) importance, approxi- mate their attention scores using the corresponding elements from SI.M M, to avoid using K cache. This approach allows for retaining important information while evicting the K cache of these tokens, effectively implementing a hicrarchical compression policy that diffcrentiates between various levels of token importance. The illustration of attention mechanism in SmallKV is shown in Figure
Method Details. In practice, to ensure the stability of matching correspondence between SLM and LLM, we dynamically determine the timing of similarity matching based on context length. Specifically, we control the token length for similarity matching within the range 100 to 200. This avoid the matching inaccuracy caused by excessively short token sequences and the distortion of similarity in high-dimensional space caused by overly long token sequences. If the current context length does not meet the minimum threshold, SmallKV delays the timing of similarity matching and KV cache eviction until the requirements are satisfied. Similarly, for context lengths exceeding the threshold, we truncate the prefill token length for similarity calculation.
The detailed procedure of SmalKKV can be referred to Algorithm TT] It is important note that SmallK V does not delete the KV cache but migrates it between GPU HBM and CPU memory, as the process of updating KV cache of critical tokens and V cache of marginal tokens in line of the Algorithm. This migration process executes in parallel with the forward of LLM to prefetch KV cache for latency reduction. Within the attention of LLM, the computation critical tokens and marginal tokens is also performed in parallel. The computation for critical tokens benefits from acceleration via Flash Attention, whereas the computation for marginal tokens involves only a matrix multiplication. The system architecture of SmallKV is shown in Figure 4

---

Algorithm SmallKV Algorithm
Input: Large language model M; Assisted small language model M,; Input æ; KV cache budget 7;
1: if Prefill then
2: Allocate KV cache budget to critical budget Te, marginal budget Te
3: Parallel forward process of M (x) and (x) to get attention matrices A, and A'
4: Establish correspondence of A, and A by similarity matching (Eq. (2) and Eq. 33
5: Output new token by (x) and append it to
6:
else
7:
Forward process of (x + t) to update A3
8: In parallel:
9: Update KV cache of critical tokens and Update V cache of marginal tokens by E f(1 Cau)
10:
for Attention layer E layers LLM do
11:
In parallel:
12:
Calculate attention output O₂ critical tokens using Flash Attention
13:
Calculate attention output On, of marginal tokens using Eq. 6
14:
Attention Output by O₂ + On end for
16:
Output new token & by M ((x + t) and append it to t
end if
Output: Output token sequences
(4
5
Experiments
5.1 Experiments Setup
Datasets.
We comprehensively evaluate the effectiveness of SmallKV in four kinds of scenarios: GSM8K [6] for mathematical reasoning, 2) BBH [34] for language understanding, 3) MT-Bench [54] for multi-turn conversation, and 4) Longbench for long-context scenario.
Baselines. We take two effective KV cache eviction methods as baselines. These methods select critical tokens according their attention scores, which is the same SmallKV.
H₂O 51:
employs the accumulative attention score as the metric to evict unimportant KV cache.
identifies that deeper layers exhibit greater redundancy and applies differentiated KV cache budgets between layers.
PyramidInfer 43:
Models. Our experiments are conducted on the Qwen and LLaMA series of LLMs with different scales. Specifically, in the benchmark results (Section 5.2), consider four pairwise combinations between the SI.M and the L.M. These combinations include: Qwen2-0.5BB with Qwen2-7B, Qwen2.5- 0.5B with Qwen2.5-144 14B, Qwen2-7B with Qwen2-72B, LLaMA 3.2-1B with LLaMA 3.1-8B. The Qwen2-72B model is quantized to INT4 data type for efficient computing. The other experiments are primarily performed on the combination of Qwen2-0.5B with Qwen2-7B.
Implementation Details. All experiments are conducted on NVIDIA A100 (80GB) GPUs. The configurations of environment include: CUDA (12.0), PyTorch (2.4.0), and huggingFace" Transformer: (4.45.1). We use greedy decoding to ensure the stability of the experimental results.
Consistent with previous study, we also allocate proportion of the KV cache budget to recent tokens. We adopt fixed ratio of 2:1:2 for allocating resources among critical tokens, recent tokens, and marginal tokens, respectively. For instance, with 20% KV cache budget, we allocate 10%, 5% and 10% budget to critical tokens, recent tokens, and marginal tokens (as marginal tokens only use cache of LLM, it actually consumes half of budget). When the available KV cache budget becomes limited so that it cannot fully accommodate all critical tokens (e.g. 5%), we proportionally reduce the allocations for recent and marginal tokens to maintain model performance.

---

Gwer2-0 Qwenz.78
Gwenz 1W0r2.5-348
Dwen2:78 Ower2-728
3.2:18 With LIMMA
CSANTK Accunccy
Accurccy
28
CONIK
GENIK-
20
Pyandrffrr
4D-
Pyamctttrr
Fyamddrrter
Pyroidirrer
Ful
20
KV cacre 16)
KV cacte
cachc 761
KV cacho (61
70-
-0
BR0 Accursicy
-0-
4D-
D001
20
R20
H20
BBH
Small 100
Pyneeiteeer
Pynamiteele
Tynwmidirler
Pynoridiiie
Fut
20-
Full
Fu1
00
40
10:
101
55 cocho
Cacho
INT- Bunch 5core 0
NTT -T--Beer Score
SAaY En Score
Ber
Smallicv
5ma RV
Smelkc H22
Sma H20
H20
Pynamatenve
H20
NY
Pynmittobee Full
Pyraiiteee
Fill
Ful
T1%
70
110
70
Figure 5: Benchmark results of SmallKV with KV cache budget varying from 100% to 5%. Full represents using the full KV cache without compression.
Longlench
Methods
Single Doc QA
Mult-Doe QA
Summarizaiion
Few shot Leurning
Code Completion
NarQA Qusper Mul.A AVG
HoLÇA ZWLQA Musque AVG
QMSum M.Neus GovRe. AVG
THLQA TREC
SAMSum AVG
LCC
Repa.
AVG
KV cuche bndger
0.3
SmallkV
20:1
42:55
48,82
37.39
46.71
51,51
23.76
40.66
23.15
23.55
19.55
22,06
86,42
48.31
68.58
57.79
55.87
56.83
H2O
22.14
33.11
45,17
33.47
44,14
48.44
2085
37.81
21.91
23.51
19.98
21.35
85.22
73
45.27
67
83
58.00
55.04
56.83
Pynmid
22.36
32.19
45.14
33333
44,14
46.01
20.77
36.97
21,33
22.97
20.08
21,46
85.42
46.01
67 82.82
58.58
55.67
$7,13
KV cache budger
SmilIYV
37.20
23.26
68.39
55.34
56.96
160
14.13
23.50
25.95
41,25
37.89
18.38
32.51
20.50
23.22
20.34
21.35
84.44
43,69
59.71
41.33
44,90
43.12
Pynanid.
25.78
16.14
32.45
20.03
20.38
59.23
41.28
46.40
43.84
Insdgrt
SmalKkV
37.54
68.34
$3.54
160
Pynmid.
9.60
34.23
19.34
19.76
43.43
Full
24.26
43.48
49.30
39.04
44.57
52.76
25.02
40.78
23.88
22.08
20.08
22.31
87.19
78
45.03
70:07
$8.26
55.64
56.05
Table 1: Performance of three methods on the LongBench with different KV cache budget (using Qwen2-0.5B with Qwen2-7B). The results of full KV cache are shown at the bottom for comparision.
5.2
Benchmark Results
Figure 5 shows the performance of four models across a range of KV cache budgets from 100% to 5% on GSM8K, BBH, and MT-Bench. The results indicate that the SmallKV method consistently outperforms bascline approaches across nearly all models and various KV cache budgets, particularly very low KV cache budgets. The marginal information compensation mechanism enables S to maintain attention information for a significant proportion of marginal tokens while using few KV cache, thereby sustaining high performance even low cache budgets. For instance, in the experiment of Qwen2-0.55 paired with Qwen2-7B, with 5% KV cache budget, the scores for H2O and PyramidInfer decline from 79.4 (at full cache) to 36.7 and 47.0, respectively, whereas the S method maintains the performance score of 73.0.
Table provides a comparative analysis of SmallKV method against baselines on Lonchench at three distinct KV cache budgets (7 0.3, 0.1, and 7 = 0.05). Our method demonstrates superior performance across all five subtasks, underscoring the efficacy of SmallKV in long-conter scenarios.
8

---

Method
Esc
Lenth
TPOT (ms)
TIFT (8)
Thr(tokens)
Accelerate (Fager) Accelerate (Flash)
59.4
22.15
62.27 (1.0x)
64
2048+256
44.6
1.09
188.23 (3.02x)
H,O (20%)
30.5
22.55
76.32 (1.23x)
SmalKKV (20%)
36.3
2.71
195,50 (3.14x)
Accelerate (Eager) Accelerate (Flash)
42.7
14.28
474.21 (1.00x)
31.3
1.41
990.39 (2.09x)
4
16384+512
H,O (20%)
20.4
14.32
689.07 (1.45x)
SmallK (20%)
24.2
1.94
1203.42 (2,54x)
Table 2: Evaluation efficiency of SmallKV. Bsz: batch size. Lenth: prefill length + decode lenth. TPOT: average time per output token in decode stage. TTFT: time first token. Thr: End-to-end throughput.
Similar to previous observations, SmallKV performs well with low KV cache budgets. For example, in subtasks such as Multi-Doc QA, Few-shot Learning, and Code Completion, baseline methods exhibit performance degradation with 5% KV cache budget, while the SmallKV method maintains performance comparable to that achieved with full KV cache. This highlights SmallKV's ability to effectively manage cache resources and deliver robust performance even under constrained conditions.
5.3 Efficiency Results
Setup. We conduct efficiency analysis on one NVIDIA A100 GPU. The evaluated model is Qwen2-7B and the SLM used in SmallKV method is Qwen2-0.5B. We use synthetic datasets for testing where all prompts are padded to the same length in the batch and the outputs are also restricted to fixed length. We consider two common scenarios: 1) Multi-user concurrency: In this scenario, the context is set to a prefill length 2048 tokens and decode length of 256 tokens, with batch size of 64. 2) Extremely long context: the context is set to a prefill length of 16384 tokens and decode length of 512 tokens, with a batch size of 4. The evaluation metric consists of average time (ms) per output token in decode stage (TPOT), time (s) to first token (TTFT), and end-to-end throughput (token/s). We use Hugging Face Accelerate and H₂O as baselines. Hugging Face Accelerate employs two attention implementation methods: eager attention and PyTorch SDPA with Flash Attention kernel M. O2O and SmallK both employ a KV cache budget of 20%. The times reported in SmallKV include the overhead of the assisted SLM.
The efficiency results are shown in Table 2 It can be observed that compared Accelerate. KV cache compression techniques (H₂O and SmallKV) significantly reduce the TPOT during the decoding. This is attributed to their reduction loading KV cache, thereby alleviating pressure on GPU memory bandwidth and decreasing the computation of attention. Specifically, introducing an additional assisted SLM in SmallKV results in increased latency during the decoding when compared to H₂O. However, due to its compatibility with memory-efficient attention methods, SmallK V's TTFT is notably lower than that of H2O, despite the overhead incurred by similarity matching during the prefill stage. In summary, although the SmallKV method incorporates overhead such as assisted SLM and similarity matching, it benefits both from the advantages of KV cache compression and compatibility with efficient attention methods. These features collectively contribute to its superior efficiency, demonstrating significantly higher throughput than baseline methods across both scenarios.
Results.
5.4 Ablation Study
We conduct ablation studies on the BBH benchmark to evaluate the contributions of the saliency shift compensation and marginal information compensation components proposed in the SmallK The experimental results are shown in Figure 6 Firstly, it is evident that the SmallK method exhibits a notable performance drop in the 10% to 40% KV cache budget range when the marginal information compensation component is absent. This observation aligns with the design purpose of marginal information compensation, which aims to compensate for marginal tokens beyond the small set of critical ones. Additionally, when the KV cache budget is under capacity to cover the critical tokens, the hierarchical compression for marginal tokens fails due to insufficient KV cache budget allocation for those tokens. This is reflected in the performance convergence between the w/o marginal information compensation and the full SmallKV V with KV cache budgets below 10%.
If we further remove the saliency shift compensation component from the w/o marginal information compensation setup, as depicted by the w/o both compensation in the figure, there an additional decline in performance. This highlights the effecctieness of saliency shift compensation, which

---

Qwen2-0.58 with Qwen2-78
Qwen2.5-14B as LLM
50
Accuracy
40
BBH Accurac)
50
30
Full
Owen2.5-0.58
20
SmallKV
Qwen2.5-1.58
w/o Marg Compensation
30
Owen2255-38
w/o both Comperisatio
Owen2.5-78
100
80
40
20
100
80
60
40
20
KV cache (%)
KV cache (5)
Figure 6: Ablation Study of SmallKV method.
Figure 7: The impact of SLM scaling on S
mitigates saliency shift issue by utilizing global information from the assisted SLM. This compensa- tion mechanism dynamically depends on context information during generation, thereby enhancing overall performance.
5.5 Impact of SLM Scaling
We also investigate the impact of scaling SLM on the performance of the SmallKV method. Theoreti- cally, as the size of the SLM increases, can provide more attention matrices, allowing the LLM to match more similar attention matrices for compensation, thereby enhancing the performance of 14B as the LLM, we experiment with four differently scaled 1.vn2.5-1155B. Qwen2.5-3B. and Qwen2.5-78 on the BBH benchmark. The results are presented Figure 7 Consistent with theoretical expectations, the performance improved gradually as the size of the SLM increased. On average, using Qwen2.5-1.5BB Qwen2.5-3B, and Qwen2.5-7B as SLMs leads to performance improvements of 4.9%, 6.8%, and 8.5% compared to using Qwen2.5-0.5B as SLM.
the SmallKV method. Using Qwen2.5- 11222.-141
SLMs: Qwen2.5-0.5B
While the performance improvements are relatively small compared to the increase in parameters of the SLM, we also note that the influence of SLM scaling becomes more pronounced with low KV cache budgets, For instance, with 5% KV cache budget, using Qwen2.5-14 Qwen2.5-7B as SLMs resulted in performance improvements of 30.2%, 63.8%, and 84.4%, respec- tively, compared using Qwen2.5-0.5B. This finding indicates that under extremely constrained KV cache budgets, it is crucial to appropriately scale up the SLM to maintain performance levels. In summary, the SmallKV method necessitates a careful trade-off between overhead introduced by scaling the SLM and the overall performance enhancement in practice.
Qwen2.5-3B, and
Conclusion
We present SmallKV, a novel small-model-assisted compensation framework that addresses two critical limitations current KV cache eviction methods: the saliency shift problem and the marginal information over-compression issue. By leveraging the high similarity of attention matrices across different model scales, SmallKV introduces two key innovations: (1) preserving globally important attention patterns through SLM assistance, and (2) accurately approximating marginal tokens using the smaller model's attention scores. Our comprehensive evaluation across multiple benchmarks (GSM8K, BBH, M'T-Bench and LongBench) demonstrates that SmalIKV consistently maintains model performance even under aggressive KV cache budgets. Notably, SmallKV achieves 1.75x- 2.56x throughput improvement over existing methods while remaining compatible with memory- efficient attention implementations like Flash Attention.
10

---

References
[1] Yushi Bai, Xin Lv, Jiajie Zhang, Hongchang Lyu, Jiankai Tang, Zhidian Huang, Zhengxiao Du, Xiao Liu, Aohan Zeng, Lei Hou, Yuxiao Dong. Jie Tang, and Juanzi Li. LongBench: A bilingual, multitask benchmark for long context understanding. In Proceedings of the 62nd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 3119-3133 3137, Bangkok, Thailand, August 2024. Association for Computational Linguistics. doi: 172.
10.18653/w1/2224.acc
org/ a 1on--ong... 172
[2] Iz Beltagy, Matthew E Peters, and Arman Cohan. Longformer: The long-document transformer. arXiv preprint arXiv:2004.05150, 2020.
[3] Charlic Chen, Sebastian Borgeaud, Geoffrey Irving, Jean- -aptisse Lespiau, Laurent Sifre, and John Jumper. Accelerating large language model decoding with speculative sampling. arXiv preprint arXiv:2300.0113/6 2023.
[4] Cheng Chen, Yichun Yin, Lifeng Shang, Xin Jiang, Yujia Qin, Fengyu Wang, Zhi Wang, Xiao Chen, Zhiyuan Liu, and Qun Liu. bert2bert: Towards reusable pretrained language models, 2021. URL https: /ttp:///axxivi atgg/bb//2110.00
[5] Yilong Chen, Guoxia Wang, Junyuan Shang, Shiyao Cui, Zhenyu Zhang, Tingwen Liu, Shuo- huan Wang. Yu Sun, Dianhai Yu, and Hua Wu. Nacl: A general and effective kv cache eviction framework for llms inference time. arXiv preprint arXiv:2408.0300 2024.
[6] Karl Cobbe, Vineet Kosaraju, Mohammad Bavarian, Mark Chen, Heewoo Jun, Lukasz Kaiser, Matthias Plappert, Jerry Tworek, Jacob Hilton, Reiichir Nakano, et al. Training verifiers to solve math word problems. arXiv preprint arXiv:2110. 141.18880,,4 2021.
[7] DeepSeek-AI. Deepseek-rl: Incentivizing reasoning capability in Ilms via reinforcement learning, 2025. URLptttss:// http://axvivv..// org abs/ 12948
[8] Harry Dong, Xinyu Yang, Zhenyu Zhang, Zhangyang Wang, Yuejic Chi, and Beidi Chen. Get more with less: Synthesizing recurrence with kv cache compression for efficient I1m inference. arXiv preprint wrXiv:2402.09398 2024.
[9] Shichen Dong, Wen Cheng, Jiayu Qin, and Wei Wang. Qaq: Quality adaptive quantization for Ilm kv cache. arXiv preprint arXiv:2403.04645 2024.
[10] Suyu Ge, Yunan Zhang, Liyuan Liu, Minjia Zhang, Jiawei Han, and Jianfeng Gao. Model tells you what to discard: Adaptive kv cache compression for llms. arXiv preprint 10.00223, 2023.
[11] Michael Gschwind, Driss Guessous, and Christian Puhrsch. Accelerated pytorch 2 transformers. https: https://pytorch.. oroooog///boo /log///lo////oo/ooo accelerated acceleraaeed-pyoorchh pytorch-2
[12] Chi Han, Qifan Wang, Hao Peng, Wenhan Xiong, Yu Chen, Heng Ji, and Sinong Wang. Lm- infinite: Zero-shot extreme length generalization for large language models. arXiv preprint arXiv:2308. 16137, 2023.
[13] Shwai He, Guoheng Sun, Zheyu Shen, and Ang Li. What matters in transformers? not all attention is needed, 2024. URL httpst:ss///a www.:/..rggg/b//b//ee abwpa/bb/////./////// 15786
[14] Coleman Hooper, Sehoon Kim, Hiva Mohammadzadeh, Monishwaran Maheswaran, June Paik, Michael W Mahoney, Kurt Keutzer, and Amir Gholami. Squeezed attention: Accelerating long context length Ilm inference. arXiv preprint arXiv:241 1,09688 2024.
[15] Coleman Hooper, Schoon Kim, Hiva Mohammadzadeh, Michael W Mahoney, Sophia Shao, Kurt Keutzer, and Amir Gholami. Kvquant: Towards 10 million context length I1m inference with kv cache quantization. Advances in Neural Information Processing Systems, 37:1270-1303, 2024.
[16] Hao Kang, Qingru Zhang, Souvik Kundu, Geonhwa Jeong, Zaoxing Liu, Tushar Krishna, and Tuo Zhao. Gear: efficient kv cache compression recipefor near-lossless generative inference of Um. arXiv e-prints, pages arXiv-2403, 2024.

---

[17] Jang- Hyun Kim, Junyoung Yeom, Sangdoo Yun, and Hyun Oh Song. Compressed context memory for online language model interaction. arXiv preprint arXiv:2312.03414 2023.
[18] Sehoon Kim, Karttikeya Mangalam. Suhong Moon, Jitendra Malik, Michael W Mahoney, Amir Gholami, and Kurt Keutzer. Speculative decoding with big little decoder. Advances Neural Information Processing Systems, 36:39236-39256, 2023.
[19] Wonbeom Lee, Jungi Lee, Junghwan Seo, and Jaewoong Sim. (InfiniGen]:: Efficient generative inference of large language models with dynamic (KV) cache management. In /8th USENIX Symposium on Operating Systems Design and Implementation (OSDI 24), pages 155-172, 2024.
[20] Yaniv Leviathan, Matan Kalman, and Yossi Matias. Fast inference from transformers via speculative decoding. In International Conference on Machine Learning, pages 19274-19286. PMLR, 2023.
[21] Yuhong Li, Yingbing Huang, Bowen Yang, Bharat Venkitesh, Acyr Locatelli, Hanchen Ye, Tianle Cai, Patrick Lewis, and Deming Chen. Snapky: Llm knows what you are looking for before generation. Advances Neural Information Processing Systems, 37:22947-22970, 2024.
[22] Aixin Liu, Bei Feng, Bing Xue, Bingxuan Wang, Bochao Wu, Chengda Lu, Chenggang Zhao, Chengqi Deng, Chenyu Zhang, Chong Ruan, et al. Deepseek-v3 technical report. arXiv preprint 2944:24712.19437, 2024.
[23] Akide Liu, Jing Liu, Zizheng Pan, Yefei He, Reza Haffari, and Bohan Zhuang. Minicache: Kv cache compression in depth dimension for large language models. Advances in Neural Information Processing Systems, 37:139997-140031, 2024.
[24] Di Liu, Meng Chen, Baotong Lu, Huiqiang Jiang, Zhenhua Han, Qianxi Zhang. Qi Chen, Chengruidong Zhang, Bailu Ding. Kai Zhang, et al. Retrievalattention: Accelerating long- context I1m inference via vector retrieval. arXiv preprint arXiv:2409. 2024.
[25] Guangda Liu, Chengwei Li, Jieru Zhao, Chenqi Zhang, and Minyi Guo. Clusterkv: Manipulating Ilm kv cache in semantic space for recallable compression. arXiv preprint arX:v2241..03213, 2024.
[26] Stephen Merity, Caiming Xiong, James Bradbury, and Richard Socher. Pointer sentinel mixture models, 2016. URL https:://axii.. org. http://rriii...r////////./
[27] Piotr Nawrot, Adrian Lańcucki, Marcin Chochowski, David Tarjan, and Edoardo M Ponti. Dynamic memory compression: Retrofitting Ilms for accelerated inference. arXiv preprint arXiv:2403.09636. 2024.
[28] OpenAL. Introducing openai ol-preview.
https://opeaal.com/ndd// com/index/
Eetroducing-openie o1- -0--011-ppp-e oo-preieeww/ 2024.
[29] OpenAl. Gpt-4 technical report, 2024. URL https:
[30] Matanel Oren, Michael Hassid, Nir Yarden, Yossi Adi, and Roy Schwartz. Transformers are multi-state mns. arXiv preprint arXiv:2401.06104 2024.
[31] Luka Ribar, Ivan Chelombiev, Luke Hudlass- Hudlass-Galley. Charlie Blake, Carlo Luschi, and Douglas Orr. Sparq attention: Bandwidth-efficien Ilm inference. arXiv preprint 2rXiv:2312.0443 049955, 2023.
[32] Wenqi Shao, Mengzhao Chen, Zhaoyang Zhang, Peng Xu, Lirui Zhao, Zhiqian Li, Kaipeng Zhang, Peng Gao, Yu Qiao, and Ping Luo. Omniquant: Omnidirectionally calibrated quantiza- tion for large language models. arXiv preprini arXiv:2308. /3137, 2023.
[33] Ying Sheng, Lianmin Zheng, Binhang Yuan, Zhuohan Li, Max Ryabinin, Beidi Chen, Percy Liang, Christopher Ré, Ion Stoica, and Ce Zhang. Flexgen: High-throughput generative inference large language models with a single gpu. In International Conference on Machine Learning, pages 11094-31116 PMLR, 2023.
12

---

[34] Mirac Suzgun, Nathan Scales, Nathanael Schärli, Sebastian Gehrmann, Yi Tay, Hyung Won Chung, Aakanksha Chowdhery, Quoc Le, Ed Chi, Denny Zhou, and Jason Wei. Challenging BIG-bench tasks and whether thain-of-thought can solve them. In Anna Rogers, Jordan Boyd- Graber, and Naoaki Okazaki, editors, Findings of the Association for Computational Linguistics: ACI. 2023, pages 13001, Toronto, Canada, July 2023. Association for Computational Linguistics. doi: 10.18653/v wiw.0/2...ninnnssaclllllll 2023. fiidininnnnal.....
URL
org/
[35] Jiaming Tang, Yilong Zhao, Kan Zhu, Guangxuan Xiao, Baris Kasikci, and Song Han. Quest: Query-aware sparsity for efficient long-context IIm inference. arXiv preprint arXiv:2406. 10774, 2024.
[36] Phi Team et al. Phi-3 technical report: A highly capable language model locally on your phone. arXiv preprint arXiv:2404.1422 2024.
[37] Zhongwei Wan, Xinjian Wu, Yu Zhang, Yi Xin, Chaofan Tao, Zhihong Zhu, Xin Wang, Siqi Luo, Jing Xiong, and Mi Zhang. D2o: Dynamic discriminative operations for efficient generative inference of large language models. arXiv preprint 24X0::24006.33355, 2024.
[38] Yumeng Wang and Zhenyang Xiao. Loma: Lossless compressed memory attention. arXiv preprint arXiv:2401.088 109466, 2024.
[39] Zheng Wang, Boxiao Jin, Zhongzhi Yu, and Minjia Zhang. Model tells you where to merge: Adaptive kv cache merging for Ilms on long-context tasks. arXiv preprint vrXiv:2407.08454 2024.
[40] Chaojun Xiao, Pengle Zhang, Xu Han, Guangxuan Xiao, Yankai Lin, Zhengyan Zhang, Zhiyuan Liu, and Maosong Sun. Inflim: Training-free long-context extrapolation for Ilms with an efficient context memory, arXiv preprint arXiv:2402.04617, 2024.
[41] Guangxuan Xiao, Yuandong Tian, Beidi Chen, Song Han, and Mike Lewis. Efficient streaming language models with attention sinks. arXiv preprint arXiv:2309. /7453, 2023.
[42] An Yang, Baosong Yang, Binyuan Hui, Bo Zheng, Bowen Yu, Chang Zhou, Chengpeng Li, Chengyuan Li, Dayiheng Liu, Fei Huang, Guanting Dong, Haoran Wei. Huan Lin, Jialong Tang, Jialin Wang, Jian Yang. Jianhong Tu, Jianwei Zhang, Jiamxin Ma, Jianxin Yang, Jin Xu, Jingren Zhou, Jinze Bai, Jinzheng He, Junyang Lin, Kai Dang, Keming Lu, Keqin Chen, Kexin Yang, Mei Li, Mingfeng Xue, Na Ni, Pei Zhang, Peng Wang. Ru Peng, Rui Men, Ruize Gao, Runji Lin, Shijie Wang, Shuai Bai, Sinan Tan, Tianhang Zhu, Tianhao Li, Tianyu Liu, Wenbin Ge, Xiaodong Deng, Xiaohuan Zhou, Xingzhang Ren, Xinyu Zhang, Xipin Wei, Xuancheng Ren, Xuejing Liu, Yang Fan, Yang Yao, Yichang Zhang, Yu Wan, Yunfei Chu, Yugiong Liu, Zeyu Cui, Zhenru Zhang, Zhifang Guo, and Zhihao Fan. Qwen2 technical report, 2024. URL. htaps:://rrxv.... org/ abs/ abs//aa///0// 2407 10671
[43] Dongjie Yang, XiaoDong Han, Yan Gao, Yao Hu, Shilin Zhang, and Hai Zhao. Pyramid- infer: Pyramid kv cache compression for thgh-throughput 11m inference. arXiv preprint arXiv:2405. 12532, 2024.
[44] June Yong Yang, Byeongwook Kim, Jeongin Bae, Beomseok Kwon, Gunho Park, Eunho Yang, Se Jung Kwon, and Dongsoo Lee. No token left behind: Reliable ky cache compression via importance-awe mixed precision quantization. arXiv preprint arXiv:2402.18096 2024.
[45] Yifei Yang, Zouying Cao, Qiguang Chen, Libo Qin, Dongjie Yang, Hai Zhao, and Zhi Chen. Kysharer: Efficient inference via layer-wise dissimilar kv cache sharing. arXiv preprint arXiv:2410.185/1 18517, 2024.
[46] Yuxuan Yue. Zhihang Yuan, Haojie Duanmu, Sifan Zhou, Jianlong Wu, and Liqiang Nie. Wkvquant: Quantizing weight and key/value cache for large language models gains more. arXiv preprint 24X. 2024.
[47] Zihao Zeng Bokai Lin, Tianqi Hou, Hao Zhang, and Zhijie Deng. In-context kv-cache eviction for Imms via attention-gate. arXiv preprint 220.:224100.276, 2024.
13

---

[48] Hailin Zhang, Xiaodong Ji. Yilin Chen, Fangcheng Fu, Xupeng Miao, Xiaonan Nie, Weipeng Chen, and Bin Cui. Pqeache: Product quantization-bast based kveache for long context Ilm inference. arXiv preprini arXiv:2407. 12820, 2024.
[49] Yanqi Zhang, Yuwei Hu, Runyuan Zhao, John Lui, and Haibo Chen. Unifying kv cache compression for large language models with leankv. arXiv preprint arXiv:2412.03111, 2024.
[50] Yuxin Zhang, Yuxuan Du, Gen Luo, Yunshan Zhong. Zhenyu Zhang, Shiwei Liu, and Rongrong Ji. Cam: Cache merging for memory-efficien Ilms inference. In Forty-first International Conference on Machine Learning, 2024.
[51] Zhenyu Zhang, Ying Sheng, Tianyi Zhou, Tianlong Chen, Lianmin Zheng, Ruisi Cai, Zhao Song, Yuandong Tian, Christopher Ré, Clark Barrett, et al. H2o: Heavy-hitter oracle for efficient generative inference of large language models. Advances in Neural Information Processing Systems, 36:34661-3477 1-47100, 2023.
[52] Junqi Zhao, Zhijin Fang. Shu Li, Shaohui Yang, and Shichao He. Buzz: Beehive-structured sparse kv cache with segmented heavy hitters for efficient IIm inference. arXiv preprint arXiv:2410.23079, 2024.
[53] Yao Zhao, Zhitian Xie, Chen Liang, Chenyi Zhuang, and Jinjie Gu. Lookahead: An inference acceleration framework for large language model with lossless generation accuracy. In Proceed- ings of the 30th ACM SIGKDD Conference on Knowledge Discovery and Data Mining, pages 6344 6355, 2024.
[54] Lianmin Zheng, Wei-Lin Chiang, Ying Sheng, Siyuan Zhuang, Zhanghao Wu, Yonghao Zhuang. Zi Lin, Zhuohan Li, Dacheng Li, Eric P. Xing, Hao Zhang, Joseph E. Gonzalez, and Ion Stoica. Judging as--ase- with mt-bench and chatbot arena, 2023. httpss///axx org/ abs/2306 05685
[55] Meizhi Zhong, Xikai Liu, Chen Zhang, Yikun Lei, Yan Gao, Yao Hu, Kehai Chen, and Min Zhang. Zigzagkv: Dynamic cache compression for long-context modeling based on layer uncertainty, arXiv preprint arXh::2412.0000 2024.
14

---

NeurIPS Paper Checklist
1. Claims
Question: Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?
Answer: [Yes]
Justification: The contributions and scope of the paper are included in the abstract and introduction (Section I
Guidelines:
The answer NA means that the abstract and introduction do not include the claims made the paper.
The abstract and/or introduction should clearly state the claims made, including the contributions made in the paper and important assumptions and limitations. A No or
NA answer to this question will not be perceived well by the reviewers.
The claims made should match theoretical and experimental results, and reflect how much the results can be expected to generalize to other settings.
is fine to include aspirational goals as motivation as long as is clear that these goals not attained by the paper.
2. Limitations
Question: Does the paper discuss the limitations the work performed by the authors?
Answer: [Yes]
Justification:
The limitations are included in the Appendix E
Guidelines:
The answer NA means that the paper has no limitation while the answer No means that the paper has limitations, but those are not discussed in the paper.
The authors are encouraged to create a separate "Limitations" section their paper.
The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (e.g., independence assumptions, noiseless settings, model well-specification, asymptotic approximations only holding locally). The authors should reflect how these assumptions might be violated in practice and what the implications would be.
The authors should reflect on the scope of the claims made, e.g., if the approach was only tested on few datasets or with few runs. In general, empirical results often depend on implicit assumptions, which should be articulated.
The authors should reflect on the factors that influence the performance of the approach. For example, facial recognition algorithm may perform poorly when image resolution is low images are taken in low lighting. Or speech-to-text system might not be used reliably to provide closed captions for online lectures because it fails to handle technical jargon.
The authors should discuss the computational efficiency of the proposed algorithms and how they scale with dataset size.
If applicable, the authors should discuss possible limitations of their approach to address problems of privacy and fairness.
While the authors might fear that complete honesty about limitations might used by reviewers as grounds for rejection, worse outcome might be that reviewers discover limitations that aren't 't acknowledged judgment and recognize that individual actions in favor of transparency play impor- tant role in developing norms that preserve the integrity of the community. Reviewers will be specifically instructed to not penalize honesty concerning limitations.
the paper. The authors should use their best
3. Theory assumptions and proofs
Question: For each theoretical result, does the paper provide the full set of assumptions and complete (and correct) proof?
Answer: [NA]
15

---

Justification:
There is no theoretical result in this paper.
Guidelines:
The answer NA means that the paper does not include theoretical results.
All the theorems, formulas, and proofs in the paper should be numbered and cross- referenced.
All assumptions should be clearly stated referenced in the statement of any theorems.
The proof' can either appear the main paper or the supplemental material, but they appear in the supplemental material, the authors are encouraged to provide short proof sketch to provide intuition.
Inversely, any informal proof provided in the core of the paper should be complemented by formal proofs provided in appendix or supplemental material.
Theorems and Lemmas that the proof relies upon should properly referenced.
4. Experimental result reproducibility
Question: Does the paper fully disclose all the information needed to reproduce the main ex- perimental results of the paper to the extent that affects the main claims and/or conclusions of the paper (regardless of whether the code and data are provided or not)?
Answer: [Yes]
Justification: We introduce the details of the experiment, which includes the information of hardware and software, models and datasets used, and the details of proposed method in
Section 5.1
Guidelines:
The answer NA means that the paper does not include experiments.
If the paper includes experiments, a No answer to this question will not be perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data are provided or not.
If the contribution is dataset and/or model, the authors should describe the steps taken make their results reproducible or verifiable.
Depending on the contribution, reproducibility can be accomplished in various ways. For example, if the contribution is novel architecture, describing the architecture fully might suffice, or if the contribution a specific model and empirical evaluation, it may be necessary either make it possible for others to replicate the model with the same dataset, or provide access to the model. In general. releasing code and data is often one good way accomplish this, but reproducibility can also be provided via detailed instructions for how to replicate the results, access to a hosted model (e.g., in the case of large language model), releasing of a model checkpoint, or other means that appropriate to the research performed.
While NeurIPS does not require releasing code, the conference does require all submis- sions to provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution. For example
(a) the contribution is primarily a new algorithm, the paper should make clear how reproduce that algorithm. (b) If the contribution is primarily new model architecture, the paper should describe the architecture clearly and fully.
(c) the contribution is a new model (e.g., a large language model), then there should either be way to access this model for reproducing the results or way to reproduce the model (e.g., with an open-source dataset instructions for how to construct the dataset).
(d) We recognize that reproducibility may be tricky in some cases, in which case authors are welcome deseribe the particular way they provide for reproducibility. In the case of closed-source models, it may be that access to the model is limited in some way (e.g., to registered users), but it should possible for other researchers to have some path to reproducing or verifying the results.
5. Open access to data and code
16

---

Does the paper provide open access to the data and code, with sufficient instruc- tions faithfully reproduce the main experimental results, as described in supplemental material?
Question:
Answer: [Yes]
Justification: We have attached the data and code used in this paper in the supplementary material.
Guidelines:
The answer NA means that paper does not include experiments requiring code.
Please see the NeurlPS code and data submission guidelines (https://niis.cc)
cublicc/uudees/Codeuubbsss wtw.ooeeeesuoiiiiooooooooiooiieo for more details.
While encourage the release of code and data, we understand that this might not be possible, so "No" is an acceptable answer. Papers cannot be rejected simply for not including code, unless this is central to the contribution (e.g., for new open-source benchmark).
The instructions should contain the exact command and environment needed to run to reproduce the results. See the NeurlPS code and data submission guidelines (https
/ wwtps/cccppllggggiiieeeeeec
Ceaeee/CCSSSuSbissS..S.. wsw.sssoiccgyyy for more details.
The authors should provide instructions on data access and preparation, including how to access the raw data, preprocessed data, intermediate data, and generated data, etc.
The authors should provide scripts to reproduce all experimental results for the proposed method and baselines. only subset of experiments are reproducible, they should state which ones are omitted from the script and why.
At submission time, to preserve anonymity, the authors should release anonymized versions (if applicable).
Providing as much information as possible in supplemental material (appended the paper) is recommended, but including URLs to data and code is permitted.
6. Experimental setting/details
Question: Does the paper specify all the training and test details (e.g., data splits, hyper- parameters, how they were chosen, type of optimizer, etc.) necessary to understand the results?
Answer: [Yes]
Justification:
We provide the details of the experimental settings in Section 5.1
Guidelines:
The answer NA means that the paper does not include experiments.
The experimental setting should be presented in the core of the paper to a level of detail that is necessary to appreciate the results and make sense of them.
The full details can be provided either with the code, in appendix, as supplemental material.
7. Experiment statistical significance
Does the report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?
Question:
Answer: [Yes]
Justification: For the main experiments, the results are stable because we use greedy decoding of LLM. For the observation experiments, we report error bars suitably in Figure (b).
Guidelines:
The answer NA means that the paper does not include experiments.
The authors should answer "Yes" if the results are accompanied by emoor bars, confi- dence intervals, or statistical significance tests, least for the experiments that support the main claims of the paper.
17

---

The factors of variability that the error bars are capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, or overall run with given experimental conditions).
The method for calculating the error bars should be explained (closed form formula, call to a library function, bootstrap, etc.)
The assumptions made should be given (e.g., Normally distributed errors).
It should be clear whether the error bar is the standard deviation or the standard error of the mean.
is OK to report 1-sigma error bars, but one should state it. The authors should preferably report a 2-sigma error bar than state that they have 96% CI, if the hypothesis of Normality of errors is not verified.
For asymmetric distributions, the authors should be careful not to show in tables or figures symmetric error bars that would yield results that are out range (e.g. negative еπτог rates).
If error bars are reported in tables or plots, The authors should explain in the text how they were calculated and reference the corresponding figures or tables text.
8. Experiments compute resources
For each experiment, does the paper provide sufficient information the com- puter resources (type of compute workers, memory, time of execution) needed to reproduce the experiments?
Question:
Answer: [Yes]
we report the details of compute resources. In Section 5.3 we evaluate the efficiency of the proposed method including average time per output token in decode stage, time to first token and end-to-end throughput.
Justification: In Section 5.1
Guidelines:
The answer NA means that the paper does include experiments.
The paper should indicate the type of compute workers CPU or GPU, internal cluster, or cloud provider, including relevant memory and storage.
The paper should provide the amount of compute required for each of the individual experimental runs as well as estimate the total compute.
The paper should disclose whether the full research project required more compute than the experiments reported the paper (e.g., preliminary or failed experiments that didn't It make into the paper).
9. Code of ethics
Question: Does the research conducted the paper conform, in every respect, with the NeurlPS Code of Ethics
Answer: [Yes]
Justification:
We have made sure that our paper conforms with the NeurlPS Code of Ethics.
Guidelines:
The answer NA means that the authors have reviewed the NeurIPS Code of Ethics.
the authors answer No, they should explain the special circumstances that require deviation from the Code of Ethics.
The authors should make to preserve anonymity (e.g., if there is special consid- eration due to laws or regulations in their jurisdiction).
10. Broader impacts
Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?
Answer: [Yes]
Justification:
We discuss the potential positive impacts proposed method in Section [
Guidelines:
The answer NA means that there is no societal impact of the work performed.
18

---

If the authors answer NA or No, they should explain why their work has no societal impact or why the paper does not address societal impact.
Examples of negative societal impacts include potential malicious or unintended uses (e.g., disinformation, generating fake profiles, surveillance), fairness considerations (e.g., deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations.
The conference expects that many papers will be foundational research and not tied to particular applications, let alone deployments. However, if there is direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point out that a generic algorithm for optimizing neural networks could enable people train models that generate Deepfakes faster.
The authors should consider possible harms that could arise when the technology is being used as intended and functioning correctly, harms that could arise when the technology is being used intended but gives incorrect results, and harms following from (intentional or unintentional) misuse of the technology.
If there are negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition attacks, mechanisms for monitoring misuse, mechanisms to monitor how system learns from feedback over time, improving the efficiency and accessibility of ML).
11. Safeguards
Does the paper describe safeguards that have been put in place for responsible release of data models that have high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)?
Question:
Answer: [NA]
Justification: There is no risk of misuse of the proposed method because its purpose is to accelerate the large language model.
Guidelines:
The answer NA means that the paper poses no such risks.
Released models that have high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model. for example by requiring that users adhere to usage guidelines or restrictions to access the model or implementing safety filters.
Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images.
recognize that providing effective safeguards challenging, and many papers do not require this, but we encourage authors to take this into account and make best faith effort.
12. Licenses for existing assets
Are the creators or original owners of assets (e.g., code, data, models), used in the paper, properly credited and are the license and terms of use explicitly mentioned and properly respected?
Question:
Answer:
[Yes]
Justification: We have cited the original paper or attached the link to the existing assets used
in Section 5.1
Guidelines:
The answer NA means that the paper does not use existing assets.
The authors should cite the original paper that produced the code package or dataset.
The authors should state which version of the asset is used and, if possible, include URL.
The name of the license (c.g., CC-BY 4.0) should be included for each asset.
19

---

For scraped data from a particular source (e.g., website). the copyright and terms of service of that souree should be provided.
assets are released, the license, copyright information, and terms use in the datasets.paperssiii paperswi thcode. www.coo//aaaaa datasets package should be provided. For popular datasets,
has curated licenses for some datasets. Their licensing guide can help determine the license of dataset.
For existing datasets that are re-packaged, both the original license and the license of the derived asset (if it has changed) should be provided.
If this information is not available online, the authors are encouraged to reach out to the asset's s creators.
13. New assets
Question: Are new assets introduced in the paper well documented and is the documentation provided alongside the assets?
Answer: [Yes]
Justification:
We have attached the introduction for running code the code repository.
Guidelines:
The answer NA means that the paper does not release new assets.
Researchers should communicate the details of the dataset/code/moodl as part of their submissions via structured templates. This includes details about training, license, limitations, etc.
The paper should discuss whether and how consent obtained from people whose asset is used.
At submission time, remember to anonymize your assets (if applicable). You can either create an anonymized URL or include an anonymized zip file.
14. Crowdsourcing and research with human subjects
Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given participants and screenshots, if applicable, as well as details about compensation (if any)?
Answer: [NA]
Justification:
This paper does not involve crowdsourcing or research with human subjects.
Guidelines:
The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.
Including this information in the supplemental material is fine, but if the main contribu- tion of the paper involves human subjects, then as much detail as possible should be included in the main paper.
According to the NeurlPS Code of Ethics, workers involved in data collection, curation, or other labor should be paid at least the minimum wage in the country of the data collector.
15. Institutional review board (IRB) approvals equivalent for research with human subjects
Question: Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or an equivalent approval/review based on the requirements of your country or institution) were obtained?
Answer: [NA]
Justification:
This paper does not involve.
Guidelines:
The answer NA means that the paper does not involve crowdsoureing nor research with human subjects.
20

---

Depending on the country in which research is conducted, IRB approval (or equivalent) may be required for any human subjects research. If you obtained IRB approval, you should clearly state this in the paper.
We recognize that the procedures for this may vary significantly between institutions and locations, and we expect authors to adhere to the NeurIPS Code of Ethics and the guidelines for their institution.
For initial submissions, do not include any information that would break anonymity (if' applicable), such as the institution conducting the review.
16. Declaration of LLM usage
Does the paper describe the usage of LLMs if is an important, original, or non-standard component of the core methods in this research? Note that if the LLM is used only for writing, editing, or formatting purposes and does not impact the core methodology, scientific rigorousness, originality of the research, declaration is not required.
Question:
Answer: [Yes]
Justification:
We describe the usage of LLMs in Section 5.1
Guidelines:
The answer NA means that the core method development in this research does not involve LLMs as any important, original, or non-standard components.
Please refer to our LLM policy for what should or should not be described.
21

---

A Overhead Analysis
In this section, analyze the additional overhead introduced by the SmallKV method, which mainly includes the cost of the SLM and the overhead associated with the V-specifcc processes. It is important note that although we provide comprehensive analysis of these overheads, the cost related to the SLM can be shared with the SLM used for speculative decoding in practice, which will significantly reduce the actual additional overhead.
A.1 Memory Overhead
The overhead of GPU memory in V comes mainly from the assisted SLM and the KV cache it stores. The storage requirement for SLM parameters is fixed and depends the size of SLM. For KV cache consumption, it can be formulated as:
MR R 2- L- NKv DE: S. B-B
Where Mk is the total GPU memory usage (in bytes) for storing KV caches. L represents the number of layers in the model. Ngo denotes the number of key-value heads used in the attention mechanism. DAu refers to the dimension size of each KV entry. S is the sequence length, indicating the number of tokens in the sequence. B stands for the batch size, which is the number of sequences processed simultaneously. C. is the number of bytes per cached value, depending on the data type used.
Qwen2-7B the SLM and Dwen2-72B as the LLM for illustrative analysis. The parameters for both models are outlined in the Table 3 Specifically, Nat denotes the number of attention heads, and Dú, represents the hidden state dimension. Given these parameters and applying the Equation 7 we can calculate that under the same batch size and sequence length, the KV cache size of Qwen2-725 72B is approximately 5.71 times that of Qwen2-7B. Therefore, if the KV cache budget for the LLM (Qwen2-744) 22BB) is set to 20%, the additional overhead introduced by the SLM (Qwen2-7B) would consume about 21.9% of the KV cache saved by reducing the LLM's KV cache consumption.
Here,
also important to note that, the analysis presented here, no optimizations have been applied to the SLM's KV cache. In practice, the SLM's KV cache can also be optimized through various methods, such as quantization and eviction. Implementing these optimizations can further reduce the KV cache overhead associated with the SLM.
Table 3: The config parameters of Qwen2-7B and Qwen2-72B.
Qwen2-7B
Qwen2-72B
L
28
80
Nko 4
8
Natt
28
64
Dp
3584
8192
Deo 128
128
A.2 Computational overhead
The additional computational overhead in SmallKV can be divided into two main parts: forward process of the SLM and similarity matching computation between SLM and LLM.
For forward process of the SLM, it affects both prefill and decode stage. For LLM with L decoder layers, the total computational cost for one batch during a forward pass is:
Crnodet = L(2ABSD) 4BS² Dz) V & 24BSLD
Where VV] denotes vocabulary size. It also can be calculated that under the same batch size and sequence length, the computational cost of LLM (Qwen2-72B) is approximately 14.9 times that of SLM (Qwen2-7B) referring to T
For the similarity matching computation between SLM and LLM, it only affects prefill stage. Assume that Seim denotes the sequence length for calculate similarity (typically 100-200 SmalIK we have the computational cost of the similarity matching for model and model b as:
22

---

Csim = NL att + ) sim + La S ~ N Lte L' L B²
(9)
att
eim
Where L and L denote the number of layers in the model and model b. Nat and Na number of attention heads respectively.
denote the
att
A.3 Quantitative Analysis and Optimization of SLM
We quantify the theoretical maximum latency and memory overhead of the model combinations used in main experiments, shown in the Table + The values 'Ratio' represent the additional overhead introduced by SLM relative to LLM (e.g., for combination Qwen2-0.5B & 7B, the KV cache of SLM is 1/4.67 that of the LLM).
Table 4: Quantitative analysis of SLM relative to LLM in latency and KV cache usage
Ratio between SLM and LLM
Qwen2-0.5B & Qwen2-7B
Qwen2.5-0.5B & Qwen2.5-14B
Qwen2-7B & Qwen2-72B
LLAMA 3,2-1B & LLAMA 3.1-8B
Latency KV cache
1/3.57
1/6.52
1/4.47
1/3.19
1/4.67
1/16.0
1/5.71
1/4.0
We propose three deployment-level optimizations effectively reduce the computational and memory overhead of the SLM, enhancing the practicality of SmallK (1) integration with speculative decoding, (2) compression the KV cache for the SLM, and (3) layer skipping (i.e., early stopping) during the attention mapping process.
(1) The integration of speculative decoding eliminate the forward pass of the SLM, significantly reduce latency introduced by the SLM. Current speculative decoding algorithms yield 3-5x speedups, which would further enhance performance.
(2) The high attention sparsity inherent in SLM enables loss-lossless reduction in memory usage through SLM KV cache compression. As shown in the Table 5 we fix the KV cache budget of the LLM (Qwen2-7B) at 20% and increase the compression ratio of the SLM (Qwen2-0.5B) to evaluate the LLM's accuracy on BBH (SmallKV ACC = 0.436 at 20% KV cache budget). Under the same memory constraint (for SmallKV we count both SLM and LLM cache), the performance SmallKV (ACC 0.462 40% KV cache budget for SLM and 20% for LLM) outperforms H2O (ACC 0.38 at 30% KV cache budget). Note that while SLM compression may slightly impair saliency shift compensation, the trade-off remains favorable in terms of overall efficiency.
(3) In an independently deployed scenario, the SLM is not required generate draft tokens, enabling the adoption of an early layer stopping strategy. This approach significantly reduces both latency and memory overhead. We fix the LLM's KV cache budget 20% and progressively reduce the number of layers utilized in the SLM for attention mapping. The results shown in Table iindicate that stopping at layer 20 (83% total layers) incurs no loss in accuracy (0.436). Furthermore, even when halting at layer 16 (67%), the accuracy of SmallKV remains superior to that of H20.
Table 5: Accuracy of L.LM under different KV cache budgets of SLM
KV Cache Budget of SLM
20%
40%
60%
80%
Full
Accuracy (BBH)
0.2873
0.4416
0.4624
0.4642
0.4359
Table 6: Accuracy of LLM under different stop layers of SLM attention mapping
Stop Layer of SLM 24 (Full)
20
12
H30
Accuracy (BBH)
0.436
0.436
0.390
0.359
0.335
0.318
Based on the implementation optimization of (2) and (3), the SLM's actual KV cache usage is calculated as: (1 (467)) 40% (SIMM compression) x (20 24) (early stopping) R 7.14% of the full LLM KV cache.
23

---

B Extended Discussion of Attention Similarity between Different-scale LLMs
The attention similarities between different-scale LLMs within same series are from our empirical observations. We first present further quantified results regarding attention similarity. Specifically, on the first 200 contexts from the Wikitext-v2 v2 dataset, record the cosine similarity between each attention head of the LLM (Qwen2-7B) and its corresponding attention matrices in SLM (Qwen2- 0.5B), which is then illustrated of the attention heads of LLM find very similar counterparts in the SLM across different contexts. Statistical analysis of the heatmap shows that 78.22% of the elements have values exceeding 0.9, 96.21 % the elements exceed 0.8, and only 0.02% have similarities lower than 0.5.
heatmap in Figure As be seen from the figure, the majority
We speculate that there are many factors that cause this similarity between different-scale LLMs within same series: 1) Although they differences in configuration parameters such as the number layers and hidden dimensions, they share the same model architecture. 2) They are pre-trained on the similar corpus [42]. 3) Some smaller models are distilled from larger models [36]. The aforementioned factors contribute different-scale LLMs learning similar semantic space representations at different level in the shallow decode layers. For deeper layers, previous rescarches (43 [3] have indicated that LLMs exhibit more attention redundancy, meaning these layers tend have simpler attention
patterns that are easy to match for the SLM, which can be evidenced by the lighter color in latter part of Figure
Similarity Heatmap
1.0
0.9
0.8
LLMAt Head
0.7
0.6
0.6
0.4
0.3
0.2
Context Index
Figure 8: Similarity heatmap between Qwen2-0.5D and Qwen2-7B on 200 Contexts from Wikitext-v2. The x-axis represents different contexts, and the y-axis represents the attention heads of the LLM (Qwen2-7B). Each cell's color intensity indicates the cosine similarity of accumulated attention scores between the corresponding attention matrices of the SLM (Qwen2-0.5B) and the LLM.
To analyze the attention similarity between different SLMs and LLMs, first visualize the attention similarity among Qwen2-7B, Qwen2-0.5B, and Llama3-1B on the Wikitext-2 dataset (as shown in Figure 9). where the y-axis represents the cumulative attention score per token across all attention heads. Results show that, due to similar training corpora and linguistic priors, models exhibit certain commonalities in their attention patterns. However, the cosine similarity between Qwen2-7B and Qwen2-0.5B reaches 0.9622, while that between Qwen2-7B and Llama3 1B is only 0.6917. This indicates that attention similarity is significantly higher within the same series models.
We further conduct a comprehensive analysis of attention pattern similarity between SL.Ms and L.MMs across diverse models, scales, benchmarks, and multi-turn dialogue scenarios. Due attention sinks,

---

Accumulated Attention Score
2.00
Qwen2-78 Attention
Qwen2-0.55 0.5 Attention
1.75
Llama3-11 Attention
1.25
1.00
0.75
0.50
0.25
0
20
30
50
60
80
Token Index
Figure 9: Accumulated attention scores of Qwen2-7B, Qwen2-0.5B, and Llama3-1B 1B (Selecting random segment from the middle the tokens to avoid attention sink).
removed the first few tokens when computing similarities to avoid too high similarity (e.g.. 0.99),
ensuring meaningful and comparable results, The results are represented in Table and Table g
Table Similarity between SLM and LLM across different settings (Part 1)
SLM
Qwen2-0.5B
Qwen2-0.55
Qwen2-0.5B
Qwen5-0.68
Qwen2.5-0.58
Qwea2.5-1.58
LLM
Qwen2-7B
Qwen2-7B
Qwen2-7B
Qwea3-8B
Qwen2.5-14B
Qwea2.5-14BB
Benchmark
BBH
GSM8K
MMLU
BBH
BBH
BBH
Similarity
0.85
0.85
0.98
0.83
0.83
0.83
Table Similarity between SLM and LLM across different settings (Continued)
SLM
Qwen2.5-3B
Qwen2.7B
Qwen2.5.7B
LlaMA3-2.1B
LLaMA3.2
LLM
Qwen2.5-14B
Qwen2-72B
Qwen2.5-14B
Qwen2-7B
LLaMA3.1-8B
Benchmark
BBH
BBH
BBH
BBH
BBH
Similarity 0.85
0.81
0.86
0.56
0.98
We evaluate the changes in similarity during multi-turn dialogue scenarios, and the results are shown These findings demonstrate the scalability attention similarities to a considerable extent.
in Table g
C Extended Analysis of Marginal Information Compensation
The effectiveness of the marginal information compensation mechanism primarily originates from two aspects: high cosine similarity constraint and low attention scores constraint. The high cosine similarity constraint derived from our observations on attention similarity detailed in Appendix ensuring that attention patterns between the LLM and SLM are closely aligned. Meanwhile, the low attention scores constraint focuses on replacing only those tokens with relatively lower attention scores compared to critical tokens, based the rationale that these lower-scoring tokens contribute less significantly to representation. Here we give derivation of the low errors of attention outputs under these two constraints:
Given an original attention vector E R", representing a row from the attention matrix, we replace with à I" The high cosine similarity constraint is defined by ensuring that the cosine similarity between a and à is very close:
a T
cos(O)
2 1 1-5,
where & 1.
||||||a|
25

---

Table 9: Attention similarity on multi turn dialogues.
global
turn1
turn2
turn3
turn4
turn5
SmallKV
0.822
0.812
0.799
0.796
0.782
Additionally, low attention scores constraint modify elements of a that are small, denoted as subset S C n.......,... The perturbation can be modeled as:
+ E,
where is the perturbation vector. Given the high cosine similarity constraint, mainly aligns along or close to a's direction:
0x0 0,
di.
E S $ S
Here, a E represents the scaling factor along o, and de is small perturbation orthogonal to a, for i S. The change in output due to the perturbation is:
ô o (a
a)V eV
Therefore,
1 ol12 = ||eV|2
Substituting e; = aa, 8; for i yields:
||ō - o|12 =
Σ SES
(aaa + 5,)0%
(aa; + (0))211u3112 +2 Σ icjES
I(aa; + 6:)(aa, + 68,0² usl
iS
we assume
||o:|| is bounded, then:
|| a1?2 < D(aa, + 8.)2
iES
Since we only modify elements a, where E S, if assume that the perturbation proportional to 0;- have:
Ele, = o²a f
Thus, the total expected squared error is:
E
[
- Σ o²
iES
Notice that:
Σ a. = 1,
but
a <
1=1
XES
Because a; is small when i E S, a is even smaller, so:
Σ < [| @
The impact on the final output is limited by:
B|llö oll S Σ d' 0
iES
Therefore, under the high cosine similarity constraint, replacing low attention scores has a minimal impact the model's output.
26

---

D More Implementation Details
We show a simplified pseudocode below to demonstrate the implementation of SmallKV. The function Smal1KV forward () consists of both SLM forward process and LLM forward process. In prefill stage, they can be parallelized to achieve speedup because there is no coupling. After prefilling and obtaining attention matrices in each model, the function establish_sim_mattt sim_matchh () is used to calculate similarity and record matching relations for subsequent steps. During the decoding, the SLM first performs its forward and updates its attention matrices. Subsequently, the LLM executes forward pass. parallel with this process, the KV cache of the LLM updated based on the updated attention matrices from the SLM, which is handled by the function update kv (). In the forward of wettattetiinnoorreeeee
LLM, the standard attention forward is replaced by the function Within this function, two key operations are performed: saliency shift compensation and marginal information compensation.
Listing I: SmallKV implementation pseudocode
def SmallKV_forward (...) if prefill: # check the token lenth for similarity matching token_lenth_ retuumeeeeeeeeeeeeeeeeeeeeeeee ():
parallel prefill for sim and 11m slm_logits, sim_attn, slm_kv,
chreading((argge=soorraa args= return_attn=True)
1lm_logits, 1lm_attn, 11m_kv,
chreadig((arget=foore
args=(11m, return_attn=True)
establish similarity matching sim_match
mat.chsll_aata.. llm_attn)
replace attention of the ilm to Smallky attention
11m. www.aatteiion.orrte forvard SmallKV SmallKy_ateention att..atttniionoorraa return 11m_logit
else:
slm_attn,
forward return_attn True)
Parallelly update XV cache (traddin(targetuuudaaa.. lim_logits, 11m_kv fo.forrard return_atta False)
args= slm_attn, ...))
return 11m_logits
def establish secablish.iinmmatccc111111 (ala_atta 1lm_attn, ...): t 1lm_attn)
sim calculate_pairvise_.eiee similaritt (elm_attn, return sim max (rettrn,innde)
def ( sim_match, slm_attn, kv_cache_budges ...):
com_attn_slm. select sim_match, layer_idx)
selected__V_cache select inder=com_att__sll.. sum topk)
marg_V_cach. select seceeseeecccceeeeeee....... selected_V_cache sub_ Sub_V_ceat v_cacee)
moox=com_attn_eee sum. top(n-2 (n-k)
sffload_CPU selected___eeeeeeeeeeee K_cche (tr
selected_ sub_ sucche
def ShallWWWatttatiio_fforraaat (layer_idxx ...):
sallencc_.compensttiooeeee
flashaattentionn(uuerry stwpette_seeec_e___c_e selected_K_cachat selected_ __cchee, ...)
com_attn_sln select sim sim_match, layer_idx)
bargimal..commmsaiioo.....t matmul (com_attn_eil sub Sub_V_cache)
return aaiiencyccomemaaaion...t + conginallcoppenaaiioo..e.
E
Limitation
There also some limitations in our approach. First, the similarity of attention patterns between the LLM and SLM cannot be guaranteed with absolute precision, which means that our method is not lossless on LLM performance. Besides, although we have analyzed potential reasons of attention similarity, we are currently unable to provide a formal theoretical proof to rigorously support these observations. Lastly, while the assisted SLM introduced by SmallKV can share overhead with speculative decoding, this additional cost still cannot be ignored when SmallKV works alone.
27

---

Case Study
Given the same prompt text in BBT Test, we visualize the output generated by Qwen2-7B under different KV cache compression methods, including the baseline of full cache, HzO [51] Pyramid- Infer 43 and our SmallKV. The results are shown in Figure TO] With only 10% of the KV cache retained, SmallKV produces outputs that are highly similar to those generated with the full cache, both follow the "Let's think step by step." instruction and calculates according to the steps provided in the prompt examples. In contrast, H2O and PyramidInfer lose their ability to follow instructions and learn from the given reasoning patterns due to compression. Even worse, they invente new words not present in the original task description during the alphabetical ordering task (c.g., "aspect"). Moreover, H2O produces meaningless repeated content at the end of its answer.
28

---

Model Input
Please reference the following examples to answer the question.
Q: Sort the following words alphabetically: List: oven costume counterpart
A: Let's think step by step.
The first letter: "oven": "o" (15). "costume": "c" (3). "counterpart": "c" (3). We now have: (3) [" ["costume" ?
"counterpart"] < (15) "oven". Now let's sort this subpart ("costume" "counterpart") by looking at their sccond letters.
The socond letter: "costume": "o" (15). "counterpart": "o" (15). We now have: (15) ["costume"?
"counterpart). Now let's sort this subpart ["costume"! "counterpart"] by looking at their third letters.
The third letter: "costume": "s" (19). "counterpart": "u" (21). We now have: (19) "costume" < (21) "counterpart,, Hence, we have ("costume" "counterpart"] "oven". So the answer is costume counterpart
oven.
Question: Sort the following words alphabeticully: List: thrill splutter punicking scorch sume dot prod obstetric malton onus drumhead delmarva barn embezzle it&t damp guru subsist entirety greene
Qwen2-7B Full Cache Ouiput
Let's think step by step.
The first letter: "thrill": "t" (20). "splutter": "s" (19). "panicking": "p" 1" (16). "scorch": "s" (19). "same": "s"
(19). "dot": "d" (4), "prod"; "p" (16). "ohstetric": "o"((1) (15). "malton": "m" (13), "onus": "o" (15).
"drumhead"; "d" (4). "delmarva": "d" (4), "bam": (2). "embezzle": "c" (5). "it&t": "I" (9). "damp": (4). "guru": "g" (7). "subsist": "s" (19). "entirety": "c" (5). "greene": "g" (7). now have: (2) ["barn"?
"dot") < (4) ["delmarva"? "drumhead"] (5) ["emhezzle"? "it&r") < (7) ["guru"? "greene"] < (9) "it&t"
(L1) ("obstetric"" "onus
Qwen2-7B H2O 10% Cache Output
1. "asaspett" *a" (1), "bar": "b" (2). "census": "c" (3). "costume": "c" (3), "credulity": "c" (3). "drumhead":
"d" (4). "embezzle": "e" (S). "entirety": "e" (5). "greene": "g" (7). "hypochlorite": "h" (11). "mugho": "m" (13). "nowt": "n" (14). "obstctrie": "o" (15). "parthia": "p" (16). "prod": "p" (16). "scorch": "s" (19). "same": *s" (19). "scismography"" "s" (19). "splutter": "s" (19). "subsist": "s" (19). "thrum": "t" (20). "tremble": "t"
(20). "tremble": "t" (20). "tremble": "f" (20). "temble": "t" (20). "tremble": "t" (20). "tremble": "t"
Qwen2-7B PyramidInfer 10% Cache Ouiput
1. 1."aspect" 2. "arson" 3. "census" 4. "drumhead" 5. "embezzle" 6. "entirety" 7. "guru" 8. "it&t" 10. "malton" 11. "mugho" 12. "panicking" 13. "prod" 14, "scorch" 15. "same" 16. "scismography" 17. "splutter" 18. "subsist" 19. "thril" 20. "turnish" 21. "universe" 22, "verity" 23. "vulgar" 24. "wagtuil" 25. "wagtail"
So the answer is arson aspect census drumhcad embezzle entirety guru it&t malton mugho panicking prod scorch same thril tarnish universe verity vulgar wagtail wagtail.
Qwen2-7B SmallKV 10% Cache Output
Let's think step by step.
The first letter: "thrill": "t" (20). "splutter": "s" (19). "panicking": ": "p" (16). "scorch": "s" (19). "same": "s"
(19). "dot": "d" (4). "prod": "p" (16). "obstetric": "o" (15). "malton": "m" (13). "onus": "o" (15). "drumhcad": *d" (4). "delmarva": "d" (4). "embezzle"; "c" (55). "it&t": "I" (9). "damp": "d" (4). "gur"": "g" (7). "subsist": *s" (19). "entircty": "c" (55). "greene": "g" (7). We now have: (4, 4, 7, 9, 13, 13, 19, 19, 20. The words sorted alphabeticully are: "dump". "dot", "entirety", "embezzle", "guru", "it&"", "obstetric", "panicking". "prod". "scorch
Figure 10: Visualized outputs of one generation example with Qwen2-7B. Results compared between the baseline model with full cache, H₂O [5T], PyramidInfer 43), and our SmallKV.
29


Reviews:
[{"review_id": "FDBaUiilhL", "reviewer": "Reviewer_7fen", "summary": "This paper introduces SmallKV, a small-model-assisted compensation framework for KV cache compression in LLMs, addressing saliency shift and marginal token over-compression issues. It leverages attention similarity between small and large models to preserve critical information and approximate marginal tokens during decoding. Experiments on GSM8K, BBH, MT-Bench, and LongBench demonstrate that SmallKV maintains high accuracy even with 5% KV cache budget, outperforming baselines like H2O and PyramidInfer. Efficiency evaluations show 1.75–2.56× throughput gains while remaining compatible with Flash Attention and speculative decoding. The method is robust across model scales (7B–72B) and series (Qwen, LLaMA), offering a practical solution for resource-constrained LLM inference.", "questions": "- How robust is the SLM compensation in multi-turn dialogues where attention patterns shift rapidly? Could you add dynamic decoding tests to verify this?\n- The current fixed definition of marginal tokens may not suit all tasks (e.g., code vs. creative writing). Would you consider task-adaptive criteria for better flexibility？\n- Your longest test is 16K tokens, but real-world applications use 128K+. How does error accumulation in SLM approximations affect output quality at such scales?\n\nThe final score depends on feedback of the rebuttal, and I would be happy to improve my final score.", "limitations": "yes", "rating": 4, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 4, "clarity": 3, "significance": 3, "originality": 3, "strengths_and_weaknesses": "Strengths:\n- The paper is well-written and easy to follow, clearly explaining the motivation, methodology, and results without unnecessary jargon.\n- Extensive experiments across multiple benchmarks and model sizes (7B–72B) convincingly demonstrate SmallKV’s superior performance, especially under aggressive KV cache budgets.\n- The proposed small-model-assisted compensation effectively addresses saliency shift and marginal token over-compression, achieving 1.75–2.56× throughput gains while maintaining accuracy, significantly outperforming prior eviction-based methods.\n\nWeaknesses:\n- While the high similarity between SLM and LLM attention matrices is empirically validated, the paper lacks rigorous theoretical analysis to explain why this consistency exists across different model scales, potentially limiting the generalizability of the approach.\n- Although the authors mention that the memory and computational costs of the SLM can be mitigated by sharing resources with speculative decoding, they do not quantify the actual overhead when SmallKV is deployed independently (e.g., increased latency or resource consumption), which may require further optimization in practice.\n- The paper demonstrates that SLM scaling significantly impacts performance under extremely low KV cache budgets (e.g., 5%), but it does not explore the feasibility of even smaller models (e.g., 0.1B), which could restrict its applicability on ultra-edge devices.", "was_revised": true, "final_justification": "The paper is clearly written, the experiments are thorough, and the rebuttal has fully addressed my concerns. Overall, this is a highly practical contribution, and I support its acceptance.", "rebuttal": "Thank you for your detailed review and valuable feedback! Below, we address the comments you raised:\n\n**Q1: How robust is the SLM compensation in multi-turn dialogues where attention patterns shift rapidly? Could you add dynamic decoding tests to verify this?**\n\n**R**: In Figure 5, we present the results of two-round dialogues over MT-Bench. The experimental results show that SmallKV significantly outperforms other baselines. To further assess the robustness of SmallKV's compensation strategy in more complex dialogue scenarios, we constructed a five-round dialogue test based on BBH test. Each round consists of questions from different subtasks. \n\nThe robustness depends on whether the attention mapping relationship between the small model (SLM) and the big model (LLM) changes during multiple rounds of prefilling and decoding. We first verify the SmallKV’s stable attention mapping on Qwen2-7B & Qwen2-0.5B. During the evaluation, we record the cosine similarity of attention patterns in the mapping dictionary constructed by SmallKV, and compare it against both the globally optimal mapping (i.e., the maximum achievable similarity) and randomly generated mappings. As summarized in the table below, the similarity achieved by SmallKV consistently stays close to the upper bound derived from the global perspective across all dialogue turns. This finding indicates that the attention mapping established by SmallKV remains robust and reliable even in extended multi-turn conversation scenarios.\n\n| |turn1|turn2|turn3|turn4|turn5|\n|-|-|-|-|-|-|\n|global|0.822|0.812|0.799|0.796|0.792|\n|SmallKV|0.822|0.809|0.792|0.791|0.787|\n|random|0.516|0.509|0.510|0.514|0.512|\n\nWe evaluate the accuracy on the five-turn dialogue BBH test. When the KV cache budget is set to 20%, SmallKV achieves an accuracy of 0.373, significantly outperforming H2O, which achieves 0.308.\n\n**Q2: The current fixed definition of marginal tokens may not suit all tasks (e.g., code vs. creative writing). Would you consider task-adaptive criteria for better flexibility?**\n\n**R**: Thanks for your insightful suggestions. We agree with you that using a fixed threshold for marginal tokens may not yield optimal results. To ensure fair comparability, in our evaluation, we adopted the same experimental parameters setting as other baselines (heavy : [recent + compensation] = 1:1). Notice that the key advantage of SmallKV lies in the small model’s ability to adaptively identify important tokens, we can dynamically define the proportions of critical tokens and marginal tokens based on the recall rate of attention scores from SLM, as illustrated in Figure 2. For example, we define the critical tokens as those accounting for the top 80% of cumulative attention scores, while marginal tokens are set to the subsequent 15% (i.e., from 80% to 95% of the cumulative sum). This approach enables effective tailoring of these proportions to different tasks.\n\n**Q3：Your longest test is 16K tokens, but real-world applications use 128K+. How does error accumulation in SLM approximations affect output quality at such scales?**\n\n**R**: Given that Qwen2 and Qwen2.5 support a maximum context length of 32K, we evaluated the impact of SLM-based approximations on output quality in extremely long contexts by padding the BBH test context with irrelevant text to reach 32K (long context) compared with original BBH test (short context). The results for Qwen2-0.5B and Qwen2-7B across different KV cache budgets are presented in the table below. Under long context scenarios (32K tokens), the accuracy of SmallKV does not degrade.\n\n|KV Cache Budget|32K BBH (long context)|Original BBH (short context)|\n|-|-|-|\n| 0.3 | 0.516| 0.466 |\n| 0.2 | 0.472 | 0.436 |\n| 0.1 | 0.343 | 0.28|\n\nIn practice, we observe that LLMs exhibit higher attention sparsity and more uniform attention patterns in long contexts—characteristics that benefit both KV cache compression and SmallKV. Consequently, the model’s performance actually improves when incorporating such long contexts.\n\n**Weakness 1: While the high similarity between SLM and LLM attention matrices is empirically validated, the paper lacks rigorous theoretical analysis to explain why this consistency exists across different model scales, potentially limiting the generalizability of the approach.**\n\n**R**: Though it is hard to provide a rigorous theoretical proof, we aim to demonstrate the similarity between SLM and LLM attention patterns across different model scales through extensive experimentation. The results of these tests are summarized in the table below. In addition to the model combinations presented in the evaluation section, we have further extended our experiments to include Qwen3 series, as well as cross-series combination between Qwen3 and Qwen2. (Note1: Because of differences of tokenizers in different series models, we compute pairwise longest common subsequences to match tokens and retain only the matched tokens for calculating cosine similarity.) (Note2: Due to the presence of attention sinks, we excluded the initial few tokens during similarity calculations to prevent artificially inflated similarity scores. (e.g. ~0.99), ensuring meaningful and comparable results.) Our results show that, although some variation exists, this similarity is consistently present across different datasets.\n\n|SLM|Qwen2-0.5B|Qwen2-0.5B|Qwen2-0.5B|Qwen3-0.6B|Qwen2.5-0.5B|Qwen2.5-1.5B|Qwen2.5-3B|Qwen2.5-7B|LLaMA3.2-1B|\n|-|-|-|-|-|-|-|-|-|-|\n|LLM|Qwen2-7B|Qwen2-7B|Qwen2-7B|Qwen3-8B|Qwen2.5-14B|Qwen2.5-14B|Qwen2.5-14B|Qwen2.5-14B|Qwen2-7B|\n|Benchmark|BBH|GSM8K|MMLU|BBH|BBH|BBH|BBH|BBH|BBH|\n|Similarity|0.85|0.85|0.98|0.83|0.83|0.83|0.85|0.86|0.56|\n\n**Weakness 2: Although the authors mention that the memory and computational costs of the SLM can be mitigated by sharing resources with speculative decoding, they do not quantify the actual overhead when SmallKV is deployed independently (e.g., increased latency or resource consumption), which may require further optimization in practice.**\n\n**R**: We quantify the theoretical maximum latency and memory overhead of the model combinations used in main experiments, as shown in the table below. The values 'Ratio' represent the additional overhead introduced by SLM relative to LLM (e.g., for combination Qwen2-0.5B & 7B, the KV cache of SLM is 1/4.67 that of the LLM).\n\n|Ratio between SLM and LLM|Qwen2-0.5B & Qwen2-7B|Qwen2.5-0.5B & Qwen2.5-14B|Qwen2-7B & Qwen2-72B|LLAMA 3.2-1B & LLAMA 3.1-8B|\n|--|--|--|--|--|\n|Latency|1/3.57|1/6.52|1/4.47|1/3.19|\n|KV cache|1/4.67|1/16.0|1/5.71|1/4.0|\n\nWe highlight the practicality of SmallKV, emphasizing how the computational overhead of the SLM can be effectively mitigated through deployment-level optimizations. To achieve this, we propose three key strategies: (1) integration with speculative decoding (SP); (2) compression of the KV cache for the SLM; and (3) layer skipping (i.e., early stopping) during the attention mapping process.\n\nRegarding latency, while the forward pass of the SLM during decoding incurs some overhead, this is mitigated by the acceleration provided through KV cache compression. As demonstrated in Section 5.3 Table 2 of the evaluation, SmallKV achieves lower latency compared to the baseline model. Additionally, the integration of SP can eliminate the forward pass of the SLM as a significant latency bottleneck. Current SP algorithms yield 3-5x speedups, further enhancing performance.\n\nRegarding memory consumption, the high attention sparsity inherent in SLM enables near-lossless reduction in memory usage through SLM KV cache compression. As shown in the table below, we fix the KV cache budget of the LLM (Qwen2-7B) at 20% and increase the compression ratio of the SLM (Qwen2-0.5B) to evaluate the LLM’s accuracy on BBH (SmallKV ACC = 0.436 at 20%). Under the same memory constraint (for SmallKV we count both SLM and LLM cache), the performance of SmallKV (ACC = 0.462 at 40% for SLM and 20% for LLM) outperforms H2O (ACC = 0.38 at 30%). Note that while SLM compression may slightly impair saliency shift compensation, the trade-off remains favorable in terms of overall efficiency.\n\n|KV Cache Budget of SLM|20%|40%|60%|80%|Full|\n|-|-|-|-|-|-|\n|Accuracy (BBH)|0.2873|0.4416|0.4624|0.4642|0.4359|\n\nFurthermore, in an independently deployed scenario, the SLM is not required to generate draft tokens, enabling the adoption of an early layer stopping strategy. This approach significantly reduces both latency and memory overhead. We fix the LLM’s KV cache budget at 20% and progressively reduce the number of layers utilized in the SLM for attention mapping. The results indicate that stopping at layer 20 (83% of total layers) incurs no loss in accuracy (0.436). Furthermore, even when halting at layer 16 (67%), the accuracy of SmallKV remains superior to that of H2O.\n\n|Stop Layer of SLM|24 (Full)|20|18|16|12|H2O|\n|-|-|-|-|-|-|-|\n|Accuracy (BBH)|0.436|0.436|0.390|0.359|0.335|0.318|\n\n**Weakness 3: The paper demonstrates that SLM scaling significantly impacts performance under extremely low KV cache budgets (e.g., 5%), but it does not explore the feasibility of even smaller models (e.g., 0.1B), which could restrict its applicability on ultra-edge devices.**\n\n**R**: The main limitation hindering the scaling of the SLM to smaller sizes (e.g., 0.1B parameters) is the absence of such compact variants within major open-source model families. To address this challenge, we plan to explore techniques such as model distillation to train smaller versions of the SLM. This will enable us to better meet the deployment requirements of SmallKV on ultra-edge devices.\n\n**Paper Formatting Concerns**\n\n**R**: Thank you for your suggestion. Revision: Phi Team et al. Phi-3 technical report: A highly capable language model locally on your phone. arXiv preprint arXiv:2404.14219, 2024.\n\nWe hope this response has adequately addressed your concerns. We look forward to the opportunity for additional discussion during the review process."}, {"review_id": "1iTFz7ghcG", "reviewer": "Reviewer_9RZw", "summary": "Efficient management of the KV-cache is critical to mitigating resource contention during LLM inference. Existing approaches often suffer from issues such as saliency shift, where token importance dynamically changes, and marginal information overcompression, where individually less important tokens are overly compressed, degrading overall model accuracy. This paper observes that a small language model can produce attention scores similar to those of a much larger LLM. Motivated by this insight, the authors propose SmallKV, a method leveraging an SLM to guide efficient compression and eviction strategies for the KV-cache. Specifically, SmallKV uses the smaller model to track global attention patterns and approximate attention for marginal tokens, significantly improving both accuracy and inference throughput over existing approaches.", "questions": "1. How sensitive is the effectiveness of SmallKV to the size and choice of the small model? Did you experiment with architectures significantly different from the large model?\n\n2. Could you elaborate on why the attention patterns between the small and large models remain highly similar across scales? Are there specific conditions under which this similarity might break down?\n\n3. How exactly were thresholds for categorizing tokens into critical, marginal, and unimportant chosen? Can you share your insight of choosing 2:1:2", "limitations": "Yes", "rating": 5, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 3, "significance": 3, "originality": 3, "strengths_and_weaknesses": "Strengths:\n\n1. The paper clearly identifies the importance of marginal tokens and conducts the empirical experiments to validate the authors' observations. \n\n2. The paper introduces a novel use of a smaller auxiliary model for dynamic KV-cache management.\n\nWeaknesses:\n\n1. The use of an auxiliary smaller model introduces additional memory and computational overhead\n\n2. While some ablations are provided, more sensitivity experiment like varying sizes of SLM is missing", "was_revised": true, "final_justification": "The paper is strong, the rebuttal results alleviated my concerns, therefore I support its acceptance.", "rebuttal": "Thank you for your detailed review and valuable feedback! Below, we address the comments you raised:\n\n**Q1: How sensitive is the effectiveness of SmallKV to the size and choice of the small model? Did you experiment with architectures significantly different from the large model?**\n\n**R**: In Section 5.5 and Figure 7, we examine the impact of small model size (ranging from 0.5B to 7B parameters) on the effectiveness of SmallKV. Our findings show that when the KV cache budget is relatively large, scaling the small model has a relatively minor effect on performance (less than 10%). However, SLM scaling significantly influences performance under extremely low KV cache budgets (e.g., 5%). However, under extremely constrained KV cache budgets (e.g., 5%), the size of the small model plays a much more significant role in performance. This suggests that an effective strategy is to select a small model of appropriate size based on the desired KV cache compression ratio.\n\nFor SLMs with architectures significantly different from the large model, two main issues prevent the direct application of them in SmallKV. First, differences in tokenizers lead to mismatched dimensions and token representations in their attention matrices, making the compensation mechanism proposed in SmallKV inapplicable. Second, we measured the attention similarity between models from different series. Results show that the average cosine similarity between Llama3-1B and Qwen2-7B on the Wikitext-2 dataset is only 0.632 (note that due to vocabulary differences, we calculate similarity only on matched tokens via longest common subsequence alignment). In contrast, using Qwen2-0.5B as the SLM yields an average similarity of 0.947. This indicates that models from different series do not exhibit significant attention similarity. \n\n**Q2: Could you elaborate on why the attention patterns between the small and large models remain highly similar across scales? Are there specific conditions under which this similarity might break down?**\n\n**R**: The attention similarities between different-scale LLMs within same series are based on our empirical findings. We speculate that many factors contribute to this similarity: 1)  although models of varying scales differ in configuration parameters such as the number of layers and hidden dimensions, they share the same underlying architecture; 2) they are typically pre-trained on similar corpora; 3) in some cases, smaller models are distilled from their larger counterparts. Beyond these considerations, we believe a key contributing factor is the inherent redundancy in the attention patterns of LLMs, which enables a compact SLM to effectively approximate the attention behavior of a larger model.\n\nTo explore the conditions under which attention similarity might degrade, we conducted a comprehensive evaluation across multiple datasets—including MMLU, BBH, and GSM8K—and various SLMs ranging from Qwen2.5-0.5B to Qwen2.5-7B. And the results are shown in the Table below. The results indicate that, while some variability exists, attention similarity remains consistently high across different datasets within the same model series. For contrast, we also include a negative comparison, reporting attention similarity between models from different series. Due to differences in tokenizers across series, we align tokens using pairwise longest common subsequences and compute cosine similarity only over matched tokens. Based on our current experiments, we have not observed any specific condition under which the attention similarity between models within the same series fully breaks down.\n\n|SLM|Qwen2-0.5B|Qwen2-0.5B|Qwen2-0.5B|Qwen3-0.6B|Qwen2.5-0.5B|Qwen2.5-1.5B|Qwen2.5-3B|Qwen2.5-7B|LLaMA3.2-1B|\n|-|-|-|-|-|-|-|-|-|-|\n|LLM|Qwen2-7B|Qwen2-7B|Qwen2-7B|Qwen3-8B|Qwen2.5-14B|Qwen2.5-14B|Qwen2.5-14B|Qwen2.5-14B|Qwen2-7B|\n|Benchmark|BBH|GSM8K|MMLU|BBH|BBH|BBH|BBH|BBH|BBH|\n|Similarity|0.85|0.85|0.98|0.83|0.83|0.83|0.85|0.86|0.56|\n\n**Q3: How exactly were thresholds for categorizing tokens into critical, marginal, and unimportant chosen? Can you share your insight of choosing 2:1:2**\n\n**R**: Our evaluation results are primarily based on a fixed allocation ratio, adopting the same parameter setting as H2O for fair comparison (heavy : recent  = 1:1). So in our experiment, heavy tokens : recent : compensation = 2:1: 2 (as compensate token only occupy half of cache compared to others). However, in practice, we observe that the optimal compensation ratio varies slightly depending on the task and model scale. For instance: (1) When the KV cache budget is extremely limited (e.g., 5%), allocating the majority of the budget to heavy tokens yields better performance. (2) When the scale gap between the SLM and LLM is larger (e.g., 70B vs. 7B, compared to 7B vs. 0.5B), slightly reducing the compensation ratio leads to improved accuracy.\n\nTherefore, we suggest that in practical deployment of SmallKV, the adaptive token importance estimation capability of the SLM can be leveraged to dynamically adjust the allocation ratios and thresholds. As illustrated in Section 3.2 Figure 2, the proportions of critical tokens and marginal tokens can be dynamically determined based on the recall of attention scores from the small model, tailored to different tasks. For instance, we define the critical tokens as those accounting for the top 80% of cumulative attention scores, while marginal tokens are set to the subsequent 15% (i.e., from 80% to 95% of the cumulative sum). \n\nWe would also like to offer some clarification regarding the discussed weaknesses:\n\n**Weakness 1:\nThe use of an auxiliary smaller model introduces additional memory and computational overhead.**\n\n**R**: We quantify the theoretical maximum latency and memory overhead of the model combinations used in main experiments, as shown in the table below. The values 'Ratio' represent the additional overhead introduced by SLM relative to LLM (e.g., for combination Qwen2-0.5B & 7B, the KV cache of SLM is 1/4.67 that of the LLM).\n\n|Ratio between SLM and LLM|Qwen2-0.5B & Qwen2-7B|Qwen2.5-0.5B & Qwen2.5-14B|Qwen2-7B & Qwen2-72B|LLAMA 3.2-1B & LLAMA 3.1-8B|\n|-|-|-|-|-|\n|Latency|1/3.57|1/6.52|1/4.47|1/3.19|\n|KV cache|1/4.67|1/16.0|1/5.71|1/4.0|\n\nWe highlight the practicality of SmallKV, emphasizing how the computational overhead of the SLM can be effectively mitigated through deployment-level optimizations. To achieve this, we propose three key strategies: (1) integration with speculative decoding (SP); (2) compression of the KV cache for the SLM; and (3) layer skipping (i.e., early stopping) during the attention mapping process.\n\nRegarding latency, while the forward pass of the SLM during decoding incurs some overhead, this is mitigated by the acceleration provided through KV cache compression. As demonstrated in Section 5.3 Table 2 of the evaluation, SmallKV achieves lower latency compared to the baseline model. Additionally, the integration of SP can eliminate the forward pass of the SLM as a significant latency bottleneck. Current SP algorithms yield 3-5x speedups, further enhancing performance.\n\nRegarding memory consumption, the high attention sparsity inherent in SLM enables near-lossless reduction in memory usage through SLM KV cache compression. As shown in the table below, we fix the KV cache budget of the LLM (Qwen2-7B) at 20% and increase the compression ratio of the SLM to evaluate the LLM’s accuracy on BBH (SmallKV ACC = 0.436 at 20% budget). Under the same memory constraint (for SmallKV we count both SLM and LLM cache), the performance of SmallKV (ACC = 0.462 at 40% for SLM and 20% for LLM) outperforms H2O (ACC = 0.38 at 30%). Note that while SLM compression may slightly impair saliency shift compensation, the trade-off remains favorable in terms of overall efficiency.\n\n|KV Cache Budget of SLM|20%|40%|60%|80%|Full|\n|-|-|-|-|-|-|\n|Accuracy (BBH)|0.2873|0.4416|0.4624|0.4642|0.4359|\n\nFurthermore, in an independently deployed scenario, the SLM is not required to generate draft tokens, enabling the adoption of an early layer stopping strategy. This approach significantly reduces both latency and memory overhead. We fix the LLM’s KV cache budget at 20% and progressively reduce the number of layers utilized in the SLM for attention mapping. The results indicate that stopping at layer 20 (83% of total layers) incurs no loss in accuracy. Furthermore, even when halting at layer 16 (67%), the accuracy of SmallKV remains superior to that of H2O.\n\n|Stop Layer of SLM|24 (Full)|20|18|16|12|H2O|\n|-|-|-|-|-|-|-|\n|Accuracy (BBH)|0.436|0.436|0.390|0.359|0.335|0.318|\n\nSo in practice, the SLM’s actual KV cache usage is calculated as: (1/4.67) × 40%(SLM compression) × (20/24)(early stopping) ≈  7.14% of the full LLM KV cache.\n\n**Weakness 2:\nWhile some ablations are provided, more sensitivity experiment like varying sizes of SLM is missing**\n\n**R**: We apologize if the connection between this experiment and the requested sensitivity analysis was not immediately clear. We hope the following clarification confirms that the impact of SLM size variation is examined and we will highlight the sensitivity experiment in the paper.\n\nThe impact of SLM scaling is investigated in Section 5.5 (Impact of SLM Scaling). In this section, we present a dedicated experiment where we systematically vary the size of the SLM. We evaluate SmallKV using a range of SLMs: Qwen2.5 [0.5B, 1.5B, 3B, and 7B], with Qwen2.5-14B serving as the LLM across all configurations.\n\nThe results in Figure 7, provide a sensitivity analysis of SmallKV performance with respect to SLM scale. Our findings indicate that while SLM size has a relatively modest effect under higher KV cache budgets, it becomes significantly more influential under extremely low KV cache conditions—an important observation highlighted in the manuscript.\n\nWe hope this response has adequately addressed your concerns. We look forward to the opportunity for additional discussion during the review process."}, {"review_id": "wRndSvCwV3", "reviewer": "Reviewer_rx3H", "summary": "The paper presents a design named SmallKV, a method for compressing Key-Value (KV) caches in large language models (LLMs) to enable efficient inference in long-context scenarios. SmallKV leverages a small language model (SLM) to address two critical limitations of existing token-level eviction methods: (1) the saliency shift issue, where dynamic changes in token importance during decoding lead to suboptimal permanent eviction, and (2) the marginal information over-compression issue, where marginally important tokens, collectively significant to performance, are treated the same as unimportant ones. SmallKV introduces saliency shift compensation, utilizing the SLM’s full cache to dynamically adjust LLM eviction strategies, and marginal information compensation, which approximates the marginal tokens’ Key cache using SLM attention scores while retaining their Value cache.", "questions": "Besides the weaknesses above:\n\nThe main experimental results lack error bars or confidence intervals. Please provide a statistical analysis and specify the method used for computation.\n\nFigure 7 illustrates that larger SLMs yield limited performance gains, accompanied by increased computational cost. Please analyze the trade-off between SLM scale and performance, specifying optimal SLM size criteria.\n\nThe selection of thresholds for marginal tokens (K and P-K) is not well justified. Please provide a theoretical or empirical analysis of threshold selection and its impact on performance.", "limitations": "yes", "rating": 4, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 2, "clarity": 3, "significance": 2, "originality": 2, "strengths_and_weaknesses": "Strengths:\n\nSmallKV utilizes an SLM to enhance KV cache compression, incorporating saliency shift and marginal information compensation mechanisms to address the limitations of traditional eviction methods. The insight of high attention similarity between SLM and LLM provides a novel perspective.\n\nJaccard similarity matching of SLM-LLM attention matrices, dynamic eviction, and marginal token approximation enable efficient compression.\n\nWeaknesses:\n\nThe method's core assumption—that attention patterns between different-scale LLMs are highly similar—is an empirical observation. The authors acknowledge they cannot provide a formal theoretical proof to guarantee this similarity, which raises questions about the method's generalizability across all models and tasks. Therefore, the technique is not lossless, and its performance depends on the quality of the attention matching.\n\nThe results presented in Table 2 show that even when using a 0.5B model, your method does not hold an advantage over H2O in terms of TPOT. This suggests that the TPOT overhead is a potential concern, especially if the SLM's parameter size were to increase.\nThe chosen baseline, H2O (2023), is somewhat dated. It would be more convincing if the authors could demonstrate that their method has advantages over more recent SOTA works.\n\nThe paper's approach to using SLM to capture global attention views may struggle to ensure comprehension of complex sentences, as the evaluation is limited to simple mathematical tasks, such as GSM8K. However, current research trends focus on enabling LLMs to tackle more challenging problems, such as those in AMIE  and AMC.", "was_revised": true, "final_justification": "The authors' response has addressed most of the points (regarding the method's foundational assumption, performance overhead, and the justification for confident design choices). The new empirical data demonstrating high SLM-LLM attention similarity across diverse models, tasks, and even multi-turn dialogues provides strong support for the method's core assumption, mitigating my concerns about its generalizability. I improved my assessment of the paper's contribution.", "rebuttal": "Thank you for your detailed review and valuable feedback! Below, we address the comments you raised:\n\n**Q1: The main experimental results lack error bars or confidence intervals. Please provide a statistical analysis and specify the method used for computation.**\n\n**R**: All baseline methods, including ours, are conducted in a deterministic setting, which is the standard evaluation approach adopted in prior KV cache research. For our experiments, we set the model's temperature to 0, leading to greedy decoding, where the model always selects the token with the highest probability at each step. This results in fully deterministic outputs for any given input and model configuration.\n\nWhen evaluating on standardized benchmarks with fixed prompts and model parameters (temperature=0), repeated runs produce identical outputs and accuracies. In such a deterministic setting, error bars or confidence intervals are not applicable, as they are designed to capture uncertainty arising from stochastic variability in results.\n\n**Q2: Figure 7 illustrates that larger SLMs yield limited performance gains, accompanied by increased computational cost. Please analyze the trade-off between SLM scale and performance, specifying optimal SLM size criteria.**\n\n**R**: Based on the experimental results shown in Figure 7, we conclude that when the KV cache budget is relatively large, scaling the small model has a relatively minor effect on performance (less than 10%). However, SLM scaling significantly influences performance under extremely low KV cache budgets (e.g., 5%). Therefore, in practical production systems, when resources are sufficient, a larger SLM scale generally leads to higher accuracy of the LLM. Under resource-constrained conditions, a trade-off between efficiency and performance must be made. For instance, in scenarios requiring extreme compression, a more suitable choice would be to increase the scale of the SLM. Conversely, when the compression ratio is small, the SLM's size can be appropriately reduced. This represents a trade-off between resources, performance, and application scenarios. \n\nFurthermore, SmallKV is compatible with speculative decoding in deployment. Therefore, the choice of SLM scale is highly deployment-dependent, and should be determined jointly based on the accuracy requirements of the draft model in speculative decoding and the available memory resources on the target hardware.\n\n**Q3: The selection of thresholds for marginal tokens (K and P-K) is not well justified. Please provide a theoretical or empirical analysis of threshold selection and its impact on performance.**\n\n**R**: Indeed, using a fixed threshold for marginal tokens may not always lead to optimal results. In our evaluation, we adopted the same experimental parameters as other baselines to ensure comparability. However, a key advantage of SmallKV lies in the SLM's ability to adaptively identify which tokens are most important. \n\nTherefore, in deployment, we can dynamically define the proportions of saliency shift compensation and marginal tokens compensation based on the recall rate of attention scores from the small model, as illustrated in Figure 2. For instance, we consider the critical tokens to be those accounting for the top 80% of cumulative attention scores, while marginal tokens are defined as those within the subsequent 15% range (i.e., from 80% to 95% of the cumulative sum). This adaptive approach allows us to fine-tune the balance between saliency and marginal tokens to better suit the specific demands of different tasks. \n\nWe would also like to offer some clarification regarding the discussed weaknesses:\n\n**Weakness 1: The method's core assumption—that attention patterns between different-scale LLMs are highly similar—is an empirical observation. The authors acknowledge they cannot provide a formal theoretical proof to guarantee this similarity, which raises questions about the method's generalizability across all models and tasks. Therefore, the technique is not lossless, and its performance depends on the quality of the attention matching.**\n\n**R**: It is hard to provide a rigorous theoretical proof, instead, we have conducted extensive tests on the attention similarity between large and small models across various scenarios to address this concern. Due to attention sinks, we removed the first few tokens when computing similarities to avoid too high similarity (e.g., ~0.99), ensuring meaningful and comparable results. \n\nFirstly, we measured the similarity across different model combinations in our main experiments on the BBH task:\n\n| SLM | Qwen2-0.5B | Qwen2.5-0.5B | Qwen2-7B | LLaMA3.2-1B | \n|-------------|-------------|-------------|-------------|-------------|\n| LLM | Qwen2-7B | Qwen2.5-14B | Qwen2-72B | LLaMA3.1-8B | \n| Similarity | 0.85 | 0.83 | 0.81 | 0.98 | \n\nAdditionally, we evaluated the attention similarity between Qwen2 7B and Qwen2 0.5B across multiple tasks:\n\n| Benchmark | BBH |  GSM8K |  MMLU |\n|-------------|-------------|-------------|-------------|\n| Similarity | 0.85 | 0.85 | 0.98 | \n\nFurthermore, we examined the changes in similarity during multi-turn dialogue scenarios:\n\n|       | turn1 | turn2 | turn3 | turn4 | turn5 |\n|-------|-------|-------|-------|-------|-------|\n| global| 0.822 | 0.812 | 0.799 | 0.796 | 0.792 |\n| SmallKV| 0.822 | 0.809 | 0.792 | 0.791 | 0.787 |\n\nThese findings demonstrate the scalability of attention similarities to a considerable extent.\n\n**Weakness 2: The results presented in Table 2 show that even when using a 0.5B model, your method does not hold an advantage over H2O in terms of TPOT. This suggests that the TPOT overhead is a potential concern, especially if the SLM's parameter size were to increase. The chosen baseline, H2O (2023), is somewhat dated. It would be more convincing if the authors could demonstrate that their method has advantages over more recent SOTA works.**\n\n**R**: In the dynamic compensation process during decoding, SmallKV requires an additional forward pass through the SLM, which cannot be fully parallelized as in the prefill stage, which leads to a slightly lower TPOT compared with H2O. However, the partial compatibility with Flash Attention makes SmallKV achieving higher overall throughput. \n\nThe decoding efficiency overhead can be effectively mitigated in deployment by integrating SmallKV with speculative decoding, where the SLM forward pass is already part of the pipeline and does not contribute to additional latency. State-of-the-art speculative decoding methods commonly achieve 3–5× speedups.\n\nWe emphasize that the primary goal of SmallKV is not to propose a new KV cache compression algorithm, but rather to provide a general compensation mechanism that improves the accuracy of existing eviction-based methods. Any resulting inference acceleration is a beneficial side effect. This also addresses your second question.\n\nWe compare against H2O and PyramidInfer because our current implementation builds upon their cumulative attention score-based eviction frameworks. Importantly, SmallKV is not mutually exclusive with other KV cache compression techniques. SmallKV is a accuracy compensation mechanism designed based on the principle of attention sparsity in LLMs, specifically targeting attention score-based KV cache compression methods. As such, SmallKV is complementary and can be applied to other SOTA KV compression approaches to further enhance their accuracy.\n\n**Weakness 3: The paper's approach to using SLM to capture global attention views may struggle to ensure comprehension of complex sentences, as the evaluation is limited to simple mathematical tasks, such as GSM8K. However, current research trends focus on enabling LLMs to tackle more challenging problems, such as those in AMIE and AMC.**\n\n**R**: Firstly, we have supplemented the attention pattern similarity results under a broader range of scenarios in response to weakness 1. These additional tests provide a more comprehensive evaluation of the scalability across different scenarios.\n\nSecondly, we would like to emphasize that the primary goal of KV Cache compression is to preserve the foundational capabilities of the model rather than to extend its limits. Benchmarks such as AMIE are often used to explore the upper bounds of model capabilities. Our extensive and comprehensive evaluation in paper already covers a wide array of common LLM tasks, including logical reasoning, mathematics, coding, multi-turn dialogue, and long-context understanding. And the results show that SmallKV ensures improvement in accuracy across diverse tasks. We believe that the accuracy compensation strategy of SmallKV can also be applied to more challenging tasks, helping models improve inference accuracy while reducing memory footprint.\n\nWe hope this response has adequately addressed your concerns. We look forward to the opportunity for additional discussion during the review process."}, {"review_id": "PH9FKUGzgw", "reviewer": "Reviewer_Dh4K", "summary": "In this paper, authors propose a new KV cache compression method named SmallKV, based on the high similarity of attention matrices between LLMs with different scales. SmallKV uses small models to assist the large model in perceiving globally important information\n of attention and approximates the attention score for large models. SmallKV could solve the saliency shift problem and the marginal information over-compression problem. Extensive experiments on benchmarks demonstrate the effectiveness of SmallKV.", "questions": "1. The implementation is not detailed. Given that using 8 A100 GPUs, which parallelism did you use?\n2. What's the exactly memory usage for your experiments?\n3. Given that your complexity is still O(n^2) for small models, It would have a theorical compression upper bound for your method, can you show this?\n4. In Figure 1(c), can you add the comparison to LLaMA3.2-1B? It would much enhance your conclusion.\n5. Why does BHH accuracy much worser for Qwen2-7B with Qwen2-72B in Figure 5？\n6. Can this method be implemented together with Speculative Sampling such as Eagle?\n7. How does the magnitude of attention pattern divergence between SLMs and LLMs correlate with downstream performance?\n\nIf my concerns are solved, I would raise my score.", "limitations": "- It's hard to be implemented on edge-device.\n- The memory usage increased is not small.\n- It have a benefit upper bound compared to other evict-based method.\n- The system optimization for such method is hard.", "rating": 4, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 3, "significance": 3, "originality": 3, "strengths_and_weaknesses": "Strengths:\n- This paper is well-written and easy to follow.\n- The insight and observation is new and interesting for me.\n- The theory analysis is rich and make sense.\n- It can be easily implemented with other methods.\n\nWeakness:\n- There's no memory cost analysis experiments, I'm skeptical about the memory cost.\n- The requirement for SmallKV to use smaller models in the same series may limit its applicability. It remains unclear whether this approach can effectively enable efficient KV caching for SLMs.\n- The experiments to approve highly consistent attention patterns between LLMs with different scaling only contain the same serial models, missing negative comparison.", "was_revised": true, "final_justification": "My main concern about this paper has been solved.\n- A substantial drop in attention similarity between models from different families.\n- The ablation study of memory cost is sufficient.\n\nSo I raise my rating to Borderline accept.", "rebuttal": "Thank you for your detailed review and valuable feedback! Below, we address the comments you raised:\n\n**Q1: The implementation is not detailed. Given that using 8 A100 GPUs, which parallelism did you use?**\n\n**R**: We utilize pipeline parallelism by setting the device_map = \"auto\" feature from the Hugging Face Accelerate library. The configuration will be explicitly described in the revised manuscript.\n\n**Q2: What's the exactly memory usage for your experiments?**\n\n**R**: We acknowledge that SmallKV introduces additional memory overhead, primarily from the KV cache of the small model. Specifically, for the four model combinations used in our main experiments, the theoretical maximum memory consumption ratio between the SLM's and LLM's KV cache is shown in the table below. The values 'Ratio' represent the additional overhead introduced by SLM relative to LLM (e.g., for combination Qwen2-0.5B & 7B, the KV cache of SLM is 1/4.67 that of the LLM).\n\n|Ratio between SLM and LLM|Qwen2-0.5B & Qwen2-7B|Qwen2.5-0.5B & Qwen2.5-14B|Qwen2-7B & Qwen2-72B|LLAMA 3.2-1B & LLAMA 3.1-8B|\n|-|-|-|-|-|\n|KV cache|1/4.67|1/16.0|1/5.71|1/4.0|\n\nHowever, we emphasize that SmallKV remains highly practical, and the overhead of the SLM can be significantly reduced through deployment-level optimizations.We propose three key optimizations: (1) integration with speculative decoding (SP), (2) KV cache compression for the SLM, and (3) layer skipping (i.e., early stopping) during attention mapping.\n\nThe high attention sparsity observed in the SLM enables nearly lossless reductions in memory usage via KV cache compression (reduce to 40% of SLM cache). As shown in the table below, we fix the KV cache budget of the LLM (Qwen2-7B) at 20% and increase the compression ratio of the SLM (Qwen2-0.5B) to evaluate the LLM’s accuracy on BBH (SmallKV ACC = 0.436 at 20%). Under the same memory constraint (for SmallKV we count both SLM and LLM cache), the performance of SmallKV (ACC = 0.462 at 40% for SLM and 20% for LLM) outperforms H2O (ACC = 0.38 at 30%). Note that while SLM compression may slightly impair saliency shift compensation, the trade-off remains favorable in terms of overall efficiency.\n\n|KV Cache Budget of SLM|20%|40%|60%|80%|Full|\n|-|-|-|-|-|-|\n|Accuracy (BBH)|0.2873|0.4416|0.4624|0.4642|0.4359|\n\nFurthermore, in the independently deployed scenario, the SLM does not need to generate draft tokens. This allows SLM to adopt an early layer stopping strategy, significantly reducing its latency and memory overhead. We fix the LLM’s KV cache budget at 20% and gradually reduce the number of layers used in the SLM for attention mapping. The results indicate that stopping at layer 20 (83% of total layers) incurs no loss in accuracy (0.436). Furthermore, even when halting at layer 16 (67%), the accuracy of SmallKV remains superior to that of H2O. \n\n|Stop Layer of SLM |24 (Full)|20|18|16|12|H2O|\n|-|-|-|-|-|-|-|\n| Accuracy (BBH)|0.436|0.436|0.390|0.359|0.335|0.318|\n\nFinally, we would like to highlight that, to date, optimizations for LLM inference have primarily focused on two directions: (1) speculative decoding for GPU utilization optimization and (2) KV cache compression for memory and time efficiency. This work represents the first attempt to efficiently combine both approaches. Moving forward, we can further leverage advancements in these two areas to enhance the performance of a unified system.\n\n**Q3：Given that your complexity is still O(n^2) for small models, It would have a theorical compression upper bound for your method, can you show this?**\n\n**R**: We apologize for not fully grasping the intent of your question. Our interpretation is that, this refers to the overall compression upper bound caused by the need to save SLM full cache. Based on this, we present the theoretical maximum memory consumption of the small model in the SLM-to-LLM KV cache ratio table in Q2, and further propose two optimization techniques to reduce this overhead. The experimental results reflect a practical upper bound on compression. For example, for Qwen2-0.5B & 7B combination, when the LLM's KV cache budget is 20%, the SLM’s actual KV cache usage—under accuracy-preserving conditions—is calculated as:(1/4.67)×40%(SLM compression)×(20/24)(early stopping)≈ 7.14% of the full LLM KV cache.\n\nIf this answer does not fully address your question, we welcome your feedback and clarification. We are willing to provide you with more detailed explanation as needed.\n\n**Q4：In Figure 1(c), can you add the comparison to LLaMA3.2-1B? It would much enhance your conclusion.**\n\n**R**: Thanks for your valuable suggestions. We have successfully added the comparison to LLaMA3.2-1B. Due to restrictions on sharing images or links here, we instead quantify this observation using average cosine similarity on the Wikitext-2 dataset. Notably, we observe a substantial drop in attention similarity between models from different families. The average similarity between Qwen2-0.5B and Qwen2-7B is 0.947, while the similarity between LLaMA3.2-1B (used as the SLM) and Qwen2-7B (the LLM) drops to 0.632. (To account for tokenizer differences, we compute pairwise longest common subsequences to align tokens and calculate cosine similarity only over matched tokens.)\n\n**Q5：Why does BHH accuracy much worser for Qwen2-7B with Qwen2-72B in Figure 5？**\n\n**R**: To investigate this issue, we measure the average attention similarity between large and small models across several model combinations on the BBH dataset, as shown in the table below. The results show a positive (nonlinear) correlation between attention similarity and task accuracy. So the lower accuracy is because of the lower similarity of Qwen2-7B and Qwen2-72B. Please refer to Q7, the discussion about attention similarity and SLM scale.\n\n|SLM|Qwen2-0.5B|Qwen2.5-0.5B|Qwen2-7B|LLaMA3.2-1B| \n|-|-|-|-|-|\n|LLM|Qwen2-7B|Qwen2.5-14B|Qwen2-72B|LLaMA3.1-8B| \n|Similarity|0.85|0.83|0.81|0.98| \n\n**Q6: Can this method be implemented together with Speculative Sampling such as Eagle?**\n\n**R**: The drafting method in Speculative Sampling can be classified into Independent Drafting and Self-Drafting[1], SmallKV is compatible with Independent Drafting, where a smaller model from the same series is uesd as the drafter [2-3]. In contrast, Eagle follows the Self-Drafting paradigm, utilizing the final hidden states of a large model as input features to train an external transformer layer for drafting. So Eagle is not compatible with SmallKV.\n\n[1] Unlocking Efficiency in Large Language Model Inference: A Comprehensive Survey of Speculative Decoding \n\n[2] Accelerating large language model decoding with speculative sampling\n\n[3] Cascade speculative drafting for even faster llm inference.\n\n**Q7: How does the magnitude of attention pattern divergence between SLMs and LLMs correlate with downstream performance?**\n\n**R**: As demonstrated in Q5, low attention similarity leads to a relative performance drop. We also test the attention similarity and the performance (under 10% KV Cache Budget) across four SLMs of varying scales, using Qwen2.5-14B as LLM. The results, summarized in the table below, reveal a clear trend: as the size of the SLM increases, both attention similarity and task performance consistently improve.\n\n|SLM|Qwen2.5-0.5B|Qwen2.5-1.5B|Qwen2.5-3B|Qwen2.5-7B| \n|-|-|-|-|-|\n|Similarity|0.832|0.834|0.846|0.855| \n|Accuracy (BBH)|0.402|0.457|0.483|0.486| \n\nTherefore, a reasonable conclusion is that attention similarity is positively correlated with downstream task performance. Ideally, a similarity score of 1.0 would yield optimal predictive KV cache compression. However, it is important to recognize that this relationship is influenced by multiple factors, including model scale, the evaluation tasks, and the specific dynamics of KV cache compression. As a result, while the correlation is evident, it is difficult to precisely quantify the extent to which divergence in attention patterns affects downstream performance.\n\nWe would also like to offer some clarification regarding the discussed weaknesses:\n\n**Weaknesse1: There's no memory cost analysis experiments, I'm skeptical about the memory cost.**\n\n**R**: We provided a detailed quantitative analysis of memory cost in Q2, and we have also implemented two methods to reduce memory overhead: KV cache compression for the small model and early stopping (see Q2 for details). We hope these explanations and solutions adequately address your concerns.\n\n**Weaknesse2: The requirement for SmallKV to use smaller models in the same series may limit its applicability. It remains unclear whether this approach can effectively enable efficient KV caching for SLMs.**\n\n**R**: Developing multiple models of varying sizes to accommodate the diverse range of central and edge devices has become a prevailing trend in mainstream LLM development (e.g., the Llama, Qwen, and Phi-3 families). Furthermore, even when a directly suitable small model is not readily available, it can be effectively obtained through distillation by training a compact model from a larger one.\n\nSmallKV has been proven to effectively enable efficient KV caching for SLMs. As response in Q2, we have implemented KV cache compression for SLM, specifically aiming to reduce the associated memory overhead.\n\n**Weaknesse3:\nThe experiments to approve highly consistent attention patterns between LLMs with different scaling only contain the same serial models, missing negative comparison.**\n\n**R**: In Q4, we conduct a negative comparison by measuring attention similarity between models from different families. The results show that the attention similarity between models from different series drops significantly. This suggests that, models from different series may not be suitable for use with SmallKV in achieving effective accuracy compensation.\n\nWe hope this response has adequately addressed your concerns. We look forward to the opportunity for additional discussion during the review process."}]

Author rebuttal:

Meta-review: ## Summary
The paper proposes SmallKV, a novel framework for KV cache compression in large language models (LLMs) that leverages a small language model (SLM) to address saliency shift and marginal token over-compression issues. By utilizing attention similarity between SLMs and LLMs, SmallKV achieves efficient compression while maintaining high accuracy across various benchmarks and model scales (7B–72B). The method demonstrates significant throughput improvements (1.75–2.56×) and compatibility with existing techniques like Flash Attention and speculative decoding.

## Strengths
1. Reviewers unanimously praise the paper’s clarity and accessibility, noting its well-written structure and clear explanation of motivation, methodology, and results. 
2. The innovative use of an SLM to guide KV cache compression, particularly in addressing saliency shift and marginal token over-compression, is highlighted as a novel and impactful contribution by all reviewers. 
3. Extensive experiments across diverse benchmarks (GSM8K, BBH, MT-Bench, LongBench) and model scales demonstrate SmallKV’s robustness and superior performance compared to baselines like H2O and PyramidInfer, especially under aggressive KV cache budgets.
4. The Jaccard similarity matching of SLM-LLM attention matrices and dynamic eviction strategies further enhance the method’s effectiveness. Additionally, SmallKV’s compatibility with existing optimization techniques makes it practical for real-world deployment.

## Weaknesses
1. Reviewers raise concerns about the lack of rigorous theoretical analysis to explain the observed high similarity in attention patterns between SLMs and LLMs, which may limit the method’s generalizability across diverse models and tasks.
2. The computational and memory overhead of using an SLM, particularly when deployed independently, is not adequately quantified, raising practical concerns about latency and resource consumption.
3. The evaluation is limited to simpler tasks (e.g., GSM8K), and its applicability to more complex problems, such as those in AMIE or AMC, remains unclear (Reviewer rx3H). Additionally, the absence of experiments with smaller SLMs (e.g., 0.1B) or negative comparisons across different model series restricts insights into SmallKV’s flexibility on ultra-edge devices or diverse architectures.
4. Finally, comparisons against dated baselines like H2O rather than more recent state-of-the-art methods weaken the paper’s competitive claims.

**After rebuttal**, reviewers appreciate that their concerns are addressed. 

Consider the provided insights of this work are interesting, and the results are remarkable, I recommend accepting this paper as a spotlight.
Decision: Accept (spotlight)