SpeCache: Speculative Key-Value Caching for Efficient Generation of LLMs
Authors: Shibo Jie
Yehui Tang
Kai Han
Zhi-Hong Deng
Jing Han
Venue: ICML 2025 poster (2025)
Keywords: Key-Value Cache
Large Language Models
Primary area: deep_learning->large_language_models

Abstract: Transformer-based large language models (LLMs) have already achieved remarkable results on long-text tasks, but the limited GPU memory (VRAM) resources struggle to accommodate the linearly growing demand for key-value (KV) cache as the sequence length increases, which has become a bottleneck for the application of LLMs on long sequences. Existing KV cache compression methods include eviction, merging, or quantization of the KV cache to reduce its size. However, compression results in irreversible information forgetting, potentially affecting the accuracy of subsequent decoding. In this paper, we propose SpeCache, which takes full advantage of the large and easily expandable CPU memory to offload the complete KV cache, and dynamically fetches KV pairs back in each decoding step based on their importance measured by low-precision KV cache copy in VRAM. To avoid inference latency caused by CPU-GPU communication, SpeCache speculatively predicts the KV pairs that the next token might attend to, allowing us to prefetch them before the next decoding step which enables parallelization of prefetching and computation. Experiments on LongBench and Needle-in-a-Haystack benchmarks verify that SpeCache effectively reduces VRAM usage while avoiding information forgetting for long sequences without re-training, even with a 10x high KV cache compression ratio.

SpeCache: Speculative Key-Value Caching for Efficient Generation of LLMs
Shibo Jie
Yehui Tang²
Kai Han Zhi-Hong Deng'
Jing Han
Abstract
Transformer-basee large language models (LLMs) have already achieved remarkable results on long- text tasks, but the limited GPU memory (VRAM) resources struggle to accommodate the linearly growing demand for key-value (KV) cache as the sequence length increases, which has become a bottleneck for the application of LLMS long sequences. Existing KV cache compression meth- ods, including eviction, merging, or quantization of the KV cache to reduce its size, result in ir- reversible information forgetting and potentially affect the accuracy subsequent decoding. In this paper, we propose SPECACHE, which takes full advantage of the large and easily expand- able CPU memory to offload the complete KV cache, and dynamically fetches KV pairs back in each decoding step based on their importance measured by low-bit KV cache copy in VRAM. To avoid inference latency caused by CPU-GPU communication, SPECACHE speculatively pre- dicts the KV pairs that the next token might at- tend to, allowing to prefetch them before the next decoding step which enables parallelization of prefetching and computation. Experiments on LongBench and Needll-in----Haystack bench- marks verify that SPECACHE effectively reduces VRAM usage while avoiding information forget- ting for long sequences without re-training, even with 10x high KV cache compression ratio.
CPU RAM
LLM
LLM
Full KV cache
SpeCache
16-bit cache
3 Low-bit
Output token
Speculative token
Figure SPECACHE use low-bit KV cache and speculative token to "guess" the top-k most relevant 16-bit KV pairs for the next token, and prefetch them before the next decoding step.
performance in tasks such document processing, retrieval- augmented generation, and in-context learning. The com- monly used transformer architecture in LLMs relies on key- value (KV) caches to avoid redundant computations during decoding. However, the size of the KV cache grows linearly with the sequence length, introducing significant memory overhead. For example, in the case of LLaMA 2-7B (Tou- vron et al., 2023) processing sequences length 2k with a batch size of 16, the KV cache size reaches 8.4B, which ex- ceeds the model's own parameter count. Since the memory of computing units (e.g., GPU VRAM) is often limited, the KV cache becomes a bottleneck restricting the deployment of LLMs, especially on edge devices.
To alleviate the memory pressure caused by the KV cache, existing solutions, include using attention layers with slower KV cache growth (e.g.. grouped query attention (Ainslie et al., 2023)) or attention mechanisms that do not require KV cache (e.g., linear attention (Katharopoulos 2020)). However, these methods alter the model architec- ture, necessitating re- re-pretraining of the LLMs. Another approach is post-training KV cache compression, includ- ing techniques like eviction (Zhang et al., 2023; Ge et al., 2024). merging (Zhang et al., 2024), and quantization (Liu et al., 2024b), but these methods are applied greedily during decoding and may result in the loss of crucial information for subsequent steps. Alternatively, KV cache can be of- floaded to larger, more scalable memory (c.g., CPU memory or even disk), but this introduces frequent and substantial CPU-GPU communication, significantly increasing infer- ence latency (Sheng et al., 2023).
1. Introduction
al.,
The ability to handle long sequences is critical for large language models (LLMs), as it substantially impacts their
'State Key Laboratory of General Artificial Intelligence, School of Intelligence Science and Technology. Peking Univer- sity "Huawei Noah's ' Ark Lab 'Shoool of Artificial Intelligence, Beijing University of Posts and Telecommunications. Correspon- dence to: Zhi-Hong Deng <zhdeng @ wkw.edu..n.. Jing Han <hanj @ @ww@@ppppppp.......
Proceedings of the 42nd International Conference Machine Learning, Vancouver, Canada. PMLR 267, 2025. Copyright 2025 by the author(s).

---

SpeCache: Speculative Key-Value Caching for Efficientt Generation of LLMS
Existing research shows that the attention mechanism in is quite sparse (Liu et at., 2023b;a; Zhang et al., 2023), meaning that during decoding, we only need to ensure that a small number of KV pairs, which are most relevant to the current query, are fully present in VRAM. Based on this observation, the latency caused by offloading can be avoided via two strategies: i) Reduce the number of KV pairs prefetched to the GPU, fetching only several most important to the current query; ii) Perform the selection of important KV pairs far before the attention layer, enabling prefetching to run in parallel with GPU computation. How- ever, achieving these two objectives without the need for retraining remains a challenge.
KV-Efficient Architecture. The size of the KV cache is di- rectly determined by the model architecture. By modifying the model structure, the size of the KV cache can be reduced. Multi-Query Attention (MQA) (Shazeer, 2019) shares the key and value across all attention heads, while Grouped- Query Attention (GQA) (Ainslie et al., 2023) groups atten- tion heads and shares the key and value only within each group. YOCO (Sun et al., 2024b) allows the latter half of the layers in LLMs to reuse the key and value computed by the earlier half of the layers. Multi-Head Latent Atten- tion (MLA) (DeepSeek-Al et al., 2024) re-parameterizes the key and value as linear projections of low-rank space vectors. Additionally, some approaches modify the attention mechanism to avoid the linearly growing KV cache, such as RWKV (Peng et al., 2023), RetNet (Sun et al., 2023), and State Space Models (Gu & Dao, 2023). These methods alter the model architecture and therefore must be applied before pre-training begins, making them unsuitable for training- free optimization during the inference stage of off-the-sheif LLMs.
In this paper, we propose SPECACHE, a training- free method to implement the aforementioned strategies. SPECACHE stores the full KV cache in CPU memory. Be- fore each attention computation begins, SPECACHE strives to ensure that the top-k most relevant KV pairs has been already prefetched into VRAM. To achieve this, at each step. SPECACHE decodes two tokens simultaneously an "'output token" to compute the model's output, with the prefetched top-k KV pairs, and a "speculative token" to guess the KV pairs most likely to be attended to in the next decoding step, with low-bit copy of the KV cache in VRAM, as shown in Figure 1. These most relevant 16-bit KV pairs for the next step are prefetched in parallel into VRAM before the attention computation of the following
Post- Bost-Training Compression. Existing post-training KV cache compression methods include eviction, merg- ing, and quantization of KV pairs. StreamLLM only re- tains the most recent KV pairs and few initial KV pairs. H2O (Zhang et al., 2023), Scissorhands (Liu et al., 2023a), and RoCo (Ren & Zhu, 2024) use attention scores to mea- sure the importance of KV pairs and greedily drops unimpor- tant pairs. FastGen (Ge et al., 2024) introduces four policies to determine which KV pairs to keep and selects the opti- mal policy combination for each attention head during the prefilling stage. CaM (Zhang et al., 2024) and D20 (Wan et al., 2024) selectively merge the KV pairs that are about to be evicted with those that are being retained. thereby pre- serving partial information. MiniCache (Liu et al., 2024a) leverages the similarity of KV caches between adjacent lay- ers merge them across layers. KIVI (Liu et al., 2024b) and KVQuant (Hooper et al., 2024) apply per-channel key quan- tization and per-token value quantization to compress the KV cache down to 2-bit. ZipCache (He et al., 2024) intro- duces couunel-separablee token-wise quantization to achieve an even higher compression rate. These methods can re- duce the KV cache size in a training-free free manner during the inference stage. However, since compression inherently leads information loss, greedy compression at the current step may discard information that could be useful for future sleps, potentially degrading the performance of LLMs.
step.
Since the decoding process of LLMs is memory-1O bound, GPU utilization is quiet low, meaning that decoding two tokens in parallel introduces almost no additional latency. Moreover, because we can prefetch the necessary KV pairs for attention one step ahead, prefetching and computation can occur in parallel, avoiding any increase in inference latency. We conducted experiments on various LLMs using the LongBench (Bai et al., 2024) and geeele-in-a-Haystar benchmarks (Greg Kamradt, 2023). The results demon- strate that, compared to existing KV cache compression methods, SPECACHE achieves better performance with even smaller KV cache size. Moreover, the performance of SPECACHE is nearly on par with that of the original KV cache, even with only 10% KV cache size. Benefiting from this, SPECACHE can increase the batch size of decoding by up to 12x, achieving 4.6x larger throughput compared to the original KV cache.
2. Related Work
Offloading & Prefetching. FlexGen (Sheng et al., 2023) offloads the KV cache to CPU memory or even disk and searches for the optimal offloading strategy. Huggingfacc'ss 's trans o ormers (Wolf et al., 2019) library also imple- ments a simple offloaded KV cache. InfLLM (Xiao al.) segments long-range context into memory units that are of- floaded, and during inference, it retrieves and loads only the
2.1. Efficient KV Cache
Existing work on optimizing the KV cache size during the inference process of transformer-basee L.MSs can be cate- gorized into the following three directions:
2

---

SpeCache: Speculative Sey-Value Caching for Efficient Generation of LLMs
1.0
Latency (ms/step)
800
Fatch KV cache
10
Non-Contiguouu
Decode bs
Contiguous
0.9
600
Decode bs 4
10-1
Query-Depende Dependent Top-k
Decode bs
Latency
10
Full cache
Hit
0.8
Greedy Cache Eviction
400
Top-1% cache
10-3
200
0.7
10"
0
0%
2%
4%
%%
20:48
4096
8192
10a
10'
10P
k% Attention
Length
# of float16 parameters
Figure 2. Left: Hit rates of query-dependent top-k attention and greedy cache eviction. Middle: The latency of a single decoding step on the GPU vs the latency of loading the KV cache from the CPU to the GPU. Right: CPU-GPU transfer latency contiguous and non-contiguous CPU memory. We highlight the full and top-1% KV cache size with context length of 32k. Mcasured using (essral---Instruut--ee on NVIDIA A6000 GPU.
units relevant the current token for attention computation. While these methods reduce VRAM usage without losing KV cache information, the frequent CPU-GPU communica- tion significantly increases inference latency. In response to this, ShadowKV (Sun et al., 2024a) also tries to reduce the amount of fetched parameters to reduce latency.
compared to full attention scores. The two sparse attention mechanisms are: i) Query-dependent top-k attention, where each query only includes the KV pairs with top-k attention scores in its attention computation. ii) Greedy eviction, sim- ilar to H2O, where KV pairs with low cumulative attention scores are evicted to ensure that each query attends to only k KV pairs.
2.2. Bottleneck of LLM Inference
Based the results shown in Figure 2 (left), we can con- clude that: i) Attention is highly sparse. Only 0.5% of the keys can cover 90% of a query's attention. zi) The sparsity of attention is query-dependent. While both methods enforce the same level of sparsity, the hit rate for greedy eviction is much lower than that of query-dependent top-k attention. This suggests that different queries tend focus on distinct sets of keys. Although greedy eviction optimizes for the current query, it fails to preserve important KV pairs for subsequent queries, potentially evicting KV pairs that are crucial for future steps. Therefore, while eviction methods can achieve sparse attention, they cannot recover evicted KV pairs, resulting lower hit rates. Consequently, dynamic prefetching is crucial for maintaining attention performance.
The inference process of LLMs can be divided into two phases: the prefilling phase, where the model generates the KV cache for the prompt and produces the first output to- ken, and the decoding phase, where the output token from the previous step serves as input to generate the next to- ken. Generally, the bottleneck in the prefilling phase is the computational speed of the GPU, making it computation- bound, while the bottleneck in the decoding phase is the speed of data transfer between High Bandwidth Memory and Static Random Access Memory, making it memory-10 bound. Many techniques, such as batching (Kwon al., 2023) and speculative decoding (Leviathan al., 2023), leverage this characteristic by simultaneously inputting mul- tiple tokens during decoding, which, although increases FL.OPs, enhances GPU utilization and mitigates significant increases in single-step latency, and ultimately enhances overall throughput.
3.2. Profiling the Efficiency of KV Cache Offloading
As shown in Figure (middle), simply offloading and prefetching the entire KV cache leads substantial CPU- GPU transfer overhead. The experiments above inspire us to prefetch only a small number of the most important KV pairs during each decoding step, thereby reducing the num- ber of transferred data. For non-contiguous memory trans- ferring, mainstream framework such as PyTorch introduces additional time overhead. Therefore, we need to conduct ex- periments to verify the efficiency advantage of transferring sparse KV cache compared to the full KV cache.
3. Method
3.1. Profiling the Sparsity of KV Cache
Existing work has already explored the sparsity of attention in LLMs (Tang et al., 2024; Singhania et al., 2024). We begin by investigating the extent of this sparsity and the potential efficiency gains from transferring only the sparse KV cache.
As shown in Figure 2 (right), although sparse CPU-GPU transfers can introduce approximately 5 times the latency, we can enhance efficiency by reducing the amount of data transferred, since the attention mechanism maintains a high hit rate even at 1% sparsity. For instance, with Mistral-7B-
We conducted experiments the ILLMA-3-8B model us- ing the PG19 dataset truncated to sequence length of 8196. We measured the hit rate, which represents the proportion of attention scores captured by two types of sparse attention
3

---

SpeCache: Speculative Vey-V Caching for Efficientt Generation of LLMs
Output
16-bit cache
Low-bit cache
LLM
LLM
Prefill
Pre Decode
16-bit & low-bit cache
CPU RAM
CPU RAM
Output token
Output
Output
Speculative token
To be prefetched
LLM
he
hl.
LLM
hnt
To be quantized offloaded
Decode T=1
Decode T=2
Ray To be evicted
CPU RAM
CPU
Identify the Indexes of top-k KV pairs
RAM
Figure Illustration SPECACHE. prefilling stage, the cache quantized and offloaded layer by layer. In pre-decoding stage, we use the first output token to compute the first speculative token and prefetch the 16-bit KV pairs needed by the first decoding step. In each step of decoding stage, we simultancously decode two tokens: the output token and the speculative token. the results of both serve as inputs for the next step. The top-k most relevant 16-bit KV pairs for the speculative token are prefetched before the next step.
Instruct-v0.2 model and context length of 32k, transferring only top-1% % of the KV pairs would reduce transfer latency by 95%.
be divided into three stages: prefilling, pre-decodeing, and decoding.
Prefilling. During the prefilling stage, adopt layer-by- layer offloading approach. After the computation of each attention layer is completed, we quantize the KV cache to low precision and offload the original 16-bit KV cache C entirely to CPU memory, freeing up space for the next layer's KV cache. Once the prefilling stage complete, we obtain accurate first output token Ti.
3.3. SPECACHE: Speculative Key- Value Caching
Although the efficiency benefits of top-k prefetching are significant, still need method to predict the attention scores of KV pairs before loading the KV cache. To enable parallel prefetching and computation, we need to begin prefetching the KV pairs as carly as possible. This brings us to a key challenge: How can we determine which KV pairs are important far before the attention operation takes place? In fact, don't need prefetch the exact top-k KV pairs; we only need to prefetch KV pairs with high hit rate, ensuring that they include the vast majority of significantly attended ones. Based on this, we propose SPECACHE, a method that speculatively predicts which KV pairs will be important for future queries and prefetches them accordingly.
Pre-decoding. Before the decoding stage begins, only the low-bit KV cache C' resides in VRAM. To prefetch the necessary 16-bit KV pairs for the first decoding step, we introduce single decoding step as pre-decodeing. We use Ti as input, generating preliminary output Ty which may not be fully accurate, as a speculative token. Simultaneously. record the indexes of k KV pairs from the attention score calculation for Ti, denoted as KI, and immediately start prefetching those 16-bit KV pairs from CPU RAM in parallel with the computation of subsequent layers. After the pre-decoding step concludes, the first decoding step follows immediately.
Speculatively predicting future attention scores requires ap- proximate representations of both the historical key cache and the future query to be available in VRAM. For the former, existing training-free KV cache quantization meth- ods can effectively address this issue; for example, we can store 2-bit or even 1-bit approximation of the KV cache in VRAM. As for the latter, we propose decoding an addi- tional "speculative token" in parallel in each decoding step to approximate the next token.
Decoding, Before the 1-th decoding step begins, VRAM contains the previous output token Ti, speculative token T' low-bit KV cache C', and the top-k 16-bit KV pairs Cx, required by T,.
We decode T. and Ty in parallel,
0 =Attn((T, Ti C' UCK,)
(I)
As shown in Figure 3, The entire inference process can
in which C' Cx, denotes replace Cke, (the subset of C'

---

SpeCache: Speculative Key-Value Caching for Efficientt Generation of LLMs
Table 1. Performance of SPECACHE across various LLMs on LongBench. "16-bit" denotes vanilla KV cache without reduction
technique, "2-hit" denotes 2-hit KTVI, and "1-bit" denotes KIVI modified for 1-hit quantization. The KV cache size calculated based on the models' maximum context length.
Bit-width of KV Cache
KV
Qasper
MF-en
HotpoiQA
2WIHMQQA
Musique
GovReport
MultiNews
PRc
LCC
RB-P
Cache Size
Average
LLaMA 2-13B-4 1 2-7 Chat
16-bit
1,00x
19.5
34.0
30.1
26.5
10.0
24.1
26.4
9.0
59.7
53.1
29.2
2-hit (g = 32)
0.22x
19.0
31.4
29.7
25.1
9.8
22.0
25.3
7.0
58.8
52.8
28.1
+ SPECACHE
0.22x
20.2
32.0
30.5
26.4
10.3
24.0
26.1
8.5
59.2
53.7
29.1 (1 1.0)
2-bit (g = 64)
0.19x
18.4
31.1
28.7
26.6
9.8
19.9
25.3
8.5
55.8
51.1
27.5
+ SPECACHE
0.19x
19.8
32,9
30.1
27.4
9.5
23.6
25.6
9.0
59.3
53.4
29.1 (1 1.6)
16-bit
1.00x
24.1
37.0
36.4
31.9
15.8
24.5
25.7
12.0
50.2
50.6
30.8
2-bit (g 32)
x.22x
23.7
36,1
35,3
31.7
14.8
22.0
25.5
12.0
49.7
48.7
30.0
SPECACHE
0.22x
24.5
36.8
35.8
32.1
15.0
24.1
26.1
12.0
49.5
50.7
30.7 0.7)
2-bit (g = 64)
0.19x
22.4
34.1
36.1
32.8
14.5
21.2
25.3
12.3
49.8
48.1
29.7
+ SPECACHE
0.19x
23.6
36.5
35.5
32.0
14.7
23.8
25.6
11.1
49.9
49.5
30.2 (1 0.5)
16-bit
1.00x
33.1
49.2
43.0
27.3
18.8
32.9
27.0
87.0
53.5
51.4
42.3
Mistral-77 7B 1----sttttt-002222
2-bit (g = 32)
0.19x
31.4
49.0
42.0
26.2
18.2
32.3
26.8
76.8
52.9
51.2
40.7
+ SPECACHE
0.19x
32.5
49.3
42.7
27.9
18.4
32.3
26.7
82.1
53.2
50.4
41.5 (1 0.8)
2-bit (g 64)
0.16x
31.6
47.5
41.9
28.0
18.8
31,5
26.6
68.6
52.1
50.6
39.7
SPECACHE
0.16x
33.0
49.1
43.6
28.2
18.4
32.1
27.0
84.6
53.2
50.5
42.0 2.3)
1-bit (g = 32)
0.13x
18.9
40.3
32,9
25.4
13.5
20.6
22.3
41.1
46.6
42.8
30.4
SPECACHE
0.13x
31,5
50.0
44.1
25.7
18.2
27.6
26.6
78.6
52.0
49.6
40.4 10.0)
1-bit (g = 64)
0.10x
18.1
38.8
31.7
23.1
13.0
21.3
22.7
34.0
44.8
41,5
28.9
SPECACHE
0.10x
31.1
49.6
43,9
26.9
18.3
27.8
26.7
76.8
52.3
50.4
40.4 (1 11.5)
16-bit
1.00x
44.3
44.4
46.6
37.0
21.5
30.0
27.7
67.0
57.1
51.4
42.7
LLAMA-3-8BD 3-B-------nsstruee Instruct
2-hit (g = 32)
0,20x
43.2
44.5
46.9
37.6
20.7
29.6
27.3
67.5
51.1
47.0
41.5
SPECACHE
0,20x
44.0
44.7
46.7
36.8
21.5
29.7
27.6
67.0
56.7
50.2
42.5 (1 1.0)
2-bit (g- 64)
0.17x
43.4
44.2
46.0
36.5
20.6
29.4
27.2
65.0
53.0
50.9
41.6
SPECACHE
0.17x
43.7
45.1
46.4
37.0
21.0
29.7
27.7
67.0
57.0
50.4
42.5 0.9)
1-bit (g = 32)
0.14x
26.0
32.4
40.9
28.9
16.9
7.8
6.1
64.0
18.6
25.3
26.7
SPECACHE
0.14x
41.8
44.6
46.4
37.2
21.4
27.0
26.6
63.5
57.1
52.2
41.8 15.1)
1-bit (g = 64)
0.11x
23.9
29.6
39.8
25.5
15.7
4.9
5.5
62.3
17.8
22.0
24.7
SPECACHE
0.11x
43.2
45.6
46.2
36.9
20.7
27.9
27.1
65.0
55.0
51.0
41.9 (1 17.2)
with index KG) with 16-bit Ck, in this computation.
original inference process, adds only a single pre-decoding step and decodes two tokens simultaneously during decod- ing. The additional pre-decoding step is negligible in the overall sentence generation. While the number of decoded tokens increases, the model weights and KV cache used for both tokens are identical. Since the decoding process of LLMs is memory-10 bound, decoding two tokens simul- tancously allows shared access to model weights and KV cache, without introducing additional latency. Moreover, as the prefetching runs in parallel with GPU operations, the overall inference latency does not significantly increase.
During the attention computation, we also record the indexes K441 of top-k KV pairs based on the attention scores of T4+1, which are likely to be needed for decoding T(+1 in the next step. Once the computation concludes, we immediately begin prefetching those 16-bit KV pairs CR. evicting the non-top-k 16-bit KV pairs from VRAM that has been used, and offloading the newly created KV pairs, All these memory operations are in parallel with the computation of subsequent layers.
After decoding, T. and T/41 generate two tokens: Ti+1 and 2- Since the 16-bit KV pairs needed by T₁ were prefetched before its attention computation, we can assume that T is accurate and use it as the model's output. In contrast, T4t2 serves as the speculative token for the next decoding step, since it may be less accurate for output.
3.4. Post- Dost-Training Quantization for GPU KV Cache
Since SPECACHE stores a low-bit copy of the KV cache in the GPU, it can be combined with any KV cache quantiza- tion method. Existing post-training KV cache quantization methods, such as KIVI (Liu et al., 2024b), can quantize the KV cache to 2-bit without calibration, while methods like
The entire inference process of SPECACHE, compared to the
5

---

SpeCache: Speculative Key-Value Caching for Efficient Generation of LLMs
Table Comparison with other methods on LongBench. SPECACHE outperforms the other methods in terms of average performance, even when using higher compression ratios.
KV
Qasper
MF-en
HotporQA
2W1kIMQA
Musique
GovReport
MuldiNew
PRe
LCC
RB-P
Method
Cache
Average
Size
Full KV cache
1.00x
33.1
49.2
43.0
27.3
18.8
32.9
27.0
87.0
53.5
51.4
42.3
Mistral -B. 1--nstrructvO.22
InfLLM
0.25x
16.8
38.4
33.9
19.2
18.2
29.6
24.7
41.4
52.8
55.3
33.0
StreamLLM
0.25x
15.1
25.4
27.7
17.4
14.6
27.4
22.1
31.6
51.8
55.9
28.9
H₂O
0.25x
24.9
44.8
35.0
19.0
17.5
28.6
24.2
82.9
53,9
52.3
38.3
SPECACHE
0.16x
33.0
49.1
43.6
28.2
18.4
32.1
27.0
84.6
53.2
50.5
42.0
InfLLMM
0.13x
12.8
33.0
29.1
16.2
13.3
27.9
23.8
26.2
54.1
53.5
29.0
StreamLLM
0.13x
11.3
22.9
22.8
12.0
10.7
24.6
19.7
16.9
53.8
56.0
25.1
H₂O
0.13x
21.4
41.1
32.8
16.8
15.9
26.2
23.0
79.5
52.8
51.8
36.1
SPECACHE
0.10x
31.1
49.6
43.9
26.9
18.3
27.8
26.7
76.8
52.3
50.4
40.4
Full KV Cache
1.00x
44.3
44.4
46.6
37.0
21.5
30.0
27.7
67.0
57.1
51.4
42.7
LLaMA-3-BR 88- Irss-ntruutt
InfLLM
0.25x
28.0
35.1
36.6
23.0
14.4
29.6
25.5
37.5
56.9
58.3
34.5
StreamLLM
0.25x
23.0
21.1
29.8
24.7
12.0
25.9
22.6
21.0
55.0
57.2
29.2
H₂O
0.25x
41.2
41.8
46.8
36.9
21.5
25.7
23.7
66.0
55.1
51.2
41.0
SPECACHE
0.17x
43.7
45.1
46.4
37.0
21.0
29.7
27.7
67.0
57.0
50.4
42.5
InfLLM
0.13x
22.0
27.0
33.1
20.5
9.3
27.5
24.4
18.0
61.4
58.4
30,2
StreamLLM
0.13x
16.6
17.4
25.7
18.5
9.8
23.4
19.9
8.0
60.4
55.8
25.6
H2O
0.13x
37.8
42.1
46.6
36.9
21.5
23.7
21.9
65.5
54.6
50.8
40.1
SPECACHE
0.11x
43.2
45.6
46.2
36.9
20.7
27.9
27.1
65.0
55.0
51.0
41.9
KVQuant (Hooper et al., 2024) achieve 1-bit quantization with the help of calibration. We utilize KIVI to quantize the
(3 min X + max X)/4, and values in the range [(min X + max 22, max are quantized to the midpoint of this interval, min + maxX)/4
GPU copy due its simplicity and the fact that does not require calibration. To further push the limits of KV cache compression, modify KIVI for 1-bit quantization.
4. Experiments
The original KIVI quantizes KV cache follows:
4.1. Implementation Details
X 2x xX
Q(X)
X' ='=Q(X) sx + 2x,
(2) Note that KIVI retains small set of recent KV pairs as residuals to perform channel-wise quantization of the keys and enhance model performance. Since SPECACHE tem- porarily loads a small number of KV pairs into the GPU, for fair comparison, we reduce the number of residual KV pairs used by SPECACHE, ensuring that SPECACHE does not inerease VRAM usage on top of KIVI. Specifi- cally, for the baseline KIVI method, we use 128 residual KV pairs and quantization group size of 32 and 64. For SPECACHE, we prefetch top-64 16-bit KV pairs from CPU memory. Since these 16-bit KV pairs will be loaded into VRAM, in order to ensure that the total size KV cache remains unchanged, we use a smaller residual length of 64.
where zx = min X is the zero-point, 8x = (max X min X)/ -1 1) is the scaling factor, and L-1 is the round- ing operation.
However, in 1-bit quantization, the elements of X' are ei- ther max X min X. This results in the quantized KV cache having an unusually large magnitude, which makes the model struggle to perform text generation effectively. To address this issue, we improve KIVI in the 1-bit scenario by assuming that the weights follow uniform distribution between the minimum min X and max X, and ensuring that the cumulative quantization error is minimized. Based on this, the zero-point and scaling factor modified follows:
4.2. Performance on LongBench
For LongBench, we report results on 10 tasks: Qasper, Mul- tiFieldQA, HotporQA, 2WikiMQA, MuSiQue, GovReport, MultiNews, PassageRetrieval, LCC,, and RepoBench-P, Full results on all the 15 tasks provided in Appendix. The maximum sequence length is set to 4k for LLaMA-2, 32k
3 min X + max X
max X min X
xx
Sx
(3)
2
which ensures that all values in the range [min X, (min.X X max X)/2) are quantized the midpoint of this interval,
6

---

SpeCache: Speculative Vay-Value Caching for Efffciiett Generation of LLMs
Table 3. Decoding throughput. Evaluated on single NVIDIA A6000 GPU using
Full KV cache
SPECACHE (2-bit, = 64)
SPECACHE (1-bit,
64)
Context
Batch Size
Throughput
Batch Size
Throughput
Speedup
Batch Size
Throughput
Speedup
2k
44
190.3
272
418.5
2.2x
410
526.3
2.8x
8k
11
47.0
82
133.7
2.8x
134
177.7
3.8x
32k
3
10.3
22
34.6
3,4x
36
47.3
4.6x
Mistral-7--instructv0..
LLaMA-3-8B-Inste
Average Accuracy
16-bit
35
32.5
30
30.0
16
128
256
16
128
256
k
1-bit
Figure 5, Ablation study on k. We use 1-bit SPECACHE with 32.
1-bit SpeCache (Ouss)
In addition to KIVI, we also compare SPECACHE with other representative KV cache compression methods, including InfLLMM (Xiao et al.), H2O (Zhang et al., 2023), and Stream- LLM (Xiao et al., 2024). For these baselines, we follow the settings from Yuan al. (2024), compressing the KV cache to 1/4 or 1/8 of its original size, and compare them with 2-bit and 1 1-bit SPECACHE, respectively. The residual length and group size for SPECACHE are set to 64. As shown in Ta- ble 2. SPECACHE outperforms the other methods in terms of average performance, even when using higher compression ratios.
Mistral 7i-InBBruct---t
Figure 4. Performance on Needle-in-a-baystack benchmark. We use g 32 for quantization, resulting the KV cache com- pression ratio of 0.13 and 0.14 for (harrll7--nnttuct-xd) and LLaMA-3-8B- Instruct, respectively.
4.3. Performance on Needle-in-a- Benchmark
for Mistral, and 8k for LLMMA-3. For LLaMA-2, we only opt for 2-bit KV cache quantization, as the model's s perfor- mance under 1-bit KV cache quantization is significantly poor.
First, we evaluate the model's long-context retrieval abil- ity after applying KV cache compression with SPECACHE on a synthetic task - the Needle-in-a-Hane Hayaa-Hayttack (NIAH) benchmark (Greg Kamradt, 2023). We report retrieval ac- curacy under three different settings the Mistral-7B. Instruct-v0.2 model with 32k context and the LLaMA- 3-8B- --s-tuutt model with an 8k context: 16-bit KV cache, where no KV cache compression is applied; 1-bit KV cache, where we use KIVI to quantizes the KV cache to 1-bit; and 1-bit KV cache + SpeCache, where SpeCache used to prefetch the 16-bit KV cache from CPU memory on top of 1-bit quantization.
According to the results are shown in Table we have several observations. As the degree of KV compression increases (with lower bit widths or larger group sizes), the model's performance declines. However, when combined with SPECACHE, the majority of the performance loss is recovered. We find that the greater the KV compression, the more pronounce the advantages of our method become, which further demonstrates that top-k KV cache recover most of the attention information. Specifically, our method can maintain a performance gap of only 2% and 1% com- pared to the 16-bit baseline for LLaMA-3-8P C-A-A----B-Insttuce Instrucct respectively, while retaining only approximately 10% of the KV cache size the GPU.
The results are shown in Figure 4. Under 1-bit KV cache quantization, LLMs' long-context retrieval ability is sig- nificantly compromised. After applying SPECACHE, the performance of LLMs under 1-bit KV cache quantization is comparable to that of the 16-bit.
B--d-----nntttt-OO.... and

---

SpeCache: Speculative Key-Value Caching for Efficientt Generation of LLMs
Table 4. Comparison with non-speculative fetching, We report the average decoding latency for a single step when using the maximum
batch size that 48GB VRAM of single NVIDIA A6000 can handle with context length of 2k.
HolporQA
2WikIMQA
GovReport
MultiNews
W/ Spee. Prefetch
Qasper
MF-en
Musique
PRe
LCC
RB-P
Latency (ms/step)
Bit-width
Average
2-bit
44.5
44.9
46.4
37.4
21.4
29.6
27.8
66.5
57.3
50.5
42.6
877
2-bit
43.7
45.1
46.4
37.0
21.0
29.7
27.7
67.0
57.0
50.4
42.5
643
X
1-bit
43.3
44.7
46.7
37.2
21.8
28.9
27.5
66.0
58.3
50.1
42.4
1144
1-bit
43.2
45.6
46.2
36.9
20.7
27.9
27.1
65.0
55.0
51.0
41,9
775
Table Ablation study on quantization method. Tested on
of transferred data. In practical applications, a trade-off for k can be made.
LLaMA-3-8B- 3-8t-Instruct.
Improved
Comparison with original 1-bit KIVI. To highlight the im- portance of the new zero point and scaling factor we propose for 1-bit KIVI, we conduct an ablation study. As shown in Table 5, when using the original KIVI, although SPECACHE provides some improvements, the model's 's performance is still very poor, regardless of whether SPECACHE is used. After modifying the quantization method, the model's per- formance significantly improves, and the advantages of SPECACHE become much more apparent.
SPECACHE
Qasper
2WikiMQA
RB-P
1-bit KIVT
X
3.7
4.9
20.3
4.1
6.5
21.6
23.9
22.0
V
43.2
36.9
51.0
4.4. Efficiency
To highlight the advantages of SPECACHE in reducing KV cache size, we test its maximum throughput during decoding. conduct tests scenarios with context lengths of 2k, 8k, and 32k, increasing the batch size to maximize GPU mem- ory usage up the 48GB VRAM of an NVIDIA A6000 GPU. The decoding throughput measured Mistral-7B- Instruct-v0.2 model with HuggingFace transformers as framework. Note that our CPU-GPU interaction code is implemented using pytorch' multi-stream mechanism and the Tensor. copy-() () method, so the parallelism achieved is not theoretically optimal. By customizing lower- level operators, the efficiency of SPECACHE can be further improved.
Comparison with non-speculative fetching. One advan- tage of our method is the ability to prefetch KV pairs one step ahead, allowing prefetching and computation to run in parallel. We compare this with another simple strategy, where instead of using speculative tokens, we use the output tokens from the previous step to estimate the top-k KV pairs and fetch them, followed by recalculating attention. this strategy, the fetching starts after the attention computation is completed and needs be finished before recalculating attention, meaning it cannot run in parallel.
As shown in Table 4, using accurate output tokens instead of speculative tokens slightly improves the model's perfor- mance, but the cost of significantly increasing the model's latency. As the batch size increases, the time required to load KV pairs from the CPU increases linearly, while the computation time increases more slowly due to the improved parallelism. Therefore, during decoding with large batch sizes, the latency of loading KV pairs becomes the domi- nant factor. This highlights the importance of parallelizing computation and prefetching.
As shown in the Table 3, with 2-bit and 1-bit quantization, SPECACHE allows the batch size to increase by up to 7x and 12x, respectively, resulting in overall throughput im- provements of 3.4x and 4.6x compared to the original setup. Notably, when the context length is large, the orig- inal KV cache can only accommodate smaller batch sizes, leading to lower parallelism. This results in more significant acceleration with SPECACHE.
5. Conclusion
4.5. Ablation Experiments
In this paper, propose SPECACHE, a low-latency, high- performance, and training free KV cache compression method, SPECACHE stores high-precision KV cache in CPU RAM, while low-bit (down to bit) KV cache stored in GPU VRAM. It utilizes speculative tokens and output tokens co-decoding anticipate and prefetch the
Ablation study k. We conduct ablation study the number of prefetched KV pairs, k. As shown in the Figure 5, even with small k, such as k = 16. SPECACHE still provides a significant improvement over the KIVI baseline (i.e., = 0). As increases, the model's performance improves, but the same time, larger increases the size
8

---

SpeCache: Speculative Key-Value Caching for Efficientt Generation of LLMs
top-k KV pairs for the next decoding step. This approach effectively recovers the information lost due to the low-bit
modeling with selective state spaces. ArXiv preprint, arXiv:2312.00752, 2023.
KV cache while enabling the parallelization of prefetching and computation.
He, Y., Zhang, L., Wu, W., Liu, J., Zhou, H., and Zhuang, B. Zipcache: Accurate and efficient KV cache quanti- zation with salient token identification. ArXiv preprint, arXiv:2405. 14256, 2024.
Impact Statement
This paper is based on LLMs, which have potential societal impacts, including concerns around bias, misinformation, and accessibility. While these issues are well-known, we aim to contribute to improving the efficiency of LLMSs. Con- tinued ethical oversight and interdisciplinary collaboration are essential as the field evolves.
Hooper, C., Kim, S., Mohammadzadeh, H., Mahoney, M. W., Shao, Y. S., Keutzer, K., and Gholami, A. Kvquant: Towards 10 million context length LLM in- ference with KV cache quantization. ArXiv preprint, arXiv:2401188799, 2024.
Katharopoulos, A., Vyas, A., Pappas, N., and Fleuret, F. Transformers are mns: Fast autoregressive transformers with linear attention. In Proceedings of ICML, 2020.
References
Ainslie, J., Lee- Thorp. J., de Jong, M., Zemlyanskiy, Y., Lebrón, F., and Sanghai, S. GQA: training generalized multi-query transformer models from multi-head check- points. In Proceedings of EMNLP, 2023.
K Kwon, W., Li, Z., Zhuang, S., Sheng, Y., Zheng, I.., Yu, C. H., Gonzalez, J., Zhang, H., and Stoica, Efficient memory management for large language model serving with pagedattention. In Proceedings of SOSP. 2023.
Bai, Y., Lv, X., Zhang, J., Lyu, H., Tang, J.. Huang, Z., Du, Z., Liu, X., Zeng, A., Hou, L.., Dong, Y., Tang, J., and Li, Longbench: A bilingual, multitask benchmark for long context understanding. In Proceedings of ACL, 2024.
Leviathan, Y., Kalman, M., and Matias, Y. Fast inference from transformers via speculative decoding. In Krause, A., Brunskill, E., Cho, K., Engelhardt, B., Sabato, S., and Scarlett, J. (eds.), Proceedings of ICML, 2023.
DeepSeek-AI, Liu, A., Feng, B., Wang, B., Wang, B., Liu, B., Zhao, C., Deng, C., Ruan, C., Dai, D., Guo, D., Yang, D., Chen, D., Ji, D., Li, E., Lin, F., Luo, F., Hao, G., Chen, G. Li, G., Zhang, H., Xu, H., Yang, H., Zhang, H., Ding, H., Xin, H., Gao, H., Li, H., Qu, H., Cai, J. L., Liang. J., Guo, J., Ni, J., Li, J., Chen, J., Yuan, J., Qiu, J., Song, J. Dong, K. Gao, K., Guan, K., Wang, I.., Zhang, I.., Xu, L., Xia, L., Zhao, L., Zhang, L., Li, M., Wang. M., Zhang, M., Zhang, M., Tang, M., Li, M., Tian, N. Huang, P., Wang, P., Zhang, P., Zhu, Q. Chen, Q., Du, Q., Chen, R. J. Jin, R. L.,, Ge, R., Pan, R., Xu, R., Chen, R., Li, S. S., Lu, S., Zhou, S., Chen, S., Wu, S., Ye, S., Ma. S., Wang, S., Zhou, S., Yu, S., Zhou, S., Zheng, S., Wang, T., Pei, T. Yuan, T., Sun, T., Xiao, W. L.,, Zeng, W., An, W., Liu, W., Liang W., Gao, W., Zhang, W., Li, X. Q., Jin, X., Wang, X., Bi, X., Liu, X., Wang, X., Shen, X., Chen, X., Chen, X., Nic, X., and Sun, X. Deepseek-v2: strong, economical, and efficient mixture-of of-exterts language model. ArXiv preprint, arXiv:2405.0443x 2024.
Liu, A., Liu, J., Pan, Z., He, Y., Haffari, G., and Zhuang. B. Minicache: KV cache compression in depth di- mension for large language models. ArXiv preprint, arXiv:2405. 14366, 2024a.
Liu, Z., Desai, A., Liao, F., Wang, W., Xie, V., Xu, Z., Kyrillidis, A., and Shrivastava, A. Scissorhands: Exploit- ing the persistence of importance hypothesis for LLM KV cache compression at test time. In Proceedings of NeurlPS, 2023a.
Liu, Z., Wang, J., Dao, T., Zhou, T., Yuan, B., Song, Z., Shrivastava, A., Zhang, C., Tian, Y., Ré, C., and Chen, B. Deja vu: Contextual sparsity for efficient 1mms at inference time. In Proceedings of ICML. 2023b.
Liu, Z., Yuan, J., Jin, H., Zhong, S., Xu, Z., Braverman, V., Chen, B., and Hu, X. KIVI: A tuning-free asymmetric 2bit quantization for KV cache. In Proceedings of ICML, 2024b.
Ge, S., Zhang. Y., Liu, L., Zhang, M., Han, J., and Gao, J. Model tells you what to discard: Adaptive KV cache compression for Ilms. In Proceedings of ICLR, 2024.
Peng, B., Alcaide, E., Anthony, Q., Albalak, A., Arcadinho, S. Biderman, S., Cao, H., Cheng, X., Chung, M., Der- czynski, L., Du, X., Grella, M., GV, K. K., He, X., Hou, H., Kazienko, P., Kocon, J., Kong, J., Koptyra, B., Lau,
Greg Kamradt. Needle in haystack pressure testing 1lms, com/
2023. URL https: http:::://htttt:///////// gkamradt BooraadtILMAN MLAMMMeetttMeddelllaaaaattttttt
H., Lin, J.. Mantri, K. S. I., Mom, F., Saito, A., Song, G., Tang, X., Wind, J. S., Wozniak, S., Zhang, Z., Zhou, Q.. Zhu, J., and Zhu, R. RWKV: reinventing mns for the transformer era. In Findings of EMNLP, 2023.
Gu, A. and Dao, T.
Mamba: Linear-time sequence

---

SpeCache: Speculative Key-Value Caching for Efficientt Generation of LLMs
Ren, S. and Zhu, K. Q. On the efficacy of eviction pol- icy for key-value constrained generative language model inference. ArXiv preprint, atXiv:2402.06262, 2024.
dynamic discriminative operations for efficient genera- tive inference of large language models. ArXiv preprint, arXiv:2406.1300 2024.
Shazeer, N. Fast transformer decoding: One write-head is all you need. ArXiv preprint, arXiv: 1911.01991000200, 2019.
Wolf, T., Debut, L., Sanh, V., Chaumond, J., Delangue, C., Moi, A., Cistac, P., Rault, T., Louf, R., Funtowicz, M., and Brew, J. Huggingface's transformers: State- of-the-art natural language processing. ArXiv preprint, arXiv: 19t..0311177,, 2019.
Sheng, Y., Zheng, L., Yuan, B., Li, Z.,, Ryabinin, M., Chen, B., Liang, P., Ré, C., Stoica, I., and Zhang, C. Flexgen: High-throughput generative inference of large language models with a single GPU. In Proceedings of ICML, 2023.
Xiao, C., Zhang, P., Han, X., Xiao, G., Lin, Y., Zhang, Z., Liu, Z., Han, S., and Sun, M. Infllm: Unveiling the in- trinsic capacity of Ilms for understanding extremely long sequences with training- free memory. ArXiv preprint, arXiv:2402.04617.
Singhania, P., Singh, S., He, S., Feizi, S., and Bhatele, A. Loki: Low-rank keys for efficient sparse attention. In Proceedings of NeurlPS, 2024.
Xiao, G., Tian, Y., Chen, B., Han, S., and Lewis, M. Effi- cient streaming language models with attention sinks. In Proceedings of ICLR, 2024.
Sun, H., Chang, L., Bao, W., Zheng, S., Zheng, N., Liu, X., Dong, H., Chi, Y., and Chen, B. Shadowkv: KV cache in shadows for high-throughput long-context LLM inference, arXiv preprint, arXiv:2410.21465, 2024a.
Yuan, J., Liu, H.,, Zhong, S., Chuang, Y., Li, S., Wang, G., Le, D., Jin, H., Chaudhary, V., Xu, Z., Liu, Z., and Hu, X. B. KV cache compression, but what must we give in return? A comprehensive benchmark of long context capable approaches. In Findings of EMNLP, 2024.
Sun, Y., Dong, L., Huang, S., Ma, S., Xia, Y., Xue, J., Wang. J., and Wei, Retentive network: successor to transformer for large language models. ArXiv preprint, MrXiv:2307.086211 2023.
Zhang, Y., Du Y., Luo. G., Zhong, Y., Zhang, Z., Liu, S., and Ji, R. Cam: Cache merging for memory-efficient Ilms inference. In Proceedingy of ICML, 2024.
Sun, Y., Dong, L., Zhu, Y., Huang, S., Wang. W., Ma, S., Zhang, Q., Wang, J., and Wei, F. You only cache once: Decoder-decoder architectures for language models. ArXiv preprint, arXiv:2405.05254, 2024b.
Zhang, Z., Sheng, Y., Zhou, T., Chen, T., Zheng, L., Cai, R., Song, Z., Tian, Y., Ré, C., Barrett, C. W., Wang, Z., and Chen, B. H2O: heavy-hitter oracle for efficient generative inference of large language models, In Proceedings of NeurlPS, 2023.
Tang, J., Zhao, Y., Zhu, K., Xiao, G., Kasikei, B., and Han, S. QUEST: query-aware sparsity for efficient long-contes LLM inference. In Proceedings of ICML, 2024.
Touvron, H., Martin, L., Stone, K., Albert, P., Almahairi, A., Babaei, Y., Bashlykov, N., Batra, S., Bhargava, P., Bhos- ale, S., Bikel, D., Blecher, L., Centon-Ferreet C., Chen, M., Cucurull, G., Esiobu, D., Fernandes, J., Fu, J., Fu, W., Fuller, B., Gao, C., Goswami, V., Goyal, N., Hartshorn, A., Hosseini, S., Hou, R., Inan, H., Kardas, M., Kerkez, V., Khabsa, M., Kloumann, I., Korenev, A., Koura, P. S., Lachaux, M., Lavril, T., Lee, J., Liskovich, D., Lu, Y., Mao, Y., Martinet, X., Mihaylov, T., Mishra, P., Molybog, I. Nie, Y., Poulton, A., Reizenstein, J., Rungta, R., Saladi, K., Schelten, A., Silva, R., Smith. M., Subramanian, R., Tan, X. F., Tang, B., Taylor, R., Williams, A., Kuan, J. X., Xu, P., Yan, Z., Zarov, I., Zhang, Y., Fan, A., Kam- badur, M., Narang, S., Rodriguez, A., Stojnic, R., Edunov, S., and Scialom. T. Llama 2: Open foundation and fine- tuned chat models. ArXiv preprint, arXiv:2307.09288. 0092888, 2023.
Wan, Z., Wu, X., Zhang, Y.. Xin, Y., Tao, C., Zhu, Z., Wang, X., Luo, S., Xiong, J., and Zhang, M. D20:
10

---

SpeCache: Speculative Value Caching for Efficient Generation of LLMs
Appendix
A. Algorithm
We provide the pseudocode for a single attention layer. For simplicity, we have omitted the residual and grouped quantization details of KIVI.
Algorithm 1 Prefilling
Input: T E RLxd
Q Tw K TW k. == TW,
q.
K' = =Quant K). V' = Quant(V)
C = (V. C' (V',K')
Offload(C) (C)
A MaskedSoflmax((Q QK)
0 AVW. 0
Output: 0 Lxd
Algorithm 2 Pre-decoding
Input: Ti R lxd
Q T.W q. K1 T,W k. = -WW,
K [K', K2), V = [V', K]]
A = Softmax( Q K
1 = (rAIopK(A)
Pre-fetch( Cx2)
0 AVW.
Output: O E lxd
Algorithm 3 Decoding
Input: T (Te, TT,T R2xd
Q, = TW q. K₁ = TW V, = TW,
K- Quant( K2), V = (u,
K = [K'U KK,, Ki), V-V'UR Vdl
A MuskedSofimux((.. (Q,KT)
K++1 ArgTopK(AA,, (A..)
Pre-fetch((Cxer)
C. Ket -f= (V K;}
C = CC, (..ooodd(.)) C² |C', C.
O = AVW.
Output: O E 22xd
B. Setting of Heeyll-in-a-Haystack Benchmark
We notice that the NIAH benchmark has many different versions. We follow the setting from Greg Kamradt (2023), inserting the sentence "The best thing to do in San Francisco is eat switch and sit in Dolores Park sunny day into Paul Graham's ss essays, and added the question. "What is the best thing to do in San Francisco? Here is the most relevant sentence in the contex:: After that, we use GPT-4 to score the model's 's response based on the following criteria:
Score 1:
The answer is completely unrelated to the reference.
Score 3:
The answer has minor relevance but does not align with the reference.
Score 5: The answer has moderate relevance but contains inaccuracies.

---

SpeCache: Speculative Key-Value Caching for Efficient Generation of LLMs
Score 7: The answer aligns with the reference but has minor omissions.
Score 10: The answer is completely accurate and aligns perfectly with the reference.
C. Full Results on LongBench
Due to space constraints, we only report the results for tasks from LongBench in the main text. The results for all 15
tasks shown in the table below. Notably, both KIVI and SPECACHE use 2-bit and 1-bit quantization with g 64.
Method
Cache Size
KV
XaurativoQA
Qaser
MF-2n
HoppoQA
2W16MON
Musique
QMSuu
MolniNews
TREC
TevaaQA
SAMSum
PRO
LCC
RB-F
Average
cache
1.00x
26.9
33.1
27.3
42.8
51.4
InfLLM
0.25x
20.9
16.8
38.4
33.9
19.2
18.2
29.6
22.2
24.7
60.5
88.3
41.3
41.4
52.8
55.3
37,5
Mistal Baaral-7B-Jaa 13-IInstrutt-0.22
L.
0.25x
19.7
15.1
25.4
27.7
17.4
14.6
27.4
20.2
22.1
61.0
83.7
39.2
31.6
51.8
55.9
34.2
H₂O
0.25x
21.6
24.9
44.8
35.0
19.0
17.5
28.6
22.8
24.2
71.0
86.7
43.8
82.9
53.9
52.3
41.9
KIVI (2-bi0
0.16x
27.0
31.6
47.5
41.9
28.0
18.8
31.5
23.6
26.6
71.0
86.2
43.6
68.6
52.1
50.6
43.2
SPECACHE
27.2
49.1
28.2
27.1
27.1
42.8
InfLLM
0.13x
20.9
12.8
33.0
29.1
16.2
13.3
27.9
21.2
23.8
60.0
86.0
40.1
26.2
54.1
53.5
34.5
StreamLLM
0.13x
16.8
11.3
22.9
22.8
12.0
10.7
24.6
19.8
19.7
56.5
79.6
38.8
16.9
56.0
30.8
H₂O
0.13x
20.9
21.4
41.1
32.8
16.8
15.9
26.2
22.6
23.0
71.0
86.3
43.8
79.5
51.8
40.4
(I-b()
21.7
23.1
22.7
34.0
32.6
SPECACHE
26.7
Full KV Cache
1.00x
21.7
44.3
44.4
46.6
37.0
21.5
30.0
22.6
27.7
74.5
90.6
42.7
67.0
57.1
51.4
45.3
InfLLM
0.25x
18.1
28.0
35.1
36.6
23.0
14.4
29.6
19.8
25.5
61.0
88.8
42.0
37.5
56.9
58.3
38.3
M 1-M----5B-Instreet Instrect
StreamLLM
0,25
17.4
23.0
21.1
29.8
24.7
12.0
25.9
19.5
22.6
60.5
85.7
40.5
21.0
35.0
57.2
34.4
H₂O
0.25x
21.8
41.2
41.8
46.8
36.9
21.5
25.7
21.4
23.7
74.0
90.6
42.4
66.0
55.1
51.2
44.0
KTVT (2-bit)
0,17x
21.0
43.4
44.2
46.0
36.5
20.6
29.4
21.9
27.2
74.0
90.1
42.9
65.0
53.0
50.9
44.4
SPECACHE
0,17
20.9
43.7
45.1
46.4
37.0
21.0
29.7
22.7
27.7
74.5
90.1
41.9
67.0
$7.0
50.4
45.0
InfLLM
0,13x
14.2
22.0
27.0
33.1
20.5
9.3
27.5
19.1
24.4
58.0
82.0
40.9
18.0
61.4
58.4
34.4
Streamt LM
0.13x
13.1
16.6
17.4
25.7
18.5
9.8
23.4
18.2
19.9
55.5
72.3
39.9
8.0
60.4
55.8
30.3
H&O
0.13x
21.3
37.8
42.1
46.6
36.9
21.5
23.7
21.1
21.9
74.0
90.5
42.9
65.5
54.6
50.8
43.4
(1-b6)
0.11x
18.6
23.9
29.6
39.8
25.5
15.7
4.9
17.5
5.5
66.0
67.1
34.8
62.3
17.8
22.0
30.1
SPECACHE
0.11x
21.0
43.2
45.6
46.2
36.9
20.7
27.9
21.3
27.1
74.0
90.4
40.1
65.0
55.0
51.0
44.4
12


Reviews:
[{"review_id": "mjXhQgJNEt", "reviewer": "Reviewer_5qwx", "summary": "This paper proposes storing low-bit KV on the GPU while offloading full-precision KV. Attention is performed between the overall full-precision KV of the top-k keys selected from the low-bit keys. \nAdditionally, the paper introduces the use of speculative tokens to speculatively prefetch the KV cache needed for the next token. Speculative tokens have been shown to be beneficial for reducing latency.", "questions": "", "limitations": "", "rating": 2, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We thank the reviewer for their thoughtful feedback! We address specific concerns and questions below.\n\n\n> Q1. Why the 1-bit improvement is so large? Performance of KVQuant + SpeCache.\n\n+ The improvement of SpeCache on 2-bit KIVI is relatively small because the performance of 2-bit KIVI on longbench is already close to that of 16-bit KV Cache, which can be considered as the performance upper bound after KV Cache compression. As a result, there is limited optimization space left. On the more complex RULER benchmark, SpeCache can provide a significant improvement to 2-bit KIVI (see Q5).\n+ In our comparison, all instances of \"1-bit KIVI\" refer to our improved KIVI implementation described in Section 3.4, since original KIVI can hardly work with 1-bit (see Table 5). The larger improvement of SpeCache on the improved 1-bit KIVI is due to the fact that the improved 1-bit KIVI is still far from the upper bound, leaving more room for optimization.\n+ Since the official KVQuant code does not include 1-bit quantization, we compared the performance of SpeCache with 2-bit KVQuant. We conducted experiments on LLaMA-2-7B and measured the PPL on Wikitext-2. The results show that SpeCache also improves the performance of KVQuant.\n\n|Method|PPL|\n|-|-|\n|Full KV|5.12|\n|2-bit KVQuant|7.03|\n|+ SpeCache|5.37|\n\n\n> Q2. Does this speculative token work layer by layer? What is pre-generated?\n\nThe pre-decoding phase is a single inference step we introduce between prefilling and decoding. Its purpose is to generate the first speculative token $T’_2$. During pre-decoding phase, $T_1$ is the model input, and the entire model output (i.e., the predicted next token) is used as $T’_2$. In the first decoding step, $T’_2$ is used alongside the $T_1$ as the query, and $T’_2$ will record the top-k KV pairs from each attention layer by layer during decoding, in order to prefetch them for use in the next step.\n\n> Q3. Is this throughput run with flash decoding?\n\nFor Full KV Cache, we consistently use Flash-Attention2. For SpeCache and KIVI, during the prefilling phase, we also use Flash-Attention2. However, in the decoding phase, due to the involvement of low-bit operations, we use the low-bit CUDA operators provided by KIVI to compute attention.\n\n> Q4. Throughput comparison between SpeCache and KIVI.\n\nThank you for your suggestion. Here, we first provide a throughput comparison between SpeCache and KIVI with context length of 32k on Mistral-7B-Instruct-v0.2.\n\n|Method|Throughput (tok/sec)|Performance on Longbench|\n|-|-|-|\n|Full KV|10.3|42.3|\n|2-bit KIVI (g=64)|36.5|39.7|\n|+ SpeCache|34.6|42.0|\n|1-bit KIVI (g=64)|50.8|28.9|\n|+ SpeCache|47.3|40.4|\n\nSpeCache only slightly reduces the throughput of KIVI. Since the computation of speculative tokens is parallelized with the output tokens, the simultaneous decoding of both tokens does not significantly increase the latency.\n\n> Q5. Add more experiment results on RULER.\n\nWe evaluate LLaMA-3.1-8B-instruct on RULER with a 32k context length, and the results are as follows:\n\n|Method|KV Size|N-S1|N-S2|N-S3|N-MK1|N-MK2|N-MK3|N-MQ|N-MV|QA-1|QA-2|VT|FWE|Average|\n|-|-|-|-|-|-|-|-|-|-|-|-|-|-|-|\n|16-bit Full KV|1.00x|100.0|100.0|100.0|99.8|99.6|99.6|98.3|98.7|76.8|50.2|98.3|87.1|92.4|\n|2-bit KIVI (g=64)|0.16x|100.0|98.6|93.8|98.8|96.0|66.2|95.7|96.8|72.8|48.2|86.8|86.1|86.7|\n|+ SpeCache|0.16x|100.0|99.8|99.6|100.0|99.0|95.8|98.9|99.0|75.0|49.4|95.7|87.3|91.6|\n|1-bit KIVI (g=64)|0.10x|11.4|12.0|1.8|7.4|2.0|0.0|1.3|0.5|38.0|25.0|2.0|20.1|10.1|\n|+ SpeCache|0.10x|100.0|94.6|94.4|97.2|65.6|34.0|80.6|82.5|59.0|34.7|74.8|74.5|74.3|\n\nWe find that 2-bit KIVI experiences significant performance degradation on RULER compared to Full KV, especially on the N-MK3 and VT tasks. However, SpeCache effectively mitigates this issue, keeping the performance gap from Full KV within 1%. While 1-bit KIVI completely fails on all tasks in RULER, SpeCache provides a significant performance boost and even reaches the Full KV cache level on certain tasks, such as N-S1.\n\n> Q6. Add a cache hit rate for speculative tokens.\n\nWe evaluate 2-bit SpeCache (g=64) and Mistral-7B-Instruct-v0.2 on two long sequence generation tasks, MultiNews and GovReport in Longbench. We track two metrics:\n+ Speculative Token Exact Hit Rate: The proportion of speculative tokens that perfectly match the next output token.\n+ Speculative Token Top-K KV Cache Hit Rate: The proportion of the top-k KV cache needed for the next output token that is hit by the top-k KV cache of the speculative token.\n\n|dataset|Exact Hit Rate|Top-K KV Cache Hit Rate|\n|-|-|-|\n|MultiNews|57.5|91.3|\n|GovReport|57.6|90.9|\n\nAlthough the exact hit rate is only around 57%, SpeCache does not directly output speculative tokens. Instead, it uses them as a medium to guess the top-k KV cache. The top-k KV cache hit rate is above 90%, which ensures the effectiveness of the speculative tokens.", "claims_and_evidence": "Most of the claims of this paper are well-supported.\nI have one question about table 3: is this throughput run with flash decoding? \nI am worried about the slow implementation of torch attention and the overhead burden of two times attention and cpu-gpu io.", "methods_and_evaluation_criteria": "see claims.", "theoretical_claims": "NA.", "experimental_designs_or_analyses": "see claims.", "relation_to_broader_scientific_literature": "na.", "essential_references_not_discussed": "na.", "other_strengths_and_weaknesses": "Strengths:\n\n1. This paper is well written with a neat idea.\n\n2. Experiments demonstrate that this method can effectively improve throughput.\n\n\nWeakness:\n\n1. The improvement w.r.t 2bit KIVI is relatively small, less than 1.5 points. I do not know why the 1-bit improvement is so large. is that due to the 1-bit kivi implementation? I do not think 1 bit is a common setting for kv quant. What is the performance of KVQuant + spec cache?\n\n2. How speculative tokens T2' generated is very confusing. I think it is the main part of this work. Does this spec token work layer by layer? What is pre-generated?\n\n3. I have one question about table 3: Is this throughput run with flash decoding? \nI am worried about the slow implementation of torch attention and the overhead burden of two times attention and cpu-gpu io.\n\nOther Suggestions:\n\n1. I suggest that the authors should add a Pareto optimal curve comparison, where the y-axis represents benchmark performance and the x-axis represents the throughput of different methods. \nThis is because Table 3 does not show a throughput comparison between the authors' method and KIVI, despite KIVI achieving almost the same performance as the authors' results.\n\n2. Add more experiment results on RULER.\n\n3. Add a cache hit rate for speculative tokens.", "other_comments_or_suggestions": "table 3: throughput -> throughput (tokens/sec)", "questions_for_authors": "see weakness.\n\nIf the author answer these questions during rebuttal, I will improve 2->3.", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "Y5RNoVUITa", "reviewer": "Reviewer_6JBf", "summary": "This paper presents SPECACHE, a novel method to address the memory bottleneck caused by key-value (KV) caches in large language models when processing long sequences. The authors propose a training-free approach that offloads the complete KV cache to CPU memory while maintaining a low-bit copy in GPU VRAM. The key innovation is a speculative mechanism that predicts which KV pairs will be most relevant for the next token, allowing these to be prefetched from CPU to GPU memory in parallel with ongoing computations. This technique avoids both the information loss associated with compression methods and the latency penalties from offloading approaches. The authors evaluate SPECACHE on LongBench and Needle-in-a-Haystack benchmarks, demonstrating that it can maintain performance comparable to full KV cache while using only 10% of the GPU memory, enabling up to 12x larger batch sizes and 4.6x higher throughput.", "questions": "", "limitations": "", "rating": 4, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We thank the reviewer for their thoughtful feedback.\n\n> Q1. How sensitive is SpeCache to changes in the distribution of attention patterns?\n\nWe add the phrase \"How many paragraph are there in the article? Translate the first sentence into German.\" to the end of a 15k-token text composed of several of Paul Graham's essays, then input it into Mistral-7B-Instruct-v0.2, and observe the output token and speculative token generated via 1-bit SpeCache (black bold text represents the erroneous prediction of speculative tokens):\n+ Output token:\n>> There are 13 paragraphs in the article above. The first sentence in German translates to \"**Es gibt zwei deutlich unterschiedliche Arten, politisch mittler zu sein: vorsätzlich und zufällig. Absichtslose Mitte stehen dafür, dass die Extreme etwa gleich weit entfernt** sind.\" (There are two distinct ways to be politically moderate: intentionally and by accident. Accidental moderates are...\n+ Speculative token:\n>> are 13 paragraphs in the article above. The first sentence in German translates to \"**Ichlieberhnte Wegeliche Arten, politisch mittellangigeben: zufzlich und zufällig.\" (There are \"Intentionenander aus eigentlich Menschen, wenn sie beideehentenfert** sind.\" (There are two distinct ways to be politically moderate: intentionally and by accident. Accidental moderates are...\n\nWe find that when switching languages and topics, speculative tokens diverge from the output tokens within some decoding step. However, as explained in our response to Q2, SpeCache does not rely on the exact match of speculative tokens. It only requires that speculative tokens have a high top-k KV cache hit rate for the next output token.\n> Q2. Accuracy of the speculation mechanism.\n\nWe evaluate 2-bit SpeCache (g=64) and Mistral-7B-Instruct-v0.2 on two long sequence generation tasks, MultiNews and GovReport. We track two metrics:\n+ Speculative Token Exact Hit Rate: The proportion of speculative tokens that perfectly match the next output token.\n+ Speculative Token Top-K KV Cache Hit Rate: The proportion of the top-k KV cache needed for the next output token that is hit by the top-k KV cache of the speculative token.\n\n|dataset|Exact Hit Rate|Top-K KV Cache Hit Rate|\n|-|-|-|\n|MultiNews|57.5|91.3|\n|GovReport|57.6|90.9|\n\nAlthough the exact hit rate is around 57%, SpeCache does not directly output speculative tokens. Instead, it uses them as a medium to guess the top-k KV cache. The top-k KV cache hit rate is above 90%, which ensures the effectiveness of the speculative tokens.\n> Q3. SpeCache requires running two forward passes per token generation. Have you explored distilling a smaller model?\n+ Throughout the inference process, we only add one step of pre-decoding, and in the subsequent decoding phase, SpeCache runs **only one forward pass for each token generation**. This is because SpeCache uses both the Output token (e.g., $T_1$) and the Speculative token (e.g., $T'_2$) as input (i.e., [$T_1$, $T'_2$]) for decoding, which generates two tokens (i.e., $T_1$ generates $T_2$, $T'_2$ generates $T'_3$) in a single forward pass. These tokens are then used as input for the next step, with $T_2$ serving as the model's official output.\n+ This is also the reason why the generation of the speculative token is a “free lunch” attached to the forward pass of the output token. Therefore, there is almost no additional overhead, and no need to distill an extra model to generate the speculative token.\n\n> Q4. How would the performance change with different hardware setups.\n\nDue to the varying computational capabilities of different GPUs and the differences in the optimization levels of their operators, it is challenging to conduct a direct ablation study on bandwidth. Therefore, we perform some theoretical analysis to address this. For example, on Mistral-7B-Instruct-v0.2, when the context length is 2K, the maximum batch size for 1-bit SpeCache supported by a 48GB GPU memory is 410. This requires transmitting 3.4GB of KV cache during each decoding step. Theoretically, since SpeCache can fully parallelize data transfer and computation, the decoding latency is 775ms/step. In other words, under optimal conditions, as long as the bandwidth is above 4.4GB/s, the decoding latency of SpeCache will not significantly change.\n> Q5. Theoretical or empirical upper bound on the batch size where \"decoding latency < fetching latency\" no longer holds?\n\nAs the batch size increases, the fetching latency grows linearly, while the decoding latency increases sub-linearly due to improved computation parallelism. This results in the overall decoding latency being lower than the latency for fetching the entire KV cache when the batch size is large. Empirically, we have observed the following behavior: For Mistral-7B-Instruct-v0.2 on an A6000 GPU, with a context length of 2K, when batch size >= 4, fetching latency > decoding latency. Additionally, when the context lengths are 8K or 32K, the boundary conditions shift to 2 and 1, respectively.", "claims_and_evidence": "The authors' primary claims are well-supported by empirical evidence:\n* The claim that KV cache is a memory bottleneck is substantiated with concrete examples in the introduction (page 1, paragraph 2), where they show that for LLaMA 2-7B processing sequences of length 2k with batch size 16, the KV cache size exceeds the model's parameter count.\n* The assertion that attention in LLMs is sparse (page 2, paragraph 1) is backed by both references to existing literature and their own analysis in Figure 2 (left), which demonstrates that only 0.5% of keys can cover 90% of a query's attention.\n* The claim that SPECACHE maintains model performance while significantly reducing memory usage is well-supported through extensive experiments across multiple models (LLaMA-2, LLaMA-3, and Mistral) on LongBench in Table 1 and Table 2. For instance, with Mistral-7B-Instruct-v0.2, they show a performance gap of only 2% compared to the 16-bit baseline while retaining only 10% of the KV cache size (page 7, paragraph 2).\n* The throughput improvements claimed (up to 4.6×) are clearly demonstrated in Table 3 with detailed measurements across different context lengths and batch sizes.\n\nHowever, I note that the claim about \"avoiding inference latency caused by CPU-GPU communication\" (page 2, paragraph 2) is slightly overstated. The method mitigates rather than eliminates latency, as shown in Table 4 where parallel prefetching reduces but doesn't eliminate the latency overhead.", "methods_and_evaluation_criteria": "The methods are sound and the evaluation criteria are appropriate for the research question:\n* The authors use a comprehensive evaluation approach, testing on the established LongBench benchmark (covering 15 diverse tasks) and the Needle-in-a-Haystack task for specific evaluation of long-context retrieval ability.\n* The baseline comparisons are thorough and fair. On page 6-7, the authors compare SPECACHE with several state-of-the-art methods including InfLLM, StreamLLM, H2O, and KIVI with varying compression ratios.\n* The experimental setup is clearly described (page 5, section 4.1), specifying implementation details like the number of residual KV pairs and quantization group sizes.\nI appreciate the realistic evaluation of throughput (Table 3) across different context lengths, which directly addresses the practical utility of the method. One minor issue is that the authors don't explicitly report statistical significance for their results, though the consistent improvements across multiple models and tasks suggest the findings are robust.", "theoretical_claims": "The authors make several theoretical claims that I've verified:\n* The analysis of attention sparsity in Figure 2 (left) is sound and aligns with previous findings in the literature. The comparison between query-dependent top-k attention and greedy cache eviction correctly illustrates why dynamic prefetching is necessary.\n* The asymptotic analysis of CPU-GPU transfer time (Figure 2, right) correctly shows the linear relationship between transfer size and latency, providing a theoretical foundation for why prefetching only the most important KV pairs is beneficial.\n* The assertion that LLM inference is memory-IO bound rather than compute-bound (page 3, end of section 2.2) is supported by both citations and their own measurements, justifying why simultaneous decoding of output and speculative tokens doesn't significantly increase latency.\n* The improved 1-bit quantization method (page 5, section 3.4) is theoretically sound, with the modified zero-point and scaling factor ensuring better approximation for uniform distributions.", "experimental_designs_or_analyses": "I checked several aspects of the experimental design and analyses:\n* The evaluation on LongBench (Tables 1 and 2) is comprehensive and well-executed, covering multiple models and settings. The authors consistently report average performance across tasks, which gives a clear overall picture.\n* The ablation studies (Figures 4 and 5, Table 4 and 5) are well-designed to isolate the impact of specific components. For example, the ablation on 'k' (number of prefetched KV pairs) in Figure 5 shows the trade-off between performance and transfer size.\n* The comparison with non-speculative fetching (Table 4) effectively demonstrates the advantage of parallelizing prefetching and computation, showing both performance and latency metrics.\n* The throughput measurements in Table 3 appropriately use maximum batch sizes that the GPU memory can handle, providing a realistic assessment of the method's practical benefits.\nOne analysis that could be improved is the mechanism for selecting the speculative token. On page 4, the authors mention using the output token to compute a speculative token, but don't fully explain how this approximates the next token. More details on this approximation would strengthen the paper.", "relation_to_broader_scientific_literature": "N/A", "essential_references_not_discussed": "N/A", "other_strengths_and_weaknesses": "Strengths:\n* The proposed method is training-free and can be applied to existing pre-trained models without modification, enhancing its practical utility.\n* The paper demonstrates impressive results across multiple models (LLaMA-2, LLaMA-3, Mistral) and a range of tasks, showing the broad applicability of the approach.\n* The improvement to 1-bit KIVI quantization (page 5, section 3.4) is a valuable contribution in itself, enabling much higher compression ratios than previously possible.\n* The method elegantly leverages the memory-IO bound nature of LLM inference to perform speculative decoding with minimal overhead.\n* The approach is particularly impactful for long contexts and larger batch sizes, where memory constraints are most significant (Table 3).\n\nWeaknesses:\n* On page 4, paragraph 2, the paper states \"speculative tokens may be less accurate for output.\" However, there's no analysis of how often the speculative token differs from the eventual output token, which would help readers understand the approach's limitations.\n* The implementation relies on PyTorch's multi-stream mechanism, which the authors acknowledge is not theoretically optimal (page 8, paragraph 1). More details on how a custom implementation could further improve efficiency would strengthen the paper.\n* While the authors show SPECACHE works well with KIVI quantization, they don't explore compatibility with other quantization methods like KVQuant or ZipCache, which might provide further improvements.\n* The paper lacks discussion of potential failure cases or limitations, such as texts with rapidly changing topics where attention patterns might be less predictable.\n* The evaluation focuses exclusively on English text processing. Testing on multilingual settings would provide a more comprehensive assessment of the method's robustness.", "other_comments_or_suggestions": "1. In Figure 3, the illustration is helpful but the distinction between \"To be prefetched\" and \"To be quantized & offloaded\" arrows could be clearer.\n2. The terminology \"SPECACHE\" is used inconsistently throughout the paper - sometimes capitalized, sometimes not (e.g., \"SpeCache\" in Figure 4 caption).\n3. On page 8, the reference to \"Table 4\" in the Ablation section should specify which aspect of Table 4 is being discussed.\n4. The paper would benefit from a brief discussion of any overhead in terms of additional CPU computation or memory requirements for managing the offloaded KV cache.\n5. Some statements like \"without the need for retraining\" are repeated multiple times throughout the paper and could be streamlined.", "questions_for_authors": "1. How sensitive is SPECACHE to changes in the distribution of attention patterns? For instance, if the text suddenly changes topic or language, how quickly can the prefetching mechanism adapt?\n2. In your experiments, how often did the speculative token differ from the actual next token, and how did this affect the quality of prefetching? This analysis would help quantify the \"accuracy\" of your speculation mechanism.\n3. Your method requires running two forward passes per token generation. Have you explored distilling a smaller model to generate the speculative token, which might reduce computation while maintaining prefetching quality?\n4. The CPU-GPU communication is a critical aspect of your approach. How would the performance change with different hardware setups (e.g., PCIe 3.0 vs 4.0, different CPU-GPU bandwidth configurations)?\n5. In Figure 2 (middle), you show that decoding latency increases with batch size but remains below fetching latency. Is there a theoretical or empirical upper bound on the batch size where this relationship no longer holds?", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "ChSOoGn1ZR", "reviewer": "Reviewer_KbMF", "summary": "This paper proposes to offload the KV cache to CPU memory and retrieve KV pairs based on the similarity between the query of a speculative token with quantized KV pairs.", "questions": "", "limitations": "", "rating": 3, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We thank the reviewer for their thoughtful feedback! We address specific concerns and questions below.\n\n> Q1. Discuss the difference between SpeCache and QUEST and the contribution of SpeCache.\n\nAlthough both QUEST and SpeCache focus on how to accurately select sparse KV pairs, their emphasis differs due to the distinct scenarios they address.\n\n+ QUEST aims to reduce computation and the loading of KV cache **from GPU HBM** by reducing the number of KV pages involved in the attention computation, thereby accelerating the model. In contrast, SpeCache focuses on how to predict and load important KV pairs in parallel when the KV cache is **offloaded to lower-level storage devices (e.g., CPU memory)** for GPU memory saving. Since the CPU-GPU bandwidth (e.g., 16GB/s) is much smaller than the GPU HBM bandwidth (e.g., 1.5TB/s), SpeCache selects a much smaller number of KV pairs to load (64) compared to QUEST (over 1024), and focuses on parallel loading and computation.\n\n+ From the experimental results, SpeCache demonstrates a more efficient use of sparse KV pairs. According to the results in QUEST, when the KV budget is less than 1024, model performance significantly declines on longbench (Fig. 7 of the QUEST paper). However, SpeCache performs well even when only 32 KV pairs are loaded (Fig. 5 in our paper). This is because the low-bit KV cache copy we employ can distinguish important KV pairs at a finer granularity, and the low-bit KV cache itself provides some coarse-grained information during decoding.\n\n> Q2. As the sequence becomes longer, will the speculative token be less accurate?\n\nWe evaluate 2-bit SpeCache (g=64) and Mistral-7B-Instruct-v0.2 on two long sequence generation tasks, MultiNews and GovReport in Longbench. We track the Speculative Token Top-K KV Cache Hit Rate -- the proportion of the top-k KV cache needed for the next output token that is hit by the top-k KV cache of the speculative token. We separate calculated the hit rate at different decoding step intervals.\n\n|dataset|[1, 10]|[11, 50]|[51, 100]|[101, 200]|[201, ]\n|-|-|-|-|-|-|\n|MultiNews|95.9|92.1|89.1|88.1|87.5|\n|GovReport|97.0|92.5|86.9|87.0|86.8|\n\nWe find that as the output grows, the hit rate gradually decreases, but after dropping to around 87%, it stabilizes and maintains a relatively high hit rate. \n\n> Q3. Does the speculative decoding introduce more latency and GPU memory usage?\n\nSince both tokens share the same model weight matrices and KV cache during decoding, speculative decoding neither introduces additional latency from loading pages into the GPU HBM nor increases VRAM usage. Furthermore, as the decoding process is IO-bound and GPU units are not fully utilized, the added computational load does not significantly increase the latency. We verify this through experiments. On Mistral-7B-Instruct-v0.2, we test the latency and VRAM usage of KIVI and KIVI+SpeCache with a batch size of 1 and a context length of 64k.\n\n|Method|Latency (ms/step)|Allocated memory (GB)|\n|-|-|-|\n|Full KV|204.3|31.1|\n|2-bit KIVI (g=64)|94.6|22.8|\n|+ SpeCache|103.6|22.8|\n|1-bit KIVI (g=64)|94.3|22.3|\n|+ SpeCache|101.3|22.3|\n\nWe can find that SpeCache only slightly increases the latency of KIVI.", "claims_and_evidence": "This paper has two claims: 1) Attention is sparse while each token requires different KV pairs. It emphasizes the importance of KV cache offloading. 2) The CPU-GPU communication significantly increases inference latency. These claims are supported by experiments in Fig.2", "methods_and_evaluation_criteria": "The paper proposes to decode an additional speculative token and load KV pairs into the GPU memory based on the speculative token before inference. The method is mainly evaluated on the LongBench dataset.", "theoretical_claims": "As far as I see, there are no theoretical claims in this paper.", "experimental_designs_or_analyses": "I checked the experimental results provided in this paper. The experiments are conducted to compare the proposed method with KV cache compression methods with ablation study about the proposed method. However, offloading KV cache to CPU memory is not a new idea, the comparison between the proposed method and the previous methods about KV cache offloading is lacked.", "relation_to_broader_scientific_literature": "This paper focuses on the KV cache offloading technique. There is no relation to broader scientific literature.", "essential_references_not_discussed": "While this paper has cited previous works like [1], the discussion of the difference between the proposed method and previous works is not thorough. Offloading the KV cache to CPU memory is not a new idea. Previous works like [1] also select significant KV pairs based on reduced keys. A discussion of the difference between the proposed method and previous works is lacking.\n\n[1] Tang,J.,Zhao,Y.,Zhu,K.,Xiao,G.,Kasikci,B.,andHan,S. QUEST: Query-aware sparsity for efficient long-context LLM inference. ICML,2024.", "other_strengths_and_weaknesses": "Strength:\n\nThe speculative token decoding for KV cache preloading seems to be an interesting and novel approach. Generally, the paper is clear and easy to follow.\n\nWeakness:\n\nWhile offloading the KV cache to CPU memory is not a new idea, there seem to be no experiments comparing the proposed method and previous KV cache offloading methods. While previous work[1] has proposed selecting KV pairs based on reduced keys, a more thorough discussion of the difference and the contribution of this paper is needed. \n\n[1] Tang,J.,Zhao,Y.,Zhu,K.,Xiao,G.,Kasikci,B.,andHan,S. QUEST: Query-aware sparsity for efficient long-context LLM inference. ICML,2024.", "other_comments_or_suggestions": "I would like to see more experiments and discussion about the difference between the SpeCache and previous KV cache offloading methods. The main contribution of the proposed method is introducing speculative decoding so that the selected KV pairs can be loaded before the decoding step. However, I still have doubts about the improvement of the proposed method over previous methods like QUEST. I look forward to further reply from the authors.", "questions_for_authors": "1) Since each speculative token is decoded based on the previous speculative token, the error may accumulate. As the sequence becomes longer, will the speculative token be less accurate? \n\n2) While it is said to decode two tokens in parallel, does the speculative decoding introduce more latency and GPU memory usage? Are there any experimental results regarding the cost brought by speculative decoding?", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "xd6flshnxr", "reviewer": "Reviewer_xLPX", "summary": "The paper introduces SpeCache, a speculative KV caching mechanism designed to enhance the efficiency of LLM inference. SpeCache mitigates these drawbacks by offloading the complete KV cache to CPU memory and dynamically fetching KV pairs back into GPU memory during decoding. To minimize CPU-GPU transfer latency, it employs a speculative mechanism to predict the KV pairs required for the next decoding step, allowing memory transfer and computation to proceed in parallel. The method does not require retraining and significantly reduces memory usage while maintaining competitive accuracy.\n\n## update after rebuttal\n\nThe authors’ response partially addresses my concerns. However, based on the discussions raised by other reviewers and overall rebuttal, I believe that my key concerns remain unresolved. Below, I elaborate on the remaining issues:\n\nWhile SpeCache proposes a KV cache offloading technique to address the long-context problem, the motivation described in the paper does not adequately support this objective. Since the method is based on offloading KV cache from the GPU to CPU memory (not merely loading from GPU DRAM), the examples given in the introduction and the maximum sequence lengths of the models evaluated in the experimental section do not convincingly demonstrate the utility of SpeCache in genuinely resource-constrained environments. As such, I find it difficult to accept the claim that SpeCache is effective under constrained memory settings.\n\nThis paper presents Table 1 and Table 2 as the main evaluation results, but there are still concerns about whether these comparisons are fair and appropriate.  In Table 1, the comparison with KIVI fixes the residual length at 64 and allocates 16-bit precision to the top-64 KV pairs. It is intuitive that assigning higher bit-precision to more important tokens would yield higher accuracy, making the comparison less informative. In Table 2, SpeCache is compared against baselines that adopt eviction strategies, while SpeCache itself stores the full KV cache in CPU memory (non-eviction). This setup favors SpeCache by design and does not ensure a fairness in accuracy comparisons.\n\nThe throughput evaluation also has room for improvement. The reported throughput gains are primarily achieved by increasing the batch size, which is similar to the benefits observed in prior work through KV cache size reduction. Moreover, the comparison is only made against FullKV, without including other CPU-GPU offloading methods. A more meaningful evaluation would involve direct comparisons with other CPU-GPU offloading methods, including the overhead from KV cache dequantization and other related costs.\n\nWhile I appreciate the authors’ additional efforts in providing further experimental results, the fundamental concerns I raised have not been fully resolved. Therefore, I will maintain my original score.", "questions": "", "limitations": "", "rating": 3, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We thank the reviewer for their thoughtful feedback! We address specific concerns and questions below.\n\n > Q1. Unclear Overhead of Parallel Prefetching and Computation\n\nSince our method is built upon KIVI, we can consider KIVI as the baseline when parallel pre-fetching and computation are not used. On Mistral-7B-Instruct-v0.2, we test the latency and VRAM usage of KIVI and KIVI+SpeCache with batch size of 1 and context length of 64k.\n\n|Method|Latency (ms/step)|Allocated memory (GB)|\n|-|-|-|\n|Full KV|204.3|31.1|\n|2-bit KIVI (g=64)|94.6|22.8|\n|+ SpeCache|101.6|22.8|\n|1-bit KIVI (g=64)|94.3|22.3|\n|+ SpeCache|103.3|22.3|\n\nIn addition, we also provide the throughput of Mistral-7B-Instruct-v0.2 with context length of 32k when the batch size is increased to the maximum capacity that can be accommodated by 48GB.\n\n|Method|Throughput (tok/sec)|batch size|\n|-|-|-|\n|2-bit KIVI (g=64)|36.5|22|\n|+ SpeCache|34.6|22|\n|1-bit KIVI (g=64)|50.8|36|\n|+ SpeCache|47.3|36|\n\nWe can find that SpeCache only slightly increase the latency of KIVI.\n\n > Q2. Questionable Relevance of the KV Cache Bottleneck Example\n\nThank you for pointing that out. Our example is indeed somewhat outdated. We will revise the description to specify the KV cache size for long sequences with a batch size of 1, such as for Mistral-v0.2-7B with a 32k context length, where the KV cache size exceeds 4B, and for Llama3.1 with a 128k context length, where the KV cache size exceeds 16B.\n\n > Q3. Peak Memory Usage Concern in Prefill Phase\n\nIn fact, during the prefilling phase, **we do not need to keep the entire KV cache in GPU memory**. This is because the computation, quantization, and offloading of the KV cache are done layer-by-layer. In other words, at any given moment, only the KV cache of a single layer needs to be fully stored in GPU memory. Before the KV cache of one layer is computed, the KV cache from the previous layer has already been quantized and offloaded. For example, Mistral-7B has 32 layers, so at any given time, we only need 1/32 of the 16-bit KV cache in GPU memory. This also allows SpeCache to handle context lengths that full cache cannot accommodate.\n\n > Q4. Experiments on LLaMA-3.1-8B-Instruct\n\n We implemented SpeCache on Llama-3.1-8B-instruct and evaluated it on longbench.\n\n|Method|KV Size|Qasper|MF-en|HotpotQA|2WikiMQA|Musique|GovReport|MultiNews|PRe|LCC|RB-P|Average|\n|-|-|-|-|-|-|-|-|-|-|-|-|-|\n|16-bit Full KV|1.00x|45.5|54.9|56.0|46.6|31.3|34.6|27.2|99.5|63.2|55.4|51.4|\n|H2O|0.13x|38.8|42.6|43.2|37.7|25.8|26.0|21.2|96.0|54.1|49.5|43.5|\n|1-bit KIVI (g=64)|0.10x|22.9|30.1|38.7|22.0|16.9|11.0|14.9|76.5|39.1|34.1|30.6|\n|+ SpeCache|0.10x|43.0|53.0|50.4|40.6|30.4|34.7|27.5|98.0|57.7|48.1|48.4|\n\nAs you expected, the performance of Full KV significantly improves compared to the 8K context at 128K context. Our method, with just 10% KV size, has a 3% gap compared to Full KV, but still shows a 17.8% improvement over the purely quantization method, KIVI.\n\n > Q5. Comparision with SnapKV\n\nWe added SnapKV as a baseline on longbench. Since SnapKV compresses the KV cache to a fixed size, to ensure a fair comparison, we dynamically calculated the SnapKV budget for each sample length, aligning its KV cache size with that of our method.\n\n|Model|Method|KV Size|Qasper|MF-en|HotpotQA|2WikiMQA|Musique|GovReport|MultiNews|PRe|LCC|RB-P|Average|\n|-|-|-|-|-|-|-|-|-|-|-|-|-|-|\n|Mistral-7B-Instruct-v0.2|SnapKV|0.10x|25.0|45.1|34.1|18.7|18.8|26.2|24.4|79.5|51.8|51.4|37.5|\n|Mistral-7B-Instruct-v0.2|1-bit SpeCache (g=64)|0.10x|31.1|49.6|43.9|26.9|18.3|27.8|26.7|76.8|52.3|50.4|40.4|\n|LLaMA-3-8B-Instruct|SnapKV|0.11x|39.1|42.9|46.2|36.1|21.8|23.2|21.5|66.0|55.1|50.2|40.2|\n|LLaMA-3-8B-Instruct|1-bit SpeCache (g=64)|0.11x|43.2|45.6|46.2|36.9|20.7|27.9|27.1|65.0|55.0|51.0|41.9|\n\nUnder the same KV cache size budget, SnapKV outperforms all other baselines. However, SpeCache still delivers superior performance. Our method is capable of working with an extremely small KV cache size (about 0.1x), whereas SnapKV tends to experience performance degradation under such conditions. This is because SnapKV compresses the KV cache all at once, which may result in the loss of information needed for subsequent tokens.\n\n > Q6. Compare throughput gains at a fixed batch size\n\nThank you for your suggestion. We have provided some relevant results in our response to Q1.", "claims_and_evidence": "The claims in the paper are generally supported by clear evidence, but there are some points that deserve closer examination. The following is an evaluation of the claims and the evidence provided in this paper:\n\n**1. Unclear Overhead of Parallel Prefetching and Computation**\n\n•\tThe paper states that \"prefetching and computation can occur in parallel, avoiding any increase in inference latency.\" However, while Table 4 presents the latency gain when fetching and computation are executed sequentially, it does not provide results on the additional overhead introduced by the parallel prefetching and computation framework compared to standard inference.\n\n•\tWithout explicit analysis of the potential trade-offs, it is unclear how much overhead the parallel execution introduces in real-world inference scenarios, particularly when considering factors such as kernel launch overhead, synchronization penalties, or impact on GPU utilization.\n\n\n**2. Questionable Relevance of the KV Cache Bottleneck Example**\n\n•\tThe introduction section states that for LLaMA-7B with a batch size of 16 and sequence length of 2k, the KV cache size reaches 8.4B parameters, and that this can be a bottleneck in memory-constrained environments like on-device inference.\n\n•\tHowever, in local on-device inference, single-batch inference is more common than large batch sizes. A more appropriate example would be a single-batch input with a sequence length of 128k, which better reflects real-world usage scenarios.\n\n•\tRecent models such as LLaMA-3.1-8B already support 128k sequence length with GQA, making the assumption that 2k is a long context less relevant. Furthermore, for a single batch with a 2k input length, the KV cache size is only about 0.26GB, which is unlikely to be a severe memory constraint in modern hardware setups. This weakens the claim that a 2k length input presents a significant long-context challenge.\n\n**3. Peak Memory Usage Concern in Prefill Phase**\n\n•\tOne limitation of SpeCache is that during the prefill phase, the entire KV cache must be deployed on the GPU at least once before any offloading occurs.\n\n•\tThis means that peak memory consumption is not reduced, which can be a critical constraint for on-device inference applications with limited memory.\n\n•\tThe paper does not discuss whether SpeCache enables inference in environments where the full KV cache would otherwise exceed memory limits, which is crucial for evaluating its feasibility in resource-constrained scenarios.", "methods_and_evaluation_criteria": "The paper introduces a novel speculative KV caching method, integrating CPU offloading with speculative token decoding. The evaluation is based on:\n\n•\tAccuracy on LongBench and Needle-in-a-Haystack benchmarks\n\n•\tMemory efficiency (compression ratio and VRAM usage)", "theoretical_claims": "The paper does not present a formal theoretical analysis but relies on empirical justification. The main theoretical intuition is that attention sparsity allows selective KV caching without performance loss. This assumption is supported by quantitative studies of attention sparsity and cache hit rates.", "experimental_designs_or_analyses": "1. The LongBench dataset includes sequences exceeding 32K tokens. However, the evaluation in the paper truncates sequences to 4K, 8K, or 32K depending on the model's maximum context length. This truncation lowers the upper bound of accuracy for full KV cache models, potentially making the reported accuracy gap between full KV cache and compressed KV cache models appear smaller than it would be with longer sequences. A more appropriate evaluation should be conducted using a model that supports 128K sequences without truncation, such as LLaMA-3.1-8B-Instruct, to ensure fair comparisons.\n\n2. The paper evaluates SpeCache against older methods such as H2O and StreamLLM, but does not compare it with more recent and competitive methods like SnapKV [1].\n\n3. Table 3 demonstrates the throughput improvements of SpeCache by increasing the maximum batch size due to KV cache compression. However, this approach does not isolate the direct impact of KV cache compression on KV cache loading time. A better approach would be to compare throughput gains at a fixed batch size, measuring the speed-up factor when using full KV cache versus SpeCache. This would more clearly demonstrate how much KV compression reduces KV cache loading time rather than conflating improvements from increased batch sizes.\n\n[1] Yuhong Li, et al., \"SnapKV: LLM Knows What You are Looking for Before Generation\", arXiv:2404.14469.", "relation_to_broader_scientific_literature": "The paper builds on and extends existing work in:\n\n•\tKV cache compression \n\n•\tOffloading techniques\n\n•\tSpeculative execution for LLMs\n\nThe method is positioned as a training-free enhancement, making it broadly applicable.", "essential_references_not_discussed": "The paper discusses all major prior works related to KV caching, attention sparsity, and speculative execution. No significant omissions were noted.", "other_strengths_and_weaknesses": "**Strength**\n\n1. The paper presents a novel speculative prefetching mechanism for KV cache in LLMs, which differentiates it from conventional KV cache compression and offloading methods. \n\n2. Unlike previous methods that rely on compression techniques such as quantization, merging, or eviction, SpeCache introduces speculative tokens to anticipate the next KV pairs required for decoding, which is an innovative approach.\n\n**Weakness**\n\n1. Unclear assumption for long context (See \tClaims and Evidence 1. and 2.)\n\n2. Peak Memory Usage Concern (See Claims and Evidence 3.)\n\n3. Unreasonable experimental results (See Experimental Designs or Analyses)", "other_comments_or_suggestions": "Typo in line 302: “min X + 3 max X)” -> “(min X + 3 max X)”", "questions_for_authors": "See Claims and Evidence and Experimental Designs and Weakness.", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Meta-review: This paper introduced an interesting approach to reduce GPU memory during long-context LLM inference by offloading full-precision KV caches to CPU memory. All reviewers agreed that the proposed paper has a novel idea and results are reasonable. There were some disagreement on the on empirical validations. After reading the rebuttal, I believe concerns from reviewer 5qwx was addressed.
Decision: Accept (poster)