Compute or Load KV Cache? Why Not Both?
Authors: Shuowei Jin
Xueshen Liu
Qingzhao Zhang
Zhuoqing Mao
Venue: ICML 2025 poster (2025)
Keywords: Prefix caching System Optimization
KV Cache
Low-Latency LLM Inference
Primary area: general_machine_learning->hardware_and_software
TL;DR: Cake is a KV cache loading system that optimally balances computation and I/O resources through a bidirectional scheduling strategy, reducing Time to First Token (TTFT) by 2.6× on average and improving long-context LLM inference efficiency.

Abstract: Large Language Models (LLMs) are increasingly deployed in large-scale online services, enabling sophisticated applications. However, the computational overhead of generating key-value (KV) caches in the prefill stage presents a major bottleneck, particularly for long-context inputs. Prefix caching mitigates this issue by storing KV caches for reuse, reducing redundant computation. Despite its advantages, prefix caching suffers from high latency due to the limited I/O bandwidth of storage devices, constraining inference efficiency. To address this challenge, we introduce Cake, a novel KV cache loading system that optimally utilizes both computational and I/O resources in parallel. Cake employs a bidirectional scheduling strategy that dynamically balances KV cache computation and loading, ensuring efficient resource utilization. Additionally, Cake incorporates an adaptive scheduling mechanism that seamlessly integrates with non-prefix caching requests, improving system throughput and adapting to fluctuating resource availabilty. Through extensive evaluations across various hardware configurations, datasets, and storage conditions, Cake achieves on average 2.6× reduction in Time to First Token (TTFT) compared to compute-only and I/O-only methods. Our findings highlight Cake as an effective and practical solution for optimizing long-context LLM inference, bridging the gap between computation and I/O efficiency in large-scale AI deployments.

Compute Or Load KV Cache? Why not Both?
Shuowei Jin Xueshen Liu
Qingzhao Zhang' Z. Morley Mao
Abstract
of large-scale online services. As these models in- creasingly adopted, ensuring efficient online inference has emerged as critical research and engineering chal- lenge (Agrawal et al., 2024; Zheng et al., 2023; Miao et al., 2024; Leviathan et al., 2023; Ning et al., 2023; Jin et al., 2025). Recent advancements in LLMs, such as the ex- pansion of context windows (openAI, 2024; Anthropic, 2024), have enabled sophisticated applications including long document understanding (Wang al., 2024), long- context Retrieval-Augmented Generation (RAG) (Jiang et al., 2024), and the creation of complex LLM-based agents (Zhang et al., 2024). However, although these ca- pabilities improve utility, they also introduce significant computational overhead during inference, due to the cost of generating key-value (KV) caches. The resulting la- tency can noticeably degrade user experience. Given this computational bottleneck, strategies to optimize LLM in- ference workflows are essential.
Large Language Models (LLMs) are increasingly deployed in large-scale online services, enabling sophisticated applications. However, the compu- tational overhead of generating key-value (KV) caches in the prefill stage presents major bottle- neck, particularly for long-context inputs. Prefix caching mitigates this issue by storing KV caches for reuse, reducing redundant computation. De- spite its advantages, prefix caching suffers from high latency due to the limited I/O bandwidth of storage devices, constraining inference effi- ciency. To address this challenge, we introduce Cake, a novel KV cache loading system that op- timally utilizes both computational and I/O re- sources in parallel. Cake employs a bidirec- tional scheduling strategy that dynamically bal- ances KV cache computation and loading, ensur- ing efficient resource utilization. Additionally, Cake incorporates adaptive scheduling mech- anism that seamlessly integrates with non- non-prefix caching requests, improving system throughput and adapting to fluctuating resource availabil- ity. Through extensive evaluations across various hardware configurations, datasets, and storage conditions, Cake achieves on average 2.6x reduc- tion in Time to First Token (TTFT) compared to compute-only and 1/O-only methods. Our find- ings highlight Cake as an effective and practical solution for optimizing long-context LLM infer- ence, bridging the gap between computation and I/O efficiency large-scale Al deployments.
In real-world applications, many tokens are reused across users and conversations, presenting an opportunity for sys- tem optimization. For instance, multi-turn conversations, follow-up queries reuse the key-value (KV cache) pairs from prior tokens, while in Retrieval-Augmenter Genera- tion (RAG) workflows, document KV cache can be shared across multiple user queries (Jin et al., 2024; Chan et al., 2024). To reduce these redundancies, prefix caching stores previously computed KV cache and loads them into GPU memory for inference when the corresponding tokens are hit. Leading LLM service providers, including OpenAI, Anthropic, and DeepSeek, have integrated prefix caching mechanisms into their inference systems, lowering infer- ence costs by over 50% (OpenAI, 2024; Anthropic, 2023; Deepseek, 2024).
Despite its advantages, deploying prefix caching at scale poses key challenge: designing a high-capacity KV cache management system across heterogeneous memory layers while ensuring low loading latency for optimal Time-to- First-Token (TTFT) during inference. State-of-the-art in- ference engines (Kwon et al., 2023; Zheng et al., 2023) typically store KV cache in GPU and CPU memory, ensur-
1. Introduction
Large Language Models (LLMs) have become corner- stone of modern Al applications, powering wide range
'Equal contribution 'University of Michigan. Correspon- dence to: Shuowei Jin jjinsw@umich..om Xueshen Liu <li- uxs @ umich.edu>..
KV cache is widely used in -fate-of-the-as inference systems to reduce computational overhead during decoding for each quest.
Proceedings of the 1nnd International Conference Machine Learning, Vancouver, Canada. PMLR 267, 2025. Copyright 2025 by the author(s).

---

Compute Or Load KV Cache? Why not Both?
novel KV cache loading system designed minimize la- tency when loading KV cache from high-capacity, low- bandwidth storage layers. Cake optimally leverages the distinct characteristics of both computational and I/O re- sources in parallel, significantly reducing TTFT by 2.6x on average in long-context prefix caching scenarios.
P PROMPT]
Bandwidth: 2TB/s
In GPU
Yes
Start Decode
Size Limit: -80GB
Memory
4
Bandwidth: 25GB/s
In CPU Memory
Yes
Load KV from CPU Memory
In addition reducing latency, Cake is designed con- sidering real-world practicality. We propose an adaptive scheduling mechanism that efficiently shares compute sources with non-prefix caching requests, improving over- all throughput by 26%. Additionally, Cake dynamically adjusts to fluctuations in network and computational re- sources, ensuring consistently optimal latency.
Size Limit: -1 -1B8TB
No
Cake
Load KV from
Bandwidth: 0.5-4GB/s
In Disk
Yes
Lecal/Remote
Disk
Size Limit: -26TB
No
Compute KV
Furthermore, we conduct extensive experiments across di- verse setups and provide a detailed analysis of Cake's ef- fectiveness. Our findings offer practical insights into when Cake achieves the greatest performance gains, serving as a valuable guide for future real-world deployments.
Figure 1. Workflow of long-context LLM inference with prefix eaching. Cake operates during the KV cache loading phase (high- lighted in blue). The configuration parameters are based on the specifications of LambdaLab GPU server (Lambda, 2024).
To the best of our knowledge, Cake is the first system to demonstrate that efficiently utilizing both computational and I/O resources can optimally reduce TTFT in long- context prefix caching scenarios. Prior approaches ei- computation solely I/O for
ing minimal latency. However, GPU and CPU memory are both limited and expensive, often requiring the eviction of KV cache for long-cone nog-ccotext requests, making them imprac- tical for large-scale inference services. To address this lim- itation, hierarchical storage systems have been proposed, leveraging CPU memory, local disks, and remote storage for managing KV cache caches (Gao et al., 2024; Liu et al., 2023; Yao et al., 2024). Figure illustrates the hierarchical workflow of LLM inference with prefix caching. Upon re- ceiving a query, the system searches for reusable KV cache caches across three storage tiers:
ther rely exclusively KV cache loading (Liu et al., 2023; Kwon et al., 2023), leaving a significant gap in understanding how to leverage the unique characteristics of both computation and I/O for efficient KV cache cache loading. Cake addresses this gap, providing a comprehensive and adaptive solution for long- context prefix caching scenarios.
2. Related Work
GPU Memory:
The fastest but most capacity-limited option (bandwidth: ~2TB/s, size: ~80GB). If the KV cache cache is found, decoding proceeds immedi- ately.
Previous work has explored two primary directions to de- velop practical prefix caching systems: (1) algorithm-level compression techniques that reduce the size of KV cache, thereby decreasing loading time, and (2) system-level KV cache management strategies that expand cache capac- ity across heterogeneous memory layers.
CPU Memory: Offers greater capacity (bandwidth:
~25GB/ size: 1..TTB but is slower than GPU mem- ory.
Disk Storage: The largest capacity option (bandwidth: 0.5-4GB/s, size: 26TBB but also the lowest bandwidth. Compute: If no KV cache cache is found across all lev- els, the system computes it from scratch.
KV cache Compression. Most work compress the KV cache through quantization, token pruning, and model architectural modifications. Quantization methods (Hooper et al., 2024; Kang et al., 2024; Liu et al., 2024b) reduce the precision of KV cache representations while maintain- ing accuracy. Token pruning approaches like LLMLin- gua (Jiang et al., 2023), ScissorHands (Liu al., 2024a), and H2O (Zhang et al., 2023) identify and remove less im- portant tokens from the KV cache. At the model architec- ture level, Grouped-Quee 2023) reduce memory footprint by sharing key-value heads across queries in the group, while Multi-head Latent At- tention (MLA) compresses KV cache into compact latent vectors to reduce KV cache size.
Due to the substantial capacity differences in the storage hi- erarchy, a significant portion of KV cache caches reside on local or remote disks. As evaluated in AttentionStore (Gao al., 2024), approximately 80% of cache hits occur at the disk level. However, fetching large KV cache from disk is constrained by the low 1/O bandwidth of PCIe or network- based remote storage, significantly impacting TTFT, which adversely affects user experience.
Attention (GQA) (Ainslie et al.,
Contribution. In this paper, we introduce Cake (Computation and I/O Aware KV cache CachE loader), a
2

---

Compute Or Load KV Cache? Why not Both?
System Optimizations. vLLM (Kwon et al., 2023) and SGLang (Zheng et al., 2023) mainly optimize KV cache management between GPU DRAM and CPU memory, en- abling low-latency loading despite limited capacity. Cache- dAttention (Gao et al., 2024) extends this by employing a hierarchical KV cache management system across memory and disk mediums, effectively support multi-turn conver- sations. CacheGen (Liu al.. 2023) addresses scenarios where KV cache is stored in remote data storages, apply- ing adaptive compression methods to reduce the latency of loading through network.
For long-context inputs, the prefill stage is highly compute- intensive, requiring substantial GPU resources for a long period. Chunk prefill (Agrawal et al., 2023; 2024) mitigates this by dividing the input sequence into smaller chunks to prefill it chunk by chunk and batching small chunk pre- fill together with decode requests. To be specific, vLLM forms a batch of requests for each inference step based on predetermined token budget. The scheduler prioritizes decode requests, allocating one token from the budget to each. Any remaining tokens in the budget are then assigned to prefill requests for chunk prefill. This prevents long prefill operations from blocking decode requests, improv- ing GPU utilization by co-locating compute- -ouud (pre- fill) and memory-bound (decode) tasks in the same batch, Currently, it is the default suggested scheduling mode in vLLM (Kwon et al., 2023).
While these approaches focus on reducing the I/O load la- tency to reduce TTFT, our work tackles the problem from a different angle: combining computation and I/O paral- lel to further reduce latency. It is an orthogonal design to existing methods.
We design Cake upon this approach, scheduling chunks for either computation or I/O. Compared to token- level scheduling, chunk-level scheduling better exploits GPU parallelism for maximum efficiency. Compared to sequence-level scheduling. it is more fine-grained, allow- ing greater flexibility for optimal scheduling strategies.
3. Background
Attention and KV cache. The attention mecha- nism (Vaswani et al., 2017) is a core component of LLMMs, allowing them to model token relationships efficiently. KV cache technique to improve attention module in- ference efficiency.
4. Design of Cake
Given an input sequence X, the attention module first transforms it into queries, keys, and values:
We design Cake to optimize the latency of loading KV cache from storage layers with limited bandwidth but large capacity, such as local disks and remote storage -where most cache hits occur prefix caching scenarios. Cake leverages both compute and I/O resourees bidirec- tionally and in parallel minimize this latency. We present Cake workflow in Figure 2.
Q = =XW: K V=
where Wo. WK. and Wy are learned projection matrices. The attention scores are then computed
In this section, we first analyze the compute and I/O capa- bility in common inference server setups. Next, we present key insights into Cake 's scheduling design, focusing on how to efficiently utilize the unique characteristics of com- pute and I/O. Finally, we discuss real-world scenarios how Cake dynamically adapts to fluctuations in compute and I/O bandwidth, ensuring optimal latency.
QKT Vdl
Attention( (Q, (Q,KV) softmax
V
where de is the key dimension.
During autoregressive decoding, this computation repeats at each step 6, generating a new token Te based on previ- ous tokens Zzt. However, the key and value vectors for Tiit remain unchanged across decoding steps. To elimi- nate redundant computation, modern systems cache these as past K and past_V, V, computing only the query for along with the new key k; and value et. This optimization. known as KV cache, significantly reduces computational overhead.
Analysis of Compute and I/O Capability. We first analyze the performance characteristics of KV cache loading and computation across common inference system configurations. We evaluate the equivalent throughput (cal- culated as KV cache file size divided by processing time) using vL.LM (Kwon et al., 2023) with a chunk size of 512 tokens on the LongAlpaca-75 model (Chen et al., 2023). Our experiments use various GPUs to process random context of 32k tokens. A more detailed analysis across var- ious configurations is provided in Section 5.
Part 1:
Chunk Prefill. Chunk prefill is a technique used to opti- mize LLM inference, particularly for long input sequences. State-of-the-art LLM inference engines (Kwon et al., 2023; Zheng et al., 2023) split inference into two phases:
Our results in Figure 3 demonstrate that computation and I/O resources achieve comparable throughput in typical in- ference scenarios. Specifically, computing KV cache with
Prefill: Computes the KV cache for the input prompt.
Decode: Generates tokens sequentially using the cached KV, updating incrementally.
3

---

Compute Or Load KV Cache? Why not Both?
Compute
Unfinished Chunks
Loaded Chunks
I/O Load
Stage0:
E
ating
cake
makes
me
happy
enjoy
eating
cakes
Stage1:
E
ating
cake
makes
me
happy
enjoy
eating
cakes
StageN-1:
E
ating
cake
makes
me
happy
enjoy
eating
cakes
StageN
E
ating
cake
makes
me
happy
enjoy
eating
cakes
Computation starts from the beginning of the sequence, while loading starts from the end. Both processes progress in parallel and merge in the middle, ensuring efficient KV cache loading and minimal latency.
Figure Workflow Cake:
Equivalent Throughput (GB/s)
NVIDIA
700
KV Cache Memory Usage (MB)
4GB/s
4GB/5
100
NVIDIA H100
600
Step Time (ms)
SSD 32 Gbps
-3
27 2 12
Memory Usage
500
2r
80
3-
2
400
17
22
60
300
2-
2r 22
200
22
&t
-GGB/s
40
100
-200MB/s
1 4
0.0
10
20
30
40
50
60
HDD
A100
H100
NVMe SSD
Network
Chunk Index (512 tokens per chunk)
Figure Chunk prefill time per step using different methods chunk index.
Figure 3. Comparison of equivalent KV cache loading bandwidth (hytes/sccond) across different storage mediums and GPU com- putation. (Bandwidth GPU computation calculated by viding the total KV cache size by processing time.)
To validate this hypothesis, we conducted long-context pre- fill experiments using chunked prefill, where each chunk contains fixed 512 tokens. As shown in Figure 4, the sults exhibit clear pattern: latency per chunk increases linearly with its index, with earlier chunks requiring less computation, while KV cache memory usage remains con- stant across all chunks.
an H100 GPU achieves similar throughput to loading from SSD and is much faster than network bandwidth used in Google Cloud Egress (" IGB/s) (Google Cloud, 2023). This observation validates our design principle leverag- ing both compute and I/O resources in parallel to minimize latency.
Leveraging this insight, Cake schedules earlier chunks for computation and later chunks for I/O loading. As illus- trated in Figure 2, upon receiving request, Cake splits the sequence into chunks and initiates two parallel processes: (1) The local GPU computes KV cache from the beginning chunk, progressing forward. (2) The data loading process fetches KV cache from the last chunk, moving in reverse direction. This bidirectional strategy continues until the two processes meet in the middle, completing KV cache generation for the entire prompt.
Part 2: Scheduling Based on the Distinet Characteris- ties of Compute and I/O. We design the Cake core sched- uler as a bidirectional KV cache loader based on the fol- lowing key insight:
Insight: Compute cost increases for later tokens, while I/O cost remains constant regardless of token position.
This is because, in the attention operation, later tokens must attend to all previous tokens, resulting in higher compu- tational cost as the sequence progresses. However, the size of kay-v vectors remain the same regardless of posi- tion, meaning I/O access cost is uniform across all tokens.
Part 3: Practical Considerations for Real-World De- ployment. In practical deployment, Cake is designed to address two key challenges: (1) Efficiently sharing com- pute resources with non-prefix caching requests. (2) Han- dling fluctuations in compute and I/O availability.
4

---

Compute Or Load KV Cache? Why not Both?
For the first challenge, real-world inference scenarios in- clude requests without saved prefix cache that require com-
Model
#Layers
KV-size/Token #KV Hd #Atta Hd
Long-Alpaca778
32
512 kB
32
32
pute resources for prefill and decoding. To ensure sys- tem throughput, Cake can prioritize other users' compute needs, allocating only available resources to Cake com- putation. Since Cake can still rely on I/O loading even when compute resources are constrained, we introduce an adaptive scheduling mode that extends vLLM's schedul- ing logic discussed in Section 3. This mode follows prioritized allocation order: 1. Decoding requests. 2. Non- prefix-cache chunked prefill. 3. Prefix-cache chunked pre- fill.
LongAlpacaa13B
40
800 kB
32
32
LLaMA 3.1-8B
32
128 kB
8
32
I.L.M 3.1-70B
80
320 kB
64
Table Comparison experimental model configs.
a user query and the generation of the first token. In other words, it reflects either the time of loading stored KV cache or computing new KV cache.
Datasets. We evaluate Cake across various context lengths using three datasets with different task types: LongChat (Li et al., 2023) for multi-turn conversations, and TriviaQA and NarrativeQA (Bai et al., 2023) for long-document question- answering tasks. According to statistical analysis from CacheGen (Liu et al., 2023), most dataset queries range be- tween 4k and 16k tokens in length. Since specific token val- ues do not impact Cake's performance evaluation (only to- ken length matters), we generate synthetic prompts by uni- formly sampling token lengths every 2k tokens within this range. Additionally, as discussed, Cake is designed to opti- mize the latency of loading cached KV cache from high- capacity, low-bandwidth storage layers. Thus, through- out our evaluation, we precompute and store all requests' KV cache in advance.
For the second challenge, since compute resources are shared across multiple users, and I/O bandwidth could fluc- tuate over time, Cake must dynamically adapt to these variations while maintaining optimal latency. However, this challenge is naturally mitigated by Cake 's bidirec- tional parallel besign-wheneeee I/O or compute availabil- ity changes, the merging point of the two processes shifts accordingly, always ensuring minimal latency.
5. Evaluation
In this section, we evaluate Cake across a variety of real- world setups and discuss its benefits and limitations in de- ployment. These insights can guide future implementations and optimizations. We begin by outlining the experiment setup and then conduct comprehensive experiments to as- sess the impact of the following factors on Cake:
Baselines. We compare Cake to two types of KV cache prefil/loading mechanisms:
1. Varying I/O bandwidth and compute configurations (Section 5.2). 2. Sequence length (Section 5.4). 3. Model architecture (Section 5.5). 4. KV cache compression tech- niques (Section 5.6). 5. Resource fluctuation scenarios (Section 5.7). 6. Adaptive Scheduling (Section 5.8). 7. System overhead (Section 5.9).
Compute-only methods, which employ chunk prefill to compute all KV cache. We use vLLM (v0.6.2) in chunk prefill mode with token budget sizes of 512 by default. In Section 5.3, we evaluate how different token budget sizes affect Cake's 's performance.
I/O Load-only, which loads saved KV cache from lo- cal/remote disks through disk/nerwork. We use LMCache (v0.1.4).
5.1. Experiment Setup
Models. We evaluate Cake on various long-context models with different architectures and sizes, including LongAlpaca-7BB and LongAlpaca- 13B (Chen et al., 2023), which are based on LLaMA 2. as well as LLaMA 3.1- 8B and LLaMA 3. 3.0.7OB. Due hardware constraints, we use the FP8-weight version for LLaMA 3.1-70B, while all other models use FP16 weights. The first two are multi- head attention (MHA) models, while the last two apply group query attention (GQA), which will introduce differ- ent computation vs. memory. We use BF16 as the default kv cache data type. The details of the model are listed in Table 1.
Hardware setting. We run our evaluation on two server configurations: 1) A server equipped with two NVIDIA A100 80GB GPUs connected via NVLink, a 64-core AMD EPYC 7763 CPU, and 2.0TB of memory. 2) A server with single NVIDIA H100 GPU, 26-core vCPU, and 200GB of memory.
1/O Bandwidth Control. To accurately manage I/O band- width and ensure reproducibility, We simulate the chunk I/O loading process by calculating the appropriate delay time based on the chunk size and network bandwidth. The simulated storage backend is then set to pause data trans- fer to GPU memory until the specified delay has elapsed. We choose a variety of bandwidth configurations as demon- strated in Table 2. We choose the I/O loading part chunk size as 128 tokens as it empirically strikes an optimal bal-
Evaluation Metrics. We use time-to- first-t-kktt (TTFT) as our primary evaluation metric. TTFT is widely used in LLM inference, measuring the time between the arrival of
5

---

Compute Or Load KV Cache? Why not Both?
Physical Config
Bandwidth
Hardware
Gbys
Ghys
Ghys
Gbys
Ghys
Google Cloud standard egress limit
7 Gbps
2xA100
12.5%
1.87
1,25 (5.06
1.8
1.999955
Google Cloud tier-1 egress limit
25 Gbps
2xA100
50%
1.27
2,07
1,002224
L465,3006
(.30
1xA100
87.5%
7.22
2.75
2.36
18332.11
1,50
Lambda Lab SSD read
32 Gbps
2xA100
100%
8.10
3,03
2.69
1.97,2.00
1,54),6.1
Samsung 980 pro SSD read Infiniband (RoCF)
56 Gbps
A100
12.5%
1.57
1.14,7.22
1,108895
1.03)14.06
022
100 Gbps
A100
501
3.35).1.46
1.85
1.55),001
1.301,4.26
L.7
A100
1.25
2.18,1888
1,00,2.10
1.5622911
1.301,4007
A100
100%
5.62 1.21
2411,881
2.00
1.99/2.71
1.30),5.52
Table 2. I/O bandwidth with corresponding physical configura- tion
H100
12.5%
1,34
1,20
1,33,690
108811008
1855,1824
H100
50%
4.13,1.38
1.94
1.72
1,45,3.62
1.27
H100
87.5%
6.2
1.5
1.26
L.74/244
3,503354
11100
100%
7.13
2.74
2,4051.75
L8412225
(409
ance between bandwidth utilization and processing granu- larity.
Table 3. Speedup over 1/O-only \ compute-only methods across different compute and I/O configurations. Chunk size: 512, Model: LongAlpaca- 13B, Sequence length: 16K.
GPU Resource Utilization. In online serving scenarios, single machine often handles multiple user requests con- currently. As result, user's prefill operation may not always have access to the full GPU resources. To evaluate different GPU resource availability conditions, we adopt wLLM's token budget scheduling policy, as discussed in $3, to represent resource utilization. For instance, if the total token budget is 512 tokens and Cake request con- sumes 256 tokens while the remaining tokens are allocated to other users, we define this scenario as 50% GPU utiliza- tion. We use this definition to simulate varying levels of GPU resource utilization.
trend. Additionally, examining Table 3 row by row, we observe that as I/O bandwidth increases, the speedup over compute-only also improves. This highlights that Cake ef- fectively balances both compute and I/O resources to en- hance overall performance.
From a broader perspective, the most favorable deploy- ment scenario for Cake occurs when its speedup is approx- imately 2x compared to both compute-only and I/O- approaches. This scenario arises when computation and I/O capabilities are well-balanced, enabling Cake to effi- ciently utilize both in parallel and achieve the best aver- age speedup over either baseline. We observe that in most cases, Cake provides a significant improvement, demon- strating its strong performance.
KV cache Compression methods. KV cache Compres- sion methods, which are orthogonal to our work. They can compress the KV cache size to make them more efficiently transferable through 1/O. In our evaluation, we combine the most common 8-bit quantization and 3-bit quantization proposed by KVQuant (Hooper et al., 2024) with Cake to further evaluate its performance.
To aid interpretation, highlight scenarios with at least 1.5x improvement over both baselines in teal, indicat- ing cases where Cake effectively leverages both resources. Conversely, we use text in red to mark cases where Cake achieves more than 10x improvement over one baseline, which suggests a severe bottleneck in the other resource. In such extreme cases, comparing with the more capable resources, Cake's ' speedup is limited.
5.2. Evaluation Across Compute and I/O Configurations
In this section, we evaluate the performance of Cake un- der varying computational resources and I/O settings. Ta- ble 3 presents the speedup achieved by Cake compared to an I/O-only approach and a compute-only approach across different GPU hardware, GPU utilization levels, and I/O configurations. We conduct tensor parallel inference using 2xA100 setup.
To ensure a fair evaluation of Cake 's average performance improvement over the two baseline methods, we exclude the data points highlighted in red, as comparing Cake speedup relative to an extremely weak baseline would be misleading. After filtering these outliers, we compute the average speedup to provide more balanced assessment of Cake 's overall efficiency. On average, Cake achieves 2.23x and 3.76x speedup over 1/O-only and compute-only meth- ods respectively, under different bandwidth and computa- tion utilization levels.
We observe that, for a fixed GPU utilization level, the speedup over I/O-only increases progressively from A100 to H100 and further to 2xA 100. For example, under 100% GPU utilization with an I/O load bandwidth of 32 Gbps (representing a common server SSD read bandwidth), the speedup over I/O-only improves from 2x on A100 to 2.23x on H100, and 2.63x on 2xA100. This aligns with our de- sign expectation that as compute resources increase, the performance advantage over an I/O-only approach contin- ues to grow. Similarly, analyzing different GPU utiliza- tion levels within the same hardware setup yields the same
5.3. Evaluation Across Varying Chunk Sizes
In this section, we evaluate the performance of Cake un- der varying chunk sizes configurations (i.c., the number of batched tokens) in vL.LM's 's chunked prefill mode.
6

---

Compute Or Load KV Cache? Why not Both?
ChunkSize
BW
LomgAlpuca-78
Long/lpaca-138
Hardware
4k Tokens
Tokens
Tokens
16k Tokeus
32G2pp
1.47
1.57
2xA100
7G0ps
12.5%
2,02
1997
1.94,209
1887
128
32Chps
18332240
L,91,2.4
2xA100
7Gpps
50%
4.75%
1.97
4.711,1244
4.27
256
32Cieps
2.06
1.76
2xA100
KGpps
87.5%
7.31
7.49
261
1.12
512
32Ghps
2,20 1.94
2.03%,191
2sA100
7Gbps
100%
9,0610.99
8.65
8,43.1088
L,10
2048
1034
320itps
32GRpp
2,31
2.75
2.04),1.70
2.18(1991
2xA100
520
12.5%
1.134441
1.195.17
1,995.68
1.186.112
2xA100
52Gabps
1.881664
1.99
1,002009
2.04
2xA100
520itps
87.5%
2.47
2.45
1.69
2.96
Table 4. Speedup over 1/O-only| \ compute-only methods across different chunk sizes of tokens and different models. Hardware IxA100, Utilization 100%, Sequence length: 16K.
2xA100
32Gspp
100%
2.67) 1.25
2.75
2.69)
2.391.59
1xA100
7Gbps
12.5%
1.62,2300
1,59
1,5826677
2.85
[xA100
7Gips
50%
3,58% 1.27
3.48
345(141
3.35),146
IxA100
7Ghps
87.5%
5.36
5.18
5.17(1211
5.06),.26
DXA100
7iipps
1005
6.31
5.93
5.6611155
162,1121
IXA100
520ibps
12.5%
1,44,6337
L.607220
1.95
IxA100
32Gbps
50%
1,38/2.10
1.4912.48
1.50
1.55
As shown in Table 4, Cake achieves an average Time-to- First-Token (TTFT) speedup of 1.96x over 1/O-only meth- ods and 2.25x over compute-only methods across different chunk sizes. Smaller chunk sizes lead underutilization of computational resources, while larger chunk sizes enable more efficient utilization. In general, the speedup over I/O- only methods increases with chunk size, reflecting Cake's growing reliance on computation. These results demon- strate Cake's adaptability to varying compute efficiencies induced by different chunk configurations, allowing to automatically optimize TTFT.
IxA100
32Gbps
87.5%
1.83
1.97
1.91
1.10
IxA100
32Gbps
1.06:
2.15
2.00Ç1.79
2.00.191
Table S. Speedup over 1/O-only \ compute-only methods across different sequence length settings, Hardware: 2xA100, Chunk size 512, Model: Long-AApaca-1.33
5.4. Evaluation Across Sequence Lengths
prefilling speed 3.34x faster than the 1/O-only method and 2.24x faster than the compute-only method.
In this section, we evaluate the performance of Cake under varying sequence length settings. Table 5 presents the speedup achieved by Cake compared to an 1/O-only approach and compute-only approach across different compute-I/O configurations and sequence length settings.
5.5. Evaluation Across Model Architectures
Table presents the speedup achieved by Cake compared to an 1/O-only approach and a compute-only approach across different compute-1/O configurations and model architec- tures. Both LongAlpaca models employ the Multi-Head Attention (MHA) mechanism, while the LLaMA 3 series utilizes Grouped Query Attention (GQA) to compress the KV cache, thereby reducing KV cache memory overhead.
We observe that, for a fixed computation and I/O configura- tion, increasing the sequence length consistently improves the speedup over compute-only methods. As discussed in Section 4, later tokens in a sequence require more com- putation. Therefore, as sequence length grows, leveraging 1/O-only methods to load tokens from later positions be- comes increasingly beneficial compared to compute-only approaches, leading to a continuous increase in speedup.
By comparing the LongAlpaca-778 and LLaMA 3.1-8B models under the same compute and 1/O settings, we ob- serve that, despite the I/O bandwidth remaining unchanged, the speedup relative to the compute-only method increases. This is due to the benefits of GQA, which reduces the KV cache size, effectively enhancing I/O efficiency.
Following the interpretation aid strategy in Section 5.2, We highlight less effective scenarios in red and highly benefi- cial scenarios in teal, observing that Cake is highly benefi- cial in many cases. Notably, there is single scenario where Cake underperforms compared to compute-only methods. This occurs when the sequence is short, as using 2xA100 for computation is significantly faster than loading data via a 7 Gbps bandwidth. However, due to Cake 's scheduling algorithm, it still assigns a portion of the workload to the /O-only method, introducing overhead. For future work, optimize this scheduling strategy by incorporating an estimation mechanism. If one resource significantly out- performs the other, Cake could adaptively fall back to a single-resource mode, utilizing only the more efficient re- source to minimize overhead.
Furthermore, comparing different model sizes within the same series reveals that the speedup for 1/O-only methods decreases as the model size grows. This indicates that com- putation requirements scale faster than I/O demands, lead- ing to diminishing relative gains from compute optimiza- tions.
We also identified four data points where Cake did not outperform the baseline, similar to the issue discussed in Section 5.4. This can be mitigated by incorporating a fall- back mechanism that dynamically adjusts resource alloca- tion when one resource significantly outperforms the other. Despite the overheads extreme scenarios, Cake on aver- age achieves 2.68x speedup over the 1/O-only method and 3.01 speedup over the compute-on) method.
However, on average, we can still observe Cake improves
7

---

Compute Or Load KV Cache? Why not Both?
LongAppac-77
LongAlpwa-113
Llmm331-81
Lamm3311770B
15000
7Gbps
1.92
1887/2.18
1.56
0,991
12.92
Token Index
12.5%
32Gbps
1,2015.98
1.18
1988119447
0,80.44.20
7Cibp=
4.79,1.50
1.25
10000
30%
52Gbps
1.25
1.80),2.4
L.3
0.92
7Cibp=
2.81
1.66,2.64
100%
52Ghps
2,791120
1.59
1.37)
1.02
Compute thread
1/0 load thread
Table 6. Speedup over 1/O-only \ compute-only across different model settings, Hardware: 2xA100, Chunk size 512, Seq-len:
0.0
0.2
0.4
0.6
1.0
1.2
Time (s)
16k.
16-bit
X-hit
3-bit
Figure 5. Cake trace under fluctuate network and available com- putation power. Hardware: A100, Model: Long-Alpaca-715 I/O
7CTpp
10001110
4.633,1.22 1.22
1.39%,163
25Cbbps
303
2.77.187
1..0)3.09
Bandwidth: 0-25Gbps, Compute Utilization: 0-512 budget, Seq- Ien: 16k.
33CThps
2.59.1.9
18552.12
1.37),3.3
3003bps
1.97),200
1,49
1.26,5.38
Table 7. Speedup over 1/O-only| compute-only under different
Token Batch Size
low-precision compression, Hardware: 2xA100 100% Utiliza- tion, Chunk size 512, Seq-len: 16k, Model: Long-Alpaca-1l
400
M 1.0
200
5.6. Incorporating KV cache Compression with Cake
Default Scheduling
Adaptive Scheduling
Table 7 presents the speedup achieved by Cake com- pared to the 1/O-only approach and compute-only approach across different I/O configurations and different compres- sion ratio. We can observe similar to the effect of GQA, now-precision compression ratio reduces the KV cache size, thus enhancing the I/O load performance, thus Cake speedup over compute-only methods continues to increase as the precision decreases.
0
0.2
0.6
1.2 1.4
Time (s)
Figure Token batch size over time under dynamic workload with default and adaptive scheduling. Adaptive scheduling of Cake flattens the spikes dynamic workloads maximizes the system throughput.
5.7. Handling Fluctuations in Resources
Using these request traces, compare the default vLLM scheduling algorithm with Cake's adaptive scheduling al- gorithm for inference execution. The result is demon- strated in Figure 6. In vLLM's default scheduling mode, our grefi--cacinn request arrives first and is processed to completion before handling subsequent tet---rfff-ccchnng requests. This approach results in suboptimal GPU utiliza- tion, as the token batch size is not fully utilized at all times.
To evaluate Cake's performance under fluctuating compute and I/O conditions, we randomly sample a compute budget trace uniformly between 0-512 tokens to represent vary- ing computational power. Similarly, we randomly sample an I/O bandwidth trace between 0-25 Gbps assess how Cake adapts its scheduling strategy.
As shown in Figure 5, Cake dynamically leverages both I/O and computational resources, regardless of fluctuations. Its bidirectional prefetching mechanism automatically identi- fies the optimal merging point to minimize TTFT, ensur- ing optimal performance even under varying resource con- straints.
In contrast, Cake 's adaptive scheduling prioritizes decod- ing and prefill operations for incoming non-pres non-prefix- prchix-caching requests while allocating the remaining compute budget for chunk prefill of the prefix-caching request. It successfully keeps the GPU busy and reduces the overall finish time from 1.5s to 1.19s, improving throughput by 26%.
5.8. Evaluation on Adaptive Scheduling
5.9. Overheads of Cake
In this evaluation, we give an example evaluate the per- formance of the adaptive scheduling algorithm Cake. We begin by sending prefix-caching request of length 16K, representing a typical long-context prefix-caching scenario in Cake. To simulate a burst of incoming requests from other users, we then randomly generate 22 additional re- quests with sequence lengths ranging from 32 to 448, fol- lowing a spiked distribution.
To evaluate the overhead introduced by Cake, we com- pare the duration of each engine step between the original vLLM and vLL.M with Cake. As shown in Figure 7, we launch a chunk prefill job on both A100 and H100 servers. The chunk prefill time of vLLM with Cake closely follows the trace of the original vLLM, indicating minimal perfor- mance impact.
8

---

Compute Or Load KV Cache? Why not Both?
design also adapts naturally distributed inference se- tups. Our evaluation with tensor-parallelism using multi- ple GPUs within single node (Table 3 and Table 5) con- firms that Cake effectively utilizes increased computational resources. For multi-node distributed environments, Cake operates independently within each inference engine, al- lowing each node to optimize its inference latency. Con- sequently, Cake remains effective even in extensive, dis- tributed inference deployments.
Time (ms)
60
NVIDIA A100
NVIDIA H100
40
5
10
20
Chunk Index
Figure 7. Per-step inference time in vLLM before and after inte- gration with Cake. The solid line represents the step time without Cake, while the? 'x' markers indicate step times with Cake.
7. Conclusion
These results demonstrate that Cake introduces negligible overhead, as it only performs a lightweight check to deter- mine whether the next chunk has been fetched at runtime.
In this paper, we introduced Cake, the first KV cache load- ing system that optimally balances computation and I/O to minimize Time to First Token (TTFT) in LLM inference. Unlike prior approaches that focus on either compute or I/O in isolation, Cake employs bidirectional scheduling strategy that dynamically adapts to resource availability, achieving an average 2.6x TTFT reduction compared to baselines. Additionally, its adaptive scheduling mechanism enhances system throughput, making it a practical and eas- ily deployable solution for LLM-serving systems. Through extensive evaluations, we provide detailed analysis of the scenarios where Cake is most beneficial, offering valuable insights for real-world deployment.
6. Discussion
Compatibility with Other Acceleration Design. Cake is orthogonal and complementary various widely adopted acceleration methods, such speculative decoding, multi- token prediction, KV cache quantization, eviction, and prefill-decode disaggregation. These methods can be cat- egorized into three main classes:
Decoding Throughput Optimization Methods: Techniques such as speculative decoding (Leviathan et al., 2023; Li al., 2024) and multi-token prediction (Cai et al., 2024) optimize the decoding phase by parallelizing token genera- tion. Cake, targeting the prefill stage, is inherently compati- ble with these methods, allowing concurrent enhancements in both phases without conflicts.
Acknowledgments
We would like to thank our anonymous reviewers for their valuable comments and feedback. This work was par- tially supported by Boeing as well as NSF under grants CMMI-2038215, CNS-2321532, and National AI Institute for Edge Computing Leveraging Next Generation Wireless Networks, 2tant-2112562
KV Cache Size Reduction Methods: Approaches like KV cache quantization (Hooper et al., 2024; Kang et al., 2024; Liu et al., 2024b) and eviction (Jiang et al., 2023) reduce memory usage by compressing or selectively removing to- kens from the KV cache. Cake treats the KV cache as data and is agnostic to its specific representation or compres- sion scheme. Our evaluation with compression methods (Section 5.6) demonstrates that Cake can seamlessly inte- grate with these techniques, further reducing latency and enhancing performance.
Impact Statement
This paper presents Cake, a system that enhances LLM in- ference by efficiently combining computation and I/O re- sources reduce latency. By improving the responsiveness of long-context LLM applications, Cake enables smoother user experiences and broader accessibility across various domains, from conversational Al to document processing. Its efficient design can help maximize the utility of exist- ing infrastructure, making large-scale models more practi- cal for real-world deployment. While increased efficiency may accelerate LLM adoption, we encourage responsible deployment with fair resource allocation and transparent system usage. Overall, Cake contributes to advancing scal- able and effective LLM serving for various applications.
System-level Acceleration Methods: Prefill-decode disag- gregation (Zhong et al., 2024) separates prefill and decode stages onto different hardware, often using separate prefill and decode servers. Cake can be integrated into such de- sign by managing bidirectional KV cache loading across these servers. The prefill server computes the KV cache from the beginning of the sequence and streams it for- ward, while the decode server concurrently manages two streams-the incoming stream from the prefill server and the reverse stream from 1/O operations. The two processes meet in the middle, ensuring efficient cache preparation.
Compatibility in Distributed Environments. Cake's
9

---

Compute Or Load KV Cache? Why not Both?
References
Hooper, C., Kim, S., Mohammadzadeh, H., Mahoney,
M. W., Shao, Y. S., Keutzer, K., and Gholami, A. Kvquant: Towards 10 million context length Ilm in- ference with kv cache quantization. arXiv preprint arXiv:2401. 18079, 2024.
Agrawal, A., Panwar, A., Mohan, J., Kwatra, N., Gulavani, B. S., and Ramjee, R. Sarathi: Efficient Ilm inference by piggybacking decodes with chunked prefills. arXiv preprint ar arv0223008 /6369, 2023.
Jiang. H., Wu, Q. Lin, C.-Y,, Yang. Y., and Qiu, L. Limlingua: Compressing prompts for accelerated in- ference of large language models. arXiv preprint arXiv:2310.05736, 2023.
Agrawal, A., Kedia, N., Panwar, A., Mohan, J., Kwatra, N., Gulavani, B. S., Tumanov, A., and Ramjee, R. Taming throughput-latency tradeoff in Ilm inference with sarathi- serve. arXiv preprint 2ati::2200022100 2024.
Jiang. Z., Ma. X., and Chen, W. Longrag: Enhancing retrieval-augmented generation with long-context 1lms. arXiv preprint arXiv:2406. 15319, 2024.
Ainslie, J., Lee-Theeeeee Lee-Thorp, Thorp, J., de Jong, M., Zemlyanskiy, Y., Lebrón, F., and Sanghai, S. Gqa: Training generalized multi-query transformer models from multi-head check- points. arXiv preprint arXiv:2305.13244 2023.
Jin, C., Zhang, Z., Jiang, X., Liu, F., Liu, X., Liu, X., and Jin, X. Ragcache: Efficient knowledge caching for retrieval-augmenner generation. arXiv preprint arXiv:2400 12457, 2024.
Anthropic. Prompt caching, caching 2023, URL https::pp/// /www. anthropic. nwws//wwww/////eeeee wwww//prompa. prompt caching.
Anthropic. Anthropic model comparison, 2024. URL ht wtt.:docs.aattrpi......coooo com cttp:////////c//////////cc ut-c htt-cllude/modee #mode wwwpariioooaoooooooe. le.
Jin, S., Liu, X., Wu, Y., Zheng, H., Zhang, Q., Prakash, A., Lentz, M., Zhuo, D., Qian, F., and Mao, M. Plato: Plan to efficiently decode for large language model in- ference. arXiv preprint arXiv:2402.. /2280v,, 2025.
Bai, Y. Lv, X., Zhang. J., Lyu, H., Tang, J., Huang, Z.,, Du, Z., Liu, X., Zeng, A., Hou, L., et al. Longbench: A bilingual, multitask benchmark for long context under- standing. arXiv preprint arXiv:2308. 14508, 2023.
Kang, H., Zhang, Q., Kundu, S., Jeong, G., Liu, Z., Kr- ishna, T. and Zhao, T. Gear: An efficient kv cache com- pression recipefor near-lossless generative inference of 1lm. arXiv preprini arXiv:2403.05522 2024.
Cai, T., Li, Y., Geng, Z., Peng. H., Lee, J. D., Chen, D., and Dao, Medusa: Simple Ilm inference acceleration framework with multiple decoding heads. arXiv preprint arXiv:2401. 10774, 2024.
Kwon, W., Li, Z., Zhuang, S., Sheng, Y., Zheng, L., Yu, H., Gonzalez, J., Zhang, H., and Stoica, Efficient memory management for large language model serving with pagedattention. In Proceedings of the 29th Sym- posium on Operating Systems Principles, pp. 611-626, 2023.
Chan, B. J., Chen, C.-T., Cheng, J.-H., and Huang, H.- H. Don't do rag: When cache-augmented generation is all you need for knowledge tasks. arXiv preprint arXiv:2412. 15605, 2024.
Lambda. Lambda lab gpu cloud specifications, 2024. URL https: / aalaalllllllllaa labs loud.
Chen, Y. Qian, S., Tang, H., Lai. X., Liu, Z., Han, S., and Jia, J. Longlora: Efficient fine-tuning of long-context large language models. arXiv preprint 2XXiv23309112007 2023.
Leviathan, Y., Kalman, M., and Matias, Y. Fast inference from transformers via speculative decoding. Inter- national Conference Machine Learning, pp. 19274- 19286. PMLR, 2023.
Deepseek. Prompt caching api, 2024. URL https://pl atform. deepseek. com s0802/.
Li, D., Shao, R., Xie, A., Sheng, Y., Zheng, L.., Gonza- lez, J., Stoica, I., Ma, X., and Zhang, H. How long can context length of open-source llms truly promise? In NeurIPS 2023 Workshop on Instruction Tuning and In- struction Following, 2023.
Gao, B., He, Z., Sharma, P., Kang, Jevdjic, D., Deng, J., Yang, X., Yu, Z., and Zuo, [Cost-Efficient) large lan- guage model serving for multi-turn conversations with (CachedAtttentoon), In 2024 USENIX Annual Technical Conference (USENIX ATC 24), pp. 111-126, 2024.
Li, Y., Wei, F., Zhang, C., and Zhang, H. Eagle: Spec- ulative sampling requires rethinking feature uncertainty. arXiv preprint arXiv:2401.15077 2024.
Google Cloud. Network bandwidth, 2023. URL https: doos / netw
// cloud. google. com/ compute eeaeee//oossso ork barkbandwiin dtth Accessed: 2023-10-30.
Liu, Y., Li, H., Du, K., Yao, J., Cheng, Y., Huang, Y., Lu, S., Maire, M., Hoffmann, H., Holtzman, A., et al.
10

---

Compute Or Load KV Cache? Why not Both?
Cachegen: Fast context loading for language model ap- plications. arXiv preprint arXi::23 2023.
Zhang, Y., Sun, R., Chen, Y., Pfister, T., Zhang, R., and Arik, S. Ö Chain of agents: Large language mod- els collaborating long-context tasks. arXiv preprint arXiv:2406.0283 2024.
Liu, Z.,, Desai, A., Liao, F., Wang, W., Xie, V., Xu, Z., Kyrillidis, A., and Shrivastava, A. Scissorhands: Ex- ploiting the persistence of importance hypothesis for Ilm kv cache compression test time. Advances in Neural Information Processing Systems, 36, 2024a.
Zhang, Z., Sheng, Y., Zhou, T., Chen, T., Zheng, L.,, Cai, R., Song, Z., Tian, Y., Ré, C., Barrett, C., al. H2o: Heavy-hitter oracle for efficient generative inference of large language models. Advances in Neural Information Processing Systems, 1..34661-34711 2023.
Liu, Z., Yuan, J., Jin, H., Zhong, S., Xu, Z., Braverman, V., Chen, B., and Hu, X. Kivi: A tuning-free asym- metric 2bit quantization for kv cache. arXiv preprint arXiv:2402.02750, 2024b.
Zheng, L., Yin, L., Xie, Z., Huang, J., Sun, C., Yu, C. H., Cao, S., Kozyrakis, C., Stoica, I., Gonzalez, J. E., et al. Efficiently programming large language models using sglang, arXiv preprint arXiv:23/2.077000 2023.
LMCache.
Lmcache, 2024. URL httttt://ggthbbb cmmLLMCCche//Iee IMCCchh
Zhong, Y., Liu, S., Chen, J., Hu, Zhu, Y., Liu, X., Jin, X., and Zhang, H. (DistServe): Disaggregating prefill and decoding for goodput-optimized large language model serving. In 18th USENIX Symposium on Operating Sys- tems Design and Implementation (OSDI 24), pp. 193- 210, 2024.
Miao, X., Oliaro, G., Zhang. Z., Cheng. X., Wang. Z., Zhang, 7.,, Wong, R. Y. Y., Zhu, A., Yang, I.., Shi, X., et al. Specinfer: Accelerating large language model serving with tree-based speculative inference and veri- fication. In Proceedings of the 29th ACM International Conference on Architectural Support for Programming Languages and Operating Systems, Volume 3. pp. 932- 949, 2024.
Ning, X., Lin, Z., Zhou, Z., Wang, Z., Yang, H., and Wang. Y. Skeleton-of th-thugghtt Large language models can do parallel decoding. Proceedings ENLSP-III, 2023.
openAI.
2024. URL hRthtttss::pplaataaaee penai. com/ dwcsd docs/nodelssgpt--22
OpenAl. Prompt caching, 2024. URL https: form. owwnaie......................... com wwt.c///occc/gu/e/eeeeeooeooaaee aching.
Vaswani, A., Shazeer, N., Parmar, N., Uszkoreit, J., Jones, L.. Gomez, A. N., Kaiser, L. u., and Polosukhin, I. At- tention is all you need. In Guyon, I,, Luxburg, U. V., Bengio, S., Wallach, H., Fergus, R., Vishwanathan, S., and Garnett, R. (eds.), Advances Neural Information Processing Systems, volume 30. Curran Associates, Inc.. 2017. URL https: http:tts///pcccceednggss neurisgss.....ss........... cc/paper_ ccc/appe_ffheee/ppeee//////////2 245577dee111dd53cccclaaaaaa df.
Wang, M., Chen, L., Fu, C., Liao, S., Zhang, X., Wu, B., Yu, H., Xu, N., Zhang, L., Luo, R., et al. Leave no docu- ment behind: Benchmarking long-context lIms with ex- tended multi-doc qa. arXiv preprint arXiv:2406. /7419, 2024.
Yao. J. Li, H., Liu, Y., Ray, S., Cheng, Y., Zhang, Q. Du, K., Lu, S., and Jiang, J. Cacheblend: Fast large language model serving with cached knowledge fusion. arXiv preprint ar055:2405. 16444, 2024.

---

Compute Or Load KV Cache? Why not Both?
A. Details of Cake Algorithm
The workflow of Cake can be described in detail as follows:
1. Upon receiving request, Cake first splits the input sequence into chunks of a predetermined size.
Compute the prefix hash of all the chunks and find the latest prefix hash that exists in the storage backend, and determine total_ookens.
Two pointers are initialized: compute-ple -pt starting the beginning of the sequence (index 0), and io-ptr the end of the sequence (index total. tokeets - 1).
3. Two parallel processes are initiated: a) A GPU computation thread that starts from compute. and moves forward. b) An I/O streaming thread that starts from io ptr and moves backward.
The GPU computation thread: Computes KV cache for chunks starting from compute-pin After each chunk computation, updates compule.plr by adding the chunk size. Continues until compule per reaches or surpasses io ptr, until the required KV cache is found in CPU memory.
5. The I/O streaming thread: Fetches pre-computed KV cache for chunks ending at io ptr from storage (local remote) to CPU memory. After each chunk fetch, it updates io_p!r by subtracting the chunk size. Continues until io_pír reaches or goes below compute ptr.
The process concludes when the two pointers meet or cross each other, indicating that KV cache for the entire sequence has been either computed or loaded.
Finally, Cake returns the complete KV cache for the entire sequence, ready for use in the subsequent inference steps.
This bidirectional approach allows Cake to efficiently utilize both computational and I/O resources simultaneously, mini- mizing idle time and optimizing the overall latency of KV cache preparation for long-context LLM inference.
Algorithm Cake Bidirectional KV cache Loading Algorithm
1:
procedure COMPUTEKV
2:
while compute ptr io ior do
3:
ISSICCPUMEARRYCCoppeeee SSANPPAAAAABrrrooouurrr....CCOOE COM..CCUUK...IIIII.CCCCCC then Signal IOO worker to stop break
4:
5:
6:
Compute KV cache for chunk starting compute.ptt using GPU
7:
compute ptr + compute ptr + COMP CHUNK SIZF.
8: procedure FETCHKV
io_pfr do Fetch KV cache chunk ending at io pfr from storage to CPU Memory 11: io ptr 4- io ptr FETCH CHUNK SIZF
9: while compute_pfss
Initialize CPU Memory, compute.pp
0, io_ptr = total
13: Start COMPUTEKV in new thread
14: Start FETCHKY in new thread
15: Wait for both threads complete
return KV cache entire sequence
B. Implementation
We implement Cake by extending LMCache (LMCache, 2024) and integrating it with vLLM (Kwon et al., 2023), adding approximately 1,000 lines of code.
B.1. Enhancements to LMCache
LMCache, originally developed as the KV cache management backend for CacheGen (Liu et al., 2023), hashes token chunks into keys for efficient KV cache retrieval. To enable Cake to continuously receive KV cache in the background, we introduce the following enhancements:
12

---

Compute Or Load KV Cache? Why not Both?
Asynchronous Retrieval We implement asynchronous get operation complement LMCache's existing asyn- chronous put functionality. This involves creating dedicated worker thread that continuously reads chunk keys from a task queue and retrieves the corresponding KV cache memory. Upon successful retrieval, the chunk's key is added to a resident dictionary for quick access.
Buffer Preallocation We modify LMCache to preallocate chunk buffers as soon as a chunk key is pushed to the queue. This optimization allows the worker to immediately write received KV cache into memory and proceed to the next chunk without delay.
B.2. Integration with LLM Serving Systems
Cake operates concurrently with LLM serving systems like vLLM. The integration process works as follows:
Upon receiving request, Cake divides it into chunks based on the scheduled token budget.
Hashed keys for these chunks are pushed to the task queue in reverse order and call batch_retrieve API.
3. While the asynchronous get worker fetches KV cache from the end of the sequence, vLLM begins chunk prefill from the start.
4. After each vLLM engine step, Cake checks if the next chunk of tokens already in the resident dictionary using the resident API.
If the chunk is resident, Cake interrupts the chunk prefill process and directs vLLM to begin token generation.
6. If the chunk is not resident, chunk prefill continues until it encounters chunk present in the dictionary.
This bidirectional approach allows Cake to efficiently utilize both I/O and computational resources, potentially reducing the Time To First Token (TTFT) for long-context LLM inference tasks.
13


Reviews:
[{"review_id": "LiVoQHmYDa", "reviewer": "Reviewer_bg9y", "summary": "This paper introduces Cake, a novel KV cache loading system that optimally utilizes both computational and I/O resources in parallel, with a bidirectional scheduling strategy, and an adaptive scheduling mechanism. The proposed method can be seamlessly integrated in existing methods, with better TTFT.", "questions": "", "limitations": "", "rating": 3, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for your insightful comments and feedback on our paper. We are pleased to address your concerns regarding the compatibility and performance of our proposed method, Cake, with other acceleration techniques and system-level optimizations. We will add the following discussion to the revised version.\n\n* **Compatibility with other acceleration methods.** Thanks to the simple and elegant design, Cake is orthogonal and complementary to the broad acceleration methods you mentioned, including speculative decoding, multi-token prediction, KV cache quantization, eviction, and Prefill-Decode disaggregation, making Cake very flexible to deploy. These methods can be categorized into three main classes:\n* + **Techniques to Accelerate LLM Decoding Throughput**:\n* + + Methods like speculative decoding and multi-token prediction are focused on optimizing the decoding stage, e.g., by leveraging parallel generation opportunities to address the inefficiencies of autoregressive decoding.\n* + + Cake, however, targets the prefill stage, i.e., generating the KV cache for the decode stage. Since the prefill and decoding stages are distinct, there is no conflict between Cake and these decoding-focused methods. They can be used together to enhance overall inference efficiency.\n* + **KV Cache Size Reduction Methods**:\n* + + Techniques such as quantization and eviction aim to reduce the memory footprint of the KV cache. Cake regards the KV cache as simple data, and it does not restrict the size of KV  cache or how the KV cache is computed. Therefore, these methods can be seamlessly integrated with Cake.\n* + + Actually, we discussed the integration of KV cache compression methods in Section 5.5 of our paper, where we evaluated Cake’s performance with compression techniques, showing that they are orthogonal and can complement Cake’s bidirectional scheduling strategy. This synergy could lead to additional reductions in TTFT, though the extent of improvement may vary depending on specific implementations and resource constraints.\n* + **System-level Acceleration Methods**:\n\n* + + **Prefill-Decode Disaggregation.** Cake is compatible with Prefill-Decode Disaggregation with a minor modification. The typical workflow of Prefill-Decode Disaggregation involves routing a request to a prefill server to generate the KV cache, which is then streamed to a decode server. Cake can fit into this scenario by splitting its bidirectional process across these servers: the prefill server handles computation, generating the KV cache from the sequence’s start and streaming it to the decode server; On the decode server, Cake simultaneously manages two streams—one is the I/O loading stream, fetching the existing KV cache from disks starting from the last tokens, and the other is the streaming process from the prefill server, starting from the first tokens—until the two processes meet in the middle on the decode server, meaning all KV cache is ready for decoding.\n\n* + + **Distributed inference.** We have evaluated Cake’s performance with tensor-parallelism using two GPUs, as detailed in Table 3 and Table 4 of our paper. The results demonstrate that Cake can effectively utilize the increased computational capabilities provided by multiple GPUs within a node. By distributing the model across GPUs, the computational power available for Cake’s bidirectional scheduling improves, allowing it to optimize TTFT more efficiently. For distributed inference settings, where multiple inference engines are deployed across various nodes in the data center and requests are routed to different engines for processing, Cake can still be applied within each inference engine. Each engine can independently optimize its own inference latency using Cake, ensuring that the method remains effective even in large-scale, distributed environments.\n\n* **About mobile devices**: Cake is designated for long-context prefix-caching enabled LLM inference systems and is beneficial as long as the simultaneous KV computation and I/O KV cache loading is possible. The enterprise-level data center is the best use case because of its sufficient GPU computational power, GPU memory capacity, and I/O bandwidth. The idea of Cake could also be applied to mobile devices, but the limited computation power and memory space limit the model size or context length, making mobile computing not the ideal use case. Cake’s deployment on mobile devices is promising but needs future work to address engineering challenges.\n\n* Thank you for your careful reading and pointing out the typos. We will correct them in the final version to ensure clarity.", "claims_and_evidence": "Most of the claims and evidence in the paper are convincing.", "methods_and_evaluation_criteria": "Most of the evaluation criterions are good.", "theoretical_claims": "There is no theoretical claims in the paper.", "experimental_designs_or_analyses": "1. The experiment designs of this paper are good. \n2. It would be better if the authors conduct experiments on more mobile devices, such as GPU for laptops (4090, 3090) and GPU for mobile phones.", "relation_to_broader_scientific_literature": "I did not find any specific relation between this paper and broader scientific literature.", "essential_references_not_discussed": "N/A", "other_strengths_and_weaknesses": "1. The most significant strength of this paper is that the proposed method does not bring any harm to model accuracy, making it useful in most applications.\n2. Please discuss whether the proposed method conflicts with other acceleration methods for KV cache and decoding, such as speculative decoding, multi-token prediction, KV cache quantization, and eviction. Intuitively, if these methods are applied, the improvements of Cake may be limited.\n3. Please discuss whether the proposed method is compatible with other system-level acceleration methods, especially for the methods that separate the prefilling and decoding stages in different hardware.\n4. Please discuss the performance of Cake when multiple GPUs and even multiple GPU nodes are employed. Does distributed inference impacts Cake?\n4. Typos: There is no ``.'' in table 4 and table 2.", "other_comments_or_suggestions": "N/A", "questions_for_authors": "N/A", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "HFLZTzNUSa", "reviewer": "Reviewer_opvL", "summary": "This paper introduces Cake, a hybrid KV cache loading system that leverages both I/O resources (for loading) and computational resources (for re-computation). The authors observe that both I/O-only approaches (e.g. LMCache) and compute-only approaches (e.g. vLLM) fall short in practice in terms of minimizing TTFT when loading from local or remote disks. Inspired by this, Cake uses both I/O and computation in parallel when loading from disks (which are high-capacity but low-bandwidth). Evaluations show that Cake can reduce TTFT by 2.6x on average for long-context prefix caching scenarios.", "questions": "", "limitations": "", "rating": 4, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thanks for your insightful feedback, and we are glad to address your concerns.\n1. **Dynamic I/O, workloads, etc.** Cake’s parameter-free design allows it to automatically adapt to dynamic conditions, such as fluctuating network bandwidth or GPU performance, as discussed in Part 3 of Section 4. This is a key strength, as shown in Figure 5 of Section 5.6, we conducted an experiment on dynamic computational power and network bandwidth. Under severe fluctuations, Cake is still able to find the optimal merging point automatically and maximize the utilization of dynamic resources. The reason behind it is that Cake does not rely on any offline profiling or any static parameters. The optimal utilization of compute and I/O is achieved by our bidirectional KV cache generation mechanism, demonstrated in Figure 2. Cake starts KV cache computing and loading from opposite directions, chunk by chunk; the two threads run in parallel asynchronously and merge at a certain point in the middle of the context. A sudden network traffic burst may slow down the I/O transfer, then the two threads will merge at a position closer to the end of the context. Similarly, if a temperature spike reduces GPU frequency and slows down the computing progress, the threads will meet at a position closer to the head of the context. The scheduler doesn’t need to change anything caused by a certain period of exceptional condition, it only needs to interrupt the two threads if the last computed chunk includes a loaded token‘s cache and switch request status to the decoding stage. This design ensures that Cake always achieves the optimal Time-To-First-Token as it fully leverages the available I/O and computing capability, even though they are dynamic.   \n2. **Potential application of CPU-GPU loading.** Cake can be applied between CPU and GPU with minor adjustments. Yet, there are two reasons why we don’t suggest applying Cake between host memory and GPU. First, as we show in Figure 3, for a context length of 32k, the KV cache computing speed of an H100 is approximately equivalent to 4GB/s I/O transfer. However, H100 supports PCIe 5.0x16, which provides up to 64GB/s host-to-device transfer bandwidth per GPU. This huge gap makes re-computing less efficient, and loading all the cache is a better option. Second, compared with SSD and remote disks, host memory is more expensive, and its storage is limited. As is profiled by AttentionScore, 80% of cache hits happen on the disk level, and we believe optimizing it will bring broader benefits to the modern LLM serving system.    \n\nWe appreciate your interest in Cake and your insightful reviews. We are happy to address any further questions about Cake.", "claims_and_evidence": "Most of the claims made by the authors are supported by citations or experiment results. Please see \"Questions For Authors\" for details.", "methods_and_evaluation_criteria": "The paper features solid and comprehensive evaluation of the proposed method, Cake.", "theoretical_claims": "N/A", "experimental_designs_or_analyses": "The paper features solid and comprehensive evaluation of the proposed method, Cake.", "relation_to_broader_scientific_literature": "This paper lies within the broad area of KV cache management systems for language model inference. The authors focus primarily on acceleration of KV cache loading from disks, which can be helpful to cloud providers in the case of prefix-cache loading or prompt-cache loading.", "essential_references_not_discussed": "As far as I know, there is no missing related work (in the area of KV cache loading / prefix caching systems) that remain undiscussed in this paper.", "other_strengths_and_weaknesses": "N/A", "other_comments_or_suggestions": "N/A", "questions_for_authors": "Thank you for submitting this paper to NeurIPS. The core idea of this paper, leveraging I/O and compute in parallel when loading KV cache from disks, is well presented and makes sense. Experiments are also solid and comprehensive. \n\nI have two major questions/concerns. \n1. In reality, I/O (network) and compute capability can vary dramatically from time to time, even with the same set of local/remote hardware, depending on if you have competing workloads, network variance, temperature of hardware, etc. Is your system assuming a fixed config of I/O and compute capability, pre-measured given the hardware stack? If this is the case, do you have real-time profiling that adjusts the capabilities so that your scheduling works well for the real-world scenario? Otherwise it seems very likely that sub-optimal scheduling decisions will occur. \n2. Do you think your system would work for CPU->GPU loading with minor adaptation? If not, what do you think is the major challenge (or maybe loading is all you need in that case)? Is it because disk->CPU/GPU might be the only case where re-computation can be sometimes more efficient (in terms of TTFT) than direct loading?", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "fiIndd99Db", "reviewer": "Reviewer_wYCE", "summary": "The paper introduces a KV cache loading system called Cake, that optimizes computation and I/O in parallel to speed up LLM inference. It uses bidirectional scheduling for efficient resource use and adaptive scheduling to handle varying workloads. Evaluations show Cake cuts TTFT by 2.6×, making it a practical solution for long-context LLMs.", "questions": "", "limitations": "", "rating": 4, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for your insightful feedback and kind words about our paper. Below, we address your suggestions regarding the interaction of Cake with Prefill-Decode (PD) disaggregation. We will add the following discussion to the revised version.\n* Cake is compatible with Prefill-Decode Disaggregation with a minor modification. The typical workflow of Prefill-Decode Disaggregation involves routing a request to a prefill server to generate the KV cache, which is then streamed to a decode server. Cake can fit into this scenario by splitting its bidirectional process across these servers: the prefill server handles computation, generating the KV cache from the sequence’s start and streaming it to the decode server; On the decode server, Cake simultaneously manages two streams—one is the I/O loading stream, fetching the existing KV cache from disks starting from the last tokens, and the other is the streaming process from the prefill server, starting from the first tokens—until the two processes meet in the middle on the decode server, meaning all KV cache is ready for decoding.\n* Thank you for your suggestions to provide more details on chunked prefill. We will add more details in the background Section in our revised version.", "claims_and_evidence": "The main claim made by this paper is \"Cake is the first system to demonstrate that efficiently utilizing both computational and I/O resources can optimally reduce TTFT in longcontext prefix caching scenarios.\"\n\nIn my opinion, this statement is clear and supported by experimmental results", "methods_and_evaluation_criteria": "The proposed method and evaluation criteria make sense for the problem considered.\n\nI also really enjoy reading the analysis of the design section and the insights of Cake", "theoretical_claims": "No theoretical claims in this paper", "experimental_designs_or_analyses": "Yes, I checked the soundness of the experimental designs.", "relation_to_broader_scientific_literature": "TTFT is a crusial SLO in LLM serving area and scheduling long context request is a very important problem.", "essential_references_not_discussed": "No", "other_strengths_and_weaknesses": "In general, I like this paper. It is simple and intuitive\n\nThe weakness in my mind is there is no discussion on how this approach interacts with PD disaggregation, which is wildly used in practice. In my understanding, even though the KV cache is computed, they still need to be transferred to decode server through RDMA.", "other_comments_or_suggestions": "I suggest introduce more details regarding chunked prefill, since Cake is built upon it. And many audience of ICML may not familar with it.", "questions_for_authors": "See the weakness part.", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "KuWiKQyEVg", "reviewer": "Reviewer_UkhW", "summary": "The authors proposed a simple but effective method dealing with the kv cache prefilling: utilizing both the GPU computation and IO loading  to get the free lunch of both, maximizing the loading speed with zero cost.", "questions": "", "limitations": "", "rating": 4, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for your interest in Cake and for providing insightful feedback. We value your constructive comments and are pleased to address your concerns.\n\nIn our original experiments, we adopted the inter-token-latency-optimized configuration in vLLM v0.6.2, where the max number of batched tokens is 512. We agree that exploring different batch sizes offers valuable insight into Cake’s adaptability under memory-bound and compute-bound conditions. Below are our newly added experiment results of Cake’s performance gain over the compute-only and I/O-only prefill on different numbers of batched tokens and different model sizes. We will include such results in our revised version. \n\n-----------------------------------------------------------------\n| BatchSize | BW     | LongAlpaca-7B | LongAlpaca-13B |\n|-----------|--------|---------------|----------------|\n| 64        | 32Gbps | 1.47\\3.47     | 1.47\\3.55      |\n| 128       | 32Gbps | 1.83\\2.40     | 1.79\\2.45      |\n| 256       | 32Gbps | 2.06\\2.09     | 1.96\\2.20      |\n| 512       | 32Gbps | 2.20\\1.94     | 2.03\\1.91      |\n| 1024      | 32Gbps | 2.15\\1.70     | 2.18\\1.91      |\n| 2048      | 32Gbps | 2.31\\1.71     | 2.04\\1.70      |\n-----------------------------------------------------------------\n\n**Table r1. Speedup of Cake over I/O-only \\ Compute-only method under different batch sizes of tokens and different models. Hardware: 1xA100, Seq-len: 16k**\n\nAs shown in Tables r1, Cake achieves an average Time-to-First-Token(TTFT) speedup of 1.96x over I/O-only methods and 2.25x over compute-only methods across various batch sizes. At smaller batch sizes, KV cache computation is memory-bound, underutilizing compute units. As batch size increases, it shifts toward compute-bound, boosting GPU efficiency. Generally, the speedup over I/O-only methods is higher with larger batch sizes, reflecting Cake’s increasing reliance on computation. These results highlight Cake’s ability to adapt to various compute efficiencies caused by different batch sizes, automatically optimizing TTFT.\n\n\nThanks again for your suggestion, and we are happy to address any further concerns.", "claims_and_evidence": "Yes", "methods_and_evaluation_criteria": "Evaluation is mainly based on the latency metrics (wall clock time). \nI don't see any inappropriate evaluation.", "theoretical_claims": "No theory claim; infra paper.", "experimental_designs_or_analyses": "it's sound enough. I like the overhead analysis, which further prove the usability of cake. \nOne simple concern: the authors didn't analyze the performance gains under memory-bounded and computing-bounded conditions by varying the batch size, which is a quite interesting study to fully know the potential of CAKE under different batch sizes and model sizes.", "relation_to_broader_scientific_literature": "it's related to the computation efficiency of Language models, including the deployment of LLMs, VLMs, and any LM archs with a heavy prefilling overhead.", "essential_references_not_discussed": "no", "other_strengths_and_weaknesses": "discussed above.", "other_comments_or_suggestions": "NA", "questions_for_authors": "NA", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Meta-review: KV fetching becomes an ever more important problem as models get longer context, and as ChatGPT-like systems become even more widely adopted. This paper proposes Cake, a KV-cache loading system that uses a clever scheduling strategy to enable more efficient loading of the KV cache. The reviewers generally agreed that the proposed approach is sensible, solves a practical problem, and backed up by solid experiments. A clear accept.
Decision: Accept (poster)