KVFlow: Efficient Prefix Caching for Accelerating LLM-Based Multi-Agent Workflows
Authors: Zaifeng Pan
AJJKUMAR PATEL
Yipeng Shen
Zhengding Hu
Yue Guan
Wan-Lu Li
Lianhui Qin
Yida Wang
Yufei Ding
Venue: NeurIPS 2025 poster ()
Keywords: LLM serving systems
multi-agent workflow
prefix caching
Primary area: infrastructure

Abstract: Large language model (LLM) based agentic workflows have become a popular paradigm for coordinating multiple specialized agents to solve complex tasks. To improve serving efficiency, existing LLM systems employ prefix caching to reuse key-value (KV) tensors corresponding to agents' fixed prompts, thereby avoiding redundant computation across repeated invocations. However, current systems typically evict KV caches using a Least Recently Used (LRU) policy, which fails to anticipate future agent usage and often discards KV caches shortly before their reuse. This leads to frequent cache misses and substantial recomputation or swap- ping overhead. We present KVFlow, a workflow-aware KV cache management framework tailored for agentic workloads. KVFlow abstracts the agent execution schedule as an Agent Step Graph and assigns each agent a steps-to-execution value that estimates its temporal proximity to future activation. These values guide a fine-grained eviction policy at the KV node level, allowing KVFlow to preserve entries likely to be reused and efficiently manage shared prefixes in tree-structured caches. Moreover, KVFlow introduces a fully overlapped KV prefetching mecha- nism, which proactively loads required tensors from CPU to GPU in background threads for agents scheduled in the next step, thereby avoiding cache miss stalls during generation. Compared to SGLang with hierarchical radix cache, KVFlow achieves up to 1.83× speedup for single workflows with large prompts, and up to 2.19× speedup for scenarios with many concurrent workflows.

KVFlow: Efficient Prefix Caching for Accelerating LLM-Based Multi-Agent Workflows
Zaifeng Pan'
Ajjkumar Patel
Yipeng Shen'
Zhengding Hu Yue Guan
Wan-Lu Li
Lianhui Qin'
Yida Wang*
Yufei Ding'
UCSD 2 AWS
Abstract
Large language model (LLM) based agentic workflows have become a popular paradigm for coordinating multiple specialized agents to solve complex tasks. To improve serving efficiency, existing LLM systems employ prefix caching to reuse key-value (KV) tensors corresponding agents' fixed prompts, thereby avoiding redundant computation across repeated invocations. However, current systems typically evict KV caches using Least Recently Used (LRU) policy, which fails to anticipate future agent usage and often discards KV caches shortly before their reuse. This leads to frequent cache misses and substantial recomputation swap- ping overhead. We present KVFlow, a workflow-awar KV cache management framework tailored for agentic workloads. KVFllw abstracts the agent execution schedule as an Agent Step Graph and assigns each agent a steps to-excuution value that estimates its temporal proximity to future activation. These values guide a fine-grained eviction policy at the KV node level, allowing KVFlow to preserve entries likely be reused and efficiently manage shared prefixes in tree-structured caches. Moreover, KVFlow introduces fully overlapped KV prefetching mecha- nism, which proactively loads required tensors from CPU to GPU in background threads for agents scheduled in the next step. thereby avoiding cache miss stalls during generation. Compared SGLang with hierarchical radix cache, K KVFlow achieves up to 1.83x speedup for single workflows with large prompts, and up to 2.19x x speedup for scenarios with many concurrent workflows.
Introduction
LLM-based agentic workflows coordinate multiple specialized agents, each defined by a fixed prompt and responsible for specific subtask, to solve complex problems in modular and interpretable way [1, 2, 3, 4, 5]. For example, MetaGPT [3] structures agent collaboration around software engineering roles such as Product Manager and Engineer. While this design improves reusability and coherence, it also leads to high inference latency due to the need to repeatedly invoke LLMs for each agent throughout the workflow.
To alleviate this overhead, existing agentic frameworks and applications [3, 6, 7, 9, 10] rely on LLM serving systems [11. 12, 13] equipped with system-level optimizations. A prevalent technique prefix caching [12, 14]. which reuses the key-value (KV) tensors produced by self-attention layers for static prompt tokens across decoding steps and requests. This is particularly beneficial in agentic workflows, where each agent is initialized with fixed prompt specifying name, responsibilities, and behavioral traits. Since these prompts remain constant across iterations, prefix caching avoids redundant computation on static content and significantly reduces per-agent inference latency.
"Corresponding author
39th Conference on Neural Information Processing Systems (NcurIPS 2025).

---

Agentic Workflow
Prefix Cache Usage Flow
Last acpess time the KV cacha
Timestamp=14
Rewninng or bneaking down
You need #eegrito
Cache Miss
Exarator Current Active Agent
Expresser
Arssoer cussioona From offerrt
Increased prefill latency!
Expresser
Least recently used, evicted!
Timestamp=13
Figure 1:
A cyclic agentic workflow abstraction consisting of four agents, Planner, Executor, Ex- presser, and Reviewer, adapted from [5]. timestamp 13, the Executor active and its KV cache is updated, which causes the Expresser's s cache to be evicted due to the LRU policy. At timestamp 14, when the Expresser becomes active again, a cache miss occurs and results in increased prefill latency.
However, prefix caching alone is insufficient in the presence of limited GPU memory. Existing systems typically adopt Least Recently Used (LRU) policy to evict KV caches that have not been accessed recently. We observe that this strategy can lead to suboptimal performance in agentic workflows. For instance, as illustrated in Figure 1, consider workflow [5] where four agents are organized into a sequential execution pipeline that is invoked iteratively. During the execution of the Executor agent shown the figure, the LRU policy identifies the Expresser's KV cache as the eviction candidate since it has not been accessed recently. This results in cache miss when the workflow proceeds to the Expresser agent, despite its imminent reuse. Such eviction behavior introduces unnecessary recomputation and degrades the overall efficiency of agentic execution.
To address the limitations of existing LLM serving systems in agentic workflows, we present KVFlow, a workflow-awaid KV cache management framework. We first introduce the Agent Step Graph, flexible abstraction that captures execution dependencies among agents and supports wide range of workflow structures, including conditional branching and synchronization barriers. Each agent node in the graph associated with computed steps-to-execction value, which estimates how soon the agent is expected to run. This value is derived through step aggregation functions that propagate across the graph, enabling KVFlow to reason about dynamic and structured execution patterns.
At runtime, KVFlow leverages this information to optimize cache behavior in two key ways. First, instead of using LRU, KVFlow adopts a workflow-aware eviction strategy that prioritizes evicting KV Since multiple agents can share common
caches belonging to agents with large ceeps-to-eeecute prefixes through a tree-structured cache, we further assign eviction priorities the cache node level to enable fine- -raigrained and efficient management. Second, KVFlow introduces a fully overlapped KV prefetching mechanism that proactively loads required KV tensors from CPU to GPU ahead of time, as we can predict the next invoked agents from the Agent Step Graph. This effectively eliminates prefix cache misses without stalling generation. Together, these optimizations significantly improve cache efficiency and reduce latency in executing agentic workflows.
In summary, this paper makes the following contributions:
We identify fundamental inefficiency in existing LLM serving systems, where the widely used based KV cache eviction strategy leads to suboptimal performance under agentic workflows.
We propose KVFlow, a workflow-aware KV cache management optimization that prioritizes eviction based on agent execution order and eliminates cache miss overhead via fully overlapped prefetching.
We conduct a comprehensive evaluation for KVFlow, showing that it significantly reduces cache miss overhead, achieving up to 1.83x and 29 speedups over SGLang with hi- erarchical radix cache under single workflows with large prompts and many concurrent workflows, respectively.

---

2 Background
Prefix Caching in LLM Serving Systems. To facilitate fine- theegraaied prefix reuse and eliminate redundant storage, modern LLM serving systems [11, 12] organize the KV cache into tree structure on the GPU, where each node stores segment of tokens and its corresponding KV tensors. Upon receiving a new request, the system matches the prefix from the root of the tree and concatenates the KV tensors along the matched path to reconstruct the full cached prefix. When GPU memory becomes insufficient, the system evicts nodes based on an LRU policy, Memory exhaustion can arise for two reasons. One common scenario is a high volume of concurrent user requests, each executing different agentic workflows, which leads to a large number of active KV cache entries. Another scenario occurs when the agent prompts are very large while the hardware capacity is limited.
Additionally, CPU memory can be configured as a secondary cache layer to back up evicted KV tensors, allowing cache swapping over PCIe. Despite the PCIe latency, swapping remains significantly faster than recomputing the KV tensors [15, 16].
Agentic Workflow. An agentic workflow [8, 10, 9, 7] is an LLM application paradigm that struc- tures multiple agents into an execution graph to collaboratively solve complex tasks. Compared to fully autonomous agents [17, 18, 19), agentic workflows leverage human domain expertise to achieve more consistent and robust performance across diverse tasks [3, 20, 21, 22, 23, 24, 25, 26]. The execution of each agent typically involves one or multiple LLM calls, with prompts composed of fixed part and a task-specific dynamic part. The fixed part usually encodes the agent's 's role, behavioral instructions, task description, and few-shot learning examples, and can be substantially large. For example, the fixed prompts of the TestBench Agent and the RTL Generator Agent in [22] contain lengthy few-shot learning examples, with over 3000 and 1000 tokens, respectively. Consequently, caching the corresponding KV of the fixed parts can significantly reduce prefill latency and improve the overall workflow execution efficiency. In contrast, the dynamic parts often contain the input questions or instructions from users, which are less valuable for caching.
3 Design of KVFlow
In this section, we present the design of KVFlow, which enhances prefix cache management for agentic workflows through two key techniques. First, we introduce a workflow-aware eviction policy that prioritizes KV nodes based on future usage, improving over the default LRU strategy. Second, propose an overlapped KV prefetching mechanism that hides CPU-GPU transfer latency via proactive loading and status-aware scheduling.
3.1 Workflow-Aware Eviction Policy
Existing LLM serving systems typically adopt an LRU eviction policy, which becomes suboptimal under agentic workflows. Specifically, an agent that is about to execute may have been idle for a long time, while agent that has just completed its execution might not be needed again in the near future. Moreover, the suffixes dynamically generated by recently executed agent often vary rapidly with task progress and are unlikely to be reused, yet they are still temporarily retained in the cache. With workflow information, we can predict the upcoming execution sequence of agents, enabling more informed eviction decisions and avoiding the inefficiencies caused by LRU.
To make eviction decisions based on workflow structure, we first need to capture the dependency relationships among agents. However, agent interactions in real-world workflows are highly diverse. As illustrated in Figure 2(a), the two workflows differ significantly: in the upper example, the Expresser agent depends on both Executorl and Executor2; in contrast, the lower workflow contains conditional branches, where Expresser can be triggered after either executor completes. Traditional abstractions such control-flow graphs (CFGs) or DAGs are insufficient to uniformly capture such diverse execution semantics.
Agent Step Graph and Steps-to-Executie
To address this, we introduce the Agent Step Graph abstraction, where each node corresponds to an agent invocation and edges encode dependency relations. Unlike conventional graphs, each node in the Agent Step Graph is associated with step aggregation function that determines how its
3

---

0
Sosps-to-xxooe topps-to-xoccuiinn
2
Step-age
Higher priority - More likely to evict
Steercher
Expoupor
Sync
Eviction Priority Propagation
+:00
Max(E1, E2jes
Plarner
+
Varying Suffixes
(Nighest Priority))
Zeval Expresser
Expouter
Merge with the Lowest Priority
Swatther
Conditonal
EONEMAA
Essizlor
+
Ponnn
MinE11, E2/11
AgHrE
agent's prampt
KV
+000
Inpur: EI OR
Expcclor
(a) Agents with their steps-to toeeccutieeeeennnee in two different Agent Step Graphs
(b) Eviction priority assignment for each KV node within cache tree. where multiple agents share partial prefix prompts.
Figure 2: Illustration of the workflow-aware eviction policy. (a) Each agentic workflow abstracted as an Agent Step Graph, where steps-to-executee values are computed using step aggregation functions over dependency edges. (b) These values are propagated through the cache tree assign eviction priorities at the KV node level. Nodes with smaller steps-to-cxecution are retained longer, reducing the chance of premature eviction.
steps-to-a porssto-exxene to-execution is derived from its predecessors. For prefix cache management, we focus solely on the earliest possible execution step of each agent and abstract away the specific type of dependency.
For example, in the upper workflow of Figure 2(a), the Expresser agent requires both upstream executors to complete, its step value is computed as max(E1, E2)
In the lower workflow, either path suffices, so the step value becomes min(E) E2) + 1. By applying these aggregation functions recursively, the Agent Step Graph enables unified computation of steps-to-cxecution across arbitrary multi-agent workflows.
Workflow-Aware Eviction Priority Assignment. Based on the steps-to-execution the Agent Step Graph, we design fine-gra graiirai eviction strategy that assigns priorities to KV cache nodes. As illustrated in Figure 2(b), agents with larger steps-to-execution are more likely to be evicted. Importantly, since agents may share common prefix segments in tree-structured cache layout, assign eviction priorities at the cache node level rather than at the agent level.
Specifically. we assign priorities only to the fixed prompt portion of each agent: all varying suffixes are always given the highest eviction priority to facilitate early eviction. For each agent, its steps- to-execution value is assigned to the last node of its fixed prompt and propagated upwards through the tree. When a node aggregates inputs from multiple agents, we assign it the minimum (i.e., least evictable) priority among its children, ensuring that shared nodes retained as long they useful to any agent in the near future.
This propagation scheme yields a priority map over the cache tree that dynamically reflects workflow- driven reuse potential. When GPU memory becomes constrained, KVFlow first evicts varying suffixes, and then incrementally evicts prefix KV nodes in descending order of their assigned priority, favoring eviction of those unlikely to be reused soon. This design naturally accommodates multiple concurrent workflows, with conflicts at shared nodes resolved by choosing the lowest (most conservative) priority across workflows.
Pseudocode. To more clearly illustrate how KVFlow integrates with KV cache management, we include pseudocode for both eviction priority assignment and the eviction procedure. When new agent request arrives with its Agent Step Graph (ASG) information, KVFlow updates the eviction priority of each cache node following Algorithm 1. When the system needs to evict cache nodes on the GPU free memory (e.g., during prefill, decode, or prefetch), proceeds from the leaf nodes in the cache tree based on their priorities, shown in Algorithm 2.

---

cache miss
Timeline
GPU
Planner
Executor
Execuror
Loading
Exckoter
CPU
Load Execular
prefelch
Fancater
GPU
Planner
Exocutor
Executor 2
Proactive Prefetching
In GPU memory
GPU
Load Exaculor
schedule In-GPU agents first'
Status KV Node
Backup CPU
Offlcading
GPU
Planner
Executor
2
Executor
Status-Aware Scheduling
Loading
CPU
Load Exeoutor
update status
Figure Illustration of overlapped KV prefetching. Compared reactive loading, KVFlow combines proactive prefetching that loads upcoming agents in advance with status-aware scheduling, minimizing the CPU-GPU transfer overhead. *The in-GPU agents can be within the same workflow or from another concurrent onc.
Algorithm 1: Priority Assignment
Algorithm 2: Eviction Procedure
Function PriorityAssign (ASG)
Function Evict (tree_cache, required)
for step, agent in ASG
leaves gea_ ge___eeaoooodeeetreeeeeeeeoe___e nodes(tree_cacche)
node=
heapify(leaves) ; // Construct max
agent.last fixed poompt_node;
heap based on priority
while node != root
while while,free_gee ftt.fe___p__emmmrrrroooooee < required
mode.counter[srer +=
node = heappop(leaves)
node.priority
free_gpu_mente += evict(node);
min(node. step); node node.parent
if node.parent becomes leaf heappush(leaves,
3.2 Overlapped KV Prefetching
While the workflow-aware eviction strategy avoids prematurely evicting agents that are about to execute, cache misses can still occur when an agent needs to run again after its KV cache has been evicted. This is particularly costly for long prompts, recomputing the KV cache from scratch incurs significant overhead. To mitigate this, we treat CPU memory as a secondary cache for storing the fixed prompt KV of evicted agents.
When CPU eaching is available, existing systems typically adopt reactive loading strategy, as illustrated in the top timeline of Figure 3. For instance, if Executor I's prefix cache has been offloaded CPU memory, the system reactively loads back only when Executor scheduled, thereby avoiding recomputation. However, CPU-to-GPU data transfers still introduce noticeable latency, especially for long prefixes.
Proactive Prefetching. To reduce this transfer overhead, we propose a proactive prefetching mechanism that leverages workflow information to asynchronously load the required KV cache advance. As shown in the second timeline Figure 3, while Planner is executing, anticipates that Executor will be invoked next and proactively prefetches its KV cache from CPU to GPU. Since the execution agents primarily involves model forward on the GPU and next token sampling (with model outputs transferred from the GPU to the CPU), and KV loading involves -PU-to-GPU transfer, the two operations utilize different hardware resources and can proceed in parallel without interference. Notably, PCIe enables full-duplex transfers, allowing bidirectional communication between CPU and GPU without contention. When the workflow contains branching. the system conservatively prefetches all agents that may be executed next based on the Step Graph, within a predefined limit on the number of concurrent prefetches.
system
However, prefetching alone is not always sufficient. When the current agent's execution time is shorter than the prefetch duration, generation may still blocked by incomplete KV loading. This common in high-coneurrency settings, where multiple workflows compete for CPU-GPU bandwidth

---

and cause queuing delays. This scenario is depicted in the second timeline of Figure 3, where Executor 1's generation is stalled despite prefetching.
Status-Aware Scheduling. To further reduce GPU idle time, we enhance the request scheduling policy with status awareness. In each scheduling step, a request's prefix cache is still in the loading process, the scheduler temporarily skips it and prioritizes other ready requests, such as Executor in Figure 3 or those from other concurrent workflows. To support this, we associate each cache node with status variable, which can be one of four states: GPU memory, backup in CPU, loading, or offloading. The scheduler inspects all nodes required by request, skips any with loading status to avoid redundant load attempts, and only dispatches the request once all dependencies are available. Upon completion, the background load thread updates the status of the cache nodes, informing readiness to the scheduler. Similarly, nodes in the offloading state are excluded from eviction decisions to avoid race conditions during memory reclaiming.
As illustrated in the third timeline of Figure 3, by combining proactive prefetching with status-aware scheduling, KVFlow effectively eliminates cache misses and fully overlaps GPU computation with prefetching, thereby hiding the CPU-GPU transfer latency.
3.3 Implementation
We implement the prototype of KVFlow based on SGLang v0.4.4 [12], an efficient LLM serving system that provides both a backend for LLM execution and a frontend interface for application development. SGLang's backend manages the prefix KV cache using a radix tree. We extend this mechanism to support our wrrkflow-wwae eviction policy and fully overlapped KV prefetching. In addition, we modify both the frontend and backend SGLang to support the transmission of agentic workflow information.
While our current prototype is integrated into SGLang's frontend API, method is not limited to SGLang. It can be adapted to other agentic workflow frameworks by modifying the HTTP requests that the frontend sends the server. Our backend optimization can also generalize to other LLM serving systems. For instance, vLLM [11] internally adopts automatic prefix caching and applies LRU the block level. Our workflow-awarc eviction logic can be applied at this block granularity by maintaining priority score for each block, enabling anticipatory eviction and prefetching.
Step Information Capture. Capturing the steps-to- texe-to-execctton information generated from the Agent Step Graph is essential for guiding our optimizations runtime. In our implementation, that each wtcchsgl..fco... function corresponds to an independent agent. During execution, we perform a just-in-time substitution of the LLM call to embed workflow metadata into the HTTP request. This metadata includes the identity of the current agent and the steps-to-executinn of all agents within the Agent Step Graph, indicating which agents may be invoked in subsequent steps. Upon receiving this information, the backend can update eviction priorities in the KV cache tree accordingly and trigger prefetching if the evictable GPU memory large enough.
assume
Prompt Segmentation. Besides capturing the step graph topology, we also need to track the last KV nodes each agent's fixed prompt, as shown in Figure 2. Distinguishing the fixed and dynamic parts of the prompt within a request presents a challenge. We offer two alternative solutions. First, we introduce a primitive interface that allows users to explicitly mark the end position of the fixed design a heuristic approach that tracks the agent's cache hit history and treats
part. Second, the consistently hit prefix as the fixed part. To avoid stale entries, KVFlow supports an adaptive mechanism that automatically removes fixed prompt nodes if they have not been reused beyond threshold period.
Client Tracking. In serving scenarios, multiple agentic workflows may execute concurrently on the same backend instance. Existing serving systems do not distinguish between request sources, potentially leading to naming conflicts. For example, two different workflows might both define an agent named "Planner". To address this, we assign unique client ID to each application. The client ID attached to every request sent to the backend, allowing the system to disambiguate agent identities and avoid interference between workflows originating from different clients.

---

3.4 Limitations
Our primary focus is on structured agentic workflows where future agent invocations can be estimated. KVFlow can handle dynamic workflows as long as the execution order of agents within a certain future window can be predicted at the current step. However, in highly unpredictable work where future execution cannot be inferred, KVFlow does not offer performance gains. In such cases, KVFlow gracefully falls back to SGLang's 's default behavior by sctting equal priority for all agent cache and disabling prefetching, without introducing additional overhead or correctness issues.
4
Evaluation
We evaluate KVFlow across range of microbenchmarks to understand its performance under different caching and execution conditions. Our experiments aim to answer the following key questions: (1) Can KVFlow reduce end-to-end latency for individual workflows with large prompt prefixes and limited GPU memory? (2) How does KVFlow perform under high concurrency, where multiple workflows run in parallel? To answer these questions, first analyze single-workflow latency in Section 4.1, and then study multi-workflow execution in Section 4.2.
Since KVFlow only modifies system-level cache management without affecting the model weights, prompts, or decoding logic, it is guaranteed to preserve the semantic correctness of the output. Therefore, our evaluation focuses exclusively on system performance metrics like the latency.
4.1 Single-Worte Workfl-W Latency
We begin by evaluating the latency of executing a single agentic workflow under batch size = 1. This single-request latency reflects interactive usage scenarios where workflows are triggered individually by user, such as in notebooks or development tools. Unlike online serving systems that rely on batching for throughput, these settings prioritize responsiveness for individual requests.
We benchmark a sequential 10-stage agentic workflow. As described in Section 2, each agent's input prompt consists of a fixed prefix (shared across invocations) and a dynamic suffix (which varies across runs). We generate synthetic input prompts by randomly sampling token sequences with controlled lengths for both parts. We evaluate two variants: (a) fully deterministic sequential workflows where each stage only has one agent, i.c., branches=1; and (b) moderately dynamic workflows, where each stage randomly selects one of two agents with partially shared prefixes, i.e., branches=2. The latter one introduces moderate unpredictability while still preserves opportunities for prefetching.
Models and testbeds. We conduct experiments on Qwen2.5-32B on an NVIDIA H100 GPU with 80GB memory and 64 GB/s PCIe Gen5 bandwidth. Qwen uses 40 attention heads and 8 KV heads. We adopt deterministic decoding (temperature = 0, greedy sampling) to ensure consistent latency measurements. This setting represents scenarios with tight GPU memory constraints when long fixed prefixes contend for cache space.
Baselines. We compare KVFlow against two SGL.ang configurations. The first, denoted as SGLang. maintains radix-structured KV cache in GPU memory without CPU backup. When GPU memory insufficient, prefix nodes are evicted and must be recomputed from scratch upon reuse. For the second configuration, denoted as SGLang w/ HiCache, we enable the hierarchical radix cache in SGLang, which SGLang's 's default CPU-based cache extension. extends SGLang's 's radix tree by asynchronously backing up frequently used cache nodes to host memory. If a node is accessed after eviction, it is loaded back from the CPU instead of being recomputed. To further reduce CPU-GPU transfer cost, SGLang with HiCache overlaps the GPU computation of layer I with the loading of layer [+1, forming a simple two-stage pipeline. Both of SGLang and SGLang w/ HiCache use LRU eviction policy when GPU memory used up. We also include vL..M [11] as baseline, which organizes KV cache into blocks and adopts an LRU--based block eviction policy.
Evaluation Methods. We first warm up the cache by executing each agent's fixed prompt multiple times, ensuring its prefix cache is constructed and backed up to CPU memory (for HiCache). We then execute the 10-stage workflow ten times, each with a varying dynamic suffix, to obtain the end-to-end latency for these ten runs. This simulates realistic usage patterns with repeated workflow invocations
7

---

Speedup over Lalang
SGLang
HiCache
VLLM
KVflow
Speedup over SGLang
SGLang
HiCache
VLLM
KVFlow
1.5
1.0
1.0
0.5
0.5
0.0
0.0
4096/32/32
4096/256/256
8192/32/32
6192/256/256
4096/32/32
4096/256/26
8192/32/32
8192/256/256
(a) Owen2.5-32B on H100, branches=1
(h) Qwen2.5-32B on H100, branches=2
Figure 4: Speedup over SGLang (GPU-only cache) on Qwen2.5-3P2 5-32B for 10-stage workflow. Left: branches=1 (deterministic sequential). Right: branches=2 (each stage randomly selects one of two agents). Horizontal axis: Fixed part token Dynamic part token Output token.
or loop-like behavior [5]. We then repeat this process ten times to report the average latency and the standard deviation.
Results. Figure 4 shows the speedup over SGLang (GPU-only cache) with error bars under different prompt configurations Fixed / Dynamic/ / Output. We intentionally test large fixed prefix sizes (e.g.. 8192 tokens) to exceed GPU memory and force evictions.
Across all settings, VFFlow consistently achieves the highest speedup, demonstrating its effectiveness both fully deterministic workflows (branches=l) and moderately dynamic workflows (branches=2). For instance, under 4096/32/32 with branches=1, KVFllw outperforms SGLang w/ HiCache by 1.24x, and the GPU-only SGLang baseline by 1.42x. This because our workflow-aware eviction and prefetch strategy effectively hides the CPU-GPU transfer overhead. While HiCache reduces recomputation overhead, it still suffers from pipeline cold-start and limited overlap when compute is shorter than transfer.
We also observe that SGLang w/ HiCache generally performs better than the GPU-only SGLang baseline, as loading from CPU is typically faster than recomputing. However, in some large-context settings on H100 (e.g., 8192/32/32), HiCache shows marginal or degraded performance. This may stem from suboptimal scheduling in SGLang's pipelining logic, where CPU-GPU transfer is not effectively overlapped under memory contention or high transfer volume.
As the number of output tokens increases, the relative gain from KVFlow diminishes. The reason is that in these settings, the LLM decoding latency dominates total runtime, reducing the proportion of time affected by cache loading. There are many works optimizing the time-consuming decoding steps for the auto-regressive LLMs, including speculative decoding [27, 28, 291, KV cache sparsity [30], and early exit [31], which are orthogonal to our work and can be co-applied with KVFlow.
Meanwhile, the speedup from KVFlow increases with fixed prompt length. When fixed tokens set to 8192, the average speedup reaches 1,30x, compared to only 1.22x at fixed = 4096. This is because the cache miss incurs higher overhead as the prefix length grows, and KVFlow's proactive cache management becomes increasingly beneficial.
Optimization Breakdown. We perform breakdown analysis of KVFlow's two optimization components on top of SGLang w/ HiCache on H100. We observe that by enabling workflow-awarc eviction alone top of HiCache provides average 1.11x x speedup. By further enabling overlapped prefetching optimization addition to eviction, we can improve the speedup to 1.29 x, demonstrating that both components contribute significantly.
Overhead Analysis. KVFlow adds negligible overhead. First, CPU-side priority assignment and prefetch scheduling overlap with GPU computation under SGLang's 's vero-overhead batch scheduler, and our profiling shows no exposed CPU cost. Second, prefetching does not interfere with decoding because token generation incurs no PCIe transfers, so KV movement overlaps with compute. Third, when prefetched KV is unused, neither PCIe bandwidth nor memory is wasted, as transfers run only during active compute, and buffers are overwritten in place.

---

Speedup over SGLang
SaLang
KVPlow
Rxcd
Dynamic
Output
HCacche
Speedup over SGLang
@wen2.5-328
Llama3-88
0.004
1.0
1.0
Density
0.003
0.022
0.5
0.5
0.0
512/201aak
0.0
1024/0-04
912/120-1aa
1024/84-Box
0.000
250
Token Number
600
Owen//16Tasst
Liama/128BTTssk
Figure 5: High-coneurrency workflow performance comparison under different tiec-pprompecconeuureeeyyy settings on an H100.
Figure 6: Token distribution for fixed, dynamic, and output parts across PEER-style work- flows.
Figure 7: Speedup of KVFlow over SGLang and HiCache on PEER- style multi-agent applica- tions.
4.2
Cogeu-onncrrnncy Workflow Performance
We further evaluate system performance under high concurrency by simultaneously launching multiple independent workflows single H100 GPU. These workflows are all sequential workflows (branches=I), and they are assumed to be non-interacting and non-sharing. As shown in Figure 5, we benchmark four configurations with models Qwen2.5-328 and Llama-3.1 each labeled by the fixed prompt length per agent and the number of concurrent workflows. The dynamic and output token lengths fixed at 256. For each setting, we choose proper concurrency that the GPU can accommodate without exhausting memory for prefix caching. If concurrency is too high, all available memory consumed by active requests, and the system can longer maintain reusable prefix caches, placing it beyond the scope of our optimization.
Results. According to Figure 5, across all settings, KVFlow consistently outperforms both SGLaang and HiCache, achieving an up to 1. 1.25x speedup. The performance improvement of KVFlow with 1024 fixed prompt tokens higher than 512, as the cache miss overhead is higher. Notably, HiCache performs particularly poorly under high concurrency, even falling behind SGLang in multiple cases. For example, it only achieves 0.57 performance of SGLang without CPU-based cache under 1024 fixed prompt tokens with 64 concurrent workflows. We suspect this is due to frequent cache misses triggering reactive load-back operations. which disrupt SGLang's schedule-compute pipeline. Additionally, due to the fragmented layout of KV storage in SGLang, the PCIe bandwidth cannot be fully utilized. While KVFlow also does not resolve the fragmentation issuc, it achieves much better overlap of PCIe transfers and GPU compute through more reasonable evictions and proactive prefetching. As result, it yields an up to 2.19x performance gain over the naive LRU-based HiCache with reactive loading.
Realistic Workflow Simulation. To better reflect real-world deployment scenarios, we simulate agentic workflows based on the PEER [5] framework. In our setup, each workflow consists of four agents, instantiated using the workflow templates provided by PEER. For each agent, we sample a role and an instruction, and prompt an LLM to generate the agent's prompt. Due to the inherent randomness in LLM sampling, the generated prompts exhibit variability even when the roles and instructions are similar. Meanwhile, since all agents operate under the same application context, their prompts often share partially overlapping prefixes. This results in workflows that are both diverse and partially redundant, reflecting the common characteristics of real-worle multi-agent applications.
We use the Financial QA dataset from PEER the workflow input. The resulting workloads are moderate in scale, with agent prompts typically ranging from a few dozen to several hundred tokens. Figure 6 shows the distribution of fixed, dynamic, and output token lengths across all agents. Figure 7 presents the performance comparison between KVFlow, SGLang, and SGLang with HiCache. KVFlow achieves clear improvements over both SGLang and HiCache, resulting in up to 1.12x and 1.08x speedups. The results demonstrate the strong practical potential of KVFlow for multi multi-application serving in realistic deployment settings.

---

Related Work
LLM Serving Optimizations. A broad line of work improves online LLM serving by optimizing request scheduling, including continuous batching (also known as iteration-level scheduling) [32], multi-level feedback queues to mitigate head-of-line blocking [33], quality-of-experienee aware schedulers tailored to streaming scenarios [34], etc. Another set of efforts focus on KV cache management. vLLM proposes PagedAttention [11] to reduce memory fragmentation via paged storage of KV tensors, while SGLang introduces Ratixittention [12] to eliminate redundancy in prefix caching. Several works also target chatbot scenarios with specialized prefix caching strategies [16, 35]. InferCept [36] predicts tool calling durations and uses a cost model to decide whether to retain, swap, or discard the KV cache of intercepted requests. These optimizations are orthogonal to VVFlow, which focuses on workflows formed by multiple agents. While Autellix [37] and ParrotServe [38] explore request scheduling agentic workflows, they do not consider prefix cache management, making their objectives complementary to ours.
Agentic Workflow Frameworks Recent works have proposed diverse multi-agent frameworks [3, 4. 6, 7, 39, 40, 41] that organize agents into structured roles to collaboratively solve complex tasks. These frameworks provide built-in mechanisms for message passing and dependency construction between agents, convenient integration of tool usage and reasoning methods within agent actions, predefined abstractions for common agent roles and behaviors, and efficient multi-threaded execution to support concurrent agent collaboration. Some of these systems abstract the agentic workflow as a computation graph, where nodes represent L.LM-invoking agents and edges denote control flow message dependencies. This abstraction enables the application of graph-level transformations, such as edge pruning [42], operator insertion [8], topology optimization [7, 10], to improve application correctness or quality. However, these frameworks remain focused on application-layer construction and rely on conventional LLM serving infrastructures handle generation. In contrast, our work leverages the structure of agentic workflows to optimize the serving system itself, targeting backend efficiency under multi-agent execution workloads.
6 Conclusion
We present KVFlow, a workflow-aware KV cache management framework for optimizing LLM serving in agentic workflows. By abstracting agent execution as a Step Graph and computing each agent's 's creps-to-execution KVFlow enables a principled eviction strategy that anticipates future usage. It further introduces fully overlapped KV prefetching mechanism proactively eliminate cache miss stalls. Our evaluations show that KVFlow significantly improves serving efficiency over existing systems in workflows with long prompts or high concurrency. While prior work multi-agen systems has predominantly focused on designing frontend application logic and interaction protocols, KVFlow highlights the importance of workflow semantics in enabling system-level optimizations.
7
Acknowledgment
We sincerely thank the anonymous NeurIPS reviewers for their valuable feedback and insightful suggestions. We also appreciate the UCSD MLSys Group for their helpful comments. This work was supported in part by NSF grant 2124039, UC AÍ LEAP fund, and the Amazon research award.
References
[1] Shunyu Yao, Jeffrey Zhao, Dian Yu, Nan Du, Izhak Shafran, Karthik Narasimhan, and Yuan Cao. React: Synergizing reasoning and acting in language models. In International Conference on Learning Representations (ICLR), 2023.
[2] Noah Shinn, Federico Cassano, Ashwin Gopinath, Karthik Narasimhan, and Shunyu Yao. Reflexion: Language agents with verbal reinforcement learning. Advances in Neural Information Processing Systems, 36:8634 8652, 2023.
[3] Sirui Hong, Xiawu Zheng, Jonathan Chen, Yuheng Cheng, Jinlin Wang. Ceyao Zhang, Zili Wang, Steven Ka Shing Yau, Zijuan Lin, Liyang Zhou, et al. Metagpt: Meta programming for multi-agent collaborative framework. arXiv preprint 200.0035220 3(4):6, 2023.
10

---

[4] Guohao Li, Hasan Hammoud, Hani Itani, Dmitrii Khizbullin, and Bernard Ghanem. Camel: Communicative agents for" mind" exploration of large language model society. Advances in Neural Information Processing Systems, 36:51991-52000 1920085 2023.
[5] Yiying Wang, Xiaojing Li, Binzhu Wang. Yueyang Zhou, Yingru Lin, Han Ji, Hong Chen, Jinshi Zhang, Fei Yu, Zewei Zhao, et al. Peer: Expertizing domain-specific tasks with a multi-agent framework and tuning methods. arXiv preprint arXiv:2407 A.iv:2407.06983, 2024.
[6] Qingyun Wu, Gagan Bansal, Jieyu Zhang. Yiran Wu, Beibin Li, Erkang Zhu, Li Jiang, Xiaoyun Zhang, Shaokun Zhang, Jiale Liu, et al. Autogen: Enabling next-gen Ilm applications via multi-agent conversation. arXiv preprint arXiv:2308.08/33 2023.
[7] Mingchen Zhuge, Wenyi Wang, Louis Kirsch, Francesco Faccio, Dmitrii Khizbullin, and Jürgen Schmidhuber. Gptswarm: Language agents optimizable graphs. In Forty-first International Conference on Machine Learning, 2024.
[8] Jiayi Zhang, Jinyu Xiang, Zhaoyang Yu, Fengwei Teng, Xionghui Chen, Jiaqi Chen, Mingchen Zhuge, Xin Cheng, Sirui Hong, Jinlin Wang, et al. Aflow: Automating agentic workflow generation. arXiv preprint 20.::2400.107000 2024.
[9] Xuchen Pan, Dawei Gao, Yuexiang Xie, Yushuo Chen, Zhewei Wei, Yaliang Li, Bolin Ding, Ji-Rong Wen, and Jingren Zhou. Very large-scale multi-agent simulation in agentscope. arXiv preprint arXiv:2407.17778 2024.
[10] Zijian He, Reyna Abhyankar, Vikranth Srivatsa, and Yiying Zhang. Cognify: Supercharging gen-ai workflows with hierarchical autotuning, arXiv preprint arXiv:2502.08056 2025.
[11] Woosuk Kwon, Zhuohan Li, Siyuan Zhuang, Ying Sheng, Lianmin Zheng, Cody Hao Yu, Joseph Gonzalez, Hao Zhang, and Ion Stoica. Efficient memory management for large language model serving with pagedattention. In Proceedings of the 29th Symposium on Operating Systems Principles, pages 61 626, 2023.
[12] Lianmin Zheng, Liangsheng Yin, Zhiqiang Xie, Jeff Huang, Chuyue Sun, Cody Hao Yu, Shiyi Cao, Christos Kozyrakis, Ion Stoica, Joseph E. Gonzalez, Clark W. Barrett, and Ying Sheng. Sglang:
Efficient execution of structured language model programs. Advances in Neural Information Processing Systems, 37:62557 2024.
2025. [14] vLLM Team, Automatic Prefix Caching. https:///ocs... vllm http:////./////aaatt///aatatse cettomtticc.pefii_cctttie htmm,, 2025.
[13] NVIDIA. TensorRT-LLLM. http:://ttaab..o////DDDTTTTeeete
[15] Chao Jin, Zili Zhang, Xuanlin Jiang, Fangyue Liu, Xin Liu, Xuanzhe Liu, and Xin Jin. Rageache: Efficient knowledge caching for strieval-augmennt generation. arXiv preprint 1rX14:2404.12457, 2024.
[16] Bin Gao, Zhuomin He, Puru Sharma, Qingxuan Kang, Djordje Jevdjic, Junbo Deng. Xingkun Yang, Zhou Yu, and Pengfei Zuo. (Cost-Effiicien large language model serving for multi- turn conversations with (CachedAttention). In 2024 USENIX Annual Technical Conference (USENIX ATC 24), pages 111-126, 2024.
[17] Joon Sung Park, Joseph O' O'Brien, Carrie Jun Cai, Meredith Ringel Morris, Percy Liang, and Michael Bernstein. Generative agents: Interactive simulacra of human behavior. In Proceed- ings of the 36th annual acm symposium on user interface software and technology, pages 1-22, 2023.
[18) Lei Wang, Chen Ma, Xueyang Feng, Zeyu Zhang, Hao Yang, Jingsen Zhang, Zhiyuan Chen, Jiakai Tang, Xu Chen, Yankai Lin, al. A survey on large language model based autonomous 2024.
agents. Frontiers of Computer Science,
[19] Shuyan Zhou, Frank F Xu, Hao Zhu, Xuhui Zhou, Robert Lo, Abishek Sridhar, Xianyi Cheng, Tianyue Ou, Yonatan Bisk, Daniel Fried, et al. Webarena: A realistic web environment for building autonomous agents. arXiv preprint arXiv:2307.13854 2023.

---

[20] Tal Ridnik, Dedy Kredo, and Itamar Friedman. Code generation with alphacodium: From prompt engineering to flow engineering. arXiv preprint arXiv:2401.08500 2024.
[21] Zhenhailong Wang, Shaoguang Mao, Wenshan Wu, Tao Ge, Furu Wei, and Heng Ji. Unleashing the emergent cognitive synergy in large language models: A task-solving agent through multi- persona self collbcollaooration. arXiv preprint arXiv:2307.05. 00300, 2023.
[22] Yujie Zhao, Hejia Zhang, Hanxian Huang, Zhongming Yu, and Jishen Zhao. Mage: A multi- agent engine for automated rtl code generation. arXiv preprint arXiv:2412.07822 2024.
[23] Chen Qian, Wei Liu, Hongzhang Liu, Nuo Chen, Yufan Dang, Jiahao Li, Cheng Yang. Weize Chen, Yusheng Su, Xin Cong, al. Chatdev: Communicative agents for software development. arXiv preprint ar wxx::230..009222 2023.
[24] Yilun Du, Shuang Li, Antonio Torralba, Joshua B Tenenbaum, and Igor Mordatch. Improv- ing factuality and reasoning in language models through multiagent debate. In Forty-first International Conference on Machine Learning, 2023.
[25] Genghan Zhang, Weixin Liang, Olivia Hsu, and Kunle Olukotun. Adaptive self-improvement IIm agentic system for ml library development. arXiv preprint arXiv:2502.02534 2025.
[26] Yile Gu, Yifan Xiong, Jonathan Mace, Yuting Jiang, Yigong Hu, Baris Kasikci, and Peng Cheng. Argos: Agentic time-series anomaly detection with autonomous rule generation via large language models. arXiv preprint 2xXiv:2501.14170 2025.
[27] Charlie Chen, Sebastian Borgeaud, Geoffrey Irving. Jean-Baptiste Lespiau, Laurent Sifre, and John Jumper. Accelerating large language model decoding with speculative sampling. arXiv preprint arXiv:2302.013311 2023.
[28] Yaniv Leviathan, Matan Kalman, and Yossi Matias. Fast inference from transformers via speculative decoding. In International Conference on Machine Learning, pages 19274-19286. PMLR, 2023.
[29] Apoorv Saxena. Prompt lookup decoding, November 2023.
[30] Guangxuan Xiao, Yuandong Tian, Beidi Chen, Song Han, and Mike Lewis. Efficient streaming language models with attention sinks. arXiv, 2023.
[31] Yichao Fu, Junda Chen, Siqi Zhu, Zheyu Fu, Zhongdongming Dai, Aurick Qiao, and Hao Zhang. Efficiently serving Ilm reasoning programs with certaindex. arXiv preprint arXiv:2412.2099) 2024.
[32] Gyeong-In Yu, Joo Seong Jeong, Geon Woo Kim, Soojeong Kim, and Byung-Gon Chun. Orca: distributed serving system for (Transformer-Basee generative models. In 16th USENIX Symposium on Operating Systems Design and Implementation (OSDI 22), pages 521-538, 2022.
[33] Bingyang Wu, Yinmin Zhong, Zili Zhang, Shengyu Liu, Fangyue Liu, Yuanhang Sun, Gang Huang, Xuanzhe Liu, and Xin Jin. Fast distributed inference serving for large language models. arXiv preprint arXiv:2300.008: 05920, 2023.
[34] Jiachen Liu, Jae- Won Chung, Zhiyu Wu, Fan Lai, Myungjin Lee, and Mosharaf Chowdhury. Andes: Defining and enhancing quality-of experrinee in llm-based text streaming services. arXiv preprint 2XXx:22444.1688. 2024.
[35] Lingfan Yu, Jinkun Lin, and Jinyang Li. Stateful large language model serving with pensieve. In Pruceedings of the Twentieth European Conference on Computer Systems, pages 144-158, 2025.
[36] Reyna Abhyankar, Zijian He, Vikranth Srivatsa, Hao Zhang, and Yiying Zhang. Infercept: Efficient intercept support for augmented large language model inference. arXiv preprint arXiv:2402.01800 2024.
[37] Michael Luo, Xiaoxiang Shi, Colin Cai, Tianjun Zhang, Justin Wong, Yichuan Wang, Chi Wang. Yanping Huang, Zhifeng Chen, Joseph Gonzalez, et Autellix: An efficient serving engine for Im agents general programs. arXiv preprint 2rXiv 13965, 2025.
12

---

[38] Chaofan Lin, Zhenhua Han, Chengruidong Zhang, Yuqing Yang, Fan Yang, Chen Chen, and Lili Qiu. Parrot: Efficient serving of (LLM-based] applications with semantic variable. In 18th USENIX Symposium Operating Systems Design and Implementation (OSDI 24), pages 929-99 945, 2024.
[39] LangChain. LangGraph.
2025. wttp://...ttrroo.c..oo.ooo http:/pppc...commeenneeeiigg
[40] Anthropic. Building effective agents. 2024.
[41] Dawei Gao, Zitao Li, Xuchen Pan, Weirui Kuang, Zhijian Ma, Bingchen Qian, Fei Wei, Wenhao Zhang, Yuexiang Xie, Daoyuan Chen, et al. Agentscope: A flexible yet robust multi-agent platform. arXiv preprint arXiv:2402.1403:
2024.
[42] Guibin Zhang, Yanwei Yue, Zhixun Li, Sukwon Yun, Guancheng Wan, Kun Wang, Dawei Cheng, Jeffrey Xu Yu, and Tianlong Chen. Cut the crap: An economical communication pipeline for lim-based multi-agent systems. arXiv preprint arXiv:2410.02506, 2024.
13

---

NeurIPS Paper Checklist
1. Claims
Question: Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?
Answer: [Yes]
Justification: Our abstract and introduction accurately reflect the paper's contributions in optimizing the prefix caching for multi-agent workflows.
Guidelines:
The answer NA means that the abstract and introduction do not include the claims made the paper.
The abstract and/or introduction should clearly state the claims made, including the contributions made in the paper and important assumptions and limitations. A No
NA answer to this question will not be perceived well by the reviewers.
The claims made should match theoretical and experimental results, and reflect how much the results can be expected to generalize to other settings.
is fine to include aspirational goals as motivation as long as is clear that these goals not attained by the paper.
2. Limitations
Question: Does the paper discuss the limitations the work performed by the authors?
Answer: [Yes]
Justification:
We explicitly include limitation section.
Guidelines:
The answer NA means that the paper has no limitation while the answer No means that the paper has limitations, but those are not discussed in the paper.
The authors are encouraged to create a separate "Limitations" section their paper.
The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (e.g., independence assumptions, noiseless settings, model well-specification, asymptotic approximations only holding locally). The authors should reflect how these assumptions might be violated in practice and what the implications would be.
The authors should reflect on the scope of the claims made, e.g., if the approach was only tested on few datasets or with few runs. In general, empirical results often depend on implicit assumptions, which should be articulated.
The authors should reflect on the factors that influence the performance of the approach. For example, facial recognition algorithm may perform poorly when image resolution is low images are taken in low lighting. Or speech-to-text system might not be used reliably to provide closed captions for online lectures because it fails to handle technical jargon.
The authors should discuss the computational efficiency of the proposed algorithms and how they scale with dataset size.
If applicable, the authors should discuss possible limitations of their approach to address problems of privacy and fairness.
While the authors might fear that complete honesty about limitations might used by reviewers as grounds for rejection, worse outcome might be that reviewers discover limitations that aren't It acknowledged judgment and recognize that individual actions in favor of transparency play impor- tant role in developing norms that preserve the integrity of the community. Reviewers will be specifically instructed to not penalize honesty concerning limitations.
the paper. The authors should use their best
3. Theory assumptions and proofs
Question: For each theoretical result, does the paper provide the full set of assumptions and complete (and correct) proof?
Answer: [NA]
14

---

Justification:
This paper does not contain any theoretical results.
Guidelines:
The answer NA means that the paper does not include theoretical results.
All the theorems, formulas, and proofs in the paper should be numbered and cross- referenced.
All assumptions should be clearly stated or referenced the statement of any theorems.
The proofs can either appear the main paper or the supplemental material, but they appear in the supplemental material, the authors are encouraged to provide short proof sketch to provide intuition.
Inversely, any informal proof provided in the core of the paper should be complemented by formal proofs provided in appendix supplemental material.
Theorems and Lemmas that the proof relies upon should be properly referenced.
4. Experimental result reproducibility
Does the paper fully disclose all the information needed to reproduce the main ex- perimental results of the paper to the extent that it affects the main claims and/or conclusions of the paper (regardless of whether the code and data are provided or not)?
Question:
Answer: [Yes]
Justification: We provide the detailed SGLang version (v0.4.4) and hardware setups for reproducibility.
Guidelines:
The answer NA means that the paper does include experiments. If the paper includes experiments, No answer to this question will not be perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data are provided or not.
If the contribution is dataset and/or model, the authors should describe the steps taken to make their results reproducible or verifiable.
Depending on the contribution, reproducibility can be accomplished in various ways. For example, if the contribution is novel architecture, describing the architecture fully might suffice, or if the contribution is a specific model and empirical evaluation, it may be necessary to either make possible for others to replicate the model with the same dataset, or provide access to the model. In general. releasing code and data is often one good way accomplish this, but reproducibility can also be provided via detailed instructions for how to replicate the results, access to a hosted model (e.g.. in the case of large language model), releasing of a model checkpoint, or other means that are appropriate to the research performed.
While NeurIPS does not require releasing code, the conference does require all submis- sions to provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution. For example
(a) If the contribution is primarily a new algorithm, the paper should make it clear how reproduce that algorithm. (b) If the contribution is primarily
model architecture, the paper should describe the architecture clearly and fully.
(c) If the contribution is a new model (e.g., a large language model), then there should either be way to access this model for reproducing the results or way to reproduce the model (e.g., with an open-source dataset instructions for how to construct the dataset).
(d) We recognize that reproducibility may be tricky in some cases, in which case authors are welcome deseribe the particular way they provide for reproducibility, In the case of closed-source models, it may be that access to the model is limited in some way (e.g., registered users), but it should be possible for other researchers to have some path to reproducing or verifying the results.
5. Open access to data and code
Question: Does the paper provide open access to the data and code, with sufficient instruc- tions to faithfully reproduce the main experimental results, as described in supplemental material?
15

---

Answer: [No]
Justification:
We will open source our codes at https://github. wtwpsttubbb.om///aaaaffnnng
KVFlow.
Guidelines:
The answer NA means that paper does not include experiments requiring code.
Please see the NeurlPS code and data submission guidelines (https://nips cc/
for more details.
While we encourage the release of code and data, we understand that this might not be possible, so "No" is an acceptable answer. Papers cannot be rejected simply for not including code, unless this is central to the contribution (e.g., for new open-source benchmark).
The instructions should contain the exact command and environment needed to to reproduce the results. See the NeurIPS code and data submission guidelines (https: //iips.cocc httppscccppulllc//geeeee Seuuiddee//oeeuuuu
ssi ssionPolicy for more details.
The authors should provide instructions on data access and preparation, including how to access the raw data, preprocessed data, intermediate data, and generated data, etc.
The authors should provide scripts to reproduce all experimental results for the new proposed method and baselines. only subset of experiments are reproducible, they should state which ones are omitted from the script and why.
At submission time, preserve anonymity, the authors should release anonymized versions (if applicable).
Providing as much information as possible in supplemental material (appended to the paper) recommended, but including URLs to data and code permitted.
6. Experimental setting/details
Question: Does the paper specify all the training and test details (e.g., data splits, hyper- parameters, how they were chosen, type of optimizer, etc.) necessary to understand the results?
Answer: [Yes]
Justification:
We discuss the experiments details in our evaluation section.
Guidelines:
The answer NA means that the paper does not include experiments.
The experimental setting should be presented in the core of the paper to level of detail that is necessary to appreciate the results and make sense of them.
The full details can be provided either with the code, in appendix, or as supplemental material.
7. Experiment statistical significance
Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?
Question:
Answer: [No]
Justification: We perform the experiments repeatedly and report the standard deviations in the main result figure.
Guidelines:
The answer NA means that the paper does not include experiments.
The authors should answer "Yes" if the results are accompanied by error bars, confi- dence intervals, or statistical significance tests, at least for the experiments that support the main claims of the paper.
The factors of variability that the error bars capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, or overall with given experimental conditions).
The method for calculating the error bars should be explained (closed form formula, call to a library function, bootstrap, etc.)
The assumptions made should be given (e.g., Normally distributed errors).
16

---

It should be clear whether the error bar is the standard deviation or the standard error of the mean.
is OK to report 1-sigma error bars, but one should state it. The authors should preferably report a 2-sigma error bar than state that they have 96% CI, if the hypothesis of Normality of errors is not verified.
For asymmetric distributions, the authors should be careful not to show in tables or figures symmetric error bars that would yield results that are out range (e.g. negative error rates).
error bars are reported in tables or plots, The authors should explain the text how they were calculated and reference the corresponding figures or tables in the text.
8. Experiments compute resources
For each experiment, does the paper provide sufficient information on the com- puter resources (type of compute workers, memory, time of execution) needed to reproduce the experiments?
Question:
Answer: [No]
Justification: We provide the specific hardware types but do not explicitly include the execution time. However, the execution time should be short as we only perform inference
jobs.
Guidelines:
The answer NA means that the paper does not include experiments.
The paper should indicate the type of compute workers CPU or GPU, internal cluster, cloud provider, including relevant memory and storage.
The paper should provide the amount of compute required for each of the individual experimental rums as well as estimate the total compute.
The paper should disclose whether the full research project required more compute than the experiments reported in the paper (e.g., preliminary or failed experiments that didn't make into the paper).
9. Code ethics
Question: Does the research conducted the paper conform, in every respect, with the NeurlPS Code of Ethics https: htttts:///eurpppi Saabblic/thhcscuideeee.
Answer: [Yes]
Justification: Our research conducted in the paper conform every respect with NeurIPS Code of Ethics.
Guidelines:
The answer NA means that the authors have reviewed the NeurIPS Code of Ethics.
If the authors answer No, they should explain the special circumstances that require deviation from the Code of Ethics.
The authors should make sure to preserve anonymity (e.g., there a special consid- eration due to laws regulations in their jurisdiction).
10. Broader impacts
Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?
Answer: [NA]
Justification: The paper presents system optimizations for accelerating executing agentic workflows, without bringing any societal impacts directly.
Guidelines:
The answer NA means that there is no societal impact of the work performed.
If the authors answer NA or No, they should explain why their work has no societal impact why the paper does not address societal impact.
17

---

Examples of negative societal impacts include potential malicious unintended uses (e.g., disinformation, generating fake profiles, surveillance), faireess considerations (e.g., deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations.
The conference expects that many papers will be foundational research and not tied to particular applications, let alone deployments. However, if there is direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point out that generic algorithm for optimizing neural networks could enable people to train models that generate Deepfakes faster.
The authors should consider possible harms that could arise when the technology is being used intended and functioning correctly, harms that could arise when the technology is being used as intended but gives incorrect results, and harms following from (intentional unintentional) misuse of the technology.
If there are negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition to attacks, mechanisms for monitoring misuse, mechanisms monitor how system learns from feedback over time, improving the efficiency and accessibility of ML).
11. Safeguards
Does the paper deseribe safeguards that have been put in place for responsible release of data or models that have a high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)?
Question:
Answer: [NA]
Justification: The paper only involves serving system optimizations and puses no such risks.
Guidelines:
The answer NA means that the paper poses no such risks.
Released models that have high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere to usage guidelines or restrictions to access the model or implementing safety filters.
Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images.
We recognize that providing effective safeguards is challenging, and many papers do not require this, but we encourage authors to take this into account and make best faith effort.
12. Licenses for existing assets
Are the creators or original owners of assets (e.g., code, data, models), used in the paper, properly credited and are the license and terms of use explicitly mentioned and properly respected?
Question:
Answer: [Yes]
Justification: We cite all assets properly.
Guidelines:
The answer NA means that the paper does use existing assets.
The authors should cite the original paper that produced the code package dataset.
The authors should state which version of the asset used and, possible, include URL.
The name of the license (e.g., CC-BY 4.0) should be included for each asset.
For scraped data from a particular source (e.g., website). the copyright and terms of service of that source should be provided.
If assets are released, the license, copyright information, and terms use in package should be provided. For popular datasets, paperswithcode. thcode. com/ datasets has curated licenses for some datasets. Their licensing guide can help determine the license of dataset.
18

---

For existing datasets that are re-packaged, both the original license and the license of the derived asset (if has changed) should be provided.
If this information is not available online, the authors are encouraged to reach out to the asset's s creators.
13. New assets
Question: Are new assets introduced in the paper well documented and is the documentation provided alongside the assets?
Answer: [NA]
Justification:
The paper does not release new assets.
Guidelines:
The answer NA means that the paper does not release new assets.
Researchers should communicate the details of the dataset/code/moddl as part of their submissions via structured templates. This includes details about training, license, limitations, etc.
The paper should discuss whether and how consent was obtained from people whose asset used.
At submission time, remember to anonymize your assets (if applicable). You can either create anonymized URL or include anonymized zip file.
14. Crowdsourcing and research with human subjects
Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given to participants and screenshots, if applicable, as well as details about compensation (if any)?
Answer: [NA]
Justification: The paper does not involve crowdsourcing nor research with human subjects.
Guidelines:
The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.
Including this information in the supplemental material is fine, but if the main contribu- tion of the paper involves human subjects, then as much detail as possible should be included in the main paper.
According the NeurIPS Code of Bthies, workers involved in data collection, curation, or other labor should be paid at least the minimum wage in the country of the data collector.
15. Institutional review board (IRB) approvals or equivalent for research with human subjects
Question: Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or equivalent approval/review based the requirements of your country or institution) were obtained?
Answer: [NA]
Justification: The paper does not involve crowdsoureing nor research with human subjects.
Guidelines:
The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.
Depending on the country in which research is conducted, IRB approval (or equivalent) may be required for any human subjects research. If you obtained IRB approval, you should clearly state this in the paper.
We recognize that the procedures for this may vary significantly between institutions and locations, and we expect authors to adhere to the NeurlPS Code of Ethics and the guidelines for their institution.
For initial submissions, not include any information that would break anonymity (if applicable), such the institution conducting the review.
19

---

16. Declaration of LLM usage
Question: Does the paper describe the usage of LLMs if it is an important, original, or non-standard component of the core methods in this research? Note that if the LLM is used only for writing, editing, or formatting purposes and does not impact the core methodology. scientific rigorousness, originality of the research, declaration is not required.
Answer: [NA]
Justification: We only use LLMs for polishing papers and helping writing several scripts.
Guidelines:
The answer NA means that the core method development in this research does not involve LLMs as any important, original, non-standard components.
Please refer to L.LM policy (https: http///eerrrppppee for what should or should not be described.
20


Reviews:
[{"review_id": "pitZLNAkmC", "reviewer": "Reviewer_Gzk9", "summary": "This paper introduces KVFlow, a novel workflow-aware KV cache management framework designed to improve the serving efficiency of LLM based multi-agent workflows. Existing LLM systems often use a Least Recently Used (LRU) policy for KV cache eviction, which can lead to suboptimal performance due to frequent cache misses and recomputation in agentic workflows where agents' fixed prompts are repeatedly invoked. KVFlow addresses this by proposing two key techniques: a workflow-aware eviction policy and a fully overlapped KV prefetching mechanism. The paper demonstrates that KVFlow significantly reduces cache miss overhead.", "questions": "n/a", "limitations": "n/a", "rating": 4, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 2, "significance": 3, "originality": 3, "strengths_and_weaknesses": "- Strengths:\n\n1. The paper identifies a crucial inefficiency in existing LLM serving systems, specifically the limitations of LRU-based KV cache eviction in the context of agentic workflows. \n2. The introduction of the \"Agent Step Graph\" and the concept of \"steps-to-execution\" is a strong conceptual contribution. This allows for a more intelligent and anticipatory eviction policy that prioritizes KV caches based on their likelihood of future reuse within the workflow, moving beyond the limitations of simple LRU. \n3. The paper describes the implementation of KVFlow as a prototype based on SGLang, highlighting its practical applicability and demonstrating how it extends existing systems. \n\n- Weaknesses and Areas for Improvement:\n\n1. While the evaluation includes sequential workflows and PEER-style simulations, exploring the performance of KVFlow on an even wider variety of agentic workflow structures (e.g., highly branching, deeply nested, or dynamic graph changes) could further solidify its generalizability. The current evaluation primarily focuses on scenarios where prefixes are likely to be reused, and a broader analysis of less predictable or more complex workflow patterns would be valuable to demonstrate robust performance across diverse real-world multi-agent applications.\n2. The current implementation of KVFlow is presented as a prototype based on SGLang. While this demonstrates its feasibility, the paper could elaborate on the generalizability of KVFlow's design principles to other LLM inference frameworks (e.g., vLLM, DeepSpeed-MII, etc.). \n3. A discussion on the architectural modifications or APIs required for plug-and-play integration into different existing multi-agent frameworks would strengthen the paper's impact and suggest broader applicability.\n4. The paper states that \"current systems typically evict KV caches using a Least Recently Used (LRU) policy,\" and then compares KVFlow to \"SGLang with hierarchical radix cache.\" While SGLang's default eviction policy might implicitly be LRU or a variant, it would strengthen the comparison if the paper explicitly discussed and presented results against a clear, standalone LRU-based KV cache eviction mechanism as a direct baseline.\n5. The paper would benefit from a more explicit ablation study to clearly demonstrate the individual contributions of KVFlow's two main components: the workflow-aware eviction policy and the fully overlapped KV prefetching mechanism.\n6. It would be beneficial to discuss how KVFlow specifically handles or integrates with KV cache management strategies during the decoding phase.", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for your insightful comments and questions. We are happy to further improve our manuscript based on your suggestions.\n\n### (W1) Evaluation Scope\n\nThank you for the suggestion. We extend the evaluation with a new 10-stage workflow, where each stage randomly selects one agent from two branches with partially shared prefixes. Other configurations are following the paper. This setup introduces moderate unpredictability while still preserving opportunities for prefetching. As shown in the table below, for Qwen2.5-32B on an H100, KVFlow achieves up to 1.459$\\times$ and 1.935$\\times$ speedups over SGLang and SGLang w/ HiCache, demonstrating its effectiveness under branched execution. \nWorkflows that are either fully deterministic or contain a limited number of branches are still common in current applications [1, 2, 3].\nWhile highly unpredictable workflows are outside our optimization scope, KVFlow can gracefully fall back to match that of HiCache by setting equal priority for all agent cache and disabling prefetching. We will add the results and discussion in Sec 4.\n\n> [1] Hong, Sirui, et al. \"MetaGPT: Meta Programming for A Multi-Agent Collaborative Framework.\"\\\n> [2] Qian, Chen, et al. \"ChatDev: Communicative Agents for Software Development.\"\\\n> [3] Wang, Yiying, et al. \"PEER: Expertizing domain-specific tasks with a multi-agent framework and tuning methods.\"\n\n**Speedups over SGLang for branched workflows with Qwen2.5-32B on an H100.**\n|Configuration|SGLang|HiCache|KVFlow|\n|-|-|-|-|\n|4096/32/32|1.000±0.016|1.099±0.024|**1.302±0.058**|\n|4096/256/256|1.000±0.012|1.001±0.017|**1.052±0.012**|\n|8192/32/32|1.000±0.001|0.754±0.014|**1.459±0.025**|\n|8192/256/256|1.000±0.003|0.924±0.008|**1.122±0.007**|\n|512/20-task|1.000±0.016|0.885±0.044|**1.225±0.021**|\n|1024/10-task|1.000±0.010|1.014±0.035|**1.140±0.007**|\n\n\n### (W2) Generalizability to Other Frameworks\n\nThank you for the comment. KVFlow builds on general cache control and prefetching concepts and can be ported to other LLM serving systems. For instance, vLLM internally adopts automatic prefix caching and applies LRU at the block level. Our workflow-aware eviction logic can be applied at this block granularity by maintaining a priority score for each block, enabling anticipatory eviction and prefetching. We will add this discussion in Sec 3.3.\n\n### (W3) API Modification\n\nKVFlow requires minimal extensions of traditional LLM call APIs. Specifically, we require (1) a client ID to distinguish requests from different workflows, and (2) the workflow information, including current activated agent name, and the steps-to-execution values for other agents. These inputs allow our runtime to assign priority scores for eviction and prefetching. We will add this illustration in Sec 3.3.\n\n### (W4) Clarification on LRU Baseline\n\nThank you for the comment, and we apologize for the confusion. Both SGLang and SGLang with HiCache adopt exact LRU eviction policies. The difference lies in how they handle evicted entries. SGLang stores KV cache only in GPU memory, so evicted entries must be recomputed when needed again. In contrast, HiCache backs up evicted entries to CPU memory, allowing them to be directly reloaded. We will revise Section 4.1 to clarify this distinction.\n\n### (W5) Optimization Ablation\n\nThank you for the suggestion. We performed a breakdown analysis of the two optimization components on top of SGLang with hierachical cache (HiCache). The table below compares the performance of using workflow-aware eviction alone with the combination of eviction and prefetching. Workflow-aware eviction alone provides an average speedup of 1.109$\\times$ over HiCache, while enabling prefetching in addition to eviction further improves the speedup to 1.288$\\times$.\n\n**Optimization breakdown for sequential workflows with Qwen2.5-32B on an H100**\n|Configuration|HiCache|+Workflow-Aware Eviction|+Prefetch|\n|-|-|-|-|\n|4096/32/32|1|1.236|1.358|\n|4096/256/256|1|1.037|1.047|\n|8192/32/32|1|1.217|1.623|\n|8192/256/256|1|1.000|1.223|\n|512/20-task|1|1.063|1.323|\n|1024/10-task|1|1.105|1.152|\n\nWe also evaluated prefetching in isolation. However, in this case, the prefetched nodes frequently conflicted with those selected for eviction, leading to significant performance degradation. This indicates that prefetching must be combined with workflow-aware eviction in order to be effective. We will add these results and discussions in Sec 4.\n\n\n\n### (W6) Cache Management Integration\n\nThank you for the suggestion. To more clearly illustrate how KVFlow integrates with KV cache management, we will include pseudocode for both eviction priority assignment and the eviction procedure in the revised version. When a new agent request arrives with its Agent Step Graph (ASG) information, KVFlow updates the eviction priority of each cache node as follows:\n\n```python\ndef PriorityAssign:\n    for step, agent in ASG:\n        node = agent.last_fixed_prompt_node  # Last node of the agent's fixed prompt\n        while node != root:\n            node.counter[step] += 1\n            node.eviction_priority = min(node.eviction_priority, step)\n            node = node.parent\n```\n\nWhen the system needs to evict cache nodes on the GPU to free memory (e.g., during prefill, decode, or prefetch), it proceeds from the leaf nodes in the cache tree based on their priorities:\n\n```python\ndef Evict:\n    leaves = get_leaf_nodes(tree_cache)\n    heapify(leaves) # Construct max heap based on eviction priority\n    while free_gpu_memory < required:\n        node = heappop(leaves)\n        free_gpu_memory += evict(node)\n        if node.parent becomes a leaf after eviction:\n            heappush(leaves, node.parent)\n```"}, {"review_id": "1aDTMwA7ZU", "reviewer": "Reviewer_Y4Pj", "summary": "The paper presents KVFlow, an optimization framework designed to enhance prefix caching efficiency in large language model (LLM)-based multi-agent workflows. Existing LLM serving systems typically employ a least-recently-used (LRU) cache eviction policy, which often leads to unnecessary recomputation and degraded performance in workflows involving multiple agents with fixed prompts. To address this, KVFlow introduces two core innovations: (1) a workflow-aware eviction policy guided by an \"Agent Step Graph,\" which accurately anticipates the execution order of agents, assigning cache eviction priorities based on their predicted temporal proximity; and (2) a proactive, fully overlapped prefetching mechanism that asynchronously transfers cached key-value (KV) tensors from CPU to GPU ahead of their anticipated reuse, thus minimizing latency. The authors implement and evaluate KVFlow within the SGLang framework, demonstrating significant latency reductions—achieving up to 1.83× speedup for individual workflows with large prompts and up to 2.19× improvement under high concurrency scenarios—compared to existing baseline approaches.", "questions": "1.  It is unclear how KVFlow handles dynamic workflows. The Agent Step Graph seems to be constructed from a known workflow structure. How would KVFlow deal with agents that are chosen at runtime (not just conditional branches) or workflows that evolve?\n2.  The core KVFlow system is implemented on a modified SGLang backend. Will the authors release this code or detailed algorithms? Adding pseudo-code for key parts (priority assignment, eviction) would improve clarity. Also, explicitly state any hyperparameters (e.g. how far ahead prefetching looks) and whether KVFlow incurs extra overhead (CPU or memory). \n3. The speedup results are compelling, but error bars or run-to-run variance are not shown. Please include standard deviations (or at least min/max) over multiple runs, especially for the main claims (Fig. 5–8). \n4.  The paper emphasizes latency reduction, but what about trade-offs? For instance, does maintaining the Agent Step Graph and doing prefetching add CPU overhead or additional memory usage? Are there cases where proactive prefetching wastes bandwidth (e.g. if prefetched KV is not used)?", "limitations": "The paper does not include an explicit Limitations section, although the authors note they target “multi-agent workflows with limited GPU memory”. It would be valuable for them to acknowledge key assumptions: for example, KVFlow assumes knowledge of future agent calls and sufficient CPU memory for offloading. Discussing cases where these assumptions break (e.g. highly unpredictable agent invocation, memory-constrained environments without CPU caching) would give readers a more balanced view.", "rating": 4, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 3, "significance": 3, "originality": 3, "strengths_and_weaknesses": "Strengths:\n1. The paper identifies a clear performance issue in multi-agent LLM workflows, namely that naive Least-Recently-Used (LRU) prefix caching often evicts KV entries just before they are needed again. The authors propose a well-motivated solution, KVFlow, which leverages workflow semantics (via an “Agent Step Graph”) to predict agent invocation order and guide cache management. The methodology is technically sound.\n2. The evaluation is thorough and well-structured. \n3. The paper is generally well-written and organized. \n4. The work shows practical impact: real deployments could serve more agentic workflows or use smaller GPUs at equal throughput.\n\nWeaknesses:\n1. Although the evaluation is extensive, it lacks statistical analysis. Reported speedups (e.g. 1.83×) appear to be averages, but no error bars or significance tests are shown. The paper does not discuss variance across runs, which is important for performance claims.\n2.  The clarity of some methodological details could be improved. For example, the mechanism by which “steps-to-execution” priorities propagate through the shared prefix tree is described in prose, but no pseudocode or algorithm is provided. Readers may find it hard to reproduce the exact cache-priority logic.", "was_revised": true, "final_justification": "The major initial concerns regarding statistical rigor and methodological clarity were fully resolved, which significantly raised my confidence and assessment of quality and reproducibility. These aspects carry substantial weight and strongly influence my overall positive recommendation. Clarification regarding dynamic workflow support was sufficiently addressed, thus maintaining a high evaluation of generalizability and practicality. The remaining minor issue on quantitative overhead measurements slightly moderates my recommendation but does not substantially affect the paper’s overall merit or novelty.\nOverall, the authors successfully addressed the most critical concerns, strongly supporting the recommended positive evaluation.", "rebuttal": "Thank you for your insightful comments and questions.\nWe are happy to further improve our manuscript based on your suggestions.\n\n### (W1) & (Q3) Statistical Analysis in Evaluation\n\nThank you for the suggestion. We report the speedups with standard deviations in the table below. The results show that KVFlow delivers highly stable performance improvements. We will revise Section 4 to include these results.\n\n**Speedups over SGLang for sequential workflows with Qwen2.5-32B on an H100**\n|Configuration|SGLang|HiCache|KVFlow|\n|-|-|-|-|\n|4096/32/32|1.000±0.002|1.022±0.074|**1.389±0.045**|\n|4096/256/256|1.000±0.004|0.996±0.022|**1.043±0.010**|\n|8192/32/32|1.000±0.043|0.738±0.030|**1.198±0.030**|\n|8192/256/256|1.000±0.013|0.928±0.006|**1.138±0.003**|\n|512/20-task|1.000±0.029|0.950±0.048|**1.256±0.010**|\n|1024/10-task|1.000±0.020|0.983±0.043|**1.132±0.015**|\n\n### (W2) & (Q2) Pseudocode\n\nThank you for the suggestion. We will include pseudocode for both the eviction priority assignment and the eviction procedure in our next version. When a new agent request arrives with its Agent Step Graph (ASG) information, KVFlow updates the eviction priority of each cache node as follows:\n\n```python\ndef PriorityAssign:\n    for step, agent in ASG:\n        node = agent.last_fixed_prompt_node  # Last node of the agent's fixed prompt\n        while node != root:\n            node.counter[step] += 1\n            node.eviction_priority = min(node.eviction_priority, step)\n            node = node.parent\n```\n\nWhen the system needs to evict cache nodes on the GPU to free memory (e.g., during prefill, decode, or prefetch), it proceeds from the leaf nodes in the cache tree based on their priorities:\n\n```python\ndef Evict:\n    leaves = get_leaf_nodes(tree_cache)\n    heapify(leaves) # Construct max heap based on eviction priority\n    while free_gpu_memory < required:\n        node = heappop(leaves)\n        free_gpu_memory += evict(node)\n        if node.parent becomes a leaf after eviction:\n            heappush(leaves, node.parent)\n```\n\nFor the prefetch distance hyperparameter, we use a value of 1 in our experiments, which is already good enough to hide data transfer. This value can be adjusted at runtime for different agents based on their workloads.\n\n\n### (Q1) Dynamic Workflows\n\nThank you for the comment. Our primary focus is on structured agentic workflows where future agent invocations can be estimated. KVFlow can handle dynamic workflows as long as the execution order of agents within a certain future window can be predicted at the current step. In such cases, KVFlow adaptively updates the steps-to-execution values at runtime and continues to make informed eviction and prefetching decisions.\n\nWe extend the evaluation with a 10-stage dynamic workflow, where each stage randomly selects one agent from two branches with partially shared prefixes. The results are shown in the following table, where all configurations other than the branching pattern follow those in the paper.\n\n**Speedups over SGLang for branched workflows with Qwen2.5-32B on an H100**\n|Configuration|SGLang|HiCache|KVFlow|\n|-|-|-|-|\n|4096/32/32|1.000±0.016|1.099±0.024|**1.302±0.058**|\n|4096/256/256|1.000±0.012|1.001±0.017|**1.052±0.012**|\n|8192/32/32|1.000±0.001|0.754±0.014|**1.459±0.025**|\n|8192/256/256|1.000±0.003|0.924±0.008|**1.122±0.007**|\n|512/20-task|1.000±0.016|0.885±0.044|**1.225±0.021**|\n|1024/10-task|1.000±0.010|1.014±0.035|**1.140±0.007**|\n\nWorkflows that are either fully deterministic or contain a limited number of branches are still common in current applications [1, 2, 3]. In highly unpredictable workflows where future execution cannot be inferred, KVFlow does not offer performance gains. In such cases, the system gracefully falls back to the default HiCache behavior by setting equal priority for all agent cache and disabling prefetching, without introducing additional overhead or correctness issues. We will include this discussion in a dedicated Limitations section.\n\n> [1] Hong, Sirui, et al. \"MetaGPT: Meta Programming for A Multi-Agent Collaborative Framework.\"\\\n> [2] Qian, Chen, et al. \"ChatDev: Communicative Agents for Software Development.\"\\\n> [3] Wang, Yiying, et al. \"PEER: Expertizing domain-specific tasks with a multi-agent framework and tuning methods.\"\n\n\n### (Q4) Overhead Analysis\n\nThank you for the question. KVFlow introduces minimal runtime overhead. First, priority assignment and prefetch scheduling are handled on the CPU side. SGLang’s zero-overhead batch scheduler [4] allows these computations to be overlapped with GPU execution, effectively hiding the associated CPU cost. According to our profiling, we do not see CPU overhead that cannot be hidden by GPU computation. Second, prefetching does not interfere with LLM decoding, as token generation does not involve CPU–GPU data transfer over PCIe. This allows the transfer of prefetched KV to proceed in parallel with computation. Third, in cases where prefetched KV is ultimately not used, no PCIe bandwidth is wasted, since prefetching only takes place during active GPU computation. It also does not introduce additional memory overhead as the memory can be directly overwritten. If a cache miss occurs, the ongoing prefetch is safely preempted. While unused prefetching may incur additional energy consumption, our design prioritizes performance and responsiveness. We will clarify this discussion in Sec 4 in the next version.\n\n\n> [4] SGLang v0.4: Zero-Overhead Batch Scheduler, Cache-Aware Load Balancer, Faster Structured Outputs."}, {"review_id": "pMkYX9qZMO", "reviewer": "Reviewer_ojFE", "summary": "This paper introduces KVFlow, a system for efficient KV (key-value) cache management in large language model (LLM)-based multi-agent workflows. The main motivation is that existing LLM serving frameworks, which often use an LRU policy for prefix KV caching, are not optimized for the structured and recurrent access patterns seen in agentic workflows. KVFlow addresses this by introducing a workflow-aware cache eviction policy grounded in an Agent Step Graph abstraction, enabling prediction-driven eviction at the KV node level. The framework also includes an overlapped prefetching mechanism to reduce CPU-GPU data transfer stalls, backed by status-aware scheduling. Experiments compare KVFlow with SGLang and SGLang's hierarchical cache extension under various agentic workflow loads, demonstrating notable speedups, especially for workloads with large static prompts and under high concurrency.", "questions": "How does KVFlow handle workflows with highly non-deterministic or user-interactive structures, where agent execution order may change unpredictably? Are there performance regressions or fallback mechanisms in such cases?\n\nCan you provide more formal evaluation or ablation on the heuristic fixed/dynamic prompt segmentation’s impact? For example, is there any case where misclassification of prefix segments leads to inefficient cache usage or performance drops?\n\nHave you assessed the overhead of maintaining and updating the Agent Step Graph and steps-to-execution metadata, especially in high-churn or large-scale deployments? Are there memory, CPU, or engineering challenges?", "limitations": "yes", "rating": 4, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 3, "significance": 2, "originality": 2, "strengths_and_weaknesses": "Strengths:\nThe paper addresses a clear inefficiency in LLM serving—LRU-based KV cache eviction poorly matches agentic workflows. KVFlow introduces a principled, workflow-aware eviction strategy using the Agent Step Graph abstraction, which models synchronization and branching dependencies (Fig. 3a). Its node-level priority assignment enables fine-grained cache management, particularly under prefix sharing scenarios (Fig. 3b). The proposed status-aware scheduling and overlapped prefetching effectively reduce CPU-GPU idle time (Fig. 4). Experiments across models, prompt sizes, and concurrency levels (Figs. 5–8) show up to 2.91× speedup over SGLang, validating performance claims. The work is reproducible in theory and relevant to real-world LLM use cases, especially in multi-agent and modular setups.\n\nWeaknesses:\nKVFlow is tailored for structured agentic workflows, limiting its applicability to unstructured or user-driven generation. Its reliance on accurate step graphs assumes robust frontend-backend integration, but fallback heuristics may not handle complex or ambiguous cases well. The paper lacks detailed evaluation on prefetching overhead and scalability in large or highly branching workflows. Experiments focus on synthetic or PEER-inspired patterns without broader real-world diversity, and statistical robustness is limited (e.g., no variance or significance analysis). Comparisons to relevant baselines like InferCept are missing. Some concepts, such as dynamic prompt detection, lack formal definitions.", "was_revised": true, "final_justification": "After considering the rebuttal, discussions, and clarifications from the authors, I find that several of my original concerns have been addressed, and the manuscript has improved in clarity and completeness.\n\nResolved Issues\nHighly Non-Deterministic Workflows: Authors clarified KVFlow’s fallback behavior and added a limitation statement. They provided additional branched-workflow experiments showing consistent gains in moderately dynamic cases.\nFixed/Dynamic Prompt Detection: Provided both manual and heuristic strategies, discussed robustness to misclassification, and clarified fallback behavior without correctness loss.\nLarge-Scale Agent Step Graph Overhead: Explained that computation is localized to a small future window and can be hidden by GPU generation, citing SGLang’s scheduler.\nStatistical Reporting: Standard deviations are now included for all performance results, demonstrating low variance and stable improvements.\nComparison to InferCept and vLLM: Clarified orthogonality with InferCept and included vLLM baseline results, showing KVFlow’s consistent advantage.\n\nRemaining Considerations\nWhile the approach handles moderately dynamic workflows well, performance in fully unstructured or highly unpredictable workflows remains inherently limited. This is acknowledged as a limitation.\nIntegration and evaluation with additional large-scale, real-world agentic workloads (beyond those in the paper) could further strengthen generality claims, but this is a minor point given the targeted scope.\n\nOverall Assessment\nThe authors’ responses are thorough, supported by additional experiments and clear explanations. The work now clearly differentiates KVFlow’s contribution, shows robust performance across scenarios, and addresses methodological transparency. Remaining limitations are inherent to the problem scope and are acknowledged. I will maintain a positive score, as the rebuttal resolves key concerns and strengthens both the technical and empirical contributions.", "rebuttal": "Thank you for your insightful comments and questions.\nWe are happy to further improve our manuscript based on your suggestions.\n\n### (Q1) Highly Non-Deterministic Workflows\n\nThank you for the question. KVFlow is designed for structured agentic workflows where the execution path can be reasonably estimated. \nKVFlow can handle dynamic workflows as long as the execution order of agents within a certain future window can be predicted at the current step. In such cases, KVFlow adaptively updates the steps-to-execution values at runtime and continues to make informed eviction and prefetching decisions.\nIn highly non-deterministic workflows, KVFlow gracefully falls back to default HiCache behavior by setting equal priority for all agent cache and disabling prefetching. We will clarify this in a distinct limitation section.\n\nWorkflows that are either fully deterministic or contain a limited number of branches are still common in current applications [1, 2, 3]. We extend the evaluation with a 10-stage dynamic workflow with branches, and the results are shown in the belowing table, where all configurations other than the branching pattern follow those in the paper.\n\n**Speedups over SGLang for branched workflows with Qwen2.5-32B on an H100**\n|Configuration|SGLang|HiCache|KVFlow|\n|-|-|-|-|\n|4096/32/32|1.000±0.016|1.099±0.024|**1.302±0.058**|\n|4096/256/256|1.000±0.012|1.001±0.017|**1.052±0.012**|\n|8192/32/32|1.000±0.001|0.754±0.014|**1.459±0.025**|\n|8192/256/256|1.000±0.003|0.924±0.008|**1.122±0.007**|\n|512/20-task|1.000±0.016|0.885±0.044|**1.225±0.021**|\n|1024/10-task|1.000±0.010|1.014±0.035|**1.140±0.007**|\n\n> [1] Hong, Sirui, et al. \"MetaGPT: Meta Programming for A Multi-Agent Collaborative Framework.\"\\\n> [2] Qian, Chen, et al. \"ChatDev: Communicative Agents for Software Development.\"\\\n> [3] Wang, Yiying, et al. \"PEER: Expertizing domain-specific tasks with a multi-agent framework and tuning methods.\"\n\n\n### (Q2) Fixed/Dynamic Prompt Detection\n\nThank you for the question. KVFlow allows users to manually specify the end position of fixed prompts based on application logic, or alternatively use a heuristic strategy. Specifically, we mark segments that are repeatedly reused by the same agent across invocations as fixed. Each agent can have multiple fixed prompts, forming a tree structure under shared prefixes. To avoid stale entries, KVFlow supports an adaptive mechanism that automatically removes fixed prompt nodes if they have not been reused beyond a threshold period.\n\nWhen the segmentation is partially incorrect, KVFlow still performs well without correctness issues. This case is analogous to workflows with a limited number of branches, as shown in the previous table, where KVFlow prefetches all candidate nodes and therefore achieves similar performance. In the extreme case where all fixed prompts are misclassified, KVFlow falls back to HiCache’s reactive cache loading or performs recomputation without performance degradation. We will add this discussion in Sec 3.3.\n\n\n### (Q3) Large-Scale Agent Step Graph\n\nThank you for the question. The overhead of maintaining the Agent Step Graph and updating steps-to-execution values is minimal. This is because we only need to consider agents within a certain step (e.g., 5) of current activated agents even when the entire graph is very large. The CPU overhead introduced by the scheduling of KVFlow can be hidden with GPU generation, thanks to SGLang's zero-overhead batch scheduler [4] design. We will add this dicussion in Sec 3.3.\n\n> [4] SGLang v0.4: Zero-Overhead Batch Scheduler, Cache-Aware Load Balancer, Faster Structured Outputs.\n\n### Statistical Analysis\n\nThank you for pointing this out. We now include standard deviations for key performance metrics across multiple runs. In all reported cases, KVFlow consistently outperforms baselines with low variance, confirming the stability of our results.\n\n**Speedups over SGLang for sequential workflows with Qwen2.5-32B on an H100**\n|Configuration|SGLang|vLLM|HiCache|KVFlow|\n|-|-|-|-|-|\n|4096/32/32|1.000±0.002|1.041±0.003|1.022±0.074|**1.389±0.045**|\n|4096/256/256|1.000±0.004|1.031±0.005|0.996±0.022|**1.043±0.010**|\n|8192/32/32|1.000±0.043|1.057±0.045|0.738±0.030|**1.198±0.030**|\n|8192/256/256|1.000±0.013|1.041±0.014|0.928±0.006|**1.138±0.003**|\n|512/20-task|1.000±0.029|1.091±0.033|0.950±0.048|**1.256±0.010**|\n|1024/10-task|1.000±0.020|1.046±0.021|0.983±0.043|**1.132±0.015**|\n\n### InferCept Baseline\n\nThank you for the comment. InferCept focuses on memory management during intra-agent interceptions such as tool calls, whereas KVFlow addresses prefix caching across agents by leveraging inter-agent execution patterns. These two approaches target different layers of the execution stack and are orthogonal in design. In fact, they could be combined: KVFlow’s eviction and prefetching mechanisms can manage inactive agents, while InferCept’s eviction policy can optimize memory usage within active, intercepted agents. We will clarify this distinction and potential complementarity in the related work section.\n\nBesides, we now include the vLLM baseline, which organizes KV cache into blocks and adopts LRU-based block eviction policy. Our experiments show that KVFlow outperforms it consistently. vLLM is slightly faster than SGLang in our experiments, but KVFlow still consistently outperforms both. Since KVFlow is built on top of SGLang with HiCache, comparing against SGLang better reflects the impact of our proposed optimizations. Our approach is general and can also be applied to vLLM or other LLM serving frameworks."}, {"review_id": "8XwbS0CAEB", "reviewer": "Reviewer_KtZF", "summary": "This paper addresses the inefficiency issues when LLM serving for multi-agent workflows. The authors propose KVFlow, a workflow-aware approach to manage prefix caches  for agents with fixed prompts. Instead of the usual least-recently-used policy, KVFlow builds an “Agent Step Graph” to estimate when each agent will run next. Using this, the system evicts less-needed KV entries and even prefetches future entries in the background. The goal is to reduce cache misses and recomputation. Experiments (on synthetic Llama-3.1-8B and Qwen2.5-32B setups) show up to about 1.8×–2.2× speedup compared to a baseline GPU-only cache system.", "questions": "1. It is not clear how heavy the overhead of graph maintenance and computing priorities is. Can the authors provide more details on the runtime overhead of KVFlow itself?\n\n2. The notion of “Agent Step Graph” assumes a known workflow in advance. How does KVFlow handle dynamic or conditional branches in a workflow that are not known beforehand?", "limitations": "N/A", "rating": 4, "confidence": 2, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 2, "significance": 3, "originality": 3, "strengths_and_weaknesses": "**Strengths**\n\n1. This is an important research problem. The paper clearly identifies a real bottleneck in agentic workflows, where naive LRU eviction causes frequent recomputation. Managing KV caches intelligently is an important problem for efficient LLM serving.\n\n2. Using a workflow graph with steps-to-execution values for cache decisions seems novel. It is interesting to combine scheduling information with cache management. The idea of fine-grained eviction and fully overlapped prefetching is creative and practical.\n\n3. The experimental results indicate noticeable speedups (up to ~2×) in relevant scenarios. The improvement is consistent across both single long workflows and many concurrent workflows, suggesting the method can benefit different settings. The quantitative gains are a strong point.\n\n**Weaknesses**\n\n1. The experiments use synthetic data (random token sequences) rather than real dialog or code tasks. Also, they compare mainly with one baseline (SGLang variants). There is little discussion of how the system performs under real-world workloads or against other existing optimizations. \n\n2. The method is tailored for a specific “agentic” workflow setting with fixed prompts. The title mentions “Key-Value Routing” but the paper mostly discusses caching rather than any routing mechanism; this confusion reduces clarity.", "was_revised": true, "final_justification": "Based on the rebuttal, I recommend that this paper be accepted.", "rebuttal": "Thank you for your insightful comments and questions.\nWe are happy to further improve our manuscript based on your suggestions.\n\n### (W1) Workload and Baselines\n\nThank you for the feedback. In addition to synthetic data, we evaluate KVFlow using dataset and agentic workflows derived from the PEER [3] paper in Section 4.2, which is a framework used widely in production. These workflows are semantically meaningful and simulate realistic agent coordination patterns.\n\nWorkflows that are either fully deterministic or contain a limited number of branches are still common in current applications [1, 2, 3]. To further strengthen generalizability, we now include a 10-stage branching workflow, where agents are randomly selected at each stage from partially overlapping prompt groups. All configurations other than the branching pattern follow those in the paper.\n\n**Speedups over SGLang for branched workflows with Qwen2.5-32B on an H100**\n|Configuration|SGLang|HiCache|KVFlow|\n|-|-|-|-|\n|4096/32/32|1.000±0.016|1.099±0.024|**1.302±0.058**|\n|4096/256/256|1.000±0.012|1.001±0.017|**1.052±0.012**|\n|8192/32/32|1.000±0.001|0.754±0.014|**1.459±0.025**|\n|8192/256/256|1.000±0.003|0.924±0.008|**1.122±0.007**|\n|512/20-task|1.000±0.016|0.885±0.044|**1.225±0.021**|\n|1024/10-task|1.000±0.010|1.014±0.035|**1.140±0.007**|\n\nRegarding baselines, in addition to SGLang and its hierarchical cache extension, we now include the vLLM baseline, which organizes KV cache into blocks and adopts LRU-based block eviction policy. The results are shown in the following table. vLLM is slightly faster than SGLang in our experiments, but KVFlow still consistently outperforms both. Since KVFlow is built on top of SGLang with HiCache, comparing against SGLang better reflects the impact of our proposed optimizations. Our approach is general and can also be applied to vLLM or other LLM serving frameworks.\n\n**Speedups over SGLang for sequential workflows with Qwen2.5-32B on an H100**\n|Configuration|SGLang|vLLM|HiCache|KVFlow|\n|-|-|-|-|-|\n|4096/32/32|1.000±0.002|1.041±0.003|1.022±0.074|**1.389±0.045**|\n|4096/256/256|1.000±0.004|1.031±0.005|0.996±0.022|**1.043±0.010**|\n|8192/32/32|1.000±0.043|1.057±0.045|0.738±0.030|**1.198±0.030**|\n|8192/256/256|1.000±0.013|1.041±0.014|0.928±0.006|**1.138±0.003**|\n|512/20-task|1.000±0.029|1.091±0.033|0.950±0.048|**1.256±0.010**|\n|1024/10-task|1.000±0.020|1.046±0.021|0.983±0.043|**1.132±0.015**|\n\n> [1] Hong, Sirui, et al. \"MetaGPT: Meta Programming for A Multi-Agent Collaborative Framework.\"\\\n> [2] Qian, Chen, et al. \"ChatDev: Communicative Agents for Software Development.\"\\\n> [3] Wang, Yiying, et al. \"PEER: Expertizing domain-specific tasks with a multi-agent framework and tuning methods.\"\n\n### (W2) Title Clarification\n\nWe apologize for the confusion. The term \"KVFlow\" in our title means **KV** cache management for agentic work**Flow**. We are open to refining the title to improve clarity.\n\n### (Q1) Overhead Analysis\n\nThanks for the question. The runtime CPU overhead introduced by KVFlow is minimal. First, even when the entire graph is very large, we only need to consider agents within a certain step (e.g., 5) of current activated agents. Second, SGLang’s zero-overhead batch scheduler [4] allows the scheduling logic on CPU to be overlapped with GPU execution, effectively hiding the associated runtime CPU cost. We observe negligible CPU overhead in our measurements. We will add this discussion in Sec 4 in the next version.\n\n> [4] SGLang v0.4: Zero-Overhead Batch Scheduler, Cache-Aware Load Balancer, Faster Structured Outputs.\n\n### (Q2) Dynamic Workflow\n\nThanks for the question. KVFlow is designed for workflows where the structure is known or can be inferred at runtime. KVFlow can handle dynamic workflows as long as the execution order of agents within a certain future window can be predicted at the current step. In such cases, KVFlow adaptively updates the steps-to-execution values at runtime and continues to make informed eviction and prefetching decisions. However, in cases where the workflow is highly unpredictable, KVFlow cannot anticipate future agent calls. In such scenarios, it falls back to HiCache’s LRU eviction and reactive cache loading. We will discuss this limitation in a dedicated section in our next version."}]

Author rebuttal:

Meta-review: This paper proposes KVFlow, a workflow-aware KV cache management framework for accelerating LLM-based multi-agent workflows. It combines an agent step graph–based eviction strategy with a fully overlapped prefetching mechanism to significantly reduce cache misses and recomputation.

Before rebuttal, the main concerns centered on evaluation scope and generalizability (Reviewer Gzk9, Reviewer Y4Pj, Reviewer ojFE, Reviewer KtZF), methodological clarity (Reviewer Gzk9, Reviewer Y4Pj, Reviewer ojFE), and runtime overhead/handling of dynamic workflows (Reviewer Y4Pj, Reviewer ojFE, Reviewer KtZF).  

Most reviewers acknowledge that rebuttal effectively addressed these issues by adding dynamic workflow experiments, clarifying LRU/vLLM baselines, and explaining overhead and fallback behavior.

Reviewers agreed that the paper identifies an important problem in LLM (Reviewer Gzk9, Reviewer KtZF), the proposed Agent Step Graph and workflow-aware eviction represent strong conceptual and practical contributions (Reviewer Gzk9, Reviewer Y4Pj, Reviewer ojFE), and the experimental results demonstrate consistent and significant speedups across scenarios with clear practical impact (Reviewer Y4Pj, Reviewer ojFE, Reviewer KtZF). All Reviewers vote positive to this paper Therefore, the AC recommend acceptance.
Decision: Accept (poster)