KVCOMM: Online Cross-context KV-cache Communication for Efficient LLM-based Multi-agent Systems
Authors: Hancheng Ye
Zhengqi Gao
Mingyuan Ma
Qinsi Wang
Yuzhe Fu
Ming-Yu Chung
Yueqian Lin
Zhijian Liu
Jianyi Zhang
Danyang Zhuo
Yiran Chen
Venue: NeurIPS 2025 poster ()
Keywords: LLM-based Multi-agent System; KV-cache reuse; Multi-context Agent Communication
Primary area: deep_learning

Abstract: Multi-agent large language model (LLM) systems are increasingly adopted for complex language processing tasks that require communication and coordination among agents. However, these systems often suffer substantial overhead from repeated reprocessing of overlapping contexts across agents. In typical pipelines, once an agent receives a message from its predecessor, the full context-including prior turns-must be reprocessed from scratch, leading to inefficient processing. While key-value (KV) caching is an effective solution for avoiding redundant computation in single-agent settings where prefixes remain unchanged, it cannot be directly reused in multi-agent scenarios due to diverging prefixes introduced by agent-specific context extensions. We identify that the core challenge lies in the offset variance of KV-caches across agents. To address this, we propose **KVCOMM**, a training-free framework that enables efficient prefilling in multi-agent inference by reusing KV-caches and aligning cache offsets of overlapping contexts under diverse prefix contexts. KVCOMM estimates and adjusts KV-caches for shared content by referencing a pool of cached examples—termed *anchors*—that store observed cache deviations under varying prefixes. The anchor pool is maintained and updated online, allowing dynamic adaptation to distinct user requests and context structures. KVCOMM achieves over 70% reuse rate across diverse multi- agent workloads, including retrieval-augmented generation, math reasoning, and collaborative coding tasks, all without quality degradation. Particularly, when each fully-connected agent receives 1K input tokens with 512 prefix tokens and 512 output tokens under a five-agent setting, KVCOMM achieves up to 7.8× speedup compared to the standard prefill pipeline, reducing TTFT from ∼430ms to ∼55ms. Code is available at https://github.com/FastMAS/KVCOMM.

KVCOMM: Online Cross-context KV-cache Communication for Efficient LLM-based Multi-agent Systems
Hancheng Ye', Zhengqi Gao?, Mingyuan Ma', Qinsi Wang', Yuzhe Fu', Ming-Yo Yu Chung', Yueqian Lin', Zhijian Liu", Jianyi Zhang', Danyang Zhuo', Yiran Chen
Duke University, MIT, SNVIDIA hancheng , yww.uke.eeeeccooo
Abstract
Multi-agent large language model (LLM) systems increasingly adopted for complex language processing tasks that require communication and coordination among agents. However, these systems often suffer substantial overhead from repeated reprocessing of overlapping contexts across agents, In typical pipelines, once agent receives message from its predecessor, the full context-including prior turns-must be reprocessed from scratch, leading to inefficient processing. While key-value (KV) caching an effective solution for avoiding redundant computation in single-agent settings where prefixes remain unchanged, it cannot directly reused in multi-agent scenarios due to diverging prefixes introduced by agent-specific context extensions. We identify that the core challenge lies in the offset variance of KV-caches across agents. To address this, we propose K KVCOMM, training- free framework that enables efficient prefilling in multi-agent inference by reusing KV-caches and aligning cache offsets of overlapping contexts under diverse prefix contexts. KVCOMM estimates and adjusts KV-caches for shared content by referencing a pool of cached examples-termer anchors-that store observed cache deviations under varying prefixes. The anchor pool is maintained and updated online, allowing dynamic adaptation to distinct user requests and context structures. KVCOMM achieves over 70% reuse rate across diverse multi- generation, math reasoning, and
agent workloads, including eetrieval-augment collaborative coding tasks, all without quality degradation. Particularly, when each fully-connected agent receives 1K input tokens with 512 prefix tokens and 512 output tokens under five-agent setting, KVCOMM achieves up 7.8 speedup compared to the standard prefill pipeline, reducing TTFT from 44430ms to ~555s. Code is available at https: // thub. com/ Fas MAS KVCOMM.
1 Introduction
Large Language Models (LLMs) such as GPT-4o [1] and Llama-3 [11] have triggered surge of interest in collaborative multi-agent systems, where several specialized agents exchange messages to collaboratively solve complex tasks such retrieval-augmented question answering, mathematical reasoning, and tool-augmented program synthesis [63, 9, 51, 1, 38, 49, 60, 54, 18). In these settings, every message processed by LLM agents must first go through the prefill stage prior decoding, during which the model encodes the full conversation history and constructs kay-value (KV) caches. Although multiple agents often share overlapping context (e.g., retrieved passages or peer outputs), they always redundantly recompute K V-caches for all input tokens, resulting in significant inefficiency of prefilling computation [27, 56, 30], which is defined as multi-context redundancy issue in the
39th Conference on Neural Information Processing Systems (NcurIPS 2025).

---

2.4
16
DKL
Key offset
9
Key offset
Value offset
6
1.2
8-
3-
0.6
4-
10
15
20
25
30
0
5
10
15
20
25
30
Layer
Layer
(a)
(b)
Figure I: (a) Mean KV-cache offset (measured by & norm) from the base-context cache the same token across ten distinct prefixes. Shaded regions indicate standard deviation across these prefixes. (b) KV-cache offsct comparison between two embedding- prefixed with alignment, respectively. "A w/o rot" and "AK2 w/o rot" refer Key offsets without alignment. "AVI" and "AV2" denote Value offsets two tokens.
tokens from the base-conter caseectntett caches when both tokens are context. "A "AK1 w/ rot" and "AK2 w/ rot" represent Key offsets of two tokens with position
multi-agent system. For example, single 8B Llama needs 4430 ms to prefill 3K-token prompt on one H100 GPU. each of M agents receives messages from all of its peers, the total prefilling complexity of these repeated computations scales O(M)), posing inefficiency in the utilization of computation and major challenge for real-time multi-agent collaboration.
Recent works attempt to reduce prefilling overhead primarily through four techniques: prompt-level reuse [10], selective recomputation [58, 27, 56), cache compression [28], kernel-level optimiza- tions [67, 46]. While effective in their target scenarios, these methods share fixed acceleration policy crafted for a particular workload. However, our empirical study reveals that the same shared text can incur vastly different KV deviations once it is preceded by different prefix contexts, e.g., system messages with different roles or upstream agents with different output lengths (see Figure la). When the acceleration policy fails to model such an offset-variand problem, cache reuse becomes misaligned, causing either large accuracy drops or a fallback to full recomputation that erodes the speed benefits.
This observation motivates prompt-adaptive paradigm that (i) dynamically determines how to reuse K V-caches at runtime for each incoming prompt given diverse prefix contexts, and (ii) requires no additional training, profiling, or model modifications, allowing easy adoption on various tasks and agent workloads. To our knowledge, no existing method simultaneously satisfies both desiderata.
In this paper, we introduce training-free online KV-cache communication (KVCOMM), a drop-in framework that accelerates multi-agent systems through shared-context reuse with adaptive KV offsetting. The key insight to treat every reuse attempt as an approximate translation problem. where the KV-cache of overlapping text becomes reusable for new prefix once the positional shift and cache offsets from similar samples are identified. As illustrated Figure 1b, the KV-cache offsets of two similar tokens prefixed with two different prompts present similar distributions across layers, where the deviation of the rotated Key cache significantly smaller than the unrotated one. Therefore, KVCOMM proposes anchor pool of previously shared samples along with their measured offsets under diverse prefixes. At inference time, the framework first locates the nearest anchor(s) for the requested segment via token similarity (Anchor Matching) and then predicts the offset by interpolating their stored deviations, avoiding a replay through the prefilling stage (Offset Approximation). For the Key cache update, the cache will first be encoded to the correct position and then biased by the estimated offset, while for the Value cache update, since it has no positional information, the offset is directly added to the cache. Mcanwhile, the anchor pool is updated online to catch up with the new input distribution. That is, once the cache of an input segment is predicted as unshareable, it will be marked as anchor and measure the cache offset under each prefix to extend the reusing range for the subsequent input samples. For the least matched anchors, they would be periodically freed up save memory consumption and computation cost.
In summary, K KCOMM represents substantial advancement in adaptively efficient KV-cache sharing in the LLM-based multi-agent system without training or recomputation, offering a practical path toward efficient agent communication. The main contribution threefold:
We identify the multi-contert redundancy as key challenge for efficient prefilling in the multi-agent scenario, and characterize the offset-variant problem that limits traditional KV sharing such a setting, which to our knowledge, has not been covered by prior work.
We propose K KVCOMM, the first training-free, prompt-aduptive mache-sharing framework for efficient prefilling of multi-agent systems, requiring only a few anchors to effectively

---

RV or Prets Connatt
KV Shandd
KV Phett
for Anent of Shared Agent
Ko Cittet tihaedd Toxt fro Agere Request
Asgetalmales so Bhases Thxt from Agont Reeuaoot
in Agert
Requert I: Tly prace 1x
Profis Cortoot
Text
Agentt
Sond Vewsage
Prefux Contedd
Text 1 from Agent
Agont
Requent- Find a wiy slows Agent
Poefic Conteot
The
Message
Prefix Contant
Test 2 from Agant
KVCCOMM Rlow
Pietil Plon
Decodlnng Plone
Original: No Cache Rouse
Prior: Cadhe Rouse with Selective Recomputaton
KYCOMM: Context awaar
Cache Offeetting
Figure 2: Comparisons with existing KV-cache reuse methods. (Iefft) The original no-cache-reuse haseline method densely prefills the tokens of all requests. (Middle) Selective recomputation methods [58, 27J select the most critical part of KV-cache for recomputation and reuse the remaining cache of each request. (Right) KVCOMM reuses all KV-eaches of the shared context introduces context-aware cache offsets to align with different prefix contexts, where the context-awarc offsct refers to the KV-eache deviation induced by the changed prefix context. Such an offset is approximated by the cround-truth ones of previous similar requests. After approximation, the model runner directly starts decoding without prefill.
approximate KV-cache offsets across different prefix. In KVCOMM, an efficient KV-cache management system is designed support fast anchor lookup.
Extensive experiments on three representative tasks with different models, including re- trieval augmented generation (RAG), math reasoning, and programming, demonstrate that KVCOMM can achieve 6.7 average prefill speed-up where each agent is deployed by NVIDIA H100 GPU. Mcanwhile, the reuse rate reaches
Llama-3. 1-8B--B-- 3B-instruct [11] 95% across 1,319 samples in a four-agent system for GSM8K [7], KVCOMM achieves comparable performance to the original workload (less than 2.5% accuracy drop).
2 Related Work
2.1 LLM-Based Multi-Agent Systems
The idea of distributing a complex task across multiple specialized LLM agents has rapidly progressed, from early frameworks such as AutoGPT [35] to mature tool augl-angeented systems for retrieval, coding, and robotics [17, 26, 39, 63, 9, 48, 22, 50, 51, 62, 55, 3, 31, 52, 24, 15, 42, 19, 40, 23]. Recent studies propose curriculum fine-tuning promote role specialization [66], graph-structured message routing (57, 69], and hierarchical decision making [32]. Yet practically, each agent still performs a full prefill pass for every turn, recomputing the KV tensors over large shared contexts. As agent graphs grow wider or deeper, the prefill complexity of these repeated computations scales quadratically, posing inefficiency computation utilization. Addressing the prefill bottleneck is thus a prerequisite for scaling multi-agent LLM applications to real-time settings.
2.2 KV -cache Acceleration and Reuse
KV-cache Sharing Scenario. Prior research has identified three principal patterns for reusing the KV-cache in transformers. (i) Multi-request sharing exploits identical prefixes across requests from different users; by copying the KV-cache of the shared prefix, servers can bypass most of the prefill compute when only the tail differs. (ii) Multi-turn sharing keeps the cache alive throughout the turns of a single conversation, thus avoiding recomputation of the history. (iii) Multi-context sharing handles inputs whose overlapping segment appears different contexts, which aims to filter out the impact of the prefixed prompt in the KV-cache and to combine current context information into the reused K KV-ccche. Most these techniques assume that every agent runs the same model vanilla RoPE-based decoder-so that a cached key can be translated by a simple rotation without re-encoding [37, 47]. DroidSpeak [27] extends the sharing from the base model to the fine-tuned one by profiling which layers remain shareable. Current industrial serving
archieecture-typicee
3

---

Agent
KV-cache Communication Module
Matching
Offset Approximation
Anchor Prediction
[Pus Phul Ba.
Placaholder KV
Po.
:: Pool
Cape Pool
Eash
Andhor
h, Lirrnichedd
Ah. Ph.xx]
No
All placeholders
Yes
Query
Hit
Akk. Av) W
Av)
Prefilling
Dense
Cache Update
PM So. 2
cb u
[Alk
Langh Uncover
OR
AND Langth Covered
Pa.:: Pr.
(Ak.
Decoding
Response Cache
[PH, Suepoton
Anchor Prodiction of Respanse KV
Agent 2
Agent
Lellia -91
Legend
Prefix KV
Prefix KV
Nov
Operted Jedamnll lepatt
2% susap
Quarted (ragprae) obeer
Placeholder KV
Placeholder KV
Example
69.11 Wa. nl Mo.ai (1
tba n (Pa, al (9g.x (5..c)
Edirrledd oftta (ookmllnn
Pracahidder
All placeholders
?
All placeholders
Forearet sty of (nsporee)
Ageni
(N) Dense
+
(Y) Sharing
(Y) Sharing
(N)
Rouse
PaL
Ongined Agent
Sheradoe placo nolde
Figure 3: Overview the KVCOMM framework in ageet scenario. Initially, each agent precomputes and stores the KV-ccche of pretix segments from its prompt template for future reuse. At runtime, upon receiving a new request, agents check placeholder shareability and query matched anchors. Matched anchors help approximate KV-cache offsets for placeholders and subsequent prefixes through embedding- The matching criteria consider length compatibility and emhedding proximity. The updated K VV-ecches are concatenated efficient decoding. After decoding, the KV-cache Communication module assesses newly generated caches for potential sharing with other agents based on the established matching rules.
interpolation.
stacks [21, 68, 5] expose the same constraint that architectural identity a prerequisite for cache reuse.
Existing methods on KV-cache acceleration primarily explore four paradigms. (1) Prompt-Level Reuse [10]. PromptCache [10] introduces Prompt Markup Language to explicitly define reusable text segments whose KV-caches are precomputed offline and directly fetched at inference, eliminating recomputation but restricted to fixed prompt structures. (2) Selective Recomputation [58, 27, 56].
CacheBlend [58] dynamically identifies and updates tokens exhibiting high variance in KV-caches. DroidSpeak [27] leverages profiling to identify critical attention layers whose KV-caches must be refreshed to maintain accuracy. KVLink [56] further extends it by fine-tuning special tokens and adjusting positional embeddings, enabling KV reuse across multiple document contexts. (3) Cache Compression [28]. CacheGen [28] compresses KV-caches into adaptive bit-streams based on available bandwidth; however, the entire token sequence still undergoes compression computations, limiting latency improvements. (4) Kernel-Level Optimizations [67, 46). PrePacking [67] employs bin-packing strategy to batch variable-length prompts into unified sequences. LoongServe [46] designs Elastic Sequence Parallelism to dynamically manage parallelism strategies and overlap cache migration with decoding steps to enhance GPU utilization. Figure 2 compares the main difference between KVCOMM and existing K KV-cache sharing methods. Generally, KVCOMM explores completely novel paradigm that can reuse all shareable KV-caches regardless of diverse prefix contexts, and align them by leveraging the context-awant cache offsets observed in previous samples.
3 Proposed Approach
3.1 Preliminaries
Large Language Models and KV-cache. Let [h...... h² denote list of token embedding sequences, with h' E N representing the input token embedding of the I-th transformer layer, where N is the number of tokens and 1) the feature dimension. For h (where n = 1,2, ........ NN, it is projected by the (-th transformer layer to Query. Key, and Value vectors for the subsequent attention computation using: q = R2 W W/hh', k, R. w h v Wi'h', where q k,, v e Rdd denotes the Q, K, V values for the n-th input token, W Wt K: Wt refer the corresponding projection weight matrices, and R,, is the position embedding at position n, such as rotary position embedding (RoPE) [37]. During autoregressive decoding, the model repeatedly attends to all past positions, and stores every (k/, v pair in GPU memory, known as KV-cache. Therefore, prefilling a prompt of N tokens costs O(Ndd) multiply-adds per layer, dominating inference latency for long contexts. Since RoPE applies the fixed rotation matrix to both Key and Query at each position, cached

---

keys remain valid across subsequent steps with no further arithmetic modification, making K KV-cache reuse the primary source of speed-ups in subsequent prefilling and decoding.
Directed-Graph Multi-agent Systems. Following [57, 69, 65], we model a multi-agent system directed graph 9 (M,E) E) whose nodes m E M agents and edges denote one-way message passing from the - agent to the m--th agent. At interaction step t. the m-th agent composes an input prompt Sin in the template consisting of (i) fixed prefix segments shared across all turns, and (ii) placeholder segments filled at runtime with user queries, tool results, or upstream agent outputs, as formulated as follows, where P(m.0) is usually a role-specific system prompt, P(m,,)) is the subsequent prefix segment of the i-th placeholder @ (8))
(m, ma) &
(m,,)
(4) ((m.0)) (m, P(m,,), (( P(m..)...... (m..)) P(mms)]
(1)
Our work targets a distinct yet practical setting: directed mulli-agent graph which each node an identical RoPE-based LLM checkpoint instantiated with a role-specific system template. Since agents differ in the length of both prefixes and incoming messages, none of the existing static policies can predict the correct positional and contextual shifts; misalignment either forces full recomputation or yields steep accuracy loss. We therefore develop a training- prompt compt-adaative cache-sharing mechanism, termed K KVCOMM which estimates the true offset the fly and maintains online anchor pool to accommodate rapidly changing interaction patterns, reducing prefilling latency without sacrificing task performance. The overall workload of K V COMM is illustrated in Figure 3, which proceeds as follows.
0. Initialization Before any user requests, all agents precompute and store the KV-caches for all prefix segments defined in their prompt templates.
1. Placeholder Readiness When a request arrives, each agent checks whether all placeholders' base KV-ccches are available. Missing bases are precomputed in parallel. Newly generated placeholder KV-caches are then sent to the anchor prediction module to search the anchor pool for similar samples and enable reuse.
2. Reuse or Fallback Once all placeholder KV-caches are ready, the agent determines whether reusable KV-cache deviations exist for each placeholder. If none found, standard dense pre- filling is used. For placeholders without reusable deviations, the agent computes the difference between their actual and base KV-caches and stores this deviation in the anchor pool to expand anchor coverage.
3. Offset Approximation If all placeholders have reusable deviations, the agent fetches the matched anchors, estimates the KV-cache deviations via Eq. (6) and Eq. (7), and updates the placeholder and prefix KV-caches in parallel.
4. Decoding The agent concatenates the updated placeholder and prefix KV-caches and initiates response decoding.
5. Anchor Update After decoding, the produced KV-cache is passed through the anchor prediction module. If a similar anchor exists, the cache is stored in shared memory for future retrieval. The agent then waits for the next request.
6. Fallback Storage If no similar anchor exists, the response KV-cache is stored in the anchor pool so dependent agents can subsequently fill deviations under their respective contexts; the agent then awaits the next request.
All inter-agent interactions occur through the KV-cache Communication Module. When multiple agents share the same user text but use different agent-specific prefixes, we avoid re-running prefilling by treating the KV-cache of the shared text under a new prefix as a context- dependet offset from its base KV-cache. We estimate this offset by interpolating from small set of anchor examples, aligning Key positions via RoPE dlerootation/r-- wewto////errrrrtonn. adding the estimated Key/Value offsets, then concatenating the adjusted segments and decoding.
Positional Alignment is Indispensable. Before analyzing two arbitrary tokens' cache correlation, we should first solve the position mismatch induced by RoPE, If a token at position n in one prompt but at + A in another, the raw keys differ by an orthogonal rotation R.. whose difference can be orders of magnitude larger than the contextual deviation we care about, as demonstrated in Figure 1b. Hence, KVCMMM always de-rotates the stored key by R_A before measuring similarity between
5

---

40
Kraa
Keat
Ka
20
Value Distance
Key Distance
30-
Spearman
0.75
10
0.50
10-
Pticr
Layer
Layer
(a)
(b)
4iey Distance
Avalue Distance
22
Alrer
0.75
Spearman
15-
0.50
0.25
Pux
Pwr
5
10
15 Layer
20
25
30
5 10
20
25
30
Layer
(c)
(d)
Figure 4: (a)(b) KV-cache proximity experiments for token pairs sharing identical prefixes: (a) shows the KV-cache distances (measured by & norm) across layers of the token pairs, which grouped into "near", "mid". and "far" by embedding distance between the two tokens in the token pairs. (b) shows the Spearman [36] correlation between embedding distances and VV-cache proximity across layers. (c)(d) KV-cache offset proximity experiments for token pairs prefixed by two distinct contexts: (c) shows the layer-wise KV-cache offset distances between tokens grouped by embedding proximity, (d) shows the Spearman correlation between embedding distance and VV-ccche offset proximity. Experimental details are shown in Appendix 6.3.
Key cache offsets of two tokens under the same context change. The following analysis assumes this alignment as completed so that the remaining deviations stem mainly from token identity and context.
3.2 Token-level Key/Value Similarity for KV Reuse
Motivation. KVCOMM hinges on the empirical observation that per-token KV vectors remain remarkably similar across distinct conversational contexts long as the model parameters are shared, Intuitively, the residual pathway in every Transformer block keeps copy of the input representation and adds the attention (Attn) and feed-forward (FFN) refinements:
h h² FFN' (h,, Attn' (h')m),
where FFN' and Attn' refer to the FFN and Attn modules in the 1-th layer. Hence, the identity information carried by the original embedding h. never overwritten but accumulates across layers, suppressing the variation of the projected keys/values. Below make this insight precise and quantify how far two distinct tokens are when prefixed with the same contexts.
Proposition (KV-Distance Between Different Tokens). Let K2 and k2 be the key vectors of two different tokens at position n at layer that are prefixed with the same token sequence. Assume Attn' a' Lipischhi FFN' is 3 -Lpschiizz [20]. Define af - 3 + nat). Then
- S CRO* (1+ 8n
6n max Sn h 1 -hill-
(3)
where CR > related to RoPE and Ck > O is related to 1-th layer projection key matrix W Similarly, the inequality also holds for the value caches of the two tokens.
The proof is deferred to Appendix 6.2.2. It can be observed that Eq. (3) bounds the KV distance by the embedding gap scaled through layers, so tokens that start closer embedding space have tighter bounds and greater cache-reuse potential. Figure 4a and 4b empirically demonstrate this insight, where the KV-caches of "near" token pairs are consistently closer to each other than the other two groups, and the K KV-cache proximity is highly correlated to the token embedding distance.
We now examine this relation multi-agent settings, where two similar tokens face different prefixes. Proposition 2 (Deviation Proximity With Different Prefixes), Let k., and k,_ be the key vectors of two different tokens at position at layer that are prefixed with prompt Pa. Similarly, k. and ke are the key vectors of the two tokens at position nh at layer I that are prefixed with prompt Ps. We denote the key cache deviation of each token layer by A = k, -K,,,'' A =kk, k. Under the same Lipschitz assumptions as Proposition I and after positional alignment.
A A S 2 Ck [) + a² yōn.;
8n, maxt
h 6/ll
(4)
Similarly, the inequality also holds for the value caches of the two tokens.

---

Proof is in Appendix 6.2.2. Eq. (4) shows that tighter embedding gaps again yield smaller deviation bounds, supporting cross-context offset reuse. Figs. 4c and 4d validate this under the same setup as Figure 4a, with each token evaluated under two distinct prefixes, where the K KV-cache offsets of "near" token pairs are also consistently closer to each other than the other two groups, and the K KV-cache offset proximity is also highly correlated to the token embedding distance.
The above propositions motivate an anchor-based KV-sharing scheme that stores representative offsets as reusable anchors for future agent interactions to reduce all redundant prefilling latency of agents.
3.3 Anchor-based K KV-ccche Communication
We now introduce our anchor-based communication framework to unify the KV-cache sharing mechanism in multi-agent systems. During setup, each agent extracts placeholder information from its role-specific prompt into a structured dictionary, indicating token positions. The naming conventions for placeholders are detailed in Appendix 6.2.3. Each placeholder initializes an individual anchor pool upon receiving the first sample.
At runtime, subsequent input samples trigger a reuse check across agent placeholders. Agents reuse KV-caches directly from corresponding anchor pools if reuse conditions are satisfied, significantly speeding up inference by skipping redundant prefilling. Otherwise, agents revert to standard prefilling, updating anchor pools with newly computed KV-caches enrich future reuse opportunities.
Anchor Pool Design. An anchor pool stores key information for each placeholder sample: (1) the base KV-cache, computed independently without external contexts: (2) offsets between the base and actual KV-caches within each agent's 's context; and (3) offsets of subsequent neighboring prefix segment's KV-cache. Thus, an anchor is represented as (ph _ph_name: base KV, agent_ _ placeho I der offset, agent_id crucial due to position-dependent KV-cchhe shifts introduced by the placeholder's context changes, as highlighted by the sink attention mechanism [53], which emphasizes local contextual dependencies.
pref ii offset). J. Neighboring prefix offsets are
Anchor Prediction. Determining whether newly-generated KV-eaches, e.g., responses, user inputs, etc., could be shared or treated as new anchors involves evaluating the ambedding-based proximity [61, 29, 64, 59] and token length compatibility with existing anchors. The prediction criterion is designed as follows:
P Panchor (Ø) (Co m A Cx) U (HoJA > 7og Apl), Haja EVeA, Wir-r log W (5)
where A refers to the anchor pool that the placeholder & belongs to, &b denotes an anchor in A, L. represents the sequence length of the sample *, HolA measures entropy of the babbedding-distanne based weights among longer anchors in the anchor pool, W2-o = so sofftaa((( (a -lohllhhhhhh hhlp-hhlll I)) tb E Agg [Ag] refers the number of anchors longer than in [A),, and y a threshold to determine how far shareable sample could be away from the anchors of A in the embedding space. Intuitively, anchors closer in embedding space yield more reliable offset predictions (validated by Prop. 2 and Figure 4c), and length compatibility ensures correct positional alignment (see Figure lb).
Anchor Update. When KV-cache sharing criteria are unmet, the wwwly-generated cache becomes a new anchor's base KV-cache. Agents relying on this unshareable placeholder revert to regular prefilling, providing agent-specific offsets for both the placeholder and its neighboring prefix segments to populate the new anchor entry. Due to GPU memory constraints, we implement an adaptive anchor pruning strategy: once anchor pools reach a predefined size V. the least frequently accessed anchor among the carliest-added entries is discarded, maintaining a relevant and efficient anchor repository.
3.4 Anchor-based Cache Update
When placeholders in an agent's 's prompt are predicted shareable, we efficiently update their K KV-caches via anchor matching and offset approximation.
Anchor Matching- We retrieve reliable anchors identified during prediction, performing parallel reads due to independent addressing, leading to negligible overhead compared to traditional prefilling,
Offset Approximation. Using matched anchors, we approximate placeholders' KV-caches within agent-specific contexts. Neighboring prefix segments are updated similarly based the placcholder
7

---

Table 1: Performance of three machhemanggement strategies under different numbers of collaborating agents, Accuracy reported for MMLU and GSM8K (Llama-3.1-8B) -nSt-ttccc) Pass is reported for HvaanEval (Qwen-2.5-coder-))
Higher is better. In addition, the Reuse Rate is reported for both VVCOMM and CacheBlend. Note that the Reuse Rate for CacheBlend is defined as the proportion of tokens reusing KV-caches in whole token sequences, while the Reuse Rate of KVCOMM is defined as the frequency of agents reusing all KV-caches in the whole serving procedure.
Agents
Dataset
Metric
Method
2
3
4
5
Original
47.1
66.7
68.0
69.9
Accuracy (%)
CacheBlend
65.4
65.4
65.4
67.3
MMLU
КУСOмM
64.7
68.6
68.0
69.9
Original
Reuse Rate (%)
CacheBlend
80
80
80
80
KVCOMM
74.5
69.9
70.1
67.6
Original
81.1
82.4
82.1
81.7
Accuracy (%)
CacheBlend
82.0
75.1
65.1
57.1
GSM8K
КAСOмM
81.5
81.7
80.6
79.6
Original
Reuse Rate (%)
CacheBlend
80
80
80
KVCOMM
79.6
77.0
71.0
Original
83.9
85.1
Pass@1 (%)
CacheBlend
31.1
21.1
30.4
32.9
HumanEval
KVCOMM
81.4
83.2
83.2
83.2
Original
0
Reuse (%)
CacheBlend
80
80
80
KVCOMM
87.6
84.7
81.1
77.8
sample's 's embedding proximity, Formally, the KV-cache of the i-th placeholder in the m-th agent is approximated as follows:
(k/V)).....)
(m..)) = (k/v) + EEE W Wo(m.))
A A(k/v) (k//))mm,))) (m,,)))
(6)
where (k/v)ocm. (mm.. refers to the approximated K/V cache for the placeholder (m..)) (k/v) (rm..))))) is the
base K/V cache for the placeholder ((m.i)- W () the softmax mapping of - h
across the anchor dimension. (kk/v) ( is the placeholder d's cache offsets in the m-th agent for the anchor th. Prefix segment updates follow an analogous process:
(k/V) P(m..) = (k/v P(m.)) (k/v)p(m.)) + EvEAo Welem.) 7 - A(k/v) (k/v))mm))) (m,))))
where (k/v)pp.. refers to the approximated K/V cache for the prefix segment P(m. (k/v) Pom.. is the base K/V cache for the prefix segment P(m.i)- A(k/v)mm,)) is the corresponding prefix segment's cache offset the m-th agent for the anchor tb.
After approximation, updated caches are concatenated and directly fed into decoding, substantially reducing prefilling latency via parallel processing. The overall algorithm listed in Appendix 6.2.5.
4
Experiments
4.1 Experimental Setup
Multi-agent System. Following GPTSwarm [69] and AgentPrune [65], construct fully- connected multi-agent system with established techniques including few-shot prompting [2], chain- of-thought [45], function calling [33], and structured outputs [34]. To precisely analyze KV-cache behaviors, we deploy open-source models using HuggingFace's framework rather than closed-source APIs used by AgentPrune. Specifically, retrieval-ause -uggneted generation (RAG) and math reasoning, and Qwen- for programming tasks. We evaluate performance across scenarios ranging from two five agents.
employ Llama-3.1 1--1--BBInntruee [11] (Llama-3.1) 1) for [16]
Benchmark Datasets. We assess RAG performance using MMLU [13], math reasoning with GSM8K [7], and programming capability via HumanEval [4].

---

Table 2: Per-agent TTFT breakdown and speedup. (Prefix token length per agent: 512; Output token
Table 3: Mean TTFT speedup using Llama-3.1
(#Agents = 3)
length: 512; Model: Llama-3.1; 1; ; #Agents = 5)
ln (Prefix sequence)
Out_len
TIPT (ms)
Agent 1
Agent 2
Agent 3
Agent
Agent
64
128
256
512
1024
Original
125.8
192.4
258.3
330.9
478.6
128
2.24x
2.31x
2.56x
3,07x
4,45%
KYCOMM
5.5
Token Decode
21.4
21.2
21.2
21.3
21.1
256
2.50x
2.51x
2.83x
3.44x
4.75%
Others
86.6
9.1
10.7
13.5
16.2
512
3.05x
3.18x
3.49x
4.09x
5.34x
Speedup
1.11x
5.06x
6.14x
6.85%
7.82%
1024
4.40x
4.48x
4.75x
5.35x
6.72x
Comparison Baselines. To our knowledge, KVCOMM is the first method enabling comprehensive KV-cache sharing tailored for open-source multi-agent frameworks. Hence, we compare primarily against CacheBlend [58]. which selectively recomputes sensitive tokens' caches for partial reuse. Given CacheBlend's tight integration with vLLM [21], we faithfully replicate its selective recomputa- tion strategy within our experimental setup, consistently recomputing the top-20% tokens exhibiting the largest KV deviations, ensuring fair baseline alignment.
Evaluation Metrics.
We report Accuracy scores MMLU and GSM8K, and Pass@1 for coding tasks. Efficiency metrics include Reuse Rate (meaning the proportion of agents employing the cache reuse scheme), individual agent To-r-To-First- Token (TTFT), and average TTFT across agents.
Implementation Details. Experiments are exccuted on a single NVIDIA H100 GPU. The maximum generation length is uniformly set to 512 tokens, with hyperparameters selected as 7 = 0.3 and anchor pool size V = 20. Further implementation specifics are detailed in Appendix 6.3.
4.2 Main Results
Table 1 compares our KVCOMM approach with the Original (no cache reuse) and CacheBlend [58] strategies across multiple agent configurations. Although our agent prompts were initially opti- mized for closed-source models, causing some performance drops with open-source deployments, KVCOMM still maintains or improves upon baseline accuracy.
On MMLU, KVCOMM achieves competitive accuracy (64.7%-69.99%) 99)), consistently outperforming or matching CacheBlend and closely tracking the original baseline. This indicates robust cross-context KV-cache alignment by K KVCOMM, CacheBlend fluctuates significantly with increasing agents. The reason why the baseline method performs poorly on the MMLU benchmark under the two-agent setting is that the first agent is designed as the knowledgeable expert to produce the related key words about the user question, and the second agent is designed as the Final Refer to analyze the predecessor agents' output and give the answer based on the previous agent's 's output, where the failure cases mainly occur when the second agent only output the answer without analysis. The prompt of each agent can be found in Appendix 6.3.2.
For GSM8K math reasoning. KVCOMM's accuracy remains stable, only declining by 1.9% from two to five agents, maintaining within =2% of the original baseline. In contrast, CacheBlend's accuracy drops dramatically from 82.0% to 57.1 %, highlighting the necessity for precise KV-cache reuse in numerical tasks.
In the HumanEval coding benchmark, KVCOMM delivers stable Pass @ scores 11.4%%88%%%% significantly surpassing CacheBlend by average margin of 53%. This underscores KVCOMM'S 's ability to preserve task- critical dependencies essential for programming tasks. The severe performance degradation of CacheBlend on Humaneval attributes to the diverse syntax separators involved in the generation process (e.g., .. i, !), which induce diverse and prefix-sensitive K KV-cache distributions.
Reuse Rate. Unlike Cleneslend's fixed reuse strategy (80%), K KVCMM adaptively determines KV-cache reuse, consistently achieving high reuse rates (70%-87.6%) This rate naturally declines as agent number increases due to more diverse contexts, but KVCOMM still effectively identifies shareable caches, confirming that adaptive reuse avoids context degradation.
4.3 Results of TTFT Speedup'
Table 2 reports TTFT per agent receiving IK tokens from user input with 512 prefix tokens and sharing the 512 response tokens with succeeding agents. The first agent, lacking upstream caches
In the original submission, the TTFT calculation for KVCOMM omitted the first token's decoding latency. The final version rectifies this.

---

(costing 86.6ms in "other" operations), shows modest acceleration x). Subsequent agents reduce prefilling dramatically to 26.9-38.6 ms via KVCOMM, achieving up x speedup (Agent 5).
Scalability in Context Length. We further examine scalability in Table 3, varying prefix (64-1K tokens) and output lengths (128-1K tokens) among three collaborating agents. KVCOMM achieves a minimum mean speedup of x.24 (shortest setting) and scales effectively 6. x.72x (longest setting), validating the approach's 's efficiency gain as context length and complexity increase.
4.4 Discussion and Ablation Study
Robustness to Request Order. Table 4 examines how request ordering affects KV- COMM'S cache alignment using MMLU.
Table Study on the robustness to varying request orders. Accuracy is reported. (#Agent = 4, Baseline
Acc = 68.0%; Model: 1lama3.1)
We test two random orders (Rand-1, Rand- 2), ascending and descending length orders. It can be observed that performance is cor-
Method
Rand-1
Rand-2
Ascending
Descending
KVCOMM
68.0
72.5
67.3
66.0
related with request order due to the designed anchor prediction criterion. Results confirm KV- COMM is robust across diverse ordering strategies, achieving consistent or slightly improved accuracy compared to the baseline, demonstrating minimal sensitivity to request sequence variability.
Contribution of Each Alignment Step. Table 5 details ablation results for three alignment components on MMLU under a four-agent setting: (1) position alignment via key rotation, (2) placeholder K KV-cache offset, and (3) prefix segment K K--ccche off- set. The results reveal that each alignment step is critical; omitting any severely de- grades accuracy. Although combining key rotation and prefix offset achieves 62.1 6%% accuracy, the response of each agent is visi-
Table 5: Ablation study on MMLU under four-agent setting. (Model: Llama-3.1)
k w/ rot
Φ w/ offset
p w/ offiet
Acc (%)
43. 1 1%
58.8%
60.1% 1%
38.6%
62. %
56.9%
68.0 %
bly less coherent with the original one (See
Appendix 6.4.6). Therefore, complete alignment is essential for robust cross-context performance.
Sensitivity to Hyperparameters. Ta-
Table 6: Hyperparameter analysis on GSM8K under the four-agent setting using Llama-3.1.
ble explores KVCOMM's sensitivity the entropy threshold y and anchor pool size V using GSM8K with four agents. = 0/0//V= = 0 refers to the original no- sharing method. It can be observed that with conservative reuse (7 0.1), ac- curacy improves slightly (1%), while mod- erate relaxation significantly boosts reuse (up to 98.2%) at minimal accuracy cost (3.3%). For V. performance is relatively stable with the increase of stored anchors, while the reuse rate finally becomes stable at 73.4%, indicating that V = 20 effectively balances efficiency and task performance.
Threshold (V 20)
Metric
0
0.1
0.3
0.5
0.7
0.9
Accuracy (%)
82.1
83.1
80.6
80.0
78.9
78.8
Reuse (%)
N/A
34.3
73.4
94.9
97.5
98.2
Maximum Anchor Num (7 0.3)
Metric
15
25
Accuracy (%)
82.1
82.0
81.4
81.2
80.6
80.6
Reuse rate (%)
N/A
44.0
60.3
66.2
73.4
73.4
5 Conclusion
In this paper, we explore KV-cache sharing for efficient communication in collaborative LLM-based MAS and introduce K KVCMM, a drop-in framework to enable efficient agent communication through shared KV-cache reuse and context-award cache offsetting. Besides, we perform analyses of KVcccche deviation across varying prefix contexts, and propose an anchor- based offiet estimator to effectively align and reuse shared context KV-caches. Extensive experiments conducted on Retrieval-Augment Generation (RAG), Math Reasoning, and Programming relatte related multi-agent systems demonstrate that our method provides an effective trade-off between prefilling efficiency and system accuracy, continuously reducing average latency as the number of agents increases. Specifically, KVCOMM can achieve ~6.7x average prefilling speedup under the three-agent setting on single H100 GPU, significantly improving the deployment efficiency of collaborative multi-agent language models.
10

---

Acknowledgments
Hancheng Ye, Jianyi Zhang, and Yiran Chen disclose the support from NSF 21 12562, ARO W911NF- 23-2-0224, and NAIRR Pilot project NAIRR240270. Danyang Zhuo discloses the support from NSF 2503010. We sincerely thank the program chairs, area chair, and reviewers for their valuable comments.
References
[1] Josh Achiam, Steven Adler, Sandhini Agarwal, Lama Ahmad, Igge Akkaya, Florencia Leoni Aleman, Diogo Almeida, Janko Altenschmidt, Sam Altman. Shyamal Anadkat, et Gpt-4 technical report. arXiv preprint arXiv:2303.0877/ 2023.
[2] Tom Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, et al. Language models are few-shot learners. Advances in neural information processing systems, 33: 13:1877-1901, 2020.
[3] Chen-Chia Chang, Chia-Tung Ho, Yaguang Li, Yiran Chen, and Haoxing Ren. Dre-coder: Automated dre checker code generation using Ilm autonomous agent. In Proceedings of the 2025 International Symposium on Physical Design, pages 143-151, 2025.
Mark Chen, Jerry Tworek, Heewoo Jun, Qiming Yuan, Henrique Ponde de Oliveira Pinto, Jared Kaplan, Harri Edwards, Yuri Burda. Nicholas Joseph, Greg Brockman, Alex Ray, Raul Puri, Gretchen Krueger, Michael Petrov, Heidy Khlaaf, Girish Sastry, Pamela Mishkin, Brooke Chan, Scott Gray, Nick Ryder, Mikhail Pavlov. Alethea Power, Lukasz Kaiser. Mohammad Bavarian. Clemens Winter, Philippe Tillet, Felipe Petroski Such, Dave Cummings, Matthias Plappert, Fotios Chantzis, Elizabeth Barnes, Ariel Herbert- Voss, William Hebgen Guss, Alex Nichol, Alex Paino, Nikolas Tezak, Jie Tang, Igor Babuschkin, Suchir Balaji, Shantanu Jain, William Saunders, Christopher Hesse, Andrew N. Carr, Jan Leike, Josh Achiam, Vedant Misra, Evan Morikawa, Alec Radford, Matthew Knight, Miles Brundage, Mira Murati, Katie Mayer, Peter Welinder, Bob McGrew, Dario Amodei, Sam McCandlish, Ilya Sutskever, and Wojciech Zaremba. Evaluating large language models trained on code. 2021.
[5] Yihua Cheng. Yuhan Liu, Jiayi Yao. Yuwei An, Xiaokun Chen, Shaoting Feng. Yuyang Huang, Samuel Shen, Kuntai Du, and Junchen Jiang. Lmeache: An efficient ky cache layer for enterprise-scale Ilm inference. arXiv preprint 100.0655,, 2025.
[6] ChuGyouk. Aime-22-25. Hugging Face Dataset, 2025. httph:tt////t. httpp::/htoginngfssscco///
ChuGyouk/A AAMEE-AAAMM-EAIAEAAA 22-25.
[7] Karl Cobbe, Vincet Kosaraju, Mohammad Bavarian, Mark Chen, Heewoo Jun, Lukasz Kaiser, Matthias Plappert, Jerry Tworek, Jacob Hilton, Reiichiro Nakano, Christopher Hesse, and John Schulman. Training verifiers to solve math word problems. arXiv preprint arXiv:21 11.1100.44. 14/68, 2021.
[8] Yuzhe Fu, Changchun Zhou, Tianling Huang. Eryi Han, Yifan He, and Hailong Jiao, Softact: A high- precision softmax architecture for transformers supporting nonlinear functions. IEEE Transactions Circuits and Systems for Video Technology, 34(9):8912 8923, 2024.
Yingqiang Gc, Wenyue Hua, Kai Mci, Jianchao Ji, Juntao Tan, Shuyuan Xu, Zelong Li, and Yongfeng Zhang. Openagi: When IIm meets domain experts. In Advances Neural Information Processing Systems (NeurIPS), 2023.
[10] In Gim, Guojun Chen, Seung-seob Lee, Nikhil Sarda, Anurag Khandelwal, and Lin Zhong. Prompt cache: Modular attention reuse for low-latency inference. Proceedings of Machine Learning and Systems, 6:325-338, 2024.
[11] Aaron Grattafiori, Abhimanyu Dubey, Abhinav Jauhri, Abhinav Pandey, Abhishek Kadian, Ahmad Al- Dahle, Aiesha Letman, Akhil Mathur, Alan Schelten, Alex Vaughan, et al. The llaaa 3 herd of models. arXiv preprint 2217224 2024.
[12] Daya Guo, Dejian Yang, Haowei Zhang, Junxiao Song, Ruoyu Zhang, Runxin Xu, Qihao Zhu, Shirong Ma, Peiyi Wang, Xiao Bi, et al. Deepseek-rl: Incentivizing reasoning capability llms via reinforcement learning. arXiv preprint arXiv:2501.1294. 2025.
[13] Dan Hendrycks, Collin Burns, Steven Basart, Andy Zou, Mantas Mazeika, Dawn Song. and Jacob Steinhardt. Measuring massive multitask language understanding, In International Conference on Learning Representations.

---

[14] Dan Hendrycks, Collin Burns, Saurav Kadavath, Akul Arora, Steven Basart, Eric Tang, Dawn Song. and Jacob Steinhardt. Measuring mathematical problem solving with the math dataset. In Tiitty-fitth Conference on Nearal Information Processing Systems Datasets and Benchmarks Truck (Round 2).
[15] Matthew Ho, Chen Si. Zhaoxiang Feng, Fangxu Yu, Zhijian Liu, Zhiting Hu, and Lianhui Qin. Arememo: Abstract reasoning composition with lifelong IIm memory. arXiv preprint arXiv:2509.0439, 2025.
[16] Binyuan Hui, Jian Yang. Zeyu Cui, Jiaxi Yang, Dayiheng Liu, Lei Zhang. Tianyu Liu, Jiajun Zhang, Bowen Yu, Kai Dang, et al. Qwen2. 5-coder technical report. arXiv preprint arXiv:2409. 1216,, 2024.
(17] Chao Huang Jiabin Tang, Tianyu Fan. AutoAgent: A Fully-Automated and Zero-Codc Framework for LLM Agents, 2025.
[18] Ting Jiang, Yixiao Wang, Hancheng Ye, Zishan Shao, Jingwei Sun, Jingyang Zhang. Zekai Chen, Jianyi Zhang, Yiran Chen, and Hai Li. Sada: Stability-guidee adaptive diffusion acceleration. arXiv preprint arXiv:2507.17335 2025.
[19] YICHEN JIANG, SUORONG YANG, SHENGJI TANG, SHENGHE ZHENG, and JIANJIAN CAO. A comprehensive survey of IIm driven collective intelligence: Past, present, and future. 2025.
[20] Hyunjik Kim, George Papamakarios, and Andriy Mnih. The lipschitz constant of self-attention. In International Conference on Machine Learning, pages 5562-5571. PMLR, 2021.
[21] Woosuk Kwon, Zhuohan Li, Siyuan Zhuang, Ying Sheng, Lianmin Zheng, Cody Hao Yu, Joseph E. Gonzalez, Hao Zhang, and Ion Stoica. Efficient memory management for large language model serving with pagedattention. In Proceedings of the ACM SIGOPS 29th Symposium on Operating Systems Principles. 2023.
[22] Guohao Li. Hasan Abed Al Kader Hammoud, Hani Itani, Dmitrii Khizbullin, and Bernard Ghanem. Camel: Communicative agents for "mind" exploration of large language model society. In Thirty-sevent Conference on Neural Information Processing Systems, 2023.
[23] Yueqian Lin, Yuzhe Fu, Jingyang Zhang, Yudong Liu, Jianyi Zhang, Jingwei Sun, Hai Li, Yiran Chen, et al. Speechprune: Context-aware token pruning for speech information retrieval. arXiv preprint arXiv:2412. 12009, 2024.
[24] Yueqian Lin, Qinsi Wang, Hancheng Ye, Yuzhe Fu, Hai Li, Yiran Chen, et al. Hippomm: Hippocampal- inspired multimodal memory for long audiovisual event understanding. arXiv preprint arXiv:2504. 2025.
[25] Aixin Liu, Bei Feng, Bin Wang, Bingxuan Wang, Bo Liu, Chenggang Zhao, Chengqi Dengr, Chong Ruan, Damai Dai, Daya Guo, et al. Deepseek- A strong, economical, and efficient mixture-of-experts language model. arXiv preprint r:Xiv:2405.04444 2024.
[26] Jijia Liu, Chao Yu, Jiaxuan Gao, Yuqing Xie, Qingmin Liao, Yi Wu, and Yu Wang. immpowereed hierarchical language agent for real-time human-ai coordination. arXiv preprint arXiv:2312. 2023.
[27] Yuhan Liu, Yuyang Huang, Jiayi Yao, Zhuohan Gu, Kuntai Du, Hanchen Li, Yihua Cheng, Junchen Jiang, Shan Lu, Madan Musuvathi, et al. Droidspeak: Kv cache sharing for fross-llm communication and multi-llm serving. arXiv preprint (:Xx::2411.028220 2024.
[28] Yuhan Liu, Hanchen Li. Yihua Cheng. Siddhant Ray. Yuyang Huang, Qizheng Zhang, Kuntai Du, Jiayi Yao, Shan Lu, Ganesh Ananthanarayanan, al. Cachegen: Ky cache compression and streaming for fast large language model serving. In Proceedings of the ACM SIGCOMM 2024 Conference, pages 38-56, 2024.
[29] Diego Mesquita, Amauri Souza, and Samuel Kaski. Rethinking pooling in graph neural networks. Advances in Neural Information Processing Systems, 33:2220-2231,, 2020.
[30] Mozhgan Navardi, Romina Aalishah, Yuzhe Fu, Yueqian Lin, Hai Li, Yiran Chen, and Tinoosh Mohsenin. Genai at the edge: Comprehensive survey on empowering edge devices. In Proceedings of the AAAI Symposium Series, volume 5, pages 180-187, 2025.
[31] Joon Sung Park, Joseph O' Brien, Carrie Jun Cai, Meredith Ringel Morris, Percy Liang, and Michael S Bernstein. Generative agents: Interactive simulacra of human behavior. In P Proceedings of the 36th annual acm symposium user interface software and technology, pages 1-22, 2023.
[32] Bharat Prakash, Tim Oates, and Tinoosh Mohsenin. Llm augmented hierarchical agents. In NewrlPS 2023 Foundation Models for Decision Making Workshop.
12

---

[33] Yujia Qin, Shihao Liang. Yining Ye, Kunlun Zhu, Lan Yan, Yaxi Lu, Yankai Lin, Xin Cong, Xiangru Tang. Bill Qian, al. Toolllm: Facilitating large language models to master 16000+ real-world apis. arXiv preprint wxXi:23307.66785 2023.
[34] Connor Shorten, Charles Pierse. Thomas Benjamin Smith. Erika Cardenas, Akanksha Sharma, John Trengrove, and Bob van Luijt. Structuredrag: Json response formatting with large language models. arXiv preprint arXiv:2408. 100888110000 11061, 2024.
[35] Significant Gravitas. AutoGPT.
[36] Charles Spearman. The proof and measurement of association between two things. The American journal of psychology, 100(3/4):441-711, 1987.
[37] Jianlin Su, Murtadha Ahmed, Yu Lu, Shengfeng Pan, Wen Bo, and Yunfeng Liu. Roformer: Enhanced transformer with rotary position embedding. Neurocomputing, 568:127063, 2024.
[38] Chanh-Tung Tran, Dung Dao, Minh-Duong Nguyen, Quoc-Viet Viet Pham, Barry O' O'Sullivan, and Hoang D Nguyen. Multi-agent collaboration mechanisms: survey of llms. arXiv preprint arXiv:2501.06522, 2025.
[39] Guanzhi Wang, Yuqi Xie, Yunfan Jiang, Ajay Mandlekar, Chaowei Xiao, Yuke Zhu, Linxi Fan, and Anima Anandkumar. Voyager: An open-ended emhodied agent with large language models. Transactions on Machine Learning Research.
[40] Qinsi Wang, Jinghan Ke, Masayoshi Tomizuka, Yiran Chen, Kurt Keutzer, and Chenfeng Xu. Dobi-svd: Differentiable svd for Ilm compression and some new perspectives. arXiv preprint a2X1v:2502.02723, 2025.
[41] Qinsi Wang. Jinghan Ke, Hancheng Ye, Yueqian Lin, Yuzhe Fu, Jianyi Zhang, Kurt Keutzer, Chenfeng Xu, and Yiran Chen. Angles don't lie: Unlocking training-efffcient rl through the model's own signals. arXiv preprint arXiv:2506.0 022281. 2025.
[42] Qinsi Wang, Bo Liu, Tianyi Zhou, Jing Shi, Yueqian Lin, Yiran Chen, Hai Helen Li, Kun Wan, and Wentian Zhao. Vision-zero: Scalable vim self-improvement via strategic gamified self-play, 2025.
[43] Qinsi Wang, Saeed Vahidian, Hancheng Yc, Jianyang Gu, Jianyi Zhang, and Yiran Chen. Coreinfer: Accelerating large language model inference with semantics-inspire adaptive sparse activation. arXiv preprint arXiv:2410. 18311, 2024.
[44] Qinsi Wang, Hancheng Ye, Ming- Yu Chung, Yudong Liu, Yueqian Lin, Martin Kuo, Mingyuan Ma, Jianyi Zhang, and Yiran Chen. Corematching: A co-adaptive sparse inference framework with token and neuron pruning for comprehensive acceleration of Dision-language models. arXiv preprint 1rXiv:2505.19235. 2025.
[45] Jason Wei, Xuezhi Wang. Dale Schuurmans, Maarten Bosma, Fei Xia, Ed Chi, Quoc V Le, Denny Zhou, al. Chain-of-though prompting elicits reasoning in large language models. Advances in neural information processing systems, 35:24824- 24837, 2022.
[46] Bingyang Wu, Shengyu Liu, Yinmin Zhong, Peng Sun, Xuanzhe Liu, and Xin Jin. Loongserve: Efficiently serving long-context large language models with elastic sequence parallelism. In Proceedings of the ACM SIGOPS 30th Symposium on Operating Systems Principles, pages 640-654, 2024.
[47] Chengyue Wu, Hao Zhang, Shuchen Xue, Zhijian Liu, Shizhe Diao, Ligeng Zhu, Ping Luo, Song Han, and Enze Xie. Fast-dllm: Training-free free acceleration of diffusion 11m by enabling kv cache and parallel decoding. arXiv preprint arXiv:2505.200 226188 2025.
[48] Qingyun Wu, Gagan Bansal, Jieyu Zhang, Yiran Wu, Beibin Li, Erkang Zhu, Li Jiang, Xiaoyun Zhang, Shaokun Zhang, Jiale Liu, et al. Autogen: Enabling next-gen 1lm applications via multi-agent conversations. In First Conference on Language Modeling.
[49] Yongii Wu, Yeehen Xu, Jingrong Chen, Zhaodong Wang, Ying Zhang, Matthw Lentz, and Danyang Zhuo Mces: A service-based approach to collective communication for multi-tenant cloud. In Proceedings of the ACM SIGCOMM 2024 Conference, pages 779-690, 2024.
[50] Chunqiu Steven Xia, Yinlin Deng, Soren Dunn. and Lingming Zhang. Agentless: Demystifying llm-based software engineering agents. arXiv preprint 240.22407001488 2024.
[51] Renqiu Xia, Haoyang Peng, Hancheng Ye, Mingsheng Li, Xiangchao Yan, Peng Ye, Botian Shi, Yu Qiao, Junchi Yan, and Bo Zhang. Structchart: On the schema, metric, and augmentation for visual chart understanding, arXiv e-prints, pages arXiv- 2023.
13

---

[52] Renqiu Xia, Bo Zhang, Hancheng Ye, Xiangchao Yan, Qi Liu, Hongbin Zhou, Zijun Chen, Peng Ye, Min Dou, Botian Shi, et al. Chartx & chartvlm: A versatile benchmark and foundation model for complicated chart reasoning. arXiv preprint axXiv:2402.12185 2024.
[53] Guangxuan Xiao, Yuandong Tian, Beidi Chen, Song Han, and Mike Lewis. Efficient streaming language models with attention sinks. The Twelfth International Conference on Learning Representations.
[54] Yechen Xu, Xinhao Kong, Tingjun Chen, and Danyang Zhuo. Conveyor: Efficient tool-aware Ilm serving with tool partial exccution. arXiv preprint arXiv:2406. 00059, 2024.
[55] Xiangchao Yan, Shiyang Feng, Jiakang Yuan, Renqiu Xia, Bin Wang, Bo Zhang, and Lei Bai. Surveyforge: On the outline heuristics, memory-drive -diive generation, and multi-dimensional evaluation for automated survey writing. arXiv preprint arXiv:2503.04629 2025.
[56] Jingbo Yang, Bairu Hou, Wei Wei, Yujia Bao, and Shiyu Chang. Kvlink: Accelerating large language models via efficient kv cache reuse. arXiv preprint arXiv:2502.1600 2025.
[57] Yuhao Yang. Jiabin Tang. Lianghao Xia, Xingchen Zou, Yuxuan Liang, and Chao Huang. Graphagent: Agentic graph language assistant. arXiv preprint (iii::242..102222 2024.
[58] Jiayi Yao. Hanchen Li, Yuhan Liu. Siddhant Ray. Yihua Cheng, Qizheng Zhang, Kuntai Du, Shan Lu, and Junchen Jiang. Cacheblend: Fast large language model serving for rag with cached knowledge fusion. In P Proceedings of the Twentieth European Conference Computer Systems, pages 94-199--99 109, 2025.
[59] Hancheng Ye, Chong Yu, Peng Ye, Renqiu Xia, Yansong Tang, Jiwen Lu, Tao Chen, and Bo Zhang. Once for both: Single stage of importance and sparsity search for vision transformer compression. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 2024.
[60] Hancheng Ye, Jiakang Yuan, Renqiu Xia, Xiangehao Yan, Tao Chen, Junchi Yan, Botian Shi, and Bo Zhang. Training-fres free adaptive diffusion with bounded difference approximation strategy. Advances in Neural Information Processing Systems, 37:306-33333 332, 2024.
[61] Zhitao Ying. Jiaxuan You, Christopher Morris, Xiang Ren, Will Hamilton, and Jure Leskovec. Hierarchical graph representation learning with diffcrentiable pooling. Advances in neural information processing systems, 31, 2018.
[62] Jiakang Yuan, Xiangchao Yan, Botian Shi, Tao Chen, Wanli Ouyang, Bo Zhang, Lei Bai, Yu Qiao, and Bowen Zhou. Dolphin: Closed-loop open-ended auto-research through thinking, practice, and feedback. arXiv preprint ar5ivv2501.002: 039//6, 2025.
[63] Aohan Zeng, Mingdao Liu, Rui Lu, Bowen Wang, Xiao Liu, Yuxiao Dong, and Jie Tang. Agenttuning: Enabling generalized agent abilities for 1lms. In Findings of the Association for Computational Linguistics ACL 2024, pages 3053-3077, 2024.
[64] Bo Zhang, Hancheng Ye, Gang Yu, Bin Wang, Yike Wu, Jiayuan Fan, and Tao Chen. Sample-centric feature generation for ccmi-supervised few-shot learning. IEEE Transactions on Image Processing, 31:2309-2320, 2022.
[65] Guibin Zhang, Yanwei Yue, Zhixun Li, Sukwon Yun, Guancheng Wan, Kun Wang, Dawei Cheng, Jef- frey Xu Yu, and Tianlong Chen. Cut the crap: An economical communication pipeline for L----b multi-agent systems. In The Thirteenth International Conference on Learning Representations, 2025.
[66] Lingzhe Zhang, Yunpeng Zhai, Tong Jia, Xiaosong Huang, Chiming Duan, and Ying Li. Agentfm: Role-aware failure management for distributed databases with lim-driven multi-agents. arXiv preprint 1eXiv:2504006611 2025.
[67] Siyan Zhao, Daniel Israel, Guy Van den Brocck, and Aditya Grover, Prepacking: A simple method for fast prefilling and increased throughput in large language models. arXiv preprint 24X4::2404409520 2024.
[68] Lianmin Zheng, Liangsheng Yin, Zhiqiang Xie, Chuyue Livia Sun, Jeff Huang, Cody Hao Yu, Shiyi Cao, Christos Kozyrakis, Ion Stoica, Joseph F. Gonzalez, et al. Sglang: Efficient execution of structured language model programs. Advances in Neural Information Processing Systems, 37:62557-622 625583. 2024.
[69] Mingchen Zhuge, Wenyi Wang, Louis Kirsch, Francesco Faccio, Dmitrüi Khizbullin, and Jürgen Schmid- huber. Gptswarm: Language agents as optimizable graphs. In Forty-first International Conference on Machine Learning, 2024.
14

---

NeurIPS Paper Checklist
1. Claims
Question: Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?
Answer: [Yes]
Justification: The motivations and contributions are well depicted and summarized in the abstract and introduction.
Guidelines:
The answer NA means that the abstract and introduction do not include the claims made the paper.
The abstract and/or introduction should clearly state the claims made, including the contributions made in the paper and important assumptions and limitations. A No
NA answer to this question will not be perceived well by the reviewers.
The claims made should match theoretical and experimental results, and reflect how much the results can be expected to generalize to other settings.
is fine to include aspirational goals as motivation as long as is clear that these goals not attained by the paper.
2. Limitations
Question:
Does the paper discuss the limitations the work performed by the authors?
Answer: [Yes]
Justification:
Limitations are discussed in Appendix A.I.
Guidelines:
The answer NA means that the paper has no limitation while the answer No means that the paper has limitations, but those are not discussed in the paper.
The authors are encouraged to create a separate "Limitations" section their paper.
The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (e.g., independence assumptions, noiseless settings, model well-specification, asymptotic approximations only holding locally). The authors should reflect how these assumptions might be violated in practice and what the implications would be.
The authors should reflect on the scope of the claims made, e.g., if the approach was only tested on few datasets or with few runs. In general, empirical results often depend on implicit assumptions, which should be articulated.
The authors should reflect on the factors that influence the performance of the approach. For example, facial recognition algorithm may perform poorly when image resolution is low images are taken in low lighting. Or speech-to-text system might not be used reliably to provide closed captions for online lectures because it fails to handle technical jargon.
The authors should discuss the computational efficiency of the proposed algorithms and how they scale with dataset size.
If applicable, the authors should discuss possible limitations of their approach to address problems of privacy and fairness.
While the authors might fear that complete honesty about limitations might used by reviewers as grounds for rejection, worse outcome might be that reviewers discover limitations that aren't 't acknowledged judgment and recognize that individual actions in favor of transparency play impor- tant role in developing norms that preserve the integrity of the community. Reviewers will be specifically instructed to not penalize honesty concerning limitations.
the paper. The authors should use their best
3. Theory Assumptions and Proofs
Question: For each theoretical result, does the paper provide the full set of assumptions and complete (and correct) proof?
Answer: [Yes]
15

---

Justification:
The theory assumptions and proofs are described in Section and Appendix
A.2.
Guidelines:
The answer NA means that the paper does not include theoretical results.
All the theorems, formulas, and proofs in the paper should be numbered and cross- referenced.
All assumptions should be clearly stated or referenced the statement of any theorems.
The proofs can either appear in the main paper or the supplemental material, but if they appear in the supplemental material, the authors are encouraged to provide short proof sketch provide intuition.
Inversely, any informal proof provided in the core of the paper should be complemented by formal proofs provided in appendix or supplemental material.
Theorems and Lemmas that the proof relies upon should be properly referenced.
4. Experimental Result Reproducibility
Does the paper fully disclose all the information needed to reproduce the main ex- perimental results of the paper to the extent that it affects the main claims and/or conclusions of the paper (regardless of whether the code and data are provided or not)?
Question:
Answer: [Yes]
Justification:
The selected models and benchmarks are clearly and fully presented in Section
4.
Guidelines:
The answer NA means that the paper does not include experiments.
If the paper includes experiments, a No answer to this question will not be perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data are provided or not.
If the contribution is dataset and/or model, the authors should describe the steps taken make their results reproducible or verifiable.
Depending on the contribution, reproducibility can be accomplished in various ways. For example, if the contribution is novel architecture, describing the architecture fully might suffice, or if the contribution a specific model and empirical evaluation, it may be necessary either make it possible for others to replicate the model with the same dataset, or provide access to the model. In general. releasing code and data is often one good way accomplish this, but reproducibility can also be provided via detailed instructions for how to replicate the results, access to a hosted model (e.g., in the case of large language model), releasing of a model checkpoint, or other means that appropriate to the research performed.
While NeurlPS does not require releasing code, the conference does require all submis- sions to provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution. For example
(a) the contribution is primarily a new algorithm, the paper should make clear how reproduce that algorithm. (b) If the contribution is primarily new model architecture, the paper should describe the architecture clearly and fully.
(c) the contribution is a new model (e.g., a large language model), then there should either be way to access this model for reproducing the results or way to reproduce the model (e.g., with an open-source dataset instructions for how to construct the dataset).
(d) We recognize that reproducibility may be tricky in some cases, in which case authors are welcome describe the particular way they provide for reproducibility. In the case of closed-source models, it may be that access to the model is limited in some way (e.g., to registered users), but it should possible for other researchers to have some path to reproducing or verifying the results.
5. Open access to data and code
16

---

Does the paper provide open access to the data and code, with sufficient instruc- tions faithfully reproduce the main experimental results, as described in supplemental material?
Question:
Answer: [Yes]
Justification:
The code is provided in supplemental material.
Guidelines:
The answer NA means that paper does not include experiments requiring code.
Please see the NeurlPS code and data submission guidelines (https://nil cc/
public/ guides/ Soniiee//ooeSuutei www.uuiissssiooiioo ssionPolicy for more details.
While we encourage the release of code and data, we understand that this might not be possible, "No" is an acceptable answer. Papers cannot be rejected simply for not including code, unless this is central to the contribution (c.g., for a new open-source benchmark).
The instructions should contain the exact command and environment needed to run to reproduce the results. See the NeurIPS code and data submission guidelines (https: á ps cc/ cc/publi c/gui des / CCodeooouuaaa ss onPol (cy) for more details.
The authors should provide instructions on data access and preparation, including how to access the raw data, preprocessed data, intermediate data, and generated data, etc.
The authors should provide scripts to reproduce all experimental results for the new proposed method and baselines. If only subset of experiments are reproducible, they should state which ones are omitted from the script and why.
At submission time, to preserve anonymity, the authors should release anonymized versions (if applicable).
Providing as much information as possible in supplemental material (appended the paper) is recommended, but including URLs to data and code is permitted.
6. Experimental Setting/Details
Does the paper specify all the training and test details (e.g., data splits, hyper- parameters, how they were chosen, type of optimizer, etc.) necessary understand the results?
Question:
Answer: [Yes]
Justification:
All details are carefully presented in Section 4.1.
Guidelines:
The answer NA means that the paper does include experiments.
The experimental setting should be presented in the core the paper to a level of detail that is necessary appreciate the results and make sense of them.
The full details can be provided either with the code, appendix, as supplemental material.
7. Experiment Statistical Significance
Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?
Question:
Answer: [Yes]
Justification: We report Spearman metric p between embedding distance and the KV-cache offset between two different tokens prefixed with different contexts.
Guidelines:
The answer NA means that the paper does not include experiments.
The authors should answer "Yes" if the results are accompanied by error bars, confi- dence intervals, statistical significance tests, least for the experiments that support the main claims of the paper.
The factors of variability that the error bars are capturing should be clearly stated (for
example, train/test split, initialization, random drawing of some parameter, or overall with given experimental conditions).
17

---

The method for calculating the error bars should be explained (closed form formula, call a library function, bootstrap, etc.)
The assumptions made should be given (e.g., Normally distributed errors).
It should be clear whether the error bar is the standard deviation the standard error of the mean.
It is OK to report 1-sigma error bars, but one should state it. The authors should preferably report a 2-sigma error bar than state that they have 96% CI, if the hypothesis of Normality of errors is not verified.
For asymmetric distributions, the authors should be careful figures symmetric error bars that would yield results that are out of range (e.g. negative rates).
show in tables or
If error bars are reported in tables or plots, The authors should explain the text how they were calculated and reference the corresponding figures tables in the text.
8. Experiments Compute Resources
For each experiment, does the paper provide sufficient information on the com- puter resources (type of compute workers, memory, time of execution) needed to reproduce the experiments?
Question:
Answer: [Yes]
Justification: The information of the employed compute resources elaborated in the implementation details of Section 4.
Guidelines:
The answer NA means that the paper does not include experiments.
The paper should indicate the type of compute workers CPU or GPU, internal cluster, or cloud provider, including relevant memory and storage.
The paper should provide the amount of compute required for each of the individual experimental runs as well as estimate the total compute.
The paper should disclose whether the full research project required more compute than the experiments reported in the paper (e.g., preliminary or failed experiments that didn't make into the paper).
9. Code Of Ethics
Question: Does the research conducted the paper conform, in every respect, with the NeurlPS Code of Ethics https:/ htt////euue /neurips.....///
Answer: [Yes]
Justification:
We make sure that the research conforms with the NeurlPS Code of Ethics.
Guidelines:
The answer NA means that the authors have reviewed the NeurIPS Code of Ethics.
If the authors answer No, they should explain the special circumstances that require deviation from the Code of Ethics.
The authors should make sure to preserve anonymity (e.g., there a special consid- eration due to laws or regulations in their jurisdiction).
10. Broader Impacts
Question: Does the paper discuss both potential positive socictal impacts and negative societal impacts of the work performed?
Answer: [NA]
Justification:
Our proposed method currently has apparent societal impacts.
Guidelines:
The answer NA means that there is no societal impact of the work performed.
If the authors answer NA or No, they should explain why their work has no societal impact why the paper does not address societal impact.
18

---

Examples of negative societal impacts include potential malicious unintended uses (e.g., disinformation, generating fake profiles, surveillance), faireess considerations (e.g., deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations.
The conference expects that many papers will be foundational research and not tied to particular applications, let alone deployments. However, if there is direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point out that a generic algorithm for optimizing neural networks could enable people to train models that generate Deepfakes faster.
The authors should consider possible harms that could arise when the technology is being used intended and functioning correctly, harms that could arise when the technology is being used as intended but gives incorrect results, and harms following from (intentional unintentional) misuse of the technology.
If there are negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition to attacks, mechanisms for monitoring misuse, mechanisms monitor how system learns from feedback over time, improving the efficiency and accessibility of ML).
11. Safeguards
Does the paper deseribe safeguards that have been put in place for responsible release of data or models that have a high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)?
Question:
Answer: [NA]
Justification:
This paper poses no such risks.
Guidelines:
The answer NA means that the paper poses no such risks.
Released models that have high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere to usage guidelines or restrictions to access the model or implementing safety filters.
Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images.
We recognize that providing effective safeguards is challenging, and many papers do not require this, but we encourage authors to take this into account and make best faith effort.
12. Licenses for existing assets
Are the creators or original owners of assets (e.g., code, data, models), used in the paper, properly credited and are the license and terms of use explicitly mentioned and properly respected?
Question:
Answer: [Yes]
Justification:
We have cited all papers that we used for experiments.
Guidelines:
The answer NA means that the paper does use existing assets.
The authors should cite the original paper that produced the code package dataset.
The authors should state which version of the asset used and, possible, include URL.
The name of the license (e.g., CC-BY 4.0) should be included for each asset.
For scraped data from a particular source (e.g., website). the copyright and terms of service of that source should be provided.
assets are released, the license, copyright information, and terms of use package should be provided. For popular datasets, papers rswi thoode. com / da sets has curated licenses for some datasets. Their licensing guide can help determine the license of dataset.
19

---

For existing datasets that are re-packaged, both the original license and the license of the derived asset (if has changed) should be provided.
If this information is not available online, the authors are encouraged to reach out to the asset's s creators.
13. New Assets
Question: Are new assets introduced in the paper well documented and is the documentation provided alongside the assets?
Answer: [NA]
Justification:
This paper does not release new assets.
Guidelines:
The answer NA means that the paper does not release new assets.
Researchers should communicate the details of the dataset/code/moddl as part of their submissions via structured templates. This includes details about training, license, limitations, etc.
The paper should discuss whether and how consent was obtained from people whose asset used.
At submission time, remember to anonymize your assets (if applicable). You can either create anonymized URL or include anonymized zip file.
14. Crowdsourcing and Research with Human Subjects
Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given to participants and screenshots, if applicable, as well as details about compensation (if any)?
Answer: [NA]
Justification: The paper does not involve crowdsourcing nor research with human subjects.
Guidelines:
The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.
Including this information in the supplemental material is fine, but if the main contribu- tion of the paper involves human subjects, then as much detail as possible should be included in the main paper.
According the NeurIPS Code of Bthies, workers involved in data collection, curation, or other labor should be paid at least the minimum wage in the country of the data collector.
15. Institutional Review Board (IRB) Approvals or Equivalent for Research with Human Subjects
Question: Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or equivalent approval/review based the requirements of your country or institution) were obtained?
Answer: [NA]
Justification: This paper does not involve research with human subjects.
Guidelines:
The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.
Depending on the country in which research is conducted, IRB approval (or equivalent) may be required for any human subjects research. If you obtained IRB approval, you should clearly state this in the paper.
We recognize that the procedures for this may vary significantly between institutions and locations, and we expect authors to adhere to the NeurlPS Code of Ethics and the guidelines for their institution.
For initial submissions, not include any information that would break anonymity (if applicable), such the institution conducting the review.
20

---

16. Declaration of LLM usage
Question: Does the paper describe the usage of LLMs if it is an important, original, or non-standard component of the core methods in this research? Note that if the LLM is used only for writing, editing, or formatting purposes and does not impact the core methodology. scientific rigorousness, originality of the research, declaration is not required.
Answer: [Yes]
Justification: This paper describe the usage of LLMs in multi-agent systems, mentioned in Section and 2.
Guidelines:
The answer NA means that the core method development in this research does not involve LLMs as any important, original, or non-standard components.
Please refer to our LLM policy (https: : //eur i ps cc/ Conferences/ 2025/ LLM) for what should should not be described.
21

---

Appendix
Due the ten-page limitation of the manuscript, we provide more details and visualizations from the following aspects:
Sec, 6.1: Limitations and Broader Impacts.
Sec. 6.2: Method Explanation.
Sec. 6.2.1: Glossary.
Sec. 6.2.2: Theoretical Proof of Proposition and
Sec. 6.2.3: Placeholder Naming Rules.
Sec. 6.2.4: KV-cache Management System.
Sec. 6.2.5: Algorithm of KV KVCOMM.
Sec. 6.3: More Experimental Details.
Sec. 6.3.1: Statistical Analysis of KV-cache Proximity and Offset Proximity.
Sec. 6.3.2: Prompts on Three Benchmarks.
Sec. 6.4: More Experimental Analysis.
Sec. 6.4.1: Evaluations on Harder Reasoning Benchmarks
Sec. 6.4.2: Analysis on the Matching Criterion.
Sec. 6.4.3: Analysis the Approximation Method.
Sec. 6.4.4: Analysis the Overhead in Long-context Anchor Matching.
Sec. 6.4.5: Analysis on the Memory Cost of KУCOMM.
Sec. 6.4.6: Visualization of Responses Generated by Different Combinations Align-
ment Strategies.
Sec. 6.4.7: Visualization of Anchor Distribution.
Sec. 6.4.8: Visualization of Difference between Prefix and Placeholder Offset Distribu- tions.
Sec. 6.4.9: Visualization of Distance between Approximated and Real Offsets.
6.1 Limitations and Broader Impacts
Currently, KVCOMM evaluated on LLM-based multi-agent systems that process text inputs. long-term vision of multi-agent systems is achieve lossless acceleration any modality input, such as image, video, or audio input. Besides, although KVCOMM can accelerate the prefilling process of each agent, the decoding latency another bottleneck for efficient collaboration between agents cannot be accelerated by KVCOMM, which will be the future work to co-optimize these two stages.
Furthermore, as mentioned in Sec. 3.1, KVCOMM is directly applicable for groups homogeneous agents. For agents with identical architectures but different weights, KVCOMM holds promise but pending further exploration. Also, in principle, KVCOMM can be applied wherever context change patterns automatic prefix caching in vLLM). Although currently KVCOMM does not cover fully dynamic, unstructured cases, recognize this as meaningful direction and will extend KVCOMM for more dynamic agentic and debate-style benchmarks future work. For the heterogeneous multi-agent systems with different attention formulations [25, 8), it remains underexplored how to use KV-cache communication to facilitate efficiency.
provided shared text segmentation is feasible (as with techniques similar to
6.2 Method Explanation
6.2.1 Glossary
Base KV-cache KV-cache for prefix/placcholder under its initial context or without external input; serves as the reference for offsets.
KV-eache offset deviation The difference between shared text's KV-cache under a new prefix and its base KV-cache (Keys require RoPE-based alignment before offsetting).
22

---

Placeholder / Prefix offset Offsets for the placeholder segment (external input) and its adjacent prefix segment (predefined context), respectively, relative their base KV-caches.
Offset variance problem KV-cache offsets for the same text can vary substantially across contexts, so static reuse is unreliable.
Positional alignment/ / Key de-rotation RoPE de-rrotation/re-roottoo to align Keys before offset- ting.
Shareability Whether request can skip prefill under anchor criteria; not, dense prefilling used.
Shared memory V-cache The shared storage that holds base K KV-caches across agents.
Dense prefilling/ generation The full computation path (prefill + decode) used when reuse not possible.
Anchor (pool) A small set of representative examples, each storing placeholder and prefix offseet, used to interpolate offsets for new contexts.
6.2.2 Theoretical Proof of Proposition 1 and 2
Throughout the proofs, we reuse the per-layer update defined in Eq. (2):
h
++1 = hi, + FFN (h', + Attn Attn'(h').) (h').).
(A.1)
We introduce the Lipschitz conditions for attention and FFN modules [20, 43, 44,
Attn' Attn'(h'), Attn' (h') nzll
h
h/ll-
(A.2)
FFN (h)) FFN
hb)
h
(A.3)
Proof of Proposition
Define the maximum divergence between two hidden states h', h' at layer
1:
A's max
[hg 1illl with A' on-
(A.4)
kS
Subtracting two instances of Eq. (A.1) and applying Eq. (A.2) yields:
[ ++1
6 < h 1 + g (1hh, -h, I| + Attn' Attn'(h'), Attn' (ttn(())).ll))
(A.5)
< (1+ 8'a Σ | hlll
(A.6)
< (1+ 8 + +noad
(A.7)
Thus, we derive:
4+ < (1+ (1+ where o' = (1+ ()+n)
(A.8)
Unrolling this recursion from layer 1 to 1, have:
1-1 A < 5, [1++
(A.9)
j=1
Projecting the key space via linear mapping W and RoPE, whose spectral norms are bounded by constants Ci and CR respectively, we have:
[K K1 S C
II o²),
(A.10)
1=1
which completes the proof of Proposition 1. Due to analogous reasoning, the bound for value vectors is similar except for the absence of RoPE projection.
23

---

Proof of Proposition 2 Let two different prompts Pa and ps be prefixed to both tokens and Denote their key vectors at positions when prefixed by Pa as kn k for token u and respectively. and at positions when prefixed by ps as kng kn, respectively. Define the two key vector deviations follows:
K nb
Knn
K
kn.
(A.11)
Then we have:
| 1ll
|kms
k/e (Knn knn )ll
(A.12)
lkh Kn ll + [knn k
(A.13)
Applying Proposition
each term, have:
[ - k2, CROkOn
[1
(A.14)
j=1
1-1
1km - -Kn 1l & CROKOn
Ins
(1
(A.15)
1=1
where 6n. max |b| higll. 8ns maxisns [1b]]
h Since h h for k S no 1 (the same prefix pa), h = h (the same token u), h = h (the same token v), we have 8n. = Ons = /// h ll. Thus, we conclude:
(the same prefix p.). h = lig for l S Tha
1-1
| A' s 2 2C on.
(A.16)
1=1
completing the proof of Proposition 2. The proof of the bound for value vectors similar, except for the absence RoPE projection.
Table A.I: K VV-cache management strategy for both anchor pools and current requests K KV-cache sharing among agents.
Anchor Manager
1st level
2nd level
3rd level
Indices
Placeholder ID, e.g.. user_que st ion
Anchor
Index,
e.g.,
Agent ID embedding, e.g., agent_l_ph.
anchor [0]
Values
Anchor List
Dict of different KV-cache off- set in different agents and the an- chor embedding tensor
KV-cache embedding ten-
sor
Shared K VV-achhe Manager
1st level
2nd level
3rd level
Indices
Agent ID User put, e.g., agent_
Placeholder id, e.g., response
Turn
Index, response [-1]
e.g.,
Values
Dict of different agents' response
Dict of response and prefix KV- caches
KV-cache list
KV and outside
input KV
6.2.3 Placeholder Naming Rules
The placeholder in each agent's prompt template in a multi-agent system can be divided into three categories: user input, tool execution results, and responses from other agents [48, 69, 65). Conse- quently, we design the name of each placeholder in the prompt template according to its category.
24

---

Algorithm 1: Anchor-based KV-cache Communication in Multi-Agent Systems (KVCOMM)
Input:
Agent set M with prompts containing placeholders (f(m.s)): Anchor pool capacity V; Entropy threshold
Output: Efficiently updated KV-caches for all agents and responses from all agents.
foreach agent E M do
Extract placeholder tokens {d(m.i) / from prompt;
Initialize anchor pool Ag for each placeholder ((m,s) if not exist;
foreach new input sample do foreach agent m E .M do if any placeholder sample is not in the shared memory then
Compute the base KV-caches for the placeholder samples absent in the shared memory and store them in the shared memory;
if all placeholders in the template are predicted as shareable according Eq. (5) then
/ Reuse Placeholder KV-caches
asyne foreach placeholder ((m,k) in agent do
// Anchor Mat Matching and Offset Approximation
Retrieve base KV-cache in the shared memory; Retrieve anchor pool Apcm..)
Identify matched anchors 1/ E A@
Compute weights Worm. .y -y =softmax(-]lbe (l hep II) across anchors;
Approximate KV-cache using Eq. (6);
Similarly, update neighboring prefix segments using Eq. (7):
Concatenate all updated { { (k/v)p(...)) for agent m;; Response and its Srcachhe-Decodnng based on the concatenated KV-cache; if the response's KV-cache reusable according to Eq. (5) then
Store the response KV-cache in the shared memory for reference of other agents; else
Store the response V-cachhe the anchor pool of the response placeholder as the base K V-cache of new anchor;
else
new anchor Response, Real KV-cache of all placeholderst Dense generation for the input sample;
Add
asyne foreach placeholder ((m.)) in agent m do
Retrieve the base KV of ((m.i) from the shared memory;
(ikk/v((m,cmmm)) the offset between the real KV-cache and the base KV-cache of ((m.));
A A(k/v) (m..pmms)) the offset between the real KV-cache and the base KV-cache of ((m,f)
Abcmni
Appon U (0(m)) (k/v)&rm.e) A(k/v) (m))c..))) A(k/v)p P
if Adrmmol V then
Prune least-frequently-used among earliest anchors in A Adcn.ni
Suppose an agent is assigned unique agent id xxx, and is succeeded to the other two agents, whose agent id are yyy and zzz respectively, the naming rule of the placeholders Agent xxx's prompt template defined as follows:
User input: (user_questiion)
Tool execution results at the current turn:
Tool execution results at the previous 1-th turn:
Response from Agent yyy at the current turn: fagentt_vy_cuurren
25

---

Response from Agent z2z at the current turm: [agent__zza_uurzent
Response from itself at the previous 1-th turn: [agent__xx_histtrrz_
Response from Agent yyy at the previous f-th turn: (agent_
Response from Agent z2z at the previous (-th turn:
Based on the above rule, it helps easily insert potential placeholders in each agent's initial prompt template and unify the addressing rule for all agents' placeholders the shared anchor pools.
6.2.4 VV-cchee Management Strategy
Regarding the cache management strategy in KVCOMM, designed three-level cache managers to achieve efficient writing and retrieving anchors' KV-caches and the current shared KV-ccches, respectively. As shown in Table A.I, based on these two cache managers, each agent can quickly retrieve their intended KV-caches and also store their generated KV-caches. Mcanwhile, we conduct the process of KV-ccche storage and retrieval asynchronously. thus further improving the efficiency.
6.2.5 Algorithm of KVCOMM
The specific details of KVCOMM are shown in Algorithm 1.
6.3
More Experimental Details
6.3.1 Statistical Analysis KV-cache Proximity and Offset Proximity
In Figure 4, we aim to evaluate the correlation between the KV-cache proximity and the token embedding proximity, well as the correlation between the KV-cache offset proximity and the token embedding proximity. This evaluation is crucial to validate the effectiveness of our approach in accurately approximating the KV-cache offsets of tokens by the embedding-closeed anchors. For Figure 4a and 4b, we randomly sample 4000 distinct vocabulary tokens and then select 300 token pairs that are closest in the embedding space to form three cqually-sized distance bins ("near", "mid", and "far"). ). For all token pairs, we prepend them with the same prefix context, test their K/V distance between each other, and compute the Spearman correlation coefficients [36] between the token distance and their K/V distances. For Figure 4c and 4d, we adopt the same setting, further prepend each token with different prefixes, and test the cache deviation distance between different tokens.
6.3.2 Prompt Design on Three Benchmarks
We show the detailed prompt template of each agent that is deployed in our experiments, which follows GPTSwarm [69] and AgentPrune [65].
MMLU We cycle through the following roles for multi- reasoning:
Knowledgeable Expert
Wiki Searcher
Critic
Mathematician
FinalRefer
Below, we show the prompt template for each role:
Knowledgeable Expert
You are a knowledgeahle expert in question answering. Please give at most six key entities that need to be searched in wikipedia to solve the problem. Key entities that need to be searched are included between two @ when output, for example: @ catfish effect@, @broken window @ffect @ Shakespeare @ there is no entity the question needs to be searched Wikipedia, you don't have provide it. sipestion)
The task is:
26

---

Wiki Searcher
You will be given question and wikipedia overview of the key entities within it. Please refer to them step by step to give your answer. And point out potential issues in other agent's 's analysis.
The task is: seserr_qusst ion)
The key entities of the problem explained in Wikipedia follows: (condition_ At the time, the oulputs of other agents
follows:
Agent role Knowledge Expert, oulput In the last round of dialogue, the outputs of other agents were: Agent 1, role is Knowledge Expert, output
Critic
You are an excellent critic,
Please point out potential issues in other agent's 's analysis point by point. The task is: [user_quest_ question) At the time, the outputs of other agents follows:
Agent role Knowledge Expert, output (agent_11 _oot_____urr____________________
Agent 2, role is Wiki Searcher, output is: fagent_?_current. In the last round of dialogue, the outputs of other agents were: Agent 1, role Knowledge Expert, output (agent_ 1 histtory--2
Agent 2, role is Wiki Searcher, output is:
(agent history--1)
Mathematician
You mathematician who good at math games, arithmetic calculation, long- term planning. The task is: seser_queetioo ion
At the same time, the outputs of other agents are as follows: Agent 1, role is Knowledge Expert, output is:
(agent_] Lereen____uureee ]
Agent 2, role Wiki Searcher, output is:
fagent_ 2 j__uurrent
Agent 3, role is Critic, output is:
[agent_3_current]
the last round dialogue, the outputs other agents were: Agent role is Knowledge Expert, output is:
(agent_] hi striittoy_-11
Agent 2, role is Wiki Searcher, output is:
Agent 3, role Critic. output
(agent history--1i
FinalRefer
You are the top decision- maker and are good at analyzing and summarizing other people's 's opinions, finding сиrоrs and giving final answers. You will receive question followed by four possible answers labeled A, B, and Only recommendations provided by the output other agents. Your response must be exactly one letters A, B, C, or D, with no additional characters or text.
correct. Choose the correct option based the analysis and
The task is: usser__e jusse__questi questiion
At the same time, the outputs of other agents follows:
Agent role Knowledge Expert, output aagent_ 1 Juuurent]
Agent 2, role is Wiki Searcher, output is: (agent_2 _________rr]]
Agent 3, role is Critie, output
27

---

(agent_3 _0____3__________
Agent 4, role is Mathematician, output is:
(agent_ 4 cuureett
last round dialogue, outputs other agents were: Agent 1, role is Knowledge Expert, output is:
[agent_1_hise wtw_hhisttry_l111
Agent 2, role is Wiki Searcher, output is:
Agent 3, role is Critie, output
(agent_3_hiss hishsttttr__11
Agent 4, role is Mathematician, output is:
(agent_ historryt-l-
GSM8K
We cycle through the following roles:
Math Solver
Mathematical Analyst
Programming Expert
Inspector
FinalRefer
Below are the prompt templates for each agent:
Math Solver
You math expert.
You will given a math problem hints from other agents.
Give your own solving process step by step based on hints.
The last line of your output contains only the final result without any units, for example: The answer is
140
You will be given some examples you may refer
Angelo and Melanie want plan how many hours over the next week they should study together for their test next week.
They have 2 chapters of their textbook to study and 4 worksheets to memorize. They figure out that they should dedicate 3 hours to each chapter of their textbook and 1.5 hours for each worksheet.
hours day, many days should they plan to study total over the next week they take a 10-minute break every hour, include 10-minute snack breaks each day, and 30 minutes for lunch cach day?.
they plan study no more
A: Let's think step by step.
Angelo Melanie think they should dedicate hours to each of the chapters, hours x 2 chapters 6 hours total.
For the worksheets they plan to dedicate 1.5 hours for each worksheet, 1.5 hours x 4 worksheets 6 hours total. Angelo and Melanie need start with planning 12 hours to study, at 4 hours day, 12 4 days.
However, they need to include time for breaks and lunch. Every hour they want to include a 10-mi (0-minute break, so 12 total hours x 10 minutes = 120 extra minutes for breaks.
They also want to include 3 10-minute snack breaks, 3 x 10 minutes = 30 minutes. And they
include 30 minutes for lunch each day, so 120 minutes for breaks 30 minutes for lunch = 180 minutes, 180 / minutes per
snack breaks + 30 minutes So Angelo and Melanie want to plan 12 hours to study + hours of breaks 15 hours total. They want to study no more than 4 hours each day, 15 hours 4 hours each day 3.75 They will need to plan to study 4 days allow for all the time they need. The answer 4
hours.
Q: Bella has two times as many marbles as frisbees. She also has 20 more friibees than deck cards.
she buys 2/5 times more of each item, what would be the total number of the items she will have if she currently has 60 marbles?
Let's think step by step
When Bella buys 2/5 times more marbles, she'l I1 have increased the number of marbles by 2/5 = 24 The total number of marbles she '1l have is 60+24 = 84
28

---

If Bella currendy has 60 marbles, and she has two times as many marbles as frisbees, she has 60/2 = 30 frisbees.
Bella buys 2/5 times more frisbees, she'll have 3/5 more frisbces. The total number of frisbees she" 'Il have will increase to 30+12 Bella also has 20 more frishees than deck cards, mcaning she has 30-20 = 10 deck cards If she buys 2/5 times more deck cards, she'll IIl have 2/5+10 4 more deck cards.
The total number of deck cards she'll lll have 10+4 14 Together, Bella will have a total 14+42488-
items amswer
Q: Susy goes to large school with 800 students, while Sarah goes a smaller school with only 300 students. At the start of the school year, Susy had 100 social media followers. She gained 40 new followers in the first week of the school year, half that in the second week, and half of that in the third week. Sarah only had 50 social media followers at the start the year, she gained 90 new followers the first week, a third of that in the second week, and a third of that in the third week. After three weeks, how many social media followers did the girl with the most total followers have? A: Let's think step by step After one week, Susy has 100+40 = 140 followers. In the second woek, Susy gains 40/2 = 20 new followers.
In the third week, Susy gains 20/2 10 new followers.
In total, Susy finishes the three weeks with 140+20+10 170 total followers. After one week, Sarah has 50+90 140 followers.
After the second week, Sarah gains 90/3
followers.
After the third week, Sarah gains 30/3 = 10 followers.
So. Sarah finishes the three weeks with 140+30+10 = 180 total followers.
Thus, Sarah is the girl with the most total followers with total of 180. The answer 180
Q:
Mathematical Analyst
are mathematical analyst. will be given a math prohlem, analysis and ende from other agents. You need to first analyze the problem-solving process step by step, where the variables are represented by letters. Then you substitute the values into the analysis process to perform calculations and get the results. The last line of your output contains only the final result without any units, for example: The answer 140 You will be given some examples you may refer to.
Q: There are 15 trees the grove. Grove workers will plant trees in the grove today. After they are done, there will be 21 trees. How many trees did the grove workers plant today?
A: ## Problem solving process analysis
There (ori tree num] trees originally.
Then there were [after [after_planted tree num) trees after some more were planted. So the number of trees planted today (today_ _atrttd_paeeedd_ee num] is the number of trees after planting @icter_plnted_____e cointed_tree_nuue tree_num] minus the number of trees before planting {ori_tree, cor__tree_num) The answer [today_planted num] (after piiiter_ppaneddd__ tree [ace__mmm-- {ori tree_ ree__num.. ## Actual analysis and solution process
this question, (ori (ori_tree_ pe_ttree_nuuee num) = 15 and [after_plamted__tree trrc__ttcceemeeee = 21.
There are 15 trees originally.
Then there were 21 trees after some more were planted. So the number of trees planted today must have been 21 15 6.
The answer 6
Q: Leah had 32 chocolates and her sister had 42, If they ate 35, how many pieces do they have left in total?
A:## Problem solving process analysis
Originally, Leah had (Leah num] Leah num chocolates.
Her sister had (sister_nume chocolates. So in total they had vall_num] [Leah_num] (sister_mmut __m] chocolates.
After eating (eating num) chocolates, the number of chocolates they have left (remain j is aall [ minus leating seating_num).
The answer (remain num) = (all pall_pumj- (eating num).
## Actual analysis and solution pracess
In this question, (Leah_num Baah___mmm___ 32, (sister_num) 42 and (all_ 35.
So, in total they had 32 42 74 chocolates originally.
29

---

After eating 35 chocolates, they had 74 35 39 chocolates.
The
39
Q: seser_question
At the same time, there are the following responses to the same question for your reference:
Agent 1 as a Math Solver his answer to this question is: [agent_]___ _____ret_____] In the last round of dialogue, there the following responses the question for your reference: Agent a Math Solver his answer to this question was:
(agent_1 1isttorr_tt_
Programming Expert
You are a programming expert. You will be given math problem, analysis and code from other agents. Integrate step-by-step reasoning and Python code to solve math problems. Analyze the question and write functions to solve the problem. The function should not take any arguments and use the final result as the return value. The last line of code calls the function you wrote and assigns the return value to the answer variable. Use Python code block write your response. For example:
'python def Eun ():
x 10
y - 20
return x y
answer = fun ()
...
Do include anything other than Python code blocks in your response. You will be given some examples you may refer to. Q: Olivia has $23. She bought five bagels for $3 each. How much money does she have left?
A:
'python
def money_left ()
money_initial 23
bagels
bagel_coot 3
money_spent bagels bagel_cost
remaining_monet = money_initial - money_spent return remaining_money
answer = money_left ...
Q: Michael had 58 golf balls. On tuesday, he lost 23 golf balls. On wednesday, he lost 2 more. How many golf balls did he have at the end wednesday?
A:
'python
def remaininggeool__ballll/
golf ballbballlinntiall 58
golf
23
golf folf_balls_lef= - sealf_balls_iiitti
2
golf__aalls_lost_tussee
golf_ball lost_ wew.edsddddeddayy
remainiing_oolf_ballls- = golf_balls_left
return remaining_golf_balll balls
answer remairiinggsoff_balls/
Q: (user_qqesttee qire_uueesioon
At the same time, there are the following responses the same question for your reference:
30

---

Agent 1 as Math Solver his answer to this question is: (agent_ [_uureett]
Agent 2 as Mathematical Analyst his answer to this question is: iirent_2_curreee
In the last round of dialogue, there the following responses to the same question for your reference:
Agent Math Solver his answer this question was:
Livent___histovee
Agent 2 Mathematical Analyst his answer this question (agent 2 hishtoryy____
Inspector
You are an Inspector. You will be given a math problem, analysis and code from other agents. Check whether the logic/calculation of the problem solving and analysis process is correct (if present). Check whether the code corresponds to the solution analysis (if present). Give your own solving process step by step based on hints. The last line of your output contains only the final result without any units, for example: The answer is 140 You will be given some examples you may refer to. Q: (user_quest At same time, there are the following responses to same question for your reference: Agent as Math Solver his answer to this question is:
(agent_2) ___uur___t______________________
Agent 2 as Mathematical Analyst his answer to this question is:
Agent 3 as Programming Expert his answer to this question is:
1osgent3__cureet
the result is (condition_] _iuriiion____entt cuureett
In the last round of dialogue, there were the following responses to the same question for your reference:
Agent as a Math Solver his answer to this question was: (agent 1 historyy-I]
Agent 2 as Mathematical Analyst his answer to this question was:
(agent 2 histoory__lii
Agent 2 as Programming Expert his answer to this question was: (agent histoory-1l)
the result is (condition_ 3_iistory__t]]
FinalRefer
are the top decision-maker. Good at analyzing and summarizing mathematical problems, judging and summarizing other people's solutions, and giving final answers to math problems. You will be given math problem, analysis and code from other agents. Please find the most reliable answer based the analysis and results of other agents. Give reasons for making decisions. The last line of your output contains only final result without any units, example: answer The task is: user__ _eer__uusttt. iiestiin ion)
At the same time, the output of other agents is as follows: Agent 1, role is Math Solver, output is:
[agent__] |_currentt
Agent 2, role Mathematical Analyst, output
(agent_ 2_current|
Agent 3, role is Programming Expert, output is: [osentt3_cuurcete
the result is (condition_3 ) 3_ourrent]
Agent 4, role is Inspector, output @osent___curree
In the last round of dialogue, the outputs of other agents were: Agent I, role is Math Solver, output is:
lagent history--1i
31

---

Agent 2, role is Mathematical Analyst, output is:
lagent historrtttyr...........
Agent 3, role Programming Expert, output is:
the result is (condition_3_ha 3_historry__1 Agent 4, role is Inspector, oulput is: (agent_ history_-1]
HumanEval Benchmark generation and debugging:
For HumanEval, the following roles are cycled for collaborative code
Project Manager
Algorithm Designer
Programming Expert
Test Analyst
FinalRefer
Below are the prompt templates for each agent:
Project Manager
You are project manager. You will be given function signature and its docstring by the user. You are responsible for overseeing the overall structure of the code, ensuring that the code is structured complete the task. Implement code concisely and correctly without pursuing over-engineering. You need to suggest optimal design patterns to ensure that the code follows best practices for maintainability and flexibility. You can specify the overall design of the code, including the classes that need to be defined (maybe none) and the functions used (maybe only one function). hope your reply will be more concise. Preferably within fifty words. Don't t list too many points.
The task is: (user_question,
Algorithm Designer
You are an algorithm designer. You will be given a function signature and its docstring by the user. You need specify the specific design of the algorithm, including the classes that may be defined and the functions used. You need to generate the detailed documentation, including explanations of the algorithm, usage instructions, and API references. When the implementation logic is complex, you can give the pseudocode logic of the main algorithm. hope your reply will more concise. Preferably within fifty words. Don't 't list too many points.
The task is: (user_ee [user_question] questien] At the same time, the outputs and feedhacks of other agents are as follows:
Agent as Project Manager:
The code written by the agent is:
tagent_l_curze
Whether it passes internal testing?
(condition_l_currt) In the last round of dialogue, the outputs and feedbacks of some agents Agent as a Project Manager:
The code written by the agent is:
(agent... wwwhisttry_____
Whether it passes internal testing?
(condit: cinditionn ti hi laao____hhtttoo________
Programming Expert
You are programming expert. You will be given function signature and its docstring by
32

---

the user. You may be able to get the output results of other agents. They may have passed internal tests, but they may not be completely correct. Write your full implementation (restate the function signature). Use Python code block to write your response. For example:
"'yython print (' Helll world!')
Do not include anything other than Python code blocks in your response. Do not change function names and input variable types in tasks.
The task is: (user_que "rr__quesiinn
At the same time, the outputs and feedbacks other agents are as follows: Agent I as Project Manager:
The code written by the agent is:
[agent_1_ peseet___cuurrte purrent]
Whether it passes internal testing?
( (condit: t
Agent 2 as a Algorithm Designer provides the following info:
(agent 2_________ [ccurren]]
In the last round dialogue, the outputs and feedbacks of some agents were: Agent I as Project Manager: The code written by the agent was:
(agent 1 hi stssoorr_1 Whether it passed internal testing?
(condi i _on_1_t hi story_-1)
Agent 2 as Algorithm Designer provided the following info: (agent. 2 history_-1)
Test Analyst
You are test analyst. You will be given function signature and its docstring by the user. You need to provide problems the current code or solution based on the test data and possible test feedback in the question. You need to provide additional special conditions, etc. that should be paid attention to when writing code. You can point out any potential errors in the code. I hope your reply will be more concise. Preferably within fifty words. Don't t list too many points.
cases, boundary
The task is: At the same time, the outputs and feedbacks of other agents are as follows: Agent 1 Project Manager:
The code written by the agent is:
Lagent___cuurenee
Whether it passes internal testing?
Econdition. o___oo_re___et]
Agent 2 as Algorithm Designer provides the following info:
(agent_ caatt___cureett_
Agent 3 as Programming Expert:
The code written by the agent is:
{agent current)
Whether it passes internal testing?
(condition_3 3___r_eec____
In the last round of dialogue, the outputs and feedbacks of some agents were:
Agent as Project Manager:
The code written by the agent was:
[agent_1_hia hi (it_iisttry__1] Whether it passed internal testing?
(conditi ioliition 1__ story_-1)
Agent 2 as an Algorithm Designer provided the following info: (agent history_-1)
33

---

Agent 3 as Programming Expert: The code written by the agent was:
(agent_ 3 wistitt history_ Whether it passed internal testing? (condition 3______r 3_hissthe mastoory_-l
FinalRefer
You are the top decision-makee and are good at analyzing and summarizing other people's opinions, finding errors, and giving final answers. And you are an Al that only responds with only Python code. You will be given a function signature and its docstring by the user. You may be given the overall code design, algorithm framework, code implementation test problems. Write your full implementation (restate the function signature). If the prompt given you contains code that passed internal testing, prompt,
you can choose most reliahle reply. there no code that has passed internal testing you can change it yourself according to the prompt. Use a Python code block to write your response. For example:
'python
print (' Hello world!')
not include anything other than Python code blocks in your response The task is: Luser_questee At the same time, the output of other agents is as follows: Agent
Project Manager:
The code written by the agent was: (agent_l_aurreds
Whether it passed internal testing?
ccndition. 1_currentttt__
Agent 2 Algorithm Designer provided the following info:
sogent_?_current
Agent 3 as a Programming Expert:
The code written by the agent was:
Whether it passed internal testing?
(condition aesoo___uuur__e 3_current]
Agent 4, role is Test Analyst, output is:
The code written by the agent was:
lagent_4 4_current]
Whether it passed internal testing?
(condit) 4 [___urertt]]
In the last round of dialogue, the outputs of other agents were:
Agent 1 Project Manager:
The code written by the agent
(agent_] hishstory_-11 Whether it passed internal testing?
(condition iis_n_nnn_________eeee_eeaaeiees histeryy-1)
Agent 2 (agent_2 f_att__2hhstor_______
Algorithm Designer provided the following info:
Agent 3 as a Programming Expert:
The code written by the agent was:
lagent__ 3_histery___
Whether it passed internal testing?
conditioon historyy-t---e
Agent 4, role is Test Analyst, output is:
The code written by the agent was:
(agent_4 4_history-_
Whether passed internal testing?
(condition www.isttrty___
34

---

Table A.2: Performance MATH500 and AIME under different numbers of collaborating agents. Higher is better for Accuracy and Reuse Rate.
# Agents
Dataset
Model
Method
2
Acc. (%) Original
41.6
39.6
42.6
Acc. (%) KVCOMM
38.0
38.6
44.2
Llama-3.1
Reuse (%) Original
0
0
Reuse Rate (%) VCOMM
59.4
40.9
30.7
MATH500
Acc. (%) Original
51.4
49.6
42.8
Acc. (%) KVCOMM
50.8
50.8
45.8
Deepseek-Quer Qwen
Reuse Rate (%) Original
0
0
Reuse Rate (%) KVCOMM
76.7
60.4
45.3
Acc. (%) Original
19.2
17.5
17.5
Acc. (%) K KVCOMM
11.7
10.8
8.3
AIME
Deepseek-Ques Qwen
Reuse Rate (%) Original
0
0
Reuse Rate (%) KVCOMM
71.3
74.6
6.4
More Experimental Analysis
In this section, we provide more analysis on K KVCOMM, including the effectiveness of the pro- posed matching criterion, the effectiveness of the norm-based approximation method, and more visualizations for understanding the effectiveness and efficiency of K KVCOMM.
6.4.1 Evaluations on Harder Reasoning Benchmarks
We further provide evaluation results on the MATH500 [14] and AIME [6] benchmarks to examine the effectiveness of KVCOMM under harder reasoning tasks. For MATH500, we tested both Llama3. 8B-instruct [11] (Llama-3.1) and Deepseek- R Distill-Qween-ee 7B [12] (Deepseek-
As shown in Table A.2, K CVCMM achieves superior or comparable performance dense computation.
Compared with relatively easy math reasoning tasks such as GSM8K, the reuse rate indeed drops more rapidly the number of agents increases. Nevertheless, the accuracy of KVCOMM remains competitive or even surpasses the baseline, indicating that referencing anchors' information can consistently help. A notable insight from Deepseek-Qwen results is that KVCOMM achieves both higher reuse rate and a higher accuracy on ceasoning-oriented models.
We also evaluate on the more challenging AIME benchmark with Deepseek- As shown in Table A.2, KVCOMM still maintains comparable accuracy to dense prefill while keeping the reuse rate above 70%. The accuracy drop here is mainly due to the token length constraint during decoding: since CCOMM accelerates prefill at the cost of extra memory, the effective decoding length per agent is reduced.
6.4.2 Analysis on the Matching Criterion
In this section, further examine the ef- fectiveness of our proposed anchor match- ing criterion, evaluating the lg bose- matching criterion. Table A.3 compares two distinct matching criteria for anchor prediction and the approximation of place- holder samples on the MMLU benchmark in the four-agent scenario. Recognizing that the length match is fundamental requirement for effective token-level approximation within placeholder samples, we explicitly evaluate the contribution of the complementary distance-based matching criterion HalA- observed that omitting the distance-based criterion can increase anchor reuse rates; however, this is accompanied by marked performance degradation, indicating
Table A.3: Ablation study of the matching criterion on MMLU under four-agent setting. (Model: Llama-3.1)
Method
Acc (%)
Reuse Rate (%)
C.
62.1
93.3
Lop & HOLA
68.0
70.1
35

---

Table A.5: Simulated smft latency (ms) with different anchor counts and sequence lengths.
#Anchor
1024-tokens
2048-tokens
4096-tokens
0.894
1.719
3,552
10
1.773
3.576
7.128
2.620
5,332
10.766
20
3.933
7.859
15.624
25
4.435
9.614
113
13:
Mest1
399 200 300 Ale 560 zie
see sés Loe
ide 360 ede sle cãe zée
sis sole
Me a60 350 ade 160 nil eee 109 taker
Ie 200 3/ sde 102 600 vie eée 30030000
(a) 5 anchors
(b) 10 anchors
(c) 15 anchors
(d) 20 anchors
Figure A.1: Memory cost (GB) across IN/OUT for different anchor counts.
that mere length matching is insufficient for optimal anchor utilization. Conversely, integrating the embedding -dittance criterion achieves an advantageous balance, maintaining high performance while providing effective anchor reuse. These results highlight the necessity of comprehensive match- ing criterion that incorporates both structural alignment (length matching) and semantic similarity sabbedding-distane (embedding distance criterion) to ensure both efficiency and task accuracy in VCOMM.
6.4.3 Analysis on the Approximation Method
We further investigate the effectiveness of different KV-cache offset approximation methods on HumanEval under four-agent setting (Table A.4). Two comparison meth- ods are introduced: one using the nearest anchor's KV-cache offset based the & norm, and another employing cosine simi- larity with so weffmma-bbsee anchor weight- ing. Results indicate the cosine-siminn similaarti--- based method achieves performance com- parable to our l2 norm-based method with slightly higher reuse rates. However, the nearest-reusing approach significantly deteriorates performance due to the error induced by the distance between the sample and the nearest anchor. Thus, soft aggregation anchors proves to be an effective approximation method.
Table A.4: Performance of different approximation
methods for KV-cache offsets on HumanEval under four-agent setting. (Model: Qwen-Coder-2255-777 Base-
line Acc: 84.45%)
Method
Acc (%)
Reuse Rate (%)
Nearest
47.20
78.9
Cosine Similarity
83.23
82.5
Ours (E2 Norm)
83.23
81.1
6.4.4 Analysis of the Overhead in Long-context Anchor Matching
In KVCOM, so softmax is operated along the anchor number dimension on the negative fo-m distance between the placeholder sample and each anchor, which reduces the Key/Value tensor into the shape of [m, 1, seq_len, 1] (m:: anchor count, seq_len: sequence length). The latency of softmax thus scales with both parameters. We quantify its latency overhead in Table A.5. Here the shape of each KV-cache tensor is [32, 8, seq len, 128]. We observe that latency remains reasonable (~-18 ms with 25 anchors and 4096 tokens per anchor).
These results are from a simulation without competing system workloads. In real multi-agent long- context scenarios, the latency also includes offloading KV-caches to the CPU. For example, as shown in Table A.6 (using Llama3.1-8B SB-instruct on the MMLU benchmark), the average softmax latency is 100+ ms when all anchors are offloaded to CPU, while total offloading per agent can reach 1260+ ms for 4K-token contexts. This indicates that the main overhead arises from data movement in the softmax operation for the long-context KV. Such communication overhead can be mitigated with systematic optimization (c.g., pipelining), which is orthogonal to the KVCOMM mechanism.
36

---

Table A.6: Latency and memory cost of 4K-tokens anchor matching using Llama3.1-8P 1-88B-inttructttt the MMLU benchmark.
Metrics
#Agent=3
#Agent=4
Softmax (ms)
104
122
Offloading (ms)
1260
1300
Acc. (%)
66.7
68.0
Reuse Rate (%)
49.7
51.0
Memory Cost (GB)
68.5
95.1
User Question
What Decologua'? Optin A. The Exodus, Optin B: The Coverant, Option C: The Tan Commandmenta Option D: The creation
Kesowellagg Euppert
Kriaaledge Expert
tge car venige ged penia (phiccku Labbeeeerd Eahie lewth 4/6
Cooorani @libbww Blbed, dag.aaarrrr
Athouie miik 4lt will KCh Go (O
guiiiccal
Wike
Wel Searcheet
Wibi Searshorr
Aeaysle Diuniiue tawe
C. Conmrei
with
Latk
addi with Pocercal
poia stamee
Sie ac ciaats decinga..
pake
deveccm glonty (042 deowm, an unthe murvee
Lncingee
planic
slack
Cylli
Cytti
Locnent Aeproom pordd righigh peiroar
The Ccvelgge deaccaa aioccs
Deart
of dours rocortt
Lao anveraaiin The Loid ploit or
Fie Eecarpge paripiaa piiin ty
Inbart duc pablet
phu cos poccali d aur antrrttnn comdie
Lupl
Fiuld Nalee
Haller
Dossira Dho-Cache Seuss)
Kay Rocaton
Key Notaton Profic KV Offacty Placcfo KV Otherts
Generation Comparison with different alignment combinations. Left: Original response of each agent without cache reuse. Middle: Responses generated by cache reuse, associated with aligning the position encoding of key caches and offsetting the prefix segments' caches. Right: Responses generated by combining all three alignment processes.
Figure A.2:
6.4.5 Analysis on the Memory Cost of KVCOMM
We also report memory overhead under different agentic configurations, varying input length (prefix), output length (response), and the number of anchors per placeholder. As shown in Figure A.1 (three-agent setting), the memory cost increases with longer IN/OUT sequences and with larger anchor counts, reflecting the storage required for mechor-specifi KV-cache deviations. Empirically. we observe that these deviation tensors are quite sparse across anchors, with on average about 50% of elements having absolute values smaller than 10-1, This indicates substantial headroom for lossless compression of anchors, which will be one of the future work to support longer contexts without sacrificing prefill speedups.
6.4.6 Visualization of Responses Generated by Different Combinations of Alignment Strategies
We further visualize the detailed response of each agent using different cache alignment strategies. As shown in Figure A.2, it can be observed that although the middle setting (Key Rotation + Prefix KV Offsets) eventually outputs the correct option "C", its reasoning chain is clearly degraded: the agents omit formal definition of "Decalogue", repeat the keywords from the Knowledge Expert agent, and make analysis on agents that do not exist, resulting a logically fragmented discourse. This "right answer for the wrong reasons" phenomenon underscores that two-level alignment alone
37

---

pertion
(a)
(b)
(c)
Figure A.3: The distribution of anchors in four-agent system on GSM8K, with cach agent fully connected to its predecessors within single conversational turn. (a) Relationship between anchor activation counts and token lengths across placeholders during inference. (b) Histogram of anchor activation counts across all placcholders.
(c) Histogram anchor token lengths across placeholders.
Table A.7: Differences between placeholder offset and prefix offset.
Aspect
Placeholder offset
Prefix offset
Definition
KV-cache offset for externally injected text (users, agents, tools, ctc.); its base K KV-cache is not preset.
KV-cache offset for predefined prompt text (system prompt, placeholder conjunctions, suffix).
Dependence
Arises mainly from changes the prefix context.
Triggered by changes the preceding placeholder (injected content).
Variance
Shows high variance across samples and
Typically more stable, since base KV-caches computed under fixed system prompt and are less influenced by preceding placcholders.
contexts.
still disrupts the causal dependency between evidence, inference, and conclusion; the combination of all three alignment processes is therefore essential for achieving a coherent explanation path that is comparable the original response.
6.4.7 Visualization of Anchor Distribution
As illustrated in Figure A.3, we visualize the anchor distribution for a four-agent system evaluated on the GSMSK dataset. Figure A.3a demonstrates clear positive correlation between anchor's token length and its activation frequency across conversational placeholders seser_question and three agent_ currret
response placeholders). Notably, the condition_3 3_cureen anchors, which are the execution results of the generated Python codes, exhibit a distinctive bimodal distribution: one group is extremely short but heavily reused (less than 10 tokens, activated over 1000 times), while another spans longer lengths with relatively sparse activations. Figure A.3b and A.3c further emphasize this long-tailed phenomenon, showing that the majority of anchors have activation counts under 100 and token lengths shorter than 200. This skewed distribution justifies our cache management strategy, prioritizing memory allocation for grggffreyeecy anchors and dynamically pruning infrequently reused ones.
6.4.8 Visualization of Difference between Prefix and Placeholder Offset Distributions
To clarify the difference between prefix and placeholder offsets, we describe them from three aspects, which are shown in Table A.7.
We further visualize the offset distributions of these two types, which is experimented in a fully- connected four-agent setting on the MMLU dataset. We tested the offset variance among ten different samples, and present them in Figure A.4, A.5, A.6, A.7, A.8, A.9, A. and A. It can be observed that while the offset of the prefix KV often larger than the placeholder one, its variance is relatively smaller than the placeholder KV, especially in deep layers (e.g., Figure A.5). The reason is that: During the precomputation of prefix KV-caches, the subsequent prefix segments are primarily correlated with the first prefix segment, typically containing system prompts; thus, their base K KV-ches are relatively stable.
6.4.9 Visualization of Distance between Approximated and Real Offsets
Figure A.12 further compares various approximation strategies HumanEval using Qwen-Coder 2.5-7B for the four-agent scenario, focusing on the similarity and error between approximated and real KV-caches. Overall, our proposed ba-ser (L2NORM) approximation exhibits
38

---

user quamdicn placenciter
user quantion prefi:
Layer 0
Layer
Layer
Kay offect.
Layer 6
Layar
Layer 10
1.90
Koy onst
Layer 12
Layer
Layer 17
1.90
Koy orber
Mall
0.00
Layer 19
Layer
Layer
1.90
Key ooyott
0.00
Layer
Layer
Layer
1.90
Koy orbee
Tomes
Ba
0.00
120
150
Hidden Statts Dim Index
Hodden Statts Dim Index
Hidden States Dim Index
Figure A.4: Key cache offset distributions of the first agent's placeholder and prefix segments in four-agent setting on the ten samples from the MMLU dataset.
39

---

user quambicn placencider user quantio prefi:
Layer 0
Layer
Layer
Veb. Wale affe
Haght
Layer 6
Layer
Layer 10
Value altet
Layer 12
Layer
Layer 17
0.47
Value offee
0.00
Layer 19
Layer
Layer
0.47
Value albett
Ay h Map Aayie
/
0.00
Layer
Layer
Layer
0.47
Value Vedte offict
0.00
Hisden Stattss Dim Index
Hodden States Dim Index
Hidden States Dim Index
Figure A.5: Value cache offset distributions of the first agent's placeholder and prefix segments in four-agent setting on the ten samples from the MMLU dataset.
40

---

agent current: placholder
conditon J. current placeholde card ton current prets
user queeion: pacehalder
agent current: pretx:
user queetion: praftc
Layer
Layer 2
Layer 4
0.00
Key atteet
198
139
Layer
Layer
Layer 10
0.00
tay dhot
1.90
-..00
Layer
Layer
Layer 17
0.00
Kay offert
1:99
Layer
Layer
Layer 23
0.00
Key plllt
-.400
-0.00
Layer
Layer
Layer
1.47
Key offeee
1.99
3..00
120
HHoden States
Index
Hidden States Dim Index
Hidden States Dkm Index
Figure A.6: Key cache offset distributions of the second agent's placeholder and prefix segments in four-agent setting on the ten samples from the MMLU dataset.
41

---

agent current: placholder
conditon current placeholder cand ton, 1. current prets
user queeiinn pacehalder
agent current: pretx:
user queetion: praftc
Layer
Layer 2
Layer 4
0.49
Veb Web.e effeee
-2.00
0.52
Layer
Layer
Layer 10
0.49
Value
-2.00
0.52
Layer
Layer
Layer 17
0.00
0.49
Value offiet
-2.00
Layer
Layer
Layer
0.00
Value oltvet
0.000
-0.00
Layer
Layer
Layer
0.29
2.00
Valre cffice
3.00
120
HHoden States Dim Index
Hidden States Dim Index
Hidden Statts Dkm Index
Figure A.7: Value cache offset distributions of the second agent's placeholder and prefix segments in a four-agent setting on the ten samples from the MMLU dataset.
42

---

apem 3 cummet placeholder apem. curment prefte
agert 2, curentt placnslder
--- user Cuaction placeholder
agert 2. ourentt preti
-2- wwu..uu..uaai.. cussion: prets
Layer
Layer
Layer 4
0.00
Key atteet
9.00
2.20
Layer
Layer
Layer 10
tay oibet
2.1
0.00
Layer
Layer
Layer 17
2.06
Kay offert
2.61
0.00
Layer
Layer
Layer 23
2.06
Bey ailut
2.11
0.68
-0.00
Layer
Layer
Layer
2.06
Key offeee
2.01
0.00
3..00
120
HHoden States Dim Index
Hidden States Dim Index
Hidden States Dkm Index
Figure A.8: Key cache offiset distributions of the third agent's placcholder and prefix segments in four-agent setting on the ten samples from the MMLU dataset.
43

---

apem 3 cummet placeholder apem. current prefte
agert 2, ourentt placnslder
--- user Cuaction placeholde
agart 2. ourent preti
-2- uwer. cussion: prets
Layer
Layer
Layer 4
Webe effeee
0.38
0.36
Layer
Layer
Layer 10
0.00
Valueelhuat
0.30
Layer
Layer
Layer 17
0.00
Value offiet
0.38
Layer
Layer
Layer
2.00
Value altvet
0.36
-0.00
Layer
Layer
Layer
0.33
0.00
Vakre office
-2.00
3..00
120
HHoddn States Index
Hidden States Dim Index
Hidden Statts Dkm Index
Figure A.9: Value cache offset distributions of the third agent's placeholder and prefix segments in four-agent setting on the ten samples from the MMLU dataset.
44

---

agent, curment aceholee agent, cumment:ppalt. pretix
spent curret: placeholder
agent J. cumment placeno
ther quastions accholder ther quastions pretx:
spent 2. current: prati
agent curments pritis
Layer 0
Layer
Layer
9.89
K0y offect
P:80
2.89
Layer 6
Layer
Layer 10
1.27
9.89
Kaj on5e
9:88
2.89
Layer 12
Layer
Layer 17
1.37
9:89
Key ofbeet
8:80
9:89
Layer 19
Layer
Layer
1.37
9:89
Key eefee
8:88
2:89
0.00
Layer
Layer
Layer
1.37
9:69
Koy arboe
P.68
8:89
0.00
Hidden Statts Dim Index
Hodden States Dim Index
Hidden States Dim Index
Figure A.10: Key cache offset distributions of the fourth agent's placeholder and prefix segments in four-agent setting on the ten samples from the MMLU dataset.
45

---

agent 1. curnent: acceaater agent curmett prott
spent curret: plecehoider
agent J. cumment placeno der
ther quasition accholeer toer quen cenittonppee preto:
spent 2. current prator
agent currents pretis
Layer 0
Layer
Layer
0.32
Vel efive
8:48
8:90
Layer 6
Layer
Layer 10
0.20
8.00
Value altut
8:88
8:98
Layer 12
Layer
Layer 17
6.52
Vakue offuet
8:08
8:99
Layer 19
Layer
Layer
6.52
Value oltvet
8:08
8:98
0.00
Layer
Layer
Layer
0.30
0.00
6.52
Value offict
8:88
8:99
0.00
Hidden Stattss Dim Index
Hodden States Dim Index
Hidden States Dim Index
Figure A.11: Value cache offset distributions of the fourth agent's placeholder and prefix segments in a four-agent setting on the ten samples from the MMLU dataset.
46

---

LONITAA
2.
25
20
25
05
30
15
20
(a)
0040
3,003
0 0005
OTac Ere Dear G.mmm
0000
UPAA S0L1 MAA
0000
0005
0000
0.000
00000
(c)
(d)
Figure A.12: Similarity and offset error analysis of approximated KV-caches versus real KV-eaches across layers of Swee--Code-225-7BB on HumanEval under four-agent setting. (a) Cosine similarity distributions between approximated and real key caches. (b) Cosine similarity distributions between approximated and real value caches. (c) Mcan norm eтrоr distributions between approximated and real key caches. (d) Mean & norm error distributions between approximated and real value caches. Labels "COSINE-AO" to "COSINE-A3" denote desiie-simiilarit--basee approximation; "L2NORM-AO" to "L2NORM-AS" denote our ly-norm-based approximation; "NEAREST-AO" to "NEAREST-A3" indicate nearest-anchor sampling approximation; "PLAIN- AO" to "PLAIN-A3" represent unaligned baseline reuse.
consistently high cosine similarities (approximately 0.92 for keys and 0.95 for values) comparable to the cosine-based approach, while maintaining minimal offset errors across all layers. Unlike simpler methods such as ncarest-reusing- which suffers substantial deviations in deeper layers (with the mean offset eπrоr exceeding 0.003 beyond layer 25) method robustly leverages weighted aggregation of multiple similar anchors to effectively estimate the target K V-caches. Additionally, the plain reuse baseline demonstrates severe similarity degradation (below 0.8 cosine similarity) and significantly elevated offset errors (above 0.004), confirming the critical importance our fine-grained anchor alignment strategies, especially in deeper transformer layers where mismatch errors tend to accumulate.
47


Reviews:
[{"review_id": "lDOvu0Cvsu", "reviewer": "Reviewer_2PaM", "summary": "The authors propose a method for reusing (prefilling) cache in multi-agent systems called KVComm.  The method is similar to CacheBlend, but is more specific to the multi-agent setting and uses 'anchors' dictated by changes in the custom prefixes/prompts for the different agents to inform where the KV cache should be shared.  Results on several benchmarks can show some degradation at large prefill rates, but the speedup for the method is large and the benefits increase with the number of agents.", "questions": "1. Could you be more specific in your claims on novelty?\n2. Could you clarify the domain you're claiming your method should work well in? (specifically, do you intend for it to only be used in single-GPU, same-architecture situations?)\n3. As someone not very familiar with the field, I'd love more intuition behind the anchors of your method and the closest related thing in multi-agent or other areas in the literature (response editing, speculative decoding)\n4. It would be great to see experiments that line up with CacheBlend claims, and clear explanations for why that method is failing so badly on HumanEval / other times that it does.  Is the method dynamically recomputing the cache at a ~15% rate? Is this the vllm version?\n5. Please bold the 'best' results rather than your results in Table 1 (e.g., CacheBlend is better on MMLU with 2 agents, but it's not obvious from the current table formatting)", "limitations": "The authors should limit the scope of their claims to single-GPU, same-architecture settings very clearly and acknowledge possible hardware communication bottleneck impact on performance in different settings, since this drives a lot of other KV cache works' considerations.", "rating": 4, "confidence": 2, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 2, "clarity": 2, "significance": 3, "originality": 2, "strengths_and_weaknesses": "## Strengths\n* The problem itself is an important one - how do you save computation when sending many of the same tokens through agents of the same model?\n* The motivation for the method and general intuition is clearly presented\n* There are a lot of interesting results / graphs presented.  If I knew the subfield better, I'd love to dig into those at a detailed level\n\n## Weaknesses / Concerns\n* Your positioning of KVComm within existing methods wasn’t clear to me:\n  * You claim: “the first training-free, prompt-adaptive cache-sharing framework for efficient prefilling of multi-agent systems” <- why doesn't CacheBlend fit this?  Could you be more clear in your claim, since methods more general than multi-agent systems exist to do this? (The claim may be that it is more efficient specifically within multi-agent systems by leveraging X\"\n  * Is the method leveraging priors around prompts in a special way?\n  * What does “context-aware” mean?\n  * You mention and emphasize the assumption that other methods make that every agent runs the same model architecture.  It seems your method needs this assumption as well – is there a reason you emphasized it?  Could you make it clear that your method is similarly constrained (and taking advantage of this structure)\n  * Given the drastic difference between your method and CacheBlend on HumanEval, I think you should (1) very clearly explain what this difference is due to, and (2) make it part of your major differentiator/impact (what other problems is this likely to have a big impact on?  Where is CacheBlend being used that it doesn’t perform well?)\n  * Building up an anchor pool feels very similar to building up a database of past tokens themselves, i.e., speculative decoding methods – could you compare to them, at least in description / scenario?\n\n* I’m concerned that the results don’t line up with the CacheBlend results and know the authors said they didn’t optimize the prompts for the open source models.  I think it’s important to rerun these experiments to match CacheBlend as best you can so the numbers are comparable between papers.\n\n* Hardware/communications are usually a bottleneck in these situations.  It seems you don't address it in your paper - could you add clarification on:\n  * The fact that this method is constrained to single-GPU settings (as far as I understand it)? (and homogenous networks, which is a common assumption)\n  * Whether you needed to do anything special to store and retrieve the cache in an efficient way that was specific to this method?\n\n## Smaller suggestions\n* Cite “prior research” in “2 KV-Cache Sharing Scenario. Prior research has identified…”.\n* In the same section, point out that your method is ‘Selective Recomputation’ (It’s kind of weird to make that #2 out of 4 – put it at the beginning or the end?  Or explain why you’re working on that one in particular in this setting and make it into a separate paragraph, perhaps? \n* `Tab.` for table is hard to read.  Preference for `Table`", "was_revised": true, "final_justification": "The paper presents a method that addresses the important setting of efficient inference in multi-agent settings.  The authors clarified many points around limitations and differentiated novelty I had not seen clearly stated in the first edition of the paper.  Given that most of my concerns were around these areas, I have bumped my score by 1, and with sufficient clarity in the final version would advocate for an accept.", "rebuttal": "Thank the reviewer for taking the time to review and providing the constructive and valuable feedback. Below, we present our point-by-point responses to each comment.\n\n## Q1: More Clarification of KVComm's Contribution\n\nKVComm’s key contributions are threefold:\n- **Problem Formulation (Novel Motivation).** Prior KV-cache reuse methods (e.g., CacheBlend) target **single-agent** settings such as RAG, where multiple retrieved chunks are concatenated into one prompt and KV-caches are precomputed without external context. In **multi-agent systems**, however, each agent’s response is repeatedly consumed by successor agents. **The resulting KV-cache unavoidably encodes its original prefix context** due to autoregressive attention. As evidenced in our work, a naive adoption of previous methods in this case can fail in filtering the previous contextual information in the KV-cache, resulting in performance degradation. **We identify and address this multi-context redundancy as a distinct challenge for efficient prefilling in the multi-agent scenario, which to our knowledge, has not been covered by prior work**.\n\n- **Empirical Insight (Context-aware KV-cache Offset Proximity).** We observe that **for two similar tokens whose KV-caches are computed under the same prefix, when the prefix changes, their KV-cache deviations remain similar**. Unlike RAG-focused findings that KV-caches are largely prefix-robust, our measurements show this robustness breaks down in general multi-context (multi-agent) scenarios (see Figure 1 and Table 1 in the main text).\n\n- **New Reuse Paradigm (Prompt-Adaptive).** Instead of token-/layer-level selective recomputation (e.g., DroidSpeak, CacheBlend), KVComm performs **prompt-adaptive selection at the segment level**: a shared text is either fully reused or fully recomputed, based on its embedding similarity to stored anchors. We further design an **efficient KV-cache storage/retrieval system** that supports fast anchor lookup and long-context multi-agent pipelines.\n\nTherefore, based on the above, **KVComm is, to our knowledge, the first training-free, prompt-adaptive cache-sharing framework for efficient prefilling in multi-agent systems.**\n\n## Q2: Priors Leveraged by KVComm\n\nThe priors around prompts we use are consistent to the baseline multi-agent framework design. That is, KVComm follows the **baseline multi-agent dependency graph** to define each agent’s placeholder set and prefix segment set. IDs (for placeholders/prefix segments) are derived directly from the dependency graph. Details are in Appendix 6.2.2 and 6.3.2 of the supplementary material.\n\n## Q3: Definition of Context-Aware Offset\n\nIn Figure 2, the **context-aware offset is a KV-cache deviation induced by the (changed) prefix context**. When the prefix changes, the required offset to align a cached segment’s KV-cache with the new context changes accordingly, hence \"context-aware\".\n\n## Q4: Domains / Assumptions for KVComm\n\nKV-cache reuse requires **identical model architecture** (tokenizer, computation graph, hidden size, #heads, #layers, etc.); otherwise KV-cache shapes/token spaces mismatch and reuse becomes infeasible without adaptation. KVComm also assumes **RoPE position encoding**, a common choice in modern LLMs, enabling explicit per-layer positional alignment across varying context lengths.\n\n## Q5: Explanation of Performance Gap between KVComm and CacheBlend on HumanEval\n\nHumanEval agents produce code with many syntax separators (e.g., `. , ; !`), which induce **diverse and prefix-sensitive KV-cache distributions**. With CacheBlend’s 20% recomputation, many sensitive tokens remain stale, degrading successor agents’ understanding. KVComm instead globally applies the best anchor-based offset to approximate these tokens’ KV-cache under the new context. When we increase CacheBlend’s recomputation ratio to 80%, as shown in the following table, performance significantly increases but remains inferior to KVComm (>81% pass@1):\n\n|Reuse Rate|#Agent=2|#Agent=3|#Agent=4|\n|:-:|:-:|:-:|:-:|\n|20%|72.05|68.94|67.70|\n|80%|31.06|21.12|30.43|\n\nThis suggests **coding tasks require near-full KV-cache updates** under context shifts, aligning with KVComm’s prompt-adaptive strategy.\n\n## Q6: Comparison with Speculative Decoding\n\nAt a high level, building up the anchor pool in KVComm might resemble building up a database of past tokens, such as those employed in speculative decoding methods. However, the fundamental concepts and underlying scenarios differ notably between KVComm and speculative decoding methods. Specifically,\n\n- **KVComm** accelerates **prefilling** across **multiple agents** by **reusing/aligning existing KV-caches** via anchor-based offsets. Each anchor in KVComm is essentially a cached representation of previously encountered segments along with their measured offsets from different prefix contexts. This allows us to approximate and reuse KV-caches dynamically at runtime.\n- **Speculative decoding** accelerates **decoding** in typically **single-agent** generation by **proposing and validating future tokens**. In speculative decoding, the model speculatively predicts based on previously generated tokens or precomputed continuations. If the prediction aligns closely with the actual model's output tokens, it accelerates decoding by accepting the speculative output. Otherwise, it discards the speculative output and recomputes fully.\n\n## Q7: Fairness of Comparison with CacheBlend\nAfter carefully double examining the CacheBlend paper and our work, we would like to clarify the fundamental differences in application scenario and the faithfulness to CacheBlend in our implementation, which justify that **the experimental comparisonn with CacheBlend is as fair and objective as best we can**.\n\n**Scenario Difference:**\n- CacheBlend is designed/evaluated for **single-agent RAG**, reusing **non-prefix** chunks with small token-level recomputation. Consequently, the experiments reported in CacheBlend are specially designed for the RAG task in a single-agent setting, **which have no experimental results for multi-agent systems as we did.**\n- Our setting is **multi-agent**, where responses propagate across agents and prefixes shift frequently. **Our scenario involves general coordination among multiple agents rather than single-agent RAG contexts.**\n\n**Faithful Implementation of CacheBlend:**\n- We implemented CacheBlend per its paper and official code. To keep the similar reusing rate, we increase the recomputation rate to 20%, which theoretically further improves the performance of CacheBlend.\n\n**Comparability of Experimental Results:**\n- Our experiments intentionally use prompts representative of multi-agent coordination scenarios. Optimizing prompts specifically to match CacheBlend’s single-agent RAG scenario would not only distort the intended application scenario of our approach but also lose the realistic aspects and complexity of multi-agent inference.\n- As reported in Table 1 of the main text, CacheBlend performs reasonably on **RAG-like** tasks (e.g., MMLU with RAG agents; see Appendix 6.3.2), which align well with the claim of CacheBlend, but struggles on **math/coding** due to the reason analyzed in **Q5**. **Therefore, we have conducted as fair comparsion with CacheBlend as best we can.**\n\n\n## Q8: Hardware / Communication Efficiency\n\nKVComm is **not** restricted to single-GPU. The two assumptions (identical architecture; RoPE) are **topology-agnostic**. We currently report single-GPU results due to framework API limitations that standard open-source inference frameworks (e.g., HuggingFace Transformers, vLLM, SGLang) focus mainly on prefix-based caching and lack stable public APIs for fine-grained, per-layer KV-cache injection across devices.\n\n**Solution to Distributed Communication Bottleneck:**\n\nRegarding the potential distributed communication bottleneck, KVComm can deliberately avoid moving entire KV-caches across GPUs/nodes. Instead, **we can synchronize only small-sized metadata (anchor indices and offset statistics) and perform KV-cache updates locally, which would remarkably reduce communication overhead**.\n\nWe are currently integrating KVComm into distributed serving frameworks like vLLM and SGLang by adding minimal KV-cache read/write hooks to fully support efficient multi-agent systems in multi-GPU and distributed environments.\n\n**Cache Management Strategy:**\n\nRegarding the cache management strategy in KVComm, we designed **two three-level cache managers** to achieve efficient writing and retrieving anchors' KV-caches and the current shared KV-caches, respectively. \n\n|Anchor Manager|1st level|2nd level|3rd level|\n|-|-|-|-|\n|Indices|placeholder id, e.g., `user_question`, |anchor index, e.g., anchor[0]|agent id / embedding, e.g., `agent_1_ph_delta`|\n|Values|Anchor list|dict of different KV-cache offset in different agents and the anchor embedding tensor|KV-cache/embedding tensor|\n\n|Shared KV-cache Manager|1st level|2nd level|3rd level|\n|-|-|-|-|\n|Indices|agent id / user_input, e.g., `agent_1`|placeholder_id, e.g., `response`|turn index, e.g., response[-1]|\n|Values|dict of different agents' response KV-caches and outside input KV-caches|dict of response and prefix KV-caches|KV-cache list|\n\nBased on these two cache managers, each agent can quickly retrieve their intended KV-caches and also store their generated KV-caches. Meanwhile, we conduct the process of KV-cache storage and retrieval in an asynchronous manner, thus further improving the efficiency. For clearer understanding, we will opensource the code repository once accepted.\n\n## Q9: Paper Structure & Formatting\n\nWe will revise Section 2.2 to clarify positioning among KV-acceleration methods, adjust `Table` formats to avoid misinterpretation, and verify all references' formats."}, {"review_id": "oAagEL4npm", "reviewer": "Reviewer_TNJW", "summary": "This paper addresses inefficient redundant KV-cache computation in multi-agent LLM systems caused by offset variance, where agents sharing overlapping contexts reprocess identical content due to divergent prefix lengths and dynamic placeholders, preventing static KV-cache reuse. The authors propose KVComm, a training-free framework enabling dynamic KV-cache reuse. KVComm employs an online anchor pool that tracks cache deviations under varying prefixes to estimate and align cache offsets for new contexts.", "questions": "1.Proposition 1 assumes RoPE-based positional encoding. How does KVComm handle non-RoPE models? Experiments on such architectures would strengthen claims.\n\n2.The prefix segmentation requires rigid templates (Eq.1), making free-form agent interactions (e.g., unstructured debates) unsupported. The framework fails when placeholders aren't predefined—shown by omitting conversational benchmarks. \n\n3.Tests use ≤1K tokens (Table 3). How does reuse rate degrade for >4K contexts where cumulative positional deviations amplify?", "limitations": "If the method only applies to strictly identical LLMs and rigidly fixed prefixes, and cannot accommodate free-form conversational interactions, the authors may wish to explicitly acknowledge these constraints in the limitations section.", "rating": 4, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 3, "significance": 3, "originality": 3, "strengths_and_weaknesses": "Strengths: The authors effectively combine theoretical analysis (Propositions 1 and 2) with extensive experiments across diverse benchmarks (MMLU, GSM8K, HumanEval). They demonstrate significant results, including a 13.7× TTFT speedup and over 70% reuse rate under realistic multi-agent workloads. Ablation studies (Table 5) validate the necessity of each component. The work further includes comprehensive robustness evaluations, assessing sensitivity to hyperparameters (Table 6), request ordering (Table 4), and context length (Table 3).\n\nweakness:\n1.The paper assumes identical LLMs for all agents, but real-world multi-agent systems often leverage heterogeneous models (e.g., domain-specific LLMs). This severely limits practical applicability.\n2.Anchor matching relies on real-time embedding similarity calculations (Eq.5), yet latency overhead for long-context inputs is unmeasured. Given the softmax over anchors, this could negate prefilling speedup for large V. Please quantify this cost.\n3.While DroidSpeak solves cross-model KV-sharing, it's only cited without comparison. This omission undermines the claimed novelty.", "was_revised": true, "final_justification": "Most concerns are resolved. I maintain my original positive score.", "rebuttal": "Thank the reviewer for taking the time to review our work and providing positive, constructive and valuable feedback. Below, we present our point-by-point responses to each comment.\n\n## Q1: Application in Heterogenous Multi-agent Systems\n\nAs for the heterogenous systems, we can categorize them into two types for detailed discussion.\n\nFor agents consisting of\n- **Identical Architecuture but Different Model Weights:** DroidSpeak has discovered the KV-cache reuse potential for this kind of KV-cache communication, where the base model and domain-specific finetuned model completely share the same prompt. Building on this, we believe that if the clustering characteristics of tokens in the finetuned model remain sufficiently similar to the base model, KVComm could facilitate KV-cache sharing for these cases. However, this may require further technical innovation and thus remains a promising direction for future research.\n- **Distinct Architectures**: In this case, it is impossible to employ KVComm as the reviewer noted.\n\nIn summary, KVComm is **directly applicable for groups of homogeneous agents and holds promise for agents with identical architectures but different weights**, pending further exploration. \n\n## Q2: Softmax Overhead in Long-context Anchor matching\n\nIn KVComm, the softmax is operated along the anchor number dimension on the negative $\\ell_2$-norm distance between the placeholder sample and each anchor, which reduces the Key/Value tensor into the shape of [$m$, 1, 1, seq_len, 1] ($m$: anchor count, seq_len: sequence length).  The latency of softmax thus scales with both parameters. And we quantify its latency overhead in Table 1. Here the shape of each KV-cache tensor is [32, 8, seq_len, 128]. We can observe that latency remains reasonable ($\\sim$ 18 ms with 25 anchors and 4096 tokens per anchor).\n\nTable 1. Simulated softmax latency (ms) with different anchor counts and sequence length.\n\n|#Anchor|1024|2048|4096|\n|-|-|-|-|\n|5|0.894|1.719|3.552|\n|10|1.773|3.576|7.128|\n|15|2.620|5.332|10.766|\n|20|3.933|7.859|15.624|\n|25|4.435|9.614|18.113|\n\nThese results are from a simulation without competing system workloads. In real multi-agent long-context scenarios, the latency also includes offloading KV-caches to CPU. For example, as shown in Table 2 (using Llama3.1-8B-instruct on the MMLU benchmark), the average softmax latency is 100+ ms when all anchors are offloaded to CPU, while total offloading per agent can reach 1260+ ms for 4K-token contexts. This indicates that **the main overhead arises from data movement, not from the softmax operation itself**. Such communication overhead is a well-known issue in long-context inference, and **can be mitigated with systematic optimization (e.g., pipelining), which is orthogonal to the KVComm mechanism**. To achieve perfect acceleration of long-context KV-cache reuse, *we are currently committed to integrate KVComm into the modern LLM serving framework such as vLLM and SGLang, which have well optimized these systematic issues yet still lack support for fine-grained, per-layer KV-cache injection across devices*.\n\nTable 2. Latency and Memory Cost of 4K-tokens Anchor Matching using Llama3.1-8B-instruct on the MMLU benchmark.\n\n|Metrics|#Agent=3|#Agent=4|\n|-|-|-|\n|Softmax (ms)|104|122|\n|Offloading (ms)|1260|1300|\n|Accuracy (%)|66.7|68.0|\n|Reuse Rate (%)|49.7|51.0|\n|Memory Cost (GB)|68.5|95.1|\n\n## Q3: Experimental Comparisons with DroidSpeak\n\nThe reason we did not originally compare directly to DroidSpeak is due to fundamental differences in application scenario:\n\n- **DroidSpeak** exploits cross-model KV-cache reuse when both base and finetuned models process **an identical prompt, so context changes are not involved**. \n- **KVComm** targets at the potential of cross-model KV-cache reuse **when only a shared segment is reused under different prefix contexts—a more general and challenging setting for multi-agent systems.**\n\nTherefore, although both designed based on KV-cache reuse, the two methods aim at solving two different problems in multi-agent systems.\n\nTo clarify, we implemented DroidSpeak in our framework and empirically compared it against KVComm, holding DroidSpeak’s reuse rate at 80% for relative fairness. It can be observed in Table 3 that DroidSpeak’s performance drops significantly in our setting, confirming that **context awareness is crucial for robust KV-cache reuse**.\n\nTable 3. Performance comparison with DroidSpeak on various benchmarks using Llama3.1-8B-instruct.\n\n|Dataset|Method|#Agent=2|#Agent=3|#Agent=4|\n|-|-|-|-|-|\n|MMLU|Original|47.1|66.7|68.0|\n||DroidSpeak|30.1|40.5|38.6|\n||KVComm|**64.7**|**68.6**|**68.0**|\n|GSM8K|Original|81.1|**82.4**|**82.1**|\n||DroidSpeak|52.2|20.6|14.5|\n||KVComm|**81.5**|81.7|80.6|\n|HumanEval|Original|**86.3**|**83.9**|**84.5**|\n||DroidSpeak|14.9|27.3|15.5|\n||KVComm|81.4|83.2|83.2|\n|Math500|Original|**41.6**|**39.6**|42.6|\n||DroidSpeak|14.8|14.8|14.8|\n||KVComm|38.0|38.6|**44.2**|\n\n## Q4: Performance on non-RoPE Models\nAs discussed in Proposition 1, KVComm’s effectiveness relies on the regular positional structure provided by RoPE-based encodings. To validate this, we tested KVComm on OPT-6.7B (absolute position encoding). As shown in Table 4, the reuse rate drops to near zero.\n\nTable 4. Performance of OPT-6.7B on the MMLU benchmark.\n\n|Method|Accuracy|Reuse Rate|\n|-|-|-|\n|Original @ 2 Agents|25.5|-|\n|KVComm @ 2 Agents|24.8|0.6%|\n|Original @ 3 Agents|26.8|-|\n|KVComm @ 3 Agents|24.2|0.2%|\n|Orignal @ 4 Agents|25.5|-|\n|KVComm @ 4 Agents|22.2|0.16%|\n\nThis is because **absolute position encodings do not preserve local positional relationships when the prefix changes, undermining cache similarity**. We will clarify in our revised manuscript that KVComm is designed specifically for RoPE-based models, and leave adaptation to other positional encodings as important future work.\n\n\n## Q5: Discussion about KVComm in Dynamic Agentic Framework\n\nThank the reviewer for highlighting the potential extension of KVComm. While our current implementation assumes predefined placeholders and templates to facilitate alignment and offset estimation, the core idea behind KVComm—context-aware offset estimation based on embedding similarity and anchor pools—is not inherently limited to static scenarios.\n\nIn principle, **it can be applied wherever context change patterns recur, provided shared text segmentation is feasible** (as with techniques similar to automatic prefix caching in vLLM). Although our current work does not cover fully dynamic, unstructured cases, we recognize this as a meaningful direction and plan to extend KVComm for more dynamic agentic and debate-style benchmarks in future work.\n\n## Q6: Reuse Rate Performance under Long-context Scenario\n\nWe have provided the experimental results in Table 2 under the 4K context setting. It can be observed that when the cumulative positional deviations amplify, the overall reuse rate drops by 20%, from $\\sim$ 70% to 50%. This aligns with expectations and the trend is similar to that of increasing placeholder number in the template, where the cumulative positional deviations will degrade the embedding similarity."}, {"review_id": "ZlfSs6a9dX", "reviewer": "Reviewer_3BgL", "summary": "The authors identify the offset-variance problem that prevents naïve KV-cache sharing across agents, and propose KVCOMM—a training-free, anchor-based framework. The anchor pool will store representative anchors on the fly, interpolates and learns the offsets across layers to align future shared tokens, and reuses the saved anchors to accelerate the prefilling. KVCOMM can achieve up to 13.7 speedup with minimal performance degradations.", "questions": "In the Table1, the performance of original settings on MMLU is quite low (47.1). Is it a typo?", "limitations": "My main concerns are in the evaluation sections. It would be better if the author can provide more quantitative results. Please see my comments in the weakness section.", "rating": 4, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 2, "clarity": 3, "significance": 3, "originality": 3, "strengths_and_weaknesses": "- **Strengths**\n  - Tackles the important problem of improving the efficiencies of multi-agent systems (MAS).\n  - Grounded with solid theory, providing formal error bounds and clear motivation.\n  - Reframes KV reuse as an *offset-alignment* problem and proposes an anchor-based interpolation mechanism.\n  - Includes extensive hyper-parameter ablation studies demonstrating the method’s robustness.\n\n- **Weaknesses**\n  - Experiments cover only a limited set of tasks and MAS setups; harder benchmarks such as Math500, AIME, and GAIA are not evaluated. \n  - Empirical comparison focuses mainly on a single baseline (cache blend). More possible baselines are helpful to get a comprehensive evaluation\n  - End-to-end (E2E) latency is not reported. It is unclear how much the speed-up of TTFT can be transferred to the E2E latencies and throughputs gains, which may be concerned more in the MAS settings.\n  - The anchor pools may introduce signifcant overhead, but this cost is not quantified. The KV cache could be very large for large models and complicated MAS trajectories. \n  - More fine-grained breakdowns are needed—for example, measuring the time to load retrieved elements into GPU memory when anchor pools are large and cannot be fully stored on-GPU.\n  - Sharing caches could raise safety or privacy concerns. Although this has already been discussed in the paper, no concrete measures is proposed to solve this concerns.", "was_revised": true, "final_justification": "In the review, authors provided detailed experiments on the breakdown of the overhead, comparison over different baselines, and experiments on harder tasks. I will raise my score by 1 since authors give detailed experimentation to resolve my concerns, yet I still notice the E2E improvement is not very exciting.", "rebuttal": "Thank the reviewer for the time to review and provide the valuable and constructive feedback. We address each of the comments in detail below.\n## Q1: Evaluations on Harder Benchmarks\n\nFor Math500, we report results using both Llama3.1-8B-instruct (Llama) and Deepseek-R1-Distill-Qwen-7B (DS-QWen). As shown in Tables 1 and 2, **KVComm achieves competitive or superior accuracy compared to the baseline method**. Notably, even as reuse rate drops with increasing agent number (due to higher context diversity), KVComm consistently matches or exceeds the baseline accuracy, suggesting that referencing anchor information helps improve answer quality. An especially encouraging observation is that **for DS-QWen, KVComm attains both a higher reuse rate and superior accuracy on challenging reasoning tasks**.\n\nTable 1. Math500 Results – Llama.\n\n|Method|Metric|#Agent=2|#Agent=3|#Agent=4|\n|-|-|-|-|-|\n|Original|Accuracy|**41.6**|**39.6**|42.6|\n|KVComm|Accuracy|38.0|38.6|**44.2**|\n||Reuse rate|59.4|40.9|30.7|\n\nTable 2. Math500 Results – DS-QWen.\n\n|Method|Metric|#Agent=2|#Agent=3|#Agent=4|\n|-|-|-|-|-|\n|Original|Accuracy|**51.4**|49.6|42.8|\n|KVComm|Accuracy|50.8|**50.8**|**45.8**|\n||Reuse rate|76.7|60.4|45.3|\n\nFor AIME, Table 3 reports the results using DS-QWen. **KVComm maintains a high reuse rate (70%+) while retaining comparable accuracy to the baseline**. The accuracy decrease largely attributes to token-length constraints during decode: as KVComm’s memory cost rises, the available context for generation per agent is reduced. We are actively addressing this with pipelining and efficient offloading techniques to support longer contexts in future work.\n\nTable 3. Performance on AIME using DS-QWen models.\n\n|Method|Metric|#Agent=2|#Agent=3|#Agent=4|\n|-|-|-|-|-|\n|Original|Accuracy|**19.2**|**17.5**|**17.5**|\n|KVComm|Accuracy|11.7|10.8|8.3|\n||Reuse rate|71.3|78.1|74.6|\n## Q2: Comparison with More Methods\n\nBelow we report experimental results for DroidSpeak alongside KVComm. DroidSpeak is designed for scenarios where **the base and finetuned models process the same entire prompt**—thus, context change is not addressed.\n\nTable 4 demonstrates that applying DroidSpeak directly in our experimental setup significantly degrades performance, **validating the necessity of context-aware cache reuse. KVComm consistently achieves higher or comparable accuracy across all datasets**.\n\nTable 4. Performance comparison with DroidSpeak.\n\n|Dataset|Method|#Agent=2|#Agent=3|#Agent=4|\n|-|-|-|-|-|\n|MMLU|Original|47.1|66.7|68.0|\n||DroidSpeak|30.1|40.5|38.6|\n||KVComm|**64.7**|**68.6**|**68.0**|\n|GSM8K|Original|81.1|**82.4**|**82.1**|\n||DroidSpeak|52.2|20.6|14.5|\n||KVComm|**81.5**|81.7|80.6|\n|HumanEval|Original|**86.3**|**83.9**|**84.5**|\n||DroidSpeak|14.9|27.3|15.5|\n||KVComm|81.4|83.2|83.2|\n|Math500|Original|**41.6**|**39.6**|42.6|\n||DroidSpeak|14.8|14.8|14.8|\n||KVComm|38.0|38.6|**44.2**|\n## Q3: End-to-End (E2E) Latency Report\nWe would like to clarify that in the KV-cache reuse literature (e.g., PromptCache, CacheBlend, DroidSpeak, KVLink), most works do not specifically target E2E latency optimization because E2E latency encompasses both prefill and decoding, whereas KV-cache reuse specifically accelerates prefill (TTFT). However, it has been shown (e.g., PromptCache) that in modern serving frameworks supporting prefill/decoding disaggregation (e.g., DistServe, Dynamo), reducing TTFT can significantly improve system throughput and user-perceived latency.\n\nNevertheless, for completeness, we provide E2E latency for various agent settings (Table 5). While the E2E latency improvement is modest (up to $\\sim$ 1.08x in long-prefill/short-decode cases), this is expected since KVComm does not target the decode stage. Still, any E2E improvement directly reflects the benefit of reduced prefill.\n\nTable 5. Latency Breakdown and Speedup vs. Dense Prefill.\n\n|IN|OUT|Agent ID|KVComm (s)|Other (s)|TTFT_Ori (s)|Decoding (s)|TTFT Speedup|E2E Speedup|\n|-|-|-|-|-|-|-|-|-|\n|64|128|0|0.083|0.005|0.068|2.430|0.770|0.992|\n|64|128|1|0.007|0.006|0.080|2.435|6.017|1.027|\n|64|128|2|0.008|0.008|0.097|2.434|6.088|1.033|\n|64|256|0|0.090|0.005|0.089|4.911|0.930|0.999|\n|64|256|1|0.008|0.007|0.115|4.919|8.199|1.020|\n|64|256|2|0.008|0.008|0.144|4.913|8.667|1.026|\n|64|512|0|0.090|0.005|0.089|9.913|0.934|0.999|\n|64|512|1|0.008|0.007|0.139|9.903|8.924|1.012|\n|64|512|2|0.009|0.010|0.213|9.910|11.130|1.020|\n|64|1024|0|0.090|0.005|0.087|19.997|0.913|1.000|\n|64|1024|1|0.010|0.009|0.204|20.007|10.762|1.009|\n|64|1024|2|0.011|0.013|0.369|20.004|15.260|1.017|\n|128|128|0|0.090|0.005|0.094|2.492|0.986|0.999|\n|128|128|1|0.007|0.006|0.110|2.499|8.203|1.038|\n|128|128|2|0.008|0.008|0.122|2.499|7.985|1.043|\n|128|256|0|0.090|0.005|0.093|4.928|0.982|1.000|\n|128|256|1|0.009|0.007|0.120|4.929|7.901|1.021|\n|128|256|2|0.008|0.008|0.149|4.927|8.987|1.027|\n|128|512|0|0.090|0.005|0.094|9.946|0.981|1.000|\n|128|512|1|0.009|0.007|0.145|9.956|9.084|1.013|\n|128|512|2|0.009|0.010|0.220|9.967|11.299|1.020|\n|128|1024|0|0.091|0.005|0.092|19.945|0.958|1.000|\n|128|1024|1|0.010|0.009|0.211|19.955|10.947|1.010|\n|128|1024|2|0.011|0.013|0.377|19.945|15.372|1.018|\n|256|128|0|0.083|0.005|0.082|2.461|0.932|0.998|\n|256|128|1|0.007|0.006|0.098|2.466|7.092|1.034|\n|256|128|2|0.008|0.008|0.115|2.467|7.281|1.040|\n|256|256|0|0.090|0.005|0.107|4.947|1.125|1.002|\n|256|256|1|0.009|0.007|0.134|4.943|8.716|1.024|\n|256|256|2|0.008|0.009|0.162|4.950|9.561|1.029|\n|256|512|0|0.090|0.005|0.108|9.891|1.129|1.001|\n|256|512|1|0.009|0.007|0.159|9.891|9.769|1.014|\n|256|512|2|0.009|0.010|0.239|9.893|12.234|1.022|\n|256|1024|0|0.092|0.005|0.106|20.190|1.091|1.000|\n|256|1024|1|0.011|0.009|0.230|20.211|11.514|1.010|\n|256|1024|2|0.012|0.014|0.399|20.160|15.698|1.018|\n|512|128|0|0.091|0.006|0.132|2.473|1.366|1.014|\n|512|128|1|0.007|0.006|0.144|2.474|10.366|1.052|\n|512|128|2|0.008|0.008|0.162|2.474|10.181|1.059|\n|512|256|0|0.090|0.006|0.131|4.980|1.377|1.007|\n|512|256|1|0.008|0.007|0.159|4.992|10.620|1.029|\n|512|256|2|0.009|0.009|0.206|4.983|11.621|1.038|\n|512|512|0|0.090|0.006|0.132|9.950|1.377|1.004|\n|512|512|1|0.009|0.007|0.202|9.955|12.329|1.019|\n|512|512|2|0.010|0.010|0.272|9.952|13.558|1.025|\n|512|1024|0|0.093|0.006|0.130|20.041|1.324|1.002|\n|512|1024|1|0.011|0.010|0.264|20.006|12.898|1.012|\n|512|1024|2|0.011|0.014|0.441|19.971|17.264|1.021|\n|1024|128|0|0.091|0.006|0.200|2.501|2.055|1.039|\n|1024|128|1|0.008|0.007|0.217|2.510|14.053|1.080|\n|1024|128|2|0.009|0.009|0.237|2.506|13.628|1.087|\n|1024|256|0|0.092|0.006|0.200|4.972|2.052|1.020|\n|1024|256|1|0.009|0.007|0.236|4.987|14.630|1.044|\n|1024|256|2|0.009|0.009|0.271|4.987|14.401|1.050|\n|1024|512|0|0.092|0.006|0.200|9.869|2.045|1.010|\n|1024|512|1|0.010|0.008|0.266|9.881|14.499|1.025|\n|1024|512|2|0.011|0.011|0.361|9.890|16.919|1.034|\n|1024|1024|0|0.092|0.006|0.195|20.055|1.992|1.005|\n|1024|1024|1|0.011|0.009|0.344|20.027|16.459|1.016|\n|1024|1024|2|0.013|0.014|0.550|19.988|20.565|1.026|\n## Q4: Memory Overhead Report Induced by KVComm\nBelow we provide the memory footprint for different agentic settings, including varying configurations of input/output length, and anchor numbers. As shown in Table 6, memory usage scales with both sequence and anchor number. Note that it is observed that **the deviation tensors stored for each anchor are highly sparse (averaging ~50% elements with absolute value < 1e-1), suggesting potential for lossless compression of anchors and further memory optimization**.\n\nTable 6. Memory Cost (GB) under the three-agent setting.\n\n|IN|OUT|#anchor=5|#anchor=10|#anchor=15|#anchor=20|\n|-|-|-|-|-|-|\n|64|128|1.67|3.31|4.95|6.58|\n|64|256|1.98|3.94|5.88|7.83|\n|64|512|2.61|5.19|7.76|10.3|\n|64|1024|3.86|7.69|11.5|15.3|\n|128|128|1.70|3.33|4.97|6.60|\n|128|256|2.01|3.96|5.91|7.85|\n|128|512|2.64|5.21|7.78|10.4|\n|128|1024|3.89|7.71|11.5|15.4|\n|256|128|1.74|3.38|5.02|6.65|\n|256|256|2.06|4.01|5.95|7.90|\n|256|512|2.68|5.26|7.83|10.4|\n|256|1024|3.93|7.76|11.6|15.4|\n|512|128|1.84|3.47|5.11|6.75|\n|512|256|2.15|4.10|6.05|8.00|\n|512|512|2.78|5.35|7.92|10.5|\n|512|1024|4.02|7.85|11.7|15.5|\n|1024|128|2.03|3.66|5.30|6.9|\n|1024|256|2.34|4.29|6.23|8.18|\n|1024|512|2.96|5.54|8.11|10.7|\n|1024|1024|4.21|8.04|11.9|15.7|\n## Q5: Finegrained Breakdown for Large Anchor Scenarios\nTable 2 in the main text has provided latency breakdowns and speedup, where the anchor loading overhead is counted at the 'Others' part in Table 2. For the large anchor scenario, we take the MMLU benchmark as an example with 4K placeholder tokens. Table 7 shows that anchor offloading can become a major latency bottleneck (up to 1.3 seconds) and that cumulative CPU memory can reach 75GB. While this is significant, *it can be mitigated by engineering improvements such as asynchronous scheduling and anchor compression, which are the focus of our ongoing system development.*\n\nTable 7. 4K-token Anchor Matching using Llama on MMLU\n\n|Metrics|#Agent=3|#Agent=4|\n|-|-|-|\n|Softmax (ms)|104|122|\n|Offloading (ms)|1260|1300|\n|Accuracy (%)|66.7|68.0|\n|Reuse Rate (%)|49.7|51.0|\n|Memory Cost (GB)|48.5|75.1|\n## Q6: Safety Considerations in Cache Sharing\nWe have reviewed our submission and confirm that safety and privacy of KV-cache sharing are not discussed in this version. While we agree that safety and privacy are of critical importance in practical multi-agent systems, a comprehensive treatment is outside the current paper’s scope. We will highlight this as a promising and important direction for future work.\n## Q7: Baseline Peroformance on MMLU\nWe have double-checked our experiments and confirm the correctness of the reported accuracy for the baseline on MMLU. The lower baseline performance under the two-agent setting arises from our agent design: the first agent is a knowledgeable expert who generates relevant keywords, and the second \"Final Refer\" is tasked to analyze the previous agent’s output and produce a final answer. Some failures are due to the second agent providing only an answer without analysis. All prompt templates are available in Appendix 6.3.2."}, {"review_id": "IN5IQpKW9t", "reviewer": "Reviewer_y5Xb", "summary": "This paper raises a key question: how can KV cache-based accelerated inference be effectively achieved in a multi-agent setting? It proposes a training-free calibration method, KVComm, which leverages KV cache offsets to enable kvcache reuse on placeholder texts with different prefixes.", "questions": "- What is the exact difference between the placeholder offset and the prefix offset? Could the author provide concrete examples to illustrate this? Why are both types of offsets necessary for multi-agent KV cache reuse?\n- Is there any quantitative experimental result demonstrating the relationship between the number of anchors, GPU memory usage, and acceleration efficiency?", "limitations": "Yes", "rating": 5, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 4, "clarity": 3, "significance": 4, "originality": 4, "strengths_and_weaknesses": "Strengths:\n- This paper provides both theoretical and empirical evidence for a compelling observation: despite variations in prefixes, identical placeholder texts often yield similar KV caches, and the discrepancy between them can be effectively captured and exploited to accelerate inference in multi-agent settings.\n- It is the first KV cache framework specifically designed for multi-agent scenarios, offering valuable infrastructure for advancing research in the multi-agent domain.\n- The proposed method achieves effective acceleration without requiring any additional training, making it lightweight and practical.\n\nWeaknesses:\n- The writing is somewhat hard to follow, and Figure 3 is overly complicated, making it difficult to grasp the overall process.\n- The paper introduces too many technical terms, making it unclear how the proposed method is concretely implemented based on the intuitive idea of “KV cache offset.”", "was_revised": true, "final_justification": "I increased the clarity score to 3 because the reviewer provided a detailed explanation of the overall process of KVCOMM in the rebuttal. I will keep my rating to 5 and advocate for acceptance.", "rebuttal": "Thank the reviewer for the time to review and provide the positive, constructive and valuable feedback. Below, we present our point-by-point responses to each comment.\n\n## Q1: Overall Process in Figure 3\nFigure 3 is designed to illustrate the overall workflow of KVComm. We summarize the process as follows:\n\n0. **Initialization**: Before any user requests, all agents precompute and store the KV-caches for all prefix segments defined in their prompt templates.\n1. **Placeholder Readiness**: When a request arrives, each agent checks whether all placeholders’ base KV-caches are available. If not, these are precomputed in parallel. Newly generated placeholder KV-caches then enter the anchor prediction module to check for similar samples in the anchor pool and enable reuse.\n2. **Reuse or Fallback**: Once all placeholder KV-caches are ready, the agent determines if there are reusable KV-cache deviations for each placeholder. If not, standard dense prefilling is used. For placeholders without reusable deviations, the agent computes the difference between their actual and base KV-caches, and stores this deviation in the anchor pool to expand anchor coverage.\n3. **Offset Approximation**: If all placeholders have reusable KV-cache deviations, the agent fetches the matched anchors, estimates the KV-cache deviations using Eq. (6) and (7), and updates the placeholder and prefix KV-caches in parallel.\n4. **Decoding**: The agent then combines the updated placeholder and prefix KV-caches to initiate response decoding.\n5.\t**Anchor Update**: After generating a response, its KV-cache passes through the anchor prediction module. If a similar anchor exists, the cache is stored in shared memory for future retrieval. The agent then awaits the next request.\n6. **Fallback Storage**: If no similar anchor exists, the response KV-cache is stored in the anchor pool for dependent agents to fill in deviations under their respective contexts, then the agent awaits the next request.\n\nIn conclude, **all agent interactions occur via the KV-cache Communication Module**. When multiple agents share the same user text but have different agent-specific prefixes, we avoid re-running prefilling by treating the KV-cache of the shared text under a new prefix as a context-dependent offset from its base KV-cache. We estimate that offset by interpolating from a small set of anchor examples, align Key positions (RoPE de-rotation / re-rotation), add the estimated Key/Value offsets, then concatenate the adjusted segments and decode.\n\nWe will clarify and streamline Figure 3 in the revised manuscript to further improve its readability. If further clarification is needed, we are happy to provide it.\n\n## Q2: Explanation of New Technical Terms\nWe will add a glossary in the revised manuscript for readability and open-source our code for reproducibility. Below are the newly introduced terms in KVComm:\n\n1. **Base KV-cache**: KV-cache for a prefix/placeholder under its initial context or without external input; serves as the reference for offsets.\n\n2. **KV-cache offset/deviation**: The difference between a shared text’s KV-cache under a new prefix and its base KV-cache. (Keys require RoPE-based alignment before offsetting.)\n\n3. **Placeholder/Prefix offset**: Offsets for the placeholder segment (external input) and its adjacent prefix segment (predefined context), respectively, relative to their base KV-caches.\n\n4. **Offset variance problem**: KV-cache offsets for the same text can vary substantially across contexts, so static reuse is unreliable.\n\n5. **Positional alignment/Key de-rotation**: RoPE de-rotation/re-rotation to align keys before offsetting.\n\n6. **Shareability**: Whether a request can skip prefill under anchor criteria; if not, dense prefilling is used.\n\n7. **Shared memory/KV-cache**: The shared storage that holds **base KV-caches** across agents.\n\n8. **Dense prefilling/generation**: Full computation path (prefill + decode) used if reuse is not possible.\n\n9. **Anchor (pool)**: A small set of representative examples, each storing placeholder and prefix offsets, used to interpolate offsets for new contexts.\n\n\n## Q3: Deeper Discussion about the Placeholder and Prefix Offsets\nThe exact difference between placeholder and prefix offset can be described from the following three aspects:\n\n1. **Definition**: \n   - Placeholder offset is defined as the KV-cache offset for externally injected text, including users, agents, and tools, etc, whose base KV-cache is not preset;\n   - Prefix offset is defined as the KV-cache offset for predefined prompt text, including system prompt, placeholder conjunctions, and suffix.\n\n2. **Dependence**:\n   - Placeholder offset arises mainly from changes to the prefix context;\n   - Prefix offset is triggered by changes to the preceding placeholder (i.e., injected content).\n\n3. **Variance**:\n   - Placeholder offset show high variance across samples and contexts;\n   - Prefix offset are more stable, as their base KV-caches are computed under a fixed system prompt and are less influenced by preceding placeholders.\n\nDue to the rebuttal format restriction, we are currently unable to share visualization results of placeholder and prefix offset distribution in the response. We will add this interesting characteristics into the revised manuscript.\n\n**Concrete Example (from Figure 3)**\n\nSuppose the prompt template of an agent is defined as follows:\n\n*\"You are a math expert. You will be given a math problem .... Q: {user_question}. Answers from other agents are: \\n Agent 2 as a mathematical analyst: {agent_2_current}, the execution result is: {condition_2_current}.
\"*\n\nIn this case,\n- “You are a math expert…” is a prefix segment (fixed context).\n- “{user_question}”, “{agent_2_current}”,  {condition_2_current}” are placeholders (external content).\n- Text following each placeholder (e.g., “Answers from…”) is a prefix segment adjacent to a placeholder.\n\nFrom this example, we can find that during the precomputation of prefix KV-caches, the subsequent prefix segments are mainly correlated to the first prefix segment, typically containing system prompts, thus their base KV-caches are relatively stable. When the preceding placeholder is injected with specific text, their actual KV-cache will contain the new information. However, the base KV-cache of prefix segments, unaware of this, must be adjusted via offset interpolation using similar anchor cases. **Otherwise, the model may receive conflicting signals: the base KV-cache of prefix segments would mislead the model decode by showing no actual placeholder text injected, while the injected placeholder text actually participate in the information fusion. Therefore, this conflict necessitates careful offset estimation for both types.** Experimental demonstration of the necessity has been provided in Table 5 of the main text.\n\nBased on the above analysis, hopefully, it would be clearer about the difference between the two offset types and the necessity of them. If further clarification is needed, we are happy to provide it.\n\n## Q4: Relationship between the number of anchors, the GPU memory usage, and TTFT speedup\nWe provide detailed experimental results in Tables 1 and 2, and analyze the relationship as follows:\n\n**Memory Cost vs. Anchor Number:**\n\nAs shown in Table 1, GPU memory usage increases approximately linearly with the number of anchors, for fixed input/output lengths and agent count. This trend holds across all sequence lengths: each additional anchor increases storage for associated KV deviations, leading to higher overall memory cost.\n\n**TTFT Speedup vs. Anchor Number:**\n\nHowever, in Table 2, the TTFT speedup does not increase monotonically with more anchors. Specifically,\nwhen increasing anchor number from 5 to 10 or 15, the TTFT speedup remains nearly unchanged or even slightly decreases across most settings. This indicates that anchor pool enlargement yields diminishing returns, and excessive anchors may even cause minor efficiency loss due to increased system overhead (e.g., lookup, memory access, or less optimal anchor matching).\n\nTable 1. Memory Cost (GB) under the three-agent setting with different maximun anchor numbers.\n\n|IN|OUT|#anchor=5|#anchor=10|#anchor=15|\n|-|-|-|-|-|\n|64|128|1.67|3.31|4.95|\n|64|256|1.98|3.94|5.88|\n|64|512|2.61|5.19|7.76|\n|64|1024|3.86|7.69|11.5|\n|128|128|1.70|3.33|4.97|\n|128|256|2.01|3.96|5.91|\n|128|512|2.64|5.21|7.78|\n|128|1024|3.89|7.71|11.5|\n|256|128|1.74|3.38|5.02|\n|256|256|2.06|4.01|5.95|\n|256|512|2.68|5.26|7.83|\n|256|1024|3.93|7.76|11.6|\n|512|128|1.84|3.47|5.11|\n|512|256|2.15|4.10|6.05|\n|512|512|2.78|5.35|7.92|\n|512|1024|4.02|7.85|11.7|\n|1024|128|2.03|3.66|5.30|\n|1024|256|2.34|4.29|6.23|\n|1024|512|2.96|5.54|8.11|\n|1024|1024|4.21|8.04|11.9|\n\nTable 2. Mean TTFT speedup under the three-agent setting with different maximun anchor numbers.\n\n|IN|OUT|#anchor=5|#anchor=10|#anchor=15|\n|-|-|-|-|-|\n|64|128|5.24|5.17|4.41|\n|64|256|5.64|5.82|4.86|\n|64|512|6.80|6.76|6.10|\n|64|1024|8.72|8.92|8.19|\n|128|128|5.50|5.40|4.77|\n|128|256|5.60|5.55|5.35|\n|128|512|6.91|6.94|6.67|\n|128|1024|9.01|9.09|8.55|\n|256|128|5.99|5.92|5.00|\n|256|256|6.15|6.12|5.52|\n|256|512|7.37|7.62|6.91|\n|256|1024|9.22|9.53|8.59|\n|512|128|7.02|6.98|5.89|\n|512|256|7.65|7.64|4.70|\n|512|512|8.77|8.65|8.19|\n|512|1024|10.38|10.52|10.11|\n|1024|128|9.97|9.66|8.56|\n|1024|256|10.18|10.42|9.05|\n|1024|512|10.80|11.09|10.55|\n|1024|1024|12.79|12.94|12.32|"}]

Author rebuttal:

Meta-review: This submission proposes a training-free framework (KVComm) for efficient KV-cache reuse in multi-agent LLM systems. In multi-agent settings, overlapping contexts often lead to redundant KV-cache computation due to diverging prefixes. This would prevent straightforward cache reuse. KVComm introduces an online anchor pool that records cache deviations under varying prefixes and uses them to align offsets dynamically. This idea helps enable effective cache sharing across agents. Experiments on show performance improvements, including up to 13× speedup in prefill latency and over 70% KV-cache reuse. 

Strengths of this submission:

1. Reviewers 2PaM, y5Xb consider that it tackles an important efficiency problem in multi-agent LLM systems. The proposed method addresses the overhead caused by repeated KV-cache recomputation in multi-agent pipelines.
2. reviewers TNJW, 3BgL are happy with its empirical results, as experiments show consistent acceleration (13× speedup) and thorough ablations and sensitivity analyses that validate the method’s design choices.
3. Reviewers y5Xb, 3BgL mentioned that the framework is supported by insights into KV-cache offset alignment while remaining training-free and applicable in real-world multi-agent scenarios.

Weaknesses of this work: 

1. Though presented with promising performance gains, three reviewers (2PaM, TNJW, 3BgL) also point out that its comparison with existing methods does not form as its strong point. Specifically, while CacheBlend is discussed, direct experimental comparisons and positioning relative to related works such as speculative decoding and DroidSpeak are insufficient. 
2. reviewers 2PaM, y5Xb also identify clarity and presentation issues. Some parts of the paper including figures and terminology are not easy to follow, making the implementation details of KV-cache offset alignment less intuitive for readers unfamiliar with the subfield.

In summary, the submission makes contributions to improving efficiency in multi-agent LLM systems and provides solid empirical results supported by insights. While there are concerns regarding baseline comparisons, applicability to heterogeneous settings, and some presentations issues, the technical soundness and performance improvements make this a valuable addition to the literature. During the rebuttal, the reviewers are generally in support of this work (all four final ratings are positive). All factors considered, this work is given an acceptance recommendation.
Decision: Accept (poster)