KVCOMM: Online Cross-context KV-cache Communication for Efficient
  LLM-based Multi-agent Systems
Authors: name: Hancheng Ye, name: Zhengqi Gao, name: Mingyuan Ma, name: Qinsi Wang, name: Yuzhe Fu, name: Ming-Yu Chung, name: Yueqian Lin, name: Zhijian Liu, name: Jianyi Zhang, name: Danyang Zhuo, name: Yiran Chen
Organization: Duke Center for Computational Evolutionary Intelligence (CEI)

Multi-agent large language model (LLM) systems are increasingly adopted for
complex language processing tasks that require communication and coordination
among agents. However, these systems often suffer substantial overhead from
repeated reprocessing of overlapping contexts across agents. In typical
pipelines, once an agent receives a message from its predecessor, the full
context-including prior turns-must be reprocessed from scratch, leading to
inefficient processing. While key-value (KV) caching is an effective solution
for avoiding redundant computation in single-agent settings where prefixes
remain unchanged, it cannot be directly reused in multi-agent scenarios due to
diverging prefixes introduced by agent-specific context extensions. We identify
that the core challenge lies in the offset variance of KV-caches across agents.
To address this, we propose KVCOMM, a training-free framework that enables
efficient prefilling in multi-agent inference by reusing KV-caches and aligning
cache offsets of overlapping contexts under diverse prefix contexts. KVCOMM
estimates and adjusts KV-caches for shared content by referencing a pool of
cached examples-termed anchors-that store observed cache deviations under
varying prefixes. The anchor pool is maintained and updated online, allowing
dynamic adaptation to distinct user requests and context structures. KVCOMM
achieves over 70% reuse rate across diverse multi-agent workloads, including
retrieval-augmented generation, math reasoning, and collaborative coding tasks,
all without quality degradation. Particularly, when each fully-connected agent
receives 1K input tokens with 512 prefix tokens and 512 output tokens under a
five-agent setting, KVCOMM achieves up to 7.8x speedup compared to the standard
prefill pipeline, reducing TTFT from ~430 ms to ~55 ms.