Title: KVLink: Accelerating Large Language Models via Efficient KV Cache Reuse
Venue: NeurIPS 2025
Accepted: yes
Keywords: LLM

Retrieval-Augmented Generation

Efficiency
Abstract: We describe KVLink, an approach for efficient key-value (KV) cache reuse in large language models (LLMs). In many LLM applications, different inputs can share overlapping context, such as the same retrieved document appearing in multiple queries. However, the LLMs still need to encode the entire context for each query, leading to redundant computation. In this paper, we investigate a new strategy to eliminate such inefficiency, where the KV cache of each document is precomputed independently. During inference, the KV caches of retrieved documents are concatenated, allowing the model to reuse cached representations instead of recomputing them. To mitigate the performance degradation when using KV caches computed independently for each document, KVLink introduces two key techniques: adjusting positional embeddings of the KV cache at inference to match the global position after concatenation, and using trainable special tokens to restore self-attention across independently encoded documents. Experiments across 7 datasets demonstrate that KVLink improves question answering accuracy by an average of 4% over state-of-the-art methods. Furthermore, by leveraging precomputed KV caches, our approach reduces time-to-first-token by up to 96% compared to standard LLM inference, making it a scalable and efficient solution for context reuse. Additionally, KVLink can be combined with KV cache compression to further save cache loading and storage overhead while outperforming the baselines.


Peer reviews:
{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper introduces KVLink, an approach designed to reusing KV caches across docs. The problem KVLink aims to solve is due to inherent inefficiency in many LLM applications, particularly those involving RAG, where different inputs often share overlapping context. Standard LLM inference requires re-encoding the entire concatenated context for each query, leading to redundant computations.\nKVLINK addresses this by precomputing the KV cache of each doc segment independently. During inference, these precomputed KV caches are concatenated, allowing the model to reuse cached representations instead of recomputing them. To counteract the performance degradation  that typically arises from independently computed KV caches. KVLink introduces positional re-encoding and trainable cross-segment link tokens.\nThe experiments validated KVLINK's effectiveness across seven datasets. Using 5 link tokens, it improved QA accuracy by 4%. Additionally, it significantly reduced TTFT\n\nSTRENGTHS AND WEAKNESSES:\n### Strength\n1. The problem of redundant computation due to repeated encoding of shared contexts in LLMs is a genuinely interesting and critical challenge. \n2. This paper introduces the KVLINK approach with positional re-encoding and trainable link tokens tackle the KV cache reuse in multi docs scenarios. The trainable link tokens are an elegant solution for restoring cross-document attention.\n3. The experiments demonstrate the effectiveness of KVLINK across diverse datasets and model scales (Llama-3.2-1B, Llama-3.2-3B, and Llama-3.1-8B). While link tokens provided improvement in smaller models.\n\n### Weakness\nOverall, this paper addresses the recomputation problem by introducing link tokens to capture relationships between documents to some extent. However, it still has the following weaknesses:\n1. Storing documents as KV Cache remains too costly at present.\n2. Based on the results shown in the paper, most of the experimental improvements are observed in 1B and 3B models, with only marginal gains for the 8B model. Can larger models (30B/70B) be evaluated to demonstrate the general applicability of link tokens?\n3. In the experiments presented in the paper, many experimental designs for certain task types, such as the NQ task, do not fully demonstrate the effectiveness of link tokens.\n\nQUESTIONS:\nQuestions have been proposed in weakness.\n\nLIMITATIONS:\nYes", "TITLE": "Official Review", "RECOMMENDATION": 4, "REVIEWER_CONFIDENCE": 3, "CLARITY": 4, "IMPACT": 3, "ORIGINALITY": 3, "DATE": "2025-06-21"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe paper proposes a method called KVLink to reuse precomputed KV caches in order to alleviate the burden of recomputing the KV cache for each retrieved document. Since simple KV cache concatenation leads to significant performance degradation, the authors propose trainable “link” tokens that can aggregate information and reconstruct the interconnection between documents. These link tokens attend to both other link tokens and to their assigned documents. From an implementation perspective, pre-RoPE is employed so that the positional indices of the KV cache can be adjusted properly. Compared to existing cache-reuse methods, the proposed approach achieves superior QA performance and summarization accuracy. Furthermore, KVLink demonstrates a significant speedup in time-to-first-token (TTFT).\n\nSTRENGTHS AND WEAKNESSES:\n**Strengths** \n\n* The concept of using link tokens is clear. It seems that parallelism can be maintained at inference time simply by adjusting the attention map properly.\n* Experiments are well-designed and detailed. The authors put an effort into fair comparisons with related work (Sections A.3, A.4). The paper evaluates KVLink across multiple tasks, including summarization. Analyses also include the GPU loading time of the KV cache in the TTFT measurements.\n\n**Weaknesses**\n\n* The most significant issue, which the authors also discuss, is that storing the entire KV cache demands an overwhelmingly large storage footprint. Even with speed gains, the jump from 5 KB to 131 MB (lines 157–158) is not practical. Although methods such as AnLLM have been explored, the manuscript does not specify how much space is saved versus how much performance is lost. In my opinion, even 5KB to 1MB is too much.\n* While extending into the domain of RAG and KV compression is meaningful, the methodology appears similar to AnLMM. KVLink seems to expand from chunk-level to document-level granularity. The manuscript should discuss these differences in detail or provide specific improvements focused on the application (e.g., RAG).\n* The on-the-fly adjustment of RoPE positions is already well known as “pre-RoPE”, in KV cache compression and quantization papers. In this perspective, the novelty of position re-embedding (rotation) is somewhat weak.\n\nQUESTIONS:\n* During training, “link” tokens take positional indices between documents; is this policy maintained at inference time, too? In other words, do link tokens possess non-consecutive positional indices in the implementation?\n* In TTFT experiments (Section 3.3), on which model and GPU were the experiments conducted, and how does TTFT vary with model size?\n* In Table 1, it would be helpful to indicate which methods require fine-tuning (BlockAttention, KVLink) and which do not, for clarity.\n\nLIMITATIONS:\nyes", "TITLE": "Official Review", "RECOMMENDATION": 4, "REVIEWER_CONFIDENCE": 3, "CLARITY": 3, "IMPACT": 3, "ORIGINALITY": 2, "DATE": "2025-07-02"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nIn RAG, data is retrieved then processed by the network during prefil. This work extends something like prefil cache-ing to the RAG system as a whole, where instead of retrieving text data, network KV cache is retrieved (note given retrieved docs can have different ordering, a special attention mask must be used in training).\n\nThis paper introduces a method already introduced in https://arxiv.org/abs/2410.07590. (they even intro special tokens to act as what this work calls \"links\").\nIts a really good idea; just already exists. Not novel :(\n\nSTRENGTHS AND WEAKNESSES:\nStrengths:\n- really good idea\n- The potential efficiency gains are f-in cracked (i'm like 80% sure the big labs do a version of this; the big labs already to disag prefil-decode; might as well stick a cache in-between the prefil and decode GPUs...)\n\nWeaknesses:\n- Not novel; missing tubrorag as related work\n- Scalability and Practical Deployment Issues\n  - Missing analysis of storage overhead costs vs. computational savings trade-offs\n    - KV caches are massive compared to text\n  - No analysis of cache management strategies for large document collections\n  - No cost analysis of storage infrastructure requirements\n- I'm not you test performance vs long context well (ie number of retrieved docs vs performance)\n\nQUESTIONS:\nplease cite: https://arxiv.org/abs/2410.07590\nPLEASE rewrite your paper comparing to TurboRAG and resubmit. please please please. I would 100% accept if it was redone that way.\n\nwhat is the size of your vector store vs the baseline where just text is stored (storing text much cheaper than storing kv-caches)\n\nLIMITATIONS:\nnot novel\nkv-caches are larger than text. this will blow up the size of your vector store.\n\nI'd also be good to talk about storage vs compute economics and cache coherence", "TITLE": "Official Review", "RECOMMENDATION": 4, "REVIEWER_CONFIDENCE": 4, "CLARITY": 4, "IMPACT": 4, "ORIGINALITY": 1, "DATE": "2025-07-02"}

comments: SUMMARY:
The paper introduces KVLINK, a method for accelerating large language models (LLMs) by enabling efficient key-value (KV) cache reuse across queries with overlapping context segments, such as in retrieval-augmented generation (RAG) scenarios. Traditional LLMs redundantly re-encode identical context segments for each query, leading to significant computational inefficiency. KVLINK addresses this by precomputing the KV cache for each document independently and then concatenating these caches at inference time, allowing for direct reuse.

STRENGTHS AND WEAKNESSES:
[+] KV cache positional re-encoding, and trainable cross-segment special (link) tokens as techniques to overcome the performance degradation of independently encoded KV caches. 

[+] Significant Efficiency Gains: KVLINK achieves up to 96% reduction in time-to-first-token latency by reusing precomputed KV caches, making it highly scalable for applications with overlapping contexts

[-] The KVLINK technique works under the assumption of the existance of the `c_{linki}` tokens. However, section 3.1 only mentions " fine-tuning them for 6,000 steps using a global batch size of 64 across 8×H100 GPUs". Does this mean that `c_{linki}` is just a special set of markers and NOT special token (OR) there is an additional mid-training phase where the tokens are baked in?

QUESTIONS:
1. How does KVLINK perform in scenarios where the overlap between context segments across queries is low? Is there a break-even point where the overhead of precomputing and managing caches outweighs the benefits?

2. Is there any degradation in model performance for tasks that require strong cross-document reasoning? Given, often times it's not possible to know ahead of time, what use case the query+documents are for.

LIMITATIONS:
yes
TITLE: Official Review
CLARITY: 3
IMPACT: 3
ORIGINALITY: 3
DATE: 2025-07-03