AdaCache: Adaptive Caching and Context Augmentation for Efficient LLM Serving
Authors: Zeng Zihao
Siyi Li
Xinyu Yan
Lei Xiao
Wei Yang Bryan Lim
Venue: ICLR 2026 Poster (2026)
Keywords: LLM Inference
Retrieval-Augmented Generation
Caching
Recomputation
Primary area: infrastructure, software libraries, hardware, systems, etc.

Abstract: Retrieval-Augmented Generation (RAG) significantly enhances Large Language Models by integrating external knowledge sources, but at the cost of substantial computational overhead from extended input sequences. 
Current RAG systems exhibit two fundamental inefficiencies: redundant processing of frequently retrieved text chunks across multiple queries, and uniform deep retrieval that over-provisions context regardless of query complexity.
We present AdaCache, an adaptive caching framework that addresses these limitations through dual optimization strategies. 
First, we introduce a cache-aware partial recomputation mechanism that profiles attention patterns to construct selective cache variants, enabling flexible reuse while preserving cross-chunk dependencies. 
Second, we develop adaptive context augmentation that dynamically determines optimal retrieval depth via lightweight confidence estimation, avoiding unnecessary overhead on simple queries.
Comprehensive experiments across diverse datasets and LLMs demonstrate that AdaCache delivers substantial improvements in Time-To-First-Token compared to state-of-the-art RAG caching systems, while preserving generation quality.

Published as a conference paper at ICLR 2026
ADACACHE: ADAPTIVE CACHING AND CONTEXT AUGMENTATION FOR EFFICIENT LLM SERVING
Zihao 2 Li', Xinyu Yan Lei Xiao?, Wei Yang Bryan Lim'
Nanyang Technological University
2 Alibaba-NTUU Global e-Sustainability CorpLab (ANGEL)
Alibaba Group
tz r hiaaaooc. zeng, bryan.limwy)enn....... wiw.....iiyeent.edd...
ABSTRACT
Retrieval-Augmennte Generation (RAG) significantly enhances Large Language Models by integrating external knowledge sources, but at the cost of substan- tial computational overhead from extended input sequences. Current RAG sys- tems exhibit two fundamental inefficiencies: redundant processing of frequently retrieved text chunks across multiple queries, and uniform deep retrieval that over-provisions context regardless of query complexity. We present AdaCache, an adaptive eaching framework that addresses these limitations through dual op- timization strategies. First, we introduce cache-aware partial recomputation mechanism that profiles attention patterns to construct selective cache variants, enabling flexible reuse while preserving cross-chune dependencies. Second, we develop adaptive context augmentation that dynamically determines optimal re- trieval depth via lightweight confidence estimation, avoiding unnecessary over- head on simple queries. Comprehensive experiments across diverse datasets and LLMS demonstrate that AdaCache delivers substantial improvements in Time-To- First-Token compared to state-of-the-arr RAG caching systems, while preserving generation quality.
INTRODUCTION
Large Language Models (LLMs) have become ubiquitous across diverse applications, from con- versational chatbots and personal assistants to specialized systems handling question answering, document summarization, and machine translation (Achiam al., 2023 Hurst et al. 2024 Guo
ct al. 2025 Yang al. 2025). Despite their impressive capabilities, LLMs suffer from hallucina- tion issues and knowledge limitations, particularly when dealing with domain-specifir or up-to-date information. Retrieval-aud letrivaalauugmentee generation (RAG) (Ram al. 2023 Siriwardhana et al.) 2023
Jiang et al. 2023) has emerged as a powerful paradigm to bridge this gap. By incorporating external knowledge bases, such Wikipedia (Cohere 2023) domain-speciff corpora, it retrieves relevant contextual information to enrich user queries. This approach has demonstrated remarkable success in improving generation quality, while enabling general-purpose LLMs tackle specialized domain problems without costly fine-tuning.
Despite these benefits, RAG introduces significant system-level challenges. The injection of re- trieved text chunks substantially increases the length of input prompts, leading to higher computation and memory requirements during the LLM inference. For instance, while a raw user query typically contains fewer than 200 tokens, augmenting it with retrieved context can push the sequence length beyond 2,000 tokens, leading to more than a 10x increase computational and memory overhead. This dramatic expansion significantly degrades Time-To-Finst-Tooee put, ultimately compromising user experience. The key objective to achieve the best of both worlds: barnessing RAG's quality improvements while preserving computational efficiency.
(TTFT) and system through-
Our observation reveals two major inefficiencics in current RAG systems. The first is cross-query context overlap, where identical text chunks from the external knowledge base are repeatedly
"Corresponding author.

---

Published as a conference paper at ICLR 2026
100
100
Ratio Quarrees (%)
Ratio Qunries (%)
80
80
Queriees
60
40
40
Retrieval
Retrieval
Retrivval Top
Tog-2
Top-2
20
Top-3 Retrieval
Tog-3 Retrievel
Top-3 Retreeel
0
20
40
60
80
100
20
40
60
80
100
20
60
80
100
Ratio Text Chunks (%)
Text Chunks (%)
TeTtst Chunks (%)
(a) MMLU.
(b) TriviaQA.
(c) NaturalQuestions--Oeenn
Figure 1: Retrieval pattern on different datasets.
5.0%
1.5%
7.5%
17.5
16.2%
MMMUU
SuperGPQA
TrivieQA
Figure 2: Distribution of minimum top-k retrieval requirements for correct responses using Llama3- 8B-Instruct model on different datasets.
trieved across multiple user queries, and a small fraction of text chunks dominate the retrieval re- quests. As shown in Fig. Ia observe power-law distributions in text chunks popularity on the MMLU dataset (Hendrycks et al. 2020), where the most frequently accessed 10% of text chunks satisfy 80% of all questions under top-1 retrieva [] This skewed access pattern indicates substan- tial redundant computation during LLM inference, as the same contextual information is processed repeatedly for different user queries. The second inefficiency stems from over-allocation of text within individual queries, regardless of their complexity. Although L.LMs consistently bene- fit from expanded contextual information, accuracy improvements follow a pattern of diminishing marginal utility with additional retrieved text chunks. We validate this intuition using Llama3-8B. Instruct (Dubey al. 2024b) on MMLU, SuperGPQA (Du al. 2025), and TriviaQA (Joshi al., By analyzing minimal knowledge requirements for accurate model predictions, we observe that over 60% of queries require only minimal context, whereas only approximately 3% need top-8 retrieval. This distribution highlights a critical inefficiency: static deep retrieval incurs unnecessary computational costs on simple queries while potentially degrading accuracy through contextual noise. These findings illuminate fundamental optimization challenge in the RAG system: How can we achieve both computational efficiency and performance gains simultaneously?
2017) datasets, shown in Fig. 2
Caching represents a promising solution to address computational redundancy in RAG systems by reusing previously computed representations (Le.. KV cache). Recent advances, including vLLM (Kwon et al. 2023), SGLang (Zheng el al. 2024). and RAGCache (Jin et al.) (002)), em- ploy prefix caching store key-value representations of processed text chunks. While maintaining generation quality equivalent to full recomputation. these methods require exact sequence matching, leading to poor hit rates with longer contexts and positional variations. Independent chunk caching approaches attempt more flexible strategies. PromptCache (Gim et al. 2024) achieves higher effi- ciency through independent chunk caching sacrifices accuracy by ignoring cross-chunk attention.
For top-2 and top-3 retrieval, we treat each unique combination of retrieved text chunks as distinct context unit for cumulative distribution analysis.
2

---

Published as a conference paper at ICLR 2026
CacheBlend (Yao al., 2025) partially restores cross-chunk attention via selective recomputation, yet applies uniform recomputation ratios across all chunks without considering the heterogeneous attention characteristics across different chunks. Furthermore, all prior work assumes static top-k lp-krn re- trieval, missing query-adaptive optimization opportunities that could enable simultaneous efficiency and aceuracy improvements.
In this paper, we present an adaptive caching framework that addresses both computational redun- dancy and contextual over-provisioning RAG systems through two complementary mechanisms. We first design a cache-aware partial recomputation method that profiles attention patterns to con- struct multiple cache variants per text chunk, selecting minimal recomputation strategies during reuse. Then, we introduce an adaptive context augmentation strategy that incrementally expands retrieval depth using lightweight confidence estimation to determine optimal context length for each user query. Evaluation across multiple models and datasets shows that we achieve 1 1.4x~5.0x TTFT reduction over of-te-of-the-arr art RAG caching systems while maintaining accuracy.
2 BACKGROUND AND RELATED WORK
Autoregressive Transformers execute inference in two distinct phases. In the prefill phase, the model processes the entire input sequence, performing self-attention across all tokens and materializing per-layer KV caches. In the subsequent decode phase, tokens are generated step by step while attending to this cached state. By reusing the stored projections of preceding tokens, the KV cache eliminates redundant recomputation of the prefix and enables efficient autoregressive generation.
RAG extends this pipeline by incorporating external evidence. A retriever encodes the user query, searches corpus, and returns the top-k passages (Ram et al., [2023 Siriwardhana et al., 2023 Jiang et al. 2023). The generator concatenates the query and retrieved passages, tokenizes the combined sequence, and applies the same deedill process: prefill constructs KV entries for all tokens, and decode reuses them to produce the answer. However, concatenation markedly lengthens the prompt, increasing both attention cost and KV overhead in proportion sequence length.
As a result, prefill dominates serving latency, raising TTFT and reducing throughput under load. Moreover, much of the additional computation is not essential for factual grounding, such as inter- actions among irrelevant passages or regions with low query attention. The fundamental bottleneck thus the cost of full prefill and KV materialization over long contexts, motivating mechanisms that preserve only query-evidence interactions while avoiding redundant computation.
General LLM Inference Systems. v.L.M (Kwon al. 2023) accelerates generic serving via Page- dAttention with block-wise KV paging and sharing: Orca (Yu et al. 2022) scales distributed decod- ing through iteration-level scheduling; prefillddcodee disaggregation, as in DistServe (Zhong et al., 2024) and SplitWise (Patel et al., 2024), separates phases across GPUs to mitigate interference; and FlexGen (Sheng et al., 2023) expands effective capacity by aggregating memory and computation from the GPU, CPU, and disk. These approaches reduce phase contention and memory pressure but treat prompts as monolithic sequences, leaving them ill-suited to RAG's retrieval-induced redun- dancy and leading to suboptimal performance.
Retrieval Optimization. Sparse retrievers such as TF-IDF (Ramos et al. 2003) and BM25 (Robert- son et al. 2009) enable efficient lexical matching, while dense retrievers leverage learned embed- dings for higher recall at greater cost (Karpukhin et al.) 2020). On top of these, rerankers refine first-stage results to improve precision with moderate overhead (Sun al.) 2023 Pradeep et al.
2023 Santhanam et al. 2021). These techniques focus on improving retrieval quality, whereas our method leaves the retrieved set unchanged and targets efficiency in post-retrieval processing.
Context Reusing, Caching mechanisms amortize the prefill cost by reusing KV states. Prefix caching, as SGLang (Zheng et al. 2024). CachedAttention (Yao et af.) 2025), and RAGCache (Jin et al. (024), achieves fidelity but relies on exact prefix matches, resulting in low hit rates under long and variable RAG prompts. To alleviate this limitation, independent chunk caching relaxes match- ing: PromptCache (Gim et al.) 2024) caches blocks independently but discards cross-chunk atten- tion, thereby compromising accuracy, while CacheBlend (Yao et al. (2255) reintroduces interactions via selective recomputation yet applies uniform ratios oblivious to heterogeneous attention patterns. Our approach addresses these gaps by incorporating attention-awan cache variants with minimal recomputation to preserve fidelity, and by employing confidence-guidees per-query adaptive expan-
3

---

Published as a conference paper at ICLR 2026
sion. This design reduces redundant computation, lowers long-context overhead, and significantly improves both TTFT and throughput.
3 ADACACHE
Sclecdive Recompuaatie
Uard Prefix Cachu Space
Saft Profix Cache Space
Independent Carbu Space
C4
C5
EW
CV
C10
C11
Hibike State
Adsppive Augmestttion
Plac
Tentchink
Logit
KY Gateo
Cfcls
Tru
Atergg K. Disagean
Exsey Eyen
Drakutt Confidencs
TearChunk
Qatr
Text Eleak
&
KV Urabe-l
Confidmer Beare
Figure 3: The Overview of AdaCache. It consists of two complementary modules. Cache-aware selective recomputation (upper) maintains three hierarchical cache spaces: (1) Hard prefix cache requires exact prefix matching and stores KV cache for all chunks along the matching path (solid boxes), enabling direct reuse without recomputation; (2) Soft prefix cache matches only effective prefix chunks requiring partial recomputation at ratio a, where solid boxes represent cached entries while dashed boxes indicate prefix dependencies without storage; (3) Independent cache performs chunk-level matching with higher recomputation ratio B. The top attention diagram shows selective recomputation where 2 out of tokens (blue solid blocks) are recomputed while the remaining 4 tokens reuse cached KV states. Adaptive context augmentation (lower) incrementally expands prompts by adding one text chunk at a time, evaluating confidence after each addition using a com- posite metric combining average KL divergence across the last few layers and output entropy, ter- minating when sufficient confidence is achieved or maximum context is reached.
3.1 CACHE- SELECTIVE RECOMPUTATION
Attention Analysis. We begin by analyzing chunk-level attention patterns to understand inter- chunk dependencies in RAG contexts. The augmented prompt segmented into discrete chunks: [system prompt, text chunk 1, text chunk k, query], and we aggregate attention weights of each layer into chunk granularity. Fig. 4 demonstrates two distinct attention distributions across model depth during Qwen3-8B model inference + Farly layers (1-18) show localized patterns where each chunk primarily attends to its predecessor, while deeper layers (19-36) exhibit attention sink phe- nomena, with certain chunks capturing most attention from subsequent chunks. This pattern reveals that only a subset of chunks serves as effective prefixes, enabling joint caching of partial prefix sequences restore cross-chunk dependencies lost in independent chunk caching.
Hierarchical Cache. Based on the observed attention patterns, we establish three-tier cache hi- erarchy that systematically balances cache utilization efficiency against generation quality. Hard
We validated these chunk-level attention patterns across Llama3-88 -BBa Instruct, Qwen3-4B, and Qwen3-8B 8B models on MMLU, TriviaQA, and SuperGPQA datasets, observing consistent hehaviors, though the specific chunk positions serving as attention sinks vary across different contexts.
4

---

Published as a conference paper at ICLR 2026
Daye:
Loor?
Lec?
Saor?
Caakhatte
Zontt
Zear!!
Inse14
Laye
Lowftf
Luee:
AMeii Wegg
Chaikt Tabton
Los
Eayo:3
Lomt
Liee'
Clack Patton
Chucs Podkann
Clank Nutze
Clask Fertm
Crul Nartan
Clanc Portee
Cluss Postdes
Chank Paston
Chust Plata
Figure 4: Chunk-level attention patterns during Qwen3-8B model inference with Top-8 retrieval. The first and last columns in each subplot correspond the system prompt and user query, respec- tively. Layers 1-18 exhibit localized attention where chunks predominantly focus on their immediate predecessors with sparse attention distant chunks. In layers 19-36, an attention sink phenomenon emerges where the 3rd chunk captures the majority of attention from subsequent chunks.
Prefix Cache requires exact prefix sequence matching, making it the most restrictive but accuracy- preserving tier. Due to the causal attention mask in autoregressive inference, exact prefix matches guarantee computational equivalence to full recomputation, thereby preserving perfect generation quality when cache hits occur. However, this strict requirement significantly constrains cache uti- lization. Soft Prefix Cache relaxes the matching constraint to effective prefix matching, where only sink chunk or predecessor chunk needs to match for cache reuse. This design leverages our attention analysis findings: since attention primarily flows from these key chunks, partial prefix matching can maintain most cross-chunk dependencies. Independent Cache provides the fallback mechanism when prefix matching fails entirely. Individual text chunks precomputed and stored independently without prefix dependencies. It maximizes cache hit rates but poses the greatest risk for accuracy preservation, cross-chunk attention dependencies must be reconstructed during LLM inference.
Cache Reusing and Recomputation. Building on previous work Yao et al. 2025), only a sub- set of tokens within each chunk exhibit significant cross-chunk attention, leading to substantial KV states deviations compared to those the independent cache. Critically, this sparsity pattern ex- hibits layer-wise consistency: tokens with the highest KV deviations in layer are likely have the highest deviations in subsequent layers. This insight enables efficient selective recomputation by identifying attention-critical tokens through first-layer analysis and applying the same selection across all layers. We determine recomputation candidates by analyzing cross-chunk attention ratios in the initial layer, selecting tokens with the highest proportion of cross-chunk attention weights.
Rather than a uniform recomputation across all chunks in context, we adapt the recomputation ratio based on available cache matches. The KV states of each chunk may have multiple cached variants stored under different prefix contexts. We retrieve from cache spaces in a hierarchical order with progressively relaxed matching constraints.
We first query the hard prefix cache for exact matches, where any chunks along the matched prefix path can directly reused without recomputation. When exact matching fails, we examine the soft prefix cache for effective prefix alignment. Note that soft prefix chunks serve only as cache key identifiers without maintaining separate cache entries. Successful soft matching requires recomput- ing fraction of tokens to restore global cross-chunk attention If no cached KV states exist in
'In the first half of model layers, effective prefixes correspond to predecessor chunks, while in the sec- ond half, they correspond to both sink chunks and predecessor chunks, We identify sink chunk positions by analyzing attention matrices transition layers: chunks before the sink chunk require mrddccesor-haseee matching, while chunks after the sink chunk use both sink chunks and predecessor chunks their effective
5

---

Published as a conference paper at ICLR 2026
Algorithm 1 Adaptive Context Augmentation
Require: System prompt s, query 9, retrieved text chunks (ci,...,
,......,,kl, Oke, confidence threshold
1: for = to l: do
2: p
[1] (1, Ca] +
Construct context-augmented prompt
3: for j 1 1o
do
4: Retrieve hash((c1..... e,]) in Hard Prefix Cache
Reuse KV states w/o recomputation
5: end for
6: if hash(effective praptiiveeeeeii((ei........ c:)) E Soft Prefix Cache then
7:
2 a
Lower recomputation ratio for Soft Prefix Cache
8: else
9:
P 8 end if
Higher recomputation ratio for Independent Cuche
10:
11:
T f fraction of tokens in c;
> Selected tokens with the highest cross-chunk attention >> Generate logits for last layers
12:
t (T)
13:
conf< +- X -K(OOI.-11,0) (1 X) ((,)))
D Evaluate the confidence
14:
if conf > r then
15: Execute decoding
Stop context augmentation
16: break
17: end if
18: Update KV states of e; with hash((ci,.....4 in Hard Prefix Cache
end for
the soft prefix cache space, we turn to the independent cache with recomputation ratio / (3 >> a) reconstruct discarded cross-chunk dependencies. This cache reuse approach achieves an optimal efficiency- accuracy trade-off through adaptive token recomputation that responds to varying prefix match conditions: exact, partial, or absent.
3.2 ADAPTIVE CONTEXT AUGMENTATION
Algorithm I presents the process of adaptive context augmentation (ACA) with cache-aware recom- putation. Rather than concatenating all top-k retrieved text chunks into the user prompt simultane- ously, we employ an incremental augmentation strategy that progressively incorporates one chunk at time until reaching the k-th chunk or achieving sufficient confidence. While this approach ne- cessitates multiple forward passes for the same query, it eliminates redundant context computation through strategic caching. At each iteration, we only recompute the KV states for the newly added chunk, storing them in the hard prefix cache space for reuse in subsequent context augmentation. This ensures that all previously processed chunks maintain cache hits, dramatically reducing compu- tational overhead. Importantly, ACA does not introduce any additional retrieval overhead. The top-k retrieval step is executed once per query, and the augmentation loop then operates solely within the prefill phase using the diready-retriewe text chunks.
To decide whether augmentation should terminate, we employ a composite confidence metric com- bining two complementary uncertainty measures. First, compute the average KL divergence between the logits of the last layers and the final layer, capturing internal reasoning consistency. If the model can accurately infer the answer from the current context, its logit distribution should con- verge early across layers. Second, we calculate the entropy of the final token distribution, reflecting output uncertainty. We normalize both the average KI. divergence and entropy to [0, 1], then com- pute weighted confidence score, with weights determined through optimization on the validation set. This dual-metric balances stability with predictive certainty, providing a more robust confidence estimate. Notably, the confidence metric is computationally lightweight. In practice, AdaCache computes logits only for the last 4 layers and for the final token rather than the full context, which keeps the overhead negligible at less than 1% of the prefill cost.
ACA reduces computational and memory demands by avoiding excessive context allocation for simple queries. Given l retrieved text chunks of length I, tokens each, query of length lg tokens,
prefixes. Including predecessor chunks prevents cumulative errors that would arise from inconsistency with the predecessor-basee matching used in the first half of layers.

---

Published as a conference paper at ICLR 2026
AdaCache (ours)
CacheBlemd
Prefix Caching
Full Recompatation
Liama3-88
Quen3-81
Qwaa3-18
70
74
Better
70
30
15
0.5
1.0
1.5
1.5
6.5
35
Bustor
Better
6.0
1.3
2.0
0.0
2.0
60
1.0
55
55
THNNBQQ
35
30
70
46-
40-
60-
6.0
05
14
1.5
2.0
0.5
1.9
1.5
2.0
60
6.5
1.0
HoppeQA
35
50
30
Figure 5: Comparison of Time-to-Tirse Toke--Toen (TTFT) and generation quality between AdaCache and baseline methods across six datasets and three models.
and early termination step t, ACA processes at most t- t- 11)) tokens. yields substantial savings in computation and memory compared to static context augmentation, which requires processing le + L tokens]
EVALUATION
4.1 EVALUATION METHODOLOGY
Models and Hardware Settings. We evaluate AdaCache using al. 2024a), Qwen3-4B, and Qwen3-8B (Yang al., 2025) models. Experiments are conducted
(Dubey
*Retrieved text chunks are typically longer than queries, with 512 tokens being a common chunk size while queries usually contain fewer than 128 tokens. For example, with top-6 retrieval, static context augmentation processes 3,200 tokens per query during the prefill phase. In contrast, given the empirical expectation of f == 2 text chunks required across the dataset, ACA processes only 1,280 tokens, achieving 2.5x reduction in computational cost.

---

Published as a conference paper at ICLR 2026
on server equipped with 128 CPU cores (2x Intel Xeon Gold 6530), 512 GB of host memory, and NVIDIA RTX 6000 Ada GPU with 48 GB memory. Data transfers between the CPU and GPU are carried out over the PCIe x.0x 16 interface.
Corpus and Datasets. We use the Wikipedia dataset as our knowledge base. Prior to embedding, all documents are segmented into chunks of size 512 tokens. Each chunk is then encoded using the e5-base-v2 embedding model. For vector search, we leverage the FAISS library to construct an inverted file (IVF) index with 1024 clusters, and set the default top-k retrieval to 6. AdaCache is evaluated on several rigorous benchmark datasets, including MMLU (Hendrycks et al. 2020), MMLU-Pro (Wang et al. 2024), SuperGPQA (Du et 2025), TriviaQA (Joshi et al. (017)) 2iiii- MultihopQA (Ho et al., 2020), and HotpotQA (Yang et al. (2018), which span general knowledge, open-domain reading comprehension, and advanced reasoning.
Baselines. We compare AdaCache with three baselines: Full Recomputation, where the raw text is fed into the LLM and the KV cache for all tokens is computed during prefill; (ii) Prefix Cache (Jin et al., 2024j, which leverages SGLang (Zheng et al. 2024) to identify frequently used prefix chunks and persist their KV caches in RAM and SSD, while non-prefix tokens are still com- puted during prefill. For fairness, we optimistically assume no delay when loading from RAM/SSI) to GPU, which favors this scheme relative to real deployments; and (iii) Selective Recomputation, which adopts CacheBlend (Yao et al., 2025) to reuse precomputed KV caches of all chunks, while selectively recomputing in each layer small subset of high-deviation tokens to restore cross-chunk attention.
Metries. We evaluate models on both accuracy and responsiveness. Accuracy is measured by Exact Match (EM), the fraction of predictions that exactly match a normalized reference answer. Responsiveness is measured by Time-To- First-Token (TTFT), the wall-clock latency from request submission to the emission of the first output token. We report results across repeated runs under controlled hardware and inference settings.
4.2 EXPERIMENTAL RESULTS
Naive RAG systems recompute KV caches for every new request and its retrieved context. Ada- Cache achieves substantial TTFT reductions of 3.12x on average and up to 6.02x compared to full recomputation while preserving nearly identical generation quality. The performance gains derive from AdaCache's 's dual optimization strategy, which simultaneously eliminates cross-request compu- tational redundancy in overlapping contexts while preventing unnecessary context augmentation for simple queries. Notably, AdaCache occasionally surpasses full recomputation in prediction accu- racy, as excessive contextual information can introduce noise that degrades model reasoning. Guided by model output confidence, AdaCache ensures that the minimal sufficient context contributes the generation process.
AdaCache demonstrates 2.69 average and up to 5.0x performance improvements over prefix caching. While prefix caching eliminates redundant computation of overlapping prefixes and main- tains identical generation quality full recomputation, exact prefix matching limits its effectiveness with longer contexts or dynamic positioning of retrieved chunks. AdaCache addresses these limita- tions with hierarchical cache architecture (i.e., hard prefix cache, soft prefix cache, and independent caches), enabling more flexible cache reuse.
CacheBlend leverages independent caching to achieve substantial improvements in cache hit rates, employing selective recomputation to maintain cross-chunk attention and preserve generation qual- ity. In comparison, AdaCache delivers 1.32x on average and up to 2.34x TTFT improvements over CacheBlend with marginally superior generation quality. AdaCache analyzes inter-chunk attention patterns across layers and constructs soft prefix caches, enabling flexible hierarchical caching that reduces token-level recomputation and decreases TTFT. Additionally, adaptive context selection duces computational waste from non-contributory text chunks.
'ee use the wikimed' a aatiaaawiiipiiiia dataset on Hugging Face, which contains cleaned articles from the official Wikipedia dumps. Fach subset corresponds to one language and consists of single training split with markdown and references removed. In our experiments, we adopt the English subset released 2023-11-00
8

---

Published as a conference paper at ICLR 2026
1.2
Caching ACA CacheBlend ACA
Caching ACA
1.6
AdaCache ACA
CacheRlend ACA
AdsCCche ACA
0.9
1.2
TIFT()) (8)
TIFT
0.6
0.8
0.3
0.4
0.0
0.0
MMLU
MMLU-PIo
SuperGPQA
InviaQA
MMLU
MMLU-PPo
SuperGPQA
InviaQQ
(a) Qwen3-4B.
(b) Qwen3-8B.
Figure 6: TTFT comparison of caching strategies combined with and without ACA across different
datasets and models.
Quey Prapation in Dunse (N)
60
MMLU
MMLUI MMLU--P0 SuperGPQA
MMLU SaperCPQA
30
MMLU-Pro
SuperGPQQA
Quay Pupartinn Dataud
1.0
Query Proportion Dunzet
50
TEiSIQA
80
TrivAQA
TriviQQA
an
50
20
20
10
5
Number Retrived Chunks
Number of Recriveed Text Chunks
Number of Retrivved Text Chunks
Lamma-8B-Instruce
(b) Qwen3-8B.
(c)
Qwen3-4B.
Figure 7: The context length distribution determined by adaptive context augmentation across dif- ferent datasets and models.
Ablation Study. To isolate the performance contributions of cache-aware selective recomputation and adaptive context augmentation (ACA), we compare four configurations: Prefix Caching com- bined with ACA, CacheBlend combined with ACA, AdaCache without ACA, and the full AdaCache system (Fig. 6). ACA integrates smoothly with different caching mechanisms, and when applied to Prefix Caching CacheBlend, it consistently reduces TTFT across datasets and models, delivering average speedups of 1.65x and 1. 1.22x, respectively.
The full AdaCache system retains substantial performance advantages ACA-enhanced base- lines, achieving average speedups of 1.76x over Prefix Caching and 1.23x over CacheBlend. No- tably, cache-aware selective recomputation alone occasionally outperforms even ACA-enhanced baselines. These results demonstrate the effectiveness of our hierarchical caching design and its tight integration with ACA.
Context Length Distribution. To better understand the performance improvements of Adaptive Context Augmentation (ACA), we analyze the distribution of context lengths identified by ACA during model inference. As shown Fig. 7 consistent pattern emerges across all three models and four datasets: the majority of queries require minimal context augmentation, while queries requiring longer contexts become increasingly rare. The sharp spike at maximum length includes queries that remain unanswerable even when provided with the complete top-6 retrieved text chunks, indicating persistently low confidence throughout the ACA process.
The performance gains from ACA correlate strongly with this distribution pattern. Datasets exhibit- ing more pronounced head-heavy distributions with smaller tail proportions yield greater improve- ments. MMLU and TriviaQA demonstrate more skewed distributions compared to MMLU-Pro and SuperGPQA, with correspondingly higher relative performance gains. Specifically, AdaCache achieves x.95 and 1,62x average TTFT reduction over CacheBlend MMLU and TriviaQA,

---

Published as a conference paper at ICLR 2026
AdtCCache (ouss)
CacheBlend
Pieftx Caching
Recompuiation
Llama3-4B
Qwen3-8B
Qwen3-4B
24
2.5
25
200
MMLDD
1.5
1.5
10
1.0
0.5
0.0
0.0
0.0
20
2.0
2.0
TriiiaQ
155
1.5
05
0.5
0.5
Number Retiiiedd Chunks
Number Retricved Text Chunks
Number Reuieved Toxt Chanks
Figure 8: Comparison of TTFT between AdaCache and baseline methods across different top-k retrieval.
respectively, across three models, compared to more modest improvements of 1.25x and 1.14x on MMLU-Pro and SuperGPQA respectively.
Performance Across top-k Retrieval Settings. Fig. g demonstrates the performance compar- ison between AdaCache, CacheBlend, Prefix Caching, and Full Recomputation across varying top-k retrieval configurations") [ At top-2 retrieval, the performance gap between Prefix Caching, CacheBlend, and AdaCache remains modest. Prefix Caching achieves a substantial TTFT reduction compared to Full Recomputation due to relatively high cache hit rates in short context scenarios.
However, as context expands, clear performance divergence emerges. Prefix Caching suffers dra- matic degradation, with TTFT improvements declining from an average of 1.76x at top-2 to merely 1.13x at top-8 retrieval, reflecting the fundamental limitation of strict prefix matching in long con- text scenarios. In contrast, AdaCache exhibits superior context scalability, with performance gains improving from an average of 2.93 to 4.67 over full recomputation. While CacheBlend's inde- pendent caching strategy substantially improves cache hit rates for long contexts compared to Prefix Caching, AdaCache achieves fundamentally better context scalability by combining hierarchical caching with adaptive context augmentation.
5 CONCLUSION
We present AdaCache, comprehensive framework that addresses fundamental computational inef- ficiencies in RAG systems through dual optimization strategies: cache-aware partial recomputation that profiles attention patterns to construct selective cache variants, and adaptive context augmenta- tion that dynamically determines optimal retrieval depth via lightweight confidence estimation. Our approach tackles two key inefficiencies observed in current RAG systems: the power-law distribu- tion of context reuse across queries, where 10% of chunks satisfy 80% of retrieval requests, and the over-allocation of context, where 60% of queries require only minimal retrieval. Comprehensive evaluation demonstrates that AdaCache achieves 1.4x~5,0x TTFT reduction over state-of-the-ee RAG caching systems while maintaining generation quality. Notably, our adaptive context augmen- tation enables seamless integration with existing caching strategies while exhibiting superior context scalability.
For AdaCache, top-k refers the maximum available context length during adaptive context augmentation.
10

---

Published as a conference paper at ICLR 2026
ACKNOWLEDGMENTS
This research is supported by the NTU startup grant and the RIE2025 Industry Alignment Fund -In- dustry Collaboration Projects (IAF-ICP) (Award 12301 E0026), administered by A*STAR, as well as supported by Alibaba Group and NTU Singapore through Alibaba-NTU Global e- e-Sustainability CorpLab (ANGEL).This research is also supported by the Ministry Education, Singapore, under its Academic Research Fund Tier 2 (Award MOE-T2EP20125-0000
ETHICS STATEMENT
Our work adheres to the ICLR Code of Ethics Our research does not involve human subjects, sensitive personal data, any identifiable information. All datasets used in experiments publicly available and widely adopted prior literature. We strictly followed the terms of use of these datasets and ensured that no proprietary or private information The methods developed are intended purely for academic research and are not designed to produce harmful applications. We are committed promoting fairness, transparency, and reproducibility in machine learning research, and we release our results in compliance with community standards of research integrity.
accessed or disclosed.
REPRODUCIBILITY STATEMENT
We provide full details to support reproducibility. The AdaCache framework, including cache- aware recomputation and adaptive context augmentation, is specified Section with with pseudocode and design assumptions. Experimental settings, datasets, preprocessing, evaluation metrics, and baseline configurations are described in Section 4 Model architectures, and hardware settings are reported to allow replication of latency and throughput measurements.
THE USE LARGE LANGUAGE MODELS (LLMs)
We used large language models as the general- purpose assistive tool during the preparation this paper. Its contributions were limited to improving grammar, polishing wording, and suggesting alternative phrasings for clarity and conciseness. The research ideas, methodological design, exper- imental implementation, analysis, and final interpretations were entirely conceived and executed by the authors.
LLMs were not used for generating novel research content, fabricating facts, or conducting scientific reasoning. All technical descriptions, results, and conclusions presented in the paper are the sole responsibility of the authors.
REFERENCES
Josh Achiam, Steven Adler, Sandhini Agarwal, Lama Ahmad, Ilge Akkaya, Florencia Leoni Ale- man, Diogo Almeida, Janko Altenschmidt, Sam Altman, Shyamal Anadkat, et al. Gpt-4 technical report. arXiv preprint arXiv:2303. 08774, 2023.
Cohere. Wikipedia en embeddings dataset. bttps://huggingee cotdatasets/////////aaa 2023. Accessed: 2025-09-13.
Xinrun Du Yifan Yao, Kaijing Ma, Bingli Wang, Tianyu Zheng, King Zhu, Minghao Liu, Yiming Liang, Xiaolong Jin, Zhenlin Wei, et al. Supergpqa: Scaling Ilm evaluation across 285 graduate disciplines. arXiv preprint arXiv:2502. 14739, 2025.
Abhimanyu Dubey, Abhinav Jauhri, Abhinav Pandey, Abhishek Kadian, Ahmad Al-Dahle, Aiesha Leuman, Akhil Mathur, Alan Schelten, Amy Yang, Angela Fan, al. The llama 3 herd of models. arXiv e-prints, pp. arXiv-2407, 2024a.
Abhimanyu Dubey, Abhinav Jauhri, Abhinav Pandey, Abhishek Kadian, Ahmad Al-Dahle, Aiesha Letman, Akhil Mathur, Alan Schelten, Amy Yang, Angela Fan, et al. The llama 3 herd of models. arXiv e-prints, pp. arXiv-2407, 2024h.
11

---

Published as a conference paper at ICLR 2026
In Gim, Guojun Chen, Seung-seob Lee, Nikhil Sarda, Anurag Khandelwal, and Lin Zhong, Prompt cache: Modular attention reuse for low-latency inference. Proceedings of Machine Learning and Systems, 6:325-338, 2024.
Daya Guo, Dejian Yang, Haowei Zhang, Junxiao Song, Ruoyu Zhang, Runxin Xu, Qihao Zhu, Shirong Ma, Peiyi Wang, Xiao Bi, et al. Deepseek-rl: Incentivizing reasoning capability in Ilms via reinforcement learning. arXiv preprint arXiv:2501. 12948, 2025.
Dan Hendrycks, Collin Burns, Steven Basart, Andy Zou, Mantas Mazeika, Dawn Song, and Jacob Steinhardt. Measuring massive multitask language understanding. arXiv preprint arXiv:2009.00 033000 2020.
Xanh Ho, comprehensive evaluation of reasoning steps. In Proceedings of the 28th Interna- qa dataset for
Duong Nguyen, Saku Sugawara, and Akiko Aizawa. Constructing multi-hop
fional Conference on Computational Linguistics, pp. 6609-6625, 2020.
Aaron Hurst, Adam Lerer, Adam P Goucher, Adam Perelman, Aditya Ramesh, Aidan Clark, AJ Os- trow, Akila Welihinda, Alan Hayes, Alec Radford, et al. Gpt-4o system card. arXiv preprint arXiv:2410.21277 2024.
Zhengbao Jiang, Frank F Xu, Luyu Gao, Zhiqing Sun, Qian Liu, Jane Dwivedi- Yu, Yiming Yang, Jamie Callan, and Graham Neubig. Active retrieval augmented generation. In Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing, pp. 7969-7992, 2023.
Chao Jin, Zili Zhang, Xuanlin Jiang, Fangyue Liu, Xin Liu, Xuanzhe Liu, and Xin Jin. Ragcache: Efficient knowledge caching for retrieval-augmennt generation. arXiv preprint arXiv:2404. 12457, 2024.
Mandar Joshi, Eunsol Choi, Daniel S Weld, and Luke Zettlemoyer. Triviaqa: A large scale distantly the 55th Annual Meet-
supervised challenge dataset for reading comprehension. In Proceedings the ing of the Association for Computational Linguistics (Volume 1: Long Papers), pp. 1601-1611, 2017.
Vladimir Karpukhin, Barlas Oguz, Sewon Min, Patrick SH Lewis, Ledell Wu, Sergey Edunov, Danqi Chen, and Wen-tau Yih. Dense passage retrieval for open-domain question answering. In EMNLP (1), pp. 6769-6781, 2020.
Woosuk Kwon, Zhuohan Li, Siyuan Zhuang, Ying Sheng, Lianmin Zheng, Cody Hao Yu, Joseph Gonzalez, Hao Zhang, and Ion Stoica. Efficient memory management for large language model serving with pagedattention. In Proceedings of the 29th symposium on operating systems princi- ples, pp. 611 1-16666, 2023.
Pratyush Patel, Esha Choukse, Chaojie Zhang. Aashaka Shah, fñigo Goiri, Saeed Malcki, and Ri- cardo Bianchini. Splitwise: Efficientt generative Ilm inference using phase splitting. In 2024 ACM/EЕЕ s/ Annual International Symposium on Computer Architecture (ISCA), pp. 118- 132. IEEE, 2024.
Ronak Pradeep. Sahel Sharifymoghaddam, and Jimmy Lin. Rankzephyr: Effective and robust zero- shot listwise reranking is a breeze! arXiv preprint arXiv:23/2.027244 2023.
Ori Ram, Yoav Levine, Itay Dalmedigos, Dor Muhlgay, Amnon Shashua, Kevin ecytonBBron and Yoay Shoham. In-context vetrieval-augmente language models. Transactions of the Association for Computational Linguistics, 1 11:1316-133) 2023.
Juan Ramos et al. Using tf-idf to determine word relevance in document queries. In Proceedings of the first instructional conference machine learning, volume 242, pp. 29 48. New Jersey, USA, 2003.
Stephen Robertson, Hugo Zaragoza, et al. The probabilistic relevance framework: Bm25 and be- yond. Foundations and Trends@ in Information Retrieval, 3(4):333-38) 2009.
Keshav Santhanam, Omar Khattab, Jon Saad-Falcon, Christopher Potts, and Matei Zaharia. Colbertv2: Effective and efficient retrieval via lightweight late interaction. arXiv preprint 1rXiv:2/12.011 12.01188. 2021.
12

---

Published as a conference paper at ICLR 2026
Ying Sheng, Lianmin Zheng. Binhang Yuan, Zhuohan Li, Max Ryabinin, Beidi Chen, Percy Liang, Christopher Ré, lon Stoica, and Ce Zhang. Flexgen: tigh-throughput generative inference of large language models with single gpu. In International Conference Machine Learning, pp- 31094-31116. PMLR, 2023.
Shamane Siriwardhana, Rivindu Weerasekera, Elliot Wen, Tharindu Kaluarachchi, Rajib Rana, and Suranga Nanayakkara. Improving the domain adaptation of retrieval augmented generation (rag) models for open domain question answering. Transactions of the Association for Computational Linguistics, 11:1-77, 2023.
Weiwei Sun, Lingyong Yan, Xinyu Ma, Shuaiqiang Wang, Pengjie Ren, Zhumin Chen, Dawei Yin, and Zhaochun Ren. Is chatgpt good at search? investigating large language models re-ranking agents. arXiv preprint arXiv:2304.09544 2023.
Yubo Wang, Xueguang Ma, Ge Zhang, Yuansheng Ni, Abhranil Chandra, Shiguang Guo, Weiming Ren, Aaran Arulraj, Xuan He, Ziyan Jiang, et al. Mmlu-pro: A more robust and challenging multi- task language understanding benchmark. Advances in Neural Information Processing Systems, 37:95266 37:95266-95290. 2024.
An Yang, Anfeng Li, Baosong Yang, Beichen Zhang, Binyuan Hui, Bo Zheng, Bowen Yu, Chang Gao, Chengen Huang, Chenxu Lv, et al. Qwen3 technical report. arXiv preprint arXiv:2505.09338,, 2025.
Zhilin Yang, Peng Qi, Saizheng Zhang, Yoshua Bengio, William W. Cohen, Ruslan Salakhutdinov, and Christopher D. Manning. HotpotQA: A dataset for diverse, explainable multi-hop question answering. In Conference on Empirical Methods in Natural Language Processing (EMNLP), 2018.
Jiay Yao, Hanchen Li, Yuhan Liu, Siddhant Ray, Yihua Cheng, Qizheng Zhang, Kuntai Du, Shan Lu, and Junchen Jiang. Cacheblend: Fast large language model serving for rag with cached knowledge fusion. In Proceedings of the Twentieth Earopean Conference on Computer Systems, pp. 94-109, 2025.
Gyeong-In In Yu, Joo Seong Jeong, Geon-Woo Woo Kim, Soojeong Kim, and Byung- Chun. Orca: A distributed serving system for ( generative models. In 16th USENIX Sympo- sium on Operating Systems Design and Implementation (OSDI 22), pp. 521-538 2022.
Lianmin Zheng, Liangsheng Yin, Zhiqiang Xie, Chuyue Livia Sun, Jeff Huang, Cody Hao Yu, Shiyi Cao, Christos Kozyrakis, Ion Stoica, Joseph E Gonzalez, et al. Sglang: Efficient execution of structured language model programs. Advances neural information processing systems, 37: 62557-62583, 2024.
Yinmin Zhong, Shengyu Liu, Junda Chen, Jianbo Hu, Yibo Zhu, Xuanzhe Liu, Xin Jin, and Hao Zhang. (DistServe): Disaggregating prefill and decoding for goodput-optimiin large language model serving. In 18th USENIX Symposhum Operating Systems Design and Implementation (OSDI 24), pp. 193-210, 2024.
13


Reviews:
[{"review_id": "fh3akxx3ov", "reviewer": "Reviewer_NEXz", "summary": "The paper proposes AdaCache, a RAG-oriented serving framework that (i) builds a 3-tier, attention-aware KV cache (hard prefix, soft prefix, independent) and selectively recomputes only attention-critical tokens, and (ii) runs an adaptive context augmentation loop that stops adding retrieved chunks once the model is confident. On QA-style RAG benchmarks and several open LLMs, it reports 1.4×–5× TTFT improvements over prior caching systems while keeping accuracy.", "questions": "- Can you report an ablation that fixes the cache to CacheBlend-style selective recompute and only adds your ACA loop, to show the incremental benefit?\n\n- How exactly is the soft-prefix cache key defined, and what prevents explosion in cache entries when the sink chunk moves?\n\n- What is the actual extra latency of computing per-layer logits for the confidence score on your hardware, and is it still a win when retrieval is remote / high-latency?", "limitations": "", "rating": 6, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "- Tackles an actually painful RAG-serving bottleneck and is compatible with existing inference stacks.\n\n- Sensible combination of attention-guided partial recomputation with incremental context — the two parts reinforce each other.\n\n- Empirical results show clear win over prefix-only caching and even over CacheBlend in long-context regimes.", "weaknesses": "- Each core idea has close prior art: RAGCache / PromptCache / CacheBlend on the caching side, and Active / Adaptive / Bandit RAG on the context-length side; the paper currently oversells novelty. \n\n- Experiments do not ablate how much gain comes from \"adaptive context\" vs \"better cache reuse\", and they do not compare to adaptive-retrieval baselines.\n\n- The proposed confidence computation and attention-sink profiling may add non-trivial overhead and may not generalize to other models/tasks.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "u7s6L0LGw8", "reviewer": "Reviewer_1wvZ", "summary": "AdaCache proposes a smarter, adaptive framework to speed up RAG in LLMs by solving two common system bottlenecks, repeatedly re-processing the same retrieved context chunks for multiple queries, and indiscriminately adding lots of context even when the query doesn’t need it. To fix this, AdaCache introduces a cache-aware partial recomputation system, profiling the attention flows so it only recalculates what’s actually needed and dynamically matches context prefixes and an adaptive context augmentation strategy based on confidence scores for each query, so simpler questions don’t get massive context of just because. In experiments across several standard datasets (MMLU, SuperGPQA, TriviaQA), AdaCache shows big wins in time-to-first-token , achieving up to 5x speedup over some baselines, all while keeping generation quality intact.", "questions": "Could the adaptive context augmentation framework be extended to multi-hop QA or dialogue, with context dependencies beyond the initial retrieval? How robust is the cache-aware recomputation under rapidly changing corpora e.g., streaming or up-to-date knowledge bases, what is the trade off? Are there edge-case queries where adaptive context addition either under-fetches (missing context) or over-fetches (noise)? Any way you could provide results for wider domain/task evaluation beyond QA/reading comprehension?", "limitations": "", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 4, "contribution": 3, "strengths": "First to combine chunk-level attention analytics and hierarchical caching with dynamic, query-by-query adaptive retrieval depth. Demonstrates large and consistent speedups (up to 5x TTFT) across varied tasks, while maintaining or improving answer quality. Empirical coverage multiple models, datasets, real hardware setup, system code/pseudocode details included. Addresses two core RAG bottlenecks context reuse, variable context needs, rather than just one. Well-justified caching logic, tested across different chunk retrieval densities.", "weaknesses": "Technical sections like attention analysis, selective recomputation ratio could use more intuitive explanation for ML domain readers. Adaptive context augmentation relies on confidence measurements that may need more validation for edge-cases/unusual queries. Tested mainly on general QA/benchmark tasks. broader evaluation with multi-turn dialogue, enterprise data, multilinguality would add a lot. Efficiency gains are only as good as the popularity distribution of text chunks. if access is more uniform, the cache hit rates drop. Hardware setup is fairly high-end. results might differ for lower-resourced deployments.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "OKdAmfISd7", "reviewer": "Reviewer_DScb", "summary": "This paper provides a efficient LLM Serving strage named AdaCache, which addresses two key inefficiencies of current RAG systems—redundant processing of frequent text chunks and uniform deep retrieval—via two optimization strategies. It uses cache-aware partial recomputation, which profiles attention patterns to build selective cache variants for flexible reuse while preserving cross-chunk dependencies. It also adopts adaptive context augmentation, dynamically determining optimal retrieval depth through lightweight confidence estimation to avoid unnecessary overhead on simple queries. Experiments across datasets and LLMs show AdaCache cuts Time-To-First-Token (TTFT) by 1.4x–5.0x versus state-of-the-art RAG caching systems, while maintaining generation quality.", "questions": "Large models preformance can be provided for 14b/30b/72b and so on.", "limitations": "", "rating": 6, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "Easy architecture and good  performance.\n\n- Efficient Resource Utilization: By reusing cached KV states of frequent text chunks and avoiding over-retrieval for simple queries, AdaCache drastically reduces redundant computation. This translates to a 1.4x–5.0x TTFT reduction, greatly improving LLM serving throughput without wasting computational resources.​\n- Quality Preservation: Its cache-aware partial recomputation preserves cross-chunk dependencies via attention pattern analysis and selective recomputation, while adaptive context augmentation uses a composite confidence metric to ensure sufficient context for accurate generation. Thus, it maintains generation quality even with efficiency gains.​\n- Strong Adaptability: The three-tier cache hierarchy (hard prefix, soft prefix, independent cache) adapts to different prefix match conditions, and adaptive context augmentation adjusts retrieval depth based on query complexity. It works across diverse datasets and LLMs, showing broad applicability.", "weaknesses": "Some points can be optimized.\n\n- Dependence on Attention Pattern Consistency.\nThe cache-aware recomputation relies on consistent layer-wise attention patterns (e.g., early localized attention, deep attention sinks). If LLMs have irregular attention distributions for certain tasks/domains, the cache variant design may fail, reducing reuse efficiency or accuracy.​\n- Increasing System Complexity. \nThe three-tier cache management, selective recomputation ratio tuning, and composite confidence metric (KL divergence + entropy) add complexity to system implementation and maintenance. This raises the barrier for integration into existing RAG pipelines, especially for teams with limited engineering resources.​\n- Potential Overhead in Incremental Augmentation.\nAdaptive context augmentation requires multiple forward passes (adding one chunk at a time) for queries needing deep retrieval. Though cached reuse mitigates this, it may still introduce more latency than static retrieval for complex queries requiring full top-k chunks, offsetting efficiency gains.​\n- Reliance on Confidence Metric Tuning.\n The confidence metric (weighted KL divergence and entropy) needs optimization on validation sets. Poor tuning (e.g., misaligned weights) could lead to premature termination (insufficient context, lower accuracy) or unnecessary augmentation (wasted resources), making performance unstable across unseen data.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "JmyRHMK7BY", "reviewer": "Reviewer_8B2T", "summary": "This paper points two major inefficiencies in current RAG systems: (1) cross-query context overlap, where identical text chunks from the external knowledge base are repeatedly retrieved across multiple user queries, and (2) a small fraction of text chunks dominate the retrieval requests. Based on these two findings, this paper aims to address both computational inefficiency simultaneously. AdaCache is then proposed to speed up the loading of contextual information. Experimental results on Llama 3 8B, Qwen 3 8B, and Qwen 3 4B show the proposed method is faster than CacheBlend and Prefix Caching in terms of TTFT with close correctness.", "questions": "* Can these method be seamlessly integrated with dense retrieval model, where the embedding of each chunk is also the cache? \n\n* The quality of Figure 3 could be further improved for readability.", "limitations": "", "rating": 4, "confidence": 5, "soundness": 2, "presentation": 2, "contribution": 2, "strengths": "* This work is well-motivated and easy to follow. \n\n* Experiments support the effectiveness of the proposed AdaCache methodology.", "weaknesses": "* The current experiments are based on MMLU, MMLU-Pro, superGPQA, and TriviaQA, where the question can be answered by single factual chunk. Evaluations on more challenging RAG scenarios that require handling long and/or complex contextual information could be added for revealing the limitation of AdaCache. \n\n* This framework requires the manipulation of KV-cache and attention values, limiting its applications to open-weight models. \n\n* Comparison with [TurboRAG](https://arxiv.org/abs/2410.07590) could be added.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (Poster)