Title: SmartCache: Context-aware Semantic Cache for Efficient Multi-turn LLM Inference
Venue: NeurIPS 2025
Accepted: yes
Keywords: Semantic Cache

Key-Value Cache

Large Language Model
Abstract: Large Language Models (LLMs) for multi-turn conversations suffer from inefficiency: semantically similar queries across different user sessions trigger redundant computation and duplicate memory-intensive Key-Value (KV) caches. Existing optimizations such as prefix caching overlook semantic similarities, while typical semantic caches either ignore conversational context or are not integrated with low-level KV cache management.
We propose SmartCache, a system-algorithm co-design framework that tackles this inefficiency by exploiting semantic query similarity across sessions. SmartCache leverages a Semantic Forest structure to hierarchically index conversational turns, enabling efficient retrieval and reuse of responses only when both the semantic query and conversational context match.
To maintain accuracy during topic shifts, it leverages internal LLM attention scores—computed during standard prefill—to dynamically detect context changes with minimal computational overhead. Importantly, this semantic understanding is co-designed alongside the memory system: a novel two-level mapping enables transparent cross-session KV cache sharing for semantically equivalent states, complemented by a semantics-aware eviction policy that significantly improves memory utilization. This holistic approach significantly reduces redundant computations and optimizes GPU memory utilization. 
The evaluation demonstrates SmartCache's effectiveness across multiple benchmarks. On the CoQA and SQuAD datasets, SmartCache reduces KV cache memory usage by up to $59.1\%$ compared to prefix caching and $56.0\%$ over semantic caching, while cutting Time-to-First-Token (TTFT) by $78.0\%$ and $71.7\%$, respectively. It improves answer quality metrics, achieving $39.9\%$ higher F1 and $39.1\%$ higher ROUGE-L for Qwen-2.5-1.5B on CoQA. The Semantic-aware Tiered Eviction Policy (STEP) outperforms LRU/LFU by $29.9\%$ in reuse distance under skewed workloads.


Peer reviews:
{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper introduces a novel method to store semantically equivalent queries and responses to reduce computation. While previous methods like Prefix caching only work for exact match queries, SmartCache searches for queries based on embedding similarity. The approach also takes history across multiple sessions into account, based on the observation that users typically ask several questions on the same topic, and so the method is able to more efficiently search a narrower space of cached queries based on the initial query. It also has a mechanism to detect context switches if the user changes topic.\n\nSTRENGTHS AND WEAKNESSES:\nStrengths:\n\nThe work is well-motivated, and the solution is logical and well-explained.\nThe paper is very clear and comprehensive in describing all the components of the solution and walking the reader through it – it was a pleasure to read!\nExperiments are carried out on 3 different LLM families and show gains across all of them.\n\nWeakness:\n\nIt feels like the authors ran out of space, and so the experimental descriptions/analysis is not detailed enough. The whole experimental setup, evaluation, analysis and conclusion are crammed into the final 1.5 pages. I still think the authors did just enough to make the paper publishable, but would encourage them to expand this section and move some earlier sections into an appendix.\n\nQUESTIONS:\nIn the experimental design, did you experiment with different similarity thresholds for the embeddings? This seems to me to be a crucial parameter.\n\nI am surprised that answer quality increased significantly using your method. I would have expected the baseline approach to perform best, i.e. when the model is free to generate an answer to every question, and that SmartCache would have sometimes mistakenly presented a cached answer when the similarity was high.\n\nLIMITATIONS:\nNone", "TITLE": "Official Review", "RECOMMENDATION": 5, "REVIEWER_CONFIDENCE": 4, "CLARITY": 4, "IMPACT": 3, "ORIGINALITY": 3, "DATE": "2025-06-23"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper proposes Smart Cache, a system level co-design framework to improve the efficiencies of multi-turn conversations that reduces the TTFT and kv cache useage. This method first cluster the conversation histories into the topic-specific tree “Semantic Forest,” performs a local-then-global ANN search over saved embeddings, and then dynamically decide whether to start a new tree or stays in the current tree based on the attention during prefill. The paper also proposes STEP policy to decide the eviction/ prefetch based on the two-level mapping table. Experiments on CoQA and SQuAD with Llama-3-8B demonstrate up to 59 % KV-memory reduction and 78 % faster time-to-first-token versus prefix caching.\n\nSTRENGTHS AND WEAKNESSES:\n**Strength**\n- The author discusses on an important topic that expand the idea of GPT-cache to the multi-turn conversations and is able to reduce both the kv cache memory and the time-to-first token  \n- Compared to the baselines outlined in the paper, this method achieves better performance than GPT-cache  \n- The author proposes system-algorithm co-design that enables efficient kv-cache reuses  \n\n**Weakness**\n- Limited Scale Evaluation: The evaluation is conducted on relatively small models and datasets. It's unclear how the approach would scale to larger models (e.g., 32B+ parameters) or handle a significant volume of concurrent requests. Some evaluations on larger models, or other large multi-turn conversations, such as Lmsys-chat, would provide a more comprehensive evaluation of this method.  \n- Response quality degradation: As shown in the Table2, even with the smart cache, the performance on the CoQA is still significantly lower than the ideal one.  \n- Privacy concerns: Though the authors have noted the privacy limitations in the paper, no concrete solutions have been proposed to address this concern.  \n- Lack of ablation and breakdown experiments: This paper doesn’t provide enough analysis on how the overhead of the retrieval, possible loading time during PCIE, and ANN search. When the prefix is short and the model is small, these overheads could be significant.  \n- Lack of baselines: More recent paper, such as CacheBlend and PromptCache, also target on the efficient caching for reducing TTFT. These baselines are not included.\n\nQUESTIONS:\nHave you examined how large the semantic tree would be confronting large volume of requests?\n\nWhat happens if the local lookup fails, but a semantically very similar node exists elsewhere in the same tree (i.e., not as a direct child)?\n\nCould you report the efficiency gains under minimal performance losses? People won’t like significant compromise of performance even with significant TTFT gains\n\nLIMITATIONS:\nI think the main limitation is for the experiment sessions. Please see my comments in the weakness parts.", "TITLE": "Official Review", "RECOMMENDATION": 4, "REVIEWER_CONFIDENCE": 3, "CLARITY": 3, "IMPACT": 3, "ORIGINALITY": 3, "DATE": "2025-06-29"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper proposes SmartCache, a novel system-algorithm co-design framework to address the inefficiencies of multi-turn LLM inference. SmartCache introduces a \"Semantic Forest\" structure to hierarchically index conversational turns, enabling efficient retrieval and reuse of responses while preserving conversational context. It further leverages internal LLM attention scores for dynamic context detection and introduces a \"two-level mapping\" for cross-session Key-Value (KV) cache sharing, complemented by a \"Semantic-aware Tiered Eviction Policy (STEP)\". The evaluation demonstrates substantial reductions in KV cache memory usage and Time-to-First-Token (TTFT), alongside improvements in answer quality on various benchmarks.\n\nSTRENGTHS AND WEAKNESSES:\nStrengths:\n\n- Introduces a hierarchical semantic cache (Semantic Forest) that respects conversational context. Bridges the gap between semantic and KV caching by linking semantic nodes directly to KV cache blocks, which is underexplored. Integrates attention-based context detection into the caching pipeline to mitigate misuses due to context-ignorant semantic retrieval.\n- Demonstrates consistent improvements on CoQA and SQuAD benchmarks using three LLMs. Reduces KV cache usage by up to 59.1%, TTFT by up to 78.0%, and improves F1/ROUGE by up to 39%.\n- The two-level KV cache mapping (session → semantic node → block) is a practical innovation that enables transparent sharing without compromising cache coherence or semantic fidelity.\n- The paper is well-organized with clear diagrams and pseudocode, easy to understand.\n\nWeaknesses\n- While the Semantic Forest provides clear benefits, the operational overhead of maintaining and updating this complex hierarchical structure in a highly dynamic, real-time serving environment could be further elaborated, especially concerning potential locking mechanisms or distributed concerns for very large-scale deployments.\n- The threshold for determining context changes based on attention scores might require fine-tuning across different LLM architectures or datasets. A more detailed analysis of its robustness would be beneficial.\n\nQUESTIONS:\nGiven that the paper mentions $t_{sim}$ for embedding similarity, could the authors clarify if a similar explicit threshold or a specific methodology for dynamically determining context changes based on LLM attention scores is employed in SmartCache, and how its sensitivity and generalizability across different LLM architectures and datasets were evaluated?\n\nLIMITATIONS:\nYes", "TITLE": "Official Review", "RECOMMENDATION": 5, "REVIEWER_CONFIDENCE": 3, "CLARITY": 2, "IMPACT": 2, "ORIGINALITY": 3, "DATE": "2025-06-30"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper introduces SmartCache, a system-algorithm co-design framework aimed at improving multi-turn LLM inference efficiency by addressing redundant computation and memory usage caused by semantically similar queries across sessions. Unlike methods that require exact matches or ignore conversational context, SmartCache integrates context-aware semantic caching with the inference engine. Its key contributions include a hierarchical semantic forest for accurate reuse of responses, an attention-based module for detecting topic shifts, and an integrated KV cache management system with cross-session sharing and a novel STEP eviction policy. Experiments show that SmartCache significantly reduces KV cache usage and TTFT while improving answer quality.\n\nSTRENGTHS AND WEAKNESSES:\n### Strengths\n1. High Significance: The paper tackles a crucial problem in LLM serving—inefficiency in multi-turn conversations caused by semantic redundancy.\n2. Novel and Holistic Design: The integration of semantic caching and KV cache management reflects a thoughtful system-algorithm co-design, offering a comprehensive solution that surpasses prior approaches in both scope and capability.\n3. Strong Empirical Results: SmartCache outperforms baseline, prefix, and flat semantic caching methods in terms of memory efficiency, TTFT, and answer quality across multiple models and datasets.\n\n### Weaknesses\n1. Lack of Overhead Quantification: While the authors claim minimal overhead, the paper does not provide a detailed breakdown of the costs associated with embedding computation, vector search, and forest maintenance relative to performance gains.\n2. Limited Evaluation Under High Concurrency: The main benefit of cross-session KV cache sharing appears in high-throughput environments, yet the paper lacks experiments simulating batch inference or parallel request scenarios.\n3. Hyperparameter Sensitivity Not Addressed: The system likely depends on hyperparameters like τ_sim, α, and β, but the paper does not include an ablation study or sensitivity analysis to show robustness.\n\nQUESTIONS:\n1. Overhead Analysis: Can the authors provide quantitative measurements on the computational and memory overhead introduced by embedding generation, vector search, and semantic forest operations?\n2. Attention-based Context Switching: Could you elaborate on how the system distinguishes between \"progressive\" and \"individual\" queries? Is there a fixed threshold on attention scores, and how robust is this heuristic across different models?\n3. Performance under High Concurrency: Have you evaluated SmartCache under high concurrency or batch inference settings? Since cross-session reuse is central to the system, understanding its behavior under realistic serving loads would be valuable.\n\nLIMITATIONS:\nYes.", "TITLE": "Official Review", "RECOMMENDATION": 5, "REVIEWER_CONFIDENCE": 4, "CLARITY": 3, "IMPACT": 3, "ORIGINALITY": 3, "DATE": "2025-07-02"}