SmartCache: Context-aware Semantic Cache for Efficient Multi-turn LLM Inference
Authors: Chengye YU
Tianyu Wang
Zili Shao
Song Jiang
Venue: NeurIPS 2025 poster ()
Keywords: Semantic Cache
Key-Value Cache
Large Language Model
Primary area: infrastructure

Abstract: Large Language Models (LLMs) for multi-turn conversations suffer from inefficiency: semantically similar queries across different user sessions trigger redundant computation and duplicate memory-intensive Key-Value (KV) caches. Existing optimizations such as prefix caching overlook semantic similarities, while typical semantic caches either ignore conversational context or are not integrated with low-level KV cache management.
We propose SmartCache, a system-algorithm co-design framework that tackles this inefficiency by exploiting semantic query similarity across sessions. SmartCache leverages a Semantic Forest structure to hierarchically index conversational turns, enabling efficient retrieval and reuse of responses only when both the semantic query and conversational context match.
To maintain accuracy during topic shifts, it leverages internal LLM attention scores—computed during standard prefill—to dynamically detect context changes with minimal computational overhead. Importantly, this semantic understanding is co-designed alongside the memory system: a novel two-level mapping enables transparent cross-session KV cache sharing for semantically equivalent states, complemented by a semantics-aware eviction policy that significantly improves memory utilization. This holistic approach significantly reduces redundant computations and optimizes GPU memory utilization. 
The evaluation demonstrates SmartCache's effectiveness across multiple benchmarks. On the CoQA and SQuAD datasets, SmartCache reduces KV cache memory usage by up to $59.1\%$ compared to prefix caching and $56.0\%$ over semantic caching, while cutting Time-to-First-Token (TTFT) by $78.0\%$ and $71.7\%$, respectively. It improves answer quality metrics, achieving $39.9\%$ higher F1 and $39.1\%$ higher ROUGE-L for Qwen-2.5-1.5B on CoQA. The Semantic-aware Tiered Eviction Policy (STEP) outperforms LRU/LFU by $29.9\%$ in reuse distance under skewed workloads.

SmartCache: Context-aware Semantic Cache for Efficient Multi-turn LLM Inference
Chengye Yu Chinese University of Hong Kong Hong Kong, China cyyu23âcse. cuhk. edu. hk
Tianyu Wang Shenzhen University Shenzhen, China tywangaszu. edu. cn
Zili Shao
Song Jiang University of Texas Arlington Arlington, USA song j jiangSuta. edu
Chinese University of Hong Kong Hong Kong, China shao8cse. cuhk. edu. hk
Abstract
Large Language Models (LLMs) for multi-turn conversations suffer from ineffi- ciency: semantically similar queries across different user sessions trigger redundant computation and duplicate memory -nttenivi Key-Vai Value (KV) caches. Existing optimizations such as prefix caching overlook semantic similarities, while typical semantic caches either ignore conversational context are not integrated with low-level KV cache management, We propose SmartCache, a system-algorithm co-design framework that tackles this inefficiency by exploiting semantic query similarity across sessions. SmartCache leverages a Semantic Forest structure to hierarchically index conversational turns, enabling efficient retrieval and reuse of responses only when both the semantic query and conversational context match. To maintain accuracy during topic shifts, it leverages internal LLM attention serrss-computer during standard prefill-to dynamically detect context changes with minimal computational overhead. Importantly, this semantic understanding is co-designed alongside the memory system: a novel two-level mapping enables transparent cross-session KV cache sharing for semantically equivalent states, com- plemented by semantics-aware eviction policy that significantly improves memory utilization. This holistic approach significantly reduces redundant computations and optimizes GPU memory utilization. The evaluation demonstrates SmartCache's 's effectiveness across multiple benchmarks. On the CoQA and SQuAD datasets, SmartCache reduces KV cache memory usage by up to 59.1% compared to pre- fix caching and 56.0% over semantic caching, while cutting Time-to-Fine First-Token (TTFT) by 78.0% and 71.7%, respectively. It improves answer quality metrics, achieving 39.9% higher Fl and 39.1% higher ROUGE-L for Owen-2.5-L.5B on CoQA. The Semantic-aware Tiered Eviction Policy (STEP) outperforms LRU/LFU by 29.99 29.9% reuse distance under skewed workloads.
1 Introduction
Transformer-basee Large Language Models (LLMs) 39 such as GPT 30 PaLM [], and Llama 37 16). are revolutionizing the world with their powerful and versatile capabilities, enabling a wide range of tasks ranging from question answering to code generation [45] 30 47). A particularly prominent application multi-turn conversation [ users engage with models over extended dialogues, often exploring specific topics in depth (3) This
18 25]. underpinning interactive systems where
39th Conference on Neural Information Processing Systems (NcurIPS 2025).

---

Questions
Geserated Answers
(aached Answers
Basiie
CPiCach
Whatin Whatis byfonmb?)
piturs
How zacil!
Telime Reattylordh
Caci
Smantacae
SnurtCacha
PyTorchis..
To PyTurch......
1tt Plore Itis Eateeve
Cocha mial
230
190
Pieee
TIT 2600
100
+
Vhatis Treacrflow!
Tessuflon
How to it
2
Netted
Srewsete
30
Getrrl Ipeeeree
Cache tabe arowort
Messyy
Cosguatc
(a)
(b)
(c)
Figure 1: I(a illustrates the functionality of SmartCache. [(b) shows an example of multi-user
multi-turn conversations. GPTCache [ would encounter false positive cache hit for Session
Turn 2 due to flat query embedding search. PrefixCache [40] fails to reuse the answer of Session 1
Turn 1 for the semantically similar query of Session Turn due to exact token match. [[c]]shows the resource consumption of different methods on SQuAD dataset with LLama-3.1-88 model.
paradigm is central to applications like collaborative content analysis and comprehension, where users interact with materials like news articles or videos through conversational interfaces.
However, current LLM serving systems suffer from a fundamental inefficiency in this setting: semantic redundancy. Different users in separate sessions often ask questions that are semantically cquivalent or highly similar. Existing systems treat each session independently. Semantically similar queries often trigger separate, full LLM inference passes, redundantly producing identical or nearly identical responses. This results in unnecessary GPU computation. Furthermore, the associated Key- Vey-Value (KV) secches-sssentalll for efficient token generation yet highly memory- memory-intensive are maintained separately for each session, resulting in excessive GPU memory usage and added overhead from memory management tasks such offloading [14 19
Existing optimization techniques are inadequate to address the problem. Prefix caching methods, proposed in RadixAttention [48] and commonly used in systems like vLLM 40) 19 27
M. leverage shared token prefixes (e.g., system prompts) share initial KV cache blocks. They rely on exact token matching and fail to recognize semantic equivalence when queries are phrased differently (c.g., "What PyTorch? vs. "Tell me about Py PyTorch" as shown in Figure l Their benefits are also largely confined to the initial prompt processing (prefill stage), offering little help for the time-consuming token-by bb-tken decoding phase. On the other hand, higher-level semantic caching systems [6] operate by storing text responses indexed by query embeddings. While they capture semantic similarity, they often struggle in multi-turn scenarios due two key limitations. (1) Context Ignorance. They often retrieve cached responses based solely on the current query's ss embedding, disregarding the preceding conversation turns. This leads to incorrect or nonsensical answers when query's meaning is context-dependent (c.g.. "How to use it?"). (2) Decoupling from Inference State. These caches operate above the LLM inference engine. cache hit provides text but does not update the LLM's internal KV cache state. A subsequent cache miss force the LLM to reprocess the entire conversation history to regenerate the KV cache, resulting in significant computational overhead and undermining potential memory savings. There is clear need for solution that understands semantics, preserves conversational context, and is tightly integrated with the underlying inference and memory management mechanisms. As depicted in Figure I(c) utilizing semantics to further reduce resource usage remains largely unexplored.
15 4
To bridge this gap, we present SmartCache a system-algorithh co-design framework purpose- built to optimize multi-turn LLM inference by leveraging cross-session semantic similarity while preserving contextual integrity. SmartCache tightly integrates novel semantic caching layer with the core LLM serving engine and its KV cache management. built upon several key design principles and mechanisms.
Semantic Forest. To ensure cached responses are used only in the appropriate context and to efficiently manage the branching nature of conversations, SmartCache employs the Semantic Forest data structure (Section 3. This hierarchy trees organizes conversational turns (query-response pairs) semantically. Lookups for incoming queries are constrained to relevant branches based on the preceding turn, ensuring contextual accuracy and efficient search. Cache misses dynamically grow the forest, capturing new conversational paths.

---

-Attnttinn-bsel Context Identification. Recognizing that conversations can dynamically shift topics, SmarCaahhe incorporates an attention- based context identification module (Section 3.2). By analyzing attention patterns during standard query prefill-a a low-overhead signal from the model's internals-SmarrCann SmartCache distinguishes between queries that extend the current context and those that begin a new one. This allows sessions to intelligently navigate or transition between different semantic trees, enhancing both cache effectiveness and response accuracy.
Semantic-aware Tiered Evietion/Prefetching, To maximize resource savings, SmartCache features a co-designed KV cache sharing and eviction mechanism that deeply integrates the semantic layer with memory management (Section 3.3). A novel two-level mapping connects semantic nodes to physical KV cache blocks, allowing multiple sessions referencing the same conversational point to transparently share underlying GPU memory. Furthermore, instead of relying on blind page-level or session-level eviction, SmartCache employs lightweight and powerful Semantic-aware Tiered PeietionPreeeeching (STEP) policy. STEP manages the lifecycle of KV cache blocks using factors derived from the Semantic Forest structure and usage patterns, including semantic depth, session affinity, and node popularity. This refined, contextually-inffrmen approach optimizes memory utilization more effectively than traditional page- or session-based methods.
To evaluate the effectiveness of SmartCache, we conduct extensive experiments across multiple benchmarks and models. On the CoQA and SQuAD datasets, SmartCache reduces KV cache memory usage by up to 59.1% compared to prefix caching and 6%.0% over semantic caching, while achieving 78.0% and 71.7% reductions in Time-to-First- -or-tt---rtt-T (TTFT). respectively. SmartCache also improves answer quality metrics, attaining 39.9% higher F1 and 39.1% higher ROUGE-L. scores for the Qwen- 2.5-1.5B model on CoQA, demonstrating its ability to preserve contextual accuracy. Furthermore, our STEP policy outperforms traditional LRU/LFU policies by up to 29.9% in reuse distance under skewed workloads, validating its efficiency in dynamic multi-session environments. These results highlight SmarCCache's 's ability to balance computational efficiency, memory optimization, and answer quality across diverse conversational scenarios.
2 Background
2.1
Related Works
Several prior works have explored various methods address the inefficiencies in multi-user multi- turn conversations. Table Tableecomme compares different caching strategies used in multi-turn conversations.
Table 1: Comparison of Caching Strategies for Multi-Turn LLM Inference.
Baseline
PrefixCache
Flat Embedding
Context-aware Semantic Cache
Feature
No Reuse
e.g. vLLM APC 40 Radix
e.g. GPTCache [6]
SmartCache (ours)
4%
SCALM 21 and
Reuse Trigger
X
Exact Prefix Match
Flat Similarity
Hierarchical Semantic
Aware
Limited'
(Hierarchical)
Resource Reused
x
KV Cache"
Text Response only
Text Response KV Cache
Computarion Suving
Limited
Integrated System/Algoritmm
(Co-designed)
Key Battleneck
High Redundancy Simplicity
No Semantic Reuse
Context Ignorunce Saves Pull Compute
Benetit Focus
Prefill Speeduu
Context Awareness Memory/Compute Elf:
GPTCache can use history for embedding, but lacks explicit conversational structure. *Only exactly matched prefix tokens are reused.
2.2 Problem Description
Let be the set of active user sessions. Each session & E S arriving at turn + has a conversational history (k) consisting of the preceding k query-responss pairs. We can represent this history as sequence or concatenation: (k) {(g.. at)) or, if treated as single string for embedding purposes, H!{) Ri I| | (qilla;), where || denotes string concatenation. When session issues the new query 1, the goal is to efficiently determine the appropriate response ag+1-
[H! ), where M is the LLM and H+()) represents the relevant history context for session s. This computation expensive.
standard LLM serving system computes d=+-1 = M
flat query-answer cache like GPTCache aims to find if there exists a previously asked query aprev (from any turn in any previous session) in a global set of all past queries Qau such that
3

---

sim (0((+)))))) E(qprew)) Tasm, where E is an embedding function, sim is a similarity metric, and Taim is a threshold. If such Sprer with associated response Oprev is found, dg++ to aprew
Find (Aprev, (prev) s.t. aprev E Qail A sim( E(Aprew)) 2 Tsim
The primary limitation is that it disregards the conversational context H! The semantic meaning of qe+11 is often dependent on H and reusing Oprep generated under different context H (K) can be incorrect, even if fi++1 and Oprey are textually similar.
One might attempt to address this by directly incorporating the conversation history (k) in the query embedding. flat semantic cache could store embeddings of the form equery E(H
(8) [[x+t)) For incoming query 4 with history H!k), the system would compute E (114/41)) and search for stored embedding Extored E(H (9gpre)) in global index, such that sém((' (imm(guery.hiin cstored) Tsim-
'query_bie hist
While this approach includes history information, it suffers from the Long History Dominance 23 32 38 [22 [0] issue, especially as the number of turns i increases. Standard embedding models struggle to effectively represent long concatenated sequences in fixed-size vector such that the final
query 4u+11 remains distinctly identifiable, As the history H(()) grows long, the embedding vector E(H (K' 114)+11) tends to be increasingly dominated by the content of the history H, itself, rather
than the specific new query Gnews i.c. E (k') (1 is closer to E (k)) but more distant from 1(
3 The Design of SmartCache
In this section, we introduce SmartCache, context-aware semantic cache designed for multi-turn LLM inference with enhanced context-awareness. First, illustrated in Fig- ure 2 SmartCache organizes and indexes con- versation contexts from different users into Semantic Forest, as detailed in Section 3.1 The semantic forest comprises multiple individual semantic trees, where each conversation turn represented a semantic node. Semantic nodes can be associated with and reused across multiple conversation sessions, enabling effi- cient context sharing. Second, in Section 3.2 SmartCache introduces a semantic identification policy that leverages attention scores from the prefilling stage of user queries to identify non- contextual queries, to support dynamic context switching, and unlocks additional sharing opportuni- ties. Finally, we present the transparent cross-session KV cache sharing design, coupled with a novel eviction policy, enabling efficient sharing of KV caches based semantic similarity in Section 3.3
Semantic Forest
Semantic Tree
Extoodii Vacter
Vepooggaalee
Rnighase
SemanticManager
Nude
SessiunID
Node
Node
LLM
KVCacheManager
Figure 2: Overview of SmartCache.
3.1 Semantic Context Manager
3.1.1 Semantic Tree
In a multi-turn conversational scenario, users often begin session with simple question specific topic, followed by a series of progressively deeper queries. Different users may explore the same topic by asking similar questions in varied ways by focusing on different aspects. Such relationships can be effectively represented using a tree structure, referred to as a semantic tree in SmartCache. Each semantic tree organizes guery-response turns semantic nodes under specific semantic context. The initial query of a new session becomes the root node of a new semantic tree, while subsequent queries create child nodes linked the corresponding semantic parents.
Conversation sessions centered on the same topic can share a common semantic tree, allowing them to leverage cached responses when similar queries have been made by other users. Each conversation

---

(a)
(b)
(c)
(d)
Query with cached response
Current position of session
Query with generated response
Semantic node
Figure 3: Example of hierarchical semantic shared by three sessions (A, B, and C).
session is associated with a semantic node, representing the last query-response turn of the session. Users can ask progressive questions based on the context represented by the semantic node. these cases, child nodes evaluated to possibly reuse responses from similar queries, avoiding the need for regeneration. When there are no reusable responses, the query is processed by the LLM using the contextual history traced from the root to the current semantic node. In the case where user poses an unrelated question and shifts to a different topic, SmartCache either locates a suitable existing semantic tree initiates
to reflect the updated context.
Figur 3 jillustrates a comprehensive example of semantic tree shared across three sessions (A, B, and C). At time T1, session A begins with query Q1, which is processed by the LLM to generate response, as it the first query on the topic. Q1 and generated response are cached as semantic node 1, which becomes the root of the newly created semantic tree. At T2, session B issues Q1', query semantically similar to Q1, and retrieves the cached response from semantic node 1. Mcanwhile, session A raises query Q2 and receives newly generated response. At T3, session C initiates Q1" and retrieves the same cached response from semantic node 1. However, when session raises query Q4, similar query is found among the children of node the current position of session B. As a result, its response is generated by the LLM, and a new branch from node 1 to node 4 is created. Session C continues reuse node 4 until it encounters cache miss at query Q6.
The semantic tree design benefits multi-turn conversations two aspects. First, by reusing semanti- cally similar nodes across different sessions, queries with identical similar semantics are computed and decoded by the I.LM only once, and their responses are shared. This approach significantly re- duces computational overhead and alleviates memory pressure, as the KV cache of common semantic nodes is shared among sessions. Second, the hierarchical structure the semantic tree preserves semantic relationships. The tree grows and the session advances level-by-level with the progress of queries. Queries are compared and retrieved only within the context of the same parent node, ensuring accuracy while limiting the search space. This design allows sessions to share several initial queries at the beginning while enabling diverse and specific questions to branch out independently as conversations progress.
3.1.2 Semantic Node
Each semantic node represents the context of single conversation turn. As shown in Figure 2] the embedding vector of the query, generated by an embedding model is used by the parent node for similarity search. Consequently, each semantic node stores the embedding vectors of its child nodes in vector index [11 24 41 19.. The KV cache for the associated context is maintained on the LLM
side. To enable transparent KV cache sharing, an additional indirect KV cache block mapping table is introduced and managed by the corresponding semantic node, which is detailed in Section 3.3
3.1.3 Semantic Forest
Multiple semantically individual trees collectively form the semantic forest. A global vector index stores the embedding vectors of the root nodes all semantic trees, which is searched by the initial query of new conversation session to locate the most relevant semantic tree. If no similar topic is found, a new semantic tree is created.
Users may occasionally change topics within a single session, such as shifting from one topic to another. In SmartCache, an active session can switch to another semantic tree no similar queries are found under the current context and the new query is deemed irrelevant to the ongoing conversation.
5

---

To determine whether query is contextual or not, SmarCCache leverages the attention score from the LLM's query prefilling stage, enabling dynamic and accurate context switching.
Algorithm Cache Operations
Require: Session New Query quews Semantic Forest .F (access to global and local Ix), Similarity threshold Teims Embedding Model E(.)
Ensure: Boolean is hit, Node Naat Response Ocached
1: function CACHELOOKUP(S, Grew: F, Tsimm)
2: Nour CurrenlNodell
3: Cnew f- F((new) )
4: is is_hit false
5:
Nhit + null
6:
acoched f null
7:
if' Neur = STA then
8:
Condidates +- Seearest.eigghor(eeer. (enews Iglobat: Ic 1, Tsim if Condidates is not empty then
9:
Nbu - Candidates (N. NNode
11:
Gcached f N
hit true
13:
else
if N Neur has local index I then Candidates N www.essseeggbborreeeeee Newr = 1, Teim)
15:
16:
Candidates not empty then
17:
Nkat +- (ondiddatss()).NVV
18:
acached f Naa-a
19:
is hit + true
return 88 hit,, Nhtty hit, aoached
3.1.4
Cache Operations
SmartCache's 's cache operation is responsible for deciding whether an existing semantic node can be reused or whether the underlying LLM must be invoked to extend the semantic tree, whenever new conversation turn arrives. The procedure (as shown in Algorithm T consists of three phases: local lookup, global fallback and miss handling.
Local Lookup. Every session s records its current semantic node Neur. The search scope is constrained to the descendants of Neur. which preserves the conversational continuity and bounds nearest-neighbor query on the local index IN..r finds the
the search space. A single-probe most similar child with minimum similarity of Tsim using the embedding vector given by the embedding model. On cache hit, the cached response aeached the hit node is streamed to the user and Newr updated.
Global Fallback. On failure of local lookup, SmartCache suspects a context shift. Before falling back to L.M generation, it performs root-level lookup on the global vector index that stores the root nodes of semantic trees. A successful match moves the session to the hit root and returns cached response.
Miss Handling. A double-miss triggers LLM inference and semantic node creation: a) Attention Peeking. After a single-pass prefilling, attention scores to historical tokens are inspected to determine context-switch. If the query contextual, the decoding preceeds. Otherwise, new semantic tree is created and the generation is restarted without context. b) Node Creation and KV Cache Materialization. SmartCache inserts new semantic node to an appropriate position with newly generated response Onews and corresponding KV cache blocks, updating vector indices and setting up correct mappings.
3.1.5 Analysis and Discussion
Let d be the embedding dimension of vector index, [F] the number of semantic trees in the forest, and [B] the average branching factor semantic node, the average generation length, and V the vocabulary size.

---

Query
25
50
75
100
125
150
175
Key
(a) A progressive query based on the context.
Query
25
50
75
100
125
150
175
Key
(h) An individual query
Figure 4: The attention score heatmaps of two different queries to the same context.
Asymptotic Time Cost. SmartCache performs at least upproximatee-nerrstt-niiee
24
search over:
gaoobal O(log [F]) or,
Hocat O(log [B)
Space Overhead. For each semantic node, SmartCache stores a) a d-dimensional embedding vector (4d Bytes in FP32 format), b) cached response dcached (O(T log [V]) Bytes) and c) local vector index /tocat consisting of index metadata (0 (O(IneeSSzee((B)) Bytes) depending on the ANN implementation and the pointers to descendant embedding vectors (8) B Bytes). Therefore, the space overhead of each semantic node is:
O (d (d+Tloog V IndexSize((BB) +l
3.2 Dynamic Context-Switching
In multi-turn conversations, the sequence length of a user query is typically much shorter than that of the generated response. Additionally, prefilling sequence is significantly faster than decoding the same number of tokens due to the batch processing nature of the prefilling stage. As a result, query prefilling consumes far less time than response generation. The attention score reveals the model's is focus on key tokens, with contextual queries exhibiting noticeably higher attention scores the context compared individual queries, as illustrated in Figure This observation motivates approach of quickly examining the attention score during the query prefilling stage determine whether a query is contextual. the query is contextual, the LLM proceeds to decode the response. Otherwise, the query is treated an individual query, and the session is redirected to another semantic tree. Since the attention score is already computed during prefilling, the performance overhead of this "peeking" mechanism is negligible.
By switching semantic tree with lighter contexts, ses- sions achieve better response times due to reduced com- putational load. Furthermore, the semantic forest expands with additional individual semantic contexts, increasing opportunities for sharing and improving overall efficiency.
Session
Session
Session
Seusion
Semantic
Tree
3.3 KV Cache Management
3.3.1
Transparent Cross-Session KV Cache Sharing
SmartCache enables transparent sharing of KV cache across semantically similar conversation sessions by in- troducing a two-level mapping table. Original paged KV cache management relies on per-session logical- to-physical token block mapping table, often referred to as block table. As illustrated Figure SmartCache enhances this approach with two additional structures: per-session context tables, which map logical tokens to contexts, and per-context block
Physical
Block
Block
Block 2
Block
Block
Block 5
Block
Block ?
Block 8
Blocks
Figure 5: Two-Level Mapping Table.
7

---

tables, maintained by each semantic node. This two-level mapping mechanism ensures efficient and flexible KV cache sharing while preserving semantic relationships.
3.3.2 Semantic-aware Tiered Eviction/Prefetching
Semantic Tree-Guided Eviction. SmartCache evaluates the eviction urgency Sesiet of semantic node by two factors: Semantic Depth Score (SDS) and Session Affinity Score (SAS). The SDS is calculated as SDS Depib(N) following the heuristic that deeper nodes in the semantic trees are prioritized to be evicted. The SAS, computed by SAS (N)) implics that nodes shared across fewer active sessions should be evicted first. The weighted eviction score is as follows:
Seviet = a. -DS-SDD- SAS
Subtree-aware Prefetching, When session advances semantic node N. SmartCache prefetches its children C' into the GPU memory based on the Node Access Frequency (Popularity) It is calculated using statistics within predefined time window, given that hot topics may change over time.
AF(C,)
#sessions accessed C. after
#sessions reached N
0wee2.51..5 1.58-lemme
Lime33 ludrot
Misuel Iestrct
200
100
100
CoRA
SQNAD
ColA
SQLAD
CoSA
SQUAD
(a) KV Cache Usage
Bailiar
Profix Cacle
GFTCxce
SourChdh
Qwent 51 1h-nmes
200
Mieral-TB--uine
T1F
100
120
50
SQUAD
SQUAD
ColjA
SQUAD
(b) Average Time To First Token
Figure 6: Resource efficiency comparison on different models and datasets.
Evaluation
4.1 Experiment Setup
Harward settings. We use server equipped with Intel Xeon Silver 4310 CPU 17 and NVIDIA RTX4090 GPU (26), connected through PCIe4.0x 16. The host has 256GB DDR4 memory and GPU has 24GB GDDR6X memory. Our server runs Linux 5,4 with CUDA 12.0 [5). We implement our
method using PyTorch 2.3 [29]. We evaluate our method on three different-sized open-source LLMs:
Qwen-2.5-1.55 -.55-.--..Bnstrre 433, Llama-3. 1.8--..-ttttI-ttccc [16], and Hiistrl-7B-IInstrece
3]
Dataset. Three datasets used in the evaluation, including the CoQA dev dataset 33 and
SQuAD2.0 dev dataset [31]. CoQA is a conversational question answering dataset where questions asked sequentially on the same short story. Later questions often depend conversation history. SQuAD (Stanford Question Answering Dataset) contains questions on Wikipedia pages. Each story or paragraph used in the experiments has original conversation sessions and two similar conversation sessions, with each session consisting of on average 5 turns of progressive question and answers.
Four methods are compared in the evaluation: 1) Baseline: KV caches not shared among different conversations sessions, 2) PrefixCache: KV caches common prefixes from different conversation sessions are shared, 3) GPTCache: an extra query-answer cache layer using the embeddings of queries as vector search keys is stacked on top of LLM inference backend, 4) SmartCache: the hierarchical and co-designed cache proposed in this paper. Conversation queries are issued interleavingly among different sessions.
Methods.
The KV cache block size is 16 tokens. BGE-M3 7 is used as the embedding model with the dimension of 1024. Embedding vectors are stored and searched using faiss [T]] vector index based on L2 distance. The similarity threshold is set 0.75,

---

Table 2: Answer Quality. Ideal represents the results of Baseline and PrefixCache.
Benchmark
CoQA (0-shot)
SQuAD2.0 (0-shot)
Model
Method
ExactMatch
FI
ROUGE-L
ExactMatch
F1
ROUGE-L
Idcal
27.5
5L.
51.3
14.2
65,5
64.0
Qwen-2.5.15 2.0-2......-Inssruurr
GPTCache
16.8
33.3
33,8
12.8
60.4
59.2
SmariCache
24.9 (+48..2%)
46,6 ()39.9%)
(+.3..1))
13.4 (+4.7%)
64.5 (+6.8%)
63.0 (+6.4%)
Idcal
36.6
59.6
59.2
14.9
67.8
66.0
Llama-3.1-88 1-8---11----steeee
GPTCache
23.0
38.8
39.1
134
62.3
60.9
SmariCache
(+8.3%)
49.5 (+20.1 %)
49.2 (+20.2%)
14.0 (+4.5%)
66.5 (+6.7%)
(+6.1
Idcal
5.5
27.0
26.9
$7.8
55.3
Mistral-BBlnnstreet
CPTCeche
3.0
17.4
17.7
52.7
50.6
SmartCache
3.1 (+3,3%)
20.0 (+14.9%)
20.0 (+13,0%)
7.7 (+11.6%)
(+6.6%)
54.6 (+7.9%)
4.2 Resource Efficiency
SmartCache significantly improves KV cache sharing by leveraging semantic relationships in multi-turn conversations, enhancing resource utilization across sessions. As shown in Figure 6(a) SmarCache reduces KV cache memory usage by up 59.1% and 56.0% compared to PrefixCache and GPTCache, respectively, demonstrating superior efficiency in reusing KV cache memory.
Memory Efficiency.
Computational Efficiency, Figure 6(b) presents the average Time to First Token (TTFT) per query. On the CoQA dataset, across different models, SmartCache achieves 78.0% and 37.7% lower TTFT than PrefixCache and GPTCache, respectively. Similarly, on the SQuAD dataset, it reduces TTFT by 71.7% and 50.6% compared to the same baselines. These results highlight SmartCache effectiveness in accelerating inference while minimizing computational overhead.
4.3 Answer Quality
The hierarchical semantic awareness of SmartCache significantly improves its answer quality, com- pared with GPTCache's flat query similarity search. As demonstrated in Table 2 SmartCache consistently enhances answer quality across different benchmarks and metrics. For the CoQA dataset, SmartCache improves Qwen-2.5-1.50 -nsttuuts 's F1 score by 39.9% and Llama-3, 3.B- 3--IIntructt ROUGE-L by 20.2%. For SQuAD2.0, SmarCCache achieves smaller but consistent gains.
Zipfian Distribution (Q-1.5)
Uniform Distribution
Pareto Distribution (k-2.0)
1.0
Conuulative Probubility
0.8
0.6
0.4
LRU: Mean-205 8, P90-466.0
LRIE: Mcan-21866 P90-490.3
LRU: 190.5.19900... 5, P90-4075
LFU: Menn-211.7, P90-484.0
LFU: Meun-222.8, P90-506.0 0
LFU:
19,.8, Mema-1ee-....
P90-410.5
0.2
Mean-265 P90=619.4
STEP: Mean=258 P90-569.0
STEP: Mcmn=247.4 PM0-563.4
0.0
0
500
1000
500
1000
1500
500
1000
1500
Reuse
Figure 7: Reuse Distances CDF of different KV Cache Management Policy
4.4 KV Cache Management Policy
We compare SmartCache's STEP policy with commonly used LRU (Least Recently Used) and LFU (Least Frequently Used) policies. 1500 conversation sessions focusing CoQA's stories follow three different distributions (zipfian, uniform and pareto) to show different real-world environments. Llama-3.1 3.1 3-B-BB--BBBBB-- Instrrcttt model is used with a GPU KV Cache budget of 4GB. Figure 7 compares the CDF (Cumulative Distribution Function) of the Reuse Distances (the number of distinct nodes accessed between
consecutive accesses the same semantic node) of different policies. STEP shows up to 29.9% and 25.7% longer average reuse distance over LRU and LFU with skewed distribution and 18.2% and 15 15.9% longer with uniform distribution.
4.5 Overhead Analysis
We quantitatively show Smart compute and memory overhead on Mistral-7B-Instructe
using CoQA. Table demonstrates the breakdown of overhead across SmartCache components. For a rache-missed new query (i.e. creating a new semantic node followed by L.MM inference), Smart- Cache adds 28.8% computational overhead (18.54 44 out of 210.37 ms end-to-end), SmanCache's

---

hierarchical semantic organization and co-design with KV cache management increase cache reuse opportunities, leading significant end-to-end gains despite modest overhead.
The total GPU memory overhead is 27.1%, primarily from the embedding model, while CPU memory overhead is negligible (=:28.05 MBB. Given the reduced inference time from cache reuse, the amortized cost of these components becomes insignificant at scale.
4.6 Performance under High Coneurrency
As shown in Figure we evaluate SmartCache under varying degrees of concurrent requests using Wistral-BBIInstruct-000. on CoQA. As the number of concurrent requests increases, SmartCache demonstrates progressively greater speedups. As concurrency increases, the LLM's s batching effi- ciency plateaus due to inherent GPU parallelism limits. However, SmartCache is able to bypass full LLM inference for many cache-hit queries, reducing the effective batch size. This not only alleviates load on the inference engine but also leverages hicrarchical KV cache reuse, leading to significant reductions in TTFT.
(ms)
3000
2000
Beseline
PrefiCachhe
7.9x speedup
Table 3: Computational and Memory Overhead Breakdown of SmartCache.
SmanCaahh
1000
Components
Computational Overheal
Memory Overhead
Embelding Generation
24
8
32 Concurrent Requests
64
Vector Search
0.11 ms
18. MB (CPL)
Vectre Insertion
1.01 ms
Shared with vector search
Figure 8: Overhead under different
Semantic Forest Maintenance
0.29
9.89 MB
number of concurrent requests.
5 Related Works
PrefixCache 40) reuses KV cache only when token prefixes match exactly, accelerating prefill but is limited handling semantically similar queries. GPTCache 6 stores and retrieves responses based on flat query embeddings, enabling semantic reuse but lacking context awareness. PromptCache (I5] accelerates inference by reusing attention (KV) states for frequently repeated prompt segments via a modular schema that preserves positional consistency, substantially reducing TTFT for long, template-like inputs. CacheBlend (44) targets RAG workloads, enabling chunk-level KV cache reuse across retrieved documents and selective recomputation of small subset of tokens refresh cross-chunk attention without incurring full prefill cost. Notably, SmartCache can be combined with CaeneBlend and PrompiCache to achieve further improvements.
6 Conclusion and Discussion
In this paper, we presented SmartCache, a context-aware semantic cache framework designed to address the inefficiencies of multi-turn LLM inference. By integrating hierarchical semantic indexing with KV cache management, SmartCache significantly reduces redundant computations and GPU memory usage while preserving conversational context.
Limitations and Future Work. First, sharing responses across user sessions introduces potential pri- vacy risks. To mitigate this, a content desensitization module could be implemented to automatically remove or anonymize sensitive information prior to caching. Second, scaling the system to support broader range of services requires a multi-instance extension, enabling more robust and distributed processing capabilities.
Acknowledgments and Disclosure of Funding
The work described in this paper is partially supported by the grants from the Research Grants Council of the Hong Kong Special Administrative Region, China (GRF 14202123, GRF 14200224).
10

---

References
[1] Google Vertex AL
Context caching overview. https://cloud. www.gooooogg.ooooo www.c///w/eeeeaaaaaee ai/
Denerative-ai/tocs/// http:///ooteet-ccccccctttttetee. caceeverrii overvies
[2] Joshua Ainslic, James Lee-Thorp, Michiel de Jong, Yury Zemlyanskiy, Federico Lebron, and Sumit Sanghai. GQA: Training generalized multi-query transformer models from multi-head checkpoints. In The 2023 Conference on Empirical Methods in Natural Language Processing, 2023.
[3] Q Jiang Albert, Alexandre Sablayrolles, Arthur Mensch, Chris Bamford, and Devendra Singh Chaplot. Mistral 7b, arXiv preprint, 2023.
[4] Anthropic. Prompt caching.
claude/
prompt-caching
[5] Rob Armstrong, Cuda toolkit 12.0 released for general availabiliity. 2022. https: http://veeolloe...eeeee con//blog/cudattolllle toolkit- 12nit--12---0
12- 0- rew.eeeeeeeeeeeee- for general- avai availability/
[6] Fu Bang. GPTCache: An source semantic cache for LLM applications enabling faster answers and cost savings. In Liling Tan, Dmitrijs Milajevs, Geeticka Chauhan, Jeremy Gwinnup. and Elijah Rippeth, editors, Proceedings of the 3rd Workshop for Natural Language Processing Open Source Software (NIP-OSS 2023), pages 212-218, Singapore, December 2023. Association for Computational I Linguisties.
[7] Jianlyu Chen, Shitao Xiao, Peitian Zhang, Kun Luo, Defu Lian, and Zheng Liu. M3-embedding: Multi- linguality, nultiifunctionaaiiyy multi- multi-graaullrity text embeddings through olf-knowledge distillation. In Wei Ku, Andre Martins, and Vivek Srikumar, editors, Findings of the Association for Computational Linguisticx: ACL 2024, pages 23335,33,, Bangkok, Thailand, August 2024, Association for Computational Linguistics.
[8]
Aakanksha Chowdhery, Sharan Narang, Jacob Devlin, Maarten Bosma, Gaurav Mishra, Adam Roberts, Paul Barham, Hyung Won Chung, Charles Sutton, Sebastian Gehrmann, et al. Palm: Scaling language modeling with pathways. Journal of Machine Learning Research, 24(240):1-113, 2023.
[9] Chroma.
The ai-native open-source embedding database.
https:// /ttp:///gttbbbbb com/chroue core/
chroma
[10] Hyung Won Chung, Thibault Fevry, Henry Tsai, Melvin Johnson, and Sebastian Ruder. Rethinking embed- ding coupling in pre-trained language models. In International Conference on Learning Representations, 2021.
[11] Matthijs Douze, Alexandr Guzhva, Chengqi Deng, Jeff Johnson, Gergely Szilvasy, Pierre-Emmanuel Mazaré, Maria Lomeli, Lucas Hosseini, and Hervé Jégou. The faiss library. IEEE Transactions on Big
Data, 2025.
[12] Haodong Duan, Jueqi Wei, Chonghua Wang, Hongwei Liu, Yixiao Fang, Songyang Zhang, Dahua Lin, and Kai Chen. BotChat: Evaluating LLMs' capabilities of having multi-turn dialogues. In Kevin Duh, Helena Gomez, and Steven Bethard, editors, Findings of the Association for Computational Linguistics: NAACL
2024, pages 3184-3200, Mexico City, Mexico, June 2024. Association for Computational Linguistics.
[13] William Fedus. Barret Zoph, and Noan Shazeer. Switch transformers: Scaling to trillion parameter models with simple and efficient sparsity. Journal of Machine Learning Research, 23(120):1-39 2022.
[14] Bin Gao, Zhuomin He, Puru Sharma, Qingxuan Kang, Djordje Jevdjic, Junbo Deng, Xingkun Yang, Zhou Yu, and Pengfei Zuo. Cost-efficient large language model serving for multi-turn conversations with cachedattention. In Proceedings of the 2024 USENIX Conference Usenix Annual Technical Conference, USENIX 24, USA, 2025. USENIX Association.
[15] In Gim. Guojun Chen. Seung-seob Lee, Nikhil Sarda, Anurag Khandelwal. and Lin Zhong. Prompt cache: Modular attention reuse for low-latency inference. In P. Gibbons, G. Pekhimenko, and C. De Sa, editors, Proceedings of Machine Learning and Systems, volume 6, pages 325-338, 2024.
[16] Aaron Grattafiori, Abhimanyu Dubey, Abhinav Jauhri, Abhinav Pandey, Abhishek Kadian, Ahmad Al- Dahle, Aiesha Leiman, Akhil Mathur, Alan Schelten, Alex Vaughan, et al. The llaaa 3 herd of models. arXiv preprint arXiv:2407.2 22733, 2024.
[17] Intel. 3rd gen intel@ xcon@) scalable processors, 2021. https: h/tp::://wwww... http:://wwwwww.n...eeeeem http://cc/c/ooottt/////eoeo us/ hn/prodducts/docoo http://occssssxxexeee/
gen xeon- scalable- wwwpppporss brief html
[18] Redwan Ibne Seraj Khan, Kunal Jain, Haiying Shen, Ankur Mallick, Anjaly Parayil. Anoop Kulkarni. Steve Kofsky, Pankhuri Choudhary, Renee St Amant, Rujia Wang, et al. Ensuring f'arr Ilm serving amid diverse applications. arXiv preprint arXiv:2411. 15997, 2024.
[19] Woosuk Kwon, Zhuohan Li. Siyuan Zhuang, Ying Sheng. Lianmin Zheng, Cody Hao Yu, Joseph Gonzalez, Hao Zhang, and lon Stoica. Efficient memory management for large language model serving with pagedattention. In Proceedings of the 29th Symposium on Operating Systems Principles. SOSP '23, page 111-626, New York, NY, USA, 2023. Association for Computing Machinery.
11

---

[20] Philippe Laban, Hiroaki Hayashi. Yingbo Zhou, and Jennifer Neville. Llms get lost in multi-turn conversa- tion. arXiv preprint arXiv:2505.061120 2025.
[21] Jiaxing Li, Chi Xu, Feng Wang, Isaac M von Riedemann, Cong Zhang, and Jiangchuan Liu. Scalm: Towards semantic eaching for automated chat services with large language models. In 2024 IEEE/ACM 32nd International Symposium on Quality of Service (IWQoS), pages 1-10. IEEE, 2024.
[22] Pierre Lison and Andrey Kutuzov. Redefining context windows for word embedding models: An ex- perimental study. In Jörg Tiedemann and Nina Tahmasebi, editors, Proceedings of the 2 Nordic Conference on Computational Linguistics, pages 284-288, Gothenburg, Sweden, May 2017. Association for Computational Linguistics.
(23) Kun Luo, Zheng Liu, Shitao Xiao, Tong Zhou, Yuho Chen, Jun Zhao, and Kang Liu. Landmark embedding: A chunking-free freng embedding method for retrieval augmented long-context large language models. In Lun-Wei Ku, Andre Martins, and Vivek Srikumar, editors. Proceedings of the 62nd Annual Meeting of the Association for Computational Linguisties (Volume 1: Long Papers), pages 3268-3281, Bangkok. Thailand, August 2024. Association for Computational Linguistics.
[24] A. Malkov and D. A. Yashunin. Efficient and robust approximate nearest neighbor search using hierarchical navigable small world graphs. IEEE Trans. Pattern Anal. Mach. Intell., 42(4))224-836 April 2020.
[25] Microsoft. Microsoft.
com http://www..aaaaaatt.o//ooaaaaaa ensuring waiiiinng-----i-ee 11m- serving- amid andddiierrr----ppeetttt.
[26] NVIDIA. Geforce rtx 4090.
40 Do-ssreesrrx--000
[27] OpenAL. Prompt caching in the api.
[28] Ofir Press. Noah Smith, and Mike Lewis. Train short, test long: Attention with linear biases enables input length extrapolation. In International Conference Learning Representations, 2022.
[29] PyTorch. Pytorch documentation. https://pytorcch.or
[30] Alee Radford, Jeffrey Wu, Rewon Child, David Luan, Dario Amodei, Ilya Sutskever, et al. Language models are unsupervised multitask learners. OpenAl blog, 1(8)9, 2019.
[31] Pranav Rajpurkar, Jian Zhang. Konstantin Lopyrev, and Percy Liang. SQuAD: 100,000+ questions for machine comprehension of text. In Jian Su. Kevin Duh, and Xavier Carreras, editors, Proceedings of the 2016 Conference Empirical Methods in Natural Language Processing, pages 2383 2392, Austin, Texas, November 2016. Association for Computational Linguistics.
[32] David Rau, Shuai Wang, Hervé Déjean, Stéphane Clinchant, and Jaap Kamps. Context embeddings for efficient answer generation in retrieval- augmented generation. Proceedings of the Eighteenth ACM International Conference on Web Search and Data Mining, WSDM '25, page 493-502, New York, NY,
USA, 2025. Association for Computing Machinery.
[33] Siva Reddy, Danqi Chen, and Christopher D. Manning. CoQA: conversational question answering challenge. Transactions of the Association for Computational Linguistics, 7:249-266 266, 2019.
[34] Luis Gaspar Schroeder, Shu Liu, Alejandro Cuadron, Mark Zhao, Stephan Krusche, Alfons Kemper, Matei Zaharia, and Joseph E Gonzalez. Adaptive semantic prompt caching with vectorq. CoRR, 2025.
[35] Jianlin Su, Murtadha Ahmed, Yu Lu, Shengfeng Pan, Wen Bo, and Yunfeng Liu. Roformer: Enhanced transformer with rotary position embedding. Neurocomputing, 568: 5:5::61:0000 127063, 2024.
[36] Romal Thoppilan, Daniel De Freitas, Jamie Hall, Noam Shazeer, Apporv Kulshreshtha, Heng-Tze Cheng, Alicia Jin, Taylor Bos, Leslie Baker, Yu Du, et al. Lamda: Language models for dialog applications. arXiv preprint 2211:22001.022..
2022.
[37] Hugo Touvron, Thibaut I Lavril, Gautier Izacard, Xavier Martinet, Maric-Anne Lachaux, Timothée Lacroix, Baptiste Rozière, Naman Goyal, Erie Hambro, Faisal Azhar, et al. Llama: Open and efficient foundation language models. arXiv preprint arXiv:2302.1397 2023.
[38] P. D. Turney and Pantel. From frequency meaning: Vector space models of semanties. Journal of Artificial Intelligence Research, 37:141-188, February 2010.
[39] Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan Gomez, Lukasz Kaiser, and Illia Polosukhin. Attention is all you need. In Proceedings of the 31 International Conference on Neural Information Processing Systems, NIPS' 17, page 6000 6010, Red Hook, NY, USA, 2017. Curran Associates Inc.
[40] vLLM Automatic prefix caching. https://docs.vllm.. wttp::/wwwwmaaaaaaaaee/eeeeeeeee http:://oossshhahaaa////eereeres
Feeeix__achiinn.o html
12

---

[41] Jianguo Wang, Xiaomeng Yi, Rentong Guo, Hai Jin, Peng Xu, Shengjun Li. Xiangyu Wang. Xiangzhou Guo. Chengming Li, Xiaohai Xu, Kun Yu, Yuxing Yuan, Yinghao Zou, Jiquan Long, Yudong Cai, Zhenxiang Li, Zhifeng Zhang, Yihua Mo, Jun Gu, Ruiyi Jiang, Wei, and Charles Xie. Milvus: purpose-built vector data management system. In Proceedings of the 2021 International Conference on Management of Data, SIGMOD '21, page Computing Machinery.
New York, NY, USA, 2021, Association for
[42] Guangxuan Xiao, Yuandong Tian, Beidi Chen, Song Han, and Mike Lewis. Efficient streaming language models with attention sinks. In The Twelfth International Conference on Learning Representations, 2024.
[43] An Yang. Baosong Yang. Beichen Zhang, Binyuan Hui, Bo Zheng, Bowen Yu, Chengyuan Li, Dayiheng Liu, Fei Huang, Haoran Wei, Huan Lin, Yang. Jianhong Tu, Jianwei Zhang, Jianxin Yang. Jiaxi Yang. Jingren Zhou, Junyang Lin, Kai Dang, Keming Lu, Keqin Bao, Kexin Yang. Le Yu, Mei Li, Mingfeng Xue, Pei Zhang. Qin Zhu, Rui Men, Runji Lin, Tianhao Li, Tingyu Xia, Xingzhang Ren, Xuancheng Ren, Yang Fan, Yang Su, Yichang Zhang. Yu Wan, Yugiong Liu, Zeyu Cui, Zhenru Zhang, and Zihan Qiu. Qwen2.5 2024.
technical report. arXiv preprint Xrxii:2412.11
[44] Jiayi Yao, Hanchen Li, Yuhan Liu. Siddhant Ray. Yihua Cheng, Qizheng Zhang, Kuntai Du, Shan Lu, and Junchen Jiang. Cacheblend: Fast large language model serving for rag with cached knowledge fusion. In Proceedings of the Twentieth European Conference on Computer Systems, EuroSys '25, page 94-109, New York, NY, USA, 2025. Association for Computing Machinery.
[45] Biao Zhang. Barry Haddow, and Alexandra Birch. Prompting large language model for machine translation: case study. In Proceedings of the 40th International Conference Machine Learning, ICML'23.
JoLR.org. 2023.
[46] Chen Zhang, Xinyi Dai, Yaxiong Wu, Qu Yang. Yasheng Wang, Ruiming Tang, and Yong Liu. A survey multi-turn interaction capabilities of large language models. arXiv preprint orXiv:2501.099. 2025.
[47] Tianyi Zhang. Faisal Ladhak, Esin Durmus, Percy Liang, Kathleen McKeown, and Tatsunori B. Hashimoto. Benchmarking large language models for news summarization. Transactions of the Association for Computational Linguistics, 12:39-52 57, 2024.
[48] Lianmin Zheng, Liangsheng Yin. Zhiqiang Xie, Chuyue Sun, Jeff Huang, Cody Hao Yu, Shiyi Cao. Christos Kozyrakis, Ion Stoica, Joseph E. Gonzalez, Clark Barrett, and Ying Sheng. Sglang: efficient execution of structured language model programs. In Proceedings of the 38th International Conference on Neural Information Processing Systems, NIPS '24, Red Hook, NY, USA, 2025. Curran Associates Inc.
13

---

A Appendix
A.1 Evaluation on Larger Model
To further evaluate SmartCache's scalability and effectiveness, we conduct additional experiments using the Swen2.5-32B-Instree model high-end hardware configuration: AMD) EPYC 7513, 512GB DRAM, and one NVIDIA A100 80GB GPU. 4 the results on the CoQA dataset.
Table 4: Results Qwen2.5-32B Instruct
Metrics / Method
TTFT (ms)
KV Cache (GB)
EM
F1
ROUGE-I
Baseline
213.9
189.2
33.3
54.8
55.0
Prefi
182.5
171.8
33.3
54.8
55.0
GPTCache
94.1
98.3
18.8
34.3
34.7
SmartCache
69.4
71.4
26.8
50.2
50.4
A.2 Computational Overhead under Concurrency
We evaluate SmartCache's computational overhead using Mistral-7B-Instruut--hh. on CoQA under varying numbers of concurrent requests. As shown in Table [ the primary cost under increased concurrency arises from embedding generation, which supports efficient batching. 8 CPU threads are used for faiss-cpu internal threading and semantic forest maintenance. Minor contention is observed in the semantic forest maintenance, especially when multiple queries attempt to insert child nodes under the same parent node. This is mitigated by batching vector insertions.
Table 5: Overhead under concurrent reguests (ms).
# Concurrent Requests
2
4
16
32
64
Embedding Generation
17.13
18.00
19.44
19.54
20.81
22.08
23.32
Vector Search
0.11
0.23
0.26
0.36
0.75
5.64
9.35
Vector Insertion
1.01
1.01
1.02
1.05
1.06
1.22
1.62
Semantic Forest Maintenance
0.29
0.31
0.31
0.33
0.67
1.34
2.97
A.3
Hyperparameter Sensitivity
We evaluate similarity threshold Taim used to determine semantic matches in the Semantic Forest, and weighting coefficients (ax, B3 that balance the Semantic Depth Score (SDS) and Session Affinity Score (SAS) in the STEP eviction policy.
A.3.1 Similarity Threshold
6 the answer quality and performance on Qwen2.5-1.5B 1.5t--rrtt- with CoQA. Increasing
Tsim monotonically improves EM/F 1/OOUGE-OULLLLL by enforcing stricter semantic matches, but reduces reuse opportunities, increasing both TTFT and KV cache usage. We adopt Tsim 0.75 for balanced operating point used in main experiments.
Table 6: Answer quality and performance across different similarity threshold.
Tsim
EM
F1
ROUGE-L
TTFT (ms)
KV Cache (GB)
0.50
17.4
35.2
35.5
26.8
4.1
0.60
21.6
41.1
41.4
27.5
4.5
0.70
23.5
45.3
45.5
30.2
6.5
0.75
24.9
46.6
47.0
31.3
7.7
0.80
25.4
47.7
47.9
49.0
8.5
0.90
25.9
48.9
49.1
83.2
12.1

---

A.3.2 Coefficients a and 3 of STEP Policy
Table 7 shows the grid search result with the same configuration as Section 4.4
We adopt a 0.6
and | 0.4 for STEP in all evaluations, as this setting attains the highest average reuse distance.
Table 7: Average reuse distance over different (ax 5) settings (zipfian distribution).
(0,
(0.1,0.9)
(0.2,0.8)
(0.3,0.7)
(0.4.0.6)
(0.5.0.5)
(0.6.0.4)
(0.7.0.3)
(0.8.0.2)
(0.9.0.1)
Reuse distance (avg)
231.8
252.0
255.9
262.3
261.6
265.4
260.5
240.9
252.5
A.4
Attention-basel Context Switching
Since different models exhibit numeric variations on attention scores, the attention score threshold is profiled for each model offline using five predefined independent queries, averaged across attention heads and query tokens for each query, excluding 8 initial tokens that behave as attention sinks (i.e., attracting disproportionate attention regardless of context 42... The same attention score calculation procedure is triggered during the prefill stage of a query when it fails from reusing its child semantic nodes. It is then compared with the profiled threshold to determine whether or not the query is independent.
Impact of Architectural Differences. The widely used Grouped-Query Attention (GQA changes per-head magnitudes but preserves the probabilistic semantics of attention. Mixture-of- Experts (MoE (T3) alters the feed-forward pathway via expert routing while leaving the attention softmax itself unchanged. Positional encodings such rotary position embeddings (RoPE (35) and attention with linear biases (ALiBi (28) modify how relative position influences attention logits, but after softmax the attention weights still form a probability simplex. Therefore, architectural modifications that change logits but preserve the probabilistic semantics do not require changing the decision logic.
A.5 Privacy Preserving
SmartCache designed support multi-user environments, where multiple sessions may reuse cached responses across conversations. While semantic reuse offers significant performance and memory advantages, also introduces potential privacy concerns. These issues can be mitigated via query sanitization, access control, and asynchronous privacy filtering.
Query Sanitization. To prevent private sensitive information from being inadvertently shared across sessions, SmartCache integrates a privacy data sanitizer module. This module inspects user queries and determines whether query-response pair should be treated private. If so, the corresponding semantic node is marked as hidden, meaning it is only visible to the originating user or authorized parties, such members of the same organization or tenant.
Access Control. Hidden semantic nodes are excluded from the global semantic forest used for cross-session lookup. These nodes are stored and reused exclusively within the user's session within authorized scopes defined by privacy policies. This ensures that personal or sensitive queries, if semantically similar to others, do not produce cache hits across different users.
Asynchronous Sanitization for Low Latency. To minimize latency, when a new query is issued, immediately processed and added to the cache hidden node by default. In the hackground, the sanitizer evaluates the query and, if is deemed non-sensitive, updates the node's 's visibility status to allow future sharing. This decouples privacy inspection from the critical path of inference, preserving the low TTFT guarantees of SmartCache while still ensuring privacy compliance.
15

---

NeurIPS Paper Checklist
The checklist is designed encourage best practices for responsible machine learning research, addressing issues of reproducibility, transparency, research ethics, and societal impact. Do not remove the checklist: The papers not including the checklist will be desk rejected. The checklist should follow the references and follow the (optional) supplemental material. The checklist does NOT count towards the page limit.
Please read the checklist guidelines carefully for information on how to answer these questions. For each question in the checklist:
You should answer [Yes], [No], or [NA].
[NA] means either that the question Not Applicable for that particular paper or the relevant information is Not Available.
Please provide short (1-2 sentence) justification right after your answer (even for NA).
The checklist answers are an integral part of your paper submission. They are visible to the reviewers, area chairs, senior area chairs, and ethics reviewers. You will be asked to also include (after eventual revisions) with the final version of your paper, and its final version will published with the paper.
The reviewers of your paper will be asked use the checklist as one of the factors their evaluation. While "[Yes] is generally preferable to "[No]" ", it is perfectly acceptable to answer "[No] providedd proper justification is given (e.g.. "error bars are not reported because it would be too computationally expensive" or "we were unable to find the license for the dataset we used"). In general, answering binary way, we
"[No] or "[NA]" is not grounds for rejection. While the questions are phrased acknowledge that the true answer is often more nuanced, please just use your best judgment and write justification to elaborate. All supporting evidence can appear either in the main paper or the supplemental material, provided in appendix. If you answer [Yes] to a question, in the justification please point to the section(s) where related material for the question can be found.
IMPORTANT, please:
Delete this instruction block, but keep the section heading "NeurIPS Paper Checklist",
Keep the checklist subsection headings, questions/answers and guidelines below.
Do not modify the questions and only use the provided macros for your answers.
1. Claims
Question: Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?
Answer: [Yes]
Justification:
Guidelines:
The answer NA means that the abstract and introduction do not include the claims made in the paper.
The abstract and/or introduction should clearly state the claims made, including the contributions made in the paper and important assumptions and limitations. A No or
NA answer to this question will not perceived well by the reviewers.
The claims made should match theoretical and experimental results, and reflect how much the results be expected to generalize other settings.
is fine include aspirational goals as motivation long is clear that these goals are not attained by the paper.
2. Limitations
Question: Does the paper discuss the limitations of the work performed by the authors?
Answer: [Yes]
Justification:
16

---

Guidelines:
The answer NA means that the paper has no limitation while the answer No means that the paper has limitations, but those are not discussed in the paper.
The authors are encouraged to create a separate "Limitations" section in their paper.
The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (e.g., independence assumptions, noiseless settings, model well-specification, asymptotic approximations only holding locally). The authors should reflect on how these assumptions might be violated in practice and what the implications would be.
The authors should reflect on the scope of the claims made, c.g., if the approach was only tested few datasets or with few runs. In general, empirical results often depend on implicit assumptions, which should be articulated.
The authors should reflect on the factors that influence the performance of the approach. For example, facial recognition algorithm may perform poorly when image resolution is low or images are taken in low lighting. Or speech-to-text system might not be used reliably to provide closed captions for online lectures because it fails handle technical jargon.
The authors should discuss the computational efficiency the proposed algorithms and how they scale with dataset size.
If applicable, the authors should discuss possible limitations of their approach to address problems of privacy and fairness.
While the authors might fear that complete honesty about limitations might be used by reviewers as grounds for rejection, worse outcome might be that reviewers discover limitations that aren't acknowledged in the paper. The authors should use their best judgment and recognize that individual actions in favor of transparency play an impor- tant role in developing norms that preserve the integrity of the community. Reviewers will be specifically instructed to not penalize honesty concerning limitations.
3. Theory assumptions and proofs
Question: For each theoretical result, does the paper provide the full set of assumptions and complete (and correct) proof?
Answer: [NA]
Justification:
Guidelines:
The answer NA means that the paper does not include theoretical results.
All the theorems, formulas, and proofs in the paper should be numbered and cross- referenced.
All assumptions should be clearly stated or referenced in the statement of any theorems.
The proofs can either appear in the main paper the supplemental material, but if they appear in the supplemental material, the authors are encouraged to provide short proof sketch provide intuition.
Inversely, any informal proof provided in the core of the paper should complemented by formal proofs provided in appendix supplemental material.
Theorems and Lemmas that the proof relies upon should properly referenced.
4. Experimental result reproducibility
Question: Does the paper fully disclose all the information needed to reproduce the main ex- perimental results of the paper to the extent that it affects the main claims and/or conclusions the paper (regardless whether the code and data are provided or not)?
Answer: [Yes]
Justification:
Guidelines:
The answer NA means that the paper does not include experiments.
17

---

If the paper includes experiments, a No answer to this question will not be perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data are provided or not.
If the contribution is dataset and/or model, the authors should describe the steps taken to make their results reproducible or verifiable.
Depending on the contribution, reproducibility can be accomplished in various ways. For example, if the contribution is a novel architecture, describing the architecture fully might suffice, or if the contribution is specific model and empirical evaluation, it may be necessary to either make it possible for others to replicate the model with the same dataset, or provide access the model. In general. releasing code and data is often one good way to accomplish this, but reproducibility can also be provided via detailed instructions for how to replicate the results, access a hosted model (e.g., in the case of a large language model), releasing of a model checkpoint, or other means that are appropriate to the research performed.
While NeurlPS does not require releasing code, the conference does require all submis- sions to provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution. For example
(a) If the contribution is primarily new algorithm, the paper should make it clear how to reproduce that algorithm. (b) the contribution is primarily a new model architecture, the paper should describe the architecture clearly and fully.
(c) If the contribution is new model (e.g., a large language model), then there should either be way to access this model for reproducing the results or way to reproduce the model (e.g., with an open-source dataset instructions for how to construct the dataset).
(d) We recognize that reproducibility may be tricky in some cases, in which case authors are welcome describe the particular way they provide for reproducibility. the case of closed-source models, it may be that access to the model is limited in some way (e.g., registered users), but it should be possible for other researchers to have some path to reproducing or verifying the results.
5. Open access to data and code
Question: Does the paper provide open access to the data and code, with sufficient instruc- tions to faithfully reproduce the main experimental results, as described in supplemental material?
Answer: [NA]
Justification:
Guidelines:
The answer NA means that paper does not include experiments requiring code.
Please see the NeurlPS code and data submission guidelines (nttps://nips.co
for more details.
While we encourage the release of code and data, we understand that this might not be possible, so "No" an acceptable answer. Papers cannot be rejected simply for not including code, unless this is central to the contribution (e.g., for new open-source benchmark).
The instructions should contain the exact command and environment needed to run to reproduce the results. See the NeurIPS code and data submission guidelines (https
7 /ttppp.cc/pbllc/guudeeee http://oeeuubiissnooooooooooo
for more details.
The authors should provide instructions on data access and preparation, including how to access the raw data, preprocessed data, intermediate data, and generated data, etc.
The authors should provide scripts to reproduce all experimental results for the new proposed method and baselines. only subset of experiments are reproducible, they should state which ones are omitted from the script and why.
At submission time, to preserve anonymity, the authors should release anonymized versions (if applicable).
18

---

Providing as much information as possible in supplemental material (appended to the paper) recommended, but including URLs to data and code permitted.
6. Experimental setting//etails
Question: Does the paper specify all the training and test details (e.g., data splits, hyper- parameters, how they were chosen, type of optimizer, etc.) necessary to understand the results?
Answer:
[Yes]
Justification:
Guidelines:
The answer NA means that the paper does include experiments.
The experimental setting should be presented the core the paper to level of detail that is necessary to appreciate the results and make sense of them.
The full details can be provided either with the code, in appendix, as supplemental material.
7. Experiment statistical significance
Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?
Question:
Answer: [No]
Justification: The reported evaluation results this paper is averaged through multiple runs of experiments.
Guidelines:
The answer NA means that the paper does not include experiments.
The authors should answer "Yes" if the results are accompanied by error bars, confi- dence intervals, statistical significance tests, least for the experiments that support the main claims of the paper.
The factors of variability that the error bars are capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, or overall run with given experimental conditions).
The method for calculating the error bars should be explained (closed form formula, call to a library function, bootstrap, etc.)
The assumptions made should be given (e.g.. Normally distributed errors).
It should be clear whether the error bar is the standard deviation or the standard error of the mean.
is OK to report I-sigma error bars, but one should state it. The authors should preferably report a 2-simma error bar than state that they have 96% CI, if the hypothesis of Normality of errors is not verified.
For asymmetric distributions, the authors should be careful not to show in tables or figures symmetric error bars that would yield results that are out range (e.g. negative error rates).
If error bars are reported in tables or plots, The authors should explain in the text how they were calculated and reference the corresponding figures or tables the text.
8. Experiments compute resources
For each experiment, does the paper provide sufficient information on the com- puter resources (type of compute workers, memory, time exccution) needed to reproduce the experiments?
Question:
Answer: [Yes]
Justification:
Guidelines:
The answer NA means that the paper does include experiments.
The paper should indicate the type of compute workers CPU or GPU, internal cluster, or cloud provider, including relevant memory and storage.
19

---

The paper should provide the amount of compute required for each of the individual experimental runs as well estimate the total compute.
The paper should disclose whether the full research project required more compute than the experiments reported in the paper (e.g., preliminary or failed experiments that didn't make into the paper).
9. Code of ethics
Question: Does the research conducted in the paper conform, in every respect, with the NeurIPS Code of Ethics httpes//beurrpp.cccoubbbi/ibiiie
Answer: [Yes]
Justification:
Guidelines:
The answer NA means that the authors have not reviewed the NeurIPS Code of Ethics.
If the authors answer No, they should explain the special circumstances that require a deviation from the Code of Ethics.
The authors should make sure to preserve anonymity (e.g.. if there special consid- eration due to laws or regulations in their jurisdiction).
10. Broader impacts
Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?
Answer: [NA]
Justification:
Guidelines:
The answer NA means that there no societal impact of the work performed.
If the authors answer NA or No, they should explain why their work has no societal impact why the paper does not address societal impact.
Examples of negative societal impacts include potential malicious unintended uses (e.g., disinformation, generating fake profiles, surveillance), fairness considerations (e.g., deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations.
The conference expects that many papers will be foundational research and not tied to particular applications, let alone deployments. However, if there is a direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that an improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point that a generic algorithm for optimizing neural networks could enable people to train models that generate Deepfakes faster.
The authors should consider possible harms that could arise when the technology is being used as intended and functioning correctly, harms that could arise when the technology being used as intended but gives incorrect results, and harms following from (intentional or unintentional) misuse of the technology.
If there are negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release models, providing defenses in addition attacks, mechanisms for monitoring misuse, mechanisms to monitor how system learns from feedback over time, improving the efficiency and accessibility ML).
11. Safeguards
Question:
Does the paper describe safeguards that have been put in place for responsible release of data or models that have a high risk for misuse (e.g., pretrained language models, image generators, scraped datasets)?
Answer: [NA]
Justification:
Guidelines:
The answer NA means that the paper poses such risks.
20

---

Released models that have high risk for misuse or dual use should be released with necessary safeguards to allow for controlled use of the model. for example by requiring that users adhere to usage guidelines or restrictions to access the model or implementing safety filters.
Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images.
We recognize that providing effective safeguards challenging, and many papers do not require this, but we encourage authors to take this into account and make best faith effort.
12. Licenses for existing assets
Are the creators original owners of assets (e.g., code, data, models), used in the paper, properly credited and are the license and terms of use explicitly mentioned and properly respected?
Question:
Answer: [Yes]
Justification:
Guidelines:
The answer NA means that the paper does not use existing assets.
The authors should cite the original paper that produced the code package dataset.
The authors should state which version of the asset used and, if possible, include URL.
The name of the license (e.g., CC-BY 4.0) should be included for each asset.
For scraped data from a particular source (e.g., website), the copyright and terms of service of that source should be provided.
If assets are released, the license, copyright information, and terms of use in the package should be provided. For popular datasets, paperswi thcode. com datasets has curated licenses for some datasets. Their licensing guide can help determine the license of dataset.
For existing datasets that are re- re-packaged, both the original license and the license of the derived asset (if it has changed) should be provided.
If this information is not available online, the authors are encouraged to reach out to the asset's 's creators.
13. New assets
Question: Are new assets introduced in the paper well documented and is the documentation provided alongside the assets?
Answer: [Yes]
Justification:
Guidelines:
The answer NA means that the paper does not release new assets.
Researchers should communicate the details of the dataset/code/model as part of their submissions via structured templates. This includes details about training, license, limitations, etc.
The paper should discuss whether and how consent obtained from people whose asset is used.
At submission time, remember to anonymize your assets (if applicable). You can either create an anonymized URL or include an anonymized zip file.
14. Crowdsourcing and research with human subjects
Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given to participants and screenshots, if applicable, as well details about compensation (if any)?
Answer: [NA]
Justification:
21

---

Guidelines:
The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.
Including this information the supplemental material is fine, but if the main contribu- tion of the paper involves human subjects, then as much detail as possible should be included the main paper.
According to the NeurIPS Code Ethics, workers involved in data collection, curation, other labor should be paid at least the minimum wage in the country of the data collector.
15. Institutional review board (IRB) approvals or equivalent for research with human subjects
Question: Does the paper describe potential risks incurred by study participants, whether such risks were disclosed the subjects, and whether Institutional Review Board (IRB) approvals (or an equivalent approval/review based on the requirements of your country or institution) were obtained?
Answer: [NA]
Justification:
Guidelines:
The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.
Depending on the country in which research is conducted, IRB approval (or equivalent)
may be required for any human subjects research. If you obtained IRB approval, you should clearly state this in the paper.
We recognize that the procedures for this may vary significantly between institutions and locations, and expect authors to adhere to the NeurIPS Code of Ethics and the guidelines for their institution.
For initial submissions, do not include any information that would break anonymity (if applicable), such as the institution conducting the review.
16. Declaration of LLM usage
Question: Does the paper describe the usage of LLMs if it is an important, original, or non-standard component of the core methods this research? Note that if the L.M used only for writing, editing, or formatting purposes and does not impact the core methodology. scientific rigorousness, or originality the research, declaration is not required.
Answer: [NA]
Justification:
Guidelines:
The answer NA means that the core method development in this research does not involve LLMs any important, original, non-standard components.
Please refer to our LLM policy htte::////orrrpsccccooooeerree/r for what should or should not be described.
22


Reviews:
[{"review_id": "ljCgp8jZpH", "reviewer": "Reviewer_h8B6", "summary": "This paper introduces SmartCache, a system-algorithm co-design framework aimed at improving multi-turn LLM inference efficiency by addressing redundant computation and memory usage caused by semantically similar queries across sessions. Unlike methods that require exact matches or ignore conversational context, SmartCache integrates context-aware semantic caching with the inference engine. Its key contributions include a hierarchical semantic forest for accurate reuse of responses, an attention-based module for detecting topic shifts, and an integrated KV cache management system with cross-session sharing and a novel STEP eviction policy. Experiments show that SmartCache significantly reduces KV cache usage and TTFT while improving answer quality.", "questions": "1. Overhead Analysis: Can the authors provide quantitative measurements on the computational and memory overhead introduced by embedding generation, vector search, and semantic forest operations?\n2. Attention-based Context Switching: Could you elaborate on how the system distinguishes between \"progressive\" and \"individual\" queries? Is there a fixed threshold on attention scores, and how robust is this heuristic across different models?\n3. Performance under High Concurrency: Have you evaluated SmartCache under high concurrency or batch inference settings? Since cross-session reuse is central to the system, understanding its behavior under realistic serving loads would be valuable.", "limitations": "Yes.", "rating": 5, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 4, "clarity": 3, "significance": 3, "originality": 3, "strengths_and_weaknesses": "### Strengths\n1. High Significance: The paper tackles a crucial problem in LLM serving—inefficiency in multi-turn conversations caused by semantic redundancy.\n2. Novel and Holistic Design: The integration of semantic caching and KV cache management reflects a thoughtful system-algorithm co-design, offering a comprehensive solution that surpasses prior approaches in both scope and capability.\n3. Strong Empirical Results: SmartCache outperforms baseline, prefix, and flat semantic caching methods in terms of memory efficiency, TTFT, and answer quality across multiple models and datasets.\n\n### Weaknesses\n1. Lack of Overhead Quantification: While the authors claim minimal overhead, the paper does not provide a detailed breakdown of the costs associated with embedding computation, vector search, and forest maintenance relative to performance gains.\n2. Limited Evaluation Under High Concurrency: The main benefit of cross-session KV cache sharing appears in high-throughput environments, yet the paper lacks experiments simulating batch inference or parallel request scenarios.\n3. Hyperparameter Sensitivity Not Addressed: The system likely depends on hyperparameters like τ_sim, α, and β, but the paper does not include an ablation study or sensitivity analysis to show robustness.", "was_revised": true, "final_justification": "SmartCache addresses a significant inefficiency in multi-turn LLM serving with a novel system-algorithm co-design that integrates semantic caching and KV cache management. My initial concerns focused on overhead quantification, scalability under high concurrency, and hyperparameter sensitivity. The authors’ rebuttal provided detailed breakdowns and new experiments demonstrating SmartCache’s additional overhead is minimal and its performance scales well with concurrency. The ablation and sensitivity analyses further strengthened the empirical evaluation. Given that the major concerns have been satisfactorily addressed, I believe the paper makes a solid technical contribution and have raised my rating from 4 to 5 and confidence from 3 to 4.", "rebuttal": "Thank you for acknowledging **the significance of the crucial problem** we address, as well as the **novel, holistic and thoughtful system-algorithm co-design**. Below, we provide point-by-point responses to your constructive and helpful questions.\n\n**Q1: Overhead Analysis of SmartCache** \n\n**Response:** Thank you for the comment. As shown in Table 1, the computational and memory overhead with Mistral-7B-Instruct-v0.2 on CoQA. The computational overhead for processing a cache-missed new query (i.e. setting up a new semantic node + performing LLM inference) is approximately 8.8% (18.54ms out of 210.37ms). Due to SmartCache's hierarchical semantic information and co-design with KV cache management, there is a higher likelyhood of reusing cached responses resulting in more substantial performance gains. SmartCache introduces about 7.1% GPU memory overhead (embedding model compared to LLM weights and compute buffer) and little CPU memory overhead (28.05MB).\n\nTable 3: Overhead Breakdown\n\n| Breakdown                    | Computational Overhead | Memory Overhead           |\n|------------------------------|------------------------|---------------------------|\n| Embedding Generation         | 17.13ms                | 1.06GB GPU memory         |\n| Vector Search                | 0.11ms                 | 18.16MB CPU memory        |\n| Vector Insertion             | 1.01ms                 | Shared with vector search |\n| Semantic Forest Maintenance | 0.29ms                 | 9.89MB CPU memory         |\n\n\n**Q2: Performance under High Concurrency** \n\n**Response:** Thank you for the insightful comment. Table 2 shows the concurrency overhead of SmartCache in the scenario with varying number of concurrent queries, based on Mistral-7B-Instruct-v0.2 and CoQA.\n\nTable 2: Overhead analysis under different number of concurrent requests\n\n| # Concurrent Request        | 1     | 2     | 4     | 8     | 16    | 32    | 64    |\n|-----------------------------|-------|-------|:------|:------|:------|:------|:------|\n| Embedding Generation        | 17.13 | 18.00 | 19.44 | 19.54 | 20.81 | 22.08 | 23.32 |\n| Vector Search               | 0.11  | 0.23  | 0.26  | 0.36  | 0.75  | 5.64  | 9.35  |\n| Vector Insersion            | 1.01  | 1.01  | 1.02  | 1.05  | 1.06  | 1.22  | 1.62  |\n| Semantic Forest Maintenance | 0.29  | 0.31  | 0.31  | 0.33  | 0.67  | 1.34  | 2.97  |\n\nThe primary source of overhead, embedding generation, supports efficient batching. 8 CPU threads are used for faiss-cpu internal threading and semantic forest maintenance. Contention mainly occurs when multiple queries attempt to add new child nodes to the same parent semantic node, which is mitigated by batching vector insertion.\n\n**Q3: Hyperparameter Sensitivity**\n\n**Response:** Thank you for the comment. The τ_sim is set to 0.75 to give a balanced trade-off between performance and quality. The α and β are selected based on the grid search results. Table 3 presents the ablation results of Qwen2.5-1.5B-Instruct on CoQA, while Table 4 shows the grid search result for α and β with the same configuration as Section 4.4.\n\nTable 3: Different similarity threshold.\n\n| Threshold | EM   | F1   | ROUGE-L | TTFT (ms) | KV Cache (GB) |\n|-----------|------|------|:--------|:----------|:--------------|\n| 0.5       | 17.4 | 35.2 | 35.5    | 26.8      | 4.1           |\n| 0.6       | 21.6 | 41.1 | 41.4    | 27.5      | 4.5           |\n| 0.7       | 23.5 | 45.3 | 45.5    | 30.2      | 6.5           |\n| 0.75      | 24.9 | 46.6 | 47.0    | 31.3      | 7.7           |\n| 0.8       | 25.4 | 47.7 | 47.9    | 49.0      | 8.5           |\n| 0.9       | 25.9 | 48.9 | 49.1    | 83.2      | 12.1          |\n\nTable 4: Different α and β, zipf distribution.\n\n| (α, β)  | (0.1, 0.9) | (0.2, 0.8) | (0.3, 0.7) | (0.4, 0.6) | (0.5, 0.5) | (0.6, 0.4) | (0.7, 0.3) | (0.8, 0.2) | (0.9, 0.1) |\n|----------------------|------------|------------|:-----------|:-----------|:-----------|:-----------|:-----------|:-----------|:-----------|\n| Reuse distance (avg) | 231.8      | 252.0      | 255.9      | 262.3      | 261.6      | 265.4      | 260.5      | 240.9      | 252.5      |\n\nBy increasing τ_sim, answer quality can be further improved. However it leads to higher TTFT and increased KV cache usage resulted from reduced opportunities to reuse similar queries.\n\n**Q4: Attention-based Context Switching**\n\n**Response:** Thank you for the comment. Since different models exhibit numeric variations on attention scores, the attention score threshold is profiled for each model offline using five predefined independent queries, averaged across query tokens for each query. The same attention score calculation procedure is triggered during the prefill stage of a qeury when it fails from reusing its child semantic nodes. It is then compared with the the profiled threshold to determine whether or not the query is independent."}, {"review_id": "mjF2kwoc3f", "reviewer": "Reviewer_C7su", "summary": "This paper proposes SmartCache, a novel system-algorithm co-design framework to address the inefficiencies of multi-turn LLM inference. SmartCache introduces a \"Semantic Forest\" structure to hierarchically index conversational turns, enabling efficient retrieval and reuse of responses while preserving conversational context. It further leverages internal LLM attention scores for dynamic context detection and introduces a \"two-level mapping\" for cross-session Key-Value (KV) cache sharing, complemented by a \"Semantic-aware Tiered Eviction Policy (STEP)\". The evaluation demonstrates substantial reductions in KV cache memory usage and Time-to-First-Token (TTFT), alongside improvements in answer quality on various benchmarks.", "questions": "Given that the paper mentions $t_{sim}$ for embedding similarity, could the authors clarify if a similar explicit threshold or a specific methodology for dynamically determining context changes based on LLM attention scores is employed in SmartCache, and how its sensitivity and generalizability across different LLM architectures and datasets were evaluated?", "limitations": "Yes", "rating": 5, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 2, "significance": 2, "originality": 3, "strengths_and_weaknesses": "Strengths:\n\n- Introduces a hierarchical semantic cache (Semantic Forest) that respects conversational context. Bridges the gap between semantic and KV caching by linking semantic nodes directly to KV cache blocks, which is underexplored. Integrates attention-based context detection into the caching pipeline to mitigate misuses due to context-ignorant semantic retrieval.\n- Demonstrates consistent improvements on CoQA and SQuAD benchmarks using three LLMs. Reduces KV cache usage by up to 59.1%, TTFT by up to 78.0%, and improves F1/ROUGE by up to 39%.\n- The two-level KV cache mapping (session → semantic node → block) is a practical innovation that enables transparent sharing without compromising cache coherence or semantic fidelity.\n- The paper is well-organized with clear diagrams and pseudocode, easy to understand.\n\nWeaknesses\n- While the Semantic Forest provides clear benefits, the operational overhead of maintaining and updating this complex hierarchical structure in a highly dynamic, real-time serving environment could be further elaborated, especially concerning potential locking mechanisms or distributed concerns for very large-scale deployments.\n- The threshold for determining context changes based on attention scores might require fine-tuning across different LLM architectures or datasets. A more detailed analysis of its robustness would be beneficial.", "was_revised": true, "final_justification": "The rebuttal provides more experiment results. The idea is novel and the paper is well-organized.", "rebuttal": "We appreciate your support of our work and your detailed detailed summary of the problem we address and novelty of our approach. Your comments are insightful and constructive to us.\n\n**Q1: Operational Overhead In Large-Scale**\n\n**Response:** Thank you for the valuable comment. The scalability of semantic forest under high concurrency is important for user experience in large-scale deployment. Table 1 shows the operational overhead of SmartCache in the scenario with varying numbers of concurrent queries, based on Mistral-7B-Instruct-v0.2 and CoQA.\n\nTable 1: Overhead analysis under different number of concurrent requests\n\n| # Concurrent Request        | 1     | 2     | 4     | 8     | 16    | 32    | 64    |\n|-----------------------------|-------|-------|:------|:------|:------|:------|:------|\n| Embedding Generation        | 17.13 | 18.00 | 19.44 | 19.54 | 20.81 | 22.08 | 23.32 |\n| Vector Search               | 0.11  | 0.23  | 0.26  | 0.36  | 0.75  | 5.64  | 9.35  |\n| Vector Insersion            | 1.01  | 1.01  | 1.02  | 1.05  | 1.06  | 1.22  | 1.62  |\n| Semantic Forest Maintenance | 0.29  | 0.31  | 0.31  | 0.33  | 0.67  | 1.34  | 2.97  |\n\nThe primary source of overhead, embedding generation, supports efficient batching. 8 CPU threads are used for faiss-cpu internal threading and semantic forest maintenance. Contention mainly occurs when multiple queries attempt to add new child nodes to the same parent semantic node, which is mitigated by batching vector insertion.\n\n**Q2: More Analysis of Attention Score Threshold on Different LLM Architectures**\n\n**Response:** Thank you for the comment. We will incorporate a more detailed analysis of different models in the revised version."}, {"review_id": "6r31h2kKYM", "reviewer": "Reviewer_P9MM", "summary": "This paper proposes Smart Cache, a system level co-design framework to improve the efficiencies of multi-turn conversations that reduces the TTFT and kv cache useage. This method first cluster the conversation histories into the topic-specific tree “Semantic Forest,” performs a local-then-global ANN search over saved embeddings, and then dynamically decide whether to start a new tree or stays in the current tree based on the attention during prefill. The paper also proposes STEP policy to decide the eviction/ prefetch based on the two-level mapping table. Experiments on CoQA and SQuAD with Llama-3-8B demonstrate up to 59 % KV-memory reduction and 78 % faster time-to-first-token versus prefix caching.", "questions": "Have you examined how large the semantic tree would be confronting large volume of requests?\n\nWhat happens if the local lookup fails, but a semantically very similar node exists elsewhere in the same tree (i.e., not as a direct child)?\n\nCould you report the efficiency gains under minimal performance losses? People won’t like significant compromise of performance even with significant TTFT gains", "limitations": "I think the main limitation is for the experiment sessions. Please see my comments in the weakness parts.", "rating": 4, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 2, "clarity": 3, "significance": 3, "originality": 3, "strengths_and_weaknesses": "**Strength**\n- The author discusses on an important topic that expand the idea of GPT-cache to the multi-turn conversations and is able to reduce both the kv cache memory and the time-to-first token  \n- Compared to the baselines outlined in the paper, this method achieves better performance than GPT-cache  \n- The author proposes system-algorithm co-design that enables efficient kv-cache reuses  \n\n**Weakness**\n- Limited Scale Evaluation: The evaluation is conducted on relatively small models and datasets. It's unclear how the approach would scale to larger models (e.g., 32B+ parameters) or handle a significant volume of concurrent requests. Some evaluations on larger models, or other large multi-turn conversations, such as Lmsys-chat, would provide a more comprehensive evaluation of this method.  \n- Response quality degradation: As shown in the Table2, even with the smart cache, the performance on the CoQA is still significantly lower than the ideal one.  \n- Privacy concerns: Though the authors have noted the privacy limitations in the paper, no concrete solutions have been proposed to address this concern.  \n- Lack of ablation and breakdown experiments: This paper doesn’t provide enough analysis on how the overhead of the retrieval, possible loading time during PCIE, and ANN search. When the prefix is short and the model is small, these overheads could be significant.  \n- Lack of baselines: More recent paper, such as CacheBlend and PromptCache, also target on the efficient caching for reducing TTFT. These baselines are not included.", "was_revised": true, "final_justification": "My main concern is in the lack of the experiments part and the lack of the breakdown parts. The author has provided additional experiments in the rebuttal and a clear breakdown of the overhead. While I raise the score from 3 to 4, I hope more efforts could be done for discussing the trade-off, since we can still see the performance degradation.", "rebuttal": "Thank you for ackownledging the **important topic** addressed in our work. Your detailed questions are constructive and valuable to us. Below, we provide detailed responses to your comments.\n\n**Q1: Larger Scale**\n\n**Response:** Thank you for the comment. Experimental results with a larger model Qwen2.5-32B-Instruct on AMD EPYC 7513, 512GB DRAM equipped with one NVIDIA A100 80GB are shown as follows.\n\nTable 1: Results of Qwen2.5-32B-Instruct on CoQA\n\n| Metrics/Methods | TTFT (ms) | KV Cache (GB) | EM   | F1   | ROUGE-L |\n|-----------------|-----------|---------------|:-----|:-----|:--------|\n| Baseline        | 213.9     | 189.2         | 33.3 | 54.8 | 55.0    |\n| PrefixCache     | 182.5     | 171.8         | 33.3 | 54.8 | 55.0    |\n| GPTCache        | 94.1      | 98.3          | 18.8 | 34.3 | 34.7    |\n| SmartCache      | 69.4      | 71.4          | 26.8 | 50.2 | 50.4    |\n\nThe results on a large dataset such as Lmsys-chat will be included in the revised version.\n\n**Q2: Efficiency Gains w.r.t Quality Loss** Could you report the efficiency gains under minimal performance losses? People won’t like significant compromise of performance even with significant TTFT gains\n\n**Response:** Thank you for the comment. Table 2 shows the answer quality and performance of SmartCache when using a high similarity threshold of 0.9.\n\nTable 2: Quality v.s. Performance\n\n|                          | EM   | F1   | ROUGE-L | TTFT (ms) |\n|--------------------------|------|------|:--------|:----------|\n| Qwen2.5-1.5B-Instruct    | 25.9 | 48.9 | 49.1    | 83.2      |\n| Llama3.1-8B-Instruct     | 30.3 | 54.7 | 54.5    | 101.7     |\n| Mistral-7B-Instruct-v0.2 | 4.6  | 23.8 | 23.7    | 118.2     |\n\nThe quality is compromised by little with a high similarity threshold.\n\n**Q3:** What happens if the local lookup fails, but a semantically very similar node exists elsewhere in the same tree (i.e., not as a direct child)?\n\n**Response:** Thank you for the in-depth comments. In such cases, SmartCache treats nodes in different branches as semantically different. Since they originate from different parent contexts, forcing a cache hit across branches could result in false positives and potentially incorrect answers. We have considered ways to accurately handle such cases, recognizing that a more complex data structure and algorithm would be required. After evaluating the trade-off between complexity and speed, we chose to allow a locally cache-missed query to either start a new tree or create a new child node under the current node.\n\n**Q4: Privacy Concerns**\n\n**Response:** Thank you for the insightful comment. The solution to the privacy issue will be clarified in the revised version. Specifically, SmartCache can incorporate a user privacy data sanitizer module to check whether the queries contain private information. Private query-response semantic nodes are hidden from other users. Only the owner or users within the same organization can access the hidden nodes, based on the privacy configurations. Various existing privacy sanitization methods can be employed as the sanitizer. To minimize potential overhead and latency cause by the privacy check, a new query can first be processed by SmartCache and inserted as an hidden semantic node. It will then be asynchronously examined by the background sanitizer to determine whether to release its hidden status.\n\n**Q5: Ablation and Breakdown Evaluations**\n\n**Response:** Thank you for the comment. As shown in Table 1, the computational and memory overhead with Mistral-7B-Instruct-v0.2 on CoQA. The computational overhead for processing a cache-missed new query (i.e. setting up a new semantic node + performing LLM inference) is approximately 8.8% (18.54ms out of 210.37ms). Due to SmartCache's hierarchical semantic information and co-design with KV cache management, there is a higher likelyhood of reusing cached responses resulting in more substantial performance gains. SmartCache introduces about 7.1% GPU memory overhead (embedding model compared to LLM weights and compute buffer) and little CPU memory overhead (28.05MB).\n\nTable 3: Overhead Breakdown\n\n| Breakdown                    | Computational Overhead | Memory Overhead           |\n|------------------------------|------------------------|---------------------------|\n| Embedding Generation         | 17.13ms                | 1.06GB GPU memory         |\n| Vector Search                | 0.11ms                 | 18.16MB CPU memory        |\n| Vector Insertion             | 1.01ms                 | Shared with vector search |\n| Semantic Forest Maintenance | 0.29ms                 | 9.89MB CPU memory         |\n\n\n**Q6: Comparison with More Baselines**\n\n**Response:** Thank you for the comment. Those works will be included in the revised version. Specifically, CacheBlend aims at speeding up the KV cache computation of different document chunks in RAG scenario. PromptCache aims at efficiently reusing the attention states like KV cache of frequently repeated input prompts, such as system prompts. SmartCache is designed to identify and reduce computational and memory overhead for similar queries in multi-user multi-turn conversations scenario. Notably, SmartCache can be combined with CacheBlend and PromptCache to achieve further improvements."}, {"review_id": "DYIJc7GH5d", "reviewer": "Reviewer_MXSA", "summary": "This paper introduces a novel method to store semantically equivalent queries and responses to reduce computation. While previous methods like Prefix caching only work for exact match queries, SmartCache searches for queries based on embedding similarity. The approach also takes history across multiple sessions into account, based on the observation that users typically ask several questions on the same topic, and so the method is able to more efficiently search a narrower space of cached queries based on the initial query. It also has a mechanism to detect context switches if the user changes topic.", "questions": "In the experimental design, did you experiment with different similarity thresholds for the embeddings? This seems to me to be a crucial parameter.\n\nI am surprised that answer quality increased significantly using your method. I would have expected the baseline approach to perform best, i.e. when the model is free to generate an answer to every question, and that SmartCache would have sometimes mistakenly presented a cached answer when the similarity was high.", "limitations": "None", "rating": 5, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 2, "clarity": 4, "significance": 3, "originality": 3, "strengths_and_weaknesses": "Strengths:\n\nThe work is well-motivated, and the solution is logical and well-explained.\nThe paper is very clear and comprehensive in describing all the components of the solution and walking the reader through it – it was a pleasure to read!\nExperiments are carried out on 3 different LLM families and show gains across all of them.\n\nWeakness:\n\nIt feels like the authors ran out of space, and so the experimental descriptions/analysis is not detailed enough. The whole experimental setup, evaluation, analysis and conclusion are crammed into the final 1.5 pages. I still think the authors did just enough to make the paper publishable, but would encourage them to expand this section and move some earlier sections into an appendix.", "was_revised": true, "final_justification": "The main limitation of this paper was that the experimental design was crammed into the final 1.5 pages so a bunch of important experimental details were not described/analysed in sufficient detail.\n\nThe authors successfully responded to this criticism with lots of further results tables addressing the questions from each reviewer, which gives me confidence that the experiments are robust. The authors should work these results/discussions into the camera ready version of the paper or the appendix.", "rebuttal": "We appreciate your acknowledgment of our work as **well-motivated** and the paper as **well-explained**. Below, we provide detailed responses to your constructive comments.\n\n**Q1: Different Similarity Threshold**\n\n**Response:** Thank you for your valuable comment. The similarity threshold τ_sim is set to 0.75 to give a balanced trade-off between performance and quality in the evaluation. Table 1 below shows the ablation result of τ_sim with Qwen2.5-1.5B-Instruct on CoQA.\n\nTable 1: Different similarity threshold.\n\n| Threshold | EM   | F1   | ROUGE-L | TTFT (ms) | KV Cache (GB) |\n|-----------|------|------|:--------|:----------|:--------------|\n| 0.5       | 17.4 | 35.2 | 35.5    | 26.8      | 4.1           |\n| 0.6       | 21.6 | 41.1 | 41.4    | 27.5      | 4.5           |\n| 0.7       | 23.5 | 45.3 | 45.5    | 30.2      | 6.5           |\n| 0.75      | 24.9 | 46.6 | 47.0    | 31.3      | 7.7           |\n| 0.8       | 25.4 | 47.7 | 47.9    | 49.0      | 8.5           |\n| 0.9       | 25.9 | 48.9 | 49.1    | 83.2      | 12.1          |\n\nBy increasing τ_sim, answer quality can be further improved. However it leads to higher TTFT and increased KV cache usage resulted from reduced opportunities to reuse similar queries.\n\n**Q2:** I am surprised that answer quality increased significantly using your method. I would have expected the baseline approach to perform best, i.e. when the model is free to generate an answer to every question, and that SmartCache would have sometimes mistakenly presented a cached answer when the similarity was high.\n\n**Response:** Thank you for the comment. To clarify, as shown in Section 4.3 (Table 2), the answer quality of SmartCache is close to the ideal and better than GPTCache. Additionally, as shown above, there is a trade-off between the performance gains and slight quality degradation compared to the ideal.\n\n**Q3:References missing conference/journal venue**\n\n**Response:** Thank you for pointing out the missing conference and journal venues in the references. We will correct the formatting in the revised version."}]

Author rebuttal:

Meta-review: This paper proposes Smart Cache, a system for improving the efficiency of multi-turn conversations by reusing cached contexts. The reviewers found that the authors addressed most concerns during the rebuttal, and the overall sentiment was positive, with all reviewers landing at borderline accept or above. The approach is viewed as promising, and the paper makes a clear contribution toward improving efficiency in multi-turn dialogue systems.

However, several points remain only partially resolved. The evaluation was conducted at a limited scale, and the qualtiy degradation and trade-offs in efficiency gains are not fully convincing. In addition, privacy implications were also raised, but the responses were somewhat insufficient. In particular, the AC believes that demonstrating effectiveness in actual multi-turn scenarios would be important. Given the mix of strengths and weaknesses, the paper will be forwarded to the SAC discussion. If accepted, it would be important for the authors to integrate the rebuttal clarifications into the main paper to strengthen its contribution.
Decision: Accept (poster)