Learned Prefix Caching for Efficient LLM Inference
Authors: Dongsheng Yang
Austin Li
Kai Li
Wyatt Lloyd
Venue: NeurIPS 2025 poster ()
Keywords: large language model inference
prefilling
prefix caching
learned caching algorithm
Primary area: deep_learning
TL;DR: It proposes a new learned eviction algorithm that predicts the conversation continuation probability to guide LLM prefix cache eviction.

Abstract: Prefix caching is a key technique for reducing Large Language Model (LLM) inference costs. However, the prevalent least-recently-used (LRU) eviction algorithm has a large gap to the optimal algorithm. This paper introduces LPC, the first learned method to perform LLM prefix cache eviction. LPC leverages conversational content analysis to provide predictive guidance for eviction, determining which conversations are likely to continue. These insights, combined with last access timestamps, inform more effective cache management. Extensive evaluations across three real-world datasets demonstrate that LPC achieves 18-47% reductions in required cache sizes for equivalent hit ratios and has an 11% improvement in LLM prefilling throughput in an emulated environment.

Learned Prefix Caching for Efficient LLM Inference
Dongsheng Yang
Kai Li Department of Computer Science Princeton University {dongsheng, a12926, li, wllloyda@princetoneedd
Austin Li
Wyatt Lloyd
Abstract
Prefix caching is key technique for reducing Large Language Model (LLM) inference costs. However, the prevalent least-recently-used (LRU) eviction algo- rithm has large gap the optimal algorithm. This paper introduces LPC, the first learned method to perform LLM prefix cache eviction. LPC leverages conver- sational content analysis to provide predictive guidance for eviction, determining which conversations are likely to continue. These insights, combined with last access timestamps, inform more effective cache management. Extensive evalu- ations across three real-world datasets demonstrate that LPC achieves 18-47% reductions in required cache sizes for equivalent hit ratios and has an 11% im- provement in LLM prefilling throughput in an emulated environment.
1 Introduction
Prefix caching has been recently proposed as an optimization for reducing inference latency and compute cost Large Language Model (LLM) serving systems [Gim 2024, Gao etall 2022]. By storing and reusing internal Key-Value (KV) states from previously processed input prefixes, prefix caches avoid redundant computation during the expensive prefilling phase of inference, especially in multi-turn conversations where historical context is repeatedly reused. cache hit on long context can reduce Time-To-First-TT In a production environment serving millions of requests, high hit ratios can significantly increase server throughput, translating to substantial hardware cost savings and accommodating thousands of additional users on the same infrastructure.
(TTFT) latency by 74% shown in Section B.
Despite these benefits, the effectiveness of current prefix caching systems [K et all 2023, Zheng et al., 20244] is constrained by their least-recently-usel (LRU) eviction strat- egy. which discards the least recently accessed KV blocks. Although simple and widely adopted, LRU's performance is far from optimal for LLM prefix caching. As illustrated in Figure MI there by LRU and an idealized Oracle policy that possesses perfect knowledge of future conversation continuations.
0.6
0.5
0.4
0.3
0.2
large gap between the hit ratios achieved
LAU
0.1
Cracle
0.0
50
75
100
125
150
Cache Size (x10' Lakens)
The success of learned caching policies in other domains, such as Content Delivery Networks (CDNs), suggests that learned cache designs can significantly outperform LRU and other heuristics 2022, Song et all, 20233). However, the fundamentally differ- nature of LLM workloads prevents direct transplantation of these techniques. LLM prefix caches manage semantically interconnected, variable-length segments of conversational context, which are far more complex than the relatively independent, often fixed-size objects handled by traditional learned caches in
Figure 1: A substantial gap between the hit ratios achieved by LRU and an idealized Oracle policy with per- fect knowledge of future conversa- tion continuation (LMSys dataset).
hit ratios [Song all, 2020 Yang al)
39th Conference on Neural Information Processing Systems (NcurIPS 2025).

---

other domains. This disparity necessitates the development of novel, specialized learned approaches tailored the unique characteristics of LLM workloads.
To bridge this gap, we introduce LPC, a lightweight learned prefix caching framework specifically designed for LLM conversational systems. LPC addresses two primary challenges inherent in this domain. The first is to achieve substantially higher cache hit ratios over LRU-based approaches, by accurately predicting the likelihood of conversation continuation. The second is ensuring the learn- ing component incurs negligible memory and computational overhead-a critical requirement for practical deployment within snource-constraing LLM serving environments in order to deliver an overall lower inference latency and higher inference throughput.
Our contributions are as follows: (1) We propose the first learned prefix cache to leverage the oppor- tunity to improve L.LM inference efficiency by increasing prefix reuse. (2) We design a method to learn and predict conversation continuation probability with negligible memory and computational overhead. (3) Our evaluations of an implemented prototype system shows that LPC consistently out- performs LRU, achieving 18-47% reductions in the required cache size achieve the same hit ratio. In an emulated disaggregated serving [Zhong et all, [022]] and reasoning [Snell et all, [[)Z environ- ment, this translates to LPC delivering up to 11% higher prefilling throughput and reducing the first token latency of up to 7% of requests by 42-75%. Furthermore, as a system-level cache management policy, LPC is orthogonal to and can be combined with other optimizations like FlashAttention or
speculative decoding. The implementation can be found at https: http:::///tthb.... www.thww.....a.a.a/aeee.oaaaa com/vangdsh/LLPG
2 Background
LLMs employ caching mechanisms to store internal representations of the input sequence, known as the KV cache and the prefix cache. This section describes the roles of the KV cache and the prefix cache in LLM inference, and presents the challenges in prefix cache eviction.
Input: What can you do?
Prefix Cache
4 Inactive KVL...
KV Cache
KVP'can"]
decode: "What can you do? I can" "chat"
Cache hit: What can
① Matched prefix:
Prefill: you do?
KV("What can")
KVPT]
Decode: can chat
KVP do?]
decode: "What can you do? I" "can"
@ prefill: "What you do?" TT
Figure 2: An example of the use of the KV cache and the prefix cache in LLM inference.
KV cache for LLM inference. The KV cache stores Key- Value (KV) states to support autoregres- sive LLM inference. These KV states are computed representations of the contextual information of each token. The KV cache is heavily used during the two phases of LLM inference. Initially, the prefilling phase processes the entire input prompt. For each token in this prompt, say loken, its corresponding KV states, ku,, are computed and stored in the KV cache. This phase concludes with the generation of the first output token. Subsequently, the decoding phase utilizes KV states in the KV cache for all previous tokens of this sequence, representing the full context seen so far, to gen- erate the next token. For efficient memory management and to reduce fragmentation, the KV cache often organizes these states into fixed-size "KV blocks", where the block size is typically around 16 tokens |K IKoon et all, |ا))] Upon completion of a sequence, the KV states associated with that sequence are typically cleared from the KV cache. The primary role of this KV cache is thus to eliminate redundant computations for token contexts within individual generation sequence.
Prefix caching. The standard KV cache typically discards KV states upon request completion. This prevents reuse across related requests, such requests continuing previous conversation, forcing costly re- pe-prefilliig of shared contexts. The prefix cache addresses this by retaining KV states from completed sequences. When new request arrives, the system queries the prefix cache for the longest matching prefix, for example tokenj.... tokenm. of the input. If its KV states (kv),,.., kum) are found, they are made available to KV cache for the current request. Prefilling is then only required for the remaining, unmatched portion of the input, from tokenm +1 onwards, significantly reducing computation. An example shown in Figure D where the prefilling of "What can" is skipped because it matches in the prefix cache.

---

Prefix cache eviction. The prefix cache has limited capacity as it uses precious GPU memory. When the prefix cache becomes full or the KV cache demands more memory, the system must evict existing KV blocks. The choice of which blocks to evict is crucial: well-designed eviction policy directly improves the prefix caches hit ratiothe fraction of future requests that find a matching prefixthereby reducing redundant computation and improving throughput and latency.
However, existing systems the LRU policy, which evicts the least-recently used blocks. This approach, while reasonably effective in traditional caching domains, falls short in LLM serving environments. Prefix cache workloads exhibit different temporal and spatial locality patterns. In human-LLM interactions, users naturally pause between turns to read, think, or compose follow-up queries, meaning useful prefixes may go untouched for extended periods despite being highly rele- vant. Furthermore, LRU fails to capture the semantic or task-related context of conversationfactors that are often more predictive future reuse than recency alone. A conversation centered around complex, unresolved task is more likely to continue than one involving brief, close-ended ex- change. These differences motivate us design new eviction strategies that go beyond recency and account for the unique behavioral and semantic patterns in LLM workloads.
3 Learned Prefix Caching
This section presents learned prefix caching approach to significantly advance the state-of-the art in cache management for LLM inference. Our goal is to improve prefix cache effectiveness by intelligently selecting which Key-Value (KV) blocks evict when space needed, thereby reducing prefilling latency and increasing throughput. To achieve this, our method must satisfy three key requirements:
High Hit Ratios: The approach should consistently and substantially outperform the LRU baseline in terms of cache hit rate across diverse workloads.
Improved Latency and Throughput: The method must be efficient without introducing bottlenecks, enabling both lower latency and higher throughput.
Minimal Overhead: The system should impose negligible overhead in terms of GPU memory usage and computational cost.
3.1 Overview of the LPC Framework
KV Cache
Conversation
LLM model
LPC Predictor
Prefix Cache
Data Parsing
Text axx. Aepeadinn
LPC eviction algorithm
MLP
Continuation probability
KV State, (o), time)
Evict
Figure 3: Overview of LPC. The LPC framework contains predictor and an eviction algorithm.
We propose LPC, learned prefix caching meet the requirements above. Figure E illustrates the design of LPC. The main difference between this approach and common prefix cache is that LPC uses predictor inform the replacement algorithm about which blocks evict, instead of picking the least recently used ones.
The LPC predictor takes the conversation history and outputs probability score indicating the likelihood of the conversation continuing. Its steps are:
1. Data parsing: extract the user prompts from the conversation history.
2. Text embedding: generate a semantic embedding vector from the parsed user prompts.
3. MI.P-based classification: feed the text embedding and other features into a Multilayer Perceptron (MLP) classifier to predict the conversation continuation probability p.

---

The LPC eviction algorithm has two rumning components:
1. Cache Insertion and Eviction: insert KV blocks into min-heap based on their updated probabilities p': evict the KV block with the smallest p' when the prefix cache is full.
2. Probability Update: periodically apply a decay function to the predicted continuation prob- ability p to get an updated probability p' that factors in the elapsed time.
3.2 LPC Predictor
This section describes the three components of the LPC predictor: data parsing, the text embedding model, and the MI.P classifier.
Data parsing. To provide the model with sufficient context for accurate predictions, text and nu- merical features are extracted from the conversation. These include the text of the new user prompt and the user prompts of the preceding N conversational turns. Our implementation uses N 4 to keep much of contextual information available while keeping resource usage low. For similar reasons we do not include model responses because they contain less new information than user prompts, a finding validated by our ablation study in the appendix. Additionally, the conversation length, represented as the number of turns already occurred, is included scalar feature, which is especially informative for conversations with more than N turns.
Text embedding model. For transforming raw text into meaningful vector representations, we uti- lize the multtiinngull-e--smaa
model [Wang et all 2024] (with MIT license) from the family of sentence transformer models. We choose this model because it is capable of representing im- portant patterns but is highly efficient. This model has 118M parameters. Such efficient models often have constraints on input length; for mul multilingual-ett e5--aal,, this 512 tokens. A key challenge, therefore, is to compress the input exceeding the length limit while preserving essential information. We employ a strategic token allocation as follows: the 512-token budget is distributed evenly across each of the N + user request turns. For each turn's 's allocated token budget, half the tokens are sourced from the beginning of the prompt, and the other half from the end. This strategy is informed by research indicating that the most important information in textual input often lies in these extremities [Lmu et all, 2023). The output of the text embedding model a vector with 384 dimensions.
MLP-based classification Model. The classification model is a 3-layer MLP, each layer with 128 hidden neurons. We select it because it is very efficient while having sufficient capability to learn for our task. It accepts concatenated feature vector as input, which consists of the 384- dimensional text embedding vector and a scalar value representing the number of conversation turns. The model outputs a single probability value, ranging from 0 1, indicating the confidence that the conversation will continue beyond the current turn.
We have made several design decisions to minimize the overhead of the predictor, First, we use small text embedding and MLP models that have low GPU memory overhead. Second, the predictor runs only once for each user request, where the request rate is typically lower than 10 per second per GPU. Third, the predictor is stand-alone component running in parallel to LLM inference. Lastly, one predictor can pair with multiple LLM inference instances to further amortize the memory overhead and computational overhead through batching.
3.3 Training
This section outlines how LPC is trained, including training data collection, training configurations, and retraining.
Training data. Training and validation data are collected from a dedicated partition of conver- sations from the target dataset (as described in Section H2).. This partition is strictly isolated and excluded from any datasets used for online evaluation. The size of the partition half of the dataset. A separate model is trained for each dataset.

---

Training configurations. The training works as follows. First, during training, only the weights of the MLP classifier are updated; the pre-trained text embedding model is kept frozen. This partial fine-tuning method allows the MI.P to adapt specifically to the continuation prediction task while ensuring the framework remains lightweight. Full fine-tuning would be operationally expensive for frequent retraining, whereas training only the tiny MLP head is extremely fast (typically less than 10 minutes), making daily adaptation feasible. Second, text embeddings for the training dataset are pre- computed and cached after the first epoch to significantly accelerate training. Subsequent epochs directly reuse these cached embeddings because the text embedding model is frozen, bypassing repeated computations and saving 90% of the training time. A binary cross-entropy loss function is employed. The loss is weighted with more weights for the minority class to handle class imbalance. The optimizer is Adam with a learning rate of 5 x 10 10"4. 4. The training runs until convergence (within 20 epochs in our evaluation), and the checkpoint with the lowest loss on the validation dataset is saved for running online inference.
Retraining, The predictor is trained offline on historical data. To adapt to gradual shifts in user behavior or evolving conversational dynamics and mitigate the risk of performance degradation from out-of-distribution (OOD) data, periodic offline retraining is the primary strategy. We propose daily retraining, standard practice for ML-based cache policies to adapt to evolving query patterns [Song et al, 20233] Training takes about 10 minutes, making this schedule affordable. For use cases with extremely high OOD risk, such as applications driven by rapidly changing current events, the system could leverage online learning to update itself incrementally. However, for typical conversa- tional workloads, the general patterns of continuation are relatively stable, making frequent offline retraining sufficient and operationally simpler choice.
3.4 Eviction Algorithm
This section introduces the LPC eviction algorithm in more detail. Our design largely follows the LRU prefix cache eviction algorithm but uses a dynamic conversation continuation probability in- stead of the last access time to rank objects.
3.4.1 Data Structure and Operations.
The key data structure is a min-heap. Blocks in the heap are prioritized based on dynamically calculated continuation probability of their associated conversation. The primary inputs associated with each sequence's ss K blocks are its initial predicted continuation probability and its last access timestamp.
The operations are:
Insertion:
When the decoding of sequence completes, its KV blocks are removed from the KV cache and inserted into the prefix cache's min-heap according to their initial pre- dicted continuation probability.
Promotion: If prefix of an incoming user request matches a sequence of KV blocks the prefix cache, these blocks are considered "hit." They are then promoted to the KV cache and are consequently removed from the prefix cache's min-heap.
Probability update: The algorithm performs a periodic probability decay to be able to evict the KV blocks with over-estimated continuation probability. At configurable fixed intervals (defaulting to 10 seconds), LPC iterates through all KV blocks in the prefix cache. For each block, its associated continuation probability is re-evaluated using a decay func- tion below. The min-heap is then reorganized to accurately reflect these updated probabili- ties.
Eviction: When the prefix cache reaches its capacity, or when memory needs to be re- claimed (e.g.. due to expansion of the KV cache), an eviction process is triggered. The KV block located at the root of the min-heap is selected and removed from the prefix cache.
3.4.2 Probability Decay Function
To handle KV blocks with over-estimated continuation probability predictions from staying in the cache forever we periodically update their probabilities. We implement decay function to adjust
5

---

the conversation continuation probability as time elapses since its last interaction. This function's s input includes the initially predicted continuation probability (prob ongiaai)) the current system time (fimecur), and the timestamp of the last access to this KV block (fimelaa)- Its output is the current, aime-adjusted probability (probeur), which is calculated as:
proboriginal x decay
prob cur
original decay (1 proboriginal
The decay factor itself is an exponential function the elapsed time:
decay- = exp(-(timeeur - time x scaal)
Here, scale is hyperparameter controlling the decay rate. It was tuned empirically, and a good rule of thumb is set to the inverse of the average turn interval. Since this interval is approx- imately 100 seconds across our datasets, we set scale = 10-2 in all experiments. This mechanism ensures that prefixes from inactive conversations receive progressively lower continuation probabil- ities, making them more likely eviction candidates.
3.5 Handling KV Blocks Shared by Multiple Conversations.
A significant challenge in prefix caching arises with KV blocks shared across multiple distinet con- versations, such as those resulting from common system prompts or identical initial user queries. Naive metadata management that simply overwrites the score of such blocks with the latest score can lead to suboptimal eviction decisions, because a popular prefix shared by multiple conversations would be incorrectly evicted by the most recent related conversation if it has low score. Our algo- rithm incorporates specific logic for these shared blocks. The rimetast for shared block is always updated to most recent access timestamp. Furthermore, when shared block is accessed as part of new or existing conversation, its effective continuation probability for eviction decisions is set to the maximum of its prob calculated now from the previously stored metadata and probecr cal- culated from the incoming metadata. This max-pooling operation across all conversations sharing the block is represented as:
prob block max mprobrob block .prob
This strategy ensures that block shared by multiple conversations, where at least one is highly likely to continue and has been recently active, is not prematurely evicted due to association with a less likely-to-contnng conversation.
4 Evaluation
To evaluate how well LPC meets its design goals, we focus on three evaluation questions:
How much does LPC improve prefix cache hit ratios compared to LRU? 4
To what extent does the higher hit ratio of LPC translate into improved throughput and reduced latency? (5.5)
What are the memory and computational overheads introduced by LPC? (..6)
4.1 Implementation and Evaluation Environments
We implement prototype of LPC on top of the vLLMM serving framework [Kwon et all, 2023)) which is licensed under Apache-2.0. The conversation continuation predictor runs as an external component, while the cache eviction policy is integrated directly into the vLLM prefix cache. Unless otherwise specified, all vLLM arguments use their default values. The implementation is based on the main branch of vLLM as of March 10, 2025.
We allocate GB of GPU memory for the predictor used LPC. This allocation breaks down as follows: approximately 560 MB for PyTorch CUDA context and kernel initialization, 240 MB for loading the multil multilingual sisiaal-e5-mmaal model weights in f6oat16 precision, and around 200 MB for runtime memory to store inputs and attention buffers.
All experiments were conducted on NVIDIA H100 GPUs, equipped with 80 GB of HBM3 mem- ory. We use the Qwen3-32B- 3 model [wwnn Team, 2025 to run inference. It is 32-billion

---

parameter reasoning model and is licensed with Apache 2.0. We use single GPU to run the model. Considering the memory used by the model and pytorch, the memory available for KV cache and Prefix cache is 40 GB in total. The other hardware usage includes 8 CPUs and 64 GB CPU memory. The baseline vL.LM's ' LRU cache eviction algorithm.
Our evaluations use the following metrics:
Hit Ratio:
The average prefix cache hit ratio across all requests.
"ime-to-FFrsstTokee (TTFT): The latency observed by the client from sending request to receiving the first response token.
Throughput: The average number of sequences prefilling per second.
To ensure a fair comparison with the LRU-based approach, deduct 1 GB from LPC's prefix cache size in all evaluations, to account for its GPU memory overhead.
4.2 Workload
We prefix cache hit ratio comparisons real-world conversational datasets. These include:
LMSys: large-scale collection of user-chatbot dialogues
Zheng al, 2023a
ShareGPT: conversations with ChatGPT [ShareGP Team. 2023).
Chatbot-Arena: platform where users compare various LLMs [Chinng et al, 2024
LMSys has a limited use license. ShareGPT has Apache-2.0 license. Chatbot-Arena has a CC license. Table II summarizes the statistics of the three datasets. The follow-up ratio is the percentage of requests that will receive future request of the same conversation.
Table 1: Dataset Properties.
Dataset
Total conversations
Avg. input length
Avg. output length
Follow-up ratio
LMSys
106
63 tokens
179 tokens
35%
ShareGPT
9.5 101
113 tokens
305 tokens
55%
Chathot-Arena
3.3 x 10a
36 tokens
155 tokens
8%
An important limitation public datasets is the lack of timestamp information, making difficult to evaluate LLM systems under realistic conversational workloads. Simple Poisson- based arrival models are insufficient: they ignore LLM generation timespotentially placing a user request before the previous response finishesand do not account for conversation start times.
To overcome this limitation, introduce two-phase strategy that separates think time model- ing from I.LM response times and schedules conversation start times to simulate realistic server load. In the pre-run phase, chat intervals within each conversation sampled from exponential dis- tribution with mean Aehat, and conversation start times are spaced to keep the number of concurrent active conversations below target threshold Noonv (default 200), emulating static load balancer.
In the runtime phase, each conversation starts at its assigned time, and subsequent requests occur after adding the sampled user interval to the actual completion time of the previous LLM response. Evaluations run for 20 minutes to avoid capturing the cool-down period at the end of the trace.
4.3 Hit Ratios Across Varying Cache Sizes
To understand how LPC performs under different memory budgets, we compare its prefix cache hit ratio with LRU's while varying total token capacity. We vary the combined size of the prefix and KV caches from 60K to 160K tokens (approximately 15.6 GB to 40 GB). They are allocated across the prefix and KV caches dynamically by vLLMM The chat interval parameter Aahae is fixed at 100.
Figure E shows that LPC consistently outperforms LRU across all datasets and cache sizes. The improvement relative LRU 13% to 38% on LMSys (Figure [a),, 14% to 98% on ShareGPT (Figure (r), and 15% to 30% on Chatbot-Arena (Figure (6)).
7

---

LAU
LRU
LAU
Nit Ratio
Hit
80
120
100
140
100
140
160
Cache Size (x10' takens)
Cache Size (x1d? tokens)
Cache Size (x10) takens)
(a) LMMys
(h) ShareGPT
(c) Chatbot-Arena
Figure 4: Hit ratio vs. cache size. LPC achieves consistently higher hit ratios than LRU, which translates to 18 47% reductions in the required cache size to achieve the same hit ratio as LRU.
To achieve the same hit ratio LRU, LPC requires significantly less memory across all three datasets. For example, on Chatbot-Arena, LPC reaches a 30% hit ratio with only 85 K tokensjust 57% of the 150 K tokens required by LRU. Similar memory savings are observed on LMSys (up to 18% less) and ShareGPT (up to 30% less). This is 7.2-18.8 GB GPU memory saving for 40 GB cache. These results show LPC substantially improves memory efficiency in prefix caching.
4.4 Impact of Chat Intervals on Hit Ratio
LRU
LRU
LRU
0.40
0.35
0.35
0.35
0.30
0.30
0.30
0.25
0.25
0.25
0.20
0.20
0.20
100
200
300
400
100
200
300
400
100
200
300
400
Chat Interval (6)
Chat interval
Interval
(a) LMSys
(b) ShareGPT
(c) Chatbot-Arens
Figure 5: Hit ratio vs. chat intervals. LPC consistently higher hit ratio than LRU, with 5-18% relative improvements.
To evaluate the robustness of LPC under varying user interaction patterns, we fix the cache size GB (the full available GPU memory) and vary the chat interval parameter Aehar to simulate different user think times. Shorter intervals represent more frequent user interactions, which may increase cache contention, while longer intervals simulate slower-paced conversations. with poten- tially higher reuse of cached prefixes.
As shown in Figure 5 LPC consistently outperforms LRU across all datasets and interval settings, achieving 5% to 18% higher hit ratios. Specifically, improvements range from 5% to 18% on LMSys (Figure 5a), 11 % 15% ShareGPT (Figure (56)), and 7% 16% on Chatbot-Arena (Figure 5d). These results show that LPC maintains strong performance across diverse conversational workloads, adapting well to varying user behaviors.
4.5 Throughput and Latency
To evaluate the impact on throughput and latency by the improved hit ratios, we use a microbench- mark that emulates prefilling-only server in disaggregated inference framework. This method widely used in industry and is becoming next-generation standard [Zhong et all 2024 Latel etall 2024, Hu 2024, Qin all 2025]. A prefilling request rate under 10 req/s is reasonable and often conservative estimate for production models with moderate prompt lengths [Zhong et all 2022!]
use a synthetic workload with longer context lengths to reflect the recent trend test-time seal- ing related to the wide adoption of reasoning models [Snelletta 2I2E Chen et all [02)] Recent

---

studies show that the generation length significantly increases from around 10 tokens generated by non-reasoning models to around 1,000 tokens generated by reasoning models [Han el all 2024. Sun etall 2025]. This increase in the generation lengths means an increase in the lengths of the matched part of the context of the input of the next conversation round.
contexts, because generated
Context: 1000 tokens
Context: 3000 tokens
15.0
Prefilling
Context Hit
Throughpun (eegi)
300
10.0
30.21
TTFT (mc)
7.5
200
5.0
100
40
tokens
500 tokens
1000 tokens
50 tokens
500 tokens
1000 tokens
Hit Ratio (56)
New Prompt Length
New Prompt Length
Figure 6: Prefilling throughput (req/s) function of hit ratio.
Figure 7:
Time To First Token (TTFT) comparison for context cache miss hit across varying context and prompt lengths.
Figure & shows the prefilling throughput gain as the hit ratio improves. In this microbenchmark. every input has a 1000-tokens context and 50-token new prompt. The x-axis the prefix cache hit ratio of the full contexts, including the new prompts that always miss. A hit ratio boost from 35 percent to 42 percent, which LPC achieves in Figure ba. leads an increase in throughput from 10.01 req/s to 11.11 1 req/s, indicating an 11.10% higher prefilling throughput for LPC over LRU in the emulated environment.
Figure 0 quantifies the reduction in TTFT achieved by a context cache hit. It shows that a context hit can keep the TTFT low regardless of the context length, which reduces TTFT by up 73% compared to context miss. It also shows that the TTFT of a hit barely increases when the new prompt length is smaller than 500, emphasizing the advantage of context hit. These result indicate that 7 percent hit ratio increase can translate to 7% of requests seeing their TTFT reduced by 42-75% in the emulated environment.
4.6 Memory and Computational Overhead
We reserved GB of GPU memory for LPC's predictor and deduct GB from LPC's prefix cache as described Section I. In this case, LPC takes about 1.25% of the GPU memory running our evaluations. A sensitivity study analyzing the impact of using varying memory for LPC is provided in the appendix.
The GPU computational overhead of LPC is minimal due to the high efficiency of the predictor model, which can process up to 1,000 predictions per second per GPU. A detailed instrumentation is also presented in the appendix.
5 Related Work
Prefix Caching and Eviction Policies. Many works have explored optimizing LLM inference with prefix caching, but none have placed an emphasis on how eviction policies adjust to the LLM workloads. PromptCache [Gum et all 2024 reuses common prompt segments with precomputed attention states. RAGCache [I ct all
2024 proposes prefix-aware Greedy Dual-Size Frequency eviction policy for RAG but for conversational workload. CacheBlend [Yao all 2025] uses pre- fix cache and cross SGLang IZheng et all 2022 builds RadixAttention to cache KV sequences in radixtree for efficient prefix matching, while vLLM won etall 2023| maps logical KV blocks to global physical cache. Both adopt least-recently used (LRU) eviction strategy. Dynamo INVIDIA Research Teaml 2025 uses first-in-irst-ouut (FIFO) eviction, while CachedAttention [Gao all 2024] presents a scheduler-aware eviction scheme based the queue of incoming requests for offline batched inference. Marconi [Pan etall 2025) factors computational savings for LRU eviction, but the approach is tailored to state-space models and does not extend to transformer architectures.

---

ML-based Caching Algorithms. Existing caching research explores machine learning an ap- proximate to optimal eviction decisions. LeCaR [[iet et all 2018| and CACHEUS [Rodnguez etall 2021] apply online learning and expert-ensemble frameworks to further learn cache eviction policies. LRB [Song at, 2020| MAT [Yang al,, 2023], and HALP [Song et 2023] leveraging machine learning in content delivery networks, demonstrates 10% to 20% hit ratio improvement and cost saving over the best heuristic algorithm and LRU. However, these algorithms cannot be directly applied to LLM prefix caches due to the large difference in their caching paradigms.
Learned Optimizations for LLM Inference. Recent L.M inference systems use learned ap- proaches improve performance. [Zheng all, 2023N] adds instructions for predicting prompt length in order to group requests with similar latencies, leading to throughput increase compared regular batched inference. al [2022]]] and Quu all [20245]]] train smaller separate models to predict output length, finding that shortest-job-fie -ortst batching reduces per-token latency significantly compared first-come- first-serve. best-comme-iss--eeve.e. Finally, lain et all [2025] explores heuristic gouristie-uuided guided reinforce- ment bearning-based router for workload- -ware scheduling. Our work on LPC complement this work by focusing its learned optimizations on different component of LLM inference: prefix caching,
Conclusion and Future Work
This paper presents the first learned prefix cache LPC to improve LLM inference efficiency. It uses MI on history data to predict conversation continuation probability to improve the cache hit ratio over LRU based approach.
Our evaluations show that LPC consistently outperforms the baseline in all three workloads, achiev- ing up 43% lower cache size utilization. The improved high-ratios translate substantially lower latency and an 11% prefilling throughput increase. This is a step towards intelligent and adaptive memory management LLM serving systems.
While our approach demonstrates significant improvements over LRU, this study has several limi- tations that plan to address in future work. First, our evaluation relies workloads augmented with synthetic timestamps, which may not fully capture real-world user behavior. Second, our com- parison is primarily against LRU; evaluating against broader range of heuristic eviction strategies would offer more comprehensive view. Third, while the principles of LPC are model-agnostic,, experiments focused architectures (e.g., Llama, Mistral) would further strengthen our claims of generality.
single model family; demonstrating its effectiveness across different
This work also opens several promising directions for future research. A finer-grained analysis of hit ratios on specific domains (e.g., coding, agentic tasks) could provide deeper insights into which conversational patterns most amenable to caching. Another exciting avenue is extending LPC to handle multi modal inputs. This could be achieved by replacing the separate text encoder with the main LLM's s own internal hidden states as a rich, holistic embedding for the predictor, making the framework highly adaptable to the future of multi-modal generative AI. These directions represent promising next steps for advancing learned prefix caching in LLM serving systems.
Acknowledgments
This material is based upon work supported by the National Science Foundation under Grant Num- ber CNS-2321723. Any opinions, findings, and conclusions or recommendations expressed this material are those of the author(s) and do necessarily reflect the views of the National Science Foundation.
References
Qiguang Chen, Libo Qin, Jinhao Liu, Dengyun Peng, Jiannan Guan, Peng Wang, Mengkang Hu, Yuhang Zhou, Te Gao, and Wanxiang Che. Towards reasoning era: A survey of long chain-of- thought for reasoning large language models. arXiv preprint 2503:25500.0566.. 2025.
Wei-Lin Chiang, Lianmin Zheng, Ying Sheng, Anastasios Nikolas Angelopoulos, Tianle Li, Dacheng Li, Banghua Zhu, Hao Zhang, Michael Jordan, Joseph E Gonzalez, et al. Chatbot
10

---

arena: An open platform for evaluating Ilms by human preference. In Forty-first International Conference on Machine Learning, 2024.
Yichao Fu, Siqi Zhu, Runlong Su, Aurick Qiao, Ion Stoica, and Hao Zhang. Efficient Ilm scheduling 15792.
by learning rank, 2024. URL
Bin Gao, Zhuomin He, Puru Sharma, Qingxuan Kang, Djordje Jevdjic, Junbo Deng, Xingkun Yang, Zhou Yu, and Pengfei Zuo. Cost-efficient large language model serving for multi-turn conversa- tions with cachedattention. In Proceedings of the 2024 USENIX Annual Technical Conference (USENIX ATC 24), pages 111-126. USENIX Association, 2024.
In Gim, Guojun Chen, Seung-seob Lee, Nikhil Sarda, Anurag Khandelwal, and Lin Zhong. Prompt cache: Modular attention reuse for low-latency inference. Proceedings of Machine Learning and Systems, 6:325-338, 2024.
Tingxu Han, Zhenting Wang, Chunrong Fang, Shiyu Zhao, Shiqing Ma, and Zhenyu Chen. Token- budget-aware IIm reasoning. arXiv preprint urXiv:24/2.18555 2024.
Cunchen Hu, Heyang Huang, Junhao Hu, Jiang Xu, Xusheng Chen, Tao Xie, Chenxi Wang, Sa Wang, Yungang Bao, Ninghui Sun, et Memserve: Context caching for disaggregated Ilm serving with elastic memory pool, arXiv preprint arXiv:2406. 11t0:22446.7766555 2024.
Kunal Jain, Anjaly Parayil, Ankur Mallick, Esha Choukse, Xiaoting Qin, Jue Zhang, fñigo Goiri, Rujia Wang, Chetan Bansal, Vietor Rühle, Anoop Kulkarni, Steve Kofsky, and Saravan Rajmo- han. Intelligent router for IIm workloads: Improving performance through workload-aware load balancing, 2025, URL https: /tap::://arxv...o wwt./oog/////aeeeooeooooo
ab// 2408 13510,
Chao Jin, Zili Zhang, Xuanlin Jiang, Fangyue Liu, Xin Liu, Xuanzhe Liu, and Xin Jin. Ragcache: Ef- ficient knowledge caching for retrieval-augmented generation. arXiv preprint arXiv:2404. 12457, 2024.
Woosuk Kwon, Zhuohan Li. Siyuan Zhuang, Ying Sheng, Lianmin Zheng, Cody Hao Yu, Joseph Gonzalez, Hao Zhang, and Ion Stoica. Efficient memory management for large language model serving with pagedattention. In Proceedings of the 29th ACM Symposium Operating Systems Principles (SOSP), pages 611-626, 2023.
Nelson F Liu, Kevin Lin, John Hewitt, Ashwin Paranjape, Michele Bevilacqua, Fabio Petroni, and Percy Liang. Lost in the middle: How language models use long contexts. arXiv preprint arXiv:2307.03172, 2023.
NVIDIA Research Team. Nvidia dynamo.
http:///itthuu..oaaaoooaaaa dynamo, 2025.
Accessed: 2025-05-12.
Rui Pan, Zhuang Wang, Zhen Jia, Can Karakus, Luca Zancato, Tri Dao, Yida Wang, and Ravi Ne- travali. Marconi: Prefix caching for the era of hybrid Ilms. In Proceedings of the 7th Conference on Machine Learning and Systems (MLSys 2025), 2025.
Pratyush Patel, Esha Choukse, Chaojie Zhang, Aashaka Shah, fñigo Goiri, Saeed Maleki, and Ri- cardo Bianchini. Splitwise: Efficient generative I1m inference using phase splitting. 2024 ACM/IEE 51 Annual International Symposium Computer Architecture (ISCA), pages 118- 132. IEEE, 2024.
Ruoyu Qin, Zheming Li, Weiran He, Jialei Cui, Feng Ren, Mingxing Zhang, Yongwei Wu, Weimin Zheng, and Xinran Xu. Mooncake: Trading more storage for less computationa {KVCache- centric) architecture for serving (LLM} chatbot. In 23rd USENIX Conference on File and Storage Technologies (FAST 25), pages 155-170, 2025.
Haoran Qiu, Weichao Mao, Archit Patke, Shengkun Cui, Saurabh Jha, Chen Wang, Hubertus Franke, Zbigniew Kalbarczyk, Tamer Başar, and Ravishankar K. Tyer. Power-aware deep learning model serving with -Serve. In 2024 USENIX Annual Technical Conference (USENIX ATC 24), pages 75-93, Santa Clara, CA, July 2024a. USENIX Association. ISBN 978-1-939133-44100 URL.
https: / /www usenix.ee... http:////ooooeeeee////e//eeeeeea

---

Haoran Qiu, Weichao Mao, Archit Patke, Shengkun Cui, Saurabh Jha. Chen Wang, Hubertus Franke, Zbigniew T. Kalbarezyk, Tamer Baar, and Ravishankar K. Iyer. Efficient interactive Ilm serving with proxy model-based sequence length prediction, 2024b. URL https: httpp::://axxvooooo org/ abs/ 2404_08500
Qwen3 Team. Qwen3, 2025. URL 12.
Accessed: 2025-05-
Liana V Rodriguez, Farzana Yusuf, Steven Lyons, Eysler Paz, Raju Rangaswami, Jason Liu, Ming Zhao, and Giri Narasimhan. Learning cache replacement with (CACHEUS). In 19th USENIX Conference on File and Storage Technologies (FAST 2/), pages 341-354, 2021. 2021.
ShareGPT Team. Sharegpt dataset. https://huggiggfe
wtt//aataatts//non233115333333 ShareGPT Vicuna unfiltered, 2023. Accessed: 2025-05-12.
Charlie Snell, Jachoon Lee, Kelvin Xu, and Aviral Kumar. Scaling Ilm test-time compute optimally can be more effective than scaling model parameters. arXiv preprint 0mx1v:2408.03314, 2024.
Zhenyu Song, Daniel S. Berger, Kai Li, and Wyatt Lloyd. Learning relaxed belady for content distribution network caching. In Proceedings of the 17th USENIX Symposium on Networked Systems Design and Implementation (NSDI 2020), 2020.
Zhenyu Song, Kevin Chen, Nikhil Sarda, Deniz Altubükenn Eugene Brevdo, Jimmy Coleman, Xiao Ju, Pawel Jurezyk, Richard Schooler, and Ramki Gummadi. (HALP): Hcuristic aided learned preference eviction policy for {YouTube} content delivery network. In 20th USENIX Symposium on Networked Systems Design and Implementation (NSDI 23), pages 1 1149- 110,,116,200000 2023.
Yi Sun, Han Wang, Jiaqiang Li, Jiacheng Liu, Xiangyu Li, Hao Wen, Huiwen Zheng, Yan Liang, Yuanchun Li, and Yunxin Liu. Time up! an empirical study of Ilm reasoning ability under output length constraint. arXiv preprint arXiv:2504. 14350, 2025.
Giuseppe Vietri. Liana V Rodriguez, Wendy A Martinez, Steven Lyons, Jason Liu, Raju Rangaswami. Ming Zhao, and Giri Narasimhan. Driving cache replacement with (ML- based) [LeCaR]. In 10th USENIX Workshop on Hot Topics in Storage and File Systems (Hot- Storage 18), 2018.
Liang Wang, Nan Yang, Xiaolong Huang, Linjun Yang. Rangan Majumder, and Furu Wei. Multilin- gual e5 text embeddings: A technical report. arXiv preprint arXiv:2402.05672, 2024.
Dongsheng Yang, Daniel S. Berger, Kai Li, and Wyatt Lloyd. A learned cache eviction framework with minimal overhead. arXiv preprint arXiv:2301. 200.288888886, 2023.
Jiayi Yao, Hanchen Li, Yuhan Liu, Siddhant Ray, Yihua Cheng, Qizheng Zhang, Kuntai Du, Shan Lu, and Junchen Jiang. Cacheblend: Fast large language model serving for rag with cached knowledge fusion. In Proceedings of the Twentieth European Conference on Computer Systems, pages 94-109, 2025.
Lianmin Zheng, Wei-Lin Chiang, Ying Sheng, Tianle Li, Siyuan Zhuang, Zhanghao Wu, Yonghao Zhuang. Zhuohan Li, Zi Lin, Eric P Xing, et al. Lmsys-chat- Im: A large-scale real-world Ilm conversation dataset. arXiv preprint arXiv:2309.11999
2023a.
Lianmin Zheng, Liangsheng Yin, Zhiqiang Xie, Chuyue Sun, Jeff Huang, Cody Hao Yu, Shiyi Cao, Christos Kozyrakis, Ion Stoica, Joseph E. Gonzalez, Clark Barrett, and Ying Sheng. Sglang: Efficient execution of structured language model programs. In Advances in Neural Information Processing Systems, volume 37, pages 62557-62583, 2024.
Zangwei Zheng, Xiaozhe Ren, Fuzhao Xue, Yang Luo, Xin Jiang, and Yang You. Response length perception and sequence scheduling: An llm-empowered I1M inference pipeline, 2023b. URL https: org/
13144.
Yinmin Zhong, Shengyu Liu, Junda Chen, Jianbo Hu, Yibo Zhu, Xuanzhe Liu, Xin Jin, and Hao Zhang. (DistServe): Disaggregating prefill and decoding for goodput-optimized large language model serving. In 18th USENIX Symposium on Operating Systems Design and Implementation (OSDI 24). pages 193-210, 2024.
12

---

NeurIPS Paper Checklist
1. Claims
Question: Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?
Answer: [Yes]
Justification: The abstract and introduction accurately state LPC's contributions (e.g., first learned LLM prefix eviction using conversational analysis) and key performance results (e.g., 18 -847 cache size reduction, 11% prefilling throughput improvement), which are detailed and supported in Sections and g
Guidelines:
The answer NA means that the abstract and introduction do not include the claims made the paper.
The abstract and/or introduction should clearly state the claims made, including contributions made in the paper and important assumptions and limitations. A No or
NA answer to this question will not be perceived well by the reviewers.
The claims made should match theoretical and experimental results, and reflect how much the results
be expected to generalize other settings. is fine to include aspirational goals motivation long it is clear that these goals are not attained by the paper.
2. Limitations
Question: Does the paper discuss the limitations of the work performed by the authors?
Answer: [Yes]
Justification: Section f explicitly discusses limitations, including reliance on synthetic timestamps for evaluation, comparison primarily against LRU, and the of single lightweight prediction model architecture.
Guidelines:
The answer NA means that the paper has limitation while the answer No means that the paper has limitations, but those are not discussed in the paper.
The authors are encouraged to create separate "Limitations" section in their paper.
The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (e.g., independence assumptions, noiseless settings, model well-specification, asymptotic approximations only holding locally). The au- thors should reflect on how these assumptions might be violated practice and what the implications would be.
The authors should reflect the scope of the claims made, e.g., if the approach was only tested on a few datasets or with a few runs. In general, empirical results often depend on implicit assumptions, which should be articulated.
The authors should reflect on the factors that influence the performance of the ap- proach. For example, a facial recognition algorithm may perform poorly when image resolution is low images are taken in low lighting. Or a speech-to-tex system might not be used reliably to provide closed captions for online lectures because it fails to handle technical jargon.
The authors should discuss the computational efficiency the proposed algorithms and how they scale with dataset size.
If applicable, the authors should discuss possible limitations of their approach ad- dress problems of privacy and fairness.
While the authors might fear that complete honesty about limitations might be used by reviewers as grounds for rejection, worse outcome might be that reviewers discover limitations that aren't 't acknowledged in the paper. The authors should use their best judgment and recognize that individual actions in favor of transparency play an impor- tant role in developing norms that preserve the integrity of the community. Reviewers will be specifically instructed not penalize honesty concerning limitations.
3. Theory assumptions and proofs
13

---

Question: For each theoretical result, does the paper provide the full set of assumptions and complete (and correct) proof?
Answer: [NA]
Justification: The paper proposes an empirical method and evaluates it experimentally. does not introduce new theoretical results that require formal proofs.
Guidelines:
The answer NA means that the paper does not include theoretical results.
All the theorems, formulas, and proofs in the paper should be numbered and cross- referenced.
All assumptions should be clearly stated or referenced in the statement of any theo- rems.
The proofs can either appear in the main paper or the supplemental material, but they appear in the supplemental material, the authors are encouraged provide short proof sketch to provide intuition.
Inversely, any informal proof provided in the core of the paper should be comple- mented by formal proofs provided in appendix or supplemental material.
Theorems and Lemmas that the proof relies upon should be properly referenced.
4. Experimental result reproducibility
Question: Does the paper fully disclose all the information needed to reproduce the main experimental results of the paper to the extent that it affects the main claims and/or conclu- sions of the paper (regardless of whether the code and data are provided or not)?
Answer: [Yes]
Justification: Key details for reproducibility are provided, including implementation envi- ronment and hardware (Section H.JI), models used (Section 6.2 I.II,, datasets (Section ( synthetic timestamp generation (Section ! predictor architecture and training parame- ters (Section
Guidelines:
The answer NA means that the paper does not include experiments.
If the paper includes experiments, a No answer to this question will not be perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data are provided or not.
If the contribution is dataset and/or model, the authors should describe the steps taken to make their results reproducible or verifiable.
Depending on the contribution, reproducibility can be accomplished in various ways. For example, if the contribution is novel architecture, describing the architecture fully might suffice, or if the contribution is specific model and empirical evaluation, may be necessary either make it possible for others to replicate the model with the same dataset, or provide access to the model. In general. releasing code and data often one good way to accomplish this, but reproducibility can also be provided via detailed instructions for how to replicate the results, access to a hosted model (e.g., in the case of a large language model), releasing of a model checkpoint, or other means that are appropriate to the research performed.
While NeurIPS does not require releasing code, the conference does require all sub- missions to provide some reasonable avenue for reproducibility, which may depend on the nature the contribution. For example
(a) the contribution primarily new algorithm, the paper should make it clear how to reproduce that algorithm.
(b) If the contribution is primarily new model architecture, the paper should describe the architecture clearly and fully.
(c) the contribution a new model (e.g., a large language model), then there should either be way to access this model for reproducing the results or way to re- produce the model (e.g., with an open-source dataset or instructions for how to construct the dataset).
14

---

(d)
We recognize that reproducibility may be tricky in some cases, in which case au- thors are welcome describe the particular way they provide for reproducibility, In the case of closed-source models, it may be that access to the model is limited in some way (e.g., to registered users), but it should possible for other researchers to have some path to reproducing or verifying the results.
5. Open access to data and code
Question: Does the paper provide open access to the data and code, with sufficient instruc- tions to faithfully reproduce the main experimental results, as described in supplemental material?
Answer: [Yes]
The datasets used are publicly available (LMSys, ShareGPT, Chatbot Arena), as cited in Section !_ The code and instructions to reproduce the main experimental results is included in the supplementary material.
Justification:
Guidelines:
The answer NA means that paper does not include experiments requiring code.
Please the NeurlPS code and data submission guidelines (https://nips.co
for more details.
While we encourage the release of code and data, we understand that this might not be possible, so No an acceptable answer. Papers cannot be rejected simply for not including code, unless this is central to the contribution (c.g., for a new open-source benchmark).
The instructions should contain the exact command and environment needed to run to reproduce the results. See the NeurIPS code and data submission guidelines (https:)
http:///uuuuueeeeeeeeeaaaaaaaaaa Pnloii for more details.
The authors should provide instructions on data access and preparation, including how access the raw data, preprocessed data, intermediate data, and generated data,
The authors should provide scripts to reproduce all experimental results for the new proposed method and baselines. If only subset of experiments are reproducible, they should state which ones are omitted from the script and why.
At submission time, to preserve anonymity, the authors should release anonymized versions (if applicable).
Providing as much information as possible supplemental material (appended the paper) is recommended, but including URLs to data and code is permitted.
6. Experimental setting/details
Question: Does the paper specify all the training and test details (e.g., data splits, hyper- parameters, how they were chosen, type of optimizer, etc.) necessary to understand the results?
Answer: [Yes]
Justification: Key training details (data splits, predictor hyperparameters in Section + embedding model specifics in Section (2) and test settings (evaluation setup including
Achat Noon in Section
decay scale tuning in Section (44.2) are provided.
Guidelines:
The answer NA means that the paper does not include experiments.
The experimental setting should be presented in the core of the paper to level of detail that is necessary to appreciate the results and make sense of them.
The full details can be provided either with the code, in appendix, or as supplemental material.
7. Experiment statistical significance
Does the paper report error bars suitably and correctly defined or other appropri- ate information about the statistical significance of the experiments?
Question:
Answer: [Yes]
Justification: The plot with error bar is included in the Appendix.
15

---

Guidelines:
The answer NA means that the paper does not include experiments.
The authors should answer "Yes" if the results are accompanied by error bars, confi- dence intervals, or statistical significance tests, least for the experiments that support the main claims of the paper.
The factors of variability that the enror bars capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, or overall
run with given experimental conditions).
The method for calculating the error bars should be explained (closed form formula, call to a library function, bootstrap, etc.)
The assumptions made should be given (e.g.. Normally distributed errors).
It should be clear whether the error bar is the standard deviation the standard error of the mean.
It is OK to report I-sigma error bars, but one should state it. The authors should prefer- ably report a 2-sigma error bar than state that they have 96% CI, if the hypothesis of Normality of errors is not verified.
For asymmetric distributions, the authors should be careful not show in tables figures symmetric error bars that would yield results that are out of range (c.g. negative error rates).
If error bars reported in tables or plots, The authors should explain in the text how they were calculated and reference the corresponding figures or tables in the text.
8. Experiments compute resources
For each experiment, does the paper provide sufficient information on the com- puter resources (type of compute workers, memory, time of execution) needed to reproduce the experiments?
Question:
Answer: [Yes]
Justification: Section memory, a single GPU for the model, CPUs, and 64GB CPU memory. Section G.l mentions predictor training takes about 10 minutes. Section H. mentions evaluations for 20 minutes. Section A.I details GPU memory allocation for the predictor (1GB total). This provides good overview of resources for reported runs.
I specifies the of NVIDIA H100 GPUs with 80GB HBM3
Guidelines:
The answer NA means that the paper does not include experiments.
The paper should indicate the type of compute workers CPU or GPU, internal cluster, or cloud provider, including relevant memory and storage.
The paper should provide the amount of compute required for each of the individual experimental runs well estimate the total compute.
The paper should disclose whether the full research project required more compute than the experiments reported in the paper (e.g., preliminary or failed experiments that didn't 't make it into the paper).
9. Code ethies
Question: Does the research conducted in the paper conform, in every respect, with the NeurlPS Code of Ethics
Answer:
[Yes]
Justification:
We conduct research conforming to the Code of Ethics.
Guidelines:
The answer NA means that the authors have not reviewed the NeurIPS Code of Ethics.
If the authors answer No, they should explain the special circumstances that require deviation from the Code of Ethics.
The authors should make sure to preserve anonymity (e.g., there a special consid- eration due to laws regulations in their jurisdiction).
10. Broader impacts
16

---

Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?
Answer: [NA]
Justification: The research focuses on improving the computational efficiency of LLM in- ference systems using publicly available conversational datasets. It does not have potential negative societal impacts.
Guidelines:
The answer NA means that there is no societal impact of the work performed.
If the authors answer NA or No, they should explain why their work has no societal impact or why the paper does not address societal impact.
Examples of negative societal impacts include potential malicious unintended uses (e.g., disinformation, generating fake profiles, surveillance), fairness considerations (e.g., deployment of technologies that could make decisions that unfairly impact spe- cific groups), privacy considerations, and security considerations.
The conference expects that many papers will be foundational research and not tied particular applications, let alone deployments. However, if there is direct path any negative applications, the authors should point it out. For example, it is legitimate point out that an improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point out that a generic algorithm for optimizing neural networks could enable people to train models that generate Deepfakes faster.
The authors should consider possible harms that could arise when the technology is being used as intended and functioning correctly, harms that could arise when the technology is being used intended but gives incorrect results, and harms following from (intentional or unintentional) misuse of the technology.
If there are negative societal impacts, the authors could also discuss possible mitiga- tion strategies (e.g., gated release of models, providing defenses addition to attacks, mechanisms for monitoring misuse, mechanisms to monitor how system learns from feedback over time, improving the efficiency and accessibility of ML).
11. Safeguards
Does the paper describe safeguards that have been put in place for responsible release of data models that have high risk for misuse (e.g.. pretrained language models, image generators, or scraped datasets)?
Question:
Answer: [NA]
Justification: The paper does not release models or datasets that would pose a high risk for misuse.
Guidelines:
The answer NA means that the paper poses no such risks.
Released models that have high risk for misuse or dual-use should be released with necessary safeguards allow for controlled use of the model, for example by re- quiring that users adhere to usage guidelines restrictions to access the model or implementing safety filters.
Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images.
We recognize that providing effective safeguards challenging, and many papers do require this, but we encourage authors to take this into account and make best faith effort.
12. Licenses for existing assets
Question: Are the creators or original owners of assets (e.g., code, data, models), used in the paper, properly credited and are the license and terms of use explicitly mentioned and properly respected?
Answer: [Yes]
17

---

Justification: The paper properly cites the original papers for the datasets (LMSys, ShareGPT. Chatbot Arena in Section !.2)) and models/frameworks used (vLLM in g.ll multilingual-e5-sman in B.2. Qwen3 in I.I... The specific licenses (e.g., Apache-2.0, MIT, CC) and terms of use for these existing assets are explicitly stated in the paper.
Guidelines:
The answer NA means that the paper does not existing assets.
The authors should cite the original paper that produced the code package dataset.
The authors should state which version of the asset used and, possible, include URL.
The name of the license (e.g., CC-BY 4.0) should be included for each asset.
For scraped data from a particular source (e.g., website), the copyright and terms of service of that source should be provided.
If assets released, the license, copyright information, and terms of use in the pack- age should be provided. For popular datasets, haperswithcode com datasets has curated licenses for some datasets. Their licensing guide can help determine the li- cense of a dataset.
For existing datasets that are re-packaged, both the original license and the license of the derived asset (if has changed) should be provided.
this information is not available online, the authors are encouraged to reach out to the assct's s creators.
13. New assets
Question: Are new assets introduced in the paper well documented and is the documenta- tion provided alongside the assets?
Answer: [NA]
Justification:
The paper does not release new asset.
Guidelines:
The answer NA means that the paper does not release new assets.
Researchers should communicate the details of the dataset/code/moodl as part of their submissions via structured templates. This includes details about training, license, limitations, etc.
The paper should discuss whether and how consent obtained from people whose asset is used.
At submission time, remember to anonymize your assets (if applicable). You can either create an anonymized URL or include an anonymized zip file.
14. Crowdsourcing and research with human subjects
Question: For crowdsourcing experiments and research with human subjects, does the pa- per include the full text instructions given to participants and screenshots, if applicable, well as details about compensation (if any)?
Answer: [NA]
Justification: The paper uses existing publicly available datasets (LMSys, ShareGPT, Chat- bot Arena) which consist of prior human-chatbot interactions. The research described does not involve conducting new crowdsourcing experiments or new direct research with human subjects.
Guidelines:
The answer NA means that the paper does not involve crowdsourcing research with human subjects.
Including this information the supplemental material is fine, but the main contri- bution of the paper involves human subjects, then as much detail as possible should be included in the main paper.
According to the NeurIPS Code of Ethics, workers involved in data collection, cura- tion, or other labor should be paid at least the minimum wage the country of the data collector.
18

---

15. Institutional review board (IRB) approvals or equivalent for research with human subjects
Question: Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or equivalent approval/review based the requirements of your country institution) were obtained?
Answer: [NA]
The research uses existing datasets and does not involve new interactions with human subjects. Therefore, IRB approval for the work presented in this paper itself is not applicable. Ethical considerations related to the original dataset collection would have been the responsibility of the original dataset creators.
Justification:
Guidelines:
The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.
Depending on the country in which research is conducted, IRB approval (or equiva- lent) may be required for any human subjects research. If you obtained IRB approval, you should clearly state this the paper.
We recognize that the procedures for this may vary significantly between institutions and locations, and expect authors to adhere to the NeurIPS Code of Ethics and the guidelines for their institution.
For initial submissions, do not include any information that would break anonymity (if applicable), such as the institution conducting the review.
16. Declaration of LLM usage
Does the paper describe the usage of LLMs if it is an important, original, or non-standard component of the core methods in this research? Note that if the L.IM used only for writing, editing, or formatting purposes and does not impact the core methodology, scientific rigorousness, or originality the research, declaration is not required.
Question:
Answer: [NA]
Justification: LLMs not used in developing implementing the methods. The algorithm itself uses a text embedding model to embed the input but it is not an LLM.
Guidelines:
The answer NA
that the method development in this research does involve LLMs as any important, original, or non-standard components.
Please refer to our LLM ottp:s////eerrrssccccooooooereee for what should or should be described.
19

---

A Additional Experiments
A.1 LPC Computational Overhead
In this section, we extend Section A.6 to address the computational overhead of our proposed method LPC.
Metrics. We evaluate GPU computational activity using three fine-graine tne-grained metrics collected by NVIDIA Data Center GPU Manager: SM Active, SM Occupancy, and Tensor Pipe Active. SM Ac- tive indicates the proportion of cycles during which at least one warp was assigned to any Streaming Multiprocessor (SM), reflecting overall SM computational engagement. SM Occupancy measures the ratio of active resident warps to the theoretical maximum warps per SM, averaged across SMs. Tensor Pipe Active represents the ratio of cycles where any tensor pipe was active.
Setup. We run three configurations: our proposed LPC method, the baseline LRU method, and the LRU w/ predictor configuration. The LRU w/ predictor configuration integrates the LPC predictor with the standard LRU policy; this setup is specifically designed to measure the predictor's ' intrinsic computational overhead while maintaining cache hit ratio comparable to the baseline LRU. The ex- periments are conducted on the LMSys dataset with a 40 GB cache size. The chat interval parameter Achu is 100.
Table 2: GPU computational activity on LMSys with 40 GB cache size.
Metric
LPC
LRU
LRU w/ predictor
SM Active (%)
39.70
41.74
42.04
SM Occupancy (%)
7.63
8.05
8.12
Tensor Pipe Active (%)
11.42
12.05
12.07
Hit Ratio (%)
46.50
40.90
40.90
Results. The results in Table Z show that LPC reduces the overall GPU computations by about 5% Specifically, LPC achieves a 4.89% decrease in SM Active, a 5.22% decrease in SM Occupancy, and 5.23% decrease in Tensor Pipe Active, underscoring its effectiveness in reducing the overall computational load due to its higher prefix cache hit ratio. Second, to assess the computational overhead introduced by the LPC predictor, we compared the LRU baseline with and without running an LPC predictor. The results indicate that the predictor's overhead is minimal: SM Active increased by only 0.72%, SM Occupancy by 0.87%, and Tensor Pipe Active by a mere 0. 0.17% when the predictor was added to LRU. This confirms that the LPC predictor can be integrated to LPC with a marginal computational overhead.
A.2 LPC Memory Overhead Sensitivity Analysis
We analyze the sensitivity of LPC's performance to its GB GPU memory overhead by scaling the reserved memory for the predictor by 0x, 1x, 2x, and 4x. Figure N shows the impact hit ratio using the same setup as Section M
The results show that the GPU memory allocated to LPC's predictor has 0.6% to 8.1% impact on its hit ratio. This observed by comparing LPC (0x) with LPC (1x). The hit ratio sees a reduction ranging from 0.6% to 2.4%, at cache size of 16 x 10 tokens (40 GB for H100 after loading Qwen3- 32B-FP8). The hit ratio impact is more pronounced at smaller cache capacities, as the reduction in hit ratio when moving from LPC (0x) to LPC (1 ranges from 2.3% to 8.1%, at cache size of 6.4 x 1 tokens (16 GB). These findings indicate that though LPC's prediction mechanism fun- damentally superior to LRU, minimizing its direct GPU memory overhead is crucial for maximizing its cache hit ratio benefits, especially when GPU memory resources are limited.
Despite the internal overhead variations, LPC consistently outperforms LRU. When varying LPC's predictor overhead from 0x to 4x, LPC achieves 4.3% to 16.7% higher hit ratios than LRU at the cache size of 16 x 10a tokens (40 GB). At the smaller cache size of 6.4 x 10a tokens (16 GB), LPC achieves 5.8% to 119.0% higher hit ratios than LRU. These findings indicate that LPCC's prediction mechanism is fundamentally superior to LRU.
20

---

0.5
Ratio
Hit Rotto
LFC (4x)
LPC (1x)
LPC (0x)
LRU
140
100
140
100
140
160
Cache Size [x10* tokens)
Cache Size (x10* tokens)
Cache Size (x10' tokens)
(a) LMSys
(b) ShareGPT
Chatbot-Arena
Figure 8: Sensitivity of LPC's cache hit ratio to its internal GPU memory overhead, compared with LRU. LPC (Nx) denotes LPC configured with N times of its actual GPU memory requirement.
A.3 LPC Predictor Accuracy and Ablation Study
We evaluate the effectiveness of the LPC predictor using the Matthews Correlation Coefficient (MCC) and maaro-aaveraged F1 score (FI macro). MCC is a robust metric for binary classifica- tion with imbalanced classes. The F1 macro computes the F1 score independently for each class and then averages them. L.Prums an ablation study using only the number of turns as input, where predictions based on the majority label of training data binned by turn count.
Table 3: Prediction correctness of LPC predictor and an ablation study using only the number of turns as the input (LPCCums). Higher values better for all metrics.
Dataset
Method
MCC
Fl macro
LMSys
LPC LPC
0.3510
0.6755
0.3863
0.6923
ShareGPT
LPC -0,0012
0.4165
LPC
0.2815
0.6407
Chatbot-Arena
LPC 0.0775
0,4741
LPC
0.3689
0.6837
The results in Table B indicate that the full LPC predictor achieves 0.28 to 0.36 MCC and is al- ways superior to the LPC ablation. This suggests that leveraging the semantic content of the conversation provides valuable signals for predicting continuation likelihood. For instance, the Chatbot-Arens dataset, LPC shows substantial improvement in MCC (0.3689 vs. 0.0775) and F1 macro (0.6837 0.4741) over LPC The negative MCC for LPCtums ShareGPT indicates that using only turn count as a predictor is no better than random guessing for that specific dataset. The LMSys dataset shows comparable MCC and F1 maero between LPC and LPClums and indicates that a light-weight predictor such as L.PGumms may be strong enough in some scenarios. Overall, the results show the effectiveness of our content-aware features in predicting the conversation continua- tion.
B Additional Ablation Studies
B.1
Ablation on Input Features: User Prompts vs. Model Responses
Our initial design choice use only user prompts as input was motivated by keeping the predictor lightweight. As shown in Table gl including the model response provides no significant, consistent improvement and can even slightly degrade performance. pact embedding model, chosen for efficiency, may lack the capacity to distill useful signals from the often longer and more varied model responses, potentially introducing noise. The user's prompt appears to be a more direct and potent signal for their immediate intent.
hypothesize this because com-
21

---

Table Ablation study on predictor input features: prompts only vs. prompts plus model responses. Performance is measured by Matthews Correlation Coefficient (MCC) and Fl macro score.
Dataset
Features
MCC
F1 macro
Chatbot Arena
Prompts + Model Response
0.3370
0.6685
Chatbot Arena
Prompts Only
0.3485
0.6738
LMSys
Prompts + Model Response
0.3843
0.6849
LMSys
Prompts Only
0.3797
0.6897
ShareGPT
Prompts + Model Response
0.2779
0.6286
ShareGPT
Prompts Only
0.2777
0.6348
B.2 Ablation on Prompt Window Length
We chose a default history window of prompts (the current prompt previous). This was based on our analysis that most conversations in the datasets are short (e.g.. 94-99% have five turns or fewer). To validate this choice, we experimented by increasing the prompt window length to 10. Ta- ble 5 shows that this did not yield an improvement and often resulted in slightly lower performance. This reinforces that the most recent conversational history is the most predictive for these workloads. This hyperparameter can be tuned for workloads with substantially longer conversations.
Table 5: Ablation study on the prompt window length for the predictor.
Dataset
Prompt window length
MCC
Fl Macro
Chatbot Arena
5
0.3485
0.6738
Chatbot Arena
10
0.3416
0.6707
LMSYS
5
0.3797
0.6897
LMSYS
10
0.3727
0.6862
ShareGPT
5
0.2777
0.6348
ShareGPT
10
0.2496
0.6167
B.3 Generalizability: Unified vs. Specialized Predictor
A crucial question for practical deployment is whether a single, general-purpon predictor can per- form as well as predictors specialized for different data distributions. "Unified" predictor
tested this by training combined dataset and comparing its performance against our default "Spe- cialized" models on each test set. The results in Table B show that the unified model competitive on the Chatbot Arena and LMSys datasets. However, its performance is notably lower ShareGPT, suggesting that while core conversational patterns are generalizable, certain datasets like ShareGPT may contain unique characteristics that benefit from domain-specinn training. A larger model might improve generalizability, but at the cost of higher overhead.
Table 6: Performance comparison of Unified predictor vs. Specialized predictors.
Test Dataset
Model Type
MCC
F1 macro
Chatbot Arena
Unified
0.3357
0.6678
Chatbot Arena
Specialized
0.3485
0.6738
LMSys
Unified 0,3802
0.6871
LMSys
Specialized
0.3797
0.6897
ShareGPT
Unified
0.1271
0.5281
ShareGPT
Specialized
0.2777
0.6348
22

---

B.4 Hit Ratio Comparisons with Oracle
To assess the upper bound for further potential improvements, we compare the hit ratio of LPC against an Oracle policy, which possesses perfect future knowledge of whether conversation will continue. Figure presents this comparison alongside LRU across the three datasets for various cache sizes. The evaluation setup is the same in Section
The results indicate that LPC consistently reduces the hit ratio gap between LRU and Oracle. On the LMSys dataset (Figure ba)), the reduction in the gap is from 22% to 30%. On the ShareGPT dataset (Figure M the reduction is from 28% to 64%. The remaining gap to Oracle is only about 0.5 times the current LPC-over-LRU gain at the smallest cache size. We hypothesize that it is because LPC has higher accuracy in high-confidence predictions, and the small cache size is only enough hold those high-confidence prefixes. For the Chatbot-Arena dataset (Figure Ed), the reduction of the hit ratio gap from LRU to Oracle achieved by LPC ranges from 25% 28%.
These observations highlight the substantial advancements LPC delivers in prefix caching. While a performance ceiling defined by the Oracle henchmark remains, contend that LPC effectively cap- tures the vast majority of predictive information available within the conversation text itself. There- fore, we believe that future efforts substantially close the remaining gap and approach Oracle-level performance will likely need to incorporate new data dimensions, such as user access patterns, mov- ing beyond sole reliance on textual content for prediction.
0.4
Rabio
Hit Ratio
0.3
LRU LPC Oracle
0.1
0.0
80
100
120
140
160
100
140
80
100
120
140
160
Cache Size [x10' tokens)
Cache Size (x10* tokens)
Cache Size (x100* tokens)
(a) LMSys
(b) ShareGPT
(c) Chatbot-Arena
Figure 9: Hit ratio vs. cache size. LPC reduces the gap from LRU to Oracle by 22% to 66%.
B.5 Experiment Statistical Significance
To assess the statistical significance and stability of our results, we repeated the experimental runs for at least times for each data point. Figure ID presents the hit ratios with error bars indicating the observed minimum and maximum values across these runs. The analysis of this data reveals that the variations are consistently small. Specifically, the maximum absolute difference between the highest and lowest observed hit ratio for any given configuratio than 0.013. The relative variation (max-min difference divided by the mean hit ratio) is also small, always below 4.6% and in most cases lower than 1 1%. This low variance across repeated experiments indicates that the performance differences observed between our proposed method and the baseline robust and not attributable random fluctuations, underscoring the statistical significance of our findings.
(algorithm, dataset, and cache size) is less
0.5
0.5
0.5
LRU
LRU LPC
LRU
0.4
LPC
0.4
0.4
LPC
Hit Ratio
0.3
0.3
Hit Rotio
0.3
0.2
HR
0.2
0.2
0.1
0.1
0.1
0.0
0.0
0.0
80
100
120
140
160
80
100
120
140
160
60
100
120
140
160
Cache Size [x10" tokens!
Cache Size (x10' tokens)
Cache Size (x10' tokens)
(a) LMSys
(b) ShareGPT
(c) Chatbot-Arena
Figure 10: Hit ratio vs. cache size with error bars. The variation in the hit ratio is smaller than 0.013.
23


Reviews:
[{"review_id": "vRUWdleKrH", "reviewer": "Reviewer_fKMe", "summary": "This work propose to replace the LRU cache eviction in LLM serving by a learnable component, to improve the kv cache hit ratio for better efficiency. Experiments show that the proposed Learned Prefix Caching can achieve 18-47% reductions in required cache sizes for equivalent hit ratios and has an 11% improvement in LLM prefiling throughput in an emulated environment.", "questions": "NA", "limitations": "NA", "rating": 4, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 3, "significance": 3, "originality": 3, "strengths_and_weaknesses": "Strength:\n1) The paper is clear and well written.\n2) As the improvement of LLMs, the serving cost is growing fast due to more usage across the world. It is very important to improve the model serving efficiency. In addition to that, the GPU memory is growing slowly but the model size is growing fast. Using memory smarter is good.\n\n\nWeakness:\n1) Since there is an additional round of training before serving LLM, there is risk that model is easier to overfit and the improvement might be way smaller on OOD data.\n2) As the community is switching to scale RL and test-time compute (more frontier models like Gemini and O3 are using thinking mode by default), more serving cost is from decoding stage instead of prefilling.\n3) It would be good to have a finer grained results like the hit ratio on the prompts from different domains (e.g. agent, coding, chatbot...)", "was_revised": false, "final_justification": "", "rebuttal": "We are very grateful to the reviewer for their time and for providing such a positive and insightful assessment of our work. We are encouraged that they found the paper \"clear and well written\" and recognized the importance of improving LLM serving efficiency. Their thoughtful comments have given us an excellent opportunity to clarify the robustness and applicability of our approach.\n\nWe address the specific weaknesses raised below.\n\n***\n\n### **1. On the Risk of Overfitting and Out-of-Distribution (OOD) Performance**\n\nWe agree this is a crucial point for any learned system deployed in a dynamic environment. The risk of performance degradation due to model staleness or out-of-distribution (OOD) data is real, but it is a well-understood challenge in production machine learning that can be effectively managed.\n\n#### **1.1 Mitigation Through Periodic Retraining**\nOur primary strategy for mitigating this risk is **periodic retraining**. On line 172 in our paper, we propose to retrain LPC daily. This is a standard practice in ML-based cache policy to continuously adapt to evolving query patterns and user behaviors. For example, as described in \"HALP: Heuristic Aided Learned Preference Eviction Policy for YouTube Content Delivery Network\" (Song et al., NSDI '23)**, retraining the ML-based cache policy weekly is sufficient for Youtube. It indicates that the cache workload is relatively stable in their production environement.\n\n#### **1.2 Adaptive Strategies for High-Risk Scenarios**\nFor use cases where the OOD risk is particularly high—such as applications driven by rapidly changing current events or highly dynamic agentic workflows—the system can leverage **online learning**. In this paradigm, the model would update itself incrementally with each new data point or mini-batch that arrives. This allows for near-real-time adaptation, ensuring the system remains robust even in the face of rapid and continuous distributional drift.\n\nWe will inlcude these discussions in our paper.\n\n***\n\n### **2. On the Relevance of Prefill vs. Decoding Stages**\n\nThis is an excellent, forward-looking point. We agree with the reviewer that the computational landscape of LLM serving is evolving, with models like Gemini and others employing more complex, multi-step reasoning that increases the relative cost of the decoding stage.\n\nHowever, we believe that optimizing the prefill stage remains a crucial and highly relevant challenge for two key reasons:\n1.  **Prefill is Universal and Frequent:** The vast majority of current LLM applications (e.g., interactive chatbots, RAG systems, summarization) are multi-turn and rely heavily on repeated prefilling. Every turn in a conversation after the first requires a new prefill operation. Improving the KV cache hit ratio directly reduces the latency and computational cost of every single one of these turns.\n2.  **Complementary, Not Competing:** Optimizations for prefilling and decoding are not mutually exclusive; they are complementary. By making the prefill stage 11% faster, as shown in our work, we free up valuable GPU resources that can then be dedicated to handling longer and more expensive decoding steps.\n\nTherefore, while the cost distribution may shift in future models, our work provides a significant and immediate efficiency gain for a fundamental operation that is ubiquitous in today's LLM serving environments.\n\n***\n\n### **3. On Finer-Grained, Domain-Specific Results**\n\nThis is a valuable suggestion. For this initial work, our goal was to establish the foundational viability of LPC across broad, widely-used conversational datasets. However, a finer-grained analysis of the hit ratio on specific domains (e.g., coding, agentic tasks, creative writing) is an excellent and logical next step for this research. Such an analysis would provide deeper insights into which types of conversational patterns are most amenable to caching. We will add this as a key direction for future work in our conclusion.\n\n***\n\nOnce again, we thank the reviewer for their constructive feedback. We believe that by incorporating these clarifications, the final paper will be substantially stronger. We hope our responses have addressed the reviewer's concerns and have reinforced the value and technical soundness of our contribution."}, {"review_id": "3CvmAGwJDI", "reviewer": "Reviewer_z2XZ", "summary": "This paper introduces a learned prefix cache framework that analyzes conversation content and evicts outdated cache information smartly. Different from LRU that evicts the least recent cache information, the method utilizes a lightweight ML predictor to analyze the historical user prompts to estimate the possibility of the conversition continuing. The framework then leverages the possibiltiy to guide eviction decisions in the prefix cache.", "questions": "1. How do you determine the value of scale? \n\n2. Can you provide more explanations about the probability decay function?\n\n3. Can you compare LPC with LRU on throughput?", "limitations": "Yes", "rating": 4, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 2, "significance": 3, "originality": 3, "strengths_and_weaknesses": "Strengths: \n\n1. The method adapts to online scenarios and can handle high query volumes, making it practical for production LLM serving systems where low latency and high throughput are critical.\n\n2. The authors introduce a novel eviction algorithm that analyzes the historical prompts to predict conversation continuation.\n\n3. The method achieves comparable hit ratios compared to LRU but reduces the cache size by 18–47%.\n\nWeakness:\n\n\n1. The method relies on the last 4 user prompts for prediction, which might fail to capture the complex interactions between users and the chatbot. Users might formulate new questions based on the LLM's responses rather than just their previous prompts. Thus, it might be biased to only use the 4 latest prompts to estimate if the chat is finished or not.\n\n2. The 4 prompt window might miss long-term conversational patterns. In a case when users seek thorough explanations from a chatbot, usually it is difficult to determine if the users' questions are fully addressed with 4 questions.\n\n\n3. The approach trains separate predictor models for each dataset (Line 161), which poses practical challenges in real-world LLM serving. Real-world LLM serving systems handle highly diverse queries across users and topics, making dataset-specific training almost impractical for production deployment. Is your proposed method data-sensitive? Is it possible to handle a more general case?", "was_revised": false, "final_justification": "", "rebuttal": "We are sincerely grateful to the reviewer for their thoughtful and positive assessment of our work. We are particularly encouraged by the recognition of LPC's practicality for production systems and its significant 18%-47% cache size reduction over LRU. The insightful questions and comments provided are invaluable and will help us to substantially improve the clarity and completeness of our paper.\n\nWe address the specific points raised below.\n\n***\n\n### **Response to Weaknesses**\n\n#### **1.1 Ablation Study on User Prompts vs. Model Responses** \n\nWe thank the reviewer for this excellent and very important point. We agree that the choice of input features is critical, and we appreciate the suggestion that model responses could potentially provide useful signals for predicting user follow-up. Our initial design choice to use only user prompts was motivated by keeping the predictor as lightweight as possible.\n\nTo investigate this empirically, we conducted a new ablation study during the rebuttal period. In this study, we trained a predictor that includes the most recent model response as an additional feature. The results are summarized in **Table 1**.\n\n**Table 1:** Prompt only vs. Prompt + model responses as input features. The predictor performance is measured by The F1 score and Matthews Correlation Coefficient (MCC) \n\n| Dataset | Features | MCC | F1 macro |\n| :--- | :--- | :--- | :--- |\n| Chatbot Arena | Prompts + Model Response | 0.3370 | 0.6685 |\n| Chatbot Arena | Prompts Only | 0.3485 | 0.6738 |\n| LMSys | Prompts + Model Response | 0.3843 | 0.6849 |\n| LMSys | Prompts Only | 0.3797 | 0.6897 |\n| ShareGPT | Prompts + Model Response | 0.2779 | 0.6286 |\n| ShareGPT | Prompts Only | 0.2777 | 0.6348 |\n\nThe results show that including the model response provides no significant, consistent improvement. We hypothesize this is because the compact embedding model (`multilingual-e5-small`), chosen for its efficiency, may lack the capacity to effectively distill useful signals from the often longer and more varied model responses, potentially introducing noise. The user's prompt appears to be a more direct and potent signal. We will add this result and discussion to the paper.\n\n#### **1.2 Ablation Study on the Prompt Window Length**\n\nWe thank the reviewer for raising the excellent question about the 4-prompt window. Our analysis of the 3 public datasets shows that the vast majority of conversations are short. For instance, in the Chatbot Arena and LMSys datasets, 99.6% and 94% of conversations have five turns or fewer, respectively. To further validate this choice, we experimented by increasing the prompt window length from 5 (our default) to 10. As shown in **Table 2**, this change did not yield an improvement and often resulted in slightly lower performance. This reinforces that the most recent history is the most predictive for the available workload. Nevertheless, this hyperparameter can be mannualy tuned if LPC is applied on a workload with substentially more turns per conversation. We will add this result and discussion to the paper.\n\n**Table 2:** Prompt window length.\n| Dataset | Prompt window length | MCC | F1 Macro |\n| :--- | :--- | :--- | :--- |\n| Chatbot Arena | 5 | 0.3485 | 0.6738 |\n| Chatbot Arena | 10 | 0.3416 | 0.6707 |\n| LMSYS | 5 | 0.3797 | 0.6897 |\n| LMSYS | 10 | 0.3727 | 0.6862 |\n| ShareGPT | 5 | 0.2777 | 0.6348 |\n| ShareGPT | 10 | 0.2496 | 0.6167 |\n\n#### **2. On the Practicality and Generalizability of the Predictor**\n\nWe thank the reviewer for raising this crucial question regarding the practical deployment and data sensitivity of LPC. We agree that a generalizable model is essential for production systems.\n\nTo directly test this, we conducted a key experiment where we trained a single \"unified predictor\" on a combined dataset and compared its performance against models trained on specialized datasets (our default).\n\n**Table 3:** Unified predictor trained on a combined dataset vs. Specialized predictor.\n\n| Test Dataset | Model Type | MCC | F1 macro |\n| :--- | :--- | :--- | :--- |\n| Chatbot Arena | Unified | 0.3357 | 0.6678 |\n| Chatbot Arena | Specialized | 0.3485 | 0.6738 |\n| LMSys | Unified | 0.3802 | 0.6871 |\n| LMSys | Specialized | 0.3797 | 0.6897 |\n| ShareGPT | Unified | 0.1271 | 0.5281 |\n| ShareGPT | Specialized | 0.2777 | 0.6348 |\n\nThe results in **Table 3** show that the unified model's performance is competitive with specialized models on the **Chatbot Arena** and **LMSys** datasets. However, on the **ShareGPT** dataset, the unified model's performance is notably lower. This suggests that while core conversational patterns are generalizable, certain datasets like ShareGPT may contain unique characteristics that benefit from domain-specific training. It is also possible that each dataset has unique request patterns that contradicts other datasets, leading to performance drop if it is trained in a mixed way. We hypothesize that a larger model or a full fine tuning of the embedding model can increase the learning capacity and make it have less performance drop on ShareGPT. However, the expensive cost prevent us from adopting the above method. Therefore, we still need a separate model for the best accuracy and overhead trade-off. We will add this important result and discussion to the paper.\n\n***\n\n### **Responses to Questions**\n\n#### **1. How do you determine the value of `scale`?**\n\nThank you for this insightful question. The `scale` hyperparameter was tuned empirically on our validation sets. We found that a good rule of thumb is to set $s$ to be the inverse of the average turn interval within a conversation. Since this interval is approximately 100 seconds across our datasets, we set the scale hyperparameter $s = 10^{-2}$ in all our experiments. We will add a sentence to Section 3.4.2 of the paper to make this tuning process and rationale more explicit.\n\n#### **2. Can you provide more explanations about the probability decay function?**\n\nThank you for this question. The probability decay function is designed to complement our ML predictor with a temporal component. Its purpose is to ensure that a prefix whose continuation probability might have been overestimated by the predictor does not remain in the cache indefinitely if it goes unused.\n\nThe function adjusts the initial predicted probability ($p_{\\text{orig}}$) based on an exponential decay factor, $d$. The current probability ($p_{\\text{cur}}$) is calculated as:\n$$p_{\\text{cur}} = \\frac{p_{\\text{orig}} \\cdot d}{p_{\\text{orig}} \\cdot d + (1 - p_{\\text{orig}})}$$The decay factor $d$ decreases as the time since the prefix was last accessed ($\\Delta t = t_{\\text{cur}} - t_{\\text{last}}$) increases:$$d = \\exp(-\\Delta t \\cdot s)$$\nAs $\\Delta t$ increases, $d$ approaches zero, which smoothly reduces the prefix's score ($p_{\\text{cur}}$) and makes it a more likely candidate for eviction.\n\n#### **3. Can you compare LPC with LRU on throughput?**\n\nYes, the reviewer is correct to ask for this comparison. We provide this analysis in Section 4.5 and Figure 6 of the main paper, where our microbenchmark evaluates prefilling throughput as a direct function of the cache hit ratio.\n\nSpecifically, Lines 293-295 and Figure 6 show that a hit ratio improvement from 35% to 42%—a gain that LPC consistently achieves over LRU in our tests (e.g., Figure 5a)—directly translates to a throughput increase from 10.01 req/s to 11.11 req/s. This demonstrates an 11% higher prefilling throughput for LPC over LRU in that environment. We will revise the text in Section 4.5 to make this comparison even more direct and explicit.\n\n***\n\nOnce again, we thank the reviewer for their constructive feedback. We believe that by incorporating these clarifications, the final paper will be substantially stronger. We hope our responses have addressed the reviewer's concerns and have reinforced the value and technical soundness of our contribution."}, {"review_id": "V8Tp9zb8DY", "reviewer": "Reviewer_X9iT", "summary": "The paper introduces Learned Prefix Caching (LPC), a new method for managing the cache used during Large Language Model (LLM) inference. The paper notes that the commonly used least-recently-used (LRU) algorithm is not ideal for this purpose, as it does not account for the conversational context. LPC uses a learned predictor to analyze conversational content and determine which conversations are likely to continue, which in turn helps decide which data to keep in the cache. The evaluations show that LPC consistently outperforms LRU, leading to a significant reduction in the required cache size for the same performance and an improvement in prefilling throughput.", "questions": "1. Besides the evaluation on cache hit ratio and throughput, conducting an evaluation of the classifier accuracy would provide a more comprehensive understanding of the predictor's performance. I am also curious why you only train the MLP part of the predictor? Full fine-tuning would likely yield better results without affecting the deployment overhead.\n\n2. A growing workload of LLM serving systems is to handle multi-modal inputs, such as images and videos. How does the proposed method handle multi-modal inputs? Is it possible to extend the LPC predictor to work with multi-modal data?", "limitations": "yes", "rating": 4, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 2, "clarity": 3, "significance": 2, "originality": 2, "strengths_and_weaknesses": "### Strengths\n\n1. This paper studies a meaningful and practical problem in LLM serving systems, specifically the inefficiency of the LRU algorithm for prefix caching. As the inference demand for LLMs grows, optimizing cache management is crucial for reducing latency and compute costs.\n\n2. Extensive evaluations are conducted across three real-world datasets (although the timestamps are synthetic), demonstrating the effectiveness of the proposed method. The results show that LPC can reduce the required cache size by 18-47% for the same hit ratio and improve LLM prefilling throughput by 11%. The paper also assesses the minimal overhead of the LPC predictor, showing that it is a practical solution.\n\n3. The paper is generally well-written and easy to follow.\n\n### Weaknesses\n\n1. The LPC predictor introduces additional complexity and overhead to the LM serving system. While the paper claims to observe substantial performance improvements on several datasets, there always exists a worst-case scenario where no shared prefix is found and all the computational overhead of the predictor is wasted. The LRU strategy, while sub-optimal, is much simpler and does not require any training.\n\n2. Although the problem setting is well-motivated and novel, the proposed solution itself is not particularly innovative. The idea of training a classifier over pre-trained embeddings has been explored in various contexts. The novelty of this work lies more in its application to LLM prefix caching rather than in the underlying methodology.", "was_revised": true, "final_justification": "This paper tackles an important problem as LLM inference traffic is growing rapidly; therefore, I increase my score to 4. However, the technical part does not endow sufficient novelty that prevents me from assigning higher scores.", "rebuttal": "We thank the reviewer for their detailed feedback and for recognizing that our work addresses a meaningful and practical problem with extensive evaluations. The insightful questions and comments provided are invaluable and will help us to substantially improve the clarity and completeness of our paper. \n\nWe address the specific points raised below.\n\n***\n\n### **Response to Weaknesses**\n\n#### **1. On Complexity and Worst-Case Overhead**\n\nWe agree with the reviewer that LRU is simpler. However, we argue this trade-off between simplicity and performance is highly favorable for LPC in its target domain.\n\nOur work focuses on **multi-turn conversational workloads**, where reusing previous turns as a prefix is inherent to the application. The \"worst-case\" scenario of no prefix reuse is therefore rare in this setting. This is validated by recent workload characterizations; for example, a study of two production traces from a large cloud provider, detailed in **\"KVCache Cache in the Wild: Characterizing and Optimizing KVCache Cache at a Large Cloud Provider\" (Li et al., arXiv '25)**, found that prefix reuse is common, with ideal hit rates of **62%** and **54%**. This confirms our problem setting is highly practical.\n\nFurthermore, there is a clear industry trend of replacing simple heuristics like LRU with more sophisticated, learned policies when the cost savings outweigh the complexity. A prominent example is the deployment of a machine learning-based caching policy in **YouTube's content delivery network, as described in \"HALP: Heuristic Aided Learned Preference Eviction Policy for YouTube Content Delivery Network\" (Song et al., NSDI '23)**. Similarly, LPC's minimal overhead is justified by its significant gains in memory efficiency and throughput.\n\n#### **2. On Novelty**\n\nWe appreciate the reviewer's perspective on novelty. We agree that the core machine learning components, such as using classifiers on pre-trained embeddings, are established techniques.\n\nThe primary novelty of our work lies in:\n1.  **Problem Formulation:** Identifying and formalizing the specific inefficiency of LRU for the stateful, conversational nature of LLM prefix caching.\n2.  **System Design:** Designing a lightweight and practical *end-to-end system* (LPC) that effectively applies a predictive approach to this new and important domain.\n3.  **Demonstrated Impact:** Proving that this application leads to significant, measurable improvements in key system metrics.\n\nTherefore, the contribution is in the novel application, problem formulation, and effective system design, rather than the invention of a new ML algorithm from scratch.\n\n***\n\n### **Responses to Questions**\n\n#### **1.1 On Predictor Accuracy**\n\nWe have evaluated the predictor's performance using the F1 score and Matthews Correlation Coefficient (MCC) on line 511 in our paper's Appendix, which confirm that our predictor using user prompt as the input is effective and necessary. We paste the results here for your convinience.\n\n| Dataset | Method | MCC | F1 macro |\n| :--- | :--- | ---: | ---: |\n| LMSys | LPC with turns input only | 0.3510 | 0.6755 |\n| LMSys | LPC | **0.3863** | **0.6923** |\n| ShareGPT | LPC with turns input only | -0.0012 | 0.4165 |\n| ShareGPT | LPC | **0.2815** | **0.6407** |\n| Chatbot-Arena | LPC with turns input only | 0.0775 | 0.4741 |\n| Chatbot-Arena | LPC | **0.3689** | **0.6837** |\n\n#### **1.2 MLP-only Training**\n\nThank you for the excellent question. We chose to train only the MLP head to ensure the entire framework remains lightweight and practical for deployment. In a production environment, the predictor would need to be periodically retrained (e.g., daily) on recent query logs to adapt to evolving usage patterns.\n\nFull fine-tuning of the entire embedding model on a large daily corpus could take several hours, making frequent retraining operationally expensive. In contrast, training only the tiny MLP head is extremely fast—typically taking only ~10 minutes. This efficiency makes daily online retraining feasible, ensuring LPC remains adaptive to user behavior without incurring significant computational cost.\n\n#### **2. On Handling Multi-modal Inputs**\n\nThis is a fantastic question and we are working on it with a new learning paradigm:\n\nThis new learning paradigm of LPC replaces the text embedding model with **the main LLM's own internal state**. To be specific, after the LLM processes a prompt, we extract the final layer's hidden states (e.g., from the KV cache) to use as a rich embedding for our predictor.\n\nThis improved method can be directly applied to multi-modal input because the LLM's internal state is a holistic representation of all inputs it has processed, whether they are text, images, or other data types. This eliminates the need for a separate encoder, reducing system complexity and overhead while providing a powerful, context-aware signal to the LPC predictor. This makes the LPC framework highly adaptable to the future of multi-modal generative AI. We will add this to our future work section.\n\n***\n\nOnce again, we thank the reviewer for their constructive feedback. We believe that by incorporating these clarifications, the final paper will be substantially stronger. We hope our responses have addressed the reviewer's concerns and have reinforced the value and technical soundness of our contribution."}, {"review_id": "SE02FOYmHo", "reviewer": "Reviewer_hPyr", "summary": "The paper introduces LPC (Learned Prefix Caching), the first learned eviction algorithm for prefix caches in Large Language Model (LLM) inference systems. Traditional prefix caching uses an LRU (Least Recently Used) policy, which underperforms due to its inability to anticipate future usage based on semantic context. LPC overcomes this by predicting the likelihood that a conversation will continue using a lightweight neural model that analyzes recent user prompts. These predictions guide the eviction policy to retain prefixes more likely to be reused, improving cache hit rates. Evaluated on real-world datasets (LMSys, ShareGPT, Chatbot-Arena), LPC achieves 18–47% reductions in required cache size for the same hit ratio compared to LRU, and up to 11% higher prefilling throughput in an emulated environment.", "questions": "1. In Line 129, the authors say that “we do not include model responses because they contain less new information than user prompts”. Could you further explain why the model response will not help judge the probability of a follow-up question? For example, if the model’s response is not complete enough, isn’t that the probability that the user will have a follow-up can be higher? Do you have any analysis or ablation on this?\n\n2. In Line 151, the authors say that “the request rate is typically lower than 10 per second per GPU”. Could you explain how did you get the 10/sec number?\n\n3. In Line 160, the authors say that they train “a separate model for each dataset”. Could you explain why do you need separate models for different datasets? Can you train a single predictor for all settings?", "limitations": "yes", "rating": 4, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 3, "significance": 3, "originality": 2, "strengths_and_weaknesses": "[Strengths]\n1. The paper proposes a learned eviction strategy tailored to prefix caching in LLM serving – LPC, identifying an important issue in efficient LLM serving while alleviating the limitations of existing methods like LRU.\n2. The paper presents adequate analysis on the performance improvements and overheads introduced by the proposed method, especially with the experiments in the appendix.\n3. The presentation and visual components of this paper are clear and easy to understand. The authors also include detailed training setups in this paper.\n\n[Weaknesses]\n1. Ablations on several important designs are missing. See questions section for more details.\n2. The cache hit ratio improvement over the LRU baseline seems to be not very significant, and the overall throughput improvement is a bit marginal.\n3. Experiments are limited to specific settings such as QWen-32B. Experiments on different model series and different model sizes may better prove the effectiveness of the design.", "was_revised": true, "final_justification": "The authors response has addressed most of my questions. I will increase my score to 4.", "rebuttal": "We sincerely thank the reviewer for their positive assessment and constructive feedback. We are encouraged that you recognized the importance of the problem, the adequacy of our analysis, and the clarity of the paper's presentation. The insightful questions and comments provided are invaluable and will help us to substantially improve the clarity and completeness of our paper.\n\nWe address the specific points raised below.\n\n***\n\n### **Response to Weaknesses**\n\n#### **1. On the Significance of Performance Improvements**\n\nWe thank the reviewer for this critical point. We realize we could have better contextualized the impact of our results. In large-scale LLM serving, seemingly small percentage gains in core metrics lead to substantial downstream effects on performance and cost. \n\nOur method LPC provides a 11% increase in prefilling throughput. In a production environment serving millions of requests, this means serving thousands of additional users per day on the same infrastructure, which can translate to millions of dollars in saved hardware costs annually.\n\nFurthermore, our approach is orthogonal to most other inference optimization techniques. As a cache eviction policy, LPC operates at the system management layer and can be seamlessly combined with architectural improvements like FlashAttention or model-level optimizations like quantization and speculative decoding. The benefits from LPC are therefore additive to gains from these other methods. We will revise the paper to better highlight the practical significance of our results.\n\n#### **2. On the Generality of Experimental Settings**\n\nThis is an excellent point. Our current experiments are focused on the Qwen-32B model. We chose it as a representative example of a modern, high-performance LLM. The fundamental principles of LPC are model-agnostic. The predictor learns patterns of *conversation continuation* only from user prompts, not specific to one model's architecture. The model output length is specified by the benchmark dataset. The model output content is not used by the baseline or our algorithm.\n\nHowever, we agree that demonstrating this effectiveness across different model families (e.g., Llama, Mistral) and sizes would greatly strengthen the paper. We will add comprehensive results in our paper.\n\n#### **3. On Missing Ablations**\n\nThank you for noting this. We provide detailed answers to your questions below, which include new empirical results for the most important design ablations: the choice to exclude model responses and the use of separate vs. unified predictors.\n\n***\n\n### **Responses to Questions**\n\n#### **1. Could you further explain why the model response will not help? Do you have any analysis or ablation on this?**\n\nThis is an excellent question. Our initial design choice to exclude model responses was motivated by keeping the predictor as lightweight as possible. To validate this empirically, we conducted an ablation study where we trained a predictor that includes the most recent model response as an additional feature.\n\n**Table 1:** Prompts only vs. Prompts + Model Respons as input features.\n\n| Dataset | Feature | MCC | F1 macro |\n| :--- | :--- | :--- | :--- |\n| Chatbot Arena | Prompts + Model Response | 0.3370 | 0.6685 |\n| Chatbot Arena | Prompts Only | 0.3485 | 0.6738 |\n| LMSys | Prompts + Model Response | 0.3843 | 0.6849 |\n| LMSys | Prompts Only | 0.3797 | 0.6897 |\n| ShareGPT | Prompts + Model Response | 0.2779 | 0.6286 |\n| ShareGPT | Prompts Only | 0.2777 | 0.6348 |\n\nAs shown in **Table 1**, including the model response provides no significant, consistent improvement and can even degrade performance (e.g., on Chatbot Arena). We hypothesize this is because the compact embedding model, chosen for efficiency, may struggle to distill useful signals from the longer and more varied model responses. The user's prompt remains the most direct and potent signal of their immediate intent. We will add this table and analysis to the appendix.\n\n#### **2. Could you explain how did you get the 10/sec number?**\n\nThank you for the question. This figure is based on performance benchmarks of modern LLM serving systems. For example, the **DistServe paper (Zhong et al., OSDI '24)** shows in their Figure 1 that for a 13B model on an A100 80GB GPU with an average input length of 512 tokens, the prefilling throughput is less than 10 req/s. Since many production models are significantly larger than 13B in the prefilling phase, this is a reasonable, and often conservative, estimate for workloads with moderate prompt lengths. We will add this citation and clarification to the paper.\n\n#### **3. Could you explain why you need separate models for different datasets? Can you train a single predictor for all settings?**\n\nThis is a crucial question about the practical deployment of LPC. To test the data sensitivity and generalizability of our method, we conducted a key experiment where we trained a single \"unified predictor\" on a combined dataset and compared its performance against the specialized models.\n\n**Table 2:** Unified predictor trained on a combined dataset vs. Specialized models.\n\n| Test Dataset | Model Type | MCC | F1 macro |\n| :--- | :--- | :--- | :--- |\n| Chatbot Arena | Unified | 0.3357 | 0.6678 |\n| Chatbot Arena | Specialized | 0.3485 | 0.6738 |\n| LMSys | Unified | 0.3802 | 0.6871 |\n| LMSys | Specialized | 0.3797 | 0.6897 |\n| ShareGPT | Unified | 0.1271 | 0.5281 |\n| ShareGPT | Specialized | 0.2777 | 0.6348 |\n\nThe results in **Table 2** show that the unified model's performance is competitive with specialized models on the **Chatbot Arena** and **LMSys** datasets. However, on the **ShareGPT** dataset, the unified model's performance is notably lower. This suggests that while core conversational patterns are generalizable, certain datasets like ShareGPT may contain unique characteristics that benefit from domain-specific training. It is also possible that each dataset has unique request patterns that contradicts other datasets, leading to performance drop if it is trained in a mixed way. We hypothesize that a larger model or a full fine tuning of the embedding model can increase the learning capacity and make it have less performance drop on ShareGPT. However, the expensive cost prevent us from adopting the above method. Therefore, we still need a separate model for the best accuracy and overhead trade-off. We will add this important result and discussion to the paper.\n\n***\n\nOnce again, we thank the reviewer for their constructive feedback. We believe that by incorporating these clarifications, the final paper will be substantially stronger. We hope our responses have addressed the reviewer's concerns and have reinforced the value and technical soundness of our contribution."}]

Author rebuttal:

Meta-review: The paper presents Learned Prefix Caching, a neural eviction policy for LLM prefix caches. Unlike LRU, LPC uses a lightweight predictor over recent user prompts to estimate continuation likelihood. Experiments show LPC reduces cache size needs by 18–47% (to keep the same hit ratios) and improves throughput by 11%. This work is practical and clearly addresses a problem in multi-turn LLM serving. However, its technical novelty is limited since the core method is a standard classifier applied in a new setting.

Reviewers agreed the problem is important, the paper is clear, and the evaluation is solid. They noted, however, that performance gains are modest, ablations were initially incomplete, and evaluation was restricted to specific LLMs. The rebuttal addressed most of these points, adding ablations and clarifying design choices, which slightly improved sentiment. Overall, while the contribution is somewhat incremental, the demonstrated efficiency benefits justify acceptance.
Decision: Accept (poster)