Title: Learned Prefix Caching for Efficient LLM Inference
Venue: NeurIPS 2025
Accepted: yes
Keywords: large language model inference

prefilling

prefix caching

learned caching algorithm
Abstract: Prefix caching is a key technique for reducing Large Language Model (LLM) inference costs. However, the prevalent least-recently-used (LRU) eviction algorithm has a large gap to the optimal algorithm. This paper introduces LPC, the first learned method to perform LLM prefix cache eviction. LPC leverages conversational content analysis to provide predictive guidance for eviction, determining which conversations are likely to continue. These insights, combined with last access timestamps, inform more effective cache management. Extensive evaluations across three real-world datasets demonstrate that LPC achieves 18-47% reductions in required cache sizes for equivalent hit ratios and has an 11% improvement in LLM prefilling throughput in an emulated environment.


Peer reviews:
{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe paper introduces LPC (Learned Prefix Caching), the first learned eviction algorithm for prefix caches in Large Language Model (LLM) inference systems. Traditional prefix caching uses an LRU (Least Recently Used) policy, which underperforms due to its inability to anticipate future usage based on semantic context. LPC overcomes this by predicting the likelihood that a conversation will continue using a lightweight neural model that analyzes recent user prompts. These predictions guide the eviction policy to retain prefixes more likely to be reused, improving cache hit rates. Evaluated on real-world datasets (LMSys, ShareGPT, Chatbot-Arena), LPC achieves 18–47% reductions in required cache size for the same hit ratio compared to LRU, and up to 11% higher prefilling throughput in an emulated environment.\n\nSTRENGTHS AND WEAKNESSES:\n[Strengths]\n1. The paper proposes a learned eviction strategy tailored to prefix caching in LLM serving – LPC, identifying an important issue in efficient LLM serving while alleviating the limitations of existing methods like LRU.\n2. The paper presents adequate analysis on the performance improvements and overheads introduced by the proposed method, especially with the experiments in the appendix.\n3. The presentation and visual components of this paper are clear and easy to understand. The authors also include detailed training setups in this paper.\n\n[Weaknesses]\n1. Ablations on several important designs are missing. See questions section for more details.\n2. The cache hit ratio improvement over the LRU baseline seems to be not very significant, and the overall throughput improvement is a bit marginal.\n3. Experiments are limited to specific settings such as QWen-32B. Experiments on different model series and different model sizes may better prove the effectiveness of the design.\n\nQUESTIONS:\n1. In Line 129, the authors say that “we do not include model responses because they contain less new information than user prompts”. Could you further explain why the model response will not help judge the probability of a follow-up question? For example, if the model’s response is not complete enough, isn’t that the probability that the user will have a follow-up can be higher? Do you have any analysis or ablation on this?\n\n2. In Line 151, the authors say that “the request rate is typically lower than 10 per second per GPU”. Could you explain how did you get the 10/sec number?\n\n3. In Line 160, the authors say that they train “a separate model for each dataset”. Could you explain why do you need separate models for different datasets? Can you train a single predictor for all settings?\n\nLIMITATIONS:\nyes", "TITLE": "Official Review", "RECOMMENDATION": 4, "REVIEWER_CONFIDENCE": 4, "CLARITY": 3, "IMPACT": 3, "ORIGINALITY": 2, "DATE": "2025-06-09"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe paper introduces Learned Prefix Caching (LPC), a new method for managing the cache used during Large Language Model (LLM) inference. The paper notes that the commonly used least-recently-used (LRU) algorithm is not ideal for this purpose, as it does not account for the conversational context. LPC uses a learned predictor to analyze conversational content and determine which conversations are likely to continue, which in turn helps decide which data to keep in the cache. The evaluations show that LPC consistently outperforms LRU, leading to a significant reduction in the required cache size for the same performance and an improvement in prefilling throughput.\n\nSTRENGTHS AND WEAKNESSES:\n### Strengths\n\n1. This paper studies a meaningful and practical problem in LLM serving systems, specifically the inefficiency of the LRU algorithm for prefix caching. As the inference demand for LLMs grows, optimizing cache management is crucial for reducing latency and compute costs.\n\n2. Extensive evaluations are conducted across three real-world datasets (although the timestamps are synthetic), demonstrating the effectiveness of the proposed method. The results show that LPC can reduce the required cache size by 18-47% for the same hit ratio and improve LLM prefilling throughput by 11%. The paper also assesses the minimal overhead of the LPC predictor, showing that it is a practical solution.\n\n3. The paper is generally well-written and easy to follow.\n\n### Weaknesses\n\n1. The LPC predictor introduces additional complexity and overhead to the LM serving system. While the paper claims to observe substantial performance improvements on several datasets, there always exists a worst-case scenario where no shared prefix is found and all the computational overhead of the predictor is wasted. The LRU strategy, while sub-optimal, is much simpler and does not require any training.\n\n2. Although the problem setting is well-motivated and novel, the proposed solution itself is not particularly innovative. The idea of training a classifier over pre-trained embeddings has been explored in various contexts. The novelty of this work lies more in its application to LLM prefix caching rather than in the underlying methodology.\n\nQUESTIONS:\n1. Besides the evaluation on cache hit ratio and throughput, conducting an evaluation of the classifier accuracy would provide a more comprehensive understanding of the predictor's performance. I am also curious why you only train the MLP part of the predictor? Full fine-tuning would likely yield better results without affecting the deployment overhead.\n\n2. A growing workload of LLM serving systems is to handle multi-modal inputs, such as images and videos. How does the proposed method handle multi-modal inputs? Is it possible to extend the LPC predictor to work with multi-modal data?\n\nLIMITATIONS:\nyes", "TITLE": "Official Review", "RECOMMENDATION": 4, "REVIEWER_CONFIDENCE": 3, "CLARITY": 3, "IMPACT": 2, "ORIGINALITY": 2, "DATE": "2025-06-28"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper introduces a learned prefix cache framework that analyzes conversation content and evicts outdated cache information smartly. Different from LRU that evicts the least recent cache information, the method utilizes a lightweight ML predictor to analyze the historical user prompts to estimate the possibility of the conversition continuing. The framework then leverages the possibiltiy to guide eviction decisions in the prefix cache.\n\nSTRENGTHS AND WEAKNESSES:\nStrengths: \n\n1. The method adapts to online scenarios and can handle high query volumes, making it practical for production LLM serving systems where low latency and high throughput are critical.\n\n2. The authors introduce a novel eviction algorithm that analyzes the historical prompts to predict conversation continuation.\n\n3. The method achieves comparable hit ratios compared to LRU but reduces the cache size by 18–47%.\n\nWeakness:\n\n\n1. The method relies on the last 4 user prompts for prediction, which might fail to capture the complex interactions between users and the chatbot. Users might formulate new questions based on the LLM's responses rather than just their previous prompts. Thus, it might be biased to only use the 4 latest prompts to estimate if the chat is finished or not.\n\n2. The 4 prompt window might miss long-term conversational patterns. In a case when users seek thorough explanations from a chatbot, usually it is difficult to determine if the users' questions are fully addressed with 4 questions.\n\n\n3. The approach trains separate predictor models for each dataset (Line 161), which poses practical challenges in real-world LLM serving. Real-world LLM serving systems handle highly diverse queries across users and topics, making dataset-specific training almost impractical for production deployment. Is your proposed method data-sensitive? Is it possible to handle a more general case?\n\nQUESTIONS:\n1. How do you determine the value of scale? \n\n2. Can you provide more explanations about the probability decay function?\n\n3. Can you compare LPC with LRU on throughput?\n\nLIMITATIONS:\nYes", "TITLE": "Official Review", "RECOMMENDATION": 4, "REVIEWER_CONFIDENCE": 3, "CLARITY": 2, "IMPACT": 3, "ORIGINALITY": 3, "DATE": "2025-07-03"}

comments: SUMMARY:
This work propose to replace the LRU cache eviction in LLM serving by a learnable component, to improve the kv cache hit ratio for better efficiency. Experiments show that the proposed Learned Prefix Caching can achieve 18-47% reductions in required cache sizes for equivalent hit ratios and has an 11% improvement in LLM prefiling throughput in an emulated environment.

STRENGTHS AND WEAKNESSES:
Strength:
1) The paper is clear and well written.
2) As the improvement of LLMs, the serving cost is growing fast due to more usage across the world. It is very important to improve the model serving efficiency. In addition to that, the GPU memory is growing slowly but the model size is growing fast. Using memory smarter is good.


Weakness:
1) Since there is an additional round of training before serving LLM, there is risk that model is easier to overfit and the improvement might be way smaller on OOD data.
2) As the community is switching to scale RL and test-time compute (more frontier models like Gemini and O3 are using thinking mode by default), more serving cost is from decoding stage instead of prefilling.
3) It would be good to have a finer grained results like the hit ratio on the prompts from different domains (e.g. agent, coding, chatbot...)

QUESTIONS:
NA

LIMITATIONS:
NA
TITLE: Official Review
CLARITY: 3
IMPACT: 3
ORIGINALITY: 3
DATE: 2025-07-12