Title: Agentic Plan Caching: Test-Time Memory for Fast and Cost-Efficient LLM Agents
Venue: NeurIPS 2025
Accepted: yes
Keywords: Caching

Memory

Serving

LLM Agents
Abstract: LLM-based agent applications have shown increasingly remarkable capabilities in complex workflows but incur substantial costs and latency due to extensive planning and reasoning requirements. 
Existing LLM caching techniques (like context caching and semantic caching), primarily designed for serving chatbots, are insufficient for agent applications where outputs depend on external data and environmental contexts. 
We propose **Agentic Plan Caching (APC)**, a novel **test-time memory** that extracts, stores, adapts, and reuses structured plan templates from planning stages of agent applications across semantically similar tasks to reduce the cost and latency of serving. 
Unlike traditional semantic caching, our system extracts plan templates from completed agent executions at test-time, employs keyword extraction to match new requests against cached plans, and utilizes lightweight models to adapt these templates to task-specific plans with contexts. 
Evaluation across multiple real-world agent applications shows that our system can reduce costs by 50.31\% and latency by 27.28\% on average while maintaining performance, offering a more efficient solution for serving LLM-based agents that complements existing LLM serving infrastructures.


Peer reviews:
{"IS_META_REVIEW": false, "comments": "SUMMARY:\nCurrent LLM caching methods (e.g., context or semantic caching) fall short for agentic tasks due to dependencies on external data and dynamic environments. To address this, the authors introduce *agentic plan caching*, a novel technique that captures, stores, and reuses structured plan templates from past executions. By leveraging keyword-based matching and lightweight adaptation models, the system efficiently tailors cached plans to new tasks with similar semantics. Experiments across real-world applications demonstrate a 46.62% average cost reduction while maintaining performance, providing a scalable and cost-effective enhancement for LLM agent deployment. This approach complements existing LLM serving systems, bridging the gap for agent-specific optimization.\n\nSTRENGTHS AND WEAKNESSES:\n**Strengths:**\n\n- The paper is well-structured, with rigorous experiments and clear methodology. The evaluation on two datasets (FinanceBench and TabMWP) provides strong empirical support for the claims.\n- The writing is clear, and the figures effectively illustrate the framework and results. The distinction between traditional caching and agentic plan caching is well-explained.\n- The idea of caching plan templates for agentic workflows is novel and interesting, which can fills a gap left by existing caching techniques, offering practical cost savings without significant performance degradation.\n\n**Weaknesses:**\n\n- The evaluation is limited to two datasets and a specific agent architecture (planner-actor). Broader applicability to more complex multi-agent systems or diverse workflows remains unexplored.\n- While cost reduction is the focus, other system metrics like latency and throughput are not discussed, which are important for real-world deployment.\n\nQUESTIONS:\n1. In line 195-199, The Figure 4a does not show the Plan-Act Framework. Maybe it should be Figure 1a?\n2. Could you include evaluation results from additional benchmarks?\n\nLIMITATIONS:\nyes", "TITLE": "Official Review", "RECOMMENDATION": 4, "REVIEWER_CONFIDENCE": 3, "CLARITY": 3, "IMPACT": 4, "ORIGINALITY": 3, "DATE": "2025-06-30"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe paper proposes agentic plan caching, a serving-time technique aimed at cutting the cost of Plan-Act LLM agents. After each completed run, the system extracts a plan template from the planner’s execution trace, stores it under a task-intent keyword, and later retrieves and lightly adapts that template—using a small model—when a semantically similar request arrives. Experiments on two data-intensive reasoning benchmarks (FinanceBench and Tabular-MWP) show a mean 46 % cost reduction while retaining 96 % of the accuracy of an all-large-LM baseline, with only ~1 % overhead for keyword extraction and cache management.\n\nSTRENGTHS AND WEAKNESSES:\nStrength:\n\n1. The paper addresses the rising serving cost of multi-step LLM agents, a practical bottleneck that is less studied than model-side optimization. The distinction between query-level (chatbot) caching and task-level (agent) caching is well argued.\n\n2. The framework rests on off-the-shelf components and could be grafted onto other Plan-Act agents. \n\nWeakness:\n\n1. Scalability and cache management are not discussed. How does cost-saving scale with cache size? Is there a point of diminishing returns?\n\n2. Cold-start and domain drift is not discussed. What happens when the cache is empty or when tasks drift over time? Could you report accuracy and cost as a function of cache warm-up and domain shift would clarify the real-world usefulness.\n\n3. While cost is reduced, the extra keyword extraction, lookup, and template adaptation may add latency. Could the paper report end-to-end wall-clock times? In addition, storing plan templates might leak sensitive workflow information\n\n4. Generality is not discussed. The solution is demonstrated with a specific large planner and an 8 B re-planner. How well does the approach transfer if those models are swapped for smaller or fundamentally different LLM families?\n\nQUESTIONS:\n1. Exact keyword matching avoids false positives but may miss near-duplicates. Have the authors explored lightweight embedding search or fuzzy matching, and how would that affect hit-rate, cost, and accuracy?\n\n2. The paper focuses on token cost; could end-to-end latency numbers be reported (cache lookup + rewrite)?\n\nLIMITATIONS:\nSee weakness.", "TITLE": "Official Review", "RECOMMENDATION": 4, "REVIEWER_CONFIDENCE": 4, "CLARITY": 3, "IMPACT": 2, "ORIGINALITY": 3, "DATE": "2025-07-01"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper proposes an agentic plan caching algorithm that reduces costs by nearly 50% and nearly maintains the original optimal accuracy. Previous caching methods are limited because they (1) have model-specific constraints; agentic flows typically use various models while caching methods are for specific models (2) reuse cached responses while agentic flows have data-dependent outputs (3) are limited in adaptability sometimes requiring exact text matches. The agentic plan caching framework, intended for use in Plan-Act frameworks that utilize a large LM for high level planning and a small LM to carry out the plan, generates a plan template on cache misses to be followed by a small LM during cache hits. Evaluation is conducted on methods optimal for accuracy and cost and semantic caching and agentic plan caching is shown to reach the best tradeoff between cost and accuracy.\n\nSTRENGTHS AND WEAKNESSES:\n**Quality:** The baselines and experiments effectively demonstrate the usefulness of agentic plan caching.\n\n**Clarity:** The paper is written clearly, with the shortcomings of previous caching approaches clearly demonstrated.\n\n**Significance:** The cost-effectiveness of this caching strategy is very significant but is limited to two-stage Plan-Act agent architectures whereas other caching strategies can be applied agnostic to architectures.\n\n**Originality:** This caching approach is a unique addition to previous approaches like KV caching and semantic caching.\n\nQUESTIONS:\n- On the discussion between exact matching versus fuzzy matching, an argument is made that fuzzy matching could make cache lookups slower and costly. An empirical evaluation of this claim would make this point stronger.\n  - Additionally, there is no study of the speed of the agentic plan caching algorithm. Surely given a plan template is generated this would be much slower than other approaches.\n- What is the major bottleneck in keeping the costs low but the accuracy high? The agentic plan caching is nearly close to optimal in some cases (Table 1 TabMWP) which is impressive but what would close this gap further?\n  - I am interested in more information regarding the rule-based filter and specific failure modes of the plan template that lead to unexpected behavior and the lower performance\n\nLIMITATIONS:\nThe authors have clearly addressed the limitations of this work.", "TITLE": "Official Review", "RECOMMENDATION": 5, "REVIEWER_CONFIDENCE": 4, "CLARITY": 4, "IMPACT": 2, "ORIGINALITY": 3, "DATE": "2025-07-03"}

comments: SUMMARY:
This paper proposes a cost-efficient serving strategy for LLM-based agents through plan caching. Instead of relying on conventional context or semantic caching, the authors introduce a method that caches structured plan templates extracted from past executions. New tasks are matched to cached plans using keyword-based similarity, and a smaller LLM is used to adapt these plans to the current context. Experiments on two agentic benchmarks demonstrate that this approach reduces serving costs significantly (by ~46%) while maintaining high task performance.

STRENGTHS AND WEAKNESSES:
## Strengths:
- The paper is overall well-written and includes clear, helpful figures.

- The analysis is thorough, especially in justifying design choices such as using keyword-based cache search instead of query-based methods, and showing the benefit of plan caching over semantic or full-history caching.

## Weaknesses:

- The TabMWP dataset feels a bit too simple. It would make the paper more impactful to evaluate on more complex math datasets like AIME.

- There is a lack of analysis on how the caching method performs as task diversity increases. For example, how would it perform on a benchmark with very diverse tasks, such as GAIA? Would it still outperform other caching strategies?

QUESTIONS:
Please see Weaknesses above.

LIMITATIONS:
Yes
TITLE: Official Review
CLARITY: 3
IMPACT: 3
ORIGINALITY: 3
DATE: 2025-07-03

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis work proposes a new technique to do plan caching. Previous works have done caching to save KV parameters of the LLM, or the whole task including all parameters and keywords. The novelty of this approach is that the abstract plan or “plan template” is cached. The keywords are abstracted or removed before doing the caching. This makes the plan applicable for a wide range of parameters, and not just the single task it was originally meant for.  \n\nThe advantage of the approach is that it saves the cost of using LLMs for plan generation every time, many of which are usually unnecessary. Experimental comparison with other caching techniques show that the cost is substantially reduced in two domains with little loss of plan quality.\n\nSTRENGTHS AND WEAKNESSES:\nThe paper is generally well written and easy to follow. The paper is also well motivated. The existing caching techniques in LLMs are explained well with figures.\n\nThe proposed method is relatively straighforward and makes sense. The concept of saving the abstract plans is not new and has been explored extensively in the case based reasoning and case based planning literature.\n\nBorrajo, Daniel, Anna Roubíčková, and Ivan Serina. \"Progress in case-based planning.\" ACM Computing Surveys (CSUR) 47, no. 2 (2015): 1-39.\n\nSpalzzi, Luca. \"A survey on case-based planning.\" *Artificial Intelligence Review* 16 (2001): 3-36.\n\nBergmann, Ralph, and Wolfgang Wilke. \"On the role of abstraction in case-based reasoning.\" In *European Workshop on Advances in Case-Based Reasoning*, pp. 28-43. Berlin, Heidelberg: Springer Berlin Heidelberg, 1996.\n\nGiven the simplicity of the technique, I was expecting a deeper dive into the issues of abstraction and I was also expecting a more elaborate experimental evaluation. \n\nWhen there are multiple possible items in the cache matching the new task, how are they ranked? Have you encountered any such cases?\n\nA planner usually takes in the current context as a part of the initial state. So, why is there no link from the context to the Planner LM in the Plan-Act flow chart?\n\nThe main weakness I see is the lack of experimental domains and analysis.\n\nQUESTIONS:\n1. When there are multiple possible items in the cache matching the new task, how are they ranked? Have you encountered any such cases?\n2. In Figure 1, why is there no link from the context to the planner?\n3. In 2.2 (1) Model Specific constraints, why would one want to use a different LLM agent for a task of one agent?\n\nLIMITATIONS:\nYes", "TITLE": "Official Review", "RECOMMENDATION": 4, "REVIEWER_CONFIDENCE": 4, "CLARITY": 3, "IMPACT": 2, "ORIGINALITY": 2, "DATE": "2025-07-05"}