Title: Generative Caching for Structurally Similar Prompts and Responses
Venue: NeurIPS 2025
Accepted: yes
Keywords: Generative Caching

Hit Rate Optimization

Repeatable Workflows

LLM Serving Systems
Abstract: Large Language Models (LLMs) are increasingly being used to plan, reason, and execute tasks across diverse scenarios. In use cases like repeatable workflows and agentic settings, prompts are often reused with minor variations while having a similar structure for recurring tasks. This opens up opportunities for caching. However, exact prompt matching fails on such structurally similar prompts, while semantic caching may produce incorrect responses by ignoring critical differences. To address this, we introduce GenCache, a generative cache that produces variation-aware responses for structurally similar prompts. GenCache identifies reusable response patterns across similar prompt structures and synthesizes customized outputs for new requests. We show that GenCache achieves 83\% cache hit rate, while having minimal incorrect hits on datasets without prompt repetition. In agentic workflows, it improves cache hit rate by $\sim$20\% and reduces end-to-end execution latency by $\sim$34\% compared to standard prompt matching.


Peer reviews:
{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper introduces GenCache, a method for leveraging previous LLM responses to handle partially matching prompts in LLM-based web agents that perform repetitive tasks. In a preprocessing stage, GenCache performs clustering of prompts in a dataset and generates a regular expression for detecting and preprocessing prompts in each cluster, and it generates code for common operations for the prompts in each cluster. During inference, it retrieves cached code and adapts it by appropriately substituting variables. Unlike existing caching methods that reuse cached responses without modification, GenCache caches code templates and dynamically adapts them to new inputs, although its current design is tightly coupled to a specific task, which may limit its applicability to broader use cases. Experimental results on the WebShop dataset show a high cache hit rate exceeding 80% and a low negative hit rate below 10%.\n\nSTRENGTHS AND WEAKNESSES:\nThe main limitation of this paper lies in its narrow focus on the web agent task, and I recommend revising the title to better reflect this scope and avoid misleading readers. Nevertheless, the paper makes a meaningful contribution toward addressing the core challenge of leveraging prior LLM responses for similar prompts and proposes a practical and effective solution. Overall, I consider this work to be above the acceptance threshold.\n\n### Strengths\n\n* This paper targets a common and fundamental challenge of leveraging previous LLM responses for similar or partially matching prompts.\n* The proposed method offers a practical and effective approach to leveraging partial matches in LLM caching. Although it is tailored to a specific use case involving web agents performing repetitive tasks, the core idea has the potential to generalize to a broader range of applications.\n\n### Weaknesses\n\n* The method is designed for a specific application\n\nThe title suggests that this paper introduces a generalizable method. However, the work specifically targets LLM-based web agents performing repetitive tasks. I recommend either revising the title to more accurately reflect this narrow scope or extending the method to support a broader range of applications.\n\nThe proposed method relies on regular expressions to detect and preprocess prompts that match cached patterns, which restricts its applicability to tasks with limited prompt diversity. This constraint significantly limits the generalizability of the approach and is the primary reason I did not assign a higher score.\n\nAlthough the current method is reasonable as an initial step, I believe the proposed framework has potential for greater robustness. For example, incorporating a preprocessing of prompts or a post-hoc revision step to modify the code using small LLMs could help generalize the method while still maintaining low computational cost.\n\n* Limited evaluation domains\n\nThe proposed method is evaluated solely on a single dataset, WebShop, which appears to be particularly well-suited to the strengths of the approach. I recommend conducting evaluations on additional datasets that involve more diverse operations in order to better understand the limitations and generalizability of the method.\n\nQUESTIONS:\nPlease let me know if there are any inaccuracies or misunderstandings in the points raised under Weaknesses.\n\nLIMITATIONS:\nYes.", "TITLE": "Official Review", "RECOMMENDATION": 4, "REVIEWER_CONFIDENCE": 4, "CLARITY": 3, "IMPACT": 3, "ORIGINALITY": 3, "DATE": "2025-06-15"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper introduces GenCache, a novel generative caching mechanism designed to address the limitations of existing prompt caching techniques for structurally similar LLM prompts with critical subtle variations. Recognizing that recurring tasks in workflows, chatbots, and agents often reuse similar prompts but require variation-aware responses, GenCache identifies and caches validated generative patterns underlying these responses as executable programs. Upon a cache hit, it executes the stored program to efficiently synthesize a new, contextually appropriate response for the new prompt, rather than returning a potentially incorrect static cached response. The authors demonstrate that GenCache achieves a high cache hit rate (83%) with minimal negative hits on datasets lacking exact prompt repetition and significantly improves performance in agentic workflows.\n\nSTRENGTHS AND WEAKNESSES:\nStrengths:\n1. The first dynamic LLM cache: Previous studies could only retrieve static LLM replies by retrieving similar caches. GenCache generates Python code through LLM to achieve more flexible cache reading. Even when facing inputs that have never appeared before, it can extract key information through Python code and generate replies.\n2. Higher hit rate and efficiency: Experiments on synthetic data and webshop datasets show that GenCache has a higher hit rate and can effectively reduce the overhead of LLM reasoning.\n\nWeakness:\n1. As the author said, Python programs usually implement scalable LLM caches through regular expressions, which means that this method is often only suitable for short dialogue outputs with format requirements. However, regular expression-based methods often have little effect on open dialogues and long text tasks, because such dialogues usually do not have a strict format, and it is difficult to extract information from long texts through regular expressions.\n\nQUESTIONS:\n1. The author mentioned that the overall size of the cache is only tens of Mb, which is very exciting, but I noticed that after the cache is built, the clusters do not disappear, and the clusters contain a large number of dialogue inputs and outputs, as well as vectorized representations of these inputs and outputs. Obviously, the size of this information is much larger than the python files in the cache, and the author should add how much space this information will take up;\n2. Other computational overhead: The author experimentally proves that GenCache can effectively reduce the computational overhead caused by LLM reasoning. However, GenCache needs to calculate the n-dimensional vector representation of all inputs and calculate its similarity with all clusters. The author should take this part of the computational overhead into consideration;\n3. Experimental details: The experimental setting is unclear. The author did not explain how the specific synthetic prompt set in Table 1 was synthesized, what the specific content is, and how the evaluation indicators were calculated. It is recommended to refer to the explanation of the evaluation indicators in GPTCache to further supplement the experimental details.\n\n\nFormat error: 1) The author mentioned \" As shown in Figure 2 with Param-w-Synonym prompts \" in line 304, but it should refer to Table 2; 2) This seems to be a more serious error. The author mentioned \" The prompt used for ValidLLM is included in the supplementary material.\" in line 206, but I did not find any appendix in the article. The author also lacked an explanation of the experimental details and prompt content of the paper. Did the author forget to attach an appendix to the paper?\n\nLIMITATIONS:\nyes", "TITLE": "Official Review", "RECOMMENDATION": 4, "REVIEWER_CONFIDENCE": 4, "CLARITY": 2, "IMPACT": 3, "ORIGINALITY": 3, "DATE": "2025-07-01"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper proposes GenCache, a client-side caching technique for LLMs that clusters prompt–response pairs based on both prompt and response embedding similarities. Once a new prompt’s embedding is close enough to a cluster center, GenCache retrieves the small Python program stored for that cluster and executes it to generate a tailored response. \n\nTo build these programs, GenCache uses a CodeGenLLM with a handful of in-context prompt–response examples to infer the consistent mapping pattern, then validates the generated code by running it on the same examples and asking a ValidLLM to semantically compare each program output to its ground truth. If at least half of the outputs match, the program is accepted and cached; otherwise, it’s retried or discarded. Each cached program takes a prompt as input and produces the correctly formatted response, enabling high hit rates with minimal LLM calls.\n\nSTRENGTHS AND WEAKNESSES:\nstrengths\n\n1. It captures the real-life setting where prompts are structurally similar yet differ in key details; exact-match or vanilla semantic caches fail here, whereas GenCache’s program cache have the potential to returns the correct, variation-aware response.\n\n2. It introduces a “cache-the-program” strategy—store a concise function that transforms the incoming prompt into the answer instead of caching the raw response—keeping the cache lightweight while generalizing across similar prompts.\n\n3. Strong empirical evidence: GenCache achieves a higher hit rate than the baselines and also achieves latency and token savings.\n\nweaknesses\n\n1. Baselines rely on relatively small-context datasets; it is unclear how the regex-driven program will scale to long, multi-paragraph inputs or to irreversible agent actions?\n\n2. CodeGenLLM and ValidLLM both run on GPT-4o/4, which may be costly; demonstrating that a smaller, open model can handle synthesis and validation would strengthen practicality.\n\n3. Accepting a program when only half of the exemplar outputs match (and still called \"error-free\") seems brittle and risks incorrect cache hits in downstream use.\n\nQUESTIONS:\n1. In line 138, the authors write “Since R often contains multiple key-value pairs… Therefore, we create an embedding array [er] by computing n-dimensional embeddings for each value of the key-value pair of R.” But if R is just free-form text (a sentence or paragraph), how should we interpret “key-value pairs”? Are they really embedding each token, or is this only for structured outputs?\n\n2. Do you apply the same multi-embedding approach to prompts, or do prompts always get a single sentencetransformer embedding? What motivated treating prompts and responses differently, especially since prompts can also be quite long?\n\n3. When you claim that “GenCache performs online clustering.” How much CPU/time overhead does this clustering add compared to simply issuing a new LLM call?\n\n4. Will giving more in-context examples (increasing ν) to CodeGenLLM always improve caching performance, or is there a point of diminishing returns?\n\n5. When you say “using the same exemplars from §3.3, we execute the program locally… to obtain program-generated responses,” are those exemplars the ones inside the cluster or the ones provided as in-context examples to CodeGenLLM?\n\n6. You describe validating a program as “error-free,” yet they accept it if only half of the outputs match. Isn’t that just 50% accuracy—why call it error-free?\n\n7. How reliable is ValidLLM’s semantic comparison in practice? Have you done any human evaluation to confirm its judgments?\n\n8. Even after a program is cached, it may still fail on new prompts. I wonder how frequently does that actually happen in their experiments?\n\nLIMITATIONS:\nyes", "TITLE": "Official Review", "RECOMMENDATION": 4, "REVIEWER_CONFIDENCE": 2, "CLARITY": 2, "IMPACT": 2, "ORIGINALITY": 3, "DATE": "2025-07-03"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe authors have introduced GenCache, a caching mechanism designed to reduce the number of LLM calls, say in agentic workflows, by caching structurally similar prompts and symbolic cache programs which generates variation-aware responses on cache hits. The core insight is that in specific usecases, structural similarity in prompts often correlates with predictable variations in their corresponding responses. To exploit this, GenCache uses additional LLM calls to identify consistent transformation patterns between prompts and responses within each cluster and learn a transformation program. These learned programs act as dynamic response generators, improving over ExactCache and GPTCache like methods which fail to capture subtle variations that demand different responses.\n\nSTRENGTHS AND WEAKNESSES:\n**Strengths**\n\n- The paper shows a clear motivation for GenCache by comparing with the limitations of existing Semantic and Exact caching mechanisms. \n- GenCache is a promising direction in better weighing the trade-off between LLM-call cost optimisation and accuracy with caching.\n- The structurally similar assumption is showcased and validated in real-life use cases of agentic workflows., where saving on LLM-call is demonstrated with GenCache.\n- The authors have also showcased the limitations of GenCache with experiments when diverse structures start to appear in prompts during caching and the mechanism experiences lower cache hits, laying out the directions for future research and improvements.\n\n\n**Weaknesses**\n- Missing ablation with respect to similarity threshold, minimum prompts per cluster used to create clusters and corresponding cached programs. For instance, experiments to show how quickly each cached program converges and what happens when clusters contain outliers are missing. \n\n- No sufficient reasoning/evidence to support design decisions during program validation. \n    - What is the reasoning behind choosing 50% as the threshold on matched program-generated responses with exemplars? What happens when mismatched cases are mostly due to outliers?\n    - Alternately, what does the mismatch between program generated responses and corresponding exemplars during program validation actually signify? Does it mean that the learned program is less generalisable or there are outliers present? Consequently, how do you make sure that while retrying program generation with reflection, you are not overfitting to outliers and generalising well?\n\nQUESTIONS:\n- See Weaknesses Section\n- How are you selecting the exemplars from the cluster to prompt the CodeGenLLM for generating the cache program? \n- While the authors acknowledge that the use case of GenCache is limited to agentic workflows where structural similarities often occur between prompts and response, it is unclear how the proposed caching would perform in agentic use-cases with dynamic workloads where the responses while being structurally similar are data dependent. Imagine a similar use case from Figure 1 but now the plan to look up AAA batteries/USB-C cables on Amazon needs to incorporate specifying min-max cost limit which is user-dependent and must be determined via an external database of user’s expenditure budgets. Can your program learning incorporate such database retrieval or tool call to collect external information?\n\nLIMITATIONS:\nYes", "TITLE": "Official Review", "RECOMMENDATION": 5, "REVIEWER_CONFIDENCE": 4, "CLARITY": 3, "IMPACT": 4, "ORIGINALITY": 4, "DATE": "2025-07-03"}