Title: Sim-LLM: Optimizing LLM Inference at the Edge through Inter-Task KV Reuse
Venue: NeurIPS 2025
Accepted: yes
Keywords: LLMs

KV cache

Task Similarity

Edge Computing
Abstract: KV cache technology, by storing key-value pairs, helps reduce the computational overhead incurred by *large language models* (LLMs). It facilitates their deployment on resource-constrained edge computing nodes like edge servers. However, as the complexity and size of tasks increase, KV cache usage leads to substantial GPU memory consumption. Existing research has focused on mitigating KV cache memory usage through sequence length reduction, task-specific compression, and dynamic eviction policies. However, these methods are computationally expensive for resource-constrained edge computing nodes. To tackle this challenge, this paper presents Sim-LLM, a novel inference optimization mechanism that leverages task similarity to reduce KV cache memory consumption for LLMs. By caching KVs from processed tasks and reusing them for subsequent similar tasks during inference, Sim-LLM significantly reduces memory consumption while boosting system throughput and increasing maximum batch size, all with minimal accuracy degradation. Evaluated on both A40 and A100 GPUs, Sim-LLM achieves a system throughput improvement of up to 39.40\% and a memory reduction of up to 34.65%, compared to state-of-the-art approaches. Our source code is available at https://github.com/CGCL-codes/SimLLM.


Peer reviews:
{"IS_META_REVIEW": false, "comments": "SUMMARY:\nSim-LLM is a novel framework exploiting the similarities across LLM inference requests to optimize the reuse of KV-cache.\nThe paper first identifies the similarity in requests across different inference tasks.\nSuch similarity allows the possibility of reusing previously generated KV-Cache.\nThe work also identifies the tradeoff between generation quality v.s reuse opportunity.\n\nSTRENGTHS AND WEAKNESSES:\nStrength\n\nWhile KV reuse is a well-explored field, this work extends the lossless reuse in previous works to the lossy reuse in this work. This enables a new tradeoff between generation quality and KV reuse opportunity.\n\nVery clear motivation with quantified similarity across different tasks.\n\nSignificant performance benefits.\n\n\nWeakness\n\nThe accuracy loss seems a little high if we want more performance benefits. So is there anyway to achieve better tradeoff between accuracy and performance?\n\nQUESTIONS:\nThis is a great work and I can also see great potential of this work in large-scale LLM infrastructure.\n\n1. While this work targets edge devices, large LLM services also face a similar problem (each LLM customer often issues many requests on similar topics). Is there anything that needs special handling if we want to apply SimLLM to large LLM clouds?\n\n2. How will the performance-accuracy tradeoff in Figure 11 look like in larger models? I believe this work should not be limited to small models and edge devices.\n\n3. Is it possible to further optimize the performance by combining SimLLM with works that exploit the KV-cache sparsity (e.g., the baseline designs you compared with), so that SimLLM reuses the KV cache and maintains only the KV-cache of important tokens. This should generate a comprehensive tradeoff space between throughput performance and accuracy, where users can pick different tradeoff points in different scenarios.\n\nLIMITATIONS:\nSee questions, can this work apply to 1) large-scale LLM infras, 2) larger LLMs, or 3) other lossy KV-cache optimization techniques?", "TITLE": "Official Review", "RECOMMENDATION": 5, "REVIEWER_CONFIDENCE": 4, "CLARITY": 4, "IMPACT": 4, "ORIGINALITY": 3, "DATE": "2025-06-17"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper presents Sim-LLM, a novel approach for optimizing LLM inference on edge computing nodes by leveraging task similarity to reduce KV cache memory consumption. The key insight is that edge computing systems often process similar tasks due to geographical and temporal locality. The system caches KV pairs from previously processed tasks and reuses them for subsequent similar tasks, using cosine similarity and locality-sensitive hashing (LSH) for efficient similarity detection. The approach supports both single-node and multi-node scenarios with cross-server KV sharing.\n\nSTRENGTHS AND WEAKNESSES:\nStrengths\n1. The observation on the edge computing systems is well-founded and empirically validated across multiple datasets.\n2. The problem that this paper addressed is important.\n3. It is the first work that reduces the KV cache memory consumption by considering the task similarities.\n4. Extensive experiments to support the idea.\n\nWeaknesses\n1. The absence of experiments on genuine edge hardware limits the study’s relevance to practical edge-computing deployments.\n2. The core insight about task similarity in edge computing is straightforward.\n3. Algorithmic details (hash size, bucket eviction, sandwich cache policy) are scattered in text instead of one clear algorithm box.\n4. For the multi-node setting, prototypes are exchanged frequently. The paper does not specify eviction, aging, or memory-bounded maintenance policies, raising questions about scalability over long-running edge workloads.\n\nQUESTIONS:\n1. How does the approach scale when the number of cached tasks becomes very large? What are the computational and memory limits?\n2. Can the system adapt to changing task distributions over time, or does it require periodic retraining of similarity models?\n3. Are there any security implications of sharing KV caches across servers, particularly in edge computing contexts?\n4. Does temperature sampling—where later tokens drift semantically—break the assumption that early-task KV is reusable?\n5. What is the memory footprint of the global prototype table over a long time of edge traffic, and how is it pruned?\n\nLIMITATIONS:\n1. Only consider a coarse granularity of the presentations among layers; layer importance also depends on the input prompt.\n2. All servers must host identical weights because KV shapes must match.\n3. If an edge node holding popular KV crashes, the benefit disappears until caches warm up again.\n4. Performance is highly dependent on actual task similarity patterns, which may vary unpredictably.\n5. The title is for optimizing LLM inference on the edge, but no tests for a single edge device.", "TITLE": "Official Review", "RECOMMENDATION": 5, "REVIEWER_CONFIDENCE": 3, "CLARITY": 3, "IMPACT": 2, "ORIGINALITY": 2, "DATE": "2025-06-30"}

comments: SUMMARY:
The paper introduces Sim-LLM, a novel method to optimize Large Language Model (LLM) inference in edge computing systems by reusing Key-Value (KV) caches from similar tasks. Sim-LLM caches top-layer KVs of processed tasks and reuses them for subsequent similar tasks, reducing redundant computations. The system employs Locality-Sensitive Hashing (LSH) for efficient task similarity identification and a KV Manager to handle cache storage and sharing across edge servers.

STRENGTHS AND WEAKNESSES:
Strengths:
1) The paper reduces KV cache memory overhead by introducing task similarity-aware reuse, which is a direction underexplored in prior work.
2) The paper is well-motivated and the observations are empirically compelling.
3) The method demonstrates up to 34.65% memory reduction and 39.40% throughput improvement on A40/A100 GPUs.

Weaknesses:
1) The multi-node KV sharing mechanism assumes low-latency inter-server communication, which may not hold in real-world edge networks.
2) While the cosine similarity threshold is justified empirically, it lacks a theoretical or intuitive explanation for why this value generalizes across diverse edge workloads.

QUESTIONS:
See weaknesses.

LIMITATIONS:
See weaknesses.
TITLE: Official Review
CLARITY: 3
IMPACT: 3
ORIGINALITY: 3
DATE: 2025-07-01

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper identifies two novel observations: task similarity and KV cache similarity among LLM tasks at the edge. The authors identify similar tasks using cosine similarity, Locality-Sensitive Hashing (LSH) or prototype learning within and across edge servers, and reuses the top-layer KV cache from previous tasks to accelerate the inference.\n\nSTRENGTHS AND WEAKNESSES:\n**Strength**\n* This paper proposes solid observations and corresponding challenges about task and KV similarity, and supports them with concrete visualizations and quantitative results. \n* The authors clearly illustrate their methods to address the challenges: using adaptive LSH for similarity detection, partial KV caching for unmatched tasks, and prototype-based routing for cross-node reuse.\n* The method targets edge deployment scenarios with constrained memory, a highly relevant use case for real-world LLM serving.\n\n**Weakness**\n* While the overall idea of leveraging task similarity for KV cache reuse is novel, the core methodology is relatively simple and straightforward. The mechanism relies on matching semantic similarity via cosine similarity or LSH and reusing top-layer KV, which lacks algorithmic innovation and theoretical depth.\n* For Challenge 2, the paper proposes skipping the KV computation of intermediate layers by caching only the top and bottom layer KV. However, the justification for this caching scheme is insufficient. The authors assume that skipping the middle layers preserves task semantics and output quality, but there is a lack of detailed theoretical analysis to support the validity.\n* The empirical evaluation lacks fine-grained error analysis. There is no quantitative analysis of the success rate of KV reuse, i.e., how often tasks are matched and accelerated in real scenarios. While average accuracy and PPL are reported, the paper does not discuss how incorrect KV reuse (e.g., misclassified similar tasks) affects output quality, which could be critical in edge deployment scenarios.\n\nQUESTIONS:\n- When fewer tasks in a batch have similar cached tasks, what is the overhead of postponing them? Have you evaluated the worst-case latency in such situations?\n- You choose 0.8 as the cosine similarity threshold. How sensitive are the results to this threshold in real-time inference scenarios? For instance, how does throughput or perplexity change with slightly higher or lower thresholds under typical workloads?\n- What eviction policy is used for managing the KV cache (e.g., FIFO, LRU)? How does this policy affect performance under bursty or highly variable load, where task arrival patterns may not be uniform?\n- In Figure 6, what type of network is used across nodes? Have you measured the communication overhead introduced by task prototype exchange and KV sharing? Could this overhead become a bottleneck in large-scale deployments?\n- ⠀[Typo] Line 148: “such as MQA and GQA used in flash-attention” is inaccurate. MQA and GQA are architectural optimizations, while FlashAttention is dataflow optimization for the attention module. FlashAttention dataflow can be applied to models using MHA, MLA, MQA and GQA, but it is incorrect to say that MQA and GQA are “used” in FlashAttention.\n\nLIMITATIONS:\nYes", "TITLE": "Official Review", "RECOMMENDATION": 3, "REVIEWER_CONFIDENCE": 4, "CLARITY": 3, "IMPACT": 2, "ORIGINALITY": 3, "DATE": "2025-07-12"}