Sim-LLM: Optimizing LLM Inference at the Edge through Inter-Task KV Reuse
Authors: Ruikun Luo
Changwei Gu
Qiang He
Feifei Chen
Song Wu
Hai Jin
Yun Yang
Venue: NeurIPS 2025 poster ()
Keywords: LLMs
KV cache
Task Similarity
Edge Computing
Primary area: deep_learning

Abstract: KV cache technology, by storing key-value pairs, helps reduce the computational overhead incurred by *large language models* (LLMs). It facilitates their deployment on resource-constrained edge computing nodes like edge servers. However, as the complexity and size of tasks increase, KV cache usage leads to substantial GPU memory consumption. Existing research has focused on mitigating KV cache memory usage through sequence length reduction, task-specific compression, and dynamic eviction policies. However, these methods are computationally expensive for resource-constrained edge computing nodes. To tackle this challenge, this paper presents Sim-LLM, a novel inference optimization mechanism that leverages task similarity to reduce KV cache memory consumption for LLMs. By caching KVs from processed tasks and reusing them for subsequent similar tasks during inference, Sim-LLM significantly reduces memory consumption while boosting system throughput and increasing maximum batch size, all with minimal accuracy degradation. Evaluated on both A40 and A100 GPUs, Sim-LLM achieves a system throughput improvement of up to 39.40\% and a memory reduction of up to 34.65%, compared to state-of-the-art approaches. Our source code is available at https://github.com/CGCL-codes/SimLLM.

Sim-LLM: Optimizing LLM Inference at the Edge through Inter-Task KV Reuse
Ruíkun Luo 1234 Changwei Gu 1234 Qiang He 1234, Feifei Chen",
Song 1u12 Hai Jin 1234, Yun Yang"
National Engineering Research Center for Big Data Technology and System
Services Computing Technology and System Lab CCuster and Grid Computing Lab
'School of Computer Science and Technology, Huazhong University of Science and Technology Deakin University "Swinburne University of Technology {rkluo, gumorming, heqiang, wusong, hjin) edu. cn, fefei. chwnddeakin edu..edu.ar au, yyangeswin. edu.an au
Abstract
KV cache technology, by storing key-value pairs, helps reduce the computational overhead incurred by large language models (LLMs). It facilitates their deployment conuurceeconttrineed edge computing nodes like edge servers. However, the complexity and size of tasks increase, KV cache usage leads to substantial GPU memory consumption. Existing research has focused on mitigating KV cache memory usage through sequence length reduction, task-specific compression, and dynamic eviction policies. However, these methods are computationally expensive for resource-constrained edge computing nodes. To tackle this challenge, this paper presents Sim-LLM, a novel inference optimization mechanism that leverages task similarity to reduce KV cache memory consumption for LLMs. By caching KVs from processed tasks and reusing them for subsequent similar tasks during inference, Sim-LLM significantly reduces memory consumption while boosting system throughput and increasing maximum batch size, all with minimal accuracy degradation. Evaluated on both A40 and A100 GPUs, Sim-LLM achieves a system throughput improvement of up to 39.40% and memory reduction of up to 34.65% compared to state-of-the-art approaches. Our source code is available at https::/gitaal com cwm/CGCL codes Pondes/SSimLLN
1 Introduction
The deployment of edge applications, such as autonomous vehicles and smart traffic management, has generated massive real-time data from numerous edge devices 2 In recent years, edge intelligence has further highlighted the advantages of edge computing in privacy preserving and latency reduction. Large language models (LLMs) can be deployed on edge servers close to users improve their experiences [3]. However, as the number of tasks increases, the memory overhead required for processing these tasks increases rapidly, making it difficult process on time. This problem becomes even more pronounced on edge servers, where resources are inherently constrained 2 4)
The memory consumption of LLMs during inference primarily arises from the model parameters and
the key-value (KV) cache S 6. KV cache a common technique for accelerating LLM inference. The core idea is reuse in subsequent token generations. Since the size of KV cache grows linearly with the sequence length, batch size, and the number of model layers, it consumes lot of GPU memory
previously computed key and value vectors from the attention mechanism for
7 g
"Corresponding author.
39th Conference on Neural Information Processing Systems (NcurIPS 2025).

---

A substantial body of research has attempted to mitigate the memory consumption of KV cache in LLMs, with most efforts aimed at reducing the sequence length stored in the cache to minimize its overall size. For example, ML.Mlin [9] employs prompt compression techniques to reduce KV cache memory consumption. Similarly, Lm-infinite 100 introduces a method for compressing specific token spans into more compact representations to save memory. Additionally, the approach proposed in []] employs a dynamic KV cache eviction policy to selectively retain only a small portion of the KV cache in memory. However, these methods were designed based on layer-wise or token-wise perspective. When the number of tasks is large, their ability to reduce memory consumption remains limited, and they also incur significant computational overhead.
Observation 1. There exists a broad similarity among LLM tasks in edge computing systems.
This paper presents novel approach named Sim- LLM to reduce the memory consumption of KV cache of edge nodes from a task-wise perspective. Its core idea is to reuse KV cache based on the sim- ilarity between previous and current inference tasks. Specifically, we have observed a considerable simi- larity among LLM tasks in edge computing systems through extensive experiments. Figure | shows the similarity proportions within and across seven widely used datasets for LLM inference in edge comput- ing systems. Among them, REDDIT [12] contains the comments of 50 high-quality subreddits from the REDDIT PushShift data dumps (from 2006 to 2023), and we use the data collected after 2020 to ensure the up-to-dateness. MMChat [33 and LCCC [14) are conversation dialogues collected from Weibo, PTT, and Douban. As shown in Figure I there is a broad similarity among LLM tasks in edge computing systems (Observation 1). The reason is edge servers' limited service coverage and geographical distribution [2 [5] The tasks handled by edge servers are more likely to reflect the hot events of the regions they serve. Thus, these tasks are likely exhibit high degree of similarity. Similar observation has been reported in previous studies [66 17].
GLUE
0.74
0:54
0.40
0.28
0.98
0.25
0.10
SICK
0.89
0.13
-00
SNLI
0:45
0:14
0.5
0.16
0.29
0.11
0.22
0:35
0.05
REDDIT
0.29
0.29
026
0.21
0.61
0.16
0.30
MMChat
0.24
0.22
0.22
0.49
0.36
-02
LCCC
0.23
0:35
023
0.14
0.18
0.42
0.44
0.0
GLUE
SICK
SNLI
WikI
REDDIT
MMChal
LCDC
Figure 1: Similarity proportion in and be- tween seven datasets
Observation 2. KV caches generated by similar tasks also share similarity.
Building upon the insights from Observation 1, we find that there is also a notable similarity among the KV caches generated by similar tasks (Observation 2). Figure 2a il- lustrates the similarity of top-layer key values across 40 similar inference tasks, with the majority concentrated above 0.7. To further quantify the degrec of KV cache similarity, Fig- ure 26 visualizes the similarity rela- tionships among 10 similar inference tasks. Both Figure 2a and Figure (b highlight the potential to reduce mem- ory usage by exploiting the similarity between KV caches. Given the sig- nificant memory consumption of KV cache, as shown in Figure Obser- vation 2 serves as the basis for our proposal of a method that leverages similar KV caches to improve system throughput and reduce KV memory usage.
24
Task 10
Task 10
(a)
(b)
Figure 2: (a) Cosine similarity scores of layer Key val- ues 40 similar tasks. (b) Visualization of the similarity relationships between the top-layer key values of 10 similar tasks. In (b), the wider the chord width of the same color, the more similar relationships the task has with other tasks.
Observation 3. The performance of the model is not significantly affected when reusing similar KVs.
In Sim-LLLM, each edge server retains the KV cache of previous inference tasks. By identifying the similarity between consecutive tasks, the KV cache from previous task can be leveraged to accelerate

---

100
Batch Size
54
Batch Stoe
SimeLIM
Batch-Soo
52-
9-ttch Size 328
Pertiety
20.8
50
Tiydlams
Sim LUN
10.6
16-
Trnuiamaa 1.18
Usma2 78
Lomaz 138
256
512
1024
2045
256
532
1024
2046
Nodel
Cache
Figure 3: Memory usage of KV cache token length 512 for different models
Figure 4: (a) Comparison of perplexity between Tiny [[8] and our model w.r.1 the cache size(number of processed tasks' se- quence & KVs)). (b) Comparison of throughput between TinyLlama and Sim-LLM w.rt the cache size. The prompt length and genera- tion length are both 2048 in (a) & (b).
subsequent inference tasks. Additionally, edge servers can communicate and exchange information about their respective tasks [19]. Similar tasks can be identified across them to provide another opportunity for inference acceleration. To ensure the feasibility of our approach, we conducted com- parative experiments on model throughput and perplexity (PPL) using TinyLlama [18]. The results presented in Figure which further highlights that exploiting KV reuse accelerate inference tasks and reduce memory usage without compromising model accuracy significantly (Observation 3). The main contributions of this paper summarized as follows:
To the best of our knowledge, Sim-LLM is the first work that leverages task similarity to accelerate LLM inference. We observe common similarity among LLM inference tasks in edge computing systems and discover that the KV caches generated by similar tasks also exhibit similarity. propose Sim-LLM, efficient inference optimization mechanism. Sim-LLM introduces a new method for the identification of similar inference tasks and a KV sharing mechanism across multiple edge servers improve system utilization.
Based observation,
Experimental results using PPL. and various downstream benchmarks demonstrate that Sim-LLM reduces GPU memory usage by up 34.65% and improves throughput by up 39.40% against state-of-the-art approaches on average, without significantly affecting model accuracy.
2 Challenges and Motivation
The memory usage of KV cache increases linearly with the sequence length, batch size, and the number of model layers. As Figure 3 demonstrates, when the batch size is 128, the memory consumption of the KV cache alone is nearly three times the size of the model. In edge computing systems, Observations 1 & 2 present opportunities to optimize KV cache memory consumption because many similar tasks can be received for LLMMs during inference, which leads to the storage of numerous similar KV caches. Leveraging these observations, Sim-LLMM aims to accelerate edge L.MM inference, Specifically, instead of releasing all KV caches after inference, Sim-LLM retains a portion of the KV cache and shares it with subsequent similar tasks. Similar KV caches can also be shared across edge servers, where communication and task information exchange among servers allow for identifying similar tasks already completed in an edge computing system. In such cases, edge servers can offload similar tasks to one another enable inference acceleration. However, Sim-LLMM must tackle three main challenges.
Aaatiie similar tasks identification: How effectively identify similarities between tasks while minimizing the overhead introduced by the identification process.
Efficient handling of unique tasks: Given that not all tasks are similar, how handle unique tasks efficiently without compromising system performance.
Reuse of inter-task KV across edge servers: How to identify similar tasks across multiple edge servers, and upon detecting similarities, how to determine the optimal data to be transmitted for efficient task offloading.
To address the first challenge, extensive research is conducted, revealing that cosine similarity provides significant advantages in identifying semantic similarities between tasks, particularly in the context of textual data. As the number of KV to be stored increases. the overhead incurred

---

from traversing the cache to check for similar tasks may diminish the benefits of utilizing similar tasks to accelerate inference. To mitigate this issue, we adopt Socality-Sensitivve Hashing (LSH) mapping approach [20] When the current task being processed is mapped to the same LSSH bucket as a previously inferred task, there is a high likelihood that they share semantic similarity. However, due to the potential distortion introduced by LSH, we implement an adaptive mapping strategy. Specifically, when the batch size of request sequences is small, we compare the current sequence and previously stored sequences' cosine similarity. For large batch sizes, we employ LSH to identify similar sequences. Since LSH may incur a distortion issue, where highly similar previous tasks are mapped to different hash values two dissimilar tasks may be mapped to the same bucket, degrading the acceleration performance, Sim-LLM merges KVs within the LSH bucket before being utilized in the inference process to address this issue.
To address the second challenge, tasks that have similar tasks can leverage the top-layer KV from these corresponding similar tasks to accelerate inference. This approach inspired by interpreting the Transformer's s stacked layer structure as an iterative process that refines token representations [21]. In this context, the representation at the top layer is considered the most informative. Thus, prioritizing the top layer when evaluating task similarity is a proper choice. However, tasks without similar counterparts requires recomputing KV from scratch during inference, resulting a significant slowdown compared to requests that can leverage previous KVs. Inspired by inter-layer KV sharing methods like YOCO [22) and CLA [23], only the KV for certain layers at the bottom and top of the model are cached. By excluding the need to compute KV for intermediate layers, and omitting the associated weight parameters WK and Wy. the overall computational and parameter overheads minimized. Furthermore, this approach retains the potential for integration with intra-layer KV sharing methods such as MQA and GQA techniques, and can benefit from dataflow optimizations like FlashAttention [5]
To address the third challenge, Sim-LLM combines LSH mapping with prototype learning [24] While LSH mapping is more compatible with and better suited to the task processing mechanism for the single-node scenario discussed in Section 3.1 it can introduce significant query overhead when searching for similar tasks repeatedly across multiple edge servers. Inspired by the concept of prototype learning [24], to overcome this, Sim-LLM extracts and merges task features processed by each edge server to generate a task prototype. Each edge server is required to maintain a global feature table. By comparing the features of an incoming task with those stored in the table, the most likely edge server with similar tasks can be identified. In cases where an edge server does not store any similar tasks, it may optionally seek assistance from its neighboring servers before proceeding with the normal inference process. This approach accelerates task identification and enhances offloading efficiency, improving the overall performance and scalability of inference in distributed edge computing system.
3 Sim-LLM
3.1 Overview
Sim-LLM aims to improve the efficiency of LLM inference while reducing KV cache consumption in edge computing systems, which consists of two distinct processes: one for single-node scenarios and another for multi-node scenarios.
In single-node scenarios, Sim-LLM leverages the similarity between tasks. After processing each batch of requests, the embeddings and top-layer KV of the processed requests are cached. The component responsible for managing this data is referred to as the KV_ MaMaggrr During task processing, the system first checks whether the cached tasks in the KV Manager are similar to the task currently being processed. If match is found, the system utilizes the cached top-layer KV to accelerate inference; otherwise, it performs a normal inference.
In multi-node scenarios, each edge server processes tasks in the same manner as in the single-node scenario. To maximize resource utilization in the system, task similarity is fully exploited. Edge servers communicate over the network to determine whether they have cached similar tasks to the one currently being processed. If similar tasks are found on one of the edge servers in the system, the task request is offloaded to that server for processing with the cached similar K.V.
Figure illustrates the overall workflow of Sim-I.L.M, which will be described thoroughly in Sec- tion 3.2 and Section 3.3 In the preprocessing phase, inference tasks are tokenized to word embeddings

---

Preprocessing
Inference
"Wher MIT!"
Edge
KV Mannogee
Nerwork
Tokenize & Hash Mapping
Eimbeednn Hash Valne
Scheduler
1.15 2.13.
Hash
0.22 1333
bottom stage
middle stage
top stage
Figure 5: Sim-LLM overview
before the Scheduler assigns them to edge servers. During the inference phase, KV replacement and sharing are employed throughout the three-stage model inference process on edge servers.
3.2 Semantic Similarity Identification
How to define whether inference task is similar to previous tasks? One commonly used approach to measure text similarity is Jaccard similarity. However, Jaccard similarity has a notable limitation. It ignores variations in text lengths. For example, the two sentences "I love you!" and "I love you! I love you! love you!" would yield a similarity score of 1, implying that they are identical, despite having different semantic meanings due the repetition in the second sentence. Metrics such as Euclidean distance, Levenshtein distance, and Jaro-Winke Winkler distance are also strongly influenced by text length when measuring similarity, making them unsuitable for evaluating tasks when the text length is unknown. Therefore, the semantic-basee method of cosine similarity is preferred. This method widely used in word vector models (e.g., Word2Vec (25), Glo (26)) and semantic models (e.g., BERT 27 ). Regarding the definition task similarity, Word2 and GloVe consider cosine similarity threshold of 0.6, Sentence- BERT and SemEval use 0.7. Through evaluation of the threshold's 's impact both on generation speed and model accuracy, Sim-LLM employs stringent threshold offset performance degradation after KV reusing while maintaining inference efficiency. considering two tasks to be similar when their cosine similarity exceeds 0.8. The detailed results can be found in Appendix 4.5
How to recognize the semantic similarity between tasks? When employing an exhaustive search approach to compare the similarity between incoming tasks and cached tasks, Sim-LLM outperforms the state-of-the-arr approaches in various downstream tasks. However, this advantage is limited to small batch sizes. As batch size increases, the performance advantage of Sim-LLM diminishes. This performance decline is caused by the cumulative computational overhead introduced by the exhaustive search for similarity calculation, which reduces the inference speed. To address this issue, Sim-LLM employs LSH mapping. During the task caching process, it stores the task's 's corresponding embedding, the top-layer KV, as well as the hash value calculated from the word embedding. By transforming the word embedding of an incoming task into a hash value and mapping it to an LSH bucket, Sim-LLM can efficiently identify cached tasks similar to the current task. By reducing high-dimensional data to lower-dimensional representations, LSH can significantly accelerate task similarity identification.
3.3
KV Manager
KV Manager stores the embeddings and top-layer KVs of previous tasks that Sim-LLM can match similar tasks and utilize similar KVs to accelerate inference. As discussed in Section 2 it leverages similar top-layer KVs to replace the conventional KV generation. Moreover, has been observed that Transformers focus syntactic information in the lower layers and semantic information in the higher layers [28]. in addition to the informative characteristic of tap-layer KV. Thus, makes sense to consider the top-layer KV for replacement between tasks sharing semantic similarity. Thus, there is no need to cache the KV from other layers. Consequently, Sim-LLM caches only the keys and values from single layer, unlike typical Transformer model that caches those from multiple layers. This significantly reduces memory consumption without introducing additional computational overhead during inference.
5

---

Global protolype
Fedwedding
A1
2 143
P2
Taka
83
Edgy Server
Edge
a,
Edge Server
Backe
Backe
1 Backet
X
X
KV Matage
KV Manage
Manager
X
KV
KV
Distasltr
x
Netoork
Figure 6: Multi-nodes share tasks with LSH and task prototype
Although the KV Manager introduces extra memory requirements for storing the embeddings and top-layer KVs of previous tasks (which will be discussed in Section 4.2 this extra memory usage is substantially lower than the memory overhead associated with computing and storing KVs of all tasks. Furthermore, KV Manager facilitates KV sharing across multiple edge nodes further reduce the additional memory overhead, as discussed in Section 3.4
As shown in Figure 5 the KV replacement process within KV_ Manager consists of five steps. When batch of tasks arrives, tokenizer first tokenizes these tasks to obtain their embedding vectors. Then these embeddings are mapped to corresponding hash values. @ The Scheduler distributes these tasks' embeddings and hash values neighboring edge servers to identify similar tasks, t KV VV_Manager stores the embedding vectors, hash values, and top-layer KV of tasks that have been inferred. Before inference begins, the tasks are mapped the LSH buckets in KV _MMnager to facilitate the search for similar tasks. If similar tasks are found, the top-layer KV stored in the KV Manager is passed to the first layer of the LLM (bottom stage). This operation accelerates the prefilling phase by eliminating the need to compute the KV from scratch at the attention module. O When the prefilling phase (top stage) for the first new token about to complete (new tasks' top-layer KV have been computed). KV V__Manager stores the top-layer KV as well as the embedding of the current task and applies an eviction algorithm to remove the KV of outdated tasks. @ Since the top-layer KV is utilized, the inference process of lower layers can be skipped. Inference can be performed directly at the final layer of the model to produce the output (top stage), thereby skipping the entire prefilling phase.
For matched tasks, the queries from all layers are paired solely with the top-layer KV pair of similar previous tasks. This eliminates the need cache or compute the KV for all layers, saving memory consumption and computational overhead. Furthermore, by using the top-layer KV to replace the KV generation process for the current task across all layers, Sim-LLMM bypasses intermediate layers and directly generates the output token from the top layer. However, this method assumes that every task within the current batch has corresponding similar task with an available top-layer KV, which is not always true. To address this, when more than half of the tasks in batch are matched, those without corresponding top-layer KV are temporarily stored for the next batch. This ensures that the remaining tasks can bypass the initial computation process, thereby enhancing the overall inference efficiency. For unmatched tasks, the goal is to accelerate their inference process by optimizing the model configuration. Sim-LLM retains the KV from the bottom three layers (bottom stage) and the top three layers (top stage) in a 'sandwich' structure.
KV Manager continuously maintains metadata for newly processed tasks. As eached tasks accumu- late, the memory footprint grows commensurately. When the cache reaches the cache size, eviction triggered to reclaim space for incoming tasks and to prevent memory exhaustion. Accordingly, (LRU) eviction policy that preferentially preserves frequently reused task KVs while removing those accessed least recently. This design aligns with the central premise of this work: reusing as many similar KVs as possible to accelerate inference.
KV D__Mangger adopts the Leaxi-Recenlly-UU

---

3.4 Across Nodes KV Sharing Mechanism
In Sim-LLM, edge servers can exchange their cached tasks to accelerate the overall inference speed. To further improve memory utilization, Sim-I.LM employs a across nodes KV sharing mechanism.
As shown in Figure 6 let us assume three edge servers in the system. Sim-LLM utilizes LSH to map the current task from edge server 81 into the LSH bucket corresponding edge server 82 and sg. The process consists of these steps: Users submit their tasks to edge servers. The new tasks are tokenized through preprocessing module. @ The scheduler assigns the tokenized embedding to 81 after comparing the user tasks' feature with the servers' task prototype in the global feature table. $1 then begins searching for similar tasks within its LSH bucket.
2 If no similar tasks are found, $1
attempts to transmit the new embeddings to its neighbor servers. ① If a neighbor server identifies a similar task, it will take over the remaining inference process.
When edge server 8, attempts to find tasks similar to the current task among its neighbor servers, begins the inference process of the current task itself. Its neighbors search for similar tasks simultaneously. When similar task is found on neighbor edge server, that server sends a signal to 8; to halt its inference and accelerates the task using the similar task. However, this approach introduces potential issue: s; may be interrupted just as its inference nears completion. This incurs resource waste. A straightforward solution to this problem is to limit the number of hops for the queries. Inspired by the concept of prototype learning (24), Sim-LLM extracts features from inference tasks on each edge server to identify task features they often handle. Each edge server maintains a global feature table that contains prototype for each server. By comparing the features of incoming tasks with those stored in the table, it can identify the edge server that is most likely to have similar tasks. With this method, Sim-LLM can locate a server with similar tasks with only one query.
Despite the across node KV sharing mechanism improves the inference performance, it also incurs communication overhead due to information exchange between edge servers. To address this issue, when similar tasks are identified on another edge server, only the sequence or the corresponding word embeddings are transmitted to the identified edge server, rather than transmitting the KV cache of the similar task to the current processing task from the identified server back the original server. However, if the identified edge server is occupied with processing other tasks, it must transfer the stored KV cache back to the original edge server for accelerating inference.
Experiments
4.1 Experimental Settings
Models. To evaluate the performance of Sim- LLM, experiments are conducted on two widely- used English LLMs, i.c.. TinyLlama- 1.IB [18], Llama2-7B and 13B [29]. as well as a bilin- gual LLM, i... InternLM2-77 7B [30], which sup- ports both Chinese and English. For the exper- iments, we utilize the chat versions of Llama2- 7B, InternLM2-7B. and Llama2-13B.
322
02
H2O
100
Zpcecha
SreurrnggLM
Arkiba
mi Heboor
Sim DiDual
100
3540
Barch
Sathh Sime
Throughput
(b) Memory
Benchmarks. The experiment is conducted in the OpenCompass evaluation framework con- tributors (31] and the Im-eval-harness frame- work (32]. The evaluation involves five key aspects, i.e., reasoning, language, knowledge, ex-
Figure 7: Comparison of throughput and memory consumption between Lim-LLM and baselines on Llama-7B
amination, and understanding, with corresponding benchmarks: (1) Reasoning: CMNLI [3]]
HellaSwag (HeSw) 34 PIQA 35); (2) Language: CHID (36), WSC [37]: (3) Knowledge:
CommonSenseQA (CSQA) 38) BoolQ 28): (4) Examination: MMLU 39]. CMMLU 40 (5)
Understanding: Race- Hicc-High/Middle (H/M) 41 XSum
42 C3
The evaluation is conducted with the official scripts from OpenCompass, employing zero-shot approach without additional training. Two evaluation modes are utilized: perplexity (PPL) and generation (GEN)². The GEN mode is used for the CHID and XSum benchmarks, while both PPL. (WSCp) and GEN (WSCa) modes are used for the WSC benchmark. The remaining benchmarks are evaluated in the PPL mode. OpenCompass subsequently converts the evaluation results for each benchmark into score, with higher scores indicating better performance.
7

---

H20
-0:
120
-G-
H20
16.0
SzeamingLoM
Strsamingu
145
Zppacne
2 15.0
14.5
14.0
Sim-LLMMOuus!
14.5
14.0
14.0
13.5
18.
13.0
114
2048
256
1024
Cache
Cache
(a) Tiama I.IB
(b) Llama-7B
(c) Llama-13B
Figure 8: PPL on Wikipedia dataset
45 at different cache sizes
Experiment Environment. The performance of Sim-LLM under single-node scenarios is evaluated on a server with a single Nvidia A100 80GB GPU. Four physical machines, each equipped with four Nvidia A40 40GB GPUs, are used as edge servers evaluate the performance across edge nodes. All experiments related to the PPL evaluation are conducted on a 10M subset of the development set from SlimPajama [44 and the Wikipedia dataset [5]
4.2 Performance Results
Experiments are conducted on each dataset, and the average score across all tasks for each aspect is reported.
The baselines include the standard Transformer models as well as the state-of-the-a art KV cache evic- tion and intra-layer compression methods, including StreamingLLMM [46). H20 [7), ZipCache [48]
and Ark Vale 449. Their details can be found in Appendix
We first evaluate the generation performance of Sim-LLM. Figure 7a and Figure 7b compare the throughput and memory consumption of the baselines and Sim-LL.M on servers with A40 GPUs with varied batch sizes. Both the prompt length and generation length are set to 2,048. Sim-LLM outperforms all baselines across all tested batch sizes, achieving a speedup of up to 39.40% (33.04% on average) and memory reduction of up to 34.65% (30.05% on average). Please note that the largest batch size does not always yield the maximum system throughput. Notably, the increase in system throughput is not solely attributed to the increased batch size. By substituting the KV computation during the prefilling phase with the replacement of top-layer KV of similar tasks, Sim-LLM effectively eliminates redundant calculations, leading both higher throughput and lower memory consumption. Additionally, we observe that as the batch size exceeds 256, system throughput no longer increases and even decreases when the batch size reaches 1,024. This suggests that the model transitions from memory-bound to compute-bound [5).
To validate the generation quality of Sim-LLM, evaluate its PPL various downstream benchmarks. Figure g demonstrates the PPL of both the baselines and Sim-LLM for the Llama series under different cache sizes. The PPL achieved by Sim-LLM is better when the cache size larger. Since Sim-LLM directly utilizes the stored KVs, the computational complexity during the generation process is reduced, result- ing in fewer inference steps and faster conver- gence to lower PPI.. When a similar task is identified, the corresponding KV of previous tasks serves as supplement, providing context from similar scenarios that Sim-LLM has previously encountered. This enables Sim-LLM to generate outputs with greater certainty, reducing its reliance on unknown contexts. Essentially, Sim-LLM becomes more familiar with task types over time, leading to decrease in PPL. Eviction-baas based methods, H2O and StreamingLLMM exhibit higher PPL compared to ZipCache, primarily due to their information loss upon token eviction. In contrast, the quantization-based method ZipCache preserves full data integrity, thereby achieving better performance. Furthermore, we evaluate zero-shot per- formance on the benchmarks mentioned in Section 4.1 using the Im-eval-harness framework [32] Table 2 shows the average score across all tasks in Appendix B.4 demonstrating that Sim-LLM does
SI-LUN Hemorr
Latenn'
5m
CAdh+ SbM
Cadfe 236
(a) TinyLJama1.1B
(b) Llama7B
Figure 9: Comparison of latency per token and memory consumption of (Tiny) and our model w.r.t. different cache sizes

---

not experience significant performance drops in any specific area. The PPL results in Figure X and downstream benchmarks scores in Table Dillustrate that Sim-LLM effectively preserves both overall and task-specific performance, maintaining reliable output quality.
To evaluate the impact of cache size on Sim-LLM, we conducted experiments using various cache size configurations. Figure iillustrates the latency and memory usage of Llama models with varying cache sizes incurred by Sim-LLM. Sim-LLM significantly reduces model inference latency and effectively decreases memory usage across most cache size configurations. As shown in Figure (9b only when the cache size reaches 4096, the memory usage of Sim-LLM becomes larger than that of Llama. It results from the stored VVs and embeddings from previous tasks, offsetting the benefits of reducing memory usage by reusing similar KVs.
Usually, the sequence length does not exceed the maximum input lengths the models are trained on. To highlight the capability of Sim-LLM support models with larger sequence lengths, we also evaluate this situation. Table 3 in Appendix B.. compares the maximum batch sizes and throughput of standard Llama models and Sim-LLM across two types of GPUs. These results confirm it and show Sim-LLM's s capacity to handle significantly larger batch sizes and achieve higher system throughput than all baselines across all settings.
4.3 Impact of Source Layer Used for KV Reusing
In this section, we evaluate the PPL and accu- racy of downstream tasks on Llama-7B (total 32 layers) when choosing different layers as a similar KV source. The position of the source layer is from top, middle, boltom. We select several layers from these positions respectively, e.g., layer to 5 for bottom, 14 to 16 for middle, and 30 to 32 for top. The cache size is set to 512. Figure 10 substantiates that the choice of top-layer KV as source reasonable. Detailed results of downstream tasks are shown in Ap- pendix B.4 can be seen that top layer outperforms bottom and middle layer in both PPL and downstream task accuracy when performing KV sharing. Additionally, it can be observed that as the position of the source layer increases, the model's performance improves, gradually approaching that of the standard model. In addition to the reasons discussed in Section [aand Section 3.3 we analyze that this phenomenon arises because, at lower layers, the KV lacks sufficient semantic information, which is inadequate to enable the model to achieve reliable performance.
59.0
16.0
-8.9
g:3
53.0
Liama
14.0
36.0
Sim
Source
Source
(a) Perplexity
(b) Accuracy
Figure 10: Perplexity on the SlimPajama dataset and average downstream task results of Llama-70 7B
4.4 Impact of Similar Task Proportion
To evaluate the impact of similarity pro- portion on Sim-LLM, we select tasks from REDDIT 11).. MMChat [13]. LCCC [14),
Sitt LiM
150
WIKI (55, GLUE [50), SICK [5T] and
SNLI 52 to create task sets with varying pro-
portions of similarity, where two tasks are con- sidered similar if their cosine similarity exceeds 0.8. Evaluations are conducted on Llama-7B using the specified task sets, with fixed prompt length of 512 and a generation length of 4,096. When the similarity proportion is zero, Sim-
034
1.5
C.. 24h
2%5
3.0
120
A34
15
211
(a) Throughput
(b) Perplexity
Figure 11: Results of task sets with different pro- portions of similarity on Llama-7B
LLM performs inference in the same manner as standard Transformers. As demonstrated in Figure with the similarity proportion in the task set increasing, Sim-LLM can more effectively leverage the reuse of KVs from similar tasks, thereby accelerating inference and improving throughput, although this may result slight degradation in accuracy.
4.5 Impact of Cosine Similarity Threshold
Since Sim-LLM accelerates inference by exploiting task similarity, we investigate the impact of the similarity threshold on performance. Specifically, we determine the optimal cosine similarity threshold for KV reuse by evaluating generation speed, perplexity, and accuracy on downstream tasks. presents the results. In Figure| 12a the prompt length and the generation length are set to
Figure 12

---

512 and 4,096, respectively. When the threshold of cosine similarity is low, throughput performance exceeds that of a higher threshold. This is because most incoming tasks can easily identify "similar" tasks' KVs for reuse, introducing minimal identification overhead and accelerating the inference process. However, such an advantage comes the expense of model accuracy. as illustrated in Figure [2b and Figure 12c When the threshold reaches 0.9, there is significant drop in generation speed due to fewer tasks being able to reuse KVs, resulting in many tasks performing inference as in the standard Transformer model. Therefore, trade-off between generation speed and model accuracy, a threshold 0.8 preferred.
Related Work
Extensive research has been dedicated to reducing the memory consumption of the KV cache for efficient inference of LLMs. Existing studies can be cat- egorized into three main approaches.
44.
Memory Space Optimization. KV caches be scheduled properly to maximize resource utilization in edge computing systems. Infinite- LLM [53] partitions the KV cache into smaller, more manageable units for independent processing and management across the global memory scheduler in a distributed system. LoongServe [54] dynamically assigns the prefilling and decoding phases tasks to instance groups, which can scale up or down in response to changing load demands, without incurring KV migration overhead.
(a) Throughput
(b) Perplexity
(c) Accuracy
Figure 12: Results of different thresholds of cosine similarity on Llama-7B. Evaluation conducted on SlimPajama 44
dataset. The downstream tasks used in (c) are CMNLI 33 HellaSwag 34. and XSum 42
Memory Usage Optimization. KVs can also be selectively retained save on system resources [46] 47 [6]. Scissorhands (56) and H2O [47 preserve only the crucial components of the KV cache based on attention scores. StreamingLLM [46] retains only the recent context window and a few initial tokens as an attention sink, and discards the rest of the past context. SnapKV [55] focuses on pruning tokens in the input prompt in response increasing input lengths.
Other KV Optimization Methods. Other methods for KV optimization include quantization g 48 57 58). eviction 49 59], and merging [7]. Quantization is a representative method for KV cache compression. It reduces the KV cache size from 16-bit to 4-bit or even lower. CacheGen is particularly relevant as it combines KV offloading with quantization, mitigating KV transfer overhead by applying quantization- based compression algorithms. Other methods have also been proposed to retain only the KV cache of important tokens while discarding others to save GPU memory [46 47 (55. However, since the importance of tokens often changes during the decoding process, discarded tokens may become crucial for further computation [59]. which potentially leads to accuracy loss. Quest (59) mitigates this issue by dynamically selecting small portion of critical KV cache for attention computation for each query token, while retaining all KV cache.
Conclusion and Future Work
This paper present Sim-LLM, novel method for reducing memory consumption and improving throughput for LLM inference edge computing systems. reduces the key-value computation over- head by reusing the KV calculated for previous tasks similar current tasks. Extensive experiments demonstrate that Sim-LLM achieves reductions of up to 34.65% in memory usage and improvements of up to 39.400 99.40% in throughput, with negligible performance degradation in accuracy.
The primary limitation of Sim-LLM lies in the necessity of storing the top-layer KV pairs, as well as corresponding embeddings of processed tasks. Although Sim-LLM performs effectively across wide range of cache size configurations, the storage required for saving additional KVs and embeddings may exceed the benefits gained from reducing the KV cache through the use of similar tasks. Furthermore, since the tapelayer KVss computed by the model are essential, the KV shape across all edge servers must remain consistent. Consequently, the models deployed across edge servers must be identical. Our future work will explore how to share KVs for heterogencous models.
10

---

Acknowledgments
We sincerely thank the AC and reviewers for their constructive and valuable feedback. This research was supported in part by the National Science Foundation of China (NSFC) under grant No. 62402188. the China Postdoctoral Science Foundation under Grant No. 2024M761017, 3025TI80422 the Postdoctoral Fellowship Program of CPSF under Grant No. GZC20240542, the Postdoctoral Project of Hubei Province under Grant Number 2024HBBHCXA026, and the HUST Kunpeng Ascend Sci-Edu Innovation and Incubation Center.
References
[1] Mahadev Satyanarayanan. The emergence of edge computing. IEFE Computer, 50(1):30-39, 2017.
[2] Weisong Shi, Jie Cao, Quan Zhang. Youhuizi Li, and Lanyu Xu. Edge computing: Vision and challenges. IEEE Internet Things, 3(5):637 646, 2016.
[3] Yuqing Tian, Zhaoyang Zhang, Yuzhi Yang, Zirui Chen, Zhaohui Yang, Richeng Jin, Tony QS Quek, and Kai-Kit Wong. An edge-cloud collaboration framework for generative ai service provision with synergetic big cloud model and small edge models. IEEE Network, 38(5):37-46, 2024.
[4] Ranesh Kumar Naha, Saurabh Garg, Dimitrios Georgakopoulos, Prem Prakash Jayaraman, Longxiang Gao, Yong Xiang, and Rajiv Ranjan. Fog computing: Survey of trends, architectures, requirements, and research directions. IEEE Access, 0579800480094 2018.
[5] Tri Dao, Dan Fu, Stefano Ermon, Atri Rudra, and Christopher Ré. Flashattention: Fast and memory- efficient exact attention with to-awarensss, In Proceedings of the Annual Conference on Neural Information Processing Systems (NeurIPS), 2022.
[6] Woosuk Kwon, Zhuohan Li, Siyuan Zhuang, Ying Sheng, Lianmin Zheng, Cody Hao Yu, Joseph Gonzalez, Hao Zhang, and Ion Stoica. Efficient memory management for large language model serving with pagedattention. In Proceedings of the ACM Symposium Operating Systems Principles (SOSP), 2023.
[7] Akide Liu, Jing Liu, Zizheng Pan, Yefei Hc, Gholamreza Haffari, and Bohan Zhuang. Minicache: KV cache compression in depth dimension for large language models. In Proceedings of the Annual Conference Neural Information Processing Systems (NeurIPS), 2024.
[8] Yuhan Liu, Hanchen Li. Yihua Cheng, Siddhant Ray. Yuyang Huang, Qizheng Zhang, Kuntai Du, Jiayi Yao. Shan Lu, Ganesh Ananthanarayanan, Michael Maire, Henry Hoffmann, Ari Holtzman, and Junchen Jiang. CacheGen: KV cache compression and streaming for fast large language model serving. Proceedings of the ACM Special Interest Group on Data Communications (SIGCOMM), 2024.
[9] Huiqiang Jiang, Qianhui Wu, Chin- Yew Lin. Yuqing Yang, and Lili Qiu. Limlingua: Compressing prompts accelerated inference large language models. Proceedings of the Conference on Empirical Methods in Natural Language Processing (EMNLP), 2023.
[10] Chi Han, Qifan Wang, Hao Peng, Wenhan Xiong, Yu Chen, Heng and Sinong Wang. Lm-infinite: Zero-shot extreme length generalization for large language models. In Proceedings of the Conference of the North American Chapter of the Association for Computational Linguisticx: Human Language Technologies (NAACL), 2024.
[11] Suyu Ge, Yunan Zhang, Liyuan Liu, Minjia Zhang, Jiawei Han, and Jianfeng Gao. Model tells you what to discard: Adaptive KV cache compression for LLMss. In Proceedings of the International Conference on Learning Representations (ICLR), 2024.
[12] Jason Baumgartner, Savvas Zannettou, Brian Keegan, Megan Squire, and Jeremy Blackburn. The pushshift reddit dataset. In Proceedings of the International AAAI Conference on Web and Social Media (ICWSM), 2020.
[13] Yinhe Zheng, Guanyi Chen, Xin Liu, and Jian Sun. Mmchat: Multi-modal chat dataset on social media. In P Proceedings of the Language Resources and Evaluation Conference (LREC), 2022.
[14] Yida Wang, Pei Ke, Yinhe Zheng, Kaili Huang, Yong liang, Xiaoyan Zhu, and Minlie Huang. large-scale chinese short-text conversation dataset. In Proceedings of the CCF International Conference on Natural Language Processing and Chinese Computing (NLPCC), 2020.
[15] Hang Wu, Lixing Chen, Cong Shen, Wujie Wen, and Jie Xu. Online geographical load balancing for energy- harvetna mobile edge computing. In Proceedings of the IEEE International Conference on Communication (ICC), 2018
11

---

[16] Shijing Li, Tian Lan, Bharath Balasubramanian, Moo-Ryong Ra. Hee Won Lee, and Rajesh Panta. El- dedup: Enabling collaborative data deduplication at the network edge. In Proceedings of the IEEE International Conference on Distributed Computing Systems (ICDCS), 2019.
[17] Ruikun Luo, Hai Jin, Qiang He, Song Wu, and Xiaoyu Xia. Enabling balanced data deduplication in mobile edge computing. IEEE Transactions on Parallel and Distributed Systems, 34(5):1420-143) 2023.
[18] Peiyuan Zhang, Guangtao Zeng. Tianduo Wang, and Wei Lu. Tinyllama: An open-source small language model. arXiv preprint urXiv:2401.0238. 2024.
[19] Jintao Wu, Xiaolong Xu, Guangming Cui, Yiwen Zhang, Lianyong Qi, Wanchun Dou, and Zhipeng Cai. falrness-awars budgeted edge server placement for connected autonomous vehicles. IEEE Transactions on Mobile Computing. 24(6):4762 4776, 2025.
[20] Anirban Dasgupta, Ravi Kumar, and Tamás Sarlós. Fast locality-sensiivee hashing. In Proceedings of the ACM International Conference on Knowledge Discovery and Data Mining (KDD), 2011.
[21] Haoyi Wu and Kewei Tu. Probabilistic transformer: A probabilistic dependency model for contextual word representation. In Proceedings of the Findings of the Association for Computational Linguistics (ACL), 2023.
[22] Yutao Sun, Li Dong. Yi Zhu, Shaohan Huang, Wenhui Wang. Shuming Ma, Quanlu Zhang, Jianyong Wang, and Furu Wei. You only cache once: Decoder-decoder architectures language models. In Proceedings of the Annual Conference on Neural Information Processing Systems (NeurIPS), 2024.
[23] William Brandon, Mayank Mishra, Aniruddha Nrusimha, Rameswar Panda, and Jonathan Ragan-Kell Kelley, Reducing transformer key-value cache size with cross-layer attention. In Proceedings of the Annual Conference on Neural Information Processing Systems (NeurIPS), 2024.
[24] Hlong-Ming Yang, Yao Zhang, Fei Yin, and Cheng-Lin Liu. Robust classification with convolutional prototype learning. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR), 2018.
[25] Tomás Mikolov, Kai Chen, Greg Corrado, and Jeffiey Dean. Efficient estimation of word representations in vector space. In Proceedings of the International Conference on Learning Representations (ICLR). 2013.
[26] Jeffrey Pemnington, Richard Socher, and Christopher Manning. GloVe: Global vectors for word representa- tion. In Proceedings of the Conference on Empirical Methods in Natural Language Processing (EMNLP), 2014.
[27] Jacob Devlin, Ming-Wo Wei Chang, Kenton Lee, and Kristina Toutanova. Bert: Pre-training of deep bidirec- tional transformers for language understanding. In Proceedings of the Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies (NAACT.) 2019.
[28] Christopher Clark, Kenton Lec, Ming- Wei Chang, Tom Kwiatkowski, Michael Collins, and Kristina Toutanova. Boolq: Exploring the surprising difficulty of natural yes/no questions. In Proceedings of the Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies (NAACL), 2019.
[29] Hugo Touvron, Louis Martin, Kevin Stone, Peter Albert, Amjad Almahairi, Yasmine Babaci, Nikolay Bashlykov, Soumya Batra, Prajjwal Bhargava, Shruti Bhosale, Dan Bikel, Lukas Blecher, Cristian Can- ton Ferrer, Moya Chen, Guillem Cucurull, David Esiobu, Jude Fernandes, Jeremy Fu, Wenyin Fu, and Brian Fuller. Llama 2: Open foundation and fine-tuned chat models. arXiv preprint arXiv:2307.09288 2023.
[30] Zheng Cai, Maosong Cao, Haojiong Chen, Kai Chen, Keyu Chen, Xin Chen, Xun Chen, Zehui Chen, Zhi Chen, Pei Chu, Xiaoyi Dong, Haodong Duan, Qi Fan, Zhaoye Fei, Yang Gao, Jiaye Ge, Chenya Gu, Yuzhe Gu, Tao Gui, and Aijia Guo. InternLM2 technical report. arXiv preprint arXiv:2403.17297 2024.
[31] Opencompass: A universal evaluation platform for foundation models.
https://github. com/
ppen--compasssppencoompa 2023.
[32] Leo Gao, Jonathan Tow, Baber Abbasi, Stella Biderman, Sid Black, Anthony DiPofi, Charles Foster, Laurence Golding, Jeffrey Hsu, Alain Noac'h, h, Haonan Li, Kyle McDonell, Niklas Muennighoff, Chris Ociepa, Jason Phang. Laria Reynolds, Hailey Schoelkopf, Aviya Skowron, Lintang Sutawika, Eric Tang. Anish Thite, Ben Wang, Kevin Wang, and Andy Zou, Lm eval harness: A framework for few-shot language model evaluation. htttps//zeeodo. wwwpor/rggrr//eeeet//rroo 126006622
2024.
12

---

[33] Liang Xu, Hai Hu, Xuanwei Zhang. Lu Li, Chenjie Cao. Yudong Li. Yechen Xu, Kai Sun, Dian Yu, Cong Yu, Yin Tian, Qianqian Dong. Weitang Liu, Bo Shi, Yiming Cui, Junyi Li, Jun Zeng, Rongzhao Wang. Weijian Xie, Yanting Li, Yina Patterson, Zuoyu Tian, Yiwen Zhang. He Zhou, Shaoweihua Liu. Zhe Zhao, Qipeng Zhao, Cong Yue, Xinrui Zhang. Zhengliang Yang, Kyle Richardson, and Zhenzhong Lan. CLUE: A Chinese language understanding evaluation benchmark. In Proceedings of the 28th International Conference on Computational Linguistics (ICCL), 2020.
[34] Rowan Zellers, Ari Holtzman, Yonatan Bisk, Ali Farhadi, and Yejin Choi. HellaSwag: Can a machine really finish your sentence? In Proceedings of the Annual Meeting of the Association for Computational Linguistics (ACL), 2019.
[35] Yonatan Bisk. Rowan Zellers, Ronan Le Bras, Jianfeng Gao, and Yejin Choi. Piqa: Reasoning about physical commonsense in natural language. In Proceedings of the Association for the Advancement of Artificial Intelligence (AAAI), 2019.
[36] Chujie Zheng, Minlic Huang, and Aixin Sun. ChID: large-scale Chinese IDiom dataset for cloze test. In Proceedings of the Annual Meeting of the Association for Computational Linguistics (ACL), 2019.
[37] Hector J Levesque, Ernest Davis, and Leora Morgenstern. The winograd schema challenge. In Proceedings of the International Conference the Principles of Knowledge Representation and Reasoning (KR), 2012.
[38] Alon Talmor, Jonathan Herzig, Nicholas Lourie, and Jonathan Berant. CommonsenseQA: A question answering challenge targeting commonsense knowledge. In Proceedings of the Conference of the North American Chapter of the Association for Computational Linguisties: Human Language Technologies
(NAACL), 2019.
[39] Dan Hendrycks, Collin Burns, Steven Basart, Andy Zou, Mantas Mazeika, Dawn Song, and Jacob Steinhardt. Measuring massive multitask language understanding. Proceedings of the International Conference on Learning Representations (ICLR), 2021.
[40] Haonan Li, Yixuan Zhang, Fajri Koto, Yifei Yang, Hai Zhao, Yeyun Gong, Nan Duan, and Timothy Baldwin, CMMLU: Mcasuring massive multitask language understanding in Chinese. In Proceedings of the Findings of the Association for Computational Linguistics (ACL), 2024.
[41] Guokun Lai, Qizhe Xie, Hanxiao Liu, Yiming Yang, and Eduard Hovy. RACE: Large-scale ReAding comprehension dataset from examinations. In Proceedings of the Conference on Empirical Methods in Natural Language Processing (EMNLP), 2017.
[42] Shashi Narayan, Shay B. Cohen, and Mirella Lapata. Don't give me the details, just the summary! topic-aware convolutional neural networks for extreme summarization. In Proceedings of the Conference on Empirical Methods in Natural Language Processing (EMNLP), 2018.
[43] Kai Sun, Dian Yu, Dong Yu, and Claire Cardie. Investigating prior knowledge for challenging Chinese machine reading comprehension. Transactions of the Association for Computational Linguistics, 8:141- 155, 2020.
[44] Daria Soboleva, Faisal Al-Khateeb, Robert Myers, Jacob R Steeves, Joel Hestness, and Nolan Dey. SlimPajama: 627B token eleaned and deduplicated version RedPajama. ltttps:// werebbra....essee.e........eeese http/hoog/slimppp ama-paaaajaaaaaaaaa token--toooo-eeee cleaned-andde handdddddulllaateddverereirrrr rewpaff jama 2023.
[45] Wikipedia dataset containing cleaned articles of all languages. httpp://ddumps. wwkinmmaa....iiiiaaaaaa.....aaa. 2023.
[46] Guangxuan Xiao, Yuandong Tian, Beidi Chen, Song Han, and Mike Lewis. Efficient streaming language models with attention sinks. In Proceedings of the International Conference on Learning Representations (ICLR), 2024.
[47] Zhenyu Zhang, Ying Sheng, Tianyi Zhou, Tianlong Chen, Lianmin Zheng, Ruisi Cai, Zhao Song, Yuandong Tian, Christopher Ré, Clark Barrett, Zhangyang 'Atlas" Wang, and Beidi Chen. H2O: Heavy-hitter oracle for efficient generative inference of large language models. In Proceedings of the Annual Conference on Neural Information Processing Systems (NeurIPS), 2023.
[48] Yefei He, Luoming Zhang, Weijia Wu, Jing Liu, Hong Zhou, and Bohan Zhuang. Zipcache: Accurate and efficient KV cache quantization with salient token identification. In Proceedings of the Annual Conference Neural Information Processing Systems (NeurIPS), 2024.
13

---

[49] Renze Chen, Zhuofeng Wang. Beiquan Cao. Tong Wu. Size Zheng, Xiuhong Li, Xuechao Wei, Shengen Yan, Meng Li, and Yun Liang. Arkvale: Efficient generative LLM inference with recallable key-value eviction. In Proceedings of the Annual Conference on Neural Information Processing Systems (NeurIPS), 2024.
[50] Alex Wang. Amanpreet Singh, Julian Michael, Felix Hill, Omer Levy, and Samuel Bowman. GLUE: multi-task benchmark and analysis platform for natural language understanding. In Proceedings of the EMNLP Workshop BlackboxNIP.. Andyzing and Interpreting Neural Networks for NLP (BlackboxNLP)), 2018.
[51] Marco Marelli, Stefano Menini, Marco Baroni, Luisa Bentivogli, Raffaella Bernardi, and Roberto Zampar- elli. A SICK cure for the evaluation of compositional distributional semantic models. In Proceedings of the Language Resources and Evaluation Conference (LREC), 2014.
[52] Peter Young, Alice Lai, Micah Hodosh, and Julia Hockenmaier. From image descriptions to visual denotations: New similarity metrics for semantic inference over event descriptions. Transactions of the Association for Computational Linguistics, 2:67-78, 2014.
[53] Bin Lin, Chen Zhang, Tao Peng, Hanyu Zhao, Wencong Xiao, Minmin Sun, Anmin Liu, Zhipeng Zhang, Lanbo Li, Xiafci Qiu, Shen Li, Zhigang Ji, Yong Li, and Wei Lin. Infinite-LLM: Efficient LLM service for long context with distattention and distributed KV cache. arXiv preprint arXiv:2401 026669, 2024.
[54] Bingyang Wu, Shengyu Liu, Yinmin Zhong, Peng Sun, Xuanzhe Liu, and Xin Jin. Loongserve: Efficiently serving long-context large language models with elastic sequence parallelism. In Proceedings of the ACM Symposium on Operating Systems Principles (SOSP), 2024.
[55] Yuhong Li, Yingbing Huang, Bowen Yang. Bharat Venkitesh, Acyr Locatelli, Hanchen Ye, Tianle Cai, Patrick Lewis, and Deming Chen. SnapKV: LLM knows what you are looking for before generation. In Proceedings of the Annual Conference Neural Information Processing Systems (NeurIPS), 2024.
[56] Zichang Liu, Aditya Desai, Fangshuo Liao, Weitao Wang, Victor Xie, Zhaozhuo Xu, Anastasios Kyrillidis, and Anshumali Shrivastava. Seissorhands: Exploiting the persistence of importance hypothesis for LLM KV cache compression test time. In Proceedings of the Annual Conference on Neural Information Processing Systems (NeurIPS), 2023.
[57] Coleman Hooper, Sehoon Kim, Hiva Mohammadzadeh. Michael Mahoney, Yakun Sophia Shao, Kurt Keutzer, and Amir Gholami. KVQuant: Towards 10 million context length LLM inference with KV cache quantization. In Proceedings of the Annual Conference on Neural Information Processing Systems (NeurlPS), 2024.
[58] Tianyi Zhang, Jonah Yi, Zhaozhuo Xu, and Aushumali Shrivastava. KV cache is I bit per channel: Efficient large language model inference with coupled quantization. In Proceedings of the Annual Conference Neural Information Processing Systems (NeurIPS), 2024.
[59] Jiaming Tang, Yilong Zhao, Kan Zhu, Guangxuan Xiao, Baris Kasikei, and Song Han. Quest: qwar-- sparsity for efficient long-context LLM inference. In Proceedings of the International Conference on Machine Learning (ICML), 2024.
14

---

NeurIPS Paper Checklist
The checklist is designed to encourage best practices for responsible machine learning research, addressing issues of reproducibility. transparency, research ethics, and societal impact. Do not remove the checklist: The papers not including the checklist will be desk rejected. The checklist should follow the references and follow the (optional) supplemental material. The checklist does NOT count towards the page limit.
Please read the checklist guidelines carefully for information on how to answer these questions. For each question in the checklist:
You should answer [Yes], , [No] or [NA].
[NA] means either that the question is Not Applicable for that particular paper or the relevant information is Not Available.
Please provide short (1-2 sentence) justification right after your answer (even for NA).
The checklist answers are an integral part of your paper submission. They are visible to the reviewers, area chairs, senior area chairs, and ethics reviewers. You will be asked to also include it (after eventual revisions) with the final version your paper, and its final version will published with the paper.
The reviewers of your paper will be asked use the checklist as one of the factors their evaluation. While "[Yes] is generally preferable to "[No] ", it is perfectly acceptable to answer "[No] proovdddd proper justification is given (e.g., "error bars are not reported because it would be too computationally expensive" or "we were unable to find the license for the dataset we used"). In general, answering "[No] or "[NA]" is not grounds for rejection. While the questions are phrased in binary way, we acknowledge that the true answer is often more nuanced, so please just use your best judgment and write justification to elaborate. All supporting evidence can appear either in the main paper or the supplemental material, provided in appendix. If you answer [Yes] to a question, in the justification please point to the section(s) where related material for the question can be found.
IMPORTANT, please:
Delete this instruction block, but keep the section heading "NeurIPS paper checklist",
Keep the checklist subsection headings, questions/answers and guidelines below.
Do not modify the questions and only use the provided macros for your answers.
1. Claims
Question: Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?
Answer: [Yes]
Justification:
The abstract and introduction accurately reflect the paper's contributions and scope.
Guidelines:
The answer NA means that the abstract and introduction do not include the claims made in the paper.
The abstract and/or introduction should clearly state the claims made, including the contributions made the paper and important assumptions and limitations. A No or NA answer to this question will not be perceived well by the reviewers.
The claims made should match theoretical and experimental results, and reflect how much the results can be expected to generalize to other settings.
is fine to include aspirational goals as motivation as long is clear that these goals are not attained by the paper.
2. Limitations
Question: Does the paper discuss the limitations of the work performed by the authors?
15

---

Answer: [Yes]
Justification: The authors have discussed the limitations of the work in the last Section of this paper.
Guidelines:
The answer NA means that the paper has no limitation while the answer No means that the paper has limitations, but those are not discussed in the paper.
The authors encouraged to create separate "Limitations" section in their paper.
The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (e.g., independence assumptions, noiseless settings, model well-specification, asymptotic approximations only holding locally). The authors should reflect on how these assumptions might be violated in practice and what the implications would be.
The authors should reflect the scope of the claims made, e.g., if the approach was only tested on a few datasets or with few runs. In general, empirical results often depend on implicit assumptions, which should be articulated.
The authors should reflect on the factors that influence the performance of the approach. For example, facial recognition algorithm may perform poorly when image resolution is low or images are taken in low lighting. Or speech-to-text system might not be used reliably provide closed captions for online lectures because it fails handle technical jargon.
The authors should discuss the computational efficiency of the proposed algorithms and how they scale with dataset size.
If applicable, the authors should discuss possible limitations of their approach to address problems of privacy and fairness.
While the authors might fear that complete honesty about limitations might be used by reviewers as grounds for rejection, worse outcome might be that reviewers discover limitations that aren't 't acknowledged judgment and recognize that individual actions in favor of transparency play an impor- tant role in developing norms that preserve the integrity of the community, Reviewers will be specifically instructed to not penalize honesty concerning limitations.
the paper. The authors should use their best
3. Theory Assumptions and Proofs
Question: For each theoretical result, does the paper provide the full set of assumptions and a complete (and correct) proof?
Answer: [NA]
Justification:
The paper does not include theoretical results.
Guidelines:
The answer NA means that the paper does not include theoretical results.
All the theorems, formulas, and proofs in the paper should be numbered and cross- referenced.
All assumptions should be clearly stated or referenced the statement of any theorems.
The proofs can either appear in the main paper or the supplemental material, but if they appear in the supplemental material, the authors are encouraged provide short proof sketch to provide intuition.
Inversely, any informal proof provided in the core of the paper should be complemented by formal proofs provided in appendix or supplemental material.
Theorems and Lemmas that the proof relies upon should properly referenced.
4. Experimental Result Reproducibility
16

---

Question: Does the paper fully disclose all the information needed to reproduce the main ex- perimental results the paper to the extent that affects the main claims and/or conclusions of the paper (regardless of whether the code and data are provided or not)?
Answer: [Yes]
Justification: The authors have disclosed all the information needed reproduce the main experimental results.
Guidelines:
The answer NA means that the paper does not include experiments.
If the paper includes experiments, a No answer to this question will not be perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data are provided or not.
the contribution is dataset and/or model, the authors should describe the steps taken to make their results reproducible or verifiable.
Depending on the contribution, reproducibility can be accomplished in various ways. For example, if the contribution is novel architecture, describing the architecture fully might suffice, or if the contribution a specific model and empirical evaluation, it may be necessary to either make it possible for others to replicate the model with the same dataset, or provide access to the model. In general. releasing code and data is often one good way accomplish this, but reproducibility can also be provided via detailed instructions for how to replicate the results, access to a hosted model (e.g.. in the case of large language model), releasing of model checkpoint, other means that appropriate to the research performed.
While NeurIPS does not require releasing code, the conference does require submis- sions to provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution. For example
(a) If the contribution is primarily new algorithm, the paper should make it clear how to reproduce that algorithm.
(b) the contribution is primarily new model architecture, the paper should describe the architecture clearly and fully.
(c) the contribution is a new model (e.g., a large language model), then there should either be way to access this model for reproducing the results or way to reproduce the model (e.g., with an open-source dataset or instructions for how to construct the dataset).
(d) We recognize that reproducibility may be tricky in some cases, in which case authors welcome describe the particular way they provide for reproducibility. In the case of closed-source models, it may be that access to the model is limited in some way (e.g., registered users), but it should be possible for other researchers to have some path to reproducing or verifying the results.
5. Open access to data and code
Question: Does the paper provide open access to the data and code, with sufficient instruc- tions to faithfully reproduce the main experimental results, as described in supplemental material?
Answer: [Yes]
Justification:
The code is open sourced.
Guidelines:
The answer NA means that paper does not include experiments requiring code.
Please see the NeurlPS code and data submission guidelines (tttps://nips.cc)
[oool////iiiiee//dddesaassssssss for more details.
17

---

While we encourage the release of code and data, we understand that this might not be possible, so "No" an acceptable answer. Papers cannot be rejected simply for not including code, unless this is central to the contribution (e.g., for new open-source benchmark).
The instructions should contain the exact command and environment needed to run to reproduce the results. See the NeurIPS code and data submission guidelines (https
httppp..c/ppblllccguuudeee/e/eee Soceeiduddeeeoeeeuuiiiiieee ssi onPolicy) for more details.
The authors should provide instructions on data access and preparation, including how to access the raw data, preprocessed data, intermediate data, and generated data, etc.
The authors should provide scripts to reproduce all experimental results for the new proposed method and baselines. If only subset of experiments reproducible, they should state which ones are omitted from the script and why.
At submission time, to preserve anonymity, the authors should release anonymized versions (if applicable).
Providing much information possible in supplemental material (appended the paper) recommended, but including URLs to data and code permitted.
6. Experimental Setting/Details
Question:
Does the paper specify all the training and test details (e.g., data splits, hyper- parameters, how they were chosen, type of optimizer, etc.) necessary understand the results?
Answer: [Yes]
Justification:
The authors have included experimental details.
Guidelines:
The answer NA means that the paper does not include experiments.
The experimental setting should be presented in the of the paper to level of detail that is necessary to appreciate the results and make sense of them.
The full details can be provided either with the code, in appendix, as supplemental material.
7. Experiment Statistical Significance
Question: Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?
Answer: [No]
Justification: Error bars are not reported because of limited computing resources.
Guidelines:
The answer NA means that the paper does include experiments.
The authors should answer "Yes" if the results are accompanied by error bars, confi- dence intervals, or statistical significance tests, least for the experiments that support the main claims of the paper.
The factors of variability that the error bars are capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, or overall run with given experimental conditions).
The method for calculating the error bars should be explained (closed form formula, call to a library function, bootstrap, etc.)
The assumptions made should be given (e.g., Normally distributed errors).
should be clear whether the error bar is the standard deviation or the standard error of the mean.
18

---

It is OK to report 1-sigma error bars, but one should state it. The authors should preferably report a 2-sigma error bar than state that they have 96% CI, if the hypothesis of Normality of emrors is not verified.
For asymmetric distributions, the authors should be careful not to show in tables or figures symmetric error bars that would yield results that out of range (e.g. negative error rates).
If error bars are reported in tables or plots, The authors should explain the text how they were calculated and reference the corresponding figures tables in the text.
8. Experiments Compute Resources
For each experiment, does the paper provide sufficient information on the com- puter resources (type of compute workers, memory, time of execution) needed to reproduce the experiments?
Question:
Answer: [Yes]
Justification:
The authors have provided information on computing resources.
Guidelines:
The answer NA means that the paper does not include experiments.
The paper should indicate the type of compute workers CPU GPU, internal cluster, or cloud provider, including relevant memory and storage.
The paper should provide the amount compute required for each of the individual experimental runs well estimate the total compute.
The paper should disclose whether the full research project required more compute than the experiments reported paper (e.g., preliminary or failed experiments that didn't make it into the paper).
9. Code Of Ethics
Question: Does the research conducted the paper conform, in every respect, with NeurlPS Code of Ethics https: htttps://euurpp..oc htt.a//wpw...cpbpa/o///cccccaaa.
Answer: [Yes]
Justification:
The authors confirm that the paper conforms with the NeurIPS Code of Ethics.
Guidelines:
The answer NA means that the authors have not reviewed the NeurIPS Code of Ethics.
If the authors answer No, they should explain the special circumstances that require deviation from the Code of Ethics.
The authors should make to preserve anonymity (e.g., if there is special consid- eration due to laws or regulations their jurisdiction..
Broader Impacts
Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?
Answer: [Yes]
Justification:
The authors have discussed broader impacts of the paper.
Guidelines:
The answer NA means that there is no societal impact of the work performed.
If the authors answer NA or No, they should explain why their work has no societal impact why the paper does not address societal impact.
19

---

Examples of negative societal impacts include potential malicious unintended uses (e.g., disinformation, generating fake profiles, surveillance), faireess considerations (e.g., deploymeet of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations.
The conference expects that many papers will be foundational research and not tied to particular applications, let alone deployments. However, if there is direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that an improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point out that a generic algorithm for optimizing neural networks could enable people to train models that generate Deepfakes faster.
The authors should consider possible harms that could arise when the technology is being used as intended and functioning correctly, harms that could arise when the technology is being used as intended but gives incorrect results, and harms following from (intentional unintentional) misuse of the technology.
If there are negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition attacks, mechanisms for monitoring misuse, mechanisms to monitor how a system learns from feedback over time, improving the efficiency and accessibility of ML).
11. Safeguards
Question: Does the paper describe safeguards that have been put in place for responsible release of data or models that have a high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)?
Answer: [NA]
Justification:
The paper poses no such risks.
Guidelines:
The answer NA means that the paper poses no such risks.
Released models that have high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere usage guidelines or restrictions to access the model or implementing safety filters.
Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images.
We recognize that providing effective safeguards challenging, and many papers do not require this, but we encourage authors to take this into account and make best faith effort.
12. Licenses for existing assets
Are the creators or original owners of assets (e.g., code, data, models), used in the paper, properly credited and are the license and terms of use explicitly mentioned and properly respected?
Question:
Answer: [Yes]
Justification:
The authors have cited the related papers.
Guidelines:
The answer NA means that the paper does not existing assets.
The authors should cite the original paper that produced the code package or dataset.
The authors should state which version of the asset is used and, if possible, include URL.
The name of the license (c.g., CC-BY 4.0) should be included for each asset.
20

---

For scraped data from a particular source (e.g., website). the copyright and terms of service of that souree should be provided.
If assets are released, the license, copyright information, and terms of use in the package should be provided. For popular datasets, papaaets.paperssit wwwpppppppwhhecccc thcode com datasets has curated licenses for some datasets. Their licensing guide can help determine the license dataset.
For existing datasets that re-packaged, both the original license and the license of the derived asset (if has changed) should be provided.
If this information is available online, the authors are encouraged reach out to the asset's s creators.
13. New Assets
Question: Are new assets introduced in the paper well documented and is the documentation provided alongside the assets?
Answer: [NA]
Justification: The paper does not release new assets
Guidelines:
The answer NA means that the paper does not release new assets.
Researchers should communicate the details the dataset/codeemoodel as part their submissions via structured templates. This includes details about training, license, limitations, etc.
The paper should discuss whether and how consent was obtained from people whose asset is used.
At submission time, remember anonymize your assets (if applicable). You either create an anonymized URL or include an anonymized zip file.
14. Crowdsourcing and Research with Human Subjects
Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given to participants and screenshots, if applicable, as well as details about compensation (if any)?
Answer: [NA]
Justification:
The paper does not involve crowdsourcing nor research with human subjects.
Guidelines:
The answer NA means that the paper does involve crowdsourcing nor research with human subjects.
Including this information the supplemental material is fine, but if the main contribu- tion of the paper involves human subjects, then as much detail as possible should be included the main paper.
According to the NeurlPS Code of Ethics, workers involved in data collection, curation, or other labor should be paid at least the minimum wage in the country of the data collector.
15. Institutional Review Board (IRB) Approvals or Equivalent for Research with Human Subjects
Question: Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or an equivalent approval/review based on the requirements of your country or institution) were obtained?
Answer: [NA]
21

---

Justification:
Given the massive amount of experiments conducted in this paper, providing error bars would be computationally prohibitive.
Guidelines:
The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.
Depending on the country in which research is conducted, IRB approval (or equivalent) may be required for any human subjects research. If you obtained IRB approval, you should clearly state this in the paper.
We recognize that the procedures for this may vary significantly between institutions and locations, and we expect authors to adhere to the NeurlPS Code of Ethics and the guidelines for their institution.
For initial submissions, do not include any information that would break anonymity (if applicable), such the institution conducting the review.
22

---

Appendix
A Baselines
The performance of Sim-LLM compared against four baselines.
StreamingLLM [46]: StreamingLLMM is a long-context streaming processing approach designed for LLMs. Its core concept centers stabilizing generation performance by retaining recent tokens and initial "attention sinks." " It maintains a fixed-size cache window that preserves only the KV of recent tokens and small number of initial tokens. Its primary limitation lies in the inherited constraint of pre-trained context window sizes, which prevents enhancement of the model's long-term memory capacity.
H20 477: With the core idea rooted in the optimization of computational efficiency during the generation process through the introduction of the "Heavy-Hitter Oracle" mechanism, H2O reduces unnccessary computations by identifying and prioritizing the most probable output tokens (i.c., heavy-hitter options), thereby enhancing the speed and efficiency of inference.
ZipCache 488]
The main concept of ZipCache is precisely identify the tokens that have significant impact on model generation, and to prioritize retaining information about these key tokens when compressing the KV cache. introduces Shannel-separable per-token quantization strategy to effectively reduce the memory overhead of quantization parameters.
Vrke organizes the tokens in the KV cache into pages and maintains summary information for these pages. Before each attention computation, Vale evaluates the importance of cach page based the current query and the summary information of the cached pages. By dynamically evicting and recalling KV pages, it addresses the issue of token importance changing during decoding steps, without requiring modifications the model architecture fine-tuning. 49 is the state-of-the-art block-level dynamic sparse attention within the vL..M.
ArkVale [4]]:
Ark
B Additional Experimental Results
B.1
Scalability and Latency of Sim-LLM Under Variable Edge Workloads
To evaluate Sim-LLM under heterogeneous edge workloads, we consider two task distri- butions: Poisson distribution and Power-law distribution. The Poisson case approximates a uniform workload, whereas the paw case captures bursty, highly variable demand. As summarized in Table I] under bursty condi- tions Sim-L..M remains scalable and does not introduce appreciable additional latency rela- tive to the -lad scenario. We attribute this robustness to the LRU eviction policy: bursty traffic induces many tasks with overlapping or identical features, and LRU adapts by retaining frequently reused task KVs, allowing a large fraction of requests to hit reusable KVs. This improves cache locality and accelerates inference, rendering LRU particularly well suited to bursty workloads.
Table I: Latency performance of Sim-LLM under differ-
ent cache sizes for bursty and uniform workload patterns
Cache Size
Poisson Latency (ms)
Power-law Latency (ms)
256
25.66
27.68
25.75
28.51
1024
26.31
29,34
2048
33.13
35.88
4096
45.35
47.72
B.2 Zero-shot Accuracy on Benchmarks
In this section, we conducted zero-shot accuracy evaluation on benchmarks discussed in Section 4.1 using the TinyLLama-I. I. Llama2-7B, and Llama2-13B with official scripts from the Im-eval-hamme -armess framework 332. The results in Table 2 show Sim-LLM is comparable to the standard transformers and the state-of-the-art methods in terms of accuracy downstream tasks.
1.
B.3 Maximum Generation Batch Size and Throughput
In this section, we conducted generation performance evaluation with different prompt and generation lengths, using the LanyLlama-1.11B, Llama2-7B, and Llama2-13B with official scripts from the OpenCompass framework 31. The results in Table the-art methods.
show that Sim-LLM outperforms the of
23

---

Table 2: Zero-shot accuracy on different benchmarks w.r.t different models
Model
Reasoning
Language
Knowledge
Examination
Understanding
Avg
TinyLlama-I., 1.IB
44.58
30.2
50,99
46.38
25.00
39,43
H2O
43.68
28.52
43.90
20.94
25.37
32.48
StreamingLLM
45.66
28.22
45.11
22.32
24.36
33.13
ZipCache
52,46
43.75
46.39
33.39
20.97
39.38
Vrle
51.68
38.52
51.90
26.94
24.37
38.68
Sim-LLM(Ours)
42.66
28.87
48.68
45.38
21.57
37.43
IJamm-7B
60.83
40.67
68.67
38.89
33.03
48.41
H2O
57,46
43.75
52.39
35.39
21.97
40.05
StreamingLLM
58.22
41.11
50.24
36.67
22.02
41.65
ZipCache
60.83
40.67
68.67
38,89
33.03
46.55
Vale
57.74
51.00
60.52
33.12
31.15
45.58
Sim-LLM(Ours)
58.45
39.61
67.78
36.19
32.43
46.89
Llama-133
62.50
60.46
75.87
46.76
49.47
58.61
H2O
60.73
54.41
71.86
36.18
44.73
52.89
Streaming
61.35
47.38
73.66
46.03
40.51
51.97
ZipCache
61.31
57.33
74.56
46.16
47.77
56.32
Vale
63.28
50.15
76.18
45.23
52.55
56.67
Sim-LLM((Our))
61.47
58.74
73.66
45.89
46.81
57.31
InternLM2-7B
61.89
71.18
76.57
64.33
69.58
68.71
Sim-L.L((Ours)
60.19
69.22
74.47
61.01
65.12
66.00
Table 3: Maximum achievable batch size and throughput across different sequence lengths on NVIDIA A40 (40GB) servers and an A100 (80GB) GPU server. Notation "u v" represents a prompt length of and generation length of v.
Banch Size
Throughput (Bokensis)
Model Soq. Length
Liama
ZipCoche
ArkVle
Sim-LLMMOUrs)
Llama
ZipCache
ArkVVle
Sim LLMrOurs)
(2.3%)
123 (2.6H)
(8x)
(1.88)
3584 (2.5x)
(2.9x)
1.18
5-2043
768
(4.8x)
(1.7x)
7428.57
10033.40 (2.0x)
5+128
128
224 (1.8k)
243 (1.99)
6:0 (5.0x)
568.56
1023.41 (1.8K)
1228.09 (2.1k)
(2.4.50
54512
2014 (8.3%)
254 (6.1x)
512 (8.26x)
354.53
1985.37 (6.6x)
2187.65 (62x)
3816.51 (10.76x)
TD
(4.8K)
(2.06)
(12.8x)
(2277))
(3.38K)
(3.8x)
512+512
52 (6.7x)
95 (10.6x)
225.31
415.79
473.15 (21x)
678.35 (3.0k)
512+1096
96 (13.7xx)
108 (15.4x)
256 (36.6x)
65.25
246.30 (1.7x)
277.16 (4.x))
522.53 (8.0x)
78
2048+2008
12% (8.5x)
141.10
421.02 (3.0x)
(32x)
(7.729
B.4 Detailed Downstream Task Results
To further elucidate the findings presented in Figure
10b we provide comprehensive statistical
evaluations of downstream task performance in Table
Table 4: Detailed downstream task results on Llama-7B
of Source Layer
HellaSwag
PIQA
BoolQ
Average
3
41.76
67.9
57.28
55.65
42.14
66.97
61.47
56.86
14
43.43
68.17
59.57
57.06
43.88
67.57
61.07
57.50
30
44.22
68.28
60.73
57.74
32
44.74
69.21
61.88
58.61
Standard model
46.58
68.93
61.46
58.99

---

Algorithm 1: Sim-LLM
Input Task batch B {t1, f2.... ................................ En cosine similarity threshold 0; LSH bucket size KV KV_ _Manager with cache size C' and LRU eviction; bottom stage num sandwich_bod
top stage num sandwich
Output: Inference results with KV optimization.
// Step 1: Preprocessing ($3.2)
foreach / E B do
tokenize and compute embedding ex;
14 - LSH(e,, k);
assign t to bucket LSH (SH_etuctthhu]:
// Step 2: Task Similarity Identification ($3.2)
foreach B do
S LSH SSSH_Buckethhy
compute sim(er, ex) for all E
3 S s.t. sim(ee; e.-) ) 2 / then
retrieve cached top-layer KV from KV_Manager [s*];
10
(.KV + KV_ <V_Manager[[s"]:
else
12
compute full KV for t from scratch;
:/ Step 3: KV Cache Management ($3.3)
13
foreach processed task do store (ex, h4, KV,) into KV ___Mageee: size( KV_MaVageN > C then apply LRU eviction;
14
15
// Step 4: Edge Server Communication ($3.4)
17 foreach / E B with no local match do
compute task feature and compare with prototypes in global feature table; remote feature match I found then offload the corresponding server;
19
20
periodically update and syne task prototypes across servers; // Step 5: Inference Execution 3..444.44
foreach do
23
if L.VV exists then
24
run inference by reusing top-layer KV;
else
26
run inference with sandwich configuration, retaining bottom sandwich_bor layers and top sandwich_tor layers KV;
C Algorithm Details
We provide the algorithm details of Sim-LLM in Algorithm [ For each batch of tasks, after preprocessing, tasks are mapped to an L.SH bucket, where similarity matching performed. match is found, the KV of the similar task is reused to accelerate inference; otherwise, normal inference is performed with sandwich configuration. The hash value, embedding value and top-layer KV for each processed task are stored in the KV _Mnagerr for future task reuse (LRU eviction is adopted when reaching cache size). After each batch processed, each server updates its task prototype and sends it to other servers maintain the global feature table.
25


Reviews:
[{"review_id": "YW3iTn0y0V", "reviewer": "Reviewer_aP8N", "summary": "This paper identifies two novel observations: task similarity and KV cache similarity among LLM tasks at the edge. The authors identify similar tasks using cosine similarity, Locality-Sensitive Hashing (LSH) or prototype learning within and across edge servers, and reuses the top-layer KV cache from previous tasks to accelerate the inference.", "questions": "- When fewer tasks in a batch have similar cached tasks, what is the overhead of postponing them? Have you evaluated the worst-case latency in such situations?\n- You choose 0.8 as the cosine similarity threshold. How sensitive are the results to this threshold in real-time inference scenarios? For instance, how does throughput or perplexity change with slightly higher or lower thresholds under typical workloads?\n- What eviction policy is used for managing the KV cache (e.g., FIFO, LRU)? How does this policy affect performance under bursty or highly variable load, where task arrival patterns may not be uniform?\n- In Figure 6, what type of network is used across nodes? Have you measured the communication overhead introduced by task prototype exchange and KV sharing? Could this overhead become a bottleneck in large-scale deployments?\n- ⠀[Typo] Line 148: “such as MQA and GQA used in flash-attention” is inaccurate. MQA and GQA are architectural optimizations, while FlashAttention is dataflow optimization for the attention module. FlashAttention dataflow can be applied to models using MHA, MLA, MQA and GQA, but it is incorrect to say that MQA and GQA are “used” in FlashAttention.", "limitations": "Yes", "rating": 3, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 3, "significance": 2, "originality": 3, "strengths_and_weaknesses": "**Strength**\n* This paper proposes solid observations and corresponding challenges about task and KV similarity, and supports them with concrete visualizations and quantitative results. \n* The authors clearly illustrate their methods to address the challenges: using adaptive LSH for similarity detection, partial KV caching for unmatched tasks, and prototype-based routing for cross-node reuse.\n* The method targets edge deployment scenarios with constrained memory, a highly relevant use case for real-world LLM serving.\n\n**Weakness**\n* While the overall idea of leveraging task similarity for KV cache reuse is novel, the core methodology is relatively simple and straightforward. The mechanism relies on matching semantic similarity via cosine similarity or LSH and reusing top-layer KV, which lacks algorithmic innovation and theoretical depth.\n* For Challenge 2, the paper proposes skipping the KV computation of intermediate layers by caching only the top and bottom layer KV. However, the justification for this caching scheme is insufficient. The authors assume that skipping the middle layers preserves task semantics and output quality, but there is a lack of detailed theoretical analysis to support the validity.\n* The empirical evaluation lacks fine-grained error analysis. There is no quantitative analysis of the success rate of KV reuse, i.e., how often tasks are matched and accelerated in real scenarios. While average accuracy and PPL are reported, the paper does not discuss how incorrect KV reuse (e.g., misclassified similar tasks) affects output quality, which could be critical in edge deployment scenarios.", "was_revised": false, "final_justification": "", "rebuttal": "We would like to express our sincere gratitude to the reviewer for your time and valuable feedback. In particular, your recognition of our “solid observations” and the comment that “the overall idea is novel” align with the positive assessments provided by other reviewers. Below, we provide clarifications in response to your concerns. Please let us know if any further explanation is needed. **(\"W\" refers to \"Weakness\" , \"Q\" refers to \"Questions\")**\n\n ---\n\n-   ##  **Response to W1**\n\n\nWe acknowledge that both cosine similarity and Locality Sensitive Hashing (LSH) are well-established techniques. Their combination, however, offers a scalable and efficient solution for task matching based on semantic similarity, significantly accelerating model inference without requiring costly architectural modifications. Cosine similarity, in particular, has a strong theoretical foundation and has been extensively studied [1] [2]. As discussed in [2], it can be interpreted as the sum of semantic similarities along embedding dimensions after Independent Component Analysis, further justifying its use for semantic comparison.\n\nWe adopt top-layer KV reuse is motivated by the interpretation of the Transformer’s stacked architecture as an iterative refinement process, where lower layers tend to encode syntactic information, while higher layers focus more on semantics [3] [4]. In **Section 4.3**, we further validate this choice through experiments that compare the performance of KV reuse from different layers, demonstrating the superiority of using top-layer KV as the source.\n\n ---\n\n-   ## **Response to W2**\n\n\n As shown in [5] [6] [7], LLMs exhibit significant layer redundancy—intermediate layers contribute minimally and can be removed with limited performance degradation. This supports our design choice: even when skipping these layers, retaining caches from the bottom and top layers preserves essential task-relevant information. We further validate this through experiments in **Section 4.3**.  \n\n \n---\n \n\n-   ## **Response to W3**\n\n\n We conducted additional experiments to examine how the KV reuse match rate changes with the number of tasks. As shown in **Table 1**, the match rate increases as the task set grows. The values in parentheses indicate the proportion of similar tasks in each set. These results suggest that a larger task pool provides more opportunities for KV reuse, thereby enhancing inference efficiency. \n\n|Number of Tasks|Match Rate (0.35)|Match Rate (0.65)|Match Rate (0.89)|\n|---|---|---|---|\n|500 |0.063|0.147|0.258|\n|1000|0.089|0.224|0.432|\n|1500|0.133|0.278|0.583|\n|2000|0.158|0.383|0.664|\n|2500|0.181|0.415|0.722|\n\n**Table 1** KV reuse match rate based on the number of tasks and varying similarity proportion of task sets.\n\n \n\nRegarding incorrect KV reuse, this situation occurs when LSH distortion happens, i.e., two dissimilar tasks are erroneously mapped to the same bucket, or similar tasks are mapped to different buckets. We first explored the probability of this occurrence, and the results, shown in **Table 2**, indicate that **this probability does not exceed 3%**. \n\n|Number of Tasks|Misclassified Task Num|Error Rate|\n|---|---|---|\n|1000|7|0.7%|\n|2000|15|0.75%|\n|3000|18|0.6%|\n|4000|37|0.925%|\n|5000|52|1.04%|\n\n**Table 2** Probability of incorrect KV reuse due to LSH distortion.\n\n \n\nNext, we simulated the impact of a higher error rate on the output quality by forcing tasks to map to the same LSH bucket. The experiments show that only when this probability exceeds 10% does it significantly affect the quality of the output. However, this probability is unlikely to occur in real-world scenarios, indicating that LSH distortion does not overly impact the model accuracy of Sim-LLM.\n\n|Error Rate|PPL|\n|---|---|\n|0.5%  |15.13|\n|0.75% |15.13|\n|1.00% |15.15|\n|2.00% |15.14|\n|3.00% |15.21|\n|5.00% |15.24|\n|7.00% |15.54|\n|10.00%|16.97|\n\n**Table 3** Impact of error rate on model output quality.\n\n \n---\n \n\n-   ##  **Response to Q1**\n\n\nTo improve overall inference efficiency, Sim-LLM postpones tasks that fail to match similar ones within a batch. This allows matched tasks to proceed with KV reuse without being delayed by unmatched tasks. If few tasks in a batch are eligible for reuse, the inference time remains comparable to that of standard processing, and no postponement occurs.\n\nTo address the reviewer’s concern about the worst-case scenario—where unmatched tasks are repeatedly delayed across batches—we conducted an experiment to measure the probability of continuous postponement. As shown in **Table 4**, the probability of a task being postponed for 5 consecutive batches is only **1.1%**, indicating that such cases are **worst-case**. Additionally, **Table 5** reports the delay overhead introduced by postponement, showing minimal impact on overall inference efficiency (e.g., average inference time remains 0.094s for sequence length 5+128).\n\n|Postponed Times|Proportion|\n|---|---|\n|1|74.7%|\n|2|12.4%|\n|3|5.6%|\n|4|1.9%|\n|5|1.1%|\n\n**Table 4** Probability of tasks being continuously postponed across multiple batches.\n\n \n\n|Similarity Proportion of Task Sets|Latency by Postponing (s)|\n|---|---|\n|0.14|0.027|\n|0.35|0.058|\n|0.53|0.064|\n|0.65|0.077|\n|0.74|0.083|\n|0.89|0.109|\n\n**Table 5** Delay overhead introduced by postponed tasks with varying similarity proportions of task sets (sequence length is 5+128).\n\n \n---\n \n\n- ## **Response to Q2**\n\n As shown in **Appendix B.1 (Impact of** **Cosine Similarity** **Threshold)**, we analyzed how the similarity threshold affects throughput, perplexity, and downstream accuracy. A lower threshold improves throughput by enabling more tasks to reuse KVs with minimal overhead. However, this comes at the cost of accuracy. Conversely, at a high threshold (e.g., 0.9), KV reuse becomes rare, leading to slower inference comparable to standard Transformers. As a trade-off, we select 0.8 as the optimal threshold, balancing generation speed and model performance. \n\n ---\n\n-   ## **Response to Q3**\n\n\n (1) We adopt the LRU strategy to maximize reuse of similar KV pairs and thus accelerate inference. LRU retains frequently reused task KVs while evicting the least reused ones.\n\n(2) To address concerns about workload variability, we evaluated LRU under two task distributions: **Poisson (uniform workload)** and **Power-law (bursty workload)**. Results in **Table 6** show that under bursty conditions, LRU maintains strong performance without significant latency increase compared to uniform loads. This is because bursty workloads tend to have many tasks sharing similar features, allowing LRU to effectively retain frequently reused KVs and boost inference speed. \n\n \n\n|Cache Size|Poisson Latency (ms)|Power-law Latency (ms)|\n|---|---|---|\n|256|25.66|27.68|\n|512|25.75|28.51|\n|1024|26.31|29.34|\n|2048|33.13|35.88|\n|4096|45.35|47.72|\n\n \n\n**Table 6** Latency comparison for Poisson and Power-law distributions under the LRU strategy.\n\n---\n\n-   ## **Response to Q4**\n\n\n We use an InfiniBand network to connect edge servers for exchanging task prototypes and KV sharing, leveraging its low-latency, high-bandwidth characteristics widely adopted in edge computing [8] [9] [10].\n\nAfter processing each batch, edge servers update and exchange prototypes to maintain a global feature table. To reduce communication overhead, servers typically exchange task sequences or embeddings rather than full KV caches, as noted in the last paragraph of **Section 3.4**.\n\nRegarding the **communication overhead** introduced by task prototype exchange and KV sharing, we conducted new experiments. The results, as shown in **Table 7**, indicate that this overhead is negligible compared to the inference cost.\n\n|Batch Size|Communication Latency (ms)|Inference Latency (ms)|\n|---|---|---|\n|4|2.37|368.35|\n|8|2.43|310.52|\n|64|2.73|304.02|\n|128|2.88|349.59|\n|256|3.59|549.73|\n|1024|5.47|700.70|\n|2048|10.23|1311.18|\n\n \n\n**Table7** communication and inference latency for different batch sizes.\n\n \n\n Additionally, we also conducted supplementary experiments to evaluate latency in **large-scale deployments**. The results presented in **Table 8** indict, although communication overhead tends to increase as the network scales up, its overall impact on inference efficiency remain limited.\n\n|Number of Nodes|Communication Latency (ms)|Inference Latency (ms)|\n|---|---|---|\n|2|1.22|483.15|\n|4|2.73|337.84|\n|6|4.12|303.51|\n|8|8.71|272.19|\n|12|12.92|232.76|\n\n**Table 8** Communication latency and inference latency with varying numbers of nodes.\n\n \n---\n  \n\n-   ## **Response to Q5**\n\n\n Thank you for pointing out the typo. What we intend to convey is the potential integration between our work and approaches using MQA or GQA techniques. We will revise the statement accordingly in the manuscript.\n\n ---\n\n## **Refs**\n\n[1] Correlation Coefficients and Semantic Textual Similarity. NAACL, 2019.\n\n \n\n[2] Revisiting Cosine Similarity via Normalized ICA-transformed Embeddings. ACL, 2025.\n\n \n\n[3] Probabilistic Transformer: A Probabilistic Dependency Model for Contextual Word Representation. ACL, 2023.\n\n \n\n[4] BoolQ: Exploring the Surprising Difficulty of Natural Yes/No Questions. NAACL, 2019.\n\n \n\n[5] ShortGPT: Layers in Large Language Models are More Redundant Than You Expect. ACL, 2025.\n\n\n\n[6] LaCo: Large Language Model Pruning via Layer Collapse. EMNLP, 2024.\n\n\n\n[7] Llm-pruner: On the structural pruning of large language models. NIPS, 2023.\n\n\n\n[8] FlowStar: Fast Convergence Per-Flow State Accurate Congestion Control for InfiniBand. ToN, 2024.\n\n \n\n[9] Alarm: An Adaptive Routing Algorithm Based on One-Way Delay for Infiniband. TNSE, 2024.\n\n \n\n[10] Enabling the CUDA Unified Memory model in Edge, Cloud and HPC offloaded GPU kernels. CCGrid, 2022.\n\n[11] Tutti: coupling 5G RAN and mobile edge computing for latency-critical video analytics. MobiCom, 2022."}, {"review_id": "0Ze56JBJwI", "reviewer": "Reviewer_G55H", "summary": "The paper introduces Sim-LLM, a novel method to optimize Large Language Model (LLM) inference in edge computing systems by reusing Key-Value (KV) caches from similar tasks. Sim-LLM caches top-layer KVs of processed tasks and reuses them for subsequent similar tasks, reducing redundant computations. The system employs Locality-Sensitive Hashing (LSH) for efficient task similarity identification and a KV Manager to handle cache storage and sharing across edge servers.", "questions": "See weaknesses.", "limitations": "See weaknesses.", "rating": 4, "confidence": 2, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 3, "significance": 3, "originality": 3, "strengths_and_weaknesses": "Strengths:\n1) The paper reduces KV cache memory overhead by introducing task similarity-aware reuse, which is a direction underexplored in prior work.\n2) The paper is well-motivated and the observations are empirically compelling.\n3) The method demonstrates up to 34.65% memory reduction and 39.40% throughput improvement on A40/A100 GPUs.\n\nWeaknesses:\n1) The multi-node KV sharing mechanism assumes low-latency inter-server communication, which may not hold in real-world edge networks.\n2) While the cosine similarity threshold is justified empirically, it lacks a theoretical or intuitive explanation for why this value generalizes across diverse edge workloads.", "was_revised": true, "final_justification": "The authors provide detailed clarifications about the inter-server communication and the similarity threshold. I have no further questions and will keep my rating.", "rebuttal": "We would like to express our sincere gratitude to the reviewer for your time and the valuable feedback. In particular, your recognition of our work as “a novel method”，“a direction underexplored in prior work” and “well-motivated” aligns with the positive assessments provided by other reviewers. Below, we provide clarifications in response to your concerns. Please let us know if any further explanation is needed. **(\"W\" refers to \"Weakness\" , \"Q\" refers to \"Questions\")**\n\n---\n\n\n\n-   ## **Response to W1**\n\nWe would like to highlight that, in practice, many **edge computing environments** are already leveraging **high-speed communication links** between servers [1] [2] [3] [4] [5], including InfiniBand which is a high performance technology aiming to provide low-latency and high-bandwith. In our paper, the edge servers are connected via an InfiniBand network to facilitate the exchange of task prototypes and KV sharing.\n\nTo minimize communication overhead due to information exchange between edge servers, when similar tasks are identified on another edge server, only the sequence or the corresponding word embeddings are transmitted to the identified edge server, rather than transmitting the KV cache of the similar task to the current processing task from the identified server back to the original server. Furthermore, regarding the inter-server communication concerns you raised, we conducted supplementary quantitative experiments, as shown in **Table 1**. These results demonstrate that the communication overhead is almost negligible compared to the inference latency.\n\n| Node Num | Communication Latency (ms) | Inference Latency (ms) |\n| -------- | :------------------------: | :--------------------: |\n| 2        |            1.22            |         483.15         |\n| 4        |            2.73            |         337.84         |\n| 6        |            4.12            |         303.51         |\n| 8        |            8.71            |         272.19         |\n| 12       |           12.92            |         232.76         |\n\n**Table 1** Communication latency and inference latency with varying numbers of nodes.\n\n---\n\n\n\n-   ## **Response to W2**\n\nIn **Appendix B.1 Impact of** **Cosine Similarity** **Threshold**, we present experimental results illustrating how throughput, perplexity (ppl), and downstream task accuracy vary with different cosine similarity threshold. When the threshold is low, throughput performance exceeds that of a higher threshold. This is because most incoming tasks can easily identify \"similar\" tasks' KVs for reuse, introducing minimal identification overhead and accelerating the inference process. However, this gain comes at the cost of model accuracy. When the threshold is incresd to 0.9, there is a significant drop in generation speed due to fewer tasks being able to reuse KVs, resulting in many tasks performing inference as in the standard Transformer model. Therefore, as a trade-off between generation speed and model accuracy, a threshold of 0.8 is preferred.\n\nThe generalizability of this threshold is supported by the observation that many edge tasks share underlying semantic similarities. As noted in **Observation 2 (Section 1)**, KV caches generated by similar tasks are themselves similar, indicating that task features exhibit consistent patterns across different edge scenarios. The selected threshold captures these shared characteristics, making it robust across varied workloads. Moreover, it is resilient to small fluctuations in task features, which are common in dynamic edge environments.\n\nTo further validate the robustness of Sim-LLM, we conducted supplementary experiments using different thresholds (0.6, 0.7, 0.8, and 0.9) across four edge servers. The results, shown in **Table 2**, align with the trends observed in **Appendix B.4 Table 2**, demonstrating that Sim-LLM maintains consistent behavior under variable threshold settings. As expected, lower thresholds enable more KV reuse and faster inference, while higher thresholds reduce reuse and slow down inference. Nevertheless, due to the semantic consistency of tasks across servers, performance remains stable over a range of threshold values, confirming the adaptability of Sim-LLM in heterogeneous edge environments.\n\n| Seq. Length | Llama Throughput (token/s) | Different Threshold Throughput (token/s) | Uniform Threshold Throughput (token/s) |\n| :---------- | :------------------------: | :--------------------------------------: | :------------------------------------: |\n| 5+128       |           568.56           |           1280.78 **(2.25x)**            |          1364.50 **(2.40x)**           |\n| 5+512       |           354.53           |           3658.23 **(10.31x)**           |          3816.51 **(10.76x)**          |\n| 5+2043      |           140.88           |            422.88 **(3.00x)**            |           534.02 **(3.8x)**            |\n| 512+512     |           225.31           |            601.28 **(2.67x)**            |           678.35 **(3.0x)**            |\n| 512+4096    |           65.25            |            477.11 **(7.31x)**            |           522.53 **(8.0x)**            |\n\n**Table 2** Throughput comparison for different sequence lengths and thresholds.\n\n---\n\n\n\n## Refs\n\n[1] Learning-Aided Computation Offloading for Trusted Collaborative Mobile Edge Computing, IEEE TMC, 2020.\n\n[2] FlowStar: Fast Convergence Per-Flow State Accurate Congestion Control for InfiniBand. IEEE/ACM ToN, 2024.\n\n[3] Tutti: coupling 5G RAN and mobile edge computing for latency-critical video analytics. MobiCom, 2022.\n\n[4] C2DN: How to Harness Erasure Codes at the Edge for Efficient Content Delivery. NSDI, 2022.\n\n[5] Argus: Enabling Cross-Camera Collaboration for Video Analytics on Distributed Smart Cameras. IEEE TMC, 2025."}, {"review_id": "MYZgruSJZn", "reviewer": "Reviewer_NQsy", "summary": "This paper presents Sim-LLM, a novel approach for optimizing LLM inference on edge computing nodes by leveraging task similarity to reduce KV cache memory consumption. The key insight is that edge computing systems often process similar tasks due to geographical and temporal locality. The system caches KV pairs from previously processed tasks and reuses them for subsequent similar tasks, using cosine similarity and locality-sensitive hashing (LSH) for efficient similarity detection. The approach supports both single-node and multi-node scenarios with cross-server KV sharing.", "questions": "1. How does the approach scale when the number of cached tasks becomes very large? What are the computational and memory limits?\n2. Can the system adapt to changing task distributions over time, or does it require periodic retraining of similarity models?\n3. Are there any security implications of sharing KV caches across servers, particularly in edge computing contexts?\n4. Does temperature sampling—where later tokens drift semantically—break the assumption that early-task KV is reusable?\n5. What is the memory footprint of the global prototype table over a long time of edge traffic, and how is it pruned?", "limitations": "1. Only consider a coarse granularity of the presentations among layers; layer importance also depends on the input prompt.\n2. All servers must host identical weights because KV shapes must match.\n3. If an edge node holding popular KV crashes, the benefit disappears until caches warm up again.\n4. Performance is highly dependent on actual task similarity patterns, which may vary unpredictably.\n5. The title is for optimizing LLM inference on the edge, but no tests for a single edge device.", "rating": 5, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 3, "significance": 2, "originality": 2, "strengths_and_weaknesses": "Strengths\n1. The observation on the edge computing systems is well-founded and empirically validated across multiple datasets.\n2. The problem that this paper addressed is important.\n3. It is the first work that reduces the KV cache memory consumption by considering the task similarities.\n4. Extensive experiments to support the idea.\n\nWeaknesses\n1. The absence of experiments on genuine edge hardware limits the study’s relevance to practical edge-computing deployments.\n2. The core insight about task similarity in edge computing is straightforward.\n3. Algorithmic details (hash size, bucket eviction, sandwich cache policy) are scattered in text instead of one clear algorithm box.\n4. For the multi-node setting, prototypes are exchanged frequently. The paper does not specify eviction, aging, or memory-bounded maintenance policies, raising questions about scalability over long-running edge workloads.", "was_revised": true, "final_justification": "The authors clearly answered all my questions in the rebuttal. Their responses helped clarify the method and confirmed the validity of their results. I now believe the paper is solid and worth accepting.", "rebuttal": "We would like to express our sincere gratitude to the reviewer for your time and valuable feedback. We are especially thankful for your recognition of our work as “the first” in this direction and for highlighting our “well-founded observation.” Your recognition of the novelty of our work is aligns with the positive assessments provided by other reviewers. Below, we provide clarifications in response to your concerns. Please let us know if any further explanation is needed. **(\"W\" refers to \"Weakness\" , \"Q\" refers to \"Questions\")**\n\n---\n\n\n\n-   ## Response to W1\n\n  In practice, edge servers are typically deployed near users. While not as powerful as cloud servers, enterprise-grade edge deployments often use high-performance hardware. For instance, the NVIDIA Jetson AGX Orin offers 275 TOPS with a 12-core ARM CPU and 64GB DDR5 memory, and the Huawei Atlas 500 Pro provides up to 420 TOPS with a 24-core KunPeng 920 CPU, 128GB DDR4 memory, and 4 Atlas 300I GPUs. In our experiments, each edge server is configured with 4 NVIDIA A40 GPUs (299 TOPS), closely mirroring real-world setups and thus ensuring that our results reflect practical edge computing capabilities.\n\n---\n\n\n\n-   ## Response to W2\n\nAlthough it may seem intuitive that similar tasks share similar KVs, **this insight has not been experimentally validated before**. In edge environments, task similarity for inference is often unknown, and we are **the first to demonstrate it experimentally**. Moreover, while most related works optimize inference at the token or layer level, our approach focuses on task-wise KV reuse, distinguishing it from conventional methods. This task-wise reuse effectively extends prior concepts of lossless reuse to lossy reuse, as noted by **Reviewer G55H and vsRh**.\n\n---\n\n\n\n-   ## Response to W3\n\nTo address your concerns, we now provide the **algorithm box** for Sim-LLM. For each task batch, tasks are mapped to LSH buckets for similarity matching. If a match is found, the corresponding KV is reused to accelerate inference; otherwise, standard inference with the sandwich configuration is applied. The hash, embedding, and top-layer KV are stored in the **KV_Manager** with **LRU** **eviction policy**. After each batch, servers update and exchange task prototypes to maintain a global feature table.\n\n---\n**Input**:\n- Task batch `B = {t1, t2, ..., tn}`\n- Cosine similarity threshold `θ`\n- LSH bucket size `k`\n- KV_Manager (with cache size `C`, eviction policy: `LRU`)\n- bottom stage num `sandwich_bot`, top stage num `sandwich_top`\n\n**Output**:\n\n- Inference results with KV optimization\n---\n**Procedure**:\n1. **Preprocessing** (**Section 3.2**)\n\n    For each task `t∈B`:\n    - Tokenize input and generate embedding `e_t`\n        - Compute LSH hash `h_t ← LSH(e_t, k)`\n        - Assign `t` to bucket `LSH_Bucket[h_t]`\n2. **Task Similarity Identification** (**Section 3.2**)\n\n    For each task `t∈B`:\n    - Search `LSH_Bucket[h_t]` for candidate tasks `S`\n        - For each candidate `s∈S`:\n        - Compute cosine similarity `sim(e_t, e_s)`\n        - If `∃s∈S` such that `sim(e_t, e_s) ≥ θ`:\n            - Retrieve cached top-layer KV from `KV_Manager[s]`\n                - Assign `t.KV ← KV_Manager[s]`\n        - Else:\n            - Compute full KV for `t` from scratch\n3. **KV Cache Management** (**Section 3.3**)\n    For each processed task `t`:\n    - Store `(e_t, h_t, KV_t)` in `KV_Manager`\n        - If cache exceeds size `C`, apply eviction policy (LRU)\n4. **Edge Server Communication** (**Section 3.4**)\n\n    For task `t` with no local match:\n    - Compute task feature `f_t` and compare with prototypes in global feature table\n        - If remote feature match `f` found:\n            - Offload `t` to corresponding server\n    - Periodically update and sync task prototypes across servers\n5. **Inference Execution** (**Section 3.3&3.4**)\n    a. For each task `t∈B`:\n    - If `t.KV` exists:\n        - Run inference by reusing top-layer KV\n    - Else:\n        - Run inference with `sandwich`configuration, retaining bottom `sandwich_bot` layers and top `sandwich_top` layers KV.\n---\n-   ## Response to W4 \n\nEach server maintains a global feature table containing task prototypes from all edge servers. After each batch, servers update their own prototypes and share them to keep global tables synchronized. These prototypes serve as lightweight indexes for efficient cross-server task matching. The global table is only updated upon receiving new prototypes and does not involve eviction.\n\nIn contrast, the **KV_Manager** stores metadata for newly processed tasks and triggers **LRU** **eviction** when the cache limit is reached. We conduct new experiment to confirm the robustness of Sim-LLM. The detailed results can be found in our **response to Q3 for Review aP8N**.\n\n---\n\n\n\n-   ## Response to Q1\n\nWe conducted supplementary experiments to evaluate the latency and memory usage of Sim-LLM under a large cache size. The results are presented in **Table 1**.\n\nAs the number of cached tasks increases, memory consumption grows accordingly. To manage this, we implement an **LRU** **eviction policy**, which effectively removes less frequently used entries, avoiding memory overflow and maintaining cache relevance.\n\nAlthough a larger cache may introduce overhead in task matching and KV lookups, we mitigate this using **locality-sensitive hashing (LSH)** for efficient approximate matching. Additionally, the **global feature table** accelerates semantic similarity matching across edge servers, keeping overall latency low. For instance, even with a cache size of 4096, Sim-LLM achieves a latency of **45.35ms**, compared to **60.61ms** for LLaMA.\n\n|Cache Size|Latency (ms)| Memory (GB) |\n|---|---|---|\n|2048|33.13|14.98|\n|2560|36.77|16.91|\n|3584|41.97|22.87|\n|4096|45.35|25.50|\n\n**Table 1** Latency and memory usage under large cache sizes.\n\n---\n\n\n\n-   ## Response to Q2 \n\n(1)  We conducted new experiments to evaluate system performance under different request distributions—Poisson (uniform) and Power-law (bursty)—focusing on both perplexity and throughput. As shown in **Table 2**, Sim-LLM maintains high performance across varying task patterns. Under bursty workloads, similar tasks occur more frequently, enabling the LRU-based KV_Manager to retain and reuse KVs more effectively, thereby accelerating inference. These results demonstrate that Sim-LLM adapts well to dynamic task distributions.\n\n|Cache Size|Throughput (token/s) (Uniform \\| Bursty)| PPL (Uniform\\| Bursty) |\n|---|---|---|\n|256|545.03\\|435.78|11.19\\|11.21|\n|512|549.58\\|434.03|10.76\\|10.89|\n|1024|547.92\\|428.09|10.49\\|10.61|\n|2048|517.92\\|426.63|10.31\\|10.44|\n\n**Table 2** Performance under different task distributions.\n\n(2) **Our method does not require periodic retraining**, as Sim-LLM relies on cosine similarity combined with LSH for semantic matching, rather than a trainable similarity model. This design significantly accelerates inference without requiring costly architectural modifications and allows the system to dynamically adapt to new tasks, ensuring both efficiency and scalability.\n\n---\n\n\n\n-   ## Response to Q3\n\n Similar to many prior works such as [1] [2] [3], our approach assumes that all edge nodes are trustworthy for KV cache sharing. Nonetheless, we acknowledge the security challenges inherent in cross-server communication within edge computing environments. Malicious nodes could threaten data integrity and confidentiality. Existing research addresses these concerns, such as [4], which analyzes potential attacks and defense strategies at the edge; and [5], which enforces secure data sharing via access control mechanisms.\n\nWhile our work does not focus on securing cross-node communication, existing security solutions can be integrated to mitigate these risks. We consider security issues orthogonal to our core contributions—task acceleration and KV reuse—and thus not directly impacting the proposed methods.\n\n---\n\n\n\n-   ## Response to Q4\n\n Temperature sampling controls output randomness by scaling logits before selecting the next token. Higher temperatures increase randomness, while lower temperatures yield more deterministic outputs.\n\nAdjusting the temperature impacts only the quality of the tokens generated during the current step and does not alter previously generated tokens. Consequently, if the existing tokens match a stored similar task, the associated KV cache can still be reused. Therefore, the reusability of early-task KVs remains unaffected.\n\n---\n\n\n\n-   ## Response to Q5 \n\nThe global prototype table stores task prototypes that serve as local indexes for efficient cross-server task matching. Its memory usage depends only on the number of edge servers. To address your concern, we conducted supplementary experiments (**Table 3**) measuring memory consumption over time. Due to the inherent similarity of task features and fluctuating traffic in edge environments, prototype updates are minimal, and memory usage stabilizes. \n\nEach edge server maintains one such table. After processing each batch, a edge server updates its prototype and shares the update with other edge servers to keep their global prototype tables synchronized. Only when an edge server leaves the network, the item corresponding to the edge server will be pruned.\n\n|Time Slot|Memory Usage (KB)|\n|---|---|\n|100th|1688|\n|200th|1727|\n|300th|1692|\n|400th|1709|\n\n**Table 3** Memory usage over time.\n\n---\n\n\n\n## Refs\n\n[1] OL-MEDC: An Online Approach for Cost-Effective Data Caching in Mobile Edge Computing Systems. TMC, 2023.\n\n[2] Joint service placement and request routing in multi-cell mobile edge computing networks. INFOCOM, 2019.\n\n[3] Service routing in multi-tier edge computing: A matching game approach. JSAC, 2022.\n\n[4] Edge Computing Security: State of the Art and Challenges. Proceedings of the IEEE, 2019.\n\n[5] Everything Under Control: Secure Data Sharing Mechanism for Cloud-Edge Computing. TIFS, 2023."}, {"review_id": "ylYJ3zDW2R", "reviewer": "Reviewer_vsRh", "summary": "Sim-LLM is a novel framework exploiting the similarities across LLM inference requests to optimize the reuse of KV-cache.\nThe paper first identifies the similarity in requests across different inference tasks.\nSuch similarity allows the possibility of reusing previously generated KV-Cache.\nThe work also identifies the tradeoff between generation quality v.s reuse opportunity.", "questions": "This is a great work and I can also see great potential of this work in large-scale LLM infrastructure.\n\n1. While this work targets edge devices, large LLM services also face a similar problem (each LLM customer often issues many requests on similar topics). Is there anything that needs special handling if we want to apply SimLLM to large LLM clouds?\n\n2. How will the performance-accuracy tradeoff in Figure 11 look like in larger models? I believe this work should not be limited to small models and edge devices.\n\n3. Is it possible to further optimize the performance by combining SimLLM with works that exploit the KV-cache sparsity (e.g., the baseline designs you compared with), so that SimLLM reuses the KV cache and maintains only the KV-cache of important tokens. This should generate a comprehensive tradeoff space between throughput performance and accuracy, where users can pick different tradeoff points in different scenarios.", "limitations": "See questions, can this work apply to 1) large-scale LLM infras, 2) larger LLMs, or 3) other lossy KV-cache optimization techniques?", "rating": 5, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 4, "significance": 4, "originality": 3, "strengths_and_weaknesses": "Strength\n\nWhile KV reuse is a well-explored field, this work extends the lossless reuse in previous works to the lossy reuse in this work. This enables a new tradeoff between generation quality and KV reuse opportunity.\n\nVery clear motivation with quantified similarity across different tasks.\n\nSignificant performance benefits.\n\n\nWeakness\n\nThe accuracy loss seems a little high if we want more performance benefits. So is there anyway to achieve better tradeoff between accuracy and performance?", "was_revised": true, "final_justification": "All concerns well solved in the rebuttal.\nThe data presented in this paper (including appendix) and the rebuttal can well justify the motivation and effectiveness of the idea.\nThe presentation can still be somewhat improved (see final suggestion).\nOverall, I am very positive about the paper.\n\nI think it is totally fine to go for a simple idea instead of a complex one when possible, especially when this simple idea is useful and educative.", "rebuttal": "We deeply appreciate your recognition and careful review of our work — it truly means a lot to us. In response to your comments, we provide the following clarifications. Please let us know if any further clarification is needed.\n\n---\n\n\n\n-   ## Response to Weakness\n\n In our current design, we prioritized performance by optimizing **throughput** and **latency**, which can sometimes result in a slight **accuracy loss**. This trade-off was made in order to meet the strict **real-time** **processing** requirements of edge computing environments, where low latency and high throughput are often more critical than perfect accuracy.\n\nTo address this trade-off, we are considering a **hybrid model architecture** that assigns high-accuracy models to critical tasks while using more efficient models for less critical ones. This would allow the system to maintain performance without sacrificing essential accuracy. Additionally, to further improve accuracy without compromising performance, we are exploring lightweight **post-processing techniques** to refine initial outputs, aiming to recover accuracy without significant performance degradation.\n\n---\n\n\n\n-   ## Response to Q1\n\nOur motivation stems from the observation of task similarities commonly present in broad edge computing scenarios. In cloud environments, the larger scale of nodes and the greater diversity of tasks may pose challenges for identifying such task similarities. However, in specialized domains—such as medicine or law—where large language models are fine-tuned for domain-specific tasks, task similarity can still be high, making Sim-LLM fully applicable in these contexts. Moreover, the presence of high-speed data center networks in cloud infrastructures enables efficient inter-node communication. As a result, even without applying the communication optimizations proposed in Sim-LLM, the system still holds strong potential for performance gains when deployed in cloud settings.\n\n---\n\n\n\n-   ## Response to Q2\n\n Given our focus on edge scenarios, the primary model used in our experiments is Llama-7B. In response to your suggestion, we conducted **supplementary experiments with Llama-13B**, and the results are presented in the table below. As you anticipated, Sim-LLM maintains strong performance with the larger model, consistently outperforming smaller models such as Llama-7B and TinyLlama-1.1B. This observation aligns with the trends illustrated in **Figure 11 of Appendix B.1**. These findings further demonstrate the scalability and effectiveness of Sim-LLM when applied to larger language models.\n\n| Cos Threshold | Throughput(token/s) |  PPL  |  Acc  |\n| ------------- | :-----------------: | :---: | :---: |\n| 0.1           |       610.45        | 15.80 | 58.12 |\n| 0.2           |       608.32        | 15.72 | 58.45 |\n| 0.3           |       602.18        | 15.63 | 59.03 |\n| 0.4           |       595.67        | 15.38 | 60.21 |\n| 0.5           |       590.12        | 15.07 | 61.08 |\n| 0.6           |       585.44        | 14.54 | 61.32 |\n| 0.7           |       580.23        | 14.12 | 61.77 |\n| 0.8           |       575.89        | 13.81 | 62.54 |\n| 0.9           |       570.01        | 13.53 | 63.01 |\n\n---\n\n\n\n-   ## Response to Q3\n\nAs mentioned in *the third paragraph* of **Section 1** and in the discussion\"*To address the second challenge*\" in **Section 2**, **Sim-LLM is inherently compatible with intra-layer** **KV** **optimization** **techniques**. While existing methods primarily focus on KV optimization from a layer-wise or token-wise perspective, our work takes a task-wise approach. These perspectives are orthogonal, and therefore complementary. A promising direction for future work is to explore the integration of Sim-LLM with these fine-grained KV optimization strategies to further reduce memory consumption and mitigate accuracy degradation in edge environments."}]

Author rebuttal:

Meta-review: The paper studies a very important topic - reducing memory consumption for the edge LLM inference. 
The reviewers engaged in fruitful discussion, with most of their concerns being addressed. The final scores are somewhat mixed (5,4,4,3), but with the majority of reviewers leaning towards acceptance. 
I have read the paper carefully, and while I agree with some of the concerns raised by the reviewer aP8N, I think that the NeurIPS community will benefit from this paper being accepted. I recommend to add carefull explanantions in the final version to address aP8N's and other reviwers concerns.
Decision: Accept (poster)