A Survey on Large Language Model Acceleration based on KV Cache Management
Authors: Haoyang LI
Yiming Li
Anxin Tian
Tianhao Tang
Zhanchao Xu
Xuejia Chen
Nicole HU
Wei Dong
Li Qing
Lei Chen
Venue: Accepted by TMLR ()

Abstract: Large Language Models (LLMs) have revolutionized a wide range of domains such as natural language processing, computer vision, and multi-modal tasks due to their ability to comprehend context and perform logical reasoning. However, the computational and memory demands of LLMs, particularly during inference, pose significant challenges when scaling them to real-world, long-context, and real-time applications. Key-Value (KV) cache management has emerged as a critical optimization technique for accelerating LLM inference by reducing redundant computations and improving memory utilization. This survey provides a comprehensive overview of KV cache management strategies for LLM acceleration, categorizing them into token-level, model-level, and system-level optimizations. 
Token-level strategies include KV cache selection, budget allocation, merging, quantization, and low-rank decomposition, while model-level optimizations focus on architectural innovations and attention mechanisms to enhance KV reuse. System-level approaches address memory management, scheduling, and hardware-aware designs to improve efficiency across diverse computing environments. 
Additionally, the survey provides an overview of both text and multimodal datasets and benchmarks used to evaluate these strategies. By presenting detailed taxonomies and comparative analyses, this work aims to offer useful insights for researchers and practitioners to support the development of efficient and scalable KV cache management techniques, contributing to the practical deployment of LLMs in real-world applications.

Published in Transactions on Machine Learning Research (05/2025)
A Survey on Large Language Model Acceleration based on KV Cache Management
Haoyang Li
Yiming Li
Anxin Tian
Tianhao Tang²
Zhanchao Xu
Xuejia
Chen1.3 Nicole Hu
Wei Dong
Qing Li
Lei Chen
saoyang-ccom..1qpoll edu.hk
atinxx,tti uuteectt..tt.it ust.hk @resham15437@@mail.ccm
ttangae@ese ust.hk hulan6link. cuh.llkkikkckkk..... edu.
zhanchaoxu0228@mai..com
wei_dongentu. edu.sg qing-prot www.opyppl....mmmmmmmm edu.hk leichen@cse ust hk
'he Hong Kong Polytechnic University The Hong Kong University of Science and Technology HHuazhong University of Science and Technology 4The Chinese University of Hong Kong "Nanyang Technological University
Reviewed on OpenReview:
Abstract
Large Language Models (LLMs) have revolutionized wide range of domains such as natural language processing, computer vision, and multi-modal tasks due to their ability to comprehend context and perform logical reasoning. However, the computational and memory demands of LLMs, particularly during inference, pose significant challenges when scaling them to real-world, long-context and real-time applications. Key-Valut (KV) cache management has emerged as a critical optimization technique for accelerating LLM inference by reducing redundant computations and improving memory utilization. This survey provides comprehensive overview of KV cache management strategies for LLM celeration, categorizing them into token-level, model-level, and system-level optimizations. Token-level strategics include KV cache selection, budget allocation, merging, quantization, and low-rank decomposition, while model-level optimizations focus on architectural inno- vations and attention mechanisms to enhance KV reuse. System-level approaches address memory management, scheduling, and hardwarc-aware designs improve efficiency across diverse computing environments. Additionally, the survey provides an overview of both text. and multi-modal datasets and benchmarks used to evaluate these strategies. By presenting detailed taxonomies and comparative analyses, this work aims to offer useful insights for rescarchers and practitioners to support the development of efficient and scalable KV cache management. techniques, contributing lo the practical deployment of LLMs in real-world applications. The curated paper list for KV cache management in: TTree/TreeeAaaaa/ Lae--Laab/wwweeee... Awesome- K CaaCCah--Maaaggnneee
1 Introduction
Large Language Models (LLMs) (Hadi al., 2023 Zhu et al. 2023), trained on massive corpora, have
revolutionized various domains such natural language processing (Naveed et al. 2023 Min al.) 2024 Xu et al.) 2024a), computer vision (Liu et al.) 2023a Zhang al. 2024c Berrios et al. 2023). and multi-modal (Zhang et al. 2024a; Cui et al.) 2024 Wu et al. 2023) tasks. Their ability to understand
context and perform logical reasoning has enabled remarkkble success in various fields, such as time series analysis (Jin el. al.) 2023 Ma et al., 2024a). recommendation (Tan & Jiang 2023 Wu et. al. 2024c).
autonomous driving Yang et al.] 2023 Chen et al. 2024b Fu et al.) (024b), and healthcare (Qiu et al.
2023 Zhou ct al. 2023b). These breakthroughs are powered by statc-of-thh-art architectures and training
"ooreepponingg Author.

---

Published in Transactions on Machine Learning Research (05/2025)
paradigms, enabling models to achieve unparalleled performance across diverse tasks. Prominent LLMs. such as GPT (Brown et al.
2020
Radford et al. 2018 2019), L.Laaa (Touvron et al. 2023 Dubey et al.
2024), DeepSeek (Dai et al.
2024 DeepSeek-A Al et al. 2024 Lu et al. 2024), Mistral (Jiang et al.) 2024a
and GLM (Zeng et al. 2023; Du et al. 2022). are built on the foundational transformer architecture (Vaswani
et al. 2017), which excels al capluring long-range dependencies in sequential data. However, despite their
powerful capabilities, the computational and memory demands of LLMs, particularly during inference, present significant challenges when scaling them to real-world, long-context, and real-time applications.
A critical bottleneck in LLM inference lies in the efficient management of Key-Value (KV) pairs. Recently, caching techniques (Gracioli et al. 2015 Podlipnig & Böszörmenyi 2003) have been extensively employed to store previously computed intermediate results, allowing their reuse in subsequent inference steps to accelerate the model, such as graph neural networks (Li & Chen 2021 Li et al.) 2023c Lin et al.) 2020). Fortunately, the auto-regressive generation mechanism inherent to LLMs presents an opportunity to leverage KV caching for efficient text generation. Spocifically, auto-regressive generation enables LLMs to produce text token by token, with each token conditioned on all previously generated ones. While this approach is highly effective for generating coherent and contextually relevant outputs, it struggles with poor scalability for long input sequences. This limitation ariscs because LLMs must compute attention values for every pair of tokens, causing the time and space complexity of the attention matrix to grow quadratically with sequence length. To address this issue, the KV cache mechanism stores the key and value matrices from previous decoding steps, allowing them to be reused. This siguificantly reduces redundant computations, as the model only computes attention values between new tokens and previously processed tokens, avoiding the need to recompute all attention values.
Several recent surveys (Zhu et al.
Zhuang et al. 2023 Park et al. 2024 Wang et al. 2024b Ding Wan et al.) 2023 Zhou et al. 2024c] Tang et al. 2024c Kachris
2023
et al., 2023 Miao et al., 2023
2024
Xu et al.) 2023 Albalak et al., 2024 Zefan- Cai 2024) have explored the domain of efficient LLMs.
These
surveys primarily examine various aspocts of LLM efficiency, presenting valuable insights while leaving room for further refinement and innovation. In particular, these works primarily focus on holistic approaches to improving LLM efficiency, examining a wide range of techniques across multiple dimensions, such as data-level optimizations (e.g., prompt engincering), model architecture-level optimizations (e.g., efficient transformer designs), and system-level optimizations (e.g., task scheduling). For instance, Ding et al. (0023) explore efficiency techniques that integrate data-level and model architecture perspectives, while Miao et al. (2023) examine efficient LLM inference from a comprehensive system-level perspective. Similarly, Tang ct al. 2024c), Wan et al.] (002)), and Xu et al. (2023) provide analyses that encompass data, model, and system-level optimizations, rellecting holistic approaches to LLM acceleration.
On the other hand, some surveys focus on more specialized aspects for LLM acceleration. For example, Zhu et al.) 2023). Park et al. (2024), Wang et al. (2024b), and Tang et al. (024c) focus on model compression as key aspect of model-level optimization. Similarly, Kachris (2024) examines hardware acceleration strategies tailored for LLMs, while Xu et al. (2023) investigates paramcter-effiiicnt tuning approaches. Albalak et al. (2024) discusses data selection stralegies to enhance the efficiency of LLM training, and Xia et al. (2024) highlights collaborative techniques, such as speculative decoding (Leviathan et al., 2023 Kim et al. 2024b), to accelerate model inference. Li et al. (2024c) focuses on prompt compression. Similar to our work, Shi ct al. (2024). Li et al.] (2024a), and Yuan et al. (2024) also explore the use of KV caches to accelerate LLMs. However, our survey both complementary and more comprehensive, offering a detailed laxonomy of KV cache management for text-based and multi-modal LLMs. We categorize techniques into leken-level, model-level, and system-level perspectives and include benchmarks for both text and multi-modal scenarios. In particular, complementing existing KV cache surveys, we provide a detailed comparison of the differences and advantages of existing models at the token-level, model-level, and system-level.
Specifically, this survey provides a comprehensive overview of the current state of KV cache management and its role in accelerating LLM inference. We begin by introducing the transformer architecture and the role of the KV cache in enabling efficient auto-regressive text generation. We then analyze the challenges associated with KV cache management, including its impact computational complexity, memory usage, and real-time performance. Following this, we present a taxonomy of existing optimization techniques, categorizing them into token-level, model-level, and system-level optimization approaches. Additionally, we
2

---

Published in Transactions on Machine Learning Research (05/2025)
Table 1: Notation Summary
Symbol
Definition
X
Input sequence of tokens
X
Dense representations of X
dg
Dimensionality of the imput embeddings.
Embedding matrix E C R Revecab
PE(X)
Positional encoding
Q,, K1, V.
Query, Key, and Value matrices
de, d,
Query/Key and Value dimension
Wo
Wx., Wv.
Weight matrices for computing K., V..
Zi
Self-attention Output
Wo
Weight malrix
W1, W2
Weight matrices
b1, bg
Bias vectors
Sequence length index
to
Number of tokens stored in the KV cache.
K,
V
Key and Value at step
K
V
Cached Key and Value
Number of attention heads per layer
Number of transformer layers
P(1++1 "1,"" ,2))
Conditional probability
discuss datasets and evaluation metrics used to benchmark these techniques and provide insights into their effectiveness across various tasks and applications.
2
Preliminary
Large language models (TLMss), pretrained vast. corpora, have demonstrated superior capabilities in context understanding and logical reasoning. These models have achieved remarkable success across wide range of tasks in various domains, including natural language processing (Navoed et al. 2023 Min et al.
2024 Xu et al., 2024a) and computer vision (Liu et al. 2023a Zhang et al. 2024c Berrios et al. 2023). Mainstream LLMs, such as GPT (Bubeck et al. 2002).. Llama (Touvron al. 2023 and DeepSeek (Dai et al.) )024), are primarily built on the transformer architecture (Vaswani et al. 20177. To explore the role of Key- Value (KV) cache management in accelerating LLM computations, first outline the core components
of the transformer model and then introduce the mechanisms for managing the KV cache to accelerate the LLMs. Important notations in this survey are summarized in Tab. 1
2.1 Transformer Architecture
Transformers Vaswani et al. 2017) have become the backbone of LLMs due to their ability to efficiently capture long-range dependencies in sequential data, such as text. This capability makes them particularly well-suited for tasks like machine translation, text generation, and image captioning. The transformer architecture follows an oncoder-decodor structure, where most LLMs utilize only the docoder component. We first introduce the core components of the Transformer decoder and then describe the critical auto- regressive generation mechanism. Particularly, we do not describe certain components in the transformer, such normalization, they do not impact the understanding of KV cache management.
2.1.1 Transformer Decoder
As shown Fig. Ia pecoder-basedd transformer architecture composed of multiple stacked Transformer blocks, cach designed process soquential data effectively. Typically, a Transformer block consists of two core components, i.c., Multi-Heal Self-Attente Attention (MHSA) mechanism and a Feed Forward Network (FFN). These blocks are arranged sequentially, where the output of one block is passed input to the next. This
3

---

Published in Transactions on Machine Learning Research (05/2025)
Output Probabilties
Linbar Softmax
Multi-head Attention
Add
Feed Forward
Concat
Add & Norm
Mult-Head Attention
Scaled Product Attention
Add & Norm
Linear
Mult--Head Attention
Full Attention
Sparse Attention
Postional Encoding
Input Embedding
Sequential Imputs
(a) The decoder-only Transformer for LLMs.
(b) The sparsity of attention matrix.
Figure 1: An architecture of transformer and two attention matrices.
iterative design allows the model to refine its understanding of the input sequence progressively, making it highly effective for tasks such as text generation and language modeling.
Positional Encoding. Before the input sequence is processed by the Transformer blocks, it undergoes a preprocessing phase. First, a tokenizer processes the input sentence X by splitting it into discrete units, such as words subwords. The resulting sequence can be represented as X (2,, ; These tokens are then mapped to dense vector representations using an embedding layer, i.e. X Ixxx where Ix E {0, 1) xaxdvacab represents the one-hot vector of tokenized input X, E Rdhocas x the embedding matrix, and X [x1, X2, , Raxd, the resulting mairix of embedded token representations. Since the Transformer architecture does not inherently account for the order of tokens in sequence, positional encodings are added to the token embeddings X to incorporate positional information. This can be ex- pressed as X = = x + PEXX), where PE(X) E Raxds represents function (Zhao et al. 2023 Zheng et al. 2021 Su et al. 2024) (e.g,, RoPF (Su et al. 2024 ) that generates positional embeddings for the input X. Note that relative positional embeddings, such as RoPE (Su et al. (024) (Rotary Positional Embedding). differ significantly from absolute positional encoding. RoPE introduces positional information each layer of the model through rotational transformations.
Transformer Block. Once the input features are prepared, they are passed through series of stacked Transformer blocks. Each block begins with the Multi-Head Self-Attention (MHSA) mechanism, which captures both local and global dependencies. For each token, the self-attention mechanism computes a weighted sum over all other tokens in the sequence, where the weights are dcrived from the similarity between the tokens. Particularly, since the operations within each transformer block are identical, we use a single transformer block as an example. Specifically, given the input to a block, denoted as X E I g xd the MHSA mechanism computes the query vectors Q E [ key vectors K; R [X|x,, and value vectors V, RlXid,, These vectors obtained through learned linear transformations as follows:
Q: = XWW K V, = =W
(1)
where Wo, Xde, WK, Xde self-attention operation is applied to each triple KQ Vi), and obtains the output of the i-th attention head Z; follows:
Xde and Wv. Ed are the learned weight parameters. Then, the
Q.K. Vdz
2, = Attention Q. V,) = -Sftmmx
V
(2)
4

---

Published in Transactions on Machine Learning Research (05/2025)
where vd is scaling factor to ensure numerical stability. To capture diverse relationships, multiple attention heads with h heads are applied to X in parallel, and their outputs are concatenated with one transformation as follows:
Z = Concat (Z, Zz...., Wo.
(3)
where Concat is the concatenation operation and Wo E gd.xd. are the trainable parameters.
Following the self-attention mechanism, the output is passed through Feed Forward Network (FFN). The FFN is fully connected neural network that applics two linear transformations separated by nonlinear activation function o(-) (e.g,, ReLU (Agarap 2018)):
bb) PFN(Z) = (ZW, + W2 + b2
(4)
where W1 E xd and W2 E RRd xd2 are two parameters, b1 E I&d and by E Rda are two bias vectors.
2.1.2 Augo-segressive Generation Mechanism
LLMs employ autoregressive mechanism to generate text token by token, with each token conditioned the previously generated ones. This iterative process ensures that the output sequence remains coherent and contextually appropriate. Formally, given an input sequence of tokens X = I2, ... ,z), the model predicts the next token T++1 at each decoding step t by modeling the conditional probability distribution as follows:
P(1r+1/21,22 #2, 24, = Softmax(h, W out + bout).
(5)
where h, E Rdh represents the hidden state of the LLM regarding X at step t. W E x is the output projection matrix, and bout is the bias vector. The softmax function converts the logits into a probability distribution ion over the vocabulary. Then, at each decoding step, the model generates the next token Z141 by sampling from the predicted probability distribution:
T++1 P(
... ;.a).
(6)
The generated token r++1 is then appended to the sequence X = [.1,... Tt, 2t41), and the process continues until a special end-of-sequence (EOS) token is generated or a predefined maximum length is reached.
2.2 Key-Value Cache in Transformer Models
ruto-regressvve generation is powerful mechanism that enables LLMs to produce high-quality, contextually coherent text. However, it presents computational challenges for long sequences, as the Keys and Values need to be recomputed for each token during the generation process. The KV cache optimization addresses this issue by storing the previously computed Keys and Values and reusing them for subsequent token generation, thereby reducing redundant computations and improving inference efficiency.
2.2.1 Auto-regressive Generation with KV Cache
Here, we describe how caching KV pairs of tokens accelerates LLM inference. Specifically, at cach decoding step i, the model performs self-attention over the entire sequence X = [= , 2.]] to generate the next token 2++1- This process requires the computation of Keys and Values matrices for all previously processed tokens in X (FI,....,zz)) Notably, when generating the token It, the LLM has already computed the Keys and Values for the tokens in X[1 : t 1] = [x1,-.. 21- [.,.-..,2-11) The KV cache optimizes this process by storing the previously computed Keys and Values matrices for X(1 L. - 1] and reusing them, thereby only requiring the computation of Keys and Values for the new token Zy. This significantly improves efficiency by climinating rodundant computations.
Formally, at each decoding step I, the new token embedding x, is used to compute the query vector q key vector k,, and value vector v as follows:
q = xx k xx v =xx, WV.
(7)
5

---

Published in Transactions on Machine Learning Research (05/2025)
The newly computed k/ and v are then appended to the cached key and value matrices from previous steps:
K Concat (K: kQ), V = Concat (V:-) Vi).
(8)
where R E R'- and { E R'-Ixd, represent the cached key and value matriccs of tokens in X[1 : - 1]. These cached matrices are then used in the scaled dot-produc attention computation for the token 24- The attention output z for the token Iy at step is calculated as:
Softmax
V
(9)
Vd
Then, similar KV reuse process can be applied to different attention heads in each layer of the LLM.
2.2.2 Time and Space Complexity Analysis
Given a transformer-basee L-layer LLM with attention heads per layer and an input sequence of length X = ..,,,... .ee,, we analyze the time saved and the space required to store cached KV pairs. For simplicity, assume that the keys and values of the tokens are stored for all heads across all LLM layers.
Computational Complexity. For cach token, the saved computation time comes from avoiding the re- peated computation of Keys and Values in Equation [ the self-attention result in Equation 2 and the linear transformation in Equation g We omit the time analysis on operations transformer that do not affect the understanding of KV cache acceleration, such as layer norm and position encoding.
QKV Computation. The time of computing Queries, Keys and Values for each token in Equation | is A1 O(2ddda + dgde).
Self-attention Result. The time complexity of computing each attention score z, in Equation 2 takes O(t(d, + d,)).
Linear Transformation. To merge the attention results in Equation 3 the time is A2 = O(hd, + dd,d)).
Therefore, for te cached tokens across h attention heads and L layers, the total saved computation time is:
O (L h to t- (d+ + do) + L h to (A1 + A2))
(10)
Thus, the saved time is directly proportional to the number of cached tokens te, significantly accelerating model computation, especially for longer sequences (when / is large).
Space Complexity, Compared to computation without caching, additional space required to store the cached KV pairs for f, tokens across h attention heads and L layers. Assuming each Key and Value is stored in Float16 precision, the total extra space needed can be expressed as:
O h le (da + dw) sizeof (Floa/16))
(11)
Thus, for the same LLM model, the extra space required to store the KV pairs primarily depends on the num- ber of cached tokens and the precision of the cached Keys and Values. To address this, existing approaches explore various techniques to reduce the extra space consumption, such as caching only the most important Keys and Values or applying quantization techniques lower the bit precision of the stored Keys and Values.
2.3 Challenges in KV Cache Management
As analyzed in Sec. 2.2.2, reusing cached KV pairs enables the LLM to avoid recomputing past tokens, resulting significant speedups during inference. However, as sequence lengths grow, the size of the KV cache increases proportionally, placing significant pressure on memory. Consequently, becomes challenging to manage this cache effectively to accelerate LLM computation without excessive space usage.

---

Published in Transactions on Machine Learning Research (05/2025)
Cache Eviction Policies: Determining which items to evict when the cache reaches its capacity is complex problem. Popular policies (Podlipnig & Böszörmenyi 2003) like Least Recently Used (LRU) or Least Frequently Used (LFU) do not align with LLM patterns, leading to suboptimal performance.
The memory required for the KV cache grows linearly with both the sequence length and the number of layers, which can quickly exceed the hardware memory limits, especially for long sequences. Consequently, managing the collaboration between different types of storage hardware (e.g., GPU, CPU, or external memory) becomes a significant challenge.
Memory Management:
Latency Bottlenccks:
Accessing and updating the cache at each docoding step can introduce latency, particularly for hardware with limited memory bandwidth.
Compression Tradc-offs: Compressing the KV cache can reduce memory usage but may degrade model performance if key information is lost.
Handling dynamic and unpredictable workloads, where access patterns and data requirements frequently change, requires adaptive caching strategies that can respond in real time.
Dynamic Workloads:
Distributed Coordination: In distributed KV caches, maintaining coordination across multiple nodes to ensure consistency, fault tolerance, and efficient resource usage adds significant complexity.
3 Taxonomy
In the above sections, we analyzed how the number of cached Key-Value (KV) pairs significantly impacts both the computation time and the additional memory required during inference. Efficient KV cache management is critical to balancing performance improvements and resource utilization, especially as sequence lengths and model sizes continue to grow. After carefully reviewing existing approaches, we categorize KV cache optimization strategies into three levels: token-level optimization, model-level optimization, and system-level optimization. Each level addresses specific aspects of the challenges associated with KV cache management and offers distinct techniques to enhance efficiency. The detailed taxonomy is illustrated in Fig.
Toven-Level Optimization refers to improving KV cache management efficiency by focusing on the fino-grained selection, organization, and compression the token level, requiring no architectural changes to the original model. While KV cache selection (Sec. 4.1b focuses on prioritizing and storing only the most relevant tokens, KV cache budget allocation (Sec. 4.2 dynamically distributes memory resources across tokens to ensure efficient cache utilization under limited memory. Furthermore, KV cache merging (Sec. 4.3 reduces redundancy by combining similar or overlapping KV pairs, while KV Cache Quantization (Sec. 4.4 minimizes the memory footprint by reducing the precision of cached KV pairs. Finally, KV cache low-rank decomposition (Sec. 4.5 uses low-rank decomposition techniques to reduce cache size.
Model-level Optimization refers to designing an efficient model structure to optimize KV cache man- agement. This can further refer to several stratogies: Attention grouping and sharing (Sec. 5.1 methods examine the redundant functionality of keys and values and group and share KV cache within or across transformer layers. Architecture alterations (Sec. 5.2 emerge to design new attention mechanisms con- struct extrinsic modules for KV optimization. Furthermore, there are also works designing or combining non-transformer architectures (Sec. 5.3) that adopt other memory-efficient designs like recurrent neural networks to optimize the KV cache in traditional transformers.
System-level Optimization refers to optimizing the KV Cache management through two classic low- level aspects: memory management (Sec. 6.1 and scheduling (Sec. 6.2). While memory management techniques focusing on architectural innovations like virtual memory adaptation, intelligent prefix sharing, and layer-aware resource allocation, scheduling strategies have evolved address diverse optimization goals through prefix-aware methods for maximizing cache reuse, preemptive techniques for fair context switching, and layer-specific mechanisms for fine-grained cache control. In addition, we provide detailed introduction for hardware accelerator design in Sec. 0.3] including single/multi-GGPt, 1/O-based solutions, heterogeneous computing and SSD-based solutions.

---

Published in Transactions on Machine Learning Research (05/2025)
Static KV Cache Selection (Scc 4.1.1
Dynamic Selection with Per- manent Eviction (Sec 1.1.2
KV Cache
Selection (Sec. 4.1
Dynamic Selection without Per- manent Eviction (Sec 1.3.3
Layer-wise Budget Allocation (Sec 4.2.
KV Cache Budget
Allocation (Sec. 4.2
Head-wise Budget Allocation (Sec 4.2.2)
Intra-layer Merging (Sec 4.3.1)
Token-level Opti- mixation (Sec.
KV Cache
Merging (Soc.
4.3
Cross-layer Merging (Sec 4.3.2
Fixed-precision Quantization (Soc
4.4.1
KV Cache
Mixed sexeedprreisson Quantization (Sec (.4.2)
Quantization (Sec.
4.4
Outlier Redistribution (Sec 4.4.3)
KV Cache Management for Large Language Models
Singular Value Decomposition (Sec
4.5.1
KV Cache Low-rank
Tensor Decomposition (Sec 4.5.2
Decomposition (Sec. 4.5
Learned Low-rank Approximation (Sec 4.5.3
Intra-Layer Crouping (Sec. 5.1.1
Attention Grouping and Sharing (Sec. 5.1
Cross-Layer Sharing (Sec.
6.1.2
Enhanced Attention (Scc. 5 1)
Model-level Opti-
Architecture Al- teration (Sec. 5 2
mization [Sec. [
Augmented Architecture (Sec. 5.2.2
Adaptive Sequence Process- ing Architecture (Sec. (.3.1)
con-traassfrrmr Ar- chitecture (Soc. 5.3
Hybrid Architecture (Sec.
5.3.2
Architectural Design (See.
6.1.
Memory Management (Sec. 6.1
Prefix-aware Design (Sec. 6.1.2 2
Prefix-aware Scheduling (Sec. 6.2.
Preemptive and Fairness- oriented Scheduling (Sec. 6.2.2
Scheduling
(Sec. 6.2
System-level Opti- mization (Sec. ?
Layer-specific and Hierarchi- cal Scheduling (Sec. (.2.3)
Single/ Single/Multiinn MMultiGGPU Design (Sec. 6.3.1
1/O-based Design (Sec. 6.3.2
Hardware-aware Design
(Sec. 6.3
Heterogencous Design (Sec.
6.3.3
SSDD-based Dessign (Sec.
6.3.4
Figure 2: Taxonomy of KV Cache Management for Large Language Models.
4 Token-level Optimization
At the token level, optimization focuses exclusively on improving the KV cache management based on the characteristics and patterns of the KV pairs of tokens, without considcring enhancements from model architecture improvements or system parallelization techniques. Generally, token-level optimization methods are guided by observations from L.LSs and sequential inputs. Existing approaches can be categorized into
8

---

Published in Transactions on Machine Learning Research (05/2025)
Statie Cacha lnction (Sec 111
Pastiiin GG at
2024 Snapky
2024b)
Compar (Davto at
Micanoon-aak Zeng ci mi
Selection (Sec.
KV Cache
Dynamic Selection with Per- manent Eviction (Soc 112
2024g
H20
Zhang ci n.. 20235
MXCL lin
BUZZ
20244
Zhao
Sciy
SmpLEM
Salection Permuzcat Eviction (Sec
Inf
202ta)
Squeesee SguseseAAsttaainn Attcation
(Hoppu
Retrieval Atterttion
20240 Bb.
LLM Loanto
Lis
ml 20240 Lokki
2024
Layer-wiie Budget Al- location (Sec 4:2.1
PyramidKV (Gat et al
20:4 Pyramidia-
fer [Yang et al 2025m
PynamicKy Anony
moil
(0/21b
Prelick
[Wanz mi
2024
KV Cache Rudgge Al- location (See 1.2
SumbayerX
Zhang ct a.
20240
Head-wise Budget Al- location (Sec 4.2.2
AdaKV [aang at al
2024 Criti-
calKy
Anorysoun
20124d
LeanKV (Chaoe
RE
Razor
Atrention
Fang
At
30210 DR0AA
sansion
Xiao at All
Intra- layer
CCM
Kil
2024a)
LoMA WWang
Merging (See
Xino
20/4)
Nuwwes B.
Kanag c1 6..
2024x
520
KV Cache Marging (Sec.
Cross-layer Merging (Sec 4.3.2
MiniCache Liu ct
2024n KVSharer
Yong
To loveleve! Opsinization
Pised-preetttoon Quan-
ZeroQuant Yan al
2022 P Flaeßen Shenng
stration (Sec 1.1
ml 2023
Kandin
2024
Cache
Mhang
2021b
Mixeci-precising Quan- sieation (See 442
KVQuant (Houper et 2024b Intoc-
JKV
Lis el
20y33
Duanmn
kt al
20224)
K.V VT Lath
2024n
Quantiaation (See. 4.4
KV Cache
ee ml
WKV
ZIPV
2623
[Ie et al 2021.) Z.puiv
Yue at al (2IS GEAl Rang
MIKY Yaang at sl
20240
2024h)
Preneiuam- (Chen
20%
[harma
C1
VVV Quaal
Harizi
Outlier Rediztris
Musaivoksivvation (Bn
2024b)
(Sed
Qualtet
2024b
2023)
+
Wel
Alliint
mi
20240
Buu
XW0
cl
2024
Ommit
Slas
65 ml
2023
Singular Value Decom- position (Sec 4.5.1)
ECKVH
Yu et. al.
2024 Elgen Atten-
sinn Sagenn af X
MPTNN ZDC
Zhang 4-
Shen
(022)
Zhanz at
(Su Cl
4024m
mang
66 al
2023
( by el al
Lokt
Sipgaanna at
KV Cache Low-rank De- comporition (Sec (3))
Tensor Decompacttion ecc 4.5.2
DecoQuant (Lim al
2024c
Learned Low-rank Ap- prostmation (Sec
LE3S Dong
2024m M
tryochean
Figure 3: Taxonomy of the Token-level Optimization for KV Cache Management.
five main types: KV cache selection, KV cache budget allocation, KV cache merging, KV cache quantization, and KV cache low-rank decomposition. The taxonomy of token-level optimization is shown in Fig. B
4.1 KV Cache Selection
9

---

Published in Transactions on Machine Learning Research (05/2025)
Table 2: Comparison of KV cache selection strategies.
Initial
Top-k
Recent
Permanent
Dynamic
Selection
Method
Remark
tokens
tokens
tokens
eviction
selection
granularity
Fast Gen
V
token
five attention structures
Ge al. 2024
Snapk V
d
token
observation window-basee
[1
2024b
L.2Compress
token
Le norm- comm-based importance
Devoto et
2024
Attention-
ate
token
learned eviction policy
Zeng et al.
2024
Streaming
LLM
4
token
initial and recent tokens
Xiao ct al.
2024c
LM-Infinite
4
token
distance ceiling
Han ct al.
2024)
H20
/
V
token
accmulative attention score
Zhang et. al.
2023b
BUZZ
4
token
beehive-like structure
Zhao 2024a
Scissorhands
token
persistence of importance
Liu 2023c
NACL
token
diversified random eviction
Chen 2024d
Keyformer
4
V
token
gumbel logit adjustment
Adnan et al.
2024
InfLLM
V
block
block-level KV management
Xiao et al.
2024a
Quest
V
block
new block representation
Tang et al.
2024b;
PQCache
block
product quantization
Zhang et al.
2024b)
Atueezed/Ateettiinee
cluster
hicrarchical clusters
Hooper al.
2024a
IettrievalAttettiiee
token ANN search
LLu 2024b)
EM-LLM
event
episodic events
Tountas al.
2024
Spar Q
token low-dimensional retrieval
Ribar et al.
2024)
InfiniGen
token
asyuchronous prefetching
LLx et al.
2024
RecycledAAtentin
V
token
periodic top-k selection
Xu et al.
2b24b)
MagicPIG
token
Local Sensitive Hash
Chen et al.
2024g)
As shown in Fig. 1b the attention matrix is sparse. KV cache selection mechanisms have emerged as a critical optimization strategy, aimed at reducing memory utilization KV caches, minimizing inference latency, and enhancing overall throughput in large language models. These optimization objectives have driven the development of various selection methodologies, which can be classified into two distinct categories: (1) static KV cache selection, which performs token filtering exclusively during the prefilling phase, with selected tokens remaining fixed throughout subsequent decoding steps; and (2) dynamic KV cache selection, which continuously updates KV cache during the decoding phase, enabling adaptive cache management. In dynamic KV cache selection approaches, KV cache tokens that are not selected may be permanently evicted or offloaded to hierarchical caching devices such as CPU memory, implementing a multi-tier storage strategy. Given that real-time KV cache seloction during decoding may incur substantial computational overhead, several studies have focused on developing optimized retrieval algorithms to enhance the efficiency of this process. These optimizations include block-level retrieval instead of token-level granularity reduce search
10

---

Published in Transactions on Machine Learning Research (05/2025)
complexity, asynchronous query mechanisms to hide latency, and parallel retrieval pipelines to accelerate the selection process. These optimization efforts aim to mitigate the computational burden while maintaining the effectiveness of token selection. The summary of the KV cache selection is listed in Tab. 2
4.1.1 Static KV Cache Selection
Static KV cache selection methods perform one-time compression the KV Cache immediately after the prefilling phase is completed. The model then uses this compressed KV cache for subsequent decoding inference. Gest (Ge et al. 2024) introduces a pattern-aware approach by identifying five fundamental attention structures and implementing targeted selection strategies. These include proximity- baase reten- tion for local attention patterns, selective preservation of critical tokens for punctuation-focusee attention, frequency- based filtering for sparse attention distributions, and complete token retention for broad atten- tion patterns. SnapKV (Li et al.] (024b) simplifies Gents approach by focusing solely on retrieving tokens based on their importance scores. It demonstrates that among all prompt tokens, only portion carries crucial information for response generation, with these tokens maintaining their significance during the generation phase. The approach employs an end-positioned observation window to detect these impor- tant contextual tokens. Their corresponding key-value pairs are then concatenated with the tokens from the observation window. 1.2Compeess (Devoto el al. 2024) proposes another simple yel. effective approach to identify important tokens during the prefilling phase using the L2 norm for key embeddings. By analyzing attention distributions in decoder-only Transformers, the authors find clear correlation between the Lę norm of key embedding and its attention score during decoding, where a low L2 norm usually leads to high attention scores. Based on this observation, they propose a strategy that compresses the KV Cache by retaining tokens with lower L2 norms. Attention-Gate (Zeng et al., 2024) introduces a learnable KV-Cache eviction mechanism that processes the entire context sequence and generates token-wise eviction decisions through a parameterized policy network, enabling dynamic in-context. memory management.
4.1.2 Dynamic Selection with Permanent Eviction
This category of methods performs frequent KV cache selection during the decoding phase, permanently removing unselected KV cache tokens from memory. Early works employ a Siiding-window mechanism to address long-text inference challenges, where tokens falling outside the window are permanently evicted and become inaccessible. StreamingLLM (Xiao et al. 2024c) uncovers a crucial phenomenon in transformer attention where preserved key-value pairs from initial sequence tokens maintain crucial model performance. This attention sink effect manifests through asymmetric attention weight accumulation at early positions, regardless of semantic significance. The approach leverages this characteristic by incorporating attention sink positions with recent context for efficient processing. LM-Infinite (Han et al., (2224) demonstrates that conventional techniques, including biddnng-wndoo patterns and relative positional encodings, fail to resolve length generalization issues. The study introduces a novel hodoloygg through the integration of A-shaped attention masking and attention distance ceiling mechanisms.
Recent works have explored leveraging attention scores criterion for selecting significant KV cache tokens. H20 Zhang et al., 2023b observes that attention computations are primarily driven by a select group of high-impact tokens, known as Heavy Hitters. This method reformulates cache optimization as a dynamic submodular problem, utilizing cunulative attention scores to guide token retention decisions. Unlike H2O, BUZZ ZZao et al. 2024a) employs a beehive-like structure that selects Heavy Hitters in local KV cache segments. NACL, (Chen et al. (024d) identifies fundamental limitation in H2O, namely their dependence on potentially biased local attention statistics. To overcome this issue, they develop an alternative approach implementing diversified random eviction strategy for token selection. Scissorhands (Liu et al., 2023c builds upon the temporal significance principle, which suggests that tokens demonstrating historical importance maintain their influence in subsequent computational steps. This obscrvation enables the preservation of repetitive allention palterns through selective token relention. Additionally, Keyformer (Adnan el al.) 2024) reveals that token removal distorts the underlying softmax probability distribution. Considering the pivotal role of softmax distributions token significance evaluation, they incorporate regularization techniques to mitigate these distributional perturbations. SepLLM (Chen et al. (2044)) observes that separator tokens (c.g.. commas, periods, and line breaks) receive disproportionately high attention scores and naturally summarize
11

---

Published in Transactions on Machine Learning Research (05/2025)
text segments. Building on this, SepLLM retains separator tokens together with initial tokens, important tokens, and recent tokens in the cache.
4.1.3 Dynamic Selection without Permanent Eviction
The aforementioned permanent eviction-based approaches face two significant limitations. First, the irre- versible eviction of tokens polentially impairs the model's performance on long-sequence tasks, particularly in needle-in-a-haystad scenarios, and these methods prove challenging to adapt to multi-turn dialogue con- texts. Second, KV cache selection during the decoding phase introduces computational overhead, adversely affecting decoding latency and compromising end-to-end acceleration. To address these challenges, several studies have focused on developing decoding-phass KV cache selection strategies without permanent evic- tion. These approaches typically employ multi-tier cache systems (e.g., CPU-GPU hierarchical caching) and leverage advanced data struclures and system-level enhancements to optimize retrieval efficiency, enabling efficient inference with reduced GPU KV cache footprint.
To accelerate the retrieval of critical tokens, several rescarch efforts have proposed index-based approaches that organize and access KV cache at block or cluster granularity, enabling efficient query and extraction operations. InfLLM (Xiao et al., 2024a) maintains full KV cache in blocks while facilitating long sequence pro- oessing through a hicrarchical storage strategy. The framework employs CPU-GPU memory orchestration, preserving essential lokens and current. computational units in GPU memory while offloading less frequently accessed units to CPU memory. To further enhance top-k block retrieval precision, the Quest (Tang et al., 2024b) framework presents refined block representation approach based minimal and maximal key val- ues in KV cache blocks. PQCache (Zhang et al. 2024b) also implements block KV cache management and identifies salient tokens through Maximum uner-PProdus Search (MIPS), leveraging Product Quant iza- tion (PQ) codes and centroids. Squeczed.Attention (Hooper et al.) 2024a) employs K-means clustering in an offline stage to group semantically similar keys, with each group represented by a centroid. During inference, it compares imput queries against these centroids to identify and load only the semantically relevant keys from the context. Similarly, RetrievalAttention (Liu et al., 2024b) manages KV cache tokens using approxi- mate nearest neighbor search (ANNS) techniques. Additionally EM-LLM (Fountas et al. 2024 dynamically segments incoming lokens into episodic events. Also, it implements a hybrid retrieval mechanism that com- bines semantic similarity matching with temporal context to efficiently access relevant KV cache segments. Similarly, ClusterKV (Lin et al. 2024c) groups tokens into semantic clusters and selectively recalls them during inference, achieving both high accuracy and efficiency for LLMs.
To accelerate top-k token identification, SparQ, (Ribar et al. 2024) identifies the r most significant elements in the incoming query vector and selectively retrieves the corresponding components along the hidden di- mension of the cached key matrix K for approximale attention computation. To overlap prefetching latency, InfiniGen (Lee et al. 2024h employs asynchronous prefetching, utilizing indices of salient KV entries selected by queries from the previous layer to retrieve KV cache entries in the current layer. To ensure maximum model performance, RecycledAAteention (Xu et al. 2024b) sustains the entire KV cache during inference computations, yielding no improvements in memory efficiency. The approach performs periodic top-k token selection to identify salient tokens. Moreover, MagicPIG (Chen et al., 2024g) shows that attention-based top-k selection may incur performance degradation. To address this limitation, they introduce novel heterogeneous computing framework leveraging Locality Sensitive Hashing (LSH) techniques. The system stores LSH hash tables and performs attention estimation on CPU. Recently, Loki (Singhania et al. 2024) is a sparse attention method motivated by the observation that attention key vectors lie in a low-dinensional space, leveraging PCA-based dimensionality reduction and dynamic top-k token selection to significantly reduce computation and memory overhead while preserving model accuracy.
4.1.4 Summary and Future Directions
Static KV cache selection algorithms demonstrate superior decoding efficiency overall; however, their efficacy remains to be thoroughly validated in multi-turn dialogues and extended decoding length scenarios. Dynamic KV cache selection algorithms, while adaptive, introduce additional computational overhead during the decoding phase due to frequent cache selection operations. Multi-tier cache architectures and prefetching schemes partially mitigate these challenges, yet. their capability to achieve rapid and accurate
12

---

Published in Transactions on Machine Learning Research (05/2025)
Table Comparison of KV cache budget allocation strategies. Extra: Extra-calibraaion
Method
Layer-wise wisd Layer-wwsseeeed-wwiie
Retrieval-head Reeellllee hnppttt-ee Input-specifie
Extra
Remark
Py
pyramid-shaped
Cal et al.
2024
PyramidInfe
pyramid-shaped
Yang et al
2024a
DynamicKV
v
maximize attention retention rate
Anonymous
2024b
PrefixKV
maximize attention retention rate
Wang et sl.
2021a;
CAKE
layer-speclfic preference score
Anonymous
2024a
SimLayerKV
KV cache compression lazy layers
Zhang et al.
2021e]
AdaKV
minimise attention computation loss
Feng
2021
CriticalKV
minimize attention computation loss
Anonymous
2024c
LeanKV
maximize attention retention rate
Zhang et al
20240
Razor Attention
ocho and induction heads
Tang ct al.
2024a
HeadKV V
retrieval and reasoning heads
et al.
2024r
DuoAttention
learned retrieval heads
Xiao et
2024b
retrieval within acceptable decoding latency constraints requires further empirical validation, particularly in real-world applications involving long sequences. Furthermore, existing selection methods predominantly rely on attention score-based lop-k selection mechanisms. However, current top-k approaches may not be able to effectively identify and extract relevant tokens in ultra-long sequence tasks.
4.2 KV Cache Budget Allocation
The hierarchical architecture of LLMs leads to diverse information extraction patterns across layers, with each layer's KV-cache contributing differently to model performance. This inherent heterogencity indicates that uniform KV-cache compression across layers may be suboptimal. KV cache budget allocation addresses this challenge by intelligently distributing memory resources based on each component's importance prediction accuracy, thereby optimizing memory utilization while minimizing accuracy degradation. It is worth noting that budget allocation approaches prioritize the effective allocation of computational budget rather than token selection strategies. Current budget. allocation strategies can be categorized into two levels of granularity: layer-wise budget allocation, which assigns different compression ratios across model layers, and the more fine-grained head-wise budget allocation, which enables precise memory distribution across individual attention heads within each layer, offering more flexible and targeted optimization opportunities. The summary of KV budget allocation listed in Tab.
4.2.1 Layer-wise Budget Allocation
In contrast to conventional approaches with uniform KV cache sizes, PyramidKV (Cai el al. 2024 em- plays a pyramid-shaped memory allocation strategy, assigning larger cache capacities to lower layers that progressively decrease in upper layers. This design is supported by the observation that lower layers exhibit uniform attention distributions across input sequences, while upper layers show concentrated attention on specific tokens. PyramidInfer (Yang et al. 2024a) also adopts pyramid-shaped budget allocation strategy
13

---

Published in Transactions on Machine Learning Research (05/2025)
while selecting tokens with high attention values each layer. Additionally, during the decoding phase, PyramidInfer dynamically maintains a set of significant tokens through frequent updates driven by attention values. Unlike previous methods, DynamicKV (Anonymous) 2024b) implements an input-adaptive budget allocation strategy by analyzing attention patterns. Specifically, it computes the average attention scores between recent. and historical tokens, identifies the top-k tokens with the highest. attention values across layers, and proportionally distributes the budget based on the density of significant tokens in each layer. Similarly, PrefixKV (Wang et al., 2024ah identifies the most important tokens for each layer by computing the average attention score of tokens within that layer. PrefixKV (Wang et al., 2024a) then uses unified threshold to determine the number of retained tokens, adaptively adjusting the retention for each layer based on its importance distribution. CAKE (Anonymous) 2024a) examines attention scores through two lenses: the spatial distribution of inter-token attention and the temporal evolution of allention focus. These measurements are combined to compute layer-specific importance scores, which further guide the alloca- tion of memory resources. Additionally, SimLayerKV (Zhang et al.) 2024ch identifies lazy layers, which are ineffective at capturing long-range dependencies. The framework then selectively preserves cache entries, maintaining only the initial and recent tokens for lazy layers, while retaining the complete KV cache for non-lazy layers.
4.2.2 Head-wise Budget Allocation
AdaKV (Feng et al., 2024) leverages the observation that attention patterns cxhibit distinct concentra- tions across different heads. It implements head-specific memory allocation by optimizing an L1 loss bound between the original and pruned multi-head attention outputs. Within the constraints of a layer-wise bud- get, the method distributes cache capacity among heads to maximize the preserved attention information collectively. Building upon AdaKV, CriticalKV (Anonymous) 2024c) introduces significant enhancements by recognizing that the importance of KV cache entries extends beyond attention weights to encompass value states and pretrained parameter matrices. Leveraging this insight, the framework implements a novel selection algorithm that identifies essential cache entries by minimizing the maximum potential output per- turbation. LeanKV (Zhang et al. 2024f) implements a fine-grained memory optimization strategy that operates independently for each attention head and input request. The method identifies the smallest subset of tokens necessary to preserve the majority of information flow, allocating cache space based on a predefined atlention score threshold and lypically maintaining 95% of the total attention mass.
Retrieval head-based methods represent a specialized category of head-wise allocation strategics that focus on identifying and prioritizing attention heads crucial for extracting key information from long soquences. This approach allocates larger cache budgets to these specialized heads, known retrieval heads (Wu et al., 2024d), due to their significant role in information extraction. RazorAttention (Tang et al. 2024a) charac- terizes two distinct categories of retricval heads: echo heads, which focus on previously occurring identical tokens, and induction heads, which attend to antecedent tokens that precede current token repetitions. This framework implements differential caching strategies, maintaining complete cache entries for retrieval heads while condensing remote tokens into consolidated compensation tokens for non-retrieval heads. HeadKV (Fu et al. 2024c) further enhances RazorAttention by introducing a novel head assessment framework that si- multaneously evaluates both retrieval and reasoning capabilities to optimize KV cache allocation strategies. DuoAttention (Xiao et al. 2024b) further introduces parameterized approach to distinguish between two calegories of allention mechanisms: retrieval heads, essential for comprehensive long-context processing, and Streaming heads, which primarily engage with recent tokens and attention sinks. This classification is achieved through learned parameters that automatically identify retricval heads requiring full attention spans.
4.2.3 Summary and Future Directions
Despite recent advances and growing attention in KV cache budget allocation rescarch, several critical chal- lenges remain unaddressed. First, the relationship between allocation strategies and model performance requires further investigation. For instance, a notable discrepancy exists between pyramid-shaped allocation strategics (Cai et al.) 20244 Yang et al.] (0224a) advocating larger budgets for lower layers, and retrieval head- studies (Tang et al.] [2024a] Fu et al. (222cc) which demonstrate that lower layers rarely ex- hibit retrieval head characteristies and thus require minimal cache resources. Additionally, the field lacks
14

---

Published in Transactions on Machine Learning Research (05/2025)
Table 4: The summary of existing KV Cache merging approaches.
Merge Layer
Model
Merge Unit
Merge Metrie
Merge Type
Training-free
Intra-layer Cross-layer
CCM
Token
Sliding Window
Mauy-to-Oue
Kim
2024n
LoMA
Token
Sliding Window
Many-to-Marg
Wang & Xlao
2024
DMC
Token
Learned Merge Indictor
Many-to-One
Nawrot et al.
2024
D20
5
Token
Cosine Similarity
Two-to-One
Wan ct al.
2024a
CaM
/
Token
Attention Score
Many-to-One
Zhang
2024g
A1M
Token
Cosine Similarity
Many-to-One
Zhong
2024b
ZeroMerge
Token
Cosine Similarity
Many-to-Oue
Liu et al.
2026
Look-M M
Wan et al
2021b
Token
Cosine Similarity
Many-to-Que
KVMerger
/
Token
Weighted Gaussian Kernel
Many-to-One
Wang et al.
2024c
CHAI
Head
Attention Score
Many-to-One
Agarwal
2024
MinCache
Token
Angular Distance
Two-to-One
Liu et al.
2024a
KVSharer
Layer
Euelidean Distance
Many-to-Oue
Yang ct, al.
2024d
comprehensive experimental comparisons, particularly regarding the compatibility and performance benefits of head-wise budget allocation strategies with ofate-of-the-ar frameworks like vLLM (Kwon et al. 2023) and FlashAttention [Dao et al. 2022 Dao 2024 Shah al. 2024). Also, existing methods, such as Pyra- midInfer (Yang al. 20244a), demonstrate some adaptability to input attention patterns. However, future research could target real-time, task-specific allocation strategies that dynamically adjust memory budgets during inference based on input characteristics, task complexity, downstream requirements.
4.3
KV Cache Merging
KV cache merging offers promising solution by compressing or consolidating KV caches without significantly degrading model accuracy. Rather than a uniform compression strategy, KV cache merging techniques leverage the inherent redundancy within and across layers dynamically optimize memory utilization. These methods aim to reduce the size of KV caches while preserving critical information necessary for accurate attention computations, enabling efficient inference in resource-constrained settings. Existing KV cache merging strategies can be categorized into two primary approaches: intra-layer merging, which focuses consolidating KV caches within individual layers to reduce memory usage per layer, and cross-layer merging. which targets redundancy across layers to eliminate unnecessary duplication. Both approaches offer complementary advantages, providing flexibility to balance memory savings and model performance degradation. The summary of the KV cache merging is listed in Tab.
4.3.1 Intra-layer Merging
As the input sequence length incrcases, the number of Keys and Values grows, leading to higher computa- tional costs for the altention process. To address this, CCM (Kim et al. 2024a), LoMA (Wang & Xiao 2024), DMC (Nawrot et al. 2024 propose to learn compression module to compress KV of tokens.
Specifically, CCM Kim et al. 2024ah inscrts special indicator token, [COMP], into the input sequence and compresses the accunulating past attention key/valus (KV) pairs in each layer wetween these indicators into a compact memory space. This compression leverages techniques inspired by the Compressive Transformer (Rac et al. 2020) and Gisting (Mu et al. 2023). Instead of computing attention across all tokens, CCM (Kim al. 20244) computes attention scores for each new token by referencing the merged token. Similarly, LoMA (Wang & Xiao 2024) inserts special token into the input sequence determine
15

---

Published in Transactions on Machine Learning Research (05/2025)
which consecutive tokens should be compressed. LoMA (Wang & Xiao 2024 performs compression using bidirectional attention, repetition zone supervision, and carefully designed attention masks and loss functions. DMC (Nawrot et al., 2024 learns a variable to decide whether to append new KV pairs to the cache when necessary or to merge them into existing KV representations using a weighted average.
Note that CCM (Kim el al. 2024a), LoMA (Wang & Xiao, 2024), and DMC (Nawrol et al. 2024 require supervised learning to learn a compression module.
Instead, CaM (Zhang et al. 2024g) KVMerger (Wang et al. 2024c). : and D20 (Wan et al 2024a are training-freee which rely observations and directly propose mule-based or heuristic- seuristic-based merging strate- gies. Specifically, they separate the Keys and Values of tokens in each layer into important (retained) and unimportant (evicted) tokens. They then keep potentially useful unimportant tokens by merging their Keys and Values with retained important tokens, ensuring that no valuable information is lost. Particularly, D20 (Wan et al. 2024a) merges the Key (or Value) of a evicted token with one retained token based on co- sine similarity. Similar to D2O, AIM (Zhong ct al., 2024b), Look-M (Wan et al. 2024b), and ZeroMerge (Liul ee al. 2025) merge Keys (resp. Values) of multiple tokens into one. CaM (Zhang et al.) 2024g) merges the Keys (or Values) of multiple evicted tokens with retained tokens based attention scores to get the final merged results. Also, KVMerger (Wang et al.) (024c) first identifics the merge token sets by clustering consecutive tokens with high cosine similarity, ensuring that only adjacent tokens with strong contextual relevance are grouped together. Then, KVMerger merges the tokens in each merge set into the pivotal token (chosen based on the highest attention score) using Gaussian kernel weights, where closer tokens contribute more to the merged state.
Instead of merging the KV of multiple tokens into one, CHAT (Agaraal et al., 2024 observes that heads in multi-head attention often produce highly correlated attention scores for tokens, particularly in the later layers of LLMs. To exploit this redundancy, CHAI (Agarwal et al. 2024) clusters attention heads within each layer that produce similar outputs and computes attention for only a single representative head in each clus- ter. Specifically, within each cluster, CHAI (Agarwal et al.) 2024) selects one representative head to perform the altention computation, and the computed allention scores are shared across all heads in the cluster.
4.3.2 Cross-layer Merging
MiniCache (Liu et al. 2024a) observes that KV caches in middle-to-deep layers exhibit high angular simi- larity, making them idcal for merging. To achieve this, MiniCache (Liu et al. 2024a) merges the Key (and Value) pairs of each token from adjacent similar layers into a single shared representation. Specifically, Mini- Cache (Liu et al.] (024a) decomposes KV vectors into magnitude and direction components, storing only the shared directional vectors, token magnitudes, and unmergeable tokens to maximize memory efficiency. Differently, KVSharer (Yang et al., 2024d) observes a counterintuitive phenomenon: when the KV caches of two layers differ significantly, sharing one layer's 's KV cache with another during inference does not cause significant performance degradation. Based on this observation, KVSharer (Yang et al. 2024d) computes the Euclidean distance between the KV caches of all layer pairs, ranks the pairs by dissimilarity, and prioritizes the most dissimilar layers for sharing. Since KVSharer (Yang et al., (024d) can share the KV cache of one layer to multiple other layers, the stored KV cache eliminated significantly.
4.3.3 Summary and Future Directions
KV cache merging represents a transformative approach to optimizing memory utilization in LLMs by consolidating or compressing KV caches while maintaining high model accuracy. However, there are several key directions and challenges for future exploration in this domain. Firstly, current KV cache merging methods are typically designed to work across wide range of tasks, but fine-tuning merging strategies for specific tasks or domains could further enhance efficiency. For example, certain tasks may tolerate more aggressive merging due to inherent redundancy in their attention patterns, while others may require more conservative approaches to preserve accuracy. Adaptive merging mechanisms that adjust compression levels on-the-fly based on task difficulty, sequence length, or available hardware resources are an exciting avenue for future work. Sccondly, sparse attention mechanisms, which alrcady reduce the computational complexity of attention by operating on subsets of tokens, could be combined with KV cache merging to achieve even greater efficiency. Exploring how merging complements sparsity-based approaches, such
16

---

Published in Transactions on Machine Learning Research (05/2025)
FP32
INT8
C
C
0.23
5.12
4.37
58
X
T X
X
T
T
1.58
3.74
-1.17
54
16
-3.26
0.67
1.01
30
33
per-tensor
per-token
per-channel
(a) The quantization of attention matrix.
(b) Three types of quantization. Then matrix X E TTCC where T is the number of tokens and C is the feature dimension.
Figure An example of quantization and three types of quantization applied each matrix.
sparse or low-rank attention, could lead novel hybrid solutions. Thirdly, while empirical results show that merging does not significantly degrade performance, providing theoretical guarantees about the preservation of critical information could enhance the reliability of these methods. Future work might focus quantifying the relationship between merging strategies, token importance, and attention accuracy to provide more formal guarantees.
4.4
KV Cache Quantization
As shown in Fig 4a quantization techniques [Ln et al. 2016 Wu et al. 2020: Kwasniewska et al. 2019:
Zhou et al. 2018 Jiang & Agrawal 2018) aim to convert full-precision values into ower-bit integers, reducing computations and storage requirements. Quantization techniques have been widely used to accelerate machine learning models from different aspects, such model parameter quantization (Frantar et al., 2022
Dettmers al. 2024] Bondarenko et al.] 2023 Cheng et al., 2017) and data fcature quantization (Zhou
al. 2023a Jegou al.) 2010). Similarly, Key-Value (KV) cache quantization emerging a highly promising solution to address the memory and computational bottlenecks in LLMss. During auto-regressive decoding, LLMs generate key-value pairs for every attention layer across all tokens in the sequence. If we store all KV pairs in the memory with full precision, this cache grows linearly with longer sequences, increasing the memory and bandwidth requirements significantly. Quantization reduces the precision of numerical representations (e.g., from FP32 to INT8 or INT4), drastically compressing the size of the KV cache. This compression can achieve up to or more memory savings, making it feasible for LLMs to operate on resource-constrained devices like GPUs with limited memory or edge devices.
However, the presence of outliers in Keys and Values poscs a significant. challenge for low-bit quantization, these extreme values can lead Lo substantial performance degradation when compressed into reduced bit rep- resentations (Dettmers et al. 2022 Bondarenko et al. 2021 Wei et al. (022).. Based on the techniques used, cxisting KV cache quantization approaches can be classified into three main categorics: Fixed-precision quantization, where all Keys and Values are quantized the same bit-width; Mixed-precision quantiza- tion, which assigns higher precision critical parts of the cache while using lower precision for less important components; and Outlier redistribution, which redistributes or smooths the outliers in Keys and Values to improve quantization quality. These methods collectively enable efficient KV cache compression while mitigating the performance degradation typically associated with low-bit quantization.
4.4.1 Fixed-precision Quantization
Fixed-precision quantization proposes quantizing different Keys (different Values) of tokens to the same bit-width. ZeroQuant Yao et al. 2022) proposes per-token quantization for Keys and Values. As shown in Fig, 4b the per-token quantization approach quantize tokens individually. Particularly, ZeroQuant (Yao ct al. 2022 dynamically computes the min-max range for each token during inference. This ensures that each loken is quantized based on its unique range, significantly reducing quantization error. Also Flex- Gen (Sheng et al.) 2023) and QJL (Zandich et al.] (024) directly perform per-token quantization for Keys and Values, where the scaling factor and zero-point are shared among all elements within the same token.
PQCache (Zhang et al. 2024b) uses product quantization approaches (Jegou et al.) 2010 Matsui et al., 2018 to compress KV pairs. However, uniform quantization approaches, which fixed bit-width for
17

---

Published in Transactions on Machine Learning Research (05/2025)
Table 5: The summary of existing mixed-
mixed-precision precision quantization models.
Important Tokens
Model
Keys
Values
Middle Recent
Outller storing Channel Reorder
KVQuant
Channel, Pre-RoPE Per-Token
[Hoopor
2024h
KTVT
Chanuel
Per-Token
Liu et al.
2024b
SKVQ Q
Dynamic outlier-wwaee
Duanmu et al.
2024
WKV Quatt
Learnable shifting
v
Yue ct al.
2024
Q
Adaptive quantixation hits
1
v
Dong
2024b
MiK
Dynamic outlier-aware
Yang ct ml.
2024c
CEARR
Dynamic outlier-aware
Kang et al.
2024
ZIPVL
Conventional
He et al.
2024a
CacheGen
Layer-wise, token-locality
et al.
20240
Atom
Group-based
Zhao al.
2024b
keys and values across all (okens, can often be suboptimal. It is because they ignore the varying importance of tokens (Zhang et al. 2024f) and account for the outlier patterns in Keys and Values (Dong et al. 2024b Hooper ct al. 2024b).
4.4.2 Mixed-precision quantization
Unlike fixed-precision quantization, where all Keys Values are quantized to the same bit-width (e.g., 4-bit or 8-bit), mixed-precision quantization assigns higher or full precision to Keys and Values of critical tokens and parts while using lower precision for less critical parts. The summary of KV mixed-precision quantization is listed in Tab. KVQuant (Hooper et al. 2024b) proposes several strategies to quantize Keys and Values smoothly based on obscrvations. Firstly, KVQuant observes that the key values exhibit outliers in specific channels prior to applying Rotary Positional Embedding (RoPE). However, after applying RoPE, the magnitudes of these outlier channels become less consistent, creating unique challenge for low-precision quantization. Thus, KVQuant (Hooper et al.] 2024b) proposes to quantize the Keys per channel before applying the RoPE operations and to quantize the Values per token. Secondly, KVQuant (Hooper et al. 2024b) observes that KV cache activations contain outliers that skew the quantization range. To address this, they isolate outlicrs per vector (c.g., per-channel or per-token), store them in a sparse format, and quantize the remaining values to a narrower range. Thirdly, LLMs disproportionatel, allocate high allention scores the first token (i.e., attention sink), and quantizing the first token will damage the performance of LLMs. Thus, KVQuant (Hooper et al. 2024b) retains the first token in full precision (FP16) while quantizing the rest of the sequence, which is also used by IntactKV (Liu et al.) 2024eb and SKVQ (Duanmu al. 2024 Similar to KVQuant (Hooper et al. 2024b), KIVT (Liu et al. 2024h) quantizes the Key cache per-channel, as certain channels exhibit large outliers, and the Value cache pokeno as there are no significant outlier patterns in the Value cache. Additionally, KIVI (Liu et al.) 2024h) retains the most recent Keys and Values in full precision, while quantizing older KVs.. This approach is based on the observation that the most recent KVs are critical for generating subscquent tokens.
Similar to KIVI [Liu et al. 2024h), WKVQnant (Yue al. 2024 temporarily retains the most recent Keys and Values in full precision, while quantizing only the past KV cache. This approach (Yue et al, 2024h helps preserve precision during computation. Similarly, ResQ (Saxena et al. 2024b) uses PCA to identify high-variance components, which are quantized in 8-bit, while the rest are quantized in 4-bit. Also, ResQ (Saxena et al. 2024b) uses random rotations to suppress outliers, improving robustness. Additionally, WKVQuant Yuue et al., 2024) introduces two-dimensional quantization strategy, which optimizes the parameter matrix to align the values in the KV cache into a smoother and more uniform range, significantly improving quantization quality. GEAR (Kang et al. 2024), MiKV (Yang et al. 2024ch, ZipCache (He et al.
18

---

Published in Transactions on Machine Learning Research (05/2025)
2024b) and ZIPVL (He et al. 2024a) quantize the KV cache based the importance of each to achieve efficient and effective compression. First, GEAR (Kang et al. 2024) applies quantization to compress the majority of less important entries (e.g., 98%) to ultra-low precision, significantly reducing memory usage. Next, GEAR (Kang et a 2024) employs a low-rank matrix to approximate residual errors, capturing structured patterns in the data. Also, GEAR (Kang et al., 2024) uses a sparse matrix to store outliers, correcting individual errors caused by these values. MiKV (Yang et al. 2024c) is a mixed-precision KV cache quantization approach. Based on the importance of each token, measured using existing methods like H2O Zhang et al. 2023a and SnapKV (Li al. 2024b), MiKV (Yang et al. 2024cb stores less important KV pairs in low precision while retaining the most important KV pairs in high precision. Instead of approximating the importance weight of each token, ZipCache (He et al. each loken. Instead of computing importance score, PrefixQuant (Chen el al. 2024ch observes that token- wise outliers frequently occur at fixed positions (e.g., initial tokens) or low-semantic-valug tokens (e.g.. '.,
2024b) accurately computes the importance of
"(n"). Based on this observation, PrefixQuant (Chen ct al., 2024c) identifics high-frequency outlier tokens in LLMs oflline and prefixes them in the KV cache, ellectively eliminating token-wise outliers. Similarly, MiniKV (Sharma et al. 2024) observes that important tokens can be identified before generation and remain consistent throughout the generation process, retaining these important tokens in high precision.
KV-AdaQuant (Hariri et al. 2025 observes the higher sensitivity of key matrices to quantization errors due to their larger norms, and therefore allocates more bits to key matrices and fewer bits to value ma- trices, optimizing both accuracy and memory efficiency. QAQ (Dong et al. 2024b) proposes a quality adaptive quantization approach to dynamically determine the suitable quantization bit for each token, based its importance and sensitivity, while handling outliers and exceptions to maintain model performance. SKVQ (Duanmu ct al.
2024) introduces the clipped dynamic quantization with channel reorder. First, SKVQ DDanmu et al."
permutation to group similar channels based on their statistical characteristics and applies clipped dynamic quantization to mitigate the outlier prob- lem. Second, SKVQ (Duanmu et al. 2024) maintains high precision for the initial tokens and the most recent tokens while quantizing older tokens. Consequently, SKVQ (Duanmu et al.] 2024) effectively reduces quantization errors and improves the accuracy of the quantized model. CacheGen (Liu et al. 2024f) and AsymKV (Tao et al.] 2024) use layer-wise asymmetric quantization, assigning higher-bit precision key matrices in sensitive carly layers and lower-bit precision to less sensitive layers, balancing memory efficiency and performance. Particularly, CacheGen (Liu el al. 2024() also exploits token-wise locality by encoding deltas (differences) between KV tensors of nearby tokens instead of raw values. Atom (Zhao et al. 2024b) identifies and separates outlier channels, reordering the matrix to group these outlier channels at the end, thereby ensuring regular memory access patterns for improved hardware utilization. Then, Atom (Zhao et al. 2024b) quantizes outliers with higher precision, while normal channels are quantized to INT4 for max- imum efficiency. In particular, Atom (Zhao et al., 2024b) applies fine-grained group quantization by dividing matrices into smaller subgroups (e.g., 128 elements per group) and performing quantization independently within each group.
2024 uses ceansorrnation-inaaiii
4.4.3 Outlier Redistribution
As previously mentioned, outliers in the Keys and Values present significant challenges for their quantization. Recent research has proposed two main approaches to address this issue: redistributing the outliers into newly appended virtual tokens or applying equivalent transformation functions to smooth the Keys and Values for improved quantization accuracy. The summary of existing outlier redistribution models are listed in Table. D
Specifically, MessiveActivation (Sun al.) 2024b) highlights the phenomenon of massive activations in large language models (LLMs), where small subset of activations is exponentially larger than the rest. To ad- dress this, MassiveActivation (Sun et al. 2024b proposes appending virtual token to the inputs, allowing LLMs to encapsulate the massive outliers within these learned keys and values for each head. On the other hand, to further address this issue, existing researchers (Ashkboos el al. 2024 Lin et al. 2024d) introduce equivalent transformation function- based approaches. Equivalent transformation functions are memaii cal transformations applied to activations that preserve the underlying information while redistributing or normalizing the values. By redistributing or scaling massive values, these approaches ensure that extreme outliers do not disproportionately affect downstream processes like compression quantization. Firstly,
19

---

Published in Transactions on Machine Learning Research (05/2025)
Table 6: The summary of outlier redistribution models in Sec.
1.4.3. Learn
Model
Operation
Formula
Remarks
Massive Act
K"
k'
Add virtual tokens
softmax
Learnable k', v'
(Sun et al.
2024b
va
Ashkhons et al.
QuaRot
2024
Hadamard rotation
XW' (XH)(H W
X
H H'H=I
Lin al.
Qserve
20244)
Hadamard rotation
XW T = (XH)(H"
x
H =
Xi et al.
Q-INT4
2023
Hadamard rotation
XWT = (XH) WT)
x
H =1
Smooth Q uant
Xlao et al.
2023
Scaling
ding(>))*) (s)
x
s E It*
(Wei et al.
QS+
2023
Sealing, Shifting
((X x) di) 1 ding(s) 2jWJ
x
E IR
AWQ
Scaling
arg min,
XW X (Q((((e))))((
v
Quantization Q()
Lin al.
2024c
OmniQ Jant
Scaling, Shifting
Qu
X Qu
@ w B &W
Learnable Qa(-), Qu
Shao et al.
2023
Lin sl.
DuQuant
2021bb
Rotation, Permutation
[XX ()) P. Real [RT2) PT. Rii, (A WT)
x
Matrices P,
Affine uant
Affine transform
arg miup Xw XP-'Q(PWT)) ||
V
Quantization Q(-)
Ma et al.
2024b
QlatQuant
Affine transform
Affineuuant P P= &
Decomposition
SSn et sl.
2024d
QuaRot (Ashkboos et al. 2024). Qserve (Lin al. 2024d), and Q-INT4 Xi et al. 2023 redistributes outlier values across all channels by Hadamard rotation, successfully lowering the maximum value of outlier tokens. The Hadamard rotation of activations can be incorporated into the preceding linear layer, thereby redistributing the outliers of Keys and Values into the parameters. Despite this improvement, outlier tokens still exhibit magnitudes hundreds of times greater than normal tokens, causing notable performance issues when using shared quantization scales across tokens (Chen el al.] 2024ch. Expanding on this idea, Spin- Quant (Liu et al., 2024g) proposes training an orthogonal matrix instead of relying on a random Hadamard matrix to achieve better performance. Similarly, DuQuant (Lin et al. 2024b) employs channel permutation to evenly distribute outliers across blocks and utilizes block rotation to further smooth outliers.
SmoothQuant (Xiao et al. 2023) leverages a key observation that different tokens show similar patterns of variation across their channels. Based on this insight, it strategically shifts the quantization complexity from activations to weights through an offline process. Specifically, SmoothQuant (Xiao el al., 2023) introduces a (()iag(s)"))((daa(( W) XW where X represents Keys or Values, and the smoothing factor s E R is used to scale X. This transformation achieves two key benefits: it smooths the distribution of Keys and Values to facilitate easier quantization, and it allows the smoothing factors to be efficiently incorporated into the parameters of previous layers during offline processing. In particular, the smooth factor is dynamically decided on based on inputs. Similarly, The OS+ (Wei et al., 2023) introduces channel- shifting to eliminate outlier asymmetry and channel-wise scaling to reduce outlier concentration. These operations are seamlessly migrated to subsequent layers, maintaining equivalence with the floating-point model while improving quantization performance.
mathematically equivalent per-channel scaling transformation: Y
Instead of using handcrafted transformations Lin et al. 2024b Wei et al. 2023 Xiao et al. 2023 to shift the quantization difficulty from activations to weights, A Affinequant (Ma et al. 2024b) uses an affine trans- formation matrix that combines both scaling and rotation transformations. This allows it to optimize weight distributions more effectively, aligning them better with the quantization function and reducing quantization errors. The affine transformation matrix provides richer flexibility compared to SmoothQuant's 's scalar-based scaling, enabling finer adjustments to the weight and activation distributions. Based on AffineQuant (Mal let al. 2024b), FlatQuant (Sun el al. 2024d) introduces a fast and learnable alfine I transformation to enhance the flatness of weights and activations, which decomposes transformation into smaller matrices to reduce memory and computational costs. Similarly, AWQ (Lin et al., (024c) and OmniQuant (Shao et al. 2023) propose differentiable and learnable equivalent transformations, which optimize the equivalent parameters (e.g., channel-wise scaling and shifting) in an end-to-end manner using gradient. descent.
20

---

Published in Transactions on Machine Learning Research (05/2025)
4.4.4 Summary and Future Directions
KV cache quantization is a crucial technique for reducing memory and computational overhead in large language models (LLMs) during auto-rogressive docoding. While significant progress has been made, this field remains dynamic and rapidly evolving, with several promising directions for future research. Finstly, one promising avenue is the development of real-time adaptive quantization methods. These techniques could dynamically adjust quantization levels during inference based on real-time metrics such as token importance, outlier presence, or sequence length. Such an approach could significantly enhance efficiency while maintaining performance, especially for processing long sequences with varying levels of complexity. Secondly, another important direction is extending KV cache quantization to multi-modal and multi-task models. Multi-modal models, which process inputs from diverse domains such as text, vision, and audio, and multi-task scenarios often exhibit highly diverse attention patterns and memory demands. This necessitates the design of more advaneed and tailored quantization strategies to balance efficiency and accuracy in these increasingly complex settings.
Thirdly, hybrid quantization techniques also hold significant potential. By combining exxedprecision mixed- procision, and outlier rodistribution methods, researchers could develop more versatile and efficient quan- tization frameworks. For instance, integrating mixed-precision allocation schemes with outlier smoothing transformations could optimize both memory usage and performance, offering a flexible approach adaptable a variety of tasks and models. Finally, addressing the challenge of outliers remains a critical area of fo- cus. Outliers can have a disproportionate impact quantization efficiency and model performance. Future research could explore ad vdvace outlier detection mechanisms or innovative encoding techniques to mil their effects. Improved handling of outliers could further enhance the effectiveness of quantization methods, enabling more robust and memory-effcient implementations.
4.5 KV Cache Low-rank Decomposition
Existing studies have demonstrated that the majority of information within KV caches can be captured by small subset of their singular values or low-rank components, making low-rank decomposition pow- erful tool for compression. By leveraging this property, KV cache low-rank decomposition techniques aim to reduce memory requirements while preserving the essential information required for accurate attention computations. Low-rank decomposition strategies can be classified into three main approaches: Singular Value Decomposition (SVD), which exploits the low-rank structure of KV matrices to retain the most critical singular values; Tensor Decomposition, which factorizes KV matrices into smaller components for minimal redundancy; and Learned Low-rank Approximation, which incorporates adaptive mecha- nisms to optimize compression based on learned representations. Fach method provides a unique balance of computational efficiency and accuracy retention, cnabling scalable and memory-effecient LLM inference.
4.5.1 Singular Value Decomposition
Recent research has demonstrated that KV caches in large language models exhibit strong low-rank prop- erties, where a small number of top singular values retain most of the information. Building upon this discovery, numerous low-rank approximation techniques have been proposed for KV cache optimization. These methods can be broadly categorized into two approaches: those that directly decompose the KV cache and those that apply low-rank approximations to the KV weight matriccs.
Among the methods that directly decompose KV caches, ECKVH (Yu et al. 2024], EigenAttention (Saxena et al. 2024a), Q-Filters (Godey et al.), and ZDC (Zhang & Shen 2024 all leverage Singular Value Decompo- sition (SVD) techniques but with distinct implementations. ECKVH (Yu et al. 2024) compresses KV caches by grouping altention heads, applying SVD, and relaining top singular values, which effectively reduces the number of KV heads while minimizing error. Similarly, EigenAttention (Saxena et al., 2024a) employs SVD to approximate keys, queries, and values with low-rank basis vectors, thereby reducing the dimensionality of KV matrices. Q-Filters (Godey et al.),, meanwhile, offers a training-free KV cache compression method that utilizes SVD-based key projections for efficient and accurate attention score approximations.
21

---

Published in Transactions on Machine Learning Research (05/2025)
Besides, Loki (Singhania et al. 2024 introduces alternative approach that first computes approximate attention scores in a reduced dimensional space to efficiently rank and select top-k keys. It then calculates exact attention scores using only the selected keys in the transformed space, thereby reducing computational and memory requirements while maintaining performance. Furthermore, ZDC (Zhang & Shen 2024) imple- ments an adaptive hybrid compression ralio mechanism that assigns higher compression to less important tokens in shallower layers while preserving more important tokens in deeper layers, effectively leveraging the similarity of token characteristics in adjacent layers.
Taking fundamentally different approach, LoRC (Zhang al. 2024d) employs low-rank approximations of KV weight matricos rather than decompasing the KV pairs themselves. Specifically, LoRC uses SVD to com- press the Keys and Values parameter matrices (i.e., W and Wi), decomposing them as W U p and W P Additionally, LoRC adopts a progressive compression strategy, applying compression conservatively in shallower layers to minimize error amplification while compressing more aggressively in deeper layers. Instead of storing complete K = X;W and V = W; matrices, it only stores K = X,U
and V = X, along with E/P: T and E.P.T achieving efficient KV cache compression without requiring model rei.aining. Palu (Chang et al. 2024 follows a comparable approach by applying SVD to compress KV weight matrices simultaneously, while ShadowKV (Sun et al. 2024a) takes a different angle by performing SVD docomposition directly on Pre-RoPE keys to reduce the dimensionality of key representations.
4.5.2 Tensor Decomposition
Tensor decompositoon (Klleshvv et al. 2015 Zhou et al. 2017 Haeffele & Vidal 2015) is a widely used
algorithm for factorizing a matrix into a scquential product of local tensors, such as Matrix Product Operator (MPO) (Liu el al. 2021] and Lurker decomposition (Malik & Beckerj 2018). Taking Matrix
Product Operator (MPO) (Liu et al. 2021)
example, the decomposition of matrix W E RIXJ using
MPO can be defined as:
11 TD(W) = II Twoldk-11 ies J=
(12)
where Tik) represents the local tensor of size dx-1 x is x jk x dx, with i is 1 and I jo = Here, n denotes the number of local tensors, collectively referred to as the decomposod tensors. As shown in Equation 12 MPO-based lensor decomposition is well-suited for KV cache compression as il reduces the memory footprint by factorizing large key and value matrices into smaller local tensors, enabling efficient storage while preserving essential information. This approach minimizes redundancy and maintains the structural integrity required for accurate attention computations. DecoQuant (Liu et al. 2024d) combines quantization with low-rank decomposition to effectively reduce quantization errors. Specifically, DecoQuant (Liu et al.) 2024d) leverages the Matrix Product Operator (MPO) to decompose matrices into smaller local tensors. The larger tensors, which contain most of the parameters, are quantized low-bit precision, while the smaller tensors relain high precision to minimize overall quantization error.
4.5.3 Learned Low-rank Approximation
LESS (Dong et al., 2024a) introduces a novel learned-kernol-aseed low-rank approximation approach to efficiently approximale the results of the softmax function. Specifically, LESS (Dong el al.] 2024a) replaces the softmax with a separable similarity metric, ((q:))((K)) where & and are row-wise functions. Here, qi E IR IxD represents the query, and K, E RRtx represents the keys at step t. To elaborate, if o and sb are such that: a; = softmax
((k then we only need to cache the hidden states H. = (K,) TV, E Rtx and the normalization factor 2, = E. ((K,).)) III for inference. Similarly, MatryoshkaKV (Lin et al. 2024a) compresses KV caches along the feature dimension by leveraging trainable orthogonal projection matrices.
K.
V ~ d(q.))(K.) TV.
vT
4.5.4 Summary and Future Directions
KV cache low-rank decomposition a powerful technique for compressing KV caches in LLMs while maintain- ing the quality of attention computations. Current methods primarily rely on fixed low-rank approximations
22

---

Published in Transactions on Machine Learning Research (05/2025)
Intra-Layer Group- ing (Sec. 5.1.1
MQA
Shazeer 2019
CQA (Ainslic
et al.
2023
AsymGQA (Chen et al.
20241
Weighted GQA (Chinnakon-
duru & Mohapatra 2024). QCQA
Attention Crouping and Sharing (Sec. 5.1
Joshi et al.
2024
KDGQA (Khan
et al.
2024
GQK VA Javadi et al.
2023
Cross-I Layer Shar-
CLA (Brandon ct al. 2024). LCKV
ing (Sec. 5.1.2
Wu & Tu
2024), SA [Liao & Var-
gas
2024].
MLKV (Zuhri et al.
2024
LISA
(Mu et al. 2024), (W
et al.
2024e
CLLA (Yang al.
2024c
DHA (Chen et al.
2024e
SVFormer (Zhou et al. 2024b
Model-level Optimization (Soc.
Enhanced Atten- tion (Sec. 5.2.1
MLA (DeepSeek-A) et al. 2024
FLASH (Hua et al. [2022] Infin-
Architecture teration (Sec. 5.2
Attention MMunkhdalai et al. 2024
Augmented
VOCO (Snn al.
2024c), CEPE
Architecture (Sec. 5.2.2
Yen al.
2024)
XC-Cache Mon-
teiro et
2024
Block Transformer
(H al.
2024
Adaptive Sequence Processing Archi- tecture (Sec. 5.3.1
RWKV (Peng
2023
Mamba
Gu Dao
2024
RetNet (Sun
al. 2023 MCSD
Yang al.
non-transformer Ar- chitecture (Sec. 5.3
2024b
Hybrid Architec- ture (Sec. 5.3.2)
MixCon (Xu & Lin 2024). GoldFinch
Goldstein et al. 2024). RecurFormer
Yan al. 2024
Figure 5: Taxonomy of the model based KV optimization for Large Language Models.
applied uniformly across all layers or tokens. However, future advancements could focus on dynamic rank adjustment, where the rank is tailored based on token importance, sequence length, layer-specific proper- ties, enabling more optimal balance between memory efficiency and performance. Additionally, real-time or streaming applications present a promising avenue for exploration. Since KV caches grow dynamically during inference, lightweight and incremental decomposition methods that can adapt efficiently to expanding sequences will be critical for supporting such scenarios without compromising latency or accuracy.
5 Model-level Optimization
In model-level optimization, architectures mechanisms are designed for transformers to allow more efficient reuse of KV cache. Typically, these methods require retraining or fine-tuning of the model to come into operation. Nevertheless, efficient transformation pipelines have also been proposed to allow for a fast deployment to new architectures. We categorize related works according to their grouping and shar- ing mechanisms, either within layers or across layers (Sec. 5.1), implementing architecture modification or augmentation (Sec. 5.2 and incorporating non-transformer architectures for optimization (Scc. 5.3 The
(axonomy of the level- optimization is shown in Fig. 5
5.1 Attention Grouping and Sharing
This section explores attention grouping and sharing methods as effective strategies for optimizing key-value (KV) management. We categorize the approaches into two distinct subtypes: intra-layer grouping (Scc. 5.1.1) that focuses on grouping query, key, and value heads within individual layers to reduce redundancy and improve efficiency, and cross-layer layer sharing (Sec. 5.1.2) that shares key, value, or attention components across
23

---

Published in Transactions on Machine Learning Research (05/2025)
Table 7: The summary of Model-based Attention Grouping and Sharing approaches.
Applied Location
Intra-layer Grouped Cross-layer Shared Component
Retraining Required
Inyra-layer Cross-layer
Component
MQA
V
Shazeer
2019
GQA
K, V
Uptrain
Ainslin ct. al.
2023
AsymGQA
K, V
Finetune
Chen et al.
20210
Weighted GQA
K, V
Uptrain & Finetune
Chlunskonduru Mobapatra
2024
QCQA
K, V
2024
KDGQA
K,
Khan
2024
GQKVA
V.
Javadi et al
2023
CLA
K, V
K,
Brandon et al.
2024
K, V
Wu & Tu
2024
SA
Attention Weight
Liao Vargas
2024
MTLV V
K,
K,
Uptrain
Zuhri
LISA
Q. K,
Lightweight adaption
Mu
2024
Wu et al.
K, V
Wu et al.
2024e
K, V
Yang et al.
2021e
DHA
1
K, V
Q, K, V
Lightweight adaption
Chen et al.
2024g
SVFormer
Zhou
2024b
layers to improve information reuse and reduce KV cache requirements. The summary of attention grouping and sharing listed in Tab. 7
5.1.1 Intra-layer Grouping
As shown in Fig. 6a Shazeer first introduced Multi-Query Attention (MQA) (Shazeer 2019) that modified the traditional multi-heat attention mechanism. In MQA, all attention heads in transformer block share single key and value. This simple strategy can greatly accelerate the decoding procedure. The experiments of the author show that MQA would gain much efficiency with only minor quality degradation occurring.
MQA is a radical strategy that would cause not just quality degradation, but also training instability. GQA (Grouped Query Attention) (Ainslie al. 2023 introdueed trade-off solution by dividing the query heads into multiple groups, while cach group shares own keys and values. In addition, an uptraining process is proposed to efficiently convert existing MHA models to GQA configurations by mean-pooling the key and value heads associated with each group. Empirical evaluations demonstrated that GQA models achieve performance close to the original MHA models while offering inference time comparable to MQA.
There are several extensions based on GQA. AsymGQA (Chen et al., 2024f exlends GQA by proposing an activation-informed merging strategy. Instead of grouping the heads by uniform clustering, AsymGQA dynamically determines the grouping of queries based on their activation similaritics during training and constructs symmetric group results, which leads to better optimization and generalization. Weighted GQA Chinnakonduru & Mohapatral 2024) introduces additional trainable weights each key and value head,
24

---

Published in Transactions on Machine Learning Research (05/2025)
Car
Ceex
Car
Dire
Memory
UPI
Care
Core
Care
Cere
Circ
Care
Care
Care
Care
Cure
Cwe
CPU 0
CPU
Lapgle
Comple
PCIF Switch
PCIE Switch
GPU 0
GPU
4 4
GPU
MHA
MQA
GQA
GPU
GPU
GPU 6
GPU
(a) Different, QKV techniques.
(b) Computer server architecture.
Figure 6: Three types of QKV and a system architecture.
which can be seamlessly integrated into existing GQA models. By tuning weights during training, improves the performance of the model without additional inference overhead. QCQA (Joshi al., (024) utilizes evolutionary algorithm to identify the optimal query head groupings for GQA, which is guided by computationally efficient fitness function that leverages the weight-sharing error and the KV cache lo evaluate text generation quality and memory capacity. KDGQA (Khan et al. 2024) argues that many variants of GQA adopt a fixed grouping strategy, thus lacking dynamic adaptability to the evolving of key-value interactions during training. Their Dynamic Key- Key-Driven GQA addresses these issues by adaptively allocating groups using key head norms during training, resulting a flexible strategy for query head grouping that enhances performance.
GQKVA (Javadi et al. 2023 advances the grouping strategy and comes up with generalized query, key and value grouping mechanism. It first introduces MKVA and GKVA, in which the key and value are grouped to share the same query. Based on this, GQKVA is proposed to separately group the query and key-value pairs. Typically, queries are partitioned into Dn groups, and keys and values are partitioned into Skv groups, and each combination of query and key-value pairs would interact using dot product attention. This results in Og gkv distinct outputs. It generalizes different group strategics on query, key and value and preserves good computational efficiency and comparable performance as MHA.
5.1.2 Cross-layer Sharing
Brandon et al. introduce Cross Layer Attention (CLA) (Brandon al. 2024h that extends the ideas of GQA and MQA by sharing the key and value heads between adjacent layers, further reducing the redundancy in the KV cache. This achieves an additional 2x KV cache size reduction compared to MQA, significantly improving memory efficiency without altering computational complexity.
LCKV WW & Tu 2024) proposes only to compute and cache the key and value for a small subset of lay- ers, even only the top layer, then let queries in bottom layers pair the saved keys and values for inference. This method not only drastically improves the inference speed and reduces memory consumption but is also orthogonal to existing memory-saving techniques, enabling straightforward integration for further opti- mization. While such mechanism makes next token computation depend top layer keys and values of previous tokens, which contradicts the parallel training of transformers, LCKV introduces an approximate training method to support parallel training.
SA (Shared Attention) [Liao Vargas 2024) proposes reuse of computed attention weights across multiple layers, rather than recalculating them for each layer. Unlike other methods focusing on sharing key-value caches, SA leverages the isotropic tendencies of attention distributions observed in pre-trained LLMs directly share attention weights, greatly reducing both computational overhead and memory usage.
25

---

Published in Transactions on Machine Learning Research (05/2025)
MLKV (Multi-Layer Key-Value) Zuhrii et al 2024p introduces a simple KV head sharing mechanism across multiple transformer layers. MLKV uses the same single KV head as MQA within layer, but it also shares this KV head with multiple layers. This extreme strategy reduces the cache size to almost 1% of normal GQA strategics, and experiments show that MLKV still has comparable performance.
LISA weight Substitute for Attention) (uu et al. 2024) makes a comprehensive analysis of the similar- ity of attention patterns across layers. Directly sharing attention weights across layers is ineffective because of the misalignment of the attention head and the sensitivity of shallow layers. LISA (Mu et al., 2024) addresses challenges by incorporating tiny feed-forward networks to align attention heads between layers and using low-rank matrices to approximate variations in layer-wise attention weights. This achieves a 6x compression of query and key parameters while maintaining high accuracy and perplexity.
Wu et al. 2024e) introduce a unified framework that systematically analyzes and optimizes the cross-layer Key-Value cache sharing mechanism. They consolidate several existing methods, explore novel variants within a cohesive struclure, and make thorough evaluations of these methods. The study finds that two times reduction to KV cache size can outperform standard transformers in throughput without substantial accuracy loss, while further reduction requires alternative design with additional training costs. With the analysis results, they offer insight into the choice of appropriate KV sharing methods based on the specific requirements or constraints.
CLLA (Cross-Layer Latent Attention) (Yang et al. 2024e) introduces an integrated framework combining multiple strategics: attention head size and dimension reduction, cross-layer cache sharing, and KV cache quantization. By unifying these strategies, CLLA achieves extreme KV cache compression to less than 2% of the original model size while maintaining performance levels comparable with uncompressed models.
DHA (Decoupled Head Attention) (Chen et al. 2024eh addresses redundancy in MHA and adaptively con- figures shared groups for key and value heads across layers, reducing KV cache requirements. Observing that clustering and fusing similar heads can reduce KV cache size without significant performance reduction, DHA designs a search, fusion, and continued pre-training [ramework that can progressively transform MHA checkpoints into DHA models through linear fusion of head parameters, preserving the pre-trained knowledge with a small pro-training budget.
Observing that later layers in traditional transformers overly rely narrow regions of allention, Zhou et al. (024bb introduce ResFormer that utilizes residual connections from the value embeddings of the first layer to all subsequent. layers, effectively approximating cross-layer attention without incurring significant computational costs. They then propose a simplified variant SVFormer that shares a single value embedding across all layers, dramatically reducing the KV cache size by nearly half while maintaining competitive performance. The proposed architectures are flexible incorporate with other KV-efficient strategies for additional memory savings.
5.1.3 Summary and Future Directions
This section explores innovative strategies for optimizing memory and computational efficiency through intra- layer grouping and cross-layer sharing mechanisms, while identifying key challenges and future directions. Maintaining performance for precision-sensitive tasks, ensuring scalability across diverse model architectures, and addressing the under-explored dynamics of attention across time and layers remain critical areas for improvement. Current approaches, like DHA (Chen et al. 2024e| and LISA Mu et al. 2024), often struggle to generalize to emerging or non-standard architectures, while static grouping and sharing stratogics fail to capture temporal and contextual attention variations. Future research should focus on developing universal, adaptable frameworks that require minimal retraining, integrating optimization techniques like quantization and pruning, and leveraging dynamic, runtime adjustments to better capture task-specific requirements. Additionally, understanding the downstream impacts on fine-tuning and transfer learning is essential for real-world applicability.
26

---

Published in Transactions on Machine Learning Research (05/2025)
Table The summary of Model-based Intra-layer approaches.
Method
Alteration Type
KV Cache Management
Retraining Requirement
Enhanced Augmented
Attention Architecture
MLA
Latent compression
DeepSeek-AL et
2024
FLASH
Lincar approximation
Hua et al. 2022
Infini-Attentin
Compressive cache
Munkhdalai et al
2024
YOCO
Single global KV cache
Sun et al. 2024c
CEPE
Parallel encoding with cross-attn
Lightweight
Yen et al.
2024
XC-Cache
Encoder cross-attention
Monteiro et al.
2024
Block Transformer
Hierarchical local KV
Lightweight
Ho et sl.
2024
5.2 Architecture Alteration
This section explores architectural modifications to optimize KV cache usage. We categorize these methods into two subsections: methods that refine the attention mechanism for KV cache efficiency (Sec. 5.2.1), and methods that introduce structural changes for better KV management (Sec. 5.2.2 Many of these works build upon the broader landscape of efficient attention mechanisms (e.g., Linear Transformer (Katharopoulos et al. 2020), Performer (Choromanski al.j 2020), LinFormer (Wang al. 2020 ctc.). Since our focus lies on methods directly impacting KV cache handling, for comprehensive overview of efficient attention mechanisms, we refer readers to dedicated surveys (Zhou et al. 2024ch. The summary of architecture altcration for KV rense is listed in Tab.
5.2.1 Enhanced Attention
DeepSeek-V2 (DeepSeek-AT et al. 2024) introdueed Multi-Head Latent Attention (MT.A) that adopts a Jow-rank KV joint compression mechanism, replacing the full KV cache with compressed latent vectors. The model adopts trainable projection and expansion matrices to do the compression. This compression mechanism significantly reduces the memory requirements of the KV cache and enables the model to handle sequences of up to 128K tokens.
FLASH Hua et al. 2022) incorporates the Gated Attention Unit (GAU) to replace the MHA mechanism in traditional transformers. GAU employs a single-head attention mechanism with gating functions that selectively modulate the importance of information flow. FLASH employs linear approximation method for attention computation through GAU module, which makes the model efficiently handle long contexts without the quadratic scaling of traditional self-attention, thus mitigating heavy KV cache issues.
Infini-Attention MMnkhddalai et al. 2024h adopts representation compression to store long-term content. Fur- thermore, they introduce a hybrid allention mechanism of masked local allention and long-term linear alien- tion. The masked local attention replaces the standard MHA to let the model only concentrate on local con- texts, while the long-term linear attention utilizes compressed memory for far-reaching dependencies and uses linear attention for efficient aggregation. Thus, infni-attention combines both local fine-grained and long- range compressed states, allowing seamless balance bet long-term and short-term context. modeling.
27

---

Published in Transactions on Machine Learning Research (05/2025)
Table The summary of Non- -on-Transformer Architectures.
Method
Key Mechanism
No Traditional KV Cache KV Compression
RWKV
Peng et al.
2023:
RNN-like with Transformer parallelism
Mamba
Gu & Dao
2024
Selective state-space model
RetNet
Sun ct al.
2023)
Retention mechanism
MCSD
Yang et al
2024b
Slope-decay fusion
MixCon
Xu & Lin
2024)
Transformer + Conba + MoE
GoldFinch
Goldstein et al.
2024
RWKV Moditied Transformer
RecurFormer
Yan et al.
2024
Mamba replacing some attention heads
5.2.2
Augmented Architecture
YOCO (Sun et al., 2024c) builds dccoder-dccoder architecture composed of two modules: a self-decoder and cross-decoder. The self-decoder efficiently encodes global key-value caches, while the cross-decoder reuses these caches via. cross-attention. This design ensures that key-value pairs are only cached once, substantially reducing GPU memory usage while maintaining global attention capabilities. YOCO's computation flow also enables the prefilling to early exit, allowing faster prefill stages without altering the final output.
CEPE. (Yen et al. 2024) interleaves additional cross-attention layers between the self-attention and feod- forward layers in the decoder model. It employs a small encoder to process long inputs chunk-by- chunk-by-chunk to encoded representations as anssstattentinn layers' inputs. In this way, CEPE can prevent the need for KV cache for every token and reduce computational cost by processing contexts in parallel. This also facilitates an existing LLM to expand its contexts while preserving the scalability and generalizability.
XC-Cache (Monteiro el al. 2024 also utilizes an encoder to interleave cross-attention layers within existing self-attention layers in pre- traineeined decoder-only models to prevent explicit prompt caching. The encoder processes the context and converts it into compact set of kcy-value pairs that summarize the essential information. It also finds that pre-trained causal decoders can be used to replace an encoder for the repre- sentations extraction, further reducing the training costs
additional encoder.
Block Transformer (Ho et al.) 2024) introduces hicrarchical global-to-local architecture by combining coarse- grained global atlention and fine-grained local atlention. In lower layers, tokens are grouped into fixed-size blocks, allowing global context modeling with reduced KV cache overhoad. In upper layers, attention operates within individual blocks, enabling lightweight, detailed token decoding with a smaller local KV cache.
5.2.3 Summary and Future Directions
This section explores research that introduces novel attention mechanisms or architectural modifications to improve KV cache management. Although these approaches demonstrate significant progress in enabling longer context windows and faster inference, several challenges remain. First, many methods, such as CEPE (Yen al., 2024) and XC-Cache (Monteiro et al. 2024) demonstrate strong performance on retricval- augmented lasks but may not generalize well across diverse workloads. This necessitates further research into task-adaptive KV cache optimization strategies that dynamically adjust caching behavior to optimize for different task demands. Secondly, integrating these novel mechanisms into existing pretrained models often requires extensive retraining, hindering their adoption in resource-constraine environments. Developing weigh, modular approaches for retrofitting efficient KV caching into existing architectures is crucial for a wider practical impact. Finally, the robustness and stability of these new mechanisms under real-world conditions, such as noisy or dynamically changing inputs, require further investigation. Addressing these limitations could improve reliability and efficiency in practical deployments.
5.3
Non- Transforfrrr Architecture
While transformers are struggling with KV cache issues, researchers have revisited principles from traditional sequential architectures, such as recurrent neural networks (RNNs) (Salehinejad et. al. 2017). which inher-
28

---

Published in Transactions on Machine Learning Research (05/2025)
ently process sequences without the need for explicit KV caches. Inspired by the lightweight and memory- efficient design of RNNs and efficient attention mechanisms, non-transformer architectures (Xu et al. 2024e;
Hasani et al. 2022 Smith et al. 2022 Wang et al. 2022 Gu & Dao 2024 Peng et al.) 2023) have emerged, such as Mamba [Gu & Dao 2024) and RWKV (Peng et al. 2023). offering promising alternatives. While
there are a large lype of new architectures, we only list methods associated with KV optimization, For further
understanding of efficient non-transformer works, please refer to these surveys (Zhou al., 2024c Xu et al.
2024d
Qu et al. 2024 Patro & Agneeswaran 2024). The summary of non-transformer is listed in Tab.
5.3.1
Adaptive Sequence Processing Architectures
RWKV (Peng et al. 2023), which means Receptance Weighted Key Value, is an architecture that combines the strengths of RNNs and transformers to achieve efficient sequence processing. RWKV integrates a linear attention mechanism, enabling parallelizable training like transformers while retaining the efficient inference characteristics of RNNs. By formulating the architecture to operate as either a transformer an RNN, RWKV achieves constant computational and memory complexity during inference, overcoming the quadratic scaling issues of transformers.
Mamba (Gu & Dao 2024] is built based on state space sequence models (SSMs) (Gu et al.
2022
2021
Inspired by the state space systems, SSMs build scalable and memory-efficient long-range sequence modeling frameworks. Mamba improves SSMs by making parameters input-dependent, allowing information to be selectively propagated or forgotten along the sequence based on the current token. This addresses the inability of traditional SSMs to elfectively handle the complexily of nonlinear dependencies in natural languages. Mamba omits attention and even MLP blocks, relying entirely on these selective state spaces for sequence modeling. It also develops a hardware-aware parallel algorithm for efficient recurrent computations in training and inference. Mamba achieves linear scaling in sequence length, demonstrating exceptional performance on sequences of up to million tokens.
RetNet (Sun al., 2023) introduces the Retentive Network, which combines elements of recurrence and attention, presenting a novel retention mechanism for sequence modeling that enables training parallelism, low-cost inference, and scalable performance. The proposed Multi-scale Retention Module (MSR) supports multiple computation paradigms: the parallel representation, similar to self-attention, allows for causal masking and parallel training; the recurrent representation, akin to RNNs, enables low-cost inference by maintaining state across sequence decoding; and the chunkwise recurrent representation constructs a hybrid of the previous two approaches, further enhancing the ability to handle long sequences.
MCSD (Yang et al., 2024b) introduces a new block called Multi-Channel Slope and Decay, which consists of two sections: the slope section, which captures local features across short temporal spans, and the decay section, which captures global features across long temporal spans. These sections are fused through element- wise operations. During inference, the process is reformulated into recurrent representation, allowing for both spatial and temporal efficiency and minimizing the need to maintain a large KV cache.
5.3.2 Hybrid Architecture
With these non-transformer architectures, some methods construct mixed models to alleviate KV cache necessities while keeping some peculiarities and merits of the self-attention mechanism.
MixCon (uu & Lin 2024) introduces a new architecture called Conba. Inspired by control theory, the Conba layer incorporates a foedback and adaptive control mechanism that can adapt to different sequence- modeling tasks and requirements dynamically with good compulational efficiency. Furthermore, MixCon integrates the Mixture of Experts (MoE) module, which dynamically selects the most relevant experts to process parts of the sequence. Combining the transformer layer, the Conba layer, and the MoE module, MixCon constructs hybrid model with a good balance between attention effectiveness and computational efficiency and significantly reduces the total size of the KV cache.
GoldFinch (Goldstcin et al. 2024 first introduces several new architectures, including the GOLD layer, which combines the Llama and RWKV channel mixer with several improvements, and the enhanced Finch model (RWKV-6) that has significantly reduced parameters without sacrificing efficiency and performance.
29

---

Published in Transactions on Machine Learning Research (05/2025)
GoldFinch also proposes a novel mechanism called TokenCat to produce highly compressed global key cache using the output of Finch layers. GoldFinch builds a hybrid architecture that constructs the key cache in the early layers and consumes the key cache to produce output without the traditional value cache in the top layers, providing a compact and reusable cache pipeline with lincar scaling.
RecurFormer (Yan et al. 2024) argues that not all transformer heads need to participate in the self-attention mechanism. The authors observe that certain attention heads exhibit recency-aware behavior, focusing on local and short-range dependencics. These heads consume computational resources but contribute little to overall performance. After identifying such heads, RecurFormer replaces them with Mamba components, resulting in straightforward KV cache reduction.
5.3.3 Summary and Future Directions
By exploring non-transformer modules such as recurrent and hybrid designs, these methods have introduced novel paradigms that balance performance with computational efficiency, and also alleviate the KV cache issues in traditional transformer architectures. Future research should focus on several key areas. First, improving the scalability of recurrent architectures, such as RWKV (Peng et al. 2023) and Mamba (Gu & Dao 2024). remains critical. Although these methods reduce memory and computational costs, their performance in capturing ultra-long-range dependencies lags behind transformers. Second, hybrid designs such as MixCon (Xu & Lin 2024) and GoldFinch (Goldstein et al., 2024) highlight the potential of integrating diverse modules, yet their complexity introduces challenges training stability and interpretability. Third, the overall generalization capabilities and robustness of non-transformer architectures need exploration for diverse imput modalities.
6 System-level Optimization
As shown in Fig. [b] existing computing server systems consist. of various components, such GPUs, CPUs, and memory storage. Optimizing LLM acccleration via KV cache at the system level is both practical and intriguing, considering the different communication and data exchange mechanisms. Recent system-level optimizations for KV cache in LLM inference can be broadly categorized into three main directions: memory management (Sec. 6.1), scheduling strategies (Sec. 6.2), and hardware-aware designs (Sec. 6.3) These complementary approaches collectively demonstrate the rich design space for system-level optimizations in
LLM inference, each addressing different aspects of the performance, efficiency, and resource utilization
challenges. The taxonomy of the system-level optimization is in Fig. 7
6.1 Memory Management
Recent advances in KV cache memory management for large language model (LLM) inference reveal three distinct approaches aimod at enhancing memory efficiency. Architectural designs, exemplified by vLLM with PagedAttention (Kwon al. 2023) and vTTensor (Xu et al. (024c), adapt classical operating system princi- ples to create flexible, dynamic memory allocation systems that optimize the use of physical memory through sophisticated mapping and virtual memory abstractions. Prefix-aware designs like Chunk (Ye et al. 2024) and MemServe (Hu et al. 2024) further refine this approach by organizing data structures to enable ef- ficient cache de-duplication and sharing of common prefixes, thereby improving both memory utilization and computational officiency. Together, these innovations illustrate the potential for significant enhanccments in LLM serving via memory management.
6.1.1 Architectural Design
The first category focuses on architectural innovations in memory management, led by vLLM with Page- dAttention (Koon el al. 2023), which adapts OS-inspired paging concepts by partitioning KV caches inlo fixed-size blocks with non- contiguots storage. PagedAttention partitions KV caches into fixed-size blocks that can be stored non-contiguously in physical memory, while vLLM (Kwon et al.) 2023) implements a virtual memory-like system that manages these blocks through sophisticated mapping mechanism. This architecture separates logical and physical KV blocks, enabling dynamic memory allocation and flexible
30

---

Published in Transactions on Machine Learning Research (05/2025)
Architectural De- sign (Sec. 6.1.1
vLLM Kwon ct al.
2023
VTensor [Xu et al.
2024c
Memory
LeanKV Zhang et al 2024
Management (Sec. 6.1
Prefix-aware
ChunkAttention Yoo ct al.
2024
sign (Sec. 6.1.2
MemServe (uu et al. 2024
Prefix-aware Schedul- ing (Sec. 6.2.1
BatchLLM (Zheng al. 2024b
RadixAttention (Zhong al.
2024a
Scheduling (Sec. 6.2
Preemptive and Pairnoss-oricnted Schoduling (Sec. 6.2.2
FastServe (Wu et al.
2024a
FastSwitch (hen et al.
2024
Layer-specific and Hlerarchical Schedul- ing (Sec. 6.2.3
LayerK (Xiong of
2024).
Cached Attention Gao et al.
2024a).
ALISA
(Zhao et al.
System-level Opti-
2024c
LAMPS (Shahout et al.
mization (Sec. Gp
2024
Single/Multii-e Design (Sec. 6.3.1
HydraGen (Juravsky al.
2024).
DeFT @Yao al. (022bb)
vLLM Kwon et al. 20223
ORCA
et et al. [002
Dist-
Serve [Zhong et al.
[024aa)
Multi-Bin Batching
(Mindo
gan ct al.
2024 Tree Atten-
tion
(Shyam ct al.
2024
1/O-based
Flash FlashAttentoon (Dao al.
Design (Sec. 6.3.2
2022
(Dao 2024) (Shah
al.
2024), Bifurcated At-
tention
(Athivaratkun et al.
2024
PartKVRee [Jiang et al.
Hardwaro-aware Design (Sec. 6.3
2024b
HCache (Gao ct al.
2024b
Cake (Jin ct al
2024
FastSwitch (Shen ct al.
2024
Heterogencous
NEO (Jiang et al.
2024c
Fast-
Design (Sec. 6.3.3
Decode [He Zhai
2024
Flexinfer [Xu ct al.
2024c
InfiniGen (Lee et al.
2024p.
Pensieve Yu & Li]
2023).
FastServe
[Wu et. al..
2024a
PartKVRoc [Jiang et al,
2024bb
HeadInfer (Luo et al.
2025
SSD-bbusdd
FlexGen (Sheng 2024
2023 In-
Design (Sec. 6.3.4
stInfer (Pan et al.
Figure 7: Taxonomy of the System-level Optimization for KV Cache Management.
block management through block tables that track mapping relationships and fill states. This memory man- agement approach enables efficient memory utilization both within and across requests, demonstrating how classical OS memory management principles can be effectively adapted for LLM inference optimization.
This approach is further enhanced by v'Tensor (Xu et al. 2024ch, which introduces virtual memory ab- straction that decouples computation from defragmentation through three key components: the vTensor Scheduler which generates memory management policies based on meta information, the veesoor Operation which translates these policies into CUDA VMM operations, and the vTesorr Pool which maintains virtual tensor mappings. VTS processes instructions and creates policies based on memory state tracking, while VTO executes these policies through asynchronous GPU operations. VTP completes the cycle by managing virtual tensor storage and updating meta information for subsequent memory operations.
31

---

Published in Transactions on Machine Learning Research (05/2025)
Table 10: Comparison of Memory Management Techniques for KV Cache Optimization.
Paged Memory
Virtual
Dynamic Sparsity
Prefix Distributed
Method
Memory
Sharing
Memory
vLLM
Kwon et al.
2023
VTensor
(Xu et al.
2024c
LeanKV
Zhang et al. 20246
ChunkAttention
Ye et al.
2024
MemServe
Hu et al. 2021)
LeanKV (Zhang et al. 2024f) combines unified paging with heterogeneous quantization and dynamic sparsity mechanisms. It implements Hetero-KV quantization store keys and values different precisions, comple- mented by per-head dynamic sparsity mechanism that adapts memory allocation based on token impor- tance across different attention heads and requests. To efficiently execute these strategies, LeanKV (Zhang ct al. 2024f) introduces an advanced on-GPU memory management system featuring three key components: unified paging for flexible memory organization, a circular free page list for efficient coordination, and a bidirectional page table for minimal metadata overhead.
6.1.2 Prefix-aware Design
Some latest works emphasize optimizing data organization structures through prefix-aware designs. ChunkAttention (Ye et al. 2024) restructures KV cache management by organizing chunks within a profix tree structure, enabling runtime detection and sharing of common prefixes. It breaks down traditional mono- lithic KV cache tensors into smaller, manageable chunks organized within a prefix tree structure, enabling efficient runtime detection and sharing of common prefixes across multiple requests. This architectural design brings two significant memory management benefits: efficient KV cache deduplication through prefix tree- based organization, and improved data locality through two-phase partition algorithm for self-attention computation. By cnabling dynamic identification and sharing of common prompt prefixcs across multiple requests, Chunk (Ye et al. (024)) optimizes both memory utilization and computational efficiency, demonstrating how intelligent chunking and prefix-ave aware cache management can significantly enhance LLM serving efficiency.
MemServe (Hu et al., 2024 extends this concept to distributed settings with its MemPool system, which orchestrates both CPU DRAM and GPU HBM resources across serving instances, managing active and historical KV caches through a comprehensive set of distributed memory pool APIs. It presents prompt token- based indexing layer for historical KV cache retrieval, cross-instance data exchange mechanisms that abstract away hardware heterogeneity, and a global scheduler implementing a prompt tree-based locality- aware policy for enhanced cache reuse, collectively resulting in significant improvements job completion time and foirst--fisttto performance.
These approaches often complement each other, suggesting potential benefits in combining multiple strate- gies. For instance, LeanKV (Zhang et al. 2024f 's integration of compression with page-based management and MemServe (Hu et al. 2024)'s 's combination of distributed memory management with prefix-aware caching demonstrate the effectiveness of hybrid approaches. The diversity of these solutions reflects both the com- plexity of KV cache management and the rich opportunity space for continued innovation in optimizing LLM inference systems. Tab 10 provides a comparison of various memory management techniques for KV Cache, highlighting key features such paged memory, virtual memory, dynamic sparsity, prefix sharing, and distributed memory.
6.1.3 Summary and Future Directions
The exploration of memory management strategics for KV caches in large language model inference reveals a promising landscape of innovations that enhance memory efficiency and overall system performance. Archi- tectural advancements, such as those seen in vLLM (Kwon et al. 2023] and LeanKV (Zhang al. 20240
32

---

Published in Transactions on Machine Learning Research (05/2025)
adapt traditional memory management principles for modern AI applications by incorporating paging and virtual memory concepts for dynamic allocation. Prefix-aware designs like ChunkAttenin (Ye et al. 2024) and Memnerve (Hu et al. 2024) optimize data organization, enabling the detection and sharing of common prefixes, which reduces redundancy and speeds up inference.
Future memory management work should prioritize adaptive hierarchies, novel compression techniques, intel- ligent prefetching, and hardware-aware optimizations utilizing new memory technologies, alongside efficient distributed cache coherence. Exploring machine learning for predictive allocation, specialized data structures, and heterogeneous memory systems is also vital for enhancing LLM inference scalability and efficiency.
6.2 Scheduling
Based on these scheduling-oricnted works, we can categorize KV cache scheduling optimizations into three main approaches: 1) prefix-aware scheduling strategies, represented by BatchLLM (Zheng et al. 2024b) and RadixAttention [Zheng et al. 2024a); 2) preemptive and fairness-oriented scheduling, exemplified by FastServe (Wu et al., (0244a) and FastSwitch (Shen et al. 2024); 3) layer-specífic and hicrarchical schoduling approaches, demonstrated by LayerKV (Xiong et al.) 2024), CtteeiAttentiin (Gao et al.) 2024a). and AL- TSA (Zhao et. al. 2024c). These approaches collectively address different. aspects of scheduling optimization, from memory efficiency to fairness and latency reduction, while specialized solutions like LAMPS (Shahout ct al. 2024 extend these concepts to specific use cases such as APL-angmented LLM requests, demonstrating the rich design space in KV cache scheduling optimization.
6.2.1 Prefix-aware Scheduling
Unlike traditional LRU-based cache management systems where shared KV contexts might be prematurely evicted or unnecessarily extended in memory, BatchLLM (Zheng et al., 2024b) implements explicit global prefix identification and coordinated scheduling of requests sharing common KV cache content. It schedules requests the granularity of prefix-sharing groups, ensuring optimal KV cache reuse while minimizing cache lifetime- requests with identical prefixes are deliberately scheduled together to maximize KV cache sharing efficiency. This scheduling approach is complemented by dynamic programming algorithm that optimizes first-level prefix patterns, enabling more eflicient KV cache management and reducing scheduling overhead.
RadixAttention (Zheng et al. 2024a) builds around a radix tree structure, replacing traditional FCFS scheduling with an intelligent cacho-aware approach that prioritizes requests based on matched prefix lengths. Ii implements dynamic memory management where cached Lokens and running requests share the same mem- ory pool, controlled by an LRU eviction policy that strategically removes leaf nodes while preserving valuable ancestor prefixes. This is complemented by reference counting mechanism that prevents eviction of ac- tively used cache entries during continuous batching while enabling efficient memory reclamation when nodes become unused.
6.2.2 Preemptive and Fairness-oriented scheduling
FastSSrvv (u et al. 2024a) implements proactive KV cache management strategy that coordinates cache movement between GPU and host memory, overlapping data transmission with computation to minimize latency impact. This is integrated with a skip-join Multi-Level Feedback Queue scheduler that makes KV cache scheduling decisions based on input length information, allowing jobs to enter appropriate priority queues directly while avoiding unnecessary demotions through higher-priority qucucs. By combining token- level preemption with sophisticated KV cache management and intelligent. queue placement, FastServe (Wul et al. 2024a) achieves significant performance improvements over traditional run-to-completion systems like vLLM Kwon et al. 2023).
FastSwitch (Shen et al. 2024) introduces a fairness-oriented KV cache scheduling system that addresses the overhead challenges of preemptive scheduling in LLM serving. There are three key mechanisms: enhancing I/O utilization through intelligent cache movement scheduling, minimizing GPU idle time during context switches, and eliminating redundant I/O operations in multi-turn conversations. Unlike traditional block- based KV cache memory policies that prioritize memory efficiency at the cost of fragmentation and gran-
33

---

Published in Transactions on Machine Learning Research (05/2025)
Table 11: Comparison of Scheduling Approaches for KV Cache Optimization.
Method
Prefix-aware Preemptive Fairness-oriented Layer-specific Hierarchical Dynamic
BatchLLM
Zheng et al.
2024b
RadixAttention
Zheng et al.
2024a
Serve
Wu et
2024a.
FastSwine witch
Shen al.
2024
LayerK K
Xiong al.
2024
Cached ention
Gao et al.
2024a
ALISA A
Zhao et. al.
2024c
LAMPS
Shahout et al.
2024
ularity limitations, FastSwitch (Shen al. 2024 implements a balanced approach that maintains efficient memory usage while facilitating smoother context switching. This integrated schoduling approach enables dynamic priority adjustments for fairness while minimizing the performance impact of context switches.
6.2.3 Layer-specific and Hierarchical Scheduling
LayerKV (Xiong et al. 2024) introduces a novel layer-wise KV cache schoduling approach to address the growing TTFT (Time to First. Token) latency challenges in large-contexl LLM serving. The contribution lies in its tine-graindd layer-specific KV cache block allocation and management strategy, which departs from traditional monolithic cache management approaches. By implementing layer-wise KV block scheduling and offloading mechanisms, LayerKV (Xiong et al., 2024) enables more efficient memory utilization and reduces queuing delays that typically occur when large context windows compete for limited GPU KV cache blocks. It complemented by an SLO-aware scheduler that optimizes cache allocation decisions based on service level objectives, allowing for dynamic management of memory resources across model layers.
AachetAAttntioon (Gao al. 2024a introduces hierarchical scheduling approach consisting of three-tier strategies: layer-wise pre-loading coordinates KV cache movement across storage hierarchies using scheduler- aware fetching and eviction policies, asynchronous saving overlaps I/O operations with GPU computation, and intelligent cache placement decisions are made based on scheduler hints to ensure frequently accessed KV caches reside in faster memory tiers. It also presents a novel positional encoding decoupling mechanism that prevents KV cache invalidation during context window overflow through effective truncation strategies.
ALISA (Zhao et al., 2024c) introduces a dual-level KV cache scheduling framework that combines algorithmic sparsity with system-level optimization. At the algorithm level, the Sparse Window Attention mechanism identifies and prioritizes the most important tokens for allention computation, creating a mixture of global dynamic and local static sparse patterns that significantly reduce KV cache memory requirements. At the sestem-level, its three-phase token-level dynamic scheduler that manages KV tensor allocation and optimizes the trade-off between caching and recomputation. The scheduler makes dynamic decisions about which to- kens to cache in GPU memory versus recompute, based on their importance and system resource constraints.
LAMPS (Shahout al. 2024) implements a predictive scheduling mechanism that estimates both pre-API outputs and optimal memory handling strategies during API calls, choosing between preserving, discarding, or swapping KV cache content based on predicted memory waste.
6.2.4 Summary and Future Directions
Tab compares scheduling approaches for KV cache optimization based on their support for prefix- awareness, preemptive scheduling, fairness, layer-specific optimizations, hierarchical structures, and
34

---

Published in Transactions on Machine Learning Research (05/2025)
dynamic adaptability. The advancements in scheduling strategies for KV cache management in large language model inference highlight a multifaceted approach to optimizing performance, memory efficiency, and fairness. By categorizing these strategies into prefix-aware, preemptive and fairness-oriented, and layer- specific scheduling, we see diverse methodologies addressing different challenges. For instance, prefix-aware strategies like BatchLLM (Zheng et al., 2024b) and RadixAutention (Zheng al. 2024a) enhance cache reuse by intelligently grouping requests based on shared prefixes, minimizing cache lifetime and reducing overhead. Meanwhile, preemptive approaches such as FastServe (Wu et al.) 2024a) and FastSwitch (Shen) et al. 2024) implement proactive management techniques that optimize cache movement and schedulingg, significantly improving latency and ensuring fairness during context switching. Layer-specific scheduling methods like LayerKV (Xiong et al.) 2024), CachedAttention (Gao et al. 2024a), and ALISA (Zhao et al., 2024c) further refine cache allocation by implementing fine-grained management strategies tailored to the unique demands of different model layers.
Future KV cache scheduling work should focus on adaptive and predictive systems, automated tuning, contexi-aware architectures, and novel coherence protocols. Integrating reinforcement learning and hardware- software co-design will further enhance LLM inference system robustness, efficiency, and adaptability. Finally, considering LLM serving (Yao et al., 2024a). different scheduling and sharing for multiple users and queries may lead to potential privacy leaks. Therefore, privacy protection techniques for LLM serving in multi-user
scenarios, such as differential privacy (Zhao & Chen 2022
Dong & Yi 2021 Dong et al. 2023a ) are worth
further investigation.
6.3 Hardware-aware Design
Recent hardware-aware optimizations for KV cache management span several key directions based on differ- eut hardware architectures and constraints. Single/Multi-GPU desigus focus on optimizing memory access patterns, GPU kernel designs for efficient attention computation, and parallel processing with load bal- ancing. TO-based designs optimize data movement across memory hierarchies through asynchronous I/O and intelligent prefetching mechanisms. Heterogeneous designs orchestrate computation and memory allo- cation across CPU-GPU tiers. SSD-based solutions have evolved from basic ofloading approaches to more sophisticated designs, with InstInfer leveraging computational storage drives (CSDs) to perform in-storage attention computation, effectively bypassing PCIe bandwidth limitations. These approaches demonstrate how hardware-aware designs can signíficantly improve LLM inference efficiency by carefully considering and exploiting the characteristics of different hardware components and their interconnections.
6.3.1 Single/Multi-GPU Design
Based on these works focusing on GPU-oriented designs, we can categorize the approaches into several key strategies for KV cache oplimization. First, shared prefix optimization approaches like HydraGen (Juravsky et al., 2024) and DeFT (Yao al.) (0224b) focus on efficient GPU memory utilization through batched prefix computations and treo-structured attention patterns. Rather than maintaining separate KV caches for cach sequence with identical prefixes, HydraGen (Juravsky et al. 2024) decomposes attention computation to leverage a single shared KV cache for common prefixes across multiple requests. It. enables efficient GPU memory utilization through two mechanisms: batched prefix KV cache access across sequences and separate handling of unique suffix KV caches. For DcFT (Yao et al.] 2024b), its core contributions are twofold: KV- Guided Grouping, which optimizes GPU memory access patterns by intelligently managing shared prefix KV caches to minimize rodundant global-to-shared memory transfers, and Flattened Tree KV Splitting, which en- sures balanced workload distribution across GPU compute units while minimizing computational redundancy.
Second, distributed processing frameworks exemplified by vLLM (woon et al. 2023 and ORCA Yu et all 2022) optimize multi-GPU scenarios through sophisticated memory management and synchronization mech- anisms. vLLM (Kwon et al.) 2023) also implements a KV cache manager that coordinates memory alloca- tion across distributed GPU workers in paraalel deployments, where each GPU handles a subset of attention heads while sharing the same logical-to-pp physscaall block mapping. This GPU-aware design enables efficient memory utilization through near-zero fragmentation and flexible KV cache sharing, while supporting Megatron-LM style tensor parallelism where GPUs execute in SPMD fashion with synchronized block-wise
35

---

Published in Transactions on Machine Learning Research (05/2025)
matrix operations. The scheduler broadcasts control messages containing input tokens and block tables to GPU workers, allowing them to independently process their assigned attention heads while maintaining memory coherence through all-reduce operations, effectively eliminating redundant memory management synchronization overhead and maximizing GPU utilization across distributed resources.
ORCA (Yu et al. )022) distributes model layers across GPUs using both intra-layer and inter-layer par- allelism, where each worker process manages multiple GPU-controlling threads and coordinates KV cache access through
Attention KV manager. ORCA's GPU-aware design minimizes CPU-GPU synchroniza- tion overhead by separating control message communication from tensor data transfer (via NCCL), allowing each GPU thrcad to efficiently access KV cache memory using request IDs and token indices.
Third, phase-aware designs like DistServe (Zhong et al. 2024a] separate prefill and decoding phases across GPU resources to optimize their distinct memory access patterns. Novel batching strategies are represented by Multi-Bin Batching (Guldogan et al. 2024). which focuses on length-aware roquest grouping for improved GPU utilization, while advanced parallel computation frameworks like Tree Attention (Shyam et al. 2024) introduce sophisticated reduction algorithms for efficient attention computation across multiple GPUs. Dist- Serve (Zhong et al., 2024al recognizes that prefill and decoding phases have distinet KV cache utilization characteristics and memory access patterns: prefill requires intensive computation with growing KV cache sizes for processing input tokens, while decoding maintains a fixed KV cache size for generating output to- kens. By physically separating these phases onto different GPUs, DistServe enables optimized GPU memory management and KV cache access patterus specific to each phase, eliminating interference between prefill's bursty memory access patterns and decoding's 's steady-state KV cache utilization. Multi-Bin Batching (Gul-) dogan et al. 2024) introduces length-aware batching strategy helps minimize GPU idle time and memory fragmentation that typically occurs when processing requests of varying lengths in the same batch, as it ensures that the KV cache memory allocated for each batch is utilized more uniformly across all requests. Tree Attention (Shyam et al. 2024) implements a tree-based reduction algorithm that fundamentally changes how attention values are computed and aggregated across GPUs, enabling more efficient handling of KV cache data through partial reductions that significantly reduce memory bandwidth requirements and peak memory usage.
These approaches can collectively demonstrate how hardware-aware dosigns can significantly improve the LLM elliciency by carefully considering GPU archilecture characteristics and memory hierarchy constraints.
6.3.2 1/O-based Design
Recent I/O-focused optimizations for KV cache management span several key dimensions, targeting different levels of the memory hierarchy. At the GPU level, approaches like F'ashAAtenttion (Dao et al.) 2022) (Dao 2024] (Sahh et al., 2024) and Bifurcated Attention (Athiwaratkun et al., 2024) optimize data movement be- tween HBM and SRAM through sophisticated liling strategies and split allention computations, while CPU- GPU data movement optimizations are addressed by systems like PartKVRee (Jiang et al., (024bb, which tackles PCIe bandwidth bottlenecks through hybrid recomputation and transfer strategics, and HCache (aoo et al. 2024b). which optimizes intermediate activation storage and restoration.
2024 (Shah et al. 2024 employs tiling strategy that carefully manages KV cache access patterns, reducing redundant memory operations by keeping frequently accessed portions of the KV cache in fast SRAM while systematically fetching and evicting data blocks to minimize HBM accesses. Bifurcated Attention (Athiwaratkun al. 2024) presents an T/O-aware approach to op- timize KV cache access patterus during shared-context batch decoding by strategically splitting attention computations into two distinct GEMM operations. It specifically targets the memory bandwidth bottleneck in high-batch scenarios with long contexts by minimizing repeated KV cache accesses, maintaining the same computational FLOPs while drastically reducing memory I/O operations. For PartKVRec (Jiang et al. 2024b), its key innovation lies in its hybrid strategy of partial KV cache recomputation on the GPU while simultaneously transferring the remaining cache data from CPU memory, effectively hiding PCLe transfer latency. The implementation employs sophisticatod I/O-aware scheduling system that analyzes input characteristics and hardware capabilities to determine the optimal balance between recomputation and data transfer, dynamically managing KV cache movement to maximize PCTe bandwidth utilization while mini-
FlashAttention (Dao et al. 2022 Dao
36

---

Published in Transactions on Machine Learning Research (05/2025)
mizing GPU idle time. HCache (Gao et al. 2024b) strategically stores and restores intermediate activations instead of complete KV cache states, implementing a bubble-free restoration scheduler that carefully balances computation and 1/O operations to maximize bandwidth utilization. A key innovation is its chunk-based storage manager that addresses the I/O pattern mismatch between saving (layer-before-token) and restora- tion (Loken-before-layrr) operations, optimixing data layout and access patterns to reduce I/O overhead. Cake (Jin et al., 2024) addresses the fundamental I/O bottleneck in loading cached KV states from disk to GPU memory. It introduces a bidirectional parallelized strategy that simultaneously leverages both compu- tational and I/O resources. This hybrid approach dynamically balances between loading cached KV states from storage and computing them on GPUs, adapting automatically to varying system conditions without manual parameter tuning.
Context management optimizations are exemplified by FastSwitch (Shen et al. 2024). which implements effi- cient context switching mechanisms for multi-user scenarios through granular memory management policies. FastSwitch (Shen et al., 2024) addresses I/O inefficiencies in traditional block- based KV cache approaches by implementing a more granular and continuous memory management policy that minimizes I/O overhead during preemption and context switching.
These approaches demonstrate how careful consideration of I/O patterns and memory hicrarchy charac- teristics can significantly improve LLM inference efficiency by minimizing data movement and maximizing bandwidth utilization across different storage tiers.
6.3.3 Heterogeneous Design
Recent heterogencous computing approaches for KV Cache demonstrate diverse strategics for optimizing CPU-GPU collaboration. Systems like NEO (Jiang et al., 2024c),, FastDecood (He & Zhai 2024) and HeadInfer (Luo et al. 2025) implement strategic workload distribution through CPU offloading of attention computations, while Flexlnfer coordination.
(Xu et al. 2024ch introduces virtual memory abstractions for optimal resource
NEO (Jiang et al. 2024c) advances heterogeneous computing for LLM inference by implementing strategic CPU offloading of attention computations and KV cache states. Through asymmetric GPU-CPU pipelining and load-aware scheduling, it optimally balances workloads across both computing platforms, enabling larger GPU batch sizes without latency penalties. For FastDecode (He & Zhai, 2024), its key contribution lies in its strategic offloading of memory-bound KV cache operations to distributed CPU resources, leveraging the aggregate memory capacity and computing power of multiple CPU nodes rather than treating CPUs mere storage devices. By utilizing CPUs for KV cache computations and storage while keeping compute- intensive operations on GPUs, it creates an efficient pipeline that maximizes resource utilization across the heterogencous infrastructure, enabling larger batch sizes and higher throughput. FlexInfer (Xu et al. 2024c) orchestrates CPU-GPU resource utilization for LLM inference by introducing the virtual memory- based abstraction vTensor. By implementing fine-grained, selective offloading of attention heads' KV cache to CPU RAM while dynamically computing attention outputs, HcadInfer (Luo al.) 2025) achieves remark- able memory efficiency without compromising computational performance. It also supports both dense and sparse attention mechanisms, integrates with pipeline parallelism for larger models. Unlike NFO (Jiang et al. 2024c), which focuses on strategic CPU offloading with asymmetric GPU-CPU pipelining, and Fast- Decode (He & Zhai 2024), which distributes KV cache operations across multiple CPU nodes, HeadInfer distinguishes itself through its granular head-wise offloading strategy that eliminates the need to fully store KV cache for any transformer layer on GPU while preserving complete mathematical equivalence without approximation.
Advanced caching and prefetching mechanisms are exemplified by InfiniGen (Lee et al. 2024h, which employs speculative prefetching for KV cache entries, and Pensieve YYu & Li 2023), which implements multi-tier caching for conversation states. For InfiniGen (Lee et al.) 2024), its key innovation lies in its prediction mechanism that operates across the heterogeneous architecture, using partial computation of attention in- puts and modified query-key weights to identify and prefetch only the most relevant KV cache entries from CPU memory to GPU. Pensieve (Yu & Li, 2023) introduces a heterogeneous computing architecture specif- ically designed for multi-turn conversation LLM serving by implementing a sophisticated multi-tier caching
37

---

Published in Transactions on Machine Learning Research (05/2025)
Table 12: Comparison of Hardware-aware Design Approaches for KV Cache Optimization.
Method
Single/Multi-GPU 1/O-aware
Heterogeneous
SSD-based
Bifurcated Attention
Athhiaaattkun al.
2024
Cake
Jin et al.
2024)
DeFT
Yao et al.
2024b
DistServe
Zhong al.
2024a
Fast.Deoode
He & Zhai
2024
FastSwitch
(Shen et al.
2024
FlexGen
Sheng al.
2023
FlexInfer
Xu et
2024ch
FlashAttention
[ao et 2022
Dau
2024
Shah ct al.
2024
HCache
Gao et al.
2024b
HydraGen
Juravsky et al. 2024
InfiniGen
Lec al.
2024
InstInfer
Pan
2024
Multi-Bin Batching
[Guldggan ct al.
2024
NEO
(Jaang et al.
2024c
ORCA
Yu ct al.
2022
PartK Ree
Jiang et al. 2024b
Pensieve
Yu & Li
2023
Tree Attention
Shyam et al.
2024
vLLM
Kwon al.
2023
strategy across GPU and CPU resources. This stateful approach manages KV cache data across the het- erogeneous memory hierarchy, maintaining conversation history states across multiple hardware tiers rather than recomputing them for each interaction.
Sophisticated scheduling and preemption strategics are demonstrated by FastServe (Wu et al. 2024a), which focuses on token-level preemption and proactive memory management, and PartK VRec (Jiang el al. 20024b) which balances data transfer and recomputation through dynamic scheduling. For FastServe (Wu et al. 2024a), its token-level preemption capability supported by sophisticated heterogeneous memory man- agement system that proactively coordinates KV cache data movement between GPU and host memory. It implements a skip-join Multi-Level Feedback Queue scheduler that manages compulational resources across the CPU-GPU boundary, optimizing both computation scheduling and data movement. PartKVRec (Jiang) ct al.
2024b employs scheduler that dynamically optimizes the distribution of tasks across the heteroge- neous hardware platform, using a profiler to analyze both hardware capabilities and workload characteristics.
These approaches collectively showcase how heterogeneous architectures can be effectively leveraged to over- come single-device limitations while maintaining efficient resource utilization and minimizing communication overhead between CPU and GPU resources.
6.3.4 Solid-state Disk (SsD)- Design
Recent SSD-based approaches for KV cache management demonstrate an evolution in storage utilization strategies, from traditional extension of the memory hierarchy to computational storage innovations. Flex- Gen (Sheng et al.) 2023) introduces an SSD-basod approach to KV cache management that extends the memory hierarchy across GPU, CPU memory, and disk storage, optimizing high-throughput LLM inference on rssource-constraine hardware through intelligent tensor storage and access pattern optimization deter- mined by lincar programming. The system's key innovations include coordinated data placement across all three storage tiers, optimized access patterns minimize SSD latency impact, aggressive 4-bit compression for both model weights and attention cache, and efficient utilization of SSD storage memory hierarchy
38

---

Published in Transactions on Machine Learning Research (05/2025)
extension for KV cache management. InstInfer (Pan et al. 2024) introduces more revolutionary approach by leveraging computational storage drives (CSDs) to perlorm attention computations directly within the storage layer, transforming SSDs from passive storage devices into active computational units and utilizing the high internal bandwidth of flash memory channels to bypass traditional PCIc bandwidth limitations.
These approaches demonstrate how storage devices can be effectively integrated into LLM inference systems, either as memory hierarchy extensions or as computational resources, enable efficient processing of large models and long sequences in resource-constrained environments. Tab [2] compares hardware-aware design approaches for KV cache optimization across four key features: Mtte//Mutti-GPP support, 1/O-awareness, heterogencous computing, and SSD-based design.
6.3.5 Summary and Future Directions
Recent advancements in hardware-aware designs for KV cache management emphasize optimizing perfor- mance based on specific hardware architectures and constraints, demonstrating significant enhancements in large language model inference efficiency. Approaches like HydraGen (Juravsky et al. 2024) and vLLM (Kwon et al. 2023) in single and multi-GPU designs focus on efficient memory access palterns and load balancing, while 1/O-based strategies such as FlashAttention (Dao et al., 2022 Dao 2024 Shah et al. 2024) and PartKVRec (Jiang et al.) (024b) tackle data movement bottlenecks through intelligent prefetching and scheduling mechanisms. Additionally, heterogeneous designs exemplified by NEO (Jiang al.
2024c and
Fast.Decode (He & Zhai 2024) effectively leverage CPU-GPU collaboration to maximize resource utilization.
Future research will focus on enhancing LLM inference systems through interconnected advancements in architectural design, hybrid systems leveraging computational storage and in-memory processing, adaptive resource allocation algorithms, advanced compression techniques, and intelligent scheduling for heterogeneous computing. These efforts aim to improve performance and scalability across diverse deployments while adapting to new hardware and demands.
7
Long-context Text and Multi-modal Benchmarks
In this section, introduce the text and multi-modal datasets used to evaluate LLM efficiency.
7.1
Text Benchmarks
We collect a lot of long-context datasets, such as NumericBench (Li al., 2025 LongBench (Bai al. 2023 We categorize these datasets into various tasks, including question answering, text summarization, text reasoning, text retricval, text generation, and aggregation. The bricf descriptions of each datasct are provided below, and the corresponding statistics are listed in Tab. 13
2025 is benchmark designed to evaluate the fundamental numerical reasoning capabilities of LLMs, em emphasizing their ability to understand the meaning of numbers.
NumericBench [L al.
2024 is a synthetic benchmark designed to evaluate the contecontext capabilities of LLMs across diverse task categories, including retrieval, multi-hop tracing, aggregation, and QA.
RULER (Hsieh al.
2025 is a multilingual benchmark that evaluates long-context language models OneRuler (Kim et al.
across 26 languages.
2023 is a benchmark for evaluating long-context language models (L.CLMs) across 20 diverse inskssincludingg closed-ended (e.g., reasoning) and open-ended (e.g., summarization) scenarios.
L-Eval (An et al.
2023) introduces new benchmark for LLMs' long-contexs comprchension, covering 36 datasets across 11 tasks and 12 domains, with input lengths from 1K to 8K words.
M4LE (wan et al.
BAMBOO (Dong al. 2023b evaluates L.LMs' long-text understanding across 5 tasks, including QA, hallucination detection, text sorting, language modeling, and code completion.
LongBench (Bai et al.] 2023) is a comprehensive bilingual benchmark designed evaluate the long- context understanding capabilities of large language models.
39

---

Published in Transactions on Machine Learning Research (05/2025)
Table 13: Long-context Text Benchmarks.
Tasks
Benchmark
Language
Q-A Summarization Reasoning Retrieval Generation Aggregation
NumcricBench
/
EN
[Li et al. 2025
RULEK E
/
EN
Hsich et al.
2024
One er
26 languages
Kim et al.
2025
Eval
V
EN
An et al.
2023
M4LE
V
EN/ZH
Kwan et al.
2023
BAMBOO
v
EN
Dong et al.
2023b)
LongBench
EN/ ZZH
Bai et al.
2023
SCROLLS
EN
Shaham al.
2022)
ZEROSCROLI
EN
Shaham al.
2023)
Loo
EN
Li et al. 2023a.
Long Eval
EN
[i* et al.
2023
Streaming Eval
EN
Xiao et al.
2024c)
SCROLLS (Shaham et al. 2022 a benchmark designed evaluate models' ability to process naturally long texts across diverse domains such literature, science, and entertainment.
ZEROSCROLLS (Shaham et al. 2023 serves as a benchmark aimed at assessing the zero-shot reasoning abilities of language models over extended texts.
LooGLE (Li et al. 2023a benchmarks LLMs' long-context reasoning using post-2022 documents, em- phasizing reasoning over memorization, though future training may impact zero-shot evaluation.
LongEval (Li* et al. 2023 is benchmark for assessing LLMs' long-context capabilities, focusing tasks requiring reasoning over extended text inputs.
StrcamingEval (Xiao et al. 2024c) is a benchmark designed to assess the ability of instruction-tuned LLMs to perform question answering in streaming contexts, where input sequences grow continuously.
7.2 Multi-modal Benchmark
Multi-modal datasets combine various data lypes, such as lext, images, audio, and video, to capture the complexity of the real world. Detailed discussions are provided for each benchmark, outlined in Table I4
LLaVA-Bench (Liu et al.
2023b is structured around image-ground ground--ruuttth truth textual description-questior: answer triplets, segmented across COCO and In-Tho-Wii datasets.
2023 serves as a bilingual multi-modal benchmark, facilitating comparative analysis VLM performance across English and Chinese linguistic contexts.
MMBench (uan Liu et al.
2024 evaluates the multi-modal long-context capabilities of LLMs, including both diagnostic and realistic evaluation sets. It emphasizes long-context and multi-image tasks.
MileBench (Song et al.
2024a) is a comprehensive benchmark for evaluating multi-modal LLMs' video comprehension, with longer videos, diverse genres, and varied assessment tasks.
MLVU Zhou et al.
40

---

Published in Transactions on Machine Learning Research (05/2025)
Table 14: Multi-modal Benchmark Tasks, Specifically, for task abbreviation, Conv: conversation task;
Desc: description task; Reas: reasoning task; Pere: perception task; Pred: prediction task; SUMM: summary task.
Tasks
Benchmark
Language
Conv
Desc
Reas
Pere
Pred Count Retrieval Order
SUMM
LL-
EN
[Liu al.
2023b
MMBench
EN / /ZH
Yuan Liu et al.
2023)
MileBench
V
EN
Song et al.
2024)
MLVU
EN
Zhou et al.
2024a
Long Video
Bench
EN
Wu et al.
2024b
Video- M
ME
EN
Fu al.
2024a
NExT QA
EN
Xiao et al.
2021
MV Bench
EN
Li et al.
2023b
MSVD
QA
EN
(uu et al.
2017
MSRV T
-QA
EN
Xu et al.
2017]
Bonn (Wu el al. 2024b) offers a framework aimed at assessing the capacity of large multi- modal models to comprehend lengthy videos with subtitles, extending up to an hour.
2024a evaluates large multi-modal models' video analysis using 900 videos across diverse domains, ranging from 11 seconds to 1 hour for broad scenario coverage.
Video-MME (Fu al.
NExT-QA (iiao et al.
2021 boasts dataset with 5,440 vidcos and approximately 52K manually annolated queslion-answer pairs, sorted into causal, temporal, and descriptive categories.
MVBench [Li et al.
2023b) comprises 200 multiple-choice question-answer (QA) pairs for cach of the
20 temporal understanding lasks, amassing a total of 4,000 QA pairs.
2017) is a collection of 1,970 video clips with descriptive captions, initially for MSVD-QA Xu et al.
video captioning.
MSRVTT-QA (Xu al. 2017 comprises 10,000 video clips with 20 human-transcribed sentences each, focusing on connecting video content with language descriptions.
7.3 Evaluation Metric
In evaluating LLMs, multiple metrics are essential to comprehensively assess performance across various tasks and application scenarios. Below is a list commonly used evaluation metrics with brief descriptions.
Exact Match (EM) (Rajpurkar et al. 2016) a strict metric assessing model accuracy by requiring predictions to exactly match the ground truth.
Partial Match (PM) metric evaluates model output similarity by allowing partial credit for overlaps with the reference, unlike strict metrics like Exact Match (EM).
Accuracy is metric used evaluate the overall performance of model by measuring the proportion of correctly predicted instances (both iositive and negative) out of the total instances.
41

---

Published in Transactions on Machine Learning Research (05/2025)
Recall measures model's ability to retrieve all relevant instances, calculated the ratio of correctly retrieved items to total relevant items.
Precision is a metric used evaluate the accuracy of model by measuring the proportion of correctly prodicted positive instances out of all predicted positive instances.
F1 combines Precision and Recall using their harmonic mean, offering a balanced evaluation by considering both false positives and negatives.
BLEU (Papineni et al. 2002) evaluates machine translation by comparing n-gram overlap with references, penalizing short outputs for fluency.
SacreBLEU (Post 2018 standardizes BLEU by fixing preprocessing steps for consistent machine trans-
lation ivaluations.
Rouge Lin 2004) and its variants measure the performance of models by calculating the overlap between the model output and the reference answer with unigram (Rouge-1) bigram(Rougee) (Rouge-2), LCS(Rouge-L).
METEOR (Denkowski & Lavie 2011 (Metric for Evaluation of Translation with Explicit ORdering) is text evaluation metric designed to assess the quality of machine translation.
BERT (Zhang et al. 2020) metric, often referred as BERTScoree, a text evaluation metric that uses contextual embeddings from the BERT model to compare similarity between generated and reference texts.
Edit Similarity measures text sequence similarity by the minimum edits needed to transform one sequence into another, derived from edit distance concepts.
Pass@k (Chen et al. 2021 evaluates the performance of a model by measuring the percentage of cases in which at least one of the top k generated outputs contains a correct solution.
Exponential Similarity is metric that measures the similarity between two items by exponentially weighting their differences, giving more importance to smaller discrepancies.
Concordance Index a metric used to evaluate the predictive accuracy of models, particularly in survival analysis ranking tasks.
Mean Reciprocal Rank (MRR) evaluates ranked results in information retrieval by averaging the reciprocal rank of the first relevant item across queries.
Relative Score evaluates multi-modal models in LLaVA-Bench by comparing outputs to a reference model like GPT-44 calculating the percentage ratio based on helpfulness, relevance, accuracy, and detail.
M-Avg Siultiple-Choice Average) is the mean accuracy across all multiple-choice tasks in the MLVU benchmark, based on the proportion of correct answers.
G-Avg (Generation Average) is the mean score of generation tasks in the MLVU benchmark, evaluated on dimensions like accuracy and relevance using GPT-4, with scores from to 5.
WUPS (K al. 2012) measures semantic similarity between words based on their laxonomy positions, using the least common ancestor.
8 Conclusion and Future Work
Advancements LLMs have driven significant progress in varions fields, but their high computational and memory demands during inference pose challenges, especially for long-context and real-time applications. KV cache management offers an effective solution by optimizing memory, reducing redundant computation, and improving performance. This survey reviews KV cache management strategics across token-level, model-level, and system-level optimizations. Token-level optimizations focus fine-grained control of KV cache through selection, budget allocation, merging, quantization, and low-rank dccomposition, enabling efficient. resource allocation without. altering model architectures. Model-level optimizations leverage architectural innovations, such as attention grouping and connfransformer designs, to enhance the efficiency of KV reuse. System-level optimizations further complement these efforts by employing advanced memory management, scheduling techniques, and hardware-aware designs to optimize resource utilization across diverse computing environments.
42

---

Published in Transactions on Machine Learning Research (05/2025)
Future directions for KV cache management research include several key areas. First, cross-category in- tegration of token-level, model-level, and system-level optimizations presents significant opportunities but remains underexplored due to the complexity of evaluating combinatorial configurations. Second, real-world case studics are essential to understand how KV cache techniques perform in production environments, shedding light on practical trade-ofls, domain-specifie priorities, and implementation challenges. Third, domain-specific optimizations, such as retaining critical tokens healtheare or applying structure-awaro strategies in legal and scientific applications, can improve efficiency by tailoring techniques to unique re- quirements. Fourth, privacy and security considerations, including privacy-aware eviction algorithms and secure isolation mechanisms, are critical for protecting sensitive data in multi-tenant environments. Finally, addressing implementation complexity, scalability, and integration with existing frameworks will be vital to ensure the widespread adoption and practical use of KV cache strategies across diverse applications.
Acknowledegments
We would like to thank the reviewers and editors of TMLR for their constructive comments. Prof. Lei Chen's work is partially supported by National Key Research and Development Program China Grant No. 2023YFF0725100, National Science Foundation of China (NSFC) under Grant No. U22B2060, Guangdong- Hong Kong Technology Innovation Joint Funding Scheme Project No. 2024A0505040012, the Hong Kong RGC GRF Project 16213620, RIF Project R6020-19, AOE Project AoE/E-603/18, Theme-based project TRS T41-603/20R, CRF Project C2004-21G, Guangdong Province Science and Technology Plan Project 2023A0505030011, Guangzhou municipality big data intelligence key lab, 2023A03J001 Hong Kong ITC ITF grants MHXX078/2 and PRPP/004//2XX Zhujiang scholar program 2021JC02X170 Microsoft Research -eebank joint research lab and 2023 HKUST Shenzhen-Hong Asia Collaborative Research Grant, Green Sustainability Special Fund, from Shui On Xintiandi and the Kong Collaborative Innovation Institute
InnoSpace GBA. Prof. Qing Li is supported by the Hong Kong Research Grants Council under General Research Fund (project no. 15200023) and Research Impact Fund (project no. R1015-23). Dr. Haoyang Li is supported by research funds PO052504 and P0053707.
References
Muhammad Adnan, Akhil Arunkumar, Gaurav Jain, Prashant J. Nair, Πyaa Soloveychik, and Purushotham Kamath. Keyformer: KV cache reduction through key tokens selection for efficient generative inference. In Proceedings of the Seventh Annual Conference on Machine Learning and Systems, MLSys 2024, Santa Claro, CA, USA, May 13-16, 2024. misys.org, 2024.
Abien Frod Agarap. Deep learning using rectified linear units. arXiv preprint arXiv:1803. 083775 2018
Saurabh Agarwal, Bilge Acun, Basil Hosmer, Mostafa. Elhoushi, Yejin Lee, Shivaram Venkataraman, Dimitris Papalliopoulos, and Carole-Jean Wu. CHAI: Clustered Head Attention for Efficient LLM Inference, April 2024. URL
httpr//abs/2403.3 abs//243300808,
Joshua Ainslie, James Tee-Thorp, Michiel de Jong, Yury Zemlyanskiy, Federico Lebrón, and Sumit Sanghai. GQA: Training Generalized Multi- Query Transformer Models from Multi-Head Checkpoints, December 2023. URL httt:://arxiv.oo
2abs/2305... 13245
Alon Albalak, Yanai Elazar, Sang Michael Xic, Shayne Longpre, Nathan Lambert, Xinyi Wang, Niklas Muennighoff, Bairu Hou, Liangming Pan, Haewon Jeong, et al. A survey on data selection for language models. arXiv preprint 2rXiv:2402.16897 2024.
Chenxin An, Shansan Gong, Ming Zhong, Xingjian Zhao, Mukai Li, Jun Zhang, Lingpeng Kong, and Xipeng Qiu. L-Eval: Instituting Standardized Evaluation for Long Context Language Models, 2023. URL https://arxiv... httpe////axivoor/abb////// www//bs//////eee/eeoo.ooeee 223// 11088
Anonymous. CAKE: Cascading and adaptive KV cache eviction with layer preferences. Submitted The Thirteenth International Conference on Learning Representations, 2024a. URL https://openreviie
het/Eforumm?.d-EEggEMAADaa under review.
43

---

Published in Transactions on Machine Learning Research (05/2025)
Anonymous. DynamicKV: Task-aware adaptive KV cache compression for long context LLMs. In Sub- milted to The Thirleenth International Conference on Learning Representations, 2024b. URL https: under review.
//openreview.net wwtpeevvieeee//ffrunuue mett/forummiddrutiffUUZZEE
Anonymous. Identify critical KV cache in LLM inference from an output perturbation perspective. Tn Submitted to The Thirteenth International Conference on Learning Representations, 2024c. URL https: under review.
//openreview
Salch Ashkboos, Amirkeivan Mohtashami, Maximilian L Croci, Bo Li, Pashmina Cameron, Martin Jaggi, Dan Alistarh, Torsten Hoefler, and James Hensman. Quarot: Outlier-free 4-bit inference in rotated llms. arXiv preprint arXix:2404. 00456,, 2024.
Ben Athiwaratkun, Sujan Kumar Gonugondla, Sanjay Krishna Gouda, Haifeng Qian, Hantian Ding, Qing Sun, Jun Wang, Jiacheng Guo, Liangfu Chen, Parminder Bhatia, Ramesh Nallapati, Sudipta Sengupta, and Bing Xiang. Bifurcated attention: Accelerating massively parallel decoding with shared prefixes Ilms, 2024. URL httpss//arxiv.. httpaaawwwiiirra//a/aaaaaaa orggabs/2403 08845
Yushi Bai, Xin Lv, Jiajie Zhang, Hongchang Lyu, Jiankai Tang, Zhidian Huang, Zhengxiao Du, Xiao Liu, Ao- han Zeng, Lei Hou, Yuxiao Dong, Jie Tang, and Juanzi Li. LongBench: A Bilingual, Multitask Benchmark for Long Context. Understanding, 2023. URL atttps://arxisn wtwp://..wr//aa////////// 14508
William Berrios, Gautam Mittal, Tristan Thrush, Douwe Kicla, and Amanprect Singh. Towards language models that can see: Computer vision through the LENS of natural language. CoRR, abs/2306. 02////6306.11100 2023. doi: 10.48550/ ARTp:///XXX...331
URL https:///oi.orgg
233500/arXiv 2306. 16410
Yelysei Bondarenko, Markus Nagel, and Tijmen Blankevoort. Understanding and overcoming the challenges of efficient transformer quantization. arXiv preprint arXiv:2109. 12948, 2021.
Yelysei Bondarenko, Markus Nagel, and Tijmen Blankevoort. Quantizable transformers: Removing outliers by helping attention heads do nothing. Advances in Neural Information Processing Systems, 36:75067- 75096, 2023.
William Brandon, Mayank Mishra, Aniruddha Nrusimha, Rameswar Panda, and Jonathan Ragan Kelly. Reducing Transformer Key-Value Cache Size with Cross- Layer Attention, May 2024. URL http://arxiv. org/ ht///4bb/40055 12981
Tom Brown, Benjamin Mann, Nick Ryder, Melanic Subbiah, Jared D Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, et al. Language models are few-shot learners. 2020.
Advances in neural information processing systems,
Sébastien Bubeck, Varun Chandrasekaran, Ronen Eldan, Johannes Gehrke, Eric Horvitz, Ece Kamar, Peter Lee, Yin Tat Lee, Yuanzhi Li, Scott Lundberg, et al. Sparks of artificial general intelligence: Farly experiments with gpt-4, arXiv preprint Xrx::23333 2023.
Zefan Cai, Yichi Zhang, Bofei Gao, Yuliang Liu, Tianyu Lin, Keming Lu, Wayne Xiong, Yue Dong, Baobao Chang, Junjie Hu, and Wen Xiao. Pyramidkv: Dynamic KV cache compression based on pyramidal information funneling. CoRR, abs/2406. bbs/2206.02068 2024.
Chi-Chih Chang, Wei-Cheng Lin, Chien-Ya Yu Lin, Chong- Yan Chen, Yu-Fang Hu, Pei-Shuo Wang, Ning-Cli Chi Huang, Luis Ceze, Mohamed S Abdelfattah, and Kai-Chiang Wu. Palu: Compressing kv-cache with low-rank projection. arXiv preprint arXiv:2407.21118, 2024.
Guoxuan Chen, Han Shi, Jiawei Li, Yihang Gao, Xiaozhe Ren, Yimeng Chen, Xin Jiang, Zhenguo Li, Weiyang Liu, and Chao Huang. Seplim: Accelerate large language models by compressing one segment into one scparator. arXio preprint or 0rxi::2412 12.14422, 2024a.
Long Chen, Oleg Sinavski, Jan Hünermann, Alice Karnsund, Andrew James Willmott, Danny Birch, Daniel Maund, and Jamie Shotton. Driving with 1mss: Fusing object-level vector modality for explainable au- tonomous driving. In 2024 IEEE International Conference on Robotics and Automation (ICRA), pp. 14093 -4100. IEFF, 2024b.
44

---

Published in Transactions on Machine Learning Research (05/2025)
Mark Chen et al. Evaluating large language models trained on code, 2021. URL https::/arxiv..t abs// 2107.03374
Mengzhao Chen, Yi Liu, Jiahao Wang, Yi Bin, Wenqi Shao, and Ping Luo. Prefixquant: Static quantization beats dynamic through prefixed outliers llms. arXiv preprint arXiv:24 10. 05265, 2024c.
Yilong Chen, Guoxia Wang, Junyuan Shang, Shiyao Cui, Zhenyu Zhang, Tingwen Liu, Shuohuan Wang, Yu Sun, Dianhai Yu, and Hua Wu. NACL: A general and effective KV cache eviction framework for Ilms at inference time. CoRR, Abs/2448.03667 2024d.
Yilong Chen, Linhao Zhang, Junyuan Shang, Zhenyu Zhang, Tingwen Liu, Shuohuan Wang, and Yu Sun. DHA: Learning Decoupled-Head Attention from Transformer Checkpoints via Adaptive Heads Fusion, June 2024c. URL httt:://arxii/ www..oggggbba.bo abbs/2466.66 065667
Yuang Chen, Cheng Zhang, Xitong Gao, Robert D. Mullins, George Anthony Constantinides, and Yiren Zhao. Optimised Grouped-Query Attention Mechanism for Transformers. In Workshop on Efficient Sys- tems for Foundation Models ICML2024. July 2024f. URL https://openreviee.nce
13MMghY6Kh
Zhuoming Chen, Ranajoy Sadhukhan, Zihao Ye, Yang Zhou, Jianyu Zhang, Niklas Nolte, Yuandong Tian, Matthijs Douze, Léon Bottou, Zhihao Jia, and Beidi Chen. Magicpig: LSH sampling for efficient LLM generation. CoRR, abs/24100 abs/2410.161199 2024g.
Jian Cheng, Jiaxiang Wu, Cong Leng, Yuhang Wang, and Qinghao Hu. Quantized cnn: unified approach accelerate and compress convolutional networks. IEEE tronsactions on neurol networks and learning systems, 29(10):4730-4743 2017.
Sai Sena Chinnakonduru and Astarag Mohapatra. Weighted Grouped Query Attention in Transformers, July 2024. URL http://arxiv, atgg/bas/20..10eeee
22/. 10855
Krzysztof Marcin Choromanski, Valerii Likhosherstov, David Dohan, Xingyou Song, Andreea Gane, Tamas Sarlos, Peter Hawkins, Jared Quiney Davis, Afroz Mohiuddin, Lukasz Kaiser, et al. Rethinking attention with performers. In International Conference on Learning Representations, 2020.
Can Cui, Yunshong Ma, Xu Cao, Wenqian Yc, Yang Zhou, Kaizhao Liang, Jintai Chen, Juanwu Lu, Zichong Yang, Kuei-Da Liao, Tianren Gao, Erlong Li, Kun Tang, Zhipeng Cao, Tong Zhou, Ao Liu, Xinrui Yan, Shuqi Mei, Jianguo Cao, Ziran Wang, and Chao Zheng. survey on multimodal large language models for autonomous driving. In IEEE/CVF Winter Conference on Applications of Computer Vision Workshops, WACVW 2024- Workshops, Waikoloa, HI, USA, January 1-6, 2024, pp. 958 979. IEEE, 2024. doi: 111109//ACWW6000 200.0000.02400066666
URI. htttps://dii. org/ 101 1ww1111999/WWWGGS 2024. .01106
Damai Dai, Chengqi Deng, Chenggang Zhao, R. X. Xu, Huazuo Gao, Deli Chen, Jiashi Li, Wangding Zeng, Xingkai Yu, Y. Wu, Zhenda Xie, Y. K. Li, Panpan Huang, Fuli Luo, Chong Ruan, Zhifang Sui, and Wenfeng Liang. Deepsockmoc: Towards ultimate expert specialization in mixture-of-experts language models. In Lun-Wei Ku, Andre Martins, and Vivek Srikumar (eds.), Proceedings of the 62nd Annual Meeting of the Association for Computational Línguistics (Volume 1: Long Papers), ACL 2024, Bangkok, Thailand, August 11-16, 2024, pp. 1280-1297. Association for Computational Linguisties, 2024. doi: 10.18653/V1/20244 /1//202AAALLLLLN....C. URL https://doi org/10. 186...1116///2/22 acl 1o1-long.70
Tri Dao. FlashAutention-2: Faster attention with better parallelism and work partitioning. In International Conference on Learning Representations (ICLR), 2024.
Tri Dao, Daniel Y. Fu, Stefano Ermon, Atri Rudra, and Christopher Ré. Fltentnt:ntion: Fast and memory- efficient exact attention with 10-awareness. In Advances in Neural Information Processing Systems (NeurIPS), 2022.
DeepSeck-A AI et al. DeepSeek- V2: A Strong, Economical, and Efficient Mixturo-of Expperss Language Model, June 2024. URL
45

---

Published in Transactions on Machine Learning Research (05/2025)
Michael Denkowski and Alon Lavie. Meteor 1.3: Automatic metric for reliable optimization and evaluation of machine translation systems. In Chris Callison-Burch, Philipp Koehn, Christof Monz, and Omar F. Zaidan (eds.), Proceedings of the Sizth Workshop on Statistical Machine Translation, pp. 85 91, Edinburgh, Scot- land, July 2011. Association for Computational Linguistics. URL http:://acccattoooooooooo//////
Tim Dettmers, Mike Lewis, Younes Belkada, and Luke Zettlemoyer. Gpt3. int8 (): 8-bit matrix multiplication for transformers at scale. Advances in Neural Information Processing Systems, 35:30318-3006 2022.
Tim Dettmers, Artidoro Pagnoni, Ari Holtzman, and Luke Zettlemoyer. Qlora: Efficient finctuning of quantized 1mss. Advances in Neural Information Processing Systems, 36, 2024.
Alessio Devolo, Yu Zhao, Simone Scardapane, and Pasquale Minervini. A simple and effective la norm-based strategy for kv cache compression, 2024. URL http:://aarri...orrr/b///4666 11430
Tianyu Ding, Tianyi Chen, Haidong Zhu, Jiachen Jiang, Yiqi Zhong, Jinxin Zhou, Guangzhi Wang, Zhihui Zhu, llya Zharkov, and Luming Liang. The efficiency spectrum of large language models: An algorithmic survey. Xrv preprint orXiv:2312.0000 2023.
Harry Dong, Xinyu Yang, Zhenyu Zhang, Zhangyang Wang, Yuejic Chi, and Beidi Chen. Get more with less: Synthesizing recurrence with kv cache compression for efficient 11m inference. arXio preprint arXiv:2402.0980 09398, 2024a.
Shichen Dong, Wen Cheng, Jiayu Qin, and Wei Wang. Qaq: Quality adaptive quantization for 1mm kv cache. arXiv preprint ar orxo:2443304000 2024b.
Wei Dong and Ke Yi. Residual sensitivity for differentially private multi-way joins. In Proceedings of the 2021 International Conference on Management of Data, pp. 432-444, 2021.
Wei Dong, Qiyao Luo, and Ke Yi. Continual observation under user-level differential privacy. In 2023 IEEE Symposium on Security and Privacy (SP), pp. 2190 2007. IEEE, 2023a.
Zican Dong, Tianyi Tang, Junyi Li, Wayne Xin Zhao, and Ji-Rong Wen. Bamboo: A comprehen- sive benchmark for evaluating long text modeling capacities of large language models. orxiv preprint arXiv:2309. 13345 13345, 2023b.
Zhengxiao Du, Yujie Qian, Xiao Liu, Ming Ding, Jiezhong Qiu, Zhilin Yang, and Jie Tang. GLM: general language model pretraining with autoregressive blank infilling. In Smaranda Muresan, Preslav Nakov, and Aline Villavicencio (eds.), Proceedings of the 60th Annual Mesting of the Association for Compu- tational Linguistics (Volume 1: Long Papers), ACL 2022, Dublin, Ireland, May 22-27, 2022, pp. 320- 335. Association for Computational Linguisties, 2022. doi: 10.18653/ (0.18633/V1/20222 /1220/22222.22-..C.CC 26
URL
attps://dois org/1o. 10. 1801336/1/2222..c1111
Haojie Duanmu, Zhihang Yuan, Xiuhong Li, Jiangfei Duan, Xingcheng Zhang, and Dahua Lin. Skyq: Sliding- window key and value cache quantization for large language models. arXiv preprint arXiv:2405.06211 2024.
Abhimanyu Dubey, Abhinav Jauhri, Abhinav Pandoy, Abhishck Kadian, Ahmad Al-Dahle, Aiesha Letman, Akhil Mathur, Alan Schelten, Amy Yang, Angela Fan, et al. The llama 3 herd of models. arXiv preprinl arXiv:247.221788 2024.
Yuan Feng, Junlin Lv, Yukun Cao, Xike Xie, and S. Kevin Zhou. Ada-kv: Optimizing KV cache eviction by adaptive budget allocation for efficient LLM inference. CoRR, abs/2407. 2024.
Zafeirios Fountas, Martin Benfeghoul, Adnan Oomerjee, Fenia Christopoulou, Gerasimos Lampouras, Haitham Bou-Ammar, and Jun Wang. Human-like episodic memory for infinite context 1lms. CoRR, abs .40700000 2024.
Elias Frantar, Salch Ashkboos, Torsten Hoefter, and Dan Alistarh. Gptq: Accurate post-training quantization for generative pre-trained transformers. arXiv preprint arXiv:2210. 17323, 2022.
46

---

Published in Transactions on Machine Learning Research (05/2025)
Chaoyou Fu, Yuhan Dai, Yondong Luo, Lei Li, Shuhuai Ren, Renrui Zhang, Zihan Wang, Chenyu Zhou, Yunhang Shen, Mengdan Zhang, et al. Video-mme: The first-ever comprehensive evaluation benchmark of multi-modal Iims in video analysis. arXiv preprint orXie:2405.21075 2024a.
Daocheng Fu, Xin Li, Licheng Wen, Min Dou, Pinlong Cai, Botian Shi, and Yu Qiao. Drive like human: Rethinking autonomous driving with large language models. In Proceedings of the IEEE/CVF Winter Conference on Applications of Computer Vision, pp. 910 919, 2024b.
Yu Fu, Zefan Cai, Abedelkadir Asi, Wayne Xiong, Yue Dong, and Wen Xiao. Not all heads matter: A head-level KV cache compression method with integrated retrieval and reasoning. CoRR, abs/2410019920 192558, 2024c.
Bin Gao, Zhuomin He, Puru Sharma, Qingxuan Kang, Djordje Jevdjic, Junbo Deng, Xingkun Yang, Zhou Yu, and Pengfei Zuo. Cost-efficient large language model serving for multi-turn conversations with cache- dattention, 2024a. URL https://arxiv. org/ httgg/bb/2400..199 19708
Shiwei Gao, Youmin Chen, and Jiwu Shu. Fast state restoration in LLM serving with hcache. CoRR, abs/24100 abs/2410.05004,, 2024b. doi: 10.48550/ARXI1 2411000.110.00001.. URL htttps://dii..o .tt.........00//......... 48550/arXiv. 2410.05004
Suyu Ge, Yunan Zhang, Liyuan Liu, Minjia Zhang, Jiawci Han, and Jianfeng Gao. Model tells you what discard: Adaptive KV cache compression for llss. In The Twelfth International Conference on Learning Representations, ICLR 2024, Vienna, Austria, May 7-11, 2024. OpenReview net, 2024.
Nathan Godey, Alessio Devoto, Yu Zhao, Simone Scardapane, Pasquale Minervini, Éric Villemonte de la Clergerie, and Benoît Sagot. Q-filters: Leveraging query-key geometry for efficient key-value cache com- pression. In Sparsity in LLMs (SLLM): Deep Dive into Mixture of Experts, Quantization, Hardware, and Inference.
Daniel Goldstein, Fares Obeid, Eric Alcaide, Guangyu Song, and Eugene Cheah. GoldFinch: High Perfor- mance /Transorreee Hybrid with Linear Pre-Fill and Extreme KV-Cache Compression, July 2024.
URL http://arxiv wtwp:///g/aa......///// 12077
Giovani Gracioli, Ahmed Alhammad, Renato Mancuso, Antônio Augusto Fröhlich, and Rodolfo Pellizzoni. A survey cache management mechanisms for real-time embedded systems. ACM Computing Surveys (CSUR), 48(2):1-33 36, 2015.
Albert Gu and Tri Dao. Mamba: Linear-Time Sequence Modeling with Selective State Spaces, May 2024.
URL http://arxiv. org/abs/// httgg/bb//311... 2tt///312..0000 00752
Albert Gu, Isys Johnson, Karan Goel, Khaled Kamal Saab, Tri Dao, Atri Rudra, and Christopher Re. Combining Recurrent, Convolutional, and Continnous-time Models with Linear State Space Layers. net/
Advances in Neural Information Processing Systems, November 2021. URL
Albert Gu, Karan Goel, Ankit Gupta, and Christopher Ré. On the Parameterization and Initialization of Diagonal State Space Models. In Sanmi Koyejo, S. Mohamed, A. Agarwal, Danielle Belgrave, K. Cho, and A. Oh (eds.), Aduances in Neural Information Processing Systems 35: Annual Conference on Neural Information Processing Systems 2022, NewrIPS 2022, New Orleans, LA, USA, November 28 December 9, 2022. arXiv, 2022.
Ozgur Guldogan, Jackson Kunde, Kangwook Lee, and Ramtin Pedarsani. Multi-bin batching for increasing LLM inference throughput, 2024. URL
Muhammad Usman Hadi, Rizwan Qureshi, Abbas Shah, Muhammad Irfan, Anas Zafar, Muhammad Bilal Shaikh, Naveed Akhtar, Jia Wu, Seyedali Mirjalili, al. survey on large language models: Applications, challenges, limitations, and practical usage. Authorea Preprints, 2023.
47

---

Published in Transactions on Machine Learning Research (05/2025)
Benjamin D Haeffele and René Vidal. Global optimality in tensor factorization, deep learning, and beyond. arXiv preprint Xiv: 07060, 2015.
Chi Han, Qifan Wang, Hao Peng, Wenhan Xiong, Yu Chen, Heng Ji, and Sinong Wang. Lm-infinite: Zero- shot extreme length generalization for large language models. In Proceedings of the 2024 Conference of the North American Chapter of the Association for Computational Linguistics: Human Longuage Tech- nologies (Volume 1: Long Papers), NAACL 2024, Merico Cily, Merico, June 16-21, 2024, pp. 1991-4008. Association for Computational Linguistics, 2024.
Mohsen Hariri, Lam Nguyen, Sixu Chen, Shaochen Zhong, Qifan Wang, Xia Hu, Xiaotian Han, and Vipin Chaudhary. More for keys, less for values: Adaptive kv cache quantization. or preprint arXiv:2502. 15075, 2025.
Ramin Hasani, Mathias Lechner, Tsun-Hsuan Wang, Makram Chahine, Alexander Amini, and Danicla Rus. Liquid structural state-space models. arXiv preprint orXiv:2209. 2022.
Jiaao He and Jidong Zhai. Fastdecode: High-throughput gpu-efficient Ilm serving using heterogeneous pipelines, 2024. URL
11421
Yefei He, Feng Chen, Jing Liu, Wenqi Shao, Hong Zhou, Kaipeng Zhang, and Bohan Zhuang. Zipvl: Efficient large vision-language models with dynamic token sparsification and kv cache compression. arXic preprint arXiv:24 10.0100..080 08584, 2024a.
Yefei He, Luoming Zhang, Weijia Wu, Jing Liu, Hong Zhou, and Bohan Zhuang. Zipcache: Accurate and efficient kv cache quantization with salient token identification. arXiv preprint rrXiv22405.12550 2024b.
Namgyu Ho, Sangmin Bae, Taehyeon Kim, Hyunjik Jo, Yireun Kim, Tal Schuster, Adam Fisch, James Thorne, and Se- Young Yun. Block Transformer: Global-to-Loct Language Modeling for Fast Inference, November 2024, URL http:///arat
org/ wwwg/abs/2406... 02657
Coleman Hooper, Sehoon Kim, Hiva Mohammadzadeh, Monishwaran Maheswaran, June Paik, Michael W. Mahoney, Kurt Keutzer, and Amir Gholami. Squeezed attention: Accelerating long context length Ilm inference, 2024a. URL
Coleman Hooper, Schoon Kim, Hiva Mohammadzadeh, Michael W Mahoney, Yakun Sophia Shao, Kurt Keutzer, and Amir Gholami. Kvquant: Towards 10 million context length Ilm inference with kv cache quantization. aix preprint orXi022401.18079, 2024b.
Cheng-Ping Hsieh, Simeng Sun, Samuel Kriman, Shantanu Acharya, Dima Rekesh, Fei Jia, Yang Zhang, and Boris Ginsburg. Ruler: What's the real context size of your long-context language models? arXiv preprint arXiv:2404.0655, 2024.
Cunchen Hu, Heyang Huang, Junhao Hu, Jiang Xu, Xusheng Chen, Tao Xie, Chenxi Wang, Sa Wang, Yungang Bao, Ninghui Sun, and Yizhou Shan. Memserve: Context caching for disaggregated I1m serving with elastic memory pool, 2024. URL htttp:://aaaat wrg/abs//4000 17565
Weizhe Hua, Zihang Dai, Hanxiao Liu, and Quoe V. Le. Transformer Quality in Linear Time. In Kamalika Chaudhuri, Stefanie Jegelka, Le Song, Csaba Szepesvári, Gang Niu, and Sivan Sabato (eds.), International Conference Machine Learning, ICML 2022, 17-23 July 2022, Baltimore, Maryland, USA, volume 162 of Proceedings of Machine Leurning Research, pp. 9099 9117. PMLR, 2022. URL mlr. www.pprsssveee htt//////a2aa...m//a//a html
Farnoosh Javadi, Walid Ahmed, Habib Hajimolahoseini, Foozhan Ataiefard, Mohammad Hassanpour, Saina Asani, Austin Wen, Omar Mohamed Awad, Kangling Liu, and Yang Liu. GQKVA: Effficient Pre-training of Transformers by Grouping Queries, Keys, and Values, December 2023. URL http:ht/tppp...rrrgpps//e www./iioorr//bba. abs/
2811.03422
Herve Jegou, Matthijs Douze, and Cordelia Schmid. Product quantization for nearest neighbor search. JEEE Iransactions on pottern, analysis and machine intelligence, 33(1):117-128, 2010.
48

---

Published in Transactions on Machine Learning Research (05/2025)
Albert Q Jiang et al. Mixtral of experts. arXiv preprint arXiv:2401.04088, 2024a.
Chaoyi Jiang, Lei Gao, Hossein Entezari Zarch, and Murali Annavaram. Efficient Ilm inference with i/o-aware partial kv cache recomputation, 2024b. URL http:///aaxxxvo.raaa//////ss 17089
Peng Jiang and Gagan Agrawal. A linear speedup analysis of distributed deep learning with sparse and quantized communication. Advances in Neural Information Processing Systems, 31, 2018.
Xuanlin Jiang, Yang Zhou, Shiyi Cao, Ion Stoica, and Minlan Yu. Noo: Saving gpu memory crisis with epu offloading for online IIm inference, 2024c. URL http:: btops://arxiv. wttpa//wwww...r////......
Ming Jin, Qingsong Wen, Yuxuan Liang, Chaoli Zhang, Siqiao Xue, Xue Wang, James Zhang, Yi Wang, Haifeng Chen, Xiaoli Li, et al. Large models for time series and spatio-temporal data: A survey and outlook. Xiri preprint 2r 10196, 2023.
Shuowei Jin, Xueshen Liu, Qingzhao Zhang, and Z. Morley Mao. Compute or load kv cache? why not both?, 2024. URL
03065
Vinay Joshi, Prashant Laddha, Shambhavi Sinha, Om Ji Omer, and Sreenivas Subramoney. QCQA: Quality and Capacity-aware grouped Query Attention, June 2024. URL http://aarav..r///aa////000 10247
Jordan Juravsky, Bradley Brown, Ryan Ehrlich, Daniel Y. Fu, Christopher Ré, and Azalia Mirhoseini. Hydragen: High-throughput 11m inference with shared prefixes, 2024. URL. https://arxiv. wwtpziii.rrg/ss//
9402.05099
Manjula Shenoy. K, K. C. Shet, and U. Dinesh Acharya. A new similarity measure for taxonomy based on edge counting, 2012. URL
Christoforos Kachris. A survey on hardware accelerators for large language models. arXiv preprint arXiv:2401.09890, 2024.
Hao Kang, Qingru Zhang, Souvik Kundu, Geonhwa Jeong, Zaoxing Liu, Tushar Krishna, and Tuo Zhao, Gear: An efficient kv cache compression recipefor near-lossless generative inference of I1m. arXie preprint arXiv:2403.055: 05527, 2024.
Angelos Katharopoulos, Apoorv Vyas, Nikolaos Pappas, and François Flcuret. Transformers are rmns: Fast autoregressive transformers with linear attention. In International conference on machine learning, pp. 5156-5165. PMLR, 2020.
Zohaib Khan, Muhammad Khaquan, Omer Tafveez, Burhanuddin Samiwala, and Agha Ali Raza. Beyond Uniform Query Distribution: Key-Driven Grouped Query Attention, August 2024. URL http://arxiv. org/abs/2408.08455
Jang- Kim, Junyoung Yeom, Sangdoo Yun, and Hyun Oh Song. Compressed context memory for online language model interaction. In The Twelfth International Conference on Learning Representations, ICLR 2024, Vienna, Austrio, May 7-11, 2024. OpenReview. net, 2024a. URL http:s:://oparrrri..eeeeee/eeees 1d=64ksvC41Pg
Sehoon Kim, Karttikeya Mangalam, Suhong Moon, Jitendra Malik, Michael W Mahoney, Amir Gholami, and Kurt Keutzer. Speculative decoding with big little decoder. Advances in Neural Information Processing Systems, 36, 2024b.
Yekyung Kim, Jenna Russell, Marzena Karpinska, and Mohit lyyer. One ruler measure them all: Bench- marking multilingual long-context language models, 2025. URL htttps://axxiv.rrrt http:/abs/520001990
Volodymyr Kuleshov, Arun Chaganty, and Perey Liang. Tensor factorization via matrix factorization. In Artificial Intelligence and Slatistics, pp. 507 516. PMLR, 2015.
Kwan, Wai-Chung, Zeng, Xingshan, Wang, Yufci, Sun, Yusen, Liand Liangyou, Shang, Lifeng, Liu, Qun, Wong, and Kam-Fai. M4le: A multi-ability multi-range multi-task multi-domain long-context evaluation benchmark for large language models. Xr preprint Xiv:23300 19240, 2023.
49

---

Published in Transactions on Machine Learning Research (05/2025)
Alicja Kwasniewska, Maciej Szankin, Mateusz Ozga, Jason Wolfe, Arun Das, Adam Zajac, Jacek Ruminski, and Paul Rad. Deep learning optimization for edge devices: Analysis of training quantization parameters. In JECON 2019-45th Annual Conference of the IEEE Industrial Electronics Society, volume 1, pp. 96 101. IEEE, 2019.
Woosuk Kwon, Zhuohan Li, Siynan Zhuang, Ying Sheng, Lianmin Zheng, Cody Hao Yu, Joseph Gonzalez, Hao Zhang, and Ion Stoica. Efficient memory management for large language model serving with page- dattention. In Proceedings of the 29th Symposium on Operating Systems Principles, SOSP 2023, Koblenz, Germany, October 23-26, 2023, pp. 611 626. ACM, 2023.
Wonbeom Lee, Jungi Lee, Junghwan Seo, and Jaewoong Sim. Infinigen: Efficient generative inference of large language models with dynamic kv cache management, 2024. URL
ha//a/bb//222.1121 19707
Yaniv Leviathan, Matan Kalman, and Yossi Matias. Fast inference from transformers via speculative decod- ing. In International Conference on Machine Learning, pp. 19274 19286. PMLR, 2023.
Dacheng Li*, Rulin Shao*, Anze Xie, Ying Sheng, Lianmin Zheng, Joseph E. Gonzalez, lon Stoicaand Xuezhe Ma, and Hao Zhang. How long can open-source Ilms truly promise on context length?, June 2023. URL
attps://1msy: http:://mmmss..g/lloo////o/// 0/tp//lloo/22200--99---- 2aa-222--1onccaa
Haoyang Li and Lei Chen. Cache-based GNN system for dynamic graphs. In CIKM '21: The 30th ACM International Conference Information and Knowledge Management, Virtual Event, Queens- land, Australia, November 1 5, 2021, pp. 937-909 437-946. ACM, 2021. doi:
URL
https: http://do..orgg//// 0ttp//00.115/345565333 348227777777
Haoyang Li, Xuejia Chen, Zhanchao XU, Darian Li, Nicole Hu, Fei Teng, Yiming Li, Luyu Qiu, Chen Jason Zhang, Qing Li, and Lei Chen. Exposing numeracy gaps: A benchmark to evaluate fundamental numerical abilities in large language models, 2025. URL https:// org/ abs 20.2.b/25021110 11075
Jiaqi Li, Mengmeng Wang. Zilong Zheng, and Muhan Zhang. LooGLE: Can Long-Conters Context Language Models Understand Long Contexts?, 2023a. URL
wtwp:///..........eeeee. 04939
Kunchang Li, Yinan He, Yi Wang, Yizhuo Li, Wenhai Wang, Ping Luo, Yali Wang, Limin Wang, and Yu Qiao. Videochat: Chat-centric video understanding. arXie preprint or xrx::23355006555, 2023b.
Yiming Li, Yanyan Shen, Lei Chen, and Mingxuan Yuan. Orca: Scalable temporal graph neural network training with theoretical guarantees. Proc. ACM Manag. Data, ((1):52:1-52:21) 2023e. doi: 10.1145/ 3588737. URL httts:://oi.org:::
1145/3588737
Yucheng Li, Huiqiang Jiang, Qianhui Wu, Xufang Luo, Surin Ahn, Chengruidong Zhang, Amir H Abdi, Dongsheng Li, Jianfeng Gao, Yuqing Yang, et al. Scbench: A kv cache-centric analysis of long-context methods. arXiv preprint arXiv:2412. 10319, 2024a.
Yuhong Li, Yingbing Huang, Bowen Yang, Bharat Venkitesh, Acyr Locatelli, Hanchen Ye, Tianle Cai, Patrick Lewis, and Deming Chen. Snapkv: Lim knows what you are looking for before generation. arXie preprint arXiv:2404.144 14469, 2024b.
Zongqian Li, Yinhong Liu, Yixuan Su, and Nigel Collier. Prompt compression for large language models: A survey. arv preprint orXiv:2410.1238. 2024c.
Bingli Liao and Danilo Vasconcellos Vargas. Beyond KV Caching: Shared Attention for Efficient LLMs, July 2024. URL http:://axxii...rrr... htab//40..... 12866
Bokai Lin, Zihao Zeng, Zipeng Xiao, Siqi Kou, Tianqi Hou, Xiaofeng Gao, Hao Zhang, and Zhijie Deng. MatryoshkaKV; Adaptive KV Compression via Trainable Orthogonal Projection, October 2024a. URL
http://arxxv.org/abb///1111
Chin-Yew Lin. ROUGE: A package for automatic evaluation of summarics. In Text Summarization Branches Out, pp. 74-81, Barcelona, Spain, July 2004. Association for Computational Linguistics. URL https:
50

---

Published in Transactions on Machine Learning Research (05/2025)
Darryl Lin, Sachin Talathi, and Sreekanth Annapureddy. Fixed point quantization of deep convolutional networks. Tn International conference on machine learning, pp. 2849-2858. PMT.R, 2016.
Haokun Lin, Haobo Xu, Yichen Wu, Jingzhi Cui, Yingtao Zhang, Linzhan Mou, Linqi Song, Zhenan Sun, and Ying Wei. Duquant: Distributing outliers via dual transformation makes stronger quantized 1lms. Tn The Thihtt-eighth Annual Conference on Neural Information Processing Systems, 2024b.
Ji Lin, Jiaming Tang, Haotian Tang, Shang Yang, Wei-Ming Chen, Wei-Chen Wang, Guangxuan Xiao, Xingyu Dang, Chuang Gan, and Song Han. AWQ: activation- aware weight quantization for on-device LLM compression and acceleration. In Phillip B. Gibbons, Gennady Pekhimenko, and Christopher De Sa (ods.), Proceedings of the Seventh Annual Conference on Machine Learning and Systems, MLSSs 2024, Santa Clara, CA, USA, May 13-16, 2024. mlsys.org, 2024c. URL https::/ mlays. org/ paper
html
Yujun Lin, Haotian Tang. Shang Yang, Zhekai Zhang, Guangxuan Xiao, Chuang Gan, and Song Han. Qserve: W4A8KV4 quantization and system co-design for efficient LLM serving. CoRR, abs/ 2405504433, 2024d. doi: 10. 10.48550 1.48550/ARXIV ARX.V..400.0455.50 04502. URL https://doi. htt.//i...rr./.....///////////
18550/arXiv, .2005. 04532
Zhiqi Lin, Cheng Li, Youshan Miao, Yunxin Liu, and Yinlong Xu. Pagraph: Scaling gnn training on large graphs via computation-aware caching. In Proceedings of the 11th ACM Symposium on Cloud Computing. pp. 201-415, 2020.
Akide Liu, Jing Liu, Zizheng Pan, Yefei He, Gholamreza Haffari, and Bohan Zhuang. Minicache: KV cache compression in depth dimension for large language models. CoRR, abs/ abs52405.1 14366, 2024a. doi: 10.48550/ARXTV 2TT.V.2405..44355 URL https::///l.oorr//////
htt.://a..vv.......... 14366.
Di Liu, Meng Chen, Baotong Lu, Huiqiang Jiang, Zhenhua Han, Qianxi Zhang, Qi Chen, Chengruidong Zhang, Bailu Ding, Kai Zhang, Chen Chen, Fan Yang, Yuqing Yang, and Lili Qiu. Retricvalattention: Accelerating long-context LLM inference via vector retrieval. CoRR, abs/2409.1055 2024b.
Guangda Liu, Chengwei Li, Jieru Zhao, Chenqi Zhang, and Minyi Guo. Clusterkv: Manipulating I1m kv cache in semantic space for recallable compression. arXiv preprint arXiv:2412.08211 2024c.
Haotian Liu, Chunyuan Li, Qingyang Wu, and Yong Jae Lee. Visual instruction tuning. In Alice Oh, Tristan Naumann, Amir Globerson, Kate Saenko, Moritz Hardt, and Sergey Levine (eds.), Advances in Newral Information Processing Systems 36: Annual Conference
Newral Information Processing Systems 2023, NewrIPS 2023, New Orleans, LA, USA December 10 - 16, 2023, 2023a. URL h nips. cc/ hash//aaa/aaaaaaa paper files/ http:///ppppeeo.o/aaa//aa//////a /ttp:///..77722222222221144.....
Satrraa---tsttaaaa---Ceeeeeeeeee Abstract Conference html
Haotian Liu, Chunyuan Li, Qingyang Wu, and Yong Jae Lee. Visual instruction tuning. In NewrIPS, 2023b.
Peiyu Liu, Ze-Feng Gao, Wayne Xin Zhao, Yuan Xie, Zhong-Yi Yi Lu, and Ji-Rong Wen. Enabling lightweight fine-tuning for pre-trained language model compression based on matrix product operators. arXiv preprint Xrx::22106. 02205, 2021.
Peiyu Liu, Ze-Feng Gao, Wayne Xin Zhao, Yipeng Ma, Tao Wang, and Ji-Rong Wen. Unlocking frea-free low- bit quantization with matrix decomposition for kv cache compression. arXiv preprint arXiv:2405. 12591, 2024d.
Ruikang Liu, Haoli Bai, Haokun Lin, Yuening Li, Han Gao, Zhengahuo Xu, Lu Hou, Jun Yao, and Chun Yuan. Intactkv: Improving large language model quantization by keeping pivot tokens intact. arXiv preprint ar FTT::2403.0244.. 2024c.
Xin Liu, Pei Lin, and Guoming Tang. Zeromerge: Parameter-free kv cache compression for memory-cfficient long-context llms, 2025. URL
Yuhan Liu, Hanchen Li, Yihua Cheng, Siddhant Ray, Yuyang Huang, Qizhong Zhang, Kuntai Du, Jiayi Yao, Shan Lu, Ganesh Ananthanarayanan, et al. Cachegen: Kv cache compression and streaming for fast large language model serving. Tn Proceedings of the ACM SIGCOMM 2024 Conference, pp. 38 56, 2024f.
51

---

Published in Transactions on Machine Learning Research (05/2025)
Zechun Liu, Changsheng Zhao, Igor Fedorov, Bilge Soran, Dhruv Choudhary, Raghuraman Krishnamoor- thi, Vikas Chandra, Yuandong Tian, and Tijmen Blankevoort. Spinquant- -Im quantization with learned rotations. Xrxiv preprint arXiv:2405116400 2024g.
Zichang Liu, Aditya Desai, Fangshuo Liao, Weitao Wang, Victor Xie, Zhaozhuo Xu, Anastasios Kyrillidis, and Anshumali Shrivastava. Scissorhands: Exploiting the persistence of importance hypothesis for LLM KV cache compression at test time. In Advances in Newral Information Processing Systems 36: Annual Conference on Neural Information Processing Systems 2023, NeurIPS 2023, New Orleans, LA, USA, December 10 16, 2023, 2023c.
Zirui Liu, Jiayi Yuan, Hongye Jin, Shaochen Zhong, Zhaozhuo Xu, Vladimir Braverman, Beidi Chen, and Xia Hu. Kivi: A luning- free asymmeerrc 2bit quantization for kv cache. Xiv preprinl arXiv:2402.02755 022500 2024h.
Haoyu Lu, Wen Liu, Bo Zhang, Bingxuan Wang, Kai Dong, Bo Liu, Jingxiang Sun, Tongzheng Ren, Zhuoshu Li, Hao Yang, Yaofeng Sun, Chengqi Deng, Hanwei Xu, Zhenda Xie, and Chong Ruan. Deepseek-vi vl: Towards roal-world vasion-language understanding. CoRR, abs/2403. 0403.05525, 2024. doi: (0.48550/ARXIY 2403.05525. URL
www./////..../////..e///..///// arXiv
Cheng Luo, Zefan Cai, Hanshi Sun, Jinqi Xiao, Bo Yuan, Wen Xiao, Junjie Hu, Jiawei Zhao, Beidi Chen, and Anima Anandkumar. Headinfer: Memory-efficien Imm inference by head-wise offloading, 2025. URL
hattps://arxe httpas///arrii.orrrr/////2/// 12574
Qianli Ma, Zhen Liu, Zhenjing Zheng, Ziyang Huang, Siying Zhu, Zhongzhong Yu, and James T Kwok. A survey on time-series pre-trained models. IEEE Transactions on Knowledge and Data Engineering, 2024a.
Yuexiao Ma, Huixia Li, Xiawu Zheng, Feng Ling, Xuefeng Xiao, Rui Wang, Shilei Wen, Fei Chao, and Rongrong Ji. Affinequant: Affine transformation quantization for large language models, orXiv preprint arXiv:2403. 12544, 2024b.
Osman Asif Malik and Stephen Becker Low-rank tucker decomposition of large tensors using tensorsketch. Advances in neural information processing systems, 31, 2018.
Yusuke Matsui, Yusuke Uchida, Hervé Jégon, and Shin'ichi Satoh. A survey of product quantization. ITE Transactions on Media Technology and Applications, 6(1):2 10, 2018.
Xupeng Miao, Gabriele Oliaro, Zhihao Zhang, Xinhao Cheng, Hongyi Jin, Tianqi Chen, and Zhihao Jia. Towards efficient generative large language model serving: survey from algorithms to systems. arXio preprint XrXi::2312. 15234, 2023.
Bonan Min, Hayley Ross, Elior Sulem, Amir Pouran Ben Veyseh, Thien Huu Nguyen, Oscar Sainz, Eneko Agirre, Ilana Heintz, and Dan Roth. Recent advances in natural language processing via large pre-trained language models: survey, ACM Comput. Surv., 56(2):30:1-30::) 30:40000,, 2024. doi: 10.1145/360599 URL https://doi... org/ 1...1.....1111 1145/3605943
João Monteiro, Élienne Marcotte, Pierre-Andre Noël, Valentina Zantedeschi, David Vázquez, Nicolas Chapa- dos, Christopher Pal, and Perouz Taslakian. XC-Cache: Cross-Attendinn to Cached Context for Efficient LLM Inference. In Yaser Al-Onaizan, Mohit Bansal, and Yun-Nung Chen (eds.), Pindings of the Asso- ciotion for Computational Linguistics: EMNLP 2024, Miami, Florido, USA, November 12-16, 2024, pp. 15284-15302. Association for Computational Linguisties, 2024. URL htttp:://cclatttool/ findings-emnlp... 896
2024.
Jesse Mu, Xiang Li, and Noah D. Goodman. Learning to compress prompts with gist to- kens. In Alice Oh, Tristan Naumann, Amir Globerson, Kate Saenko, Moritz Hardt, and Sergey Levine (eds.), Advances in Neural Information Processing Systems 36: Annual Confer- ence on Neural Information Processing Systems 2023, NeurIPS 2023, New Orleans, LA, USA, December 10 16, 2023, 2023. URL http:://papers.. nips. Coneeracct-cooffeene.e.... html
GGTTceddccf1133a//5644444eeed222 wtw.://deo--aatatt--oCoaooo Conf
52

---

Published in Transactions on Machine Learning Research (05/2025)
Yongyu Mu, Yuzhang Wu, Yuchun Fan, Chenglong Wang, Hengyu Li, Qiaozhi He, Murun Yang. Tong
Xiao, and Jingbo Zhu. Cross-layer Attention Sharing for Large Language Models, 2024. URL https:
//arxiv. wwwp.b//g/.......///////// (00bb//900000800
Tsendsuren Munkhdalai, Manaal Faruqui, and Siddharth Gopal. Leave No Context Behind: Efficient Infinite
Context Transformers with infini-attentin August 2024. URL http::/aaxxv..rrg// wwt.:oo/g/aba/ea.ee. ht////bb///444 2202200049093
Humza Naveed, Asad Ullah Khan, Shi Qiu, Muhammad Saqib, Saced Anwar, Muhammad Usman, Nick Barnes, and Ajmal Mian. A comprehensive overview of large language models. CoRR, abs/ 2307063006 2023. doi: 10. 10.48550/ARXIY 2307.00435.. URL https://doi. oeaeioii.oorooooooooo org/ 100 005444555/aai/ arriv. 0330706443
Piotr Nawrot, Adrian Lancucki, Marcin Chochowski, David Tarjan, and Edoardo M. Ponti. Dynamic Memory Compression: Retrofitting LLMs for Accelerated Inference. In Forty-First International Conference on Muchine Learning, ICML, 2024, Vienna, Austria, July 21- 21-27, 2024. OpenReview.nel, 2024. URL https:
//openreview.o Ctgpaproiisv..eee/cooooo//dddtdd
Xiurui Pan, Endian Li, Qiao Li, Shengwen Liang, Yizhou Shan, Ke Zhou, Yingwei Luo, Xiaolin Wang, and Jie Zhang, Instinfer: In-storage attention offloading for cost-effective long-context Ilm inference, 2024.
URL petps://arxivy www.xiv.org/aaae /409 .49992
Kishore Papineni, Salim Roukos, Todd Ward, and Wei-Jing Zhu. Blew: a method for automatic evaluation of machine translation. In Pierre Isabelle, Eugene Charniak, and Dekang Lin (eds.), Proceedings of the 40th Annual Meeting of the Association for Computational Linguistics, pp. 311 318, Philadelphia, Pennsylvania, USA, July 2002. Association for Computational Linguistics. doi: 011155/177308..100 Tacclanthologg..orr//o-----
URL https:
Seungcheol Park, Jachyeon Choi, Sojin Lee, and U Kang. comprehensive survey of compression algorithms for language models. arXiv preprint arXiv:2401.15344 2024.
Badri Narayana Patro and Vijay Srinivas Agneeswaran. Mamba-360: Survey of state space models transformer alternative for long sequence modelling: Methods, applications, and challenges. arviv preprinl arXiv:2404.16112 2024.
Bo Peng et al. RWKV: Reinventing RNNs for the Transformer Era, December 2023. URL http://arxiv. org/ 2305. 13048
Stefan Podlipnig and Laszlo Böszörmenyi. A survey of web cache replacement strategies. ACM Computing Surveys (CSUR), 35(4):374 -9837399, 2003.
Matt Post. A call for clarity in reporting BLEU scores. In Ondrej Bojar, Rajen Chatterjee, Christian Federmann, Mark Fishel, Yvette Graham, Barry Haddow, Matthias Huck, Antonio Jimeno Yepes, Philipp Kochn, Christof Monz, Matteo Negri, Aurélic Név6ol, Mariana Neves, Matt Post, Lucia Specia, Marco Turchi, and Karin Verspoor (eds.), Proceedings of the Third Conference on Machine Translation: Research Papers, pp. 186- 191, Brussels, Belgium, October 2018. Association for Computational Linguisties. doi: 10.18653/vl 11//18-66319 URL htttps:/aaclaathhology) htt.://attoooloooooo//1//1---- org/N18-631
Jianing Qiu, Lin Li, Jiankai Sun, Jiachuan Peng, Peilun Shi, Ruiyang Zhang, Yinzhao Dong, Kyle Lam, Frank P-W Lo, Bo Xiao, et al. Large ai models in health informatics: Applications, challenges, and the future. IEFE Journal of Biomedical and Health Informaties, 2023.
Haohao Qu, Liangbo Ning, Rui An, Wenqi Fan, Tyler Derr, Hui Liu, Xin Xu, and Qing Li. A survey of mamba. are preprint arxiv:2408. 01 01129, 2024.
Alec Radford, Karthik Narasimhan, Tim Salimans, Ilya Sutskever, et al. Improving language understanding by generative pre-training. 2018.
Alec Radford, Jeffrey Wu, Rewon Child, David Luan, Dario Amodei, Iyy Sutskever, et al. Language models are unsupervised multitask learners. OpenAl I blog, 188)=9, 2019.
53

---

Published in Transactions on Machine Learning Research (05/2025)
Jack W. Rac, Anna Potapenko, Siddhant M. Jayakumar, Chloe Hillier, and Timothy P. Lillicrap. Com- pressive transformers for long-range sequence modelling. In 8th International Conference on Learning Representations, ICLR 2020, Addis Ababa, Ethiopia, April 26-30, 2020. OpenReview.net, 2020. URL https://openreviie net fyd
Pranav Rajpurkar, Jian Zhang, Konstantin Lopyrev, and Percy Liang- Squad: 100, 000+ questions for 05250
machine comprehension of text. CoRR, abs/ abs/1606.05250, 2016. URL http:p///aaaii..ogg/bbbbeee/////
Luka Ribar, Ivan Chelombiev, Luke Hudlass-Galle Charlic Blake, Carlo Luschi, and Donglas Orr. Sparq at- tention: Bandwiddu--eficientt LLM inference. In Forty-first International Conference on Machine Learning,, ICML 2024, Vienna, Austria, July 21-27, 2024. OpenReview net, 2024.
Hojjat Salchinejad, Sharan Sankar, Joseph Barfett, Errol Colak, and Shahrokh Valace. Recent advances recurrent neural networks. arXiv preprint arXiv:1801.01078, 2017.
Utkarsh Saxena, Gobinda Saha, Sakshi Choudhary, and Kaushik Roy. Eigen attention: Attention in low-rank space for kv cache compression. arXiv preprint ar Xix:24408 05660,656666 2024a.
Utkarsh Saxena, Sayeh Sharify, Kaushik Roy, and Xin Wang. Resq: Mixed-precciion quantization of large language models with low-rank residuals. arXic preprint arXiv:2412.1436. 2024b.
Jay Shah, Ganesh Bikshandi, Ying Zhang, Vijay Thakkar, Pradeep Ramani, and Tri Dao. Flashattention-:: Fast and accurate attention with asynchrony and low-precision, 2024. URL. https://aarat wwwpsiiiioorre/eee abs/
08608
Uri Shaham, Elad Segal, Maor Ivgi, Avia Efrat, Ori Yoran, Adi Haviv, Ankit Gupta, Wenhan Xiong, Mor Geva, Jonathan Berant, and Omer Levy. SCROLLS: Standardized CompaRison Over Long Language Sequences, 2022. URL httpp:///aaxii.or/bbbbb///////// 03533
Uri Shaham, Maor Ivgi, Avia Efrat, Jonathan Berant, and Omer Levy. ZeroSCROLLS: A Zero-Shot Bench- mark for Long Text Understanding, 2023. URL httpp/:///aaxii.og//aab/////////
14196
Rana Shahout, Cong Liang, Shiji Xin, Qianru Lao, Yong Cui, Minlan Yu, and Michael Mitzenmacher. Fast inference for augmented large language models, 2024. URL httpps://axxiv... org/ abs/a/////// 24100.221.... 18248
Wenqi Shao, Mengzhao Chen, Zhaoyang Zhang, Peng Xu, Lirui Zhao, Zhiqian Li, Kaipeng Zhang, Peng Gao, Yu Qiao, and Ping Luo. Omniquant: Omnidirectionally calibrated quantization for large language models. XrXiv preprint arXiv:2008.. 13137, 2023.
Akshat Sharma, Hangliang Ding, Jianping Li, Neel Dani, and Minjia Zhang. Minikv: Pushing the limits of Ilm inference via 2-bit diverniiseeiiiiative kv cache, 2024. URL httpps org/ hts//bbs/24411 18077
Noam Shazeer. Fast Transformer Decoding: One Write-Head is All You Need, November 2019. URL http://arrivvoorr/aa///11..010
Ao Shen, Zhiyao Li, and Mingyu Gao. Fastswitch: Optimizing context switching efficiency in fairness-aware large language model serving, 2024. URL https://arxiv..rrr attgg///////......// 18424
Ying Sheng, Lianmin Zheng, Binhang Yuan, Zhuohan Li, Max Ryabinin, Beidi Chen, Percy Liang, Christo- pher Ré, lon Stoica, and Ce Zhang. Flexgen: High-throughput generative inference of large language models with a single GPU. In Andreas Krauso, Emma Brunskill, Kyunghyun Cho, Barbara Engelhardt, Sivan Sabato, and Jonathan Scarlett (eds.), International Conference on Machine Learning, ICML 2023, 23-29 July 2023, Honolulu, Hawaři, USA, volume 202 of Procendings of Machine Learning Research, pp. 31094 31116. PMLR, 2023. URL ftttps:/prroceedi::
mlr. wwwpppprwaeee//e/e shwng23a. html
Luohe Shi, Hongyi Zhang, Yao Yao, Zuchao Li, and Hai Zhao. Keep the cost down: A review on methods optimize Imm's kv-cache consumption. arXiv preprint arxin:2407. 18003, 2024.
54

---

Published in Transactions on Machine Learning Research (05/2025)
Vasudev Shyam, Jonathan Pilault, Emily Shepperd, Quentin Anthony, and Beren Millidge. Tree attention: Topology-aware decoding for long-context attention on gpu clusters, 2024. URL https://arxiv.org orrgg
Prajwal Singhania, Siddharth Singh, Shwai Hc, Soheil Feizi, and Abhinav Bhatcle. Loki: Low-rank keys for efficient sparse allention, 2024. URL https://arxiv: http://aaxxvv..r//bbbaaaae// arggabb//4406... 22042222002222222
Jimmy TH Smith, Andrew Warrington, and Scott W Linderman. Simplified state space layers for sequence modeling. arXiv preprint 020:2208.04033, 2022.
Dingjic Song, Shunian Chen, Guiming Hardy Chen, Fei Yu, Xiang Wan, and Benyon Wang. Milebench: Benchmarking milmss in long context. arXiv preprial arXiv:2404. 18532, 2024.
Jianlin Su, Murtadha Ahmed, Yu Lu, Shengfeng Pan, Wen Bo, and Yunfeng Liu. Roformer: Enhanced transformer with rotary position embedding. Neurocomputing. 568:127063, 2024.
Hanshi Sun, Lin Chang, Wenlei Bao, Size Zheng, Ningxin Zheng, Xin Liu, Harry Dong, Yuejie Chi, and Beidi Chen. Shadowkv: Kv cache in shadows for tigh-throughput long-context Ilm inference. arXiv preprint or Xiv:24 10.2444,,,, 2024a.
Mingjie Sun, Xinlei Chen, J. Zico Kolter, and Zhuang Liu. Massive activations in large language models. CoRR, abs/ abs/2402.17762, 2024b. doi: 10.48550/ 1.48550/ARXIV 2772...770... URL 0ttp://00...0//1.4455500
arXiv. 17762)
Yutao Sun, Li Dong, Shaohan Huang, Shuming Ma, Yuqing Xia, Jilong Xue, Jianyong Wang, and Furu Wei. Retentive Network: A Successor to Transformer for Large Language Models, August 2023. URL
http://arxiv www. httg//bb/3307.... 2t///
Yutao Sun, Li Dong, Yi Zhu, Shaohan Huang, Wenhui Wang, Shuming Ma, Quanlu Zhang, Jianyong Wang, and Furu Wei. You Only Cache Once: Recoder-Decodde Architectures for Language Models, May 2024c.
URL http://arxiv. http:://axivvorr//abb//000 org/abs/24055
05254
Yuxuan Sun, Ruikang Liu, Haoli Bai, Han Bao, Kang Zhao, Yuening Li, Jiaxin Hu, Xianzhi Yu, Lu Hou, Chun Yuan, et al. Flatquant: Flatness matters for I1m quantization. arXiv preprint or Xiv:22 10. 09426, 2024d.
Zhaoxuan Tan and Meng Jiang. User modeling in the era of large language models: Current research and future directions. arXiv preprint arXiv:231 115188 2023.
Hanlin Tang, Yang Lin, Jing Lin, Qingsen Han, Shikuan Hong, Yiwu Yao, and Gongyi Wang. Razorattention: Efficient KV cache compression through retrieval heads. CoRR, abs/ /bs//247..n 448911 2024a.
Jiaming Tang, Yilong Zhao, Kan Zhu, Guangxuan Xiao, Baris Kasikci, and Song Han. QUEST: query-aware sparsity for efficient long-context LLM inference. In Forty-first International Conference on Machine Learning, ICML 2024, Vienna, Austria, July 21-27, 2024. OpenRevicw.net, 2024b.
Yehui Tang, Yunhe Wang, Jianyuan Guo, Zhijun Tu, Kai Han, Hailin Hu, and Dacheng Tao. A survey on transformer compression. arXiv preprint arXiv:2402. 05964, 2024c.
Qian Tao, Wenyuan Yu, and Jingren Zhou. Asymkv: Enabling 1-bit quantization of kv cache with layer-wise asymmetric quantization configurations. arXiv preprint 1...:2210.12211111 2024.
Hugo Touvron, Thibaut Lavril, Gautier Izacard, Xavier Martinet, Marie-Anne Lachaux, Timothée Lacroix, Baptiste Rozière, Naman Goyal, Eric Hambro, Faisal Azhar, Aurélien Rodriguez, Armand Joulin, Edouard Grave, and Guillaume Lample. Llama: Open and efficient foundation language models. CoRR, abs/2300.1399 2023. doi: 10.48550/ARXIV 2307.13971.
URL https :ctps://dol. ://doi, org/10. 48550/arXiv.
2302. 13971
55

---

Published in Transactions on Machine Learning Research (05/2025)
Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Lukasz Kaiser, and Illia Polosukhin. Attention is all you need. Advances in neural informalion processing systems, 30, 2017.
Zhongwei Wan, Xin Wang, Che Liu, Samiul Alam, Yu Zheng, Zhongnan Qu, Shen Yan, Yi Zhu, Quanlu Zhang, Mosharaf Chowdhury, et al. Efficient large language models: A survey. arXiv preprint arXiv:2312. 03863, 1, 2023.
Zhongwei Wan, Xinjian Wu, Yu Zhang, Yi Xin, Chaofan Tao, Zhihong Zhu, Xin Wang, Siqi Luo, Jing Xiong, and Mi Zhang. D20: dynamic discriminative operations for efficient generative inference of large language models. CoRR, abs/2406. 13035, 2024a. doi: (18550//AXTT 1306. URL https://doi.c. org/10. 48550/arXiv. 2406. 13035
Zhongwei Wan, Ziang Wu, Che Liu, Jinfa Huang, Zhihong Zhu, Peng Jin, Longyue Wang, and Li Yuan. LOOK-M: look-once optimization in KV cache for efficient multimodal long-context inference. In Yaser Al-Onaizan, Mohit Bansal, and Yun-Nug Chen (eds.), Findings of the Association for Computational Linguistics: EMNLP 2024, Miami, Florida, USA, November 12-16, 2024, pp. 4065-4078. Association for or2/2024. findings-emnlp 235
Computational Linguistics, 2024b. URL
Ao Wang, Hui Chen, Jianchao Tan, Kefeng Zhang, Xunliang Cai, Zijia Lin, Jungong Han, and Guiguang Ding. Prefixkv: Adaptive prefix kv cache is what vision instruction-followin models need for efficient generation, 2024a. URL httpa///aaaxxvvrr/aaabbabeeeee//
03409
Junxiong Wang, Jing Nathan Yan, Albert Gu, and Alexander M Rush. Pretraining without attention. aiv preprint XrXi::2212 10544, 2022.
Sinong Wang, Belinda 7 Li, Madian Khabsa, Han Fang, and Hao Ma. Linformer: Self-attention with linear complexity. arxiv preprint ar arXiv:2006.00 768, 2020.
Wenxiao Wang, Wei Chen, Yicong Luo, Yongliu Long, Zhengkai Lin, Liye Zhang, Binbin Lin, Deng Cai, and Xiaofei He. Model compression and efficient inference for large language models: A survey. arXic preprint arXiv:2402.099 09744, 2024b.
Yumeng Wang and Zhenyang Xiao. LoMA: Lossless Compressed Memory Attention, February 2024. URL http://arxiv http//axxvvvooor/aaa/ abs/ /0tp:/201.00000 09486
Zheng Wang, Boxiao Jin, Zhongzhi Yu, and Minjia Zhang. Model tells you where to merge: Adaptive KV cache merging for 1mms on long-context tasks. CoRR, abs/2407.08444 2024c. doi: (0.48550/ARXIV.22 08454. URL
08454
Xiuying Wei, Yunchen Zhang, Xiangguo Zhang, Ruihao Gong, Shanghang Zhang, Qi Zhang, Fengwei Yu, and Xianglong Liu. Outlier suppression: Pushing the limit of low-bit transformer language models. Advances in Neural Information Processing Systems, 35:17402 17414, 2022.
Xiuying Wei, Yunchen Zhang, Yuhang Li, Xiangguo Zhang, Ruihao Gong, Jinyang Guo, and Xianglong Liu. Outlier suppression+: Accurate quantization of large language models by equivalent and optimal shifting and scaling, arXiv preprint Xrvvv:2304. 09145, 2023.
Bingyang Wu, Yinmin Zhong, Zili Zhang, Shengyu Liu, Fangyue Liu, Yuanhang Sun, Gang Huang, Xuanzhe Liu, and Xin Jin. Fast distributed inference serving for large language models, 2024a. URL https:
/arxiv ieciiv.org//ee atg/abs/2305..0 05920
Hao Wu, Patrick Judd, Xiaojie Zhang, Mikhail Isaev, and Paulius Micikevicius. Integer quantization for deep learning inference: Principles and empirical evaluation. arXiv preprint arXiv:200400600, 2020.
Haoning Wu, Dongxu Li, Bei Chen, and Junnan Li. Longvideobench: A benchmark for long-context inter- leaved video-language understanding, 2024b. URL http://awxxxv.oo//a////////
56

---

Published in Transactions on Machine Learning Research (05/2025)
Haoyi Wu and Kewei Tu. Layer-Condensee KV Cache for Efficient Inference of Large Language Models. In Lum-We Ku, Andre Martins, and Vivek Srikumar (eds.), Proceedings of the 62nd Annual Meeting of the Association for Computational Linguisties (Volume 1: Long Papers), ACL 2024, Bangkok, Thailand, August 11-16, 2024, pp. 11175-11188. Association for Computational Linguistics, 2024. doi: 10.18653/
V1/2024.ACI-LONGG 602. URL https://doi... http:///ooo.org//// wwt.//...0//...... 100.1/86//11/222...aa acw.o22...1111...00000 602
Jiayang Wu, Wensheng Gan, Zefeng Chen, Shicheng Wan, and Philip S. Yu. Multimodal large language models: A survey. In Jingrui He, Themis Palpanas, Xiaohua Hu, Alfredo Cuzzocrea, Dejing Dou, Dominik Slezak, Wei Wang, Aleksandra Gruca, Jerry Chun-Wei Lin, and Rakesh Agrawal (eds.), IEEE International Conference on Big Dala, BigDala 2023, Sorrento, Iaaly, December 15-18, 2023, pp. 2247-2256. TEFE, 2023. doi: BLLLO0/BIIGDAATA5004440000 10386743. URL
2023. 10386743
Likang Wu, Zhi Zheng, Zhaopeng Qiu, Hao Wang, Hongchao Gu, Tingjia Shen, Chuan Qin, Chen Zhu, Hengshu Zhu, Qi Liu, al. A survey on large language models for recommendation. World Wide Web, 27(5):60, 2024c.
Wenhao Wu, Yizhong Wang, Guangxuan Xiao, Hao Peng, and Yao Fu. Retrieval head mechanistically explains long-context factuality. CoRR, abs/2404.15574, 2024d.
You Wu, Haoyi Wu, and Kewei Tu. A Systematic Study of Cross-Layer KV Sharing for Efficient LLM Inference, October 2024e. URL http:///axiv..ot/ wwwpeiiii.oor/abaa wtwp//////b//aaaa/a////////// 14442
Haocheng Xi, Changhao Li, Jianfei Chen, and Jun Zhu. Training transformers with 4-bit in- tegers. In Alice Oh, Tristan Naumann, Amir Globerson, Kate Saenko, Moritz Hardt, and Sergey Levine (eds.), Advances in Newral Information Processing Systems 36: Annual Confer- ence on Neural Information Processing Systems 2023, NeurIPS 2023, New Orleans, LA, USA, December 10 16, 2023, 2023. URL hhttp://papers cwappprrs.ipp.... cc.psccrrrpprrrrep https/caapeeetlel//pppppeeeee 2003/haah/ BFfcBb4468117733100000744303 -ttp://ooaaaaooooootootttee---e- Conferenceeeeccceceeeeeeeeoeeeee html
Heming Xia, Zhe Yang, Qingxiu Dong, Peiyi Wang, Yongqi Li, Tao Ge, Tianyu Liu, Wenjie Li, and Zhi- lang Sui. Unlocking efficiency in large language model inference: A comprehensive survey of speculative decoding. arXiv preprint ar 2xxx:2401.07888 2024.
Chaojun Xiao, Pengle Zhang, Xu Han, Guangxuan Xiao, Yankai Lin, Zhengyan Zhang, Zhiyuan Liu, and Maosong Sun. Infllm: Training- free long-context extrapolation for llms with an efficient context memory, 2024a. URL https: // arxiv. org/abs/ 2402. 04617
Guangxuan Xiao, Ji Lin, Mickaël Seznec, Hao Wu, Julien Demouth, and Song Han. Smoothquant: Ac- curate and efficient post-training quantization for large language models. In Andreas Krause, Emma Brunskill, Kyunghyun Cho, Barbara. Engelhardt, Sivan Sabato, and Jonathan Scarlett (eds.), Interna- tional Conference on Machine Learning, ICML 2023, 23-29 July 2023, Honolulu, Hawaři, USA, vol- ume 202 of Proceedings of Machine Learning Research, pp. 38087-38099. PMLR, 2023. URL https: //proceedings. mlr. wtt..prree/vvv///// w0wp://00/xx2..xxxxoa html
Guangxuan Xiao, Jiaming Tang, Jingwei Zuo, Junxian Guo, Shang Yang, Haotian Tang, Yao Fu, and Song Han. Duoattention: Efficient. long-context. LLM inference with retrieval and streaming heads. CoRR, 0ts//2110.1081, 2024b.
Guangxuan Xiao, Yuandong Tian, Beidi Chen, Song Han, and Mike Lewis. Efficient streaming language models with attention sinks. In The Twelfth International Conference on Learning Representations, ICLR 2024. Vienna, Austria, May 7-11, 2024. OpenReview.net, 2024c. URL httt::://ooenreriee.eeee/// 1d=
Junbin Xiao, Xindi Shang, Angela Yao, and Tat-Seng Chua. Next-qa: Next phase of question-answering explaining temporal actions. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pp. 9786, June 2021.
57

---

Published in Transactions on Machine Learning Research (05/2025)
Yi Xiong, Hao Wu, Changxu Shao, Ziqing Wang, Rui Zhang, Yuhong Guo, Junping Zhao, Ke Zhang, and Zhemxuan Pan. Layerkv: Optimizing large language model serving with layer-wise kv cache management, 2024. URL httpp:://axxiw.or//bbbee.
00428
Dejing Xu, Zhou Zhao, Jun Xiao, Fei Wu, Hanwang Zhang, Xiangnan He, and Yueting Zhuang. Video question answering via gradually refined attention over appearance and motion. In ACM Mullimedia, 2017.
Derong Xu, Wei Chen, Wenjum Peng, Chao Zhang, Tong Xu, Xiangyu Zhao, Xian Wu, Yefeng Zheng, Yang Wang, and Enhong Chen. Large language models for generative information extraction: A survey. Frontiers of Computer Science, 18(6):186357, 2024a.
Fangyuan Xu, Tanya Goyal, and Eunsol Choi. Recycled attention: Efficient inference for long-context language models, 2024b. URT, https: /tatpps://rxiv.... wwwpexiiii.orvveeee abs/2411. 05787
Jiale Xu, Rui Zhang, Cong Guo, Weiming Hu, Zihan Liu, Feiyang Wu, Yu Feng, Shixuan Sun, Changxu Shao, Yuhong Guo, Junping Zhao, Ke Zhang, Minyi Guo, and Jingwen Leng. vtensor: Flexible virtual tensor management for elficient Ilm serving, 2024c. URL
Lingling Xu, Haoran Xie, Si-Zhao Joe Qin, Xiaohui Tao, and Fu Lee Wang. Parameter-effiiiet fine- tuning methods for pretrained language models: critical review and assessment. arXiv preprint arXiv:2312.1211 12148, 2023.
Rui Xu, Shu Yang, Yihui Wang, Bo Du, and Hao Chen. A survey on vision mamba: Models, applications and challenges. arXie preprint 2rXiv:240..18862 2024d.
Xin Xu and Zhouchen Lin. MixCon: A Hybrid Architecture for Efficient and Adaptive Sequence Modeling, In Ulle Endriss, Francisco S. Melo, Kerstin Bach, Alberto Bugarin-Diz, José M. Alonso-Moral Senén Barro, and Fredrik Hcintz (eds.), Frontiers in Artificial Intelligence and Applications. IOS Press, October 2024. ISBN 178-1-64438-548-9 doi: 10.3233 (0.3233/PAAA24059) URL htttps://eboooss wtw.:ksssssssrsss......... m1/doi/10. 3233/FAIA240593
Xiongxiao Xu, Yueqing Liang, Baixiang Huang, Zhiling Lan, and Kai Shu. Integrating mamba and trans- former for long-short range time series forecasting, ar preprint orXiv:2404.14757, 2024e.
Ruiqing Yan, Linghan Zheng, Xingbo Du, Han Zou, Yufeng Guo, and Jianfei Yang. RecurFormer: Not All Transformer Heads Need Sell-Attention, October 2024. URL httt:://araae http:///axxiv..oo/aa/////// 12850
Dongjie Yang, Xiaodong Han, Yan Gao, Yao Hu, Shilin Zhang, and Hai Zhao. Pyramidinfer: Pyramid KV cache compression for high-throughput LLM inference. In ACL, pp. 3258 3270. Association for Compu- tational Linguistics, 2024a.
Hua Yang, Duohai Li, and Shiman Li. MCSD: An Efficient Language Model with Diverse Fusion, July 2024b. URL http:// http:///axxv...// org/ ab/////s//2200 12230
June Yong Yang, Byeongwook Kim, Jeongin Bae, Beomseok Kwon, Gunho Park, Eunho Yang, Se Jung Kwon, and Dongsoo Lee. No token left behind: Reliable kv cache compression via importance-aware mixed precision quantization. arXie preprint arXiv:2402.0 18096, 2024c.
Yifei Yang, Zouying Cao, Qiguang Chen, Libo Qin, Dongjie Yang, Hai Zhao, and Zhi Chen. Kvsharer: Efficient inference via layer-wise dissimilar KV cache sharing. CoRR, abs/ 22102211..111.. 2024d. doi: 10. 48550/ARXIV. 20T.X.V..410.18.00 2410118517. URL
org/10. 48550/arXiv 2410 18517
Zhen Yang, J. N. Han, Kan Wu, Ruobing Xie, An Wang, Xingwu Sun, and Zhanhui Kang. Lossless KV Cache Compression to 2%, October 2024e. URL http://axxxvv..///bb/
15252
Zhenjie Yang, Xiaosong Jia, Hongyang Li, and Junchi Yan. Lim4drive: survey of large language models for autonomous driving. In NeurIPS 2024 Workshop on Open- World Agents, 2023.
58

---

Published in Transactions on Machine Learning Research (05/2025)
Jiayi Yao, Hanchen Li, Yuhan Liu, Siddhant Ray, Yihua Cheng, Qizheng Zhang, Kuntai Du, Shan Lu, and Junchen Jiang. Cacheblend: Fast large language model serving with cached knowledge fusion. arXiv preprint arXie:2405.16444 2024a.
Jinwei Yao, Kaiqi Chen, Kexun Zhang, Jiaxuan You, Binhang Yuan, Zeke Wang, and Tao Lin. Deft: Decoding with flash tree-attention for efficient troe-structured Ilm inference, 2024b. URL https: org/ htg/abb/24000 00242
Zhewei Yao, Reza Yazdani Aminabadi, Minjia Zhang, Xiaoxia Wu, Conglong Li, and Yuxiong He. Zero- quant: Efficient and affordable post-training quantization for large-scale transformers. Advances in Newral Information Processing Systems, 35:27168-277 2022.
Lu Ye, Ze Tao, Yong Huang, and Yang Li. Chunkattention: Efficient self-attention with prefix-aware kv cache and two-phase partition, 2024. URL http:///aaxxi...gaabbs////s 15220
Howard Yen, Tianyu Gao, and Danqi Chen. Long-Context Language Modeling with Parallel Context En- coding. In Lun-Wei Ku, Andre Martins, and Vivek Srikumar (eds.), Proceedings of the 62nd Annual (Volume 1: Long Papers), ACL 2024, Bangkok, Thailand, August 11-16, 2024, pp. 2588- 2618- Association for Computational Linguisties, 2024. doi: URL https://doi..rgg/ 1863300110011111//2022 acl-lon.. 142
Meeting of the Association for Computational Linguistics (
(0.18633/V1/2024.0 WAL0022AACI--LONG.GAC
Gyeong-In In Yu, Joo Seong Jeong, Geon Woo Kim, Soojeong Kim, and Byung-Gon Chun. Orca: A distributed serving system for transformer- based generative models. In Marcos K. Aguilera and Hakim Weatherspoon (eds.). 16th USENIX Symposium on Operaling Systems Design and Implementation, OSDI 2022, Carlsbad, CA, USA, July 11-13, 2022, pp. 521-538. USENIX Association, 2022. URL https: wwwwix.seei...ommm org/
Hao Yu, Zelan Yang, Shen Li, Yong Li, and Jianxin Wu. Effectively compress kv heads for Ilm. preprint Xiv: 07056, 2024.
arXiv
Linglan Yu and Jinyang Li. Stateful large language model serving with pensieve. ColR,, abs 00ss/2311.05116. 2023. doi: 00.4850/ARRA 2.....2..11..166 URL https://doi.org/ 000.0.8555///r00 arr/v..Xiivii... 201.60//..ii.211......
Jiayi Yuan, Hongvi Liu, Shaochen Zhong, Yu-Neng Chuang, Songchen Li, Guanchu Wang, Duy Le, Hongye Jin, Vipin Chaudhary, Zhaozhuo Xu, et al. Kv cache compression, but what must we give in return? comprehensive benchmark of long context capable approaches. arXiv preprint 2XXi::2077.01227 2024.
Yuanhan Zhang Yuan Liu, Haodong Duan et al. Mmbench: Is your multi-modal model an all-around player? arX1v:230.006881, 2023.
Yuxuan Yue, Zhihang Yuan, Haojie Duanmu, Sifan Zhou, Jianlong Wu, and Liqiang Nie. Wkvquant: Quan- tizing weight and key/valuu vvalue cache for large language models gains more. arXiv preprint 0rXiv24402.1066 2024.
Amir Zandieh, Majid Daliri, and Insu Han. Qil: 1-bit quantized jl transform for kv cache quantization with zero overhead. arXiv preprint arxiv:2406. 03482, 2024.
etc Zefan-Cal
2024. URL Arresone-Imm--v-cachee A curated list of awesome Ilm inference papers with codes, at
-VV---CC Open-source software available
Aohan Zeng, Xiao Liu, Zhengxiao Du, Zihan Wang, Hanyu Lai, Ming Ding, Zhuoyi Yang, Yifan Xu, Wendi Zheng, Xiao Xia, Weng Lam Tam, Zixuan Ma, Yufei Xue, Jidong Zhai, Wenguang Chen, Zhiyuan Liu, Peng Zhang, Yuxiao Dong, and Jie Tang. GLM-130B: an open bilingual pre-trained model. In The Eleventh International Conference on Learning Representations, ICLR 2023, Kigali, Rwanda, May 1-5, 2023. OpenReview.net, 2023. URL https://
Zihao Zeng, Bokai Lin, Tianqi Hou, Hao Zhang, and Zhijie Deng. In-context kv-cache eviction for 1lms via attention-gate. CoRR, abs/2410.122 12876, 2024.
59

---

Published in Transactions on Machine Learning Research (05/2025)
Duzhen Zhang, Yahan Yu, Jiahua Dong, Chenxing Li, Dan Su, Chenhui Chu, and Dong Yu. Mm-llms: Recent advances in multimodal large language models. In Lun-Wei Ku, Andre Martins, and Vivek Srikumar (eds.); Findings of the Association for Computational Linguistics, ACL 2024, Bangkok, Thailand and virtual meeting, August 11-16, 2024, pp. 12401-12430. Association for Computational Linguistics, 2024a. doi: 10. ACL.738. URL httpss//doii .tt.dii.orgg. org/10.
findings-acl.733
Hailin Zhang, Xiaodong Ji, Yilin Chen, Fangcheng Fu, Xupeng Miao, Xiaonan Nie, Weipeng Chen, and Bin Cui. Pqcache: Product quantization-basse kvcache for long context Ilm inference. arXiv preprint (xXi:2447.12820 2024b.
Jingyi Zhang, Jiaxing Huang, Sheng Jin, and Shijian Lu. Visiun-anguuae models for vision tasks: A survey. TEFF Trons. Pattern Anal. Mach. Intell., (6(8):5625-5644 2024c. doi: 10.1109/TPA.M.20 20t0:TPAMM.222.33300000 URL attpss///do org/10. 1109/TPAMI 2024. 3369699
Rongzhi Zhang, Kuang Wang, Liyuan Liu, Shuohang Wang, Hao Cheng, Chao Zhang, and Yelong Shen. Lorc: Low-rank compression for lmss kv cache with a progressive compression strategy. arxiv preprinl arXiv:2410. 03111, 2024d.
Tianyi Zhang, Varsha Kishore, Felix Wu, Kilian Q Weinberger, and Yoav Artzi. Bertscore: Evaluating text generation with bert, 2020. URL http:://aaxxvv.org//a///////
Xuan Zhang, Cunxiao Du, Chao Du, Tianyu Pang, Wei Gao, and Min Lin. Simlayerkv: A simple framework for layer-level KV cache reduction. CoRR, abs/ has/2410013846 2024e.
Yanqi Zhang, Yuwei Hu, Runyuan Zhao, John C. S. Lui, and Haibo Chen. Unifying kv cache compression for large language models with leankv, 2024f. URL https: //arxiv. org/ httg/ab//2412.0055 03131
Yuxin Zhang, Yuxuan Du, Gen Luo, Yunshan Zhong, Zhenyu Zhang, Shiwei Liu, and Rongrong Ji. Cam: Cache merging for memory-efficiet 1mms inference. In Forty-first International Conference on Machine Learning, ICML 2024, Vienna, Austria, July 21-27, 2024. OpenReview.nct, 2024g. URL
htttss///penrrviiew.. http://wwww..e.foomurorddttttnpp
Zeyu Zhang and Haiying Shen. Zero-delay qkv compression for mitigating kv cache and network bottlenecks in Ilm inference. arXiv preprint Xrxiv:2408. 04107, 2024.
Zhenyu Zhang, Ying Sheng, Tianyi Zhou, Tianlong Chen, Lianmin Zheng, Ruisi Cai, Zhao Song, Yuandong Tian, Christopher Ré, Clark Barrett, et al. H2o: Heavy-hitter oracle for efficient generative inference of large language models. Advances in Neural Information Processing Systems, 14.34661-34711 2023a.
Zhenyu Zhang, Ying Sheng, Tianyi Zhou, Tianlong Chen, Lianmin Zheng, Ruisi Cai, Zhao Song, Yuandong Tian, Christopher Ré, Clark W. Barrett, Zhangyang Wang, and Beidi Chen. H2O: heavy-hitter oracle for efficient generative inference of large language models. In Alice Oh, Tristan Naumann, Amir Globerson, Kate Saenko, Moritz Hardt, and Sergey Levine (eds.), Advances in Neural Information Processing Systems 36: Anruua Conference on Neural Information Processing Systems 2023, NewrIPS 2023, New Orleans, LA, USA, December 10 16, 2023, 2023b. URL http::/papees....../////// nips ccciis.cc/pppe wwcppappp.fleeeeeeeeeee siiles/paper/2000 wwwtrattt-oooeerraaaaaaoeee-eeee
html
Junqi Zhao, Zhijin Fang, Shu Li, Shaohui Yang, and Shichao He. Buzz: Bechive-structuree sparse kv cache with segmented heavy hitters for efficient Ilm inference, 2024a. URL https://arxiv.org org/abs///10.....
23079
Liang Zhao, Xiaocheng Feng, Xiachong Feng Bin Qin, and Ting Liu. Length extrapolation of transformers: A survey from the perspective of position encoding. arXiv preprint arXio:2312.170/ 2023.
Yilong Zhao, Chien-Yre Yu Lin, Kan Zhu, Zihao Ye, Loqun Chen, Size Zheng, Luis Ceze, Arvind Krishnamurthy, Tianqi Chen, and Baris Kasikci. Atom: Low-bit quantization for efficient and accurate Ilm serving. Proceedings of Machine Learning and Systems, 6:196-200 2024b.
60

---

Published in Transactions on Machine Learning Research (05/2025)
Ying Zhao and Jinjun Chen. A survey on differential privacy for unstructured data content. ACM Computing Surveys (CSUR), 54(10s):1122) 28, 2022.
Youpeng Zhao, Di Wu, and Jun Wang. Alisa: Accelerating large language model inference via sparsity-aware kv caching, 2024c. URL httts://arxiv..rr//bbbbb. 240./ 17312
Jianqiao Zheng, Sameera Ramasinghe, and Simon Lucey. Rethinking positional encoding. ar preprinl arXiv:2107.02501, 2021.
Lianmin Zheng, Liangsheng Yin, Zhiqiang Xie, Chuyue Sun, Jeff Huang, Cody Hao Yu, Shiyi Cao, Christos Kozyrakis, Ion Stoica, Joseph E. Gonzalez, Clark Barrett, and Ying Sheng. Sglang: Efficient execution of structured language model programs, 2024a. URL httph:ttp/////t //appii///xxv....// wwwxxiiv.rgg/ee /tt/////iwi.rr////////2////...00 07104
Zhen Zheng, Xin Ji, Taosong Fang, Fanghao Zhou, Chuanjie Liu, and Gang Peng. Batchllm: Optimizing large batched 11m inference with global prefix sharing and throughpul-oriented token batching, 2024b.
URL https://arxxt www..oog//gaooo wxggabbs/2422/ 221.0//24113099941
Yinmin Zhong, Shengyu Liu, Junda Chen, Jianbo Hu, Yibo Zhu, Xuanzhe Liu, Xin Jin, and Hao Zhang, Distserve: Disaggregating prefill and decoding for goodput-optimized large language model serving. In Ada Gavrilovska and Donglas B. Terry (eds.), 18th USENIX Symposium on Operating Systems Design and Implementation, OSDI 2024, Santa Clara, CA, USA, July 10-12, 2024, pp. 193 210. USENIX Association, 2024a. URL https://www... /wwpp/////ww.uuiixx..mmm usenix.enn........... wwwp:/uusnix.....eeeeee
Yiwu Zhong, Zhuoming Liu, Yin Li, and Liwei Wang. Aim: Adaptive inference of multi-modal 1lms via token merging and pruning. arXie preprint 3rXiv:2412.032/4 2024b.
Daquan Zhou, Kai Wang, Jianyang Gu, Xiangyu Peng, Dongze Lian, Yifan Zhang, Yang You, and Jiashi Feng. Dataset quantization. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pp. 17205 17216, 2023a.
Hongjaan Zhou, Fenglin Liu, Boyang Gu, Xinyu Zou, Jinfa Huang, Jinge Wu, Yiru Li, Sam S Chen, Peilin Zhou, Junling Liu, et al. A survey of large language models in medicine: Progress, application, and 2023b.
challenge. aixi preprint arXiv:2311.0511
Junjie Zhou, Yan Shu, Bo Zhao, Boya Wu, Shitao Xiao, Xi Yang, Yongping Xiong, Bo Zhang, Tiejun Huang, and Zheng Liu. Mlvu: A comprehensive benchmark for multi-task long video understanding. Xiviv preprint ar Xix:224006. 04264, 2024a.
Pan Zhou, Canyi Lu, Zhouchen Lin, and Chao Zhang. Tensor factorization for low-rank tensor completion. IEEE Transoctions on Image Processing, 27(3):1152-116; 2017.
Yiren Zhou, Sehsd-Mohsen Moosavi-Dezfooli, Ngai-Man Cheung, and Pascal Frossard. Adaptive quantization for deep neural network. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 32, 2018.
Zhanchao Zhou, Tianyi Wu, Zhiyun Jiang, and Zhenzhong Lan. Value Residual Learning For Alleviating 17897
Attention Concentration In Transformers, December 2024b. URL
Zixuan Zhou, Xuefci Ning, Ke Hong, Tianyu Fu, Jiaming Xu, Shiyao Li, Yuming Lou, Luning Wang, Zhihang Yuan, Xiuhong Li, et al. A survey on efficient inference for large language models. arXic preprint arXiv:2404.14 14294, 2024c.
Xunyu Zhu, Jian Li, Yong Liu, Can Ma, and Weiping Wang. A survey model compression for large language models. arXiv preprint arXim:2308. 07633, 2023.
Bohan Zhuang, Jing Liu, Zizheng Pan, Haoyu He, Yuetian Weng, and Chunhua Shen. A survey on efficient training of transformers. arXie preprint Xrx:2:300 01 01707,110000110001 2023.
Zayd Muhammad Kawakibi Zuhri, Muhammad Farid Adilazuarda, Ayu Purwarianti, and Alham Fikri Aji. MLKV: Muyti-Layer Value Heads for Memory Efficient Transformer Decoding, October 2024. URL
http://arxiv. .rg/ abs /etp://440609222
61


Reviews:
[{"review_id": "eqiCCBEvLX", "reviewer": "Reviewer_2duU", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Strengths:\n\n1. This paper is organized well, with figures explaining each section's taxonomy.\n\n2. The authors outline detailed future directions and provide a succinct summary of existing works.\n\nWeaknesses:\n\n1. The paper is unnecessarily long, with a few sections that are not directly related to the KV cache.", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "Yes", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "This paper covers a wide range of KV cache management techniques for LLMs, from low-rank compression to system-level optimizations. The KV cache becomes a major bottleneck during LLM inference, and this paper highlights a range of works that attempt to tackle this.", "claims_and_evidence_yn": "Yes", "claims_explanation": "", "audience_yn": "Yes", "audience_explanation": "", "requested_changes": "1. In Section 4.5.1: \"Firstly, ECKVH Yu et al. (2024), EigenAttention Saxena et al. (2024), and ZDC Zhang & Shen (2024) shows\nthat KV caches have a low-rank property, where a small number of top singular values retain most of the\ninformation. Using Singular Value Decomposition (SVD), the method compresses KV caches by grouping\nheads, applying SVD, and retaining top singular values, effectively reducing the number of KV heads with\nminimal error.\" The second sentence is ambiguous and does not describe what Eigen Attention does. Please edit this to specify which method the authors are discussing.\n\n2. There are repeated references (in Introduction) and keywords (eg, \"Equation\" in 2.2.2) throughout the paper.\n\n3. Please include the following works in the relevant sections: \n\n[1] P. Singhania et al., \"Loki: Low-rank Keys for Efficient Sparse Attention\", arXiv (2024).\n\n[2] C. Luo et al., \"HEADINFER: Memory-Efficient LLM Inference by Head-wise Offloading\", arXiv (2025).\n\n[3] Saxena et al., \"ResQ: Mixed-Precision Quantization of Large Language Models with Low-Rank Residuals\", arXiv (2024).\n\n4. Section 7 on datasets is not needed in this review and makes the paper unnecessarily long. In my opinion, it should be removed and/or summarized by citing related review works on LLM evaluation benchmarks/datasets.", "broader_impact_concerns": "NA", "additional_comments": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "jmQFBQ88W6", "reviewer": "Reviewer_6Kho", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "## Strengths:\n\n- **Comprehensive and Timely Coverage**: The paper provides an extensive review of KV cache management techniques across multiple dimensions (token-level, model-level, system-level). It also covers many recent works, making it a timely survey of the rapidly evolving field.\n\n- **Well-Structured Taxonomy**: The hierarchical categorization of methods creates a clear framework for understanding the relationships between different approaches, helping readers navigate the landscape of KV cache optimization.\n\n- **Comparative Analysis**: The paper presents detailed technical descriptions of each method, including mathematical formulations, algorithmic ideas, and implementation considerations.\nWithin each category, the authors provide comparative tables and analyses that highlight the similarities, differences, and trade-offs between different techniques.\n\n- **Future Research Directions**: Each section concludes with thoughtful discussions on research gaps and potential directions for future work, providing valuable guidance for researchers in the field.\n\n\n## Weaknesses:\n\n- **Limited Critical Analysis**: While the paper catalogs many techniques, it provides relatively limited critical analysis comparing the effectiveness of different approaches under various conditions or workloads. For example, how the token-level KV cache optimizations behave for prefill and decoding phrases. \n\n- **Insufficient Performance Benchmarks**: The paper lacks a unified benchmark comparison of different KV cache management techniques, making it difficult to identify which approaches perform best in which scenarios. The survey could also better highlight the gap between theoretical proposals and practical implementations, especially identifying which methods ae suitable for real-world deployment (e.g, some methods are not batching friendly and not able to deploy at scale).\n\n- **Limited Discussion on Hybrid Approaches**: There could be more exploration of how different optimization levels (token, model, system) can be combined synergistically. Is there any work that co-optimizes the KV cache from token-model or model-system perspectives?", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "Yes", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "This survey paper provides a comprehensive overview of Key-Value (KV) cache management strategies for accelerating Large Language Model (LLM) inference. It introduces a taxonomy categorizing techniques into token-level, model-level, and system-level optimizations.\n\nThe paper establishes foundational concepts, explaining transformers, auto-regressive generation, and the role of KV caches in reducing redundant computations. It analyzes KV cache operations' time and space complexities and highlights key management challenges.\n\nToken-level optimizations focus on techniques like KV cache selection, budget allocation, merging, quantization, and low-rank decomposition, offering detailed analyses and future directions. Model-level optimizations explore architectural innovations, including attention grouping, architecture adjustments, and non-transformer models like Mamba and RWKV. System-level optimizations address memory management, scheduling, and hardware-aware designs like virtual memory abstractions and heterogeneous computing.\n\nFinally, the paper compiles text and multi-modal datasets for evaluating LLM efficiency, providing a helpful reference for researchers in KV cache optimization.", "claims_and_evidence_yn": "Yes", "claims_explanation": "", "audience_yn": "Yes", "audience_explanation": "", "requested_changes": "## Critical Changes:\n\n- **Unified Benchmark Results**: Include a comprehensive benchmark comparison of representative methods from each category using standard datasets and metrics. This would significantly enhance the practical value of the survey by helping readers identify which approaches are most effective for specific scenarios. For example, in section 5.1, efficiency analysis, such as theoretical computation complexity for sharing or group, given notation in section 2, and summary of empirical results. One suggestion is that section 7 is largely unnecessary; an alternative is to group the evaluation datasets into long or short contexts and summarize the main KV cache optimization results, highlighting tradeoffs (e.g., more compression, more task performance loss). \n\n- **Expanded Critical Analysis**: Strengthen the comparative analysis between different approaches by more explicitly discussing their limitations, trade-offs, and scenarios where they underperform. This would provide a more balanced view of the field. For example, budget allocation techniques seem similar to cache selection, any core differences? e.g., PyramidKV, PyramidInfer, and DynamicKV also select KV caches and maybe merge into a single section. For cross-layer merging, the section is mostly ideas based discussion, any insights into why they work or not, and any empirical summary of findings, especially for KVSharer?\n\n- **Cross-Category Integration**: Add a dedicated section discussing how techniques from different categories (token-level, model-level, system-level) can be integrated for multiplicative benefits. This is important for practical implementations that often combine multiple approaches.\n\n- **Performance vs. Accuracy Trade-offs**: Enhance the discussion about how different KV cache optimization techniques impact model performance (speed, memory) versus accuracy, as this is a critical consideration for practitioners. For example, any tradeoff summary between dynamic and static cache selection? which scenarios were static or dynamic cache selections are better? Any detailed analysis of dynamic selection overhead or end-to-end efficiency gains? For head-wise budge allocation, are there speed ups, or require special software runtime support for head wise compression?\n\n## Strengthening Changes:\n\n- **Summarizing insights**: It would be better to summarize insights/findings for the KV cache optimizations in the introduction section. \n\n- **Case Studies**: If there are any, include case studies of KV cache management in production systems to bridge the gap between research and practice.\n\n- **Quantitative Comparison Framework**: Propose a standardized framework for evaluating and comparing KV cache management techniques, which would help future research in this area.\n\n- **Ethical and Privacy Implications**: Strengthen the discussion on privacy and security considerations related to KV cache management, particularly for multi-user scenarios.\n\n- **Visualization of Techniques**: The comparison tables are already a good visualization. However, adding more visualization diagrams or figures to illustrate the key concepts and differences between various KV cache management approaches would make the paper more accessible.\n\n- **Implementation Complexity**: Provide more discussion on the implementation complexity and deployment challenges of different techniques, as this impacts their practical adoption.\n\n- **Domain-Specific Optimizations**: Include a section on how KV cache management can be specialized for specific domains (e.g., healthcare, finance, legal) where context requirements may differ.\n\n- **Open Source Implementations**: Compile links to open-source implementations of key techniques, which would be valuable for researchers and practitioners looking to build upon existing work.\n\n- **Expanded Multi-Modal Discussion**: Given the growing importance of multi-modal models, expand the discussion on KV cache management for multi-modal contexts where different modalities may have different caching requirements.\n\n- **Writing Changes**: Section 2.1 can be compressed a lot, authors need to briefly discuss keys and values and how they are used during inference, and minimal context for KV cache is good.", "broader_impact_concerns": "NA", "additional_comments": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "aCQdbXAKkS", "reviewer": "Reviewer_85cD", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "**Strengths**:\n- It was not trivial to design a decent categorization given that there are so many methods, and some of them inevitably overlap. Yet, the way the paper is organized makes a lot of sense.\n- The survey is very very exhaustive.\n- Fig.2 helps a lot framing the topic and consulting the work, even for readers that might not be from the the field \n\n**Weaknesses**\n- In the preliminary section I got the feeling that some parts were written a bit \"in a rush\". \n   - On page 2. \"As the computational and memory requirements grow quadratically with sequence length.\"  -> this sentence is a bit ambiguous. The memory for storing the attention matrix is quadratic but the memory for KV Cache is linear wrt to sequence length. I would clarify this. \n   - On page 3. The positional embedding section mentions RoPE very quickly and spends more time on absolute positional encoding that are not used anymore. I wold explain that RoPE is quite different because it is added at each layer and might have impact on kv cache compression.\n   - On page 6, section \"saved time\". I wouldn't call this section \"saved time\" as what it is actually doing is estimating the saved FLOPs, which of course impact the time. However, the impact depends on many factors, like hardware, and this can be confusing. Maybe a better name for the sections would be computational complexity & memory complexity. \n\n- Figure 2. It looks a bit strange to have the Dataset and Benchmarks in Fig. 2, given that the plot is about KV Cache compression methods. I would regard that section more as a \"bonus\" rather than placing it on the same level as KV Cache management methods.\n\n- On page 17. “f we store all KV pairs in the memory with full precision, this cache grows exponentially with longer sequences, increasing the memory and bandwidth requirements significantly.” Why does the cache grow exponentially ?\n\n- On page 20. This sentence is not clear: “Then, we introduce the equivalent transformation function-based approaches”\n\n- The citation for FlashAttention v2 and v3 is missing \n\n- Some works for KV Cache compression are missing in the token eviction and quantization section: \n     - L2 norm based compression: https://arxiv.org/pdf/2406.11430 \n- These two works came out after the submission but they are very related:\n     - SVD based compression: https://arxiv.org/abs/2503.02812\n     - more for keys less for values: https://arxiv.org/abs/2502.15075 \n\n- In the datasets and benchmark sections, another popular dataset is ruler: https://github.com/NVIDIA/RULER , that was also recently proposed for multilingual: https://arxiv.org/abs/2503.01996v1 (I believe the extension was published after the submission of the manuscript)", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "Yes", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "This is a broad survey about the topic of KV Cache compression. This is a crucial aspect of efficient LLM inference, and while there are other surveys on the same topic (e.g. https://arxiv.org/pdf/2407.18003) , this work provides a broader overview and considers not only high level methods, but also low level kv cache management and engineering methods.", "claims_and_evidence_yn": "Yes", "claims_explanation": "", "audience_yn": "Yes", "audience_explanation": "", "requested_changes": "See weaknesses.", "broader_impact_concerns": "I don't have any concerns.", "additional_comments": "", "extra_scores": {}, "extra_text": {}}]

Author rebuttal:

Meta-review: The review process was relatively short, and all comments from the reviewers were included in the new revision. I personally agree that the topic is very interesting, with lots of new research coming out every month. As such, a highly comprehensive survey like this one can be valuable. However, I have some concerns that should be addressed:

1. Please provide a very strong proofreading of the entire paper, as there are several typos and errors, e.g., "by focusing on fine-grained the careful selection", "While KV cache selection (Sec. 4.1) focuses on prioritizing and storing only the most relevant tokens." (should be a comma), "KV cache Architecture alterations (Sec. 5.2" (missing closing bracket), etc.
2. The paper is unnecessarily long, with many repeated parts (e.g., all the "Summary and Future Directions" subsections share similar text), many parts that are only tangentially related to KV cache management, long bullet point lists (e.g., the dataset one). I would suggest to shorten multiple sections as much as possible to make for a clearer, more concise survey. Combining similar subsections is also a valid strategy.
Decision: Accept (poster)