MiniCache: KV Cache Compression in Depth Dimension for Large Language Models
Authors: Akide Liu
Jing Liu
Zizheng Pan
Yefei He
Gholamreza Haffari
Bohan Zhuang
Venue: NeurIPS 2024 poster (2024)
Keywords: KV Cache
Large Language Models
Efficiency AI
Primary area: natural_language_processing
TL;DR: We introduce MiniCache, a novel method to compress the KV cache between adjacent layers from a depth perspective, achieving superior compression ratios, high throughput, and near-lossless performance.

Abstract: A critical approach for efficiently deploying computationally demanding large language models (LLMs) is Key-Value (KV) caching. The KV cache stores key-value states of previously generated tokens, significantly reducing the need for repetitive computations and thereby lowering latency in autoregressive generation. However, the size of the KV cache grows linearly with sequence length, posing challenges for applications requiring long context input and extensive sequence generation. In this paper, we present a simple yet effective approach, called MiniCache, to compress the KV cache across layers from a novel depth perspective, significantly reducing the memory footprint for LLM inference. Our approach is based on the observation that KV cache states exhibit high similarity between the adjacent layers in the middle-to-deep portion of LLMs. To facilitate merging, we propose disentangling the states into the magnitude and direction components, interpolating the directions of the state vectors while preserving their lengths unchanged. Furthermore, we introduce a token retention strategy to keep highly distinct state pairs unmerged, thus preserving the information with minimal additional storage overhead. Our MiniCache is training-free and general, complementing existing KV cache compression strategies, such as quantization and sparsity.  We conduct a comprehensive evaluation of MiniCache utilizing various models including LLaMA-2, LLaMA-3, Phi-3, Mistral, and Mixtral across multiple benchmarks, demonstrating its exceptional performance in achieving superior compression ratios and high throughput. On the ShareGPT dataset, LLaMA-2-7B with cross-layer merging achieves a compression ratio of $1.53\times$. Additionally, since MiniCache is orthogonal to existing quantization techniques, it can achieve a compression ratio of up to $5.02\times$ when combined with the 4-bit quantization technique, enhancing inference throughput by approximately $5\times$ and reducing the memory footprint by $41\%$ compared to the FP16 full cache baseline, all while maintaining near-lossless performance. Project is available at https://minicache.vmv.re .

MiniCache: KV Cache Compression in Depth Dimension for Large Language Models
minicache
Akide Liu!
Jing Liu'
Zizheng Pan'
Yefei He²
Gholamreza Haffari!
Bohan Zhuang!2
'IP Lab, Monash University, Australia
ZZIP Lab, Zhejiang University, China
Abstract
A critical approach for efficiently deploying computationally demanding large lan- guage models (LLMs) is Key-Value (KV) caching. The KV cache stores key-value states of previously generated tokens, significantly reducing the need for repetitive computations and thereby lowering latency in autoregressive generation. However, the size of the KV cache grows linearly with sequence length, posing challenges for applications requiring long context input and extensive sequence generation. In this paper, present a simple yet effective approach, called MiniCache, to compress the KV cache across layers from novel depth perspective, significantly reducing the memory footprint for LLM inference. Our approach is based on the observation that KV cache states exhibit high similarity between the adjacent layers the middle-to-deep portion of LLMs. To facilitate merging, we propose disentangling the states into the magnitude and direction components, interpolating the directions of the state vectors while preserving their lengths unchanged. Furthermore, introduce a token retention strategy to keep highly distinct state pairs unmerged, thus preserving the information with minimal additional storage overhead. Our MiniCache training-free and general, complementing existing KV cache com- pression strategies, such quantization and sparsity. We conduct comprehensive evaluation of MiniCache utilizing various models including LLaMA-2, LLaMA-3, Phi-3, Mistral, and Mixtral across multiple benchmarks, demonstrating its excep- tional performance in achieving superior compression ratios and high throughput. On the ShareGPT dataset, LLaMA-2-7B with cross-layer merging achieves com- pression ratio of 1.53x. Additionally, since MiniCache is orthogonal to existing quantization techniques, can achieve a compression ratio of up 5.02x when combined with the 4-bit quantization technique, enhancing inference throughput by approximately 5x and reducing the memory footprint by 41% compared to the FP16 full cache baseline, all while maintaining near losslesss performance. Project is available https://minicache. vmv.re.
Introduction
Large Language Models (LLMs), exemplified by the GPT series [1, 2, 3] and the LLaMA series [4, 5, 6), have emerged as pivotal innovations within the artificial general intelligence, significantly enhancing the capabilities of natural language processing. However, these models are meticulously trained using extensive computational resources [7] and massive datasets [8], which enables them to
Corresponndig author. Email: bohan. zhuang gmaii..oomaaocommm
38th Conference on Neural Information Processing Systems (NcurIPS 2024).

---

(a) Cross-layer similarity
0.9
Input
Prevs.
V
0.9
Cache.
Coatre Skrbberty
0.7
Layer
Compression
Pruning Quant.
0.6
0.5
0.4
Layer
Decoding
0.3
CODA Dataget
0.2
TruhhuuQ Dataset
Atiention
Index:
MiniCache
0.80
Layer 3
EM Score GS40K
KV Cache
R X X
0.70
Compression
Layer
0.65
Cross Layer Merging
0.60
T+]
0.55
MasCacte
Layer -
X
28
0.50
Decoding
Attention
10 15 25 35 40 50 60 80 Number Layers Merged on LLaMA-3-700
LM Hend
(b) Merged layers EM score GSM8K
(e) Comparison between MiniCache and previous methods
Figure 1: Overview of our MiniCache strategy and example results: (a) shows the observation that the KV cache states between two adjacent layers are highly similar, particularly across the middle to deep layers. The x-axis uses index/2 to represent the similarities for each pair of layers. (b) compares the performance of MiniCache, and the mean baseline, which simply averages the KV caches of two layers, using the LLaMA-3-70B model [6] on the GSM8K dataset [10]. MiniCache, which begins merging from the half-layer depth, achieves near-lossless performance. (c) highlights the primary difference between MiniCache and previous approaches. MiniCache investigates the
inter-layer redundancy of KV caches along the depth dimension of LLMss an aspect overlooked by antra-layer-basee methods. Here, T refers to the last timestamp of pre-filling, and T des the first timestamp of decoding.
produce text that effectively mimics human writing styles and conducts complex reasoning analysis, yet raises the challenge of efficient deployment and serving. Within the inference framework of LLMs, KV caches [9] are crucial for storing pre-computed keys and values, thus avoiding repeated calculations over the preceding context and substantially enhancing LLMs' deployment efficiency. However, the increasing demand for longer sequence lengths results in voluminous cached states, leading to significant memory consumption during generation. For instance, 175B GPT-3 model [2], with batch size of 64 and a sequence length of 4,096 tokens (both prefilled and generated), requires approximately 1,208GB of GPU memory. This requirement is 3.45 greater than the memory used to store the model's weights. In this context, KV cache compression is of paramount importance due to its clear benefits: 1) largely reduces the memory footprint allowing for faster generation and larger batch serving; 2) it significantly lowers the cost per token, demonstrating substantial commercial benefits.
Existing KV cache compression efforts can be roughly categorized into two types, namely quantization and sparsity. The quantization approaches [11, 12] propose storing the KV states in low-bit numerical values. Typically, FlexGen [13] demonstrates that 4-bit KV cache quantization can achieve lossless performance. contrast, sparsity-driven methods aim to retain only the salient tokens while evicting the rest, either heuristically [1 [14, 15] adaptively [16]. Some approaches [11] [] explore the intersection of these two types, by assigning high-bit to salient tokens and extremely low-bit to the rest of the tokens, achieving more aggressive memory gain. Despite these innovations, existing literature merely consider the intra-layer redundancy, while neglecting another important complementary direction the inter-layer redundancy, illustrated in the Figure I(c).

---

Our amalysis begins by exploring the redundancy of KV caches along the depth dimension, as shown in Figure ((a). We observe that KV cache states exhibit high similarity between neighbouring layers in the middle-to-deep portions of LLMs. This intriguing property suggests that states paired by position between adjacent layers can be accurately merged into a single state space with a strong performance guarantee, as illustrated in Figure 1(b). This approach significantly reduces the memory footprint without the need to retain individual states for each attention layer. Note that these observations are pertinent to dynamic inference strategies such as ofxtureeof-depths [17] and layer-wise early exiting [18, 191, which optimize computational paths by skipping non-critical layers to enhance training and inference efficiency. Furthermore, layer pruning methods [20] highlight considerable redundancy in deeper layers. However, despite these advancements, the redundancy KV caches along the depth dimension has largely been overlooked.
In this paper, we propose MiniCache, a simple yet effective cross-layer KV cache compression method aimed at advancing the inference efficiency of LLMs. MiniCache consists of two essential components. Firstly, we introduce an accurate cache merging strategy, employing a reparameteri- zation of state vectors that decompose them into the magnitude and direction components, akin to weight normalization [21]. This approach allows for effective interpolation of the directional compo- nent in polar coordinates while preserving the original state norms to retain much information as possible. This interpolation refers to the cross-layer merging as shown in the Figure 1 (c). Secondly, recognize that small subset of state pairs, characterized by low similarities but carrying largely distinct semantic meanings, are unsuitable for inter-layer merging. To address this, we propose a token retention strategy to minimize performance degradation, which involves separately retaining these outlier pairs. Our framework is notably memory-efficieme requiring storage for only a single high-dimensional directional component, along with minimal extra memory overhead. The over- head consists of a few unmergeable tokens and their corresponding indexes, as well as token-wise magnitudes accurately restore the original states.
We conduct extensive experiments with representative LLMs, including Mixtral-8x777 [22], Phi- 3-Mini [23], and LLaMA-3 [6] 8B and 70B, respectively. Our method is benchmarked across diverse range of question answering and generation datasets [24, 25, 26, 27, 28, 29, 30, 31] using the an-eval-harness [32]. Additionally,
evaluate our results on LongBench [33] for oong-eeqennce generation. The results demonstrate that MiniCache can reduce the memory footprint required for LLM inference by up to 41%, while simultaneously enhancing throughput by approximately 5x
compared to fully cached baseline, clearly surpassing existing methods [11, 12, 14, 15].
Our contributions are summarized as follows:
We introduce MiniCache, a simple yet highly effective framework for KV cache compression. MiniCache pioneers the exploration of KV cache compression along the depth dimension, thereby significantly expanding its capabilities.
We observe a fascinating characteristic of cross-layer KV cache states: high similarity between adjacent layers in the middle later stages of LLMs. Additionally, find that not all state pairs are suitable for merging.
We propose an accurate and memory-efficient method for cross-layer cache merging, comprising a reparameterizztiion strategy and token retention mechanism. Our method complements existing KV cache compression approaches, further enhancing LLM serving efficiency.
Our MiniCache performs favourably against the state-of-the-art methods. Notably, our 4-bit MiniCache achieves a strong compression ratio of up to 5.02x, 5x higher inference throughput, and 41% memory reduction compared to the FP16 full cache baseline with near-lossless performance.
2
Related Work
Efficient inference for LLMs. Large Language Models (LLMs) are constrained by considerable computational and memory requirements during inference, particularly in resource-constrained environments. To mitigate these challenges, variety of efficient inference techniques have been developed. For instance, dynamic inference methods [18, 34, 35, 36, 37, 38), represented by mixture- of-experts (MoE) [39, 40, 41, 42, 43), adaptively select specific sub-structures of the model during the inference process based on the input data, significantly improving inference efficiency while keeping model capacity. Techniques like Multi- Multi-Query Query Attention [44, 45], Kernel-drived attentions
3

---

[46, 47, 48, 49], and low-rank attentions [43, 50, 51, 52] approximate the functionality of traditional attention mechanisms but with more efficient implementations. Quantization strategies [53, 54, 55, 56] involve converting the model's s weights and activations into low bit-widhh format, thereby reducing memory footprint and computational intensity. Sparsification approaches [14, 15, 57, 58] eliminate unnecessary elements in both model weights and token representations, further enhancing efficiency. Some closely related works, such as MoD [17] and LayerSkips [19], considered the dynamic inference nature to ignore unimportant layers according to input [59]. However, these methods require an additional fine-tuning process or carefully designed pre-training stages, which reduces the adaptivity of these methods. MiniCache relies inter-layer similarity observations to perform cross-layer merging, significantly reducing memory demand.
Model merging. Merging compression involves the aggregation of model's parameters and activations at various granularities. This process enhances the efficiency of inference in large models and facilitates huge redundancy [60]. Linear Mode Connectivity (LMCC) [61] enables the fine-tuning of models from a shared pre-trained base. Commonly, weight averaging [62] employed as an efficient technique to perform merge compression. Notably, Model Soup [63] utilizes linear averaging in this context. Advanced methods like TIES Merging [64], Model Breadcrumbs [65], and DARE [66] further enhance this process by sparsifying and combining model parameters, enabling the merging of models without sacrificing performance capabilities. Additionally, Spherical Linear intERPolation (SLERP) [67] extends beyond simple weight averaging by interpolating between model parameters. The Fisher information matrix [68] and RegMean-basee methods [69] further optimize merges to produce ideal weights, minimizing the & distance to generation outputs while preserving the privacy of the training data. However, most existing works focus on merging model parameters, with the concept of depth-wise mergeability not being thoroughly explored in prior research. MiniCache focuses on the KV cache token merging in the depth dimensional of LLMs.
Motivation
In the below, we present new observations in novel cross-layer perspective.
OpenBookOA
50
Mean KV
0.6
40
Cosine Siiilarity
PIDA
Dort Match
16
30
Layer 18 19
0.88
20
0.0
Layer 20 -2
Layer
22 -3
0.0 0
UhQA
10
Layer 26
-0.4
Layer 28
Mama 78
LLama 308
Laaa 88
Mistral 8x78
-0.6
30 32
MiniCache
20
40
60
80
100
RTE
Baseline
Models
Tokens Index
Mean
(a) Simple average baseline vs. full cache on GSM8K
(b) Pairwise similarity in adjacent layers KV cache
Winogrande
(c) Benchmark on five QA datasets
Figure 2: Overall of our explorations and observations (a) shows the strong bascline by performing average merging on the KV cache. (b) shows the pairwise similarity of cache states between adjacent layers. (c) compares the MiniCache, simple average, and full cache baseline across five different datasets.
3.1 Cross-Layer Redundancy in KV Cache
Prior studies have revealed the ineffectiveness of middle-to-deep layers in LLMs [20]. Thus, layer- wise early exiting in this case can effectively avoid the redundant computation with minor effect on LLM performance [19, 70]. Inspired by this, we explore layer-wise merging of KV cache in LLMs, starting with a simple baseline by averaging full tokens across adjacent layers. We provide our key observations follows.
Observation I: KV cache shares a high similarity between adjacent layers. Based on LLaMA-3- 70B [6], we conduct zero-shot inference on the validation sets of three widely recognized benchmarks: COQA [71], GSM8K [10] and TruthfulQA [72]. general, we find that KV cache the shallow layers exhibits low similarity, whereas those in the middle-to-deep layers are very similar to one

---

another based on angular distance, as shown in Figure 1(a). Next, we merge the KV cache across adjacent layers by conducting five-shot inference with LLaMA-2-7B, LLaMA-2-133 [5], LLaMA- 3-8B [6], and Mixtral-8x7B [22] on GSM8K [10]. Specifically, starting from the middle layer of each model, we merge the KV cache in the adjacent two layers. As shown in Figure 2(a), we observe a favourable performance for different LLMs, which reveals the huge potential for efficiency improvement by sharing the KV cache across adjacent layers during LLM decoding.
Observation 2: Not all tokens are equally important to merge, a few distinct pairs require retention. Recent works [15, 16] in KV cache compression have found that keeping a few salient tokens at each layer, which contribute the majority of attention scores, could be sufficient to maintain LLM performance. In our case, we speculate that certain token pairs in adjacent layers also exhibit outlier behaviours, showing strong semantic differences that make them unsuitable for merging. Based on COQA [71] and LLaMA-2-7H [5], we investigate the similarities at the level of token pairs. As shown in Figure 2(b), we find a significant portion of token pairs share high similarities across adjacent layers. However, we observe a few outlier pairs, such as indices 0 and 15, with large margins of difference. We consider these tokens non-mergeable due to their significant differences. We also show that merging these distinct tokens results in performance degradation, corresponding to 0 row in Table 2. Thus, while cross-layer merging is a promising strategy for reducing memory burdens, it must be implemented with careful consideration of token-level similarities to ensure optimal performance, shown in Figure 2(c).
4 Method
In this section, we introduce our MiniCache, a simple yet effective method aimed at trimming the KV cache redundancy the depth dimension. This framework exploits the high similarity of KV cache states between adjacent layers and consists of two primary components: reparameterization-ae based merging strategy and a token retention mechanism. The merging strategy compresses the KV cache states in adjacent layers aggregate them into single shared memory space, beginning from the middle of the model. The token retention mechanism mitigates information lost by retaining the highly distinct state pairs with minimal additional memory cost. With the merged cache, retention tokens, and magnitudes, we can accurately restore the original cache states for token decoding.
4.1 Cross-Layer Compression
Our method commences with the identification of an optimal starting layer S. Observations in Section 3.1 indicate that the KV cache from middle-to-deep layers consistently exhibits patterns of high similarity across adjacent layers. Consequently, we select the starting layer from the middle of the LLM, specifically S = L/2. From this layer onward, the KV pairs are assumed to be sufficiently similar across adjacent layers to warrant their consolidation. Central to this approach is merge function, F, which is designed to integrate the KV caches of consecutive layers into a single, unified cache. We define æ as the vectorized cache state of single token, where the superscript indicates the layer index and the subscripts k and denote the keys and values, respectively. Specifically, for pair of key/value tokens at the same position in layers I and I 1, the merged cache is computed as
1.1-1 F(m xx/1/ d.' =-='' x
This consolidation process effectively eliminates the need to store and process the original memory- intensive keys and values in each layer independently. Instead, it approximates shared cache across the adjacent layers.
4.2 KV Cache Merging and Restoration
cache merging. To perform the pairwise merging, one solution is to directly average a pair of KV tokens, analogous to model merging [63, 64]. However, we observe that direct averaging can cause significant information loss. We conjecture that the distance between activations can be larger than that of weights due to the presence of outlier activation channels with extremely large magnitudes in LLMMs [73, 74], while weights typically have relatively quite small magnitudes. A potential method to compensate for this information loss is to project from to r'11
Reparameterixation-bn
5

---

KV cache compression layer
Fetch for layer I and - 1
Keep
KV
original KV Cache
merged KV Cache
Rescale
Store C
retention token
Recover
cache magnitude
operation
Keep
(a) Cross-layer Compression
(b) Restoration
Figure 3: The illustration of the proposed method MiniCache. (a) depicts the cross-layer compression process. We fetch the KV caches, from layers / and 1, and merge them into shared states via Eq. (3). Additionally, we compute the & norm for the caches to obtain their magnitudes. Furthermore, select unmergable tokens for retention, then store merged cache, retention tokens, and magnitudes at layer / in C. (b) illustrates the restoration process for layers and 1, which includes magnitude rescaling in Eq. (2) and retention token recovery.
and x', then rescale the projected vectors based on their relative magnitudes to exactly restore the original states. However, this approach requires extensive additional storage and computations; for example, restoring x g needs both and x', which undermines the benefits of cache merging. To efficiently merge token pairs, we draw inspiration from weight normalization [21]. which disentangles model parameters into the magnitude and direction components to accelerate the convergence of stochastic gradient descent. Additionally, we take cues from DoRA [75], which employs similar way to resemble the learning behavior of parameter-efficient fine-tuning compared to full fine-tuning. In our case, the reparameterization can be formulated as follows:
al = c
||"|||| ||eli-=1||> -/-1 = cli-1
100 ||e!
(2)
2d-1 where e is the directional vector. This decomposition ensures that Ieet T is a unit vector, and allows the restored states to match the Eg norm of the original states, thereby preserving the cache's information as much possible. The restoration is shown as Figure 3(b). For brevity, we omit the subscripts & and v. as keys and values are decomposed in the same way. For estimating the directional component 1t we follow SLERP [67], which adaptively handles the interpolation, which often resembles rotation-like transformations. The choice of SLERP the merging function is strategic, as facilitates interpolation along the shortest path on the unit sphere between two high-dimensional vectors, thereby preserving their geometric integrity, which refers to merge operation in Figure 3(a). This is crucial for maintaining the semantic and syntactic properties of the KV caches. The formula for SLERP in our context is:
d.--1 sin((1 - -))
x'-1
sin(t((--1) (i
sin(OW-1) |
x
represents the angle between vectors x and a/11, and sin(-) (-) the sine function. / is interpolation hyperparameter that adjusts the relative influence of each vector on the resulting direction, tailored to the layer depth and specific characteristics the KV pairs. Note that when set 0.5. it will become average merging along the geometry surface, which consider special cases in Eq. (A). The merged cache for each token pair is then a concatenation the directional vector, magnitude and 1 denoting as c! = [e 1|
where Q4//11 arccos
|| |||
||>'''||, "1, ||=d|], -2 cached components as shown in Figure 3(a). Note that apart from storing the merged directional vector, we only need to store additional token- wise magnitude and angle scalars, which is memory efficient. In this way, we achieve substantial memory efficiencies through reduced redundancy while ensuring the retention of the critical functional characteristics of the original KV pairs across transformer layers.
Unmergeable token retention. Highly distinct pairs are sensitive to merging operations, leading us to propose unmergeable token retention, shown in Figure 3(a). Despite the high similarity between

---

KV cache states across neighbouring layers, few sensitive distinct pairs remain that are significantly difficult to merge and share. Aligned with previous studies, these distinct tokens carry substantial semantic meanings [15, 16]. We observe that merging sensitive tokens, which results in the loss of layer-specific information, can lead to significant performance degradation. Therefore, it crucial to properly disentangle the shared and unique information between adjacent layers. To address this issue, we designed a token retention strategy to selectively retain tokens that cannot be merged based on their angular distance, defined as: d(w', _I= L. For the KV caches, the minimum and maximum angular distances are determined to identify the unmergeable tokens.
The set of required token indices to keep, I, is obtained by:
{i d; < dmin + (dmax - damin) ,7),
where y a predefined hyperparameter that controls the retention threshold. The tokens with indices in I are retained and not compressed during the merge, which ensures that performance does not decline by preventing the loss of unmergeable tokens.
Next, let X E R be either the key or value cache at one attention layer, where n denotes the number of tokens and is the number of hidden dimensions, and E E Ig*xh be the shared KV cache states. For each pair of neighbouring two layers, the unmergeable tokens are selected along with the token dimension by R [I]. R [II] then restoring our compressed caches by x = - R'-1, as shown in Figure 3(b). Overall, we share the final cache for the two layers as C |EU-1, R¹, R -|x |I 1]. This cache includes the shared KV cache states, retention of unmerged tokens, magnitude vectors for each layer, and the eekennkeeping index, respectively. These additional components are quite lightweight. Thus compared to full-layer caches, our method remains memory effirieet,, as discussed in Sec. 4.3.
Cache restoration. After obtaining the shared cache C further need to approximately restore the original cache states for the current token decoding, as shown in Fig. 3(b). Specifically, to restore X we first rescale the directional shared states with the corresponding magnitude vector along the token dimension, denoted as E ||x] J.. Subsequently, we perform retention token recovery by placing the sensitive tokens according their token indices.
4.3 Efficiency Discussion
Compression efficiency. We primarily analyze our memory efficiency in terms of the number of tokens used. Next, let be the number of layers and and b is the batch size, and are input and output sequence length respectively. We consider the FP16 storage for KV cache. The full cache memory usage is given by 4brh(s + n). In study, begin merging layers from the middle to the deeper layers, consolidating the KV cache states of every two layers into a single shared state space. As result, effectively reduce the GPU memory usage in the decoding inference to hb + n), demonstrating a significant compression rate.
Restoration efficiency. We then analyze the additional memory cost incurred during the restora- tion process, which During the magnitude rescaling phase, we save an additional norm vector for the corresponding layers in the KV cache. It is important to note that the norm vector is in the shape of xxb which has a single channel dimension compared to the fully ranked original KV states. Additionally, we suppose that the retention threshold can be set to 0.05. Therefore, we have (0.(0.)5 + n)) tokens retained without compression. Finally, our overall memory requirement is given by (3.1h 2)br(s + n). The detailed derivation is shown in the Appendix E.
5 Experiments
We demonstrated that our MiniCache can perform merging compression on the latter half of the layers of LLMs with minimal performance degradation.
Implementation details. Our experiments are based on representative model families of LLMs. including a compact LLM Phi-3-Mini [23] and an MoE LLM Mixtral-8x7B [22]. Additionally, we
adopt LLaMA-33 A-3 [6] 8B and 70B models to explore how our method generalizes to larger LLMs. We
sample ten tasks from Im-eval-harness [32], including COPA [24], MathQA [25], OpenBookQA [26],
PIQA [27], RTE [28], WinoGrande [29], XSUM [30], and CNN/Daily Mail [31]. We also evaluate
long-sequence generation on LongBench [33]. We compare method with fully cached baseline,
7

---

and other methods such as round-to to-nnerest quantization (RTN) [76], SmoothQuant [73] and KIVI [11].
For the proposed MiniCache, we set the interpolation parameter t. to 0.6, indicating that the merged results have a smaller rotation angle the next layer. Furthermore, set the token retention threshold to 0.05, according to the statistics of unmergeable tokens across multiple datasets. In addition to our merging method, we also consider a strong baseline of average merging. For sequential loading of large models, we utilize NVIDIA 4 A100 80GB GPUs, more details refers to Appendix D.
Main results. We evaluate MiniCache by merging KV caches across all layers on GSM8K, COQA, and TruthfulQA. The results are shown in Figure 4. In general, we demonstrate the general effective- layers across different sized LLMs. Moreover, the
ness of merging KV caches from middle-to-des proposed MiniCache demonstrates a consistent and significant advantage over the averaging baseline. We also illustrate the performance of merging KV caches across half of the layers with the blue lines, where MiniCache still maintains a robust performance and achieves the best compression ratio. Besides, we find that our method is even more effective for larger LLMs. For instance, based on LLaMA-3-70B, MiniCache shows nearly zero performance drop even with the KV cache in 87.5% of the layers merged on the COQA dataset. This highlights the adaptability and efficiency of our approach in handling large-scale models while ensuring minimal performance degradation.
LongBench. We also conduct experiments to evaluate performance and quality in long- long-sequence gen- eration using the LongBench dataset [33], as shown in Table 1. Our experiments applicd MiniCache over several models: ALLMMA-2-7B-Chan I.L.MA-2-133 Cha- Mistral-7B, and Mistral-7B-Inntteen is important to note that our MiniCache method maintains orthogonality with all existing quan- tization and sparsity (refers to Table A) methods at both the model and token-wise levels. When combined with KIVI-4bit KV cache quantization, our approach achieves a compression rate of 5.02x, with minimal impact on accuracy across various challenging long-context generation tasks. The combination of MiniCache and KIVI-4bit KV cache quantization demonstrates significant memory savings without compromising the model's 's ability to handle long sequences effectively. This high-
COMEK Med
CGHBC Mi77
GSMRE
COHSE, LUA00-3--000
0.00
01.00
0.35
2.04
6:50
0.75
0.3
3.70
2.3
0.85
202
26-3400 Paar
0.30
Motral 0.10 Naan
(LIMA 5:55 Mope
0.95
Boadine
0.25 Brodfine ne
0.50
10121444800220
Navkerct Layer Merget on Ps May
Mavia of lage Meapel on
Lappe le aged Liaite
CODA. Ph: 3-Mb
C006.
0.63
0.00
0.6
0.00
0.33
LLan 180 MeaDache
9.635
1.00
M Mirta- 2:70 Mrr Cadw
11985 726 MorCare
B45
E34
LLAMA
0.43
5525
4.4. a DIIZERI CETESRINNRRRIIIEIIIN 411
0.2 e SOLZSTAAENOEAAAOOOO
0.2.4 3000540000022403333
h1mber Laver Marged LLANA 700
Tutti Laatt
76.33/96 Mars
Moral 0075 Hean
LLA4 5:55 Phyn
LL 755 Moon
Seadin
Pergen PF
SAKON Sap Mergne M4 6075
RaVEer Lapel Meiget LEARA 110
WATOO Laper 758
Figure 4: Performance comparisons between our proposed MiniCache with the "averaging baseline" and the "unmerged full cache baseline" on multiple datasets with Phi3-Mini, Mixtral-8x7B, L.aMA- 3-8B, and LLaMA-3-70BB, More result details are shown in Appendix F. The x-axis indicates the number of layers merged. As more layers are merged, a greater reduction in memory usage is achieved.
8

---

Table 1: Evaluation of different KV cache compression methods on LongBench. MiniCache builds on top of 4-bit KIVI [11] and achieves the best performance with the strongest compression rate.
Model
Melhod
LCC
RepuBench-IP
PR-en
TREC
Zwlkimqa
GocReport
MQA-zh
Average
Compression Ratio
Baseline
58.16
52.19
10.12
64.90
31.12
27.09
10.12
36.41
1x
RTN [76]
15.44
8.76
0.79
4.00
0.30
1.93
0.07
4.90
3.21x
Ulama-2-7B-CUn
SmunthQuumt [73]
2.15%
KIVI-2 [11]
43.71
3.9%
MiniCache
$8.03
52.01
9.00
61.00
30.58
25.32
10.13
35,44
5,02x
Baseline
48.06
50.08
14.25
68.50
13.09
27.76
7.23
32.71
1x
RTN [76]
20.89
18.62
0.33
0.00
0.52
1.68
0.16
6.03
3.21x
Ulama-2-13B8
SuaethQuant [73]
16
2.15%
[it]
7.01
MiniCache
48.75
08.59
13.00
68.00
14.36
26
7.99
32.61
5,02x
Baseline
65.06
60.16
17.71
68.00
10.87
20.09
17.10
37.33
1x
KTN [76]
27.98
26.18
3.34
13,90
1.11
2.49
0.45
10.51
3,21X
Mistral 7B
SuoothQuaax [73]
40.63
35.14
3.40
30.50
6.03
5,00
4,12
17.55
2.15%
KIVI-2 [11]
MiniCache
60.98
13.92
67.00
10.50
18.06
7,88
35.75
5.02x
Baseline
55.51
48.96
60.00
71.00
27.33
42.74
48.32
1x
KTN [76]
32.36
33.23
0.67
1.00
2.25
10.03
2.30
11.55
3,218
Mistral Instruct
SuoothQuam [73]
43.84
38.63
4.79
39,50
10.34
23.61
8.33
24.43
2.15%
KIVI-2 [11]
51.02
24.97
lights the potential of our method to optimize large language models for tasks requiring extensive context, making them more efficient and scalable real-world applications.
Efficiency analysis. To assess the acceleration capabilities MiniCache, we conduct evaluations based on the methodologies employed in vLLM [77] and KIVI [11]. We generate synthetic workloads derived from ShareGPT, which include real input and output texts from LLM services. The dataset features an average input prompt length of 161 tokens and output length of 338 tokens. Using the LLaMA-2-7B model on a single 80GB NVIDIA A 100 GPU, we benchmark our method in batch-serving scenario, comparing peak memory usage and throughput among 2-bit KIVI, 4-bit MiniCache, and an FP16 baseline. As illustrated in Figure 5, with batch size of 128, MiniCache reduces memory usage by 25GB, achieving 41% memory saving. In terms of throughput, MiniCache outperforms the FP16 baseline by approximately 5x. Additionally, despite utilizing 4-bit quantization, MiniCache benefits from merging and sharing KV caches across adjacent layers, resulting in 111.2 x.29x higher throughput compared to the 2- KIVI. These results demonstrate that MiniCache offers state-of-the art trade-off between efficiency and performance.
80
Pook Momory Usagon (G8)
ghout (aMenaa22d
0.500-
1.0
2500
0.450
50
2000
0.425
40
1500
Raseline FP16
Saselne FP16
0.375
20
MINICache
KN
1300
MINiCache
ØVI
0.350
0.3
0.4
0.5
0.6
0.7
0.0
250
150
Batch
Batchh
Figure 6: LLaMA-3-8B [6] to experiment on the GSM8K [10]. The right axis is the normalized frequency the relative magni- tude ratio. Optional shows
(a) BS. Peak Memory Usage (b) BS. Decoding Throughput
Figure 5: Memory usage and throughput comparison between 4-bit MiniCache, 2-bit KIVI, and 16-bit Baseline. MiniCache can achieve higher throughput by enabling a larger batch size while reducing memory footprints via LLaMA-2-7B [5].
strong correlation with frequency.
Ablation Study
Table Comparisons of various token retention thresholds by LLaMA-2-7B [5] on three benchmarks.
The effect of interpretation parameter t. We explore the effects of the interpretation parameter L on perfor- mance, particularly in relation to the relative magnitude ratio of adjacent layers, as shown in Figure 6. We maintain
COQA
GSMSK
TruthfulQA
0.603
0.108
29.813
0.01
0.620
0.126
30.226
all settings constant, starting from layer S 16 (halfway
0.02
0.630
0.143
33.903
0.05
0.647
0.152
33.213
0.1
0.643
0.152
33,903
0.643
0.159
33.743

---

through the layers of LLaMA-3-8B). and vary the interpretation parameter t from 0.3 to 0.7. Our findings reveal several key points. When 0.5, the process resembles average merging, which is less effective for cross-layer merging. In contrast, when / = 0.6 is optimal, the merged representation exhibits the most robust performance, while indicating that more information derived from the second term (æ') of the SLERP.
The frequency results also indicate that the high frequencies are clustered around 0.4 and 0.6, corroborating our optimal f. Moreover, there is a strong correlation between the optimal & and the high frequency the relative magnitude ratio of adjacent layers. This finding provides an opportunity to utilize the relative magnitude ratio to dynamically determine the interpretation parameter t. Dynamic t allows for more flexible weight control in SLERP merging for each layer-wise operation, thereby showing potential for further exploration.
The effect of token retention threshold y. We investigate the impact of the token retention threshold on model performance across the three datasets, as shown in Table A larger t generally means retaining more tokens for improved performance, but this comes the cost of increased memory demand. The results suggest that setting to 0.05 achieves the best balance between performance and efficiency.
Conclusion and Future Work
This paper presents a pioneering exploration of KV cache compression in the depth dimension, addressing significant memory bottleneck in LLMs. Our proposed MiniCache offers simple, effective, and training-free approach compressing KV caches by leveraging the notable high similarities between KV caches in neighboring layers, starting from the midpoint of LLMss. We have demonstrated that MiniCache can significantly reduce the memory footprint required for LLM inference by up to 41% %,, while simultaneously enhancing throughput by approximately five times compared to the FP16 baseline. In conclusion, MiniCache significantly advances the field KV cache compression, offering state-of-the-au balance between efficiency and performance. Future work will focus on enhancing the compression ratio by woos-mulliple-layyr merging, developing advanced merging algorithms such as Spherical Cubic Interpolation [78], and further optimizing memory usage for large-scale deployments in diverse application scenarios.
8
Acknowledgement
This research is partially supported by the ARC Future Fellowship (FT190100039) to G.H. Additional support was partially provided by the DARPA Assured Neuro-Symbolic Learning and Reasoning (ANSR) program under award number FA8750-23-2-1016.
References
[1] T. Brown, B. Mann, N. Ryder, M. Subbiah, J. D. Kaplan, P. Dhariwal, A. Neelakantan, P. P.Shyam, G. Sastry, A. Askell, et al., "Language models are few-shot learners," " in NeurIPS, vol. 33, pp. 1877-1901, 2020.
[2] OpenAI, J. Achiam, S. Adler, S. Agarwal, L. Ahmad, I. Akkaya, F. L. Aleman, D. Almeida, J. Altenschmidt, S. Altman, and OTHERS, "Gpt-4 technical report," 2023.
[3] L. Ouyang, J. Wu, X. Jiang, D. Almeida, C. Wainwright, P. Mishkin, C. Zhang, S. Agarwal, K. Slama, A. Ray, et al., "Training language models to follow instructions with human feedback," in NeurIPS, vol. 35, pp. 27730-27744, 2022.
[4] H. Touvron, T. Lavril, G. Izacard, X. Martinet, M.-A. Lachaux, T. Lacroix, B. Rozière, N. Goyal, E. Hambro, F. Azhar, et al., "Llama: Open and efficient foundation language models," arXiv preprint 2rXiv: 2023.
[5] H. Touvron, L. Martin, K. Stone, P. Albert, A. Almahairi, Y. Babaei, N. Bashlykov, S. Batra, Bhargava, S. Bhosale, el al., "I.lama 2: Open foundation and fine-tuned chat models," arXiv preprint arXiv:2307.092888 2023.
10

---

[6] "Introducing meta Ilama 3: The most capable openly available Ilm to date." httpthhttttttaneee meta. hom/blog/meen aatlo/meta-llmma3333 1ataaa-laaaa---- 3/, 2024. Accessed: 2024-05-04.
[7] Kaplan, S. McCandlish, T. Henighan, T. Brown, B. Chess, R. Child, S. Gray, A. Rad- ford, J. Wu, and D. Amodei, "Scaling laws for neural language models," arXiv preprint arXiv:2001.08360 2020.
[8] G. Team, R. Anil, S. Borgeaud, Y. Wu, J.-B. Alayrac, Yu, Soricut, J. Schalkwyk, A. M. Dai, A. Hauth, et al., "Gemini: a family of highly capable multimodal models," arXiv preprint
arXiv:2312.11880
2023.
[9] R. Pope, S. Douglas, A. Chowdhery, J. Devlin, J. Bradbury, J. Heek, K. Xiao, S. Agrawal, and J. Dean, "Efficiently scaling transformer inference," Proceedings of Machine Learning and Systems, vol. 5, 2023.
[10] K. Cobbe, V. Kosaraju, M. Bavarian, M. Chen, H. Jun, L. Kaiser, M. Plappert, J. Tworek, J. Hilton, R. Nakano, el al., "Training verifiers to solve math word problems," arXiv preprint arXiv:21 10. 14168, 2021.
[11] Z. Liu, J. Yuan, H. Jin, S. Zhong, Z. Xu, V. Braverman, B. Chen, and X. Hu, "Kivi: Plug- and-play 2bit kv cache quantization with streaming asymmetric quantization," arXiv preprint arXiv:2402.02750 2024.
[12] H. Kang. Q. Zhang, S. Kundu, G. Jeong. Z. Liu, T. Krishna, and T. Zhao, "Gear: An efficient kv cache compression recipefor near-lossless generative inference of Ilm," arXiv preprint arXiv:2403.05527, 2024.
[13] Y. Sheng. L. Zheng, B. Yuan, Z. Li, M. Ryabinin, B. Chen, P. Liang, C. Ré, I. Stoica, and C. Zhang. "Flexgen: High-throughhut generative inference of large language models with a single gpu," in ICML, pp. 1109--31166. PMLR, 2023.
[14] Z. Zhang. Y. Sheng, T. Zhou, T. Chen, L. Zheng, R. Cai, Z. Song, Y. Tian, C. Ré, C. Barrett, al., "H2o: Heavy-hitter oracle for efficient generative inference of large language models," in NeurIPS, vol. 36, 2024.
[15] G. Xiao, Y. Tian, B. Chen, S. Han, and M. Lewis, "Efficient streaming language models with attention sinks," " in ICLR, 2024.
[16] S. Ge. Y. Zhang, Liu, M. Zhang, J. Han, and J. Gao, "Model tells you what to discard: Adaptive kv cache compression for llss," ICLR, 2024.
[17] D. Raposo. S. Ritter, B. Richards, T. Lillicrap, P. C. Humphreys, and A. Santoro, "Mixtuee-of- depths: Dynamically allocating compute transformer-bass language models," arXiv preprint arXiv:2404.02255 2024.
[18] W. Zhou, C. Xu, T. Ge, J. McAuley, K. Xu, and F. Wei, "Bert loses patience: Fast and robust inference with early exit," in NeuriPS, vol. 33, pp. 18330-18341, 2020.
[19] M. Elhoushi, A. Shrivastava, D. Liskovich, B. Hosmer, B. Wasti, L. Lai, A. Mahmoud, B. Acun, S. Agarwal, A. Roman, et al., "Layer skip: Enabling early exit inference and self-speculative decoding," arXiv preprint
[tfi:2404.1671000 2024.
[20] A. Gromov, K. Tirumala, H. Shapourian, P. Glorioso, and D. A. Roberts, "The unreasonable ineffectiveness of the deeper layers," arXiv preprini arXiv:2403. /7887, 2024.
[21] Salimans and D. P. Kingma, "Weight normalization: simple reparameterizzation to accelerate training of deep neural networks." in NeurIPS, vol. 29, 2016.
[22] A. Q. Jiang, A. Sablayrolles, A. Roux, A. Mensch, B. Savary, C. Bamford, D. S. Chaplot, D. d. I. Casas, E. B. Hanna, F. Bressand, et al., "Mixtral experts," arXiv preprint arXiv:2401.04088 2024.
[23] M. Abdin, S. A. Jacobs, A. A. Awan, J. Aneja, A. Awadallah, H. Awadalla, N. Bach, A. Bahree, A. Bakhtiari, H. Behl, el al., "Phi-3 technical report: A highly capable language model locally on your phone," arXiv preprint arXiv:2404.142111 2024.

---

[24] M. Roemmele, C. A. Bejan, and A. S. Gordon, "Choice of plausible alternatives: An evaluation of commonsense causal reasoning." " in AAAI spring symposium: logical formalizations of commonsense reasoning, pp. 90-95, 2011.
[25] A. Amini, S. Gabriel, S. Lin, R. Koncel-Kedziorski, Y. Choi, and H. Hajishirzi, "Mathqa: Towards interpretable math word problem solving with operation formalisms." in NAACL. pp. 2357-2367, 2019.
[26] T. Mihaylov, P. Clark, T. Khot, and A. Sabharwal, "Can suit of armor conduct electricity? new dataset for open book question answering," in EMNLP, 2018.
[27] Y. Bisk, R. Zellers, R. L. Bras, J. Gao, and Y. Choi, "Piqa: Reasoning about physical common- sense in natural language," " in AAAI, 2020.
[28] A. Wang, A. Singh, Michael, F. Hill, O. Levy, and S. R. Bowman, "Glue: multi- task benchmark and analysis platform for natural language understanding." arXiv preprint arXiv:1804.0744 074411, 2018.
[29] K. Sakaguchi, R. L. Bras, C. Bhhaavvttula, and Y. Choi, "Winogrande: An adversarial winograd schema challenge at scale," Communications of the ACM, vol. 64, no. 9, pp. 99-106, 2021.
[30] S. Narayan, S. B. Cohen, and M. Lapata, "Don't give me the details, just the summary! topic-aware convolutional neural networks for extreme summarization," arXiv preprint 1:Xiv:1808.08745 2018.
[31] R. Nallapati, B. Zhou, C. Gulcehre, B. Xiang, et al., "Abstractive text summarization using sequence-to-sequencc rnns and beyond," arXiv preprint arXiv: 160.v01602.00023 2016.
[32] L. Gao, J. Tow, B. Abbasi, S. Biderman, S. Black, A. DiPofi, C. Foster, L. Golding, J. Hsu, A. Le Noac' h, H. Li, K. McDonell, N. Muennighoff, C. Ociepa, J. Phang, L. Reynolds, H. Schoelkopf, A. Skowron, L. Sutawika, E. Tang, A. Thite, B. Wang, K. Wang, and A. Zou, "A framework for few-shot language model evaluation," 12 2023.
[33] Y. Bai, X. Lv, J. Zhang, H. Lyu, J. Tang, Z. Huang, Z. Du, X. Liu, A. Zeng, L. Hou, et al., "Longbench: A bilingual, multitask benchmark for long context understanding." arXiv preprint arXiv:2308. 14508, 2023.
[34] L. Del Corr,, A. Del Giorno, S. Agarwal, B. Yu, A. Awadallah, and S. Mukherjee, "Skipdecode: Autoregressive skip decoding with batching and caching for efficient IIm inference," arXiv preprint arXiv:2307.02622 2023.
[35] T. Schuster, A. Fisch, J. Gupta, M. Dehghami, D. Bahri, V. Tran, Y. Tay, and D. Metzler, "Confident adaptive language modeling," " in NeurlPS, vol. 35, pp. 17456-17472, 2022.
[36] H. Wu and K. Tu, 'Layer-condensed kv cache for efficient inference of large language models," 2024.
[37] J.-H. Kim, Yeom, Yun, and H. O. Song. "Compressed context memory for online language model interaction," arXiv preprint arXiv:2312.03414, 2023.
[38] A. Bulatov, Y. Kuratov, and M. Burtsev, "Recurrent memory transformer," Advances in Neural Information Processing Systems, vol. 35, pp. 11079-11091, 2022.
[39] W. Fedus, B. Zoph, and N. Shazeer, "Switch transformers: Scaling to trillion parameter models with simple and efficient sparsity," Journal of Machine Learming Research, vol. 23, no. 120, pp. 1 1-39, 2022.
[40] D. Lepikhin, H. Lee, Y. Xu, D. Chen, O. Firat, Y. Huang, M. Krikun, N. Shazeer, and Z. Chen, "Gshard: Scaling giant models with conditional computation and automatic sharding," " arXiv preprint arXiv:2006.16663 2020.
[41] D. Dai, C. Deng, C. Zhao, R. Xu, H. Gao, D. Chen, J. Li, W. Zeng, X. Yu, Y. Wu, el al., "Deepseekmoe: Towards ultimate expert specialization in mixture-of-experts language models." arXiv preprint 2xXi::2401.06066 2024.
12

---

[42] C. Hwang, W. Cui, Y. Xiong, Z. Yang. Z. Liu, H. Hu, Z. Wang, R. Salas, Jose, P. Ram, et al., "Tutel: Adaptive mixture-of-ex experesss at scale," Proceedings of Machine Learning and Systems, vol. 5, 2023.
[43] DeepSeck-AI, "Deepseek-v2. strong, economical, and efficient mixture-of-experts language model." arXiv preprint arXiv:2405. 04434, 2024.
[44] Ainslie, J. Lee-Thorp. M. de Jong. Y. Zemlyanskiy, F. Lebrón, and S. Sanghai, "Gqa: Training generalized multi-query transformer models from multi-head checkpoints," arXiv preprint orXiv22305.13245 2023.
[45] N. Shazeer, "Fast transformer decoding: One write-head is all you need," arXiv preprint 1rX1v:1911.02150, 2019.
[46] K. Choromanski, V. Likhosherstov, D. Dohan, X. Song, A. Gane, T. Sarlos, P. Hawkins, J. Davis. A. Mohiuddin, L. Kaiser, et al., "Rethinking attention with performers," arXiv preprint arXiv:2009. 14794, 2020.
[47] H. Peng, N. Pappas, D. Yogatama, R. Schwartz, N. A. Smith, and L. Kong, "Random feature attention," " in ICLR, 2021.
[48] T. Dao, D. Fu, S. Ermon, A. Rudra, and C. Ré, "Flashattention: Fast and memory-efficient exact attention with io-awareness," in NeurIPS, vol. 35, pp. 16344-16359, 2022.
[49] T. Dao, "Flashattention 2: Faster attention with better parallelism and work partitioning," arXiv preprint arXiv:2307.08699, 2023.
[50] S. Wang, B. Z. Li, M. Khabsa, H. Fang, and H. Ma, "Linformer: Self-attention with linear complexity," arXiv preprint arXiv:2006. 047668 2020.
[51] X. Ma, X. Kong, S. Wang, C. Zhou, J. May, H. Ma, and L. Zettlemoyer, "Luna: Linear unified nested attention," in NeurIPS, vol. 34, pp. 2441-2453, 2021.
[52] J. Lee, Y. Lee, J. Kim, A. Kosiorek, S. Choi, and W. Teh, "Set transformer: A framework for attention-basee based permutation-invarian neural networks," in ICML, pp. 3753, PMLR, 2019.
[53] Lin, J. Tang, H. Tang, S. Yang, X. Dang, and Han, "Awq: Activation-aware weight quantization for Ilm compression and acceleration," arXiv preprint arXiv:2306.00978 2023.
[54] T. Dettmers, A. Pagnoni, A. Holtzman, and L. Zettlemoyer, "Qlora: Efficient finetuning of quantized Ilms," in NeurIPS, vol. 36, 2023.
[55] Dettmers, Lewis, Y. Belkada, and Zettlemoyer: intm:: 8-bit matrix multiplication for transformers at scale," in NeurIPS, vol. 35, pp.
2022.
[56] Liu, R. Gong, X. Wei, Z. Dong, J. Cai, and B. Zhuang, "Qllm: Accurate and efficient low-bitwidth quantization for large language models," in ICLR, 2024.
[57] M. Zhang, C. Shen, Z. Yang, L. Ou, X. Yu, B. Zhuang, et al., "Loraprune: Pruning meets low-rank parameter-efffcient fine-tuning," in ACL 2024.
[58] N. Kitaev, F.. Kaiser, and A. Levskaya, "Reformer: The efficient transformer." arXiv preprint arXiv:2001.04451, 2020.
[59] S. Bac, J. Ko. H. Song, and S.-Y. Yun, "Fast and robust carly-exiting framework for autoregres- sive language models with synchronized parallel decoding," arXiv preprint arXiv:23 10.0552,, 2023.
[60] K. Ainsworth, J. Hayase, and S. Srinivasa, re-bsin: Merging models modulo permutation symmetries," in ICLR, 2023.
[61] R. Entezari, Sedghi, O. Saukh, and B. Neyshabur, "The role of permutation invariance in linear mode connectivity of neural networks," arXiv preprint
2021.
13

---

[62] T. Garipov, P. Izmailov, D. Podoprikhin, D. P. Vetrov, and A. G. Wilson, "Loss surfaces, mode connectivity, and fast ensembling of dnns." in NeurIPS, vol. 31, 2018.
[63] M. Wortsman, G. Ilharco,
S. Y. Gadre, R. Roelofs, R. Gontijo-Lopes, A. S. Morcos, H. Namkoong, A. Farhadi, Y. Carmon, S. Korbblith, et al., "Model soups: averaging weights of multiple fine-tumed models improves accuracy without increasing inference time," in ICML, pp. 23965-239992 PMLR, 2022.
[64] Yadav, D. Tam, L. Choshen, C. Raffel, and M. Bansal, "Ties- mier-ig Resolving interfer- ence when merging models," in NeurlPS, vol. 36, 2023.
[65] M. Davari and E. Belilovsky, "Model breadcrumbs: Scaling multi-task model merging with sparse masks," arXiv preprint arXiv:2312.0679, 2023.
[66] L. Yu, B. Yu, H. Yu, F. Huang, and Y. Li, "Language models are super mario: Absorbing abilities from homologous models as free lunch," arXiv preprint arXiv:2311.0.33 2023.
[67] K. Shoemake, "Animating rotation with quaternion curves," in Proceedings of the 12th annual conference Computer graphics and interactive techniques, pp. 245-254, 1985.
[68] M. S. Matena and A. Raffel, "Merging models with fishht- averaging." " in NeurIPS, vol. 35, pp. 17703-17716, 2022.
[69] X. Jin, X. Ren, D. Preotiuc-Piees -Peetro, and P. Cheng, "Dataless knowledge fusion by merging weights language models," arXiv preprint arXiv:2212.09849, 2022.
[70] Y. Chen, X. Pan, Y. Li, B. Ding, and J. Zhou, "Ee-llm: Large-scale training and inference of early-exit large language models with 3d parallelism," arXiv preprint arXiv:2312.06 04916, 2023.
[71] S. Reddy, D. Chen, and C. D. Manning, "Coqa: A conversational question answering challenge." Transactions of the Association for Computational Linguistics, vol. 7, pp. 249-266, 2019.
[72] S. Lin, Hilton, and O. Evans, "Truthfulqa: Mcasuring how models mimic human falsehoods," arXiv preprint arXiv:2109.0799 2021.
[73] G. Xiao, J. Lin, M. Seznec, H. Wu. J. Demouth, and S. Han, "Smooothuant: Accurate and efficient post-training quantization for large language models," in International Conference on Machine Learning, pp. 38087-38099, PMLR, 2023.
[74] Liu, R. Gong, X. Wei, Z. Dong, J. Cai, and B. Zhuang, "QIlm: Accurate and efficient low-bitwidth quantization for large language models," in ICLR, 2024.
[75] S.-Y. Liu, C.-Y. Wang. H. Yin, P. Molchanov, Y.-.. F. Wang, K.-T. Cheng, and M.-H. Chen, low-rank adaptation," " arXiv preprint arXiv:2402.09333, 2024.
"Dora:
[76] M. Nagel, R. A. Amjad, M. Van Baalen, C. Louizos, and T. Blankevoort, "Up or down? adaptive rounding for post-training quantization." " in International Conference on Machine Learning, pp. 7197-7206, PMLR, 2020.
[77] W. Kwon, Z. Li. S. Zhuang, Y. Sheng, L. Zheng, C. H. Yu, J. Gonzalez, H. Zhang, and I. Stoica, "Efficient memory management for large language model serving with pagedattention." in Proceedings of the 29th Symposium Operating Systems Principles, pp. 61 611-626, 2023.
[78] D. H. Eberly, "Quaternion algebra and calculus," 2002.
[79] G. Xiao, Y. Tian, B. Chen, S. Han, and Lewis, "Efficient streaming language models with attention sinks," arXiv preprint arXiv:2309. 17453, 2023.
[80] "Stanford erfm." http::///ccttsssttoooooeeee/e// http://saaaaataeaa/////////////a 2024. Accessed: 2024-05-04.
html,
[81] Y. Liu, H., K. Du, J. Yao, Y. Cheng, Y. Huang, S. Lu, M. Maire, H. Hoffmann, A. Holtzman, et al., "Cachegen: Fast context loading for language model applications," arXiv preprint arXiv:2310.07240, 2023.
14

---

[82] L. Chen, H. Zhao, T. Liu, S. Bai, J. Lin, C. Zhou, and B. Chang, "An image is worth 1/2 tokens after layer 2: Plug-and-pla inference acceleration for large vision-language models," arXiv preprini MxXiv:2403006760 2024.
[83] S. Wei, T. Ye, S. Zhang, Y. Tang, and J. Liang, "Joint token pruning and squeezing towards more aggressive compression vision transformers," in CVPR. pp. 2092-2101, 2023.
[84] L. Gao, J. Tow, S. Biderman, S. Black, A. DiPoñi, C. Foster, L.. Golding, J. Hsu, K. McDonell, N. Muennighoff, J. Phang, L. Reynolds, E. Tang, A. Thite, B. Wang, K. Wang, and A. Zou, "A framework for few-shot language model evaluation," 2021.
[85] W. Kwon, Z. Li. S. Zhuang, Y. Sheng, L. Zheng, C. H. Yu, J. Gonzalez, H. Zhang, and I. Stoica, "Efficient memory management for large language model serving with pagedattention." in Proceedings of the 29th Symposium on Operating Systems Principles, pp. 61 611-626, 2023.
15

---

Appendix
A Additional Experiment Results
Comparisons with token sparsity methods. We also compare MiniCache with the sparsity- method H2O [14]. Our method outperforms H2O in most tasks on LongBench, Notably, our approach focuses on reducing inter-layer redundancy, whereas H2O focuses on intra-layer redundancy. Our approaches are orthogonal to sparsity- sparsity-based methods.
Table A: Comparison between MiniCache with token sparsity based method H2O, using mistral-7B- instruct LongBench dataset.
Metheds
NrvQ4
MFF-n
HorpoQA
2WILMOAA
Misigue
Car-eepert
QMSum
MuniNNox
TW3C
TriveRQ
SAMSum
PCou
FRr
Lo:
Budiie
26.82
53.05
49.28
42.75
27.33
19:27
32.85
24:25
27.06
71.0
$6.23
42.08
2.55
85.05
55.51
(200114]
29.05
7M)
Minitacher
22.04
$9.99
4191
28.0
11:46
76.6
71.0
96.91
79.56
Execution time comparison for difference sequence length. We benchmark the latency of LlaMA- 2-7B on an NVIDIA A 100 GPU using different sequence lengths ranging from 1024 to 4096 with batch size of 16, shown in table B. We compare with H2O, which requires calculating full attention scores to estimate token importance. In contrast, MiniCache performs reparameterization: based merging and token restoration using simple matrix manipulations, resulting in more lightweight computations and lower latency. Specifically, when the sequence length 4096, MiniCache shows 36.83% reduction latency compared H2O.
Table Execution time comparison for different sequence lengths. We benchmarked the latency of LLaMA-2-7B on an NVIDIA A100 GPU using sequence lengths ranging from 1024 to 4096 with batch size 16.
1024
2048
3072
4096
H2O
9083.08 ms
19,274.46 ms
26,508.00 ms
45,850.708 ms
MiniCache
8605.53 ms
14,843.40 ms
23,111.93 ms
33,414.56 ms
Trade-off between accuracy and efficiency. We conducted experiments by varying the ratio of retention tokens, shown in table C. We observe that accuracy improves up to a certain point before plateauing as the token retention ratio increases. Retaining 20% the tokens necessary to ensure all salient tokens preserved without performance degradation. In contrast, we only need to retain the top 5% the salient tokens using a dynamic distance-based threshold, as proposed in our paper. Note that efficiency decreases more tokens retained. This demonstrates that our distance-based approach better balances performance and efficiency than the fixed retention ratio counterpart.
Ablation for relations between performance and ratio of merged tokens. We are adding another column in terms compression ratio to represent the efficiency trade-offs, as shown table D. The results suggest that setting r to 0.05 achieves the best balance between performance and efficiency.
Table C:
Performance comparison of LLaMA- 2-7B under different retention ratios overall tokens.
Table D: Performance and compression ratios of LLaMA-2-7B at various retention thresh-
5% token
COQA
GSM8K
TruthfulQA
olds 7 in Eq. (4).
COQA
GSMBK
TrathfulQA
Compression Ratio
0.603
0.108
29.813
0.603
0.108
29.813
5.191
0.01
0.607
0.114
29.997
30,226
5.156
0.05
0.614
0.137
30.174
0.02
0.630
0.143
31.903
5.122
0.1
0.625
0.139
32.661
5,023
0.1
0.643
0.152
33.903
4,866
0.2
0.641
0.152
33.174
3.115
0.643
0.159
33.743
16

---

Benchmark of the computational overhead of different components. As shown in Table table E, the reparametrization process, including the computation of magnitude and direction, takes 0.093 ms (0.031 ms + 0.062 ms), and the restoration process, including the computation of the distance matrix and token replacement, takes 0.116 ms (0.061 ms + 0.055 ms). These times indicate a negligible computational overhead compared to the overall attention computation, which takes 9.756 ms. The running time is measured in milliseconds per attention layer with a batch size of on LLaMA-2-7B.
Table E: We benchmark the different components in the reparameterization and restoration stages using LLaMA-2-7B.
Component
Running Time (ms)
Standard Variation (ms)
Magnitude
0.031
0.007
Direction
0.062
0.013
Distance
0.061
0.022
Token Replacement
0.055
0.010
Attention (Overall)
9.756
0.513
Efficiency comparison without quantization. table F demonstrates that MiniCache's cross-layer merging without quantization achieves almost lossless performance compared to the FP16 haseline, with compression ratio of 1.53. In contrast, 2-bit quantization causes performance degradation. For instance, on the GSM8K dataset, performance drops from 0.159 to 0.127, but the compression ratio improves 3.95. Our MiniCache method, focusing on the depth dimension, can complement any quantization and existing KV cache compression methods. The results indicate that combining cross-layer merging with 4-bit quantization achieves the optimal balance between performance and efficiency.
Table F: Performance comparison and compression ratios across different methods. MiniCache is orthogonal to existing quantization techniques.
Method
COQA
GSMSK
TruthfulQA
Compression Ratio
FP16
0.643
0.159
33.743
1
KIVI-2
0.631
0.127
33.950
3.95
KIVI-4
0.638
0.154
33.864
3.29
Cross-layer Merging
0.644
0.159
33.713
1.53
Cross-layer Merging
0.647
0.152
33.213
5.023
4-bit quantization
B Additional Related Work
Preliminaries of KV cache in LLMs. LLM inference has been significantly improved in recent works [9, 13, 77] by optimizing the KV cache management. Overall, this line of research is typically done in two steps: First, at the prefilling stage, LLM calculates the initial KV caches at each attention layer based on the input sequence and decodes the first token. Second, 2) at the decoding stage, LLM autoregressively predicts the next token, where KV cache at each attention layer added to the overall caches. Existing works have compressed KV cache in different aspects (e.g.. quantization [11, 12], token pruning [14, 16]).
KV cache compression. In the prior study, various strategies for enhancing efficient transformer architectures are discussed, covering a spectrum of techniques aimed optimizing performance and managing resource constraints. These methods include attention optimization [48, 49, 80], grouping queries [44, 45], sparse KV caching [16, 81, 82]. shrinking tokens [15, 83), and improving long-context generation. Significant contributions come from projects such as H2O [15], GEAR [15], and KIVI Additionally, efforts alleviate KV cache bottlenecks include strategies like multi-query attention [44] and multi-group attention [45], which propose reducing the number of heads in the KV cache. However, these methods often necessitate retraining or fine-tuning models. Other approaches focus diminishing the size of the KV cache by selectively evicting less important
17

---

tokens [14] and enhancing the system architecture through technologies like offloading the KV cache [84] or integrating techniques such as virtual memory and paging [85] into the attention mechanism.
C Discussions and Limitations
Table G: Comparison of performance using different cross-layer merging strategies. The experiment shows that SLERP has the best performance across three datasets.
Strategy
COQA
GSM8K
TruthfulQA
Average
0.541
0.081
28.162
Max Norm
0.607
0.104
30.150
SLERP
0.647
0.152
33.213
Alternative merging function. During our preliminary exploration, we initially considered an alter- native, simpler merge function for cross-layer compression: maximum norm-preservie interpolation. This function is designed to maintain the maximum norm of the vectors involved, ensuring that the most significant features are preserved during the merging process. The maximum norm-preeerving interpolation in terms of Fax can be defined as follows:
F (x', x1-1)=
Max [|<2"| [|))
(A)
Here -!.-1 represents the average vector between x' and -/ The function Fmax ensures that the merged vector preserves the direction of the average vector while scaling it to the maximum norm of SLERP-based merge function, Fmax has less computational
original KV states. Compared overhead and lower memory consumption. However, it is less accurate than SLERP. The choice between using FSLERP or Fmax depends on the specific requirements the application. In our study, primarily use SLERP maximize performance, also shown in Table G.
Societal impact. Our work shows preliminary exploration of KV cache Compression in the depth dimension, relatively unexplored yet critically bottlenecked area in large language models (LLMs). The proposed MiniCache provides solution to improve the efficiency of LLM generation and is adaptable to existing intra-layer-basee based KV cache pruning and quantization technologies. Additionally, we proposed a simple yet effective approach that merges similar KV cache states in a cross-layer manner and effectively restores performance through a novel restoration technique. Our observations indicate that cross-layer similarity and mergeability can be applied in broad-efficient applications for post-training optimization in low-resource scenarios, such deployment on mobile devices. Moreover, with effective optimization of KV cache memory demand, MiniCache further enhances long-context generation, which is crucial paradigm for real-world applications, such as understanding concepts in textbooks. We aim for our work to advance the boundaries of two key challenges in the LLM industry and research: batch inference and long-context generation.
Furthermore, addition the significant contributions of MiniCache for KV cache Compression, several challenges remain that are common to LLMs. Issues such as the truthfulness and security of LLMS are still unresolved. Ensuring the accuracy and reliability of generated content is critical, as L.MSs can sometimes produce plausible but incorrect or misleading information. Additionally, safeguarding against security vulncrabilities, such as adversarial attacks or data leakage, is paramount to maintaining the integrity and confidentiality of user interactions. Addressing these challenges requires ongoing research and development to enhance the robustness and trustworthiness of LLMs. This effort must proceed alongside advancements in computational efficiency and performance, as exemplified by innovations like MiniCache.
Limitations. The current merging algorithm based on Spherical Linear Interpolation (SLERP) has its limitations. SLERP suitable only for merging two vectors and uses an interpolation approach that restricts our algorithm from merging multiple layers simultaneously and maximizing the compression ratio further states. This limitation impacts the overall efficiency of KV cache compression and underscores the need for advanced techniques capable of handling more complex merging scenarios.
18

---

Future research should focus on developing more sophisticated algorithms that can overcome these constraints, thereby enhancing the compression capabilities and overall performance of LLMs.
D Additional Implementation Details
Overview of the inference algorithm. The MiniCache inference implementation, as shown in Alg. involves several key steps. When the interface starts, In the prefilling phase, we define the merging starting layer S. Before reaching layer S, the inference uses the original attention and cache logic. From layer S onward, we implement our merging algorithm, which operates in cross-layer manner and is applied only at odd-numbered layers. During merging, we fetch the KV cache from the previous layer and save the merged shared states into the current layer's KV cache. To reduce memory usage, we then remove the cache of the previous layer. Additionally, we store the magnitudes vector and iitention-sensiivve tokens for each layer. During the decoding phase, two scenarios arise after layer S. For even-numt numbbedeeeedd layers (first round), since the KV cache has been removed during the prefilling phase, we refer to the next layer (1 1) to fetch the shared KV cache states. We then perform approximated scale restoration and retention token recovery. The new KV states from this phase are stored for use the next round. In the second round, which involves odd-numbered layers, we use the new KV tokens from both the previous and current layers. After the restoration phase, we perform the merge operations and update the shared KV cache states in the stack.
Cross- merging and restoration algorithm. As outlined in Alg. 2, MiniCache algorithm involves several crucial steps to ensure efficient memory usage while maintaining the integrity of the Key-Value (KV) Cache states. Initially, given the KV cache E norm values || &, unmerged
tokens R k. retention indices Tk,,, and the next tokens t', t'-1, the algorithm proceeds by rescaling the magnitude of the KV pairs. Specifically, X and X! are computed by multiplying the normalized KV pairs E with their respective magnitude norms | Following this, the algorithm restores unmerged tokens using the retention indices, updating X and X accordingly. Next, the new tokens tx and tu are concatenated to the rescaled KV pairs along the token dimension. This augmented KV cache undergoes a softmax attention mechanism where the attention scores A are computed by taking the dot product of the query token b, with the transposed keys (X()). The output token to is then obtained by multiplying the attention scores A with the values X!. In cases where the previous token t!-1 exists, the algorithm performs a compression step. It concatenates the existing KV cache E/,e with the merged tokens resulting from the current and previous layers, effectively reducing redundancy and optimizing memory. If t'-1 is not available, the KV cache is updated by simply concatenating FL., with the new tokens t deferring the compression until the next iteration. The final output token to is then returned, concluding the decoding process. In the merging function, the algorithm normalizes the KV pairs from both the current and previous layers. It calculates the angular distance Ω between the normalized vectors, ensuring that the interpolation occurs along the shortest path on the unit sphere. The merged KV cache is then obtained by weighted interpolation of the normalized vectors, preserving the geometric and semantic integrity of the original states. This comprehensive process allows MiniCache to achieve substantial memory efficiencies while maintaining the functional characteristics of the KV pairs across transformer layers.
logic for the MiniCache framework, which incorporates cross-layer merging and error suppression to achieve memory effi- ciency and maintain functional integrity. Initially, in Step 1, the KV cache is fetched from the previous layer (Layer I-- 1) during the pre-filling phase. In Step 2, the fetched KV pairs from the current layer x" are merged with the KV pairs from the preceding layer x reducing redundancy through a merge function. Subsequently, in Step 3, the merged KV pairs are cached for future use, representing a consolidated data set from multiple layers. During the decoding phase. Step 4 involves deleting unnecessary or redundant KV pairs to optimize memory usage. In Step 5, the decoder fetches the required KV pairs from the cache for output generation. Step 6 applies error suppression mechanisms to the fetched KV pairs, including rescaling and retention recovery, to minimize errors introduced during the merging and compression processes. Finally, in Step 7, the cache is updated with the final KV pairs post-error suppression and adjustments, ensuring the cache contains the most accurate and efficient representation of the KV pairs for subsequent layers. This comprehensive approach guarantees substantial memory efficiencies while preserving the critical functional characteristics of the original KV pairs across transformer layers.
MiniCache execution flow. Figure A delineates the pre- pre-filling and decoding
19

---

KV cache
compression layer I
K, VII
Keep
x
2
Merge
Cross-Layer Merging
Prefilling
Fetch
Layer 1
Layer
Time
Delete
3
Cache
Decoding
5
Fetch
Fetch
6
K
Rescale
Recovery
Retention Recovery +
Attention
Contact New Token +
Attention
Recovery
Cross Layer Merging
Figure A: Overall prefilling and decoding logic for MiniCache involves performing cross-layer
merging and recovery within framework.
E Detailed Efficiency Derivation
In this section, we provide a detailed derivation of the memory efficiency improvements outlined in Section 4.3.
First, we consider the original KV cache memory usage, which is given by:
4brh(s + m.)
Here, r is the number of layers, b is the batch size, h is the hidden size, is the input sequence length, and n is the output sequence length. To improve efficiency, we begin merging layers starting from the midpoint, S = fr. by consolidating the KV cache states of every two layers into a single shared state space. The memory usage derivation proceeds as follows: for the unmerged part of the cache (from layer to S):
20

---

Algorithm 1: The MiniCache Inference Algorithm
procedure MiniCache Inference:
Input: Input Tokens: T fi Xd number of layers L, merging beginning layer S
Output: Output Tokens: O E Rauua Xd
for < to S do
procedure Standard Prefill:
Standard Attention & Standard Cache
procedure Standard Decoding:
Standard Attention & Standard Cache
// Start Merging from layer 5 for < S to do
// Perform Merging in every two layers 1%2 = 1
if 1%2 == 1 and prefilling then
procedure MiniCache Prefill:
Input: KV cache from Current layer X k, KV cache from Previous layer
1- 1: X k token retention threshold:
10
Delete KV cache of the I 1-th layer // layer 1, 1 shares one Cache
Standard Attention & Standard Cache
// Perform Merging in the second layer if and decoding then if 1%2 == 0 then
12
13
// first round in the cross-layer merging, fetch shared
cache states from Cl+1
14
procedure MiniCache Decoding:
Input: KV cache: k2v Norm: || k.ull. Unmerged Tokens: R
Retention indices: 44,,, Next Token: t
15
else
// second round in cross-layer merging, while fl-1 exist procedure Minie Decoding:
16
Input: KV cache: C Norm: | Unmerged Tokens: R Retention
indices: 1kk,, Next Token: t/1
return O
4brh(s + n). 2brh(s + n)).
For the merged part of the cache (from layer + r):
4brh(s n) 1 brh(s + n).
Combining these two parts, the total memory usage is:
2brh(s + n) + brh(s (s + n) = 3brh(s + n2).
Next, consider the additional memory cost incurred during the restoration process. During this phase, we save additional normalized vectors for the layers in the KV cache. These vectors are of shape Roxex1, which means they have single channel dimension compared to the fully ranked original KV states.
The additional normalized vectors for layers from S onwards are given by:
21

---

Algorithm 2: The MiniCache Prefill & Decoding Compression Algorithm
Hyperparameter: number of layers I., merging beginning layer procedure MiniCache Prefill:
Input: KV cache from current layer I: X E 1p xd KV cache from previous layer
1: X R2x (xxppxx xd token retention threshold:
E
X
I
X
11, S4 & Merge( X X {);
E
X
Ω2 <- Merge X X x)=);;
E
R. fromn
1xI-111 11 IR. 1pxmppp
compression outpui
for rescaling
x kk,0 = S4, // distance metrics
TR. = (i d3
< dmin + (dmax dmin) }} // retention indices
R
X
1 [ R
R
// unmerged tokens
return E
1x(111 111-1 R I
R
2,0
procedure MiniCachi Decoding:
Input: KV cache: E E 2 xd Norm: |X k.u E R2 Iuan x Unmerged Tokens:
R2 pr xd Retention indices: llg. p
Next Token:
t. R Rlxd 11-1 Rixd
10
X E |
X E
[ E
// magnitude rescale
E
X [ R X Rd,
token restoration
X f Concat X tk. dim=token) X + Concat(X,, t,, dim=token)
12
Softmax(t,, (x() to A X iitt exists then
KV cache Concat( E Merge (t) ( dim=token) // perform compression
14
else
KV cache
Concat( E t/ dim=token) // wait for compression
return to
function MiniCache Merge X X1-1
17
X
18
X
x
X!-1
X x
Ω arccos
||*|+11| x||
E sin((1 (()(() X
(12
X 1-1
sin((()
(1)
22
return E, || 11,1, X Ω
br(s + n)) 2 2br(s + n.).
We also introduce a retention threshold, which we set to 0.05. This means that 5% of the KV cache tokens are retained without compression:
br (0.00. + n)).
Combining these terms, the total additional memory for the restoration process is:
2br(s n) + 0.1brh(s + n).
Finally, summing the compressed memory usage and the restoration memory cost, the overall memory requirement is:
22

---

3brh(s + n) 2br(s + n) + 0.1brh(s n).
This can be simplified by grouping the common factors:
(r n) (3h + 2 0.1h).
Simplifying the expression inside the parentheses, we get:
br(s n) (3.1h + 2).
Therefore, the total memory cost for the KV cache in the MiniCache Framework is:
br(s + n3 2).
This detailed derivation confirms the efficiency improvements discussed in Section 4.3, highlighting the significant reduction in memory usage achieved through our layer merging and restoration strategies.
F
Detailed Experiment Results
Table H: Detailed performance comparison on GSM8K dataset with LLaMA-3-70B.
Layer
LLaMA-3-70B MiniCache
LLaMA-3-70B Mean
0
0.799
0.729
5
0,800
0.640
10
0.792
0,578
15
0,815
0.545
20
0,801
0.560
25
0.812
0.544
30
0.799
0.556
35
0,810
0.557
40
0.790
0.551
45
0.725
0.539
50
0.638
0.541
55
0,638
0.501
60
0,625
0,497
65
0,635
0.511
70
0,623
0.497
75
0.615
0.493
23

---

Table I: Detailed performance comparison on COQA dataset with LLaMA-3-70B.
Layer
LLaMA-3-70B MiniCache
LLaMA-3-70B Mean
0
0.705
0.706
4
0.705
0.699
8
0.706
0.696
12
0.704
0.691
16
0.704
0.690
20
0.703
0.690
24
0.701
0.690
28
0.702
0,690
32
0.702
0.688
36
0.703
0.688
40
0.697
0.687
44
0.698
0,685
48
0,699
0.678
52
0.699
0.672
56
0.701
0.668
60
0.704
0.657
64
0.706
0.635
68
0.691
0.61
0.689
0.565
76
0.641
0.526
Table J: Detailed performance comparison on TruthfulQA dataset with LLaMA-3-70B,
Layer
LLaMA-3-70B MiniCache
LLaMA-3-70B Mean
0
22.615
23.130
4
22.512
22.005
8
22.451
21.876
12
22.413
21.303
16
22.387
21.209
20
22.387
20.752
24
22.387
20.657
28
22.276
20.501
32
22.130
20.479
36
22.130
20.335
40
22.073
19,834
44
21.356
17.024
48
21.356
12.440
52
21.333
9.127
56
21.316
3,255
60
21.172
2.349
64
21.153
2,250
68
21.002
1.721
72
20.940
1.119
76
20.683
0.784
24

---

Table K: Detailed performance comparison on GSM8K dataset with LLaMA-3-8B.
Layer
LLaMA-3-8B MiniCache
LLaMA-3-8B Mean
0,488
0.467
0.476
0.369
0,489
0.388
0.487
0.387
0.489
0.359
10
0.479
0.388
0,486
0.384
14
0.472
0.368
0.477
0.343
18
0,446
0.291
20
0,447
0.271
22
0,433
0.234
24
0.399
0.155
26
0.396
0.140
28
0.395
0,052
30
0.391
0.024
32
0.397
0.025
Table L: Detailed performance comparison on COQA dataset with LLaMA-3-8B.
Layer
LLaMA-3-8B MiniCache
LLaMA-3-8B Mean
0.676
0.676
0.676
0.571
0.675
0.566
0.674
0.564
0.674
0.561
10
0.673
0.560
0.672
0.560
14
0.670
0.559
0.670
0.558
18
0.669
0.555
20
0.669
0.552
22
0,668
0.549
24
0.667
0.543
26
0.667
0.537
28
0.666
0.536
30
0.666
0.531
32
0.665
0.528
25

---

Table M: Detailed performance comparison on TruthfulQA dataset with LLaMA-3-8B.
Layer
LLaMA-3-8B MiniCache
LLLMA-3-8B Mean
32.520
32.524
32.231
28.431
31.645
28.197
31,485
27.894
31.008
27.796
10
30.964
27.704
30.798
27.371
14
30.798
27.093
30.798
26.643
18
30.798
26.517
20
30.798
26.355
22
30.798
26.011
24
30.798
25.044
26
30.798
15.254
28
30.798
14.791
30
30.765
9.419
32
30.390
6.068
Table N: Detailed performance comparison on GSM8K dataset with Mixtral-8x7B.
Layer
Mixtral-8x7B MiniCache
Mixtral-8x7B Mean
0
0.589
0.575
2
0.592
0.480
0.593
0.491
6
0.591
0.469
8
0.580
0.472
10
0.592
0.492
12
0.582
0.485
14
0.572
0.480
16
0,562
0.462
18
0.522
0.432
20
0.526
0.426
22
0.540
0.416
24
0.519
0.398
26
0.515
0.436
28
0.502
0.401
30
0.515
0.386
32
0.490
0.258
26

---

Table O: Detailed performance comparison on COQA dataset with Mixtral-8x
Layer
Mixtral-8x7B MiniCache
Mixtral-8x7B Mean
0
0.672
0.675
2
0.671
0,612
0.670
0.601
6
0.672
0.590
0.674
0.582
10
0.671
0.571
12
0.674
0.561
14
0.670
0.546
16
0.672
0.544
18
0,672
0.530
20
0.675
0.522
22
0.671
0.512
24
0.660
0.455
26
0.657
0.447
28
0.640
0,440
30
0.634
0.424
32
0.459
0.430
Table P: Detailed performance comparison on TruthfulQA dataset with Mixtral-8x7B
Layer
Mixtral-8x7B MiniCache
Mixtral-8x7B Mean
0
21.686
19.465
2
21.385
19.405
21,368
19.251
6
21.038
19.094
8
21.038
18.265
10
20.216
17.019
12
20.026
15.902
14
19.723
15.505
16
19.641
15.028
18
19.641
14.723
20
19.546
14.543
22
18.756
14.122
24
18,402
13.834
26
18.366
13.789
28
17.738
12.091
30
16.827
12,008
32
16.635
0.430
27

---

Table Q: Detailed performance comparison on GSM8K dataset with Phi-3-Mini.
Layer
Phi-3-Mini MiniCache
Phi-3-Mini Mean
0
0.774
0.774
2
0.765
0.667
0.757
0.661
6
0.754
0.659
0.748
0.657
10
0.750
0.645
12
0.750
0.616
14
0.752
0.575
16
0.739
0.491
18
0.742
0,417
20
0.692
0,272
22
0.685
0.206
24
0.640
0.110
26
0.545
0.061
28
0,500
0.039
30
0.460
0.036
32
0.447
0.028
Table R: Detailed performance comparison on COQA dataset with Phi-3-Mini.
Layer
Phi-3-Mini MiniCache
Phi-3-Mini Mean
0
0.665
0.665
2
0,662
0.562
0.657
0.557
6
0,656
0,556
0.656
0.556
10
0.654
0.554
12
0.646
0.546
14
0.648
0.538
16
0.647
0.537
18
0.637
0.527
20
0.627
0.487
22
0.591
0.461
24
0,567
0.437
26
0.548
0.408
28
0.527
0.407
30
0.506
0.406
32
0.503
0.403
28

---

Table S: Detailed performance comparison on TruthfulQA dataset with Phi-3-Mini.
Layer
Phi-3-Mini MiniCache
Phi-3-Mini Mean
0
19.686
19,465
19.385
19.365
4
19.368
19.221
19.100
18.255
8
19.038
17.019
10
19,500
15.912
12
19.216
15.525
14
20.026
15.195
16
19.641
15.058
18
18.756
14.763
20
17.738
14.593
22
19.546
14.182
24
19.723
13.954
26
18.366
13.919
28
18.402
12.231
30
16.827
12.158
32
16.635
10.333
NeurIPS Paper Checklist
1. Claims
Question: Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?
Answer: [Yes]
Justification:
We have properly made the main claims in the abstract and introduction.
Guidelines:
The answer NA means that the abstract and introduction do not include the claims made in the paper.
The abstract and/or introduction should clearly state the claims made, including the contributions made in the paper and important assumptions and limitations. A No or
NA answer to this question will not perceived well by the reviewers.
The claims made should match theoretical and experimental results, and reflect how much the results can be expected to generalize to other settings.
It is fine to include aspirational goals as motivation as long as it is clear that these goals are not attained by the paper.
2. Limitations
Question:
Does the paper discuss the limitations of the work performed by the authors?
Answer: [Yes]
Justification: Yes, include the limitations in the Appendix.
Guidelines:
The answer NA means that the paper has no limitation while the answer No means that the paper has limitations, but those are not discussed in the paper.
The authors are encouraged to create a separate "Limitations" section their paper.
The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (e.g., independence assumptions, noiseless settings, model well well-specification, asymptotic approximations only holding locally). The authors should reflect on how these assumptions might be violated in practice and what the implications would be.
29

---

The authors should reflect on the scope of the claims made, e.g., if the approach was only tested on few datasets or with few runs. In general, empirical results often depend on implicit assumptions, which should be articulated.
The authors should reflect the factors that influence the performance of the approach. For example, facial recognition algorithm may perform poorly when image resolution is low images are taken in low lighting. Or speech-to-text system might not be used reliably to provide closed captions for online lectures because it fails to handle technical jargon.
The authors should discuss the computational efficiency of the proposed algorithms and how they scale with dataset size.
If applicable, the authors should discuss possible limitations of their approach to address problems of privacy and fairness.
While the authors might fear that complete honesty about limitations might be used by reviewers as grounds for rejection, worse outcome might be that reviewers discover limitations that aren't 't acknowledged in the paper. The authors should use their best judgment and recognize that individual actions in favor of transparency play an impor- tant role in developing norms that preserve the integrity of the community, Reviewers will be specifically instructed not penalize honesty concerning limitations.
3. Theory Assumptions and Proofs
Question: For each theoretical result, does the paper provide the full set of assumptions and a complete (and correct) proof?
Answer: [Yes]
Justification:
See sections in methodology and appendixes.
Guidelines:
The answer NA means that the paper does not include theoretical results.
All the theorems, formulas, and proofs in the paper should be numbered and cross- referenced.
All assumptions should be clearly stated or referenced the statement of any theorems.
The proofs can either appear in the main paper or the supplemental material, but they appear in the supplemental material, the authors encouraged provide short proof sketch to provide intuition.
Inversely, any informal proof provided in the core of the paper should be complemented by formal proofs provided in appendix or supplemental material.
Theorems and Lemmas that the proof relies upon should properly referenced.
4. Experimental Result Reproducibility
Does the paper fully disclose all the information needed to reproduce the main ex- perimental results the paper to the extent that affects the main claims and/or conclusions of the paper (regardless of whether the code and data are provided not)?
Question:
Answer: [Yes]
Justification: We provide detailed experiments setup on Experiment sections, we also have detailed algorithm appendix for Reproducibility.
Guidelines:
The answer NA means that the paper does not include experiments.
If the paper includes experiments, a No answer to this question will not be perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data provided or not.
If the contribution is dataset and/or model, the authors should describe the steps taken to make their results reproducible or verifiable.
Depending on the contribution, reproducibility can be accomplished in various ways. For example, if the contribution a novel architecture, describing the architecture fully might suffice, or if the contribution is specific model and empirical evaluation, it may be necessary to either make possible for others to replicate the model with the same datasct, provide access to the model. In general. releasing code and data is often
30

---

reproducible, they one good way to accomplish this, but reproducibility can also be provided via detailed instructions for how replicate the results, access a hosted model (e.g., in the case of large language model), releasing of a model checkpoint, or other means that are appropriate to the research performed.
While NeurIPS does not require releasing code, the conference does require submis- sions to provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution. For example
(a) If the contribution is primarily a new algorithm, the paper should make it clear how to reproduce that algorithm.
(b) If the contribution is primarily new model architecture, the paper should describe the architecture clearly and fully.
(c) If the contribution is a new model (e.g., a large language model), then there should either way to access this model for reproducing the results or way to reproduce the model (e.g., with an open-source dataset or instructions for how to construct the dataset).
(d) We recognize that reproducibility may be tricky in some cases, in which authors are welcome to describe the particular way they provide for reproducibility. In the case of closed-source models, it may be that access to the model is limited in some way (e.g., to registered users), but it should be possible for other researchers have some path reproducing or verifying the results.
5. Open access to data and code
Question: Does the paper provide open access to the data and code, with sufficient instruc- tions to faithfully reproduce the main experimental results, as described in supplemental material?
Answer: [Yes]
Justification:
We provide code after acceptance.
Guidelines:
The answer NA means that paper does not include experiments requiring code.
Please see the NeurlPS code and data submission guidelines (https://nips. co/
mublic/guides/ Stw.idde//Cdeeeumissiiiiiii
for more details.
While we encourage the release of code and data, we understand that this might not be possible, so "No" an acceptable answer. Papers cannot be rejected simply for not including code, unless this is central to the contribution (e.g., for new open-source benchmark).
The instructions should contain the exact command and environment needed to run to reproduce the results. See the NeurlPS code and data submission guidelines (https:
/ //wppn.cccc///////////o///oot/ct
httgg//ddd//oeuubbiiiii// /Cooe/ubeeeeeeeeeeeeee wwwsonoolll oniiooioo)y) for more details.
The authors should provide instructions on data access and preparation, including how to access the raw data, preprocessed data, intermediate data, and generated data, etc.
The authors should provide scripts to reproduce all experimental results for the new proposed method and baselines. only subset of experiments should state which ones are omitted from the script and why.
At submission time, to preserve anonymity, the authors should release anonymized versions (if applicable).
Providing as much information as possible in supplemental material (appended to the paper) recommended, but including URLs to data and code permitted.
6. Experimental Setting/Details
Question: Does the paper specify all the training and test details (e.g., data splits, hyper- parameters, how they were chosen, type of optimizer, etc.) necessary to understand the results?
Answer: [Yes]
Justification:
We provide implementation details.
Guidelines:
31

---

The answer NA means that the paper does not include experiments.
The experimental setting should be presented in the core of the paper to level of detail that is necessary to appreciate the results and make sense of them.
The full details can be provided either with the code, in appendix, or as supplemental material.
7. Experiment Statistical Significance
Question: Does the paper report error bars suitably and correctly defined other appropriate information about the statistical significance of the experiments?
Answer: [No]
We verify our results on huge size language models, repeat computation are expensive.
Justification:
Guidelines:
The answer NA means that the paper does not include experiments.
The authors should answer "Yes" if the results accompanied by error bars, confi- dence intervals, or statistical significance tests, at least for the experiments that support the main claims of the paper.
The factors of variability that the error bars are capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, or overall run with given experimental conditions).
The method for calculating the error bars should be explained (closed form formula, call to a library function, bootstrap, etc.)
The assumptions made should be given (e.g.. Normally distributed errors).
should be clear whether the error bar is the standard deviation or the standard error of the mean.
OK to report 1-sigma error bars, but one should state it. The authors should preferably report a 2-sigma error bar than state that they have 96% CI, if the hypothesis of Normality of errors is not verified.
For asymmetric distributions, the authors should be careful not to show in tables or figures symmetric error bars that would yield results that out of range (e.g. negative error rates).
If error bars are reported in tables or plots, The authors should explain in the text how they were calculated and reference the corresponding figures or tables in the text.
8. Experiments Compute Resources
For each experiment, does the paper provide sufficient information on the com- puter resources (type of compute workers, memory, time of execution) needed to reproduce the experiments?
Question:
Answer: [Yes]
Justification: We give the competing requirements in the experiment setup.
Guidelines:
The answer NA means that the paper does not include experiments.
The paper should indicate the type of compute workers CPU GPU, internal cluster, or cloud provider, including relevant memory and storage.
The paper should provide the amount of compute required for each of the individual experimental runs as well as estimate the total compute.
The paper should disclose whether the full research project required more compute than the experiments reported in the paper (e.g., preliminary failed experiments that didn't make into the paper).
9. Code Of Ethics
Question: Does the research conducted in the paper conform, in every respect, with the NeurlPS Code of Ethics https h/nnurrris.... www.riiss.ccpuuullieee htcppublic/tthicuuieeeee
Answer: [Yes]
32

---

Justification:
Yes we do.
Guidelines:
The answer NA means that the authors have not reviewed the NeurIPS Code of Ethics.
If the authors answer No, they should explain the special circumstances that require deviation from the Code of Ethics.
The authors should make sure to preserve anonymity (e.g.. there is a special consid- eration due to laws or regulations in their jurisdiction).
10. Broader Impacts
Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?
Answer: [Yes]
Justification:
We have included the broader impacts the Conclusion.
Guidelines:
The answer NA means that there is no societal impact of the work performed.
If the authors answer NA or No, they should explain why their work has no societal impact or why the paper does not address societal impact.
Examples of negative societal impacts include potential malicious unintended uses (e.g., disinformation, generating fake profiles, surveillance), fairness considerations (e.g., deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations.
The conference expects that many papers will be foundational research and not tied to particular applications, let alone deployments. However, if there is a direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that an improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point out that a generic algorithm for optimizing neural networks could enable people to train models that generate Deepfakes faster.
The authors should consider possible harms that could arise when the technology is being used as intended and functioning correctly, harms that could arise when the technology being used as intended but gives incorrect results, and harms following from (intentional or unintentional) misuse of the technology.
If there negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition to attacks, mechanisms for monitoring misuse, mechanisms to monitor how system learns from feedback over time, improving the efficiency and accessibility of MI...
11. Safeguards
Does the paper describe safeguards that have been put in place for responsible release of data models that have high risk for misuse (c.g., pretrained language models, image generators, or scraped datasets)?
Question:
Answer: [NA]
Justification:
Our methodology not release new data or models.
Guidelines:
The answer NA means that the paper poses such risks.
Released models that have high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere to usage guidelines or restrictions to access the model or implementing safety filters.
Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images.
We recognize that providing effective safeguards is challenging, and many papers do not require this, but we encourage authors to take this into account and make best faith effort.
33

---

12. Licenses for existing assets
Question:
Are the creators or original owners of assets (e.g., code, data, models), used in the paper, properly credited and the license and terms of use explicitly mentioned and properly respected?
Answer: [Yes]
Justification: Yes, we give proper citations and reference for used resources.
Guidelines:
The answer NA means that the paper does not use existing assets.
The authors should cite the original paper that produced the code package dataset.
The authors should state which version of the asset used and, possible, include URL.
The name of the license (e.g.. CC-BY 4.0) should be included for each asset. For scraped data from a particular source (e.g., website), the copyright and terms of service of that source should be provided.
If assets are released, the license, copyright information, and terms of use in the package should be provided. For popular datasets, paperswi thcode. com datasets has curated licenses for some datasets. Their licensing guide can help determine the license dataset.
For existing datasets that are re-packaged, both the original license and the license of the derived asset (if it has changed) should be provided.
this information is available online, the authors are encouraged to reach out to the asset's s creators.
13. New Assets
Question: Are new assets introduced in the paper well documented and is the documentation provided alongside the assets?
Answer: [NA]
Justification:
Our method is training free, can accelcrate the LLMs inference.
Guidelines:
The answer NA means that the paper does not release new assets.
Researchers should communicate the details the dataset/code/model as part of their submissions via structured templates. This includes details about training, license, limitations, etc.
The paper should discuss whether and how consent was obtained from people whose asset used.
At submission time, remember to anonymize your assets (if applicable). You can either create anonymized URL or include an anonymized zip file.
14. Crowdsourcing and Research with Human Subjects
Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given participants and screenshots, if applicable, as well as details about compensation (if any)?
Answer: [NA]
Justification:
This paper does not study human subjects.
Guidelines:
The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.
Including this information the supplemental material is fine, but if the main contribu- tion of the paper involves human subjects, then as much detail as possible should be included in the main paper.
According to the NeurlPS Code of Ethics, workers involved in data collection, curation, or other labor should be paid at least the minimum wage in the country of the data collector.
34

---

15. Institutional Review Board (IRB) Approvals or Equivalent for Research with Human Subjects
Question: Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or equivalent approval/review based the requirements of your country institution) were obtained?
Answer: [NA]
Justification:
We are not subject to the approval of IRB.
Guidelines:
The answer NA means that the paper does not involve crowdsourcing nor research with
human subjects.
Depending on the country in which research is conducted, IRB approval (or equivalent)
may be required for any human subjects research. If you obtained IRB approval, you should clearly state this in the paper.
We recognize that the procedures for this may vary significantly between institutions and locations, and we expect authors to adhere the NeurIPS Code of Ethics and the guidelines for their institution.
For initial submissions, do not include any information that would break anonymity (if applicable), such as the institution conducting the review.
35


Reviews:
[{"review_id": "zfhIJsDozP", "reviewer": "Reviewer_TyMG", "summary": "This paper introduces MiniCache, a novel approach to compressing the Key-Value (KV) cache in large language models (LLMs) to enhance inference efficiency. The KV cache is crucial in storing key-value states of previously generated tokens, significantly reducing redundant computations and lowering latency during autoregressive generation. However, as the sequence length increases, the KV cache's size also grows linearly, leading to substantial memory consumption. MiniCache addresses this issue by compressing the KV cache across layers from a depth perspective, leveraging the observation that KV cache states exhibit high similarity between adjacent layers in the middle-to-deep portions of LLMs. The proposed method involves disentangling the states into magnitude and direction components, interpolating the directions while preserving the lengths, and retaining highly distinct state pairs unmerged to minimize information loss. MiniCache is a training-free, general approach that complements existing KV cache compression strategies like quantization and sparsity.\n\nThe authors conducted comprehensive evaluations using various models, including LLaMA-2, LLaMA-3, Phi-3, Mistral, and Mixtral, across multiple benchmarks. The results demonstrated that MiniCache achieves superior compression ratios and high throughput, with LLaMA-2-7B showing a compression ratio of up to 5.02×, a 5× increase in inference throughput, and a 41% reduction in memory footprint compared to the FP16 full cache baseline, all while maintaining near-lossless performance. The paper highlights the potential of MiniCache to significantly reduce memory requirements and enhance the efficiency of LLM inference, making it a promising solution for applications requiring long context inputs and extensive sequence generation.", "questions": "1. **Distance-Based Threshold for Retention (Line 226)**\n   Why did you choose a distance-based threshold instead of the merging ratio of overall tokens as the control for retention? Can you show the effect on accuracy and efficiency as the ratio of retention tokens varies, highlighting this trade-off?\n\n2. **Ratio of Merged Tokens in Ablation Study**\n   In the second ablation study, can you show the ratio of merged tokens and the efficiency trade-offs for different settings?\n\n3. **Implementation Source Code**\n   Can you include the implementation source code?\n\n4. **Justification for SLERP**\n   Can you justify your choice of SLERP over other interpolation methods?", "limitations": "There should be a section discussing the limitation of this work and its social impact.", "rating": 7, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "The paper presents a substantive contribution to the field of efficient machine learning by introducing a novel, high-quality, and clearly explained method for KV cache compression in large language models. The significance of the work is underscored by its potential to improve the practicality and scalability of LLMs, addressing a critical bottleneck in their deployment. The originality of the approach, combined with comprehensive evaluations and clear exposition, makes this paper a valuable addition to the literature on efficient ML techniques.\n\n**Originality**\n\n- **Novel Compression Approach**: The paper introduces a unique method for KV cache compression in large language models by exploring the depth dimension, which is a previously overlooked area. This novel perspective of compressing across layers rather than within layers demonstrates a creative combination of existing ideas in a new, impactful way.\n- **Reparameterization Strategy**: The method's use of reparameterization to disentangle state vectors into magnitude and direction components for interpolation is innovative. This approach preserves important information while effectively reducing memory usage.\n\n**Quality**\n\n- **Comprehensive Evaluation**: The authors provide a thorough evaluation of MiniCache across various models and benchmarks, including LLaMA-2, LLaMA-3, Phi-3, Mistral, and Mixtral. The extensive experiments validate the method's effectiveness and robustness.\n- **Performance Metrics**: The results show significant improvements in compression ratios, inference throughput, and memory footprint reduction, with metrics such as a 5.02× compression ratio and a 41% reduction in memory usage while maintaining near-lossless performance. These metrics highlight the quality and practicality of the proposed solution.\n\n**Clarity**\n\n- **Detailed Exposition**: The paper is well-written, with a clear and detailed exposition of the methodology. Figures and tables are effectively used to illustrate key concepts, observations, and results, aiding in the understanding of the approach and its benefits.\n- **Step-by-Step Explanation**: The authors provide a step-by-step explanation of the MiniCache method, from the initial observations to the final implementation, making the paper accessible even to those less familiar with the intricacies of KV cache compression.\n\n**Significance**\n\n- **Addressing a Critical Issue**: The paper addresses a significant challenge in the deployment of large language models – the growing memory consumption of KV caches with increasing sequence lengths. By reducing the memory footprint and improving inference efficiency, MiniCache has the potential to make LLMs more practical and scalable in real-world applications.\n- **Broad Applicability**: The approach is general and training-free, making it applicable to a wide range of models and scenarios. This broad applicability enhances the significance of the work, as it can be integrated into existing systems with minimal modification.", "weaknesses": "1. **No Implementation Source Code Provided**: The paper does not include the implementation source code, which is a significant limitation. Releasing the source code would facilitate further research and enable other researchers to replicate and build upon the work. Providing the code upon publication would enhance the paper's impact and encourage broader adoption of the proposed method.\n\n2. **Insufficient Justification for SLERP**: The introduction of Spherical Linear Interpolation (SLERP) in the paper feels abrupt and lacks sufficient justification. While SLERP is used for interpolating between vectors, the paper does not provide enough rationale for why this specific technique was chosen over other interpolation methods. More ablation studies should be conducted to demonstrate the effectiveness and necessity of using SLERP in this context. These studies could compare SLERP with alternative interpolation techniques to show its advantages and validate the authors' choice.\n\n### Minor Writing Improvements\n\n1. **Figure Clarity**: \n   - **Figure 1(a)**: The resolution of Figure 1(a) could be higher, or the figure could be replaced with a vector graphic to improve clarity and readability. Enhancing the visual quality would make the figure easier to understand and more professional.\n\n2. **Typographical Corrections**:\n   - **Line 308**: The word \"interpretation\" should be corrected to \"interpolation\".\n   - **Line 328**: The phrase \"A larger t\" should be corrected to \"A larger \\(\\gamma\\)\".", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thanks to the reviewer for the valuable comments.\n\n**Q1: Distance-Based Threshold for Retention (Line 226)**. **Why did you choose a distance-based threshold instead of the merging ratio of overall tokens as the control for retention? Can you show the effect on accuracy and efficiency as the ratio of retention tokens varies, highlighting this trade-off?**\n\n**A1:**\nAccording to our Observation 2, token pairs with lower similarity scores are more critical for restoration, and different adjacent layers exhibit distinct patterns.\n\nOur proposed **dynamic distance-based threshold** allows us to selectively retain salient tokens according to low similarity scores. Setting the retention ratio of the overall tokens as a hyperparameter is impractical because determining the optimal retention ratio for each layer manually is challenging due to the dynamic nature across different layers.\n\nTo illustrate the trade-off regarding accuracy and efficiency, we conducted experiments by varying the ratio of retention tokens, as shown in Table I in the rebuttal PDF. We observe that as the overall token retention ratio increases, accuracy improves up to a certain point before plateauing. Retaining 20% of the tokens is necessary to ensure all salient tokens are preserved without performance degradation. In contrast, using a **dynamic distance-based threshold**, as proposed in our paper, we only need to retain the top 5% of the salient tokens. Note that efficiency decreases as more tokens are retained. This demonstrates that our distance-based approach better balances performance and efficiency than the fixed retention ratio counterpart.\n\n**Q2: Ratio of Merged Tokens in Ablation Study. In the second ablation study, can you show the ratio of merged tokens and the efficiency trade-offs for different settings?**\n\n**A2:**\nWe are adding another column in terms of compression ratio to represent the efficiency trade-offs, as shown in Table II in the rebuttal PDF. The results suggest that setting γ to 0.05 achieves the best balance between performance and efficiency.\n\n**Q3: No Implementation Source Code Provided. Can you include the implementation source code?**\n\n**A3:**\nWe will definitely release our source code upon acceptance.\n\nTo ensure reproducibility, we have included comprehensive pseudocode in the Appendix, specifically in Algorithm 1 and Algorithm 2.\n\n- **Algorithm 1:** This algorithm outlines the overall logic of the MiniCache Inference process, covering both the prefilling and decoding stages. It provides a step-by-step breakdown of the core processes involved in our approach.\n- **Algorithm 2:** This algorithm details the compression and restoration processes. It elaborates on how the KV cache is compressed and subsequently restored, highlighting the technical intricacies and optimizations employed in our method.\n\nWe believe that the inclusion, along with the detailed comments, will significantly aid the community in grasping the logic and mechanics of our implementation.\n\n**Q4: Justification for SLERP. Can you justify your choice of SLERP over other interpolation methods?**\n\n**A4:**\n1. Motivation for Using SLERP:\nWe conceptualize the KV Cache as activations, which consist of two factors: magnitude and direction in the spherical feature space. SLERP allows us to use a more compact form to represent the original KV Cache by merging states effectively. We can compute the overall magnitude vector and store it in a channel with a dimension equal to 1, significantly reducing the memory overhead, as shown in *section 4.2*. \n\n2. Performance Metrics and Empirical Evidence:\nIn our initial experiments, we considered average merging as the preliminary method; however, the performance in terms of accuracy was not promising, as shown in Figure 2(a). Subsequently, we conducted experiments using maximum norm-preserving interpolation, but it demonstrated lower accuracy compared to SLERP, as detailed in Appendix Section C (Line 579). Ultimately, we selected SLERP [64] as our final solution due to its multiple advantages in key performance metrics, including compression ratio, accuracy, and computational efficiency.\n\nTo further substantiate our claim, we conducted additional ablation studies comparing SLERP with average merging and maximum norm-preserving interpolation. The results, as shown in Table VI in the rebuttal PDF, reaffirm that SLERP provides superior performance in information preservation.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "ityAqDUDSX", "reviewer": "Reviewer_fj43", "summary": "The paper proposes a novel method called MiniCache for compressing the KV cache in large language models (LLMs) by merging cache states across layers. The authors argue that this approach significantly reduces the memory footprint and enhances inference throughput without significant performance loss. The paper includes evaluations on various models and benchmarks, demonstrating the effectiveness of MiniCache in terms of compression ratio and throughput improvement.", "questions": "1. Beyond memory compression, what is the impact of MiniCache on latency?", "limitations": "As mentioned above.", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 2, "strengths": "1. The idea of compressing the KV cache by merging states across layers introduces a new perspective on reducing memory usage during inference. This approach could inspire further research in inter-layer redundancy exploitation.\n2. Memory consumption and inference speed are critical challenges in deploying LLMs. The proposed MiniCache method addresses these issues directly, offering a potentially valuable tool for practitioners working with resource-constrained environments.\n3. The method does not require additional training, making it easy to integrate into existing inference pipelines without the need for extensive retraining or fine-tuning of models.", "weaknesses": "**Major Weekness 1: Lack of Baseline Comparisons**   \nIn Table 1, the authors only include quantization methods for performance comparison, neglecting other KV cache eviction methods such as those proposed in [1] and [2].      \n\n**Major Weekness 2: Lack of Evaluation for Instruction-following Benchmarks**   \nGiven that instruction-tuned models are more generalizable for downstream applications, it is essential to evaluate how MiniCache performs on instruction-following benchmarks such as MT-Bench [3].\n\n\n[1] H2O: Heavy-Hitter Oracle for Efficient Generative Inference of Large Language Models, NeurIPS 2023   \n[2] Efficient Streaming Language Models with Attention Sinks, ICLR 2024   \n[3] Judging LLM-as-a-Judge with MT-Bench and Chatbot Arena, NeurIPS 2023", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thanks to the reviewer for the valuable comments.\n\n**Q1: Lack of Baseline Comparisons. In Table 1, the authors only include quantization methods for performance comparison, neglecting other KV cache eviction methods such as those proposed in [14] and [15].**  \n\n**A1:** We have included the baseline comparison with H2O [14] in Appendix Section A, along with additional experimental results. We further add the Attention Sink [15] benchmark, as shown in Table III in the rebuttal PDF.\n\nAccording to this table, MiniCache consistently outperforms H2O [14] and Attention Sink [15]. Additionally, MiniCache explores the KV Cache in a **novel depth perspective**, making it **orthogonal to existing quantization and sparsity techniques** for further improvements.\n\n**Q2: Lack of Evaluation for Instruction-following Benchmarks. Given that instruction-tuned models are more generalizable for downstream applications, it is essential to evaluate how MiniCache performs on instruction-following benchmarks such as MT-Bench [A].**\n\n**A2:**\nWe benchmark the MiniCache using the MT-Bench dataset, which focuses on instruction-following tasks, as shown in Table VIII in the rebuttal PDF. The models employed for this benchmarking include LLaMA-2-7B-Chat, Mistral-7B-Instruct, and LLaMA-3-8B-Instruct. Our MiniCache-4bit exhibits a reasonably slight performance reduction caused by 4-bit quantization. However, MiniCache can achieve almost lossless performance via standalone cross-layer merging. \n\n**Q3: Latency Benchmark. Beyond memory compression, what is the impact of MiniCache on latency?**\n\n**A3:**\nWe benchmark the latency of LlaMA-2-7B on an NVIDIA A100 GPU using different sequence lengths ranging from 1024 to 4096 with a batch size of 16, as shown in Table IV in the rebuttal PDF. We compare it with H2O, which requires calculating full attention scores to estimate token importance. In contrast, MiniCache performs reparameterization-based merging and token restoration using simple matrix manipulations, resulting in more lightweight computations and lower latency. Specifically, when the sequence length is 4096, MiniCache shows a 36.83% reduction in latency compared to H2O.\n\nReferences:\n\n[A] Judging LLM-as-a-Judge with MT-Bench and Chatbot Arena, NeurIPS 2023.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "M8rHvAUtGI", "reviewer": "Reviewer_ZDdv", "summary": "Authors proposed a KVCache Compression Method, Minicache, is introduced as a method to efficiently compress the Key-Value (KV) cache in large language models (LLMs) by leveraging the high similarity of KV states between adjacent layers in the middle-to-deep portion of LLMs. This compression is achieved by disentangling states into magnitude and direction components and interpolating directions while retaining distinct state pairs to minimize storage overhead. MiniCache is training-free, complements existing compression strategies, and demonstrates significant performance improvements, including up to a 5.02x compression ratio, a 5x enhancement in inference throughput, and a 41% reduction in memory footprint, all while maintaining near-lossless performance across various models and benchmarks.", "questions": "See weaknesses.", "limitations": "The authors address the limitations inherited from the SLERP method and propose future research directions to enhance the adaptability of the merge compression strategy.", "rating": 7, "confidence": 5, "soundness": 4, "presentation": 3, "contribution": 4, "strengths": "1.[Novel Approach, Interesting Observation] The authors explore KV cache compression from a novel perspective by focusing on the depth dimension, which is previously unexplored. They identify a significant finding: KV cache states exhibit high similarity between adjacent layers in the middle-to-later stages of LLMs. This discovery is backed by a thorough analysis and visualization of layer-wise similarity statistics, leading to a well-motivated methodology for compressing the KV cache. This approach has substantial implications for the development of KV cache compression techniques.\n\n2.[Sound Methodology] The authors propose a training-free strategy for cross-layer merging, formulating a reparametrization compression technique to enhance inference efficiency. This method carefully considers outliers, retaining these specific tokens and restoring them through a well-constructed strategy.\n\n3.[Strong Experiments, Well Compatibility] Extensive experiments across multiple datasets with four different models demonstrate that the proposed MiniCache consistently performs effective compression without compromising performance, achieving a superior compression ratio of ~5× and outperforming existing methods. Furthermore, the proposed method is compatible with existing quantization strategies, highlighting its practical values.\n\n4.The paper is well-written, which is easy to understand.", "weaknesses": "1.I am concerned about the computational overhead during the reparametrization and restoration stages. Does this compression strategy increase computational overhead?\n2.The authors selected the middle layers for cross-layer merging compression. Have the authors considered any principled methods to further increase the compression ratio by compressing the shallow layers in future work?", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thanks to the reviewer for the valuable comments.\n\n**Q1 : Concern about computational overhead during reparametrization and restoration stages. Does this compression strategy increase computational overhead?**\n\n**A1:**\nReparametrization-based compression involves computing magnitude and direction vectors, and applying the SLERP algorithm through simple matrix manipulation. Additionally, we compute pairwise distances to identify salient tokens, as detailed in Section 4.2.\n\nFor restoration, we utilize an in-place index-based token replacement operation, where the computational overhead is managed by the PyTorch index selection procedure, as mentioned in line 216.\n\nAs shown in Table V of the rebuttal PDF, the reparametrization process, including the computation of magnitude and direction, takes 0.093 ms (0.031 ms + 0.062 ms), and the restoration process, including the computation of the distance matrix and token replacement, takes 0.116 ms (0.061 ms + 0.055 ms). These times indicate a negligible computational overhead compared to the overall attention computation, which takes 9.756 ms. The running time is measured in milliseconds per attention layer with a batch size of 1 on LLaMA-2-7B.\n\n**Q2: The authors selected the middle layers for cross-layer merging compression. Have the authors considered any principled methods to further increase the compression ratio by compressing the shallow layers in future work?**\n\n**A2:**\nYes, we have considered and proposed methods to further improve the compression ratio.\n\nOur observations indicate that certain shallow layers exhibit low similarities, suggesting the presence of differences and residual errors. To address this, we suggest employing a meta-learning-based approach to approximate these differences in the shallow layers. Once these approximations are made, we can bridge the gaps in the shallow layers, enabling their effective merging and compression. The possibility of merging shallow layers has been demonstrated in concurrent work [B], though it trains from scratch.\n\nReferences:\n\n[B] You only cache once: Decoder-decoder architectures for language models. Arxiv 2024.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "T0wCZ0JAy7", "reviewer": "Reviewer_GYW9", "summary": "This paper proposes a KV cache compression method by merging keys and values of consecutive layers. Based on the empirical observation that keys and values of consecutive layers after the mid-depth layer have high cosine similarity, the authors propose a merging strategy using angular interpolation. Additionally, they emphasize the importance of retention tokens based on their empirical findings. Extensive experiments across multiple state-of-the-art models and various evaluation benchmarks demonstrate the effectiveness of the proposed compression method.", "questions": "- What will be the results when applying the merging function to more than two layers?\n- How about applying the merging function based on cosine similarity? For instance, only apply merging when the cosine similarity is higher than a specified threshold. This might address the issues mentioned in the first weakness above.\n- Could you compare the proposed method with early layer-exiting methods?\n- In Eq (4), is the retention index defined per layer? If not, how do you measure d_i, d_{min}, and d_{max}?\n- Could you provide an intuitive analysis of why some tokens have severely low cosine similarity in Figure 2 (b)? Are there any common characteristics of these tokens?\n- Do you apply merging per attention head or to the entire key/value vectors?", "limitations": "In appendix.", "rating": 6, "confidence": 5, "soundness": 2, "presentation": 2, "contribution": 3, "strengths": "- The proposed approach of merging keys and values can have a significant practical impact.\n- The method has low computational overhead while providing meaningful efficiency improvements.\n- The proposed approach is training-free.\n- Experiments conducted on various models and benchmarks demonstrate the effectiveness of the proposed method.\n- The writing is overall clear.", "weaknesses": "- Some methodological design choices seem ad-hoc. For example, the method starts merging after the mid-layer and only merges two consecutive layers. I believe the paper could be technically improved by addressing these design choices.\n- The authors compare the efficiency of the method after applying quantization (MiniCache 4-bit) to the FP16 baseline. This comparison overstates the method's effectiveness because the contribution of the paper does not involve quantization. The paper should measure efficiency improvement without using quantization. Currently, lines L22-24, L302-303, etc., can mislead readers.\n- A comparison to early layer-exiting approaches seems more relevant than comparisons to KV quantization methods, which the current draft focuses on.\n   - e.g., \"Fast and Robust Early-Exiting Framework for Autoregressive Language Models with Synchronized Parallel Decoding\", EMNLP, 2023\n\n**minor comments**  \n- Figure 1, index -> layer index\n- Line 140, Figure 1 (b) -> Figure 1 (a)\n- For related works, it would be informative to include token compression methods: \n  - \"Compressed Context Memory for Online Language Model Interaction\", ICLR 2024\n  - \"Recurrent memory transformer\", NeurIPS 2022", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thanks to the reviewer for the valuable comments.\n\n**Q1: Ad-hoc design choices (mid-layer merging, only two layers) need improvement.**\n\n**A1:**\nIt is worth noting that MiniCache is the pioneer work to explore KV cache compression along the depth dimension (Lines 48-50). This insight has been highly recognized by Reviewer TyMG (*\"...novel perspective of compressing across layers\"*), Reviewer fj43 *(“The idea ... introduces a new perspective”)* and Reviewer ZDdv (*“This novel perspective, previously unexplored”*).\n\nWith this new motivation, the design of MiniCache is initially based on strong empirical evidence as shown in Figure 1(a), where KV cache states exhibit high similarity between the adjacent layers in the middle-to-deep portion of LLMs. Thus,\n\n1. Merging after the mid-layer is **a reasonable and good starting point for our early exploration** as it would ideally keep most performance of LLMs.\n2. More importantly, **we have already shown that MiniCache can perform cache-merging across more layers in Figure 4**, e.g. from shallow to deep layers. We have also discussed cross-layer merging beyond two layers using an advanced merging algorithm termed Spherical Cubic Interpolation in further work (Lines 339-342).\n\nFurthermore, we agree that our current approach could be slightly simple as an early effort. However, at this stage, we have already conducted comprehensive experiments to demonstrate the effectiveness of our main idea of cross-layer KV cache merging. Notably, this has been **recognized by all reviewers.** Thus, we will leave more advanced design for future works, as also recognized by Reviewer fj43 (\"could inspire further research in inter-layer redundancy exploitation.\").\n\n**Q2: Results of merging more than two layers?**\n\n**A2:**\nWe have considered multiple-layer merging functions; however, in the current version of MiniCache, we employ SLERP (Spherical Linear Interpolation) as our merging function. This choice is due to SLERP's inherent ability to merge only two vectors.\n\nTo extend this capability to more than two layers, please refer to general response **Q3**.\n\nAs discussed in Q1, we will explore merging across more than two layers in future work.\n\n**Q3: Merge based on cosine similarity with a threshold?**\n\n**A3:**\nIn our approach, we used angular distance as the threshold for merging the KV cache.\nMathematically, cosine similarity $\\cos(\\theta)$ can be converted to angular distance $d$ using the following relationship:\n$d = \\frac{1}{\\pi}\\arccos(\\cos(\\theta))$\nThus, merging based on angular distance is equivalent to merging based on cosine similarity.\n\n\n\n**Q4: Compare with early layer-exiting methods?**\n\n**A4:**\nBoth early-exiting and MiniCache aim to accelerate inference by addressing the depth dimension. Early-exiting is more effective for compute-bound operations, whereas MiniCache is advantageous for memory-bound operations. Furthermore, early-exiting methods typically require extensive training, whereas MiniCache is training-free, making it broadly applicable to a wide range of off-the-shelf models and scenarios.\nIn future work, we will explore combining these two paradigms. The relevant reference [C] will be included in the revised paper.\n\n**Q5: Is the retention index defined per layer in Eq (4)? How to measure d_i, d_{min}, and d_{max}?**\n\n**A5:**\nIn Eq. (4), the retention index is **shared for the paired KV states by position between adjacent layers**. Specifically, $d_i$ represents the angular distance between the paired tokens, with $d_{min}$ and $d_{max}$ denoting the minimum and maximum distances across all paired tokens between two adjacent layers, respectively. The rationale behind this approach is to retain paired tokens when they exhibit low similarity, which corresponds to a large distance $d_i$ exceeding the retention threshold.\n\n**Q6: Why do some tokens have low cosine similarity in Fig 2(b)? Are there any common characteristics?**\n\n**A6:**\nWe conducted a thorough analysis and observed that the salient tokens often include the initial token and punctuation, as illustrated in Figure I and Table IX of the rebuttal PDF. \n\nWe have observed that the numerical values of these salient tokens are relatively large compared to other tokens. After passing through the attention layers, these salient tokens demonstrate significant numerical differences, as they receive strong attention despite not being semantically important. Similar phenomena have been observed in previous studies [D][E]. Consequently, the significant numerical differences between these tokens result in low similarity, rendering them unmergeable in our algorithm. We will include the analysis in the revised paper.\n\n**Q7: Merging per attention head or entire key/value vectors?**\n\n**A7:**\nWe apply compression to the entire key and value vectors.\n\n**Q8: Efficiency comparison without quantization is needed to avoid misleading readers.**\n\n**A8:**\nTable VII in the rebuttal PDF demonstrates that MiniCache's cross-layer merging without quantization achieves almost lossless performance compared to the FP16 baseline, with a compression ratio of 1.53. In contrast, 2-bit quantization causes performance degradation. For instance, on the GSM8K dataset, performance drops from 0.159 to 0.127, but the compression ratio improves to 3.95.\n\nOur MiniCache method, focusing on the depth dimension, can complement any quantization and existing KV cache compression methods. The results indicate that combining cross-layer merging with 4-bit quantization achieves the optimal balance between performance and efficiency.\n\nTo avoid confusion, we will revise the description as ``On the ShareGPT dataset, LLaMA-2-7B with cross-layer merging achieves a compression ratio of 1.53. Additionally, since MiniCache is orthogonal to existing quantization techniques, it can achieve a compression ratio of up to 5.02x when combined with the 4-bit quantization technique.”\n\n**Note: Please refer to the reference list in the general response.**", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:
## Response to all reviewers

We sincerely thank all reviewers for their valuable comments.

All reviewers agree that:

**The Novel Approach:**

- "The paper introduces a unique method …. This novel perspective impacts the field in a new, impactful way." (Reviewer TyMG)
- "The idea ... introduces a new perspective ... . This approach could inspire further research ..."(Reviewer fj43)
- "This novel perspective, previously unexplored, identifies a significant finding ..." (Reviewer ZDdv)

**The Insightful Observation:**

- "They identify a significant finding ... A thorough analysis and visualization back this discovery... ." (Reviewer ZDdv)
- "The authors provide a step-by-step explanation ... , from the initial observations to the final implementation." (Reviewer TyMG)

**The Sound Methodology:**

- "The method is innovative, preserving important information while effectively reducing memory usage." (Reviewer TyMG)
- "The authors propose a training-free strategy ... . This method ... a well-constructed strategy" (Reviewer ZDdv)
- "The method does not require additional training, making it easy to integrate into existing inference pipelines ..." (Reviewer fj43)

**The Practical Impact:**

- "MiniCache offers a potentially valuable tool." (Reviewer fj43)
- "The proposed approach has a significant practical impact." (Reviewer GYW9)
- "The approach is general and training-free ... have broad applicability ..." (Reviewer TyMG)

**The Comprehensive Experiments:**

- "The authors provide a thorough evaluation across various models and benchmarks. The extensive experiments validate the method's effectiveness and robustness." (Reviewer TyMG)
- "Extensive experiments demonstrate that MiniCache consistently performs effective compression without compromising performance." (Reviewer fj43)

## General Response

**Q1: Comparative Analysis and Benchmarks (Reviewers GYW9 and fj43)**

- We include comparisons with other KV cache eviction methods (e.g., those proposed in H2O and Attention Sinks) to strengthen our evaluation, as shown in Table III in the rebuttal PDF.
- We evaluate our method on instruction-following benchmarks like MT-Bench to demonstrate its generalizability for downstream applications, as shown in Table VIII in the rebuttal PDF.

**Q2: Efficiency and Overhead (Reviewers GYW9 and fj43)**

- We address concerns regarding computational overhead by measuring the different stages introduced during the reparametrization-based merging and token restoration, as shown in Table V in the rebuttal PDF.
- We evaluate the latency of MiniCache and compare latency with a baseline method H2O, as shown in Table IV in the rebuttal PDF. We will include these results in the revised paper.

**Q3: Future work and direction (Reviewers GYW9 and ZDdv)**

- Advanced Merging Techniques: Techniques such as Spherical Cubic Interpolation [75], mentioned in the further work Section 7, allow for the interpolation of multiple vectors. This method enables the effective merging of more than two layers.
- Multiple-Round Based Merging Strategy: This strategy involves iteratively merging layers by first combining two layers, then merging the resultant layer with a third one. However, this approach is less efficient and may introduce additional computational complexity.
- Approximation-Based Methods: To merge the KV cache in shallow layers, we plan to develop a meta-learning method to approximate their differences and merging errors. As shown in Figure 1 (a), some shallow layers exhibit low similarity, indicating significant differences. By approximating these differences, we can effectively merge and compress the KV cache in shallow layers, thereby enhancing the overall compression ratio.

Additionally, we thank all minor writing improvements given by Reviewers TyMG and GYW9. We will take these suggestions into account and improve them in the final manuscript.

**Reference:**

[A] Judging LLM-as-a-Judge with MT-Bench and Chatbot Arena. NeurIPS 2023.

[B] You only cache once: Decoder-decoder architectures for language models. Arxiv 2024.

[C] Fast and Robust Early-Exiting Framework for Autoregressive Language Models with Synchronized Parallel Decoding. EMNLP 2023

[D] Quantizable Transformers: Removing Outliers by Helping Attention Heads Do Nothing. NeurIPS 2023.

[E] Efficient Streaming Language Models with Attention Sinks. ICLR 2024.

Meta-review: The AC has carefully read all reviewers' comments and the authors' response. The overall sentiment to the paper is quite positive. The main reasons for accepting the paper are:
1) The novel approach -- KV cache compression along the depth dimension (3 out of 4 reviewers).
2) Insightful observations -- KV cache at certain layers exhibit high similarity (2 out of 4 reviewers).
3) Sound and easy-to-us methodology (3 out of 4 reviewers). 
4) Comprehensive evaluation -- across multiple models and benchmarks (2 out of 4 reviewers). 

The main concerns from the reviewers are:
1) Lack of justification of design choices, e.g., SLERP.  
2) Missing important baselines. 
3) Ad-hoc design, e.g., layer selection policy.

Some of the concerns were addressed via the rebuttal, such as (1) and (2). The other is indeed weakness but can be further improved via follow-up work. 

Overall, this is an interesting paper that explores a previous less looked dimension for compressing KV cache. It would be great if the authors can also add the actual memory usage, latency, throughput with batch size >1 comparison between MiniCache and alternative methods.
Decision: Accept (poster)