MiniCache: KV Cache Compression in Depth Dimension for Large Language Models

Authors: akide  liu
jing  liu
zizheng  pan
yefei  he
gholamreza  haffari
bohan  zhuang
prevs  minicache
Venue: NeurIPS_cc_2024_Conference
Published: 2024-05-03 08:56:23

Abstract: A critical approach for efficiently deploying computationally demanding large language models (LLMs) is Key-Value (KV) caching. The KV cache stores key-value states of previously generated tokens, significantly reducing the need for repetitive computations and thereby lowering latency in autoregressive generation. However, the size of the KV cache grows linearly with sequence length, posing challenges for applications requiring long context input and extensive sequence generation. In this paper, we present a simple yet effective approach, called MiniCache, to compress the KV cache across layers from a novel depth perspective, significantly reducing the memory footprint for LLM inference. Our approach is based on the observation that KV cache states exhibit high similarity between the adjacent layers in the middle-to-deep portion of LLMs. To facilitate merging, we propose disentangling the states into the magnitude and direction components, interpolating the directions of the state vectors while preserving their lengths unchanged. Furthermore, we introduce a token retention strategy to keep highly distinct state pairs unmerged, thus preserving the information with minimal additional storage overhead. Our MiniCache is training-free and general, complementing existing KV cache compression strategies, such as quantization and sparsity.  We conduct a comprehensive evaluation of MiniCache utilizing various models including LLaMA-2, LLaMA-3, Phi-3, Mistral, and Mixtral across multiple benchmarks, demonstrating its exceptional performance in achieving superior compression ratios and high throughput. On the ShareGPT dataset, LLaMA-2-7B with cross-layer merging achieves a compression ratio of $1.53\times$. Additionally, since MiniCache is orthogonal to existing quantization techniques, it can achieve a compression ratio of up to $5.02\times$ when combined with the 4-bit quantization technique, enhancing inference throughput by approximately $5\times$ and reducing the memory footprint by $41\%$ compared to the FP16 full cache baseline, all while maintaining near-lossless performance. Project is available at https://minicache.vmv.re .

Main paper: 
 introduction: 
 Large Language Models (LLMs), exemplified by the GPT series [1, 2, 3] and the LLaMA series [4, 5, 6] , have emerged as pivotal innovations within the artificial general intelligence, significantly enhancing the capabilities of natural language processing. However, these models are meticulously trained using extensive computational resources [7] and massive datasets [8] , which enables them to Figure 1 : Overview of our MiniCache strategy and example results: (a) shows the observation that the KV cache states between two adjacent layers are highly similar, particularly across the middle to deep layers. The x-axis uses index/2 to represent the similarities for each pair of layers. (b) compares the performance of MiniCache, and the mean baseline, which simply averages the KV caches of two layers, using the LLaMA-3-70B model [6] on the GSM8K dataset [10] . MiniCache, which begins merging from the half-layer depth, achieves near-lossless performance. (c) highlights the primary difference between MiniCache and previous approaches. MiniCache investigates the inter-layer redundancy of KV caches along the depth dimension of LLMs, an aspect overlooked by intra-layer-based methods. Here, T refers to the last timestamp of pre-filling, and T + 1 des to the first timestamp of decoding.produce text that effectively mimics human writing styles and conducts complex reasoning analysis, yet raises the challenge of efficient deployment and serving. Within the inference framework of LLMs, KV caches [9] are crucial for storing pre-computed keys and values, thus avoiding repeated calculations over the preceding context and substantially enhancing LLMs' deployment efficiency. However, the increasing demand for longer sequence lengths results in voluminous cached states, leading to significant memory consumption during generation. For instance, a 175B GPT-3 model [2] , with a batch size of 64 and a sequence length of 4,096 tokens (both prefilled and generated), requires approximately 1,208GB of GPU memory. This requirement is 3.45× greater than the memory used to store the model's weights. In this context, KV cache compression is of paramount importance due to its clear benefits: 1) it largely reduces the memory footprint allowing for faster generation and larger batch serving; 2) it significantly lowers the cost per token, demonstrating substantial commercial benefits.Existing KV cache compression efforts can be roughly categorized into two types, namely quantization and sparsity. The quantization approaches [11, 12] propose storing the KV states in low-bit numerical values. Typically, FlexGen [13] demonstrates that 4-bit KV cache quantization can achieve lossless performance. In contrast, sparsity-driven methods aim to retain only the salient tokens while evicting the rest, either heuristically [14, 15] or adaptively [16] . Some approaches [11] explore the intersection of these two types, by assigning high-bit to salient tokens and extremely low-bit to the rest of the tokens, achieving more aggressive memory gain. Despite these innovations, existing literature merely consider the intra-layer redundancy, while neglecting another important complementary directionthe inter-layer redundancy, as illustrated in the Figure 1(c) .Our analysis begins by exploring the redundancy of KV caches along the depth dimension, as shown in Figure 1(a) . We observe that KV cache states exhibit high similarity between neighbouring layers in the middle-to-deep portions of LLMs. This intriguing property suggests that states paired by position between adjacent layers can be accurately merged into a single state space with a strong performance guarantee, as illustrated in Figure 1(b) . This approach significantly reduces the memory footprint without the need to retain individual states for each attention layer. Note that these observations are pertinent to dynamic inference strategies such as mixture-of-depths [17] and layer-wise early exiting [18, 19] , which optimize computational paths by skipping non-critical layers to enhance training and inference efficiency. Furthermore, layer pruning methods [20] highlight considerable redundancy in deeper layers. However, despite these advancements, the redundancy of KV caches along the depth dimension has largely been overlooked.In this paper, we propose MiniCache, a simple yet effective cross-layer KV cache compression method aimed at advancing the inference efficiency of LLMs. MiniCache consists of two essential components. Firstly, we introduce an accurate cache merging strategy, employing a reparameterization of state vectors that decompose them into the magnitude and direction components, akin to weight normalization [21] . This approach allows for effective interpolation of the directional component in polar coordinates while preserving the original state norms to retain as much information as possible. This interpolation refers to the cross-layer merging as shown in the Figure 1(c ). Secondly, we recognize that a small subset of state pairs, characterized by low similarities but carrying largely distinct semantic meanings, are unsuitable for inter-layer merging. To address this, we propose a token retention strategy to minimize performance degradation, which involves separately retaining these outlier pairs. Our framework is notably memory-efficient, requiring storage for only a single high-dimensional directional component, along with minimal extra memory overhead. The overhead consists of a few unmergeable tokens and their corresponding indexes, as well as token-wise magnitudes to accurately restore the original states.We conduct extensive experiments with representative LLMs, including Mixtral-8x7B [22] , Phi-3-Mini [23] , and LLaMA-3 [6] 8B and 70B, respectively. Our method is benchmarked across a diverse range of question answering and generation datasets [24, 25, 26, 27, 28, 29, 30, 31] using the lm-eval-harness [32] . Additionally, we evaluate our results on LongBench [33] for long-sequence generation. The results demonstrate that MiniCache can reduce the memory footprint required for LLM inference by up to 41%, while simultaneously enhancing throughput by approximately 5× compared to fully cached baseline, clearly surpassing existing methods [11, 12, 14, 15] .Our contributions are summarized as follows:• We introduce MiniCache, a simple yet highly effective framework for KV cache compression.MiniCache pioneers the exploration of KV cache compression along the depth dimension, thereby significantly expanding its capabilities.• We observe a fascinating characteristic of cross-layer KV cache states: high similarity between adjacent layers in the middle to later stages of LLMs. Additionally, we find that not all state pairs are suitable for merging.• We propose an accurate and memory-efficient method for cross-layer cache merging, comprising a reparameterization strategy and a token retention mechanism. Our method complements existing KV cache compression approaches, further enhancing LLM serving efficiency.• Our MiniCache performs favourably against the state-of-the-art methods. Notably, our 4-bit MiniCache achieves a strong compression ratio of up to 5.02×, 5× higher inference throughput, and 41% memory reduction compared to the FP16 full cache baseline with near-lossless performance. 
related work: 
 Efficient inference for LLMs. Large Language Models (LLMs) are constrained by considerable computational and memory requirements during inference, particularly in resource-constrained environments. To mitigate these challenges, a variety of efficient inference techniques have been developed. For instance, dynamic inference methods [18, 34, 35, 36, 37, 38] , represented by mixtureof-experts (MoE) [39, 40, 41, 42, 43] , adaptively select specific sub-structures of the model during the inference process based on the input data, significantly improving inference efficiency while keeping model capacity. Techniques like Multi-Query Attention [44, 45] , Kernel-driven attentions [46, 47, 48, 49] , and low-rank attentions [43, 50, 51, 52] approximate the functionality of traditional attention mechanisms but with more efficient implementations. Quantization strategies [53, 54, 55, 56] involve converting the model's weights and activations into a low bit-width format, thereby reducing memory footprint and computational intensity. Sparsification approaches [14, 15, 57, 58] eliminate unnecessary elements in both model weights and token representations, further enhancing efficiency. Some closely related works, such as MoD [17] and LayerSkips [19] , considered the dynamic inference nature to ignore unimportant layers according to input [59] . However, these methods require an additional fine-tuning process or carefully designed pre-training stages, which reduces the adaptivity of these methods. MiniCache relies on inter-layer similarity observations to perform cross-layer merging, significantly reducing memory demand.Model merging. Merging compression involves the aggregation of a model's parameters and activations at various granularities. This process enhances the efficiency of inference in large models and facilitates huge redundancy [60] . Linear Mode Connectivity (LMC) [61] enables the fine-tuning of models from a shared pre-trained base. Commonly, weight averaging [62] is employed as an efficient technique to perform merge compression. Notably, Model Soup [63] utilizes linear averaging in this context. Advanced methods like TIES Merging [64] , Model Breadcrumbs [65] , and DARE [66] further enhance this process by sparsifying and combining model parameters, enabling the merging of models without sacrificing performance capabilities. Additionally, Spherical Linear intERPolation (SLERP) [67] extends beyond simple weight averaging by interpolating between model parameters. The Fisher information matrix [68] and RegMean-based methods [69] further optimize merges to produce ideal weights, minimizing the ℓ 2 distance to generation outputs while preserving the privacy of the training data. However, most existing works focus on merging model parameters, with the concept of depth-wise mergeability not being thoroughly explored in prior research. MiniCache focuses on the KV cache token merging in the depth dimensional of LLMs.Question: Does the paper discuss the limitations of the work performed by the authors?Answer: [Yes]Justification: Yes, we include the limitations in the Appendix.Guidelines:• The answer NA means that the paper has no limitation while the answer No means that the paper has limitations, but those are not discussed in the paper. • The authors are encouraged to create a separate "Limitations" section in their paper.• The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (e.g., independence assumptions, noiseless settings, model well-specification, asymptotic approximations only holding locally). The authors should reflect on how these assumptions might be violated in practice and what the implications would be.one good way to accomplish this, but reproducibility can also be provided via detailed instructions for how to replicate the results, access to a hosted model (e.g., in the case of a large language model), releasing of a model checkpoint, or other means that are appropriate to the research performed. • While NeurIPS does not require releasing code, the conference does require all submissions to provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution. , with an open-source dataset or instructions for how to construct the dataset). (d) We recognize that reproducibility may be tricky in some cases, in which case authors are welcome to describe the particular way they provide for reproducibility.In the case of closed-source models, it may be that access to the model is limited in some way (e.g., to registered users), but it should be possible for other researchers to have some path to reproducing or verifying the results. 
motivation: 
 In the below, we present our new observations in a novel cross-layer perspective. Prior studies have revealed the ineffectiveness of middle-to-deep layers in LLMs [20] . Thus, layerwise early exiting in this case can effectively avoid the redundant computation with minor effect on LLM performance [19, 70] . Inspired by this, we explore a layer-wise merging of KV cache in LLMs, starting with a simple baseline by averaging full tokens across adjacent layers. We provide our key observations as follows.Observation 1: KV cache shares a high similarity between adjacent layers. Based on LLaMA-3-70B [6] , we conduct zero-shot inference on the validation sets of three widely recognized benchmarks: COQA [71] , GSM8K [10] and TruthfulQA [72] . In general, we find that KV cache in the shallow layers exhibits low similarity, whereas those in the middle-to-deep layers are very similar to one another based on angular distance, as shown in Figure 1(a) . Next, we merge the KV cache across adjacent layers by conducting five-shot inference with LLaMA-2-7B, LLaMA-2-13B [5] , LLaMA-3-8B [6] , and Mixtral-8x7B [22] on GSM8K [10] . Specifically, starting from the middle layer of each model, we merge the KV cache in the adjacent two layers. As shown in Figure 2 (a), we observe a favourable performance for different LLMs, which reveals the huge potential for efficiency improvement by sharing the KV cache across adjacent layers during LLM decoding.Observation 2: Not all tokens are equally important to merge, a few distinct pairs require retention. Recent works [15, 16] in KV cache compression have found that keeping a few salient tokens at each layer, which contribute the majority of attention scores, could be sufficient to maintain LLM performance. In our case, we speculate that certain token pairs in adjacent layers also exhibit outlier behaviours, showing strong semantic differences that make them unsuitable for merging. Based on COQA [71] and LLaMA-2-7B [5] , we investigate the similarities at the level of token pairs. As shown in Figure 2 (b), we find a significant portion of token pairs share high similarities across adjacent layers. However, we observe a few outlier pairs, such as indices 0 and 15, with large margins of difference. We consider these tokens non-mergeable due to their significant differences. We also show that merging these distinct tokens results in performance degradation, corresponding to γ = 0 row in Table 2 . Thus, while cross-layer merging is a promising strategy for reducing memory burdens, it must be implemented with careful consideration of token-level similarities to ensure optimal performance, as shown in Figure 2(c ). 
method: 
 In this section, we introduce our MiniCache, a simple yet effective method aimed at trimming the KV cache redundancy in the depth dimension. This framework exploits the high similarity of KV cache states between adjacent layers and consists of two primary components: a reparameterization-based merging strategy and a token retention mechanism. The merging strategy compresses the KV cache states in adjacent layers to aggregate them into a single shared memory space, beginning from the middle of the model. The token retention mechanism mitigates information lost by retaining the highly distinct state pairs with minimal additional memory cost. With the merged cache, retention tokens, and magnitudes, we can accurately restore the original cache states for token decoding.Our method commences with the identification of an optimal starting layer S. Observations in Section 3.1 indicate that the KV cache from middle-to-deep layers consistently exhibits patterns of high similarity across adjacent layers. Consequently, we select the starting layer from the middle of the LLM, specifically S = L/2. From this layer onward, the KV pairs are assumed to be sufficiently similar across adjacent layers to warrant their consolidation. Central to this approach is a merge function, F , which is designed to integrate the KV caches of consecutive layers into a single, unified cache. We define x as the vectorized cache state of a single token, where the superscript indicates the layer index and the subscripts k and v denote the keys and values, respectively. Specifically, for a pair of key/value tokens at the same position in layers l and l -1, the merged cache is computed asEQUATIONThis consolidation process effectively eliminates the need to store and process the original memoryintensive keys and values in each layer independently. Instead, it approximates a shared cache across the adjacent layers.Reparameterization-based cache merging. To perform the pairwise merging, one solution is to directly average a pair of KV tokens, analogous to model merging [63, 64] . However, we observe that direct averaging can cause significant information loss. We conjecture that the distance between activations can be larger than that of weights due to the presence of outlier activation channels with extremely large magnitudes in LLMs [73, 74] , while weights typically have relatively quite small magnitudes. A potential method to compensate for this information loss is to project from c to x l-1 3). Additionally, we compute the ℓ 2 norm for the caches to obtain their magnitudes. Furthermore, we select unmergable tokens for retention, then store merged cache, retention tokens, and magnitudes at layer l in C. (b) illustrates the restoration process for layers l and l -1, which includes magnitude rescaling in Eq. ( 2) and retention token recovery. and x l , then rescale the projected vectors based on their relative magnitudes to exactly restore the original states. However, this approach requires extensive additional storage and computations; for example, restoring x l-1 needs both c and x l , which undermines the benefits of cache merging. To efficiently merge token pairs, we draw inspiration from weight normalization [21] , which disentangles model parameters into the magnitude and direction components to accelerate the convergence of stochastic gradient descent. Additionally, we take cues from DoRA [75] , which employs a similar way to resemble the learning behavior of parameter-efficient fine-tuning compared to full fine-tuning. In our case, the reparameterization can be formulated as follows:EQUATION)where e is the directional vector. This decomposition ensures that e l,l-1 ∥e l,l-1 ∥ is a unit vector, and allows the restored states to match the ℓ 2 norm of the original states, thereby preserving the cache's information as much as possible. The restoration is shown as Figure 3(b) . For brevity, we omit the subscripts k and v, as keys and values are decomposed in the same way. For estimating the directional component e l,l-1 , we follow SLERP [67] , which adaptively handles the interpolation, which often resembles rotation-like transformations. The choice of SLERP as the merging function is strategic, as it facilitates interpolation along the shortest path on the unit sphere between two high-dimensional vectors, thereby preserving their geometric integrity, which refers to merge operation in Figure 3(a) . This is crucial for maintaining the semantic and syntactic properties of the KV caches. The formula for SLERP in our context is:EQUATIONwhere Ω l,l-1 = arccosx l •x l-1∥x l ∥∥x l-1 ∥ represents the angle between vectors x l and x l-1 , and sin(•) is the sine function. t is an interpolation hyperparameter that adjusts the relative influence of each vector on the resulting direction, tailored to the layer depth and specific characteristics of the KV pairs. Note that when we set t = 0.5, it will become an average merging along the geometry surface, which we consider special cases in Eq. (A). The merged cache for each token pair is then a concatenation of the directional vector, magnitude and Ω l,l-1 , denoting as c l,l-1 = [e l,l-1 , ∥x l-1 ∥, ∥x l ∥, Ω l,l-1 ], cached components as shown in Figure 3(a) . Note that apart from storing the merged directional vector, we only need to store additional token-wise magnitude and angle scalars, which is memory efficient. In this way, we achieve substantial memory efficiencies through reduced redundancy while ensuring the retention of the critical functional characteristics of the original KV pairs across transformer layers.Unmergeable token retention. Highly distinct pairs are sensitive to merging operations, leading us to propose unmergeable token retention, as shown in Figure 3(a) . Despite the high similarity between KV cache states across neighbouring layers, a few sensitive distinct pairs remain that are significantly difficult to merge and share. Aligned with previous studies, these distinct tokens carry substantial semantic meanings [15, 16] . We observe that merging sensitive tokens, which results in the loss of layer-specific information, can lead to significant performance degradation. Therefore, it is crucial to properly disentangle the shared and unique information between adjacent layers. To address this issue, we designed a token retention strategy to selectively retain tokens that cannot be merged based on their angular distance, defined as: d(x l , x l-1 ) = 1 π Ω. For the KV caches, the minimum and maximum angular distances are determined to identify the unmergeable tokens.The set of required token indices to keep, I, is obtained by:EQUATIONwhere γ is a predefined hyperparameter that controls the retention threshold. The tokens with indices in I are retained and not compressed during the merge, which ensures that performance does not decline by preventing the loss of unmergeable tokens.Next, let X ∈ R n×h be either the key or value cache at one attention layer, where n denotes the number of tokens and h is the number of hidden dimensions, and E ∈ R n×h be the shared KV cache states. For each pair of neighbouring two layers, the unmergeable tokens are selected along with the token dimension byR l = X l [I], R l-1 = X l-1 [I], then restoring to our compressed caches byXl [I] = R l , Xl-1 [I] = R l-1, as shown in Figure 3(b) . Overall, we share the final cache for the two layers asC l,l-1 = [E l,l-1 , R l , R l-1 , ∥X l-1 ∥, ∥X l ∥, I].This cache includes the shared KV cache states, retention of unmerged tokens, magnitude vectors for each layer, and the token-keeping index, respectively. These additional components are quite lightweight. Thus compared to full-layer caches, our method remains memory-efficient, as discussed in Sec. 4.3.Cache restoration. After obtaining the shared cache C l,l-1 , we further need to approximately restore the original cache states for the current token decoding, as shown in Fig. 3 (b). Specifically, to restore X l , we first rescale the directional shared states with the corresponding magnitude vector along the token dimension, denoted as E l,l-1 ∥X l ∥. Subsequently, we perform retention token recovery by placing the sensitive tokens according to their token indices.Compression efficiency. We primarily analyze our memory efficiency in terms of the number of tokens used. Next, let r be the number of layers and and b is the batch size, s and n are input and output sequence length respectively. We consider the FP16 storage for KV cache. The full cache memory usage is given by 4brh(s + n). In our study, we begin merging layers from the middle to the deeper layers, consolidating the KV cache states of every two layers into a single shared state space. As a result, we effectively reduce the GPU memory usage in the decoding inference to 3brh(s + n), demonstrating a significant compression rate.Restoration efficiency. We then analyze the additional memory cost incurred during the restoration process, which During the magnitude rescaling phase, we save an additional norm vector for the corresponding layers in the KV cache. It is important to note that the norm vector is in the shape of R b×s×1 , which has a single channel dimension compared to the fully ranked original KV states. Additionally, we suppose that the retention threshold can be set to 0.05. Therefore, we have brh(0.05(s + n)) tokens retained without compression. Finally, our overall memory requirement is given by (3.1h + 2)br(s + n). The detailed derivation is shown in the Appendix E. 
experiments: 
 We demonstrated that our MiniCache can perform merging compression on the latter half of the layers of LLMs with minimal performance degradation.Implementation details. Our experiments are based on representative model families of LLMs, including a compact LLM Phi-3-Mini [23] and an MoE LLM Mixtral-8x7B [22] . Additionally, we adopt LLaMA-3 [6] 8B and 70B models to explore how our method generalizes to larger LLMs. We sample ten tasks from lm-eval-harness [32] , including COPA [24] , MathQA [25] , OpenBookQA [26] , PIQA [27] , RTE [28] , WinoGrande [29] , XSUM [30] , and CNN/Daily Mail [31] . We also evaluate long-sequence generation on LongBench [33] . We compare our method with a fully cached baseline, and other methods such as round-to-nearest quantization (RTN) [76] , SmoothQuant [73] and KIVI [11] .For the proposed MiniCache, we set the interpolation parameter t to 0.6, indicating that the merged results have a smaller rotation angle to the next layer. Furthermore, we set the token retention threshold γ to 0.05, according to the statistics of unmergeable tokens across multiple datasets. In addition to our merging method, we also consider a strong baseline of average merging. For sequential loading of large models, we utilize NVIDIA 4 A100 80GB GPUs, more details refers to Appendix D.Main results. We evaluate MiniCache by merging KV caches across all layers on GSM8K, COQA, and TruthfulQA. The results are shown in Figure 4 . In general, we demonstrate the general effectiveness of merging KV caches from middle-to-deep layers across different sized LLMs. Moreover, the proposed MiniCache demonstrates a consistent and significant advantage over the averaging baseline.We also illustrate the performance of merging KV caches across half of the layers with the blue lines, where MiniCache still maintains a robust performance and achieves the best compression ratio.Besides, we find that our method is even more effective for larger LLMs. For instance, based on LLaMA-3-70B, MiniCache shows nearly zero performance drop even with the KV cache in 87.5% of the layers merged on the COQA dataset. This highlights the adaptability and efficiency of our approach in handling large-scale models while ensuring minimal performance degradation.LongBench. We also conduct experiments to evaluate performance and quality in long-sequence generation using the LongBench dataset [33] , as shown in Table 1 . Our experiments applied MiniCache over several models: LLaMA-2-7B-Chat, LLaMA-2-13B-Chat, Mistral-7B, and Mistral-7B-Instruct.It is important to note that our MiniCache method maintains orthogonality with all existing quantization and sparsity (refers to Table A ) methods at both the model and token-wise levels. When combined with KIVI-4bit KV cache quantization, our approach achieves a compression rate of 5.02×, with minimal impact on accuracy across various challenging long-context generation tasks. The combination of MiniCache and KIVI-4bit KV cache quantization demonstrates significant memory savings without compromising the model's ability to handle long sequences effectively. This high- lights the potential of our method to optimize large language models for tasks requiring extensive context, making them more efficient and scalable for real-world applications.Efficiency analysis. To assess the acceleration capabilities of MiniCache, we conduct evaluations based on the methodologies employed in vLLM [77] and KIVI [11] . We generate synthetic workloads derived from ShareGPT, which include real input and output texts from LLM services. The dataset features an average input prompt length of 161 tokens and an output length of 338 tokens. Using the LLaMA-2-7B model on a single 80GB NVIDIA A100 GPU, we benchmark our method in a batch-serving scenario, comparing peak memory usage and throughput among 2-bit KIVI, 4-bit MiniCache, and an FP16 baseline. As illustrated in Figure 5 , with a batch size of 128, MiniCache reduces memory usage by 25GB, achieving a 41% memory saving. In terms of throughput, MiniCache outperforms the FP16 baseline by approximately 5×. Additionally, despite utilizing 4-bit quantization, MiniCache benefits from merging and sharing KV caches across adjacent layers, resulting in a 1.29× higher throughput compared to the 2-bit KIVI. These results demonstrate that MiniCache offers a state-of-the-art trade-off between efficiency and performance. The effect of interpretation parameter t. We explore the effects of the interpretation parameter t on performance, particularly in relation to the relative magnitude ratio of adjacent layers, as shown in Figure 6 . We maintain all settings constant, starting from layer S = 16 (halfway through the layers of LLaMA-3-8B), and vary the interpretation parameter t from 0.3 to 0.7. Our findings reveal several key points. When t = 0.5, the process resembles average merging, which is less effective for cross-layer merging. In contrast, when t = 0.6 is optimal, the merged representation exhibits the most robust performance, while indicating that more information is derived from the second term (x l ) of the SLERP.Question: Does the paper provide open access to the data and code, with sufficient instructions to faithfully reproduce the main experimental results, as described in supplemental material?Answer: [Yes]Justification: We provide code after acceptance.Guidelines:• The answer NA means that paper does not include experiments requiring code.• Please see the NeurIPS code and data submission guidelines (https://nips.cc/ public/guides/CodeSubmissionPolicy) for more details. • While we encourage the release of code and data, we understand that this might not be possible, so "No" is an acceptable answer. Papers cannot be rejected simply for not including code, unless this is central to the contribution (e.g., for a new open-source benchmark). • The instructions should contain the exact command and environment needed to run to reproduce the results. See the NeurIPS code and data submission guidelines (https: //nips.cc/public/guides/CodeSubmissionPolicy) for more details. • The authors should provide instructions on data access and preparation, including how to access the raw data, preprocessed data, intermediate data, and generated data, etc. • The authors should provide scripts to reproduce all experimental results for the new proposed method and baselines. If only a subset of experiments are reproducible, they should state which ones are omitted from the script and why. • At submission time, to preserve anonymity, the authors should release anonymized versions (if applicable). • Providing as much information as possible in supplemental material (appended to the paper) is recommended, but including URLs to data and code is permitted. 
ablation study: 
 The frequency results also indicate that the high frequencies are clustered around 0.4 and 0.6, corroborating our optimal t. Moreover, there is a strong correlation between the optimal t and the high frequency of the relative magnitude ratio of adjacent layers. This finding provides an opportunity to utilize the relative magnitude ratio to dynamically determine the interpretation parameter t. Dynamic t allows for more flexible weight control in SLERP merging for each layer-wise operation, thereby showing potential for further exploration.The effect of token retention threshold γ. We investigate the impact of the token retention threshold γ on model performance across the three datasets, as shown in Table 2 . A larger t generally means retaining more tokens for improved performance, but this comes at the cost of increased memory demand. The results suggest that setting γ to 0.05 achieves the best balance between performance and efficiency.Question: Does the paper specify all the training and test details (e.g., data splits, hyperparameters, how they were chosen, type of optimizer, etc.) necessary to understand the results?Answer: [Yes] Justification: We provide implementation details.Guidelines:• The answer NA means that the paper does not include experiments.• The experimental setting should be presented in the core of the paper to a level of detail that is necessary to appreciate the results and make sense of them. • The full details can be provided either with the code, in appendix, or as supplemental material. 
conclusion and future work: 
 This paper presents a pioneering exploration of KV cache compression in the depth dimension, addressing a significant memory bottleneck in LLMs. Our proposed MiniCache offers a simple, effective, and training-free approach to compressing KV caches by leveraging the notable high similarities between KV caches in neighboring layers, starting from the midpoint of LLMs. We have demonstrated that MiniCache can significantly reduce the memory footprint required for LLM inference by up to 41%, while simultaneously enhancing throughput by approximately five times compared to the FP16 baseline. In conclusion, MiniCache significantly advances the field of KV cache compression, offering a state-of-the-art balance between efficiency and performance. Future work will focus on enhancing the compression ratio by cross-multiple-layer merging, developing advanced merging algorithms such as Spherical Cubic Interpolation [78] , and further optimizing memory usage for large-scale deployments in diverse application scenarios.Question: Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?Answer: [No] Justification: We verify our results on huge size of language models, repeat computation are expensive.Guidelines:• The answer NA means that the paper does not include experiments.• The authors should answer "Yes" if the results are accompanied by error bars, confidence intervals, or statistical significance tests, at least for the experiments that support the main claims of the paper. • The factors of variability that the error bars are capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, or overall run with given experimental conditions). • The method for calculating the error bars should be explained (closed form formula, call to a library function, bootstrap, etc.) • The assumptions made should be given (e.g., Normally distributed errors). • It should be clear whether the error bar is the standard deviation or the standard error of the mean. • It is OK to report 1-sigma error bars, but one should state it. The authors should preferably report a 2-sigma error bar than state that they have a 96% CI, if the hypothesis of Normality of errors is not verified. • For asymmetric distributions, the authors should be careful not to show in tables or figures symmetric error bars that would yield results that are out of range (e.g. negative error rates). • If error bars are reported in tables or plots, The authors should explain in the text how they were calculated and reference the corresponding figures or tables in the text. 
acknowledgement: 
 This research is partially supported by the ARC Future Fellowship (FT190100039) to G.H. Additional support was partially provided by the DARPA Assured Neuro-Symbolic Learning and Reasoning (ANSR) program under award number FA8750-23-2-1016. Trade-off between accuracy and efficiency. We conducted experiments by varying the ratio of retention tokens, as shown in table C. We observe that accuracy improves up to a certain point before plateauing as the token retention ratio increases. Retaining 20% of the tokens is necessary to ensure all salient tokens are preserved without performance degradation. In contrast, we only need to retain the top 5% of the salient tokens using a dynamic distance-based threshold, as proposed in our paper. Note that efficiency decreases as more tokens are retained. This demonstrates that our distance-based approach better balances performance and efficiency than the fixed retention ratio counterpart.Ablation for relations between performance and ratio of merged tokens. We are adding another column in terms of compression ratio to represent the efficiency trade-offs, as shown in table D. The results suggest that setting γ to 0.05 achieves the best balance between performance and efficiency. Question: For each experiment, does the paper provide sufficient information on the computer resources (type of compute workers, memory, time of execution) needed to reproduce the experiments?Answer: [Yes] Justification: We give the competing requirements in the experiment setup.Guidelines:• The answer NA means that the paper does not include experiments.• The paper should indicate the type of compute workers CPU or GPU, internal cluster, or cloud provider, including relevant memory and storage. • The paper should provide the amount of compute required for each of the individual experimental runs as well as estimate the total compute. • The paper should disclose whether the full research project required more compute than the experiments reported in the paper (e.g., preliminary or failed experiments that didn't make it into the paper). 
code of ethics: 
 Question: Does the research conducted in the paper conform, in every respect, with the NeurIPS Code of Ethics https://neurips.cc/public/EthicsGuidelines?Answer: [Yes]  
 
 Appendix: Preliminaries of KV cache in LLMs. LLM inference has been significantly improved in recent works [9, 13, 77] by optimizing the KV cache management. Overall, this line of research is typically done in two steps: 1) First, at the prefilling stage, LLM calculates the initial KV caches at each attention layer based on the input sequence and decodes the first token. Second, 2) at the decoding stage, LLM autoregressively predicts the next token, where its KV cache at each attention layer is added to the overall caches. Existing works have compressed KV cache in different aspects (e.g., quantization [11, 12] , token pruning [14, 16] ).In the prior study, various strategies for enhancing efficient transformer architectures are discussed, covering a spectrum of techniques aimed at optimizing performance and managing resource constraints. These methods include attention optimization [48, 49, 80] , grouping queries [44, 45] , sparse KV caching [16, 81, 82] , shrinking tokens [15, 83] , and improving long-context generation. Significant contributions come from projects such as H2O [15] , GEAR [15] , and KIVI [11] . Additionally, efforts to alleviate KV cache bottlenecks include strategies like multi-query attention [44] and multi-group attention [45] , which propose reducing the number of heads in the KV cache. However, these methods often necessitate retraining or fine-tuning models.Other approaches focus on diminishing the size of the KV cache by selectively evicting less important tokens [14] and enhancing the system architecture through technologies like offloading the KV cache [84] or integrating techniques such as virtual memory and paging [85] into the attention mechanism. This function is designed to maintain the maximum norm of the vectors involved, ensuring that the most significant features are preserved during the merging process. The maximum norm-preserving interpolation in terms of F max can be defined as follows:F max (x l , x l-1 ) = xl,l-1 ∥ xl,l-1 ∥ • Max(∥x l ∥, ∥x l-1 ∥). (A)Here xl,l-1 represents the average vector between x l and x l-1 . The function F max ensures that the merged vector preserves the direction of the average vector while scaling it to the maximum norm of the original KV states. Compared to the SLERP-based merge function, F max has less computational overhead and lower memory consumption. However, it is less accurate than SLERP. The choice between using F SLERP or F max depends on the specific requirements of the application. In our study, we primarily use SLERP to maximize performance, as also shown in Table G .Societal impact. Our work shows a preliminary exploration of KV cache Compression in the depth dimension, a relatively unexplored yet critically bottlenecked area in large language models (LLMs). The proposed MiniCache provides a solution to improve the efficiency of LLM generation and is adaptable to existing intra-layer-based KV cache pruning and quantization technologies. Additionally, we proposed a simple yet effective approach that merges similar KV cache states in a cross-layer manner and effectively restores performance through a novel restoration technique. Our observations indicate that cross-layer similarity and mergeability can be applied in broad-efficient applications for post-training optimization in low-resource scenarios, such as deployment on mobile devices. Moreover, with effective optimization of KV cache memory demand, MiniCache further enhances long-context generation, which is a crucial paradigm for real-world applications, such as understanding concepts in textbooks. We aim for our work to advance the boundaries of two key challenges in the LLM industry and research: batch inference and long-context generation.Furthermore, in addition to the significant contributions of MiniCache for KV cache Compression, several challenges remain that are common to LLMs. Issues such as the truthfulness and security of LLMs are still unresolved. Ensuring the accuracy and reliability of generated content is critical, as LLMs can sometimes produce plausible but incorrect or misleading information. Additionally, safeguarding against security vulnerabilities, such as adversarial attacks or data leakage, is paramount to maintaining the integrity and confidentiality of user interactions. Addressing these challenges requires ongoing research and development to enhance the robustness and trustworthiness of LLMs. This effort must proceed alongside advancements in computational efficiency and performance, as exemplified by innovations like MiniCache.Limitations. The current merging algorithm based on Spherical Linear Interpolation (SLERP) has its limitations. SLERP is suitable only for merging two vectors and uses an interpolation approach that restricts our algorithm from merging multiple layers simultaneously and maximizing the compression ratio in further states. This limitation impacts the overall efficiency of KV cache compression and underscores the need for advanced techniques capable of handling more complex merging scenarios.Future research should focus on developing more sophisticated algorithms that can overcome these constraints, thereby enhancing the compression capabilities and overall performance of LLMs.Overview of the inference algorithm. The MiniCache inference implementation, as shown in Alg. 1, involves several key steps. When the interface starts, In the prefilling phase, we define the merging starting layer S. Before reaching layer S, the inference uses the original attention and cache logic. From layer S onward, we implement our merging algorithm, which operates in a cross-layer manner and is applied only at odd-numbered layers. During merging, we fetch the KV cache from the previous layer and save the merged shared states into the current layer's KV cache. To reduce memory usage, we then remove the cache of the previous layer. Additionally, we store the magnitudes vector and retention-sensitive tokens for each layer. During the decoding phase, two scenarios arise after layer S. For even-numbered layers (first round), since the KV cache has been removed during the prefilling phase, we refer to the next layer (l + 1) to fetch the shared KV cache states. We then perform approximated scale restoration and retention token recovery. The new KV states from this phase are stored for use in the next round. In the second round, which involves odd-numbered layers, we use the new KV tokens from both the previous and current layers. After the restoration phase, we perform the merge operations and update the shared KV cache states in the stack.Cross-Layer merging and restoration algorithm. As outlined in Alg. 2, MiniCache algorithm involves several crucial steps to ensure efficient memory usage while maintaining the integrity of the Key-Value (KV) Cache states. Initially, given the KV cacheE l k,v , norm values ∥X l k,v ∥ unmerged tokens R l k,v , retention indices I k,v, and the next tokens t l , t l-1 , the algorithm proceeds by rescaling the magnitude of the KV pairs. Specifically, Xl k and Xl v are computed by multiplying the normalized KV pairs E l k,v with their respective magnitude norms ∥X l k,v ∥. Following this, the algorithm restores unmerged tokens using the retention indices, updating Xl k and Xl v accordingly. Next, the new tokens t k and t v are concatenated to the rescaled KV pairs along the token dimension. This augmented KV cache undergoes a softmax attention mechanism where the attention scores A are computed by taking the dot product of the query token t q with the transposed keys ( Xl k ) ⊤ . The output token t O is then obtained by multiplying the attention scores A with the values Xl v . In cases where the previous token t l-1 exists, the algorithm performs a compression step. It concatenates the existing KV cache E l k,v with the merged tokens resulting from the current and previous layers, effectively reducing redundancy and optimizing memory. If t l-1 is not available, the KV cache is updated by simply concatenating E l k,v with the new tokens t l k,v , deferring the compression until the next iteration. The final output token t O is then returned, concluding the decoding process. In the merging function, the algorithm normalizes the KV pairs from both the current and previous layers. It calculates the angular distance Ω between the normalized vectors, ensuring that the interpolation occurs along the shortest path on the unit sphere. The merged KV cache is then obtained by weighted interpolation of the normalized vectors, preserving the geometric and semantic integrity of the original states. This comprehensive process allows MiniCache to achieve substantial memory efficiencies while maintaining the functional characteristics of the KV pairs across transformer layers.Figure A delineates the pre-filling and decoding logic for the MiniCache framework, which incorporates cross-layer merging and error suppression to achieve memory efficiency and maintain functional integrity. Initially, in Step 1, the KV cache is fetched from the previous layer (Layer L -1) during the pre-filling phase. In Step 2, the fetched KV pairs from the current layer χ L are merged with the KV pairs from the preceding layer χ L-1 , reducing redundancy through a merge function. Subsequently, in Step 3, the merged KV pairs are cached for future use, representing a consolidated data set from multiple layers. During the decoding phase, Step 4 involves deleting unnecessary or redundant KV pairs to optimize memory usage. In Step 5, the decoder fetches the required KV pairs from the cache for output generation. Step 6 applies error suppression mechanisms to the fetched KV pairs, including rescaling and retention recovery, to minimize errors introduced during the merging and compression processes. Finally, in Step 7, the cache is updated with the final KV pairs post-error suppression and adjustments, ensuring the cache contains the most accurate and efficient representation of the KV pairs for subsequent layers. This comprehensive approach guarantees substantial memory efficiencies while preserving the critical functional characteristics of the original KV pairs across transformer layers. In this section, we provide a detailed derivation of the memory efficiency improvements outlined in Section 4.3.First, we consider the original KV cache memory usage, which is given by:4brh(s + n).Here, r is the number of layers, b is the batch size, h is the hidden size, s is the input sequence length, and n is the output sequence length. To improve efficiency, we begin merging layers starting from the midpoint, S = 1 2 r, by consolidating the KV cache states of every two layers into a single shared state space. The memory usage derivation proceeds as follows: for the unmerged part of the cache (from layer 1 to S): Input: KV cache from current layer l: X l k,v ∈ R 2×tprompt×d , KV cache from previous layer l -1: X l-1 k,v ∈ R 2×tprompt×d , token retention threshold:γ 2 E l k , ∥X l k ∥, ∥X l-1 k ∥, Ω k ← Merge(X l k , X l-1 k ) ; 3 E l v , ∥X l v ∥, ∥X l-1 v ∥, Ω v ← Merge(X l v , X l-1 v ) ; 4 E l k,v ∈ R tprompt×d compression output , ∥X l,l-1 k,v ∥ ∈ R 4×tprompt×1 norms for rescaling ; 5 d(X l , X l-1 ) k,v = 1 π • Ω k,v // distance metrics 6 I k,v = {i | d i < d min + (d max -d min ) • γ} // retention indices 7 R l,l-1 k ← X l,l-1 k [I k ], R l,l-1 v ← X l,l-1 v [I v ] // unmerged tokens 8 return E l k,v , ∥X l,l-1 k,v ∥, R l,l-1 k , R l,l-1 v , I k,v9 procedure MiniCache Decoding: Input: KV cache: E l k,v ∈ R 2×tprompt×d , Norm: ∥X l k,v ∥ ∈ R 2×tprompt×1 , Unmerged Tokens: R l k,v ∈ R 2×γ•tprompt×d , Retention indices: I k,v ∈ R 2×γ•tprompt×1 , Next Token: t l ∈ R 1×d , t l-1 ∈ R 1×d 10 Xl k ← E l k • ∥X l k ∥ ∥E l k ∥ Xl v ← E l v • ∥X l v ∥ ∥E l v ∥ // magnitude rescale 11 Xl k [I k ] = R l k Xl v [I v ] = R l v // token restoration 12 Xl k ← Concat( Xl k , t k , dim=token) Xl v ← Concat( Xl v , t v , dim=token) A ← Softmax(t q • ( Xl k ) ⊤ ) t O ← A • Xl v if t l-1 exists then 13 KV cache ← Concat(E l k,v , Merge(t l k,v , t l-1 k,v ), dim=token) //⃗ X l ← X l ∥X l ∥ 19 ⃗ X l-1 ← X l-1 ∥X l-1 ∥ 20 Ω ← arccos X l T •X l-1 T ∥X l T ∥∥X l-1 T ∥ 21 E ← sin((1-t)Ω) sin(Ω) ⃗ X l + sin(tΩ) sin(Ω) ⃗ X l-1 22 return E, ∥X l ∥, ∥X l-1 ∥, Ω br(s + n) • 2 = 2br(s + n).We also introduce a retention threshold, which we set to 0.05. This means that 5% of the KV cache tokens are retained without compression: brh(0.05(s + n)).Combining these terms, the total additional memory for the restoration process is:2br(s + n) + 0.1brh(s + n).Finally, summing the compressed memory usage and the restoration memory cost, the overall memory requirement is:3brh(s + n) + 2br(s + n) + 0.1brh(s + n).This can be simplified by grouping the common factors:br(s + n) (3h + 2 + 0.1h) .Simplifying the expression inside the parentheses, we get:br(s + n) (3.1h + 2) .Therefore, the total memory cost for the KV cache in the MiniCache Framework is:br(s + n)(3.1h + 2).This detailed derivation confirms the efficiency improvements discussed in Section 4.3, highlighting the significant reduction in memory usage achieved through our layer merging and restoration strategies. Justification: We have properly made the main claims in the abstract and introduction.Guidelines:• The answer NA means that the abstract and introduction do not include the claims made in the paper. • The abstract and/or introduction should clearly state the claims made, including the contributions made in the paper and important assumptions and limitations. A No or NA answer to this question will not be perceived well by the reviewers. • The claims made should match theoretical and experimental results, and reflect how much the results can be expected to generalize to other settings. • It is fine to include aspirational goals as motivation as long as it is clear that these goals are not attained by the paper. 


Decision: The AC has carefully read all reviewers' comments and the authors' response. The overall sentiment to the paper is quite positive. The main reasons for accepting the paper are:
1) The novel approach -- KV cache compression along the depth dimension (3 out of 4 reviewers).
2) Insightful observations -- KV cache at certain layers exhibit high similarity (2 out of 4 reviewers).
3) Sound and easy-to-us methodology (3 out of 4 reviewers). 
4) Comprehensive evaluation -- across multiple models and benchmarks (2 out of 4 reviewers). 

The main concerns from the reviewers are:
1) Lack of justification of design choices, e.g., SLERP.  
2) Missing important baselines. 
3) Ad-hoc design, e.g., layer selection policy.

Some of the concerns were addressed via the rebuttal, such as (1) and (2). The other is indeed weakness but can be further improved via follow-up work. 

Overall, this is an interesting paper that explores a previous less looked dimension for compressing KV cache. It would be great if the authors can also add the actual memory usage, latency, throughput with batch size >1 comparison between MiniCache and alternative methods.

Reviews: review: paper_summary: paper_summary: This paper introduces MiniCache, a novel approach to compressing the Key-Value (KV) cache in large language models (LLMs) to enhance inference efficiency. The KV cache is crucial in storing key-value states of previously generated tokens, significantly reducing redundant computations and lowering latency during autoregressive generation. However, as the sequence length increases, the KV cache's size also grows linearly, leading to substantial memory consumption. MiniCache addresses this issue by compressing the KV cache across layers from a depth perspective, leveraging the observation that KV cache states exhibit high similarity between adjacent layers in the middle-to-deep portions of LLMs. The proposed method involves disentangling the states into magnitude and direction components, interpolating the directions while preserving the lengths, and retaining highly distinct state pairs unmerged to minimize information loss. MiniCache is a training-free, general approach that complements existing KV cache compression strategies like quantization and sparsity.

The authors conducted comprehensive evaluations using various models, including LLaMA-2, LLaMA-3, Phi-3, Mistral, and Mixtral, across multiple benchmarks. The results demonstrated that MiniCache achieves superior compression ratios and high throughput, with LLaMA-2-7B showing a compression ratio of up to 5.02×, a 5× increase in inference throughput, and a 41% reduction in memory footprint compared to the FP16 full cache baseline, all while maintaining near-lossless performance. The paper highlights the potential of MiniCache to significantly reduce memory requirements and enhance the efficiency of LLM inference, making it a promising solution for applications requiring long context inputs and extensive sequence generation.
strength_weakness: strength_weakness: 1. **No Implementation Source Code Provided**: The paper does not include the implementation source code, which is a significant limitation. Releasing the source code would facilitate further research and enable other researchers to replicate and build upon the work. Providing the code upon publication would enhance the paper's impact and encourage broader adoption of the proposed method.

2. **Insufficient Justification for SLERP**: The introduction of Spherical Linear Interpolation (SLERP) in the paper feels abrupt and lacks sufficient justification. While SLERP is used for interpolating between vectors, the paper does not provide enough rationale for why this specific technique was chosen over other interpolation methods. More ablation studies should be conducted to demonstrate the effectiveness and necessity of using SLERP in this context. These studies could compare SLERP with alternative interpolation techniques to show its advantages and validate the authors' choice.

### Minor Writing Improvements

1. **Figure Clarity**: 
   - **Figure 1(a)**: The resolution of Figure 1(a) could be higher, or the figure could be replaced with a vector graphic to improve clarity and readability. Enhancing the visual quality would make the figure easier to understand and more professional.

2. **Typographical Corrections**:
   - **Line 308**: The word "interpretation" should be corrected to "interpolation".
   - **Line 328**: The phrase "A larger t" should be corrected to "A larger \(\gamma\)".
questions: questions: 1. **Distance-Based Threshold for Retention (Line 226)**
   Why did you choose a distance-based threshold instead of the merging ratio of overall tokens as the control for retention? Can you show the effect on accuracy and efficiency as the ratio of retention tokens varies, highlighting this trade-off?

2. **Ratio of Merged Tokens in Ablation Study**
   In the second ablation study, can you show the ratio of merged tokens and the efficiency trade-offs for different settings?

3. **Implementation Source Code**
   Can you include the implementation source code?

4. **Justification for SLERP**
   Can you justify your choice of SLERP over other interpolation methods?
limitations: limitations: There should be a section discussing the limitation of this work and its social impact.
paper_summary: paper_summary: The paper proposes a novel method called MiniCache for compressing the KV cache in large language models (LLMs) by merging cache states across layers. The authors argue that this approach significantly reduces the memory footprint and enhances inference throughput without significant performance loss. The paper includes evaluations on various models and benchmarks, demonstrating the effectiveness of MiniCache in terms of compression ratio and throughput improvement.
strength_weakness: strength_weakness: **Major Weekness 1: Lack of Baseline Comparisons**   
In Table 1, the authors only include quantization methods for performance comparison, neglecting other KV cache eviction methods such as those proposed in [1] and [2].      

**Major Weekness 2: Lack of Evaluation for Instruction-following Benchmarks**   
Given that instruction-tuned models are more generalizable for downstream applications, it is essential to evaluate how MiniCache performs on instruction-following benchmarks such as MT-Bench [3].


[1] H2O: Heavy-Hitter Oracle for Efficient Generative Inference of Large Language Models, NeurIPS 2023   
[2] Efficient Streaming Language Models with Attention Sinks, ICLR 2024   
[3] Judging LLM-as-a-Judge with MT-Bench and Chatbot Arena, NeurIPS 2023
questions: questions: 1. Beyond memory compression, what is the impact of MiniCache on latency?
limitations: limitations: As mentioned above.
paper_summary: paper_summary: Authors proposed a KVCache Compression Method, Minicache, is introduced as a method to efficiently compress the Key-Value (KV) cache in large language models (LLMs) by leveraging the high similarity of KV states between adjacent layers in the middle-to-deep portion of LLMs. This compression is achieved by disentangling states into magnitude and direction components and interpolating directions while retaining distinct state pairs to minimize storage overhead. MiniCache is training-free, complements existing compression strategies, and demonstrates significant performance improvements, including up to a 5.02x compression ratio, a 5x enhancement in inference throughput, and a 41% reduction in memory footprint, all while maintaining near-lossless performance across various models and benchmarks.
strength_weakness: strength_weakness: 1.I am concerned about the computational overhead during the reparametrization and restoration stages. Does this compression strategy increase computational overhead?
2.The authors selected the middle layers for cross-layer merging compression. Have the authors considered any principled methods to further increase the compression ratio by compressing the shallow layers in future work?
questions: questions: See weaknesses.
limitations: limitations: The authors address the limitations inherited from the SLERP method and propose future research directions to enhance the adaptability of the merge compression strategy.
paper_summary: paper_summary: This paper proposes a KV cache compression method by merging keys and values of consecutive layers. Based on the empirical observation that keys and values of consecutive layers after the mid-depth layer have high cosine similarity, the authors propose a merging strategy using angular interpolation. Additionally, they emphasize the importance of retention tokens based on their empirical findings. Extensive experiments across multiple state-of-the-art models and various evaluation benchmarks demonstrate the effectiveness of the proposed compression method.
strength_weakness: strength_weakness: - Some methodological design choices seem ad-hoc. For example, the method starts merging after the mid-layer and only merges two consecutive layers. I believe the paper could be technically improved by addressing these design choices.
- The authors compare the efficiency of the method after applying quantization (MiniCache 4-bit) to the FP16 baseline. This comparison overstates the method's effectiveness because the contribution of the paper does not involve quantization. The paper should measure efficiency improvement without using quantization. Currently, lines L22-24, L302-303, etc., can mislead readers.
- A comparison to early layer-exiting approaches seems more relevant than comparisons to KV quantization methods, which the current draft focuses on.
   - e.g., "Fast and Robust Early-Exiting Framework for Autoregressive Language Models with Synchronized Parallel Decoding", EMNLP, 2023

**minor comments**  
- Figure 1, index -> layer index
- Line 140, Figure 1 (b) -> Figure 1 (a)
- For related works, it would be informative to include token compression methods: 
  - "Compressed Context Memory for Online Language Model Interaction", ICLR 2024
  - "Recurrent memory transformer", NeurIPS 2022
questions: questions: - What will be the results when applying the merging function to more than two layers?
- How about applying the merging function based on cosine similarity? For instance, only apply merging when the cosine similarity is higher than a specified threshold. This might address the issues mentioned in the first weakness above.
- Could you compare the proposed method with early layer-exiting methods?
- In Eq (4), is the retention index defined per layer? If not, how do you measure d_i, d_{min}, and d_{max}?
- Could you provide an intuitive analysis of why some tokens have severely low cosine similarity in Figure 2 (b)? Are there any common characteristics of these tokens?
- Do you apply merging per attention head or to the entire key/value vectors?
limitations: limitations: In appendix.

Comments: title: Thank you for the response
Thank you for the response
Thank you for the rebuttal
Follow-Up on Rebuttal
comment: I appreciate the authors' responses, and most of my concerns have been addressed. I will keep my evaluation for acceptance
Dear Reviewer GYW9,

Thank you for your valuable feedback and for raising the score. We are pleased that the new experiments and additional details have enhanced the clarity and positioning of our work. 

We understand the reviewer's concern regarding the criteria for layer selection. We will incorporate your suggestion and include a more detailed trade-off analysis based on the experimental results shown in Figure 4. This will also explore the potential for a more dynamic layer selection strategy in our future work.

Best regards, 
Authors of #1613
My concerns are fully addressed and I would like to keep my score.
Thank you for your detailed responses to my questions. I agree that the paper presents interesting observations. However, the approach still feels ad-hoc. It's unfortunate that no consideration was given to which layers should be merged. My suggestion in Q3 was to decide which layers to merge based on specific criteria. The authors only merge consecutive layer pairs after the mid-layer (please correct me if I’ve misunderstood). Regarding Q8, I believe a 1.5x improvement is indeed very significant, and I appreciate the authors for sharing this result. On the condition that the exaggerated claims in L21, 81, 94, etc. (regarding the proposed method achieving a 5x improvement) are revised accordingly, I raise my score to weak accept.
Dear Reviewer GYW9,

We would like to thank you for your valuable feedback to improve our work. We are wondering whether our response has addressed your questions and can improve your opinion of our work. We are committed to ensuring that our work meets the expectations of our esteemed reviewers.

Kindly let us know if you have any other concerns, and we will do our best to address them.

Best regards,
Authors of #1613