Keep the Cost Down: A Review on Methods to Optimize LLM’s KV-Cache Consumption

Authors: shi  luohe
zhang  hongyi
yao  yao
li  zuchao
zhao  hai
Venue: colmweb_org_COLM_2024_Conference
Published: 2024-03-22 08:43:34

Abstract: Large Language Models (LLMs), epitomized by ChatGPT's release in late 2022, have revolutionized various industries with their advanced language comprehension. However, their efficiency is challenged by the Transformer architecture's struggle with handling long texts. KV-Cache has emerged as a pivotal solution to this issue, converting the time complexity of token generation from quadratic to linear, albeit with increased GPU memory overhead proportional to conversation length.  With the development of the LLM community and academia, various KV-Cache compression methods have been proposed. In this review, we dissect the various properties of KV-Cache and elaborate on various methods currently used to optimize the KV-Cache space usage of LLMs. These methods span the pre-training phase, deployment phase, and inference phase, and we summarize the commonalities and differences among these methods. Additionally, we list some metrics for evaluating the long-text capabilities of large language models, from both efficiency and capability perspectives. Our review thus sheds light on the evolving landscape of LLM optimization, offering insights into future advancements in this dynamic field.

Main paper: 
 introduction: 
 Since the release of ChatGPT, Large Language Models (LLMs) are gradually having a profound impact on people's lives and are standing out in various fields (Wu et al., 2023; Roumeliotis & Tselikas, 2023b; a) . These LLMs face a computational challenge: their Decoder-Only Transformer architecture has a quadratic time complexity when processing text sequences. During inference, the auto-regressive decoding mechanism amplifies this issue, as it repeats the process for each token generated. KV Cache, by storing the keys and values tensor in attention module generated by past tokens, can reduce the time complexity required to generate each token to linear, greatly improving inference efficiency. However, the use of KV Cache is not all advantageous. KV Cache will increase linearly with the length of Layer specific evictionℎ 0 × ℎ 3 ℎ 2 ℎ 4 ×Head level eviction X X X 0 1 2 3 4 5 6 7 8 9Attention/Entropy score i as the hidden state of i-th token, after l-th layer (here we treat the embedding layer as the 0-th layer). At last, H (l) represents the matrix formed by concatenating n h (l) i vectors. Each Transformer Decoder Block consists of two parts: Self-Attention and Feed-Forward Network (FFN) . Each part has its own residual connection and Normalization (Norm) operation. In this paper, we only focus on the Self-Attention part. In the Decoder Block, each h (l) i is mapped to three new vectors q (l) i , k (l) i , and v (l) i using three trainable matrices W (l) q , W (l) k , paired with position embedding matrix R i , and W (l) v . In other words, H (l) is mapped to Q (l) , K (l) , and V (l) . Another matrix W (l) o is then used to map the output of the Self-Attention layer to the input h ′ (l) i for a single token, and H ′ (l) for the entire sequence. The process is given by Formula 1.H ′ (l) = softmax mask Q (l) • K (l) T √ d • V (l) • W (l) o (1)Modern LLMs utilize multi-head attention (MHA). The idea is to split q, k, and v into n h smaller blocks, named heads, each with d h = d/n h dimensions. For the j-th head of l-th layer, we denote {Q, K, V, W q , W k , W v , q i , k i , v i } (l,j) for the corresponding partition of these matrices or vectors. In MHA, Causal Mask mask(•) was used to prevent earlier tokens attend on later ones, by adding -inf to the upper right triangle of the pre-softmax attention matrix. This crucial property ensures that the K and V computed by preceding tokens will not be affected by subsequent tokens. Therefore, for each token newly added during auto-regression decoding, we only need to update previous K and V, as {K (l) , k l) , and perform Formula 2. The K and V which we retained are called KV Cache.(l) n } -→ K (l) and {V (l) , v (l) n } -→ V (h ′ n (l) = Concatenate n h j=1 softmax mask q (l,j) n • K (l,j) T √ d h • V (l,j) • W (l) o(2) o is not depicted here.EQUATION 
training stage optimization: 
 For LLMs that adopt the traditional Decoder-Only Transformer architecture, the most effective KV Cache compression method emerges during the pre-training phase. This is because, in this phase, the model possesses the greatest plasticity. The primary adjustment in this phase is to the model architecture, which, while still retaining the excellent properties of Attention, reduces the size of the generated Keys and Values vectors to a quarter or even less. Shazeer (2019) proposed Multi-Query Attention (MQA) based on Multi-Head Attention (MHA). Shazeer claimed that even if we retain only one head for keys and values, we can still achieve satisfactory model performance. In this case, different query heads calculate attention scores with the same key head. Although there is only one head left for value, it will receive n h different combination weights, resulting in n h combinations. Doing so can instantly optimize the KV Cache space usage to 1/n h of the original.Reducing the number of keys and values heads from n h to 1 is undoubtedly an aggressive strategy. Hence, Ainslie et al. (2023) proposed Grouped Query Attention (GQA), a method that can better balance speed and performance. In GQA, query heads are divided into n g groups. Each group shares one key head, serving as the n h /n g different combination weights for the corresponding value head. Ultimately, the KV Cache we need to store will be reduced to n g /n h . Compared to MHA and MQA, GQA introduces an adjustable parameter n g . When n g = 1, we get MQA; when n g = n h , we get MHA. When n g is between 1 and n h , efficiency and performance achieve a more granular balance. Moreover, GQA can save a substantial number of parameters within the Attention module, with the ratio of savings η = 0.5 + 0.5 (2024) introduce more significant changes: they generate the all layers' KV Cache using linear models, like RetNet (Sun et al., 2023a) and RWKV (Peng et al., 2023a) . However, it's essential to note that these methods do not optimize the memory bandwidth bottleneck of the KV Cache. Cross-layer reuse results in scattered access to the same KV Cache, and only meticulously crafted CUDA kernels can effectively reduce data exchanges between different cache levels, if such reduction is possible at all.We can consider reuse as a form of low-rank compression, and complete low-rank compression should have even greater potential. The method proposed by DeepSeek-AI et al. ( 2024), called Multi-Head Latent Attention (MLA), exemplifies this. MLA achieves KV Cache decompression by utilizing an additional dimension-expanding matrix, rather than merely reusing the existing content. However, it's essential to note that this method still does not address the memory bandwidth bottleneck, as the cache still needs to be expanded into large KV vectors. Yen et al. (2024) proposed a new idea for the LLM architecture. CEPE, a framework that combines the pre-trained LLM with an Encoder module that serves as context compressor.In RAG scenario, the ultra-long context will greatly increase the need to store KV Cache. CEPE's additional Encoder compresses these reference texts, and then inputs them into the Decoder through cross-attention. The compressed text has a shorter sequence length than before, so the KV Cache is saved. This method can use the pre-trained LLM, but the added decoder and cross-attention layer still require a lot of extra training. 
deploy-stage optimization: 
 From the perspective of inference systems, KV Cache presents significant challenges in two key areas. First, during the continuous {K, k n } -→ K and {V, v n } -→ V operations, memory is repeatedly allocated and released, resulting in substantial fragmentation. Second, the inability to batch-process KV Cache exacerbates severe memory bandwidth bottlenecks, leading to computational inefficiency. Addressing these two issues has become a central concern for inference systems. Kwon et al. (2023) introduced the Paged Attention mechanism and the vLLM framework. Paged Attention draws on the page memory mechanism widely used in CPU memory, and uses an additional mapping table to map the KV Cache that used to be stored continuously to discontinuous GPU memory. During inference, in order to generate the next token, it is necessary to call KV Cache for calculation, and this process can also be efficiently completed through a set of custom CUDA kernels. The use of the Paged Attention mechanism results in almost no unused memory fragments and efficient inference. 
eviction and merging: 
 Eviction methods are about the policies to discard unnecessary tokens. Two lines of approaches exist: static policies, which are designed before inference and remains consistent across every inference request, and dynamic policies, which utilize the information generated while inference to identify important tokens.Static policies A straightforward approach for maintaining a fixed-size KV Cache is to retain recent tokens, known as slide window attention (Beltagy et al., 2020) . However, the model will collapse when sequence length exceeds cache size (Xiao et al., 2023) . Recently, Xiao et al. (2023) and Han et al. (2024) proposed that initial tokens almost consistently receive high attention weights across layers and heads. As a result, keeping KV Cache of both initial and recent tokens can maintain model performance for long contexts.Dynamic policies Policies based on attention weights have been widely explored by researchers, as they believe that attention weights can provide insights into the importance of individual tokens. This can be leveraged in the design of dynamic strategies for eviction. Since it is impossible to predict which token will be necessary for future generation, the question arises whether it is possible to estimate the importance of a token based on history information.Liu et al. ( 2023) empirically finds existence of Repetitive Attention Pattern. This pattern suggests that for two different tokens, there are similarities in what they are attending to and ignoring. So a natural hypothesis is that only the tokens which are important in previous steps will be significant in the future.This provides us with the possibilities to estimate future significance of a token based on history information of attention weights. Token Omission Via Attention (TOVA) (Oren et al., 2024) proposes a simple greedy method to discard useless tokens. While keeping a fixed-length of KV-Cahce, TOVA evicts at each decoding step the tokens with minimal attention weights layer-wise. (Zhang et al., 2023) uses accumulative normalized attention scores to decide which token to stay and at the same time keeps the recent tokens since they may show strong correlations with current tokens.PyramidInfer (Yang et al., 2024a ) uses a layer-wise approach with recent tokens occupying more weights. Apart from this, Yang et al. (2024a) believes that deeper layer contains more context redundancy, so it defines a decay ratio to shorten the length of KV Cache in deeper layers and thus forming a "pyramid".However, while discarding unnecessary tokens seems to have minor influence on the original inference process, Adnan et al. (2024) observes that when evicting more tokens during the inference process, the distribution of normalized attention scores becomes uneven among the remaining ones. Therefore, Keyformer( Adnan et al., 2024) introduces additional methods to smooth the uneven distribution and approximate the original one when using full KV Cache.FastGen (Ge et al., 2023) uses a hybrid strategy to optimize token selection for discarding. In prompt encoding phase, it selects the best policy for each head in KV Cache, and then uses these policies to decide which token to discard in decoding process. The policies include: a) keeping special tokens, b) keeping punctuation tokens, c) keeping recent tokens and d) keeping tokens with attention-weight-based policies.SparQ Attention (Ribar et al., 2023) tries a different approach. Instead of reducing memory capacity, it aims at reducing the amount of data transferred. It first uses the norm of q The darker the color, the higher the token attention weight.Dynamic Policies While attention scores provide ample information, they are incompatible with certain optimization methods, such as Flash Attention (Shah et al., 2024) . Therefore, investigating how to evict tokens without relying on attention scores is a novel research topic. Several approaches address this challenge. Tang et al. (2024) leverages the prior-known properties of attention heads to determine whether to evict. Yao et al. (2024) uses token entropy to decide whether to remove tokens. Devoto et al. (2024) relies on the L 2 norm of keys to determine token eviction. All these methods can be combined with Flash Attention. Notably, Devoto et al. ( 2024)'s approach demonstrates excellent granularity and controllability.Merging, as a relatively recent method, represents a natural extension of Eviction, transitioning from hardmax to softmax. The current key challenges in this area include determining the merging region and selecting an appropriate scheme for combining keys and values. Another commonly utilized method for compressing Key-Value (KV) cache is through quantization. This approach effectively compresses data by mapping tensor values, originally in full precision, to discrete levels and storing them at a reduced precision. There are two primary categories of KV Cache quantization: Full quantization and KV Cache-only quantization. Full quantization involves compressing both the model weights and the KV Cache, thereby reducing the memory footprint of the entire model. On the other hand, KV Cache-only quantization specifically targets the KV Cache activations, selectively compressing them to conserve memory while leaving the model weights in their original state. This tailored approach can offer a balance between model efficiency and performance. Inspired by recurrent networks, LESS utilizes a constant-sized low-rank cache by replacing the softmax with a separable similarity metric calculated by learnable row-wise functions. By doing so, LESS obtains the ability to accumulate history token information before they are discarded from KV Cache, allowing for continued access to previous information. Yang et al. (2024b) proposd Mixed-precision KV Cache (MiKV), a reliable cache compression method that retains evicted KV pairs in reduced precision to preserve information and maintains important KV pairs in higher precision to ensure generation quality. MiKV employed Dynamic Outlier Awareness to dynamically balance the outliers manifested in the query and keys to reduce quantization error. Dynamic Outlier Awareness multiplies and divides a channel balancer to the keys and queries to mitigate the impact of outliers. Dong et al. (2024b) proposed the Quality Adaptive Quantization (QAQ). QAQ utilized separate quantization strategies for key caches and value caches, ensuring that the key cache, which is more sensitive, is quantized in a way that less affects model performance. QAQ also uses an attention window to predict future attention scores based on the history of attention values. This ensures that the quantization does not overly compress tokens that may become more important in subsequent generation steps. Kang et al. (2024) proposed GEAR (Generative Inference with Approximation Error Reduction) which integrates three techniques: (1) Uniform quantization for the majority of entries.(2) Low-rank matrix approximation for quantization residuals.(3) Sparse matrix to handle errors from outlier entries.Full quantization Sheng et al. (2023) proposed FlexGen which compresses both model weights and attention cache to 4 bits without significant accuracy loss. Yue et al. (2024) took FlexGen a step further and proposed WKVQuant. For weight quantization, WKVQuant ultize the OmniQuant (Shao et al., 2023) . For KV Cache quantization, WKVQuant employs following three strategies: Past Only Quantization, Two-dimensional Quantization and Cross-block Reconstruction Regularization. Different from KVQuant (Hooper et al., 2024) which focus on calibrating for per-channel(token) thresholds to ignore outlier, WKVQuant focuses on aligning and smoothing each channel and token. 
evaluation: 
 In this section, we first introduce the datasets commonly used in KV Cache optimization, followed by a presentation of frequently used evaluation metrics.Long context benches Currently, there are several benchmarks that rely on extracting information from extremely long texts to answer questions. These benchmarks typically require models to read a substantial reference document, often containing 4,000 to 20,000 tokens, and then respond to questions. In some cases, the longest texts may even approach 200,000 tokens. Typical examples are LongBench (Bai et al., 2023) , ZeroSCROLLS (Shaham et al., 2023) , L-Eval (An et al., 2023) , BAMBOO (Dong et al., 2024c) , XL 2 bench( Ni et al., 2024) , InfiniteBench (Zhang et al., 2024b) , and LooGLE (Li et al., 2023) . However, these benchmarks face challenges related to model-specific knowledge requirements, fixed text lengths that are difficult to adjust, and the uncontrollable and subjective nature of content generated by the models.The other method for evaluating the ability of models to handle extremely long texts involves directly extracting a specified key from unrelated context. This approach is simpler, more flexible, and more controllable. Importantly, it does not rely on the model's existing knowledge. Examples are Needle in a Haystack (Kuratov et al., 2024) , Passkey Retrieval (Mohtashami & Jaggi, 2023) , and RULER (Hsieh et al., 2024) . However, critics argue that this method is overly simplistic. Examples of these tasks can be found in Appendix B Figure 4 and Figure 5 .In addition to datasets specifically designed for long texts, the length of traditional shorter test sets can be extended through a few-shot (Brown et al., 2020) format or by simulating multi-turn dialogues, in order to test the model's capabilities with long texts. However, in long-text tests based on few-shot, texts at a greater distance only serve as a reference and do not contain information that the model needs to extract or conditions for inference, so the reference value of the results is limited.Per Token GPU-Memory Usage For KV Cache, the most intuitive optimization indicator would be the memory space occupied by each token. The LLaMA2-7B model, as a typical example, theoretically occupies 0.5MB of memory for each KV Cache entry. Please note that when measuring this indicator rigorously, the fragment space generated by the token should also be taken into account, that is, it is best to measure according to the actual memory occupancy, rather than a value calculated through a string of parameters.Throughput and Latency Throughput and latency are important indicators for measuring the time efficiency of a model. Throughput, usually measured in tokens per second (token/s), represents how many new tokens the model can generate per second. The higher the throughput, the better the model efficiency. In the Decoding phase, latency is usually considered to be the time required to generate each new token, which is the reciprocal of the throughput, typically in milliseconds.The Perplexity (PPL) is: for each token, the model calculates the natural logarithm likelihood value of the probability distribution predicted based on its previous tokens, takes the average, and then calculates the value as the exponent of e, mathematically given by Formula 3, where ANLL refers to the average natural logarithm likelihood.EQUATIONPPL can provide a rough reference for the performance changes of the model. If PPL rises sharply, it usually means that the model's ability has significantly decreased, such as completely losing language ability, etc. 
key takeaways: 
 This review, following the footsteps of prior work, takes a closer look into KV Cache optimization, uncovering its complexities and proposing strategies for its optimization. We hope to propose some insights and suggested directions for future research that are not just reflections of current trends but also an invitation to explore the uncharted territory of LLMs.Principles of KV Cache Optimization: At the core of optimizing KV Cache lies the principle of reducing memory consumption. This can be achieved by further compressing 'K' (Keys) or 'V' (Values) in the KV pairs. Techniques to compress these components directly impact the efficiency of the models, especially in terms of memory usage and processing speed.Trade-offs in Deletion vs. Compression: Whether to delete less important KV pairs to save memory or to compress the KV Cache without deletion remains an open question.While deletion might offer immediate memory relief, it could potentially compromise the model's performance. In contrast, better compression techniques strive to retain information integrity while reducing memory footprint.A more radical approach could involve storing the KV Cache externally, possibly on a different storage medium. This method would transform KV Cache management into a retrieval challenge, where the relevant KV pairs are fetched and reintegrated into the model as needed. While this could reduce memory usage on primary devices, it would introduce complexities in retrieval and integration processes.Future Directions in Storage and Retrieval Technologies: These discussions point towards an evolving future where storage and retrieval technologies might become as crucial as the computational models themselves. Innovations in how KV Cache is managed, stored, and accessed could open up new avenues for making LLMs more efficient and versatile. 
conclusion: 
 In this review, we highlighted the significance and the multifaceted nature of optimizing KV Cache in Large Language Models (LLMs). This review traversed through a variety of methods, from the training and deployment stages to post-training optimizations. Each method, whether it be the architectural changes in the pre-training phase, framework optimizations during deployment, or dynamic strategies like eviction and quantization in the post-training phase, offers unique insights into mitigating the challenges posed by the KV Cache's memory-intensive nature. Moreover, the exploration of different evaluation metrics for long-text performance underpins the importance of a balanced approach that considers both efficiency and model capabilities. As the field continues to evolve, it is clear that the optimization of KV Cache will remain a critical area of focus, offering a pathway towards more efficient and environmentally responsible use of LLMs. We hope this review can be a roadmap to guide learners venturing through this dynamic and rapidly progressing field. 
 
 Appendix: Mary moved to the bathroom. John went to the hallwayWhere is Mary?Senjō no Valkyria 3 : Unrecorded Chronicles ( Japanese : 戦場のヴァルキュリア3, lit. Mary moved to the bathroom.Valkyria of the Battlefield 3 ), commonly referred to as Valkyria Chronicles III outside Japan, is a tactical role @-@ playing video game developed by Sega and Media.Vision for the PlayStation Portable.Released in January 2011 in Japan, it is the third game in the Valkyria series.Employing the same fusion of tactical and real @-@ time gameplay as its predecessors, the story runs parallel to the first game and follows the " Nameless ", a penal military unit serving the nation of Gallia during the Second Europan War who perform secret black operations and are pitted against the Imperial unit " Calamaty Raven ". … Due to low sales of Valkyria Chronicles II, Valkyria Chronicles III was not localized, but a fan translation compatible with the game's expanded edition was released in 2014.Media.Vision would return to the franchise with the development of Valkyria : Azure Revolution for the PlayStation 4. John went to the hallway. As with previous Valkyira Chronicles games, Valkyria Chronicles III is a tactical role @-@ playing game where players take control of a military unit and take part in missions against enemy forces.Stories are told through comic book @-@ like panels with animated character portraits, … In addition to the above methods, there are also some methods that modify the Attention mechanism to completely eliminate KV Cache. Furthermore, by directly compressing the prompt or the embedding, the length of KV Cache can also be reduced by decreasing the number of tokens. 2023) applies a special attention mask. The gist token is inserted between instructions and inputs, acts as a bridge for transferring instruction information to inputs and thus forcing gist token to compress information.LLMLingua (Jiang et al., 2023a ) uses a budget controller, an iterative token-level compression algorithm and a small model to estimate which token is important. LongLLMLingua (Jiang et al., 2023b) further improves LLMLingua in long context scenarios.Lin et al. (2024) further developed the idea of Paged Attention. Through the DistAttention it proposed and the DistKV-LLM built on it, KV Cache was able to achieve distributed deployment on multiple servers. This significantly improved the efficiency of providing LLM services using large-scale cloud servers.Ye et al. (2024) aims to reuse KV Cache between different dialogues, achieving acceleration of the pre-fill stage and optimization of GPU memory occupancy. By establishing a dictionary tree for all historical dialogues, and finding the longest common prefix in the dictionary tree when a new dialogue request is received and reusing the corresponding KV Cache for this part, the ChunkAttention made the model avoid repeated calculation of some tokens in the pre-fill stage, speeding up the response speed of the deployment system. Moreover,Gao et al. (2024) extend this idea by introducing a hierarchical system that utilizes multiple storage devices.Jin et al. (2024) andLee et al. (2024) offload KV Cache to CPU with speculation. The core idea is to offload the primary portion of the KV Cache to the CPU, retaining only critical components. During inference, by speculatively reloading a small amount of KV Cache onto the GPU using the limited preserved information, we can save memory space on high-speed computational devices without compromising performance, all while avoiding extensive data exchanges.He & Zhai (2024) introduces a novel approach: by collaboratively utilizing the CPU for attention computations, we can enhance GPU utilization efficiency, improve overall effective computational power, and accelerate inference speed.Additionally,Qin et al. (2024) provides a comprehensive technical report from an enterprise perspective, optimizing the entire inference system. 


Decision: The reviewers agree this paper is on a timely topic and seems to be very thorough, especially for a conference paper. The authors have promised to add some additional discussion and synthesize their different threads more closely, including adding a table to compare the methods in question. If they make these changes, the paper is clearly strong enough to accept.

[At least one review was discounted during the decision process due to quality]

Reviews: review: paper_summary: paper_summary: This paper presents a review of KV-caching techniques for Decoder-only LLMs, describing methods and strategies pertaining to multiple steps along the life of a model, as well as briefly discussing evaluation metrics and proposing avenues for future work. Overall, the work is timely and necessary, and there will definitely be benefits for researchers new to this topic who choose to read it instead of (or before) diving into the growing literature.
strength_weakness: strength_weakness: * The non-core sections are substantially weaker. "Evaluation" (S.6) is mostly unnecessary and the space used for it could have been better spent on covering more methods, extending the descriptions for some, or integrating interesting bits from Appendices, mostly C. Not only is S6 describing at length well-known resources, it does not connect back to any of the methods described in the core sections, and the results table isn't referred to, let alone analyzed. This is followed by a "take-aways" section (S7) which does not follow comfortably from the preceding ones. Overall, there's a noticeable disconnect at the 5-6 section border.
* Section 2 (Preliminaries and Notation(s)) is a good idea but not executed carefully enough (see below).
* While some illustrations (Fig. 2,3) are useful and illustratory, Figure 1 is mostly confusing and inconsistent. It does not lend intuition of the topic or set expectations for the survey's contents.
questions: questions: ### Questions
* In S2, does $p_i$ denote a distribution, as suggested twice, or a single probability, as suggested once and inferred by convention?
* In S3, how do $n_h$ combination weights result in $n_h$ combinations? It feels like an entire dimension is absent in the former.
* In S3, setting $n_g$ to $n_h$ in the $\eta$ equation results in 1.5, which cannot be true (it needs to be 1). What's the correct formula?

### Required fixes
* Please refrain from personifying models - "understanding" in line 4 of the intro should be changed. Especially since this is a didactic work which may be used in introductory classes.

### Grammar and Suggestions
* There are some foci of grammatical errors - four in the last three lines of the first intro paragraph, the caption for figure 2, "utilize(s)"in p.5, "quer(y/ies)" in p.6. The last sentence of the intro is not understandable.
* Appendices A and B look useful, you should probably alert the reader to their existence in the main text.
* The transpose notation in eq. (1) and (2) is not the standard "top" but a cross, please change.
* Please don't call things "horrible" (top of S4). "undesirable" would be more suitable.
* In p.5, *Static policies* paragraph, it seems like Xiao et al. and Han et al. *find* rather than *suggest* (unless they really only suggest, in which case somebody better run that analysis).
* Last but not least. "(the) KV cache" is a noun phrase and shouldn't be hyphenated unless used as a modifier.
paper_summary: paper_summary: This submission is a review of the compression methods of KV Cache. It organizes existing works in chronological order, including the training phase, deployment phase, and run-time phase with post-training compression. The training stage mainly involves some novel architecture designs, such as Multi-Query Attention (MQA), and Grouped-Query Attention (GQA). For the deployment-related techniques, this paper covers some system optimization like Paged Attention and DistKD-LLM. For the post-training optimization, several training-free methods are discussed, including Window Attention and Attention Sink. Besides, some pruning, and quantization methods are mentioned. Further, this paper summarises the datasets and evaluation metrics for the KV cache.
strength_weakness: strength_weakness: 1. The primary reason for rejection is the lack of connections between different methods within the paper. While the taxonomy is structured around different stages, such as pre-training, deployment, and run-time optimization, there is a missed opportunity to highlight similarities and shared concepts among various methods. For instance, the Attention Sink and Repetitive Attention Pattern both emphasize the importance of focusing on specific tokens, yet this connection is not adequately explored or summarized in the review. As a review paper, it's crucial to provide a synthesis of such related ideas to enhance clarity and coherence.
2. Despite claiming to introduce metrics for assessing LLMs' performance on long texts, the evaluation section falls short by only listing general and widely used metrics like memory usage, throughput, latency, and perplexity. The paper lacks a comprehensive assessment of different methods, failing to delve into specific metrics tailored to the nuances of long text processing. A more thorough evaluation framework is needed to effectively gauge the effectiveness of the reviewed methods and provide valuable insights for future research and development in this domain.
questions: questions: Please refer to the weaknesses. My main concern lies in the lack of in-depth analysis of similar methods.
paper_summary: paper_summary: This paper provides a review of KV-cache optimization for LLMs. The reviews are grouped according to the stage when cache optimization is performed: training, deployment or post-training. Training-stage optimization mainly changes the model's architecture for efficient caching, deployment-stage optimization mainly studies better inference system, and post-training optimization mainly perform token eviction or quantization. Moreover, related datasets and evaluation metrics are also discussed.
strength_weakness: strength_weakness: - This paper is mainly a literature review without empirical experiments exploring which methods are most effective in which scenarios.
- It would be nice to provide a table listing the advantages and disadvantages of different methods with suggestions on their best usage.
- Prompt compression techniques are quite related to this topic and related work should be discussed.
paper_summary: paper_summary: This paper performs a literature review on optimizing KV-cache usage in LLMs. The paper is well written and ontology on classifying the current research and engineering effort is always welcome. While the topic is an important and active research area, I feel that the paper could have covered a rather deep review and discussion on the topic. The authors didn't discuss financial implications of different approaches while the title says 'Keep the Cost Down'.
strength_weakness: strength_weakness: In depth, discussion and review is missing since it is a survey paper. Work could be supplemented by experimental benchmarks.
paper_summary: paper_summary: The manuscript presents an up-to-date overview of approaches for optimizing the KV-Cache, which is a method for speeding up, and reducing the memory load for LLMs.
strength_weakness: strength_weakness: The papers is in some places too condensed.
questions: questions: Some acronyms are not explained: GQA (page 3), RAG (page 4), please consider explaining.

Page 4: "but each time it only increases a little, which is a horrible property" - the 'horrible' epithet is not helpful. The case is not 'horrible' but surely can be made more efficient. 

Page 7, a typo: "obtains the ability to accumulates history" -- should be 'accumulate'

Appendix A: "Mixture of Experts model" - please explain what is that.

Appendix C 'Other methods' deserves to be a section in the body of the paper, and deserves more elaboration. If you get more pages for the paper, consider moving this section. 

The following papers in the References section have no venue/publication data: 
Arora et al., Beltagy et al., Han et al. 
Please check completeness of the references.

Comments: comment: Thanks for your response and please update the paper in later versions accordingly.
Thank you for the responses. If the manuscript is accepted, adjust the clarifications in the text.
Thanks for taking extra effort for the Needle-in-haystack eval! It is nice have such eval and is useful to the community. I am willing to increase my score to 5.