Title: CAOTE: Efficient Caching through Attention Output based Token Eviction
Venue: NeurIPS 2025
Accepted: no
Keywords: token eviction

LLMs

attention output

constraint KV cache generation
Abstract: While long context support of large language models has extended their abilities, it also incurs challenges in memory and compute which becomes crucial bottlenecks in resource-restricted devices. 
Token eviction, a widely adopted post-training methodology designed to alleviate the bottlenecks by evicting less important tokens from the cache, typically uses attention scores as proxy metrics for token importance.
However, one major limitation of attention score as a token-wise importance metrics is that it lacks the information about contribution of tokens to the attention output.
In this paper, we propose a simple eviction criterion based on the contribution of cached tokens to attention outputs. Our method, CAOTE, optimizes for error due to token eviction, by seamlessly integrating attention scores and value vectors. This is the first method to use information from the value vector on top of attention-based eviction scores. Additionally, CAOTE can act as a meta-heuristic method with flexible usage with any token eviction method. 
We show that CAOTE, when combined with state-of-the-art attention score-based methods, always improves accuracies on the downstream task for $L{\small LAMA}$3 and $Q{\small WEN}$2.5 model families, indicating the importance of leveraging information from values during token eviction process.


Peer reviews:
{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe paper introduces CAOTE, a KV cache eviction method that based on the L2 difference of attention output which is different from previous methods that only considered attention scores. Additionally, the paper also proposes a faster version which directly use the average of the value vectors rather than the weighted version. Both CAOTE and its faster variant can be applied directly on top of existing KV Cache eviction method. The paper further shows the empirical evaluation results on Llama 3.1-8B-Instruct, Llama 3.2-3B-Instruct and, Qwen 2.5-3B-Instruct, Qwen 2.5-7B-Instruct over multiple language modeling tasks to show the effectiveness of their method.\n\nSTRENGTHS AND WEAKNESSES:\nStrengths\n\n- The sparse attention problem is important in the long context or long chain-of-thought reasoning setup. \n- The empirical evaluations are comprehensive with multiple models and datasets. \n\nWeaknesses:\n\n- The evaluation section lacks lots of important details: see the question section.\n- There is at least one prior work [1] as far as I am aware of that also focus on attention output deviation based KV cache eviction, which is not covered by this work.\n- Pure KV cache eviction based methods still have a large performance gap compared to the dense baseline and the improvements provided by the proposed method is mostly marginal (or even worse as in Table 2,3) with respect to the baseline cache eviction algorithm like TOVA or SnapKV. And the retrieval accuracy is far from ideal for the relatively easier needle-in-the-haystack task. The accuracy drop might be more significant in more challenging tasks like RULER or reasoning tasks. \n\nReference:\n\n1. Feng, Yuan, et al. \"Identify Critical KV Cache in LLM Inference from an Output Perturbation Perspective.\" arXiv preprint arXiv:2502.03805 (2025).\n\nQUESTIONS:\n1. For each dataset, what is the average prompt length and generation length? It is possible that for some tasks a token budget from {2k, 4k, 6k, 8k} is basically full attention without this specification. \n2. How do you deal with Group Query Attention which is used in the models you have evaluated with. Are you averaging the computed CAOTE score within the same KV group? \n3. Are you doing KV cache eviction for every attention layer in the model?\n4. How is your method different from [1]?\n5. Even though the method you proposed can be easily applied on top of existing cache eviction method based on their customized score, this might deviate from your theoretical foundation derived in section 3.1 as the token scoring from H2O cannot be translated to the attention output deviation of the current decoding step. Do you have any motivation why such generalization is reasonable?  \n\nReference:\n\n1. Feng, Yuan, et al. \"Identify Critical KV Cache in LLM Inference from an Output Perturbation Perspective.\" arXiv preprint arXiv:2502.03805 (2025).\n\nLIMITATIONS:\nYes", "TITLE": "Official Review", "RECOMMENDATION": 2, "REVIEWER_CONFIDENCE": 4, "CLARITY": 2, "IMPACT": 1, "ORIGINALITY": 2, "DATE": "2025-06-20"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe paper proposes CAOTE, a post-training token-eviction strategy for LLM KV caches that augments traditional attention-score heuristics with information from value vectors. Existing methods treat the attention weight as a proxy for token importance, ignoring how each token’s value actually shapes the attention output, and are therefore sub-optimal.\n\nCAOTE addresses this limitation by deriving a closed-form “eviction error’’—the expected L2 change in the attention output when a token is removed—and using it as a new importance score. The resulting CAOTE score fuses the attention weight with the value vector in a single analytic expression, yielding a simple yet principled criterion for cache eviction.\n\nBecause CAOTE is formulated as a meta-heuristic, it can be layered on top of any existing policy. Experimental results on LLama-3 and Qwen-2.5 across LongBench, perplexity evaluation and Needle-in-a-Haystack show that adding CAOTE systematically improves strong baselines such as H2O, TOVA and SnapKV, sometimes even reducing token count while raising accuracy.\n\nSTRENGTHS AND WEAKNESSES:\nStrengths\n+ Novel contribution: CAOTE augments traditional, attention-only heuristics with value-vector information by explicitly minimizing the change in the attention output, yielding a conceptually clear and previously unexplored criterion for KV-cache eviction.\n+ Solid theory: Theorem 3.2 formally shows that the CAOTE score equals the mean-squared error incurred in the attention output after evicting a token, giving the method a rigorous foundation.\n+ Convincing empirical study: Extensive tests on LongBench, perplexity, and Needle-in-a-Haystack with Llama-3 and Qwen-2.5 demonstrate consistent improvements over strong baselines such as H2O, TOVA, and SnapKV.\n\nWeaknesses\n- Single-token assumption: The derivation assumes only one token is evicted, while the experiments use block-wise prefilling that discards multiple tokens at once. The theoretical consequences of joint eviction remain unanalysed.\n- Overhead not quantified: The paper does not report the additional computation or latency introduced by computing CAOTE scores at inference time.\n- FastCAOTE is heuristic: Replacing the attention output with the mean value vector is empirically effective but lacks a principled justification or error analysis.\n- Limited model coverage: All experiments are on Llama-3 and Qwen-2.5; results on larger MoE models would strengthen the claim of generality.\n- Incomplete related work: Recent methods such as CAKE (ICLR 2025) are not compared, leaving readers uncertain about CAOTE’s relative standing in the latest literature.\n\nQUESTIONS:\n+ Your theoretical framework is built on single-token eviction. For the multi-token eviction used during block-wise prefill, could you elaborate on the sub-optimality of your greedy, independent-eviction assumption?\n+ Could you provide a quantitative analysis of the computational overhead (e.g., extra FLOPs or percentage increase in latency per step) of both CAOTE and FastCAOTE? \n+ As eviction decisions are made per layer, how does this error propagate and potentially compound in deeper models? Does your per-layer optimization effectively mitigate this compounding effect more than baselines?\n+ How does CAOTE interact with adaptive budget allocation methods (e.g., AdaKV) that dynamically adjust cache size per layer?\n\nLIMITATIONS:\nNA", "TITLE": "Official Review", "RECOMMENDATION": 3, "REVIEWER_CONFIDENCE": 4, "CLARITY": 3, "IMPACT": 3, "ORIGINALITY": 3, "DATE": "2025-06-25"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis article primarily focuses on compressing the KV cache of long-context LLMs using a token-eviction-based approach (i.e., removing tokens from the context that have minimal impact on subsequent generation). The authors observe that many classic existing methods rely solely on attention scores to measure the importance of tokens. However, in the transformer’s attention layer, the output is a weighted sum of value states based on the attention scores. To address this, they propose a method that considers both attention scores and value states. Experimental results demonstrate the effectiveness of their approach.\n\nSTRENGTHS AND WEAKNESSES:\nStrengths\n- The KV cache compression is important since long-context is more and more necessary recently.\n- The motivation is clear and significant. I agree with the authors’ view that value states should be taken into account when assessing the importance of tokens.\n\nWeaknesses\n\n- The writing is somewhat poor, especially in the Introduction and Method sections. Specifically, the Introduction devotes too much space to background and previous methods, with a serious lack of introduction to the authors’ own approach. For example, it merely states the motivation for using value states in addition to attention scores, but doesn’t explain at all how value states are actually utilized. In the Method section, Figure 1 only illustrates motivation, while the details of the method remain completely unclear.\n- The experiments are rather lacking, as reflected in several aspects:\n  - Baselines: As far as I know (though I may not be fully up-to-date on the latest work), there are many token-eviction-based methods that are not mentioned in this paper, including but not limited to StreamingLLM [1], Quest [2], CaM [3], and others. The baselines considered by the authors need to be more thoroughly investigated.\n  - The scale of the experiments is also somewhat outdated. Current long-context LLMs are already capable of handling very long contexts (up to 1M tokens), so it is necessary to explore at least 128K tokens. Specifically, it is important to conduct experimental validation on InftyBench [4].\n  - The depth of the experiments is insufficient. The authors do not analyze why CAOTE works—in other words, they do not investigate which specific changes compared to baseline methods (such as H2O) lead to performance improvements. The authors neither provide a qualitative analysis through case studies nor offer any quantitative experiments to support their claims. Furthermore, I observe that in Figure 3, CAOTE does not show any significant improvement over the baseline methods.\n  - The authors do not discuss the impact of their method on inference latency.\n  - There is a lack of adaptation to existing inference engines. Currently, there are many frameworks, such as VLLM and SGLANG, that can significantly improve LLM inference speed, but this method is not easily adaptable to them.\n\n[1] Xiao, Guangxuan, et al. \"Efficient streaming language models with attention sinks.\" arXiv preprint arXiv:2309.17453 (2023).\n\n[2] Tang, Jiaming, et al. \"Quest: Query-aware sparsity for efficient long-context llm inference.\" arXiv preprint arXiv:2406.10774 (2024).\n\n[3] Zhang, Yuxin, et al. \"Cam: Cache merging for memory-efficient llms inference.\" Forty-first International Conference on Machine Learning. 2024.\n\n[4] Zhang, Xinrong, et al. \"ınftyBench: Extending Long Context Evaluation Beyond 100K Tokens.\" ACL (1). 2024.\n\nQUESTIONS:\nSee the Weaknesses.\n\nLIMITATIONS:\nGiven the incomplete nature of the experiments, I believe there are still some limitations that have not been addressed. However, to some extent, this method is novel. In addition, the authors have not discussed the social impact of their work.", "TITLE": "Official Review", "RECOMMENDATION": 2, "REVIEWER_CONFIDENCE": 4, "CLARITY": 2, "IMPACT": 2, "ORIGINALITY": 3, "DATE": "2025-07-05"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe paper present CAOTE (KV Caching through Attention Output Error-based Token Eviction), a post-training token eviction method for reducing memory usage in long-context LLM inference. Unlike existing methods that rely solely on attention scores to determine token importance, CAOTE directly minimizes the change in attention output (eviction error) caused by removing a token. This error can be computed in closed-form using both attention scores and value vectors. CAOTE can also be combined with existing score-based eviction strategies (e.g., H2O, TOVA, SnapKV) via a normalization trick, and an efficient approximation called FastCAOTE is proposed. Extensive experiments across LongBench, BookSum, and Needle-in-a-Haystack show that CAOTE consistently improves accuracy and perplexity, with up to 30–60% higher retrieval accuracy at the same memory budget\n\nSTRENGTHS AND WEAKNESSES:\nstength:\n- The method is mathematically solid becasue it defines eviction error as the L2 distance between full and evicted attention outputs and shows this equals a closed-form score computable from attention weights and values.\n- enables more effective KV-cache trimming under tight memory constraints. For example, H2O+CAOTE boosts accuracy by over 30% on LongBench compared to H2O alone at 2k budget\n\nweakness:\n- CAOTE assumes single-token eviction at each step; during prefilling (multi-token insertions), the independence assumption may reduce optimality\n- Some notation is dense and can benefit from simplification or intuitive explanation, the role of each term in Eq. (8) could be explained better\n\nQUESTIONS:\n- Could you provide statistical significance or variability measures to validate that CAOTE’s performance gains\n- how would CAOTE extend to multi-token eviction scenarios\n\nLIMITATIONS:\nno negative societal impact", "TITLE": "Official Review", "RECOMMENDATION": 3, "REVIEWER_CONFIDENCE": 3, "CLARITY": 2, "IMPACT": 3, "ORIGINALITY": 3, "DATE": "2025-07-14"}