Title: Accurate KV Cache Eviction via Anchor Direction Projection for Efficient LLM Inference
Venue: NeurIPS 2025
Accepted: yes
Keywords: Large Language Models

KV Cache Eviction

LLM Inference Acceleration
Abstract: Key-Value (KV) cache eviction---which retains the KV pairs of the most important tokens while discarding less important ones---is a critical technique for optimizing both memory usage and inference latency in large language models (LLMs).
However, existing approaches often rely on simple heuristics---such as attention weights---to measure token importance, overlooking the spatial relationships between token value states in the vector space.
This often leads to suboptimal token selections and thus performance degradation.
To tackle this problem, we propose a novel method, namely **AnDPro** (**An**chor **D**irection **Pro**jection), which introduces a projection-based scoring function to more accurately measure token importance.
Specifically, AnDPro operates in the space of value vectors and leverages the projections of these vectors onto an *``Anchor Direction''*---the direction of the pre-eviction output---to measure token importance and guide more accurate token selection.
Experiments on $16$ datasets from the LongBench benchmark demonstrate that AnDPro can maintain $96.07\\%$ of the full cache accuracy using only $3.44\\%$ KV cache budget, reducing KV cache budget size by $46.0\\%$ without compromising quality compared to previous state-of-the-arts.


Peer reviews:
{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper proposes AnDPro (Anchor Direction Projection), a novel KV cache eviction strategy for large language models that goes beyond traditional attention-weight-based heuristics by incorporating geometric information from value vectors. It defines token importance via a projection-based scoring function, $\\( s_i = a_i \\cdot y^\\top v_i \\)$, where $\\( y \\)$ is the pre-eviction output direction. This approach is derived from a sparse optimization formulation and efficiently implemented with support for cross-head cache budgeting. Experiments on LongBench and Needle-in-a-Haystack benchmarks show that AnDPro outperforms prior methods, achieving 96.07% of full-cache accuracy using only 3.44% of the cache, reducing memory by up to 46% without quality loss.\n\nSTRENGTHS AND WEAKNESSES:\n[Strengths]\n1. The paper proposes a novel projection-based token importance score  $\\( s_i = a_i \\cdot y^\\top v_i \\)$​, incorporating value vector geometry beyond attention weights.\n2. The paper provides comprehensive theoretical analysis/proof and intuitive explanation for the proposed method.\n3. The authors evaluate the proposed method across different accuracy benchmarks with a wide range of LLMs, demonstrating the effectiveness of their optimizations, achieving 96.07% of full-cache accuracy using only 3.44% of the cache, reducing memory by up to 46% without quality loss.\n\n\n[Weaknesses]\n1. The discussion of implementation details should be improved.\n\nQUESTIONS:\n1. How is y computed in Section 3.4? Is it computed with KV cache in the local window and what is the window size? Do you compute y before each step of generation?\n2. Do different attention heads select the same tokens to keep? Or they can choose to preserve different tokens?\n3. Will the evicted token be permanently deleted from the KV cache? Or they can still be selected and attended to at future generation steps?\n4. What is the chunk size in Section 3.4?\n\nLIMITATIONS:\nyes", "TITLE": "Official Review", "RECOMMENDATION": 5, "REVIEWER_CONFIDENCE": 4, "CLARITY": 3, "IMPACT": 3, "ORIGINALITY": 4, "DATE": "2025-06-11"}

comments: SUMMARY:
This paper proposes AnDPro, a novel KV cache eviction method for efficient LLM inference. Unlike prior approaches that rely solely on attention weights to measure token importance, AnDPro leverages the geometric relationships between token value vectors in the embedding space. It introduces a projection-based scoring function, where token importance is quantified by projecting value vectors onto an anchor direction. Evaluated on 16 LongBench datasets, AnDPro achieves 96.07% of full-cache accuracy using only 3.44% of the KV cache budget.

STRENGTHS AND WEAKNESSES:
S1: The theoretical analysis is novel and solid.

S2: The experimental comparison is comprehensive.

S3: The writing of this paper is easy to follow.

W1: The evaluation is only conducted on relatively small-sized LLMs.

W2: The specific computational overhead of AnDPro is not provided.

QUESTIONS:
Q1: What would AnDPro perform on larger-sized LLMs (e.g., 32B, 70B)?
Q2: What is the specific computational overhead of AnDPro? How does AnDPro’s overhead compare to the overhead of existing methods?

LIMITATIONS:
Yes
TITLE: Official Review
CLARITY: 2
IMPACT: 3
ORIGINALITY: 3
DATE: 2025-06-30

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper proposes AnDPro (Anchor Direction Projection), a new method for KV cache eviction in large language models (LLMs). Traditional methods often rely on attention weights to determine token importance, but these heuristics ignore the geometric structure of value vectors, potentially leading to suboptimal eviction decisions. AnDPro introduces a projection-based scoring function that measures the importance of each token by projecting its value vector onto an “anchor direction”—the direction of the pre-eviction output. The paper formulates the eviction process as an integer optimization problem and derives a relaxed convex formulation to inspire the design of the scoring function. Experimental results on two models (Mistral-7B and LLaMA3-8B) and across 16 tasks in LongBench and the Needle-in-a-Haystack benchmark demonstrate that AnDPro achieves superior accuracy and efficiency, maintaining over 96% of full-cache performance using only 3.44% of the cache.\n\nSTRENGTHS AND WEAKNESSES:\nStrength:\n1. The authors model KV cache eviction as a constrained combinatorial optimization problem (P0-C), then relax it to a sparse convex problem (P1). This provides a mathematically grounded approach to derive a new scoring function based on the vector space geometry of the value states, rather than relying on attention weights alone.\n2. The authors evaluate AnDPro on two strong LLMs across diverse tasks and memory budgets, demonstrating consistent improvement over prior methods including SnapKV, Pyramid, Ada-KV, and CriticalKV. Notably, it achieves SOTA performance even at low memory budgets.\n3. They also provided a comprehensive analysis in the appendix.\n\nWeakness:\n1. The gap between the original non-convex objective (P0-C) and the relaxed convex form (P1) is not analyzed. Specifically, there is no theoretical or empirical discussion of how close the solution to P1 is to the true solution of P0-C.\n2. The reference of some methods, e.g. PyramidKV are missing.\n\nQUESTIONS:\n1. Can this method be extended to decoding phase? As it mainly focuses on the prefilling stage.\n2. What is the computational overhead of computing projections and scores per token? While the authors claim negligible latency, quantitative timing comparisons of the Update KV phase across methods would support the claim.\n\nLIMITATIONS:\nYes", "TITLE": "Official Review", "RECOMMENDATION": 4, "REVIEWER_CONFIDENCE": 4, "CLARITY": 3, "IMPACT": 3, "ORIGINALITY": 3, "DATE": "2025-07-02"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe paper introduces AnDPro (Anchor Direction Projection), a novel method for improving Key-Value (KV) cache eviction in Large Language Models (LLMs). By proposing a projection-based scoring function, AnDPro aims to accurately measure token importance in the context of KV cache management, thus optimizing memory usage and inference latency. The authors argue that traditional methods relying on attention weights overlook spatial relationships between token value states in vector space, leading to suboptimal token selections. Through extensive experiments on 16 datasets from the LongBench benchmark, AnDPro demonstrates its effectiveness by maintaining high accuracy with significantly reduced KV cache budgets compared to state-of-the-art techniques.\n\nSTRENGTHS AND WEAKNESSES:\n### Strengths:\n1) AnDPro introduces an anchor direction projection technique that considers the spatial relationships within vector spaces, providing a more nuanced way to evaluate token importance beyond simple heuristics like attention weights, which is Innovative.\n2) The paper offers a thorough theoretical analysis, including derivations and propositions, which underpin the design of the proposed scoring function.\n\n### Weaknesses:\n1) The idea of considering value states in the vector space is excellent. However, given that random vectors in high-dimensional spaces are almost orthogonal, and the vector space of value states is a very high-dimensional space, have you observed whether the value state vectors are orthogonal? If they are nearly orthogonal, then the illustration in Figure 1 could be misleading because the anchor direction would almost be dominated by attention weights, thereby reducing the accuracy gains obtained by the proposed method. If they are not orthogonal, please provide experimental results.\n2) In the implementation, is the anchor direction updated during each forward pass of the LLM, and what is the associated computational overhead? Clarification is needed on whether the anchor direction is recalculated or updated during each forward pass of the LLM. The computational cost associated with this update should also be quantified to understand its impact on overall efficiency.\n3) Could the compressed KV cache mislead the anchor direction compared to its uncompressed state? If this is the case, then the practical effectiveness of AnDPro might deviate from the theoretical derivation presented in Section 3. \n4) Experiments were only conducted on Mistral-7B-Instruct-v0.2, which is insufficient to demonstrate that the method can generalize to other series of models or models of different sizes. Additionally, conducting evaluations in more scenarios (e.g., mathematical reasoning) to evaluate the impact of AnDPro would be beneficial.\n\nQUESTIONS:\nSee weaknesses.\n\nLIMITATIONS:\nSee weaknesses.", "TITLE": "Official Review", "RECOMMENDATION": 4, "REVIEWER_CONFIDENCE": 4, "CLARITY": 3, "IMPACT": 3, "ORIGINALITY": 4, "DATE": "2025-07-02"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper aims to design an improved scoring policy for KV cache eviction that leverages error in the output to identify the optimal KV cache set to retain. Their approach is complementary to existing KV cache compression strategies. They first present theoretical analysis for the assignment problem, demonstrating how an ideal method would select the optimal set to minimize the degradation in the output vector. They then propose a computationally efficient method that projects value vectors weighted by attention score onto anchor direction (the original unperturbed output) to better estimate token importance. Their approach leads to a reduction in accuracy degradation relative to existing KV cache approaches, without adding substantial noticeable inference efficiency overheads.\n\nSTRENGTHS AND WEAKNESSES:\nStrengths:\n- They present a novel strategy based on minimizing output degradation which leverages strong theoretical foundations to identify the optimal key set to retain.\n- They present a computationally efficient method for identifying the optimal key set to retain based on projecting the attention score-weighted value vectors onto an anchor direction (the original output vector). \n- They outperform a range of strong baseline methods (as evaluated on LongBench + Passkey Retrieval)\n- Their approach has low overhead for the Update KV phase, as highlighted in Appendix C.4.\n\nWeaknesses:\n- Their method is only applicable to compressing the prefill KV cache for generation. It is unclear whether the update step is fast enough to run online during generation.\n- They evaluate their method only on short decoding tasks (e.g. LongBench). Some analysis of performance on longer decoding tasks (e.g. LLM reasoning) would be useful for understanding the broader applicability of their method.\n- The KV cache merging method (which is similar to pooling) is not well justified - the ablations in C.5 are hard to follow. Some explanation of this approach (or empirical evaluation) would be helpful to support this, including a comparison between chunking and pooling methods as well as intuitive explanation for why this is helpful and superior to pooling.\n\nQUESTIONS:\n- Is there any guarantee on how close the solution obtained through projection is to the optimal solution from the initial problem (P0-C or P1)?\n\nLIMITATIONS:\nyes", "TITLE": "Official Review", "RECOMMENDATION": 4, "REVIEWER_CONFIDENCE": 3, "CLARITY": 2, "IMPACT": 3, "ORIGINALITY": 3, "DATE": "2025-07-03"}