Title: PrefixKV: Adaptive Prefix KV Cache is What Vision Instruction-Following Models Need for Efficient Generation
Venue: NeurIPS 2025
Accepted: yes
Keywords: large vision-language models

KV cache compression
Abstract: Recently, large vision-language models (LVLMs) have rapidly gained popularity for their strong generation and reasoning capabilities given diverse multimodal inputs. However, these models incur significant computational and memory overhead during inference, which greatly hinders the efficient deployment in practical scenarios. The extensive key-value (KV) cache, necessitated by the lengthy input and output sequences, notably contributes to the high inference cost. Based on this, recent works have investigated ways to reduce the KV cache size for higher efficiency. Although effective, they generally overlook the distinct importance distributions of KV vectors across layers and maintain the same cache size for each layer during the next token prediction. This results in the significant contextual information loss for certain layers, leading to notable performance decline. To address this, we present PrefixKV. It reframes the challenge of determining KV cache sizes for all layers into the task of searching for the optimal global prefix configuration. With an adaptive layer-wise KV retention recipe based on binary search, the maximum contextual information can thus be preserved in each layer, facilitating the generation. Extensive experiments demonstrate that our method achieves the state-of-the-art performance compared with others. It exhibits superior inference efficiency and generation quality trade-offs, showing promising potential for practical applications. Code is available at
https://github.com/THU-MIG/PrefixKV.


Peer reviews:
comments: SUMMARY:
The paper introduces PrefixKV, an innovative method for KV cache compression in LVLMs. It adjusts layer-wise KV cache retention ratio based on varying importance distribution of KV vectors across layers. By utilizing offline binary search to determine the optimal prefix configuration, PrefixKV aims to enhance inference efficiency while maintaining high-quality generation. The paper presents extensive experiments, showing superiority in terms of inference speed and performance.

STRENGTHS AND WEAKNESSES:
Strengths:

1.It is interesting to leverage the varying layer-wise KV importance for developing a KV cache budget strategy with the analyses from the Lorenz curve and Gini coefficient perspectives. The offline estimation of the global prefix configuration ensures that the method does not incur additional inference costs, making it practical for deployment. 
2. The method achieves state-of-the-art performance across various compression ratios, maintaining high performance even at low compression ratios. The experiments are conducted across multiple models and benchmarks, which present comprehensive analyses and comparisons and verify the efficacy of the proposed method. 

Weaknesses:
1. The related work section is somewhat short. A more detailed introduction and discussion of recent KV cache compression techniques would provide a clearer context.
2. The "prefix" in PrefixKV might be somewhat misleading. Specifically, "prefix" usually denotes the initial segment of a sequence, but the proposed method preserves tokens based on significance, rather than position in the sequence.

QUESTIONS:
See the weaknesses.

LIMITATIONS:
Yes
TITLE: Official Review
CLARITY: 3
IMPACT: 3
ORIGINALITY: 3
DATE: 2025-06-30

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe paper proposes a method for efficient attention key-value caching in the context of LVMs. In particular, the authors propose a variable-rate caching mechanism based on the relative importance of cached values across layers. For a fixed ``budget” compression rate, the method aims to optimally estimate the best amount of key-value pairs to be cached per layer, with such optimal ratio found through binary search. The authors apply their method across a variety of models including LlaVa and MM-Vet, as well as in Qwen-VL and LLM models. A set of ablation studies show the efficiency of their proposed approach. The proposed method shows to produce state of the art results for high levels of compression, illustrating the benefit of the proposed approach in low-regime scenarios.\n\nSTRENGTHS AND WEAKNESSES:\nStrengths:\n\n\nThe paper proposes a simple yet efficient idea, namely that of varying the number of cached tokens per layer, according to their estimated relevance to future attention maps. \n\n\nI find it interesting the fact that offline results are on par with online results, illustrating that some pre-computed relevance is enough to have good and efficient compression. \n\n\nThe results show that for high compression levels their proposed approach is better than alternative methods like H2O or Elastic.\n\n\nThe paper includes a good amount of ablation studies including the tradeoff between compression ratio and results, the effect of online vs offline caching, or the effect of merging rather than discarding tokens. \n\n\nWeaknesses:\n\n\nThe idea, while overall efficient, is relatively simple and the novelty is a bit limited. I see that in Table 4, the proposed approach is almost on par with PyramidKV except for very big compression rates. The comparison novelty-results against PyramidKV is in my opinion a bit limited. \n\n\n\nI find it a bit surprising that the Offline estimation produces nearly exactly the same results as that of online caching. This is very surprising as one would expect that online caching should always be related to current sequences and therefore some variability would be expected in these results in favor of online caching. In a similar way, I find it really surprising that a single sample is enough for the ratio optimality search in an offline setting, with results that are pretty much the same as those of online caching. A single example might contain a rather different semantic information and provoke completely different activations in the attention maps at different layers. The results suggest that the percentage of attention values that are relevant is almost fixed for each layer and is independent of the input content. In other words, the same percentage of tokens will be relevant at each layer for a completely variety of samples, regardless the entropy of each input sequence. This requires further investigation or explanation because it is rather surprising. \n\n\n\n\nThe results in Table 7 also suggest that merging tokens is a better option than directly disregarding them. This requires further investigation against methods for token merging.\n\nQUESTIONS:\nMy questions are embedded in the weaknesses section above. In particular:\n\n- Novelty/results tradeoff against PyramidKV\n\n- Offline estimation being on par with online estimation\n\n- Merging tokens better than discarding them.\n\nLIMITATIONS:\nNo, there is not such section in the paper. The authors should add such section in the paper.", "TITLE": "Official Review", "RECOMMENDATION": 4, "REVIEWER_CONFIDENCE": 4, "CLARITY": 3, "IMPACT": 3, "ORIGINALITY": 2, "DATE": "2025-07-02"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper tackles the KV cache overhead in large vision-language models (LVLMs). The core argument is that the importance of KV cache vectors is heterogeneous across layers, a claim supported by an analysis of attention distributions. To leverage this, the authors propose PrefixKV, a method that assigns a unique, adaptive compression ratio to each layer. These layer-specific ratios are determined by searching for a single information retention threshold that satisfies a global compression budget. Crucially, this configuration is calculated offline, imposing no overhead during inference.\n\nSTRENGTHS AND WEAKNESSES:\n\tStrengh:\n(1) The paper's approach is well-motivated and principled, grounded in a rigorous analysis of KV cache importance heterogeneity. The use of Lorenz curves provides a strong quantitative justification for the proposed layer-wise compression.\n(2) The PrefixKV method is straightforward and elegant. It cleverly reframes a complex multi-layer optimization problem into a simple binary search for a global threshold, and its ability to be pre-computed offline makes the solution highly practical for deployment.\n(3) The experiments show that the method surpasses previous methods significantly.\n\n\tWeakness:\n(1) While the paper presents a well-engineered and effective solution, the core idea of layer-wise KV cache allocation is not entirely novel. The authors acknowledge prior works [65, 19, 27] that explore this direction. The primary contribution thus shifts from the conceptual breakthrough of \"adaptive allocation\" to the specific algorithm for achieving it (binary searching for a global cumulative priority threshold), which, while elegant, can be seen as a strong incremental improvement rather than a fundamentally new paradigm.\n(2) The practicality of the method hinges on the claim that an optimal prefix configuration can be determined offline from a very small number of samples (as few as one, per Table 6). While experimental results are stable, this assumption might not hold under more diverse, out-of-distribution, or adversarial scenarios. The paper lacks a rigorous analysis of the conditions under which this offline estimation remains robust, which is a critical aspect for real-world reliability.\n(3) The experimental setup for ROUGE evaluation, where the uncompressed reference is generated with non-zero temperature while compressed models use temperature 0 (lines 237-239), complicates interpretation. This leads to reported ROUGE scores for the proposed method that are higher than the \"full cache\" baseline (e.g., 0.76 vs 0.62 in Table 1), which is counter-intuitive. A more direct comparison using the same generation parameters for both the baseline and the proposed method would provide a clearer measure of quality preservation.\n(4) The most direct competitor, PyramidKV [65]—which employs a non-uniform, \"manually designed\" layer-wise allocation—is only compared in a supplementary analysis (Table 4). For a stronger claim, PrefixKV should be benchmarked against PyramidKV and other potential adaptive baselines within the main result tables (Tables 1 and 2). The current presentation, which primarily contrasts with uniform allocation methods, may overstate the performance gains attributable to the proposed search strategy versus the general benefit of adaptivity.\n\nQUESTIONS:\n(1) A major practical advantage of your method is the offline estimation of the global prefix configuration from a small sample set. Could you provide more analysis on the robustness of this process? For instance, how much does the derived configuration vary when using different small sets of calibration samples, especially samples from different domains or with different instruction complexities? A convincing analysis, even on a small scale, demonstrating the stability of the derived configuration would significantly strengthen the paper's practical claims.\n(2) The use of different temperature settings for the baseline (temp > 0) and the proposed method (temp = 0) leads to counter-intuitive ROUGE scores where PrefixKV surpasses the full-cache model (Table 1). Could you please provide results from a revised experiment where both the full-cache baseline and PrefixKV use the exact same deterministic generation settings (i.e., temperature 0)? This would provide a more direct and interpretable measure of information preservation.\n(3) The comparison with PyramidKV [65] in Table 4 is crucial, as it isolates the benefit of your search-based adaptive strategy from a manually-designed one. Could you elaborate on your interpretation of these results? Specifically, what is the core mechanism that allows your method to so clearly outperform PyramidKV, especially at aggressive compression ratios (e.g., 10-30%)? A more detailed analysis would help to better situate the novelty of your contribution, moving it beyond \"layer-wise adaptation is good\" to \"our method for finding the adaptation strategy is superior.\"\n(4) The W-shaped curve for retained cache sizes in Figure 4 is an interesting finding. Is there any intuition as to why shallow and deep layers require more cache than middle layers? Is this pattern consistent across different model families, or is it specific to the LLaMA architecture? A brief discussion on this could offer valuable insights into the inner workings of LVLMs and enhance the paper's significance.\n\nLIMITATIONS:\nyes", "TITLE": "Official Review", "RECOMMENDATION": 4, "REVIEWER_CONFIDENCE": 3, "CLARITY": 3, "IMPACT": 3, "ORIGINALITY": 2, "DATE": "2025-07-03"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper proposes PrefixKV, a method for adaptive KV cache compression in large vision-language models (LVLMs), targeting efficient generation during inference. The core idea is to optimize the retained key-value (KV) cache size per layer, rather than applying a uniform retention policy.  PrefixKV introduces a binary search mechanism to derive a global prefix configuration that maximizes contextual information retention while respecting a global compression budget. The method is evaluated on several models (LLaVA-1.5-7B, 13B, Qwen-VL) and datasets (LLaVA-Description, MM-Vet), showing superior performance to state-of-the-art methods such as H2O and Elastic Cache under a range of compression budgets\n\nSTRENGTHS AND WEAKNESSES:\nStrengths.\n\n-  The paper provides strong motivation for layer-wise KV retention, backed by empirical observations such as Lorenz curves and Gini coefficients. The insight into variable importance distributions across layers is well-founded and novel.\n- The method provides substantial inference speedups (e.g., 1.8× on LLaVA-1.5-7B) while retaining performance, making it highly relevant for real-world deployment.\n\nWeaknesses.\n- While the binary search approach is efficient, the paper does not deeply analyze the sensitivity or convergence behavior of the algorithm (e.g., robustness to approximation thresholds, search steps).\n- Some figures (e.g., Fig. 1 and Fig. 3) are packed and slightly difficult to interpret. More concise visuals or splitting into sub-figures could improve clarity.\n- The term “prefix” is overloaded in the Transformer literature and may confuse readers. A brief clarification early on (e.g., prefix in sorted importance vs. positional prefix) would help.\n\nQUESTIONS:\n1. Could you expand on why 10 samples suffice for offline estimation across different datasets and tasks? Are there scenarios where this assumption might break?\n2. How well does a prefix configuration estimated on one dataset (e.g., LLaVA-Description) generalize to another (e.g., MM-Vet)? Have you tested cross-task portability?\n\nLIMITATIONS:\nyes", "TITLE": "Official Review", "RECOMMENDATION": 4, "REVIEWER_CONFIDENCE": 3, "CLARITY": 3, "IMPACT": 2, "ORIGINALITY": 2, "DATE": "2025-07-05"}