Title: Less Is More, but Where? Dynamic Token Compression via LLM-Guided Keyframe Prior
Venue: NeurIPS 2025
Accepted: yes
Keywords: Efficient Large Multimodal Models

Video Large Language Models

Token Pruning
Abstract: Recent advances in Video Large Language Models (VLLMs) have achieved remarkable video understanding capabilities, yet face critical efficiency bottlenecks due to quadratic computational growth with lengthy visual token sequences of long videos. While existing keyframe sampling methods can improve temporal modeling efficiency, additional computational cost is introduced before feature encoding, and the binary frame selection paradigm is found suboptimal. Therefore, in this work, we propose **Dy**namic **To**ken compression via LLM-guided **K**eyframe prior (**DyToK**), a training-free paradigm that enables dynamic token compression by harnessing VLLMs' inherent attention mechanisms. Our analysis reveals that VLLM attention layers naturally encoding query-conditioned keyframe priors, by which DyToK dynamically adjusts per-frame token retention ratios, prioritizing semantically rich frames while suppressing redundancies. Extensive experiments demonstrate that DyToK achieves state-of-the-art efficiency-accuracy tradeoffs. DyToK shows plug-and-play compatibility with existing compression methods, such as VisionZip and FastV, attaining 2.5x faster inference while preserving accuracy across multiple VLLMs, such as LLaVA-OneVision and Qwen2.5-VL. Code and models will be made publicly available.


Peer reviews:
{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper addresses the inefficiency of Video Large Language Models (VLLMs) in processing long videos, where computational costs rise due to redundant visual information. The authors propose DyToK, a training-free paradigm for dynamic token compression during inference. It leverages attention scores from VLLMs to distinguish between key frames and less relevant ones, adapting each frame’s token retention ratio based on its estimated importance. The method is plug-and-play compatible with existing compression techniques and applicable across multiple VLLM families. The authors conduct extensive experiments, visualizations, and analyses, showing that DyToK effectively identifies relevant key frames and achieves superior efficiency–accuracy trade-offs across various video benchmarks.\n\nSTRENGTHS AND WEAKNESSES:\nStrengths:\n- Quality: The proposed method is technically sound and thoroughly evaluated through extensive experiments across multiple VLLMs and token compression techniques. The authors also provide visualizations and analyses to support their claims.\n- Clarity: The paper is clearly written and easy to follow. Technical details are well-explained, with helpful insights that facilitate a clear understanding.\n- Significance: The work addresses the critical challenge of improving efficiency in long-video understanding for VLLMs. The analysis of attention maps across different layers for keyframe identification is insightful and potentially valuable for future research.\n- Originality: The training-free, plug-and-play paradigm and its seamless integration with existing compression techniques offer a novel and practical perspective on advancing token compression in VLLMs.\n\nWeakness:\n- Quality: The method relies on an auxiliary lightweight model from the same architectural family to approximate the original model. The effectiveness and generalizability of this approach are not fully validated and warrant further analysis and discussion.\n- Clarity: There are several noticeable typos in the paper, such as the placeholder “on XX benchmarks” in Line 65. The authors should proofread the manuscript more carefully.\n- Significance: The practical benefits of token reduction, such as inference speed-up or extended video support under fixed computational budgets, are not sufficiently demonstrated in the paper.\n\nQUESTIONS:\n1. The auxiliary lightweight assistant model used to obtain attention maps is derived from the same architectural family as the main model. However, I believe its feasibility and generalizability require further clarification. a) In addition to the final benchmark results, it would be helpful to include visualizations or statistical analyses. b) This design choice may also limit the method’s generalizability, especially for models that lack a corresponding lightweight version or where the lightweight version differs significantly in architecture. c) What would happen if a lightweight model from a different architectural family were used?\n2. The practical benefits of reducing the number of tokens can be better demonstrated. For example, the actual speed-up in inference time with the same number of input frames, or the performance gains under a fixed computational budget.\n3. Although the authors conducted extensive experiments and analyses, the presentation could be improved. For instance, performance trends across different token compression ratios could be presented more clearly, and visualizing the keyframe scores more intuitively in the main text would help better support the proposed method.\n4. a) Using only the last token of the query to score the frames may overlook some important information. b) If the query does not explicitly point to specific visual content (e.g., a general descriptive query), can this method still effectively select important frames in the video?\n\nMy overall score may be adjusted based on the authors’ responses to these questions.\n\nLIMITATIONS:\nIn the limitations section, the authors mention that, to balance performance and efficiency, they introduced an auxiliary lightweight assistant model to approximate the original model. I believe the effectiveness and generalizability of this approach require further analysis and discussion, as noted in the questions above.", "TITLE": "Official Review", "RECOMMENDATION": 4, "REVIEWER_CONFIDENCE": 4, "CLARITY": 2, "IMPACT": 3, "ORIGINALITY": 3, "DATE": "2025-06-26"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper introduces DyTok , a training-free dynamic token compression method that leverages LLM-guided keyframe priors. Given a sequence of visual tokens, DyTok employs a LLM to assess the importance of each token by computing its similarity with the textual query tokens. Based on these importance scores, the method selectively retains the most informative tokens while maintaining a balanced token distribution across frames. DyTok is evaluated on multiple video understanding benchmarks.\n\nSTRENGTHS AND WEAKNESSES:\nStrengths:\n* The paper is well-written and clearly structured, making it easy to follow.\n* The proposed idea is conceptually straightforward, computationally efficient, and can be integrated into various methods.\n* Extensive experimental results demonstrate the effectiveness and robustness of DyTok across multiple tasks and datasets.\n\nWeaknesses:\n* The paper computes attention scores between the final textual query token and the visual tokens. However, it would be helpful to include an analysis on the choice of textual token representation. While the last token may capture global query information, in some cases, it might lack sufficient semantic richness. \n* Although the auxiliary lightweight assistant model helps reduce computational cost, it still relies on strong pre-trained VLMs, such as LLaVA-OneVision and Qwen2.5-VL. This dependency may limit the generalizability and accessibility of DyTok across different frameworks or less powerful backbones.\n* The selection of deep layers appears to be based on specific architectural choices. It is unclear whether this selection is model-agnostic or tailored to particular backbones. If it is not, then the analysis presented in Table 3 may need to be repeated for each new model architecture. Is there evidence supporting the transferability of this layer selection strategy?\n* In the appendix Table 7, instead of only computing prefilling time, what about the inference time of the whole pipeline? As an additional model is used, I'd expect the DyTok pipeline to have longer inference time overall.\n\nQUESTIONS:\nPlease refer to the weaknesses part for my concerns. My initial judgment is BR, and I will adjust my rating accordingly based on the authors' response.\n\nLIMITATIONS:\nyes", "TITLE": "Official Review", "RECOMMENDATION": 4, "REVIEWER_CONFIDENCE": 4, "CLARITY": 3, "IMPACT": 2, "ORIGINALITY": 2, "DATE": "2025-07-01"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe paper introduces DyToK, a training-free dynamic token compression framework for Video Large Language Models (VLLMs). By extracting query-conditioned keyframe priors from deep attention layers of a lightweight assistant model, DyToK adaptively allocates per‐frame token budgets and applies any compatible pruning strategy. This plug-and-play approach seamlessly integrates with existing compression backbones—both encoder feature– and attention–based—achieving state-of-the-art efficiency-accuracy trade-offs across multiple long-video benchmarks.\n\nSTRENGTHS AND WEAKNESSES:\nStrengths:\n* The motivation for dynamic compression is presented concisely, highlighting inefficiencies of uniform frame pruning.\n* The ablation studies systematically evaluate compression ratios, attention‐layer selection, and assistant model size, providing strong empirical insights.\n* Algorithm 1 is presented with pseudocode that precisely describes the adaptive token allocation and compression steps.\n* DyToK’s training-free paradigm and modular compression function enable straightforward integration with existing token-pruning methods.\n* The paper demonstrates insightful finding that depth of attention is more critical than model scale for importance estimation.\nDyToK reliably boosts accuracy under aggressive pruning across diverse benchmarks and methods.\n\nWeaknesses:\n*  While existing binary keyframe sampling of VLLM is discussed, the method also closely relates to the retrieval-based video understanding methods, such as rag-based methods(Video-rag: Visually-aligned retrieval-augmented long video comprehension). It would add clarity if the authors discuss and compare with that line of methods.\n*  Eq. (1) quantifies total FLOPs, but the paper does not present calculated FLOP savings for DyToK relative to baselines.\n*  It remains unclear how different assistant models (a wider ranges besides 7B and 0.5B) influence the performance, and how mispredictions of assistant models propagate to the primary VLLM.\n\nQUESTIONS:\nDoes the reported efficiency improvement include the lightweight assistant model’s forward pass?\n\nLIMITATIONS:\nyes", "TITLE": "Official Review", "RECOMMENDATION": 4, "REVIEWER_CONFIDENCE": 4, "CLARITY": 4, "IMPACT": 3, "ORIGINALITY": 3, "DATE": "2025-07-02"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper proposes a dynamic token compression scheme that leverages the inherent attention mechanisms of VLLM. Experimental results show that the attention module effectively learns keyframe priors from the training data. By adopting this scheme, the algorithm dynamically adjusts the number of tokens on a frame-by-frame basis, adaptively removing redundant information. Extensive experiments demonstrate that DyToK achieves state-of-the-art trade-offs between efficiency and accuracy.\n\nSTRENGTHS AND WEAKNESSES:\n### Strengths\n\n- Extensive evaluations on long-video benchmarks convincingly demonstrate that DyToK outperforms existing state-of-the-art methods in terms of efficiency-accuracy tradeoffs.\n\n- The paper provides a compelling finding that attention layers in vision-language large models (VLLMs) inherently encode query-conditioned keyframe priors. This insight enables effective identification of task-relevant information without requiring additional supervision.\n\n- DyToK shows excellent generalization and compatibility across different backbone models and token compression strategies. This makes it a versatile component that can be easily integrated into existing frameworks.\n\n- The authors conduct extensive experiments and ablation studies across various datasets and model settings, providing strong evidence for the effectiveness and robustness of the proposed approach.\n\n### Weaknesses\n- Lack of Comparison with Recent State-of-the-Art Methods: The paper omits comparison with several recently proposed token compression or pruning techniques, limiting the evaluation’s completeness and making it hard to assess the relative progress.\n\n- No Evaluation Against Structured Compression Baselines: Methods such as LoRA, quantization-aware training, or parameter-efficient fine-tuning techniques are not included in the comparison, which raises concerns about the fairness and rigor of the benchmarks.\n\n- Missing End-to-End Latency on Real Hardware: While theoretical FLOPs are reported, practical runtime metrics (e.g., latency on GPU inference) are not provided, making it unclear how beneficial the proposed approach is in deployment scenarios.\n\n- Weak Justification for the Attention-Based Importance Estimation: The proposed method for selecting important tokens is intuitive but lacks theoretical grounding or sufficient ablation studies to validate why it outperforms simpler baselines like entropy- or magnitude-based filtering.\n\nQUESTIONS:\n- **Line 65: Placeholder Reference**  \n  The text refers to “XX datasets,” which appears to be a placeholder. This should be replaced with the actual dataset names to clearly indicate the evaluation scope and improve the paper’s professionalism.\n\n- **Definition of Keyframe and Adaptive Rate**  \n  The paper does not clearly define what qualifies as a “keyframe” within the proposed method. Additionally, the mechanism for determining the adaptive token rate for keyframes versus normal frames is not sufficiently explained. A detailed description or visualization of this process would enhance understanding.\n\n- **Runtime Efficiency**  \n  Although the paper emphasizes token redundancy reduction, it lacks reporting of runtime efficiency metrics. Including inference latency or throughput on real hardware would significantly strengthen the case for practical applicability, particularly in real-time or latency-sensitive environments.\n\nLIMITATIONS:\nYes.", "TITLE": "Official Review", "RECOMMENDATION": 3, "REVIEWER_CONFIDENCE": 3, "CLARITY": 2, "IMPACT": 3, "ORIGINALITY": 2, "DATE": "2025-07-05"}