Less Is More, but Where? Dynamic Token Compression via LLM-Guided Keyframe Prior
Authors: Yulin Li
Haokun GUI
Ziyang Fan
Junjie Wang
Bin Kang
BIN CHEN
Zhuotao Tian
Venue: NeurIPS 2025 poster ()
Keywords: Efficient Large Multimodal Models
Video Large Language Models
Token Pruning
Primary area: applications

Abstract: Recent advances in Video Large Language Models (VLLMs) have achieved remarkable video understanding capabilities, yet face critical efficiency bottlenecks due to quadratic computational growth with lengthy visual token sequences of long videos. While existing keyframe sampling methods can improve temporal modeling efficiency, additional computational cost is introduced before feature encoding, and the binary frame selection paradigm is found suboptimal. Therefore, in this work, we propose **Dy**namic **To**ken compression via LLM-guided **K**eyframe prior (**DyToK**), a training-free paradigm that enables dynamic token compression by harnessing VLLMs' inherent attention mechanisms. Our analysis reveals that VLLM attention layers naturally encoding query-conditioned keyframe priors, by which DyToK dynamically adjusts per-frame token retention ratios, prioritizing semantically rich frames while suppressing redundancies. Extensive experiments demonstrate that DyToK achieves state-of-the-art efficiency-accuracy tradeoffs. DyToK shows plug-and-play compatibility with existing compression methods, such as VisionZip and FastV, attaining 2.5x faster inference while preserving accuracy across multiple VLLMs, such as LLaVA-OneVision and Qwen2.5-VL. Code and models will be made publicly available.

Less Is More, but Where?
Dynamic Token Compression via LLM-Guided Keyframe Prior
Yulin Li
Haokun Gui
Ziyang Fan' Junjie Wang'
Bin Kang
Bin Chen 1.3
Zhuotao Tian
Harbin Institute of Technology (Shenzhen) 2Shenzhen Loop Area Institute
University Chinese Academy of Sciences "The Hong Kong University of Science and Technology
Abstract
Recent advances in Video Large Language Models (VLLMs) have achieved re- markable video understanding capabilities, yet face critical efficiency bottlenecks due to quadratic computational growth with lengthy visual token sequences long videos. While existing keyframe sampling methods can improve temporal modeling efficiency, additional computational cost is introduced before feature encoding, and the binary frame selection paradigm is found suboptimal. Therefore, in this work, we propose Dynamic Token compression via LLM-guided Keyframe prior (DyToK), training-free paradigm that enables dynamic token compression by harnessing VLLMs' inherent attention mechanisms. Our analysis reveals that
VLLM attention layers naturally encoding query-conditioned keyframe priors, by which DyToK dynamically adjusts per-frame token retention ratios, prioritizing semantically rich frames while suppressing redundancies. Extensive experiments demonstrate that DyToK achieves state-of-the t--hhe-ar art efficiency-accuree tradeoffs. DyToK shows plug-and-play compatibility with existing compression methods, such as VisionZip and FastV, attaining 4.3 x faster inference while preserving accuracy across multiple VLLMs, such LLaVA-One Vision and Qwen2.5-VL. Code is available at http:://gitub..comyy-iii-ii-ii ToK.
Introduction
have demonstrated re- markable capabilities in processing complex video content. However, their practical deployment faces critical challenges when handling long videos [5] o 71. including excessive computational overhead, slow inference speeds, and performance degradation caused by redundant visual information.
Recent advancements in Video Large Language Models (VLLMs) 2 4
Existing solutions primarily focus token compression through saliency metrics derived from either LLM attention patterns [8] [0] or visual encoder features [ [3]. While LLM attention-baan
methods selectively retain prompt-relevant tokens, their effectiveness heavily relies layer-specifis attention maps, introducing instability as shallow layers yield noisy signals while deeper layers negate computational benefits [14 15). Conversely, encoder feature-based approaches exploit aitra-frame
token sparsity through CLS token attention [66 or inter-patch correlations [17] but uniformly apply fixed compression ratios across frames, ignoring temporal dynamics critical for video understanding.
"Equal Contribution
Corresponding Author
39th Conference on Neural Information Processing Systems (NcurIPS 2025).

---

part of video
does the red parrat appear?
Question &Video
A. The parrot does not appear.
8. End of the video.
C. Beginning the video.
D. Middle the video.
Dynamic Per-Frame
Static Per-Frame
GT Keyframe
Dynamic Taken Compression
Taken Allocation
Token Allocation
Importance
Correct Answer:
VLLM Temporal
GT Keyframe
Top-& Keyframe Importance
Impotance
1 oop
Uniform Token Compression
Wrong Answer:
Frame Num
Figure 1: Unveiling the keyframe prior in VLLMs. LLaVA- -ne Vision's 's answers to video QA tasks are shown on the left. On the right, we plot the averaged attention from the final text token to visual tokens across all layers and within each frame. The top-8 frames by attention scores are arranged in time order, and the Ground Truth (GT) keyframes are highlighted in red. We observe that even when the model answers incorrectly, its attention still pinpoints the relevant frames, revealing a strong desk-depeddent keyframe prior.
Motivation. Unlike token compression for images, which applies single ratio, video frames contain varying spatiotemporal information are critical for answering user queries, while others may hold less relevance. This inherent variation suggests that the shared compression ratios may be inappropriate for processing video frames. Consequently, a key question emerges: How can we dynamically retain task-relevant information while filtering out the irrelevant components?
An intuitive solution to this question is to perform the keyframe selection where frames deemed semantically relevant user queries are prioritized while others are discarded, theoretically improv- ing VLLMs' inference efficiency. However, existing works [18 19] typically rely on pretrained lssionlanguage models or auxiliary modules to perform frame selection before the feature encoding. introducing additional computational overhead that undermines the efficiency gains from frame reduction. Moreover, the binary selection paradigm proves suboptimal as irrevocably discards po- tentially useful visual cues in unselected frames while retaining redundant information within chosen frames, creating an efficiency-utii -tillty trade-off that restricts video understanding. This observation suggests the need for a more nuanced approach to spatial-temporal information compression for better preserving the rele-relevant details within the constrained computational budget.
Our solution. To address this challenge, in this work, we propose Dynamic Token compression via LLM-guided Keyframe prior (DyToK). simple yet effective training-free free method that performs dynamic frame token compression for enhancing the efficiency of VLLMs. Instead of simply discarding or retaining frames outright, DyToK introduces a dynamic compression approach tailored for VLLMs. By leveraging the inherent attention mechanisms of VLLMs, DyToK selectively preserves the most critical tokens across frames, prioritizing those with higher importance. This ensures that keyframes retain more tokens, optimizing both efficiency and model performance. The modular design of DyToK ensures seamless compatibility with existing token pruning techniques, enabling plug-and-play integration without compromising efficiency.
Extensive evaluations on long-video benchmarks demonstrate DyToK's superiority over state-of-the- art methods. Under 20% token retention, our approach surpasses uncompressed baselines by 2.6% accuracy on Bideeeee...... At extreme compression ratios (10% retention), DyToK achieves a 24.0% performance gain over the competitors. To summarize, our contributions are as follows:
We empirically reveal that the attention layers in VLLMs inherently encode query- conditioned keyframe priors that can be used for identifying the task-relevant information.
We propose DyToK,, training-free compression paradigm that dynamically adjusts per- frame token retention ratios based on the model's intrinsic attention scores, achieving adaptive temporal compression while preserving task-critical semantics.
* DyToK demonstrates strong plug-and-play compatibility across different models and com- pression methods, attaining -tate-of-the-art fffciency-acccuray tradeoffs on three bench- marks specifically designed for long-video analysis.

---

2 Background and Motivation
In the following, we provide a concise overview foundational concepts that underlic this study in Sec. [. and highlight the key observations in Sec. 2.2. which offer valuable insights for motivating the proposed approach.
2.1 Preliminaries
are typically composed of three core com- ponents: vision encoder, modality projector, and LLM backbone. The vision encoder, often pre-trained on large-scale image datasets (e.g., CLIP [6), SigLip [17]). processes each video frame into a sequence of visual tokens, with some variants incorporating video-specific pretraining (20 21 for temporal modeling. The projector aligns these tokens with the LLM's textual embedding space, enabling cross mods- fusion. Finally, the LLM integrates the aligned visual and textual tokens to generate contextually relevant responses.
Architecture of VLLMs. Modern VLLMS 1 3
Computational bottleneck analysis. Though VLLMs have shown promising performance, the computational overhead impedes their practical applications. Specifically, the computational com- plexity of VILMM dominated by the self-attention mechanism and perr-rd Networks (FFNs) in transformer layers. For model with T transformer layers, the total Floating Point Operations (FLOPs) can be formulated as:
Total FLOPs = T x (4nd² + 2n'd d + 2ndm),
where n denotes the input sequence length, d is the hidden dimension, and m represents the FFN's intermediate size. In video tasks, is dominated by visual tokens nlvis which often exceed textual tokens Vlays + Rquestion by orders of magnitude through frame accumulation [12 22). Reducing 7ivis is thus crucial for improving inference efficiency.
Recent advancements in efficient inference for VLLMs can be categorized into two paradigms: LLM attention -ased token pruning [10] [3] and encoder feature-based token selection [23 24 The former (Fig. ([a)) dynamically eliminates redundant visual tokens during LLM inference by leveraging cross- modal attention patterns from intermediate layers, yet suffers from unstable pruning decisions due to malloow-lyerr attention noise. The latter (Fig- (b) statically selects tokens at the encoder output using feature correlations, but disregards temporal dependencies critical for video comprehension.
Efficient inference paradigms for VLLMs.
Notably, existing keyframe selection techniques [19 [8] primarily address issues arising from uni- form input sampling in long videos, rather than optimizing for computational acceleration. Moreover, their binary frame selection approach risks discarding useful tokens of the discarded frames, poten- tially harming overall performance. Differently, in this study, we leverage a novel keyframe-aware compression strategy (Fig.(c)) that actively reduces inference latency for different frames, by which more tokens will be retained in the semantically important frames and fewer will be kept in those less relevant, thereby enhancing efficiency without compromising the coherence among the temporal semantics. A more comprehensive review of related works is provided in Appendix D
2.2 Key Observations
13 9 10 have demonstrated promising results, key chal- directly adapt image-domain techniques, processing each frame independently while overlooking inter-frame correlations. How- ever, in practice, video frames exhibit strong temporal dependencies some contain query-critical information, while others contribute minimally. Therefore, to enable the dynamic frame-level token compression, we need to establish an accurate estimation of temporal importance across video frames.
Although prior compression methods 8
lenges persist. Most video understanding approaches [ 2 12
We start by considering whether the attention map can serve an indicator of the importance of frames. As shown Fig. [ we observe that the model consistently assigns peak attention scores to query-relevant keyframes, irrespective of answer correctness. This persistent pattern indicates an inherent bias toward semantically salient frames the model's attention mechanism. A detailed analysis of keyframe prior in VLLMs is presented in Appendix B
3

---

LLM Layer (K+1)~N
Visual
Visual
Keyframe Prior
Tokens
Tokers
Keyframe Price
Pruning
Large Language Model
Large Language Model
LLM Layer O-K
Pruning
Pruning
Vision Encoder
Embedding
Vision Encoder
Embedding
Vision Encoder
Embedding
Please describe this
Pietse deseribe video
Please describe this
(a) Attention-Basee
Encoder Feature-Rused
(C) Keyframe (Ours)
Figure 2: Efficient inference methods for VLLMs. (a) LLM attention-baser methods perform token pruning during LLM inference by selecting visual tokens through cross-modal attention maps from specific layers, hence suffer from constrained pruning accuracy due their reliance noisy shallow-layer attention patterns. (b) Encoder feature-based methods prune tokens post-encoder using inter-patch feature correlations, but neglect temporal dynamics essential for video understanding. (c) Our approach uniquely exploits the keyframe priors embedded within LLMs to dynamically allocate frame-specific compression ratios, enabling plug-and and-play enhancement of temporal perception capabilities in existing efficient VLLMs.
Deep attention layers provide good keyframe priors. An intuitive approach aggregate atten- tion patterns from all LLM layers for frame importance estimation. However, despite its effectiveness, this method necessitates full inference through the LLM, inevitably introducing computational in- models show that shallow layers primarily capture local features, whereas deeper layers parse more abstract, high-level semantics.
efficiency. Prior studies 25 26 on transformer-
To investigate this, we conduct series of experiments. The results in Tab, reveal that deeper layers exhibit more semantically meaningful and task-aware attention distributions, suggesting their potential as reliable keyframe indicators. However, directly using deep-layer attention to guide early layers within the same model introduces significant computational overhead for instance, using the deepest layer to guide the first layer nearly doubles inference costs. To this end, new challenge arises: How can we leverage deep-layer attention keyframe priors for dynamic token compression while maintaining computational efficiency?
3 Methodology
3.1
Overview
To address the above issues, we propose DyToK, training-free framework that adaptively allocates per-frame compression ratios by leveraging the keyframe prior obtained from LLM. In Sec. 3.2 DyToK starts by Temporal Importance Estimation, where lightweight assistant model computes frame-level importance scores, achieving decent trade-off between efficiency and performance. Then, in Sec. [3.3] we present Dynamic Frame-Level Compression to distribute the overall token budget across frames according these weights. This two-stage process yields a simple, effective, and general framework that integrates seamlessly with both encoder feature-based and L.M attention- based pruning paradigms. The overall framework of DyToK is demonstrated in Fig. 3
3.2 Temporal Importance Estimation
Given a constrained budget for video token compression, necessary to allocate more tokens to critical frames. Thus, evaluating frame-level temporal importance scores is crucial to determine their relevance the query, such that the token compression can be performed adaptively. achieve this, we leverage cross-men cross-modal attention weights from the LLM quantify frame importance.

---

Temporal Importance Estimation
Visual/Text
Tokens
Tokens
Projector Pooling
Embedding
Primary LLM
User Prompts
Sequence
Sequence
Vision Encoder
Lightweight Assistant
Import-
Temporal
Frame
Framn
Please describe
ance
this detail.
Frame1
Framoti
Dynamic Frame-Level Compression
Figure 3: Illustration of DyToo. We adaptively compress video tokens through two synergistic components: (1) Temporal Importance Estimation leverages cross-modal attention from lightweight assistant model to identify keyframes, followed by (2) Dynamic Frame-Level Compression that proportionally allocates token budgets to preserve salient content. This training- paradigm achieves superior efficiency-accurag tradeoff's by dynamically adjusting compression ratios per frame while maintaining compatibility with diverse token pruning methods.
Specifically, compute attention scores between the last textual query token and visual tokens. Given an input video of F frames, query and key vectors at attention layer are represented as Q RIxD and K, E RVD, respectively. V denotes the number of visual tokens, and D is the emhedding dimension per head. The importance wy of frame f is then computed as:
1 C
QiKq
wf
Softmax
VD
where is the layers considered for attention aggregation.
However, directly utilizing deep-layer attention to guide early layers within the same model introduces significant computational overhead due to the unavoidable re-computation of earlier layers. To address this, we propose leveraging an auxiliary lightweight assistant model derived from the same architectural family as the primary model to facilitate the generation of temporal resampling strategies. Experimental results in Tab. 4 reveal that this compact assistant model achieves nearly identical frame perception accuracy the primary model despite being 14x smaller, while occasionally demonstrating superior keyframe identification performance at certain compression ratios.
Furthermore, as discussed in Sec. 2.2 deeper layers tend to exhibit more semantically meaningful and task-relevant attention distributions, providing superior priors for frame importance evaluation. To leverage this property, aggregate attention scores exclusively from the deep layers in the assistant model. To this end, the calibrated temporal importance score w is obtained as:
w
Softmax
L
VD
LEC
where L represents the selected subset of deep layers. The analysis of different layers, and the comparison between wy and w are provided in Sec. 4.2
3.3 Dynamic Frame-Level Compression
Based the temporal importance scores, in this section, allocate token budgets proportionally achieve frame-level dynamic compression, enhancing efficiency while preserving key information.
Token budget allocation. Specifically, the initial token assignment for each frame computed as a _ x T. where w is the calibrated temporal importance score obtained in Eq. (3). a denotes the preliminary token allocation for frame f, and Tiotal defines the global token budget. Given F' frames within the input video, the set of remaining tokens Trem after the initial allocation is:
F
Tiem Tiotal Σ af.
(4)
f=1
5

---

To fairly distribute these remaining tokens, we calculate fractional remainders ry (w) x [otal) for each frame. Frames are then sorted by descending order of y, and the remaining tokens rem are sequentially allocated to the top-ranked frames until exhausted. This ensures frames that were closest to receiving an additional token in the initial allocation get priority.
Then, to prevent excessive token allocation per frame, tokens exceeding the per-frame limit Timax are reallocated frames with available capacity according to their importance rankings. This yields the final token assignment as: f = min(ay, Tmax), with Σÿ= 0f Tiotal ensuring budget adherence.
Dynamic token compression. To perform token compression given the allocated budget, define a modular compression function Compression(Tj, visual features (e.g., patch tokens) for frame ʃ. and a represents the allocated token budget. This function can instantiate any compatible token pruning strategy, including both encoder feature-based approaches [2 131 and LLM attention- based token pruning methods 9 10 the function is the compressed token sequence 2 for frame f:
Specifically, 2y denotes the raw
The output of
2 Compression(111 (z), ay).
(5)
This design ensures that token reduction is tailored per frame based on its importance, while preserving compatibility with various compression backbones.
As summarized in Alg. | by adaptively allocating more token budget to the task-relevant critical frames, this strategy balances computational efficiency and performance, enabling dynamic adapta- tions to diverse video content characteristics. More implementation details and compatibility with different token compression methods are discussed in Sec. 4.1
Algorithm 1 Adaptive Token Allocation and Compression Based on Frame Weights
Require: Frame data {-,}f=ı- importance weights { total budget Tiotals per-frame token
upper limit Timax
Ensure: Compressed tokens {
1: Initialize allocation: a i Trotal
2: Compute remaining tokens: Trem + Teotal Es af
3: Calculate fractional remainders: +- (wg X Trotal) -
4: Sort frames by descending order of rf
5: while Trem > 0 do
6: for frame f in sorted order do
7: if y < Tmax then
8:
a < a + 1, Tram +- T Trem
9:
if T = then break
end if
11: end if
12: end for
13: end while
14: Redistribute excess tokens to frames below T Tax based on importance
15: for each frame f E (1, do
16: z + Compression(zy,aq)
b Apply dynamic per-frame compression
17: end for
4
Experiments
To evaluate DyToK, we conduct extensive experiments across multiple benchmarks and baseline VLLMs. The evaluation encompasses diverse video understanding tasks and token compression methods to assess both performance and generalizability. The subsequent section details the evaluation tasks, implementation settings, and key findings that align with our method.
4.1 Experimental Setup
We evaluate our method on three widely-used video understanding benchmarks: VideoMME 6 Long VideoBench 77 and MLVU (27], covering durations ranging from minutes to hours. Our

---

#Retained Tokens
VideoMME
LongVideo Video
Average
Method
MLVU
Short
Medium
Long
Ovemil
Bench
Score
%
Vanilla (TXLR)
6272
70.0
56.7
48.8
58.5
56.6
47.1
54.1
100
VisionZip (CVPR2025)
68.9
56.2
48.2
57.8
55.4
45.0
52.7
97.4
VisionZip'
71.4
55.2
49.0
58.6
56.0
46.0
53.5
98.9
4704
+ DyYToK
(L 25%)
48.9
59.0
53.9
46.6
53.8
99.4
(+-2.0)
DyCoke (CVPR2025)
70.9
56.4
48.8
58.7
55.3
47.5
53.8
99.4
+ DyYTK
71.9
56.0
49.1
59.0
56.5
47.7
54.4
100.6
(+1.2)
VisionZip (CVPR2025)
66.8
56.6
48.2
57.2
53.8
43.7
51.6
95.4
VisionZip'
70.3
56.8
49.2
58.8
56.0
45.5
53.4
98.7
3136
+ DyToK
(I 50%)
71.9
56.1
49.3
59.1
56.4
46.2
53.9
99.6
(+4.2)
DyCoke (CVPR2025)
70.0
55.8
49.3
58.4
55,4
46.6
53.5
98.9
+ DyToK
55.9
58.3
57.1
47.5
54.3
100.4
(+1.5)
VisionZip (CVPR2025)
62.2
53.0
47.4
54.2
51.4
41.2
48.9
90.4
VisionZip'
68.8
57.3
48.1
58.1
55.8
44.8
52.9
97.8
1568
+ DyTTK
(4 75%)
56.8
48.9
58.3
55.4
46.3
53.3
98.5
(+8.1)
DyCoke (CVPR2025)
68.2
56.1
47.7
57.3
54.6
43.5
51.8
95.7
+ DyToK
69.6
54.7
47.1
57.1
55.2
45.1
52.5
97.0
(+1.3)
VisionZip (CVPR2025)
60.2
51.0
45.7
52.3
47.7
36.5
45.5
84.1
VisionZip'
67.2
55.9
50.1
57.7
54.8
42.2
51.6
95.4
1120
DyToK
(1 80%)
69.2
56.4
47.8
57.8
56.0
45.2
53.0
98.0
(+13.9)
DyCoke (CVPR2025)
63.4
55.0
47.7
55.4
52.6
45.3
51.1
94.5
+ DyToK
55.6
53.2
44.6
51.1
94.5
VisionZip (CVPR2025)
47.0
44.2
42.2
44.5
41,4
29.8
38.6
71.3
VisionZip"
54,9
49.7
44.9
49.8
47.8
37.6
45.1
83.4
DyToK
(L 90%)
61.0
51.1
47.6
53.2
50.4
42.8
48.8
90.2
(+18.9)
DyCoke (CVPR2025)
64.0
51.4
45.9
53.8
50.4
40.9
48.4
89.5
DyToK
62.3
51.6
46.6
53.5
50.2
43.6
49.1
90.8 (+1.3)
Table 1: Performance of DyToK integrated into encoder feature-based methods. The vanilla setup processes 32 frames with 196 tokens each. The final column reports the average score across benchmarks and the accuracy relative to the unpruned baseline. VisionZip' denotes our pooling- compatible variant, and DyCoke here applies pruning exclusively the encoder side.
evaluation follows the standard settings LMMs-Eval (28). Due to space limitations, we present only key results for LLaVA-One Vision [1 in the main text. To validate generalizability across different VLLM inference acceleration paradigms, we integrate DyToK into three state-of-the-art methods: FastV (8] (LLM abaentton-based),, VisionZip [12 (encoder feature-based), and DyCoke [13 (hybrid pruning). For fair comparison, we align computational budgets to ensure equivalent FLOPs across methods (Appendix G.2p. Comprehensive results are provided in the Appendix, including full experiments on 32-frame LLaVA-One Vision (Appendix A.II, evaluations of Qwen2.5 LL (Appendix A.2). analyses with extended video lengths (Appendix A.3) studies on broader model sizes (Appendix A.4), efficiency analysis (Appendix C). and implementation details (Appendix G
4.2 Main Results
Effectiveness encoder feature-based methods. We first evaluate DyToK encoder feature- based VisionZip and DyCoke (encoder), which performs static token selection using encoder features. Notably, since VisionZip originally discards the spatial information of retained visual tokens, it is incompatible with 2D pooling methods, which are widespread in today's 's VLLMs. To address this limitation, adapt VisionZip by deferring pruning until completely passing through the projection and pooling modules. Specifically, inter-patch correlations among pooled tokens serve as the pruning metric. We denote our improved variant as VisionZip'.
We report the average score all the benchmarks and also in percentage format for comparative analysis, with the vanilla model's accuracy serving as the 100% upper limit to comprehensively assess the performance. As shown in Tab. I DyToK improves VisionZip by 4.2% average accuracy across
7

---

#Retained Tokens
VideoMME
Video
Average
Method
MIVV
Short
Medium
Long
Overall
Bench
Score
%
Vanilla (TXLR)
6272
70.0
56.7
48.8
58.5
56.6
47.1
54.1
100.0
FastV (BCDV2024)
69.7
55.7
47.6
57.6
57.1
46.5
53.7
99.3
DyToK
70,4
56.7
48.2
58.4
56.8
46.8
54.0
99.8
(+0.5)
4704
DyCoke (CVPR2026)
(1 25%)
70.9
56.4
48.8
58.7
55.3
47.5
53.8
99.4
DyToK
70.9
55.9
49.0
58.6
55,9
47.5
54.0
99.8 (+0.4)
FastV (BCCV2024)
69.1
55.4
47.1
57.2
57.1
44.7
53
98.0
DyToK
71.1
55.6
48.8
58.5
57.2
46.3
54.0
99.8
(+1.8)
3136
DyCoke (CVPR2025)
70.0
55.8
49.3
58.4
55.4
46.6
53.5
98.9
50%
DyToK
71.0
56.1
49.2
58.5
56.0
46.9
53.8
99.4 (++0.5
FastV (ECCV2024)
66,2
54.7
47.1
56.0
56.6
43.7
52.1
96.3
DyToK
67.7
54.0
47.1
56.3
55.7
47.8
53.3
98.5 (+2222
1568
DyCoke (CVPR2025)
68.2
56.1
47.7
57.3
54,6
43.5
51.8
95.7
(1 75%)
DyTTK
67.1
55.8
48.3
57.1
54.8
45.4
52.4
96.9 (+1.2)
FastV (ECCV2024)
58.0
51.7
43.6
51.1
51.2
38.3
46.9
86.7
DyToK
64.6
54.2
45.8
54.8
52.6
43.2
50.2
92.8 (+6.1)
896
DyCoke (CVPR2026)
65.1
53.6
45.3
54.7
52.2
42.5
49.8
92.1
(4 85%)
DyTTK
66.1
52.8
46.3
55.1
52.4
42.9
50.1
92.6 (+0.5)
DyCoke (CVPR2026)
448
64.0
51.4
45.9
53.8
50.4
40.9
48.4
89.5
+DD DyTTo
(1 90%))
64.9
52.3
45.4
54.2
50.4
41.2
48.6
89.8 (+0.3)
Table Performance of Dy DyToK integrated into LLM attention-based methods. The evaluation follows the same setup above. DyCoke here only applies pruning at the LLM side.
benchmarks 50% compression, demonstrating that our dynamic keyframe aware compression complements static fcature-based approaches. The performance gap widens as compression ratios increase (18.95 9% improvement 90% compression), confirming that temporal importance awareness becomes critical under aggressive pruning.
Effectiveness on LLM attention-based methods. To enhance the compatibility of the proposed method with a broader range of existing pruning techniques, we also evaluate DyToK on LLM attention-basee based Methods, such as FastV and DyCoke (LLM). Following the same experimental setup. we report both the average score and the percentage relative to the unpruned LLaVA-One Vision baseline. As shown Tab. 2, DyToK improves performance across both pruning rates. an 85% compression rate, DyToK improves FastV by 6.1% % in accuracy across benchmarks, further demonstrating the effectiveness and robustness of our method. Notably, 90% pruning rate, our token budget allocation (Appendix DyToKK with no tokens to reallocate, so results for this case are omitted.
G.2) reduces FastV's 's retained visual tokens to zero and leaves
4.3 Ablation Studies
Effect of layer-level attention location on keyframe prior. As hypothesized in Sec. 2.2 we
conduct detailed analysis further illustrate this core finding. To evaluate the impact of different layers' attention patterns on keyframe prior, we fix the retention ratio at 20%-a moderate setting that effectively reduces redundancy without being overly aggressive. As shown in Tab. we can clearly observe that deeper layers provide significantly better keyframe priors compared to shallower ones, with layers 20 and achieving the best performance. The observed trend aligns with our hypothesis: model performance improves deeper layers are used for keyframe prior extraction, which further proves the effectiveness of our method. To ensure generalizability without manual tuning, we uniformly average the attention scores from the last one-third of layers for all models in practical implementations. Please refer to Appendix (A.6 for further details.
Effect of the model size on keyframe prior. As discussed in Sec. 2.2 using the full-size model to extract attention maps for token pruning is computationally expensive. We therefore evaluate whether a smaller model in the same VL.LM family can assist pruning. We compare the pruning performance of the standard 7B model with that of its smaller counterpart across several video understanding tasks. Tab, shows that the smaller model achieves comparable performance with significantly lower

---

VideoMME
LongVides Bench
Average
Layer
MLVU
Short
Medium
Long
Overall
Score
%
Vanilla (TMLS)
70.0
56.7
48.8
58.5
56.6
47.1
54.1
100.0
0
62.6
53.1
46.4
54.0
52.0
40.8
48.9
90.5
4
63.6
53.3
49.0
55.3
53.5
42.3
50.4
93.1
8
64.1
53.4
47.3
55.0
53.4
41.5
50.0
92.3
12
65.7
54.9
47.8
56.1
53.3
43.3
50.9
94.1
16
66.2
55.3
48.2
56.6
53.4
44.9
51.6
95.4
20
68.1
56.6
47.8
57.5
55.5
46.3
53.1
98.2
23
66.3
56.3
48.7
57.1
53.8
45.1
52.0
96.1
Table 3: Performance of DyToK under different layer configurations. We conduct experiments using retention ratio of 20%, with the number of frames set to 32. In comparison, the baseline LLa VL-VO- Vision retains its original configuration without any pruning or modification. The best result highlighted in red, while the second-best shown in bold.
#Retained Tokens
VideoMME
Long Video Bench
Average
Method
MLVU
Short
Medium
Long
Overall
Score
%
Vanilla (THLR)
6272
70.0
56.7
48.8
58.5
56.6
47.1
54.1
100.0
Base
4704
71.9
56.8
48.8
59.1
55.7
47.5
54.1
100.0
Tiny
(1 25%)
70.8
57.2
48.9
59.0
55.9
46.6
53.8
99.4 (-0.6)
Base
3136
71.8
57.2
49.0
59.3
55.9
46.0
53.7
99.3
Tiny
50%
71.9
56.1
49.3
59.1
56.4
46.2
53.9
99.6 (+0.33
Base
1568
70.4
57.1
48.1
58.6
57.2
46.6
54.1
100.0
Tiny
() 75%)
69.2
56.8
48.9
58.3
55.4
46.3
53.3
98.5 (-1.5)
Base
1120
70.3
56.4
48.1
58.3
55.8
46.5
53.5
98.9
Tiny
(L 80%)
69.2
56.4
47.8
57.8
56.0
45.2
53.0
98.0 (-0.9)
Base
448
63.9
52.7
47.3
54.6
52.7
41.5
49.6
91.7
Tiny
(1 90%)
61.0
51.1
47.6
53.2
50.4
42.8
48.8
90.2 (-1.5)
Table Performance of Dy with different LLM sizes for keyframe prior. The Base setting uses LLa Visionn- to provide the keyframe prior, while the Tiny setting employs LLaVA. VA- Visionnn- for the same purpose. All experiments are conducted with 32 input frames.
computational cost (14x) while incurring minimal performance degradation. More ablations on token allocation upper limit (Appendix (A.7), importance estimation strategies (Appendix (A8). and textual token selection (Appendix A.9 are detailed in the Appendix.
Concluding Remarks
Summary. this paper, we analyze attention maps in VLLMs and observe a strong correlation between attention scores and keyframes, along with the insight that deeper layers provide more effective priors for keyframe selection. Building on these findings, we propose Dy ToK, a simple yet effective method that leverages the inherent priors of LLMs to reduce vision tokens while preserving video understanding capabilities. DoKo is a plug-and aad--ndd-plyy method compatible with both attention- and encoder-based approaches, demonstrating broad applicability across different architectures.
Limitation and future work. Although employing a lightweight assistant model to approximate the original model for keyframe prior generation improves performance while ensuring efficiency, this work has not yet to propose a better method avoid introducing additional models. Future efforts will focus further optimization to address this issue.

---

Acknowledgement. This work was supported by Guangdong Basic and Applied Basic Re- search Foundation (2025A151501 by Shenzhen Science and Technology Innovation Program (CCY200248811005500 KIZID20240003 10000900000000000000 and by Science and Technology Project of Shenzhen 1WW.DD220220011110000000000
References
[1] Bo Li, Yuanhan Zhang, Dong Guo, Renrui Zhang. Feng Li, Hao Zhang, Kaichen Zhang, Peiyuan Zhang, Yanwei Li, Ziwei Liu, et al. Llava-onevision: Easy visual task transfer. arXiv preprint arXiv:2408.0333 2024.
[2] Yuanhan Zhang, Jinming Wu, Wei Li, Bo Li, Zejun Ma. Ziwei Liu, and Chunyuan Li. Video instruction tuning with synthetic data. arXiv preprint arXiv:2410.02713, 2024.
[3] Shuai Bai. Kegin Chen, Xuejing Liu, Jialin Wang, Wenbin Ge, Sibo Song, Kai Dang, Peng Wang. Shijie Wang, Jun Tang, et al. Qwen2.5-vl technical report. arXiv preprint arXiv:2502. 13923, 2025.
[4] Yi Wang. Xinhao Li, Ziang Yan, Yinan He, Jiashuo Yu, Xiangyu Zeng, Chenting Wang, Changlian Ma, Haian Huang. Jianfei Gao, al. Internvide22. 5: Empowering video milms with long and rich context modeling. arXiv preprint arXiv:2501. 12386, 2025.
[5] Keshigeyan Chandrasegaran, Agrim Gupta, Lea M Hadzic, Taran Kota, Jimming He, Cristóbal Eyzaguirre, Zane Durante, Manling Li, Jiajun Wu, and Fei-Fei Li. Hourvideo: 1-hour video-language understanding. Advances in Neural Information Processing Systems, 2024.
[6] Chaoyou Fu, Yuhan Dai, Yongdong Luo, Lei Li, Shuhwai Ren, Renrui Zhang, Zihan Wang, Chenyu Zhou, Yunhang Shen, Mengdan Zhang, et al. Video-mme: The first-ever comprehensive evaluation benchmark of multi-modal ilms in video analysis. arXiv preprint arXiv:2405.21075, 2024.
[7] Haoning Wu, Dongxu Li, Bei Chen, and Junnan Li. Longvideobench: A benchmark for long-context interleaved video-language understanding. Advances in Neural Information Processing Systems, 2024.
[8] Liang Chen, Haozhe Zhao, Tianyu Liu, Shuai Bai, Junyang Lin, Chang Zhou, and Baobao Chang. An image is worth 1/2 tokens after layer 2: Plug-and-play inference acceleration for large vision-language models. In European Conference Computer Vision. Springer, 2024.
[9] Shaolei Zhang, Qingkai Fang. Zhe Yang, and Yang Feng. Llava- mini: Efficient image and video large multimodal models with one vision token. arXiv preprint arXiv:2501 250.:99500030955 2025.
[10] Xiaohu Huang. Hao Zhou, and Kai Han. Prunevid: Visual token pruning for efficient video large language models. arXiv preprint mXXvv:2412.1611 2024.
[11] Yanwei Li, Chengyao Wang. and Jiaya Jia. Llama-vid: An image worth tokens in large language models. In European Conference on Computer Vision, 2024.
[12] Senqiao Yang. Yukang Chen, Zhuotao Tian, Chengyao Wang, Jingyao Li, Bei Yu, and Jiaya Jia. Visionzip: Longer is better but not necessary in vision language models. arXiv preprint arXiv:2412.0445 2024.
[13] Keda Tao, Can Qin, Haoxuan You, Yang Sui, and Huan Wang. Dycoke: Dynamic compression of tokens for fast video large language models. arXiv preprint arXiv:2411.15024 2024.
[14] Wangbo Zhao, Yizeng Han, Jiasheng Tang, Zhikai Li, Yibing Song, Kai Wang. Zhangyang Wang, and Yang You. A stitch in time saves nine: Small vlm is a precise guidance for accelerating large vims. arXiv preprint arXiv:2412.03324 2024.
[15] Zhenwei Shao, Mingyang Wang, Zhou Yu, Wenwen Pan, Yan Yang. Tao Wei, Hongyuan Zhang, Ning Mao, Wei Chen, and Jun Yu. Growing twig accelcrate large vision-language models. arXiv preprint arXiv:2503.1403 2025.
[16] Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, al. Learning transferable visual models from natural language supervision. In International conference on machine learning, 2021.
[17] Xiaohua Zhai, Basil Mustafa, Alexander Kolesnikov, and Lucas Beyer. Sigmoid loss for language image pre-training. In Proceedings of the IEEE/VF international conference computer vision, 2023.
[18] Kai Hu, Feng Gao, Xiaohan Nie, Peng Zhou, Son Tran, Tal Neiman, Lingyun Wang, Mubarak Shah, Raffay Hamid, Bing Yin, et al. M-Ilm based video frame selection for efficient video understanding. arXiv preprint 2rXiv:2502. 19680, 2025.
10

---

[19] Xi Tang, Jihao Qiu, Lingxi Xie. Yunjie Tian, Jianbin Jiao, and Qixiang Ye. Adaptive keyframe sampling for long video understanding. arXiv preprint arXiv:2502.2127,, 2025.
[20] Bin Lin, Yang Ye. Bin Zhu, Jiaxi Cui, Munan Ning. Peng Jin, and Li Yuan. Video-Ilava: Learning united visual representation by alignment before projection. arXiv preprint arXiv:2311.10122 2023.
[21] Yi Wang, Kunchang Li, Xinhao Li, Jiashuo Yu, Yinan He, Guo Chen, Baoqi Pei. Rongkun Zheng, Zun Wang. Yansong Shi, et al. Internvido02: Scaling foundation models for multimodal video understanding. In European Conference Computer Vision, 2024.
[22] Yuan Zhang, Chun-Kai Fan, Junpeng Ma, Wenzhao Zheng, Tao Huang, Kuan Cheng, Denis Gudovskiy, Tomoyuki Okuno, Yohei Nakata, Kurt Keutzer, al. Sparsevlm: Visual token sparsification for efficient vision-language model inference. arXiv preprint arXiv:2410.04417, 2024.
[23] Yuetian Weng. Mingfei Han, Haoyu He, Xiaojun Chang. and Bohan Zhuang. Longvlm: Efficient long
video understanding via large language models. European Conference on Computer Vision, 2024.
[24] Xiaoqian Shen, Yunyang Xiong, Changsheng Zhao, Lemeng Wu, Jun Chen, Chenchen Zhu, Zechun Liu, Fanyi Xiao, Balakrishman Varadarajan, Florian Bordes, et Longvu: Spatiotemporal adaptive compression for long video- Ladoo-lag understanding. arXiv preprint arXiv:2410. 2024.
[25] Mingyu Jin, Qinkai Yu, Jingyuan Huang, Qingcheng Zeng, Zhenting Wang, Wenyue Hua, Haiyan Zhao, Kai Mci, Yanda Meng, Kaize Ding, Fan Yang. Mengnan Du, and Yongfeng Zhang. Exploring concept depth: How large language models acquire knowledge and concept at different layers? In International Conference on Computational Linguistics, 2025.
[26] Wei Shi, Sihang Li, Tao Liang, Mingyang Wan, Gojun Ma, Xiang Wang, and Xiangnan He. Route sparse autoencoder to interpret large language models. ArXiv, 2025.
[27] Junjie Zhou, Yan Shu, Bo Zhao, Boya Wu, Shitao Xiao, Xi Yang. Yongping Xiong, Bo Zhang, Tiejun Huang, and Zheng Liu. Mlvu: A comprehensive benchmark for multi-task long video understanding. arXiv preprint ar 2iw.::22200.266... 2024.
[28] Kaichen Zhang, Bo Li, Peiyuan Zhang, Fanyi Pu, Joshua Adrian Cahyono, Kairui Hu, Shuai Liu, Yuanhan Zhang, Jingkang Yang, Chunyuan Li, et al. Lmms-eval: Reality check on the evaluation of large multimodal models. arXiv preprint 1tti:22000.127722 2024.
[29] Muhammad Maaz, Hanoona Rasheed. Salman Khan, and Fahad Shahbaz Khan. Video-chatgpt: Towards detailed video understanding via large vision and language models. arXiv preprint 04X442230.004224 2023.
[30] Han Wang. Yuxiang Nie, Yongjie Ye, Deng YuanYu, Yanjie Wang, Shuai Li, Haiyang Yu, Jinghui Lu, and Can Huang. Dynamic-vim: Simple dynamic visual token compression for videollm. arXiv prepriat arXiv:2412.09530, 2024.
[31] Timothée Darcet, Maxime Oquab, Julien Mairal, and Piotr Bojanowski. Vision transformers need registers. arXiv preprint 1XXi:23399.16888 2023.
[32] Tong Shao, Zhuotao Tian, Hang Zhao, and Jingyong Su. Explore the potential of clip for training frree open vocabulary semantic segmentation. In European Conference on Computer Vision, pages 139-156. Springer, 2024.
[33] Junjie Wang, Bin Chen. Yulin Li, Bin Kang. Yichi Chen, and Zhuotao Tian, Declip: Decoupled learning for open-vocabulary dense perception. In Proceedings of the Computer Vision and Pattern Recognition Conference, pages 14824-14834, 2025.
[34] Guangxuan Xiao, Yuandong Tian, Beidi Chen, Song Han, and Mike Lewis. Efficient streaming language models with attention sinks. arXiv preprint arXiv:2309.1 (1ti::2309.17453,, 17453, 2023.
[35] Junjic Wang, Keyu Chen, Yulin Li, Bin Chen, Hengshuang Zhao, Xiaojuan Qi, and Zhuotao Tian. Generalized decoupled learning for enhancing open-vocabulary dense perception. arXiv preprint arXiv:2508.1 1256, 2025.
[36] Hugo Touvron, Thibaut Lavril, Gautier Izacard, Xavier Martinet, Marie-Anne Lachaux, Timothée Lacroix, Baptiste Rozière, Naman Goyal, Eric Hambro, Faisal Azhar, Aur' elien Rodriguez, Armand Joulin, Edouard Grave, and Guillaume Lample. Llama: Open and efficient foundation language models. ArXiv, 2023.
[37] An Yang, Baosong Yang. Beichen Zhang, Binyuan Hui, Bo Zheng. Bowen Yu, Chengyuan Li, Dayiheng Liu, Fei Huang. Haoran Wei, et al. Qwen2. technical report. arXiv preprint 1551 2024.
11

---

[38] OpenAl Josh Achiam, Steven Adler, Sandhini Agarwal, Lama Ahmad, Ilge Akkaya, Florencia Leoni Aleman, Diogo Almeida, Janko Altenschmidt, and et al. Sam Altman. Gpt-4 technical report. 2023.
[39] Gemini Team, Rohan Anil, Sehastian Borgeaud, Jean- Alayrac, Jiahui Yu, Radu Soricut, Johan Schalkwyk, Andrew M. Dai, and et al. Anja Hauth. Gemini: A family of highly capable multimodal models, 2025.
[40] Junjie Wang. Bin Chen, Bin Kang, Yulin Li, Weizhi Xian, Yichi Chen, and Yong Xu. Ov-dquo: Open- vocabulary detr with denoising text query training and open- unknown objects supervision. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 39, pages 7762-7770, 7770, 2025. 2025.
[41] Bin Kang, Bin Chen, Junjie Wang. Yulin Li, Junzhi Zhao, Junle Wang, and Zhuotao Tian. Calibelip: Contextual calibration of dominant semantics for text-driven image retrieval. In Proceedings of the 33rd ACM International Conference on Multimedia, 2025.
[42] Junnan Li, Dongxu Li. Silvio Savarese, and Steven C. H. Hoi. Blip-2: Bootstrapping language-iman immge pre-training with frozen image encoders and large language models. In International Conference on Machine Learning, 2023.
[43] Haotian Liu, Chunyuan Li, Qingyang Wu, and Yong Jae Lee. Visual instruction tuning. ArXiv, abs/2304.08485, 2023.
[44] Haotian Liu. Chunyuan Li, Yuheng Li, Bo Li, Yuanhan Zhang, Sheng Shen, and Yong Jae Lee. Llava-next: Improved reasoning, ocr, and world knowledge, January 2024.
[45] Haicheng Wang. Zhemeng Yu, Gabriele Spadaro, Chen Ju, Victor Qu' etu, and Enzo Tartaglione. Folder: Accelerating
large language models with enhanced performance. ArXiv, 2025.
[46] Saeed Ranjbar Alvar, Gursimran Singh, Mohammad Akbari, and Yong Zhang. Divprune: Diversity-based visual token pruning for large multimodal models. ArXiv, 2025,
[47] Yuzhang Shang. Mu Cai, Bingxin Xu. Yong Jae Lee, and Yan Yan. Llava-prumerge: Adaptive token reduction for efficient large multimodal models. ArXiv, 2024.
[48] Bingxin Xu, Yuzhang Shang, Yunhao Ge, Qian Lou, and Yan Yan. freepruner: A training-free approach for large multimodal model acceleration. ArXiv, 2024.
[49] Zichen Wen, Yifeng Gao. Shaobo Wang, Junyuan Zhang, Qintong Zhang, Weijia Li, Conghui He, and Linfeng Zhang. Stop looking for important tokens in multimodal language models: Duplication matters more. ArXiv, 2025.
[50] Leqi Shen, Guoqiang Gong, Tao He, Yifeng Zhang, Pengzhang Liu, Sicheng Zhao, and Guiguang Ding. Fastvid: Dynamic density pruning for fast video large language models. ArXiv, 2025.
[51] Sicheng Yu, Chengkai Jin, Huan Wang, Zhenghao Chen, Sheng Jin, Zhongrong Zuo, Xioalei Xu, Zhenbang Sun, Bingni Zhang, Jiawei Wu, Hao Zhang, and Qianru Sun. Frame-voyager: Learning to query frames for video large language models. ArXiv, 2024.
[52] Hao Liang, Jiapeng Li, Tianyi Bai, Xijie Huang, Linzhuang Sun, Zhengren Wang, Conghui He, Bin Cui. Chong Chen, and Wentao Zhang. Keyvideollm: Towards large-scale video keyframe selection. ArXiv, 2024.
[53] Wei Han, Hui Chen, MingSung Kan, and Soujanya Poria. Self-adaptive sampling for efficient video crestion-answering on image text models. 2023.
[54] Yongdong Luo, Wang Chen, Xiawu Zheng, Weizhong Huang. Shukang Yin, Haojia Lin, Chaoyou Fu, Jinfa Huang, Jiayi Ji, Jiebo Luo, and Rongrong Ji. Quota: Query-oriented token assignment via cot query decouple for long video comprchension. ArXiv, 2025.
[55] Karttikeya Mangalam, Raiymbek Akshulakov, and Jitendra Malik. Egoschema: A diagnostic benchmark for very long-form video language understanding. Advances in Neural Information Processing Systems. 2023.
[56] Anurag Arnab, Mostafa Dehghani, Georg Heigold, Chen Sun, Mario Lucic, and Cordelia Schmid. Vivit: A video vision transformer. 2021 JEEE/CVF International Conference on Computer Vision (CCVV), 2021.
[57] Ze Liu, Jia Ning, Yue Cao, Yixuan Wei, Zheng Zhang, Stephen Lin, and Han Hu. Video swin transformer. 2022 JEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), 2021.
12

---

Supplementary Material
Contents
More Experimental Results
14
A.1
Full Experiments on 32 32-Frame LLa VLa Vision
14
A.2
Experiments Qwen2.5-VI
14
A.3
Experiments on Extended Video Length
17
A.4
Experiments on Broader Model Size
19
A.S
Experiments on General Descriptive Queries
19
A.6
Ablation on Keyframe Prior Layer Selection
19
A.7
Ablation on Token Allocation Upper Limit
20
A.8
Ablation on Importance Estimation Strategies
22
A.9
Ablation on Textual Token Selection
22
Detailed Analysis of Keyframe Prior in VLLMS
23
B.
Temporal Attention Outlier Phenomenon
23
B.2
Visualizing and Characterizing Temporal Attention Outliers ................
27
B.3
Exploring Keyframe Bias VLLMS
27
Efficiency Analysis
28
Related Works
29
Efficient inference paradigms for VLLMS
30
Evaluation Benchmarks
Implementation Details
31
G.1 Experimental Settings
Token Budget Alignment
31
Visualization of Per-Framm Token Compression
33
13

---

10%
15%
SOTA, DyColi
49.8
$3.3
20%
34g
25%
38.0
43.0
45.3
540
5096
34.3
48.9
1300
51.5
53.3
511
54.1
300
$1
4669
51.3
53.5
70%
75%
25%
50%
Figure 4: Performance gains of DyToK under various retention ratios. Performance comparison of SOTA acceleration methods with and without DyToK on LLLVA-OmeVision under 32-frame input. Experiments conducted on VideoMME, Long VideoBench, and MLVU across varying retention ratios show that integrating DyToK consistently improves accuracy, demonstrating its effectiveness in en- hancing long-video understanding. The scores presented in the figure represent average performance across the three benchmarks. For detailed results, please refer to Tab. 5 and Tab. 6
A More Experimental Results
A.1 Full Experiments on 32-Frame LLaVA-OmeVision
In the Tab. Tand Tab. of the main text, evaluate the effectiveness of DyToK using lightweight assistant model to generate keyframe priors for token pruning under a 32-frame setting with LLaVA- Vision. To further validate the generalizability and efficacy of DyToK, this section expands upon those findings by employing the primary model's attention scores directly as temporal importance indicators dynamically allocate token budgets. Experiments are conducted across three widely recognized benchmarks: VideoMME, Long VideoBench, and MLVU. Results corresponding to identical experimental conditions in the main text are aligned for consistency.
As demonstrated Tab. and Tab. 6 DyToK consistently enhances performance across different VLLM inference acceleration paradigms, specifically encoder feature-based and LLM attention- based methods. Under relatively moderate token reduction ratios (25%-50%). DyToK improves upon baseline pruning methods by effectively preserving keyframe tokens, resulting in notable gains of up to 4.2% (VisionZip. 50% token reduction) and up to 2.4% (FastV, 50% token reduction) in average scores. Notably, more aggressive compression settings (75%-90% token reductions), the advantage of DyToK becomes even more pronounced, delivering substantial accuracy improvements up to 20.4% for VisionZip at 90% reduction and 9.6% 75% reduction, clearly highlighting DyToK's capability to significantly mitigate accuracy degradation under extreme token pruning scenarios.
A.2 Experiments on Qwen2.5VVL
To further demonstrate the generalization capability of our method, conduct experiments the recently released Qwen2.5-VL [ model, which has gained prominence the current VLM community. This model introduces architectural components not aligned with previous pruning methods, including sliding window attention and 3D convolutions- both which are rarely present in earlier models. As result, adapt the logic of existing state-of-ther the-art pruning methods to suit this architecture. The experiments are conducted using uniform sampling of 32 frames with the Qwen2.5-VL. 7B model. Detailed results presented in Tab. 7
Notably, across retention ratios when used with FastV, our method achieves a substantial per- formance gain exceeding 10 points in most settings. Remarkably, it even outperforms the vanilla
14

---

VideoMME
LongVideo Bench
Average
Method
MLVU
Short Medium Long Overall
Score
%
Upper Bound, 6272 tokens
Vanilla (THLR)
70.0
56.7
48.8
58.5
56.6
54.1
100
Retain 4704 tokens () 25%)
VisionZip (CVPR2025)
68.9
56.2
48.2
57.8
55.4
45.0
52.7
97.4
VisionZip'
71.4
55.2
49.0
58.6
56.0
46.0
53.5
98.9
DyToK
70.8
57.2
48.9
59.0
55.9
46.6
53.8
99.4
(+2.0)
DyToK (7B)
71.9
48.8
55.7
47.5
54.1
100
(-2.6)
DyCoke (CVPR2025)
70.9
56.4
48.8
58.7
55.3
47.5
53.8
99.4
DyToK
71.9
56.0
49.1
59.0
56.5
47.7
54.4
100.6
(+1.2)
DyToK (7B)
72.0
55.9
49.0
59.0
56.3
47.6
54.3
100.4
(+1.0)
Retain 3136 tokens (1 50%)
VisionZip (CVPR2025)
66.8
56.6
48.2
$7.2
53.8
43.7
51.6
95.4
VisionZip'
70.3
56.8
49.2
58.8
56.0
45.5
53.4
98.7
DyYTK
71.9
56.1
49.3
59.1
56.4
46.2
53.9
99.6
(+-4.2)
DyToK (7B)
71.8
57.2
49.0
59.3
55.9
46.0
53.7
99.3
(+-3.)
DyCoke (CVPR2025)
70.0
55.8
49.3
58.4
55.4
46.59
53.5
98.9
+ DyToK
70.2
55.9
48.9
58.3
57.1
47.5
54.3
100.4
(+1.5)
+DyToK(7B)
71.0
56.7
49,2
59.0
56.6
47.1
54.2
100.2
(+1.3)
Retain 1568 tokens (L 75%)
VisionZip (CVPR2025)
47.4
54.2
51.4
41.2
48.9
90.4
VisionZip'
68.8
57.3
48.1
58.1
55.8
44.8
52.9
97.8
+ DyToK
69.2
56.8
48.9
58.3
55.4
46.3
53.3
98.5
(+8.1)
DyToK (7B)
70.4
57.1
48.1
58.6
57.2
46.6
54.1
100
(-9.6)
DyCoke (CVPR2025)
68.2
56.1
47.7
57.3
54.6
43.5
51.8
95.7
DyToK
69.6
54.7
47.1
55.2
52.5
97.0 (+1.3)
+ +DTToK7BB)
68.0
54.9
47.2
56.7
53.7
45.6
52.0
96.1 (+0.4)
Retain 120 tokens (L 80%)
VisionZip (CVPR2025)
60.2
51.0
45.7
52.3
47.7
36.5
45.5
84.1
VisionZip'
67.2
55.9
50.1
57.7
54.8
42.2
51.6
95.4
+ &yyToK
69.2
56.4
47.8
57.8
56.0
45.2
53.0
98.0
(+13.9)
+ DyToK (7B)
70.3
56.4
48.1
58.3
55.8
46.5
53.5
98.9
(+14.8)
DyCoke (CVPR2025)
63.4
55.0
47.7
55.4
52.6
45.3
51.1
94.5
DyToK
65.1
54.2
47.3
55.6
53.2
44.6
51.1
94.5 (+0.0)
+ DyToK (7B)
53.4
46.7
55.6
52.4
44.9
94.3 (-0.2)
Retain 448 tokeny (L 90%)
VisionZip (CVPR2025)
47.0
44.2
42.2
44.5
41.4
29.8
38.6
71.3
VisionZip'
54.9
49.7
44,9
49.8
47.8
37.6
45.1
83.4
DyToK
61.0
51.1
47.6
53.2
50.4
42.8
48.8
90.2
(+18.9)
DyToK (7B)
$2.7
47.3
$4.6
52.7
41.5
49.6
91.7
(+200.4)
DyCoke (OVP82025)
64.0
51.4
45.9
53.8
50.4
40.9
48.4
89.5
DyToK
62.3
51.6
46.6
53.5
50.2
43.6
49.1
90.8
(+1.3)
DyToK (7B)
61.4
52.1
46.1
53.2
50.6
41.3
48.4
89.5
(+0.0)
Table 5: Full experiments of Dy'ToTTo ToK integrated into encoder feature-based methods on 32-frames
LLa LLaVA- VA-One Vision. The vanilla setup processes 32 frames with 196 tokens each. The final column
reports the average score across benchmarks and the accuracy relative to the unpruned baseline. VisionZip" denotes our compression ratio allocation exclusively at the encoder side.
variant, and DyCoke here applies dynamic per-frame
15

---

VideoMME
LongVideo Video Bench
Average
Method
MLVU
Short Medium Long
Overall
Score
Upper Bound, 6272 tokens
Vanilla (TNL))
70.0
56.7
48.8
58.5
56.6
47.1
54.1
100
Retain 4704 tokens
25%)
FastV (BCCV2024)
69.7
47.6
57.6
57.1
46.5
99.3
+ DyToK
70.4
56.7
48.2
58.4
56.8
46.8
54.0
99.8
(+0.5)
+ DyToK (7B)
70.1
55.9
48.4
58.1
57.6
46.4
54.0
99.8 (+0.5)
DyCoke (CVPR2025)
70.9
56.4
48.8
58.7
55.3
47.5
53.8
99.4
DyToK
70.9
55.9
49.0
58.6
55.9
47.5
54
99.8
(+-..4)
+ DyToK (7B)
70.9
55.7
58.4
55.7
47.7
53.9
99.6 (+0.22
Retain 3136 tokens (L 50%)
FastV (ECCV2024)
69.1
55.4
47.1
57.2
57.1
44.7
53
98.0
DyToK
71.1
55.6
48.8
58.5
57.2
46.3
54
99.8
(+11.)
+ DyToK (7B)
56.3
47.6
58.3
57.4
47.1
54.3
100.4
(+2.4)
DyCoke (CVPR2025)
70.0
55.8
49.3
58.4
55.4
46.6
53.5
98.9
+ DyToK
71.0
56.1
49.2
58.5
56.0
46.9
53.8
99.4 (+0.5)
+ DyToK (7B)
70.7
55.9
48.9
58.5
56.1
46.7
53.8
99.4 (+0.5)
Retain 1568 tokens
75%)
FastV (BCCV2024)
66.2
54.7
47.1
56.0
56.6
43.7
52.1
96.3
+ DyToK
67.7
54.0
47.1
56.3
55.7
47.8
53.3
98.5
(+-2.2)
+ DyToK (7B)
69.4
56.1
47.2
57.6
56.0
45,8
53.1
98.2
(+1.9)
DyCoke (CVPR2025)
68.2
56.1
47.7
57.3
54.6
43.5
51.8
95.7
+ DyToK
55.8
48.3
57.1
54.8
45.4
52.4
96.9
(+1.2)
+ DyToK (7B)
68.6
55.2
49.0
57.6
55.2
46.2
98.0 (+1.1)
Retain 896 tokens (1 85%)
FastV (BC0V2024)
58.0
51.7
43.6
51.1
51.2
38.3
46.9
86.7
DyToK
64.6
54.2
45.8
54.8
52.6
43.2
50.2
92.8
(+6.1)
+ DyToK (7B)
56.1
54.0
44.0
95.0 (+8.3)
DyCoke (CVPR2025)
65.1
53.6
45.3
54.7
52.2
42.5
49.8
92.1
+ DyToK
66.1
52.8
46.3
55.1
52.4
42.9
50.1
92.6
(+0.5)
+ DyToK (7B)
65.4
52.7
46.3
54.8
52.7
41.9
49.8
92.1 (+0.0)
Retain tokens (L 90%)
DyCoke (CVPR2025)
64.0
51.4
45.9
53.8
50.4
40.9
48.4
89.5
+ DyToK
64.9
52,3
45.4
54.2
50.41
41.2
48.6
89.8
(+0.3)
+ DyToK (7B)
64.3
51.9
45.1
53.8
51.3
42.5
49.2
90.9 (+1.4)
Table 6: Full experiments of DyToK integrated into LLM attention-based methods on 32-frames The evaluation follows the same setup as above. DyCoke here only applies dynamic per- fer-frame compression ratio allocation at the LLM side.
LLa
16

---

Average
Method
VidcoMME Long LongVideoBench MLVU
Score
%
Upper Bound
Vanilla (TMLR)
60.7
57.7
44.7
54.4
Retain 75% tokens
VisionZip
59.7
56.5
44.6
53.6
98.5
+ DyToTK (7B)
59.3
57.0
43.1
53.1
97.6
(-0.9)
FastV (BCDV2024)
58.7
56.1
41.5
52.1
96.3
+ DyToK (7B)
60.7
58.3
45.4
54.8
100.1
(43.8)
Retain 50% tokens
VisionZipl
59.6
56.8
45.0
53.8
98.9
+ DyToK (7B)
56.8
43.8
53.6
98.5(-0.4) (.4)
FastV (BCOV2024)
55.7
53.2
36.8
48.6
89.3
+ DyToK(7B)
60.8
57.9
44.1
54.3
99.8 (+10.5)
Retain 25% tokens
VisionZipt
59.2
55.1
44.3
52.9
97.2
+ DyToK (7B)
59.9
55.3
44.5
53.2
97.8 (+0.6)
FastV (ECCV2024)
53.1
49.3
34.0
45.5
83.6
+ DyToK(7B)
58.8
54.5
43.3
52.2
96.0 (+12.4)
Retain 15% tokens
VisionZip
58.3
41.6
51.8
95.2
+ DyToK (7B)
58.4
54.1
42.5
51.7
95.0
(-0.2)
FastV (#C0V2024)
48.9
47.1
32.8
42.9
78.9
DyToK (7B)
56.9
51.1
39.8
49.3
90.6
(4-11.7)
Retain 10% tokens
VisionZip'
55.8
53.2
41.7
50.2
92.3
+ DyToK (7B)
57.1
53.1
43.6
51.3
94.3 (+2.0)
FastV (#C0V2024)
42.3
42.3
24.6
36.4
66.9
+ DyToK (7B)
42.3
42.3
24.6
36.4
66.9 (+00.0)
Table 7: Performance of DyToK integrated into VisionZip and FastV Qwen2.5-VL. The vanilla setup processes 32 frames. The final column reports the average score across benchmarks and the accuracy relative to the unpruned baseline. VisionZip denotes our pooling-compatibe variant.
(non-pruned) setting 75% retention ratio by 0.1%, demonstrating the strong effectiveness of our approach. In the extreme case of 10% retention ratio, our method combined with VisionZip also yields 2% improvement, further confirming its robustness under aggressive compression.
A.3 Experiments on Extended Video Length
To further verify DyToK's effectiveness on longer video sequences, we conduct supplementary experiments using uniform sampling of 64 frames with VaVionneiissoon as presented in Tab, S This experiment complements the findings in the main text by evaluating DyToK's scalability and robustness when dealing with increased temporal context.
The experimental results demonstrate DyToK's consistent superiority across various token retention ratios. At moderate compression levels (25%-50%). DyToK notably enhances performance, achieving up to 7.0% improvement in average accuracy compared to baseline methods at 50% token retention ratio. Remarkably, under more severe compression conditions (75%-90% token reduction), DyToK's advantage becomes even more pronounced. Specifically, at a retention ratio of just 10% of the original tokens (896 tokens), DyToK achieves substantial accuracy gains of up to 24.0% over baseline methods, highlighting its capability to preserve essential temporal information efficiently.
17

---

VideoMME
Long Video Bench
Average
Method
MLVU
Short Medium Long Overall
Score
%
Upper Bound, 12544 tokens
Vanilla (TXLB)
70.8
56.9
48.7
58.8
57.7
55.5
Retain 9408 tokens (L 25%)
VisionZip (CVPR 2025)
70.3
55.7
50.1
58.7
56.0
43.9
52.9
95.3
VisionZip'
71.1
56.8
48.9
58.9
58.0
49.6
55.5
100
DyTok
71.0
56.6
49.7
59.1
58.2
48.6
55.3
99.6
(+4.3)
+ DyTok (7B)
70.8
57.2
48.4
58.5
58.3
49.1
55.3
(+-4.3)
Retain 6272 tokens () 50%)
VisionZip (CVPR 2025)
69.0
57.1
49.1
58.4
55.2
42.3
52.0
93.7
VisionZip
72.4
56.0
48.4
59.0
57.7
48.7
55.1
99.3
DyToo
72.1
57.8
49.2
59.7
58.5
48.1
55.4
99.8
(+6.1)
+ DyTok (7B)
72.2
58.1
49.0
59.8
58.3
49.7
55.9
100.7
(47.0)
Retain 3136 tokens () 75%)
VisionZip (CVPR 2025)
65.6
53.4
48.2
55.7
51.5
42.3
49.8
89.7
VisionZip!
70.3
55.8
48.1
58.1
56.6
47.4
54.0
97.3
+ DyTok
71.6
57.4
49.1
59.4
57.8
46.5
54.6
98.4
(+8.7)
+ DyToo (7B)
72.7
57.7
48.1
59.5
58.4
47.4
55.1
99.3
(49.6)
Retain 2240 tokens (L 80%)
VisionZip (CVPR 2025)
61.6
52.9
46.1
53.5
48.5
37.1
46.4
83.6
VisionZip
70.1
56.4
48.0
58.2
55.5
46.8
53.5
96.4
+ DyTok
70.1
56.3
48.7
58.4
56.1
46.1
53.5
96.4
(+12.88
+ DyTok (7B)
71.6
57.1
47.8
58.8
59.2
47.7
55.2
99.5
(+15.9)
Retain 1792 tokens (L 85%)
VisionZip (CVPR 2025)
58.1
50.6
44.3
51.0
46.7
34.0
43.9
79.1
VisionZip
67.4
54.8
47.1
56.4
54.5
42.9
51.3
92.4
+ DyTok
69.0
55.8
48.1
57.6
55.3
45.0
$2.6
94.8
(+115.7)
+ DyTok (7B)
70.6
56.9
47.9
58.4
57.7
46.5
54.2
97.7
(+18.6)
Retain 896 tokens () 90%)
VisionZip (CVPR 2025)
47.8
45.3
41.9
45.0
42.1
29.0
38.7
69.7
VisionZip!
56.7
52.1
45.9
51.6
47.9
36.8
45.4
81.8
+ DyTok
64.0
52.9
47.2
54.7
53.6
42.6
50.3
90.6
(+-20.9)
+ DyTok (7B)
65.9
55.8
46.3
56.0
55.1
45.0
$2.0
93.7
(+-24.0)
Table Performance Dy'To on 64-frames L.LVVA-Oe OnAA-Onevssonn The vanilla setup processes 64 frames with 196 tokens each. The final column reports the average score across benchmarks and the accuracy relative to the unpruned baseline. VisionZip denotes our pooling-compatible variant. DyTok(7B) indicates that token pruning guided by 7B model.
18

---

Method
100%
25%
20%
15%
Vanilla
41.4
N/A
N/A
N/A
FastV
N/A
23.6
22.3
17.1
DyToK (3B guides 32B)
N/A
35.5
35,4
29.9
DyToK (7B guides 32B)
N/A
36.3
34.9
33.9
Table 9: Experiments broader model size. We conduct experiments utilizzig Owen2.5-VLL (3B, 7B, and 32B) on the MLVU benchmark. Specifically, models are evaluated using 32 input frames, with retention ratios ranging from 15% 25 55,, based on the FastV pruning method.
Overall, these findings confirm Dy DyToK's effectiveness significantly mitigating performance degra- dation under extreme compression scenarios, reinforeing its general applicability and robustness for longer video sequences.
A.4 Experiments on Broader Model Size
Beyond the 0.5B guiding 7B experiments using VALaVA-O already presented in the main text, we conduct additional extensive experiments using Qwen2.5-Vi 5-VL on the MLVU benchmark, which covers a wider range commonly used model sizes (3B, 7B, and 32B), as shown in Tab. 9 For the experiments in this section, adopt 4-bit NormalFloat (NF4) quantization configuration with double quantization and 6 compute, and enable Fltentott during inference. The use of these heterogeneous quantization regimes is pragmatic choice dictated by hardware- resource constraints, primarily regarding device memory capacity and memory bundwidtth/hhoogghuut characteristics,
evaluation platforms.
These experimental results provide two further insights. First, video models inherently possess keyframe priors regardless of model size. Second, such priors are not constrained by model scale, as even lightweight models can exhibit keyframe perception capabilities comparable to those of larger models. Notably, at 20% retention ratio, the 3B Qwen2.5-VL assistant model surpasses the 7B variant by substantial margin, further validating the feasibility of using lightweight assistant model to provide keyframe priors in DyToK.
A.5 Experiments General Descriptive Queries
To assess DyToK's ability to select keyframes for general descriptive queries (i.e., the query does not explicitly point specific visual content), conduct captioning experiments VideoChatGPT (29 using SLaVA-OneVision (7B), which includes prompts like "What is happening in the video?"
Following Dynamic-VLM [30
we use GPT-3.5-Turbo-0613 for quantitative scoring, with the results
summarized in Tab. 10
We attribute Dy DoK'ss advantage to its effectiveness in identifying both "semantic keyframes" (relevant to query semantics) and "content keyframes" (capturing key scene or event transitions), making well-suited for general descriptive queries. This distinguishes DyToK from CLIP-based methods, which rely solely on semantic matching. To support our claim, we replace Dy'To''s cross-modal attention-basee frame weight estimation with CLIP-based semantic similarity. Implementation details follow AKS 19 Results remain consistent, as shown in Tab.
A.6 Ablation on Keyframe Prior Layer Selection
In the main text (Tab, different layers of LLM. To comprehensively investigate the impact of various layer configurations, this section provides detailed results from full of ablation studies on layer selection. We explore multiple configurations, including individual layers as well as combinations of consecutive layers segmented into thirds and sixths of the model's depth, to verify the potential advantage aggregating multiple consecutive layers for enhancing keyframe priors. This extended analysis further supports the observation discussed in Sec. 2.2 specifically highlighting that deeper attention layers yield more reliable keyframe priors.
partially discuss the effectiveness of extracting keyframe priors from
19

---

Retention Ratio
Average
Method
TU
Score
%
Vanilla
100%
3.11
3.17
2.79
2.83
2.20
2.82
100.0
VisionZip
2.91
3.08
2.72
2.74
2.03
2.70
95.7
50%
DyToK
3.04
3.15
2.79
2.80
2.17
2.79
98.9
VisionZip
2.62
2.86
2.50
2,42
1.81
2.44
86.5
25%
DyToK
3.01
3.11
2.76
2.68
2.10
2.73
96.8
VisionZip 15%
2,20
2.48
2.14
2.08
1.61
2.10
74.5
DyToK
2.91
3.02
2.67
2.54
1.97
2.62
VisionZip 10%
1.85
1.51
1.24
1.53
54.3
DyToK
2.62
2.75
2.38
2.24
1.84
2.37
84.0
Table 10: Experiments general descriptive queries. We conduct experiments VideoChatGPP using LLaVA-One Vision (7B) with 32 input frames, based on the VisionZip pruning method at
retention ratios ranging from 10% 50%.
Retention Ratio
Method
Short Medium
Long Overall
Relative (%)
Vanilla
100%
69.9
56.7
48.9
58.5
100.0
DyToK (w/ CLIP)
68.3
56.9
49.1
58.1
99.4
25%
DyToK
69.2
56.8
48.9
58.3
99.7
DyToK (w/ CLIP)
64.6
53.9
48.4
55.6
95.1
15%
DyToK
66.3
55.0
48.7
56.7
97.0
DyToK (w/ CLIP)
57.6
50.9
45.8
51.4
87.9
10%
DyToK
61.0
51.1
47.6
53.2
91.0
Table 11: Comparison of Dy DyToK and CLIP-based keyframe selection. We conduct experiments on LLaVA-One Vision (7B) using 32 input frames, with retention ratios ranging from 10% to 25%, based on the VisionZip pruning method.
Tab. 12 reports performance across alternative layer-selection strategies. We find that aggregating attention from the deepest third layers (layers 16 to 23) achieves strong results, reaching 98. 08.0% of baseline performance under a 20% retention ratio. Although its accuracy is slightly lower than that of the west-performing single layer (layer 20), its fixed percentage-base configuration generalizes better across models compared manually tuning layer selection for each individual model. More broadly, accuracy improves with increasing layer depth: configurations leveraging later attention layers consistently exceed those restricted to earlier shallower selections, indicating that temporally salient information is concentrated toward the top the network. These findings substantiate the hypothesis that deeper attention layers effectively capture relevant temporal cues under pruning-aware settings.
A.7 Ablation on Token Allocation Upper Limit
As discussed in Appendix B.I and Sec. 3.3 to ensure token budgets allocated using frame weights remain within reasonable bounds for each frame and mitigate the adverse effects of temporal atten- tion outliers, introduce predefined upper limit for tokens per frame. If the allocated tokens exceed this upper limit, excess tokens are truncated and redistributed to other frames. This section comprehensively evaluates the influence of various upper limit settings on DyToK.
Tab. 13 presents detailed results across multiple upper limit configurations. Our findings reveal that setting moderately restrictive upper limit significantly enhances performance. Particularly, an upper
20

---

VideoMME
Average
Layer
Long VideoBench MLVU
Short
Medium
Long
Overall
Score
Vanilla (TNLR)
70.0
56.7
48.8
58.5
56.6
47.1
54.1
100
0
62.6
53.1
46,4
54.0
52.0
40.8
48.9
90.5
4
63.6
53.3
49.0
55.3
53.5
42.3
50.4
93.1
8
64.1
53.4
47.3
55.0
53,4
41.5
50.0
92.3
12
65.7
54.9
47.8
56.1
53.3
43.3
50.9
94.1
16
66.2
55.3
48.2
56.6
53.4
44.9
51.6 95.4
20
68.1
56.6
47.8
57.5
55.5
46.3
53.1 98.2
23
66.3
56.3
48.7
57.1
53.8
45.1
52.0 96.1
0.7
64.7
53.3
48.4
55.5
53.5
43.7
50.9 94.0
8.15
55.4
48.0
56.5
54.8
51.9 96.0
16.23
69.2
56.4
47.8
57.8
56.0
45.2
53.0 98.0
0.3
63.2
54.0
48.7
55.3
54.2
42.4
50.6 93.5
4.7
65.6
53.8
48.8
56.0
54.5
43.9
51.5 95.2
8.11
66.9
55.3
49.7
57.3
55.2
44.9
52.5 97.0
12.15
66.3
57.1
48.3
57.3
53.8
41.6
50.9 94.1
16.19
67.0
56.2
46.7
56.6
54.8
42.7
51.4
94.9
20.23
69.3
55.7
48.7
57.9
54.5
46.1
52.8 97.6
Table 12: Performance of DyToK under different layer configurations. We conduct experiments using retention ratio of 20%, with the number of frames set to 32. In comparison, the baseline LLaVA-OneVisia retains its original configuration without any pruning or modification. The best result is highlighted in red, while the second-best is shown in bold.
VideoMME
Egoschema
Upper Limit/Frame
Long VideoBench MLVU
Average
Short
Medium
Long
Overall
Subset
Total
Vanilla (THLR)
70.0
56.7
48.8
58.5
62.0
60.0
56.6
47.1
56.6
196
69.2
56.4
47.8
57.8
62.2
59.8
56.0
45.2
54.7
176
69.0
56.3
47.9
57.7
62.2
59.8
55.8
45.0
54.6
147
68.7
56.3
48.1
57.7
56.2
44.7
118
68.0
56.0
48.0
57.3
62.0
59.9
55.9
44.7
54.5
98
68.4
55.8
47.9
57.4
62.8
59.9
55.8
46.6
54.9
78
68.2
55.3
49.0
57.5
62.0
59.8
56.4
45.5
54,8
49
68.0
56.7
49.0
57.9
63.2
60.2
55.1
45.2
54.6
Table 13: Performance of DyToK under different upper limit configurations. We conduct experiments using retention ratio of 20%, with the number of frames 32. In comparison, the bascline LLaVA-One Vision retains its original configuration without any pruning or modification. The best result is highlighted in red, while the second-best is shown bold.
21

---

Retention Ratio
LongVideo Bench
Average
Method
VideoMMM
MLVU
Score
%
Vanilla
100%
58.5
56.6
47.1
54.1
100.0
Attention Entropy Based
58.3
55.9
45.6
53.3
98.5
Feature Entropy-Baad Based
58.3
54.8
42.9
52.0
96.1
25%
Feature Magnitude-Bann -Baseddeeeedee
58.3
56.0
44.8
53.1
98.1
DyToK
58.3
53.3
98.5
Attention Entropy-
52.8
50.5
93.2
Feature Entropy-Basee -Based
55.2
52.7
44.9
50.9
94.1
15%
Feature Magnitude-Band Based
54.7
53.3
43.8
50.6
93.5
DyToK
56.7
54.2
43.4
53.3
96.0
Attention Entropy- -assed
50.6
48.1
38.6
45.7
84.5
Feature Entropy-Based
50.7
48.8
38.4
45.9
84.9
10%
Feature Magnitude-Bast -ased
50.6
47.4
37.9
45.3
83.7
DyToK
53.2
50.4
42.8
48.8
90.2
Table 14: Ablation on importance estimation strategies. We conduct experiments on LLaVA- Vision (7B) using 32 input frames, with retention ratios ranging from 10% to 25%, based on the VisionZip pruning method.
limit of 98 tokens per frame achieves the highest average performance of 54.9, closely followed by the configuration with tokens per frame at 54.8. These settings indicate that tighter upper limits efficiently suppress temporal attention outliers without excessively restricting token allocation to keyframes, thereby maintaining robust model accuracy.
A.8 Ablation on Importance Estimation Strategies
To investigate the effects of various importance estimation strategies the performance, conduct ablation studies comparing our proposed cross-modal attention-based frame weighting method (utilized in the Temporal Importance Estimation stage) with entropy- -based and magnitude-based approaches. Specifically, we explore the following alternative importance estimation strategies.
Attention Entroopy-Based For each frame, compute the attention weights of the last query token over visual tokens. Calculate the entropy of these attention weights and normalize the results to obtain the frame weights.
Feature Entropy-Based Compute the information entropy of visual tokens along the feature dimension. For each frame, average the entropy of all tokens and normalize the values to derive the frame weights.
Feature Magnitude-Based. Calculate the L2 norm of visual tokens for each frame. Average the norms and normalize the results obtain the frame weights.
Tab. 14 shows that DyToK consistently outperforms entropy and magnitude-based methods across different retention ratios.
A.9 Ablation on Textual Token Selection
To explore whether relying solely on the last textual token can fully capture the sequence's 's semantics, conduct ablations multiple long video benchmarks using LLaVA-One Vision (7B) under various retention ratios.
at higher retention ratios (e.g., 75%, 50%). using only the last query token performs better. more aggressive ratios (e.g., 25% 15%), using query tokens yields further gains. We hypothesize that the last token acts precise but narrow selector, while all tokens offer
As shown in Tab. 15
22

---

Retention Ratio
LongVideo Bench
Average
Method
VideoMME
MLVU
Score
Vanilla
100%
58.5
56.6
47.1
54.1
100.0
FastV
57.6
57.1
46.5
53.7
99.3
DyToK (all query tokens)
75%
58.1
57.1
45.4
53.5
98.9
DyToK (last query token)
58.4
56.8
46.8
54.0
99.8
FastV
57.2
57.1
44.7
53.0
98.0
DyToK (all query tokens)
50%
58.2
58.0
45.6
53.9
99.6
DyToK (last query token)
58.5
57.2
46.3
54.0
99.8
FastV
56.0
56.6
43.7
52.1
96.3
DyToK (all query tokens)
25%
56.9
58.0
47.3
54.1
100.0
DyToK (last query token)
56.3
55.7
47.8
53.3
98.5
FastV
51.1
51.2
38.3
46.9
86.7
DyToK (all query tokens)
15%
54.1
53.1
44.4
50.5
93.3
DyToK (last query token)
54.8
52.6
43.2
50.2
92.8
Table 15: Ablation on textual token selection. We conduct experiments on LLaVA-OneVision (7B)
using 32 input frames, based on the FastV pruning method.
broader coverage with higher recall. When visual evidence is abundant, precision reduces noise; when it is scarce, broader semantic coverage becomes essential.
B Detailed Analysis of Keyframe Prior in VLLMs
In this section, we provide an in-depth description and qualitative analysis of the keyframe prior phenomenon observed in VLLMs. We believe our findings will aid future research in understanding and addressing related challenges.
B.1 Temporal Attention Outlier Phenomenon
Motivation and methodology. As described in Sec. 2.2 effeective token compression should dynamically allocate tokens according to each frame's 's relevance to the query, i.e., assigning more tokens informative frames and fewer to less relevant ones. Accurate indicators of frame contribution are thus essential. We investigate whether the VLLM attention map can serve as such an indicator. Specifically, compute attention weights from the last textual token to all visual tokens each layer the VL.MM, producing fine-grained layer-frame correlations. Averaging these attention weights across frames and layers yields temporal importance score, quantifying each frame's overall relevance. Similarly, averaging attention weights across the entire visual token sequence per layer indicates each layer's contribution video comprehension.
Experimental observations. We conduct experiments with 32-frame and 64-frame inputs, as illustrated in Fig. Fig.5) and Fig. Both LLaVA-One' Vision [ and LLaVA- Video [2] consistently exhibit accurate identification of keyframes relevant to specific queries. For example, when asked, "Which of the following outcomes occurs when the boy with brown hair and no glasses indicates the feather?", both models prominently attend to keyframe 23. However, also notice a consistent bias toward disproportionately high attention allocation at the initial and final frames, regardless their semantic irrelevance
absence of meaningful information.
This phenomenon, termed Temporal Attention Oullier, becomes significantly pronounced with 64- frame inputs, wherein an additional, highly prominent outlier frame appears consistently in the middle of the sequence. Remarkably, this outlier frame remains constant across different queries for the same video and generally lacks relevant semantic content.
23

---

Input Vidco
Layer-Frame Correlation
Key Frames
Key Layers
Question: activities do
stadents engage in within the room?
LLaVA-Ov: Practicing spell.
Question:
Which of the following when the boy with brown hair and glasses indicates the feather?
MaVAA-V: The foather floats
Question:
What activities do
Shudents engage in within the room?
LLaVA-Videu: Pructicing spell.
Question: of the following outcomes when the boy with brown hair and no glasses indicates the feather?
LLAVA-Vido: The feather floats
Figure Analysis VLLMMs with 32-frame inputs. We visualize the attention behavior LLaVA- 32-frame video input under different queries. Each row shows the model's predicted answer, the layer-frame correlation heatmap, the frame-wise attention distribution, and the layer-wise attention weights. Both models exhibit consistent and accurate localization of task-relevant keyframes. However, we also observe a recurring bias toward the initial and final frames, where attention is disproportionately high despite their limited relevance.
One Vision and LLaVA-Video
24

---

Input Video
naRA VACOIE
DOG
IBRA
Layer-Frame Correlation
Key Frames
Key Layers
What color the cell membrane of the viras-nvvded cell in the video?
Question:
L.L.VA-VV: Red.
Question:
Where mRNA come
the video?
LLaVA-OV: COVID-19 virus.
What color the cell membrane of the vires-ini invaded cell video?
Question:
LLaVA-Video: Red.
Question: Where from the video?
LLaVA-Video: COVID-19 virus.
Figure 6: Analysis of VLLMS with 64-frame inputs. This figure presents the behavior of LLaVA- One Vision and LLLVA--Vdde 64-frame inputs. Compared to the 32-frame case, the temporal attention bias becomes more prominent. In addition to edge-frame outliers, a strong and persistent attention peak emerges at middle-frame location across different queries. This central outlier does not consistently contain meaningful visual content yet dominates attention allocation, revealing new form of modality-invarin attention artifact we term as Temporal Attention Outlier.
25

---

Layer 0
Layer
Layer
Layer 3
Layer
Layer
Layer 6
Layer
Layer
Layer
Layer 10
Layer 11
Layer 12
Layer 13
Layer
Layer
Layer
Layer 17
Layer
Layer
Layer 20
Layer 21
Layer 22
Layer 23
Layer 24
Layer 25
Layer 26
Layer 27
Figure 7: Frame-level attention maps of LLa Vision across 28 layers. Attention visual- izations for the mid-sequence outlier frame reveal a pronounced and consistent activation pattern from layers to 20, all focusing on the same non-semantic background patch. The similarity across layers reveals copy-paste- pet--p-atl-kkeee like effect, indicating temporal attention outlier that persists across the model hierarchy.
Related work and interpretation. Similar attention outlier phenomena have been observed in other domains. Registers [31] identify certain tokens visual Transformers exhibiting extraordinarily high output norms despite lacking semantic content. Such tokens aggregate global information rather
than local patch representations. Analogously, CLIPtrase 322 and DeCLIP (33 report "proxy/global token" effects in vision language models, where specific tokens gain disproportionate attention, diluting local semantic relationships and negatively impacting downstream dense prediction tasks. StreamingLLM [3 introduces the "attention sink" concept, observing structural anomalies whereby non-semantic initial tokens dominate attention distributions due normalization constraints imposed by softmax operations.
Collectively, these studies highlight general drawback in transformer attention mechanisms: tain non- conionformatie tokens inevitably absorb disproportionate attention to maintain computational stability under softmax normalization. The temporal attention outlier phenomenon observed here temporal manifestation of this mechanism, distorting accurate attention allocation based on semantic relevance. This misallocation undermines the robustness and effectiveness of dynamic token compression strategies.
Mitigation strategies. Considering this detrimental effect, we suggest explicit strategies to mitigate attention outliers. As detailed Sec. 3.3 imposing an upper limit T Tmaz on tokens per frame during budget allocation can effectively suppress outlier frames' influence by reallocating surplus tokens to genuinely important frames. Experimental results summarized Tab. T3 demonstrate that lowering Tmaz enhances post-pruning model accuracy.
26

---

Layer 0
Layer
Layer
Layer 3
Layer
Layer 5
Layer 6
Layer
Layer
Layer
Layer 10
Layer 11
Layer 12
Layer 13
Layer
Layer
Layer
Layer 17
Layer
Layer
Layer 20
Layer 21
Layer 22
Layer 23
Layer 24
Layer 25
Layer 26
Layer 27
Figure 8: Frame-level attention maps of VL-VA-iddeo across 28 layers. Similar LLaVA- Vision, distinct attention concentration is observed from layers 12 to 20, fixated on a spatially narrow and semantically irrelevant region. The uniformity of these patterns across layers highlights the presence of systematic temporal attention outlier within VLLMs.
B.2 Visualizing and Characterizing Temporal Attention Outliers
To further characterize temporal attention outliers, we visualize attention distributions at the frame- token level across layers (Fig. 7 and Fig. 8). For 64-frame inputs, distinct bright horizontal line appears consistently at layers 12-20, corresponding precisely to the mid-sequence attention outlier frame. Detailed visualization reveals these layers exclusively attend to single background patch within the outlier frame, with strikingly similar attention patterns across layers, resembling "copy-paste" effect.
This comprehensive visualization reinforces the understanding that temporal attention outliers consistently manifest specific layers, target non-semantic regions, and represent systematic rather than random anomaly within the VLLM architecture.
B.3 Exploring Keyframe Bias in VLLMs
The discovery of temporal attention outliers raises a fundamental question: Does an inherent bias toward specific frames ("keyframe bias") exist within VLLMs? To investigate this hypothesis, present LLa VA-One Vision with semantically empty inputs, including black, white, and noise videos, matching the original video's frame count and resolution.
As illustrated in Fig. significant attention peaks consistently appear the initial, final, and mid- sequence frames, even without semantic content. This result strongly suggests an inherent structural bias within VLLMs toward these frames. Potential explanations include mechanisms similar to "attention sinks" 34 which manage excess attention distribution, global patches [3] 32 33 35
used aggregate information across sequences, facilitating processing under causal masking. Future research should further clarify these mechanisms.
27

---

Input Video
Layer-Frame Correlation
Key Frames
Key Layers
Figure 9: Visualization of the keyframe bias in VLLMs. Each corresponds variant of synthetic input (black/white/noise videos) with minimal semantic content. The middle column presents the layer-frame correlation heatmaps. where a distinct vertical stripe indicates mid- sequence attention outlier across layers 12 to 20. The right two plots show the corresponding frame-wise and layer-wise importance scores, with sharp peak at the outlier frame. These consistent patterns across diverse non-semantic inputs reveal the systematic nature of keyframe bias VLLMs.
We hope that our detailed exploration of keyframe bias and temporal attention outliers may provide useful insights for future research, contributing to more robust and interpretable VLLMs.
C Efficiency Analysis
In this section, we present additional experiments evaluating the efficiency improvements of DyToK, demonstrating significant reductions in GPU memory usage, FLOPs, and inference time with minimal impact model accuracy.
GPU memory usage. We first analyze the GPU memory efficiency of DyToK using LLaVA- Vision (7B) the VideoMME benchmark with 32 3r-frame inputs. Experiments conducted across various token retention ratios, ranging from 10% to 75%. As shown in Tab. T6 GPU memory consumption decreases substantially with lower retention ratios, achieving maximum memory reduction of 77.9%. Memory savings are significantly influenced by model size, stabilizing around 17.3 GB at retention ratios of 50% and below, which approximates the minimal GPU memory footprint limited by the model's weights.
Inference time analysis. evaluate DyToK's inference efficiency, we measure the prefilling time on the VideoMME benchmark using LLaVA-One Vision (7B) with 32 input frames. As indicated in
28

---

Retention Ratio
Vineo Bench
Method
FLOPs (T)
Memory (GB)
Latency (ms)
VideoMME
MLVU
Avg.
Vanilla
100%
40.8
22.2
48.3
58.5
56.6
47.1
54.1
DyToK
75%
32.6
20.2
44.1
59.0
55,9
46.6
53,8
DyToK
50%
21.9
17.3
23.5
59.1
56.4
46.2
53,9
DyToK
25%
12.2
17.3
11.3
58.3
55.4
46.3
53.3
DyToK
20%
9.6
17.3
10.7
58.4
55.4
43.7
52.5
DyToK
15%
8.4
17.3
10.4
56.7
54.2
43.4
51.4
DyToK
10%
5.9
17.3
10.8
53.2
50.4
42.8
48.8
Table 16: Analysis of DyToK on FLOPs, GPU memory, and prefilling time. We conduct experiments on the LLaVA-Onel Oik--nneisionn model using 32 input frames, with pruning rates ranging from 10% to 75%, based the VisionZip pruning method. For comparison, the baseline LLaVA- One retains its original configuration without any pruning or modification.
Tab. 16 ToK notably accelerates inference speed as the token retention ratio decreases. Specifically, at 50% retention ratio, DyToK achieves a 2.1x speedup. improving further 4.3x at a 25% retention ratio compared to the vanilla model.
-ccuracy-efffcienncy trade-off. DyToK provides balanced trade-off between accuracy and com- putational efficiency. At moderate 50% retention ratio, DyToK retains 99.6% of the original model accuracy, accelerates inference by 2.1x, and reduces GPU memory usage by 22.1 1.. At lower retention ratio of 25%, DyTTK maintains 98.5% accuracy alongside a 4.3 inference speedup. Even at the extreme retention ratio of 10%, equivalent to retaining approximately 14 tokens per frame, DyToK still preserves 90.2% accuracy, underscoring its robustness under aggressive compression.
D Related Works
Vision-Language Models. Inspired by the success of large language models (LLMs)
36
391, recent Vision Language Models (VLMs) combine visual encoder [66 pretrained LLM [37
17
40
41 with
36 to enable large-scale multimodal reasoning. Pioneering systems such as 2 and LLaVA 43 pass visual features through lightweight projector before concatenating with textual tokens. While effective, this design suffers from a significant token- lenen-length bottleneck: single high-resolution image (672x672 in LLa [4 yields over 2,000 patch tokens, overwhelming the textual prompt. Multi-image or video inputs 2 3 further increase the sequence length. Therefore, maintaining model performance under constrained visual token budget is critical for the continued advancement of VLMs.
BLIP-2
Efficient Languae Models. the development VLMMs, two main approaches have been proposed to accelerate inference, targeting the two core components of the system. One common strategy involves pruning on the encoder side [12 45 46
These methods typically identify dominant tokens and merge contextual tokens to preserve information from pruned tokens. Another prevalent approach focuses on pruning within the LLM itself. For example, FastV S selects important vision tokens based attention scores but neglects the contribution of low-attention tokens. Sparse [22] incorporates text guidance via cross cross-modal attention, yet faces similar limitations. DART 49 addresses this by considering not only high-priority tokens but also applying token reduction techniques to retain information from low-attention tokens. These methods collectively aim reduce the KV cache size and improve inference efficiency in LLMss.
47 48),
Efficient Video- Video-Language Models. Due to the large number of vision tokens in video inputs, recent video compression methods [0 13 50 have attracted growing attention. In early stages, most pruning techniques were adapted from image-based approaches, which inevitably overlooked the temporal dependencies inherent video data. For example, PruneVID IC clusters video tokens and selects those most relevant to the query tokens, yet it fails to fully capture temporal continuity. More

---

recent approaches have begun to address this limitation. FastVID [50], for instance, incorporates both temporal and visual context to perform spatiotemporal token pruning at the input stage, offering a more holistic approach to video token reduction.
Frame Selection Methods. Given the varying information density across video frames, most existing VLLMS [3
2 20 adopt fixed-duration sampling strategies. However, this uniform sampling often leads to sub-optimal performance, as it fails to account for the relevance of specific frames to the query. Several recent works (2 [33] aim to improve frame selection by clustering input frames to minimize redundancy, but these methods typically incur high computational overhead during the selection process. An alternative line of work [54] dynamically allocates tokens each frame based on their query relevance. Quo [54], for instance, employs a lightweight model to evaluate the relationship between each frame and the query in order to assign token budgets, but this introduces significant latency prior to inference. In contrast, our approach eliminates redundancy before inference by leveraging attention weight scores indicators for token allocation.
1
E
Efficient inference paradigms for VLLMs
Recent advancements in efficient inference for VLLMs can be categorized into two paradigms: LLM
attention-base based token pruning [T0 [3] and encoder feature-based token selection 23 [4]. These
paradigms differ fundamentally in their operational stages, criteria for token importance, and stability under varying conditions.
LLM Attention-Ban Based Token Pruning. As shown in Fig. aa), this paradigm dynamically prunes redundant visual tokens during the LLM inference stage. It operates by ranking token importance intermediate transformer layers within the LLM, typically using attention scores as criteria. Tokens with lower importance scores are progressively discarded in subsequent layers. Theoretically, the computational cost reduction can be modeled as:
K E/ C(ni) +E C(f))
Σt1 C(m)
where C(n) denotes the FLOPs of a transformer layer processing n tokens, is the pruning start layer, and hn S ny represents the retained token count after pruning.
While this approach aligns token retention with task-specific textual prompts, its efficacy hinges critically on the quality of attention maps from designated pruning layers. Shallow layers often produce noisy attention signals due to insufficient semantic integration, whereas deeper layers yield delayed pruning decisions that diminish computational savings. This layer-specific dependency introduces instability, particularly for long video sequences requiring multi-layer reasoning.
Encoder Teature-Based Token Selection. Differently, this paradigm reduces redundancy at the visual encoder's 's output stage by selecting tokens based on intrinsic feature importance before feeding them to the LLM, as depicted in Fig. b). Importance is quantified through static criteria derived from encoder self-attention patterns or feature activations, such as the average attention received by each token across all spatial or temporal positions. The token set is then truncated to subset of size k & n, achieving a theoretical FLOPSs reduction ratio of:
Cproj LLM (k)
Cproj LLM (1))
where Oproj LLM(-)) encompasses both projector and LLM computations. This paradigm avoids layer-specific dependencies by leveraging stable encoder-derived features, ensuring consistent token selection regardless of downstream LLM depth.
Our method can be seamlessly integrated into both of these VLLM inference acceleration paradigms in a training-fres frre and Alug-and-pla manner. By leveraging the inherent keyframe prior within weights to guide underlying token compression
VLLMs, Dy ToK provides temporal trame-importane strategies. This allows more tokens to be retained for keyframes that are critical to the current task while reducing token allocation for redundant frames. Under limited token budgets, DyToK helps focus the model's s attention on truly informative keyframes and enhances the temporal perception capabilities of VLLMs.
30

---

F Evaluation Benchmarks
We conducted experiments these widely used long video understanding benchmarks.
Video.MM.. VideoMME 6 comprises 2,700
multiple-choice questions drawn from 900 videos (~s 254h) across visual domains and 30 sub-fields; the evaluation therefore covers wide spectrum of spatial scenes and temporal durations, probing models' ability to track multimodal cues (frames audio + subtitles) from I--second clips up to 1-hour stories.
comprises 6,678 human-curated multiple-choice questions built on 3,763 web videos (up to Ih) and their subtitles, organized into 17 fine-grained categories that revolve around a novel "referring-reasoning" task, thereby requiring models to localize, retrieve, and integrate visual-linguistic evidence over very long pratial-teppon contexts.
Long BideoBench. LongVideoBench
EgoSchema. EgoSchema [55] comprises around 5,000 five-option multiple-chot choice questions col- lected from 250h of egocentric footage, emphasizing very long-form temporal reasoning- each three-minute clip demands tracing objects and actions over intrinsic time spans that 5-10x longer than prior datasets, thereby stressing both spatial perception and extended temporal coherence.
comprises 3,102 multiple-choice questions that span 9 distinct long-video understanding tasks, challenging models to handle videos ranging from 3 minutes to 2 hours and to reason over plot, temporal order, event retrieval, and other facets that jointly test fine-grained spatial recognition and long-rut oag-range temporal logic.
MLVU. MLVU (27
G Implementation Details
G.1 Experimental Settings
The vanilla LLaVA-OneVision employs S0400M [17
its vision encoder and Qwen2 [37 as its language model. The bilinear pooling strategy reduces the number of vision tokens per frame from 729 196. We use two settings for the total number of frames -32 and 64 consistent with prior work [56 57 This results in 6,272 tokens for 32 frames and 12,544 tokens for 64 frames.
To validate the generalization ability of DyToK different VLLMM inference acceleration paradigms, we integrate DyToK into the existing SOTA methods, FastV g. VisionZip [12], and DyCoke [3] which perform visual token pruning inside LLM, between the visual encoder and LLM,, and both sides, respectively. Specifically, FastV performs token pruning based on the text-to-visual attention extracted from a specific layer in LLM, VisionZip utilizes inter-patch feature correlations to apply token pruning after the visual encoder and before LLM, while DyCoke prunes visual tokens before and after entering into LLM, where the Token Temporal Merging (TTM) strategy compresses the redundancy in the temporal dimension by token merging on the encoder side, and the Dynamic Pruning (DP) strategy prunes visual tokens the decoding stage based on the last query-to-vist attention on the LLM side.
G.2 Token Budget Alignment
To ensure a fair comparison, we limit all methods to the same token budget. Based on this budget, calculate the actual compression ratios specific to each one FastV, DyCoKe, and VisionZip, so that the FLOPs of the computation of visual tokens during inference remain strictly consistent across methods.
To this end, we use a simple yet effective strategy to align the computational cost of different pruning strategies by keeping the total number of visual tokens processed by L.LM across all layers the same. Assuming an LIM with I. layers, token pruning occurs at layer K. Let N represent the initial number of visual tokens per layer before pruning, and M denote the number of tokens retained after pruning at layer K. To quantify the average computational cost per layer, we introduce Rt satisfying the following equation:
K KxN+ K) x M = LX R.
(8)
Thus, two methods can be considered computationally equivalent if their respective R values match.
31

---

Dominant
Contextual
Retention Ratio
Numerical Sum
Approximate Sum
168
28
100%
196
196
126
21
75%
147
147
84
14
50%
98
98
42
7
25%
49
49
30
5
20%
39.2
35
4
15%
29.4
2
10%
19.6
Table 17: Token budget allocation under different retention ratios in sionZip. A fixed 6:1 ratio maintained between dominant and contextual tokens. The total number of retained tokens is rounded down to the nearest multiple of 7 to meet this ratio, resulting in lower actual token counts, especially under extreme compression.
Vi-
Adaptation of VisionZip. When integrating DyToK into VisionZip, standardize hyperparameter settings across experiments, we maintain a fixed Given that LLaVA-OneVision originally processes 196 tokens per frame, exact division isn feasible, necessitating approximate rounding. To deliberately introduce a challenging scenario highlighting DyToK's robustness, we uniformly round down token counts to multiples of seven (see Tab. G.2). Token budgets used in subsequent methods are also aligned with the values presented in this table. The rounding strategy significantly reduces the effective token budget, particularly at extreme compression levels. For instance, at 10% retention ratio, this rounding strategy yields approximately 40% fewer tokens compared to standard calculations without rounding. Consequently, accuracy scores under identical nominal retention ratios in this paper may appear slightly lower than those reported in comparable literature. However, this deliberate choice does not obscure the clear superiority of DyToK's performance.
token ratio of 6:1 per frame.
Adaptation of DyCoke. We hypothesize that token pruning in LLM only occurs the prefilling stage for computational overhead alignment. However, DyCoke doesn't perform token pruning in LLM during the prefilling phase in order to obtain the KV Cache of all visual tokens across all layers for dynamic pruning in the decoding stage. The prefilling stage is crucial for some benchmarks like VideoMME which LLM generates short responses (e.g. one token). Thus, we transform the dynamic pruning strategy in the second stage of DyCoke into static and only perform pruning in the prefilling stage for fair comparison.
Adaptation of token compression methods to Qwen2.5-VL. Qwen2,5-VL adopts a sliding- window computation strategy that supports video frames of arbitrary resolution. At the end of its encoder, it introduces a patch merger module that reduces computational overhead by merging every four spatially adjacent visual tokens into one. However, this architecture poses significant challenges for the adaptation of token compression methods designed for other VLLMs.
Unlike the common 2D pooling techniques (e.g.. bilinear or average pooling) used in models like Onee-Oneiiiion and LLLVA- Video (2) Qwen2.5-VL. 3 performs token merging by reshaping the token layout and feeding it into an MIP to fuse features in the embedding space. To accommodate this mechanism in encoder fcature-based methods such as VisionZip. we retain the MLP-based merging strategy for the hidden states. However, to align the attention weights and importance scores, originally computed before merging with the post-merging hidden states, we apply spatially localized averaging to these metrics using the same compression ratio.
In addition, Qwen2.5-V VL employs Multimodal Rotary Position Embedding (MRoPE), a positional encoding scheme that encodes each visual token based on its precise temporal, height, and width indices. Compression methods like VisionZip. which discard original token positions during pruning, are therefore inherently incompatible with mRoPE. To address this issue, we degrade the multimodal mRoPF a standard ID ROPE, consistent with the approach used for L. VA -nOOee Vision, ensuring compatibility without altering the model architecture.
32

---

H
Visualization of Per-Frame Token Compression
To intuitively illustrate the effectiveness of dynamic token compression via LLM-guided keyframe prior, we visualize per-frame compression results on video-question pairs randomly sampled from the widely used VideoMME [6] benchmark. Each input video uniformly sampled to 8 frames, with the ground-truth keyframes marked by red pentagrams and all other frames denoted by yellow circles temporal order.
Fig. Fig. and Fig. 13 present results for an encoder feature-based method, Vi- sionZip under extreme token retention ratio of 10%. When the token budget uniformly allocated across frames. VisionZip fails to sufficiently preserve information from critical keyframes while retaining excessive tokens from less relevant frames. This leads to substantial semantic distraction and ultimately incorrect answers. In contrast, our LLM-guided keyframe priors assign token budgets based on frame-level importance. As result, DyToK effectively prioritizes tokens for keyframes that are most relevant to the question, while ag- gressively compressing redundant frames, leading enhanced temporal perception under constrained computational budgets.
Fig. 10
version leverages
Fig. 14 Fig. 15 Fig. 16 and Fig. 17 show similar comparisons for LLM attention-based method, Fast V 8] evaluated under a 15% retention ratio. Uniform token allocation again results a loss of key semantic content and an accumulation redundant context. DyToK consistently mitigates these issues by dynamically adjusting token budgets across frames, demonstrating its general applicability across different VLLM inference acceleration paradigms.
To accurately illustrate the actual attention distribution guided by the keyframe prior in the token budget allocation process, applied smoothing techniques certain temporal attention outliers, details of which can be found in Appendix lightweight, and pluggable module capable of dynamically assigning compression ratios based on frame-query relevance. Our visualizations clearly demonstrate its effectiveness in dynamically allocating token budgets. However, the precise distribution and exact positions of retained tokens within individual frames depend on the underlying pruning method, such as VisionZip or Fast
[B.]] [BI Importantly, DyToK designed as a training-free free,
33

---

video, facial expression look like when sees the scorpion skewer?
Input Video
Keyframe Prior
VisionZip X
A: Joy.
w/ DyToK
Diogust.
Figure 10: Example comparing VisionZip and VisionZip w/ DyToK under 10% retention ratio. The DyToK -enhaneddd method better preserves keyframe tokens and suppresses temporal redundancy under the limited budget.
Q: When zodiacal light visible from the video?
Input Video
Keyframe Prior
X
VisionZip
A. March 24.
DyTaK V
A. On March 19,
Figure 11: Example comparing VisionZip and VisionZip w/ DyToK under 10% retention ratio.
34

---

What kind clothes speaker wearing?
Iaput Video
Keyframe Prior
VisionZip X
WISSI DO SONC JOWN
Blue jacket.
3
DyToK V
As Green plaid shirt.
Figure 12: Example comparing VisionZip and VisionZip w/ DyToK under 10% retention ratio.
Q: What the specific sentence in smart phone that man embarrassed?
Input
Key Keyframe Prior
VisionZip
X
A: Smake Sig 2.0.
w/ DyToK V
BTW.joo something in your teetk!
Figure 13: Example comparing VisionZip and VisionZip w/ DyToK under 10% retention ratio.
35

---

What does the panda to fight with enemies in the video?
Input Video
Keyframe Prior
Fasty
X
bamboo.
FastV
Ar stick.
Figure 14: Example comparing FastV and FastV w/ DyToK under 15% retention ratio.
The variant achieves better focus on keyframe regions and reduces attention to temporally redundant content under the constrained token budget.
Q: many does interviewed girl appear the video?
Input Video
Keyframe Prior
FastV
x
A:
w/ D/DyTK V
Figure 15: Example 2 comparing FastV and FastV w/ DyToK under 15% retention ratio.
36

---

Q: Which object does the creature hold in its hand?
Input
Keyframe Prior
FastV
X
Sunglasses.
DyToK
A: A pistol.
Figure 16: Example comparing FastV and FastV w/ DyToK under 15% retention ratio.
STALL
Q: Which country do the athletes shown at beginning of the come from?
Input Video
Keyframe Prior
STRINN FEOR
FastV
X
A: Brazil.
DyToK
Germany.
Figure 17: Example 4 comparing FastV and FastV w/ DyToK under 15% retention ratio.
37

---

NeurIPS Paper Checklist
The checklist is designed to encourage best practices for responsible machine learning research, addressing issues of reproducibility, transparency, research ethics, and societal impact. Do not remove the checklist: The papers not including the checklist will be desk rejected. The checklist should follow the references and follow the (optional) supplemental material. The checklist does NOT count towards the page limit.
Please read the checklist guidelines carefully for information on how to answer these questions. For each question in the checklist:
You should answer [Yes], [No], or [NA].
[NA] means either that the question is Not Applicable for that particular paper or the relevant information is Not Available.
Please provide short (1-2 sentence) justificcton right after your answer (even for NA).
The checklist answers are an integral part of your paper submission. They are visible to the reviewers, area chairs, senior area chairs, and ethics reviewers. You will be asked to also include (after eventual revisions) with the final version of your paper, and its final version will be published with the paper.
The reviewers of your paper will be asked to use the checklist as one of the factors in their evaluation.
While "[Yes] " is generally preferable to "[No]", ", it is perfectly acceptable to answer "[No] provided proper justification is given (e.g., "error bars are not reported because it would be too computationally expensive" or "we were unable to find the license for the dataset we used"). In general, answering "[No] or "[NA] is not grounds for rejection. While the questions are phrased in binary way, we acknowledge that the true answer often more nuanced, so please just use your best judgment and write justification to elaborate. All supporting evidence can appear either in the main paper or the supplemental material, provided in appendix. If you answer [Yes] to a question, in the justification please point to the section(s) where related material for the question can be found.
IMPORTANT, please:
Delete this instruction block, but keep the section heading "NeurIPS Paper Checklist",
Keep the checklist subsection headings, questions/answers, and guidelines below.
Do not modify the questions and only use the provided macros for your answers.
1. Claims
Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?
Question:
Answer: [Yes]
Justification: Claims in the abstract and introduction align with our key motivation in Sec. 2.2 and our experiment result in Sec.
Guidelines:
The answer NA means that the abstract and introduction do not include the claims made the paper.
The abstract and/or introduction should clearly state the claims made, including the contributions made the paper and important assumptions and limitations. A No or
NA answer to this question will not be perceived well by the reviewers.
The claims made should match theoretical and experimental results, and reflect how much the results can be expected to generalize to other settings.
It is fine to include aspirational goals as motivation as long as it is clear that these goals are not attained by the paper.
2. Limitations
Question: Does the paper discuss the limitations of the work performed by the authors? Answer: [Yes]
38

---

Justification:
We have clarified our limitations in this paper.
The answer NA means that the paper has no limitation while the answer No means that the paper has limitations, but those are not discussed in the paper.
The authors are encouraged to create separate "Limitations" section in their paper.
The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (e.g., independence assumptions, noiseless settings, model well-specification asymptotic approximations only holding locally). The authors should reflect on how these assumptions might be violated in practice and what the implications would be.
The authors should reflect on the scope of the claims made, e.g., if the approach was only tested few datasets with few runs. In general, empirical results often depend on implicit assumptions, which should be articulated.
The authors should reflect on the factors that influence the performance of the approach. For example, facial recognition algorithm may perform poorly when image resolution is low or images are taken in low lighting. Or speech-to-text system might not be used reliably to provide closed captions for online lectures because it fails to handle technical jargon.
The authors should discuss the computational efficiency of the proposed algorithms and how they scale with dataset size.
If applicable, the authors should discuss possible limitations of their approach to address problems of privacy and fairness.
While the authors might fear that complete honesty about limitations might be used by reviewers as grounds for rejection, worse outcome might be that reviewers discover limitations that aren't 't acknowledged in the paper. The authors should use their best judgment and recognize that individual actions in favor of transparency play an impor- tant role in developing norms that preserve the integrity of the community. Reviewers will be specifically instructed not penalize honesty concerning limitations.
3. Theory assumptions and proofs
Question: For each theoretical result, does the paper provide the full set of assumptions and complete (and correct) proof?
Answer: [Yes]
Justification: More details of the theoretical analysis and evaluation settings can be found in the appendix.
Guidelines:
The answer NA means that the paper does not include theoretical results.
All the theorems, formulas, and proofs in the paper should be numbered and cross- referenced.
All assumptions should be clearly stated or referenced in the statement of any theorems.
The proofs can either appear in the main paper or the supplemental material, but if they appear in the supplemental material, the authors are encouraged to provide a short proof sketch to provide intuition.
Inversely, any informal proof provided in the core of the paper should be complemented by formal proofs provided in appendix or supplemental material.
Theorems and Lemmas that the proof relies upon should be properly referenced.
4. Experimental result reproducibility
Question: Does the paper fully disclose all the information needed to reproduce the main ex- perimental results of the paper to the extent that it affects the main claims and/or conclusions the paper (regardless whether the code and data are provided or not)?
Answer: [Yes]
Justification: We comprehensively clarify our method DyToK and present the detailed algorithm pseudo code. Our implementation details and specific parameter settings can be found in the appendix.
Guidelines:
39

---

The answer NA means that the paper does not include experiments.
If the paper includes experiments, a No answer to this question will not be perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data are provided or not.
If the contribution is dataset and/or model, the authors should describe the steps taken to make their results reproducible or verifiable.
Depending on the contribution, reproducibility can be accomplished in various ways. For example, if the contribution is a novel architecture, describing the architecture fully might suffice, or if the contribution is specific model and empirical evaluation, it may be necessary to either make it possible for others to replicate the model with the same dataset, provide access to the model. In general. releasing code and data is often one good way to accomplish this, but reproducibility can also be provided via detailed instructions for how to replicate the results, access to a hosted model (e.g., in the case of a large language model), releasing of a model checkpoint, or other means that are appropriate to the research performed.
While NeurIPS does not require releasing code, the conference does require all submis- sions to provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution. For example
(a) the contribution is primarily a new algorithm, the paper should make clear how reproduce that algorithm. (b) If the contribution is primarily new model architecture, the paper should describe the architecture clearly and fully.
(c) If the contribution is a new model (e.g., a large language model), then there should either be way to access this model for reproducing the results or way to reproduce the model (e.g., with an open- source dataset instructions for how to construct the dataset).
(d) We recognize that reproducibility may be tricky in some cases, in which case authors are welcome deseribe the particular way they provide for reproducibilityy In the case of closed-source models, it may be that access to the model is limited in some way (e.g., registered users), but it should be possible for other researchers to have some path to reproducing or verifying the results.
5. Open access to data and code
Question: Does the paper provide open access to the data and code, with sufficient instruc- tions to faithfully reproduce the main experimental results, as described in supplemental material?
Answer: [Yes]
Justification:
Our code and models will be made publicly available after publication.
Guidelines:
The answer NA means that paper does not include experiments requiring code.
Please see the NeurlPS code and data submission guidelines (https://nips. cc/
public/guides//odeSumiii http://odeSuumiisssooooooooooo for more details.
While we encourage the release of code and data, we understand that this might not be possible, "No" is an acceptable answer. Papers cannot be rejected simply for not including code, unless this is central to the contribution (e.g., for a new open-source benchmark).
The instructions should contain the exact command and environment needed to run to reproduce the results. See the NeurlPS code and data submission guidelines (https CodeeSumiiii wwwiiisssooosiiiiio
for more details.
The authors should provide instructions on data access and preparation, including how to access the raw data, preprocessed data, intermediate data, and generated data, etc.
The authors should provide scripts to reproduce all experimental results for the new proposed method and baselines. If only subset of experiments are reproducible, they should state which ones are omitted from the script and why.
At submission time, to preserve anonymity, the authors should release anonymized versions (if applicable).
40

---

Providing as much information as possible in supplemental material (appended to the paper) recommended, but including URLs to data and code permitted.
6. Experimental setting//etails
Question: Does the paper specify all the training and test details (e.g., data splits, hyper- parameters, how they were chosen, type of optimizer, etc.) necessary to understand the results?
Answer: [Yes]
Justification: First, DyToK is new training-free MLLM acceleration paradigm. Second, in our experiment, we clearly present our experiment setting and the token budget alignment strategy for fair comparison.
Guidelines:
The answer NA means that the paper does not include experiments.
The experimental setting should be presented in the core of the paper to level of detail that is necessary to appreciate the results and make sense of them.
The full details can be provided either with the code, in appendix, or as supplemental material.
7. Experiment statistical significance
Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?
Question:
Answer: [Yes]
Justification: More experimental details and results will be present in the appendix and supplementary materials.
Guidelines:
The answer NA means that the paper does not include experiments.
The authors should answer "Yes" if the results are accompanied by error bars, confi- dence intervals, or statistical significance tests, least for the experiments that support the main claims of the paper.
The factors of variability that the error bars capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, or overall
with given experimental conditions).
The method for calculating the error bars should be explained (closed form formula, call to a library function, bootstrap, etc.)
The assumptions made should be given (e.g., Normally distributed errors).
It should be clear whether the error bar is the standard deviation the standard error of the mean.
It is OK to report I-sigma error bars, but one should state it. The authors should preferably report a 2-sigma error bar than state that they have 96% CI, if the hypothesis of Normality of errors is not verified.
For asymmetric distributions, the authors should be careful figures symmetric error bars that would yield results that are out of range (e.g. negative error rates).
show in tables or
If error bars are reported in tables or plots, The authors should explain the text how they were calculated and reference the corresponding figures tables in the text.
8. Experiments compute resources
For each experiment, does the paper provide sufficient information on the com- puter resources (type of compute workers, memory, time of execution) needed reproduce the experiments?
Question:
Answer: [Yes]
Justification: The detailed experimental setting and inference platform can also be found in the appendix supplementary materials.
Guidelines:
41

---

The answer NA means that the paper does not include experiments.
The paper should indicate the type of compute workers CPU or GPU, internal cluster, or cloud provider, including relevant memory and storage.
The paper should provide the amount compute required for each of the individual experimental runs well estimate the total compute.
The paper should disclose whether the full research project required more compute than the experiments reported in the paper (e.g., preliminary or failed experiments that didn't make into the paper).
9. Code of ethics
Question: Does the research conducted in the paper conform, in every respect, with the NeurIPS Code of Ethics http:///w//aaaoo.c.coo////////aa
Answer: [Yes]
Justification: This work complies with NeurlPS Ethics Guidelines (no human subjects involved).
Guidelines:
The answer NA means that the authors have not reviewed the NeurIPS Code of Ethics.
If the authors answer No, they should explain the special circumstances that require deviation from the Code of Ethics.
The authors should make sure to preserve anonymity (e-g., there a special consid- eration due to laws or regulations in their jurisdiction).
10. Broader impacts
Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?
Answer: [No]
Justification: This work aims to accelcrate the inference efficiency of VLLMs while main- taining their performance. No social impacts involved.
Guidelines:
The answer NA means that there no socictal impact of the work performed.
If the authors answer NA or No, they should explain why their work has no societal impact why the paper does not address societal impact.
Examples of negative societal impacts include potential malicious unintended uses (e.g., disinformation, generating fake profiles, surveillance), fairness considerations (e.g.. deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations.
The conference expects that many papers will be foundational research and not tied to particular applications, let alone deployments. However, if there is direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that an improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point out that a generic algorithm for optimizing neural networks could enable people to train models that generate Deepfakes faster.
The authors should consider possible harms that could arise when the technology is being used as intended and functioning correctly, harms that could arise when the technology being used intended but gives incorrect results, and harms following from (intentional or unintentional) misuse of the technology.
If there are negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition attacks, mechanisms for monitoring misuse, mechanisms to monitor how system learns from feedback over time, improving the efficiency and accessibility of MI.)
11. Safeguards
Question: Does the paper describe safeguards that have been put in place for responsible release of data models that have a high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)?
42

---

Answer:
[No]
Justification:
We don't 't present high-risk models.
Guidelines:
The answer NA means that the paper poses no such risks.
Released models that have high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere usage guidelines or restrictions to access the model or implementing safety filters.
Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images.
We recognize that providing effective safeguards is challenging, and many papers do require this, but we encourage authors to take this into account and make best faith effort.
12. Licenses for existing assets
Question: Are the creators or original owners of assets (c.g., code, data, models), used in the paper, properly credited and are the license and terms of use explicitly mentioned and properly respected?
Answer: [No]
Justification: Our work only presents a novel training-free free VLLM acceleration paradigm.
Guidelines:
The answer NA means that the paper does not use existing assets.
The authors should cite the original paper that produced the code package dataset.
The authors should state which version the asset used and, possible, include URL.
The name of the license (e.g., CC-BY 4.0) should be included for each asset. For scraped data from a particular source (e.g., website), the copyright and terms of service of that source should be provided.
If assets are released, the license, copyright information, and terms of use in the package should be provided. For popular datasets, paperswi thcode. com datasets has curated licenses for some datasets. Their licensing guide can help determine the license of dataset.
For existing datasets that re-packaged, both the original license and the license of the derived asset (if has changed) should be provided.
If this information is not available online, the authors are encouraged to reach out to the asset's s creators.
New assets
Question: Are new assets introduced in the paper well documented and is the documentation provided alongside the assets?
Answer: [No]
Justification:
We don't 't present new dataset new models.
Guidelines:
The answer NA means that the paper does not release new assets.
Researchers should communicate the details the dataset/code/moodll as part their submissions via structured templates. This includes details about training, license, limitations, etc.
The paper should discuss whether and how consent was obtained from people whose asset used.
At submission time, remember to anonymize your assets (if applicable). You can either create an anonymized URL or include an anonymized zip file.
14. Crowdsourcing and research with human subjects
43

---

Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given to participants and screenshots, if applicable, as well as details about compensation (if any)?
Answer: [No]
Justification: This work is purely algorihmmic/thorrticaal and does not involve any human subjects crowdsourcing.
Guidelines:
The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.
Including this information the supplemental material is fine, but if the main contribu- tion of the paper involves human subjects, then as much detail as possible should be included the main paper.
According to the NeurIPS Code of Ethics, workers involved in data collection, curation, other labor should be paid at least the minimum wage in the country of the data collector.
15. Institutional review board (IRB) approvals or equivalent for research with human subjects
Question: Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or an equivalent approval/review based on the requirements of your country or institution) were obtained?
Answer: [Yes]
Justification: the paper does not involve crowdsourcing nor research with human subjects.
Guidelines:
The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.
Depending on the country in which research is conducted, IRB approval (or equivalent) may be required for any human subjects research. If you obtained IRB approval, you should clearly state this in the paper.
We recognize that the procedures for this may vary significantly between institutions and locations, and expect authors to adhere to the NeurIPS Code of Ethics and the guidelines for their institution.
For initial submissions, do not include any information that would break anonymity (if applicable), such as the institution conducting the review.
16. Declaration of LLM usage
Question: Does the paper describe the usage of LLMs if it is an important, original, or non-standard component of the core methods this research? Note that if the LLM used only for writing, editing, or formatting purposes and does not impact the core methodology. scientific rigorousness, or originality the research, declaration is not required.
Answer: [Yes]
Justification: DyToK uses assistant model derived from the same model family but with extra small parameters to generate frame weights, guiding us to perform dynamic pruning.
Guidelines:
The answer NA means that the core method development in this research does involve LLMs as any important, original, or non-standard components.
Please refer to our LLM policy http::::hearrppsssc/oeoeeeeeeeee for what should or should be described.
44


Reviews:
[{"review_id": "N8NnFnyO64", "reviewer": "Reviewer_HiJ7", "summary": "This paper proposes a dynamic token compression scheme that leverages the inherent attention mechanisms of VLLM. Experimental results show that the attention module effectively learns keyframe priors from the training data. By adopting this scheme, the algorithm dynamically adjusts the number of tokens on a frame-by-frame basis, adaptively removing redundant information. Extensive experiments demonstrate that DyToK achieves state-of-the-art trade-offs between efficiency and accuracy.", "questions": "- **Line 65: Placeholder Reference**  \n  The text refers to “XX datasets,” which appears to be a placeholder. This should be replaced with the actual dataset names to clearly indicate the evaluation scope and improve the paper’s professionalism.\n\n- **Definition of Keyframe and Adaptive Rate**  \n  The paper does not clearly define what qualifies as a “keyframe” within the proposed method. Additionally, the mechanism for determining the adaptive token rate for keyframes versus normal frames is not sufficiently explained. A detailed description or visualization of this process would enhance understanding.\n\n- **Runtime Efficiency**  \n  Although the paper emphasizes token redundancy reduction, it lacks reporting of runtime efficiency metrics. Including inference latency or throughput on real hardware would significantly strengthen the case for practical applicability, particularly in real-time or latency-sensitive environments.", "limitations": "Yes.", "rating": 3, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 2, "significance": 3, "originality": 2, "strengths_and_weaknesses": "### Strengths\n\n- Extensive evaluations on long-video benchmarks convincingly demonstrate that DyToK outperforms existing state-of-the-art methods in terms of efficiency-accuracy tradeoffs.\n\n- The paper provides a compelling finding that attention layers in vision-language large models (VLLMs) inherently encode query-conditioned keyframe priors. This insight enables effective identification of task-relevant information without requiring additional supervision.\n\n- DyToK shows excellent generalization and compatibility across different backbone models and token compression strategies. This makes it a versatile component that can be easily integrated into existing frameworks.\n\n- The authors conduct extensive experiments and ablation studies across various datasets and model settings, providing strong evidence for the effectiveness and robustness of the proposed approach.\n\n### Weaknesses\n- Lack of Comparison with Recent State-of-the-Art Methods: The paper omits comparison with several recently proposed token compression or pruning techniques, limiting the evaluation’s completeness and making it hard to assess the relative progress.\n\n- No Evaluation Against Structured Compression Baselines: Methods such as LoRA, quantization-aware training, or parameter-efficient fine-tuning techniques are not included in the comparison, which raises concerns about the fairness and rigor of the benchmarks.\n\n- Missing End-to-End Latency on Real Hardware: While theoretical FLOPs are reported, practical runtime metrics (e.g., latency on GPU inference) are not provided, making it unclear how beneficial the proposed approach is in deployment scenarios.\n\n- Weak Justification for the Attention-Based Importance Estimation: The proposed method for selecting important tokens is intuitive but lacks theoretical grounding or sufficient ablation studies to validate why it outperforms simpler baselines like entropy- or magnitude-based filtering.", "was_revised": true, "final_justification": "Thank you to the authors for the detailed rebuttal. While the explanation of the attention-based importance estimation is helpful, the overall novelty of the proposed method remains limited. The lack of comparison with more recent state-of-the-art approaches—relying only on VisionZip and DyCoke as baselines—further restricts the demonstrated impact and potential for broader applicability. Given that some concerns remain unaddressed, I will maintain my rating as Borderline reject.", "rebuttal": "We sincerely appreciate your comments. All suggestions will be incorporated into the paper, and typos will be fixed. Our responses are as follows.\n\n&nbsp;\n\n> **Response to Weakness 1 (Comparison with SOTA Methods)**\n> \n\nAs described in Figure 2 (page 4) of the manuscript, existing token compression methods can generally be categorized into two paradigms: encoder feature-based and LLM attention-based. Accordingly, we have conducted comparisons with VisionZip[1] (state-of-the-art encoder feature-based method) and DyCoke[2] (state-of-the-art LLM attention-based method), both recently published at CVPR 2025. \n\nUnder identical token budgets and across a comprehensive range of retention ratios (10%, 15%, 20%, 25%, 50%, 75%), DyToK has consistently achieved superior performance on diverse benchmarks for long video understanding (VideoMME, LongVideoBench, MLVU). Detailed comparative radar plots and complete experimental results are provided in Figure 1 (Appendix, page 2) and Tables 1 and 2 (Appendix, pages 18–19). If there are specific works that you believe we should additionally compare with, please kindly let us know, and we will do our best to include the comparisons in the discussion phase.\n\n[1] VisionZip: Longer is Better but Not Necessary in Vision Language Models (CVPR 2025)\n\n[2] DyCoke: Dynamic Compression of Tokens for Fast Video Large Language Models (CVPR 2025)\n\n&nbsp;\n\n> **Response to Weakness 2 (Evaluation Against Structured Compression Baselines)**\n> \n1. **Training-Free vs. Training-Based:** DyToK is designed as a training-free dynamic token compression method; therefore, our evaluation primarily targets other training-free methods. As the structured compression approaches you have mentioned require training, the comparison with them is beyond the scope of our current comparative framework.\n2. **Inference Acceleration vs. Training Acceleration:** DyToK is a training-free method that aims at accelerating inference in video large language models through dynamic token compression, whereas methods such as LoRA or parameter-efficient fine-tuning primarily target reductions in training resource consumption. Due to these distinct goals, a direct comparison is inherently challenging.\n\n&nbsp;\n\n> **Response to Weakness 3 & Question 3 (End-to-End Latency on Real Hardware)**\n> \n\nTo demonstrate the real-world efficiency gains provided by our method with minimal performance compromise, we have evaluated DyToK using LLaVA-OneVision (7B) on VideoMME with 32-frame inputs. Below we summarize GPU inference latency, FLOPs, and GPU memory consumption across various token retention ratios (10% to 75%). More comprehensive efficiency analyses and experimental results can be found in Appendix D.     \n\n| Method | Retention Ratio | FLOPs (T) | GPU Memory (GB) | Latency (ms) | VideoMME | LongVideoBench | MLVU | Avg. |\n| --- | --- | --- | --- | --- | --- | --- | --- | --- |\n| Vanilla | 100% | 40.8 | 22.2 | 48.3 | 60.7 | 57.7 | 44.7 | 54.4 |\n| DyToK | 75% | 32.6 | 20.2 | 44.1 | 59.3 | 57.0 | 43.1 | 53.1 |\n| DyToK | 50% | 21.9 | 17.3 | 23.5 | 60.2 | 56.8 | 43.8 | 53.6 |\n| DyToK | 25% | 12.2 | 17.3 | 11.3 | 59.9 | 55.3 | 44.5 | 53.2 |\n| DyToK | 20% | 9.6 | 17.3 | 10.7 | 58.4 | 55.4 | 43.7 | 52.5 |\n| DyToK | 15% | 8.4 | 17.3 | 10.4 | 58.4 | 54.1 | 42.5 | 51.7 |\n| DyToK | 10% | 5.9 | 17.3 | 10.8 | 57.1 | 53.1 | 43.6 | 51.3 |\n\n*Note: The extent of memory savings largely depends on model size.*\n\nAs demonstrated, DyToK provides an excellent trade-off between accuracy and efficiency:\n\n- **GPU Memory and Inference Savings:** DyToK achieves a 2.1× inference acceleration at a 50% retention ratio, and this acceleration increases to 4.3× at a 25% retention ratio. Starting from a 50% retention ratio, GPU memory usage remains around 17.3 GB, which essentially represents the minimum memory footprint determined solely by the model weights.\n- **Accuracy-Efficiency Trade-Off:** At a moderate retention ratio of 50%, DyToK retains 98.5% of the original accuracy, reduces inference time by 2.1×, and decreases GPU memory usage by 22.1%. At a 25% retention ratio, DyToK maintains 97.8% accuracy with a 4.3× speedup. Even at the extreme retention ratio of 10% (only ~14 tokens per frame), DyToK maintains 94.3% accuracy.\n\nTo more comprehensively evaluate DyToK’s efficiency, we further conduct long-sequence captioning experiments on VideoChatGPT. Following Dynamic-VLM[1], we use GPT-3.5-Turbo-0613 for quantitative scoring. All statistics include both the assistant and primary models.\n\n| Method | Retention Ratio | Latency (s) | CO | CU | CI | DO | TU | Avg. |\n| --- | --- | --- | --- | --- | --- | --- | --- | --- |\n| Vanilla | 100% | 1.41 | 3.11 | 3.17 | 2.79 | 2.83 | 2.20 | 2.82 |\n| DyToK | 75% | 1.39 | 3.05 | 3.14 | 2.76 | 2.80 | 2.20 | 2.79 |\n| DyToK | 50% | 1.07 | 3.04 | 3.15 | 2.79 | 2.80 | 2.17 | 2.79 |\n| DyToK | 25% | 0.66 | 3.01 | 3.11 | 2.76 | 2.68 | 2.10 | 2.73 |\n| DyToK | 20% | 0.57 | 2.96 | 3.01 | 2.71 | 2.60 | 2.00 | 2.67 |\n| DyToK | 15% | 0.54 | 2.91 | 3.02 | 2.67 | 2.54 | 1.97 | 2.62 |\n| DyToK | 10% | 0.27 | 2.62 | 2.75 | 2.38 | 2.24 | 1.84 | 2.37 |\n\n[1] Dynamic-VLM: Simple Dynamic Visual Token Compression for VideoLLM\n\n&nbsp;\n\n> **Response to Weakness 4 (More Justification for the Attention-Based Importance Estimation)**\n> \n\nThank you for highlighting this issue. We fully agree that different importance estimation strategies warrant further investigation. Following your suggestions, we conduct additional ablation studies by comparing our cross-modal attention-based frame weighting method (Temporal Importance Estimation stage) with entropy-based and magnitude-based approaches:\n\n- **Attention Entropy-Based:** For each frame, compute the attention weights of the last query token over all visual tokens. Calculate the entropy of these attention weights and normalize the results to obtain the frame weights.\n- **Feature Entropy-Based:** Compute the information entropy of visual tokens along the feature dimension. For each frame, average the entropies of all tokens and normalize the values to derive the frame weights.\n- **Feature Magnitude-Based:** Calculate the L2 norm of visual tokens for each frame. Average the norms and normalize the results to obtain the frame weights.\n\n| Method | Retention Ratio | VideoMME | LongVideoBench | MLVU | Avg. | Relative Avg. (%) |\n| --- | --- | --- | --- | --- | --- | --- |\n| Vanilla | N/A | 58.5 | 56.6 | 47.3 | 54.1 | 100.0 |\n| Attention Entropy-Based | 25% | 58.3 | 55.9 | 45.6 | 53.3 | 98.5 |\n| Feature Entropy-Based | 25% | 58.3 | 54.8 | 42.9 | 52.0 | 96.1 |\n| Feature Magnitude-Based | 25% | 58.3 | **56.0** | 44.8 | 53.1 | 98.1 |\n| DyToK | 25% | **58.3** | 55.4 | **46.3** | **53.3** | **98.5** |\n| Attention Entropy-Based | 15% | 55.2 | 52.8 | 43.4 | 50.5 | 93.2 |\n| Feature Entropy-Based | 15% | 55.2 | 52.7 | **44.9** | 50.9 | 94.1 |\n| Feature Magnitude-Based | 15% | 54.7 | 53.3 | 43.8 | 50.6 | 93.5 |\n| DyToK | 15% | **56.7** | **54.2** | 43.4 | **53.3** | **96.0** |\n| Attention Entropy-Based | 10% | 50.6 | 48.1 | 38.6 | 45.7 | 84.5 |\n| Feature Entropy-Based | 10% | 50.7 | 48.8 | 38.4 | 45.9 | 84.9 |\n| Feature Magnitude-Based | 10% | 50.6 | 47.4 | 37.9 | 45.3 | 83.7 |\n| DyToK | 10% | **53.2** | **50.4** | **42.8** | **48.8** | **90.2** |\n\nIt can be observed that DyToK consistently outperforms entropy and magnitude-based methods across different retention ratios. We appreciate your valuable feedback and will include further analysis and discussion in the revised paper.\n\n&nbsp;\n\n> **Response to Question 2 (Definition of Keyframe and Adaptive Rate)**\n> \n- **Definition of Keyframe:** A keyframe is defined as a crucial frame that requires particular attention for accurate video understanding. For a comprehensive explanation, please refer to the Preliminary section (lines 94–101) and the cited literature there.\n- **Mechanism of Adaptive Token Allocation:** DyToK allocates initial token quantities per frame by multiplying frame weights from the temporal importance estimation stage with the specified token budget. It subsequently trims tokens based on an upper limit to mitigate temporal attention outliers (see Appendix B.1) and redistributes remaining tokens according to frame weights in descending order. Implementation details are described in Algorithm 1 of the Methodology section.\n\nWe have completed additional visualizations that provide a clearer explanation of our method. As image attachments are not permitted in the rebuttal phase, these visualizations will be included in the revised paper."}, {"review_id": "6jLbNI6SUZ", "reviewer": "Reviewer_nbsC", "summary": "The paper introduces DyToK, a training-free dynamic token compression framework for Video Large Language Models (VLLMs). By extracting query-conditioned keyframe priors from deep attention layers of a lightweight assistant model, DyToK adaptively allocates per‐frame token budgets and applies any compatible pruning strategy. This plug-and-play approach seamlessly integrates with existing compression backbones—both encoder feature– and attention–based—achieving state-of-the-art efficiency-accuracy trade-offs across multiple long-video benchmarks.", "questions": "Does the reported efficiency improvement include the lightweight assistant model’s forward pass?", "limitations": "yes", "rating": 4, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 4, "significance": 3, "originality": 3, "strengths_and_weaknesses": "Strengths:\n* The motivation for dynamic compression is presented concisely, highlighting inefficiencies of uniform frame pruning.\n* The ablation studies systematically evaluate compression ratios, attention‐layer selection, and assistant model size, providing strong empirical insights.\n* Algorithm 1 is presented with pseudocode that precisely describes the adaptive token allocation and compression steps.\n* DyToK’s training-free paradigm and modular compression function enable straightforward integration with existing token-pruning methods.\n* The paper demonstrates insightful finding that depth of attention is more critical than model scale for importance estimation.\nDyToK reliably boosts accuracy under aggressive pruning across diverse benchmarks and methods.\n\nWeaknesses:\n*  While existing binary keyframe sampling of VLLM is discussed, the method also closely relates to the retrieval-based video understanding methods, such as rag-based methods(Video-rag: Visually-aligned retrieval-augmented long video comprehension). It would add clarity if the authors discuss and compare with that line of methods.\n*  Eq. (1) quantifies total FLOPs, but the paper does not present calculated FLOP savings for DyToK relative to baselines.\n*  It remains unclear how different assistant models (a wider ranges besides 7B and 0.5B) influence the performance, and how mispredictions of assistant models propagate to the primary VLLM.", "was_revised": false, "final_justification": "", "rebuttal": "We sincerely appreciate your comments. All suggestions will be incorporated into the paper. Our responses are as follows.\n\n&nbsp;\n\n> **Response to Weakness 1 (More Discussion with RAG-Based Methods)**\n> \n\nThank you for your insightful suggestion. Video-RAG primarily focuses on enhancing video comprehension through auxiliary textual information extracted by external speech recognition, OCR, and object detection modules. In contrast, our method specifically targets efficient long-video reasoning and extended video support within a given computational budget, making a direct comparison challenging. A commonality between the two methods is that Video-RAG employs CLIP-based keyframe extraction in its object detection stage.\n\nTo facilitate a clearer comparison regarding the handling of keyframes, we incorporate Video-RAG’s keyframe extraction module into DyToK’s temporal importance estimation stage. Specifically, we utilize CLIP’s cross-modal semantic similarity scores to determine frame weights, dynamically allocating token retention ratios for each frame. We present experimental results on VideoMME using LLaVA-OneVision as the base model under different retention ratios below. We will incorporate these discussions and experimental results into our paper.\n\n| Method  | Retention Ratio | Short | Medium | Long | Overall | Relative Acc. (%) |\n| --- | --- | --- | --- | --- | --- | --- |\n| Vanilla | N/A | 69.9 | 56.7 | 48.9 | 58.5 | 100 |\n| Video-RAG | 25% | 68.3 | **56.9** | **49.1** | 58.1 | 99.4 |\n| DyToK | 25% | **69.2** | 56.8 | 48.9 | **58.3** | **99.7** |\n| Video-RAG | 15% | 64.6 | 53.9 | 48.4 | 55.6 | 95.1 |\n| DyToK | 15% | **66.3** | **55.0** | **48.7** | **56.7** | **97.0** |\n| Video-RAG | 10% | 57.6 | 50.9 | 45.8 | 51.4 | 87.9 |\n| DyToK | 10% | **61.0** | **51.1** | **47.6** | **53.2** | **91.0** |\n\n&nbsp;\n\n> **Response to Weakness 2 (Relative FLOPs Saving Calculation)**\n> \n\nWe appreciate your valuable feedback on presenting the relative computational savings. To intuitively demonstrate the practical efficiency gains achieved with minimal performance trade-offs, we conduct experiments on VideoMME using LLaVA-OneVision (7B) as the base model, evaluating GPU inference latency, FLOPs, and memory usage across a broad retention ratio range (10%-75%) with 32-frame input. The summarized results are presented in the following table, with detailed efficiency analysis provided in Appendix D. All statistics include both the assistant and primary models.\n\n| Method | Retention Ratio | FLOPs (T) | GPU Memory (GB) | Latency (ms) | VideoMME | LongVideoBench | MLVU | Avg. |\n| --- | --- | --- | --- | --- | --- | --- | --- | --- |\n| Vanilla | 100% | 40.8 | 22.2 | 48.3 | 60.7 | 57.7 | 44.7 | 54.4 |\n| DyToK | 75% | 32.6 | 20.2 | 44.1 | 59.3 | 57.0 | 43.1 | 53.1 |\n| DyToK | 50% | 21.9 | 17.3 | 23.5 | 60.2 | 56.8 | 43.8 | 53.6 |\n| DyToK | 25% | 12.2 | 17.3 | 11.3 | 59.9 | 55.3 | 44.5 | 53.2 |\n| DyToK | 20% | 9.6 | 17.3 | 10.7 | 58.4 | 55.4 | 43.7 | 52.5 |\n| DyToK | 15% | 8.4 | 17.3 | 10.4 | 58.4 | 54.1 | 42.5 | 51.7 |\n| DyToK | 10% | 5.9 | 17.3 | 10.8 | 57.1 | 53.1 | 43.6 | 51.3 |\n\n*Notably, the extent of memory savings significantly depends on the model size.*\n\nFrom the table above, DyToK achieves an excellent trade-off between accuracy and efficiency:\n\n- **GPU Memory and Inference Savings:** DyToK achieves a 2.1× inference acceleration at a 50% retention ratio, and this acceleration increases to 4.3× at a 25% retention ratio. Starting from a 50% retention ratio, GPU memory usage remains around 17.3 GB, which essentially represents the minimum memory footprint determined solely by the model weights.\n- **Accuracy-Efficiency Trade-Off:** At a moderate retention ratio of 50%, DyToK retains 98.5% of the original accuracy, reduces inference time by 2.1×, and decreases GPU memory usage by 22.1%. At a 25% retention ratio, DyToK maintains 97.8% accuracy with a 4.3× speedup. Even at the extreme retention ratio of 10% (only ~14 tokens per frame), DyToK maintains 94.3% accuracy.\n\nTo more comprehensively evaluate DyToK’s efficiency, we further conduct long-sequence captioning experiments on VideoChatGPT. Following Dynamic-VLM[1], we use GPT-3.5-Turbo-0613 for quantitative scoring. All statistics include both the assistant and primary models.\n\n| Method | Retention Ratio | Latency (s) | CO | CU | CI | DO | TU | Avg. |\n| --- | --- | --- | --- | --- | --- | --- | --- | --- |\n| Vanilla | 100% | 1.41 | 3.11 | 3.17 | 2.79 | 2.83 | 2.20 | 2.82 |\n| DyToK | 75% | 1.39 | 3.05 | 3.14 | 2.76 | 2.80 | 2.20 | 2.79 |\n| DyToK | 50% | 1.07 | 3.04 | 3.15 | 2.79 | 2.80 | 2.17 | 2.79 |\n| DyToK | 25% | 0.66 | 3.01 | 3.11 | 2.76 | 2.68 | 2.10 | 2.73 |\n| DyToK | 20% | 0.57 | 2.96 | 3.01 | 2.71 | 2.60 | 2.00 | 2.67 |\n| DyToK | 15% | 0.54 | 2.91 | 3.02 | 2.67 | 2.54 | 1.97 | 2.62 |\n| DyToK | 10% | 0.27 | 2.62 | 2.75 | 2.38 | 2.24 | 1.84 | 2.37 |\n\n[1] Dynamic-VLM: Simple Dynamic Visual Token Compression for VideoLLM\n\n&nbsp;\n\n> **Response to Weakness 3(a) (Broader Model Size)**\n> \n\nThank you for your valuable suggestion. Beyond the **0.5B** guiding **7B** experiments using LLaVA-OneVision already presented in the main text, we conduct additional extensive experiments using Qwen2.5-VL on the MLVU benchmark, which covers a wide range of commonly used model sizes (**3B**, **7B,** and **32B**), as shown below.\n\n| Method | 100% | 25% | 20% | 15% |\n| --- | --- | --- | --- | --- |\n| Vanilla | 41.4 | N/A | N/A | N/A |\n| FastV | N/A | 23.6 | 22.3 | 17.1 |\n| DyToK (3B guides 32B) | N/A | 35.5 | **35.4** | 29.9 |\n| DyToK (7B guides 32B) | N/A | **36.3** | 34.9 | **33.9** |\n\nThese experimental results provide two further insights: \n\n1. **Video models inherently possess keyframe priors regardless of model size.** \n2. **Such priors are not constrained by model scale, as even lightweight models can exhibit keyframe perception capabilities comparable to those of larger models.** Notably, at a 20% retention ratio, the 3B Qwen2.5-VL assistant model surpasses the 7B variant by a substantial margin, further validating the feasibility of using a lightweight assistant model to provide keyframe priors in DyToK.\n\n&nbsp;\n\n> **Response to Weakness 3(b) (More Analysis on Assistant Model Misprediction Propagation)**\n> \n\nThank you for raising this critical point. In the DyToK framework, the assistant model’s role is strictly confined to extracting frame weights based on keyframe priors, guiding dynamic token retention without directly passing predictions to the primary model.\n\nAs illustrated in Figure 1 of the main text, we have empirically found that **even when the assistant model’s predictions are incorrect, its cross-modal attention remains accurately aligned with keyframes**. Consequently, the transferred frame weights remain generally reliable, reducing the risk of error propagation. This robustness underlies DyToK’s consistent effectiveness across various base models, benchmarks, and retention ratios.\n\nFollowing your suggestion, we will expand discussions on misprediction propagation in the main text and provide further visual analysis beyond the current visualizations in Appendix Figures 7-14.\n\n&nbsp;\n\n> **Response to Question (Efficiency Improvement Confirmation)**\n> \n\n**Yes.** To ensure fairness, efficiency gains reported in our study (Appendix D) include the forward passes of both assistant and primary models."}, {"review_id": "pm2ei5vS8T", "reviewer": "Reviewer_mobj", "summary": "This paper introduces DyTok , a training-free dynamic token compression method that leverages LLM-guided keyframe priors. Given a sequence of visual tokens, DyTok employs a LLM to assess the importance of each token by computing its similarity with the textual query tokens. Based on these importance scores, the method selectively retains the most informative tokens while maintaining a balanced token distribution across frames. DyTok is evaluated on multiple video understanding benchmarks.", "questions": "Please refer to the weaknesses part for my concerns. My initial judgment is BR, and I will adjust my rating accordingly based on the authors' response.", "limitations": "yes", "rating": 4, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 3, "significance": 2, "originality": 2, "strengths_and_weaknesses": "Strengths:\n* The paper is well-written and clearly structured, making it easy to follow.\n* The proposed idea is conceptually straightforward, computationally efficient, and can be integrated into various methods.\n* Extensive experimental results demonstrate the effectiveness and robustness of DyTok across multiple tasks and datasets.\n\nWeaknesses:\n* The paper computes attention scores between the final textual query token and the visual tokens. However, it would be helpful to include an analysis on the choice of textual token representation. While the last token may capture global query information, in some cases, it might lack sufficient semantic richness. \n* Although the auxiliary lightweight assistant model helps reduce computational cost, it still relies on strong pre-trained VLMs, such as LLaVA-OneVision and Qwen2.5-VL. This dependency may limit the generalizability and accessibility of DyTok across different frameworks or less powerful backbones.\n* The selection of deep layers appears to be based on specific architectural choices. It is unclear whether this selection is model-agnostic or tailored to particular backbones. If it is not, then the analysis presented in Table 3 may need to be repeated for each new model architecture. Is there evidence supporting the transferability of this layer selection strategy?\n* In the appendix Table 7, instead of only computing prefilling time, what about the inference time of the whole pipeline? As an additional model is used, I'd expect the DyTok pipeline to have longer inference time overall.", "was_revised": true, "final_justification": "My initial judgment is on the borderline. The authors' rebuttal addressed my concerns, especially the transferability of the layer selection. Although I'm still on the borderline, I lean towards to the positive side, and my final rating is BA.", "rebuttal": "We sincerely appreciate your comments. All suggestions will be incorporated into the paper. Our responses are as follows.\n\n&nbsp;\n\n> **Response to Weakness 1 (Additional Analysis on Textual Token Selection)**\n\nThank you for the insightful suggestion. We agree that relying solely on the last textual token may not fully capture the sequence’s semantics. To explore this, we conduct ablations on multiple long video benchmarks using LLaVA-OneVision (7B) under various retention ratios.\n\n|Method|Retention Ratio|VideoMME|LongVideoBench|MLVU|Avg.|Relative Avg. (%)|\n|---|---|---|---|---|---|---|\n|Vanilla|N/A|58.5|56.6|47.1|54.1|100|\n|FastV|75%|57.6|57.1|46.5|53.7|99.3|\n|DyToK (all query tokens)|75%|58.1|**57.1**|45.4|53.5|98.9|\n|DyTok (last query token)|75%|**58.4**|56.8|**46.8**|**54.0**|**99.8**|\n|FastV|50%|57.2|57.1|44.7|53.0|98.0|\n|DyToK (all query tokens)|50%|58.2|**58.0**|45.6|53.9|99.6|\n|DyToK (last query token)|50%|**58.5**|57.2|**46.3**|**54.0**|**99.8**|\n|FastV|25%|56.0|56.6|43.7|52.1|96.3|\n|DyToK (all query tokens)|25%|**56.9**|**58.0**|47.3|**54.1**|**100**|\n|DyTok (last query token)|25%|56.3|55.7|**47.8**|53.3|98.5|\n|FastV|15%|51.1|51.2|38.3|46.9|86.7|\n|DyToK (all query tokens)|15%|54.1|**53.1**|**44.4**|**50.5**|**93.3**|\n|DyTok (last query token)|15%|**54.8**|52.6|43.2|50.2|92.8|\n\n\n\nAs shown above, at higher retention ratios (e.g., 75%, 50%), using only the last query token performs better. At more aggressive ratios (e.g., 25%, 15%), using all query tokens yields further gains. We hypothesize that the last token acts as a precise but narrow selector, while all tokens offer broader coverage with higher recall. When visual evidence is abundant, precision reduces noise; when it is scarce, broader semantic coverage becomes essential.\n\nWe will incorporate these analyses into the final version and add visualizations comparing the distributions of frame weights derived from both strategies.\n\n&nbsp;\n\n> **Response to Weakness 2 (Concern on Cross-Architecture Transferability and Weaker Backbone Adaptability)**\n\nWe sincerely appreciate your insightful suggestion. We agree that validating DyToK's transferability across different model architectures is crucial to demonstrate its generalizability and accessibility. To address this, we conduct experiments on VideoMME, using assistant and primary models with distinctly different architectures under varying retention ratios.\n\nTo maximize the differences in architectures and thus rigorously test DyToK's robustness, we select LLaVA-OneVision (OV) and Qwen2.5-VL (Qwen), two open-source frameworks exhibiting significant architectural differences. We believe that strong performance under such settings provides meaningful evidence of generalizability.\n\nPreliminary results are shown below, with OV using 0.5B/7B and Qwen using 3B/7B as assistant and primary models.\n\n|Method|Retention Ratio|Short|Medium|Long|Overall|Relative Acc. (%)|\n|---|---|---|---|---|---|---|\n|Vanilla (OV)|N/A|69.9|56.7|48.9|58.5|100.0|\n|VisionZip|75%|68.9|56.2|48.2|57.8|98.8|\n|DyToK (OV guides OV)|75%|70.8|**57.2**|**48.9**|**59.0**|**100.9**|\n|DyToK (Qwen guides OV)|75%|**71.4**|57.0|48.3|58.9|100.7|\n|VisionZip|50%|66.8|56.6|48.2|57.2|97.8|\n|DyToK (OV guides OV)|50%|**71.9**|56.1|**49.3**|**59.1**|**101.0**|\n|DyToK (Qwen guides OV)|50%|71.6|**56.6**|48.1|58.7|100.3|\n|VisionZip|25%|62.2|53.0|47.4|54.2|92.6|\n|DyToK (OV guides OV)|25%|69.2|**56.8**|**48.9**|58.3|99.7|\n|DyToK (Qwen guides OV)|25%|**71.3**|56.6|48.1|**58.9**|**100.7**|\n|VisionZip|20%|60.2|51.0|45.7|52.3|89.4|\n|DyToK (OV guides OV)|20%|**69.2**|**56.4**|47.8|**57.8**|**98.8**|\n|DyToK (Qwen guides OV)|20%|68.7|55.9|**48.2**|57.6|98.5|\n|VisionZip|15%|55.9|49.9|44.3|50.0|85.5|\n|DyToK (OV guides OV)|15%|66.3|**55.0**|48.7|**56.7**|**96.9**|\n|DyToK (Qwen guides OV)|15%|**67.3**|53.7|**48.9**|56.6|96.8|\n|VisionZip|10%|47.0|44.2|42.2|44.5|76.1|\n|DyToK (OV guides OV)|10%|**61.0**|51.1|47.6|53.2|90.9|\n|DyToK (Qwen guides OV)|10%|59.2|**53.3**|**47.9**|**53.5**|**91.5**|\n\n|Method|Retention Ratio|Short|Medium|Long|Overall|Relative Acc. (%)|\n|---|---|---|---|---|---|---|\n|Vanilla (Qwen)|N/A|72.6|61.4|49.9|61.3|100.0|\n|FastV|75%|70.0|57.4|50.4|59.3|96.7|\n|DyToK (Qwen guides Qwen)|75%|72.6|**60.6**|**51.0**|**61.4**|**100.1**|\n|DyToK (OV guides Qwen)|75%|**72.7**|60.1|50.2|61.0|99.5|\n|FastV|50%|63.9|52.7|48.0|54.9|89.5|\n|DyToK (Qwen guides Qwen)|50%|**72.0**|60.2|**51.7**|**61.3**|**100.0**|\n|DyToK (OV guides Qwen)|50%|71.8|**60.4**|51.0|61.1|99.6|\n|FastV|25%|59.9|49.1|46.8|51.9|84.7|\n|DyToK (Qwen guides Qwen)|25%|**71.7**|**58.9**|51.3|**60.6**|**98.9**|\n|DyToK (OV guides Qwen)|25%|71.3|58.3|**51.8**|60.5|98.7|\n|FastV|20%|57.8|48.4|45.6|50.6|82.5|\n|DyToK (Qwen guides Qwen)|20%|**70.8**|**57.7**|**51.4**|**60.0**|**97.8**|\n|DyToK (OV guides Qwen)|20%|**70.8**|57.2|51.2|59.7|97.5|\n|FastV|15%|54.9|45.0|44.2|48.0|78.4|\n|DyToK (Qwen guides Qwen)|15%|**68.4**|**56.3**|**50.1**|**58.3**|**95.1**|\n|DyToK (OV guides Qwen)|15%|67.2|55.1|49.7|57.3|93.5|\n\nThe results show that DyToK remains robust even with cross-architecture assistant-primary model pairs. In some cases, using an assistant from a different architecture even outperforms one from the same series under certain compression ratios.\n\nWe attribute this to the assistant’s key role in estimating frame importance. As long as this estimation is accurate, architectural alignment is not essential. Additionally, when the models share similar architectures or training recipes, aligned attention distributions may further boost performance.\n\nThese comprehensive experimental findings broaden DyToK's practical applicability in various scenarios:\n\n- **No lightweight version available**: A lightweight variant from another model, such as LLaVA-OneVision 0.5B, can serve effectively as an assistant model, given its proven efficacy and extremely small parameter count.\n- **Inadequate capacity of lightweight models**: If a lightweight model lacks sufficient capability to generate accurate frame weights, a lightweight variant from another pre-trained video large language model can readily serve as the assistant model.\n\nWe hope this addresses your concerns regarding DyToK’s cross-architecture transferability and robustness with weaker backbones. Given the importance of this issue to DyToK’s versatility, we will elaborate on this in the main text and add visualizations in the appendix to illustrate attention distribution differences across assistant architectures.\n\n&nbsp;\n\n> **Response to Weakness 3 (Additional Analysis on Deep Layer Selection)**\n\nAs stated in Section 3.2 (line 154), we define deep layers as the last 25% of the assistant model’s layers to ensure generalizability without architecture-specific tuning. This proportional strategy adapts well across models like LLaVA-OneVision and Qwen2.5-VL, making it unnecessary to repeat Table 3 experiments for each new model.\n\nThe ablation in Table 3 serves to validate our key observation from Section 2.2 (line 113) that “deep attention layers provide good keyframe priors.” Appendix A.4 (Table 5) further analyzes individual layers and combinations divided into quarters and sixths, confirming the benefits of aggregating deep layers.\n\nThese results support, but do not affect, our practical choice of using the last 25% layers. We appreciate the reviewer’s suggestion and will clarify the distinction between our empirical analyses and adopted strategy in Section 3.2 to avoid misunderstanding.\n\nTo further illustrate the transferability of our approach, we also conduct additional layer analyses on LLaVA-OneVision and Qwen2.5-VL.\n\n*LLaVA-OneVision*\n\n||VideoMME|LongVideoBench|MLVU|Avg.|\n|---|---|---|---|---|\n|All Layers|57.1|54.3|44.7|52.0|\n|Shallow 25%|55.5|53.6|43.7|50.9|\n|Mid-Shallow 25%|56.4|54.1|43.7|51.4|\n|Mid-Deep 25%|57.5|55.1|42.7|51.8|\n|Deep 25%|**57.8**|**56.0**|**45.1**|**53.0**|\n\n*Qwen2.5-VL*\n\n||VideoMME|LongVideoBench|MLVU|Avg.|\n|---|---|---|---|---|\n|All Layers|58.0|54.3|41.9|51.4|\n|Shallow 25%|57.6|54.2|42.1|51.3|\n|Mid-Shallow 25%|57.9|54.0|**42.6**|51.5|\n|Mid-Deep 25%|58.7|54.9|42.2|51.9|\n|Deep 25%|**59.0**|**55.8**|41.7|**52.2**|\n\nThe experimental results demonstrate that our layer selection strategy is model- and architecture-agnostic, exhibiting generalizability by eliminating the need for repeated layer analyses when adapting to new models.\n\n&nbsp;\n\n> **Response to Weakness 4 (Inference Time Clarification)**\n\nThe “prefilling time” reported in Appendix Table 7 reflects the combined inference time of the assistant and primary models during the generation of the first token in DyToK. We focus on this metric because DyToK’s additional computation occurs entirely during prefilling. In contrast, decoding involves only the primary model operating on pruned tokens, leading to consistent speedups. Measuring total generation time would understate DyToK’s actual overhead, as the assistant’s cost becomes negligible over longer sequences. Therefore, prefilling time offers a fair and transparent efficiency measure.\n\nWe further validate this choice through additional experiments on long-sequence captioning with VideoChatGPT, which confirm our analysis.\n\n|Method|Retention Ratio|Latency (s)|CO|CU|CI|DO|TU|Avg.|\n|---|---|---|---|---|---|---|---|---|\n|Vanilla|100%|1.41|3.11|3.17|2.79|2.83|2.20|2.82|\n|DyToK|75%|1.39|3.05|3.14|2.76|2.80|2.20|2.79|\n|DyToK|50%|1.07|3.04|3.15|2.79|2.80|2.17|2.79|\n|DyToK|25%|0.66|3.01|3.11|2.76|2.68|2.10|2.73|\n|DyToK|20%|0.57|2.96|3.01|2.71|2.60|2.00|2.67|\n|DyToK|15%|0.54|2.91|3.02|2.67|2.54|1.97|2.62|\n|DyToK|10%|0.27|2.62|2.75|2.38|2.24|1.84|2.37|"}, {"review_id": "ayIlGhDgzX", "reviewer": "Reviewer_bZfF", "summary": "This paper addresses the inefficiency of Video Large Language Models (VLLMs) in processing long videos, where computational costs rise due to redundant visual information. The authors propose DyToK, a training-free paradigm for dynamic token compression during inference. It leverages attention scores from VLLMs to distinguish between key frames and less relevant ones, adapting each frame’s token retention ratio based on its estimated importance. The method is plug-and-play compatible with existing compression techniques and applicable across multiple VLLM families. The authors conduct extensive experiments, visualizations, and analyses, showing that DyToK effectively identifies relevant key frames and achieves superior efficiency–accuracy trade-offs across various video benchmarks.", "questions": "1. The auxiliary lightweight assistant model used to obtain attention maps is derived from the same architectural family as the main model. However, I believe its feasibility and generalizability require further clarification. a) In addition to the final benchmark results, it would be helpful to include visualizations or statistical analyses. b) This design choice may also limit the method’s generalizability, especially for models that lack a corresponding lightweight version or where the lightweight version differs significantly in architecture. c) What would happen if a lightweight model from a different architectural family were used?\n2. The practical benefits of reducing the number of tokens can be better demonstrated. For example, the actual speed-up in inference time with the same number of input frames, or the performance gains under a fixed computational budget.\n3. Although the authors conducted extensive experiments and analyses, the presentation could be improved. For instance, performance trends across different token compression ratios could be presented more clearly, and visualizing the keyframe scores more intuitively in the main text would help better support the proposed method.\n4. a) Using only the last token of the query to score the frames may overlook some important information. b) If the query does not explicitly point to specific visual content (e.g., a general descriptive query), can this method still effectively select important frames in the video?\n\nMy overall score may be adjusted based on the authors’ responses to these questions.", "limitations": "In the limitations section, the authors mention that, to balance performance and efficiency, they introduced an auxiliary lightweight assistant model to approximate the original model. I believe the effectiveness and generalizability of this approach require further analysis and discussion, as noted in the questions above.", "rating": 4, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 2, "clarity": 2, "significance": 3, "originality": 3, "strengths_and_weaknesses": "Strengths:\n- Quality: The proposed method is technically sound and thoroughly evaluated through extensive experiments across multiple VLLMs and token compression techniques. The authors also provide visualizations and analyses to support their claims.\n- Clarity: The paper is clearly written and easy to follow. Technical details are well-explained, with helpful insights that facilitate a clear understanding.\n- Significance: The work addresses the critical challenge of improving efficiency in long-video understanding for VLLMs. The analysis of attention maps across different layers for keyframe identification is insightful and potentially valuable for future research.\n- Originality: The training-free, plug-and-play paradigm and its seamless integration with existing compression techniques offer a novel and practical perspective on advancing token compression in VLLMs.\n\nWeakness:\n- Quality: The method relies on an auxiliary lightweight model from the same architectural family to approximate the original model. The effectiveness and generalizability of this approach are not fully validated and warrant further analysis and discussion.\n- Clarity: There are several noticeable typos in the paper, such as the placeholder “on XX benchmarks” in Line 65. The authors should proofread the manuscript more carefully.\n- Significance: The practical benefits of token reduction, such as inference speed-up or extended video support under fixed computational budgets, are not sufficiently demonstrated in the paper.", "was_revised": true, "final_justification": "My initial judgment was borderline. The authors have provided additional results that reveal the underlying relationships between attention patterns across different architectures and models, which support the proposed method. Although my assessment remains borderline, I now lean toward the positive side, and my final rating is BA.", "rebuttal": "We sincerely appreciate your insightful comments. All suggestions will be incorporated into the paper, and typos will be fixed. Our responses are as follows.\n\n&nbsp;\n> **Response to Weakness 1 & Question 1(b, c) (Concern about Cross-Architecture Transferability)**\n\nTo better demonstrate DyToK's transferability, we conduct experiments on VideoMME, using assistant and primary models with distinctly different architectures under varying retention ratios.\n\nTo maximize the differences in architectures and thus rigorously test DyToK's robustness, we select LLaVA-OneVision (OV) and Qwen2.5-VL (Qwen), two open-source frameworks exhibiting significant architectural differences.\n\nPreliminary results are shown below, with OV using 0.5B/7B and Qwen using 3B/7B as assistant and primary models.\n\n|Method|Retention Ratio|Short|Medium|Long|Overall|Relative Acc. (%)|\n|---|---|---|---|---|---|---|\n|Vanilla (OV)|N/A|69.9|56.7|48.9|58.5|100.0|\n|VisionZip|50%|66.8|56.6|48.2|57.2|97.8|\n|DyToK (OV guides OV)|50%|**71.9**|56.1|**49.3**|**59.1**|**101.0**|\n|DyToK (Qwen guides OV)|50%|71.6|**56.6**|48.1|58.7|100.3|\n|VisionZip|25%|62.2|53.0|47.4|54.2|92.6|\n|DyToK (OV guides OV)|25%|69.2|**56.8**|**48.9**|58.3|99.7|\n|DyToK (Qwen guides OV)|25%|**71.3**|56.6|48.1|**58.9**|**100.7**|\n|VisionZip|20%|60.2|51.0|45.7|52.3|89.4|\n|DyToK (OV guides OV)|20%|**69.2**|**56.4**|47.8|**57.8**|**98.8**|\n|DyToK (Qwen guides OV)|20%|68.7|55.9|**48.2**|57.6|98.5|\n|VisionZip|15%|55.9|49.9|44.3|50.0|85.5|\n|DyToK (OV guides OV)|15%|66.3|**55.0**|48.7|**56.7**|**96.9**|\n|DyToK (Qwen guides OV)|15%|**67.3**|53.7|**48.9**|56.6|96.8|\n|VisionZip|10%|47.0|44.2|42.2|44.5|76.1|\n|DyToK (OV guides OV)|10%|**61.0**|51.1|47.6|53.2|90.9|\n|DyToK (Qwen guides OV)|10%|59.2|**53.3**|**47.9**|**53.5**|**91.5**|\n\n|Method|Retention Ratio|Short|Medium|Long|Overall|Relative Acc. (%)|\n|---|---|---|---|---|---|---|\n|Vanilla (Qwen)|N/A|72.6|61.4|49.9|61.3|100.0|\n|FastV|50%|63.9|52.7|48.0|54.9|89.5|\n|DyToK (Qwen guides Qwen)|50%|**72.0**|60.2|**51.7**|**61.3**|**100.0**|\n|DyToK (OV guides Qwen)|50%|71.8|**60.4**|51.0|61.1|99.6|\n|FastV|25%|59.9|49.1|46.8|51.9|84.7|\n|DyToK (Qwen guides Qwen)|25%|**71.7**|**58.9**|51.3|**60.6**|**98.9**|\n|DyToK (OV guides Qwen)|25%|71.3|58.3|**51.8**|60.5|98.7|\n|FastV|20%|57.8|48.4|45.6|50.6|82.5|\n|DyToK (Qwen guides Qwen)|20%|**70.8**|**57.7**|**51.4**|**60.0**|**97.8**|\n|DyToK (OV guides Qwen)|20%|**70.8**|57.2|51.2|59.7|97.5|\n|FastV|15%|54.9|45.0|44.2|48.0|78.4|\n|DyToK (Qwen guides Qwen)|15%|**68.4**|**56.3**|**50.1**|**58.3**|**95.1**|\n|DyToK (OV guides Qwen)|15%|67.2|55.1|49.7|57.3|93.5|\n\nResults show that DyToK remains robust even with cross-architecture assistant-primary model pairs. In some cases, using an assistant from a different architecture even outperforms one from the same series under certain compression ratios.\n\nWe attribute this to the assistant’s key role in estimating frame importance. As long as this estimation is accurate, architectural alignment is not essential. Additionally, when the models share similar architectures or training recipes, aligned attention distributions may further boost performance.\n\n&nbsp;\n> **Response to Weakness 3 & Question 2 (More Experimental Settings)**\n\n**(1) Inference speed-up with fixed input frames**\n\nUsing LLaVA-OneVision (7B) on VideoMME with 32-frame inputs, we have evaluated DyToK across retention ratios from 10% to 75%. All statistics include both the assistant and primary models. Results show substantial GPU memory and latency reductions with minimal accuracy loss. At 50% retention, DyToK achieves 2.1× speed-up, 22.1% memory savings, and 98.5% accuracy. Even at an aggressive 10% retention, it retains 94.3% accuracy with 4.5× speed-up.\n\n|Method|Retention Ratio|FLOPs (T)|GPU Mem (GB)|Latency (ms)|VideoMME|LongVideoBench|MLVU|Avg.|\n|---|---|---|---|---|---|---|---|---|\n|Vanilla|100%|40.8|22.2|48.3|60.7|57.7|44.7|54.4|\n|DyToK|75%|32.6|20.2|44.1|59.3|57.0|43.1|53.1|\n|DyToK|50%|21.9|17.3|23.5|60.2|56.8|43.8|53.6|\n|DyToK|25%|12.2|17.3|11.3|59.9|55.3|44.5|53.2|\n|DyToK|20%|9.6|17.3|10.7|58.4|55.4|43.7|52.5|\n|DyToK|15%|8.4|17.3|10.4|58.4|54.1|42.5|51.7|\n|DyToK|10%|5.9|17.3|10.8|57.1|53.1|43.6|51.3|\n\n*Please note: The extent of memory savings largely depends on the model size.*\n\n**(2) Performance gains under fixed computational budgets**\n\nWe evaluate DyToK’s capability to support longer video sequences under the same token budget as the uncompressed baseline, using Qwen2.5-VL (7B) on MLVU (token budget alignment details in Appendix H.2).\n\n*Token Budget Alignment*\n|Method|Retention Ratio|#Frames|MLVU|\n|---|---|---|---|\n|Vanilla|100%|32|44.7|\n|DyToK|57.1%|56|45.6|\n|DyToK|50%|**64**|**47.9**|\n\nWe attribute the accuracy gains to DyToK’s token compression, which reduces tokens per frame and enables more frames under a fixed token budget, improving temporal granularity.\n\nTo address efficiency concerns from the assistant model, we also evaluate DyToK under a fixed FLOPs budget matching the uncompressed baseline, ensuring fair computational comparison. As Qwen2.5-VL yields variable token counts due to dynamic resolution, we use LLaVA-OneVision (7B) for consistent FLOPs estimation.\n\n*FLOPs Budget Alignment*\n|Method|Retention Ratio|#Frames|MLVU|\n|---|---|---|---|\n|Vanilla|100%|64|48.0|\n|DyToK|57.1%|96|48.8|\n|DyToK|49.5%|**106**|**49.5**|\n\n&nbsp;\n> **Response to Question 1(a) & Question 3 (More Visualization)**\n\nWe appreciate your constructive feedback regarding visualization. Due to rebuttal format limits, we cannot include visuals now but will add the following in the final version:\n\n- Attention distribution across different assistant model architectures\n- Performance trends across varying token retention ratios\n- Selected keyframe visualizations moved from Appendix (Figs. 7-14) to the main text\n- Comparative frame weight distributions from all query tokens vs. only the last query token\n- Attention distributions for general description queries\n\nPlease feel free to suggest additional visualizations.\n\n&nbsp;\n> **Response to Question 4(a) (Additional Analysis on Textual Token Selection)**\n\nWe agree that relying solely on the last textual token may not fully capture the sequence’s semantics. To explore this, we conduct ablations on multiple long video benchmarks using LLaVA-OneVision (7B) under various retention ratios.\n\n|Method|Retention Ratio|VideoMME|LongVideoBench|MLVU|Avg.|Relative Avg. (%)|\n|---|---|---|---|---|---|---|\n|Vanilla|N/A|58.5|56.6|47.1|54.1|100|\n|FastV|75%|57.6|57.1|46.5|53.7|99.3|\n|DyToK (all query tokens)|75%|58.1|**57.1**|45.4|53.5|98.9|\n|DyTok (last query token)|75%|**58.4**|56.8|**46.8**|**54.0**|**99.8**|\n|FastV|50%|57.2|57.1|44.7|53.0|98.0|\n|DyToK (all query tokens)|50%|58.2|**58.0**|45.6|53.9|99.6|\n|DyTok (last query token)|50%|**58.5**|57.2|**46.3**|**54.0**|**99.8**|\n|FastV|25%|56.0|56.6|43.7|52.1|96.3|\n|DyToK (all query tokens)|25%|**56.9**|**58.0**|47.3|**54.1**|**100**|\n|DyTok (last query token)|25%|56.3|55.7|**47.8**|53.3|98.5|\n|FastV|15%|51.1|51.2|38.3|46.9|86.7|\n|DyToK (all query tokens)|15%|54.1|**53.1**|**44.4**|**50.5**|**93.3**|\n|DyTok (last query token)|15%|**54.8**|52.6|43.2|50.2|92.8|\n\nAs shown above, at higher retention ratios (e.g., 75%, 50%), using only the last query token performs better. At more aggressive ratios (e.g., 25%, 15%), using all query tokens yields further gains. We hypothesize that the last token acts as a precise but narrow selector, while all tokens offer broader coverage with higher recall. When visual evidence is abundant, precision reduces noise; when it is scarce, broader semantic coverage becomes essential.\n\n&nbsp;\n> **Response to Question 4(b) (Concern about General Descriptive Queries)**\n\n**Quantitative Analysis.** To assess DyToK’s ability to select keyframes for general descriptive queries, we conduct captioning experiments on VideoChatGPT using LLaVA-OneVision (7B), which includes prompts like “What is happening in the video?” Following Dynamic-VLM[1], we use GPT-3.5-Turbo-0613 for quantitative scoring.\n\n|Method|Retention Ratio|CO|CU|CI|DO|TU|Avg.|Relative Avg. (%)|\n|---|---|---|---|---|---|---|---|---|\n|Vanilla|N/A|3.11|3.17|2.79|2.83|2.20|2.82|100.0|\n|VisionZip|50%|2.91|3.08|2.72|2.74|2.03|2.70|95.7|\n|DyToK|50%|**3.04**|**3.15**|**2.79**|**2.80**|**2.17**|**2.79**|**98.9**|\n|VisionZip|25%|2.62|2.86|2.50|2.42|1.81|2.44|86.5|\n|DyToK|25%|**3.01**|**3.11**|**2.76**|**2.68**|**2.10**|**2.73**|**96.8**|\n|VisionZip|15%|2.20|2.48|2.14|2.08|1.61|2.10|74.5|\n|DyToK|15%|**2.91**|**3.02**|**2.67**|**2.54**|**1.97**|**2.62**|**92.9**|\n|VisionZip|10%|1.46|1.85|1.51|1.58|1.24|1.53|54.3|\n|DyToK|10%|**2.62**|**2.75**|**2.38**|**2.24**|**1.84**|**2.37**|**84.0**|\n\n**Qualitative Analysis.** In visualization experiments, we have observed that DyToK effectively identifies both “semantic keyframes” (relevant to query semantics) and “content keyframes” (capturing key scene or event transitions), making it well-suited for general descriptive queries. This demonstrates an advantage over CLIP-based methods that rely solely on semantic matching. While visualizations cannot be included due to rebuttal constraints, they will appear in the final version along with detailed results.\n\nTo support our claim during the rebuttal phase, we replace DyToK’s cross-modal attention-based frame weight estimation with CLIP-based semantic similarity. Implementation details follow AKS[2]. Results remain consistent.\n\n|Method|Retention Ratio|Short|Medium|Long|Overall|Relative Avg. (%)|\n|---|---|---|---|---|---|---|\n|Vanilla|N/A|69.9|56.7|48.9|58.5|100|\n|DyToK (w/ CLIP)|25%|68.3|**56.9**|**49.1**|58.1|99.4|\n|DyToK|25%|**69.2**|56.8|48.9|**58.3**|**99.7**|\n|DyToK (w/ CLIP)|15%|64.6|53.9|48.4|55.6|95.1|\n|DyToK|15%|**66.3**|**55.0**|**48.7**|**56.7**|**97.0**|\n|DyToK (w/ CLIP)|10%|57.6|50.9|45.8|51.4|87.9|\n|DyToK|10%|**61.0**|**51.1**|**47.6**|**53.2**|**91.0**|\n\n[1] Dynamic-VLM: Simple Dynamic Visual Token Compression for VideoLLM\n\n[2] Adaptive Keyframe Sampling for Long Video Understanding"}]

Author rebuttal:

Meta-review: This paper proposes a training-free, LLM-guided dynamic token compression for long-video VLLMs by exploiting attention-based keyframe priors; it offers plug-and-play compatibility with existing pruners and yields strong efficiency-accuracy trade-offs. Strengths are the novel insight, thorough ablations, cross-architecture transferability, and real-hardware latency results. Reviewer HiJ7 is the only reviewer who gives a negative opinion, mainly focusing on the lack of comparison experiments. During the rebuttal, authors have given sufficient comparison with VisionZip and dycoke, and end-to-end latency on real hardware. During the following discussion, reviewer HiJ7 did not give any response, which reduces the confidence of his/her rating.  Based on the above reasons, this paper is recommended for acceptance.
Decision: Accept (poster)