FastVID: Dynamic Density Pruning for Fast Video Large Language Models
Authors: Leqi Shen
Guoqiang Gong
Tao He
Yifeng Zhang
pengzhang liu
Sicheng Zhao
Guiguang Ding
Venue: NeurIPS 2025 poster ()
Keywords: Video Large Language Models
Inference Acceleration
Primary area: deep_learning

Abstract: Video Large Language Models have demonstrated strong video understanding capabilities, yet their practical deployment is hindered by substantial inference costs caused by redundant video tokens. 
Existing pruning techniques fail to effectively exploit the spatiotemporal redundancy present in video data. 
To bridge this gap, we perform a systematic analysis of video redundancy from two perspectives: temporal context and visual context. 
Leveraging these insights, we propose Dynamic Density Pruning for Fast Video LLMs termed FastVID.
Specifically, FastVID dynamically partitions videos into temporally ordered segments to preserve temporal structure and applies a density-based token pruning strategy to maintain essential spatial and temporal information.
Our method significantly reduces computational overhead while maintaining temporal and visual integrity. 
Extensive evaluations show that FastVID achieves state-of-the-art performance across various short- and long-video benchmarks on leading Video LLMs, including LLaVA-OneVision, LLaVA-Video, Qwen2-VL, and Qwen2.5-VL.
Notably, on LLaVA-OneVision-7B, FastVID effectively prunes $\textbf{90.3\%}$ of video tokens, reduces FLOPs to $\textbf{8.3\%}$, and accelerates the LLM prefill stage by $\textbf{7.1}\times$, while maintaining $\textbf{98.0\%}$ of the original accuracy. 
The code is available at https://github.com/LunarShen/FastVID.

Fast Dynamic Density Pruning for Fast Video Large Language Models
Leqi Shen 122
Guoqiang Gong Tao He
Yifeng Zhang?
Pengzhang Liu
Sicheng Zhao Guiguang Ding!.21
School of Software, Tsinghua University
BNRist, Tsinghua University
3 JD.com
GRG Banking Equipment Co., Ltd.
Abstract
Video Large Language Models have demonstrated strong video understanding capabilities, yet their practical deployment is hindered by substantial inference costs caused by redundant video tokens. Existing pruning techniques fail to effectively exploit the spatiotemporal redundancy present in video data. To bridge this gap, we perform systematic analysis of video redundancy from two perspectives: temporal context and visual context. Leveraging these insights, we propose Dynamic Density Pruning for Fast Video I.LMs termed FastVIID. Specifically, FastViI dynamically partitions videos into temporally ordered segments preserve temporal structure and applies a density-based token pruning strategy to maintain essential spatial and temporal information. Our method significantly reduces computational overhead while maintaining temporal and visual integrity. Extensive evaluations show that FastVID achieves ffite-of-the---er of-the-art performance across various short- and long- video benchmarks on leading Video LLMs, including LLa One Vision, LLaVA- Notably,
Video, Qwen2-VL, and Qwen2.5- effectively prunes 90.3% % of video tokens, reduces FLOPS to 8.3 %, and accelerates the LLM prefill stage by 7.1x, while maintaining 98.0 % of the original accuracy. The code is available at httttss:: /ttp///gttttt//.///////eeeee.. www.//unarrSenn/aatt//.
Vision-7B. VIs
Introduction
Video Large Language Models (Video LLMs) 7 22
19
57 45 have shown strong performance in
video understanding but incur substantial inference costs. While several methods [38
28
48 [8] 37
60] explore training-time compression to mitigate this issue, they often require expensive retraining. In this work, we focus on an inference-time acceleration strategy that enhances efficiency without requiring additional training.
This computational burden is primarily caused by the high volume of video tokens, making effective token compression essential. While prior image compression methods 6 32
56
51
reduce frames. As
5
redundancy within single image, they fail exploit temporal dependencies result, spatiotemporal redundancy remain insufficiently explored. In this work, we systematically analyze video redundancy from two key perspectives: temporal context and visual context.
Temporal context is fundamental video understanding, as the order and continuity of frames directly influence semantic interpretation. As depicted in Figure a). disrupting frame order (shuffled) or omitting frames (incomplete) leads to incorrect comprehension, highlighting the necessity of pre- serving temporal structure. To achieve this, segment the video into temporally ordered segments, grouping highly similar consecutive frames. Pruming is applied within each high-redundancy segment but not across segments, preserving the essential temporal structure for accurate video understanding.
"Tunarshen@mmail.comm com, work done at JD.com. Project lead.
Corresponding authors.
39th Conference on Neural Information Processing Systems (NcurIPS 2025).

---

Q: How do the players celebrato scoring the first goal in this video?
Anchor Token
Original Image
Each token is assigned to its nearsst anchor token.
Ordered
A: High-fiving with the tearmmates.
Shuffled
x
A: Running on the field.
incomplete X A: Knooling on the field and screaming.
(b1) Uniform Sampling
(b2) Deasity-Based Sampling (ours)
Temporal Context
(b) Visual Context
Figure 1: Effective token compression in Video LLMs relies on preserving both temporal and visual context. (a) shows the impact of temporal disruptions, emphasizing the importance of temporal structure preservation. (b) show the effects of different token selection strategies for visual merging, where patches that share the same inner and border color are merged. Density- Sampling retains both distinctive and representative context while effectively reducing redundancy.
Given high-redundancy segments, we aim to preserve visual context by effectively consolidating spatial and temporal information within each segment. state-of-the-art method VisionZip (51) applies uniform token sampling followed by merging redundant tokens [5] shown in Figure lb1).
However, its token selection strategy is content- aggeont-agnottcc, potentially leading to the loss of important details, such the guitar body being incorrectly merged with the background. To address this, propose a density-baat -based token sampling in Figure Specifically, high-density tokens, surrounded by numerous similar tokens, serve candidates for selection. We select density peak tokens as anchors, which have a higher density than their neighbors and by a relatively large distance from tokens with higher densities (30) This strategy ensures that the selected tokens are both representative and distinctive, effectively preserving segment visual context while reducing redundancy.
Building upon these insights, propose Dynamic Density Pruning for Fast VIDeo LLMs mini- mizing spatiotemporal redundancy while preserving essential semantics. We begin with Dynamic Temporal Segmentation, which adaptively partitions videos into segments. Within each segment, introduce Density Spatiotemporal Pruning to retain both global visual context and salient details. FastVID significantly accelerates inference while preserving both temporal and visual integrity.
To evaluate the generalization capability of our approach, we evaluate on leading Video LLMs, LLaVA-One Vision [19, VALaVA-Vide [57], Qwen2-VL 45 and Qwen2.5-VV VL. 4 To further vali- date its effectiveness, we perform extensive experiments on MVBench 21.. Long VideoBench [46), MLVU (58) and VideoMME 212] These benchmarks cover wide range of video complexities and durations, ensuring a comprehensive evaluation. Notably, on LLaVA-One Visiooon--Viii--- effectively prunes 90.3% % of video tokens, reduces FLOPs to 8.3%, and accelerates the LLM prefill stage by 7.1x, while maintaining 98.0% % of the original accuracy across all benchmarks.
Vision- 7B, FastVID
The main contributions are summarized as follows: (1) We analyze Video LLM compression from both temporal and visual context perspectives, emphasizing the importance of maintaining temporal and visual integrity.
(2) We propose Fast novel pruning framework that employs Dynamic Temporal Segmentation to partition videos into temporally ordered segments and Density Spatiotemporal Pruning to retain global segment information and key details. (3) Our FastVID achieves state-of-the-art performance across diverse Video LLMs and benchmarks, while maintaining robust accuracy even under extreme compression.
2 Related Work
15 14], there has been growing interest in Video LLMs. These models can be categorized based on how they process video tokens: general Video L.MMs and Video LLMs with training-time compression.
Video LLMs. With the rapid advancement of LMMs I 42 and MLLMs 43 25 26 20
General Video L.MMs 24
45 directly process raw video tokens or apply pooling. leverages shared projection layers obtain unified visual representations, LLaVA-
Video-LLaVA 22

---

Vision 19 demonstrates strong video understanding through task transfer from images. LLaVA-
To better capture Video [57] creates high-quality synthetic dataset for video introduce additional designs for video the spatiotemporal structure of video, some models [57 45
positional information. I.L.VA Vided introduces newline tokens to distinguish spatial and temporal positions effectively.
37 60 aim significantly reduce the progressively clusters visual tokens and provides multi-scale features. LongVU [37 employs cross-modal query and inter- frame dependencies to adaptively reduce video redundancy. Apollo [60 explores scaling consistency and uses the Perceiver Resampler [17].
Video LLMs with training-time compression 38
28
48
18
number of video tokens, enabling longer video sequences. Chat-UniV [1]
However, general Video L.LMs remain the dominant paradigm, with LLaVA-One Vision and the VL. series being widely adopted due to its adaptability and superior performance. Therefore, we focus on inference-time acceleration for general Video L.LMs.
Token Compression. Token compression has emerged as an effective approach to reduce com-
putational complexity in transformer architectures, such as ViT [10] and CLIPP 29 ToMe
progressively merges fixed spatial tokens, while TempMe [55 extends this concept by merging neighboring clips to minimize temporal redundancy.
Recent studies 66
32 56 51
47 23 59 primarily focus on spatial token reduction to accelerate Im-
age LLMs. FastV selects text-relevant tokens at shallow layers of the LLM. VLLVVAA-uMMrre [2] uses attention scores from the [CLS] token to prune spatial redundancy. Sparse VLM [56] proposes token recycling strategy to aggregate and reconstruct tokens. VisionZip [ST] reduces visual redun- dancy in the vision encoders. However, these methods overlook the temporal relationships across frames.
Due to the high volume of video tokens in Video L.Mss, recent video compression methods 13
16 36
34 40 have gained increasing attention. FrameFusion [3] applies both merging and pruning across successive shallow LLM layers, but repeated pruning operations adversely affect overall efficiency. DyCoke [4] merges tokens across frames and applies dynamic KV cache reduction. However, its pruning during the prefill stage struggles to achieve substantial token reduction while maintaining accuracy. PruneVID [T6] clusters video tokens and selects those most relevant to query tokens, but this dependency on clustering introduces significant latency during compression. Despite these advances, efficient and accurate pruning under large token reduction remains unsolved.
3 Methodology
3.1 FIstVID
In this paper, we focus on preserving temporal and visual context at the prefill stage. By reduc- ing video tokens before LLM processing, our FastVID significantly enhances computational efficiency while facilitating easy deployment, in- cluding compatibility with FlashAttention [9]. KV cache, multi-ture turt conversations, and plug- and-play design for seamless integration into ex- isting Video LLMs. Our method achieves robust performance even under extreme compression rates, offering a practical solution for fast Video LLMs.
FastVID
Densly Spatitampoo
Vision Encodlee
Dynande DraamicTtaraan Terpara
Impee
Fromms
Projector
Pooling
Vidko Tilhers
Segnentation
Phunle
Query
Figure 2: The overview of our FastVID.
Figure presents an overview of FastVID. Given an input video, F frames are uniformly sampled (e.g., 32 in LLaVA-OneVision 64 in LLaVA- I.LVV--Vdeo), Each frame is individually processed by the vision encoder. The extracted tokens are projected and pooled into a video token sequence. With FastVID, we effectively eliminate redundant tokens while preserving critical information. Specifically. Dynamic Temporal Segmentation (DySeg) in Section 3.2 dynamically partitions video tokens into temporally ordered, high-redundancy segments, while Density Spatiotemporal Pruning (STPrune) in
htttps://github..com/Laaaaa
3

---

(a) Original Sampled Frumes
13
Yeur
(b) Fixed-Lnterva Segmentaaton
Your
(c) Cluster-bssed Segmentation
(d) Dynamic Temporal Segmeriation louss)
Figure Visualization of different segmentation methods on 12 sampled frames from a video example. (b) Fixed-interval Segmentation struggles to maintain high intra-segment similarity, leading to visually diverse frames within the same segment. (c) Cluster-based Segmentation disrupts temporal order, grouping frames from different time periods to the same segment. (d) Our DySeg adaptively partitions the video, preserving both temporal structure and high intra-segment similarity. Please refer to Table 8 for detailed quantitative comparison.
Section 3.3 performs density- bbased pruning within each segment, enhancing efficiency with minimal loss key information. The final retained video tokens, combined with query tokens, then fed into the LLM to generate the response.
3.2 Dynamic Temporal Segmentation
As shown in Figure b) and 3 c), there are two common static segmentation methods for sampled frame sequences. In Figure 3 (), Fixed-interall Segmentation partitions the sequence into segments of a fixed length, preserving temporal order but potentially grouping visually dissimilar frames. TigureBcc shows Cluster-based Segmentation, where frames are grouped into three clusters based on frame similarity. However, suffers from a predefined cluster number, leading to ineffective segmentation when video complexity varies. As result, the first segment contains similar objects but different scenes. Furthermore, clustering may disrupt the temporal order by ignoring critical temporal information, such as omitting key frames (e.g., the 9th frame) and incorrectly grouping frames from different time periods into the first segment.
To address the limitations of static methods, propose Dynamic Temporal Segmentation, simple yet effective method that adaptively refines segment boundaries according to video complexity. DySeg achieves both temporal structure and high intra-segment similarity, generating fewer partitions for simple scenes and finer ones for more complex scenes.
To enable effective pruning, DySeg induces high spatiotemporal redundancy within each segment by minimizing similarity between adjacent segments. Thus, we segment the video based on transition similarity between adjacent frames. Specifically, we utilize global frame features f to compute the cosine similarity:
t; (f,, f(41), ==
T
(1)
where 4 denotes the transition similarity between the i-th and (i + 1)-th frame. T denotes the F - transition similarities for F sampled frames. To achieve dynamic segmentation, we select transitions that satisfy the following conditions:
S arg min 1
T. - < T),
S S, S2.
(2)
where denotes the minimum number of segments and denotes the threshold for transition similarity,
S denotes the 1 most dissimilar frame transitions, while S, denotes transitions where similarity falls below a fixed threshold 7. Each transition in the union S marks a boundary between segments. In simple videos with minimal shot transitions, S, is often empty. In such cases, S₁ enables finer segmentation by distinguishing subtle temporal changes. For complex videos, S is typically subset of S2, where S, ensures that adjacent frames with similarity below 7 are assigned to different segments. In Figure our DySeg effectively segments videos by dynamically adjusting granularity in simple yet effective manner, outperforming Fixed-interval and Clustering- based methods.

---

Denniy Sased Samyling Similanil Sactaaillllleeeaaae. wew--lle Assigumest
oudho-onltic quregalioo
Merging
Density- hased Token M Kenging (ouns)
Figure 4: Comparison of our proposed DTM and Cluster-based Token Merging. (a) Cluster-based methods aggregate tokens within each cluster and concatenate them, leading information and disruption of the spatiotemporal structure in video data. (b) Our DTM updates anchor tokens while preserving their original positional information, maintaining structural coherence. Please refer to Table for a detailed quantitative comparison.
loss of positional
3.3 Density Spatiotemporal Pruning
After obtaining segments with highly similar frames, we introduce STPrunc to reduce redundant tokens. It consists of two key modules: Density-ban -based Token Merging (DTM) for segment visual context and Attention-bal based Token Selection (ATS) for salient details. For segment of P frames, we retain rPN tokens in total, where N the number of tokens per frame, and r denotes the retention ratio. These tokens are generated by DTM and ATS. Specifically, dr PN tokens are generated by DTM and (1 d)rPN tokens are generated by ATS, where d controls the token allocation between the two modules.
Density-based Token Merging. For segment visual context, first identify anchor tokens and to-be-merged tokens. Selecting anchor tokens from the entire segment would disrupt their spatial relationships, as anchor tokens corresponding to different components of the same object might be distributed across multiple frames. To mitigate this, we restrict anchor token selection to specific frames. Specifically, we sample anchor frames at fixed interval p and select anchor tokens from these frames. The remaining tokens in the segment are treated as to-be-merged tokens. Specifically, the number of anchor frames [P/p]. From each anchor frame, dr drPN//[/p tokens are selected as anchor tokens. Please see Figure 5 for visualization of DTM applied to video segments.
To further clarify DTM, Figure[4t illustrates DTM when the segment length 1 and shows that it consists of three key steps. First, we follow density peaks clustering algorithms [30. to compute the density score. For each token in the anchor frame [1i, U2, uN.. calculate its local density ps and its distance to the closest higher-density token ò;, obtaining the final density score Pt X 5g:
Pi exp(-
1
Σ
d(us, 0;))),
(3)
min 3:p5>9;
d(v,,
if if s.t, Pj > pi
6:
(4)
max d( my), otherwise
where d(v;, Uz) denotes the Euclidean distance. Density peak tokens with high p x 6; serve as anchor tokens, indicating that they are surrounded by neighbors with lower local density while remaining relatively distant from other high-density tokens. This selection ensures that anchor tokens are both representative and distinctive. Next, for each anchor frame. Similarity-based Assignment assigns each token in the segment to the nearest anchor token using cosine similarity. Finally, we apply Anchor-centric Aggregation merge the assigned tokens into their respective anchors, preserving key visual structures through representative tokens. For an anchor and its associated tokens [b,,... b,). the updated is computed as: a = 3a
3
bis
(5)
where controls the balance between the anchor token and its associated tokens. In Figure 4 we compare our DTM with Cluster-based Token Merging. In L.MMs, RoPE (39 encodes relative positional relationships between tokens, making positional information essential for maintaining the spatiotemporal structure of video tokens. While prior methods [16] rely Cluster-based Token
5

---

Table Comparison of state-of- thethe-art methods on VLaVA-One [9] The A%/B% retention
ratio indicates that A% of the LLM input tokens are retained, and subsequently compressed to B% during the LLM forward pass. The best performance among those with similar retention ratios highlighted in bold. TFLOP related to video tokens are reported (see Appendix A for details).
Method
Retension
Rato R
TELOPs
MVBench
LongVido Bench
MLVU
Cherall I-600mi
Short
VidroMME
Mediam
Lang 30-60min
Score
Avg. Acc.
%
Duration
166
I--00min
1200mm
1-3min
3--30mm
Vanilla
100%
48.82
$6.9
56.4
65.2
58.6
70.3
56.6
48.8
59.3
100
Eveer2s
PautV ( 1chcc
61.3
6800
VisionZip
51 ewriss
25%
10.25
53.7
51.2
58.5
54,1
61.6
53.4
47.2
54.4
91.7
VisionIZip"
25%
10.73
56.6
55.7
64.8
58,0
68.5
57.7
47.7
58.8
99.1
DyCoke 41
25%
10.75
49.5
48 1
55.8
51.0
61.1
48.6
43.2
51.1
86.2
Fast ID
25%
10.75
56.5
56.3
61.1
58.0
69.9
47.7
58.7
99.0
FastV [6 01
36.6
SGLT
Visice/Pip
19.9%
8.46
53.0
50.0
57.1
60.8
90.0
Vision?lip" [S] levaros
19.9%
8.46
55.8
55.4
64.2
58,0
68.6
57.0
48.3
58.4
98.5
FDstVD
19.9%
8.46
56.3
$7.1
63.9
57.9
69.3
$6.7
47.7
58.8
99.1
FasiV 00
100%6115%
9.35
53.2
34.9
59.3
65.1
43.7
92.99
6.23
50.3
46-9
54.4
VisionZig" CVTR225
55.5
FD
14.8%
6.23
56.0
56.2
63.2
$7.7
69.3
56.2
47.4
58.3
98.3
PaiVV [5 24
7.36
51.7
S21
57.7
$2.4
S1.4
45.0
31.5
M112
VisionZip [1]
9.7%
4,04
44.4
43.5
51.5
46.0
50.4
45.8
41.8
46.4
78.3
VisionZip"
4.04
51.7
48.3
59.7
46.9
89.6
PraneVily Mc122
4.23
54.2
95.4
4.04
56.3
67.4
98.0
PrunVVID
[6
Actas
10.152235%
2.58
54.1
51.8
623
35.3
67.1
51.8
47.7
55.9
94.3
V
0.7933665
2.47
55.0
53.6
61.9
56.3
66.1
54.8
48.1
56.7
95.6
Merging, they discard positional information, causing RoPE to struggle with encoding spatiotemporal structure. In contrast, our DTM maintains the positional information of merged tokens, enhancing visual context understanding.
Overall, our DTM offers three key advantages: (1) Density-based Sampling selects density peak tokens as anchors, ensuring representative visual context. (2) We maintain the original positional information of updated anchor tokens, preserving spatiotemporal structure. (3) Anchor-centric Aggregation emphasizes representative tokens, enhancing feature representation.
Attention-based Token Selection. In addition to segment visual context obtained through DTM, we introduce ATS to capture salient visual details.
Motivated by previous studies [32 51 44 55). we utilize [CLS] attention scores to identify salient visual information. However, Video LLMs, which commonly use SigLIP (53) as their vision en- coder, [CLS] attention scores cannot be obtained. This is because Video LLMs utilize the penultimate layer of LigLIP, omitting the SigLIP head where the [CLS] token is generated. To overcome this, we reintegrate a pretrained SigLIP head into the vision encoder, allowing the model to compute [CLS] attention scores. Since the SigLIP head is lightweight (15.2M parameters), this modification incurs minimal computational overhead compared to the full Video L.M.M
Specifically, extract the [CLS] attention score from the pretrained SigLIP head, A R where and W are the spatial dimensions of frame tokens. Since Video LLMs incorporate pooling (see Figure 2 we apply the same operation to A, resulting in A E IR #W where // and W are the pooled spatial dimensions. Finally, for each frame, we select the top (1 d)rN tokens with the highest attention scores to preserve critical visual details.
4 Experiments
4.1 Experimental Settings
Benchmarks. We evaluate method on several widely used video understanding benchmarks: MVBench [I] Long VideoBench [46), MLVU [88, and VideoMME (wo sub.) [12]. Specifically, VideoMME is officially divided into short, medium, and long subsets. These benchmarks contain videos of varying durations and complex scenarios, providing a comprehensive evaluation of our method's effectiveness and generalization.
Implementation Details.
We apply our method leading Video L.Mss: L.L. VA-OoeVision [T9], LLa LLaVA-Video [57]. Qwen2-VL [45] and Qwen2.5-VL 4.. Unless otherwise specified, we adopt

---

Table 2: Comparison of state-of-the methods on LLaVA-Video [57 TFLOPs" calculations
include both video and newline tokens.
Method
Retention Ratio R
New line
TFLOPs
MVBench
LongVidoo Bench
MLVU
VideoMME
Avg.
Acc.
Tokens M
Overall
Shoet
Long
Score
Vanilla
100%
832
103.2
60.4
59.6
70.3
64.1
76.9
53.4
63.6
100
DyCoke
Jeveerss
32.1%
256
27.1
59.3
57.9
65.7
61.6
74.6
51.1
61.1
96.1
832/568.7
58.0
VisionZáp
ST oopees
24.9%
64
19.5
56.4
54.1
62.1
58.6
66.3
51.2
57.8
90.9
VisionZip"
[31
24.9%
64
19.5
58.3
58.3
66.6
61.9
73.3
52.2
61.5
96.7
DyCake
53.0
56.1
FastY1D
24.9%
64
19.5
59.3
$8.3
67.7
62.6
74.9
52.0
62.0
97.5
FastVID
24.9%
715.5
24.5
39.9
$7.4
63.6
74.9
53.7
62.4
98.1
V (6 soevits
10W95/10%
832/311.8
16.2
35.8
35.4
58.9
57.9
67.6
48.6
$7.0
89.6
Visioon/ip
5D PVRRus
9.5%
64
7.3
46.3
46.6
52.2
49.5
54.2
44.3
48.7
76.6
VisionZip"
[11 covress
56.6
53.6
58.7
FastVDD
9.5%
7.3
58.3
56.2
63.9
59.6
70.9
50.7
99.5
93.6
FastYID
9.5%
508.2
10.5
58,5
86.5
64,9
60.7
71.7
51.2
60.2
94.7
Table 3: Comparison of state-of-the-art methods on Qwen2-VL. 45
We set the maximum number of video tokens fed into the LLM to 16384, and the maximum number sampled frames 768. FastV performs pruning based on LLM attention scores, but its cager-attention implementation materializes the full attention matrix in memory, leading OOM errors due the large number of video tokens.
VideoMME
Method
Token
TFLOPs
Short
Medium
Long
Overall
Vanilla
13447.1
100%
124.0
100%
74.1
60.4
54.3
63.0
100%
FastV [6 2CCV124
13447.1/336..8
100%/25%
31.3
25.3%
Out of Memory
VisionZip" [1 22ee
3349.3
24.9%
24.1
19.4%
70.9
56.3
48.3
58.5
92.9%
VIuD
[6 AC1:25
3460.6
25.7%
25.0
20.1%
66.7
54.0
48.1
56.3
89.4%
FastVID
3349.3
24.9%
24.1
19.4%
72.7
58.3
50.7
60.6
96.2%
the hyperparameter setting e = 8,r = 0.9, = 0.4, 4,3 0.6 for all experiments. For LLaVA-One 32 sampled frames generate with E {25%, 20%, 15%. 10%). For LLa LA Video,, 64 sampled frames generate a 64 x 169 token input. We experiment with r E (25%, 10%). For the Qwen- VI. series, which samples up to 768 frames, we discard highly redundant frames and set T to its optimal value. We conduct all evaluations using LvMs-Eval [54 on A100 GPUs
x 196 token input to the LLM. We experiment
Compared Baselines. (1) For image compression, we adopt both the widely used classic method FastV [6] and the current state-of-the-art method VisionZip [5]]. VisionZip prunes tokens in the last layer of the MLLM's ViT, conflicting with pooling in Video LLMs and degrading performance. To address this, we implement VisionZip', which applies pruning after pooling.
(2) For video
compression, we compare two recent methods, DyCoke 44 and PruneVID [16]. Our FastVID focuses on pruning during the prefill stage. To ensure fairness, we reimplement these baselines without pruning in the decode stage. VID applies two-stage pruning both input tokens and the LLM's 10th layer. Prune VID* refers to the variant that prunes only input tokens. More implementation details of these baselines are provided in Appendix
4.2 Comparisons with State-of-the-Art Methods
For comprehensive evaluation, we compare our FastVID with state-of-the-at methods on bench- marks with diverse video durations. We perform evaluations across different retention ratios Rt to systematically assess model performance. In the case of DyCoke, video frames are evenly divided into 4-frame segments, retaining all tokens from the first frame while pruning the rest. Consequently, its lowest retention ratio R is 25%.
Results on LLaVA-OneVisian LLaVA-One Vision. While FastV and VisionZip perform well R 25%, but their performance declines sharply as R decreases. This indicates the limitations of spatial compression alone, which struggles to preserve essential temporal information under extreme pruning. Notably, even after pruning 90.3% of the tokens, our FastVID preserves 98.0 % of the vanilla model's performance. To compare with Prun VID, we additionally apply FastV at the 10th layer of the L.M.. Fast FastVIDFFastV achieves 95.6 % of the original accuracy at 2.47 (5.1 %) TFLOPs.
In Table I we evaluate our FIstVID against other methods on
7

---

Table Comparison of state-of ohe-----tttttttt methods on Qwen2.5-VL. 4
We set the maximum number
of video tokens fed into the LLM to 16384. and the maximum number of sampled frames to 768.
VideoMME
Method
Token
TFLOPs
Short
Medium
Long
Overall
Vanilla
13447.1
100%
124.0
100%
76.7
68.2
56.9
67.3
100%
VID [66 16L22
3294.5
24.5%
23.7
19.1%
67.0
59.4
51.6
59.3
88.1%
VID
3240.7
24.1%
23.2
18.7%
74.3
61.3
52.8
62.8
93.3%
Table Efficiency Comparison on LLaVA-One Vision
19
The prefill time, defined as the latency
to the first generated token, is measured on VideoMME using an A100 GPU.
Prefill Time (ms)
Method
# Token
TFLOPs
Arg. Acc.
Segmentation
Compression
LLM Forward
Total
Vanilla
6272 (100%)
48.82 (100%)
476.3
59.3 (1.0g)
PruneVID> [66
635.6 (10.1%
4.23 (8.7%)
5.2
32.0
64.3
101.3 (4.7%)
(95
FastVID
608 (9.7%)
4.04 (8.3%)
5.6
611
67.2 (7.1x)
58.1 (98.0%)
Results on LLa LLaVA-Video adopts a unique design by inserting newline tokens after each height-wise position in every frame, producing 64 x 13 x (13 + 1) tokens. In Table [ for all methods, when the positional information of retained tokens is preserved, we also retain the associated newline tokens. For a fair comparison with VisionZip, evaluate FastVID*, where only one newline token retained per frame. Notably, our FastVID consistently outperforms all baselines across various retention ratios.
Results on Qwen2-VL..
Unlike LLaVA-One [9] and LLaVA-Vided
Qwen2-VL LLL 45
employs distinct architecture that samples 768 frames per video and processes them using 3D convolutions. It introduces M-RoPE, which decomposes rotary embeddings into temporal, height, and width components. The model dynamically adjusts the resolution of each frame. In Table 3 our FIstVID outperforms other baselines on Qwen2-VL. VL under similar model complexity. particular, FastVID achieves 2.4 accuracy improvement on the long subset.
Results on QwL To further demonstrate its generalization ability, we report results on Qwen2.5-VL VL 4 in Table E Unlike the LLaVA series and Qwen2-VL, Qwen2.5-VL employs vision encoder with window attention and Qwen2.5 LLM [50], representing substantially different architecture. Compared to the recent SOTA VIuneVID, FastVID achieves a 3.5 improvement under comparable retention ratio. shown Tables
1 2
and our FastVID's consistent superiority
across these distinct Video LLM architectures further validates its strong generalizability and practical effectiveness.
Efficiency Comparison. Table 5 compares the efficiency of our method against the state-of wi-thh--r art video compression method Prune VID. Latency and accuracy are evaluated while maintaining similar model complexity. Pru relies heavily on time-consuming clustering algorithms during both video segmentation and compression. In contrast, FastVID leverages efficient transition similarity to achieve efficient segmentation. Although density score computation (see Eq. consuming, we restrict this step to anchor frames and parallelize its execution, thereby accelerating the compression process. As result, FastVID achieves a 7.1% x speedup while preserving 98.0 % accuracy. Further comparisons with VID are provided Appendix C
3 is time-
4.3 Ablation Study
By default, we conduct ablation studies on LLaVA-OneVision r 10%. Further hyperparameter analysis is provided in Appendix
Ablation study on DySeg. To compare different video segmentation methods, we present qual- itative and quantitative results in Figure 3 and Table 8 respectively. In Table g Fixed-interval Segmentation with an interval of 4 generates 32/4 segments, while Cluster-based Seg- mentation generates 8 clusters, ensuring consis-
Table 8: Ablation study on video segmentation.
Segmentation
MVBench
LongVideo- Bench
MLVU
VideoMME
Avg. Acc.
Hixe--intrval Cliste--bsed Our DySeg
53.6
55.4
95.3%
91.9%
98.0%
8

---

Table Ablation Study on in STPrune. Table 7: Ablation study on different token merging strategies. Figure lbb)) presents Uniform. Fig- ure a) presents Cluster-based.
STPrune consists of DTM and ATS, A fraction of the retained tokens is from DTM, while the remaining d is from ATS.
Taken Merging
MVBench
LongVido-- Bench
MLVU
VideoMME
Acc.
Avg.
in STPrume
MVHench
Long Vidzo- VingV Bench
MLVU
VideoMME
Avg. Acc.
Caiform Closee-hused
55.0
ILaVAOOneislon
$4.7
62.0
56.4
96.2%%
QNATS
15.3
15.3
62.4
55.0
96,25
55.6
58.2
62.4
57.3
97.25
55.0
53.6
62.1
56.1
95.7%
Our DTM
55.9
56.3
62.7
57.3
98.0%
62.7
57.3
98.0%
Video
54.5
62.0
Caiform
91.5%
35.6
96.79
Cluster based
55.3
55.]
62.9
60.1
91.8%
1.D
S4.1
50.1
61.5
55.9
93.3%
Our I7M
58.5
$6.5
64.9
60.7
94.7%
Table 9: Ablation study on the SigLIP head in ATS.
Table 10: Improved length extrapolation via Fast
Method
Frame
# Token
VideoMME
Method
VideoMME
Vanilla
32
6272
58.6
100%
FastVID w/o the SigLIP head
$7.3
FastVID (r=25%)
128
6272
60.4
103.1%
56.3
FastVID (r=10%)
320
6080
61.4
104.8%
lency with the minimal segment number è in our DySeg. Fixed-interval preserves temporal order between segments, whereas Cluster-based maintains intra-segment similarity. The results show that Fixed-interval outperforms Cluster-based by 1.4%, highlighting the importance of temporal structure preservation in Video LLM pruning. Additionally, high intra-segment similarity is also essential for effective pruning. Our proposed DySeg successfully integrates both ad advantages while enabling dynamic pruming, achieving a superior average performance of 98.0%.
Ablation study on STPrune. STPrune is proposed to prune tokens within each segment and consists of two key components: DTM and ATS. DTM employs density-based sampling to merge redundant tokens, preserving segment visual context. ATS leverages [CLS] attention scores to highlight important visual details. T presents an ablation study on d, which controls the token distribution between DTM and ATS. The results show that ATS alone (when d 0.0) outperforms DTM alone (when d = 1.0). However, the best performance is achieved at d = 0.4, demonstrating that balanced integration of both modules is crucial for preserving essential video information. This integration boosts performance by 1.8% % and 4.5% points over ATS and DTM alone, respectively.
Ablation study on DTM. In Table 7 we compare different token merging strategies for segment visual context. Uniform, used in VisionZip. selects anchor tokens in a content-agnoste manner, which limits its ability distinguish semantically meaningful objects. Cluster-based discards essential positional information, resulting in inferior performance, particularly in LLaVA- By leveraging Density-basee -based Sampling and Anchor-centric Aggregation, our DTM achieves superior results, especially in LLaVA-Vidae Video, where DTM outperforms other methods by 2.9% %.
Ablation study on the SigLIP head ATS. We reintroduce the original, un-finetuned SigLIP head to generate attention scores in ATS. This creates potential parameter gap between the SigLIP encoder and the original SigLIP head. However, we believe this gap has limited impact for the following reasons.
First, the SigLIP head is pretrained jointly with the encoder on large-scale vision-language data, enabling strong generalization and transferability. Second, since the first-stage training of Video LLMs also aims to align visual and textual representations, the SigLIP head is naturally compatible with this alignment objective. Finally, shown in Table results show that using the SigLIP head leads to a 1.0 improvement, indicating its effectiieness. This likely because the SigLIP head is explicitly trained to aggregate patch tokens for semantic alignment, making its attention scores better suited for identifying semantic salient tokens.
In addition, we emphasize that our VIs generalizes to vision encoders without pretrained heads or [CLS] tokens (e.g., the Qwen-VL. series). In such cases, [CLS] attention scores are computed at the Video LLM''s ViT final layer using pseudo [CLS] tokens derived by averaging patch tokens. Our VIs achieves strong performance on Qwen2-VI. (Table 3) and Qwen2.5-VL (Table 4. significantly outperforming previous methods.

---

Input
DTM
Retained Tokens
(a)
DIMon
Input
DTM
DTM
Retained Tokens
(b)
DTM
Figure 5: Visualization VastVID with 0 0.9, 0.4.p 2. We retain a total of 40 tokens, including 16 tokens in DTM (highlighted in red) and 24 in ATS (highlighted in blue). In DTM, patches that share the same inner and border color are merged.
Improved length extrapolation. In Table TO by applying FastVID with different retention ratios, extend the number of sampled frames from 32 to 320 while maintaining a comparable total number of input video tokens. This consistently improves performance, which demonstrates that FIstVID only compresses video tokens efficiently but also enables effective length extrapolation.
4.4 Qualitative Results
Figure visualizes the proposed DySeg and STPrune. In Figure 5 (a), all frames grouped into single segment, with the 1st and 3rd frames selected as anchor frames, where DTM is applied. In Figure 5(b), the frames are divided into two segments, with the 1st, 3rd, and 4th frames selected as anchor frames. Blue tokens represent those selected by ATS due to high [CLS] attention. These tokens readily cluster together and do not always correspond object regions, suggesting that while they reflect salient [CLS] information, they lack broader visual context. In contrast, DTM's 's red tokens effectively aggregate visually similar content across frames, significantly reducing spatiotemporal redundancy while preserving visual context. Together, ATS and DTM offer complementary benefits for effective video compression. More qualitative results are provided in Appendix B
Conclusion
In this paper, we introduced FastVID, novel inference-time pruning framework designed to accel- erate Video LLMs by effectively reducing spatiotemporal redundancy. Through a comprehensive analysis of video tokens from both temporal context and visual context, FastVID dynamically par- titions videos into temporally ordered segments and employs density-based token pruning within each segment. Extensive experiments across multiple Video LLMs and benchmarks demonstrate its generalization ability and effectiveness. Crucially, FastVID maintains high performance even under extreme compression rates, enabling practical deployment of fast Video LLMss
Acknowledgments
This work was supported by National Natural Science Foundation of China (Nos. 62525103, 62571294), Beijing Natural Science Foundation (No. 1.252009), CCF-DiDi GAIA Collaborative Research Funds, and the Postdoctoral Science Foundation of China (No. 2024M750565).
10

---

References
[1] Josh Achiam, Steven Adler, Sandhini Agarwal, Lama Ahmad, Ilge Akkaya, Florencia Leoni Aleman, Diogo Almeida, Janko Altenschmidr, Sam Altman, Shyamal Anadkat, et al. Gpt-4 technical report. arXiv preprint arXiv:2303.08774, 2023.
[2] Joshua Ainslie, James Lee-Thorp. Michiel De Jong, Yury Zemlyanskiy, Federico Lebrón, and Sumit Sanghai. Gqa: Training generalized mueti-query transformer models from multi- heati-hedd checkpoints. arXiv preprint arXiv:2305. 13245, 2023.
[3] Jean-Baptiste Alayrac, Jeff Donahue, Pauline Luc, Antoine Miech, lain Barr, Yana Hasson, Karel Lenc, Arthur Mensch, Katherine Millican, Malcolm Reynolds, et al. Flamingo: a visual language model for few-shot learning. Advances in Neural Information Processing Systems, 2022.
[4] Shuai Bai, Keqin Chen, Xuejing Liu, Jialin Wang, Wenbin Ge, Sibo Song, Kai Dang, Peng Wang. Shijie Wang, Jun Tang, al. Qwen2. 5-vl technical report. arXiv preprint arXiv:2502.13922, 2025.
[5] Daniel Bolya, Cheng- Yang Fu, Xiaoliang Dai, Peizhao Zhang, Christoph Feichtenhofer, and Judy Hoffman. Token merging: Your vit but faster. In Proceedings of the International Conference on Learning Representations, 2023.
[6] Liang Chen, Haozhe Zhao, Tianyu Liu, Shuai Bai, Junyang Lin, Chang Zhou, and Baobao Chang. An image is worth 1/2 tokens after layer 2: Plug-and-play inference acceleration for large vision-language models. In Proceedings of the European Conference on Computer Vision, 2024.
[7] Zhe Chen, Weiyun Wang, Yue Cao, Yangzhou Liu, Zhangwei Gao, Erfei Cui, Jinguo Zhu, Shen- glong Ye, Hao Tian, Zhaoyang Liu, al. Expanding performance boundaries of open- multimodal models with model, data, and test-time scaling. arXiv preprint 4rXiv:2412.05271, 2024.
[8] Wei-Lin Chiang, Zhuohan Li, Zi Lin, Ying Sheng, Zhanghao Wu, Hao Zhang, Lianmin Zheng, Siyuan Zhuang, Yonghao Zhuang, Joseph E Gonzalez, et al. Vicuna: An open-source chatbot impressing gpt-4 with 90%* * chatgpt quality, 2023. BtLLhhttts///vcuumt Imsys. org
[9] Tri Dao, Dan Fu, Stefano Ermon, Atri Rudra, and Christopher Ré. Flashattention: Fast and memory-efficieee exact attention with io-awareness. Advances in Neural Information Processing Systems, 2022.
[10] Alexey Dosovitskiy, Lucas Beyer, Alexander Kolesnikov, Dirk Weissenborn, Xiaohua Zhai, Thomas Unterthiner, Mostafa Dehghani. Matthias Minderer, Georg Heigold, Sylvain Gelly, et al. An image is worth 16 words: Transformers for image recognition scale. arXiv preprint 2020.
arXiv:2010.1 19999 1929,
[11] Mingjing Du Shifei Ding, and Hongjie Jia. Study on density peaks clustering based on k-nearest neighbors and principal component analysis. Knowledge-Bald BBased Systems, 99:135-145 145, 2016.
[12) Chaoyou Fu, Yuhan Dai, Yongdong Luo, Lei Li, Shuhuai Ren, Renrui Zhang, Zihan Wang, Chenyu Zhou, Yunhang Shen, Mengdan Zhang, et al. Video-mme: The first-ever comprehensive evaluation benchmark of muulti-moda Iims in video analysis. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, 2025.
[13] Tianyu Fu, Tengxuan Liu, Qinghao Han, Guohao Dai, Shengen Yan, Huazhong Yang, Xuefei Ning, and Yu Wang. Framefusion: Combining similarity and importance for video token reduction on large visual language models. arXiv preprint arXiv:2501.0/90 0009090........... 2024.
[14] Zhangwei Gao, Zhe Chen, Erfei Cui, Yiming Ren, Weiyun Wang, Jinguo Zhu, Hao Tian, Shenglong Ye, Junjun He, Xizhou Zhu, et al. Mini-internvl: a flexible-transfer pocket multi- modal model with 5% parameters and 90% performance. Visual Intelligence, 2(1):32, 2024.

---

[15] Wei Huang, Xingyu Zheng, Xudong Ma. Haotong Qin, Chengtao Lv, Hong Chen, Jie Luo, Xiaojuan Qi, Xianglong Liu, and Michele Magno. An empirical study of Ilama3 quantization: From llms to milms. Visual Intelligence, 2(1):36, 2024.
[16] Xiaohu Huang, Hao Zhou, and Kai Han. Prunevid: Visual token pruning for efficient video large language models. arXiv preprint arXiv:2412. 16117, 2024.
[17] Andrew Jaegle, Felix Gimeno, Andy Brock, Oriol Vinyals, Andrew Zisserman, and Joao Car- reira. Perceiver: General perception with iterative attention. In Proceedings of the International Conference on Machine Learning, 2021.
[18] Peng Jin, Ryuichi Takanobu, Wancai Zhang, Xiaochun Cao, and Li Yuan. Chat-univi: Unified visual representation empowers large language models with image and video understanding. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, 2024.
[19] Bo Li, Yuanhan Zhang, Dong Guo, Renrui Zhang, Feng Li. Hao Zhang, Kaichen Zhang, Peiyuan Zhang, Yanwei Li, Ziwei Liu, et al. Llava-onevision: Easy visual task transfer. arXiv preprint arXiv:2408. 03326, 2024.
[20] Junnan Li, Dongxu Li, Silvio Savarese, and Steven Hoi. Blip-2: Bootstrapping language-image pre-training with frozen image encoders and large language models. In Proceedings of the International Conference on Machine Learning, 2023.
[21] Kunchang Li, Yali Wang, Yinan He, Yizhuo Li, Yi Wang, Yi Liu, Zun Wang. Jilan Xu, Guo Chen, Ping Luo, et al. Mybench: A comprehensive multi-modal video understanding benchmark. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 22195-222066 2024.
[22] Bin Lin, Yang Ye, Bin Zhu, Jiaxi Cui, Munan Ning, Peng Jin, and Li Yuan. Video-llava: Learning united visual representation by alignment before projection. arXiv preprint arXiv:23 1. 2023.
[23] Zhihang Lin, Mingbao Lin, Luxi Lin, and Rongrong Ji. Boosting multimodal large language models with visual tokens withdrawal for rapid inference. In Proceedings of the AAAI Confer- ence on Artificial Intelligence, 2025.
[24] Aixin Liu, Bei Feng, Bin Wang, Bingxuan Wang, Bo Liu, Chenggang Zhao, Chengqi Dengr, Chong Ruan, Damai Dai, Daya Guo, et al. Deepseek-v2: A strong, economical, and efficient mixture-of-experts language model. arXiv preprint arXiv:2405.. 240.04400 2024.
[25] Haotian Liu, Chunyuan Li, Yuheng Li, and Yong Jae Lee. Improved baselines with visual instruction tuning. In Proceedings of the IEEE Conference on Computer Vision and Puttern Recognition, 2024.
[26] Haotian Liu, Chunyuan Li, Yuheng Li, Bo Li, Yuanhan Zhang, Sheng Shen, and Yong Jae Lee. Llava-next: Improved reasoning, ocr, and world knowledge, January 2024. URL https
7 //1lava-vl, wiw.vaawllgggtuubb
[27] Haotian Liu, Chunyuan Li, Qingyang Wu, and Yong Jae Lee. Visual instruction tuning. Advances in Neural Information Processing Systems, 2024.
[28] Muhammad Maaz, Hanoona Rasheed, Salman Khan, and Fahad Shahbaz Khan. Video-chatgpt: Towards detailed video understanding via large vision and language models. In Association for Compulational Linguistics, 2024.
[29] Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, et al. Learning transferable visual models from natural language supervision. In Proceedings of the International Conference on Machine Learning, 2021.
[30] Alex Rodriguez and Alessandro Laio. Clustering by fast search and find of density peaks. science, (44(4441)1111142221444 2014.
12

---

[31] Amir Shahroudy, Jun Liu, Tian-Tsong Ng. and Gang Wang. Ntu rgb+ d: A large scale dataset for 3d human activity analysis. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, 2016.
[32] Yuzhang Shang, Mu Cai, Bingxin Xu, Yong Jae Lee, and Yan Yan. prameeer Adaptive token reduction for efficient large multimodal models. arXiv preprini arXiv:2403. 15338, 2024.
[33] Noam Shazeer. Glu variants improve transformer. arXiv preprint arXiv:2002.05202, 2020.
[34] Leqi Shen, Guoqiang Gong, Tianxiang Hao, Tao He, Yifeng Zhang, Pengzhang Liu, Sicheng Zhao, Jungong Han, and Guiguang Ding. Discovla: Discrepancy reduction vision, lan- guage, and alignment for parameter-efficien video-text retrieval. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, 2025.
[35] Leqi Shen, Tianxiang Hao. Sicheng Zhao, Yifeng Zhang, Pengzhang Liu, Yongjun Bao, and Guiguang Ding. Tempme: Video temporal token merging for efficient text-video retrieval, In Proceedings of the International Conference on Learning Representations, 2025.
[36] Leqi Shen, Tao He, Guoqiang Gong, Fan Yang, Yifeng Zhang, Pengzhang Liu, Sicheng Zhao, and Guiguang Ding. Llava-mlb: Mitigating and leveraging attention bias for training-five free video Ilms. arXiv preprint arXiv:2503. /205, 2025.
[37] Xiaoqian Shen, Yunyang Xiong, Changsheng Zhao, Lemeng Wu, Jun Chen, Chenchen Zhu, Zechun Liu, Fanyi Xiao, Balakrishnan Varadarajan, Florian Bordes, al. Longvu: Spatiotem- poral adaptive compression for long video-language understanding. In Proceedings of the International Conference on Machine Learning, 2025.
[38] Enxin Song, Wenhao Chai, Guanhong Wang, Yucheng Zhang, Haoyang Zhou, Feiyang Wu, Haozhe Chi, Xun Guo, Tian Ye, Yanting Zhang, al. Moviechat: From dense token to sparse memory for long video understanding. In Proceedings of the IEEE Conference on Computer Vision and Puttern Recognition, 2024.
[39] Jianlin Su, Murtadha Ahmed, Yu Lu, Shengfeng Pan, Wen Bo, and Yunfeng Liu. Roformer: Enhanced transformer with rotary position embedding. Neurocomputing, 568:127063. 2024.
[40] Fengyuan Sun, Leqi Shen, Hui Chen, Sicheng Zhao, Jungong Han, and Guiguang Ding. Adatp: Attention-debiased token pruning for video large language models. arXiv preprint 25Xiv:2505.20100 2025.
[41] Keda Tao, Can Qin, Haoxuan You, Yang Sui, and Huan Wang. Dycoke: Dynamic compression of tokens for fast video large language models. In Proceedings of the IEEE Conference Computer Vision and Pattern Recognition, 2025.
[42] Rohan Taori, Ishaan Gulrajani, Tianyi Zhang, Yann Dubois, Xuechen Li, Carlos Guestrin, Percy Liang, and Tatsunori B Hashimoto. Stanford alpaca: An instruction-following Ilama model, 2023.
[43] Gemini Team, Rohan Anil, Sebastian Borgeaud, Jean- Baan-Bapistte Alayrac, Jiahui Yu, Radu Soricut, Johan Schalkwyk, Andrew M Dai, Anja Hauth, Katie Millican, et al. Gemini: family of highly capable multimodal models. arXiv preprint arXiv:2312. / 2023.
[44] Ao Wang, Fengyuan Sun, Hui Chen, Zijia Lin, Jungong Han, and Guiguang Ding, [cls] token tells everything needed for training efficient milms. arXiv preprint 4rX 2024.
[45] Peng Wang, Shuai Bai, Sinan Tan, Shijie Wang, Zhihao Fan, Jinze Bai, Keqin Chen, Xuejing Liu, Jialin Wang, Wenbin Ge, et al. Qwen2-vl: Enhancing vasion-language model's perception of the world at any resolution. arXiv preprint arXiv:2409. 12191, 2024.
[46] Haoning Wu, Dongxu Li, Bei Chen, and Junnan Li. Longvideobench: A benchmark for long- context interleaved video-language understanding. Advances in Neural Information Processing Systems, 37:28828-28888 288577, 2025.
13

---

[47] Long Xing, Qidong Huang, Xiaoyi Dong, Jiajie Lu, Pan Zhang, Yuhang Zang, Yuhang Cao, Conghui He, Jiaqi Wang, Feng Wu, et al. Pyramiddrop: Accelerating your large lasounlanguage models via pyramid visual redundancy reduction. In Proceedings of the IEFF Conference on Computer Vision and Puttern Recognition, 2025.
[48] Lin Xu, Yilin Zhao, Daquan Zhou, Zhijie Lin, See Kiong Ng, and Jiashi Feng. Pllava: Parameter-free free llava extension from images to videos for video dense captioning. arXiv preprint arXiv:2404.1699 16994, 2024.
[49] An Yang, Baosong Yang, Binyuan Hui, Bo Zheng, Bowen Yu, Chang Zhou, Chengpeng Li, Chengyuan Li, Dayiheng Liu, Fei Huang, al. Qwen2 technical report. arXiv preprini arXiv:2407.100 10671, 2024.
[50] An Yang, Baosong Yang, Beichen Zhang, Binyuan Hui, Bo Zheng, Bowen Yu, Chengyuan Li, Dayiheng Liu, Fei Huang, Guanting Dong, al. Qwen2.5 technical report. arXiv preprint arXiv:2412.15115, 2024.
[51] Senqiao Yang, Yukang Chen, Zhuotao Tian, Chengyao Wang, Jingyao Li, Bei Yu, and Jiaya Jia. Visionzip: Longer better but not necessary in vision language models. In Proceedings of the IEEE Conference Computer Vision and Pattern Recognition, 2025.
[52] Zhou Yu, Dejing Xu, Jun Yu, Ting Yu, Zhou Zhao, Yueting Zhuang, and Dacheng Tao. Activitynet-qa: A dataset for understanding complex web videos via question answering. In Proceedings of the AAAI Conference Artificial Intelligence, 2019.
[53] Xiaohua Zhai, Basil Mustafa, Alexander Kolesnikov, and Lucas Beyer. Sigmoid loss for language image pre-training. In Proceedings of the International Conference on Computer Vision, 2023.
[54] Kaichen Zhang, Bo Li, Peiyuan Zhang, Fanyi Pu, Joshua Adrian Cahyono, Kainui Hu, Shuai Liu, Yuanhan Zhang, Jingkang Yang, Chunyuan Li, et al. Lmms-eval: Reality check on the evaluation of large multimodal models. Findings of the Association for Computational Linguistics: NAACL, 2025.
[55] Qizhe Zhang, Aosong Cheng, Ming Lu, Zhiyong Zhuo, Minqi Wang. Jiajun Cao, Shaobo Guo, Qi She, and Shanghang Zhang. [cls] attention is all you need for training- visual token pruning: Make vlm inference faster. arXiv preprint 2rXiv:24/2.0188
2024.
[56] Yuan Zhang, Chun-Kai Fan, Junpeng Ma, Wenzhao Zheng, Tao Huang, Kuan Cheng, Denis Gudovskiy, Tomoyuki Okuno, Yohei Nakata, Kurt Keutzer, et al. Sparsevlm: Visual token sparsification for efficient vision- language model inference. In Proceedings of the International Conference on Machine Learning, 2025.
[57] Yuanhan Zhang, Jinming Wu, Wei Li, Bo Li, Zejun Ma, Ziwei Liu, and Chunyuan Li. Video instruction tuning with synthetic data. arXiv preprint 1rXiv:2410.02713, 2024.
[58] Junjie Zhou, Yan Shu, Bo Zhao, Boya Wu, Shitao Xiao, Xi Yang, Yongping Xiong, Bo Zhang, Tiejun Huang, and Zheng Liu. Mlvu: A comprehensive benchmark for multi-task long video understanding. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, 2025.
[59] Jiedong Zhuang, Lu Lu, Ming Dai, Rui Hu, Jian Chen, Qiang Liu, and Haoji Hu. St3: Accelerating multimodal large language model by spatial-temporal visual token trimming. In Proceedings of the AAAI Conference on Artificial Intelligence, 2025.
[60] Orr Zohar, Xiaohan Wang. Yann Dubois, Nikhil Mehta, Tong Xiao, Philippe Hansen-Estruch Licheng Yu, Xiaofang Wang, Felix Juefei-Xu, Ning Zhang, et al. Apollo: An exploration of video understanding large multimodal models, arXiv preprint arX.v.2412.10360, 2024.
14

---

NeurIPS Paper Checklist
1. Claims
Question: Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?
Answer: [Yes]
Justification:
The main claims clearly reflect the paper's contributions and scope.
Guidelines:
The answer NA means that the abstract and introduction do not include the claims made in the paper.
The abstract and/or introduction should clearly state the claims made, including the contributions made in the paper and important assumptions and limitations. A No or
NA answer to this question will not perceived well by the reviewers.
The claims made should match theoretical and experimental results, and reflect how much the results be expected to generalize other settings.
is fine include aspirational goals as motivation as long is clear that these goals are not attained by the paper.
2. Limitations
Question: Does the paper discuss the limitations of the work performed by the authors?
Answer: [Yes]
Justification: Please refer to the appendix for the discussion of limitations.
Guidelines:
The answer NA means that the paper has no limitation while the answer No means that the paper has limitations, but those not discussed in the paper.
The authors are encouraged to create separate "Limitations" section in their paper.
The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (e.g., independence assumptions, noiseless settings, model well-specification. asymptotic approximations only holding locally). The authors should reflect how these assumptions might be violated in practice and what the implications would be.
The authors should reflect on the scope of the claims made, e.g., if the approach was only tested on few datasets or with few runs. In general, empirical results often depend on implicit assumptions, which should be articulated.
The authors should reflect the factors that influence the performance of the approach. For example, facial recognition algorithm may perform poorly when image resolution low images are taken in low lighting. Or speech-to-text system might be used reliably to provide closed captions for online lectures because it fails to handle technical jargon.
The authors should discuss the computational efficiency of the proposed algorithms and how they scale with dataset size.
If applicable, the authors should discuss possible limitations of their approach address problems of privacy and faimess.
While the authors might fear that complete honesty about limitations might be used by reviewers as grounds for rejection, a worse outcome might be that reviewers discover limitations that aren't 't acknowledged in the paper. The authors should use their best judgment and recognize that individual actions in favor of transparency play an impor- tant role in developing norms that preserve the integrity of the community. Reviewers will be specifically instructed to not penalize honesty concerning limitations.
3. Theory assumptions and proofs
Question: For each theoretical result, does the paper provide the full set of assumptions and complete (and correct) proof?
Answer: [NA]
15

---

Justification:
Our work does not include theoretical results.
Guidelines:
The answer NA means that the paper does not include theoretical results.
All the theorems, formulas, and proofs in the paper should be numbered and cross- referenced.
All assumptions should be clearly stated referenced in the statement of any theorems.
The proofs can either appear in the main paper or the supplemental material, but they appear in the supplemental material, the authors are encouraged to provide short proof sketch provide intuition.
Inversely, any informal proof provided in the core of the paper should be complemented by formal proofs provided in appendix or supplemental material.
Theorems and Lemmas that the proof relies upon should be properly referenced.
4. Experimental result reproducibility
Question: Does the paper fully disclose all the information needed to reproduce the main ex- perimental results of the paper to the extent that affects the main claims and/or conclusions of the paper (regardless whether the code and data are provided or not)?
Answer: [Yes]
Justification: Sections
3 and provide sufficient details for reproducibility.
Guidelines:
The answer NA means that the paper does not include experiments.
If the paper includes experiments, a No answer to this question will not be perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data are provided or not.
If the contribution is dataset and/or model, the authors should describe the steps taken to make their results reproducible or verifiable.
Depending on the contribution, reproducibility can be accomplished in various ways. For example, if the contribution is a novel architecture, describing the architecture fully might suffice, or if the contribution is a specific model and empirical evaluation, it may be necessary to either make possible for others to replicate the model with the same dataset, or provide access to the model. In general. releasing code and data is often one good way accomplish this, but reproducibility can also be provided via detailed instructions for how to replicate the results, access to a hosted model (e.g.. in the case of large language model), releasing of a model checkpoint, or other means that are appropriate to the research performed.
While NeurIPS does not require releasing code, the conference does require all submis- sions to provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution. For example
(a) If the contribution is primarily a new algorithm, the paper should make clear how to reproduce that algorithm. (b) the contribution is primarily a new model architecture, the paper should describe the architccture clearly and fully.
(c) If the contribution is new model (e.g., a large language model), then there should either way to access this model for reproducing the results or way to reproduce the model (e.g., with an open-source dataset instructions for how to construct the dataset).
(d) We recognize that reproducibility may be tricky in some cases, in which case authors are welcome to describe the particular way they provide for reprodueibility. the case of closed-source models, may be that access to the model is limited in some way (e.g., to registered users), but it should be possible for other researchers have some path to reproducing or verifying the results.
5. Open access to data and code
Question: Does the paper provide open access to the data and code, with sufficient instruc- tions to faithfully reproduce the main experimental results, as described in supplemental material?
16

---

Answer: [Yes]
Justification: Due to time constraints, we provide code in the attached zip file to reproduce the main results in Table I The full code will be open-sourced once finalized.
Guidelines:
The answer NA means that paper does not include experiments requiring code.
Please see the NeurIPS code and data submission guidelines (https://nips... cc/
for more details.
While we encourage the release of code and data, we understand that this might not be possible, so "No" is an acceptable answer. Papers cannot be rejected simply for not including code, unless this is central to the contribution (e.g., for new open-source benchmark).
The instructions should contain the exact command and environment needed to to reproduce the results. See the NeurlPS code and data submission guidelines (https
ssicnlolicy) for more details.
The authors should provide instructions on data access and preparation, including how to access the raw data, preprocessed data, intermediate data, and generated data, etc.
The authors should provide scripts to reproduce all experimental results for the new proposed method and baselines. only subset of experiments are reproducible, they should state which ones are omitted from the script and why.
At submission time, preserve anonymity, the authors should release anonymized versions (if applicable).
Providing as much information as possible in supplemental material (appended to the paper) recommended, but including URLs to data and code permitted.
6. Experimental setting/details
Question: Does the paper specify all the training and test details (e.g., data splits, hyper- parameters, how they were chosen, type of optimizer, etc.) necessary to understand the results?
Answer: [Yes]
Justification:
We specify all necessary details in Section and the attached zip file.
Guidelines:
The answer NA means that the paper does not include experiments.
The experimental setting should be presented in the core of the paper to level of detail that is necessary to appreciate the results and make sense of them.
The full details can be provided either with the code, in appendix, or as supplemental material.
7. Experiment statistical significance
Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?
Question:
Answer: [No]
Justification: We focus on inference-time acceleration. Following prior works [1,51]] we use LMMs-Eval [54) for evaluation to ensure fairness and stability of the results.
Guidelines:
The answer NA means that the paper does not include experiments.
The authors should answer "Yes" if the results are accompanied by emror bars, confi- dence intervals, or statistical significance tests, at least for the experiments that support the main claims of the paper.
The factors of variability that the error bars capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, or overall with given experimental conditions).
The method for calculating the error bars should be explained (closed form formula, call to a library function, bootstrap, etc.)
The assumptions made should be given (e.g., Normally distributed errors).
17

---

It should be clear whether the error bar is the standard deviation or the standard error of the mean.
is OK to report 1-sigma error bars, but one should state it. The authors should preferably report a 2-sigma error bar than state that they have 96% CI, if the hypothesis of Normality of errors is not verified.
For asymmetric distributions, the authors should be careful not to show in tables or figures symmetric error bars that would yield results that are out range (e.g. negative enror rates).
If error bars reported in tables plots, The authors should explain in the text how they were calculated and reference the corresponding figures or tables in the text.
8. Experiments compute resources
For each experiment, does the paper provide sufficient information on the com- puter resources (type of compute workers, memory, time of execution) needed reproduce the experiments?
Question:
Answer: [Yes]
Justification: Details on compute are provided Section 4
Guidelines:
The answer NA means that the paper does not include experiments.
The paper should indicate the type of compute workers CPU or GPU, internal cluster, or cloud provider, including relevant memory and storage.
The paper should provide the amount compute required for each of the individual experimental runs well estimate the total compute.
The paper should disclose whether the full research project required more compute than the experiments reported in the paper (e.g., preliminary failed experiments that didn't make into the paper).
9. Code ethics
Question: Does the research conducted the paper conform, in every respect, with the NeurlPS Code of Ethics
Answer: [Yes]
Justification: We have carefully reviewed the NeurIPS Code of Ethics and confirm that this paper aligns with it.
Guidelines:
The answer NA means that the authors have not reviewed the NeurIPS Code of Ethics.
If the authors answer No, they should explain the special circumstances that require deviation from the Code of Ethics.
The authors should make sure to preserve anonymity (e.g., there a special consid- eration due to laws or regulations in their jurisdiction).
10. Broader impacts
Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?
Answer: [Yes]
Justification: Please refer to the appendix for discussion of broader impacts.
Guidelines:
The answer NA means that there no societal impact of the work performed.
If the authors answer NA or No, they should explain why their work has no societal impact why the paper does not address societal impact.
Examples of negative societal impacts include potential malicious or unintended uses (e.g., disinformation, generating fake profiles, surveillance), fairness considerations (e.g., deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations.
18

---

The conference expects that many papers will be foundational research and not tied particular applications, let alone deployments. However, if there is direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that an improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point out that a generic algorithm for optimizing neural networks could enable people to train models that generate Deepfakes faster.
The authors should consider possible harms that could arise when the technology is being used as intended and functioning correctly, harms that could arise when the technology being used as intended but gives incorrect results, and harms following from (intentional or unintentional) misuse of the technology.
If there are negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition attacks, mechanisms for monitoring misuse, mechanisms to monitor how system learns from feedback over time, improving the efficiency and accessibility ML).
11. Safeguards
Question:
Does the paper describe safeguards that have been put in place for responsible release of data or models that have a high risk for misuse (e.g., pretrained language models, image generators, scraped datasets)?
Answer: [NA]
Justification: This work focuses on accelerating inference for existing Video LLMs. We do not release any new data or models.
Guidelines:
The answer NA means that the paper poses such risks.
Released models that have high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere usage guidelines or restrictions to access the model or implementing safety filters.
Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images.
We recognize that providing effective safeguards is challenging, and many papers do require this, but we encourage authors to take this into account and make best faith effort.
12. Licenses for existing assets
Question: Are the creators or original owners of assets (e.g.. code, data, models), used in the paper, properly credited and are the license and terms of use explicitly mentioned and properly respected?
Answer: [Yes]
Justification: Please refer the appendix.
Guidelines:
The answer NA means that the paper does not use existing assets.
The authors should cite the original paper that produced the code package dataset.
The authors should state which version of the asset is used and, if possible, include URL.
The name of the license (c.g., CC-BY 4.0) should be included for each asset.
For scraped data from a particular source (e.g., website), the copyright and terms of service of that source should be provided.
If assets are released, the license, copyright information, and terms of use in the Datasets.paaersstt paperswi thcode. cow/ datasets package should be provided. For popular datasets,
has curated licenses for some datasets. Their licensing guide can help determine the license of dataset.
For existing datasets that are re-packaged, both the original license and the license of the derived asset (if it has changed) should be provided.
19

---

If this information is not available online, the authors are encouraged to reach out to the asset's creators.
13. New assets
Question: Are new assets introduced in the paper well documented and is the documentation provided alongside the assets?
Answer: [Yes]
Justification: Please refer the attached zip file.
Guidelines:
The answer NA means that the paper does not release new assets.
Researchers should communicate the details of the dataset/code/mmoel as part of their submissions via structured templates. This includes details about training, license, limitations, etc.
The paper should discuss whether and how consent obtained from people whose asset is used.
At submission time, remember to anonymize your assets (if applicable). You can either create an anonymized URL or include an anonymized zip file.
14. Crowdsoureing and research with human subjects
Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given to participants and screenshots, if applicable, as well details about compensation (if any)?
Answer: [NA]
Justification: There were no such experiments or research.
Guidelines:
The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.
Including this information in the supplemental material is fine, but if the main contribu- tion of the paper involves human subjects, then as much detail as possible should be included the main paper.
According the NeurIPS Code of Ethics, workers involved in data collection, curation, or other labor should be paid at least the minimum wage in the country of the data collector.
15. Institutional review board (IRB) approvals or equivalent for research with human subjects
Question:
Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or an equivalent approval/review based on the requirements of your country or institution) were obtained?
Answer: [NA]
Justification:
There were such participants.
Guidelines:
The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.
Depending the country in which research is conducted, IRB approval (or equivalent) may be required for any human subjects research. If you obtained IRB approval, you should clearly state this in the paper.
We recognize that the procedures for this may vary significantly between institutions and locations, and expect authors to adhere to the NeurlPS Code of Ethics and the guidelines for their institution.
For initial submissions, do not include any information that would break anonymity (if applicable), such the institution conducting the review.
16. Declaration of LLM usage
20

---

Question: Does the paper describe the usage of LLMs if it is important, original, or non-standard component of the core methods this research? Note that if the LLM used only for writing, editing, or formatting purposes and does not impact the core methodology, scientific rigorousness, or originality of the research, declaration is not required.
Answer: [NA]
Justification: The LLM was only used for editing purposes (e.g., grammar, spelling, word choice).
Guidelines:
The answer NA means that the method development in this research does involve LLMs as any important, original, or non-standard components.
Please refer to our LLM policy for what should or should be described.
21

---

Appendix
This appendix provides additional details and results to support our main paper:
Appendix A
presents additional experimental settings, including reproduction details of the compared baselines and an estimation of the computational cost.
Appendix B presents additional experimental results, such as ablation studies on key hyperpa- rameters and further quantitative and qualitative results.
Appendix discusses limitations of our approach and its potential broader impacts.
A
Additional Experimental Settings
A.1 Reproduction Details of Compared Baselines
All experiments are conducted using LMMs-Eva
54 for consistency. The performance of the LLaVA- Video [57]. and Qwen2- [45] differs
vanilla versions of LLaVA-OUe Lne Vision 19
slightly from their reported results, remaining within an acceptable margin of error. We reimplement all baseline methods using LMMs-Eval, following their official implementations:
Fast V 6 (ECCV 2024). FastV performs token pruning at the K-th layer of the LLM
using attention scores, with filtering ratio R. We reimplement it with K 2, using R (75%, 80%, 85%, 90%} in Table
[ and R E (75%, 90%} in Table
VisionZip [51] (CVPR 2025). Visionzip performs pruning at the vision encoder's output, which conflicts with pooling operations in Video LLMs and degrades performance. To address this, we implement VisionZip', which applies pruning after pooling. Following the original settings, each frame retains both dominant and contextual tokens a 54: 10 ratio. We define as the proportion of tokens retained per frame. We set r E (25%, 20%, 15%, 10%} in Table ] and r E (25%, 10%} in Table 2
DyCoke 41 (CVPR 2025). DyCoke prunes during both the prefill and decode stages. For fair comparison, we only evaluate its prefill stage. The pruning rate in the TTM module is set K E {0.9, 1.0] in both Table and Table 2
Prune 16 (ACL 2025). PruneVID includes both input-stage and intra-LLM pruning in the prefill phase, along with decoding-stage pruning. For fair comparison, evaluate two variants: Viu (input-stage pruning only) and Prune VID (full prefill-stage pruning). Following the original settings,
a threshold 0.8, temporal segment ratio 7 0.25, token selection ratio a = 0.4, and attention calculations use the 10th layer. The cluster ratio / controls compression in the input-stage pruning. We use 3 11% % in Table and Table
A.2 Computational Cost Estimation
Following prior works 6 47, report the theoretical FLOPs of the LLM related to visual/video
tokens. Specifically, LLaVA-One Vision [19], LLaVA- Video [57], and Qwen2- VL [45] are all built on Qwen2 4 which employs grouped-quers attention 2 and a SwiGLU-base three-layer feed- The per-layer FLOOPs of the LLM are computed as:
forward network 33
2nD(haod) + D² + 2nd D + 3'D
(6)
where n is the number of video tokens, D is the hidden state size, D' is the FFN intermediate size, fuo is the number of key/value heads, and dl is the head dimension.
httpps//ggthabb.oo//s//oviiggg/g /ambs-mmmm---/m eval
MIT License
bttp:://gthhhb.coo./LLa.//,L,,,t NeXT Apache License 2.0
http:://gttabb..oem////////////p Apache License 2.0
Apache License 2.0
http::///hthbbbbcc//00000001100 Apache License 2.0
http:://ggtabboooa///aaaaaaaa/lr CC BY-NC-SA 4.0 License
22

---

59
100
100
Avg. ACc.
98.0
59
VideOMME
98.0
VideoMME
58
Avg. Acc.
VideoMME
Avg. Acc.
57.3
57.3
57-
95
95
Avg. Acc. VideoMME
56
12
16
32
0.5
0.6
0.7
0.95
(a)
(b)
Figure 6: Ablation study on e and T in DySeg. denotes the minimum number of segments in video, whereas 7 denotes the transition similarity threshold. Both parameters jointly control the segmentation granularity.
Avg.
98.0
VideOMME
98.0
VideoMME
Avg. Acc.
VideoMME
Avg. Acc.
57.3
95
95
Avg. Acc. VideoMME
56
56
2
4
8
0.0
0.2
0.4
1.0
p
B
(a)
(b)
Figure 7: Ablation study on p and DTM. p denotes the interval for anchor frame selection,
whereas 3 controls the merging weight for anchor tokens and their associated tokens in Eq. 5
B
Additional Experimental Results
B.1 Ablation study on and in DySeg
We evaluate the effect of and 7 in Eq. (2) of DySeg. Figure ((a) shows results with varing c. In simple video scenarios with high transition similarity, e regulates segmentation. When c segmentation relies solely on т. = 32 divides the video into single-frame segments. Performance remains stable for < c < 16, but the decline at c disregards temporal relationships. The optimal performance at 8 indicates the benefit of minimum cluster number. Figure([lb shows results with varing т. When 7 is small, most transitions in S are selected by As increases, more transitions fall below the threshold. The best performance achieved at 0.9, effectively grouping redundant frames while separating non-redundant ones.
suggests that excessive segmentation
B.2 Ablation study on p and 3 in DTM
In Figure aa), we evaluate the effect of p on anchor frame selection, from which anchor tokens are subsequently sampled. When p 1, anchor tokens are evenly distributed across all frames within a segment. As p increases, fewer frames are selected as anchors, while more anchor tokens are allocated each anchor frame. Given the high similarity between frames segment, a sufficient number of anchor tokens per anchor frame is essential to capture visual context. However, if is too large, all anchor tokens come from the first frame, limiting temporal information. We find that p yields the best performance by effectively capturing spatiotemporal context. Figure (Tb) shows the effect of 3 Eq. ( When 3 = 0.0, matching tokens are averaged. / = 1.0 discards all non-anchor tokens, removing segment visual context and leading to performance drop. Notably, Anchor-centric Assignment (when 3 0.6) yields optimal results, highlighting the importance of representative tokens and visual context.
23

---

Table 11: Additional quantitative results on VideoMME. We report the overall performance of LLaVA-Ore Vision.
Table 12: Additional quantitative re- sults ANet-QA. We report the per- formance of LLaVA-OneVision
Method
Retention Ratio R
TELOPs
VideoMME
ANet QA
Vanilla
100%
48.82
58.6
Method
TELOPs
Accurney
Score
Pyramiel Deop CVER
14.70
51.8
Spurse VLM
56 CyR/2s
100%/10.45. 35.
7.06
53.3
Vanilla
48.82
$4.4
3.57
LLaVA LLAAA-Pruuaee PruMerge
322 Iccv22
9.7%
4.04
49.9
Prranidroop
14.70
41.2
3.07
FIstVID
9,7%
FIs
4.04
53.1
3.52
B.3 Additional Quantitative Results
In Table conduct additional comparative experiments against previous methods.
Following its official settings, we prune tokens at the 8th, 16th, and 24th layers with retention ratios of 4.1%, 2.0%, and 1.0%, respectively. Despite aggressive pruning, PyramidDrop still incurs high FLOPs due to full video token processing in the first 8 layers. In contrast, our VID compresses tokens from the input stage, reduces computation significantly, and outperforms PyramidDrop under extreme pruning.
Comparison with PyramidDrop 47
Comparison with VLM [56]. Following its official setup, we prune tokens at the 3rd, 7th, and 16th layers with retention ratios of 10.4%, 5.3%, and 2.9%. respectively. Although SparseVI.M aggressively prunes later layers, it still incurs high FLOPs due to full token processing in the early layers. In contrast, VIs compresses tokens from the input stage, leading to significantly lower computational cost. Moreover, VID achieves better performance due to its more effective video token pruning strategy.
Comparison with LLaVA- PruMMeree [2]. Although both LLaVA- PPrumeree and our ATS select tokens based on attention scores, our FastVID is specifically designed better preserve both temporal and visual context in Video LLMs. A direct quantitative comparison is provided in Table Under the same video token budget of 608 tokens (=99.7% retention ratio), method significantly outperforms LLaVA-PruMerge.
B.4 Results for long-context generation
FastVID prunes redundant video tokens while preserving critical semantic information, which supports both short- and long-context generation tasks. We select ANet-QA [52] for the video captioning task reported in Table [2] Metrics are computed using 99T-3.5-Turbo-01 FastVID achieves 97.6 % accuracy and 98.6 % score, demonstrating its effectiveness in long-context generation.
B.5 Additional Qualitative Results
Figures 10 show additional qualitative results on VideoMME. We use the following settings: 0,7 0.9, d 0.4, p 2, retaining an average of 10 tokens per frame. Figure presents static scenes, while Figures 9 and 10 present dynamic scenes. These visualizations demonstrate FastVID's ability to adaptively segment videos across varying temporal dynamics. Within each segment, the merged region boundaries loosely align with object shapes, highlighting the effectiveness of the proposed DTM in preserving visual context.
Figures b)) and 9 showcase failure cases where the query is relevant only a small subset of frames. In Figure B)), only few frames depict the subject eating banana, while in Figure Daa) only a few frames correspond to the entrance scene. In such scenarios, when detailed questions are posed, our query-agnostic pruning strategy becomes less effective. Under high compression rates, most retained tokens are allocated to query-irrelevant frames, making challenging for the model to provide the necessary visual evidence and produce accurate responses.
24

---

Additional Discussions
C.1 Comparison with PruneVID
VID [[6] and FastVID differ fundamentally in both design and implementation. Below, we detail the key technical distinctions and their practical impact on efficiency and accuracy.
Methodological Differences
(a) Segmentation Strategy: PruneVID uses the original density-based clustering to partition the video into segments, with a hard upper bound on segment number (S sampled frames Y = 0.25). This constraint may cause semantically diverse frames to be grouped together in complex videos, reducing the effectiveness of subsequent pruning. contrast, FIstVID uses frame transition similarity to adaptively segment the video, ensuring high intra-segment similarity, thereby facilitating more efficient and accurate token pruning.
(b) Compression Strategy: PruneVID performs clustering- -based merging (Figure 4 aa) on both dynamic and static tokens within each segment, without considering token representativeness or positional structure. In contrast, our proposed density-basee based token merging DTM selects anchor tokens only from anchor frames and merges the remaining tokens within each segment. The number of anchor tokens per frame adaptive to segment length. Importantly, our DTM explicitly emphasizes anchor tokens located at density peaks. Positional information of anchor tokens preserved to maintain the spatiotemporal structure, which is particularly benefi- cial for the Qwen-VL series that adopts M-RoPE. We further apply anchor-centric aggregation instead simple average pooling to highlight representative anchor tokens.
2. Practical Impact
(a) Efficiency: In Table we conduct a detailed efficiency comparison. The primary bottleneck in efficiency lies in the density score computation. Our segmentation based on frame transition similarity is significantly faster than PruneVID's clustering. During compression, Prune clusters dynamic and static tokens separately per segment. Crucially, the number of tokens varies across segments, which leads to repeated density computations. In contrast, we computes density scores in parallel across all anchor frames, requiring only single pass. As result, pruning speed is 6.1x x faster (6.1ms vs. 37. 37.2ms).
(b) Accuracy: We compare FruneVID across multiple Video LLMs, including LLaVA- Vision (Table D Qwen2-VL (Table 3) and Qwen2.5-VL. (Table 4 Under both the 32 fixed-frame sampling setting ILLaVA-OneVine Vision) and dynamic frame sampling (Qwen-VI. series), our method consistently outperforms VID at the similar compression ratio. We attribute this improvement to our more effective dynamic segmentation strategy, which ensures high intra-segment similarity. Furthermore, our token merging strategy highlights representative tokens while retaining thier positional information, making it well-suited for diverse models.
In summary, FastVID introduces a novel framework that integrates dynamic temporal segmentation with density spatiotemporal pruning, effectively preserving both temporal and visual context. achieves consistent improvements in both efficiency and accuracy over prior methods across diverse Video LLMs.
C.2 Limitations
FastVID achieves strong performance on LLaVA-OneVisiion [9 (32 frames/video), maintaining com-
parable accuracy with only 8.3% of the FLOOPs. However, LLa [57] (64 frames/video), Qwen2-VI. [45] (768 frames/video) and Qwen2.5-VI. 4 (768 frames/video). although FastVII consistently outperforms existing SOTA baselines, a noticeable accuracy drop occurs compared to
the original model.
This degradation mainly arises from the nature of our query-agnostic pruning strategy. Our aggressive pruning retains only a small number of tokens. When temporal spans are long, few of these retained tokens relate to the query, leading performance drops. To overcome these challenges, future work may explore integrating query-guiee guided keyframe selection to better support long-frame Video LLMs 45 [57].
However, query-agnostic pruning has a key advantage: the pruned KV cache can reused across multiple dialogue turns with different queries, making it more suitable for multi-turn conversations.
25

---

Both query- and query ageossig methods have their own advantages, and the choice can be made based on specific application scenarios.
C.3 Broader Impacts
FastVID accelerates inference for existing Video LLMs without modifying their parameters or training new models, thereby minimizing the risk of introducing new biases or unintended behaviors. However, VastVID inherits any potential negative societal impacts of the original models, such as representational bias or potential misuse. Despite this, FastVID maintains strong performance even under extreme token compression. This makes the use of Video LLMs more practical in environments with limited computational resources, promoting broader and more sustainable deployment.
26

---

Retained Tokens
DTM on the 1st frame
DTM on the 3rd frame
DTM the 5nd trame
DTM on the 7nd frame (a)
Retained Tokens
DTM on the st frame
DTM the 3rd frame
DTM on the 5nd frame
DTM on the 7nd frame (b) Figure 8: Segment boundaries are marked by brown vertical lines. Tokens generated by DTM
and ATS highlighted in red and blue, respectively. Anchor frames, indicated by red boxes, processed by DT'M individually. DTM, patches with matching inner and border colors are merged.
27

---

Retained Tokens
DTM on the 1st frame
DTM on the 3rd frame
DTM on the 5nd frame
DTM on the 7nd frame
Retained Tokens
DTM on the 1st Trame
DTM the 7nd frame
DTM the Trame
DTM on the 4nd frame
BDn
DTM on the 5nd frame
(b)
Figure 9: Segment boundaries are marked by brown vertical lines. Tokens generated by DTM and ATS highlighted in red and blue, respectively. Anchor frames, indicated by red boxes, processed by DT'M individually. DTM, patches with matching inner and border colors are merged.
28

---

Salaes
Carfee
Retained Tokens
DTM on the 1st frame
DTM on the 3rd frame
DTM on the 4nd frame
OFD
DTM the 6nd frame
DTM on the frame
Retained Tokens
DTM on the 1st frame
DTM the 3rd frame
DTM the 4nd frame
DTM on the ond frame
DTM on the 7nd frame
(b)
Figure 10: Segment boundaries are marked by brown vertical lines. Tokens generated by DTM and ATS are highlighted in red and blue, respectively. Anchor frames, indicated by red boxes, processed by DT'M individually. DTM, patches with matching inner and border colors are merged.
29


Reviews:
[{"review_id": "ecHbSjD8PW", "reviewer": "Reviewer_JwZp", "summary": "This work addresses the significant computational cost of inference in Video LLMs, which is primarily caused by the large number of redundant video tokens. This work proposes FastVID, an inference-time, plug-and-play token pruning framework that requires no retraining. The core idea is to systematically reduce spatiotemporal redundancy while preserving essential information. FastVID consists of two main stages: (1) Dynamic Temporal Segmentation (DySeg), which adaptively partitions the video into temporally ordered segments of high visual similarity, and (2) Density Spatiotemporal Pruning (STPrune), which operates within each segment to prune tokens. STPrune consists of two modules: Density-based Token Merging (DTM) to retain representative visual context and Attention-based Token Selection (ATS) to capture salient details. Extensive experiments are conducted on two Video LLMs (LLaVA-OneVision and LLaVA-Video) across several challenging benchmarks. The results demonstrate that FastVID can prune over 90% of video tokens, leading to a 7.1x speedup in the prefilling stage, while maintaining 98.0% of the original model's accuracy, significantly outperforming existing token compression methods.", "questions": "1.  Experiments on the LLaVA series models are convincing. However, since these models share a similar architectural backbone, demonstrating its effectiveness on a more diverse set of models would significantly improve the claims of generalizability.\n\n2. The current experiments are conducted on inputs of up to 64 frames. Would FastVID still work when scaled to much longer video inputs (e.g., hundreds of frames)? Furthermore, could your method potentially aid in length extrapolation at inference time, allowing models trained on shorter contexts to process longer videos more effectively by compressing the input sequence?\n\n3. The ablation study thoroughly investigates the impact of the parameter d. Could you provide a similar analysis or discussion on the sensitivity of the other key hyperparameters?\n\n4. While the overall performance gap is small, any pruning method inevitably has limitations. Could you provide some qualitative analysis of failure cases? For instance, what types of videos or questions cause FastVID to underperform compared to the vanilla model?", "limitations": "yes", "rating": 5, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 3, "significance": 3, "originality": 3, "strengths_and_weaknesses": "**Strengths**\n1. This method recognizes the non-uniform distribution of information across the temporal dimension of videos. The proposed Dynamic Temporal Segmentation algorithm is a well-designed for this insight.\n\n2. The authors evaluate FastVID on a wide array of benchmarks that cover short, long, and complex videos.\n\n3. The comparison includes recent and relevant state-of-the-art methods (DyCoke, VisionZip, PruneVID). This work re-implements baselines (e.g., VisionZip*) to ensure a fair comparison (pruning after pooling), which adds to the credibility of the results.\n\n4. The reported results—achieving a massive 7.1x speedup with only a 2% drop in accuracy—are highly impressive.\n\n**Weaknesses**\n\n1. While the authors evaluate their method on two Video LLMs (LLaVA-OneVision and LLaVA-Video), these models share a nearly identical architecture, with the primary difference being the training data. To more convincingly demonstrate the generalization capability of FastVID, it would be beneficial to test it on a strong, open-source Video LLM with a more distinct architecture, such as Qwen2.5-VL. This would provide stronger evidence that the method is truly model-agnostic.\n\n2. The experiments in the paper use a maximum of 64 sampled frames. This input length may not be sufficient to fully evaluate the method's ability to compress temporal redundancy in long videos. The effectiveness of temporal segmentation and pruning might change as the temporal context becomes significantly more extended and complex.\n\n3. The proposed method introduces five hyperparameters, yet the ablation studies primarily focus on d. A sensitivity analysis for the other key parameters is missing. For instance, it is unclear how robust the performance is to changes in the segmentation threshold τ or the anchor sampling interval p. Furthermore, the paper could explore whether some hyperparameters are avoidable. For example, could the merging of anchor and associated tokens be simplified by using a standard pooling operation instead of introducing the balance parameter α?\n\n4. The pruning process in FastVID is content-agnostic, meaning it does not leverage information from the user's instruction or query. This could be a limitation, as tokens relevant to the query might be pruned away if they are not considered salient by the general-purpose [CLS] attention or density-based merging.\n\n5. The training of both LLaVA-Video and LLaVA-OneVision involves fine-tuning the pretrained SigLIP encoder. However, for ATS, the proposed training-free method reintroduces the original, un-finetuned SigLIP head to generate attention scores. This creates a potential parameter gap between the vision encoder used by the Video LLM and the head used for token selection. This discrepancy might have a subtle but non-negligible impact on performance, as the attention scores may not perfectly align with the feature space the fine-tuned model has adapted to. A discussion of this potential issue would strengthen the paper.", "was_revised": true, "final_justification": "The authors address all my concerns, including new comparison on Qwen2.5-VL, length extrapolation and other ablations. I will raise my rating. And I suggest the authors should include the new discussion proposed in the rebuttal.", "rebuttal": "Thank you for your positive and insightful comments, which has been instrumental in improving the overall quality of our work. In response, we have performed further experiments and analyses. All clarifications and results will be included in the revised paper.\n\n***\n\n**Question 1**: Experiments on the LLaVA series models are convincing. It would be beneficial to test it on a strong, open-source Video LLM with a more distinct architecture, such as Qwen2.5-VL. \n\n**Response**: Thank you for the insightful suggestion. To further demonstrate its generalization ability, we now include results on Qwen2.5-VL in Table 1. Unlike the LLaVA series and Qwen2-VL (Table 7 of the appendix), Qwen2.5-VL employs a vision encoder with window attention and a Qwen2.5 LLM, representing a substantially different architecture. Compared to the recent SOTA PruneVID (ACL25), FastVID achieves a **3.5** improvement under a comparable retention ratio. This supports that FastVID is model-agnostic and generalizes well across diverse Video LLM architectures.\n\n#### **Table 1: Results on Qwen2.5-VL.**\n|Qwen2.5-VL|Retention Ratio|TFLOPs|VideoMME|\n|:---:|:---:|:---:|:---:|\n|Vanilla|100%|124.0|_67.3_|\n|PruneVID* (ACL25)|24.5%|23.7|59.3|\n|FastVID|24.1%|23.2|**62.8**|\n\n***\n\n**Question 2**: The current experiments are conducted on inputs of up to 64 frames. Would FastVID still work when scaled to much longer video inputs (e.g., hundreds of frames)?\n\n**Response**:  Thank you for raising this important point. FastVID is fully compatible with long video inputs. In fact, both Qwen2-VL (Table 7 in the appendix) and Qwen2.5-VL (response to Q1) adopt a dynamic resolution and frame sampling strategy, allowing up to 768 sampled frames per video. Specifically, on VideoMME, Qwen2.5-VL processes an average of 549.8 sampeld frames per video. FastVID achieves SOTA performance on both Qwen2-VL and Qwen2.5-VL, demonstrating its effectiveness in handling long and complex temporal contexts.\n\n***\n\n**Question 3**: Could your method potentially aid in length extrapolation at inference time, allowing models trained on shorter contexts to process longer videos more effectively by compressing the input sequence?\n\n**Response**: Thank you for the insightful question. As shown in Table 2, by applying FastVID with different compression ratios, we extend the number of sampled frames from 32 to 320 while maintaining a comparable total number of input video tokens. This consistently improves performance, which demonstrates that FastVID not only compresses video tokens efficiently but also enables effective length extrapolation at inference time.\n\n#### **Table 2: Improved length extrapolation via FastVID.**\n|LLaVA-OneVision|# Frame|# Token|VideoMME|\n|:---:|:---:|:---:|:---:|\n|Vanilla|32|6272|58.6|\n|FastVID (r=25%)|128|6272|60.4|\n|FastVID (r=10%)|320|6080|**61.4**|\n\n***\n\n**Question 4**: The ablation study thoroughly investigates the impact of the parameter d. Could you provide a similar analysis or discussion on the sensitivity of the other key hyperparameters?\n \n**Response**: We appreciate your concern about hyperparameter sensitivity. Due to space limits, we report comprehensive analyses of other hyperparameters in the appendix (Sections B.1, B.2; Figures 6, 7). We apologize for the typo in Section B.2. The symbol $\\beta$ refers to the balance parameter $\\alpha$ in Eq. (5). Specifically, when $\\alpha=0$, the merging degenerates into a standard pooling operation (line 843), addressing your suggestion for a simpler alternative.\n\n***\n\n**Question 5**: This creates a potential parameter gap between the vision encoder used by the Video LLM and the head used for token selection. A discussion of this potential issue would strengthen the paper.\n\n**Response**: Thank you for the insightful comment. We acknowledge the potential parameter gap between the fine-tuned SigLIP encoder and the original SigLIP head. However, we believe this gap has limited impact for the following reasons. \n\nFirst, the SigLIP head is pretrained jointly with the encoder on large-scale vision-language data, enabling strong generalization and transferability. Second, since the first-stage training of Video LLMs also aims to align visual and textual representations, the SigLIP head is naturally compatible with this alignment objective. Finally, as shown in Table 3, results show that using the SigLIP head leads to a 1.0 improvement, indicating its effectiveness. This is likely because the SigLIP head is explicitly trained to aggregate patch tokens for semantic alignment, making its attention scores better suited for identifying semantic salient tokens.\n\n#### **Table 3: Ablation study on the SigLIP head.**\n|LLaVA-OneVision|VideoMME|\n|:---:|:---:|\n|FastVID|**57.3**|\n|w/o the SigLIP head|56.3|\n\nIn addition, we emphasize that our FastVID generalizes to vision encoders without pretrained heads or [CLS] tokens (e.g., Qwen-VL series). In such cases, [CLS] attention scores are computed at the final layer using pseudo [CLS] tokens derived by averaging patch tokens. Our FastVID achieves strong performance on Qwen2-VL (Table 7 in the appendix) and Qwen2.5-VL (response to Q1), significantly outperforming previous methods.\n\n***\n\n**Question 6.1**: The pruning process in FastVID is content-agnostic. This could be a limitation, as tokens relevant to the query might be pruned away.\n \n**Response**: Thank you for pointing out the limitation of query content-agnostic pruning. Our aggressive pruning retains only a small number of tokens. When temporal spans are long, few of these retained tokens relate to the query, leading to performance drops (discussed in Section C.1 Limitations). However, query-agnostic pruning has a key advantage: the pruned KV cache can be reused across multiple dialogue turns with different queries, making it more suitable for multi-turn interactions. Both query-aware and query-agnostic methods have their own advantages, and the choice can be made based on specific application scenarios.\n\n***\n\n**Question 6.2**: Could you provide some qualitative analysis of failure cases? For instance, what types of videos or questions cause FastVID to underperform compared to the vanilla model?\n\n**Response**: Thank you for the suggestion. As discussed in our response to Q6.1, we analyze a typical failure case of our FastVID, particularly when the query relates to only a few frames. In such cases, most of the retained tokens after pruning may not be related to the query. Unfortunately, we are unable to include qualitative examples during the rebuttal phase due to formatting constraints, but we will provide detailed failure cases in the revised version."}, {"review_id": "uZJNZ38CWf", "reviewer": "Reviewer_QvJS", "summary": "This paper introduces FastVID, a dynamic token pruning method designed to reduce the cost of Video LLMs by addressing spatiotemporal redundancy. FastVID analyzes video redundancy through temporal and visual contexts, then applies a two-part strategy: (1) temporal segmentation to preserve sequence structure, and (2) density-based pruning to retain essential visual tokens. Evaluated on models like LLaVA-OneVision, FastVID achieves state-of-the-art efficiency-performance tradeoffs across various video benchmarks.", "questions": "- (maybe I missed sth. somewhere) in line 225: why adopts two representative Video LLMs? what is the tradeoff for each base model?\n(Thanks the author for explaining the reason.)", "limitations": "yes", "rating": 4, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 2, "clarity": 3, "significance": 2, "originality": 2, "strengths_and_weaknesses": "Strengths:\n- focus on efficiency of video processing, a key bottleneck of video LLMs;\n- strong tradeoff on performance vs efficiency (98% of accuracy with 90.3% pruned tokens and 8.3% of FLOPs);\n- figures are intuitive to understand the benefits of the proposed method;\n- diverse benchmarks and detailed ablation studies;\n\nWeaknesses:\n- some arguments are too strong: i) in abstract, “fully exploit”: probably no work(includes this paper) can do fully but only improve? ii) line 26: “nature of video redundancy”, there’s probably no such nature exists, video redundancy may just be about having high-frequency changes that human does not care much at this stage? iii) line 134: “without loss of key information”, without is too strong (the definition of key information may vary and there’s always a loss of information for any neural network?)\n- general presentation issue: i) in abstract etc. “visual context”: visual is very general and covers temporal, here it probably means spatial; ii) Figure 1 (b), (b1) and (b2) are not easy to understand at the beginning, probably because lacking description/legend about what does purple/green etc. area mean? iii) Sec. 3.1 sounds like a tone of another abs/intro.\n- limited novelty, compress video tokens/making efficiency tradeoff is very common in video LLMs.", "was_revised": true, "final_justification": "All my concerns are addressed: eg, 1, revision the writing to be more objective on arguments and (no more strong over claim); 2, explained reasons of adopting two baselines. Will keep the score unchanged given the impact of the main subject of this paper.", "rebuttal": "Thank you for your positive and insightful comments, which has been instrumental in improving the overall quality of our work. In response, we have performed further analyses. All clarifications will be included in the revised paper.\n\n***\n\n**Question 1**: Some arguments are too strong. 1. \"Fully exploit\". 2. \"Nature of video redundancy\". 3. \"Without loss of key information\".\n\n**Response**: Thank you for your detailed suggestions regarding our wording. We agree that some phrases can be overly strong and will revise them accordingly. Specifically:\n\n 1. \"Existing pruning techniques fail to _effectively exploit_ the spatiotemporal redundancy present in video data.\"\n 2. \"As a result, _spatiotemporal redundancy_ remain insufficiently explored.\"\n 3. \"..., enhancing efficiency _with minimal loss_ of key information.\"\n\nAdditionally, we will revise the full manuscript to improve clarity and precision, aided by writing tools and consulting native speakers.\n\n***\n\n**Question 2**: General presentation issue. 1. \"Visual context\". 2. Figure 1(b) lacks descriptions. 3. Sec. 3.1 sounds like a tone of another abs/intro.\n\n**Response**: Thank you for the detailed and constructive suggestions.\n\n 1. We refer to _visual context_ as the both spatial and temporal information within each video segment. After partitioning the video into temporally ordered, high-redundancy segments, we propose density spatiotemporal pruning to preserve the _visual context_ within each segment.\n 2. We will revise both the caption and the legend in Figure 1 to improve clarity. Specifically, we will emphasize that regions with the same color are merged together. For example, the purple area indicates that patches from the hand region are merged.\n 3. In Sec. 3.1, we will remove lines 111–121 and revise lines 122–135 to focus solely on introducing the framework of FastVid, ensuring it reads as a technical section rather than a second abstract.\n\n***\n\n**Question 3**: limited novelty, compress video tokens/making efficiency tradeoff is very common in video LLMs.\n\n**Response**: Thank you for your comment. While token compression is indeed a common goal in Video LLMs, we highlight that our method introduces a novel framework that combines dynamic temporal segmentation with density spatiotemporal pruning to preserve both temporal and visual context. \n\nMost prior methods (e.g., Dycoke CVPR25, PruneVID ACL25) rely on fixed-length segmentation or clustering with global constraints, which often group semantically diverse frames into the same segment. This leads to pruning important context or merging visually distinct tokens. In contrast, our FastVID adaptively identifies highly redundant segments and preserves representative and distinctive tokens, enabling efficient yet accurate pruning. Our FastVID significantly improves both efficiency and accuracy over prior methods across diverse Video LLMs.\n\n***\n\n**Question 4.1**: Why adopts two representative Video LLMs?\n\n**Response**: Thank you for your question. We apply FastVID to LLaVA-OneVision (Table 1 in the main paper), LLaVA-Video (Table 2 in the main paper), Qwen2-VL (Table 7 in the appendix), and Qwen2.5-VL (response to Reviewer JwZp's Q1). These models are selected to demonstrate that our method is model-agnostic and robust across different architectures. FastVID consistently achieves SOTA compression performance on all of them, confirming its general applicability and effectiveness.\n\n***\n\n**Question 4.2**: What is the tradeoff for each base model?\n\n**Response**: Thank you for your question. The LLaVA series serves as commonly used academic baselines, while the Qwen-VL series comprises powerful open-source MLLMs. Each base model is widely used in the Video LLM community and exhibits distinct designs. LLaVA-Video inserts newline tokens after each height-wise position within each frame. The Qwen-VL series adopts a dynamic resolution and frame sampling strategy, combined with M-RoPE to better handle video tokens. Unlike Qwen2-VL, Qwen2.5-VL employs a vision encoder with window attention and a Qwen2.5 LLM. Our FastVID demonstrates strong generalization and effectiveness across all these mainstream models, validating its robustness under diverse model designs."}, {"review_id": "O9g2h1L5Q4", "reviewer": "Reviewer_vfkR", "summary": "This paper introduce FastVID, a novel inference-time pruning framework designed to accelerate Video LLMs by effectively reducing spatiotemporal redundancy. It achieves a 7.1× speedup while preserving stable accuracy. However, more models, benchmarks and prior works should be considered.", "questions": "please refer to weaknesses", "limitations": "yes", "rating": 4, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 3, "significance": 3, "originality": 3, "strengths_and_weaknesses": "Strengths\n\n1. The paper is well-organized.\n\n2. The figures are visually good and easy to understanding.\n\n3. The method solves the problem well. It efficiently prunes the redundant visual tokens in video and achieves 7x acceleration for video LLM.\n\nWeaknesses\n\n1. More video LLMs should be considered in experiments (e.g. Qwen2.5-vl). LLava-onevision and qwen2-vl employ the same LLM (qwen2), it is better for authors that testing their method in other MLLMs based on various LLM. \n\n2. The datasets are almost VideoQA, what about the performance on other long context generation datasets? Because the method proposed also prune KV Cache, will it cause a decrease in quality on long context  generation(like video captioning task ActivityNet, VideoDetailCaption e.g.) \n\n3. Some prior works VTW[1], ST3[2], pyramiddrop[3] should be discussed in related work or compared in experiments:\nThe FastV is a old baseline, so the recent method blow should be consider in comparison.\n\n[1] Boosting Multimodal Large Language Models with Visual Tokens Withdrawal for Rapid Inference. AAAI25\n\n[2] ST3: Accelerating Multimodal Large Language Model by Spatial-Temporal Visual Token Trimming. AAAI25\n\n[3] PyramidDrop: Accelerating Your Large Vision-Language Models via Pyramid Visual Redundancy Reduction. CVPR25", "was_revised": true, "final_justification": "Thanks for your rebuttal, my concerns are resolved and I will maintain my score. I suggest authors include the new discussions and experiments in their revision.", "rebuttal": "Thank you for your positive and insightful comments, which has been instrumental in improving the overall quality of our work. In response, we have performed further experiments and analyses. All clarifications and results will be included in the revised paper.\n\n***\n\n**Question 1**: More video LLMs should be considered in experiments (e.g.  Qwen2.5-vl). LLava-onevision and qwen2-vl employ the same LLM (qwen2), it is better for authors that testing their method in other MLLMs based on various LLM.\n\n**Response**: Thank you for the valuable suggestion. To evaluate the generalization ability of FastVID, we report results on Qwen2.5-VL in Table 1. Unlike the LLaVA series and Qwen2-VL, Qwen2.5-VL employs a vision encoder with window attention and a Qwen2.5 LLM, representing a substantially different architecture. Compared to the recent SOTA PruneVID (ACL25), FastVID achieves a **3.5** gain under a comparable retention ratio. This supports that FastVID is model-agnostic and generalizes well across diverse Video LLM architectures.\n\n#### **Table 1: Results on Qwen2.5-VL.**\n|Qwen2.5-VL|Retention Ratio|TFLOPs|VideoMME|\n|:---:|:---:|:---:|:---:|\n|Vanilla|100%|124|_67.3_|\n|PruneVID* (ACL25)|24.5%|23.7|59.3|\n|FastVID|24.1%|23.2|**62.8**|\n\n***\n\n**Question 2**: The datasets are almost VideoQA, what about the performance on other long context generation datasets? Because the method proposed also prune KV Cache, will it cause a decrease in quality on long context generation(like video captioning task ActivityNet,  VideoDetailCaption e.g.)\n\n**Response**: Thank you for the insightful question. FastVID prunes redundant video tokens while preserving critical semantic information, which supports both short- and long-context generation tasks. Due to time constraints, we select ActivityNet for the video captioning task reported in Table 2. Metrics are computed using GPT-3.5-Turbo-0125. FastVID achieves **97.6%** accuracy and **98.6%** score, demonstrating its effectiveness in long-context generation.\n\n#### **Table 2: Comparisons with PyramidDrop on VideoMME and ActivityNet.**\n|LLaVA-OneVision|Retention Ratio|TFLOPs|VideoMME|ActivityNet||\n|:---:|:---:|:---:|:---:|:---:|:---:|\n||||Overall|Accuracy|Score|\n|Vanilla|100%|48.82|_58.6_|_54.4_|_3.57_|\n|PyramidDrop (CVPR25)|100%/4.1%/2.0%/1.0%|14.70|51.8|41.2|3.07|\n|FastVID|9.7%|4.04|**57.3**|**53.1**|**3.52**|\n\n***\n\n**Question 3**: Some prior works VTW[1], ST3[2], pyramiddrop[3] should be discussed in related work or compared in experiments. The FastV is a old baseline, so the recent method blow should be consider in comparison.\n\n[1] Boosting Multimodal Large Language Models with Visual Tokens Withdrawal for Rapid Inference. AAAI25\n\n[2] ST3: Accelerating Multimodal Large Language Model by Spatial-Temporal Visual Token Trimming. AAAI25\n\n[3] PyramidDrop: Accelerating Your Large Vision-Language Models via Pyramid Visual Redundancy Reduction. CVPR25\n\n**Response**: Thank you for highlighting these relevant works. In addition to FastV, we already compare against several recent SOTA baselines, including VisionZip (CVPR25), DyCoke (CVPR25), and PruneVID (ACL25). We will include VTW[1], ST3[2], and PyramidDrop[3] in the revised version of our paper.\n\nVTW removes visual tokens at a specific layer, arguing they are redundant in deep layers of MLLMs. ST3 dynamically adjust the number of visual tokens across LLM layers and generation steps. PyramidDrop performs multi-stage visual token dropping to improve efficiency. However, none of these methods consider temporal relationships across video frames, which are critical for Video LLMs.\n\nDue to time constraints, we include PyramidDrop as a baseline in Table 2. Following its official settings, we prune tokens at the 8th, 16th, and 24th layers with retention ratios of 4.1%, 2.0%, and 1.0%, respectively. Despite aggressive pruning, PyramidDrop still incurs high FLOPs due to full video token processing in the first 8 layers. In contrast, our FastVID compresses tokens from the input stage, reduces computation significantly, and outperforms PyramidDrop under extreme pruning on both VideoMME and ActivityNet."}, {"review_id": "ZPWOssTQaO", "reviewer": "Reviewer_GXf1", "summary": "This paper proposes a token pruning method called FastVID, which effectively improves the efficiency of video LLMs (LLaVA-OV and LLaVA-Video) across a wide a range of video benchmarks. Specifically, FastVID segments the video into segments and conducts token pruning within each one, achieving superior performance compared to existing methods.", "questions": "1. In Tab.2, FastVID achieves significant efficiency improvement compared to the vanilla model. I would like to know which attention version the vanilla model is using.", "limitations": "Yes.", "rating": 4, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 2, "clarity": 3, "significance": 2, "originality": 2, "strengths_and_weaknesses": "**Strengths**\n1. The paper writing is clear, and the method is easy to follow.\n2. FastVID achieves SOTA performance compared to previous methods.\n\n**Weaknesses**\n\n1. **Technical Novelty and Comparison with Prior Work:**  \n   The proposed method appears to combine existing approaches, specifically dynamic temporal segmentation, density-based token merging, and attention-based token selection. From my understanding, the concepts of dynamic temporal segmentation and density-based token merging are already central to PruneVid [1]. The main technical distinction highlighted in this paper is that PruneVid [1] uses clustering to find dynamic anchors for token merging (which introduces additional latency), whereas the current paper uses fixed anchors.  \n   However, the discussion in the related work section is brief and does not make it clear whether anchor selection is the **only** substantive difference between the proposed method and PruneVid [1] or if there are other key innovations. I would appreciate a more detailed technical comparison, clarifying (i) all methodological differences and (ii) their practical impact on efficiency and accuracy.  \n   \n   Furthermore, the attention-based token selection in this paper appears very similar to the approach used in LLaVA-Prumerge [2]. While this similarity is acknowledged in the related work section, there is no direct quantitative comparison provided with LLaVA-Prumerge [2]. A side-by-side performance evaluation would be necessary to assess the contribution and effectiveness of the proposed method relative to LLaVA-Prumerge [2].\n\n2. **Comparative Evaluation:**  \n   Other relevant token pruning methods, such as SparseVLM [3], are only mentioned briefly and not included in the experimental comparison. Incorporating these methods into the evaluation would provide a more comprehensive and convincing assessment of the strengths and weaknesses of the proposed approach.\n\nIn summary, while the Related Work section makes some connections to prior work, it does not fully address (a) the precise technical distinctions between the proposed method and PruneVid [1], nor (b) provide sufficient experimental comparison with LLaVA-Prumerge [2] and other relevant methods like SparseVLM [3]. Addressing these points would help clarify the technical novelty and practical value of the paper.\n\n[1] \"PruneVid: Visual Token Pruning for Efficient Video Large Language Models.\" arXiv preprint arXiv:2412.16117 (2024).\n\n[2] \"Llava-prumerge: Adaptive token reduction for efficient large multimodal models.\" arXiv preprint arXiv:2403.15388 (2024).\n\n[3] \"Sparsevlm: Visual token sparsification for efficient vision-language model inference.\" arXiv preprint arXiv:2410.04417 (2024).", "was_revised": true, "final_justification": "The authors have addressed my concerns. So, I will keep my positive rating.", "rebuttal": "Thank you for your positive and insightful comments, which has been instrumental in improving the overall quality of our work. In response, we have performed further experiments and analyses. All clarifications and results will be included in the revised paper.\n\n***\n\n**Question 1**: The precise technical distinctions between the proposed method and PruneVid [1]. I would appreciate a more detailed technical comparison, clarifying (i) all methodological differences and (ii) their practical impact on efficiency and accuracy.\n\n[1] PruneVid: Visual Token Pruning for Efficient Video Large Language Models. ACL25.\n\n**Response**: Thank you for the thoughtful feedback. PruneVID (ACL25) and FastVID differ fundamentally in both design and implementation. Below, we detail the key technical distinctions and their practical impact on efficiency and accuracy.\n\n1. Methodological Differences\n\t- Segmentation Strategy: PruneVID uses the original density-based clustering to partition the video into segments, with a hard upper bound on segment number ($\\leq$ sampled frames $\\times$ $\\gamma$ = 0.25). **This constraint may cause semantically diverse frames to be grouped together in complex videos**, reducing the effectiveness of subsequent pruning. In contrast, FastVID uses frame transition similarity to adaptively segment the video, **ensuring high intra-segment similarity**, thereby facilitating more efficient and accurate token pruning.\n\t - Compression Strategy: PruneVID performs clustering-based merging (Fig. 4(a) in the main paper) on both dynamic and static tokens within each segment, **without considering token representativeness or positional structure**. In contrast, our proposed density-based token merging DTM (lines 176–180) selects anchor tokens only from anchor frames and merges the remaining tokens within each segment. The number of anchor tokens per anchor frame is adaptive to segment length, rather than fixed. Importantly, **our DTM explicitly emphasizes anchor tokens located at density peaks** (lines 193-203). Positional information of anchor tokens is preserved to maintain the spatiotemporal structure, which is particularly beneficial for the Qwen-VL series that adopts M-RoPE. We further apply anchor-centric aggregation instead of simple average pooling to highlight representative anchor tokens.\n2. Practical Impact\n\t- Efficiency: In Table 3 of the main paper, we conduct a detailed efficiency comparison (lines 262–266). The primary bottleneck in efficiency lies in the density score computation. Our segmentation based on frame transition similarity is significantly faster than PruneVID’s clustering. During compression, PruneVID clusters dynamic and static tokens separately per segment. Crucially, the number of tokens varies across segments, which leads to repeated density computations. In contrast, we computes density scores in parallel across all anchor frames, requiring only a single pass. As a result, our pruning speed is **6.1×** faster (**6.1ms** vs. 37.2ms).\n\t- Accuracy: We compare PruneVID across multiple Video LLMs, including LLaVA-OneVision (Table 1 in the main paper), Qwen2-VL (Table 7 in the appendix), and Qwen2.5-VL (response to Reviewer JwZp's Q1). Under both fixed frame sampling (LLaVA-OneVision) and dynamic frame sampling (Qwen-VL series), our FastVID consistently outperforms PruneVID at the similar compression ratio, demonstrating its strong compression capability.\n\nIn summary, FastVID introduces a novel framework that integrates dynamic temporal segmentation with density spatiotemporal pruning, effectively preserving both temporal and visual context. It achieves consistent improvements in both efficiency and accuracy over prior methods across diverse Video LLMs.\n\n***\n\n**Question 2**: The attention-based token selection (ATS) in this paper appears very similar to the approach used in LLaVA-Prumerge [2]. While this similarity is acknowledged in the related work section, there is no direct quantitative comparison provided with LLaVA-Prumerge [2].\n\n[2] Llava-prumerge: Adaptive token reduction for efficient large multimodal models. ICCV25.\n\n**Response**: Thank you for the observation. As stated in line 206, our ATS is indeed inspired by LLaVA-PruMerge (which we cite as [27]). However, our FastVID is specifically designed to better preserve both temporal and visual context in Video LLMs. A direct quantitative comparison is provided in Table 1. Under the same video token budget of 608 tokens ($\\approx$ 9.7% retention ratio), our method significantly outperforms LLaVA‑PruMerge.\n\n#### **Table 1: Comparisons with LLaVA-Prumerge and SparseVLM.**\n|LLaVA-OneVision|Retention Ratio|TFLOPs|VideoMME|\n|:---:|:---:|:---:|:---:|\n|Vanilla|100%|48.82|_58.6_|\n|SparseVLM (ICML25)|100%/10.4%/5.3%/2.9%|7.06|53.3|\n|LLaVA-PruMerge (ICCV25)|9.7%|4.04|49.9|\n|FastVID|9.7%|4.04|**57.3**|\n\n***\n\n**Question 3**: Other relevant token pruning methods, such as SparseVLM [3], are only mentioned briefly and not included in the experimental comparison. \n\n[3] Sparsevlm: Visual token sparsification for efficient vision-language model inference. ICML25.\n\n**Response**: Thank you for the suggestion. We include a direct comparison with SparseVLM in Table 1. Following its official setup, we prune tokens at the 3rd, 7th, and 16th layers with retention ratios of 10.4%, 5.3%, and 2.9%, respectively. Although SparseVLM aggressively prunes later layers, it still incurs high FLOPs due to full token processing in the early layers. In contrast, FastVID compresses tokens from the input stage, leading to significantly lower computational cost. Moreover, FastVID achieves better performance due to its more effective video token pruning strategy.\n\n***\n\n**Question 4**: FastVID achieves significant efficiency improvement compared to the vanilla model. I would like to know which attention version the vanilla model is using.\n\n**Response**: Thank you for your question. In the efficiency comparison (Table 3 of the main paper), all models, including the vanilla model, use FlashAttention (version 2.7.2.post1) to ensure a fair evaluation."}]

Author rebuttal:

Meta-review: This paper introduces a dynamic token-pruning approach that exploits spatiotemporal redundancy to reduce computing cost for video LLMs. Initial reviews were on the positive side, and after rebuttal all reviewers recommended acceptance. In spite of minor concerns about some technical and experimental details, the reviewers highlighted the technical innovation, strong empirical results, and high quality presentation. The AC concurs with the reviewers' post-rebuttal assessment and recommend acceptance. The AC encourages the authors to incorporate the rebuttal materials into the camera-ready version.
Decision: Accept (poster)