HoliTom: Holistic Token Merging for Fast Video Large Language Models
Authors: Kele Shao
Keda TAO
Can Qin
Haoxuan You
Yang Sui
Huan Wang
Venue: NeurIPS 2025 poster ()
Keywords: Video LLMs
Token Compression
Video Understanding
Primary area: deep_learning
TL;DR: HoliTom introduces a training-free holistic outer-inner token merging framework for video LLMs, significantly accelerating inference with negligible performance degradation.

Abstract: Video large language models (video LLMs) excel at video comprehension but face significant computational inefficiency due to redundant video tokens. Existing token pruning methods offer solutions. However, approaches operating within the LLM (inner-LLM pruning), such as FastV, incur intrinsic computational overhead in shallow layers. In contrast, methods performing token pruning before the LLM (outer-LLM pruning) primarily address spatial redundancy within individual frames or limited temporal windows, neglecting the crucial global temporal dynamics and correlations across longer video sequences. This leads to sub-optimal spatio-temporal reduction and does not leverage video compressibility fully. Crucially, the synergistic potential and mutual influence of combining these strategies remain unexplored. To further reduce redundancy, we introduce HoliTom, a novel training-free holistic token merging framework. HoliTom employs outer-LLM pruning through global redundancy-aware temporal segmentation, followed by spatial-temporal merging to reduce visual tokens by over 90%, significantly alleviating the LLM's computational burden. Complementing this, we introduce a robust inner-LLM token similarity-based merging approach, designed for superior performance and compatibility with outer-LLM pruning. Evaluations demonstrate our method's promising efficiency-performance trade-off on LLaVA-OneVision-7B, reducing computational costs to 6.9% of FLOPs while maintaining 99.1% of the original performance. Furthermore, we achieve a 2.28× reduction in Time-To-First-Token (TTFT) and a 1.32× acceleration in decoding throughput, highlighting the practical benefits of our integrated pruning approach for efficient video LLMs inference.

HoliTom
Holistic Token Merging for Fast Video Large Language Models
Kele Shao
Keda Tao Can Qin',
Haoxuan You²,
Yang Sui
Huan Wang
1Zhejiang University Shanghai Innovation Institute
Westlake University
Salesforce AI Research $Columbia University Rice University
https: //t::///ttttbb
Holiiom
Language Response
Large Language Model
Avg. Score
Spatial Merging
Inner-LLM Merging
Tokenizer
S4
Temporal Marging
Prompt Input
LLaW-OV 7B
PruneVa (1n0224122)
<system> What's Tom doing?
62
Faxty (TCDV24)
Pliee (0VP122)
DyCoke (CVPR/25)
Halitan (wla
VilionZ (CUPR225)
Projector
Visien Encoder
Video Input
Prefiling FLOPs (T)
Figure 1: Left: We introduce HoliTom, training-fiee free holistic token merge method for fast video LLMs. Its key innovation lies in its global, redundancy-avee -awae outer-LLM spatio-temporal compres- sion and robust, token similarity-base -based inner-LLM compression. Right: The Efficieney/Perrormaacc trade-off curve of multiple training-free methods on four widely used video understanding bench- marks: MVBench, EgoSchema, Long Bide, and VideoMME. Our method, HoliTom, surpasses the SoTA approaches by maintaining 99. % average performance while reducing FL.OPs to 6.9%.
Abstract
Video large language models (video LLMs) excel at video comprehension but face significant computational inefficiency due to redundant video tokens. Existing token pruning methods offer solutions. However, approaches operating within the LLM (inner-LLM pruning), such FastV, incur intrinsic computational overhead in shallow layers, In contrast, methods performing token pruning before the LLM Couter-LLM pruning) primarily address spatial redundancy within individual frames or limited temporal windows, neglecting the crucial global temporal dynamics and correlations across longer video sequences. This leads to sub-optimal spatio- temporal reduction and does not leverage video compressibility fully. Crucially, the synergistic potential and mutual influence of integrating these strategies remain unexplored. To further reduce redundancy, we introduce HoliTom, a novel training- free holistic token merging framework. HoliTom employs outer-LLM pruning through global redundancy-awart temporal segmentation, followed by spatial- temporal merging reduce visual tokens by over 90%, significantly alleviating the LLM's computational burden. Complementing this. we introduce a robust inner- LLM token similarity- based merging approach, designed for superior performance and compatibility with outer-LI.M pruning. Evaluations demonstrate our method's promising
trade-off on V.L.VA-One Vision--- reducing computational costs to 6.9% of FLOPs while maintaining 99. % of the original performance. Furthermore, we achieve a 2.28 reduction in Time-To-First- Token (TTFT) and 1.32x acceleration in decoding throughput, highlighting the practical benefits of our integrated pruning approach for efficient video LLMs inference.
*Corresponding authors: wanghuan @ edu.cc
39th Conference on Neural Information Processing Systems (NcurIPS 2025).

---

Introduction
Video large language models (video LLMs) [25, 70, 56, 8, 30, 63, 67, 18, 51J show remarkable potential in understanding complex video content. However, their practical deployment is hindered by significant computational inefficiency. This inefficiency stems from processing high volumes of video tokens generated by encoding sampled frames, leading to substantial overhead, particularly due to the quadratic complexity of the attention mechanism in the LLMs. For videos with numerous frames, the input token count can easily reach tens of thousands, making inference computationally expensive. While prior works [6, 65, 61, 48, 19, 34, 36] have explored model compression and token pruning, achieving desirable balance between efficiency and performance in video tasks remains an open challenge. Thus, developing effective methods to reduce video token redundancy while preserving critical semantic information is crucial for the widespread adoption of video LLMs.
Token pruning is a promising direction. These ap- proaches generally fall into two categories depending on where pruning occurs. Inner-LLM pruning meth- ods, such as V [6]. TopV [64], and PDrop [61], operate within the LLM layers. However, they in- cur intrinsic computational and memory costs in the initial layers before pruning takes effect, limiting overall FLOPs reduction. Outer-LLM pruning meth- ods process tokens before the main LLM compu- tation. Some methods address spatial redundancy (VisionZip [65], PruMerge [42]), others tackle tem- poral aspects within limited temporal windows (Dy- Coke [48]. Prune Vid [21]), thus preventing a global understanding of video dynamics and comprehensive spatio-temporal optimization. Furthermore, despite the potential for synergy, prior work has systemat- ically explored integrating inner-LLM and outer-LLM pruning strategies or analyzed their mutual benefits. The current methods, while offering some benefits, still leave room for improvement.
Compression scope of vision- language model acceleration methods. This table outlines where different methods apply compression. Spatial and Temporal refer to compression of the input visual data, while Inner-LLM indicates compression mecha- nisms applied within the model's 's processing.
Table 1:
Methods
Spatial
Temporal
Inner LLM
FastV
PDrop [61]
LLaVA-PruMerge [42]
VisionZip [65]
DyCoke [48]
VID
To address these limitations, we propose a holistic token pruning for video L.MSs that leverages external and internal strategies. Our method first tackles temporal redundancy through global redundancy-awaan video segmentation process, followed by spatio-temporal merging. This external step reduces visual tokens less than 10%, significantly alleviating the computational burden on the subsequent LLM. Complementing this, we introduce a new and robust inner-LLM token similarity- based merging method, specifically designed for integration with outer-LLM pruning method, enabling mutual benefits. This integrated strategy offers a more holistic and efficient solution to handle long videos with LLMs, as summarized in Tab. which contrasts the compression achieved by our approach in both the ppatio-temporaa domain and within the inner-LLM against other methods.
Empiriaal evaluations validate the effectiveness of our proposed method in achieving compelling effiiincy--peffommance performance trade-off. Specifically, as shown Fig. (right), on the VA VA-nee Vision- 7B model [25], our approach reduces computational costs to just 6.9% of the original FLOPSs while remarkably preserving 99.1% of the original model's s performance. Moreover, we observe significant gains in inference efficiency, achieving 2.28x x reduction in
Fi-FFiittTTken (TTFT) and 1.32x x acceleration in decoding throughput. These results clearly demonstrate the substantial practical advantages of our holistic token merging framework for efficient video LLM inference.
Our key contributions are summarized as follows:
1. We analyze the phenomenon of temporal redundancy in the context of video LLMs and propose global redundancy- aware temporal merging method to effectively address the inefficiency in video LLMs before LLM processing in a and--and-pla fashion.
2. We introduce a robust inner-LLM similarity-based merging technique specifically designed for integration with the outer-LLMM pruning method, facilitating synergistic optimization.
3. Extensive evaluations on L.aa -AaVV-One VA-One and -LdVooVideo demonstrate that our integrated pruning framework achieves a state-of-the-art reducing computational costs and accelerating inference while preserving model performance.
perrernny-- trade-off, significantly

---

2 Related Work
2.1 Video Large Language Models
The rapid progress of multimodal large language models has led to the integration of video encoders, creating video LLMs that excel in video understanding and question answering tasks [63, 17, 25, 2, 56, 3, 27, 28, 30, 67, 47, 50, 23, 1, 46, 29]. However, the substantial number of tokens generated by processing numerous video frames hinders inference efficiency, thereby impeding the widespread adoption of video LLMs. Existing approaches have attempted to mitigate this issue. For instance, VideoLLaMA [67] employs Q-Former module [26] to aggregate video tokens, while MovieChat [47] introduces a memory module to merge and store token representations. Although pooling mechanisms in LLaVA-One Vision [25] reduce token counts, each video frame still produces hundreds of tokens for downstream processing. Consequently, handling tens of thousands of visual tokens for long video inputs substantially increases inference time and memory consumption. While works such as VILA [32] and NVILA [35] aim to optimize token usage, these methods often require model fine-tuning, demanding considerable hardware resources [22, 30, 32, 35, 55, 33]. This underscores a critical need for developing more efficient, training-free token compression methods specifically for video LLMs, bypassing the need for costly model adaptations and significant hardware investment.
2.2 Visual Token Compression
Token compression [43] has emerged as an effective strategy for reducing token redundancy in vision transformers and large language models. ToMe [4] merges similar tokens in ViTs alleviate spatial redundancy, while TempMe [45] focuses on minimizing temporal redundancy by merging adjacent video clips. TESTA [40] achieves up 75% reduction in processed tokens by employing temporal and spatial aggregation modules. For MILLMs, FastV [6] prunes non-essential visual tokens in early layers of LLM. TopV [64] proposes an optimization framework prune unnecessary visual tokens. DyMU [57] introduces token merging in the visual encoder and virtual unmerging in the LLM decoder. PDrop [61] performs progressive pruning of tokens at different stages within the LLM, LLaVA-PruMerge [42] and VisionZip [65] leverage attention weight analysis in visual encoders to eliminate spatial redundancy. However, the inherent temporal dependencies between video frames necessitate specialized compression designs. Consequently, recent methods specifically for video token compression have gained increasing attention. DyCoke [48] consolidates tokens across frames and implements dynamic key-value cache reduction. PruneVID [21] clusters video tokens, whereas FastVID [44] enhances compression by combining temporal segmentation with spatio-temporal token merging. In this paper, we propose new token merging strategy specifically designed for video LLMs, which fully considers spatio-temporal characteristics to maximize performance retention.
3 Method
3.1
Background on Video LLMs Inference
The inference process of video LLMs involves three key stages: before LLM, prefilling, and decoding.
(1) Before LLM. Given an input video with B frames, a vision encoder processes each frame to produce N,, embedding vectors. These are projected into the text embedding space, yielding visual
tokens H., E Bd, where d represents the dimension of the hidden state space. A text prompt T= {t,) { is tokenized and embedded into text tokens H, E NR,, xd similarly. Finally, the visual and text tokens are concatenated to form H concat H,, Hy), which serves as the LLM input.
(2) Prefilling Stage. During prefilling, each transformer layer of the LLM performs self-attention compute query Q' = HW q. key K' HW and value V = HWW, where W W W/ E Rdxa are learnable projection matrices. The resulting key-value pairs (K' and V are then cached (KV cache) to enhance the efficiency of token generation during the subsequent decoding phase.
operations on the concatenated input H. It begins with linear transformations
leveraging the KV cache. At each time step I, only the new token hy is processed to compute its key and value representations, avoiding recalculating attention weights over the entire history. The KV cache is updated by appending the new key-value pairs: K < kK,,,W. mechanism substantially reduces the computational complexity of the generation process.
(3) Decoding Stage. The decoding stage generates tokens autoregressively.
<- [V, Wy). This caching
3

---

Stage Spatial Merging
Stage IlI: Inner-LLM Merging
TextTokend
Transformer Block K+1
Vision Tokens
Marged Tokens
Non- Non-Topk Attnenioo Tekens
Clattenbaned Hargin
000
Topk Attantion Tokers
Aturioo-buaed Seace
Transformer Block
Nerged Tokeess
Stage I: Temporal Merging
Large Language Model
Pruned Tickens
Temporal Merged Tokens
Toien One Frame
Spatial Marging
Tokenizer
Pruned Taken of One Frame
Tedundert
Temporal Marging
Prompt Input
Nenhhatenteet
ssystemo What Tem doing?
Temporal Redundancy Merging
Tokens in Frame 1-N
rojector
Redundancy--neee Aware Segmentatio
Vision Encoder Input
Figure 2: Overview of our HoliTom method. HoliTom compresses video LLMs across three scopes; the first two are outer-LLM pruning. Temporal Merging maximizes temporal compression via global redundancy-awar aware segmentation, merging similar tokens into their first occurrence. Spatial Merging further reduces redundancy by applying tailored spatial compression based on the characteristics of remaining temporal variations. Inner-L.LM Merging leverages attention within the L.M identify key tokens and merges less important, similar tokens, streamlining information within the LLM.
3.2 Global Redundancy-Awar Temporal Merging
Temporal redundancy describes feature persistence at fixed spatial locations across consecutive frames. We identify this redundancy for the k-th feature between frames and m + 1 using their respective feature vectors lm,,k and 1.+1 A feature considered temporally redundant if its normalized similarity, sim(hm. hmm+ I,R), exceeds defined threshold E [0,
For temporal segment defined by a start frame t, and an end frame te (covering [t,, te))., the total number of prunable tokens, g(t,,ta t,), is calculated. This involves counting tokens (t,,, te) that are consecutively redundant across all frames from to to te and then multiplying by the number of subsequent frames (te t, 1) within the segment where these tokens can be pruned. Our method prunes the redundant by merging these subsequent occurrence tokens into their first appearance start frame f,, treating them temporal redundant tokens, as shown in Fig. 2. The formulation is:
No 1
((t,,te)) =
Σ
(ssim(hm 1m > T) MEt
II
(te - 1),
N(ta.ta)
where N. is the total number of features per frame, and II(-) the indicator function.
Given video of B frames, objective is to find segmentation into K consecutive segments [f,, (44) (with f = 1, tk+1 = 1, and tú < 1 that maximizes the total prunable features:
max K.
Σ gta.ttt1).
(2)
This optimization solved using dynamic programming achieve global optimization. Let dp[i] be the maximum prunable features for video ending at frame (exclusive, i.c., considering frames 1...., i 1), where frame marks the exclusive end of the segment. The value prev stores the optimal starting frame j" of this final segment [j", i). The state transition is given by:
dpli] max [dpli] g(. +)), with prev|i] arg magmax(dppy + o))
(3)
15j<s
The base case is dp [] 0. The maximum prunable features for the entire video are dp + 1]. The optimal segmentation is reconstructed by backtracking from B + 1 using the prev array.

---

3.3 Spatial Merging
After temporal merging, tokens are classified as non-redundant or redundant temporal tokens. We first process the former. Inspired by works [42, 65, 69, 52], we utilize the CLS tokens for spatial feature selection. For vision encoders like Siglip [66] that do not have an explicit CLS token, method is detailed to derive CI.-eqqivallen attention. Specifically, we compute the attention matrix:
A Softmax(Qk" E BBNNN....
(4)
where d is the state dimension. Token importance is quantified by averaging the attention weights each token receives from all other tokens within the same frame in the vision tower, yielding a score vector Anv E BxNN Tokens receiving higher average attention are considered more salient.
Consistent with video LLMs of applying spatial pooling (c.g., after the projector to reduce tokens), we reshape Aug to its original spatial grid dimensions (H x W N.) and apply analogous pooling operation. This results in a spatially downsampled importance map Aavg E IR Bxxx Ultimately, we select the visual features corresponding to the highest scores in Aavg as the representative and most informative spatial tokens, known as iitention-based select, discarding all others.
The computation of vision tower attention is intra- frrmm. frames lacks theoretical justification, invalidating the Averaging these attention weights across method for redundant temporal tokens. To process these features, we employ a cluster-based merging method utilizing density peak clustering based k-nearest neighbors (DPC-KNN) [13, 41]. Given set of N redundant temporal tokens [un, u2,....NN' within the first frame of the segmentation. For each token U; we calculate its local density Pis distance the closest higher-density token 6, and the final density score 74 ps x 8,;
max d(v,, Uy) 17
if P: max Pk:
Pi exp
Σ ()ENN)()))
d(us, 03)3
5,
(5)
K
min d(v,, Uz)
otherwise
Tokens with high "/j selected cluster centers. After selecting the cluster centers, each remaining feature is assigned to the cluster whose center is closest in feature space. Finally, the representative feature for each cluster is then computed by averaging the features assigned to it. Ultimately, the compressed features, derived from this clustering process, along with the non-redundant features, are concatenated according to their original spatial order, thereby preserving positional characteristics.
3.4 Inner-LLM Merging
Inefficient visual attention in large vision language models has been widely discussed [6, 61]. Existing methods, such Fast' [6] and PDrop [61], directly discard redundant visual tokens, which may lead to performance degradation due to information loss. Unlike these approaches, our proposed method addresses it by merging the information from potentially redundant tokens instead of simply discarding them. Specifically, at the K-th layer of the LLM,, to reduce the number of visual tokens by R% we employ a token selection strategy based on attention scores. We use the attention weights of the last token to rank all vision tokens at layer K. The R% of visual tokens exhibiting the lowest attention scores are identified as candidates for merging. We find its most similar visual token within the set of tokens designated for retention. For retained token U, and its associated set of low attention tokens Vn {Ump Umg m.. Um,, ). The updated retained token v/. is:
average(Er: Uma Dm..)
(6)
This selective merging preserves relevant features from tokens that would otherwise be removed, mitigating information loss while achieving the desired token reduction.
Experimental Results
4.1 Experimental Settings
Benchmarks. We evaluate our method on four widely-used video understanding benchmarks: MVBench [28], EgoSchema [37], Beng [59], and VideoMME [17]. Comprising videos of varying lengths and complex scenarios, these benchmarks provide comprehensive testbed for assessing the effecttivnness and generalization of our method.
5

---

Table 2: Comparison state-of-the methods across benchmarks, Best and most efficient results in bold, second best underlined. Here, "HoliTom" means the full version of our method; "HoliTom (w/o M)" means our method without inner-LLM merging, for reference.
Method
Prefilling FLOPs (T)
Refore L.M
MVRench
EgoSchema
LongVideo Bench
VideoMME
Avg.
Ratio
Retained Ratio
Score
%
LLaVA-O--7B
40.8
100%
100%
58.3
60.4
56.4
58.4
100
FastV [6]
9.3
22.8%
100%
55.9
57.5
56.7
56.1
56.5
96.7
PDrop [61]
10.5
25.7%
100%
56.1
58.0
54.1
56.2
96.2
DyCoke [48]
8.7
21.3%
25%
53.1
59.5
49.5
54.3
54.1
92.6
VisionZip [65]
8.7
21.3%
57.9
60.3
56.5
58.2
58.2
99.7
PruneVid [2]]
8.7
21.7%
25%
57.4
59.9
55.7
57.4
57.6
98.6
[44]
8.7
HoliTom (w/o M)
8.7
21.3%
25%
58.5
60.8
56.5
59.1
58.7
100.5
HoliTom
58.8
100.7
VisionZip [65]
7.0
17.2%
20%
57.7
59,8
55,2
$7.9
57.7
98.8
PruncVid [21]
7.0
17.2%
20%
57.2
59.7
54.7
56,9
57.1
97.8
FastVID [44]
7.0
17.2%
20%
56.3
57.1
57.9
HoliTom (w/o M)
7.0
17.2%
58.5
60.7
56.3
58.5
100.2
HoliTom
5.8
14.2%
20%
58.7
61.0
57.1
58.6
58.8
100.7
VisionZip [65]
5.2
12.7%
15%
56.5
59.8
54.4
56.1
56.7
97.1
PruneVid [21]
5.2
12.7%
15%
56.8
59.7
55.4
56.6
57.1
97.8
FastVDD [44]
5.2
12.7%
56.0
56.2
57.7
HoliTom (w/o MD
5.2
12.7%
15%
58.1
61,0
57.0
58.1
58.5
100.2
HoliTom
4.3
10.5%
58.1
61.2
56.4
57.3
58.2
99.7
VisionZip [65]
3,4
8.3%
10%
53.5
$8.0
49.3
53.4
53.5
91.6
PruncVid [21]
3.4
8.3%
10%
56.2
59,8
54.5
56.0
56.6
96.9
FasVVD [44]
3,4
8.3%
10%
55.9
56.3
57.3
HoliTom (w/o M)
3.4
8.3%
10%
56.9
61.1
56.5
56.9
57.8
99.0
HoliToom
2.8
6.9%
10%
57.3
61,2
56.3
56.8
57,9
99.1
Compared Methods. We compare our proposed HoliTom against strong training-free baselines: 1) FastV [6]. identifies key tokens during prefilling using attention scores between predicted and vision tokens; PDrop [61], prunes visual tokens within partitioned LLM stages, guided by image
and instruction tokens; 3) Visionzip [65], prunes tokens before LLM via spatial token merging: 4)
DyCoke [48], employs temporal merging before LLM and dynamic KV cache pruning in decoding; 5) Vid [21], minimizes video redundancy via spatio-temporal token clustering; and 6) FastVID [44], a concurrent work, partitions videos and applies density-based token pruning. Due to lack of public code, compare FastVID to its reported results. For all other baselines and our method, experiments use their open-source code under identical hardware condition.
Inference Cost Evaluation. We evaluate the inference cost of transformer layers, each composed of multi-head attention (MHA) and feed-forward network (FFN) modules. Following previous work [6, 61, 48], the FLOPs for processing vision tokens in layer i, with hidden state size d and FFN intermediate size m, are defined as 4n,d² + 2 d + 2n,dm. For an LLM with T transformer layers, the total FLOPs span the prefilling and decoding phases, calculated as:
T (n + 2n2d 2m,dm) R((Ad + 2dm) + 2(nn, d( (1)))
1
Prefilling FLOPs per
Decoding FLOPs per layer
For consistency, the decoding calculation is fixed for predicting R 100 tokens, accounting for the the KV cache. In video LLMs, the decoding phase FLOPs contribute only approximately 2% of the total. Consequently, our primary optimization focus is on the prefilling stage. When considering prefilling optimization, Lnner-LMM pruning methods like FastV [6]. the FLOPs incurred the first 2 shallow layers can amount to 2.9 TFLOPs in LLaVA-One Visionn-- Even pruning 100% token in the layer, these methods cannot match the potential efficiency compared to outer-LLM pruning methods. Thus, outer-LLM pruning offers more impactful optimization approach for this domain.
Implementation Details. Our method is implemented on LLAVA-Video-77 [70] models. Evaluation uses NVIDIA 1 GPUs, while inference is on an RTX A6000. Inference cost measured by prefilling FLOPs, with baselines configured for comparable FLOPs (details in the appendix A). The default is 0.8; for 10% compression, 7 is 0.65. Following official practice, L.L.VA-OneVe Vision models utilize 32 input video frames (N, 196), while LLaVA- Video uses 64 frames (N, 169). All benchmarks are conducted using LMMs-EW Eval [68, 24].
Visioonn
[25] and

---

Table 3: Cross-backbone method comparison. Performance comparison of our method against
state-of of-the-art art methods across different backbones, demonstrating consistent effectiveness.
Model
Method
Prefilling FLOPs (T)
Ratio I.
FLOPs
Retained Ratio
Before LLM
MVBench
EgoSchema
Long Video
Bench 1
VideoMME
Score
Avg.
%
Vanilla
429.3
100%
100%
60.9
61.1
62.7
65.7
62.6
100
FastV
61.2
92.5
LLaVA-
Vissouzip [65]
59.0
13.7%
15%
58.4
59.3
57.4
63.8
59.7
95.4
OneVision-P22 728
PruneVid [21]
59.0
13.7%
15%
56.8
57.7
57.
62.7
58.6
93.6
HolfTom (wie M)
59.0
13.7%
15%
58.5
60.0
57.8
64.1
60.1
96.0
HolfTom
51.6
12.0%
15%
58.7
60.1
57.2
64.3
96.0
100%
58.9
100
FadtY [6]
17.1
21.3%
1005
54.3
54.1
55.0
58.8
92.4
LLaVA- Video-7B
PDrop [6]]]
19.5
24.3%
10015
55.9
$4.3
54.7
61.9
56.7
94.2
VisionZip [65]
9.3
11.6%
15%
56.7
54.7
54.7
60.7
56.2
94.2
HoliTom (wol M)
9.3
11.6%
15%
57.8
54.8
55.6
61.9
95.5
57.2
56.2
95.8
Average Soore
58
Hollfom
V
$6.1
67.9
57.7
67.7
57.2
56.6
FastV Holifom
60
58.9
66.6
Avenge
Vanilla VisionZip
55.7
55.7
54.5
56
55.2
PruncVid
HoliTom M)
64
2
14
18
21
0.3
0.4
0.5
0.8
0.7
16
32
64
128
Number of Layers (K)
Praning Rate (F)
Number of Frames
Figure 3: Left: Performance of our method vs. FastVV when pruning various layers rate R=50%. Right: Performance comparison with varying pruning rates at a fixed layer (K=14).
Figure 4: Performance vs. number of frames for method and other token compression methods.
4.2 Main Results
Comparison with State-of-the-Art Methods. Tab. benchmarks Holitom against state-of-the-art approaches on the LLaVA-OneVision-1 model, analyzing performance and inference cost (FLOPs) various token retention ratios (25%, 20%. 15%, and 10%) prior to LLM processing. Inner-LLM pruning methods, such as FastV [6] and PDrop [61]. often struggle to balance performance and efficiency, especially at lower token retention ratios (25%). DyCoke [48]. which segments video frames into groups of 4 and prunes all but the first frame, is limited by its design, capping its lowest retention ratio 25%. Spatial pruning methods like VisionZip [65] show significant performance drop (up to 8.4%) at 10% retention. This decline stems from relying solely on spatial compression, less effective at preserving crucial temporal information needed for performance under aggressive pruning. Crucially, even without our inner- LLM merging technique, our method achieves of the-art performance and efficiency consistently across the evaluated retention ratios. This highlights the superior robustness and adaptability of our approach compared to prior methods. Our inner-LLM merging method further enhances efficiency, driving optimization further. For instance, we retain only 6.9% of the original FLOPs, while preserving 99.1% of the bascline performance.
Performance Comparison Across Different Backbones. Tab. 3 assesses our method's performance model, sensitive to aggressive
across various backbones. For the powerful compression, our approach reduces computational cost to 11.3% keeping 96% of its original perfor- mance. LLaVA- VA-Video-7B presents a greater compression challenge due to its higher initial pooling rate (169 vs. 196 tokens/frame in LA One Vision). Despite this, our method achieves a reduction to just 9.5% of the original FLOPS, retaining 95.8% performance and outperforming existing methods. Overall, achieving significant token compression with minimal performance drop is indeed tougher for LLL Visionnn- and LLaVA-Video-TB than for L.L
Video-7B
HoliTom vs. FastV under Outer-LLM Compression. Building on the challenges faced by inner- LLM pruning methods discussed in Section 4.1, we compare our inner-LLM merging method with FastV, specifically scenarios where Luter-LLM compression already applied. In this compressed context, the property "an image is worth 1/2 tokens after layer 2" [6] is not consistently observed. This is because outer-LLM compression concentrates information, making trivial token discarding more difficult using attention mechanisms. As illustrated in Fig. 3, our method demonstrates superior performance compared to FastV when pruning 50% at shallower layers. Furthermore, at equivalent layers, our approach consistently surpasses FastV across wide range of pruning rates, underscoring its effectiveness. This effectiveness stems from our inner-LLM merging method, which better preserves information rather than directly discarding it.
7

---

Tower
Decoding Throughput
Score
FastV
500
45
372
50.8
56.5 (-1.9)
PDrop
45
365
36.7
(-2.)
VislonZip
500
45
271
37.5
57.7 (+0.7)
PruneVid
500
75
214
38.2
57.1 (-1.3)
HoliTom M)
175
38.8
(-0.8
HoliTam
500
155
2.28*
57.9 (-0.5)
Full Tokens
500
4.
251.5
1,322
58.4
250
760
1000
1500
28
40
Token breskdown (ms)
Throughput of decoding stage (tokens/san)
Figure 5: Achieving superior inference. "Other" indicates token pre-processing time (e.g.. pooling). Our proposed method reduces Mime--T--Frr----- decoding throughput, outperforming all other token compression methods and the vanilla model.
(TTFT) by 2.28 and achieves 1.32x higher
Table
Ablation study on merging modules. Our temporal merging module reduces FLOPs to 75.7% without performance loss, alleviates the performance degradation caused by aggressive spatial pruning. The integration of all 3 modules achieves the best performance--fficieney trade-off.
Method
Prefilling FLOP: (T)
FLOPs
Before LLM
MVBench
EgoSchema
LongVideo Bench f
VideoMME
Avg.
Ratio
Retained Ratio
1
Score
%
Vanilla
40.8
100%
100%
58.3
60.4
56.4
58.4
100
Only Temporal
30.9
75.7%
79%
58.9
60.5
56.5
59.1
58.8
100.7
Only Spatial
5.2
12.7%
15%
57.9
60.8
54.2
56.8
57.4
98.3
HoliTom (w/o M)
5.2
12.7%
15%
58.1
61.0
57.0
58.1
58.5
100.2
HoliTom
4,3
10.5%
15%
58.1
61.2
56.4
57.3
58.2
99.7
Performance Scaling with more frames. Our method scales performance robustly with increasing input frames (Fig. 4). A challenge for video LLMs is that uniformly sampled frames may miss crucial information required for accurate answers. Therefore, an effective token pruning method is essential process more frames and capture sufficient context. Fig. 4 shows our approach consistently outperforms other compression methods across frame rates. At 16 frames, where less temporal redundancy exists, our method, while slightly below the vanilla, still outperforms all other compression techniques. With 64 frames, our method is more efficient and achieves superior performance over the vanilla model. Furthermore, when processing 128 frames, our token compression approach avoids the maximum context length limitations that bottleneck vanilla models. This capability is particularly beneficial for tasks that require an extensive temporal context or to answer complex questions with long text, resulting in improved performance.
Discussion: Improved Performance after Token Compression Tabs. 2, 4, and Fig. 4 present key finding: models employing our token compression technique outperform the original models on various benchmarks. This surprising result underscores a fundamental principle for achieving superior performance at the input stage: the value of key information over exhaustive information. Excessive, irrelevant, or redundant data acts as noise, obscuring essential signals critical for effective processing. This information overload impedes the capacity of the model to accurately identify and process critical details, thereby degrading understanding and response generation. By providing a refined input that retains pertinent information while shedding redundant information, our compression method facilitates deeper comprehension and yields more accurate, relevant outputs. Collectively, these results underscore the efficacy of our technique in distilling key information and demonstrate that intelligent input refinement is crucial for superior model performance.
4.3 Efficiency Results
Fig. 5 summarizes the impact of various token compression methods on the inference efficiency of video LLMs. As shown, all the evaluated methods demonstrably reduce LLM pretilling time. Our method, in particular, reduces it to just 13.7% of the original. For VisionZip [65], Prune [21], and our HoliTom require token pre-processing, which introduces additional "other" time. Furthermore, both Prune and our method produce variable number of tokens per frame, complicating batch processing, which contributes to extra overhead. Our method, designed to maximize temporal redun- dancy pruning. leads to finer-graine segmentation, further influencing this observation. Nevertheless, our method achieves the maximum reduction Time-To-Fine Cio-FFirs-TTooee Token latency, reducing it by 2.28 x, while maintaining optimal performance. Although we did not specifically optimize for decoding, our model's 's decoding speed still benefits from the reduced number of vision tokens. Our throughput increased by 1.32x compared to the original model, the highest among all methods evaluated.
8

---

ench
EgoSchema
LongVidesBend: ench
VidecMME
Court
20-
Mean: 43.0%
100
Meen: 9.3%
20
Meen: 19.18
50
Neen: 10.5%
20
10
Ratio
Figure 6: Histogram of temporal pruning rates across four benchmarks 0.80). The average pruning ratio for each benchmark is annotated in the top right. MVBench (16s duration) exhibits the highest ratio, reflecting greater temporal redundancy, while EgoSchema the least.
Table 5: Ablation study on video segmentation methods. This table compares different video segmentation strategies: Fixed-interval segmentation partitions the video at equal inter- vals; DySeg adaptively segments based on transition similarity; and our proposed global aedundancy-awart segmentation.
LongMideeBencr
0.5
0.6
0.7
0.8
0.9
Methods
MVBench
EgoSehoma
LongVideo Bench
VidcoMME
Avg.
Figure 7: Ablation study on Per- formance of our method analyzed with varying T at target before
Fixed interval
$7.0
60.9
53.8
56.4
$7.0
DySeg [44]
56.8
60.8
$4.1
56.6
$7.1
HoliTom (wlo M)
56.9
61.1
56.5
56.9
57.8
LLM retained ratio of 15%.
4.4
Ablation Study
Ablation study merging modules. Tab. 4 provides detailed ablation study on the contribution of our proposed merging modules. We first evaluated the temporal merging module (r = 0.8), designed to eliminate temporal redundancy, which demonstrated efficiency gains while preserving performance. Across the four benchmarks, our method achieved 100.7% the baseline performance while reducing FLOPs to 75.7%. Note that the reported average pruning rate is calculated over four datasets. The average pruning rate varied across datasets illustrated in Fig. 6. For instance, MVBench(1666 with its shortest duration, exhibits the highest temporal redundancy, allowing approximately 43% pruning, whereas EgoSchema contains the least, permitting only about 9.3%. We then investigate combining temporal with spatial pruning. Applying our temporal pruning method significantly mitigates the performance degradation typically associated with aggressive spatial pruning alone. Furthermore, incorporating the inner merging module allowed us push the efficiency boundaries even further, ultimately retaining 99.7% performance with a mere 10.5% of the original FLOPS.
Ablation study on temporal segmentation method. Tab. 5 compares different temporal segmenta- tion methods. Fixed-int eixeedinterval Segmentation generates 8 segments with an interval of 4. DySeg [44] selects segment start points using the 8 largest inter-frame differences and includes frames below 0.90 similarity threshold. Our proposed global redundancy-aweer segmentation maximally leverages spatial redundancy and achieves better performance.
Ablation study on т. The hyperparameter т controls the sensitivity of the temporal pruning mecha- nism, with lower values leading more aggressive pruning. For fixed retained ratio, 7 also governs amount of spatial and temporal pruning applied. Fig. 7, demonstrates the easy 0.8. This value is adopted uniformly
the balance between
tunability of 7, with peak performance observed around without performance degradation, as shown in Tab. 4. For a 10% pruning target, we set 7 = 0.65 to mitigate performance degradation from aggressive spatial pruning.
5 Conclusion
This paper presents HoliTom, a new training-free holistic token merging framework for boosting the efficiency of video LLMs by effectively handling redundant visual tokens. HoliTom achieves this through synergistic integration of outer-LLM mattio-tmmporl reduction, drastically reducing initial token counts, and robust inner-LLM token merging mechanism tailored for compatibility and further optimization. Evaluated on prominent video LLMs, HoliTom achieves state-of-the-an trade-off, substantially reducing computational costs (e.g., to 6.9% FLOPs) while preserving high performance (e.g., 99.1 1 1% accuracy), and accelerating inference (2.28x TTFT, 1.32x throughput). These results underscore the effectiveness HoliTom in enabling practical and efficient video LLMs inference for complex, form video understanding.

---

Acknowledgment
This paper supported by Young Scientists Fund of the National Natural Science Foundation of China (No. 62506305), Zhejiang Leading Innovative and Entrepreneur Team Introduction Program (No. 2024R01007), Key Research and Development Program of Zhejiang Province (No. 2025C01026), Scientific Research Project of Westlake University (No. WU2025WF0033 Chinese Association for Artificial Intelligence (CAAI) & Ant Group Research Fund AGI Track (No. 2025CAAI-ANT-13), and the Special Support Talents Program of "Xi Hu Ming Zhu Program" in Hangzhou.
References
[1] Kirolos Ataallah, Xiaoqian Shen, Eslam Abdelrahman, Essam Sleiman, Deyao Zhu, Jian Ding, and Mohamed Elhoseiny. Minigpt4-video: Advancing multimodal Ilms for video understanding with interleaved visual-textual tokens. arXiv preprint urXiv:2404.03415 2024.
[2] Jinze Bai, Shuai Bai, Shusheng Yang, Shijie Wang, Sinan Tan, Peng Wang, Junyang Lin, Chang Zhou, and Jingren Zhou. Qwen-vl: A versatile vision- language model for understanding, localization, text reading, and beyond. arXiv preprins arXiv:2308. /2966, 2023.
[3] Shuai Bai, Kegin Chen, Xuejing Liu, Jialin Wang, Wenbin Ge. Sibo Song. Kai Dang, Peng Wang, Shijie Wang. Jun Tang, Humen Zhong, Yuanzhi Zhu, Mingkun Yang, Zhaohai Li, Jianqiang Wan, Pengfei Wang, Wei Ding, Zheren Fu, Yiheng Xu, Jiabo Ye, Xi Zhang, Tianbao Xie, Zesen Cheng, Hang Zhang, Zhibo Yang, Haiyang Xu, and Junyang Lin. Qwen2.5-vl technical report. arXiv preprint arXiv:2502. / 13923, 2025.
[4] Daniel Bolya, Cheng-Yane Fu, Xiaoliang Dai, Peizhao Zhang, Christoph Feichtenhofer, and Judy Hoffman. Token merging: Your vit but faster. ICLR, 2023.
[5] Joya Chen, Zhaoyang Lv, Shiwei Wu, Kevin Qinghong Lin, Chenan Song, Difei Gao, Jia- Wei Liu, Ziteng Gao, Dongxing Mao, and Mike Zheng Shou. Videollm-online: Online video large language model for streaming video. In CVPR, 2024.
[6] Liang Chen, Haozhe Zhao, Tianyu Liu, Shuai Bai, Junyang Lin, Chang Zhou, and Baobao Chang. An image is worth 1/2 tokens after layer 2: Plug-and-play inference acceleration for large vision-language models. In ECCV, 2024.
[7] Xueyi Chen, Keda Tao, Kele Shao, and Huan Wang. Streamingtom: Streaming token compres- sion for efficient video understanding. arXiv preprint arXiv:25100 00261............. 2025.
[8] Zhe Chen, Weiyun Wang, Yue Cao, Yangzhou Liu, Zhangwei Gao, Erfei Cui, Jinguo Zhu, Shen- glong Ye, Hao Tian, Zhaoyang Liu, et al. Expanding performance boundaries of open- open-source multimodal models with model, data, and test-time scaling, arXiv preprint 21Xiv:241..05271, 2024.
[9] Zesen Cheng, Sicong Leng, Hang Zhang, Yifei Xin, Xin Li, Guanzheng Chen, Yongxin Zhu, Wenqi Zhang, Ziyang Luo, Deli Zhao, et al. Videollama 2: Advancing spatial-temporal modeling and audio understanding in video-llms. arXiv preprint arXiv:2406.07477 074766 2024.
[10] Rohan Choudhury, Guanglei Zhu, Sihan Liu, Koichiro Niiumaa, Kris Kitani, and László Jeni. Don't look twice: Faster video transformers with run-length tokenization. NeurIPS, 2024.
[11] Tri Dao. PlashAttentionn2:: Faster attention with better parallelism and work partitioning. In ICLR, 2024.
[12] Tri Dao, Daniel Y. Fu, Stefano Ermon, Atri Rudra, and Christopher Ré. FlashAttention Fast and memory-
exact attention with IO-awareness. In NeurlPS, 2022.
[13] Mingjing Du, Shifei Ding, and Hongjie Jia. Study on density peaks clustering based on k-nearest neighbors and principal component analysis. Knowledge-Bused Systems, 99: 19915,5445, 2016.
[14] Sicheng Feng. Kaiwen Tuo, Song Wang, Lingdong Kong, Jianke Zhu, and Huan Wang. Re- wardmap: Tackling sparse rewards fine-grained visual reasoning via multi-stage reinforcement learning. arXiv preprint arXiv:2510.0222
2025.
[15] Sicheng Feng, Song Wang, Shuyi Ouyang, Lingdong Kong, Zikai Song, Jianke Zhu, Huan Wang, and Xinchao Wang. Can milms guide me home? a benchmark study on fine-grained 2025.
visual reasoning from transit maps. arXiv preprint
10

---

[16] Elias Frantar, Salch Ashkboos, Torsten Hoeferr, and Dan Alistarh. GPTQ: Accurate test-training compression for generative pretrained transformers. ICLR, 2023.
[17] Chaoyou Fu, Yuhan Dai, Yongdong Luo, Lei Li, Shuhuai Ren, Renrui Zhang, Zihan Wang, Chenyu Zhou, Yunhang Shen, Mengdan Zhang, et al. Video-mme: The first-ever comprehensive evaluation benchmark of multi-modal Ilms in video analysis. arXiv preprint arXiv:2405.21075, 2024.
[18] Zhangwei Gao, Zhe Chen, Erfei Cui, Yiming Ren, Weiyun Wang. Jinguo Zhu, Hao Tian, Shenglong Ye, Junjun He, Xizhou Zhu, et al. Mini-internvl: a flexible-transfer pocket multi- modal model with 5% parameters and 90% performance. Visual Intelligence, 2(1):32, 2024.
[19] Yuhang Han, Xuyang Liu, Zihan Zhang, Pengxiang Ding, Junjie Chen, Donglin Wang, Hong- gang Chen, Qingsen Yan, and Siteng Huang. Filter. correlate, compress: Training-free token reduction for mllm acceleration. arXiv preprint arXiv:2411.17688 2024.
[20] Wei Huang. Xingyu Zheng, Xudong Ma, Haotong Qin, Chengtao Lv, Hong Chen, Jie Luo, Xiaojuan Qi Xianglong Liu, and Michele Magno. An empirical study of Ilama3 quantization: From ilms to milms. Visual Intelligence, 2(1):36, 2024.
[21]
Xiaohu Huang, Hao Zhou, and Kai Han. Prunevid: Visual token pruning for efficient video large language models. arXiv preprint vrXiv:2412.16111 2024.
[22] Peng Jin, Ryuichi Takanobu, Wancai Zhang, Xiaochun Cao, and Li Yuan. Chat-univi: Unified visual representation empowers large language models with image and video understanding. In CVPR, 2024.
[23] Yang Jin, Zhicheng Sun, Kun Xu, Liwei Chen, Hao Jiang, Quzhe Huang, Chengru Song, Yuliang Liu, Di Zhang, Yang Song, et Video-lavit: Unified video-language pre-training with decoupled visual-motional tokenization. In ICML, 2024.
[24] Bo Li, Peiyuan Zhang, Kaichen Zhang, Fanyi Pu, Xinrun Du, Yuhao Dong, Haotian Liu, Yuanhan Zhang, Ge Zhang, Chunyuan Li, and Ziwei Liu. Lmms-eval: Accelerating the development of large multimoal models, 2024.
[25] Bo Li, Yuanhan Zhang, Dong Guo, Renrui Zhang, Feng Li, Hao Zhang, Kaichen Zhang, Peiyuan Zhang, Yanwei Li, Ziwei Liu, et al. Llava-onevision: Easy visual task transfer. arXiv preprint arXiv:2408.0m 03326. 2024.
[26] Junnan Li, Dongxu Li, Silvio Savarese, and Steven Hoi. Blip-2: Bootstrapping language-image pre-training with frozen image encoders and large language models. In ICML, 2023.
[27] KunChang Li, Yinan He, Yi Wang, Yizhuo Li, Wenhai Wang, Ping Luo, Yali Wang, Limin Wang. and Yu Qiao. Videochat: Chat-centric video understanding. arXiv preprint arXiv:2305 063555 2023.
[28] Kunchang Li, Yali Wang, Yinan Hc, Yizhuo Li, Yi Wang, Yi Liu, Zun Wang. Jilan Xu, Guo Chen, Ping Luo, et al. Mybench: A comprehensive multi-modal video understanding benchmark. In CVPR, 2024.
[29] Xinhao Li, Yi Wang, Jiashuo Yu, Xiangyu Zeng, Yuhan Zhu, Haian Huang, Jianfei Gao, Kunchang Li, Yinan He, Chenting Wang, et al. Videochat-flash Hierarchical compression for long-context video modeling. arXiv preprint arXiv:2501.0060 00574, 2024.
[30] Yanwei Li, Chengyao Wang, and Jiaya Jia. Llama-vid: An image is worth 2 tokens in large language models. In ECCV, 2024.
[31] Ji Lin, Jiaming Tang, Haotian Tang, Shang Yang. Wei-Ming Chen, Wei-Chen Wang, Guangxuan Xiao, Xingyu Dang, Chuang Gan, and Song Han. Awq: Activation weight quantization for on-device I1m compression and acceleration. In MLSys, 2024.
[32] Ji Lin, Hongxu Yin, Wei Ping, Pavlo Molchanov, Mohammad Shoeybi, and Song Han. Vila: On pre-training for visual language models. In CVPR, 2024.
[33] Xuyang Liu, Yiyu Wang, Junpeng Ma. and Linfeng Zhang. Video compression comman- der: Plug-and-play inference acceleration for video large language models. arXiv preprint arXiv:2505.1444 2025.
[34] Xuyang Liu, Ziming Wang, Junjie Chen, Yuhang Han, Yingyao Wang, Jiale Yuan, Jun Song, Linfeng Zhang, Siteng Huang, and Honggang Chen. Global compression commander: Plug-and- play inference acceleration for high-resolution large vasion-l models. arXiv preprint 200.02250. 2025.
11

---

[35] Zhijian Liu, Ligeng Zhu, Baifeng Shi, Zhuoyang Zhang. Yuming Lou, Shang Yang, Haocheng Xi, Shiyi Cao. Yuxian Gu, Dacheng Li, et al. Nvila: Efficient frontier visual language models. arXiv preprini arXiv:24/2.04440 2024.
[36] Yongdong Luo. Wang Chen. Xiawu Zheng, Weizhong Huang, Shukang Yin, Haojia Lin, Chaoyou Fu, Jinfa Huang, Jiayi Ji. Jiebo Luo, et al. Quota: Query-oriented token assignment via cot query decouple for long video comprehension. arXiv preprint (5tv:2533.088899 2025.
[37] Karttikeya Mangalam, Raiymbek Akshulakov. and Jitendra Malik. Egoschema: A diagnostic benchmark for very long-form video language understanding. In NeurIPS, 2023.
(38) Rui Qian, Shuangrui Ding, Xiaoyi Dong, Pan Zhang, Yuhang Zang, Yuhang Cao, Dahua Lin, and Jiaqi Wang. Dispider: Enabling video Iims with active real-time interaction via disentangled perception, decision, and reaction. In CVPR, 2025.
[39] Rui Qian, Xiaoyi Dong, Pan Zhang, Yuhang Zang, Shuangrui Ding, Dahua Lin, and Jiaqi Wang. Streaming long video understanding with large language models. In NeurIPS, 2024.
[40] Shuhuai Ren, Sishuo Chen, Shicheng Li, Xu Sun, and Lu Hou. Testa: Temporal-spatial token aggregation for long-form video-language understanding. In EMNLP, 2023.
[41] Alex Rodriguez and Alessandro Laio. Clustering by fast search and find of density peaks. science, 344(6191): 149199)):944))111 499-1196, 2014.
[42] Yuzhang Shang. Mu Cai, Bingxin Xu, Yong Jae Lee, and Yan Yan. Llava-prumerge: Adaptive token reduction for efficient large multimodal models. arXiv preprint urXiv:2403.15338 2024.
[43] Kele Shao, Keda Tao, Kejia Zhang, Sicheng Feng, Mu Cai, Yuzhang Shang, Haoxuan You, Can Qin, Yang Sui, and Huan Wang. When tokens talk too much: A survey of multimodal long-context token compression across images, videos, and audios. arXiv, 2025.
[44] Leqi Shen, Guoqiang Gong, Tao He, Yifeng Zhang, Pengzhang Liu, Sicheng Zhao, and Guiguang Ding. Fastvid: Dynamic density pruning for fast video large language models. arXiv preprini arXiv:2503.1 11177, 2025.
[45] Leqi Shen, Tianxiang Hao. Sicheng Zhao, Yifeng Zhang, Pengzhang Liu, Yongjun Bao, and Guiguang Ding. Tempme: Video temporal token merging for efficient text-video retrieval. In ICLR, 2025.
[46] Yan Shu, Zheng Liu, Peitian Zhang, Minghao Qin, Junjie Zhou, Zhengyang Liang, Tiejun Huang, and Bo Zhao. Video-xl: Extra-long vision language model for hour-scale video understanding. In CVPR, 2025.
[47] Enxin Song, Wenhao Chai, Guanhong Wang, Yucheng Zhang, Haoyang Zhou, Feiyang Wu, Haozhe Chi, Xun Guo, Tian Ye, Yanting Zhang, al. Movicchat: From dense token to sparse memory for long video understanding. In CVPR, 2024.
[48] Keda Tao, Can Qin, Haoxuan You, Yang Sui, and Huan Wang. Dycoke: Dynamic compression of tokens for fast video large language models. In CVPR, 2025.
[49] Keda Tao, Kele Shao, Bohan Yu, Weigiang Wang, Huan Wang, et al. Omnizip: Audio- guided dynamic token compression for fast omnimodal large language models. arXiv preprint arXiv:2511.14550 2025.
[50] Keda Tao, Haoxuan You, Yang Sui, Can Qin, and Huan Wang. Plug-and-play 1. x-bit kv cache quantization for video large language models. arXiv preprint 2rXiv:2503.16255: 2025.
[51] Xiaoguang Tu, Zhi He, Yi Huang, Zhi-Hao Zhang, Ming Yang, and Jian Zhao. An overview of large ai models and their applications. Visual Intelligence, 2(1):34, 2024.
[52] Ao Wang, Fengyuan Sun, Hui Chen, Zijia Lin, Jungong Han, and Guiguang Ding, [cls] token tells everything needed for training- efficient milms. arXiv preprint arX1v:2412.05819, 2024.
[53] Haicheng Wang, Zhemeng Yu, Gabriele Spadaro, Chen Ju, Victor Quétu, Shuai Xiao, and Enzo Tartaglione. Folder: Accelerating multi-modal large language models with enhanced performance. arXiv preprint arXiv:2501.024533 2025.
[54] Hanzhen Wang, Jiaming Xu, Jiayi Pan, Yongkang Zhou, and Guohao Dai. Specprune-vla: Accelerating linion-languagg-action models via action-aware self-speculative pruning. arXiv preprint 2509.02500905111 2025.
12

---

[55] Jiawei Wang, Liping Yuan, Yuchen Zhang, and Haomiao Sun. Tarsier: Recipes for training and evaluating large video description models. arXiv preprint arXiv:2407.000644 2024.
[56] Peng Wang. Shuai Bai, Sinan Tan, Shijie Wang, Zhihao Fan, Jinze Bai, Keqin Chen, Xuejing Liu, Jialin Wang, Wenbin Ge, Yang Fan, Kai Dang, Mengfei Du, Xuancheng Ren, Rui Men, Dayiheng Liu, Chang Zhou, Jingren Zhou, and Junyang Lin. Qwen2-vl: Enhancing vision- language model's perception of the world at any resolution. arXiv preprint arXiv:2409.12191, 2024.
[57] Zhenhailong Wang, Senthil Purushwalkam, Caiming Xiong, Silvio Savarese, Heng Ji, and Ran Xu. Dymu: Dynamic merging and virtual unmerging for efficient vlms. arXiv preprint a50iv:2504. 170000 2025.
(58) Meng Wei, Chenyang Wan, Xiqian Yu, Tai Wang, Yuqiang Yang, Xiaohan Mao, Chenming Zhu, Wenzhe Cai, Hanqing Wang, Yilun Chen, et al. Streamvin: Streaming iisionandd-anguage navigation via slowfast context modeling. arXiv preprint arXivv2507.055240 2025.
[59] Haoning Wu, Dongxu Li, Bei Chen, and Junnan Li. Longvideobench: A benchmark for long-context interleaved video-language understanding. NeurIPS, 2024.
[60] Guangxuan Xiao, Ji Lin, Mickael Seznec, Hao Wu, Julien Demouth, and Song Han. Smoothquant: Accurate and efficient post-training quantization for large language models. In ICML, 2023.
[61] Long Xing, Qidong Huang, Xiaoyi Dong, Jiajie Lu, Pan Zhang, Yuhang Zang, Yuhang Cao, Conghui He, Jiaqi Wang. Feng Wu, et al. Pyramiddrop: Accelerating your large vision-language models via pyramid visual redundancy reduction. In CVPR, 2025.
[62] Jiaming Xu, Jiayi Pan, Yongkang Zhou, Siming Chen, Jinhao Li, Yaoxiu Lian, Junyi Wu, and Guohao Dai. Specee: Accelerating large language model inference with speculative early exiting. In ISCA, 2025.
[63] Lin Xu, Yilin Zhao, Daquan Zhou, Zhijie Lin, See Kiong Ng, and Jiashi Feng. Pllava: Parameter-free llava extension from images to videos for video dense captioning. arXiv preprint arXiv:2404. 16994, 2024.
[64] Cheng Yang, Yang Sui, Jinqi Xiao, Lingyi Huang, Yu Gong, Chendi Li, Jinghua Yan, Yu Bai, Ponnuswamy Sadayappan, Xia Hu, al. Topv: Compatible token pruning with inference time optimization for fast and low-memory multimodal vision language model. In CVPR, 2025.
[65] Senqiao Yang, Yukang Chen, Zhuotao Tian, Chengyao Wang, Jingyao Li, Bei Yu, and Jiaya Jia. Visionzip: Longer is better but not necessary in vision language models. In CVPR, 2025.
[66] Xiaohua Zhai, Basil Mustafa, Alexander Kolesnikov, and Lucas Beyer. Sigmoid loss for language image pre-training In ICCV, 2023.
[67] Hang Zhang, Xin Li, and Lidong Bing. Video-Ilama: An instruction-tunel audio- visual language model for video understanding. In EMNLP, 2023.
[68] Kaichen Zhang, Bo Li, Peiyuan Zhang, Fanyi Pu, Joshua Adrian Cahyono, Kairui Hu, Shuai Liu, Yuanhan Zhang, Jingkang Yang. Chunyuan Li, and Ziwei Liu. Lmms-eval: Reality check on the evaluation of large multimodal models, 2024.
[69] Qizhe Zhang, Aosong Cheng, Ming Lu, Zhiyong Zhuo, Mingi Wang. Jiajun Cao, Shaobo Guo, Qi She, and Shanghang Zhang. [cls] attention is all you need for training-free visual token pruning: Make vim inference faster. arXiv preprint 12.
2024.
[70] Yuanhan Zhang, Jinming Wu, Wei Li, Bo Li, Zejun Ma, Ziwei Liu, and Chunyuan Li. Video instruction tuning with synthetic data. arXiv preprint 10.00224 2024.
13

---

NeurIPS Paper Checklist
1. Claims
Question: Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?
Answer: [Yes]
The abstract and introduction clearly outline the problem, propose a new solution combining external and internal pruning, and list key contributions that align with the methods and goals described in the text.
Justification:
Guidelines:
The answer NA means that the abstract and introduction do not include the claims made the paper.
The abstract and/or introduction should clearly state the claims made, including the contributions made in the paper and important assumptions and limitations. A No or
NA answer to this question will not be perceived well by the reviewers.
The claims made should match theoretical and experimental results, and reflect how much the results be expected to generalize other settings.
is fine include aspirational goals as motivation long is clear that these goals are not attained by the paper.
2. Limitations
Question:
Does the paper discuss the limitations of the work performed by the authors?
Answer: [Yes]
Justification: Appendix details the limitations our current method and outlines future research directions.
Guidelines:
The answer NA means that the paper has limitation while the answer No means that the paper has limitations, but those are not discussed in the paper.
The authors are encouraged to create separate "Limitations" section in their paper.
The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (e.g., independence assumptions, noiseless settings, model well-specification, asymptotic approximations only holding locally). The authors should reflect on how these assumptions might be violated in practice and what the implications would be.
The authors should reflect the scope of the claims made, e.g., if the approach was only tested on a few datasets or with few runs. In general, empirical results often depend on implicit assumptions, which should be articulated.
The authors should reflect the factors that influence the performance of the approach. For example, facial recognition algorithm may perform poorly when image resolution low or images are taken in low lighting. Or speech-to-text system might not be used reliably to provide closed captions for online lectures because it fails to handle technical jargon.
The authors should discuss the computational efficiency the proposed algorithms and how they scale with dataset size.
If applicable, the authors should discuss possible limitations of their approach to address problems of privacy and fairness.
While the authors might fear that complete honesty about limitations might used by reviewers as grounds for rejection, worse outcome might be that reviewers discover limitations that aren't 't acknowledged in the paper. The authors should use their best judgment and recognize that individual actions in favor of transparency play impor- tant role in developing norms that preserve the integrity of the community. Reviewers will be specifically instructed to not penalize honesty concerning limitations.
3. Theory assumptions and prool's
Question: For each theoretical result, does the paper provide the full of assumptions and complete (and correct) proof?
14

---

Answer:
[NA]
Justification:
No formal theorem or proposition is provided in the main manuscript.
Guidelines:
The answer NA means that the paper does not include theoretical results.
All the theorems, formulas, and proofs in the paper should be numbered and cross- referenced.
All assumptions should be clearly stated or referenced in the statement of any theorems.
The proofs can either appear in the main paper or the supplemental material, but if they appear in the supplemental material, the authors are encouraged to provide a short proof sketch provide intuition.
Inversely, any informal proof provided in the core of the paper should be complemented by formal proofs provided in appendix or supplemental material.
Theorems and Lemmas that the proof relies upon should be properly referenced.
4. Experimental result reproducibility
Does the paper fully disclose all the information needed to reproduce the main ex- perimental results of the paper to the extent that it affects the main claims and/or conclusions of the paper (regardless of whether the code and data are provided or not)?
Question:
Answer: [Yes]
Justification: We have made our best efforts to ensure the reproducibility of our method. The algorithmic details are provided Section 3, and the experimental setup and procedures are detailed in Section 4.1 and Appendix A.
Guidelines:
The answer NA means that the paper does not include experiments. If the paper includes experiments, a No answer to this question will not be perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data are provided or not.
If the contribution is dataset and/or model, the authors should describe the steps taken to make their results reproducible or verifiable.
Depending on the contribution, reproducibility can be accomplished in various ways. For example, if the contribution is novel architecture, describing the architecture fully might suffice, or if the contribution a specific model and empirical evaluation, it may be necessary to either make it possible for others to replicate the model with the same dataset, or provide access to the model. In general. releasing code and data is often one good way accomplish this, but reproducibility instructions for how to replicate the results, access to a hosted model (e.g.. in the case of large language model), releasing of a model checkpoint, other means that appropriate to the research performed.
also be provided via detailed
While NeurIPS does not require releasing code, the conference does require all submis- sions to provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution. For example
(a) If the contribution is primarily a new algorithm, the paper should make it clear how to reproduce that algorithm.
(b) the contribution is primarily new model architecture, the paper should describe the architecture clearly and fully.
(c) If the contribution is a new model (e.g., a large language model), then there should either be way to access this model for reproducing the results or way to reproduce the model (e.g., with an open-source dataset or instructions for how to construct the dataset).
(d) We recognize that reprodueibility may be tricky in some cases, in which authors are welcome to describe the particular way they provide for reproducibility. In the case of closed-source models, it may be that access to the model is limited in some way (e.g., to registered users), but it should be possible for other researchers have some path reproducing or verifying the results.
5. Open access to data and code
15

---

Question: Does the paper provide open access to the data and code, with sufficient instruc- tions faithfully reproduce the main experimental results, as described in supplemental material?
Answer: [No]
Justification: The datasets are from publicly available online sources. We guarantee the code will be open-sourced after the review period.
Guidelines:
The answer NA means that paper does not include experiments requiring code.
Please see the NeurIPS code and data submission guidelines (https://nips cc/
for more details.
While we encourage the release of code and data, we understand that this might be possible, so "No" is an acceptable answer. Papers cannot be rejected simply for not including code, unless this is central to the contribution (e.g., for new open-source benchmark).
The instructions should contain the exact command and environment needed to to reproduce the results. See the NeurlPS code and data submission guidelines (https:
// //nips. wwtpcccppppppppeeeeececcccc ppwppc/gguddessseepppppppppopale gwtgg/uiiee///// (CodeSubmissiee wwwiiissssspoiiioiiioi for more details.
The authors should provide instructions on data access and preparation, including how to access the raw data, preprocessed data, intermediate data, and generated data, etc.
The authors should provide scripts to reproduce all experimental results for the new proposed method and baselines. only subset of experiments are reproducible, they should state which ones omitted from the script and why.
At submission time, to preserve anonymity, the authors should release anonymized versions (if applicable).
Providing as much information as possible in supplemental material (appended to the paper) recommended, but including URLs to data and code permitted.
6. Experimental setting/details
Question: Does the paper specify all the training and test details (e.g., data splits, hyper- parameters, how they were chosen, type of optimizer, etc.) necessary to understand the results?
Answer: [Yes]
Justification: The experimental details, including the model structure, hyperparameters, and comparison methods, have been elaborated in detail in Section 4 and Appendix A.
Guidelines:
The answer NA means that the paper does not include experiments.
The experimental setting should be presented in the core of the paper to a level of detail that is necessary appreciate the results and make sense of them.
The full details can provided either with the code, appendix, or as supplemental material.
7. Experiment statistical significance
Question: Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?
Answer: [No]
Justification: For evaluation, we did not report error bars, consistent with previous work in this area. Due to the computational cost, cannot replicate the experiments multiple times. This said, we have evaluated our method on multiple datasets and models (see Tab. 2, 3) to confirm the performance robustness of method.
Guidelines:
The answer NA means that the paper does not include experiments.
The authors should answer "Yes" if the results are accompanied by emoor bars, confi- dence intervals, or statistical significance tests, least for the experiments that support the main claims of the paper.
16

---

The factors of variability that the error bars are capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, or overall run with given experimental conditions).
The method for calculating the error bars should be explained (closed form formula, call to a library function, bootstrap, etc.)
The assumptions made should be given (e.g., Normally distributed errors).
It should be clear whether the error bar is the standard deviation or the standard error of the mean.
is OK to report 1-sigma error bars, but one should state it. The authors should preferably report a 2-sigma error bar than state that they have 96% CI, if the hypothesis of Normality of errors is not verified.
For asymmetric distributions, the authors should be careful not to show in tables or figures symmetric error bars that would yield results that are out range (e.g. negative еπτог rates).
If error bars are reported in tables or plots, The authors should explain in the text how they were calculated and reference the corresponding figures or tables text.
8. Experiments compute resources
For each experiment, does the paper provide sufficient information on the com- puter resources (type of compute workers, memory, time of execution) needed to reproduce the experiments?
Question:
Answer: [Yes]
Justification: In Section 4.1, we have detailed the computational resources required for our evaluation and inference tests.
Guidelines:
The answer NA means that the paper does not include experiments.
The paper should indicate the type of compute workers CPU GPU, internal cluster, or cloud provider, including relevant memory and storage.
The paper should provide the amount of compute required for each of the individual experimental runs well estimate the total compute.
The paper should disclose whether the full research project required more compute than the experiments reported in the paper (e.g., preliminary or failed experiments that NeurIPS Code of Ethics https:/ //tepr///////u..../// cc wwtpppcccc////ccccoccccccctotltt http:////hhhistueeeeeeeeeeeeeeee
didn't make into the paper).
9. Code of ethics
Question: Does the research conducted in the paper conform, in every respect, with the
Answer:
[Yes]
Justification:
We believe have followed the Code of Ethics.
Guidelines:
The answer NA means that the authors have not reviewed the NeurlPS Code of Ethics.
If the authors answer No, they should explain the special circumstances that require deviation from the Code of Ethics.
The authors should make sure to preserve anonymity (e.g., there a special consid- eration due to laws regulations in their jurisdiction).
10. Broader impacts
Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?
Answer: [Yes]
Justification: As shown in appendix E, the societal impact of our method has been fully discussed.
Guidelines:
The answer NA means that there is no societal impact of the work performed.
17

---

If the authors answer NA or No, they should explain why their work has no societal impact or why the paper does not address societal impact.
Examples of negative societal impacts include potential malicious or unintended uses (e.g., disinformation, generating fake profiles, surveillance), fairness considerations (e.g., deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations.
The conference expects that many papers will be foundational research and not tied to particular applications, let alone deployments. However, if there is direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point out that generic algorithm for optimizing neural networks could enable people to train models that generate Deepfakes faster.
The authors should consider possible harms that could arise when the technology is being used as intended and functioning correctly, harms that could arise when the technology is being used as intended but gives incorrect results, and harms following from (intentional or unintentional) misuse of the technology.
If there are negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition attacks, mechanisms for monitoring misuse, mechanisms to monitor how system learns from feedback over time, improving the efficiency and accessibility of ML).
11. Safeguards
Does the paper describe safeguards that have been put in place for responsible release of data models that have high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)?
Question:
Answer: [NA]
Justification:
Our method is training-free free and plug-and aung--and-ly,, which means this is no risks.
Guidelines:
The answer NA means that the paper poses no such risks.
Released models that have high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere to usage guidelines or restrictions to access the model or implementing safety filters.
Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images.
We recognize that providing effective safeguards challenging, and many papers do not require this, but we encourage authors to take this into account and make best faith effort.
12. Licenses for existing assets
Are the creators or original owners of assets (e.g.. code, data, models), used in the paper, properly credited and the license and terms of use explicitly mentioned and properly respected?
Question:
Answer: [Yes]
Justification:
We have cited all datasets and baseline models used in our experiment.
Guidelines:
The answer NA means that the paper does not use existing assets.
The authors should cite the original paper that produced the code package dataset.
The authors should state which version of the asset used and, possible, include URI.
The name of the license (e.g., CC-BY 4.0) should be included for each asset.
For scraped data from a particular source (e.g., website), the copyright and terms of service of that source should be provided.
18

---

If assets are released, the license, copyright information, and terms of use in the package should be provided. For popular datasets, paperswi thcode. com/dataseta has curated licenses for some datasets. Their licensing guide can help determine the license of dataset.
For existing datasets that are re-packaged, both the original license and the license of the derived asset (if it has changed) should be provided. If this information is available online, the authors encouraged to reach out the asset's s creators.
13. New assets
Question: Are assets introduced in the paper well documented and is the documentation provided alongside the assets?
Answer: [NA]
Justification: Our work does not provide assets.
Guidelines:
The answer NA means that the paper does not release new assets.
Researchers should communicate the details of the dataset/code/model as part of their submissions via structured templates. This includes details about training, license, limitations, etc.
The paper should discuss whether and how consent was obtained from people whose asset used.
At submission time, remember to anonymize your assets (if applicable). You can either create anonymized URL or include an anonymized zip file.
14. Crowdsourcing and research with human subjects
Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given to participants and screenshots, if applicable, as well as details about compensation (if any)?
Answer: [NA]
Justification: Our work does not involve any crowd sourcing experiments with human subjects.
Guidelines:
The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.
Including this information in the supplemental material is fine, but if the main contribu- tion of the paper involves human subjects, then much detail as possible should be included the main paper.
According to the NeurlPS Code of Ethics, workers involved in data collection, curation, other labor should be paid at least the minimum wage in the country of the data collector.
15. Institutional review board (IRB) approvals or equivalent for research with human subjects
Question: Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or an equivalent approval/review based on the requirements of your country or institution) were obtained?
Answer: [NA]
Justification: Our work does not involve experiments with human subjects.
Guidelines:
The answer NA means that the paper does involve crowdsourcing nor research with human subjects.
Depending the country in which research is conducted, IRB approval (or equivalent) may be required for any human subjects research. If you obtained IRB approval, you should clearly state this in the paper.
19

---

We recognize that the procedures for this may vary significantly between institutions and locations, and we expect authors adhere the NeurIPS Code of Ethics and the guidelines for their institution.
For initial submissions, do not include any information that would break anonymity (if applicable), such as the institution conducting the review.
16. Declaration of LLM usage
Does the paper describe the usage of LLMs if it is an important, original, or non-standard component of the core methods in this research? Note that if the LLM is used only for writing, editing, formatting purposes and does not impact the core methodology, scientific rigorousness, or originality of the research, declaration is not required.
Question:
Answer: [NA]
Justification: The method development in this research does not involve LLMs as any important, original, or non-standard components.
Guidelines:
The answer NA means that the core method development in this research does not involve LLMs as any important, original, non-standard components.
Please refer to our LLM policy pectp::/neeurpge http://p//.//oooeeeeee////////// for what should or should not be described.
20

---

Table Comparison state-of-the-art methods on Qwen2.5-VL 7B. Qwen2.5-V VL accepts video input at 2 fps, capped at maximum of 768 frames, maximum video token limit to 24,576 (as the technical report recommends). A/B in the Token column indicates that tokens are first provided to the LLM,, and then compressed to B tokens during the LLM forward pass. The token number is derived from the average video token count per task within VideOMME. FastV performs full attention matrix calculation in memory, causing OOM errors due to the large number of video tokens.
VideoMME
Method
# Token 4
TFLOPs 4
Short
Medium
Long
Overall
Owen2.5-VI.-7B
18442
100%
377.2
100%
77.4
68.1
55.6
67.0
100%
FastV [6]
18442 / 1921
100% / 50.0%
170.4
45.2%
OOM
Visionzip [65]
9221
50.0%
154.5
41.0%
74.9
66.6
55.7
65.7
98.1%
PruneVid [21]
9173
49.7%
153.5
40.7%
72.3
64.8
54.7
63.9
95.4%
HoliTom (wlo M)
6513
34.9%
102.0
27.0%
74.4
66.4
56.4
65.8
98.2%
V [6]
18442 / 4610
100% / 25.0%
90.7
24.0%
OOM
Visionzip [65]
4610
25.0%
68.7
18.2%
73.1
63.3
55.9
64.1
95.7%
Prune Vid [21]
4632
25.1%
69.1
18.3%
69.3
61.1
53.2
61.2
91.3%
HoliTom (w/o M)
4504
24.4%
66.9
17.7% %
72.7
65.7
56.1
64.8
96.7%
A Supplemental Implementation Details
Our method is implemented on the models. Evaluation utilized NVIDIA A100 (80GB) GPUs; inference was performed on an NVIDIA RTX A6000 GPU. To ensure fair comparison of computational cost, we used total prefilling FLOPs as the primary metric. Baselines are configured for comparable FLOPs: Fast [6] prunes 80% of tokens at layer 2; PDrop [61] retains 50%, 25%, and 12.5% of vision tokens at layers 2, 7, and 14, respectively; VisionZip [65] and Prune Vid [21] maintain a consistent proportion of input tokens with our method. Performance results for Fast [44] are adopted directly from their original paper. For our proposed method, the default threshold r is 0.8. In the specific experiments conducted on Qwen2.5-VL [3] with maximum sampling of 768 frames, a lower threshold of T = 0.2 was used. This adjustment accounts for the higher temporal redundancy present when sampling is dense. In experiments targeting a 10% compression ratio, т was set to 0.65. Experimental setups include pruning K=18 layers of the 7B model and K=60 layers of the 72B model, both at ratio of R=50%. Following the official VA-OOne Vision specifications, the default input video frames are 32 and N. 196. For LLaVA- Video, the default input consisted of 64 video frames with N. 169. All benchmark evaluations are performed using the L.MM--Eall [68, 24].
[25] and LLaVA-Video-71 [70]
B Supplemental Experimental Results
B.1 Experiments on Qwen2.5-VL with High Frame Sampling
Existing models like LLaVA-OV [25] and LLaVA-Video [70] utilize fixed input of 32/64 video frames, each resized to static resolution (Tab. 3). In contrast, frontier models, such as Qwen2.5- VL [3], introduce advanced features including FPS frame sampling, which extend input sequences (up to 768 frames), and dynamic resolution support. These new capabilities pose new challenges to existing token compression methods in maintaining performance for video understanding tasks.
As shown in Tab. 6, HoliTom surpasses ofate-of-the-art methods across both token compression rates, especially for long videos. Due to Out-of- (OOM) issues arising from the full attention matrix calculation, we were unable to report results for FastV and the inner-LLM merging execution.
B.2 Impact of Token Compression on Fine-Grained Object Understanding
HoliTom applies aggressive video token compression. Does this aggressive compression impair the model's s ability to comprehend fine-grained details? Tab. presents the performance results on selected subtasks from the MVBench benchmark (originally detailed in Table 2).
21

---

Table 7: Fine-grained object tasks. Performance on MVBench object subtasks (object existence (OE), object interaction (OI), and object shuffle (OS)) im- proves with more aggressive token compression.
Table 8: Improved performance with hanced efficiency. Using more imput frames, token compression boosts performance while controlling computational overhead.
MVBench OS
Method
Frame
FLOPs (%)
Avg. Score
Method
FLOPs (%)
OE
O1
Overall
32
100
58.4
OV 7B
100
57.5
84.0
35.5
58.3
TooliTom
32
12.7
58.5
+HoliTom
17.4
61.0
83.5
36.5
58.4
+HoliTom
64
26.5
58.9
+HoliTom
14.2
63.0
84.0
37.5
58.7
TooliTom
128
56.6
59.3
+HoliTom
10.5
60.5
84.5
38.0
58.1
EgoSchema
LangVidcoBench
56,2%
Mean: 20.6%
Meam: 26.5%
Nean: 17.8%
Countt
20
20
0.2
0.4
0.6
0.8
0.2
0.4
0.6
0.2
0.4
0.6
0.8
0.2
0.4
0.6
Frune Ratio
Frune Rabio
Prune Ratio
Pruna Ratio
Figure 8: Histogram of temporal pruning rates across four benchmarks (T = 0.65). The average pruming ratio for each benchmark is annotated in the top right. MVBench (16s duration) exhibits the highest ratio, reflecting greater temporal redundancy, while VideoMME is the least (r = 0.65).
At compression rates ranging from 10% to 25%, HoliTom demonstrates increases performance. This result would be improbable if HoliTom discards critical information about small objects. Instead, this outcome provides evidence of HoliTom's 's robust ability to retain fine-grained details.
B.3 Enhanced Performance with Reduced Overhead
Tab. 8 extends the findings presented in Fig. 4. By sampling more frames with HoliTom while maintaining constant or even reduced total FLOPs, we achieve better performance compared to a vanilla model operating on fewer frames. We also observe that as the number of input frames increases, the computational overhead contributed by the vision encoder becomes a nog-neglig factor. This performance and efficiency trade-off is further illustrated in Fig. 5 of our paper.
B.4 Supplemental Ablation Study on T
LangVideoBench
Avg
In section 4.4, we discussed the selection (7 0.8) and the corresponding histogram of temporal pruning rates on four benchmarks for retain ratio of 15%. Next, we detail the selection of the hyperpa- rameter T for 10% retain ratio and present the cor- responding histogram. As illustrated in Fig. 9, peak performance observed around 5 0.65. The Fig. 8 presents the histogram of temporal pruning rates on the four datasets when controlling r regulates the aggressiveness of temporal pruning; a larger 7 results in more aggressive pruning.
57-
56.5
LongWoederd
58-
57.7
57.8
58
55.4
54.8
54.8
54
57
0.50
0.60
0.65
0.70
0.80
0.65. It is evident that
Figure 9: Ablation study on Performance of our method is analyzed with varying 7 a target before LLM retained ratio of 10%.
B.5 Ablation Study Merge Strategy
The core of apatio-tempont merging in HoliTom is a hybrid strategy that integrates attention attention-guided compression with similarity-guit guided clustering (DPC-KNN). As shown in Tab. our mixed strategy achieves the best performance. The key to this lies in distinguishing between the two token types encountered during the merging process. For non-redundant tokens (within a single frame), the encoder's s self-attention scores are excellent priors, making attention-guided compression strategy highly effective. However, for redundant tokens (formed by merging tokens from adjacent frames), the original fnale-framc self-attention scores lack theoretical basis as merging metric. Therefore,
22

---

Table 9: Ablation Study on Merge Strategy. Our mixed strategy achieves the best performance.
Method
MVBench
EgoSchema
Long BingeeBBBen
VideoMME
Avg. Score
Attention 58.4
60.9
55.9
57.4
58.1
DPC-KNN
57.4
59.6
53.9
56.6
56.9
HoliTom
58.4
60.9
56.2
58.3
58.5
DPC-KNN clustering is adopted to group these merged tokens based on feature similarity, which provides a more principled and effective approach in this specific spatio-temporal context.
C Compatible with Flash Attention
Our approach introduces two distinct merging strategies: inner-LLM and outer-LLM. The inner-LLM strategy, similar to prior work [6, 61, 48], is designed for integration with highly optimized attention implementations (e.g., Flash Attention [12, 11).. This requires obtaining attention scores from specific layer only once during the prefilling stage, an operation introducing negligible computational overhead compared to total inference cost. In contrast, our outer-LLM merging strategy operates externally to the model, decoupled from the attention mechanisms of LLM.
D Limitations and Future Work
While work demonstrates adequate acceleration of video LLMs by token merging for inference, it is important to outline its current limitations. First, the approach is primarily designed for fixed- length video clips and does not natively support online, arbitrary-length streaming video input. This poses challenges for real-time processing [5, 39, 38] and maintaining long-term context understanding. Second, as shown in Fig. 5, similar other methods [6, 61, 65, 48] in the token pruning area, our approach does not optimize the latency of the vision tower. Further work, such quantization [31, 60, 16, 20], methods accelerate the vision tower [10, 53, 57] and new application areas [58, 54, 62, 49, 7, 15, 14], is worth exploring for further optimization.
E Broader impacts
This work significantly enhances video LLM efficiency, addressing key barrier to deployment and scalability, By reducing computational needs, it broadens access to advanced video AI, enabling wider application and fostering innovation.
More Visualizations
23

---

chlor object statonary?
green moving
LLaVA-OV-78
40.8T
The object ts stationary purple.
The green ball towards rightside of frame. The green ball moves towards the right of the frame. moving towards the right side of frame. The green ball towards left side of frame.
FastV
9.3T
The object ts stationery is purple.
VisionZio
8.7T
The object that stationry green.
The green
HoliTom (w/
8.7T
The object ih stationary purple.
HoliTom
7.1T
The object th stationary purple.
The green ball moves towards the left side of the frame.
User:
FLOPS
Where the hidden object the end the occlusion game from the camera's point of view?
LLaVA-OV-78
40.8T
notebook. The hidden object under the black notebook. The hidden object is under the black notebook cup.
object
FastV
9.3T
Top:
Dynamic Object Tracking
VisionZip
8.7T
HoliTom M)
object The hidden object under the middle cup.
Memory; Motion Understandlng
HoliTom
7.1T
Figure 10: Comparison Challenging Video Understanding. Green: correct results, Red: incorrect results. Our method is able to produce correct answers on challenging video tasks.
User
Describe whats happaning this video.
LLaVA-OV-7B
The video features standing
black shirtand dark pocket for? appears bottom screen, indicating curiosity about the purpose of speaific pocket the person's clothing. The text changes to I've always war deree that as the hand hand Nano 0:. The derson holes up the od Nano showing lo the audierce. The video concludes with close-up the person holding iPad Nano 0, emphesizing is des and features.
background,
hand
cofero maching
the personts area. text Anyone
40.8 TFLOPS
100% Token
attached
identified
HoliTom
background, always wondered (Tve always wondered) displayed the corner screen, suggesting sensa curlosity or anticisaton. As the hand moves closer, the text changes to well now we know twell now know), indicating a revelation discovery. The hand
The video person.
standing
dressed dark outfit
4.3 TFLOPS
155 Token
Nano). which text I Nano' (Pod Nano) appeers below the device, confirming identtiy, video concludes with close-up of person holding the iPod Nano, emphestzing q the signficance the product.
small device lsseled this ad (his
dircular
small The
Figure 11: Qualitative generation comparison. Green indicates correctly detailed descriptions. Our method achieves high-quality, accurate text generation when retaining only 15% of input tokens.
24


Reviews:
[{"review_id": "uCxuxVNety", "reviewer": "Reviewer_rsKW", "summary": "The paper introduces HoliTom, a training-free token merging method to enhance the efficiency of video LLMs by compressing redundant visual tokens during video understanding tasks. The motivation stems from the high computational inefficiency of video LLMs due to the large number of tokens generated from numerous frames. HoliTom proposes a three-stage method: global redundancy-aware temporal merging to reduce temporal redundancy, spatial merging to compress spatial information, and inner-LLM merging to optimize token processing within the LLM. As a result, they reduce FLOPs to 6.9% while maintaining 99.1% of the baseline performance across various benchmarks, and ablation studies confirm the effectiveness of each module and segmentation strategy.", "questions": "See above", "limitations": "See the weaknesses", "rating": 3, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 2, "clarity": 3, "significance": 3, "originality": 2, "strengths_and_weaknesses": "Strengths:\n1. The paper’s use of a combined temporal, spatial, and inner-LLM merging strategy. It is technically sound as it provides a comprehensive solution.\n2. The paper achieves a 6.9% FLOPs reduction while retaining 99.1% performance, making it practical for real-world applications.\nWeakness:\n1. Instead of LLaVA series, applying the method to other video llm structure is necessary to showcase the generalization of the idea. Otherwise, I might suspect that it only works on simple (LLaVA) structures.\n2. The paper lacks the explanation of why without inner-LLM could sometimes get the best performance.\n3. The threshold for in line134 is manually crafted which is not a general method intrinsically.\n4. While FLOPs and TTFT are measured, there’s no mention of actual inference time on different hardware setups. I’d like to see a comparison of real-time latency across GPUs to better assess practical deployment feasibility—can the authors provide this in their rebuttal?\n5. The method handles 64 frames well, but it’s unclear how it performs with extremely long videos. I’m puzzled by this gap—could the authors test HoliTom on longer sequences to evaluate scalability and potential bottlenecks", "was_revised": false, "final_justification": "", "rebuttal": "Thank you so much for the detailed and constructive comments. We address your concerns as follows. \n\n---\n\n> **Q1**: \"Instead of LLaVA series, applying the method to other video llm structure is necessary to showcase the generalization of the idea. Otherwise, I might suspect that it only works on simple (LLaVA) structures.\"\n\n**A1**: We select the frontier Qwen2.5-VL-7B as our base model, which notably supports dynamic resolutions and long video inputs of up to 768 frames. In a controlled comparison with other training-free token compression methods, we standardize the input to 128 frames and a fixed resolution, aligning the token count with that of LLaVA-Video (11648 tokens).\n\n| Method      | Token |      | FLOPs  |      |       |        |      | VideoMME | Avg. |\n|-------------|-------|------|--------|------|-------|--------|------|----------|------|\n|             |       | %    | TFLOPs | %    | Short | Medium | Long | Overall  | %    |\n| Qwen2.5-VL  | 11256 | 100  | 197.8  | 100  | 76.2  | 67.2   | 55.0 | 66.1     | 100  |\n| +Visionzip  | 2814  | 25.0 | 39.9   | 20.2 | 71.4  | 59.8   | 52.3 | 61.2     | 92.6 |\n| +PruneVid   | 2823  | 25.0 | 40.0   | 20.2 | 65.8  | 55.8   | 49.6 | 57.0     | 86.2 |\n| +HoliTom|**2800**|**24.9**|**39.7**|**20.1**|**72.0**|**61.3**|**52.9**|**62.1**|**93.9**|\n\n**Observation**: HoliTom achieves superior efficiency, retaining 93.9% of the original performance with only 20.1% of the original FLOPs. This demonstrates HoliTom's exceptional efficiency and its robust adaptability to new architectures. We believe this finding strongly supports the contributions of our work.\n\n---\n\n> **Q2**: \"The paper lacks the explanation of why without inner-LLM could sometimes get the best performance.\"\n\n**A2**: Without inner-LLM merging, a *greater* number of the original, less-compressed tokens are retained within the LLM. For a powerful LLM, it can be capable of understanding a larger number of tokens and can deliver better performance when provided with more of the original, uncompressed tokens.\n\nBesides, sometimes, retaining fewer tokens can also lead to better performance. Concurrent studies [*1] have also observed this phenomenon. We also provide an explanation for this phenomenon in line 264 of the paper, summarized as follows:\n\n> **The value of key information over exhaustive information.** As we state: Excessive, irrelevant, or redundant data acts as noise, obscuring essential signals critical for effective processing. This information overload impedes the capacity of the model to accurately identify and process critical details, thereby degrading understanding and response generation.\n\n- [*1] 2025, Arxiv, VideoChat-Flash: Hierarchical Compression for Long-Context Video Modeling\n\n---\n\n> **Q3**: The threshold for in line 134 is manually crafted, which is not a general method intrinsically.\n\n**A3**: The threshold τ is grounded in a clear physical interpretation. It quantifies the similarity between tokens at the same spatial location in adjacent frames. A higher similarity score directly indicates that the feature representations of these frames are more alike, signifying temporal redundancy.\n﻿\nBesides, we add sensitivity analysis experiments to further clarify this.  The table below shows that the results of the average performance on the four test datasets are *robust* to different choices of this hyperparameter -- As seen, at the setup of 15% token compression rate, the 5 different τ's achieve similar average performance (the std is only 0.2). This low sensitivity demonstrates that while τ is the only hyperparameter, it does not require meticulous tuning for each specific case, thus underscoring the generalizability of our method.\n\n| τ         | 0.5  | 0.6  | 0.7  | 0.8  | 0.9  | Standard deviation (std)  |\n|-----------|------|------|------|------|------|-----|\n| Ratio=15% | 58.1 | 57.9 | 58.1 | 58.5 | 58.2 | 0.2 |\n\n---\n\n> **Q4**: \"While FLOPs and TTFT are measured, there’s no mention of actual inference time on different hardware setups. I’d like to see a comparison of real-time latency across GPUs to better assess practical deployment feasibility—can the authors provide this in their rebuttal?\"\n \n**A4**: The number of decoded tokens is variable. To provide a standardized and useful metric, our paper reports the average latency per decoded token. This allows for an estimation of the total inference time for any given output length using the formula:\n\n Total Inference Time ≈ TTFT + (Average Latency per Token × Number of Decoded Tokens)\n \nTo directly address your concern, we add an evaluation on the VideoMME benchmark to provide a comparison of total inference time. The table below presents the results for our method, HoliTom, versus the baseline on an A6000 GPU. \n\n| A6000       | TTFT    | sec/token | Latency | Avg. Score |\n|-------------|---------|-----------|---------|------------|\n| LLaVA-OV-7B | 1687 ms | 33.9 ms   | 191min  | 58.4       |\n| +Holitom  |**741 ms**|**25.8 ms** |**158min**| 57.9       |\n\nThe results show that our method not only achieves a significant reduction in Time to First Token (TTFT) and latency per token but also reduces the total inference time for the entire benchmark by 33 minutes. This demonstrates a substantial performance advantage in a practical inference scenario, underscoring the real-world applicability and efficiency of our approach.\n\nIt is also important to note that this evaluation was performed using the `lmms-eval` framework. Within this framework, a majority of the processing time is consumed by video decoding (pre-processing), which leads to low GPU utilization. Consequently, the latency improvements presented here are a conservative measure. We are confident that in more GPU-intensive applications, the performance gains from our method would be even more pronounced.\n\nWe are committed to thoroughly validating our work, and we will be expanding our tests to include a wider range of hardware setups in future work.\n\n---\n \n> **Q5**: The method handles 64 frames well, but it’s unclear how it performs with extremely long videos. I’m puzzled by this gap—could the authors test HoliTom on longer sequences to evaluate scalability and potential bottlenecks\n\n**A5**: Thanks for the insightful suggestion! We add the evaluations on Qwen2.5-VL. \n\nSpecifically, we follow the recommendations from the Qwen2.5-VL technical report, employing dynamic resolution and increasing the maximum input to 768 frames. We cap the video token count at 24,576, as suggested by the report. This inherently imposes a stricter representational constraint of 32 tokens per frame (24,576 tokens / 768 frames), a significant reduction compared to LLaVA-Video's 169 tokens per frame.\n\nResults are below:\n\n| Method     | Token |       | FLOPs  |           |       |        |      | VideoMME | Avg. |\n|------------|-------|-------|--------|-----------|-------|--------|------|----------|------|\n|            |       | %     | TFLOPs | %         | Short | Medium | Long | Overall  | %    |\n| Qwen2.5-VL | 18442 | 100   | 377.2  | 100       | 77.4  | 68.1   | 55.6 | 67.0     | 100  |\n| +Visionzip | 9221  | 50.0  | 154.5  | 41.0      |**74.9**|**66.6**| 55.7 | 65.7     | 98.1 |\n| +PruneVid  | 9173  | 49.7  | 153.5  | 40.7      | 72.3  | 64.8   | 54.7 | 63.9     | 95.4 |\n| +HoliTom   |**6513**|**34.9**|**102.0**|**27.0**| 74.4  | 66.4   |**56.4**| **65.8** | **98.2** |\n| +Visionzip | 4610  | 25.0  | 68.7   | 18.2      |**73.1**| 63.3  | 55.9 | 64.1     | 95.7 |\n| +PruneVid  | 4632  | 25.1  | 69.1   | 18.3      | 69.3  | 61.1   | 53.2 | 61.2     | 91.3 |\n| +HoliTom   |**4504**|**24.4**|**66.9**|**17.7** | 72.7  |**65.7**|**56.1**| **64.8**| **96.7** |\n\n\nAs seen, even under these stringent conditions, HoliTom demonstrates very promising performance. It maintains 98.2% of the original performance while using only 27% of the FLOPs. In an even more extreme scenario, it retains 96.7% of the performance with a mere 17.7% of the computational cost. Notably, for the long-video sub-task of VideoMME, HoliTom surpasses the original baseline's performance under both compression settings, underscoring its potential for long-video processing. In this latter case, the average number of tokens per frame is compressed to 8.\n\n---\n\n*Thank you again for helping us improve our paper! We are **actively available** during the next author-reviewer discussion period. Should you have any further questions, please let us know!*"}, {"review_id": "XcajjBZ8dH", "reviewer": "Reviewer_N3Be", "summary": "In this work, the authors propose a training-free Holistic Token Merging method to boost computation efficiency for video MLLMs. Basically, it consists of three key parts, including Temporal Merging, Spatial Merging, and Inner-LLM Merging.", "questions": "Please see the weakness section in details.\n\n1 Limited Novelty: Please clarify the key contributions in the proposed merging method, compared to the existing methods, e.g., VideoChat-Flash, Video-XL, MovieChat, etc. \n\n2 Insufficient Comparison: To further show its effectiveness, it is necessary to show compasion with the sota methods metioned above.\n\n3 More Investigation: Video QA is only one task in long video understanding. It is interesting to show if this method works for other long video understanding tasks like temporal grounding or  Needle-in-A-Video Haystack.", "limitations": "Please see the weakness section.", "rating": 4, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 3, "significance": 2, "originality": 2, "strengths_and_weaknesses": "*Strengths\n\n1 The efficiency of video MLLMs is an important research problem, especially for long video understanding. The authors attempt to address this issue via token merging.\n\n2 The paper is well organized with good structure.\n\n*Weaknesses\n\n1 The novelty is actually limited. The merging strategies ( including temporal, spatial and inner-LLM) in this work are straightforward, according to similarity comparsion. They are not foundamentally different from those merging strategies of the existing methods e.g., VideoChat-Flash, Video-XL, MovieChat, etc. Please clarify the key contributions in the proposed merging method.\n\n2 Moreover, in the experiment section, comparison with these sota methods (e.g., VideoChat-Flash, Video-XL, MovieChat, etc) is missing.  To further show its effectiveness, it is necessary to show such compasion.\n\n3 Besides of video QA, does this method work for other long video understanding tasks like temporal grounding (e.g., Charades-STA)  or  Needle-in-A-Video Haystack tasks in VideoChat-Flash.", "was_revised": true, "final_justification": "Thanks for the rebuttal. The authors have addressed most of my main concerns. I change my rating as borderline accept.", "rebuttal": "Thank you so much for the detailed and constructive comments. We address your concerns as follows. \n\n---\n\n> **Q1**: The novelty is limited. The merging strategies (including temporal, spatial, and inner-LLM) in this work are straightforward, according to similarity comparison. They are not fundamentally different from those merging strategies of the existing methods, e.g., VideoChat-Flash, Video-XL, MovieChat, etc. Please clarify the key contributions in the proposed merging method.\n\n**A1**: We understand the concern that the merging scheme used by other works (VideoChat-Flash, Video-XL, MovieChat) may make our work \"appear\" similar to them. However, our work is actually distinct from these works. Before detailing our novel contributions, it is worth noting the key differences that separate HoliTom from the methods mentioned above:\n\n- VideoChat-Flash, Video-XL, and MovieChat are primarily **training-based** methods. Their performance is a result of the entire model architecture, the training data, and the specific training strategies employed, as well as their token compression techniques. The associated **training costs are non-trivial**.\n\n- In contrast, HoliTom is proposed not as a new standalone model, but as a **training-free**, **parameter-free**, and **plug-and-play** token compression framework. Its purpose is to be seamlessly inserted into existing, powerful video LLMs to help them achieve a superior balance between performance and efficiency. \n\nIn light of this crucial distinction, direct comparison between our training-free method vs. their training-based methods might not fully capture our contribution.\n\nWith this clarification, we would like to address your primary concern by detailing the key contributions and novelty within our proposed methodology:\n\n**Contributions**:\n\n- **Optimal Temporal Merging via Dynamic Programming**: We introduce a more principled approach to segmenting and compressing visual tokens. Existing methods rely on naive fixed-cliping or clustering on abstracted frame tokens (a pooled or [CLS] token), which ignore the rich, underlying visual tokens. Our key innovation is to solve this by formulating it as a global optimization problem and solving it with Dynamic Programming (DP). By operating directly on the complete set of fine-grained, raw visual tokens, our DP algorithm finds the optimal solution that simultaneously achieves the dual objectives of maximum temporal redundancy pruning and the most coherent semantic video segmentation.\n- **Synergistic Inner-LLM Merging that Preserves Information**: Common techniques like token dropping, while simple, risk irreversible information loss. Synergy with our outer-LLM merging and considering information loss, we merge related tokens in the LLM’s deeper layers. This condenses rich details into representative tokens and mitigates the performance degradation caused by dropping.\n- **SoTA Efficiency-Performance Trade-off**: Extensive evaluations on LLaVA-OneVision and LLaVA-Video demonstrate that our integrated pruning framework achieves a state-of-the-art efficiency-performance trade-off, significantly reducing computational costs and accelerating real-world inference while preserving model performance.\n\n---\n\n> **Q2**: Insufficient Comparison: To further show its effectiveness, it is necessary to show a comparison with the SOTA methods mentioned above.\n\n**A2**: Per the suggestion, we add comparisons on Qwen2.5-VL. Results are below. \n\n| Method               | T/F     | Res.   | Token ratio (%) | VideoMME | Charades-STA mIoU |\n|----------------------|---------|--------|-----------------|----------|-------------------|\n| MovieChat            |         |        | -               | 38.2     | <10               |\n| Video-XL             |         |        | -               | 55.5     | -                 |\n| VideoChat-Flash @224 | 16      | 50176  | -               | 64.0     | 48.4              |\n| VideoChat-Flash @448 | 16      | 200704 | -               | 65.3     | 48.0              |\n| Qwen2.5-VL+Visionzip | 16      | 25088  | 50.0            | 65.7     | 49.2              |\n| Qwen2.5-VL+PruneVid  | 16      | 25088  | 49.7            | 63.9     | 28.8              | \n| Qwen2.5-VL+HoliTom   | **~11** | 25088  | **34.9**        | **65.8** | **49.7**          |\n| Qwen2.5-VL+HoliTom   | 8       | 25088  | 24.4            |   64.8   | 45.9              |\n\nAs seen, with less input image resolution, fewer tokens/frame, and potentially fewer input frames, Qwen2.5-VL+HoliTom **surpasses** VideoChat-Flash @448 on both the VideoMME and Charades-STA benchmarks. \n\nThis result highlights HoliTom's strength as a plug-and-play module that enables powerful video LLMs to achieve a superior balance between performance and computational cost, *even being training-free*.\n\n---\n\n> **Q3**: Besides video QA, does this method work for other long video understanding tasks like temporal grounding (e.g., Charades-STA) or Needle-in-A-Video Haystack tasks in VideoChat-Flash.\n\n**A3**: \n**Regarding the temporal grounding task** - Charades-STA, we are pleased to report that our approach surpasses the current SoTA training-free token compression methods at both evaluated compression ratios. We attribute this success to our temporal compress strategy, which, unlike methods such as PruneVid that disrupt temporal order, consciously preserves the temporal information of video frames, a critical aspect for this task. \n\nWe do observe a performance decline at higher compression rates. The reasons are analyzed below. \n\n- **Analyses**: General video understanding (like QA) relies on interpreting what content is present in the video. Training-free compression methods benefit from the preliminary semantic alignment between visuals and text by the vision encoder. The attention scores in the encoder and clustering mechanisms can then effectively guide the condensation of this **existing semantic understanding**. In contrast, temporal grounding is about pinpointing when an event occurs. This requires the joint training of the vision encoder and LLM with the help of calibrated data. Neither vision encoder nor LLM can do this alone. This capability is not something that can be expected to emerge in a zero-shot manner; it **must be learned**.\n\n**Regarding Needle-in-a-Video-Haystack** tasks: We also appreciate the reviewer mentioning this challenging benchmark. At present, the backbone of our method, Qwen2.5-VL, is constrained to a maximum input of 768 frames. The significant computational overhead required for processing 1k-10k frames makes it infeasible for us to conduct these experiments on this task for the current submission. We are grateful for the suggestion and see this as an important future direction.  \n\n---\n\n*Thank you again for helping us improve our paper! We are **actively available** during the next author-reviewer discussion period. Should you have any further questions, please let us know!*"}, {"review_id": "CvFyhKf8Kd", "reviewer": "Reviewer_jPEX", "summary": "This paper introduces HoliTom, a holistic token merging method for video LLMs that integrates spatial, temporal, and inner-LLM token merging. HoLiTom employs: (1) Global redundancy-aware temporal merging, which reduces temporal redundancy by merging tokens into their first occurrence; (2) Spatial merging via clustering tokens based on global feature representations; and (3) Inner-LLM merging, where redundant tokens are averaged rather than discarded. Evaluated on multiple video understanding benchmarks, HoLiTom achieves high efficiency and strong performance.", "questions": "* Please refer to the Weaknesses section.\n* If have time, an interesting extension would be evaluating HoliTom on training-free video LLMs (e.g., image-trained models like LLaVA-1.6 applied to videos, e.g. SF-LLaVA [1] or TS-LLaVA). Since these models lack explicit temporal knowledge from video training, does the token merging strategy still prove effective? \n\n\n[1] Mingze Xu, Mingfei Gao, Zhe Gan, Hong-You Chen, Zhengfeng Lai, Haiming Gang, Kai Kang, Afshin Dehghan. (2024) SlowFast-LLaVA: A Strong Training-Free Baseline for Video Large Language Models.\n\n[2] Tingyu Qu, Mingxiao Li, Tinne Tuytelaars, Marie-Francine Moens. (2024) TS-LLaVA: Constructing Visual Tokens through Thumbnail-and-Sampling for Training-Free Video Large Language Models", "limitations": "yes", "rating": 4, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 3, "significance": 2, "originality": 2, "strengths_and_weaknesses": "**Strengths**\n* The paper is clearly-written and well-organised. The method is quite intuitive.\n* The method considers multiple aspects while improving efficiency, including spatial, temporal and inner-LLM redundancy. Overall, the method offers a comprehensive solution to token redundancy in video understanding tasks.\n* Performance wise, strong performance and high efficiency are obtained as compared to previous methods when evaluating with video LLMs.\n\n**Weaknesses**\n* If I understand correctly, the core idea is to merge/average out redundant information based mainly on the global representations of the visual tokens (either the similarity calculation in temporal merging, CLS replacement in spatial merging or the inner-LLM merging). This approach seems effective for redundancy reduction and is highly favourable in terms of efficiency, but could it risk losing fine details (e.g., small objects) or become overly aggressive for longer videos?\n* Following up on the longer videos, I noticed that on LongVideoBench HoliTom either matches or underperforms compared to prior methods, unlike its superior results on shorter videos (e.g., MVBench). Could you elaborate on this? Is this a limitation inherent to the method’s design, or are there other factors at play?\n* I’m curious about HoliTom’s performance on tasks requiring fine-grained object-level reasoning. Given the method’s aggressive merging strategy, could you discuss whether it preserves sufficient detail for such scenarios? Additional analyses would be helpful.", "was_revised": true, "final_justification": "The rebuttals have addressed most of my concerns, and I believe the paper makes a valuable contribution to the subfield of video understanding. Having read the rebuttals and other reviews, I remain somewhat on the borderline, but I support the paper’s acceptance. If accepted, it would be a welcome addition to the field.", "rebuttal": "Thank you so much for the detailed and constructive comments. We address your concerns as follows. \n\n---\n\n> **W1**: If I understand correctly, the core idea is to merge/average out redundant information based mainly on the global representations of the visual tokens (either the similarity calculation in temporal merging, CLS replacement in spatial merging or the inner-LLM merging). This approach seems effective for redundancy reduction and is highly favourable in terms of efficiency, but could it risk losing fine details (e.g., small objects) or become overly aggressive for longer videos?\n\n> **W3**: I’m curious about HoliTom’s performance on tasks requiring fine-grained object-level reasoning. Given the method’s aggressive merging strategy, could you discuss whether it preserves sufficient detail for such scenarios? Additional analyses would be helpful.\n\n**A1,3**: (The above two weaknesses are similar. We respond to them collectively here.) Thank you for your interest and accurate summary of our method. We appreciate the opportunity to elaborate on this important point.\n\n**Details Protection**\n\nYour concern about the potential loss of fine-grained details is indeed a critical consideration for any token compression strategy. However, HoliTom not only preserves these details but can actually enhance the model's ability to perceive them.\n\nTo substantiate this, we would like to draw your attention to our performance on the **3 object-level subtasks** (OE, OI, OS) in MVBench. With compression rates of 10-25%, HoliTom leads to a **notable increase** in performance. This outcome would be unlikely if our method are discarding critical information about small objects. Instead, it serves as strong evidence of our method's ability to retain fine-grained details effectively.\n\n> OE: Object Existence - Example: Are there any moving green objects when the video ends? (A) not sure (B) yes (C) no \n\n> OI: Object Interaction - Example: Which object was tidied up by the person? (A) broom (B) cabinet (C) blanket (D) table\n\n> OS: Object Shuffle - Example: Where is the hidden object at the end of the game from the person's point of view? (A) Under the first object from the left. (B) Under the third object from the left. (C)Under the second object from the left.\n\n|             | FLOPs Ratio | OE   | OI   | OS   | Overall |\n|-------------|-------------|------|------|------|---------|\n| LLaVA-OV-7B | 100%        | 57.5 | 84.0 | 35.5 | 58.3    |\n| +HoliTom    | 17.4%       | 61.0 | 83.5 | 36.5 | 58.4    |\n| +HoliTom    | 14.2%     |**63.0**| 84.0 | 37.5| **58.7** |\n| +HoliTom    | 10.5%      | 60.5 |**84.5**|**38.0**| 58.1 |\n| +HoliTom    |**6.9%**     | 60.0 | 81.0 | 37.0 | 57.3    |\n\nThis counterintuitive performance boost is discussed in our paper (line 264): \n> **The value of key information over exhaustive information.** As we state: Excessive, irrelevant, or redundant data acts as noise, obscuring essential signals critical for effective processing. This information overload impedes the capacity of the model to accurately identify and process critical details, thereby degrading understanding and response generation.\n\nFrankly, I concede that extreme token compression will inevitably degrade performance across the board, including detail-oriented understanding. We cannot expect a handful of tokens to carry the informational capacity of a massive dataset.\n\n**Concerns about the method's ability for longer video understanding**\n\nTo test Holitom's ability on long video understanding, we followed the recommendations from the Qwen2.5-VL technical report, employing dynamic resolution and increasing the maximum input to 768 frames. We capped the video token count at 24,576, as suggested by the report. This inherently imposes a stricter representational constraint of 32 tokens per frame (24,576 tokens / 768 frames), a significant reduction compared to LLaVA-Video's 169 tokens per frame.\n\nEven under these stringent conditions, HoliTom demonstrated **remarkable** performance. Notably, for the long-video sub-task of VideoMME, HoliTom **surpassed** the original baseline's performance under both compression settings, underscoring its potential for long-video processing. In this latter case, the average number of tokens per frame was compressed to 8.\n\n| Method     | Token |       | FLOPs  |           |       |        |      | VideoMME | Avg. |\n|------------|-------|-------|--------|-----------|-------|--------|------|----------|------|\n|            |       | %     | TFLOPs | %         | Short | Medium | Long | Overall  | %    |\n| Qwen2.5-VL | 18442 | 100   | 377.2  | 100       | 77.4  | 68.1   | 55.6 | 67.0     | 100  |\n| +Visionzip | 9221  | 50.0  | 154.5  | 41.0      |**74.9**|**66.6**| 55.7 | 65.7     | 98.1 |\n| +PruneVid  | 9173  | 49.7  | 153.5  | 40.7      | 72.3  | 64.8   | 54.7 | 63.9     | 95.4 |\n| +HoliTom   |**6513**|**34.9**|**102.0**|**27.0**| 74.4  | 66.4   |**56.4**| **65.8** | **98.2** |\n| +Visionzip | 4610  | 25.0  | 68.7   | 18.2      |**73.1**| 63.3  | 55.9 | 64.1     | 95.7 |\n| +PruneVid  | 4632  | 25.1  | 69.1   | 18.3      | 69.3  | 61.1   | 53.2 | 61.2     | 91.3 |\n| +HoliTom   |**4504**|**24.4**|**66.9**|**17.7** | 72.7  |**65.7**|**56.1**| **64.8**| **96.7** \n\n---\n\n> **W2**: Following up on the longer videos, I noticed that on LongVideoBench HoliTom either matches or underperforms compared to prior methods, unlike its superior results on shorter videos (e.g., MVBench). Could you elaborate on this? Is this a limitation inherent to the method’s design, or are there other factors at play?\n\n| Model          | OV-7B | VisionZip | PruneVid | FastVID | HoliTom | OV-72B | VisionZip | PruneVid | HoliTom | Vid-7B | VisionZip | HoliTom |\n|----------------|-------------|------------|-----------|----------|----------|--------------|------------|-----------|----------|--------------|------------|----------|\n| MVBench        | 58.3        | 56.5       | 56.8      | 56.0     | **58.1** | 60.9         | 58.4       | 56.8      | **58.7** | 60.4         | 56.7       | **57.7** |\n| LongVideoBench | 56.4        | 54.4       | 55.4      | 56.2     | **56.4** | 62.7         | **57.4**   | **57.4**  | 57.2     | 58.9         | 54.7       | **56.2** |\n\n**A2**: Thanks for the question, but there might be a misunderstanding here. \n\nAs shown in the paper (Table 2,3 ), on 3 different LLM backbones and at multiple ratios, HoliTom *surpasses* the previous SoTA method in both MVBench and LongVideoBench benchmarks in the majority when tested. This result underscores its exceptional ability to maintain high-fidelity comprehension for videos of all lengths.\n﻿\n---\n\n> **Q1**: If have time, an interesting extension would be evaluating HoliTom on training-free video LLMs (e.g., image-trained models like LLaVA-1.6 applied to videos, e.g., SF-LLaVA or TS-LLaVA). Since these models lack explicit temporal knowledge from video training, does the token merging strategy still prove effective? \n\n**Q1-Ans**: Thank you for your insightful suggestion! We believe that our method, HoliTom, would indeed remain effective for training-free video LLMs.\n\nWe find this to be a fascinating direction for empirical validation. Due to device constraints, we are unable to provide experimental results for this specific setting within the rebuttal period. We give our analysis below:    \n\nThe effectiveness of our approach is rooted in principles that are not dependent on the model having explicit temporal knowledge from video-specific training. We can break this down based on our two-stage compression:\n\n- **Outer-LLM Token Compression**: This stage targets the inherent redundancy within the visual data itself. For instance, in a model like LLaVA-OneVision that uses a vision encoder like SigLIP, video frames are typically treated as a batch of images (i.e., the `batch_size` dimension in a tensor like `[batch_size, head_num, seq_len, head_dim]`). The frames are encoded without explicit cross-frame computation, meaning the model lacks inherent temporal knowledge at this stage. Our outer-LLM compression operates on these encoded representations after they have been processed by the vision encoder. The redundancy it compresses is therefore the semantic and visual overlap between frames, which is a fundamental property of the video data itself, not a feature learned by the model.\n- **Inner-LLM Token Compression**: This stage leverages the **sparse attention** phenomenon common in large vision-language models. We suggest that this sparsity primarily arises from two factors: the misalignment between visual and text, and the redundant visual representations. These factors are characteristic of the multi-modal data and architecture themselves, rather than being dependent on whether the model was trained on video datasets. Therefore, since the underlying causes of sparse attention are not unique to video-trained models, we expect the inner-LLM compression to be effective in identifying and merging non-essential tokens in a training-free context as well.\n \n---\n\n*Thank you again for helping us improve our paper! We are **actively available** during the next author-reviewer discussion period. Should you have any further questions, please let us know!*"}, {"review_id": "J6kQuXb1q8", "reviewer": "Reviewer_uHRM", "summary": "This paper proposes HoliTom, a training-free token-compression framework that improves the inference-time efficiency of video LLMs. HoliTom combines outer-LLM reduction—which performs global optimization–based temporal merging and attention/cluster-based spatial merging—with inner-LLM reduction, which merges visual tokens at every layer using a fixed retention ratio instead of simply dropping redundant tokens. The method attains state-of-the-art results while requiring fewer FLOPs than prior token-compression approaches.", "questions": "Apart from three questions in weakness part, I have additional following questions:\n\n4. For inner-LLM compression, the method uses the attention score of the “last token.” Does this refer to the last prompt token (visual + text) or the last visual token only? Have the authors compared the two variants, and if so, what did you observe?\n5. Suppose we sample more frames yet keep overall FLOPs constant (eg. 100%) after applying HoliTom. Would this yield better performance than using fewer frames with the vanilla model?\n\nIf the authors address the concerns above, I believe the paper will be further solidified, and I will raise my score.", "limitations": "- Multi-round conversations. Can HoliTom handle multi-turn dialogue in video LLMs? If a follow-up query arrives, does it affect the validity of inner-LLM merging?\n- Extremely long videos. Have the authors tested HoliTom on very long inputs (e.g., Qwen 2.5-VL with 768 frames)? What efficiency and accuracy gains does HoliTom provide compared with no reduction in that setting?", "rating": 5, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 4, "significance": 3, "originality": 3, "strengths_and_weaknesses": "**Strengths**\n1. **Effectiveness and efficiency.** Both outer-LLM and inner-LLM merging clearly improve performance, as evidenced by the ablations in Table 4. The temporal-merging objective is novel, and solving it via dynamic programming is elegant and effective.\n2. **Comprehensive analysis.** Section 4 offers detailed experiments that thoroughly validate the proposed method. The inference-efficiency study highlights HoliTom’s advantage over previous techniques.\n3. **Clarity of writing.** The paper is well written and easy to follow.\n\n**Weaknesses**\n1. **Missing spatial-merging ablation.** For non-redundant temporal tokens, the paper uses an attention-based strategy; for redundant tokens, it adopts DPC-KNN clustering. An ablation justifying these design choices is needed.\n2. **Lack of qualitative visualizations.** Visual examples showing token reduction at each stage would help readers better understand how each component works.\n3. **Equation (5) needs more explanation.** The derivation and intuition behind this equation should be clarified.\n\nMinor Issues\n- L156: “an” → “a”\n- L198: “spatial-temporal” → “spatio-temporal”", "was_revised": true, "final_justification": "Overall, this paper proposes three key designs to achieve stronger training-free token compression for video VLMs.\n\nIn my view, the novelty gap between this work and prior or concurrent papers is not marginal. Focusing on a training-free token-compression paradigm, the paper introduces three key innovations that together deliver state-of-the-art performance and efficiency. Given the limited toolbox for further improving token compression and the intense competition in this area, these advances are both meaningful and valuable.\n\nThe rebuttal also provides convincing evidence of the method’s generalization capability and supplies the necessary technical details. Accordingly, I have kept my rating and recommend acceptance.", "rebuttal": "Thank you so much for the detailed and constructive comments. We address your concerns as follows. \n\n---\n\n>**W1**: Missing spatial-merging ablation. For non-redundant temporal tokens, the paper uses an attention-based strategy; for redundant tokens, it adopts DPC-KNN clustering. An ablation justifying these design choices is needed.\n\n**A1**: We thank the reviewer for this valuable suggestion. We agree that an ablation study is essential to justify our design choices for the spatial-merging strategy. Following this advice, we have conducted the experiment, and the results are presented below.\n\nHere, Attn refers to using only the attention-based strategy for all tokens, while DPC-KNN uses only the clustering algorithm. Mixed is our proposed hybrid strategy.\n\n| Method  | MVBench | EgoSchema | LVBench | VideoMME | Avg. Score |\n|---------|---------|-----------|---------|----------|------------|\n| Attn    |**58.4**| **60.9**   | 55.9    | 57.4     | 58.1       |\n| DPC-KNN | 57.4    | 59.6      | 53.9    | 56.6     | 56.9       |\n| Mixed   |**58.4**| **60.9** | **56.2**  | **58.3** | **58.5**   |\n\nThe results clearly show that our mixed strategy achieves the best performance. Our design principle is as follows:\n\nFor non-redundant tokens, the encoder's self-attention scores serve as an **excellent prior** for visual importance within a single frame. A compression strategy guided by these intra-frame scores is therefore highly effective.\n\nHowever, redundant tokens are different, as they are formed by merging tokens from adjacent frames. Since the self-attention mechanism operates in encoder (siglip) only within a single frame, using these attention scores as a metric for the new, merged tokens lacks a solid theoretical basis. For this reason, we adopted DPC-KNN clustering to group these merged tokens based on their feature similarity, which is a more principled approach for this context. As noted in the paper (line 163), this design consideration is central to our method.\n\nWe sincerely appreciate the reviewer's feedback. This supplementary experiment has allowed us to substantiate the rationale for our method more rigorously and has significantly strengthened our paper.\n\n---\n\n>**W2**: Lack of qualitative visualizations. Visual examples showing token reduction at each stage would help readers better understand how each component works.\n\n**A2**: We completely agree that visualizations would significantly enhance the reader's understanding of our proposed token reduction process. We were unable to include figures due to the NeurIPS'25 rebuttal guidelines, but we are willing to incorporate these key visualizations in the next version. We appreciate you pointing out this opportunity for improvement.\n\n---\n\n>**W3**: Equation (5) needs more explanation. The derivation and intuition behind this equation should be clarified.\n\n**A3**: This method for selecting cluster centers is based on the well-established Density Peak Clustering (DPC) algorithm, originally proposed by Rodriguez and Laio in Science (2014). The fundamental idea is that cluster centers are characterized by two key properties:\n\n- They are surrounded by neighbors with lower local density (i.e., they are local density maxima).\n\n- They are at a relatively large distance from any other points with a higher local density.\n\nThe equations you referenced are designed to mathematically identify points that satisfy these two conditions.\n\nIntuition Breakdown\nLocal Density $ρ_i$: This term measures how dense the feature space is around a given token $v_i$. A high value of $ρ_i$ indicates that the token is in a crowded region, surrounded by many other similar tokens. This suggests it's part of a potential cluster.\n\nSeparation Distance $δ_i$: This term measures the minimum distance from token $v_i$ to any other token with a higher local density. A large $δ_i$ is significant: it means that even though $v_i$ is in a dense area, it is also the densest point in its local neighborhood, making it a \"peak\" in the density landscape. For the globally densest point, $δ_i$ is its maximum distance to any other token.\n\nFinal Score $γ_i=ρ_i×δ_i$: The final score $γ_i$ effectively combines these two concepts. To be chosen as a cluster center, a token must not only be in a dense region (high $ρ_i$) but also be a clear density peak (high $δ_i$). By multiplying them, we find tokens that are simultaneously local density maxima and are well-separated from other, even denser points. These are the ideal candidates for cluster centers.\n\nIn summary, this method provides a robust and non-parametric way to automatically find cluster centers by identifying the peaks in the data's density distribution. We will add this detailed explanation and the proper citation to the manuscript to eliminate any confusion.\n\n---\n\n>**Q1**: For inner-LLM compression, the method uses the attention score of the “last token.” Does this refer to the last prompt token (visual + text) or the last visual token only? Have the authors compared the two variants, and if so, what did you observe?\n\n**A1**: The \"last token\" we utilize for attention-based compression is indeed the final token of the full input sequence, comprising both visual and textual elements.\n\nOur reason for this choice is that the final token of the complete sequence is positioned **closest** to the tokens that are to be generated. We believe its hidden state therefore encapsulates a more comprehensive understanding of the entire query (both visual and textual context), making it the most suitable anchor for calculating attention scores for compression. While we have not performed a direct empirical comparison, we appreciate you bringing this to our attention, as it is a valuable consideration for future analysis.\n\n---\n\n>**Q2**: Suppose we sample more frames yet keep overall FLOPs constant (eg. 100%) after applying HoliTom. Would this yield better performance than using fewer frames with the vanilla model?\n\n| Method  | Frames | Prefilling FLOPs (T)  | FLOPs Ratio  | Avg Score |\n|---------|--------|-----------------------|--------------|-----------|\n| ov-7b   | 32     | 40.8                  | 100%         | 58.4      |\n| Holitom | 32     | 5.2                   | 12.7%        | 58.5      |\n| Holitom | 64     | 10.8                  | 26.5%        | 58.9      |\n| Holitom | 128    | 23.1                  | 56.6%        | 59.3      |\n\n**A2**: **Yes**, sampling more frames with HoliTom, while keeping the total FLOPs constant or even less, yields significantly better performance than a vanilla model operating on fewer frames.\n\nWe also note that as the number of input frames grows, the computational overhead from the vision encoder becomes non-negligible. This trade-off can be seen in Figure 5 of our paper.\n\n---\n\n>**L1**: Multi-round conversations. Can HoliTom handle multi-turn dialogue in video LLMs? If a follow-up query arrives, does it affect the validity of inner-LLM merging?\n\n**A1**: We appreciate you pointing this out. The validity of the merged tokens in a multi-turn scenario is indeed a key consideration that we carefully addressed in our work. The core of the issue lies in the distinction between our two token compression strategies:\n\n- **Inner-LLM Merging**: As you rightly suggest, this approach can be affected by follow-up queries. Because it computes joint attention over both visual and text tokens, the resulting compressed representation is influenced by the initial query. A new turn in the dialogue would ideally require re-computation.\n- **Outer-LLM Merging**: In contrast, this method is designed to be question-agnostic. The compression is performed independently of the language input, making it robust and consistently valid across multiple conversational turns.\n\n**Recognizing the distinct implications** of these two methods for multi-turn dialogue, we report the results for HoliTom (which uses inner-LLM merging) and HoliTom(w/o M) **separately** in our paper. Our intention is to provide the research community — especially those focused on downstream multi-turn Q&A tasks — with better understanding.\n\n>**L2**: Extremely long videos. Have the authors tested HoliTom on very long inputs (e.g., Qwen 2.5-VL with 768 frames)? What efficiency and accuracy gains does HoliTom provide compared with no reduction in that setting?\n\n| Method     | Token | Ratio | FLOPs  | FLOPs (%) |       |        |      | VideoMME | Avg. |\n|------------|-------|-------|--------|-----------|-------|--------|------|----------|------|\n|            |       | %     | TFLOPs | %         | Short | Medium | Long | Overall  | %    |\n| Qwen2.5-VL | 18442 | 100   | 377.2  | 100  | 77.4  | 68.1   | 55.6 | 67.0     | 100  |\n| +Visionzip | 9221  | 50.0  | 154.5  | 41.0 | **74.9**  | **66.6**   | 55.7 | 65.7     | 98.1 |\n| +PruneVid  | 9173  | 49.7  | 153.5  | 40.7 | 72.3  | 64.8   | 54.7 | 63.9     | 95.4 |\n| +HoliTom   | **6513**  | **34.9**  | **102.0**  | **27.0** | 74.4  | 66.4   | **56.4** | **65.8**     | **98.2** |\n| +Visionzip | 4610  | 25.0  | 68.7   | 18.2 | **73.1**  | 63.3   | 55.9 | 64.1     | 95.7 |\n| +PruneVid  | 4632  | 25.1  | 69.1   | 18.3 | 69.3  | 61.1   | 53.2 | 61.2     | 91.3 |\n| +HoliTom   | **4504**  | **24.4**  | **66.9**   | **17.7** | 72.7  | **65.7**   | **56.1** | **64.8**     | **96.7** |\n\n**A2**: We tested HoliTom on very long inputs (768 frames) on the comprehensive VideoMME benchmark, which includes a dedicated sub-evaluation set for long videos. This allows for a direct assessment of performance on extended sequences.\n\nOur findings on this \"Long\" video subset confirm the effectiveness of our approach. Specifically, HoliTom achieves a 0.5% performance increase over the baseline, even while retaining only 25% of the original tokens.\n \n---\n\n*Thank you again for helping us improve our paper! We are **actively available** during the next author-reviewer discussion period. Should you have any further questions, please let us know!*"}]

Author rebuttal:

Meta-review: This paper presents HoliTom a training-free token merging framework for video LLMs. It performs an integration of outer-LLM spatio-temporal reduction, reducing initial token counts, and an inner-LLM token merging mechanism.  The paper received 1 borderline reject, 2 borderline accept, and 1 accept final ratings.  The reviewers are positive about the novelty of the approach and good performance.  Concerns regarding novelty compared to training-based methods and missing experiments were mostly addressed adequately by the rebuttal.  The borderline reject reviewer was not satisfied with the rebuttal, but was vague in their reason ("Some explanations are still unclear to me. Many details are missing in the paper and rebuttal.").  The AC deemed the rebuttal to address their main concerns, especially with the experiments using Qwen2.5-VL-7B.  After carefully considering the paper, reviews, and discussion, the AC recommends accept.
Decision: Accept (poster)