Title: HoliTom: Holistic Token Merging for Fast Video Large Language Models
Venue: NeurIPS 2025
Accepted: yes
Keywords: Video LLMs

Token Compression

Video Understanding
Abstract: Video large language models (video LLMs) excel at video comprehension but face significant computational inefficiency due to redundant video tokens. Existing token pruning methods offer solutions. However, approaches operating within the LLM (inner-LLM pruning), such as FastV, incur intrinsic computational overhead in shallow layers. In contrast, methods performing token pruning before the LLM (outer-LLM pruning) primarily address spatial redundancy within individual frames or limited temporal windows, neglecting the crucial global temporal dynamics and correlations across longer video sequences. This leads to sub-optimal spatio-temporal reduction and does not leverage video compressibility fully. Crucially, the synergistic potential and mutual influence of combining these strategies remain unexplored. To further reduce redundancy, we introduce HoliTom, a novel training-free holistic token merging framework. HoliTom employs outer-LLM pruning through global redundancy-aware temporal segmentation, followed by spatial-temporal merging to reduce visual tokens by over 90%, significantly alleviating the LLM's computational burden. Complementing this, we introduce a robust inner-LLM token similarity-based merging approach, designed for superior performance and compatibility with outer-LLM pruning. Evaluations demonstrate our method's promising efficiency-performance trade-off on LLaVA-OneVision-7B, reducing computational costs to 6.9% of FLOPs while maintaining 99.1% of the original performance. Furthermore, we achieve a 2.28× reduction in Time-To-First-Token (TTFT) and a 1.32× acceleration in decoding throughput, highlighting the practical benefits of our integrated pruning approach for efficient video LLMs inference.

HoliTom : Holistic Token Merging for Fast Video

Abstract

Video large language models (video LLMs) excel at video comprehension but face significant computational inefficiency due to redundant video tokens. Existing token pruning methods offer solutions. However, approaches operating within the LLM (inner-LLM pruning), such as FastV, incur intrinsic computational overhead in shallow layers. In contrast, methods performing token pruning before the LLM (outer-LLM pruning) primarily address spatial redundancy within individual frames or limited temporal windows, neglecting the crucial global temporal dynamics and correlations across longer video sequences. This leads to sub-optimal spatiotemporal reduction and does not leverage video compressibility fully. Crucially, the synergistic potential and mutual influence of integrating these strategies remain unexplored. To further reduce redundancy, we introduce HoliTom , a novel trainingfree holistic token merging framework. HoliTom employs outer-LLM pruning through global redundancy-aware temporal segmentation, followed by spatialtemporal merging to reduce visual tokens by over 90%, significantly alleviating the LLM's computational burden. Complementing this, we introduce a robust innerLLM token similarity-based merging approach, designed for superior performance and compatibility with outer-LLM pruning. Evaluations demonstrate our method's promising efficiency-performance trade-off on LLaVA-OneVision-7B, reducing computational costs to 6.9% of FLOPs while maintaining 99.1% of the original performance. Furthermore, we achieve a 2.28 × reduction in Time-To-First-Token (TTFT) and a 1.32 × acceleration in decoding throughput, highlighting the practical benefits of our integrated pruning approach for efficient video LLMs inference.

∗

1 Introduction

Video large language models (video LLMs) [25, 70, 56, 8, 9, 27, 30, 63, 67, 18, 51] show remarkable potential in understanding complex video content. However, their practical deployment is hindered by significant computational inefficiency. This inefficiency stems from processing high volumes of video tokens generated by encoding sampled frames, leading to substantial overhead, particularly due to the quadratic complexity of the attention mechanism in the LLMs. For videos with numerous frames, the input token count can easily reach tens of thousands, making inference computationally expensive. While prior works [6, 65, 61, 48, 19, 34, 36] have explored model compression and token pruning, achieving a desirable balance between efficiency and performance in video tasks remains an open challenge. Thus, developing effective methods to reduce video token redundancy while preserving critical semantic information is crucial for the widespread adoption of video LLMs.

Token pruning is a promising direction. These approaches generally fall into two categories depending on where pruning occurs. Inner-LLM pruning methods, such as FastV [6], TopV [64], and PDrop [61], operate within the LLM layers. However, they incur intrinsic computational and memory costs in the initial layers before pruning takes effect, limiting overall FLOPs reduction. Outer-LLM pruning methods process tokens before the main LLM computation. Some methods address spatial redundancy (VisionZip [65], PruMerge [42]), others tackle temporal aspects within limited temporal windows (DyCoke [48], PruneVid [21]), thus preventing a global understanding of video dynamics and comprehensive spatio-temporal optimization. Furthermore, despite the potential for synergy, no prior work has systemat-

Methods

Spatial

Temporal

Inner-LLM

FastV [6]

PDrop [61]

LLaVA-PruMerge [42]

VisionZip [65]

DyCoke [48]

FastVID [44]

Ours

ically explored integrating inner-LLM and outer-LLM pruning strategies or analyzed their mutual benefits. The current methods, while offering some benefits, still leave room for improvement.

To address these limitations, we propose a holistic token pruning for video LLMs that leverages external and internal strategies. Our method first tackles temporal redundancy through a global redundancy-aware video segmentation process, followed by spatio-temporal merging. This external step reduces visual tokens to less than 10%, significantly alleviating the computational burden on the subsequent LLM. Complementing this, we introduce a new and robust inner-LLM token similaritybased merging method, specifically designed for integration with our outer-LLM pruning method, enabling mutual benefits. This integrated strategy offers a more holistic and efficient solution to handle long videos with LLMs, as summarized in Tab. 1, which contrasts the compression achieved by our approach in both the spatio-temporal domain and within the inner-LLM against other methods.

Empirical evaluations validate the effectiveness of our proposed method in achieving a compelling efficiency-performance trade-off. Specifically, as shown in Fig. 1 (right), on the LLaV A-OneVision7B model [25], our approach reduces computational costs to just 6.9% of the original FLOPs while remarkably preserving 99.1% of the original model's performance. Moreover, we observe significant gains in inference efficiency, achieving a 2.28 × reduction in Time-To-First-Token (TTFT) and a 1.32 × acceleration in decoding throughput. These results clearly demonstrate the substantial practical advantages of our holistic token merging framework for efficient video LLM inference.

Our key contributions are summarized as follows:

We analyze the phenomenon of temporal redundancy in the context of video LLMs and propose a global redundancy-aware temporal merging method to effectively address the inefficiency in video LLMs before LLM processing in a plug-and-play fashion.

We introduce a robust inner-LLM similarity-based merging technique specifically designed for integration with the outer-LLM pruning method, facilitating synergistic optimization.

Extensive evaluations on LLaVA-OneVision and LLaVA-Video demonstrate that our integrated pruning framework achieves a state-of-the-art efficiency-performance trade-off, significantly reducing computational costs and accelerating inference while preserving model performance.

2 Related Work

2.1 Video Large Language Models

The rapid progress of multimodal large language models has led to the integration of video encoders, creating video LLMs that excel in video understanding and question answering tasks [63, 17, 25, 2, 56, 3, 27, 28, 30, 67, 47, 50, 23, 1, 46, 29]. However, the substantial number of tokens generated by processing numerous video frames hinders inference efficiency, thereby impeding the widespread adoption of video LLMs. Existing approaches have attempted to mitigate this issue. For instance, VideoLLaMA [67] employs a Q-Former module [26] to aggregate video tokens, while MovieChat [47] introduces a memory module to merge and store token representations. Although pooling mechanisms in LLaVA-OneVision [25] reduce token counts, each video frame still produces hundreds of tokens for downstream processing. Consequently, handling tens of thousands of visual tokens for long video inputs substantially increases inference time and memory consumption. While works such as VILA [32] and NVILA [35] aim to optimize token usage, these methods often require model fine-tuning, demanding considerable hardware resources [22, 30, 32, 35, 55, 33]. This underscores a critical need for developing more efficient, training-free token compression methods specifically for video LLMs, bypassing the need for costly model adaptations and significant hardware investment.

2.2 Visual Token Compression

Token compression [43] has emerged as an effective strategy for reducing token redundancy in vision transformers and large language models. ToMe [4] merges similar tokens in ViTs to alleviate spatial redundancy, while TempMe [45] focuses on minimizing temporal redundancy by merging adjacent video clips. TESTA [40] achieves up to a 75% reduction in processed tokens by employing temporal and spatial aggregation modules. For MLLMs, FastV [6] prunes non-essential visual tokens in early layers of LLM. TopV [64] proposes an optimization framework to prune unnecessary visual tokens. DyMU [57] introduces token merging in the visual encoder and virtual unmerging in the LLM decoder. PDrop [61] performs progressive pruning of tokens at different stages within the LLM. LLaVA-PruMerge [42] and VisionZip [65] leverage attention weight analysis in visual encoders to eliminate spatial redundancy. However, the inherent temporal dependencies between video frames necessitate specialized compression designs. Consequently, recent methods specifically for video token compression have gained increasing attention. DyCoke [48] consolidates tokens across frames and implements dynamic key-value cache reduction. PruneVID [21] clusters video tokens, whereas FastVID [44] enhances compression by combining temporal segmentation with spatio-temporal token merging. In this paper, we propose a new token merging strategy specifically designed for video LLMs, which fully considers spatio-temporal characteristics to maximize performance retention.

3 Method

3.1 Background on Video LLMs Inference

The inference process of video LLMs involves three key stages: before LLM, prefilling, and decoding.

(1) Before LLM. Given an input video with B frames, a vision encoder processes each frame to produce N v embedding vectors. These are projected into the text embedding space, yielding visual tokens H v ∈ R BN v × d , where d represents the dimension of the hidden state space. A text prompt T = { t i } N q i =1 is tokenized and embedded into text tokens H q ∈ R N q × d similarly. Finally, the visual and text tokens are concatenated to form H = concat [ H v , H q ] , which serves as the LLM input.

Prefilling Stage. During prefilling, each transformer layer l of the LLM performs self-attention operations on the concatenated input H . It begins with linear transformations to compute query Q l = H W l Q , key K l = H W l K , and value V l = H W l V , where W l Q , W l K , W l V ∈ R d × d are learnable projection matrices. The resulting key-value pairs ( K l and V l ) are then cached (KV cache) to enhance the efficiency of token generation during the subsequent decoding phase.

Decoding Stage. The decoding stage generates tokens autoregressively, leveraging the KV cache. At each time step t , only the new token h t is processed to compute its key and value representations, avoiding recalculating attention weights over the entire history. The KV cache is updated by appending the new key-value pairs: K ← [ K , h t W K ] , V ← [ V , h t W V ] . This caching mechanism substantially reduces the computational complexity of the generation process.

Temporal Redundancy Merging

1

5

2

4

3

6

1

5

2

4

Temporal Merged Tokens

3

6

Stage I: Temporal Merging

1

2

3

1

2

3

sim>

τ

Redundant

Non-Redundant

1

1

5

2

4

Spatial Merged Tokens

3

6

Stage II: Spatial Merging

1

Redundant

Non-Redundant

1

1

1

Cluster-based Merging

Attention-based Select

Vision Encoder

Tokenizer

Tokens in Frame 1-N

Transformer Block K+1

Transformer Block K

Projector

Stage III: Inner-LLM Merging

…

Text Tokens

…

1

Token of One Frame

Pruned Token of One Frame

merge

…

Non-TopK Attnention Tokens

TopK Attention Tokens

Merged Tokens

Vision Tokens

Pruned Tokens

Temporal Merging

<system> What is Tom doing?

3.2 Global Redundancy-Aware Temporal Merging

Temporal redundancy describes feature persistence at fixed spatial locations across consecutive frames. We identify this redundancy for the k -th feature between frames m and m +1 using their respective feature vectors h m,k and h m +1 ,k . A feature is considered temporally redundant if its normalized similarity, sim ( h m,k , h m +1 ,k ) , exceeds a defined threshold τ ∈ [0 , 1] .

For a temporal segment defined by a start frame t s and an end frame t e (covering [ t s , t e ) ), the total number of prunable tokens, g ( t s , t e ) , is calculated. This involves counting tokens N ( t s , t e ) that are consecutively redundant across all frames from t s to t e -1 , and then multiplying by the number of subsequent frames ( t e -t s -1) within the segment where these tokens can be pruned. Our method prunes the redundant by merging these subsequent occurrence tokens into their first appearance at start frame t s , treating them as temporal redundant tokens , as shown in Fig. 2. The formulation is:

where N v is the total number of features per frame, and I ( · ) the indicator function.

Given a video of B frames, our objective is to find a segmentation into K consecutive segments [ t i , t i +1 ) (with t 1 = 1 , t K +1 = B +1 , and t i < t i +1 ) that maximizes the total prunable features:

This optimization is solved using dynamic programming to achieve global optimization. Let dp [ i ] be the maximum prunable features for a video ending at frame i (exclusive, i.e., considering frames 1 , ..., i -1 ), where frame i marks the exclusive end of the last segment. The value prev [ i ] stores the optimal starting frame j ∗ of this final segment [ j ∗ , i ) . The state transition is given by:

The base case is dp [1] = 0 . The maximum prunable features for the entire video are dp [ B +1] . The optimal segmentation is reconstructed by backtracking from B +1 using the prev array.

3.3 Spatial Merging

After temporal merging, tokens are classified as non-redundant or redundant temporal tokens. We first process the former. Inspired by works [42, 65, 69, 52], we utilize the CLS tokens for spatial feature selection. For vision encoders like Siglip [66] that do not have an explicit CLS token, a method is detailed to derive CLS-equivalent attention. Specifically, we compute the attention matrix:

where d is the state dimension. Token importance is quantified by averaging the attention weights each token receives from all other tokens within the same frame in the vision tower, yielding a score vector A avg ∈ R B × N v . Tokens receiving higher average attention are considered more salient.

Consistent with video LLMs of applying spatial pooling (e.g., after the projector to reduce tokens), we reshape A avg to its original spatial grid dimensions ( H × W = N v ) and apply an analogous pooling operation. This results in a spatially downsampled importance map A avg ∈ R B × H × W . Ultimately, we select the visual features corresponding to the highest scores in A avg as the representative and most informative spatial tokens, known as attention-based select , discarding all others.

The computation of vision tower attention is intra-frame. Averaging these attention weights across frames lacks theoretical justification, invalidating the attention-based method for redundant temporal tokens . To process these features, we employ a cluster-based merging method utilizing density peak clustering based on k -nearest neighbors (DPC-KNN) [13, 41]. Given a set of N redundant temporal tokens [ v 1 , v 2 , ..., v N ] within the first frame of the segmentation. For each token v i , we calculate its local density ρ i , distance to the closest higher-density token δ i and the final density score γ i = ρ i × δ i :

̸

Tokens with high γ i are selected as cluster centers. After selecting the cluster centers, each remaining feature is assigned to the cluster whose center is closest in feature space. Finally, the representative feature for each cluster is then computed by averaging the features assigned to it. Ultimately, the compressed features, derived from this clustering process, along with the non-redundant features, are concatenated according to their original spatial order, thereby preserving positional characteristics.

3.4 Inner-LLM Merging

Inefficient visual attention in large vision language models has been widely discussed [6, 61]. Existing methods, such as FastV [6] and PDrop [61], directly discard redundant visual tokens, which may lead to performance degradation due to information loss. Unlike these approaches, our proposed method addresses it by merging the information from potentially redundant tokens instead of simply discarding them. Specifically, at the K-th layer of the LLM, to reduce the number of visual tokens by R%, we employ a token selection strategy based on attention scores. We use the attention weights of the last token to rank all vision tokens at layer K. The R% of visual tokens exhibiting the lowest attention scores are identified as candidates for merging. We find its most similar visual token within the set of tokens designated for retention. For a retained token v r and its associated set of low attention tokens V m = { v m 1 , v m 2 , ..., v m n } . The updated retained token v ′ r is:

This selective merging preserves relevant features from tokens that would otherwise be removed, mitigating information loss while achieving the desired token reduction.

4 Experimental Results

4.1 Experimental Settings

Benchmarks. We evaluate our method on four widely-used video understanding benchmarks: MVBench [28], EgoSchema [37], LongVideoBench [59], and VideoMME [17]. Comprising videos of varying lengths and complex scenarios, these benchmarks provide a comprehensive testbed for assessing the effectiveness and generalization of our method.

Compared Methods. We compare our proposed HoliTom against 6 strong training-free baselines: 1) FastV [6], identifies key tokens during prefilling using attention scores between predicted and vision tokens; 2) PDrop [61], prunes visual tokens within partitioned LLM stages, guided by image and instruction tokens; 3) Visionzip [65], prunes tokens before LLM via spatial token merging; 4) DyCoke [48], employs temporal merging before LLM and dynamic KV cache pruning in decoding; 5) PruneVid [21], minimizes video redundancy via spatio-temporal token clustering; and 6) FastVID [44], a concurrent work, partitions videos and applies density-based token pruning. Due to the lack of public code, we compare FastVID to its reported results. For all other baselines and our method, experiments use their open-source code under identical hardware condition.

Inference Cost Evaluation. We evaluate the inference cost of transformer layers, each composed of multi-head attention (MHA) and feed-forward network (FFN) modules. Following previous work [6, 61, 48], the FLOPs for processing n i vision tokens in layer i , with hidden state size d and FFN intermediate size m , are defined as 4 n i d 2 +2 n 2 i d +2 n i dm . For an LLM with T transformer layers, the total FLOPs span the prefilling and decoding phases, calculated as:

For consistency, the decoding calculation is fixed for predicting R = 100 tokens, accounting for the the KV cache. In video LLMs, the decoding phase FLOPs contribute only approximately 2% of the total. Consequently, our primary optimization focus is on the prefilling stage. When considering prefilling optimization, inner-LLM pruning methods like FastV [6], the FLOPs incurred in the first 2 shallow layers can amount to 2.9 TFLOPs in LLaVA-OneVision-7B. Even pruning 100% token in the layer, these methods cannot match the potential efficiency compared to outer-LLM pruning methods. Thus, outer-LLM pruning offers a more impactful optimization approach for this domain.

Implementation Details. Our method is implemented on LLaVA-OneVision-7B/72B [25] and LLaVA-Video-7B [70] models. Evaluation uses NVIDIA A100 GPUs, while inference is on an RTX A6000. Inference cost is measured by prefilling FLOPs, with baselines configured for comparable FLOPs (details in the appendix A). The default τ is 0.8; for 10% compression, τ is 0.65. Following official practice, LLaV A-OneVision models utilize 32 input video frames ( N v = 196), while LLaVAVideo uses 64 frames ( N v = 169). All benchmarks are conducted using LMMs-Eval [68, 24].

Model

Method

Prefilling

FLOPs (T)

↓

FLOPs

Ratio

↓

Before LLM

MVBench

↑

EgoSchema

↑

LongVideo

Bench

↑

VideoMME

↑

↑

Score

%

LLaVA-

OneVision-72B

Vanilla

429.3

100%

100%

60.9

61.1

62.7

65.7

62.6

100

FastV [6]

86.4

20.1%

100%

56.1

57.1

57.0

61.2

57.9

92.5

Visionzip [65]

59.0

13.7%

15%

58.4

59.3

57.4

63.8

59.7

95.4

PruneVid [21]

59.0

13.7%

15%

56.8

57.7

57.4

62.7

58.6

93.6

HoliTom (w/o M)

59.0

13.7%

15%

58.5

60.0

57.8

64.1

60.1

96.0

HoliTom

51.6

12.0%

15%

58.7

60.1

57.2

64.3

60.1

96.0

LLaVA-

Video-7B

Vanilla

80.2

100%

100%

60.4

57.2

58.9

64.3

60.2

100

FastV [6]

17.1

21.3%

100%

54.3

54.1

55.0

58.8

55.6

92.4

PDrop [61]

19.5

24.3%

100%

55.9

54.3

54.7

61.9

56.7

94.2

VisionZip [65]

9.3

11.6%

15%

56.7

54.7

54.7

60.7

56.7

94.2

HoliTom (w/o M)

9.3

11.6%

15%

57.8

54.8

55.6

61.9

57.5

95.5

HoliTom

7.6

9.5%

15%

57.7

54.8

56.2

62.1

57.7

95.8





















































































































4.2 Main Results

Comparison with State-of-the-Art Methods. Tab. 2 benchmarks Holitom against state-of-the-art approaches on the LLaVA-OneVision-7B model, analyzing performance and inference cost (FLOPs) at various token retention ratios (25%, 20%, 15%, and 10%) prior to LLM processing. Inner-LLM pruning methods, such as FastV [6] and PDrop [61], often struggle to balance performance and efficiency, especially at lower token retention ratios (25%). DyCoke [48], which segments video frames into groups of 4 and prunes all but the first frame, is limited by its design, capping its lowest retention ratio at 25%. Spatial pruning methods like VisionZip [65] show a significant performance drop (up to 8.4%) at 10% retention. This decline stems from relying solely on spatial compression, less effective at preserving crucial temporal information needed for performance under aggressive pruning. Crucially, even without our inner-LLM merging technique, our method achieves state-ofthe-art performance and efficiency consistently across the evaluated retention ratios. This highlights the superior robustness and adaptability of our approach compared to prior methods. Our inner-LLM merging method further enhances efficiency, driving optimization further. For instance, we retain only 6.9% of the original FLOPs, while preserving 99.1% of the baseline performance.

Performance Comparison Across Different Backbones. Tab. 3 assesses our method's performance across various backbones. For the powerful LLaVA-OneVision-72B model, sensitive to aggressive compression, our approach reduces computational cost to 11.3%, keeping 96% of its original performance. LLaVA-Video-7B presents a greater compression challenge due to its higher initial pooling rate (169 vs. 196 tokens/frame in LLaVA-OneVision). Despite this, our method achieves a reduction to just 9.5% of the original FLOPs, retaining 95.8% performance and outperforming existing methods. Overall, achieving significant token compression with minimal performance drop is indeed tougher for LLaVA-OneVision-72B and LLaVA-Video-7B than for LLaVA-OV-7B.

HoliTom vs. FastV under Outer-LLM Compression. Building on the challenges faced by innerLLM pruning methods discussed in Section 4.1, we compare our inner-LLM merging method with FastV, specifically in scenarios where outer-LLM compression is already applied. In this compressed context, the property " an image is worth 1/2 tokens after layer 2 " [6] is not consistently observed. This is because outer-LLM compression concentrates information, making trivial token discarding more difficult using attention mechanisms. As illustrated in Fig. 3, our method demonstrates superior performance compared to FastV when pruning 50% at shallower layers. Furthermore, at equivalent layers, our approach consistently surpasses FastV across a wide range of pruning rates, underscoring its effectiveness. This effectiveness stems from our inner-LLM merging method, which better preserves information rather than directly discarding it.









































 



 

























































































Performance Scaling with more frames. Our method scales performance robustly with increasing input frames (Fig. 4). A challenge for video LLMs is that uniformly sampled frames may miss crucial information required for accurate answers. Therefore, an effective token pruning method is essential to process more frames and capture sufficient context. Fig. 4 shows our approach consistently outperforms other compression methods across frame rates. At 16 frames, where less temporal redundancy exists, our method, while slightly below the vanilla, still outperforms all other compression techniques. With 64 frames, our method is more efficient and achieves superior performance over the vanilla model. Furthermore, when processing 128 frames, our token compression approach avoids the maximum context length limitations that bottleneck vanilla models. This capability is particularly beneficial for tasks that require an extensive temporal context or to answer complex questions with long text, resulting in improved performance.

Discussion: Improved Performance after Token Compression Tabs. 2, 4, and Fig. 4 present a key finding: models employing our token compression technique outperform the original models on various benchmarks. This surprising result underscores a fundamental principle for achieving superior performance at the input stage: the value of key information over exhaustive information . Excessive, irrelevant, or redundant data acts as noise, obscuring essential signals critical for effective processing. This information overload impedes the capacity of the model to accurately identify and process critical details, thereby degrading understanding and response generation. By providing a refined input that retains pertinent information while shedding redundant information, our compression method facilitates deeper comprehension and yields more accurate, relevant outputs. Collectively, these results underscore the efficacy of our technique in distilling key information and demonstrate that intelligent input refinement is crucial for superior model performance.

4.3 Efficiency Results

Fig. 5 summarizes the impact of various token compression methods on the inference efficiency of video LLMs. As shown, all the evaluated methods demonstrably reduce LLM prefilling time. Our method, in particular, reduces it to just 13.7% of the original. For VisionZip [65], PruneVid [21], and our HoliTom require token pre-processing, which introduces additional "other" time. Furthermore, both PruneVid and our method produce a variable number of tokens per frame, complicating batch processing, which contributes to extra overhead. Our method, designed to maximize temporal redundancy pruning, leads to finer-grained segmentation, further influencing this observation. Nevertheless, our method achieves the maximum reduction in Time-To-First-Token latency, reducing it by 2.28 × , while maintaining optimal performance. Although we did not specifically optimize for decoding, our model's decoding speed still benefits from the reduced number of vision tokens. Our throughput increased by 1.32 × compared to the original model, the highest among all methods evaluated.

0.2

0.4

0.6

0.8

0

10

20

Count

Mean: 43.0%

MVBench

0.0

0.2

0.4

0.6

0

50

100

Mean: 9.3%

EgoSchema

0.0

0.2

0.4

0.6

0

10

20

Mean: 19.1%

LongVideoBench

0.0

0.2

0.4

0.6

0

50

Mean: 10.5%

VideoMME

4.4 Ablation Study

Ablation study on merging modules. Tab. 4 provides a detailed ablation study on the contribution of our proposed merging modules. We first evaluated the temporal merging module ( τ = 0 . 8 ), designed to eliminate temporal redundancy, which demonstrated efficiency gains while preserving performance. Across the four benchmarks, our method achieved 100.7% of the baseline performance while reducing FLOPs to 75.7%. Note that the reported average pruning rate is calculated over four datasets. The average pruning rate varied across datasets is illustrated in Fig. 6. For instance, MVBench(16s), with its shortest duration, exhibits the highest temporal redundancy, allowing approximately 43% pruning, whereas EgoSchema contains the least, permitting only about 9.3%. We then investigate combining temporal with spatial pruning. Applying our temporal pruning method significantly mitigates the performance degradation typically associated with aggressive spatial pruning alone. Furthermore, incorporating the inner merging module allowed us to push the efficiency boundaries even further, ultimately retaining 99.7% performance with a mere 10.5% of the original FLOPs.

Ablation study on temporal segmentation method. Tab. 5 compares different temporal segmentation methods. Fixed-interval Segmentation generates 8 segments with an interval of 4. DySeg [44] selects segment start points using the 8 largest inter-frame differences and includes frames below a 0.90 similarity threshold. Our proposed global redundancy-aware segmentation maximally leverages spatial redundancy and achieves a better performance.

Ablation study on τ . The hyperparameter τ controls the sensitivity of the temporal pruning mechanism, with lower values leading to more aggressive pruning. For a fixed retained ratio, τ also governs the balance between the amount of spatial and temporal pruning applied. Fig. 7, demonstrates the easy tunability of τ , with peak performance observed around τ = 0 . 8 . This value is adopted uniformly without performance degradation, as shown in Tab. 4. For a 10% pruning target, we set τ = 0 . 65 to mitigate performance degradation from aggressive spatial pruning.

5 Conclusion

This paper presents HoliTom , a new training-free holistic token merging framework for boosting the efficiency of video LLMs by effectively handling redundant visual tokens. HoliTom achieves this through a synergistic integration of outer-LLM spatio-temporal reduction, drastically reducing initial token counts, and a robust inner-LLM token merging mechanism tailored for compatibility and further optimization. Evaluated on prominent video LLMs, HoliTom achieves a state-of-the-art efficiency-performance trade-off, substantially reducing computational costs (e.g., to 6.9% FLOPs) while preserving high performance (e.g., 99.1% accuracy), and accelerating inference (2.28 × TTFT, 1.32 × throughput). These results underscore the effectiveness of HoliTom in enabling practical and efficient video LLMs inference for complex, long-form video understanding.























































Acknowledgment

This paper is supported by Young Scientists Fund of the National Natural Science Foundation of China (No. 62506305), Zhejiang Leading Innovative and Entrepreneur Team Introduction Program (No. 2024R01007), Key Research and Development Program of Zhejiang Province (No. 2025C01026), Scientific Research Project of Westlake University (No. WU2025WF003), Chinese Association for Artificial Intelligence (CAAI) & Ant Group Research Fund - AGI Track (No. 2025CAAI-ANT-13), and the Special Support Talents Program of "Xi Hu Ming Zhu Program" in Hangzhou.

References

Kirolos Ataallah, Xiaoqian Shen, Eslam Abdelrahman, Essam Sleiman, Deyao Zhu, Jian Ding, and Mohamed Elhoseiny. Minigpt4-video: Advancing multimodal llms for video understanding with interleaved visual-textual tokens. arXiv preprint arXiv:2404.03413 , 2024.

Jinze Bai, Shuai Bai, Shusheng Yang, Shijie Wang, Sinan Tan, Peng Wang, Junyang Lin, Chang Zhou, and Jingren Zhou. Qwen-vl: A versatile vision-language model for understanding, localization, text reading, and beyond. arXiv preprint arXiv:2308.12966 , 2023.

Shuai Bai, Keqin Chen, Xuejing Liu, Jialin Wang, Wenbin Ge, Sibo Song, Kai Dang, Peng Wang, Shijie Wang, Jun Tang, Humen Zhong, Yuanzhi Zhu, Mingkun Yang, Zhaohai Li, Jianqiang Wan, Pengfei Wang, Wei Ding, Zheren Fu, Yiheng Xu, Jiabo Ye, Xi Zhang, Tianbao Xie, Zesen Cheng, Hang Zhang, Zhibo Yang, Haiyang Xu, and Junyang Lin. Qwen2.5-vl technical report. arXiv preprint arXiv:2502.13923 , 2025.

Daniel Bolya, Cheng-Yang Fu, Xiaoliang Dai, Peizhao Zhang, Christoph Feichtenhofer, and Judy Hoffman. Token merging: Your vit but faster. In ICLR , 2023.

Joya Chen, Zhaoyang Lv, Shiwei Wu, Kevin Qinghong Lin, Chenan Song, Difei Gao, Jia-Wei Liu, Ziteng Gao, Dongxing Mao, and Mike Zheng Shou. Videollm-online: Online video large language model for streaming video. In CVPR , 2024.

Liang Chen, Haozhe Zhao, Tianyu Liu, Shuai Bai, Junyang Lin, Chang Zhou, and Baobao Chang. An image is worth 1/2 tokens after layer 2: Plug-and-play inference acceleration for large vision-language models. In ECCV , 2024.

Xueyi Chen, Keda Tao, Kele Shao, and Huan Wang. Streamingtom: Streaming token compression for efficient video understanding. arXiv preprint arXiv:2510.18269 , 2025.

Zhe Chen, Weiyun Wang, Yue Cao, Yangzhou Liu, Zhangwei Gao, Erfei Cui, Jinguo Zhu, Shenglong Ye, Hao Tian, Zhaoyang Liu, et al. Expanding performance boundaries of open-source multimodal models with model, data, and test-time scaling. arXiv preprint arXiv:2412.05271 , 2024.

Zesen Cheng, Sicong Leng, Hang Zhang, Yifei Xin, Xin Li, Guanzheng Chen, Yongxin Zhu, Wenqi Zhang, Ziyang Luo, Deli Zhao, et al. Videollama 2: Advancing spatial-temporal modeling and audio understanding in video-llms. arXiv preprint arXiv:2406.07476 , 2024.

Rohan Choudhury, Guanglei Zhu, Sihan Liu, Koichiro Niinuma, Kris Kitani, and László Jeni. Don't look twice: Faster video transformers with run-length tokenization. In NeurIPS , 2024.

Tri Dao. FlashAttention-2: Faster attention with better parallelism and work partitioning. In ICLR , 2024.

Tri Dao, Daniel Y. Fu, Stefano Ermon, Atri Rudra, and Christopher Ré. FlashAttention: Fast and memory-efficient exact attention with IO-awareness. In NeurIPS , 2022.

Mingjing Du, Shifei Ding, and Hongjie Jia. Study on density peaks clustering based on k-nearest neighbors and principal component analysis. Knowledge-Based Systems , 99:135-145, 2016.

Sicheng Feng, Kaiwen Tuo, Song Wang, Lingdong Kong, Jianke Zhu, and Huan Wang. Rewardmap: Tackling sparse rewards in fine-grained visual reasoning via multi-stage reinforcement learning. arXiv preprint arXiv:2510.02240 , 2025.

Sicheng Feng, Song Wang, Shuyi Ouyang, Lingdong Kong, Zikai Song, Jianke Zhu, Huan Wang, and Xinchao Wang. Can mllms guide me home? a benchmark study on fine-grained visual reasoning from transit maps. arXiv preprint arXiv:2505.18675 , 2025.

Elias Frantar, Saleh Ashkboos, Torsten Hoefler, and Dan Alistarh. GPTQ: Accurate post-training compression for generative pretrained transformers. In ICLR , 2023.

Chaoyou Fu, Yuhan Dai, Yongdong Luo, Lei Li, Shuhuai Ren, Renrui Zhang, Zihan Wang, Chenyu Zhou, Yunhang Shen, Mengdan Zhang, et al. Video-mme: The first-ever comprehensive evaluation benchmark of multi-modal llms in video analysis. arXiv preprint arXiv:2405.21075 , 2024.

Zhangwei Gao, Zhe Chen, Erfei Cui, Yiming Ren, Weiyun Wang, Jinguo Zhu, Hao Tian, Shenglong Ye, Junjun He, Xizhou Zhu, et al. Mini-internvl: a flexible-transfer pocket multimodal model with 5% parameters and 90% performance. Visual Intelligence , 2(1):32, 2024.

Yuhang Han, Xuyang Liu, Zihan Zhang, Pengxiang Ding, Junjie Chen, Donglin Wang, Honggang Chen, Qingsen Yan, and Siteng Huang. Filter, correlate, compress: Training-free token reduction for mllm acceleration. arXiv preprint arXiv:2411.17686 , 2024.

Wei Huang, Xingyu Zheng, Xudong Ma, Haotong Qin, Chengtao Lv, Hong Chen, Jie Luo, Xiaojuan Qi, Xianglong Liu, and Michele Magno. An empirical study of llama3 quantization: From llms to mllms. Visual Intelligence , 2(1):36, 2024.

Xiaohu Huang, Hao Zhou, and Kai Han. Prunevid: Visual token pruning for efficient video large language models. arXiv preprint arXiv:2412.16117 , 2024.

Peng Jin, Ryuichi Takanobu, Wancai Zhang, Xiaochun Cao, and Li Yuan. Chat-univi: Unified visual representation empowers large language models with image and video understanding. In CVPR , 2024.

Yang Jin, Zhicheng Sun, Kun Xu, Liwei Chen, Hao Jiang, Quzhe Huang, Chengru Song, Yuliang Liu, Di Zhang, Yang Song, et al. Video-lavit: Unified video-language pre-training with decoupled visual-motional tokenization. In ICML , 2024.

Bo Li, Peiyuan Zhang, Kaichen Zhang, Fanyi Pu, Xinrun Du, Yuhao Dong, Haotian Liu, Yuanhan Zhang, Ge Zhang, Chunyuan Li, and Ziwei Liu. Lmms-eval: Accelerating the development of large multimoal models, 2024.

Bo Li, Yuanhan Zhang, Dong Guo, Renrui Zhang, Feng Li, Hao Zhang, Kaichen Zhang, Peiyuan Zhang, Yanwei Li, Ziwei Liu, et al. Llava-onevision: Easy visual task transfer. arXiv preprint arXiv:2408.03326 , 2024.

Junnan Li, Dongxu Li, Silvio Savarese, and Steven Hoi. Blip-2: Bootstrapping language-image pre-training with frozen image encoders and large language models. In ICML , 2023.

KunChang Li, Yinan He, Yi Wang, Yizhuo Li, Wenhai Wang, Ping Luo, Yali Wang, Limin Wang, and Yu Qiao. Videochat: Chat-centric video understanding. arXiv preprint arXiv:2305.06355 , 2023.

Kunchang Li, Yali Wang, Yinan He, Yizhuo Li, Yi Wang, Yi Liu, Zun Wang, Jilan Xu, Guo Chen, Ping Luo, et al. Mvbench: A comprehensive multi-modal video understanding benchmark. In CVPR , 2024.

Xinhao Li, Yi Wang, Jiashuo Yu, Xiangyu Zeng, Yuhan Zhu, Haian Huang, Jianfei Gao, Kunchang Li, Yinan He, Chenting Wang, et al. Videochat-flash: Hierarchical compression for long-context video modeling. arXiv preprint arXiv:2501.00574 , 2024.

Yanwei Li, Chengyao Wang, and Jiaya Jia. Llama-vid: An image is worth 2 tokens in large language models. In ECCV , 2024.

Ji Lin, Jiaming Tang, Haotian Tang, Shang Yang, Wei-Ming Chen, Wei-Chen Wang, Guangxuan Xiao, Xingyu Dang, Chuang Gan, and Song Han. Awq: Activation-aware weight quantization for on-device llm compression and acceleration. In MLSys , 2024.

Ji Lin, Hongxu Yin, Wei Ping, Pavlo Molchanov, Mohammad Shoeybi, and Song Han. Vila: On pre-training for visual language models. In CVPR , 2024.

Xuyang Liu, Yiyu Wang, Junpeng Ma, and Linfeng Zhang. Video compression commander: Plug-and-play inference acceleration for video large language models. arXiv preprint arXiv:2505.14454 , 2025.

Xuyang Liu, Ziming Wang, Junjie Chen, Yuhang Han, Yingyao Wang, Jiale Yuan, Jun Song, Linfeng Zhang, Siteng Huang, and Honggang Chen. Global compression commander: Plug-andplay inference acceleration for high-resolution large vision-language models. arXiv preprint arXiv:2501.05179 , 2025.

Zhijian Liu, Ligeng Zhu, Baifeng Shi, Zhuoyang Zhang, Yuming Lou, Shang Yang, Haocheng Xi, Shiyi Cao, Yuxian Gu, Dacheng Li, et al. Nvila: Efficient frontier visual language models. arXiv preprint arXiv:2412.04468 , 2024.

Yongdong Luo, Wang Chen, Xiawu Zheng, Weizhong Huang, Shukang Yin, Haojia Lin, Chaoyou Fu, Jinfa Huang, Jiayi Ji, Jiebo Luo, et al. Quota: Query-oriented token assignment via cot query decouple for long video comprehension. arXiv preprint arXiv:2503.08689 , 2025.

Karttikeya Mangalam, Raiymbek Akshulakov, and Jitendra Malik. Egoschema: A diagnostic benchmark for very long-form video language understanding. In NeurIPS , 2023.

Rui Qian, Shuangrui Ding, Xiaoyi Dong, Pan Zhang, Yuhang Zang, Yuhang Cao, Dahua Lin, and Jiaqi Wang. Dispider: Enabling video llms with active real-time interaction via disentangled perception, decision, and reaction. In CVPR , 2025.

Rui Qian, Xiaoyi Dong, Pan Zhang, Yuhang Zang, Shuangrui Ding, Dahua Lin, and Jiaqi Wang. Streaming long video understanding with large language models. In NeurIPS , 2024.

Shuhuai Ren, Sishuo Chen, Shicheng Li, Xu Sun, and Lu Hou. Testa: Temporal-spatial token aggregation for long-form video-language understanding. In EMNLP , 2023.

Alex Rodriguez and Alessandro Laio. Clustering by fast search and find of density peaks. science , 344(6191):1492-1496, 2014.

Yuzhang Shang, Mu Cai, Bingxin Xu, Yong Jae Lee, and Yan Yan. Llava-prumerge: Adaptive token reduction for efficient large multimodal models. arXiv preprint arXiv:2403.15388 , 2024.

Kele Shao, Keda Tao, Kejia Zhang, Sicheng Feng, Mu Cai, Yuzhang Shang, Haoxuan You, Can Qin, Yang Sui, and Huan Wang. When tokens talk too much: A survey of multimodal long-context token compression across images, videos, and audios. arXiv , 2025.

Leqi Shen, Guoqiang Gong, Tao He, Yifeng Zhang, Pengzhang Liu, Sicheng Zhao, and Guiguang Ding. Fastvid: Dynamic density pruning for fast video large language models. arXiv preprint arXiv:2503.11187 , 2025.

Leqi Shen, Tianxiang Hao, Sicheng Zhao, Yifeng Zhang, Pengzhang Liu, Yongjun Bao, and Guiguang Ding. Tempme: Video temporal token merging for efficient text-video retrieval. In ICLR , 2025.

Yan Shu, Zheng Liu, Peitian Zhang, Minghao Qin, Junjie Zhou, Zhengyang Liang, Tiejun Huang, and Bo Zhao. Video-xl: Extra-long vision language model for hour-scale video understanding. In CVPR , 2025.

Enxin Song, Wenhao Chai, Guanhong Wang, Yucheng Zhang, Haoyang Zhou, Feiyang Wu, Haozhe Chi, Xun Guo, Tian Ye, Yanting Zhang, et al. Moviechat: From dense token to sparse memory for long video understanding. In CVPR , 2024.

Keda Tao, Can Qin, Haoxuan You, Yang Sui, and Huan Wang. Dycoke: Dynamic compression of tokens for fast video large language models. In CVPR , 2025.

Keda Tao, Kele Shao, Bohan Yu, Weiqiang Wang, Huan Wang, et al. Omnizip: Audioguided dynamic token compression for fast omnimodal large language models. arXiv preprint arXiv:2511.14582 , 2025.

Keda Tao, Haoxuan You, Yang Sui, Can Qin, and Huan Wang. Plug-and-play 1. x-bit kv cache quantization for video large language models. arXiv preprint arXiv:2503.16257 , 2025.

Xiaoguang Tu, Zhi He, Yi Huang, Zhi-Hao Zhang, Ming Yang, and Jian Zhao. An overview of large ai models and their applications. Visual Intelligence , 2(1):34, 2024.

Ao Wang, Fengyuan Sun, Hui Chen, Zijia Lin, Jungong Han, and Guiguang Ding. [cls] token tells everything needed for training-free efficient mllms. arXiv preprint arXiv:2412.05819 , 2024.

Haicheng Wang, Zhemeng Yu, Gabriele Spadaro, Chen Ju, Victor Quétu, Shuai Xiao, and Enzo Tartaglione. Folder: Accelerating multi-modal large language models with enhanced performance. arXiv preprint arXiv:2501.02430 , 2025.

Hanzhen Wang, Jiaming Xu, Jiayi Pan, Yongkang Zhou, and Guohao Dai. Specprune-vla: Accelerating vision-language-action models via action-aware self-speculative pruning. arXiv preprint arXiv:2509.05614 , 2025.

Jiawei Wang, Liping Yuan, Yuchen Zhang, and Haomiao Sun. Tarsier: Recipes for training and evaluating large video description models. arXiv preprint arXiv:2407.00634 , 2024.

Peng Wang, Shuai Bai, Sinan Tan, Shijie Wang, Zhihao Fan, Jinze Bai, Keqin Chen, Xuejing Liu, Jialin Wang, Wenbin Ge, Yang Fan, Kai Dang, Mengfei Du, Xuancheng Ren, Rui Men, Dayiheng Liu, Chang Zhou, Jingren Zhou, and Junyang Lin. Qwen2-vl: Enhancing visionlanguage model's perception of the world at any resolution. arXiv preprint arXiv:2409.12191 , 2024.

Zhenhailong Wang, Senthil Purushwalkam, Caiming Xiong, Silvio Savarese, Heng Ji, and Ran Xu. Dymu: Dynamic merging and virtual unmerging for efficient vlms. arXiv preprint arXiv:2504.17040 , 2025.

Meng Wei, Chenyang Wan, Xiqian Yu, Tai Wang, Yuqiang Yang, Xiaohan Mao, Chenming Zhu, Wenzhe Cai, Hanqing Wang, Yilun Chen, et al. Streamvln: Streaming vision-and-language navigation via slowfast context modeling. arXiv preprint arXiv:2507.05240 , 2025.

Haoning Wu, Dongxu Li, Bei Chen, and Junnan Li. Longvideobench: A benchmark for long-context interleaved video-language understanding. In NeurIPS , 2024.

Guangxuan Xiao, Ji Lin, Mickael Seznec, Hao Wu, Julien Demouth, and Song Han. Smoothquant: Accurate and efficient post-training quantization for large language models. In ICML , 2023.

Long Xing, Qidong Huang, Xiaoyi Dong, Jiajie Lu, Pan Zhang, Yuhang Zang, Yuhang Cao, Conghui He, Jiaqi Wang, Feng Wu, et al. Pyramiddrop: Accelerating your large vision-language models via pyramid visual redundancy reduction. In CVPR , 2025.

Jiaming Xu, Jiayi Pan, Yongkang Zhou, Siming Chen, Jinhao Li, Yaoxiu Lian, Junyi Wu, and Guohao Dai. Specee: Accelerating large language model inference with speculative early exiting. In ISCA , 2025.

Lin Xu, Yilin Zhao, Daquan Zhou, Zhijie Lin, See Kiong Ng, and Jiashi Feng. Pllava: Parameter-free llava extension from images to videos for video dense captioning. arXiv preprint arXiv:2404.16994 , 2024.

Cheng Yang, Yang Sui, Jinqi Xiao, Lingyi Huang, Yu Gong, Chendi Li, Jinghua Yan, Yu Bai, Ponnuswamy Sadayappan, Xia Hu, et al. Topv: Compatible token pruning with inference time optimization for fast and low-memory multimodal vision language model. In CVPR , 2025.

Senqiao Yang, Yukang Chen, Zhuotao Tian, Chengyao Wang, Jingyao Li, Bei Yu, and Jiaya Jia. Visionzip: Longer is better but not necessary in vision language models. In CVPR , 2025.

Xiaohua Zhai, Basil Mustafa, Alexander Kolesnikov, and Lucas Beyer. Sigmoid loss for language image pre-training. In ICCV , 2023.

Hang Zhang, Xin Li, and Lidong Bing. Video-llama: An instruction-tuned audio-visual language model for video understanding. In EMNLP , 2023.

Kaichen Zhang, Bo Li, Peiyuan Zhang, Fanyi Pu, Joshua Adrian Cahyono, Kairui Hu, Shuai Liu, Yuanhan Zhang, Jingkang Yang, Chunyuan Li, and Ziwei Liu. Lmms-eval: Reality check on the evaluation of large multimodal models, 2024.

Qizhe Zhang, Aosong Cheng, Ming Lu, Zhiyong Zhuo, Minqi Wang, Jiajun Cao, Shaobo Guo, Qi She, and Shanghang Zhang. [cls] attention is all you need for training-free visual token pruning: Make vlm inference faster. arXiv preprint arXiv:2412.01818 , 2024.

Yuanhan Zhang, Jinming Wu, Wei Li, Bo Li, Zejun Ma, Ziwei Liu, and Chunyuan Li. Video instruction tuning with synthetic data. arXiv preprint arXiv:2410.02713 , 2024.

NeurIPS Paper Checklist

1. Claims

Question: Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?

Answer: [Yes]

Justification: The abstract and introduction clearly outline the problem, propose a new solution combining external and internal pruning, and list key contributions that align with the methods and goals described in the text.

Guidelines:

The answer NA means that the abstract and introduction do not include the claims made in the paper.

The abstract and/or introduction should clearly state the claims made, including the contributions made in the paper and important assumptions and limitations. A No or NA answer to this question will not be perceived well by the reviewers.

The claims made should match theoretical and experimental results, and reflect how much the results can be expected to generalize to other settings.

It is fine to include aspirational goals as motivation as long as it is clear that these goals are not attained by the paper.

2. Limitations

Question: Does the paper discuss the limitations of the work performed by the authors?

Answer: [Yes]

Justification: Appendix D details the limitations of our current method and outlines future research directions.

Guidelines:

The answer NA means that the paper has no limitation while the answer No means that the paper has limitations, but those are not discussed in the paper.

The authors are encouraged to create a separate "Limitations" section in their paper.

The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (e.g., independence assumptions, noiseless settings, model well-specification, asymptotic approximations only holding locally). The authors should reflect on how these assumptions might be violated in practice and what the implications would be.

The authors should reflect on the scope of the claims made, e.g., if the approach was only tested on a few datasets or with a few runs. In general, empirical results often depend on implicit assumptions, which should be articulated.

The authors should reflect on the factors that influence the performance of the approach. For example, a facial recognition algorithm may perform poorly when image resolution is low or images are taken in low lighting. Or a speech-to-text system might not be used reliably to provide closed captions for online lectures because it fails to handle technical jargon.

The authors should discuss the computational efficiency of the proposed algorithms and how they scale with dataset size.

If applicable, the authors should discuss possible limitations of their approach to address problems of privacy and fairness.

While the authors might fear that complete honesty about limitations might be used by reviewers as grounds for rejection, a worse outcome might be that reviewers discover limitations that aren't acknowledged in the paper. The authors should use their best judgment and recognize that individual actions in favor of transparency play an important role in developing norms that preserve the integrity of the community. Reviewers will be specifically instructed to not penalize honesty concerning limitations.

3. Theory assumptions and proofs

Question: For each theoretical result, does the paper provide the full set of assumptions and a complete (and correct) proof?

Answer: [NA]

Justification: No formal theorem or proposition is provided in the main manuscript.

Guidelines:

The answer NA means that the paper does not include theoretical results.

All the theorems, formulas, and proofs in the paper should be numbered and crossreferenced.

All assumptions should be clearly stated or referenced in the statement of any theorems.

The proofs can either appear in the main paper or the supplemental material, but if they appear in the supplemental material, the authors are encouraged to provide a short proof sketch to provide intuition.

Inversely, any informal proof provided in the core of the paper should be complemented by formal proofs provided in appendix or supplemental material.

Theorems and Lemmas that the proof relies upon should be properly referenced.

4. Experimental result reproducibility

Question: Does the paper fully disclose all the information needed to reproduce the main experimental results of the paper to the extent that it affects the main claims and/or conclusions of the paper (regardless of whether the code and data are provided or not)?

Answer: [Yes]

Justification: We have made our best efforts to ensure the reproducibility of our method. The algorithmic details are provided in Section 3, and the experimental setup and procedures are detailed in Section 4.1 and Appendix A.

Guidelines:

The answer NA means that the paper does not include experiments.

If the paper includes experiments, a No answer to this question will not be perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data are provided or not.

If the contribution is a dataset and/or model, the authors should describe the steps taken to make their results reproducible or verifiable.

Depending on the contribution, reproducibility can be accomplished in various ways. For example, if the contribution is a novel architecture, describing the architecture fully might suffice, or if the contribution is a specific model and empirical evaluation, it may be necessary to either make it possible for others to replicate the model with the same dataset, or provide access to the model. In general. releasing code and data is often one good way to accomplish this, but reproducibility can also be provided via detailed instructions for how to replicate the results, access to a hosted model (e.g., in the case of a large language model), releasing of a model checkpoint, or other means that are appropriate to the research performed.

While NeurIPS does not require releasing code, the conference does require all submissions to provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution. For example

(a) If the contribution is primarily a new algorithm, the paper should make it clear how to reproduce that algorithm.

(b) If the contribution is primarily a new model architecture, the paper should describe the architecture clearly and fully.

(c) If the contribution is a new model (e.g., a large language model), then there should either be a way to access this model for reproducing the results or a way to reproduce the model (e.g., with an open-source dataset or instructions for how to construct the dataset).

(d) We recognize that reproducibility may be tricky in some cases, in which case authors are welcome to describe the particular way they provide for reproducibility. In the case of closed-source models, it may be that access to the model is limited in some way (e.g., to registered users), but it should be possible for other researchers to have some path to reproducing or verifying the results.

5. Open access to data and code

Question: Does the paper provide open access to the data and code, with sufficient instructions to faithfully reproduce the main experimental results, as described in supplemental material?

Answer: [No]

Justification: The datasets are from publicly available online sources. We guarantee the code will be open-sourced after the review period.

Guidelines:

The answer NA means that paper does not include experiments requiring code.

Please see the NeurIPS code and data submission guidelines ( https://nips.cc/ public/guides/CodeSubmissionPolicy ) for more details.

While we encourage the release of code and data, we understand that this might not be possible, so 'No' is an acceptable answer. Papers cannot be rejected simply for not including code, unless this is central to the contribution (e.g., for a new open-source benchmark).

The instructions should contain the exact command and environment needed to run to reproduce the results. See the NeurIPS code and data submission guidelines ( https: //nips.cc/public/guides/CodeSubmissionPolicy ) for more details.

The authors should provide instructions on data access and preparation, including how to access the raw data, preprocessed data, intermediate data, and generated data, etc.

The authors should provide scripts to reproduce all experimental results for the new proposed method and baselines. If only a subset of experiments are reproducible, they should state which ones are omitted from the script and why.

At submission time, to preserve anonymity, the authors should release anonymized versions (if applicable).

Providing as much information as possible in supplemental material (appended to the paper) is recommended, but including URLs to data and code is permitted.

6. Experimental setting/details

Question: Does the paper specify all the training and test details (e.g., data splits, hyperparameters, how they were chosen, type of optimizer, etc.) necessary to understand the results?

Answer: [Yes]

Justification: The experimental details, including the model structure, hyperparameters, and comparison methods, have been elaborated in detail in Section 4 and Appendix A.

Guidelines:

The answer NA means that the paper does not include experiments.

The experimental setting should be presented in the core of the paper to a level of detail that is necessary to appreciate the results and make sense of them.

The full details can be provided either with the code, in appendix, or as supplemental material.

7. Experiment statistical significance

Question: Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?

Answer: [No]

Justification: For evaluation, we did not report error bars, consistent with previous work in this area. Due to the computational cost, we cannot replicate the experiments multiple times. This said, we have evaluated our method on multiple datasets and models (see Tab. 2, 3) to confirm the performance robustness of our method.

Guidelines:

The answer NA means that the paper does not include experiments.

The authors should answer "Yes" if the results are accompanied by error bars, confidence intervals, or statistical significance tests, at least for the experiments that support the main claims of the paper.

The factors of variability that the error bars are capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, or overall run with given experimental conditions).

The method for calculating the error bars should be explained (closed form formula, call to a library function, bootstrap, etc.)

The assumptions made should be given (e.g., Normally distributed errors).

It should be clear whether the error bar is the standard deviation or the standard error of the mean.

It is OK to report 1-sigma error bars, but one should state it. The authors should preferably report a 2-sigma error bar than state that they have a 96% CI, if the hypothesis of Normality of errors is not verified.

For asymmetric distributions, the authors should be careful not to show in tables or figures symmetric error bars that would yield results that are out of range (e.g. negative error rates).

If error bars are reported in tables or plots, The authors should explain in the text how they were calculated and reference the corresponding figures or tables in the text.

8. Experiments compute resources

Question: For each experiment, does the paper provide sufficient information on the computer resources (type of compute workers, memory, time of execution) needed to reproduce the experiments?

Answer: [Yes]

Justification: In Section 4.1, we have detailed the computational resources required for our evaluation and inference tests.

Guidelines:

The answer NA means that the paper does not include experiments.

The paper should indicate the type of compute workers CPU or GPU, internal cluster, or cloud provider, including relevant memory and storage.

The paper should provide the amount of compute required for each of the individual experimental runs as well as estimate the total compute.

The paper should disclose whether the full research project required more compute than the experiments reported in the paper (e.g., preliminary or failed experiments that didn't make it into the paper).

9. Code of ethics

Question: Does the research conducted in the paper conform, in every respect, with the NeurIPS Code of Ethics https://neurips.cc/public/EthicsGuidelines ?

Answer: [Yes]

Justification: We believe we have followed the Code of Ethics.

Guidelines:

The answer NA means that the authors have not reviewed the NeurIPS Code of Ethics.

If the authors answer No, they should explain the special circumstances that require a deviation from the Code of Ethics.

The authors should make sure to preserve anonymity (e.g., if there is a special consideration due to laws or regulations in their jurisdiction).

10. Broader impacts

Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?

Answer: [Yes]

Justification: As shown in appendix E, the societal impact of our method has been fully discussed.

Guidelines:

The answer NA means that there is no societal impact of the work performed.

If the authors answer NA or No, they should explain why their work has no societal impact or why the paper does not address societal impact.

Examples of negative societal impacts include potential malicious or unintended uses (e.g., disinformation, generating fake profiles, surveillance), fairness considerations (e.g., deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations.

The conference expects that many papers will be foundational research and not tied to particular applications, let alone deployments. However, if there is a direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that an improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point out that a generic algorithm for optimizing neural networks could enable people to train models that generate Deepfakes faster.

The authors should consider possible harms that could arise when the technology is being used as intended and functioning correctly, harms that could arise when the technology is being used as intended but gives incorrect results, and harms following from (intentional or unintentional) misuse of the technology.

If there are negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition to attacks, mechanisms for monitoring misuse, mechanisms to monitor how a system learns from feedback over time, improving the efficiency and accessibility of ML).

11. Safeguards

Question: Does the paper describe safeguards that have been put in place for responsible release of data or models that have a high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)?

Answer: [NA]

Justification: Our method is training-free and plug-and-play, which means this is no risks.

Guidelines:

The answer NA means that the paper poses no such risks.

Released models that have a high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere to usage guidelines or restrictions to access the model or implementing safety filters.

Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images.

We recognize that providing effective safeguards is challenging, and many papers do not require this, but we encourage authors to take this into account and make a best faith effort.

12. Licenses for existing assets

Question: Are the creators or original owners of assets (e.g., code, data, models), used in the paper, properly credited and are the license and terms of use explicitly mentioned and properly respected?

Answer: [Yes]

Justification: We have cited all datasets and baseline models used in our experiment.

Guidelines:

The answer NA means that the paper does not use existing assets.

The authors should cite the original paper that produced the code package or dataset.

The authors should state which version of the asset is used and, if possible, include a URL.

The name of the license (e.g., CC-BY 4.0) should be included for each asset.

For scraped data from a particular source (e.g., website), the copyright and terms of service of that source should be provided.

If assets are released, the license, copyright information, and terms of use in the package should be provided. For popular datasets, paperswithcode.com/datasets has curated licenses for some datasets. Their licensing guide can help determine the license of a dataset.

For existing datasets that are re-packaged, both the original license and the license of the derived asset (if it has changed) should be provided.

If this information is not available online, the authors are encouraged to reach out to the asset's creators.

13. New assets

Question: Are new assets introduced in the paper well documented and is the documentation provided alongside the assets?

Answer: [NA]

Justification: Our work does not provide new assets.

Guidelines:

The answer NA means that the paper does not release new assets.

Researchers should communicate the details of the dataset/code/model as part of their submissions via structured templates. This includes details about training, license, limitations, etc.

The paper should discuss whether and how consent was obtained from people whose asset is used.

At submission time, remember to anonymize your assets (if applicable). You can either create an anonymized URL or include an anonymized zip file.

14. Crowdsourcing and research with human subjects

Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given to participants and screenshots, if applicable, as well as details about compensation (if any)?

Answer: [NA]

Justification: Our work does not involve any crowd sourcing or experiments with human subjects.

Guidelines:

The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.

Including this information in the supplemental material is fine, but if the main contribution of the paper involves human subjects, then as much detail as possible should be included in the main paper.

According to the NeurIPS Code of Ethics, workers involved in data collection, curation, or other labor should be paid at least the minimum wage in the country of the data collector.

15. Institutional review board (IRB) approvals or equivalent for research with human subjects

Question: Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or an equivalent approval/review based on the requirements of your country or institution) were obtained?

Answer: [NA]

Justification: Our work does not involve experiments with human subjects.

Guidelines:

The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.

Depending on the country in which research is conducted, IRB approval (or equivalent) may be required for any human subjects research. If you obtained IRB approval, you should clearly state this in the paper.

We recognize that the procedures for this may vary significantly between institutions and locations, and we expect authors to adhere to the NeurIPS Code of Ethics and the guidelines for their institution.

For initial submissions, do not include any information that would break anonymity (if applicable), such as the institution conducting the review.

16. Declaration of LLM usage

Question: Does the paper describe the usage of LLMs if it is an important, original, or non-standard component of the core methods in this research? Note that if the LLM is used only for writing, editing, or formatting purposes and does not impact the core methodology, scientific rigorousness, or originality of the research, declaration is not required.

Answer: [NA]

Justification: The method development in this research does not involve LLMs as any important, original, or non-standard components.

Guidelines:

The answer NA means that the core method development in this research does not involve LLMs as any important, original, or non-standard components.

Please refer to our LLM policy ( https://neurips.cc/Conferences/2025/LLM ) for what should or should not be described.

A Supplemental Implementation Details

Our method is implemented on the LLaVA-OneVision-7B/72B [25] and LLaVA-Video-7B [70] models. Evaluation utilized NVIDIA A100 (80GB) GPUs; inference was performed on an NVIDIA RTX A6000 GPU. To ensure a fair comparison of computational cost, we used total prefilling FLOPs as the primary metric. Baselines are configured for comparable FLOPs: FastV [6] prunes 80% of tokens at layer 2; PDrop [61] retains 50%, 25%, and 12.5% of vision tokens at layers 2, 7, and 14, respectively; VisionZip [65] and PruneVid [21] maintain a consistent proportion of input tokens with our method. Performance results for FastVID [44] are adopted directly from their original paper. For our proposed method, the default threshold τ is 0.8. In the specific experiments conducted on Qwen2.5-VL [3] with a maximum sampling of 768 frames, a lower threshold of τ = 0 . 2 was used. This adjustment accounts for the higher temporal redundancy present when sampling is dense. In experiments targeting a 10% compression ratio, τ was set to 0.65. Experimental setups include pruning K=18 layers of the 7B model and K=60 layers of the 72B model, both at a ratio of R=50%. Following the official LLaVA-OneVision specifications, the default input video frames are 32 and N v = 196 . For LLaVA-Video, the default input consisted of 64 video frames with N v = 169 . All benchmark evaluations are performed using the LMMs-Eval [68, 24].

B Supplemental Experimental Results

B.1 Experiments on Qwen2.5-VL with High Frame Sampling

Existing models like LLaVA-OV [25] and LLaVA-Video [70] utilize a fixed input of 32/64 video frames, each resized to a static resolution (Tab. 2, 3). In contrast, frontier models, such as Qwen2.5VL [3], introduce advanced features including FPS frame sampling, which extend input sequences (up to 768 frames), and dynamic resolution support. These new capabilities pose new challenges to existing token compression methods in maintaining performance for video understanding tasks.

As shown in Tab. 6, HoliTom surpasses state-of-the-art methods across both token compression rates, especially for long videos. Due to Out-of-Memory (OOM) issues arising from the full attention matrix calculation, we were unable to report results for FastV and the inner-LLM merging execution.

B.2 Impact of Token Compression on Fine-Grained Object Understanding

HoliTom applies aggressive video token compression. Does this aggressive compression impair the model's ability to comprehend fine-grained details? Tab. 7 presents the performance results on selected subtasks from the MVBench benchmark (originally detailed in Table 2).

0.2

0.4

0.6

0.8

0

20

Count

Mean: 56.2%

MVBench

0.2

0.4

0.6

0

50

Mean: 20.6%

EgoSchema

0.2

0.4

0.6

0.8

0

5

10

Mean: 26.5%

LongVideoBench

0.2

0.4

0.6

0

20

40

Mean: 17.6%

VideoMME

At compression rates ranging from 10% to 25%, HoliTom demonstrates increases in performance. This result would be improbable if HoliTom discards critical information about small objects. Instead, this outcome provides evidence of HoliTom's robust ability to retain fine-grained details.

B.3 Enhanced Performance with Reduced Overhead

Tab. 8 extends the findings presented in Fig. 4. By sampling more frames with HoliTom while maintaining constant or even reduced total FLOPs, we achieve better performance compared to a vanilla model operating on fewer frames. We also observe that as the number of input frames increases, the computational overhead contributed by the vision encoder becomes a non-negligible factor. This performance and efficiency trade-off is further illustrated in Fig. 5 of our paper.

B.4 Supplemental Ablation Study on τ

In section 4.4, we discussed the selection of τ ( τ = 0 . 8 ) and the corresponding histogram of temporal pruning rates on four benchmarks for a retain ratio of 15%. Next, we detail the selection of the hyperparameter τ for a 10% retain ratio and present the corresponding histogram. As illustrated in Fig. 9, peak performance is observed around τ = 0 . 65 . The Fig. 8 presents the histogram of temporal pruning rates on the four datasets when τ = 0 . 65 . It is evident that controlling τ regulates the aggressiveness of temporal pruning; a larger τ results in more aggressive pruning.

B.5 Ablation Study on Merge Strategy

The core of spatio-temporal merging in HoliTom is a hybrid strategy that integrates attention-guided compression with similarity-guided clustering (DPC-KNN). As shown in Tab. 9, our mixed strategy achieves the best performance. The key to this lies in distinguishing between the two token types encountered during the merging process. For non-redundant tokens (within a single frame), the encoder's self-attention scores are excellent priors , making an attention-guided compression strategy highly effective. However, for redundant tokens (formed by merging tokens from adjacent frames), the original single-frame self-attention scores lack a theoretical basis as a merging metric. Therefore,























































DPC-KNN clustering is adopted to group these merged tokens based on feature similarity, which provides a more principled and effective approach in this specific spatio-temporal context.

C Compatible with Flash Attention

Our approach introduces two distinct merging strategies: inner-LLM and outer-LLM. The inner-LLM strategy, similar to prior work [6, 61, 48], is designed for integration with highly optimized attention implementations (e.g., Flash Attention [12, 11]). This requires obtaining attention scores from a specific layer only once during the prefilling stage, an operation introducing negligible computational overhead compared to total inference cost. In contrast, our outer-LLM merging strategy operates externally to the model, decoupled from the attention mechanisms of LLM.

D Limitations and Future Work

While our work demonstrates an adequate acceleration of video LLMs by token merging for inference, it is important to outline its current limitations. First, the approach is primarily designed for fixedlength video clips and does not natively support online, arbitrary-length streaming video input. This poses challenges for real-time processing [5, 39, 38] and maintaining long-term context understanding. Second, as shown in Fig. 5, similar to other methods [6, 61, 65, 48] in the token pruning area, our approach does not optimize the latency of the vision tower. Further work, such as quantization [31, 60, 16, 20], methods to accelerate the vision tower [10, 53, 57] and new application areas [58, 54, 62, 49, 7, 15, 14], is worth exploring for further optimization.

E Broader impacts

This work significantly enhances video LLM efficiency, addressing a key barrier to deployment and scalability. By reducing computational needs, it broadens access to advanced video AI, enabling wider application and fostering innovation.

F More Visualizations

User:

FLOPS

What color is the object that is stationary?

What direction is the green ball moving in?

----------------------------------------------------------------

LLaVA-OV-7B

40.8T

The object that is stationary is

purple

.

The green ball moves towards the

right

side of the frame.

FastV

9.3T

The object that is stationary is

purple

.

The green ball moves towards the

right

side of the frame.

VisionZip

8.7T

The object that is stationary is

green

.

The green ball is moving towards the

right

side of the frame.

HoliTom (w/o M)

8.7T

The object that is stationary is

purple

.

The green ball moves towards the

left

side of the frame.

HoliTom

7.1T

The object that is stationary is

purple

.

The green ball moves towards the

left

side of the frame.

User:

FLOPS

Where is the hidden object at the end of the occlusion game from the camera's point of view?

----------------------------------------------------------------

LLaVA-OV-7B

40.8T

The hidden object is

under the black notebook

.

FastV

9.3T

The hidden object is

under the black notebook

.

VisionZip

8.7T

The hidden object is

under the black notebook

HoliTom (w/o M)

8.7T

The hidden object is

under the middle cup.

HoliTom

7.1T

The hidden object is

under the middle cup.

Top

:

Dynamic Object Tracking

Bottom

:

Memory; Motion Understanding

User:

Describe what's happening in this video.

-------------------------------------------------------------------------

LLaVA-OV-7B

The video features a person standing on a stage with a dark background, wearing a black shirt and dark pants. A large hand appears from the top

left corner, reaching towards the person's waist area. The text 'Anyone know what this pocket is for?' appears at the  bottom of the screen,

indicating curiosity about the purpose of a specific pocket on the person's clothing. The text changes to 'I've always wondered that' as the hand

continues to reach out. Eventually, the hand reveals a small white device attached to the pocket,

which is identified as 'this is the new iPod

Nano 0'. The person holds up the iPod Nano 0, showing it to the audience. The video concludes with a close-up of the person holding the iPod

Nano 0, emphasizing its design and features.

HoliTom

The video features a person standing on a stage, dressed in a dark outfit. A large hand appears from the background, reaching towards the

person. The text 'I've always wondered' (I've always wondered) is displayed at the top left corner of the screen, suggesting a sense of curiosity or

anticipation. As the hand moves closer, the text changes to 'well now we know' (well now we know), indicating a revelation or discovery. The

hand then reveals a small device labeled 'this new iPod Nano' (this new iPod Nano), which is white with a circular button and a small screen. The

text 'iPod Nano' (iPod Nano) appears below the device, confirming its identity. The video concludes with a close-up of the person holding the

iPod Nano, emphasizing the significance of the new product.

40.8

TFLOPS

100%  Token

4.3

TFLOPS

15%

Token

Orgpazkguda

Ivealwoys

Peer reviews:
{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper proposes HoliTom, a training-free token-compression framework that improves the inference-time efficiency of video LLMs. HoliTom combines outer-LLM reduction—which performs global optimization–based temporal merging and attention/cluster-based spatial merging—with inner-LLM reduction, which merges visual tokens at every layer using a fixed retention ratio instead of simply dropping redundant tokens. The method attains state-of-the-art results while requiring fewer FLOPs than prior token-compression approaches.\n\nSTRENGTHS AND WEAKNESSES:\n**Strengths**\n1. **Effectiveness and efficiency.** Both outer-LLM and inner-LLM merging clearly improve performance, as evidenced by the ablations in Table 4. The temporal-merging objective is novel, and solving it via dynamic programming is elegant and effective.\n2. **Comprehensive analysis.** Section 4 offers detailed experiments that thoroughly validate the proposed method. The inference-efficiency study highlights HoliTom’s advantage over previous techniques.\n3. **Clarity of writing.** The paper is well written and easy to follow.\n\n**Weaknesses**\n1. **Missing spatial-merging ablation.** For non-redundant temporal tokens, the paper uses an attention-based strategy; for redundant tokens, it adopts DPC-KNN clustering. An ablation justifying these design choices is needed.\n2. **Lack of qualitative visualizations.** Visual examples showing token reduction at each stage would help readers better understand how each component works.\n3. **Equation (5) needs more explanation.** The derivation and intuition behind this equation should be clarified.\n\nMinor Issues\n- L156: “an” → “a”\n- L198: “spatial-temporal” → “spatio-temporal”\n\nQUESTIONS:\nApart from three questions in weakness part, I have additional following questions:\n\n4. For inner-LLM compression, the method uses the attention score of the “last token.” Does this refer to the last prompt token (visual + text) or the last visual token only? Have the authors compared the two variants, and if so, what did you observe?\n5. Suppose we sample more frames yet keep overall FLOPs constant (eg. 100%) after applying HoliTom. Would this yield better performance than using fewer frames with the vanilla model?\n\nIf the authors address the concerns above, I believe the paper will be further solidified, and I will raise my score.\n\nLIMITATIONS:\n- Multi-round conversations. Can HoliTom handle multi-turn dialogue in video LLMs? If a follow-up query arrives, does it affect the validity of inner-LLM merging?\n- Extremely long videos. Have the authors tested HoliTom on very long inputs (e.g., Qwen 2.5-VL with 768 frames)? What efficiency and accuracy gains does HoliTom provide compared with no reduction in that setting?", "TITLE": "Official Review", "RECOMMENDATION": 5, "REVIEWER_CONFIDENCE": 4, "CLARITY": 4, "IMPACT": 3, "ORIGINALITY": 3, "DATE": "2025-06-21"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper introduces HoliTom, a holistic token merging method for video LLMs that integrates spatial, temporal, and inner-LLM token merging. HoLiTom employs: (1) Global redundancy-aware temporal merging, which reduces temporal redundancy by merging tokens into their first occurrence; (2) Spatial merging via clustering tokens based on global feature representations; and (3) Inner-LLM merging, where redundant tokens are averaged rather than discarded. Evaluated on multiple video understanding benchmarks, HoLiTom achieves high efficiency and strong performance.\n\nSTRENGTHS AND WEAKNESSES:\n**Strengths**\n* The paper is clearly-written and well-organised. The method is quite intuitive.\n* The method considers multiple aspects while improving efficiency, including spatial, temporal and inner-LLM redundancy. Overall, the method offers a comprehensive solution to token redundancy in video understanding tasks.\n* Performance wise, strong performance and high efficiency are obtained as compared to previous methods when evaluating with video LLMs.\n\n**Weaknesses**\n* If I understand correctly, the core idea is to merge/average out redundant information based mainly on the global representations of the visual tokens (either the similarity calculation in temporal merging, CLS replacement in spatial merging or the inner-LLM merging). This approach seems effective for redundancy reduction and is highly favourable in terms of efficiency, but could it risk losing fine details (e.g., small objects) or become overly aggressive for longer videos?\n* Following up on the longer videos, I noticed that on LongVideoBench HoliTom either matches or underperforms compared to prior methods, unlike its superior results on shorter videos (e.g., MVBench). Could you elaborate on this? Is this a limitation inherent to the method’s design, or are there other factors at play?\n* I’m curious about HoliTom’s performance on tasks requiring fine-grained object-level reasoning. Given the method’s aggressive merging strategy, could you discuss whether it preserves sufficient detail for such scenarios? Additional analyses would be helpful.\n\nQUESTIONS:\n* Please refer to the Weaknesses section.\n* If have time, an interesting extension would be evaluating HoliTom on training-free video LLMs (e.g., image-trained models like LLaVA-1.6 applied to videos, e.g. SF-LLaVA [1] or TS-LLaVA). Since these models lack explicit temporal knowledge from video training, does the token merging strategy still prove effective? \n\n\n[1] Mingze Xu, Mingfei Gao, Zhe Gan, Hong-You Chen, Zhengfeng Lai, Haiming Gang, Kai Kang, Afshin Dehghan. (2024) SlowFast-LLaVA: A Strong Training-Free Baseline for Video Large Language Models.\n\n[2] Tingyu Qu, Mingxiao Li, Tinne Tuytelaars, Marie-Francine Moens. (2024) TS-LLaVA: Constructing Visual Tokens through Thumbnail-and-Sampling for Training-Free Video Large Language Models\n\nLIMITATIONS:\nyes", "TITLE": "Official Review", "RECOMMENDATION": 4, "REVIEWER_CONFIDENCE": 4, "CLARITY": 3, "IMPACT": 2, "ORIGINALITY": 2, "DATE": "2025-06-24"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nIn this work, the authors propose a training-free Holistic Token Merging method to boost computation efficiency for video MLLMs. Basically, it consists of three key parts, including Temporal Merging, Spatial Merging, and Inner-LLM Merging.\n\nSTRENGTHS AND WEAKNESSES:\n*Strengths\n\n1 The efficiency of video MLLMs is an important research problem, especially for long video understanding. The authors attempt to address this issue via token merging.\n\n2 The paper is well organized with good structure.\n\n*Weaknesses\n\n1 The novelty is actually limited. The merging strategies ( including temporal, spatial and inner-LLM) in this work are straightforward, according to similarity comparsion. They are not foundamentally different from those merging strategies of the existing methods e.g., VideoChat-Flash, Video-XL, MovieChat, etc. Please clarify the key contributions in the proposed merging method.\n\n2 Moreover, in the experiment section, comparison with these sota methods (e.g., VideoChat-Flash, Video-XL, MovieChat, etc) is missing.  To further show its effectiveness, it is necessary to show such compasion.\n\n3 Besides of video QA, does this method work for other long video understanding tasks like temporal grounding (e.g., Charades-STA)  or  Needle-in-A-Video Haystack tasks in VideoChat-Flash.\n\nQUESTIONS:\nPlease see the weakness section in details.\n\n1 Limited Novelty: Please clarify the key contributions in the proposed merging method, compared to the existing methods, e.g., VideoChat-Flash, Video-XL, MovieChat, etc. \n\n2 Insufficient Comparison: To further show its effectiveness, it is necessary to show compasion with the sota methods metioned above.\n\n3 More Investigation: Video QA is only one task in long video understanding. It is interesting to show if this method works for other long video understanding tasks like temporal grounding or  Needle-in-A-Video Haystack.\n\nLIMITATIONS:\nPlease see the weakness section.", "TITLE": "Official Review", "RECOMMENDATION": 4, "REVIEWER_CONFIDENCE": 4, "CLARITY": 3, "IMPACT": 2, "ORIGINALITY": 2, "DATE": "2025-06-30"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe paper introduces HoliTom, a training-free token merging method to enhance the efficiency of video LLMs by compressing redundant visual tokens during video understanding tasks. The motivation stems from the high computational inefficiency of video LLMs due to the large number of tokens generated from numerous frames. HoliTom proposes a three-stage method: global redundancy-aware temporal merging to reduce temporal redundancy, spatial merging to compress spatial information, and inner-LLM merging to optimize token processing within the LLM. As a result, they reduce FLOPs to 6.9% while maintaining 99.1% of the baseline performance across various benchmarks, and ablation studies confirm the effectiveness of each module and segmentation strategy.\n\nSTRENGTHS AND WEAKNESSES:\nStrengths:\n1. The paper’s use of a combined temporal, spatial, and inner-LLM merging strategy. It is technically sound as it provides a comprehensive solution.\n2. The paper achieves a 6.9% FLOPs reduction while retaining 99.1% performance, making it practical for real-world applications.\nWeakness:\n1. Instead of LLaVA series, applying the method to other video llm structure is necessary to showcase the generalization of the idea. Otherwise, I might suspect that it only works on simple (LLaVA) structures.\n2. The paper lacks the explanation of why without inner-LLM could sometimes get the best performance.\n3. The threshold for in line134 is manually crafted which is not a general method intrinsically.\n4. While FLOPs and TTFT are measured, there’s no mention of actual inference time on different hardware setups. I’d like to see a comparison of real-time latency across GPUs to better assess practical deployment feasibility—can the authors provide this in their rebuttal?\n5. The method handles 64 frames well, but it’s unclear how it performs with extremely long videos. I’m puzzled by this gap—could the authors test HoliTom on longer sequences to evaluate scalability and potential bottlenecks\n\nQUESTIONS:\nSee above\n\nLIMITATIONS:\nSee the weaknesses", "TITLE": "Official Review", "RECOMMENDATION": 3, "REVIEWER_CONFIDENCE": 3, "CLARITY": 3, "IMPACT": 3, "ORIGINALITY": 2, "DATE": "2025-07-03"}