ST-SimDiff: Balancing Spatiotemporal Similarity and Difference for Efficient Video Understanding with MLLMs
Authors: Bingjun Luo
Tony Wang
Chaoqi Chen
Xinpeng Ding
Venue: ICLR 2026 Poster (2026)
Keywords: Video Understanding
Visual Token Reduction
Multimodal Large Language Models
Primary area: applications to computer vision, audio, language, and other modalities
TL;DR: We propose a training-free method that builds a spatio-temporal graph to efficiently select video tokens by incorporating similarity for redundancy reduction and difference for key event detection.

Abstract: Multimodal Large Language Models (MLLMs) face significant computational overhead when processing long videos due to the massive number of visual tokens required. To improve efficiency, existing methods primarily reduce redundancy by pruning or merging tokens based on importance or similarity. However, these approaches largely overlook a critical dimension of video content, i.e., changes and turning points, and they lack a collaborative model for spatio-temporal relationships.
To address this, we propose a new perspective: similarity is for identifying redundancy, while difference is for capturing key events. Based on this, we designed a training-free framework named ST-SimDiff. We first construct a spatio-temporal graph from the visual tokens to uniformly model their complex associations. Subsequently, we employ a parallel dual-selection strategy: 1) similarity-based selection uses community detection to retain representative tokens, compressing static information; 2) temporal difference-based selection precisely locates content-changing points to preserve tokens that capture key dynamic shifts. This allows it to preserve both static and dynamic content with a minimal number of tokens. Extensive experiments show our method significantly outperforms state-of-the-art approaches while substantially reducing computational costs.
Our code is available in [https://github.com/bingjunluo/ST-SimDiff](https://github.com/bingjunluo/ST-SimDiff).

Published as a conference paper at ICLR 2026
ST-SIMDIFF: BALANCING SPATIOTEMPORAL SIMI- LARITY AND DIFFERENCE FOR EFFICIENT VIDEO UN- DERSTANDING WITH MLLMS
Bingjun Luo', Tony Wang', Chaoqi Chen";
Xinpeng Ding
Tsinghua University
Shenzhen University
Xiiiian University
bing jonouuuuoookk com,
tonymang4444mmall.coom com
Sqcchn11994@@ma111 kdingaffsconeettusttt ust.
ABSTRACT
Multimodal Large Language Models (MLLMs) face significant computational overhead when processing long videos due to the massive number of visual tokens required. To improve efficiency, existing methods primarily reduce redundancy by pruning or merging tokens based on importance or similarity. However, these ap- proaches largely overlook a critical dimension of video content, i.e., changes and turning points, and they lack a collaborative model for spatio-temporal relation- ships. To address this, we propose a new perspective: similarity is for identifying redundancy, while difference is for capturing key events. Based on this, we de- signed a training-free framework named ST-SimDiff. We first construct spatio- temporal graph from the visual tokens to uniformly model their complex associ- ations. Subsequently, we employ parallel dual-selection strategy: 1) similarity- based selection uses community detection to retain representative tokens, com- pressing static information; 2) temporal difference- based selection precisely lo- cates content-changing points preserve tokens that capture key dynamic shifts. This allows it to preserve both static and dynamic content with minimal num- ber of tokens. Extensive experiments show method significantly outperforms state-of-the-ard approaches while substantially reducing computational costs. Our code is available in http::/siiibbb.ooobiiiiuuuuuSSii
INTRODUCTION
The rise of Large Language Models (LLMs) has significantly propelled advancements of Large Vision-Language Models (LVLMss) which have demonstrated remarkable capabilities in image and video understanding (Zhang et al., 2024c; Liu et al. 2024c). For video processing, current LVLMs typically sample video into sequence frames and then convert each frame into hundreds or thousands of visual tokens for the LLM (Bai et al. 2025 Li et al. 2024). While this paradigm is effective, the number of tokens grows explosively with increasing video duration and resolution. This leads to prohibitive computational and storage burdens, severely limiting the application of LVLMs scenarios such as long-video analysis and real-time interaction (Fu et al. 20245).
To address this challenge, various methods are proposed to enhance the efficiency of LVLMs. These approaches can be categorized into two types. One category is importance- based pruning. As ob- served by VastV (Chen al. 2024). there redundancy in the attention distribution across different layers when LVL.Ms process visual information. Therefore, visual tokens with lower attention scores in the deeper layers of the model can be pruned reduce the computational load. The other category is similarity- based merging/selection. Existing works have found high similarity among visual to- kens, either between adjacent video frames or within single frame. FrameFusion (Fu al. (024b) reduces redundancy by merging similar tokens from adjacent frames, while VisionZip (Yang et al. 2025) directly selects dominant tokens at the vision encoder level.
"Corresponding author.

---

Published as a conference paper at ICLR 2026
Spatiotemporal Similarity
Temporal Difference
Timeline
Figure 1: The core motivation of ST-SimDiff. We posit that efficient video understanding requires handling two scenarios simultaneously: Spatiotemporal Similarity (left) can be utilized to compress redundant information both spatially within a frame and temporally across adjacent frames. Tempo- ral Difference (right) should be detected to capture the key actions or events that define the plot.
Despite the success of existing methods in visual token reduction, they face key limitations. First, existing token pruming methods often focus spatial correlations within the same frame or temporal correlations at the same positions across different frames. This lack of joint modeling and analysis of spatio-temporal correlations prevents them from effectively capturing complex dynamic events. Furthermore, these methods share a potential blind spot: they mainly focus on informational commonalities, like similarity and importance, while neglecting crucial changes and differences. Since the narrative of the video is often driven by turning events, a model that only seeks similarity might smooth over sudden action, leading to misinterpretation of the content.
Based on this, we propose a new perspective: similarity is for identifying redundancy, while dif- ference is for capturing key moments. We believe that an ideal video token compression algorithm should achieve two goals simultancously: representing the stable content video with the fewest tokens, and precisely preserving the key changes. To this end, we propose ST-SimDift, dual to- ken selection framework based on a spatio-temporal graph. Specifically, the framework consists of representative token selection: we all visual tokens
two parts. The first is similarity- of a video as nodes and construct graph based on their feature similarity in the spatio-temporal dimension. Using a graph community detection algorithm, we find tightly connected token clus- ters in the graph, which represent stable and persistent visual elements in the video. We select few central tokens with higher centrality from each cluster as representatives. The second part difference- event token selection: we pay special attention to the edges connecting along the time axis in the graph. When the similarity between corresponding tokens of adjacent frames drops sharply, consider it turning point and mark these tokens as critical event tokens that must be retained. Finally, we merge these two sets of tokens to be input into the LLM. Our contributions can be summarized as follows:
We are the first to propose that the similarity and difference of importance VLM efficiency research.
tokens should be given equal
We design spatio-temporal graph framework to uniformly model the complex spatio- temporal relationships between video tokens. We propose a novel dual token selection strategy that can simultaneously screen for both representative and pivotal event tokens.
We conduct extensive experiments video understanding benchmarks, and the results show that ST-SimDiff significantly reduces the visual token budget while maintaining or even improving performance.
2

---

Published as a conference paper at ICLR 2026
2 RELATED WORK
2.1 LARGE VISION LANGUAGE MODELS
The rapid advancement of Large Language Models (LLMs) (Touvron et al.) 2023 Achiam et al. 2023 Ouyang et al., 22022 has significantly propelled progress in multimodal understanding, lead- ing to the emergence of numerous prominent Large Vision Language Models (LVLMs) (Liu et al.
2024b Bai al. 2023). These LVLMs typically process images video frames by converting them into visual tokens via pre- pre-trained visual encoders, which are then fed into LLMs. Various alignment modules commonly connect the visual encoder and the LLM, including MLPs (Liu et al. 2024a bb
and Q-Formers (Li et al. 2023 Dai et al. 2023). This architecture enables LVLMs i to exhibit excep- tional capabilities in mul multimodal tasks (Xi al.) 2025) Holdaway et al., 2024 Zhang et 2025a).
Particularly in video understanding, they show promise in processing longer. higher-resolution com-
plex videos (Bai al., 2025 Li et al. 2024) Zhang et al. 2024c Liu et al. 2024c).
However, this powerful capability also introduces significant computational challenges, especially in video understanding scenarios. Due to the need to process continuous sequences of frames, the number of visual tokens grows exponentially, far exceeding that of static images, posing severe challenge to the training and inference efficiency of LVLMs. Therefore, how to efficiently process massive video visual tokens and reduce computational overhead remains critical issue in current LVLM research.
2.2 VISUAL TOKEN COMPRESSION
Driven by the substantial overhead of video processing, visual token compression has become a critical method for improving the efficiency of Large Visual Language Models (LVLMs). Existing methods primarily approach compression from two aspects: token importance and similarity.
Earlier methods primarily focused on importance-based token pruning, aiming to identify and re- move visual tokens that contribute less to model performance. These methods typically utilize at- tention scores or other feature metries to quantify token importance, such as FastV (Chen et al. 2024) and FasterVLM (Zhang et al. [0024)). Subsequent works (Shang et al. 2024 Ye et al.) 2025) have started to further consider token similarity to more effectively reduce redundancy and preserve critical information. These methods recognize that even important tokens can exhibit high similar- ity, leading to informational redundancy. For instance, FrameFusion (Fu et al.] [2024b] combines similarity-bant -base merging with importance-based pruming. VisionZip (Yang al. 2025) reduces redundancy by selecting dominant tokens and merging contextual information. VisPruner (Zhang al. 2025b) leverages visual cues remove redundant items.
Despite significant progress in visual token redundancy, existing work hardly addresses the intricate, global spatio-temporal similarity relationships between tokens. In video scenarios, visual informa- tion is redundant both spatially and temporally. Existing similarity-driven methods, like FrameFu- sion, mainly only on the temporal similarity between adjacent frames, failing to utilize complex spatio-temporal relevant information. This limits their ability to fully exploit inherent video redun- dancy, restricting performance extreme token reduction scenarios.
3 PROBLEM DEFINITION
Given a video V and a text query Q, a standard Large Vision-Languaaee Model (LVLMM) first en- codes the video into full sequence of N visual tokens, Tim = (t1, t2......... (t).. This process is computationally demanding, as the self-attention mechanism's 's complexity scales quadratically with the token count N, i.e., O(N2). Our task is therefore to design an efficient token selection function, f(-). that takes the full sequence Tiul and a compression ratio r to produce a much smaller subset Tub = (Tiun r). The objective is maximize the LVLM's downstream task performance using this subset, subject to the constraint that its size Tasb is approximately r N. The core challenge is to design the selection function f(-) to retain the most critical information within the subset T Tub-
3

---

Published as a conference paper at ICLR 2026
Input Video
Similarity-ban -based Representative Token Selection
PentioTeempoal Graph
Selest
Tepp
Visual Lncodle
Community
Reprosstative
Language Model
Og
Response
Difference- based Event Token Selection
Selectt
Add
Specisal Fdlgs F5 Tempural Fddess By
Tenpural Stumilarit wtf <Tayyy
Figure 2: The overview framework of ST-SimDiff, which consists of three parts: Spatio- Temporal Graph Construction, Similarity-based Representative Token Selection, and Difference-based Event Token Selection.
4 ST-SIMDIFF
4.1 МЕTHOО OVERVIEW
As illustrated in Figure 2 the proposed ST-SimDiff method comprises two key components that operate in parallel on a apatio-temporal graph constructed from the video's visual tokens: Similarity- based Representative Token Selection (SRTS) and Difference-basee Event Token Selection (DETS). First, in the SKTS module, we identify and condense the video's stable and redundant content. By applying community detection algorithms to the graph, we group highly similar tokens into clusters, which correspond to persistent visual elements like static backgrounds. From each cluster, we then select a few highly central tokens to form a representative set, Tisp- Second, the DETS module is designed to capture the video's crucial dynamic shifts. It analyzes the temporal edges of the graph, identifying moments where the similarity between corresponding tokens in adjacent frames drops sharply. These points of high difference signify turning points, and the tokens framing these transitions are preserved as critical event tokens, forming the set Trans By synergistically combining these two components, ST-SimDiff generates a final token subset Suub Teop Juaaa which is both highly compact and information-rien rich, retaining stable content while precisely capturing key events.
4.2 SPATIO-TEMPORAL GRAPH CONSTRUCTION
Given a video, the vision encoder transforms it into a sequence of tokens T f2.......... }- Each token to E T' corresponds to a specific spatio-temporal location and is associated with feature vector Xg. We define the location of each token 4 by its frame index 7(4)), its spatial height index H(2)), and its spatial width index W W(+).
We model the relationships between these tokens using a spatio-temporal graph G (V, E), where the vertex set V represents all tokens in T. The edge set E is the union of two distinct subsets, E = Es ET, which capture spatial and temporal relationships respectively. The spatial edges Es connects tokens that spatially adjacent within the same frame:
Es [ U;) E V x V T(t) T(t;) and
(t.) (t,))) + (W(+,) - W(t))
The temporal edges Fr connects tokens that are at the same spatial position but in adjacent frames, capturing temporal continuity:
Ey =((,, 12) EV x V H(t.) H(t,). W(t,) W(t),, and [Tt,)) T(t)))=
The weight of any edge w(v,, v3) E is defined by the cosine similarity the feature vectors of the corresponding tokens:
w(r, U3) = x; Xj
(3)
X Xx,
4

---

Published as a conference paper at ICLR 2026
This sparse graph structure efficiently encodes local spatial relationships and temporal continuity, providing unified foundation for our dual-path token selection strategy.
4.3 SIMILARITY-BASE BASSED REPRESENTATIVE TOKEN SELECTION
Video content inherently characterized by substantial spatio-temporal redundancy. For instance, in static scene that persists for several seconds, the background tokens exhibit high similarity over time. Similarly, for an object moving across the screen, its corresponding tokens, despite changing spatial positions, maintain high degree of semantic correlation. Existing methods often address temporal and spatial similarities isolation. contrast, our spatio-temporal graph designed to capture these joint similarities. Within this graph, semantically related tokens form dense "commu- nities" or "clusters" based on their feature similarity, irrespective of their specific frame or location. Therefore, selecting representatives from each community emerges as highly efficient strategy for compressing redundant information while preserving core semantics.
To accurately identify strongly correlated token clusters, first threshold the graph G. We set similarity threshold Faim and retain only the edges with weights above this threshold, forming a new graph G' (V, F'), where F' {(1, vy) E E w(0; 0;) > Tsim )- Next, community detection algorithm (e.g.. the Louvain method (Blondel et al. 2008)) on G' to identify token clusters C {c1, (2, ................................ C. For each community Ck E C, we rank and filter its internal tokens based on their centrality. The centrality score Se(ta) of a token &a E Ck is defined as its average similarity with all other tokens within the community:
apply graph
Sa(ta) =
Σ (s dokoa
w(t.: tb)
(4)
Ckl
Subsequently, we set the intra-community retention rate to the globally preset compression ratio r. and preserve the top [[cu r] tokens with the highest centrality scores from each community Cg. This uniform filtering process naturally handles all communities, including single-node com- munities composed of unique tokens (in which case the sole node retained). Formally, the set of representative tokens, Tep, is defined as:
Trep U
TopK (feckl- rr]
(5)
A==1 LECL Score
where the TopK funetion returns the set of elements with the highest scores from a given set. In summary, this strategy efficiently compresses the static and persistent content of the video by identifying and retaining the most central tokens within each semantic cluster.
4.4 DIFFERENCE-BASED EVENT TOKEN SELECTION
If similarity defines the "norm" of video, then difference defines its "events". A model focusing only on similarity may excel at understanding "what is" but struggle with "what happened". A video's plot is driven by key events, and the essence of event is change-the appearance of new object, the start of an action, or scene transition. In our model, these changes manifest as sharp difference in the features of temporally adjacent tokens. Therefore, accurately capturing these "turning points" of significant difference is crucial for understanding the video's 's dynamic content and correctly answering questions like "when" and "why".
Difference, particularly along the temporal dimension, often signals the occurrence of key event. For example, the entry of new object, a scene change, or the beginning/end of action will cause drastic changes in the visual features at corresponding positions in adjacent frames. We specifically analyze the temporal edges (Er) in our spatio-temporal graph. For any temporal edge (Uk, vx) E ET in the graph, which connects two temporally adjacent tokens t& and we dynamic threshold T (c.g., the 95th percentile of all temporal edge difference scores). When the difference score of temporal edge (Vk, m)) exceeds this threshold, i.e., w(t,, 4)) Tair, consider the subsequent token & as a critical event token and retain it. Formally, the set of event tokens, T Tevm, is defined as:
Tevent =(4 36 s.t. (UR, vr) E ET, T(t) > T(tx), and w(Uk, vi) < <aift)
(6)
5

---

Published as a conference paper at ICLR 2026
With this strategy, tokens that signify moments of significant temporal change be preserved. By capturing these key transitions, we ensure that the crucial dynamic aspects of the video's narrative not overlooked during the compression process.
4.5 OVERALL REDUCTION PROCESS
In the proposed framework, we first compute the representative token set Tepp and the event token set Tevent in parallel, and take their union, Tamdiidate = Trep U as the initial candidate set.To precisely meet the target token count Nurget [r .N7, we introduce a final pruning step. the size of the candidate set, [T canddiddee exceeds Narget, we remove the caanddidmel Noarett tokens with the lowest importance. Following existing work (Chen et al., 2024), the importance of token is determined by its attention score in the shallow layers of the LLM. This step ensures that our method can flexibly meet any computational budget while preserving critical information.
In summary, overall method adeptly combines two strategies: graph-based token selection and attention-based dynamic pruning. The former leverages the intrinsic structure of the video content (similarity and difference) ensure the retention of core information, while the latter provides a flexible mechanism to precisely meet any given computational budget. This design makes our token compression process both principled and adaptive, thereby achieving robust balance between efficiency and performance.
4.6 COMPUTATIONAL COMPLEXITY ANALYSIS
The computational overhead of our proposed ST-SimDiff framework can be analyzed in three main stages: Spatio-Temporaa Graph Construction, Similarity-based Representative Token Selec- tion (SRTS), and Difference-basee Event Token Selection (DETS).
The initial stage involves building the graph. For each of the N visual tokens, we compute the cosine similarity with its spatially and temporally adjacent neighbors. As each token has small, constant number of neighbors, this step requires a single pass over all tokens, resulting in complexity of O(Nd), where d is the feature dimension of each token.
Following graph construction, the SRTS module identifies token clusters. As detailed in our im- plementation (Section 5.2), employ a connected components algorithm, which operates with near-linear time complexity of O(N + EE). Since the number of edges [E] is at most 3.N, this simplifies O(N). The subsequent filtering step within each community could be computation- ally intensive. A naive approach of calculating all-pairs similarity within each community c& would have a complexity of O(CC)- |cul2-. d). To prevent this, we impose a constraint on the maximum size of any community. In the rare event that a community exceeds a predefined threshold (c.g., lc&l > VN), we partition it. This ensures the filtering process remains efficient and does not dominate the overall complexity. While other mainstream graph clustering algorithms such as the Louvain and Leiden methods offer more complex community definitions, their computational com- plexity is higher, approaching O(N log N). Therefore, we opted for the simpler and highly efficient connected components algorithm to ensure minimal processing overhead.
The DETS module involves a single pass through temporal edges in the graph to identify sig- nificant changes by comparing token similarities against threshold. This process has a linear complexity of O(Nd).
In summary, the total complexity of our ST-SimDiff framework is governed by these linear-time operations, culminating in an overall complexity of O(Nd). This is substantially more efficient than the quadratic complexity of the self-attention mechanism, O(N2 used in the Large Lan- guage Model. Therefore, our method provides significant token reduction with only a negligible computational footprint relative the model's main inference cost.

---

Published as a conference paper at ICLR 2026
VideoMMM
VongeBeeen
Perception
Method
Medium
Overall
Relation
Overall
EgoSchema
Short
Long
Upper Bound (Full Performance)
LLaVA-Video
63.3
58.2
57.3
Token Retain Ratio r 30%
FastV
68.2
58.6
51.2
59.3
48.0
59.9
53.5
51.3
PruMerge
70.2
57.3
52.2
59.9
49.6
60.5
54.7
50.9
FasterVIM
71.3
57.6
51.4
60.1
51.0
61.3
55.8
52.6
VisionZip
68.4 4
55.9
50.4
58.3
46.9
60.3
53.2
53.0
FrameFusion
74.0
59.8
50.0
61.3
49.7
63.3
56.0
53.0
ST-SimDiff (Ours)
74.7
61.9
53.0
63.2
52.5
63.2
57.5
56.0
Token Retain Ratio r
50%
Fast
73.9
61.4
51.3
62.2
50.1
62.2
55.7
54.7
PruMerge
72.3
58.7
52.8
61.3
51.4
63.2
56.9
54.6
FasterVLM
74.0
59.2
51.8
61.7
51.3
62.2
56.4
56.2
VisionZip
72.6
57.7
51.6
60.6
51.3
63.2
56.8
54.2
FrameFusion
74.6
61.2
52.0
62.6
51.7
64.2
57.6
55,8
ST-SimDiff' (Ours)
76.2
62.4
52.7
63.8
52.3
64.3
57.9
57.3
Table I: Performance comparison on long-form video understanding benchmarks on ALaVA-Video- 7B for 64 input frames (%). The best performance among all methods emphasized bold.
EXPERIMENTS
EXPERIMENTAL SETUP
Baselines To comprehensively evaluate method's effectiveness, use LLAVA-Vides (Zhang et al. 2024c) and NVILA (Liu et al. 2024c) as our base models and compare our approach against range of state-of-the-ard video token compression techniques. These baselines cover diverse strate- gies, including importance-basel reduction methods like FastV (Chen et al. 2024) and Faster- VLM (Zhang al.] (022bb). and hybrid reduction methods like VisionZip Yang FrameFusion (Fu et al., (024b). and PruMerge (Shang et al., (024). To ensure a fair and complete comparison, we also include the original uncompressed model (Vanilla) as performance upper bound and Random Sampling of an equivalent number of tokens lower bound.
(025),
Benchmarks To test our method's capabilities in long-vidn video understanding, we adopt three chal- lenging benchmark datasets: VideoMME (Fu et al. (2024a). LongVideoBench (Wu et al.) 2025) and EgoSchema (Mangalam et al.] 2024). VideoMME
comprehensive benchmark for foun- dational video understanding. featuring diverse video types and three length categories: Short, Medium, and Longg LongVide VideoBench evaluates the ability to retrieve and reason about details in long videos through a "referring reasoning" task, with videos four lengths: 15s, 60s, 600s, and 3600s. EgoSchema is a question-answering dataset focused on long-term, first-person videos that tests the model's ability to understand character intentions and causal chains of actions.
5.2 IMPLEMENTATION DETAILS
We use NVIDIA L20 GPUs with 48GB VRAM Ubuntu 22.04. The inference evaluation conducted based on the imms-eval (Zhang et al. 2024a) library. For LLaVA-Vidhe we follow its original setting of 64 input frames. To ensure fair comparison of computational efficiency and resource usage under similar conditions, we establish consistent setup by also setting the input frame count for NVILA A 64, As for the hyperparameters, set the similarity threshold 7sim 0.8.

---

Published as a conference paper at ICLR 2026
VideoMMM
VongeBeeen
Perception
Method
Medium
Overall
Relation
Overall
EgoSchema
Short
Long
Upper Bound (Full Performance)
WVILA-Vided
61.5
56.3
52.9
Token Retain Ratio r 30%
FastV
69.4
55.3
48.8
57.9
50.8
55.6
53.0
49.7
PruMerge
71.0
53.7
49.9
58.2
49.3
58.1
53.4
47.5
FasterVIM
72.6
56.9
51.0
60.1
48.7
57.8
53.0
49.3
VisionZip
71.4
56.2
49.7
59.1
46.3
56.2
50.9
48.9
FrameFusion
72.1
55.7
48.7
58.8
51.3
59.3
54.9
51.3
ST-SimDiff (Ours)
73.3
57.7
49.7
60.2
51.7
59.2
55.2
51.7
Token Retain Ratio r
50%
Fast
71.9
58.3
49.3
58.9
49.5
58.8
53.9
50.2
PruMerge
71.2
54.8
46.8
57.6
49.7
58.7
53.9
48.9
FasterVLM
74.6
57.8
50.1
60.8
48.5
58.1
53.0
50.5
VisionZip
72.8
58.1
50.6
60.5
50.4
58.9
54.4
50.3
FrameFusion
72.7
57.2
48.3
59.4
51.0
59.2
54.8
52.6
ST-SimDiff' (Ours)
73.9
59.7
51.4
61.7
52.3
61.3
56.5
52.5
Table 2: Performance comparison on long-form video understanding benchmarks on NVILA-Video 8B for 64 input frames (%). The best performance among all methods emphasized bold.
and the difference threshold Tair = 0.2. Experiments are conducted with token compression rates of 30% and 50%, respectively. In practical implementation, considering the trade- off between time consumption and performance improvement, we opt to use connected component finding community detection algorithm.
5.3 COMPARISON WITH STATE-OF-THE-ANT
To comprehensively evaluate proposed ST-SimDiff framework, we compare it against range of mainstream efficient video language model (VLM) compression methods. All experiments conducted three widely-used long-form video understanding benchmarks: VideoMME, BengVideoBench, and EgoSchema. To validate the generalization ability and robustness of our method, we report detailed performance data NVILA-8B, under token retain ratios of 30% and 50%. The experimental results are presented in Table and Table 2
different base models,
and
(1) ST-SimDiff' consistently outperforms all competing methods across all test configurations. A particularly noteworthy finding is that 50% token retain ratio, the overall performance of our method on both base models not only surpasses other compression algorithms but, some bench- marks, even matches or exceeds that of the original model using 100% of the tokens. This result validates the effectiveness and novelty of ST-SimDiff and prompts deeper analysis of the strategies and limitations of existing baselines.
(2) In importance-based baseline methods (e.g., FastV, FasterVLM), FasterVLM typically exhibits relatively stronger performance. However, their common limitation lies in the inefficient handling of temporal redundancy prevalent in videos, they tend to retain important yet repetitive tokens, thereby limiting information compression efficiency. Hybrid baseline methods (e.g., PruMerge, VisionZip, FrameFusion) demonstrate proaches, with FrameFusion generally showing the most outstanding overall results. Nevertheless, the common optimization goal of these methods is still to identify and preserve representative "com-
competitive performance than importance-only ap-
8

---

Published as a conference paper at ICLR 2026
Method
VideoMME
LongVideoBench
EgoSchema
Token Retain Ratio r = 30%
Bascline
60.3
56.2
54.8
Spatial
61.5
56.5
55.2
+ Sim
Temporal
61.7
56.8
53,1
Spa. Tem.
62.6
57.0
55.3
++ Diff (Proposed)
63.2
57.5
56.0
Token Retain Ratio r
50%
Baseline
63.2
56.7
56.5
Spatial
63.3
57.0
56.7
Sim
Temporal
63.7
57.2
56.9
Spa. Tem.
63.7
57.8
57.2
++ Diff (Proposed) 63.8
57.9
57.3
Table 3: Ablation results of different components (%).
monality" information, with theoretical blind spot of potentially overlooking key narrative infor- mation driven by "turning points" and "abrupt events".
(3) In summary, existing baselines reflects deepening understanding of "redundancy" in videos, yet they still face two core challenges: first, a lack of synergistic analysis of spatio-temporal joint similarity between visual tokens, making it difficult capture complex redundancies: and second, general neglect of "difference" detection, failing to actively preserve key changes that define plot de- velopment. The performance gain of ST-SimDiff stems from addressing these issues by constructing spatio-temporal graph to uniformly model complex spatio-temporal relationships and innovatively introducing difference detection.
5.4 ABLATION STUDY
To validate the effectiveness of each core component within our ST-SimDiff framework, we con- ducted series of detailed ablation studies. We started with Baseline strategy that includes only fundamental importance-bast pruning and progressively introduced our proposed similarity selee- tion module (+Sim) and difference selection module (++Diff) to quantify their respective contri- butions. All experiments were conducted at token retain ratios 30% and 50% with the results presented in T
(1) First, we evaluated the effectiveness of the +Sim module. The experimental results clearly in- dicate that introducing similarity modeling top of the baseline consistently improves model per- formance. We further broke down the similarity strategy into three types: spatial-only (Spatial), temporal-only (Temporal), and spatio-temporal joint (Spa. Tem.). The data shows that collabora- tively modeling the spatio-temporal associations between visual tokens is the most effective way to compress redundancy.
(2) After integrating the optimal spatio-temporal joint similarity module, we introduced the dif- ference selection module (++Diff) form our complete franework. The addition of this module provides a significant performance lcap, particularly under the high compression ratio of 30%. This demonstrates its vital complementary role: while the aggressive +Sim module focuses on redun- dancy, the ++fiff module acts as safety net to identify and preserve unique event tokens that define the video's 's narrative. In contrast, the module's marginal gain is smaller a 50% retention ratio. This because the more lenient +Sim module is more likely have already captured these event tokens, and the overall model performance is already approaching its upper bound. This analysis confirms motivation: the framework's strength lies in its ability to balance similarity based redun- dancy compression with the crucial preservation of difference- based events, especially in demanding high-compression scenarios.

---

Published as a conference paper at ICLR 2026
Time Cost
Memory Cost
31.7%
29.356
35.0
27.21
50.3
Average Time (6)
11%
Memory (GB)
25.0
20,1 M
20.22
Reducting
2
2.28
1.56
10-
%2
64
12k
96
1.28
Frames
Frames
Bascline
ST-SimDift
Reduction Ratio
Figure 3: Computational cost comparison between method and the baseline ILLVA-Video model average inference time and peak GPU memory usage under a 30% token budget.
5.5 COMPUTATIONAL COST ANALYSIS
To quantify the efficiency advantages of our proposed ST-SimDiff method in practical applications, we conducted detailed tests on inference time and peak GPU memory usage under a 30% token budget. We compared it against the bascline model (Baseline LLaVA- -ideo) without any compres- sion. The experimental results in Figure cover video inputs lengths from 32 to 128 frames.
Inference Time ST-SimDiff' demonstrates significant speedup. As the number of input video frames increases, the average inference time of the baseline model grows sharply from 1.56 seconds (32 frames) 6.50 seconds (128 frames). In contrast, our method effectively curtails this growth, with an average time of only 4.54 seconds when processing 128 frames. This means that ST-SimDiff achieves an increasingly higher time-saving rate for longer videos, improving from 23.0% for 32 frames to 30.2% for 128 frames.
Memory Cost ST-SimDiff also performs exceptionally well. The peak GPU memory usage of the baseline model climbs linearly with video length, from 20.9 GB (32 frames) to 35.0 GB (128 frames). Our method significantly reduces memory pressure by substantially decreasing the number of tokens fed into the large language model. When processing 128-frame videos, ST-SimDiff's peak memory usage is only 23.9 GB, saving 31.7% of the memory space compared to the baseline. This advantage enables our model to process longer videos on hardware with limited memory, greatly expanding its potential applications.
In summary, ST-SimDiff not only achieves leading performance but also demonstrates significant advantages in computational efficiency (both time and memory), proving its efficiency for video understanding.
CONCLUSION
In this paper, we address the computational inefficiency of Multimodal Large Language Models when processing long videos, a problem rooted in the excessive number of visual tokens and the neglect of critical content changes. We propose ST-SimDiff, framework requiring no training that builds a spatiotemporal graph to model complex associations between tokens. The method utilizes a dual selection strategy where similarity based selection identifies redundancy via community de- tection and temporal difference based selection preserves tokens key transition points. Extensive experiments on VideoMME and Long VideoBench show that ST-SimDiff significantly reduces pro- cessing time and memory usage while consistently surpassing state of the art performance. This work introduces a new research direction by balancing content representativeness with transitional events create more efficient representation of video data.
10

---

Published as a conference paper at ICLR 2026
REFERENCES
Josh Achiam, Steven Adler, Sandhini Agarwal, Lama Ahmad, lige Akkaya, Florencia Leoni Ale- man, Diogo Almeida, Janko Altenschmidt, Sam Altman, Shyamal Anadkat, al. Gpt-4 technical report. arXiv preprint wwxiv:2333008774, 2023.
Jinze Bai, Shuai Bai, Shusheng Yang, Shijie Wang, Sinan Tan, Peng Wang, Junyang Lin, Chang Zhou, and Jingren Zhou. Qwen-vl: A versatile vision-language model for understanding, local- ization, text reading, and beyond. arXiv preprint arXiv:2308.12990 1(8), 2023.
Shuai Bai, Keqin Chen, Xuejing Liu, Jialin Wang, Wenbin Ge, Sibo Song, Kai Dang, Peng Wang, Shijie Wang, Jun Tang, et al. Qwen2. 5-vl technical report. arXiv preprint arXiv:2502.1392. 2025.
Vincent D Blondel, Jean-Loup Guillaume, Renaud Lambiotte, and Etienne Lefebvre. Fast unfolding of communities in large networks. Journal of statistical mechanics: theory and experiment, 2008 (10):P10008, 2008
Liang Chen, Haozhe Zhao, Tianyu Liu, Shuai Bai, Junyang Lin, Chang Zhou, and Baobao Chang- An image is worth 1/2 tokens after layer 2: Plug-and-pla inference acceleration for large vision- language models. In European Conference on Computer Vision, pp. 19-35. Springer, 2024.
Wenliang Dai, Junnan Li, Dongxu Li, Anthony Meng Huat Tiong, Junqi Zhao, Weisheng Wang, Boyang Li, Pascale Fung, and Steven Hoi. Instructblip: Towards general-purpose models with instruction tuning, 2023.
purpose vision-language
Chaoyou Fu, Yuhan Dai, Yongdong Luo, Lei Li, Shuhuai Ren, Renrui Zhang. Zihan Wang, Chenyu Zhou, Yunhang Shen, Mengdan Zhang, et al. Video-mme: The first-ever comprehensive evalua- tion benchmark of multi-modal Iims in video analysis. arXiv preprint arXiv:2405.21075 2024a.
Tianyu Fu, Tengxuan Liu, Qinghao Han, Guohao Dai, Shengen Yan, Huazhong Yang. Xuefei Ning, and Yu Wang. Framefusion: Combining similarity and importance for video token reduction on large visual language models. arXiv preprint arxiv:2501.01900 2024b.
Holdaway et al. Meta-designing quantum experiments with language models. Nature Machine Intelligence, 2024. doi: 10. 00.10088//4225--03--115333
Bo Li, Yuanhan Zhang. Dong Guo, Renrui Zhang, Feng Li, Hao Zhang, Kaichen Zhang, Peiyuan Zhang, Yanwei Li, Ziwei Liu, et al. Llava-onevision: Easy visual task transfer. arXiv preprint arXiv:2408. 03326, 2024.
Junnan Li, Dongxu Li. Silvio Savarese, and Steven Hoi. Blip-2: Bootstrapping language-image pre-training with frozen image encoders and large language models. In International conference machine learning, pp. 19730-19743 PMLR, 2023.
Haotian Liu, Chunyuan Li, Yuheng Li, and Yong Jae Lee. Improved baselines with visual instruction tuning. Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recogni- tion, pp. 26266-26306, 2024a.
Haotian Liu, Chumyuan Li, Yuheng Li, Bo Li, Yuanhan Zhang, Sheng Shen, and Yong Jae Lee. Llava-next: Improved reasoning, oer, and world knowledge, January 2024b. URL https://
Flava-vl. // 221p:///222222102200
llava next/
Zhijian Liu, Ligeng Zhu, Baifeng Shi, Zhuoyang Zhang, Yuming Lou, Shang Yang, Haocheng Xi, Shiyi Cao, Yuxian Gu, Dacheng Li, al. Nvila: Efficient frontier visual language models. arXiv
preprint 24.0:22412004466 2024c.
Karttikeya Mangalam, Raiymbek Akshulakov, and Jitendra Malik. Egoschema: A diagnostic bench- mark for very lorg-form video language understanding. Advances in Neural Information Process- ing Systems, 36, 2024.
Long Ouyang. Jeffrey Wu, Xu Jiang. Diogo Almeida, Carroll Wainwright, Pamela Mishkin, Chong Zhang, Sandhini Agarwal, Katarina Slama, Alex Ray, et al. Training language models to fol- low instructions with human feedback. Advances in neural information processing systems, 35: 27730 2022.
11

---

Published as a conference paper at ICLR 2026
Yuzhang Shang. Mu Cai, Bingxin Xu, Yong Jae Lee, and Yan Yan. Llava-pru Adaptive token reduction for efficient large multimodal models. arXiv preprint arXiv:2403.15388 2024.
Hugo Touvron, Thibaut Lavril, Gautier Izacard, Xavier Martinet, Marie-Anne Lachaux, Timothée Lacroix, Baptiste Rozière, Naman Goyal, Eric Hambro, Faisal Azhar, al. Llama: Open and efficient foundation language models. arXiv preprint arXiv:2302. 13971, 2023.
Haoning Wu, Dongxu Li, Bei Chen, and Junnan Li. Longvideobench: A benchmark for long-contex, interleaved video-language understanding. Advances Neural Information Processing Systems, 01128828-288577 2025.
Haoning Xi, Zhiqi Shao, David A Hensher, John D Nelson, Huaming Chen, and Kasun Wijayaratna. A multi-task transformer with mixture-of for personalized periodic predictions of individ- ual travel behavior in multimodal public transport. Transportation Research Part C: Emerging Technologies, 179: 105217,,77 2025.
Senqiao Yang, Yukang Chen, Zhuotao Tian, Chengyao Wang, Jingyao Li, Bei Yu, and Jiaya Jia. Visionzip: Longer better but not necessary in vision language models. In Proceedings of the Computer Vision and Pattern Recognition Conference, pp. 19792-198099 2025.
Weihao Ye, Qiong Wu, Wenhao Lin, and Yiyi Zhou. Fit and prune: Fast and training-frre visual token pruming for multi-modal large language models. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 39, pp. 22128 111222221666 22136, 2025.
Zhang al. Solving the many-electron schrödinger equation with transformer-based framework.
Nature Communications, 2025a. doi:
Kaichen Zhang, Bo Li, Peiyuan Zhang, Fanyi Pu, Joshua Adrian Cahyono, Kairui Hu, Shuai Liu, Yuanhan Zhang, Jingkang Yang. Chunyuan Li, and Ziwei Liu. Lmms-eval: Reality check on the evaluation of large multimodal models, 2024a. WtR.itttss///axiv..co
12772
Qizhe Zhang, Aosong Cheng, Ming Lu, Zhiyong Zhuo Minqi Wang, Jiajun Cao, Shaobo Guo, Qi She, and Shanghang Zhang. [cls] attention is all you need for training-free visual token prun- ing: Make vlm inference faster. arXiv e-prints, pp. arXiv-2412, 2024b.
Qizhe Zhang, Aosong Cheng, Ming Lu, Renrui Zhang, Zhiyong Zhuo, Jiajun Cao, Shaobo Guo, Qi She, and Shanghang Zhang. Beyond text-visual attention: Exploiting visual cues for effective token pruning in vlms. arXiv preprint arXi::2412.01811 2025b.
Yuanhan Zhang, Jinming Wu, Wei Li, Bo Li, Zejun Ma, Ziwei Liu, and Chunyuan Li. Video instruction tuning with synthetic data, arXiv preprint arXiv22410.02713, 2024c.
12

---

Published as a conference paper at ICLR 2026
A VISUALIZATION OF THE TOKEN SELECTION PROCESS
Original Video
Cluster in SRTS
Cluster 2
SRTS
Final Results
Original Video
Cluster in SRTS
Cluster in SRTS
Final Results
Original Video
Cluster 1
SRTS
Cluster in SRTS
Final Results
Figure 4: Visualization of the ST-SimDiff process. The visualization breaks down the token selec- tion into communities detected by SRTS (shown grids in Cluster and 2) and event-driven tokens detected by DETS. The final result highlights the synergy between sparse representative tokens (yel- low) for stable content and dense event tokens (red) for dynamic actions.
13

---

Published as a conference paper at ICLR 2026
To provide more intuitive understanding of the ST-SimDiff framework, present some visualiza- tion samples of the intermediate and final token selection results in Figure The figure illustrates the process sample video sequence featuring dynamic object manipulation task. First, regard- ing the Similarity-based Representative Token Selection (SRTS), the rows labeled "Cluster 1* and "Cluster 2" demonstrate how our graph community detection algorithm functions. It successfully groups spatially and temporally redundant regions, such as the static background and stationary ob- jects, into cohesive communities. This visualization confirms that SRTS effectively identifies stable visual elements and compresses them by selecting only few central "representative tokens," which appear sparse yellow boxes in the final results. This mechanism allows the model handle massive background redundancy efficiently.
Complementing this, the Difference- based Event Token Selection (DETS) plays critical role during dynamic moments. As seen in the "Final Results" row, tokens triggered by high temporal difference are explicitly marked with dense red bounding boxes. These red boxes align perfectly with the mov- ing hand and the shifting blue liquid, verifying that our method precisely locates "turning points" where the temporal difference exceeds the threshold (Taiff). This ensures that fine-grained action details and rapid visual changes preserved, allowing the model to capture the essence of the event without being overwhelmed by static data.
Finally, the synergistic effect of these two components is evident in the overlay of the final results. The visualization demonstrates a powerful balance: the sparse yellow tokens represent the stable background, while the dense red tokens capture the rapid motion. This contrast provides direct visual evidence of our core motivation. ST-SimDiff avoids the blind spots of similarity-oni, approaches by densely sampling during rapid changes, while simultaneously pruning the spatiotemporal redundan- cies that importance-basse methods often miss. Consequently, our framework achieves an optimal balance between computational efficiency and the preservation critical visual information.
B COMPARISON RESULTS ON QWEN2.5-VL VL
VideoMME Medium
Long LongVideoBench Perception
Method
EgoSchema
Short
Long
Overall
Relation
Overall
Upper Bound (Full Performance)
Qwen2.5-V1.co V1
74.2
62.7
51.8
62.9
54.2
64.8
59.2 60.4
63.0)
Qwen2.5-V1--228
77.4
65.8
56.1
66.4
56.2
65.1
63.9
Token Retain Ratio
30%
ST-SimDiff--4
73.8
61.7
51.8
62.4
52.2
62.1
56.8
62,7
ST-SimDiff-128
75.4
64.7
55.2
65.1
53.1
65.0
58.6
64.4
Token Retain Ratio
50%
ST-SimDiff-64
74.2
62.1
52.2
62.9
54.2
63.2
58.4
62.7
Si-DiDDif--128
76.8
66.3
55.8
66.3
54.9
65.3
59.8
64.5
Table 4: Performance comparison on long-form video understanding benchmarks on Qwen2.5-VL- 7B for 64 and 128 input frames (%).
To further validate the generalization capability of our proposed framework, we conducted additional experiments on the recently released Qwen2.5-V VL model. These evaluations were performed on the VideoMME and Long VidcoBench benchmarks confirm that our method's efficacy is not limited to specific model architecture but can be broadly applied.
The empirical results, presented in Table demonstrate the strong performance of ST-SimDiff on the Qwen2.5-VL. model. At token retention ratio of 30%, our method achieved scores of 62.4 on VideoMME and 56.8 on Long VideeBencn When the token budget was increased to 50% retention ratio, the performance improved to 62.9 on VideoMME and 58.4 4 on Long VideoBench. is particularly noteworthy that at the 50% ratio, the performance on VideoMME matches the upper-bound performance achieved using all tokens, and the comparable to its full- pelforerformanc counterpart. These findings affirm the excellent generalization ability of our framework.
Long BidcoBench is highly
14

---

Published as a conference paper at ICLR 2026
C
ABLATION STUDY RESULTS ON NVILA
Method
VideoMME
LongVideoBench
EgoSchema
Token Retain Ratio r = 30%
Bascline
59.3
53.8
50.2
Spatial
59.5
54.5
50.9
+ Sim
Temporal
59.6
54.3
51.2
Spa. Tem.
59.9
54.6
51.3
++ Diff (Proposed)
60.2
55.2
51.7
Token Retain Ratio r
50%
Baseline
60.9
55.3
51.5
Spatial
61.1
55.7
51.9
Sim
Temporal
61.3
55.6
51.8
Spa. Tem.
61.4
56.0
52.3
++ Diff' (Proposed)
61.7
56.5
52.5
Table 5: Ablation results of different components on NVILA-Video-81 model (%).
To further validate the generalization ability of each component in our framework across different base models, we also conducted a series of detailed ablation studies on the EVILA-Video- model (Liu et al.) 2024c). The experimental setup is consistent with the ablation study in the main paper; we start with a Bascline strategy that includes only fundamental importance-basse pruning and progressively introduce the similarity selection module (+ Sim) and the difference selection module (++ Diff).
As shown in Table the experimental results once again validate the effectiveness of our framework design, and the trends are highly consistent with the performance on LLaVA Video-7B. First, after introducing the similarity module (+ Sim) on top of the baseline, the model's performance shows a steady improvement at both 30% and 50% token retain ratios. Within the similarity module, we compared three strategies: spatial-only (Spatial), temporal-only (Temporal), and spatio-temporal joint (Spa. + Tem.). The results show that spatio-temporal joint modeling (Spa. + Tem.) is the most effective. For example, with r= 50%, the spatio-temporal joint strategy achieved a score of 61.4% on VideoMME, outperforming the spatial-only and temporal-onie strategies. This again demonstrates the importance of collaboratively considering ppatio-temporal associations for effectively identify- ing redundant information. After integrating the optimal ppatio-tempora joint similarity module, we further introduced the dissimilarity selection module (++ Diff) to form our complete ST-SimDiff framework. The table clearly shows that the addition of this module brought the most significant performance gains. This result strongly demonstrates the effectiveness and generalization ability of the two core modules in framework (similarity and dissimilarity selection) and once again validates our core idea of simultaneously handling redundancy and change.
D ABLATION RESULTS OF Tsim AND Taiss ON LLAVA-VIDER
In our framework, the similarity threshold Taam and the difference threshold Taisf are two key hy- perparameters that respectively influence the selection of representative and transitional tokens. To investigate their impact, we conducted series of ablation studies, with the results shown in Figure 6 The experiments show that the impact of both parameters on model performance follows a similar trend, first rising and then falling, while demonstrating good robustness within certain range. For Teims value that is too low leads to imprecise community detection, while value that is too high can disrupt the integrity semantic clusters. For Taiff value that too low can introduce noise due to over-sensitivity, whereas value that is too high may cause the model miss key events. According to the experimental results, the model achieves optimal performance at Txim = 0.8 and Taifs = 0.2. Therefore, adopted this optimal configuration for all other experiments.
15

---

Published as a conference paper at ICLR 2026
66.0
66.0
VideoMME I Long LongVideoBenr VideoBench
VideoMME Long VideoBench
64.0
63.2%
64.0
63.2%
Performance (%)
Performance (%)
62.0
62.0
60.0
60.0
58.0
57.5%
58.0
57.5%
56.0
56.0
0.6
0.8
0.85
0.95
0.1
0.2
0.3
0.4
Taim
Tdiff
(3)
Tasm
(b) Taiff
Figure 5: Ablation study results for different values of Taifs and Tsim VideoMME and LongVin VideoBencc.
E ABLATION RESULTS OF Tsim AND Taffs ON NVILA-VIDRO
64.0
64.0
VidcoMME Long VideoBench
VideoMME
62.0
62.0
Long Video
60.2%
60.2%
Performance (%)
60.0
Performance (%
60.0
58.0
58.0
56.0
55.2%
56.0
55,2%
54.0
54.0
52.0
52.0
0.6
0.7
0.8
0.85
0.9
0.95
0.1
0.2
0.3
0.4
Tasm
(b) Taiff
Figure 6: Ablation study results for different values of Taiff and Taim on NVIIA-Video-B8
To determine the robustness of the key hyperparameters in our method, we conducted series of ablation studies on the NVILA- ViBeo-8B model. As shown in Fig. O we evaluated the impact of the similarity threshold, Tsims and the difference threshold, Tairs on the model's 's performance across two benchmarks: VideoMME and Long VideoBench.
Impact of Similarity Threshold Tsim We tested different values for Tsm within the range of [0.6, 0.95). The experimental results show that the model's performance is relatively insensitive to changes in this threshold. On the VideoMME dataset, peak performance of 60.2% was achieved at Tsim 0.8. Similarly, the Long VideoBench dataset, performance was also optimal 7sim 0.8, reaching 55.2%. While performance slightly decreased when the threshold was too high or too low, the overall fluctuation was minimal, confirming that our method maintains good stability across wide range of similarity thresholds.
Impact of Difference Threshold Tairt We tested Tair in the range of [0.1, 0.4). Similar to Tsm- the model's performance demonstrated strong robustness to variations in Tain- In both bench- marks, the optimal performance was achieved around Tair = 0.2, with VideoMME at 60.2% and Bidee at 55.2%. This indicates that setting the difference threshold around 0.2 is most
16

---

Published as a conference paper at ICLR 2026
effective for capturing key events in the video, thereby maximizing model performance while com- pressing tokens.
In summary, these ablation studies confirm the robustness of our proposed method with respect to its key hyperparameters. The results validate our choice using Tsim 0.8 and Tan = 0.2 in our main experiments, these values consistently yield optimal performance across different benchmarks and different base models.
17


Reviews:
[{"review_id": "K5zlDLx05w", "reviewer": "Reviewer_1bh7", "summary": "The paper presents a novel and insightful framework for video token compression, effectively balancing similarity-based redundancy reduction with difference-based key event capture. This training-free, dual-path approach is technically sound and demonstrates strong, generalizable performance across multiple models and significant computational savings.", "questions": "1.Which exact shallow layers are used in the final attention-based pruning step? What is the specific aggregation function for attention scores across heads and layers?\\\n2.Provide qualitative visualizations that highlights which tokens are preserved by the proposed method to help build intuition and identify failure modes.\\\n3.Why do you choose the connected components algorithm compared with other mainstream community detection methods, especially in terms of both understanding performance and processing efficiency?\\\n4.Could the framework's performance be further enhanced by making parts of it learnable? What are the potential benefits versus the cost?\\\n5.The SRTS and DETS paths are currently combined, seemingly uniformly. Is it possible to dynamically weight these two paths for better performance?", "limitations": "", "rating": 8, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "1.The paper makes a significant contribution by addressing the need for efficient video understanding through the dual focus of redundancy compression (via similarity) and key event capture (via difference). This approach moves beyond prior work that primarily focused on similarity or importance-based pruning alone.\\\n2.The proposed ST-SimDiff framework is technically sound, employing a spatio-temporal graph to model token relationships uniformly. The dual-path SRTS and DETS effectively implement the core motivation, demonstrating significant and consistent performance improvements across a range of state-of-the-art, model-agnostic baselines, including LLaVA-Video, NVILA-Video, and Qwen2.5-VL.\\\n3.As a training-free framework, ST-SimDiff introduces minimal overhead and operates with substantially improved efficiency. Its computational complexity is analyzed as O(Nd), much more efficient than the $O(N^2d)$ self-attention it aims to reduce. Empirical results show significant reductions in inference time and peak GPU memory usage.", "weaknesses": "1.The final pruning step, where the initial candidate set exceeds the target budget, is underexplained in Section 4.5. It is unclear which layers are selected, how the scores are aggregated, and the impact of this step on performance.\\\n2.The paper lacks qualitative examples, such as visualizations showing which tokens are selected. This makes it challenging to build intuition about the method’s behavior and understand its potential failure modes.\\\n3.There is no comparative analysis or justification to explain why the connected components algorithm is chosen in SRTS. The impact of this choice is less investigated.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "phVmksTTrH", "reviewer": "Reviewer_f9gs", "summary": "This paper presents ST-SimDiff, a balanced framework that explicitly models the trade-off between spatial-temporal similarity and difference in video representations. The method aims to improve video understanding by aligning similar semantics while preserving meaningful diversity across frames. The authors introduce novel similarity–difference guided modules that can be integrated into existing video backbones with minimal modifications. Extensive experiments on standard benchmarks demonstrate consistent improvements over strong baselines, showing superior performance across multiple tasks with manageable computational overhead.", "questions": "Can the authors provide qualitative visualization results, especially highlighting where the similarity–difference mechanism fails or produces ambiguous interpretations?\nWhere are the performance boundaries of this framework? Are there specific video tasks/types that ST-SimDiff is ill-equipped to handle?\nHow does the framework solve the input misalignment problem after pruning?\nIf the model makes an incorrect judgment on a video, how can you attribute the failure to the similarity module versus the difference module?", "limitations": "", "rating": 6, "confidence": 5, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "This well-executed, clearly written paper addresses the critical issue of balancing temporal consistency and diversity in video representation learning. The ST-SimDiff framework is elegant, conceptually sound, and easily integrates into existing architectures, delivering a significant performance boost without the need for additional supervision or costly retraining.\nThe paper effectively highlights how previous methods either overemphasize similarity or difference, while ST-SimDiff strikes a balanced, principled approach. The motivation is compelling, the technical formulation is clear, and the presentation flows smoothly.\nThe design of interpretable components effectively clarifies how spatial-temporal cues interact. The experimental validation is comprehensive, covering diverse datasets, multiple backbones, and various evaluation metrics, with ablations that isolate the contribution of each component. Results are strong and consistent.", "weaknesses": "The paper lacks crucial qualitative visualizations. It does not show failure cases or ambiguous scenarios, making it difficult to understand the mechanism's practical limitations or how it behaves when it produces an incorrect interpretation.\nThe framework's performance boundaries are not explored. The paper fails to investigate video types or specific tasks where the proposed balancing paradigm might be suboptimal or ill-equipped.\nThe paper does not address the critical misalignment problem that arises from sequence pruning. While pruning/dropping is a common efficiency method, it creates a gap between the dense data the model was trained on and the sparse data it receives at inference, for which the model has no explicit handling mechanism.\nThe paper lacks a method for error attribution, making it difficult to determine whether a specific failure originates from the similarity module, the difference module, or their interaction.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "3WsMIJacnE", "reviewer": "Reviewer_TViS", "summary": "This paper addresses the computational overhead MLLMs face when processing long videos. The authors propose ST-SimDiff, a training-free token reduction framework. The core idea is that video compression must balance two aspects: using similarity to identify and compress redundant static content, and using difference to preserve key dynamic events or turning points. The method constructs a spatio-temporal graph to model token relationships. It then uses community detection for similarity and temporal difference thresholding for key events to generate a compact, information-rich token subset. Experiments show SOTA performance on multiple benchmarks.", "questions": "See weaknesses above.", "limitations": "", "rating": 6, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "1. The paper is well-written and clearly articulates a significant and practical problem. The core concept of striking a balance between spatiotemporal similarity and temporal difference constitutes a valuable contribution to this field.\n2. The proposed framework is training-free, making it highly practical, and easily applicable as a plug-and-play module. The use of a spatio-temporal graph to uniformly model complex token relationships is good way to implement the core idea.", "weaknesses": "1. Discrepancy in baseline performance. The paper's validation on Qwen2.5-VL (Appendix, Table 4) suffers from a baseline inconsistency. The paper reports the \"Upper Bound (Full Performance)\" for Qwen2.5-VL as 62.9 on VideoMME and 59.2 on LongVideoBench. However, the official Qwen2.5-VL technical report states a score of 65.1 on VideoMME and 56.0 on LongVideoBench. The authors should address or explain this discrepancy.\n2. Lack of Image-based Validation. The paper focuses exclusively on video datasets. However, the core mechanism, particularly the Similarity-based Representative Token Selection (SRTS, is designed to handle spatio-temporal redundancy. The spatial component of this logic should be directly applicable to compressing redundant tokens in static images. Experiments on standard image-based VLM benchmarks could showcase the robustness of the similarity-based compression module.\n3. Qualitative Visualization. The paper's core hypothesis relies on the intuitive concepts of \"similarity\" (static content) and \"difference\" (key events). While the quantitative results are strong, the paper provides no qualitative visualizations. It would be highly beneficial to include figures that show, for specific video examples, which tokens are selected by the SRTS module versus the DETS module. This would provide direct and intuitive evidence that the framework is truly capturing static backgrounds and dynamic turning points as intended.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "yAxP96Cg4N", "reviewer": "Reviewer_34qR", "summary": "This paper aims to address the problem of extremely high computational costs caused by the massive number of visual tokens when multimodal LLMs process long videos. It makes three main contributions: 1) It is the first to focus on inter-frame differences in videos, emphasizing that change is key to video understanding. 2) It proposes a method for constructing spatiotemporal graphs that uniformly models spatial similarity and temporal continuity. 3) It introduces a new token selection strategy: for token clusters with high similarity in the graph, only a few representative tokens are retained, while tokens with extremely high dissimilarity on temporal edges are preserved.", "questions": "1. Does the community detection algorithm actually use Louvain or connected components? If it's the latter, why not use the superior Louvain algorithm?\n2. In Table 1, the ST-SimDiff results at Token Retain Ratio (r=50%) even exceed the Token Retain Ratio (Full Performance) results (63.3). Why does this phenomenon occur where the compressed video performs better than the original video? The paper should provide a discussion on this.\n3. Could you provide some concrete examples showing which tokens are selected by SRTS and which are selected by DETS?\n4. When selecting difference frames, the video only considers changes between a few adjacent frames, which may lead to the model being insufficiently sensitive to slowly changing scenes.", "limitations": "", "rating": 6, "confidence": 4, "soundness": 4, "presentation": 4, "contribution": 4, "strengths": "This paper proposes ST-SimDiff, a training-free framework for video token compression in multimodal large language models. The core innovation lies in simultaneously leveraging similarity (to identify redundancy) and dissimilarity (to capture key events) for token selection. The method constructs a spatiotemporal graph and achieves dual-path parallel selection by combining community detection and temporal difference detection. Experiments demonstrate state-of-the-art performance on VideoMME, LongVideoBench, and EgoSchema, while significantly reducing computational costs. The work exhibits strong novelty, effectively achieves its research objectives, and thoroughly validates the method's effectiveness across multiple models and datasets.", "weaknesses": "1.Does the compressed video still retain the audio modality? If yes, how is the alignment between video and audio achieved? If not, does the absence of audio bring negative impacts? If so, how can the negative effects caused by the missing audio be mitigated?\n2.For the same input, this framework seems to produce only one type of output regardless of how the text query instruction varies (rather than automatically adapting based on instruction changes). For different video analysis objectives, is this fixed, unified video pruning framework insufficiently flexible?\n3.Constructing the spatiotemporal graph requires traversing all visual tokens. For videos with extremely high resolution or extremely long duration, is this framework still applicable? In other words, what is the estimated upper limit of visual tokens that this framework can handle?\n4.Considering only changes between adjacent frames may lead to the model being insufficiently sensitive to slowly changing scenes.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (Poster)