SURGE: Surprise-Guided Token Reduction for Efficient Video Understanding with VLMs
Authors: Chong Tang
Sannara Ek
Dirk Koch
Robert D. Mullins
Alex S. Weddell
Jagmohan Chauhan
Venue: ICLR 2026 Poster (2026)
Keywords: Efficient Video Understanding
Vision-Language Models
Token Pruning
Redundancy Reduction
Predictive Coding
Primary area: foundation or frontier models, including LLMs

Abstract: Videos contain rich information but also high redundancy, as consecutive frames often share similar backgrounds and predictable motions. Current video-language models (VLMs) are unable to exploit this redundancy and therefore perform a significant amount of superfluous computation, processing thousands of patch tokens even when little new information is present. What is missing is an on-the-fly, model-agnostic signal of temporal predictability to decide whether tokens carry unpredictable information that merits computation. We propose SURGE, a training-free and backbone-agnostic method that measures surprise in token space. Surprise scores are defined by the prediction error of each token from its recent history; high-surprise tokens are retained, while predictable ones are pruned. Aggregating scores over time produces a surprise curve that highlights key events, which can be further refined with CLIP-based query relevance to form a compact spatio-temporal mask. Experiments on multiple video understanding benchmarks show that SURGE reduces tokens by up to 7$\times$ and prefill cost by 86–98\%, while maintaining accuracy within $\pm$1 point of full-token baselines. By aligning computation with novelty, SURGE enables video VLMs to handle long contexts efficiently and without retraining.

Published as a conference paper at ICLR 2026
SURGE: SURPRISE-GUDDED TOKEN REDUCTION FOR EFFICIENT VIDEO UNDERSTANDING WITH VLMs
Chong Tang"
Sannara Ek
University of Southampton UCL Al Centre
University Cambridge Cambridge, United Kingdom se521 cam. ac uk
Southampton, United Kingdom chong. tang sot on ac. uk
Dirk Koeh
Robert Mullins
University of Manchester Manchester, United Kingdom
University of Cambridge Cambridge, United Kingdom
dirk. www..koocchccceeeeeeeeee ac.
robert www.mmmllmmm...m.mmm.m. cam ac uu
Alex Weddell
Jagmohan Chauhan
University of Southampton Southampton, United Kingdom aswêecs soton. ac uk
University College London
UCL AI Centre London, United Kingdom ja jagmohan. chwuhannucl. uk
ABSTRACT
Videos contain rich information but also high redundancy, as consecutive frames often share similar backgrounds and predictable motions. Current video-language models (VLMs) are unable to exploit this redundancy and therefore perform a significant amount of superfluous computation, processing thousands of patch tokens even when little new information is present. What is missing is an on- the-fly, model-agnostic signal of temporal predictability to decide whether tokens carry unpredictable information that merits computation. We propose SURGE, a training-free and backbone-agnostic method that measures surprise in token space. Surprise scores defined by the prediction error of each token from its recent history; high-surprise tokens are retained, while predictable ones are pruned. Ag- gregating scores over time produces surprise curve that highlights key events, which can be further refined with CLIP-based query relevance to form a com- pact spatio-temporal mask. Experiments marks show that SURGE reduces tokens by to 7x and prefill cost by 86-98% % while maintaining accuracy within +1 point of full-token baselines. By aligning computation with novelty, SURGE enables video VLMs to handle long contexts efficiently and without retraining. hhtpps: / com/ Banp//aarraang2/////// SURGE. git
multiple video understanding bench-
INTRODUCTION
VLMs face fundamental scalability problem in video understanding tasks. Even short clips expand into thousands of visual tokens, and longer videos quickly overwhelm memory and computational resources. The quadratic complexity of attention makes long-context reasoning especially expen- sive, and practical deployments add further constraints such as streaming inputs, limited hardware, and the inability retrain models. Recent large-scale systems, including Intern (Chen al. 2024b), Qwen series VL (Wang et al. (2024a] Bai et al. 2025), and LLa VA-Next (Li et al. [2024a), demonstrate strong video understanding but expose the steep computational price for long inputs.
"corresponding author

---

Published as a conference paper at ICLR 2026
To address this bottleneck, research has explored efficiency along temporal, representational and token-level directions. Temporal methods reduce cost by selecting a subset of frames (Hu et al, 2025 Park et al.] 2024). which can be effective for long-video QA but may miss fine-grained events within frames. depresentatioonfocusee approaches instead compress each frame into fewer tokens, for example by distilling features into compact query embeddings or by learning hierarchical rep- resentations that adjust granularity to budget (Li et al. 2023] (025). At the token level, merging methods collapse redundant patches across space and time (Bolya al., 2023 Lee al., 22024 Choudhury et al., 2024), while pruning methods remove less useful tokens during inference (Zhang al. 2024). These strategies are effective, yet many introduce complexity by requiring auxiliary selectors, fine-tuning of the backbone, or access to internal attention maps that are not always avail- able in deployed systems. What still missing is a training-free, can allocate computation more flexibly as video unfolds.
agebooee-agnostic criterion that
natural way to think about such a signal to ask: which parts of a video actually surprise the model as it evolves over time? We define surprise as the prediction error of token given its recent history: low error indicates predictable, redundant content, while high error signals novel events that merit computation. In videos, where continuity is common, surprise provides clear criterion for allocating effort. Based on this idea, we introduce SURGE, a training-free free and backbone-agnostic surprise mask that directs computation to new content. After the vision encoder produces patch tokens for each frame, SURGE applies a lightweight temporal predictor to estimate each token from its recent history. The surprise score the difference between the estimate and the actual embedding: high-surprise tokens are kept, while predictable ones are pruned. Aggregating token scores over time yields a surprise curve which, after smoothing and peak picking, provides key event windows. The surprise mask is usable as is for efficient inference, and can be further refined with CLIP-based query relevance: we reweight event windows by query-key query-keyframe similarity so the final spatio-temporal mask prioritizes content that is both new and on-topic. This reduces tokens before the language interface, adds negligible overhead, and fits naturally with other methods (e.g. AKS (Tang et al.) (025)). Across comprehensive experimental analysis, SURGE reduces tokens by up to 7x and prefill cost by 86-98% while keeping accuracy within =1 point of full-token baselines. The pipeline of SURGE is shown in Fig. 2 Our contributions can be summarized as:
A training-free method that measures temporal predictability in token space, yielding sur- prise curves and token masks that work with any ViT-based backbone without retraining.
A compact spatio-temport masking scheme, effective on its own and further refinable with CLIP for query- aware event selection.
Extensive experiments across models, benchmarks and pruning budgets, demonstrating SURGE's robustness and its ability to combine with complementary strategies (e.g., keyframe selection) for additional gains.
RELATED WORK
Sparsity as Selection. A common strategy for VLM efficiency is to process only a sparse sub- set of the visual stream, either across time at the token level. Temporal sparsity assumes most frames are redundant, so a small set of keyframes suffices. Methods such as SeViLA (Li et al.) 2023 Yu et al. 2023), ViLA (Wang al. 2024b), and AKS (Tang al. 2025) follow this idea, improving long-video QA but sometimes missing fine-grained evidence when relevance estimates are imperfect (Park et al., 2024 Hu al. 2025). Recent work such as BOLT (Liu al. 2025)
and ViLAMP (Cheng et al. (025) extend this by jointly selecting relevant frames and condensing information hierarchically. Token sparsity instead prunes less useful patches within frames. Ap- proaches like Fast (Shu et al. (202)). IVTP (Huang et al., 2024), ATP-LLaVA (Ye et al.) 2025).
and SLM (Zhag et al. 20024 typically rely on attention thresholds or relevance scores. More recent methods like DyCoke (Tao et al. 2025) and DivPrune (Alvar et al., 2025) aim to dynamically retain diverse or cache-critical tokens, achieving strong accuracy without retraining. These methods are effective, yet often need careful calibration of thresholds and policies, and some depend on in- ternal attention maps often unavailable in deployed systems (Wei et al. 2023 Chen et al., 2024a). Realistically, sparsity proxies highlight what seems important at the current step but not what has changed, so redundant but on-topic segments may still absorb computation while unseen events can be overlooked.
2

---

Published as a conference paper at ICLR 2026
Par wpt- Rerteery
patch
patch
patch 358
potch
Token surprise vs. pixel change. For four random patches, we compare pixel dissimilarity (blue), raw surprise (orange; Eq. B). and normalized surprise (purple; Eq. (5. Normalization sup- presses drift and aligns peaks with true content changes.
Figure 1:
Representation Compression. Another strategy is to compress visual representations fewer tokens are processed per frame or interval. Some methods replace dense patches with compact learned embeddings: LLaMA-VII encodes each frame into just two tokens via cross-attention (Li ett al. 2024b), Video-XI. learns adaptive summarization tokens (Shu et al., 2025). and Matryoshka generates nested coarse-to-line token sets for dynamic accuracy-efficiency trade-offs (Cai et al. 2024). These designs achieve large tokens reductions but require retraining, architectural changes. or additional hyperparameters. Another complementary approach token merging (ToMe) (Bolya it al. 2023), which progressively merges similar patches during inference to reduce redundancy. Ex tensions to video include methods such as vid-TLDR (Choi et al., 2024) and TempMe (Shen et al.) 2024a), as well as spatio-temporal schemes like STTM (Feng et al. 2024 Lee et al. [2024] Choud- hury et al. 2024). Recent works like VisionZip Yang et al. (2025). 5. Long VU (Shen et al. 2024b) and Chat- Uni Vi (Jin et al., 2024) further combine spatial and temporal compression enable long- horizon video processing. Two-stage pipelines like PruMerge (Shang et al. 2024) combine pruning and clustering for up 14x reductions. While effective, these approaches depend architectural modifications, merge schedules, or retrained summarizers that are often model- or dataset-specific.
Predictability and Surprise. Beyond sparsity and compression, longstanding idea in cognitive science and machine learning is to use prediction error (surprise) as a signal of information value. Predictive coding suggests that expected inputs are suppressed while unexpected ones receive deeper processing (Itti & Baldi 2005), and curiosity modules in reinforcement learning reward high pre- diction error to drive exploration (Pathak al., (0017). In video, frame-level errors have been used for anomaly detection (Liu et al., 2018). The common principle is that novelty, defined as a "devi- ation from expectation" deserves computation. By contrast, most VLM efficiency methods rely on proxies such as attention weights, similarity scores, or trained selectors (Zhang et al., [2024] Huang jet al., 2024 Ye et al. 2025), which add complexity and often depend on model-specific thresholds or attention maps that may not transfer across datasets or deployment settings.
We argue that more intuitive criterion to test predictability itself: tokens that align with prior context contribute little new information, while those that diverge from expectation indicate gen- uine change. Unlike attention weights or similarity scores, this signal directly measures novelty as it unfolds, offering lightweight and training-free mechanism that complements sparsity and compression by aligning computation with new information.
3 FROM PREDICTABILITY TO SURPRISE IN TOKEN SPACE
3.1 PRELIMINARIES AND TOKEN DYNAMICS
Given sequence of frames I Txx HxW the vision tower partitions each frame into m spatial
cells and produces patch embeddings 7, = [2(21) ,...,,? = {(m)] E mm where 2(") E IRd the token for spatial index j. Formally, -(2) f (Tz), where f; is the encoder map for cell j. Natural videos
evolve smoothly: consecutive frames satisfy [4+1 R T, + AI, with bounded perturbation AT, Since
fy is differentiable, first-order Taylor expansion gives
2. ( + Jr, (I:) AL,
241
where Jí, (T)) is the Jacobian of ʃz at Tt. This implies approximately linear dynamics in token space:
(
4+1 22
2 0.
3

---

Published as a conference paper at ICLR 2026
Response:
Surprise-Peaks Keyframes
Low-Surprise
Multimodal
Frames
EL
X
E2
E:3
X
Tn
User Query
CLIP Query-Aware I Focusing
O: How fong does take boll potatoes?
E1
E3
2
Surprise
E2
Surrpiie traction
St
Linear Projector
00
E1
VIT Tower
Surprise Curve Key Events
surpeiso
(1)
8
60
70
Temporal Cnit (Chunl)
Figure 2: SURGE pipeline. Tokens are predicted with constant-velocity model, then detrended and variance-normalized to yield surprise scores. This surprise computation is performed directly on the raw patch embeddings output by the vision tower before projection into the language model. We retain the top-p tokens, aggregate them into surprise curve to segment key events, and optionally rank events with CLIP (Top-K). Finally, only high-surprise tokens from key events are forwarded to the LLM, while others are pruned. Note, the SURGE block appears after the projection layer to reflect where the masking is applied, but the surprise scores themselves are computed earlier, on pre-projection features.
Eq. 2 encodes constant-velocit prior: smooth motions satisfy it, while abrupt events yield large deviations interpreted as surprise. This connects to representation learning approaches that model videos as smooth stochastic processes (e.g., Brownian bridges) (Zhang et al. (023) and to frame interpolation methods that rely on constant-velocity or higher-order motion priors (Zhong et al.) 2024). We adopt the same principle at the token level, using deviations from constant-velocity prediction as surprise signals that highlight semantically meaningful events.
3.2 TRAINING-FREE TEMPORAL PREDICTION
causal constant-velocit predictor, common in tracking and interpolation, to estimate tokens from recent displacements.
Building Sec. 3.1
Constant-velocity extrapolation. For token let 2zt")
.0) ( denote its displacement
between consecutive steps. A natural causal predictor is to carry forward the most recent displace- ment,
2(()
+
801 8 = -
For < 2, we initialize 8(2) 0 and set 2(6 2 This autoregressive extrapolation is consis- tent with the austant-velocity prior in Eq. 2 is causal, training-free, and applies uniformly across spatial indices, ensuring compatibility with any ViT backbone. However, this naive form cannot dis- tinguish novelty from large-scale coherent motion, arising from camera panning, scene-level shifts, or other global transformations, which induce nearly uniform changes across tokens (Lian et al. 2023).
Global drift correction. Such drift is spatially smooth and can dominate residuals, causing false surprise (Lian et al. 22233 James et al. 2023). To suppress this effect, we approximate the displace- ment field with affine function of spatial coordinates, standard choice in motion compensation and stabilization (Nie et al. 2024). Let (2,, denote the normalized position token j. and define
AZ( RJ co + Cy Mjj
with coefficients co, Co, Cy E Rd fit by least squares. Concretely, let X [1, y] R stack token coordinates and AZ, Rmxd stack displacements row-wise; the closed-form is
4

---

Published as a conference paper at ICLR 2026
Ö (X AZ Raxd with rows corresponding to cj C2 c.g. The detrended dis- placement then
+ CC Wj). This removes global translation (cn) and first-order planar flow (Cz: Cy), ensuring that residuals re- flect true surprise rather than scene-level drift. Special tokens (e.g., /CLS] or register tokens) excluded from this fit and always retained. The 3 x 3 least-squares solve per frame adds negligible ((md) cost. Finally, Eq. 3 is applied using the detrended displacement i-1 to produce the causal prediction.
(4)
Surprise scoring. Given the causal prediction =("), we compute the surprise vector: e/ =
2t which measures the prediction error of token j. Under smooth motion, the vector remains small but new information makes them large. To obtain a scalar measure, we define the surprise score of token j by normalizing the squared magnitude with a running estimate its variance:
||E 12
2.5 O +
where di 2.00 an exponential moving average of past vector variances for token j, and > pre- vents division by zero. This auriacc-nnrmallin formulation provides calibrated statistic: under locally linear-Gaussian dynamics with no change, s (f) stays near its expectation, while significant deviations directly indicate surprise.
As a sanity check, Fig. shows that token-space surprise tracks pixel change, validating predictor residuals proxy. Specifically, the raw constant-velocity predictor (Eq. often fires spuriously under global shifts such as camera pans. After applying drift detrending (Eq. and variance nor- malization (Eq. 5), surprise spikes align more closely with true content changes, suppressing noise signal.
and yielding a stable,
4 FROM TOKEN SURPRISE TO A SPATIO-TEMPORAL MASK
Given per-token surprise scores s (Sec. 3
we produce compact spatio-temporal mask in two stages: (i) global percentile thresholding keep the most surprising tokens across the sequence, and (ii) construction of surprise curve per temporal unit to segment key events. For query-focused applications, then rank these events using CLIP similarity (Radford al. 2021) peak frames and concentrate computation on the top-K relevant ones.
4.1 ADAPTIVE TOKEN MASKING
Within each sequence, we adaptively keep the most surprising tokens according to a global per- centile. Let B denote the current buffer of tokens (the full clip offline, or the observed prefix in streaming), and collect all surprise scores S3 { : (u,j) B.. The global p-percentile is (()) = Quantile,(Ss) for E (0, 1). The binary mask is then
Mu. KK{ s( 2 {pp)
This global selection retains the top p == (1 - fraction where content changes: dynamic frames contribute more, redundant frames contribute less. Note that special tokens (e.g. (CLS)) are always preserved. The mask is applied after the vision encoder and positional encodings, and before the lan- guage model, making the procedure training-free backbone-agnostic, and compatible with standard ViT-based pipelines.
4.2 SURPRISE CURVE AND KEY EVENTS
After generating token-level masks, we aggregate surprise over time to detect salient events. For each temporal unit frame or a chunk), we count how many tokens exceed the global threshold: S, Σj=1 Mu.j. cquivalently the fraction S,/ m. The sequence {S.} forms the surprise curve.
We smooth it with exponential moving average, S., S.-1 + (1 S,, to reduce noise.
5

---

Published as a conference paper at ICLR 2026
Peak-based event segmentation. Let P found with minimum separation A and a small prominence threshold. Let U be the total number of units. We define event boundaries at midpoints between adjacent peaks:
{n <... <<..<<T be the local maxima (peaks) of Su,
bp
bk
Te Tett
(k- I,..., 1),
bn U,
and assign each peak & the event interval Is [b by ), 1=k= 1,.... Thus, durations between peaks are taken as key events.
Query-aware event focusing. To align events with text query 9. peak frames. Let Ur, be the frame-level embedding at peak Th and define r'K = sim (g, Ury). We rank events by {rk] and keep the top-K: &K (rr). We then concentrate computation on {I& k E } by applying Ma. within these intervals, while maintaining only a small context floor elsewhere. Although this adds one CLIP pass over the set of peak frames, it further reduces tokens forwarded to the language model and improves focus in very long-context retrieval. Finally, based on the surprise mask Ma.jj event intervals {Ix), and CLIP-selected indices εk, define the event indicator A., = X { E UKERR In J. With small context floor Cu.j (e.g., the top-keax tokens by Su when Av = 0), the final mask is (see Appendix A.4 and A.5 for additional visualizations): (1)
CLIP similarity at the
M = Au Mag + (1 A.) Cu.--
(8)
5 EXPERIMENTS
Benchmarks & Baselines. We evaluate SURGE on five representative video-language bench- marks: (1) Video-MME (Fu et al.1 (025): overall QA on short-long clips; (2) MLVU (Zhoul
itt al. 2025): long-video multi-task evaluation (M-Avg/G-Avg) including Needle QA, grounding,
and summarization; (3) MMBench- (Fang al. 2024): curated multi-step QA probing com- positional/temporal reasoning; (4) TempCompass (Liu et al. 2024): structured temporal QA on short clips; (5) LongVideoBenee (Wu et al. 2024): very long videos emphasizing retrieval and cross-event reasoning. As baselines, we compare against random token pruning. FastV (token prun- ing/aggregation) (Shu al. 2025) and AKS (adaptive keyframe selection) (Tang et al., 2025) ) cho- sen because they are publicly available, replicable, and applicable across model families J We also report AKS w/ SURGE and AKS w/ SURGE" to assess complementarity with temporal selection. Full details and justifications are in Appendix A.3 and Appendix A.2
Models. To demonstrate SURGE's breadth, we evaluate three flagship VLMs: (i) InternVL-3.55
VL (Wang al. 2025) (8B; 15k-token context): the latest open-source model with strong general- (Lin et al.] 2023) (7B;
purpose video QA and extended text window; (ii) Video-LLaVA-Qwer
64-frame cap): a mid-size video variant with strict frame budget, highlighting token efficiency; (iii) Qwen2.5-VI. (Bai et al. (025) (7B; 13/k-token context): a long-context model able to process hours-long vidcos, ideal for testing SURGE's scalability. Together these models provide a compre- hensive generality test. Further implementation details and justifications are in Appendix A.I and Appendix A.3
Evaluation pipeline & Hyperparameters. We use VLMEvalKit (Duan et al. 2024) for bench- marks, applying SURGE purely inference. Experiments run on 1-8 A100 80GB GPUs. By default, we retain the top P = 0.25 tokens (75th percentile), smooth surprise curves with EMA (7 = 0.9), and enforce a minimum separation A units between peaks. For query-focused re- trieval, we select the Top-5 events by CLIP (ViT-B/32) similarity at peak frames. No context floor (Eq. B is used in the main experiments, isolating surprise-driven gains.
5.1
PERFORMANCE
Large token reductions with stable accuracy. At the default 0=0.25, SURGE retains only ~26%-27% of visual tokens (~4xx fewer), and SURGE* with CLIP Top-5 further reduces this to 14%% (~77x fewer). Yet benchmark scores remain within +1 point of the baseline (Tab. validating our hypothesis that predictability in token space is a reliable proxy for redundancy
'Several other methods currently support only specific VLMS. For FastV, we follow its official implemen- tation and adapt to Qwen2 LM by replacing the k-1 attention layer with cager version to access attentions.

---

Published as a conference paper at ICLR 2026
@: are the guurds gosstiioing family??
Q: Which animals ottack the explorers on the bridger?
Blink afress
Middlt: quardz are guossianing a nann
lavima
Base Model: The explorers are offached by beboons on the bradges.
SURGE: guarsh are gosstioning a group of people including a in suni, who appears to be magician performer, and sxomun in block aless.
SURGE:
The animals that atlack explorers on bridges are babcons.
SURGE": The animals that ottack the explorers the bridges
SURGE": The gnard ore quertioning group of people inchuding a mun black drert The the ritt explatning nomething
baboons,
and
animaals are
woman black
part
w/SURGE: They anlacted by baboons.
the cunversation.
Random:
Gorillas
AKS: guards are questioning a suu and woman block
idress The mun idencffied in the mugicion, and the womun his axristunt
guards are questtoning a group of people factity. Thi tows shows a woman in black drez and man in nami being quesioned the guorls
Figure 3: Qualitative examples. SURGE and SURGE* even provide more accurate and detailed answers than the base model.
Random: The comon appsars to be the foom of the queationing, as she
being held by the non and the being directly odddeesedd by quards.
Table Benchmark results (64 frames). SURGE retains a global top-p=0.25 of tokens; SURGE* adds CLIP Top-5 event focusing. "Tokens" reports average fisual-ffixed text tokens per sample 64 frames. t Exceeds Intern VL-3.5LL- 15k-token context; inputs were truncated by the runtime, so scores reflect truncated context. Best performance in bold.
Model
Tokens
V-MME
MLVU (M/G)
MMB-V
T-Compass
L.B
VntemnV--355VL (8B)
17,124
66.0
71.7/
1.54
68.9
61.3
+ SURGE
4.674
64.9
7..5/
1.57
69.0
61.7
+ SURGE"
2,932
71.7 /3.69
1.57
69.7
62.2
+ AKS
17,124
66.1
71.6/3.48
1.54
68.6
62.2
+ AKS w/ SURGE
4,819
65.9
71.7/3.39
1.56
68.6
62.0
AKS w/ SURGE"
2,390
66.2
1.55
68.4
61.9
Lin--LLLV--Qoor (7B)
12,246
63.4
72.9 3.30
1,53
66.9
58.3
+ -URGEE
3,324
63.1
72.9
1,53
67.0
59.1
+ SURGE"
1,884
72.7 3.20
1.60
66.9
61.9
+ FastV
3,300
58.1
52.3 3.11
1.29
61.7
55.4
+ AKS
12,246
64.7
72.7 / 3.30
1.55
66.9
61.4
AKS w/ SURGE
3,157
64.9
7
1.55
66.7
62.3
+ AKS SURGE'
1,961
62.4
72.8/3.56
1.61
66.9
62.2
Qwen2.5-VL. (7B)
41,590
62.2
45.8/4.26
1.60
70.5
60.0
+ SURGE
10,992
60.9
65.7 4.26
1.72
70.5
59.4
+ SURGE"
5,207
62.7
66.1/4.24
1.70
67.7
61.3
+ AKS
41,590
62.0
65.8 / 4.26
1.67
70.7
59.9
+ AKS w/ SURGE
11,588
61.9
15.8/4.25
1.68
71.1
60.3
+ AKS SURGE"
6,204
62.7
66.0/ 4.26
1.73
70.9
60.4
The qualitative examples (Fig.) further illustrate this: even when the base model gives in- complete answers, SURGE and SURGE* recover more accurate and detailed responses by focusing on novel and relevant tokens. In contrast, random pruning often drops critical evidence, producing unstable outputs, while surprise-driven selection preserves informativeness.
(Sec. 3).
Performance across contexts. On benchmarks stressing long videos and cross-event reasoning, SURGE* not only preserves but surpasses full-token baselines: e.g., Tab. Intern L-3.5-VVL... on LVB (+0.9) and MIVU G-Avg (+0.25), with consistent improvements T-Compass (+0.8) and MMB-V (+0.03). These are precisely the conditions where surprise- reallocation and query- aware focusing are most usefull confirming that SURGE effectively identifies and prioritizes new, relevant content (Sec. 4p. For short-form QA benchmarks such as V-MME, where redundancy is lower, SURGE induces only small fluctuations (typically within +1 point). Importantly, SURGE* often recovers or slightly improves over the baseline (e.g., +1.1 on Video-LLaVA-Qwen under- scoring that SURGE is safe to apply even in short-video settings with relatively low redundancy.
Comparison to pruning and keyframe baselines. Against FastV (attention based pruming), SURGE is markedly more reliable the same budgets, e.g., Video-LLaaA VAV--ve,, V-MME 64.5

---

Published as a conference paper at ICLR 2026
Table 2: Token-accuracy trade-off on Qwen2.5-VL. p is the fraction of visual tokens retained
(1.0=full tokens). "+%" indicates the maximum relative deviation (5 runs), reflecting result stability.
Method
V-MME
MLVU (M/G)
MMB-V
T-Compass
LVB
Qwen2.5VVL (7B)
1.00
62.2
65.8/ /..2
1.60
70.5
60.0
SURGE +0.8%
0.75
62.3
65.8 / 4,22
1.60
70.3
60.5
SURGE" =1.1%
0.75
62.1
66.4 4,25
1.60
70.2
58.1
+ Random 13.7%
0.75
61.9
65.7 / 4.26
1.61
69.7
50.1
SURGE +%
0.50
62.0
65.8 4.26
1.66
72.3
59.7
+ SURGE* +0.5%
0.50
62.1
66.2 / 4,22
1.70
71.7
57.9
Random 1.2%
0.50
62.0
64.9 / 4.24
1.62
66.1
57.0
SURGE +0.4%
0.25
60.9
65.7 4.26
1.72
70.5
59.4
SURGE" +0.6%
0.25
62.0
66.1 / 4.24
1.70
67.7
56.3
Random 13.2%
0.25
53.6
55.9 3.32
0.96
57.4
49.7
SURGE +0.7%
0.10
60.3
65.8 / 4.22
1,58
70.2
58.6
Random 233.9%
0.10
36.8
37.4 3.11
0.94
50.9
46.0
SURGE 1.00%
0.01
58.7
65.8 4.22
1,55
70.2
55.7
Random 19.8%
0.01
29.8
3.10
0.84
50.6
37.3
Table 3: Top-K CLIP event focusing evaluation on Qwen2.5-VL. (7B). Baseline=full tokens, SURGE=percentlle masking (p=0.25), SURGE'=SURGGE (p=0.25) + CLIP Top-K K event selection.
Method
K
V-MME
MLVU (M/G)
MMB-V
T-Compass
LVB
Qwen2.5-VL (7B)
62.2
65.8/4.26
1.60
70.5
60.0
+ SURGE
60.9
65.7 /4226
1.72
70.5
59.4
SURGE"
51.7
37.9
1.09
23.9
40.1
+ SURGE'
59.0
/..5
1.15
59.7
49.6
+ SURGE"
62.0
66.1 /4.24
1.70
67.7
56.3
+ SURGE"
62.3
66.8/4.30
1.65
70.5
60.2
+ SURGE"
10
62.7
66.8 /.226
1.71
71.0
60.2
vs. 1, MLVU 72.7/3.20 vs. 52.3/3.11 11 (Tab. indicating that temporal surprise can be stronger criterion than attention magnitude for deciding which visual tokens to keep in video understanding tasks. In contrast, AKS selects fewer frames instead of pruming tokens, and can match surpass baselines on line-video metrics. Combined with SURGE, the two act complementarily, obtaining further gains in efficiency and accuracy, showing SURGE is both competitive alone and composable with temporal selection.
5.2 BBFICIENCY-ACCURACY TRADE-OFFS
Token Efficiency Analysis. At moderate pruning (p=0.50-0.75), SURGE maintains accuracy within #1 of bascline, while improves it for T-Compass (+1.8) and MMB-V (+0.1-0.2). This suggests that surprise-guide guided masking effectively discards predictable background while preserving transition-hest heavy evidence. CLIP focusing further helps in some cases (e.g., MIVU, +0.3 M-Avg), validating that query-aware peak selection reallocates budget toward relevant content.
The random pruming baseline provides an informative contrast: at moderate levels (p20.50), can approximate baseline accuracy, confirming that redundancy is indeed present in visual tokens. However, once more than 75% of tokens are dropped, performance becomes highly unstable, with maximum relative deviations exceeding 20% (Tab. 2 reflecting frequent loss of critical informa- tion. By comparison, SURGE stays within + 1.1% across all pruning levels, remaining stable even under aggressive settings (p=0.10-0.01). This yields usable accuracy while exposing a clear com- pute-accuracy trade-off.
Effect of Top-K CLIP Focusing. We investigate CLIP event focusing by varying K, the number of peak events retained (Table Extremely small K (e.g.. K=1) collapses coverage and causes severe accuracy drops, while larger K steadily restores performance: with K=5-10, SURGE* of-
8

---

Published as a conference paper at ICLR 2026
ten matches or exceeds the full-token baseline, especially on long-context benchmarks like MLVU and LVB. This illustrates the precision-coverage trade-off: small K risks missing relevant events, whereas moderate K balances query alignment with sufficient context. In practice, K=5-7 provides a robust setting, confirming that query-aware focusing complements surprise masking by allocating compute both new and relevant content.
Long-Context Performance. On MLVU with Qwen2.5-V1 we test up to 3600 frames (Fig.
Accuracy rises for both baseline and SURGE up to 256 frames. Beyond 230 frames, the baseline exceeds A100 80GB VRAM, while SURGE/SURGE" remain executable, extending to 1024 and 3600 frames. Performance drops when truncation dominates (e.g.. 52.3/3.61 1024, 35.4/3.30 at 3600), but the key result is that SURGE raises the practical upper bound, enabling over an order- of-magnitude longer videos to be processed on fixed hardware, while maintaining near-baseline accuracy within the feasible range.
70-
>131k
>111 context lime
05-
C0-
MLVU G-hug
4.--
MIVU M.
2.-
SUNGE pmo.25
SUNGE p-o.25
SURGE* p-0 TTpp
3.6
SURGE p-0.25 (Top
SURGE p-0.10
SURGE p-0.10
SURGE
3.4
SUAGE p-0.01
64
120 256
1k
3.6k
64
128 256
1k
Frames
Frames
Figure 4: Long-context evaluation on MLVU using Qwen2.5-V. M-Avg and G-Avg across frame counts; shaded area exceeds the ~ -31kkttoenn limit. SURGE and SURGE* extend capacity to 3600 frames while staying competitive below the limit.
Efficiency Analysis. On Video-MME with Qw (7B), SURGE achieves substantial prefill
savings: at 0=0.55, FL.PP//atency drop by 86%/79% (Fig. ((a). respectively, while generation FL.OP//atency also show modest reductions (-38 1-14% % (Fig. {()).))). At extreme pruning (p=0.01), prefill costs shrink by over 98%, and generation FI.OPs are roughly halved. Total visual tokens reduce by -72% (p=0.25) to -96% (p=0.01). Together with accuracy stable at moderate pruning (Tab. 2 this shows SURGE translates token savings into substantial FLOP/latency gains, especially in the compute-heavy prefill stage. For SURGE*, a CLIP pass over 5-8 peaks adds roughly 0.63-1.0 TFL0P11102-1191 ms per query, while pre-LLM pruning still removes most KV-cache and prefill load, keeping both variants effective under memory or throughput limits.
(a) Preftl FLOPs (T)
(b) Ganeration FLOPs (C)
(c) Profil Labency (ms)
(d) Canaration Latercy (ms)
4000
80-
12.5
60
29
2000
40
15
50
10
20
3-
00
0.75 0.50
0.01
Basc 0.75 0.25 0.10 0.01
Base 0.25 0.01
Base
Figure 5: Efficiency on Video-MME with QwenZ.5-VL (7B). X-axis in all panels: token retention Approximate total token counts per setting: 41.6k, 31.6k, 21.5k, 11.6k, 5.5k, 1.8k.
5.3 COMPONENT ABLATIONS
Table Treports ablations on: MLVU (long multi-task), TempCompass tine-ggrined temporal), and MMB-V (multi-step QA), chosen to cover complementary reasoning settings without redundancy. Removing drift detrend or variance normalization yields moderate but consistent drops, showing
Extended 131k context window, ~23 frames; beyond this inputs truncated though outputs remain.

---

Published as a conference paper at ICLR 2026
Component ablations at fixed budget. Each row removes single ingredient from SURGE (global affine drift detrending, variance normalization, or the causal temporal predictor), while keeping the global percentile mask and event segmentation unchanged.
Table 4:
Variant (Qwen2.5-VL, 0=0225)
MLVU (M/G)
T-Compass
MMB-V
SURGE
65.7 4.226
70.5
1.72
wfo drift detrend (Eq. 4
64.9 4.18
69.4
1.65
variance norm (Eq. 5
65.1 / 4.22
1.70
w/o temporal predictor (rrme-difff only Eq.
63.4/4.17
66.9
1.55
their role in stabilizing surprise. The largest degradation comes from discarding the temporal pre- dictor, where same-differences mistake smooth motion for novelty. Overall, each component con- tributes to robustness. In addition, also sweep hyperparameters and find SURGE to be stable: EMA smoothing 7 E [0.7, 0.95]. peak separation A E [4, 12). in Tab. and percentile Tab. With default settings, SURGE (7=0.9, A 8, K 5, p=0.25) offers good balance.
CONCLUSION
We introduced SURGE, training-free and backbone-agnosta method that allocates computation to high-surprise content. SURGE addresses the scalability of video VLMs by cutting the cost of long inputs and expanding the cemmory-bounder effective context: by pruming predictable visual to- kens before the multimodal LLMs, it reduces embedding/attention activations and K VV-cache growth, which are typically constrained by the device VRAM//sstem memory and latency budgets. In prac- tice, SURGE reduces token counts by up to 7x and cuts prefill cost by nearly 90%, while keeping accuracy within +1 point of full-token baselines. On commodity accelerators, this enables pro- cessing sequences that would otherwise hit out-of-memory (or unacceptable latency) for the same model; SURGE also composes with keyframe selection or query-aware focusing for added effi- ciency. A current limitation is that SURGE* requires extra CLIP pass and is sensitive K and query phrasing. Future work will explore lighter relevance models, adaptive event selection, and in-context alignment improve robustness.
ACKNOWLEDGMENTS
This work was supported by the Engineering and Physical Sciences Research Council (EPSRC), under the project Perfect Recollection for Clearer Insight (grant number EP/Y036077/1) and the computing resources and software from the NVIDIA Academic Grant program.
REPRODUCIBILITY STATEMENT
We have taken several steps to ensure reproducibility. All models used in this work (Intern 3.5-VL, Video-LLVVA-Qwen, and Qwen2.5-VL) are publicly available through HuggingFace, and Appendix A.I details where SURGE is integrated into each codebase. Our implementation lies only on open-source tools and libraries, including HuggingFace Transformers, OpenAl CLIP, and VLMEvalKit for standardized evaluation VidEo-MME, MLVU, MMBench- MiiBench-Vidoo, Temp- Compass, and LongoideoBench Hyperparameter ranges, default values, and ablation settings are reported in Section and the appendix. Hardware (NVIDIA A A100 80GB GPUs) and evaluation protocols (latency, FLOPs, token counts, and accuracy) are likewise fully specified. We will release our SURGE/SURGE* modules, modified model wrappers, and experiment scripts upon acceptance to enable exact reproduction of all tables and figures. https:://// pttps://gin /tt:://ggttu..../ ctm.t/..ooooaaaaaa Baanm/BarrTTaag221
SURGE. qit
ETHICS STATEMENT
This work focuses on improving the efficiency of video VLMs at inference. We do not collect new datasets or involve human subjects; all models and benchmarks used are publicly available under
10

---

Published as a conference paper at ICLR 2026
their respective licenses. Our method reduces computation and memory usage, which can lower the energy footprint of large-scale inference. Potential risks stem from the underlying pretrained models (e.g.. biases misuse), which SURGE does not alter. We encourage responsible deployment of VLMS and will release our code to facilitate transparent and reproducible research.
REFERENCES
Saeed Ranjbar Alvar, Gursimran Singh, Mohammad Akbari, and Yong Zhang. Divprune: Diversity- based visual token pruning for large multimodal models. In Proceedings of the Computer Vision and Pattern Recognition Conference, pp. 9392-9401, 2025.
Shuai Bai, Keqin Chen, Xuejing Liu, Jialin Wang, Wenbin Ge, Sibo Song, Kai Dang, Peng Wang, Shijie Wang, Jun Tang, et al. Qwen2. 5-vl technical report. arXiv preprint arXiv:2502.1 132233, 2025.
Daniel Bolya, Cheng- Yang Fu, Xiaoliang Dai, Peizhao Zhang, Christoph Feichtenhofer, and Judy Hoffman. Token merging: Your vit but faster. In ICLR, 2023.
Mu Cai, Jianwei Yang, Jianfeng Gao, and Yong Jae Lee. Matryoshka multimodal models. In Workshop on Video-Language Models@ NeurlPS 2024, 2024.
Liang Chen, Haozhe Zhao, Tianyu Liu, Shuai Bai, Junyang Lin, Chang Zhou, and Baobao Chang. An image is worth 1/2 tokens after layer 2: Plug-and-play inference acceleration for large vision- language models. In European Conference on Computer Vision, pp. 19-35. Springer, 2024a.
The Chen, Jiannan Wu, Wenhai Wang. Weijie Su, Guo Chen, Sen Xing, Muyan Zhong, Qinglong Zhang, Xizhou Zhu, Lewei Lu, et al. Internvl: Scaling up vision foundation models and aligning for generic visual-linguistic tasks. In Proceedings of the IEEE/CVE conference on computer vision and pattern recognition, pp. 24185-24198, 2024b.
Chuanqi Cheng, Jian Guan, Wei Wu, and Rui Yan. Scaling video-language models to 10k frames via hierarchical differential distillation. In Forty-second International Conference on Machine Learning. 2025.
Joonmyung Choi, Sanghyeok Lee, Jaewon Chu, Minhyuk Choi, and Hyunwoo Kim. Vid-tldr: Training free token merging for hight-weight video transformer. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 18771-18781, 2024.
Rohan Choudhury, Guanglei Zhu, Sihan Liu, Koichiro Niinuma, Kris Kitani, and Lászłó Jeni. Don't look twice: Faster video transformers with run-length tokenization. Advances in Neural Informa- fion Processing Systems, 37:22127-28144 2024.
Haodong Duan, Junming Yang. Yuxuan Qiao, Xinyu Fang, Lin Chen, Yuan Liu, Xiaoyi Dong. Yuhang Zang, Pan Zhang, Jiaqi Wang, et al. Vimevalkit: An open-source toolkit for evaluat- ing large multi-moddlity models. In Proceedings of the 32nd ACM International Conference on Multimedia, pp. 1118-112211 2024.
Xinyu Fang, Kangrui Mao, Haodong Duan, Xiangyu Zhao, Yining Li, Dahua Lin, and Kai Chen. Mmbench-video: A long-form multi-shot benchmark for holistic video understanding. Advances in Neural Information Processing Systems, 37:89098-93 010098889124 89124, 2024.
Zhanzhou Feng, Jiaming Xu, Lei Ma, and Shiliang Zhang. Efficient video transformers via spatial- temporal token merging for action recognition. ACM Transactions on Multimedia Computing, Communications and Applications, 20(4):1-21, 2024.
Chaoyou Fu, Yuhan Dai, Yongdong Luo, Lei Li, Shuhuai Ren, Renrui Zhang, Zihan Wang, Chenyu Zhou, Yunhang Shen, Mengdan Zhang, et al. Video-mme: The first-ever comprehensive evalu- ation benchmark of multi-modal 11ms video analysis. In Proceedings of the Computer Vision and Pattern Recognition Conference, pp. 24 24108-24118, 2025.
Kai Hu, Feng Gao, Xiaohan Nie, Peng Zhou Son Tran, Tal Neiman, Lingyun Wang, Mubarak Shah, Raffay Hamid, Bing Yin. al. M-llm based video frame selection for efficient video understanding. In Proceedings of the Computer Vision and Pattern Recognition Conference, pp- 13702-13712, 2025.
11

---

Published as a conference paper at ICLR 2026
Kai Huang, Hao Zou, Ye Xi, BoChen Wang. Zhen Xie, and Liang Yu. Ivtp: Instruction -eaietiion-uidedd visual token pruning for large vision-language models. In European Conference on Computer Vision, pp. 21-2230. Springer, 2024.
Laurent Itti and Pierre Baldi. Bayesian surprise attracts human attention. Advances in neural infor- mation processing systems, 18, 2005.
Jerin Geo James, Devansh Jain, and Ajit Rajwade. Globalflownet: Video stabilization using deep distilled global motion estimates. In Proceedings the IEEE/CVF winter conference on applica- tions of computer vision, pp. 5078-5087, 2023.
Peng Jin, Ryuichi Takanobu, Wancai Zhang, Xiaochun Cao, and Li Yuan. Chat-umivi: Unified visual representation empowers large language models with image and video understanding. In Pro- ceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 13700- 13710, 2024.
Sehoon Kim, Sheng Shen, David Thorsley, Amir Gholami, Woosuk Kwon, Joseph Hassoun, and Kurt Keutzer. Learned token pruning for transformers. In Proceedings of the 28th ACM SIGKDD Conference on Knowledge Discovery and Data Mining (KDD '22), 2022.
Seon-Ho Lee, Jue Wang, Zhikang Zhang, David Fan, and Xinyu Li. Video token merging for long video understanding. Advances in Neural Information Processing Systems, 37:13851-13871 2024.
Feng Li, Renrui Zhang, Hao Zhang, Yuanhan Zhang, Bo Li, Wei Li, Zejun Ma, and Chunyuan Li. Llava-next-interleeae: Tackling multi-image, video, and 3d in large multimodal models. CoRR, 2024a.
Junnan Li, Dongxu Li, Silvio Savarese, and Steven Hoi. Blip-2: Bootstrapping language-image pre-train pre-training with frozen image encoders and large language models. In International conference on machine learning, pp. 19730-19742. PMLR, 2023.
Xinhao Li, Yi Wang, Jiashuo Yu, Xiangyu Zeng, Yuhan Zhu, Haian Huang, Jianfei Gao, Kunchang Li, Yinan He, Chenting Wang. et al. Videochat- flash: Hierarchical compression for con-ccontext video modeling, CoRR, 2025.
Yanwei Li, Chengyao Wang, and Jiaya Jia. Llama-vid; An image is worth 2 tokens in large language models. In European Conference on Computer Vision, pp. 323-340. Springer, 2024b.
Long Lian, Zhirong Wu, and Stella X Yu. Bootstrapping objectness from videos by relaxed common fate and visual grouping. In Proceedings of the IEEE/CVF Conference Computer Vision and Pattern Recognition, pp. 14582-14591, 2023.
Bin Lin, Bin Zhu, Yang Ye, Munan Ning, Peng Jin, and Li Yuan. Video-llava: Learning united visual representation by alignment before projection. arXiv preprint arXiv:2311.10122 2023.
Shuming Liu, Chen Zhao, Tianqi Xu, and Bernard Ghanem. Bolt: Boost large vision-language model without training for long-form video understanding. In Proceedings of the Computer Vision and Pattern Recognition Conference, pp. 3318-3327, 2025.
Wen Liu, Weixin Luo, Dongze Lian, and Shenghua Gao. Future frame prediction for anomaly detection-a a new baseline. In Proceedings of the IEEE conference on computer vision and pattern recognition, pp. 6536-6545, 2018.
Yuanxin Liu, Shicheng Li, Yi Liu, Yuxiang Wang, Shuhuai Ren, Lei Li, Sishuo Chen, Xu Sun, and Lu Hou. Tempcompass: Do video Ilms really understand videos? In ACL (Findings), 2024.
Lang Nie, Chunyu Lin, Kang Liao, Yun Zhang, Shuaicheng Liu, Rui Ai, and Yao Zhao. Eliminating warping shakes for unsupervised online video stitching. In European Conference on Computer Vision, pp. 390-407, 2024.
Jongwoo Park, Kanchana Ranasinghe, Kumara Kahatapitiya, Wonjeong Ryoo, Donghyun Kim, and Michael S Ryoo. Too many frames, not all useful: Efficient strategies for long-form video qa. CoRR, 2024.
12

---

Published as a conference paper at ICLR 2026
Deepak Pathak, Pulkit Agrawal, Alexei A Efros, and Trevor Darrell. Curiosity- exploration by self-supervised prediction. In International conference on machine learning, pp. 7778-7787. PMLR. 2017.
Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, al. Learning transferable visual models from natural language supervision. In International conference on machine learning, pp- 8748-8763. PmLR, 2021.
Yuzhang Shang. Mu Cai, Bingxin Xu, Yong Jae Lee, and Yan Yan. Llava-prumerge: Adaptive token reduction for efficient large multimodal models. arXiv preprint arXiv:2403.15388 2024.
Leqi Shen, Tianxiang Hao, Tao He, Sicheng Zhao, Yifeng Zhang, Yongjun Bao, Guiguang Ding, et al. Tempme: Video temporal token merging for efficient text-video retrieval. In The Thirteenth International Conference Learning Representations, 2024a.
Xiaoqian Shen, Yunyang Xiong, Changsheng Zhao, Lemeng Wu, Jun Chen, Chenchen Zhu, Zechun Liu, Fanyi Xiao, Balakrishnan Varadarajan, Florian Bordes, et al. Longvu: Spatiotemporal adaptive compression for long video-language understanding. arXiv preprint arXiv:2410.1 17434, 2024b.
Yan Shu, Zheng Liu, Peitian Zhang, Minghao Qin, Junjie Zhou, Zhengyang Liang, Tiejun Huang, and Bo Zhao. Video-xl: Extra-long vision language model for hour-scale video understanding. In Proceedings of the Computer Vision and Pattern Recognition Conference, pp. 26160-26169, 2025.
Xi Tang, Jihao Qiu, Lingxi Xie, Yunjie Tian, Jianbin Jiao, and Qixiang Ye. Adaptive keyframe sam- pling for long video understanding. In Proceedings of the Computer Vision and Pattern Recogni- tion Conference, pp. 291 18-29128, 2025.
Keda Tao, Can Qin, Haoxuan You, Yang Sui, and Huan Wang. Dycoke: Dynamic compression of tokens for fast video large language models. Proceedings of the Computer Vision and Pattern Recognition Conference, pp. 18992-19001, 2025.
Peng Wang, Shuai Bai, Sinan Tan, Shijie Wang, Zhihao Fan, Jinze Bai, Keqin Chen, Xuejing Liu, Jialin Wang, Wenbin Ge, et Qwen2-vl: Enhancing vision-language model's 's perception of the world at any resolution. CoRR, 2024a.
Weiyun Wang, Zhangwei Gao, Lixin Gu, Hengjun Pu, Long Cui, Xingguang Wei, Zhaoyang Liu, Linglin Jing, Shenglong Ye, Jie Shao, et al. Internv13. 5: Advancing open-source multimodal models in versatility, reasoning, and efficiency. arXiv preprint vrXiv:2508.18266 2025.
Xijun Wang, Junbang Liang, Chun-Kai Wang, Kenan Deng, Yu Lou, Ming C Lin, and Shan Yang- Vila: Efficient video-language alignment for vidco question answering. In European Conference on Computer Vision, pp. 186-204. Springer, 2024b.
Siyuan Wei, Tianzuu Ye, Shen Zhang, Yao Tang. and Jiajun Liang. Joint token pruning and squeez- ing towards more aggressive compression of vision transformers. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pp. 2092-2101, 2023.
Zichen Wen, Yifeng Gao, Weijia Li, Conghui He, and Linfeng Zhang. Token pruning in multimodal large language models: Are we solving the right problem? In Findings of the Association for Computational Linguistics, 2025.
Haoning Wu, Dongxu Li, Bei Chen, and Junnan Li. Longvideobench: A benchmark for long-conters interleaved video-language understanding. Advances in Neural Information Processing Systems, 77:28828-288577 2024.
Sengiao Yang, Yukang Chen, Zhuotao Tian, Chengyao Wang, Jingyao Li, Bei Yu, and Jiaya Jia. Visionzip: Longer is better but not necessary in vision language models. In Proceedings of the Computer Vision and Pattern Recognition Conference, pp. 19792-19802, 2025.
13

---

Published as a conference paper at ICLR 2026
Xubing Ye, Yukang Gan, Yixiao Ge, Xiao-Ping Zhang, and Yansong Tang. Atp-llava: Adaptive token pruning for large vision language models. Proceedings of the Computer Vision and Pattern Recognition Conference, pp. 24972-24982, 2025.
Shoubin Yu, Jaemin Cho, Prateek Yadav, and Mohit Bansal. Self-chained image-language model for video localization and question answering. Advances in Neural Information Processing Systems, 36:76749-76777 2023.
Heng Zhang. Daqing Liu, Qi Zheng, and Bing Su. Modeling video as stochastic processes for fine- grained video representation learning. In Proceedings of the IEEE/CVF Conference on Computer Vision and Puttern Recognition, pp. 2225-2234, 2023.
Yuan Zhang, Chun-Kai Fan, Junpeng Ma, Wenzhao Zheng, Tao Huang, Kuan Cheng, Denis A Gudovskiy, Tomoyuki Okuno, Yohei Nakata, Kurt Keutzer, al. Sparsevim: Visual token sparsi- fication for efficient vision-language model inference. In Forty-second International Conference on Machine Learning, 2024.
Zhihang Zhong, Gurunandan Krishnan, Xiao Sun, Yu Qiao, Sizhuo Ma, and Jian Wang. Clearer frames, anytime: Resolving velocity ambiguity in video frame interpolation. In European Con- ference on Computer Vision, pp. 346-363, 2024.
Junjie Zhou, Yan Shu, Bo Zhao, Boya Wu, Zhengyang Liang, Shitao Xiao, Minghao Qin, Xi Yang, Yongping Xiong, Bo Zhang, al. Mlvu: Benchmarking multi-task long video understanding. In Proceedings of the Computer Vision and Pattern Recognition Conference, pp. 13691-13701, 2025.
14

---

Published as a conference paper at ICLR 2026
A APPENDIX
USE OF LARGE LANGUAGE MODELS
We used large language models (LLMs) solely to assist with grammar checking, wording sugges- tions, and polishing of the manuseript text. No LLMs were used for idea generation, experimental design, analysis, or results.
A.I IMPLEMENTATION DETAILS
We implement SURGE as lightweight masking module between the vision encoder and the lan- guage model, requiring no retraining or modification of backbone weights. For InternVL-335. (8B), we use the official HuggingFace release and insert SURGE after the vision encoder outputs, before token projection to the LLM. For Video-LLaVA-Qwen (7B), build on the Hugging- Face Transformers implementation of LLaVA- Video, applying SURGE to patch embeddings from the BigLIPSS040N. vision encoder before they are passed into the multimodal backbone. For Qwen2.5-VL (7B), we extend the HuggingFace Transformers classes for the vision transformer and multimodal generation incorporate SURGE masking in the visual embedding pipeline. For query-aware focusing (SURGE"), ), we add a CLIP relevance scorer, applied only to candidate peak frames.
For baseline comparisons, we adopt V (Chen et al. 2024a) (attention-based pruning) and AKS (Tang al., 2025) (adaptive keyframe selection), using their official open-source implemen- tations within the HuggingFacel Facarrassoooorerrs ecosystem. FastV is integrated on Video-LLaVA Qwen via its attention pruning hooks, while AKS performs frame selection upstream of the vision encoder and is applicable to all models.
All benchmarks are evaluated using VLMEvalKit (Duan et al. 2024). standardized open-souree source toolkit for multimodal evaluation. We adopt its official protocols, prompt templates, and de- coding settings for Video-MME, MLVU, MMBench- Video, TempCompass and Long VideoBench, ensuring consistent and comparable results. All models, baselines, and toolkits are open-source and we will release our SURGE/SURGE* modules and wrapper code for each model upon acceptance. Results are averaged over 5 random seeds ((41, 79, 138, 534, 963}) for robustness.
A.2 BASELINE METHODS
FastV. FastV (Chen et al. 2024a) targets inefficiencies in VLM attention by observing that image and video tokens receive vanishingly small attention in deeper layers of models like LLaVA and VL. To reduce redundant computation, FastV adaptively prunes visual tokens after a chosen layer, guided by their average attention scores. This plug-and -andplay pruning avoids both self-attention and FFN costs discarded tokens, gaining large FLOP savings with little performance drop. Simi- lar attention-guided pruning strategies exist (Zhang et al.) 2024 Ye et al. 2025 Huang et al. 2024). but they rely internal attention maps that may not be exposed in deployed systems. In contrast, SURGE is training-free and attention attention-agnosticc: it measures novelty directly via token prediction error, enabling pruning without accessing internal gradients hidden states. This makes SURGE more robust across backbones and applicable even when attention maps are inaccessible.
Adaptive Keyframe Sampling (AKS). AKS (Tang et al. 2025) reduces temporal redundancy by selecting subset of video frames prior to encoding. It balances relevance (frame- query similarity) and coverage (temporal diversity) through a recursive gudge--ndd-splis strategy, yielding high-quality keyframes under fixed context budgets. While highly effective for long-video QA, AKS operates only at the frame level, meaning all patch tokens from selected frames are preserved. SURGE complementary: it operates within frames, pruning predictable patches after encoding. Combin- ing AKS and SURGE leverages both temporal and token-level sparsity, balancing efficiency and accuracy across compute budgets.
Random Pruning. Random pruning removes a fixed fraction of visual tokens uniformly at random. Despite its simplicity, it is often surprisingly competitive baseline: Wen et al. (Wen et al., 2025) show that random selection or simple pooling can match or even outperform attentioon-base pruning methods such as V and Sparse VLM several benchmarks, due to position bias and instability
15

---

Published as a conference paper at ICLR 2026
in attention based importance scores. More broadly, Transformers are known to be robust random token dropping at moo-erat ratios, as demonstrated by Kim et al. (Kim et al. 2022) which highlights the redundancy and sparsity inherent in visual tokens. However, because random pruning ignores content, it risks discarding critical information as pruning increases. In contrast, SURGE explicitly measures temporal predictability: it prunes only redundant tokens while retaining novel ones, achieving efficiency without sacrificing essential content.
A.3 BENCHMARKS AND MODELS: DETAILS AND JUSTIFICATION
A.3.1 BENCHMARKS
We evaluate on five public video-language benchmarks that together span breadth (general QA), depth (temporal reasoning), and length (long-context). This diversity avoids redundancy and ensures SURGE is tested under complementary stressors.
(1) Video-MME (Fu et al., 2025): covers short and long clips with multimodal inputs and diverse QA types, and is widely used as general evaluation suite for VLMs. We include it both in main results and the basis for FLOP//atency profiling (Fig. 5). since efficiency patterns are primarily model-driven and do not require redundant measurements across multiple datasets.
(2) MLVU (Zoou et al., 2025): a long-video, multi-task benchmark reporting M-Avg/G-Avg across heterogeneous tasks such as needle QA, grounding, and summarization. Because it stresses both extended context and task variety, we use it broadly: in main results, in Oblation/hyperpprammtee sweeps, and for long-context scaling (Fig. 4 where its long videos make it the natural choice to probe robustness under extreme input lengths.
(3) MMBench-Vidde (MMB-V) (Fang et al. (024): emphasizes multi-step and compositional rea- soning on multi-shot videos, complementing MLVU's multi-task design with a focus on logical con- sistency and stepwise inference. We use it in main results and ablations with Video-LLa VA-QWen,, since its free-form QA format is particularly sensitive to token loss.
(4) TempCompass (Liu et al.) 2024]: isolates temporal reasoning (order, speed, duration) by con- structing nearly identical clips that differ only in motion attributes. We include it to verify that pruning does not compromise event chronology. weakness of many sparsity methods. It appears in main results and ablations as a targeted "stress test" for SURGE.
(5) LongViideBBenh (LVB) (Wu al.) 2024): targets very long vidcos with referring/retrieval style questions, stressing extreme long-range reasoning. We use it in main results to demonstrate SURGE's scalability, and in hyperparameter studies with Qwen2.5-VL. since only models with extended context can operate in this regime.
Benchmark coverage. All five benchmarks are included in the main results with all three models. Ablations and hyperparameters focus on Mideo-LLaVA-Qwer with MLVU, TempCompass, and MMB-V, V, these sets are most diagnostic for pruning behavior. Efficiency analysis is done Video-MME with Qwen2.5-VL. and long-context scaling on MLVU with Qwen2.5-VI, since these the only realistic pairings for those analyses.
A.3.2 MODELS
We select three open-source VLMs that represent distinct integration styles and compute regimes, ensuring that SURGE evaluated across both high-capacity and practical backbones.
(1) Intern VL-3.5-VL (8B) (Chen et al. 2024b); a flagship open model with strong general VLM performance and adaptive vision encoding. We include it in the main results to show SURGE re- mains effective high-capacity backbone at the frontier of open performance.
(2) Video-LLa VALLLVA-QQE VA-Qwen (7B) (Li et al. 2024a): a widely used LLaVA Video variant coupling a Qwen LLM with SigLIP encoder, capped at 64 frames. Because of its popularity and stable infer- ence pipeline, we use it extensively for ablations and hyperparameter sweeps, making comparisons with prior sparsity methods more direct.
(3) Qwen2.5-V VL (7B) (Bai et al. 2025) Wang et al. 2024a): the latest long-context VL model with efficient vision encoding and ~ 131k token window. It is the only backbone among the three that
16

---

Published as a conference paper at ICLR 2026
extended contexts beyond 230 frames standard hardware, which is why most efficiency breakdowns, scaling studies, and token token-accuracy trade-offs are conducted on
Model coverage. Intern VL-3.5-VL represents flagship, high-capacity baseline; Video-LLaVA- Qwen covers a video-specialized, widely adopted LLaVA-style architecture; and Qwen2.5-VLL rep- resents the state of the art in long-context efficiency. This spread justifies why not all models appear in every experiment, while ensuring SURGE is validated across distinct backbones and deployment scenarios.
A.4 ADDITIONAL VISUALIZATIONS OF SURGE MASKING
We provide qualitative examples of SURGE across different video domains. Figures 6 show raw and smoothed surprise curves alongside a subset of the processed frames for clarity. Red boxes indicate peak events, and shaded patches denote tokens retained by SURGE. In our implementation, the first frame always treated as full-surprise and fully preserved, since no temporal history exists for prediction. Results demonstrate that SURGE highlights novel content across natural, animated, and lecture-style videos.
Start from t>10
Surprise Fraction
0.4
0.3
0.2
0.1
Raw surprise
Smoothed surprise
0.0
50
100
150
200
Temporal Unit (Chunk)
Figure 6: SURGE visualization on an animal documentary clip. SURGE emphasizes novel motion events (e.g., cheetah appearance, tiger interaction), with smoothed surprise peaks aligning with key scene changes.
A.5 ADDITIONAL QUALITATIVE EXAMPLES
Across these examples, SURGE/SURGE" perform par with the base model, successfully retaining key evidence tokens. For clarity, we display only a subset of processed frames; in SURGE*, red boxes mark peak-event selections, and we show the first frame of each event.
We further illustrate SURGE and SURGE" on video-QA tasks (Figs, 1
17

---

Published as a conference paper at ICLR 2026
Start from P-10
Surprise Fraction
0.3
N
WMM
0.2
0.1
Raw surprise
Smoothed surprise
0.0
20
40
60
80
100
120
140
160
Temporal Unit (Chunk)
Figure 7: SURGE visualization on cartoon (SpongeBob) clip. Surprise spikes capture exaggerated expressions and character interactions, while redundant frames are pruned.
York
York
0.5
Start from t
Surprise Fraction
0.4
0.3
0.2
Raw surprise
Smoothed surprise
0.0
20
40
60
80
100
120
140
160
180
Temporal Unit (Chunk)
Figure 8: SURGE visualization on TED talk clip. Peaks correspond to speaker appearance changes and audience reactions, whereas static slides yield low surprise.
18

---

Published as a conference paper at ICLR 2026
Q: As depicted the video, which tool not necessary to make a rubber band car?
ENERGY
2
Pind
He Leoie
A. Straw
B. Pencil
C. Scissors
D. Hammer
Base Model: B. Pencil SURGE: B. Pencil
SURGE": B. Pencti
Figure 9: Both SURGE and SURGE* retain the key content and match the base model's correct prediction.
19

---

Published as a conference paper at ICLR 2026
Q: When demonstrating the Germany modern Christmas tree is initially decorated with apples, candles and berries, which kind of the decoration has the largest number?
NEW AT 6
Paradise treeş
13
A. Apples
B. Candles
C. Berries
D. The three kinds are of the same number
Base Model:
Barries
SURGE: C. Barries
SURGE*: C. Barries
This task requires counting the dominant decoration type. SURGE and SURGE* preserve key frames showing dense berries, enabling them match the base model's s correct answer despite heavy pruning.
Figure 10:
Q: Which of following reasons motivated archaeologists excavale the tomb?
3
lott to the MO// L'MNOETOO viire DM Socil
A. Because it's from Ming Dynasty and of specific archaeological significance. Because railway line will be built nearby.
C. Because there were treasures inside the tomb.
D.. Highway realignment.
X
Base Model:
SURGE: C SURGE*:A 4
Figure 11: The correct reason ("highway realignment") is non-visual and not inferable from frames alone. All models, including the base, fail here.
20


Reviews:
[{"review_id": "UfWpSlehup", "reviewer": "Reviewer_ZWDa", "summary": "In this paper, a training-free and model-agnostic token reduction algorithm, called SURGE, is proposed. The proposed algorithm estimates the surprising scores based on the prediction error of each token from its recent history. Then, it remains the tokens with high surprising score, while prunes the tokens with low surprise. Additionally, it utilizes the CLIP similarities for query-focused applications. Experimental results on video understanding datasets show that the proposed algorithm achieves comparable or even better performance despite a lower computational budget.", "questions": "My main concern is that the paper lacks sufficient performance comparison with existing token reduction methods. If this issue is properly addressed in the rebuttal, I would be willing to raise my score. Please also refer to the Weaknesses section for my other concerns.", "limitations": "", "rating": 4, "confidence": 4, "soundness": 2, "presentation": 3, "contribution": 2, "strengths": "- This paper is well-written and easy to follow.\n- The paper includes sufficient implementation details, which enhances its reproducibility.\n- The proposed method is motivated by a solid rationale and is technically well justified.", "weaknesses": "- From Table 1, the comparison appears to be limited to a single baseline—fastV + Video-LLaVA-Qwen at one fixed budget. If comparisons with additional methods are difficult to include, it would still be helpful to report results across a broader range of budgets to better illustrate robustness and trade-offs.\n- L354-357 mentions that AKS and the proposed SURGE are complementary to each other. However, in Table 1, when comparing the results at similar token budgets, some results slightly improve while others decrease, and the others remain almost unchanged, so the overall performance appears to be roughly comparable. \n- It would be beneficial if the experimental results on benchmark datasets, such as Next‑QA or EgoSchema, are included.\n- This paper appears to lack a comparison with recent training-free token reduction methods. Including such baselines would help clarify the relative advantages of the proposed approach. For example, there are the following recent works:\n\n    [1] Multi-Granular Spatio-Temporal Token Merging for Training-Free Acceleration of Video LLMs, ICCV 2025\n\n    [2] Video, How Do Your Tokens Merge?, CVPR Workshop 2025\n\n    [3] Don’t Look Twice: Faster Video Transformers with Run-Length Tokenization, NeurIPS 2024\n\n- Typo\n  - L49 ”surprise” : the quotation marks appear to be incorrectly oriented.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "R4Id6fVh62", "reviewer": "Reviewer_xZY7", "summary": "language models named SURGE. Specifically, SURGE includes two core modules: Surprise Estimation and Spatio-temporal Masking. Surprise Estimation leverages a constant-velocity predictor in token space with global affine drift removal and variance normalization to measure temporal predictability, where high-surprise tokens are identified by large prediction errors indicating novel content. Spatio-temporal Masking applies global percentile thresholding to retain the most surprising tokens and aggregates scores over time to produce a surprise curve for key event segmentation.\n\nThe experimental evaluation in this paper assessed the performance of the proposed SURGE on five public benchmarks (Video-MME, MLVU, MMBench-Video, TempCompass, and LongVideoBench), comparing it with various approaches.", "questions": "- In SURGE, it seems the decision to retain a token is based solely on its individual surprise score. If part of an object’s tokens are removed due to low surprise, could the remaining tokens lose semantic completeness and break the spatial continuity of the object representation? Why is there only a retain-or-delete operation at the token level, without a merging/fusion mechanism to preserve semantic integrity?", "limitations": "", "rating": 4, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "- The definition of token surprise is considered novel.\n- The method’s performance has been validated across multiple base models, enhancing the credibility of the approach.", "weaknesses": "- SURGE models temporal dynamics using constant-velocity prediction plus affine drift fitting. Compared to optical flow or more advanced temporal prediction methods, this simple predictor may be insufficient for complex motions or non-linear changes (e.g., rapid acceleration/deceleration, severe local deformations). Could there be missed detections of key tokens in high-complexity videos?\n- SURGE introduces several hyperparameters (γ, ∆, K, ρ), but the paper provides limited ablation discussion on their effects. Although Sec. 5.3 claims stability within reasonable ranges; in different tasks or video distributions, these parameters may require retuning.\n- SURGE⋆ adds an additional CLIP scoring step. Does this introduce extra computational overhead?\n- The FLOPs and latency analysis lack comparison with other methods.\n- The paper lacks comparison against other approaches that reduce tokens before the LLM stage, such as VisionZip [1] or LLaVA-Scissor [2].\n\n[1] VisionZip: Longer is Better but Not Necessary in Vision Language Models\n\n[2] LLaVA-Scissor: Token Compression with Semantic Connected Components for Video LLMs", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "KKvCwrPsMt", "reviewer": "Reviewer_XSco", "summary": "This paper introduces SURGE, a training-free, and model-agnostic method designed to enhance the long video understanding by dynamically reducing redundant input tokens. It uses a simple feature predictor to estimate the next frame's visual features based on the current one. If the surprise score is low, the corresponding tokens are pruned, significantly reducing the VLM's computational load. Then it refines with CLIP-based query relevance. SURGE reduces tokens by up to 7× and prefill cost by 86–98%, while keeping accuracy within ±1 point of full-token baselines.", "questions": "+ Previous studies have shown that sampling more frames generally leads to better performance. Ideally, if redundant tokens are removed while key frames are preserved, the model should achieve improved results under the same context budget. Have the authors tested their method under the same context budget as the base model to verify whether token redundancy reduction effectively enhances context utilization and leads to better performance?\n+ As shown in Figure 2, why is the surprise score computed on the visual features after the linear projection? Since the projection aligns the features with the LLM’s input space rather than the pure visual space, wouldn’t this affect the intended focus? Have the authors conducted any experiments using features extracted directly from the ViT backbone?\n+ Additionally, is the frame–text similarity based on CLIP features also calculated after the linear projection?\n+ How does the proposed method perform compared with DyCoKe, VisionZip, and BOLT?\n+ The proposed surprise scoring mechanism is primarily designed to capture pixel-level changes. However, CLIP features are trained using a vision–language contrastive objective in a semantic space, which may not align well with this goal. Have the authors considered using vision-centric models, such as DINO, to compute the surprise scores instead?", "limitations": "", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 2, "strengths": "+ The method is training-free and thus can be applied to any LVLMs.\n+ It reduces tokens by up to 7× and prefill cost by 86–98% while preserve the accuracy within ±1 point.\n+ The authors provide comprehensive results based on several LVLMs across multiple video understanding benchmarks.", "weaknesses": "+ The paper lacks a discussion of related work on video token compression and frame selection methods [1–5].\n+ Although the proposed method improves efficiency, it does not yield a clear accuracy gain. Ideally, removing redundant tokens should free up contextual capacity to accommodate more informative content, thereby enhancing the model’s ability to answer queries.\n+ The proposed surprise scoring mechanism is intended to capture pixel-level changes, yet it relies on CLIP features trained in semantic space.\n+ Missing comparison with training-free token pruning methods [1,2,5].\n\n[1] DyCoke: Dynamic Compression of Tokens for Fast Video Large Language Models, CVPR 2025.\n[2] VisionZip: Longer is Better but Not Necessary in Vision Language Models, CVPR 2025.\n[3] Chat-UniVi: Unified Visual Representation Empowers Large Language Models with Image and Video Understanding, CVPR 2024.\n[4] LongVU: Spatiotemporal Adaptive Compression for Long Video-Language Understanding, ICML 2025.\n[5] BOLT: Boost Large Vision-Language Model Without Training for Long-form Video Understanding, CVPR 2025.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "WYBeFGsTsV", "reviewer": "Reviewer_ky99", "summary": "This paper introduces SURGE, a training-free and model-agnostic method for improving the efficiency of VLMs. The authors propose defining \"surprise\" as the prediction error of a visual token based on its recent history. A lightweight constant-velocity temporal predictor, enhanced with affine drift correction and variance normalization, is used to calculate a surprise score for each token at each frame.\n\nBased on these scores, SURGE employs a two-stage process to create a sparse spatio-temporal mask, including token pruning and event segmentation. For query-based tasks, the method can be extended to SURGE⋆, which uses CLIP to rank these key events by their relevance to the text query, further focusing computation. The authors conduct extensive experiments on three state-of-the-art VLMs (InternVL-3.5-VL, Video-LLaVA-Qwen, Qwen2.5-VL) across five diverse video understanding benchmarks. The results show that SURGE can reduce token counts by up to 7x and prefill costs by 86–98%, while maintaining accuracy within ±1 percentage point of the full-token baseline, demonstrating its effectiveness and practicality.", "questions": "Please see weakness.", "limitations": "", "rating": 4, "confidence": 4, "soundness": 3, "presentation": 2, "contribution": 2, "strengths": "- Soundness of the core idea: The central contribution is the formulation of \"surprise\" as a criterion for token retention. Using a simple, physics-inspired constant-velocity predictor in the token embedding space to measure novelty is both original in this context and conceptually elegant. It provides a principled way to distinguish new information from redundant content, moving beyond heuristics like attention weights which can be noisy or biased.\n\n- Practicality: SURGE is training-free and backbone-agnostic. This is a major strength, as it can be plugged into virtually any ViT-based VLM without costly retraining or architectural modifications. This makes the method highly practical and lowers the barrier to adoption for improving inference efficiency.\n\n- Impressive empirical performance: The paper demonstrates substantial efficiency gains while maintaining accuracy.", "weaknesses": "- The submission conflicts with the format requirement: \"The table number and title always appear before the table.\" I am not sure whether it should be a reason for a desk rejection.\n\n- The paper claims \"negligible overhead\" for the SURGE module, but this is not sufficiently quantified.\n\n- The method introduces too many new hyperparameters but does not justify how these hyperparameters are chosen and how they influence the performance.\n\n- The constant-velocity model with affine drift correction is a core component. While effective for many scenarios, it has inherent limitations. It may struggle with complex, non-linear motions such as rapid rotations, non-rigid deformations, or fast zooms, potentially leading to inaccurate surprise scores. Besides, the temporal predictor uses only the displacement from the immediate previous step. This limited temporal window makes the model susceptible to noise and unable to capture more complex, non-linear dynamics such as acceleration or periodic motions.\n\n- There have been many token-level compression methods proposed in the past year, such as ToMe in the related work, and [1] [2]. Please see the survey [3] for more details. The paper should compare with these methods.\n\n- The proposed method operates on the patch embeddings after they have been processed by the vision encoder. While it effectively reduces the computational load on the subsequent large language model, it does not address the significant upstream cost of encoding every single frame of a long video. For extremely long videos, the vision encoder itself can become a major bottleneck. It would be valuable to explore whether the core idea of \"surprise\" could be adapted to operate at a lower level—perhaps on raw pixels or early-layer features—to enable dynamic frame dropping or resolution adjustment before the expensive full ViT forward pass.\n\n- The method relies on a \"global percentile\" threshold and peak-based event segmentation, both of which seem to require processing the entire video clip offline to establish a global context of surprise scores. This \"two-pass\" nature makes its direct application to true streaming or real-time video analysis challenging, where decisions must be made causally and on-the-fly with only past and present information.\n\n[1] ViLAMP: Scaling Video-Language Models to 10K Frames via Hierarchical Differential Distillation\n[2] Divprune: Diversitybased visual token pruning for large multimodal models\n[3] When Tokens Talk Too Much: A Survey of Multimodal Long-Context Token Compression across Images, Videos, and Audios", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (Poster)