Scaling Video-Language Models to 10K Frames via Hierarchical Differential Distillation
Authors: Chuanqi Cheng
Jian Guan
Wei Wu
Rui Yan
Venue: ICML 2025 poster (2025)
Keywords: Vision Language Models
Video Understanding
Primary area: deep_learning->foundation_models
TL;DR: We enhance the model's performance in understanding long videos through hierarchical compression.

Abstract: Long-form video processing fundamentally challenges vision-language models (VLMs) due to the high computational costs of handling extended temporal sequences. Existing token pruning and feature merging methods often sacrifice critical temporal dependencies or dilute semantic information. We introduce differential distillation, a principled approach that systematically preserves task-relevant information while suppressing redundancy. Based on this principle, we develop ViLAMP, a hierarchical video-language model that processes hour-long videos at "mixed precision" through two key mechanisms: (1) differential keyframe selection that maximizes query relevance while maintaining temporal distinctiveness at the frame level and (2) differential feature merging that preserves query-salient features in non-keyframes at the patch level. Hence, ViLAMP retains full information in keyframes while reducing non-keyframes to their most salient features, resembling mixed-precision training. Extensive experiments demonstrate ViLAMP's superior performance across five video understanding benchmarks, particularly on long-form content. Notably, ViLAMP can process ultra-long videos (up to 10K frames) on a single NVIDIA A100 GPU, achieving substantial computational efficiency while maintaining state-of-the-art performance. 
Code and model are available at https://github.com/steven-ccq/ViLAMP.

Scaling Video-Language Models to 10K Frames via Hierarchical Differential Distillation
Chuanqi Cheng * 12
Jian Guan x2
Wei Wu
Rui Yan 113
Abstract
00M
80000
GPU Memory (MB)
Long-form video processing fundamentally chal- lenges vision-language models (VLMs) due to the high computational costs of handling extended temporal sequences. Existing token pruning and feature merging methods often sacrifice critical temporal dependencies dilute semantic infor- mation. We introduce differential distillation, a principled approach that systematically preserves task-relevant information while suppressing re- dundancy. Based on this principle, we develop VILAMP. a hierarchical video-language model that processes hour-long videos at "mixed preci- sion" through two key mechanisms: (1) differ- ential keyframe selection that maximizes query relevance while maintaining temporal distinctive- ness at the frame level and (2) differential feature merging that preserves query-salient features in non-keyframes the patch level. Hence, VIL- AMP retains full information in keyframes while reducing non- non-key keyframes to their most salient fea- tures, resembling mixed-pret sexxed-preciioo training (Mi- cikevicius et al., 2018). Extensive experiments demonstrate VILAMP's superior performance across five video understanding benchmarks, par- ticularly on long-form content. Notably, VIL- AMP process ultra-long videos (up 10K frames) on single NVIDIA A100 GPU, achiev- ing substantial computational efficiency while maintaining state-of the-art performance. Code and model are available at https: / com/ steven coq/ cccvVViAC LVMMP..
60000
40000
IK Number of Framess
I0C
Figure 1. Comparison of the GPU memory consumption with vary- ing input lengths (from O to 10K frames) on our VideoNIAH benchmark (c.f. $5.3). OOM indicates out-of-memory errors. All models are evaluated using one NVIDIA A100 GPU,
1. Introduction
The rapid advancement of vision-language models (VLMs) continues to drive remarkable progress in processing both visual and textual content (OpenAI, 2024; Liu et al., 2024a; Chen al., 2024a; Zhang et al., 2024b). However, as mod- els expand their capabilities from processing single images (typically requiring 729 visual tokens (Zhai et al., 2023)) to handling long videos, they inevitably encounter a funda- mental challenge: the visual token sequence from video can easily exceed the context length of L.MMs, imposing prohibitive costs in both computational resources and in- ference latency. For instance, a one-minute video clip at 24 frames per second (FPS) could generate over million visual tokens (24 60 729), far surpassing the context capacity of most mainstream LLMs, which typically ranges from 4K to 128K tokens (Dubey et al., 2024; Yang et al., 2024a). The challenge becomes even more critical as video lengths extend to an hour or beyond, despite the prevalence of such lengthy videos real-world applications, including long-form video analysis (Lin al., 2023) and continuous robot learning (Wu al., 2023; Baker et al., 2022).
'Equal contribution 'Gaoling School of Artificial Intel- ligence. Renmin University of China 'Ant Group "School of Artificial Intelligence, Wuhan University. Correspon- dence to: Wei Wu <wuwci19850318 101550011@mmal..omm. gmmil Rui Yan <ruiyan@rue.ddu.om
Prior works have explored two main approaches to handle long videos efficiently: (1) token pruning through uniform or content-aware sampling strategies (Ye et al., 2023; Shen et al., 2024), and (2) feature merging via heuristic or learn- able mechanisms (Lin et al., 2024; Bai al., 2023). How-
Proceedings of the 42nd International Conference Machine Learning. Vancouver, Canada. PMLR 267, 2025. Copyright 2025 by the author(s).

---

Scaling Vidco- Video-Language Models to 10K Frames via Hierarchical Differential Distillation
Depoth Noolle
20-
20
06
80
84
82-
100
IX
I0K X
3k
<<
2K.
IK
%
10k
10C
Length of Haystack (Tranes))
Length of Hlystack (frames)
Length of Huystack (frumes)
(a) LLaMA-VID
(h) VideoChat-Flasse
(c) VIL.AMP (Ours)
Figure 2. Comparison of ultra-long video understanding capabilities on our VideoNIAH benchmark (c.f. $5.3). Each heatmap shows model performance across different haystack lengths and needle depths. The depth refers to the relative position of the needle video within the haystack with 0% the front and 100% at the end.
ever, these methods face fundamental limitations: token pruning risks losing critical temporal dependencies, while feature merging often leads to information dilution (Liu et al., 2024c). The challenge of balancing computational efficiency and semantic preservation thus remains unsolved.
which identifies keyframes by maximizing query relevance while ensuring temporal distinctiveness. (2) At the patch level, introduce the Differential Feature Merging (DFM) strategy for non-keyframe compression, which employs differentiable learning mechanism to preserve query- features while suppressing redundant visual information.
In this work, we explore scaling video-language models to handle significantly longer videos by systematically dis- tilling salient signals into minimal visual tokens without sacrificing information integrity. We begin by addressing fundamental question: how to identify truly salient signals from long videos? While the saliency of visual signals is traditionally measured by query relevance (Li et al., 2024d), videos often contain substantial temporal correlations and repeated content (Shen et al., 2024). This suggests that merely considering query relevance might lead to selecting redundant signals. To validate this hypothesis, we conduct comprehensive analysis at both frame and patch levels (c.f., $3). Our analysis reveals two key findings: (1) At the frame level, 990% of query-induced attention weights concentrate on merely 5% of frames, with these attention-heay frames showing significant internal similarities. (2) At the patch level, ~00 of patches from less-attended frames command 80% of model attention while exhibiting high visual similar- ity to those in attention-heavy frames, indicating redundant attention to similar visual patterns. These findings inspire our disferential distillation principle for efficient video mod- eling: distilling salient information that exhibits maximal correlation with the query intent while maintaining minimal redundancy with respect to the vidco's temporal context.
We conduct extensive evaluations of VILAMP across five video understanding benchmarks, spanning from minutes to nearly hours. VILAMP outperforms state-of-the- baselines on five benchmarks. Notably, on the "Long" sub- set of Video-MME, which features average duration of 2,386 seconds, VILAMP achieves substantial 3.0% and 4.8% absolute accuracy improvement over the previous top- performing model in non-subtitled and subtitled settings, respectively. To rigorously assess VLMs' capability in ultra- long video understanding, we introduce VideoNIAH. a chal- lenging benchmark inspired by the Reedle-in-a-Haystar (NIAH) evaluation paradigm from LLM research (Fu et al., 2024b; Kuratov et al., 2024). This benchmark requires mod- els to both locate and comprehend specific needle videos embedded within extensive video sequences (haystacks) while answering targeted queries about these needles. As demonstrated in Figure and Figure 2, when scaling ex- isting models to process 10K frames (approximately 2.7 hours at IFPS), they either fail entirely due to memory con- straints or suffer severe performance drops compared to the 2K-frame counterpart (e.g., 224500% accuracy degradation for VideoChat- Flash (Li et al., 2024c)). In contrast, VIL- AMP not only successfully processes videos containing up to 10K frames on a single NVIDIA A100 GPU, but also maintains remarkably stable performance with an 12.82% absolute improvement over VideoChat-Flaan Flash'. These re- sults convincingly demonstrate that VILAMP establishes promising direction for both effective and efficient long- form video understanding.
Building on the principle, we introduce VILAMP, an effi cient Lideo-LAnguage model specifically designed to pro- cess long videos using "Mixed Precision" - allocating computational resources according to information saliency. VILAMP implements this mixed-precision strategy through- out a hierarchical compression framework that operates at two levels: (I) At the frame level, VILAMP maintains full visual token representations for salient frames (i.e., keyframes) while applying aggressive compression in- termediate frames (i.c., non-keeframes), reducing each to single token. Frame saliency is determined through our novel Differential Keyframe Selection (DKS) mechanism,
In summary, our primary contributions are threefold:
'Ulike NIAH tasks in L.M research where answers can he directly extracted from the needle text. VideoNIAH requires com- prehensive video understanding capabilities interpret visual- temporal events, making inherently more challenging to achieve the near-perfect accuracy (c.g., 99%) observed in LLM evaluations.
2

---

Scaling Video-Language Models to 10K Frames via Hierarchical Differential Distillation
I. We introduce the differential distillation principle that systematically identifies and preserves task-relevant infor- mation while actively suppressing redundancy, establishing principled approach for efficient long video understanding.
frame sampling, i.e., selecting fixed-interval frames. Re- cent works propose more sophisticated content-aware prun- ing approaches: Goldfish (Ataallah et al., 2024) employs database- caaabase-inspredd retrieval mechanisms, MovieChat (Song et al., 2024) implements dynamic memory management, and LongVU (Shen et al., 2024) introduces content-aware frame grouping and similarity- patch selection. However, such pruning risks significant information loss.
II. We propose VILAMP, a hierarchical architecture that operationalizes differential distillation through keyframe selection and non-keyframe compression. reducing non- keyframes to their most salient features.
(3) Feature Merging. Recent research has explored merg- ing video features at various representational levels for video compression, through either heuristic methods (c.g., downsampling (Lin et al., 2024), pooling (Xu et al., 2024; Zhang et al., 2024e: Li et al., 2025), similarity guiidd merg- ing (Bolya et al., 2023; Jin et al., 2024; Li al., 2024c)) or learnable approaches (c.g.. Q-Former (Li et al., 2023; Bai et al., 2023; Zhang et al., 2025), Perceiver Resampler (Yao et al., 2024)). However, these strategies often struggle to maintain semantic fidelity (Liu et al., 2024e).
III. Experiments show that VILAMP outperforms similar- sized VLMS on five video understanding benchmarks while efficiently scaling to 10K frames on a single GPU, opening new opportunities for processing ultra-long videos.
2. Related Work
2.1. Vision-Language Models
Advances in large language models (LLMs) (Dubey al., 2024; Yang et al., 2024a) has catalyzed the development of VLMs that can process visual and textual inputs, broadly classified into two categories: (I) models jointly trained on visual and textual modalities from scratch, such as Flamingo (Alayrac et al., 2022) and KOSMOS (Peng et al., 2023a;b); and (2) models leveraging pre-trained vision en- coders like CLIP (Radford et al., 2021b) SigLIP (Zhai et al., 2023), connected to LLMs through projection mod- ules. This design, pioneered by LLaVA (Liu et al., 2024a) with a simple two-layer MLP connector, has been widely adopted in Qwen-VL VL (Bai et al., 2023), MiniCPM (Yao et al., 2024), and InternVL. (Chen et al., 2024c).
In contrast, VILAMP preserves salient information through differential distillation, achieving both computational effi- ciency and information preservation.
3. Preliminary Studies
Video understanding tasks process a sequence of frames V = (f,,, /2, ... JNx)² to answer a query Q. Each frame fn is divided into M patches: fn = (p), P...... : .P ) each patch p is encoded into a unique visual token en through a vision encoder. Consequently, a video sequence generates MN visual tokens in total. This section analyzes information distribution patterns at both frame level ($3.1) and patch level ($3.2), leading to our differential distillation principle ($3.3).
) where
While recent VLMs can handle multiple images (Wang et al., 2024a; Li et al., 2024b) and videos (Zhang et al., 2024c; Anonymous, 2024; Zhang et al., 2024e), they face significant limitations: multi-image models ignore temporal dependencies, while video models struggle with computa- tional costs for long videos. This highlights the need for more efficient approaches to video understanding.
3.1. Frame-Level Redundancy Analysis
We analyze attention patterns of several representative VLMs, including LLaVA-One Vision (Li et al.. 2024b), LLLVA-Video (Zhang et al., 2024e), Qwen2-VL (Wang et al., 2024a) and Long VA (Zhang et al., 2024c), on Video- MME (Fu al., 2024a) by sampling N=128 frames per video. For each frame Sn, we compute its receiving attention weight a, by aggregating cross-attention weights between -M
2.2. Long-form Video Understanding
Current approaches to addressing the computational chal- lenges of long video understanding fall into three categories:
(1) Context Window Extension. This line of research fo- cuses on enhancing models' capacity for extended tempo- ral contexts. Liu et al. (2024b) gradually increases input length during training, while VA (Zhang et al., 2024a) transfers long-context capabilities from text to visual under- standing. However, these approaches still face efficiency limitations with extensive videos.
query Q and visual tokens {!7 JMM Jm an >m== a E [0, 1]. The token-level attention weight a. E [0, 1] is ob- tained by averaging across all query tokens, all layers, and all attention heads.
Figure 3 reveals two key patterns. First, about 90% of at- tention mass concentrates on less than 5% of frames, as evi- denced by the steep rise in the cumulative attention curves
(2) Token Pruning. Various pruning strategies have emerged to reduce computational burden. Early models (Lin et al., 2023; Ye et al., 2024; Cheng et al., 2024) used uniform
POur study focuses visual information, excluding subtitles or audio tracks.
3

---

Scaling Vidco-Langee Video-Language Models to 10K Frames via Hicrarchical Differential Distillation
Inten Inten-Patee Simllariy (Av
Comulabee Alteniiu Weight
by
ILLVA--One Vision
(b) LLaVA-Video
Percendic Patches Ranked Attention
Figure 4. Patch-level attention distribution of LLaVA-OneVision and cross-frame patch similarity. s-axis: percentile of non- keyframe patches ranked by attention weights from the query (%). Red curve: cumulative attention (normalized across all non- keyframe patches). Bars: similarity to nearest keyframe patches. Dashed line: random inter-patch similarity baseline (0.61).
(c) Qwen2-VL
(d) LongVA
Figure 3. Frame-level attention distribution and frame similarities. --axis: percentile of frames ranked by attention weights from the query (%). Red curves: cumulative attention. Bars: average simi- larity with neighboring three frames of similar attention. Dashed lines: random frame pair similarity baseline (0.54~0.61).
show strong visual similarity their keyframe counterparts, with just 50% of patches contributing 80% of total atten- tion. This correlation between attention weights and visual similarity indicates that substantial computation is spent processing redundant visual patterns across frames.
(red lines). This pronounced attention skew suggests query- relevant information clusters around specific keyframes. Second, and more intriguingly, these attention-heavy frames show high visual similarity (cosine similarity:>0.8) within themselves, substantially exceeding the random baseline similarity (dashed lines). This phenomenon indicates that frames receiving high attention weights contain signifi- cant visual redundancy. Together, these findings provide compelling empirical evidence for developing more effi- cient video processing strategies by eliminating redundant computation on visually similar, high-attention frames.
3.3. Differential Distillation Principle
Our frame- and patch-level analyses motivate our differen- tial distillation principle, fundamentally rethinking how to identify and preserve salient information while reducing re- dundant computation. The key insight that truly important information should simultaneously satisfy two criteria: (1) high relevance the query intent, and (2) low redundancy with respect to its temporal context. Formally, for any video component (frame, patch, or feature) and query Q, we define its differential information saliency score:
3.2. Patch-Level Redundancy Analysis
While frame-level redundancy analysis suggests processing only keyframes (Ye et al., 2023; Shen et al., 2024) for effi- ciency, this approach faces two critical limitations: (1) poten- tial loss of pivotal moments due to selection uncertainty, and (2) missing temporal context essential for tasks like action recognition and causal reasoning. Although this necessitates effective compression strategies for non-keyframes, naive feature merging approaches (e.g., pooling (Xu et al., 2024)) can be suboptimal due to substantial information overlap between fey-keefsames and keyframes, risking attention misattribution to redundant information.
D(v) = R(v, Q) T(e, ((v)),
(1)
where R(v, Q) measures query relevance and T(e C(w)) captures temporal redundancy with context features C(v). A higher D(v) indicates more unique, task-relevant infor- mation deserving
computational resources.
This formulation provides two key advantages: (1) natural adaptation to varying temporal granularities, enabling uni- fied frame- and patch-level operations, and (2) query-awan processing that dynamically allocates computation based on task requirements.
To verify the hypothesis, we analyze video frames by se- lecting the top 32 attention-weighted frames as keyframes. com-
4. Methodology
For each patch in the remaining 96 non-keyframes, pute its query attention weight and visual similarity to its corresponding patch in the preceding keyframe (or the first keyframe for earlier non- oon-keyrrames. The visual similarity is measured using cosine similarity between the model's visual embeddings. Figure 4 reveals that high-attention patches
Following the differential distillation principle, we propose VILAMP, a hierarchical architecture for mixed-ppecisse video processing, as shown in Figure 5. VILAMP com- bines frame-level selection via Differential Keyframe Se- lector (DKS) ($4.1) and patch-level compression via Dif-

---

Scaling Video-Language Models to 10K Frames via Hierarchical Differential Distillation
ferential Feature Merger (DFM) ($4.2), enabling efficient resource allocation based information saliency.
Chen al., 2024c), we propose a differential feature merg- ing (DFM) approach that compresses non-keyframes to sin- gle tokens while preserving task-relevant information.
4.1. Differential Keyframe Selection
Specifally, for non-keyframe fn. = (p,,
P and
The differential keyframe selection mechanism identifies frames with high query relevance while maintaining tem- poral distinctiveness. Given frames V = (f1, fa..... fx) and query Q, we first encode both the query and each frame into normalized d-dimensional embeddings using a CLIP encoder (-) (Radford et al., 2021a), and then compute the frame-level query relevance score Ry(Sn, h Q) as the cosine similarity between these embeddings:
its nearest preceding keyframe f& = (p Pk- Pk: /. we compute patch-level embeddings using the VLM's visual encoder Ep(-), that generates a d-dimensional embedding for each patch. The differential information saliency D, of each patch pM is determined by two factors, i.e., its query relevance R2 and temporal redundancy Tp:
M
Dp(pm Rp(p" Q) XTp (Pp.PPPPP)
(5)
Rp(p" Q) cos(p"" .q):
(6)
Rf Q) cos (f:( 9) fn = Ey(()) E Rd. q (Q) ER",
(2)
Tp(p" C(p")) cos(p," PE
(3)
(p" my M
m==1
M Fop(fn).
{PF 1 = Ep(fa)
(8)
Then we instantiate the temporal redundancy function in Eq. at the frame level as the maximum cosine similarity between frame fn and its context frames C(fa):
where m is the patch index, k/n refers to the keyframe/non- keyframe index, À balances the trade-off between query relevance and temporal uniqueness, and p"' /p"" and q rep- resent the encoded patch and query embeddings, respec- tively. C(p")) is set to {oX² } to compute temporal redun- dancy only with respect to the corresponding patch position in the keyframe, leveraging the strong spatial correspon- dence prior in video content while maintaining computa- tional efficiency. We set T = 0 for non-keyframes without preceding keyframes.
C((n),C(Jn)) = max cos(fr (f) fEC(fa)
(4)
While one could naively set C(fn) = V. {fa), i.e., mea- suring frame-level redundancy by comparing each frame against all others, this would incur prohibitive quadratic complexity 3 Moreover, selecting frames that directly maximize yTy without proper prioritization of two terms risks selecting frames that are temporally distinct but irrele- vant to the query intent. Therefore, we propose an efficient greedy algorithm (Algorithm 1). It ranks frames by query relevance and selects those maintaining sufficient temporal distance (threshold r), achieving (max (N K, N log N)) complexity for K keyframes while ensuring both semantic relevance and temporal diversity.
The saliency scores guide our adaptive token merging strat- egy through differentially weighted pooling operation:
t, ΣM
Wilh P
Σm=
M
with
"x = softmax (Dp(e))) DDWN
(10)
Algorithm Differential Keyframe Selection Data: V == (f1, f2. ... fN):: Video frames; Q: Query; K: Max- imum keyframes; 7: Similarity threshold.
where in is the compressed representation for the non- keyframe fn, wh is the pooling weights, and a controls the sharpness of the weight distribution. This formulation ensures that patches receive high-importance weights when they contain query-relevant information that is not already captured by the corresponding keyframe patches.
Result: XC: Keyframe Set.
// Sort all frames by query-relevance descending order (fi, f2.... ( = sorted(V. key = lambda fn : (J,, Q))
// Initialize 1. with the most relevant frame
-C= -())
4.3. Multimodal Learning
// Iterate through the sorted frames while n <- to N do
We utilize two-stream visual connector architecture to integrate heterogeneouss representations from keyframes and compressed non-keyframes into the language model:
if [[| K and (f,,,, KC) < then f.add(f.)
return KC
h MLPx(pT)) h (t,,
(11)
4.2. Differential Feature Merging
where h is the projected embedding for the m-th patch of keyframe fa and h, is the embedding for the compressed representation of non-keyframe fn. processed through sepa- rate learnable two-layer MI.Ps. The model is then trained
While DKS identifies salient keyframes, non keyframes pro- vide crucial temporal context. Instead of discarding these frames as mainstream approaches (Yang et al., 2024a;
5

---

Scaling Video-Language Models to 10K Frames via Hicrarchical Differential Distillation
Large Language Modiel
Fonee Sortad
BB
eB0 ae
2010
Seytrumm Selector
Differential Fealure ging
BB BB
Dvary Betevane
Tampore Backedency
Srybemer
Gryloree
byboer
Grytoree
Diffirrntial Keyframe Selection (DKS)
Diliinetial Koyframm Selection (DKS)
(e Diliieniiall Nimpinng (DFM)
Hour-Long Video Input
Figure 5. Overview of VILAMP, which first identifies representative keyframes from videos using the differential keyframe selection algorithm (a) and then adaptively merges visual features from oon-keyframess through the differential feature merging mechanism (b).
using a language modeling objective over the concatenated sequence of visual embeddings and the prompt:
Implementation Details. VILAMP employs the same ar- chitecture as VLaVV-OOee Vision (Li et al., 2024a), utilizing SigLIP-So400m' as the visual encoder, a two-layer MLP as the vision-language connector and Qwen2-7B* as the lan- guage model. We process frames at x84 resolution and empirically set in Alg. to 0.85, the keyframe count K to 32, À in Eq. 5 to 1, and in Eq. 10 to 10 unless otherwise specified. We employ CLIP-VITI HIP---T---32" as E,(-) in Eq. 3. More training details are provided in Appendix A.3.
C C= log log P(Al(h)) I E K) U (h.lSn & K.) Q), (12)
where the keyframe and non-keyframe embeddings are ar- ranged in their temporal order, Q and A represent the query and target answer, respectively.
This design of VILAMP offers three key advantages. First, dramatically reduces the total number of visual tokens from MN to MK + (N - K) where < N, enabling ef- ficient processing of extended video sequences. Second, the full-token keyframe representations serve as natural anchors for learning temporal relationships, facilitating the model's interpretation of compressed intermediate frames. Third, the end-to-end optimization of DFM parameters through direct supervision ensures that our compression mechanism adapts to preserve task-relevant information effectively.
5.2. Main Results
Table 1 presents comprehensive evaluation results across five benchmarks. Among open-source models of sim- ilar size, VILAMP establishes new state-of-the-art per- formance across diverse temporal scales, from short- form (EgoSchema) to medium (Long VideoBench and MLVU) and long-form (L.Bennch and Video-MME) video understanding tasks. Remarkably, VILAMP performs petitively with or surpasses substantially larger open-source models (~70B parameters) on these benchmarks. This supe- rior performance particularly evident long-form video understanding: On MLVU, VILAMP achieves 72.6% ac- curacy, even outperforming the proprietary GPT-4o model (64.6%); On the long-video subset of Video-MME, VIL- AMP achieves 67.5% in non-subtitled setting and 73.5% in subtitled setting, outperforming existing similar-sized mod- els by a large margin. These consistent strong results across varying temporal scales, particularly in challenging long- form scenarios, validate VILAMP's robust video modeling capabilities.
5. Experiments
5.1. Experimental Setup
Benchmarks. We thoroughly evaluate VILAMP five video understanding benchmarks spanning diverse tem- poral scales and tasks (See Appendix A.1 for details): LVBench (Wang et al., 2024b) for long-term decision- making, EgoSchema (Mangalam et al., 2023) for natural scenario understanding, VideeBeBeen (Wu et al., 2024) for referred reasoning, MLVU (Zhou al., 2024) for multi- task question answering, and Video-MME (Fu et al., 2024a) for comprehensive video understanding.
htttps://uuginggiiccccoo/// si siggipp-so400mnppaccch--222 "nttps:/hnugingffce..o
Baselines. We conduct comprehensive comparisons with sfate-of-the-art baselines spanning proprietary VLMs, open- source imate VLMs, and open- video- Lado-language models. Please refer to Appendix A.2 for more details.
sutpps:/bugginggcce..c/
6

---

Scaling Video-Language Models to 10K Frames via Hicrarchical Differential Distillation
Table Accuracy (%) on five video understanding benchmarks. Size indicates the number parameters. Frames denotes either fixed number of frames sampled from each video (e.g., 32 means sampling 32 frames regardless of video length) or the frame sampling rate (e.g.. FPS means sampling 2 frames per second). The best and second best results among open-soeneee source models similar size (7~9BB are in bold and underlined, respectively. M-Avg refers to the average score across all multi-choice tasks. For Video-MME, we report the performance on the whole benchmark ("Overall"" and the long-form video subset ("Long", videos > 39 minutes). All baseline results are collected from their original papers. "." indicates results not found.
LVBench
EgoSchema
LongVideo
MLVU
Video-MME (wo sub)
Models
Size
Frames
Overall
Validation
Validation
M-Avg
Overall
Long
Average Duration
4101s
180s
473s
651s
1,010s
2,386s
Proprictary Vision-Language Models
GPT4-V (Achiam al., 2023)
Undisclosed Undisclosed
59.1
49.2
59.9/63.3
53.5 56.9
GPT4-o (OpenAI, 2024)
30.8
66.7
64.6
71.9/77.2
65.
Gemini-1.5 P.o-Pro (Reid al., 2024)
Undisclosed
33.1
64.0
75.0/81.3
67.4/77.4
Open-Source Multi-Image Vision-Language Models
LLaVA-OneVision (Li 2024b)
72B
32
62.0
61.3
66.4
66.3
60.0/
IntemVL2 (Chen et al., 2024c)
76B
16
61.0
69.9
61.2/67.8
V.aVV-OneVisionn (I.i et al., 2024a)
7B
32
60.1
56.5
64.7
58.2/
Oryx-1.5 (Liu et al., 2024d)
7B
128
56.3
67.5
58.8 64.2
IntemVL2 (Chen el al., 2024c)
8B
16
54.6
64.0
56.3
IntemVL.2.5 (Chen al., 2024b)
8B
64
60.0
68.9
64,2 66.9
Chat-Univi (Jin et al., 2024)
7B
64
40.6 45.9
35.8 41.8
MiniCPM-v2.66 (Yao al., 2024)
8B
64
54.9
37.3
60.9/63.7
51.8/ 56.3
mPLUG-OW13 (Ye et al., 2024)
7B
16
52.1
63.7
59.3/
50.1/
Qwen2-VL. ( (Yang et al., 2024a)
7B
2FPS
55.6
69.0
NVILA (Liu et al., 2024c)
7B
256
70.1
64.2 70.0
54.8/ / 63.3
Open-Source Video-I andea-- Models
VideoLLLMM22 (Cheng et al., 2024)
72B
16
63.9
61.2
62.4/
59,6
LLaVA- (Zhang et al., 2024c)
7B
IFPS
65.6
58.2
70.8
63.3/69.7
M.A (I.i al., 2024d)
7B
IFPS
23.9
38.5
33.2
25.9
Video-XL (Shu et al., 2024)
7B
2,048
49.5
64.9
/
49.2
Video-LLaVA (Lin al., 2023)
7B
8
38.4
37.6
47.3
/9,9/416
36.2 38.1
aMA2 (Zhang et al., 2023)
7B
16
51.7
48.5
47.9
VideoChat2 /KunChang et al.. 2023)
7B
16
54.4
36.0
47.9
54.6/
39.2
Video-CCAM (Fei et al., 2024)
9B
96
58.5
53.2/574
46.7/49.9
PLLaVA (Xu ct al., 2024)
7B
16
40.2
Movie-Chat (Song et al., 2024)
7B
2,048
22.5
Kangaroo (Liu et al., 2024c)
8B
64
62.7
54.8
61.0
56.0/
4 59.3
Long VU (Shen el al., 2024)
7B
67.6
65.4
60.6
Long VA (Zhang et al., 2024c)
7B
128
56.3
52.6/54.3
46.2
VILA (Chen al., 2024a)
7B
256
67.7
57.1
6
VILAMP(OUrs)
7B
1FPS
45.2
70.2
61.2
72.6
67.5
73..5
57.8 68.1
5.3. Scaling Up to 10K Frames
by sampling long-form videos from Video-MME to create "haystacks" ranging from 2K to 10K frames (at IFPS). We then insert "needle" video clips (30~120 seconds) within these haystacks at random positions. Both the needle videos and their corresponding query-answer pairs sampled from the original Video-MME dataset. Models must not only locate these needles but also comprehend their tempo- ral content to answer targeted queries. We create 3K test cases that distributed across five haystack lengths (2K, 4K, 6K, 8K, and 10K frames), with carefully balanced question types to ensure comprehensive evaluation. The testing was conducted in a non-subtilled setting.
The ability to process hour-long videos is crucial for real- world applications. However, evaluating models' long video understanding capabilities remains challenging. While ex- isting works often adapt the iaalleiin---Hstackkk (NIAH) paradigm from LLM research to assess VLMs (Zhang et al., 2024a; Shen et al., 2024), these adaptations typically ex- hibit two key limitations: (1) using simplified single-frame "needles" that fail to capture temporal dynamics, and (2) limiting haystack sizes to at most 3K frames, inadequately testing true long-form processing capabilities.
To rigorously address above limitations, we develop Video- NIAH, more challenging variant of the NIAH task specifi- cally designed for video content. We construct VideoNIAH
Efficacy. Figure 2 compares VILAMP against LLaMA- VID (Li et al., 2024d) and VideoChat-Flash (Li et al., 2024c)
7

---

Scaling Vidco- Video-Language Models to 10K Frames via Hierarchical Differential Distillation
75
Table 2. Ablation study results.
Model
MLVU
Video-MME:
70-
LLaVA-OneVision
56.5
64.7
58.2
Accuracy (%)
+DKS (Ours)
$7.0
67.3
61.8
65
+Query-Guided Sampling
54,8
63.7
55.6
+Uniformly Sampling
57.6
66.8
60.2
60
+DFM (Ours)
60.5
72.6
65.1
+Q-oormer
62.6
59.2
+Mean-Pooling
56.3
67.3
62.1
.55
50
10
30
40
60
Number of Keyframes
across varying haystack lengths, two state-of-the-ar mod- els specifically designed for ultra-long video understanding. The results reveal several key insights: (1) While all mod- els show some performance degradation as context length increases, VILAMP maintains significantly higher average accuracy at 10K frames (58.15% vs. 47.25% of VideoChat- Flash). (2) VILAMP exhibits remarkably stable perfor- mance scaling. with 12.82% less drop of average accuracy from 2K to 10K frames compared to VideoChat- Flash, sug- gesting superior scalability to ultra-long videos.
Figure 6. Analysis of model performance with varying numbers of keyframes across different video benchmarks. The test for Video- MME is conducted in non-subtitled setting.
Results in Table demonstrate the effectiveness of both components. For frame selection, DKS shows particularly strong improvements
longer videos (MLVU and Video- MME), while uniform sampling becomes less effective with increased video length due to missing key information, and query-guided sampling underperforms due to locally con- centrated frame selection. For feature merging, DFM con- sistently outperforms alternatives across all datasets by ef- fectively preserving contextual information, whereas mean- pooling suffers from redundant information and Q-former is limited by fixed embedding constraints.
Efficiency. Figure 1 shows that VILAMP exhibits mini- mal memory growth with increased input length, achieving ~50% lower memory consumption than baselines when processing 10K frames. To comprehensively evaluate com- putational efficiency, we additionally measure the required computational load (FLOPs) and inference latency (Time- Fiiist-Token, TTFT) under varying input lengths. As reported in Appendix B.1, VIL.MMP shows substantial com- putational efficiency, particularly for longer inputs with FLOPs reduced by >80% compared to baseline models at 8,192 frames while maintaining comparable inference speed.
Influence of Keyframes. To determine the optimal trade- off between computational efficiency and model perfor- mance, we analyze VILAMP'ss behavior with varying num- bers of keyframes. Results in Figure 6 show consistent performance improvements as keyframe count increases, until reaching dataset-specific saturation points. Short-form videos (Video-MME-SHI -Short and Long VideoBench) plateau around 16 keyframes, while longer videos (Video-MME. Medium/Loong and MLVU) require 32 keyframes for optimal performance. This aligns with our findings in $ 3.1 that in- formative frames can be sparsely sampled, though optimal sampling density varies with video length.
5.4. Discussions
In this section, we conduct a comprehensive investigation to validate our design choices. We first evaluate the effec- tiveness of DKS and DFM, and then analyze the sensitivity of model performance to the number of keyframes. Ap- pendix B.2 further discusses other configuration settings.
Effectiveness other video tasks Although VILAMP's effectiveness has been verified on video question answering benchmarks, it is still unclear whether this method can truly preserve necessary information for more temporal-sensitive tasks like action recognition, video grounding or temporal reasoning. Therefore, we add some experiments.
Ablation Study. We evaluate the effectiveness of DKS and DFM through comparative experiments. For frame selec- tion, we compare DKS with two baselines: (1) query-guided sampling, which selects 32 frames with the highest query relevance scores (Eq. 2) without considering inter-frame redundancy, and (2) uniform sampling, which extracts 32 frames at fixed intervals. For feature merging, we com- pare DFM against (1) Q-former (Li al., 2023), which integrates 729 visual token sequences into fixed-length embeddings through a learnable cross-attentic module, and (2) mean-pooling which averages token embeddings within each key-keyeeame into one token. All models are trained on the same dataset as mentioned Appendix A.3.
For the video grounding task, we perform experiments on the widely used QVHighlights (Lei et al., 2021) benchmark. This dataset provides annotations for videos, marking the timestamps of query-relevant segments and the positions of keyframes. We evaluated VILAMP using the Hit@l k metric, which measures the probability of the ground-trun keyframe appearing in the top-k keyframes selected by our
8

---

Scaling Video-Language Models to 10K Frames via Hierarchical Differential Distillation
method. The results are as follows:
Acknowledgments
This work was supported by the Beijing Outstanding Young Scientist Program NO. BJJWZYJHO (TTHH001211100000000000 and Intelligent Social Governance Platform, Major Innovation & Planning Interdisciplinary Platform for the "Double-Firs. Class" Initiative, Renmin University of China, the Funda- mental Research Funds for the Central Universities, and Public Computing Cloud, Renmin University of China, the fund for building world-class universities (disciplines) of Renmin University of China.
Table 3. Evaluation results on QVHighlights.
Hit@1
Hit@5
Hit@10
@0
26.1
47.3
60.0
83.2
As shown in Table 3, VILAMP achieves 60% accuracy for @it@ 10 and 83% for @it@ 30. providing strong empirical support for its Dynamic Keyframe Selection (DKS) pro- cess. The current method effectively captures query-relevare video clips.
Impact Statement
For the action recognition task, we incorporate two base- lines: Adaframe (Wu et al., 2019) and MGSampler (Zhi
This paper aims to propose a video-language model that efficiently handles ultra-long video inputs with relatively low computational cost. We would like to highlight a po- tential impact related to the generated output of our model, which may contain inaccuracies, biases, or offensive content. These issues may stem from various factors, including in- herent biases in the training datasets, or the complex nature of video-language interpretation. Such outputs could po- tentially lead to misinformation propagation or unintended consequences in real-world applications. Therefore, users are expected to implement rigorous validation protocols when deploying this model.
et al., 2021), )) and evaluate them alongside VILAMM on three benchmarks: ActivityNet (Caba Heilbron et al., 2015),
Something-Someethig V2 (Sth-V2) (Goyal al., 2017), and UCF-101 (Soomro et al., 2012). Since VILAMP is generative model, we reformulate the original classification task as a multiple-choice problem during testing. The results are as follows:
Table Evaluation on action recognition tasks, The results
Adaframe and MGSampler are sourced from their papers.
Method
Net (mAP %)
Sth-V2 (Acc. % UCF-101 (Aec. %))
Adairame
71.5
MCSSampler
60.1
95.2
References
VILAMP(OUr)
78.6
97.6
Achiam, J., Adler, S., Agarwal, S., Ahmad, L., Akkaya, I., Aleman, F. L., Almeida, D., Altenschmidt, J., Altman, S., Anadkat, S., et al. Gpt-4 technical report. arXiv preprint arXiv:2303.08772 2023.
Notably, although VILAMP is not trained on these datasets, it demonstrates robust performance. This may be attributed to the generalizable knowledge acquired by the large-scale pretrained model, enabling effective zero-shot transfer to action recognition tasks.
Alayrac, J.-B., Donahuc, J., Luc, P., Micch, A., Barr, I., Hasson, Y., Lene, K., Mensch, A., Millican, K., Reynolds, M., et Flamingo: visual language model for few- shot learning. Advances in neural information processing systems, 35:23716-23723 2022.
6. Conclusion
In this work, we address the critical challenge of scaling video-language models (VLMs) to process ultra-long videos efficiently. We present VILAMP, video-language model guided by differential distillation. By combining Differ- ential Keyframe Selection (DFS) and Differential Feature Merging (DFM), VILAMP demonstrate superior perfor- mance across five video benchmarks, particularly in long- form scenarios. On the challenging VideoNIAH benchmark, VILAMP processes videos up to 10K frames (~2.7 hours) on single NVIDIA A100 GPU, outperforming state-of- the-art models by 12.82% in accuracy while significantly reducing computational overhead. The hierarchical com- pression strategy enables stable performance across varying video lengths, with minimal memory growth and latency. Our work establishes a scalable paradigm for long-form video understanding, bridging the gap between computa- tional constraints and real-world demands.
Anonymous. LongVILA: guage models for long videos. In Submitted to The Thir- teenth International Conference on Learning Represen- lations, 2024. URL https: / openrevien .nt// under review.
Scaling long-conters visual lan-
Ataallah, K., Shen, X., Abdelrahman, E., Sleiman, E., Zhuge, M., Ding, J., Zhu, D.,, Schmidhuber, J., and El- hoseiny, M. Goldfish: Vision-language understanding of arbitrarily long videos. arXiv preprint wxXiv22407112699 2024.
Bai, J., Bai, S., Yang, S., Wang, S., Tan, S., Wang, P., Lin, J., Zhou, C., and Zhou, J. Qwen-vl: A versatile vision- language model for understanding, localization, text read-
9

---

Scaling Video-Language Models to 10K Frames via Hierarchical Differential Distillation
ing, and beyond. arXiv preprint arXiv:2308. 12966, 1(2): 3, 2023.
Farré, M., Marafioti, A., Tunstall, L., Von Werra, L.,, and co/ 2024.
Wolf, T. Finevideo.
datasets/
Bain, M., Nagrani, A., Varol, G., and Zisserman, Frozen in time: A joint video and image encoder for end-to-end retrieval. In IEEE International Conference on Computer Vision, 2021.
Fei, Li, D., Deng, Z., Wang. Z., Liu, G., and Wang, understand-
H. Video-ccam: Enhancing ing with causal cross-attention masks for short and long videos, 2024. URL https: arxiv org/ /abs/ 2408 100...403..
Baker, B., Akkaya, I., Zhokov, P., Huizinga, J., Tang, J., Ecoffer, A., Houghton, B., Sampedro, R., and Clune, J. Video pretraining (VPTy: Learning to act by watching unlabeled online videos. In Oh, A. H., Agarwal, A., Belgrave, D., and Cho, K. (eds.), Advances Neurul Information Processing Systems, 2022. URL https: //openreview pwt/ffrntiteeeeeee
Fu, C., Dai, Y., Luo, Y.. Li, L., Ren, S., Zhang, R., Wang, Z., Zhou, C., Shen, Y., Zhang, M., et al. Video-mme: The first-ever comprehensive evaluation benchmark of multi-modal IIms in video analysis. arXiv preprint 2024a.
Bolya, D., Fu, C.-Y., Dai, X., Zhang, P., Feichtenhofer, C., and Hoffman, J. Token merging: Your vit but faster. In The Eleventh International Conference Learning Representations, 2023. https: //ppp/eeprr.eeeeeeeeeeeee net/ Eorumiidumi Trw.-JJooRaaaaaaa
Fu, Y., Panda, R., Niu, X., Yue, X., Hajishirzi, H., Kim, Y., and Peng. H. Data engineering for scaling language models to 128k context. arXiv preprint arXiv:2402. 2024b.
Goyal, R., Ebrahimi Kahou, S., Michalski, V., Materzynska, J., Westphal, S., Kim, H., Haenel, V., Fruend, I., Yianilos, P., Mueller-Freitag, M., et al. The" something something" video database for learning and evaluating visual com- mon sense. In Proceedings of the IEEE international conference computer vision, pp.
Caba Heilbron, F., Escorcia, V., Ghanem, B., and Car- los Niebles, J. Activitynet: A large-scale video bench- mark for human activity understanding. In Proceedings of the ieee conference on computer vision and pattern recognition, pp. 961-970, 2015.
2017.
Chen, Y., Xue, F., Li, D., Hu, Q., Zhu, L., Li, X., Fang, Y., Tang, H., Yang, S., Liu, Z., He, E., Yin, H., Molchanov, P., Kautz, J., Fan, L., Zhu, Y., Lu, Y., and Han, Longvila: Scaling long-context visual language models for long videos, 2024a. URL https:/ /ttp/aaiivv..///.e//eee/eaaa orgoorgggoooooo abs/ 2408. 10188.
Jia, B., Lei, T., Zhu, S.-C., and Huang. S. Egotaskqa: Under- standing human tasks in egocentric videos. Advances in Neural Information Processing Systems, 35:3343-3360, 2022.
Jin, P., Takanobu, R., Zhang, W. Cao, X., and Yuan, L. Chat-univi: Unified visual representation empowers large language models with image and video understanding. In Proceedings of the IEEF/CVF Conference on Computer Vision and Pattern Recognition, pp. 13700-13710, 2024.
Chen, Z., Wang, W. Cao, Y., Liu, Y., Gao. Z., Cui, E., Zhu, J., Ye, S., Tian, H., Liu, Z., et al. Expanding per- formance boundaries of open-source multimodal models with model, data, and test-time scaling. arXiv preprint arXiv:2412.05277. 2024b.
KunChang, L., Yinan, H., Yi, W., Yizhuo, L., Wen- hai, W., Ping, L.,, Yali, W., Limin, W., and Yu, Q. Videochat: Chat-centric video understanding. arXiv preprint arXiv:2305.06335 2023.
Chen, 7.,, Wu, J., Wang, W., Su, W., Chen, G., Xing, S., Zhong. M., Zhang, Q.. Zhu, X., Lu, L., et al. Internvl: Scaling up vision foundation models and aligning for generic visual- lingul-ing tasks. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 24185-24198, 2024c.
Kuratov, Y., Bulatov, A., Anokhin, P., Sorokin, D., Sorokin, A., and Burtsev, M. In search of needles in 11m haystack: Recurrent memory finds what Ilms miss, 2024. URL https: / /aaaiiiiiv. orq/ wttpa////......
Cheng, Z., Leng, S., Zhang, H., Xin, Y., Li, Chen, G., Zhu, Y., Zhang, W., Luo, Z., Zhao, D., et al. Videollama 2: Advancing spatial-temporrl modeling and audio under- standing in video-Ilms. arXiv preprint 24Xx:2406.07476. 2024.
Langley, P. Crafting papers on machine learning. In Langley, P. (ed.), Proceedings of the 17th International Conference on Machine Learning (ICML,2000) pp. 1207-1216, Stan- ford, CA, 2000. Morgan Kaufmann.
Dubey, A.. Jauhri, A., Pandey, A., Kadian, A., Al-Dahle, A., Letman, A., Mathur, A., Schelten, A., Yang, A., Fan, A., et al. The Ilama herd of models. arXiv preprint arXiv:2407.21788 2024.
Lei, J., Berg, T. L., and Bansal, M. Detecting moments and highlights in videos via natural language queries. Advances in Neural Information Processing Systems, 34: 1 11846-11855 2021.
10

---

Scaling Video-Language Models to 10K Frames via Hierarchical Differential Distillation
Li, B., Zhang, Y., Guo, D., Zhang, R., Li, E, Zhang, H., Zhang, K., Zhang, P., Li, Y., Liu, Z., and Li, C. Llava- onevision: Easy visual task transfer, 2024a. URL / //arxiv. org/ abs/ / 2400. 03326.
Liu, Z., Zhu, L.,, Shi, B.,, Zhang, Z., Lou, Y., Yang, S., Xi, H., Cao, S., Gu, Y., Li, D., Li, X., Fang, Y., Chen, Y., Hsieh, C.-Y., Huang, D.-A., Cheng, A.-C., Nath, V., Hu, J., Liu, S., Krishna, R., Xu, D., Wang. X., Molchanov, P., Kautz, J., Yin, H., Han, S., and Lu, Y. Nvila: Efficient frontier visual language models, 2024c. URL https:
Li, B., Zhang, Y., Guo, D., Zhang, R., Li. F., Zhang, H., Zhang, K., Zhang. P., Li, Y., Liu, Z., et al. Llava- onevision: Easy visual task transfer. arXiv preprint arXiv:2408.03320 2024b.
//arxiv. org/ abs/////////// 2412 044
Mangalam, K., Akshulakov, R., and Malik, J. Egoschema: A diagnostic benchmark for very long-form video lan- guage understanding. Advances in Neural Information Processing Systems, 36:46212 2023.
J., Li, D., Savarese, S., and Hoi, S. Blip-2: Bootstrapping language-ing image pre- pre-training with frozen image encoders and large language models. In International conference on machine learning, pp. 19730-19742, PMLR, 2023.
Micikevicius, P., Narang, S., Alben, J., Diamos, G., Elsen, E., Garcia, D., Ginsburg, B., Houston, M., Kuchaiev, O., Venkatesh, G., and Wu, H. Mixed precision training. In International Conference on Learning Representations, 2018. URL https: /pwpr/eeeeeeeeeeeeeeeeeee.e.eeee net net.:orunttf forum?
Li, X., Wang, Y., Yu, J., Zeng. X., Zhu, Y., Huang, H., Gao, J., Li. K., He, Y., Wang, C., al. Videochat-flan Hi- erarchical compression for long-context video modeling- arXiv preprint arXiv:2501.005:: 2024c.
i id=r1gs9JgR3
Li, Y., Wang, C., and Jia, Llama-vid: An image is worth 2 tokens in large language models. 2024d.
Nan, K., Xie, R., Zhou, P., Fan, T., Yang, Z., Chen, Z., Li, X., Yang, J., and Tai, Y. Openvid- m: A large-scale high-quality dataset for text-to-video generation. arXiv preprint 24Xiv:2407.0237// 2024.
Li, Y., Wang, C., and Jia, Llama-vid: tokens in large language models. In European Conference Computer Vision, pp. 323-340. Springer, 2025.
An image is worth 2
OpenAl. Hello gpt-4o. In Blog, 2024. Utthttps:
/ / //openai...
Lin, B., Ye, Y., Zhu, B., Cui, J., Ning, M. Jin, P., and Yuan, L. Video-llava: Learning united visual represen- tation by alignment before projection. arXiv preprint 231.:2311.10122 2023.
Patraucean, V., Smaira, L., Gupta, A., Recasens, A., Mar- keeva, L.., Banarse, D., Koppula, S., Malinowski, M., Yang, Y., Doersch, C., et al. Perception test: A diagnos- tic benchmark for multimodal video models. Advances in Neural Information Processing Systems, 36:42748- 42761, 2023.
Lin, J., Yin, H., Ping, W., Molchanov, P., Shoeybi, M., and Han, S. Vila: On pre-training for visual language models. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 26689- 26699, 2024.
Peng, Z., Wang. W., Dong, L., Hao, Y., Huang. S., Ma, S., and Wei, F. Kosmos-2: Grounding multimodal large language models the world. arXiv preprint arXiv:2306. 14824, 2023a.
Liu, H., Li, C., Wu, Q.. and Lee, Y. Visual instruction tun- ing, Advances in neural information processing systems, 36, 2024a.
Peng, Z., Wang. W., Dong. L., Hao, Y., Huang. S., Ma, S., and Wei, F. Kosmos-2: Grounding multimodal large language models to the world, 2023b. URL https: //arxiv. org/ abs 2236 14824.
Liu, H., Yan, W., Zaharia, M., and Abbeel, P. World model mintt video and language with blockwise ringattention, 2024b. URL
Radford, A., Kim, J. W., Hallacy, C., Ramesh, A., Goh, G., Agarwal, S., Sastry, G., Askell, A., Mishkin, P., Clark, J., Krueger, G.. and Sutskever, I. Learning transferable visual models from natural language su- pervision. In Meila, M. and Zhang, T. (eds.), Pro- ceedings of the 38th International Conference on Mu- chine Learning, volume 139 of Proceedings of Ma- chine Learning Research, pp. 8748-8763. PMLR, 18- 24 Jul 2021a. URL https: http:p:::/ppocceeeee http///ooocdnnnn/e/ee mlr. press/v139 raadoordffddaaeee presss/v139/rrddfor221...... html.
Liu, J., Wang, Y., Ma, H. Wu, X., Ma, X., Wei, X., Jiao, J., Wu, E., and Hu, J. Kangaroo: A powerful laneo-language model supporting long-context video input, 2024c. URL https: //t///axxxa www.i.vvvaoo http://wwwvvog////////// abs 2408. .10.8.555444 15542.
Liu, Z. Dong, Y., Liu, Z., Hu, w., Lu, J., and Rao, Y. Oryx mlim: On-demand spatial-tempon understanding at arbitrary resolution. arXiv preprint arXiv:2409.1296. 2024d.

---

Scaling Video-Language Models to 10K Frames via Hierarchical Differential Distillation
Radford, A., Kim, J. W., Hallacy, C., Ramesh, A., Goh, G., Agarwal, S., Sastry, G., Askell, A., Mishkin, P., Clark, J., et al. Learning transferable visual models from natural language supervision. In International conference on machine learning, pp. 1738-8763. PMLR, 2021b.
Wang, Y., He, Y., Li, Y., Li, K., Yu, J., Ma, X., Li, X., Chen, G., Chen, X., Wang, Y., et al. Internvid: A large- scale video-text dataset for multimodal understanding and generation. arXiv preprint WxXiv2337.069422 2023.
Wu, B. and Star, S. Y. A benchmark for situated reasoning in real-world vidcos. NeurlPS Datasets and Benchmarks, 2021.
Rawal, R., Saifullah, K., Farré, M., Basri, R., Jacobs, D., Somepalli, G., and Goldstein, T. Cinepile: A long video question answering dataset and benchmark. arXiv preprint 1rXiv:2405.088133 2024.
Wu, H., Li, D., Chen, B., and Li, J. Longvideohench: A benchmark for long-context interleaved video-language understanding, 2024. URL httts:://axii..rr abs /ttp/2407.15575
Reid, M., Savinov, N., Teplyashin, D., Lepikhin, D., Lilli- crap, T., Alayrac. J.-b., Soricut, R., Lazaridou, A., Firat, O., Schrittwieser, J., et al. Gemini 1.5: Unlocking multi- modal understanding across millions of tokens of context. arXiv preprint ar 24X:v:2403.053300 2024.
Wu, P., Escontrela, A., Hafner, D., Abbeel, P., and Goldberg, K. Daydreamer: World models for phys- ical robot learning. In Liu, K., Kulic, D., and Ich- nowski, J. (eds.), Proceedings of The 6th Conference on Robot Learning, volume 205 of Proceedings of Machine Learning Research, pp. 2226-2226422 2240.. PMLR, 14-18 Dec 2023. URL https: httthttp://peeee http:///pooceeeeeee mlr, press/ v205 / www3c/ html.
Shen, X., Xiong, Y., Zhao, C.,, Wu, L., Chen, J., Zhu, C., Liu, Z., Xiao, F., Varadarajan, B., Bordes, F., et al. Longvu: Spatiotemporal adaptive compression for long video-language understanding. arXiv preprint arXiv:2410.174344 2024.
Wu, Z., Xiong, C., Ma, C.-Y., Socher, R., and Davis, L. Adaframe: Adaptive frame selection for fast video recog- nition. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 1278- 1287, 2019.
Shu, Y., Zhang, P., Liu, Z. Qin, M., Zhou, J., Huang, T., and Zhao, B. Video-xl: Extra-long vision language model for hour-scale video understanding. arXiv preprint arXiv:2409. 14485, 2024.
Xiao, J., Shang, X., Yao, A., and Chua, T.-S. Next-qa: Next phase of question- answering to explaining temporal actions. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pp. 9777-9786, June 2021.
Song, E.,, Chai, W., Wang. G., Zhang, Y., Zhou, H., Wu, F., Guo, X., Ye, T., Lu, Y., Hwang. J.-N., et al. Moviechat: From dense token to sparse memory for long video un- derstanding. arXiv preprint Stti:2307.114499 2023.
Song. E., Chai, W., Wang, G., Zhang. Y., Zhou, H., Wu, F. Chi, H., Guo, X., Ye, T., Zhang, Y., al. Moviechat: From dense token to sparse memory for long video under- standing. In Proceedings of the IEEE/CVE Conference on Computer Vision and Puttern Recognition, pp. 18221- 18232, 2024.
Xu, L., Zhao, Y., Zhou, D., Lin, Z., Ng. S. K., and Feng, J. Pllava: Parameter-free llava extension from images to videos for video dense captioning. arXiv preprint arXiv:2404.1699 2024.
Yang, A., Yang, B., Zhang, B., Hui, B., Zheng, B., Yu, B., Li, C., Liu, D., Huang, F., Wei, H., et al. Qwen2. 5 technical report. arXiv preprint arXiv:2412. 2024a.
Soomro, K., Zamir, A. R., and Shah, M. Ucf101: A dataset of 101 human actions classes from videos in the wild. arXiv preprint arXiv: 1212.040,, 2012.
Yang, D., Huang, S., Lu, C., Han, X., Zhang, H., Gao, Y., Hu, Y., and Zhao, H. Vript: A video is worth thousands of words. Advances in Neural Information Processing Systems, 37:57240 57200057261112 2024b.
Wang, P., Bai, S., Tan, S., Wang, S., Fan, Z., Bai, J., Chen, K., Liu, X., Wang, J., Ge, W., Fan, Y., Dang. K., Du, M., Ren, X., Men, R., Liu, D., Zhou, C., Zhou, J., and Lin, J. Qwen2-vl: Enhancing vision-language model's perception of the world at any resolution. arXiv preprint arXiv:2409. 12099.122200 12191, 2024a.
Yao, Y., Yu, T., Zhang, A., Wang, C., Cui, Zhu, H., Cai, T., Li, H., Zhao, W., He, Z., et al. Minicpm-v: A gpt-4v level milm on your phone. arXiv preprint arXiv:2408.0I1 8000, 2024.
Wang, W., He. Z., Hong, W., Cheng, Y., Zhang, X., Qi, Gu, X., Huang, S., Xu, B., Dong, Y., et al. Lvbench: An extreme long video understanding benchmark. arXiv preprint 2006v 08035, 2024b.
Ye, J., Xu, H., Liu, H., Hu, A., Yan, M., Qian, Q., Zhang, J., Huang, F., and Zhou, J. mplug-ow13: Towards long understanding in multi-modal large lan- guage models. arXiv preprint 0044404008.04440 2024.
12

---

Scaling Video-Language Models to 10K Frames via Hicrarchical Differential Distillation
Ye, Q. Xu, H., Xu, G., Ye, J., Yan, M., Zhou, Y., Wang. J. Hu, A., Shi, P., Shi, Y., et al. mplug-owl: Modulariza- tion empowers large language models with multimodality. 2023.
Zhi, Y., Tong, Z., Wang, L., and Wu, G. Mgsampler: An plainable sampling strategy for video action recognition. In Proceedings of the IEEE/CVF International conference on Computer Vision, pp. 1513-1522, 2021.
arXiv preprint arXiv:2304.14111
Yi, K., Gan, C., Li, Y., Kohli, P., Wu, J., Torralba, A., and Tenenbaum, J. B. Clevrer: Collision events for video representation and reasoning. arXiv preprint arXiv:1910.01442, 2019.
Zhou, J., Shu, Y., Zhao, B., Wu, B., Xiao, S., Yang. X., Xiong, Y., Zhang, B., Huang, T., and Liu, Z. Mlvu: A comprehensive benchmark for multi-task long vidco understanding. arXiv preprint arXiv:2406. 04264, 2024.
Zala, A., Cho, J., Kottur, S., Chen, X., Oguz, B., Mehdad, Y., and Bansal, M. Hierarchical video-moment retrieval and step-captioning
Proceedings of the IEEE/CVF Conference Computer Vision and Pattern Recognition, pp. 23056-2363 23065, 2023.
Zhai, X., Mustafa, B., Kolesnikov, A., and Beyer, L. Sig- moid loss for language image pre-training. In Proceed- ings of the IEEE/CVF International Conference on Com- puter Vision, pp. 11975-11988 2023.
Zhang. H., Li, X., and Bing, L. Video-Ilama: An instruction- tuned audio-visual language model for video understand- ing. arXiv preprint wxXiv:2306.02858, 2023. URL https:/
/tttps://axivv.../ otg/ http://ss/2233.00. 02858.
Zhang, P., Zhang, K., Li, B., Zeng. G., Yang. J., Zhang, Y., Wang. Z., Tan, H., Li, C., and Liu, Z. Long con- text transfer from language to vision. arXiv preprint 27X1v:2406.1685,, 2024a.
Zhang, P., Zhang, K., Li, B., Zeng, G., Yang, J., Zhang, Y., Wang, Z., Tan, H., Li, C., and Liu, Z. Long context transfer from language to vision, 2024b. URL https: //a //arxiv. orro/rggaao abs/2aa// /2w. 12446.16600 16852.
Zhang, P., Zhang, K., Li, B., Zeng, G., Yang, J., Zhang, Y., Wang, Z., Tan, H., Li, C., and Liu, Z. Long con- text transfer from language to vision. arXiv preprint arXiv:2406. 16852, 2024c. URL attps://arxit org/abs/ aalaaab/......0 16852....
Zhang, R., Gui, L., Sun, Z.., Feng, Y., Xu, K., Zhang, Y., Fu, D., Li, C., Hauptmann, A.. Bisk, Y., et al. Di- rect preference optimization of video large multimodal models from language model reward. arXiv preprint arXiv:2404.01258, 2024d.
Zhang, S., Fang, Q. Yang, Z., and Feng, Y. Llava-mini: Efficient image and video large multimodal models with one vision token, 2025. URL. https::///// http://axxvvvoo// abs/2501. 03895.
Zhang, Y., WU, J., Li, W. Li, B., Ma, Z., Liu, Z., and Li, C. Video instruction tuning with synthetic data. arXiv preprint arXiv:2410.027 2024e.
13

---

Scaling Video-Language Models to 10K Frames via Hicrarchical Differential Distillation
A. Experiment Details
A.1. Benchmarks
LVBench (Wang et al., 2024b) consists of publicly sourced long videos with an average duration of approximately 101 seconds, significantly longer than existing datasets. The videos are categorized into six major categories and 21 subcategories, covering a wide range of topies such sports, live streams, TV shows, documentaries, and animations. The dataset is annotated through combination of manual effort and model assistance.
EgoSchema (Mangalam et al., 2023) is diagnostic benchmark aimed at assessing the long-form video-language understanding capabilitiies of modern multimodal systems. Derived from the Ego4D dataset, EgoSchema consists of over 5000 human-curated multiple-choice question-answer pairs, covering more than 250 hours of high-quality egocentric video data (180 seconds in average). The dataset is designed to require models to understand and reason over extended temporal contexts, with each question based on a thrce-minute video clip.
Long VencoBenhh (Wu et al., 2024) comprises 3,763 videos with varying lengths, ranging from seconds to 1 hour, and 6,678 numan-annotated multiple-choice questions. The videos cover wide range of themes, including movies, news, life and knowledge. The benchmark introduces a novel task called "referring reasoning" address the single-frame bias in existing video understanding metrics. This task requires models to process more frames improve performance.
MLVU (Zhou et al., 2024) consists of 2,593 samples across 9 task categories, designed to comprehensively assess MLLMs' capabilities in long video understanding. The tasks are categorized into holistic LVU (e.g., topic reasoning, anomaly recognition), single-detail LVU (e.g., needle question answering, ego reasoning). and multi-detail LVU (e.g., action order, action count), including both multiple-choice and free-form generation formats. Videos range from 3 minutes to over hours, with an average length of 651 seconds.
VideoMME (Fu et al., 2024a) consists of 900 videos spanning primary visual domains (Knowledge, Film & Television, Sports Competition, Artistic Performance, Life Record, and Multilingual) with 30 subfields. The videos vary length from 1l seconds to I hour, covering short, medium, and long durations. Each video is annotated with 3 multiple-choice questions, resulting in total of 2,700 question-answer pairs. It also includes subtitles and audio tracks for 744 and 900 videos.
A.2. Baselines
A.2.1. PROPRIETARY VISION-LANGUACE MODELS
GPT4-V (Achiam et al., 2023) is advanced multimodal AI model developed by OpenAI, combining the power of natural language processing with visual understanding. Building the capabilities of GPT-4, GPT-4-V integrates vision-based functionalitics, enabling it to generate responses based on both text and visual inputs.
GPT4-o (OpenAI, 2024) is the latest iteration in the Generative Pre-trained Transformer series. Building on the strengths of its predecessors, GPT-4-o offers enhanced capabilities in understanding multimodal inputs.
Gemini-L.5-Pro (Reid et al., 2024) is Google's latest breakthrough in artificial intelligence, representing a significant leap in multi-modal and long-context understanding.
A.2.2. OPEN-SOURCE MULTI- VISION-LANGUAC MODELS
LLaVA-OneVision (Li et al., 2024b) leverages strong transfer learning across different modalities, pushing performance boundaries in tasks such as video understanding and complex reasoning through unified model architecture.
Oryx-1.5 (Liu et al., 2024d) is a novel multi-modal large language model (MLLM) designed for on-demand spatial- temporal understanding of diverse visual inputs, including images, videos, and 3D scenes. It processes visual inputs at arbitrary resolutions and temporal lengths through pre-trained OryxViT model and a dynamic compressor module.
InternVI VL Series (Chen et al., 2024c;b) is characterized by its large-scale visual encoder, Intern ViT-6B, and language middleware, LLLaMA, which enables to handle wide range of lasounl tasks. Intern VI. 2.5 is the latest version
14

---

Scaling Video-Language Models to 10K Frames via Hicrarchical Differential Distillation
in the Intern VL series, and it significantly enhances its performance through improvements in training and testing strategies, well data quality. The model demonstrates outstanding performance across multiple benchmarks.
Chat-Univi (Jin et al., 2024) uses of dynamic visual tokens to represent images and videos in a unified framework, allowing it to capture spatial details of images and temporal relationships of videos efficiently. Additionally, the model employs a multi-scale representation that enables to perceive both high-level semantic concepts and low-level visual details.
MiniCPM-v2.6 (Yao et al., 2024) achieves strong performance comparable to GPT-4V while being optimized for efficiency. has high-resolution image perception, strong OCR capabilities, multilingual support for over languages, and low hallucination rates, making it suitable for real-world applications with limited computational resources.
mPLUG-OW13 (Ye et al., 2024) is a versatile multi-modal large language model designed to enhance long image-sequence understanding. It introduces Hyper Attention blocks, which integrate vision and language efficiently by allowing parallel cross-attention and self-attention within the transformer architecture.
Qwen2-VI. VL (Wang et al., 2024a) is an advanced vision-language model that enhances the perception of visual information at any resolution. Qwen2-VL introduces Naive Dynamic Resolution mechanism, which allows the model to process images of varying resolutions into different numbers of visual tokens, and the Multimodal Rotary Position Embedding (M-RoPE), which effectively fuses positional information across text, images, and videos.
NVILA (Liu et al., 2024e) builds on VILA and enhances its architecture by scaling up spatial and temporal resolutions before compressing visual tokens, enabling efficient processing of high-resolution images and long videos. Additionally, NVILA incorporates system-wide optimizations, including dataset pruning, FP8 training, and specialized inference engines, to achieve significant improvements in training speed, memory usage, and inference efficiency, while maintaining competitive performance across various benchmarks.
A.2.3. OPEN-SOURCE VIDEO VISION-LANGUAGE MODELS
LLaVA Video (Zhang et al., 2024e) is trained on high-quality synthetic dataset LLaVA- Video- 178K. It processes frames using an optimized video representation technique called LLaVA- Video SlowFast, which allows for more detailed temporal understanding compared to previous models.
LLaMA-VID (Li al., 2024d) represents each frame with two distinct tokens-a context token that captures the overall image context based user input, and a content token that encapsulates visual details. This dual-token strategy significantly reduces the computational burden of processing long videos while preserving critical information, enabling VLMS to handle hour-long videos.
Video-XL (Shu al., 2024) leverages the inherent key-value sparsification capability of large language models to condense visual inputs through a new special token called the Visual Summarization Token (VST). This approach enables the model to handle long videos efficiently by summarizing visual information into compact representations, while dynamic compression and curriculum learning strategies further enhance its performance and training effectiveness.
Video-LLaVA (Lin et al., 2023) uses LanguageBind encoder to align visual signals from images and videos into a unified visual feature space, followed by joint training of images and videos to enhance multi-modal interactions.
Lad2 (Cheng et al., 2024) is designed to enhance spatial-temporal modeling and audio understanding in video and audio-oriented tasks. introduces a Spatial-Temport Convolution (STC) connector that effectively captures the intricate spatial and temporal dynamics of video data, and an integrated Audio Branch that enriches the model's multimodal understanding capabilities by seamlessly incorporating audio cues.
VideoChat2 (KunChang et al., 2023) utilizes the UMT-L visual encoder, which is specifically designed for spatiotemporal representation learning. also adopt Q-former to compresses redundant visual tokens into fewer tokens and aligns them
15

---

Scaling Video-Language Models to 10K Frames via Hierarchical Differential Distillation
with text tokens. VideoChat2 excels in tasks that require understanding of temporal sequences, such as action prediction, action sequence, and moving direction.
Video-CCAM (Fei et al., 2024) is designed to enhance video-language understanding for both short and long videos. It uses Causal Cross-Attention Masks (CCAMs) within the cross-attention layers, which allows the model to effectively process large number of visual tokens while preserving temporal order in videos.
PLLaVA (Xu et al., 2024) is novel and efficient approach to adapt existing image-language pre-trained models for dense video understanding. PLLaVA VA employs a simple yet effective pooling strategy to smooth the feature distribution along the temporal dimension, reducing the impact of high-norm visual features that often lead to performance degradation.
Movie-Chat (Song et al., 2024) model, which employs a combination of short-term and long-term memory to efficiently process and understand long videos.
introduces innovative memory mechanism inspired by the Atkinson- memory
Kangaroo (Liu et al., 2024c) is designed for long-context video input. Kangaroo addresses the challenges of limited training data and excessive compression of visual features in long videos by developing a data curation system to build a large-scale high-quality dataset for vision-language pre-training and instruction tuning. It also employs curriculum training pipeline that gradually increases the resolution and number of input frames enhance the model's ability to process long videos.
LongVU (Shen et al., 2024) introduces spatiotemporal adaptive compression mechanism designed for long video- language understanding. LongVU leverages cross-modee queries and inter-frame dependencies reduce the number of video tokens while preserving visual details, making it possible to process hour-long videos within the context length constraints of large language models (LLMs).
LA (Zhang et al., 2024c) leverages a unique approach called "long context transfer," where it extends the context length of language model by training it on longer text data and then aligns this extended model with visual inputs using short image data. This enables Long VA to process over 2,000 frames or more than 200K visual tokens without explicit long video training.
LongVILA (Anonymous, 2024) adopts a five-stage training pipeline that incorporates multi-modaa alignment, large-scale utilizes a novel
pre-training, supervised fine-tuning, context extension, and long video fine-tuning. Additionally, Multi-Modal Sequence Parallelism (MM-SP) system, which efficiently parallelizes training and inference for long videos.
A.3. Implement Details
Stage
Datasets
Size
Pretrain
WebVid (Bain et al., 2021), Intern (Wang et al., 2023), ShareG- 7.4M
PTVVide (Zhang et al., 2024d), OpenVid (Nan et al., 2024), Vript (Yang et al., 2024b)
Short Video Understanding
VideoChat2-IT (KunChang al., 2023), Tas (Jia et al., 2022), 1.3M CLEVRER (Yi et al., 2019), LLaVA- Video- 178K (Zhang et al., 2024e), MovieChat (Song et al., 2023), Perception (Patraucean et al., 2023), STAR (Wu & Star, 2021), NExTQA (Xiao et al., 2021)
Longer Video Understanding
Fine Video (Farré et al., 2024), CinePile (Rawal et al., 2024)
500K
Training Datasets We employ three-stage training strategy fine-tune VILAMP and enhance its video comprehension capabilities. During the pretraining phase, the primary objective is to adapt VILAMM to the interleaved input structure com- bining keyframes and non-keyframess Our training data comprises approximately 7.4M video-caption pairs collected from multiple open-source datasets including Web Vid (Bain et al., 2021), Intern (Wang et al., 2023), ShareGPT (Zhang et al., 2024d), OpenVis Vid (Nan et al., 2024), and Vript (Yang et al., 2024b), ensuring broad coverage of diverse visual scenarios. This comprehensive collection trains the model develop a foundational understanding of video inputs.
16

---

Scaling Video-Language Models to 10K Frames via Hicrarchical Differential Distillation
Additionally, we further expand the data scale and enhance the model's capabilities by incorporating diverse types of QA datasets, including VideoChat2-IT (KunChang et al., 2023), EgoTaskQA (Jia et al., 2022), CLEVRER (Yi al., 2019), LLaVA-Video-I78k (Zhang et al., 2024e), MovieChat (Song et al., 2023), PerceptionTest (Patraucean et al., 2023), HiREST (Zala et al., 2023). STAR (Wu & Star, 2021), and NExTQA (Xiao et al., 2021). ultimately constructing a dataset of 1.3M samples. This dataset improves the model's Dnstructionffollwwii downstream tasks. Specifically, for LLLVA-Video- 178K, we utilize the academic and youtube subsets.
ability, enabling it to better adapt to various
Finally, to improve the model's performance on longer videos, we incorporate Fine Video (Farré et al., 2024) and CinePile (Rawal al., 2024), two datasets with longer average video durations (280s and 160s, respectively), and get approximately 500K training samples.
Training Parameters As mentioned in $ B.2, we configure = 1 and T 0.85 to achieve balanced effectiveness. During training, we set learning rate to 2 x 10 for the visual encoder and 10-5 for the remaining components. We use adamW as the optimizer. The optimization follows cosine learning rate scheduler with a warmup ratio of 0.03. We set the batch size to and the gradient accumulation steps to 4. We train the model for epoch, which costs approximately two weeks with 32 NVIDIA A100 GPUs.
B. Discussions
B.1. Computational Efficiency Result
We compare against four representative video-language models: LongVA, LLaVA-Video, LLaMA-VID, and VideoChat- Flash. As shown in Table 5, VILAMP demonstrates significant advantage in memory efficiency. Unlike typical models whose memory usage increases drastically with input length, VILAMP shows only a marginal increase in memory demand as the input length grows. This characteristic enables VILAMP to handle extremely long inputs under limited memory constraints, which can be attributed to the effective removal of redundant information through our differential distillation. Furthermore, VILAMP exhibits slow growth rate FLOPs with input expansion. demonstrating remarkable superiority in handling long sequences. When the input length reaches the thousand-frame level, VILAMP outperforms all baseline models, requiring merely 18.4% of the FLOPs consumed by VideoChat-Flash when processing 8,192 frames.
Moreover, VILAMP also demonstrates a significant speed advantage, with TTFT results indicating that it exhibits a response latency that is merely 50% or less of typical models. When the number of frames reaches 8,192, VILAMP exhibits marginally slower performance compared to VideoChat-Flash This is attributed to the pruning operations implemented in the Large Language Model (LLM) component of VideoChat-Flashh which specifically optimizes inference speed. Nevertheless, VILAMP achieves comparable performance levels without employing such optimizations.
B.2. Influence of Hyperparameter Configuration
We conduct a series of experiments under various settings of a and r. As reported Table 6, the overall performance of the model first improves and then decreases as 7 increases. This trend can be attributed to the fact that a too small 7 leads to insufficient sampling, while an excessively large results in overly concentrated sampling, both of which negatively impact the model's effectiveness. We find that setting 7 0.85 provides a robust balance between semantic preservation and temporal diversity. Further analysis of reveals similar pattern. To some extent, reducing a allows the weight distribution to focus more on the target tokens, which helps in eliminating redundant information. However, when is too small, the weights concentrate on only few tokens, leading to information loss that adversely affects the model's performance. Through extensive experimentation, we find that setting a = 10-2 ensures robust performance across diverse video content.
17

---

Scaling Video-Language Models to 10K Frames via Hicrarchical Differential Distillation
Table 5. Computational efficiency comparison across different input lengths.
Model
FLOPs (T)
Memory (MB)
TTFT (ms)
Input 64 Frames
Long VA
316.0
62,262
798
VA-Viee
101.8
32.362
203
ALaMA-VID
112.3
46,402
373
VideoChat-Flash
29.6
18,688
521
VILAMP
61.0
23,201
80
Input 256 Frames
Long VA
2,946.3
OOM
00M
LLaVA-Video
1,221.6
71,495
535
LLaMA-VID
245.9
47,294
492
VideoChat-Flash
113.3
22,665
586
VILAMP
120.3
23,647
157
Input 1,024 Frames
LongVA
27,605.6
0OM
0OM
18,324.0
0OM
0OM
LLaMA-VIL
1,045.3
48,678
785
489.6
38,427
694
VILAMP
356.8
24,339
462
Input 8, 89192 Frames
LongVA
2,153,901.7
0OM
0OM
LLa LLaVA-Video
1,643,892.3
00M
00M
LLaMA-VID
20,906.0
91,438
5,080
ChatoChat
13,896.6
92,034
2,730
VILAMP
2,565.2
45,819
3,374
Table 6. Performance varying with hyperparameters and
LongVideoBench
MLVU
Video-MME
0.35
55.6
65.9
59.4
0.50
58.3
69.5
61.2
1
0.85
58.9
70.5
62.7
1.00
51.3
64.5
58.0
0.35
57.2
63.3
60.5
0.50
58.0
68.9
62.6
10
0.85
59.1
71.5
64.5
1.00
51.1
63.6
57.3
0.35
56.8
65.3
60.2
0.50
59.6
69.7
63.2
10
0.85
60.5
72.6
65.1
1.00
50.9
62.3
58.8
0.35
49.3
61.6
45.6
0.50
52.2
63.4
47.0
10
0.85
54.4
65.6
48.2
1.00
46.5
60.9
43.6
18


Reviews:
[{"review_id": "SAxvY0koKT", "reviewer": "Reviewer_uDcJ", "summary": "This paper introduces a new long-video LLM framework called ViLaMP to incorporate a differential distillation mechanism to compress the redundant video frame sequence's information into a more compact representation for better trade-off between performance and computational cost. Specifically, this work mainly designs a differential keyframe selection module and a differential feature merging module to compress the video features by the principle of maximizing query relevance while minmizing context redundancy. Extensive experiments and in-depth analyses demonstrate the motivation and effectiveness of the proposed method.", "questions": "", "limitations": "", "rating": 4, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "> Although this method's effectiveness has been verified on video question answering benchmarks for multimodal large language models, it is still unclear whether this method can truly preserve necessary information for more temporal-sensitive tasks like action recognition, video grounding or temporal reasoning. I suggest the authors to add some experiments and anslyses on this aspect.\n\n> There have been many existing frame sampling research in video understanding domain such as [1, 2] that can serve as reasonable baselines for comparative analyses. I suggest the authors to add some comparisons with these frame sampling methods.\n\nThank you for your valuable feedback. We have conducted additional experiments in response to your suggestions, as detailed below:\n\n**For the Video Grounding task**, we performed experiments on the widely used QVHighlights[1] benchmark (267 citations). This dataset provides annotations for videos, marking the timestamps of query-relevant segments and the positions of keyframes. We evaluated ViLAMP using the Hit@k metric, which measures the probability of the ground-truth keyframe appearing in the top-*k* keyframes selected by our method. The results are as follows:\n\n| **Hit@1** | **Hit@5** | **Hit@10** | **Hit@30** |\n|:---------:|:---------:|:----------:|:----------:|\n|    26.1   |    47.3   |    60.0    |    83.2    |\n\nAs shown, ViLAMP achieves 60% accuracy for Hit@10 and 83% for Hit@30, providing strong empirical support for its Dynamic Keyframe Selection (DKS) process. The current method effectively captures query-relevant video clips.\n\n**For the action recognition task**, we incorporated the two baselines suggested: **Adaframe** and **MGSampler**, and evaluated them alongside ViLAMP on three benchmarks: ActivityNet[2], Something-Something V2 (Sth-V2)[3], and UCF-101[4]. Since ViLAMP is a generative model, we reformulated the original classification task as a multiple-choice problem during testing. The results are as follows:\n\n|           | **ActivityNet (mAP %)** | **Sth-V2 (Acc. %)** | **UCF-101 (Acc. %)** |\n|:---------:|:---------------:|:----------:|:-----------:|\n|  Adaframe |       71.5      |      -     |      -      |\n| MGSampler |        -        |    60.1    |     95.2    |\n| ViLAMP (Ours)|     **78.6**    |  **86.9**  |   **97.6**  |\n\nThe results for Adaframe and MGSampler are sourced from their papers. Notably, although ViLAMP was not trained on these datasets, it demonstrates robust performance, significantly outperforming both baselines. This may be attributed to the generalizable knowledge acquired by the large-scale pretrained model, enabling effective zero-shot transfer to action recognition tasks.\n\nWe will incorporate these results into our revised draft. Please let us know if you have any further questions or suggestions.\n\n[1] Lei J, Berg T L, Bansal M. Detecting moments and highlights in videos via natural language queries[J].\n\n[2] Caba Heilbron F, Escorcia V, Ghanem B, et al. Activitynet: A large-scale video benchmark for human activity understanding[C]\n\n[3] Goyal R, Ebrahimi Kahou S, Michalski V, et al. The\" something something\" video database for learning and evaluating visual common sense[C]\n\n[4] Soomro K, Zamir A R, Shah M. UCF101: A dataset of 101 human actions classes from videos in the wild[J].", "claims_and_evidence": "The claims are empirically supported by experiments and observations.", "methods_and_evaluation_criteria": "This method is useful in improving the capability of current video LLMs to process longer video inputs.", "theoretical_claims": "None.", "experimental_designs_or_analyses": "The experimental designs and analyses are intuitively reasonable.", "relation_to_broader_scientific_literature": "None.", "essential_references_not_discussed": "None.", "other_strengths_and_weaknesses": "Strengths:\n\n1.The paper is well-written and easy to understand.\n\n2.Abundant experimental results and empirical analyses are included in the manuscript, which make the motivation and method design clearer. The basic assumption of the method related to query relevance and context uniqueness for compression is reasonable.\n\n3.One of the advantage of this method is that both of the frame selection and patch merging modules are learnable to some extent since they rely on the feature-based intermediate scores for operation. This is more flexible compared to previous hard selection strategy.\n\n4.Experiments show that the method is able to achieve competitive trade-off between computational cost and model performance, and its performance in long-form video scenarios seems compelling.\n\nWeaknesses:\n\n1.Although this method's effectiveness has been verified on video question answering benchmarks for multimodal large language models,  it is still unclear whether this method can truly preserve necessary information for more temporal-sensitive tasks like action recognition, video grounding or temporal reasoning. I suggest the authors to add some experiments and anslyses on this aspect.\n\n2.There have been many existing frame sampling research in video understanding domain such as [1, 2] that can serve as reasonable baselines for comparative analyses. I suggest the authors to add some comparisons with these frame sampling methods.\n\nReferences\n\n[1] Adaframe: Adaptive frame selection for fast video recognition, In CVPR 2019.\n\n[2] MGSampler: An Explainable Sampling Strategy for Video Action Recognition, In ICCV 2021.", "other_comments_or_suggestions": "Please refer to strengths and weaknesses.", "questions_for_authors": "Please refer to strengths and weaknesses.", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {"ethical_review_concerns": "None."}}, {"review_id": "ql5gU3X5Ko", "reviewer": "Reviewer_3pJA", "summary": "This paper presents a novel video token compression strategy named Hierarchical Differential Distillation, designed for long video understanding. Specifically, the hierarchical differential distillation process is achieved through two key mechanisms: Differential Keyframe Selection (DKS) and Differential Feature Merging (DFM). Building upon this strategy, the paper introduces the VILAMP model. Experimental results demonstrate that VILAMP achieves superior performance across five common video understanding benchmarks.\n\n##Update after rebuttal: I appreciate the authors' detailed response, most of my concern is settled. I will keep the original rating as \"Accept\".", "questions": "", "limitations": "", "rating": 4, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for your time and effort, here are our replies:\n\n> Q1: Do the authors consider comparing LongVU with some training-free token compression methods[1-4]?\n\nSince VisionZip[1], FasterV[2], and FastV[4] are based on the LLaVA-NeXT model, which is optimized for single image scenarios, they cannot be directly applied to video input. Due to time constraints, we only reproduce DyCoke[3], and the results are as follows:  \n\n|        | **EgoSchema** | **MLVU** | **Video-MME** |\n|:------:|:-------------:|:--------:|:-------------:|\n| DyCoke |   **69.8**    |   63.2   |      57.9     |\n| LongVU |      67.6     | **65.4** |    **60.6**   |  \n\nWe will include these results in our revised draft.  \n\n> Q2: The DKS presets a fixed number of keyframes K, which may not be suitable for all videos. Intuitively, videos with less content variation may require fewer keyframes, while more dynamic and content-rich videos would benefit from selecting more keyframes.\n\nThank you for your valuable feedback. We would like to clarify that the parameter K serves as an upper boundary for the number of keyframes rather than a required quantity to achieve. In practical applications, when processing videos with less content variation, the DKS algorithm may automatically select fewer keyframes than the specified K value, as outlined in line 4 of Algorithm 1 (Page 5).\n\nAs demonstrated in Figure 6, which illustrates the impact of K on model performance, we observe that the model's performance tends to saturate when K exceeds 32. Taking into account both the model's performance and the length constraints of the base model, we ultimately set K to 32.\n\n> Q3: VILAMP requires model finetuning, which may limit its portability and practical value.\n\nIndeed, ViLAMP requires finetuning. This is primarily for performance considerations. Training-free methods such as FastV, PruMerge, and DyCoke can only achieve performance comparable to the base model. We finetune ViLAMP with additional data to attain superior results, enabling it to better adapt to compressed video inputs.\n\n> Q4: The hierarchical differential distillation relies on multiple hyperparameters, and as shown in Table 4 of the supplementary material, these hyperparameters significantly impact the model’s performance.\n\nAs mentioned in Q3, our method does involve finetuning that introduces certain hyperparameters. We fully acknowledge the potential impact of hyperparameter selection on model performance and have conducted systematic experiments and corresponding analyses to offer practical guidance for users. The results demonstrate that the model maintains stable performance with slight deviations from the preset optimal values, and significant performance degradation only occurs under extreme parameter configurations.\n\n> Q5: Why does DFM choose to compress non-keyframes into a single token? Would increasing the number of retained tokens during DFM improve performance?\n\nThis is an important question. In our pilot study, we explored various configurations by compressing non-keyframes into 1 token, 3 tokens, and 9 tokens, respectively. \n\nThe result shows that using more tokens to represent non-keyframes do not yield performance improvements. We attribute this observation to two main reasons: (1) Useful patches are sparse in frames. As shown in Table 4, DFM’s parameter $\\alpha$ achieves the best performance when set to $10^{-2}$. This indicates that the patch scores are amplified by 100 before the Softmax operation (see Eq. 10). Such sharpening concentrates the weights on a small subset of patches, suggesting that focusing on only a few critical patches is sufficient for achieving desirable results. (2) Keyframes already contain sufficient details, while non-keyframes primarily serve to complement missing information. In this case, increasing the number of tokens for non-keyframes may not lead to noticeable gains.\n\nTherefore, for efficiency considerations, we ultimately compressed non-keyframes into a single token. We will include the discussion ablve in our revised draft.\n\n> Q6: Is it possible to adaptively select the number of keyframes?\n\nAs mentioned in Q2, our current DKS keyframe selection algorithm employs a dynamic selection mechanism, though it currently operates with a fixed upper limit K for the maximum number of keyframes. To enhance model adaptability across diverse video inputs, we plan to develop a dedicated module in subsequent experiments. This module will dynamically calculate the optimal K value by considering inter-frame relevance and variation, and video duration, thereby enabling more intelligent adaptation.", "claims_and_evidence": "Yes. This paper claimed that the DKS strategy can select keyframes by maximizing query relevance while ensuring temporal uniqueness, whereas the DFM strategy preserves query-relevant features while suppressing redundant visual information, thereby merging non-keyframes.", "methods_and_evaluation_criteria": "Yes. The evaluation benchmarks utilized in this paper include the most common video understanding benchmarks, including ActivityNetQA, Egoschema, LongVideoBench, MLVU, and Video-MME.", "theoretical_claims": "This paper does not make theoretical claims.", "experimental_designs_or_analyses": "The experimental analysis covers a comparison between state-of-art approaches, ablation studies among components, and keyframe numbers. And more experiments are provided in the supplementary materials. I have no objection to this part.", "relation_to_broader_scientific_literature": "This paper presents two novel strategies for video token compression, which hold the potential to advance not only VLMs but also the broader field of video processing.", "essential_references_not_discussed": "Do the authors consider comparing LongVU with some training-free token compression methods[1-4]?\n\n[1] VisionZip: Longer is Better but Not Necessary in Vision Language Models (2024)\n\n[2] [CLS] Attention is All You Need for Training-Free Visual Token Pruning: Make VLM Inference Faster (2024)\n\n[3] DyCoke : Dynamic Compression of Tokens for Fast Video Large Language Models (2024)\n\n[4] An Image is Worth 1/2 Tokens After Layer 2: Plug-and-PLay Acceleration for VLLM Inference (2024)", "other_strengths_and_weaknesses": "### Strength\n1. The ideas behind the DKS and DFM strategies are considered novel.\n2. Experimental results demonstrate that VILAMP achieves superior performance across multiple benchmarks.\n\n### Weakness\n1. The DKS presets a fixed number of keyframes K, which may not be suitable for all videos. Intuitively, videos with less content variation may require fewer keyframes, while more dynamic and content-rich videos would benefit from selecting more keyframes.\n2. VILAMP  requires model finetuning, which may limit its portability and practical value.\n3. The hierarchical differential distillation relies on multiple hyperparameters, and as shown in Table 4 of the supplementary material, these hyperparameters significantly impact the model’s performance.", "other_comments_or_suggestions": "It would be better to provide a more detailed derivation and analysis of the computational complexity of the DKS strategy.", "questions_for_authors": "1. Why does DFM choose to compress non-keyframes into a single token? Would increasing the number of retained tokens during DFM improve performance?\n\n2. Is it possible to adaptively select the number of keyframes?", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "pMktU66C9s", "reviewer": "Reviewer_VLKM", "summary": "Long video processing needs to exploit spatio-temporal redundancies. The core idea of this paper is hierarchical differentiation by key-frame selection and non-keyframe compression leading to processing abiility upto 10K frames on a single GPU and shows competitive performance on long-video benchmarks.", "questions": "", "limitations": "", "rating": 4, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for your valuable feedback. Here is our reply to your main concern:\n\n> Can the authors shed some lights of its possible extension to a streaming framework?\n\nWe sincerely appreciate the reviewer’s insightful question. While our proposed DKS and DFM processes significantly reduce token counts, enabling ViLAMP to handle long video inputs, the current architecture does not yet support streaming videos where input length increases incrementally over time. However, we believe our method can be adapted to streaming scenarios through some adjustments. One feasible solution involves:\n\n- **Keyframe Candidate Pool**: A candidate pool for keyframes is maintained. When a new frame arrives, it is identified as a keyframe based on two criteria: (1) Its similarity to the query exceeds a threshold. (2) Its similarity to existing keyframes in the pool is below a threshold. If both conditions are met, the frame is added to the pool, and an existing keyframe is evicted based on criteria such as (a) query similarity, (b) similarity to the newly inserted frame, or (c) the frame’s retention duration in the pool.\n\nWhile this approach theoretically supports streaming scenarios, several implementation challenges require further exploration. A critical issue lies in kv-cache management: Modifying historical information (e.g., updating keyframes) currently does not update the kv-cache, leading to an approximate rather than exact kv-cache during inference. This approximation may impact performance, and rigorous empirical validation is necessary to quantify its effects. We plan to address these nuances in future work, with a focus on efficient cache updates and experimental validation.", "claims_and_evidence": "Claims and evidences are clear.", "methods_and_evaluation_criteria": "Results shown on 5 different benchmarks.", "theoretical_claims": "NA", "experimental_designs_or_analyses": "I liked the way the experimental choices had been made.", "relation_to_broader_scientific_literature": "Spatio-temporal redundancy is the key to process long-video, be it video compression or video classification or video-llm, any kind of model and application.", "essential_references_not_discussed": "The paper restricted their discussions and comparative performances for 7-8B models, not beyond that. I believe this is ok.", "other_strengths_and_weaknesses": "I think this is smart to do the frame-level and patch-level analysis on 4 video-llm models on Video-MME benchmark. The paper proposes an efficient scaling mechanism of video-llm models for long-videos, and results shown on different benchmarks show that the redundancy exploitation techniques motivated by intermediate results from Video-MME benchmark also hold true for other benchmarks.  \n\nThe only shortcoming of this method is that its differential keyframe selection may not easily be extended for even larger videos. It assumes offline processing. Can the authors shed some lights of its possible extension to a streaming framework?", "other_comments_or_suggestions": "It's a nicely written paper, easy to understand and follow.", "questions_for_authors": "Clearly written paper, self-explanatory.", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Meta-review: This paper works on improving the high computational costs for long-form video processing in VLMs. It receives unanimous positive reviews due to 1) the novel idea; 2) extensive experiments; 3) good writing. Although there are some concerns at the beginning, including 1) missing experiments on more temporal-sensitive tasks; 2) missing comparison with some related works; 3) the fixed number of keyframes K may not be suitable for all videos; 4) the fintuening of VILAMP may limit its practical value; 5) the hyperparameters. Most of they were well addressed by the rebuttal, and all reviewers are happy in the end. The AC finds no reason to disagree with the reviewers' recommendation, and thus recommends acceptance. The authors should follow the reviewers' feekback and reflect the rebuttal in the final version.
Decision: Accept (poster)