VideoChat-Flash: Hierarchical Compression for Long-Context Video Modeling
Authors: Xinhao Li
Yi Wang
Jiashuo Yu
Xiangyu Zeng
Yuhan Zhu
Haian Huang
Jianfei Gao
Kunchang Li
Yinan He
Chenting Wang
Yu Qiao
Yali Wang
Limin Wang
Venue: ICLR 2026 Poster (2026)
Keywords: long video understanding
video language model
Primary area: foundation or frontier models, including LLMs
TL;DR: Comprehensive Solution for Performance-Leading and Highly Efficient Long-Video Understanding Models

Abstract: Long-context video modeling is critical for multimodal large language models
(MLLMs), enabling them to process movies, online video streams, and so on.
Despite its advances, handling long videos remains challenging due to the difficulty
in efficiently understanding the extremely long video context. This paper aims to
address this issue from aspects of the model architecture, training data, training
strategy, and evaluation benchmark. First, we propose a novel Hierarchical video
token Compression (HiCo) method, which leverages visual redundancy in long
videos to compress long video context from Clip-level to Video-level, reducing the
computation significantly while preserving essential details, achieving an extreme
compression ratio of approximately 1/50 with almost no performance loss. Second,
we introduce a multi-stage short-to-long learning scheme, a large-scale dataset of
real-world long videos named LongVid, and a challenging “Multi-Hop Needle-In-
A-Video-Haystack” benchmark. Finally, we build a powerful video MLLM named
VideoChat-Flash, which shows a leading performance on both mainstream long
and short video benchmarks at the 2B and 7B model scales. It first gets 99.1%
accuracy over 10,000 frames in NIAH among open-source models.

Published as a conference paper at ICLR 2026
VIDEOCHAT-FLASH: HIERARCHICAL COMPRESSION FOR LONG-CONTEXT VIDEO MODELING
Xinhao Li Yi Wang lot Jiashuo Yu
Xiangyu Zeng".'
Yuhan Zhu?
Haian Huang! Jianfei Gao! Kunchang Li
Yinan He
Chenting Wang*
Yu Qiao" Yali Wang*. Limin Wang
Shanggaai AI Laboratory *Nanjing University
Shenzhen Institutes of Advanced Technology. Chinese Academy of Sciences
https: /ttps://gite tuub. com cttmm//oeeeniiaaaa/ceeccccaaaaaa
ABSTRACT
Long-context video modeling is critical for multimodal large language models (MLLMs), enabling them to process movies, online video streams, and so on. Despite its advances, handling long videos remains challenging due to the difficulty in efficiently understanding the extremely long video context. This paper aims to address this issue from aspects of the model architecture, training data, training strategy, and evaluation benchmark. First, we propose novel Hierarchical video token Compression (HiCo) method, which leverages visual redundancy in long videos to compress long video context from Clip-level to Video-level, reducing the computation significantly while preserving essential details, achieving an extreme compression ratio of approximately 1/50 with almost performance loss. Second, we introduce a multi-stage short-to-long learning scheme, a large-scale dataset of real-world long videos named LongVid, and challenging "Multi-Hop Needle-In- A Hideo-Haysterd
benchmark. Finally, we build powerful video MLLM named VideoChat Chat-Flash, which shows leading performance on both mainstream long and short video benchmarks at the 2B and 7B model scales. It first gets 99.1 % accuracy over 10,000 frames in NIAH among open-soures models.
INTRODUCTION
Long-context video modeling stands one of the most crucial capabilities within multimodal large language models (MLLMs). This capability empowers MILMS to proficiently manage hours-long movies, documentaries, and online video streams, all of which demand sophisticated long video
processing. Recent advances in MLLMs (Wang et al., 2022; 2024c: Li et al. [025 2024c Zhang al. 2023a; Cheng al. 20224] Zhang et al. 2024c Lin et al. [024] Xu et al. 2024 Li al.
2024a Bavishi et al. [0224] Li et al. (023a) perform well in short video understanding. However, it
remains challenging to build MLLMs for processing extremely long videos (lasting for hours or even
longer). The difficulty lies in how to enable MLLMs to efficiently understand the extremely long
video context brought by long videos.
Inspired by large language models (LLMs) with long context, modeling multimodal long context is widely studied from several perspectives. Some work (Reid al. 2024]] Xue et al.) 2024) represented by Deminii-.5-PPr (Reid et al., 2024) address it by training well- pel--peffrreed MLLMs on long-form corpus e.g. lengthy text and thousands of frames from videos, minimizing the gap between the evaluation and learning. Although the progress in system construction and hardware has made it possible to train and infer with super-long multimodal contexts, such super-long multimodal contexts have significantly reduced the training and inference efficiency of models. (For Gemini- 1. 1.5-Pro (Reid et al., 2024), one-hour video will be converted into 921, 221,600 tokens). Meanwhile, the high redundancy in long video context makes it particularly difficult for models understand. Some previous efforts (Song et al., 2024 Li et al. 2024d: Shen et al., 2024) have been made to compress video tokens in order achieve higher training and inference efficiency for long videos. However, the compression of visual content inevitably leads to the loss of detailed information. In
Equal contribution. Corresponding authors.

---

Published as a conference paper at ICLR 2026
GPT44
Geminil Pro
PiPrevoous SOTA (78 stre
VideoChat Flach (Curg)
1150x
74.0
76.2
71.0
42.9
64.6
68.4
66.7
64.7
65.3
48.2
48.4
25.7
50
MV/ench
Perception Test
LongVideoBench
VideoMME
LVBench
Chardes-STA Temporal Grounding
AuroraCap Caption
TELOPS
Short Video
Long Video
+
w/ 1K Frames
Figure 1: Comparison results on various generic video-linguistic tasks.
some long video understanding benchmarks, certain current long video models even perform worse than some image MLLMs. Therefore, how to strike balance between performance and efficiency remains a significant challenge. In this paper, we attempts to address the above issues from the model architecture, training data, training strategy, and evaluation benchmark.
First, we propose a novel Hierarchical video token Compression method (HiCo) to model the long video context efficiently, which defines the compression of the long video context into two stages. First, we segment the long video into multiple clips. Then, at the Clip-level, we utilize the spatio-temporal attention of the video encoder and the similar token merging aggregate the key information between frames, thereby reducing the redundancy of inter- frame features. Subsequently, we take advantage of the sparsity of attention when the LLM processes long video tokens, further discard the video tokens that are irrelevant to the current task at the Video-level. HiCo could achieve an extreme compression ratio approximately 1/50 with almost no performance loss. Additionally, we have conducted thorough explorations of other designs such as video sampling and timestamp awareness prompt.
Second, to further enrich the existing long video training corpus, we construct LongVid dataset that contains 114,228 long videos and 3,444,849 question- caastion-nnswerigg pairs. With LongVid we have designed: multi-stage training strategy named short-to-long learning. The main idea is to first utilize image and short video data to learn basic visual perception abilitics. Then, through the joint training of short video and long video data, the model is enabled to handle videos of different lengths and different types of tasks. In addition, we design a new evaluation benchmark named "Multi-Hop Needle In A Video Haystack", which more challenging and can better examine the model's complex reasoning abilities regarding long videos.
Finally, we develop powerful video MILM named VideoChat-Flash, as shown in fig. T which achieves remarkably leading performance with extremely high efficiency on various video understand- ing benchmarks. Even with 7B size, it outperforms closed-source models such as GPT-4o (OpenAl 2024) and feminii1.55Ppp (Reid et al. 2024).
2 RELATED WORK
Multimodal Large Language Models for Video Understanding. Recent advancements in multi- modal large language models (MLLMs) have shown significant promise in video understanding. Most
of them (Li et al. 2025 2024c Wang et al. 2024e Lin et al. 2024 Zhang et al., 2023a 2024c d) focus on the understanding of levul videos, and some works (Reid et 2024 Song et al.
2024 Wang et al. 2024d Shen et al. 2024 Xue et al. 2024 Shu et al. 2024 Huang et al. 2024) have further tried handle longer heur-level videos. To address the challenge processing long videos, researchers focus on two key strategies: (1) extending the context window of the LLM (Reid
al., 2024, Zhang et al. 2024a] Xue et al., 2024 Wei Chen 2024 and (2) compressing the video tokens (Li et al.] 2024d Fei et al. 2024; Weng et 2225 lan et al., 2024 [Song et al. 2024 Shu et al. 2024 Zeng et al. 2024). For context extension, although the approach of expanding the context window enable the possibility of long video understanding, it falls short of reducing the high computational burden and processing costs induced by long videos, thereby imposing limitations on its practical application. For token compression, Methods represented by Llama- Vid (Li et al.) 2024d) use a highly compact representation while preserving key information. The high compression ratio makes it difficult for such methods achieve excellent long video understanding performance,
2

---

Published as a conference paper at ICLR 2026
Video Enceder
LLM
Connector
share
share
inber EVTT
MLM Lager
M Lagor
LLM Layer
LLM Layer
claav
Video Encoder
Connector
Timestamp
Prompt Quussiion
Duration based Sampling
Video Encoding Clip-level Compression
Video-level Compression
Output
Figure 2: Framework of VideoChat-Flash with Hierarchical Video Token Compression. Video tokens will be compressed at the Clip-level by leveraging the local redundancy of the video modality during video encoding. Subsequently, during LLM processing, they will compressed at the Video-level by taking advantage of the sparsity in the interaction between the text and the video.
and they may even be inferior to some MLLMs designed for image modeling. Therefore, how to build Video MI.LM that can balance both efficiency and performance remains difficult challenge.
Evaluation of Long Video Understanding. In order to evaluate the ability of Video MLLMs to
understand long videos, previous works (Song et al., 2024 Zhang al.) 2023b) Mangalam al.
2023 Rawal et al., 2024 Wu et al. 2024: Zhou et al. 2024 Fu et al. 2024; Chandrasegaran et al.
2024 Wang et al. 2024c) have achieved this by collecting long videos and then designing various multiple-choiee questions related to the content of these long videos. This approach is closer to real-world applications and can effectively examine the model's ability to understand and reason about long videos. However, when comes examining the model's capabilities for videos of different lengths, this method is not intuitive enough. Inspired by the popular "Needle A Haystack" (NIAH) evaluation long text context evaluation, some recent works (Zhang et al.) 20244] Zhao al. 2024) have attempted NIAH for Video haystack. Nevertheless, it is difficult to assess complex reasoning abilities, and there may be information leakage. In this paper, we propose more challenging "Multi-Har Hop Needle- Meedle-In-A- Video-Hol iaaet-Hayttcck is designed to address the above issues.
3 METHOD
3.1 HICO: EFFICIENT LONG VIDEO MODELING
To enable MLLMs to handle thousands of input frames, propose video context compression paradigm named hierarchical compression (HiCo). This paradigm decomposes video context com- pression into two main stages: Clip-level compression for unimodal visual redundancy during the encoding of long videos. Video-level compression for cross-modal visual redundancy within the context interaction the LLM. Based on this framework, we have designed an innovative efficient Video MLLM architecture, VideoChat-Flas as illustrated in fig. Below, we elaborate on our specific design details from data input model output.
Duration-based Sampling. First, we need to perform frame sampling on the original video. Specifically, we sample raw video with a duration of D obtain T frames as input. Considering that the requirements for understanding short and long videos often differ, we aim to conduct dense sampling on short videos to capture detailed motions and sparse sampling on long videos focus on event understanding. To this end, we have designed Duration-based Sampling strategy:
T = min [ max. max (D, T )m.
Simultaneously, we define the sampling density & as follows:
(T, = T min max; (D, Tmin)
That is, for short videos where < Tmins Φ = /min/ D. which increases as the vidco length decreases. For long videos where D > T Tmax, T maxx / D. which decreases as the video length increases.
3

---

Published as a conference paper at ICLR 2026
Spatio-Temporal Compression Encoding for Clips. Considering the substantial redundant and repetitive information, such as that of backgrounds and objects, present between adjacent frames in natural videos, segment the original video frames into several equally sized clips. Each clip is then encoded using a Video Encoder with Tennoral Attention, which effectively captures both key information and temporal redundancies within the clips, thereby significantly improving compression efficiency. We briefly offer some theoretical explanations here. First, we define the video features within a clip of length 7 Y (Y,, 2:--.. Yr). the compressed feature as Z = C(Y), where C denotes a deterministic compression operation (e.g., average pooling). For commonly used Image Encoder (such as SigLIP (Zhai et al., (023), etc.), they assume inter-frame independence, i.e., I( Y II== p( .)). We can define the information loss caused by compression using the conditional entropy of the original features given the compressed feature:
T
Lime H(Y Ying,
Z
H(Y HZ).
For the Video Encoder, it models the joint distribution p(Y,,, 1. Yr) using spatio-temporal atten- tion, and its information loss be expressed
Iidid
H(Y Y.d.
Y Z) H(YY Y Y (Z)
T
(4)
Σ ( I(Y; Y Y Y - (Z).
where I(Yrid, Y Y
Y represents the mutual information between the t-th frame and the previous t - 1 frames, i... the redundant information between the f-th frame and its preceding frames. For most videos, it is evident that T(Yyd, Yvid Y , Yvid (-11 > 0. Assuming the same (Z) and Σ H(Yimr)= ~ Σi= H(Y)")), we thus have L jmng > L A detailed proof is provided in the Appendix. In terms of specific implementation, we use ToMe (Bolya et al., 2022) as C. Benefiting from the effectiveness of the video encoder in modeling spatio-temporal interactions, we achieve an extremely heavy compression while well retaining the key information, with each video frame being compressed to an average of only 16 tokens.
The compressed features from different clips are chronologically merged to form the final visual con- text. This representation is then aligned with the feature space of the LLM through video-language projection. Furthermore, to reduce the cost of timestamp encoding, unlike previous approaches (Ren) et al. 2024; Chen et al. (2224) that depend on auxiliary modules or insert textual annotations into every frame. which introduces substantial computational overhead when processing long videos, we introduce a lightweight timestamp prompt appended after the video context: "The video lasts for N seconds, and I frames are uniformly sampled from it." We find that this straightforward approach is sufficient enable the model to perceive the timestamps of the input video, achieving excellent performance on timestamp sensitive tasks such as temporal grounding (see (able Finally, the entire video context Xv can be represented as:
X, = (on) F((,,,) X Ximestamp;;
(5)
where No is the number of clips, F a MLP projection serving as video-language connector.
Progressive Visual Dropout in LLM. Although clip-level compression has been carried out before, due to the possibility of longer-range visual redundancies in long videos (e.g. surveillance videos), and when an LLM responds to specific instructions regarding the visual input, it may not be necessary to continuously focus on the entire long video context. We consider conducting further video-level compression compression during the LLM inference stage. Recent works (Chen et al. 2023 2024b) have explored acceleration strategies for MILMss when processing short visual contexts. Most of them drop visual tokens based on the correlation between text tokens and visual tokens. In contrast, we find that when the LLM processes a long video context, it pays attention to the entire long video context at the shallow layers of the LLM, while focusing on the details of certain local moments at the deep layers (see the Appendix for specific visualizations). Based this observation, we have designed a progressive visual dropout strategy, which is divided into two stages. At the shallow layers of the LLM,, we uniformly drop small number of video tokens (i.e. uniform drop), reducing the
4

---

Published as a conference paper at ICLR 2026
computation while maintaining the original spatio-temporal structure of the video context. At the deep layers of the LLM, we rely on the correlation between text tokens and video tokens to retain the most critical relevant information (i.e. text-guided select). We have found that this operation not only effectively improves the computational efficiency of the model but also slightly enhances the understanding performance of the model by reducing irrelevant visual noise.
3.2 LARGE-SCALE CORPUS FOR LONG VIDEO TRAINING
One of the challenges in long video model training is the shortage of large-scale, high-quality data. Though recent advances have mitigated this by long-form datasets of video-text pairs, these lack the cnttructtio-followiiii paradigm, such as (video, instruction, answer) triplets, crucial for multimodal reasoning. To address this, introduce large-scale long video instruction-tuning dataset named LongVid. It comprises 1 114,228 long videos (with an average duration of 367.3 seconds) and 3,444,849 question-answering (QA) pairs, covering five distinct task types: long video captioning, temporal grounding, event relation recognition, scene relation recognition, and video event counting. Long LongVid significantly surpasses previous datasets in both scale and average video length, enabling models to tackle a wide range of long video scenarios.
To construct the LongVid dataset, we follow three core steps: (1) first, for data source selec- tion, we leverage diverse existing long video datasets that include Ego4D (Grauman et al.. 2022). HowTo100M (Miech et al.) 2019). HD-VILA [uee et al., 2022). and MiraData (Ju et al. 2024) ; these datasets collectively cover multiple video types (e.g.. movies, egocentric videos, news, interviews, how-to videos, and other in-the-wild long videos) to ensure the dataset's diversity. (2) Second, for event label curation, key step that involves generating dense event labels for each long video, we first utilize high-quality short video captions tailored to each source dataset (c.g., Panda-700 70M (Chen al. 2024c) for HD-VILA, CosMo (Wang al. 2024a) for HowTo100M. Ego4D-HCap (Islam et al. 2024) for Ego4D, and the original captions for MiraData), then filter consecutive short video segments that can be reorganized into single long video sequence, and subsequently construct timestamped event label sequences for each long video based on the aforementioned captions; specifically, for datasets with pre-existing high-quality event-level annotations (e.g., HT-Step (Afouras al. 2024) for HowTo 10M, Ego4D-HCap for Ego4D), we directly leverage these annotations as event labels, while for datasets lacking such annotations, we extract core events from the captions using large language model (LLM). (3) Finally, for QA pair construction, we build multiple types of long video question-anssering (QA) pairs using the video captions, event labels, and timestamps of the short video segments, with further details about the entire construction process provided in the Appendix.
3.3 MULTI-STAGE SHORT-TO-LONE LEARNING
Unlike studies (Zhang al., 2024a Xue al. 2024 that use long-form text extend the context window, we prefer that direct training on long- form videos minimizes the gap between training and testing, leading to better downstream evaluations. The training data are detailed in the Appendix.
Stage-1: 1: Video-Language Alignment. In this stage, we freeze the visual encoder and the large language model while training the compressor and the MLP to align the language with the compressed visual features. We use 0.5 million image-text pairs and 0.5 million short video-text pairs, and sample 4 frames from each video in training.
Stage-2:
Short Video Pre-training. To enhance the model's understanding of visual concepts, we conduct visual pre-training using 3.5 million images and 2.5 million short video-text pairs. During this stage, we sample 8 frames from each video.
Stage-3: Joint Short& Long Video Instruction Tuning. To enable the model to handle a wide variety of video tasks, we collect 3.5 million instruction fine-tuning samples, including 1.IM images, 1.7M short videos (under 60 seconds), and 0.7M long videos (60~3600 seconds). We mix the short and long video data ensure the model retains fine- aie-ggrinedd understanding while expanding its comprehension of long videos. The sampling method used is the duration based sampling described in section 3.1 with the number of sampled video frames ranging from 64 to 512.
5

---

Published as a conference paper at ICLR 2026
50
Ranfom CAP
L.bmaVVD CAP
45-
Random
Thams in
LongVA in CAP
VideoChon- Flasb CAP
Long in QA
VidesCtusoFlalii QA
XI
X3
20
beigh foetna
luud withe Katth
15
corteet (1) Koral fe Bod 2id. ( Liv en adters nu gaes
(A) Roring
10
(C) (Stendingena amoun
4000
Figure 3: An example of our Multi-Hop Nee- dle in a Video Haystack. The right path 2. for finding the needle while the wrong path (X1, X2, X3) is for distraction. MLLMs are asked to both find the needle (Q1) and answer its re-
Figure 4: Results on the Multi-Hop NIAH with 10,000 frames. "CAP" represents the ac- curacy of finding the correct needle, while "QA" denotes the accuracy of answering the questions related to the correct needle while also finding the correct needle.
lated question (Q2).
Stage-4: Efficient Video Encoding Enhancement. To enable the model to perceive higher reso- lutions, we employ an efficient post-finetuning strategy to adapt the original cow-resouution video encoder higher-resolution inputs. Specifically, we increase the input resolution of the video encoder from 224 to 448, freeze the L.M,, and directly utilize 25% of the stage-3 data for post-finetuning the video encoding. We find that this simple, full-data strategy effectively enhances the video encoder's adaptability to higher-resolution video inputs.
3.4 MULTI-HOP NEEDLE IN A VIDEO HAYSTACK
Previous works (Zhang al. 2024a; Xue al., 2024) utilize the "Needle in a Video Haystack" (NIAH-Video) to evaluate the long video context understanding ability of models. Specifically, an image (commonly referred to as the "needle") was inserted into a long video and then the model under test was required to input the entire video and answer questions related to the needle. NIAH- assesses the model's capability drawbacks. Firstly, it is difficult to prevent images and questions similar to the needle from appearing the model's training data, which leads to information leakage. Secondly, merely examining the model's 's visual retrieval ability is insufficient and lacks discrimination for evaluating its long video context understanding ability (many models can achieve an accuracy rate over 99%). There is a need to further evaluate its reasoning ability regarding the content.
retrieve information from long videos. However, it has several
To address the above issues, we have designed a new evaluation task called "Multi-Hop Needle in a Video Haystack" (MH-NIAH Video).. As shown in fig. 3 insert reasoning path composed of multiple images into the video haystack. Each image in this path has a randomly inserted position and the corresponding textual clues help find the next image. Given the starting point of the reasoning path, the model needs to follow this path to find the needle and answer questions related to it. What's 's more, to prevent the model from skipping the step of finding the needle by relying on information leakage or memorizing the content of all images, we insert multiple wrong reasoning paths simultaneously while inserting the correct reasoning path. The model needs to find the correct needle (Q1) along the correct reasoning path based on the given starting point and then answer questions related to the needle (Q2). way, our multi-hop approach offers a much more robust evaluation of the long context understanding ability in Multimodal Large Language Models (MLLMs) compared to the previous NIAH- Video. In practice, all images are sourced from MS-COCO (Lin of its
et al. 2014), making noted that even the model can perfectly remember the content of AIS-COCO, it will not be of much help in finding the needle, significantly reducing the likelihood of successful "cheating".
captions and question-
pairs. It should be

---

Published as a conference paper at ICLR 2026
Table I: Results on comprehensive video-linguistic benchmarks.
Medd
Size
Nrg tobore
per Froms
MVlrrach
Ang
Perception Test
V01
TangVirellenhh
Vol
MI.U
36-4r5
VideoMMM onto a isA)
USwrll
1.ng
EVilork Arz
CharadeeSTA
mitel
AumumCaa
Ang
Agg. Dursion
16+
25c
4735
651s
103ls
2356s
41015
30
256
Promnogg Moadl
7505133
6740744
Seall Siy
MCTMN
DAre2-11
Were al
bmm
70
1931
61.2
550004
Charerii
20240
28
256
68.8
46.0
01.
$1.8541
VidesChut-hpaa V--4
28
16
70L0
70.5
S8.5
66.7
57,06399
44.35488
43.9
45.3
63.4
49.41
LLiVA-UneVaadd
196
95.1
56.5
64.7
18.2211
135
314
LL UneVers
7201
1966
10.9
61.5
Miee
78
076
53.6
67.9
58.
70.
63.300
390
VITALS
78
256
56.
IntreVL
38
256
65.8
54.6
GLD
SALIS669
372
708
255
00.6
6L.
61.822
256
72.0
889
54286.9
620
16.9
41A
703
43.6
11.MA-YI
Larral
1224d b334d
41.9
64.5
25.2
25.0
23.9
309
LergVU
She PeTT
78
64
66.9
05.4
H006
159.
LengVA
78
144
56.3
#63476
34,5
LengVil.
2000
78
196
67.1
36.1
57.1
MLINSS
410811
Kanzao
88
256
61.0
54.5
6LD
350/516
46.71903
30.4
712
75.
691999
158814
4K4
74.
76,2
74.
35,4833
48.0
419
4 EXPERIMENTS
Implementation details. We employ UMT-L (Li et al. 2023b), token merging with MLP, and Qwen2-7B as visual encoder, connector, and LLM, respectively. When processing a long video, divide it into shorter clips, each consisting of 4 frames. Each clip is compressed into 64 tokens, meaning that, on average, each frame is represented by 16 tokens. Regarding video-level compression, while we found that enabling during training would slightly impair performance, so only employ it during inference. We use only one-fourth of the full dataset for ablation. See Appendix for details.
4.1 GENERAL VIDEO UNDERSTANDING EVALUATION
Leading performance. We evaluate our model on six general video understanding benchmarks in question-answering format, including two short video benchmarks: MVBench (Li al. (024c) and Perception Test (Patraucean et al., 2024), and three long video benchmarks: Long VideoBench (Wu iet al., (024).. MLVU (Zhou et al. 2024) and LVBench (Wang et al., 2024c), and a comprehensive benchmark. VideoMME (Fu et al. 2024), covering videos ranging from minute-level to hour- level durations. We further evaluate the temporal grounding and video caption tasks, using the Charades-STA (Gao et al., 2017) and AuroraCap (Islam al. 2024). As depicted in table I our VidcoChat-Flaske achieves the best results on diverse VideoQA benchmarks within the 2B and 7B size category, significantly outperforming other approaches. Remarkably, its performance even eclipses that of models with substantially larger scales, such as InternVL.2-66,, as well as proprietary models like GPT-4o and Gemini Crmmini-.55Prre 1.5-Pro. Even when merely supplying timestamp information via a text prompt, our model has achieved remarkable performance in temporal grounding. Mcanwhile, it also significantly outperforms other models in the video captioning task, even surpassing the proprietary GPT-4o and Gemini- 1.5 Pro. This demonstrates the effectiveness of the comprehensive design of our model, data, and training strategies.
4.2 LONG VIDEO CONTEXT EVALUATION
Baseline. Long VA Zhhag et al. 2024a) and LLama-VIn (Li et al., 2024d) are used as baselines. Lo trains MLLMs using long text data, transfering the long context of LLM from text to video. LLama- VID accomplishes efficient inference of long videos by compressing each frame to only two tokens. Our model benefits from these two,
take them as baselines.
As shown in fig. 5 follow the protocols in Long VA (hang et al.) 2024a) for Single-Hop NIAH, we source a long video and sample frames uniformly from it. Then we add needles (indicating images) into the sampled image sequence at different positions. MLLMs
Single-Hop NIAH.
7

---

Published as a conference paper at ICLR 2026
60
86
(a) Long VA Zhang et al. (2024a), accuracy=]1 91.80%-911%% at 3k Trames.
(b) LL VID Li ct al. (2024d), Accuracy=55.0% at 10k Trames.
(c) SdlcoChat-Flass (ours), accu-
racy=99.1% % at 10k frames.
Figure 5: Results on the Single-Hop NIAH evaluation with 10,000 frames.
fed with this long image sequence and answer the corresponding questions to the indicating images. We evaluate all models over 10,000 frames. Note our VideoChat- Flash delivers 99.1% % success rate in accurately retrieving the correct indicating image and answering the related question even across 10,000 frames. In comparison, LongVA VA gives a decent result close to 92% within 3000 frames while ILLaa-VII only achieves 55% accuracy. It demonstrates VideoChat-Flan Flash's state-of-ther performance in long multimodal context modeling.
Table 2: Performance comparison on MH-NIAH benchmark. Evaluated under 1000-frame setting with 266k input tokens. "Cap Score" denotes captioning score, and "QA Score" denotes question answering score.
Model
Thinking
Token Per Frame
Cap Score
QA Score
Random
25%
6.25%
Vid
20%
7%
LongVA
144
25%
18%
VideoChat-Flast (Ours)
16
33%
27%
Gemini2.5 Flash
258
35%
31%
258
60%
54%
Multi-Hop NIAH. this evaluations, MLLMs need to trace along the chain of indicating images, locate the needle, and answer its question. Two metrics "CAP" and "QA" used denote the accuracy of finding the correct needle and the accuracy of answering the questions related to the needle well as finding the needle successfully, respectively. As shown fig. 4 our VideoChat-Flaa still beats all baselines. Specifically, VideoChat-Flec on average, higher than LongVA by around points. can be seen that compared with the single-hop NIAH, the multi-Hop NIAH presents much more difficult challenge, which can better reflect the real gap between the capabilities different models. As shown in table we also have supplemented the results of Gemini 2.5 Flash and Gemini 2.5 Flash thinking our MH-NIAH benchmark. Due to budget constraints, we only evaluated its performance under the 1000-frame setting, with an input token count of approximately 266k. Surprisingly, without enabling the reasoning mode, Gemini 2.5 Flash achieved score only slightly higher than that of our VideoChat-BB -hats- in the Multi-Hop Hopi NIAH. However, its score significantly improved when the thinking mode was activated. This further validates the fact that the Multi- Hop NIAH task we designed truly requires more complex video reasoning capabilities rather than mere retrieval abilities for successful completion.
gives 31.3% and 25.4% in "CAP" and "QA"
4.3 ABLATION & ANALYSIS
Effect of compression ratio. As shown in fig. ba| lower compression ratio (i.e., fewer encoded visual tokens per frame on average) confers a substantial efficiency advantage for both short and long video inputs, and this advantage becomes Notably, as shown in fig. 6bb under our well-designed compression strategy, the detrimental impact of compression on video task performance less significant than anticipated. For both short-video and long-video understanding tasks, appropriate compression can even enhance task performance; even under an extreme compression ratio of 2%, the model can still retain 95% of its performance. This finding gives confidence in developing state-of-tha -oee------hhrrr video understanding models with low compression ratios.
pronounced as the input length increases.
8

---

Published as a conference paper at ICLR 2026
10
MVBood
danggA))
10
4
VieuMM
1ELO75
.8
13.34
10
15.2-
10100
100%
Euput
Compression Ratio Compresion Nam.
Rffect on efficiency.
(h) Effect on performance.
Figure 6: Effect of compression ratio.
Table 5: Effectiveness of visual dropout. The Qwen2-7B we used has a total of 28 layers. "Uni." and "Attn." represent uniform drop and attention select respectively.
Table 4: Effectiveness of video encoder. We use the popular image encoder (SigLIP (Zhai al. 2023)) and video encoder (UMT (Li al. 2023bb) for comparison.
Visual Enceder
FLOP' LVtenny/MVBennt MLVU M-Asg
Arg
VideoMME Overall
Drop typofhepp ratin Drop layer
FLOPs
(G)
Latency MENU
(5)
M-Avg
VideoMME Overal
341.4
2.6
71.8
61.2
Ptodons por frome- mainnig data
SigLIPs (844)
UMT-L822 224
2679
596
79.7
11.8
6L.1(+0.9)
160.2
60.0
62.0
$3.5
53.2(-0.3)
Uni.0.5
Amm.41.5
242.8
242.8
1.9
1.9
71.2
70.7
60.4
60.8
Uni.0.5
18
295-2
2.2
71.7
61.8
Stodons per frame= training data sicesMM
Amm.115
18
295.2
2.2
72.1(40.3) 61.704055)
Sigl Peo 2679
79.7
UMT 224
596
11.8
(3.5(+2.)) 73.7 (2.3()
Alm (1. A0.2
40.25
4,18
245.8
1.9
71.4
60.9
Uni A122
4,18
245.8
1.9
72.0++0.3 61.10001)
Table 3: Ablation of various designs on data model. and resolution, The baseline employs SigLiP-So400M (Zhai et al. (023) as the vision encoder and Spatial donwsampling (196 tokens per frame) the connector. It adopts two-stage train- ing strateay with image and short video following LLaVA (Liu et al. 2023
Ablation of various designs. As shown in ta- ble 3 we have conducted comprehensive ab- lation studies model, it can be observed that HiCo significantly reduces the computation (from 196 to 16 to- kens per frame) while barely compromising the performance. Meanwhile, duration-based sam- pling and timestamp prompts play crucial roles in enhancing the performance. The further leap in performance mainly stems from the training strategy in short-to-long learning and a better mixture of training data.
each design. In terms of the
Settings
MVB
YMMK Overal
Charades
Avg
M-Avg
mlooi
Baseline
60.2
63.7
5288
* HiCo
61.1
60.6
$3.2
short videa pretraining
66.5
62.4
53.9
+ duratico-based based sampling
67.0
64.5
55.5
+ Long Vid data
66.5
68.3
55,8
+ Joint short & long sil
73.2
74.5
64.0
48.4
Effectiveness of spatio-temporal compression encoding. As shown in table[4] we have tested the most popular image encoder, SigLIP (Zhai al. 2023), and the short video encoder, UMT (Li et al.) 2023b), for encoding clips with heavy compression. We found that even when the computational cost is significantly lower, UMT can still achieve better performance on the short video task MV Moreover, as the size of the training data increases from 2 million to million, UMT outperforms SigLIP distinctly across various benchmarks. We believe that this is attributed to the spatio-temporal attention employed by UMT, which can aggregate the key information from different frames within clip, thus enabling the learning of more compact compression features.
High-res
74.0
74.7
65,3
48.0
timestamp prompt
73.4
73.2
Effectiveness of progressive visual dropout. As shown in the table at the shallow layers the LLM, uniform dropout performs better than attention select on long video tasks. However, at the deep layers of the LLM, attention select shows better performance. Performing visual dropout at the deep layers not only improve the computational efficiency but also enhance the performance. Combining uniform dropout and attention select can achieve a good balance between performance and efficiency. More relevant analyses and comparative experiments can be found in the Appendix.

---

Published as a conference paper at ICLR 2026
73
20.00
71,40
7098
71,70
72.50
0000
67
68:50
67
Accurccy
65
MME
65
VideoMME
MVBanch
1
63
MVBench
61
61
60660
60.30
58.50
58.0
58,00
39,30
8
32
64
128
64
128
256
512
Tan
Timax
(a) Effect of T (fmx==18))
(b) Effect of Timxx (Tmi=64)..
Figure 7: Ablation of dtration-bbsed sampling.
Effectiveness of duration-bass sampling. As shown in fig. 7 A relatively large Tinn (64) enables the model to better learn to model the fine actions and rapid movements in short videos during training, thereby enhancing the performance of short video understanding. Increasing T max from 64 to 256 leads to stable improvement in the understanding performance both short and long videos. This indicates that more sampled frames extract more accurate information from our long video data. When T mix reaches 512, there is a slight decline in the performance of short videos. Overall, achieves balance between the performance of short and long videos.
Table 6: Ablation results of Progressive Visual Dropout in training/inference stages.
Training
Evaluation
VideoMME MVBench
No Dropout
No Dropout
53.2
61.1
Progressive Visual Dropout
No Dropout
53.6(+0.4)
58.4(-2.7)
Progressive Visual Dropout
52.2(-1.0)
57.9(-3.3)
Impact of progressive visual dropout on training. As shown in ta ablation studies verify that the train-inference mismatch of Visual Drop (training with dropout vs. inference without) does not lead to performance degradation. Training-stage Visual Drop yields no significant performance gains and incurs slight performance attenuation, which we ascribe its inherent mechanism of identifying and retaining key visual tokens via text-guided attention. We therefore apply Visual Drop exclusively during the inference phase based on these empirical results.
Table 7: Results with different video encoder.
MVBench PerceptionIest
Ving
MLVU
VideoMME (w/o sub.)
LVBench
Video encoder
Avg 16s
Val
Val
M-Avg
Overall
Avg
Avg. Duration
23s
473s
651s
1010s
4101s
UMT-L
73.2
75.6
64.2
74.5
64.0
48.4
Video2-1 1B
74.3(+1.1)
76.3(+0.7)
64.5(+0.3)
73.4(-1.1)
65.2(+1.2)
48.7(+0.3)
Results with InternVideo' As shown in table Intern Video2- IB (Wang et al.) (024c) as the video encoder. We found that stronger video encoder can lead to better compressed representations.
7 We also attempted use the more powerful
5 CONCLUSIONS
In this paper, we address the challenge of long-context video modeling in MLLMs from the model architecture, training data, training strategy and evaluation benchmark. We design an efficient architecture for video MILMss by introducing a hierarchical long video context compression method, which achieves an extreme compression ratio with nearly no performance loss. Regarding data and training, we propose new long video training corpus and short-to-long learning strategy, which effectively enhances the model's 's understanding ability for videos of various lengths. Additionally, developed a and more challenging long video context evaluation benchmark. Our model demonstrated outstanding performance on various video understanding benchmarks, which validates the effectiveness of proposed methods.
10

---

Published as a conference paper at ICLR 2026
6 REPRODUCIBILITY STATEMENT
We provide all the necessary details to reproduce experiments in section 4 and Appendix.
ACKNOWLEDGEMENTS
This work is supported by the National Key R&D Program of China (No. 2022/D0160000),, the Basic Research Program of Jiangsu (No. BK20250009), the Shanghai Artificial Intelligence Laboratory and the Collaborative Innovation Center of Novel Software Technology and Industrialization.
REFERENCES
Triantafyllos Afouras, Effrosyni Mavroudi, Tushar Nagarajan, Huiyu Wang, and Lorenzo Torresani. Ht-step: Aligning instructional articles with how-to videos. Advances in Neural Information Processing Systems, 36, 2024.
Jinze Bai, Shuai Bai, Yunfei Chu, Zeyu Cui, Kai Dang, Xiaodong Deng, Yang Fan, Wenbin Ge, Yu Han, Fei Huang, et al. Qwen technical report. arXiv preprint arXiv:2309. 2023.
Shuai Bai, Keqin Chen, Xuejing Liu, Jialin Wang, Wenbin Ge, Sibo Song, Kai Dang, Peng Wang, Shijie Wang, Jun Tang, et al. Qwen2. 5-vl technical report. arXiv preprint urXiv:2502.1392. 2025.
Max Bain, Arsha Nagrani, Gül Varol, and Andrew Zisserman. Frozen in time: A joint video and image encoder for end-to-end retrieval. In Proceedings of the IEEE/CVF international conference computer vision, pp. 1728-1738. 2021.
Rohan Bavishi, Erich Elsen, Curtis Hawthorne, Maxwell Nye. Augustus Odena, Arushi Somani. and Sadgnak Tasılar. Fuyu-8b: A multimodal architecture for ai agents, 2024. URL https www. www.aaapp..aabbbbbaaaaa http///looo/uuu--8333 Fuyu-8b
Daniel Bolya, Cheng- Yang Fu, Xiaoliang Dai, Peizhao Zhang, Christoph Feichtenhofer, and Judy Hoffman. Token merging: Your vit but faster. arXiv preprint arXiv:2210.09461, 2022.
Keshigeyan Chandrasegaran, Agrim Gupta, Lea M Hadzic, Taran Kota, Jimming He, Cristóbal Eyzaguirre, Zane Durante, Manling Li, Jiajun Wu, and Li Fei-Fei. Hourvideo: 1-hour video- language understanding. arXiv preprint arXiv:2411.04990 2024.
Guiming Hardy Chen, Shunian Chen, Ruifei Zhang, Junying Chen, Xiangbo Wu, Zhiyi Zhang, Zhihong Chen, Jianquan Li, Xiang Wan, and Benyou Wang. Allava: Harnessing gpt4v-synthesized data for a lite vision-language model. arXiv preprint arXiv:2402.1 16684, 2024a.
Jieneng Chen, Luoxin Ye, Ju He, Zhao Zhao-Yang Wang, Daniel Khashabi, and Alan Yuille. Lla- volta: Efficient multi-modal models via stage-wise visual context compression. arXiv preprint arXiv:2406.20092, 2024b.
Liang Chen, Haozhe Zhao, Tianyu Liu, Shuai Bai, Junyang Lin, Chang Zhou, and Baobao Chang. An image is worth 1/2 tokens after layer 2: Plu--and-play inference acceleration for large vision- language models. In European Conference on Computer Vision, pp. 19-35. Springer, 2025.
Tsai-Shien Chen, Aliaksandr Siarohin, Willi Menapace, Ekaterina Deyneka, Hsiang-wee wein Chao, Byung Eun Jeon, Yuwei Fang, Hsin- Ying Lee, Jian Ren, Ming-Hsuan Yang, al. Panda-70m: Captioning 70m videos with multiple mross-modalityy teachers. In Proceedings of the IEEE/CVF Conference Computer Vision and Pattern Recognition, pp. 11331, 2024c.
Zhe Chen, Weiyun Wang. Yue Cao, Yangzhou Liu, Zhangwei Gao, Erfei Cui. Jinguo Zhu, Shenglong Ye, Hao Tian, Zhaoyang Liu, et al. Expanding performance boundaries open-source multimodal models with model, data, and test-time scaling. arXiv preprint (XXi::2412005222 0,5227,, 2024d.
11

---

Published as a conference paper at ICLR 2026
Zhe Chen, Weiyun Wang, Hao Tian, Shenglong Ye, Zhangwei Gao, Erfei Cui, Wenwen Tong, Kongzhi Hu, Jiapeng Luo, Zheng Ma, Ji Ma, Jiaqi Wang. Xiaoyi Dong, Hang Yan, Hewei Guo, Conghui Hc, Botian Shi, Zhenjiang Jin, Chao Xu, Bin Wang, Xingjian Wei, Wei Li, Wenjian Zhang, Bo Zhang, Pinlong Cai, Licheng Wen, Xiangchao Yan, Min Dou, Lewei Lu, Xizhou Zhu, Tong Lu, Dahua Lin, Yu Qiao, Jifeng Dai, and Wenhai Wang. How far the gap to commercial multimodal models with open-source suites, CoRR, abs/2404. 16821, 2024e.
gpt-4v? closing
Zesen Cheng. Sicong Leng, Hang Zhang, Yifei Xin, Xin Li, Guanzheng Chen, Yongxin Zhu, Wenqi Zhang, Ziyang Luo, Deli Zhao, and Lidong Bing. Videollama 2: Advancing spatial-temporal modeling and audio understanding in video-llms. CoRR, abs/2406.07476. 2024.
Tri Dao. Flashattention--2: Faster attention with better parallelism and work partitioning. arXiv preprint arXiv:2307.08691, 2023.
Jiajun Fei, Dian Li, Zhidong Deng, Zekun Wang, Gang Liu, and Hui Wang. Video-ccam: Enhancing video-language understanding with causal cross-attention masks for short and long videos. arXiv preprint arXiv:2408. 14023, 2024.
Chaoyou Fu, Yuhan Dai. Yondong Luo, Lei Li, Shuhuai Ren, Renrui Zhang, Zihan Wang, Chenyu Zhou, Yunhang Shen, Mengdan Zhang, et al. Video-mme: The first-ever comprehensive evaluation benchmark of multi- Ilms video analysis. arXiv preprint arXiv:2405. 20055, 2024.
Chaoyou Fu, Haojia Lin, Xiong Wang, Yi-Fan Zhang, Yunhang Shen, Xiaoyu Liu, Yangze Li, Zuwei Long, Heting Gao, Ke Li, et al. Vita- 1:5: Towards gpt-4o level real-time vision and speech interaction. arXiv preprint arXiv:2501 0015570.0177.2001 2025.
Jiyang Gao, Chen Sun, Zhenheng Yang, and Ram Nevatia. Tall: Temporal activity localization via language query. In Proceedings of the IEEE international conference on computer vision, pp. 5267-5275, 2017.
Kristen Grauman, Andrew Westbury, Eugene Byrne, Zachary Chavis, Antonino Furnari, Rohit Girdhar, Jackson Hamburger, Hao Jiang, Miao Liu, Xingyu Liu, et al. Ego4d: Around the world in 3,000 hours of egocentric video. In Proceedings of the IEEE/CVE Conference on Computer Vision and Puttern Recognition, pp. 18995-19012, 2022.
Zhenpeng Huang, Xinhao Li, Jiaqi Li, Jing Wang, Xiangyu Zeng, Cheng Liang, Tao Wu, Xi Chen, Liang Li, and Limin Wang. Online video understanding: comprehensive benchmark and memory-augmented method. arXiv preprint arXiv:2501.003: 2024.
Md Mohaiminul Islam, Ngan Ho, Xitong Yang. Tushar Nagarajan, Lorenzo Torresani, and Gedas Bertasius. Video recap: Recursive captioning of hour-long videos. In Proceedings of the IEEE/CVF Conference Computer Vision and Pattern Recognition, pp. 18198-18208, 2024.
Xuan Ju, Yiming Gao, Zhaoyang Zhang, Ziyang Yuan, Xintao Wang, Ailing Zeng, Yu Xiong, Qiang Xu, and Ying Shan. Miradata: A large-scale video dataset with long durations and structured captions. arXiv preprint 24Xi::2407.06338 2024.
Will Kay, Joao Carreira, Karen Simonyan, Brian Zhang, Chloe Hillier, Sudheendra Vijayamarasimhan, Fabio Viola, Tim Green Trevor Back, Paul Natsev, et al. The kinetics human action video dataset. arXiv preprint arXiv: 000.//05550691 2017.
Bo Li, Peiyuan Zhang. Jingkang Yang. Yuanhan Zhang, Fanyi Pu, and Ziwei Liu. Otterhd: A high-resolution multi-modality model. arXiv preprint ArXiv:2311.042/1 2023a.
Bo Li, Yuanhan Zhang, Dong Guo, Renrui Zhang, Feng Li, Hao Zhang, Kaichen Zhang, Yanwei Li, Ziwei Liu, and Chunyuan Li. Llava-onevision: Easy visual task transfer. CoRR, abs/2408.03326. 2024a.
Feng Li, Renrui Zhang, Hao Zhang, Yuanhan Zhang, Bo Li, Wei Li, Zejun Ma, and Chunyuan Li. llva--eex-interleeae: Tackling multi-image, video, and 3d in large multimodal models. arXiv preprint 0rXiv:2407.07885, 2024b.
Kunchang Li, Yali Wang, Yizhuo Li, Yi Wang, Yinan He, Limin Wang, and Yu Qiao. Unmasked teacher: Towards training-efffcient video foundation models. In ICCV, 2023b.
12

---

Published as a conference paper at ICLR 2026
Kunchang Li, Yali Wang, Yinan He, Yizhuo Li, Yi Wang, Yi Liu, Zun Wang, Jilan Xu, Guo Chen, Ping Lou, Limin Wang, and Yu Qiao. Mvbench: A comprehensive multi-modal video understanding benchmark. In CVPR, pp. 22195-22200 IEEE, 2024c.
Kunchang Li. Yinan Hc, Yi Wang, Yizhuo Li, Wenhai Wang, Ping Luo, Yali Wang, Limin Wang, and Yu Qiao. Videochat: chat-centric video understanding. Sei. China Inf. Sci. 68(10), 2025. doi: 10. 1007/S1 (0SSS11442-2--4 442--024-3321-9 http:///////ttps:///ooo/o///////
Yanwei Li, Chengyao Wang, and Jiaya Jia. Llama-vid: An image is worth 2 tokens in large language models. In ECCV, volume 15104 of Lecture Notes in Computer Science, pp. 323-340. Springer, 2024d.
Bin Lin, Yang Ye, Bin Zhu, Jiaxi Cui, Munan Ning, Peng Jin, and Li Yuan. Video-llava: Learn- ing united visual representation by alignment before projection. In EMNLP, pp. 5971-5984. Association for Computational Linguistics, 2024.
Tsung- Yi Lin, Michael Maire, Serge Belongie, James Hays, Pietro Perona, Deva Ramanan, Piotr Dollár, and C Lawrence Zitnick. Microsoft coco: Common objects in context. In ECCV, pp. 740-755. Springer, 2014.
Haotian Liu, Chunyuan Li, Qingyang Wu, and Yong Jae Lee. Visual instruction tuning. In NeurIPS, 2023.
Jiajun Liu, Yibing Wang, Hanghang Ma, Xiaoping Wu, Xiaoqi Ma, Xiaoming Wei, Jianbin Jiao, Enhua Wu, and Jie Hu. Kangaroo: A powerful video-language model supporting long-context video input. arXiv preprint arXiv:2408. 15542, 2024.
Muhammad Maaz, Hanoona Rasheed, Salman Khan, and Fahad Khan. Vidcogpt+: Integrating image and video encoders for enhanced video understanding. arXiv preprint arXiv:2406.094 09418, 2024.
Karttikeya Mangalam, Raiymbek Akshulakov, and Jitendra Malik. Egoschema: A diagnostic benchmark for very long-form video language understanding. Advances in Neural Information Processing Systems, 36:46212-46333 2023.
Antoine Miech, Dimitri Zhukov, Jean-Baptiste Alayrac, Makarand Tapaswi, Ivan Laptev, and Josef Sivic. Howtol00m: Learning a text-video embedding by watching hundred million narrated video clips. In Proceedings of the IEEE/CVF international conference on computer vision, pp.
2630-2640, 2019.
Mathew Monfort, Sou Young Jin, Alexander Liu, David Harwath, Rogerio Feris, James Glass, and Aude Oliva. Spoken moments: Learning joint audio-visual representations from video descriptions. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp-
14871-14833 2021.
OpenAl. GPT-4 technical report. CoRR, abs/2303.08774, 2023.
OpenAl. Gpt-4o. https://ppenai..cm///nee//////// 1otellllooppt-oo//1 May 2024.
Viorica Patraucean, Lucas Smaira, Ankush Gupta, Adria Recasens, Larisa Markeeva, Dylan Banarse, Skanda Koppula, Mateusz Malinowski, Yi Yang, Carl Doersch, al. Perception test: A diagnostic benchmark for multimodal video models. NIPS, volume 36, 2024.
Jeff Rasley, Samyam Rajbhandari, Olatunji Ruwase, and Yuxiong He. Deepspeed: System optimiza- tions enable training deep learning models with over 100 billion parameters. In Proceedings of the 26th ACM SIGKDD International Conference on Knowledge Discovery & Data Mining, pp- 3505-3506, 2020.
Ruchit Rawal, Khalid Saifullah, Miquel Farré, Ronen Basri, David Jacobs, Gowthami Somepalli, and Tom Goldstein. Cinepile: A long video question answering dataset and benchmark. arXiv preprint arXiv:2405.088// 2024.
13

---

Published as a conference paper at ICLR 2026
Machel Reid, Nikolay Savinov, Denis Teplyashin, Dmitry Lepikhin, Timothy P. Lillicrap. Jean- Baptiste Alayrac, Radu Soricut, Angeliki Lazaridou. Orhan Firat, Julian Schrittwieser, loannis Antonoglou, Rohan Anil, Sebastian Borgeaud, Andrew M. Dai, Katie Millican, Ethan Dyer, Mia Glaese, Thibault Sottiaux, Benjamin Lee, Fabio Viola, Malcolm Reynolds, Yuanzhong Xu, James Molloy, Jilin Chen, Michael Isard, Paul Barham, Tom Hennigan, Ross Mcllroy, Melvin Johnson, Johan Schalkwyk, Eli Collins, Eliza Rutherford, Erica Moreira, Kareem Ayoub, Megha Goel, Clemens Meyer, Gregory Thornton, Zhen Yang. Henryk Michalewski, Zaheer Abbas, Nathan Schucher, Ankesh Anand, Richard Ives, James Keeling, Karel Lenc, Salem Haykal, Siamak Shakeri, Pranav Shyam, Aakanksha Chowdhery, Roman Ring. Stephen Spencer, Eren Sezener, and et al. Gemini 1.5: Unlocking multimodal understanding across millions of tokens of context. CoRR, abs/2403 055330, 2024.
Shuhuai Ren, Linli Yao, Shicheng Li, Xu Sun, and Lu Hou. Timechat: A time-sensitive multimodal large language model for long video understanding. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 14313-14322 2024.
Share. Sharegemini: Scaling up video caption data for multimodal large language models, June 2024. URL httphhtts///thhhhe com Sharel 4/ //Shaarreeemnn
Xiaoqian Shen, Yunyang Xiong, Changsheng Zhao, Lemeng Wu, Jun Chen, Chenchen Zhu, Zechun Liu, Fanyi Xiao, Balakrishnan Varadarajan, Florian Bordes, et al. Longvu: Spatiotemporal adaptive compression for long video-language understanding, arXiv preprint arXiv:2410.174533 2024.
Yan Shu, Peitian Zhang, Zheng Liu, Minghao Qin, Junjie Zhou, Tiejun Huang, and Bo Zhao Video-xl: Extra-long vision language model for hour-scale video understanding. arXiv preprint arXiv:2409.14488 2024.
Enxin Song. Wenhao Chai, Guanhong Wang, Yucheng Zhang, Haoyang Zhou, Feiyang Wu, Haozhe Chi, Xun Guo, Tian Ye, Yanting Zhang, et al. Moviechat: From dense token sparse memory for long video understanding. In Proceedings of the IEEE/CVF Conference Computer Vision and Puttern Recognition, pp. 18221-18232, 2024.
Guangyu Sun, Archit Singhal, Burak Uzkent, Mubarak Shah, Chen Chen, and Garin Kessler. From frames to clips: Efficient key clip selection for long-form video understanding. arXiv preprint arXiv:2510.02262. 2025.
Reuben Tan, Ximeng Sun, Ping Hu, Jui-hsien Wang, Hanieh Deilamsalehy, Bryan A Plummer, Bryan Russell, and Kate Saenko. Koala: Key frame-conditioned long video-llm. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 13581-13591, 2024.
Xi Tang, Jihao Qiu, Lingxi Xie, Yunjie Tian, Jianbin Jiao, and Qixiang Ye. Adaptive keyframe sampling for long video understanding. In Proceedings of the Computer Vision and Patlern Recognition Conference, pp. 29118-29128, 2025.
Alex Jinpeng Wang, Linjie Li, Kevin Qinghong Lin, Jianfeng Wang, Kevin Lin, Zhengyuan Yang, Lijuan Wang, and Mike Zheng Shou. Cosmo: Contrastive streamlined multimodal model with interleaved pre-training arXiv preprint arXiv:2401.00849 2024a.
Peng Wang, Shuai Bai, Sinan Tan, Shijie Wang, Zhihao Fan, Jinze Bai, Keqin Chen, Xuejing Liu, Jialin Wang, Wenbin Ge, et al. Qwen2-vl: Enhancing vision-language model's perception of the world at any resolution. arXiv preprint 1401122409.12191, 2024b.
Weihan Wang, Zehai He, Wenyi Hong. Yean Cheng, Xiaohan Zhang, Ji Qi, Xiaotao Gu, Shiyu Huang, Bin Xu, Yuxiao Dong, et al. Lvbench: An extreme long video understanding benchmark. arXiv preprint ar arXiv:2406.0880
2024c.
Xidong Wang, Dingjie Song, Shunian Chen, Chen Zhang, and Benyou Wang. Longllava: Scaling molti-modal Iims to 1000 images efficiently via hybrid architecture. CoRR, abs/2409.02889 2024d.
Yi Wang. Kunchang Li. Yizhuo Li, Yinan He, Bingkun Huang, Zhiyu Zhao, Hongjie Zhang, Jilan Xu, Yi Liu, Zun Wang, et al. Intervvideo: General video foundation models via generative and discriminative learning. arXiv preprint arXiv:2212.03111 2022.
14

---

Published as a conference paper at ICLR 2026
Yi Wang, Kunchang Li, Xinhao Li, Jiashuo Yu, Yinan He, Guo Chen, Baoqi Pei, Rongkun Zheng, Jilan Xu, Zum Wang, et al. Internvide02: Scaling vidco foundation models for multimodal video understanding. In ECCV, 2024e.
Hongchen Wei and Zhenzhong Chen. Visual context window extension: A new perspective for long video understanding. arXiv preprint arXiv:2409.200/8. 2024.
Yuetian Weng, Mingfei Han, Haoyu He, Xiaojun Chang, and Bohan Zhuang. Longvlm: Efficient long video understanding via large language models. In ECCV, pp. 453-470. Springer, 2025.
Haoning Wu, Dongxu Li, Bei Chen, and Junnan Li. Longvideobench: A benchmark for long-context interleaved video-language understanding. arXiv preprint arXiv:2407.15754. 2024.
Lin Xu, Yilin Zhao, Daquan Zhou, Zhijie Lin, See-Kiong Ng, and Jiashi Feng. Pllava: Parameter-free llava extension from images to videos for video dense captioning. CoRR, abs/2404. 16994, 2024.
Fuzhao Xue, Yukang Chen, Dacheng Li, Qinghao Hu, Ligeng Zhu, Xiuyu Li, Yunhao Fang, Haotian Tang, Shang Yang, Zhijian Liu, et al. Longvila: Scaling long-context visual language models for long videos. arXiv preprint arXiv:2408. 10188, 2024.
Hongwei Xue, Tiankai Hang. Yanhong Zeng, Yuchong Sun, Bei Liu, Huan Yang. Jianlong Fu, and Baining Guo. Advancing high-resolution video-language representation with large-scale video transcriptions. Proceedings of the IEEE/CVF Conference on Computer Vision and Puttern Recognition, pp. 5036-5045, 2022.
Dongjie Yang. Suyuan Huang, Chenggiang Lu, Xiaodong Han, Haoxin Zhang, Yan Gao, Yao Hu, and Hai Zhao. Vript: A video is worth thousands of words. arXiv preprint arXiv:2406.0600 2024.
Xiangyu Zeng, Kunchang Li, Chenting Wang, Xinhao Li, Tianxiang Jiang, Ziang Yan, Songze Li, Yansong Shi, Zhengrong Yue, Yi Wang. et al. Timesuite: Improving milms for long video understanding via grounded tuning. arXiv preprint 0xXiv:2410.19700 2024.
Xiaohua Zhai, Basil Mustafa, Alexander Kolesnikov, and Lucas Beyer. Sigmoid loss for language image pre-training. In ICCV, 2023.
Hang Zhang, Xin Li, and Lidong Bing. Video-Ilama: An instruction-tune -unned audio-visual language model for video understanding. arXiv preprint arXiv:2306. 02858, 2023a.
Hongjie Zhang, Liu, Lu Dong, Yifei Huang, Zhen-Hua Ling, Yali Wang, Limin Wang, and Yu Qiao. Movqa: A benchmark of versatile crestton--nsswriigg for long-form movie understanding, arXiv preprint 2xX1v:2312.04817, 2023b.
Peiyuan Zhang, Kaichen Zhang, Bo Li, Guangtao Zeng, Jingkang Yang, Yuanhan Zhang, Ziyue Wang, Haoran Tan, Chunyuan Li, and Ziwei Liu. Long context transfer from language to vision. CoRR, abs/2406. 168522 2024a.
Ruohong Zhang, Liangke Gui, Zhiqing Sun, Yihao Feng, Keyang Xu, Yuanhan Zhang, Di Fu, Chunyuan Li, Alexander Hauptmann, Yonatan Bisk, et al. Direct preference optimization of video large multimodal models from language model reward. arXiv preprint arXiv:2404..0 002.0/ 2024b.
Yuanhan Zhang, Bo Li, haotian Liu, Yong jae Lee, Liangke Gui, Di Fu, Jiashi Feng, Ziwei Liu, and Chunyuan Li. Llava-next: A strong zero-shot video understanding model, April 2024c, URL /ettps://llavae
qwt.tuthhhh........ 1 /./bb/b 1tt.:/0020---------1a------e-e -videt//tvvvvv////
Yuanhan Zhang, Jinming Wu, Wei Li, Bo Li, Zejun Ma, Ziwei Liu, and Chunyuan Li. Video instruction tuning with synthetic data. arXiv preprint arXiv:2410.02713. 2024d.
Zijia Zhao, Haoyu Lu, Yuqi Huo, Yifan Du, Tongtian Yue, Longteng Guo, Bingning Wang, Weipeng Chen, and Jing Liu. Needle in video haystack: A scalable synthetic framework for benchmarking video milms. arXiv preprint 7xXiv:2406.09367 2024.
Junjie Zhou, Yan Shu, Bo Zhao, Boya Wu, Shitao Xiao, Xi Yang, Yongping Xiong, Bo Zhang, Tiejun Huang, and Zheng Liu. Mivu: A comprehensive benchmark for multi-task long video understanding. arXiv preprint 1.Xiv:2446604264 2024.
Zirui Zhu, Hailun Xu, Yang Luo, Yong Liu, Kanchan Sarkar, Zhenheng Yang, and Yang You. Focus: Efficient keyframe selection for long video understanding. arXiv preprint 27X1v:2510.27280, 2025.
15

---

Published as a conference paper at ICLR 2026
A
APPENDIX
In this appendix, we provide more details from the following aspects:
A.
Statement on the Use of LLM.
A.2
Supplementary proof of Clip-Compression.
A.3
More Results & Discussions of VideoChat- Flash.
A.4
Implementation Details of VideoChat-Flash.
A.5
Dataset Details of Long Vid.
A.6
Qualitative Results of VideoChat-Flash.
A.1
STATEMENT ON THE USE OF LLM IN WRITING
LLM used for polishing and review during the writing of this article.
A.2
SUPPLEMENTARY PROOF OF CLIP-COR COMPREESION
To complete the derivation section 3.1 we elaborate on the key steps using fundamental information- theoretic properties, including conditional entropy, joint entropy, and mutual information.
Step 1: Conditional Entropy for Compression Loss The information loss due to compression defined as the conditional entropy of the original features given the compressed feature, i.e., L H((Y,) Z). By the definition of conditional entropy, for any random variables A and B:
H(A B) H(A, B) H(B),
(6)
where H(A, B) is the joint entropy of A and B, and H (BB is the marginal entropy of B. For our setting, let A (.....,, Yr (original features) and (compressed feature). The information loss thus becomes:
L. = -({,)) HH)
Step 2: Simplifying H(Y.)), 2) for Deterministic Compression The compression operation C deterministic, meaning uniquely determined by (Y.) (i.c., Z = ((YY))). For deterministic functions, the conditional entropy of the output given the input is zero:
H (Y,)) 0,
(8)
since uncertainty remains about Z once (Y,) is known.
By the chain rule of joint entropy:
H ((YY), Z) H(Y))) + H(Z (Y.)).
(9)
Substituting H(Z (Y.)) = 0, we get:
(10)
Combining eq. 7 and eq. (10), the information loss simplifies to:
L. H((Y.)) (Z))
(11)
Step 3: Image Encoder Loss Derivation Image encoders assume inter-frame independence, i.e., the joint probability factorizes as:
T
pC ing
inng
p Y
ing
(12)
16

---

Published as a conference paper at ICLR 2026
For independent random variables, the joint entropy equals the sum of marginal entropies:
H(Y
in
Yimg!
Σ HY
(13)
Substituting eq.
13
into eq.
(the general loss formula), the Image Encoder loss becomes:
Le jimg
Σ I(Y. HZ),
(14)
which matches eq.
Step 4: Video Encoder Loss Derivation Video encoders model inter-frame dependencies via spatiotemporal attention, so they do not assume independence. Their joint entropy is expanded using the chain rule of joint entropy:
H
Y T
H(Y id
Y ..... Y
(15)
where H(Y frames.
Y
Y is the conditional entropy of the I-th frame given all previous
By the definition of mutual information, for random variables X and y:
I(X:)) -())))>()))) - H - )) = H(X) I(x:)).
(16)
Applying this to the conditional entropy in eq. 15 (letting X Y and y = (Y
Y }):
H(Y
vii
Yvid
H(Y) - I(Yrid, Yrid
Y
(17)
Substituting eq.
17 into eq.
15
the joint entropy for Video Encoders becomes:
H H(Y
Yvid
H(Y I(Yrid, Y ..... Y
(18)
Finally, substituting eq.
[8
into eq.
(the general loss formula), the Video Encoder loss becomes:
Lydd
[((Y))))) I(Y)d), Y ..... Y H()),
(19)
which matches eq.
Step 5: Comparing T.img and I.yld For most videos, consecutive frames correlated (e.g., static backgrounds or smooth motion), so the cumulative mutual information is positive:
T(Yri, Y ,.... y O AL 2.
(20)
This implies:
T Σ 1=1
T vid.
T Σ H(Y:
H(Y
Σ 10
(21)
Given that Image and Video Encoders process the same video (so ΣT HYY and share the same compressed feature (so H (Z) is identical), we conclude:
img
H(Y
iin
vid
(22)
17

---

Published as a conference paper at ICLR 2026
Table 8: Comparison of connectors.
MVBench MLVU
VideoMME Overall
Connector
Avg
Avg
M-Avg
#tokens per frame=729, compression ratio= 100%
MLP (Uncompressed)
59.4
64
55.3
59.6
#tokens per frame= 196, compression ratio=27%
Spatial Downsampling
60.2
63.7
52.8
58.9(-0.7)
Uneven Downsampling
60.9
62.5
54.9
59.4(-0.2)
Spatio-temporal Resampler
59.5
61.9
51.9
57.8(-18)
Similar Token Merging
62.8
66.7
56.8
62.1 62.1(+2.5)
#tokens per frame=49, compression ratio= 7%
Spatial Downsampling
60.2
61.8
53.6
58.5(-1.1)
Uneven Downsampling
59.8
62.8
54.3
59.0(-0.6)
Spatio-temporal Resampler
55.5
58.1
51.1
54.9(-4.7)
Similar Token Merging
61.4
63.3
55.3
60.0(+0.44
#tokens per frame= 16, compression ratio
2%
Spatial Downsampling
58.1
61.1
50.1
56,4(-3.2)
Uneven Downsampling
58.3
60.0
52.3
56.9(-2.7)
opatioltemporal Resampler
51.4
54.7
47.7
51.3(-8.3)
Similar Token Merging
60.2
62.4
53.5
58.7(-0:9)
(a) Spatial Downsampling
(b) Uneven Downsampiing
XV
(c) poatl Resampler
(d) Similar Token Merging
Figure 8: Comparison of different connectors.
A.3 MORE RESULTS & DISCUSSIONS VIDEOCHAT-FLASH
A.3.1 DETAILED RESULTS VISUAL COMPRESSION
Different connectors and compression ratio. As shown in the table 8 we consider three differrnt
numbers of tokens per frame after compression (16, 49, 196) and four popular token compression
strategies: spatial downsampling (Zhang et al.) 2022cc Chen et al. 2024e), uneven downsampling (Wei) Chen 2024), spatio-temporal resampler (Li et al. 2024c| Tan et al. 2024). and similar token merging (Bolya al. 2022 Weng et al. 2025) (more details can be found in the Appendix). It can be seen that compared with other methods, the parameter- free similar token merging operation achieve remarkably low compression ratio and even obtain better performance than without compression. Even the extreme case 2% compression ratio, it can still maintain most of the performance.
18

---

Published as a conference paper at ICLR 2026
Table 9: Comparison of FLOPs and Cuda memory. The FLOPs and inference memory estimated using one NVIDIA A A100-80G GPU with one sample, and the training is estimated using 32 NVIDIA A100-80G GPUs with DeepSpeed ZeRO-3 (Rasley et al. 2020). We assume that the visual features have been extracted and stored in advance, so we only consider the FLOPs and memory of the LLM.
Input frames
Avg tokens
FLOPs (T)
Memory(G)
Model
per frame
Train
Infer
LongVILA Xue et al.
2024
196
224.8
15.4
16.7
64
Long VA (Zhang et
2024a
144
155.9
12.3
16.3
VidcoChat-Flash
16
14.8
4.8
15.4
LongVII.A (Xue et al. 2024
196
1467.5
50.1
21.0
256
LongVA VA ZZhang ct al.
2024a
144
930.4
37.8
19.5
VideoChat-F
16
63.0
7.6
15.7
Long (Xue al.
2024
196
14336.9
oom
37.7
1000
LongV VA Zhang et al.
2024a
144
8278.9
oom
31.8
VideoChat-Flashe
16
303.3
18.6
17.1
Long VILA (Xue et al.
2024
196
1184250.0
oom
00m
10000
Long VA (Zhang ct al.
2024a
144
644632.0
VideoChat-F
16
9969.5
0om
33.6
Training cost. The main training advantage of our method lies in the fact that compression reduces the context sequence length fed into the LLM. First, consider the impact of reduced context sequence length on training under the condition of the same training data volume. Here, we cite the comparison of training system throughput under different sequence lengths provided by the state-of- the-art long video training system Long VILA Xue et al. (2024) as a reference- this comparison is conducted on 64 H100 GPUs and measured in time per iteration (seconds).
Table 10: Training system throughput comparison (cited from LongVILA). The data is measured on 64 H100 GPUs, with the metric being time per iteration.
Model
Token Per Frame
Max Frames
Sequence Length
Training Time per Iteration (s)
VideoChat-Flash
512
8k
Not provided Long
16
2048
32k
4.24
1122
256
512
128k
16.0
256
2048
512k
66.1
As shown in table 10 it can be observed that despite the excellent acceleration optimizations for long-sequence training, changes in sequence length still significantly impact training speed. Our method can substantially reduce the training cost of long videos. Regarding the specific training cost, our method roughly requires 32 A100 GPUs for to days of training, which significantly lower than that of mainstream models such as Intern VL and QwenVL.
Inference cost. As in table even when processing short videos, the compute load of our model only one-tenth that of previous models. Meanwhile, the number of input frames increases, the difference becomes more and more pronounced. Only our model can complete the inference on 10,000 frames on single A 1100-80G. Concretely, VideoChat-Fllahh compute load is orders of magnitude lower than that of LongVILA (Xue et al., 2024) (9,969.5 vs. 1,184,250.0 TFLOPs). We also present the real inference costs using HuggingFace Pipeline and Flash-Attention table II] under the same video frame input and resolution, the FLOPs, GPU memory consumption, and latency of our VideoChat-Flaan are all significantly lower than those of Qwen2.5-VLL VL- one of the representative mainstream MLLLMsddmmonstraite the high efficiency of our method.
As shown in
A.3.2 VISUAL. DROPOUT IN LLM
Visual token redundancy in LLM inference. As shown in fig. 9 we find that even when half of the tokens discarded the shallow layers of the I.M,, the performance of long video understanding
19

---

Published as a conference paper at ICLR 2026
Table 11:
Inference cost comparison (measured via HuggingFace Pipeline with Flash-
Attention2).
Model
Input Frames Nom. Visual Tokens
FLOPs
GPU Memory latency (Vision Encoder)
latency (LLM) latency (Total)
Qwen2.5.VL
8192
158.65
18.66
1.115
2.308
3.422
VideoChat-Flash
44.32
0.096
0:419
Qwen2.5VVL
256
32768
632.29
28.07
4.621
6.211
10.832
VodooChat-Flash
4096
175.03
20.41
1,188
0.337
1.526
Qwen2.5-VL
1024
131072
2526.80
65.72
18,930
34,473
53,403
VideoChat-Flash
1024
16384
697.81
35.92
4.697
1.42
6,140
74
Uniform drop
73
Attention drop
MLVU Accuracy (%
No drop
72
71
70
69
68
4
6
8
10
14 Drop layer
12
16
18
20
22
24
Figure 9: Visual redundancy in long context across layers. We conduct experiments on Qwen2-7B (28 layers) and test the impact of dropping 50% of the visual tokens from shallow to deep layers.
only degrades marginally. This indicates that despite high compression at the clip level (encoding each frame into only 16 tokens), there remains considerable redundancies between clips when their representations are interacted in the LLM. Furthermore, we find the overall understanding performance gets better as the dropout happens in deeper layer of the model. Remarkably, at approximately two-thirds of the LLM's depth, the performance even surpasses that of the no-discard baseline. This might suggest that the deeper layers of the network, an excess of visual tokens may interfere with the model's 's reasoning process. For the drop type, observe that uniform drop often outperforms attention- baseed selection in the shallow layers. We suppose, at these layers, the LLM has not yet fully determined the specific locations focus As result, relying attention may introduce bias.
Visualization of visual attention map. As shown in the fig. T0 for long video context, the attention of text tokens is relatively dispersed in the shallow layers the network. However, as layers deepen, the attention gradually becomes focused on specific regions. Thus, we believe that the attention scores in the deeper layers are more reliable, while those in the shallow layers may be prone to bias.
The Premise:
Effective dropping requires the model to accurately determine which visual tokens are semantically relevant the text.
At the early stages of training, the MLLM has not yet established robust alignment between textual and visual modalities. Consequently, cannot reliably identify "key" visual information. Forcing the model to drop tokens based on unlearned or unstable attention weights prevents from effectively learning the full video representation.
The Conflict:
20

---

Published as a conference paper at ICLR 2026
layer
300
300
Index
Layer
100
200
Layer
Taken Index
Layer
300
200
Index
Layer 20
0.033
0.008
Attention
Lack las Alteeal
Token Index
Figure 10: Visualization of the attention scores from the last textual token to visual tokens at each layer of the network.
21

---

Published as a conference paper at ICLR 2026
A.3.3 RESULTS ON IMAGE UNDERSTANDING BENCHMARKS
Our model is specifically designed for video understanding. However, according to the newly- evaluated results of image benchmarks, our model can still outperform the strong image- MLLM, LLaVA-NeXT (Zhang al. (024c), with significantly lower computational cost: MMMU (45.2 vs. 35.3), MME (1843.4 vs 1603.7).
A.3.4 THE IMPACT OF VISUAL COMPRESSION ON EFFICIENCY AND PERFORMANCE
emphasize that the advantages of our compression strategy are twofold, benefiting both efficiency and performance:
From an efficiency perspective: With the same number of input frames, our method significantly reduces computational costs without a marked decline performance, as demonstrated in the second row of table
From a performance perspective: In scenarios where computational resources and LLM context length are constrained-a common bottleneck long-video understanding where standard sparse sampling often leads to severe information cossceompresioon allows the model to ingest much larger number video frames. This enables the model to achieve better performance by scaling up the input video context. To further empirically validate this, we present the additional data below. As shown in the table T2 under a fixed budget of total Vision Tokens (4,096), compressing the per-frame representation (from 256 to 64 tokens) allows us to significantly increase the number of input frames (from 16 to 64). This enhanced temporal context leads to substantial performance gains.
Table 12: Performance results under different Token per frame and Input Frames configura- tions.
Token Per Frame
Input Frames
Vision Token
VideoMME
MotionBench
256
16
4096 4096
51.3
44.8
64
64
59.3(+8.0)
46.2(+1.4)
A.3.5 FURTHER ELABORATION ON NOVELTY.
A critical distinction between our work and prior art lies in our objective: we pursue efficient long-video modeling to build long-video MILMS from scratch, whereas most existing works aim to accelerate existing MI.LMs (e.g., by freezing most modules or adopting training-free approaches).
We believe that learning native compressed representations for long-video modeling essential. To this end, we propose the Hierarchical Compression (HiCo) paradigm. This is an effective system- level design rather than merely a specific component. HiCo decouples compression into two stages: handling visual redundancy at the Clip-level (within the ViT) and textual semantics at the Video-level (within the LLM). Furthermore, regarding specific design details, our approach offers key differences and novel insights compared to previous work:
Clip-level Compression: We propose leveraging spatio-temporal attention to capture local inter-frame redundancy. This allows the Video Encoder to learn how to compress representa- tions via parameter updates, rather than relying on hand-crafted compression modules. This self-adaptive mechanism is key to achieving our exceptionally high compression rate.
Video-level Compression: We introduce new improved visual drop scheme specifically optimized for the long-video understanding domain.
Timestamp Prompts: We demonstrate that concise text prompts remain highly effective even under extreme visual compression. This avoids the computational overhead associated with adding text timestamps to every frame, a common practice in prior work.
Duration-based Sampling: We validate a more effective sampling strategy: applying dense sampling for short videos and sparse sampling for long videos. This differs from
22

---

Published as a conference paper at ICLR 2026
previous works, which predominantly utilize fixed FPS sampling or rely exclusively sparse sampling.
In summary, by combining these improvements, we are the first to demonstrate that heavy- compression-based methods can achieve long-video understanding performance comparable to, or even surpassing, closed-source context-extension-nee models.
A.3.6 DISCUSSION ON THE TRAINING-FREE FRAME SELECTION METIOI
Numerous training-free frame selection approaches Tang et al. (025): Zhu al. (2025); ; Sun et al. (2025) have been employed to enhance the long-video understanding of MLLMs. These methods target the selection of key frames or clips most relevant to a specific task, thereby avoiding the direct input of the full long-video context. This improves efficiency and often boosts performance, tasks.
particularly given that most MLLMs are currently optimized for short-context rather than long-context
While we acknowledge that training-free frame selection is an effective, low-cost solution for specific application scenarios, we argue that it possesses inherent limitations when viewed from the perspective of constructing a general general-purposs purpose foundation model for long video understanding:
Task Formulation & Intrinsic Capability: Fundamentally, frame selection simplifies long-video task into a short-context problem; it does not enhance the MLLM's intrinsic ability to process long temporal contexts. in contrast, our approach reduces the visual token count via clip-level compression and employs a short-to-long training strategy. This allows the MI.LM efficiently and holistically comprehend the compressed long-video context.
Granularity & Precision: External selection modules typically have a limited performance ceiling. A text-guided selector may struggle to accurately interpret complex instructions. Conversely, method utilizes video-level compression to leverage the L.LM's inherent capabilities in identifying critical visual information. This achieves precise token-level selection, which offers significantly finer granularity than coarse frame-level selection.
Information Retention: Frame selection inevitably requires discarding significant portion of the raw data, making unsuitable for tasks requiring dense, global understanding (c.g., dense video captioning detailed video description). Our method retains essential semantic information, demonstrating superior performance on these information-intensin tasks.
In summary, we regard frame selection and our proposed method as distinct paradigms tailored for different objectives: the former is suitable for adapting MLLMs in a training-free manner, while the latter is designed to train a native long-video MLLM.
A.4 IMPLEMENTATION DETAILS OF VIDEOCHAT-FLASI
A.4.1 VIDEO-LANUUAGE CONNECTORS
As shown in fig. from video clips:
8 we consider four popular token compression strategies to compress the features
Spatial Downsampling. Applying spatial operations (pooling (Xu et al. 2024). interpola- tion (Zhang et al. 2024c). and convolution (pixel shuffle) (Chen 2004e)) to each video frame for downsampling has been demonstrated in previous work (Xu et al. 2024 Maaz (et al.] (224)) an effective method to reduce the number of video tokens. However, due to the lack of temporal interaction, this approach fails to leverage the relation between frames. We use pixel shuffle in our experiments.
Uneven Downsampling. Considering the similarities between adjacent frames, it is un- necessary to retain full details for every frame. We can apply wwwn-sampling operations with different sizes across frames within clip. Specifically. lower down-sampling size is applied to the first frame, while higher down-sampling sizes are used for the remaining frames. Similar approaches have been validated in recent study (Wei & Chen 2024).
Resampler. Using a learnable compressor, such as a Q-Former (Li et al.) 2024ch cross-attention layer, to compress spatiotemporal tokens. However, this approach
23

---

Published as a conference paper at ICLR 2026
requires large amount of data for effective learning. In training, observe that the Q- Former barely converges well in our setting. So in our ablations, we adopt a single-layer cross-attention instead.
Similar Token Merging, We directly merge similar tokens, using the ToMe (Bolya et al. 2022 approach.
A.4.2 VISUAL DROPOUT IN LLM
Herein, we present the detailed implementation visual dropout in LLM: For uniform drop, we uniformly drop a proportion of visual tokens and reassign position ids to the retained visual tokens. For text-guided selection, since Flash-Attention2|] Dao (023) does not support returning valid attention maps, we instead compute the attention scores between text tokens and vision tokens independently for text-guided selection.
A.4.3 TRAINING HYPERPARAMETERS.
As shown in Table 1, the training details and hyperparameters for each stage of our VideoChat- -lash model are presented.
Table 13: Training details of each training stage for the SaatooChtt-Flash--- model.
Stuge-1
Stuge-2
Stage-3
Stage-4
Resolution Num. frames
Num..
234
224 x
224xx (64--511) 16x(644-512)
224xx (64--512))
16x8
16x(66--512)
ANO
Dataset
Image & Short Video
Image & Short Video
(Mull)) Image & Short/Long Video
(Mulk)) Image ShomfLong Video
ASamples
0.3M
Thanug Model
Trainable
Projector 20:M
Medel
VTT&&Prgector
7.68 LLM
7:98
7.98
0.38
Size
256
256
LR of vision encnder
1x10-3 1x10-3
2x110-6
2xx 10-4
2x-10-1
x10--
x010
x110-
Epoch
A.4.4 TRAINING DATA
Stage 1: Video-Language Alignment. this stage, we use 558k image-text pairs from LCS-
558K (Liu et al. 2023) and 481k short video-text pairs from S-MiT (Monfort et al. (021).
Stage 2:
Short Video Pre-training. To enhance the model's understanding visual concepts, we conduct visual pre-training using 3.5 million images and 2.5 million short video-text pairs.
Video Description Data. We utilize the video description data recaptioned with VideoChat2 (Li al., 2024c) from WebVid2M (Bain et al. 202T).
Detailed Video Description Data. We employ the 323k detailed video description data recap- tioned with Gemini (Reid et al., 2024 from WebVid (Bain et al. 2021) and Kinetics (Kay et al. 2017). as in previous work (Share 2024).
Detailed Image Description Data. We use the 3.5 million detailed image description data recaptioned with LLava-NeXT-34B (Zhang et al.) 2024c) from the following datasets: COCO118K, BLIP558K, and CC3M, as provided by previous work (Li et al. 2024a).
Text Data. To enhance the model's language understanding capabilities, we incorporate 143K samples from the Evo-Instruct dataset (Chen et al. 2024a).
Stage 3: Joint Short& & Long Video Instruction tuning. To enable the model to handle wide
variety of video tasks, we collect 3.5 million instruction fine-tuning samples, including 1.1M images, 1.7M short videos (under 60 seconds), and 0.7M long videos (60~3600 seconds).
Long Video Instruction data. We primarily utilized long video instruction data from MoiveChat (Song et al. 2024 Vript Yang et al. 2024) and subset of Long
24

---

Published as a conference paper at ICLR 2026
Distribution by Category
Entertiiment (10.754, 9.40a
(AMS, 6. 1563
(4,005, 5.450)
(5,159, 4.5%
4.%
(5,000 4955
Scense Technology
Total Videos:
3.8%6)
253%
114,228
54%
Braucation
Artee Vebadde
Fil Ausimation
(3,872,344% 3,4%)
Ganing
Casadd 22,55,
Thavd Evente (2,025, 1.8%)
Animate (1.905, 1 756 Monpoocts (900, 08%))
(793, (799,0.7%) 0.7%5
Figure 11: Distribution by categroy.
Short Video Instruction data. We primarily utilized short video data from VideoChat2 [Li et al. 2024c) and InternVideo' (Wang et al. 2024e) for instruction fine-tuning. Addi-
tionally, we incorporated data annotated with GPT4-0 from previous works, including
ShareGPT4o (Chen et al., 2024e Wang al.
2024e VideoChatGPT-PIur (Maaz et al.
2024h, LLaVA Video 178K (Zhang et al. 2024d) and LLava-Hound (Zhang et al. 2024b).
Image Instruction data We primarily utilized single-image instruction data from LLava- NeXT (Zhang et al. 2024c), Allava (Chen et al.) 2024a),, and ShareGPT4-o (Chen et al. Additionally, incorporated multi-image data provided by
2024e
Wang al.] 2024e).
LLaVA-IInerleae
(Li ciet al. 2024b).
A.5 DATASET DETAILS OF LONGVID
The videos Long Vid are curated from open-source video datasets: Ego4D (Grauman et al. 2022 How 100M (Miech al. 2019), HD-VILA (Xue et al., 2022), and MiraData (Ju et al. 2024). We provide statistics and details of the data construction pipeline for each dataset as follows.
Dutribatisn of Vidcos by Duration
Die Butiia Qustiie Anewar Pairs Thik Categery
15.4%
1a%
24.5%
5.3%
Wileo
18-3reaa
24%%
Revt Maless
(36786
31.2%
33.51
Figure 12: Distribution of videos by dura- tion.
Figure 13: Distribution of Question-Answer Pairs by Task Category.
25

---

Published as a conference paper at ICLR 2026
A.5.1 STATISTICS AND EXAMPLES
As shown in fig. I1 fig. 12 fig. 13 we' ve shown the data and QA type distribution for Long Vid. Next, we will provide examples for the five tasks.
Task 1: Video Captioning
Question:
Which option best describe artistic style, visual and photographic aspects for this video, such as realistic, cyberpunk, and cinematic style?
Options&Annserr:
(A) The video features a high-definition, realistic graphic style with detailed textures and vibrant lighting effects, emphasizing futuristic and immersive cyberpunk aesthetic.
(B) The video features degindefiniion,, realistic graphic style with detailed textures and dynamic lighting, emphasizing a gritty, futuristic aesthetic that is both immersive and visually engaging.
(C) The video features high-definition, realistic artistic style with focus on detailed textures and vibrant lighting, enhancing the immersive medieval game setting.
(D) The video features realistic yet distinctly stylized graphic design typical of modern vidco games, with vibrant colors and detailed environments that enhance the immersive experience.
Task 2: Temporal Grounding
Question: When does person is walking down street in video game?
Antwons&Answer:
(A) 00:08:56.833 00:09:56.367
(B) 00:03:03.033 00:04:09.367
(C) 00:07:21.900 00:08:07.633
Task 3: Event Relation Recognition
Question:What is the event in the video between "fold the wrapper loosely around the
filling" and "repeat until there are three or four folds on each side"?
Options&A
(A) wrap the dough and let sit 10-30 minutes
(B) mix the vegetables and ground meat together
(C) add the potstickers the hot oil
(D) fold piece of dough at corner
( peel and chop other herbs and vegetables
Task 4: Scene Relation Recognition
Question: Which option can best describe the scenery of the video?
&ptions&Annwer::
(A) lush forest - sunset
(B) rural suburban
(C) city night - streetlights + + buildings trees
(D) indoors makeshift + operations room
Task S: Video Event Counting
Question: According to the video, how many steps does the chef take during cooking?
Answer:
(A) 6
26

---

Published as a conference paper at ICLR 2026
A.5.2 EGO4D
For ego-centric videos, we adopt 3,662 long videos from the Ego4d (Grauman al. 2022) and lever- age Ego4DHcap (Illa et al. 2024) as the corresponding captions. Ego4DHcap gives hierarchical captions for short, medium, and long video segments. For the short video captioning task, we directly utilize these captions, while for the dense caption task, we concatenate captions in the lower level to form dense one. For example, we merge all short video captions in a medium video segment to create a dense medium-leve one, and the dense caption of long video segments can be formed by concatenating multiple medium-level video captions.
We also build event relation recognition and temporal grounding tasks based on captions of short video segments. For the event relation recognition task, models are required to choose the right order of an event sequence. Since we find the captions of short videos are highly concise and event-oriented, we use them as the event labels and serially put the short captions a medium-level video segment as the ground-truth event relationship. For the temporal grounding task, the short video captions with the corresponding timestamps as the ground-truth, and randomly select other timestamps in the current medium video segments the false options.
A.5.3 MIRADATA
MiraData (Ju et al. 2024) provides multi-level captions for large-scale minute-level movic segments. Apart from short and dense captions that are used for short and dense video captioning tasks, also provides multiple fine-grat fine-grained captions that focus on various specific perspectives, such as the main object, background, camera movements, and video style. We use an open-source LLM (Qwen-72b (Bai al. )023)) extract the event and background labels from the main object and background captions, respectively, and we put the labels ground truth of the event/background relation recognition task. For the temporal grounding task, we use the event label with the corresponding timestamp as the ground-truth option.
long video in the right order as the
A.5.4 HOWTo100M
HowTo100m (Miech et al. 2019) includes more than million long-duration how-to videos. We adopt How Tolnterreiin (Wang al., (024a),, video captioning dataset that provides refined interleaved video captions of HowTo 100M videos as short and dense video captions. For the event relationship recognition and temporal grounding tasks, we use HTStep (Afouras et al. 2024). large-scale dataset containing temporal annotations of instructional steps in HowTol 10000M videos.
A.5.5 HD-VILA
While previous datasets focus on long videos in specific domains, we also select part of the videos from HD-VILA (Xue al. 2022). large-scale video dataset that includes various in-the-wild videos. We argue that adding these videos into training could enhance the model's ability to process long videos in some uncommon domains. For HD-VILA videos, we adopt the captions of Panda- 70M (Chen et al. 2024c). Specifically, we filter consecutive video segments that can be re-constructed into more than 60s long videos from the 10M training subset and utilize these captions as the video short/dense captioning and temporal grounding tasks. The event labels are also extracted from these captions in the same way as Miralata (Ju et 2024).
A.6
QUALITATIVE RESULTS OF VIDEOCHAT- HOOEOSHAT-FLASH
We perform qualitative comparisons of model with the proprietary model Gemini-1.5 Pro (Reid
et al. 2024 and the open-source LUngVU (Shen et al. (2024) and Videol (Cheng et al. 2024) across three tasks: fine-grained understanding of short videos figs. 14a 15 and long video
understanding ( figs. 16 and l7p.
We use the newest Gemini-1.5 Pro-002 for evaluation.
27

---

Published as a conference paper at ICLR 2026
26:11 31
00.24
Time
The peeson uses sultiple sirilar jejtcts to play an occlusion pume, Mieree the hideen objuct as the gane the passon spoint Clofi, or zight?
Gennet Pre
O
from pacson's soint view.
Free pesnt hooden the left side the the
hinder ooject under miodls cup the parsen's point
Figure 14:
Fine-grained Understanding of Short Videos: Fast Motion. By adopting a dense sampling strategy for short videos, model effectively captures fast motion within the video, enabling it to accurately determine the final position of the object under the cup.
Launch once
Launch beice
Lsunch thme times
6102
ite sany ttare did the peneor launes objects in the talle?
person abject
Lengl'
taurched acceeding
The Launched objects the table three times.
Figure 15: Fine-grained Understanding of Short Videos: Action Count. Our model can accurately capture actions in short videos while also recording their occurrence frequency.
28

---

Published as a conference paper at ICLR 2026
Burglry
00.40
Is thrre possiale crisinal behawiee in this sirveiltance video? 16 there 16, stat type colmssal bevarder 28?
aspeurs that converter thate oatalytic crtas. Catalytic convrrters vallabbl contaze platines, galladian, aur ctudion. lhese ectels are exsersive, and satalyUic corvecters are relatively
Paggest
The video does shau any explictt crinieal bataxdar, bet does show pezzzn matking actixiit or stmply nakecrg.
fased the gives Informatias, there is na cntticatton of any cotatast betavtor in tht auvelllacee video. The video only ahoua cara pactedd along strest nest budldingg night time
Yes, there possiible criatnal behariee ln this surveltlance video person unlkinng down street and theft.
Figure 16: Long video understanding: Anomaly Detection In Surveillance Videos. Our model can detect anomalous behaviors in surveillance videos and provide corresponding inferences.
Carried on herss by another individual
Tiree
Reuats ixpaned aet indu the sike starts argars souviid seruttic much
led. tume the siate wan is coctee the tho con coeetec the addet after the sen that for waver sntted mes metume and wiltiees The Butter bitt and as ciemy, spives nomire fied ment
exchunding
light
O
The person Lying An the bed in the capp wus sazzied there by another person
Logio
The video does informstion
the bad acriived
individul.
Figure 17: Long video understanding: Moive Understanding, Our model can understand the plot of movie and retain detailed visuals.
29


Reviews:
[{"review_id": "KbA5DLMepg", "reviewer": "Reviewer_rnXS", "summary": "This paper presents VideoChat-Flash, a Multimodal Large Language Model (MLLM) designed to efficiently handle extremely long video contexts. The authors identify high computational cost and information redundancy as the primary bottlenecks in current long-video models. To address this, the paper introduces a comprehensive, four-part contribution: 1.A novel Hierarchical video token Compression method. This operates at two levels: (1) a Clip-level compression during encoding that uses a spatio-temporal video encoder to exploit local redundancy, and (2) a Video-level \"progressive visual dropout\" during LLM inference that prunes tokens based on attention patterns . 2.A new, large-scale dataset for long-video instruction tuning. 3.A multi-stage short-to-long learning scheme. 4.A new, more challenging \"Needle in a Haystack\" benchmark, the \"Multi-Hop Needle-In-A-Video-Haystack\".", "questions": "1. Following on from my point in \"Weaknesses,\" could you please elaborate on the performance drop when only HiCo is added (Table 2, row \"+ HiCo\")? How much of the final SOTA performance should be attributed to the sheer efficiency of HiCo (allowing for processing long contexts) versus the new LongVid dataset and short-to-long training strategy (which seem to compensate for an initial compression-induced quality loss)?\n2. The authors mention that the video-level \"Progressive Visual Dropout\" is used \"only during inference\" due to \"challenges in compatibility with training acceleration\". Would there be issues such as training-inference mismatch ? However the results show this method is helpful. Could the author explain a bit why such mismatch does not lead to performance drop ?", "limitations": "", "rating": 4, "confidence": 3, "soundness": 2, "presentation": 3, "contribution": 2, "strengths": "1.The core strength is the HiCo compression framework. Achieving a 1/50 compression ratio (16 tokens/frame) while simultaneously achieving SOTA performance is a breakthrough for practical long-video MLLMs .The design is well-motivated.\n2. The introduction of the Multi-Hop NIAH benchmark is a significant contribution.\n3. The contributions of the paper are multifold including model design, data creation, and new evaluation.", "weaknesses": "1. The biggest concern is the novelty of the proposed method. While the system as a whole is novel and highly effective, its constituent parts are largely clever integrations of existing ideas.\n2. The ablation in Table 2, while extensive, presents a slightly confusing narrative for HiCo. The baseline (196 tokens/frame) achieves a 63.7 on MLVU. The \"+ HiCo\" model (16 tokens/frame) drops to 60.6 on MLVU. This seems to contradict the \"almost no performance loss\" claim from the abstract. The performance is only recovered after applying the new training strategies and data. This suggests HiCo does incur a performance cost, which is then compensated for by the data and training recipe.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "Iw5ry6fDhf", "reviewer": "Reviewer_cNh3", "summary": "The paper proposes a hierarchical video token compression method designed to efficiently handle long-video contexts without compromising performance. It begins with duration-based sampling, which adapts to the differing requirements of short and long videos. This is followed by clip-level compression to aggregate key information across frames, and finally, a progressive visual dropout mechanism that allocates attention differently across shallow and deep layers. The authors also introduce a training dataset aimed at enhancing video understanding capabilities along with a multi-stage training strategy. In addition, a new challenging evaluation benchmark is presented to better assess complex reasoning abilities. Finally, the model is evaluated on multiple benchmarks, demonstrating the effectiveness of the proposed method, with ablation studies further validating the contribution of each component.", "questions": "1. Regarding the creation of the Multi-Hop NIAH evaluation task, is there any restriction on the maximum number of frames that can be inserted?\n2. For the image encoder, what is the rationale for using SigLIP instead of SigLIP2, which is generally considered a stronger image encoder? Similarly, for the video encoder, why not choose VideoPrism?\n3. Why does the accuracy drop when increasing $T_{min}$ from 64 to 128 in Figure 7 (a)? Intuitively, performance should improve as more frames are used to capture finer details, as suggested by the trend in Figure 7 (b).", "limitations": "", "rating": 8, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 4, "strengths": "This is a well-structured and thoroughly executed paper. It presents extensive and insightful experiments and analyses that may also benefit future work in video understanding. The proposed HiCo framework demonstrates strong and consistent performance across multiple benchmarks. Additionally, the authors provide a new training dataset and evaluation benchmark, which will further benefit the community and advance progress in video understanding. The comprehensive ablation studies clearly illustrate the contribution of each component.", "weaknesses": "1. Some recently released baseline video models [1][2][3][4] are not included in the comparison. It would improve readability if the authors could explicitly indicate which model represents the previous state of the art in Figure 1.\n2. It would be beneficial to include more advanced video models as baselines for comparison on the multi-hop NIAH task. Presenting the performance of closed-source models, such as Gemini 2.5 Pro, could also provide valuable context for readers to comprehend the complexity of this task. \n3. It would be helpful to provide more details about the text-guided selection mentioned in L214.\n\n[1] Comanici, G., Bieber, E., Schaekermann, M., Pasupat, I., Sachdeva, N., Dhillon, I., Blistein, M., Ram, O., Zhang, D., Rosen, E. and Marris, L., 2025. Gemini 2.5: Pushing the frontier with advanced reasoning, multimodality, long context, and next generation agentic capabilities. arXiv preprint arXiv:2507.06261.\n\n[2] Zhang, B., Li, K., Cheng, Z., Hu, Z., Yuan, Y., Chen, G., Leng, S., Jiang, Y., Zhang, H., Li, X. and Jin, P., 2025. Videollama 3: Frontier multimodal foundation models for image and video understanding. arXiv preprint arXiv:2501.13106.\n\n[3] Wang, X., Si, Q., Wu, J., Zhu, S., Cao, L. and Nie, L., 2025. Adaretake: Adaptive redundancy reduction to perceive longer for video-language understanding. arXiv preprint arXiv:2503.12559.\n\n[4] Xu, M., Gao, M., Li, S., Lu, J., Gan, Z., Lai, Z., Cao, M., Kang, K., Yang, Y. and Dehghan, A., 2025. Slowfast-llava-1.5: A family of token-efficient video large language models for long-form video understanding. arXiv preprint arXiv:2503.18943.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "hAFFx0xuQS", "reviewer": "Reviewer_TNAS", "summary": "This work aims to efficiently understand the extremely long video context with multimodal large language models (MLLMs). They propose a novel Hierarchical video token Compression (HiCo) method, which leverages visual redundancy in long videos to compress long video context from Clip-level to Video-level, reducing the computation significantly while preserving essential details. Then, they introduce a multi-stage short-to-long learning scheme and also a large-scale video dataset called LongVid as well as a benchmark. They build VideoChat-Flash that shows a leading performance on both mainstream long and short video benchmarks at the 2B and 7B model scales.", "questions": "N/A. \nOne tiny suggestion: fig -> Fig, tab -> Tab", "limitations": "", "rating": 6, "confidence": 5, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "1. The topic is meaningful to the community: we need to find a better way to understand long videos with MLLMs efficiently. \n2. The paper is well-organized and easy to follow. The experiments are solid and clear. \n3. I really appreciate the authors for the LongVid dataset and benchmark. Open-sourced data is important in this community.", "weaknesses": "1. The paper claims “almost no performance loss” under extreme 1/50 compression, yet the quantitative analysis (Fig. 6b) provides limited breakdown across different downstream tasks. It would be helpful if they can provide the results on temporal grounding or motion-sensitive tasks besides the three benchmarks. \n2. The claimed efficiency benefits (1/50 token reduction) are shown in FLOP counts but not in end-to-end wall-clock latency or GPU memory consumption during real long video inference. It would be helpful to report them in the paper as well! \n3. Although the LongVid dataset is described as large-scale and diverse, construction heavily relies on caption-based pseudo-labeling using other LLMs. There is limited discussion of data noise, filtering quality, or potential overlap with evaluation benchmarks.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "NOUL25vWfD", "reviewer": "Reviewer_m8kd", "summary": "This paper introduces VideoChat-Flash, a new model for long-context video modeling. It is trained on LongVid, a novel dataset combining Ego4D, HowTo100M, HD-Vila, and MiraData, with additional dense and timestamped event labels. The core of their method is hiCo (Hierarchical Compression), a multi-step technique to reduce both clip-level and video-level redundancy. It involves sampling clips based on video duration, using a vision encoder with spatio-temporal attention to compress visual and temporal information within clips, and then merging these compressed features into a final vision context aligned with LLM representations via a projection head. The training process is multi-staged: video-language alignment, pre-training with short videos, short and long video instruction tuning, and finally, post-finetuning for higher-resolution video encoding. The authors also present a new \"Needle in a Haystack\" benchmark. Unlike traditional benchmarks that insert a single needle, their approach leverages model reasoning by embedding multiple clues in a video that the model must follow to find the final objective. VideoChat-Flash significantly outperforms models like GPT4, Gemini 1.5, and other open-source models on various benchmarks, including LongVideoBench, MLVU, VideoMME, and LVBench, demonstrating strong performance across long video contexts. An ablation study was conducted to analyze different compression ratios and architectural and training criterion designs.", "questions": "- Do the authors plan to open-source models and benchmarks?\n- For the Multi-Hop Needle-In-A-Video-Haystack eval, are the clues always going forward in the video or can they also be backward? Let's say you have 3 hops, one at 5s, another at 8s, and the last one at 3s before having all the clues to get to the final haystack.", "limitations": "", "rating": 8, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "- Paper well-written\n- Lot of artefacts such as new model, new training data mix, and new evaluation protocole\n- Really like the idea of a reasoning based needle in a video haystack evaluation\n- Extensive experimental protocole and comparison against lot of different methods\n- Performance significantly higher than the concurrent methods\n- In depth ablation studies over the compression effect, sampling strategies, video encoder and design choices\n- Good appendix with details on hyper-parameters that were used, additional ablations studies and additional training data mix information.", "weaknesses": "- Would have expected some discussions over the overall training/inference cost of this method in contrast to other.\n- Missing some discussion on the training-free frame selection method such as AKS (Adaptive Keyframe Sampling for Long Video Understanding)\n- Missing information about the different video data licenses. A datasheet would have been appreciated.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (Poster)