LongVU: Spatiotemporal Adaptive Compression for Long Video-Language Understanding
Authors: Xiaoqian Shen
Yunyang Xiong
Changsheng Zhao
Lemeng Wu
Jun Chen
Chenchen Zhu
Zechun Liu
Fanyi Xiao
Balakrishnan Varadarajan
Florian Bordes
Zhuang Liu
Hu Xu
Hyunwoo J. Kim
Bilge Soran
Raghuraman Krishnamoorthi
Mohamed Elhoseiny
Vikas Chandra
Venue: ICML 2025 poster (2025)
Keywords: Long Video Understanding;Video-Language;Spatiotemporal
Primary area: deep_learning->foundation_models
TL;DR: Large language model for long video language understanding

Abstract: Multimodal Large Language Models (MLLMs) have shown promising progress in understanding and analyzing video content. However, processing long videos remains a significant challenge constrained by LLM's context size. To address this limitation, we propose \textbf{LongVU}, a spatiotemporal adaptive compression mechanism that reduces the number of video tokens while preserving visual details of long videos. Our idea is based on leveraging cross-modal query and inter-frame dependencies to adaptively reduce temporal and spatial redundancy in videos. Specifically, we leverage DINOv2 features to remove redundant frames that exhibit high similarity. Then we utilize text-guided cross-modal query for selective frame feature reduction. Further, we perform spatial token reduction across frames based on their temporal dependencies. Our adaptive compression strategy effectively processes a large number of frames with little visual information loss within given context length. Our LongVU consistently surpass existing methods across a variety of video understanding benchmarks, especially on hour-long video understanding tasks such as VideoMME and MLVU. Given a light-weight LLM, our LongVU also scales effectively into a smaller size with state-of-the-art video understanding performance.

LongVU: Spatiotemporal Adaptive Compression for Long Video-Language Understanding
Xiaoqian Shen 1,2"
Yunyang Xiong" "
Changsheng Zhao' Lemeng Wu' Jun Chen'
Chenchen Zhu'
Zechun Liu"
Fanyi Xiao*
Balakrishnan Varadarajan' Florian Bordes' Zhuang Liu' Hu Xu
Hyunwoo J. Kim
Bilge Soran
Raghuraman Krishnamoorthi Mohamed Elhoseiny?" Vikas Chandra"
Meet AI
"King Abdullah University of Science and Technology (KAUST) Korra University
https://vision cair weaarrggtuub... LonnggVVV
Abstract
Oucation: Viere did pet Lic white cup?
Multimodal Large Language Models (MLLMs) have shown promising progress in understanding and analyzing video content. However, process- ing long videos remains significant challenge constrained by LLM's context size. To address this limitation, we propose LongVU, U, spatiotem- poral adaptive compression mechanism that re- duces the number of video tokens while preserv- ing visual details of long videos. Our idea based leveraging cross-modal query and inter- frame dependencies to adaptively reduce tempo- ral and spatial redundancy in videos. Specifically, we leverage DINOv2 features to remove redun- dant frames that exhibit high similarity. Then we utilize text-guided cross-modns moddll query for se- lective frame feature reduction. Further, we per- form spatial token reduction across frames based their temporal dependencies. Our adaptive compression strategy effectively processes a large number of frames with little visual information loss within given context length. Our Long VU consistently surpass existing methods across a variety of video understanding benchmarks, es- pecially on hour-long video understanding tasks such VideoMME and MLVU. Given light- weight LLM, our LU also scales effectively into smaller size with state-of-the-art video un- derstanding performance.
0000
0034
01.55
00:37
13:3
Uniorm
LongtU [Cum]
Acaptive
You left the whate cup on the tablee
Figure I. Effectiveness of LongVU commonly-used uni- form sampling and dense sampling. Uniform sampling overlooks critical frames due its sparse nature. Dense sampling may sur- pass the maximum context length, Icading to truncation of tokens from targeted frames. In contrast, our method can adaptively conduct spatiotemporal compression, accommodating long video sequences while preserving more visual details.
1. Introduction
Large Language Models (LLMs) (Brown, 2020; Ouyang et al., 2022: OpenAI, 2022: Achiam et al., 2023; Chiang et al., 2023; Touvron al., 2023; Jiang et al., 2024) manifest universal capabilities that are instrumental in progress towards general intelligence. Through the integration of modality alignment and visual instruction tuning, Multi- modal Large Language Models (MLLMs) (Alayrac et al., 2022; Li et al., 2023b; Zhu et al., 2023; Liu et al., 2024b; Ye et al., 2023; Bai et al., 2023; Chen et al., 2023c; Dong et al., 2024) have demonstrated exceptional competencies in tasks such as captioning and visual question-ande guestion-anssering. Recent literature has initiated explorations of extending MLLMs for the comprehension of video content (Li et al., 2023e; Zhang et al., 2023; Maaz et al., 2023a; Lin et al., 2023; Wang et al., 2024: Liu et al., 2024c). Despite exhibiting po- tentials across specific benchmarks. effectively processing and understanding of exceedingly lengthy videos remains a
'Work done during the internship at Meta Al "Project lead Correspondence to: Yunyang Xiong <yunyang@ meta.com>, Xi- aoqian Shen <xiaoqian shen kaust.edu.sa>.
Proceedings of the nd International Conference Machine Learning. Vancouver, Canada. PMLR 267, 2025. Copyright 2025 by the author(s).

---

LongVU; Spatiotemporal Adaptive Compression for Long Video-Language Understanding
significant challenge.
To evaluate our method, we conduct extensive experiments across various video understanding benchmarks, including EgoSchema (Mangalam et al., 2024), MVBench (Li et al., 2024b), VideoMME (Fu et al., 2024), and MLVU (Zhou et al., 2024). Our LongVU significantly outperforms several recent open-source video LLM models, such as VideoChar2 (Li et al., 2024b), Long VA (Zhang et al., 2024a), and LLaVA-One Vision (Li et al., 2024a), by a large mar- gin. For example, our LongVU outperforms strong open- source baseline, LLa Vision (Li et al., 2024a) by approximately ~5% in average accuracy. We also observed that our light-weight LongVU, basing Llama3. 3l-33 (Llama, 2024) as the language backbone, significantly improves over previous state-of-the-art small video-LLMMs e.g., Phi- 3. 55-vision-inssrruc--ee (Abdin et al., 2024), by 3,4% on VideoMME Long subset. Our LongVU established new state-of-the results video understanding benchmarks among video-language models. We believe that our pro- posed approach marks a meaningful progression towards long-video understanding MLLMs.
One primary reason is that it is impractical to process all the information for hour-long videos, given that advanced MLLMs represent a single image using hundreds of tokens. For instance, 576 ~ 2,880 tokens per image are used in LLa (Liu et al., 2024a) and 7,290 tokens are used in Va-One monly used and computationally manageable context length for multimodal training is 8k, which limits processing 125 frames (2-minutes video) even at 64 tokens per frame, while an hour-long video could require over 200k tokens. Con- sequently, in video scenarios with an extra temporal di- mension, it is intractable for training due to the demand of excessive GPU memory. Various studies have attempted to establish a balance between the number of tokens and the frequency of frame sampling. Most of these studies (Li et al., 2024a; Cheng et al., 2024; Zhang et al., 2024c; Chen et al., 2024a) opt for a uniform sampling of fixed number of video frames as input. However, these methods naively overlook non- non-uniform content, c.g., static vs dynamic scenes within the video, as shown in Figure Other approaches (Li et al., 2023c;d; Jin al., 2023) employ intensive resampling modules that significantly decrease the quantity of visual tokens, leading to considerable loss of essential visual information.
(Li et al., 2024a). However, a com-
2. Related Work
2.1. Vision Language Models
Early visual language models (VLMs) such CLIP (Rad- ford et al., 2021), are trained with a contrastive loss to project both vision and language embeddings to a shared representation space. SigLIP (Zhai et al., 2023) takes a sigmoid loss instead, allowing further scaling the training batch size with better performance.
In this paper, we propose VU that aims to preserve as much frame information as possible while accommodat- ing lengthy videos without exceeding the context length of commonly used LLMs. Video, by its nature, contains significant temporal redundancy. MovieChat (Song al., 2024) employs a similarity- frame-level feature selec- tion using CLIP visual representation (Radford et al., 2021). While we argue that DINOv2 (Oquab et al., 2023), through self-supervised training with a feature similarity objective on vision-centric tasks, captures subtle frame differences and low-level visual features more effectively than vision- language contrastive methods (Radford et al., 2021; Zhai et al., 2023), as shown in Figure 5. Hence, (1) we apply temporal reduction strategy on the frame sequence by leveraging similarity from DINOv2 (Oquab et al., 2023) fea- tures to remove redundant video frames. In addition, (2) we jointly capture the detailed spatial semantic and long-range temporal context by performing selective feature reduction via cross-modal query, where we preserve full tokens for frames that are relevant to the given text query, while ap- plying spatial pooling to reduce the remaining frames to (3) A spatial token reduction mechanism based on temporal dependencies is applicd for excessively long vidcos. As a result, our model capable of processing ffps sampled video input with high performance, which can adaptively reduce the number of tokens per frame to 2 on average to accommodate an hour- long video for MLLM within 8k context length.
The development of LLMs has significantly advanced VLMs. Kosmos-1 (Huang et al., 2023; Peng et al., 2023) introduces end-to-end framework that integrates visual input with LLM in a cohesive training regime. Flamingo (Alayrac et al., 2022) and BLIP-2 (Li et al., 2023a) merge visual and linguistic features through cross-attention and a Q-Former module, respectively. MiniGPT-4 (Zhu et al., 2023) and LLaVA (Liu et al., 2024b) simplify inte- gration by projecting visual features directly into the LLM embedding space using an MLP.
Later studies (Chen et al., 2023b; Peng et al., 2023; Wang et al., 2023; Chen et al., 2023a) have expanded LMM ap- plications to broader multi-modal tasks, enhancing spatial perception through visual grounding. Recent efforts (Liu et al., 2024a; Zhang et al., 2024b; Dong et al., 2024) aim to create general models that unify diverse tasks, employ- ing sophisticated optimization techniques, high-quality mul- titask datasets, and complex training strategies to boost performance across extensive vision-language tasks. Cam- brian (Tong et al., 2024) combines features from multiple vision encoders with Spatial Vision Aggregator (SVA) for a more capable MLLM.
a low-resolution token representation.
2

---

Long VU: Spatiotemporal Adaptive Compression for Long Vidco-Language Understanding
Spatial Token Compression
Large Language Model
Text Query
Spatiotemporal Adaptive Token Compression
Spatial Compression
V
V/2
V/
...
Vx-2
VN-1
Vi
Text Query
Temporal Reduction
Temporal Reduction
did put on the pizza?
SigLIP + DINOv2
Figure 2. Architecture of Long VU. Given densely sampled video frames, first utilize DINOv2 (Oquab et al., 2023) prior to remove redundant frames, and fuse the remaining frame features from both SigLIP (Zhai et al., 2023) and DINOv2 (Oquab et al., 2023), deseribed in Section 3.1. Then selectively reduce visual tokens via cross- modal query, detailed in Scction 3.2. Finally, demonstrated in Scction 3.3, we conduct spatial token compression based on temporal dependencics to further meet the limited context length LLMs.
2.2. Video Large Language Models
into shorter clips, processes each segment independently. and retrieves the most relevant segment in response to user queries. MA-LMM (He et al., 2024) maintains a memory bank aggregate long-term video without exceeding the context length of LLMs. LongVILA (Xue al., 2024) ex- tends the number of video frames 2048 enabling 2M context-length training.
Recent advancements in MILMs have broadened their appli- cation to video understanding tasks. Video LMMs process videos by extracting and encoding frames, then rearranging these as final video features. Several works (Li al., 2023c; 2024b; Cheng al., 2024; Korbar al., 2024), use the Q-Former-like (Li al., 2023a) to merge visual and text features, while others (Lin et al., 2023; Luo al., 2023; Ataallah et al., 2024a) concatenate frame features directly.
Our work focuses maximizing the preservation of frames in video content (1fps) within given context length by proposing spatiotemporal compression of video tokens.
When processing lengthy videos, the constraint on context length inevitably causes trade-off between the number of tokens per frame and the number of frames input. Most existing works (Li al., 2023c: Ataallah al., 2024a; Cheng al., 2024; Zhang al., 2024c; Li et al., 2024a) address this challenge by uniformly sampling frames from the video, which, however, results in a significant loss of visual details within the video. Video-ChatGPT (Maaz et al., 2023b) employs pooling modules to reduce data dimen- sions, enhancing processing efficiency. Other works try to preserve the maximum number of frames in video content. LLaMA-VID (Li et al., 2023d) employs additional text decoder embed the text query for cross-attention between frame features and compress the context token to one to- ken per frame, while MovieChat (Song et al., 2023) and TimeChat (Ren et al., 2023b) develop memory modules and timestamp-aware encoders to capture detailed video con- tent. Golfish (Ataallah al., 2024b) segments long videos
2.3. Video Token Compression
Recent methods have explored dynamic image tokens (Ma et al., 2023; Xu et al., 2022; Bolya al., 2022) or video tokens (Lee et al., 2024; Ren al., 2023a; Choi al., 2024) within the Transformer (Vaswani, 2017) framework. Chat-UniVi (Jin al., 2023) extends dynamic tokens for visual features in MILMs by merging K-nearest neighbor tokens across frame features of the video input. SlowFast- LLaVA (Xu et al., 2024) uniformly samples 8 frames for high-resolution tokens, while performing spatial pooling to decrease the number of tokens frames sampled at higher frame rate. our work, we propose spatiotemporal adaptive token reduction strategy that leverages both cross- modal query and inter-frame dependencies. This approach effectively mitigates temporal redundancy in video content, thereby enabling the accommodation of long videos within
3

---

LongVU: Spatiotemporal Adaptive Compression for Long Vidco-Language Understanding
limited context length.
gregate visual features from multiple vision encoders. We denote the fused frames features as V {V] .. VT).
3. Method
3.2. Selective Feature Reduction via Cross-modal Query
We propose spatiotemporal adaptive compression in three steps to effectively process long video, as shown in Fig- ure Initially, we implement a temporal reduction strategy on the frame sequence by leveraging the prior knowledge from DINOv2 (Oquab et al., 2023) (Section 3.1). Then, we selectively preserve full tokens for key frames via cross- modal query, while applying spatial pooling to reduce the remaining frames into low-resolution token representations (Section 3.2). Furthermore, we implement a spatial token reduction mechanism based on inter-frame temporal depen- dencies (Section 3.3).
After temporal reduction, we obtain set of fused frame features from both vision encoders, denoted as V {V), [V.,..,V') E [BTT where HA x Wh. de- notes the spatial dimension of the frame features, and D, indicates the channel dimension of the frame feature after SVA. If the concatenated frame features exceed the limited context length, i.e., T x Hn, X Wh. 2 Lmmz. we develop a selective compression strategy for certain frames, in order to capture both the detailed spatial semantic and long-range temporal context.
To achieve this, we propose using text query to help reduce spatial tokens of certain frames from H4, x Wh to Hr x Wi. Given the LLM embedding of the text query Q E IR R.,xDD,, , where Ly is the length of text query and D, is the dimensionality of LLM's s embedding space, we strategically choose N₂, frames to preserve their original token resolution, while the remaining undergoes a process of spatial pooling to achieve a reduced resolution. The selection mechanism is based on the cross-modal attention scores between each frame feature and the text query. The number of frames to keep original resolution can be formulated as,
3.1. Frame Feature Extractor and Temporal Reduction
DINOv2 (Oquab et al., 2023), through its self-supervised (SSL.) training with a feature similarity objective on vision- centric tasks, can effectively capture subtle frame differ- ences and low-level visual features. In contrast, CLIP- based (Zhai et al., 2023; Radford et al., 2021) models are trained with tic space, excelling at language alignment while sacrificing low-level features as shown in Figure 5. Moreover, Cam- brian (Tong et al., 2024) discovered that combining features from both SigLIP (Zhai al., 2023) and DINOv2 (Oquab et al., 2023) leads to significant performance boost in vision-centric tasks. Therefore, we pioneer to leverage both base model DINOv2 (Oquab et al., 2023) with vision- language contrastive-based model SigLIP (Zhai et al., 2023) as frame feature extractors for MLLM in video understand- ing task.
contrastive loss in the seman-
To
1
ΣF(V)TT
H Wh hiwdd
(I)
N₁ = max (0, T-max -- THW
HAW, - H.W
where Lmar is the maximum context length, F(-) denotes a multi-layer perceptron (MLP)-based multimodal adapter designed to align visual features with the input space of the LLM. Note that we omit the system prompt in the in- struction template for Equation I simplification. If N,, O, indicating that no frames are selected for retention at their original resolution, we will skip the computation of atten- tion scores and will directly perform spatial pooling across all the frames to the lower resolution.
Note that processing the entire long video can be com- putationally expensive. Given lfps-sampled video with N frames, denoted as I = [I",..,,'' J. we first use DINOv2 (Oquab et al., 2023) to extract features from each frame, leading to a set of DINO features (Vino...... }. We then calculate the average similar- ity sim' = J- Ej=1oz; sim dima: Vilno) within each non- overlapping window with J = 8 frames and reduce frames that exhibit high similarity with other frames. This step significantly reduces video redundancy by temporally com- pressing the original N frames to T frames, which reduces approximately half of the video frames, as detailed in Sec- tion 4.6.
....,
3.3. Spatial Token Compression
As previously discussed, there are cases where the concate- nated visual features with low resolution tokens still exceeds the maximum context length, i.c., T H₁ x Wi 2 Limas Under these circumstances. further token compression is necessary. We partition the sequence of frame features into non-overlapping segments with sliding window of size K < T. within which we conduct spatial token compres- sion (STC). The first frame in each window retains its full token resolution. We then compute the cosine similarity between the first frame and subsequent frames within the
We then extract features of the remaining T frames using SigLIP (Zhai et al., 2023) vision encoder, resulting in T features (Velg--., VT }- Subsequently, following Cam- brian (Tong et al., 2024), we combine these two types of visual features via Spatial Vision Aggregator (SVA) (Tong al., 2024) that employs learnable queries spatially ag-
4

---

LongVU; Spatiotemporal Adaptive Compression for Long Video-Language Understanding
window, conducting an element-wise comparison of spatial tokens between the first frame and its successors. Spatial tokens that exhibit a cosine similarity sim(:.. greater than the threshold 0 with the corresponding tokens of the first frame at the same spatial location will be pruned, which can be formulated as,
2024) (1 min ~ 1 hour) and MLVU (Zhou al., 2024) (3 mins 2 hours) are long video benchmarks for assessing long video understanding ability. For VideoMME (Fu et al., 2024), videos are officially split based on duration, which contains a subset of long videos ranging from 30 minutes to hour. We perform standardized evaluations using greedy decoding (num_beams= beams=1) 1) and benchmark our results against other open-source and proprictary models.
o8 4- vz (h(h,w) w) sim(v1 (h, w); ca(h,,)) 0
4.3. Implementation Details
otherwise
vh E [1, Hi), we E Wi), [2,4]
We use SigLIP (Zhai et al., 2023) (so400m-patch I 14-384) and DINOv2 (Oquab et al., 2023) as the vision encoder while choose Qwen2-7B (Qwen, 2024) and Llama3.2 2- 3B (Llama, 2024) as our language foundation model. We only compute cross-entrope loss for autoregressive text gen- eration. We use Adam (Loshchilov, 2017) optimizer with a cosine schedule for all the trainings. In the image-language pre-training stage, we train the model for one epoch with global batch size of 128. The learning rate is set to 1c-5, and the warmup rate is 0.03. The number of tokens per image are set to 576. For the video-language finetuning stage, we train the model for one epoch with global batch size of 64. The learning rate is set to 1c-5, and the warmup rate is 0.03. The maximum number of tokens per frame are set to 144 (H, = Wn = 12), while each might be reduced by our proposed adaptive compression approach (s 64, Hi W1 = 8). The DINO threshold is set 0.83 and the STC reduction threshold is 0 0.75. The sliding window size K = 8. Our model is trained on 64 NVIDIA H100 GPUs.
Given that videos often contain significant pixel-level redun- dancy, particularly in static background, this method allows spatial tokens reduction via temporal dependencies. We chose the first frame in each sliding window for comparison, assuming DINOv2 (Oquab et al., 2023) has effectively re- duced video redundancy across frames, making each frame less similar. We also tested alternative strategies, like using the middle frame or adaptively selecting based on frame changes (Section 4.5), but these provided similar perfor- mance and compression rates. Therefore, we chose the first-frame strategy in each sliding window for its simplicity and effectiveness.
4. Experiments
4.1. Datasets
We adopt two stages of training in our experiments: image- language pre- pre-trrining and video-language finetuming. For the image-language pre-training stage, previous meth- ods (Chen et al., 2023b; Peng et al., 2023; Wang et al., 2023; Chen et al., 2023a; Liu et al., 2024a; Dong et al., 2024) usu- ally use two steps for aligmment and finetuning. For simplic- ity, we combine these two steps in one stage using Single- Image data from LLaVA-One Vision (Li et al., 2024a). For video-language finetuning, we utilize large-scale video- text pairs sourced from several publicly accessible databases. The video training data contains a subset of VideoChat2- IT (Li et al., 2024b), which includes TextVR (Wu et al., 2025), Youcook2 (Zhou et al., 2018), Kinetics-710 (Kay et al., 2017), NEXTQA (Xiao al., 2021), CLEVRER (Yi et al., 2019), EgoQA (Fan, 2019). TGIF (Li et al., 2016), WidV (Yang et al., 2021), ShareGPPTVideer (Chen et al., 2024a), and MovieChat (Song et al., 2024) as the long video complementary. All the training datasets are listed in Table 6.
4.4. Video Understanding
Quantitative Results. Table presents our experimen- tal results on multiple video understanding benchmarks. Our results compares favorably to all the baselines across various video understanding benchmarks. For example, on VideOMME (Fu et al., 2024), our LongVU outper- forms VideoChat2 (Li et al., 2024b), LLaVA-One (Li et al., 2024a) by 6.0% and 2.4% respectively. Notably, on VideoMME Long subset (Fu et al., 2024), our model sur- passes LLa LLaVA-Onee One Vision (Li et al., 2024a) by 12.8%. These results indicate the strong video understanding capabilities of our model. Note that our model achieves significant im- proved performance with a much smaller training dataset, comparing to LLVA-OOn Vision (Li et al., 2024a) trained One Vision- 1. 1.6M (multi-image, video) that has not yet been made publicly available'. With the same video training dataset from VideoChat2-IT IT (Li et al., 2024b), our Vu
4.2. Benchmarks and Metrics
'LLaVA-One Vision (Li et al., 2024a) only release ingle-image set at the time of current submission. http://uggnngfee..cccaatas////// LaV.l.LeVVV-O-iiiiiiii Data/discussions/6
We evaluate our model on EgoSchema (Mangalam et al., 2024), MVBench (Li et al., 2024b), VideoMME (Fu et al., 2024) and MLVU (Zhou et al., 2024). VideoMME (Fu et al.,
5

---

LongVU: Spatiotemporal Adaptive Compression for Long Vidco-Language Understanding
VideoMME
Models
Size
Context Length
#Frames
EgoSchema
MV AABennch
MIVU
Overall
Long
Duration
179.8
16
3~~120 min
1~60 min
30~60 min
Proprietary Models
GPT4-V (OpenA1, 2023)
lfps 1tps
55.6
43.7
60.7
56.9
GPT4-o (OpenAl, 2024)
72.2
64.6
77.2
72.1
Open-Source Video MI..Ms
Vidco-LLaVA (Lin al., 2023)
38.4
41.0
47.3
40.4
38.1
LLaMA-VID (Li 2023d)
1fps
38.5
41.9
Chat-UniVi (Jin ct al., 2023)
7B
4k
64
45.9
41.8
SharcGPT4Video (Chen et al., 2024a)
8B
8k
16
51.2
46.4
43.6
37.9
LLaVA-NeXT-Viddo (Zhang et al., 2024c)
43.9
33.7
46.5
VidcoLLaMMA2 (Cheng al., 2024)
51.7
54.6
48.5
46.6
43.8
LongVA (Zhang al., 2024a)
7B
224k
128
56.3
54.3
47.6
VidcoChat2 (Li ct al., 2024b)
7B
8k
16
54.4
60.4
47.9
54.6
39.2
LLaVA-One Vision (Li et al., 2024a)
56.7
64.7
58.2
46.7
Long VU (Ours)
8k
1ips
67.6
66.9
65.4
60.6
59.5
Table 1. Results comprchensive video understanding benchmarks
shows much higher performance than VideoChat2 (Li al., 2024b), 100 accuracy improvement in average.
Interestingly, we also find that our model can even beat pro- prietary model GPT4-V (I,enAI, 2023) on MVBench (Li et al., 2024b) with densely sampled video input and reduce accuracy gap compared to proprietary models on other video benchmarks.
00:33
you scentsffy the direction yellow sphere's viidect
The yel-ow sphere in eooting aeay tran the red
(a) Spatisl-tempora Orientation Awareniss
We also scale our LongVU with lightweight LLM, Llama3.2-3B (Llama, 2024), to further demonstrate the strong video understanding capabilities. We observe con- sistent improvement of our wighht VU over base- lines in Table Our method outperforms Phi-3.5-vision- instruct (Abdin et al., 2024) on VideoMME (Long) by a margin of accuracy of 3.4%. This set of experiments val- idates the effectiveness of our method even scaling to smaller size.
00:00:0
00:0022
Describe the video in detasl.
featuring fartastical settire, supgesting narrative adventure contiset. Liet, rarst cla cittrr dresses in yellow and eartial arts unifers with wosk, alder ean with vhibe bears and a bus robe, appears sirpeised or concerned. The lardscape,, indicating ing natural, possibly structures sountsiouus
characters
envirorecit.
(b) Video Detailed Descriation
We now provide the qualitative results in Figure: 3. Specifically, we demonstrate various video understanding abilities in the examples, such as accurately recognizing the orientation moving objects in Figure 3(a), providing detailed video descriptions in Figure 3(b), identi- fying inserted needle frames and conducting action counting in Figure 3(c), and responding precisely to questions about specific frames in an hour-long video in Figure 3(d). These results demonstrate that our model has competing video- language understanding capabilities.
Qualitative Results.
20 00
00:1033
this video, Fow nany timos does the scone the "cleasing toilet" actton appear
(c) Action Action Counting
boach ar the vudoo?
The vides provides in aertal view of the beach.
Video Understanding
4.5. Ablation Studies
Effects of the number of tokens per frame. We ablate the number of tokens in our uniform -amoomismmpiing baselines.
Figure 3. Examples for various video understanding capabilities of Long model. We showcase that our LongVU is able to completing different types video understanding tasks.
6

---

LongVU; Spatiotemporal Adaptive Compression for Long Vidco-Language Understanding
VideoMME
Models
EgoSchema
MVBench
MLVU
Overall
Long
InternV1.2 (IntemL.M2-1.BB) (OpenGVLab, 2024)
60.2
47.3
42.6
VideoChal2 (Phi-3-mini-44B) (Li et al., 2024b)
56.7
55.1
Phi-3.5-vision-instue (Phi-3-mini-4B) (Abdin et al., 2024)
50.8
43.8
LongVU (Ours) (Llama3.2-33B
59.1
60.9
51.5
47.2
55.9
Table 2. Results of small-size video language models across video understanding benchmarks.
There is a trade-off between the number of tokens per frame and the sampling frequency of frames. Table 3 shows the experimental results when using different number of to- kens with different sampling. When applying uniforming sampling. 144 tokens per frame shows better performance than 64 tokens in an 8k context length VideoMME (Fu et al., 2024) and MLVU (Zhou et al., 2024) while worse EgoSchema (Mangalam et al., 2024). With 144 tokens per frame, it preserves more visual details, but restricts the total number of frames, i.e., less than 60 frames within 8k context length. This demonstrate that adaptive tokens are needed for better performance across different video benchmarks.
Methods
Context Length
#Tokems
EguSchema
VideoMME
MLVU
Laifomn
64
57.56
60.87
Caifoon
8k
144
66.25
58.84
63.25
Caifomn
10k
144
67.12
00.01
64.70
TR (DINO)
144
TR (SigLIP)
8k
64
66.04
58.63
62.17
TR (DINO)
8k
64
66.20
99.90
62.54
TR (DINC) Query
64/44
67.50
60.08
65.05
(DINO) Query STC
dyramic
Table 3. Ablation studies number of tokens per frame, different context lengths, and our spatiotemporal compression components. TR denotes temporal reduction.
DINOv2 vs SigLIP. Our results in Table 3 verify that DI- NOv2 (Oquab et al., 2023) features more effective than SigLIP (Zhai et al., 2023) features. As expected, we also find that using DINO-based features for temporal frame reduction outperforms uniform sampling. Therefore, DI- NOv2 (Oquab et al., 2023) an useful vision-centric feature extractor to help perform temporal reduction.
gies for determining anchor frames: the first/middle one in each sliding window, or the frame that exhibits significant changes compared its adjacent frames. In Table 5, our results indicate that taking the first frame in each sliding window gives slightly better performance with similar reduction rates across all strategies.
Query guided selection. We apply text-guided frame se- lection after temporal reduction, where relevant frames are maintained at full token capacity (144 tokens), while oth- ers are reduced to 64 tokens. This helps preserve essential visual features and accommodates more long-range con- text within the context length. In Table 3, we observe the improvement with query guided frame selection across all benchmarks. Moreover, in Table 4, the results of each sub- task in MLVU (Zhou et al., 2024) show significant per- formance improvements when using cross-modal queries, particularly for frame-retrieval tasks such as counting and needle detection.
Please refer to additional ablation studies on threshold, con- text size, and window size in Appendix
4.6. Spatiotemporal Compression Analysis
Compression analysis. We sampled hundreds of videos to demonstrate the distribution of frame/token reduction rate. Figure (a) presents the number of frames before and after temporal reduction based on the similarity of DINOv2 features across frames. We find that ~45.9% of the frames are maintained after temporal reduction average. Figure 4 (b) shows the number of tokens before and after spatial token compression (Section 3.3). We observe that 4000.4% tokens are reduced on average. These results demonstrate the effective video token compression with temporal and spatial token reduction.
Spatial token compression. We further apply spatial to- ken compression after query guided selection. We find that spatial token compression (STC) not only enhances perfor- mance within 8k context length, but also achieve results comparable or slightly better than 16k context length in Ta- ble 3. We also note some improvements for most subtasks in MLVU (Zhou et al., 2024).
Long context analysis. Recently, the Keedle-in-a-Haysttee task (Hsieh et al., 2024; Kamradt., 2023) has been used to assess the ability of Large Language Models (LLMs) to retrieve long context information. We follow (Zhang et al., 2024a) to conduct a video needle-in-at -aadle-i----hasstaae experiment to demonstrate the effectiveness of our compression strategy on identifying the needle frame within an hour-long video.
Different strategies for spatial token compression. We now ablate different strategies of our spatial token compres- sion mechanism. This analysis explores different strate-

---

LongVU: Spatiotemporal Adaptive Compression for Long Vidco-Language Understanding
Stratgy
count
ego
needle
order
plotQA
anomaly
reasoning
Avg
DINO
24.15
59.09
68.16
52.89
71.24
74.00
86.36
62.54
DINO+Query
28.98
55.39
78.87
56.37
72.35
75.50
87.87
65.05
DINO+Query+STC (default)
28.98
59.37
76.33
58.30
71.61
76.00
87.50
65.44
Table Ablation study on each subtask MLVU (Zhou et al., 2024).
Model
Short
Medium
Long
Overall
Reduction rate
1s frame in sliding window (default)
64.7
58.2
59.5
60.9
55.47%
(K/2)th frame in sliding window
64.7
58.7
58.6
60.7
54.97%
frame with high changes
64.7
58.2
58.3
60.4
55.62%
Table 5. Different strategies for spatial token compression on VideoMME (Fu et al., 2024).
compoeasion
1000
20000
20000
Numberd of tomee
Number of Takors
10000
6000
213
Video Duration
afte DINO
(a)
(b)
Figure randomly sample hundreds videos demonstrate frames/tokens level reduction rate. (a) The number frames before/after temporal reduction based on DINOv2 features (Section 3.1). (b) The number of tokens before/after spatial token compression (Section 3.3).
To facilitate this evaluation, we randomly select an one-hour- long test video from MLVU. We then insert each image from of VQA problems as needle frame into this long video for creating challenging search task. We sample the video at FPS and control the frame length ranging from 200 to 3.6k frames. We also vary the needle frame insertion depth from 0% to 100% of the total input frames.
5. Conclusion
We introduced LongVU, a MLLM that can address the significant challenge of long video understanding within limited context length. To achieve this, we proposed a spatiotemporal adaptive compression scheme of LongVU for helping reduce video tokens without losing much vi- sual details of long videos by leveraging cross-modal query and inter-frame similarities. Experiments various video understanding benchmarks consistently validate the advan- tages of our model. We also demonstrate that our method helps build quality light-weight video language under- standing model based on Llama3.2-33 2-3BB, which suggests that VU has many potential applications in the vision- language community.
conduct experiments with 8k context length and com- pare our adaptive token compression the one without applying query-guided selection (w/o Query) and spatial token compression (w/o STC) after temporal reduction. Fig- ure 6 demonstrates that our adaptive compression mech- anism could accurately resolve the needle VQA problem of 1k frames within 8k context length and improve score with more frames. This demonstrates the advantage of our method for long context video understanding.
8

---

LongVU: Spatiotemporal Adaptive Compression for Long Vidco-Language Understanding
Impact Statement
Chen, K., Zhang, Z., Zeng, W., Zhang, R., Zhu, E, and
Zhao, R. Shikra: Unleashing multimodal Ilm's s referential dialogue magic. arXiv preprint arXiv: 15195, 2023b.
Our work introduces a spatiotemporal adaptive compression mechanism that reduces the number of video tokens while preserving visual details of long videos. Our open-sourced model paves the way for future research in video compres- sion tailored for MLLM-based applications, enabling more effective long-video, media, and streaming video under- standing.
Chen, L., Wei, X., Li, J., Dong, X., Zhang, P., Zang, Y., Chen, Z., Duan, H., Lin, B., Tang, Z., et al. Sharegpt4video: Improving video understand- ing and generation with better captions. arXiv preprint arXiv:2406.04433 04335. 2024a.
Chen, L.,, Zhao, H., Liu, T., Bai, S., Lin, J., Zhou, C., and Chang, B. An image is worth 1/2 tokens after layer 2: Plug-and-play inference acceleration for large vision- language models. In European Conference on Computer Vision, pp. 19-35. Springer, 2024b.
References
Abdin, M., Jacobs, S. A., Awan, A. A., Aneja, J., Awadallah, A., Awadalla, H., Bach, N., Bahree, A., Bakhtiari, A., Behl, H., et al. Phi-3 technical report: A highly capable language model locally on your phone, arXiv preprint arXiv:2404. 14219, 2024.
Chen, Z. Wu, J., Wang, W., Su, W., Chen, G., Xing, S., Muyan, Z., Zhang, Q., Zhu, X., Lu, L., et al. In- ternvl: Scaling up vision foundation models and align- ing for generic visual-linguistic tasks, arXiv preprint 23..v:23/2.14220 2023c.
Achiam, J., Adler, S., Agarwal, S., Ahmad, L.,, Akkaya, I., Aleman, F. L., Almeida, D., Altenschmidt, J., Altman, S., Anadkat, S., ct al. Gpt-4 technical report. arXiv preprint 1.Xi::2303.08774.. 2023.
Cheng, Z., LLng, S., Zhang, H., Xin, Y., Li, X., Chen, G., Zhu, Y., Zhang, W., Luo, Z., Zhao, D., et al. Videollama 2: Advancing spatial-temporal modeling and audio under- standing in video-Ilms. arXiv preprint arXiv:2406. 074766 2024.
Alayrac, J.- Donahue, J., Luc, P., Miech, A., Barr, I., Hasson, Y., Lene, K., Mensch, A., Millican, K., Reynolds, M.. et al. Flamingo: visual language model for few- shot learning. Advances in neural information processing systems, 35:23716 1997166-37666 2022.
Chiang, W.-L., Li, Z., Lin, Z., Sheng, Y., Wu, Z., Zhang, H., Zheng, L., Zhuang, S., Zhuang, Y., Gonzalez, J. E., Stoica, I., and Xing, E. P. Vicuna: An open- source chatbot impressing gpt-4 with 90%* chatgpt quality, 2023, URL https: // Imsysmms.ss.... org.llog/g// 2023 103-00330-vioee 33-00033--vvuua///
Ataallah, K., Shen, X., Abdelrahman, E.. Sleiman, E., Zhu, D., Ding, J., and Elhoseiny, M. Minigpt4-video: Advancing multimodal lims for video understanding with interleaved visual-textual tokens. arXiv preprint arXiv:2404.00 034133, 2024a.
Ataallah, K., Shen, X., Abdelrahman, E., Sleiman, E., Zhuge, M., Ding, J., Zhu, D., Schmidhuber, J., and EI- hoseiny, M. Goldfish: Vision-language understanding of arbitrarily long videos. arXiv preprint arXiv:2407.12679, 2024b.
Choi, J., Lee, S., Chu, J., Choi, M., and Kim. H. J. vid- tldr: Training free token merging for light-weight video transformer. In Proceedings of the IEEF/CVF Conference on Computer Vision and Pattern Recognition, pp. 18771- 18781, 2024.
Bai, J., Bai, S., Yang, S., Wang. S., Tan, S., Wang, P., Lin, J., Zhou, C., and Zhou, J. Qwen-vl: A frontier large vision- language model with versatile abilities. arXiv preprint arXiv:2308. 1 12966, 2023.
Dong, X., Zhang, P., Zang, Y., Cao, Y., Wang, B., Ouyang, L., Wei, X., Zhang, S., Duan, H., Cao, M., et al. Internlm- xcomposer2: Mastering free-form text-image composi- tion and comprehension in vision- languaee large model. arXiv preprint arXiv:2401.. 16420 2024.
Bolya, D., Fu, C.-Y., Dai, X., Zhang, P., Feichtenhofer, C., and Hoffman,, J. Token merging: Your vit but faster. arXiv preprint arXiv:22 Kiv:22/00094/ 00.6611, 2022.
Fan, C. Egovqa-an egocentric video question answering benchmark dataset. In Proceedings of the IEEF/CVF International Conference on Computer Vision Workshops, pp. 0-0, 2019.
Brown. T. B. Language models are few-shot learners. arXiv preprint arXiv:2005. 141655, 2020.
Chen, J., Zhu, D., Shen, X., Li, X., Liu, 7.,, Zhang, P., Krish- namoorthi, R., Chandra, V., Xiong, Y., and Elhoseiny, M. Minigpt-v2: large language model as a unified interface for vision- Langugee multi-task learning. arXiv preprint arXiv:2310. 10.09478, 2023a.
Fu, C., Dai, Y., Luo, Y., Li, L.,, Ren, S., Zhang, R., Wang, Z., Zhou, C., Shen, Y., Zhang, M., et al. Video-mme: The first-ever comprehensive evaluation benchmark of multi-modal 1lms in video analysis. arXiv preprint mxXiv:2405.21075 2024.

---

LongVU: Spatiotemporal Adaptive Compression for Long Vidco-Language Understanding
He, B., Li, H., Jang, Y. K., Jia, M., Cao, X., Shah, A., Shrivastava, A., and Lim, S.-N. Ma-Imm: Memory- augmented large multimodal model for long-term video understanding. In Proceedings of the IEEE/CVF Confer- ence on Computer Vision and Puttern Recognition, pp- 13504-13514, 2024.
Li, J., Li, D., Savarese, S., and Hoi, S. Blip-2: Bootstrapping language-iman pre-training with frozen image encoders and large language models. In International conference on machine learning, pp. 19730-19742. PMLR. 2023a.
Li, J., Li, D., Savarese, S., and Hoi, S. Blip-2: Boot- strapping language-image pre-training with frozen im- age encoders and large language models. arXiv preprint arXiv: 200::2301.129997 12597, 2023b.
Hsieh, C.-P., Sun, S., Kriman, S., Acharya, S., Rekesh, D., Jia, F., and Ginsburg, B. Ruler: What's 's the real context size of your long-cor context language models? arXiv 2024.
Li, K., He, Y., Wang, Y., Li, Y., Wang, W., Luo, P., Wang, Y., Wang, L.,, and Qiao, Y. Videochat: Chat-centrie video understanding. arXiv preprint arXiv:2305.0633
preprint arXiv:2404.0666
Huang. S., Dong, L., Wang, W., Hao, Y., Singhal, S., Ma, S., Lv, T., Cui, L., Mohammed, O. K., Patra, B., et al. Language not all you need: Aligning perception with language models. Advances in Neural Information Pro- cessing Systems, 36:72096-72109 2023.
2023c.
Li, K. Wang, Y., He. Y., Li, Y., Wang, Y., Liu, Y., Wang, Z., Xu, J., Chen, G., Luo, P., et al. Mybench: A compre- hensive multi-modal video understanding benchmark. In Proceedings of the IEEE//VV Conference on Computer Vision and Pattern Recognition, pp. 22195-22206, 2024b.
Jiang, A. Q. Sablayrolles, A., Roux, A., Mensch, A., Savary, B., Bamford, C., Chaplot, D. S., Casas, D. d. I., Hanna, E. B., Bressand, F,, et al. Mixtral of experts. arXiv:2401.0m0 0400888888 2024.
Li, Y., Song, Y., Cao, L., Tetreault, J., Goldberg, L., Jaimes, A., and Luo, J. Tgif: A new dataset and benchmark on animated gif description. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp. 4641-4650, 2016.
Jin, P., Takanobu, R., Zhang, C., Cao, X., and Yuan, L. Chat-univi: Unified visual representation empowers large language models with image and video understanding. arXiv preprint (rXx::2311.08045 2023.
Li, Y., Wang, C., and Jia, J. Llama-vid: An image is worth 2 tokens in large language models. arXiv preprint arXiv:2311.7044 2023d.
Kamradt., G. Needle in a haystack pressure testing Ilms, 2023. URL https://gittiei com/ http://aaa//ttt..o.///// gkamradt/ Seaassteeeeeeeeeeaaaateeeee
Lin, B., Zhu, B., Ye. Y., Ning, M., Jin, P., and Yuan, L. Video-llava: Learning united visual representa- tion by alignment before projection. arXiv preprint 1.Xiv:231110012 2023.
Kay, W., Carreira, J., Simonyan, K., Zhang, B., Hillier, C., Vijayanarasimhan, S., Viola, F., Green, T., Back, T., Natsev, P., et al. The kineties human action video dataset. arXiv preprint
Liu, H., Li, C., Li, Y., Li, B., Zhang, Y., Shen, S.. and Lee, Y. J. Llava-next: Improved reason- ing, and world knowledge, January 2024a. URL https://llava-ue http://llaaav...iitu..... 2024 01-00-1111a-----------
2017.
Korbar, B., Xian, Y.. Tonioni, A., Zisserman, A., and Tombari, F. Text-conditioned resampler for long form video understanding. In European Conference on Com- puter Vision, pp. 271-288. Springer, 2024.
Liu, H., Li, C., Wu, Q., and Lee, J. Visual instruction tun- ing. Advances in neural information processing systems, 36. 2024b.
Langley, P. Crafting papers on machine learning. Langley, P. (ed.), Proceedings of the 17th International Conference Machine Learning (ICML 2000), pp. 1207-1216, Stan- ford, CA, 2000. Morgan Kaufmann.
Liu, H., Yan, W., Zaharia, M., and Abbeel, World model on million-length video and language with ringattention. arXiv preprint arXiv:2402.0888 082688 2024c.
Lee, S., Choi, J., and Kim, H. Multi-criteria token fu- sion with one-step-ahead attention for efficient vision transformers. In Proceedings of the IEEE/CVF Confer- ence on Computer Vision and Pattern Recognition, pp. 15741-15750, 2024.
Llama, M. Llama 3.2. 2024.
URL https:
/ //huggingface mangginnface.commetee ww/.c//m///////// aamaa Llamm-3. 2-3B.
Loshchilov, I. Decoupled weight decay regularization. arXiv preprint arXiv: 171 1.05 2017.
Li, B., Zhang, Y., Guo, D., Zhang, R., Li, F., Zhang, H., Zhang, K., Li, Y., Liu, Z., and Li, C. Llava- onevision: Easy visual task transfer. arXiv preprint arXiv:2408. 03326, 2024a.
Luo, R. Zhao, Z. Yang, M., Dong, J., Qiu. M., Lu, P. Wang, T., and Wei, Z. Valley: Video assistant with
10

---

LongVU: Spatiotemporal Adaptive Compression for Long Vidco-Language Understanding
large language model enhanced ability, arXiv preprini wrxiv:2306.07207, 2023.
language supervision. In International conference on machine learning, pp. 8748 8763, 2021.
Ma, X., Zhou, Y., Wang, H., Qin, C., Sun, B., Liu, C., and Fu, Y. Image as set of points. In ICLR, 2023.
Ren, S., Chen, S., Li, S., Sun, X., and Hou, TESTA: Temporal-spatee token aggregation for long- form video-language understanding. arXiv preprint arXiv:2310. /9060, 2023a.
Maaz, M., Rasheed, H., Khan, S., and Khan, F. S. Video-chatgpt: Towards detailed video understanding via large vision and language models. arXiv preprint 3xXiv:3306.05422 2023a.
Ren, S., Yao, L., Li, S., Sun, X., and Hou, L. Timechat: A time-sensitive multimodal large language model for long video understanding. arXiv preprint arXiv:2312.0205/ 2023b.
Maaz, M., Rasheed, H., Khan, S., and Khan, F. S. Video-chatgpt: Towards detailed video understanding via large vision and language models. arXiv preprint arXiv:2306.0544 054244 2023b.
Song, E., Chai, W., Wang, G., Zhang, Y., Zhou, H., Wu, F., Guo, X., Ye, T., Lu, Y., Hwang, J.-N., et al. Moviechat: From dense token to sparse memory for long video un- derstanding. arXiv preprint arXiv:2307.16449 2023.
Mangalam, K., Akshulakov, R., and Malik, J. Egoschema: diagnostic benchmark for very long-form video lan- guage understanding. Advances in Neural Information Processing Systems, 36, 2024.
Song, E., Chai, W. Wang, G., Zhang, Y., Zhou, H., Wu, F., Chi, H., Guo, X., Ye, T., Zhang, Y., et al. Moviechat: From dense token sparse memory for long video under- standing. In Proceedings of the IEEE/CVF Conference Computer Vision and Pattern Recognition, pp. 18221- 18232, 2024.
OpenAI.
Introducing
chatgpt.
https://openaiico//llo////eeeee 2022.
OpenAL. Gpt-4v(ision) system card 2023.
URL https: /ttp:///ppp....////e//eo// cow.//searc///c/////
gpt-Av-systent card.
Tao, K., Qin, C., You, H., Sui, Y., and Wang, H. Dycoke: Dynamic compression of tokens for fast video large lan- guage models. arXiv preprint meXiv:2411.15000 2024.
OpenAl. Gpt-4o system card, 2024. URL https://
openai com/ www.//ooo..///////ee///ee/ee wennddeeeeellee htwdd///ehllggpto///////
OpenGVLab, T. Internv12: Better than the best expandnng performance boundaries of open-source multimodal models with the progressive scaling strategy, 2024. URL /ttpp::/innerrv11, http:/l..giittt wwt..g.tthh..og io/blog/////////ooo 2024-07-02-
Tong, S., Brown, E., Wu, P., Woo, S., Middepogu, M.,
Akula, S. C., Yang, J., Yang. S., Iyer, A., Pan, X., et al. Cambrian-1: A fully open, vision-centric exploration of multimodal llms. arXiv preprint arXiv:2406. 16860, 2024.
Touvron, H., Martin, L., Stone, K., Albert, P., Almahairi, A., Babaei, Y., Bashlykov, N., Batra, S., Bhargava, P., Bhosale, S., et al. Llama Open foundation and fine- tuned chat models. arXiv:2307.09288. 2023.
Oquab, M.. Darcet, T., Moutakanni, T., Vo, H., Szafraniec, M., Khalidov, V., Fernandez, P., Haziza, D., Massa, F., El- Nouby, A., al. Dinov2: Learning robust visual features without supervision. arXiv preprint arXiv:2304.07199, 2023.
Vaswani, A. Attention all you need. Advances Neural Information Processing Systems, 2017.
Ouyang. L., WU, J., Jiang, X., Almeida, D.,. Wainwright, C., Mishkin, P., Zhang, C., Agarwal, S., Slama, K., Ray, A., et al. Training language models to follow instructions with human feedback. Advances in neural information processing systems, 35:27730-27744 2022.
Wang. W., Lv, Q., Yu, W., Hong, W., Qi, Wang, Y., Ji, J., Yang, 7.,, Zhao, L.,, Song, X., et al. Cogylm: Visual expert for pretrained language models. arXiv preprint arXiv:2311.03077) 2023.
Peng, Z.., Wang, W., Dong, I.,, Hao, Y., Huang, S., Ma, S., and Wei, F. Kosmos-2: Grounding multimodal large language models to the world. arXiv preprint arXiv:2306. 14824, 2023.
Wang. Y., Li. K., Li, X., Yu, J., Hc, Y., Chen, G.. Pei, B., Zheng, R., Xu, J., Wang. Z., et al. Internvideo2: Scaling video foundation models for multimodal video understanding. arXiv preprint arXiv:2403. 15377, 2024.
Qwen, T. Qwen2 technical report, 2024.
Wu, W., Zhao, Y., Li, Z., Li, J., Zhou, H., Shou, M. Z., and Bai, X. large cross- moos- video retrieval dataset with reading comprehension. Pattern Recognition, 157: 1108 18, 2025.
Radford, A., Kim, J. W., Hallacy, C., Ramesh, A., Goh, G., Agarwal, S., Sastry, G., Askell, A., Mishkin, P., Clark, J., et al. Learning transferable visual models from natural

---

LongVU: Spatiotemporal Adaptive Compression for Long Vidco-Language Understanding
Xiao, J.. Shang, X., Yao, A., and Chua, T.-S. Next-qa: Next phase of question-answerigg explaining temporal actions. In Proceedings of the IEEE/CVF conference computer vision and pattern recognition, pp. 9777-9786, 2021.
Zhang, R., Gui, L., Sun, Z., Feng, Y., Xu, K., Zhang, Y., Fu, D., Li, C., Hauptmann, A., Bisk, Y., al. Di- rect preference optimization of video large multimodal models from language model reward. arXiv preprint urXiv:2404.01256 2024b.
Xu, J., De Mello, S., Liu, S., Byeon, W., Breuel, T., Kautz, J. and Wang, X. GroupViT: Semantic segmentation emerges from text supervision. In CVPR, pp. 18134- 18144, 2022.
Zhang, Y., Li, B., Liu, h., Lee, Y. j., Gui, L.,, Fu, D., Feng, J., Liu, Z., and Li, C. Llava-next: A strong zero-shot video understanding model, April 2024c. URL 30-11ava weelllaaaaaeettteee next-video/.
io/
Xu, M., Gao, M., Gan, Z., Chen, H.-Y., Lai, Z., Gang, H., strong
Zhou, J., Shu, Y., Zhao, B., Wu, B., Xiao, S., Yang, X., Xiong, Y., Zhang, B., Huang, T., and Liu, Z. Mlvu: A comprehensive benchmark for multi-task long video understanding. arXiv preprint arXiv:2406. 240.044400 2024.
Kang, K., and Dehghan, A. Slowfast-llawe training-free baseline for video large language models. arXiv preprint ar #rXi:2447.15844/ 2024.
Xue, F., Chen, Y., Li, D., Hu, Q.. Zhu, L., Li, X., Fang, Y., Tang, H., Yang, S., Liu, Z.. et al. Longvila: Scaling long- context visual language models for long videos. arXiv preprint arXiv:2408. 10188, 2024.
Zhou, L., Xu, C., and Corso. J. Towards automatic learning of procedures from web instructional videos. In Proceed- ings of the AAAI Conference on Artificial Intelligence, 2018.
Yang, A., Miech, A.. Sivic. J., Laptev, I., and Schmid, C. Just ask: Learning to answer questions from millions of narrated videos. In Proceedings of the IEEE/CVF international conference on computer vision, pp. 1686- 1697, 2021.
Zhu, D., Chen, J., Shen, X., Li, X., and Elhoseiny, M. Minigpt-4: Enhancing vision-language understanding with advanced large language models. arXiv preprint arXiv:2304 10592, 2023.
Yang. S., Chen, Y., Tian, Z., Wang, C., Li, J., Yu, B., and Jia, J. Visionzip: Longer is better but not necessary in vi- sion language models. arXiv preprint arXiv:2412.04467, 2024.
Ye, Q., Xu, H., Xu, G., Ye, J., Yan, M., Zhou, Y., Wang. J. Hu, A., Shi, P., Shi, Y., Li, C., Xu, Y., Chen, H., Tian, J., Qi, Q., Zhang, J. C., and Huang, F. mplug-owl: Modularization empowers large language models with multimodality, arXiv preprint arXiv:2304. 14178, 2023.
Yi, K. Gan, C., Li, Y., Kohli, P., Wu, J., Torralba, A., and Tenenbaum, J. B. Clevrer: Collision events for video representation and reasoning. arXiv preprint 1011: 2019.
Zhai, X., Mustafa, B., Kolesnikov, A., and Beyer, L. Sig- moid loss for language image pre-training. In Proceed- ings of the IEEE/CVF International Conference on Com- puter Vision, pp. 11975-11986 2023.
Zhang. H., Li, X., and Bing, L. Video-llama: An instruction- tuned audio-visual language model for video understand- ing. arXiv preprint 03282830660 2023.
Zhang, P., Zhang, K., Li, B., Zeng, G., Yang, J., Zhang, Y., Wang. Z., Tan, H., Li, C., and Liu, Z. Long con- text transfer from language to vision. arXiv preprint arXiv:2406. 16852, 2024a.
12

---

LongVU: Spatiotemporal Adaptive Compression for Long Vidco-Language Understanding
A. Training Datasets
For the image-language training stage, previous methods (Chen et al., 2023b; Peng et al., 2023; Wang et al., 2023; Chen et al., 2023a; Liu al., 2024a; Dong al., 2024) usually use two stages of alignment and finetuning. For simplicity, we combine and alignment in one stage using single image version of LLaVA-Onie One--on (Li al., 2024a) data. For vidco-language training. utilize large-scale video-text pairs sourced from several publicly accessible databases. The video training data is subset of VideoChat2-IT (Li et al., 2024b), which includes TextVR (Wu et al., 2025), Youcook2 (Zhou et al., 2018), Kinetics-710 (Kay et al., 2017). NExTQA (Xiao et al., 2021), CLEVRER (Yi et al., 2019), EgoQA (Fan, 2019), TGIF (Li et al., 2016), WebVidQA (Yang et al., 2021), ShareGPT4Video (Chen et al., 2024a), in addition to above, we use MovieChat (Song et al., 2024) as long video complementary. All the training data is demonstrated in Table 6.
Modality
Task
Samples
Dataset
Image-Text
Single-Image
3.2M
LLaVA-OneVisionn
Captioning
43K
TextVR, MovieChat, YouCook2
Classification
IK
Kinetics-710
NEXTQA, CLEVRER, EgoQA,
Video-Text
VQA
424K
TGIF, WebVidQA, DiDeMo
Instruction
85K
ShareGPT4Video
Table Training data statistics.
Model
Size
Frames
Short
Medium
Long
Overall
Video-LLaVA (Lin et al., 2023)
7B
46.1
40.7
38.1
41.6
ShareGPT4Video (Chen al., 2024a)
8B
16
53.6
39.3
37.9
43.6
Chat-Univi-v1..5 (Jin et al., 2023)
7B
64
51.2
44.6
41.8
45,9
VideoLLaMA2 (Cheng al., 2024)
7B
16
59.4
47.6
43.8
50.3
VideoChat2 (Li al., 2024b)
7B
16
52.8
39.4
39.2
43.8
LongVA (Zhang al., 2024a)
7B
128
61.6
50.4
47.6
54.3
LLaVA-OneVision (Li al., 2024a)
7B
69.1
53.3
46.7
58.2
VU (Ours)
7B
1fps
64.7
58.2
59.5
60.9
Table 7. Comparison with other video LMMs VideoMME (Fu al., 2024) benchmark.
B. Frame-level Position Encoding
To alleviate potential confusion arising from frame-hy-frame feature concatenation, we incorporate a frame-level position encoding to enforce the temporal boundaries across frames and capture inter-dependencies within each frame. Given that we temporally reduce several frames, straightforward concatenation of all frames renders the model unaware of the relative timestep across frames. Furthermore, our dynamic token sampling strategy does not delineate clear boundaries between each frame. To address this, we incorporate frame-level positional embeddings (FPE) that correspond to the absolute timestep of each frame, utilizing shared sinusoidal position encoding (Vaswani, 2017) for frames at time shown in Equation 3.
PE 21) sin(t/100000*/4) PF cos(t/10000
The ablation shows in Table and Table 9 that adding the FPE does not affect much to the overall performance across several benchmarks. Therefore, we decide not to include it in our default setting.
13

---

LongVU: Spatiotemporal Adaptive Compression for Long Video-Language Understanding
Methods
Context Length
#Tokens
EgoSchema
VideoMME
MLVU
DINO Query
8k
64/144
67.30
60.08
65.05
DINO + Query + STC (default)
8k
dynamic
67.62
60.56
63.44
DINO + Query + STC FPE
8k
dynamic
67.87
60.89
64.56
Table 8. Ablation study with without FPE.
Stratgy
count
ego
needle
order
plotQA
anomaly
reasoning
Avg
DINO
24.15
59.09
68.16
52.89
71.24
74.0
86.36
62.54
Quer+Query
28. 28.98
55.39
78.87
56.37
72.35
75.5
87.87
65.05
DINO+Query+STC (default)
28.98
59.37
76.33
58.30
71.61
76.0
87.50
65.44
DINO+Query+STCC+ FPE
29.46
60.79
74.08
52.12
71.79
74.5
86.74
64.56
Table 9. Strategy ablations each subtask in MLVU (Zhou et al, 2024).
C. DINOv2 v.s. SigLIP
DINOv2 (Oquab et al., 2023), through self-super training with a feature similarity objective on visually-centric tasks, captures subtle frame differences and low-level visual features more effectively than vision-language contrastive methods (Radford al., 2021; Zhai al., 2023), shown in Figure
SigLIP
015/56
DINOv2
0.9727
Figure S. Similarity comparison between SigLIP (Zhai al., 2023) and DINOv2 (Oquab et al., 2023) features. The similarity is calculated between the first frame and the remainings. DINO concentrating on vision centric task effectively capture subtle frame differences compared with SigLIP (Zhai al., 2023) which aligned on semantic space.
D. Needle-In-A-Viide--Haays-
We conducted experiments using an 8k context length to evaluate our default setting, which incorporates our adaptive compression, against configurations without spatial token compression (w/o STC) and without querying guided reduction (w/o Query), as depicted in Figure 6. By integrating cross-modal query to selectively retain full tokens of frames relevant the text query, the model significantly enhances its ability accurately identify key frames when the total number of video frames is fewer than 1.4k. Moreover, adaptive token compression mechanism further boosts VQA accuracy with increased frames.
E. Inference Time
To evaluate the computational overhead introduced by our proposed spatiotemporal compression approach, we compare with various baselines using input videos of the same length (20 minutes) sampled at FPS. The experiments were conducted
14

---

LongVU: Spatiotemporal Adaptive Compression for Long Vidco-Language Understanding
(a) Query
(b) Ours w/o STC
(o) Ours (default)
Figure Hocdie-i-AAVideeoHHaatttkk results. Our spatiotemporal adaptive token compression scheme improves the score for locating the needle frame.
Model
GQA
MMVP
POPE
RealWorldQA
Cambrian-1
64.04
32.67
58.95
LLaVA-NeXT
65.2
38.7
86.5
64.2
Ours (Before video SFT, w/o DINO)
61.9
31.3
85.6
59.5
Ours (Before video SFT)
62.26
51.33
86.65
61.06
Ours (After video SFT)
60.83
32.00
81.23
47.65
Ours (After video SFT + image SFT)
69.65
52.66
86.22
62.10
Table 10. We mainly focus on video understanding task and use video-only data for video SFT stage. We observe a decrease in performance image understanding after video SFT stage.
on an A100 GPU with 80 GB memory. LLaMA-VID (Li et al., 2023d) encounters CUDA out-of-memory (OOM) issue when processing 20-minute videos as input. To address this, we need to precompute the video features in advance and then load the entire model for inference. Our method demonstrates faster performance compared to the token compression approach of Chat-UniVi (Jin et al., 2023), which relies on a KNN-based strategy to merge similar tokens. Furthermore, is more efficient than the resampler- method VideoChat2 (Li al., 2023c), which compresses video inputs using learnable queries in Q-Former. When compared to methods without compression, such LLa VA-OOeOne Vision (Li et al., 2024a), approach is slightly slower, requiring 1.27x the processing time.
Models
LLaMA-VID
Chat-UniVi
VideoLLaMA2
VideoChat2
VLaVVAOneViin
Ours
Time (sec)
55.3
49.06
58.62
45.22
25.84
32.96
Table 11. Inference time comparison on 20 minutes videos. All models take frames sampled at 1 input, approximately 1200 frames.
15

---

Lo Spatiotemporal Adaptive Compression for Long Vidco-Language Understanding
We begin by using the DINOv2 vision encoder to extract features from all frames and then reduce redundant frames based DINO feature similarity. After this reduction, the remaining frames are processed using SigL.P. One significant advantage of method is that the DINO-based frame reduction step substantially decreases the computation required for the remaining frames in subsequent steps. As shown in the table below, the primary computation lies in frame feature extraction, which, in real-world applications, can be preprocessed offline. Notably, our proposed compression component contributes only a small portion to the overall inference overhead.
Component
Extract all frame features
DINO similarity
Extract remaining feature for SigLIP
Query
STC
Time (sec)
22.2
1.05
4.32
0.27
0.18
Table 12. Inference time of each component.
F. Experiments
F.1. Comparison with training-free token compression methods.
Methods
MVBench
MLVU
VidcoMME
FastV (Chen et al., 2024b)
56.1
62.6
57.3
VisionZip (Yang et al., 2024)
56.9
62.5
57.8
DyCoke (Tao et al., 2024)
58.2
63.8
60.4
Long V (Ours)
66.9
65.4
60.6
Table 13. Comparison with training-free token compression methods.
F.2. Context Length
We conducted an ablation study compare the impact different context lengths on our method Table 14. By default, use an 8K context length, it aligns with most baselines and is the standard for commonly used MLLMs, ensuring fair comparison. When reducing the context length 6K, performance suffers due to the restricted context window. Conversely, testing model with extended context lengths of 12K and 16K yielded results comparable to the 8K setting, This demonstrates that our spatiotemporal compression approach effectively minimizes redundant video tokens, maintaining performance even with longer contexts. In addition, currently, there is scarcity of long videos with high-quality annotations that can support scaling our methods to longer context lengths. Most videos in our training dataset, VideoChat2-II, are limited minutes.
Context length
EgoSchema
MVBench
MLVU
VideoMME
6k
67.82
66.71
62.33
59.54
8k (default)
67.62
66.95
65.44
60.56
12k
67.14
66.83
63.54
60.12
16k
67.20
66.86
64.40
60.20
Table 14. Context length ablation.
F.3. DINO Threshold
We investigate the impact of DINO threshold selection on performance Table 15. lower DINO threshold results fewer frames being discarded (lower compression rate). For example, in MLVU, decreasing the threshold leads to performance drop as more redundant frames are retained given context length.
16

---

LongVU: Spatiotemporal Adaptive Compression for Long Vidco-Language Understanding
DINO threshold
EgoSchema
MVBench
MLVU
VideoMME
0.9
67.64
66.88
64.33
60.3
0.85
67.66
66.86
63.12
59.9
0.83 (default)
67.62
66.95
65.44
60.56
0.8
67.18
66.86
63.51
60.34
0.75
67.22
66.86
63.16
60.38
Table 15. DINO threshold ablation.
F.4. STC Threshold
We analyze the impact of STC threshold selection on performance and observe a trade-off with the compression rate as shown in Table 16. Increasing the threshold, meaningly reducing the compression rate, leads drop performance, particularly on the long-video benchmarks MLVU and VideoMME. Conversely, lowering the threshold results more aggressive compression, which also negatively impacts performance.
STC threshold
EgoSchema
MVBench
MLVU
VideoMME
0.85
67.56
66.88
64.00
59.98
0.8
67.30
66.86
63.51
59.83
0.75 (default)
67.62
66.95
65.44
60.56
0.7
67.50
66.86
64.03
60.27
0.65
67.42
66.86
63.91
60.34
Table 16. STC threshold ablation.
F.5. Sliding Window Size K
K represents the size of the sliding window, a hyperparameter used to divide the frame sequence into chunks for spatial token compression. We show the ablation results for different values of K in Table 17. Adjusting the sliding window size will affect the MLVU performance, while having minimal impact for other benchmarks. In STC, each subsequent frame is compared with first frame as anchor within the sliding window. We assume that larger context window can weaken the strong temporal dependency when the subsequent frame is far from the initial frame in the window.
Sliding window K
EgoSchema
MVBench
MLVU
VideoMME
4
67.38
66.86
63.74
60.45
8 (default)
67.62
66.95
65.44
60.56
16
67.22
66.86
62.18
60.42
32
67.20
66.86
60.69
60.82
Table 17. Sliding window K ablation.
F.6. Sliding Window Size
represents the size of the sliding window, a minor hyperparameter used to divide the video into chunks. We present ablation results demonstrating the effects different values for J Table 18. Adjusting the sliding window size has minimal impact on the benchmarks.
17

---

LongVU: Spatiotemporal Adaptive Compression for Long Vidco-Language Understanding
Sliding window
EgoSchema
MVBench
MLVU
VideoMME
4
67.54
66.88
63.79
60.6
8 (default)
67.62
66.95
65.44
60.56
16
67.56
66.86
64.3
60.16
32
67.54
66.83
63.38
60.23
Table 18. Sliding window J ablation.
F.7. Effect STC
The primary goal of STC is to reduce tokens while preserving visual information, rather than achieving a significant performance improvement. The table below shows the retained token ratio before and after STC, demonstrating that while the number of token is significantly reduced, the model's performance effectively preserved.
Methods
Retained Ratio
EgoSchema
VideoMME
MLVU
LongVU VU w/o STC Lo
23.9%
67.3
60.1
65.1
14.2%
67.6
60.6
65.4
Table 19. Sliding window J ablation.
G. Limitation
Our research primarily concentrated on video understanding tasks, for which we employ video-only data during the video supervised fine-tuning (SFT) stage. As evidenced in Table 10,, our image understanding capabilities. However, after fine-tuning on video tasks, modality gap leads decline in image performance. Interestingly, when the video SFT model is further fine-tuned on the image data from the pretraining stage, the image performance is effectively restored. This indicates that our approach to video understanding can also enhance image understanding capabilities. The recovered image understanding performance shown in below table is comparable to MLLM baselines, such as Cambrian-1 (Tong et al., 2024) and -LeTV--eXTT (Liu et al., 2024a), which primarily focus on image understanding and operate with a similar number of image tokens. A potential remedy could involve integrating mix of image, imatte,,,ee,,eee and video data during training. However, due to constraints in GPU resources, we leave it future work with larger datasets for stronger unified image and video models.
model demonstrates good
Our method spatiotemporally reduces video frames/tokens and concatenates tokens all together to form the overall video representation. However, this approach does not encode the temporal location of individual frames. While we experimented with frame-level positional embeddings to alleviate this drawback. the model still struggles with tasks like temporal grounding, meaningly identifying the precise start and end times of events.
think that a well-designed frame-level positional embedding could help address this issue. Alternatively, explicitly adding <frame_i> text to demonstrate the timestamp of each frame or overlaying visual text on the frames indicate their timestamps could also be potential solution.
18


Reviews:
[{"review_id": "0ainuissBC", "reviewer": "Reviewer_7Nad", "summary": "The paper proposes LongVU, a method that addresses the challenge of processing long videos within multimodal language models' limited context by implementing a three compression approaches: reducing temporal redundancy through inter-frame similarity, leveraging cross-modal dependencies, and eliminating spatial token redundancy. This framework successfully processes hour-long videos within standard 8k context lengths, outperforming competing methods across multiple benchmarks while maintaining high-quality video understanding.", "questions": "", "limitations": "", "rating": 4, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "**1. LLama-VID OOM issue.**\n\nThanks for your concern. We encountered an OOM issue while running on an 80GB A100 GPU for long videos, using the same settings as other comparison models. To address this, we need to precompute the video features in advance and then load the entire model for inference, as shown in LLaMA-VID ([long-video-inference]([https://github.com/dvlab-research/LLaMA-VID?tab=readme-ov-file#long-video-inference](https://github.com/dvlab-research/LLaMA-VID?tab=readme-ov-file#long-video-inference))). The inference time for 20-minute videos is 55.3 seconds, which will be updated in Table 11.\n\n**2. The main contribution has already been proposed in prior work for either image VLMs or other Video VLMs.**\n\n**2.1. For example, both [1] and [2] use DINO features alongside the CLIP features.**\n\nWhile MoF [1] and Cambrian-1 [2] utilize both DINO and CLIP features, neither explores their combination for video training and video token compression—where careful design is crucial for practical training and good performance. Simply applying both vision features to all video frames is computationally demanding, making long-video training impractical.\n\nOur work goes beyond merely using both encoders for video understanding—we introduce a novel strategy to reduce computation demand and conduct adaptive video token compression for MLLMs. We identify a key insight: vision-centric encoders trained with feature similarity objectives, e.g., DINOv2, excel at frame reduction in the visual space, while CLIP-based features, optimized for vision-language alignment, are suboptimal for this task. To our knowledge, this has not been explored before for video token reduction. In addition, by leveraging DINOv2 to selectively filter redundant frames before extracting SigLIP features, we significantly reduce computational costs and make long video training feasible.\n\n\n**2.2. Using text query to select optimal tokens is also covered in [3, 4] that the authors do not seem to discuss.**\n\nThe claim that Goldfish uses text query to select optimal tokens is inaccurate, as its selection mechanism does not operate on visual representations. Instead, it retrieves text descriptions based on query similarity. **As we have already discussed in lines 151-154**, Goldfish chunks long videos into short clips, generates descriptions with VideoLLM, and encodes them using OpenAI’s text-embedding-3-small. Then it retrieves most relevant descriptions based on similarity between the query's embedding and clip embeddings for the final answer. In contrast, whereas our method selects video tokens, making its approach fundamentally different from ours.\n\nIn [3], the Text-Conditioned Resampler (TCR) employs learnable queries to resample a long sequence of video frame features into a fixed-length sequence, which is then fed into a language model. This approach is quite similar to QFormer [5] but is applied in the video domain. However, TCR enforces long video compression into a limited and fixed token space, limiting its ability to represent complex visual information and ignoring temporal variations—resulting in redundancy in static scenes and loss of detail in complex ones. \n\nIn contrast to [3], our method dynamically compresses video tokens based on intrinsic dynamics, adapting to each video's complexity.\nWe will integrate the above discussion in our revision.\n\n[5] Blip-2: Bootstrapping language-image pre-training with frozen image encoders and large language models\n\n**3. The unique contribution that is different from previous work?**\n\n**(1) Vision-centric visual encoders for video understanding.** To the best of our knowledge, no previous work has explored leveraging a vision-centric visual encoder like DINOv2 for frame reduction. While methods like MovieChat and Chat-UniVi reduce redundancy by calculating frame-level feature similarities, they rely on CLIP features. We argue that language-supervised methods like CLIP fail to capture the nuanced differences between frames.\n\n**(2) Context-aware dynamic token compression.** Many token compression methods, like TCR, VideoChat, and VideoChat2, rely on QFormer-based modules, which limit video representation by transforming it into a fixed number of trainable queries. These approaches overlook the video's dynamic and informational richness, regardless of their complexity. In contrast, our method adaptively compresses the video both spatially and temporally, dynamically adjusting the number of tokens based on the video's inherent visual complexity and redundancy.\n\n**(3) Stronger performance with lower cost.** Compared to Chat-UniVi, which extends ToMe-based dynamic token merging for visual features in MLLMs, our method is both more efficient and more effective. As shown in Table 11, it processes over 1K video frames in 32.96 seconds, compared to Chat-UniVi’s 49.06 seconds, while achieving a substantial 17.7% accuracy gain in VideoMME long.", "claims_and_evidence": "The major claims about the method's performance on long videos are supported by results across multiple benchmark datasets. Additionally, the authors have provided detailed ablations on all their components to provide more insight into their method.", "methods_and_evaluation_criteria": "Yes, all major benchmark datasets are covered. However, I am concerned about the inference reported in the supplementary material. Authors report that LLama-VID is OOM on 20-minute videos. But the LLama-VID paper reports performance on hour-long videos. Is there any error here?", "theoretical_claims": "The mathematical representations are fine. No major theoretical proofs in the paper.", "experimental_designs_or_analyses": "The experimental design and analysis are all technically sound and thorough. All major datasets concerning long video evaluations are covered and experimental analysis is provided in detail.", "relation_to_broader_scientific_literature": "This is my major point of concern. The main contribution of the paper revolves around token reduction based on dino features and adaptive reduction based on the text query. These components have already been proposed in prior work for either image VLMs or other Video VLMs. For example, both [1] and [2] use DINO features alongside the CLIP features. Similarly, the idea of using text query to select optimal tokens is also covered in previous published work on videos [3, 4] that the authors do not seem to discuss.\n\nThis reduces the proposed work to a more engineering solution that combines ideas from previous image or video-based work. There is no doubt that the results are outstanding, but the core contributions leading to strong performance have already been proposed in a similar form in previous work, or adapted for similar work in image-based models.\n\n[1] Eyes Wide Shut? Exploring the Visual Shortcomings of Multimodal LLMs\n\n[2] Cambrian-1: A Fully Open, Vision-Centric Exploration of Multimodal LLMs\n\n[3] Text-Conditioned Resampler For Long Form Video Understanding\n\n[4] Goldfish: Vision-Language Understanding of Arbitrarily Long Videos", "essential_references_not_discussed": "I believe that there is a missing discussion on two works, that also propose a similar text-conditioned token reduction for videos.\n\n[1] Text-Conditioned Resampler For Long Form Video Understanding\n\n[2] Goldfish: Vision-Language Understanding of Arbitrarily Long Videos", "other_strengths_and_weaknesses": "Strengths:\n\nThe paper is well-motivated and well-written. The overall method achieves strong performance on all major benchmark datasets. The method is also supported by detailed ablations regarding each component of the method.\n\nWeaknesses:\n\nMy major concern is only regarding the technical novelty of the method. As of right now, it seems that the method is a strong engineering solution to the problem of long video understanding in VLMs, but the major components have all been proposed in previous published work. Whether it is the concept of combining DINO and CLIP features [1, 2], the idea of selecting tokens based on text queries [3, 4], or even the concept of reducing temporal resolution before processing the video [5]. Therefore, I believe that the novelty of the method is quite limited with all components proposed in some form in previous work.\n\n[1] Eyes Wide Shut? Exploring the Visual Shortcomings of Multimodal LLMs\n\n[2] Cambrian-1: A Fully Open, Vision-Centric Exploration of Multimodal LLMs\n\n[3] Text-Conditioned Resampler For Long Form Video Understanding\n\n[4] Goldfish: Vision-Language Understanding of Arbitrarily Long Videos\n\n[5] 3D CNNs with Adaptive Temporal Feature Resolutions", "other_comments_or_suggestions": "N/A", "questions_for_authors": "Can the authors discuss each component of their method in light of previous work on similar components? What has already been and what is the unique contribution that is different from previous work? Is the unique contribution really a major research contribution of incremental in nature? Because this is my only major concern right now. Other than that, the results are good and the work is well presented.", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {"ethical_review_concerns": "N/A"}}, {"review_id": "P3HvJoRQ4W", "reviewer": "Reviewer_ow6N", "summary": "This paper introduces a novel spatiotemporal token compression method named LongVU, designed for long video understanding. Specifically, LongVU divides the video token compression process into three stages: Temporal Reduction, Selective Feature Reduction, and Spatial Token Compression.\n\nIn the Temporal Reduction stage, an additional DINOv2 visual encoder is introduced to eliminate frame-level redundancy based on similarity. The Selective Feature Reduction stage incorporates the user’s query to identify the most relevant frames, preserving their resolution while reducing the resolution of less relevant frames. Finally, the Spatial Token Compression stage partitions the video into temporal windows and performs spatial compression at corresponding positions within each window.\n\nThe experimental evaluation in this paper assesses the performance of the proposed LongVU on four public long video understanding benchmarks, comparing it with other state-of-the-art methods.\n\n##Update after rebuttal: I appreciate the authors' detailed response, most of my concern is settled. I will keep the original rating as \"Accept\".", "questions": "", "limitations": "", "rating": 4, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "**1. The drawbacks of uniform sample and dense sampling.**\n\nNumerous studies, including LLaVA-OneVision, LongVA, and SlowFast-LLaVA, have explored the trade-offs between these approaches. Below, we present results of the baseline LongVA using either uniform sampling or dense sampling (1fps, with truncation at the end). LongVA extends the context length to 224K, yet it still cannot fully process long videos at 1 fps, requiring truncation. Performance peaks around 128-frame sampling and then decreases.\n\n| #Frames  | VideoMME (Long) | VideoMME (Avg) |\n| --- | --- | --- | \n| 32 (uniform) | 45.4 | 51.8 |\n| 64 (uniform) | 45.0 | 52.4 | \n| 128 (uniform) | 46.2 | 52.6 |\n| 384 (uniform) | 46.1 | 51.8 |\n| 1fps (dense) | 44.9 | 51.5 |\n\n\n\n**2. Lacks a thorough discussion on the visual encoders, and analysis on DINOv2/SigLIP solely for feature extraction.**\n\nWe appreciate the reviewer's concern regarding the discussion on visual encoders. However, we emphasize that our approach primarily focuses on the adaptive video token reduction, rather than an exhaustive exploration of visual encoders. \n\nIn the initial stage, we explored various vision encoders, including SAM, DINOv2, and SigLIP. Due to space constraints, we present the ablation studies conducted during the image pretraining phase, where we compare the performance of SigLIP alone with that of a SigLIP and DINOv2 combination. \n\n| Method | GQA  | MMVP | POPE | RealWorldQA |\n| --- | --- | --- | --- | --- | \n| SigLIP | 61.9 | 31.3 | 85.6 | 59.5 |\n| SigLIP + DINOv2 | 62.3 | 51.3 | 86.7 | 61.1 |\n\nThe results clearly demonstrate that combining both vision encoders leads to superior performance, resulting in a more robust image understanding model. SigLIP alone ranked as the runner-up. In contrast, DINOv2 and SAM, trained without language supervision, consistently showed an average accuracy more than 5% lower than that of SigLIP.\n\n**3. Comparing with training-free token compression methods.**\n\nThank you for sharing these methods. We reported the results in the table below and will incorporate them into our revision.\n\n| Model | MVBench | MLVU | VideoMME |\n| -------- | ------- | ------- | ------- | \n| FastV | 56.1 | 62.6 | 57.3 |\n| VisionZip | 56.9 | 62.5 | 57.8 |\n| DyCoke | 58.2 | 63.8 | 60.4 |\n| LongVU (Ours) | 66.9 | 65.4 | 60.6 |\n\n**4. Quantitative statistics on the number of tokens and the compression ratio.**\n\nWe already illustrated the number of tokens before and after reduction for different video durations in Figure 4. Below we provide the average compression ratio for the VideoMME dataset below for each compression step.\n\n| Temporal Frame Reduction | Query-based Token Reduction | STC |\n| -------- | ------- | ------- | \n| 45.9% | 52.1% (23.9%) | 59.6% (14.2%) |\n\nAt the temporal frame reduction stage, 45.9% of tokens remain. In the query-based token reduction, 52.1% remain, and at the STC stage, 59.6% are retained, yielding the final retain ratio of 14.2%. This breakdown shows the adaptive nature of our approach and the significant compression achieved throughout the process.\n\n\n**5. The order of experiments within the table does not align with the sequence in the paper.**\n\n\nThe last three rows of Table 3 outline the sequence of our method: first, DINOv2 for temporal reduction, followed by query-based selection, and finally, STC reduction. The table also includes ablation studies on context length, tokens per frame, and the impact of using SigLIP or DINOv2 for temporal reduction. We apologize for any confusion caused by the table’s condensed format. To enhance clarity, we will reorganize it in the revision—grouping “Uniform” together, renaming \"DINO\" to \"Temporal Reduction (DINO),\" and \"SigLIP\" to \"Temporal Reduction (SigLIP).\"\n\n\n**6. The impact statement is missing.**\n\nOur work introduces a spatiotemporal adaptive compression mechanism that reduces the number of video tokens while preserving visual details of long videos. This innovation paves the way for future research in video compression tailored for MLLM-based applications, enabling more effective long-video, media, and streaming video understanding. Additionally, we contribute to the open-source community by providing datasets, benchmarks, and models to support the advancement of AI-driven video analysis. We envision a future where MLLMs can directly process compressed video formats aligned with VLLMs that account for spatiotemporal redundancy.\n\nWe will add an impact statement section in our revision.", "claims_and_evidence": "The paper claims that existing sampling schemes have drawbacks, namely that uniform sampling may cause some frames to be missed, and dense sampling may lead to the number of tokens reaching the limit prematurely, resulting in the video being truncated early. While this claim makes intuitive sense, it lacks further validation both qualitatively and quantitatively.", "methods_and_evaluation_criteria": "Yes. The evaluation benchmarks utilized in this paper include most common video understanding benchmarks, including Egoschema, mvbench, MLVU and Video-MME.", "theoretical_claims": "This paper does not make theoretical claims.", "experimental_designs_or_analyses": "I am satisfied with the most of the experiments in this paper except for the visual encoder part. \n\n1. Since this paper introduces an additional DINOv2 encoder, it highlights the importance of the visual encoder. However, the paper lacks a thorough discussion on the visual encoders, including why DINOv2 and SIGLIP were selected (beyond the brief description in Sec 3.1, A more comprehensive analysis, such as quantitative ablation studies on different visual projectors, would be better).\n\n2. Furthermore, the discussion of DINOv2 and SIGLIP in Tab. 3 is limited to their role in feature extraction during the temporal reduction stage. However, prior to the Selective Feature Reduction stage, the SVA is still employed for feature fusion (I hope I understand it correctly). Thus it lacks quantitative analysis when relying solely on DINOv2/SIGLIP for feature extraction throughout the pipeline.", "relation_to_broader_scientific_literature": "This paper proposes several novel solutions for feature extraction in long videos, which could be beneficial not only to the field of VLMs but also to the broader, more general domain of video processing.", "essential_references_not_discussed": "Does the authors consider comparing LongVU with some training-free token compression methods[1-4]?\n\n[1] VisionZip: Longer is Better but Not Necessary in Vision Language Models (2024)\n\n[2] [CLS] Attention is All You Need for Training-Free Visual Token Pruning: Make VLM Inference Faster (2024)\n\n[3] DyCoke : Dynamic Compression of Tokens for Fast Video Large Language Models (2024)\n\n[4] An Image is Worth 1/2 Tokens After Layer 2: Plug-and-PLay Acceleration for VLLM Inference (2024)", "other_strengths_and_weaknesses": "## Strength\n1. The ideas behind the spatiotemporal adaptive compression are considered intuitive and useful.\n2. In general, the paper is easy to follow.\n\n## Weakness\n1. I understand that the tokens filtered out are considered less important compared to those retained. However, the filtered tokens may still contain some useful information. Have the authors considered providing a discussion on the potential implications of discarding these tokens?\n2. LongVU requires retraining the model, which may limit its portability and practical value.", "other_comments_or_suggestions": "1. Although the authors conducted ablation experiments on the threshold and sliding window in the supplementary material, it would be more convincing to provide quantitative statistics on the number of tokens and the compression ratio after each stage of compression based on these ablation experiments.\n\n2. The description of Table 3 is somewhat unclear, as the order of experiments within the table does not align with the sequence in the paper. Also, the unclear description of specific experimental setup makes me somewhat confused.\n\n3. The impact statement is missing. It is recommended that the author add it.", "questions_for_authors": "I have no further questions.", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "hWrkasZszC", "reviewer": "Reviewer_9x5j", "summary": "This paper proposes to reduce long video context by: 1) temporal frame reduction based on DINO feature similarity to extract keyframes (DINO module), 2) cross-modal query (Query module) to capture important tokens, 3) Spatial Token Compression (STC module) to further reduce tokens for excessively long videos.\nThe technical novelty of the Query and STC module is limited. The ablation studies demonstrate that the improvements of STC module are weak and it sacrifices fine-grained abilities.\n\nGiven that the improvements of DINO module and Query module are significant, I lean towards weak accept if the concerns in STC module and advantages against FastV are solved.", "questions": "", "limitations": "", "rating": 3, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "**1. The core idea of Query and STC lacks substantial novelty. Cross-modal query are known to reduce sequence length long time ago[1], and sparsity in MLLM has already been explored by FastV [2]. This paper differs from FastV in that it compresses tokens before feeding into LLM, but the authors did not demonstrate its advantages either.**\n\nFastV is a token reduction method designed for image understanding in MLLMs, operating after the second layer of MLLMs by reranking visual tokens based on attention scores from remaining tokens. In DyCoke [3], FastV was compared as a baseline for long video understanding tasks. The results show that while FastV effectively compresses tokens to speed up inference, it sacrifices performance in long-video comprehension compared to the base model. \n\nAs a plug-and-play approach, we applied FastV to our LongVU model to compare its compression with our method.\n\n| Models | TFLOPs | VideoMME | \n| -------- | ------- | ------- |\n| LongVU w/o reduction + FastV | 34.4 | 55.1 |\n| LongVU w/ reduction (default) | 18.5 | 60.6 |\n\nFor our case, the limitation of FastV becomes more pronounced. Our approach performs dense sampling and reduces tokens before input to LLM, ensuring they fit within the LLM’s context length with lower complexity. In contrast, FastV still requires processing all video tokens (100K) in the first two layers, leading to a significant exceeding beyond the LLM's context size (8K). This discrepancy affects attention reliability for extremely long contexts that extend beyond the model’s training distribution. In addition, its attention-based token selection fails to account for the temporal nature of video.\n\nFastV is an effective token compression approach for image understanding, but not designed optimal for long video compression. In contrast, our method effectively compresses video tokens by considering the spatiotemporal dependencies inherent in video data, which not only reduces the number of tokens but also improves performance in long-video understanding tasks.\n\n[3] DyCoke: Dynamic Compression of Tokens for Fast Video Large Language Models, CVPR 2025\n\n**2. Improvements of the STC module are not significant. STC harms fine-grained temporal understanding abilities.**\n\nWe propose that STC is not primarily aimed at improving performance but at reducing video redundancy to minimize the number of tokens input into the LLM.\n\n| Methods | Retained Ratio |  EgoSchema | VideoMME | MLVU | \n| -------- | ------- | ------- | ------- | ------- |\n| w/o STC | 23.9% | 67.3 | 60.1 | 65.0 |\n| w/ STC | 14.2% | 67.6 | 60.6 | 65.4 |\n\nBy applying STC, we achieve a 40.4% reduction in tokens, ultimately retaining only 14.2% of the original tokens. It demonstrates that while the number of tokens is significantly reduced, the model’s performance is effectively preserved.", "claims_and_evidence": "Yes", "methods_and_evaluation_criteria": "Yes", "theoretical_claims": "No theoretical claims in this paper.", "experimental_designs_or_analyses": "Yes", "relation_to_broader_scientific_literature": "In terms of video understanding.", "essential_references_not_discussed": "No.", "other_strengths_and_weaknesses": "**Strength**\n- For long video understanding, the authors developed  1) temporal frame reduction based on DINO feature similarity to extract keyframes (DINO feature), 2) cross-modal query (Query module) to capture important tokens, 3) Spatial Token Compression (STC module) to further reduce tokens for excessively long videos.\n\n- The effectiveness of all the above modules is verified by ablation studies.\n\n\n\n**Weakness**\n\n- The core idea of Query and STC lacks substantial novelty. Cross-modal query are known to reduce sequence length long time ago[1], and sparsity in MLLM has already been explored by FastV [2]. This paper differs from FastV in that it compresses tokens before feeding into LLM, but the authors did not demonstrate its advantages either.\n- Improvements of the STC module are not significant. From Table 3, performance gain on EgoSchema, VideoMME, and MLVU ranges from 0.3%~0.5%. In MLVU, simply changing seed, resolution, or number of sampled frames brings much more variations.\n- STC harms fine-grained temporal understanding abilities. In Table 4, performance in Needle QA and Plot QA drops significantly when introducing STC. Both subtasks are on the fine-grained temporal understanding, especially Needle QA. In another saying, the paper sacrifices fine-grained abilities since they account for only small portions on the benchmark.\n\n\n\n[1] X-Pool: Cross-Modal LanguageVideo Attention for Text-Video Retrieval. CVPR 2022.\n\n[2] An Image is Worth 1/2 Tokens After Layer 2: Plug-and-Play Inference Acceleration for Large Vision-Language Models. ECCV 2024.", "other_comments_or_suggestions": "N/A", "questions_for_authors": "1) Since dropping / compression tokens in MLLM is not a new idea, the authors should demonstrate their advantages against previous methods like FastV. I think efficiency might be a point, as LongVU compresses tokens before feeding LLM.", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "MWC3AbgQfE", "reviewer": "Reviewer_YjAF", "summary": "This paper introduces LongVU, a video-LLM designed for Long Video-Language Understanding.  LongVU is built on three main components: (1) DINOv2 Features, which is used to remove redundant frames that exhibit high similarity, (2)Text-Guided Cross-Modal Query, which can selective frame feature reduction and (3)Spatial Token Reduction, which allows spatial tokens reduction via temporal dependencies. Experiments mainly focus on video understanding tasks, testing the model’s Spatial-temporal Orientation Awareness, Video Detailed Description, Action Counting, and Hour-long Video Understanding, demonstrating the superiority of LongVU for long video understanding.", "questions": "", "limitations": "", "rating": 3, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "**1. The paper claims that STC helps compress video tokens, but it does not provide quantitative details on how many tokens are actually reduced.**\n\n\nPlease refer to Figure 4(b), where we illustrate the reduction in the number of tokens after applying STC across different video durations. On average, we observe a 40.4% reduction in tokens during STC. The primary goal of STC is to reduce tokens while preserving visual information, rather than achieving a significant performance improvement. The table below shows the retained token ratio before and after STC, demonstrating that while the number of token is significantly reduced, the model’s performance is effectively preserved.\n\n| Methods | Retained Ratio |  EgoSchema | VideoMME | MLVU | \n| -------- | ------- | ------- | ------- | ------- |\n| w/o STC | 23.9% | 18.5 | 67.3 | 60.1 | 65.0 |\n| w/ STC | 14.2% | 67.6 | 60.6 | 65.4 |\n\n\n**2. Limited Task Scope. Other video understanding tasks like: Video captioning.**\n\nOur video training data, sourced from VideoChat2, includes extensive captioning annotations, as shown in Table 6. Our model is capable of generating detailed video captions, as demonstrated in the qualitative results presented in Figure 3. \n\nRecent long video benchmarks adopt the VideoQA format for easier and more stable evaluation, since automatic caption evaluation is challenging, and GPT-4 based evaluation is not cost friendly. MLVU offers subsets like Video Summarization and Sub-Scene Captioning for captioning tasks. Below, we report MLVU video captioning results, with G-Avg representing the average performance across generation tasks.\n\n\n| Models | MLVU (G-Avg) ↑ |\n| -------- | ------- |\n| ShareGPT4Video | 3.7 |\n| VideoLLaMA2 | 3.9 |\n| VideoChat2 | 3.9 |\n| LongVU (Ours) | 4.1 |\n\n\n**3. Lack of list of main contributions.**\n\nWe believe we developed notable contributions that have been thoughtfully developed and carefully integrated to achieve strong performance. We are clarifying the contributions here.\n\n+ **Vision-centric visual encoders for video understanding.** We identify a key insight: vision-centric encoders trained with feature similarity objectives, e.g., DINOv2, excel at frame reduction in the visual space, while CLIP-based features, optimized for vision-language alignment, are suboptimal for this task. To our knowledge, this has not been explored before for video token reduction.\n\n+ **Context-aware dynamic token compression.** Our method adaptively compresses the video both spatially and temporally by dynamically adjusting the number of tokens based on the video's inherent visual complexity and redundancy, considering both spatiotemporal nature of video and semantic relevance to the user’s query.\n\n+ We conducted extensive experiments across several video understanding benchmarks, including EgoSchema, MVBench, VideoMME, and MLVU. Our LongVU method significantly outperforms multiple recent open-source video LLM models and also demonstrates strong effectiveness in smaller models (3B).", "claims_and_evidence": "Claims supported by clear and convincing evidence:\n\nClaim: The proposed LongVU significantly improves the performance of existing visual-language models (VLMs) on video understanding tasks.\n\nEvidence: The paper provides extensive experimental results in Table 1 and Table 2, demonstrating performance gains on multiple benchmarks such as EgoSchema, MVBench, MLVU and VideoMME. The comparison with baseline models clearly shows the superiority of the LongVU.", "methods_and_evaluation_criteria": "Proposed Methods:\nThe proposed method, LongVU, aims to enhance video understanding by:\nInjecting temporal information into large vision-language models (VLMs). Minimizing additional computational overhead while improving performance. Generalizing across various video understanding tasks.\nVideo understanding inherently requires capturing both spatial and temporal features, which LongVU addresses. Reducing the cost of adaptation without compromising performance is crucial for large-scale VLMs.", "theoretical_claims": "N/A", "experimental_designs_or_analyses": "**Strengths**\n\nThe paper provides a reasonably well-structured experimental design, with the following positive aspects:\n\n**Baseline Comparisons:**\nThe papaer compare LongVU with several state-of-the-art large vision-language models (VLMs).\n\n**Multiple Datasets:**\nThe experiments are conducted on well-known datasets like EgoSchema, MVBench, and VideoMME.\nThese datasets are widely accepted benchmarks for video understanding tasks, making the evaluation relevant and convincing.\n\n**Evaluation Metrics:**\nThe primary metric used is Accuracy (Acc), which is standard for VideoQA tasks.\nAlthough limited, the choice of this metric is reasonable for the problem setting.\n\n**Ablation Study:**\nThe paper conducts ablation studies to isolate the contribution of LongVU. This strengthens the validity of their claims.\n\n**Weaknesses**\n\n**No Clear Analysis of STC’s Token Reduction**\nThe paper claims that STC helps compress video tokens, but it does not provide quantitative details on how many tokens are actually reduced.\nTable 3 shows that DINO + Query and DINO + Query + STC have very similar performance (only 0.3% - 0.5% difference), raising the question of whether STC meaningfully contributes to efficiency or accuracy.\nWithout token reduction statistics, it is unclear whether STC is significantly improving computational efficiency or if it has a negligible effect.\n\n**Limited Task Scope**\nThe paper claims LongVU is generally applicable to various video understanding tasks.\nHowever, all experiments focus solely on VideoQA tasks, without evaluating performance on other video understanding tasks like: Video captioning (describing video content). Action recognition (classifying actions from videos). This narrow task scope weakens its claim of generalizability.", "relation_to_broader_scientific_literature": "N/A", "essential_references_not_discussed": "N/A", "other_strengths_and_weaknesses": "**Strengths**\n\nThe paper presents a novel model LongVU, which effectively reduces video tokens while preserving crucial visual details. This is an innovative approach compared to existing uniform or dense sampling strategies.\nThe use of cross-modal query-based compression is a unique way to adaptively reduce spatial and temporal redundancy in videos, an aspect that has not been fully explored in prior works.\n\n**Weaknesses**\n\nLack of a Clearly Stated Contribution Section\nThe paper does not explicitly list its main contributions in a separate section or clearly structured paragraph in the introduction.While the methodology is described in detail, the lack of a concise summary of contributions makes it difficult to quickly understand what is novel and significant about this work.", "other_comments_or_suggestions": "N/A", "questions_for_authors": "Would the compression strategy require re-tuning for different types of videos (e.g., static surveillance footage vs. fast-moving sports videos)?", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Meta-review: This paper presents an adaptive compression algorithm for long video understanding. It reveals that by compressing spatiotemporal visual information, the modern MLLMs can get more video context as input and thus improve long video understanding. This is a solid contribution to the community. Considering the unanimous acceptance across all reviewers, the AC finds no reason to overturn the decision.
Decision: Accept (poster)