Long Context Transfer from Language to Vision
Authors: Peiyuan Zhang
Kaichen Zhang
Bo Li
Guangtao Zeng
Jingkang Yang
Yuanhan Zhang
Ziyue Wang
Haoran Tan
Chunyuan Li
Ziwei Liu
Venue: Accepted by TMLR ()

Abstract: Video sequences offer valuable temporal information, but existing large multimodal models (LMMs) fall short in understanding extremely long videos. Many works address this by reducing the number of visual tokens using visual resamplers. Alternatively, in this paper, we approach this problem from the perspective of the language model. By simply extrapolating the context length of the language backbone, we enable LMMs to comprehend orders of magnitude more visual tokens without any video training. We call this phenomenon \textit{long context transfer} and carefully ablate its properties. To effectively measure LMMs' ability to generalize to long contexts in the vision modality, we develop V-NIAH (Visual Needle-In-A-Haystack), a purely synthetic long vision benchmark inspired by the language model'
s NIAH test. Our proposed Long Video Assistant (LongVA) can process 2000 frames or over 200K visual tokens without additional complexities. With its extended context length, LongVA achieves state-of-the-art performance on Video-MME and MLVU among 7B-scale models by densely sampling more input frames.

Published in Transactions on Machine Learning Research (07/2025)
Long Context Transfer from Language to Vision
Peiyuan Zhang"
peiguan.zhanggmnie edu.sg
Lab & S-Lob, Nonyang Technological University
Kaichen Zhang'
zhan0564 Sentu...d
Labs & L-b, Nanyang Technological University
Bo Li
libo0013 (0lt00010e..ttll Ge..... calu..edu...
LMMs- Lab 8 S- Lab, Nanyang Technological University
Guangtao Zeng
guanglao _ecei_eeegggg_ zeng seng@myyail.ssuu sutd. edu.s9
Singapore University of Technology and Design
Jingkang Yang
yangjinggkan0000 @grmail com
LMMs-Lob Lab & S-Lab, Nangang Technological University
Yuanhan Zhang
juanhann0028enntu.cdeu
LMbs & S-bob, Nanyang Technological University
Ziyue Wang
Eigue.wargg De. we... sg
S-Lab, Nanyong Technologieal University
Hoaran Tan
haoran. Cetoan.taan@Baoo w.w.mmii.lololll.cooo
S-Lab, Nanyang Technological University
Chunyuan Li LMMs-Lab
lichungguan24 @grmail. com
Ziwei Liu
ziwei. liu@tuu edu.sg
LMMs-Lab & Lab, Nanyang Technological University
Reviewed on OpenReview: https: // openreview. net/ forum? id= 3ORAWQVG1
Abstract
Vidoo sequences offer valuable temporal information, but existing large multimodal models (LMMs) fall short in understanding extremely long videos. Many works address this by reducing the number of visual tokens using visual resamplers. Alternatively, in this paper, approach this problem from the perspective of the language model. By simply extrapolating the context length of the language backbone, we enable LMMs to comprehend orders of magnitude more visual tokens without any video training. We call this phenomenon long contezt transfer and carefully ablate its properties. To effectively measure LMMs ability to generalize to long contexts in the vision modality, we develop V-NIAH (Visual Needle- In (t-A-Haystack); & purely synthetic long vision benchmnrkk inspired by the language model' s NIAH test. Our proposed Long Video Assistant (LongVA) can process 2000 frames or over 200K visual tokens without additional complexities. With its extended context length, LongVA achieves state-of-the the--ar--hh-rtttt performance on Video-MME and MLVU among 7B-scale models by densely sampling more input frames.
Equal contribution.
1

---

Published in Transactions on Machine Learning Research (07/ (225))
1 Introduction
Driven by the progress of Large Language Models (LLMs) (Brown et al., 2020; Team, 2023; Touvron et al., 2023; Gemini, 2024; Ormazabal ct al., 2024; Mistral, 2024; Cohere, 2024), multiple studics are conducted to extend their capability to understand images and videos (Li et al., 2023b; Dai et al., 2023; Qwen, 2024; Liu et al., 2023c). With modality alignment and visual instruction tuning, these Large Multimodal Models (LMMs) have shown impressive abilities such as captioning and visual question-ansseringg While current. LMMs have demonstrated promising performance tasks involving single images and short videos (Song et al., 2024b; Lin et al., 2023a; Maaz et al., 2023; Zhang et al., 2023), effectively processing and understanding extremely long vidcos remains a significant challenge (Wang ct al., 2024b).
One primary reason for this challenge is the excessive number of visual tokens generated by the vision encoder. For instance, LLa (Liu et al., 2024b) can produce 576 to 2880 visual tokens for a single image. The number of visual tokens increases significantly with the addition of more frames. To address this problem, numerous methods have been proposed to reduce the number of visual tokens. One popular direction is to modify the visual resampler that connects the vision encoder and LLM, aiming to extract fewer tokens (Li et al., 2023b;d; Cai et al., 2024; Cheng et al., 2024). Alternative approaches (Chen et al., 2024a; Shang et al., 2024; Jin et al., 2024; Zhou et al., 2024b) employ heuristic techniques to prune or merge the visual features. However, despite these efforts, Table demonstrates that the majority of current LMMs are still limited in their ability to process a large number of frames effectively.
Another issue hindering the development of perforrrfee long video LMMs is the lack of high-quality long video datasets. In Table 2, we list the average video length of existing video instruction tuning data. Most datasets consist of video clips within I minute. Even if some datasets do contain longer videos, the corresponding text. pairs are generated by annotating only several frames within that video, lacking long and dense supervision signals.
Given the circumstance, in this paper, instead of reducing the visual tokens, we identify the more critical issue limiting the visual context length in existing LMMs: the context length of the language model backbone. Given a language model, we first extend its context length by training on longer text data. We then use this context-extended LM as the backbone to perform modality alignment and visual instruction tuning without any long video text pairs. By training this way, the context length of the language model is directly transferred to that of the LMMs. We further propose UniRes, unified encoding scheme that represents videos as extended images, enhancing the capability fusion between images and videos. To facilitate benchmarking and accurately assess the context length in the visual domain, we created V-NIAH, a synthetic visual benchmark based on the srelll-in-a-haystacc test. (Gregory, 2024) used in language models. Our model, Long Video Assistant (LongVA), is capable of accurately retricving visual information from 2000 frames or more than 200K visual tokens. Experiments show that additional frames during inference lead to improved performance on long video
benchmarks, and LongVA achieves sate-of-theeart performance among 7B models on the Vidco-MME (Fu et al., 2024a) and MLVU (Zhou et al., 2024a) datasct. In summary, our paper makes the following contributions:
(1) Long Context Transfer: We discovered the long contect transfer phenomenon where the context of the language model can be directly transferred to the modality-aligned multi-modal models.
(2) Visual Needle-In-A-Haystacl (V-NIAH): We propose the V-NTAH benchmark to test LMMs ability in locating and retrieving visual information over extremely long contexts.
(3) Long Video Assistant (LongVA):
With long contezt transfer and UniRes, we developed LongVA that can perceive more than 200K visual tokens, achieving SoTA performance on the Video-MME, and MIVU datasot.
2 Related Work
Vision Language Connector in Large Multimodal Models Existing studies explore different archi- tectures to extract and inject visual features into LLMs. One line of work (Alayrac al., 2022; Li al., 2023a; Awadalla et al., 2023; Laurençon et al., 2023), pioneered by Flamingo (Alayrac et al., 2022), adopts
2

---

Published in Transactions on Machine Learning Research (07/2025)
Model
Tokens /Taaness
Training Max Fraanes"
LM Backboue
LM Context Length
MPLUG-Owl-viieo (Ye et ol,, 2024)
266
LLaMA
4K'
MovieChat [Song al., 2024b)
32
Vicuma-v0
Viden-LLaV (Zhang al,, 2023)
49
Virana-1.5
4K
VideoChat (Li et al,, 2024b)
32 196
Vieuma-vi
2K
LLaVA-NeXT-Viae (Zhang es al., 202/b)
144
16
Vicuna-1.5
1K
ST-LLM (Liu al, 2024c)
256
16
Vicuna-1.1 1.1
2K
Video-LLaMA (Cheng et sl., 2021)
32
32
LLaMA-2
1K
Chat-UniVi (Jin et al., 2023)
112
64
Vicama-1.5
4K
TimeClut (Ben et al., 2024)
96
LLaMA-2
Video-ChatGPT (Masz al. 2023)
256
100
Vieuna-111
2K
MLMA--VD (Li et al,, 2023d)
300
Vicuma-1.5 1.5
4K
LongVA (Ours)
144
Qwen2- Qwen2-Exxended
22-K++
Table 1: To enable longer video inputs, previous works train fewer visual tokens increase the maximum frames during training. Our LongVA, the other hand, enables long video capability by extending the backbone language model. "We report it based the best available information from their paper released codebase.
Table 2: Existing Video SFT Datasets
Table 3: Video Benchmarks
Dotasct Namse
Vileo Length (sec.)
Text Length
Beschurack Name
Video Length (scc.)
VideoChatCPT-nog (Mass et al,, 2023)
123.4
68.0
VideoChatGPT (Mana ot sl, 2723)
108.0
MaVA-Homdd22550 (Zhong
202(x)
52.4
37.6
Ne/TQA (Nio al, 20016)
PPaeeoo et al., 2024b)
26.8
273.3
EgoSchema (Manglaa al, 2023)
170.8
Timell (Ron et al,, 2024)
19018
62.5
VidenMME (Pu et al., 20126a]
1017.0
VideoChat el at, 20203)
50.0
V-NIAH (Ours)
resampler to compress the visual feature and inserts cross-gated attention layers into the LLM. Some other works still use a reampler (Li et al., 2023b; Zhu et al., 2023; Qwen, 2024) while directly feeding the image feature into the imput layer of the language model. The LLaVA series (Liu et al., 2024b; 2023b;c) use simple and scalable design to directly project the image features into language model without any pooling resampling. When the field moves from image-only models to include multi-image and video inputs, more modifications to the visual language connector were proposed. Zhang et al. (2024b) and Cai et al. (2024) use a simple average pooling. Jin et al. (2024) dynamically drop the visual tokens. Cheng et al. (2024) adopt a spatial-temporal convolution to better capture the dynamics of video data and reduce feature size. Our proposed context transfer from text image is orthogonal to those works and can further enable LMMs to understand more frames.
Context Extrapolation in Transformer Transformer does not directly work on sequences longer than its training length. To alleviate that, various RoPF-based (Su et al., 2023) extension techniques (Chen et al., 2023a; bloc97, 2024; Rozière et al., 2024; Peng et al., 2023; Ding et al., 2024) have been proposed to allow for training-fiee free context extrapolation. Efforts have also been made on data curation (Fu al., 2024b; Xiong et al., 2023; Bai et al., 2024) and system optimization (Li et al., 2023c; Liu et al., 2023a; Jacobs et al., 2023) during long context training. There has been limited exploration of the context extrapolation in the domain of LMMs. Liu et al. (2024a) are closest to our work and train LMM with long context language models, but they do not benchmark the effective visual context length of their model.
Video Language Benchmarks Recent years have witnessed significant progress in Video Question- Auswering Antol et al. (2015). To accurately measure the progress of the video LMMs performance, researchers have developed various benchmarks encompassing a broad spectrum of tasks. These range from fundamental visual perception tasks such as activity recognition Yu et al. (2019a), concept detection (Xu et al., 2017), and counting (Jang et al., 2017), to more complex visual rcasoning tasks including compositional (Grunde- McLaughlin et al., 2021), causal (Xiao et al., 2021a; Yi et al., 2019; Xu el al., 2021), and situated reasoning (Wu et al., 2021). However, most of those benchmarks focus on short videos, lacking data and metrics to test LMMs' capability over a long context. Inspired by the NIAH test (Gregory, 2024) the language model community, we propose V-NIAH to benchmark LMMs ability over long visual inputs with the minimum overhead of data collection and human annotation. Several concurrent works also developed multimodal
3

---

Published in Transactions on Machine Learning Research (07 (07/202) 2025)
Long- Long-Context Language Model
Language Model
Language
Instruction
(b) LongVA Training Stage
Language Instruction
Frame
Compress
Module Q-Fommet)
Long-Context Language Model
Language Instruction
(a) Previous Solutions
(c) LongVA Inference Stage
to develop long vision models, previous studics proposed better visual resamplers to reduce the number of visual tokens. Right: LongVA approaches this problem from the angle of the language model. We leverage image data (short visual imput) align long-context LLM with vision. During the test time, LongVa VA can zero-shot process extremely long videos, thanks to the property of long context transfer.
Figure 1: Left:
versions of the Needle-in-a-hayaee hhysttacc test (Wang et al., 2024c; Zhou al., 2024a; Song et al., 2024a; Wang et al., 2024a). However, they only measure on several hundreds of frames and lack strong baseline to properly analyze the properties of visual context length.
3 Long Video Assistant
As in Figure 1, this paper centers around the hypothesis that if the modality of vision and language can be truly aligned, the capability to handle long contezts could also transfer from tezt to vision, and this could happen even without explicit long video training. Our methodology is thus very straightforward. Given a language model, we first perform long context training purely on language to extend its text context (Section 3.1). We then detail how augment. this language model with long visual capabilities by training solely short image data in Section 3.2.
3.1 Training Long Language Model
We use Qwen2-7B-Insttre
(Team, 2024) as the backbone language model and perform continued pretraining with a context length of 224K1 over a total of 900M lokens. We follow Xiong et al. (2023) to increase RoPE (Su et al., 2023) base frequency during the continued pertaining and specifically set it to 1B. A constant learning rate of lc-5 maintained for batch size of one million tokens across 1,000 training steps. Following Fu et al. (2024b), we construct the dataset used for long context training from Slimpajama (Cerebras, 2023) by upsampling documents longer than 4096 and keeping the domain mixture ratio unchanged. Multiple documents are packed into single sequence separated by BOS token.
We employed several optimization strategies to perform training on such long sequences. These includes FlashAttention-22 (Dao, 2023), Ring Attention (Liu et al., 2023a; et al., 2023c), activation checkpointing, and parameter offload (Rajbhandari et al., 2020). To balance the load across different GPUs, we shard the
with A100-80OG for Qwen-2-7B. We find that the embedding size significantly impacts the maximum sequence length optimized codebase. Qwen2 huge vocabulary of 152K tokens. For LLaMA2 with 32K vocahulary, we train it with 700K context length.
224K is the maximum
4

---

Published in Transactions on Machine Learning Research (07/2025)
Encoding Strategy
Used By
Number of Grids
Max Tokens
Video support
Rope
AnyRes
LLNNA-NeXT
4
2880
No
1D
UniRes
Ours
49
7056
Yes
ID
Native Res
Qwen2-VT.
NA
NA
Yes
3D
Table 4: Comparison of UniRes with existing encoding strategies. Unlike AnyRes and Native Res, our method compresses tokens via pooling, enabling higher token capacity and supporting video input.
sequence in vigzag way (Zhu, 2024) in ring attention. The resulting training framework is memory efficient. and maintains very high GPU occupancy. Note that we do not use any parameter-cffiient methods such LoRA (Hu et al., 2021) approximate attention (Child et al., 2019). With those optimizations, the compute used in long context training is minimal compared to that of language model pretraining, making it feasible for academic budgets. The long context training can finish in 2 days with 8 A100 GPUs.
In Figure 7, we evaluate the extended Qwen2 with the Needle-in-a-haystade (NIAH) test (AI, 2023; Gregory, 2024). It achieves perfect results within the training context length (224K) and generalizes even further. We find the vanilla NIAH to be relatively trivial benchmark and further test it with 5 distractors randomly inserted into the documents. The detailed configuration can be found in Scction 9.
3.2 Aligning Long Language Model Using Short Vision Data
Inspired by the AnyRes encoding scheme in LLaVA-NeXT (Liu et al., 2024b; Li et al., 2024a), we designed UniRes that provides a unified encoding scheme for both images and videos, as shown in Figure 2. Unlike AnyRes which retains a small base image and flattens ViT patches across the grids, UniRes removes the base image, flattens patches within each grid, and 2x2 pool the visual features by default (Section 10). This approach allows us to maintain consistent representation when extending image data into vidcos where multiple frames are viewed as multiple grids in a row.
Specifically, UniRes divides an imput image of resolution a x into smaller grids, each with a resolution of 336 x 336 pixels. This results in (a//336) x (b//336) grids. For very high-resolution images, we limit the maximum number of grids to 49, resizing images larger than this threshold. Each grid separately encoded using CLIP-VIT-L-336px (Radford et al., 2021) and then projected through 2-layer MLP to match the LM's input dimension, resulting in 576 features per grid. We then apply 2x2 average pooling, finally converting an a x b image into (a//330 (a//336) x (b// /336) x 144 tokens. During inference, this visual encoding scheme allows videos to be represented as very long images (even though we do not train on videos). An N-frame video is treated as an image of size 336* x (336 x N). divided into N grids where each grid corresponds to a video frame. Using CLIP encoding, MLP projection, and average pooling, an N-frame video is encoded into 144N visual lokens. To clearly ablate the long context transfer phenomenon from language vision, we adopt train short, test long protocol where we only use image-text data during training, but test on long videos. We trained our model using the same data recipe and two-stage training approach as VL. Our experiments show that compared to AnyRes, UniRes has slightly lower scores on low-resolution image benchmarks (Table 9) but performs better on V-NIAH (Figure 4) and Video-MME (Table 6). We believe the unified encoding scheme for images and videos is crucial, thus choosing this as the encoding scheme of LongVA. The image- text alignment can be finished in 1.5 days. With days for long context training on text, the total training cost. of LongVA is 3.5 days on 8x A100-80G. Compared to approaches such as VA-NVA- VidoooZadn et al., 2024b) that is trained on 128 H100, Long requires far less compute. In Table 5, we compare the training token counts of various video- language models, highlighting the efficiency of LongVA VA.
It is worth noting previous work largely inspired the design choice of LongVA. For Gaample,Xions et al. (2023) first demonstrates the elfectiveness of long context continued pretraining with increased RoPE base frequency (thus decreasing the rotation angles). We sample the long text data following the guidance of (Fu et al., 2024b). We adopt the same vision encoder and training data as that of V.-L. (Liu al., 2024b). We try to keep our methods as simple as possible to clearly show the phenomenon of long context transfer without other confounders.
5

---

Published in Transactions on Machine Learning Research (07/2025)
Model
Dataset Tokens
Training Max Frames
SPPLUG-Owlviiee
104B
VideoChat
VA-Ve-Video
8B
16
VideoLLaMA
10B
32
LongVA
Table comparison video-language models in terms of estimated training tokens and maximum training frames, based on data reported in their original publications.
LLM
576 Tokens per Grid
144 Tokens per Grid
Original Images
Image as Multi-Grid
Split
Encode
Flatten
Average Pool
Original Video
Video Image with Multi-Grid 576 Tokens per Frame
144 Tokens per Frame
Figure 2: UniRes's unified encoding scheme of images and videos. During training, images are divided into multiple grids. During inference, videos treated extended images with each frame considered grid.
4
V-NIAH
To measure the context length of language models extremely long input, earlier works calculate perplexity scores long documents. Recently, many have started using the Needlo-n--a-Hyysaach (NIAH) test to benchmark LLMs' ability to retrieve long context information precisely. We note that there is so far no benchmark to measure the visual context length of LMMs. To evaluate Long VA's capacity to locate and retrieve long-range visual information, we extend the NIAH test from text to vidco and propose V-NIAH.
As shown in Table 11, we designed video quest question--nswwring problems as the needle and inserted each single frame into hours-long videos. We sampled the videos at FPS as the visual input. The image of the needle is sourced from existing VQA benchmarks Al-generated to avoid any contamination. The Al-generated images and questions purposely chosen to be "counterfactual" or ccomter-commonsen ensuring the model cannot answer based on language knowledge alone. Each question includes "locating prompt" so that a capable system or human can locate the needle frame from the video haystack and answer the question.
When testing LongVA with visual inputs of up to 3000 frames, one difficulty we encountered was that processing 200K-token input requires up to 100GB of GPU memory for the KV cache for a 7B LM like LLaMA. Even with advanced LM serving systems like vLLM (Kwon et al., 2023) with tensor parallelism to shard the KV cache across multiple GPUs, the sampling process remains extremely slow due to limited memory and batchsize. To address this, we used "perplexity-baeed" evaluation to mcasure the correctness of the model output. We first encode all frames and save their corresponding visual embeddings. During the evaluation, we only load the language model from LongVA VA and concatenate the visual embeddings, question tokens, and answer tokens for single forward pass with ring attention. This approach makes the workload compute-bound and eliminates the need to cache the KV state. The model's output is considered correct. only if the highest output logits index of all tokens in the answer span matches the correct answer.

---

Published in Transactions on Machine Learning Research (07 (07/202) /025)
Needle (vartous positions)
Question: Find the frame of couple in wedding. Inside
the frame there is balloon on the bridegroom's head.
What is the color of that balloon?
Answer the question using a single word or phrase.
Answer: Yellow
Figure 3: V-NIAH consists of haystack video, needle image, and question related to the needle. The needle is inserted at various positions in the haystack video.
(i) LongVA
(ii) LongVA (AnyRes)
1.00
0.0
Frame Depth
0.75
40.0
Score
0.50
60.0
80.0
0.25
100.0
0.2K
0.6k
1.0k
222k
2.6k
3.0K
0.2k
0.6k
1.kk
2.6K
3.0K
0.00
Num. Frame
Num. Frame
(iii) LLaVA-NeXTVVideo332
(iv) LLVVA-NeXT-viieo-22KK RoPE= 100M
1.00
0.0
Frame Depth
20.0
0.75
40.0
0.50
60.0
80.0
0.25
100.0
0.2K
0.6K
1.0k
1.4K
1.8k
2.2K
2.6K
3.0K
0.2k
0.6k
1.0k
1.4k
1. .k
2.2K
2.6K
3,0K
0.00
Num. Frame
Num. Frame
Figure 4: The V-NIAH results of LAngVA and its basclines. The x-axis represents the total number of frames
in the video haystack. The y-axis shows the position where the needle image located. For instance, a frame depth of 0% would place the needle image at the very beginning of the video. The black dotted line denotes the training length of the backbone language model, with each frame corresponding 144 tokens.
5 Experiments
We primarily assess the long visual capability of LongVA on two benchmarks: V-NIAH (Section 5.1 and Video- MME (Fu et al., 2024a) (Section 5.2). V-NIAH provides quick signals about the visual context length of LAngVA. However, only tests the model's ability to retrieve infor- mation and does not cover other abilities necessary for worldworld long video assistant. Therefore, also include Lo'sVA's performance on Vidco-MME, comprchensive evaluation suite for video LMMs that includes diverse data types and qualitative annotations. Video-MME is an ideal benchmark for assessing LMMs' ability to handle long vidcos real-world scenarios, given its average vidco duration of 1017 scconds and the inclusion of short, medium, and long subsets. We further include the benchmark results on MLVU(Zhou al., 2024a) in Table 10.
Model
Qwen2-224K UniRes
LougVA (Angfes)) LougVA
Table 7: LongVA and its baselines.

---

Published in Transactions on Machine Learning Research (07/2025)
Model
LLM Params
Frames
Short
Medium
Long
Overall
AntenVL-Chat-Vee (Chen et al,, 2023b)
60.2
46.4
Videen34BB LaVVA-NeXT NeX-VVee--d (Zhang et al,, 2024b)
34B
32
61.7
50.1
44.3
52.0
VILA-155 (Lin al., 2023b)
34B
8
68.1
58.1
50.8
59.0
Qwen-VL-Chat (Qwen, 2024)
7B
4
46.9
38.7
37.8
41.1
Vidoo-LLaVA (Lin al., 2023a)
7B
45.3
38.0
36.2
39.9
ST-LLM (Liu et al, 2024c]
7B
64
45.7
36.8
31.3
37.9
VideoChat2-
et 2024b)
7B
48.3
37.0
39.5
Chat-UniVi-V1.5 (Jin et al,, 2023)
7B
45.7
40.3
35.8
VideoLLaMA2 (Cheng et al., 2024)
8B
16
56.0
45.4
42.1
47.9
LaVA-NeeTT-ueen
7B
32
59.8
48.2
44.7
50.9
8
56.6
46,0
41.7
47.7
16
59.9
47.0
43.8
50.2
32
61.7
49.1
45.9
52.2
Long VA
7B
64
61.8
51.7
44.6
52.7
61.6
50.6
47.1
53.1
60.9
49.9
Table 6: Performance comparison of various LMMs on Video-MME (Fu et al., 2024a) without subtitles.
LongVA achieves state-of-the-art results among 7B models. Its performance also increases with denser sampling of video frames.
We mainly compare LongVA against other image and video LMMs. To validate the phenomenon of long contert transfer, we trained LLaVA-Next-Qwenz: a baseline model based on Qwen2-7BBInnsrruc using the LLa VA-NeXT (Liu et al., 2023b; Li al., 2024a) training recipe. Additionally, trained LongVA (AnyRes) to showcase the advantages of our UniRes encoding scheme. The difference between LongVA VA and our baselines can be found in Table 7.
5.1 V-NIAH Results
Long context transfers from language to vision Figure 4 shows the V-NIAH performance of Long VA and other LMMs. Specifically, Figure 4 (iii) demonstrates that the visual context length of LLaVA-NeXT- Video-32K (Zhang et al., 2024b) is constrained by the 32K context length of its language backbone, Mistral- TB-Instruct-v0.22 (Jiang et al., 2023), equivalent to approximately 200 frames. Beyond this limit, the V-NIAH accuracy drops significantly. As a stronger baseline, we include the results of LLaVA--eeTT-vide---2 enhanced with a training-fir free length extrapolation algorithm (bloc97, 2024) by increasing its RoPE base frequency. We empirically determine the optimal extrapolation frequency by choosing from [3M, 10M, 30M, 100M, 300M, 1BJ. As indicated in Figure 4 (iv), although this training-free extrapolation allows the model to process information across an extended context, the improvement is marginal. These findings led us to develop LongV4 VA, a model that unlocks the visual context by extending the language model purely on text. As shown Figure 4 (i), LongVA can almost perfectly retrieve information and answer the needle question for input frames fewer than 2000. Although we only trained LongVA's language backbone on context length of 224K (equivalent to 1555 frames), it generalizes well beyond that, maintaining satisfactory performance within 3000 frames. Those results clearly corroborate of hypothesis of long contert transfer.
Unified encoding enables better visual context extrapolation We also present the V-NIAH heatmap of LongVA trained with AnyRes encoding scheme, keeping all other factors unchanged in Figure 4 (ii). LongV VAngVA-AnyRess demonstrates strong retrieval capabilities. However, its performance still lags behind VA trained with UniRes. We believe that the unifted representation of images and videos in UniRes, where video encoded in the same way as a long image, enhances the long context transfer from language to vision. This approach also facilitates effective training with short vision data (images) and enables zero-shot understanding of long videos during inference.
5.2 Video Evaluation
V-NIAH is synthetic by design to allow controlled and scalable probing of the length of the visual context. To further illustrate that LongVA capable of handling long and realistic videos, we evaluated LongVA on

---

Published in Transactions on Machine Learning Research (07/2025)
Model
NeXTQA
ActivityNetQA Score
VideoChatGPP
Video-DD
frames
MC
OE
Consistency
Correctness
Detail
Context
Temporal
Score
LLaVA-NeXT-Vide
32
57.93
26.90
3.20
3.12
3.39
3.29
3.92
2.60
3.32
LongVA
8
50.78
27.71
2.73
3.73
3.00
3.14
3.72
2.39
3.19
LongVA VA
16
61.61
27.87
2.78
3.61
3.13
3.15
3.75
2.40
3.22
LongVA
32
67.08
27.87
2.80
3.65
3,08
3.10
3.74
2.28
3.19
LongVA
64
68.27
27.81
2.84
3.64
3.05
3.09
3.77
2.44
3.14
Long VA DPO
32
69.26
28.02
2.80
4.07
3.55
3.32
4.09
2.86
3.58
Table 8: Video evaluation results for LongVA on various short video benchmarks with comparison to 7B scale models.
real-world benchmarks such as Video-MME(Pu et al., 2024a) and MLVU(Zhou et al., 2024a) which cover diverse realistic video scenarios. LongVa VA achieves state-of -het-artt performance among 7B models on both, despite lack of video training.
On Video-MME (Table 6), LongVA achieves slale-of-lhe-arl performance among LMMs under 10B parameters, rivaling much larger ones such as -LaVA-NeXT-Video-Je V1.5 (Chen et al., 2023b). Notably, VA is trained without any video data, so its performance on video can be considered zero-shot. As the number of sampled frames increases, LongVA shows improved performance on the long subset, handling up to 384 frames?. Even though LongVA's 's score slightly drops when we upsample from 128 to 384 frames, it maintains a competitive performance. To our knowledge, Long VA is the only open-source model that can handle such large input frames on Vidco-MME. Those findings highlight the long conlext. transfer effect, where LongVA, originating from long context language model, can process significantly more frames than its baseline, despite being trained the same multimodal data. While 384 frames may not capture all long-term dependencies, but it's significant step up from prior work, which is less than 64 frames. While more frames are generally better, our choice is limited by hardware and budget constraints.
(Zhang et al., 2024b) and InternVL-Chaa-
We also tested LongVa VA on shorter benchmarks with average video durations under 120 seconds. As indicated in Table 8, although LongVA scores higher with more densely sampled frames on datasets such as NeXTQA (Xiao et al., 2021b) and ActivityNetQA (Yu et al., 2019b), the gains quickly plateau and not as significant. those observed in Vidco-MME, which can be attributed to the shorter duration of these datascts. On the VideoChaiGPT (Maaz et al., 2023) and Video Detailed Description (Video-DD) (Maaz et al., 2023) benchmarks, increasing frames does not lead to better performance, and Long VA generally achieves lower scores compared to KLAVA-NcXT-Video believe their metrics are closely related to the answering format. To address this, we perform a light Direct Preference Optimization (DPO) on the significantly improved performance for LongVA-DPO, confirming the findings in Zhang et al. (2024a).
Since both benchmarks use OpenAl's GPT API as a judge, wo
(Zhang et al., 2024a) dataset. We observe
5.3 Image Evaluation
Model
A12D
ChartQA
DocVQA
Info
RealworldQA
MMMU
LLaVA-1.66 Vicuna
66.6
54.8
74.4
37.1
57.8
35.1
MaVA-NeXTTLLLaMAS
71.6
69.5
78.2
37.6
60.0
41.7
LaVA-NeXT-Qwann
73.5
74.0
81.3
42.0
61.6
41.9
LongVA (AnyRes)
73.1
74.4
81.5
43.3
62.4
42.1
LongVA (UniRes)
70.7
70.4
80.8
49.4
60.0
42.6
Table Image evaluation results for LongVA on multiple benchmarks. Compared to other image multimodal models, our methods maintain high performance and achieve better scores on InfoVQA(Mathen) et al., 2020), We further evaluate our model on various image benchmarks to investigate the image performance of LongVA (Table 9). Compared to the LongVA (AnyRes) baseline, LongVA with Unitles achieves significantly increased performance on InfoVQA (Mathew et al., 2020), while the scores drop to some extent on AI2D (Kembhavi
We limited analysis 384 frames due computational memory constraints detailed in Section 4.

---

Published in Transactions on Machine Learning Research (07/202) /025)
AI2D Itest)
ChartOA (best)
InfoVOA (val)
1800
1400
14000
1600
1200
12000
1400
Wathh
1000
Imags With
1200
Ainae Vilu
10000
1000
600
6000
400
600
4000
200
400
600
1000 1200 Image Height
800
1400
200
300
400
500
600
700
800
2000
4000
6000
8000
Image Height
Image Height
Figure 5: The -D-histogamm of the image width and height of different image benchmarks. Info VeooQQQ(Matiaa et al., 2020) consists of many high-resolution images compared to other benchmarks.
et al., 2016) and ChartQA (Masry al., 2022). On DocVQA (Mathew et al., 2020), RealworkIQA (xAT, 2024), and MMMU (Yue et al., 2023), LongVA VA is able match its baseline's performance. To better understand this phenomenon, we recorded and analyzed the image size of those datasets, shown in Figure 5. We found that InfoVQA consists of higher-resolution images, while many images in AI2D and ChartQA are smaller than 768x 768. Compared to Anyres, UniRes operate 2x2 average pooling on each image, reducing to 1/4 visual tokens per image grid. However, the grid upper bound is set to 49 for UniRes while 4 for AnyRes, so UniRes may produce more image grids if the input images are of higher resolution. By using more grids per image, UniRes allocates more visual tokens datasets such as InfoVQA, achieving superior performance compared to the previous 7B LLaVA model. However, most of the images in ChartQA and A12D require fewer than 4 grids to represent. This may explain why the image performance decreases those benchnarks.
6 Qualitative Results
The qualitative results of LongVA-DPO are illustrated in Figure 6. The short video example comes from (Xie et al., 2023) and the two long videos are sourced from linkl and link2, respectively. the figure, LongVA accurately describes the short, humorous video involving individuals playfully interacting with condiments. It also identifies specific details in long videos, such as the color of train and the colors of umbrellas used in a scene, showcasing its proficiency in retrieving and interpreting visual information over extended vidco contexts. These capabilities highlight. LongVA's 's polential to overcome the challenges associated with processing and understanding extremely long videos.
7
MLVU Results
Holistic AR
Single Detail
Detail SSC
Averages M-Avg
Methods
Input
TR
VS
NQA
ER
PQA
AO
AC
G-Avg
GPT-4o (OpenAI, 2024)
0.5 fps
87.4
74.5
4.90
64.8
57.1
65.1
6.69
56.7
46.3
64.6
5.80
LLaVA-166 (Liu ot al., 2024b)
16 frm
60.6
41.0
2.11
43.1
38.4
41.0
4.35
25.5
25.7
39.3
3.23
Int (Chen al., 2028b)
16
67.0
3.16
52.7
32.8
23.5
50.4
4.02
MovieChat (Song et al, 2024b)
2018 frm
29.5
25.0
2.35
24.2
24.7
25.8
3.23
28.6
22.8
25.8
2.78
TimeChat (Ren et sl, 2024)
96 fum
23.1 1
27.0
2.51
21.5
28. 1
25.8
4.29
24.7
32.0
30,9
3.12
LIMMA-VID (Iá m al., 20234)
fps
34.5
3.22
30.1
32.7
32.5
27.5
33.2
MA-LMM (He ol., 2024)
1000 frm
51.9
35.5
2.12
43.1
38.9
35.8
4.80
25.1
24.3
36.4
3.46
SharoGPT4Vided (Chen et at, 2024b)
16 frm
75.8
51.5
2.52
47.6
43.2
48.4
5.02
34.0
23.3
46.4
3.77
VideoChat2 HD (Li et al. 2024b)
16 frm
77.3
60.5
3.38
16.2
48.9
50.1
4.39
23.2
29.1
47.9
3.99
VideoLJaMA2 (Cheng
2024)
16 frm
74.6
64.5
2.79
43.8
45.1
5.18
34.0
27.4
48.5
3.99
LongVA (ours)
256 frm
83.3
58.5
3.39
69.3
50.0
67.2
5.26
38.6
27.2
56.3
4.33
Table 10: Evaluation results by the authors of MTVU (Zhou et al., 2024a). The highest scores excluding
GPT-40 are bolden. TR: Topic Reasoning. AR: Anomaly Recognition. VS: Video Summary; NQA: Needle
QA; ER: Ego Reasoning; PQA: Plot QA; SSC: Sub-Scene Captioning; AO: Action Order; AC: Action Count; A-Avg: the average performance of multiple-choice tasks; G-Avg; the average performance of generation tasks.
10

---

Published in Transactions on Machine Learning Research (07 (07/202) 2025)
seconds
Please describe the funny video in detail.
The video depicts a lively and animated social scene where group of people are enjoying a meal together. The setting appears to be casual indoor space with comfortable seating, possibly living room or lounge area adorned with some artwork paintings easels in the background.
In the foreground, there's central figure who seems be playfully doling out condiments, likely ketchup another type of sauce, from bottles onto the faces hands of other individuals in the group. This activity is causing laughter and amusement the participants react with expressions surprise and delight, sometimes annoyance. The individuals involved have expressive faces and engaged in the moment...
34 min 51 seconds
Woloee Bit
wis
What is the color of the train in the video?
The train in the video is red.
27 min 13 seconds
In this video, two umbrellas were used. What the colors of the umbrellas?
The colors of the two umbrellas in the video are blue and green.
Figure 6: Qualitative Result of the LongVA-DPO. Our model demonstrates superior capability in under- standing both short and long videos.
11

---

Published in Transactions on Machine Learning Research (07/2025)
Table 10 includes the evaluation results by the authors of MLVU (Zhou et al., 2024a) their benchmark. LongVA achieves stale-of-lhe-ee results among open-source models and is only second to GPT-4o.
8 Visual Needle In A Haystack Test
Table lists the five VQA needles we used for V-NIAH. The 5 visual questions and answers are the only places where human annotation involved in the construction of V-NIAH, making it an ideal testbed benchmark LMMs long context capability.
V-NIAH Needles
Question:
Find the frame of the 'While You Were Out note.
What the name of the university on that note?
University Californis,
Angeles
Univensity California, San Diego
C. University of Califoraia, Berkeley
University Californis, Santa Barbara
Answer with option's from the given choices directly.
Answer:
Find the frame a couple in wedding, Inside the frame, there balloon bridegroom's head. What the oolor of that belloon? Answer the question using a single word or plrase.
Question:
Answer: Yellow
Question: Find the frume with the image of Seleniou tablets.
How many mg does each tablet contain? Answer question using single word or phrase.
SELENIUM
Answer: 200
Question:
Find the frome of a scieutist, The acientist is a...
Bird
B. Elephant
Panda
Dog
Answer with the option's letter from the given choices directly.
Question: Find the frame of seddy bear. Where is this teddy bear?
A. Times Square
B. Eiffel Tower
Taj Mahal
Sydney Opera House
Answer with the option's letter from the given choices directly.
Answer:
Table 11: The design of the 5 visual question anwwerrng problems used as the needle in V-NIAH.
12

---

Published in Transactions on Machine Learning Research (07 (025)
distractor 0.99
distractor 3, 0.93
distractor 0.92
20.0
0.8
Depth Percent
40.0
0.6
60.0
64k
128k
192K
250k
320k
384K
448K
512k
64k
128k
192K
256k
320k
384K
448K
512K
6dk
128k
192k
258k
320K
384K
448K
512K
0.0
Context Tokens
Figure The NIAH results of
after long context training.
9 Needle In A Haystack Test
When evaluating the Needle In A Haystack task (Gregory, 2024), we focus specifically casicr-to-evaluate variant. (AT, 2023) that involves identifying and relrieving random numbers associated with various randomly assigned cities from the context. The input to the language model has below template:
This is a very long story book:
<book> (haystack + needle haystack) </book>. An Based on the content the book, Question: What the special magic Singapore number? Answer: The special magic Singapore number is:
We insert a neodle with the key Singapore and a 7-digit randomly sampled magic number as the value into the haystack of Paul Graham's Essays. The needle has the following format:
AnThe special magic (City) number is: (XXXXXXX).X)
We iterate over various document depths (where the needle is placed) and context lengths measure the performance. For each depth and context length, conducted the test 5 times, each time with different 7-digit needle. We also come up with harder version where also insert several (3 5) other needles with the same format but different city name as distractors. The results are shown in Figure 7.
10 UniRes Encoding Scheme
AnyRes
UniRes-Image
UniRes- Video
2
2
3
Figure 8: The difference between AnyRes and UniRes, assuming the image divided into 2x2 grids and the video has 4 frames. The number indicates the flattening order. Additionally, UniRes applies 2x2 average pooling to both images and vidcos after the MLP projector between the vision encoder and the language model.
Figure 8 indicates the difference between AnyRes and UniRes. Given a high-resolution image and assuming CLIP-VIT-L--336 the vision encoder, both AnyRes and UniRes will divide it into multiple grids,
13

---

Published in Transactions on Machine Learning Research (07/2025)
each with the size 336x336. However, AnyRes will have smaller version of the full image as the base image and prepended before the high-resolution image grids. Additionally, UniRes flattens the encoded image feature in a raster-order within each grid, while Anyltes combines all the grids as a big feature map and flattens them across the border of the grid. UniRes also apply 2x2 average pooling on the image feature. As shown in the mosh part of Figure 8, the design of UniRes allows us to unifiedly encode videos as well. A video treated as an extended image where each frame considered as an image grid.
11 Conclusion
This work addresses the challenges of understanding long videos in Large Multimodal Models. By extending the language model on text and then aligning this extended model with visual imputs, we significantly improved the capability of LMMs to handle long videos thanks to the long contert transfer phenomenon. Our model, VA, shows improved performance with more input frames and achieves state-of-tho-art results on Video-MME. Additionally, we introduce synthetic benchmark, V-NIAH, to effectively measure the visual context length of video LMMs. We hope this work inspires further research in the field of long video LMMs and multimodal agents.
12 Limitations
Our current setup varies the position of the correct needle frame and measures the model's retrieval accuracy, but it does not yet evaluate whether the model can identify the correct timestamp of the needle frame. We will explore this feature in future versions of V-NIAH.
Evaluation of V-NIAH
Temporal Understanding We represent the video as 144xN visual tokens (where is the number of frames), without introducing additional tokens to explicitly delimit the frame boundaries. In our preliminary experiments, adding such delimiters did not lead to performance improvements. We believe that this may reflect an inherent limitation of the vision encoder, for example, is known to behave similarly to oag-of-words model in spatial undorstanding Yuksckgonul et al. (2023), and recent works suggest that this bag-of-words behavior may also extend to VLMs in both spatial and temporal reasoning Li el al. (2024c). We plan to explore more structured temporal representations in future work.
13 Broader Impact
We believe that understanding long videos can introduce privacy, bias, and ethical concerns. Specifically, we acknowledge that techniques such as visual needle search could raise surveillance or misinformation risks if misused. However, we argue that our benchmark, V-NIAH, provides a safer and more controlled environment for studying these challenges, as is fully synthetic and designed minimize exposure to sensitive or real-world data. Additionally, we commit. to releasing all datasets and code with appropriate licenses and clear usage guidelines to discourage unethical applications.
Hallucination remains a common challenge for models, particularly in long-vidco question answering. When a model fails to retrieve accurate details or is overly influenced by the text prompt, it may produce incorrect or misleading responses. Promising approaches, such as Sparse AutoEncoder (SAE) and advances in the internal mechanisms of LLMs, offer potential pathways to steer model behavior toward greater reliability and eventually enable more controllable outputs. Provenance concerns, such as the potential misuse of video understanding models to support generative techniques like deepfakes, are also relevant. technology advances. Although it may be difficult to fully control end-user behavior, believe watermarking techniques could play critical role in addressing these risks by allowing the identification of generated content and distinguishing it from authentic media.
Acknowledgments
This study is supported by the Ministry of Education, Singapore, under its MOE AcRF Tier 2 (MOE- T2FP20221-0012, MO0E-T2EP20023-0022 and under the RIE2020 Industry Aligument Fund Industry Collaboration Projects (IAF-ICP) Funding Initiative, as well as cash and in-kind contribution from the industry partner(s).'
14

---

Published in Transactions on Machine Learning Research (07 2025)
References
Arize Al. Lim testaredllinabbaystakk, 2023.
URL httpp://gtthaab.cm///tttaaaaaaaa
NeedleInAHaastack. 5, 13
Jean-Baye Bann--tttstteeeee Alayrac, Jeff Donahuc, Pauline Luc, Antoine Mioch, lain Barr, Yana Hasson, Karel Lenc, Arthur Mensch, Katherine Millican, Malcolm Reynolds, Roman Ring, Eliza Rutherford, Serkan Cabi, Tengda Han, Zhitao Gong, Sina Samangooei, Marianne Montciro, Jacob L Menick, Sebastian Borgeaud, Andy Brock, Aida Nematzadeh, Sahand Sharifzadeh, Mikot aj Binkkowski, Ricardo Barreira, Oriol Vinyals, Andrew Zisserman, and Karén Simonyan. Flamingo: visual language model for few-shot learning. In S. Koyejo, S. Mohamed, A. Agarwal, D. Belgrave, K. Cho, and A. Oh (eds.), Advances in Newral Information Processing Systems, volume 35, pp. 23716 23736. Curran Associates, Inc., 2022. URL https: /ttcttp::/ppooeeo http:///eceeedg...eeeee neurips. Saep://8000-Pap---oooeeeee pateeeereeeeeee.......... 2
Stanislaw Antol, Aishwarya Agrawal, Jiasen Lu, Margaret Mitchell, Dhruv Batra, Lawrence Zitmick, and Devi Parikh. Vqa: Visual question answering. In Proceedings of the IEEF international conference computer vision, pp. 2425-2423 2015.
Anas Awadalla, Irena Gao, Josh Gardner, Jack Hessel, Yusuf Hanafy, Wanrong Zhu, Kalyani Marathe, Yonatan Bitton, Samir Gadre, Shiori Sagawa, Jenia Jitsev, Simon Kornblith, Pang Wei Koh, Gabriel Ilharco, Mitchell Wortsman, and Ludwig Schmidt, Openftamingo: An open-source framework for training large autoregressive vision-language models. ar preprint arXiv:2308.011 01390, 2023. 2
Yushi Bai, Xin Lv, Jiajie Zhang, Yuze He, Ji Qi, Lei Hou, Jie Tang, Yuxiao Dong, and Juanzi Li. Longalign: A recipe for long context alignment of large language models, 2024. 3
bloc97. Ntk-aware scaled rope allows Ilama models to have extended (8k+) context size without any fine-tuning and minimal perplexity degradation. 2024. URL https: reddit t. com/z/LocalLLMM/ MaoMLLLLM////mmess/// comments/ hopeedrro_______ allows awwaasalllllaaaaaaaaaaaaaaaaaaae sastaammmmdeel___eeee tacoilse_tthaae____e 3,
Tom Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared D Kaplan, Prafulla Dhariwal, Arvind Neclakantan, Pranav Shyam, Girish Sastry, Amanda Askell, Sandhini Agarwal, Ariel Herbert- Voss, Gretchen Krueger, Tom Hemighan, Rewon Child, Aditya Ramesh, Daniel Ziegler, Jeffrey Wu, Clemens Winter, Chris Hesse, Mark Chen, Eric Sigler, Mateusz Litwin, Scott Gray, Benjamin Chess, Jack Clark, Christopher Berner, Sam McCandlish, Alec Radford, 1lya Sutskever, and Dario Amodei. Language models are few-shot learners. In H. Larochelle, M. Ranzato, R. Hadsell, M.F. Balcan, and H. Lin (eds.). Advances in Neural Information Processing Systems, volume 33, pp- 1901- Curran Associates, Inc., 2020. URL https: httpss//prroeeddinga file//ill//////// /ttp////////..000000000000111111 -2pe44 Paper pdf. 2
Mu Cai, Jianwei Yang, Jianfeng Gao, and Yong Jae Lee. Matryoshka multimodal models, 2024. 2, 3
Cerebras. cerebras 6limpajama-627b, 2023. URL http:///nnggiigffeeecooo/etttttr
SlimPajama 627B.
Liang Chen, Haozhe Zhao, Tianyu Liu, Shuai Bai, Junyang Lin, Chang Zhou, and Baobao Chang. An image is worth 1/2 tokens after layer 2: Plug-and-play inference acceleration for large vision-language models, 2024a. 2
Lin Chen, Xilin Wei, Jinsong Li, Xiaoyi Dong, Pan Zhang, Yuhang Zang, Zehui Chen, Haodong Duan, Bin Lin, Zhenyu Tang, Li Yuan, Yu Qiao, Dahua Lin, Feng Zhao, and Jiaqi Wang. Sharegpt4video: Improving video understanding and generation with better captions, 2024b. 3, 10
Shouyuan Chen, Sherman Wong, Liangjian Chen, and Yuandong Tian. Extending context window of large language models via positional interpolation. arXiv preprint arXiv:2306.1553 2023a. 3
Zhe Chen, Jiannan Wu, Wenhai Wang, Weijic Su, Guo Chen, Sen Xing, Zhong Muyan, Qinglong Zhang, Xizhou Zhu, Lewei Lu, et al. Internvl: Scaling up vision foundation models and aligning for generic visuua-linguistic tasks. ar Xie preprint ar 2rxi::232.. 14238, 2023b. 8, 9, 10
15

---

Published in Transactions on Machine Learning Research (07 (07/2025)
Zesen Cheng, Sicong Leng, Hang Zhang, Yifei Xin, Xin Li, Guanzheng Chen, Yongxin Zhu, Wenqi Zhang, Ziyang Luo, Deli Zhao, and Lidong Bing. Videollama 2: Advancing spatial-temporal modeling and audio understanding in video-llms, 2024. 2, 3, 8, 10
Rewon Child, Scott Gray, Alec Radford, and Tlya Sutskever. Generating long sequences with sparse transformers, 2019. 5
Cohere. Introducing command r+: A scalable Ilm built for business, 2024. URL TRtLttts///cohhere.com////
command-r r-s--s--uuu-ee calcooootaauuree....
Wenliang Dai, Junnan Li, Dongxu Li, Anthony Meng Huat Tiong, Junqi Zhao, Weishong Wang, Boyang Li, Pascale Fung, and Steven Hoi. Instructblip: Towards general-purpose vision-language models with instruction tuning, 2023. 2
Tri Dao. Flashattemtion-2: Faster atlention with better parallelism and work partitioning, 2023.
Yiran Ding, Li Lyna Zhang, Chengruidong Zhang, Yuanyuan Xu, Ning Shang, Jiahang Xu, Fan Yang, and Mao Yang. Longrope: Extending Ilm context window beyond 2 million tokens. orXiv preprint ar Xiv:22402. 13753, 2024. 3
Chaoyou Fu, Yuhan Dai, Yondong Luo, Lei Li, Shuhuai Ren, Renrui Zhang, Zihan Wang, Chenyu Zhou, Yunhang Shen, Mengdan Zhang, Peixian Chen, Yanwei Li, Shaohui Lin, Sirui Zhao, Ke Li, Tong Xu, Xiawu Zheng, Enhong Chen, Rongrong Ji, and Xing Sun. Video-mme: The first-ever comprehensive evaluation benchmark of multi-modal llms in video analysis, 2024a. 2, 3, 7, 8,9
Yao Fu, Rameswar Panda, Xinyao Niu, Xiang Yue, Hannaneh Hajishirzi, Yoon Kim, and Hao Peng. Data engineering for scaling language models to 128k context, 2024b. 3, 4, 5
Gemini. Gemini: A family of highly capable multimodal models, 2024.
Kamradt Gregory. Limtest needleinahaystack, 2024. URL httpa/////tthubcoo/gggge com/ htm///kkaaratttLLMeettt. http://ggggaboogggaaaaaaaMMMMtt NeeddlennAHaystackk 2, 3, 5, 13
Madeleine Grande-MeLaughlin Ranjay Krishna, and Maneesh Agrawala. Agqa: A benchmark for composi- tional spatio-temporal reasoning. In Proceedings of the IEEE/CV Conference on Computer Vision and Pattern Recognition, pp. 11287-11297, 2021.
Bo He, Hengduo Li, Young Kyun Jang, Menglin Jia, Xuefei Cao, Ashish Shah, Abhinav Shrivastava, and Ser-Nam Lim. Ma-lmm: Memory-augmented large multimodal model for long-term video understanding, 2024. URL htttp:://axxiv.rrr/ ab2p//44.///0/...//// 05726. 10
Edward J. Hu, Yelong Shen, Phillip Wallis, Zeyuan Allen-Zhu, Yuanzhi Li, Shean Wang, Lu Wang, and Weizhu Chen. Lora: Low-rank adaptation of large language models, 2021. 5
Sam Ade Jacobs, Masahiro Tanaka, Chengming Zhang, Minjia Zhang, Shuaiwen Leon Song, Samyam Rajbhandari, and Yuxiong He. Deepspeed ulysses: System optimizations for enabling training of extreme long sequence transformer models, 2023. 3
Yunscok Jang, Yale Song, Youngjae Yu, Youngjin Kim, and Gunhee Kim. Tgif-qa: Toward spatio-temporal reasoning in visual question answering. In Proceedings of the IEEE conference on computer vision and pattern recognition, pp. 2758-2766 2017. 3
Albert Q Jiang, Alexandre Sablayrolles, Arthur Mensch, Chris Bamford, Devendra Singh Chaplot, Diego de las Casas, Florian Bressand, Gianna Lengyel, Guillaume Lample, Lucile Saulnier, Lélio Renard Lavand, Marie-Anne Lachaux, Pierre Stock, Teven Le Scao, Thibaut Lavril, Thomas Wang, Timothée Lacroix, and William El Sayed. Mistral 7b, 2023. 8
Peng Jin, Ryuichi Takanobu, Caiwan Zhang, Xiaochun Cao, and Li Yuan. Chat-univi: Unified visual representation empowers large language models with image and video understanding. ar preprint arXiv:2311.08046 2023. 3,
16

---

Published in Transactions on Machine Learning Research (07/)025)
Peng Jin, Ryuichi Takanobu, Wancai Zhang, Xiaochun Cao, and Li Yuan. Chat-umivi: Unified visual representation empowers large language models with image and video understanding, 2024. 2, 3
Aniruddha Kembhavi, Mike Salvato, Eric Kolve, Minjoon Seo, Hannanch Hajishirzi, and Ali Farhadi. A diagram is worth dozen images, 2016. 9
Woosuk Kwon, Zhuohan Li, Siyuan Zhuang, Ying Sheng, Lianmin Zheng, Cody Hao Yu, Joseph E. Gonzalez, Hao Zhang, and lon Stoica. Efficient memory management for large language model serving with page- dattention. In Proceedings of the ACM SIGOPS 29th Symposium on Operating Systems Principles, 2023.
6
Hugo Laurengon, Lucile Saulnier, Léo Tronchon, Stas Bekman, Amanpreet Singh, Anton Lozhkov, Thomas Wang, Siddharth Karamcheti, Alexander M. Rush, Douwe Kiela, Matthieu Cord, and Victor Sanh. Obelics: An open web-scale filtered dataset of interleaved image-text documents, 2023. 2
Bo Li, Yuanhan Zhang, Liangyu Chen, Jinghao Wang, Jingkang Yang, and Ziwei Liu. Otter: A multi-modal model with in-context instruction luning, 2023a. 2
Bo Li, Hao Zhang, Kaichen Zhang, Dong Guo, Yuanhan Zhang, Renrui Zhang, Feng Li, Ziwei Liu, and Chunyuan Li. Llava-next: What else influences visual instruction tuning beyond data?, May 2024a. URL https://1lavavv1, thub. 202///002222400--25555 1lava- wett--llaaeee------aeeeeeaaeeeee meaexttabltiooss. 5, 8
Junnan Li, Dongxu Li, Silvio Savarese, and Steven Hoi. Blip-2: Bootstrapping language image pre-training with frozen image encoders and large language models, 2023b. 2,
KunChang Li, Yinan He, Yi Wang, Yizhuo Li, Wenhai Wang, Ping Luo, Yali Wang, Limin Wang, and Yu Qiao. Videochat: Chat-centric video understanding, 2024b. 3, 8, 10
Lei Li, Yuanxin Liu, Linli Yao, Peiyuan Zhang, Chenxin An, Lean Wang, Xu Sun, Lingpeng Kong, and Qi Liu. Temporal reasoning transfer from text to video, 2024c. URL attps://arxivv www. hwwgg/b//241...e 0tt.//211.......... 06166. 14
Shenggui Li, Fuzhao Xue, Chaitanya Baranwal, Yongbin Li, and Yang You. Sequence parallelism: Long sequence training from system perspective. In Anna Rogers, Jordan Bryderraber, and Naoaki Okazaki (eds.), Proceedings of the 61sl Annual Meeting of the Association for Compulational Linguistics (Volume 1: Long Papers), pp. 2391-2404, Toronto, Canada, July 2023c. Association for Computational Linguisties. doi: /t5p:///112233ccl-//1 134. URL https: /taps://accli htt///acaaaatlooggo// 1ow.oloogoo.rr/202233
org/2023........ooo acl acl-long 134. 3, 4
Yanwei Li, Chengyao Wang, and Jiaya Jia. Llama-vid: An image is worth 2 tokens in large language models, 2023d. 2, 3, 10
Bin Lin, Yang Ye, Bin Zhu, Jiaxi Cui, Munan Ning, Peng Jin, and Li Yuan. Video-llava: Learning united visual representation by alignment before projection, 2023a. 2, 8
Ji Lin, Hongxu Yin, Wei Ping, Yao Lu, Pavlo Molchanov, Andrew Tao, Huizi Mao, Jan Kautz, Mohammad Shoeybi, and Song Han. Vila: On pre-training for visual language models, 2023b. 8
Hao Liu, Matei Zaharia, and Pieter Abbeel. Ring attention with blockwise transformers for near-infinite context, 2023a. 3, 4
Hao Liu, Wilson Yan, Matei Zaharia, and Pieter Abbeel. World model on million-length video and language with ringattention. areie preprint, 2024a. 3
Haotian Liu, Chunyuan Li, Yuheng Li, and Yong Jac Lec. Improved baselines with visual instruction tuning, 2023b. 3, 8
Haotian Liu, Chunyuan Li, Qingyang Wu, and Yong Jae Lee. Visual instruction tuning, 2023c. 2, 3
Haotian Liu, Chunyuan Li, Yuheng Li, Bo Li, Yuanhan Zhang, Sheng Shen, and Yong Jac Loc. Llava-next: Improved reasoning, ocr, and world knowledge, January 2024b. URL cctps://llaa--vl.giite 1 blog/2024-01-30-1lavv --a----a------aaaaeeee/e//e 2, 3, 5, 10
17

---

Published in Transactions on Machine Learning Research (07/2025)
Ruyang Liu, Chen Li, Haoran Tang, Yixiao Ge, Ying Shan, and Ge Li. St-llm: Large language models are effective temporal learners, 2024c. 3, 8
Muhammad Maaz, Hanoona Rasheed, Salman Khan, and Fahad Shahbaz Khan. Video-chatgpt: Towards detailed video understanding via large vision and language models, 2023. 3, 9
Karttikeya Mangalam, Raiymbek Akshulakov, and Jitendra Malik. Egoschema: diagnostic benchmark for very long-form video language understanding, 2023. 3
Ahmed Masry, Do Xuan Long, Jia Qing Tan, Shafiq Joty, and Enamul Hoque, Chartqa: A benchmark for question answering about charts with visual and logical reasoning, 2022. 10
Minesh Mathew, Dimosthenis Karatzas, R Manmatha, and CV Jawahar. Docvqa: A dataset for vqa document images. corr 0bss/200.00033) (2020). arXiv preprint arXiv:2007.00808 2020. 9, 10
Mistral. Mixtral 8x22b: Cheaper, better, faster, stronger, 2024. mixtral-8x22b/.2
URL https://mistral. /nw/www/////
OpenAl. Hello gpt-4o, 2024. URL
10
Aitor Ormazabal, Che Zheng, Cyprien de Masson d' d'Autmme Dani Yogatama, Deyu Fu, Donovan Ong, Eric Chen, Eugenie Lamprecht, Hai Pham, Isaac Ong, et al. Reka core, flash, and edge: A series of powerful multimodal language models. arXiv preprint orXiv:2404.1238) 2024. 2
Bowen Peng, Jeffrey Quesnelle, Honglu Fan, and Enrico Shippole. Yarn: Elficient context window extension of large language models. In The Twelfth International Conference on Learning Representations, 2023. 3
Qwen. Introdueing qwen-vl, 2024. URL
qww.o///////v///.// 2, 3,
Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, Gretchen Krueger, and Tlya Sutskever. Learning transferable visual models from natural language supervision, 2021.
Samyam Rajbhandari, Jeff Rasley, Olatunji Ruwase, and Yuxiong He. Zero: Memory optimizations toward training trillion parameter models, 2020.
Shuhuai Ren, Linli Yao, Shicheng Li, Xu Sun, and Lu Hou. Timechat: A time-sensitive multimodal large language model for long video understanding, 2024. 3, 10
Baptiste Rozière, Jonas Gchring, Fabian Glocckle, Sten Sootla, Itai Gat, Xiaoqing Ellen Tan, Yossi Adi, Jingyu Liu, Romain Sauvestre, Tal Remez, Jérémn Rapin, Artyom Kozhevnikov, Ivan Evtimov, Joanna Bitton, Manish Bhatt, Cristian Canton Ferrer, Aaron Grattafiori, Wenhan Xiong, Alexandre Défossez, Jade Copet, Faisal Azhar, Hugo Touvron, Louis Martin, Nicolas Usunier, Thomas Scialom, and Gabriel Synnaeve. Code Ilama: Open foundation models for code, 2024. 3
Yuzhang Shang, Mu Cai, Bingxin Xu, Yong Jae Lee, and Yan Yan. Llava prameeer Adaptive token reduction for efficient large multimodal models, 2024. 2
Dingjie Song, Shunian Chen, Guiming Hardy Chen, Fei Yu, Xiang Wan, and Benyou Wang. Milebench: Benchmarking mllms in long context, 2024a. URL attps://arxitss orgooogg/aao aaaaaab///44////////// 18532. 4
Enxin Song, Wenhao Chai, Guanhong Wang, Yucheng Zhang, Haoyang Zhou, Feiyang Wu, Haozhe Chi, Xun Guo, Tian Ye, Yanting Zhang, Yan Lu, venq-Neng Hwang, and Gaoang Wang. Moviechat: From dense token to sparse memory for long video understanding, 2024b. 2, 10
Jianlin Su, Yu Lu, Shengfeng Pan, Ahmed Murtadha, Bo Wen, and Yunfeng Liu. Roformer: Enhanced transformer with rotary position embedding, 2023. 3,
PaLM Team. Palm 2 technical report, 2023. 2
Qwen2 Team. Qwen2 technical report, 2024. 4
18

---

Published in Transactions on Machine Learning Research (07/2025)
Hugo Touvron, Thibaut Lavril, Gautier Izacard, Xavier Martinet, Marie-Anne Lachaux, Timothée Lacroix, Baptiste Rozière, Naman Goyal, Eric Hambro, Faisal Azhar, Aurelien Rodriguez, Armand Joulin, Edouard Grave, and Guillaume Lample. Llama: Open and efficient foundation language models, 2023. 2
Hengyi Wang, Haizhou Shi, Shiwei Tan, Weiyi Qin, Wenyuan Wang, Tunyu Zhang, Akshay Nambi, Tanuja Ganu, and Hao Wang. Multimodal needle a haystack: Benchmarking long-contexl capability of multimodal large language models, 2024a. URL https:// www.v.org//att attaaabb..//..///// 11230.
Weihan Wang, Zehai He, Wenyi Hong, Yean Cheng, Xiaohan Zhang, Ji Qi, Shiyu Huang, Bin Xu, Yuxiao Dong, Ming Ding, and Jie Tang. Lvbench: An extreme long video understanding benchmark, 2024b. 2
Weiyun Wang, Shuibo Zhang, Yiming Ren, Yuchen Duan, Tiantong Li, Shuo Liu, Mengkang Hu, Zhe Chen, Kaipeng Zhang, Lewei Lu, Xizhou Zhu, Ping Luo, Yu Qiao, Jifeng Dai, Wenqi Shao, and Wenhai Wang. Needle in a multimodal haystack, 2024c.
Bo Wu, Shoubin Yu, Zhenfang Chen, Joshua B Tenenbaum, and Chuang Gan. Star: benchmark for situated reasoning in real-world vidcos. In Thirty-fifth Conference Neural Information Processing Systems Dalasels and Benchmarks Track (Round 2), 2021. 3
xAI. Grok-1.5 vision preview, apr 2024. URL https::///// xixxx.iiiii wwwpaa/ogggooo.....111 10
Junbin Xiao, Xindi Shang, Angela Yao, and Tat-Seng Chua. Next-qa: Next phase of cresston-annswerin to explaining temporal actions. Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pp. 9777 9786, 2021a. 3
Junbin Xiao, Xindi Shang, Angela Yao, and Tat-Seng Chua. Next-qanext phase of question-answering to explaining temporal actions, 2021b. 3,
Binzhu Xie, Sicheng Zhang, Zitang Zhou, Bo Li, Yuanhan Zhang, Jack Hessel, Jingkang Yang, and Ziwei Liu.
Funqa: Towards surprising video comprehension. GitHuu repository, 2023. 10
Wenhan Xiong, Jingyu Liu, Igor Molybog, Hejia Zhang, Prajiwal Bhargava, Rui Hou, Louis Martin, Rashi Rungla, Karthik Abinav Sankararaman, Barlas Oguz, Madian Khabsa, Han Fang, Yashar Mehdad, Sharan Narang, Kshitiz Malik, Angela Fan, Shruti Bhosale, Sergey Edunov, Mike Lewis, Sinong Wang, and Hao Ma. Effective long-context scaling of foundation models, 2023. 3, 4,
Dejing Xu, Zhou Zhao, Jun Xiao, Fei Wu, Hanwang Zhang, Xiangnan He, and Yueting Zhuang. Video question answering via gradually refined attention over appearance and motion. In Proceedings of the 25th ACM international conference on Mullimedia, pp. 1645 1653, 2017. 3
Li Xu, He Huang, and Jun Liu. Sutd-trafficqa: A question answering benchmark and an efficient network for video reasoning over traffic events. In Proceedings of the IEEE/CVF Conference Computer Vision and Poltern Recognition, pp. 9878 9888, 2021. 3
Qinghao Ye, Haiyang Xu, Guohai Xu, Jiabo Ye, Ming Yan, Yiyang Zhou, Junyang Wang, Anwen Hu, Pengcheng Shi, Yaya Shi, Chenliang Li, Yuanhong Xu, Hehong Chen, Junfeng Tian, Qi Qian, Ji Zhang, Fei Huang, and Jingren Zhou, mplug-owl: Modularization empowers large language models with multimodality, 2024.
Kexin Yi, Chuang Gan, Yunzhu Li, Pushmect Kohli, Jinjun Wu, Antonio Torralba, and Joshua B Tenenbaum. Clevrer: Collision events for video representation and reasoning. arXiv preprint arXiv:191000144. 2019. 3
Zhou Yu, Dejing Xu, Jun Yu, Ting Yu, Zhou Zhao, Yueting Zhuang, and Dacheng Tao. Activitynet-qa: A dataset for understanding complex web videos via question answering. In Proceedings of the AAAT Conference on Artificial Intelligence, volume 33, pp. 9127 9134, 2019a. 3
Zhou Yu, Dejing Xu, Jun Yu, Ting Yu, Zhou Zhao, Yueting Zhuang, and Dacheng Tao. Activitynet-q:: A dataset for understanding complex web videos via question answering, 2019b.
19

---

Published in Transactions on Machine Learning Research (07 (07/2025)
Xiang Yue, Yuansheng Ni, Kai Zhang, Tianyu Zheng, Ruoqi Liu, Ge Zhang, Samuel Stevens, Dongfu Jiang, Weiming Ren, Yuxuan Sun, Cong Wei, Botao Yu, Ruibin Yuan, Renliang Sun, Ming Yin, Boyuan Zheng, Zhenzhu Yang, Yibo Liu, Wenhao Huang, Huan Sun, Yu Su, and Wenhu Chen. Mmmu: A massive
multi-discipline multimodal understanding and reasoning benchmark for expert agi, 2023. 10
Mert Yuksekgonul, Federico Bianchi, Pratyusha Kalluri, Dan Jurafsky, and James Zou. When and why vision-languagg models behave like bags-of-words, and what to do about it? 2023. URL https://arxiv. org/abs/220) 221/aba//21.... http://2............ 01936. 14
Hang Zhang, Xin Li, and Lidong Bing. Video-llama: An instruction-tuned audio-visual language model for video understanding, 2023. 2,
Ruohong Zhang, Liangke Gui, Zhiqing Sun, Yihao Feng, Keyang Xu, Yuanhan Zhang, Di Fu, Chunyuan Li, Alexander Hauptmann, Yonatan Bisk, and Yiming Yang. Dircct preference optimization of video large multimodal models from language model reward, 2024a. 3,
Yuanhan Zhang, Bo Li, haotian Liu, Yong jae Lee, Liangke Gui, Di Fu, Jiashi Feng, Ziwei Liu, and Chunyuan Li. Llava-next: strong zero-shot video understanding model, April 2024b. URL https: //tlava-vl.gi github. wtt.gggtttt....ooggg///oo http://////222------1--aaaaa----
3, 5, 9
Junjie Zhou, Yan Shu, Bo Zhao, Boya Wu, Shitao Xiao, Xi Yang, Yongping Xiong, Bo Zhang, Tiejun Huang, and Zheng Liu. Mlvu: A comprehensive benchmark for multi-task long video understanding, 2024a. 2, 4, 7, 9, 10, 12
Xingyi Zhou, Anurag Arnab, Shyamal Buch, Shen Yan, Austin Myers, Xuehan Xiong, Arsha Nagrani, and Cordelia Schmid. Streaming dense video captioning, 2024b. 2
Deyao Zhu, Jun Chen, Xiaoqian Shen, Xiang Li, and Mohamed Elhoseiny. Minigpt-4: Enhancing vision- language understanding with advanced large language models, 2023.
Zilin Zhu. Ring flash attention, 2024. https://github.com/ wwwpbbuzliizzziiiinno.mm /ring- wiwg-gflsh--ateeee flash-attention. 5
20


Reviews:
[{"review_id": "Tr3On2Roxn", "reviewer": "Reviewer_eYB8", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Strength:\n1. The paper focuses on an important problem.\n2. The paper is well written and easy to follow\n3. The paper proposed a novel view to design the method.\n4. The proposed method makes sense to me.\n\nWeakness:\n1. I wonder to know whether the proposed method is good at multi-image understanding or reasoning?\n2. The improvement compared with existing methods is kind of limited.", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "Yes", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "This paper focuses on long video understanding. It proposes a new method, where it is a long input LLM for video input. Then the authors design a new training strategy. Extensiven experiments on multiple datasets are conducted and the results show the effectiveness of the proposed method.", "claims_and_evidence_yn": "Yes", "claims_explanation": "", "audience_yn": "Yes", "audience_explanation": "", "requested_changes": "see weakness", "broader_impact_concerns": "None", "additional_comments": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "nhJIMd8ZCb", "reviewer": "Reviewer_QZgK", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Strengths:\n- The novelty of the long context transfer concept is significant, presenting a fresh approach compared to traditional token-reduction methods.\n- The introduction of V-NIAH as a benchmark fills an important gap, providing a practical evaluation of visual context retrieval capabilities.\n- LongVA's performance sets a strong new baseline for long-context multimodal tasks, particularly notable for its zero-shot capability on long videos.\n- The method demonstrates efficiency, requiring minimal additional training, making it feasible for long context optimization.\n\nWeaknesses:\n- Generalization about Long-context evaluation beyond the synthetic benchmarks is uncertain, as real-world video complexities might not be fully captured by synthetic tests like V-NIAH.\n- The method based on the Qwen2 with 224K context length, does the long context LM is important in this method? How's the performance and computation cost compare with Qwen2-VL with Long-context VL training? This make the comparison being more convincing.\n- V-NIAH focuses on time-aware retrieval tasks. Can this evaluation provide timestamp correspondence information? For example, time-based evidence to clarify whether the model has found the answer within a specific timestamp interval?\n- The reliance on high GPU memory for handling long sequences (3.5 days on 8×A100 GPUs) still heavy and could limit broader academic adoption.", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "Yes", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "This paper addresses the problem of long context transfer in large multimodal models (LMMs), showing how extending the context length of a language model can directly transfer to improved handling of long video sequences without additional long-video training data. Specifically, the paper contributes:\n- Long Context Transfer: Demonstrating that language models with extended context can naturally enhance visual modality context handling.\n- Visual Needle-In-A-Haystack (V-NIAH): A novel synthetic benchmark for testing multimodal models' ability to retrieve visual information from extremely long contexts.\n- Long Video Assistant (LongVA): A multimodal model capable of processing over 200K visual tokens, achieving state-of-the-art results among 7B-scale models on benchmarks like Video-MME and MLVU.", "claims_and_evidence_yn": "Yes", "claims_explanation": "", "audience_yn": "No", "audience_explanation": "", "requested_changes": "Please see the Weakness section, beside:\n\n- The key difference between AnyRes by LLaVA-OV, UniRes still unclear to me, please add more clarification such as a Table to clearly indicate their difference. Also, please add 3D-ROPE of Qwen2-VL for a comprehensive discussion.\n\n- Long video setting In Table 4, 384 frames, is this setting dense enough to capture details in long videos?\n\n- I appreciate the comparison in Tab.7, looks like the UniRes drop the performance in series of image tasks beside InfoVQA and MMMU. Could author provide more clarification why performance drop on other benchmarks?", "broader_impact_concerns": "The paper does not explicitly discuss broader ethical implications. Discussion of ethical implications could be V-NIAH, whether the authors have addressed the data issues such as licensing.", "additional_comments": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "BcdJZ4PBAz", "reviewer": "Reviewer_7nrL", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "### Strengths\n1. Long video understanding is a challenging and timely problem to tackle, particularly given the recent success of LMMs.\n\n2. The paper's contributions of long context transfer of language models and improvement of text-video alignment to perform long video understanding are technically sound and meaningful. The paper establishes new baselines for future video understanding models. It will likely interest the broader scientific communities working on language models, video understanding, and their intersections.\n\n3. The V-NIAH benchmark is a very relevant benchmark for long video understanding. It can potentially be developed into a community standard for evaluating Q&A tasks on long videos.\n\n### Weaknesses\n1. In Sec. 3.2, the authors describe their tiling approach, which results in a $336 \\times (336 \\times N)$ tensor, which they then transform into $144N$ visual tokens for text alignment. Since the frame length $N$ remains the same before and after tokenization, does this imply that the tokens do not take into account any temporal dependencies across the $N$ frames? If so, how can such tokens be used for questions requiring temporal understanding (e.g., \"what action is person A doing?\" -> \"walking\")?\n\n2. How are the grid upper bounds 49 and 4 chosen for UniRes and AnyRes, respectively? Are these computational limits for the systems on which the authors experimented? Can the authors comment on the impact these upper bounds would have on model performance if it were possible to increase them further?\n\n3. In Sec. 8, it is unclear from the text whether the five VQA needles shown in Table 9 comprise the full set of needles in the proposed V-NIAH benchmark or whether they are five examples. Also, what are the distributions of the videos in which they are placed for benchmarking, e.g., min, max and mean # frames of the videos, somewhat dissimilar and very dissimilar video contents compared to the needle, etc.? Providing some difficulty gradings of the benchmark along these lines can help inform its utility for different use cases.", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "Yes", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "The paper presents a method for long video understanding by expanding the context length of the language backbones of LMMs. The proposed approach, which the authors name long context transfer, consists of training language models on longer text data compared to existing training setups, followed by aligning these longer-context trained models with the visual data and fine-tuning on visual instructions. To improve the alignments between the text and the visual data, the authors also propose an approach to represent videos as \"extended\" images by tiling their constituent image frames. The authors perform quantiative experiments and ablation studies to evaluate both their long-context training of the language model and improving the text-video alignment on existing open-source video Q&A datasets. Further, to evaluate long video understanding more systematically, the authors present a visual needle-in-a-haystack (V-NIAH) benchmark where, given a question, the relevant content in the video is located within a long window of otherwise irrelevant content.", "claims_and_evidence_yn": "Yes", "claims_explanation": "", "audience_yn": "Yes", "audience_explanation": "", "requested_changes": "1. Please address the concerns under the \"Weaknesses\" section.\n\n2. The authors refer to \"Appendix 9\" at the end of Sec. 3.1, but I found no appendices attached to or linked with the paper. Did the authors provide the appendices?\n\n3. Typos, for example, mixed verb tenses, such as \"We further proposed UniRes\" -> \"We further propose UniRes\" (Sec. 1), \"We proposed the V-NIAH benchmark\" -> \"We propose the V-NIAH benchmark\" (Contribution # 2), \"We then detailed how we augment\" -> \"We then detail how we augment\" (Sec. 3).", "broader_impact_concerns": "The paper can benefit from some discussion on the ethical implications of achieving long video understanding. For example, what level of detail in videos would become accessible for the LMMs with the proposed enhancement of LMM capabilities (privacy concerns), how sensitive is the text-video alignment process to the contents of the text and the video and whether the LMMs can learn spurious alignments (bias concerns), and whether the longer-context learning can be leveraged for deepfake generation (provenance concerns). Much of these ethical implications are shared with existing works on video understanding, which the authors can consider using to ground their discussion.", "additional_comments": "", "extra_scores": {}, "extra_text": {}}]

Author rebuttal:

Decision: Accept (poster)