Long Context Transfer from Language to Vision

## Abstract

Abstract Video sequences offer valuable temporal information, but existing large multimodal models (LMMs) fall short in understanding extremely long videos. Many works address this by reducing the number of visual tokens using visual resamplers. Alternatively, in this paper, we approach this problem from the perspective of the language model. By simply extrapolating the context length of the language backbone, we enable LMMs to comprehend orders of magnitude more visual tokens without any video training. We call this phenomenon long context transfer and carefully ablate its properties. To effectively measure LMMs’ ability to generalize to long contexts in the vision modality, we develop V-NIAH (Visual Needle-In-A-Haystack), a purely synthetic long vision benchmark inspired by the language model’
s NIAH test. Our proposed Long Video Assistant (LongVA) can process 2000 frames or over 200K visual tokens without additional complexities. With its extended context length, LongVA achieves state-of-the-art performance on Video-MME and MLVU among 7B-scale models by densely sampling more input frames. Our work is open-sourced at https://github.com/EvolvingLMMs-Lab/LongVA .

## 1 Introduction

Driven by the progress of Large Language Models (LLMs) [, , , , , , ], multiple studies are conducted to extend their capability to understand images and videos [, , , ]. With modality alignment and visual instruction tuning, these Large Multimodal Models (LMMs) have shown impressive abilities such as captioning and visual question-answering. While current LMMs have demonstrated promising performance on tasks involving single images and short videos [, , , ], effectively processing and understanding extremely long videos remains a significant challenge [].

One primary reason for this challenge is the excessive number of visual tokens generated by the vision encoder. For instance, LLaVA-1.6 [] can produce 576 to 2880 visual tokens for a single image. The number of visual tokens increases significantly with the addition of more frames. To address this problem, numerous methods have been proposed to reduce the number of visual tokens. One popular direction is to modify the visual resampler that connects the vision encoder and LLM, aiming to extract fewer tokens [, , , ]. Alternative approaches [, , , ] employ heuristic techniques to prune or merge the visual features. However, despite these efforts, Table 1 demonstrates that the majority of current LMMs are still limited in their ability to process a large number of frames effectively.

Another issue hindering the development of high-performance long video LMMs is the lack of high-quality long video datasets. In Table 3, we list the average video length of existing video instruction tuning data. Most datasets consist of video clips within 1 minute. Even if some datasets do contain longer videos, the corresponding text pairs are generated by annotating only several frames within that video, lacking long and dense supervision signals.

Given the circumstance, in this paper, instead of reducing the visual tokens, we identify the more critical issue limiting the visual context length in existing LMMs: the context length of the language model backbone. Given a language model, we first extend its context length by training on longer text data. We then use this context-extended LM as the backbone to perform modality alignment and visual instruction tuning without any long video text pairs. By training this way, the context length of the language model is directly transferred to that of the LMMs. We further proposed UniRes, a unified encoding scheme that represents videos as extended images, enhancing the capability fusion between images and videos.
To facilitate benchmarking and accurately assess the context length in the visual domain, we created V-NIAH, a synthetic visual benchmark based on the Needle-in-a-haystack test [] used in language models. Our model, Long Video Assistant (LongVA), is capable of accurately retrieving visual information from 2000 frames or more than 200K visual tokens. Experiments show that additional frames during inference lead to improved performance on long video question-answering benchmarks, and LongVA achieves state-of-the-art performance among 7B models on the Video-MME [] and MLVU [] dataset.
In summary, our paper makes the following contributions:

(1) Long Context Transfer: We discovered the long context transfer phenomenon where the context of the language model can be directly transferred to the modality-aligned multi-modal models.

(2) Visual Needle-In-A-Haystack (V-NIAH): We proposed the V-NIAH benchmark to test LMMs ability in locating and retrieving visual information over extremely long contexts.

(3) Long Video Assistant (LongVA): With long context transfer and UniRes, we developed LongVA that can perceive more than 200K visual tokens, achieving SoTA performance on the Video-MME and MLVU dataset.

**Table 1: To enable longer video inputs, previous works train fewer visual tokens to increase the maximum frames during training. Our LongVA, on the other hand, enables long video capability by extending the backbone language model. ^*We report it based on the best available information from their paper or released codebase.**
| Model | Tokens/Frames^* | Training Max Frames^* | LM Backbone | LM Context Length |
| --- | --- | --- | --- | --- |
| MPLUG-Owl-video [] | 256 | 4 | LLaMA | 4K |
| MovieChat [] | 32 | 8 | Vicuna-v0 | 2K |
| Video-LLaVA [] | 49 | 8 | Vicuna-1.5 | 4K |
| VideoChat [] | 32$/$196 | 8 | Vicuna-v0 | 2K |
| LLaVA-NeXT-Video [] | 144 | 16 | Vicuna-1.5 | 4K |
| ST-LLM [] | 256 | 16 | Vicuna-1.1 | 2K |
| Video-LLaMA [] | 32 | 32 | LLaMA-2 | 4K |
| Chat-UniVi [] | 112 | 64 | Vicuna-1.5 | 4K |
| TimeChat [] | 4 | 96 | LLaMA-2 | 4K |
| Video-ChatGPT [] | 256 | 100 | Vicuna-1.1 | 2K |
| LLaMA-VID [] | 2 | 300 | Vicuna-1.5 | 4K |
| LongVA (Ours) | 144 | - | Qwen2-Extended | 224K+ |

## 2 Related Work

#### Vision Language Connector in Large Multimodal Models

Existing studies explore different architectures to extract and inject visual features into LLMs. One line of work [, , , ], pioneered by Flamingo [], adopts a resampler to compress the visual feature and inserts cross-gated attention layers into the LLM. Some other works still use a reampler [, , ] while directly feeding the image feature into the input layer of the language model. The LLaVA series [, , ] use a simple and scalable design to directly project the image features into language model without any pooling or resampling. When the field moves from image-only models to include multi-image and video inputs, more modifications to the visual language connector were proposed. [] and [] use a simple average pooling. [] dynamically drop the visual tokens. [] adopt a spatial-temporal convolution to better capture the dynamics of video data and reduce feature size. Our proposed context transfer from text to image is orthogonal to those works and can further enable LMMs to understand more frames.

**Table 2: Existing Video SFT Datasets**
| Dataset Name | Video Length (sec.) | Text Length |
| --- | --- | --- |
| VideoChatGPT-100K [] | 123.4 | 068.0 |
| LLaVA-Hound-255K  [] | 052.4 | 037.6 |
| ShareGPT4Video[] | 026.6 | 273.3 |
| TimeIT [] | 190.8 | 052.5 |
| VideoChat [] | 009.5 | 059.0 |

#### Context Extrapolation in Transformer

Transformer does not directly work on sequences longer than its training length. To alleviate that, various RoPE-based  [] extension techniques [, , , , ] have been proposed to allow for training-free context extrapolation. Efforts have also been made on data curation [, , ] and system optimization [, , ] during long context training. There has been limited exploration of the context extrapolation in the domain of LMMs. [] are closest to our work and train LMM with long context language models, but they do not benchmark the effective visual context length of their model.

#### Video Language Benchmarks

Recent years have witnessed significant progress in Video Question-Answering[].
To accurately measure the progress of the video LMMs’ performance, researchers have developed various benchmarks encompassing a broad spectrum of tasks. These range from fundamental visual perception tasks such as activity recognition[], concept detection [], and counting [], to more complex visual reasoning tasks including compositional [], causal [, , ], and situated reasoning [].
However, most of those benchmarks focus on short videos, lacking data and metrics to test LMMs’ capability over a long context. Inspired by the NIAH test [] in the language model community, we proposed V-NIAH to benchmark LMMs’ ability over long visual inputs with the minimum overhead of data collection and human annotation. Several concurrent works also developed multimodal versions of the Needle-in-a-haystack test [, , , ]. However, they only measure on several hundreds of frames and lack a strong baseline to properly analyze the properties of visual context length.

## 3 Long Video Assistant

As in Figure 1, this paper centers around the hypothesis that if the modality of vision and language can be truly aligned, the capability to handle long contexts could also transfer from text to vision, and this could happen even without explicit long video training. Our methodology is thus very straightforward. Given a language model, we first perform long context training purely on language to extend its text context (Section 3.1). We then detailed how we augment this language model with long visual capabilities by training solely on short image data in Section 3.2.

Figure: Figure 1: Left: to develop long vision models, previous studies proposed better visual resamplers to reduce the number of visual tokens. Right: LongVA approaches this problem from the angle of the language model. We leverage image data (short visual input) to align long-context LLM with vision. During the test time, LongVA can zero-shot process extremely long videos, thanks to the property of long context transfer.
Refer to caption: extracted/5701582/image/longva_plot.png

### 3.1 Training Long Language Model

We use Qwen2-7B-Instruct [] as the backbone language model and perform continued pretraining with a context length of 224K(^1^11224K is the maximum we can fit with 8$\times$A100-80G for Qwen-2-7B. We find that the embedding size significantly impacts the maximum sequence length in our optimized codebase. Qwen2 has a huge vocabulary of 152K tokens. For LLaMA2 with 32K vocabulary, we can train it with 700K context length.) over a total of 900M tokens. We follow [] to increase RoPE [] base frequency during the continued pertaining and specifically set it to 1B. A constant learning rate of 1e-5 is maintained for a batch size of one million tokens across 1,000 training steps. Following [], we construct the dataset used for long context training from Slimpajama [] by upsampling documents longer than 4096 and keeping the domain mixture ratio unchanged. Multiple documents are packed into a single sequence separated by a BOS token.

We employed several optimization strategies to perform training on such long sequences. These includes FlashAttention-2 [], Ring Attention [, ], activation checkpointing, and parameter offload []. To balance the load across different GPUs, we shard the sequence in a zigzag way [] in ring attention. The resulting training framework is memory efficient and maintains very high GPU occupancy. Note that we do not use any parameter-efficient methods such as LoRA [] or approximate attention []. With those optimizations, the compute used in long context training is minimal compared to that of language model pretraining, making it feasible for academic budgets. The long context training can finish in 2 days with 8 A100 GPUs.

In Figure 7, we evaluate the extended Qwen2 with the Needle-in-a-haystack (NIAH) test [, ]. It achieves perfect results within the training context length (224K) and generalizes even further. We find the vanilla NIAH to be a relatively trivial benchmark and further test it with 5 distractors randomly inserted into the documents. The detailed configuration can be found in Appendix A.

### 3.2 Aligning Long Language Model Using Short Vision Data

Inspired by the AnyRes encoding scheme in LLaVA-NeXT [, ], we designed UniRes that provides a unified encoding scheme for both images and videos, as shown in Figure 2. Unlike AnyRes which retains a small base image and flattens ViT patches across the grids, UniRes removes the base image, flattens patches within each grid, and 2x2 pool the visual features by default (Appendix B). This approach allows us to maintain consistent representation when extending image data into videos where multiple frames are viewed as multiple grids in a row.

Specifically, UniRes divides an input image of resolution $a\times b$ into smaller grids, each with a resolution of $336\times 336$ pixels. This results in $(a//336)\times(b//336)$ grids. For very high-resolution images, we limit the maximum number of grids to 49, resizing images larger than this threshold. Each grid is separately encoded using CLIP-ViT-L-336px [] and then projected through a 2-layer MLP to match the LM’s input dimension, resulting in 576 features per grid. We then apply 2x2 average pooling, finally converting an $a\times b$ image into $(a//336)\times(b//336)\times 144$ tokens. During inference, this visual encoding scheme allows videos to be represented as very long images (even though we do not train on videos). An $N$-frame video is treated as an image of size $336\times(336\times N)$, divided into $N$ grids where each grid corresponds to a video frame. Using CLIP encoding, MLP projection, and average pooling, an $N$-frame video is encoded into $144N$ visual tokens.

To clearly ablate the long context transfer phenomenon from language to vision, we adopt a train short, test long protocol where we only use image-text data during training, but test on long videos. We trained our model using the same data recipe and two-stage training approach as LLaVA-1.6. Our experiments show that compared to AnyRes, UniRes has slightly lower scores on low-resolution image benchmarks (Table 7) but performs better on V-NIAH (Figure 4) and Video-MME (Table 4). We believe the unified encoding scheme for images and videos is crucial, thus choosing this as the encoding scheme of LongVA. The image-text alignment can be finished in 1.5 days. With 2 days for long context training on text, the total training cost of LongVA is 3.5 days on 8$\times$A100-80G.

It is worth noting previous work largely inspired the design choice of LongVA. For example,[] first demonstrates the effectiveness of long context continued pretraining with increased RoPE base frequency (thus decreasing the rotation angles). We sample the long text data following the guidance of []. We adopt the same vision encoder and training data as that of LLaVA-1.6 []. We try to keep our methods as simple as possible to clearly show the phenomenon of long context transfer without other confounders.

Figure: Figure 2: UniRes’s unified encoding scheme of images and videos. During training, images are divided into multiple grids. During inference, videos are treated as extended images with each frame considered as a grid.
Refer to caption: extracted/5701582/image/uniresv2.png

## 4 V-NIAH

To measure the context length of language models on extremely long input, earlier works calculate perplexity scores over long documents. Recently, many have started using the Needle-in-a-Haystack (NIAH) test to benchmark LLMs’ ability to retrieve long context information precisely. We note that there is so far no benchmark to measure the visual context length of LMMs.
To evaluate LongVA’s capacity to locate and retrieve long-range visual information, we extend the NIAH test from text to video and propose V-NIAH.

As shown in Table 9, we designed 5 video question-answering problems as the needle and inserted each as a single frame into hours-long videos. We sampled the videos at 1 FPS as the visual input. The image of the needle is sourced from existing VQA benchmarks or AI-generated to avoid any contamination. The AI-generated images and questions are purposely chosen to be "counterfactual" or "counter-commonsense", ensuring the model cannot answer based on language knowledge alone. Each question includes a "locating prompt" so that a capable system or human can locate the needle frame from the video haystack and answer the question.

When testing LongVA with visual inputs of up to 3000 frames, one difficulty we encountered was that processing a 200K-token input requires up to 100GB of GPU memory for the KV cache for a 7B LM like LLaMA. Even with advanced LM serving systems like vLLM [] with tensor parallelism to shard the KV cache across multiple GPUs, the sampling process remains extremely slow due to limited memory and batchsize. To address this, we used "perplexity-based" evaluation to measure the correctness of the model output. We first encode all frames and save their corresponding visual embeddings. During the evaluation, we only load the language model from LongVA and concatenate the visual embeddings, question tokens, and answer tokens for a single forward pass with ring attention. This approach makes the workload compute-bound and eliminates the need to cache the KV state. The model’s output is considered correct only if the highest output logits index of all tokens in the answer span matches the correct answer.

Figure: Figure 3: V-NIAH consists of a haystack video, a needle image, and a question related to the needle. The needle is inserted at various positions in the haystack video.
Refer to caption: x1.png

Figure: Figure 4: The V-NIAH results of LongVA and its baselines. The x-axis represents the total number of frames in the video haystack. The y-axis shows the position where the needle image is located. For instance, a frame depth of 0% would place the needle image at the very beginning of the video. The black dotted line denotes the training length of the backbone language model, with each frame corresponding to 144 tokens.
Refer to caption: extracted/5701582/image/video_niah.png

## 5 Experiments

**Table 4: Performance comparison of various LMMs on Video-MME [] without subtitles. LongVA achieves state-of-the-art results among 7B models. Its performance also increases with denser sampling of video frames.**
| Model | LLM Params | Frames | Short | Medium | Long | Overall |
| --- | --- | --- | --- | --- | --- | --- |
| InternVL-Chat-V1.5 [] | 20B | 010 | 60.2 | 46.4 | 45.6 | 50.7 |
| LLaVA-NeXT-Video-34B [] | 34B | 032 | 61.7 | 50.1 | 44.3 | 52.0 |
| VILA-1.5 [] | 34B | 008 | 68.1 | 58.1 | 50.8 | 59.0 |
| Qwen-VL-Chat [] | 07B | 004 | 46.9 | 38.7 | 37.8 | 41.1 |
| Video-LLaVA [] | 07B | 008 | 45.3 | 38.0 | 36.2 | 39.9 |
| ST-LLM [] | 07B | 064 | 45.7 | 36.8 | 31.3 | 37.9 |
| VideoChat2-Mistral [] | 07B | 016 | 48.3 | 37.0 | 33.2 | 39.5 |
| Chat-UniVi-V1.5 [] | 07B | 064 | 45.7 | 40.3 | 35.8 | 40.6 |
| VideoLLaMA2 [] | 08B | 016 | 56.0 | 45.4 | 42.1 | 47.9 |
| LLaVA-NeXT-Qwen2 | 07B | 032 | 58.0 | 47.0 | 43.4 | 49.5 |
| LongVA | 07B | 008 | 55.1 | 46.3 | 42.1 | 47.9 |
| 016 | 59.0 | 46.6 | 43.6 | 49.7 |  |  |
| 032 | 61.1 | 48.8 | 45.4 | 51.8 |  |  |
| 064 | 61.4 | 50.9 | 45.0 | 52.4 |  |  |
| 128 | 61.1 | 50.4 | 46.2 | 52.6 |  |  |
| 384 | 60.3 | 48.9 | 46.1 | 51.8 |  |  |

**Table 5: LongVA and its baselines.**
| Model | Qwen2-224K | UniRes |
| --- | --- | --- |
| LLaVA-Next-Qwen2 | $\times$ | $\times$ |
| LongVA (AnyRes) | $\checkmark$ | $\times$ |
| LongVA | $\checkmark$ | $\checkmark$ |

We primarily assess the long visual capability of LongVA on two benchmarks: V-NIAH (Section 5.1 and Video-MME [] (Section 5.2). V-NIAH provides quick signals about the visual context length of LongVA. However, it only tests the model’s ability to retrieve information and does not cover other abilities necessary for a real-world long video assistant. Therefore, we also include LongVA’s performance on Video-MME, a comprehensive evaluation suite for video LMMs that includes diverse data types and qualitative annotations. Video-MME is an ideal benchmark for assessing LMMs’ ability to handle long videos in real-world scenarios, given its average video duration of 1017 seconds and the inclusion of short, medium, and long subsets. We further include the benchmark results on MLVU[] in Appendix C.

We mainly compare LongVA against other image and video LMMs. To validate the phenomenon of long context transfer, we trained LLaVA-Next-Qwen2, a baseline model based on Qwen2-7B-Instruct using the LLaVA-NeXT [, ] training recipe. Additionally, we trained LongVA (AnyRes) to showcase the advantages of our UniRes encoding scheme. The difference between LongVA and our baselines can be found in Table 5.

### 5.1 V-NIAH Results

Long context transfers from language to vision Figure 4 shows the V-NIAH performance of LongVA and other LMMs. Specifically, Figure 4 (iii) demonstrates that the visual context length of LLaVA-NeXT-Video-32K [] is constrained by the 32K context length of its language backbone, Mistral-7B-Instruct-v0.2 [], equivalent to approximately 200 frames. Beyond this limit, the V-NIAH accuracy drops significantly. As a stronger baseline, we include the results of LLaVA-NeXT-Video-32K enhanced with a training-free length extrapolation algorithm [] by increasing its RoPE base frequency. We empirically determine the optimal extrapolation frequency by choosing from [3M, 10M, 30M, 100M, 300M, 1B]. As indicated in Figure 4 (iv), although this training-free extrapolation allows the model to process information across an extended context, the improvement is marginal. These findings led us to develop LongVA, a model that unlocks the visual context by extending the language model purely on text. As shown in Figure 4 (i), LongVA can almost perfectly retrieve information and answer the needle question for input frames fewer than 2000. Although we only trained LongVA’s language backbone on a context length of 224K (equivalent to 1555 frames), it generalizes well beyond that, maintaining satisfactory performance within 3000 frames. Those results clearly corroborate of hypothesis of long context transfer.

Unified encoding enables better visual context extrapolation
We also present the V-NIAH heatmap of LongVA trained with AnyRes encoding scheme, keeping all other factors unchanged in Figure 4 (ii). LongVA-AnyRes demonstrates strong retrieval capabilities. However, its performance still lags behind LongVA trained with UniRes. We believe that the unified representation of images and videos in UniRes, where a video is encoded in the same way as a long image, enhances the long context transfer from language to vision. This approach also facilitates effective training with short vision data (images) and enables zero-shot understanding of long videos during inference.

### 5.2 Video Evaluation

On Video-MME (Table 4), LongVA achieves state-of-the-art performance among LMMs under 10B parameters, rivaling much larger ones such as LLaVA-NeXT-Video-34B [] and InternVL-Chat-V1.5 []. Notably, LongVA is trained without any video data, so its performance on video can be considered zero-shot. As the number of sampled frames increases, LongVA shows improved performance on the long subset, handling up to 384 frames(^2^22We limited our analysis to 384 frames due to computational and memory constraints as detailed in Section 4.). Even though LongVA’s score slightly drops when we upsample from 128 to 384 frames, it maintains a competitive performance. To our knowledge, LongVA is the only open-source model that can handle such large input frames on Video-MME.
These findings highlight the long context transfer effect, where LongVA, originating from a long context language model, can process significantly more frames than its baseline, despite being trained on the same multimodal data.

We also tested LongVA on shorter benchmarks with average video durations under 120 seconds. As indicated in Table 6, although LongVA scores higher with more densely sampled frames on datasets such as NeXTQA [] and ActivityNetQA [], the gains quickly plateau and are not as significant as those observed in Video-MME, which can be attributed to the shorter duration of these datasets. On the VideoChatGPT and Video Detailed Description (Video-DD) benchmarks, increasing frames does not lead to better performance, and LongVA generally achieves lower scores compared to LLaVA-NeXT-Video-7B.
Since both benchmarks use OpenAI’s GPT API as a judge, we believe their metrics are closely related to the answering format. To address this, we perform a lightweight Direct Preference Optimization (DPO) on the LLaVA-Hound-DPO [] dataset. We observe significantly improved performance for LongVA-DPO, confirming the findings in [].

**Table 6: Video evaluation results for LongVA on various short video benchmarks with comparison to 7B scale models.**
| Model |  | NeXTQA [] | ActivityNetQA [] | VideoChatGPT [] | Video-DD [] |  |  |  |  |  |
| --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- |
|  | frames | MC | OE | Score | Consistency | Correctness | Detail | Context | Temporal | Score |
| LLaVA-NeXT-Video [] | 32 | 57.93 | 26.90 | 3.20 | 3.12 | 3.39 | 3.29 | 3.92 | 2.60 | 3.32 |
| LongVA | 08 | 50.78 | 27.71 | 2.73 | 3.73 | 3.09 | 3.14 | 3.72 | 2.39 | 3.19 |
| LongVA | 16 | 61.61 | 27.87 | 2.78 | 3.61 | 3.13 | 3.15 | 3.75 | 2.40 | 3.22 |
| LongVA | 32 | 67.08 | 27.87 | 2.80 | 3.65 | 3.08 | 3.10 | 3.74 | 2.28 | 3.19 |
| LongVA | 64 | 68.27 | 27.81 | 2.84 | 3.64 | 3.05 | 3.09 | 3.77 | 2.44 | 3.14 |
| LongVA-DPO | 32 | 69.26 | 28.02 | 2.80 | 4.07 | 3.55 | 3.32 | 4.09 | 2.86 | 3.58 |

### 5.3 Image Evaluation

**Table 7: Image evaluation results for LongVA on multiple benchmarks. Compared to other image multimodal models, our methods maintain high performance and achieve better scores on InfoVQA[].**
| Model | AI2D [] | ChartQA [] | DocVQA [] | InfoVQA [] | RealworldQA [] | MMMU [] |
| --- | --- | --- | --- | --- | --- | --- |
| LLaVA-1.6-Vicuna [] | 66.6 | 54.8 | 74.4 | 37.1 | 57.8 | 35.1 |
| LLaVA-NeXT-LLaMA3 [] | 71.6 | 69.5 | 78.2 | 37.6 | 60.0 | 41.7 |
| LLaVA-NeXT-Qwen2 | 73.5 | 74.0 | 81.3 | 42.0 | 61.6 | 41.9 |
| LongVA (AnyRes) | 73.1 | 74.4 | 81.5 | 43.3 | 62.4 | 42.1 |
| LongVA (UniRes) | 70.7 | 70.4 | 80.8 | 49.4 | 60.0 | 42.6 |

Figure: Figure 5: The 2D-histogram of the image width and height of different image benchmarks. InfoVQA[] consists of many high-resolution images compared to other benchmarks.
Refer to caption: extracted/5701582/image/ai2d_image_plot.jpg

We further evaluate our model on various image benchmarks to investigate the image performance of LongVA (Table 7). Compared to the LongVA (AnyRes) baseline, LongVA with UniRes achieves significantly increased performance on InfoVQA [], while the scores drop to some extent on AI2D [] and ChartQA []. To better understand this phenomenon, we recorded and analyzed the image size of those datasets, as shown in Figure 5. We found that InfoVQA consists of higher-resolution images, while many images in AI2D and ChartQA are smaller than 768$\times$768.
Compared to Anyres, UniRes operate 2$\times$2 average pooling on each image, reducing to $1/4$ visual tokens per image grid. However, the grid upper bound is set to 49 for UniRes while 4 for AnyRes, so UniRes may produce more image grids if the input images are of higher resolution. By using more grids per image, UniRes allocates more visual tokens on datasets such as InfoVQA, achieving superior performance compared to the previous 7B LLaVA model. However, most of the images in ChartQA and AI2D require fewer than 4 grids to represent. This may explain why the image performance decreases on those benchmarks.

## 6 Qualitative Results

The qualitative results of LongVA-DPO are illustrated in Figure 6. The short video example comes from [] and the two long videos are sourced from [link1](https://www.bilibili.com/video/BV1Kn4y1d7G7/?buvid=c066506d947f48e761c78e00991fd406&from_spmid=main.space-contribution.0.0&is_story_h5=false&mid=804D54DYmsNyGN1vdgHxnA%3D%3D&p=1&plat_id=116&share_from=ugc&share_medium=iphone&share_plat=ios&share_session_id=35F6AE42-FAD2-436C-B164-107EF91E283D&share_source=WEIXIN&share_tag=s_i&spmid=united.player-video-detail.0.0&timestamp=1719191857&unique_k=AOiQBaI&up_id=23947287&share_source=weixin) and [link2](https://www.bilibili.com/video/BV1F4411v7rj/?spm_id_from=333.1007.top_right_bar_window_custom_collection.content.click&vd_source=a8b9ec3a8fda7923ed9b151c46917050), respectively. In the figure, LongVA accurately describes the short, humorous video involving individuals playfully interacting with condiments. It also identifies specific details in long videos, such as the color of a train and the colors of umbrellas used in a scene, showcasing its proficiency in retrieving and interpreting visual information over extended video contexts. These capabilities highlight LongVA’s potential to overcome the challenges associated with processing and understanding extremely long videos.

Figure: Figure 6: Qualitative Result of the LongVA-DPO. Our model demonstrates superior capability in understanding both short and long videos.
Refer to caption: x2.png

## 7 Conclusion

This work addresses the challenges of understanding long videos in Large Multimodal Models. By extending the language model on text and then aligning this extended model with visual inputs, we significantly improved the capability of LMMs to handle long videos thanks to the long context transfer phenomenon. Our model, LongVA, shows improved performance with more input frames and achieves state-of-the-art results on Video-MME. Additionally, we introduce a synthetic benchmark, V-NIAH, to effectively measure the visual context length of video LMMs. We hope this work inspires further research in the field of long video LMMs and multimodal agents.

## 8 Acknowledgements

This research/project is supported by the National Research Foundation, Singapore under its AI Singapore Programme (AISG Award No: AISG2-PhD-2022-01-029). Besides, this project is supported by NTU NAP, MOE AcRF Tier 2 (MOE-T2EP20221-0012), and under the RIE2020 Industry Alignment Fund – Industry Collaboration Projects (IAF-ICP) Funding Initiative, as well as advise from the industry partner(s).

## Appendix

## Appendix A Needle In A Haystack Test

Figure: Figure 7: The NIAH results of Qwen-7B-Instruct after long context training.
Refer to caption: extracted/5701582/image/text_niah.jpg

When evaluating the Needle In A Haystack task [], we focus specifically on an easier-to-evaluate variant [] that involves identifying and retrieving random numbers associated with various randomly assigned cities from the context. The input to the language model has below template:

We insert a needle with the key Singapore and a 7-digit randomly sampled magic number as the value into the haystack of Paul Graham’s Essays. The needle has the following format:

We iterate over various document depths (where the needle is placed) and context lengths to measure the performance. For each depth and context length, we conducted the test 5 times, each time with a different 7-digit needle. We also come up with a harder version where we also insert several (3 or 5) other needles with the same format but different city name as distractors. The results are shown in Figure 7.

## Appendix B UniRes Encoding Scheme

Figure: Figure 8: The difference between AnyRes and UniRes, assuming the image is divided into 2x2 grids and the video has 4 frames. The number indicates the flattening order. Additionally, UniRes applies 2x2 average pooling to both images and videos after the MLP projector between the vision encoder and the language model.
Refer to caption: extracted/5701582/image/anyres_unires.png

Figure 8 indicates the difference between AnyRes and UniRes. Given a high-resolution image and assuming we use CLIP-ViT-L-336px as the vision encoder, both AnyRes and UniRes will divide it into multiple grids, each with the size 336x336. However, AnyRes will have a smaller version of the full image as the base image and prepended before the high-resolution image grids. Additionally, UniRes flattens the encoded image feature in a raster-order within each grid, while AnyRes combines all the grids as a big feature map and flattens them across the border of the grid. UniRes also apply 2x2 average pooling on the image feature. As shown in the rightmost part of Figure 8, the design of UniRes allows us to unifiedly encode videos as well. A video is treated as an extended image where each frame is considered as an image grid.

## Appendix C MLVU Results

**Table 8: Evaluation results by the authors of MLVU []. The highest scores excluding GPT-4o are bolden. TR: Topic Reasoning. AR: Anomaly Recognition. VS: Video Summary; NQA: Needle QA; ER: Ego Reasoning; PQA: Plot QA; SSC: Sub-Scene Captioning; AO: Action Order; AC: Action Count; M-Avg: the average performance of multiple-choice tasks; G-Avg: the average performance of generation tasks.**
| Methods | Input | Holistic | Single Detail | Multi Detail | Averages |  |  |  |  |  |  |  |
| --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- |
| TR | AR | VS | NQA | ER | PQA | SSC | AO | AC | M-Avg | G-Avg |  |  |
| GPT-4o [] | 0.0.5 fps | 87.4 | 74.5 | 4.90 | 64.8 | 57.1 | 65.1 | 6.69 | 56.7 | 46.3 | 64.6 | 5.80 |
| LLaVA-1.6 [] | 0016 frm | 60.6 | 41.0 | 2.11 | 43.1 | 38.4 | 41.0 | 4.35 | 25.5 | 25.7 | 39.3 | 3.23 |
| InternVL-1.5 [] | 0016 frm | 78.8 | 67.0 | 3.16 | 52.7 | 43.5 | 54.4 | 4.88 | 32.8 | 23.8 | 50.4 | 4.02 |
| MovieChat [] | 2048 frm | 29.5 | 25.0 | 2.33 | 24.2 | 24.7 | 25.8 | 3.23 | 28.6 | 22.8 | 25.8 | 2.78 |
| TimeChat [] | 0096 frm | 23.1 | 27.0 | 2.54 | 24.5 | 28.4 | 25.8 | 4.29 | 24.7 | 32.0 | 30.9 | 3.42 |
| LLaMA-VID [] | 0001 fps | 50.8 | 34.5 | 3.22 | 30.1 | 32.7 | 32.5 | 5.22 | 23.9 | 27.8 | 33.2 | 4.22 |
| MA-LMM [] | 1000 frm | 51.9 | 35.5 | 2.12 | 43.1 | 38.9 | 35.8 | 4.80 | 25.1 | 24.3 | 36.4 | 3.46 |
| ShareGPT4Video [] | 0016 frm | 75.8 | 51.5 | 2.52 | 47.6 | 43.2 | 48.4 | 5.02 | 34.0 | 23.3 | 46.4 | 3.77 |
| VideoChat2_HD [] | 0016 frm | 77.3 | 60.5 | 3.38 | 46.2 | 48.9 | 50.1 | 4.59 | 23.2 | 29.1 | 47.9 | 3.99 |
| VideoLlaMA2 [] | 0016 frm | 74.6 | 64.5 | 2.79 | 49.9 | 43.8 | 45.1 | 5.18 | 34.0 | 27.4 | 48.5 | 3.99 |
| LongVA (ours) | 0256 frm | 83.3 | 58.5 | 3.39 | 69.3 | 50.0 | 67.2 | 5.26 | 38.6 | 27.2 | 56.3 | 4.33 |

Table 8 includes the evaluation results by the authors of MLVU [] on their benchmark. LongVA achieves state-of-the-art results among open-source models and is only second to GPT-4o.

## Appendix D Visual Needle In A Haystack Test

Table 9 lists the five VQA needles we used for V-NIAH. The 5 visual questions and answers are the only places where human annotation is involved in the construction of V-NIAH, making it an ideal testbed to benchmark LMMs’ long context capability.

Table: Table 9: The design of the 5 visual question-answering problems used as the needle in V-NIAH.

## Peer Reviews

confidence: 3
contribution: 3
initial_rating: 6
presentation: 2
questions: What would be the performance of VILA-1.5 on the V-NIAH benchmark?
soundness: 3
strengths: 1. The proposed method achieves the state of the art on multiple video understand benchmarks. 

2. The proposed method is simple and effective. The continuous pretraining on large context window only needs text data, while the multimodal alignment stage only requires image-text data. The uniRes encoding is straightforward extension of AnyRes to videos. Nevertheless, the model seems to generalize surprisingly well to videos. 

3. The Needle-in-a-Haystack test seems to be novel in the context of long video understanding. This new task could be an interesting testbed for long video understanding in the future.
summary: Current LLMs struggle to understand very long videos because of the context length limitation. This paper addresses this issue by proposing a new method that extends the input context length of LLMs. The method includes three key ingredients: 1) continuous pretraining the LLM with a large context window on text-only dataset, 2) aligning the long-context LLM with image data, 3). tokenizing images (or videos) with the so called UniRes that divides images into grids and flatten the visual tokens in each grid. To evaluate the effectiveness of the method on long video understanding, they introduce a new benchmark called V-NIAH where the task is to retrieve one frame that is randomly inserted into a long video. They also evaluate the method on other video question answering benchmarks. On all the benchmarks, their model "LongVA" demonstrates state-of-the-art performance.
weaknesses: 1. Despite the impressive empirical results, there are not much explanations. In particular, I am interested in understanding why the long-context pretraining on text-only data could transfer to long video understanding.

2. It is not clear on how the image-text alignment is done e.g., training data and losses. Most of the content in the Sec. 3.2 is about the UniRes which is relatively straightforward. 

3. The number of grids during training is limited to 49 (Line 205), which is effectively equivalent to 49 frames. Why does the model could generalize to handle more than 2K frames give that it is trained with only 49 frame inputs?

4. The results in Table 4 show that LongVA outperforms the methods that are much larger and trained on videos e.g., LLaVA-NeXT-Video-34B. Meanwhile, VILA-1.5 achieves the best results with only 8 frames. Those results seem to imply that the input frames and video-specific training are not that important for video understanding. I found this phenomenon counter intuitive. But this might be caused by the different LLM backbones. One interesting experiment I would suggest is to ablate context length and video specific training with the same LLM. 

5. Temporal understanding is arguably an important aspect in video understanding. Being able to process a large number of video frames does not indicate a strong capability in temporal understanding. Most of the experiments shown in the paper can be solved by "selecting" one frame or very few frames. The model might not be able to understand fine-grained motions/actions, for example Something-Something V2 dataset.

---

confidence: 5
contribution: 2
initial_rating: 5
presentation: 2
questions: (1) Can training only be performed on a single image? How is the long text obtained in image-text training? Because the trained LLM backbone has a context length of 224K. The training details are not provided.

(2) According to the display of Figure 2, the author treats each frame of the video as a grid of the image during testing, so 576 tokens are formed for inference. In order to input more frames during inference, my understanding is that during image-text training, the image here needs to have a very high resolution? In order to segment more grids, so as to adapt to more frames during inference. How are high-resolution images obtained? For the collection process of the training set, this seems to be a method of exchanging spatial information annotation for temporal information annotation.

(3) Unless the details of the training are given, including the amount of data, image resolution, and text length, it is difficult to understand why simple image-text training can be transferred to the reasoning of videos with very long frames?

(4) Why can image training achieve long-context video MLLM? The author did not discuss and analyze the principle.

(5) The experimental results in Table 4 show that inputting more frames into LongVA does not continuously enhance the performance of Video-MME? Especially in long duration videos, why is this? And as the number of frames continues to increase, it gets worse in short, medium and long. Although other experiments show that LongVA performs well in VNIAH, demonstrating the ability of information retrieval, it does not show the advantage of this model in long video understanding.

(6) In the experimental results of Table 7, in 6 benchmarks, UniRes is better than AnyRes in 4 tasks, which does not prove that UniRes is a better visual encoding method. Why use UniRes instead of AnyRes?
soundness: 2
strengths: (1) Explored a method to implement long-context video MLLM from the perspective of long-context text training. The idea is very interesting.  
 
(2) The evaluation of long-context text retrieval (NIAH) and video retrieval (V-NIAH) is very complete and well presented.   

(3) The experimental results are rich. LongVA was evaluated on different multi-modal benchmarks and achieved SOTA performance on Video-MME and MLVU.
summary: This paper proposes a novel way to implement long-context Video MLLM, which only requires long context training of plain text and image-text alignment training.
- The trained long-context LLM is used for image-text training. Through the designed UniRes visual information encoding method, the long-context LLM can be extended to the long-context video MLLM without video-text training.
- A Visual Needle-In-A-Haystavk (V-NIAH) benchmark is proposed to test the capabilities of long-context video MLLM.
- By inputting more frames for reasoning, the proposed LongVA achieves SOTA performance on Video-MME and MLVU.
weaknesses: (1) The results of LongVA on some benchmarks are not ideal, such as ActivityNetQA and VideoChatGPT, which is reflected in the fact that more frames do not show better video understanding ability. This makes people suspect that LongVA only has the ability to retrieve long-context video information, but lacks the ability to understand long-context videos.

(2) The paper does not give detailed training details, such as the training details of the long-context LLM of pure text, the training details of image-text, and the datasets used in these two training processes.

(3) Some experimental results in the article do not show that a long-context MLLM with good video understanding ability has been achieved. And the proposed UniRes has not been proven to be better than the previous AnyRes.

---

confidence: 4
contribution: 3
initial_rating: 8
presentation: 4
questions: See "Weakness" 1 & 2.
soundness: 4
strengths: 1. Novel Perspective: The paper presents a fresh perspective on handling long video sequences in LMMs by identifying the language model's context length as the primary bottleneck rather than following the conventional approach of reducing visual tokens. This reframing of the problem leads to a novel solution pathway.

2. Technical Innovation: The discovery of the "long context transfer" phenomenon represents a significant contribution to the field. The ability to transfer extended context capabilities from language to vision without explicit long video-text pair training is both elegant and practical, potentially influencing future research directions in multimodal learning.

3. Benchmark Contribution: The introduction of V-NIAH provides the community with a much-needed evaluation framework for assessing visual context length capabilities in LMMs. This standardized benchmark addresses a gap in existing evaluation metrics and will facilitate future research in long-form video understanding.

4. Practical Impact: The achieved results are impressive, with LongVA demonstrating the ability to process over 200K visual tokens and showing state-of-the-art performance on established benchmarks. The scalability of the approach (showing improved performance with more input frames) suggests real-world applicability.

5. Methodological Clarity: The proposed UniRes encoding scheme offers a unified approach to handling both images and videos, promoting better integration of these modalities and potentially simplifying the architecture of multimodal systems.
summary: This paper focuses on the challenge of processing long video sequences in Large Multimodal Models (LMMs). Instead of focusing on reducing visual tokens, the authors identify the language model's context length as the key limitation and propose a novel approach called "long context transfer." The method first extends the context length of the language model through training on longer text data, then performs modality alignment without requiring long video-text pairs. They introduce UniRes, a unified encoding scheme representing videos as extended images. They propose V-NIAH, a synthetic benchmark based on the Needle-in-a-haystack test, to evaluate visual context length capabilities. Their resulting model, Long Video Assistant (LongVA), can effectively process over 200K visual tokens (approximately 2000 frames) and achieves state-of-the-art performance on Video-MME and MLVU datasets among 7B-parameter models, demonstrating that increased frame input during inference leads to improved performance on long video question-answering tasks.
weaknesses: 1. Visual Encoding Ablations: The paper lacks comprehensive analysis of different visual encoding schemes' impact on performance. Specifically:
   - The comparison between UniRes, AnyRes, and Higher-AnyRes is not thoroughly explored
   - The effect of including or excluding base image encoding is not clearly demonstrated
   - The impact of different layout configurations for extended images (1 x N, N x 1, etc.) is not investigated
   These ablation studies would provide valuable insights into the robustness and generalizability of the proposed approach.

2. Limited Context Analysis: Given that the paper's core contribution is the long context transfer phenomenon, a more detailed analysis of how different context lengths in the language model correlate with visual understanding capabilities would strengthen the claims. The mechanism behind this transfer phenomenon could be better explained.

3. Typos: The paper contains several typographical errors, specifically in the references on lines 123 and 125, where parentheses and spaces are missing. While minor, such oversights affect the paper's professional presentation and should be addressed in the final version.

---

confidence: 4
contribution: 2
initial_rating: 5
presentation: 3
questions: Refer to weakness.
soundness: 3
strengths: * The paper is well-written and easy to understand. The figure illustration and captions are informative.
* The authors contributed the V-NIAH benchmark, which is an effective benchmark to test LMMs' ability to locate and retrieve visual information over extremely long contexts.
*  Comprehensive evaluations on two video understanding benchmarks (Video-MME and MLVU) beat most of the baselines.
summary: The paper addresses a significant limitation in large multimodal models (LMMs) concerning their ability to process extremely long video sequences. The existing approaches primarily focused on reducing the number of visual tokens using visual resamplers, this work proposes a novel perspective by extending the context length of the language model backbone, which allows LMMs to interpret a substantially larger number of visual tokens—up to 2000 frames or over 200,000 visual tokens—without requiring additional video training.  The authors also introduce V-NIAH, a synthetic benchmark designed to assess LMMs' generalization capabilities in handling long visual contexts. The results indicate that the proposed method achieves state-of-the-art performance on the Video-MME task among 7B-scale models.
weaknesses: * The technical contributions of this paper are limited. It appears that the authors' only contribution is the benchmark used to evaluate long-context understanding capabilities, which may not be sufficient for a paper presented at the main conference of ICLR.
* The authors' design to enhance the capability of video understanding in long contexts only involves using a large language model (LLM) with long-context capabilities. Such an improvement is obvious and does not provide new insights.
* Where is the **transfer** mentioned in the title of the article reflected? The authors do not seem to have made any special design considerations for transfer, aside from using a long-context LLM as the base model.

---

confidence: 5
contribution: 3
initial_rating: 5
presentation: 3
questions: Limitations of Long Context Transfer: Can the authors provide more insights into the limitations of the long context transfer phenomenon? Are there scenarios where this approach might not be effective?
soundness: 4
strengths: Novel Approach: The paper introduces a novel method for enabling LMMs to process extremely long videos by extending the context length of the language model, rather than reducing the number of visual tokens. This shift in perspective is innovative and opens new avenues for research in multimodal learning.

Long Context Transfer Phenomenon: The discovery of the long context transfer phenomenon is significant. It demonstrates that the extended context capabilities of a language model can directly benefit the vision modality without additional long video training, which is a valuable insight.

V-NIAH Benchmark: The development of V-NIAH provides a useful synthetic benchmark for evaluating LMMs' ability to handle long visual contexts. This can serve as a valuable tool for the research community to assess and compare models.

Strong Experimental Results: LongVA achieves state-of-the-art performance among 7B-scale models on the Video-MME and MLVU datasets. The experimental results are comprehensive and support the claims made in the paper.

Ablation Studies: The paper includes thorough ablation studies that analyze the properties of long context transfer and the impact of different components of the model, adding depth to the evaluation.
summary: This paper addresses the challenge of enabling Large Multimodal Models (LMMs) to process and understand extremely long videos. While previous works have focused on reducing the number of visual tokens through visual resamplers, this paper approaches the problem from the perspective of the language model. By extending the context length of the language backbone, the authors enable LMMs to process significantly more visual tokens without any additional video training. They term this phenomenon "long context transfer," where the extended context capabilities of the language model transfer to the vision modality.

To evaluate this capability, the authors introduce V-NIAH (Visual Needle-In-A-Haystack), a synthetic benchmark inspired by the NIAH test for language models, designed to test LMMs' ability to locate and retrieve information from extremely long visual contexts. They propose LongVA (Long Video Assistant), an LMM that, through context length extrapolation in the language model, can process over 200K visual tokens or 2000 frames without additional complexities or long video training. LongVA achieves state-of-the-art performance among 7B-scale models on the Video-MME and MLVU datasets by densely sampling more input frames.
weaknesses: Latest LLMs like llama3.2 already has context length up to 128k, which makes this work less valuable. 

Limited Real-World Evaluation: While V-NIAH is a useful synthetic benchmark, it may not fully capture the complexities of real-world long video understanding. The reliance on synthetic data could limit the assessment of the model's practical applicability in real-world scenarios.

Potential Limitations: The paper could delve deeper into potential limitations or failure cases of the long context transfer phenomenon. Understanding scenarios where this approach might not be effective would strengthen the work.

Clarity in Presentation: While the paper is generally well-written, some sections could benefit from clearer explanations. For example, more detailed descriptions of technical aspects and improved figures could enhance overall readability.

Meta-review: This paper proposes an alternate approach to long-video understanding with VLMs: The authors train the language model with longer textual context lengths, and show how this transfers to the visual modality: ie a language model trained with larger context lengths can perform better on video understanding tasks, even without training on long videos. In addition, the authors also proposed a Visual Needle-in-a-Haystack (V-NIAH) benchmark, and an extension of the Qwen2 VLM that achieves better results on the Video-MME and MLVU datasets.

Reviewers opinions were quite divergent on the paper: Some reviewers appreciated the overall approach which suggests that we don't need video-specific training data, but can simply train with long-context text tasks to perform well for long-context video tasks. However, other reviewers were concerned about the lack of technical novelty, and the fact that although the authors show impressive (and perhaps counterintutive) results on Video-MME and MLVU, there is not much analysis of why the proposed "long context transfer" works. For example, what the failure cases are, in what domains of problems can we expect textual long-context to transfer, and for what domains we actually need video-specifc training data? Although the authors compare to the state-of-the-art in Tables 4 through 7, there is not much analysis of why the proposed method works.

Furthermore, although the authors perform Needle-in-a-haystack experiments, there are also a number of recent works in this area which were not discussed at all [A, B, C, D]. In addition to Video-MME, there are also more recent, real-world video understanding datasets which could also provide more understanding about the "long context transfer" phenomenon. Examples include LVBench (longer, real-world videos) and Egoschema (the videos are not as long, but in contrast to needle-in-a-haystack tasks, require looking at the whole video).

Therefore, after some deliberation, the AC and Senior AC believe that this paper could do with further iteration and encourage the authors to revise the paper in light of reviewers' comments and resubmit to another venue.


[A] Zhao et al. Needle in a video haystack: A scalable synthetic framework for benchmarking video mllms

[B] T Yuan et al. Lv-eval: A balanced long-context benchmark with 5 length levels up to 256k

[C] T Wu et al. Visual haystacks: Answering harder questions about sets of images.

[D] Gemini Team. Gemini 1.5: Unlocking multimodal understanding across millions of tokens of context.

Decision: Reject