Long Context Transfer from Language to Vision

Authors: Peiyuan Zhang, Kaichen Zhang, Bo Li, Guangtao Zeng, Jingkang Yang, Yuanhan Zhang, Ziyue Wang, Haoran Tan, Chunyuan Li, Ziwei Liu
Primary area: foundation or frontier models, including LLMs
Keywords: Vision Language Model, Long Context Model

Abstract:
Video sequences offer valuable temporal information, but existing large multimodal models (LMMs) fall short in understanding extremely long videos. Many works address this by reducing the number of visual tokens using visual resamplers. Alternatively, in this paper, we approach this problem from the perspective of the language model. By simply extrapolating the context length of the language backbone, we enable LMMs to comprehend orders of magnitude more visual tokens without any video training. We call this phenomenon long context transfer and carefully ablate its properties. To effectively measure LMMs' ability to generalize to long contexts in the vision modality, we develop V-NIAH (Visual Needle-In-A-Haystack), a purely synthetic long vision benchmark inspired by the language model's NIAH test. Our proposed Long Video Assistant (LongVA) can process 2000 frames or over 200K visual tokens without additional complexities. With its extended context length, LongVA achieves state-of-the-art performance on Video-MME among 7B-scale models by densely sampling more input frames.

Decision: reject
Reviewer scores: 6, 5, 8, 5, 5

Reviews and discussion:
<thread>
creation date:
1729990125218

thread_type:
official_review

<thread_object>
creator: reviewer_dyq4

<content>
summary:
This paper addresses the challenge of enabling Large Multimodal Models (LMMs) to process and understand extremely long videos. While previous works have focused on reducing the number of visual tokens through visual resamplers, this paper approaches the problem from the perspective of the language model. By extending the context length of the language backbone, the authors enable LMMs to process significantly more visual tokens without any additional video training. They term this phenomenon "long context transfer," where the extended context capabilities of the language model transfer to the vision modality.

To evaluate this capability, the authors introduce V-NIAH (Visual Needle-In-A-Haystack), a synthetic benchmark inspired by the NIAH test for language models, designed to test LMMs' ability to locate and retrieve information from extremely long visual contexts. They propose LongVA (Long Video Assistant), an LMM that, through context length extrapolation in the language model, can process over 200K visual tokens or 2000 frames without additional complexities or long video training. LongVA achieves state-of-the-art performance among 7B-scale models on the Video-MME and MLVU datasets by densely sampling more input frames.

strengths:
Novel Approach: The paper introduces a novel method for enabling LMMs to process extremely long videos by extending the context length of the language model, rather than reducing the number of visual tokens. This shift in perspective is innovative and opens new avenues for research in multimodal learning.

Long Context Transfer Phenomenon: The discovery of the long context transfer phenomenon is significant. It demonstrates that the extended context capabilities of a language model can directly benefit the vision modality without additional long video training, which is a valuable insight.

V-NIAH Benchmark: The development of V-NIAH provides a useful synthetic benchmark for evaluating LMMs' ability to handle long visual contexts. This can serve as a valuable tool for the research community to assess and compare models.

Strong Experimental Results: LongVA achieves state-of-the-art performance among 7B-scale models on the Video-MME and MLVU datasets. The experimental results are comprehensive and support the claims made in the paper.

Ablation Studies: The paper includes thorough ablation studies that analyze the properties of long context transfer and the impact of different components of the model, adding depth to the evaluation.

weaknesses:
Latest LLMs like llama3.2 already has context length up to 128k, which makes this work less valuable. 

Limited Real-World Evaluation: While V-NIAH is a useful synthetic benchmark, it may not fully capture the complexities of real-world long video understanding. The reliance on synthetic data could limit the assessment of the model's practical applicability in real-world scenarios.

Potential Limitations: The paper could delve deeper into potential limitations or failure cases of the long context transfer phenomenon. Understanding scenarios where this approach might not be effective would strengthen the work.

Clarity in Presentation: While the paper is generally well-written, some sections could benefit from clearer explanations. For example, more detailed descriptions of technical aspects and improved figures could enhance overall readability.

questions:
Limitations of Long Context Transfer: Can the authors provide more insights into the limitations of the long context transfer phenomenon? Are there scenarios where this approach might not be effective?
</content>
</thread_object>

</thread>
<thread>
creation date:
1730638507794

thread_type:
official_review

<thread_object>
creator: reviewer_kgwu

<content>
summary:
The paper addresses a significant limitation in large multimodal models (LMMs) concerning their ability to process extremely long video sequences. The existing approaches primarily focused on reducing the number of visual tokens using visual resamplers, this work proposes a novel perspective by extending the context length of the language model backbone, which allows LMMs to interpret a substantially larger number of visual tokens—up to 2000 frames or over 200,000 visual tokens—without requiring additional video training.  The authors also introduce V-NIAH, a synthetic benchmark designed to assess LMMs' generalization capabilities in handling long visual contexts. The results indicate that the proposed method achieves state-of-the-art performance on the Video-MME task among 7B-scale models.

strengths:
* The paper is well-written and easy to understand. The figure illustration and captions are informative.
* The authors contributed the V-NIAH benchmark, which is an effective benchmark to test LMMs' ability to locate and retrieve visual information over extremely long contexts.
*  Comprehensive evaluations on two video understanding benchmarks (Video-MME and MLVU) beat most of the baselines.

weaknesses:
* The technical contributions of this paper are limited. It appears that the authors' only contribution is the benchmark used to evaluate long-context understanding capabilities, which may not be sufficient for a paper presented at the main conference of ICLR.
* The authors' design to enhance the capability of video understanding in long contexts only involves using a large language model (LLM) with long-context capabilities. Such an improvement is obvious and does not provide new insights.
* Where is the **transfer** mentioned in the title of the article reflected? The authors do not seem to have made any special design considerations for transfer, aside from using a long-context LLM as the base model.

questions:
Refer to weakness.
</content>
</thread_object>

</thread>
<thread>
creation date:
1730704992913

thread_type:
official_review

<thread_object>
creator: reviewer_yach

<content>
summary:
This paper focuses on the challenge of processing long video sequences in Large Multimodal Models (LMMs). Instead of focusing on reducing visual tokens, the authors identify the language model's context length as the key limitation and propose a novel approach called "long context transfer." The method first extends the context length of the language model through training on longer text data, then performs modality alignment without requiring long video-text pairs. They introduce UniRes, a unified encoding scheme representing videos as extended images. They propose V-NIAH, a synthetic benchmark based on the Needle-in-a-haystack test, to evaluate visual context length capabilities. Their resulting model, Long Video Assistant (LongVA), can effectively process over 200K visual tokens (approximately 2000 frames) and achieves state-of-the-art performance on Video-MME and MLVU datasets among 7B-parameter models, demonstrating that increased frame input during inference leads to improved performance on long video question-answering tasks.

strengths:
1. Novel Perspective: The paper presents a fresh perspective on handling long video sequences in LMMs by identifying the language model's context length as the primary bottleneck rather than following the conventional approach of reducing visual tokens. This reframing of the problem leads to a novel solution pathway.

2. Technical Innovation: The discovery of the "long context transfer" phenomenon represents a significant contribution to the field. The ability to transfer extended context capabilities from language to vision without explicit long video-text pair training is both elegant and practical, potentially influencing future research directions in multimodal learning.

3. Benchmark Contribution: The introduction of V-NIAH provides the community with a much-needed evaluation framework for assessing visual context length capabilities in LMMs. This standardized benchmark addresses a gap in existing evaluation metrics and will facilitate future research in long-form video understanding.

4. Practical Impact: The achieved results are impressive, with LongVA demonstrating the ability to process over 200K visual tokens and showing state-of-the-art performance on established benchmarks. The scalability of the approach (showing improved performance with more input frames) suggests real-world applicability.

5. Methodological Clarity: The proposed UniRes encoding scheme offers a unified approach to handling both images and videos, promoting better integration of these modalities and potentially simplifying the architecture of multimodal systems.

weaknesses:
1. Visual Encoding Ablations: The paper lacks comprehensive analysis of different visual encoding schemes' impact on performance. Specifically:
   - The comparison between UniRes, AnyRes, and Higher-AnyRes is not thoroughly explored
   - The effect of including or excluding base image encoding is not clearly demonstrated
   - The impact of different layout configurations for extended images (1 x N, N x 1, etc.) is not investigated
   These ablation studies would provide valuable insights into the robustness and generalizability of the proposed approach.

2. Limited Context Analysis: Given that the paper's core contribution is the long context transfer phenomenon, a more detailed analysis of how different context lengths in the language model correlate with visual understanding capabilities would strengthen the claims. The mechanism behind this transfer phenomenon could be better explained.

3. Typos: The paper contains several typographical errors, specifically in the references on lines 123 and 125, where parentheses and spaces are missing. While minor, such oversights affect the paper's professional presentation and should be addressed in the final version.

questions:
See "Weakness" 1 & 2.
</content>
</thread_object>

</thread>
<thread>
creation date:
1730721369722

thread_type:
official_review

<thread_object>
creator: reviewer_92h6

<content>
summary:
This paper proposes a novel way to implement long-context Video MLLM, which only requires long context training of plain text and image-text alignment training.
- The trained long-context LLM is used for image-text training. Through the designed UniRes visual information encoding method, the long-context LLM can be extended to the long-context video MLLM without video-text training.
- A Visual Needle-In-A-Haystavk (V-NIAH) benchmark is proposed to test the capabilities of long-context video MLLM.
- By inputting more frames for reasoning, the proposed LongVA achieves SOTA performance on Video-MME and MLVU.

strengths:
(1) Explored a method to implement long-context video MLLM from the perspective of long-context text training. The idea is very interesting.  
 
(2) The evaluation of long-context text retrieval (NIAH) and video retrieval (V-NIAH) is very complete and well presented.   

(3) The experimental results are rich. LongVA was evaluated on different multi-modal benchmarks and achieved SOTA performance on Video-MME and MLVU.

weaknesses:
(1) The results of LongVA on some benchmarks are not ideal, such as ActivityNetQA and VideoChatGPT, which is reflected in the fact that more frames do not show better video understanding ability. This makes people suspect that LongVA only has the ability to retrieve long-context video information, but lacks the ability to understand long-context videos.

(2) The paper does not give detailed training details, such as the training details of the long-context LLM of pure text, the training details of image-text, and the datasets used in these two training processes.

(3) Some experimental results in the article do not show that a long-context MLLM with good video understanding ability has been achieved. And the proposed UniRes has not been proven to be better than the previous AnyRes.

questions:
(1) Can training only be performed on a single image? How is the long text obtained in image-text training? Because the trained LLM backbone has a context length of 224K. The training details are not provided.

(2) According to the display of Figure 2, the author treats each frame of the video as a grid of the image during testing, so 576 tokens are formed for inference. In order to input more frames during inference, my understanding is that during image-text training, the image here needs to have a very high resolution? In order to segment more grids, so as to adapt to more frames during inference. How are high-resolution images obtained? For the collection process of the training set, this seems to be a method of exchanging spatial information annotation for temporal information annotation.

(3) Unless the details of the training are given, including the amount of data, image resolution, and text length, it is difficult to understand why simple image-text training can be transferred to the reasoning of videos with very long frames?

(4) Why can image training achieve long-context video MLLM? The author did not discuss and analyze the principle.

(5) The experimental results in Table 4 show that inputting more frames into LongVA does not continuously enhance the performance of Video-MME? Especially in long duration videos, why is this? And as the number of frames continues to increase, it gets worse in short, medium and long. Although other experiments show that LongVA performs well in VNIAH, demonstrating the ability of information retrieval, it does not show the advantage of this model in long video understanding.

(6) In the experimental results of Table 7, in 6 benchmarks, UniRes is better than AnyRes in 4 tasks, which does not prove that UniRes is a better visual encoding method. Why use UniRes instead of AnyRes?
</content>
</thread_object>

</thread>
<thread>
creation date:
1730750680908

thread_type:
official_review

<thread_object>
creator: reviewer_9tbs

<content>
summary:
Current LLMs struggle to understand very long videos because of the context length limitation. This paper addresses this issue by proposing a new method that extends the input context length of LLMs. The method includes three key ingredients: 1) continuous pretraining the LLM with a large context window on text-only dataset, 2) aligning the long-context LLM with image data, 3). tokenizing images (or videos) with the so called UniRes that divides images into grids and flatten the visual tokens in each grid. To evaluate the effectiveness of the method on long video understanding, they introduce a new benchmark called V-NIAH where the task is to retrieve one frame that is randomly inserted into a long video. They also evaluate the method on other video question answering benchmarks. On all the benchmarks, their model "LongVA" demonstrates state-of-the-art performance.

strengths:
1. The proposed method achieves the state of the art on multiple video understand benchmarks. 

2. The proposed method is simple and effective. The continuous pretraining on large context window only needs text data, while the multimodal alignment stage only requires image-text data. The uniRes encoding is straightforward extension of AnyRes to videos. Nevertheless, the model seems to generalize surprisingly well to videos. 

3. The Needle-in-a-Haystack test seems to be novel in the context of long video understanding. This new task could be an interesting testbed for long video understanding in the future.

weaknesses:
1. Despite the impressive empirical results, there are not much explanations. In particular, I am interested in understanding why the long-context pretraining on text-only data could transfer to long video understanding.

2. It is not clear on how the image-text alignment is done e.g., training data and losses. Most of the content in the Sec. 3.2 is about the UniRes which is relatively straightforward. 

3. The number of grids during training is limited to 49 (Line 205), which is effectively equivalent to 49 frames. Why does the model could generalize to handle more than 2K frames give that it is trained with only 49 frame inputs?

4. The results in Table 4 show that LongVA outperforms the methods that are much larger and trained on videos e.g., LLaVA-NeXT-Video-34B. Meanwhile, VILA-1.5 achieves the best results with only 8 frames. Those results seem to imply that the input frames and video-specific training are not that important for video understanding. I found this phenomenon counter intuitive. But this might be caused by the different LLM backbones. One interesting experiment I would suggest is to ablate context length and video specific training with the same LLM. 

5. Temporal understanding is arguably an important aspect in video understanding. Being able to process a large number of video frames does not indicate a strong capability in temporal understanding. Most of the experiments shown in the paper can be solved by "selecting" one frame or very few frames. The model might not be able to understand fine-grained motions/actions, for example Something-Something V2 dataset.

questions:
What would be the performance of VILA-1.5 on the V-NIAH benchmark?
</content>
</thread_object>

</thread>
<thread>
creation date:
1732212710698

thread_type:
official_comment

<thread_object>
creator: authors

<content>
title:
General Response

comment:
We sincerely thank all reviewers for their comments and are encouraged by the shared positive feedback  in the following areas:

1. Introduction of a novel approach that extends the context length of language models for long video understanding, reframing the problem from traditional token reduction.
2. Discovery and analysis of the long context transfer phenomenon, highlighting the transferability of extended context capabilities from language to vision.
3. Development of the V-NIAH benchmark as a valuable tool for assessing long visual context handling in LMMs.
4. Comprehensive experimental evaluations demonstrating state-of-the-art performance on Video-MME and MLVU benchmarks.


We mainly add the following experimental results:

For reviewers’ common concern over AnyRes v.s. UniRes, we further evaluate LongVA(AnyRes) and report them below:

| Model           | Frames | Short  | Medium | Long  | Total   |
|------------------|--------|--------|--------|-------|---------|
| LongVA (AnyRes)  | 32     | 58.7   | 46.9   | 44.8  | 50.111  |
| LongVA (UniRes)  | 32     | 61.1   | 48.8   | 45.4  | 51.8    |

The results demonstrate that UniRes enables better video understanding capabilities in a setup where there is no video during training.

To ablate the impact of the backbone language model’s context length on video understanding, we further run LLaVA-Next-Qwen2 on V-NIAH. 
The LLaVA-Next-Qwen2's heatmap result is here: https://postimg.cc/v4dSXYYd. 
Please refer to Table 5 for the relation between  LLaVA-Next-Qwen2, LongVA, and LongVA(AnyRes). LongVA, and LongVA(AnyRes)'s results are in Paper Figture 4. All three models are trained with exactly the same data and hyperparameters during image-text alignment. 

The results show that increasing the text context length alone has the biggest improvement of visual context length on V-NIAH. Changing AnyRes to UniRes further increases the visual length extrapolation capabilities.

We will address other concerns and questions individually. We will update our manuscript based on the reviewers’ comments.
</content>
</thread_object>

</thread>


Metareview:
metareview:
This paper proposes an alternate approach to long-video understanding with VLMs: The authors train the language model with longer textual context lengths, and show how this transfers to the visual modality: ie a language model trained with larger context lengths can perform better on video understanding tasks, even without training on long videos. In addition, the authors also proposed a Visual Needle-in-a-Haystack (V-NIAH) benchmark, and an extension of the Qwen2 VLM that achieves better results on the Video-MME and MLVU datasets.

Reviewers opinions were quite divergent on the paper: Some reviewers appreciated the overall approach which suggests that we don't need video-specific training data, but can simply train with long-context text tasks to perform well for long-context video tasks. However, other reviewers were concerned about the lack of technical novelty, and the fact that although the authors show impressive (and perhaps counterintutive) results on Video-MME and MLVU, there is not much analysis of why the proposed "long context transfer" works. For example, what the failure cases are, in what domains of problems can we expect textual long-context to transfer, and for what domains we actually need video-specifc training data? Although the authors compare to the state-of-the-art in Tables 4 through 7, there is not much analysis of why the proposed method works.

Furthermore, although the authors perform Needle-in-a-haystack experiments, there are also a number of recent works in this area which were not discussed at all [A, B, C, D]. In addition to Video-MME, there are also more recent, real-world video understanding datasets which could also provide more understanding about the "long context transfer" phenomenon. Examples include LVBench (longer, real-world videos) and Egoschema (the videos are not as long, but in contrast to needle-in-a-haystack tasks, require looking at the whole video).

Therefore, after some deliberation, the AC and Senior AC believe that this paper could do with further iteration and encourage the authors to revise the paper in light of reviewers' comments and resubmit to another venue.


[A] Zhao et al. Needle in a video haystack: A scalable synthetic framework for benchmarking video mllms

[B] T Yuan et al. Lv-eval: A balanced long-context benchmark with 5 length levels up to 256k

[C] T Wu et al. Visual haystacks: Answering harder questions about sets of images.

[D] Gemini Team. Gemini 1.5: Unlocking multimodal understanding across millions of tokens of context.