Exploring the Design Space of Visual Context Representation in Video MLLMs

Authors: Yifan Du, Yuqi Huo, Kun Zhou, Zijia Zhao, Haoyu Lu, Han Huang, Xin Zhao, Bingning Wang, weipeng chen, Ji-Rong Wen
Primary area: applications to computer vision, audio, language, and other modalities
Keywords: Video Multimodal Large Language Model, Scaling Law

Abstract:
Video Multimodal Large Language Models~(MLLMs) have shown remarkable capability of understanding the video semantics on various downstream tasks. Despite the advancements, there is still a lack of systematic research on visual context representation, which refers to the scheme to select frames from a video and further select the tokens from a frame. In this paper, we explore the design space for visual context representation, and aim to improve the performance of video MLLMs by finding more effective representation schemes. Firstly, we formulate the task of visual context representation as a constrained optimization problem, and model the language modeling loss as a function of the number of frames and the number of embeddings (or tokens) per frame, given the maximum visual context window size. Then, we explore the scaling effects in frame selection and token selection respectively, and fit the corresponding function curve by conducting extensive empirical experiments. We examine the effectiveness of typical selection strategies and present empirical findings to determine the two factors. Furthermore, we study the joint effect of frame selection and token selection, and derive the optimal formula for determining the two factors. We demonstrate that the derived optimal settings show alignment with the best-performed results of empirical experiments. The data and code are available at: https://github.com/RUCAIBox/Opt-Visor.

Decision: accept (poster)
Reviewer scores: 6, 8, 6

Reviews and discussion:
<thread>
creation date:
1730578262940

thread_type:
official_review

<thread_object>
creator: reviewer_wrue

<content>
summary:
This paper analyses the input design space of Video Multimodal Large Language Models (MLLMs) by scaling two components: the number of frames and the number of tokens per frame.
Given the quadratic computational complexity of the attention mechanism in the number of input tokens, MLLMs are limited in the size of their input. This paper studies the impact of varying the number of frames and the number of tokens per frame, as well as the sampling and compression strategies to obtain the final tokens.

strengths:
The paper offers a valuable analysis of visual context representation design in video MLLMs. The approach is straightforward, with clearly explained experimental settings and well-presented results, and the conclusions are well-supported by experiments.

It is interesting to see the contribution of mean pooling versus uniform sampling, both for selecting frames and selecting tokens.

The authors promise to release code and data for reproducing their results.

weaknesses:
Comparison to SOTA results on the benchmarks.
It is understood that the goal of this paper is not to obtain SOTA results on the evaluated benchmarks, but rather to perform an analysis on the design space of visual context representation in video MLLMs. Nevertheless, it would be informative to the reader to have the information of how far the experimental recipes presented in this paper are from the SOTA.

questions:
Figure 2b and 3b: In addition to the performance curves on each benchmark, it would be valuable to also plot the average over all of them.

Section 3: At the other end of the scaling spectrum, it would be interesting to also report the performance without any visual token at all (M=0) to see what would be the comparative performance of a "blind" MLLM.

Minor typo:
L325: "increasing the frames"
</content>
</thread_object>

</thread>
<thread>
creation date:
1730728454025

thread_type:
official_review

<thread_object>
creator: reviewer_4wan

<content>
summary:
Given a fixed Video-LLM window size, this paper examines the optimal selection of the number of video frames, the number of video tokens, and the compression/sampling methods used for frames and embeddings. The authors include experiments on both loss and benchmark evaluations.

strengths:
1. The paper is clearly written and follows a logical structure.  

2. The paper includes extensive experiments and offers a thorough analysis. Notably, the experiments that consider fixed frames, fixed embeddings per frame, and the trade-off between embeddings and frames are particularly useful for following research.

weaknesses:
1. Is the design space for embedding and frame operations too narrow? The findings within this design space may be biased towards different LLM backbones and vision encoders. We suggest that the authors conduct experiments across a variety of LLM backbones and vision encoders to ascertain the generalizability of their findings. Or at least one more different LLM backbone and vision encoders.

2. In the experiments depicted in Figure 2 and Tables 1 & 2, the authors utilize the same T = 32 (L168) for all experiments, which results in video tokens of varying lengths within the same LLM context window size. It is generally assumed that longer tokens within a pretrained LLM context window size lead to better performance. How can it be determined whether longer visual embeddings after the vision encoder or longer visual tokens in the LLM contribute more significantly to performance improvements? 

3. Since this paper aims to study the scaling effect, conducting a sufficient number of experiments with training models is crucial to mitigate the impact of randomness bias. The authors should clarify the number of models trained for each data point presented in experiments.

questions:
My major concerns lie in the soundness of the experiments. Please find more details in the 'Weaknesses' section
</content>
</thread_object>

</thread>
<thread>
creation date:
1730867748380

thread_type:
official_review

<thread_object>
creator: reviewer_uyut

<content>
summary:
This paper mainly discusses the video frame and visual token per frame selection strategies for video MLLMs. Based on LLaVA-like architecture, this paper conducts several experiments, e.g., varying frame number while fixing visual token per frame and altering visual token per frame while fixing total frame number, to fit the loss curves. Finally, it proposes a fitting function of the two factors and draw an optimal combination conclusion.

strengths:
1. The motivation and idea of this paper is straightforward.
2. The writing and structure organization of this paper is clear and easy to follow.
3. The experimental results are clearly clarified, and the statistics are persuasive.

weaknesses:
1. My main concern about this paper is the generalization of the final conclusion. As we all know, video MLLM is very large and complex. This paper uses only one model structure (SigLip + Qwen2-7B) to conduct experiments on several benchmarks. Is the final best setting ($T_{opt} \approx 118$ and $M_{opt} \approx 51$) generalizable to other models and benchmarks? If not, must we conduct several experiments to fit a new curve to find the best factor combination?
2. Based on the above question, if this paper is just an experiment report, readers cannot obtain enough inspiration from this paper.

questions:
Besides the question in Weakness, how can you determine the function formation of Eqs. 3, 4, 5, and 6? Just observe from figures 2a, 3a, and 4?
</content>
</thread_object>

</thread>
<thread>
creation date:
1732288193137

thread_type:
official_comment

<thread_object>
creator: authors

<content>
title:
Revision of our Manuscript

comment:
We sincerely thank the reviewers for their thoughtful feedback and constructive suggestions, which greatly help us improve our work. In this work, we aim to explore an important but under-explored problem for video-MLLM, i.e., the design of visual context encoding part, and conduct comprehensive experiments to formulate useful rules to determine its design. We follow the reviewer's suggestion and update the paper, and the main revision contents are listed as follows:

- Figure 2 (b) and Figure 3 (b): We add the averaged performance in them.
- Appendix B: We add the experiment on other LLM and vision encoder backbones.
- Section 4 and Appendix C: We generalize our conclusion to various video context lengths.
- Appendix D: We add the expriments of scaling visual tokens in the vision encoder and compare it with scaling visual tokens input to LLM.
- Appendix E: We add the experiments under different seeds and the predictive validation of the scaling law.
- Appendix F: We scale the video data and compare our model with several SOTA video MLLMs.
- Appendix G: We add the experiments without any frames and visual embeddings as a reference.

We hope that the above works are able to address the reviewers' concerns, and are also very willing to have more discussion with all reviewers.
</content>
</thread_object>

</thread>
<thread>
creation date:
1732667470712

thread_type:
official_comment

<thread_object>
creator: authors

<content>
comment:
Dear Reviewers,

Thank you for your time and effort in reviewing our manuscript. We sincerely appreciate the thoughtful feedback you provided, which has been invaluable in improving our work.

In our rebuttal, we have made a concerted effort to address your concerns through detailed responses, supplementary experiments, and revisions to the manuscript. As the discussion phase draws to a close, we would be very grateful if you could take a moment to review our response.

If you find that our revisions have adequately addressed your concerns, we kindly ask if you might consider reevaluating your initial assessment and adjusting the score accordingly. Should there be any remaining issues or points requiring further clarification, we are more than happy to address them promptly.

Thank you again for your valuable time and insights.

Best regards,

Authors
</content>
</thread_object>

</thread>
<thread>
creation date:
1732729351428

thread_type:
official_comment

<thread_object>
creator: area_chair_l4cz

<content>
title:
Please take a look at the authors' rebuttal and start a discusssion if needed

comment:
Dear Reviewers,

Thanks for your contributions in reviewing this paper.

As the author-reviewer discussion deadline is approaching, please could you take a look at the authors' rebuttal (if not yet) and see if it addressed your concerns or if you have any further questions. Please feel free to start a discussion.

Thanks,

AC
</content>
</thread_object>

</thread>


Metareview:
metareview:
In this paper, the authors presented a study about the design space of visual context representation in video multimodal large language models (MLLMs). Specifically, two components were considered for the analysis: the number of frames and the number of embeddings/tokens per frame. By exploring the scaling effects within these aspects, empirical findings were presented with an optimal formula for determining the two components. The main strengths of this paper are:
- The paper is well motivated and the idea is interesting, offering a valuable perspective for visual context representation design in video MLLMs. The idea is simple and straightforward, with a clear presentation.
-  Extensive experimental analysis was presented, showing several interesting and useful findings, e.g. the trade-off between embeddings and frames, the contribution of mean pooling vs. uniform sampling, etc. The analysis presented in this paper could be useful for follow-up research in the community.
- The paper is well-written, well-organised, and easy to follow.

The main weaknesses of the paper were mainly around the generalisation ability, the impact of the corresponding derived conclusions, unclear analysis of the contributions from different components, and comparison with state-of-the-art.

After the authors' rebuttal, followed by a few rounds of discussions, the above major concerns were addressed, and the reviewers were satisfied with the supplemented evidence. Two reviewers decided to raise their ratings, and the paper ended up with 1 Accept and 2 borderline Accept. Considering the consistent positive ratings and the potential contributions this paper could bring to the community, the AC is happy to recommend an Accept.
But the authors are strongly recommended to incorporate the additional experiments and discussions into their final version.