Exploring the Design Space of Visual Context Representation in Video MLLMs
Authors: Yifan Du
Yuqi Huo
Kun Zhou
Zijia Zhao
Haoyu Lu
Han Huang
Xin Zhao
Bingning Wang
weipeng chen
Ji-Rong Wen
Venue: ICLR 2025 Poster (2025)
Keywords: Video Multimodal Large Language Model
Scaling Law
Primary area: applications to computer vision, audio, language, and other modalities

Abstract: Video Multimodal Large Language Models~(MLLMs) have shown remarkable capability of understanding the video semantics on various downstream tasks. Despite the advancements, there is still a lack of systematic research on visual context representation, which refers to the scheme to select frames from a video and further select the tokens from a frame. In this paper, we explore the design space for visual context representation, and aim to improve the performance of video MLLMs by finding more effective representation schemes. Firstly, we formulate the task of visual context representation as a constrained optimization problem, and model the language modeling loss as a function of the number of frames and the number of embeddings (or tokens) per frame, given the maximum visual context window size. Then, we explore the scaling effects in frame selection and token selection respectively, and fit the corresponding function curve by conducting extensive empirical experiments. We examine the effectiveness of typical selection strategies and present empirical findings to determine the two factors. Furthermore, we study the joint effect of frame selection and token selection, and derive the optimal formula for determining the two factors. We demonstrate that the derived optimal settings show alignment with the best-performed results of empirical experiments. The data and code are available at: https://github.com/RUCAIBox/Opt-Visor.

Published as a conference paper at ICLR 2025
EXPLORING THE DESIGN SPACE OF VISUAL CONTEXT REPRESENTATION IN VIDEO MLLMS
Yifan Dut; Yuqi Huo"", Kun Zhou Zijia Zhao', Haoyu Lu', Han Huang Wayne Xin Zhao'; Bingning Wang² Weipeng Chen', Ji-Rong Wen'
'Gaoling School of Artificial Intelligence, Renmin University of China Baachhun Inc.
School of Information, Renmin University of China "Institute of Automation, Chinese Academy of Sciences {yi fandu1l 9999, batmanfly) gma i com (ariellebatle chhaicihan-inne com
ABSTRACT
Video Multimodal Large Language Models (MLLMs) have shown remarkable capability of understanding the video semantics on various downstream tasks. Despite the advancements, there is still a lack of systematic research on visual context representation, which refers to the scheme to select frames from a video and further select the tokens from frame. In this paper, explore the de- sign space for visual context representation, and aim to improve the performance of video MLLMs by finding more effective representation schemes. Firstly, we formulate the task of visual context representation as a constrained optimization problem, and model the language modeling loss function of the number of frames and the number of embeddings (or tokens) per frame, given the maximum visual context window size. Then, explore the scaling effects in frame se- lection and token selection respectively, and fit the corresponding function curve by conducting extensive empirical experiments. We examine the effectiveness of typical selection strategies and present empirical findings to determine the two factors. Furthermore, we study the joint effect of frame selection and to- ken selection, and derive the optimal formula for determining the two factors. We demonstrate that the derived optimal settings show alignment with the best- performed results of empirical experiments. The data and code are available at: wtt.:ttuw.coo///CAAOO///OOOO////
INTRODUCTION
Recent advancements in video Multimodal Large Language Models (video MLLMs) have shown the great potential in extending L.MMs (Zhao et al. 2023) to process video data (Lin et al., 2023). Typically, a video MILM is developed based on a pre-trained LLM,, and an image encoder will be attached the LLM via a modality projector, which links the textual and visual semantic spaces. In this way, we can prompt the video MLLM with textual instruction and visual embeddings, to generate the natural language response for fulfilling the video-based task, e.g., video question an- swering (Xu et al. 2017) and video captioning (Caba Heilbron et al. 2015). Despite the success, it still challenging for existing video MLLMs to handle complex or long videos, due the limited model capacities.
To develop effective video MILMMs, previous research work mainly focuses on two aspects, either improving the model architecture (Wang et al. 2024b) enhancing the model training (Zhang et al.) 2024b, Liu et al. (024b)) However, another important aspect has been missing in the related litera- ture, i.e., visual context representation. In this work, visual context refers to the visual embeddings in the prompt of video MLLMs. Unlike text and images, it is not very straightforward to represent video. In existing approaches, widely used way to sample number of frames from video
"quual contribution.
"orresponning author.

---

Published as a conference paper at ICLR 2025
(frame selection) and then further sample generate number of embeddings for each selected frame (embedding selection). However, it is unclear how each factor affects the performance of video MLLMs, and how both factors jointly contribute to the performance improvement within the limited context length of the underlying L.LM.
Considering this issue, in this paper, take the initiative to explore the design space for visual context representttinn,, and derive more effective representation schemes to improve the performance of video MLLMs. Specifically, we firstly formulate the studied task constrained optimization problem: given the maximum visual context window size, we model the language modeling loss as function of the number of frames and the number of embeddings (or tokens) per frame. Such a formulation is useful to help understand the competitive relationships between frame selection and embedding selection. Subsequently, we conduct extensive empirical experiments to explore the scaling effects in frame and embedding selection respectively, and fit the corresponding function to describe the performance trend. Our findings show that: (1) overall increasing the number of visual embeddings (either tokens or frames) would enhance the performance, while scaling the frames can lead to consistently improved performance; (2) the compression-basee
method can effectively preserve more semantic information with fewer visual embeddings. Furthermore, study the joint effect of the two factors and propose the method to find the optimal allocation given the limited context length, which is further supported by empirical experiments.
The major contributions of our work are as follows:
To the best of our knowledge, this the first work systematically study the design of visual context, which is an important yet under-explored problem for developing capable video MLLMs. We provide both theoretical formulations and empirical findings to approach this problem.
We study the scaling effects of model performance w.r.f. the number of selected frames and the number of selected embeddings per frame respectively. We fit the corresponding function curve, and compare different strategies (i.e., sampling- and compression- based methods) for both factors.
We explore the trade-off relationships for frame and embedding selection, and suggest the optimal formula for determining the two factors. We demonstrate that the derived optimal settings show alignment with the best-performed results of empirical experiments.
PRELIMINARY
In this section, we introduce the background for building the base model in our work.
Model Architecture. Following existing works (Liu et al. 2024a Zhang et al. 2024c), we adopt the LLaVA-like model architecture, consists of a visual encoder, an LLM, and a projector that maps the visual embeddings to the semantic space of the LLM. Formally, given vidco with T frames {Te)== each frame is encoded by the image encoder fo to obtain M visual embeddings [ where v E Rd denotes the i-th visual embedding in the f-th frame, and d, is the dimensionality of the visual embedding. Then the projector fo projects these visual embeddings into the semantic space of the LLM,, producing hf E Rd. These visual embeddings are concatenated with the embeddings of a textual prompt (e; J where e, E IRd is the embedding of the j-th token in the prompt. The concatenated sequence fed as input the LLM fo to generate the output:
fo h .... hir..... .... bf b : e1, eNl)) (o
V1
yK
During training, we optimize the parameters (0, vh 0} by minimizing the next-token prediction loss.
Training Data. Based on existing instruction datasets, we mix several widely used image instruc- tion and video instruction sets to construct new instruction dataset. For the image instruction set, we adopt Cauldron (Laurençon et al.) 2024b). which is large image instruction set based on 50 vision-language datasets. For the video instruction set, we collect the instructions from AideohhatGPT-100S (Muhammad Maaz & Khan 2023), ShareGPT4Vides (Chen al.) (024), ShareGPTVideo (Zhang et al.) (2024b), VIM (Du et al. 2024), as well as some instruction data from VideoChat2 (Li et al. 2024b). The statistics of each instruction are listed in Table
2

---

Published as a conference paper at ICLR 2025
Model Architecture
Embedding neveell Operation
Frame- revel Operation
Resporse
Compressing
Sompling- bodedd method
Compressing- boseed method
(4,4,9
size (4,44)
Large Longuage Model
size (4,4)
Size (22)
Operation
Kernel Bize
(:3,)
(2.4.4)
Sampling
#ize (2,4.4)
Embedding level Oparahon
Sempling- bosed method
size (4.4)
Size - (2,2)
size
(32)
Figure I: Overview of the LLaVA -ike architecture for rideo-MLLM, and our used frame-leve and embedding- operations for adjusting the visual context window size.
Implementation Details. We adopt SigLIP (Zhi et al., 2023) the image encoder, Qwen2- 7B (Yang et al.] 2024) as the base LLM, and a two-layer MLP as the projector. We train all the models with the training data listed in Table for epoch. We have tried include pre-training stage before the visual instruction tuning, using the 558K pre-training data and only updated the parameters the MLP following LLaVA (Liu et al. 2024a), but found no obvious difference. All the experiments are conducted on 32 Nvidia H800, with the detailed hyperparameters listed in Table 7
Evaluation Setup. To quantitatively assess the scaling effect of visual context in video MLLMs, we consider the following two metrics for evaluation:
Language modeling loss. is a continuous measure of model performance in predicting the next token, used to estimate the parameters of the scaling law function. Following Chinchilla (Hoffmann al., 2022), each model is trained for one epoch to ensure that training samples are unseen when calculating the loss for evaluation. We conduct experiments with five random seeds under a widely used configuration to guarantee the robustness of our results, with detailed results in Appendix E.
Zero-shot Accuracy. The zero-shot accuracy can reflect the performance of the model in the real- world application. We select several long video understanding benchmarks for evaluation, including Event-Bendh (Du et al. (2044) (only with the challenging episodic reasoning task). VNBench (Zhao ct al. 2024). MLVU (Zhou et al., (2024), and VidcoMME (Fu et al., 2024). All the questions in these accuracy as the evaluation metric.
benchmarks multiple-choice and
3 SCALING LAW OF VISUAL CONTEXT
3.1 PROBLEM FORMULATION
As introduced in Section 2 existing video LLMs typically follow the vision-language model archi- tecture (Liu et al. 2024a Zhang et al. 2024c), which represents a video into multiple representative frames. Further, each Trame will be encoded into a number of visual tokens or embeddings. The aggregation of the visual embeddings from all selected frames is referred to as visual context in this work. To set the visual context, is essential to determine two aspects when the base architecture fixed: (1) how to select the frames from video (frame selection), and (2) how to select the visual embeddings from an input frame (embedding selection). Since the base architecture is developed existing LLM. the length of visual context is naturally limited by its context length, i.e., the maximum length of input tokens. The two aspects would be competitive in input length allocation: the more the selected frames, the fewer the visual embeddings per selected frame, and vice versa.
In this work, we study the optimal allocation relationship the visual context for given video. Formally, model the language modeling loss L(T, M) function of the number of frames T and the number of embeddings (or tokens) per frame M. Given the maximum visual context window size I., the number of frames T and visual embeddings per frame M should satisfy the constraint:
3

---

Published as a conference paper at ICLR 2025
T M L, we aim to find the optimal solution in minimizing C(T,M) under this constraint:
T Tog ( L., Mapt
argmin ((T,
T,M s.L. T&M&L
where Top and (L) represent the optimal allocation strategy for the frame and visual em- bedding, respectively, with the input limit L. To approach it, in the following, we will explore the scaling effect of frame and embedding selection in Section 3.2 and Section 3.3 respectively.
3.2 SCALING EFFECT OF THE VISUAL EMBEDDINGS
We first analyze the scaling effect of visual embeddings in frame for a fixed number frames. Specifically, we utilize two methods to select (or generate) the visual embeddings in a frame: the sampling- and compression- based method.
3.2.1 SAMPLING-BASEE METHOD
Experimental Setup. In this part, each image is first converted to 27 27 embeddings by the image encoder, then we vary the number of sampled visual embeddings. Specifically, we uniformly sample {12, 22, 32, 42, 52, 6², 72, _9, ]42] embeddings from the 27 27 embeddings. as illustrated in Figure Other sampling methods like block sampling et. 2023ch will be explored in future work. We set T 32 the constant, and uniformly sample frames from each video and keep all other factors the same to train 9 video MLLMs with this setup of visual embeddings.
Fitting Function.
We propose the following function to fit the scaling law of visual embeddings:
Mu M
((M) = LM +
(3)
We fit the language modeling loss with respect to the number of visual embeddings M using the scipy curvefit function, obtaining T. 0.48, Mo 1.16 10-5 our 0.1, with R² 0.927 indicating a good fit. The fit curve in Figure 2a shows that L(M) decreases with increasing M, following power-law like trend. We calculate the squared error between the actual loss and predicted loss, obtaining value of 0.0001, which indicates very low fitting error.
0.800
compresiae-tatee
Sampinggeant based mechod
Accuracy
40
Curve of sam sampling based method
0.750
30
Event-Bench Bench
LiM
0.725
MLVU
0,700
20
VideoMME wo subs
LongVideo-Bende
0.675
Average
25
50
75
100
125
150
175
200
Number tokens (M)
0
25
300
75
100
122
130
175
200
(b) The relationship between the number of em- beddings per frame and the benchmark accu- racy.
(a) The scaling curve of visual embeddings.
Figure 2: The scaling law of visual embeddings, reflected by the language modeling loss and the zero-shot accuracy on video understanding benchmarks.
Benchmark Performance Analysis. Table [ shows the results of scaling visual embeddings on the evaluation benchmarks, Overall, the model performance improves as the number of visual em- beddings increases, especially when it varies from 1 to 4. The improvement becomes with increasingly more visual embeddings. However, when exceeds some threshold, the per- formance starts to decrease. For example, using 196 tokens is worse than using 49 tokens. An
marginal
4

---

Published as a conference paper at ICLR 2025
Table 1: Results of sampling-bes bbate method under different number of visual tokens per frame.
Embed./ Frames
Event- Bench
LongVides Bench
VideoMME wo/w-subs
# Frames
VNBench
MLVU
Avg.
32
21.67
14.96
48.67
40.56
42.48/52.04
36.73
23.33
21.11
50.72
44.88
45.44/54.81
40.05
32
25.33
24.15
52.42
43.82
45.85/54.67
41.04
32
16
23.00
27.85
52.29
46.70
49.22/57.11
42.20
32
25
22.33
26.22
55.18
47.08
50.44/58.22
43.25
32
36
20.33
29.48
56.05
47.76
50.07/58.04
43.62
23.33
28.52
55.41
48.90
51.30/59.74
44.53
32
81
24.00
29.41
55.12
48.90
50.04/57.30
44.13
32
196
26.00
27.78
56.53
47.69
48.63/55.59
43.70
Table 2: Results of compression-bassee method under different number of visual tokens per frame.
Embed./ Frames
Event- Bench
Long Vidco- Bench
VideoMME wo/w-subs
# Frames
VNBench
MLVU
Avg.
32
1
18.67
18.4
49.45
39.88
41.33/49.15
36.15
32
4
24.33
27.04
52.77
43.44
46.11/55
41.60
22.33
27.41
53.83
45.11
47.78/55.44
41.98
20.33
28.96
55.04
46.32
49.85/58.15
43.11
32
25
23.00
28.67
54.21
46.02
49.85/58.
43.31
32
36
27.33
30.00
53.73
48.45
50.33/58.74
44.76
32
49
21.33
29.93
54.84
47.16
49.96/58.37
43.60
23.33
27.33
57.59
48.60
52.00/59.37
44.70
32
196
29.00
31.56
56.81
52.24
53.56/59.48
47.11
interesting finding is that the language modeling loss with 196 embeddings is significantly smaller than that of the model trained with 49 embeddings, as shown in Figure [2 which indicates that model loss might not directly reflect the performance downstream tasks.
3.2.2 COMPRESSION-BASE METHOD
Experimental Setup. We utilize the MeanPooling Yao 2024a) strategy for compressing the visual embeddings, which has been widely used in visual information processing. Another advantage is that does not introduce extra parameters, avoiding the influence new factors in the experiments. We apply MeanPooling with different kernel sizes the feature map produced by the image encoder and obtain the condensed representation of the image. Specifically, each image encoded into 27 x 27 visual embeddings, on which we apply p x mean pooling with stride (p (2,3, 4, 5, 6, 7, 9, 14, 27)), obtaining (12 22, 32, 52,52, 6,,62, 72, 9 142} condensed embeddings per image. All the other factors are kept the same for fair comparison.
Fitting Function. We use Equation to fit the scaling law and obtain LM = 0.57, Mo = 0.01, aM 0.39, with R² 0.987. The mean squared error between the predicted loss and the actual loss is 5.32 x 10-5 indicating good fit. Compared to the parameters of sampling-based based method in Section S..2. where am = 0.1, the am of the compression-bes larger, implying that increasing the number of embeddings using the compression based method will result in faster loss decrease, as shown in Figure 2a Additionally, the compression-be based method consistently yields lower loss than the sampling-basee method for the same number of visual embeddings. This is because the compression- based method does not directly discard visual embed- dings but instead aggregates information from them, which preserves
method is significantly
spatial information.
Benchmark Performance Analysis. For the benchmark evaluation in Table 2 the overall accu- racy consistently increases as the number of visual embeddings increases. This finding is signifi- cantly different from that in Table [ This result highlights the advantage of the compression based method, which can preserve more information than the sampling-basee based method. The conclusion drawn from the benchmark evaluation aligns with that concluded from the language modeling loss.
5

---

Published as a conference paper at ICLR 2025
Table Experimental results under different number of frames, with the sampling- based method.
# Embed./ Frames
Event- Bench
Long Bench
VidcoMME wo/w-subs
Frames
VNBench
MLVU
Avg.
8
49
21.67
15.70
46.30
44.73
44.85/S2.74
37.67
16
49
22.67
23.33
52.53
46.78
49.74/57.59
42.11
32
49
21.33
29.93
54.84
47.16
49.96/58.37
43.60
48
49
22.67
34.15
56.22
48.75
52.81/59.11
45.62
64
49
25.33
32.59
57.23
47.08
52.59/58.93
45.63
96
49
26.67
37.26
60.97
48.60
53.26/60.85
47.94
128
49
25.67
39.70
61.44
51.40
56.11/61.63
49.33
Take-away Findings
Increasing the number of visual embeddings can significantly enhance the performance, with the sampling-basee method achieves the peak at 49 tokens while the compression- based method does not saturate even with 196 tokens.
When the visual context window size is limited, the compression- based method can effec- tively preserve more visual information with fewer visual embeddings.
3.3
SCALING EFFECT OF THE SELECTED FRAMES
Next, continue explore the scaling effect of the selected frames by varying its number T while fixing the number of embeddings per frame M. We also consider utilizing sampling-bas baasin and compression-baset methods.
3.3.1 SAMPLING-BASEE METHOD
Experimental Setup. existing works, has become widely used practice to sample frames uniformly from the original video to accommodate for the context length of LLM. Based on this method, we sample different numbers of frames from the video to explore the scaling effect, by varying T in {1, 16, 32, 48, 64, 96, 128). In Section we further increase T to 162 to explore the limit of scaling frames. The maximum context length allowed by the computation memory is 8K, corresponding
128 frames and 49 visual embeddings per frame. As result, adopt 4 MeanPooling to keep 49 embeddings per frame and train 8 video MILMs by varying the number of frames from to 128.
Fitting Function. Similarly, we use the following function fit the scaling law of frames:
To
a7
C(T)= LT +
We fit the losses with the number of frames T and obtain L1 0. 5.37 10 0.04, with R² 0.892. The fitted curve in Figure [a shows that L(T) decreases with increasing following plwer-law like trend. We calculate the mean squared eтror between the actual loss and predicted loss, obtaining a value of 0.0001, which indicates very low fitting error.
Benchmark Performance Analysis. The results in Table and Figure 3b show that as the num- ber of frames increases, the model consistently improves on all benchmarks, with clear saturation point, even at 128 frames, which exceeds the maxinum frame count of most video MLLMs. Among all the benchmarks, VNBench shows that the most pronounced improvements (from 15.70 to 39.70), suggesting that the Needle- In-the- mee-the-Hayee Haystack-Seena -Sarchhh (NIAH) task benefits most from extended tem- poral context. However, the Bench shows significant improvement beyond 64 frames, and a detailed inspection reveals that all questions Event- Benth focus on episodic reasoning, task that cannot be effectively learned by video MI.LMs simply by increasing the number of frames (Li

---

Published as a conference paper at ICLR 2025
60
Sempling-hesed
0.72
Guroe sumplingeaedd methed
Cempresiion-baeed mathed
Curve compresionn basod method
0.70
Accuracy
0.68
30
VNBench
MIVU
LongVideo-Berdd
VideoMME we- subs
Average
20
40
100
120
(b) The relationship between the number of frames and the benchmark accuracy.
(a) The scaling curve of frames.
Figure 3:
The scaling law of frames, reflected by the language modeling loss and the zero-shot accuracy on video understanding benchmarks.
Table Model performance with the compression-based method under different numbers of frames. For the model trained with 128 frames, since Taax T 128 in this setting, the temporal pooling kernel size is
128 =
1, resulting in the same outcome as the sampling-bass based method.
128
# Embed./
Event- Bench
Long Video Bench
VideoMME wo/w-subs
Frames
VNBench
MLVU
Avg.
Frames
8
49
25.67
20.89
53.04
46.32
50.48/57.41
42.30
16
49
27.00
27.04
55.77
48.07
50.44/58.30
44.44
32
49
25.33
29.78
59.37
48.52
53,81/60.93
46.29
48
49
24.33
37.41
59.31
47.61
52,07/59.81
46.76
64
49
29.00
36.30
61.03
47.61
53.70/60.56
48.03
128
49
25.67
39.70
61.44
51.40
56.11/61.63
49,33
al. 2023d). Overall, compared to scaling the visual embeddings per frame (Figure 2b), increasing the frames is more beneficial for improving the model performance.
Performance Compensating for Compressing Visual Embeddings. Another interesting finding that the performance degradation caused by compressing visual embeddings can be compensated by increasing the number frames. Specifically, Table Table[sshow shows that reducing the number visual embeddings per frame from 196 to 49 leads to a performance drop across all benchmarks. However, we simultaneously increase the number frames to 128, the accuracy returns, even surpassing the model with 196 embeddings (comparing the last rows of Table 2 and Table 3 both setups use total 6272 visual embeddings, but one utilizes 32 frames with 196 embeddings per frame, while the other employs 128 frames with 49 embeddings per frame). These results suggest that when constrained by the visual context length, we can increase the number of frames while decreasing the embeddings per frame achieve better performance, as will be further demonstrated in Section 4
3.3.2 COMPRESSION COMPRESSION-BASET METHOD
Experimental Setup. Compressing frames along the temporal dimension has been widely discussed in the field of video representation learning but remains underexplored in video MLLMs (Cheng al. 2024]. Similar the compression strategy used in Section [3.2] we uti- lize MeanPooling here to reduce the number of frames input to the LLM. Specifically, we uniformly sample T Tmax frames from the video and encode them with image encoder. Then, we apply MeanPooling along the temporal dimension to compress the video into T frames, where the tem- poral pooling kernel size is determined by Trax and T: 1 Tam }. Due to the limitation of 128 and T {8, 16, 32, 48, 64, 128) to explore the scal-
computational memory, we set T T
If the original video duration T' T Tmxx uniformly sample T" frames from it; otherwise, uniformly sample Tims frames, which common practice for video MI..Ms..

---

Published as a conference paper at ICLR 2025
ing law, which is significantly larger than existing sfate-of-the-eeee video MLLMs that mostly 32 or 64 frames as imput (Li et al., 2024a [Cheng et al.] 2024). To ensure a fair comparison with the sampling-bases
method, we also reduce the number of visual embeddings per frame to 49. In practice, we utilize three-dimensional MeanPooling, instead of first performing spatial MeanPooling
followed by temporal MeanPooling, to avoid over-smoothing of the feature maps.
Fitting Function. Different from the previous experiments, a power-law like function can't san't fit the data points this part. Instead, we find that a simple linear function can well describe the function relationship, which is defined follows:
C(T) a +
(5)
We fit the losses with the number of frames T' and obtain 0.0002, b 0.651, with πt 0.807. The fit curve is shown in Figure 3a We calculate the mean squared error between the actual loss and predicted loss 1.753 10 which indicates very low fitting error.
Benchmark Performance Analysis. Comparing the curve of the sampling- based and the compression- methods, the latter consistently results lower loss. This phenomenon re- veals the temporal redundancy in video data, showing that temporal information can be effec- tively preserved even when compressed into fewer frames. The evaluation results on the bench- marks are shown in Table 4 Overall, increasing the number of frames consistently improves model performance. Furthermore, the compression-bast basedd method achieves higher accuracy than the sampling- -ased method for the same number of frames. This aligns with the phenomenon that the compression-basee method generally gives lower training loss as depicted in Figure 3a
Take-away Findings
Increasing the number of frames consistently improves the performance, even compensat- ing for the performance degradation caused by compressing visual embeddings per frame.
When the visual context window size is limited, the compression- based method can pre- serve more temporal information than sampling-basse method with fewer frames.
To validate the reliability of our scaling law, we perform predictive validation by holding out the last data point, refitting the scaling function using the remaining points, and then predicting the loss for the hold-out point. As shown in Table 14 in the B.2] the predicted losses align closely with the actual losses, with prediction errors below 0.01. This validates the reliability of our scaling law and ensures the robustness of our findings.
TRADE-OFF BETWEEN VISUAL EMBEDDINGS AND FRAMES
Section 3.2 and Section 3.3 have discussed the scaling effect of visual embeddings and frames separately. In this section, we explore the joint effect of the two factors, and study the problem: how to jointly determine the numbers of visual embeddings and and frames under the constrain of maximum input length of LLM or deployment resource?
Fitting Function of the Two Factors. Following Hoffmann et al. (002)). we fit the losses by considering the numbers of embeddings M and frames T as follows:
((M,T) = CM x M a + +Cr x T +
(6)
Specifically, we set the number of visual embeddings as {25, 81), and set the number of frames as {48, 64, 80, 96). train 2
models in total. To extend the data points, we also include the 17 models trained in Section 3 and finally obtain 25 models in total. We obtain Cи = 0.25, a = 0.26, Cr 0.13, 3 0.21, Lo 0.50, with R² 0.884. The fit curve along the axes of T and M is shown in Figure With the decreasing of both T and M, the loss C(M,T) will consistently increase, reaching the highest loss the data point T 32, M 4 in experiment. In contrast,
8

---

Published as a conference paper at ICLR 2025
6499
0 800
Maximum Point: (32,4)
(TM (32,1)
00775
Minimum Poini: (128,49)
0.725
(128.49)
700 9
Tangent along?
0.675
0370
0.650
00.62
Tangent along
140
Number visual tokens (M)
20
Number of frames (1)
8g 2000
40
20 40 5g
60
Figure 4: The scaling law of the visual embeddings and frames, We also display the maximum point and the minimum point in our experiments.
Table 5: Model performance under the visual context length of 6K.
Embed./ Frames
Total Embed.
Event- Bench
Long Video Bench
VideoMME wo/w-suhs
# Frames
Loss
VNBench
MLVU
Avg.
729
5832
0.78
18.33
16.30
50.98
43.44
46.22/53.67
38.
30
196
5880
0.71
28.67
31.11
54.97
48.90
53. 19/60/6..
46.17
72
81
5832
0.70
24.33
37.56
58.37
50.34
53.04/61.11
47.46
120
49
5880
0.68
29.67
38.44
59.06
49.81
55.15/61.67
48.97
5832
0.70
33.00
40.67
62.83
50.04
55.19/62.00
50.62
as the M and T increase, the loss gradually decreases, and T 128, M 49 reaches the lowest loss. The computed gradient via the fitting function can help determine whether to increase M or T ac to achieve lower loss. For example, the derivatives at 7 32, MM are 7M -0.01, 37
ac
-0.004, indicating that descends faster along the M direction. Therefore, increasing the number of embeddings is promising to obtain lower loss, which aligns with our experiments.
Finding Optimal Setting. In practice, we are interested in the question that "given the visual context window L, what is the best choice of M and T that achieves the lowest loss T)" To answer this question, we utilize the Lagrange multiplier method to obtain the minimum point of Equation under the constraint M x T Tı
1-o
1,CT a.M
1- T
Topt
Mopt
BCT aC's
I
To verify the effectiveness of this principle, we set L as 6K and obtain (Topt Mops) s (118, 51) according Equation 7 For the experiment, frames simultaneously under a fixed visual context length, yielding five (T, .M) configurations: (8, 729), (30, 196), (72, 81), (120, 49), (162. 36). We then train five video MLLMs based on these configurations. The results in Table 5 show that the minimum loss is achieved with 120 frames and 49 visual embeddings per frame, which quite near to (Taps, Maa) (118, 51). As for the bench- mark evaluation, scaling the number of frames consistently improved overall performance without saturation, even with 162 frames and 36 visual embeddings. This phenomenon occurs because there remains gap between the mext-token-predicffi but the theoretical minimum point can serve as a strong starting point for subsequent optimization. To verify the generalization of our conclusion, we conduct experiments LLM backbones Appendix B We also derive the Tom and Mmg for MLLMs with different context lengths, ranging from 4K to I 126K, and verify the conclusion in Appendix
vary the number of visual embeddings and
loss and final performance on downstream tasks,
other vision encoders and

---

Published as a conference paper at ICLR 2025
5 RELATED WORK
Scaling Law. In the field of LLMs, scaling model parameters and training data have been shown to consistently enhance model capacity (Radford et al.] 2019: Brown 2020 Touvron et al.] (023). As a result, it is necessary to build a quantitative relationship between these scaling factors and the final performance, which is called the scaling law. Two representative scaling laws for LLM are proposed by Kaplan et al. (2020) and Hoffmann et al. (2022), where the former one models the relationship between the loss and model size, dataset size, and the amount of computation budget independently, and the follower one models the relationship between loss and model size, dataset size jointly. Inspired by these works, subsequent studies show that the scaling law also holds for
different model architectures (Clark et al.] 22022) training strategies (Goo et al.) 2023). and can be transferred to other domains like computer vision (Zhai et al. 2022
Dehghani et al. 2023 and
multi-modal (Radford et al. 202 Alayrac et al. 2022).
2023) with long video understanding ability is challenging task and remains underexplored. One line of work focuses on enabling long video training from the perspectives of training systems (Xue et al.) 2024), training strategies (Liu et al. 2024b), and model architectures (Wang et al. 2024b). For example, LongVILA (Xue et al. 2024) proposes the first Multi-Modal Sequence Parallelism system for long- context training and inference. Kangaroo (Liu al., 2024b) utilizes curriculum training pipeline to gradually increase the number of frames during training. LongLLaVA (Wang et al. (2024b) adapts the model architecture to a hybrid of Mamba (Gu & Dao 2023) and Transformer (Vaswani et al.) 2017) blocks. Another line of work aims to enable long video understanding during inference (Song) ct al. 2024 Zhang et al.) 2024a). For example, MovieChat (Song et al., (024) proposes a memory mechanism that includes rapidly updated short-term memory and compact long-tern term memory to store representations of long videos. VA (Zhang et al., (024a) extends the context window of an LLM and demonstrates that long video understanding can be directly transferred from MLLM without any video-specific training.
Video MILM. Training an MILMM Liang et al. (2024): Yao et al. (024b); Du et al.
Visual Embedding Compression. Most MLLMs consist of vision encoder, an LLM, and a vi- sual projector to project the image embeddings into the semantic space of the LLM. Early works like Flamingo Alaayrac et al. 2022) adopt resampler, which inserts a cross-attention module into the
LLM layer to extract visual features, and this is followed by IDEFICS (Laurençon et al.) 2024a and
Otter (Li et al. 2023a). Similarly, BLIP-2 (Li et al.) 2023b) and InstructBLIP (Dai et al. 2023 uti-
lize a pross-attennionn module called Q-Former to compress the image embeddings before inputting them into the LLM. Another line of work, represented by LLaVA (Liu et al.] (0024)) directly projects the image embeddings into the semantic space of the LLM with MLP, achieving decent perfor- mance and converging quickly. Extensions like Yao et al. 2024a Cai et al. 2024) add pooling modules after the MLP to reduce visual embeddings. However, projector design for video tasks is less explored. L.LaV--Ne'T-Vidae use mean pooling bilinear interpolation to aggregate visual embeddings, while neglecting the temporal dependency of video frames. To model the temporal dependency, VideoLLaMA2 (Cheng al. 2024) introduces downsampling module and spatial-temporal convolution module.
(Zhang et al.] (024c) and LLaVA -One Vision (Li et al. 2024a)
CONCLUSION
In this work, we explored the basic design space of visual context representation in video MLLMs from two major aspects: the number of frames per video (frame selection) and visual embeddings per frame (embedding selection). Using widely-used Video-MLLM architecture, we tested vari- ous sampling and compression strategies, collecting data points by varying frames and embeddings. We formulated the studied task as a constrained optimization problem, and then studied the scal- ing effects for frame selection and embedding selection. Then we fitted the performance function curve w.r.f. the two factors, and derived several important empirical findings to determine the two factors. Finally, we modeled their joint effects, derived the optimal setting, and validated the effec- tiveness with empirical experiments. Our findings highlight the significant impact of visual context representation on video MLLM performance, which is worth more research attention. Future work will explore advanced strategies for frame and embedding selection and develop architectures better suited for long video representation.
10

---

Published as a conference paper at ICLR 2025
ACKNOWLEDGEMENTS
This work was partially supported by National Natural Science Foundation of China under Grant No. 92470205 and 62222215, Beijing Municipal Science and Technology Project under Grant No. 120100000,22000, and Beijing Natural Science Foundation under Grant No. L233008. This research was also supported by Baichuan and the Outstanding Innovative Talents Cultivation Funded Programs 2024 of Renmin University of China. Xin Zhao and Bingning Wang are the corresponding authors.
REFERENCES
Cean-Baptiste Alayrac, Jeff Donahue, Pauline Luc, Antoine Miech, lain Barr, Yana Hasson, Karel Lene, Arthur Mensch, Katherine Millican, Malcolm Reynolds, et al. Flamingo: a visual language model for few-shot learning. Advances in neural information processing systems, 35:23716- 23736, 2022.
Jinze Bai, Shuai Bai, Shusheng Yang, Shijie Wang, Sinan Tan, Peng Wang, Junyang Lin, Chang Zhou, and Jingren Zhou. Qwen-vl: A frontier large vision-language model with versatile abilities. CoRR, abs/2308. 00s//308.12966, 2023.
Tom B Brown. Language models are few-shot learners. arXiv preprint arXiv:2005.1416. 2020.
Fabian Caba Heilbron, Victor Escorcia, Bernard Ghanem, and Juan Carlos Niebles. Activitynet:
A large-scale video benchmark for human activity understanding. In Proceedings of the ieee conference on computer vision and pattern recognition, pp. 961-970, 2015.
Mu Cai, Jianwei Yang, Jianfeng Gao, and Yong Jae Lee. Matryoshka multimodal models. arXiv preprint arXiv:2405.17433 2024.
Lin Chen, Xilin Wei, Jinsong Li, Xiaoyi Dong. Pan Zhang, Yuhang Zang, Zehui Chen, Haodong Duan, Bin Lin, Zhenyu Tang, al. Sharegpt4video: Improving vidco understanding and genera- tion with better captions. arXiv preprint arXiv:2406.0433 2024.
Zesen Cheng, Sicong Leng, Hang Zhang, Yifei Xin, Xin Li, Guanzheng Chen, Yongxin Zhu, Wenqi Zhang, Ziyang Luo, Deli Zhao, et al. Videollama 2: Advancing ppatial-temporal modeling and audio understanding in video-llms. arXiv preprint arXiv:2406.07476 2024.
Aidan Clark, Diego de Las Casas, Aurelia Guy, Arthur Mensch, Michela Paganini, Jordan Hoff- mann, Bogdan Damoc, Blake Hechtman, Trevor Cai, Sebastian Borgeaud, et al. Unified scaling laws for routed language models. In International conference on machine learning, pp. 4057- 4086. PMLR. 2022.
Wendi Dai, Junnan Li, Liyuan Zhang, Dongxu Zhang, Zhecan Chen, Yu Liu, Yue Xie, Lu Yuan, Lijuan Zhang, and Michael Zeng. Instructblip: Towards general-purpose vision-language models with instruction tuming. arXiv preprint arXiv:2305.0m 06500, 2023.
Mostafa Dehghani, Josip Djolonga, Basil Mustafa, Piotr Padlewski, Jonathan Heek, Justin Gilmer, Andreas Peter Steiner, Mathilde Caron, Robert Geirhos, Ibrahim Alabdulmohsin, et al. Scaling vision transformers 22 billion parameters. In International Conference on Machine Learning, pp. 7480-7512. PMLR, 2023.
Yifan Du, Hangyu Guo, Kun Zhou, Wayne Xin Zhao, Jinpeng Wang, Chuyuan Wang, Mingchen Cai, Ruihua Song, and Ji-Rong Wen. What makes for good visual instructions? synthesizing com- plex visual reasoning instructions for visual instruction tuning. arXiv preprint arXiv:2311.01453 2023.
Yifan Du, Kun Zhou, Yuqi Huo, Yifan Li, Wayne Xin Zhao, Haoyu Lu, Zijia Zhao, Bingning Wang. Weipeng Chen, and Ji-Rong Wen. Towards event-oriented long video understanding. arXiv preprint arXiv:2406. 141.2999.., 2024.
Jiajun Fei, Dian Li, Zhidong Deng, Zekun Wang, Gang Liu, and Hui Wang. Video-ccam: Enhancing video-language understanding with causal cross-attention masks for short and long videos. arXiv preprint Ar0iv:2408 14023, 2024.
11

---

Published as a conference paper at ICLR 2025
Chaoyou Fu, Yuhan Dai, Yondong Luo, Lei Li, Shuhuai Ren, Renrui Zhang, Zihan Wang, Chenyu Zhou, Yunhang Shen, Mengdan Zhang, et al. Video-mme: The first-ever comprehensive evalua- tion benchmark of multi-modde modall Ilms in video analysis. arXiv preprint arXiv:2405..1075. 2024.
Leo Gao, John Schulman, and Jacob Hilton. Scaling laws for reward model overoptimization. In International Conference on Machine Learning, pp. 10835-10866. PMLR, 2023.
Albert Gu and Tri Dao. Mamba: Linear-time sequence modeling with selective state spaces. arXiv preprint arXiv:2312.00752,, 2023.
Jordan Hoffmann, Sebastian Borgeaud, Arthur Mensch, Elena Buchatskaya, Trevor Cai, Eliza Rutherford, Diego de Las Casas, Lisa Anne Hendricks, Johannes Welbl, Aidan Clark, et al. Train- ing compute-optimal large language models. arXiv preprint arXiv:2203. 155.5526003.5.55,, 2022.
Jared Kaplan, Sam McCandlish, Tom Henighan, Tom B Brown, Benjamin Chess, Rewon Child, Scott Gray, Alec Radford, Jeffrey Wu, and Dario Amodei. Scaling laws for neural language models. arXiv preprint arXiv:2001.0883 2020.
Hugo Laurençon, Lucile Saulnier, Léo Tronchon, Stas Bekman, Amanpreet Singh, Anton Lozhkov, Thomas Wang, Siddharth Karamcheti, Alexander Rush, Douwe Kiela, et al. Obelics: An open web-scale filtered dataset of interleaved image-text documents. Advances in Neural Information Processing Systems, 36, 2024a.
Hugo Laurençon, Léo Tronchon, Matthieu Cord, and Victor Sanh. What matters when building vision-language models? arXiv preprint arXiv:2405.02244 2024b.
Bo Li, Yuanhan Zhang, Liangyu Chen, Jinghao Wang, Jingkang Yang. and Ziwei Liu. Otter: multi-modal model with in-context instruction tuning. corr .03/2305.03726 (2023), 2023a.
Bo Li, Yuanhan Zhang, Dong Guo, Renrui Zhang, Feng Li, Hao Zhang, Kaichen Zhang, Yanwei Li, Ziwei Liu, and Chunyuan Li. Llava-onevision: Easy visual task transfer. arXiv preprint arXiv:2408. 03326, 2024a.
Junnan Li, Dongxu Li, Silvio Savarese, and Steven Hoi. Blip-2: Bootstrapping language-image pre- pre-training with frozen image encoders and large language models. In International conference on machine learning, pp. 19730-19742. PMLR, 2023b.
Kunchang Li, Yali Wang, Yinan He, Yizhuo Li, Yi Wang, Yi Liu, Zun Wang, Jilan Xu, Guo Chen, Ping Luo, et al. Mybench: comprehensive multi- modal video understanding benchmark. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp- 22195-22206 2024b.
Xianhang Li. Zeyu Wang, and Cihang Xie. An inverse scaling law for CLIP training. In Alice Oh, Tristan Naumann, Amir Globerson, Kate Saenko, Moritz Hardt, and Sergey Levine (eds.), Advances in Neural Information Processing Systems tion Processing Systems 2023, NeurIPS 2023, New Orleans, LA, USA, December 10 16, 2023, 2023c.
36: Annual Conference on Neural Informa-
Yanwei Li, Chengyao Wang, and Jiaya Jia. Llama-vid: An image is worth 2 tokens in large language models. CoRR, abs/2311. 2023d. doi: 1044550/ARXXV2231111772 URL https: 2311 12311.17043
Prriiorr/100 10. 48550/arXiv
Zijing Liang, Yanjie Xu, Yifan Hong, Penghui Shang, Qi Wang, Qiang Fu, and Ke Liu. A survey of multimodel large language models. In Proceedings of the 3rd International Conference on Computer, Artificial Intelligence and Control Engineering, pp. 405-409, 2024.
Bin Lin, Bin Zhu, Yang Ye, Munan Ning, Peng Jin, and Li Yuan. Video-llava: Learning united visual representation by alignment before projection. arXiv preprint arXiv:2311.10123 2023.
Haotian Liu, Chunyuan Li, Qingyang Wu, and Yong Jae Lee. Visual instruction tuning. Advances in neural information processing systems, 36, 2024a.
12

---

Published as a conference paper at ICLR 2025
Jiajun Liu, Yibing Wang, Hanghang Ma, Xiaoping Wu, Xiaoqi Ma, Xiaoming Wei, Jianbin Jiao, Enhua Wu, and Jie Hu. Kangaroo: A powerful video- language model supporting long-context video imput. arXiv preprint 240.:2200..1554,, 2024b.
Ruyang Liu, Chen Li, Haoran Tang, Yixiao Ge, Ying Shan, and Ge Li. St-llm: Large language models are effective temporal learners. In European Conference on Computer Viston, pp. 1-18. Springer, 2025.
Al Meta. Introducing meta llama 3: The most capable openly available Ilm to date, 2024. URL hulps:/ai.. melu. com/blag/meta-llanne Accessed on April, 26, 2024.
Salman Khan Muhammad Maaz, Hanoona Rasheed and Fahad Khan. Video-chatgpt: Towards detailed video understanding via large vision and language models. ArXiv 2306. 2023.
OpenAl. GPT-4 technical report. CoRR, abMr2a33.0007774,,444 2023.
OpenAL. Hello gpt-4o. 2024. sttpe///ppeeaal..cmm/nnee///////
Alec Radford, Jeffrey Wu, Rewon Child, David Luan, Dario Amodei, Ilya Sutskever, et al. Language models unsupervised multitask learners. OpenAl blog, 1(8):9, 2019.
Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, et al. Learning transferable visual models from natural language supervision. In International conference on machine learning, pp. 7748-8766 PMLR, 2021.
Machel Reid, Nikolay Savinov, Denis Teplyashin, Dmitry Lepikhin, Timothy P. Lillicrap. Jean- Baptiste Alayrac, Radu Soricut, Angeliki Lazaridou, Orhan Firat, Julian Schrittwieser, Ioannis Antonoglou, Rohan Anil, Sebastian Borgeaud, Andrew M. Dai, Katie Millican, Ethan Dyer, Mia Glaese, Thibault Sottiaux, Benjamin Lee, Fabio Viola, Malcolm Reynolds, Yuanzhong Xu, James Molloy, Jilin Chen, Michael Isard, Paul Barham, Tom Hennigan, Ross Mcllroy, Melvin Johnson, Johan Schalkwyk, Eli Collins, Eliza Rutherford, Erica Moreira, Kareem Ayoub, Megha Goel, Clemens Meyer, Gregory Thornton, Zhen Yang. Henryk Michalewski, Zaheer Abbas, Nathan Schucher, Ankesh Anand, Richard Ives, James Keeling, Karel Lene, Salem Haykal, Siamak Shakeri, Pranav Shyam, Aakanksha Chowdhery, Roman Ring, Stephen Spencer, Eren Sezener, and al. Gemini 1.5: Unlocking multimodal understanding across millions of tokens of context. CoRR, 0t//2403.055300 2024.
Enxin Song, Wenhao Chai, Guanhong Wang, Yucheng Zhang, Haoyang Zhou, Feiyang Wu, Haozhe Chi, Xun Guo, Tian Ye, Yanting Zhang, et al. Moviechat: From dense token to sparse memory for long video understanding. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 18221-18221 2024.
Hugo Touvron, Thibaut Lavril, Gautier Izacard, Xavier Martinet, Marie-Anne Lachaux, Timothée Lacroix, Baptiste Rozière, Naman Goyal, Eric Hambro, Faisal Azhar, et al. Llama: Open and efficient foundation language models. arXiv preprint arXiv:2302. 13971, 2023.
Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Eukasz Kaiser, and Illia Polosukhin. Attention is all you need. In Advances in neural information processing systems, pp.
2017.
Peng Wang, Shuai Bai, Sinan Tan, Shijie Wang, Zhihao Fan, Jinze Bai, Keqin Chen, Xuejing Liu, Jialin Wang, Wenbin Ge, et al. Qwen2-vl: Enhancing aision-languag, model's perception of the world at any resolution. arXiv preprint arXiv:2409. /2191, 2024a.
Xidong Wang, Dingjie Song, Shunian Chen, Chen Zhang, and Benyou Wang. Longllava: Scaling multi-modal Ilms to 1000 images efficiently via hybrid architecture. arXiv preprint arXiv:2409.02889 2024b.
Dejing Xu, Zhou Zhao, Jun Xiao, Fei Wu, Hanwang Zhang, Xiangnan He, and Yueting Zhuang. Video question answering via gradually refined attention over appearance and motion. In Pro- ceedings of the 25th ACM international conference Multimedia, pp. 1645-1653, 2017.
13

---

Published as a conference paper at ICLR 2025
Fuzhao Xue, Yukang Chen, Dacheng Li, Qinghao Hu, Ligeng Zhu, Xiuyu Li, Yunhao Fang, Haotian Tang, Shang Yang, Zhijian Liu, et al. Longvila: Scaling long-context visual language models for long videos. arXiv preprint ar.iv:2408..01111 2024.
An Yang, Baosong Yang, Binyuan Hui, Bo Zheng, Bowen Yu, Chang Zhou, Chengpeng Li, Chengyuan Li, Dayiheng Liu, Fei Huang, et al. Qwen2 technical report. arXiv preprint arXiv:2407.10671 2024.
Linli Yao, Lei Li, Shuhuai Ren, Lean Wang. Yuanxin Liu, Xu Sun, and Lu Hou. Deco: Decou- pling token compression from semantic abstraction in multimodal large language models. arXiv preprint arXiv:2405. 20985, 2024a.
Yuan Yao, Tianyu Yu, Ao Zhang. Chongyi Wang, Junbo Cui, Hongji Zhu, Tianchi Cai, Haoyu Li, Weilin Zhao, Zhihui He, et al. Miniepm-v: A gpt-4v level milm your phone. arXiv preprint arXiv:2408.00 01800, 2024b.
Xiaohua Zhai, Alexander Kolesnikov, Neil Houlsby, and Lucas Beyer. Scaling vision transformers. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pp-
10004--2113,, 2022.
Xiaohua Zhai, Basil Mustafa, Alexander Kolesnikov, and Lucas Beyer. Sigmoid loss for language image pre-training. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pp. 11975-11986, 2023.
Peiyuan Zhang, Kaichen Zhang, Bo Li, Guangtao Zeng, Jingkang Yang, Yuanhan Zhang, Ziyue Wang, Haoran Tan, Chunyuan Li, and Ziwei Liu. Long context transfer from language to vision. arXiv preprint arXiv:2406. 16852, 2024a.
16852
Ruohong Zhang, Liangke Gui, Zhiqing Sun, Yihao Feng. Keyang Xu, Yuanhan Zhang, Di Fu, Chun- yuan Li, Alexander Hauptmann, Yonatan Bisk, and Yiming Yang. Direct preference optimization of video large multimodal models from language model reward, 2024b.
Yuanhan Zhang, Bo Li, haotian Liu, Yong jae Lee, Liangke Gui, Di Fu, Jiashi Feng. Ziwei Liu, and Chunyuan Li. Llava-next: A strong zero-shot video understanding model, April 2024c. URL
https: //1l://a/avaaaaaa vl.. gi github............ i /b blloblog///00 200// 04-30-0----00 11 ava - nee-teeett--e - vi deo/
Yuanhan Zhang, Jinming Wu, Wei Li, Bo Li, Zejun Ma, Ziwei Liu, and Chunyuan Li. Video instruction tuning with synthetic data. arXiv preprint arXiv:2410.027133 2024d.
Wayne Xin Zhao, Kun Zhou, Junyi Li, Tianyi Tang, Xiaolei Wang, Yupeng Hou, Yingqian Min, Beichen Zhang, Junjie Zhang, Zican Dong, et al. A survey of large language models. arXiv preprint XxXv:23333.18233 1(2), 2023.
Zijia Zhao, Haoyu Lu, Yuqi Huo, Yifan Du, Tongtian Yue, Longteng Guo, Bingning Wang, Weipeng Chen, and Jing Liu. Needle in video haystack: A scalable synthetic framework for benchmark- ing video milms. arXiv preprint arXiv:2406. 243.027400 2024.
Junjie Zhou, Yan Shu, Bo Zhao, Boya Wu, Shitao Xiao, Xi Yang. Yongping Xiong, Bo Zhang, Tiejun Huang, and Zheng Liu. Mlvu: A comprehensive benchmark for multi-task long video understanding, arXiv preprint 240.:22006042644 2024.
14

---

Published as a conference paper at ICLR 2025
A IMPLEMENTATION DETAILS
We list all the training data we used in T and the training hyperparameters are in Table 7
Table The statistics of our training data, including 1.8M image-text instructions and 0.7M video- text instructions.
Modality
Dataset
Samples
Image-Text
Cauldron
1.8M
VideoChatGPT-100 100K
100K
ShareGPT4Video
40K
ShareGPTVideo
255K
VIM
32K
Video-Text
NEXT-QA
40K
SthSth
40K
STAR
40K
TextVR
40K
CLEVRER
80K
Kinctics-710
40K
Total
2.5M
Table 7: Training hyperparameter.
Hyperparameter
Value
Global batch size
64
Gradient clipping
1
Weight decay
0
Warmup ratio
0.03
LLM Ir
2e-5
Projector
le-4
Vision encoder Ir
2e-6
Ir schedule
cosine
B EXPERIMENTS ON OTHER BACKBONES
To verify the generalization of our conclusion in Section 4 we utilize other representative base
models: LLaMA3-SB (Metal 2024) and CLIP-VTT-L-336px (Radford al. (021). Experiments conducted under the same settings as described in Section 4 The experimental results in Table
show that both the minimum loss and the optimal benchmark performance
achieved with 120
frames and 49 visual embeddings per frame, consistent with the conclusion in Section 4 This demonstrates the generalizability of our conclusion and are expected inspire future developments in video MI..Ms.
Table 8: Model performance under the same number of visual embeddings. The backbone models LLaMA3-8B and LLIP-VVT-L-336px..
# Embed./ Frames
# Total Embed.
Event- Bench
Long Video Bench
VideoMME wo/w-subs
Frames
Loss
VNBench
MLVU
Avg.
10
576
5760
0.90
20.00
18.52
49.95
45.80
44.22/43.37
36.98
30
196
5880
0.91
23.33
26.15
52.98
45.80
45.74/46.41
40.07
5832
19.33
26.67
54.99
46.03
46.59/47.26
40.15
120
49
5880
0.85
24.33
30.67
$7.68
47.16
46.93/47.89
42.44
162
36
5832
0.86
22.00
28.74
56.80
47.99
49.15/49.67
42.39
15

---

Published as a conference paper at ICLR 2025
C GENERALIZATION TO DIFFERENT VISUAL CONTEXT LENGTHS
Besides the optimal settings for 6K context window length, we also derive the Topt and Maps for MLLMS with different context lengths, ranging from 4K 126K, in Table To verify this conclu- sion, we conduct additional experiments with a visual context window length of 4K. The results in Table TO show that the optimal performance is achieved with 96 frames and 36 visual embeddings frame, which closely aligns with the predicted (T, .M = (97, 41). Due to memory constraints, leave the validation of longer context lengths as future work.
Table Estimated optimal visual tokens and frames for various visual context lengths.
Visual Context Length
Visual Tokens (M)
Frames (7)
Optimal Ratio
4,000
41
97
2.4
6,000
51
118
2.3
14,000
78
178
2.3
30,000
116
258
2.2
62,000
169
367
2.2
126,000
243
517
2.1
Table 10: Model performance under the visual context length of 4K.
# Embed./ Frames
# Total Embed.
Event- Bench
LongVideo Bench
VideoMME wo/w-subs
Frames
Loss +
VNBench
MLVU
Avg.
18
196
3528
0.8010
25.00
24.81
54.75
48.68
50.15/50.41
42.30
70
49
3430
0.7581
23.00
32.52
59.60
49.36
54.44/55.33
45.71
96
3456
0.7539
28.00
35.63
61.64
49.51
53.26/53.78
46.97
138
25
3450
0.7554
22.33
37.93
60.06
47.84
52.26/53. 19
45.60
D SCALING THE VISUAL EMBEDDINGS IN VISION ENCODER
In Section 3.2
we fix the number of frames and vary the number of visual embeddings input to the LLM study the effect of scaling visual embeddings. Besides varying the visual embeddings input to the LLM, we can also vary the visual embeddings produced by the vision encoder. In this section, we explore which strategy contributes more to the performance improvements.
Generally speaking, two methods can increase the number of visual embeddings produced by ViT- based vision encoder: (1) increasing the resolution of the frame, or (2) increasing the number of patches in each frame. To explore the contributions of longer visual embeddings after the vision encoder versus longer visual tokens input to the LLM, we conducted two sets of experiments based on these two methods.
Increasing the resolution of the frames. To increase the encoded resolution of the frames while keeping other factors the same, we utilize SigLIP-basee base-patch11-2 2-66 and Daa.l-P--aa--pach1l6 as the vision encoders and design the following three experiments:
Expl (Baseline): Standard setup with 256 visual tokens after the vision encoder and tokens (setting kernel size as 2) input to the LLM.
Exp2: Increasing the number of tokens after the vision encoder to 1024 while keeping the number of tokens input the LLM constant at 16.
Exp3: Keeping the number of tokens after the vision encoder at 256 while increasing the number tokens input to LLM to 64.
From the results in Table II observe: (1) Exp3 consistently improves performance across all benchmarks, demonstrating the benefit of increasing the number visual tokens input to the LLM. (2) Exp2 shows slight performance degradation some benchmarks compared Expl, which is
16

---

as the Published as a conference paper at ICLR 2025
Table 11: Comparing the influence of scaling visual embeddings in vision encoder versus in LLM. We increase the resolution of the frames to produce more visual embeddings in vision encoder.
# Emb. in VE
# Emb.
Event- Bench
Long Video Bench
VideoMME wo/w-subs
VNBench
MLVU
Avg.
for LLM
Exp1
256
16
19.33
25.26
51.98
45.04
47.93/48.1 19 39.62
Exp2
1024
16
23.67
24.89
53.74
44.97
47.70/48.78
40.63
Exp3
256
64
20.33
26.96
54.04
46.56
48.59/49.48
40.99
Table 12: Comparing the influence of scaling visual embeddings in vision encoder versus LLM. We increase the number of patches in each frame produce more visual embeddings in the vision encoder.
# Emb. in VE
# Emb. for LLM
Event- Bench
Long Video Bench
VideoMME wo/w-subs
VNBench
MLVU
Avg.
Expl
49
16
25.0
24.67
50.65
45.87
44.22/46.00
39.40
Exp2
196
16
13.33
22.22
47.83
40.12
41.85/40.93
34.38
Exp3
49
49
22.0
24.81
49.84
47.16
47.15/46.19
39.53
expected since more tokens after the vision encoder introduce greater information loss during token compression/samplingg to fit within the fixed LLM input size.
Increasing the number patches per frame. To increase the number of patches per frame while keeping other factors the same, we utilize clip-base-patchee patch32-224 and vision encoders and design the following three experiments:
Expl (Baseline): Standard setup with 49 tokens after the vision encoder and 16 tokens (setting kernel size as 2) input to the LLM.
Exp2: Increasing the number of tokens after the vision encoder to 196 while keeping the number of tokens input to the LLM constant at 16.
Exp3: Keeping the number of tokens after the vision encoder at 49 while increasing the number of tokens input the LLM to 49.
From the results in Table T2 we observe: (1) Exp2 shows consistent performance degradation across all benchmarks. The reason the same in the above experiment: more visual tokens after the vision encoder means that we may lose more information during token samplingccoppresion in order to keep the number of visual tokens input LLM the same. (2) Exp3, which increases the number of tokens input to the LLM, improves the performance on most benchmarks, further emphasizing the benefits directly optimizing the LLM input size.
The results from both sets of experiments consistently show that increasing the number of visual tokens input to the LLM is significantly more effective than increasing the number of visual tokens after the vision encoder. The findings highlight the importance of optimizing the LLM input for improved performance, as excessive compression of tokens after the vision encoder leads to infor- mation loss that cannot be recovered downstream.
E VERIFICATION OF THE SCALING EXPERIMENTS
E.1 ROBUSTNESS VALIDATION WITH MULTIPLE SEEDS
To confirm the consistency of our results, we trained the model with a widely used configuration (T. M) = (32, 196) using five different random seeds. The results in Table 13 demonstaate minimal variability in both loss and accuracy, with a particularly low standard deviation for the loss (0.0002). This confirms that our training process is robust and not significantly affected by randomness. Fur- thermore, the fluctuations in loss are smaller than those accuracy on downstream tasks. Therefore, used the loss stable metric estimate the parameters of the scaling law function.
17

---

Published as a conference paper at ICLR 2025
Table 13: The evaluation results of models trained with different random seeds.
Seed
Loss
ACC (%)
0.6377
47.20
2
0.6382
47.01
0.6377
46.62
4
0.6381
46.75
0.6379
47.11
Avg.
0.6379
46.94
Std.
0.0002
0,219
Table 14: The predicted loss according to the scaling function and the actual loss.
Setup
Hold-out point
Predicted loss
Actual loss
Sampling tokens
M 196
0.678
0.679
Compressing tokens
M = 196
0.630
0.638
Sampling frames
T 128
0.628
0.638
Compressing frames
T T= T=128
0.629
0.622
E.2 PREDICTIVE VALIDATION OF THE SCALING LAW
We conducted predictive validation by holding out the last data point, refitting the scaling function with the remaining points, and predicting the loss for the hold-out point. The predicted losses were then compared to the actual losses. As shown in Table the predicted losses consistently align with the actual values, with prediction errors remaining below 0.01. These results validate the reliability of our scaling law and confirm the robustness of our findings.
F COMPARION WITH EXISTING MLLMS
We compare our model with series of representative MLLMs, including four proprictary MLLMs:
GPT-4o (OpenAl 2024). Gemini-1.5 1emini-1.5-Par. 1.5-Pro (Reid et al. 2024). GPT-4V (OpenAl 2023). Qwen-
VL-Max (Bai al. [203)) as well as ten open-source MLLMs: Video-CCAM (Fei et 2024).
LLaMA-VVD-long (Li et al. 2023d), MovieChat (Song et al.
[024). ST-LLM (Liu et al., 2025
Video (Cheng et al. 2024), LongVA (Zhang et al. 2024a), and Long ViLA (Xue et al, 2024). I.LaV- Vineee Vision (Li et al. 2024a). V.LaVA-Video (Zhang et al.] (024d), and Qwen2- VL (Wang et al. 2024a). We evaluate our model and the baseline models on the long video under- model
standing benchmarks. The results in Table 15 show that there is still large gap between and SOTA results, primarily due to differences in training data. To mitigate this gap and demonstrate the superiority of visual context representation scheme, video instruction collection, ViKe (Zhang et al. 2024d), to replace the instructions used in re-train the model with (T, MM (120, 49)
incorporate the recently proposed
original experiments. Under this configuration. (denoted as "Ours+ Video 78K"). This adjustment yielded substantial performance improvements: our results on VNBench and Long Beng surpassed the SOTA, while Video-MME became comparable to the SOTA. However, MLVU and Event-Bench remain below SOTA levels. The dis- parity on MLVU and Event-Bench can be attributed the fact that the model /Zhang al, (2024d) is built upon powerful image MILMM pre-trained approximately 8M image instructions, scale not yet reflected our experimental setup. Nonetheless, these results underscore the effectiveness of our proposed experimental recipes and their potential when paired with richer data. In the fu- ture, we plan to incorporate more diverse and extensive image instructions to further enhance our video MLLM's capabilities and close the remaining gaps with SOTA. Notably, model even outperforms GPT-4V on certain benchmarks, such as Event-Bench and MLVU, demonstrating the effectiveness of optimal visual context representation scheme.
18

---

Published as a conference paper at ICLR 2025
Table 15: Experiment results on representative long video understanding benchmarks.
Training Data
Event- Bench
VidcoMME wo/w-subs
Models
VNBench
MLVU
Long VingeBeeen
Proprietary MLLMs
GPT-4o
Unknown 37.33
64.4
71.9/77.2
64.6
66.7
Gemini-1.5-Pro
Unknown
38.67
66.7
75.0/81.3
64.0
GPT-4V
Unknown
27.00
48.9
59.9/63.3
Qwen-Vit VL-Max
Unknown
51.3/51.2
42.2
Open-source MILMS
Video-CCAM-14B
4.4M
53.2/57.4
63.1
JaMA-VVD-llon--7B
1.6M
0.00
10.8
MovieChat-7B
Unknown
20.33
25.8
ST-LLM-7B
Unknown
16.67
22.7
37.9/42.3
LaMA2278
13.4M
46.6/-
48.5
LongVA-7B
1.3M
$2.6/-
56.3
Von--B
Unknown
50.5/-
Qwen2-VL VL-7B
Unknown
33.9
63.3/69.0
56.8
LLaVA-OneVisson-775
7.7M
28.7
35.7
58.2/61.5
64.7
56.5
LLa LLaVA-Video-7B
9.3M
43.0
40.7
63.3/69.7
66.9
58.2
Ours
2.6M
29.7
55.2/61.7
49.8
Ours+ Video 78K
3.4M
35.0
67.3
61.7/66.9
66.0
58.4
G EXPERIMENTAL RESULTS WITHOUT ANY FRAMES AS INPUT
We also report the results without any frames as the input. The results in Table 16 show that: (1) Event-Bench and VNBench the most reliant visual input. (2) Comparing the results with Section 3 the model achieves substantial gains across all benchmarks when visual inputs are included. These findings highlight the critical role of visual tokens in enabling video MLLMs to achieve high performance on diverse benchmarks.
Table 16: Evaluation results without any frames as the input.
Event-
Long Vidco
VideoMME
# Frames Bench
VNBench
MLVU
Avg.
Bench
wo/w-subs
0
19.30
7.56
40.48
36.64
34.89/36.48
29.23
19


Reviews:
[{"review_id": "iD6I1eGiSl", "reviewer": "Reviewer_UYUt", "summary": "This paper mainly discusses the video frame and visual token per frame selection strategies for video MLLMs. Based on LLaVA-like architecture, this paper conducts several experiments, e.g., varying frame number while fixing visual token per frame and altering visual token per frame while fixing total frame number, to fit the loss curves. Finally, it proposes a fitting function of the two factors and draw an optimal combination conclusion.", "questions": "Besides the question in Weakness, how can you determine the function formation of Eqs. 3, 4, 5, and 6? Just observe from figures 2a, 3a, and 4?", "limitations": "", "rating": 6, "confidence": 4, "soundness": 2, "presentation": 3, "contribution": 3, "strengths": "1. The motivation and idea of this paper is straightforward.\n2. The writing and structure organization of this paper is clear and easy to follow.\n3. The experimental results are clearly clarified, and the statistics are persuasive.", "weaknesses": "1. My main concern about this paper is the generalization of the final conclusion. As we all know, video MLLM is very large and complex. This paper uses only one model structure (SigLip + Qwen2-7B) to conduct experiments on several benchmarks. Is the final best setting ($T_{opt} \\approx 118$ and $M_{opt} \\approx 51$) generalizable to other models and benchmarks? If not, must we conduct several experiments to fit a new curve to find the best factor combination?\n2. Based on the above question, if this paper is just an experiment report, readers cannot obtain enough inspiration from this paper.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "yTEfBhcRnX", "reviewer": "Reviewer_4WaN", "summary": "Given a fixed Video-LLM window size, this paper examines the optimal selection of the number of video frames, the number of video tokens, and the compression/sampling methods used for frames and embeddings. The authors include experiments on both loss and benchmark evaluations.", "questions": "My major concerns lie in the soundness of the experiments. Please find more details in the 'Weaknesses' section", "limitations": "", "rating": 8, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "1. The paper is clearly written and follows a logical structure.  \n\n2. The paper includes extensive experiments and offers a thorough analysis. Notably, the experiments that consider fixed frames, fixed embeddings per frame, and the trade-off between embeddings and frames are particularly useful for following research.", "weaknesses": "1. Is the design space for embedding and frame operations too narrow? The findings within this design space may be biased towards different LLM backbones and vision encoders. We suggest that the authors conduct experiments across a variety of LLM backbones and vision encoders to ascertain the generalizability of their findings. Or at least one more different LLM backbone and vision encoders.\n\n2. In the experiments depicted in Figure 2 and Tables 1 & 2, the authors utilize the same T = 32 (L168) for all experiments, which results in video tokens of varying lengths within the same LLM context window size. It is generally assumed that longer tokens within a pretrained LLM context window size lead to better performance. How can it be determined whether longer visual embeddings after the vision encoder or longer visual tokens in the LLM contribute more significantly to performance improvements? \n\n3. Since this paper aims to study the scaling effect, conducting a sufficient number of experiments with training models is crucial to mitigate the impact of randomness bias. The authors should clarify the number of models trained for each data point presented in experiments.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "qrPan2PgMW", "reviewer": "Reviewer_WRue", "summary": "This paper analyses the input design space of Video Multimodal Large Language Models (MLLMs) by scaling two components: the number of frames and the number of tokens per frame.\nGiven the quadratic computational complexity of the attention mechanism in the number of input tokens, MLLMs are limited in the size of their input. This paper studies the impact of varying the number of frames and the number of tokens per frame, as well as the sampling and compression strategies to obtain the final tokens.", "questions": "Figure 2b and 3b: In addition to the performance curves on each benchmark, it would be valuable to also plot the average over all of them.\n\nSection 3: At the other end of the scaling spectrum, it would be interesting to also report the performance without any visual token at all (M=0) to see what would be the comparative performance of a \"blind\" MLLM.\n\nMinor typo:\nL325: \"increasing the frames\"", "limitations": "", "rating": 6, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 2, "strengths": "The paper offers a valuable analysis of visual context representation design in video MLLMs. The approach is straightforward, with clearly explained experimental settings and well-presented results, and the conclusions are well-supported by experiments.\n\nIt is interesting to see the contribution of mean pooling versus uniform sampling, both for selecting frames and selecting tokens.\n\nThe authors promise to release code and data for reproducing their results.", "weaknesses": "Comparison to SOTA results on the benchmarks.\nIt is understood that the goal of this paper is not to obtain SOTA results on the evaluated benchmarks, but rather to perform an analysis on the design space of visual context representation in video MLLMs. Nevertheless, it would be informative to the reader to have the information of how far the experimental recipes presented in this paper are from the SOTA.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (Poster)