Needle In A Video Haystack: A Scalable  Synthetic Evaluator for Video MLLMs
Authors: Zijia Zhao
Haoyu Lu
Yuqi Huo
Yifan Du
Tongtian Yue
Longteng Guo
Bingning Wang
weipeng chen
Jing Liu
Venue: ICLR 2025 Poster (2025)
Keywords: video MLLM
Primary area: datasets and benchmarks

Abstract: Video understanding is a crucial next step for multimodal large language models (MLLMs).
Various benchmarks are introduced for better evaluating the MLLMs.
Nevertheless, current video benchmarks are still inefficient for evaluating video models during iterative development due to the high cost of constructing datasets and the difficulty in isolating specific skills.
In this paper, we propose VideoNIAH (Video Needle in A Haystack), a benchmark construction framework through synthetic video generation. 
VideoNIAH decouples video content from their query-responses by inserting unrelated visual 'needles' into original videos. 
The framework automates the generation of query-response pairs using predefined rules, minimizing manual labor.  The queries focus on specific aspects of video understanding, enabling more skill-specific evaluations. The separation between video content and the queries also allow for increased video variety and evaluations across different lengths.
Utilizing VideoNIAH, we compile a video benchmark, VNBench, which includes tasks such as retrieval, ordering, and counting to evaluate three key aspects of video understanding: temporal perception, chronological ordering, and spatio-temporal coherence. We conduct a comprehensive evaluation of both proprietary and open-source models, uncovering significant differences in their video understanding capabilities across various tasks. Additionally, we perform an in-depth analysis of the test results and model configurations. Based on these findings, we provide some advice for improving video MLLM training, offering valuable insights to guide future research and model development.

Published as a conference paper at ICLR 2025
NEEDLE IN A VIDEO HAYSTACK: A SCALABLE SYNTHETIC EVALUATOR FOR VIDEO MLLMS
Zijia Zhao!, 1.2, Haoyu Lu Yuqi Huo", Yifan Du², Tongtian Yue! Longteng Guol?? Bingning Wang', Weipeng Chen', Jing Liu Institute Automation, Chinese Academy of Sciences
1.2f
School of Artificial Intelligence, University of Chinese Academy of Sciences 3 &Gooling School of Artificial Intelligence, Renmin University of China "Baichuan Inc.
ABSTRACT
Video understanding is crucial next step for multimodal large language models (MI.LMs). Various benchmarks are introduced for better evaluating the MILMss. Nevertheless, current video benchmarks are still inefficient for evaluating video models during iterative development due to the high cost of constructing datasets and the difficulty in isolating specific skills. In this paper, we propose Video- NIAH (Video Needle In A Haystack), benchmark construction framework through synthetic video generation. VideoNIAH decouples video content from their query-responses by inserting unrelated visual 'needles' into original videos. The framework automates the generation of query-response pairs using predefined rules, minimizing manual labor. The queries focus on specific aspects of video understanding, enabling more skill-specific evaluations. The separation between video content and the queries also allow for increased video variety and evaluations across different lengths. Utilizing VideoNIAH, we compile a video benchmark, VNBench, which includes tasks such as retrieval, ordering, and counting evalu- ate three key aspects of video understanding: temporal perception, chronological ordering, and spatio-temporal coherence. We conduct a comprehensive evaluation of both proprictary and open-source models, uncovering significant differences in their video understanding capabilities across various tasks. Additionally, we perform an in-depth analysis of the test results and model configurations. Based on these findings, we provide some advice for improving video MILM training, offering valuable insights to guide future research and model development.
INTRODUCTION
Multimodal Large Language Models (MLLMs) (Li ct al. 2023d Liu et al.
2023 Lu et al. 2024
OpenAl 2023 Team et al. 2023 Wang et al. 2023 Liu et al. (024b) have recently made significant
strides in understanding visual content. Video understanding is one of the most crucial next steps to mirror real-world scenarios, and numerous video-centric MILMs (Li et al.
2023e Maaz et al. 2023
Luo et al. 2023 Lin et al. 2023 Reid et al. 2024) have been proposed. Recent research has further established video benchmarks (Li et al. [023f Ning et al.) 2023 Chen et al. 2023 Liu et al. 2024e)
to assess specific aspects of video understanding in these models.
Most existing video benchmarks are designed assess the general understanding capabilities of video models and are curated to reflect real-world data. However, we argue that these benchmarks are inefficient for evaluating video models during the iterative development process. The inefficiency arises from two main issues: the high cost of constructing quality datasets and the inability to decouple different aspects of video comprehension, which makes it challenging to pinpoint specific weaknesses in the models.
The first challenge is the construction of high-quality, realistic video benchmarks, a process that remains time-consuming and complex, illustrated in Fig. [a). Selecting videos based on targeted
capabilities is crucial (Mangalam et al. 2024 Yu et al. 2211).. Additionally, creating these bench-
marks often requires labor- inbor-intensive tasks such prompt engineering (Li et al. 2023t Ning et al.
"Equal contribution. Corresponding author.

---

Published as a conference paper at ICLR 2025
(a) Previous Work
Raw Videas
Video Selection
Prompt Generate
Response Annotate
Filtering
Video Benchmark
Answer: Techniques mastering foreign language
short period of time.
Required Ability (hard decouple):
OCR (e.g. text each frame)
Object Recognization (eg things in each frame)
Knowledge (egg. landucapes represent countries)
Chronological Order (e.g. the sequence of events)
Reasoning (e inference from visual contents) and
Sample from realistic benchmark VideoMME
(b) VideONIAH Framework
Needle Selection
Video
Text Neadles
Image Needles
Neadles Insert, Rule Based Query
Without Selection
Response Automatic Generate
(e.g. Content, Order, Number, .8tc)
Synthetic Video Benchmark
Row Videos
VideoNIAH
Question:
What the order which objects
appear in the video?
Options:
bost, doz. flowerr bloytle dog, boat, flower, bicycle
flower, dog, blcyde bicycle, boat, flwwer
Required Ability (focus on single aspect):
Object Recognization (e.g Identify object simple)
Sample made by VideoNIAH
Chronological Order (eg. the sequence of objects)
Figure Comparison between VideoNIAH and de facto paradigm. Previous methods require extensive model/manual design to construct real-world data, and the resulting question-answer pairs often assess multiple video understanding capabilities simultaneously, making it difficult to decouple specific skills. In contrast, VideoNIAH framework automates data construction through predefined rules. These rules correspond distinct aspects of video understanding, enabling the precise identification of weaknesses in specific capabilities. We show detailed rules in Appendix A.4
2023
Chen et al. 2023
Liu et al. 2024e),
manual annotation (Jang et al. 2017 Mangalam et al. Chen et al., (203)), and data filtering (iiao al. [022]] Li
2024
Ning al. 2023
Yu et al.) 2019
2019) to ensure accurate alignment between query-answe pairs and video content. Furthermore, the risk of data leakage arises when using query-response pairs derived from real-world videos, as these videos may have been previously used during model training, compromising the fairness and objectivity of benchmark evaluations.
ct al. 20231 Liu et al. 2024e Yu et al.
The second limitation of existing benchmarks is their comprehensive nature (Fu et al.) 2024 Zhou al. 2024 He al.) [024] Liu al. 2024e Du et al. 20244] Wu al., 2024), which requires models to address multiple aspects of video content simultaneously. For instance, as shown in Fig. a[a. the example from VideoMME (Fu al. 2024) benchmark demands broad range of capabilities, including Optical Character Recognition (OCR), object detection, word knowledge, chronological order and temporal reasoning, etc. This comprehensive nature of such benchmarks makes difficult to evaluate specific weaknesses in any single capability, complicating the skill-specific improvement during the model iteration process.
To address these challenges, propose VideoNIAH (Video Needle In A Haystack), novel and scalable framework for constructing video benchmarks using synthetic video generation. This approach is inspired by advancements language model evaluations (Kamradt 2023 Song al.
2

---

Published as a conference paper at ICLR 2025
2024 Hsich et al. 2024). VidcoNIAH introduces innovative method that decouples test videos from their corresponding query-response pairs by embedding unrelated image or text "needles" into the original video "haystacks". This technique enables the use of diverse video sources with flexible lengths, offering significant scalability and adaptability. Moreover, VideoNIAH allows for the automated design of video understanding probing tasks by inserting multiple spatio-temporal "needles" into videos and generating the corresponding query-response pairs based on predefined rules. These rules are tailored to probe specific aspects of video understanding, ensuring that the evaluation of one capability is minimally influenced by other abilities. This framework significantly reduces the need for human labor while providing precise assessments of targeted model skills.
Utilizing VIdeoNIAH, we compile a decoupled video benchmark, VNBench, which includes tasks such as retrieval, ordering, and counting. These three tasks independently point to the three most important aspects in video understanding: temporal perception, identify chronological order and understanding spatio-temporal coherence. By isolating these abilities, VNBench enables a more focused evaluation of specific video comprehension capabilities, facilitating clearer assessment of model performance across distinct dimensions. Additionally, since the video sampled in VideoNIAH method is decoupled with task task-specific query-response pairs, we add various videos in any length and domain into the test set, which makes video evaluation under any context length possible. We evaluate 12 video understanding MLLMs on VNbench, including proprietary models and open-source models. We observe a significant performance gap between the proprietary and open- source models on temporal tasks (retrieval and ordering), with the proprictary models showing clear advantages. Moreover, most models perform poorly in apatial-temporal tasks (counting), suggesting that current video models are still far from perfect. Leveraging the flexibility of the VideoNIAH framework, we further investigate the impact of various components within VNBench, including video context length, the number, position, and characteristics of inserted needles. Moreover, we examine the effects of video model training settings and offer valuable advice for improving video MLLM training practices. Our contributions are summarized as follows:
We proposed VideoNIAH, simple, flexible and scalable synthetic framework for designing skill- skill-targeted video benchmark, which can be used to explore the different aspects of video understanding without heavy annotation resource cost.
To the best of our knowledge, we are the first to propose a synthetic video benchmark VNBench, which thoroughly examines three import aspects in video understanding: retrieval, ordering and counting, on a board range of context length.
We conduct a thorough analysis of both proprietary and open-source video models on VNBench, providing insights into the effects of context length, needle number, type, and position on model performance. Additionally, we offer practical advice for improving video MLLM training practices on comprehensive model analysis, with the aim of inspiring future research.
2
RELATED WORKS
2.1 VIDEO MLLMS AND VIDEO BENCHMARKS
Recently, various multimodal large language models (MLLMs) have been developed for video understanding. VideoChat (Li et al. 2023e) and Video-LLaMA (Zhang et al. 2023) are pioneering efforts that utilize large language models (LLMs) for this purpose. Subsequent models (Lin et al.)
2023 Li et al. 2023g Liu et al. 2024d Zhang et al. 2024). following training strategies similar to those of Valley (Luo et al. 2023) and VideoChatGPT Maaz et al., 2023). have been developed using open-source video data. MovieChat (Song et al. 2023) innovates by designing a complex mechanism that incorporates both short-term and long-term memory, enhancing Video-I..aMA' capability for understanding extended videos. Several video MLLMs (Wang al. 2024b] Li ct al. 2024 Zhao ctt al. 2023) apply more video SFT data to achieve more comprehensive ability or integrate external modality information to enhance performance. Meanwhile, proprietary models like GPT-4V (OpenAl 2023) and Gemini 1.5 Pro (Reid et al.) 2024) demonstrate superior video understanding capabilities through larger model parameters and more comprehensive training procedures.
In order to evaluate the video understanding capability of these MLLMs, several video benchmarks have been proposed. Moving beyond traditional video question-answering (QA) datasets (Xu et al..
3

---

Published as a conference paper at ICLR 2025
Video Haystack Type: Arbitrary Content, Arbitrary Length
Query Type:
Q Retrieval: Asking for the needle inf information
Required Ability: Temporal Perception
E. Ordering: Asking for the needle order
Required Ability: Chronological Order
Needle Type:
Counting: Asking for the needle number
Required Ability: Temporal Coherence
The zecret word Allie.
Subcitle
Image
Video Clip
Figure 2: Construction Overview of VNBench within the VideoNIAH Framework. Each sample in VNBench composed of several inserted needles, query-response pairs generated by predefined rules, and randomly selected video haystack.
2016
Caba Heilbron et al. 2015 Chen Dolan 201 Grunde- Mcl Laughlin al. 2021 Jang et
2017 Liu al. [2018] 8), more comprehensive benchmarks (Li ct al. [223f]] Maaz al. [0023] Liu ct al.
2024c; Chen et al. 2023 Ning et al. 2023 Fu et al. 2024| Zhou et al. 2024 Du et al. 2024] have
been proposed to encapsulate the diverse characteristics of video data comprehensively Additionally. specific benchmarks (Patraucean et al.) 2024 Li et al. 2023c Mangalam et al. [0024] Xiao et al. [021]]
[Song et al. 2023) have been designed evaluate models' proficiency in understanding long- cong-context videos within a QA framework. However, these video benchmarks, which are based on realistic videos, may encounter data leakage issues and require laborious annotation processes. VideoNIAH is a scalable synthetic video benchmark, that avoids the above issues and evaluates different video understanding abilities on boarder range of context lengths.
2.2 SYNTHETIC BENCHMARKS
Synthetic benchmarks (Kamradt 2023 al. 2023b Liu al. 2024a Reid al. 2024) provide
more control over factors like sequence length and task complexity. They are largely unaffected by the parametric knowledge acquired during model training, thereby eliminating the risk of data leakage. Needle in Haystack (Kamradt) 2023) (NIAH) first introduces synthetic framework for evaluating the in-context retrieval capabilities long-context large language models (LLMs). This method
involves embedding random synthetic statement within an unrelated long context and subsequently querying the model to retrieve this statement, Other approaches (Mohtashami & Jaggi (023) utilize special tokens strategies develop synthetic benchmarks tailored for LLMss. Counting-stars (Song) et al.] 2024)) and RULER (Hsieh et al., 2024) enhance the original 'needle in a haystack" task by introducing more complex settings and emphasizing the long-range dependencies of the inserted statements. VideoNIAH represents the first holistic synthetic benchmark for video understanding with diverse needle types, various query formats and long context length.
3 EVALUATION DATA RECIPE
3.1 VIDEONIAH: A FLEXIBLE SYNTHETIC FRAMEWORK
VideoNIAH is synthetic framework assess video model comprehension inspired by the "needle in a haystack (NIAH)" test (Kamradt 2023). Each sample in VideoNIAH involves "needles" representing inserted information, "haystack" for the original video context, and "query" directing the extraction of needles. The "needle" information in VideoNIAH is independent of the video content in "haystack". This independence necessitates focus on the synthetic design of the "needle" and the rule generate specific query-response pairs to accurately assess video understanding capabilities.
We provide a comprehensive VideoNIAH recipe tailored for the natural characteristics of videos considering the following reasons. 1) The inherent spatio-temporal relationships in videos motivate the use of both intra- intra-frame and inter- frame "needles". We employ two strategies to construct these spatio-temporal "needles": editing within individual video frames and inserting visual content across
4

---

Published as a conference paper at ICLR 2025
multiple frames. These needles can take the form of textual subtitles, static images, dynamic video clips, etc. 2) Multiple segments should be captured in a video at once. Consequently, we advocate for an increase in the number of "needles" from one to several in VideoNIAH. 3) The presence of extensive long-range associations
lengthy videos underscores the need to introduce more challenging queries that depend on long-term dependencies. In conclusion, the VideoNIAH synthetic framework is characterized by its combination of spatial and temporal analysis, an increased number of needles, and the introduction of long-deepp Long-dependency queries.
3.2 VNBENCH: A BASIC VIDEO UNDERSTANDING BENCHMARK
We construct video understanding benchmark VNBench with the above synthetic method, which shown in Fig. 2 VNBench contains three distinct tasks to address various aspects of video understanding, including short-dependency task (retrieval) and two long-dependency tasks (ordering and counting). Additionally, each task incorporates various types of "needles", considering both intra- frame editing and inter-frame inserting methods enrich the comprehensiveness of the evaluation.
Retrieval task is the basic NIAH task aimed at evaluating the long-context video model's ability to retrieve single needle. This task specifically assesses the model's capability for temporal perception and understanding across the time dimension. The retrieval task in VNBench is categorized into two types based the nature of the "needle": intra-frame editing needles and inter-frame inserting needles. For ferra-frame editing, we use inserted subtitles as the needle, while for inter- frame insertion, inserting needles are further divided into
static images are employed. Additionally, the two difficulty levels: simple and hard images.
Ordering task more challenging compared to the retrieving task men- tioned above. the ordering task, the model needs to identify the correct chronological order of all inserted needles. Ordering demonstrates the ability to comprehend temporal dy- namies and sequence events accu- rately in video, which reflects the model's proficiency in temporal rea- soning. Similar to the retrieval task, the ordering task in VNBench is di- vided into two sub-tasks based on needle type: intra-frame editing nee- dles and inter-frame inserting needles. And the inter-frame inserting needles also further divided into two diffi- culty levels.
Table 1: Task Statistics in VNBench
Needle Position
Task Name
Needle Type
#Sample
Edit Region
Insert Frrame
Short depondency Abillry
Retrieval-E Retrieval-11 Retrieval-12
Single Subtitle
150
Single Fruit Image
150
Single Landmark Image
150
Long dependency Abiliry
Ordering E Ordeting-11 Ordering-12
Four Subtitle
150
Four Fruit Image
150
Four Object Image
150
Counting El
Multiple Subside
150
Counting-F2 Counting I
Multiple Object Image
Multiple Object Image
150
Toral
1350
Insrr
Edi-ueel
EdhLevll
60-
Sample Number
Sample Number
Counting is also a challenging task for oong-context video models. In the
counting task, the model is required to provide the number of times the spec- ified object appears in video. Suc- cessfully counting recurring content over an extended video demonstrates the model's 's ability to recognize and track spatio-temporal coherence pat- terns, which is closely tied to its proficiency in maintaining temporal consistency and an internal representation of identified elements across different segments of the video. The counting task in vNBench is also divided into two sub-tasks: intra-frame editing needles and inter-frame inserting needles. Unlike the previous tasks, the counting task's difficulty is divided into two levels within the editing sub-task: subtitles and local image regions. In counting E2 task, the local image regions are applied to partial areas of the original video frames, testing the model's ability to recognize spatio-temporal coherence and maintain consistency over time.
6
Needle Depth
Needle Number
Figure 3: Needle Depth Dis- Figure 4: Needle Number Dis-
tribution in Retrieval task
tribution in Counting task
5

---

Published as a conference paper at ICLR 2025
As shown in T the VNBench benchmark contains three tasks, each with three sub-tasks focusing on different aspects. We sampled 150 video haystacks from three video data sources: MSRVTT (Xu jt al. 2016), NeXT Videos (Xiao al. 202I). and ActivityNet (Caba Heilbron et al. 2013) These videos contain various scenarios and range from 10 to 180 seconds in duration. We create nine task samples for each video haystack, resulting in total of 1350 samples for the entire test set. Additionally, we have created an extremely long video test set called VNBench-Long comprising four tasks, where the videos range from 10 30 minutes length. Each sample consists of video with inserted needles, question, four answer options, and one ground tround-truth answer. The duration of each needle set second. For the retrieval task, the answer options are sampled from the entire set of ground-truth candidates. For the ordering task, the negative answer options are shuffled versions of the ground-truth. For the counting task, we sample three negative options from a normal distribution centered around the ground-truth number. Further details on the construction of VNBench can be found in Appendix A
3.3 AUTOMATIC FILTER IN NEEDLE SELECTION
To prevent confusion between needle images and the video haystacks, we employ a straightforward yet effective filtering method for needle selection. We adopt CLIP model compute the image similarity between potential needle images and frames extracted from the video haystack. the maximum similarity exceeds predefined threshold (0.7, in VNBench), the candidate needle images are discarded from the current round of data generation.
3.4 EVALUATION STRATEGY
We define all tasks as multiple-choice questions. For each query, we provide four choices, with only one being correct. To reduce randomness in multiple-choice questions, adopt circular evaluation strategy. Each sample is evaluated four times, with the options shuffled each time. A sample is considered correctly answered only if the model selects the correct option all four times. We report the accuracy of different tasks.
4 EVALUATION RESULTS
4.1 MODELS
We select 12 video understanding models, including 9 open-source models and 3 proprictary models. The proprictary models are Gemini 1.5 Pro (Reid et al.) 2024) and the GPT-4 se- ries O 2023), accessed via the official API. We evaluate 2 different GPT-4 version include 1oot-4-turbo-2000 20--0---09999 and gpt 40---4-2222 20------22-------
The open-source video MLLMs include LLa Ve Video (Zhang et al. 2024), ST-LLM (Liu et al. (024d), LLaMA-VID (Li et al. 2023g), Video-LLaVA (Lin et al. 2023), VideoChatGPT (Maaz et al. 2023), VideoChat2
et al. 2023f Video -.Moo (Zhang et al. 2023), Qwen2-VI. VI. (Wang et al. 2024b) and LLaVA-
One Vision
(i et al. 2024). The detailed model setting is shown in Appendix C.I
4.2
MAIN RESULTS
In Table TT we report the whole result on VNBench tasks and the overall score for each model on the main split of VNBench. We summarize the result as follows:
1) Proprietary models perform better than open-source models on most VNBench tasks. In terms of overall accuracy, the highest performance among open-source models (58.7% for LLaVA- One Vision- 72B) and the highest performance among proprietary models (66.7% for Gemini 1.5 Pro) differ by 8.0%. Additionally, the average accuracy of proprictary models is also significantly higher than that of open-source models.
2) Performance on multiple-neede long-dependency tasks is lower than on single-needle short- dependency tasks. When comparing the accuracy across different tasks, we find that most models
"The results are provided in the Appendix D sawaaalllli-vii--b--pttthhhee

---

Published as a conference paper at ICLR 2025
Table 2: Evaluation Results on VNBench. VNBench comprises three synthetic tasks constructed
using the VideoNIAH method, with each task divided into three splits. "E" denotes intra-frame editing needles, while "T" represents inter-frame inserting needles. The numbers "1" and "2" refer to the difficulty levels of the sub-tasks, with "I" indicating simple and "2" indicating hard. In total, we evaluated 3 proprietary models and open-source models across these tasks.
Retrieval
Ordering
Counting
Video MLLLMs
Overall
B
L1
1-2
Avg
1-1
1.2
Avg.
E-1
E:2
Avg.
Proprietory Models
Gemini Pro Reid
2024
100.0
96.0
76.0
90.7
90.7
95.3
32.7
72.9
60.7
7.3
42.0
36.7
66.7
GPT-4o OpenAl
222
100.0
98.0
87.3
95.3
88.4
86.6
45.2
73.4
36.8
0.0
36.1
24.5
64.4
GPT-4-curbo OpenAl
2023
100.0
99.3
82.0
93.7
42.6
22.8
23.0
29.5
37.6
0.0
32.4
23.3
48.9
Open Open-source MLLMs
VidcoChaGPP
Maaz
2023
3.3
0.0
4.0
Vidoo-LaMM2
Zhang ctal.
2023
1.2
26.0
6.0
11.1
0.0
0.0
0.0
0.0
2.0
4.7
0.7
2.4
4.5
LLaMA-VID-7B
2023g
0.0
2.7
10.8
Vidko-LLVVA778
Lin ot
2023
26.0
28.0
17.3
23.8
0.7
0.7
2.0
1.1
16.7
0.7
20.0
12.4
12.4
VideoChar2 (ietal
32.7
1.3
0.7
8.0
12.4
MLaVA-NeXi-Viddos
Zhing et al.
2024
56.7
56.7
19.3
44.2
0.7
0.0
0.7
0.4
6.7
14.6
25.3
15.5
20.1
ST-LM (Liu 20244
64.7
51.3
0.0
21.3
1.3
27.3
16.7
22.7
LLaVA-OneVisioos-....
Lietal
2004
88.7
80.7
22.0
63.8
2.7
1.3
2.7
2.2
6.7
9.3
18.7
11.6
25.9
Qwen2-VI-7B Wang
2024h
76.0
69.1
12.7
8.7
26.0
9.3
24.7
20.0
33.9
L.L.V MlaVA-OneVisioo--7 13 al.
2024
88.7
87.3
55.3
77.1
70.0
50.0
37.3
52.4
41.3
8.7
27.3
25.8
51.8
ILaVA.OneVisiooa722
Li
2024
90.7
86.7
57.3
78.2
78.0
74.0
54.0
68.7
42.7
14.7
30.7
29.3
58.7
perform much better on retrieval tasks than on ordering and counting. For most proprietary models, they can retrieve almost all the inserted information (for example, 100% accuracy on Retrieval-F task). For some open-source models (such as ST-LLM, ILLaV--NeT--Vide Vision), the retrieval accuracy is also significantly higher than the other two tasks.
Qwen2-VI,, LLaVA-
3) The gap between open and proprietary models in the ordering task is enormous. The most advanced proprietary models are far ahead of other models in the ordering task (with Gemini 1.5 Pro at 72.9% accuracy on ordering task and GPT-4o at 73.4%), while most open-source models are nearly incapable of completing the ordering task with the exception of the LLaVA-One Vision series. This may be due to most open-source models' training processes overlooking the modeling of temporal sequences, thereby impairing the models' ability to process temporal relationships.
4) Counting is difficult, especially when counting hard-to-identify needles. Even on the more advanced proprietary models, the performance in counting is not very good. Moreover, the Counting-E-B E-2 task (detecting and tracking information deeply embedded within specific spatial areas of video segments), all models perform poorly, suggesting that current video models still lack the capability to deeply understand and model the fine- tne-grained
relationships in videos.
5 RESULT ANALYSIS
In this section, further analyze the evaluation results VNbench, conducting an in-depth analysis of different video understanding capabilities.
Effect of Haystack Length Since the query-resporne pairs constructed through VideoNIAH are unrelated to the original video content, we can fairly compare the models' video understanding ability to handle videos of different lengths by dividing them according to the length of the sample videos. In Fig. [ we divide the VNBench data into three splits based on the video haystack duration: short (10-30s), medium (30-60s), and long (60-180s). We observe that as the duration length of the videos processed changes, the performance of the proprietary models does not fluctuate significantly, thanks to their longer context processing windows (128k tokens for GPT-4 and 1M tokens for Gemini 1.5 Pro). However, for open-source models, handling donger-duration videos proves difficult, and models such as VideoChat2, LLa VA-NeXT- Video, and ST-LLM show a significant performance decline. This indicates that current models are still limited in the duration of video they effectively handle.

---

Published as a conference paper at ICLR 2025
Gemii Pro
GPT4g
Vidco LLaVA
STILM
LLAVA Onevision 76
GP1--turbo
VideoChaz2
Querg-ML-11
Retrieval-Edi
Retriecaa-nnset--Levll
Betrieval-hnerrteeeee
Drdering-Edd
Ordefing-insert--.o evell
Arcunsty
Accuracy
Accursst
Arcuncyy
40
medium
medium
Iong
short
medium
ston
(20-30))
301
1000
120-02
Time
Interval
Ordering-issett--wwe
Counting-Edt-levvi
Counting-&il-ievvel
Counting-Insert
Overall
Accunct
Accuacy
Accurnty
66-
60-
40-
20-
(10 3001
(10-0041
[10-500]
(33-000)
[00-1600)
[10
(30-000)
110
(10-624)
100g
110-30
230-C0M
100-1000
Interval
Time Interval
Intervei
Time Interval
Interval
Figure 5: Task performance on different video durations. We divide all VNBench videos into 3 splits: short( (00-3s,0 medium(30-60s) and long(60-1 180s). All the numerical results are in the Table II
Counting-Edit-Lovel
100
Counting- Insert
100
Retrieval
100
Ordering
B0
80-
Lovert
Accuracy
Accuracy
Accuraty
Accuracy
60
60
40
40-
20
20
47540
0-1
GP
Gemini
GPT-40
Needle Number
Model
Figure 6: Effect of needle number in Figure 7: Effect of needle type in UNBench-Retriene and VNBench-Counting task.
VNBench-Ordering task.
Effect of Needle Number In Fig. 6 we show the effect of the needle number in counting tasks, where the number of inserted needles varies among different samples. We observe that as the number of needles increases, the model's performance in the counting task significantly declines. This result highlights deficiencies in video understanding models regarding tracking and memory of objects within video content. It indicates that current video understanding models still need further optimization and improvement in understanding spatio-temporal relationships, attention mechanisms, and long-term term memory processing.
Effect of Recognizing Ability Visual recognition is fundamental ability for video MLLMs, as comprehensively analyzing visual contents on all frames is essential to understand their interrelation- ships. In Fig. 7 we illustrate the impact of different needle types on the retrieval task, highlighting the importance of robust visual recognition skills by comparing various needle categories. The retrieval task encompasses different sub-tasks, each probing a distinct aspect of visual recognition. The Retrieval-E sub-task evaluates the model's ability to identify specific local patches. The Retrieval-I-1 sub-task involves recognizing common objects. Conversely, the Retrieval-I-2 sub-task demands extensive world knowledge and keen eye for detail to identify landmark images, challenging the model's fine-grained visual recognition capabilities. We note that proprietary models like Gemini 1.5 Pro and GPT-4o excel recognizing subtitles and common images. However, more complex images tend to introduce confusion within the video. leading to noticeable drop in accuracy in the Retrieval-I-2 sub-task compared to Retrieval---1. This variation highlights the necessity of strong visual recognition abilities for advanced video MLLMs.
Effect of Needle Position We also explore the effect needle position in Fig. g We fix the video haystack and query-response pair in this position test Retrieval-I-1 task, just modifying the haystack length and needle position. The video haystack varies from 10 to 180 seconds, while the needle is inserted depths ranging from 0% to 90%. We evaluate the average accuracy for each samples, varying the needle depth and haystack
position using 32 different VNBench-Retrieval- length. For each sample, haystacks of different lengths are randomly cut from the same long video.
8

---

Published as a conference paper at ICLR 2025
(a) Gemini 1.5 Pro
GPT-to-1pp
IO LLaVA-
140 LLaMA-IDD-1pp
38
Curedat
Vdea Dostion
(e) Vide schat2-hhhhhc oframe
Video 16frame
(g) Video LaVe-rrrmee
(hi ST-LLM-32framm
10
Video
Video Dussiie
Vidao Dostion
Figure& 8: Results of varying depth and context length in VNBench-RRerieval The x-axis represents the video duration, while the y-axis indicates the context depth where the needle resides. Different color indicates different average accuracy of 32 different samples.
Ovenl
80
60-
60
Accurary
40-
Accuracy
80-
0.5B
78
728
253649
196
816
32
128
Model Size
Frame Token Number
Frame Sample Number
Figure 9: Model analysis model size. We evaluate LLa Visionn model fam- ily with different model sizes.
Figure 10: Model analysis on token number per frame. All models are trained with 32 frames.
Figure 11: Model analysis on frame number. All models are trained with different frames and 49 frame token numbers.
We observe three distinct patterns in this test. For the proprietary models, the test intervals we used are much shorter than their context window lengths, allowing these models to precisely recall all inserted information (Fig. 8 a.
For open-source models using sequential sampling, such as LLaMA- VID and LLaVA-NeXT-Vice a phenomenon similar to 'lost in the middle' (Liu et al.) (224c) in language models is evident (Fig. R e,d), where the models tend recall information at the beginning and end of long sequences, but pay less attention to the middle. For open-source models that employ uniform sampling strategy, such as Vidco-LLai VA and VideoChat2, they uniformly sample the input video to ensure a fixed number of frames are fed into the model. Thus, their recall results in position tests are more closely related to the sampling strategy, displaying a similar "bar-shaped" phenomenon (Fig. e-hh), where the successful recall of needle is tightly linked to whether it was sampled.
MODEL ANALYSIS
In this section, we mainly analyze several important items we think may influence the video MLLM's ability on spatial and temporal understanding. We use VNBench to validate the influence of these model settings. And in all experiments in this section, the training recipe of our model is kept same. We list the detailed training recipe Appendix G
Model Size We studied the impact of model parameters on the LLaVA-OneVision (Li et al. 2024 family, with a particular focus on the parameters related to the language model. In Fig. 9 we find that model parameters increase, performance across all VNBench sub-tasks improves, with the ordering task showing the most significant gains. This suggests that enhanced language abilities strengthen the model's ability perceive chronological order. However, observed only marginal improvement in the coherence task (counting), which aligns with the well-known difficulty language models face with counting tasks.

---

Published as a conference paper at ICLR 2025
Token Number Per Frame We explore the impact of varying the number of tokens per frame. Intuitively, increasing the token density per video frame should enhance spatial understanding. To investigate this, apply simple adaptive pooling kernel to compress the frame features into different token lengths, and train the model with varying numbers of frame tokens. The experimental results, as shown in Fig. [0 demonstrate minimal variation in performance across all sub-tasks in VNBench. This aligns with our initial design goal, which is to use VNBench primarily for evaluating the model's ability to understand temporal relationships in long contexts, rather than its spatial comprehension.
Frame Sampling Number Due to the constraints imposed by the current model's window size, most models rely on frame sampling to process video sequences. The frame sampling number determines how many raw video frames can be input into the model. We conducted an investigation into the effect of varying the number of sampled frames the model's temporal understanding capabilities. As illus- trated in Fig. [1] an increase the number of sampled frames leads to performance improvements in the retrieval and ordering tasks on VNBench. This suggests that future research should focus on opti- mizing video models to process a greater number of frames and overcome current window limitations. On the other hand, the counting task exhibits only marginal improvements, indicating that while nereasing the number of input frames enhances perceptual understanding. further progress in modeling higher-level and more complex tem- poral relationships (e-g.. spatio-temporal coher- ence) will require additional model optimiza- tion.
Table 3: Model analysis on temporal prompt. (idx) } means the index of the sampled frame/image. HH:MM:SS means the timestamp of the sampled frames in the raw video.
Prompt
RET
ORD
CNT
Overall
<imagg>>
62.4
12.2
13.3
29.3
We also investigated the impact of temporal prompts on enhancing the temporal awareness of video models. Temporal prompts provide explicit textual cues that inform the language model about time-related aspects
Temporal Prompt
image (idk): <image>
18.7
19.3
image fidx]: <image>
frame [idx]: <frame>
65.5
19.9
18.4
34.7
image: <image>
frame time HH MM.SS <frames
64.0
21.9
23.5
36.5
of the video, such as the order of frames or their corresponding timestamps. Our findings indicate that this straightforward approach significantly improves the model's s performance in both ordering and counting tasks in Table 3 This suggests that the temporal modeling capabilities of video models can be effectively enhanced by incorporating prompts that inject time-related information.
7 LIMITATIONS
VideoNIAH offers both scalability and flexibility, enabling video model researchers to design increas- ingly complex construction rules tailored to their needs, facilitating more comprehensive evaluations of video models. We hope our work serves as a catalyst for the rapid iteration and optimization of video model capabilities. On the other hand, traditional comprehensive benchmarks still hold irreplaceable value. They are based on real-world videos with human-validate questions that reflect authentic scenarios, which synthetic data generated by the VideoNIAH framework cannot fully replicate. We believe that real-world benchmarks should complement synthetic ones created using the VideoNIAH framework, working together to advance research in video models.
CONCLUSION
In this paper, we propose a scalable synthetic framework for benchmarking video MLLMs, named VideoNIAH, which decouples the relationship between video content and query-response pairs. It also separates different aspects of video understanding skills, allowing us to probe the strengths and weaknesses of video MLLMs. VideoNIAH can evaluate various dimensions of video comprehension and can be applied to diverse video sources and lengths. Utilizing this framework, we construct the first synthetic video benchmark, VNBench, which assesses video model capabilities (temporal perception, chronological order, spatio-temporal conherence) across a broad range of video contexts. We provide a comprehensive evaluation of both proprietary and open-source video MLLMs, revealing that they still struggle with long- long-distance dependency tasks on VNBench. Additionally, we conduct model-level analysis and offer valuable insights for improving video MLLM training. We believe our work will inspire future advancements in the field.
10

---

Published as a conference paper at ICLR 2025
9 ACKNOWLEDGMENT
This research is supported by Artificial Intelligence National Science and Technology Major Project(0222200121222 the National Natural Science Foundation of China (No. 62437001, 62436001), the Key Research and Development Program of Jiangsu Province under Grant BE2023016- the Natural Science Foundation of Jiangsu Province under Grant BK20243051.
REFERENCES
Fabian Caba Heilbron, Victor Escorcia, Bernard Ghanem, and Juan Carlos Niebles. Activitynet: A large-scale video benchmark for human activity understanding. In Proceedings of the ieee conference on computer vision and pattern recognition, pp. 961-970, 2015.
David Chen and William B Dolan. Collecting highly parallel data for paraphrase evaluation. In Proceedings of the 49th annual meeting of the association for computational linguistics: human language technologies, pp. 190-200, 2011.
Xiuyuan Chen, Yuan Lin, Yuchen Zhang, and Weiran Huang. Autoeval-video: An automatic benchmark for assessing large vision language models in open-ended video question answering. arXiv preprint arXiv:2311.14900 2023.
Yifan Du, Kun Zhou, Yuqi Huo, Yifan Li, Wayne Xin Zhao, Haoyu Lu, Zijia Zhao, Bingning Wang, Weipeng Chen, and Ji-R Ji-Rong Wen. Towards event-oriented long video understanding. arXiv preprint arXiv:2406. 14129, 2024.
Chaoyou Fu, Yuhan Dai. Yondong Luo, Lei Li, Shuhuai Ren, Renrui Zhang, Zihan Wang, Chenyu Zhou, Yunhang Shen, Mengdan Zhang, et al. Video-mma mm:: The first-ever comprehensive evaluation benchmark of multi- ilms video analysis. arXiv preprint arXiv:2405.21075 2024.
Raghav Goyal, Samira Ebrahimi Kahou, Vincent Michalski, Joanna Materzynska, Susanne Westphal, Heuna Kim, Valentin Haenel, Ingo Fruend, Peter Yianilos, Moritz Mueller-Freitai et al. The" something something" " video database for learning and evaluating visual common sense. In Proceedings of the IEEE international conference on compuler vision, pp. 5842-5850, 2017.
Madeleine Grunde-McLaughlin Ranjay Krishna, and Maneesh Agrawala. Agqa: A benchmark for compositional spatio-temporal reasoning. In Proceedings of the IEEE/CVE Conference on Computer Vision and Pattern Recognition, pp. 11287-11297, 2021.
Xuehai He, Weixi Feng, Kaizhi Zheng, Yujie Lu, Wanrong Zhu, Jiachen Li, Yue Fan, Jianfeng Wang. Linjie Li, Zhengyuan Yang. et al. Mmworld: Towards multi-discipline multi-faceted world model evaluation in videos. arXiv preprint arXiv:2406. S0T.::2000000007,, 2024.
Cheng-Ping Hsieh, Simeng Sun, Samuel Kriman, Shantanu Acharya, Dima Rekesh, Fei Jia, and Boris Ginsburg. Ruler: What's the real context size your long-context language models? arXiv preprint wwXiv:2404.066544 2024.
Yunseok Jang, Yale Song, Youngjae Yu, Youngjin Kim, and Gunhee Kim. Tgif-qa: Toward spatio- temporal reasoning in visual question answering. In Proceedings of the IEEE conference on computer vision and pattern recognition, pp. 2758-2766 2017.
G Kamradt. Needle in haystack presssur testing 1lms, 2023.
Bo Li, Yuanhan Zhang, Dong Guo, Renrui Zhang, Feng Li, Hao Zhang, Kaichen Zhang, Yanwei Li, Ziwei Liu, and Chunyuan Li. Llava-onevision: Easy visual task transfer. arXiv preprint arXiv:2408. 033326 2024.
Bohao Li, Yuying Ge. Yixiao Ge, Guangzhi Wang. Rui Wang, Ruimao Zhang, and Ying Shan. Seed- bench-2: Benchmarking multimodal large language models. arXiv preprint arXiv.2311.1709.. 2023a.
Dacheng Li, Rulin Shao, Anze Xie, Ying Sheng, Lianmin Zheng, Joseph Gonzalez, Ion Stoica, Xuezhe Ma, and Hao Zhang. How long can context length of open-source ilms truly promise? In NeurlPS 2023 Workshop Instruction Tuning and Instruction Following, 2023b.
11

---

Published as a conference paper at ICLR 2025
Jiapeng Li, Ping Wei, Wenjuan Han, and Lifeng Fan. Intentqa: Context-aware video intent reasoning. In Proceedings of the IEEE/CVF International Conference on Computer Vision. pp. 11963-11974, 2023c.
Junnan Li, Dongxu Li, Silvio Savarese, and Steven Hoi. Blip-2:
Bootstrapping language-image pre-training with frozen image encoders and large language models, 2023d.
KunChang Li, Yinan He, Yi Wang, Yizhuo Li, Wenhai Wang, Ping Luo, Yali Wang, Limin Wang. and Yu Qiao. Videochat: Chat-centric video understanding. arXiv preprint arXiv:2305.0633 2023e.
Kunchang Li, Yali Wang. Yinan He, Yizhuo Li, Yi Wang. Yi Liu, Zun Wang. Jilan Xu, Guo Chen, Ping Luo, et al. Mvbench: A comprehensive multi-modal video understanding benchmark. arXiv preprint arXiv:2311.17055, 2023f.
Yanwei Li, Chengyao Wang, and Jiaya Jia Llama-vid: An image is worth 2 tokens in large language models. arXiv preprint arXiv:231 I. 1714,, 2023g.
Bin Lin, Bin Zhu, Yang Ye, Munan Ning, Peng Jin, and Li Yuan. Video-Ilava: Learning united visual representation by alignment before projection. arXiv preprint arXiv:2311. 2023.
Feng Liu, Tao Xiang, Timothy M Hospedales, Wankou Yang, and Changyin Sun. ivqa: Inverse visual question answering. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp. 8611-8619, 2018.
Hao Liu, Wilson Yan, Matei Zaharia, and Pieter Abbeel. World model on million-length video and language with ringattention. arXiv preprint arXiv:2402.08226, 2024a.
Haotian Liu, Chunyuan Li, Yuheng Li, Bo Li, Yuanhan Zhang, Sheng Shen, and Yong Jae Lee. Llava-next: Improved reasoning, ocr, and world knowledge, January 2024b. URL https
//llava vl. githuu...b.....
00-30-0-00000000 11 llvvvlvaaaa Lavaanext//
Nelson F Liu, Kevin Lin, John Hewitt, Ashwin Paranjape, Michele Bevilacqua, Fabio Petroni, and Percy Liang. Lost in the middle: How language models use long contexts. Transactions of the Association for Computational Linguistics, 12:157-173, 202,cc
Ruyang Liu, Chen Li, Haoran Tang, Yixiao Ge, Ying Shan, and Ge Li. St-llm: Large language models are effective temporal learners. arXiv preprint arXiv:2404.00333 2024d.
Shilong Liu, Hao Cheng, Haotian Liu, Hao Zhang. Feng Li, Tianhe Ren, Xueyan Zou, Jianwei Yang, Hang Su, Jun Zhu, et al. Llava-plus: Learning use tools for creating multimodal agents. arXiv preprint arXiv:231 1.05 05443, 2023.
Yuanxin Liu, Shicheng Li, Yi Liu, Yuxiang Wang, Shuhuai Ren, Lei Li, Sishuo Chen, Xu Sun, and Lu Hou. Tempcompass: Do video lims really understand videos? arXiv preprint arXiv:2403.000 00476, 2024e.
Haoyu Lu, Wen Liu, Bo Zhang, Bingxuan Wang, Kai Dong, Bo Liu, Jingxiang Sun, Tongzheng Ren, Zhuoshu Li, Yaofeng Sun, et al. Deepseek-vl: towards real-world lasgun-language understanding. arXiv preprint ar X40::2403.05525. 2024.
Ruipu Luo, Ziwang Zhao, Min Yang, Junwei Dong, Minghui Qiu, Pengcheng Lu, Tao Wang, and Zhongyu Wei. Valley: Video assistant with large language model enhanced ability. arXiv preprint arXiv:2306.0r7 07207, 2023.
Muhammad Maaz, Hanoona Rasheed, Salman Khan, and Fahad Shahbaz Khan. Video-chatgpt: Towards detailed video understanding via large vision and language models. arXiv preprint AxXiv23006.05424.. 2023.
Karttikeya Mangalam, Raiymbek Akshulakov, and Jitendra Malik. Egoschema: diagnostic benchmark for very long-form video language understanding, Advances in Neural Information Processing Systems, 36, 2024.
Amirkeivan Mohtashami and Martin Jaggi. Landmark attention: Random-access infinite context length for transformers. arXiv preprint ar0iv:2305. /6300, 2023.
12

---

Published as a conference paper at ICLR 2025
Munan Ning. Bin Zhu, Yujia Xie. Bin Lin, Jiaxi Cui, Lu Yuan, Dongdong Chen, and Li Yuan. Video-bench: A comprehensive benchmark and toolkit for evaluating video-basee large language models. arXiv preprint arXiv:2311.161000 2023.
OpenAl. Gpt-4 technical report, 2023.
Viorica Patraucean, Lucas Smaira, Ankush Gupta, Adria Recasens, Larisa Markeeva, Dylan Banarse, Skanda Koppula, Mateusz Malinowski, Yi Yang, Carl Doersch, et al. Perception test: diagnostic benchmark for multimodal video models. Advances in Neural Information Processing Systems, 36, 2024.
Machel Reid, Nikolay Savinov, Denis Teplyashin, Dmitry Lepikhin, Timothy Lillicrap, Jean-baptiste Alayrac. Radu Soricut, Angeliki Lazaridou, Orhan Firat, Julian Schrittwieser, et al. Gemini 1.5: Unlocking multimodal understanding across millions of tokens of context, arXiv preprint arXiv:2403.055300 2024.
Enxin Song, Wenhao Chai. Guanhong Wang, Yucheng Zhang, Haoyang Zhou, Feiyang Wu, Xun Guo, Tian Ye, Yan Lu, Jenq-Neng Hwang, et al. Moviechat: From dense token to sparse memory for long video understanding. arXiv preprint arXiv:2307.164444 2023.
Mingyang Song, Mao Zheng, and Xuan Luo. Counting-starss A simple, efficient, and reasonable strategy for evaluating long-context large language models. CORR, abs/2403. 2024. doi: 10. 48550/ARXIV.2403.1 1802. URLpttps: //woi//oi//// org../...111100 httpg//110.44550//.v...o
11802
Gemini Team, Rohan Anil, Sebastian Borgeaud, Yonghui Wu, Jcan-Baptiste Alayrac, Jiahui Yu, Radu Soricut, Johan Schalkwyk, Andrew M Dai, Anja Hauth, et al. Gemini: family of highly capable multimodal models. arXiv preprint arXiv:2312. 1805, 2023.
Haochun Wang. Sendong Zhao, Zewen Qiang. Nuwa Xi, Bing Qin, and Ting Liu. Beyond the answers: Reviewing the rationality of multiple choice question answering for the evaluation of large language models. arXiv preprint arXiv:2402.01349 2024a.
Peng Wang. Shuai Bai, Sinan Tan, Shijie Wang. Zhihao Fan, Jinze Bai, Keqin Chen, Xuejing Liu, Jialin Wang, Wenbin Ge, et al. Qwen2-vl: Enhancing vision- -lngugge model's 's perception of the 2024b.
world at any resolution. arXiv preprint
Weihan Wang, Qingsong Lv, Wenmeng Yu, Wenyi Hong. Ji Qi, Yan Wang, Junhui Ji, Zhuoyi Yang, Lei Zhao, Xixuan Song, et al. Cogvlm: Visual expert for pretrained language models. arXiv preprint 1:Xiv:2311.03077 2023.
Sheng- Wei, Cheng- Kuang Wu, Hen-Hsen Huang, and Hsin-Hsi Chen. Unveiling selection biases: Exploring order and token sensitivity in large language models. arXiv preprint xYXiv:2406.03009, 2024.
Haoning Wu, Dongxu Li, Bei Chen, and Junnan Li. Longvideobench: A benchmark for long-context interleaved video-language understanding. arXiv preprint arXiv:2407.15754 2024.
Junbin Xiao, Xindi Shang, Angela Yao, and Tat-Seng Chua. Next-qa: Next phase of question- answering explaining temporal actions. In Proceedings of the IEEE/CVF conference computer vision and patlern recognition, pp. 9777-9786, 2021.
Jun Xu, Tao Mei, Ting Yao, and Yong Rui. Msr-vtt: A large video description dataset for bridging video and language. In Proceedings of the IEEE conference on computer vision and pattern recognition, pp. 5288-5296, 2016.
An Yang, Baosong Yang, Binyuan Hui, Bo Zheng, Bowen Yu, Chang Zhou, Chengpeng Li, Chengyuan Li, Dayiheng Liu, Fei Huang, et al. Qwen2 technical report. arXiv preprint arXiv:2407. 10671, 2024.
Zhou Yu, Dejing Xu, Jun Yu, Ting Yu, Zhou Zhao, Yueting Zhuang, and Dacheng Tao. Activitynet -a: A dataset for understanding complex web videos via question answering. In Proceedings of the AAAI Conference Artificial Intelligence, volume 33, pp. 9127-9134, 2019.
13

---

Published as a conference paper at ICLR 2025
Xiaohua Zhai, Basil Mustafa, Alexander Kolesnikov, and Lucas Beyer. Sigmoid loss for language image pre-training. In Proceedings of the IEEE/CVE International Conference on Computer Vision, pp. 11975-11986, 2023.
Hang Zhang, Xin Li, and Lidong Bing. Video-Ilama: An instruction-tuned audio-visual language model for video understanding. arXiv preprint arXiv:2306.000 02858, 2023.
Yuanhan Zhang, Bo Li, haotian Liu, Yong jae Lee, Liangke Gui, Di Fu, Jiashi Feng, Ziwei Liu, and Chunyuan Li. Lexva-next: A strong zero-shot video understanding model, April 2024. URL
https:// vl. github.t................ htt.:ttuu...obbog/////// wttbll/g//024-0---3333333 30 1lava next video/
Zijia Zhao, Longteng Guo, Tongtian Yue, Sihan Chen, Shuai Shao, Xinxin Zhu, Zehuan Yuan, and Jing Liu. Chatbridge: Bridging modalities with large language model as a language catalyst. arXiv preprint 0rXIv:2305.16103, 2023.
Junjie Zhou, Yan Shu, Bo Zhao, Boya Wu, Shitao Xiao, Xi Yang, Yongping Xiong, Bo Zhang, Tiejun Huang, and Zheng Liu. Mlvu: A comprehensive benchmark for multi-task long video understanding. arXiv preprint arXiv:2406. 042644 2024.
14

---

Published as a conference paper at ICLR 2025
Overview of Appendix
Details in VNBench Construction
Task Correlation Analysis
Evaluation Details
Evaluation Results on VNBench-Long
Evaluation Results VNBench-Act
Evaluation Robustness Analysis
Training Setting in Model Analysis
Complete Results on VNBench
Task Samples of VNBench
Consistency Evaluation
A DETAILS VNBENCH CONSTRUCTION
A.1 SUBTITLE NEEDLE
The subtitles we used have unified format:
The secret word is NAME.
while the candidate names are listed below.
Name Candidates
"Alice", "Bob", "Carol", "Dave", "Eve", "Frank", "Grace", "Harry", "Ivy", "Jack", "Kate", "Lco",
"Mary", "Nick", "Olivia", "Paul", "Quentin", "Rachel", "Sam", "Tom", "Uma", "Victor", "Wendy",
"Xander", "Yvonne", "Zach"
A.2 IMAGE NEEDLE
The fruit images we used are shown below:
Figure 12: Fruit Image Candidates.
Furthermore, we have also gathered object images from the MSCOCO dataset and landmark images from Seed-Bench2.
A.3 RESPONSE GENERATION
For Retrieval tasks, negative answer options are randomly selected from the pool of needle candidates. For ordering tasks, the negative answer options consist of shuffled version of the ground ground-truth answers. For counting tasks, we employ sampling method based on a normal distribution centered around the ground-truth number, thereby increasing the difficulty of the multiple-choice problem.
15

---

Published as a conference paper at ICLR 2025
Retrieval
RehrivaalEElf
Relifeval-Insert
Guentor: what the secrel word
In the video?
Option
ftot
C. Coro
Dave
Opsine:
word
cppls pwrr
bonano
stowbery
Ordering
Ordering Edit
Ordering-Inten
Question: what's ordor of the scorct word the vidcos
Option:
Corol:
Quandion:
Boo. Dovo Alcc Caro
Option
coro, Davo Alcc. Bob
lemon.
The secret word Alice.
CoroLAlca
8. lemon. biieoeyy mango, apple C. mongo, mongo, leron bluebery opple mango
Counting
Counting-Insen
Counting Edit-Levell
Counting-Edikn Fdiling-Edil-Leveee
Gueeiie monr limen does the sppple
Guestion How mone fimiss doos the socre! appear
aрpиer the vidwol
Calllone
Oprion: 3.3
Optiin:
A
8
A.3
B.
Figure 13: Illusions of different tasks in VNBench. Each sample in VNBench consists of several inserted needles, query-response pairs generated by pre-defined rules, and randomly selected video.
A.4 CONSTRUCTION DETAILS
human-made subtitle as the needle, which is similar the method used in (Reid et al., 2024). We sample a name word as the keyword and append it on the frames of random video clip in the format "The secret word is NAME". The query in this task asks for the secret word stated in the inserted subtitle.
Retrieval-Edi
Retrieval- Insert uses image the needle. Unlike subtitles appended video frames, these images are inserted between existing frames as static video clips. Furthermore, we divide this task into two levels according image recognizability. The level-1 task uses common images, such as fruit images, as the image needle, while the level-2 task adopts more challenging images, e.g., landmark images from SEED-Bench2 (Li al. 2023a).
Ordering-Eate uses human-made made subtitles as the needles. We sample four different names used in the Retrieval-E task as the needles and ask the model to determine the correct order of these unique inserted names.
Ordering-Insed uses images as the needles. Four images are sampled to be inserted between existing frames as static video clips. Then, the model required give the correct temporal order of these image needles. The task is also divided into two levels according to image recognizability.
Counting-Ead Edit asks the model count the appearing time of the object appended on the edited video segment. human-made subtitles the needles. We sample several names used in the Retrieval-E task as the needles and ask the model to provide the number of times the inserted subtitles appear. The level-2 task is the most challenging in VNBench. We choose one image from the candidate image
divided into 2 levels based on the task difficulty. The level-1 task uses
16

---

Published as a conference paper at ICLR 2025
set and append it to four random video clips. In each video clip, this image can appear to four times in different regions of the frame randomly. The model asked to count the total number of appearances of this specified object, requiring counting in both spatial and temporal dimensions. Counting-Inse Insert uses images the needles. We choose one image category and randomly sample several images from the image needles. We ask the model to provide the correct count of how many times one type of object appears in the vidco.
17

---

Published as a conference paper at ICLR 2025
B TASK CORRELATION ANALYSIS
B.1 SUB-TASK CORRELATION ANALYSIS
VNBench is constructed with the premise that different tasks can expose unique aspects in model performance. To affirm the legitimacy of these task categories and to facilitate the identification of key tasks, we conduct task correlation analysis. Our evaluation encompasses ten distinct models, with each task being characterized by vector that encapsulates the models' performance across range of context sizes. These nine task vectors are subsequently organized through an agglomerative clustering technique, where the correlation coefficient is utilized the measure of distance. As depicted in Fig. [[5] the tasks within each the two main categories (Retrieval and Ordering) tend to cluster together cohesive manner, devoid of overlap. On the other hand, the sub-tasks within the Counting category appear to be more autonomous, phenomenon attributed to the distinct types of 'needles' employed in these tasks. The
task leverages subtitles as needles to detect subtlcties within individual frames, while the Counting-Inseri task employs static frames as needles to gather information from within single frame. Conversely, the Counting-Edit-Levne task demands the insertion of multiple image needles across various frames, necessitating an advanced capacity for capturing both spatial and temporal details.
1.0
Ret-E
Ret-I1
0.9
Ret-12
Ord-E
Ord-11
Ord-12
0.7
Cnt-E1
0.6
Cnt-I
Cnt-E2
Cnt-E2
Cnt-I
Cnt-E1
Ord-12
Ord-11
Ord-E
Ret-12
Ret-Il
Ret-E
Figure 14: Sub-task correlation among 9 VNBench tasks.
18

---

Published as a conference paper at ICLR 2025
B.2 CROSS-TASE CORRELATION ANALYSIS
Here, we also analyzed the correlation between VNBench and other real-world video understand- ing benchmarks. Specifically, we selected six models with different numbers of sampled frames (Section()) 60. including models with 16, 32, 48, 64, 96, and 128 frames as input. These models were evaluated on real-world video understanding benchmarks. including VideoMME Fu et al. (2024). MLVU Zhou et al. (2024), and Long LongVideoBench Wu et al. (2024).
For each benchmark, we calculated the correlation coefficients of the corresponding performance vectors. The results show that all benchmarks achieved correlation coefficients above 0.75. Notably, VNBench demonstrated correlation coefficients higher than 0.9 with MLVU and VideoMME,, two commonly used real-world benchmarks. This indicates that VNBench can, certain extent, reflect a model's ability to process real-world videos.
MLVU LongVideoBench
VideoMME
VNBench
VNBench
Correlation
MLVU
Coefficient
LongVideoBench
VideoMME
as
Figure 15: Cross-task correlation among VNBench and real-world video understanding tasks.
VNBench
MIVU
Bone
VideoMME
VNBench MLVU
1. 1.000000
0.951426
0.823635
0.912259
0.951426
1.000000
0.782742
0.897072
Long Bench
0.823635
0.782742
1.000000
0.908773
VideoMME
0.912259
0.897072
0.908773
1.000000
Table 4: Cross-task correlation matrix among VNBench and real-world video understanding tasks.
#Frames
VNBench
MLVU
Long VideoBench
VideoMME
16
23.33
52.53
46.78
49.74
32
29.93
54.84
47.16
49,96
48
34.15
56,22
48.75
52,81
64
32.59
57.23
47.08
52.59
96
37.26
60.97
48.60
53.26
128
39.70
61.44
51.40
56.11
Table 5: Evaluation results VNBench and real-world benchmarks.
19

---

Published as a conference paper at ICLR 2025
C EVALUATION DETAILS
C.I BASELINE MODELS
We evaluate 10 video MLLMs, including 7 open-source models and proprietary models.
Gemini 1.5 Pro is generative model capable of processing contexts up to 1 million tokens, accom- modating videos as long as hour. We employ sampling strategy of frame per second, and all frames are fed into the model along with their timestamps.
GPT-4 is multimodal generative model with context window of approximately 128k tokens, capable of handling low-quality videos of about 5 minutes length. We utilize frame per second sampling rate for GPT-4, processing all frames through the model. The version of GPT-4 used including gpt-4- -uttuuro-222244 10002-004-5 and gpt-4o- 2024 -25-4305-3, represented as GPT-4o and GPT-4-turbo.
VideoChatGPT is multimodal generative model, equipped with a context window of approximately 4k tokens. employs the CLIP ViT-L/14 model for frame extraction, sampling 100 frames from the entire video.
Video-LLaMA2 is built top of BLIP-2 and MiniGPT-4. It is composed of core compo- nents: Vision- Language (VL) Branch and Audio-Language (AL) Branch. In our research, we have exclusively utilized the VL branch, which processes 8 input frames.
LLaMA-VID incorporates a 4K context window. It employs EVA-CLIP-Giant to extract one context and one content token for each specified frame. adopt a sampling rate of 1fps for the given video.
Video-LLaVA VA also employs a context window of 4k tokens. It leverage LanguageBind to extract the features of uniformly sampled frames.
VideoChat2 leverages a video encoder UMT-L to extract the features of uniformly sampled frames. use frame our main evaluation experiment. We use position interpolation to our input frame number.
ST-LLM leverages BLIP-2 extract the features of uniformly sampled frames. We use 32 frame in our main evaluation experiment.
LLa -LaVA-NeXT--ideo incorporates a context window comprising tokens and employs CLIP ViT-
I/14 to extract features from 32 evenly distributed frames. In our experiment, we sample video frames with 1fps sampling strategy. If the video contains more than 32 frames, we uniformly sample 32 frames from them. In our main evaluation experiment, we utilize the 7B model.
Qwen2-VL supports arbitrary image resolutions, mapping them into dynamic visual tokens for human-like visual processing. It uses Multimodal Rotary Position Embedding (M-ROPE) to handle positional information across text, images, and videos. We use 224x224 frame resolution and 1-fps frame sampling strategy for all video inputs.
LLaVA-One Vision an open large multimodal model designed based on insights from the LLaVA- NeXT series. It pushes the performance boundaries in single-image, multi-image, and video scenarios simultaneously. The model excels at transfer learning across different modalities, showcasing strong video understanding and cross scenario capabilities. We a unified 64-frame sampling strategy for 0.5B, 7B and 72B models
C.2 INFERENCE SETTING
For most video MILMMs, we use unified prompt to get the answer for each sample:
Unitied Inference Prompt Template
<QUETTION>
A. <OPTIONI>
B. <OPTION2>
20

---

Published as a conference paper at ICLR 2025
C. <OPTION>>>
D. <OPTIONN4>
Answer with the option's 's letter from the given choices directly.
For most models, we use a rule-based matching strategy to extract option letter from their response. However, some models such as VideoChatGPT and Video-LLaMA2, can not follow the instructions to output letter. They are tend to output direct answers for the questions. Thus we use GPT-3.5 (version gpt-3. 5ttor-o--oo--0 613) as the judge to identify whether they can correctly answer the question.
GPT Judge Prompt Template
SYSTEM:
You intelligent chatbot designed for evaluating the correctness of generative outputs for question- answer pairs. Your task is to compare the predicted answer with the correct answer and determine they match meaningfully. Here's how you can accomplish the task:
#HINSTRUCTIONS:
Focus the meaningful match between the predicted answer and the correct
Consider synonyms paraphrases as valid matches.
Evaluate the correctness of the prediction compared to the answer.
USER:
Please evaluate following video-based question-ansenn answer pair:
Question: <QUESTION>
Correct Answer: <GT ANSWER
Predicted Answer: <PREDICTED ANSWER>
the predicted answer expresses the same meaning as the correct answer, please output l; otherwise,
output
NOT PROVIDE ANY OTHER OUTPUT TEXT EXPLANATION, Only provide or
21

---

Published as a conference paper at ICLR 2025
D EVALUATION RESULTS ON VNBENCH-LONG
VNBench-Long a synthetic benchmark contructed with VideoNIAH method. It contains 4 tasks in VNBench, including Retrieval-Edit, Retrieval-Insert Ordering-Edit and Ordering-Insert. Unlike the primary test set, the video haystack in VNBench-Long ranges from 10 to 30 minutes, surpassing the context window limitations of most video MLLMs, except for Gemini 1.5 Pro, which can handle up to IM tokens simultaneously. Therefore, we primarily report task performance on Gemini 1.5 Pro. Performance comparisons across different time intervals are shown in Fig. 16 We observe that Gemini 1.5 Pro maintains stable performance even when video durations extend to 30 minutes, demonstrating robust long-context video understanding capabilities.
Retrieva E
Retrieval
Ordering-E
Ordering-
02
.88
22
23
99
93
Accuracy
80
60-
0
10-30s
30.60s
60-180s
10min
20min
30min
Video Duration
Figure 16: Evaluation Results on VNBench-Long
E EVALUATION RESULTS ON VNBENCH-ACT
VNBench-Ace is synthetic benchmark constructed using the VidcoNIAH method. In VNBench-Act, we aim evaluate the insertion of short video clips containing continuous natural frames from Something-Something V2 Goyal et al. (2017). This ensures that the inserted needle not only carries static image-level semantics but also conveys short-term action meanings.
It is worth noting that when the needles are extended to short video clips, the task becomes more challenging, especially for ordering tasks. This highlights limitations in modeling temporal action sequences effectively.
Method
Ret-Action
Ord-Action
Cnt-Action
Video-II La VA Lin al. (003)
27.3
0.7
7.3
LLaVA-NeXTI Video Zhang et al. 2004
42.7
0.7
11.3
ST-LLM Liu al. 2024d
54.7
0.0
22.7
GPT-4o OpenAl 2023)
85.6
11.3
10.0
Table Evaluation Results on VNBench-Act.
22

---

Published as a conference paper at ICLR 2025
F EVALUATION ROBUSTNESS ANALYSIS
F.1 CIRCULAR EVALUATION
In this analysis, assess the robustness of the top two models in our test, Gemini 1.5 Pro and GPT-4o. We adjusted the number of iterations in our circular test from 1 to 4 to demonstrate the robustness of video MILM inference, as shown in Table We observed that increasing the number of iterations effectively reduces randomness in the MLLM's inference process, leading to a more accurate and fair evaluation.
Table 7: Evaluation robustness analysis on VNBench. We report the top-2 model on VNBench, including Gemini 1.5 Pro and GPT-4o. For each model, we report the task accuracy I to 4 iteration number in circular evaluation.
Video
Retrieval
Ordering
Counting
Overall
MLLMs
1-2
Avg.
E
1-1
1-2
Avg.
E-1
E-2
Avg.
Proprietary Models
Gemini 1.5 Pro 4try
100.0
76.0
90.7
90.7
95.3
32.7
72.9
60.7
7.3
42.0
36.7
66.7
Gemini 1,5 Pro 3tyy
100.0
98.0
78.0
92.0
94.0
95.3
44.6
78.0
64.7
10.6
46.0
40.4
70.1
Gemini 1.5 Pro 2try
100.0
98.0
80.0
92.6
96.7
96.0
60.7
84.4
71.3
16.7
51.3
46.4
74.5
Gemini 1,5 Pro Ltry
100.0
98.0
87.3
95.1
98.0
96.7
72.0
88.9
80.7
24.7
58.0
$4.4
79.5
GPT-44 4try
100.0
87.3
95.3
86.6
45.2
73.4
36.8
36.1
24.5
64.4
GPT-4o Stry
100.0
98.7
87.3
95.3
91.2
90.6
53.3
78.3
44.2
2.7
38.8
28.6
67.4
GPT-4o 2try
100.0
98.7
88.7
95.7
92.5
92.7
62.8
82.7
50.3
11.4
47.5
36.4
71.6
GPT-4o Itry
100.0
98.3
92.0
97.1
95.2
96.0
76.3
89.2
61.7
28.9
55.6
48.7
78.3
F.2 NEEDLE CONTENT
In VNBench, goal is to decouple needle identification from the video understanding abilities aim to evaluate (e.g., temporal ordering). To this end, we use easy-to-identify objects as needles in several VNBench sub-tasks (e.g., Retrieval-E, Retrieval-11, Ordering-E, Ordering-Il, Counting-El] To verify the robustness of these inserted needles, we replace the needle content and insert them into the same positions within video haystacks.
Specifically, for insertion tasks, we replace fruit images (as shown in Fig. with animal images.
Figure 17: Animal Image Candidates.
Additionally, we replace subtitles (as shown in Appendix A.I with newly curated subtitle in editing tasks:
The private key is OBJECT.
where the object name is randomly sampled in candidates below:
23

---

Published as a conference paper at ICLR 2025
Object Candidates
"apple", "banana", "cherry", "desert", "eagle", "forest", "garden", "harmony", "island", "jungle", "kite",
"lemon", "mountain", "nectar", "ocean", "planet", "quartz", "river", "sunset", "tulip", "umbrella",
"village". "waterfall", "xylophone", "yogurt", "zebra"
We compare the evaluation results across different types of needle content in Table g We calcu- lated the correlation coefficients between the results before and after replacing the needle content. Experimental results indicate that the impact of needle content on test outcomes is minimal. This demonstrates the effectiveness of our decoupling strategy, ensuring that the evaluation focuses on the video understanding capabilities aim to assess rather than being influenced by simple object recognition.
Table 8: Impact of needle content. We evaluated the impact of needle content on test results across five tasks: Retrieval-E, Retrieval-11, Ordering-E, Ordering-I1, and Counting-El) sre refers the original VNBench, and new refers to the test results after replacing the needle content. We computed task vectors for the five models' test results these tasks and calculated the correlation coefficients between src and new.
Video MILMs
Retrieval-E src
Retrieval-I-1
Ordering-E new
Ordering-ID new
Counting E1
new
src
new
src
src
src
new
Gemini 1.5 Pro
100.0
100.0
96.0
98.0
90.7
92.0
32.7
31.3
60.7
59.3
GPT-4o LLaVA-NeXT-Viieo-77
100.0
100.0
98.0
96.0
88.4
90.7
45.2
46.0
36.8
36.0
56.7
55.3
56.7
60.0
0.7
0.7
0.7
0.7
6.7
3.3
ST-LLM
58.0
59.3
66.7
0.0
0.0
0.0
21.3
20.6
Video-LLaVA-72
26.0
22.0
28.0
26.0
0.7
0.7
2.0
2.0
16.7
15.3
Correlation Coefficient
0.9990
0.9965
0.9999
0.9993
0.9990
F.3 SAMPLE NUMBER
In this section, we investigate the effect of sample size on VNBench. The original VNBench consists of 1350 samples. To evaluate the impact of larger dataset, we curated new benchmark using the same methodology, which includes 2700 samples total. We tested five models on the expanded benchmark and compared the results with those from the original split, shown in Table 9 Our analysis reveals a relatively high correlation coefficient between the source dataset and the enlarged benchmark, indicating the robustness of our benchmark with respect the current sample size.
Table Impact of sample number. We evaluated the impact of sample size on the average accuracy across three VNBench tasks: Retrieval, Ordering, and Counting, sre refers to the original VNBench, which consist of 1350 samples, while mo re refers to the newly curated benchmark, which follows the same methodology but includes 2700 samples. We computed task vectors for the test results of five models across these tasks and calculated the correlation coefficients between sre and more.
Video MILMs
Retrieval
Ordering
Counting
Overall
src
more
Gemini 1.5 Pro
90.7
89.2
72.9
73.2
36.7
35.7
66.7
66.0
GPT-4o LLaVVe-eeT-Viido-7BB
95.3
96.7
73.4
75.2
24.5
24.3
64.4
65.4
44.2
42.4
0.4
2.3
13.5
20.1
19.4
ST-LLM
51.3
49.4
0.0
0.5
16.7
15.7
22.7
21.9
Video-LLaaVA77B
23.8
23.8
1.1
2.6
12.4
11.1
12.4
12.5
Correlation Coefficient
0.9998
0.9577
0.9990
0.9996
24

---

Published as a conference paper at ICLR 2025
G TRAINING SETTING IN MODEL ANALYSIS
For our trained MLLM, we used 1yglip-so400m--o -att000mmathh---8- 14-384 (Zhai et al., 2023) as the visual encoder for frames, with mean pooling and an MLP as the connection layer. The visual features from different frames were concatenated and combined with textual instructions before being input into the LLM, which utilized Qwen2-7B Yang et 2024).
We sample the input videos into several frames and encode each frame into fixed length of visual features with the visual encoder. An MLP modality projector then maps these visual features to visual tokens.
X = MLP(PPolling( (Fl )).... (PP(Pooling( FF
These visual tokens, along with human- pooaaddrovided textual instructions, are fed into large language model (LLM) to perform advanced video understanding tasks.
We train the model with visual instructions and responses to better understand human instructions in visual contexts. The loss function is defined as follows:
Loss
log Po (xt Xv, Xins,<as Xans, <i)
where is the sequence length, Xans,cs and iiss< represent the tokens from the answer and instruction sequences preceding the i-th token, and denotes the entire of model parameters.
We trained the video MLLM using the dataset shown in Table 10 with 8192 token context length. The training hyperparameters included global batch size of 64, with the learning rates set to 2c-5 for the LLM,, le-4 for the MIP, and 2e-6 for the visual encoder. A cosine learning rate schedule was applied. All models trained on 16 NVIDIA A100 GPUS with epoch.
Modality
Dataset
Samples
Image-Text Text
Cauldron
1.8M
VideoChatGPT- -O0K
100K
ShaceGPT4Video
40K
ShareGPTVideo
255K
VIM
32K
Video-Text
NEXT-QA
40K
SthSthV2
40K
STAR
40K
TextVR
40K
CLEVRER
80K
Kinetics-710
40K
Total
2.5M
Table 10: The statistics of our training data, including 1.8M image-text instructions and 0.7M video- text instructions.
25

---

Published as a conference paper at ICLR 2025
H COMPLETE RESULTS ON VNBENCH
Table 11: Complete evaluation results on VNBench. VNBench includes 3 synthetic tasks constructed with VideoNIAH method, while each task is divided into 3 splits. We evaluate proprietary models and open-source models in total.
Retrieval
Ordering
Counting
Video MILMs
Overall
E
I-1
1-2
Avg.
E
1-1
1-2
Avg.
E-1
B-2
Avg.
Video Haystack Length:
10-30s
Gemini Pro
100.0
92.7
92.0
32.0
74.0
68.4
GPT-4o
100.0
98.0
90.0
96.0
100.0
93.9
57.1
83.7
46.0
0.0
48.0
31.3
70.3
GPT-4-turoo
100.0
100.0
86.0
95.3
38.0
20.4
26.5
28.3
34.0
40.0
24.7
49,4
LLaMA-VID
8.0
18.0
28.0
18.0
0.0
0.0
0.0
0.0
18.0
2.0
14.0
11.3
9.8
Video-LLaVA
40.0
40.0
30.0
36.7
0.0
2.0
6.0
2.7
26.0
2.0
28.0
18.7
19.3
VideoChar2
82.0
76.0
26.0
61.3
0.0
4.0
1.3
3.3
22.0
LLaVA-NeXT-Vido--B
90.0
74.0
36.0
66.7
0.0
0.0
0.0
0.0
12.0
16.0
30.0
19.3
28.7
ST-LLM
70.7
0.0
21.3
30.7
Qwen2-VL--7 VL--BB
100
78.0
44.0
74.0
6.0
12.0
4.0
7.3
20.0
8.0
28.0
18.7
33.3
96.0
72.7
4.7
22.0
12.7
LLaVA-One LLavA-OneViiion-7B 7
100.0
100.0
72.0
90.7
94.0
78.0
58.0
76.7
64.0
12.0
32.0
36.0
67.8
LLaVA-OneVision-PP
100.0
98.0
76.0
91.3
96.0
66.0
86.7
52.0
12.0
36.0
33.3
70.4
Video Haystack Length:
30-60s
Gemini 1.5 Pro
100.0
96.0
80.0
92.0
90.0
92.0
32.0
71.3
60.0
8.0
42.0
36.7
66.7
GPT-4o
100.0
100.0
88.0
96.0
91.8
86.0
52.0
76.6
42.0
40.0
27.3
66.6
GPT-4-turbo
100.0
100.0
80.0
93.3
55.1
18.0
18.0
30.4
38.0
26.0
49.9
LLaMA-VID
4.0
14.0
16.0
11.3
0.0
0.0
0.0
0.0
10.0
0.0
14.0
8.0
6.4
Video-LLaVA
16.0
14.0
0.0
0.0
0.0
16.0
22.0
12.7
8.9
VideoChat2
38.0
34.0
14.0
28.7
0.0
0.0
0.0
0.0
6.0
0.0
10.0
5.3
11.3
Vide
ST-LLMM
60.0
80.0
30.0
56.7
0.0
0.0
0.0
0.0
22.0
0.0
32.0
18.0
24.9
Qwen2-VL-7B
70.0
30.0
66.0
18.0
14.0
14.0
15.3
24.0
14.0
30.0
22.7
34.7
LLaVA-OneVisione Vision-O.5B
100.0
90.0
26.0
72.0
0.0
0.0
4.0
1.3
12.0
4.0
22.0
12.7
28.7
LLaVA-OneVisioonTT
100.0
100.0
56.0
85.3
92.0
58.0
42.0
64.0
46.0
10.0
36.0
30.7
60.0
LLaVA One Vision 72B
100.0
100.0
56.0
85.3
96.0
94.0
70.0
86.7
$2.0
22.0
38.0
37.3
69.8
Video Haystack Length: 1N0s
Gemini 1.5 Pro
100.0
94.0
68.0
87.3
90.0
96.0
34.0
73.3
56.0
10.0
38.0
34.7
65.1
GPT-4o
100.0
98.0
84.0
94.0
73.5
80.0
60.0
22.3
20.4
14.9
56.3
GPTT-4turro
100.0
98.0
80.0
92.7
34.7
30.0
24.5
29.7
38.8
0.0
18.4
19.1
47.2
I.LMA-VID
16.0
0.7
Video-LLaVA
20.0
28.0
14.0
20.7
2.0
0.0
0.0
0.7
8.0
0.0
10.0
6.0
9.1
VideoChat2
10.0
10.0
8.0
0.0
0.0
3.3
3.8
LLaVA-NeXT-Vidoo
26.0
36.0
10.0
24.0
2.0
0.0
2.0
1.3
0.0
20.0
12.0
10.7
12.0
ST-LLM
26.0
38.0
16.0
26.7
0.0
0.0
0.0
14.0
14.0
10.7
12.4
Qwen2-VL. VL.-BB
96.0
80.0
26.0
67.3
24.0
12.0
8.0
14.7
34.0
16.0
18.7
33.6
LLaVA-One OneVision-O.5
66.0
56.0
18.0
46.7
2.0
0.0
0.0
0.7
2.0
12.0
14.0
9.3
18.9
LLaVA-One Vision-7B
62.0
38.0
55.3
24.0
14.0
12.0
16.7
14.0
14.0
10.7
27.6
LLaVA-One LLaVA-OOneiision-722
72.0
62.0
40.0
58.0
42.0
30.0
26.0
32.7
24.0
10.0
18.0
17.3
36.0
Video Haystack Length: 10 minutes
Gemini Pro
100.0
Video Hoystark Length: 20 minutes
Gemini 1.5 Pro
100.0
93.3
86.7
93.3
Video Hoystack Length: 30 minutes
Gemini 1.5 Pro
100.0
93.3
93.3
93.3
26

---

Published as a conference paper at ICLR 2025
I TASK SAMPLES OF VNBENCH
The whole VNBench contain tasks. Here we show samples of generated task data.
Retrieval-Edi Task Sample
Question:
What is the secret word in this video?
Options:
A. Rachel, Carol, C. Mary, D. Nick
Answer:
B. Carol.
Retrieval-Insert-LLeelll Task Sample
Question:
What is the fruit that appears in this video?
Options:
A. Apple, B. Pear, C. Peach, Lemon
Answer:
B. Pear.
27

---

Published as a conference paper at ICLR 2025
Retrieval- Sittivvall-nser-Leeeell Task Sample
Question:
What is the landmark that appears in this video?
Options:
A. Kagoshima Prefectural Kamoike Athletic Stadium
B. Stadtkirche (Thun)
C. Waterkant, Paramaribo
D. Templo Khadro Ling (Budismo Tibetano), Treas Coroas, Brasil Stadtkirche (Thun).
Answer:
Ordering- Edit Task Sample
The SEOE DIZE DES
IEE Pa
Question:
What is the order of the secret words that appeared in the video?
Options:
A. Xander, Kate, Grace, Olivia
B. Kate, Xander, Olivia, Grace
C. Olivia, Xander, Kate, Grace
D. Kate, Grace, Xander, Olivia
Answer:
D. Kate, Grace, Xander, Olivia.
28

---

Published as a conference paper at ICLR 2025
Ordering Cedering-Insert-Lone Insert-Levell Task Sample
Question:
What is the order of fruits appearing in the video?
Options:
A. banana, grapes, lemon, pear banana, lemon, grapes, pear
C. grapes, lemon, banana, pear
D. banana, pear, lemon, grapes
Answer:
B. banana, lemon, grapes, pear.
Insertie-nssrr-LLeeee2 Task Sample
GO
00
Question:
What the order of images appearing in the video?
Options:
A. chair, sofa, horse, bus
B. chair, sofa, bus, horse
C. chair, bus, sofa, horse
D. horse, sofa, bus, chair
Answer:
D. horse, sofa, bus, chair.
29

---

Published as a conference paper at ICLR 2025
Counting-Edit-Levell Task Sample
Question:
How many secret words appeared in the video?
Options:
A. 3, B. C. D.
Answer:
A.
Counting-Edit-Levell Task Sample
Question:
Some boats were inserted into small sections of the video. How many boats appeared in the video total?
Options:
A. 9, B. 11. C. 10, D. 12
Answer:
C. 10
30

---

Published as a conference paper at ICLR 2025
Counting-Inser Task Sample
WON Lodk nig BOEA
WON Lonk allie Thaee
atele Thaesred
Look atiic Than
Teo hcary
Too hear
Yea heavy
Question:
How many dogs appeared in the video?
Options:
A. 5, B. 4, C. 3, 6
Answer:
B. 4
31

---

Published as a conference paper at ICLR 2025
J CONSISTENCY EVALUATION
Since our tasks are in multi-choice QA format, and most video MLLMs (excluding VideoChat- GPT Maaz et al. 2023) and VideoLLaMA2 Zhang et al. (003)) which directly output sentence and must GPT to evaluate) have strong instruction-foll follooiin capabilities, their output is constrained to the four provided option letters. This means that even if the model cannot select the correct answer, it will still output one of the option letters.
As mentioned in Wang et al. 2024a); Wei al. 2024). LLMs inherently have preferences for certain options, and different models show varying sensitivity the order of options. When options shuffled (in our circular evaluation) and a model lacks the ability to analyze the question but forced output answer, it tends to rely on internal biases select its preferred option letter rather than making a choice based on option content. This can lead to situations where the model outputs the same option for a shuffled version of the question, even though the content corresponding to that option differs.
We analyzed this phenomenon on 9 sub-tasks in Tables 12 14 First, we examined the option consistency of VAVA--OneVisie series, Gemini 1.5 Pro, and GPT-4 series. Since our circular evaluation involves four iterations with shuffled option orders, only predictions where the option content remains the same across all four iterations are considered consistent. We define Consist. as the proportion of consistent predictions across these four iterations, and Acc. as the accuracy of the model's 's predictions under circular evaluation.
In this setup, two outcomes are possible:
1. The predictions are consistent, meaning the model may predict the same answer as the GroundTruth (correct) or different answer (incorrect).
2. The predictions are inconsistent, which results in the answer being marked as incorrect in circular evaluation.
Thus, accurate predictions are a subset of consistent predictions. We compute the conditional probability Acc.Coonsist, which represents the accuracy of the model's prediction given that the results are consistent.
Our conclusions are follows:
1. When the conditional probability Acc.IConsist. is significantly higher than 25% (roughly equivalent to random guessing). the accuracy can effectively reflect the model's 's ability to handle the task. In Counting- E1 and -ounting-I, the results from the model performance comparison are valuable.
2. When the conditional probability Acc.IConsist. fluctuates around 25% or falls below 25%, comparisons of accuracy have limited significance. In this case, the conclusion is that the models unable to solve the task, meaning that all models perform unsatisfactorily Counting-E2. This represents a common issuc across all models and indicates an area where VideoMLLMs need optimization.
3. For GPT-series models, we find that compared to Gemini 1.5 Pro and LLaVA-One Vision series models, their Consistency Rate on unsolvable tasks (e.g., Counting -E2) is very low (5.3% for GPT-4o, 3.6% for GPT-4-turbo). This suggests that these models are highly sensitive the order of options, resulting in 0.0% accuracy in circular evaluation. While other models also fail to provide correct answers (reflected by the conditional probability Acc.IConsist. around 25%. roughly equivalent random guessing), their higher Consis- tency Rate leads to slightly higher Acc. values. However, as concluded in point 2, such comparisons are not significant. This phenomenon can also be validated in Table 7 where GPT-4o shows performance closer to Gemini 1.5 Pro with fewer evaluation iterations.
4. The low Consistency performance of GPT-series models is seen only in tasks they cannot solve. On tasks they can handle, their Consistency Rate is high (in Table 12 and Table 13).
The above conclusions suggest that for VNBench, comparisons at high Acc. levels reflect the models' abilities, while low Acc. levels indicate common issue for all Video MLLMs.. In fact, this conclusion
32

---

Published as a conference paper at ICLR 2025
holds true for most multi-choice QA benchmarks. Meanwhile, we believe that Acc./Consist. may also be an important metric in circular evaluation.
Table 12: Accuracy and consistant rate of LLde in multi-try circular evaluations of VNBench Retrieval Tasks.
Retrieval-E1
Retrieval-11
Retrieval-12
Video MLLMs
Acc.
Consist.
Ace IConsist
Acc.
Consast
Acc Consist.
Ace.
Consist.
Acc Connsist
candomm choiee
25.0
25.0
25.0
Gemini 1.5 Pro
Reid et al. 2024
100.0
100.0
100.0
96.0
96.7
99.3
76.0
80.0
95.0
GPT-40 OpenAl
2023
100.0
100.0
100.0
98.0
100.0
98.0
87.3
88.0
GPT-4-urbo OpenAA 2023
100.0
100.0
100.0
99.3
99.3
100.0
82.0
$4.0
97.6
O.a
Li
2024
88.7
91.3
97.2
80.7
88.7
91.0
22.0
50.7
43.4
LLaVA-OneVisioon
et
2024
88.7
92.0
96.4
87.3
94.7
92.2
55.3
72.0
OL.WAo-OeeVisioonPP 72B Li
2024
90.7
95.3
95.2
86.7
96.7
89.7
57.3
64.7
88.6
Table 13: Accuracy and consistant rate of VideoMLLLs in multi-try circular evaluations of VNBench Ordering Tasks.
Ordering-E
Ordering-11
Ordering-12
Video MLLMs
Acc.
Consist.
Acc. Conniss
Acc.
Consist.
Acc. AcnsiCansist
Acc.
Consist.
Acc. ACosiSons
random choice
Gemini 1.5 Pro (Reid
2024
90.7
90.7
100.0
95.3
95.3
100.0
32.7
33.3
98.2
GPT-4o OpenAl
2023
88.4
88.4
100.0
86.6
87.2
99.3
45.2
46.6
97.0
GPT-4.turbo OpenAL 2023
42.6
43.2
98.6
22.8
24.8
91.9
23.0
24.3
95.8
LLaVA-OneVision-O.55.
Li at
2004
2.7
5.3
50.9
1.3
3.3
39.4
2.7
5.3
50.9
LLaVA-OneVision-PP
2024
73.3
LLaVA-DneVisoon-222 Liet al
2024
78.0
80.7
96.7
74.0
76.0
97.4
54.0
56.7
95.2
Table 14: Accuracy and consistant rate of VideoMILMMs in multi-try circular evaluations of VNBench Counting Tasks.
Counting-El E1
Counting-E2 E2
Counting-1
Video MLLMs
Acc. Consist.
AcniCsnssst
Acc.
Consist.
Acc Consist
Acc.
Consist.
AcciConsist.
Tundom choice
25.0
25.0
25.0
Gemini 1.5 Pro Read atal 2024
60.7
67.3
90.2
7.3
27.3
26.7
42.0
$4.0
77.8
OpenAl
GPT-4.turb OpenAl 2023
37.6
44.9
83.7
0.0
3.6
0.0
32.4
36.2
89.5
LLaVA-One Viseeiinnn-.B.
LLaVA LLaVA-OneVision OneVision 7B
Li et al
2024
41.3
72.0
57.3
8.7
48.7
17.9
27.3
92.0
29.6
33


Reviews:
[{"review_id": "4saRzb1dhS", "reviewer": "Reviewer_aagQ", "summary": "This paper proposes a scalable synthetic framework for benchmarking video MLLMs. It decouples video content from query-response pairs and separates different aspects of video understanding skills. The authors construct a synthetic video benchmark with tasks like retrieval, ordering, and counting to evaluate video models' temporal perception, chronological order, and spatio-temporal coherence. The experiment on VNBench reveals performance gaps and weaknesses in current video models.", "questions": "Please see weakness.", "limitations": "", "rating": 6, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "1. VideoNIAH offers a novel approach to evaluating video models by decoupling video content and query-response pairs.\n\n2. VNBench provides a scalable and flexible way to evaluate video understanding across different dimensions and video lengths, overcoming some limitations of traditional benchmarks.\n\n3. The paper conducts a thorough evaluation of multiple models, analyzing various factors such as haystack length, needle number, and model settings, providing valuable insights for model improvement.", "weaknesses": "1. While VNBench assesses important aspects of video understanding, it may not cover all possible video-related tasks and scenarios.\n\n2. Despite the relatively low cost of the evaluation method proposed in this paper, it is vulnerable to being gamed by models. Models may develop strategies to perform well on the tasks without truly understanding the video content, thus compromising the reliability of the evaluation.\n\n3. The overall tasks in the proposed evaluation framework are relatively simplistic. After targeted training, models can achieve high scores with relative ease, which may not accurately reflect their ability to handle more complex and diverse real-world video understanding scenarios. This simplicity could lead to an overestimation of the models' capabilities and may not provide a comprehensive assessment of their true understanding and generalization ability in practical applications.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "FWj96kZdja", "reviewer": "Reviewer_FQGX", "summary": "This paper introduces Video Needle-In-a-Haystack (VideoNIAH), a framework for synthetic video generation that utilizes intra-frame and inter-frame edits to create specific modifications within video segments. These modifications, referred to as \"needles,\" are used to assess the capabilities of video MLLMs in handling challenging video content. Additionally, the authors present VNBench, a benchmark compiled from these synthetic videos, which evaluates video MLLMs across three video understanding tasks: retrieval, ordering, and counting. The benchmark is tested on 12 MLLMs, both proprietary and open-source.", "questions": "- **Counting E2 Task:** What factors might explain the poor performance of Gemini and GPT-4 on the counting E2 task (Figure 5)?\n- **Consistency Check:** Could the authors verify the data in Figure 5 and Table 2? For instance, Video-LLaVa and VideoChat2 seem to perform better than Gemini on the counting E2 task in Figure 5, but Table 2 suggests otherwise.\n- **Haystack Length Variation:** In Section 5, where the \"Effect of Needle Position\" is discussed, the authors mention that the haystack is fixed. Could the authors clarify how the length of the haystack is adjusted in this scenario?", "limitations": "", "rating": 6, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "- The paper presents a framework to create a synthetic (i.e.: edited video) video that can be automated based on certain predetermined rules and tasks. These edits are posed as “needles” in a “haystack” (the original video segment), and the query for the MLLM is based on the inserted needles. The proposed method is easily scalable given the minimal labeling/definition process that needs to be set manually.\n- The curated VNBench benchmark can be used for evaluation of MLLM in video understanding, primarily spatial and temporal understanding. This is presented through tasks of retrieval, counting and ordering based queries from the curated video benchmark.\n- The paper evaluates 12 MLLMs on the proposed benchmark with extensive evaluation on the “needle” placements and difficulty.", "weaknesses": "- The rule-based automation process for dataset curation is unclear. For example, Figure 1(b) lacks clarity in illustrating this process, which is essential to understanding the scalability benefits of this approach, which can be improved.\n- While the method is scalable and captures the temporal dependency well, the scope of video understanding through this framework seems limited. For instance, the ability of the model in understanding actions, fine-grained action recognition cannot be ascertained, especially since MLLMs are used in this context extensively.\n- Given the nature of the benchmark, the sampling strategy of the video MLLM I believe plays a significant role in its performance. For instance, if any of the frames with a “needle” is not sampled, the model fails to correctly answer that query. Clarification on how this issue is addressed, or suggested solutions, would strengthen the framework’s reliability.\n- Can the authors explain if each cell in Figure 8 corresponds to a single test sample? It would be better to enhance this image with a heat map to better understand how the depth of the needle impacts the MLLMs performance. In addition, is there a reason that a depth higher than 10% was not considered?\n- Section 6: The mention of the \"training recipe\" lacks context on what is being trained, creating some confusion. A clearer introduction to this section would aid comprehension.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "JYlxpLIkeS", "reviewer": "Reviewer_ozpA", "summary": "This paper proposed a synthetic framework called VideoNIAH to construct the benchmark for evaluating video MLLMs. This framework aims to decouple video content from query-response pairs by corrupting the original videos with \"needles\" that are irrelevant to the videos.  Based on this frame, a video benchmark named VNBench is further built to evaluate different capabilities of proprietary and open-source video MLLMs, such as temporal perception, chronological ordering, and spatio-temporal coherence.", "questions": "1. The paper mentioned the used 'needles' are not related to the original videos. Did you conduct the similarity calculation between each video frame and the candidate \"needle\" before inserting them? \n2. Inserting the unrelated visual \"needle\" may not make the benchmark very challenging. Did you try the \"needles\" that are similar to the original video frame to a certain extent but are from a different category or situation? Maybe it will make the benchmark more challenging.", "limitations": "", "rating": 6, "confidence": 3, "soundness": 3, "presentation": 2, "contribution": 2, "strengths": "1. The proposed VideoNIAH is a simple and flexible framework for constructing the video benchmark by avoiding expensive human annotation and leveraging existing video datasets. \n2. The VNBench seems to be the first synthetic video benchmark for evaluating video MLLMs through three skills including retrieval,\nordering and counting. \n3. Comprehensive evaluation experiments are conducted to evaluate close- and open-source MLLMs, further giving some insights for improving the model development and training.", "weaknesses": "1. Although the authors mentioned the proposed framework is inspired by language model evaluation. It is still not very clear about the motivation for proposing such a synthetic framework for video benchmarking and why the proposed one is necessary or important. \n2. The proposed framework mainly adopts subtitles and unrelated images as the \"needle\", it is better to discuss whether there are other types of \"needle\" that can be used. \n3. A small number of video samples and the types of selected video datasets are used to build the benchmark. In addition, only three tasks are adopted to evaluate the video MLLMs. It is better to make the benchmark more scalable and diverse with comprehensive evaluation capabilities.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "IGhGImM7oQ", "reviewer": "Reviewer_9VkG", "summary": "This paper introduces VideoNIAH, a novel framework for constructing scalable synthetic benchmarks for evaluating video multimodal large language models (MLLMs).  The core idea is to inject unrelated visual or textual \"needles\" into existing videos (\"haystacks\") and then pose questions designed to test specific video understanding skills, such as temporal perception, chronological ordering, and spatio-temporal coherence.  The authors create VNBench, a benchmark based on VideoNIAH, and evaluate 12 proprietary and open-source video MLLMs, revealing significant performance gaps, particularly in long-dependency tasks.  They also conduct a detailed analysis of model performance across various factors like video length, needle number, and position.", "questions": "1. How does the performance on VNBench correlate with performance on established real-world video understanding benchmarks?  Can you provide a comparative analysis?\n2. Have you explored the sensitivity of the results to the CLIP similarity threshold used for needle filtering?\n3. How robust are the results to variations in the types and complexities of the \"needles\"?\n4. Could the VideoNIAH framework be adapted to evaluate other multimodal tasks beyond video understanding?", "limitations": "", "rating": 5, "confidence": 4, "soundness": 2, "presentation": 2, "contribution": 3, "strengths": "1. In VideoNIAH, the decoupling of video content from query-response pairs addresses a limitation of existing real-world video benchmarks, namely the high cost of data creation and the difficulty in isolating specific skills.  The synthetic approach mitigates data leakage concerns. The framework's design allows for easy generation of diverse and large-scale benchmarks, adaptable to various video lengths and complexities.  \n2. The authors evaluate a wide range of models (both proprietary and open-source), providing a comprehensive comparison of their performance on different tasks and under varying conditions. The analysis goes beyond simple performance metrics, exploring the impact of various factors (video length, needle number, position, etc.) on model accuracy.  This provides valuable insights into the strengths and weaknesses of current video MLLMs.", "weaknesses": "1. While the synthetic nature is a strength in terms of scalability and avoiding data leakage, it also limits the ecological validity of the benchmark.  Performance on VNBench doesn't guarantee real-world performance. The authors acknowledge this, but a correlation computation between the given benchmark and other real-world ones is needed.\n2. The reliance on CLIP for filtering needle images introduces a potential bias.  The performance of the CLIP model itself could influence the results, especially on synthetic videos.  A sensitivity analysis or exploration of alternative filtering methods would be valuable.\n3. Minor issues\n- Wrong single quotation marks in L18, P1.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (Poster)