IV-Bench: A Benchmark for Image-Grounded Video Perception and Reasoning in Multimodal LLMs
Authors: David Ma
Yuanxing Zhang
JinCheng Ren
Jiawei Guo
Yifan Yao
Zhenlin Wei
Zhenzhu Yang
Zhongyuan Peng
Boyu Feng
Jun Ma
顾潇
King Zhu
Zhoufutu Wen
Yancheng He
Meng Cao
Wangchunshu Zhou
Shiwen Ni
Jiaheng Liu
Wenhao Huang
Ge Zhang
Xiaojie Jin
Venue: ICLR 2026 Poster (2026)
Keywords: Image-Grounded Video Perception and Reasoning
Multimodal llms
Benchmark
Primary area: datasets and benchmarks
TL;DR: This paper introduces IV-Bench, the first comprehensive benchmark for evaluating Multimodal Large Language Models on image-grounded video perception and reasoning.

Abstract: Current benchmarks for Multimodal Large Language Models (MLLMs) predominantly rely on text-only queries, overlooking the essential role of images as visual context for enhancing video comprehension and facilitating natural human-AI interaction. To bridge this gap, we introduce \textbf{IV-Bench}, the first comprehensive benchmark for evaluating MLLMs on Image-Grounded Video Perception and Reasoning. IV-Bench comprises 966 videos paired with 2,560 meticulously annotated image-text queries across 13 tasks (7 perception and 6 reasoning tasks) spanning 5 distinct categories. We extensively evaluate state-of-the-art MLLMs, including open-source models (e.g., InternVL2.5, Qwen2.5-VL) and closed-source models (e.g., GPT-4o, Gemini2.0 series), revealing substantial performance gaps, with the best-performing model achieving only 28.9\% accuracy. Ablation studies demonstrate that incorporating images significantly enhances video understanding and highlight key model design factors influencing performance. Our findings provide valuable insights and guidance for future research. The code and dataset are available at \url{https://github.com/multimodal-art-projection/IV-Bench}.

Published as a conference paper at ICLR 2026
IV-BENCH: A BENCHMARK FOR IMAGE-GROUNDED VIDEO PERCEPTION AND REASONING IN MULTI- MODAL LLMS
David Ma'; Yuanxing Zhang?; Jincheng , Jiawei Guo', Yifan Yao", Zhenlin Wei Zhenzhu Yang'. Zhongyuan Peng', Boyu Feng' Jun Ma', Xiao Gu', Zhoufutu Wen', King Zhu' Yancheng He Meng Cao², Shiwen NI45, Jiaheng Liu Wenhao Huang
Ge Zhang'; 1 Xiaojie Jin
M-A-P Kuuiihoo Technology
MBZUAI
4 SARR,SUAT
SSAT,CAS
NJU
Sports (97)
Film & Television (250)
Terio
Eet
Ph----mlti
Clanne
Keyt
Spertal Bal.
IntamW12.5-10
ViceoLLM63
Art. Pert.
Rey, Syet
Owenz 39L-220
(154)
(red
(6112)
Peeporiig
Coup.
Plrreeatio Merowariion (150B)
Constr OCn 2201
(214)
Lif
Knowledge (246)
Coust:
(d) (e/2) poccee
loon
Dat Evente
tent)
Summerry
Art. Chenge
(a) Video categories
(b) Task Distribution
(c) Model Performance
Figure 1: (a) Video Categories. IV-Bench includes videos spanning five representative categories, ensuring diverse topical coverage. (b) Task distribution in IV-Bench. IV-Bench consists of total of 13 tasks, which are categorized into two main types: 6 reasoning tasks and 7 perception tasks. (c) Model Performance on IV-Bench. All evaluated MLLMs exhibit limited performance on IV-Bench. Even on the best-pent best-performing task (Natural Language Inference), the highest achieved accuracy is merely 64.7%, with other tasks resulting substantially lower scores.
ABSTRACT
Current benchmarks for Multimodal Large Language Models (MLLMs) predom- inantly rely on text-only queries, overlooking the essential role of images as vi- sual context for enhancing video comprehension and facilitating natural human- AI interaction. To bridge this gap, we introduce IV-Bench, the first compre- hensive benchmark for evaluating MLLMs on Image-Grounded Video Percep- tion and Reasoning. IV-Bench comprises 966 videos paired with 2.560 metic- ulously annotated image-text queries across 13 tasks (7 perception and rea- soning tasks) spanning 5 distinct categories. We extensively evaluate state-of- the-art MILMM, including open-source models (e.g., InternVi.2., Qwen2.5-VL) and closed-source models (e.g., GPT-4o, Gemini2.0 series), revealing substan- tial performance gaps, with the best-performing model achieving only 28.9% accuracy. Ablation studies demonstrate that incorporating images significantly enhances video understanding and highlight key model design factors influenc- ing performance. Our findings provide valuable insights and guidance for fu- ture research. The code and dataset are available at tibps://oge htgitthtttbu thub. com/ Bench
multimodal -rt--arretttetiiooooo
Equual contribution. Corresponding author.

---

Published as a conference paper at ICLR 2026
INTRODUCTION
Building upon the remarkable success of Large Language Models (LLMs) across various AI tasks (Young et al. 2024 Zhang et al.] 2024b). Multimodal Large Language Models (MLLMs) have recently demonstrated impressive capabilities in comprehending and integrating multimodal infor- mation from diverse sources such as text, images, and videos (Liu et al. 2023 Chen et al. 2024c
Guo et al., 2024). Consequently, numerous evaluation benchmarks Yue et al. 2024; Zhang et al,, 2024a Hu et al. 2025: Cheng et al. 2025 Wu et al. 2024a b Zhu et al.) 2024) have emerged to systematically assess their multimodal understanding. integration, and mask-solving capabilities.
Existing video benchmarks predominantly focus text-based queries, which fundamentally strug- gle to convey precise visual detail. Language is inherently too abstract for specifying fine-grained attributes (e.g., subtle design variations) identifying specific instances (e.g., person's face). This limitation is not just theoretical; it is underscored by the massive real-world adoption of visual search (Zhang et al., 2018). For instance, Pinterest's Lens was handling 600 million visual searches per month as early as 2018 (Zhai et al. 2017). and research shows 62% of young consumers desire visual search capabilities precisely because they excel where words fail (Dagan et al. (023). A more detailed discussion is provided in Appendix A.5 An external image query overcomes this ambiguity by providing a concrete visual anchor, a capability critical in applications from e-commerce to media analysis. We define this fundamental challenge as amage-grounder video perception and reasoning. Despite its proven importance and widespread applicability, this capability remains systematically overlooked in current evaluation frameworks.
To bridge this significant research gap, we propose IV-Bench, the first benchmark specifically de- signed for evaluating MLLMs in image-grounded video perception and reasoning. IV-Bench in- cludes 966 diverse videos paired with 2,560 meticulously annotated image-text queries distributed across 13 tasks (7 perception and 6 reasoning) and 5 categories: Knowledge, Film & Television, Sports Competition, Artistic Performance, and Life Record (see FigureTaa) Critically, IV-Bench's use of externally sourced reference images ensures that the evaluation transcends simple frame- matching task. This prevents information leakage and forces models to rely on generalized knowl- edge rather than perceptual similarity, offering a more rigorous and valid test of their real-world understanding.
Through extensive evaluations on 28 MLLMs-including 23 advanced open-source models like InterVLL2.5 (Chen et al. 2024b). Owen2.5- VL series (Bai et al. 2025). and 5 leading closed-source models such as GPT-4o (OpenAi 2024) and Gemini-2.0 series (Team et al., 20044- -we uncover substantial performance deficiencies. A significant disparity is evident between model and human capabilities, with the best models achieving only 28.9% accuracy, in stark contrast to the 88.8% achieved by humans. These limitations become even more pronounced in complex reasoning tasks such as Temporal Reasoning. Our analysis provides key insights to guide future research. Ablation studies reveal that the ability utilize visual context is critically dependent on model scale, with larger models showing significant benefits while smaller models often fail to do so. We further dissect the impact of visual token allocation, demonstrating that performance is generally more sensitive to increasing the number of video frames than to enhancing resolution. These findings, combined with our discovery that placing the image query after video frames yields the best results, provide crucial insights for advancing the capabilities of MILMS on image-grounded video tasks.
In summary, our work has three-fold contributions:
1. We present IV-Bench, the first comprehensive benchmark for image-grounded video percep- tion and reasoning in MLLMs. IV-Bench comprises 966 videos paired with 2,560 meticulously annotated image-text queries, where the images, collected from external sources rather than ex- tracted from the videos themselves, provide the essential context required to accurately answer the queries. The dataset spans 5 major categories and covers 7 perception and 6 reasoning tasks, ensuring substantial diversity across various scenarios and task types. Moreover, ity control-one ensuring clarity, accuracy, and category labeling, and another confirming that both image and video are required answer correctly, ensuring the high quality of IV-Bench.
round qual-
2. We evaluate 28 ofate-of-the-art MLLMs, including the latest closed-source models (c.g., GPT- 4o and Gemini-2.0-Par and open-source models (e.g.. VL2.5 and Qwen2.5-Ve VL series). Our experiments reveal that current models perform poorly on IV-Bench, with the best model
2

---

Published as a conference paper at ICLR 2026
Existence
Query image
contents eQuery Images appear
Query Image
content
in the video?
is similarta to the one the cQuery Images? black
Second from right
D: First from left
red
B: First from the right
white C.blue
R yelloo
Perception Tasks
Spatial Relationship
Keyframe Extraction
Query Image
Where similar content from the <Query Images the frame video??
Query Image
At what Line does content similar that in the <Query Images appear the video?
Below
D: Right side
0:27
5:19
Upper left
C: Lower left
C: 0:06
Attribute Change
Space-Timm Computing
16
Query Image
player weartng the jersey shown in <Query Image> scores?
What happens
Query Image
mentioned the meaning of the <Query Images?
regarding
PHILA 9
Sad
A:14-16
B:8-11
2:23-25
B: 10s
E: 33s
Reasoning
Instruction Understunding
Tasks
Summary
Query Image
What is the content in the video made using similar from the <Query Images? A: Bracelet
Query Image
What the main story of the person In <Query
Images when apgear in the video?
D: Sun hat
A: He experienced sudden mutation...
Mashed polatoes
grill
criminal. f He senses danger, defeats police...
C: Bed
Figure 2: Representative examples from IV-Bench. Each sample consists of video paired with an image-text query, comprising query image and corresponding query text. The correct answer is marked in green, with relevant video frames also highlighted in green.
achieving only 28.9% overall securacy-lleati indicating an big research gap for enhancing the capablilities of MLLMs in this direction.
3. Our analysis provides key insights to guide future research. Ablation studies indicate that increas- ing frame number and video resolution positively affect performance. Moreover, larger models significantly benefit from image contexts presented after the video, while smaller models show minimal improvements.
IV-BENCH
2.1 OVERVIEW
IV-Bench is designed to evaluate image-grounded video perception and reasoning, aiming to assess the capabilities of MLLMs in utilizing external visual cues for localization, reasoning, and com- prehension of video content. IV-Bench consists of 966 diverse videos paired with 2,560 image-text queries, with each image providing indispensable contextual cues necessary for correctly answering the queries. The dataset spans distinct tasks. Key features of IV-Bench include:
Image-Tr Text Queries. For each video, design multiple image-text queries. Each query in- cludes an externally sourced image-no not extracted from the video itself- -nd an associated textual question. These externally sourced images guarantee greater visual diversity and better simulate
3

---

Published as a conference paper at ICLR 2026
real-world usage scenarios, providing critical contextual cues necessary accurately answer the queries.
Diverse Video Categories. The dataset covers wide array of categories including Knowledge, Film & Television, Sports Competitions, Artistic Performances, and Life Records, ensuring exten- sive content diversity for various research purposes. Each video, with minimum duration of five minutes, provides sufficient depth for comprehensive analysis.
Diverse Evaluation Tasks. As illustrated in Figure[]bb, IV-Bench offers 13 distinct evaluation tasks grouped into perception and reasoning categories. These tasks comprehensively assess the capability of MILMs perform srageegrounded video perception and reasoning, spanning diverse set of perceptual and reasoning skills.
2.2 TASK DEFINITIONS
IV-Bench comprises 7 perception tasks and 6 reasoning tasks, with representative examples of se- lected tasks shown in Figure 2 Additional examples of the remaining tasks are provided in Figure These tasks address various aspects of image-gronn grooudegd video comprehension, spanning from basic perception to complex reasoning. Detailed descriptions of each task are presented below.
Perception Tasks: These tasks evaluate the model's capability directly extract and interpret fun- damental visual information from the video. They primarily focus on recognizing objects, people, scenes, and spatial relationships by leveraging contextual cues from the reference image. In essence, perception tasks assess the model's 's ability to accurately 'see" and identify content within the video. 1) Existence: Identify which objects or people in the reference image appear in the video. 2) Reverse Existence: Identify objects or people present in the image but absent in the video. 3) Natural Lan- guage Inference: Determine which scenes in the video are similar to a specific scene in the image. 4) Spatial Relationship: Identify absolute or relative spatial relationships among objects people in the video, grounded by the reference image. 5) Keyframe Extraction: Identify precise timestamps or segments within the video where objects people depicted in the image appear. 6) Constrained OCR: Recognize text-based content in the video, constrained by conditions explicitly defined by the reference image, such spatial alignment, temporal correspondence, and semantic relevance. 7) Detailed Events: Identify specific events or actions within the video directly related to content depicted in the reference image.
These tasks require models to engage in higher-order cognitive functions by integrating visual cues with contextual and temporal information. They assess the model's capac- ity to analyze, synthesize, and infer meaningful conclusions beyond simple visual recognition. 7) Counting: Count occurrences of person, object, or action depicted in the video, grounded by the reference image. 2) Space- Time Computing: Calculate event durations or distances between ob- jects/people in the video, using the image as contextual guidance. 3) Summary: Generate brief description summarizing a person, object, or event depicted in the video, informed by the reference image. 4) Instruction Understanding: Understand the functionality or creation process of objects de- picted in the video. guided by the reference image. 5) Attribute Change: Detect changes in attributes (e.g.. clothing, size, color) of objects or people throughout the video, referenced by the image. 6) Temporal Reasoning: Infer precise start and end timestamps of target events using temporal cues and world knowledge, as introduced in (Huang et al. 2024).
Reasoning Tasks:
2.3 ANNOTATION AND QUALITY CONTROL
The high quality of IV-Bench is ensured by rigorous, two-stage quality control process. Initially, annotators select external images from non-video sources and formulate questions with multiple plausible distractors. The first quality control round verifies fundamental clarity and accuracy. The second round is more stringent: we eliminate samples solvable without the image and, critically, ensure each question includes least two "effective" distractors. These distractors designed to be plausible yet incorrect, but would become correct answers if paired with a different image, thus confirming the necessity of the provided image for accurate reasoning.
4

---

Published as a conference paper at ICLR 2026
2.3.1 ANNOTATION PROCESS
Video Collection: A total of 966 videos, each exceeding five minutes in length, are carefully se- lected to ensure broad topical coverage. These videos span five distinct categories: Knowledge, Film & Television, Sports, Artistic Performances, and Life Records, providing diverse content suitable for various research applications.
Task Assignment: Annotators first watch each video in its entirety before performing image and text annotations. They then assign an appropriate task type from 13 predefined categories. This initial task assignment guides subsequent annotations, ensuring alignment with task-specific requirements.
Image and Question Annotation: After task assignment, annotators manually retrieve relevant external image from online sources, explicitly ensuring it is not extracted from the video itself. The selected image must be closely related to specific keywords, individuals, or themes present in the video. Annotators then formulate a text question leveraging both the video content and contextual cues provided by the image. The diversity of these external images is presented in Appendix AA.6 and their necessity for the task is justified in Appendix A.7
Answer and Distractor Design: Annotators craft the correct answer by carefully analyzing mul- timodal information from both the video and the image. Additionally, they generate up to nine plausible yet incorrect distractors to increase the question's 's difficulty while ensuring contextual rel- evance. For certain questions, fewer distractors may be provided depending on content constraints.
2.3.2 QUALITY CONTROL
Ensuring the quality and consistency of the data crucial for creating a reliable dataset. Our qual- ity control process is conducted in two main rounds (see Appendix A.2 for more detail of quality control):
The first round quality check focuses the structure and content standardization of evaluation questions. It mainly checks whether the query and options are clearly described, whether the answer is correct, and whether the distractors can effectively mislead test- takers. So we verify the clarity, precision, and unambiguity of each question. We also confirm that the correct answers and distractors are both plausible and contextually relevant, ensuring each query can be accurately answered based on the provided video and image. Furthermore, we check task categorization accuracy, correcting any misclassifccation maintain consistency across all 13 predefined tasks.
First Round Quality Control:
Since some questions can be answered using only common sense or video content-we conduct second round of quality check. During this phase, any query that can resolved without the reference image or video is simply removed, and any text query that inadvertently reveals visual content is rewritten to eliminate leakage. We also pinpoint ineffective distractors-tth those easily dismissed using video alone- and manually introduce least two effective distractors per question; These distractors are crafted so that, although incorrect for the current image, they would serve as the correct answers to alternative questions sharing the same text query but paired with different image teeereyy ensuring that the image is necessary for each sample in IV-Bench.
Second Round Quality Control:
COMPARISON WITH OTHER VIDEO BENCHMARKS
As shown in Table existing approaches can be broadly categorized into two groups: benchmarks with text-only queries and those with combined image-text queries. Benchmarks with text-only queries primarily emphasize evaluating video understanding guided solely by textual instructions. Existing benchmarks- from short-video suites like MMBench-Videe (Fang al. (025) and Video- Bench (Ning et al., (023) to long-range collections such as Long Bong (Wu et al. (025a),, MLVU (Zhou al. 2024), and Video-MME (Hu et al. 2025) span varied durations and modal- ities but none employ image-text queries, limiting their ability to evaluate perception and reasoning.
video
Although V2P-Bench (Zhao et al. 2025) and VideoRefer- Bencch Yuuan et al. 2025) both employ image-text queries, all images sourced from within the videos themselves; nonetheless, relying
5

---

Published as a conference paper at ICLR 2026
Table 1: A comparison of representative video benchmarks. ImgSre, ImgNec, and VidNec are abbreviations for Image Source, Image Necessity, and Video Necessity, respectively.
Query Modality
Benchmark
AVideos
Duration. (s)
Flasks
4QA Pairs
Anno.
#Ag Opt.
ImgSre
IngNee VidNec
MMRench-Videp
609
165
26
1,938
M
Video Bench
Viden-Bench
5,917
56
10
17,036
M& A
EguSchemaa
5,063
180
5,063
M&A
AutoEval Video
Video
TemppCongan
M&A
LongViil-Beench
17
6,678
MLVU
1,730
930
3,102
M & A
4
Video MME
900
1,01999
12
2,700
M
Textelmage
VideoReter Benehh?
Video Vi-idoo
Tost-Emage
V2P-Rench
Text-ollyTeut--nnee
Videx-MMMMU
Out-of-Viddo
Texi+Image
IV-Bench
966
13 2,560
Out-of-Video
solely on frame-extracted visuals constrains content diversity and real-world applicability. Video- MMMU (Hu et al. 2025 is the only video-QA benchmark employing external images, but there the video serves merely y as an auxiliary to boost image-only or text-only performance and isn' required answer the questions. In contrast, IV-Bench is the first manually annotated image-grounded video perception and reasoning benchmark: each query pairs an external image with its corresponding video-both of which are indispensable for deriving the correct answer. Refer to the Appendix for illustrative examples that compare IV-Bench and VideoMMMU.
4 EXPERIMENTS
This section evaluates several representative MLLMs on IV-Bench. We begin by outlining the mod- els and experimental setup, then analyze the performance of the 28 MLLMs in IV-Bench, and finally examine how inference patterns and how the number of visual tokens influences results.
4.1 SETTINGS
We evaluate 5 commercial models: Doubao-1.5- -.risson--o (Doubao Team 2025
Claude-3.7
sonnet (Anthropic 2025). GPT-40 (OpenAi 2024), Gemini 2 Flash (Team et al. [2024) and Gemini 2 Pro (Team et al. 2024). For open-source models, we select 23 representative MLLMs, includ- ing the Qwen2.5-VV VL series, IntermV2..5- L2.5-VVL series, and VideoLLaMA3 (Zhang et al. 2025a). We employ uniform sampling strategy process video frames. For all models, we uniformly set the frame number to 32. The default evaluation input format is 'video frames + image + question' with prompts, indicating that video frames are provided first, followed by the image. Since test samples in IV-Bench are multiple-choice questions, accuracy is computed by directly matching the model's output to the correct answer.
4.2 MAIN RESULTS
The performance across the 13 IV-Bench tasks-7 perception tasks and 6 reasoning tasks pre- sented in Table 2 We report individual task performances along with average results for perception tasks (P-Avg) and reasoning tasks (R-Avg). From the results, we derive two primary conclusions:
Image-Grounded video perception and reasoning is Challenging: In stark contrast to human per- formance, which achieves average accuracy of 88.8% (see Appendix A.8), effectively perform- ing image-grounded video perception and reasoning continues to pose significant challenges for MLLMs. For instance, among models under 10B parameters, the top-performing Qwen2.5-VL-7B achieves merely 18.5% accuracy, whereas random guessing would yield 11.11%. Even larger mod- els such Intern VLB and Qwen2.5-VL-77) achieve just 28.6% and 28.9%, respectively. Moreover, the best-performing commercial model, Gemini-2.0-Pro,, reaches only 27.7%. This vast gap to human-level performance underscores the significant untapped potential in leveraging im- age contexts to improve video comprehension. Overall, perception tasks generally prove easier for models than reasoning tasks, though both remain far from human proficiency. For example, Intern Intem VL2.5-78B achieves perception task average (P-Avg) of 33.4%, compared to reasoning task average (R-Avg) of only 21.9%. Even Gemini-2-Pro which performs relatively better on rea- soning tasks, manages only 24.9%. Temporal reasoning remains especially challenging, with the

---

Published as a conference paper at ICLR 2026
Table 2: The performance of MLLMS on IV-Bench across 13 tasks, comprising perception tasks and 6 reasoning tasks. For each task, the best performance is indicated in bold, and the second- best performance is indicated with underlining. Note that 'P-Avg' and 'R-Avg' denote the average results on perception and reasoning tasks, respectively.
Models
Overall
Perception KF
Reasoning STC
Exiet.
P-Aug
Cit.
Sum.
R--Agg
98.1
Open
Hamaviil
13.3
10.6
12.0
9.5
MalVA-Min
20250
14.0
MAmnoTH
20144
3.3
4.7
3.0
10.5
12.7
13.1
14.5
5.2
15.0
11.7
NVILA
209210
2.4
17.4
2.6
13.2
17.3
14.7
3.8
17.1
11.5
18.0
15.9
15.0
Pai 11500 (Mlrooous 1024
15.2
5.2
12.9
10.8
25.0
16.1
13.6
14.8
Pti-4-mllimobil
A014
16.6
11.
12.5
9.8
12.5
21.2
17.
10.0
170
135
14.8
20/9
16.3
2.8
14.
5.9
27.5
12.8
24.2
15.7
18.
17.5
156
14.5
14.7
20.3
12.3
ImemmV12-48 (Dendiiie
16.3
7.1
106
19.0
25.5
12.2
23.9
22.5
17.1
18.
2066
6.7
14.5
14.7
17.
16.3
YAMRA Ren
16.9
13.7
11.7
13.7
26.1
12.9
27.9
23.0
17.
15.
214
12.
136
133
16.5
15.5
Miniqui-r Lem00010 2004
17.2
7.1
15.5
9.2
30.1
14.3
52.5
22.5
18.6
15.7
159
122
15.0
13.3
17.5
15.3
Minecpm-
14.8
8.5
14.6
51.2
18.4
13.1
135
173
15.2
VadzoLLaMat
N05
17.
11.9
136
17.7
26.1
1644
28.9
20.0
19.0
16.5
15.1
140
17.5
13.7
14.9
1memV12..588
Charera
17.4
5.4
15.7
16.4
76.
10.1
$1.4
25.3
17.8
15.7
22.1
14.1
16.3
20.6
16.5
Qwan2.5-VL-78 Parreaa MIPN
18.5
8.1
13.6
21.6
76.8
15.7
S1.7
18.5
18.9
16.5
25.0
15.4
17.5
24.1
17.9
Opon Source MLLMN's 108)
Adaa Lictal 2124h
17.4
9.5
9.5
11.8
24.8
17.4
36.7
20.5
18.6
16.9
16.7
15.6
14.0
18.9
13.7
15.8
Intem 1..5-200
20.6
12.8
14.4
33.3
36
13.6
$2.5
24.5
22.4
15.0
23.0
14.4
14.0
25.2
19.8
18.1
11aWA-OreWiiinn-71
02.3
15.6
18.9
40.5
78.8
20.1
$3.9
27.0
25.3
17.6
17.5
189
15.3
21.1
20.5
18.1
Qwen2 --V1-128
23.0
19.3
13.8
28.8
31.6
13.8
43.7
27.1
24.9
20.5
23.0
12.3
14.7
25.4
23.6
20.2
1acmNV12..-888
26.6
24.6
19.3
56.9
34.00
19.5
38.8
33.5
30.4
20.2
22.2
6.7
22.0
27.3
22.6
21.1
28.6
28.0
20.5
50.1
43.1
19.5
39.5
40.5
33.4
21.4
26.2
12.2
13.1
29.4
27.8
21.9
Quaa2.-VV.-7288
28.9
38.9
13.3
64.7
34.9
16.0
99.4
35.0
33.7
17.9
22.2
10.1
15.7
30.5
32.4
21.9
Closed Scoree MCLAMs
Dalhus Swyieeee Dadler Tram 3025
20.1
Chadl1 7.ammm Antresua
33.0
17.6
12.5
11.2
15.1
OpenA)
21.5
11.8
23.4
33.8
16.4
23.6
19.7
top- pop-perfomming model achieving only 16.7% accuracy. Conversely, NLI and existence tasks appear relatively easier, with Qwen2.5-VL-722 scoring 38.9% and 64.7%, respectively, yet these scores still highlight the profound difficulty for models compared to humans.
Moderate Gains with Larger Models: Increasing model scale results in modest performance im- provements across various tasks. For example, in the V1.2.5 series, the 8B model achieves 17.4% accuracy, while the larger 26B, 38B, and 78B variants reach 20.6%, 26.6%, and 28.6%, respectively. Similarly, for the Qwen2.5-VL series, scaling from the 7B model (18.5%) to the 72B model (28.9%) results in noticeable performance gains. This improvement is particularly pro- nounced few perception-bas based tasks. For example, Qwen2.5-VV VL-72B demonstrates a substantial 30.8% improvement over Qwen2.5-VLL-2 VL-7BB on existence tasks and a notable 43.1% gain on NLI tasks. However, the benefits of scaling are notably smaller for reasoning-intent ample, counting tasks, W surpasses Qwen2.5-VL-77 by merely 1.4%, while IntemVLL..5-788 exhibits just a 5.7% improvement over Intern Intern LL2.5-8.. The limited benefits ob- served on reasoning crasoning-ittensive intensive tasks may be attributed to the fact that increasing model size tends to enhance memorization and shallow pattern recognition more significantly than improves reasoning ability. More detail can be seen in Appendix A.
tasks. For ex-
The experimental results confirm that amage-grounded video perception and reasoning continue to pose significant challenges for MILMs, especially for beasoning-intensin tasks. Furthermore, in- creasing model size yields only moderate improvements, suggesting that merely scaling models is insufficient to fully overcome these challenges. Future research should prioritize developing spe- cialized mechanisms for video reasoning, such as enhanced temporal modeling techniques.
4.3 THE IMPACT OF INFERENCE PATTERN
In this section, comparatively analyze performance across three inference patterns: image-first, video-first (both under image-text query settings), and text-only queries (without image input). We evaluate both the MiniCPM-v/o and InternV22. model series, with results presented in Figure (]).
Smaller models are less proficient in image-grounded video perception and reasoning. MiniCPM-v/o achieve its highest scores under the text-only setting, whereas incorporating im-

---

Published as a conference paper at ICLR 2026
30
Video
32-
HArrAZZ 205
27
Accuracy (%)
Accuracy
22
5-
MCPMV MCPMo
IVL-88
IVL-26B
IVL-38B
IVL-78B
12-
16
32
64
(a)
(b)
Figure 3: Comparison of model performance: (a) across different inference patterns and (b) with varying numbers of frames. MCPMv/o represent MiniCPMv/o, IVL is the abbreviation of Intern
Owee G4--3 216-19
27
30
27
Accuracy (96)
Accuracy (96)
24
21
21
18-
15
72p
108p
144p
240p
15
9/720p
18'480p
32'360p
64/240p 128r144p 25e-108p 512'72p
(a)
(b)
Figure 4: Comparison of model performance: (a) across different video resolutions and (b) across various trame-resolutee combinations.
ages whether before after the video-leads to drop in performance. This suggests that lack of sufficient image-grou grounddd video training data, combined with limited model capacity, ren- ders visual context unhelpful even detrimental for smaller architectures. Intern V1.2.588B likewise shows only marginal gains when images are added, reinforcing this trend.
For larger models, placing the image after video frames results in superior performance com- pared to positioning it beforehand. Across InternVL2.5-V variants of 26B, 38B, and 78B pa- rameters, observe steadily improving results when the image is appended after the sequence of video frames. We hypothesize that images presented early in the token stream are more easily 'for- gotten' amid subsequent video tokens, leading under-utiiizatin of crucial visual information phenomenon similarly reported in (Ma et al. 2024). More analysis can be seen in Appendix A.
4.4 ANALYSIS OF THE NUMBER OF VISUAL TOKENS
This section addresses two key questions regarding the number of visual tokens supplied to multi- modal lasion-language models:
Scaling Effect. How does model performance change as we increase the total number of visual tokens-bh by varying either frame number or resolution?
Token Allocation. When the total number of visual tokens is held constant, does allocating tokens more frames or higher resolution produce greater gains?
8

---

Published as a conference paper at ICLR 2026
4.4.1 ISOLATED SCALING OF FRAME NUMBER AND RESOLUTION
In this subsection, we analyze how allocating additional visual tokens- both temporally (by increas- ing frame number) and spatially (by raising resolution)-impacts model performance. We conduct experiments on six top-performing models Aria, MiniCPM-O, JntemWL-88B/66B, and Qwen2.5- VL-7B/72B under two settings. We find:
Model performance consistently improves frame number increases. We vary frame number E {8, 16, 32, 64) at a fixed resolution (results in Figure (()).. Across all six models, increasing the number of frames steadily boosts accuracy, demonstrating that additional temporal visual tokens enhance image-groundes video perception and reasoning.
Performance consistently improves as resolution increases-most markedly at low resolutions, though gains diminish at higher resolutions. We fix frame number at 32 and vary resolution (72p, 108p, 144p, 240p) (results in Figure 4a)). When varying spatial detail, performance im- proves most between 72p and 144p, but further upgrades to 240p yield marginal gains, indicating diminishing returns beyond a mid-range resolution.
4.4.2 TOKEN ALLOCATION: FRAMES vs. RESOLUTION
To disentangle temporal and spatial contributions under fixed budget of visual tokens, we evalu- ate seven frame-ree resolution pairs- (8, 720p), (16, 480p), (32, 360p). (64, 240p), (128, 144p), (256, 108p), and (512, (asp-chosen to yield roughly equivalent token number. Figure 4b) reveals con- trasting behaviors between model scales. For Qwen2.5-VL- VL-7B, performance rises primarily with frame number, while resolution plays a secondary role. In contrast, Qwen2.5-VLL-725 exhibits near- constant performance across all combinations, indicating its capacity to flexibly trade temporal for spatial information. These findings suggest that smaller models rely more on temporal cues to compensate for limited spatial encoding, whereas larger models extract complementary signals from both dimensions interchangeably.
5 RELATED WORK
5.1
MULTIMODAL LARGE LANGUAGE MODELS
Multimodal Large Language Models (MLLMs) fuse LLM backbones with visual encoders via visual
instruction tuning for richer multimodal understanding (Liu et al. [023 2024a Zhu et al.) 2023 Ma 2024c; Bai al. (025). In video, they leverage instruction datasets and 2024] Li el al. 2023) e.g.. Video-LLaMA's ViT+Q-Former (Zhang et al.,
al. 2024 Zhang et al.
projectors (Cheng et al.
2023) and LLaMA- Vid's frame compression et al., 2024d) and can handle diverse tasks from
captioning to temporal localization. However, despite training on massive image/video corpora (c.g., Qwen2.5-VL), they still lack targeted examples that ground video queries external image contexts, IV-Bench.
limiting their ability to resolve cross-source visual
5.2
VIDEO UNDERSTANDING BENCHMARKS
Recent benchmarks target temporal perception (Wu al. 2024). action understanding (Wang al.,
2023
Wu et al. 2024), and reasoning (Xiao et al. 2021), exemplified by MVBench for vidrt
QA (Li et al. 2024c). Long VideoBench's s hour-long long referring reasoning (Wu et al.) 2025a), Video- MME's 11 h multimodal analysis (Fu et al. 2024), and V2P-Bench's 980 videos with 1,172 visual-prompa QA pairs (Zhao et al.) 2025). However, these efforts focus only on intra-video signals and overlook the challenge of grounding video understanding in external image contexts. None, however, assess image-grounder video perception and reasoning. To fill this gap, we introduce IV- Bench the first benchmark dedicated to evaluating smage-grounder video perception and reasoning using images sourced externally from the videos themselves.
6 CONCLUSION
In this work, we introduce IV-Bench, the first benchmark explicitly designed to evaluate models on image-grounded video perception and reasoning tasks. Our extensive evaluation of both open-source
9

---

Published as a conference paper at ICLR 2026
and closed-source multimodal large language models reveals significant limitations, particularly in effectively leveraging image contexts for accurate video comprehension, notably in temporally sen- sitive perception and reasoning tasks. We observe that smaller models demonstrate limited capa- bility of image grounded video perception and reasoning. Furthermore, our analysis indicates that increasing the number of video frames generally has compared to enhancing video resolution. We hope IV-Bench will inspire future research to sub- stantially advance the capabilities of multimodal large language models in image-grounded video perception and reasoning.
pronounced impact performance
ETHICS STATEMENT
We have undertaken thorough review of the ethical considerations related the creation and deployment of the IV-Bench benchmark. Our ethical framework addresses three primary areas: data sourcing. potential biases, and human labor.
Data Sourcing. The videos and external images comprising IV-Bench are sourced from publicly available online platforms. We have taken deliberate steps to filter this content, ensuring its suit- ability for general research audience and excluding any material that is explicitly copyrighted, sensitive, or private. The use of this data is intended strictly for academic and non-commercial research purposes, in alignment with fair use principles.
Potential Biases. We acknowledge that any dataset curated from web-based sources may inherit biases present in the original content. IV-Bench may reflect cultural, demographic, or subject-matter skews inherent its sources. We transparently disclose this limitation alert downstream users to the potential fairness risks. Models that perform well on IV-Bench may not generalize equitably across all cultural contexts or populations, and we encourage researchers to consider these factors when interpreting results.
The development of IV-Bench involved human participation in two key stages: the classification of external images and the human performance evaluation. All annotators involved are experts with extensive experience in evaluating Multimodal Large Language Models (MLLMs) and complex visual reasoning tasks. They were fully informed about the scope and purpose of their tasks and provided their consent to be involved. All participants were compensated fairly for their expertise and time, accordance with ethical standards for academic research and to respectfully acknowledge their valuable contributions.
Human Labor.
REPRODUCIBILITY STATEMENT
We are fully committed to ensuring the reproducibility of our research. facilitate the verification of our findings and to encourage future work, we make the IV-Bench dataset and our code publicly available at httttts://
com muu iimotillalll- art -ot.oaaalarrrrrooooeeooooooooo/o itt.:eeeett////////eeeeeee
In the main body of our paper, particularly in the experimental setup section, we provide detailed description of our evaluation protocol. This includes key hyperparameters and settings required for faithful replication, such the number of sampled video frames (32), the input resolution (c.g., 480p), the specific input sequence used (video > image - text), and details of the models evaluated. We are confident that the public release of our benchmark and the comprehensive documentation of our methodology will allow the research community to easily reproduce our results and build upon contributions.
REFERENCES
Marah Abdin, Jyoti Aneja, Harkirat Behl, Sébastien Bubeck, Ronen Eldan, Suriya Gunasekar, Michael Harrison, Russell J Hewett, Mojan Javaheripi. Piero Kauffmann, al. Phi-4 techni- cal report. arXiv preprint arXiv:2412.08900 2024.
Anthropic. Claude 3.7 sonnet, 2025.
10

---

Published as a conference paper at ICLR 2026
Shuai Bai, Keqin Chen, Xuejing Liu, Jialin Wang, Wenbin Ge, Sibo Song, Kai Dang, Peng Wang, Shijie Wang, Jun Tang, et al. Qwen2. 5-vl technical report. arXiv preprint arXiv:2502.13923, 2025.
Daniel Bolya, Yao Huang, Peize Sun, Jang Hyun Cho, Andrea Madotto, Chen Wei, Tengyu Ma, Jiale Zhi, Jathushan Rajasegaran, Hanoona Rasheed, et al. Perception encoder: The best visual embeddings are not at the output of the network. arXiv preprint arXiv:2504. 13181, 2025.
Liang Chen, Haozhe Zhao, Tianyu Liu, Shuai Bai, Junyang Lin, Chang Zhou, and Baobao Chang- An image is worth 1/2 tokens after layer 2: Plug-and-play inference acceleration for large vision- language models. In European Conference on Computer Vision, pp. 19-35. Springer, 2024a.
Zhe Chen, Weiyun Wang, Yue Cao, Yangzhou Liu, Zhangwei Gao, Erfei Cui, Jinguo Zhu, Shen- glong Ye, Hao Tian, Zhaoyang Liu, et al. Expanding performance boundaries of open-source multimodal models with model, data, and test-time scaling. arXiv preprint arXiv:2412.05271, 2024b.
The Chen, Jiannan Wu, Wenhai Wang, Weijie Su, Guo Chen, Sen Xing, Muyan Zhong, Qinglong Zhang, Xizhou Zhu, Lewei Lu, et al. Internvl: Scaling up vision foundation models and aligning for generic visual-linguistic tasks. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pp. 24185 28198, 2024c.
Xianfu Cheng, Wei Zhang, Shiwei Zhang, Jian Yang, Xiangyuan Guan, Xianjie Wu, Xiang Li, Ge Zhang, Jiaheng Liu, Yuying Mai, et al. Simplevqa: Multimodal factuality evaluation for multimodal large language models. arXiv preprint arXiv:2502. 13059, 2025.
Zesen Cheng, Sicong Leng, Hang Zhang, Yifei Xin, Xin Li, Guanzheng Chen, Yongxin Zhu, Wenqi Zhang, Ziyang Luo, Deli Zhao, and Lidong Bing. Videollama 2: Advancing ppatial-temport modeling and audio understanding in video-llms. arXiv preprint arXiv:2406.0744 077447 2024.
Arnon Dagan, Ido Guy, and Slava Novgorodov. Shop by image: characterizing visual search in e-commerce. Information Retrieval Journal, 26(1):2, 2023.
Doubao Team. Doubao 1.5 pro, 2025.
Xinyu Fang, Kangrui Mao, Haodong Duan, Xiangyu Zhao, Yining Li, Dahua Lin, and Kai Chen. Mmbench-video: A long-form multi-shot benchmark for holistic video understanding. Advances in Neural Information Processing Systems, 17:89098-899 89124, 2025.
Zixun Fang, Wei Zhai, Aimin Su, Hongliang Song, Kai Zhu, Mao Wang, Yu Chen, Zhiheng Liu, Yang Cao, and Zueng-Jun Zha. Vivid: Video virtual try-on using diffusion models. arXiv preprint arXiv:2405. 1179, 2024.
Chaoyou Fu, Yuhan Dai, Yongdong Luo, Lei Li, Shuhuai Ren, Renrui Zhang, Zihan Wang, Chenyu Zhou, Yunhang Shen, Mengdan Zhang, Peixian Chen, Yanwei Li, Shaohui Lin, Sirui Zhao, Ke Li, Tong Xu, Xiawu Zheng, Enhong Chen, Rongrong Ji, and Xing Sun. Video-mme: The first-ever comprehensive evaluation benchmark of multi-modal 1lms in video analysis, 2024.
Jarvis Guo, Tuney Zheng, Yuelin Bai, Bo Li, Yubo Wang, King Zhu, Yizhi Li, Graham Neubig, Wenhu Chen, and Xiang Yue. Mammoth-vl: Eliciting multimodal reasoning with instruction tuning at scale. arXiv preprint arXiv.2412.05237, 2024.
Songhao Han, Wei Huang. Hairong Shi, Le Zhuo, Xiu Su, Shifeng Zhang, Xu Zhou, Xiaojuan Qi, Yue Liao, and Si Liu. Videoespresso: A large-scale chain-of-thought dataset for fine-grained video reasoning via core frame selection. In Proceedings of the Computer Vision and Pattern Recognition Conference, pp. (6181-26199) 2025.
Kairui Hu, Penghao Wu, Fanyi Pu, Wang Xiao, Yuanhan Zhang, Xiang Yue, Bo Li, and Ziwei Liu. Video- diloommmu:: Evaluating knowledge acquisition from multi-discipline professional videos. arXiv preprint arXiv:2501.1382 2025.
De-An Huang. Shijia Liao, Subhashree Radhakrishnan, Hongxu Yin, Pavlo Molchanov, Zhiding Yu, and Jan Kautz. Lita: Language instructed ence on Computer Vision pp. 202-218. Springer, 2024.
assistant. In European Confer-
11

---

Published as a conference paper at ICLR 2026
Zeyinzi Jiang, Zhen Han, Chaojie Mao, Jingfeng Zhang, Yulin Pan, and Yu Liu. Vace: All-in-one video creation and editing. arXiv preprint arXiv:2503.07598, 2025.
Bo Li, Yuanhan Zhang, Dong Guo, Renrui Zhang, Feng Li, Hao Zhang, Kaichen Zhang, Yanwei Li, Ziwei Liu, and Chunyuan Li. Llava-onevision: Easy visual task transfer, 2024a.
Dongxu Li, Yudong Liu, Haoning Wu, Yue Wang, Zhiqi Shen, Bowen Qu, Xinyao Niu, Guoyin Wang, Bei Chen, and Junnan Li. Aria: An open multimodal native mixture-of-experts model. arXiv preprint ar arXiv:2410.05990 2024b.
KunChang Li, Yinan He, Yi Wang, Yizhuo Li, Wenhai Wang, Ping Luo, Yali Wang, Limin Wang, and Yu Qiao. Videochat: Chat-centric video understanding. arXiv preprint MXXi::335500635: 2023.
Kunchang Li, Yali Wang, Yinan He, Yizhuo Li, Yi Wang, Yi Liu, Zun Wang, Jilan Xu, Guo Chen, Ping Luo, et al. Mybench: A comprehensive multi-modal video understanding benchmark. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp- 22195-222061 2024c.
Yanwei Li, Chengyao Wang, and Jiaya Jia. Llama-vid: An image is worth 2 tokens in large language models. In European Conference on Computer Vision, pp. 323-340, Springer, 2024d.
Jia.
Haotian Liu, Chunyuan Li, Qingyang Wu, and Yong Jae Lee. Visual instruction tuning. Advances in neural information processing systems, 36:34892-34916, 2023.
Haotian Liu, Chunyuan Li, Yuheng Li, Bo Li, Yuanhan Zhang, Sheng Shen, and Yong Jac Lee. Leava-next: Improved reasoning, ocr, and world knowledge, January 2024a.
Zhijian Liu, Ligeng Zhu, Baifeng Shi, Zhuoyang Zhang, Yuming Lou, Shang Yang, Haocheng Xi, Shiyi Cao, Yuxian Gu, Dacheng Li, al. Nvila: Efficient frontier visual language models. arXiv preprint arXiv:2412.04466 2024b.
Yongdong Luo, Xiawu Zheng, Xiao Yang, Guilin Li, Haojia Lin, Jinfa Huang, Jiayi Ji, Fei Chao, Jiebo Luo, and Rongrong Ji. Video-rag: Visually-aligned retrieval- laugmeted long video com- prehension. arXiv preprint arXiv:2411. 2024.
Fan Ma, Xiaojie Jin, Heng Wang, Yuchen Xian, Jiashi Feng, and Yi Yang. Vista-llama: Reducing hallucination in video language models via equal distance to visual tokens. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 13151-13160, 2024.
Ziyu Ma. Chenhui Gou, Hengcan Shi, Bin Sun, Shutao Li, Hamid Rezatofighi. and Jianfei Cai. Drvideo: Document retrieval based long video understanding. In Proceedings of the Computer Vision and Pattern Recognition Conference, pp.
2025.
Microsoft. Phi-3.5-mini-instrue. 2024.
Munan Ning, Bin Zhu, Yujia Xie, Bin Lin, Jiaxi Cui, Lu Yuan, Dongdong Chen, and Li Yuan. Video-hench: A comprehensive benchmark and toolkit for evaluating video-based large language models. arXiv preprint arXiv:2311.16103, 2023.
OpenAi. Gpt-4o, 2024.
OpenGVLab. Internv12-8b 8b, n.d. Hugging Face model.
Weiming Ren, Wentao Ma. Huan Yang, Cong Wei, Ge Zhang, and Wenhu Chen. Vamba: Under- standing hour-long videos with hybrid mamba-transformers. arXiv preprint 11X5 2025.
Xiaoqian Shen, Yunyang Xiong, Changsheng Zhao, Lemeng Wu, Jun Chen, Chenchen Zhu, Zechun Liu, Fanyi Xiao, Balakrishman Varadarajan, Florian Bordes, Zhuang Liu, Hu Xu, Hyun- woo J. Kim, Bilge Soran, Raghuraman Krishnamoorthi, Mohamed Elhoseiny, and Vikas Chan- dra. Longvu: Spatiotemporal adaptive compression for long video-language understanding. 1rXiv:2410.1744 17434, 2024.
12

---

Published as a conference paper at ICLR 2026
Shuai Tan, Biao Gong, Xiang Wang, Shiwei Zhang, Dandan Zheng, Ruobing Zheng, Kecheng Zheng, Jingdong Chen, and Ming Yang. Animate-x: x: Universal character image animation with enhanced motion representation. arXiv preprint X1v:24100110306 2024.
Gemini Team, Petko Georgiev, Ving lan Lei, Ryan Burnell, Libin Bai, Anmol Gulati, Garrett Tanzer, Damien Vincent, Zhufeng Pan, Shibo Wang, et al. Gemini 1.5: Unlocking multimodal under- standing across millions of tokens of context. arXiv preprint arXiv:2403.0553 0353030300,,,, 2024.
Omkar Thawakar, Muzammal Naseer, Rao Muhammad Anwer, Salman Khan, Michael Felsherg, Mubarak Shah, and Fahad Shahbaz Khan. Composed video retrieval via enriched context and discriminative embeddings. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 28896,, 2024.
Lucas Ventura, Antoine Yang, Cordelia Schmid, and Gül Varol. Covr: Learning composed video re- trieval from web video captions. In Proceedings of the AAAI Conference on Artificial Intelligence, pp. 5270-5279, 2024.
Peng Wang, Shuai Bai, Sinan Tan, Shijie Wang. Zhihao Fan, Jinze Bai, Keqin Chen, Xuejing Liu, Jialin Wang. Wenbin Ge, Yang Fan, Kai Dang. Mengfei Du, Xuancheng Ren, Rui Men, Dayiheng Liu, Chang Zhou, Jingren Zhou, and Junyang Lin. Qwen2-vl: Enhancing vision-language model's perception of the world at any resolution. arXiv preprint arXiv:2409.12199 2024.
Zhenhailong Wang, Ansel Blume, Sha Li, Genglin Liu, Jaemin Cho, Zineng Tang, Mohit Bansal, and Heng Ji. Paxion: Patching Action Knowledge in Video-Language Foundation Models. arXiv e-prints, art. arXiv:2305. 10683, May 2023, doi: 10.48550%axxxv.2255.6666
Bo Wu, Shoubin Yu, Zhenfang Chen, Joshua Tenenbaum, and Chuang Gan. STAR: Benchmark for Situated Reasoning in Real-World Videos. arXiv e-prints, art. arXiv:2405.09711, May 2024. doi: 10.48550/acxiv.22455.009 11..11.
Haoning Wu, Dongxu Li, Bei Chen, and Junnan Li. Longvideobench: A benchmark for long- long-context interleaved lideo-language understanding. Advances in Neural Information Processing Systems, 37:28828-2888 28857, 2025a.
Siwei Wu, Yizhi Li, Kang Zhu, Ge Zhang, Yiming Liang, Kaijing Ma, Chenghao Xiao, Haoran Zhang, Bohao Yang, Wenhu Chen, al. Scimmir: Benchmarking scientific multi-modal infor- mation retrieval. arXiv preprint arXiv:2401. 13478, 2024a.
Siwei Wu, Kang Zhu, Yu Bai, Yiming Liang, Yizhi Li, Haoning Wu, JH Liu, Ruibo Liu, Xing- wei Qu, Xuxin Cheng, al. Mmra: A benchmark for evaluating guaniigrannllritty and multi-image relational association capabilities in large visual language models. arXiv preprint arXiv:2407.17379 2024b.
Weijia Wu, Zeyu Zhu, and Mike Zheng Shou. Automated movie generation via multi-agent cot planning, arXiv preprint 1.Xiv:2503307314. 2025b.
Junbin Xiao, Xindi Shang, Angela Yao, and Tat-Seng Chua. NEXT-QA:Next Phase of Question- Answering to Explaining Temporal Actions. arXiv e-prints, art. arXiv:2105.08276, May 2021. doi: 10.48550/a2xv.21155.08
Zhongcong Xu, Jianfeng Zhang, Jun Hao Liew, Hanshu Yan, Jia- Wei Liu, Chenxu Zhang, Jiashi Feng, and Mike Zheng Shou. Magicanimate: Temporally consistent human image animation using diffusion model. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 1481-1490, 2024.
Zuhao Yang, Jiahui Zhang, Yingchen Yu, Shijian Lu, and Song Bai. Versatile transition generation with inage-to-video diffusion. arXiv preprint arXiv:2508. 0/6688 2025.
Yuan Yao, Tianyu Yu, Ao Zhang, Chongyi Wang, Junbo Cui, Hongji Zhu, Tianchi Cai, Haoyu Li, Weilin Zhao, Zhihui He, et al. Minicpm-v: A gpt-4v level milm on your phone. arXiv preprint arXiv:2408. 8000 2024.
13

---

Published as a conference paper at ICLR 2026
Alex Young, Bei Chen, Chao Li, Chengen Huang, Ge Zhang, Guanwei Zhang, Guoyin Wang, Heng Li, Jiangcheng Zhu, Jianqun Chen, et al. Yi: Open foundation models by 01. ai. ar preprint arXiv:2403.04662, 2024.
Yuqian Yuan. Hang Zhang, Wentong Li, Zesen Cheng, Boqiang Zhang, Long Li, Xin Li, Deli Zhao, Wengiao Zhang, Yueting Zhuang. Jianke Zhu, and Lidong Bing. Videorefer suite: Advancing spatial-temporal object understanding with video 11m, 2025. URL https: orrg
abs/2501. 00599
Xiang Yue, Yuansheng Ni, Kai Zhang, Tianyu Zheng, Ruoqi Liu, Ge Zhang, Samuel Stevens, Dongfu Jiang, Weiming Ren, Yuxuan Sun, et al. Mmmu: A massive multi-discipline multi- modal understanding and reasoning benchmark for expert agi. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 9556-9567, 2024.
Andrew Zhai, Dmitry Kislyuk, Yushi Jing, Michael Feng, Eric Tzeng, Jeff Donahue, Yue Li Du, and Trevor Darrell. Visual discovery at pinterest. Proceedings of the 26th international conference world wide web companion, pp. 515-524, 2017.
Boqiang Zhang, Kehan Li, Zesen Cheng, Zhiqiang Hu, Yuqian Yuan, Guanzheng Chen, Sicong Leng. Yuming Jiang, Hang Zhang, Xin Li, et al. Videollama 3: Frontier multimodal foundation models for image and video understanding. arXiv preprint arXiv:2501.13166, 2025a.
Ge Zhang, Xinrun Du, Bei Chen, Yiming Liang, Tongxu Luo, Tianyu Zheng, Kang Zhu, Yuyang Cheng. Chunpu Xu, Shuyue Guo, et al. Cmmmu: A chinese massive multi-discipline multimodal understanding benchmark. arXiv preprint arXiv:2401.11944 2024a.
Ge Zhang, Scott Qu, Jiaheng Liu, Chenchen Zhang, Chenghua Lin, Chou Leuang Yu, Danny Pan, Esther Cheng, Jie Liu, Qunshu Lin, al. Map-neo: Highly capable and transparent bilingual large language model series. arXiv preprint 2XXi::2405.19327 2024b.
Hang Zhang, Xin Li, and Lidong Bing. Video-llama: An instruction-tuned audio-visual language model for video understanding. arXiv preprint arXiv:2.306. 02858, 2023.
Haoji Zhang, Yiqin Wang, Yansong Tang, Yong Liu, Jiashi Feng, Jifeng Dai, and Xiaojie Jin. Flash-vstream: Memory-based real-time understanding for long video streams. arXiv preprint arXiv:2406.0mm 08085, 2024c.
Peiyuan Zhang, Kaichen Zhang. Bo Li, Guangtao Zeng, Jingkang Yang, Yuanhan Zhang, Ziyue Wang, Haoran Tan, Chunyuan Li, and Ziwei Liu. Long context transfer from language to vision. arXiv preprint 2xXx:24466./882 2024d.
Shaolei Zhang, Qingkai Fang, Zhe Yang, and Yang Feng. Llava-mini: Efficient image and video large multimodal models with one vision token. arXiv preprint arXiv:2501.03895 2025b.
Yanhao Zhang, Pan Pan, Yun Zheng, Kang Zhao, Yingya Zhang, Xiaofeng Ren, and Rong Jin. Visual search at alibaba. In Proceedings of the 24th ACM SIGKDD international conference on knowledge discovery & data mining, pp. 993 1001, 2018.
Yongheng Zhang, Xu Liu, Ruihan Tao, Qiguang Chen, Hao Fei. Wanxiang Che, and Libo Qin. Vit- cot: Video-text interleaved ffain-of-thougeee for boosting video understanding in large language models. arXiv preprint arXiv:2507 09876, 2025c.
Yiming Zhao, Yu Zeng. Yukun Qi, Yao Yang Liu, Lin Chen, Zehui Chen, Xikun Bao, Jie Zhao, and Feng Zhao. V2p-bench: Evaluating video-language understanding with visual prompts for better human-model interaction. arXiv preprint arXiv:2503. /736, 2025.
Junjie Zhou, Yan Shu, Bo Zhao, Boya Wu, Shitao Xiao, Xi Yang. Yongping Xiong, Bo Zhang, Tięjun Huang, and Zheng Liu. Mlvu: A comprehensive benchmark for multi-task long video understanding. arXiv preprint 2406::2000.02664 2024.
Deyao Zhu, Jun Chen, Xiaoqian Shen, Xiang Li, and Mohamed Elhoseiny. Minigpt-4: En- hancing vinion-language understanding with advanced large language models. arXiv preprint arXiv:2304. 10592, 2023.
14

---

Published as a conference paper at ICLR 2026
King Zhu, Qianbo Zang, Shian Jia, Siwei Wu, Feiteng Fang, Yizhi Li, Shawn Gavin, Tuney Zheng, Jiawei Guo. Bo Li, Haoning Wu, Xingwei Qu, Jian Yang. Zachary Liu, Xiang Yue. H. Liu, Chenghua Lin, Min Yang, Shiwen Ni, Wenhao Huang, and Ge Zhang. Lime: Less is more for mllm evaluation, 2024.
15

---

Published as a conference paper at ICLR 2026
A
APPENDIX
A.1
ANNOTATION TUTORIAL.
A.1.1
QUESTION TYPE
Question Type Details
Representative examples of 13 task categories in IV-Bench are shown in Figure and Figure
Summary
These questions aim to test the model's ability, using the provided image, to understand the main narrative thread of the video and summarize the main plot character stories from global perspective. The model needs leverage clues context from the image, go beyond understanding single frames segments, grasp the core content video, and comprehend and extract a plot summary.
Spatial Relationship
These questions aim test the model's 'ss ability, using the provided image, understand the spatial positions of objects and their relationships within the video scene. The model needs to identify specific objects or areas based on the image, locate them video scene, and describe the surrounding spatial layout and relationships with other objects.
Existence
These questions aim to test the model's ability, using the provided image, to identify spe- cific object within the image and search within the video content determine that object appears or is used the vidco.
Reverse Existence
These questions aim to test the model's ability, using the provided image (often showing set), to perform comparative analysis of the set and identify missing elements within the video content. model needs to identify all items the image set, compare against the video content, and identify those items that do not appear the video.
Natural Language Inference
These questions aim to test the model's ability, using the provided image. to perform consis- tency reasoning between the visual content of the image and the video. The model needs understand image's visual information and video's content determine the image is semantically consistent with the video content.
Detailed Events Questions
These questions aim to test the model's ability, using the provided image, to identify spe- cific object scene in the image, locate related events within the video, and extract specific detail information (e.g.. price, time, location) from those events.
Instruction Understanding
These questions aim to test the model's ability, using the provided image, to understand the attributes of the object shown in the image and perform associative analysis connecting them with explanatory content in vidco (such as introductions descriptions). Based on the video content, the model needs to understand creation process related to specific attributes of the object shown in the image.
reason, definition, function, impact, or
Keyframe Extraction
These questions aim the model's ability integrate textual understanding with visual analysis of the provided image identify specific object state. The model then locate the corresponding keyframe(s) or segment within the video timeline, as depicted in the image, demonstrating comprehension of the 'keyframe" concept and the ability to correlate visual cues with temporal positioning.
Counting
test model's ability, using the provided image, to identify the spe- cific object category designated in the image and count all instances of that object within the video scene(s). The model needs to accurately identify and differentiate between individual instances and report the total count.
These questions
16

---

Published as a conference paper at ICLR 2026
Space- Space-Time Computing
These questions aim to test the model's ability, using the provided image (which might include scale information, map, specific reference objects), to understand spatial scales and track the movement trajectory objects/people within the video, ultimately calculating the actual distance traveled. The model needs utilize scale or reference points from the image to analyze the motion depicted in the video.
Space-Timee Time Computing
These questions aim to test the model's ability, using the provided image (which might reference specific time points, events, or individuals), to understand and analyze temporal information and sequential events within the video. The model needs to use the clues from the image to perform temporal calculations (like duration), comparisons (such length), or pinpoint events specific times within the video.
Constrained OCR
These questions aim to test the model's text recognition capabilities under specific con- straints (like artistic lettering or particular fonts), using provided image that showeases specific text style or example. The task involves scarching for and extracting text content from the video that matches the specified style.
Attribute Change
These questions aim test the model's ability, using the provided image (which typically designates the object/person to track), to continuously follow that specific target across dif- ferent segments of the video and analyze/describe how its attributes (e.g., color, state, loca- tion) change over time.
Temporal Reasoning
These questions aim to test the model's ability. using the provided image (which might reference event types, participants, or scenes), to understand the sequence of recurring events within the vidco and perform temporal reasoning locate the specific time point or time interval corresponding to the Nth occurrence of particular event that sequence.
A.1.2 DATA ANNOTATION STEPS
Operations
Watch Video and Determine Question Type:
After watching the video content to annotated in its entirety, select most suitable and valuable question type from the 14 pre-defined types based on the video content, and then brainstorm the question direction accordingly to prepare for subsequent question stem and answer design.
Design Question Stem and Answer:
Question Stem Design Requirements:
Close Relevance: Easure the question stem closely related to the content of the video and paired images.
Information Confidentiality: The question stem should avoid revealing any direct information about the video images, retaining only necessary hints.
Assessment Significance: Question stem design should have assessment significance, avoiding simple questions purely objective facts.
Concise and Clear Language: Use concise and clear language describe the ques- tion, avoiding ambiguity or redundancy.
Answer Design Requirements:
Unique Clarity: For the question stem, there must be unique and clearly correct
answer.
The content of the correct answer itself must not di- rectly reveal any information about the paired images.
Information Confidentiality:
Video Granularity: Specify the smallest video unit required to answer the question,
such as:
17

---

Published as a conference paper at ICLR 2026
Perception Tasks
Reasoning Tasks
Nutural Language Inference
Tempurul Reusoning
Query Image
scamn cQuery Imageo matches
Query Image When vidrro fully display the content
the style of asseeee in the video?
showw in the <Query Images?
Second from the left
D: Secoad from the right
FAAD
A: 01:26-01:20
D: 02:08-02:10
from the right
E: first from the left
8: 12:01:24-011 01:26
E: 2:01:20-0::2 01:20001:22
Reverse Existence
Counting
Query Imagee
person the <Query Image> appear in the video?
Query Image
The video blogper recommended a couple books, order which book the content the <Query Image> adapted from?
First from left...
from the...
THE
B: Second from the right...
B:2
E:
C.Second from left...
3
Constrained OCR
Query Image
text image in the <Query Images> ?
B: GREATNESS
Slay The DRAGON
PRFECT
Figure 5: Five remaining IV-Bench task categories: Natural Language Inference, Constrained OCR, Reverse Existance, Counting, and Temporal Reasoning. Each example requires using text, image, and video together.
Frame: The answer located at a specific frame in the video.
Clip: The answer located continuous clip of the video.
" Full Video: The answer requires understanding the full video content.
Video Range: Clearly indicate the specific segment range in the vidco where the an-
swer is located to quickly verify the accuracy of the answer.
COLLECT PAIRED IMAGES:
Diversity of Image Sources: Widely collect images that meet the question requirements
from the internet or video resources, avoiding single and over-reused image sources to ensure image diversity.
Non- Video Screenshots: Directly capturing frames from the current test video paired images is prohibited.
IMAGE QUALITY ASSURANCE:
Texture Clarity: The texture of the image must be clearly distinguishable, avoiding blurri- ness to ensure the effectiveness of visual information.
Subjeet Consistency: The characters or objects the image must be completely consistent with or visually highly similar in texture to the characters or objects appearing the video.
The target object in the image should occupy the main position of the image, highlighting subject reducing background interference easy observation and identification.
Subject Prominence:
Visual Information Validity: Images must meet pre-defined "visual information validity requirements."
Annotation Validity Requirement Number: Based on the specific basis for selecting im- ages, annotate the corresponding number ceqqurrement_number of the "visual information validity requirements" that the image meets. (Please provide pre-defined list of "visual information validity requirement" numbers accurate annotation).
18

---

Published as a conference paper at ICLR 2026
DESIGN DISTRACTOR OPTIONS (9 OPTIONS):
Each question needs design misleading incorrect op- tions, plus correct answer, for a total 10 options.
Number of Distractor Options:
Avoid patterned option design. Incorrect answers should be set from different angles and dimensions increase the discrimination and difficulty of the questions. Common confusion strategies include:
Diversified Confusion Strategies:
Conceptual Confusion: Use options with concepts similar to the correct answer but with subtle differences in meaning test the precise understanding of concepts.
Partial Correctness:
The content described in the option partially matches the video content, but does not fully answer the question raised in the question stem.
Irrelevant Information: The option content has low direct relevance to the video content, but may have some connection with the question daily cognition common sense, setting up interference.
Conclusions obtained from logical reasoning that able but actually incorrect based on the video content are used as distractor options. visual information presented in paired images to design options that match the images but not the video, forming interference. Conclusion Divergence: For same question stem, change different paired images to make the conclusion of the options change with the images, and use this as dis- tractor item.
Incorrect Reasoning:
Image Misdirection: Use objective
Design options that have certain significance or ra- lionality in real life based on the images, but do not conform to the video content, inducing
Real-World Significance Trap:
answer based common sense rather than the video.
Avoid Fabrication: Avoid designing options with overly obvious traces of fabrication, ensuring that distractors have certain degree of mislcadingness and avoiding casy exclusion by users.
OPTION FORMAT REQUIREMENTS:
Length Consistency: Ensure that the lengths of the 10 options (including the correct answer and distractors) approximately equal to avoid answer information leakage due to option length differences.
The language expression of distractor options should be concise and clear, avoiding unnecessary complex sentence structures and rare vocabulary. At the same time, the descriptive language of distractor options should avoid revealing any information about the paired images.
Concise Language:
A.1.3 METHODS FOR DESIGNING DISTRACTORS
Distractors Design Methods
Visual Replacement: Replace a visual element in the video (such as the color, shape, or texture of an item) with visual information that is similar but inaccurate to the actual content.
Quantitative Replacement: Replace a numerical detail the video (such as quantity, time, distance, etc.) with an incorrect numerical value.
Spatial Replacement: Incorrectly describe the location where event occurs, for example, misdescribing "in the kitchen" as "in the living room" another space.
Incorrectly deserihe the time point when an event occurs, for example, misdeseribing "morning" "evening."
Temporal Replacement:
Delibcrately add non-existent events or information to the vidco content, such as fabricating a plot or detail.
Addition of Information:
Missing Information: Delete important information
that exist in the video, for
example, deliberately omitting key information, leading incomplete information.
Detail Replacement: Incorrectly replace key information involving characters, events, or details in the video, for example, replacing the profession age attributes of character, incorrectly describing the details of an event.
19

---

Published as a conference paper at ICLR 2026
Sequential Replacement: Arrange series of actions events that actually occurred in the video in the wrong order, disrupting their chronological relationship.
Reality Trap: Based on people's common sense or logic in real life, design options that
have a certain meaning rationality in reality, but these options do not match the actual content of the video.
Conclusion Divergence: For the same video content and question stem, by changing differ- ent paired images, the conclusion of the options changes with the paired images.
Video Content Guidelines and Question Screening Process
Video Content Guidelines:
Include Similar Distractors:
The video must include least figures objects of the same type the image target to create visual confusion.
Meet any of the following conditions to emphasize the importance of texture:
Condition 1: Texture-Dominant Definition
The image target can only he identified in the video through texture characteristics.
3. For example: Only present close-up texture details of a human face, weakening features like contours and clothing.
Conniiionn 2: Key Feature Differentiation
*
Distractors of the same type in the video differ from the image target in at least one key visual feature. For example: Shoes the same style but different colors, the same person wearing different styles of clothing.
Condition 3: Multiple Feature Description Requirement
image larget requires least four visual features to be fully described. Emphasize the complexity of the target's visual information, requiring multi- dimensional features for accurate understanding.
Question Screening Process:
Generate Detailed Description:
After annotation, MLLM to generate Detailed Description based the image and question, controlling the granularity of the description.
Assess Answerability:
Use MLLM human evaluation, combined with the Detailed Description video, to attempt to answer the question.
Question Screening:
Eliminate questions that can be answered correctly based solely on the Detailed De- scription + video.
Retain questions to ensure that image texture information is crucial for a correct an- swer.
A.2 QUALITY CONTROL PROCESS DETAILS
The quality inspection of V--Benchmark comprises two rounds:
Round focuses on standardizing problem structures and content validity.
Round 2: addresses advanced quality requirements ensure task rigor.
20

---

Published as a conference paper at ICLR 2026
A.2.1 ROUND QUALITY CONTROL
Purpose
Ensure basic structural integrity and content standardization, including unambiguous ques- tion formulation, verifiable answers, reasonable distractors, and data quality.
Quality Assessment Dimensions
Clarity Validation: Verify grammatical correctness and unambiguous expression of ques- tions/options.
Answer Validity Validation: Confirm answers are deducible video content (eliminate labeling errors).
Task Categorization Calibration: Validate proper classification of question types.
Contextual Validation Contextual Validation: Ensuring answers and distractors are plau- sible and contextually relevant.
Image Quality Assurance: Check query image resolution visibility critical informa- tion.
Option Completeness Validation: Verify coverage of plausible alternatives (c.g., critical
missing options in multi-choice questions).
A.2.2
ROUND 2 QUALITY CONTROL
Purpose
Ensure task validity by verifying the necessity of multimodal components and the contextual
plausibility of distractors, thereby mitigating evaluation bias caused by design flaws.
Methods
Multimodal Necessity Check: Retain only questions requiring combined analysis of tex-
Vimage/video.
Information Leakage Detection: Identify text queries that inadvertently reveal visual con- tent through textual cues (e.g.. explicit object descriptions, positional references) and elimi- Icakage by rewriting queries preserve task intent.
Commonsense Dependency Screening: Eliminate questions answerable through general knowledge alone (c.g.. "the rises in the cast"))
Distractor Optimization: Redesign meaningless distractors on video content. Pre- ferred: Distractors should match answer categories or create confusion (e.g., use actual OCR text from vidcos for text-related questions).
Object Uniqueness Verification: Ensure non-unique targets in questions (e.g.. "What color is the person's 's clothing shown in both image and video?" requires multiple persons in video).
Quality control was primarily the responsibility of our author team and conducted in multiple stages. These stages encompassed strict manual checks against all defined quality standards and require- ments. The entire process predominantly manual; relied expert judgment for all final quality decisions, ensuring data validity and accuracy. Although we explored auxiliary tools, the ultimate quality assessment and decision-making entirely depended on our human expertise. This direct, manual oversight was key to guarantecing the quality and complexity of the benchmark.
A.2.3 STATISTICAL ANALYSIS OF QUALITY CONTROL.
To quantify the rigor of our manual review process and ensure the reliability of BIV--BBrd Bench. imple- ment "I-Generation, Verification" pipeline. This involves tracking retention rates at each stage and calculating inter-annotator agreement among experts.
21

---

Published as a conference paper at ICLR 2026
First Round Quality Check
Second Round Quality Check
Querry
Which
mentoroee
Query Image
XOnggraa Query: Accurding to video, what barn oew the tools in the picture muinly used desssambile?
the video?
The fitth The sinth
Sharting
opdero
main purpose of the items the -Qveriingee?
/ Bevieed Query: Acconding to the strees what the
Revise
Added option to easure completensss
Ravbn
Revise itens with information lealage in thei queetion query.
where the qquar finst warmen?
Query Image
What color coat the persen the waaring in the video?
RRrised Answers Leftside
Black Orange
E: Upper left
XOrignal
Green
Restse
Rovius prodennciccpiioeeeeoooooooo ancsed lebling ernurs.
Delete
Uliinate Dems stolate the Object Unigureees criterton.
Querry Image
ttem sQuary imaquo appair vidao?
Image
According
the of Is?
Yc
& Meptiiza Aurrntees
Maphictsphalo
tosufffottttttt dieeeminativeeeeeeeeeeeeeeeeee power
Figure 6: Representative data examples from Two Round Quality Check. Each round includes modifications and deletions to queries and images.
Pipeline Statistics and Retention Rates. Our multi-stage filtering process acts as funnel to ensure only high-quality samples remain. We begin with an initial pool of 1,500 videos and approx- imately 4,500 questions.
Round (Standardization): We filter out 285 videos and 1,072 questions due structural flaws or ambiguity, resulting in retention rate of 81.0% for videos.
Round 2 (Leakage & Necessity): We further remove 249 videos and 868 questions that fail strict multi-modal necessity checks or contain unfixable leakage.
The final benchmark comprises 966 videos and 2,560 questions. As detailed Table the overall retention rates are 64.4% for videos and 56.9% for questions, reflecting our high standard for data validity.
Table Statistics of the Quality Control Pipeline. The rigorous filtering process ensures that only high-quality, non-leaking, and strictly maage-grounder samples remain.
Videos
Questions
Stage
Count
Filtered
Retention
Count
Filtered
Retention
Initial Pool
1,500
4,500
Round I (Standardization)
1,215
285
81.0%
3,428
1,072
76.2%
Round 2 (Leakage & Necessity)
966
249
79.5%
2,560
868
74.7% 7%
Overall Retention
534
64.4% %
1,940
56.9 %
Inter-Annotator Agreement. To ensure consistency in expert judgment (particularly for sub- jective criteria like "distractor effectiveness" and "image necessity"). ). we calculate the agreement among the three independent inspectors for each verification decision. We compute Fleiss' Kappa
22

---

Published as a conference paper at ICLR 2026
across the validation process and achieve score of:
0.86
This value indicates strong inter-annotator agreement, validating that quality control stan- dards are applied objectively and consistently across the dataset.
A.3 INFERNCE PROMPT
Multiple-choice questions are constructed by pairing each instance with one correct answer and several distractors. During inference, the answer choices are randomly shuffled to ensure that the correct answer appears in different positions. The default placement order is video, image, and text. For some models, we also experiment with placing the image after the video. Therefore, we have two different text prompts: placed after the video. In the prompt, we explicitly specify the order the video and image, the video length, and instruct the model to answer the question based both the video and the image, providing only the options.
with the order of video, image, and text, and another with the image
Prompt
Video-first prompt: <VIDEO> <IMAGE> We provide you with an image placed at the very heginning, followed by video that has heen divided into evenly spaced frames across its seconds. duratio on. Please answer the question based on the content from both the image and the extracted video frames.
Image-first prompt: <IMAGE> <VIDEO> We provide you with a video that has been divided into frame num evenly spaced frames across its duraticn seconds duration, followed an image. Please answer the question based the content from both the video frames and the image.
A.4 RESOURCE
All experiments on open-source MLLMs are conducted on cloud servers with NVIDIA A800 80G GPUs and Intel Xcon Platinum 8336C @ 2.30 GHz CPUs. For open source MLLMs, we obtain their results where their API services. To avoid unknown impacts on results, do conduct batched inference and set batch size=1 for all MILMs.
A.5 JUSTIFICATION FOR IMAGE-GROUNDES VIDBO PERCEPTION REASONING
The introduction of our benchmark for Image-Grounded Video Perception and Reasoning is not an isolated academic pursuit but a direct response to convergence of massive, real-world user behav- iors and critical backend technical needs. This appendix provides comprehensive justification for the task's 's necessity, demonstrating that it is grounded in both large-scale, commercially vital user habits and range of established and emerging technological applications.
A.5.1 THE USER-DRIVED MANDATE: IMAGE QUERIES DOMINANT BEHAVIOR
The necessity of understanding visual queries is unequivocally demonstrated by their massive scale in production systems and clear user preference. This is not anecdotal use-case but foundational aspect of modern digital interaction.
Massive Scale and Commercial Vitality: User-facing evidence confirms that visual search is large-scale and commercially vital behavior. early as 2018, Pinterest's Lens platform was handling 600 million image searches per month Zhai et al. (2017). while Alibaba's image search application grew to over 17 million daily active users Zhang et al. (2018). These figures establish an undeniable foundation for the necessity of models that can ground their understanding in a user- provided image.
Strong User Preference: This massive adoption is driven by strong user sentiment. Re- search reports that 62% of Millennial and Gen Z. consumers wish for visual search
23

---

Published as a conference paper at ICLR 2026
capabilities, and 85 % of young respondents place more importance visual informa- tion than text when shopping. This preference is rooted in visual search's 's unique ability to handle queries that are difficult to verbalize Dagan et al. (0223) As "key driver for user engagement" that provides "a significant lift in key business metrics" (Zhai et al. (2017), the evidence is clear: image-grounder interaction is user expectation.
A.5.2 THE TECHNICAL IMPERATIVE: IMAGE-GROUNDES UNDERSTANDING AS A CORE CAPABILITY
The large-scale user demand for visual queries fuels a wide range of backend technical applica- tions where image-grounded video understanding not just beneficial but essential. The advent of MLLMs has significantly enhanced performance in these areas, making them more prevalent. Key applications include:
To generate relevant video from given multimodal inputs, such as in all-in-one video creation and editing platforms, robust joint understanding of images and videos is required [iang et al. (2025). For instance, in script-based movie gen- eration, visual information about main subjects backgrounds must first be acquired from images and then used for co-reference throughout the script, demanding a comprehensive integration of image and video contexts Wu et al. (2025b).
Video Editing and Creation:
Image-to- Video Generation: This task requires an MLLM to comprehend the relationship between static frames and the complete video dynamic. which is critical for creating coher- ent mange-text-viiee. triplets for training Yang al. (2025). This core capabiliit is essential across various sub-tasks, including image animation, which animates subject while pre- serving identity TTan et al.) (2024); Xu et al. (2024), and virtual try-on for E-commerce, which demands a precise understanding of image details to prevent distortion Fang et al. (2024).
Video retrieval is cornerstone for training specialized MLLMs, video generation models, and recommendation systems. When the desired con- tent is difficult describe with text, direct image-based retrieval offers superior alter- native. While historically based on embedding methods [olly et al. (2025), the rapid advancement of RAG technology has led a new wave of multimodal query-based video retrieval systems that require more sophisticated image- groggeeg reasoning Ventura et al. (2024) Thawakar et al. (0044).
Image-based Video Retrieval:
Image-Aided Video Understanding: A mainstream approach to long-video understand- ing involves first identifying keyframes or crucial visual elements from the video. These extracted visual cues (images) are then fed into an MILM alongside the original video for more focused and comprehensive analysis Han et al.) (2025): Luo et al.) (2024); Ma jet al. (2025). This concept evolving into more complex reasoning tasks, with a grow- ing body of work incorporating the idea of "thinking with images" directly into the video understanding process Zhang et al. (2025c).
In conclusion, the combination of overwhelming user demand for visual interaction and the foun- dational role of image-grounded understanding across numerous advanced Al applications provides a powerful and timely justification for our work. The ability to perceive and reason about video content based on an external image is not a niche capability but central challenge for the next generation of multimodal systems.
A.5.3 QUANTITATIVE ANALYSIS ON THE SEMANTIC GAP AND IMAGE NECESSITY
While the sections above establish the broad utility of image-grounded video understanding, we further conduct a comprehensive quantitative analysis to rigorously justify the structural design of IV-Bench. Specifically, we demonstrate why external images with diverse semantic gaps-ranging from near-frame extraction to high-level semantic abstraction are essential for a robust evaluation.
Experimental Setup. We quantify the "semantic gap" by calculating the cosine similarity between the external image query and uniformly sampled video frames using CLIP- ViT-L/14. Based on the similarity scores across 2,560 samples, we divide the dataset into three groups:
24

---

Published as a conference paper at ICLR 2026
High-Similarite (Top 25%, 0.567): Scenarios akin to near frame extraction direct visual matching.
Mid-Similarity (Mid 50%, 0,427-0.567): Scenarios with a moderate semantic gap.
Low-Similarity (Bottom 25%, 0.427): Scenarios with high semantic gap, requiring strong reasoning.
Insight 1: Diverse Gaps are Essential for Comprehensive Evaluation. We evaluate state-of- the-art MILMs across these groups (Table A). The results reveal divergent model behaviors that single-type dataset fails to capture. Some models (e.g.. Intem
show a clear pref-
erence for High-Sim scenarios (+3.29%), indicating strong visual pattern matching but weaker semantic generalization. Conversely, models like Qwen3-VL-32B excel in Low-Sim scenarios (-6.58% diff), demonstrating superior capability in handling abstract semantic connections. This underscores that holistic assessment must include diverse semantic gaps to evaluate both fine- grained visual correspondence and semantic reasoning capabilities.
Table 4: Model performance across semantic gaps. A positive difference denotes a preference for High-Sim Fear-Framm inputs, while negative difference denotes better performance on Low-Sim (Reasoning) inputs.
Model
High-Sim
Mid-Sim
Low-Sim
Diff (H-L)
Aria-32B
19.59
15.22
19.91
-0.31
Qwen2.5-VI.-77
20.85
16.71
19.28
+1.57
Qwen3-VL-B8 VL-BB
19.28
16.71
20.22
-0.94
Qwen3-VL-32B
20.53
23.29
27.12
-6.58
VL2.5222
22.10
19.29
21.94
+0.16
Intern VL3-8B
21,94
16.63
20.69
+1.25
IntemVL33- 14B
25.39
24.47
27.12
-1.72
Intern VL3-38B
26.49
24.47
27.43
-0.94
VL.3.5LL.3.888
21.79
17.80
19.75
+2.04
Intern --P
22.26
18.35
18.97
+3,29
Insight 2: Task-Specifi Dependencies and Discriminative Power. The necessity of diverse im- ages is further corroborated by task-level analysis (Table Tasks relying on direct visual verifi- cation (e.g., Detailed Events, Existence) benefit significantly from high visual similarity (+20.64% gap). However, reasoning-intensive tasks (e.g., Auribute Change, Natural Language Inference) per- form substantially better with larger semantic gaps. Notably, Attribute Change achieves 10.47% higher accuracy in low-similarity settings. This suggests that high high-similarity images may introduce bias by fixing attention static states, whereas external images with a larger semantic gap encour- age the model to focus on attribute transformations.
Furthermore, low-similarity samples demonstrate 39.7% higher standard deviation in model performance compared to high-similarity samples, indicating that scenarios with larger semantic gaps provide stronger differentiation power for evaluating advanced model capabilities.
A.6 DIVERSITY ANALYSIS OF EXTERNAL IMAGES
To ensure that IV-Bench robustly evaluates MLLMs across a wide spectrum real-world scenar- ios, the external grounding images were curated to exhibit substantial diversity in both content and complexity. This section provides quantitative analysis of this diversity.
A.6.1 METHODOLOGY FOR CLASSIHICATION
To systematically categorize the visual content, recruited three annotators with significant ex- perience in evaluating Multimodal Large Language Models (MLLMs) to classify all 2,560 external images into seven distinct categories based their primary subject matter. This classification pro- vides a clear overview the visual contexts represented the benchmark, confirming that models tested against varied set of grounding inputs.
25

---

Published as a conference paper at ICLR 2026
Table Performance by Task Type across Semantic Gaps. Tasks are sorted by their dependency on high visual similarity (Diff H-L).
Task Type
High-Sim
Mid-Sim
Low-Sim
Diff (H-L)
Detailed Events
35.37%
21.44%
14.73%
+20.64%
Summary
26.38%
18.28%
16.39%
+9.99%
Instruction Understanding
17.96%
21.86%
11.62%
+6.34%
Existence
20.00%
15.70%
16.18%
+3.82%
Reverse Existence
16.51%
14.70%
13.39%
+3.12%
Keyframe Extraction
16.81%
14.05%
13.93%
+2.88%
Counting
20.45%
18.11%
19.66%
+0.80%
Space-Time Computing
18.53%
12.95%
18.18%
+0.35%
Temporal Reasoning
12.03%
11.55%
12.50%
-0.47%
Spatial Relationship
32.66%
25.52%
34.85%
-2.19%
Constrained OCR
31.49%
29.55%
36.64%
-5.16%
Natural Language Inference
28.57%
28.89%
36.36%
-7.79%
Attribute Change
18.02%
22.48%
28.48%
-10.47%
A.6.2 DISTRIBUTION OF IMAGE CATEGORIES
The distribution the external images across the seven content-basel categories detailed in Ta- ble 6 The results show balanced yet comprehensive collection, with strong representation of both object-centric and person-centric scenes.
Table 6: Distribution of external images across seven content-based categories. The variety ensures that models are evaluated on wide range visual grounding scenarios.
Category
Single Person
Multi Person
Single Object
Multi Object
Landscape
Text-Dominant
Others
Count
553
293
792
296
187
303
136
As the data illustrates, these seven categories span broad range of visual contexts. They range from simple compositions like single-person portraits and isolated object shots to more complex scenarios involving multi-person and multi-object scenes, as well as ambient landscapes and information-rich, text-dominant images. This variety in both structural and semantic content ensures that IV-Bench's external images rigorously challenge a model's image-grounded video perception and reasoning capabilities. By preventing models from overfitting narrow type of visual query, this diversity promotes the development of more generalized and robust systems.
A.7 ABLATION STUDY IMAGE VS. TEXTUAL CAPTION GROUNDING
To validate the necessity of using external images for grounding in IV-Bench, conduct an ablation study to investigate whether the proposed tasks can be simplified into a text-only video question- answering problem. Specifically, evaluate naive two-step baseline where the external image is first converted into machine-generate textual description (a caption), which then serves as the textual context alongside the video.
A.7.1 EXPERIMENTAL SETUP
We evaluate this approach on two powerful open-source models, Qwen2.5-VL--22B and Qwen2.5- VL--2BB, under three distinct settings:
1) Video-Only: The model receives only the video frames and the text query, with no image-based context. This serves our lower-bound baseline.
2) Video Caption: The external image is replaced by auto generated caption. To ensure fair comparison, the caption for each image is generated by the model under evaluation itself.
3) Video + Image: The model is provided with the full input triplet: video, the raw external image, and the text query. This represents the standard setting for IV-Bench.
26

---

Published as a conference paper at ICLR 2026
Consistent with the main paper's evaluation protocol, all experiments conducted using 32 video frames at a 480p resolution.
A.7.2 RESULTS AND ANALYSIS
The performance comparison across the three settings summarized in Table
Table 7: Performance comparison of different input modalities IV-Bench. The Video Caption setting demonstrates that while textual descriptions of the image provide some benefit over a Video- Only baseline, they are significantly outperformed by the direct use of the raw image (Video Image).
Model
Video-Only
Video + Caption
Video + Image
Qwen2.5-VL Qwen2.5-VL-72B
18.0%
20.9% (+2.9%)
23.0% 28.9%
18.6%
24.8% (+6.2%)
The results clearly indicate that while adding a textual caption of the image provides a notable performance uplift over the Video-Only baseline (+2.9% the 32B model and +6.2% for the 72B model), it is insufficient to match the performance of true image grounding. A significant performance gap persists between the Video + Caption and Video + Image conditions (2. % for 32B and 4.1% for 72B).
This persistent gap underscores a core motivation for IV-Bench: textual summaries inevitably omit critical visual information that is essential for correct answers. Nuances such as fine-grained ob- ject attributes, specific human facial features, and complex spatial details are often lost during the captioning process. Therefore, although captioning direct image input can fully capture the depth of visual perception and reasoming that our benchmark designed to evaluate. This experiment confirms that the challenges posed by IV-Bench cannot be circumvented by relying on text-only shortcuts and require genuine multimodal reasoning grounded in direct visual evidence.
be considered strong naive baseline, only
A.8 HUMAN TEST
To explore the gap between Multimodal Large Language Models (MLLMs) and human performance on IV-Bench, we recruit ten experienced annotators who, over four days, answer all IV-Bench ques- tions. To facilitate a comprehensive understanding and accurate responses, we provide the annotators with the complete video, corresponding images, and the full text associated with each question. The evaluation is conducted under strict protocol where we instruct annotators to base their answers solely on the provided materials, without the use of any external tools such as search engines or other aids. The average of the human performance is then calculated and included in paper for comparison with the model's 's performance.
A.9 ROOT CAUSE ANALYSIS: INFORMATION AVAILABILITY vs. TEMPORAL CAPABILITY
A significant observation from our main evaluation is the limited performance on frame-sensitive tasks, such as Temporal Reasoning, Counting, and Space-Time Computing. A natural hypothesis that this limitation stems from sparse frame sampling (typically 8-32 frames), which may miss critical temporal details required for fine-gra fine-grained understanding.
To investigate whether the bottleneck lies in insufficient visual information (i.c., the sampling rate) or inherent model capabilities, we conducted controlled experiment using Qwen3-VI--8 We com- pared performance under standard setting (32 frames) versus high-density setting (128 frames) to isolate the impact of information availability.
Experimental Setup: Quantifying Information Gain. First, we verified that inereasing the frame count indeed provides more valid visual information. We define two metrics:
Temporal Coverage: The percentage of samples where at least one sampled frame falls within the ground truth (GT) relevant time segment.
27

---

Published as a conference paper at ICLR 2026
Frame Density: The average number of frames sampled per second.
As shown in T increasing frames from 32 to 128 results a substantial increase in information availability, with Temporal Coverage improving by nearly 10 percentage points and Frame Density increasing by 3.2x.
Table 8: Comparison of information availability between 32 and 128 frames.
Metric
32 Frames
128 Frames
Improvement
Temporal Coverage
76.09%
85.94%
+9.85 pp (+13.0%)
Frame Density
0.13 fps
0.41 fps
+3.2x (+216%)
The Divergence. We focused our analysis on four Prame-sensitive tasks that exhibited similarly low baseline performance (< 20%). The results, presented in Table reveal a critical divergence in how models respond to increased information.
Performance Analysis:
Table 9: Performance comparison frame-sensitive tasks using Qwen3-VL-S
Task Category
32 Frames
128 Frames Change
Counting
14.98%
16.48%
+1.50%
Space-Time Computing
14.02%
16.36%
+2.34%
Keyframe Extraction
18.47%
17.77%
-0.70%
Temporal Reasoning
13.33%
11.11%
-2.22%
While tasks relying information aggregation (e.g., Counting, Space- Computing) benefit modestly from the increased frame count, performance on Temporal Reasoning notably declines.
The 'Static Bias' Bottleneck. This paradox-where more relevant visual informa- tion leads to worse reasoning performance-stronee suggests that the primary bottleneck is not the lack of visual cues, but rather inherent deficiency the model's temporal modeling mechanism.
Discussion:
1. Information Overload vs. Utilization: If the limitation were simply missing frames, Tem- poral Reasoning should have improved alongside Counting. The decline implies that the additional frames introduced "information overload" rather than useful signals for sequen- tial logic.
2. Static Bias:
Our findings align with the concept of "Static Bias" identified in recent studies (e.g., MVBench Li et al. (2024c)). Current MLLMs predominantly rely on static image encoders (e.g.. CLIP) trained to align single images with text. This design encourages the model to process video modeling temporal flow or causality.
"bag of frames," effectively averaging static content rather than
3. Temporal-Spatial Performance Gap: Consequently, we observe a significant gap where increasing frame density fails to bridge the reasoning deficit. Without explicit mechanisms to model inter-frame dynamics, simply feeding more frames exacerbates the noise for rea- soning tasks, highlighting the urgent need for architectures that move beyond static visual encoding to genuine causal modeling.
A.10 SYMMETRIC IMPACT OF MODEL SCALING: PERCEPTION REASONING
To investigate whether simply increasing model capacity solves the challenges in image- grounded video understanding, we conduct a detailed analysis of scaling effects. We categorize the tasks into Perception Tasks (c.g., Existence, Detailed Events) and Reasoning Tasks (e.g.. Counting, Temporal Reasoning) and evaluate performance gains across five major model families.
Experimental Results. Table summarizes the performance of models ranging from 7B to 78B parameters. We observe a consistent and significant divergence in how scaling benefits different task types.
28

---

Published as a conference paper at ICLR 2026
Table 10: Scaling Effects on Perception vs. Reasoning. Larger models achieve massive gains in perception but show diminishing returns in reasoning. (Gain denotes the absolute improvement from the smallest to the largest model in the family).
Model Family & Size
Overall
Perception
Reasoning
Qwen2.5-VV. 21..2........ Series
Qwen2.5-VL-78 Qwen2.5-VL-25 Qwen2.5-VL. VLL-2B
18.5%
18.9%
17.9%
23.0%
24.9%
20.2%
28.9%
33.7%
21.9%
Gain (7B - 72B)
+10.4%
+14.8 %
+4.0% %
Qwen3-VV VL Series
Qwen3-VL--8B Qwen3-VL-328 VL-32B
18.2%
20.1%
15.4%
23.5%
27.7%
17.4%
Gain (8B - 32B)
+5.3%
+7.6%
+2.0%
InterVV12..3 Series
Intem VL2.5-8B
17.4%
17.8%
16.5%
Intern V12.5-12.5-26B
20.6%
22.4%
18,
Intem L125mVL2.5-38B
26.6%
30.4%
21.1%
Intern 782
28.6%
33.4%
21.9%
Gain (8B 78B)
+11.2%
+15.6%
+5.4%
Intern Series
Inter VL3-8B
18.9%
20.0%
17.3%
Intern VL3-14B
25.3%
27.2%
22.5%
Intern VL3-38B
25.6%
27.2%
23.4%
Gain (8B -> 38B)
+6.7%
+7.2%
+6.1% %
Intern Series
Inter LL3.5
19.3%
18.9%
19.8%
Intern V
26.0%
27.3%
24.0%
Gain (8B > 38B)
+6.7%
+8.4%
+4.2%
Analysis:
Perception Dominates, Reasoning Lags. Our analysis reveals two critical trends:
1. Perception Dominates Scaling Gains: Across almost all model families, perception tasks witness massive improvements as model size increases. For instance, the Qwen2.5-VL series sees a +14.8% % jump, and InternVL2.. sees +15.6%. This confirms that increasing parameters significantly enhances the model's capacity to "see," identify, and match static visual content.
2. Reasoning Encounter Diminishing Returns: In sharp contrast, reasoning tasks show much more modest gains (e.g., only +4.0% for Qwen2.5-V LL and +2.0% for Qwen3-VLJ VLL). Even the largest 70B+ models struggle to break past the 22-23% accuracy mark reason- ing tasks, despite their strong perception capabilities (> 33%).
Implication. This finding highlights critical limitation of current scaling laws in the context of IV-Bench. While scaling effectively improves visual recognition, it encounters severe diminishing returns for complex reasoning. This suggests that future advancements must look beyond simple parameter scaling and focus on architectural innovations (c.g., temporal causal modeling) or data- centric approaches (e.g., wwssonnng-specffie curriculum learning) bridge this gap.
A.11 IMPACT OF TEXT ORDERING ON IV- BENCH
Given the complex multimodal context in IV-Bench (Video External Image Text), the sequential ordering of inputs plays crucial role. While our default setting follows the widely adopted "Vision- then-Text" paradigm (i.e., Video - Image + Question), we conduct controlled experiment to investigate the sensitivity of MILMS to inverted text ordering (i.e., Question - Video - Image).
Experimental Setup. two input configurations:
We evaluate two distinct model families, MiniCPM and InternVL2.5. under
29

---

Published as a conference paper at ICLR 2026
1. Text After Vision (Standard): The standard training configuration for most recent MLLMs Chen et al. ) et al. (2024a): Wang et al. (2024). where the textual query appears at the end of the sequence.
2. Text Before Vision (Inverted): The textual query is prepended to the visual tokens, forcing the model to retain textual intent across the long visual sequence.
Results and Analysis. As shown in Table 11 inverting the text order leads substantial perfor- mance degradation across all tested models.
Table 11: Impact of Text Ordering. Placing the text query before visual content results in signifi- cant performance drops, highlighting the sensitivity of current MLLMs to input sequence ordering.
Model
Text After Vision
Text Before Vision
Performance Drop
MiniCPM-V
17.2%
13.4%
-3.8%
MiniCPM-o
17.1%
14.1%
-3.0%
InterVV2..5-1
26.6%
15.9%
-10.7%
Intern VL2.5-78B
28.6%
16.5%
-12.1 %
Key Observations.
Universal Degradation: Placing text before vision causes accuracy drops ranging from 3.0% to 12.1%. This confirms that the relative positioning of textual and visual modalities is not trivial and significantly impacts multimodal integration.
Model Sensitivity: Different architectures exhibit varying degrees of robustness. The In- VL.25 series is particularly sensitive, with drops exceeding 10%, whereas MiniCPM shows relatively better stability.
Distribution Shift and Attention Dilution: We attribute this degradation to two primary factors. First, the "Text Before Vision" order deviates from the standard pre-training ob- jective of most MLLMs (Next Token Prediction based on visual prefix), causing a distribu- tion shift. Second, consistent with the "forgetting" phenomenon discussed in Section A0 placing text before lengthy sequence of visual tokens (video frames + image) likely causes attention dilution, where the model under-utilizes the early textual instructions.
These findings empirically justify our choice of the "Vision-then Text" ordering as the optimal set- ting for image-grounded video understanding.
A.12 MECHANISM VERIFICATION: ATTENTION SCORE ANALYSIS
To quantitatively investigate the mechanism behind the input ordering effects observed in Sec- tion 4.3 we analyzed the attention distribution within the ILt-r and 14B models. We compared two distinct configurations: placing the external image before the video frames versus placing after the video frames (closer to the textual query).
Methodology. We calculated the attention weights by aggregating scores from all attention heads across all transformer layers during the decoding phase. Specifically, for each generated text token, computed the average attention weight it assigned the visual tokens (image vs. video) and averaged these values over the entire IV-Bench dataset.
Results. The results, summarized in Table 12 reveal a significant shift in attention distribution based on input ordering.
Analysis and Key Findings.
Drastic Increase in Image Attention: When the image is placed after the video (closer to the final prediction token), the average attention weight image tokens increases by
30

---

Published as a conference paper at ICLR 2026
Table 12: Attention Distribution Analysis. Placing the image after the video frames (closer the query) drastically increases the attention weight assigned to the image, correlating with improved accuracy.
Model
Configuration
Accuracy
Image Attn.
Video Attn.
Image Before Video
16.19%
0.0051
0.0865
IntemVL3-885
Image After Video 20.21%
0.0217
0.0730
Image Before Video
21.73%
0.0041
0.0937
Intern VL3-14B
Image After Video
24.60%
0.0218
0.0781
approximately 4x to 5x (e.g., rising from 0.0051 to 0.0217 for InternVL3-88 This con- firms that proximity to the query significantly enhances the model's ability attend to the external grounding image.
Correlation with Performance: This surge in image attention strongly correlates with
improvement in accuracy (rising from 16.19% to 20.21% for the 8B model). This empirical evidence suggests that the "Image Before" configuration leads to unlli-ttii of the grounding signal due the long context of the intervening video.
Verification of Positional Bias: These statistics empirically verify the "positional bias" or "forgetting" effect. When the image appears carly in the context window, suffers from attention dilution caused by the subsequent video sequence. Placing the image later ensures it remains in the model's s active focus, facilitating better visual grounding.
Consistency with Sparse Attention: We also observe that the absolute attention scores for visual tokens generally low (e.g., ~0.02). This aligns with findings in FastV Chen et al. (2024a), which revealed that visual tokens often receive sparse attention in deeper layers. Our results confirm this sparsity but highlight that even small shifts in this limited budget (from 0.005 to 0.02) can drive significant performance gains.
31

---

Published as a conference paper at ICLR 2026
A.13 COMPARISON WITH HIDEO-MMMU
Fiour the main differences between IV-Bench and Video-MMMU. IV-Bench is designed to require all three modalities- image, and video for every question. In contrast, Video-MMMU often allows questions to be answered using only text, making video or image redundant. Specifi- cally:
Image Necessity:In Video-MMMU, image-text prompts often include full descriptions that make the image redundant. For instance, the caleium-hyydroxide question's text describes both solutions in detail, so viewing the picture of the bottles does not change the answer. In --Bennh, each image contains distractor objects that are not mentioned in the text. Only by examining the image can one distinguish these distractors-eA, e.g. in the "Which contents... quesstion, you must look at the picture to see which item never appears the video.
Video Necessity: In Video-MMMU (left of Fig 7. the real-interest-rate question includes both text and video, but it can be answered using only the text, making the video unnecessary. In fact, in the very first Vidco-MMMU example, neither the image
the video is needed to find the correct
answer. By contrast, in IV-Bench (right of Fig 7. the question "When did the items in the picture first appear in the video?" cannot be answered without watching the video, since only the video reveals the exact timestamp.
Query Modality: Vidco-MMMU uses pure-text for two-thirds of its items: in the Fig 7 "Compre- hension" and "Perception" examples (the second row), both queries are text-only, accounting for 2/3 of the dataset. IV-Bench, however, pairs text with an image in every query. For example, in the "Which contents in the image do not appear in the video" item, the model must read the text prompt and inspect the image together to identify the correct distractor.
Video MMMU
IV Bench
Adaptation: image Comprebension:
Perception:
Perception: Image+Text
Reasoning: Image- Text
Image Necessity
Video Necessity
VVieeo Necessity
J Imago Necessity
MMMU Adaptation
Bench
Textslone is sufficeent - image and video redundant
All modalities are necessary - none rellunannt
<Vide>>>
<ideo>
<image>
Soherimse A and conceting ofraccum hystratte
<hroge>
When the torre the video?
disnttedd vaive ciragy b-Schution acilem
Iydrocid: solution fut coscentu aisu 0,350
D207
03-47
00:122
00:52
Iydronid: dischral make solution...
C 00.32
The <Tat>> section describes the content of the <Image>> detail.
<lasges, cVilend, <Taa thrse requirad.
<leet>
Arswen
Video Comprebension Pecception
Bench
is sulficient video redundass
All three modalities are necessiry - none is redundant
<Tideo>>
wwwpcto
<image>
Which costinns image appoar
Clecs
aieiinaiintii mihho??
Subtrect inlution rato Eromo the nosiiall
Second left first Irem Bi firstion the left secord
Di Ditine the..
B. Subriact the rominal interest rate from the inlltioon rite
B. fisttion the
C. Add the inilutios rate to the nominal intorat Bivile the rovined interet rase by the inforios vate
from
Necnorary
Recnsary
The quaction can be arowerd without the <Videncs
<insge>, <Videoo, <TarD three are requized.
Answer:
<image 24 <Vides>> icTlats
Arsonent A
Figure 7: Side-by-side comparison of Video-MMMU (left) and IV-Bench (right). The left side shows that in Video-MMMU many questions can be answered using only text, so the image or video input is often not needed. The right side shows that IV-Bench always requires using the video, image, and text together to answer each question. IV-Bench enforces true multimodal reasoning by adding visual distractors and making each modality necessary, so no single modality (like text alone) answer the questions.
32


Reviews:
[{"review_id": "V7fU2Gh9RP", "reviewer": "Reviewer_AHVG", "summary": "IV-Bench is one of the first benchmark for image-grounded video perception and reasoning in MLLMs, featuring 966 videos and 2,560 external image-text queries across 13 diverse tasks. Unlike other benchmarks, all queries use external images as anchors, increasing real-world complexity. Evaluations on 28 leading MLLMs show performance is much lower than humans, especially in reasoning. The paper analyzes model size, reasoning methods, and token allocation, and both dataset and code are open-source.", "questions": "1. On subjective scoring consistency and ground-truth diversity: For reasoning tasks with subjective or diverse answers, how is human label consistency and answer distribution measured? How are multi-answer questions evaluated?\n\n2. On the irreplaceability of images: Has image replacement testing been conducted across all task categories? If an image is replaced by another with similar semantics but different visuals, does the answer change? If not, how can it be demonstrated that the image truly “grounds” the reasoning process?\n\n3. On prompt strategy: In Table 2, do all models use the optimal prompt order (image after video)? If not, have you considered rerunning the experiments for fairness?", "limitations": "", "rating": 8, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 4, "strengths": "- This is one of the first systematic benchmark proposal for image-grounded video perception and reasoning, filling a notable gap in existing evaluation ecosystems.\n\n- The benchmark features high data quality and a robust evaluation framework. All images are sourced externally, preventing information leakage and accurately reflecting the difficulty of cross-source visual grounding in complex tasks such as search and retrieval, which strongly enhances the validity of the assessment.\n\n- Experimental analyses are thorough and the findings are insightful for follow-up research. The paper details the effects of image token order, scale, resolution, and frame count on model performance, and clearly points out the extremely limited reasoning capabilities of current MLLMs in image-grounded video scenarios. It also shows model scaling has minimal impact on reasoning, which offers clear direction for future advances.", "weaknesses": "- Some tasks (e.g., Instruction Understanding, Summary) feature only weak connections to the visual “grounding,” where images act in a mainly auxiliary role and do not fully embody the core definition of image grounding.\n\n- The paper lacks granular error analysis. It reports only accuracy without dissecting which sub-tasks exhibit systematic failure (e.g., universal shortcomings in temporal reasoning or attribute change), and does not offer failure case examples. It is suggested to add error type breakdowns (e.g., missed visual cues, text comprehension errors, temporal confusion) and provide typical failed cases versus human labels in the appendix.\n\n- The discussion around the semantic gap and selection logic for external images should be strengthened. Although the necessity and realism of using external images is emphasized, there is no structured description or case analysis of their diversity/difficulty. A quantitative contrast between tasks with highly similar external images (near frame extraction) versus high-gap scenarios should be considered.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "tOeABMG4fU", "reviewer": "Reviewer_MRHL", "summary": "This paper introduces IV-Bench, the first comprehensive benchmark designed specifically for evaluating Multimodal Large Language Models (MLLMs) on image-grounded video perception and reasoning tasks. The benchmark comprises 966 videos and 2,560 meticulously annotated image-text queries, spanning 13 distinct tasks (7 perception and 6 reasoning) across 5 video categories. The authors conduct an extensive evaluation of 28 state-of-the-art MLLMs, revealing that even the best-performing model (Qwen2.5-VL-72B) achieves only 28.9% overall accuracy, which is significantly lower than human performance (88.8%). Furthermore, through ablation studies, the authors analyze the impact of factors such as the order of image input, number of video frames, and resolution on model performance, providing valuable insights for future model design.", "questions": "1.\tDuring the construction of IV-Bench, did you consider incorporating dynamic image sequences (e.g., GIFs or short video clips) as the grounding signal to better simulate real-world scenarios where users provide visual context?\n2.\tRegarding the particularly poor performance on tasks like \"Temporal Reasoning,\" have you conducted further analysis into the root causes? Is it related to the models' ability to understand long videos or their inherent temporal modeling mechanisms?", "limitations": "", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "1. IV-Bench is the first benchmark specifically dedicated to image-grounded video understanding tasks. It effectively addresses the limitation of existing video benchmarks that rely solely on text queries, thereby advancing the field of multimodal reasoning evaluation.\n2.The benchmark's high quality and validity are ensured through a two-round quality control process, the use of externally sourced images, and diverse task design. The inclusion of \"effective distractors\" is particularly noteworthy, as it forces models to rely on the image information.\n3. The systematic evaluation of 28 models thoroughly exposes the significant shortcomings of current MLLMs on this task. The ablation studies offer practical design recommendations, such as the finding that placing the image after the video frames yields better performance.", "weaknesses": "1.Although covering multiple categories, the total of 966 videos is smaller than some existing video understanding benchmarks (e.g., Video-Bench with 5,917 videos), which might affect the benchmark's broad representativeness.\n2.The current work focuses only on the triplet input of image-text-video. It does not explore more complex multimodal grounding signals, such as audio, multiple images, or dynamic image sequences (e.g., GIFs).\n3.While emphasizing \"image-grounding,\" the benchmark may not fully account for whether a single static image is sufficient to represent dynamic changes in a video. Some tasks might still lean towards static matching rather than genuine spatio-temporal reasoning.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "QR1vx4wy9G", "reviewer": "Reviewer_2qaq", "summary": "This paper proposes a benchmark named IV-Bench that requires the models to perceive and reason the contents from a reference image which is not involved in the input video. The dataset is manually labored and elaborated 7 main findings from evaluating 28 MLLMs.", "questions": "- How do RL-trained models perform on this benchmark? Are there any inductive bias observed from visual encoders? Are there any underlying biases or trends from the training set? These points are revealed that these two points matter [2,3].\n- Authors analyzed the order of video and image matters. Did the ordering of text inputs change performance?\n- I believe that analysis on attention score can somehow verify the finding in Section 4.3 (line 400). How are the attention scores distributed towards image tokens?\n- How can the evaluated models have forgetting issues (line 401) in the input sequence? Different from LSTM models that sequentially feed tokens to update hidden state, Transformers architecture processes the input tokens in parallel rather than sequentially. Isn’t the “forgetting” happens only when the input tokens are truncated according to the size of the context window? The size of the context window should be reported to make this claim concrete. In other words, how many previous tokens can the models condition on.\n\nReferences\n\n[1] Fang et al. MMBench-Video: A long-form multi-shot benchmark for holistic video understanding. NeurIPS D&B Track 2024.\n\n[2] Wang et al. VideoHallucer: Evaluating Intrinsic and Extrinsic Hallucinations in Large Video-Language Models. arxiv:2406.16338\n\n[3] Li et al. Vidhalluc: Evaluating Temporal Hallucinations in Multimodal Large Language Models for Video Understanding. CVPR 2025.", "limitations": "", "rating": 4, "confidence": 5, "soundness": 4, "presentation": 4, "contribution": 2, "strengths": "- The curation of the dataset does not rely on automated annotation tools (e.g., GPT-4V) and ensures high quality by manually annotations.\n- Assessing model capability to utilize reference images is a novel motivation. Also, excellent presentation with high overall readability and nice figures.\n- Impressively extensive evaluation (28 MLLMs) is reported, providing actionable diagnostics. Also, the performance gap between humans and models is huge.", "weaknesses": "- The main concern is that the findings are not surprising compared to the quality of the dataset. Comparisons across the model capacity, video fps, and resolution are trivial according to the scaling law [1].\n- Although it is not necessary, the analysis lacks experiments on meta-data (e.g., subtitles).\n- All tasks are based on multi-choice questions, not requiring open-ended questions even for the “reasoning” category.\n- Inter-annotator agreement and quality-control rejection rates on both rounds are missing.\n- The paper rigorously describes the model performance by model families (e.g., line 349 to 360). I believe that reporting this with averaged gain by model scaling and adding more verifications and allocating more descriptions of author’s findings can make this paper much more stronger.\n\nI will increase the score once current concerns are addressed.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "qiFFtzu2M5", "reviewer": "Reviewer_Zjuc", "summary": "This paper presents IV-Bench, the first large-scale benchmark specifically designed to assess image-grounded video reasoning and understanding in multimodal large language models (MLLMs). IV-Bench comprises 966 videos paired with 2,560 manually annotated image–text queries, covering 13 tasks across 5 diverse domains. Through an extensive evaluation of state-of-the-art MLLMs, the study finds that current models achieve no more than 28.9% accuracy, revealing substantial performance gaps.", "questions": "I have no questions.", "limitations": "", "rating": 6, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "1. The paper tackles an important and previously overlooked problem—image-grounded video understanding—by establishing the first comprehensive benchmark in this area, thereby filling a crucial gap in existing multimodal evaluation benchmarks.\n\n2. The paper highlights the limitations of current MLLMs in handling such tasks, offering clear guidance and future directions for advancing MLLM capabilities.", "weaknesses": "The analyses in the paper are not sufficiently deep or insightful. For example, the reported phenomenon of \"moderate performance gains with larger models\" lacks both theoretical grounding and concrete case studies. Assertions such as \"increasing model size primarily enhances memorization and shallow pattern recognition rather than reasoning ability\" are not adequately supported by evidence. Similar shortcomings appear throughout other analytical sections. Moreover, the paper fails to propose practical solutions or methodological advances to address the identified challenges.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (Poster)