V2P-Bench: Evaluating Video-Language Understanding with Visual Prompts for Better Human-Model Interaction
Authors: Yiming Zhao
Yu Zeng
Yukun Qi
YaoYang Liu
Xikun Bao
Lin Chen
Zehui Chen
Qing Miao
Chenxi Liu
Jie Zhao
Feng Zhao
Venue: ICLR 2026 Poster (2026)
Keywords: LVLMs
Video Understanding
Visual Prompting
Primary area: datasets and benchmarks
TL;DR: A Video-Language Understanding Benchmark for Evaluating LVLMs on Human-Model Interaction through Visual Prompts

Abstract: Large Vision-Language Models (LVLMs) have made significant strides in the field of video understanding in recent times. Nevertheless, existing video benchmarks predominantly rely on text prompts for evaluation, which often require complex referential language. To address this limitation, we propose V2P-Bench, a robust and comprehensive benchmark for evaluating the ability of LVLMs to understand Video Visual Prompts in human–model interaction scenarios. V2P-Bench consists of 980 videos and 1172 well-structured high-quality QA pairs, each paired with manually annotated visual prompt frames. The benchmark spans three main tasks and twelve categories, thereby enabling fine-grained, instance-level evaluation. Through an in-depth analysis of current LVLMs, we identify several key findings: 1) Visual prompts are both more model-friendly and user-friendly in interactive scenarios than text prompts, leading to significantly improved model performance and enhanced user experience. 2) Models are reasonably capable of zero-shot understanding of visual prompts, but struggle with spatiotemporal understanding. Even o1 achieves only 71.8%, far below the human expert score of 88.3%, while most open-source models perform below 60%. 3) LVLMs exhibit pervasive hack phenomena in video question answering, which intensify with longer videos and lower frame sampling density, artificially inflating performance scores. We anticipate that V2P-Bench will not only shed light on these challenges but also serve as a foundational tool for advancing human–model interaction. The code and datasets are available at https://github.com/gaotiexinqu/v2p-bench.

Published as a conference paper at ICLR 2026
V2P-BENCH: EVALUATING VIDEO-LANGUAGE UN- DERSTANDING WITH VISUAL PROMPTS FOR BETTER HUMAN-MODEL INTERACTION
Yiming Zhao', Yu Zeng', Yukun Qi Yao Liu', Xikun Bao',
Lin Chen', Zehui Chen', Qing Miao*, Chenxi Liu', Jie Zhao?, Feng Zhao
1 University of Science and Technology of China
2 Huawei Noah's 's Ark Lab
3 Xi' a' Jiaotong University
ABSTRACT
Large Vision-Language Models (LVLMs) have made significant strides in the field of video understanding in recent times. Nevertheless, existing video benchmarks predominantly rely on text prompts for evaluation, which often require complex referential language. To address this limitation, we propose V2P-Bench, robust and comprehensive benchmark for evaluating the ability of LVLMs to understand Video Visual Prompts in human-model interaction scenarios. V2P-Bench consists of 980 videos and 1172 well-structured high-quality QA pairs, each paired with manually annotated visual prompt frames. The benchmark spans three main tasks and twelve categories, thereby enabling fine-grained, instance-level evaluation. Through an in-depth analysis of current LVLMs, we identify several key findings: and user-friendly in interactive
I) Visual prompts both more scenarios than text prompts, leading to significantly improved model performance and enhanced user experience. 2) Models reasonably capable of zero-shor understanding of visual prompts, but struggle with spatiotemporal understanding. Even ol achieves only 71.8%, far below the human expert score of 88.3%, while most open-source models perform below 60%. 3) LVLMs exhibit pervasive hack phenomena in video question answering. which intensifyy with longer videos and lower frame sampling density, artificially inflating performance scores. We antici- pate that V2P-Bench will not only shed light on these challenges but also serve as foundational tool for advancing human- buman-model model interaction. The code and datasets are available at /ttps: //gpthbb..ttttt../ com/ wwwtiexinqui...o wee.xing//vppbbectee
INTRODUCTION
Large Vision-Language Models (VLMs) have recently achieved remarkable success across wide
range of multimodal tasks, including image captioning (Zeng et al., (025b), visual question answering
Q al. 2025; Pan al. (2025; Huang al.
2025 2026a). self self-supervision (Zeng et al. 2025a).
deep research (Huang et al. [022bb) Zeng et al. 2026). document understanding ( (Wang et al. 2025
multimodal generation (Zhang et al. 2025b c Ma et al.] 2025 Zhang et al.) 2025a). retrieval- augmented generation (Wang et al. 2026), etc. further extends the model's versatility across general tasks. Notable models like Gemini-2.55Pa (Team al., 2024) and LLaVA- Video (Zhang et al.
2024b) have set new performance benchmarks. In response, numerous benchmarks have emerged to evaluate these models comprehensively across diverse tasks (Li et al., 2024c) Mangalam et al. 2023 Fu et al. 2024). These benchmarks support LVLM evaluation, enabling nuanced assessments of their strengths and weaknesses real-world applications. This expanding landscape fosters rigorous testing and drives further innovation in LVLM development.
However, most benchmarks utilize texts for human-model interaction, which inevitably introduces certain inherent limitations. As shown in Figure I text prompts usually fail to provide precise spatial
Corresponding author

---

Published as a conference paper at ICLR 2026
Complex Multi- Targets
Remarkably Similar Targets
Unknown Target Attributes
Visuel prompts eliminates the redundont encoding and decoding process user and model respectively, making friendly for both.
person outlined doing?
Where the girl outtined stonding at the beginning of the vides?
What is the use of the outling marking tool according to the video?
The person outlined yellow is wearing a long black gown with a simple yet elegent design. She...
The girl out lined yellow stands on the left during the introduction segment at the beginning.
The tool with the ou outline morking is useful for blowing dust off delicate surfoces, such os..
Pure text prompts requires complex reference language for referning, making humon-model interaction unfriienly.
What is the seveniii per person from the left in the first row stage doing
In the middle the video, where wos the girl on the right standing baginning?
What is the use of the fourth tool from the left the demonstrecion at the beginning of the video?
the beginning of the video?
The seventh person from the left in the first row on stage wearing black tuxedo, paired with...
I'm unable determine the positions individuals pasittions one constantly changing
The fourth tool from right left a pair pliers. Its primary ane to grip, band, and out objecta (
Figure I: Comparison of text prompts and visual prompts for users and models. Pure text prompts suffer from complex encoding and decoding, requiring users to describe targets with intricate language and models to infer the intended referents, which often leads misalignment, especially in complex scenes with multiple or similar targets. In contrast, visual prompts directly indicate targets on video frames, providing a more intuitive and user- and model-friendly interaction.
and temporal references, resulting in difficulties when assessing the ability of LVLMs to understand specific areas moments in videos, particularly in complex multi-object scenarios. For users, significant amount of referential language is required to specify targets. For the model, it first needs to comprehend the user's referential language, making it prone to confusion at the initial step.
In contrast, as a frontier approach to multimodal human-model interaction, visual prompts offer simpler and more precise mechanism that enhances model understanding and better aligns with human intuitive cognition. Prior works (Cai et al. [0244] Yang et al.] 2023) explore visual prompts in the image domain and demonstrate their advantages over text- bextd prompts. However, limited studies in the video modality hinder further advances in multimodal human- interaction.
To bridge this gap, we propose V2P-Bench, comprehensive benchmark for evaluating LVLMs' video understanding in human-model interaction scenarios. As shown in Figure 2 V2P-Bench covers three main tasks, twelve categories, twenty video types, eight visual prompt types, and durations ranging from three seconds to two hours. Each query contains single visual prompt annotation, emphasizing fine-gre graineai spatial and temporal understanding to comprehensively assess LVLMs' video capabilities. All videos are manually curated to ensure high quality.
We conduct a comprehensive evaluation on V2P-Bench. user study comparing visual and text prompts shows that visual prompts significantly improve both model performance and human- interaction. We further evaluate 15 LVLMs, including 3 closed-source and 12 open-soure models. Results reveal that even state-of-the-ar models perform poorly (e.g., o1 (OpenAl 2024) achieves 71 71.8%), far below the human expert score of 88.3%, highlighting limitations in video visual prompt understanding. Additional analysis exposes widespread hacking behaviors in video question answering, which worsen with longer videos and lower frame sampling rates.
In nutshell, our contributions are as follows:
V2P-Bench has been meticulously designed, comprising twelve categories covering wide range of video types and diverse visual prompts. Collection and annotation process undergoes rigorous human validation, aiming to provide the community with high-quality benchmark for multi-mode human-model interaction.
We demonstrate the superiority of visual prompts over text prompts. Experimental results reveal that current models exhibit substantial limitations in comprehending video visual prompts and display evidence of hacking behaviors in video question-answerti tasks.
V2P- BeP-Bench pioneeringly applies visual prompts in video understanding evaluation for numan--oded interaction, addressing critical limitations in existing text-based evaluation frameworks. We seek to advance the field of vidco visual prompt understanding evaluation and establish foundation for more intuitive human-model interaction.
2

---

Published as a conference paper at ICLR 2026
Feattuee Mapping
Cousol Relstionship
Object Direction
Car
Avar
Arta
85
64.8
Movie
Sports
Understanding
Attriute
Trovel
VLOG
Fashion
News
Purcaph Bench
ae
Counterfactual
84 Object
Animation
MIVU
Video HAEE
Music
Inference
Attribute
bacumentory
Television
Science
681
644
Video Sons
Cooking
Egecantric
Life
Temporal
Forward
97
84.
727
71.
92.0
Counting
General
Genee-255 5-Pro
Internit.360
(LeVA.OV.1
5-VL-288
Reverge Tempord
Spatial Relationship
Action Sequence
Figure 2: (Left) Datasets and categories. Our dataset is derived from twelve datasets and contains twenty restructured categories. (Right) Performance radar chart. We report the performance of different models on by dimension. SOTA for each dimension is given.
2
RELATED WORK
2.1 LVLMS FOR VIDEO UNDERSTANDING
The rapid development of masee-based LVLMs (Liu et al. 2024b 2023 2024a Li et al. 2024a Chen
jtt al. 2024a d) Bai et al. 2023) has significantly enhanced the potential of video understanding and
question answering tasks, injecting new vitality into the field of artificial intelligence. VideoChat (Li et al. 20235) and Video-ChatGPT (Maaz et al., (023) are preliminary attempts in the realm of video understanding. Notable recent works include Cog VLL22MVV (Hong et al., 2024), InternVL2 (Chen et al. (224dd) and LLaVA- Video (Zhang et al., 2024b), which treat videos as sequences of images and leverage the powerful image comprehension capabilities to process video modality. Furthermore, the high computational and memory demands required for handling high frame rates and long videos have spurred advancements video compression technologies. For instance, Intern Video2 (Wang al. 2024c) and Video-LLaMA (Zhang et al. 2023) utilize QFormer (Li et al., (202aa) for efficient video feature extraction. Despite promising results, current LVLMs primarily rely on text prompts and still face challenges in fine-graines spatial and temporal understanding when given visual prompts as input.
2.2 VIDEO UNDERSTANDING BENCHMARKS
Traditional video understanding benchmarks, such MSRVTT-QA (tu al. 2017). ActivityNet-QU2 (Yu et al. 2019), and NEXT-QA (Xiao et al. 2021), focus on basic action recognition and video question answering. Recently, more benchmarks have been proposed. MMBench (Liu al. 2024c), SEED-Bench (Li et al. 2024b), and MVBench (Li et al. 2024c) mainly concentrate on short video clips for evaluation. EgoSchema (Mangalam
2023) and MovieQA (Tapaswi al. 2016) provide insights into narrative and thematic understanding. Long VideoBench (Wu et al. 2024), Video-MME (Fu et al. 2024). and LVBench (Wang al. (024b) offer longer videos and a broader variety of tasks. Additionally, recent works like INST-IT (Peng al.) 2024 and VideoRefer (Yuan et al. 2024) have introduced instance-level video question answering benchmarks. However, constrained by insufficiently robust and comprehensive, they still fail to adequately simulate real-world interactions. To address this limitation, we introduce -2P-Bench, allowing for a comprehensive evaluation of LVLMs that simulates multimodal muman--oohe interaction in realistic settings.
2.3 VISUAL PROMPT AS USER-FRIENDLY SOLUTION
Compared to text prompts, visual prompts offer a simple and effective means facilitating interaction between users and models. Visual prompts have been widely utilized in image understanding. ViP- LLaVA (Cai et al.] 2024) enhances the ability of LVLMs to comprehend local image regions by overlaying arbitrary visual prompts images. Draw-and-Understan (L et al. 2024 employs
3

---

Published as a conference paper at ICLR 2026
Table 1: Comparison of different datasets. Answer Type indicates whether the QA pair open- eOded(OE)) or multipe--choice(MCC) Multi Level represents whether the videos cover multiple duration levels. Open Domain indicates whether the video source is diversified. Visual Prompt represents whether the video contains visual prompts, Refer to Appendix for more features.
Benchmarks
Videos
Samples
Tacks
Arg duration
Amosation
Anewer Type
Multi Level
Open Domain
Visual Prompt
MSVI=QA Nf
2018
Asto
GI
MSRVTT- bar anm 23r
72821
Asto
GI
AuthiiyNa No 201
111.44
Manual
QE
NEXT-QA Xings Same
1000
39.5,
MC
Perceptro 133 mo 2004
11000
23.0s
Auso&Mannal
MC
MLVU (bon
112mnn
VOGIM
OF
1641
MC
HourVide
MC
LVRand Spareveue
Manudl
MC
EgeSchal
5063
180.0
MC
Vidco MMP
2700
17.0min
Manual
MC
INST Bench Peng 1025
1000
14.2s
Amso&Manal
OL&MC
VilcoRefar Benci 2024
13.8s
Manual
MC
Benchenccrsurs)
1372
12
Manual
MC
two-stage training approach to improve performance in pixel-level tasks. Set-of-Mark (Yang et al. 2023) introduces a novel visual prompting method to enhance the performance of LVLMs in visual localization tasks. However, research visual prompts in the context of video remains limited. INST-IT (Peng et al.) (204)) introduces instruction tuning with visual prompts to enhance instance- level understanding in LVLMs. VideoRefer Suite (Yuan al. 2024) creates large instance-level video instruction dataset to assist LVLMs in understanding spatiotemporal information in videos.
V2P BENCH
Table [ compares the key difference of V2P- Bench with previous benchmarks. The first two blocks list traditional pure text video understand- ing benchmarks, which primarily understand videos at holistic level and lack instance- level comprehension. Instance-level understanding is crucial as it focuses on the specific elements of greatest interest to us, requiring a more nuanced understanding and consistency.
Figure 3: Various visual prompt types.
As shown in the third block, although INST-IT Bench (Peng et al. 2024) and VideoRefer 2 (Yuan et al. 2024) use visual prompts for question-answeringg their: 7) visual prompts are annotated on all frames, rendering them unsuitable for human-mode interaction scenarios; 2) all data are sourced exclusively from VIS datasets (Yang et al., 2019 Pont- Pont-Tuset et al. [0177] Ding et al. [2023]. thereby exhibiting limitations in both robustness and comprchensiveness, meaning a) Shorter video durations( 14.2s and 13.8s); b) Single continuous shots; c) Limited video sources; d) Objects of interest not be suitable for question-answering
3.1 TASK DEFINITION
To facilitate fine-grained evaluation of LVLMs from various perspectives, we categorize the questions according to dimensions. Our dimension design strives to ensure both comprehensiveness and orthogonality, and ultimately includes three main tasks and twelve dimensions. Definitions for tasks and dimensions are as follows:
Basic Perception focuses on understanding the intrinsic attributes of objects and humans in the visual prompt. This task includes: /) Object Attribute (OA); 2) Human Attribute (HA). Temporal Understanding emphasizes comprehension and processing of dynamic information and chronological sequences in videos. This task includes: 1) Object Direction (OD); 2) Feature Mapping (FM); 3) Forward Temporal (FT); 4) Reverse Temporal (RT); 5) Action Sequence (AS); 6) Spatial Relationship (SR): 7) General Counting (GC).
High-level Reasoning extends beyond perception and temporal understanding, requiring logical inference and judgment to derive new conclusions or answers. This task includes: 1) Causal Relationship (CR); 2) Plot Understanding (PU); 3) Counterfactual Inference (CI).
4

---

Published as a conference paper at ICLR 2026
[OA] Object
(HA]
Object
What lether isede the rechorgle?
Whan is cortezn character with a red det no?
Which dinech on doss the dbject in red nore
Up to right. Up
(A) Natchhes (8) Peker
(A) Watching TV. Chalting Playing gomoe. Steeping
(C)
Shop.
Cornect Answer
Connect Answer
Mapping
Temporal
(RT) Temporal
Arong the four people norked, who the one stancing in The front the bogierin??
What did the person in the vide do ofter picking up the objozt pa paintedd the erro??
What did the solected person blue pincle before the perfornance bogl?
Persond (8) Person 2
Open and sot.
(A)
[C] Person (b) Parson
Search phone. (0) Correct Answer
ground.
(8) dewr.
(4) move desely to the camera.
Anawer (0)
(AS) Action Sequence
[SR] Spotial Relstionship
[GC] General Counting
Assording content:
How mory dees woman
pidding the
(A] The First
(8) The third.
Pottory.
Bookss
(6) Cornect Anawer
Feed.
(D) Cel phones.
(b) sitting not picch bell Cornect Asswer:
Asswer: (8)
[CR] Relationship
[PU] Understonding
Counterfactual Inference
Why are the two pooplc morket the box surn sunded by ammed man?
Why ore the two markel
What night happen you oncy stject srrow parning
the box surrcunded emaed mos?
(A) gooled
Becouse they killed poroonn. (C) Becouss they tiggenee the aermm desice.
recut
gool. Correct
her belance.
Answer: (C)
Correct Arawer (8)
Figure 4: Examples for each dimension of V2P-Bench.
Detailed elaborations and examples each dimension are provided Appendix A.2
3.2 DATASET CONSTRUCTION
3.2.
VIDEO COLLECTION
To create dataset, start from existing video benchmarks, they already have wide distribution of durations and diverse video types. We categorize the video durations into short, medium, and long videos. Additionally, reclassify all the videos, resulting twenty video categories, shown in Figure left). Our final dataset covers multiple video domains while maintaining relative balance video lengths.
3.2.2 QA AND VISUAL PROMPT ANNOTATION
After completing the collection process, we conduct the annotation of QA pairs and visual prompts to evaluate the capabilities of LVL.Ms video understanding with visual prompts. The annotation work is carried out by researchers proficient in English. To ensure data quality, provide thorough training for the annotators and conduct pilot annotations to assess their annotation capabilities.
While annotating the QA pairs, annotators are also required to perform visual prompt annotations. To better approximate real-world distributions, we adopt fully manual approach for annotating visual prompts, with each QA pair constrained to visual prompt frame. We predefine various types of visual prompts as follows: rectangle, mask contour, ellipse, triangle, scribble, point, arrow and SoM, shown in Figure
5

---

Published as a conference paper at ICLR 2026
3.2.3 POST PROCESSING
To ensure the quality of the dataset. conduct rigorous review of the annotated data after completion, including both VLMs and manual review processes.
Blind LLMs Filtering. Inspired by MMStar (Chen al. 2024b), perform plain text filtering on the QA pairs to ensure that questions could only be answered correctly by viewing the videos. Specifically, we provide only the pure text QA pairs the most powerful closed-source models GPT-4o and Gemini-2.5-Pre We set the sampling temperature to 0.2 and conduct two rounds of inference, then exclude samples for which both rounds provided correct answers.
Manual and Rule-based Review. After that, we perform male-based check and manual review of the remaining data. We exclude samples where the length disparity between different options was too significant. Additionally, we shuffle the order of multiple-choice options to ensure a uniform distribution of answer choices, thereby eliminating potential biases of different models toward specific options. The filtering ratios are reported in the Appendix AA.4 The final balanced proportions of the four options are 28.0%, 23 23.9%, 25.0% and 23.1%. Through this rigorous dataset construction process, strive to provide a high-quality, diverse, and balanced dataset that will benefit researchers in the field of buman-model interaction.
4 EXPERIMENTS
4.1 EXPERIMENT SETUP
Evaluation Models. To thoroughly evaluate the effectiveness of V2P-Bench, we conduct assessments
on multiple models, including 3 closed-source models: ol (OpenAI, 2024). GPT-4o (Hurst et al.
2024) and Gemini-2.5-PP (Team et al. 2024); 12 open-source models: LLa On-
72B)
(Li al. 2024a). Intern VL3-8B (Zhu et al. 2025), mPLUG-OW13-77 Ye al. 2024), LLaVA- Video 7B 72B) (Zhang et al. 2024b), MiniCPM-V V 2.6-8B Yao et al. 2024), Qwen2.5 VLLBBB
72B) Wang et al. (2224)), MiMo VL-BB (Team et al. 2025).
(Liu et al.) 2024a)
and NeX--NNTT-NSSTTET currently available.
(Peng et al. 2024). This essentially covers the mainstream L.L.Ms
Implementation Details. For open-source models, we select the sampling frame rate based on the context length of each model. Specifically, we average 128 frames from the video for LLaVA- One Vision 72B), Intern VL3-8B, mPLUG-OW1377B, LLaVA-Video(7B 72B). MiniCPM-V 2.6-8B, Qwen2.5-VL(7B 72B), MiMo-VL-7B, 4 frames for LLaVA-NeXT-7B and -LXT--NeXT-ISSTIIT 7B. For ol and GPT-4o, we average 64 frames. For Gemmini-2.5-Pre the raw video was uploaded directly without audio track. The visual prompt frame is placed after the video for all models. For other hyperparameters, we follow the settings in VLMEvalKit (Duan et al.) 2024).
4.2 QUANTITATIVE RESULTS
Table prresent the comprehensive evaluation results of V2P-Bench across different dimensions and video durations. These results encompass the performance of human experts, the blind answering task, and evaluations of 15 different models. We can conclude that ol achieves the highest overall score; however, its performance is not consistently superior across all dimensions, particularly in Object Direction and Action Sequence. As shown at the top of Table 2 human experts achieve an accuracy of 88.3% representing the upper bound of human performance. For the blind answering task, we observe that all three models perform below 10% on this task, with GPT-4o at 1 1.4%, Gemini- Pr5-Pro at 9.6%, and Qwen2.5- VLL-BB 3.0%. This demonstrates that our post-process processing pipeline effectively filters out commonsense-bann based questions, thereby ensuring high quality and robustness.
Below we summarize our key findings as follows:
Which prompt type works better for humans and models? Most benchmarks rely solely on text prompts, requiring models to answer questions based on textual questions and visual context. To investigate which type of prompt(textual or visual) is more conducive in human model interaction scenarios, we conduct a prompt comparison experiment and a real-user study. Results are reported in

---

Published as a conference paper at ICLR 2026
Table 2: Evaluation Results on V2P-Bench across Dimensions. We report results for 12 open-source models, 3 closed-source models, 3 blind answering results and human performance on V2P-Bench across dimensions. The best results are bold and the second- -est are underlined.
Method
OA
HA
OD
FM
CR
PU
CI
FT
RT
AS
SR
GC
Avg
Human Performance
92.2
91.7
84.8
89.5
85.7
83.2
91.9
87.4
84.1
75.4
92.0
95.8
88.3
Pure Text as Input
GPT-4o
2.3
1.2
0.0
4.6
1.8
1.7
0.0
0.0
4.6
1.4
Gemini-2.55 P.5-PPo
15.6
12.0
6.5
3.9
9.2
10.9
8.1
5.2
13.6
1.8
8.0
7.4
9.6
Qwen2.5-VL--225
3.9
4.0
0.0
0.0
7.3
3.6
0.0
0.0
6.8
5.4
0.0
0.0
3.0
Closed
Closed-souree source Models
85.2
78.4
23.1
78.1
71.6
78.7
69.1
73.1
50.0
64.1
51.2
71.8
GPT-4o
76.6
68.9
41.3
60.8
67.0
73.3
67.6
68.1
70.5
50.0
54.0
48.4
65.4
Gemini-2.5-PP
84.0
72.4
68.2
71.8
75.0
73.3
22.6
66.7
72.7
47.4
67.5
63.6
69.8
Open- source Models
LLaVA-NeXI-7B
56.6
34.8
43.0
48.6
42.6
42.2
28.
42.0
30.5
46.0
LLWA-NCXT-INSTIIT-B
57.4
58.4
26.1
42.4
43.0
49.2
31.6
49.2
42.2
26.3
42.0
27.4
46.3
LLaVA-OV-7B
57.1
52.1
28.3
47.1
63.8
59.1
41.0
42.1
35.6
63.2
62.8
43.2
52.8
LLVVA-OV72BB
65.5
59.9
34.7
47.0
63.8
43.2
38.5
50.0
41.1
66.3
66.9
45.9
56.7
IntemVLL3-8B
73.9
39.1
60.8
58.1
65.9
41.0
52.6
41.1
61.1
69.7
64.9
61.7
mPLUG-OW[3-7B
61.3
54.4
28.3
60.0
50.0
51.3
60.5
34.4
55.8
58.6
37.8
$2.6
LLaVA-Video-72
60.5
58.1
37.0
49.0
62.9
54.5
41.0
52.6
48.9
57.9
56.6
40.5
54,8
LLaVA- 7ideo-
62.2
60.8
30.4
54.9
61.0
54.5
43.6
47.4
42.2
70.5
71.0
59.5
58.6
MiniCPM-V 2,6-8B
68.9
59.4
26.1
56.9
58.1
50.0
33.3
50.0
34.4
57.9
67.6
43.2
55.3
Qwen2.5-VLL-2
60.5
56.7
17.4
45.1
47.6
40.9
52.6
32.2
47.4
50.3
35.1
48.1
Swen2.5-VI-7225
69.7
72.4
43.5
52.9
49.5
59.1
53.8
55.3
44.4
57.9
64.1
51.4
59.8
MiMe-VL.
67.2
57.6
37.0
45.1
47.6
52.3
59.0
55.3
41.1
50.5
61.4
43.2
53.8
Table 3: Comparison of text and visual prompts for humans and models.
Table We can draw the following observations: (1) Vi- sual prompts are more model-friendly than text prompts. As shown in Table [aa] simply converting visual prompts into text prompts leads to a substantial drop in accuracy across all participating models, with the most pronounced decline 15.1% observed in Gemini-2.5-PPro This is because text prompts require the model to decode the target from the text, which increases the difficulty of comprehension. Moreover, textual references can sometimes be ambiguous, illustrated in Figure [ In contrast, visual prompts can precisely localize the target within video frames, bypassing both the user's need to encode intentions in text and the model's need to decode them. (2) Visual prompts are more user-friendly than text prompts. We recruit 20 volunteers to participate in the experiment. Specifically, the interac- tion process consists of: watching a video, formulating a meta-question, rewriting the question into both text and a visual-pronme prompt version, completing the QA session, and then indicating their preference(text record the order which users write the text version and the visual prompt version of each question. Each participant is required to produce 10 text and 10 visual prompt questions. All questions are open-ended but designed have unique correct answers to minimize potential subjectivity or eval- uation bias. The correctness of model responses is assessed by our annotators. All videos are randomly sampled from the V2P-Bench dataset we construct. Throughout the study,
(a) Model performance.
Model
Text
Visual
GPT-4o
53.0
65.4
Gemini-2.5-Pro
54.7
69.8
LLaVA- Video-BB
42.4
54.8
Qwen2.5.VL VL-BB
43.1
52.4
Mimo VL-BB
46.7
55.6
(b) User experience study.
Metric
Text
Visual
Acc
57.0
69.5
Cost Time
25.2
18.1
User Satisfaction
5.3
7.5
(c) User preference for prompt type.
visual prompts). We also
Preference
Text
Visual
None
Nums
57
129
14 7.0%
Percentage
28.5%
64.5%
(d) Question order on user responses.
Order
Text First
Visual First
Nums
64
136 68.0%
Percentage
34.0%
Gemini-2.5-PPr serves as the conversational agent. To systematically evaluate performance, we record five key metries: answer accuracy (max 100), completion time (seconds), user satisfaction (max 10), user preference, and question order. As shown in Table 3bl users complete tasks more quickly in the visual prompt interaction setting, with the time reduced from 25.2s to 18.1s, saving 7.7s on average. In addition. the improvement in model performance means more satisfactory responses for users. Together, these factors contributed to an average user satisfaction score of 8.2. 2.4 points higher than under the text-only prompt condition. This indicates that visual prompts provide clear
7

---

Published as a conference paper at ICLR 2026
Hack Phonomone Perfornance on 2-5V--2---
Phenoment Performance Mino-VL-78
144
14
10
3) 12
Prens
francr
Figure 5: Hack Phenomena and Model Performance on V2P-Bence
advantages in muman-model interaction, improving both overall satisfaction and efficiency. Besides, 64.5% of participants explicitly preferred using visual prompts, while only 28.5% preferred text-only input, and 7.0% had no preference. What's 's more, 68% of the questions were initiated using visual prompts. This indicates that for real users, visually selecting the target is more intuitive and less effortful than crafting natural- languagg description, especially for objects, locations, or people that are difficult to describe precisely with text. Refer to Appendix E..2 for the implementation details.
Can the models effectively comprehend visual prompts, and how do they perform? In Table 2 we report the performance of human experts and models across dimensions, leading to the following observations: (1) Models are reasonably capable of zero-shot understanding of visual prompts. Except for LLaVA-NeXT-INST-TTS
the open-source models have not been trained on visual prompt datasets, yet all achieve performance above 45 points, surpassing the random-guess baseline (25%) by over 20%. Notably, in the Basic Perception task, every model achieves over 50%. This phenomenon arises because LVLMs are typically trained massive image- and video-text text pairs, enabling them to learn broad risual-semanti associations and thus exhibit strong zero-shot transfer capabilities. (2) Models struggle with spatiotemporal understanding. Object Direction requires models to identify the direction of an object's movement, while Spatial Relationship requires models understand the dynamic spatial positions. The average scores on the two dimensions are only 34.4% and 45.7%, respectively, indicating that spatiotemporal tasks remains a weakness of current models and requires further improvement. (3) Closed-source models and large-scale parameter models possess stronger capabilities. As shown in Table 2 three closed- source models outperform all open- source counterparts. This disparity highlights the persistent challenges in advancing open-source models. Moreover, for the same open-source model with varying scales, for example Qwen2.5- VL 7B and 72B, the larger-parameter variant demonstrates stronger capabilities and achieves higher scores, which is consistent with the established scaling laws of LVLMs.
Does the expert model exhibit the anticipated superior performance? ? tuned on INST-IT dataset based on LL. yet the results show only a marginal improvement of 0.3%. There are two main reasons for this limited performance gain: (1) Limited diversity of visual prompt types. ALaVA-NeXT-INSTA.
is fine-
is trained exclusively on SoM data. As observed in Table 13 while the model achieves a 15.0% improvement on SoM prompts, its performance drops on other types of visual prompts, indicating forgetting phenomenon. This suggests that models' training should cover a broad range of visual prompt types to better meet the demands of real-world scenarios. (2) Differences in data format. Unlike V2P-Bench, INST-IT dataset annotates visual prompts on every sampled frame, as shown in Figure 9 (left). This redundant annotation does not account for the constraints of user interactions in real numan-moder interaction scenarios, which contributes to the suboptimal performance. A detailed analysis of this phenomenon provides important insights for building future datasets and training strategies. Refer to Appendix Dooo more information.
Are vision-language models truly understanding videos, merely exploiting hacks?
Due to the sparsity of frame sampling and the upper limits of model perception, the models may guess answers rather than rely on vi- sual context. We randomly shuffle the videos and questions and performed inference with 0wwn2.55VL-77 and MiMo-VL-7B. As shown in Table only 6.4% and 3.9% of the cases trigger a refusal, respectively. while all others follow the instructions to select op- tion. This indicates that current models are trained to be "test-takers" often neglecting basic factual information. To investigate to what extent models exhibit hack behavior
Table 4: Results with shuffled video and question pairs.
Model
Trigger Ratio
Qwen2.5-VL-7B MiMo-VI.-BB
6.4%
3.9%
8

---

Published as a conference paper at ICLR 2026
on V2P- BeP-Bench, conduct experiments in which the models are instructed to reject when they cannot reach a conclusion. Results are reported in Figure [5] From our analysis, we draw the following conclusions: (1) Hack Phenomena exist in video benchmark evaluations. Both Qwen2.5-VL-7B and MiMo-VL--7B exhibit positive Hack Ratios across all settings, accompanied by varying degrees of performance degradation. (2) Longer videos exacerbate Hack Phenomena. For instance, under a 4-frame sampling setting, the Hack Ratios of Qwen2.5-VL-7B for short, medium, and long videos are 11.1% 1.11, 23.0%, and 33.8%, respectively. This pattern is consistent across other experiments as well. (3) Fewer sampled frames increase Hack Phenomena. For example, when reducing the number of sampled frames for Owen2.5-VLL77 from 128 to 4, the average Hack Ratio steadily rises from 8.0% to 18.7% 1877. The presence of Hack Phenomena can be attributed to three factors: (1) Insufficient information, as existing sparse sampling strategies fail to provide the model with enough information to get the answer; (2) Limited model perception, where excessive visual context may overwhelm the key information; (3) Training strategy. Models are trained as instruction inslloction-folloiinng agents, leading them to prioritize following instructions over grounding their responses in factual information. Efforts should be made to actively enhance model capabilities and explore novel visual representation mechanisms that capture broader range of effective context. When visual context is insufficient, models should proactively request clarification. We also report the results of other benchmarks in Appendix L.3 which are consistent with our observation on V2P-Bench.
How LVLMs are robust to varied video duration?
Table 5: Evaluation results on V2P-Bench across durations. The best results are bold and the second-best are underlined.
In Table we compare the performance of differ- ent models on short, medium, and long videos, from which we can conclude that: Performance degrades as video length increases. All models exhibit a sig- nificant drop in performance. For example, ol's 'ss per- formance on long videos decreases by 23.5% com- pared to the medium. This decline is primarily due to the increased sparsity of frame sampling as video length grows, which reduces the amount of effective visual content. Such sparsity prevents models from retaining all relevant "ssual-emmatic information, thereby hindering accurate predictions. However, all models perform worse on short videos compared to medium-length videos unexpectedly. This is likely because over half of the short videos are drawn from Perception Test, MVBench and TVBench, which have high information density and contain challeng- ing questions related to spatiotemporal questions.
Method
Short
Medium
Long
Avg
Human Performance
91.6
87.3
$4.0
$8.3
Pure Text Ispur
GPT-4o
1.1
1.9
1.6
1,4
Gemini-2.5-P P.5-Pro Owen2. 5-VL-72B
10.0
10.6
8.1
9.6
2.7
5.1
2.5
3.0
Clowd-sounee Modaly
ol
75.2
83.9
60.4
71.8
GPT-4o
67.3
70.8
59.3
65.4
Gemini-2.5.Prr
23.8
86.3
54.5
69.8
Open soueee Medels
LLaVA-NeXT-72 7B
47.0
47.1
43.8
46.0
LLLVA.NeXT.INTT.TT.
48.6
51.1
39.5
LLaVA-OV-78
51.3
63.0
47.3
52.8
LLaVA-OV-72B
51,5
70.4
19.0
56.7
55.6
mPL MPL.DGG--W15-78
52.2
62.5
44.9
52.6
LLaVA Video-7B
51.1
65.7
46.5
51.8
LLaVA 72d
54.3
58.6
MiniCPM-V 26-38
53.3
66.2
50.2
58.3
V-V-225-VL-7B
48.0
53.2
43.6
48.1
Qww-2255VVL-2BB
62.4
50.2
MiMo-VL-7R
56.8
58.8
42.4
53.8
How does the structure of visual prompts affect model performance? To investigate the effect of the visual query structure itself on the performance of visual prompts, we randomly sampled 217 data instances. For each muestion-answer pair, we annotated the corresponding visual prompt frames with multiple types of visual prompts, while strictly keeping all other settings identical, including the number of frames, prompts. The results are shown in Table G
For the same visual prompt type, the doodle-style shapes are slightly weaker than the standard shapes. When switching from standard shapes to doodle shapes, the performance of Owen2.5-VL.-77 and MiMo-VL-71B decreases by 0.7% and 0.8%, respectively. This is reasonable, since most training data are synthetic, whereas hand-drawn doodles often have unstable boundaries, making it harder for the model to extract consistent visual prompts. Regardless of whether standard or doodle shapes are used, the overall performance trend remains Rectangle SoM > Arrow. Among these, rectangles appear most frequently in training data and can fully enclose the target region, providing a stable and explicit spatial localization signal. In contrast, arrows are typically smaller and carry lower information density, making it more difficult for the model to capture key spatial relationships and therefore resulting in the weakest performance.
Table 6: Performance on Different Visual Prompts.
Standard Shapes
Doodle Shapes
Mothod
Rectangle
Anrow
SoM
Rectangle
Arrow
SoM
Qwen2.5-VL-7B
47.3
43.6
45.1
46.7
42.9
44.4
-L- 54.2
51.2
52.7
S3.6
50.3
51.9

---

Published as a conference paper at ICLR 2026
4.3 ERROR ANALYSIS
In this section, we analyze the error patterns of LVLMs in video visual prompt understanding. We examine and categorize 470 model predictions from Qwen2.5-VL-7BB identifying three main representative eπror types. These patterns are illustrated in Figure to Figure 12 and their
44%
hot Plece wed
435
Iaformer us
Haae Ereer etas
Kaurniig Erenl
13.3%
Vincal Froept 14.95
distribution is presented in Figure
Spetil Erre
Provided with sufficient visual context, the model still produces errors due to deficiencies in its perception capabilities. Perception errors account for 57.7%, making them the most prevalent error type.
Perception Error.
57.7%
Courtig 47%
Teepord
25.1%
The model demonstrates a range of logical reasoning failures, including plot understanding. counterfactual reasoning and causal errors. Some of these errors arises from deficiencies in perception, which further undermine the model's ' reasoning ability.
Reasoning Error.
Figure 6: Distribution of Error Types.
Hack Error. Constrained by the sparse frame sampling strategy and the upper limit of the model's perceptual capability, the model fails to recognize sufficient visual semantics and consequently resorts to arbitrary guessing, which ultimately leads to erroneous predictions.
Analyzing and mitigating these errors is crucial for enhancing the performance of LVLMs in video visual prompt question understanding. This analysis provides opportunity target specific error types, thereby improving the model's overall capability.
5 CONCLUSION
In this study, we introduce derstanding capabilities of LVLMs through visual prompts human-model interaction scenarios. Through a rigorous construction and evaluation process, V2P-Bench enables systematic evaluation and useful insights into the issues within the current models. Our experiments demonstrate three key findings: 1) visual prompts outperform text prompts, substantially improving accuracy and interaction efficiency: 2) a notable performance gap remains between LVLMs and human experts, especially in spatiotemporal understanding: and 3) Hack Phenomena are prevalent, exacerbated by longer videos and sparser frame sampling. These results underscore the need for more robust evaluation protocols and model designs. We envision V2P-Bench as both diagnostic tool and a roadmap for advancing LVLMs toward more reliable, human-aligned video understanding and interaction.
a comprehensive benchmark for evaluating the video un-
ACKNOWLEDGEMENTS
This work was supported by the Anhui Provincial Natural Science Foundation under Gran12108085UD12. We acknowledge the support of GPU cluster built by MCC Lab of Infor- mation Science and Technology Institution, USTC. The AI driven experiments, simulations and model training were performed on the robotic Al-Scientist platform of Chinese Academy of Sciences.
7 ETHICS STATEMENT
This work complies with ethical research standards in data collection, model training, and evaluation. All datasets used in this study are publicly available research datasets, collected and released under their respective licenses. No private personally identifiable information (PII) used.
REPRODUCIBILITY STATEMENT
We committed to ensuring the reproducibility of our results. All prompts used during the evaluation are provided in the appendices. In addition, the datasets and evaluation scripts used in this paper have been publicly released.
10

---

Published as a conference paper at ICLR 2026
REFERENCES
Jinze Bai, Shuai Bai, Shusheng Yang, Shijie Wang, Sinan Tan, Peng Wang, Junyang Lin, Chang Zhou, and Jingren Zhou. Qwen-vl: frontier large lannuale model with versatile abilities. arXiv preprint arXiv:2308. 0..3308.22666 /2966, 2023.
Mu Cai, Haotian Liu, Siva Karthik Mustikovela, Gregory Meyer, Yuning Chai, Dennis Park, and Yong Jae Lee. Vip-Ilava: Making large multimodal models understand arbitrary visual prompts. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 12914-12922 2024.
Nicolas Carion, Laura Gustafson, Yuan-Ting Hu, Shoubhik Debnath, Ronghang Hu, Didac Suris, Chaitanya Ryali, Kalyan Vasudev Alwala, Haitham Khedr, Andrew Huang, Jie Lei, Tengyu Ma, Baishan Guo, Arpit Kalla, Markus Marks, Joseph Greer, Meng Wang, Peize Sun, Roman Rädle, Triantafyllos Afouras, Effrosyni Mavroudi, Katherine Xu, Tsung- Han Wu, Yu Zhou, Liliane Momeni, Rishi Hazra, Shuangrui Ding, Sagar Vaze, Francois Porcher, Feng Li, Siyuan Li, Aishwarya Kamath, Ho Kei Cheng, Piotr Dollár, Nikhila Ravi, Kate Saenko, Pengchuan Zhang, and Christoph Feichtenhofer. Sam 3: Segment anything with concepts, 2025. URL
https://arxiv www. httpg///bb//211.... 2511 16719
Keshigeyan Chandrasegaran, Agrim Gupta, Lea M Hadzic, Taran Kota, Jimming He, Cristóbal Eyzaguirre, Zane Durante, Manling Li, Jiajun Wu, and Li Fei-Fei. Hourvideo: 1-hour video- language understanding. arXiv preprint arXiv:2411.04990 2024.
Lin Chen, Jinsong Li, Xiaoyi Dong, Pan Zhang, Conghui He, Jiaqi Wang, Feng Zhao, and Dahua Lin. Sharegpt4v: Improving large multi-modal models with better captions. In European Conference on Computer Vision, pp. 370-387. Springer, 2024a.
Lin Chen, Jinsong Li, Xiaoyi Dong, Pan Zhang, Yuhang Zang, Zehui Chen, Haodong Duan, Jiaqi Wang, Yu Qiao, Dahua Lin, et al. Are we on the right way for evaluating large vision-language models? arXiv preprint arXiv:2403. 20330, 2024b.
Lin Chen, Xilin Wei, Jinsong Li, Xiaoyi Dong, Pan Zhang, Yuhang Zang, Zehui Chen, Haodong Duan, Bin Lin, Zhenyu Tang, Sharegpt4video: Improving video understanding and generation with better captions. arXiv preprint arXiv:24066 04325,, 2024c.
Zhe Chen, Jiannan Wu, Wenhai Wang. Weijie Su, Guo Chen, Sen Xing, Muyan Zhong, Qinglong Zhang, Xizhou Zhu, Lewei Lu, et al. Internvl: Scaling up vision foundation models and aligning for generic visual-linguistic tasks. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 24185 24198, 2024d.
Henghui Ding, Chang Liu, Shuting He, Xudong Jiang, and Chen Change Loy. Mevis: A large-scale benchmark for video segmentation with motion expressions. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pp. 2694-2703, 2023.
Haodong Duan, Junming Yang, Yuxuan Qiao, Xinyu Fang, Lin Chen, Yuan Liu, Xiaoyi Dong, Yuhang Zang, Pan Zhang, Jiaqi Wang, al. Vlmevalkit: An open-source toolkit for evaluating large multi-modality models. In Proceedings of the 32nd ACM International Conference on Multimedia, pp. 11198 1201, 2024.
Chaoyou Fu, Yuhan Dai, Yongdong Luo, Lei Li, Shuhuai Ren, Renrui Zhang, Zihan Wang, Chenyu Zhou, Yunhang Shen, Mengdan Zhang, et al. Video-mme: The first-ever comprehensive evaluation benchmark of multi-modal llms in video analysis. arXiv preprini arXiv:2405.21073 2024.
Wenyi Hong, Weihan Wang, Ming Ding, Wenmeng Yu, Qingsong Lv, Yan Wang, Yean Cheng, Shiyu Huang, Junhui Ji, Zhao Xue, et al. Cogvlm2: Visual language models for image and video understanding. arXiv preprint arXiv: 16500, 2024.
Shiting Huang, Zhen Fang, Zehui Chen. Siyu Yuan, Junjie Ye, Yu Zeng. Lin Chen, Qi Mao, and Feng Zhao. Critictool: Evaluating self- self-critique capabilities of large language models in cool-calling error scenarios. Proceedings of the 2025 Conference on Empirical Methods in Natural Language Processing, pp. 26683-26699 2025.
11

---

Haotian Liu, Chunyuan Li, Qingyang Wu, and Yong Jae Lee. Visual instruction tuning. Advances in Published as a conference paper at ICLR 2026
Shiting Huang, Zecheng Li, Yu Zeng, Qingnan Ren, Zhen Fang, Qisheng Su, Kou Shi, Lin Chen, Zehui Chen, and Feng Zhao. Internaiizing meta-experience into memory for guided reinforcement learning in large language models. arXiv preprint arXiv:2602.10224, 2026a.
Wenxuan Huang, Yu Zeng, Qiuchen Wang, Zhen Fang, Shaosheng Cao, Zheng Chu, Qingyu Yin, Shuang Chen, Zhenfei Yin, Lin Chen, et al. Visionndeepreseaacch: Incentivizing deepresearch capability in multimodal large language models. arXiv preprint arXiv:2601 22060, 2026b.
Aaron Hurst, Adam Lerer, Adam Goucher, Adam Perelman, Aditya Ramesh, Aidan Clark, AJ Os- trow, Akila Welihinda, Alan Hayes, Alec Radford, et al. Gpt-4o system card. arXiv preprint arXiv:2410.21276. 2024.
Bo Li, Yuanhan Zhang, Dong Guo, Renrui Zhang, Feng Li, Hao Zhang, Kaichen Zhang, Peiyuan Zhang, Yanwei Li, Ziwei Liu, et al. Llava-onevision: Easy visual task transfer. arXiv preprint arXiv:2408. 03326, 2024a.
Bohao Li, Yuying Ge, Yixiao Ge, Guangzhi Wang, Rui Wang, Ruimao Zhang, and Ying Shan. Seed-bench: Benchmarking multimodal large language models. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 13299-13308, 2024b.
Junnan Li, Dongxu Li, Silvio Savarese, and Steven Hoi. 2 Bootstrapping language- image pre-training with frozen image encoders and large language models. In International Conference on Machine Learning, pp. 19730-19742. PMLR, 2023a.
KunChang Li, Yinan He, Yi Wang, Yizhuo Li, Wenhai Wang. Ping Luo, Yali Wang, Limin Wang, and Yu Qiao. Videochat: Chat-centric video understanding. arXiv preprint Stti:::3330063555 2023b.
Kunchang Li, Yali Wang, Yinan He. Yizhuo Li, Yi Wang, Yi Liu, Zun Wang, Jilan Xu, Guo Chen, Ping Luo, et al. Mvbench: A comprehensive multi-modal video understanding benchmark. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp- 22195-22200 2024c.
Weifeng Lin, Xinyu Wei, Ruichuan An, Peng Gao, Bocheng Zou, Yulin Luo, Siyuan Huang, Shang- hang Zhang, and Hongsheng Li. Draw-and and-uunerstand: Leveraging visual prompts enable milms to comprehend what you want. arXiv preprint arXin:2403.20227,,20 2024.
Haotian Liu, Chunyuan Li, Yuheng Li, and Yong Jae Lee. Improved baselines with visual instruction tuning. arXiv preprint drXiw:2310.03744, 2023.
Haotian Liu, Chunyuan Li, Yuheng Li, Bo Li, Yuanhan Zhang, Sheng Shen, and Yong Jae Lee. Leava-next: Improved reasoning, ocr, and world knowledge, January 2024a. URL https://
Ilava-vl. ww..gitubbb......... ht/booog//o/----00 01 30- l-e-llllaaeeee
Neural Information Processing Systems, 36, 2024b.
Yuan Liu, Haodong Duan, Yuanhan Zhang, Bo Li, Songyang Zhang, Wangbo Zhao, Yike Yuan, Jiaqi Wang, Conghui He, Ziwei Liu, et al. Mmbench: Is your multi-modal model an all-around player? In European Conference on Computer Vision, pp. 1116-23..20 Springer. 2024c.
Xiaoxiao Ma. Haibo Qiu, Guohui Zhang, Zhixiong Zeng, Siqi Yang, Lin Ma, and Feng Zhao. Stage: Stable and generalizable grpo for autoregressive image generation. arXiv preprint (rXiv:2509.25027 2025.
Muhammad Maaz, Hanoona Rasheed, Salman Khan, and Fahad Shahbaz Khan. Video-chatgpt: Towards detailed video understanding via large vision and language models. arXiv preprint arXiv:2306.05424. 2023.
Muhammad Maaz, Hamoona Rasheed, Salman Khan, and Fahad Khan. Vidcogpt+: Integrating image and video encoders for enhanced video understanding. arXiv preprint arXiv:2406.0944 09418, 2024.
Karttikeya Mangalam, Raiymbek Akshulakov, and Jitendra Malik. Egoschema: A diagnostic benchmark for very long-form video language understanding. Advances in Neural Information Processing Systems, 36:46212-446 2023.
12

---

Published as a conference paper at ICLR 2026
OpenAl. Introducing openai o1, 2024., 2024. /ILLittps///opeaaa.com///
Yaning Pan, Qianqian Xie, Guohui Zhang, Zekun Wang, Yongqian Wen, Yuanxing Zhang, Haoxuan Hu, Zhiyu Pan, Yibing Huang, Zhidong Gan, et al. Mt-video-bench: A holistic video under- standing benchmark for evaluating multimodal Ilms in multi-turn dialogues. arXiv preprint arXiv:2510. 17722, 2025.
Viorica Patraucean, Lucas Smaira, Ankush Gupta, Adria Recasens, Larisa Markeeva, Dylan Banarse, Skanda Koppula, Mateusz Malinowski, Yi Yang, Carl Doersch, et al. Perception test: A diagnostic benchmark for multimodal video models. Advances in Neural Information Processing Systems, 36, 2024.
Wujian Peng, Lingchen Meng, Yitong Chen, Yiweng Xie, Yang Liu, Tao Gui, Hang Xu, Xipeng Qiu, Zuxuan Wu, and Yu-Gang Jiang. Inst-it: Boosting multimodal instance understanding via explicit visual prompt instruction tuning. arXiv preprini arXiv:2412.03560 2024.
Jordi Pont-Tuset, Federico Perazzi, Sergi Caelles, Pablo Arbeláez, Alex Sorkine -ornung, and Luc Van Gool. The 2017 davis challenge on video object segmentation. arXiv preprint arXiv: (XX4::770.00077 00675, 2017.
Yukun Qi, Yiming Zhao, Yu Zeng, Xikun Bao, Wenxuan Huang, Lin Chen, Zehui Chen, Jie Zhao, Zhongang Qi, and Feng Zhao. Ver-bench: comprehensive evaluation framework for video chain-of of- thothhtgtt reasoning. arXiv preprint arXiv:2504.07955 2025.
Makarand Tapaswi, Yukun Zhu, Rainer Stiefelhagen, Antonio Torralba, Raquel Urtasun, and Sanja Fidler. Movieqa: Understanding stories in movies through question-answerte In Proceedings of the IEEE Conference on Computer Vision and Puttern Recognition, pp. 4631-4640, 2016.
Core Team, Zihao Yue, Zhenru Lin, Yifan Song, Weikun Wang, Shuhuai Ren, Shuhao Gu, Shicheng Li, Peidian Li, Liang Zhao, Lei Li, Kainan Bao, Hao Tian, Hailin Zhang, Gang Wang, Dawei Zhu, Cici, Chenhong He, Bowen Ye, Bowen Shen, Zihan Zhang, Zihan Jiang, Zhixian Zheng, Zhichao Song, Zhenbo Luo, Yue Yu, Yudong Wang, Yuanyuan Tian, Yu Tu, Yihan Yan, Yi Huang. Xu Wang, Xinzhe Xu, Xingehen Song, Xing Zhang, Xing Yong. Xin Zhang, Xiangwei Deng, Wenyu Yang, Wenhan Ma, Weiwei Lv, Weiji Zhuang. Wei Liu, Sirui Deng, Shuo Liu, Shimao Chen, Shihua Yu, Shaohui Liu, Shande Wang, Rui Ma, Qiantong Wang, Peng Wang, Nuo Chen, Menghang Zhu, Kangyang Zhou, Kang Zhou, Kai Fang, Jun Shi, Jinhao Dong, Jiebao Xiao, Jiaming Xu, Huaqiu Liu, Hongshen Xu, Heng Qu, Haochen Zhao, Hanglong Lv, Guoan Wang, Duo Zhang, Dong Zhang, Di Zhang, Chong Ma, Chang Liu, Can Cai, and Bingquan Xia. Mimo-vl technical report, 2025. URL htttt:://arxxiii org/ abs/ 2506. 03569
Gemini Team, Petko Georgiev, Ving Ian Lei, Ryan Burnell, Libin Bai, Anmol Gulati, Garrett Tanzer, Damien Vincent, Zhufeng Pan, Shibo Wang, et al. Gemini 1.5: Unlocking multimodal understanding across millions of tokens of context. arXiv preprint arXiv:2403.05530 2024.
Peng Wang, Shuai Bai, Sinan Tan, Shijie Wang. Zhihao Fan, Jinze Bai, Keqin Chen, Xuejing Liu, Jialin Wang, Wenbin Ge, Yang Fan, Kai Dang, Mengfei Du, Xuancheng Ren, Rui Men, Dayiheng Liu, Chang Zhou, Jingren Zhou, and Junyang Lin. Qwen2-vl: Enhancing vision-language model's perception of the world at any resolution. arXiv preprint arXiv:2409.12199 2024a.
Qiuchen Wang, Ruixue Ding, Yu Zeng, Zehui Chen, Lin Chen, Shihang Wang, Pengjun Xie, Fei Huang, and Feng Zhao. Vrag-rl: Empower vision settt--peeeetnbbaseeeeeee rag for visually rich in- formation understanding via iterative reasoning with reinforcement learning. arXiv preprint arXiv:2505.22019 2025.
Qiuchen Wang, Shihang Wang, Yu Zeng, Qiang Zhang, Fanrui Zhang, Zhuoning Guo, Bosi Zhang, Wenxuan Huang, Lin Chen, Zehui Chen, et al. Vimrag: Navigating massive visual context in augmented generation via multimodal memory graph. arXiv preprint 127755 2026.
Weihan Wang, Zehai He, Wenyi Hong. Yean Cheng, Xiaohan Zhang, Ji Qi, Xiaotao Gu, Shiyu Huang, Bin Xu, Yuxiao Dong, et al. Lvbench: An extreme long video understanding benchmark. arXiv preprint 4rXiv:2406. 24X022406.08035.. 2024b.
13

---

Published as a conference paper at ICLR 2026
Yi Wang, Kunchang Li, Xinhao Li, Jiashuo Yu, Yinan He, Guo Chen, Baoqi Pei, Rongkun Zheng, Zun Wang, Yansong Shi, et al. Internvide02: Scaling foundation models for multimodal video understanding. In European Conference on Computer Vision, pp.
Springer, 2024c.
Haoning Wu, Dongxu Li, Bei Chen, and Jumnan Li. Longvideobench: A benchmark for long-context interleaved video-language understanding. arXiv preprint arXiv:2407.15754 2024.
Junbin Xiao, Xindi Shang, Angela Yao, and Tat-Seng Chua. Next-qa: Next phase of question- answering to explaining temporal actions. In Proceedings of the IEEE/CVF Conference Computer Vision and Pattern Recognition, pp. 9777-9786 2021.
Dejing Xu, Zhou Zhao, Jun Xiao, Fei Wu, Hanwang Zhang, Xiangnan He, and Yueting Zhuang. Video question answering via gradually refined attention over appearance and motion. In Proceedings of the 25th ACM International Conference Multimedia, pp. 1645-1653, 2017.
Jianwei Yang. Hao Zhang, Feng Li, Xueyan Zou, Chunyuan Li, and Jianfeng Gao. Set-of- mark prompting unleashes extraordinary visual grounding in gpt-4v, 2023. URL http://axxi. org/abs/2310. 1441, 2023.
Linjie Yang, Yuchen Fan, and Ning Xu. Video instance segmentation. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pp.
Yuan Yao, Tianyu Yu, Ao Zhang, Chongyi Wang, Junbo Cui, Hongji Zhu, Tianchi Cai, Haoyu Li, Weilin Zhao, Zhihui Hc, et al. Minicpm-v: A gpt-4v level milm on your phone. arXiv preprint arXiv:2408.0 01800, 2024.
Jiabo Ye, Haiyang Xu, Haowei Liu, Anwen Hu, Ming Yan, Qi Qian, Ji Zhang, Fei Huang, and Jingren Zhou, mplug-ow13: Towards long imago-sequence understanding in multi-modal large language models. arXiv preprint arXiv:2408.04488 2024.
Zhou Yu, Dejing Xu, Jun Yu, Ting Yu, Zhou Zhao, Yueting Zhuang, and Dacheng Tao. Activitynet-qa: A dataset for understanding complex web videos via question answering. In Proceedings of the AAAI Conference Artificial Intelligence, volume 33, pp. 9127-9134, 2019.
Yuqian Yuan, Hang Zhang, Wentong Li, Zesen Cheng, Boqiang Zhang, Long Li, Xin Li, Deli Zhao, Wenqiao Zhang, Yueting Zhuang, et al. Videorefer suite: Advancing -ptiial-tempora object understanding with video Ilm. arXiv preprint wxxiv2501.00099 2024.
Yu Zeng. Wenxuan Huang, Shiting Huang, Xikun Bao, Yukun Qi. Yiming Zhao, Qiuchen Wang, Lin Chen, Zehui Chen, Huaian Chen, et al. Agentic jigsaw interaction learning for enhancing visual perception and reasoning in vision-languag models. arXiv preprint arXiv:2510.01304. 2025a.
Yu Zeng, Yukun Qi, Yiming Zhao, Xikun Bao, Lin Chen, Zehui Chen, Shiting Huang, Jic Zhao, and Feng Zhao. Enhancing large vision- vision-language models with ultra-detailed image caption generation. In Proceedings of the 2025 Conference on Empirical Methods in Natural Language Processing, pp. 16670-26672) 2025b.
Yu Zeng, Wenxuan Huang, Zhen Fang, Shuang Chen, Yufan Shen, Yishuo Cai, Xiaoman Wang, Zhenfei Yin, Lin Chen, Zehui Chen, et al. Vieioenddepprseaacch benchmark: Rethinking visual and textual search for multimodal large language models. arXiv preprint arXiv: 02 185, 2026.
Guohui Zhang, Jiangtong Tan, Linjiang Huang, Zhonghang Yuan, Mingde Yao, Jie Huang, and Feng Zhao. Infoscale: Unleashing training- free variable-scaled image generation via effective utilization of information. arXiv preprint arXiv:2509.0142) 2025a.
Guohui Zhang, Hu Yu, Xiaoxiao Ma, Yaning Pan, Hang Xu, and Feng Zhao. Maskfocus: Focusing policy optimization critical steps for masked image generation. arXiv preprint 21772 2025b.
Guohui Zhang, Hu Yu, Xiaoxiao Ma. JingHao Zhang. Yaning Pan, Mingde Yao, Jie Xiao, Lin- jiang Huang, and Feng Zhao. Group critical-token policy optimization for autoregressive image generation. arXiv preprint 22.4452509.22485, 2025c.
14

---

Published as a conference paper at ICLR 2026
Hang Zhang, Xin Li, and Lidong Bing. Video-Ilama: An instruction-tuneel -uunde audio-visual language model for video understanding. arXiv preprint arXivv2.306. 02858, 2023.
Youcai Zhang, Xinyu Huang, Jinyu Ma, Zhaoyang Li, Zhaochuan Luo, Yanchun Xie, Yuzhuo Qin, Tong Luo, Yaqian Li, Shilong Liu, et al. Recognize anything: A strong image tagging model. In Proceedings of the IEEE/CVF Conference on Computer Vision and Puttern Recognition, pp. 1724 1732, 2024a.
Yuanhan Zhang, Jinming Wu, Wei Li, Bo Li, Zejun Ma, Ziwei Liu, and Chunyuan Li. Video instruction tuning with synthetic data. arXiv preprint arXiv:2410.02773. 2024b.
Junjie Zhou, Yan Shu, Bo Zhao, Boya Wu, Shitao Xiao, Xi Yang. Yongping Xiong, Bo Zhang, Tiejun Huang, and Zheng Liu. Mlvu: comprehensive benchmark for multi-task long video understanding. arXiv preprint 2rXiv:2406004266 2024.
Jinguo Zhu, Weiyun Wang, Zhe Chen, Zhaoyang Liu, Shenglong Ye, Lixin Gu, Hao Tian, Yuchen Duan, Weijie Su, Jie Shao, et al. Internv13: Exploring advanced training and test-time recipes for open-source multimodal models. arXiv preprint arXiv:2504. /0479, 2025.
15

---

Published as a conference paper at ICLR 2026
APPENDIX OVERVIEW
Section A
More Dataset Details.
Section B
Dataset Scaling Up.
Section
More Experiments.
Section
Qualitative Examples.
Section
Implementation Details.
Section
Error Analysis.
Section
Application Scenarios for Visual Prompts.
Section
Discussion.
Section Additional Results.
Section
Prompt Template.
A MORE DATASET DETAILS
A.1 DATASET STATISTICS
In Table I. we have already presented the main characteristics of V2P-Bench. Overall, the proposed defines three main tasks and twelve dimensions, encompassing 980 unique videos and 1,172 QA pairs sourced from twelve existing video datasets, covering twenty video categories. The average duration of the videos is 19.0 minutes, which represents wide range of video lengths, differing from most benchmarks. The format of the QA pairs is multiple-choice with options. Below introduce more detailed analysis of benchmark:
Wide distribution of durations. Figure (ef() shows the detailed duration distributions on V2P-Bench. We follow Video-MME (Fu et al.) 2024) in categorizing video lengths into short minutes), medium (3-30 minutes), and long videos (30-120 minutes), with respective proportions of 46.8%, 22.0%, and 31.2%.
Diverse video types and comprehensive tasks. Figure left) shows various datasets and categories on V2P-Bench. Figure 7 right) shows the detailed distribution of each dimension.
Diverse Targets and Visual Prompts. Figure 3 shows various targets and visual prompts on V2P-Bench, benefiting from diverse video sources.
Comprehensive Shot Types. V2P-Bench includes both continuous and transition videos, the latter of which significantly increases the difficulty of reference, implying that the model must perform temporal and spatial grounding in different scenes.
Distribution of Videos Durations
Distribution of QA Pairs Across Dimensions
Sei't Adeas
Vilee
Per sloett
Seoet Atrilon
Plmaed Fagure
bsstnalp
Seewe Coolng
Bapiyy
Arte Depurte
Sone Exiizado
fremor
Figure 7: Distribution of video durations and dimensions.
16

---

Published as a conference paper at ICLR 2026
A.2 ELABORATION ON DIMENSIONS
Table 7 presents detailed information on the three main tasks and twelve dimensions of V2P-Bench.
Table 7: Our proposed three main tasks and twelve dimensions with explanation.
Basic Perception
Object Attribute
This dimension evaluates the model's ' ability to perceive the visual and motion attributes of objects indicated by visual prompts, such color, shape, position, and movement.
Human Attribute
This dimension evaluates the model's obility to recognize the actions and attrihutes of individuals indicated by visual prompts, such as their activities, clothing, and appearunce.
Temporal Understanding
Forward Temporal
This dimension assesses the model's ability to accurately locate the visual prompt and track subsequent events that follow the natural chronological order of the video.
Object Direction
This dimension examines the model's ability to perceive and interpret the motion trajectory of objects pointed by visual prompts, with particular focus movement direction.
Feature Mapping
This dimension examines the model's 's capability to extract distinctive features of objects indicated by visual prompts and consistently track them across the entire video.
Reverse Temporal
This dimension evaluates the model's 's capacity to comprehend the temporal struc- ture of the video by identifying events that precede the visual prompt, demonstrating understanding of temporal precedence.
Action Sequence
This dimension evaluates model's 's ability to grasp the overall temporal flow of the video, particularly in understanding and reasoning about the temporal dynamics of multiple action sequences of individuals or objects, as indicated by visual prompts.
Spatial Relationship
This dimension assexses the model's ability to discern and comprehend the spatial relationships between instances highlighted by visual prompts within the video scene.
General Counting
This dimension evaluates the model's ability to perceive and accurately repeated actions or objects within video, as indicated by visual prompts, testing its capacity for detailed content understanding and comprehensive scene analysis.
High-level Reasoning
Causal Relationship
This dimension assesses the model's 's ability to perceive the causal relationships between actions and events, identifying the underlying intentions of actions and the causes of subsequent events. The visual prompt points the action executor.
Plot Understanding
This dimension examines the model's 's ability to analyze narrative progression and logically infer subsequent developments based on the given plot. The visual prompt executes the protagonist of the plot.
Counterfactual Inference
This dimension evaluates model's 's ability to reason under hypothetical scenarios that deviate from actual video content, with visual prompts guiding the devi- ation, assessing its capacity to infer potential outcomes based on counterfactual assumptions.
17

---

Published as a conference paper at ICLR 2026
A.3
ANNOTATION GUIDELINES
A.3.1 VISUAL PROMPT ANNOTATION
Visual Prompt Annotation Guidelines
Interaction Constraint. To remain consistent with real-world interactive applications, each QA pair is restricted to a single visual prompt frame, This design emphasizes the simplicity of user prompt creation and minimizes annotation burden. Therefore, annotators must select the most representative frame that anchors the key moment or location of the event.
Visual Prompt Type. Annotators should select the appropriate type of visual prompt according to the target and strictly follow the eight predefined categories.
Uniqueness. Visual prompts must precisely point to specific object or region in the video, avoiding ambiguity multiple referents.
Consistency. Visual prompts must strictly align with the question text, ensuring a one-to-one correspondence between the annotated target and the object referred to in the question, without mismatches or misleading references.
Multi-target Differentiation. When multiple prompts appear the same frame, different shapes or colors should be used to clearly distinguish between targets.
A.3.2 QA PAIRS ANNOTATION
QA Pairs Annotation Guidelines
Scenario Realism. Questions must based on the actual video content rather than hypo- thetical or fictional scenarios to ensure relevance to the real world and avoid fabricated or unrelated plots.
Correct : Asking What is the object held by the marked person? when annotating a frame where person raises cup.
Incorrect If he is holding beer, what will he do next? (overly speculative).
Answer Uniqueness. Questions should be concise and straightforward, avoiding ambiguous references or subjective judgments to ensure a unique answer.
Correct What color of clothes is the target inside the rectangle wearing?
Incorrect What does this person look like? (ambiguous and hard to standardize).
Avoid Reliance on Common Sense. Do not design questions that can be answered solely using common sense; ensure that the model must refer the visual content answer.
Correct What color is the label of the target inside the ellipse?
Incorrect Do people usually use a knife to cut vegetables? (answerable without watching the video).
Fine- Fine-grained Observation and Diversity. Questions should emphasize detailed observation and cover diverse objects (people, animals, tools, etc.), rather than being restricted to specific target.
Correct: What color jersey is the circled target wearing?
Incorrect What activity is happening in the video? (too broad).
Dependence Visual Prompts. Questions must rely visual prompts be answered, which may require multiple targets to appear in the video, rather than inferring from context common sense. Avoid questions unrelated the prompts.
18

---

Published as a conference paper at ICLR 2026
Correct What object is the larget pointed by the arrow holding?
Incorrect Is the person in the video cooking? (lacking visual prompt).
Minimize Feature Descriptions. Use only the visual prompts refer to the target; avoid describing the target's s appearance in words.
Correct: : When annotating frame where an arrow points to person sitting, ask:
What did the target do before sitting?
Incorrect: What did the person wearing beige clothes and black pants do before sitting? (contains appearance description).
Concise Language and Distractor Design. Questions and options should be kept concise to avoid noise from long descriptions and prevent models from learning biases unrelated to visuals. Options in multiple-choice questions should follow consistent style, avoiding hints from length, tone, or format.
Misleading yet incorrect distractors. Besides common types (e.g., quantity substitution, causal reversal), annotators must carefully control the distractor's misleading nature ensure answer uniqueness.
A.3.3 DESIGN OF DISTRACTORS
Standards for Designing Distractors
Quantity Substitution. Replacing the correct numerical information (such quantities, years, percentages, number of flags, etc.) with incorrect values.
Tool / Object Replacement. Swapping the actual tool used for another similar but different item- for example, writing hoe instead rake, or switching the functions peeler and knife.
Exchanging the duties or action order of participants- instance, changing "the cameraman cleans the table while the assistant kneads the dough" to the opposite distribution of tasks.
Role Reversal.
Step / Sequence Inversion. Reordering the correct procedure omitting key steps-for example, writing the actions as "pass then plant" instead of the correct "plant then pass."
Identity / Name Error. Substituting the real person's 's name, nationality, or position with another-for example, writing Nathan instead of Eric, labeling Canadian athlete as British.
Attribute Replacement. Changing the attribute of clothing, objects, chart lines to another common color example, describing black clothing as green.
Orientation Gesture Reversal. Reversing directions or gestures for example, writing "the right hand holds the cup" or "both hands hold it" instead of "the left hand holds the cup, and the right hand returns to the table."
Emotion Substitution. Replacing the true such as "surprised" or "smil- ing"-witt emotions like "angry," "sad," or "bored."
Misplaced Manse-and-Effect Substituting the real reason with another seemingly plausible explanation for example, the girl covering her ears is truly because "there are too many mice," but the distractor states "neighbors are renovating."
19

---

Published as a conference paper at ICLR 2026
Location Replacement. Changing the venue similar but different place- for instance, describing "a prison" as "a hospital" or "a hotel."
Relationship Misjudgment. Describing friends as lovers, enemies, or strangers, or labeling father-daughter pair as siblings, and so on.
Misattributed Result Follow-up Action. Giving an incorrect description subsequent actions impacts for example, writing "the cameraman bows in thanks" instead of "waves and steps down." "
Addition of Extraneous False Details, Introducing nonexistent actions or objects such as "handing the host torch" create confusion through redundant information.
A.4 QUALITY CONTROL
In Section 3.2.3 to ensure the quality of the dataset, we conducted a rigorous review of the annotated data after completion. The review process included Blind LLMs filtering and manual and rule-based review. Initially, we had 1,747 QA pairs. After applying Blind L.LMs filtering, 1,653 pairs remained (94 filtered). Subsequently, we performed comprehensive manual and rule-based review, resulting in final count of 1,172 QA pairs (481 filtered). Table shows some representative data examples from the post-pre post-processing step.
Can be answered solely by the question, without video frames and visual prompts.
Quesiton don't contain visual prompts.
What the cartoon mouse place at the door after opening
Who is the winner of the 2023 Nobel Prize Literature?
(A) Claudia Goldin
Revised Question:
Jon Fosse
What did the object circled
(C) Louis Brus
place at the door after opening
it?
Cornect Answer: (B)
Obscure the referential message visual prompts.
Deleted as be answered by common sense.
Figure 8: Representative data examples from the post-processing step.
A.5 CERTIFICATE LENGTH
Following EgoSchema Mangalam et al. (2023), we compute the temporal certificate length and the temporal certificate ratio (the proportion the minimal effective content duration relative to the total duration) based on the video length and task type in our dataset. The results are shown in Table
Table 8: Certificate Length and Certificate Length Percentage across Different Video Durations.
Short TU
Medium TU
Long TU
BP
HR
BP
HR
BP
HR
Certificate Length Certificate Length Percentage
31s
27s
1s
204s
169%
1s
1147s 35.3%
984s
1.9%
61.4%
52.9%
0.2%
44.7%
37.2%
0.003%
30.3%
The Basic Perception task is designed solely to assess a model's fundamental ability to perceive visual prompts, and thus it can completed using a single prompted frame. In contrast, Temporal Understanding and High-level Reasoning focus dynamic information and chronological relation- ships in videos, and on logical inference and higher-level judgment, respectively. As a result, both tasks require multiple video frames to be properly resolved.
20

---

Published as a conference paper at ICLR 2026
A.6 DATASET BIAS
Our benchmark is constructed from 12 publicly available video datasets, all of which were originally annotated with pure text-based QA pairs. The annotators of these datasets did not assume that "users will provide visual prompts" when creating the annotations, meaning that the data are naturally more aligned with text descriptions. The visual-prompt version of the benchmark is a rewrite we created on top of these original annotations. Therefore, from the perspective of data provenance: If any bias exists, it should favor pure text, not visual prompts.
Furthermore, to ensure a fair comparison between text prompts and visual prompts, we uniformly rewrote all text prompts rather than directly reusing the original textual annotations. Since the 12 constituent datasets vary widely in linguistic style, granularity, and descriptive conventions, directly comparing visual prompts with the original text would introduce additional bias. To avoid this, rewrote all questions into a standardized text-prompt format and ensured the text prompts contained the same information as the visual prompts.
B DATASET SCALING UP
The core value of benchmark dataset lies in being high-quality, trustworthy, and diagnostic, rather than merely covering much data as possible. To this end, we rely trained human annotators instead of automated generation, ensuring that each question maintains a striet causal linkage between the video evidence, visual prompts, and the correct answer. Thus, the benchmark does not prioritize "scaling up data volume" as its primary goal.
On the other hand, to enable the construction of large-scale visual-prompt datasets and to enhance existing models' ability to understand visual prompts, we also provide a fully automated data- generation pipeline, including:
1) Using RAM++ Zhang et al. (202aa) to automatically extract potential target objects from video.
2) Applying SAM3 Carion et al. 2025 for cross-frame object tracking and mask propagation to obtain temporally consistent visual annotations.
3) Following automatic data-synthesis strategies from LLaVA- Video Zhang et al. (022b) and ShareGPT4Video Chen et al.] (2024c) to bind model-generated QA pairs to target regions, thus producing large quantities of high-quality video-visual-promptt supervision data.
C MORE EXPERIMENTS
C.I CONSISTENCY ANALYSIS OF MCQ AND DIALOGUE TASKS
We adopt a MCQ format as MCQ-based evaluation easy to automate, highly controllable, and effective at avoiding the ambiguity and vagueness that commonly arise in open-ended responses.
This is also why existing mainstream undersunderst benchmarks (Video-MME) Fu et al. (2024).
MVBench [i et al. (2024c), EgoSchema Mangalam al. (2023). LVBench Wang et (024bb)
ett)widelyy adopt QA-style design. In contrast, free-form dialogue introduces substantial subjectivity and is difficult to score in consistent and reproducible manner. Moreover, relying on LLM-as-a- judge may introduce evaluation bias, further weakening the faimess, reliability, and comparability of the assessment. However, real human- modell interaction scenarios are inherently dialogue tasks. To examine the extent to which MCQ task can represent dialogue task, we further analyze the consistency between the two. Specifically, we remove the multiple-choice constraints from the original questions, convert them into open-ended ones and re-evaluate the models using GPT-4o Hurst et al. 2024) and human annotators as the judge.
As shown in Table 9 under the open-ended setting, the Pearson consistency coefficient between MCQ and OE (GPT-0 Judge) reaches 0.95, and the coefficient between MCQ and OE (Human Judge) reaches 0.98. This demonstrates strong consistency between QA-style evaluation and free-form dialogue, indicating that Qa-based assessments can reliably reflect a model's 's capabilities in open-ended conversational scenarios.
21

---

Published as a conference paper at ICLR 2026
Table 9: Consistency Evaluation Between MCQ and Open-Ended Tasks.
Model
MCQ
OE (GPT-40 Judge)
OE (Human Judge)
Gemini-2.55 69.8
48.3
51.6
Qwen2.5-VL-Th 48.1
30.2
33.7
MiMo-
45.7
32.7
35.6
C.2 HACK PHENOMENA IN OPEN-ENDEL EVALUATION
Existing benchmarks commonly adopt the MCQ format because is casy to evaluate and highly controllable, and avoids the ambiguity often seen in open-ended responses. Our experimental results show that models indeed exhibit such behavior: they tend to game the MCQ structure to obtain higher scores instead of performing pursuing higher benchmark scores, future models should also prioritize genuine understanding and groundedness key optimization objectives.
video comprehension. This finding suggests that, in addition to
To examine whether open-ended generation helps reduce hack behaviors, we conducted two experi- ments in which we removed the multiple-choice constraints and answer options from the original prompts, converting them into open-ended question-answeringg tasks:
Random Shuffling Experiment. We randomly shuffle the videos and questions, meaning the model can no longer obtain the answer from the video content. If a model is sufficiently honest, it should refuse answer all questions. We perform inference using Qwen2.5- - and MiMo VL-7B. The Trigger Ratio denotes the proportion of cases in which the model refuses to answer. For the MCQ setting, we perform standard reasoning and count all responses that do not select any option; for the OE setting, we manually evaluate whether the model refused to respond. The results shown in Table
Table 10: Hack Phenomena under MCQ and Open-Ended Evaluation.
MCQ Trigger Ratio (%)
OE Trigger Ratio (%)
Model
Short
Medium
Long
Avg
Short
Medium
Long
Avg
Qwen2.5-VIL-7B
6.2
6.9
6.3
6.4
95.8
96.8
96.1
96.7
MiMo-VL-7B 3.6
4.2
4.0
3.9
92.2
94.6
94.8
93.6
We observe that under the MCQ setting, Qwen2.5-VL-7B and MiMo- exhibit refusal rates of only 6.4% and 3.9%, respectively. This indicates that even when the video and question are completely mismatched, the models still select option in the vast majority of cases, and the trigger rates remain almost identical across different video lengths. In contrast, under the OE setting, the refusal rates rise sharply to 96.7% and 93.6%, again showing little variation across video lengths. These results clearly demonstrate that open-ended generation effectively mitigates hack behaviors to large extent.
Model Performance on V2P-Bench. Besides, we follow the same setup as the random shuffling experiment, with the only difference being that shuffled version. The experimental results are shown in Table
the original benchmark data rather than the
Table 11: Hack Rates under MCQ and Open-Ended Evaluation.
MCQ Hack Ratio (%) Medium
OE Hack Ratio (%)
Model
Short
Long
Avg
Short
Medium
Long
Avg
Qwen2.5-VL-7B
8.1
11.5
14.4
10.2
2.1
2.6
3.4
2.7
MiMo-VL-7B
5.3
7.2
7.4
6.2
1.7
2.3
2.8
2.2
We observe that after converting the MCQ task into open-ended question task, the Hack Ratios Qwen2.5-VL-7B and MiMo-VLL decrease by 7.5% and 4.0%, respectively. This further demonstrates that open-ended generation can substantially mitigate hack behaviors.
22

---

Published as a conference paper at ICLR 2026
C.3 IMPACT OF SAMPLING FRAME RATES
We evaluate Qwen2.5-VL-7B across different task types and video durations to investigate how varying sampling frame rates affect model performance. Here, frame indicates that only the visual-prompt frame is provided to the model. The results are shown in Table 12
Table 12: Performance under Different Sampling Frame Rates across Video Durations.
Short TU
Medium TU
Long TU
Model
Frames
BP
HR
BP
HR
BP
HR
62.1
28.6
29.8
64.3
30.4
31.1
47.6
27.4
33.2
Qwen2.5-V1-77B
8
61.8
38.3
43.0
59.1
43.6
44.2
44.8
41.7
45.7
64
60.8
40.1
46.8
56.1
47.9
51.8
43.1
41.7
50.0
We observe that for the BP task, which relies solely on single-frame information, increasing the sampling frame rate actually leads a certain degree of performance degradation. This is because more frames enlarge the temporal search space, making it harder for the model to accurately locate the visual prompt frame. In contrast, TU and HR tasks inherently require multi frameefreeee temporal information. As the frame rate increases, the model gains access to richer dynamic cues, resulting in clear performance improvement that eventually saturates.
D QUALITATIVE EXAMPLES
D.1 COMPARISON WITH INST-IT
Unlike V2P- 22P-Bench, INST-IT dataset contains visual prompt annotations on every frame of the video, as shown in Figure which is both unrealistic and practically unachievable in real human-model interaction scenarios.
INST-IN IIT Dataset
V2P-Bench
What new becomes visible by the frame at <4>?
In the video, how many arrows did the woman in the detection box shoot?
impossible in human-madel
the frome timestamp performing action opening window.
Just frame for interaction.
interaction
(B): [7] raises both hands to engage with the window frame.
(A) 1.
(B) 3.
scenarios.
(C) (D) 7.
Correct Answer: (B)
Correct Answer: (C)
Figure 9: Comparison of INST-IT- Dataset and v2P-Bench.
E IMPLEMENTATION DETAILS
E.
HUMAN EXAM AND BLIND LLMS ANSWERING.
Human Exam. For the human evaluation, all questions were assigned to three human experts. To mitigate the risk of data leakage, we took special care to ensure that the experts involved in the evaluation had never participated in the annotation process. The human experts were instructed to watch the videos along with the visual prompt frames and to respond by selecting only the letter corresponding to their chosen option. The results scored using a script, maintaining consistency with the model evaluation process.
recorded in JSON file. The evaluation was
Blind LLMs Answering. For the blind answering task, report the performance of three models: GPT-4o, Gemini-2.5-Par -ro, and Qwen2.5-VV VL. Although all three models are in fact LVLMs, we provide them only with the textual QA pairs, that their visual encoders remain inactive. The models
23

---

Published as a conference paper at ICLR 2026
prompted to output "Z" honestly instead of selecting option whenever the QA pair requires access to visual context, thereby preventing the models from guessing.
E.2 COMPARISON OF TEXT AND VISUAL PROMPTS FOR HUMANS AND MODELS.
Text prompt evaluation. After completing the annotation of the visual-prompt dataset, we uniformly rewrote all text prompts, rather than directly reusing the original textual annotations. The twelve source datasets differ significantly in linguistic style, granularity, and description patterns; therefore. directly comparing their original questions would introduce additional bias. To ensure fairness, we rewrote all questions into standardized text-promp format and guaranteed that the amount of information conveyed matched that of the visual- proupp version.
Concretely, we converted the visual-pre poompp references in each QA pair into natural-language descrip- tions of the target's appearance. For example, a question such "What does the arrow-pointed person do after getting off the car?" was rewritten as "What does the person weaormuing a black suit and black hat do after getting off the car?" In contrast, in the original text-only dataset, the same question appeared as "What does the main character in the video do after getting off the car?", a formulation that requires the model to first understand the video content and identify the main character. This would be unfair when comparing text prompts to visual prompts, since the latter already explicitly localize the target. Therefore, we performed a unified rewriting of all text prompts to eliminate this bias and ensure a fair comparison between text-based and coomp--rompp-baaeee evaluation.
To guarantee fairness in the comparison, we also include the original video frame corresponding to the visual prompt (without any visual prompts) later in the video, ensuring that in both experimental settings, the model receives an identical amount of visual context. Finally, we adopt the same inference and evaluation procedures as in the visual prompt setting, ensuring comparability of results and enabling rigorous assessment of model performance in pure text QA tasks.
User experience study. In fact, we reused the Gradio-based annotation interface originally built for constructing the dataset, so the volunteers operated the same environment as the annotators. The UI supports displaying, playing, and pausing the video at the top; when the video is paused, clicking again enlarges the frame and launches MS Paint, allowing users to draw visual prompts directly on the frame. The UI also displays the original QA pair and allows the annotator to input a new question. After the participant finishes formulating the question, the video, the visual prompt frame, and the question are sent to the Gemini API to obtain response. We slightly improved the system display the model's 's reply within the UI.
To avoid possible biases preferences arising from annotators who were familiar with the dataset's purpose, we recruited volunteers to conduct the user study. Before formally participating, each volunteer received training from one of the annotators and completed guided interaction session. Only after this training were the volunteers allowed to proceed with the actual experiment.
24

---

Published as a conference paper at ICLR 2026
F ERROR ANALYSIS
Counting Number Error
Question: How many Pepsi-Cola bottles like the one the green box the table next to?
A: There six Pepsi cans the table.
B: There four Pepsi
the table.
C: There were two Pepsi on the table.
There were eight Pepsi cans on the table.
Visual Prmopt Frame
Looking video, we can see white rolling cart next to an open refrigerator. top shelf this cart, One is partially obscured are three cans clearly visible in row. 3. Behind these three cans, there are two more cans, So, the total count is the refrigerator, and the green box highlights some Pepsi cans inside the refrigerator. The question, however, cans the video) the table next the refrigerator. Therefore, option is correct. The final onswer
blue cans the Pepsi logo, Let's count large red bag of chips the left. 2. Next
there
2 cans. The visual cue frame shows the inside of
Gemini- 2.5-Pro
about the Pepsi-Cola bottles (or cans, as they appear to be
Direction Error
Question: Which direction does the object in the pink box move the video?
A: Down and to the right.
B: Up and to the right.
Up and the left.
Down to the left.
Visual Prmopt Frame
To determine the direction movement for the object the pink box (the green sphere):
**Observe the initial position and subsequent movement**; Initially, the green sphere is positioned near the blue cylinder. As the video progresses, the green sphere moves away from the starting point. 2. **Track the trajectory**. right.
The green sphere moves diagonally upward and to the
40
**Cross- with the direction options**: The movement matches Option B: Up and the right. Thus, the correct answer is Up and to the right.
Figure 10: Typical failure modes of the model regarding Perception Errors.
25

---

Published as a conference paper at ICLR 2026
Plot Understanding Error
Question: What is the purpose of the man in the video processin
the object with red graffiti?
A: Processing the blue plastic barrel for feeding cattle.
Processing the blue plastic barrel for feeding ducks.
C: Processing the blue plastic barrel for feeding chicken.
D: Processing the blue plastic barrel for feeding pigs.
Visual Prmopt Frome
To determine the purpose of the man processing the blue plastic barrel with red graffiti, let's analyze the video step by step:
**Contextual Clues**:
The setting appears be farm an agricultural environment.
There are various onimals present different scenes, including ducks, chickens, and pigs.
2. **Specific Scenes**: In one scene, there are ducks gathered around feeding area.
Another scene shows chickens in a coop. There is also scene with pigs in a fenced area.
*MProcessing Activity**: The man is modifying the barrel, likely cutting or shaping it. This modification suggests that the barrel being repurposed for specific function. **Feeding Troughs
troughs, which are typically used for feeding animals. The size and shope of the barrel suggest it could be used for larger animals group of smaller animals.
barrel being transformed
Qwen-VL- 2.5-72B
5. "Matchhin with *nimals**: Ducks often require water and food troughs that can hold both liquid and solid feed. Chickens also use troughs for feeding, but they are usually smaller. Pigs can use larger troughs for both water and food.
ducks in the video and the size of barrel, most likely being processed into troughs for feeding ducks. Ducks typically need larger troughs to accommodate their feeding habits and the amount of water they consume. Therefore, the correct onswer is **8: Processing the blue plastic barrel into troughs for feeding ducks. **Final Answer: ******
**Conclusion**:
Given the presence
Figure 11: Typical failure modes of the model regarding Reasoning Errors.
Insufficient Information
How does the person marked with green graffiti feel when he is hit by egg?
Question:
Happy
B: Sad
C: Angry
D: Neutral
Visual Prmopt Frame
The question asks about the emotional state the person marked with green graffiti when hit by an egg. However, the image provided does not show any person being hit by an egg any green graffiti. Instead, it shows man and woman sitting on couch in dimly lit room, engaged
conversation. The man wearing brown jacket and black shirt with cartoon character on it, while the woman is wearing white dress. There is no indication of any egg being thrown any green graffiti in the image. \nAnTherefore, it is not possible to determine the emotional state of the person marked with green graffiti when hit by an based on the given image. The correct answer cannot be determined from the image provided. Since user required to provide an answer, correct answer **g: * Answer: **g**
Qwen-VL- 2.5-72B
Figure 12: Typical failure modes of the model regarding Insufficient Information Errors.
26

---

Published as a conference paper at ICLR 2026
G APPLICATION SCENARIOS FOR VISUAL PROMPTS
In real-world scenarios, the video modality demonstrates greater applicability and value, specifically including but not limited to the following situations:
Interaction with Mobile Devices. Users watch videos mobile devices such as smartphones and computers, easily creating visual prompts with their fingers or a mouse. This functionality has been successfully implemented in systems like GPT-4o and Video Refer, allowing users to interact with video content more intuitively, enhancing their viewing experience.
Smart Device Wearables. Imagine users wearing advanced smart devices (like Apple Vision Pro), immersed in sea of video content. These users could generate visual prompts through natural gestures. It undoubtedly offers limitless possibilities for future interaction methods, significantly enhancing the immersive experience of watching videos.
Interaction with Smart Robots. Consider the interaction between users and physical smart robot terminals equipped with visual prompt screens. It will revolutionize the way we communicate with smart technology, making interactions more vivid, engaging, and intuitive.
In these diverse scenarios, users can interact smoothly with local or cloud-based LVLMs based on video content. Developers only need adjust the front-end system efficiently capture video frames and visual prompts to achieve this goal. Such integration will create a more immersive experience for users, transforming video watching from passive observation into an engaging and interactive journey.
H DISCUSSION
Limitation. Although our V2P-Bench comprehensively evaluates the capabilities of LVLMs in video-language understanding with visual prompts for multimodal human-model interaction, it only focuses on visual and textual inputs, lacking audio input, and supports evaluations only on offline videos, which leaves a gap compared the ultimate form of multimodal human-model interaction in real world. we plan to develop V2P- '2P-Bench v2, which will support all types of video understanding, incorporate full-n mol--moyyyyty inputs, and enable the evaluation of multi-turn dialogue and interruptible interactions.
Broader Impact. V2P-Bench has built comprehensive visual prompt dataset for evaluating video visual prompt question answering in the multimodal domain, which will help more thoroughly validate the video understanding capabilities large visual language models and enhance their performance in the field of video understanding. We have released the dataset, evaluation code, and leaderboard.
I ADDITIONAL RESULTS
I.1
RESULTS ACROSS VISUAL PROMPTS
See Table
1.2 RESULTS ACROSS FRAME RATES
See Table [4] to Table 2
I.3
HACK PHENOMENA ON OTHER BENCHMARKS
See Figure 13
27

---

Published as a conference paper at ICLR 2026
J
PROMPT TEMPLATE
Prompt template for model inference
System prompt:
First, you will receive series of images sampled at regular intervals from a video. Then, you will receive a visual prompt frame, which is screenshot from the same video and have been manually annotated with visual prompts. Your task to answer the question based on the video and visual prompt frame. Select the best option that accurately addresses the question. Give only your option letter, no other words.
User prompt:
<video> <vp_frame> <question>
Prompt template for the blind answering task
System prompt:
You will receive question and four options. Please respond based on the question. Select the best option that accurately addresses the question. Give only your option letter, no other words.
If the question cannot be answered, output Z.
User prompt:
<question>
Prompt template for model inference without hack
System prompt:
First, you will receive series of images sampled regular intervals from video. Then, you will receive a visual prompt frame, which is screenshot from the same video and have been manually annotated with visual prompts. Your task to answer the question based on the video and visual prompt frame. Select the best option that accurately addresses the question. Give only your option letter, no other words.
the video does not provide sufficient information, or none of the options correct, don't try to make up answer but output Z directly.
User prompt:
<video> <vp_ framf> <question>
28

---

Published as a conference paper at ICLR 2026
& Furformence Willench
Mask Pranonsnnc Porformance Video
Hack Prevree Performrcc Mideffece
Ang Rurtormaac
coiking
Figure 13: Hack Phenomena and Performance Qwen2.5-VL-7B on MVBench, Video-MME and Long Bene.Behnnn
Table 13: Evaluation results on V2P-Bench across visual prompt types.
Method
Rectangle
Mask
Ellipse
Triangle
Scribble
Point
Arrow
SoM
Avg
Human Performance
86.6
88.1
92.5
86.9
89.7
90.2
88.3
Closed-souee source Models
o1
74.4
70.1
68.9
71.9
72.1
71.8
GPT-40
68.5
63.6
65.4
62.3
66.0
63.2
69.8
59.6
65.4
Gemini-2.5-Pr
73.2
63.6
71.4
65.6
69.5
72.4
62.8
66.3
69.8
Open-source Models
LLaVA-NeXT-7BB
44.3
40.2
47.9
49.1
48.3
45.4
45.1
47.7
46.0
LLaVA-NcXT-NSTTII-7
42.2
41.5
45.0
45.6
45.3
43.7
44.2
62.7
46.3
LLaVA-OV-7B
54.7
48.1
49.2
59.0
44.9
72.1
52.7
52.9
52.8
LLaVA-OV-72B
55.1
$4.5
54.1
55.7
58.2
60.5
56.7
54.8
56.7
InternV1.3-8B
62.6
$7.1
61.1
62.3
65.3
55.8
59.6
65.4
61.7
nPLJUG-OW33-7B
51.6
55.8
47.6
$2.5
49.0
53.5
59.1
51.9
52.6
LLaVA-Video-7B
50.4
54.5
53.5
55.7
62.2
58.1
56.2
54.8
54.8
LLAVA-Video-72B
59.1
59.7
57.3
49.2
67.3
55.8
58.6
56.7
58.6
MiniCPM-V 2.6-8B
56.3
48.1
53.5
65.6
51.0
53.5
57.6
53.8
55,3
Qwen2.5-VL-7B
47.6
55.8
42.2
42.6
48.0
48.8
50.2
51.9
48.1
Qwen2.5 VLL-72B
63.4
61.0
57.8
54.1
55.1
58.1
61.1
61.5
59,8
MiMo-VL-7B
49.6
51.9
55.1
55.7
50.0
48.8
56.2
61.5
53.8
Table 14: Results LLaVA-One with different frame-rate sampling strategies.
Model
Frames
Dimension
Duration
Avg
AS
GC
Mediumm
Long
49.6
52.5
30.4
45.1
51.4
47.4
36.7
45.3
57.2
35.1
$1.1
43.1
40.3
48.0
51.3
51.2
32.6
45.1
56.2
59.1
41.0
47.4
37.8
48.4
55.2
43.2
52.4
50.0
41.2
49.2
LL
58.8
53.0
28.3
45.1
35.9
34.2
37.8
55.8
58.6
43.2
52.0
53.7
43.6
50.4
64
56.3
54.4
28.3
47.1
60.0
59.1
41.0
42
41.1
55.8
63.4
43.2
51.9
63.4
45.3
52.7
57.1
52.1
28.3
47.1
63.8
39.1
41.0
12.1
35.6
63.2
62.8
43.2
51,3
63.0
17.3
52.8
Table 15: Results on LLaVA-One Vision- 72B with different frame-rate sampling strategies.
Model
Dimension
Duration
Avg
0D
FM
CR
PU
C1
FT
Shoet
Medium
Long
48.7
55.8
37.0
47.1
55.2
50.0
38.5
57.9
33.3
60.0
60.0
32.4
50.6
57.4
46.1
51.0
58.0
55.3
39.1
47.1
57.1
43.2
33.3
52.6
35.6
62.1
57.9
45.9
52.6
59.7
44.4
52.1
1LAVA-OV-228
62.2
60.8
30.4
45.1
63.8
45.5
41.0
52.6
35.6
62.1
67.6
45.9
54.9
65.3
49.4
55.8
64
65.5
60.4
30.4
45.1
61.9
43.2
38.5
47.4
43.3
65.3
66.2
45.9
53.8
71.8
48.1
56.2
65.5
59.9
34.7
47.0
63.
43.2
38.5
50.0
41.1
66.3
54.5
70.4
56.7
Table 16: Results on Intern VI.3-BB with different frame-rate sampling strategies.
Duration
Model
Frames
Avg
DA
HA
0D)
FM
CR
PLI
CI
FI
AS
SR
OKC
Shart
Mediuu
Long
66.4
65.0
28.3
56.9
51.4
59.1
28.2
52.6
34.4
57.9
63.4
61.9
56.8
58.8
51.9
56.0
59.1
IntemV113-SB
16
69.7
67.7
37.0
54.9
61.9
63.6
35.9
52.6
37.8
60.0
68.3
67.6
60.3
68.5
52.3
60.1
32
69.7
67.3
39.1
52.9
63.8
63.6
43.6
50.0
400
56.8
70.3
62.2
59.8
68.1
55.1
60.4
128
73.9
69.1
60.8
58.1
65.9
41.0
52.6
41.1
61.1
69.7
64.9
61.7
68.5
55.6
61.7
29

---

Published as conference paper at ICLR 2026
Table 17: Results on mPLUG-Owi3: with different frame-rate sampling strategies.
Model
Frames
Dimenssoo
Duration Medium
Avg
QA
HA
0D
FM
CR
PC
C1
FT
RT
AS
SR
GC
Shor
Long
61.3
53.5
32.6
41.2
53.3
50.0
18.7
55.3
26.7
51.6
53.8
37.8
51.7
50.5
43.6
49.5
544
mPLUG-OW1B-7B
16
59.7
51.6
304
51.0
54.
13.5
18.7
53.3
31.1
51.7
60.0
40.5
51.3
50.9
32
60.5
55.3
32.6
45.1
56.2
45.5
48.7
50.0
28.9
54.7
61.4
35.1
$1.7
58.8
44.0
51.4
55.5
54.8
348
54.9
6000
50.0
46.2
50.0
33.3
58.9
58.6
35.1
52.2
59.3
45.7
52.1
128
544
28.3
490
34.4
55.8
58.6
37.8
52.6
Table 18: Results on LLa Videoo 7B with different frame-rate sampling strategies.
Model
Frames
Dimension
Duration
Avg
0A
HA
0D
FM
CR
PU
C1
FT
RT
AS
SR
GC
Shoet
Medium
Long
55.5
54.8
30.4
49.0
54.3
59.1
38.5
55.3
43.3
54.7
53.1
37.8
50.6
57.4
46.9
51.2
IlVVA-Vides-B 16
61.3
57.6
32.6
19.0
57.1
51.5
38.5
$2.6
1L.1
56.8
59.3
13.2
53.4
62.0
16.5
$3.6
58.8
58.5
34,8
52.9
58.1
56.8
$1.3
55.3
42.2
56.8
55.2
40.5
54.3
63.4
44.9
54.0
128
60.5
58.1
37.0
19.0
62.9
51.5
11.0
52.6
18.9
57.9
56.6
10.5
54.1
65.7
46.5
51.8
Table 19: Results LLaVA- with different frame-rate sampling strategies.
Model
Frames
Dimension
Duration
Avg
SR
OC
Medium
Long
52.9
54.8
43.5
54.9
55.2
56.8
43.6
57.9
32.2
61.1
64.8
54.1
55.2
53.9
8
58.0
51.4
34.8
52.9
61.0
56.8
41.0
47.4
43.3
63.2
61.4
56.8
55.2
62.5
46.9
54.8
LLaVA-Videu-222 16
61.3
55.8
34,8
51.0
60.0
56.8
13.6
52.6
40.0
65.3
68.3
62.2
56.1
65.3
19.1
56.6
57.4
48.7
69.5
72.4
58.0
128
62.2
60.8
30.4
54.9
61.0
34.5
43.6
47.4
42.2
70.5
71.0
59.5
57.5
06.2
54.3
53.6
Table 20: Results on MiniCPM-V 2.6 with different frame-rate sampling strategies.
Model
Frames
Dimension
Duration
Avg
0D
FM
CR
PU
C1
Shoet
Medium
Long
59.7
54.8
28.3
41.2
42.9
52.3
28.2
52.6
33.3
51.5
59.3
40.5
50.3
51.4
44.0
49.0
63.0
55.3
26.1
33.3
52.4
43.2
30.8
11.7
34.
4.2
61.1
10.5
19.9
55.1
43.5
49.5
ManiCPM-V
65.5
57.1
28.3
49.0
58.
50.0
33.3
50.0
43.3
55.8
50.7
45.9
54.5
61.6
45.3
53.8
64
67.2
62.2
26.1
52.9
57.1
43.2
28.2
50.0
40.0
58.9
64.8
45.9
54.3
63.7
47.7
55.2
68.9
59.4
56.9
58.1
50.0
33.3
50.0
34.4
57.9
533
66.2
50.2
55.3
Table 21: Results on Qwen2.5-VL-7B with different frame-rate sampling strategies.
Model
Frames
Dimension
Duration
Avg
DA
HA
OD
FM
CR
PU
CI
FT
AS
Short
Medium
Long
62.2
56.2
17.4
35.3
45.7
43.2
53.8
57.9
36.7
41.1
47.6
47.6
49.1
43.2
46.9
Qwen2.5-VL-7B
16
56.3
56.3
54.8
53.9
26.1
26.1
47.1
43.1
47.6
52.4
52.
56.8
38.5
25.6
52.6
55.3
32.2
36.7
44.2
43.2
46.2
49.0
43.2
40.5
50.1
49.2
48.1
48.6
40.3
42.4
47.4
47.5
48.7
128
60.5
53.7
17.4
45.1
47.6
40.9
48.7
52.6
32.2
47.4
50.3
35.1
48.0
53.2
43.6
48.1
Table 22: Results on Qwen2.5-VL-72B with different frame-rate sampling strategies.
Model
Dimension
Duration
Avg
Long
64.7
65.0
28.3
56.9
41.0
40.9
30.8
55.3
46.7
50.5
59.3
29.7
55.4
55.6
44.0)
52.7
65.5
62.2
50.4
49.0
42.9
52.3
38.5
55.3
37.8
54.7
59.3
40.5
55.7
52.8
46.5
52,9
Owen2.5-VLL722
57.9
69.7
68.2
39.1
5499
47.6
50.0
51.3
50.0
44.4
63.2
68.3
48.6
61.0
62.0
51.4
59.0
128
69.7
72.4
43.5
52.9
49.5
59.1
53.8
55.3
44.4
57.9
64.1
51.4
62.4
63.9
50.2
59.8
30


Reviews:
[{"review_id": "Lc0mTOHcfe", "reviewer": "Reviewer_rfAK", "summary": "The paper introduces V2P-Bench, a benchmark designed to evaluate large vision-language models (LVLMs) using visual prompts instead of text prompts for video understanding, enabling fine-grained assessment of spatial, temporal, and reasoning abilities. Results show that visual prompts improve both model accuracy and user experience, but current LVLMs still lag behind humans, particularly in spatiotemporal comprehension and robustness.", "questions": "See weaknesses.", "limitations": "", "rating": 4, "confidence": 4, "soundness": 2, "presentation": 2, "contribution": 2, "strengths": "- Introduces a visual prompt–based benchmark that improves realism in human–model interaction evaluation.\n- Provides a comprehensive dataset with multi-level tasks and rigorous annotation ensuring high quality and diversity.\n- Offers clear empirical insights into LVLM weaknesses, including hack phenomena and spatiotemporal reasoning gaps.", "weaknesses": "1. The benchmark currently focuses on synthetic QA tasks rather than natural, conversational interactions, which limits its generalization to real-world human–model dialogues. How would you integrate free-form conversational or multimodal dialogue settings to better simulate authentic human–AI interaction? If not, it is really not geberalisable. \n\n2. The dataset construction process is highly manual and time-consuming, making it difficult to scale to larger domains or real-time applications. How can we incorproate scalability?\n\n3. Several analyses, such as the user experience evaluation and hack behavior detection, rely on qualitative interpretation rather than standardized quantitative frameworks. Cou.d the authors design more rigorous, quantitative evaluation protocols and reproducible metrics to strengthen the reliability and comparability of findings?\n\n4. The benchmark primarily evaluates static performance outcomes but does not assess model adaptability or learning progression over time.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "Zn9KPBA4KB", "reviewer": "Reviewer_9ULh", "summary": "The paper proposes V2P-Bench, a robust and comprehensive benchmark for evaluating the ability of LVLMs to understand video visual prompts in human–model interaction scenarios. The authors argue that existing video benchmarks primarily rely on text prompts, which often involve complex referential language and, in turn, reduce the accuracy and efficiency of human–model interaction. To address this, they introduce more user-friendly visual prompts. The authors conduct a comprehensive analysis and highlight several key findings, such as performance on spatiotemporal understanding and the prevalence of the hack phenomenon.", "questions": "1. The hack phenomenon is concerning, especially as it increases rates as the video length grows. I am curious if the authors convert their benchmark to an open-ended generation one instead of MCQ-based, would that help in reducing some of this? \n\n2. I am curious how does the structuring of the visual query itself impact the performance? For instance, what if you have scribbled unstructured bounding boxes instead of perfect rectangles or other more natural markers that are more user-friendly, how does that impact overall performance? \n\n3. From many of the query examples provided in the paper, it appears that most could be resolved using only a few frames, with the main bottleneck being the grounding of the visual query frame. Could the authors elaborate on this point, perhaps by quantifying it using temporal certificates [1] or frame rate ablations? Specifically, do the models on their benchmark perform better as more visual information is provided, or can the benchmark generally be answered using just a few frames?\n\n[1] EgoSchema: A Diagnostic Benchmark for Very Long-form Video Language Understanding", "limitations": "", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "1. The paper is well-motivated and studies a timely problem of visual prompt-based evaluation instead of unnecessarily overcomplicated textual prompts. I particularly liked that the authors clearly established this with a human study as well.\n2. The benchmark is carefully curated with multiple automated and human checks.\n3. The authors conduct comprehensive experiments to study different models, and go beyond standard evaluation to study interesting hack phenomenon and other ablations.", "weaknesses": "While I don't foresee any major weaknesses, I have some questions about experiment design and further experiments that I would like to see before I raise my score further.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "xG47oj0bA1", "reviewer": "Reviewer_8395", "summary": "This paper proposes V2P-Bench, a new benchmark addressing the inefficiency of text-based evaluation in video-language understanding. By using visual prompts directly on video frames, it assesses model performance across perception, temporal, and reasoning tasks. Results show that visual prompts improve model accuracy while exposing weaknesses in long video and spatiotemporal reasoning.", "questions": "I have several critical questions that the authors should address to clarify the scientific validity of the work.\n\nFirst, the central claim, that visual prompts are a more human-aligned form of interaction, remains unsubstantiated. Could the authors provide theoretical grounding or empirical evidence (e.g., from cognitive studies or user experiments) rather than relying on intuitive justification?\n\nSecond, the benchmark design is still based on offline video QA, which seems inconsistent with the paper’s framing around “human–model interaction.” How do the authors ensure that this static setup meaningfully reflects real interactive understanding?\n\nThird, the error analysis is largely descriptive and lacks mechanism-level insight. Why do models consistently fail on spatiotemporal reasoning tasks, and how can the authors be sure that the evaluation protocol effectively eliminates “hack” behaviors rather than concealing them? \n\nAddressing these questions is essential to assessing the scientific rigor and actual contribution of the work.", "limitations": "", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "This paper introduces the novel idea of using visual prompts for video-language evaluation, providing a more intuitive and human-aligned alternative to text-based prompts. The dataset and tasks are well designed, with careful annotation and broad experimental coverage across major LVLMs. The presentation is clear and well-structured, supported by effective figures and analyses.", "weaknesses": "The main weaknesses of this paper lie in its theoretical and methodological depth. While the idea of visual prompts is novel, the work lacks a stronger theoretical or cognitive grounding to explain why this approach better reflects human interaction. The evaluation remains limited to offline video QA with a relatively small dataset and does not include audio or multimodal signals, which limits its realism and scalability. In addition, the analysis is mostly descriptive, without deeper investigation into why models fail in spatiotemporal reasoning or specific error types.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "wsCsPNvT10", "reviewer": "Reviewer_VvkZ", "summary": "The paper introduces a new video benchmark that tests models’ abilities at visual question answering while allowing for visual prompting in addition to text.  With visual prompting, users are allowed to annotate the video and ask questions that refer to the annotation (eg., what is the person with the box around them doing?).  The benchmark is built on existing video collections and includes a range of different types of questions, ranging from low level questions about, for example, object attributes, to higher level questions, about plot or causal relationships.  It seems that all questions are designed to require identification of specific people or objects that might be difficult to otherwise describe.  A large number of open and closed models are identified on the benchmark, which still demonstrate performance that is below human level.", "questions": "Please provide more details on how the questions were chosen.  How do you ensure that you are not selecting questions that will bias performance in favor of visual prompting?\n\nExperiments in Table 3 are not clearly explained.  How are the text and visual prompts generated?  In Table 3b, what interface is provided for visual prompts?  Is training required?  The supplementary material addresses some of this, but I still found it unclear.  This should be clearly explained in the body of the paper.", "limitations": "", "rating": 2, "confidence": 3, "soundness": 2, "presentation": 3, "contribution": 2, "strengths": "The evaluations seem thorough.  Visual prompting does not seem to have been studied adequately in video.  The work seems very comprehensive.", "weaknesses": "First, it is not clear how important this problem is.  The questions in the dataset seem designed to benefit from visual prompting.  But how often do users really want to ask questions about people or objects that are difficult to describe?  This might happen often, but there is no evidence given that this is the case.\n\nSecond, it is not clear that the results that visual prompting is more effective and efficient than text prompting are truly valid, since the questions developed seem designed to favor visual prompting.  This would be more convincing if these questions arose organically from users attempting to solve real tasks.  \n\nThird, the fact that the benchmark contains high and low level questions does not seem very significant.  Visual prompting is used entirely to make it easier for a user to specify a person or object.  The only real question is how much more effective this is than specifying people or objects with text.  If the model can interpret the visual prompting, failures to answer higher level questions have nothing to do with the prompt, but more to do with higher level reasoning. \n\nFour, one of the key questions in video understanding is the extent to which models can integrate information from different parts of the video.  Video prompting seems to be primarily used here with questions that are temporally localized, and could perhaps be answered using a single frame of the video.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (Poster)