A Shortcut-aware Video-QA Benchmark for Physical Understanding via Minimal Video Pairs
Authors: Benno Krojer
Mojtaba Komeili
Candace Ross
Quentin Garrido
Koustuv Sinha
Nicolas Ballas
Mido Assran
Venue: Accepted by TMLR ()

Abstract: Existing benchmarks for assessing the spatio-temporal understanding and reasoning abilities of video language models are susceptible to score inflation due to the presence of shortcut solutions based on superficial visual or textual cues. This paper mitigates the challenges in accurately assessing model performance by introducing the Minimal Video Pairs (MVP) benchmark, a simple shortcut-aware video QA benchmark for assessing the physical understanding of video language models. The benchmark is comprised of 55K high-quality multiple-choice video QA examples focusing on physical world understanding. Examples are curated from nine video data sources, spanning first-person egocentric and exocentric videos, robotic interaction data, and cognitive science intuitive physics benchmarks. To mitigate shortcut solutions that rely on superficial visual or textual cues and biases, each sample in MVP has a minimal-change pair — a visually similar video accompanied by an identical question but an opposing answer. To answer a question correctly, a model must provide correct answers for both examples in the minimal-change pair; as such, models that solely rely on visual or textual biases would achieve below random performance. Human performance on MVP is 92.9%, while the best open-source state-of-the- art video-language model achieves 40.2% compared to random performance at 25%.

Published in Transactions on Machine Learning Research (12/2025)
A Shortcut-aware Video-QA QA Benchmark for Physical Under- standing via Minimal Video Pairs
Benno Krojer*
benso.krooer@mmillggueel quebec
PAIR at Meta
Mila
McGill University
Mojtaba Komcili
FAIR at Meto
Candace Ross FAIR at Meta
Quentin Garrido
PAIR Meta
Koustuv Sinha
FAIR Meta
Nicolas Ballas
FAIR Meta
Mahmoud Assran
FAIR at Meta
Work done during internship.
Reviewed on OpenReview: https: openreview. net/ forum? id= guFgNJcSSu
Abstract
Existing benchmarks for assessing the spatio-temporal understanding and reasoning abilities of video language models are susceptible to score inflation due to the presence of shortcut solutions based on superficial visual or textual cues. This paper mitigates the challenges in accurately assessing model performance by introducing the Minimal Video Pairs (MVP) benchmark, a simple shortcut-aware video QA benchmark for assessing the physical derstanding of video language models. The benchmark is comprised of 55K high-quality multiple-choice video QA examples focusing physical world understanding, Examples are curated from nine video data sources, spanning first-person egocentric and exocentric videos, robotic interaction data, and cognitive science intuitive physics benchmarks. To mitigate shortcut solutions that rely on superficial visual or textual cues and biases, each sample MVP has a minimal-change pair a visually similar video accompanied by an identical question but an opposing answer. To answer a question correctly, model must provide correct answers for both examples in the minimal-change pair; such, models that solely rely on visual or textual biases would achieve below random performance. Human performance on MVP is 92.9%, while the best open-source state-of-thee the-art video-language model achieves around 40% compared to random performance at 25%.

---

Published in Transactions on Machine Learning Research (12/2025)
Capcioned robot videos
Manual categurical flltering (váa meta-data and fasonomics)
Automatic pairing of minimal video pairs
Automatic flteriin single- frame-solvable examples
Intultive phystes videos (coguiliie literature)
Kesepl relaak object
Stadurty horntt
cagineenn Vaal imper
Sagleingg Vtadiig
Egocentric videos
Video Pool
Newe
Proer'
MVP
Minimal Video Pairs
Benchmurk
Wlich rubot instruction deseribes the actions I..I!
Ia video plausible accortinn intuiline physics [-.]?
What action dil the person fail to complete
Al Move the Mae ciergle koove Do Modks
A) More the blo siangl dlove the bloks ai triangle
You evnyyling beknving scerding to ieaalive plaskal
Yes,s eveyylngg bekwing accrliin issalive plasies [-] saaeerattiu vices
Al Possas lecal slies betwenn ingediins, nor [-] i0 Porson stacked beaad
Al Possn locad sioes beta en tegreieess wot [-] is dacked bexad aizze
Bi the manple
sematiinu vidza
VideoLLM
V.deoL..M
VidooLM
VidolLLM
VideolLLM
VidooLLM
Move
Move the blke blocks
something video violates
samnething video stelatios
bread
stacked bread sllees under [--]
bovee red blocks
X
V
Minimall Pair Score
Minimal Pair Score (Moda cack to pesdit coreet))))
Minimall Pair Score (Sooii produn balk carocts)
(Modd pedeettecccnnettt
Figure 1: Illustrating MVP with its curation steps (top) and examples of our Minimal Pair Scoring (bottom).
1 Introduction
Moravec's paradox highlights a counterintuitive phenomenon: high-level reasoning lasks, often perceived as complex, are typically casier for AT agents to solve than sensorimotor and perception tasks, which are seemingly effortless for humans (Moravec) 1988).
Recently, large vision-language models have emerged as promising paradigm for enabling perception capa- bilitics in AT agents, demonstrating impressive progress question-answering tasks across various domains including movies, docunents, charts, and sciences (Alayrac et al., 2022 Team et al. 2024 Dubey et al. 2024 Wang et al. ). This progress raises a natural question: do these models possess the spatiotemporal understanding and reasoning abilities essential for an agent to interact within the physical world, or do they buttress Moravec's paradox?
Various visual QA datasets have been proposed by the community to assess the spatiotemporal understanding of video-language models (Tapaswi al. 2016 Maharaj et al. 2017
Li al. 2024b Patraucean et al.
2023 Zhang et al. 2023c Xie et al. 2025 Wang et al.) 2023c Yi* et al. 2020 one of the most popular,
MV MlVBench [Li et al. 2024b) combines 11 video datasets into single video QA benchmark.
While recent state-of-the-art video-language models obtain performance far superior to a random baseline on these benchmarks (Wang et al. 2024a Shen et al., 2024 Li et al.) 2024a), our investigation reveals that existing models can achieve strong performance on these tasks by relying on superficial visual or textual cucs biases. This validated using simple basclines that discard the visual input temporal aspect, yet. achieve non-trivial performance. Similarly, recent work (Cores al. 2024] shows that some of these tasks (Li et al. 2024b fail to accurately measure the temporal understanding of model.
In this work, we take inspiration from works in natural language processing (Levesque ct al. 2012: Sakaguchi BL al,, 2021) and image processing (Thrush et al. 2022 Yuksekgonul et al. 2022) addressing visual and textual biases evaluation, and introduce MVP, a video- QA benchmark containing cinimall-hanne video pairs (MVP). Specifically, cach vidco-question-anssee sample the benchmark accompaniod by visually similar video possessing an identical question but an opposing answer (Figure 2 To answer a question correctly, model must also provide the correct answer for its minimal-change pair while processing them
2

---

Published in Transactions on Machine Learning Research (12/2025)
100
92.9
80
Arcuracy
60
40
37.4
39.6
29.3
29.9
29.6
32.5
Random
-005
20
0
LLaVA-OneVision
Long-VU
Owen2-VL-7B
Tarsler-34B
InteraVL2.5544 5-8B
Gemin11 3-Pro
GPT-0
Human
Figure 2: Performance of the strongest evaluated VideoLLMs on MVP (minii-vrrion), compared to human perfor- mance.
independenlly. Many lypes of shorteul solutions are penalized under the minimal-pair scoring framework model relying on superficial visual or textual cues or biases would incorrectly output the same answer for both the samples in the pair.
While recent. work created small sets minimal-change video pairs for course-grained lemporal reason- ing (Zhang et al., 2024a Liu et al., 2024), ; our key insight is that these pairs can be efficiently mined from existing video sources to test for several model capabilities through an automated process relying on visual embeddings and video meta-data. We propose automatic process to find minimally different video pairs with limited human intervention, and then build these into video-question-anssee tuple with identical questions and opposing answers, enabling the scaling of the benchmark to a broad set of videos spanning diverse situations. We further process the mined samples using a model ensemble to filter out single-frame solvable examples - questions that can be answered using any single randomly sampled frame from the video to encourage a stronger focus on video understanding. We build MVP by running our process on nine video sources spanning intuitive physics understanding, spatiotemporal reasoning, action anticipation, and robotic manipulation, leading to total of 54, 828 multiple-choice video QA examples with minimal-change pairs; i.e. 27, 414 minimal-change pairs.
Next, we assess recent proprietary and open-source state-of-the-arl video-language models using MVP. Specifically, we evaluate 2 closed-source models (GPT4-o (Achiam et al. 2023 and Gemini-1.5 Pro Team
et al 2024)), and open-source lideo-language models: LLa VA-OneViiioiono LL et al. 2024ab, VideoChat2 (Li
et al. 2024b), Mini-CPM (Yao et al.] (024), Qw (Bai et al. (022). Tarsier (Wang al.) 2024a Long Vu (Shen et al., 2024), InternVI.-22 (Chen al. 2024b). We find that even proprietary models are only slightly above random and that the best accuracy achieved across models is only at around 4 in stark contrast to human baseline performance at 92.9% accuracy. These findings suggest that Andeollanuage models may still struggle with seemingly simple physical reasoning tasks, despite achieving relatively high accuracy on standard spatio-temporal reasoning benchmarks.
In short, we make the following contributions:
Analyze potential shortcut solutions on all 11 datasets the popular MVBench (Li et al. 2024b) benchmark suite, using simple basclines consisting of language-only models, ainle-frram// image models, and Socratic LLMs.
2. Introduce MVP, video QA benchmark for physical world understanding comprising minimally different videos the largest of its kind by order of magnitude with ~55K examples.
1 40.2% MVP-full and 39.6% smaller representative MVP-mini split we adopt compare expensive API models
3

---

Published in Transactions on Machine Learning Research (12/2025)
3. Benchmark closed-source and open-source state-of-the-art models and identify gap in physical world understanding; human performance on MVP is around 92.9%, while even GPT4-o and Gemini achieve around 30% compared to random performance at 25%.
We are publicly releasing MVP well as representative subset of the benchmark for faster inference (MVP-mini), together with leaderboard and casy download scripts.
2 Robustness Analysis of Existing Video-QA Tasks
2.1 Shortcut solutions in video understanding
What is classified as a shorteut depends on the skills a task is meant to test, e.g., whether we deem certain features as causal or merely spurious (Geirhos et al. 2020); detecting and mitigating shorteut learning therefore remains an open problem [Geirhos ct al.) 2020 In our sctting, we focus on shortcuts that models exploit in video lasks intended to test spatio-lemporal or intuilive physics understanding. A standard way to reveal such shortcuts is to remove or alter parts of the imput. For example, if a VideoQA model performs well with no video frames, then the task fails to measure for grounded visual understanding. Below, we investigate four such shorteut strategies, illustrated in Figure with an example from the existing MVBench dataset [Li et al. 2024b) for each of the four strategies.
2.2 Empirical investigation
We begin by examining robustness of existing video QA benchmarks to shorteut solutions based on visual or textual cues or biases. Specifically, our analysis focuses on CLEVRER (Johnson et al.
2017), Perception
Test (Patraucean et al.] 2023), STAR (Wu et al., 2021), PAXION (Wang et al. 2023c)
Moments in Time
V1 Monfort et al. 2020 FunQA (Xie et al.J 2025h, Charades-STA, (Gao et al. 2017
MoVQA (Zhang
et al. 2023b), NTU RGB+D (Liu et al.) (2220), VLN-CE (Krantz et al. 2020 and TVQA
Lei et al. 2018
which are all included in the widely adopted MVBench (Li et al. 2024b) benchmark suite.
Empirical Setup. MVBench is comprised of 20 tasks from 11 datasets, collected in a multiple-choice video QA format, where model is required to choose answer a; from tuple of question, video, and answer candidates (g, v. (ai, -.).- Following standard practice (Goyal et al.] 2017b), we study robustness to shortcuts by perturbing the task inputs, e.g., requiring the model to select an answer candidate without seeing the video or perhaps without reading the question, and compare to the accuracy achieved by video LLM without perturbing the task inputs. We study 4 types of shortcut solutions by evaluating language-only models, video-only models, single-frame models, and simple Socratic LLMs. Results are reported in Table using the original skill taxonomy outlined in MVBench.
Language only. Langunge-only models do not observe the video, and therefore select answer candidate by only considering the textual inputs q and the answer candidates 82. We leverage the Llama3-8B and Llama3- 70B models due to their competitive performances (Dubey et al. 2024). In Table | we find that a Llama3-8B outperforms a random baseline by 6%, and a larger Tlama3-70B outperforms a random baseline by 8%, suggesting that only a small subset of examples can be solved without considering the video input. However, digging into the individual datasets and sub-tasks in Table D reveals strong language-only performance on Action Antonym, where LLaMA3-70 achieves 78% compared to random baseline 50%. Upon closer inspection of the original dataset, we observe that many questions can be correctly selected by choosing the answer candidate with the highest marginal likelihood. For instance, given an example with answer candidates "book falling like a rock versus "book rising like rock," an LLM,, just like human, can rely on its language bias to infer that the former is probably the correct description without observing the video. We show a concrete example in Figure g where the LLM can rely on knowledge about TV shows without the video.
Video only. Video-only models do not observe the question, and therefore select an answer candidate by only considering the video input and the answer candidates [a1, -2, Table ] shows that video
4

---

Published in Transactions on Machine Learning Research (12/2025)
B
LCM
IregeLLM
Text-only Baseline
Single-frame Baseline
Video-only Baseline
Socratic LLM Baseline
Examples from different MVBench splits
Episodic Reasoning split
Fine-prained Action split
Counterfactual Inference split
Counterfactual Inference split
Counterfactual ("whithout red...") he ignored answer
LLM knows about TV without looking at the video
Single frame enough to see "sketching" action
Coarse-grained representation of video is enough to answer
Pooee
Poeeo 11
Princ
Replace video with mas opens drawer desk."
Perme
Pieeo
Q: Why did Rachel pretend to not hear when Ross was talking to her about the ring?
Q: Which one of these correctly matches the actions in the video?
Q: Without the red object, which event will happen?
Q: What will the person do next?
Rachel was emburrused that Ross had found about engagement
skotekong painting
Cube collides with purple cylinder object purple cyltnder
Take the paperinotebool table
collide
Rachel had infection
An overview of four plausible shortcut baseline models might rely on for video QA tasks: Ignoring the text (first), relying solely on single-frame understanding (second), ignoring the question (third), or relying on a task-independent coarse-grained representation of the image (fourth), iee. a task- agnostic caption replacing the vidco itself.
Figure 3:
LLM (VideoChat2-Mistra) can solve most sub-tasks without access to the question, reaching 50% overall accuracy; by comparison the same model achieves an accuracy of 61% when given the question in addition the video, while a random baseline is at 30%. These findings indicate that the answer candidates for each question [ar, a2, are not sufficiently task-specifie, as the model is able discard the incorrect answers without knowing question. Recent work has found similar trends in language understanding QA benchmarks such MMLU-Pro (Wang et al. 2024c where models are found to reach high scores without access to the question (Chandak et al. 2025).
This trend is particularly interesting on the counterfactual inference sub-task, where the counterfactual scenario such as "What happens if the red object is removed?" can only be known from the question. Manual inspection reveals that the correct answer in this task, e.g. "Cube collides with purple cylinder", (based CLEVRER (Yi* et al.) (2020)) often occurs in the video regardless of the counterfactual scenario, e.g., the two objects in question will collide regardless of the causal intervention. We visually illustrate this in Figure 3
Single-frame only. Single-frame models do not observe the entire vidco, but rather are provided only a single frame f. form the video. These models must therefore select an answer candidate by considering the frame fú, the textual inputs g, and the answer candidates (a1, a2, We take fa to be the center
5

---

Published in Transactions on Machine Learning Research (12/2025)
Task
Avg
AA
AC
AL
AP
A8
CI
CO
EN
ER
FA
FP
MA
MC
MD
OE
01
08
ST
SC
UA
Random Chence
0.30
0.50
0.33
0.25
0.25
0.25
0.31
0.33
0.25
0.20
0.25
0.25
0.33
0.25
0.25
0.50
0.25
0.33
0.25
0.33
0.25
GPT-4V 1
0.44
0.72
0.39
0.41
0.64
0.56
0.11
0.52
0.31
0..9
0.47
0.48
0.23
0.11
0.11
0.19
0.59
0.30
0.84
0.45
0.74
VideoChat2 (Mistral)
0.61
0.86
0.37
0.44 0.55
0.76
0.72
0.49
0.36
0.40
00.50
0.64
00.88
0.69
0.49
0.87
0.75
0.41
0.85
0.50
0.62
Langwage only: Model considers question and answer choices, withont access to the video.
Llama 3-8B Llanu 3-70B
0.36
0.63
10.38
00:27 0.25
0.28
10.35
0.44
0.29
0..43
0.299
0.29
0.38
10:27
[0.21
0.46
0.29
0.36
0.52
0.40
0.52
0.38
0.78
0.39
0.32 0.26 0.26
0.43
0.47
0.28
0.46
0.26
0.27
0.41
0.29
0.20
0.48
0.29
0.32
0.48
0.45
0.58
Video only: Model considors video and answer choioes only, without access the guestion..
VideoChat2 (Mistral)
0.50
(..0)00.8.... (0.25 0.49| 0.68 [0.74| 0.44 0.28 0.339 |0.53 0.65 0.47 |0.29 |0.26 0.53
|0.75| 0.34 0.81 [..32
0.55
Single-Pranne only: Model comsidders queation, answer chorces and a single leey frame, withontt access to the full video.
Idefies3
0.47 0.72 0.37 [0.31 0.52 0..48 0..4 0.54 0.31 00.48 |0.40 0.44
0.55
0..42 |0.34
0.49
0..50 0.37 0.78
0.48
0.60
Qwen2-VL
0.51 0.87 0.37 0..31 0.55 0.541 0.37 0.59 0.10 0.15 0.11 0.53 0.6 00:13 [0.37 0.53
0.54 0.39
0.74
0.12
0.68
Simple Socractic LLM: Model considers the question. answer choices and short generric description of the video.
Llama 3-8B
0.44
0.56
0.38
10.28
0.49
0.57
|0.35 0.53
0.29
0..42
0..30
0.35
0.56
00:42
00:32
0.50
0.56
0.35
0.68
0.44
0.56
Llama 3-70B
0.46
0.67 0.32
0.35
0.40 0.55
0.38 0.55
0.24
0.45
0.36
0.41 0.56
0.46 0.32 0.57
0.62
0.35
0.70
0.39
0.54
Table 1: Shortcut Analysis on the 20 MVBench tasks from 11 datasets: Optimal performance on these spatio-temporal reasoning benchmarks is frequently achieved by models relying on visual or textual biases (Single- Framee only, Video only, Simple Socratic LLM). 1; GPT-1V accuracy from [Li et al. [2021b]. Tasks: AA (Action Antonym), AC (Action Count], AL (Action Localisation), AP (Action Prediction), AS (Action Sequenee]. CI CTCCanateraactull Inferencr), CO (Character Order), EN (Egocentric Navigation), ER (Episodic Reasoming). FA (Fine-grained Action), FP (Fine-grained Pose), MA (Moving Attribute), MC (Moving Count), MD (Moving Direction), OE (Object Existence), O1 (Object Interaction), OS (Object Shuffle), ST (Scene Transition), SC (State Change), UA (Unexpected Action).
frame from the video and leverage Idefics3-8B Laurençon et al., 2024) and (0wen2-VL-71) (Wang et al. 2024b) for the single-frame baselines. In Table 1 Idefics3-8B achieves an overall accuracy of 47% and Qwen2-VI.-BB achieves an overall 51% accuracy, which is comparable to the performance of full-fledged VideoLLMss Morcover, on Action Antonym, Action Prediction, Character Order, Egocentric Navigation, Episodic Reasoning, Fine-grained Action, State Transition, and Unexpected Action, the single-frame models are on par with (or even exceed) the performance of the VideoLLMs. Concurrent work (Cores et al.. 2024) also studies the related bag-of-frame bias by shuffling the video frames.
Simple Socratic LLM. A Simple Socratic LLM (Zhan el. al. 2023a Zeng et al. 2023 replaces the video
input a with a short caption e, that can only convey a low-bandwide wiith description of the video. In practice, co is or 2 sentence-long caption generated by a separate VideoLLM (Zhang et al. 2024b) in a task-indeeendeen manner. The Socratic LLMs therefore select an answer candidate by only considering the bonddandwitth caption Ce, the question q: and the answer candidates (a1, -2]- Following the text-only baselines, we use Llama3-8B and 70B. The performance of the Simple Socratic LLMs in Table D is significantly above random, with 44% for the LLaMA3-SB and 47% for the LLaMA-70B, suggesting that many sub-tasks (e.g. Character Order, Episodic Reasoning, Scene Transition) do not require fine-grained scene understanding.
Summary. The shortcut analysis reveals that existing models can often achieve strong performance on spatio-temporal reasoning benchmarks by relying on language cues (Tanguage only shortcut) or visual cues, (Video only shorteut), and may not need to perform temporal reasoning SSingle-Framan only shortcut), or possess fine-grained visual features (Simpliffed Socralic LLM). The first two baselines, text-only and single- frame, are what one might call "hard requirements" for video understanding benchmark: no example should be solvable in these scttings. On the other hand, video-only (selecting most probably answer candidate without access to the question) and Simplified Socratic LLM (replacing video with a short caption) baselines are akin to "soft requirements": they might point to issues for certain examples for which you expect to require this information (e.g., MVBench's 's counterfactual sub-task should not be solvable without reading the question), but it does not imply a flaw in all cases.
3 Testing Physical World Understanding via Minimal Change Pairs
In this section we discuss the construction of MVP to mitigate shortcut solutions based on visual and textual biases. MVP is comprised of 828 video QA examples covering various aspects of physical world
6

---

Published in Transactions on Machine Learning Research (12/2025)
DOMAINS
VIDEOQA-FXAMPLES EXAMPLES ROBOTICS
TOTAL
NATURAL VIDEOS
INTUITIVE PHYSICS
SYNTHETIC VIDEOS
MINIMIALLY DIFE. VIDEOS
PROCEDURAL
SINGLE- FRAME SINGLE-FRAME
BENCHMARK
BIAS FILTERING
FORMAT
CLEVRER
76.3K
21.4K
OK
76.3K
X
X
MC-QA
PERCEPTION TEST
11.5K
11.5K
0-0.2K
0K
OK
X
X
MC-QA
MVBENCH
4K
2.8K
OK
0.2K
1.2K
X
X
MC-QA
TVBENCH
2.5K
1.9K
0K
0.2K
0.6K
X
X
MC-QA
VINOGROUND
1K
1K
OK
0K
OK
V
X
GROUP-SCORE
TEMPCOMPASS
0.5K
0.5K
0K
OK
OK
V
X
GROUP-SCORE
MVP
54.8K
22.3K
25.8K
32.6K
/
PAIR MC-QA
Table 2: We compare with recent benchmarks that focus on similar skills. Note that some videos may fall within several categories (e.g., synthetic intuitive physics videos). MVP contains minimally different videos at much larger scale and across more diverse domains. From these benchmarks, MVP the first to procedurally filter out examples due to single-frame bias. GROUP-SCORE = Present one video + two captions, and two videos one caption. CLEVRER's intuitive physics entry grayed as it only covers a narrow subset of intuitive physics concepts, largely based on collisions.
Benchmark Category
Sources (# paired video-Q2 QA ex- amples)
Example
Fine-grained human-object in- teractions
Perception Test
[Patraucean et al.
Q: What stops the motion of the object placed on the slanted plane after being released f...]? A) Person or collision with another object B) High friction with sur- face
2023) (3.5K), Something Something v2
Goyal et al.
2017a (3.6K)
Fino-grained robot-object in- teractions
Language Table Corcy (129K))
2022
Q: Which robot instruction best describes the actions in the video? A) Move the green blocks vertical line below blue cube B) Move the green blocks and blue cube in vertical line
Intuitive physics and collisions
IntPhys InfLevel GRASP Jassim et al. 2024
Riachet et
2022
Q: this video physically phousibbe//possiine accord- ing to your understanding of c.g. object permanence, (2.0K), gravity, [...] A) Yes, everything is behaving accord-
(0.2K),
Weihs et al. 2022
(2.6K),
CLEVRER (Y** et al.
2020
(1.2K)
ing to human intuitive physics understanding B) No, something in the video is off/strange or violates :-.]
Coarso-grained temporal soning
STAR
Wu ct al. 2021
(1.0K),
Q: What is the best caption for this video? A) The kayak flips downwards B) The kayak flips over from facing down- wards towards facing upwards
Vinoground
[Zhang
2024a
from facing upwards towards facing
(0.5K)
Table 3: Overview of MVP. Each answer option A/B is correct for only one video in the minimal-change pair, while acting as hard negative for the other video (by curation design). Note that we show the number of paired video-QA examples, thus the number of videos in our data is twice that amount.
understanding. including spatial reasoning, temporal understanding. human-object interaction, memory, counterfactuals, anticipation, and intuitive physics.
3.1 Task Definition
Designing MVP as a multiple-choice QA task ensures simple unified setup that is casy to adopt by the community. At the same time, multiple-choice QA is highly flexible since most lasks can be reformulated as QA (several videos we source examples from were not originally designed for QA, such as Language Table). As result any VideoLLM can be evaluated on MVP off-tho-sholf. By contrast, other task formats, such as localization, have distinct advantages that may allow pinpointing the exact temporal or spatial failures (Cheng al. 2025), however they do not allow the same ease of adoption and flexibility: applying

---

Published in Transactions on Machine Learning Research (12/2025)
VideoLLM to such task can require careful attention to the CoT prompts for different sub-tasks, and involve mix of more complex metries such as "mean visual Intersection over Union."
Task formulation. To improve robustness to the various shortcut solutions described in the previous section, we adopt a minimal-change pair approach (Levesque et al.) 2012 Sakaauchi et al. 2021). An example in MVP consists of two video QA pairs (71 U1, [11, a2]) and (92, 2/2, [2]) containing identical questions 41 = 42 visually similar videos 01 ~ U2, and two mutually exclusive (i.e.; contradicting) answer candidates an and 02.
Minimal-change Pair Scoring. model relying on superficial visual or textual cues or biases to solve task will tend produce the same output for each sample in the minimal-change pair. Thus, to penalize models for latching onto shortcuts, we only provide a positive score if the correct answer is produced for both minimal-change samples; the model receives each example (1, V1, [81,82) and (q, 12 [a1, a2]) ) in isolation. Following a multiple choice QA framework, the model has to output single answer letter (A or B) via task- specific prompts. In this setup, random baseline achieves an accuracy of 25%.
3.2 Data Curation
Question Taxonomy. We wish to understand whether video LLMs possess the spatio-temporal under- standing and reasoning abilities essential for agent to interact within the physical world. As such we consider a coarse-grained taxonomy of question categories encompassing:
Fino-grained human-object interactions,
Fino-grained robot-object interactions,
Intuitive Physics understanding,
Coarso-grained temporal understanding.
We intentionally construct samples that are not overly reliant cultural knowledge (Raaal et al. 2024 He
et al. 2024 Li el al. 2024c (e.g,, movies) or specific domain knowledge (Tang el al. 2019 (e.g., delailed tasks where language bias could contribute to the general performance.
recipes)
We first manually filter vidcos from the sources described Table based on manual inspection (cf. Sec- tion C.1. 1, then convert. them into a question-answer formal based on the associated mela-data (the tex- tual captions for Language Table, the class labels for Vinoground, STAR, and CLEVERER, and the concept labels for IntPhys, InfLevel, and GRASP), yielding starting set of 548K video QA examples.
QA annotations for PT,
Minimal-change Pair Mining. Next procedurally identify minimal-change pairs from the 548K video QA examples produced from the previous stage. We note that 16% of the videos in our final benchmark (~ 8.8K examples) already possess explicit minimal visual pairs (even though most of these videos are not in a suitable format for video QA benchmarking, they can be converted into a minimal-change pair question- answer format based on the associated meta-data). For the remaining 84% of the videos, we leverage the following procedure to construct visual minimal-change pairs. In this process, we search for samples that have visually similar videos, identical questions (based on semantic matching), and contradictory answers. To then determine whether two videos with the same question are suitable minimal pairs, and neural rules to determine video similarity and b) entailment detection (Bowman et al. 2015 Dagan et al.] 2013) between the correct answers of cach video. Whether we rely more on symbolic or neural rules of similarity depends on the data source: If a dataset has rich annotations (positions or altributes of objects) or structured captions (such as CLEVRER or Something Something-v22 rules and the NLP toolkit spacy (Honnibal & Montani 2017) to narrow down the candidate pool of minimal pairs. This step would match videos with large intersection of objects or attributes mentioned in the annotation/cappe /caption, leading to highly similar videos (e.g., the same objects appearing in both videos). Once
a) symbolic
we use hand-crafted
8

---

Published in Transactions on Machine Learning Research (12/2025)
we have narrowed down the pool of candidate pairs, in the final step we rank video pairs by their cosine similarity in the VICLIP (Wang et al. 2023b) video embedding space. We then select the top-ranked minimal video pairs such that each question or skill-type is sufficiently represented. At the same time, we ensure that the correct answers for samples in a minimal-change pair are sufficionlly different, as the correct answer of one element in the pair must be a truly negative (negallive) answer candidate for the other element, and vice versa: To avoid cases where both answers could be true at the same time (e-g., synonyms or more subtle cases) we define a set of textual rules to detect entailment for a subset of datasets. To illustrate this, in the Fine-grained Robot-object interactions category, our entailment-detection would discard the following pair of answers: A) "Move the blue cube towards the red heart" and B) "Move the blue cube to the left of the red heart", since A entails B. After this minimal-pair mining, we are down to 70K QA examples; cf. Section for technical details of the minimal pair mining process.
Single-frame Bias Filtering. Finally, to address single-frame bias, we remove examples that can be solved without the temporal information in the video; i.e., using only a single frame. We note that the input frame for this filtering stage should not be selected in a "smart way," since key-frame selection can be regarded as basic form of temporal reasoning. In practice, five state-of-the-art multi-modal LLMs (LLama3.2 2- BBB (Dubey el al. MLaVA-OneVision
2024), Molmo-7B (Deitke el al., 2024), Pixtral-12B (Agrawal et al.) 2024). (Li et al. 2024a), Tdefics3-8B (Laurençon et al. 2024)) are prompted to answer the video-QA questions and "give their best commonsense guess given a single frame sampled from a video." If at least out of 5 models in the ensemble predict the correct answer given the same frame, then we flag that frame as soluable. The minimal-change pair is then discarded if 30% of the frames in both videos are deemed solvable. This houristic process removes around 20% of the samples from the previous stage.
MVP Statistics. We end up with 54, 828 examples in MVP, grouped into 27, minimal- video QA pairs. A breakdown of these examples is shown in Table 2 and Table B with a rcasonably balanced split between natural videos, synthetic videos, robotics videos, and intuitive physics videos. An average video is 8.8 seconds long, the answer candidates contain an average of 8.1 words, and the datasets contains 2355 unique words in the questions and answers. Note that the word diversity is much less than MVBench (Li et all, 2024b), which has only 4K examples but twice the number of unique words (4338), reflecting focus in testing for physical world understanding and not linguistically-diven tasks with cultural or domain knowledge. Instead the task difficulty arises from the physical and perceptual aspects of MVP.
MVP Data Quality Inspection. In this section we briefly examine the data quality of MVP, i.e., how many examples are noisy or ambiguous. While we regularly inspected paired examples during pipeline design, it is important to also assess the final benchmark. Our overall human bascline of 92.9% (see Section 4 is a strong indicator: most paired QA examples are clean and solvable by humans.
On STAR, humans struggled for two reasons in rare cases: (i) the relevant action is sometimes not clearly visible (often in kitchen or living room scenes), or (ii) answer options in minimal pairs are too similar when the object is only partly visible. For example: Q: What did the human pick up? A) The sandwich B) The dish. Similarly, in Percoption Test, rare cases were unsolvable due to ambiguous questions or because our answer matching produced two valid options. For instance, two candidate list might differ ouly in order but boil down to the same meaning: Q: [.../ What repeated actions did the person do? A) juggling, folding, [...] vs. B) folding, fuggling, (--/. Finally, we also investigate on Language Table whether our hand-crafted rules for selecting minimal pairs works intended: Out of 4 pairs of examples that human annotators had not successfully solved, one was simply human error while the other three were issues with the underlying source data itself. In the minimal pair of "move the red circle left to green star" and "move the red circle diagonally top left to green star", while the first caption mentions "left to green star" in reality the vidco showed more of "top left", thus making it ambiguous. We largely miligate such non-exclusive candidates, as with Language Table's hand-crafted rules (see Appendix Section Still, some edge cases remain, as illustrated above.
4 Empirical Results on MVP
We evaluate several saate-of-the-ars open-source VideoLLMs on MVP, summarized in Table LLaVa- OneVision [Li et al. 2024a), VideoChat2 (Li et al. 2024b), Mini-CPM-v 2.6 Yao et al. 2024 Qwen2-
9

---

Published in Transactions on Machine Learning Research (12/2025)
Model
MVP (macro-avg)
Fine-grained human-object interactions
Fine-grained
Intuitive physice collisions
Cuarse-grained
robot-object interactions
temporal reasoning
Random
25.0 (25.0)
25.0 (25.0)
25.0 (25.0)
25.0 (25.0)
25.0 (25.0)
Any text model
0.0 (0.0)
0.0 (0.0)
0.0 (0.0)
0.0 (0.0)
0.0 (0.0)
Single-Frand Fralle Baseline (occess to question, answer choices, and a single key frome from the video.)
LLaVA-OV (Quea2-1B)
11.8 (11.7)
14.7 (12.2)
8.7 (10.5)
2.0 (2.3)
21.6 (21.9)
Qwen2-V VL (7B)
16.7 (15.7)
16.9 (13.6)
20.1 (19.9)
3.7 (4.4)
26.3 (24.8)
InternVL2.558B
19.1 (18.0)
19.2 (15.1)
18.3 (17.8)
9.8 (11.7)
29.3 (27.2)
Video-only Baseline (access to video and answer choices, but removing the question.)
Qwen2-VT, (7R)
27.3 (26.2)
30.8 (28.5)
27.4 (25.8)
6.0 (6.9)
45.0 (43.5)
InternV1.2.5888
37.3 (35.1)
38.9 (32.8)
36.5 (32.9)
16.0 (17.5)
58.1 (57.2)
baseline (access question, answer choices and short generic description of the sidoo)
Llama3-8h
(19.7) (15.5)
(20.3)
(27.3)
(12.5)
(18.9)
Llama3-70B
(16.9)
(11.7)
(10.6)
(22.7)
VideoLLMs (full acceas to the video, question, and answer choices.)
LLaVA-OV (Qwen2-7B)
20.7 (20-5)
24.3 (21.8)
5.2 (5.2)
5.8 (6.8)
47.5 (48.2)
VideoChat2 (Mattad-7B)
23.3 (22.0)
25.7 (21.0)
21.4 (20.1)
10.1 (11.5)
35.8 (35.3)
Miui-CPM-v 2.6
21.7 (20.2)
21.3 (19.0)
18.0 (14.1)
9.2 (8.9)
38.3 (38.7)
Qwen2-VL. VL (7B)
30.0 (29.9)
27.1 (30.0)
27.6 (25.7)
20.0 (19.1)
45.2 (44.7)
VU (LAaMA3-8B)
20.6 (20.6)
15.8 (14.1)
14.8 (16.0)
16.2 (16.7)
35.4 (35.8)
LongVU VU (Qwen2 7D)
29.9 (29.3)
28.9 (26.3)
21.5 (21.8)
20.5 (22.3)
48.6 (46.7)
Tarsier-7B 7B
26.0 (24.3)
31.3 (24.5)
18.7 (18.2)
15.0 (16.3)
38.9 (38.2)
Tarsier-3IB
38.8 (37.4)
45.2 (38.7)
36.3 (36.6)
21.0 (22.1)
52.7 (52.4)
InternV1.2.5-88
40.2 (39.6)
43.7 (37.8)
40.2 (39.1)
22.8 (23.9)
54.4 (57.7)
Gemini-1.5 Pro
(29.6)
(43.1)
(15.5)
(19.6)
(40.2)
GPT4-0
(32.5)
(36.1)
(32.8)
(16.2)
(45.0)
Human
92.9
91.3
91.7
97.6
90.9
Table 4: Accuracy on MVP and MVP-mini in parentheses. MideoLLM-performenn is slightly greater than random chance, while humans achieve greater than 90% accuracy on all categories. Results for closed- source models are only shown on MVP-mini due to API costs. Performance is mcasured via Minimal Pair Score, wherein a model obtains a score iff the prediction for both QA examples of the pair is correct. t; If temperature of the LLM is zero, since from a text-side both examples in the minimal pair look the same.
VL Wang et al. 2024b), Tarsier (Wang et al., 2024a 7B/34B LongVU Shen et al. 2024 InternVL2.5-
8B CCen et al. 2024b), Gemini-1.5 Pro (Team et al. 2024), and GPT4-o (Achiam et al. 2023). Most notably these models differ in their generality: The models we evaluate are either generalist models (GPT4-0, Gemini 1.5), specialized for any visual inputs (LLa VaLLVaa-OeVisiin Mini-CPM, Qwen2-VI, InternVI), specialized primarily for videos (VideoChat2, LUngVU). We also consider the same baselines in Section 2 text-only, single-frame, vidco-only (i.c. removing the question) and "Socratic LLM" (feeding a generic thort/medium length caption to an LLM Note that we additionalyy evaluate on a smaller balanced version of MVP, dubbed MVP-min,, with around 1/3 of the original size 3 For a fair comparison, we adopt the default parameters for all models, see Section for details and for the full prompt to the models.
4.1 Main Results
Overall performance of VideoLLMs. Despite their strong performances other video QA bench- marks (Li et al. 2024b Liu et al. 2024 Mangalam et al. 2024 Xiao et al. 2021), Table shows that most models perform around random chance (25% accuracy) with the exception of the Tarsier-34B model and InternVL2.5, reaching an average accuracy of 38 and 40.2 respectively. This is in contrast to human performances which obtain an average accuracy of 92.9% on a representative subset of MVP (cf. Section
While average performance is close to random for most models, we do observe non-trivial performance on several sub-tasks and data sources. In particular, VidcoLLMs achieve better than random performance
We captions from Qwe72-VI--BB after testing both 1.2.58VT.2.588B. and Qwen2-VT.-7BB captioners
We release MVP-mini for faster evaluation and lower costs of API models.
10

---

Published in Transactions on Machine Learning Research (12/2025)
Course-grained temporal reasoning, meaning they possess some ability to distinguish the order of events in video.
All models fall short on Fine-gruined robot-object interactions, which involves understanding fine-grained object manipulation on a table with a robotic arm. This is particularly interesting given the proliferated usage of multi-modal LLMss for learning large-scale visuomotor control policies (Driess al., 2023 Jiang et all 2023). Most notably, the Infuitive physics category of MVP is by far the hardest with sub-random scores. As highlighted by previous works, intuitive physies concepts such object permanence
gravity are easy for humans but remain hard for Al systems (Riochet et al. 2022 Jassim et al. 2024 Weihs et al,
2022 Du et al. 2023 Garrido et al., 2025 Bordes et al. 2025). While some methods have shown non trivial performance recently, LLMs in particular tend to achieve low performance (Garrido et al. 2025
Finally we observe that shortcut- thortcut-based baselines perform poorly on MVP (see upper rows in Table 4 confirming our efforts to mitigate shortcuts the benchmark. Only the video-only baseline (Vidco LLM must pick the most likely answer candidate without access the question) reaches close the standard VideoLLM accuracy, e.g., InternVL2.5-2 reaches 39.6% on MVP-mini and 35.1% when removing the question (recall that random accuracy is 25%). However this only marginal performance degradation is mainly driven by the fact that accuracy on the Coarse-grained temporal reasoning split (sourced from STAR and Vinoground) remains stable in the video-only baseline: Here the question is indeed often not needed since both answer options are essentially captions. For many tasks the question might provide some context but is not strictly needed for capable models, e.g., "What action is being performed in the video? A) running, B) walking".
Fine-grained failure analysis. this section we provide further insights where exactly models are falling short of human performance. We quantify failures more fine-grained level via subsets of MVP: Some sources in MVP are further divided into fine-grained splits by their original designers, where each split tests for specific ability (e.g., object permanence, shape consistency, motion consistency, elc.). We show detailed model performance on the fine-grained splits in Appendix Section and discuss the most interesting cases here.
While performance on all intuitive sources mostly below random, we find that some stronger models obtain non-trivial performance on four types of examples: Gravity- Continuity (LongVU (Qwen2-7B), 39.1%), Un- changeableness VU (LLaMA-3.2): 35.2%, LongVU (Qwen2-7B): 42.2%); Gravity-Suppor (Tarsicr-34B: 35.2%) Continuity (InternVL2.5-8B. 42.2%). We illustrate these numbers with two interesting examples: Figure 5 shows an example from Unchangeableness where both LongVU variants perform above random. However in another very simliar example in the Unchangeableness category shown in Figure 7 accuracy of all models is near or exactly 0%. Both examples showcase that objects cannot change color after being occluded for a short period of time. In the first example the ball is moving behind the wall while in second example the wall itself is moved up and down. These two examples highlight that even these rare success cases intuitive physics not particularly robust under slight perturbations to the setup.
A) Yes, everything is behaving according to human intuitive physics understanding
B) No, something in the video off/strange or vi- olates human intuitive physics understanding
Figure 6: One of the few intuitive physics categories where some models perform above random: A MVP sample with videos sourced from the GRASP dataset. In the two videos the ball passes the central wall and cither changes color (right vidco, violating intuitive physics) or remains the same color (left vidco).
Even some of the weaker models can achieve performance far above random on the MVP split Fine-graine human-object interactions, when looking closer into Perception Test categorics such as Counterfactual (c.g.. Qwen2-Vis VL: 46.5%) and Memory (c.g., LongVU (Qwen2): 40.4%). We note that the questions in these two categories often involve significant symbolic skills next to the spatio-temporal aspect, such as reversing
11

---

Published in Transactions on Machine Learning Research (12/2025)
A) Yes, everything is behaving according to human intuitive physics understanding
B) No, something in the video is off/strange or vi- olates human intuitive physics understanding
Figure 7: One of the many intuitive physics categories where all models perform around 0% An MVP sample with videos sourced from the GRASP dataset. In the two videos a wall is moved up and down to either reveal that one object has changed color (right video, violating intuitive physics) or remains the same video (left vidco).
lists ("What if objects are put in reverse order?"), or OCR abilities. Thus if the spatio-temporal aspect is relatively straightforward, the LLM alone might get far with its abstract reasoning.
Finally, to the human eye examples where vidcos are sourced from Language Table and CLEVRER in MVP might look very simílar (see appendix Figure 10 and Figure 12 : They are both simple environments containing basic objects with canonical shapes. However, Language Table performance is generally around random while CLEVRER above 50% in most cases; this counterintuitive as CLEVRER questions should require more reasoning such counting ("How many objects were moving at the end?") or prediction ("What will happen when the video ends?"). We olfer several explanations: First, we observe that the video pairs in Language Table are closer to true minimal pairs; this is likely due to the fact that the Language Table video source is 10 times as large, making it easier for our pipeline to find hard minimal-change pairs. Second, due to the more complex reasoning in CLEVRER, there is also more room for biases to creep in. Third, Language Table videos carry subtle difficulties because it has "messier" distractor movements: humans teleoperate a robot arm and sometimes bump into other objects before taking the actual action. Finally, many VidcoLLMs train on videos from the training split of CLEVRER and also the original CLEVR dataset (Johnson et al. 2017), while Language Table is robotics dataset, and it is less likely that these robotics videos have crept into the training pipeline of common VidcoLLMs.
4.2 Ablations and Analyses
Importance of Data Curation. In Table we explore the effects of the minimal-change pair mining and single-frame bias filtering on model performance. For this exploration we use the smaller MVP-mini (see Section B) and report the average performance of five VideoLLMs F
When pairing vidcos randomly instead of using minimal-change pairs, the average accuracy across tasks is at 45.4%, far superior to random chance. Using mininal-change pairs, the average VideoLLM performance significantly drops to 27.3%. This result shows the importance the minimal-pair framework and suggests that VidcoL.LMs can frequently leverage shortout solutions or spurious features to solve QA tasks. Addition- ally, the average VidcoLLM performance drops again by another 2.2% to 25.1% by removing single-frame solvable videos, with much larger drops on certain subsets. Note that while Fine-gruined robot-object inter- actions and the Intuitive physics and collisions categories contain almost single frame biases, we see significant drops of 3.5% and 3.3% for the other two categorics (Fine-grained human-object interactions and Coarse-grunned lemporal reasoning) with this additional filtering step. Overall, Table 5 confirms that the minimal-change pair mining and single-frame filtering pipeline is effective at mitigating potential shortcut solutions in MVP.
Influence of single-letter output format. When evaluating VideoLLMs on MVP, we ask models to produce single-letter answer in the format Answer: A/B (full prompt in Section H. This design keeps inference simple and uniform across diverse APIs and architectures. However one critique might be that models would benefit from open-ended output formats that encourage reasoning before answering. To test
A..VVA-VV, VidenChat, Qwen2-VT, LongVU (Qwen2), Tarsier-BB
12

---

Published in Transactions on Machine Learning Research (12/2025)
Fine-grained
Fine-grained
Intuitive physies and collisions
Coarse-grained Temporal reasoning
Model
Overall
human-object Interactions robat-object. Interactions
Pairing of random videos fwith question)
Avg. VideoLLM Acc.
45.4
36.8
40.9
19.7
84.3
Pairing minimally different videos
27.3
28.7
18.6
16.7 J3.0
45.1
Arg. VideoLLM Ace.
118.1
48.1
122.3
J39.2
Remove fomete-fume-soosable exomples final version of MVP
25.1 .2.2
25.2
18.3
15.2
41.8 13.3
Avg. VideoLLM Ace.
13.5
10.3
J1.5
We ablate the effect of our main curation steps. Both the automatic pairing of minimal pairs and the single-frane-bias filtering lead to lower average model performance, with especially large drop once introduce the minimal pair setup.
Table 5:
this, we update the prompt for Gemini 1.5 to explicitly request reasoning before committing to final answer (Short reasoning prompt):
Short reasoning:
Based on your observations, reason about the following question [...] in 1-3 sentences: {QUESTION OPTIONS). Then after the reasoning, select the best option [...]
Somewhat surprisingly, performance decreases, in line with recent work on output formatting in LLMs (Long ot all
2025 see Table
To investigate further with non-API model, provide our strongest open-source VideoLLM (InternVL2.55 8B) with the same Short reasoning prompt variant and additional step-by-step Long reasoning prompt:
Long reasoning:
Reason about the answer step by step, and identify key positions of objects and the times of key events in the video. Analyze the spatial relationships, temporal dynamics, and causal connections in the scene. Based on your detailed observations and step-by-step reasoning, address the following question: {QUESTION + OPTIONS). After your detailed analysis, select the best option [...]
While performances does not decreases for InternVL2.5-8B,, remains the same (Short reasoning) or only increases minimally (Long reasoning), see Table G Overall, these results suggest that the primary bottleneck might not be symbolic reasoning but fundamental perception.
Model Prompt
MVP (macro-avg)
Human-Obj Robot-Obj
Int. Phys.
Temp. Reason.
Gemini 1.5 (defanlt)
30.0
27.1
20.0
21.0
15.2
Gemini 1.5 (+ Short reason)
21.2 (-8.8)
22.5 (-4.6)
23.9 (+3.9)
9.9 (-11.1)
28.5 (-16.7)
S.kernVL.2255-885 (default)
39.6
37.8
39.1
23.9
57.7
L.ternLL2.5-881 (+ Short)
39.6 (+0.0)
37.6 (-0.2)
38.8 (-0.3)
24.2 (+0.3)
58.0 (+0.3)
InternVL..5-8B (+ Long)
39.8 (+0.2)
38.0 (+0.2)
38.6 (-0.5) 24.9 (+1.0)
57.9 (+0.2)
Table 6: Effect reasoning prompts on VideoLLM accuracy. Default rows (gray) serve as baselines.
Variants report raw accuracy and relative to their bascline.
Modee
MVP (moero-ang)
Human-Ubi
Robot-Okt
lst. Plys.
Temp. Hasson.
Model
MVP (marrowwug)
Human-Oki
Robot-Ubb
Int. Plys.
Thmmp, Reason.
Mini CPM-v 2.5
37.5
57.3
52.5
52.7
67.5
Mini CPM v 2.6
74,9
76.5
76.8
87.7
58.4
QuenZ VL. (71)
02.6
629
60.1
57.4
70.0
Quen2.V VL (71)
65.6
65.7
Q8.7
76.5
G1.4
InternV1.2.5.88
Intor
Table 7: Single-example accuracy MVP-mini.
Table 8: Same-answer rate MVP-mini.
13

---

Published in Transactions on Machine Learning Research (12/2025)
Additional metrics. We further analyze MVP-mini with representative set of open-source MidsoLLMs, ranging from lower-performing (MiniCPM-v 2.6), to mid-level (Qwen2-VT. 7B), to higher-performing (In- ternVL2.5-88B.. Recall that our main metric, the Minimal Pair Score, assigns a point only if the model answers both examples in a pair correctly). We complement this with two auxiliary metrics: single-erample accuracy and same-answer rale (the fraction of pairs where the model predicts the same option for both items).
The latter captures a common failure mode: insensitivity to the min- imal change, leading to identical responses for both videos. Both metrics are shown in Table and Table g Single-example accu- racy helps reveal that the Minimal Pair Score captures more than isolated correctness
Celibretion curve for Single example Accuracy
0.8
it reflects whether models latch onto con- stant features across both videos. For instance, Qwen2-VL. achieves a single-example accuracy of 62.6, an a Minimal Pair Score of 29.9, which is considcrably lower, indicating that crrors are not indepen- dent. Similarly, same-answer rate is far above "random chance" of 50% in most cases.
tegge-oanggeee accurar
02-
We also analyze the relationship between model confidence and single-example accuracy, plotting reliability diagrams for two repre- sentative models in Figure S For each question, we extract the prob abilities of the answer tokens "A" and "B," interpret the probability of the predicted option as the model's confidence, and normalize PA_, } Note that this value is by definition e.g. f max PAtPP' patpa at minimum 0.5, as shown in Figure Binning these confidences reveals how well the self-reported probabilities align with empirical accuracy. InternVL2.5-SB exhibits particularly strong calibration, whereas Qwen2-VL tends to be over-confident in the mid-confidence range.
Qwer78
MorVL
0.0
Confidence predicted
Figure 8: Calibration curve for two models (Qwen2-VL and InternVL2.5). plotting model confidence against ac- tual accuracy.
Relationship between accuracy and difficulty proxies. We ask whether certain proxies of how dif- ficult an example in MVP is negatively correlate with performance. For example on the vision side, we measure whether a longer video is harder for models on average. On the text side, we use two standard proxies, computed from the question and answer options (averaged): (i) Lexical frequency (Avg Zipf), us- ing wordfree's
- Zipf scale, where higher values indicate more common words; and (ii) Readability (Avg FleschKincaid grade"), where higher values indicate harder text (e.g. more words per sentence). We report Spearman correlations between single-cxample accuracy and each proxy (overall and per subtask). Overall, we find small or negligible correlations, except for the sentence difficulty, see Table g Moreover there is stark differences across the different data splits. Ultimately, there is no single easy proxy for task difficulty: We have seen that models do poorly on intuitive physics examples (Table 4), yet many intuitive physics examples are short videos. A concept like object permanence can be captured in a short vidco containing few objects, and does not need many words to be probed yet models struggle.
Difficulty Proxy
Overall
Human-Obj Robot-Obj Int. Phys.
Temp. Reason.
Video duration
-0.072
-0.095
-0.013
-0.225
0.165
Zipf FleschKincaid
-0.024
0.039
-0.033
-0.102
-0.134
-0.204
0.055
-0.002
-0.244
-0.139
Table 9: Correlation of difficulty proxies with model accuracy. proxies with model predictions aggregated from different models.
We compute correlations of
Dyppioorg/project//orrd
githab.com/ wwwoan//wwww///ooeee wwords//lesch-
14

---

Published in Transactions on Machine Learning Research (12/2025)
5 Related Work
Language biases in Vision-Language models. Vision-and-languag benchmarks, such Visual Ques- tion Answering (VQA) (Antol et al. 2015 Goyal et al. 2017b; Marino et al. 2019) have been found to be vulnerable to language blases as evidenced by the performance of "blind" language-only models. Blind mod- els are routinely shown to be efficient. at solving many of the vision-and-language tasks (Goyal et al.) 2b;
Zeng et al. 2023 Chen et al.
2024a], and can also solve several imago-text retrieval benchmarks (Yuksck-
gonul el. al. 2022
Hsieh et al.
2024) using language biases (Lin el al. 2023). Visual Question Answering
the video-language domain
(ideo-QA) (Li et al.) 2024b He al. 2024 Xiao et al. [021] Lei et al. 2018)
Majumdar et al. 2024 Tapaswi et al. 2016 Rawal et al.
2024) also exhibits language biases, as shown in
the performance of strong language-only baselines (Zhang et al. 2023a Cores et al. 2024
Vision-centric biases in Vision-Language models. State of the art vision-language models are shown to be surprisingly unaware of the vision inputs, where they often struggle with simple questions due to incorrect visual grounding (Tong et al. 2024), despite leveraging sufficiently powerful visual embeddings. VLMs are shown to be imprecise at spatial information understanding and geometry (Rahmanzadchgervi et al. 2024 Kamath al. 2023). Similar biases exists in video-and- languuuge tasks, where LideoLLMs typically exhibit single-frame bias (Buch et al., 2022; Lei et al. 2023 spatial bias (Cores et al. 2024p. where either single frame is enough to solve the task, or the ordering of the frames is not important. To overcome this bias, benchmarks propose computing temporal certificate sets (Mangalam et al. 2024). key-frame bias (Buch et all 2022), or investigate temporal understanding through shuflled frame inpuls (Cores et al.) 2024). In MVP, we operationalize a looser definition of temporal understanding for our filtering pipeline (Section 3) in that we keep an example if it is only solvable given the right key-frame, but discard it if it can be solved with any randomly sampled frame- the intuition being that key-frame identification can already involve temporal reasoning.
Benchmarks addressing vision-and- biases. Several approaches are proposed in the literature to reduce the aforementioned biases in Vision-Languaa systems. One promising approach is to use minimally
different pairs of inputs (hrush et al. 2022
Yuksekgomul et al. 2022 Hsieh et al., 2024 Krojer et al.] 2022
Wang et al. 2023a), also known as Contrast Sets (Gardner et al. 2020 which stem from related work in
natural language processing (Levesque et al., 2012 Sakaguchi et al. 2021 McCoy et al. 2019). Minimally different input pairs restrict the models' abilities to use these biases, as both samples in the pair must be answered correctly to achieve a non-zero score. Similar to MVP, some highly adopted examples of such image-language benchmarks build on top of existing image sources (ARO (Yuksekgonul et al. 2022 or fix them explicitly (SugarCREPE (Hsieh et al. 2024)). ). Commonly, the focus is on teztual minimal -aaanal--caageeee pairs, e.g., providing several answer candidates for a question with only slight variations in word order YYuksekgonul gt all, 2022 Cores et al. 2024 Park et al. 2022 Li ct al.) 2023 Cai ct al. 2024). However, even textual
minimal-change pairs can be susceptible to the same language biases (Hsieh el al. 2024 Wu et al.,
2023), which why we focus on visual minimal pairs in MVP. Other works, such as in Video-QA. focuses on visual minimal-change pairs. TempCompass (Liu et al.) minimally different videos by manipulating the original video, e.g., playing the video in reverse, at faster speed, or playing one video above the other. Vinoground (Zhang et al. 2024a) scrapes 0.5K minimally different video pairs from YouTube with the majority following the same pattern: event A before B vs. event B before A. Our work differs in several aspects from these (summarized Table 2), notably as well in terms of the scale of curation by showing that minimal video pairs can be procedurally extradted from existing video sources. While our Minimal Pair Score is inspired by Winoground (Thrush ct al. 2022), unlike Vinoground, we intentionally do not adopt the Winoground metric directly since we want MVP P to be agnostic to whether models can process several videos in one forward pass.
2024 creates small set of less than 0.5K artificial
The language biases in existing vision-language benchmarks often stem from the over-reliance on world knowledge and plausible co-occurrences (Hsieh el al. 2024 Goyal el al. 2017b). Thus, MVP focuses on short videos with "basic" perceptual skills (spatial, temporal, or intuitive physics), which requires understanding of physical world propertics (Yi* et al. 2020 Chen et al.] 2022 Jassim et al. 2024: RRochet et al., 2022; Bear et al., 2021 Margoni et al.) 2024 Bailargeon et al. 1985) reducing the space for blind LLMs to rely their cultural knowledge.
15

---

Published in Transactions on Machine Learning Research (12/2025)
6 Discussion and Limitations
Going back to our initial question, our results suggest suggest that VideoLLMs do not yet perceive and understand the world as reliably as humans. After evaluating varions state-of-the-art VideoLLM models for physical world understanding on MVP, the best model obtains only 40.2% average accuracy, while human performance is 92.9%. Yet, Mss are not completely blind. On some sub-categories of spatio-temporal understanding and intuitive physics, VidcoLLMs can perform significantly better than random chance. Overall, our empir- ical evaluation shows that current VideoL.LMs are still far from matching human performances all tested tasks, calling for more research in this direction to develop better training data for world modelling, as well novel learning criteria and model architectures. We anticipate MVP to help the development of the next generation of visual systems to perceive the world as robustly as humans.
Limitations: No benchmark comes without limitations. First, it is possible that more elaborate prompting strategies for free-form reasoning (CoT) and higher frame rates could improve performance. Additionally, using an automated curation approach will not be able fully remove noisy examples; through manual inspection, we found some of the examples to be too simple, and a few others be ambiguous, although we note that these noisy samples only represent small subset of the overall data.
References
Josh Achiam, Steven Adler, Sandhini Agarwal, Lama Ahmad, Ilge Akkaya, Florencia Leoni Aleman, Diogo Almeida, Janko Altenschmidt, Sam Altman, Shyamal Anadkat, et al. Gpt-4 technical report. arXiv preprint arXi::3303 2023.
Pravesh Agrawal, Szymon Antoniak, Emma Bou Hanna, Baptiste Bout, Devendra Chaplot, Jessica Chud- novsky, Diogo Costa, Baudouin De Monicault, Saurabh Garg, Theophile Gervet, et al. Pixtral 12b. arXiv preprint ar 1n.:2440007755 2024.
Jean-Baptiste Alayrac, Jelf Donahue, Pauline Luc, Antoine Miech, Tain Barr, Yana Hasson, Karel Lene, Arthur Mensch, Katherine Millican, Malcolm Reynolds, et al. Flamingo: a visual language model for fow-shot learning. Advances in neural information processing systems, 35:23716 127366 2022.
Stanislaw Antol, Aishwarya Agrawal, Jiasen Lu, Margaret Mitchell, Dhruv Batra, C Lawrence Zitnick, and Devi Parikh. Vqa: Visual question answering. In Proceedings of the IEEE international conference on computer vision, pp. 2425-2433, 2015.
Jinze Bai, Shuai Bai, Shusheng Yang, Shijie Wang, Sinan Tan, Peng Wang, Junyang Lin, Chang Zhou, and Jingren Zhou. Qwen-vl: A frontier large vision-language model with versatile abilities. arXiv preprint arXiv:2308. 129666, 2023.
Rence Baillargeon, Elizabeth S Spelke, and Stanley Wasserman. Object permanence in five-month-old infants. Cognition, 20(3):191-208, 1985.
Daniel Bear, Elias Wang, Damian Mrowca, Felix Binder, Hsiao- Yu Tung, Pramod RT, Cameron Holdaway, Sirui Tao, Kevin Smith, Fan- Yun Sun, Fei-Fei Li, Nancy Kanwisher, Josh Tenenbaum, Dan Yamins, and Judith Fan. Physion: Evaluating physical prediction from vision in humans and machines. In J. Vanschoren and S. Yeung (cds.), Proceedings of the Neural Information Processing Systems Track Datasets and Benchmarks, volume 1, 2021. URL http:p///ettttstteeaaacasssppppe Filess/aperr/222//111 http://2110000444440000000000000
www.neuuui...e www.ipp..ccpaapeeee... cc/paper pdf..
Florian Bordes, Quentin Garrido, Justine T Kao, Adina Williams, Michael Rabbat, and Emmanuel Dupoux. Intphys 2: Benchmarking intuitive physics understanding in complex synthetic environments. arXiv preprint arXiv:2506.0900 09849, 2025.
Samuel R. Bowman, Gabor Angeli, Christopher Potts, and Christopher D. Manning. A large annotated corpus for learning natural language inference. In Lluis Mårquez, Chris Callison-Burch and Jian Su (eds.), Proceedings of the 2015 Conference on Empirical Methods in Natural Language Processing, pp-
16

---

Published in Transactions on Machine Learning Research (12/2025)
632-642, Lisbon, Portugal, September 2015. Association for Computational Linguistics. doi: 10.18653/ v1/D15-1075. URL. httpps:/aadlanttoooooooo////////
Shyamal Buch, Cristóbal Eyzaguirre, Adrien Gaidon, Jiajun Wu, Li Fei-Fei, and Juan Carlos Niebles. Re- visiting the" video' in video-language understanding. In Proceedings of the IEEE/CVF conference computer vision and pattern recognition, pp. 2917-2927, 2022.
Mu Cai, Reuben Tan, Jianrui Zhang, Bocheng Zou, Kai Zhang, Feng Yao, Fangrui Zhu, Jing Gu, Yiwu Zhong, Yuzhang Shang, et al. Temporalbench: Benchmarking fine-grained temporal understanding for multimodal video models. arXiv preprinl ar XrXiv2240.. 1...2410.10888. 2024.
Nikhil Chandak, Shashwat Goel, Ameya Prabhu, Moritz Hardt, and Jonas Geiping. Answer matching outperforms multiple choice for language model evaluation. arXiv preprint arXiv:2507.02856, 2025.
Lin Chen, Jinsong Li, Xiaoyi Dong, Pan Zhang, Yuhang Zang, Zehui Chen, Haodong Duan, Jiaqi Wang, Yu Qiao, Dahua Lin, and Feng Zhao. Are we on the right way for evaluating large vision-language models? arXiv fes. CVJ, March 2024a. URL htLL:ttp:///// http://arxiv.. htttp://arriv.orr/ab////0000 org/abs/24033 /2002220022 20330
Zhe Chen, Weiyum Wang, Yue Cao, Yangzhou Liu, Zhangwei Gao, Erfei Cui, Jinguo Zhu, Shenglong Ye, Hao Tian; Zhaoyang Liu, et al. Expanding performance boundaries of open-source multimodal models with model, data, and test-time scaling. arXiv preprint arXiv:2412.05877 05271, 2024b.
Zhenfang Chen, Kexin Yi, Yunzhu Li, Mingyu Ding, Antonio Torralba, Joshua Tenenbaum, and Chuang Gan. Comphy: Compositional physical reasoning of objects and events from videos. In International Conference Learning Representations, 2022.
Zixu Cheng, Jian Hu, Ziquan Liu, Chenyang Si, Wei Li, and Shaogang Gong. V-star: Benchmarking video- Ilms video spatio-temporal reasoning. arXiv preprinl arXiv:2503.114 11495, 2025.
Daniel Cores, Michael Dorkenwald, Manuel Mucientes, Cees GM Snoek, and Yuki M Asano. Tybench: Redesigning video-language evaluation. arXiv preprint orXiv:2410.0775. 2024.
Lynch Corey, Wahid Ayzaan, Tompson Jonathan, Ding Tianli, Betker James, Baruch Robert, Arm- strong Travis, and Florenc Pete. Interactive language: Talking to robots real time. orXiv preprint arXiv:2210.06/07 2022.
Ido Dagan, Dan Roth, Fabio Zanzotto, and Mark Sammons. Recognizing teztual entailment: Models and applications. Morgan & Claypool Publishers, 2013.
Matt Deitke, Christopher Clark, Sangho Lee, Rohun Tripathi, Yue Yang, Jae Sung Park, Mohammadreza Salchi, Niklas Muennighoff, Kyle Lo, Luca Soldaini, et al. Molmo and pixmo: Open weights and open data for stale-of-the-art multimodal models. arXiv preprinl ar arxiv:2409. 17146, 2024.
Danny Driess, F. Xia, Mehdi S. M. Sajjadi, Corey Lynch, Aakanksha Chowdhery, Brian Ichter, Ayzaan Wahid, Jonathan Tompson, Quan Ho Vuong, Tianhe Yu, Wenlong Huang, Yevgen Chebotar, Pierre Sermanet, Daniel Duckworth, Sergey Levine, Vincent Vanhoucke, Karol Hausman, Mare Toussaint, Klaus Greff, Andy Zeng, Igor Mordatch, and Peter R. Florence. Palm-c: An embodicd multimodal language model. In International Conference on Machine Learning, 2023. URL https://api. sewanticscoool..r.. Cot.oopuuiI11533453...300
Yilun Du, Mengjiao Yang, Pete Florence, Fei Xia, Ayzaan Wahid, Brian Ichter, Pierre Sermanet, Tianhe Yu, Picter Abbeel, Joshua B. Tenenbaum, Leslie Kaelbling, Andy Zeng, and Jonathan Tompson. Video language planning, 2023. URL http:///ax...ooogaaa///000... 10625
Abhimanyu Dubey, Abhinav Jauhri, Abhinav Pandey, Abhishek Kadian, Ahmad Al-Dahle, Aiesha Letman, Akhil Mathur, Alan Schelten, Amy Yang, Angela Fan, et al. The llama 3 herd of models. arXiv preprinl arXiv:2407.22 210072 2024.
J. Gao, Chen Sun, Zhenheng Yang, and Ramakant Nevatia. Tall: Temporal activity localization via language query. In ICCV, 2017.
17

---

Published in Transactions on Machine Learning Research (12/2025)
Matt Gardner, Yoav Artzi, Victoria Basmov, Jonathan Berant, Ben Bogin, Sihao Chen, Pradeep Dasigi. Dheeru Dua, Yanai Flazar, Ananth Gottumukkala, Nitish Gupta, Hannaneh Hajishirzi, Gabriel IIharco, Daniel Khashabi, Kevin Lin, Jiangming Liu, Nelson F. Liu, Phoebe Mulcaire, Qiang Ning, Sameer Singh, Noah A. Smith, Sanjay Subramanian, Reut Tsarfaty, Eric Wallace, Ally Zhang, and Ben Zhou. Eval- uating models' local decision boundaries via contrast sets. Trevor Cohn, Yulan He, and Yang Liu (ods.); Findings of the Association for Compulational Linguisties: EMNLP 2020, pp. 1307-1323, Online, November 2020. Association for Computational Linguistics. doi: 10.18653/v1/20222 (0065/////2200.indnin----ee findings--mnlp.117. URL attps://aclanthologg) org/2020 findings-n--iii emnlp. .177
Quentin Garrido, Nicolas Ballas, Mahmoud Assran, Adrien Bardes, Laurent Najman, Michael Rabbat, Emmanuel Dupoux, and Yann LeCun. Intuitive physics understanding emerges from self-supervieed pre- training on natural videos. arXiv preprint wrXiv22502.1882 2025.
Robert Geirhos, Jörn-Henrik Jacobsen, Clandio Michaelis, Richard Zemel, Wieland Brendel, Matthias Bethge, and Felix A Wichmann. Shortcut lcarning in deep neural networks. Nature Machine Intelli- gence, 2(11):665 673, 2020.
Raghav Goyal, Samira Ebrahimi Kahou, Vineent Michalski, Joanna Materzynska, Susanne Westphal, Heuna Kim, Valentin Haenel, Ingo Fruend, Peter Yianilos, Moritz Mueller-Freitage Florian Hoppe, Christian Thurau, Ingo Bax, and Roland Memisevic. The 'something something' video database for learning and evaluating visual common sense. In Proceedings of the IEEE International Conference on Computer Vision (ICCV), Oct 2017a.
Yash Goyal, Tejas Khot, Douglas Summers-Ste Stay, Dhruv Batra, and Devi Parikh. Making the in vqa matter: Flevating the role of image understanding in visual question answering. In Proceedings of the IEEE conference on computer vision and pattern recognition, pp. 6904 6913, 2017b.
Xuchai He, Weixi Feng, Kaizhi Zheng, Yujie Lu, Wanrong Zhu, Jiachen Li, Yue Fan, Jianfeng Wang, Linjie Li, Zhengyuan Yang, et al. Mmworld: Towards multi-discipline multi-faceted world model evaluation in videos. arxiv preprinl ar Xrx::22066.0 08407, 2024.
Matthew Honnibal and Ines Montani. spaCy 2: Natural language understanding with Bloom embeddings, convolutional neural networks and incremental parsing. To appear, 2017.
Cheng-Yu Yu Hsieh, Jieyu Zhang, Zixian Ma, Aniruddha Kembhavi, and Ranjay Krishna. Sugarcrepe: Fixing hackable benchmarks for vision-language compositionality. Advances in newral information processing systems, 36, 2024.
Sorwan Jassim, Mario Holubar, Annika Richter, Cornclius Wolff, Xenia Ohmer, and Elia Bruni. Grasp: A novel benchmark for evaluating language grounding and situated physics understanding in multimodal language models. In Kate Larson (ed.), Proceedings of the Thirty- Third International Joint Conference on Artificial Intelligence, IJCAI-24, pp. 6297-6305. International Joint Conferences on Artificial Intelligence URL
Organization, 8 2024. doi: /11/9/66///cai22224/43 Main Track.
http///.............////////aaa. 2024//996
Yunfan Jiang, Agrim Gupta, Zichen Zhang, Guanzhi Wang, Yongqiang Dou, Yanjun Chen, Li Fei-Fei, Anima Anandkumar, Yuke Zhu, and Linxi Fan. Vima: General robot manipulation with multinodal prompts. In Fortieth International Conference Machine Learning, 2023.
Justin Johnson, Bharath Hariharan, Laurens Van Der Maaten, Li Fei-Fei, C Lawrence Zitnick, and Ross Girshick. Clevr: A diagnostic datasct for compositional language and clementary visual reasoning. In Proceedings of the IEEE conference on computer vision and pattern recognition, pp. 2901-2910, 2017.
Amita Kamath, Jack Hessel, and Kai-Wei Chang. What's "up" with vision-language models? investigating their struggle with spatial reasoning. In Houda Bouamor, Juan Pino, and Kalika Bali (eds.). ), Proceedings of the 2023 Conference on Empérical Methods in Natural Language Processing, pp. 9161-9175, Singapore, December 2023. Association for Computational Linguistics. doi: 10.18653/v hwt////1/222/eee/ m6in.568. URL http://aacaantooooy..o/////////a emw.::maa.mmmmmm....m..mm.m 568
18

---

Published in Transactions on Machine Learning Research (12/2025)
Jacob Krantz, Erik Wijmans, Arjum Majumdar, Dhruv Batra, and Stefan Lec. Beyond the nav-graph: Vision-and-lanuuage navigation in continuous environments. In ECCV, 2020.
Benno Krojer, Vaibhav Adlakha, Vibhav Vineet, Yash Goyal, Edoardo Ponti, and Siva Reddy. Image retrieval from contextual descriptions. In Smaranda Muresan, Preslav Nakov, and Aline Villavicencio (eds.), Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pp. 3426-3440, Dublin, Ireland, May 2022. Association for Computational Linguisties. orgg2022.acll.
doi: 10.18653/v1 a2022.ccl-ooog 2411 URL htttps://cclantholl
241
Hugo Laurengon, Andrés Marafioti, Victor Sanh, and Léo Tronchon. Building and better understanding vision-language models: insights and future directions. arXiv preprint or Xix:24008. 1 12637, 2024.
Jie Lei, Licheng Yu, Mohit Bansal, and Tamara Berg. TVQA: Localized, compositional video question answering. In Ellen Riloff, David Chiang, Julia Hockenmaier, and Jun'icht Tsujii (cds.), Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing, pp. 1369 1379, Brussels, Belgium, October-November 2018. Association for Computational Linguistics. doi: 10.18653/vv /D18/ 1167. URL https://aclanthooogyy org/ 1167
Jie Lei, Tamara Berg, and Mohit Bansal. Revealing single frame bias for video-and-languase learning. In Anna Rogers, Jordan Boyd-Graber, and Naoaki Okazaki (eds.), Proceedings of the 61 Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pp. 487 507, Toronto, Canada, July 2023. Association for Computational Linguisties. doi: (0.18653/v1 /22223..ccl-oong. URL https:
www.aantthoooy..rrg///// ac1--02....1oooooo acl-long.22. 29.
Hector Levesque, Ernest Davis, and Leora Morgenstern. The winograd schema challenge. In Thirteenth international conference on the principles of knowledge representation and reasoning, 2012.
Bo Li, Yuanhan Zhang, Dong Guo, Renrui Zhang, Feng Li, Hao Zhang, Kaichen Zhang, Yanwei Li, Ziwei Liu, and Chunyuan Li. Llava-onevision: Easy visual task transfer. arXiv preprint Xrx::22408.00 08326, 2024a.
Kunchang Li, Yali Wang, Yinan He, Yizhuo Li, Yi Wang, Yi Liu, Zun Wang, Jilan Xu, Guo Chen, Ping Luo, et al. Mvbench: A comprchensive multi-modal vidco understanding benchmark. In Proceedings of the IEEE/CVE Conference on Computer Vision and Pallern Recognilion, pp. 22195 22206, 2024b.
Shicheng Li, Lei Li, Shuhuai Ren, Yuanxin Liu, Yi Liu, Rundong Gao, Xu Sun, and Lu Hou. Vitatees: A diagnostic dataset for temporal concept understanding of video-language models. ar preprint arXiv:2311.17404, 2023.
Yunxin Li, Xinyu Chen, Baotian Hu, Longyue Wang, Haoyuan Shi, and Min Zhang. Videovista: A versatile benchmark for video understanding and reasoning. ArXiv, abs/2406.11303, 2024c. URL https://api.
semanticscholar. wtwpaarooroooroo.00000055555555.
Zhiqiu Lin, Xinyue Chen, Deepak Pathak, Pengchuan Zhang, and Deva Ramanan. Revisiting the role of language priors in vision-language models. arXiv preprint XrX:::2306.6 01879, 2023.
Jun Liu, Amir Shahroudy, Mauricio Perez, Gang Wang, Ling-Yo Yu Duan, and Alex C Kot. Ntu rgb+d 120: A large-scale benchmark for 3d human activity understanding. TPAMI, 2020.
Yuanxin Liu, Shicheng Li, Yi Liu, Yuxiang Wang, Shuhuai Ren, Lei Li, Sishuo Chen, Xu Sun, and Lu Hou. TempCompass: Do video LLMs really understand videos? In Lun Wei Ku, Andre Martins, and Vivek Srikumar (ods.), Findings of the Association for Computational Linguistics: ACL 2024, pp. 8731-8772, Bangkok, Thailand, August 2024. Association for Computational Linguisties. doi: 10.16653/v1/2024. findings-acl.55 a11. URL
wwwpioooooooorg/22222 org/2024. (1..02..finninng findings-acll acl.
Do Xuan Long, Ngoc-Hai Nguyen, Tiviatis Sim, Hieu Dao, Shafiq Joly, Kenji Kawaguchi, Nancy F. Chen, and Min- Yen Kan. LLMs are biased towards output formats! systematically evaluating and mitigating output format bias of LLMs. In Luis Chiruzzo, Alan Ritter, and Lu Wang (cds.), Proceedings of the 2025 Conference of the Nations of the Americas Chapter of the Association for Computational Linguisties: Hu- man Language Technologies (Vobume 1: Long Papers), pp. 299 -30,,,,, Albuquerque, New Mexico, April 2025.
19

---

Published in Transactions on Machine Learning Research (12/2025)
Association for Computational Lingussss.. ISBN 979-8-89176-88888 doi: 10.18653/v1/2025.mm naacl-lannnl URL httppe//acclattblooy...o.////22 www.:-accllaaaa long--onnn-11 157
Tegan Maharaj, Nicolas Ballas, Anna Rohrbach, Aaron Courville, and Christopher Pal. A dataset and explo- ration of models for understanding video data through fill-in-the-blan question-answering. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp. 6884-6893, 2017.
Arjun Majumdar, Anurag Ajay, Xiaohan Zhang, Pranav Pulta, Sriram Yenamandra, Mikael Henaff, Sneha Silwal, Paul Movay, Oleksandr Maksymets, Sergio Arnaud, et al. Openeqa: Embodied question answering in the era foundation models. In Proceedings of the IEEE/CVF Conference on Computer Vision and 2024.
Pattern Recognition, pp.
Karttikeya Mangalam, Raiymbek Akshulakov, and Jitendra Malik. Egoschema: diagnostic benchmark for very long-form video language understanding. Advonces in Neural Information Processing Systems, 36, 2024.
Francesco Margoni, Luca Surian, and Renée Baillargeon. The violation-d-exxeecaatiin paradigm: A con- ceptual overview. Psychological Review, 181(3):716-745, April 2024. ISSN 1939-1471, 0033-295X. doi: 10.1037 1110377rev000045000 URL https://doi. apa. wtwp///////ooooo//////oo/goo.. 1037/rev0000450
Kenneth Marino, Mohammad Rastegari, Ali Farhadi, and Roozbeh Mottaghi. Ok-vqa: A visual question an- swering benchmark requiring external knowledge. In Proceedings of the IEEE/cof conference on computer vision and pattern recognition, pp.
2019.
Tom McCoy, Ellie Pavlick, and Tal Linzen. Right for the wrong reasons: Diagnosing syntactic heuristics in natural language inference. In Anna Korhonen, David Traum, and Lluís Mårquez (eds.), Proceedings of the 57th Annual Meeling of the Associalion for Computational Linguisties, pp. 3428 3448, Florence, Italy, July 2019. Association for Computational Linguistics. doi: 10.18653/v1/P19 1334. V/aclanthology.orr//PP wwt.:oo-g/////....
URL https:
Mathew Monfort, Bolei Zhou, Sarah Adel Bargal, Alex Andonian, Tom Yan, Kandan Ramakrishnan, Lisa M. Brown, Quanfu Fan, Dan Gutfreund, Carl Vondrick, and Aude Oliva. Moments in time dataset: One million videos for event understanding. TPAMI, 2020.
Hans Moravec. Mind children: The future of robot and human intelligence. Harvard UP, 1988.
Jae Sung Park, Sheng Shen, Ali Farhadi, Trevor Darrell, Yejin Choi, and Anna Rohrbach. Exposing the limits of video-text models through contrast sets. In Proceedings of the 2022 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, pp. 3574-3586, 2022.
Viorica Patrauccan, Lucas Smaira, Ankush Gupta, Adria Recasens Continente, Larisa Markeeva, Dylan Sunil Banarse, Skanda Koppula, Joseph Heyward, Mateusz Malinowski, Yi Yang, Carl Doersch, Tatiana Mate- jovicova, Yury Sulsky, Antoine Miech, Alexandre Fréchette, Hanna Klimczak, Raphael Koster, Junlin Zhang, Stephanie Winkler, Yusuf Aytar, Simon Osindero, Dima Damen, Andrew Zisserman, and Joao Carreira. Perception test: A diagnostic benchmark for multimodal video models. In Thirty-seventh Conference on Neural Information Processing Systems Dalasels and Benchmarks Track, 2023. URL https://openreviee http://pppeeeeeeeeeeeeettttuuuee waw.rruunnnd--EEEXPPPPP
Pooyan Rahmanzadehgervi, Logan Bolton, Mohammad Reza Taesiri, and Anh Totti Nguyen. Vision language models are blind. arXiv preprint arXiv:8407. 06581, 2024.
Ruchit Rawal, Khalid Saifullah, Ronen Basri, David Jacobs, Gowthami Somepalli, and Tom Goldstein. Cinepile: A long video question answering dataset and benchmark. ArXio, 0as//2405.0811 2024. URL
https://app wwwppppeamaaticcccooooooooogoooo wttpolopaa..rr/oopsusstllllt 269761335
Ronan Riochet, Mario Ynocente Castro, Mathieu Bernard, Adam Lerer, Rob Fergus, Véronique Izard, and Emmanuel Dupoux. Intphys 2019: A benchmark for visual intuitive physics understanding. IEFE Trans- actions on Pattern Analysis and Machine Intelligence, 44(9):5016-500 50255, 2022. doi: N0.1109/TPAM..222..
3083839.
20

---

Published in Transactions on Machine Learning Research (12/2025)
Keisuke Sakaguchi, Ronan Le Bras, Chandra Bhagavatula, and Yejin Choi. Winogrande: An adversarial winograd schema challenge at scale. Commaniications of the ACM, 64(9):99-106, 2021.
Xiaoqian Shen, Yunyang Xiong, Changsheng Zhao, Lemeng Wu, Jun Chen, Chenchen Zhu, Zechun Liu, Fanyi Xiao, Balakrishnan Varadarajan, Florian Bordes, et al. Longvu: Spatiotemporal adaptive compression for long video-language understanding. arXiv preprint arXic:2410.1 17434, 2024.
Yansong Tang, Dajum Ding, Yongming Rao, Yu Zheng, Danyang Zhang, Lili Zhao, Jiwen Lu, and Jie Zhou. Coin: A large-scale dataset for comprehensive instructional video analysis. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), June 2019.
Makarand Tapaswi, Yukun Zhu, Rainer Stiefelhagen, Antonio Torralba, Raquel Urtasun, and Sanja. Fi- dler. Movieqa: Understanding stories in movies through crestion-answering. In Proceedings of the IEEE conference on compuler vision and pallern recognilion, pp. 4631-4640, 4640, 2016.
Gemini Team, Petko Georgiev, Ving Tan Lei, Ryan Burnell, Libin Bai, Anmol Gulati, Garrett Tanzer, Damien Vincent, Zhufeng Pan, Shibo Wang, et al. Gemini 1.5: Unlocking multimodal understanding across millions of tokens of context. ar preprint arXie:2403.0555 2024.
Tristan Thrush, Ryan Jiang, Max Bartolo, Amanprect Singh, Adina Williams, Douwe Kicla, and Candace Ross. Winoground: Probing vision and language models for visio-linguistic compositionality. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 5238-5248, 2022.
Shengbang Tong, Zhuang Liu, Yuexiang Zhai, Yi Ma, Yann LeCun, and Saining Xie. Eyes wide shut? exploring the visual shortcomings of multimodal llms. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 9568 9578, 2024.
Jiawei Wang, Liping Yuan, and Yuchen Zhang. Tarsier: Recipes for training and evaluating large video description models. arXiv preprint arXi:2407.00633, 2024a.
Peng Wang, Shuai Bai, Sinan Tan, Shijie Wang, Zhihao Fan, Jinze Bai, Keqin Chen, Xuejing Liu, Jialin Wang, Wenbin Ge, et al. Qwen2-vl: Enhancing vision-language model's perception of the world at any resolution. arXiv preprint orXiv:2409.12100 2024b.
Tan Wang, Kevin Lin, Linjie Li, Chung-Ching Lin, Zhengyuan Yang, Hanwang Zhang, Zicheng Liu, and Li- juan Wang. Equivariant similarity for vision-language foundation models. In Proceedings of the IEFE/CW /CVF International Conference on Computer Vision, pp. 11998-12008, 2023a.
Yi Wang, Yinan He, Yizhuo Li, Kunchang Li, Jiashuo Yu, Xin Ma, Xinhao Li, Guo Chen, Xinyuan Chen, Yaohui Wang, et al. Internvid: A large-scale video-text dataset for multimodal understanding and gener- ation. arXiv preprint 230702307700942, 2023b.
Yubo Wang, Xueguang Ma, Ge Zhang, Yuansheng Ni, Abhranil Chandra, Shiguang Guo, Weiming Ren, Aaran Arulraj, Xuan He, Ziyan Jiang, et al. Mmlu-pro: A more robust and challenging multi-task language understanding benchmark. Advances in Neural Information Processing Systems, 37:95266 95290, 2024c.
Zheuhailong Wang, Ansel Blume, Sha Li, Genglin Liu, Jaemin Cho, Zineng Tang, Mohit Bansal, and Heng Ji. Paxion: Patching action knowledge in video-language foundation models. Advances in Neural Information Processing Systems, 36:20729-20744 20749, 2023c.
Luca Weihs, Amanda Yuile, Renée Baillargcon, Cynthia Fisher, Gary Marcus, Roozbeh Mottaghi, and Aniruddha Kembhavi. Benchnarking progress to infant-level physical reasoning in AI. Transactions on Machine Learning Research, 2022. ISSN 2835-8856. URL https://openrevierr wwwparrvviiiinneeffuuiuee wwt:/ferum?ii
9N1qD9148M
Bo Wu,, Shoubin Yu, Zhenfang Chen, Joshua B Tenenbaum, and Chuang Gan. Star: A benchmark for situated reasoning in real-world videos. In Thirty- Conference on Neural Information Processing Systems (NeurIPS), 2021.
21

---

Published in Transactions on Machine Learning Research (12/2025)
Chenwei Wu, Li Erran Li, Stefano Ermon, Patrick Haffner, Rong Ge, and Zaiwei Zhang. The role of linguistic priors in measuring compositional generalization of vision-language models. In Proceedings on, pp. 118 126. PMLR, 2023.
Junbin Xiao, Xindi Shang, Angela Yao, and Tat-Seng Chua. Next-qa: Next phase of question-answering to explaining temporal actions. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognilion, pp.
2021.
Binzhu Xie, Sicheng Zhang, Zitang Zhou, Bo Li, Yuanhan Zhang, Jack Hessel, Jingkang Yang, and Ziwei Liu. Funqa: Towards surprising video comprehension. In European Conference on Compuler Vision, pp- 39-57. Springer, 2025.
Yuan Yao, Tianyu Yu, Ao Zhang, Chongyi Wang, Junbo Cui, Hongji Zhu, Tianchi Cai, Haoyu Li, Weilin Zhao, Zhihui Hc, et al. Minicpm-v: gpt-4v level milm on your phone. arXiv preprint orXiv:2408.01 0100 2024.
Kexin Yia, Chuang Gan Yunzhu Li, Pushmeet Kohli, Jiajun Wu, Antonio Torralba, and Joshua B. Tenen- baum. Clevrer: Collision events for video representation and reasoning. International Conference on Learning Representations, 2020. URL
Mert Yuksekgonul, Federico Bianchi, Pratyusha Kalluri, Dan Jurafsky, and James Zou. When and why vision- language models behave like bags-of-words, and what to do about it? arXiv preprint arXiv:2210.01 01909 2022.
Andy Zeng, Maria Attarian, brian ichter, Krzysztof Marcin Choromanski, Adrian Wong, Stefan Welker, Federico Tombari, Aveek Purohit, Michael S Ryoo, Vikas Sindhwani, Johnny Lee, Vincent Vanhoucke, and Pete Florence. Socratic models: Composing zero-shot multimodal reasoning with language. In The Eleventh International Conference on Learning Representations, 2023. URL httpps://openreviev..oo
forum//d==62222Maavooo
Ce Zhang, Taixi Lu, Md Mohaiminul Islam, Ziyang Wang, Shoubin Yu, Mohit Bansal, and Gedas Bertasius. arXiv preprint arXiv:2312. 172355311.17335,
A simple Ilm framework for long-range video
17235, 2023a.
Hongjie Zhang, Yi Liu, Lu Dong, Yifei Huang, Zhen-Hua Ling, Yali Wang, Limin Wang, and Yu Qiao. Movqa: A benchmark of versatile question-answerin for forg-form movie understanding. ArXiv, abs/2312.048111 2023b.
Hongjie Zhang, Yi Liu, Lu Dong, Yifei Huang, Zhen-Hua Ling, Yali Wang, Limin Wang, and Yu Qiao. Movqa: A benchmark of versatile question-answering for long-form movie understanding. arXiv preprint arXiv:2312.044 04817, 2023c.
Jianrui Zhang, Mu Cai, and Yong Jae Lee. Vinoground: Scrutinizing Immn over dense temporal reasoning with short videos. ai preprint 24 10. 0..02663 2024a.
Pan Zhang, Xiaoyi Dong, Yuhang Zang, Yuhang Cao, Rui Qian, Lin Chen, Qipeng Guo, Haodong Duan, Bin Wang, Linke Ouyang, et al. mnterllmxccomppsr-22555 versatile large vision language model supporting long-contextual input and output. Xixie preprint urXiv:2407.0333 2024b.
22

---

Published in Transactions on Machine Learning Research (12/2025)
Outline
Section A
shows the license for each source in MVP and discusses legal aspects of the release.
Section B explains why and how created smaller version of MVP (MVP-mini).
Section goes into all the nitty-gritty details of how we curated MVP.
Section contains implementation details and further explorations for the shortcut analysis Section 2 that motivated building MVP.
Section B focuses the taxonomics from all the seed sources in MVP. We show more fine-grained results than our main results in Table by showing performance categorized by these taxonomies (e.g. reasoning types in Perception Test).
Section shows samples to provide readers with a realistic distribution of question and videos in MVP.
Section explains how we arrived at the human baseline accuracy on MVP.
Section provides details on how exactly we run inference, e.g. the exact prompt VideoLLMs (or single- image VLMs for the shortcut baselines) and how extract answers.
Section T (Behind the Scenes) shows not just the final product (this paper) but also how we arrived here, what we discarded, and some personal reflections.
A Licensing and legal aspects
We only source videos for MVP with a permissible license. For legal reasons, we cannot directly release the videos that ended up in MVP after all curation steps. Instead are providing easy to run download script that goes each source and downloads only the relevant videos. Similarly, when showing examples in the paper we ensure no faces of people are visible or other privacy concerns.
Dataset
Source
License
Perception Test
github. htt.th/.ww/ooggglldddeaaaa///rpr test
Apache 2.0
CLEVRER
MIT CCO
STAR
Dobbysu.com/57.0
Apache 2.0
InfLevel
github. g/thub.com//lllena//infff
Apache 2.0
IntPhys
intphya..ognit.ivemm.fff
CC BY 4.0
Vinoground
vinoground.github.it
A Apache 2.0
GRASP
gi
MIT
CLEVRER
Llevrer.csail.mit.cc
CCO
Language-Table
github.cmm Bilub..com/googlee sthpa..co//ogggerreeerrrr/aaaaaa
A Apache-2.0
Something- Something V2
Something-Somethinng
Qualcomm License Agreement PDF
Custom (Research Use)
Table 10: Datasets with sources and licenses.
B MVP-mini
Next to the full MVP, we also release MVP-mini downsampled in subsct-aware manner to 18,290 vidco-QA examples (thus 9,145 pairs). MVP-mini will be faster to use, while MVP-full allows researcher to filter and curate derivatives at a large scale. For the most part our process simply involves selecting a random subset, except that we place additional conditions such that no dataset (or subset of dataset) is underrepresented due to subsampling (i.e. never going below 500 examples). For example, IntPhys is already quite small with 360 videos don't ton't remove any examples in this case.
23

---

Published in Transactions on Machine Learning Research (12/2025)
C
Details: The Curation of MVP
As illustrated in Section
follow three main steps curate MVP: 1) Manual categorical filtering 2) Aulomatic pairing of minimally different examples 3) Aulomatic filtering of single-franee-ollbble examples. We also need to QAify 5 out of the 9 datasets.
While there are commonalities across datascts for how we implement Step 1 (categorical filtering) and Step 2 (pairing), there also some differences that. we describe here for reproducibility, Note that Step 3 is exactly the same across datasets, and is described in sufficient detail in Section 3 Hence, we focus on the first and second step.
C.1 Manual categorical filtering
For out of datasets we select subsets and categories of questions suitable for MVP.
We manually annotate the 132 question types in Perception Test. Specifically we filter out question that cither do not require temporal understanding ("Where the person?") or are ill-defined. Around 20% of question types are discarded.
Perception Test:
Language Table: We select the human-captioned and human-controlled split of Language Table which constitules 440K. Thus we exclude other splits where the robot arm is aulomatically controlled and/or the robotic actions are synthetically captioned and not by a human. Additionally, we exclude any videos where the caption only mentions a single object such as "Move the arm to the left" to ensure complex enough interactions.
We find an issue in the counterfactual split of CLEVRER and exclude these examples based the meta-data associated with each video, c.g. object attributes and the exact position of cach object at each frame. The issue is that most of the time, the object mentioned (laryet object) in the question ("What happens if the cube is not there?") is not actually involved in any collisions. As a result, the correct answer (e.g. "The red and yellow cube collide") is often depicted, whereas in a proper counterfactual example the correct answer should never be depicted but only happen in an "alternative world". Thus we filter out an example if the target object is never near any other moving objects (i.e. a collision) based on their coordinates. Moreoever, we filter out an example if the two objects mentioned in the correct answer are in fact already colliding in the video, based on their coordinates.
CLEVRER:
InfLevel:
We only use the split with real-life videos where humans conduct the experiments in front of a camera, similar to experimental designs in psychology (Weihs et al. 2022).
STAR: We exclude the Feasibility and Interaction splits since they are often ill-defined, lead to strong language biases or are too easy.
C.2 Automatic Pairing of Minimally Different Examples
We apply this step to only 5 out of the 9 datasets, since Vinoground (Zhang et al. 2024a) and the 3 intuitive physics datasets are already structured inlo mininal-change video pairs. Our pairing boils down to finding highly similar pairs, which we base on symbolic or visual similarity, and at the same time ensuring that both answers cannot be true at the same time, i.c. mutually exclusive. Espocially the latter task, also known as entailment detection or natural language inference, has many nuanced edge cases.
This step conducted on QA examples æ consisting of a question, a video, and answer candidates: (g. u, [a1, ....
Perception Test/STAR.
1. Group QA examples into sets with the same question:
P (X Vor, E X,9; b.
24

---

Published in Transactions on Machine Learning Research (12/2025)
2. For given X, examples z; and zj grouped into potential pairs if they have opposite (mutually exclusive) correct answers:
p' [(11 23) as + az}.
3. From this set of potential pairs P', we choose the top-k for a given question based on visual similarity. measured via cosine similarity of embeddings from the video encoder ViCLIP-ViT-L (Wang et al. 2023b
Pk TT)............................. sim(U)... V2m)) ) 2 sim(U)m-r )-
In practice, we choose 50 for each question.
Additionally, we use dapaset- rules after manual inspection, e-g., for the Perception Test, require that for two potential pairs z; and Iy, neither correct answer a, aj is "Both the other options'.
Language Table. Note that all examples in Language Table have the same question "Which robot in- struction best describes the actions in the video?".
We group QA examples into sets such that a) both correct answers mention the same objects (e.g., both involve a "red triangle" and "green heart") and b) the set of tokens in a and aj have a large enough overlap:
P {aa az) obj(a,) obj(a;) < loken diff(a;; az) < <4).
Due to the finite number attributes and objects in Language Table, obj(.) checks for these attribute and object key-words.
Examples that would be matched:
(a) move green star to the center left move green star the center Reason: i) all objects overlap, ii) set of words overlap except for one word ("left" only in first sentence)
(b) move green star to the top right of the red circle +> move the red circle to the top of green star Reason: all objects overlap (just in different order), ii) set of words overlap except for one token ("right" only in first sentence)
2. We narrow this set of potential pairs P with a visual similarity threshold, measured via cosine similarity of embeddings from the video encoder ViCLIP-ViT-L (Wang et al. 2023b):
P' = (()1,,)) sim(v, u3) 0.9}.
3. Finally, we ensure that answers are mutually exclusive, i.c., f aj and 0; + a, In practice, this involves several hand-crafted rules after inspecting failure cases: If the order of objects mentioned different, there is no entailment (e.g., "Move yellow triangle to blue heart" and "Move blue heart to yellow triangle"); otherwise one answer contains a general direction such as "towards", "to" or "into" but the other answer contains specific direction such as "left" or "above", there is entailment (we discard the example).
To illustrate, we would discard the following:
move the towards Y move X to the left of Y move the closer to the
move the X to the Y
move the X to left of Y move the X to the top left of Y
25

---

Published in Transactions on Machine Learning Research (12/2025)
Something Something v2. Note that Something Something v2 is video caption dataset where each caption contains either one two objects and a simple action, with in total 174 types of such actions. We QAify these examples with the question "Which action is being performed in the videog, and use the caption with comething-placcholdee
instead of objects as the answer a, c.g., "dropping something".
1. We group QA examples a, and aj into pairs such that the action in a, is well-defined antonym of the action in Oz:
P = { Q,) antonym(a,,a)) az)}.
In practice we identify a subset of 82 action types (47% of all actions) that have well-defined opposite, e.g., "spinning something so it contimues spinning" and "spinning something that quickly stops spinning".
2. We narrow down pairs further by selecting a pair aq and zj if the videos contain the samne object (s) based on their captions:
P' ((;;, obj(u,) = obj(u,)}-
If no pairs fulfill this strict criterion, relax it such that only object must overlap:
P' p²= ((r, obj(e;) nobi(v;) + 00.
From this set of potential pairs P', we choose the top-k based on visual similarity, measured via cosine similarity of embeddings from the video encoder ViCLIP-ViT-L (Wang al. 2023b):
Ps {Ti Tz)...) (Ti))
sim Ujm) 2 sim(vim-ur 1 Ujm+1)}
In practice, we choose l 4000.
Note that CLEVRER has detailed meta-data with a list of all objects throughout the video and their attributes (color, shape, material), with many videos featuring five or more objects.
CLEVRER.
Group QA examples into sets with the same question:
P [X Vr,, X. = 9.}.
2. For a given X, examples and zy are grouped into potential pairs if they have opposite (mutually exclusive) correct answers:
p'= ((x,,,,,)) as * a,
In the special case that the answers are both numerical, require them both to be 1 apart, e.g., "How many objects are moving when the vidco ends? A) 2 B)3". 3".
3. We further filter the set P' by requiring a large overlap of objects with the exact same attributes in both videos. Specifically, we keep pair if the set of objects in o; is "fuzzy subset" of the objects in Uj vice versa:
(obbe__(bbtt(obo(:)) obi(t,)) p" = p" ((x,, 2z)
fuzzy (ob___sssst(bb((t.....oo obj(s))
Here, muzzy_subscte allows one mismatch between the sets of objects and their attributes. To illustrate with a simple example (most actual videos contain more objects):
Videol contains Objl (red, triangle, metal), Obj2 (blue, triangle, metal)
and Video2 Obj1 (red, triangle, metal), Obj2 (blue, triangle, rubber), Obj3(blue, cube metal)
While Videol not an exact subset of Video2, it is almost subset if we ignore that Obj2 does not match with any of the objects (which we call fuzzy subset(.,)).
26

---

Published in Transactions on Machine Learning Research (12/2025)
D Details: Shortcut Analysis
We provide additional experiments for the shortcut analysis on MVBench datasets from Section
2
Language only shortcuts. Similar to (Li ct al. 2024b) also tested VidcoLLM as language-only baseline by blacking out the video (replacing it with zeros). With VideoChai2 this gave slightly worse results with both LLM-versions: 33.0% (Mistral) and 34.6 (Vicuna).
Video only shortcuts. For the video only we remove the question and only provide the answer candidates to the model. In detail, we tested several ways of removing the question (empty string, replace with "what?" explain to answer without question, etc) and found that replacing the question with "REDACTED]" yiclded the best performance.
Single-frame only shortcuts. First we study how performance varies when selecting frames at different first, middle, last, random and finally key-frame. We choose a key-frame based on the highest CLIP similarity among all frames and all answer candidates (with the question prefixed to the candidate). We find that the performance differs only by 1-2% among these selection strategies except for the first frame which performed more than 5% worse than the rest with Idefics3. Since middle has the highest MVBench accuracy, excluding key-framie, we show middle frame results in the paper. In the main paper we show results for Idefies3 and Qwen2-VL, models that mostly focus on non-video tasks. We also test VideoChat2 variants but found performance to be worse, with either showing a single frame once copying it 16-times as "video".
positions:
Simple Socratic LLM shortcuts. In this shorteut we test how well models can still perform when the video is replaced by a much lower bandwidth representation and presented to a text-only LLM: short generic caption of the video. We generate these captions with 2024b). We investigate how model performance differs when increasing the bandwidth of this caption from short, medium to long caption: "[...] Briefly describe this video in one sentence.", ; "[...] Describe this video in 1-2 sentences." and "[...] Describe this video in as much detail and length as possible.". We also ask whether focus on objects or actions helps, i.e. by prompting the captioning model to list the objects or actions in the video. While the long caption variant achieves the highest performance when provided to LLaMA3 8B and 70B, followed by action caption, we choose to show the medium caption variant (i.e. asking the model to caption the video in 1-2 sentences) in the main paper since this is most in the spirit of short (1-2 sentences) and generic (not asking for anything specific) caption a simple bascline.
(Zhang et al,
Additional robustness experiments beyond main paper. As a sanity check, we also study how well perplexity baseline and answer frequencies perform on MVBench. For the perplexity baseline we compute the perplexity of each answer candidate sequence based on LLM (LLaMA3-7B), i.e. how plausible this string is by itself. For example, are common objects or scenarios more often the correct answer? This would be reflected in such a baseline. However we find that this baseline performs around random overall. Next, we also compute statistics to determine some subtasks of MVBench have a skewed distribution of answer Trequencies, i.e. whether option A is more often correct than the other option B, C, etc. or if it is more often "yes" than "no", etc. Here we also also find very little evidence of any issues in terms of frequencies.
E Details: Accuracies on sub-splits of each source in MVP
MVP is sourced from 9 seed datasets and 6 come of them with their own fine-grained laxonomy. In this section we provide a further breakdown of performance across these taxonomies.
For the other throe sources (SSv2, Language Table, Vinoground), we provide performance in a single table:
TODO
27

---

Published in Transactions on Machine Learning Research (12/2025)
Model
Something Something Language Table Vinoground
LLAVA-OV (Qwen2-7B)
26.8
5.2
36.5
VideoChat2 (Mistral-7B)
39.1
21.4
16.9
Mini-CPM-v 2.6
18.1
18.0
20.9
Qwen2- VL (7B)
32.6
27.6
16.5
LongVU (LLaMA3.2)
19.4
14.8
12.2
LongVU (Qwen2-7B)
38.1
21.5
25.6
Tarsier-7B
45.4
18.7
20.2
Tarsier-34B
65.4
36.3
34.2
InternVT.25 8B
52.8
40.2
39.5
Table 11: Accuracy on three sources of MVP that do not have further fine-grained splits.
Perception Test. The Perception Test dataset groups examples along two axes: Skill Area and Type of Reasoning.
Skill Areas: Memory (e.g. change detection seeing table early in video and at the end), Abstraction (e.g. counting), Physics (e.g. collisions), Semantics (e.g. distractor actions).
Types of Reasoning: Descriptive, Explanatory, Predictive, Counterfactual.
We show results of all models these splits in Table 12
CLEVRER. CLEVRER distinguishes into question types (as types of reasoning in Perception Test) and question sublypes, and we add Moving Direction as in (Li et al. 2024b):
Types: Descriptive, Explanatory, Predictive, Counterfactual.
Subtypes: Count, Exist, Query_Shape, MateraMa Query_Colorr, Moving Direction.
We show results of all models these splits in Table 13
IntPhys. IntPhys contains three types of intuitive physics examples: object permanence, shape constoncy and spatio-temporal continuity. We show results of all models on these splits in Table 14
InfLevel. IntPhys contains three types of intuitive physics examples: object permanence, shape constoncy and spatio-temporal continuily. We show results of all models these splits in Table 15
GRASP. GRASP contains much more categories than the other two intuitive physics datascts with total of 15. We show results of all models on these 15 splits in Table 16
STAR. As we adopted STAR, has only two splits (since we removed two others the filtering): Sequence
and Prediction. We show results of all models on these splits in Table 17
Abs- traction
Phy- sics
Sem- antics
Descrip- tive
Explana- tory
Predic- tive
Counter- factual
Model
Memory
LLaVA-OV (Qwen2-7B)
35.6
18.8
16.8
30.2
21.5
20.8
12.7
32.5
VideoChat2 (Mistral-7B)
25.1
7.4
12.2
15.4
9.8
18.6
14.5
22.8
Mini-CPM-v 2.6
20.8
21.9
19.8
14.3
20.0
18.7
33.3
20.8
Qwen2-VL (7B)
30.9
18.7
18.1
29.4
21.2
17.4
19.1
46.5
VU (LLaMA3.2)
20.7
8.0
7.9
23.1
10.2
12.1
14.5
38.6
LongVU (Qwen2-7B)
40.4
15.0
13.9
29.7
18.1
20.6
13.6
43.9
Tarsier-7B
20.4
10.8
18.3
21.3
14.3
22.1
10.9
26.3
Tarsier-34B
32.0
18.5
24.9
32.4
21.5
32.7
24.5
41.2
Intern LL,, 8B
29.2
29.5
29.0
27.2
30.9
27.7
33.3
29.2
Table 12: Accuracy for subsets of the Perception Test dataset.
28

---

Published in Transactions on Machine Learning Research (12/2025)
Query Shape
Query Material
Query Color Direction
Moving Descriptive Explanatory Predictive
Model
Count Exist
LLaVA-OV (Qwen2-7B)
23.3
23.4
42.2
33.3
56.5
2.8
27.9
27.6
45.0
VideoChat2 (Mistral-7B)
50.4
71.9
65.6
69.8
68.5
10.2
57.2
13.8
80.0
MiniCPM-v 2.6
17.1
34.4
37.8
12.7
60.2
4.6
28.7
17.2
60.0
Qwen2-VL VL (7B)
61.6
84.4
87.8
73.0
86.1
16.7
70.3
24.1
83.3
LongVU (LLaMA3.2)
45.1
14.1
34.4
19.0
26.9
13.0
38.6
6.9
26.7
LongVU (Qwen2-7B)
61.5
52.3
76.2
78.7
66.1
10.3
68.3
Tarsier-7B
56.0
65.1
81.5
62.4
31.0
83.3
Tarsicr-34B
71.5
92.2
90.5
93.5
37.0
79.1
27.6
95.0
InternVL2.5-88
81.5
93.8
88.9
96.8
96.3
99.1
88.1
24.1
93.3
Table 13:
Accuracy for subsets of the CLEVRER dataset.
Model
Shape constancy Spatio-temporal continuity Object permanence
LLaVA-OV (Qwen2-7B)
0.0
0.0
0.0
VideoChat2 (Mistral-7B)
0.0
0.0
0.0
MiniCPM-v 2.6
1.8
Qwen2-Ve VL (7B)
14.7
4.7
6.8
VU (LLaMA3.2)
0.0
VU (Qwen2-7B)
14.7
2.3
4.5
Tarsier-7B
20.6
18.6
18.2
Tarsier-31B
5.9
0.0
6.8
InternVL2..2 LB2.58B
12.1
14.5
6,8
Table 14: Accuracy for subsets of the IntPhys dataset.
Model
Continuity
Solidity
Gravity
VA-OV (Qwen2-7B)
0.0
0.0
VideoChat2 (Mistral-7IB)
0.0
0.2
0.0
MiniCPM-v 2.6
0.0
0.0
0.0
Qwen2-VL. (7B)
11.3
8.8
13.3
LongVU (LLaMA3.2)
8.0
5.9
7.8
LongVU (Qwen2-7B)
7.0
6.8
9.6
Tarsier-7B
7.5
2.7
0.6
Tarsier-34B
5.0
14.0
14.5
Table 15: Accuracy for subsets of the InfLevel dataset.
29

---

Published in Transactions on Machine Learning Research (12/2025)
Unchangesainn bienese2
Objert Permanence
Solidity contimuity?
Solddit continuity
Objoct Permancnce2
Objert Permancneed
Cravity Coutinuity
Gravity Support
Gravity Inertla.
Unchangeablena
Graviity Inertia2
Coutiauitty
Inertia
Inertio2
Collision
Gravity
Model
LLaVA-OV (Qwen2-7B)
0.8
2.3
0.0
0.0
0.0
10.2
0.8
9.8
0.0
1.6
0.0
5.5
0.0
0.0
8.6
0.8
VideoChat2 (Mistral-7BB
0.0
0.0
0.0
0.0
0.0
0.0
0.0
0.0
0.0
0.0
0.0
0.0
0.0
0.0
0.0
MiniCFM-v 2.6
3.1
10.9
10.2
0.8
0.0
0.8
4.7
5.5
6.2
6.3
0.8
3.9
3.1
2.3
23.4
14.8
Qwen2.
7.0
LongVU (LIaMA3.2)
0.8
0.8
35.2
0.0
4.7
0.0
0.8
0.0
2.3
0.0
0.0
0.0
3.1
0.0
24.2
2.3
LongVU (Qwen2-7B)
9.4
16.4
42.2
3.1
7.8
1.6
0.0
4.7
2.3
12.5
39.1
10.9
Tarsier-BB
10.9
10.2
5.5
0.0
3.1
8.6
3.9
7.3
3.1
3.1
166
0.8
0.0
0.0
3.1
11.7
Taniier34B
35.2
1.6
0.8
0.8
4.7
1.6
11.8
0.0
0.0
6.2
7.0
0.8
16
0.8
15.6
2.3
InternV1.2.58BB
10.2
42.2
3.1
22.0
19.5
9.4
2.3
13.3
6.2
9.4
4.7
6.2
3.9
3.1
3.9
3.1
Table 16: Accuracy for subsets of the GRASP dataset.
Model
Prediction
Sequence
LLVVA-OV (Qwen2-7B)
39.9
59.8
VideoChat2 (Mistral-7B)
33.3
53.3
MiniCPM-v 2.6
54.3
54.0
Qwen2-VL (7B)
54.3
64.2
LongVU (LLaMA3.2)
52.2
50.5
LongVU (Qwen2-7R)
58.0
65.0
Tarsier-BB
39.9
50.7
Tarsier-34B
43.5
64.0
InternVL.2.5-
67.4
75.5
Table 17: Accuracy for subsets of the STAR dataset.
F Illustrative examples from MVP
We show 2 examples for each of the uine sources in MVP, ranging from Figure to Figure 17
B) The lighting device did not work most likely be- cause it was already unplugged the beginning of the video.
A) The device worked when the person tried to turn it on.
Question: If the person tried to turn the lighting device and it not working, what the reason for that?
A) yellow-white book Question: If the object were launched with greater force the friction was smaller, with which object would the launched object collide?
B book
Figure 9: Two random samples (i.e. not cherry-picked) from the Perception Test split MVP. We show pairs of videos where the answer for one video is the negative candidate for the other video.
30

---

Published in Transactions on Machine Learning Research (12/2025)
Free A)
the red circle into the green star
B) move the green star towards the red circle Question: Which robot instruction best describes the actions in the video?
heet hewll
Powe
haw
A) move the yellow hexagon diagonal to the top right side the red star
B) move the yellow hexagon diagonal to the top left side of the red star
Question: Which robot instruction best describes the actions in the vidco?
Figure 10: Two random samples (i.e. not cherry-picked) from the Language Table split of MVP. We show pairs of videos where the answer for one video is the negalive candidate for the other video.
Iew
A)
unfolding something
B) folding something Question: Which action is being performed in the video?
hume
Neal
hew
Home
Tram
Newr
Nime
A) pushing something from right to left
B) pulling something from left to right Question: Which action is being performed in the video?
Figure 11: Two random samples (i.e. not cherry-picked) from the Something Something v2 split of MVP. We show pairs videos where the answer for one video the negative candidate for the other video.
B) the presence of the yellow sphere Question: Which of the following is not responsible for the yellow cube's colliding with the brown object?
A) the cylinder's entering the scene
A) no
B) yes
Question: Are there any stationary objects when the sphere enters the scene?
Figure 12: Two random samples (i.e. not cherry-picked) from the CLEVRER split of MVP. We show pairs of videos where the answer for one video is the negative candidate for the other video.
31

---

Published in Transactions on Machine Learning Research (12/2025)
Fawid
Tomm
Pames
A) No, something in the video is off /ttrnne or vi-
B) Yes, everything is behaving according to human intuitive physics understanding
olates human inluitive physics understanding Question: this video physically plausibleppossibee according your understanding of e.g. object per- manence, shape constancy (objects maintain shape time), continuous trajectorics of objects? Assume it is the normal laws of physics. Your answer should be based on the events in the video and ignore the quality of the simulation engine. The rising wall is part of the experiment setup and should not be judged for plausibility.
Fawi
Tomn
Pimas
A) No, something in the video is ofr/stra or vi-
B) Yes, everything is behaving according to human intuitive physics understanding
olates human intuitive physics understanding Question: Is this video physically plausible/ /possiele according to your understanding of e.g. object perma- nence, shape constancy (objects maintain shape over time), continuous trajectorics of objects? Assume it is the normal laws of physics. Your answer should be based the events in the video and ignore the quality of the simulation engine.
Two random samples (i.e. not cherry-picked) from the IntPhys split of MVP. We show pairs of videos where the answer for one video the negative candidate for the other video.
Figure 13:
Few B) No, something the video off /strgga or vi- olates human intuitive physics understanding Question: Is this video physically plausible/posse according to your understanding of e.g. object perma- nence, gravity, trajectories of objects? (Assume it is the normal laws of physics and there Sometimes the video might start with visually showing the general setup of objects before the important actions happen)
A) Yes, everything is behaving according human intuitive physics understanding
magic tricks.
Irw B) No, something the video off /strange vi- olates human intuitive physics understanding
Frewar
A) Yes, everything is behaving according human intuitive physics understanding Question: Is this video physically plausible/ plausible/possible possible according to your understanding of e.g. object perma- nence, gravity, trajectories of objects? (Assume it is the normal laws of physics and there are no magic tricks. Sometimes the video might start with visually showing the general setup of objects before the important actions happen)
Two random samples (i.e. not cherry-picked) from the InfLevel split of MVP. We show pairs of videos where the answer for video the negative candidate for the other video.
Figure 14:
32

---

Published in Transactions on Machine Learning Research (12/2025)
Pree
Trem
B) No, something in the video is off/strange vi- olates human intuitive physics understanding Question: The video you'l re seeing was generated by a simulator. Given how objects behave on earth, is the final position of the ball plausible? Your answer should be based on the events the vidco and ignore the quality of the simulation.
A) Yes, everything is behaving according to human intuitive physics understanding
Tree
A) Yes, everything is behaving according human intuitive physics understanding Question: The video you're seeing generated by simulator. Given how objects behave earth, is the final position of the ball plausible? Your answer should be based on the events in the video and ignore the quality of the simulation.
B) No, something the video off ssranng vi- olates human intuitive physics understanding
Figure 15: Two random samples (i.e. not cherry-picked) from the GRASP split of MVP. We show pairs of videos where the answer for one video the negative candidate for the other video.
The
hew
New
B) Took the dish.
A) Took the sandwich.
Question: What happened after the person sat at the table?
hew
A) The paper/notcbook
head
Puwe Eat
Nom
B) The book.
Question: Which object did the person put down after they held the shoc?
Figure 16: Two random samples (L.e. not pherry- from the STAR split of MVP. We show pairs of videos where the answer for one video is the negative candidate for the other video.
33

---

Published in Transactions on Machine Learning Research (12/2025)
A) the man was not welding before he uses his weld- ing gun and flames come out.
B) the man uses his welding gun and flames come out before he was not welding
Question: What is the best caption for this video?
A) oil is drained before the cap is loosened
How
B) the cap is loosened before oil is drained Question: What is the best caption for this video?
Tren
Yrace
Figure 17: Two random samples (i.e. not cherry-pickod) from the Vinoground split of MVP. We show pairs of videos where the answer for one video is the negative candidate for the other video.
G Human Annotation
We assigned over 600 vidcos rescarchers from lab, recorded their answer responses, and then computed the benchmark metric using the pair-wise scoring. Each person was assigned one video from the pair at a time, thus avoiding any advantage over VideoLLMs that would come from seeing both videos in a minimal-change pair (i.c., avoiding any knowledge that the answer can only be AB or BA).
In addition to the main splits of MVP, we also provide a breakdown of human accuracy across the 9 data sources we used:
Perception Test: 87.8%
Something Something V2: 94.7%
Language Table: 91.7%
CLEVRER: 100%
InLevvl:: 100%
IntPhys: 96.3%
GRASP: 94%
STAR: 81.8%
Vinoground: 100%
Details on prompting multi-modal LLMs
For a fair comparison (HuggingFace, API), we use each model's default inference setup (e.g. greedy decoding for all models) which are quite similar anyways. For example, videos are similarly processed (resolution, number of frames) and then fed through CLIP or ViT-based video encoders. Most models use either 8 or frames by default, e.g. Tarsier-34B uses 16 which we believe enough for most MVP examples, yet the model still far below humans.
We the following prompt for all VideoLLMs (filled with example question from benchmark):
34

---

Published in Transactions on Machine Learning Research (12/2025)
VideoLLM Prompt
You are an expert video understanding AT system. Carefully watch the video and pay attention the cause and sequence of events, the details and movements of objects, and actions of people. Based your observations, select the best option that accurately addresses the following question:
Q: (Question)
A) [Correct answer for video1}
B) {Correct answer for video2}
Even when unsure, always answer with a single letter from A or B, format exactly like: 'Answer: A/B:
We extract the answer letter via simple regex and find that this approach fails in only less than 1% of examples.
Behind the Scenes
In this section we go beyond what usually goes into paper and discuss how the paper came about, what did not work, or what motivated the authors so in essence: all the things that are usually deemed too subjective or "unscientific", yet would help other researchers, especially those joining the field, often much more than the polished narrative of the main paper.
1.1 Motivation and timeline
Several of the authors who work video modeling felt growing frustration with existing benchmarks that often rewarded the wrong things. So the dircction of the project was quickly set after short period of brainstorming: Quantify in what ways existing benchmarks are broken, and then fix it. We then spent few months staring at hundreds examples from the MVBench datasets, scouting for glaring issues or shorteuts and manually annotating lots of data. First, we tested the simple baselines with respect to frequency, text-only or single-frame biases, and soon included the less often discussed video-only (remove question) had philosophical discussions about benchmark design
and Simple Socratie LLM shortcuts. In between what it means for video task to be truly temporal: is it temporal if two-
(bottom-down vs. top-down) frames are needed, or if a single frame is needed but it has to be a key-frame (needle in haystack), or ...? Regarding benchmark design, should we adopt other paper's taxonomies or design or own? Should one collect all kinds of examples and ad-hoc define a taxonomy (bottom-up), or should one define a taxonomy, then systematically collect examples to fit the taxonomy (top-down)? From the beginning the idea of minimal video pairs generated excitement among us: Minimal visual pairs have led to much progress in the field of vision-and language compositionality (e.g. Winoground), yet had not been explored much in the realm of moving images.
There were, and perhaps still are, plans to crowd-source human shortcut performance, i.e. how much better are humans at solving video tasks when given single frames? At scale this could also be used to filter out examples with more precision than our five-model ensemble approach. Doing human crowd-sourcing well is not trivial, it is time-intensive and requires dedication but il can lead to much stronger insights than relying purely on automatic metrics and black-box models.
After the exploration phase, executed the benchmark building: From the start we had identified several promising datasets to mine minimal video pairs and contimouuly added new source roughly every week. Perception Test was the first to go through our curation pipeline and hence took the longest as we were still refining the pipeline steps. Language Table was very hard to do well with many edge cases in the entailment detection, and also with its scale 440K video-caption examples (imagine looking for potential pairs of videos, i.e. 440, 0002 combinations).
1.2 Observations and lessons learned
There are too many edge cases to catch every single at this scale of data curation.
35

---

Published in Transactions on Machine Learning Research (12/2025)
2. Paper writing is smooth when the story and contribution is clear from the beginning of the project (this was not the case in the first author's last paper so it was nice to observe the contrast...
The intuitive physics datasets are (to the subjective taste of the first author) the cleanest and most fascinating sources in MVP, even though they are not directly suitable for video QA, and only make up small fraction of the benchmark.
At the same time, Something Something v2 truly stood the test of time as a great video understand- ing dataset due to its low noiso-ratio a scale of 200K examples, and coverage of interesting yet simple phenomena.
5. Frame rate plays a big role for solving many examples in MVP. In order to push the field further we are now asking the models more and more nuanced questions, and the answer may lie only in a short span of a less than second. However many models may not have access this short span principle as they represent video 16 uniformly sampled frames.
6. Parsing outputs from LLMs into a structured format such as answer options can feel like the wild west sometimes: Could models perform better if we prompt them better, or have more flexible ways of extracting the answer?
1.3 Advice for others working similar direction
Video-QA is becoming increasingly popular topic; it is very exciting direction with enough dimensions for everyone to innovate on: long video benchmarking, intuitive physics, social common-sense/ waa.o---seseeeeoe--ootttoo Mind/narratives, novel simulation engines, and so on.
Despite our best efforts studying shortcuts, we probably missed some shortcuts or issues in MVP. It is good to think two steps ahead what kind of shoricuts future more capable models could take. Tt easier said than done, but in retrospect older video benchmarks from 2015-2020 might have at least been able to address single-frame biases, priori, during benchmark design. Instead now the field took years to identify and clean up benchmarks.
36


Reviews:
[{"review_id": "N4iJxQjoYj", "reviewer": "Reviewer_wWdq", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "No", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "**Summary of Contributions:**\n+ This work proposes the Minimal Video Pairs (MVP) benchmark, a large-scale shortcut-aware VideoQA benchmark for measuring the physical understanding and reasoning capabilities of video language models. It further introduces an automated pipeline to mine minimally different video pairs from existing video sources by leveraging visual embeddings and video meta-data.\n\n+ Presents an analysis of potential shortcut solutions on the 20 tasks encompassed by MVBench benchmark using a variety of simple baselines.\n\n+ Evaluate both open-source and closed-source SOTA models on MVP, signalling a gap in physical world understanding to human performance vs GPT4-o and Gemini.", "claims_and_evidence_yn": "No", "claims_explanation": "+ While the paper excessively uses the term “shortcuts” and has also done a shortcut analysis on MVBench – it fails to coherently explain what a shortcut actually is. I understood this in much later in the paper. A well-motivated example with a figure could have made things more clear.\n\n+ While the scale of MVP is huge as compared to prior attempts (VINOGROUND and TempCompass) it still restricts to multi-choice QA instead of also including other task formats thereby hindering the understanding of how temporal perception may vary across different tasks – a limitation pointed by these prior works themselves. (Cheng et.al.,V-STaR: Benchmarking Video-LLMs on Video Spatio-Temporal Reasoning, arXiv, 2025) tells that (what/when/where) fails and prompts the models to give localized instances of the same instead of simple binary answers. (Zhou et.al., TUMTraffic-VideoQA: A Benchmark for Unified Spatio-Temporal Video Understanding in Traffic Scenes, arXiv, 2025) shows that robust spatio-temporal reasoning requires multiple tasks like referred object captioning, etc. MVP’s minimal pair evaluation, however,  does not show why the model failed.\n\n+ The paper misses mentioning the quantity of cases where the dataset examples are noisy / ambiguous. While the paper reports the accuracy, there is no failure case analysis as to why models fail on certain examples. A section on the same can make the contribution of MVP more intuitive.\n\n+ The paper could have also included multiple examples of the minimal change pairs for the sake of seeing diversity across visual changes included in the video pairs", "audience_yn": "Yes", "audience_explanation": "The work is interesting and relevant to the large computer vision community, particularly those focused on video understanding.", "requested_changes": "+ Provide a clear and coherent definition of “shortcuts”, supported by a well-motivated example and an illustrative figure.\n\n+ Expand the benchmark beyond multi-choice QA to include additional task formats (e.g., captioning, localization, temporal ordering) to better capture temporal perception across diverse tasks. (Optional)\n\n+ Discuss related findings from V-STaR (Cheng et al., 2025) and TUMTraffic-VideoQA (Zhou et al., 2025) to highlight the need for richer task types and failure localization, and clarify how MVP addresses or differs from these limitations.\n\n+ Explicitly mention the quantity and proportion of noisy or ambiguous dataset cases, along with steps (if any) taken to mitigate their effect.\n\n+ Add a failure case analysis section explaining why models fail on certain examples, complementing the reported accuracy metrics.\n\n+ Include multiple diverse examples of minimal change pairs in the main text or appendix to showcase the range of visual variations represented in the dataset.", "broader_impact_concerns": "The MVP benchmark raises broader impact concerns by potentially oversimplifying temporal reasoning through limited task formats and unclear shortcut definitions. Without addressing dataset noise and failure analysis, there is a risk of overestimating model competence, which could misguide the community and hinder progress toward robust, real-world video reasoning systems.", "additional_comments": "Minor comments:\nThe caption for Figure 1 is not informative.\n\nThe presentation of the paper can be improved by including more intuitive examples and figures with self-explanatory captions.", "extra_scores": {}, "extra_text": {}}, {"review_id": "7lQXy4NT1H", "reviewer": "Reviewer_D1CL", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "No", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "This paper introduces the Minimal Video Pairs (MVP) benchmark, which aims to address the problem of score inflation in existing Video QA benchmarks where models achieve score inflation by exploiting superficial visual or textual cues. MVP contains approximately 55,000 multiple-choice video QA examples, focusing on physical world understanding. To effectively resist shortcut solutions, each sample in MVP contains a minimal-change pair: a visually similar video with opposite answers. The model can only score if it correctly answers both questions in the minimal-change pair at the same time. The paper analyzes the shortcut solutions of the existing MVBench benchmark and, by introducing MVP, reveals the huge gap in physical world understanding of current VideoLLMs, where human performance reaches 92.9%, while the state-of-the-art models only reach 40.2%.\n\n### Strengths\n\n1. The authors conduct an in-depth analysis of potential shortcut solutions across all 11 datasets in the popular benchmark, using simple baselines such as language-only models, single-frame/image models, and Socratic LLMs.\n2. The authors introduce MVP, a video QA benchmark for physical world understanding. It features minimally diverse videos and is an order of magnitude larger than similar benchmarks, with approximately 55,000 samples.\n3. The authors benchmark closed-source and open-source state-of-the-art models, revealing significant gaps in physical world understanding.", "claims_and_evidence_yn": "No", "claims_explanation": "1. The paper mentions that the majority of minimal-change pairs were mined through an automated process. While this allows the benchmark to achieve a large scale, it may also introduce noise. For example, entailment detection in the Language Table involves several handcrafted rules, and the robustness and generalization of these rules warrant further investigation. The definition of \"fuzzy subset\" mentioned in Appendix B.2 is also somewhat vague. Errors in these automated steps could result in some minimal pairs not being truly \"minimally different\" or \"mutually exclusive,\" thus affecting the validity of the benchmark.\n2. Section 4 of the paper discusses the impact of the single-letter output format and tests the performance of Gemini 1.5 after adding a reasoning prompt, showing a slight performance degradation. However, for such a benchmark, prompt engineering may have a far greater impact on VideoLLMs' performance than this. Exploring more diverse prompting strategies (e.g., Chain-of-Thought, few-shot prompting, etc.) and their impact on model performance on MVP will help more comprehensively assess the model's true capabilities.\n3. The paper evaluated multiple state-of-the-art VideoLLMs, but most performed poorly on MVP. While this highlights the challenge, future research may require a deeper analysis of the specific modes of failure of these models, such as whether they suffer from systematic deficiencies in spatial reasoning, temporal understanding, or intuitive physics. Currently, simply providing overall and subcategory scores may not be sufficient for understanding model weaknesses and guiding improvements.", "audience_yn": "Yes", "audience_explanation": "This work proposes a shortcut-aware video question answering benchmark (MVP), which systematically exposes the \"shortcut\" bias of the existing Video-LLM evaluation using minimal pairs and single-frame filtering; this is of direct value to TMLR readers who are interested in multimodal evaluation, benchmark design, and video understanding modeling.", "requested_changes": "1. In addition to the Minimal Pair Score, it is recommended to supplement single-question accuracy, pair agreement rate, uncertainty measures (such as temperature sampling variance), confidence-accuracy calibration curves, and report the pair difficulty distribution.\n2. Randomly sample N pairs from each major category for double-blind verification, reporting the noise ratio and major ambiguity types (occlusion, language ambiguity, and video quality).\n3. Use a non-LLM single-frame classifier (such as a classic action classifier or temporal consistency classifier) to compare with the current LLM ensemble, reporting the agreement rate and difference examples.\n4. If affordable, expand the GPT-4o and Gemini evaluation to a representative subset of the MVP-full dataset, standardizing the frame rate, resolution, and prompt. Alternatively, provide performance curves for the inference budget to avoid biased conclusions due to \"different parameters.\"\n5. In Appendix B.2, provide more specific examples or descriptions of the \"several hand-crafted rules\" used in constructing the Language Table dataset. Also, provide a clearer explanation of the definition of the fuzzy_subset function used in pairing the CLEVRER dataset (particularly the meaning of \"one mismatch\"). It is recommended to supplement the results of a small-scale manual validation, such as randomly sampling minimal pairs generated by the automated process, and report the quality ratio of human judgment to quantify the reliability of the automated process.\n6. Currently, the performance of models in the Intuitive Physics category is generally close to random. Please further analyze the specific reasons for this extremely low performance, for example, whether the models generally have systematic deficiencies in specific physical concepts such as object permanence, gravity, and collision dynamics. If possible, provide a qualitative analysis of the errors of the best-performing models in this category, and provide one or two specific error cases to help the community understand the specific patterns of model failure.\n7. The exploration of prompt engineering in the paper is limited to the effects of Gemini 1.5 after the addition of reasoning prompts. Please also conduct similar reasoning prompt experiments for at least one open-source SOTA VideoLLM. Alternatively, briefly discuss other common prompting strategies (e.g., few-shot examples).", "broader_impact_concerns": "The main ethical risks lie in licensing compliance and potential privacy issues associated with the re-annotation and redistribution of multi-source videos (including egocentric ones), as well as the representativeness and bias introduced by sample distribution. It is recommended to supplement the license matrix and privacy handling instructions for each data source. Another risk is that single-frame filtering relying on an MLLM ensemble may encode model bias into the data and induce overfitting of the minimal-pair architecture. This requires increased bias auditing and usage specifications.", "additional_comments": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "pv5MTxZO78", "reviewer": "Reviewer_4W8E", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "Yes", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "## Summary\nThis paper first evaluates a few shortcuts in existing video QA benchmarks. It reveals that some of current benchmarks suffer from language shortcut, video shortcut, single-frame shortcut, and course-grained caption shortcut. Then, the paper introduces Minimal Video Pairs (MVP), a video QA benchmark that mitigates the shortcuts. Eash sample in MVP includes a pair of similar videos with the same question but opposite answers. The videos are sourced from nine existing video QA benchmarks under four categories. And the video pairs are matched by an automatic procedure. The paper evaluates eleven multimodal LLMs and the best performance is slightly better than random chance and significantly worse than human performance. Especially, on the intuitive physics and collisions category, all the models perform worse than random chance.\n\n## Strengths:\n1. The paper reveals four types of shortcuts in existing video QA benchmarks, providing insights about these benchmarks to the community.\n2. The proposed MVP benchmark avoids the language shortcut by collecting video pairs and mitigates the single-frame shortcut by filtering.\n3. Although MVP benchmark requires paired videos, they search for the paired videos from existing benchmarks in an automatic procedure. This makes it easier to scale up the MVP benchmark and results in 27k video pairs.\n4. The paper conducts comprehensive evaluations on eleven multimodal LLMs, including open-source and proprietary models. Human performance is also provided as a reference.\n\n## Weaknesses:\n1. While the proposed MVP is a video benchmark, the paper does not include much visualization of it. The only visualization is in the figure on page 1, where three video pairs are provided and only one frame is presented for each video. I would suggest that the paper visualizes at least one sample for each of the four categories.\n\n2. All the evaluation results are quantitative but not qualitative. It would be better if the paper could include qualitative results to show a) How the models leverage the shortcuts in existing benchmarks and b) How the models fail in MVP.\n\n3. For four out of the nine video sources, the MVP benchmark directly uses the video pairs provided by the sources. However, the details about this part of videos are unclear, especially whether they share the same property as the newly matched video pairs. For example, do their video pairs share the same question while having opposite answers? How visually similar are these video pairs?\n\n4. While the paper evaluates four types of shortcuts of existing video QA benchmarks, the proposed MVP benchmark is only designed to mitigate two of these shortcuts (language shortcut and single-frame shortcut). While it is acceptable to only address part of the shortcuts, the readers would be curious about whether the MVP benchmark also suffers from the video shortcut and the course-grained caption shortcut. However, the paper does not evaluate the video-only baseline or the Socratic LLM baseline.", "claims_and_evidence_yn": "Yes", "claims_explanation": "1. The paper claims that there are shortcuts in existing video QA benchmarks. This is supported by the evaluation results of a few baselines leveraging shortcuts.\n2. The paper claims that the proposed MVP mitigates the shortcuts. This is supported by the detailed dataset curation process and the evaluation of language-only and single-frame baselines.\n3. The paper claims that the state-of-the-art multimodal LLMs still struggle with seemingly simple physical reasoning tasks. This is supported by the evaluation on the MVP benchmark.", "audience_yn": "Yes", "audience_explanation": "This paper is about multimodal LLMs and video understanding. Both have a large population of audience.", "requested_changes": "1. The performance of InternVL2.5-8B on MVP is 39.9% in Table 4 but 39.0% in Figure 1. Please fix the typo. Moreover, the abstract and introduction report the best performance of 40.2% on the entire MVP benchmark, but Figure 1 only reports the result on the mini version. This would make the readers confused. I would suggest reporting the performance on both versions in Figure 1.\n\n2. In \"VideoLLMs performance on dataset sub-tasks\", Section 4, the paper only mentions a few numbers for a few models and sub-tasks. The paper should includes a complete table listing all the sub-task performance in either the main paper or the appendix.\n\n3. Evaluate the video-only baseline or the socratic LLM baseline on MVP.\n\n4. Include more figures to demonstrate the MVP benchmark and the qualitative results of the models.", "broader_impact_concerns": "No broader impact concerns.", "additional_comments": "", "extra_scores": {}, "extra_text": {}}]

Author rebuttal:

Decision: Accept (poster)