TOMATO: Assessing Visual Temporal Reasoning Capabilities in Multimodal Foundation Models
Authors: Ziyao Shangguan
Chuhan Li
Yuxuan Ding
Yanan Zheng
Yilun Zhao
Tesca Fitzgerald
Arman Cohan
Venue: ICLR 2025 Poster (2025)
Keywords: visual temporal reasoning
video understanding
benchmark
vision-language benchmark
video-language models
evaluation
Primary area: datasets and benchmarks

Abstract: Existing benchmarks often highlight the remarkable performance achieved by state-of-the-art Multimodal Foundation Models (MFMs) in leveraging temporal context for video understanding.
However, *how well do the models truly perform visual temporal reasoning?*
Our study of existing benchmarks shows that this capability of MFMs is likely overestimated as many questions can be solved by using a single, few, or out-of-order frames. 
To systematically examine current visual temporal reasoning tasks, we propose three principles with corresponding metrics:
(1) *Multi-Frame Gain*,
(2) *Frame Order Sensitivity*,
and (3) *Frame Information Disparity*.
Following these principles, we introduce **TOMATO**, **T**emp**O**ral Reasoning **M**ultimod**A**l Evalua**T**i**O**n, a novel benchmark crafted to rigorously assess MFMs' temporal reasoning capabilities in video understanding.
TOMATO comprises 1,484 carefully curated, *human-annotated* questions spanning *six* tasks (i.e. *action count, direction, rotation, shape & trend, velocity & frequency, and visual cues*), applied to 1,417 videos, including 805 self-recorded and -generated videos, that encompass human-centric, real-world, and simulated scenarios. 
Our comprehensive evaluation reveals a human-model performance gap of 57.3% with the best-performing model.
Moreover, our in-depth analysis uncovers more fundamental limitations beyond this gap in current MFMs. While they can accurately recognize events in isolated frames, they fail to interpret these frames as a continuous sequence.
We believe TOMATO will serve as a crucial testbed for evaluating the next-generation MFMs and as a call to the community to develop AI systems capable of comprehending the human world dynamics through the video modality.

Published as a conference paper at ICLR 2025
TOMATO: ASSESSING VISUAL TEMPORAL REASONING CAPABILITIES IN MULTIMODAL FOUNDATION MODELS
Ziyao Shangguan" !
Chuhan Li
Yuxuan Ding'
Yanan Zheng'
Yilun Zhao Tesca Fitzgerald' Arman Cohan
Yale University Allen Institute for Al
shangguan chuhan. 1i. ctt.:ii...115yyyyllllll. com com/yall--
ABSTRACT
Existing benchmarks often highlight the remarkable performance achieved by state-of-the-art Multimodal Foundation Models (MFMs) in leveraging temporal context for video understanding. However, how well do the models truly perform visual temporal reasoning? Our study of existing benchmarks shows that this capability of MFMs is likely overestimated as many questions can be solved by using a single, few, or out-of-order frames. To systematically examine current visual temporal reasoning tasks, we propose three principles with corresponding metrics: (1) Multi-Frame Gain, (2) Frame Order Sensitivity, and (3) Frame Infor- mation Disparity. Following these principles, we introduce TOMATO, TempOral Reasoning MultimodAl EvaluaTiOn, novel benchmark crafted to rigorously assess MFMs' temporal reasoning capabilities in video understanding. TOMATO comprises 1,484 carefully curated, human-annotated questions spanning six tasks (i.e., action count, direction, rotation, shape & trend, velocity & frequency, and visual cues), applied to 1,417 videos, including 805 self-recorded and -generated videos, that encompass human-centrie real real-world, and simulated scenarios. Our comprehensive evaluation reveals numan-model performance gap of 57.3% with the best- pest-performing model. Moreover, our in-depth analysis uncovers more funda- mental limitations beyond this gap current MFMs. While they can accurately recognize events in isolated frames, they fail to interpret these frames as continu- ous sequence. We believe TOMATO will serve as crucial testbed for evaluating the next-generation MFMs and as call to the community to develop Al systems capable of comprehending the human world dynamics through the video modality.
INTRODUCTION
Visual temporal reasoning, an important aspect of human perception, refers to the cognitive process of understanding and interpreting sequences of visual information over time, such recognizing patterns of motions, detecting changes in scenery, and following the progression of events (Kahneman ct al. 1992). Currently, state-of-the-at methods for addressing visual temporal reasoning are centered on the use of Multimodal Foundation Models (MFMs) (OpenAl 2024 Anthropic 2024b Wang al. [2244] et al., (024), which have shown remarkable performance across numerous temporal reasoning video benchmarks (Li et al. 2023 Liu al.) 2024 Li et al.) (0224)) Chen et al. (024a). However, despite these impressive performances, our study $5 has shown that the models' true capabilities in visual temporal reasoning are likely overestimated.
This study examines four existing widely-used temporal reasoning video benchmarks (Li et al. 2023 2024b; Liu et al.. 2024| Chen et al. (2024)), illustrated in Figure [ We identify patterns in their question formulation lttion that allow models to exploit shortcuts, enabling them to answer correctly using single, few, or out-of-order frames. To rigorously evaluate whether benchmark effectively assesses MFMs' visual temporal reasoning ability, we propose three principles with corresponding metrics:
"Equal Contribution.

---

Published as a conference paper at ICLR 2025
Existing Benchmarks
VITATECS (Li 2023]:
Which of the following bext describes the content of the video?
receoeed spread
being sliced
Vadeo ID: 7966
AVBencch (Li al., 2024):
behind
when she in chair tulking to
A. A Atelevision
B. Jaes's white dog sculpture
C.An American flag.
basket laundry
Video ID: fands didal0 sugto ddp, 11,
Tempoommar Compass (Liu 2024):
the woman athletes doing?
A. Cycling Swimming
Running
D. Dancing
ReXTimme (Chen et al., 2024):
Why is the woman preparing ingredients and utensils?
Tb make herself a sandwich
B. To cook dinner for her family
Vidro li: vatte
C. bake cake
D. To prepare salad
TOMATO (Ours)
In which direction(s) Ping Pong
A. Clockwise throughoul
Counter clockwise clockwise
C. Counter-clockwwte throughout
Clockwise counte--llcckwse
Figure Examples of several VideoQA benchmarks. We examine four existing representative benchmarks and ours: VITATECS, MVBench, TempCompass, ReXTime, and TOMATO, In the four existing henchmarks, we use bounding boxes highlight frames that models can exploit as shortcuts, allowing questions to be answered using any one of highlighted frames. In TOMATO TO, however, models must reason both between consecutive frames and across all frames as a continuous sequence.
(1) FraltieFrameeee Gain, where a higher value indicates the task is less solvable by single frame (.1.1) (2) Frame Order Sensitivity, where a higher value indicates the task is more reliant on the correct order of frames ($3.2), and (3) Frame Information Disparity, where lower value indicates information is more evenly distributed across the frames ($3.)) Our analysis reveals that, on average, existing benchmark exhibit a Multi-Frame Gain of less than 5%, compared 66.3% for ours ($5.1); Frame Order Sensitivity of less than 8%, compared to 34.1 % for ours ($5.2); and a Frame Information Disparity of over 27% while ours only 4.6% ($5.3). These results suggest that tasks from existing benchmarks are relatively more solvable by single frame, rely less the order of the frames, and allow for more unevenly distributed information across the frames. Therefore, models" visual temporal reasoning capabilities, crucial to MFMs' video comprehension, are likely overestimated.
Following the three aforementioned principles for building a more effective visual temporal reasoning benchmark, we introduce TOMATO TOO TempOral Reasoning MultimodAl EvaluaTiOn, novel video understanding benchmark designed to explicitly assess MFMs' temporal reasoning capabilities. TOMATO comprises 1,484 carefully curated, human- annata-n multiple-choie questions spanning six distinct temporal reasoning tasks (action count, direction, rotation, shape & trend, velocity & frequency, and visual cues) applied 1,417 videos encompassing human-centric, real-world, and simulated scenarios 1.. 'OMATO AT features a diverse collection of videos, sourcing from YouTube, four existing video datasets (Jang et al., 2017 Yi et al. 2020 Li et al. 2022 Paträucean ctt al. (023),, as well as self-recorded and -generated videos ($4.2). To enhance diversity, we carefully edit the sourced YouTube and self-recorded videos, incorporating various characteristics such as counter-factual scenarios, composite motions, and zoomed-in views. In creation of the 805 original videos. we explicitly include both existing videos that capture human interactions or generated synthetic scenes. The question-answering (QA) pairs are meticulously designed to ensure that models need reason about the transitions across all frames, making visual temporal reasoning essential for solving the tasks.
and simulated scenarios to address the lack of
We conduct a comprehensive evaluation of 21 open-source models and 10 proprietary models. Notably, the best-performing open-source model, Qwen2-VL-72B. which achieves 37.9% overall accuracy, outperforms all proprietary models, including GPT-40, which achieves 37.7% overall accuracy. However, both types of models remain significantly below human-level performance, which reaches 95.2% using full videos and 79.7% with 16 frames. Moreover, our analysis goes beyond highlighting these performance gap, and it exposes deeper, more fundamental limitations in current
2

---

Published as a conference paper at ICLR 2025
MFMs' capabiliiiees ($6.3). Specifically, show that these models: (1) lack the basic ability to interpret the frames as a continuous sequence, (2) fail to truthfully leverage the visual input while being over-reliant on common sense, and (3) are highly susceptible to noisy information. We hope that our findings can provide useful insights for future work guiding development of improved MFMs.
In summary, our contributions are:
We identify inflated performance of MFMs on existing visual temporal reasoning benchmarks and establish three principles with metrics to evaluate the effectiveness of benchmarks assessing models' visual temporal reasoning capabilities.
We introduce TOMATO, a novel benchmark for assessing MFMs' capabilities on visual temporal reasoning tasks, spanning six reasoning types and three video scenarios, including 805 self-created and -generated videos.
We present a comprehensive evaluation of 21 open-source models and 10 proprietary models on TOMATO, revealing a substantial gap between human-level and MFM-enabled visual temporal reasoning capabilities.
We provide in-depth error case analysis, uncovering more fundamental shortcomings in MFMs' visual temporal reasoning capabilities that go beyond the human-model performance gap.
2 RELATED WORK
General video understanding benchmarks. Video understanding capability plays pivotal role in multimodal learning and is key step toward achieving artificial general intelligence. Prior to the era of MFMs, early benchmarks (Heilbron et al. 2015 Goyal et al., 2017: focus primarily action recognition. However, more recent benchmarks (Jang et al. 2017; Yi et al. 2020) represent shift towards evaluating models" ability to reason about temporal dynamics and causal events. The rise of MFMs has further propelled the field toward more complex, human- like understanding tasks. These tasks include (1) long-fore form video understanding (Zhang et al. 2023
et 2018)
Rawal et al. 2024 Zhou et al. 2024 Nagrani al. 2024). (2) multi-disciplinary video understanding
(He et al.] 2024). and (3) comprehensive evaluation across various tasks (Ning et al.) 2023 Li
et al.) 2024d|c c Fu al., (024). Building upon this remarkable progress, MMBench- Video (Fang et al. benchmarks are rendered rather "static," but its reasoning dimension such Attribute Recognition,
2024) advocates the needs for more temporal questions since many questions in existing
Object Recognition. and OCR still remain static. This notion motivates the creation of TOMATO, benchmark specifically designed evaluate MFMs' visual temporal reasoning capabilities.
Visual temporal reasoning benchmarks. Several benchmarks have been developed to specifically evaluate models' visual temporal reasoning capabilities. For instance, VITATECS (Li et al. 2023) introduces six temporal reasoning tasks (e.g.. "A man is putting asks models to distinguish between the correct and counter factual caption. Addressing the lack of tasks diversity in VITATECS, TempCompass (Liu et al. 2024) expands tasks types include multiple-choice QA, yes/no QA, caption matching, and caption generation tasks. Aiming to cover a wide-range of femporal-sensitivn videos, MVBench (Li et al. 2024b) defines nine core temporal tasks with 20 subtasks, each of which cannot be solved using a single frame. Similarly. ReXTime (Chen al. (2224)) targets comprehensive temporal reasoning tasks and puts a special emphasis on cause and effect samples. However, despite these efforts, as illustrated in Figure I we observe that many questions in these benchmarks can be answered correctly using single, few, or out-of-ordes frames ($5), limiting their effectiveness to evaluate models' true visual temporal reasoning capabilities. To address these shortcomings, we introduce TOMATO, benchmark designed to provide a more rigorous evaluation of visual temporal reasoning.
tie or putting off tie?") and
3 BENCHMARKING PRINCIPLES FOR VISUAL TEMPORAL REASONING TASKS
In this section, we define three key principles and corresponding metrics for assessing how rigorously benchmark targets visual temporal reasoning rather than static image understanding.
3

---

Published as a conference paper at ICLR 2025
3.1 MULTI-FRAME GAIN
Key principle: A visual temporal reasoning task should require reasoning across multiple frames, making it impossible for models to solve the task using frame.
Requiring models to reason across multiple frames ensures that the tasks are distinguished from static image recognitions. This requirement aligns with prior works (Lei et al. 2022 Fang et al., 2024). which highlight that many video understanding tasks rely heavily static visual information.
To assess this principle, we define Multi Frame Gain K, which measures the relative performance gain from using a single frame m frames as input. When considering single frame, examine both settings: (1) a random frame and (2) a handpicked, highly informative frame specific to the question. The frames are uniformly sampled equal intervals from the videos. Using Acc(m frames) to denote the model's accuracy in solving benchmark tasks using m frames, and e a small positive constant, we define x as:
Ace(m frames)
Acc(1 frame) +
A lower A value indicates that the question can be more accurately answered using a single frame, while a higher value indicates a necessity to reason across multiple frames.
3.2 FRAME ORDER SENSITIVITY
Key principle: A visual temporal reasoning task, when given multiple frames, should enforce constraints on maintaining the correct order of frames.
Shuffling video frames disrupts the temporal information throughout the video, demonstrated in the prior works (Misra et al. 2016 Dorkenwald et al., [0022] Hao et al., 2022). a task is solvable by shuffled frames, time dependencies across frames absent, and reasoning along the time dimension unnecessary. which disqualifies the task from being temporal.
To quantify this principle, we introduce Frame Order Sensitivity т, which measures the relative performance gain from using the shuffled m frames to the ordered m frames. Using Acc(m frames) denote the model's accuracy in solving benchmark tasks using frames, and small positive constant, we define 7 as:
Acc(m frames)
T Acc(shuffled m frames) +
1
lower value indicates the question can be answered more accurately using out-of-order frames, while a higher value suggests stronger reliance on the original order of the frames.
3.3 FRAME INFORMATION DISPARITY
Key principle: visual temporal reasoning task, when given multiple frames, should ensure that each frame contributes relatively evenly to solving the task.
Even contribution suggests that no single frame provides disproportionately more information. Even in tasks involving sequential events, where the number of events aligns with the number of frames necessary to answer the question accurately, models should not achieve significantly higher accuracy by relying handpicked single frame over any random single frame. This principle aligns with prior works (Huang et al. 2018 Liu et al.) (202T) in which they emphasize the necessity of leverage temporal information from all frames to ensure an accurate and robust temporal video understanding.
To quantitatively evaluate this principle, we introduce Frame Information Disparity which measures the relative performance gain achieved by switching from a random single frame to handpicked single frame. Using Acc(m frames) to denote the model's 's accuracy in solving benchmark tasks using m frames, and a small positive constant, we define p as:
Acc(handpicked frame)
Acc(random-sampled frame) +
4

---

Published as a conference paper at ICLR 2025
A higher o value indicates that the question be more accurately answered by a handpicked single frame compared to a random single frame, while a lower value indicates a more even distribution of informativeness across the multiple frames. In other words, ideally, benchmark with perfectly even distribution of informativeness among all the frames should achieve a value of O.
4 TOMATO: A VISUAL TEMPORAL REASONING BENCHMARK
We introduce TOMATO, new visual temporal reasoning benchmark that satisfies all three afore- mentioned principles, addressing issues where tasks from existing benchmarks are relatively more solvable by single frame, rely less on the order of frames, and allow for more unevenly distributed information across the frames ($5). TOMATO comprises 1,484 carefully curated, human-amnotated questions spanning six visual temporal reasoning tasks, applied to 1,417 videos, including 805 self- recorded and -generated videos, that encompass cammnn-centric real-world, and simulated scenarios. In the following sections, we describe temporal tasks in TOMATO ($4.I), video collection ($4.2). and question annotation
The main statistics of TOMATO are presented Table 9
4.1 TEMPORAL TASKS IN TOMATO
We introduce six visual temporal reasoning tasks, each them requiring multi-frame visual temporal reasoning: (1) Rotation: Determine the rotational direction of of the subject; (2) Direction: Identify the direction of the subject's movement; (3) Velocity & Frequency: Detect changes in the subject's movement speed variations in the frequency of repeated actions; (4) Shape & Trend: Analyze the
subject's s trajectory, such as the shape or general trend of its movement; (5) Visual Cues: Discern key visual signals to determine the sequence or timing of actions without audio; and (6) Action Count: Count how many times a specific action has been performed. Examples are provided in T
To ensure comprehensive coverage across various scenarios, we categorized each video into one of three demonstration categories: (1) Human-centric: Involving human interactions, where actions or intentions are observed; (2) Real-world: Focuses on actions involving objects various real-world scenes; and (3) Simulated: Depicting simplified, simulated environments representing temporal actions. The distribution of demonstration categories across each task is shown in Figure in EE2
4.2 VIDEO COLLECTION
TOMATO features a diverse range of videos from three distinct sources: YouTube, existing video datasets, and self- recordedd and -enerratdd benchmark-specte videos. To enhance diversity, we collect and create videos of three scenarios: real-world, human-centric, and simulated. Additionally, we edit videos to incorporate counterfactual scenes, composite motions, and zoomed-in views, aiming to investigate the impact of these characteristies on the performance of MFMs (66.)). License information for all videos is detailed in $
We chose YouTube as our primary source to ensure diverse and representative real-world scenarios our benchmark. Given specific visual temporal reasoning task, we task human annotators with searching for YouTube videos that best represented the corresponding task definition ($4.)) The selected videos cover a wide array of topics, including science experiments, outdoor activities, educational tutorials, and artistic performances. To prevent models from relying on commonsense knowledge, such as assuming the second hand of a watch rotates clockwise, we edit videos to create counterfactual scenarios. These edits, including reversing, concatenating, adjusting speed, and mirroring, ensure that models must fully analyze video content to answer questions correctly. In total, we collect 171 source videos before editing and 398 videos after editing.
YouTube videos.
Existing video datasets. It is essential for Al agents to comprehend the human world through detecting the point of change in actions (Visual Cues) and discern the number of actions occurred (Action Count). Therefore, to address the lack of diversity in YouTube videos for these reasoning types, we incorporate four established datasets, each contributing to unique domains and scenarios: (1) Music-AVQA (Li et al. 2022) featuring inlti-insttument performances; (2) CLEVRER (Yi et al.) 2020) presenting synthetic scenes with multiple moving objects; (3) TGIF-QA (Jang et al. 2017)
focusing action counting in varied scenes; and (4) Perception Test (Päträucean al.
2023
5

---

Published as a conference paper at ICLR 2025
addressing indoor action counting tasks, such as clapping and object moving. For Music-AVQA and CLEVER, we re-amnotate and re- we-write questions to specifically inquire the temporal contexts of these videos ( $..3). In total, we incorporate 214 videos: 70 from Musi 50 from CLEVRER, 50 from TGIF-QA AA and 44 from Perception Test.
Self-recorded and -generated benchmark-speciffcc videos. For an Al agent truly comprehend the human world dynamics, it is essential for it to understand scenarios where a person is actively interacting with the agent, treating the agent as the "other" participant of an interaction. However, YouTube videos and existing datasets lack such human-centric interactive scenes for various reasoning types in TOMATO ($..I). To address this limitation, we record videos such as a human conveying "turn-around" by twirling their wrist while maintaining a pointing gesture, or drawing a shape in the air using their arm. Furthermore, to aid data collection and study the effect of simulated videos, generate more simple, abstract representations of both real-world objects and human. Thus, expand our dataset by (1) creating simulated videos using Keynote featuring objects moving in different patterns; and (2) generating 3D human model videos through the VIBE (Kocabas al.) 2019) and SMPL (Bogo et al., (016)) frameworks. To study the effect of different video characteristics. we edit videos to incorporate counter-factus scenes, composite motions, and zoomed-in views. In total, we create 298 videos before editing and 805 videos after editing, spanning all six tasks.
4.3 QUESTION-ANSWER ANNOTATION AND QUALITY CHECK.
QA annotation. To address the limitation that tasks from existing benchmarks are solvable by single, few, out-of-order frames (discussed further in $ TOMATO focuses crafting questions that require reasoning about transitions across all frames. Our annotation process varied by video sources. For YouTube videos, self-created benchmark-speciffc videos, and their edits, human annotators composed QA pairs targeting specific temporal reasoning tasks ($4.1)
Quality check. To ensure high-quality QA annotations, we implemented a three-stage process: initial annotation by annotators, followed by cross-checking and verification among annotators, and finally, collective resolution of disagreements with a final review (see $ This rigorous approach ensured consistency and accuracy across all annotated QAs.
COMPARISONS AMONG VISUAL TEMPORAL REASONING BENCHMARKS
In 3 we defined three key principles with corresponding metrics assess how effectively benchmark addresses visual temporal reasoning. Using these metrics, we compare TOMATO with four recent visual temporal reasoning benchmarks: VITATECS (Li al., (023). MVBench (Li) et al. 2024b), TempCompass (Liu et al. 2024), and ReXTime (Chen et al. 2024a). To conduct this comparison, we randomly sampled approximately 200 QA from these benchmarks, using two state-of-the-art MFMs: GPT-4o (OpenAl) 2024) and Qwen2-VL--72B (Wang et al.) (024a). For metrics that requires handpicking frames, we present annotators with the full video and corresponding question, and ask them to select the most informative frame for each benchmark. For metrics requiring multiple frames, we set m = 16, as our study across m = 1, 8, 16, 32 demonstrates that 16 frames provide sufficient window for effective analysis E.
5.1 MULTI-FRAME GAIN
We present results on Multi-frame Gain for both (1) random frame (Table and (2) handpicked, highly informative frame (Table As shown the two tables, TOMATO achieves a significantly higher value in using both random single frame and handpicked single frame. This significant relative performance gain from the single frame input setting to the 16 frames input shows the necessity in tasks to reason using multiple frames. In comparison of the two tables, observe an expected decrease in the x value in using handpicked single frame compared to random single frame existing benchmarks, indicating the relative ease in answering the benchmark questions
https://support apple. com/keynote
'Sampling varies slightly: TempCompass: 200 samples. MVBench: 204 samples (excluding overlapping reasoning types). VITATECS: 203 samples. ReXTime: 180 samples from QVHighlight's validation set.

---

Published as a conference paper at ICLR 2025
Table Performance using single-frame and 16- 16-frame inputs.
VITATECS
MVBench
TempCompass
ReXTime
TOMATO
Frames
16
1
16
K
16
K
16
16
4P
70.0
68.2
26.1
47.1
62.3
32.3
52.5
71.5
36.2
61.7
78.3
26.9
21.2
37.7
78.0
Qwen2-VT.
71.4
86.2
20.7
47.1
63.2
34.4
50.0
79.0
58.0
63.9
81.7
27.8
20.6
37.9
84.0
Average
70.7
87.2
23.4
47.1
62.7
33.3
51.3
75.3
47.1
62.8
80.0
27.4
20.9
37.8
81.0
Table 2: Performance using handpicked single-frame (denoted by 1(H) and 16-frame inputs.
VITATECS
MVBench
TempCompass
ReXTime
TOMATO (200)
I[H]
16
K
1[H]]
K
1[1]]
K
1[1]]
16
K
1[H]
K
GPT-4o
84.2
88.2
4.7
59.8
62.3
4.1
64.5
71.5
10.9
86.7
78.3
-9.7
21.5
37.0
72.1
Qwen2-VL
Average
86.0
87.2
1.5
58.8
62.7
6.7
64.0
75.3
17.6
86.1
80.0
-7.1
22.8
37.8
66.3
Table 3: Performance using shuffled 16-frame (denoted by 16[S] and 16-frame inputs.
VITATECS
MVBench
TempCompass
ReXTime
TOMATO
Frames
16/S]
161S]
r
16/81
r
GPT-4o
85.7
88.2
2.9
59.8
62.3
4.1
59.0
71.5
21.2
77.8
78.3
25.8
37.7
46.2
Qwen2-VI.
83.3
86.2
3,6
58.8
63.2
7.5
64.0
79.0
23.4
81.7
81.7
31.1
37.9
21.9
Average
75.3
Table 4: Performance using single frame and handpicked single-frame (denoted by 1(H) inputs.
VITATECS
MVBench
TempCompass
ReXTime
TOMATO (200)
# Frames
I[I]]
P4
I[I]]
p
I[I]]
1[I]]
P
GPT-4o
Qwen2. VI.
71.4
87.7
22.7
47.1
57.8
22.9
50.0
63.5
27.0
63.9
85.6
33.9
23.0
24.0
4.1
Average
70.7
86.0
21.6
47.1
58.8
25.0
5133
64.0
24.9
62.8
86.1
37.2
21.8
22.8
4.6
using handpicked single frame. Interestingly, the values are negative in the handpicked single frame setting ReXTime; this negative performance gain in using more frames might stem from noise introduced by additional frames.
5.2 FRAME ORDER SENSITIVITY
In our shuffled 16 frames setting, apply random shuffling on the ordered 16 frames ensure that the same set of frames are used in both settings. As shown Table TOMATO achieves signifi- cantly higher т, demonstrating that benchmark imposes stricter requirement maintaining the order of the frames to accurately answer the question.
5.3 FRAME INFORMATION DISPARITY
As shown in Table 4 the performance gain from random single frame to handpicked single frame is the lowest on TOMATO, indicating relatively more consistent informativeness across all frames compared other existing benchmarks.
6 EVALUATING VISUAL TEMPORAL REASONING IN ADVANCED MFMs
In introducing TOMATO, present comprehensive evaluation of 31 MFMs, including 10 proprietary models and 21 open-source models, to assess their visual temporal reasoning capabilities. In the following sections, detail the experimental setup ($6.1). evaluation results ($..2) and
7

---

Published as a conference paper at ICLR 2025
Table 5: Evaluation results on TOMATO. Unless otherwise specified, each model processes 16 frames. Bold and underlined numbers indicate the best and second-best performance in each category. respectively. See Table for detailed model configurations. Models that can process video directly. :; Models that can only process eight frames. Details for these categories are shown in $4.1
Velocity Frequency (210)
Shape & Trend (223)
Visual Cues (70)
Action Count (292)
Model
Rotation
Direction (403)
All
(286)
(1,484)
Buxelines
Human (Videos)
93.5
95.4
94.1
100.0
95.0
93.6
95.2
Human (Frames)
60.9
93.9
85.3
91.7
60.0
70.2
79.7
Random (GPT-4o)
16.8
17.4
32,9
23.1
Random Choice (42)
22.0
17.6
22.9
17.9
18.6
13.4
18.5
Frequent
Proprictary Multimodal Foundation Models (MFMss
GPT-4o
24.5
45.2
31.9
42.6
58.6
36.0
37,7
Gemini pro
25.9
40.7
35.2
41.3
37.1
36.1
Gemini Flash Exp GPT-to-mini Clande Opus
48.6
33.7
19.9
32.8
28.1
29.6
41.4
28.8
28.8
31.1
23.3
32.4
27.8
28.6
29.5
28.2
Claude 3.5 Sonnet
27.3
25.6
26.2
27.8
32.9
31.2
27.8
Gemini 1,5 Flash
22.4
30.3
31.0
26.9
30.0
27.7
27.8
Clande Haiku
25.9
19.4
31.0
33.2
30.0
26.4
26.2
Reka. Flash
19.6
21.5
32,9
20.5
Reka Core
Open-Source Multimodal Foundation Models (MFMs)
Qwen2-VL-
26.9
38.2
43.8
36.3
48.6
42.8
37,9
Qwen2
37.1
31.5
Video C 14B
32.2
26. I
29.5
27.4
44.3
35.6
30.7
InternVL
23.4
32.0
15.7
41.7
34,3
29.0
LLaVA-Vidko-22B
18.9
30.0
16.7
32.3
50.0
34.6
28.2
Video-CCAM-v1..
22.4
25.6
25.7
26.0
34.3
33.6
27.0
InderViideo 2 8B
31.8
24.1
23.3
25.6
35.7
25.0
26.4
Qwen2-VL-28
26.9
22,8
31.4
23,8
32.9
25.7
26.0
Video-CCAM-vl.
21.7
24.3
19.0
27.4
32.9
31.5
25.3
LLaVA-Vidoo-BB
20.1
29.1
41.4
24.9
VILA 138
29.0
19,6
19.0
27.4
32.9
24.7
LLaVA Video (B.eo-BB..e Video
15.4
24.1
19.5
31.4
38.6
25.7
23,9
Video LLLVA 7B
29.4
17.9
27.1
23.3
34,3
23.6
VideoLLaMA
14.3
24.6
22.4
26.5
27.1
23.5
InternVL. 2 268
18.5
29.3
10.5
31.4
11.4
25.7
23.3
LLaVA-NeXT- M.VV--NcXT-Video--28
20.6
26.3
24.2
30,0
22.7
2
17.1
25.1
9.0
28.7
31.4
22.9
21.7
AuromaCap-7B-Va
19.6
17.1
26.7
20.6
25.7
17.1
20.6
VideoLLaMA 2 7B
10.1
22.8
15.7
18.8
31.4
19.5
18.5
a multi-faceted analysis ($6.1 6.3 considering factors such as model architectures, reasoning type correlations, frame counts, and video characteristics.
6.1 EXPERIMENTAL SETUP
Models. We evaluate diverse set of general-purposs MFMs. For open-source MFMs, test:
VILA (Lin et al. 2023b), Intern VI. (Chen al. 2024b) Phi 3.5 Vision (Team] 2024), Video L.LVV
Lin et al.) 2023a). Intern Video WWang et al. 2024bb, LLa VA-NeXTT Video Zhang et al.) 2024a). Li al. 2024a). VideoLLaMA2 (Cheng et al. 2024), Qwen2 VL Wang et al.
LLaVA-One
2024a VideoCCAM (Fei et al. 2004)) AuroraCap (Chai et al. 2024), and LLaVA Video ZZang
et al. 2024b). We also evaluate the following proprietary MFMs: GPT-4o (OpenAl 2024), Claude
(Anthropic] [2024 b), Reka (Reka 2024).. and Gemini (Gemini 2024). For all models, provide
generation configuration in Table
Baselines. We include text-only baseline, denoted Random (GPT-4o). in which prompt GPT- 4o to guess the answer without access to the videos (prompt in (D3)). Additionally, we report results for Random Guess and Frequent Guess baselines. Furthermore, evaluate human performance
8

---

Published as a conference paper at ICLR 2025
TOMATO, reporting results for humans using video input, i.e., Human (Video)) and 16 frames as input, ie,, Human (Frames). Annotator biographies are provided in Table 10 in GG.I
6.2 EXPERIMENTAL RESULTS
We provide quantitative results on TOMATO for all models in Table [ To better understand where models fail, select set of representative models (OpenAI 2024 Anthropic [024b] Wang et al.
2024a Fei et al.] 2024; Chen et al. (024b) and present examples of failure cases in $I J SK
M and
Analysis of video scenarios (Counterfactual, zoomed-in, and first-person)
Widespread difficulty in visual temporal reasoning. Our evaluation (Table 5 underscores the significant challenges of TOMATO across all tested models. The leading open-source model, Sive2-VLL-221g achieves 37.9% accuracy, slightly outperforming GPT-4o's 's 37.7% 7%. However, this still leaves a substantial 57.3% performance gap compared to human accuracy of 95.2%. While this result demonstrates the competitive potential of open-source models in video understanding, many still fall below 30.0%, indicating weaknesses in TOMATO despite their decent performance
in existing benchmarks (Päträucean et al. 2023 Li et al.
2024b
Fu et al. 2024).
6.3 ANALYSIS
Models lack the basic ability to interpret frames as continuous sequence. While MFMs demonstrate remarkable performance in understanding sequential events in videos (Chen et al. 2024 our benchmark exposes a more fundamental limitation: models struggle reason across multiple time steps and to interpret the frames as a continuous sequence. As shown in error case I1.2.1 GPT-4o correctly generates captions for each consecutive change in the moon's movement, showcasing its ability to reason at individual time steps. However, it fails to infer based on the captions that the overall sequence represents a clockwise rotation. This issue is not limited rotation similar shortcomings are observed in direction ($ ($3333). action count ($N.2.5). ) etc..
($1.1.1 I
Models fail to truthfully leverage the visual input while being over-reliant on common sense. In our evaluations, despite explicit instructions ($D.I) to rely on the visual input rather than common sense, we find that models frequently hallucinate based on information from single frames rather than utilizing true visual reasoning. For instance, in error case $K.2.4 GPT-4o incorrectly concludes that an object is dropping due to the presence of motion blur in some of the frames. However, this is a reversed video where the object actually only moving upward- a conclusion that can only be reached if the video modality where truthfully utilized. A similar instance of the limitation occurs in error case $J.1.2] where GPT-4o likely assumes the person raises their hand first to reach the posture depicted in the first 8 single frames. In reality, the person's 's hand remains relatively stationary throughout these frames, but the model fails to make accurate visual comparisons across these frames.
Models are highly susceptible to noisy information in the input. As demonstrated in error case (I.3.3] where a block moves downwards, models are especially vulnerable to noisy information, such as misleading text on the block. In particular. while GPT-4o correctly describes the block's downward motion based on its relative position to the screen, it incorrectly concludes that the block is moving upward, likely influenced by the false information presented in the text written on the block. Similarly, in error case (N2.5) butterflysshpped laser spot is moving in triangle shape. However, the unique butterfly shape likely causes the models to lose focus on the trajectory of the laser spot, and resort to random guessing in their conclusions.
Explicitly incorporating time-aware positional encoding can likely enhance visual temporal reasoning. The Qwen2-VL family (Wang al. 2024a), which consistently achieves the highest scores across multiple categories, leverages Multimodal Rotary Positional Encoding (M-RoPE) within its visual encoders. M-RoPE explicitly encodes temporal information into visual tokens, allowing models to retain critical temporal context throughout the entire pipeline. In contrast, models without such temporal-awer awwre positional encoding scheme appear to lose critical time-related context after visual encoding, such as Causal Cross-Atter Aross-Attentinn Masks (CCAM) used in vision- lingua-language alignment (Fei et al.] 2024). Unmasked Video Token Reconstruction during training (Wang et al. (2024b), and Spatial-Temporra Convolution (STC) connector applied after frozen visual encoder (Cheng al.

---

Published as a conference paper at ICLR 2025
LLaNONAT cmoebe Na
50
A
21.9%
0.8
Inkyetited
44.6
Stmutane Iluman
Accuracy
(%)
40
37.6
7.8%
0.6
30
25.7 27.7
0.4-
20-
10-
1
16
24
32
GPT-4o
Qwen2-VL-72B
(a)
(b)
(c)
Figure 2: (a) Human and models' performance on TOMATO with different number of frames. (b) Real human and simulated human's performance on TOMATO. (c) Example of real human vs. simulated human.
2024). Performance of Qwen2- VL on TOMATO suggests that explicitly incorporating temporal- aware positional encoding like M-RoPE is likely essential for enhancing MFMs' visual temporal reasoning capabilities. Future improvements in open-source models could benefit from adopting similar strategies to close the gap between the open-source and proprietary models.
Existing models are limited understanding events that are interpretable in S 8 frames. We assess four MFMs' performance across different number of frames on TOMATO, as illustrated in Figure 2 (a). The consistent improvement in accuracy of human performance suggests that our benchmark rely the additional frames convey more temporal information. Notably, although models exhibit performance increase transitioning from frame to 8 frames, the performance plateaus beyond this point. This suggests that even if models able reason the transitions between the frames before 8 frames, they cannot utilize the additional temporal information obtained by the added frames. Therefore, we conclude that the overall performance in their visual temporal reasoning capabilities, and there is still room for improvements in MFMs' ability to leverage the additional information on the frame transitions introduced by added frames.
the four MFMs remain suboptimal
Models perform better on simulated human than real human scenarios. To investigate the extent to which cleaner, more abstract representation of video content (Figure 2 (c)) influences models' temporal reasoning abilities, evaluate two leading MFMs. Specifically, this evaluation contrasts real human scenarios and their corresponding simulated counterparts, across five reasoning types (i.e., action count, direction, rotation, shape & trend, and velocity & frequency), covering the same 101 QA pairs in both scenarios. As a result, GPT-4o marks a noticeable improvement of 21.9% from real human to simulated human scenarios, underscoring the potential of enhancing models' temporal reasoning capabilities in video understanding through semantic video abstraction. Conversely, Qwen2-VL-72B displays modest increase of 7.8% transitioning from real human to simulated human. While it marginally outperforms GPT-4o in the overall evaluation on TOMATO, its visual temporal reasoning capabilities simulated scenarios still show room for improvement (Figure 2 (b)). Future work can target further enhancing models' temporal reasoning abilities for real human videos by exploring their generalization capabilities through leveraging automatically generated simulated 3D human motion data (Guo et al. 2022).
7 CONCLUSION
Existing benchmarks likely overestimate the true visual temporal reasoning capabilities of MFMs. In response, we establish three key principles and corresponding metrics to systematically examine visual temporal reasoning tasks. Building upon these principles, we introduce TOMATO, TO, novel video understanding benchmark to rigorously assess MFMs' true visual temporal reasoning capabilities. Besides revealing previously underestimated human-model performance gap, our comprehensive evaluation highlights critical limitation: MFMs fail to interpret videos as continuous sequences, instead resorting to understanding isolated frames, which severely undermines their visual temporal reasoning capabilities. This work sheds the light for developing Al systems capable of comprehending changing scenes in real life through the video modality.
10

---

Published as a conference paper at ICLR 2025
ACKNOWLEDGMENTS
We are grateful for compute credits provided by Google through the TRC program. We thank Professor Alex Wong, Professor Rex Ying. and Professor Dylan McKay for their valuable feedback. We also appreciate the insightful discussions and support from members of the Yale NLP Lab, Yale Inquisitive Robotics Lab, Yale Graph and Geometric Learning Lab, and our peers in the Yale two-year MSCS program.
REFERENCES
Anthropic. Introducing the next generation of claude, 2024a. URL https::/www.aatttrppcc.omm
Anthropic. Claude 3.5 sonnet, 2024b.
URL httos://ww.aattiroiic.ccmmm//
claude-3-5-some
Federica Bogo, Angjoo Kanazawa, Christoph Lassner, Peter V. Gehler, Javier Romero, and Michael J. Black. Keep it SMPL: automatic estimation of 3d human pose and shape from single image.
CoRR, abs/1 007.66772222 081228, 2016. htt.:/tt:://arav..or//aa////////
000007.88120 08128
Wenhao Chai, Enxin Song, Yilun Du, Chenlin Meng, Vashisht Madhavan, Omer Bar- Tal, Jeng-Neng Hwang, Saining Xie, and Christopher D. Manning. Auroracap: Efficient, performant video detailed captioning and new benchmark, 2024. URL /ttpss////aax http:iii.wrr//////////e 2410. 2410.03051
Jr-Jen Chen, Yu-Chien Liao, Hsi-Che Lin, Yu-Chu Yu, Yen-Chun Chen, and Yu-Chiang Frank Wang. Rextime: A benchmark suite for ceanninn-acros--iii in videos, 2024a. URL https //arxiv.org/abs//4466. 19392.
Zhe Chen, Weiyun Wang, Hao Tian, Shenglong Ye. Zhangwei Gao, Erfei Cui, Wenwen Tong, Kongzhi Hu, Jiapeng Luo, Zheng Ma, et al. How far are we to gpt-4v? closing the gap to commercial multimodal models with open-source suites, arXiv preprint arXiv:2404.168:
2024b.
Zesen Cheng, Sicong Leng, Hang Zhang, Yifei Xin, Xin Li, Guanzheng Chen, Yongxin Zhu, Wenqi Zhang, Ziyang Luo, Deli Zhao, and Lidong Bing. Videollama 2: Advancing spatial-temporal modeling and audio understanding in video-Ilms, 2024. URL https:///axxiv.or//bbae 07476
Michael Dorkenwald, Fanyi Xiao, Biagio Brattoli, Joseph Tighe, and Davide Modolo. Sevrl: Shuffled contrastive video representation learning, 2022. URL https:///axxvv wwrg/abs/2205.2 11710
Xinyu Fang, Kangrui Mao, Haodong Duan, Xiangyu Zhao, Yining Li, Dahua Lin, and Kai Chen. Mmbench-video: long-form multi-shot benchmark for holistic video understanding, 2024. URL
http:///aaxxv..o//abb////
14515
Jiajun Fei, Dian Li, Zhidong Deng, Zekun Wang, Gang Liu, and Hui Wang. Video-ccam: Enhancing video-language understanding with causal cross-attention masks for short and long videos, 2024.
URL /tR.https://axxiv.com ort.:/ss.rgr//////////o///.o/ss 2408. 14023
Chaoyou Fu, Yuhan Dai, Yongdong Luo, Lei Li, Shuhuai Ren, Renrui Zhang, Zihan Wang, Chenyu Zhou, Yunhang Shen, Mengdan Zhang, Peixian Chen, Yanwei Li, Shaohui Lin, Sirui Zhao, Ke Li, Tong Xu, Xiawu Zheng, Enhong Chen, Rongrong Ji, and Xing Sun. Video-mme: The first- ever comprehensive evaluation benchmark of multi-modal lmms in video analysis, 2024. URL https://arxiv.. http://aarxiv.oog/bbbbbb/ wttp://////...//22222/ 2105.21075
Gemini. Gemini 1.5: Unlocking multimodal understanding across millions of tokens of context, 2024.
Raghav Goyal, Samira Ebrahimi Kahou, Vincent Michalski, Joanna Materzynska, Susanne Westphal, Heuna Kim, Valentin Haenel, Ingo Fründ, Peter Yianilos, Moritz Mueller- Freitar-Freitag, Florian Hoppe, Christian Thurau, Ingo Bax, and Roland Memisevic. The "something something" video database for learning and evaluating visual common sense. 2017 IEEE International Conference Computer Vision (ICCV), pp. 5843-5851, 2017. URL https: //api. semanticscholar. www.:orgg/oooooo org/CorpusiD; 834612

---

Published as a conference paper at ICLR 2025
Chuan Guo, Shihao Zou, Xinxin Zuo, Sen Wang. Wei Ji, Xingyu Li, and Li Cheng. Generating diverse and natural 3d human motions from text. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 5152-5161, 2022.
Jiachang Hao, Haifeng Sun, Pengfei Ren, Jingyu Wang. Qi Qi, and Jianxin Liao. Can shuffing video benefit temporal bias problem: novel training framework for temporal grounding, 2022. URL
https://arx...org/bbs////
14698
Xuehai He, Weixi Feng, Kaizhi Zheng, Yujie Lu, Wanrong Zhu, Jiachen Li, Yue Fan, Jianfeng Wang. Linjie Li, Zhengyuan Yang. Kevin Lin, William Yang Wang, Lijuan Wang, and Xin Eric Wang. Mmworld: Towards multi-discipline multi-faceted world model evaluation videos, 2024. URL
https://arxiv.org//bba/ 0842/
Fabian Caba Heilbron, Victor Escorcia, Bernard Ghanem, and Juan Carlos Niebles, Activitynet: A large-scale video benchmark for human activity understanding. In 2015 IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pp. 961-970, 2015. doi: 10.1109/CVPR.2055
7298698.
De-An Huang, Vignesh Ramanathan, Dhruv Mahajan, Lorenzo Torresani, Manohar Paluri, Li Fei-Fei, and Juan Carlos Niebles. What makes video a video: Analyzing temporal information in video understanding models and datasets. In Proceedings of the IEEE Conference Computer Vision and Pattern Recognition (CVPR), June 2018.
Yunseok Jang, Yale Song, Youngjae Yu, Youngjin Kim, and Gunhee Kim. Tgif-qa: Toward spatio- temporal reasoning in visual question answering, 2017. URL https:://axxv..or///bbat 04497
Daniel Kahneman, Anne Treisman, and Brian Gibbs. The reviewing of object files: Object- specific integration of information. Cognitive Psychology, 24(2):15--19,, 1992. ISSN 0010-0285. doi: https://bbooogg////...ooooo00021 Scienee//rricee//////00122999999
http::/::://ww http:://wwwwwciieeeerreettttttt. com/
Muhammed Kocabas, Nikos Athanasiou, and Michael J. Black. VIBE: video inference for human body pose and shape estimation. CoRR, abs/1912.006657 2019. URL http://arxiv.org/ wrtp:xiivi..ov//baaaa / 256566
Jie Lei, Tamara L. Berg, and Mohit Bansal. Revealing single frame bias for video-and-language learning, 2022. URL htt.h:ttss://rrttt wttp:///w..rggb///2eee/// 03428
Bo Li, Yuanhan Zhang, Dong Guo, Renrui Zhang, Feng Li, Hao Zhang, Kaichen Zhang, Yanwei Li, Ziwei Liu, and Chunyuan Li. Llava-onevision: Easy visual task transfer, 2024a. URL
https://arxias htt.:/org/bbb///// 2408/24488010 03326
Guangyao Li, Yake Wei, Yapeng Tian, Chenliang Xu, Ji-Rong Wen, and Di Hu. Learning to answer questions dynamic audio-visual scenarios, 2022. URL httph:t/tts:xi...tttt org/ abw./200..11.42 14072
Kunchang Li, Yali Wang. Yinan He, Yizhuo Li, Yi Wang, Yi Liu, Zun Wang, Jilan Xu, Guo Chen, Ping Luo, Limin Wang, and Yu Qiao. Mvbench: A comprchensive multi-modal video understanding benchmark, 2024b.
httttt:ss///iiiii org/ 2311/2311.117 17005
Shicheng Li, Lei Li, Shuhuai Ren, Yuanxin Liu, Yi Liu, Rundong Gao, Xu Sun, and Lu Hou. Vitatecs: A diagmostic dataset for temporal concept understanding of video-language models, 2023. URL 17404
Xinhao Li, Zhenpeng Huang, Jing Wang, Kunchang Li, and Limin Wang. Videoeval: Comprehensive benchmark suite for low-cost evaluation of video foundation model, 2024c. https:/ts:///
org/abs/2407. 204224477796611
Yuncheng Li. Yale Song, Liangliang Cao, Joel Tetreault, Larry Goldberg, Alejandro Jaimes, and Jiebo Luo. Tgif: A new dataset and benchmark on animated gif description, 2016. URL https
htwpxiv.orr//b//160000
12

---

Published as a conference paper at ICLR 2025
Yunxin Li, Xinyu Chen, Baotian Hu, Longyue Wang, Haoyuan Shi, and Min Zhang. Videovista: A versatile benchmark for video understanding and reasoning, 2024d. URL httpss://aariv..rrg. abs/2406. 11303
Bin Lin, Yang Ye, Bin Zhu, Jiaxi Cui. Munan Ning, Peng Jin, and Li Yuan. Video-Ilava: Learning united visual representation by alignment before projection, 2023a. URL https://aaxxi org abs/2311. 10122
Lin, Hongxu Yin, Wei Ping, Yao Lu, Pavlo Molchanov, Andrew Tao, Huizi Mao, Jan Kautz, Mohammad Shoeybi, and Song Han. Vila: On pre-training for visual language models, 2023b.
Xin Liu, Silvia L. Pintea, Fatemeh Karimi Nejadasl, Olaf Booij, and Jan C. van Gemert. No frame left behind: Full video action recognition, 2021. URL httthtttt//axiiiii wttp://www...rr/////////... 15395
Yuanxin Liu, Shicheng Li, Yi Liu, Yuxiang Wang, Shuhuai Ren, Lei Li, Sishuo Chen, Xu Sun, and Lu Hou. Tempcompass: Do video Ilms really understand videos?, 2024. URL https warxiv.org//abll hwtgg//a/////....../ 2403. 00476
Ishan Misra, C. Lawrence Zitnick, and Martial Hebert. Shuffle and learn: Unsupervised learning using temporal order verification, 2016. URL https://arxii....tmtmmmm org/ hwwggg/a////ee 110.. 08561
Arsha Nagrani, Mingda Zhang, Ramin Mehran, Rachel Hormung, Nitesh Bharadwaj Gundavarapu, Nilpa Jha, Austin Myers, Xingyi Zhou, Boqing Gong, Cordelia Schmid, Mikhail Sirotenko, Yukun Zhu, and Tobias Weyand. Neptune: The long orbit to benchmarking long video understanding. 2024.
Munan Ning, Bin Zhu, Yujia Xie, Bin Lin, Jiaxi Cui, Lu Yuan, Dongdong Chen, and Li Yuan. Video-bench: A comprehensive benchmark and toolkit for evaluating video-based large language models, 2023. htttp:://arxlv.org/bb//////
OpenAl. Hello gpt-4o, 2024. URL htt.:ttpse//ggeeai.cc/////////// gpptooo
Viorica Päträuccan. Lucas Smaira, Ankush Gupta, Adrià Recasens Continente, Larisa Markeeva, Dylan Banarse, Skanda Koppula, Joseph Heyward, Mateusz Malinowski, Yi Yang, Carl Doersch, Tatiana Matejovicova, Yury Sulsky, Antoine Miech, Alex Frechette, Hanna Klimczak, Raphael Koster, Junlin Zhang, Stephanie Winkler, Yusuf Aytar, Simon Osindero, Dima Damen, Andrew Zisserman, and João Carreira. Perception test: A diagnostic benchmark for multimodal video models, 2023. URL https:://arxww.or//bbbaa 2305 13786
Ruchit Rawal, Khalid Saifullah, Ronen Basri, David Jacobs, Gowthami Somepalli, and Tom Goldstein. Cinepile: long video question answering dataset and benchmark, 2024. URL /https:///xxiiv
org//bs/2ae 088133
Reka. Reka core, flash, and edge: A series of powerful multimodal language models, 2024. URL https://arxiv. www.://gg//////////oooooo 12387
Gunnar A. Sigurdsson, Abhinav Gupta, Cordelia Schmid, Ali Farhadi, and Karteek Alahari. Actor and observer: Joint modeling of first and third-person videos, 2018. httppttts://arxiw.rrg
0bs/1804.00 09627
Zhi Rui Tam, Cheng-Kuang Wu, Yi-Lin Tsai, Chieh- Yen Lin, Hung yi Lee, and Yun-Nung Chen. Let speak freely? study the impact of format restrictions performance of large language models, 2024. URL htttp:://aaxxii http://axxxvv..o///bb//s// www.oogg//bbbbeeeeee/ 2408. 02442
Team. Phi-3 techniccl report: A highly capable language model locally on your phone, 2024. URL http://aaaxiv..og/abb///////
14219
Peng Wang. Shuai Bai, Sinan Tan, Shijie Wang, Zhihao Fan, Jinze Bai, Keqin Chen, Xuejing Liu, Jialin Wang. Wenbin Ge, Yang Fan, Kai Dang, Mengfei Du, Xuancheng Ren, Rui Men, Dayiheng Liu, Chang Zhou, Jingren Zhou, and Junyang Lin. Qwen2-vl: Enhancing lasion-language model's perception of the world at any resolution, 2024a.
http://aaxxi....r/////./99..01. 12191
13

---

Published as a conference paper at ICLR 2025
Yi Wang, Kunchang Li, Xinhao Li, Jiashuo Yu, Yinan He, Chenting Wang, Guo Chen, Baoqi Pei, Ziang Yan, Rongkun Zheng, Jilan Xu, Zun Wang, Yansong Shi, Tianxiang Jiang, Songze Li, Hongjie Zhang, Yifei Huang, Yu Qiao, Yali Wang, and Limin Wang. Intemvideo2: Scaling foundation models for multimodal video understanding, 2024b.
Kexin Yi, Chuang Gan, Yunzhu Li, Pushmeet Kohli, Jiajun Wu, Antonio Torralba, and Joshua B. Tenenbaum. Clevrer: Collision events for video representation and reasoning, 2020. URL
https://arxxv.orr//aat
.0142
Hongjie Zhang, Liu, Lu Dong, Yifei Huang, Zhen-Hua Ling, Yali Wang, Limin Wang, and Qiao. Movqa: A benchmark of versatile wanwwerrng for long-form movie understanding, 2023.
URL /tattps//arxiv....
04817
Yuanhan Zhang, Bo Li, haotian Liu, Yong jae Lee, Liangke Gui, Di Fu, Jiashi Feng, Ziwei Liu, and Chunyuan Li. Llava-next: A strong zero-shot video understanding model, April 2024a. URL
https://llawa-vl.. github. io/blog///////oo -024-04-2220 30- 10--000-111000 11ava daxva-next-videeee
Yuanhan Zhang. Jinming Wu, Wei Li, Bo Li, Zejun Ma. Ziwei Liu, and Chunyuan Li. Video instruction tuning with synthetic data, 2024b. URL httth: orgg////24// 20tpp//2210.01131
Junjie Zhou, Yan Shu, Bo Zhao, Boya Wu, Shitao Xiao, Xi Yang, Yongping Xiong, Bo Zhang, Tiejun Huang, and Zheng Liu. Mlvu: A comprehensive benchmark for multi-task long video understanding, 2024. htttp:://aaii..rrrrrr/tt
04264
14

---

Published as a conference paper at ICLR 2025
CONTENTS
Tasks Examples of TOMATO
19
Scenario Analysis
20
Experiment Setup
21
C.1 Model Configuration
21
Implementation Details for Model Inference
Prompts
22
D.I Evaluation Prompt
22
D.2 Answer Extracting Prompt
22
D.3 Video-Free Random Guess Prompt
23
Data Analysis
24
Statistics
24
Video Source Distribution
25
E.3
Performance Comparisons across Different Number of Frames
................................
26
Model Performance with < 8 Frames
26
Video Duration Distribution
27
E,6
Response Length Distribution
28
icense Information.
30
Annotation Details
G.1 Annotator Biographies
31
Pilot Annotation
G 3 Full-Scale Annotation
31
H Data Annotation Platform
32
Common Failure Cases Rotation
33
Human
33
................................
Example
33
1.2
Object
................................
34
I.2.1
Example
34
1.2.2
Example 2
35
1.2.3
Example 3
37
1.2.4
Example 4
38
1.2.5
Example 5
39
1.2.6
Example 6
40
15

---

Published as a conference paper at ICLR 2025
.2.77
Example 7
41
.2.8
Example 8
42
1.2.9
Example 9
44
Simulated
46
I.3.1
Example
46
I.3.2
Example
47
I.3.3
Example 3
48
13.4
Example 4
49
L33.5
Example 5
51
Common Failure Cases Direction
52
0.1
Human
52
J.1.1
Example 1
52
J.1.2
Example 2
54
J.1.3
Example 3
56
J.2
Object
58
J.2.1
Example
58
J.2.2
Example 2
60
J.2.3
Example
1.2.4
Example 4
63
J.2.5
Example 3
65
J.2.6
Example 6
66
J.3
Simulated
67
J.3.1
Example
67
J.3.2
Example 2
68
J.3.3
Example 3
69
J.3.4
Example 4
70
J.3.5
Example 5
1.3.6
Example 6
72
Common Failure Cases of Velocity & Frequency
73
Human
73
K.1.1
Example 1
73
K.2
Object
75
K.2.1
Example
75
K.2.2
Example 2
77
K.2.3
Example 3
78
K.2.4
Example 4
79
K.2.5
Example 5
81
16

---

Published as a conference paper at ICLR 2025
K.2.6
Example 6
82
K.2.7
Example 7
83
K.2.8
Example 8
84
K.2.9
Example 9
85
K.2.10
Example 10
86
K.2.11
Example 11
87
K.2.12 Example 12
89
3
Simulated
90
K.3.1 Example
90
Common Failure Cases Shape & Trend
Human
91
L.1.1
Example
L.2
Simulated
93
L.2.1
Example
..2.2
Example 2
94
L.2.3
Example 3
95
L.2.4
Example 4
96
L.2.5
Example 5
97
L.2.6
Example 6
98
22.77
Example 7
99
M Common Failure Cases of Visual Cues
100
M.
Object
100
M.I.1
Example
100
M.1.2
Example 2
101
M.1.3
Example 3
103
M.1.4
Example 4
104
M.1.5
Example 5
105
M.1.6
Example 6
106
M.1.7
Example 7
107
M.I.8 Example g
109
M.1.9 Example 9
110
M.I.10 Example 10
112
M.1.11 Example
114
M.1.12 Example 12
115
M.I.13 Example 13
116
M.I.14 Example 14
117
M.1.15 Example 15
119
17

---

M.1.16 Example 16
120
Common Failure Cases of Action Count
121
N.1
Human
121
N.1.1
Example
121
N.1.2
Example 2
123
N.1.3 Example 3
125
N.1.4
Example 4
127
N.2
Object
129
N.2.1
Example 1
129
N.2.2
Example 2
131
N.2.3
Example 3
132
N.2.4
Example 4
134
N.2.5
Example 5
136
Simulated
137
N.3.1
Example
137
N.3.2
Example 2
139
N.3.3
Example 3
141
N.3.4
Example 4
142
Back to Table of Contents
18
Back to the First Page

---

A TASKS EXAMPLES OF TOMATO
For Music-AVQA and CLEVRER, we re-annotated QA pairs to emphasize temporal aspects, such as "which musical instrument plays first" for Music-AVQA and "how many collisions are there in the video" for CLEVRER. For TGIF-QA and Perception Test, we retained their existing questions but generate additional numerical answer options close the groundtruth.
Table 6: Task examples of TOMATO. Some videos are collected from existing video datasets, including Music-AVQA (Li et al. 2022), CLEVRER (Yi et al. 2020), TGIF-QA (Jang et al. 2017). and Perecption Test (Päträucean et al. 2023). 1: Tasks requiring re-annotation
Temporal Tasks
Video Sources
Examples
In which direction's) does the object rotute?
Rolation
YouTube
(A) Clockwise (B) Counter clockwise (C) Clockwise then counter counter-clockwwie
(19.3%)
Self-created
(D) Coumter-lockwise
(E) rotation
divection(si si does the person's 's hand move?
Direction (27.2%)
YouTube & Self-created
(A) Left (B) Right (C) First the left then the right
(D) First to the right then the left (E) No movements
Velocity & Frequency (14.2%)
YouTube & Self created
What the speed pattern of the train? (N) Accelerating (B) Develerating (C) Constant Speed (D) No movemmnt
Shape Trend (15.0%)
YouTube Self-created
What the shape of the abject that the person drows the air?
(A) Circle (B) Triangle (C) Squarehectangle (D) Trapezoid
(ES Diamond (F) Not drawing all
Which musical instrument playa first?
Visual Cues (4.7%)
Music-AVQA
(A) Accordion Saxophone (C) Both instruments play simultaneously
(D) Neither instrument produces any sound
CLEVRER
How many collizion(s) s) are there in the video?
(A) (C) (D)
(F)
many times does the lick water tap?
Action Count (19.75)
TGIF QA
(A) 2
(E)
How many times does the person launch the object the slanted plane?
Perception Test
(A) (B) (C) (D) (E) (F)
How tropeznld(() does the person draw in the air?
Self-created
(A) (B) (C) (D) (E) (F)
Back to Table of Contents
19
Back to the First Page

---

B SCENARIO ANALYSIS
Table 7: MFMs performance on Counterfactual, Zoomed-in, and Pirst-Person Perspective QAs. "True" and "False" indicate whether the questions meet the criteria specified in the merged cell above the respective column (e.g. Counterfactual, Zoomed-In, and First Person Perspective).
Counterfactual
Zoomed-In
Perst-Person Perspective
False
True
4%
False
True
A%
False
True
A
GPT-4o Qwen2-VL-72B
38.5
24.0
-37.7
37.8
40.1
+6.0
38.8
47.5
+22.5
45.3
32.0
-29.4
31.6
30.3
-4.2
30.4
61.3
+101.6
Qwen2-VL-7B
37.1
33.0
24.9
+1.9
24.7
42.5
+72.1
Video-CCAM-v1.. 14B
37.5
29.0
-22.6
24.7
24.4
-1.0
24.1
35.0
+45.2
In this section, we present our analysis across various video scenarios. While the videos are not explicitly designed to rigorously validate the findings, we believe these insights are valuable for highlighting general trends observed in the experimental results for future studies.
More capable models are more reliant on common sense. In curating the counterfactual QAs, we employ video editing techniques (e.g. reversing, rotating, cropping) to produce contents that are impossible to observe in real life. As detailed in Table although all four models demonstrate similar performance connteuuterfattull QAs, the shift to counterfactual examples reveals significant performance drop, particularly for the best general-purpose model, GPT-4o, with a decrease of 37.7%, and the leading open-source model, Qwen2-Vin VL.-2-- with 29.4% decrease. These results suggest that more capable models are more prone to exploit shortcuts within the task's background information and heavily rely on pre-trained knowledge, rather than truthfully understanding the video content, even when explicitly instructed to not rely on commonsense reasoning in solving the tasks.
Zoomed-in views offer limited performance improvement challenging human scenarios. Although performance standard views varies across different models (Table 7). providing manual zoomed-in views yields only modest performance gains: 6.0% for GPT-4o and 1.9% for Qwen2-Ve 2- 7B. Surprisingly, the zoomed-in views
the performance of Video-CCA.M-vI. 14B and Qwen2-VL- VL-72B by 1.0% and 4.2%, respectively. The limited aid provided by the zoomed-in view indicates that the challenges inherent to the tasks not be tackled through zooming in, but require deeper temporal understanding of the videos beyond enhancements in visual focus.
Models excel first-person over third-person perspective temporal reasoning video understand- ing. In comparing 80 first-person perspective QAs to larger set of 668 third-person perspective QAs, our goal is to explore the general trends in model performances based on the presence absence of main subject in the video. The results suggest that the absence of main subject does not hinder model performance. In fact, we observe significantly better model performance on the first-person perspective tasks (Table Notably, the Qwen2-VL VL models achieve remarkable performance gains of 101.6% and 72.1%, respectively, with Qwen2-VL-72B VL-72B scoring 61.3 outperforming GPT-4o by 28.9% under the same conditions. These findings also highlight the potential of open-source models to surpass more capable general-purpont proprietary models on temporal reasoning video understanding tasks.
Back to Table of Contents
20
Back to the First Page

---

C EXPERIMENT SETUP
C.I. MODEL CONFIGURATION
Table 8: Model configurations for evaluation. Unset values indicate that their default values being
used. 1: Models that can process video directly. ": Models that can only process eight frames. t; Set token.kept.ratio 0.61. Configurations are based on official model repositories where available. Temp.: temperature.
Checkpoint/ / Checkpoint
New Tokens
Temp.
Top-P
Seed
Sample
Proprictary Multimodal Foundation Models (MFMs)
GPT- mo-m-i
gpt-4o-mini-28244977
215
GPT-4o
5ot-40-2824-00-88
1024
215
Claude 3 Opus
28249229
1024
Claude 3.5 Sonnet
-Laud=-3-5-sommet-2200
1024
Claude 3 Haiku
2aude=33haakku-2224333
1024
Gemini
genini-1.5-fleehh-881
Gemini 1.5 Pro
gemini-11S-pro-881
1024
Gemini Exp
5emini-2.8-flast-eeg
Reka Flash
rekk--22as-22222022
1024
Reka Core
reka-core-2024855
1024
Open-Source Multimodal Foundation Models (MFMs)
InterVV1. 2. 8B
OpenGVL UpenGvLab/IInee htt.:aablntatttt--8900
Fal.se
1024
IntemVL. 26B
False
1024
InterVT.. 2 40B
OpenGVI ht//Innernl1
False
1024
Vidco Chat 8B
False
1024
Video-CCAM--1111
IarcoTHt/VYLee--CCAA--111111
Video-CCAM-v1119 9B
TarronTl//Videee-CCi--8
Folse
1024
1ea--CCMM-1.111
LLaVA NeXT Video 32B
Fal.ee
1024
LLaVA OneVision
False
1024
Video LLaVA 7B
Languagettind//iiiee--LLLVV---
False
1024
VILA 138
EfficienttLarre-Hodde//lLL---
False
1024
nicrosotuuPt.335sssiiiinnnntteee
Qwen2-VL-77 VL-
Quen//Ween2-00-77-Tnnntte
Folse
1024
Owen2-VT--72B
DANO-ILP-SG/ViddooLLLM---2
VideoLLaMMA2 72B
DAMO-NLF
MAMD-NLP-SG/VidooLLLHM2---
False
1024
ViVe--Viden-72R
False
1024
Video-BB
Video-
BB-ee-/7--VViioono Video-Only
mams-llab/LLaV--vdddee
False
1024
AuroraCap-7B-VID
uchai/AuroraCap-112 1MM---tte-----
False
1024
C.2 IMPLEMENTATION DETAILS FOR MODEL INFERENCE
All MFMs are evaluated using a zero-shot strategy across all benchmarks, including TOMATO, to ensure fair comparison. Whenever possible, we use the provided code from each model's official page for video preprocessing. Notably, Videol I LaMA 2 7B and Videol LaMA 72B (Cheng et al. 2024) processes video content directly without splitting into frames, so we input the videos without modification. Gemini 1.5 Flash (Gemini 2024) also processes video content directly with FPS. As noted by Tam et al. 2024), restricting the output format of MFMs can hinder performance; therefore, we do not impose any output restrictions. However, this approach can complicate answer parsing using simple heuristics. To address this issue, we employ GPT-4o-ming to extract multiple-choice answers from each model's 's response when standard parser fails, where prompts provided in D We use NVIDIA A10 GPUs for all non-APl-based evaluation.
Back to Table of Contents
21
Back to the First Page

---

D PROMPTS
D.1 EVALUATION PROMPT
You will be provided with (num_frames) separate frames uniformly sampled from video, the frames are provided in chronological order of the video. Analyze these frames and provide the answer to the question about the video content. Answer the multiple-choice question about the video content.
You must use these frames to answer the multiple-choice question; do not rely on any external knowledge or commnnsense.
<question> {question} </question>
<options> (index2ans} </options>
Even if the information these separate frames is not enough answer the question, PLEASE TRY YOUR BEST TO GUESS AN ANSWER WHICH YOU THINK WOULD BE THE MOST POSSIBLE ONE BASED ON THE QUESTION.
DO NOT GENERATE ANSWER SUCH AS 'NOT POSSIBLE DETERMINE."
D.2 ANSWER EXTRACTING PROMPT
You are given a response, a list of multiple-choice options, and a index2answer mapping. You are required to extract the letter option from the GPT.
<response> (response} </response>
<all_choices> (all_choices) </all_choices>
<index2answer> (index2ans} </index2answer>
Only output the single parsed letter from the response. No other texts are needed.
If you think no options can match the index2answer dictionary, randomly select one letter.
Your extracted letter is:
Back to Table of Contents
22
Back to the First Page

---

D.3 VIDEO-FREE RANDOM GUESS PROMPT
Randomly guess reasonable answer based on the question only.
<question> {question} </question>
<options> (index2ans) </options>
DO NOT GENERATE ANSWER SUCH AS 'NOT POSSIBLE TO DETERMINE."
Back to Table of Contents
23
Back to the First Page

---

E DATA ANALYSIS
E.1 STATISTICS
Table 9: TOMATO main statistics.
Statistics
Value
Total Questions
1,484
Total Videos
1,417
Demonstration Type
Human
588 (41.4%)
Ohject
596 (42.1%)
Simulated
233 (16.4%)
Source
Self-recorded and -generated
805 (56.8%)
YouTube
398 (28.1%)
Existing Video Datasets
214 (15.1%)
Unique Source Videos
683
Duration (Seconds, avg/max)
9.21 72.74
Resolution (avg/max)
1332 x 1076 / 1080 1920
Number of Reasoning Tasks
Number of Demonstration Categories
3
Question Length (avg/max)
11.71 22
Single Choice Length (avg/max)
3.69 10
Choices per Question (avg/ myg/max)
5.19/7
Back to Table of Contents
24
Back to the First Page

---

E.2
VIDEO SOURCE DISTRIBUTION
object 6.97%
object 15.01%
human 19.86%
simulated 3,49%
Rotation 21.38%
Direction
28.20%
human 2. 2.88%
Visuel Cues 5.31%
Velovits Aidoitts&Fremqunne Ferqueeccy
object 5.31%
Action Count 20.55%
Shape& Trend 14.25%
10. 3131%
object 12.43%
human 4.85%
object 5.16%
human 4.32%
simulated
human
3. 3.79%
13.19%
Figure 3: Video source distribution of TOMATO.
Back to Table of Contents
25
Back to the First Page

---

E.3 PERFORMANCE COMPARISONS ACROSS DIFFERENT NUMBER OF FRAMES
llava-next
qwen2vl
intervideo22
gpt-4o
VITATECS
TempCompss
MVBennh
ReXTime
TOMATO
1.00
Accuracy
0.75
0.50
Frames
# Frames
Frames
Frames
Frames
Figure 4: Performance comparisons of VITATECS Li al. 2023). TempCompass (Liu et al. 2024
MVBench (Li et al. 2024b), (Chen et al. 2024a). and
TOMATO across different number
of frames.
E,4 MODEL PERFORMANCE WITH < 8 FRAMES
TempCompass
0.8
VITATECS"
0.7
MVBench
ReXTime
Accuracy
0.6
TOMATO
0.5
0.4
0.3
0.2
0
1
2
3
4
5
6
7
8
Number of Frames
Figure 5: Performance of Qwen2-VL-72B across five benchmarks with input restricted to fewer than 8 frames.
Back to Table of Contents
26
Back to the First Page

---

E.5 VIDEO DURATION DISTRIBUTION
500
Overall
400
10
20
50
40
50
25
Count
25
100
Direction
50
100
Rotation
Number of Videos
80
15
20
25
35
Shape Trend
40
Volocity & Frequency
20-
15
20
25
50
30-
Visual Cucs
20
10
20
40
50
Vidco Duration (scconds)
Figure 6: Video duration distribution of TOMATO
Back to Table of Contents
27
Back to the First Page

---

E.6 RESPONSE LENGTH DISTRIBUTION
Clodde
100
1.0
175
100
Clande Sonnet
400
200
1.00
Prequency
200
9%
1447
245
294
34G
461
490
VL. 408
200
0-
90
135
180
270
315
360
405
450
IntentVL
200
40
60
120
160
200
2
2400
520
3no
400n
InsenVidee Chus
410
200
19
11
12
13
14
15
17
15
LLaVA NeXT Video 32B
66
59
110
152
Response Length (Tokens)
Figure Response length distribution, part
Back to Table of Contents
28
Back to the First Page

---

LLaVA OneVisio 7B
1000
Phi 3.5 Vision Inntrcct
1000
Qweaz VL ALW
1000
Qwen? VL.TR
500
VILA
1000
Frequency
0-
2
5
10
11
Video OCAM
1000
0-
12
14
16
500
CCAM
0-
5
7
10
11
12
14
200
Video
100
0
176
264
932
440
528
616
704
792
500
0
12
16
20
24
28
32
36
40
46
VidksLL
138
184
230
276
322
368
414
460
Response Length (Tokens)
Figure Response length distribution, part
Back to Table of Contents
29
Back to the First Page

---

F LICENSE INFORMATION.
We, the authors, bear all responsibility case of violation of rights. TOMATO is intended only used for academic research purposes. Commercial use TOMATO in any form is strictly prohibited. The videos included in TOMATO are subject to the following licensing conditions:
YouTube videos. All videos sourced from YouTube are licensed under Creative Common The original YouTube video links are provided in our dataset, and proper attribution is given in accordance with the license terms. We do not hold the copyright to any of the You videos; we ask that you respect the rights of the original video creators. If you believe that any content in TOMATO infringes on your rights, please contact the authors immediately, and we will remove the video accordingly.
CLEVRER Yi et al. 2020) is licensed under CC TGIE-QA Jang et al. (2017) is built upon TGIF (Li et al. 2016). which is available for non-commereial only Perception Test (Päträucean al. 2023) is licensed under Creative Commons Attribution 4.0 International
Existing video datasets.
License (CC-BY) y Music-AVQA (Li et al. 2022) is licensed under Creative Commons Attribution-
NonCommercial 4.0 International (CC BY-NC 4.0
Self-recorded and -generated benchmark-speet -ppeiifc videos. Videos created specifically for TOMATO are licensed under CC BY-NC-SA These videos are intended to be freelly used for academic purposes, with proper attribution required. We kindly ask that all individuals appearing in our self-recorded videos are respected.
https://wwwyyoutubbeccom
https://rratt/veccmmonssoo
en
Back to Table of Contents
30
Back to the First Page

---

G ANNOTATION DETAILS
G.1 ANNOTATOR BIOGRAPHIES
Table 10: Biographies for human annotators in TOMATO.
ID
Year
Field of Study
Second- Year Master's Student
Mathematics
2
Second-Yeas Year Master's Student
Computer Science & Mathematics
Recent Master's ' Graduate
Mechanical Engincering
4
Second- Year Ph.D. Student
Computer Science
G.2 PILOT ANNOTATION
To determine the visual temporal reasoning types in TOMATO, we conduct a pilot study aimed at exploring and refining task definitions. Each annotator is tasked with brainstorming potential task types relevant to visual temporal reasoning. Once new reasoning type is proposed, each annotator is assigned compose 20 question-answer (QA) pairs of this reasoning type. To ensure that the QA pairs of this reasoning type adhere to the principles outlined in } we utilize GPT-4o to verify each metrics score. result of this pilot process, we identify and finalize six distinct visual temporal reasoning tasks: Rotation, Direction, Velocity & Frequency, Shape & Trend, Visual Cues, and Action Count. These task types form the foundation the full-scale annotation of our benchmark, ensuring comprehensive coverage of visual temporal reasoning.
G.3 FULL-SCALE ANNOTATION
Initial annotation. For YouTube videos, annotators are tasked with selecting videos that cor- responded specific reasoning type (as outlined in $4.1). Given the targeted reasoning type, annotators carefully search for vidcos from YouTube and compose questions that exemplified the temporal reasoning patterns (Table For TGIF-QA (Jang et al. 2017), annotators are instructed to select video-question pairs that they could easily answer. This is necessary because many questions in TGIF-QA are challenging for humans to answer due the low resolution low frame rate the videos. For CLEVRER Yi et al. (2020), annotators are responsible for writing the question (i.e., "How many distinct collision(s) occur throughout data of each video. For Music-AVQA (Li et al. 2022). annotators are tasked with curating videos that exemplified the Visual Cues reasoning types and composing corresponding question-ans -answer pairs. For self-recorded and -generated videos, both annotators and authors responsible for recording and generating videos on specific reasoning types, as well as composing corresponding question-answer pairs.
entire video?"", and we extract answers from the meta
Cross-checkings and verification. Each annotator reviews the work of another annotator by care- fully re-annotating the answer. without being provided with the original annotator's response. At the end of the process, the two sets of annotations are compared and checked for consistency. If discrepancies arise between the two annotators' answers, both annotators and the authors engage in collective resolution to reconcile the differences. During this stage, out of 1,484 questions-answer pairs, 44 questions require collective resolution during this verification phase.
Collective resolution. In cases where the annotators do not reach consensus during the cross- checking phase, a discussion held to resolve the disagreement. This phase enables both authors and annotators determine whether the discrepancy stems from lack of careful annotation from poor video quality. Out of the 44 mismatched answers, we replace four videos and correct the answers for the remaining cases.
Back to Table of Contents
31
Back to the First Page

---

H DATA ANNOTATION PLATFORM
Foamre aequerca:: to Right, to Down
How many triangle(s) does the person draw the air throughout the entire video?
Figure 9: Screenshot of our human evaluation platform. As shown, the annotator is presented with question (top right) alongside the corresponding video, displayed in a grid of 16 frames (4 x 4). The annotator is then required to select the correct answer from the provided options.
Back to Table of Contents
32
Back to the First Page

---

1 COMMON FAILURE CASES OF ROTATION
L.1 HUMAN
1.1.1 EXAMPLE
rotation 8
Which direction(s) does the person's 's hand rotate in?
A. Counter-clockwise throughout.
B. No rotation.
C. Clockwise throughout.
Counter-clockwise then clockwise.
E. Clockwise then counter-clockwise.
Answer: C. Clockwise throughout.
Based the provided frames, the person's hand appears rotating in circular motion. Initially, hand pointing upwards, then moves the right, downwards, finally the left before returning to the starting position. This suggests counter-clockwise rotation.
GPT-4o:
Therefore, the most likely answer is:
"*Option A: Counter-clockwier throughout."
Based analyzing the sequence of images provided, the most likely answer the question "Which direction(s) does the person's hand rotate in?" is:
Claude 3.5 Sonnet:
E: Clockwise then counter-clockwise.
The images show the person's hand and movement in series of frames. At the beginning. the hand is raised with the index finger pointing upward. As sequence progresses, the hand appears rotate clockwise, moving down and to the side. Then in later frames, the hand motion reverses, rotating counter-clnckwise back up to the initial raised position with the finger pointing up.
While it's difficult be certain of the full motion from still images alone, the clockwise followed by counter-clockwise rotation seems to best match the arm and hand positions shown across the frame sequence provided.
Qwen2 VL 72B instruct AWQ: E
Qwen2 VI. 7B Instruct: D
Video-CCAM-vl.1 14B: Answer is A.
InternVI. 2 40B: Based on the frames provided, the person's 's hand appears to rotate in counter-clockwise dircction throughout the video.
Answer: A
Back to Table of Contents
33
Back to the First Page

---

1.2 OBJECT
1.2.1 EXAMPLE
rotation 42 YouTube
https: https://wwww.. https://www.ooutuuee w
What direction(s) does the moon rotate relative to the Earth?
Counter-clockwise throughout.
B. No rotation.
Clockwise throughout.
D. Clockwise then counter-clockwise
Counter-clockwise then clockwise.
Answer: C. Clockwise throughout.
Based on the provided frames, the moon appears to be rotating around the Earth in a counter-clockwise direction. This is evident from the sequential positions of the moon it moves from the right side of the Earth, downwards, the left side, and then upwards, completing counter- clotkr--lockwise orbit.
GPT-4o:
Therefore, the most possible answer based on the frames is:
**'A': Counter-clockwise throughout
Claude 3.5 Sonnet: Based on analyzing sequence 16 frames provided, the answer to question "What direction(s) does the moon rotate relative to the Earth?" appears to be:
A. Counter-clockwis throughout.
The images show the moon orbiting around the Earth a counter-clockwise direction when viewed from above the North Pole. The moon starts on the right side of the Earth and moves counter-clockwise around the circular orbit, completing full revolution. Throughout the entire sequenee, the moon maintains this counter-clockwise motion relative the Earth.
Qwen2 VL 72B instruct AWQ: A
Qwen2 VL 7B Instruct: A
Video-CCAM-rl.1 14B: Answer is A.
InternVi VL 2 40B: Based on the provided frames, the moon appears to rotate in counter-clockwise direction relative the Earth throughout the video.
Therefore, the most possible answer is:
s* A: Counter-clokwwise throughout #s
Back to Table of Contents
34
Back to the First Page

---

1.2.2 EXAMPLE 2
rotation 50 YouTube
https://ww https//wwww.yoooooocccc////o//t/ oetVI
What direction(s) does the Ping Pong ball rotate in?
A. Clockwise throughout.
B. No rotation.
C. Clockwise then counter-clockwise,
Counter-clockwise throughout.
E. Counter-clockwise then clockwise.
Answer: D. Counter-clockwise throughout.
Based on the provided frames, the ping pong ball appears to rotating in consistent direction. The ball moves around the plate in a circular motion, which suggests rotational movement.
GPT-4o:
From the sequence of frames, it that the ball is rotating clockwise direction through- out video. There indication a change the direction of rotation.
Therefore, the most likely answer is:
** **Option A: Clockwise throughout.
**
Claude 3.5 Sonnet: Based on analyzing the sequence of frames provided, the most likely answer to the question "What direction(s) does the Ping Pong ball rotate in?" is:
C: Clockwise then counter-clockwise.
The reasoning for this conclusion is:
In initial frames, the ball appears to be rotating clockwise as it moves around the edge the plate in counter-clockwise direction.
2. As the ball approaches the bottom of the plate, its rotation slow down and then verse.
3. In the later frames, the ball appears to be rotating counter-clockwise as it continues its circular path around the plate.
This change rotational direction is consistent with the physics a ball rolling along a curved surface, where friction and the changing angle of contact can cause the ball to reverse its spin. The clockwise rotation followed by counter-clockwise rotation best matches the observed behavior of the ball in the provided image sequence.
Qwen2 VL 72B instruct AWQ: A
Qwen2 VL 7B Instruct: C
Video-CCAM-vv.11 14B: Answer C.
Back to Table of Contents
35
Back to the First Page

---

InternVL 40B: Based the provided frames, the Ping Pong ball appears to rotate in clockwise direction throughout the video.
Answer: A
Back to Table of Contents
36
Back to the First Page

---

1,2.3 EXAMPLE
rotation 57 YouTube
http::/www...uttu..e comm/watchh2==z233mm.kk
In which direction(s) does the person rotate on the horizontal bar?
A. Clockwise then counter-clookwise. B. Clockwise throughout.
C. Counter-clockwise throughout. D. No rotation.
E. Counter-clockwise then clockwise.
Answer: B. Clockwise throughout.
Based on the provided frames, the person appears to be rotating in counter-clockwise direction throughout the sequence. The frames show the person starting from seated position the bar, moving through an inverted position, then returning the seated position, in counter-clockwise motion.
GPT-4o:
Therefore, the most likely answer is:
#**C:: CCounter-clckkwiee throughout. *
Claude 3.5 Sonnet: Based on the sequence of images provided, the answer to the question "In which direction(s) does the person rotate on the horizontal bar?" appears be:
Clockwise throughout.
The images show the person starting in an upright seated position on the horizontal bar. then tating forward (clockwise) through various positions including hanging upside down, continuing the rotation until returning the starting position. The rotation seems consistently clockwise throughout the entire sequence of frames, without any counter-clockwise movement visible.
Qwen2 VL 72B instruct AWQ: A
Qwen2 VL 7B Instruct:
Video-CCAM-v1.1 14B:
Answer is C.
Intern VI. 40B: Based on the frames provided, the person on the horizontal har appears to rotate a counter-clnckwise direction throughout the video.
Therefore, the most possible answer is:
'C": Counter-clockwise throughout.
Back to Table of Contents
37
Back to the First Page

---

1,2.4
EXAMPLE 4
rotation 58 YouTube
OADIKAL
FIRIUNAL
CAGIUNAL
RIINAAL
METAI
CIRUINAL
FERGINAL
FATIINALL
SAW WAL
EEVIBBLL
ELTIBAL
GADIRAL
CARDINAAL
hittps:/wwwn youtube.
In which direction(s) does the gymnastic athlete rotate on the horizontal bar?
A. Clockwise throughout.
B. Counter-clockwiss then clockwise.
Clockwise then counter-clockwise
D. Counter-clockwise throughout.
No rotation.
Answer:
Clockwise throughout.
GPT-4o: Based on the provided frames, the gymnastic athlete appears to be rotating in counter-clockwise direction throughout the sequence. The athlete starts from a position above the bar and moves downward and backward, indicating counter-clockwise rotation.
Therefore, the most likely answer
**''' Counter-clockwise throughout,
Based on the sequence images provided, the gymnast appears rotating counter-clockwise throughout the routine on the horizontal bar. The athlete starts in a position with their body extended away from the bar. then swings under and around the bar in counter-clockwise direction. This counter-clockwise rotation continues through subsequent swings and movements shown in the frames.
Claude 3.5 Sonnet:
Therefore, the most likely answer to the question "In which direction(s) does the gymmastic ath- lete rotate on the horizontal bar?" is:
D: Counter-clockwise throughout.
The images consistently show the gymnast moving in counter-elockwisc direction around the horizontal bar, from the initial swing through to the later parts of the routine captured in the frames.
Qwen2 VL 72B instruct AWQ:
Qwen2 VL 7B Instruct: C
Video-CCAM-vv.1 14B: Answer A.
InternVL 40B: Based the frames provided, the gymnastic athlete appears rotate in counter- clockwise direction throughout the routine the horizontal bar.
Therefore, the most possible answer is:
'D': Counter-clockwise throughout.
Back to Table of Contents
38
Back to the First Page

---

1,2.5
EXAMPLE 5
rotation 144 YouTube
https:
www..youtube.commresul.. ts httpa//cncccncno...eeeeaaaaaaaaa
In which direction(s) does the move?
A. No rotation.
B. Counter-clockwise then clockwise.
Counter-clockwise throughout.
D. Clockwise throughout. Clockwise then counter-clockwise.
Answer: D. Clockwise throughout.
Based on the provided frames, the car appears to be rotating. The direction of rotation can be inferred from the position and orientation of the car in each frame.
GPT-4o:
From the first frame the last frame, the car seems tion. There is no indication that the car changes direction of rotation.
rotating in counter-clockwise dirce-
Therefore, the most likely answer is:
"Counter-clockwe throughout. s
Based on the series of images provided, can observe small robotic device on tiled floor. The device appears be rotating slightly between frames, but it's difficult determine consistent direction of rotation from these static images alone.
Claude 3.5 Sonnet:
Given the question and options provided, will make best guess based on the limited infor- mation available:
most likely answer seems to Counter-clockwise throughout.
This is because in some of the frames, there appears to be slight counter-clockwise rotation of the device when comparing its orientation the tile lines the floor. However, want emphasize that this educated guess based on limited information, the rotation subtle definitively clear from these static images.
Qwen2 VL 72B instruct AWQ: E
Qwen2 VL 7B Instruct:
Video-CCAM-v1.1 14B: Answer is C.
Intern 2 40B: Based on the provided frames, the car appears to move in counter-clockwise direction throughout.
Answer: C
Back to Table of Contents
39
Back to the First Page

---

1.2.6
EXAMPLE 6
rotation 64 YouTube
https: hwt. www.t//uuwooouuubbeeee com wwwp://tttiiiiiaa..RRRRR...
In what direction(s) does the person rotate?
A. No rotation.
B. Clockwise then counter-clockwise.
C. Counter-clockwise then clockwise.
Counter-clockwise throughout.
E. Clockwise throughout.
Answer: E. Clockwise throughout.
Based the provided frames, the person appears to be rotating in counter-clockwise direction initially and then changes direction rotate clockwise. This be inferred from the change in the orientation of the person's 's body and the direction they are facing.
GPT-4o:
Therefore, the most likely answer is:
* **Option C: Counter-clockwise then clockwise." **
Claude 3.5 Sonnet: Based on analyzing the sequence of images provided, the answer to the question "In what direction(s) does the person rotate?" appears to be:
D: Counter-clockwise throughout.
The images show the sequence progresses,
standing circular platform and holding wheel-like object. As
observe that the person's body orientation changes gradually in counter-clockwise direction. The person starts facing away from the camera and slowly rotates to face different directions, always turning to their left (counter-clockwise) throughout the sequence of frames. There is no evidence of clockwise rotation change in rotation direction in the provided images.
Qwen2 VL 72B instruct AWQ:
Qwen2 VL 7B Instruct: C
Video-CCAM-vv.1 14B: Answer D.
InternVL 40B: Based the frames provided, the person appears to rotate in counter-clockwiss direction throughout video.
Answer: D
Back to Table of Contents
40
Back to the First Page

---

1,2.7 EXAMPLE
rotation 84 YouTube
https://ww..youle comm/watchhy--111tt-v-v
In which direction(s) does the wheel rotate?
A. Clockwise throughout.
B. Counter-clockwiss throughout.
C. Counter-clockwise then clockwise.
Clockwise then counter-clockwise.
E. No rotation.
Answer: B. Counter-clockwise throughout.
Based the provided frames, the wheel appears be rotating in consistent direction it moves over the rock. The movement of the wheel suggests that rotating a clockwise direction throughout the sequence frames.
GPT-4o:
Therefore, the most likely answer is:
* **Option A: Clockwise throughout. **
Based analyzing the sequence of images provided, my best assessment is that the wheel rotates clockwise throughout the video. The images show Jeep tire rocky terrain, and while it's difficult see clear rotation in still frames, the progression suggests the wheel is moving forward over the rocks a clockwise direction. Therefore, would select option A: "Clockwise throughout."
Claude 3.5 Sonnet:
The images don't show any obvious change in rotation direction, the options involving counter- clockwise rotation changes in direction seem less likely. And the wheel does appear moving rather than stationary, ruling out
rotation" option.
While there's some uncertainty given the limitations of still images, clockwise rotation through- out seems the most probable answer based on the visual information provided in the frame sequence.
Qwen2 VL 72B instruct AWQ:
Qwen2 VL 7B Instruct: D
Video-CCAM-v1..1 14B: Answer A.
Intern VL 2 40B: rotation. The frames show the wheel in different positions, but they do not provide enough information to determine the direction of rotation. Therefore, the most possible answer is:
Based frames provided, is possible to determine direction of wheel's
'E': No rotation.
Back to Table of Contents
41
Back to the First Page

---

1.2.8 EXAMPLE 8
rotation 74 YouTube
hettp::/www.youle h
com/watch?v= (ene:/===JJR011AAA
In which direction(s) does the rope rotate?
A. Clockwise then counter-clockwise.
B. Clockwise throughout.
C. Counter-clockwise throughout.
D. Counter-clockwise then clockwise.
E. No rotation.
Answer:
Counter-clockwise throughout.
Based the provided frames, the rope appears he rotating consistent direction throughout the sequence. The motion the rope and the position of person suggest continuous rotation.
GPT-4o:
The most likely answer is:
"B': "Clockwise throughout."
Based on the sequence images provided, I can see a shirtless man in gray pants and black shoes performing what appears to be jump rope exercise. The rope itself is not clearly visible in most frames, but the man's 's and body movements consistent with jump roping.
Claude 3.5 Sonnet:
Regarding the specific question about direction rope rotation, with certainty from these still images. However, hased on the typical mechanics of jump roping and the man's arm positions in the frames, would make an educated guess that the rope likely rotating:
difficult to determine
Counter-clockwise throughout.
This is because in standard jump rope technique, the rope typically rotates forward over the head and under the feet in counter-clockwise direction when viewed from the jumper's right side, which is the perspective we see these images. The man's arm positions slight forward lean are consistent with this standard counter-clockwise rotation.
However, want emphasize that this is inference based on limited information from still im- ages, and without seeing the actual motion of the rope, it's impossible to be 100% certain of the rotation direction.
Qwen2 VL 72B instruct AWQ: B
Qwen2 VL 7B Instruct: B
Video-CCAMM-vI.1 14B:
Answer B.
Based the provided frames, it not possible determine the exact direction of the rope's 's rotation. The frames not provide clear view of rope's movement confidently answer the question. Therefore, the most possible answer would be:
InternVI. 2 40B:
Back to Table of Contents
42
Back to the First Page

---

'E": No rotation.
Back to Table of Contents
43
Back to the First Page

---

1.2.9
EXAMPLE 9
rotation 91 YouTTube
https://wwwi http::///ww..yuuuuee com/watch Ag
In which di rection((s) does the object rotate?
No rotation.
B. Counter-clockwise throughout.
Counter-clockwise then clockwise.
D. Clockwise throughout. Clockwise then counter-clockwise.
Answer: D. Clockwise throughout.
GPT-4o:
Based on the provided frames, the object appears to be rolating in consistent direction. The frames show gradual movement of the object in circular path.
From the sequence of frames, it can be observed that the object is rotating in a counter-clockwise direction throughout the video.
Therefore, the most likely answer is:
**'B": Counter-clockwise throughout." **
Claude 3.5 Sonnet: Based on the sequence images provided, I can analyze rolation of object on the circular platform. The object appears to be a rod with cube-shaped ends that is mounted on a central pivot point.
Looking the progression the images, we can observe that the rotates counter-clockwise direction initially. starts from a horizontal position and gradually rotates counter-clockwise through various angles.
However, the sequence of images does show complete 360-degree rotation any reversal the direction of rotation. The seen rotating counter-clockwise throughout frames provided.
Given the options and the information available from these frames. the most likely answer to the question "In which direction(s) does the object rotate?" would be:
B: Counter-cloekwicc throughout.
This answer best matches the observed motion in the sequence of images, where the rod consis- tently rotates in counter-clockwise direction without any apparent change in rotational direction.
Qwen2 VL 72B instruct AWQ: D
Qwen2 VL 7B Instruct: C
Video-CCAMMyyI.1 14B: Answer is B.
Back to Table of Contents
44
Back to the First Page

---

InternVL 2 40B: Based the provided frames, it appears that the object rotates in counter-clockwiss direction throughout the video.
Therefore, the most possible answer is:
"B': Counter-clokkwise throughout.
Back to Table of Contents
45
Back to the First Page

---

1.3 SIMULATED
1.3.1 EXAMPLE
rotation 141 YouTube
In which direction(s) does cube rotate?
Clockwise then counter-clockwise. Clockwise throughout.
C. Counter-clockwise then clockwise. D. No rotation. counter-clockwise throughout.
Answer: Counter-clockwise throughout.
Based on the provided frames, the cube appears to be rotating a consistent direction throughout the sequence. The rotation seems to be clockwise when observed from fixed point.
GPT-4o:
Therefore, the most likely answer is:
'B': "Clockwise throughout."
Based analyzing the sequence of frames provided, the cube appears clockwise throughout video. The cube starts in one orientation and gradually rotates clockwise direction across the frames, without changing direction. Therefore, the most likely answer to the question "In which direction(s) does cube rotate?" is:
Claude 3.5 Sonnet:
Clockwise throughout.
The cube maintains a consistent clockwise rotation from the beginning to the end of the frame sequence, without any apparent reversal change in rotational direction.
Qwen2 VL 72B instruct AWQ:
Qwen2 VL 7B Instruct: A
Video-CCAM-v1.1 14B: Answer
InternVL. 2 40B: Based the provided frames, it appears that the cube does exhibit any significant rotation. The cube remains a consistent orientation throughout the frames.
Therefore, the most possible answer is:
'D': 'No rotation.'
Back to Table of Contents
46
Back to the First Page

---

1.3.2 EXAMPLE
rotation 180 YouTube
http://.oouuube..co//wtatchhhh wwt/wwtcchvsJe...u. Wlovw
In which direction(s) does the minute hand move on the clock face?
A. Clockwise then counter-clockwise.
B. Clockwise throughout.
C. No rotation.
D. Counter-clockwise throughout.
E. Counter-clockwise then clockwise.
Answer: E. Counter-clockwise then clockwise.
GPT-40: Based the provided frames, the minute hand of the clock appears in clockwise direction throughout the sequence. There no indication the minute hand moving counter-clockwise any the frames.
Therefore, the most possible answer is:
"B": 'Clockwise throughout."
Claude 3.5 Sonnet: Based on analyzing the sequence of clock images provided. the answer to the question "In which direction(s) does the minute hand move the clock face?" is:
Clockwise throughout.
The images show the minute hand of the clock moving in clockwise direction over time, start- through various positions.
ing 12 o' o'clock and progressing around the clock face in clockwise There no evidence the minute hand moving counter-clockwise at any point sequence images. The minute hand consistently moves clockwise as time advances across the frames.
Qwen2 VL 72B instruct AWQ: B
Qwen2 VL 7B Instruct:
Video-CCAMMvv.11 14B:
Answer is B.
InternVL. 2 40B: Based on the provided frames, the minute hand on the clock face appears to move in a clockwise direction throughout. Therefore, the most possible answer is:
'B': Clockwise throughout.
Back to Table of Contents
47
Back to the First Page

---

1.3.3 EXAMPLE
rotation 62
In what direction(s) does the minute hand move on the clock face?
A. Clockwise throughout.
B. Counter-clockwiss then clockwise.
C. No rotation.
D. Counter-clockwise throughout.
E. Clockwise then counter-clockwise.
Answer: D. Counter-clockwise throughout.
GPT-4o: Based on the provided frames, the minute hand of the clock does not appear to move all. It remains the 12 o'clock position throughout all the frames.
Given the options, the most appropriate answer is:
'C:: 'No rotation.'
Claude 3,5 Sonnet: Based on analyzing the sequence of clock images provided. the answer to the question "In what direction(s) does the minute hand move the clock face?" is:
A: Clockwise throughout.
The minute hand (the longer red hand) can be observed moving in clockwise direction across the series of images, starting There no evidence the minute hand moving counter-clockwise at any point the sequence.
o'clock and progressing around the clock face in clockwise
Qwen2 VL 72B instruct AWQ: A
Qwen2 VL 7B Instruct: D
Video-CCAM-v1.1 14B: Answer is A.
Intern 2 40B: Based on the provided frames, the minute hand on the clock face appears to move a clockwise direction throughout.
Therefore, the most possible answer is:
'A': "Clockwise throughout.'
Back to Table of Contents
48
Back to the First Page

---

1.3.4
EXAMPLE 4
rotation 103 YouTube
https: / /wwp///www///// youtube. com/wwataiyy
In what direction(s) is the wheel rotating?
A. Counter-clockwise throughout.
B. Counter-clockwise then clockwise.
C. Clockwise then counter-clockwise.
D. Clockwise throughout.
E. No rotation.
Answer: Clockwise throughout.
Based the provided frames, it appears that the wheel is rotating in counter-clockwise direction initially and then changes clockwise direction. This be inferred from the position of the colored segments and the central cross in cach frame.
GPT-4o:
Therefore, the most likely answer is:
""Option B: Counter-cloekwise then clockwise. **
Claude 3.5 Sonnet: Based on analysis of the frames provided, most likely answer to the question "In what direction(s) is the wheel rotating?" is:
A: Counter-clockwise throughout.
Reasoning:
1. The wheel in the images consists a central black circle with four purple rectangular spokes and an outer ring with alternating cyan and black segments.
Observing position spokes across frames, there appears to be slight but consis- tent counter-clockwise movement.
3. The cyan segments the outer ring also seem to be moving in counter-cleckwise direction relative the fixed "M" letter in the top-left corner of cach frame.
4. There is no apparent change in the direction of rotation throughout the sequence of frames.
While the movement is subtle and the frames static, the overall impression from analyzing the sequence suggests counter-clockwise rotation. Therefore, option A: "Counter-clockwise throughout" seems to the most appropriate answer based on the given information.
Qwen2 VI. 72B instruct AWQ:
Qwen2 VI. 7B Instruct:
Back to Table of Contents
49
Back to the First Page

---

Video-CCAM-v1111 14B: Answer is A.
Intern VI, 2 40B: Based on the provided frames, it is not possihle to determine the direction of the wheel's s rotation. The frames do not show clear indication of the wheel's movement rotation. Therefore, the most possible answer is:
"E:; No rotation.
Back to Table of Contents
50
Back to the First Page

---

1.3.5
EXAMPLE 5
rotation 206 YouTube
youtube. wtw.:yuuuuue.oe/tttttttttooe http///atttnt/-xxxxxleerrggg
What direction(s) is the blue circle moving in?
A. Clockwise then counter-clockwise. B. Clockwise throughout.
C. Counter-clockwise then clockwise. D. Counter-clockwise throughout.
E. No rotation.
Answer: C. Counter-clockwise then clockwise.
Based on provided frames, blue circle appears rotating counter-clockwise direction throughout the sequence. The position of the blue circle relative the green center point and the orientation of the attached yellow and blue arms indicate consistent counter-clockwee motion.
GPT-40:
Therefore, the most likely answer is:
'D': "Counter-clockwise throughout."
Claude 3.5 Sonnet: Based the sequence of images provided, the blue circle appears he rotating counter-clockwise direction throughout the video. The yellow blue amms attached to blue circle consistently rotate counter-clockwise from frame to frame.
Starting from the first image and progressing through the sequence, of the blue circle move in counter-clockwise manner. There is evidence of the circle changing its direction rotation at any point the given frames.
observe that the arms
Therefore, the most likely answer to the question "What direction(s) is the blue circle moving in?" is:
D: Cluwter-clockwise throughout.
Qwen2 VL 72B instruct AWQ: D
Qwen2 VL 7B Instruct:
Video-CCAMMyy.1 14B:
Answer B.
InternVi VL 40B: Based on the provided frames, the blue circle appears to be moving in clockwise direction throughout the video.
Answer:
Back to Table of Contents
51
Back to the First Page

---

J COMMON FAILURE CASES OF DIRECTION
J.1. HUMAN
J.1.1 EXAMPLE 1
direction
In which direction(s) did the person's s hand move?
First the right then the left.
B. First to the left then to the right.
C. Right.
D. Not moving at all
E. Left.
Answer: E. Left
GPT-4o: Based on the provided frames, the person's 's hand starts pointing to right and then moves the left.
Therefore, the most possible answer is:
"*'B": FFrst to the left then to the right."**
Claude 3,5 Sonnet: Based on the sequence of images provided, the answer to the question "In which direction(s) did the person's 's hand move?" is:
Right.
The images show the person starting with their hand at their side. then moving it outward to their right and holding it there. There is clear movement of the hand to the left back and forth. The hand consistently and stays the right side of the person's body throughout the sequence of frames.
Qwen2 VL 72B instruct AWQ:
Qwen2 VL 7B Instruct: C
Video-CCAM-v1.11 14B: Answer C.
Back to Table of Contents
52
Back to the First Page

---

InternVL 2 40B: Based the frames provided, the person's 's hand appears move from right to left. The correct answer
AA:: "First the right then the left.'
Back to Table of Contents
53
Back to the First Page

---

J.1.2 EXAMPLE 2
direction
In which direction(s) did the person's s hand move?
A. Right.
B. First to the right then to the left.
C. Left.
D. Not moving all
E. First to the left then to the right.
Answer: A. Right.
GPT-4o:
Based on the provided frames, the person's hand0 starts in an upright position and then moves to the left. The hand then returns to the right. This indicates movement to the left and then back to the right.
Therefore, the most likely answer is:
**Option E: First to the left then to the right.
Claude 3.5 Sonnet: Based on analyzing the sequence of images provided, the answer to the question "In which direction(s) did the person's hand move?" appears to be:
First to the right then the left.
The images show the person's hand starting in raised position their right side, then moving across their body to the left, and then back towards the right. This back-and-ford motion is consistent with option B, which states the hand moved right then left.
Qwen2 VI. 72B instruct AWQ: E
Qwen2 VL 7B Instruct: The person's 's hand moved from the left side the right side of their body. Therefore, the answer is:
'E': 'First to the left then to the right.'
Video-CCAMMvl.I 14B: Answer is B.
Back to Table of Contents
54
Back to the First Page

---

InternVL 2 40B: Based the frames provided, the person's 's hand appears move from left to right.
The most possible answer is:
"E": First to the left then to the right.
Back to Table of Contents
55
Back to the First Page

---

J.1.3 EXAMPLE 3
direction 118
What instruction did the person give to the camera?
A. Move to the right.
B. Move closer.
C. Stay still.
D. Move to the left.
E. Move further.
Answer: E. Move further.
Based on the sequence of frames, appears that the person is making hand gestures that could be interpreted as giving instructions the camera. The person raises their hands and moves them slightly forward, which could be interpreted as signal for the camera to move closer.
GPT-4o:
Therefore, the most likely answer based on the frames provided is:
#*'B": 'Move closer,** **
Claude 3.5 Sonnet: Based on analyzing the sequence of images provided, the most likely instruction the person gave to the camera appears to be:
C: Stay still.
reasoning for conclusion is:
1. The person remains in the same position throughout all 16 frames, standing on the steps in front of the door.
background and surroundings do not change, indicating camera is not moving.
3. While the person's arms and hands move in various gestures, their overall body position and location stay constant.
There are indications the camera moving closer, further away, or either side.
5. The consistent framing and stable background strongly suggest the camera remained stationary while capturing these images.
Therefore, the instruction to "Stay still" seems to he most plausible option given the evidence the image sequence provided.
Qwen2 VL 72B instruct AWQ: C
Qwen2 VL 7B Instruct: C
Video-CCAMM-v1.1 14B: Answer
Back to Table of Contents
56
Back to the First Page

---

Intern VL 2 40B: Based the frames provided, the person appears to be standing in front of door, with their hands raised in front of their face. There is clear indication of any specific instruction being given the camera. However, since the person is stationary and not making any gestures that suggest movement, the most plausible answer would he:
CC": Stay still.
Back to Table of Contents
57
Back to the First Page

---

J.2 OBJECT
J.2.1 EXAMPLE 1
direction 242 YouTube
hitpss//www. https::/www.youttbee. wttp:yyouuuuuueeoo/mtttette
In which direction(s) did the drone move during the video?
A. Leftwards then upwards. B. Rightwards then upwards.
C. Downwards then leftwards. D. Rightwards then downwards.
E. Leftwards then downwards. Rightwards then leftwards.
Answer: F. Rightwards then leftwards.
Based on the provided frames, the drone appears to be moving rightwards initially and then upwards. This can be inferred from the drone's ' position relative to the person and the table. In the first few frames, the drone is moving the right, and in the later frames, it appears to be rising.
GPT-4o:
Therefore, the most likely answer is:
**'B: Rightwards then upwards.***
Claude 3,5 Sonnet: Based on analyzing the sequence of images provided, the most likely answer to the question "In which direction(s) did the drone move during the video?" is:
D. Rightwards then downwards.
The reasoning for this conclusion is:
In the initial frames, the drone appears be hovering in relatively stable position the ble front of the person.
2. As the sequence progresses, there are subtle changes in the drone's position relative to the per- son's con's hands and the table edge, suggesting some movement.
The drone seems to shift slightly to right (from the viewer's perspective) the middle frames.
4. In the later frames, the drone appears be lower in position compared earlier frames, sug- gesting a downward movement.
5. The person's 's hand gestures and gaze direction also provide cues that they are following the drone's movement the right and then down.
While difficult to determine precise movements from still images, the subtle changes posi- tioning across the frame sequence suggest rightward movement followed by downward movement as the most plausible interpretation of the drone's 's motion.
Qwen2 VL 72B instruct AWQ:
Qwen2 VL 7B Instruct: B
Back to Table of Contents
58
Back to the First Page

---

Video-CCAM--1.1 14B: Answer is B.
Intern VI. 2 40B: Based on the frames provided, the drone appears to move in the following directions:
Initially, the drone is seen moving towards the left side the frame.
2. After moving leftwards, the drone seen ascending upwards.
Therefore, the most likely answer is:
A*A: Leftwards upwards. **
Back to Table of Contents
59
Back to the First Page

---

J.2.2
EXAMPLE 2
direction 269 YouTube
https://www. youtube www.yyuuub..../aaaata.e------ee.
What directional shifts can be observed in the helicopter's motion from the footage?
Upwards then downwards. B. Leftwards then rightwards,
C. Leftwards then upwards. D. Downwards then rightwards.
E. Rightwards then upwards.
F. Rightwards then downwards.
Answer:
Rightwards then downwards.
Based on the provided frames, the helicopter appears to initially in the air and then gradually descends to land. The sequence of frames shows the helicopter moving downwards and slightly to the right approaches the ground.
GPT-4o:
Therefore, the most plausible answer is:
**Option D: Downwards then rightwards.
Claude 3.5 Sonnet: Based on the sequence images provided. the most likely answer question "What directional shifts can observed in the helicopter's motion from the footage?" is:
E. Rightwards then upwards.
The images show a yellow ambulance helicopter initially on the ground. the later frames, the helicopter appears to be lifting off vertically. While it's difficult to discern clear horizontal movement from still images, there seems to be slight rightward shift in the helicopter's position relative to the background buildings it begins rise.
The upward motion is more clearly evident in the later frames where the helicopter is seen hover- ing above the ground. The rightward shift, while subtle, can be inferred from the changing perspective of the helicopter against the stationary background elements.
important to note that without seeing the actual video footage, this analysis is based on the limited information provided by the still frames. The rightward movement less obvious than the upward motion. but option E best fits the observed changes in the helicopter's position across the image sequence.
Qwen2 VL 72B instruct AWQ:
Qwen2 VI. 7B Instruct: A
Video-CCAM-vy..1 14B: Answer is B.
The helicopter in the video appears to be moving in leftwards direction initially and then upwards. Based on the frames provided, the most likely directional shift observed in the helicopter's motion
InternVL 2 40B:
s*C: Leftwards then upwards.**
Back to Table of Contents
60
Back to the First Page

---

J.2.3 EXAMPLE 3
direction 243 YouTube
https: htyps://www. youtube. com/watchh 12Hks
In which direction(s) did the drone move during the video?
A. Downwards then rightwards.
B. Upwards then downwards.
C. Rightwards then upwards. D. Rightwards then leftwards.
E. Leftwards then upwards. F. Rightwards then downwards.
Answer: B. Upwards then downwards.
GPT-40: Based on the provided frames, the drone's movement can analyzed as follows:
1. The drone starts near the center of the frame.
2. It moves slightly to the left and upwards.
3. It continues moving upwards and to the left.
4. reaches the top left corner.
5. It then starts moving downwards.
6. It continues moving downwards.
7. It moves back towards the center.
continues moving downwards and slightly the right.
9. moves further downwards.
10. continues moving downwards and slightly to the right.
11. It moves back towards the center.
12. continues moving downwards.
13. moves further downwards.
14. continues moving downwards and slightly to the right.
15. It moves back towards the center.
16. It continues moving downwards and slightly to the right.
From this analysis, drone initially moves leftwards and upwards, then moves downwards slightly to the right.
The most fitting option based the frames is: "E': "Leftwards then upwards."
Claude 3.5 Sonnet: Based on analyzing the sequence of frames provided, the most likely answer the question "In which direction(s) did the drone move during the video?" is:
C. Rightwards then upwards.
The reasoning for this conclusion is:
1. In the initial frames, the drone appears as a small blurry object near the center of the cur- tain.
2. As the frames progress, the drone's position shifts gradually towards the right side of the tain, indicating rightward movement.
3. In the later frames. the drone's position appears to move higher up on the curtain, suggesting upward movement.
Back to Table of Contents
61
Back to the First Page

---

4. The final frames show the drone positioned higher and more to the right compared to its initial location.
This rightward then upward movement pattern best matches option from the given choices. While there may be some subtle variations or brief movements in other directions, the overall trend visible across the frame sequence supports this answer as the most probable description of the drone's primary movement directions.
Qwen2 VL 72B instruct AWQ: B
Qwen2 VL 7B Instruct:
Video-CCAMMyy.11 14B: Answer is F.
Intern 2 40B: Based on the frames provided, the drone appears to move in the direction of rightwards
then downwards.
Therefore, the most possible answer is:
'F": Rightwards then downwards.
Back to Table of Contents
62
Back to the First Page

---

J.2.4 EXAMPLE 4
direction 277 YouTube
http:://www.youtubee comm/watcoiv==BNa@@yyRRRR
What was the directional trend of the toy car's movement throughout the clip?
Upwards then rightwards. Downwards then leftwards, C. Leftwards then rightwards. D. Rightwards then twards.
E. Leftwards then upwards. F. Downwards then upwards.
Answer:
Rightwards then leftwards.
GPT-4o:
Based on the provided frames, the toy car appears to be moving in generally leftward direction initially and then changes direction to move rightward.
Here the analysis of frames:
1. The car starts from position closer to the right side of the frame.
2. It moves leftward in the subsequent frames.
3. Around the middle frames, the starts to change direction.
4. the later frames, the car appears to he moving rightward.
Given this observation, the most plausible answer is:
** **Option C: Leftwards then rightwards.
#*
Claude 3,5 Sonnet: Based on analyzing the 16 frames provided, the most likely answer to the question
"What was the directional trend of the toy car's movement throughout the clip?" is:
Leftwards then rightwards.
Reasoning:
1. In the initial frames, the toy car appears to be positioned more towards the right side of the image.
2. As the sequence progresses, the car
slightly towards the left, getting closer the center of
the frame.
3. In the later frames, there's 's a subtle shift where the car appears to move hack towards the right side of the image.
While movement not dramatic, there does seem a slight leftward trend followed by a rightward trend. This makes option C the most plausible answer among the given choices, based solely on the information provided in these static frames.
It's important note that the movement is very subtle, and without seeing the actual video, it's challenging to absolutely certain. However, given instruction to make a best guess based on the available information, option appears to be the most fitting answer.
Qwen2 VL 72B instruct AWQ: D
Qwen2 VL 7B Instruct: C
Video-CCAMM-v..1 14B: Answer B.
Back to Table of Contents
63
Back to the First Page

---

InternVL 2 40B: Based the frames provided, the toy appears move in a leftward direction initially and then tumss rightward. Therefore, the most possible answer is:
'C': Leftwards then rightwards.
Back to Table of Contents
64
Back to the First Page

---

J.2.5 EXAMPLE 5
direction 349 YouTube
In what direction did the car move?
A. Downwards. B. Rightwards. C. Upwards.
D. No movements. E. Leftwards.
Answer: E. Leftwards.
Based on the sequence of frames provided, car appears to be moving left side the frame to the right side of the frame. This is evident from the car's initial position on the left and its final position the right.
GPT-4o:
Therefore, the most likely answer
**Option B: Rightwards.
Claude 3.5 Sonnet: Based on the sequence of images provided, the answer to the question "In what direction did the car move?" is:
B. Rightwards.
frames show car moving across a grassy field with trees the background. As the se- quence progresses, we can see the car consistently moving from left to right in the frame. The car kicking up dust and at times even becoming airborne as it travels across the uneven terrain, but its overall direction of movement is clearly rightwards through the field.
Qwen2 VL 72B instruct AWQ: B
Qwen2 VL 7B Instruct: B
Video-CCAM-vv..1 14B: The car moving to right. answer is rightwards.
Intern 2 40B: Based on the frames provided, the car appears to be moving upwards, as it is shown in various stages of jump. Therefore, the most possible answer is:
'C': Upwards.
Back to Table of Contents
65
Back to the First Page

---

J.2.6 EXAMPLE 6
direction 294 YouTube
https:
In what direction did the camera move?
A. Backward
B. No movement C. Forward
Answer: A. Backward
Based the provided frames, it appears that the camera is moving forward. The person in the black outfit and the dog ahead of them getting closer in each subsequent frame, indicating forward movement.
GPT-4o:
Therefore, the most likely answer is:
"*Option C: #orward**
Claude 3.5 Sonnet: Based on analyzing the sequence of images provided, the most likely answer to the question "In what direction did the camera move?" is:
Forward
The images show consistent perspective of following behind person walking on forest path with dog. The camera appears be moving forward along the path, maintaining similar distance behind the person throughout the sequence. This gives the impression forward movement of camera along with the walking subjects.
Qwen2 VI. 72B instruct AWQ: C
Qwen2 VI. 7B Instruct:
Video-CCAM-v1.1 14B:
Answer is C.
InternVI. 40B: Based on the frames provided, it appears that the camera is moving backward. The perspective shows clear view of the path and the people walking away from the camera, indicating that the moving in the opposite direction of the subjects.
Therefore, the most possible answer is:
"A": 'Backward'
Back to Table of Contents
66
Back to the First Page

---

J.3 SIMULATED
J.3.1 EXAMPLE 1
direction 235
What is the direction of the red di amaod'' movement from the video?
A. Leftwards then rightwards. B. Rightwards then downwards.
C. Downwards then rightwards. D. Rightwards then upwards.
E. Downwards then upwards. F. Upwards then downwards.
Answer: F. Upwards then downwards.
Based on the provided frames, the diamond starts bottom of the frame and moves upwards. then moves slightly to right as it continues move upwards.
GPT-4o:
Therefore, the most likely answer is:
**Option D: Rightwards then upwards.** ***
Claude 3.5 Sonnet: Based on the 16 frames provided. cannot determine any movement of the red diamond shape. The diamond appears to be in the same position in the center of each frame, without any apparent change in its location orientation.
However, since I'm instructed to make guess even if there's not enough information, I'll choose the option that seems most plausible given the question:
Rightwards upwards.
This is purely a guess based on the question asking about movement, even though movement is visible the provided frames. want to emphasize that this answer not based any observed motion in the images, but simply arbitrary selection from the given options.
Qwen2 VL 72B instruct AWQ: E
Qwen2 VL 7B Instruct: B
Video-CCAM-v1.1 14B: Answer D.
Intern VI 40B: Based on frames provided, the diamond appears move upwards and then downwards. Therefore, the most possible answer is:
#*E: Downwards then u
Back to Table of Contents
67
Back to the First Page

---

J.3.2 EXAMPLE 2
direction 238 YouTube
https://www.yotteee wwt.///tc//wc///..///o SWA.CC/RRECCCOOOLEEEEMMEEEE2----
What is the direction of the game character's movement from the video?
A. Downwards then leftwards. B. Leftwards then rightwards.
C. Downwards then rightwards. D. Downwards then upwards.
E. Leftwards then downwards. F. Rightwards then downwards.
Answer: F. Rightwards then downwards.
Based the provided frames, the game character starts the top center of the field and moves downwards. As the frames progress, the character then towards the right side of the field.
GPT-4o:
Therefore, the direction of the game character's 's movement from the video is:
Option 'C:: Downwards then rightwards.
Claude 3.5 Sonnet: Based on analyzing the sequence frames provided, the direction of the game character's movement appears to be:
C: Downwards then rightwards
The character starts near the top of the green field in the early frames, then moves downward to- wards the center. In the later frames, the character can be seen moving towards the right side of the field. This downward then rightward movement pattern best matches option C from the given choices.
Qwen2 VL 72B instruct AWQ:
Qwen2 VL 7B Instruct: The direction of the game character's movement from the video is downwards then rightwards.
Video-CCAM-vL.11 14B:
Answer E.
InternVL 2 40B: Based on the frames provided, the game character appears to be moving downwards and then rightwards.
Therefore, the correct is:
#*C: Downwards then rightwards**
Back to Table of Contents
68
Back to the First Page

---

J.3.3 EXAMPLE 3
direction 361
How did the blue text block progress?
A. Upwards.
B. Downwards. C. Leftwards.
D. Rightwards. E. No movements.
Answer: B. Downwards.
Based on the provided frames, the blue text block appears to be moving upwards. In the initial frames, the block is positioned higher the screen, and the frames progress, it moves downwards. This indicates that the block moving upwards the video.
GPT-4o:
Therefore, the most possible answer is:
**'A': Upwards.** **
Claude 3.5 Sonnet: Based on the 16 frames provided, can answer the question about how the blue text block progressed.
answer Upwards.
In all 16 frames, we see blue rectangular block containing the text "The block is moving up- wards". The block remains in the same position in the center of cach frame. While don't see the block itself moving upwards across the frames, the text within the block explicitly states that it moving upwards.
Given that we are instructed to use only the information provided in the frames to answer the question, and not rely external knowledge, the most logical conclusion based on the text in the blue block is that the block is progressing upwards, though we don't visually this movement in the static frames provided.
Qwen2 VL 72B instruct AWQ: B
Qwen2 VL 7B Instruct: A
Video-CCAM-vL.1 14B: Answer
The blue text block in the video appears to be moving upwards. This can be inferred from the consistent position of the text block in the frames provided, which shows it progressively moving upwards.
Intern 40B:
Therefore, the correct
A: Upwards.
Back to Table of Contents
69
Back to the First Page

---

J.3.4 EXAMPLE 4
direction 233
What is the direction of the black sweater icon's movement from the video
A. Leftwards then rightwards.
B. Lef then downwards.
C. Downwards then leftwards.
D. Downwards then upwards.
E. Upwards then rightwards.
F. Downwards then rightwards
Answer: B. Leftwards then downwards.
Based on the provided frames, the black sweater icon starts the top right of the frame and moves downwards and then the left.
GPT-4o:
Therefore, the most likely answer is:
**Option C: Downwards then leftwards.
Based on sequence frames provided, the black sweater icon appears redmove in diagonal direction from the bottom right corner towards the top left corner of the image. The movement seems to be primarily upwards and leftwards.
Claude 3.5 Sonnet:
Given the options provided, the closest match to this observed movement would
E: Upwards then rightwards.
While this option is perfect description of the movement in the frames (as the icon moves leftwards rather than rightwards), the closest match among the given choices. The upward movement is correctly captured, and it's the only option that includes an upward direction.
It's important note that this answer based solely the information provided in the image frames and given options, as instructed, The movement not entirely clear from limited frames, but this appears to be the best guess based on the available information and options.
Qwen2 VI. 72B instruct AWQ: B
Qwen2 VI. 7B Instruct: The direction the black sweater icon't movement from the video downwards then rightwards.
Video-CCAM-vi 14B: Answer is D.
The black sweater icon in the vidco moves from the top-right cornrr to the bottom-left comerr This movement can be described moving downwards and then leftwards. Therefore, the correct answer is:
InternVI. VL 2 40B:
#eC: Downwards then *
Back to Table of Contents
70
Back to the First Page

---

J.3.5 EXAMPLE 5
direction 278 YouTube
https://wwx. /yt.tuuwb..ccm//aaaaccyyyya...l.
What was the directional trend of the car's movement throughout the clip?
Downwards then leftwards. Upwards then downwards. C. Downwards then upwards. D. Rightwards then leftwards. Leftwards then downwards. Downwards then rightwards.
Answer: D. Rightwards then leftwards.
Based on the provided frames, the car initially moves downwards and then starts to turn right. This is evident from the car's position and the activation of the right turn signal lights.
GPT-4o:
Therefore, the most likely answer is:
Option F: Downwards then rightwards
Claude 3.5 Sonnet: Based on the sequence frames provided, the most likely answer the question "What was the directional trend car's movement throughout the clip?"
F: Downwards then rightwards
Explanation:
1. In the initial frames, the car appears to be moving straight along the road.
2. As the sequence progresses, the car starts to tilt, suggesting change in direction.
3. The tilting of the car becomes more pronounced, with its front end pointing downwards and the right. straighten out again, but has moved position further right the
4. In the later frames, the starting position.
road compared
This sequence of movements suggests that the car first moved downwards (tilting forward) and then rightwards, making option F the most plausible answer among the given choices.
Qwen2 VL 72B instruct AWQ:
Qwen2 VL 7B Instruct: C
Video-CCAM-vl.11 14B: Answer
The directional trend of the car's movement throughout the clip can be observed by analyzing the positions of the car the frames provided.
InternVL 40B:
1. In the first frame, the car is positioned at the bottom of the road.
2. subsequent frames, the Finally, the car seen moving downwards towards the bottom road again.
upwards along the road.
Based on this observation, the most accurate answer is:
"B': Upwards then downwards.
Back to Table of Contents
71
Back to the First Page

---

J.3.6 EXAMPLE 6
direction 239 YouTube
httpp::/www.youeer wwt.///www/wc//ccccooooccccsc Sww.aacchh22.-00H0333MMM---22-2
What is the direction of the game character's movement from the video?
A. Leftwards then rightwards. B. Leftwards then upwards.
C. Downwards then rightwards. D. Downwards then leftwards.
E. Upwards then rightwards. F. Leftwards then downwards.
Answer: B. Leftwards then upwards.
Based on the provided frames, the game character starts at the bottom right corner and moves leftwards along the bottom edge of the field. The character then turns and moves upwards along the left edge field.
GPT-4o:
Therefore, the direction of the game character's movement is:
'B': Leftwards then upwards.
Claude 3.5 Sonnet: Based on analyzing the sequence of frames provided, the direction of the game character's movement appears to be:
Leftwards then rightwards.
The character starts in the bottom right corner of the green field. It then moves leftwards along the bottom edge of the field. After reaching the bottom left corner, the character changes direction and moves rightwards back along the bottom edge. This leftwards then rightwards movement pattern matches option from the given choices.
Qwen2 VL 72B instruct AWQ: D
Qwen2 VL 7B Instruct: The direction the game character's movement the video 'Leftwards then downwards.'
Video-CCAM-vLi1 14B: Answer
InternVI. 2 40B: Based on the frames provided, the game character appears to move from bottom left corner to the bottom right corner. This suggests that the character is moving downwards and then to the right.
Therefore, the most possible answer is:
"D': Downwards then leftwards
Back to Table of Contents
72
Back to the First Page

---

K
COMMON FAILURE CASES OF VELOCITY & FREQUENCY
K.1
HUMAN
K.1.1 EXAMPLE
velocity&ffrequencc
What is the pattern of the person's rotation speed?
A. Decelerating.
B. Accelerating.
C. Constant speed.
D. Not moving at all.
Answer: A. Decelerating.
GPT-4o: Based on the provided frames, the person appears to be rotating. The sequence of frames shows the person turning from facing away from the camera facing the camera and then back facing away again. The rotation seems to happening at consistent pace throughout frames.
Given the options: ('A': 'Dcceleratigg'. 'B': AAcelerating... 'C'; CConstant speed.", 'D': "Not moving at all.'
The most plausible answer based on frames is:
C CConstant speed.'
Based on the provided frames, cannot definitively determine the pattern the person's 's rotation speed. The images show the person in various static poses, facing different directions, but do not capture continuous motion rotation. However, since I'm asked to make best guess based on the available information, I would choose:
Claude 3.5 Sonnet:
C: Constant speed.
This is because the frames show the person in different orientations that appear be evenly spaced (front view, side view, back view, etc.), which could suggest a constant rotation if we assume the frames are sampled regular intervals. However, this is purely speculative given the limited information the static images.
Qwen2 VL 72B instruct AWQ:
Qwen2 VL 7B Instruct: C
Back to Table of Contents
73
Back to the First Page

---

Video-CCAM-vL11 14B: Answer is C.
Intern 2 40B: Based on the provided frames, the person appears be standing in different positions, facing different directions. There clear indication of rotation movement in the frames. Therefore, the most possible answer is:
D: Not moving at all.
Back to Table of Contents
74
Back to the First Page

---

K.2 OBJECT
K.2.1 EXAMPLE 1
veloeity&&rrequency 42 You'Tube
http::/wwww.oooouu..cco/aaaaaata tsMXc
What is the pattern of the plane's speed on the runway?
A. Accelerating. B. Decelerating.
C. Not moving all. D. Constant speed.
Answer: B. Decelerating.
GPT-4o:
Based on the 16 frames provided, the plane appears to be moving on the runway. The background and the position of the wing relative the ground change slightly between frames, indicating motion. However, the changes are drastic, suggesting that the plane not accelerating rapidly.
Given the subtle changes the frames, the most plausible answer is that the plane is moving at a constant speed.
Therefore, the answer
**D: Constant speed.
Claude 3.5 Sonnet: Based on analyzing the 16 frames provided, the likely answer the question "What is the pattern of the plane's speed on the runway?" is:
A: Accelerating.
While difficult determine the exact speed from still images, there are few clues sug- gest the plane is likely accelerating:
1. The frames show consistent view of the runway and surrounding area, indicating the plane is moving forward along the runway.
2. In typical takcoff procedures, planes accelerate down the runway to reach the necessary speed for liftoff.
The scenery appears moving past the window at an increasing rate in later frames, though this is subtle.
4.
signs of deceleration stopping, which would be expected if the plane slowing down maintaining constant speed.
There
Given these observations and the typical behavior of planes during takeoff, acceleration seems to be the most probable pattern of speed for the plane in this sequence of images. However, want to emphasize that this educated guess based limited information from still frames, and without more dynamic data, impossible state this with absolute certainty.
Qwen2 VL 72B instruct AWQ: A
Back to Table of Contents
75
Back to the First Page

---

Qwen2 VL 7B Instruct: A
Video-CCAM-v1.1 14B: Answer A.
Based frames provided, appears the plane moving along the runway. frames show the plane's wing and the runway in the background, indicating that the plane is in motion. Therefore, the most possible answer is:
Intern VL 2 40B:
Accelcrating.
Back to Table of Contents
76
Back to the First Page

---

K.2.2 EXAMPLE 2
veloeity&&frequency 46 YouTube
http:///ww..oouuuue www.:yyuubb..co/waaaahchhhccceee http://wttch/=ddokkkmeddddd 5nE
What is the pattern of the roller coaster's speed the rail?
Accelerating.
Not moving all.
C. Decelerating.
D. Constant speed.
Answer: C. Decelerating.
Based the provided frames, the roller coaster appears be ascending hill. The frames show a consistent upward movement along the track, which suggests that the roller coaster being pulled up by a chain mechanism. This typically means the roller coaster is moving at constant speed during the ascent.
GPT-4o:
Given this information, the most likely answer is:
* **Option D: Constant speed. **
Claude 3.5 Sonnet: Based on the series of images provided, which show first-person view from roller coaster car ascending steep incline, I would conclude that the most likely answer to the question "What is the pattern roller coaster's speed on rail?"
A: Accelerating.
The images consistently show the roller coaster car climbing up steep incline. As the car moves up this initial hill, it likely being pulled by chain or similar mechanism. This would result in a steady acceleration as the car gains both height and speed while ascending the track.
While cannot the actual speed directly, the consistent upward motion and the gradually changing perspective of track and surrounding structures suggest increasing velocity. This acceleration is typical for the initial climb of many roller coasters, building up potential energy that will converted to kinetic energy for the rest of the ride.
Qwen2 VL 72B instruct AWQ:
Qwen2 VL 7B Instruct: A
Video-CCAM-vl.1 14B:
Answer A.
Based the frames provided, the roller coaster appears to he stationary at the top of the track. Therefore, the most possible answer is:
Intern VI, 2 40B:
B: Not moving at all.
Back to Table of Contents
77
Back to the First Page

---

K.2.3 EXAMPLE 3
veloeity&&frequency 44 YouTube
Southwest
Southwest
Southwest
http:::/wwww..uue.. www.:yyuuuube..../aaettee.ee httpp///ew.o/eeaaaaoeee////////
What is the pattern of the plane's speed on the runway?
Decelerating.
Not moving all.
C. Constant speed. D. Accelerating.
Answer: A. Decelerating.
Based the provided frames, the plane's position relative the background and the runway markings does not change significantly. This suggests the plane not moving or is moving very slowly.
GPT-4o:
Given the options:
I'A': 'Decelerating'' "B': 'Not moving all.' 'C: 'Constant speed.', 'D': "Accelerating.'
The most plausible answer based on frames is:
"B:: 'Not moving at all.'
Claude 3.5 Sonnet: Based on series images provided, would say most likely answer the question "What is the pattern of the plane's 's speed on the runway?" is:
C: Constant speed.
The images show a Southwest Airlines plane on runway, but there is no clear indication of ac- celeration or deceleration. The plane appears to be in similar position relative to the background in most frames, suggesting it is moving at steady pace. While we can't I determine the exact speed, the visual evidence points the plane maintaining constant speed
along the runway.
Qwen2 VL 72B instruct AWQ:
Qwen2 VL 7B Instruct: D
Video-CCAMM-v1.1 14B: Answer A.
Intern VL 2 40B: Based on frames provided, appears that plane moving steadily along the run- way. There no visible change in the plane's 's position relative to the background, suggesting a constant speed.
Therefore, the most possible answer is:
Constant speed.
Back to Table of Contents
78
Back to the First Page

---

K.2.4 EXAMPLE 4
velocity&frrequeeccy 66 YouTube
com//watchh-=vv0000yiiee
What is the pattern of the object's speed in the video?
A. Not moving at all. B. Constant speed.
C. Decelerating.
D. Accelerating.
Answer: C. Decelerating.
Based the provided frames, it appears that the object (a cup with liquid) is initially moving downward and then upward. The motion suggests that the ohject being dropped and then caught swung back up. The the images indicates motion, and distance hetween the object and the hand changes over time.
GPT-4o:
Given the sequence of frames. the object gravity and then decelerating as eaught or swung back up.
be accelerating it downward due
Therefore, the most plausible answer based on the frames is:
**Option D: Accelerating.*
**
Claude 3.5 Sonnet: Based on analyzing sequence of frames provided, the pattern of the object's speed in the vidco appears to be:
D. Accelerating.
reasoning for this conclusion is:
1. In the initial frames, the red object (which appears to be a small container with liquid) is not visible or barely visible the bottom the frame.
2. As the sequence progresses, the object becomes more visible and moves upward in the frame.
3. The rate which the object moves upward seems to increase throughout the sequence of frames.
4. the final frames, the object clearly visible near the top of the frame, and appears to be moving more rapidly compared to earlier frames.
The blurring of object in later frames also suggests increased speed.
This pattern of movement, starting slow and gradually increasing in speed as it moves upward, is consistent with acceleration. The object appears to be gaining speed throughout the video sequence, rather than moving constant speed slowing down.
Qwen2 VL 72B instruct AWQ:
Qwen2 VL 7B Instruct:
Back to Table of Contents
79
Back to the First Page

---

Video-CCAM-vL11 14B: Answer is C.
Intern VI, 2 40B: Based the frames provided, the object appears to be moving downward in a consistent manner. There no indication of sudden changes in speed or direction. Therefore, the most likely pattern of the object's speed the video
"B": : Constant speed.
Back to Table of Contents
80
Back to the First Page

---

K.2.5 EXAMPLE 5
veloeity&&frequency 72 YouTube
https://www.youttlee www.:yyouuuue.coo///aaem conp//atcchhv-sAOMMLr.....
What is balls' speed pattern?
Decelerating.
Not moving all. D. Constant speed.
C. Accelerating.
Answer: A. Decelerating.
Based the provided frames, the ball appears to be moving from the left side of the frame the right side. The distance covered by the ball each subsequent frame seems to be increasing, indicating that the ball is accelerating.
GPT-4o:
Therefore, the most likely answer is:
**''": Accelerating. **
Claude 3.5 Sonnet: Based on the sequence of images provided, I can analyze the movement of the green and orange arrows infer the ball's speed pattem. The appear to represent the ball's motion, with the green arrow indicating forward movement and the orange arrow indicating backward movement.
Throughout the sequence, the relative sizes of the green and orange arrows remain consistent. This suggests that the ball is moving back and forth at steady rate, without any noticeable accelcration deceleration.
Given this observation, the most likely answer to the question "What is balls' speed pattern?" is:
Constant speed.
The consistent size and positioning of the arrows across the frames indicate that the ball is mov- ing at constant speed, oscillating back and forth without changing its velocity.
Qwen2 VI. 72B instruct AWQ:
Qwen2 VL 7B Instruct: C
Video-CCAM-v1.11 14B: Answer A.
InternVI. VI. 2 40B: Based the provided frames, is possihle to determine the speed pattern of the balls as there are no visible balls frames. The frames show person standing on structure a pirk-like setting. Therefore. the most possible answer would be:
"B': Not moving at all.
Back to Table of Contents
81
Back to the First Page

---

K.2.6 EXAMPLE 6
veloeity&freequnccy 135
How does the frequency of the laser pointer movement change?
A. Decelerating.
B. Not moving all.
C. Accelerating.
D. Constant speed.
Answer: A. Decelerating.
Based the provided frames, the laser pointer appears to be moving in circular pattern. The distance between the laser pointer's positions in consecutive frames seems be increasing, indicating that the speed the laser pointer increasing over time.
GPT-4o:
Therefore, the most likely answer is:
#*'C:: Accelerating.
**
Claude 3.5 Sonnet: Based on analyzing the 16 frames provided, would say the most likely answer is C: Accelerating.
images show a pink laser pointer drawing what appears cartoon or animal face on white wall or board. While the fish shape itself remains fairly constant, the blurred trail behind it seems to change in length and intensity across the frames.
the carlier frames, the blurred trail is relatively short and faint. the sequence progresses, the trail becomes longer and more pronounced in many the frames. This suggests the laser pointer is being moved more quickly over time, leaving more prominent trail.
The increasing length and intensity of the blur trails implies the laser pointer's movement is speeding up accelerating throughout video sequence. While there is some variation between individual frames, the overall trend points towards faster movement and longer trails in later frames compared to earlier ones.
Therefore, based solely the visual information provided in these 16 sample frames, the most likely answer how the frequency of laser pointer movement changes is C: Accelerating.
Qwen2 VI. 72B instruct AWQ: C
Qwen2 VL 7B Instruct: C
Video-CCAM-vI.11 14B: Answer A.
Intern VL. 2 40B: Based on the provided frames, the laser pointer appears to move in consistent manner across the whiteboard. There no significant change in the speed or direction of the movement. Therefore, the most likely answer is:
Constant speed.
Back to Table of Contents
82
Back to the First Page

---

K.2.7 EXAMPLE 7
veloeity&ffreuueccy 136 YouTube
https//wwww.youuubee.cc/m/ataaao
What is the pattern of the pendulum's movement speed throughout the entire video?
A. Not moving at all. B. Constant speed. Decelerating.
C. Accelerating.
Answer: C. Accelerating.
GPT-4o:
Based on the provided frames, appears that the pendulum is moving. The pendulum's 's position changes slightly from frame to frame, indicating movement. However, the changes in position are subtle and do not suggest significant acceleration deceleration.
Given the uniformity of the movement across the frames, the most plausible answer is that the pendulum is moving at constant speed.
Therefore, the is:
**'B": Constant speed.
Claude 3.5 Sonnet: Based the provided frames, cannot determine the pattern pendulum's
movement speed, there is no pendulum visible in these images. frames show aerial view of what appears to be a room or hall with furniture and seating.
However, since I'm instructed to provide guess based the given options, would choose:
B: Constant speed.
This is purely guess and not based on any visual evidence from the provided frames. In reality, there is no information about pendulum its movement in these images.
Qwen2 VL 72B instruct AWQ:
Qwen2 VL 7B Instruct: C
Video-CCAM-vl.11 14B: Answer
InternVL 2 40B: Based the provided frames. not possible to determine pattern of the pendulum's movement speed throughout the entire video. The frames do not show any pendulum or its movement. Therefore, the most possible answer is:
'NOT POSSIBLE TO DETERMINE:
Back to Table of Contents
83
Back to the First Page

---

K.2.8
EXAMPLE 8
veloeity&frreueeccy 139 YouTube
https://www, www.uube com
What is the object's rotational speed pattern?
Decelerating.
Not moving all.
C. Accelerating.
D. Constant speed.
Answer: A. Decelerating.
Based the 16 frames provided, the object appears be rotating constant speed. The positions of object in frame suggest uniform angular displacement between consecutive frames, indicating that the rotational speed is not changing.
GPT-4o:
Therefore, the most likely answer is:
* **Option D: Constant speed. **
Based the sequence of images provided, I can observe rotating arm white platform. The appears be rotating constant speed throughout the frames. There are no clear indications acceleration deceleration in motion the arm across the sequence images.
Claude 3.5 Sonnet:
Given the question "What is the object's rotational speed patter?"' and the available options, the most likely answer based solcly the visual information in these frames is:
D: Constant speed.
The arm seems to maintain steady rotational movement without any apparent changes in speed from frame frame. However, it's important that making definitive determination about speed patterns from images can be challenging, especially without precise timing information between frames.
Qwen2 VL 72B instruct AWQ: D
Qwen2 VL 7B Instruct: D
Video-CCAM-vl.1 14B: Answer is A.
InternVL 2 40B: Based on the provided frames, is not possible to determine the exact rotational speed pattern of the object. However, we can observe that the object is moving in circular path. Given this information, the most possible answer would be:
"D': Constant speed.
Back to Table of Contents
84
Back to the First Page

---

K.2.9 EXAMPLE 9
veloeity&ffreuuency 140 YouTube
wwwp:/ http:/uwwwe..o//aataa///sssso//a
What is the object's rotational speed pattern?
A. Not moving all. B. Decelerating.
C. Constant speed. D. Accelerating.
Answer: D. Accelerating.
GPT-40:
Based the provided frames, the object appears be rotating constant speed. The position object each frame suggests that rotational speed does not change significantly over time.
Therefore, the most likely answer is:
CConstant speed."
ased on the series of images provided, can see person demonstrating something with pink object, likely some kind of toy educational prop. The person is consistently holding and manipulating this object throughout the frames. However, from these static images alone, cannot definitively determine the rotational speed pattern of object.
Claude 3.5 Sonnet:
Given the question and available options, I will make best guess based on the limited infor- mation:
The most likely answer seems to be C: Constant speed.
My reasoning is that the person's 's hand positions and the object's appearance remain fairly consis- across the frames, suggesting steady, continuous motion rather than acceleration, deceleration, movement at However, want to emphasize that this is an educated guess based on limited information from static images, and may not reflect the actual motion in the video.
Qwen2 VL 72B instruct AWQ: C
Qwen2 VL 7B Instruct: D
Video-CCAM-yl.1 14B: Answer C.
Based on the provided frames, it appears that the object is pink object being held by person. The object does not to be moving rotating in any of the frames. Therefore, the most possible answer the question about the object's rotational speed pattern
InternVL. 2 40B:
A: Not moving at all.
Back to Table of Contents
85
Back to the First Page

---

K.2.10 EXAMPLE 10
veloeity&ffrequency 151 YouTube
https://wwes http///w.....ubee..../.. comm/watcci/=-oVVEEjj...
What is the disk's rotational speed pattern?
Constant speed.
Not moving all. D. Accelerating.
C. Decelerating.
Answer: D. Accelerating.
Based the provided frames, it appears that the disk slowing down time. The position of the red mark on the disk changes less between each frame sequence progresses, indicating decrease rotational speed.
GPT-4o:
Therefore, the most likely answer is:
Option Decelerating.
Claude 3.5 Sonnet: Based on analyzing the sequence of 16 frames provided, the disk's rotational speed pattern appears be accelerating. In the initial frames, the golden cylindrical object the disk is stationary moving slowly. the sequence progresses, the motion of this object becomes more pronounced and blurred, suggesting the disk is rotating faster. By the final frames, the golden object significantly blurred due to rapid rotation, indicating an increase in speed over time.
Therefore, the likely answer the question "What is the disk's rotational speed patter?""
D: Accelerating.
This conclusion drawn solely from observing the changing position and blur the golden object on the disk across the provided image sequence, without relying on any external knowledge.
Qwen2 VL 72B instruct AWQ: A
Qwen2 VL 7B Instruct:
Video-CCAM-v1.1 14B:
Answer is A.
Based on the provided frames, the disk appears to he rotating at a constant speed. The position of the object the disk remains consistent throughout the frames, indicating that the disk is not accelerating, decelerating, stopping. Therefore, the most likely answer is:
Intern VI. 2 40B:
A: Constant speed.
Back to Table of Contents
86
Back to the First Page

---

K.2.11 EXAMPLE 11
velocity& 157 YouTube
https://www.y> ywwtwb.....ubee...moooo http://wwyouuuuo.oo/////ttatttaa htt////wwhtchwt--S222geae
What is the trend in the camera's movement speed across the entire video?
A. Not moving all. B. Decelerating.
C. Constant speed. D. Accelerating.
Answer: B. Decelerating.
Based on the 16 frames provided, it appears that the camera not moving at all. The position of the zebras, trees, and other elements the scene remain consistent across all frames, indicating that the camera is stationary.
GPT-4o:
Therefore, the most likely answer is:
"A': 'Not moving at all.'
Claude 3.5 Sonnet: Based analyzing the 16 frames provided, the most likely the question "What trend in camera's movement speed across the entire video?" is:
C: Constant speed.
The reasoning for this conclusion is:
1. The overall composition of the scene remains very consistent across all 16 frames. The posi- tioning of key elements like the trees, zebras, and background landscape stays nearly identical.
There no noticeable change ground or background, which would typically occur the camera was accelcrating or decelerating significantly.
detail or apparent size of objects fore-
The zebras' positions relative each other and fixed elements like show only minor changes between frames, consistent with their natural movement rather than rapid camera motion.
4. The clarity and focus of the images remain constant, suggesting steady camera movement rather than acceleration which might motion blur.
There are no obvious signs of zooming or out, panning, or other camera movements would indicate changing speed.
While it's difficult be absolutely certain without seeing the actual video, the evidence from these static frames strongly suggests that the camera moving a constant speed throughout captured sequence, making option C the most probable answer.
Qwen2 VL 72B instruct AWQ: C
Qwen2 VL 7B Instruct: D
Video-CCAM-v1.1 14B: Answer B.
Back to Table of Contents
87
Back to the First Page

---

InternVL 2 40B: Based the provided frames, it appears that the camera's movement speed relatively constant throughout the video. The frames show consistent view of the zebras and the landscape without any noticeable changes in speed or direction.
Therefore, the most possible answer is:
'C": Constant speed.
Back to Table of Contents
88
Back to the First Page

---

K.2.12 EXAMPLE 12
veloeity&frreuueccy 159 YouTube
https://www.youuee www.yyyuuuee.coo/eeeee com////ttcillooyyy//ru/umm
What is the hula hoop's rotational speed pattern?
Not moving all. Decelerating.
C. Constant speed. D. Accelerating.
Answer: D. Accelerating.
Based the provided frames, the hula hoop appears be rotating consistent speed around the person's waist. There visible indication the hula hoop slowing down speeding up significantly the sequence of frames.
GPT-4o:
Therefore, the most likely answer is:
**'C": Constant speed. **
Claude 3.5 Sonnet: Based on the series of images provided, can analyze the hula hoop's 's rotational speed pattern. The images show person in pink shirt using hula hoop outdoors.
Looking at the sequence of frames, the hula hoop appears to be rotating at a fairly consistent speed throughout. There are no obvious signs of the hoop slowing down or speeding up significantly from frame to frame. The person maintains steady rhythm and posture, which suggests they are able to keep the hoop spinning a relatively constant rate.
Given the options provided and based solely on what can be observed these image frames, the most likely answer to the question "What is the hula hoop's 's rotational speed pattern?" is:
Constant speed.
The hoop seems to maintain a steady rotational speed across the sequence frames, without any clear indication of acceleration or deceleration. Of course, without more precise measurements a longer video sequence, it's difficult rule minor fluctuations in speed entirely. However, from the visual information provided, a constant speed appears to be the best characterization the hula hoop's rotational pattern.
Qwen2 VL 72B instruct AWQ: C
Qwen2 VI. 7B Instruct:
Video-CCAM-vl.1 14B:
Answer is C.
InternVIL 2 40B: Based on the provided frames, the hula hoop appears to maintain consistent motion around the girl's waist. There is visible change in the speed or pattern of the hoop's rotation. Therefore, the most likely answer
C Constant speed.
Back to Table of Contents
89
Back to the First Page

---

K.3 SIMULATED
K.3.1 EXAMPLE 1
veleeity&&frequency 81
What is the pattern of the car's s speed?
A. Decelerating. B. Accelerating.
Constant speed.
Not moving all.
Answer: A. Decelerating.
Based on the provided frames, the car appears to be moving consistently across the frames. The position of the car relative to the frame edges does not show significant changes that would indicate acceleration deceleration.
GPT-4o:
Therefore, the most likely answer is:
**C: Constant speed,
Claude 3.5 Sonnet: Based on sequence frames provided, the most likely answer the question "What is the pattern of the car's ' speed?" is:
C: Constant speed.
Reasoning:
1. The red car appears to be moving from left to right across the frames.
2. The distance the car travels hetween each frame seems be relatively consistent.
3. There's apparent change in the car's position that would suggest acceleration deceleration. car clearly moving,
not stationary.
While it's difficult to determine the exact speed without more precise measurements, the consis- tent movement of the car across the frames suggests it's maintaining steady pace. Therefore, based on the visual information provided, the most likely answer is that the moving constant speed.
Qwen2 VL 72B instruct AWQ: C
Qwen2 VL 7B Instruct: C
Video-CCAM-v1..1 14B:
Answer A.
Intern 2 40B: Based on the frames provided, the car appears to be moving from left to right across the screen. The car's position changes in each frame, indicating that it is moving. However, the frames do not provide enough information to determine the exact pattern of the car's speed (whether it is accelerating. decelerating, moving constant speed).
Given the options provided, the most possible answer would be:
B: Accelerating.
Back to Table of Contents
90
Back to the First Page

---

L COMMON FAILURE CASES OF SHAPE & TREND
L.1 HUMAN
L.1. EXAMPLE
shape&trend 0
What is the shape of the object that the person drew in the air?
A. Trapezoid.
B. Diamond.
C. Square/rectangle.
D. Circle.
E. Triangle.
F. Not drawing at all.
Answer: A. Trapezoid.
GPT-4o: Based on the sequence of frames, the person appears to be drawing a shape in the air. The movement of the hand suggests a continuous motion that could form a closed shape.
hand starts
top (Frame 1).
2. Moves to the right (Frame 3).
3. Moves downwards (Frame 5).
4. Moves to the left (Frame 7).
5. Moves upwards (Frame 9).
Returns to the starting point (Frame 11).
This sequence of movements is consistent with drawing square or rectangle.
Therefore, the most likely answer
Option 'C': Square/rectangle.
Based on the sequence of images provided, it appears the person is not drawing specific shape in the air. Rather, they making various gestures with their hand and arm, including pointing upwards, moving their arm horizontally, and positioning their hand near their face. The movements do not seem to trace out any clear geometric shape like those listed in the options.
Claude 3.5 Sonnet:
However, since I'm instructed to provide answer from the given options, would guess:
Back to Table of Contents
91
Back to the First Page

---

C: Square/rectangle.
This is because could potentially he interpreted as tracing out straight lines that form corners, which are characteristic of squares or rectangles. But want to emphasize that this is a very tentative guess based on limited information, as the images do not clearly show the person drawing any specific shape.
of the movements, particularly the horizontal and vertical motions,
Qwen2 VL 72B instruct AWQ:
Qwen2 VL 7B Instruct: B
Video-CCAM-yl.1 14B: Answer is B.
InternVL 2 40B: Based the frames provided, the person appears to be drawing shape in the air with their finger. The shape seems to straight line followed by a curve, which characteristic of circle. Therefore, the most possible answer is:
'D': Circle.
Back to Table of Contents
92
Back to the First Page

---

L.2 SIMULATED
L.2.1 EXAMPLE
&hape&trend 174
What shape is formed by the black circle's trajectory as it moves?
A. Diamond.
В. Triangle.
C. Circle. D. Square/rectangle.
E. Trapezoid. F. Not drawing at all.
Answer: B. Triangle.
Based the provided frames, the black circle appears to move in path that forms square rectangle. The circle starts at the bottom left, moves to the left, then to the top right, and finally to the bottom right before returning to the starting position.
GPT-4o:
Therefore, the shape formed by the black circle's trajectory is:
**D: Square/rectangle.* **
Claude 3.5 Sonnet: Based on the sequence of frames provided, the black circle appears to be moving in a circular trajectory. The circle starts in the bottom left corner, moves to the top center, then down to the bottom right, and finally back towards the bottom left. This circular motion is repeated across the frames.
Therefore, the most likely answer to the question "What shape formed by black circle's tra- jectory as it moves?" is:
C: Circle.
Qwen2 VI. 72B instruct AWQ: D
Qwen2 VI. 7B Instruct: C
Video-CCAM-v1.11 14B: Answer is A.
Intern 2 40B: The black circle's 's trajectory forms shape that resembles a diamond.
Answer: A
Back to Table of Contents
93
Back to the First Page

---

L.2.2 EXAMPLE
shape&trend 175
What shape is formed by the black circle's trajectory as it moves?
A. Diamond.
В. Not drawing at all.
C. Trapezoid. D. Circle.
E. Triangle. F. Square/reetangle.
Answer: C. Trapezoid.
Based the provided frames, black circle appears to moving in pattern that suggests tracing out circular path. The circle starts the bottom, moves to left, then upwards, to right, back down, completing a loop.
GPT-40:
Therefore, the most likely answer is:
**'D": Circle.
Based on the sequence frames provided, the black circle appears to moving a circular pattern. The circle starts in the bottom left corner, moves to the right, then up towards the center, then to the left, and finally back down towards the starting position. This movement pattern suggests that the shape formed by the black circle's 's trajectory most likely circle.
Claude 3.5 Sonnet:
Therefore, the answer to the question "What shape is formed by the black circle's trajectory as it moves?" is:
D. Circle.
Qwen2 VL 72B instruct AWQ: D
Qwen2 VL 7B Instruct: The black circle's trajectory forms a cirele.
Video-CCAM-vl.11 14B:
Answer is D.
Intern 2 40B: The black circle's trajectory forms shape that resembles a diamond.
Answer: A
Back to Table of Contents
94
Back to the First Page

---

L.2.3 EXAMPLE
shape&trend 182
What shape is formed by the yellow star's trajectory as it moves?
A. Diamond.
B. Triangle.
C. Circle. D. Square/rectangle.
E. Not drawing at all. F. Trapezoid.
Answer: B. Triangle.
GPT-4o:
Based on the provided frames, the yellow star appears move in pattern that forms closed shape. The starts the bottom left, moves up, then the right, down, and back the left, completing loop. This movement suggests a shape with four sides.
Given the options, the most likely shape formed by the yellow star's trajectory is:
* **Option D: Square/rectangle.**
Based on the sequence of images provided, the yellow star appears to he moving in a circular trajectory. The star's position changes gradually across the frames, moving from the bottom left, the center, to the top right, and then back towards the bottom left. This circular motion is consistent frames.
Claude 3.5 Sonnet:
throughout
Therefore, the most likely answer to the question "What shape is formed by the yellow star's tra- jectory it moves?" is:
C: Circle.
Qwen2 VI. 72B instruct AWQ:
Qwen2 VL 7B Instruct: C
Video-CCAM-vl.1 14B: Answer is A.
InternVL 2 40B: The yellow star's trajectory forms shape that resembles diamond.
Answer: A
Back to Table of Contents
95
Back to the First Page

---

L.2.4 EXAMPLE shape&trend 180
What shape is formed by the black circle's trajectory as it moves?
A. Circle.
B. Infinity-shape
C. Square/rectangle. D. Trapezoid.
E. Diamond.
F. Not drawing at all.
Answer: B. Infinity-shape.
GPT-40: Based on the provided frames, the black circle appears to moving pattern that forms a closed loop. The circle moves from the bottom center to the left, then the top, then the right, back to the bottom center. This pattern suggests a circular trajectory.
Therefore, the most likely answer is:
A": Circle.
Claude 3.5 Sonnet: Based on analyzing frames provided, which show position black circle at different points in time, can make an educated guess about the shape of the circle's 's trajectory.
The black circle appears frame. starts the upper left, moves the right, then down to the lower right, back to the left, and up again. This circular motion repeated across multiple frames.
in roughly circular oval path around the center of the
Given the options provided, the most likely answer is:
Circle.
The trajectory forms approximately circular shape the black circle moves around the frame in loop. While may not be perfect circle, this option best matches the observed motion compared the other choices given.
Qwen2 VL 72B instruct AWQ: A
Qwen2 VL 7B Instruct: B
Video-CCAM-v1..1 14B:
Answer A.
Intern VL 2 40B: Based on the provided frames, the black circle appears to move in a way that forms a shape that resembles a diamond. The trajectory of the circle's 's movement creates the outline of a diamond shape.
Therefore, the correct answer
'E': Diamond.
Back to Table of Contents
96
Back to the First Page

---

L.2.5 EXAMPLE 5
shape&trend 184
What shape is formed by the yellow star's trajectory as moves?
A. Square/rectangle. B. Not drawing at all.
C. Circle.
D. Trapezoid.
E. Diamond.
F. Triangle.
Answer: D. Trapezoid.
Based on the provided frames. the yellow star appears to be moving in circular trajectory. The starts at the bottom left, moves to the top left, then to the top right, and finally to the bottom right, before returning to the bottom left. This pattern suggests circular motion.
GPT-4o:
Therefore, the most likely answer is:
**'C:: Circle.** **
Claude 3.5 Sonnet: Based analyzing the 16 frames provided, the most likely answer the question "What shape formed by the yellow star's trajectory moves?" is:
C. Cirele.
The yellow appears to in circular pattern across the frames. It starts in the bottom left corner, moves upward and to right, then back down the left, tracing what seems circular trajectory. While we don't 't see the complete circle in every frame, the overall movement pattern strongly suggests circular path.
This circular motion is the consistent with the star's positions compared the other shape options provided. star does not appear to move straight lines that would form a square, rectangle, trapezoid, diamond, or triangle. The smooth curved motion most indicative a circular trajectory.
the different frames,
Qwen2 VL 72B instruct AWQ:
Qwen2 VL 7B Instruct: C
Video-CCAM-vl.. 14B: Answer is E.
InternVL 2 40B: The yellow star's trajectory forms shape that resembles circle.
Answer: C
Back to Table of Contents
97
Back to the First Page

---

L.2.6 EXAMPLE 6
shape&trend 186
What shape is formed by the yellow star's trajectory it moves?
B. Infinity-shape. C. Not drawing at all. D. Circle.
A. Triangle.
E. Trapezoid. F. Diamond.
Answer: B. Infinity-shape
GPT-4o:
Based on the provided frames, the yellow star appears to be moving in circular pattern. The star starts at the bottom left, moves then to the right, down, and back to the left, completing loop.
Therefore, the shape formed by the yellow star's trajectory as it moves is:
"*Option D: Circle.**
Claude 3.5 Sonnet: Based on the sequence of frames provided, the yellow star appears to be moving circular pattern. The star's position changes gradually from the bottom left corner, to the center, to top right, and then back towards the bottom left, completing what seems to be circular trajectory.
Therefore, the most likely answer to the question "What shape is formed the yellow star's tra- jectory as it moves?"
D. Circle.
The star's movement pattern across the frames strongly suggests circular path, making this the most plausible option among the given choices.
Qwen2 VL 72B instruct AWQ: A
Qwen2 VL 7B Instruct: The yellow star's trajectory forms circle.
Video-CCAM-v1.1 14B:
Answer is A.
Intern 2 40B: The ycllow star's trajectory forms shape that resembles a triangle.
Answer: A
Back to Table of Contents
98
Back to the First Page

---

L.2.7 EXAMPLE
shape&trend 183
What shape is formed by the yellow star's trajectory as it moves?
A. Not drawing at all. B. Diamond.
C. Circle.
D. Trapezoid.
E. Triangle. F. Square/rectangle.
Answer: F. Square/rectangle.
Based on the provided frames, the yellow star appears move in pattern that forms square rectangle. The starts at the bottom left, moves up the top left, then the top right, down the bottom right, and finally back the bottom left.
GPT-4o:
Therefore, the shape formed by the yellow star's trajectory is:
** **Option F: Squarc/rcctangle.**
Claude 3.5 Sonnet: Based on the sequence of images provided, the yellow star appears to he moving in a circular trajectory. The star's position changes across the frames, starting from the bottom left, moving the top center, then the top right, and finally back towards the bottom left. This movement pattern suggests circular motion.
Therefore, the most likely answer to the question "What shape is formed by the yellow star's tra- jectory it moves?" is:
C. Cirele.
Qwen2 VI. 72B instruct AWQ:
Qwen2 VL 7B Instruct: The yellow star's trajectory forms diamond shape.
Video-CCAM-v1.11 14B:
Answer
Intern 2 40B: The yellow star's trajectory forms shape that resembles diamond.
Answer: B
Back to Table of Contents
99
Back to the First Page

---

M COMMON FAILURE CASES OF VISUAL CUES
M.1 OBJECT
M.1.1 EXAMPLE
visual cues O Music-AVQA
Which musical instrument sounds last?
Cello. B. None of them produces any sound.
C. Flute. D. Violin.
E. All instruments sound the same time.
Answer: C. Flute.
GPT-4o:
Based on the provided frames, appears that the three musicians are playing their instruments simultaneously throughout the video. There is no clear indication that one instrument stops playing before the others. Therefore, the most plausible answer based on the frames is:
" "*Option E: All instruments sound the same time.
Based on the series of images provided, can see three musicians performing together outdoors night. They are playing three different instruments: flute, cello, and violin. Through- out sequence of images, all three musicians appear to be actively playing their instruments simultaneously.
Claude 3.5 Sonnet:
Given the question "Which musical instrument sounds last?" and the provided options, the most likely answer based the visual information would be:
E: All instruments sound at the same time.
This is because in all frames, the three musicians shown in playing positions with their instru- ments, suggesting they performing together ensemble. There's visual indication that any instrument stops playing before the others continues after others have finished. Therefore, most reasonable assumption from the given information is that all instruments are sounding simultaneously throughout the performance.
Qwen2 VL 72B instruct AWQ:
Qwen2 VI. 7B Instruct: A
Video-CCAM-vLl 14B: Answer is A.
Based on the provided frames, it is not possible to determine which musical instrument sounds last. The frames show three women playing different instruments (flute, cello, and violin), but they do not provide information about the timing of their sounds. Therefore, the most possible answer is:
Intern VL 2 40B:
'E': All instruments sound at the same time.
Back to Table of Contents
100
Back to the First Page

---

M.1.2 EXAMPLE
visual cues 28 Music-AVQA
Which musical instrument plays first?
Cello. B. Violin.
C. Both instruments begin simul taneously. Neither instrument produces any sound.
D.
Answer: B. Violin.
Based on the provided frames, both cello the violin players appear to preparing to play their instruments. There is no clear indication from the frames alone which instrument starts playing first. However, given that both musicians are in position and appear ready to play, it is reasonable to guess that they might begin simultancously.
GPT-4o:
Therefore, the most plausible answer based frames is:
**Option C: Both instruments hegin simultancously. **
Claude 3.5 Sonnet: Based the series of images provided, it's difficult determine with certainty which instrument plays first. However, make my best guess based on the visual information available.
The images show two individuals, one playing cello and the other playing violin. They ap- pear to be practice performance setting. with music stands front of them. Both musicians be in playing positions throughout the sequence of images.
Given that both musicians appear ready to play from the heginning of the sequence, and there's no clear indication of one starting before the other, my best guess would be:
Both instruments begin simultaneously.
This answer seems most plausible because both musicians are in playing position throughout the sequence, suggesting they might be performing duct playing together. Without audio information more detailed visual cues about who starts first, it's reasonable assume they begin playing the time.
Qwen2 VL 72B instruct AWQ: A
Qwen2 VI. 7B Instruct: A
Back to Table of Contents
101
Back to the First Page

---

Video-CCAM-v1.11 14B:
Answer is A.
Intern 2 40B: Based on the provided frames, it appears that both the cello and the violin are heing played simultaneously. Therefore, the most possible answer is:
C: Both instruments begin simultaneously.
Back to Table of Contents
102
Back to the First Page

---

M.1.3 EXAMPLE
visual cues 31 Music-AVQA
Which musical instrument plays first?
A. Both instruments begin sanouuan B. Flute.
C. Ukulele. D. Neither instrument produces any sound.
Answer: B. Flute.
Based on the provided frames, it appears that both the flute player and the ukulele player are engaged in playing their instruments throughout the sequence. There is no clear indication that one starts before the other. Therefore, the most plausible answer based frames is:
GPT-4o:
**Option A: Both instruments begin simultancously.**
Claude 3.5 Sonnet: Based the images provided, would guess that the the question "Which musical instrument plays first?" is:
C. Ukulele.
My reasoning that most the frames, both musicians appear to be actively playing their struments simultancously. However, in the last two frames, we see close-up just the ukulele player's s hands on the instrument, suggesting focus on that instrument. This could indicate that the ukulele starts the piece has solo introduction before the flute joins in.
While can't certain without hearing audio, visual emphasis on the ukulele in final frames leads me to believe it's the most likely instrument to play first in this musical performance.
Qwen2 VL 72B instruct AWQ: C
Qwen2 VL 7B Instruct: C
Video-CCAM-vl.I 14B: isswer C.
Intern VL 2 40B: Based on the frames provided, it appears that both the flute and the ukulele are being played simultaneously. Therefore, the most possible answer is:
A: Both instruments hegin simultancously.
Back to Table of Contents
103
Back to the First Page

---

M.1.4 EXAMPLE
visual cues 38 Music-AVQA
Which violin is the first sounding instrument?
A. Both violins sound at the same time. B. The right one.
C. The left one. D. Neither instrument produces any sound.
Answer: B. The right one.
Based on the provided frames, it appears that both violinists are playing their instruments simultancously throughout the sequence. There is clear indication that one starts playing before the other. Therefore, the most plausible answer based the frames is:
GPT-4o:
'A": "Both violins sound at the same time.'
Claude 3.5 Sonnet: Based on the series images provided, it's difficult to determine with certainty which violin the first sounding instrument. However, I'll make my best guess based on the visual information available.
From the images, we can see two violinists playing together throughout the sequence. They both appear to be actively playing their instruments in all frames, suggesting they might be playing simultaneously very close to each other in timing.
Given that we need to choose answer, considering the question asks specifically about which violin sounds first, would guess:
A: Both violins sound the same time.
This answer seems most plausible based on the visual evidence, as both violinists appear engaged in playing their instruments from the very first frame. There's 's no clear indication that one starts before the other. While it's impossible to hear the sound from these images alone, the synchronized movements of both players suggest they might be starting together.
Qwen2 VL 72B instruct AWQ:
Qwen2 VL 7B Instruct: C
Video-CCAMM-v..1 14B: Answer
Intern VL 40B: Based on the frames provided, it not possible determine which violin is the first sounding instrument. The frames show both women playing the violin simultancously, and there is no clear indication of which one started playing first.
Back to Table of Contents
104
Back to the First Page

---

M.1.5 EXAMPLE
visual cues Music-AVQA
Which musical instrument plays first?
A. Nei ther instrument produces any sound.
B. Both instruments begin simuutane
C.
Violin. D. Piano.
Answer: D. Piano.
Based on the provided frames, both the boy playing the piano and the girl playing the violin appear to be engaged in playing their instruments simultaneously. There is clear indication that one starts before the other. Therefore, the most plausible answer based the frames is:
GPT-4o:
""Option B: Both instruments begin simultancously."
Claude 3.5 Sonnet: Based the sequence of images provided, it's difficult determine with certainty which instrument plays first. However, I'll make my best guess based information available in the frames.
The images show two children, one playing violin and the other playing piano. Both appear be in playing positions throughout the sequence. There's clear indication of one instrument starting before the other.
Given that both children seem ready to play from the first frame, and their postures remain con- sistent throughout, would guess that the most likely answer is:
B: Both instruments begin simultancously.
This answer assumes that the children start playing together, as they both appear prepared to play from the beginning of the sequence. However, it's important that without audio detailed visual cues, this educated guess based the limited information provided the image frames.
Qwen2 VL 72B instruct AWQ: D
Qwen2 VL 7B Instruct: C
Video-CCAM--v..1 14B: Answer C.
InternVL 2 40B: Based on the provided frames, it is not possible to determine which musical instrument plays first. The frames show both the girl playing the violin and the boy playing the piano, but they do not provide enough information to determine the sequence of their playing.
Back to Table of Contents
105
Back to the First Page

---

M.1.6 EXAMPLE 6
visual cues 7 Music-AVQA
Which musical instrument plays first?
A. Nei ther instrument produces any sound. B. Pipa.
C. Erhu.
D. Both instruments begin simultaneously.
Answer: B. Pipa.
GPT-4o:
Based on the provided frames, it appears that the person playing the Erhu (on the left) starts moving their bow before the person playing the Pipa (on the right) starts plucking the strings. This suggests that the Erhu might he the first instrument produce sound.
Therefore, the most likely answer is:
Option C: Erhu.
Claude 3.5 Sonnet: Based on the series of images provided, I'll try my best to answer the question about which musical instrument plays first. However, it's important to note that these are still images, so we can't t actually hear any sound or determine the exact order of play. I'll make educated guess based on the visual cues in the frames.
Looking at the images, we can see two musicians holding traditional Chinese instruments. The person the left is holding an erhu (a two-stringed bowed instrument), while the person on the right is holding pipa (a four-stringed plucked instrument).
Throughout the sequence of images, both musicians appear to be in playing positions, holding their instruments ready to perform. There's no clear indication that one starts before the other based solely these visual cues.
Given the question and options provided, and considering both musicians seem equally prepared to play in all frames, would guess:
D: Both instruments begin simultaneously.
This answer assumes that since both musicians are consistently shown playing positions, they likely start playing their instruments at the same time. However, want to emphasize that this is an educated guess based on limited visual information, and without audio, it's impossible to determine with certainty which instrument actually plays first.
Qwen2 VL 72B instruct AWQ: C
Qwen2 VL 7B Instruct: C
Video-CCAMM-v..1 14B:
Answer B.
Based on the provided frames, not possible to determine which musical instrument plays first. The frames do not provide any information about the sequence of the instruments playing.
Intern VI. 2 40B:
Back to Table of Contents
106
Back to the First Page

---

M.1.7 EXAMPLE
visual cues 14 Music-AVQA
Which musical instrument finishes the song?
A. None of them produces any sound. B. Cello the middle.
C. All instruments sound at the same time. D. Violin on the left.
E. Violin on the right.
Answer: Cello on the middle.
GPT-40:
Based on the 16 frames provided, it appears that the video features trio of musicians: two violinists (one the left and the right) and cellist in the middle. The frames show the musicians playing their instruments throughout the video.
Given the options and the visual information from the frames, the most plausible answer is:
**Option D: Violin the left. s*
This is a guess based on the visual cues, as the frames do not provide explicit information about which instrument finishes the song.
Claude 3.5 Sonnet: Based on the provided frames, will make my best guess to answer the question "Which musical instrument finishes the song?"
From analyzing the frames, we can see string performing two violins on the left and one on the right) and cello the middle. All three musicians appear to be actively playing their instruments throughout the sequence of images.
Since the question asks specifically about which instrument finishes the song, and don't have audio information or a clear indication when the piece ends, I'll make an educated guess based on common musical practices and the visual cues available.
In many musical compositions, especially for string ensembles. it's common for all instruments play until the end, often finishing together on final chord note. However, if had to choose one instrument that might be more likely to have the final note or phrase, I would guess:
E: Violin on the right.
This is because:
1. Violins often carry the melody in string ensembles.
2. The right-side violinist seems to be in slightly more prominent position in the frame.
3. In some compositions, the highest voice (which would typically be violin) might have the final note flourish.
However, want to emphasize that this is guess based on limited visual information, and with- out audio more specific cues about the end of the piece, it's impossible to be certain.
Qwen2 VL 72B instruct AWQ: E
Qwen2 VL 7B Instruct: C
Back to Table of Contents
107
Back to the First Page

---

Video-CCAM-v1.1 14B: Answer is B.
Intern VI. 40B: Based on the given frames, it is not possible to determine which musical instrument finishes the song. The frames show the Crystal Strings Trio playing their instruments together, and there is no clear indication of which instrument finishes the song. Therefore, most possible answer
CC': All instruments sound at the same time.
Back to Table of Contents
108
Back to the First Page

---

M.1.8 EXAMPLE
visual cues 30 Music-AVQA
Which musical instrument plays first?
A. Guitar,
B. Both instruments begin simul taneously.
C. Clarinet.
D. Neither instrument produces any sound.
Answer: A. Guitar.
Based on the provided frames, both the clarinet player and the guitarist appear to be playing their instruments simultancously in all frames. There is no clear indication that one instrument starts before the other.
GPT-4o:
Given this observation, the most plausible answer is:
**'B:: Both instruments begin simultaneously.t* th
Based on the series of images provided, it appears that both musicians are actively playing their instruments throughout the frames. The woman is consistently shown playing the clarinet, while the man is playing the guitar. Since both musicians seem to be engaged in playing simultancously across all frames, and there's no clear indication of one instrument starting before the other, the most likely answer the question "Which musical instrument plays first?"
Claude 3.5 Sonnet:
B: Both instruments begin simultaneously.
This conclusion is drawn from the consistent positioning and active playing stance of both musi- cians across all the frames, suggesting they are performing together from the beginning of the captured sequence.
Qwen2 VL 72B instruct AWQ:
Qwen2 VL 7B Instruct: C
Video-CCAM-vv.1 14B: Answer C.
Intern 2 40B: Based on the frames provided, it appears that both the guitar and the clarinet are being played simultaneously. Therefore, the most possible answer is:
B: Both instruments begin simultaneously.
Back to Table of Contents
109
Back to the First Page

---

M.1.9 EXAMPLE
visual cues 55 Music-AVQA
Which musical instrument plays first?
A. Piano. B. Both instruments begin simul taneously.
C. Nei ther instrument produces any sound. D. Violin.
Answer: B. Both instruments begin simoltane
Based the provided frames, both the violinist and the pianist are in position and appear to be playing their instruments. However, there clear indication from the frames alone which instrument starts playing first. Given the context of typical musical performance, an introduction or accompaniment before the violin joins in.
GPT-4o:
for the piano provide
Therefore, the most plausible answer based on the frames is:
"A': Piano.
Claude 3.5 Sonnet: Based on the images provided, the most likely answer to the question "Which musical instrument plays first?" is:
D. Violin.
The reasoning for this answer is:
frames, we see person actively playing the violin. They are holding the violin and moving the how across the strings.
Throughout
The person the piano is seated but does not appear be actively playing in any of the frames. Their hands are not visible on the keys.
3. The violinist seems to be the main focus of the performance, standing in front of music stand and microphone.
consistent positioning and active playing the violin across frames suggests is the primary instrument being played in this scene.
While cannot hear the audio confirm definitively, based the visual evidence provided in these frames, it appears that violin the instrument playing first or primarily this performance setting.
Qwen2 VL 72B instruct AWQ:
Qwen2 VL 7B Instruct: D
Video-CCAM-vy..1 14B: Answer D.
InternVi VL 2 40B: Based on the frames provided, possible to determine which musical instrument plays first. video captures continuous performance where both violin and piano are being played simultaneously. Therefore, the most possible answer is:
Back to Table of Contents
110
Back to the First Page

---

B: Both instruments begin simultaneously.
Back to Table of Contents
111
Back to the First Page

---

M.1.10 EXAMPLE 10
visual cues 62 Music-AVQA
Which musical instrument plays first?
A. Accordion. B. Flute.
C. Neither instrument produces any sound. Both instruments begin simul taneously.
D.
Answer:
Both instruments begin simultane
Based on the provided frames, it appears that both the accordion player and the flute player are preparing to play their instruments. However, there is no clear indication from the frames alone which instrument starts playing first.
GPT-4o:
Given the options visual cues, will make an educated guess:
The accordion player seems to be actively pressing the keys and moving the bellows in the ini- tial frames.
The flute player holding flute a playing position does not show clear signs starting play first.
Therefore, the most plausible answer based on the frames is:
** **Option A: Accordion.
s
Claude 3.5 Sonnet: Based on the sequence of images provided, would guess that the answer to the question "Which musical instrument plays first?" is:
Accordion.
My reasoning for this guess is:
Throughout all the frames, we see man holding seemingly playing an accordion. His posture and hand positions on the instrument remain consistent, suggesting he actively playing it.
2. The woman with the flute appears to be in playing position, but her posture and the position of the flute don't ' change significantly across the frames. This could indicate she might be preparing to play joining later.
3. The accordion player seems more engaged with his instrument from the beginning, which might suggest he starts playing first.
4. While we can't hear the audio, the consistent positioning of the accordion player across all frames gives the impression that he might have begun playing first or is the lead musician in this scenario.
Without being able to hear the audio However, based visual cues the provided frames, the accordion seems the most likely instrument have started playing first.
detailed motion, it's impossible be certain.
Qwen2 VL 72B instruct AWQ:
Qwen2 VL 7B Instruct: A
Back to Table of Contents
112
Back to the First Page

---

Video-CCAM--111 14B: Answer is A.
Intern VI. 2 40B: Based on the provided frames, it is not possible to determine which musical instrument plays first. Both the accordion and the flute are being played simultaneously in the video. Therefore, the most possible answer is:
D: Both instruments begin simultancously.
Back to Table of Contents
113
Back to the First Page

---

M.1.11 EXAMPLE 11
visual cues 63 Music-AVQA
Which musical instrument plays first?
A. Both instruments begin simultaneously,
B. Neither instrument produces any sound.
C. Guzheng. D. Accordion.
Answer: A. Both instruments begin simul
Based the provided frames, it appears that the person playing the Guzheng actively plucking the strings, while the person with the accordion holding the instrument but not yet playing This suggests that the Guzheng might be the first instrument to produce sound.
GPT-4o:
Therefore, the most likely answer is:
**'C": Guzheng.* s*
Claude 3.5 Sonnet: Based on the provided frames, it's difficult to determine with certainty which instrument plays first. However, I'll make my best guess based the information available in the images.
The frames show two musicians, one playing an accordion and the other playing what appears to be guzheng (a Chinese plucked string instrument). Both musicians seem to be in performance positions throughout the sequence of images.
Given that question asks which instrument plays first, and considering that both musicians ap- pear ready to play in all frames, would guess:
C: Guzheng.
My reasoning for this guess is that the guzheng player's posture seems slightly more active in the earlier frames, with their hands positioned over the strings, suggesting they might be about to play or have just started playing. The accordion player's 's posture remains relatively consistent throughout the frames.
guess based on limited visual information, we can't Itn't hear the audio or see the exact moment when each instrument begins playing.
However, want emphasize that
Qwen2 VL 72B instruct AWQ: D
Qwen2 VI. 7B Instruct: D
Video-CCAM-vl.1 14B: Answer is D.
Intern VL 2 40B: Based on the provided frames, it is not possible to determine which musical instrument plays first. The frames show both the accordion and the guzheng being played simultaneously. without any clear indication of which one started first. Therefore, the most possible answer
'A: Botth instruments begin simultancously.
Back to Table of Contents
114
Back to the First Page

---

M.1.12 EXAMPLE 12
visual cues 3 Music-AVQA
Which instrument plays first?
A. Both instruments begin simultaneously. B. Banjo.
C. Violin.
D. Neither instrument produces any sound.
Answer: C. Violin.
GPT-4o:
Based on the provided frames, it appears that the person with the violin is preparing play first. In the initial frames, the person with the violin is seen positioning the bow and the violin, while the person with the banjo not yet actively playing.
Therefore, the most likely answer is:
Option C: Violin.
Claude 3.5 Sonnet: Based on the sequence of images provided, would guess that the answer to the question "Which instrument plays first?" is:
Banjo.
My reasoning is as follows:
1. In the early frames, both musicians holding their instruments (banjo and violin) but not actively playing.
2. As the sequence progresses, the banjo player appears to become more animated and engaged first. His posture changes and he seems to be moving his hands on the instrument in a playing motion.
The violin player's posture and engagement seems follow slightly after the banjo player in the sequence of images.
4. In the later frames, both musicians appear to be actively playing and singing/vocalizing to- gether.
While it's impossible to certain without audio, the visual cues suggest that the banjo player ini- tiates the performance, with the violin joining in shortly after. Therefore, my best guess based solely on these images is that the banjo plays first.
Qwen2 VL 72B instruct AWQ:
Qwen2 VL 7B Instruct: B
Video-CCAM-v1.1 14B:
Answer B.
InternVL 2 40B: Based the frames provided, it appears that both the banjo and the violin being played simultaneously. Therefore, most possible answer is:
A: Both instruments begin simultaneously.
Back to Table of Contents
115
Back to the First Page

---

M.1.13 EXAMPLE 13
visual cues 19 Music-AVQA
Which guitar is the first sounding instrument?
A. The left one. B. Neither instrument produces any sound.
C. Both guitars sound at the same time. D. The right one.
Answer: D. The right one.
GPT-4o:
Based on the provided frames, it appears that both guitarists are actively playing their instruments. However, the guitarist the right seems to more animated and engaged in playing, suggesting that he might be the first to produce sound.
Therefore, the most likely answer is:
**D: The right one **
Claude 3.5 Sonnet: Based on the sequence of images provided, it appears that both guitarists are playing their instruments simultaneously throughout the video. From the first frame to the last, can see both musicians actively engaged with their guitars, suggesting they performing together.
While it's impossible to determine the exact timing when each instrument first produces sound from these still images alone, the visual evidence suggests that both guitarists are playing concurrently. Therefore, the most likely answer to the question "Which guitar is the first sounding instrument?" is:
Both guitars sound the same time.
This answer assumes that since both guitarists are shown playing from the beginning, they likely started their performance together. Without audio information, this seems to be the most reasonable conclusion based the visual information provided in the frames.
Qwen2 VL 72B instruct AWQ: A
Qwen2 VL 7B Instruct: A
Video-CCAMM-v..1 14B:
Answer is A.
InternVI. 40B: Based on the frames provided, appears that both guitars are being played simultaneously. Therefore, the most likely answer is:
'C': Both guitars sound at the same time.
Back to Table of Contents
116
Back to the First Page

---

M.1.14 EXAMPLE 14
visual cues 25 Music-AVQA
Which musical instrument plays first?
A. Gui tar.
B. Neither instrument produces any sound.
C. Pipa.
D. Both instruments begin simultaneously.
Answer: A. Guitar.
Based on the provided frames, it appears that both musicians are preparing to play their instruments. The person with the guitar seen singing and strumming the guitar, while the person with the pipa is adjusting and plucking the strings.
GPT-4o:
Given the visual cues, it seems that the person with the guitar is more actively engaged play- ing and singing, suggesting that the guitar might be the first instrument to produce sound.
Therefore, the most likely answer
A': Guitar.**
Claude 3.5 Sonnet: Based on the series of images provided, would guess that the answer to the question "Which musical instrument plays first?" is:
Pipa.
My reasoning for this guess is:
1. The images show musicians, playing a pipa (a Chinese lute-like instrument) and the other playing a guitar.
2. In several of the early frames. the pipa player appears to be more actively engaged with their instrument, holding in playing position and sceming to strum pluck the strings.
The guitar player, while holding the instrument, docsn't seem to be as actively playing the initial frames.
4. There's progression in the images where both musicians become animated, suggesting they eventually play.
5. Given that the pipa player seems more ready to play in the earlier frames, it's reasonable to guess that this instrument might have started first.
However, it's important to note that this an educated guess based solely on the visual informa- tion provided in the still frames. Without audio or more detailed timing information, it's impossible to be certain which instrument actually played first.
Qwen2 VL 72B instruct AWQ:
Qwen2 VL 7B Instruct: C
Video-CCAMMv1..1 14B: Answer C.
Back to Table of Contents
117
Back to the First Page

---

Based on the provided frames, it is not possible to determine which musical instrument plays first. The frames show both the guitar and the pipa being played simultaneously. without any clear indication of which one started first.
VL 2 40B:
Therefore, the most possible answer is:
Both instruments begin simultaneously.
Back to Table of Contents
118
Back to the First Page

---

M.1.15 EXAMPLE 15
visual cues 26 Music-AVQA
Which violin is the first sounding instrument?
A. Both violins sound at the same time. B. The right one.
C. Neither instrument produces any sound. D. The left one.
Answer: B. The right one.
GPT-4o:
Based on the provided frames, it appears that the person the right (the right violinist) is consistently in playing position with the bow the strings, while the person the left (the left violinist) not always playing position sometimes the bow away from the strings.
Given this observation, the most likely answer is:
B*'B: The right one.** **
Based on the sequence of images provided, it appears that both violinists start playing their instruments at approximately the same time. In the first few frames. both musicians are positioning their violins and bows, and by the third or fourth frame, they both seem be actively playing. There's clear indication that violinist starts before the other.
Claude 3.5 Sonnet:
Given the options provided and the information available in the images, the most likely answer is:
Both violins sound the same time.
This answer best fits what can be observed from the sequence of frames, where both violinists appear to begin playing simultaneously very close the same moment.
Qwen2 VL 72B instruct AWQ:
Qwen2 VL 7B Instruct: D
Video-CCAM-v1.11 14B: Answer D.
Based on the provided frames, it possible determine which violin is the first sounding instrument as frames do not provide any audio information. The visual content only shows the two women playing their violins, but it does not indicate which one started playing first. Therefore, the most possible answer would be:
Intern VI. 2 40B:
"A': 'Both violins sound at same time."
Back to Table of Contents
119
Back to the First Page

---

M.1.16 EXAMPLE 16
visual cues 53 Music-AVQA
Which musical instrument plays first?
A. Neither instrument produces any sound.
B. Both instruments begin simultaneously.
C. Guitar. D. Ukulele.
Answer: Both instruments begin simul tamuotaan
Based on the provided frames, it appears both guitar ukulele are being played simultaneously throughout the vidco. There is no clear indication that one instrument starts before the other. Therefore, the most plausible answer based on the frames is:
GPT-40:
** **Option B: Both instruments begin simultaneously. **
Claude 3.5 Sonnet: Based on the series of images provided, would guess that the answer to the question "Which musical instrument plays first?" is:
Ukulele.
My reasoning for this guess is:
1. In all the frames, see people playing musical instruments one with smaller instru- ment that appears to be ukulele, and the other with larger instrument that looks like an acoustic guitar.
2. Both musicians seem to be actively playing their instruments in all frames, their hands are positioned the instruments in playing positions.
3. Since the question asks which instrument plays first, and we don't have any audio information, we have to make an educated guess based on the visual cues.
4. The person with the ukulele is consistently shown the left side of the frame, which in West- ern reading and viewing patterss, tends to indicate precedence or coming first.
5. In some frames, the ukulele player seems to be looking at slightly turned towards the guitar player, which could suggest they might be leading starting the performance.
While this not a definitive answer, given the constraints the task and the information avail- able in the images, this seems like the most plausible guess among the options provided.
Qwen2 VL 72B instruct AWQ:
Qwen2 VL 7B Instruct: D
Video-CCAM-vv.1 14B: Answer D.
InternVL 2 40B: Based on the frames provided, it is not possible to determine which musical instrument plays first. The video does provide enough information the question with certainty.
Back to Table of Contents
120
Back to the First Page

---

N COMMON FAILURE CASES OF ACTION COUNT
N.1 HUMAN
N.1.1 EXAMPLE 1
count
How many triangle(s) does the person draw in the air throughout the entire video?
A.
3
B.
C.
D.
1
E. 2
F. 0
Answer: C. 4
GPT-40: Based the provided frames, appears that the person drawing shapes in the air. The sequence of movements suggests the drawing of triangles. Here is the analysis:
1. Frame 1: Starting point at the top.
2. Frame 2: Still at the top.
Frame Moving to the right.
4. Frame 4: Moving downwards.
5. Frame 5: Moving to the left.
6. Frame 6: Completing the first triangle.
Frame Starting the second triangle the top.
Frame 8: Moving to the right.
9. Frame 9: Moving downwards.
10. Frame 10: Moving to the left.
11. Frame 11:
Completing the second triangle.
12. Frame Starting the third triangle the top.
13. Frame 13: Moving to the right.
14. Frame 14: Moving downwards.
15. Frame 15: Moving the left.
16. Frame 16: Completing the third triangle.
From the frames, it can be inferred that the person draws three triangles in the air.
Therefore, the most likely answer is: "3"
Claude 3.5 Sonnet: Based on the series images provided, can see the person making various hand gestures and movements. However, don't It see any clear evidence of the person drawing complete triangles
Back to Table of Contents
121
Back to the First Page

---

the air. The images show the individual pointing upwards, moving their hand downwards, and making other gestures, but there obvious triangle shapes being formed.
Given the limited information available from these static images, and the instruction to provide a guess if the information is not conclusive, I would estimate that the most likely answer is:
E:
This is a guess based on the fact that there are some hand movements that could potentially be parts of triangle shapes, but not enough to confidently say there are more than 2 complete triangles drawn, However, I want to emphasize that this is an educated guess given the constraints of the task, and not definitive answer based clear evidence from the images provided.
Qwen2 VL 72B instruct AWQ: A
Qwen2 VL 7B Instruct: A
Video-CCAMMvv..1 14B: Answer is
Intern VI. 2 40B: Based on the provided frames, appears the person is drawing triangle the The frames show the person's 's hand making a triangular shape multiple times.
Given the options: 'A": '3', BB: '5', 'C: '4, DD:: '1, 'E:: '2", 'F:: '0'
The most plausible answer based on the frames is:
'D''1'
Back to Table of Contents
122
Back to the First Page

---

N.1.2
EXAMPLE 2
count 4
How many diamond shape(s) does the person draw in the air throughout the entire video?
A.
3
B.
0
C.
1
D. 5
E.
F.
Answer: A. 3
GPT-4o: Based on the provided frames, the person appears to be drawing a shape the air. The sequence of movements suggests the person is drawing diamond shape. The frames show the person pointing at different positions that could correspond to the vertices of diamond shape.
Here is the analysis of the frames:
1. Frame 1: Pointing at the top vertex.
2. Frame 2: Pointing the right vertex.
3. Frame 3: Pointing the bottom vertex.
Frame Pointing at the left vertex.
5. Frame 5: Pointing at the top vertex again.
6. Frame 6: Pointing at the right vertex again.
7. Frame 7: Pointing the bottom vertex again.
Frame Pointing the left vertex again.
9. Frame 9: Pointing at the top vertex again.
10. Frame 10: Pointing at the right vertex again.
11. Frame 11: Pointing at the hottom vertex again.
12. Frame 12: Pointing at the left vertex again.
13. Frame Pointing at top vertex again.
14. Frame 14: Pointing at the right vertex again.
15. Frame 15:
Pointing at the bottom vertex again.
16. Frame 16: Pointing at the left vertex again.
The sequence movements suggests that the person drawing diamond shape multiple times. Based on the frames, it appears that the person completes the diamond shape four times.
Therefore, the most likely answer is: 'E" '4'
Claude 3.5 Sonnet: Based on the provided images, do not any clear evidence of the person drawing diamond shapes in air. The images show individual making various hand gestures and movements, but none of them definitively depict drawing diamond shapes.
Back to Table of Contents
123
Back to the First Page

---

However, since I'm asked to make best guess based on the question and options provided, would choose:
C:
This is because while I don't see clear diamond shape being drawn, it's possible that of the hand movements could interpreted as drawing single diamond shape that isn't fully captured these still images. The option of diamond shape seems like the most conservative guess given the limited information available from the frames provided.
Qwen2 VL 72B instruct AWQ: C
Qwen2 VL 7B Instruct: A
Video-CCAM-v1.1 14B: Answer is C.
Vntern 2 40B:
Based on the provided frames, it appears that the person is making gesture with their hand, hut it is not clear if they are drawing diamond shape the air. The frames do not provide enough information to confidently determine number of diamond shapes drawn.
However, since I must provide an answer, I will make an educated guess based on the available information.
*nswer:**
sef: 2
Back to Table of Contents
124
Back to the First Page

---

N.1.3
EXAMPLE 3
count 83
How many square(s) does the person draw the air throughout the entire video?
A.
3
B.
4
C.
D.
0
E.
F.
Answer: F. 2
Based the provided frames, it appears that the person is making series of hand movements that could be interpreted as drawing shapes in the air. The sequence of movements suggests the person is drawing multiple squares.
GPT-4o:
From the frames, it looks like the person completes the motion of drawing square multiple times. Here the breakdown:
1. The person starts with their hand in the top-right position.
2. Moves to the bottom-right.
3. Moves to the bottom-left.
4. Moves to the top-left.
Returns to the top-right.
This sequence is repeated multiple times throughout the frames.
Given the frames,
the person completes the motion of drawing square times.
Therefore, the most likely answer is:
=*'B": *4***
Claude 3.5 Sonnet: Based the sequence of images provided, cannot definitively determine how many squares the person draws the air throughout the entire vidco. The frames show the per- son making various arm and hand gestures, but do not clearly depict the drawing of multiple complete squares.
However, given the question and available options, I will make my best guess based the lim- ited information:
My estimate that the person draws square in the air throughout the video.
Therefore, my answer is:
This is an educated guess based on observing the person's arm movements across the frames, which suggest they may complete at least square gesture, but likely multiple squares given the limited number of distinct motions shown.
Qwen2 VL 72B instruct AWQ:
Qwen2 VL 7B Instruct: The person draws 2 squares in the air throughout the entire video.
Back to Table of Contents
125
Back to the First Page

---

Video-CCAM-vL11 14B: Answer is D.
InternVI. 2 40B: Based on the frames provided, the person appears to be drawing square in the air. The frames show the person's hand making consistent motion that suggests drawing square.
Given the options: "A': '3', "B: '4' 'C: '5, DD: '0, "T' "2"
The most plausible answer is that the person draws one square in the air throughout the entire video.
Answer: E
Back to Table of Contents
126
Back to the First Page

---

N.1.4 EXAMPLE 4
count 12
How many times does the person swing their hand to the right throughout the entire video?
A. 2
B.
0
C.
D.
F.
Answer: 6
GPT-4o: Based the provided frames, the person swings their hand the right in the following frames:
1. Frame 1
2. Frame 4
3. Frame 6
4. Frame 8
Frame 10
6. Frame 12
Therefore, the person swings their hand to the right 6 times throughout the entire video.
correct answer
'E': '6'
Claude 3.5 Sonnet: Based the analysis of the frames provided, can that the person swings their hand to right multiple times throughout the video. Counting distinct right-hand swings visible in the frames, observe 4 clear instances where the hand extended to the right side.
Therefore, my answer to the question "How many times does the person swing their hand the right throughout the entire video?" is:
F:4
This be the accurate count based the visual evidence provided in the frame samples from video.
Qwen2 VL 72B instruct AWQ:
Qwen2 VL 7B Instruct: A
Back to Table of Contents
127
Back to the First Page

---

Video-CCAM-v1.1 14B: Answer is A.
Intern VI, 2 40B: Based on the provided frames, the person swings their hand to the right in the following frames: 2, 3, 5, 7, 8, 9, 10, 11. 12, 13, 14, 15.
Counting the number of times the hand swings to the right, it appears to be 15 times.
However, since the options provided are limited, the closest answer is:
5
Back to Table of Contents
128
Back to the First Page

---

N.2 OBJECT
N.2.1
EXAMPLE 1
count 72
A
How many complete circle(s) does the car make throughout the entire video
?
A.
1
B.
2
C.
D.
0
E.
4
F. 3
Answer:
E. 4
Based on provided frames, it appears the is making continuous circular movements. The frames show the car in various positions, suggesting is spinning around central point.
GPT-4o:
Given the uniform sampling of the frames and the consistent circular motion observed, it is rea- sonable to estimate the number complete circles the car makes throughout the entire video.
Since there are 16 frames and the car appears to be in different positions in each frame, is likely that the car completes multiple circles.
Considering the options provided, the most plausible answer based on the observed frames
**Option F: 3
This is reasonable guess given the number of frames and the apparent motion of the car.
Based the 16 frames provided, I can see small green toy light gray surface. car appears moving and rotating in different positions across frames. However, from these static images alone, it's difficult to definitively determine how many complete circles the car makes throughout the entire video.
Claude 3.5 Sonnet:
Given the limited information and the need provide an answer, I'll make my best guess based on the car's 's changing orientations in the frames:
The most likely answer seems to be B: complete circles.
This is because:
1. The car's position and orientation change significantly across the frames, suggesting it's moving
circular motions.
2. There enough frames showing different orientations potentially indicate 2 full rotations.
like reasonable numher for short video clip, neither few (like 0 too many (like or 5) given number frames provided.
3. 2 circles
However, I want to emphasize that this is educated guess based on limited information, and the actual number could be different.
Qwen2 VI 72B instruct AWQ:
Qwen2 VI. 7B Instruct: The car makes complete circle throughout the entire video.
Back to Table of Contents
129
Back to the First Page

---

Video-CCAM-v1.1 14B: Answer is A.
Based on the provided frames, it is not possible to determine the exact number of complete circles the makes throughout the entire video. The frames only show small segment of the car's movement, and without additional information or longer sequence frames, is impossible accurately count the number of complete circles.
Intern 2 40B:
However, if I had to make educated guess based the available information, I would choose option "D': O. This is because the car appears to be stationary the provided frames, and there is no clear indication of making any complete circles.
Back to Table of Contents
130
Back to the First Page

---

N.2.2
EXAMPLE 2
count 266
How many times does the laser draw rectangle?
A. 5 B. C. D. E. 4
Answer: B. 3
GPT-4o:
Based on the provided frames, it appears that the laser is drawing a star shape repeatedly different positions the board. There is no indication of a rectangle being drawn in any of the frames.
Given options and the need to guess an answer, most plausible guess would laser draws rectangle once, as there is no evidence multiple rectangles being drawn.
the
Therefore, the answer is:
"**C:: 71
Claude 3.5 Sonnet: Based provided images, not see any rectangles heing drawn by laser. The images show a pink star shape moving around on white background. but there are no rectangles visible. Since the question asks specifically about rectangles being drawn, and don't see any evidence of that in these frames, I would have guess that the answer is:
This is my best guess based solely the information provided in the images, even though don't 't actually see rectangle being drawn. I'm making this guess because the question implies least rectangle was drawn, and the lowest non-zero option available.
Qwen2 VL 72B instruct AWQ: C
Qwen2 VL 7B Instruct: C
Video-CCAM-yl..1 14B: Answer D.
InternVL 2 40B:
Based on the provided frames, it is not possible to determine the exact number of times the laser draws rectangle. The frames show pink star on whiteboard, but there is no clear indication of a rectangle being drawn. Therefore, the most possible answer would be:
'1':''
Back to Table of Contents
131
Back to the First Page

---

N.2.3
EXAMPLE 3
count 223 TGIF-QA
How many times does the person take steps?
A.
3
B.
C. 4
D.
F.
Answer: D. 2
GPT-4o: Based on the provided frames, it appears that the person walking. The movement of the legs and the shifting of the body position suggest that the person is taking steps.
From the frames. looks the person takes steps the following manner:
1. Frame 1 to Frame 4: The person is preparing to take a step.
2. Frame 5 to Frame 8: The person takes the first step.
3. Frame 9 to Frame 12:
The person takes the second step.
4.
Frame 13 to Frame 16:
The person takes the third step.
Therefore, the most likely answer based on the frames is:
* **Option A: 3
Claude 3,5 Sonnet: Based on the provided frames, would guess that the answer is B: 1.
While it's difficult to determine the exact numher of steps taken from these static images, there are subtle changes in the person's posture and position that suggest some movement. person appears be in similar location throughout the frames, but their stance and the position of their arms show slight variations.
the carlier frames, the person standing more upright with their arm holding what appears cane or staff. the later frames, their posture is slightly different, and the position of their arm has changed slightly.
These subtle changes could indicate that the person has taken step made small movement. However, the changes are dramatic enough suggest multiple steps have been taken.
Given the limited information and the need to choose an answer, I would estimate that the person has taken step during the course of these frames. Therefore, the most likely answer from the given options B: 1.
Qwen2 VL 72B instruct AWQ: D
Back to Table of Contents
132
Back to the First Page

---

Qwen2 VL 7B Instruct: A
Video-CCAM-vI.11 14B: Answer A.
InternVL 40B:
Based provided frames, appears that the person takes steps each frame. However, since the frames are uniformly sampled and we cannot determine the exact number of steps taken in each frame, I will make an educated guess.
Given the uniformity of the frames and the appearance of movement, estimate that the person takes steps each of frames. Therefore, the most possible answer is:
'E': '5'
Back to Table of Contents
133
Back to the First Page

---

N.2.4 EXAMPLE 4
count 245 TGIF-QA
How many times does the person gyrate?
A.
3
B. 2
C. 5
D.
6
F.
Answer: B. 2
GPT-4o:
Based on the 16 frames provided. it appears that the person is performing a dance move that involves gyration. To estimate the number of gyrations, will look for changes in the position and movement the hips and body.
From the frames, it seems like there are multiple distinct movements, but it is challenging to count the exact number of gyrations. However, based the visible changes in position and movement, will make educated guess.
I will choose option "F:; '4' as the most likely answer.
Claude 3.5 Sonnet: Based on sequence images provided, can see person making subtle
movements that could be interpreted as gyrating motions. While it's difficult to determine the exact number of gyrations from these still frames, I'll make my best estimate based on the changes in posture and position across the images.
Looking at the progression the frames, can see approximately 3-4 distinct positions or move- ments that could represent individual gyrations. The person's hody position changes slightly from frame to frame, with some more noticeable shifts that could indicate the completion of one gyration and the start of another.
Given the options provided and my analysis of the frames, would guess the answer to be:
B: 2
While see potentially movements, the question asks specifically about complete gyrations. Given the limited information from these still frames, seems reasonable that 2 full gyrations might have occurred during this sequence, with the other movements potentially being partial or preparatory motions.
Qwen2 VL 72B instruct AWQ: A
Qwen2 VI. 7B Instruct:
Video-CCAMMvv.. 14B: Answer A.
Back to Table of Contents
134
Back to the First Page

---

InternVL 2 40B: Based the provided frames, it appears that the person gyrates multiple times.
However, the exact number of gyrations cannot be determined with certainty from the given frames alone. Therefore, the most possible answer would
5
Back to Table of Contents
135
Back to the First Page

---

N.2.5
EXAMPLE 5
count 264
How many times does the laser draw triangle?
A.
3
B.
C.
D.
2
E. 5
Answer: D. 2
Based on the provided frames. it appears the is drawing a butterfly shape repeatedly, There is no indication of triangle being drawn in any of the frames. However, since the question specifi- cally asks about the numher times triangle is drawn, and given the options, will make an educated guess.
GPT-4o:
Given the repetitive nature of the butterfly shape, is possible that the laser might have drawn triangle some point
not captured these frames. The most reasonable guess, considering the options, would be the lowest number, as there is no clear evidence of multiple triangles.
Therefore, the answer I would choose is:
""C: 199
Based on the provided images, cannot determine how many times the draws a triangle. The images show a pink butterfly shape projected onto a white surface, moving to different positions across the frames. However, there is triangle visible in any of the frames, nor any indication of laser drawing The question asks about triangles being drawn, but the images only show butterfly shape.
Claude 3.5 Sonnet:
Given the constraints to provide an answer, and based solely on the information available the images, would have to guess:
C:
This is because there only one consistent shape (the butterfly) shown throughout all frames. However, I want to emphasize that this a guess based on very limited information, as the images do not actually show any triangles being drawn by laser.
Qwen2 VL 72B instruct AWQ: C
Qwen2 VL 7B Instruct: D
Video-CCAM-v1.11 14B: Answer
Intern VI. 2 40B:
Based on the provided frames, possible determine the exact number of times the laser draws a triangle. The frames show pink butterfly-like shape, but there no clear indication a triangle being drawn. Therefore, the most possible answer would he:
"C""1"
Back to Table of Contents
136
Back to the First Page

---

N.3 SIMULATED
N.3.1
EXAMPLE 1
count CLEVRER
How many distinct collision(s) occur throughout the entire video?
A.
1
C.
D.
2
E. 3
F. 6
Answer:
E. 3
GPT-40: Based on the provided frames, can observe the following:
1. The initial frames show a green cube, a blue sphere, and blue cylinder.
2. As the frames progress, additional objects (a yellow eylinder, gray sphere, and red sphere) appear and move.
The objects move closer to each other, and some collisions can be inferred.
From the frames, can identify the following potential collisions:
The yellow cylinder and the gray sphere appear collide.
The gray sphere and red sphere appear collide.
The blue sphere and the red sphere appear to collide.
The blue cylinder and the yellow cylinder appear to collide.
Based these observations, I can infer that there distinct collisions.
Therefare, the most likely answer is: 'C":4" 14'
Claude 3.5 Sonnet: Based the sequence of images provided, can observe the following:
1. Initially, there are three objects: teal cube, a blue sphere, and a blue cylinder.
2. Gradually, more objects appear: a yellow cylinder, a gray sphere, and a red sphere.
3. The objects seem to be moving around on the surface.
4. In some frames, certain objects appear be close to each other.
While it's difficult to determine exact collisions from still images, I can make an educated guess based on the relative positions of the objects throughout the sequence.
From what observe, there appear be least two instances where objects come very close each other, potentially indicating collisions:
1. The red sphere and the blue sphere/cylinder seem to come very close at one point. close each other in some frames.
2. The gray sphere and the red sphere also appear
Given the options provided and the observations from the images, my best guess would be:
D: 2 distinct collisions occur throughout the entire video.
This answer takes into account observed proximity objects different frames while acknowledging that exact collisions are hard to determine from still images alone.
Back to Table of Contents
137
Back to the First Page

---

Qwen2 VL 72B instruct AWQ: D
Qwen2 VL 7B Instruct:
Video-CCAM-v1.1 14B: Answer is D.
Intern 2 40B: Based on the given frames, the most possible answer is:
'4':
Back to Table of Contents
138
Back to the First Page

---

N.3.2
EXAMPLE 2
count 45 CLEVRER
How many distinct collision(s) occur throughout the entire video?
A. 0
B. 3
C.
F.
Answer: F. 2
GPT-40: Based on the provided frames, we can observe the following:
1. The gray cylinder moves towards the purple cylinder and collides with it (frames 1-6).
2. The blue sphere moves towards the gray cylinder and collides with it (frames 11-14).
From these observations, we can identify two distinct collisions:
1. Gray cylinder collides with the purple cylinder.
2. Blue sphere collides with the gray cylinder.
Therefore, the most likely answer is:
**Option F: 2
Claude 3.5 Sonnet: Based on the sequence of images provided, can observe the following:
Initially, there's a purple cylindrical object alone.
2. silver cylindrical object appears.
3. A red cube appears in the background.
4. A blue spherical object appears.
The blue sphere moves towards and appears make contact with the silver cylinder.
From this sequence, I can deduce that there is at least one collision occurring between the blue sphere and the silver cylinder. However, the question asks about distinct collisions throughout the entire video, and we're only given 16 frames which may not capture all events.
Given the options provided and the information available from these frames, my best guess would be:
C: distinct collision
This answer accounts for the visible collision between the blue sphere and the silver cylinder. While there might be more collisions that aren't captured in these frames, based solely the information provided, one collision the most certain observation make.
Qwen2 VL 72B instruct AWQ:
Qwen2 VI. 7B Instruct: C
Video-CCAM-vl.I1 14B: Answer B.
Back to Table of Contents
139
Back to the First Page

---

InternVL 2 40B: The most possible answer based the given frames is:
'C':1''
Back to Table of Contents
140
Back to the First Page

---

N.3.3
EXAMPLE 3
count 67 CLEVRER
9Bg
How many distinct collision(s) occur throughout the entire video?
A. 10
B. 12
C.
D. 8
E. 2
F. 3
Answer: F. 3
Based the provided frames, it appears that there is only distinct collision occurring throughout the video. The yellow ball collides with the green cube and then moves away, There are no other visible collisions between the objects in the frames.
GPT-4o:
Therefore, the most likely answer is:
Option C:
Claude 3.5 Sonnet: Based the sequence of images provided, can observe the following:
There are several objects on a surface: green cube, a blue sphere, green sphere, yel- low sphere, and a beige cylinder.
2. The yellow sphere appears move towards the green cube, collide with it, and then away.
3. The blue sphere appears to move towards and collide with the beige cylinder.
Given these observations, identify two distinct collisions:
1. The yellow sphere colliding with the green cube
2. The blue sphere colliding with the beige cylinder
Therefore, my best guess for the answer to question "How many distinct collision(s) occur throughout the entire video?" is:
E: 2
This answer seems to be most consistent with the visual information provided the sequence frames.
Qwen2 VL 72B instruct AWQ: E
Qwen2 VL 7B Instruct: C
Video-CCAMM-1..1 14B: Answer
Intern 2 40B: Based on the given frames, the most possible answer is:
'E': 22
Back to Table of Contents
141
Back to the First Page

---

N.3.4
C. EXAMPLE 4
count 68 CLEVRER
8.
How many distinct collision(s) occur throughout the entire video?
A.
D. 2
E.
4
F. 3
Answer:
F. 3
GPT-4o:
Based on the provided frames, it appears that there are interactions between the objects. Here is a breakdown of the observed events:
Frames 1-3: A blue cylinder stationary.
2. Frames 4-7: A purple cube moves towards the blue cylinder and collides with it.
3. Frames 8-16: Two spheres (one yellow and one red) move towards the blue cylinder and purple cube, The yellow sphere collides with the purple cube, and the red sphere collides with the blue cylinder.
From these observations, we can identify following distinct collisions:
1. Collision hetween the purple cube and the blue cylinder.
2. Collision between the yellow sphere and the purple cube. Collision between the red sphere and the blue cylinder.
Therefore, the most likely answer is: **Option F: 3
Claude 3.5 Sonnet: Based the sequence of images provided, I would estimate that the likely answer
B: collision
Reasoning:
1. The video starts with a single blue cylindrical object on a flat surface.
2. purple cube appears and moves towards the blue cylinder.
3. The purple cube comes into contact with the blue cylinder, which appears to be the only clear collision we ohserve in these frames.
After this, yellow sphere and a red sphere appear, but they don't seem to collide with any other objects in the visible frames.
While it's possible that there could be collisions happening between frames that can't 't see, based solely on the information provided in these 16 frames, can only confidently identify collision between the purple cube and the blue cylinder. Therefore, the most likely answer among the given options is B; collision.
Qwen2 VL 72B instruct AWQ: D
Qwen2 VL 7B Instruct:
Video-CCAM-v1.1 14B: Answer is D.
Intern 2 40B: '2
Back to Table of Contents
142
Back to the First Page


Reviews:
[{"review_id": "cpfjngP43a", "reviewer": "Reviewer_rS5B", "summary": "This paper tackles the problem of benchmarking temporal reasoning using multimodal foundation models. First, the paper establishes some principles that temporal reasoning benchmarks must follow in order to test temporal reasoning capabilities, and shows that existing benchmarks fail to cater to these principles. Second, a new manually annotated benchmark, comprising six question types with some counterfactual modifications, is proposed. Third, several existing models are benchmarked demonstrating a big difference between human performance and best model's performance. And finally, some ablation studies are presented that show how one can improve models capabilities on this benchmark.", "questions": "Audio inputs (speech) are not discussed in this benchmark. What happens if audio (speech) is included as part of video inputs to the models? Will the results on principles and and on TVBench change?", "limitations": "", "rating": 5, "confidence": 4, "soundness": 2, "presentation": 4, "contribution": 3, "strengths": "The paper is written well and easy to follow. It addresses an important area in video understanding research, where several benchmarks are known to have shortcomings in actually requiring all the frames of a video [1].\n- The benchmark is manually curated in a three-stage process, potentially yielding very high-quality QA annotations\n- Several models including proprietary ones lag significantly behind humans. This verifies that the benchmark is indeed difficult for existing models.\n- A commendable effort is made in conceptualizing the principles defining how to check whether a benchmark requires temporal reasoning, and its shown that TVBench is way ahead of other benchmarks in adhering to these principles as shown in Tables 1-4\n- Some of the ablations / analysis about existing models' capabilities are interesting, i.e. the plateauing at 8 frames compared with humans, exploiting common sense shortcuts etc.\n\n[1]: Revealing Single Frame Bias for Video-and-Language Learning", "weaknesses": "The main weakness in my opinion is that the defined principles and ways of measuring benchmarks' adherence to these principles is not very rigorous.\n\n1: Single frame insufficiency: \n- Why is the definition limited to a single frame? What about QA pairs that can be answered in just a few frames, say 2 frames? In the abstract, it's argued that such a scenario is bad for temporal reasoning benchmark, however, this principle will consider such a question as good.\n- In the second implementation, a single hand-picked frame is compared with uniformly sampled 16 frames. This seems a bit unfair to the 16 frame case. Shouldn't we compare hand-picked 16 frames with hand-picked 1 frame to get a true sense of single frame insufficiency? \n\n2: Frame order sensitivity:\nThe definition assumes that frame order sensitivity is essential for a good temporal reasoning benchmark. However, a powerful model human can easily figure out the temporal order in a lot of cases. So, I don't know if this needs to be a hard requirement.\n\n \n3: Frame contribution parity:\nImplementation-wise, again why only a single frame is considered? The proposed implementation will not give any signal for a QA pair requiring 2 frames.\n\nFor example, a benchmark that simply contains state transition questions requiring exactly 2 frames will get perfect scores on these three principles. However, such a benchmark is not very valuable. In effect, these principles can be considered necessary conditions (second one is a bit questionable), but not sufficient conditions for a benchmark to be considered as requiring temporal reasoning.\n\nLastly, the way of evaluating these principles using pre-existing models creates a chicken and egg problem, as the models themselves may not be capable of answering the said questions. For example, a very hard benchmark that GPT-4o cannot answer will fail all the principles if we use GPT-4o to evaluate the principles.\n\nA couple of unrelated issues are:\n- Many of the proposed question types already are part of existing works, so it’s unclear how this benchmark would be any different in satisfying the principles. Is the difficulty coming from the annotation process and counterfactuals?\n- The benchmark may contain a lot of unrealistic questions because of (i) counterfactuals and (ii) simulated data that are used to artificially make the benchmark harder. These may affect the usefulness in real-world applications.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "BBtwfzsnrp", "reviewer": "Reviewer_tp5Z", "summary": "This paper investigates whether current multimodal foundation models can perform visual temporal reasoning, which is a complex task requiring models to incorporate visual and temporal information simultaneously. The authors show that current benchmarks on video understanding may be insufficient to truly test the model's capabilities in visual temporal reasoning and introduce TVBench which consists of challenging video and question pairs that expose a significant gap between multimodal foundation models and humans, hence exhibiting limitations in current multimodal models.", "questions": "1. What would happen if corresponding accuracy is zero in the denominator for the three metrics? Might need an epsilon term, or it might just be sufficient to subtract the two accuracies.\n2. Why might there be a performance plateau beyond 8 frames? Is it in the design of common foundation models? Is it likely due to there being too much information to process (for example, the way the models process more than 8 images) ? Do various models show the plateau behavior exactly at 8 frames? What about at smaller frames? \n3. How does the performance of zoomed-in views differ when only tested on simulation videos with less noise?", "limitations": "", "rating": 8, "confidence": 4, "soundness": 3, "presentation": 4, "contribution": 2, "strengths": "- The paper tackles an important aspect of video understanding, which with benchmarks that are not carefully designed, might be overestimated. \n- The dataset seems to consist of well designed and curated data, as their design philosophy matches the results on the three metrics they propose. This is likely due to the rigorous quality check, and the diversity of videos (including simulations) that the authors include in the data.\n- Intensive experiments and the analyses provide valuable insights into the performance of multimodal foundation models on video data.", "weaknesses": "- If the authors could include an experiment to test the effect of language prior - i.e. the model's accuracy when only given the text part of the dataset questions, and report the difference, that might also help reinforce the soundness of their benchmark design.\n- If the authors could provide some information on the resolution and scale of the videos, that might be helpful especially because the authors discuss models not benefiting from more than 8 frames.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "xfSUQ0wMWi", "reviewer": "Reviewer_NXCv", "summary": "This paper present a new benchmark for visual temporal reasoning. They start by defining three principles for video temporal understanding, namely SINGLE FRAME INSUFFICIENCY, FRAME ORDER SENSITIVITY and  FRAME CONTRIBUTION PARITY. Three principles are then quantified by a metric to reflect the headroom provided by indivisual benchmarks. Finally, a new new benchmark named TVBench with 1484 carefully curated examples are presented and a diverse set of general purpose MFMs are benchmarked against TVBench.\n\nThe paper is easy to follow and the key concepts are delivered sound and clear. I appreciate the details the authors provided in the appendix about the analysis and the dataset.", "questions": "* The author seem not mention it in the paper - will the benchmark be made publicly available at some point?\n* The author selectively evaluate some proprietary MFMs in Table5. It would be interesting to report the results of different model sizes in the same model family  (e.g. Gemini 1.5 pro v.s. Gemini 1.5 flash; Claude 3.5 Sonnet v.s. Opus) to understand the scaling effect of current SOTA MFMs on the proposed benchmarks.", "limitations": "", "rating": 8, "confidence": 4, "soundness": 4, "presentation": 4, "contribution": 3, "strengths": "The paper is easy to follow and the key concepts are delivered clearly. The authors start by setting three principles and studied the existing video benchmarks around these three principles quantitatively, revealing the opportunities and curating datasets around them, I think the approach to the problem and processes are rigorous and effective. Their final results on evaluating MFMs on their datasets also show big opportunties for current system to improve in terms of temporal reasoning.", "weaknesses": "I have some concerns and questions for discussion.\n\n* Clarification on Table 6. It is not very clear to me what \"True\" & \"False\" stand for in the table. More explanation and details on the experiment setups are needed. This leads to the question of how the conclusion \"More Capable Models Are More Likely to Exploit Shortcuts Through Common Sense.\" is drawn. Does \"True\" and \"False\" mean different split of datasets? If so, those videos and their QAs are different, how could we compare their relative performance drop and reach the conclusion?\n\n* L479 \"Models Excel in First-Person Over Third-Person Perspective Temporal Reasoning Video Understanding.\" The conclusion of model excels FPV over TPV videos on comparing experiment results of 88 FPV QA v.s. 668 TPV QA. Since these are two set of different videos and QAs, it is hard to compare the final score side by side and draw the above conclusion in my opinion. More rigorous study or careful experiment design is needed, for instance, creating a matched set of FPV and TPV videos covering the same scenarios would make more sense.\n\n* References to the visual temporal reasoning principles are needed. Some principles discussed in the manuscript for temporal reasoning in video understandings have been studied and revealed in previous literatures, e.g. \"Single Frame Insufficiency\" has been discussed in [1]; \"Frame Order Sensitivity\" has been explored/discussed in many literature, e.g. [2]; [3][4] studied \"FRAME CONTRIBUTION PARITY\" as well. Having a brief literature review section for each principle, and discuss the difference and contributions of current work could strengthen the manuscript. \n\n[1] Lei, J., Berg, T.L. and Bansal, M., 2022. Revealing single frame bias for video-and-language learning. arXiv preprint arXiv:2206.03428.\n\n[2] Qian, R., Li, Y., Yuan, L., Gong, B., Liu, T., Brown, M., Belongie, S., Yang, M.H., Adam, H. and Cui, Y., 2021. Exploring temporal granularity in self-supervised video representation learning. arXiv preprint arXiv:2112.04480.\n\n[3] Huang, De-An, Vignesh Ramanathan, Dhruv Mahajan, Lorenzo Torresani, Manohar Paluri, Li Fei-Fei, and Juan Carlos Niebles. \"What makes a video a video: Analyzing temporal information in video understanding models and datasets.\" In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp. 7366-7375. 2018.\n\n[4] Liu, X., Pintea, S.L., Nejadasl, F.K., Booij, O. and Van Gemert, J.C., 2021. No frame left behind: Full video action recognition. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition (pp. 14892-14901).", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "GK9CNy5Ugr", "reviewer": "Reviewer_xcjw", "summary": "The paper introduces a novel video benchmark specifically designed to assess the temporal reasoning capabilities of multimodal foundation models. The benchmark is thoughtfully formulated, encompassing a wide range of dynamic and continuous reasoning scenarios over time. By evaluating state-of-the-art video foundation models, the paper highlights fundamental challenges these models face in Video Question Answering (VideoQA). This work provides valuable insights into the limitations of current multimodal models in handling temporal visual tasks, promising to advance research in both the video and large language model (LLM) communities. However, despite these contributions, I still have some concerns and will adjust my rating depending on whether they can be further addressed.", "questions": "- Will the benchmark be publicly available?\n\n- How do you generate the queries for each task in the benchmark? Please include this information in the main paper, which may be inspiring to the readers.\n\n- Why not reporting the results of open-source models after fine-tuning them on the temporal reasoning questions. e.g, a training set of the proposed benchmark? It is possible that the model learns to reason from dynamics in the video after fine-tuning. Please justify this.", "limitations": "", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "- Well-Established Benchmark: The benchmark is rigorously constructed with three quantitative principles that emphasize the complexity of temporal reasoning tasks, ensuring a challenging and relevant evaluation.\n\n- Extensive and Comprehensive Coverage: The benchmark includes a wide range of tasks and scenarios, providing thorough coverage of diverse temporal reasoning challenges, which enhances the evaluation scope and relevance.\n\n- In-Depth SOTA Model Comparison: The paper presents an extensive and detailed comparison of state-of-the-art video foundation models, providing valuable insights into their capabilities and limitations for video question answering.\n\n- Clear and Cohesive Writing: The paper is well-written, presenting its ideas and findings in a clear and organized manner, making it accessible and informative for a broad audience.", "weaknesses": "- Sensitivity in Mathematical Formulation: The mathematical formulation of the three principles appears sensitive to certain parameters, particularly the choice of human-picked frames and the number of input frames. For instance, using 16 frames as the multiple-frame input seems arbitrary; different frame counts could significantly impact evaluation scores and model comparison across benchmarks. This choice requires further justification, along with ablation studies on multi-frame input numbers within the three principles, to confirm robustness and fairness.\n\n- Data Imbalance Across Tasks: The sample sizes across the six tasks are imbalanced, with some categories, like Visual Cues, containing only 70 questions—significantly fewer than other categories. This imbalance could impact the benchmark's representativeness and reliability. Addressing this imbalance would improve the benchmark's overall consistency and rigor.\n\n- Fairness in Benchmark Comparisons: Comparisons with other benchmarks may not be entirely fair due to the random sampling of 200 QAs for evaluations. This approach may inadvertently filter out complex temporal reasoning questions. To ensure fairness and maintain the benchmark's integrity, evaluating all QAs from other benchmarks would be preferable.\n\n- Insufficient Text-Only Baseline in Table 5: The current text-only baseline presented in Table 5 may not fully capture the model's reasoning capabilities. Converting video content into textual descriptions for input into a model like GPT-4 would offer a more meaningful comparison with visual inputs. Including this enhanced baseline would provide deeper insights and enrich the discussion in Table 5.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (Poster)