PhysBench: Benchmarking and Enhancing Vision-Language Models for Physical World Understanding
Authors: Wei Chow
Jiageng Mao
Boyi Li
Daniel Seita
Vitor Campagnolo Guizilini
Yue Wang
Venue: ICLR 2025 Oral (2025)
Keywords: vision-language
multi-modal understanding
Primary area: datasets and benchmarks
TL;DR: We propose PhysBench to evaluate VLMs' physical understanding, highlighting their limitations and introducing PhysAgent to enhance VLMs' physical understanding.

Abstract: Understanding the physical world is a fundamental challenge in embodied AI, critical for enabling agents to perform complex tasks and operate safely in real-world environments. While Vision-Language Models (VLMs) have shown great promise in reasoning and task planning for embodied agents, their ability to comprehend physical phenomena remains extremely limited.
To close this gap, we introduce PhysBench, a comprehensive benchmark designed to evaluate VLMs' physical world understanding capability across a diverse set of tasks. 
PhysBench contains 10,002 entries of interleaved video-image-text data, categorized into four major domains: physical object properties, physical object relationships, physical scene understanding, and physics-based dynamics, further divided into 19 subclasses and 8 distinct capability dimensions.
Our extensive experiments, conducted on 75 representative VLMs, reveal that while these models excel in common-sense reasoning, they struggle with understanding the physical world---likely due to the absence of physical knowledge in their training data and the lack of embedded physical priors.
To tackle the shortfall, we introduce PhysAgent, a novel framework that combines the generalization strengths of VLMs with the specialized expertise of vision models, significantly enhancing VLMs' physical understanding across a variety of tasks, including an 18.4\% improvement on GPT-4o.
Furthermore, our results demonstrate that enhancing VLMs' physical world understanding capabilities can help embodied agents such as MOKA.
We believe that PhysBench and PhysAgent offer valuable insights and contribute to bridging the gap between VLMs and physical world understanding. [Project Page is here](https://physbench.github.io/)

Published as a conference paper at ICLR 2025
PHYSBENCH: BENCHMARKING AND ENHANCING VISION-LANGUAC MODELS FOR PHYSICAL WORLD UNDERSTANDING
Wei Chow Jiageng Mao" Boyi Li2, Daniel Seita', Vitor Guizilini", Yue Wang' University Southern California, PCC Berkeley, Toyota Research Institute
ABSTRACT
Understanding the physical world is fundamental challenge in embodied AI, crit- ical for enabling agents to perform complex tasks and operate safely in real-world environments. While Vision-Language Models (VLMs) have shown great promise in reasoning and task planning for embodied agents, their ability to comprehend physical phenomena remains extremely limited. To close this gap, we introduce PhysBench, comprehensive benchmark designed to evaluate VLMs' physical world understanding capability across diverse set of tasks. PhysBench contains 10,002 entries of interleaved video-image-text data, categorized into four major domains: physical object properties, physical object relationships, physical scene understanding, and physics- based dynamics, further divided into 19 subclasses and distinct capability dimensions. Our extensive experiments, conducted on 75 representative VLMs, reveal that while these models excel in common-sense rea- soning, they struggle with understanding the physical world- likely due to the ab- sence of physical knowledge in their training data and the lack of embedded phys- ical priors. To tackle the shortfall, we introduce PhysAgent a novel framework that combines the generalization strengths of VLMs with the specialized expertise of vision models. significantly enhancing VLMs' physical understanding across a variety of tasks. including an 18.4% improvement on GPT-4o. Furthermore. our results demonstrate that enhancing VLMs' physical world understanding ca- pabilities can help embodied agents such as MOKA. We believe that PhysBench and Phys/Agent offer valuable insights and contribute to bridging the gap between VLMs and physical world understanding. Project Page is here
INTRODUCTION
Understanding the physical world is fundamental challenge in embodied AI (Gupta et al. 2021
Srivastava et al. 2021). Embodied agents are required to understand the physical properties of objects (e.g., mass, stiffness) accurately interact with these objects. They also need to understand the relationships of physical objects to operate efficiently in cluttered environments, understand the structure of physical scenes for safe navigation and manipulation, and anticipate the outcomes of interactions and physics-based dynamics for better planning and preventing accidents. These capabilities of intuitive physics (McCloskey et al. 1983 Carey 2000) are innate humans and can also greatly benefit embodied agents, allowing them to perform complex tasks and operate safely in real-world scenarios (Kill Kim 2020).
Vision-language models (VLMs) (Liu et al. 2024c Achiam et al. 2023 Team et al. 2023) have emerged as promising solutions for building embodied agents (Liu al. 2024a; Nasiriany et al.) 2024 Huang et al.) 2023a). Trained on large amounts of human knowled wewlegee these models have
developed strong capabilities in reasoning and task planning (Yue et al.) 2024 Lu et al.) 2024b Kim et al. 2024 Niu et al.) 2024 Zhen et al. 2024). However, relying solely these capabilities is insufficient for developing generalist embodied agents. A series studies have highlighted a gap in understanding the physical world, leading to operational errors (Liu et al. 2024a), such as mishandling fragile objects (Wang et al. (023c) or failing to recognize appropriate grasping affor- dances (Guo al., 2024). Since these agents operate in and interact with the real world, VLMs must possess a comprehensive understanding of the physical world-a critical yet underex- plored domain. This deficiency in physical world understanding limits the effective deployment of
VLMs in embodied applications (Liu et al. 2024a
Guo et al. 2024
Gao et al. 2024a)
"qqual contribution.

---

Published as a conference paper at ICLR 2025
Common VQA
Physical Object Property
Physical Object Relationships
Q Which object has greater classicity?
Q What is the object closest the teacup in the Figure?
A. Green ball B. White ball
A. The pastry
B. The pench
C. Same clasticity
C. The knife The spoon
D. Cannot determine
Physical Scene Understanding
A Physics-based Dynamics
What are things should he cautious about when visit here?
A When visiting the pier over the lake, there few things you should cautious about. First, casure that you
How does the viewpoint alter?
Q Which object will the cart hit first? red cube / B. blue
A. Moves downward
B. Rotates the right
Moves upward
Rotates upward
C. The green cube
Db. The yellow eube
Figure I: Common VQA tasks typically involve questions about visual content and general knowl- edge. PhysBench emphasizes understanding the physical world, encompassing 4 dimensions.
To further investigate this issue, we pose two fundamental questions: (1) Do VLMs possess an understanding of the physical world, and if not, what factors contribule to this limitation? (2) How can we enhance VLMs' physical world understanding capabilities and facilitate the effective deployment of embodied agents like MOKA?
To answer the above questions and comprehensively assess the extent of the gap between VLMs and physical world understanding, we introduce PhysBench, dataset comprising 10,002 inter- leaved video-image-texy entries. Given the difficulty of acquiring such data, where expressing spe- cific properties often requires multiple images, we undertook five-step process, spending a total of 4,000 hours on annotation. We systematically evaluate 75 representative VLM across four do- mains phytsial object properties, physical object relationships, physical scene understanding, and physics-based eynompis-encommpasiin encompassing 19 sub-tasks, as shown in Figure []] Our extensive exper- iments reveal that (1) most current VLMs exhibit poor understanding of the physical world, par- ticularly in physical scene understanding and physics-based dynamics, with closed-source models significantly outperforming open-source ones; and (2) the training data for VLMs is likely a major factor contributing to their subpar performance, as it often lacks the necessary physical knowledge. Notably, when VLMs were fine-tuned on our physically grounded data, their performance improved.
To further improve VLM's physical world understanding capabilities, we propose PhysAgent, a unified framework that incorporates vision foundation models and physics knowledge memory. By analyzing the sources of errors for VLMs on PhysBench, we identified perceptual inaccuracies and insufficient knowledge as the primary causes of mistakes. To address these issues, we incorpo- rated vision foundation models to enhance perceptual capabilities and assist VLMs in handling tasks they typically struggle with, such as depth estimation and numerical distance calculation. Addition- ally, we integrated a knowledge memory module to embed essential knowledge about the physical world, which can be selectively invoked by PhysAgent. Unlike previous methods designed for phys- ical reasoning (Zheng et al. 2024b; Tung et al. 2023), PhysAgent retains the strong generalization abilitics of VLMs and their capacity to solve open-ended problems, without relying on manually predefined processing logic or being limited to specific tasks. Experimental results demonstrate that PhysAgent improves GPT-4o's zero-shot performance on PhysBench by 18.4%. Furthermore, we investigate how physical world understanding helps the deployment of embodied agents through extensive robotic manipulation experiments on MOKA (Liu et al.) (024a).. Specifically, we employ two approaches: fine-tuning the VLM with PhysBench and utilizing PhysAgent for zero-shot infer- ence across five representative manipulation tasks. The improvement in those tasks further validates that PhysBench and PhysAgent can facilitate the deployment of embodied agents like MOKA.
We hope this work offers valuable insights and contributes to bridging the gap between VLMs and physical world understanding, ultimately advancing embodied Al toward human-level capabilities. In summary, this paper has two technical contributions: (1) We present PhysBench, a large-scale benchmark for evaluating the performances of vision-language models in physical world understand- ing. We identify the key challenges through extensive studies and provide insights into why the exist- ing VLMs have insufficient physical world understanding capabilities. (2) We propose PhysAgent, unified approach that improves VLMs' physical world understanding abilities. Through extensive experiments, we demonstrate that enhancing VLMs' comprehension of physical environments can significantly facilitate the deployment of embodied agents.
2

---

Published as a conference paper at ICLR 2025
2 RELATED WORK
Physical Comprehension Datasets. Early benchmarks (Riochet et al., 2018 Rajani et al. 2020) were developed primarily for vision-only models, while more recent efforts (Yi et al. 2019 Chen ict al. 2022: Wang et al. 2024g) have predominantly focused on simple visual primitives, such as spheres, cubes, and rigid object collision events, often restricted to limited set of simulated sce- narios (Zengg et al., 2024b; Tung et al. 2023). We summarize the key features of these various benchmarks and compare them against
benchmark in Table [ However, existing VQA datasets assessing physical knowledge (Lu et al. 2022 He et al. 2024) mainly focus on commonsense rea- soning rather than physical world perception. Spatial VQA benchmarks (Chen al. 2024a Lyu et al. 2024 Bonnen et al. 2024 Wang et al. 2024d) emphasize geometric relationships in 3D sence, which represent only part of the physical. In contrast, PhysBench is the first comprehen- sive dataset designed to evaluate models' understanding of the physical world, encompassing a wide variety of scenarios and tasks not covered by previous benchmarks.
Physical Reasoning Models. Models for understanding the physical world generally fall into two categories. The first comprises
models (Guen Thome 2020; Duan et al.] 2022). which are typically limited to predicting the next state and are not applicable to other tasks. The second includes physical oracle models (Zheng et al., 2024b Tung et al. 2023), which are suit- able for only narrow range of tasks due to their reliance require training additional modules like R-CNN, and their probabilistic outputs restrict them to clas- sification tasks, limiting their ability to handle open-ended questions. In contrast, Phys/Agent offers greater flexibility and adaptability across broader spectrum of problems without these limitations.
predefined rules. These models often
Vision-Language Models. Vision-Language Models (VLMs) are large-scale models that integrate visual modalities with language understanding (Wu et al. 2023b Zhan et al.) 2024 Dai et al.) (024). In recent years, there has been surge of work leveraging VLMs agents for embodied AT (Liu
et al. 2024a Nasiriany et al., 2024). Although these approaches are generalizable, they face chal- lenges due to weak physical world understanding capabilities (Liu et al. 2024a Guo et al. 2024). By employing PhysBench and PhysAgent, these shortcomings can be mitigated, enhancing the physical world understanding capabilities of VLMs and enabling more reliable robotic control. Additionally, spatial VLMs (Bonnen et al. (024) have identified that most VLMs lack 3D spatial reasoning capa- bilities due to insufficient data. However, since spatial reasoning represents only a subset of physical world understanding, our work aims to provide more comprehensive evaluation and improvement of VLMs' physical world understanding abilities. For additional related work, see Appendix G
Table 1: A comparison between PhysBench and other physical understanding question-answerte benchmarks. PhysBench is a comprehensive dataset, covering a wide range of tasks related to phys- ical world understanding.
Phipery
Atrilen
Lootin
Mediee
Torpprmuo
Virapiii
Tight
Crilldie
Masipistton
Tlit
Inselennd
Qte
Viiv cit
V.taa po19 Suly
3,311
X
X
8
..188
Canity
Myyam
2
7
17.200
Mlysen
X
2.000
supetii.
1.301
Itollaad
10/00
PHYSBENCH
To assess VLMs' physical world understanding ability, first define the concept of physical world
understanding and introduce PhysBench in Section 3.1 Next, we provide a detailed description of the data collection process in Section [3.2] Utiliing PhysBench,
conduct experiments to determine whether VLMS can effectively comprehend the physical world in Section 3.3 Finally, in Section 3.4 discuss the potential reasons for poor performance.
3.1 OVERVIEW OF PICYBENNC
Understanding the physical world is essential yet fundamentally challenging for embodied AI, as systems must perceive, interpret, and predict the properties and dynamics of objects and environ- ments. This involves comprehending object properties and relationships, interpreting environmental scenes, and anticipating interaction outcomes based on visual cues and core physical principles to ensure safe and effective operation.
3

---

Published as a conference paper at ICLR 2025
Physical Object Property
1 Physical Object Relationships
Y Distance
What distance yellow (The blue cabe has width of ems, A About Fem.
bue
Number has the largest numbhrr? A Bize balls.
Y Location
9 egg? A Mustrooms.
de Depth
Q Which marked object
closest to the camera! Option
TOe Attribure
Mass
Q
appliad
Q What the mase relationsiip between the three ping pong balls??
furce is the same, which
object the images has higher stiffness?
* Size larest
of
Q Which has higher speed?
ping-pong halls is adantical.
$ Velocitty
image.
lettmost speccuum! Red.
Physical Scene Understanding
Physical-based Dynamics
Throwing
+ Cullision
O Which can is the ball
Q Which soonc, depieted in the imegos, occurs first?
most likely land in?
? Temperature
12 Viewpoint
+ Light
Manipalation
Q the phenomenon obeanved in the video cold hat water? A Hott water.
How does the focal change!
O How might the light source
the image have charged!" appears have shilled image the right side.
soguence ofineges make gilt box contaning the perfume botte?
Q What
A The Socal longth incrosecs.
fins, image followed image finzily image
Pigas
Piguus
Figus
z Air Pressure
What the change combuution
t Fluid
cup?
Which object the
viscosity?
air presuue the cup.
liquid.
Figure 2: Sampled PhysBench examples from four major dimensions mentioned in Section 3.1 Duc space constraints, present only the correct answers (as each question in our dataset four- option multiple-choice with one correct answer) and defer additional examples to Appendix
However, existing datasets often focus solely image content and commonsense reasoning. glecting the four fundamental aspects of the physical world mentioned above. To address this gap, we propose PhysBench, which comprehensively evaluates VLMs perception the physical world across four major task categories of the physical world: (1) Physical Object Property: Assessment of physical attributes of objects such as mass, size, density, tension, friction, bending stiffness, elasticity, and plasticity. (2) Physical Object Relationships: Evaluation of spatial relationships in- volving objects' relative or absolute positions and motions. (3) Physical Scene Understanding: Interpretation of environmental factors, including light sources, viewpoints, temperature, elc. (4) Physics-basee Dynamics: Understanding of physical events like collisions, throwing, fluid dynam- ics, explosions, and other phenomena. Each category corresponds
specific sub-task types and ability types, whose distributions are shown in Figures 3 Detailed examples of specific tasks are illustrated in Figure 2 with additional examples provided Appendix H. comprehensive descrip- tion of sub-task types and ability types is available Appendix
PhysBench is structured as multiple-choie questionnaire, presenting four options for each ques- tion, with only one correct answer. The primary statistics of PhysBench are presented in Table and detailed in Appendix D Recognizing that different types of tasks possess unique characteristics. we utilize videos and multiple images to effectively convey features that are difficult to capture in single image-stte such as elasticity, mass, density, and environmental factors like temperature, hu- midity, light source, and viewpoint. The dataset also includes objects with similar initial states but differing properties, leading to different future outcomes. This enriches the dataset and allows for wider range of observable physical behaviors. Consequently. PhysBench draws its data from the internet, real-world captures, and simulations, making a mixed-formi benchmark that integrates text, images, and videos. For convenience, PhysBench-test consists of 10,002 entries, organized into 19 subclasses, as the test set, and 200 entries as the validation set for parameter choosing. We also present 89,998 entries for further research. The experimental results presented in this paper, un- less otherwise specified, are based on the test set. The performance VLMs on PhysBench-vall can be found in Appendix F.4 Benchmark release details can be found in Appendix B.8

---

Published as a conference paper at ICLR 2025
others
Bud
Statistie
Nomaeer
Alid
percoption
Total questions
10,002
attribute
collision
cemparioon
only one image
1,766
only one video intesfeave
2,749 (44.8%)
dynamics
aynamics
prediction
1,902 (18)
number
property
mangguattion
property
Unique mumber of inages
10,058
identify
Unique number videos
3,260
678
color
mass
scene
aocation
depth
scene
dynanic
Maximani leigth
4%
light
viewpoint
motion
reasonin
static
Maximum choice length
20
Average question length
16.5
Averae choice longth
4,4
Table 2: Key statistics.
Figure 3: Subtype distribution and ability distribution
3.2 DATASET COLLECTION PROCESS
To ensure data quality, all questions were manually annotated by graduate students in STEM fields and further refined through rigorous review process after collecting and clipping the raw images or videos. To maintain consistency in annotations, we implemented multiple rounds of cleaning and validation throughout the following steps. We have preserved intermediate outputs from the annota- tion process, such as depth and reflectance maps for sinulator-generated data and human-annotated physical principles for many web-sourced videos. The process involves the following sequential steps: (a) Video Collection. Videos and images are gathered from web searches, simulations, and real-world captures. The collection process uses predefined simulation rules, LLM-guided queries, and other strategies to find related images or videos (see Appendix AA.. Human annotators further refine the data by clipping and annotating physical principles in the images or videos. (b) Video Captioning. Human- annotated raw videos are processed through automatic filtering, followed by GPT-4o annotations that generate captions with human check. (c) Questions Design. For videos annotated with physical principles, we generate physics-related questions using both manual design and GPT-4o, following predefined rules. An automated filter and manual review processes eliminate irrelevant questions. (d) File Organization. The remaining valid questions are categorized by task, sub-task, and ability type by human experts. (e) Quality Check. The organized dataset undergoes human review to ensure that the questions are physical world relevant, rely on all input information, are not grounded in common sense, and are accurately categorized with clear questions and corre- sponding answers. Due to space limitations, the collection guidelines are provided Appendix B
3.3 CAN VLMS UNDERSTAND THE PHYSICAL WORLD
To assess whether VLMs can understand the physical world, we evaluated 75 representative VLMs on PhysBench and found that a significant performance gap remains between VLMs and human- level understanding. The primary results are presented in Table while detailed analyses of sub-task performance and ability types across the four task categories are provided in Appendix F33
Setup. Our evaluation was conducted under three configurations: (a) Image VLMSs, which support only single-image input (e.g., LLL and BLIP-2);; (b) Video VI.Ms, designed for video compre- hension (c.g., Chat-UniVi and PLLaVA); and (c) General VLMs, which support multiple images and interleaved inputs (e.g., VILA-1.5 and GPT-4o). It is important to note that the data used for eval- uating setups (a) and (b) subset of PhysBench test subset with interleaved QA pairs removed, whereas setup (c) was evaluated on the full dataset. For most models, we followed the standard protocol outlined in VLMEvalKit Contributors (2023), setting the temperature O. For models that do not support multiple images input, we employed two methods: the merge method, where video frames are concatenated into a single image (Fu et al., 2024: Zhang et al. [024a] [aang al.) 2024), and the seq method, where video frames input sequentially as individual images. Notably, only models using the seq setup can handle interleaved text-image sequences. For details on VLM prompts and hyperparameters,
Appendix B
VLMs exhibit a limited understanding of the physical world. Our evaluation indicates that most models achieve an average accuracy of approximately 40%, which is significantly below human- level performance. Even the pert-performing model, GPT-4o, attains only 49.49% accuracy, under- scoring the substantial gap between current VL.Ms and true comprehension of the physical world.
5

---

Published as a conference paper at ICLR 2025
1010
160
100
199
Figure 4: (a) Correlation map between tasks in PhysBench and 15 other vision- Lasion- bench-
marks. (b) The visualization of model performance across 19 sub-tasks is presented, where different colors represent the respective categories. The four colors, from left to right, represent physical
object property, physical object relationships, physical scene, and physical- dynamics.
Size Format Property
Relationships
Scene
A Dynamics
Avg
Random Choice
25.00
25.00
25.00
25.00
25.00
Humma
97.10
95.67
94.91
95.68
Image VLM
Instructi5LIP-ER
Dai 2024
4B
merge
35.35
36.67
37.45
35.95
36.24
InstructBLIP LIP-IBLIIP----000
Dit 2024
12B
merge
41.11
38.47
37.89
36.42
38.51
InstrucBLLIP-70 DDaeeta 2024
7B
merge
21.94
29.00
19.53
27.45
23.82
InstruciBLIP 13-
merge
31.69
33.19
23.13
30.64
BLIP-2 Li al. 202.50
12B
merge
41.70
40.83
36.25
36.93
38.61
LLaVA-I.3-7B
al. 2023a
7B
38.44
41.53
38.60
42.69
40.09
LLaVA-1.5-131B (Lu all
207.3
13B
menge
41.31
42.50
34.40
44.38
40.45
LLaVA16-mistte [Lau al..
2024b
7B
merge
29.77
22.22
20.58
20.30
L.VAAI.6vvicua
20224
merge
59.72
38.60
42.65
42.28
Qwen-VI--Cat (Bar
9B
merge
35.97
43.33
26.47
41.27
35.63
IntemVLLCChtt..5 Chen
2024c
26B
53.08
70.14
37.01
44.78
47.51
Cambrian-8B Tong af 2024
8B
merge 23.27
17.92
23.02
29.29
24.61
Clande-3--pus Anthropic
merge
31.97
40.97
30.63
36.50
37.00
Claude-3-sonee Anthroppi 2124
merge
37.86
40,00
32.23
36.89
36.18
Claude-3-huiku Anthropic
merge
43.28
53.33
30.06
39,93
39.44
Claude 3.5seeet (Anthropic
2024
46.46
41.11
27.89
37.60
38.05
Vidco VLM
Video-LLaVA LLn et
202.3g
7B
seq
36.82
36.11
33.69
40.52
37.04 04
Chat-Univi-7B Uin al.
seq
19.28
20.97
28.46
22.19
Chat-Univi-1 Um
2023
13B
sey
4.30
11.53
15.67
11.47
10.36
PLLaVA-7B Xu
2024
7B
seq
38.02
35.83
36.34
39.89
37.94
PLLaVA-13B Xuu et
2024
13B
39.91
38.33
31.52
40.76
37.70%
General VLM Interleaved data
LLaVA-intereeave (Li et 2024d
7B
sey
47.23
44.62
35.64
37.21
41.00
LLaVA-imeerleave-rrg
al.
2024d
7B
seq
47.97
42.67
33.73
38.78
(Lin VILA--15-3B
seq
32.40
33.02
34.84
35.78
34.11
VILA-I15-3B--- (Ln c.
3B
33.14
30.26
35.72
33,00
33.07
VILA-115-8B (Ln et a. 2023b
8B
seq
33.41
29.88
30.85
35.91
32.85
VILA-1.5-13B Lin al.
20235
seq
40.53
40.15
31.96
36.07
37.15
Phi-3V (Abdin
seq
43.67
37,92
34,93
36.92
38.42
Nv /hang e
2024h
7B
seq
38.33
30.83
34.00
37.17
35.42
LLaVA-NV-apo Zhang et al.
2024b
7B
seq
38.83
44,31
33.86
37.21
37.43
Idetis- (lang ala 2024
seq
41.97
41.44
29.53
36.56
37.39
aaV (hang
seq
44.48
30.45
34.73
36.69
Mantis-siglip-llam Urang
2024
8B
sey
42.47
32.78
36.83
37.51
37.64
Mantis-clip-llamat (Jiang el
2124
8B
40.61
35.11
32,45
38.36
36.92
GPT-4V (Achiam et al. 2025
49,59
45.77
26.34
42.15
41.26
GPT-4o (Achiam el 2023
sey
56.91
64.80
30.15
46.99
49,49
GPT-40-mm (Achiam
sey
53.54
44.24
30.59
42.90
43.15
Gemini-1.5-ftaag
seq
57.41
$2.24
34.32
40.93
46.07
Gemini-1.5-ppn p.5-pro TTam
20233
seq
57.26
63.61
36.52
41.56
49.11
Table 3: Evaluation results for 39 VLMs. The evaluation of General VLMs is based on the data
from Video and Image VLM evaluations, with the addition of interlcaved data. "Seq" refers to sequential input of frames of videos, while "merge" refers to merging video frames into a single image. Bold indicates the best result, and underline indicates the second best each group.

---

Published as a conference paper at ICLR 2025
As shown in Figure 4 b), considerable room for improvement remains, particularly in tasks related to physical scene understanding and physics- based dynamics.
Closed-source models generally perform better. As shown in Figure 5 b). the GPT series and Gemini-1.5 models significantly outperform open-source models. Notably, GPT-4 surpasses the best open-source model, LLa VA-interleave, by 20.7%, indicating substantial gap between open- source and flosed-source models. However, we did not observe a clear advantage with Claude, a
finding that aligns with results from other benchmarks (Cao al. 2024
Wu al. 2024c).
digtt
Ioction
$.00
dolanee
OF14c
49.49
odlor
46.07
Proprettory
Jarpene
48.99 15
Avg
Veepoont
41.0
ctwn
37.64
Open
chemidtyy
VL-4 1100
37:15
35 Aog. 37
17.03
LLAWN NATT Wires EIR
call mCc
Perormarce
Figure 5: (a) The performance of 8 representative open-source General VL.Ms across 19 sub-tasks in PhysBench, which support interleaved inputs. The closer it to the circular boundary, the better. (b) The overall performance of those 8 VLMs. Closed-source models generally perform better.
3.4 WHY Do VLMS STRUGGLE WITH PHYSICAL WORLD UNDERSTANDING
To further investigate why VL.Ms struggle with physical world understanding. we analyzed Phys- Bench and discovered that it differs significantly from common VQA tasks. Additionally, we found that the performance of larger model size or more training data does not result in clear improvements on PhysBench, which may be due lack of physical world knowledge in the training data. Fur- thermore, we found that many errors stem from this deficiency; when we augmented the models with physical world knowledge, their performance improved. This further suggests that the gap between VLMs and physical world understanding may be attributed to limitations in the training data.
Physical world understanding differs significantly from common VQA tasks. To assess the relationships between our tasks and other VL.M benchmarks, we adopted the methodology proposed by (Tong al. 2024 Fang et al. 2024) to construct a correlation map. shown Figure 4(a). Details on the construction of the correlation map are provided in Appendix F.6 Our analysis reveals that PhysBench differs significantly from traditional VLM benchmarks, exhibiting closer alignment with POPE (Li et al.) (203h) in tasks such as hallucination detection, while also showing that performance does not consistently improve with increased data or model scale.
VLMs's physical world understanding ability does not scale with model size, data, or frames. (1) Model Size Scalability Figure (fa) shows that increasing model size using the same dataset significantly enhances performance
common QA tasks. However, this improvement does not extend to PhysBench, where gains are limited or even negative. For instance, while VILA-1.5's performance improves by 7.1% on common QA tasks when scaling from 3B to 7B parameters,
LLaVA-1.5
LLaWA-1:5
VILA-1.5
Performance
60
PLLaVA
Performance
PILAMA
Performance
GPT-4o-mri Phi-3V
Mants 88
PLLAVA-78
30
38
156
24
LLM Size
Reldiondsp
Dynarics
Frames
Figure 6: (a) Model size scalability. The solid line shows the average performance across 14 com- mon QA tasks (Table 23 while the dashed line represents PhysBench results. (b) Data scalability. VILA and PLLaVA expand upon LLaVA's architecture by utilizing more data. (c) Frame scalability.
7

---

Published as a conference paper at ICLR 2025
decreases by 3.8% PhysBench. (2) Data Scalability. As shown in Figure b(b), scaling up the dataset offers limited benefits for physical comprehension. PLLaVA and VILA- larger-data vari- ants of LLaVA-1.55 exhibit minimal improvement even decline in performance
PhysBench compared to L.La VA-1.5. Analysis of the additional data (Appendix (.2) reveals it is predominantly descriptive, focusing content description rather than enhancing physical understanding. Nev- ertheless, VILA-1.5's spatial reasoning abilities have significantly improved, aligning with trends observed in other benchmarks (Yu et al. 2023b Li al. 2023a). (3) Frame Scalability, Figure indicates that the three open-source models are insensitive to the number of frames, performing similarly to single-frame inputs, with performance sometimes decreasing as frames increase. This suggests that current models cannot effectively utilize multi-frame information. Notably, increasing the number of frames led Mantis to frequently fail to follow instructions or refuse to answer, and expanding beyond eight frames did not yield further improvements.
Perceptual and knowledge gaps constitute the majority of errors. To investigate the poor perfor- mance of VLMs PhysBench, we randomly selected 500 questions and obtained explanations from three models-GPT-40 Phi-3V, and Gemini-1.5-flash. Expert annotators classified the root causes of the mispredictions into six categories: perception errors, reasoning errors, lack of knowledge, refusal to answer, failure to follow instructions, and annotation errors in the dataset. The distribu- tion of these error types is shown in Figure 7 with selected cases and detailed analyses provided in Appendix []] The error distribution reveals that perceptual errors account for 37%, 40%, and 45% of the mistakes made by GPT-4o. Gemini-1.5 1.5 and Phi-3V, respectively, while lack of knowledge constitutes 34%, 35%, and 23% of errors for these models. This analysis suggests that perceptual er- rors and knowledge gaps are the primary sources of mispredictions, indicating that while the models are adept at extracting information from text and visual inputs, their physical world understanding and complex reasoning abilities remain limited.
GPT-4o
Gemini-1.5
Phi-3V
6%
14%
17S
Perceplual Error
Lack of Knowledge
Ressoning
Textual Underalanding
37%
NR
27%
Rehuse to Answer
Annotation Error
Figure 7: Distribution of error types for GPT-4V, Gemini-1. 1.5ni---.--fah,, Phi-3V.
Can VLMs transfer physical world knowledge? Our error analysis revealed that inadequate phys- ical world knowledge and reasoning capabilities were key contributors to the models' poor perfor- mance. To investigate whether introducing additional examples could enhance performance, we conducted tests on 200 entries of PhysBench, pairing each with a similar example. These additional examples were incorporated through fine-tuning or in-context learning. As shown Figure (T))) the performance improvements after adding physical world knowledge examples indicate that VLMs can transfer physical knowledge to some extent. This suggests that the original data's 's lack of physi- cal world knowledge significant factor in the models' suboptimal performance.
PHYSAGENT
Recognizing perceptual inaccuracies and knowledge gaps as key sources of error shown in Sec- tion 3.4 we introduce PhysAgent in Section 4.1 to improve VLMs' understanding of the physical world by integrating vision foundation models for enhanced perception and incorporating physi- cal knowledge memory. To verify whether enhaneing VLMs' physical understanding facilitates the deployment of embodied agents, we conducted five embodied agent tasks as detailed in Section 4.2
4.1 How TO ENHANCE VLMS FOR PHYSICAL WORLD UNDERSTANDING
We propose PhysAgent, novel framework that integrates knowledge memory and vision foun- dation models to enhance physical world understanding in VLMs. This framework is inspired by findings in Section 3.4 where we identified perceptual emrors and insufficient knowledge as the
8

---

Published as a conference paper at ICLR 2025
primary causes of mistakes in VLMs. To address these shortcomings, we establish knowledge memory that provides prior physical world knowledge and rules. Additionally, we utilize vision foundational models namely Depth Anything (Yang al. 2024b), SAM (Kirillov et al. 2023), ) and GroundingDING (Liu et al. 2023b) to achieve enhanced visual perception. These models enable us to identify object types and spatial locations, and further acquire information about objects' dynam- ics through VLM reasoning or retrieval from memory. They also help solve problems that VLMs cannot address, such as estimating depth and numerical distances. Unlike prior physical reason- ing models that are confined to specific tasks and struggle to adapt to natural language queries, our method aims to fully leverage the reasoning and generalization capabilities of VLMs. Experiments on PhyBench show that PhysAgent improves performance by 18.4% on GPT-4o.
As illustrated in Figure g given question, PhysAgent follows three key steps: (1) Task-specific Prompt Activation: PhysAgent first classifies the question (manually or automatically) and activates task-specific prompts, incorporating relevant physical knowledge for different tasks. For instance, for question about light, it retrieves knowledge on the relationship between light source movement and shadow direction to assist the VLMs. (2) Foundation Models Integration: PhysAgent processes the foundation model's 's outputs, leveraging VLM reasoning capabilities. For example, it identifies objects in the scene using GroundingDINO and retrieves relevant attributes from the knowledge memory. (3) Thhoughhss Reasoning: PhysAgent then engages in chain-of-thoughh reasoning. conducting a self-verification step to ensure logical consistency before providing the final answer.
How does the light move in the vidco?
PhysAgent
T Knowledge Memory
C....
① Taskk-peelffe Prompt Activation
"The sladow the oppacite dinection light source 'The picture beighters as the light source gess closer
[Prompt) You expert, determine the type
...
[VLM] This question is about light [Knowledge] <the related knowledge poumpe>
@ Thain-of-Thouuhhss Reasoning [VLM] As the shadow moves so the light
L
Foundation Models
2 Foundation Models Integration
should
[Grounding DIND]: Bandages, foutballs
moves .... shadow will moves... (venficatioon
[NLM]
[Knowlalge] Retrieee and find footholls clastic
[SAM] Shadow in differrntt picture...
Answer
[VLM] The shadow gradhall moves closer the objects, parallel the line...
C. Move parallel to the line berween the bandage football, and closer these objects
Depth Anything
SAM
Greunding DINO
Figure 8: Architecture of Phys/gent. PhysAgent employs three-step reasoning process to address the problem: activating task misk-specific prompts, integrating foundation models, and reasoning.
Baselines. We utilized three prompt methods: Chain of Thought (CoT) (ojiiaa et al. 2023). Desp- CoT (Wu et al.) 2023d), and Pure Language Reasoning (PLR), in addition to an oracle method, ContPhy (Zheng et al. 2024b), which served as our baseline. Detailed descriptions of the prompt strategies and the implementation of ContPhy are provided in Appendix and Appendix E.7
(!) Prompting methods are unstable, and using pure language yields catastrophic results. As observed, the CoT strategy has minimal impact, while both Desp-CoTT and PLR show a decline in performance. This suggests that descriptive prompts are not particularly effective for addressing the questions, implying that our dataset requires deeper understanding of the videos images to answer accurately. (2) ContPhy even worsens performance. In three out of four tasks, ContPhy underperforms compared to its base model. GPT-4o, due to suboptimal module invocation and limited ficxibility in its logical templates, which struggle to adapt to diverse scenarios. Additionally, ContPhy relies on models like RCNN to process visual information instead of directly leveraging GPT-40, leading potential informa- tion loss and subsequent performance degradation. (3) PhysAgent consistently improves zero-shot performance, notably achieving a 49.5% improvement for GPT-4o in Scene. Compared to the CoT, Desp-CoT, and PLR prompting strategies, our method demonstrates consistent improvements.
Results. The results in Figure 9 a). lead to the following conclusions:
4.2 CAN PHYSICAL WORLD UNDERSTANDING HELP IN EMBODIED APPLICATIONS
Despite gaining significant attention in recent years for their strong generalization capabilities, VLMs as embodied agents (Liu et al., 2024a) still exhibit fundamental operational errors during physical world interactions. In this section, we investigate whether enhancing the physical world perception abilities of VLMs can improve their performance in downstream embodied agent tasks.

---

Published as a conference paper at ICLR 2025
Property Reietveslipss Scear Dananes
Pie-ty
Affendaxe Forze Cite Locatin Tiok.
41.5
31.9
54.9
36.9
revontox
SIDK
Cor
Desp OuT
42.7
42.5
16.6
15.3
14.5
12.6
36.2
20.8
16.7
36.7
34.9
14.0
GPT-4a
TEDEITELE emar.
0.3
06
0.3
0.2
00.
80
0.6
Cuuiito
PhysAgeat
44.5
42.0
88.6
37.1
PlasAgont
60
0.3
02
0.3
182
B1
0.2
81
PARDONT
Cor
0.1
0111
0.
Physkgrat
FLR
33.6
$1.3
20
Prosett
Resatomatss
Boote
Dynamis
EDRERTAES CIIEE
0.8
0.3
077
(a)
(b)
Figure 9: (a) Performance comparison of various methods. (b) Analysis of physical world knowl- edge transfer. (c) Performance evaluation across five embodied tasks as described Figure T0
To evaluate embodied agents, we designed five fundamental manipulation tasks as shown Fig- ure [0 a). The specifics of these tasks, along with the corresponding testing methods and language instructions, can be found in Appendix F.3 These tasks require the agents to possess a basic un- derstanding of spatial relations and the physical properties of objects. Specifically,
utilized Mu- JoCo (Todoroy et al.) 2012) and the 7-DoF Franka Emika robotic arm from Menagerie (Zakka et al.) 2022 building our simulation platform based MOKA (Liu et al. 2024a) as the embodied agent approach to test these embodied tasks. The VLM we used in these tasks is GPT-4o.
Affordance Grasp pot, knife,, spoon, monitor, tennis racket.
Force
Grasp fragile items (egg, ripe persimmon), soft items (jelly, plastic cup), and rigid objects (iron ball).
Color
Grasp the specific color cube.
Location
Grab the cube in specific location and move
the plate.
Tool
Grasp specific tools. depending the problem.
graep
target
waypoints
function
(a)
Figure 10: (a) Description of each of the testing tasks. (b) Marked observation, predicted affor- dances, and motion in MOKA. MOKA leverages a VLM to generate key points and waypoints, and then converts these affordance representations into executable motions for the robotic arm.
As illustrated in Figure [0]b), MOKA prompts the VLM to generate key points and additional attributes for affordance representation based on free-form language instructions and visual ob- servations of the environment. Since the five tasks we tested were relatively basic and did not require decomposition into subtasks, we could directly invoke the VLM in crestion-answertie format to address the operational challenges. This approach ensures seamless compatibility be- tween the pipelines of PhysAgent and MOKA. Once the key points and waypoints were obtained from the VLM, MOKA converted these affordance representations into executable motions for the robotic arm. To evaluate the impact of enhanced physical world understanding on embodied tasks, applied two methods to MOKA's VLM: (1) fine-tuning it with PhysBench, and (2) employing PhysAgent to zero-shot assist reasoning about affordance representations.
As shown in Figure c),, we observe consistent improvements after fine-tuning with subset of PhysBench, indicating that the benchmark's ' data is of high quality and suitable for use as demonstra- tion data in open-world roboties tasks. Additionally, PhysAgent consistently yields stable zero-shot gains across all five tasks, with significant progress observed in the force task in Figure 10 a).
5 CONCLUSION
In conclusion, we introduce PhysBench, a benchmark designed to assess Vision-Lannuuae Models' understanding of the physical world. Through experiments 75 models, identified significant gaps in physical world understanding, particularly in open-source models, due to inadequate training data. To address this, we developed PhysAgent, novel framework that improves physical reasoning by 18.4% on GPT-4o. Additionally, we demonstrated the utility of our dataset and approach in robotic tasks, helping to advance the understanding of the physical world machine intelligence.
Statement. We provide detailed discussion of limitations, broader impacts, ethical considerations, and reproducibility in Appendix
10

---

Published as a conference paper at ICLR 2025
ACKNOWLEDGMENTS
Jiageng Mao and Yue Wang acknowledge funding supports from Toyota Research Institute, Dolby, and Google DeepMind. Yue Wang is also supported by a Powell Faculty Research Award.
REFERENCES
Marah Abdin, Sam Ade Jacobs, Ammar Ahmad Awan, Jyoti Aneja, Ahmed Awadallah, Hany Awadalla, Nguyen Bach, Amit Bahree, Arash Bakhtiari, Harkirat Behl, et al. Phi-3 technical report: A highly capable language model locally on your phone. arXiv, 2024.
Josh Achiam, Steven Adler, Sandhini Agarwal, Lama Ahmad, Ilge Akkaya, Florencia Leoni Ale- man, Diogo Almeida, Janko Altenschmidt, Sam Altman, Shyamal Anadkat, al. Gpt-4 technical report. arXiv, 2023.
01. AI, : Alex Young, Bei Chen, Chao Li, Chengen Huang, Ge Zhang, Guanwei Zhang, Heng Li, Jiangcheng Zhu, Jianqun Chen, Jing Chang. Kaidong Yu, Peng Liu, Qiang Liu, Shawn Yue, Senbin Yang. Shiming Yang, Tao Yu. Wen Xie, Wenhao Huang, Xiaohui Hu, Xiaoyi Ren, Xinyao Niu, Pengcheng Nie, Yuchi Xu, Yudong Liu, Yue Wang. Yuxuan Cai, Zhenyu Gu, Zhiyuan Liu, and Zonghong Dai. Yi: Open foundation models by 01.ai, 2024.
Jean-Baptiste Alayrac Jeff Donahue, Pauline Luc, Antoine Miech, lain Barr, Yana Hasson, Karel Lenc, Arthur Mensch, Katherine Millican, Malcolm Reynolds, et al. Flamingo: a visual language model for few-shot learning. Advances in neural information processing systems, 35:23716- 23736, 2022.
Zahid Ali, Chesser Luke, and Carbone Timothy. Unsplash, 2023.
Anthropic. About claude models.
https: /toos://docs.. htte:://foesssttttroocc...c////s about clw.deeemeeeeeeeseeeee 2024. Accessed: 2024-09-03.
Daman Arora, Himanshu Gaurav Singh, and Mausam. Have ilms advanced enough? challenging problem solving benchmark for large language models, 2023.
Tayfun Ates, M Samil Atesoglu, Cagatay Yigit, Ilker Kesen, Mert Kobas, Erkut Erdem, Aykut Erdem, Tilbe Goksun, and Deniz Yuret. Craft: A benchmark for causal reasoning about forces and interactions. arXiv, 2020.
AzureML. Phi-3.5-vision instruct (128k).
http://gttttttecemmm//a/atttaeet ls/
instruct] 2024. Architecture: Phi-3.5-vision has 4.2B pa- rameters with image encoder, connector, projector, and Phi-3 Mini language model. Inputs: Text and Image (best suited for chat format). Context length: 128K tokens. GPUs: 256 A100-80G. Training time: 6 days. Training data: 500B tokens (vision + text tokens). Outputs: Generated text. Trained between July and August 2024. License: MIT. Release date: August 20, 2024. Sta- tus: Static model with offline text dataset cutoff on March 15, 2024.
azureml
Jinbin Bai, Chunhui Liu, Feiyue Ni, Haofan Wang, Mengying Hu, Xiaofeng Guo, and Lele Cheng. Lat: latent translation with cycle-consistnncy for video-text retrieval. arXiv, 2022.
Jinbin Bai, Wei Chow, Ling Yang, Xiangtai Li, Juncheng Li, Hanwang Zhang, and Shuicheng Yan. Humanedit: A high-quality human 2024a.
human-rewarded rewarded dataset for instruction- image editing. arXiv,
Jinbin Bai, Tian Ye, Wei Chow, Enxin Song, Qing-Gur Chen, Xiangtai Li, Zhen Dong, Lei Zhu, and Shuicheng Yan. Meissonic: Revitalizing masked generative transformers for efficient high- resolution text-to-image synthesis. arXiv, 2024b.
Jinze Bai, Shuai Bai, Yunfei Chu, Zeyu Cui, Kai Dang, Xiaodong Deng, Yang Fan, Wenbin Ge, Yu Han, Fei Huang, Binyuan Hui, Luo Ji, Mei Li, Junyang Lin, Runji Lin, Dayiheng Liu, Gao Liu, Chengqiang Lu, Keming Lu, Jianxin Ma, Rui Men, Xingzhang Ren, Xuancheng Ren, Chuanqi Tan, Sinan Tan, Jianhong Tu, Peng Wang, Shijie Wang, Wei Wang, Shengguang Wu, Benfeng Xu, Jin Xu, An Yang. Hao Yang, Jian Yang, Shusheng Yang, Yang Yao, Bowen Yu, Hongyi Yuan,
11

---

Published as a conference paper at ICLR 2025
Zheng Yuan, Jianwei Zhang, Xingxuan Zhang, Yichang Zhang, Zhenru Zhang. Chang Zhou, Jingren Zhou, Xiaohuan Zhou, and Tranhang Zhu. Qwen technical report. arXiv, 2023a.
Jinze Bai, Shuai Bai, Shusheng Yang, Shijie Wang, Sinan Tan, Peng Wang, Junyang Lin, Chang Zhou, and Jingren Zhou. Qwen-vl: A frontier large vision-language model with versatile abilities. arXiv, 2023b.
Max Bain, Arsha Nagrani, Gül Varol, and Andrew Zisserman. Frozen in time: joint video and image encoder for end-to-end retrieval. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pp. 1728-1738, 2021.
Hritik Bansal, Zongyu Lin, Tianyi Xie, Zeshun Zong, Michal Yarom, Yonatan Bitton, Chenfanfu Jiang, Yizhou Sun, Kai-Wei Chang, and Aditya Grover. Videophy: Evaluating physical common- sense for video generation. arXiv, 2024.
Fabien Baradel, Natalia Neverova, Julien Mille, Greg Mori, and Christian Wolf. Cophy: Counter- factual learning of physical dynamics. arXiv, 2019.
Daniel M Bear, Elias Wang, Damian Mrowca, Felix Binder, Hsiao-W Yu Fish Tung, RT Pramod, Cameron Holdaway, Sirui Tao, Kevin Smith, Fun Sun, et al. Physion: Evaluating physical prediction from vision in humans and machines. arXiv, 2021.
Ali Furkan Biten, Ruben Tito, Andres Mafla, Lluis Gomez, Marçal Rusinol, Ernest Valveny, CV Jawahar, and Dimosthenis Karatzas. Scene text visual question answering. In Proceedings of the IEEE/CVF international conference on computer vision, pp. 4291-4301, 2019.
OC Blender. Blender-a a 3d modelling and rendering package. Retrieved. represents the sequence of Constructs| lo, 2018.
Tyler Bonnen, Stephanie Fu, Yutong Bai, Thomas O'Connell, Yoni Friedman, Nancy Kanwisher, Joshua Tenenbaum, and Alexei A Efros. Evaluating multiview object consistency in humans and image models. arXiv, 2024.
Minwoo Byeon, Beomhee Park, Haecheon Kim, Sungjun Lee, Woonhyuk Back, and Sachoon Kim. Coyo-700m: Image-text pair dataset, 2022.
Holger Caesar, Varun Bankiti, Alex H. Lang, Sourabh Vora, Venice Erin Liong, Qiang Xu, Anush Krishnan, Yu Pan, Giancarlo Baldan, and Oscar Beijbom. nuscenes: A multimodal dataset for autonomous driving. arXiv, 2019.
Xu Cao, Bolin Lai, Wenqian Ye, Yunsheng Ma, Joerg Heintz, Jintai Chen, Jianguo Cao, and James M Rehg. What is the visual cognition gap between humans and multimodal Ilms? arXiv, 2024.
Susan Carey. The origin of concepts. Journal of Cognition and Development, 1(1):37-41,
Keshigeyan Chandrasegaran, Agrim Gupta, Lea M. Hadzic, Taran Kota, Jimming He, Cristobal Byzaguirre, Zane Durante, Manling Li, Jiajun Wu, and Fei-Fei Li. Hourvideo: 1-hour video- language understanding. In Advances in Neural Information Processing Systems, volume 37, 2024.
Angel Chang, Angela Dai, Thomas Funkhouser, Maciej Halber, Matthias Niessner, Manolis Savva, Shuran Song, Andy Zeng, and Yinda Zhang. Matterport3d: Learning from rgb-d data in indoor environments. arXiv, 2017.
Boyuan Chen, Zhuo Xu, Sean Kirmani, Brain Ichter, Dorsa Sadigh, Leonidas Guibas, and Fei Xia. Spatialvlm: Endowing vasion-language models with spatial reasoning capabilities. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 14455-14465. June 2024a.
Lin Chen, Jisong Li, Xiaoyi Dong, Pan Zhang, Conghui He, Jiaqi Wang, Feng Zhao, and Dahua Lin. Sharegpt4v: Improving large multi-modal models with better captions. arXiv, 2023a.
Qiguang Chen, Libo Qin, Jin Zhang, Zhi Chen, Xiao Xu, and Wanxiang Che. M A novel benchmark for multi-domat multi-step multi-moda chain th-tuugh,,, 2024b. URL https: www.xxx..rrg/aa/////a 16473
12

---

Published as a conference paper at ICLR 2025
Xi Chen, Xiao Wang. Lucas Beyer, Alexander Kolesnikov, Jialin Wu, Paul Voigtlaender, Basil Mustafa, Sebastian Goodman, Ibrahim Alabdulmohsin, Piotr Padlewski, Daniel Salz, Xi Xiong, Daniel Vlasic, Filip Pavetic, Keran Rong, Tianli Yu, Daniel Keysers. Xiaohua Zhai, and Radu Soricut. Pali-3 vision language models: Smaller, faster, stronger, 2023h. URL. https: 09199
Zhe Chen, Jiannan Wu, Wenhai Wang, Weijie Su, Guo Chen, Sen Xing, Muyan Zhong, Qinglong Zhang, Xizhou Zhu, Lewei Lu, Bin Li, Ping Luo, Tong Lu, Yu Qiao, and Jifeng Dai. Internv:: Scaling up vision foundation models and aligning for generic visual-linguistic tasks. arXiv, 2023c.
Zhe Chen, Weiyun Wang. Hao Tian, Shenglong Ye, Zhangwei Gao, Erfei Cui, Wenwen Tong, Kongzhi Hu, Jiapeng Luo, Zheng Ma, et al. How far are we to gpt-4v? closing the gap to commercial multimodal models with open-source suites. arXiv, 2024c.
Zhenfang Chen, Kexin Yi, Yunzhu Li, Mingyu Ding, Antonio Torralba, Joshua B Tenenbaum, and Chuang Gan. Comphy: Compositional physical reasoning of objects and events from videos. arXiv, 2022.
An-Chieh Cheng, Hongxu Yin, Yang Fu, Qiushan Guo, Ruihan Yang, Jan Kautz, Xiaolong Wang, and Sifei Liu. Spatialrgpt: Grounded spatial reasoning in vision-language models. In NeurIPS, 2024.
Wei-Lin Chiang, Zhuohan Li, Zi Lin, Ying Sheng, Zhanghao Wu, Hao Zhang, Lianmin Zheng, Siyuan Zhuang, Yonghao Zhuang, Joseph E. Gonzalez, Ion Stoica, and Eric P. Xing. Vicuna: An open-source chatbot impressing gpt-4 with 90%* chatgpt quality, March 2023. URL https:
2003--023-2222 03 30 vicuna/
Wei Chow, Juncheng Li, Qifan Yu, Kaihang Pan, Hao Fei, Zhiqi Ge, Shuai Yang. Siliang Tang, Hanwang Zhang, and Qianru Sun. Unified generative and discriminative training for multi-modal large language models. arXiv, 2024.
Hyung Won Chung, Le Hou, Shayne Longpre, Barret Zoph, Yi Tay, William Fedus, Yunxuan Li, Xuezhi Wang. Mostafa Dehghani, Siddhartha Brahma, al. Scaling catructtin-finttunee lan- guage models. Journal of Machine Learning Research, 25(70):1-53, 2024.
OpenCompass Contributors. Opencompass: A universal evaluation platform for foundation models.
GitHuu repository, 2023.
Chenhang Cui, Yiyang Zhou, Xinyu Yang. Shirley Wu, Linjun Zhang, James Zou, and Huaxiu Yao. Holistic analysis of hallucination in gpt-4v(ision): Bias and interference challenges, 2023, URL
https://arxis: btggabbs/33111 03287
abs /toos/231111
Angela Dai, Angel X Chang, Manolis Savva, Maciej Halber, Thomas Funkhouser, and Matthias Nießsner Scannet: Richly-annotated 3d reconstructions of indoor scenes. In Proceedings of the
IEEE conference on computer vision and pattern recognition, pp. 3828-5839, 2017.
Wenliang Dai, Junnan Li, Dongxu Li, Anthony Meng Huat Tiong, Junqi Zhao Weisheng Wang, Boyang Li, Pascale N Fung, and Steven Hoi. Instructblip: Towards general-purpose vision- language models with instruction tuning. Advances in Neural Information Processing Systems, 36, 2024.
Abhishek Das, Satwik Kottur, Khushi Gupta, Avi Singh, Deshraj Yadav, José MF Moura, Devi Parikh, and Dhruv Batra. Visual dialog. In Proceedings of the IEEE conference on computer vision and patlern recognition, pp. 926-335. 2017.
Matt Deitke, Christopher Clark, Sangho Lee, Rohun Tripathi, Yue Yang, Jae Sung Park, Moham- madreza Salehi, Niklas Muennighoff, Kyle Lo, Luca Soldaini, al. Molmo and pixmo: Open weights and open data for state-of-the-arr multimodal models. arXiv, 2024.
Mengfei Du, Binhao Wu, Zejun Li, Xuanjing Huang, and Zhongyu Wei. Embspatial bench: Bench- marking spatial understanding for embodied tasks with large vision- languae models. arXiv, 2024.
13

---

Published as a conference paper at ICLR 2025
Jiafei Duan, Samson Yu, Soujanya Poria, Bihan Wen, and Cheston Tan. Pip: Physical interaction prediction via mental simulation with span selection. In European Conference on Computer Vision, pp. 0055-21..20 Springer, 2022.
Desmond Elliott, Stella Frank, Khalil Sima' an, and Lucia Specia. Multi30k: Multilingual english- german image descriptions. arXiv, 2016.
Taoran Fang, Wei Zhou, Yifei Sun, Kaiqiao Han, Lvbin Ma, and Yang Yang. Exploring correlations of self-supervised tasks for graphs. arXiv, 2024.
Taoran Fang, Tianhong Gao, Chunping Wang, Yihao Shang, Wei Chow, Lei Chen, and Yang Yang. Kaa: Kolmogorov-arnon attention for enhancing attentive graph neural networks. arXiv, 2025.
Yuxin Fang, Wen Wang, Binhui Xie, Quan Sun. Ledell Wu, Xinggang Wang, Tiejun Huang, Xinlong Wang, and Yue Cao. Eva: Exploring the limits of masked visual representation learning at scale. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp- 19358-19369. 2023.
Xingyu Fu, Yushi Hu, Bangzheng Li, Yu Feng, Haoyu Wang, Xudong Lin, Dan Roth, Noah A Smith, Wei-Chiu Ma, and Ranjay Krishna. Blink: Multimodal large language models can see but not perceive. arXiv, 2024.
Chuang Gan, Jeremy Schwartz. Seth Alter, Damian Mrowca, Martin Schrimpf, James Traer, Julian De Freitas, Jonas Kubilius, Abhishek Bhandwaldar, Nick Haber, et al. Threedworld: A platform for interactive multi-modal physical simulation. arXiv, 2020.
Jensen Gao, Bidipta Sarkar, Fei Xia, Ted Xiao, Jiajun Wu, Brian Ichter, Anirudha Majumdar, and Dorsa Sadigh. Physically grounded vision-language models for robotic manipulation. In Interna- tional Conference on Robotics and Automation. IEEE, 2024a.
Zhangwei Gao, Zhe Chen, Erfei Cui, Yiming Ren, Weiyun Wang, Jinguo Zhu, Hao Tian, Shenglong Ye, Junjun He, Xizhou Zhu, et al. Mini-intervvl: A flexible-transee pocket multimodal model with 5% parameters and 90% performance. arXiv, 2024b.
Zhiqi Ge, Juncheng Li, Qifan Yu, Wei Zhou, Siliang Tang, and Yueting Zhuang. Demon24: Acm mm24 demonstrative instruction following challenge. In Proceedings of the 32nd ACM Interna- tional Conference on Multimedia, pp. 1426--1 14228, 2024.
James J Gibson. The ecological approach to visual perception: classic edition. Psychology press, 2014.
Rohit Girdhar and Deva Ramanan. Cater: A diagnostic dataset for compositional actions and tem- poral reasoning. arXiv, 2019.
Raghav Goyal, Samira Ebrahimi Kahou, Vincent Michalski, Joanna Materzynska, Susanne West- phal, Heuna Kim, Valentin Haenel, Ingo Fruend, Peter Yianilos, Moritz Mueller-Freitag et al. The" something something" video database for learning and evaluating visual common sense. In Proceedings of the IEEE international conference on computer vision, pp. 5842-5850, 2017a.
Yash Goyal, Tejas Khot, Douglas Summers-Stay, Dhruv Batra, and Devi Parikh. Making the in vqa matter: Elevating the role of image understanding in visual question answering. In Proceedings of the IEEE conference on computer vision and pattern recognition, pp. 6904-6913, 2017b.
Kristen Grauman, Andrew Westbury, Eugene Byrne, Zachary Chavis, Antonino Furnari, Rohit Gird- har, Jackson Hamburger, Hao Jiang, Miao Liu, Xingyu Liu, et al. Ego4d: Around the world in 3,000 hours of egocentric video. In Proceedings of the IEEE/CVF Conference Computer Vision and Pattern Recognition, pp. 18995-19012, 2022.
Cylingo Group. Xinyuan-vl-2b: A high perfomeeformance multimodal large model. https:// hugging httpe//oooyyyyllcceec nywan VI-22 2024. Accessed: 2024-12-02
14

---

Published as a conference paper at ICLR 2025
Shuhao Gu, Jialing Zhang, Siyuan Zhou, Kevin Yu, Zhaohu Xing, Liangdong Wang. Zhou Cao, Jintao Jia, Zhuoyi Zhang, Yixuan Wang. Zhenchong Hu, Bo-Wen Zhang. Jijie Li, Dong Liang, Yingli Zhao, Yulong Ao, Yaoqi Liu, Fangxiang Feng, and Guang Liu. Infinity-mm: Scaling multimodal performance with large-scale and high-quality instruction data, 2024. httppt::
18558
Vincent Le Guen and Nicolas Thome. Disentangling physical dynamics from unknown factors for unsupervised video prediction. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pp. 11474-1148/ 2020.
Dingkun Guo, Yuqi Xiang, Shuqi Zhao, Xinghao Zhu, Masayoshi Tomizuka, Mingyu Ding, and Wei Zhan. Phygrasp: Generalizing robotic grasping with physics-ine physics-informed large multimodal models, 2024. URL httpp:://arzi..rrr3 abs 2402
Agrim Gupta, Silvio Savarese, Surya Ganguli, and Li Fei-Fei. Embodied intelligence via learning and evolution. Nature communications, 12(1):5721, 2021.
Danna Gurari, Qing Li, Abigale Stangl, Anhong Guo, Chi Lin, Kristen Grauman, Jiebo Luo, and Jeffrey Bigham. Vizwiz grand challenge: Answering visual questions from blind people. In Proceedings of the IEEE conference computer vision and pattern recognition, pp. 3608-3617 2018.
John K Haas. A history of the unity game engine. Diss. Worcester Polytechnic Institute, 2014.
Chaoqun He, Renjie Luo, Yuzhuo Bai, Shengding Hu, Zhen Leng Thai, Junhao Shen, Jinyi Hu, Xu Han, Yujie Huang, Yuxiang Zhang, et al. Olympiadbench: A challenging benchmark for promoting agi with olympiad-level bilingual multimodal scientific problems. arXiv, 2024.
Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep residual learning for image recog- nition. In Proceedings of the IEEE conference on computer vision and pattern recognition, pp- 770-778, 2016.
Kaiming He, Georgia Gkioxari, Piotr Dollár, and Ross Girshick. Mask r-cnn. In Proceedings of the IEEE international conference computer vision, pp. 2961-2969, 2017.
Dan Hendrycks, Collin Burns, Steven Basart, Andy Zou, Mantas Mazeika, Dawn Song, and Jacob Steinhardt. Measuring massive multitask language understanding. arXiv, 2020.
Jinchang Hou, Chang Ao, Haihong Wu, Xiangtao Kong, Zhigang Zheng, Daijia Tang, Chengming Li, Xiping Hu, Ruifeng Xu, Shiwen Ni, and Min Yang. E-eval: A comprehensive chinese k-12 education evaluation benchmark for large language models, 2024.
Shengding Hu, Yuge Tu, Xu Han, Chaoqun He, Ganqu Cui, Xiang Long, Zhi Zheng, Yewei Fang, Yuxiang Huang, Weilin Zhao, et al. Minicpm: Unveiling the potential small language models with scalable training strategies. arXiv, 2024.
Wenlong Huang, Chen Wang, Ruohan Zhang, Yunzhu Li, Jiajun Wu, and Fei-Fei. Voxposer: Composable 3d value maps for robotic manipulation with language models. arXiv, 2023a.
Xuanwen Huang, Wei Chow, Yang Wang, Ziwei Chai, Chunping Wang, Lei Chen, and Yang Yang. One graph model for cross-domail dynamic link prediction. arXiv, 2024.
Yuzhen Huang, Yuzhuo Bai, Zhihao Zhu, Junlei Zhang, Jinghan Zhang, Tangjun Su, Junteng Liu, Chuancheng Lv, Yikai Zhang, Jiayi Lei, Yao Fu, Maosong Sun, and Jumxian He. C-eval: A multi-level multi-discipline chinese evaluation suite for foundation models, 2023b.
Drew A Hudson and Christopher D Manning. Gqa: A new dataset for real-world visual reasoning and compositional question answering. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pp. 6700-6709, 2019.
Aaron Jaech, Adam Kalai, Adam Lerer, Adam Richardson, Ahmed El-Kishky, Aiden Low, Alec Helyar, Aleksander Madry, Alex Beutel, Alex Carney, et al. Openai ol system card, arXiv, 2024.
15

---

Published as a conference paper at ICLR 2025
Baoxiong Jia, Yixin Chen, Huangyue Yu, Yan Wang, Xuesong Niu, Tengyu Liu, Qing Li, and Siyuan Huang. Sceneverse: Scaling 3d vision-language learning for grounded scene understanding. In European Conference on Computer Vision, pp. 289-310 Springer. 2025.
Albert Q Jiang, Alexandre Sablayrolles, Arthur Mensch, Chris Bamford, Devendra Singh Chaplot, Diego de las Casas, Florian Bressand, Gianna Lengyel, Guillaume Lample, Lucile Saulnier, al. Mistral 7b. arXiv, 2023.
Dongfu Jiang, Xuan He, Huaye Zeng. Cong Wei, Max W.F. Ku, Qian Liu, and Wenhu Chen. Mantís: Intericaved multi-image instruction tuning. arXiv, 2024.
Peng Jin, Ryuichi Takanobu, Caiwan Zhang, Xiaochun Cao, and Yuan. Chat-univi: Unified visual representation empowers large language models with image and video understanding. arXiv, 2023.
Justin Johnson, Bharath Hariharan, Laurens Van Der Maaten, Li Fei-Fei, C Lawrence Zitnick, and Ross Girshick. Clevr: A diagnostic dataset for compositional language and elementary visual reasoning. In Proceedings of the IEEE conference on computer vision and pattern recognition, pp. 2901-2910, 2017.
Kushal Kafle and Christopher Kanan. An analysis of visual question answering algorithms. In Proceedings of the IEEE international conference on computer vision, pp. 1965-1973, 2017.
Andrej Karpathy and Li Fei-Fei. Deep visual-semantic alignments for generating image descrip- tions. In Proceedings of the IEEE conference on computer vision and pattern recognition, pp- 1128-3137, 2015.
Will Kay, Joao Carreira, Karen Simonyan, Brian Zhang, Chloe Hillier, Sudheendra Vijaya- narasimhan, Fabio Viola, Tim Green, Trevor Back, Paul Natsev, et al. The kinetics human action video dataset. arXiv, 2017.
Alexander Khazatsky, Karl Pertsch, Suraj Nair, Ashwin Balakrishna, Sudeep Dasari, Siddharth Karamcheti, Soroush Nasiriany, Mohan Kumar Srirama, Lawrence Yunliang Chen, Kirsty Ellis, Peter David Fagan, Joey Hejna, Masha Itkina, Marion Lepent, Yecheng Jason Ma, Patrick Tree Miller, Jimmy Wu, Suncel Belkhale, Shivin Dass, Huy Ha, Arhan Jain, Abraham Lee, Young- woon Lee, Marius Memmel, Sungjae Park, Ilija Radosavovic, Kaiyuan Wang, Albert Zhan, Kevin Black, Cheng Chi, Kyle Beltran Hatch, Shan Lin, Jingpei Lu, Jean Mercat, Abdul Rehman, Pan- nag R Sanketi, Archit Sharma, Cody Simpson, Quan Vuong, Homer Rich Walke, Blake Wulfe, Ted Xiao, Jonathan Heewon Yang, Arefeh Yavary, Tony Z. Zhao, Christopher Agia, Rohan Bai- jal, Mateo Guaman Castro, Daphne Chen, Qiuyu Chen, Trinity Chung, Jaimyn Drake, Ethan Paul Foster, Jensen Gao, David Antonio Herrera, Minho Heo, Kyle Hsu, Jiaheng Hu, Donovon Jack- son, Charlotte Le, Yunshuang Li, Kevin Lin, Roy Lin, Zehan Ma, Abhiram Maddukuri, Suvir Mir- chandani, Daniel Morton, Tony Nguyen, Abigail O' N'eell,, Rosario Scalise, Derick Seale, Victor Son, Stephen Tian, Emi Tran, Andrew F. Wang, Yilin Wu, Annie Xie, Jingyun Yang, Patrick Yin, Yunchu Zhang, Osbert Bastani, Glen Berseth, Jcannette Bohg. Ken Goldberg, Abhinav Gupta, Abhishek Gupta, Dinesh Jayaraman, Joseph J Lim, Jitendra Malik, Roberto Martin-Martin, Sub- ramanian Ramamoorthy, Dorsa Sadigh, Shuran Song, Jiajun Wu, Michael C. Yip. Yuke Zhu, Thomas Kollar, Sergey Levine, and Chelsea Finn. Droid: A large-scale teth--wwldd robot manip- ulation dataset. 2024.
Jihyung Kil, Zheda Mai, Justin Lee, Zihe Wang, Kerrie Cheng, Lemeng Wang, Ye Liu, Arpita Chowdhury, and Wei-Lun Chao. Compbench: comparative reasoning benchmark for multi- modal lls.. arXiv, 2024.
Carina Kill and Ohnshim Kim. Mental mechanics: How humans reason through a physical world. Mental, 2020.
Moo Jin Kim, Karl Pertsch, Siddharth Karamcheti, Ted Xiao, Ashwin Balakrishna, Suraj Nair, Rafael Rafailov, Ethan Foster, Grace Lam, Pamnag Sanketi, Quan Vuong, Thomas Kollar, Ben- jamin Burchfiel, Russ Tedrake, Dorsa Sadigh, Sergey Levine, Percy Liang, and Chelsea Finn. Openvla: An open-source vision-language-action model. arXiv, 2024.
16

---

Published as a conference paper at ICLR 2025
Alexander Kirillov, Eric Mintun, Nikhila Ravi, Hanzi Mao, Chloe Rolland, Laura Gustafson, Tete Xiao, Spencer Whitehead, Alexander C Berg, Wan-Yen Lo, et al. Segment anything. In Proceed- ings of the IEEE/CVF International Conference Computer Vision, pp. 4015-4026, 2023.
Takeshi Kojima, Shixiang Shane Gu, Machel Reid, Yutaka Matsuo, and Yusuke Iwasawa. Large language models are zero-shot reasoners. Advances in neural information processing systems, 35:22199-22213, 2022.
Takeshi Kojima, Shixiang Shane Gu, Machel Reid, Yutaka Matsuo, and Yusuke Iwasawa. Large language models are zero-shot reasoners, 2023. URL httpttt:///aiii..rrgtt/
11916
Eric Kolve, Roozbeh Mottaghi, Winson Han, Eli VanderBilt, Luca Weihs, Alvaro Herrasti, Matt Deitke, Kiana Ehsani, Daniel Gordon, Yuke Zhu, et al. Ai2-thor: An interactive 3d environment for visual ai. arXiv, 2017.
Jonathan Krause, Justin Johnson, Ranjay Krishna, and Li Fei-Fei. A hierarchical approach for generating descriptive image paragraphs. In Proceedings of the IEEE conference on computer vision and pattern recognition, pp. 317-325, 2017.
Zihang Lai, Senthil Purushwalkam, and Abhinav Gupta. The functional correspondence problem. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pp. 15772-15781, 2021.
Paul Lerner, Olivier Ferret, Camille Guinaudeau, Hervé Le Borgne, Romaric Besançon, José G Moreno, and Jesús Lovón Melgarejo. Viquae a dataset for knowledge-based visual question answering about named entities. In Proceedings of the 45th International ACM SIGIR Conference on Research and Development in Information Retrieval, pp. 3108 -120,, 2022.
Aitor Lewkowycz, Anders Andreassen, David Dohan, Ethan Dyer, Henryk Michalewski, Vinay Ra- masesh, Ambrose Slone, Cem Anil, Imanol Schlag, Theo Gutman Solo, Yuhuai Wu, Behnam Neyshabur, Guy Gur-Ari, and Vedant Misra. Solving quantitative reasoning problems with lan- guage models, 2022.
Bohao Li, Rui Wang, Guangzhi Wang, Yuying Ge, Yixiao Ge, and Ying Shan. benchbench: Bench- marking multimodal lims with generative comprehension. arXiv, 2023a.
Bohao Li, Yuying Ge, Yixiao Ge, Guangzhi Wang, Rui Wang, Ruimao Zhang, and Ying Shan. Seed-bench: Benchmarking multimodal large language models. In Proceedings the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 12299-13308, 2024a.
Boyi Li, Yue Wang, Jiageng Mao, Boris Ivanovic, Sushant Veer, Karen Leung, and Marco Pavone. Driving everywhere with large language model policy adaptation. In Proceedings of the IEEE/CVF Conference Computer Vision and Pattern Recognition, pp. 14948-14957, 2024b.
Boyi Li, Ligeng Zhu, Ran Tian, Shuhan Tan, Yuxiao Chen, Yao Lu, Yin Cui, Sushant Veer, Max Ehrlich, Jonah Philion, et al. Wolf: Captioning everything with a world summarization frame- work. arXiv, 2024c.
Feng Li, Renrui Zhang, Hao Zhang, Yuanhan Zhang, Bo Li, Wei Li, Zejun Ma, and Chunyuan Li. Llava-next-imterleave: Tackling multi-image, video, and 3d in large multimodal models, 2024d.
URL httpss::://axxi org/ abw//407 07895
Juncheng Li, Kaihang Pan, Zhiqi Ge. Minghe Gao. Wei Ji, Wenqiao Zhang, Tat-Seng Chua. Siliang Tang. Hanwang Zhang, and Yueting Zhuang. Fine-tuning multimodal 1lms to follow zero-shot demonstrative instructions. In The Twelfth International Conference Learning Representations, 2023b.
Junnan Li, Dongxu Li. Silvio Savarese, and Steven Hoi. Blip-2: Bootstrapping language-imast pre-training with frozen image encoders and large language models. In International conference on machine learning, pp. 19730-19742. PMLR, 2023c.
KunChang Li, Yinan He, Yi Wang, Yizhuo Li, Wenhai Wang, Ping Luo, Yali Wang, Limin Wang, and Yu Qiao. Videochat: Chat-centric video understanding. arXiv, 2023d.
17

---

Published as a conference paper at ICLR 2025
Kunchang Li, Yali Wang, Yinan He, Yizhuo Li, Yi Wang, Yi Liu, Zun Wang, Jilan Xu, Guo Chen, Ping Luo, Limin Wang, and Yu Qiao. Mvbench: A comprehensive standing benchmark, 2023e.
video under-
Kunchang Li, Yali Wang, Yinan He, Yizhuo Li, Yi Wang, Yi Liu, Zun Wang, Jilan Xu, Guo Chen, Ping Luo, et al. Mybench: A comprehensive multi-modal video understanding benchmark. In Proceedings of the IEEF/CVF Conference on Computer Vision and Pattern Recognition, pp- 22195-222066 2024c.
Lei Li. Yuwei Yin. Shicheng Li, Liang Chen, Peiyi Wang. Shuhuai Ren, Mukai Li, Yazheng Yang, Jingjin Xu, Xu Sun, Lingpeng Kong, and Qi Liu. Mit: A large-scale dataset towards multi- modal multilingual instruction tuning. arXiv, 2023f.
Yanwei Li, Chengyao Wang, and Jiaya Jia. Llama-vid: An image is worth 2 tokens in large language models. arXiv, 2023g.
Yifan Li, Yifan Du, Kun Zhou, Jinpeng Wang, Wayne Xin Zhao, and Ji-Rong Wen. Evaluating object hallucination in large vision- vision-language models. arXiv, 2023h.
Yuncheng Li, Yale Song, Liangliang Cao, Joel Tetreault, Larry Goldberg, Alejandro Jaimes, and Jiebo Luo. Tgif: A new dataset and benchmark on animated gif description. In Proceedings of the IEEE Conference on Computer Vision and Puttern Recognition, pp. 4641 4641-4650, 2016.
Yunzhu Li, Jiajun Wu, Russ Tedrake, Joshua B Tenenbaum, and Antonio Torralba. Learning particle dynamics for manipulating rigid bodies, deformable objects, and fluids. arXiv. 2018.
Zekun Li, Xianjun Yang, Kyuri Choi, Wanrong Zhu, Ryan Hsieh, HyeonJung Kim, Jin Hyuk Lim, Sungyoung Ji, Byungju Lee, Xifeng Yan, et al. Mmsci: A multimodal multi-discipline dataset for phd-level scientific comprehension. arXiv, 2024f.
Bin Lin, Bin Zhu, Yang Ye, Munan Ning, Peng Jin, and Li Yuan. Video-llava: Learning united visual representation by alignment before projection. arXiv, 2023a.
Ji Lin, Hongxu Yin, Wei Ping, Yao Lu, Pavlo Molchanov, Andrew Tao, Huizi Mao, Jan Kautz, Mohammad Shoeybi, and Song Han. Vila: On pre-training for visual language models, 2023b.
Xiongkun Linghu, Jiangyong Huang, Xuesong Niu, Xiaojian Ma, Baoxiong Jia, and Siyuan Huang. Multi-modal situated reasoning in 3d scenes. arXiv, 2024.
Fangehen Liu, Kuan Fang, Pieter Abbeel, and Sergey Levine. Moka: manipulation through mark-based visual prompting. arXiv, 2024a.
robotic
Haotian Liu, Chunyuan Li. Yuheng Li, and Yong Jae Lee, Improved baselines with visual instruction tuning, 2023a.
Haotian Liu, Chunyuan Li, Yuheng Li, Bo Li, Yuanhan Zhang, Sheng Shen, and Yong Jae Lee. Llava-next: Improved reasoning, ocr, and world knowledge, January 2024b. URI. (ttts//ts://
1 flava-vl.g .w..iiliiii. www.b............. lo/blogg/024401-22222
Haotian Liu, Chunyuan Li, Qingyang Wu, and Yong Jae Lee. Visual instruction tuning. Advances in neural information processing systems, 36, 2024c.
Junpeng Liu, Yifan Song, Bill Yuchen Lin, Wai Lam, Graham Neubig, Yuanzhi Li, and Xiang Yue. Visualwebbench: How far have multimodal 1lms evolved in web page understanding and grounding?, 2024d.
Shilong Liu, Zhaoyang Zeng, Tianhe Ren, Feng Li, Hao Zhang, Jie Yang, Chunyuan Li, Jianwei Yang. Hang Su, Jun Zhu, et al. Grounding dino: Marrying dino with grounded pre-training for open-set object detection. arXiv, 2023b.
Yuan Liu, Haodong Duan, Yuanhan Zhang, Bo Li, Songyang Zhang, Wangbo Zhao, Yike Yuan, Jiaqi Wang, Conghui He, Ziwei Liu, et al. Mmbench: your multi-modal model an all-around player? arXiv, 2023c.
18

---

Published as a conference paper at ICLR 2025
Yuanxin Liu, Shicheng Li, Yi Liu, Yuxiang Wang, Shuhuai Ren, Lei Li, Sishuo Chen, Xu Sun, and Lu Hou. Tempcompass: Do video lims really understand videos? arXiv, 2024e.
Zhijian Liu, Ligeng Zhu, Baifeng Shi, Zhuoyang Zhang, Yuming Lou, Shang Yang, Haocheng Xi, Shiyi Cao, Yuxian Gu, Dacheng Li, Xiuyu Li, Yunhao Fang. Yukang Chen, Cheng-Ya Yu Hsieh, De-An Huang, An-Chieh Cheng, Vishwesh Nath, Jinyi Hu, Sifei Liu, Ranjay Krishna, Daguang Xu, Xiaolong Wang, Pavlo Molchanov, Jan Kautz, Hongxu Yin, Song Han, and Yao Lu. Nvila: Efficient frontier visual language models, 2024f. URL httph::t////c www.v..iiii....... abs/242..
04468
Haoyu Lu, Wen Liu, Bo Zhang, Bingxuan Wang. Kai Dong, Bo Liu, Jingxiang Sun, Tongzheng Ren, Zhuoshu Li, Yaofeng Sun, Chengqi Deng, Hanwei Xu, Zhenda Xie, and Chong Ruan. Deepseek- vI: Towards real-world vision- laiiuaee understanding. 2024a.
Pan Lu, Swaroop Mishra, Tanglin Xia, Liang Qiu, Kai-Wei Chang, Song-Chun Zhu, Oyvind Tafjord, Peter Clark, and Ashwin Kalyan. Learn to explain: Multimodal reasoning via thought chains for science question answering. Advances in Neural Information Processing Systems. 35:2507-2553 2022.
Pan Lu, Hritik Bansal, Tony Xia, Jiacheng Liu, Chunyuan Li, Hannaneh Hajishirzi, Hao Cheng, Kai- Wei Chang, Michel Galley, and Jianfeng Gao. Mathvista: Evaluating mathematical reasoning of foundation models in visual contexts. In International Conference on Learning Representations, 2024b.
Ruiyuan Lyu, Tai Wang, Jingli Lin, Shuai Yang, Xiaohan Mao, Yilun Chen, Runsen Xu, Haifeng Huang. Chenming Zhu, Dahua Lin, et al. Mmscan: A multi-modal 3d scene dataset with hierar- chical grounded language annotations. arXiv, 2024.
Xiaojian Ma, Silong Yong, Zilong Zheng, Qing Li, Yitao Liang, Song-Chun Zhu, and Siyuan Huang. Sqa3d: Situated question answering in 3d scenes, In International Conference Learning Rep- resentations, 2023. URL hetps: http//ppeeevviieeee
net/
7BC38
Muhammad Maaz, Hanoona Rasheed, Salman Khan, and Fahad Shahbaz Khan. Video-chatgpt: Towards detailed video understanding via large vision and language models. arXiv, 2023a.
Muhammad Maaz, Hanoona Rasheed, Salman Khan, and Fahad Shahbaz Khan. Video-chatgpt: Towards detailed video understanding via large vision and language models. arXiv:2306.05424, 2023b.
Arjun Majumdar, Anurag Ajay. Xiaohan Zhang, Pranav Putta, Sriram Yenamandra, Mikael Henaff, Sneha Silwal, Paul Mevay, Oleksandr Maksymets, Sergio Arnad, Karmesh Yadav, Qiyang Li, Ben Newman, Mohit Sharma, Vincent Berges, Shiqi Zhang, Pulkit Agrawal, Yonatan Bisk, Dhruv Batra, Mrinal Kalakrishnan, Franziska Meier, Chris Paxton, Sasha Sax, and Aravind Rajeswaran. Openeqa: Embodied question answering in the era of foundation models. In Conference on Computer Vision and Pattern Recognition, 2024.
Karttikeya Mangalam, Raiymbek Akshulakov, and Jitendra Malik. Egoschema: A diagnostic bench- mark for very formf video language understanding. Advances in Neural Information Process- ing Systems, 36, 2024.
Jiageng Mao, Yuxi Qian, Junjie Ye, Hang Zhao, and Yue Wang. Gpt-driver: Learning to drive with gpt. arXiv, 2023a.
Jiageng Mao, Junjie Ye, Yuxi Qian, Marco Pavone, and Yue Wang. A language agent for autonomous driving. arXiv, 2023b.
Jiageng Mao, Siheng Zhao, Siqi Song, Tianheng Shi, Junjie Ye, Mingtong Zhang, Haoran Geng, Jitendra Malik, Vitor Guizilini, and Yue Wang. Learning from massive human videos for universal humanoid pose control. arXiv, 2024.
Kenneth Marino, Mohammad Rastegari, Ali Farhadi, and Roozbeh Mottaghi. Ok-vqa: A visual question answering benchmark requiring external knowledge. In Conference on Computer Vision and Pattern Recognition, 2019.
19

---

Published as a conference paper at ICLR 2025
Minesh Mathew, Dimosthenis Karatzas, and CV Jawahar. Doevqa: A dataset for vqa document images. In Proceedings of the IEEE/CVF winter conference on applications of computer vision, pp. 2200-2209, 2021.
Michael McCloskey, Allyson Washburn, and Linda Felch. Intuitive physics: the straight-down belief and its origin. Journal of Experimental Psychology: Learning, Memory, and Cognition, 94):636, 1983.
Fanqing Meng, Jin Wang, Chuanhao Li, Quanfeng Lu, Hao Tian, Jiaqi Liao, Xizhou Zhu, Jifeng Dai, Yu Qiao, Ping Luo, Kaipeng Zhang, and Wenqi Shao. Mmiu: Multimodal multi-image understanding for evaluating large vision-language models, 2024. URL
stg/abs/2408.. 02718
Antoine Miech, Dimitri Zhukov, lean-Baptiste Alayrac, Makarand Tapaswi, Ivan Laptev, and Josef Sivic. Howtol00m: Learning a text-video embedding by watching hundred million narrated video clips. In Proceedings of the IEEE/CVF international conference on computer vision, pp.
2630-2640, 2019.
Anand Mishra, Shashank Shekhar, Ajeet Kumar Singh, and Anirban Chakraborty. Ocr-vqa: Visual question answering by reading text in images. In International Conference on Document Analysis and Recognition, pp. 947-952. IEEE, 2019.
Niklas Muennighoff, Luca Soldaini, Dirk Groeneveld, Kyle Lo, Jacob Morrison, Sewon Min, Weijia Shi, Pete Walsh, Oyvind Tafjord, Nathan Lambert, Yuling Gu, Shane Arora, Akshita Bhagia, Dustin Schwenk, David Wadden, Alexander Wettig, Binyuan Hui, Tim Dettmers, Douwe Kiela, Ali Farhadi, Noah A. Smith, Pang Wei Koh, Amanpreet Singh, and Hannanch Hajishirzi. Olmoe: Open mixture-of-exxpone language models, 2024. URL httph:t///////tt www.a..w..igo............. abgg/////b///s//// 02060
Soroush Nasiriany. Fei Xia, Wenhao Yu, Ted Xiao, Jacky Liang, Ishita Dasgupta, Annie Xie, Danny Driess, Ayzaan Wahid, Zhuo Xu, Quan Vuong, Tingman Zhang, Tsang-Wei Edward Lee, Kuang- Huei Lee, Peng Xu, Sean Kirmani, Yuke Zhu, Andy Zeng, Karol Hausman, Nicolas Heess, Chelsea Finn, Sergey Levine, and Brian Ichter. Pivot: Iterative visual prompting elicits actionable knowledge for vims. arXiv, 2024.
Dantong Niu, Yuvan Sharma, Giscard Biamby, Jerome Quenum, Yutong Bai, Baifeng Shi, Trevor Darrell, and Roei Herzig. Llarva: Vision-action instruction tuning enhances robot learning. arXiv, 2024.
Octo Model Team, Dibya Ghosh, Homer Walke, Karl Pertsch, Kevin Black, Oier Mees, Sudeep Dasari, Joey Hejna, Charles Xu, Jianlan Luo, Tobias Kreiman, You Liang Tan, Lawrence Yunliang Chen, Pannag Sanketi, Quan Vuong, Ted Xiao, Dorsa Sadigh, Chelsea Finn, and Sergey Levine. Octo: An open-source generalist robot policy. In Proceedings of Robotics: Science and Systems, Delft, Netherlands, 2024.
Kaihang Pan, Siliang Tang, Juncheng Li, Zhaoyu Fan, Wei Chow, Shuicheng Yan, Tat-Seng Chua, Yueting Zhuang, and Hanwang Zhang. Auto-encoding morph-tokens for multimodal Ilm. arXiv, 2024.
Maitreya Patel, Tejas Gokhale, Chitta Baral, and Yezhou Yang. Cripp-vqa: Counterfactual reasoning about implicit physical properties via video question answering. arXiv, 2022.
Zhiliang Peng, Wenhui Wang, Li Dong. Yaru Hao, Shaohan Huang, Shuming Ma, and Furu Wei. Kosmos-2: Grounding multimodal large language models to the world. ArXiv, abs/2306. 14824, semant www.chlll.....oooooooooo.o...... wwwaaroorr//eee Corr/CCrrpsTTTi ID: 62226
2023. URL httpt:tt::
Charles R Qi, Hao Su, Kaichun Mo, and Leonidas J Guibas. Pointnet: Deep learning on point sets for 3d classification and segmentation. In Proceedings of the IEFE conference on computer vision and pattern recognition, pp. 652-660, 2017.
Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, et al. Learning transferable visual models from natural language supervision. In International conference on machine learning, pp. PMLR, 2021.
20

---

Published as a conference paper at ICLR 2025
Nazneen Fatema Rajani, Rui Zhang, Yi Chern Tan, Stephan Zheng, Jeremy Weiss, Aadit Vyas, Ab- hijit Gupta, Caiming Xiong, Richard Socher, and Dragomir Radev. Esprit: Explaining solutions to physical reasoning tasks. arXiv, 2020.
Tianhe Ren, Shilong Liu, Ailing Zeng, Jing Lin, Kunchang Li, He Cao, Jiayu Chen, Xinyu Huang, Yukang Chen, Feng Yan, et al. Grounded sam: Assembling open open-world models for diverse visual tasks. arXiv, 2024.
Ronan Riochet, Mario Ynocente Castro, Mathieu Bernard, Adam Lerer, Rob Fergus, Véronique Izard, and Emmanuel Dupoux. Intphys: A framework and benchmark for visual intuitive physics reasoning. arXiv, 2018.
Paul-Edouard Sarlin, Daniel DeTone, Tomasz Malisiewicz, and Andrew Rabinovich. Superglue: Learning feature matching with graph neural networks. In Proceedings of the IEEE/CVF confer- ence on computer vision and pultern recognition, pp. 4938-4947, 2020.
Piyush Sharma, Nan Ding, Sebastian Goodman, and Radu Soricut. Conceptual captions: A cleaned, hypernyme,, image alt-text dataset for automatic image captioning. In Proceedings of ACL, 2018.
Oleksii Sidorov, Ronghang Hu, Marcus Rohrbach, and Amanpreet Singh. Textcaps: a dataset for image captioning with reading comprehension. Computer Vision-ECCV 2020: 16th European Conference, Glasgow, UK, August 23-28, 2020, Proceedings, Part II 16, pp. 742-758 Springer, 2020.
Karen Simonyan and Andrew Zisserman. Very deep convolutional networks for large-scale image recognition. arXiv, 2014.
Amanpreet Singh, Vivek Natarajan, Meet Shah, Yu Jiang, Xinlei Chen, Dhruv Batra, Devi Parikh, and Marcus Rohrbach. Towards vqa models that can read. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pp. 8317-8326, 2019.
Noah Snavely, Steven M. Seitz, and Richard Szeliski. Photo tourism: Exploring photo collections in 3d. In SIGGRAPH Conference Proceedings, pp. 835-846, New York, NY, USA, 2006. ACM Press. ISBN 1-59593-364-6.
Sanjana Srivastava, Chengshu Li, Michael Lingelbach, Roberto Mart, Fei Xia, Kent Vainio, Zheng Lian, Cem Gokmen, Shyamal Buch, Karen Liu, Silvio Savarese, Hyowon Gweon, Jiajun Wu, and Li Fei-Fei. Behavior: Benchmark for everyday household activities in virtual, interactive, and ecological environments. In Conference in Robot Learning, pp. accepted, 2021.
Andreas Steiner, André Susano Pinto, Michael Tschannen, Daniel Keysers, Xiao Wang, Yonatan Bitton, Alexey Gritsenko, Matthias Minderer, Anthony Sherbondy, Shangbang Long, Siyang Qin, Reeve Ingle, Emanuele Bugliarello, Sahar Kazemzadeh, Thomas Mesnard, Ibrahim Alab- dulmohsin, Lucas Beyer, and Xiaohua Zhai. Paligemma 2: family of versatile vlms for transfer. arXiv,, 2024.
Alane Suhr, Mike Lewis, James Yeh, and Yoav Artzi. A corpus of natural language for visual reasoning. In Proceedings of the 55th Annual Meeting of the Association for Computational Linguistics, pp. 217-223, 2017.
Alane Suhr, Stephanie Zhou, Ally Zhang, Iris Zhang, Huajun Bai, and Yoav Artzi. corpus for reasoning about natural language grounded in photographs. arXiv, 2018.
Deborah Sulsky, Shi-Jian Zhou, and Howard L Schreyer. Application of certicle-innceld method to solid mechanics. Computer physics communications, 87(1-2)2236-55 1995.
Ddac Surs, Sachit Menon, and Carl Vondrick. Vipergpt: Visual inference via python execution for reasoning. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pp. 11888-1 1199,, 1898, 2023.
Ryota Tanaka, Kyosuke Nishida, and Sen Yoshida. Visualmre: Machine reading comprehension document images. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 35, pp. 13878-13888 13888, 2021.
21

---

Published as a conference paper at ICLR 2025
Gemini Team, Rohan Anil, Sebastian Borgeaud, Yonghui Wu, Jean- Cean-Bapisste Alayrac, Jiahui Yu, Radu Soricut, Johan Schalkwyk, Andrew Dai, Anja Hauth, et al. Gemini: a family of highly capable multimodal models. arXiv, 2023.
Gemma Team, Morgane Riviere, Shreya Pathak, Pier Giuseppe Sessa, Cassidy Hardin, Surya Bhu- patiraju. Léonard Hussenot, Thomas Mesnard, Bobak Shahriari, Alexandre Ramé, et al. Gemma 2: Improving open language models at a practical size. arXiv. 2024.
Emanuel Todorov, Tom Erez, and Yuval Tassa. Mujoco: A physics engine for model-based control. In 2012 IEEE/RSJ international conference intelligent robots and systems, pp. 5026-5033. IEEE, 2012.
Shengbang Tong, Ellis Brown, Penghao Wu, Sanghyun Woo, Manoj Middepogu, Sai Charitha Akula, Jihan Yang, Shusheng Yang, Adithya Iyer, Xichen Pan, Austin Wang, Rob Fergus, Yann LeCun, and Saining Xie. Cambrian-l: A fully open, vision-centric exploration of multimodal Ilms. Xrviv, 2024.
Hugo Touvron, Thibaut Lavril, Gautier Izacard, Xavier Martinet, Marie-Anne Lachaux, Timothée Lacroix, Baptiste Rozière, Naman Goyal, Eric Hambro, Faisal Azhar, et al. Llama: Open and efficient foundation language models. arXiv, 2023.
Fish Tung, Mingyu Ding, Zhenfang Chen, Daniel M. Bear, Chuang Gan, Joshua B. Tenenbaum, Daniel L. K. Yamins, Judith Fan, and Kevin A. Smith. Physion++: Evaluating physical scene understanding that requires online inference of different physical properties. arXiv, 2023.
Andong Wang, Bo Wu, Sunli Chen, Zhenfang Chen, Haotian Guan, Wei-Ning Lee, Li Erran Li, and Chuang Gan. Sok-bench: situated video reasoning benchmark with aligned open-world knowl- edge. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 13384-13394, 2024a.
Chen Wang, Linxi Fan, Jiankai Sun, Ruohan Zhang, Li Fei-Fei, Danfei Xu, Yuke Zhu, and Anima Anandkumar. Mimicplay: Long-horizon imitation learning by watching human play. arXiv, 2023a.
Fei Wang, Xingyu Fu, James Y Huang, Zekun Li, Qin Liu, Xiaogeng Liu, Mingyu Derek Ma, Nan Xu, Wenxuan Zhou, Kai Zhang, et al. Muirbench: A comprehensive benchmark for robust multi-image understanding. arXiv, 2024b.
Peng Wang, Shuai Bai, Sinan Tan, Shijie Wang, Zhihao Fan, Jinze Bai, Keqin Chen, Xuejing Liu, Jialin Wang, Wenbin Ge, Yang Fan, Kai Dang, Mengfei Du, Xuancheng Ren, Rui Men, Dayiheng Liu, Chang Zhou, Jingren Zhou, and Junyang Lin. Qwen2-vl: Enhancing vision-language model's perception of the world at any resolution. arv, 2024c.
Tai Wang, Xiaohan Mao, Chenming Zhu, Runsen Xu, Ruiyuan Lyu, Peisen Li, Xiao Chen, Wenwei Zhang, Kai Chen, Tianfan Xue, Xihui Liu, Cewu Lu, Dahua Lin, and Jiangmiao Pang. Em- bodiedscan: A holistic multi-modal 3d perception suite towards embodied ai. In Conference on Computer Vision and Pattern Recognition, 2024d.
Weihan Wang, Qingsong Lv, Wenmeng Yu, Wenyi Hong, Ji Qi, Yan Wang, Junhui Ji, Zhuoyi Yang, Lci Zhao, Xixuan Song, et al. Cogvlm: Visual expert for pretrained language models. arXiv, 2023b.
Weiyun Wang, Zhe Chen, Wenhai Wang, Yue Cao, Yangzhou Liu, Zhangwei Gao, Jinguo Zhu, Xizhou Zhu, Lewei Lu, Yu Qiao, and Jifeng Dai. Enhancing the reasoning ability of multimodal large language models via mixed preference optimization. arXiv, 2024e.
Xiaoxuan Wang, Ziniu Hu, Pan Lu, Yangiao Zhu, Jieyu Zhang, Satyen Subramaniam, Arjun R. Loomba, Shichang Zhang, Yizhou Sun, and Wei Wang. Scibench: Evaluating college-level sci- entific problem-solving abilities of large language models, 2024f.
Xingrui Wang, Wufei Ma, Angtian Wang, Shuo Chen, Adam Kortylewski, and Alan Yuille. Com- positional 4d dynamic scenes understanding with physics priors for video question answering. arXiv, 2024g.
22

---

Published as a conference paper at ICLR 2025
Yi Ru Wang, Jiafei Duan, Dieter Fox, and Siddhartha Srinivasa. Newton: Are large language models capable of physical reasoning? arXiv, 2023c.
Jason Wei, Maarten Bosma, Vincent Y Zhao, Kelvin Guu, Adams Wei Yu, Brian Lester, Nan Du, Andrew M Dai, and Quoe V Le. Finetuned language models are zero-shot learners. arXiv, 2021.
Johannes Welbl, Nelson F. Liu, and Matt Gardner. Crowdsourcing multiple choice science ques- tions. In Leon Derezynski, Wei Xu, Alan Ritter, and Tim Baldwin (eds.), Proceedings of the 3rd Workshop Noisy User-generated Text, pp. 94-106, Copenhagen, Denmark, Septem- ber 2017. Association for Computational Linguistics. doi: 10. 10.18653/11/W11-44411 URL. httts:://aclaattee antho www.tthwooo......oooo www..orrg/17--000
Bo Wu, Shoubin Yu, Zhenfang Chen, Joshua B Tenenbaum, and Chuang Gan. Star: A benchmark for situated reasoning in real-world videos. arXiv, 2024a.
Haoning Wu, Zicheng Zhang, Erli Zhang, Chaofeng Chen, Liang Liao, Annan Wang, Chunyi Li, Wenxiu Sun, Qiong Yan, Guangtao Zhai, al. Q-bench: A benchmark for general-purpose foundation models on low-level vision. arXiv, 2023a.
Haoning Wu, Dongxu Li. Bei Chen, and Junnan Li. Longvideobench: A benchmark for long-context interleaved video-language understanding. arXiv, 2024b.
Jiajun Wu, Joseph Lim, Hongyi Zhang, Joshua B Tenenbaum, and William T Freeman. Physics 101: Learning physical object properties from unlabeled videos. In BMVC, volume 2, pp. 7, 2016.
Qiucheng Wu, Handong Zhao, Michael Saxon, Trung Bui, William Yang Wang, Yang Zhang, and Shiyu Chang. Vsp: Assessing the dual challenges of perception and reasoning in spatial planning tasks for vlms. arXiv, 2024c.
Shengqiong Wu, Hao Fei, Leigang Qu Wei Ji, and Tat-Seng Chua. Next-gpt: Any-to-any multi- modal Ilm. ArXiv, abs/2300.05519, 2023b.
http://app..emmtttccccoooooollle org/
(orpuusDD:26166655)
Weijia Wu, Yuzhong Zhao, Zhuang Li, Jiahong Li, Hong Zhou, Mike Zheng Shou, and Xiang Bai. A large eross-modal video retrieval dataset with reading comprehension. arXiv,, 2023c.
Yifan Wu, Pengchuan Zhang, Wenhan Xiong, Barlas Oguz, James C. Gee, and Yixin Nie. The role of chain thhougtf- complex vision-language reasoning task, 2023d. URL https://
09199
Yuxin Wu, Alexander Kirillov, Francisco Massa, Wan-Yen Lo, and Ross Girshick. Detectron2. 2019.
Junbin Xiao, Xindi Shang, Angela Yao, and Tat-Seng Chua. Next-qa: Next phase of question- answering to explaining temporal actions. In Proceedings of the IEEE/CVF conference com- puter vision and pattern recognition, pp. 9777-9786, 2021a.
Junbin Xiao, Xindi Shang, Angela Yao, and Tat-Seng Chua. Next-qa: Next phase of question- answering to explaining temporal actions. In Pruceedings of the IEEE/CVF conference on com- puter vision and pattern recognition, pp. 9777-9786, 2021b.
Binzhu Xie, Sicheng Zhang, Zitang Zhou, Bo Li, Yuanhan Zhang, Jack Hessel, Jingkang Yang, and Ziwei Liu. Funqa: Towards surprising video comprehension. arXiv. 2023.
Dejing Xu, Zhou Zhao, Jun Xiao, Fei Wu, Hanwang Zhang, Xiangnan He, and Yueting Zhuang. Video question answering via gradually refined attention over appearance and motion. In Pro- ceedings of the 25th ACM international conference on Multimedia, pp. 1645-1653, 2017.
Jun Xu, Tao Mei, Ting Yao, and Yong Rui. Msr-vtt: A large video description dataset for bridging video and language. In Proceedings of the IEEE conference on computer vision and pattern recognition, pp. 5288-5296, 2016.
23

---

Published as a conference paper at ICLR 2025
Lin Xu, Yilin Zhao, Daquan Zhou, Zhijie Lin, See Kiong Ng, and Jiashi Feng. Pllava Parameter- free llava extension from images to videos for video dense captioning, 2024.
Antoine Yang, Antoine Miech, Josef Sivic, Ivan Laptev, and Cordelia Schmid. Just ask: Learning to answer questions from millions of narrated videos. In Proceedings of the IEEE/CVF international conference on computer vision, pp. 1686-1697, 2021.
Jihan Yang, Shusheng Yang. Anjali W Gupta, Rilyn Han, Li Fei-Fei, and Saining Xie. Thinking in space: How multimodal large language models see, remember, and recall spaces. arXiv, 2024a.
Lihe Yang, Bingyi Kang, Zilong Huang, Zhen Zhao, Xiaogang Xu, Jiashi Feng, and Hengshuang Zhao. Depth anything v2. arXiv:2406.0944 2024b.
Yuan Yao, Tianyu Yu, Ao Zhang, Chongyi Wang. Junbo Cui, Hongji Zhu, Tianchi Cai, Haoyu Li, Weilin Zhao, Zhihui He, Qianyu Chen, Huarong Zhou, Zhensheng Zou, Haoye Zhang, Shengding Hu, Zhi Zheng, Jie Zhou, Jie Cai, Xu Han, Guoyang Zeng, Dahai Li, Zhiyuan Liu, and Maosong Sun. Minicpm-v: A gpt-4v level milm on your phone. arXiv, 2024. URL btttps:/araae aa///bbb/2400 01800
Jiabo Ye, Haiyang Xu, Haowei Liu, Anwen Hu, Ming Yan, Qi Qian, Ji Zhang, Fei Huang, and Jingren Zhou. mplug-ow13: Towards long image-sequence understanding in multi-modal large language models, 2024. URL
wwt.ooogg/ao...o..ooooooo 2408 04840
Kexin Yi, Chuang Gan, Yunzhu Li, Pushmeet Kohli, Jiajun Wu, Antonio Torralba, and Joshua B Tenenbaum. Clevrer: Collision events for video representation and reasoning. arXiv, 2019.
Shukang Yin, Chaoyou Fu, Sirui Zhao, Ke Li, Xing Sun, Tong Xu, and Enhong Chen. A survey on multimodal large language models. arXiv, 2023.
Kaining Ying, Fanqing Meng, Jin Wang, Zhiqian Li, Han Lin, Yue Yang, Hao Zhang, Wenbo Zhang, Yuqi Lin, Shuo Liu, Jiayi Lei, Quanfeng Lu, Runjian Chen, Peng Xu, Renrui Zhang, Haozhe Zhang, Peng Gao, Yali Wang, Yu Qiao, Ping Luo, Kaipeng Zhang, and Wenqi Shao. Mmt-bench: A comprehensive multimodal benchmark for evaluating large vision-language models towards multitask agi, 2024.
Qifan Yu, Juncheng Li, Yu Wu, Siliang Tang, Wei Ji, and Yueting Zhuang. Visually-prompted language model for fine-grained scene graph generation in an open world. In Proceedings of the JEEE/CVF International Conference on Computer Vision, pp. 21560-21571, 2023a.
Qifan Yu, Wei Chow, Zhongqi Yue, Kaihang Pan, Yang Wu, Xiaoyang Wan, Juncheng Li, Siliang Tang, Hanwang Zhang, and Yueting Zhuang. Anyedit: Mastering unified high-quality image editing for any idea. arXiv, 2024a.
Qifan Yu, Juncheng Li, Longhui Wei, Liang Pang. Wentao Ye, Bosheng Qin, Siliang Tang, Qi Tian, and Yueting Zhuang. Hallucidoctor: Mitigating hallucinatory toxicity in visual instruction data. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, 2024b.
Weihao Yu, Zhengyuan Yang, Linjie Li, Jianfeng Wang, Kevin Lin, Zicheng Liu, Xinchao Wang, and Lijuan Wang. Mm-vet: Evaluating large multimodal models for integrated capabilities. arXiv, 2023b.
Zhou Yu, Dejing Xu, Jun Yu, Ting Yu, Zhou Zhao, Yueting Zhuang, and Dacheng Tao. Activitynet- qa: A dataset for understanding complex web videos via question answering. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 33, pp. 9127-9134, 2019.
Shenghai Yuan, Jinfa Huang, Yongqi Xu, Yaoyang Liu, Shaofeng Zhang, Yujun Shi, Ruijie Zhu, Xinhua Cheng, Jiebo Luo, and Li Yuan. Chronomagic-bench:: A benchmark for metamorphic evaluation of text-to-time-lapp video generation. arXiv, 2024.
Xiang Yue, Yuansheng Ni, Kai Zhang, Tianyu Zheng, Ruoqi Liu, Ge Zhang, Samuel Stevens, Dongfu Jiang, Weiming Ren, Yuxuan Sun, Cong Wei, Botao Yu, Ruibin Yuan, Renliang Sun, Ming Yin, Boyuan Zheng, Zhenzhu Yang, Yibo Liu, Wenhao Huang, Huan Sun, Yu Su, and Wenhu Chen. Mmmu: A massive multi-discipline multimodal understanding and reasoning benchmark for expert agi. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, 2024.
24

---

Published as a conference paper at ICLR 2025
Kevin Zakka, Yuval Tassa, and MuJoCo Menagerie Contributors. MuJoCo Menagerie: A col- lection of high-quality simulation models for MuloCo, 2022. URL http://githui: com/
pnoogle-deepmindd/eee joco menagerie
Michal Zawalski, William Chen, Karl Pertsch, Oier Mees, Chelsea Finn, and Sergey Levine. Robotic control via embodied chain-of-thought reasoning. arXiv, 2024a.
Michal Zawalski, William Chen, Karl Pertsch, Oier Mees, Chelsea Finn, and Sergey Levine. Robotic control via embodied chain-of -fthoghht reasoning. arXiv, 2024b.
Xiaohua Zhai, Basil Mustafa, Alexander Kolesnikov, and Lucas Beyer. Sigmoid loss for language image pre-training, 2023.
Jun Zhan, Junqi Dai, Jiasheng Ye, Yunhua Zhou, Dong Zhang, Zhigeng Liu, Xin Zhang. Ruibin Yuan, Ge Zhang. Linyang Li, Hang Yan, Jie Fu, Tao Gui. Tianxiang Sun, Yu- gang Jiang, and Xipeng Qiu. Anygpt: Unified multimodal 11m with discrete sequence mod- cling. ArXiv, abs/2402.12226, 2024. URL https: /stps://api... www.ntiisccootoooooooo..
org/
Corpus_D:26775011
Hang Zhang, Xin Li, and Lidong Bing. Video-llama: An instruction-tuned audio-visual language model for video understanding. arXiv, 2023. URL https://axx.vvorg//b/////2
02858
Jieyu Zhang, Weikai Huang, Zixian Ma, Osear Michel, Dong He, Tanmay Gupta, Wei-Chiu Ma, Ali Farhadi, Aniruddha Kembhavi, and Ranjay Krishna. Task me anything. arXiv, 2024a.
Yuanhan Zhang, Bo Li, haotian Liu, Yong jae Lee, Liangke Gui, Di Fu, Jiashi Feng. Ziwei Liu, and Chunyuan Li. Llava-next: A strong zero-shot video understanding model, April 2024b. URL vl. gi thub.
30--300-10 1lava nexta-nextvve video/
Haoyu Zhen, Xiaowen Qiu, Peihao Chen, Jincheng Yang, Xin Yan, Yilun Du, Yining Hong, and generative world model. arXiv, 2024.
Chuang Gan. 3d-vla: 3d dision-language-nn
Chujie Zheng, Zhenru Zhang, Beichen Zhang, Runji Lin, Keming Lu, Bowen Yu, Dayiheng Liu, Jingren Zhou, and Junyang Lin. Processbench: Identifying process errors in mathematical rea- soning, 2024a. URL htttss://axxiiii
abs /21...212... 2412.06559
Zhicheng Zheng, Xin Yan, Zhenfang Chen, Jingzhou Wang, Qin Zhi Eddie Lim, Joshua Tenen- baum, and Chuang Gan. Contphy: Continuum physical concept learning and reasoning from videos. International Conference on Machine Learning. PMLR, 2024b.
Junjie Zhou, Yan Shu, Bo Zhao, Boya Wu, Shitao Xiao, Xi Yang, Yongping Xiong, Bo Zhang, Tiejun Huang, and Zheng Liu. Mlvu: A comprehensive benchmark for multi-task long video understanding. Xrv,v, 2024.
Luowei Zhou, Chenliang Xu, and Jason Corso. Towards automatic learning of procedures from web instructional videos. In Proceedings of the AAAI Conference on Artificial Intelligence, 2018.
Shuyan Zhou, Frank F Xu, Hao Zhu, Xuhui Zhou, Robert Lo, Abishek Sridhar, Xianyi Cheng, Yonatan Bisk, Daniel Fried, Uri Alon, et al. Webarena: A realistic web environment for building autonomous agents. arXiv, 2023.
Wanrong Zhu, Jack Hessel, Anas Awadalla, Samir Yitzhak Gadre, Jesse Dodge, Alex Fang, Young- jae Yu, Ludwig Schmidt, William Yang Wang, and Yejin Choi. Multimodal c4: An open, billion- scale corpus of images interleaved with text. Advances in Neural Information Processing Systems, 36, 2024.
25

---

Published as a conference paper at ICLR 2025
CONTENTS
Detailed Dataset Collection Process
28
Simulation
28
A.2 Web
28
A.3 Roaldworld
29
Data Annotation Protocol
29
General Guidelines
29
B,2
Data Format and Structure
30
B.3
Quality Control and Validation
30
B.4
Handling Ambiguilies
31
Ethical Considerations
B
6
Data Contamination Considerations
31
B.
7.7
Annotation Platform
31
8.8
Benchmark Preparation and Release
31
B.9
More Details of the Annotation Pipeline
32
Detailed Task Description
33
Ability Description
33
Physical Object Property Sub-task
33
Physical Object Relationships Sub-task
33
Physical Scene Understanding Sub-task
35
C,5 Physics-basse based Dynamics Sub-task
35
D More Dataset Analysis
36
D. Global Staties
36
D.2 Word Staties and Word Cloud
37
More Details on the Setup
39
Prompt for LLMs
39
3D Assets
41
Model Hyperparameters
41
Image VLM
Video VLM
44
E.3.3
General VLM
44
E,4
Prompt for VLM test
47
E.5
Reference Datascts Summary
48
6.6
Prompt Strategics
48
+.
7
ViperGPT Implementation
49
E
8
Human Performance
50
More Experiments Results
50
26

---

Published as a conference paper at ICLR 2025
F1
GroundinDDNNO Contiguration
50
Effect Visual Prompting
50
More PhysBench Results
PhysBench-val Results
59
R55
Emhodied Tasks Detailed Description
60
F6
Correlation Map
61
F7
Performance Related Benchmarks
62
G
More Related Works
63
H More Examples
65
H.1 Physical Object Property Sub-task
65
H.2 Physical Object Relationships Sub-task]
65
Physical Scene Understanding Sub-task
65
H.4
Physics-hased Dynamics Sub-task
65
Error Study
79
Detailed Staties
79
.22 Main Reason Analysis
79
Case Study
79
Discussion and Statement
141
Limitation
141
Boarder Impact
141
Ethics Statement
142
4
Reproducibility Statement
142
Latest Results
142
27

---

Published as a conference paper at ICLR 2025
A DETAILED DATASET COLLECTION PROCESS
A.1 SIMULATION
We use (Blender 2018) as our simulation platform. We utilized 679 objects and 470 HDR images to generate simulated videos and images. During each simulation, we concurrently save images or videos of depth, normal, and albedo, as well as the corresponding configuration files, which include the position, angle, movement and other properties of the object light source.
Image generation. addition to ambient lighting, we employed two point lights and one sunlight. To ensure data diversity, the positions of the camera and the arrangement of objects (ensuring no overlap and that all objects are captured by the camera) were randomized to some extent. Drawing from the object attribute annotation methods described in Newton (Wang et al. 2023c), we cleaned and re-annotated our data to develop a comprehensive table of objects and their attributes. Utiliz- ing this table, delineate the relational semantics between different objects and the corresponding queries. Following the approach in BLINK (Fu et al.) (024), each object in the simulated images is demarcated with bounding box rather than being explicitly mentioned in the text, thereby enhanc- ing the evaluation of the model's image comprehension capabilities.
Despite imposing considerable constraints there remains the possibility of minor object overlaps or incomplete captures of objects by the cam- era. To ensure that objects in the data are clearly identifiable, we employed GroundingDINO (Liu et al. 2023b) for detection. We only accept images where the labels detected by GroundingDINO match exactly in content and quantity with the generated labels. This process also provides us with the bounding boxes of objects for subsequent annotation. During the later stages of annotation, manual inspection of the images is conducted to ensure accuracy. To improve the detection success rate of GroundingDINO and reduce the probability of false detections, we set the box_threshold to 0.2 and the thxt_threshold to 0.2. Specifically, these parameter settings were obtained through a grid search, with detailed results presented in Table 14
code and meticulously annotating the 3D assets,
Videos with varying lighting conditions. We used only one point light source and arranged objects on plane to render shadows. The variations in lighting include three aspects: the color of the light, the position of the light source, and the intensity of the light. In terms the light source position, the movement involves translations along the x. y, and z axes. To avoid ambiguity in lateral directions (Du et al., 2024), during the dataset generation, the movement questions are typically framed in terms of moving along the line connecting two objects rather than simply asking for the direction of movement.
Videos with varying camera conditions. We used the same lighting and other configurations as in the image generation process. During video recording, we randomly altered the camera's 's position or shooting angle to capture the videos.
Fulid. We used assets from ContPhy (Zheng et al. 2024b) and Unity (Haas 2014) to generate videos across four types: fluid, rope, cloth, and ball, with 350, 250, 200, and 200 videos respectively. The videos were then manually annotated.
A.2
WEB
For web data collection, we primarily use predefined topics (e.g., gases) to retrieve relevant videos or images from the internet (such as middle school physics experiments). After filtering and clean- ing the data, we proceed with annotation. Additionally, we leverage large language models (LLMs) to generate suitable descriptions of physics-related concepts, which we then use to search for corre- sponding videos, followed by further cleaning and annotation.
In addition to the network data collection process described Section 3.2 we employ the following methods to gather data.
Unsplash. We use high-quality and high-resolution images from Unsplash Ali et al. (203). 57,859 images are downloaded, and finally we use only about 6,000 images.
28

---

Published as a conference paper at ICLR 2025
Manipulation. We sampled upproximately 500 videos from DROID (Khazatsyy al. 2024). Ego4D (Grauman et al. 2022), and MimicPlay (Wang et al. 2023a), providing detailed annota- tions to generate QA pairs categorized under object-manipulation tasks. The primary focus of these questions is to determine the appropriate sequence of actions based on given instructions, which derived from the original datasets" descriptions of actions. Figure 42|s both first and second examples provide an example of this task. First, we filtered the videos to select those with a strong alignment between the instructions and the visuals, ensuring that the videos were clear, unambigu- ous, and matched the instructions well. Next, we identified 3-4 keyframes from these videos. The task involved sorting these key frames in the correct order execute the instructions properly. Ad- ditionally, we used FunKPoint (Lai et al. 202T) to annotate the affordance (Gibson 2014) points in individual images from the original dataset. Specific examples of these annotations can be found in Figure 42 third and fourth examples.
nuScenes. We cropped and annotated videos from the nuScenes (Caesar al. 2019 mini and test datasets, ultimately obtaining 1,356 QA pairs for spatial movement tasks. We categorized the questions into types such as left turn, straight, and right turn, and included arrows on the images to indicate the direction. The questions asked participants to identify which image they might see based on the indicated direction, as illustrated in Figure 28
Visual Prompt. In certain tasks, we utilized Visual Prompts (Fu et al. 2024), and through experi- mentation, we identified an alternative annotation method, detailed in Appendix F2 For tasks using Visual Prompts, set the image size 1024 1024 pixels. In this scale, standardized the Visual Prompt to red circle with a 30-pixel radius and white text options with a font size of 65 pixels. The positions of the options' centers in the dataset are recorded the following format:
"A": 734, 922 J, "B": 2222, 898 "C": [43,, 115 1. "D": 4410, 559
Visual Correspondences. Drawing inspiration from Fu et al. 2024); Sarlin et al. (020)) we also annotated portion of the corresponding point data using visual prompts. Specific examples can be found in Figure 24
A.3 REAL-WORLD
We also collected some real-world videos and images, primarily covering sub-tasks related to light, camera, and physical dynamics such as collisions. An iPhone Pro Max was used as the recording device, and all images in RGBD format.
B DATA ANNOTATION PROTOCOL
B.1 GENERAL GUIDELINES
As previously discussed, there significant gap in existing benchmarks, which primarily as- sess vision-languag models (VLMs) based on descriptive tasks without adequately addressing their physical perception and reasoning abilities. To bridge this gap, our benchmark, PhysBench, is de- signed to provide comprehensive evaluation framework for physical perception, integrating visual understanding with the assessment of physical propertics, spatial relationships, and dynamic phe- nomena. This approach aims to advance AI systems toward more general-purye -urpoose capabilities in real-world physical environments. Our benchmark follows the guidelines outlined below for data collection:
General Principles:
Annotations must be accurate, consistent, and adhere to a high standard of academic rigor.
covers multiple tasks and topics to mirror real-world applications.
It incorporates diverse visual contexts and physics knowledge to foster a well-rounded evaluation.
- offers varying levels of challenge to effectively probe and uncover the potential limitations of current models.
29

---

Published as a conference paper at ICLR 2025
It provides robust evaluation settings for deterministic assessments.
Specific Instructions:
All questions must contain one or more images.
All questions should be written in English.
All questions should meet the college-level difficulty.
Questions should not be ambiguous and must be answerable with one of the given options.
Clearly categorize each question.
Annotate all fields, including the question, answer options and other things that follow the format requirement.
Review Process: Ensure that every annotation undergoes a peer review to maintain high standards and minimize errors.
Annotations such as physical properties, spatial relationships, dynamic interactions, and environ- mental factors also collected, providing detailed examples that demonstrate the physical percep- tion and reasoning capabilities of the models for further analysis and usage.
B.2 DATA FORMAT AND STRUCTURE
Detailed examples of annotated question examples shown in Figure 11 are provided in the guid- ance to serve as a reference for the annotators.
The structured JSON format will include fields for number, question type, question text, answer options (for multiple-choice), correct answer, question diffi- culty, and explanation (if there exists).
JSON File Format:
Naming Conventions:
Each collected sample will be stored in separate ISON file following standard naming rule: subject. [ooject_[Nuuberr [Nmbber]) }. json
Image Files: image. QueeNNum -[ImageNum] ppn .png
"scone": "black background",
"object":
"glass",
Following the video, which option's corresponding picture will happen first?
"web", "file SHINChiSLAgn.cc
A
questton":: "Folloxing the content of the cvsdess, which option's corresponding picture will happen riestiins. stragesinB. etaagesAnnC ctrage LnD. eirages",
"answer" "x"
"task type "dyceaics", "sub Eype": "tallisim", "ability type's "prediction", "Gescription": null, "mode" "general"
Figure 11: A question-answer pair case in PhysBench and its JSON representation.
B.3 QUALITY CONTROL AND VALIDATION
A secondary review team will rigorously vet annotations for quality and adherence to guide- lines.
Regular audits of random samples from the dataset will be conducted to ensure sustained quality and consistency.
Periodic training sessions will be held to update annotators on best practices and any changes in annotation guidelines.
Feedback mechanisms will be established to promptly address and rectify any identified emrors inconsistencies in the annotations.
30

---

Published as a conference paper at ICLR 2025
B.4 HANDLING AMBIGUITIES
Instances of ambiguity or unclear data should be flagged for detailed review. These instances will be collaboratively examined during team meetings establish standardized approach for annotation.
B.5 ETHICAL CONSIDERATIONS
Copyright and Licensing: Adherence to copyright and licensing regulations is strictly en- forced. Data from sources that prohibit copying or redistribution will be explicitly avoided.
Data Privacy: Compliance with privacy laws and ethical standards in data handling is paramount. Annotators must avoid collecting questions that contain any private informa- tion.
Ethical Data Usage: All data collection and usage must respect ethical guidelines. This includes avoiding biased harmful content and ensuring that the datasets promote fairness and inclusivity.
B.6 DATA CONTAMINATION CONSIDERATIONS
The risk of data contamination can be mitigated by assigning annotators to carefully select questions that extend beyond straightforward queries with easily accessible answers. It is essential that tasks rely on provided videos or images for answers rather than the common knowledge large language models. This approach is beneficial for creating benchmarks that genuinely test the model's ability to comprehend and synthesize information from diverse and challenging sources.
B.7 ANNOTATION PLATFORM
We developed GUI-based annotation platform, as shown in Figure T2 designed to assist human experts in the data annotation process. Through this program, experts easily view various media content, such as videos and images, and perform annotations and edits directly within an intuitive interface. The streamlined layout enhances the user experience, ensuring that experts can complete annotation tasks efficiently and accurately, thereby improving the quality and efficiency of the an- notations. The purpose of this tool simplify the complex annotation workflow, reduce manual effort, and make the annotation process more efficient.
Vide and images
Overview
Otesth
Sub Type
Mede
Bt
Figure 12: Annotation Platform.
B.8 BENCHMARK PREPARATION AND RELEASE
For convenience, PhysBenchh-ess consists 10,002 entries, organized into 19 subclasses. as the test set, and 200 entries the validation set for parameter choosing (PhysBench-val) since the answers
31

---

Published as a conference paper at ICLR 2025
for the PhysBench will not be publicly available and are hosted similarly to Yue al. 2024) We also present 89,998 entries for further research. The experimental results presented in this paper, unless otherwise specified, are based on the test set. Importantly, the answer labels for the remaining set will not be publicly released to prevent data leakage, and we will maintain an online evaluation platform. It should be noted that the scores we refer to for PhysBench are based on the entire test. dataset, including the validation split.
To ensure that each source dataset is well represented in the validation split and that the distribution of sub-task types and ability types in the validation set is similar to that of the entire dataset, we adopted the following sampling strategy: Randomly sample questions to ensure that the distribu- tion of sub-task and ability types in the validation set matches the full dataset. 2. Randomly sample the remaining questions from each source dataset based on its proportion in the entire dataset. Additionally, we conducted several quality checks to address any potential errors.
B.9 MORE DETAILS OF THE ANNOTATION PIPELINE
As described in Section 3.2 all questions were manually annotated by graduate students STEM fields and subsequently refined through a rigorous review process. The detailed workflow is depicted in Figure [3] where the icon A denotes stages involving human participation. The annotators were divided into three groups, each comprising six individuals. During the annotation process, either GUI interface similar to Figure or direct editing of JSON files was employed. The first group was responsible for video collection, the final group handled quality checks, and the intermediate group completed the remaining steps. A comprehensive explanation of this process is provided below.
Video Collection. Videos and images were sourced through web searches, simulations, and real- world recordings. The collection process utilized predefined simulation rules, LLM-guided queries, and other strategies to identify relevant content. Specifically, for data that could be simulated, we generated it using pre-written simulation scripts, as described in Section A Additionally, we con- ducted YouTube searches for videos, leveraging captions generated by GPT to guide annotators in collecting videos. We also sought compilations on topics like "interesting physical phenomena." For real-world recordings, we pre-designed scenarios and objects to capture. Given the complexity of collecting such data, we employed GPT as heuristic tools to expand the search scope and enrich the dataset's diversity. Finally, annotators curated the videos by trimming them to retain only relevant segments and provided detailed physical descriptions, such as the direction of shadow movement or the causes of observed events.
Question Design. In the previous step, utilized GPT to generate annotations for videos. This approach was adopted after observing that directly inputting a video and prompting GPT to gen- erate question-answer pairs in in-context learning format often led to suboptimal adherence to instructions. Instead of focusing on generating questions, GPT tended to explain the video con- tent. Additionally, inputting videos directly consumed more tokens, increasing computational costs. Furthermore, annotating videos with captions was considered beneficial for subsequent research leveraging our dataset. Given the complexity of collecting such data, we employed GPT as a heuris- tic strategy to broaden the scope of search and enhance the diversity of our dataset. GPT generated questions using an in-context learning approach, with the templates provided in Appendix [E.] No- tably, all final questions were curated and verified by human annotators, with GPT serving only as reference.
File Organization. We presented annotators with examples and detailed classification criteria, as outlined in Appendix [ and Appendix H Annotators were then instructed to categorize tasks ac- cordingly and structure each task in the JSON format shown in Figure
Quality Check. After organizing the dataset, we conduct a two-stage review process to ensure its quality. The dataset undergoes human verification to confirm that the questions are relevant to the physical world, rely on all provided input information, are not solely based on common sense, and accurately categorized with clear questions and corresponding answers. The first stage involves filtering and refining the questions, while the second stage focuses on thorough validation to ensure data accuracy and consistency.
Given the difficulty of acquiring such data, where expressing spe- cific properties often requires multiple images, undertook five-step process, spending total of 4,000 hours annotation.
32

---

Published as a conference paper at ICLR 2025
Rew Video
Aurotated
Atronted
QA
Orgminad (beoe
Filerr
S
Delizn
Denike
Modily
Rales Ag
Simnlation
Cipy Aunezate
Astittation
Filter
Real-worle
Videos
Ansotated Dutract
Ammoliidd
QA
Organioul
Video Collection
Video caption
QA Design
Organization Quality Check
Figure 13: Annotation Pipeline. Icon & indicate stages involving human participation.
C DETAILED TASK DESCRIPTION
This section primarily introduces the definitions of the various capabilities and sub-tasks. Specific examples can be found in Appendix H As illustrated in Figure humans not only accurately com- prehend visual content but also draw upon knowledge to explain and reason about the scenes they observe. This is the primary goal of most VQA datasets. Building on this foundation, several stud- ies have focused on the commonsense understanding of the geometric relationships and properties within the 3D world. However, the real physical world encompasses not only 3D geometric rela- tionships but also includes object properties (e.g., elasticity, ductility). physical object relationships (e.g., velocity, acceleration, depth). physical scene and environmental factors understanding (e.g., temperature, camera parameters, lighting conditions), and the principles and mechanisms of physical dynamics (e.g., the sequence of collisions, energy transfer, fluid flow, simple physical and chemi- cal reactions, optical and clectromagnetic dynamics). To address this gap, we propose PhysBench, which aims to bring Vision-Langune Models closer to achieving spatial intelligence (Gupta et al.
2021 Srivastava et al. 202T).
C.1 ABILITY DESCRIPTION
Table presents the description of capability types for PhysBench. The tasks categorized into four types: physical properties, physical object relationships, physical scene, and physics-based dynamics. Each task type corresponds different capability types such as recognition, compari- son, prediction, judgment, reasoning, and perception, with detailed descriptions provided for each. Specifically. the physical properties task type includes the recognition of object properties and com- parisons between the physical properties of different objects. The physical object relationships task type distinguishes between static and dynamic relationships of objects. The physical scene un- derstanding task type involves the prediction, judgment, reasoning, and perception of changes in environmental conditions. The physics- based dynamics task type evaluates the ability to predict, judge, reason, and perceive various physics-basee dynamics or phenomena.
C.2 PHYSICAL OBJECT PROPERTY SUB-TASK
Table 5) describes the sub-tasks for object types in PhysBench. The object types are divided into four subcategories: number, mass, color, and attributes. Specifically, the "number" subcategory in- volves the count of certain objects or changes in their quantity; the "mass" subcategory focuses on approximate changes object mass, mass estimation, or mass comparison; the "color" subcategory describes the color of objects or color changes; and the "attributes" subcategory covers object char- acteristics such as rigidity, fluidity, gas, hardness, malleability, elasticity, smoothness, and sharpness.
Notably, our dataset imposes more rigorous evaluations on conventional attributes like mass and color. For instance. in the case of counting tasks, previous works (Kafle & Kanan [017;: YYi al., 2019 Chen et al., 2022) typically only require identifying the number of objects in an image. In contrast, our dataset often ties quantities to specific object attributes (e.g.. "How many objects of certain color are outside the plate?" or "How many objects not blurred by the camera?").
C.3 PHYSICAL OBJECT RELATIONSHIPS SUB-TASK
Table 6 presents the descriptions of physical object relationships sub-tasks in PhysBench. The re- lationships types divided into five subcategories: size, location, depth, distance, and movement.
33

---

Published as a conference paper at ICLR 2025
Table 4: Ability Type Description for PhysBench.
Task type
Ability type
Description
Identify Comparison
Identify physical property of an object.
Physical Object Property
Comparison of the same physical property between different
objects, or changes in specific property over time.
Statie Dynamic
The static spatial properties of object.
Physical Object Relationships
The dynamic spatial properties of an object. meaning the spatial properties are changing dynamically.
Prediction
Predict what might happen certain environmental condition changed, what will happen Judge what will different if certain environmental condition modified.
Physical
Scene
Judgment
Understanding
Reasoning Perception
Explain occurrence reason condition of the environment. Understand what environment condition change has occurred and what its definition is, or determine the environmental at- tributes that caused the phenomenon to occur.
Prediction
Predict what might happen certain object or object attribute in the video is changed, or what will happen next. Judge what will change if a certain attribute is modified, or, for example, the sequence in which actions occur.
Physics- based
Judgment
Dynamics
Reasoning Perception
Explaining the occurrence reason or condition phenomena. Understand what phenomenon has occurred and what def- inition is, or determine the physical attributes that caused the phenomenon occur.
Table 5: Sub-task Description for Physical Object Property Type in PhysBench.
Sub type
Description
Number
The number of certain objects or changes in the number of ob- jects.
Mass
Approximate changes in mass, mass estimation, or mass com- parisons.
Color
Color of an object or changes in color.
Attribute
The attributes types of objects, such rigid body, fluid, gas,
stiffness, malleability, elasticity, smoothness, sharpness, etc.
Specifically, the "size" subcategory relates to the dimensions of an object; the "location" subcategory describes the absolute and relative positions of objects, including tasks related to object localization and spatial information processing: the "depth" subcategory focuses an object's 's depth relative to the camera or depth comparisons between different objects; the "distance" subcategory involves the comparison estimation of distances between objects as well as their absolute size; and the "move- ment"" subcategory addresses the analysis of movement direction, changes in speed, and changes in acceleration.
Table 6: Sub-task Description for Physical Object Relationships Type PhysBench.
Sub type
Description
Size
The size of the object.
Location
Positional relationships (absolute and relative), including di- rectly or indirectly locating objects and other tasks involving spatial information.
Depth
Depth of an object relative to the camera or depth comparisons between different objects.
Distance
Comparison or estimation of distances between objects or their absolute sizes.
Motion
Motion, velocity, acceleration and the direction of movement, changes in speed, or changes in acceleration.
34

---

Published as a conference paper at ICLR 2025
C.4 PHYSICAL SCENE UNDERSTANDING SUB-TASK
Table 7 describes the sub-tasks for physical scene understanding types in PhysBench. The physical scene understanding types are divided into four subcategories: temperature, camera, gas, and light. Specifically, the "temperature" subcategory involves temperature and its fluctuations, as well as dynamics caused by these changes; the "camera" subcategory focuses on changes in camera position and the resulting effects; the "gas" subcategory covers conditions of the gas environment, such as high pressure, low pressure, or vacuum states; and the "light" subcategory describes the color tone of the light source (warm or cool), changes in the position of the light source, light intensity, and the nature of the light source (point or surface).
Table 7: Sub-task Description for Physical Scene Understanding Type in PhysBench.
Sub type
Description
Temperature
The temperature and changes, as well as phenomena caused by temperature fluctuations.
Viewpoint
The position of the camera and its changes, along with phenom- ena caused by shifts in camera position. The air environment encompasses conditions such air pres- sure, humidity, airflow direction, and intensity.
Air
Light
Includes the color tone of the light source (warm or cool), changes in the position of the light source, its intensity, and the nature of the light source (point or surface).
C.5 PHYSICS-BASED DYNAMICS SUB-TASK
Table the sub-tasks for physics-based dynamics types in PhysBench. The physics-based dynamics types divided into six subcategories: collision, throwing, manipulation, fluid, chem- istry, and other physics-based dynamics. Specifically, the "collision" subcategory includes physics- based dynamics such friction between objects, collisions, and explosions; the "throwing" sub- category involves physical world dynamics and phenomena such as throwing and falling; the "ma- nipulation" subcategory focuses on the manipulation of deformable objects and affordance- sequence arrangements; the "fluid" subcategory covers fluid motion, shapes, and other fluid-related dynamics; the "chemistry" subcategory involves basic chemical reactions other dynamics phe- nomena related to chemistry in the physical world; and the "other" subcategory includes various other dynamics related to the physical world.
Table 8: Sub-task Description for Physics-based Dynamics Type in PhysBench
Sub type
Description
Collision
Friction between objects, collisions, explosions, and similar dy- namics.
Throwing
Throwing, falling, and other physical world dynamics. Manipulation of deformable objects and affordance-hased se- quence arrangements.
Manipulation
Fluid
Fluid motion, shapes. and other fluid- dynamics. Basic chemical reactions other dynamics involving chemical knowledge in the physical world.
Chemistry
Others
Other dynamics related to the physical world.
All of our questions designed multiple-choice, with only correct answer among the four options. Referring to BLINK (Fu et al.) 2024). the visual prompt is set as a red circle with a 10px size, which has been found to be the most suitable.
35

---

Published as a conference paper at ICLR 2025
D MORE DATASET ANALYSIS
D.1 GLOBAL STATICS
Question Distribution. Figure 14 fürther elucidates the distribution word counts, highlighting the diverse patterns of questions. The average word of the questions within PhysBench is 16.53, while the maximum number of words in question reaches 48.
10%
Median 15.00
8%
Percentage (%)
6%
4%
2%
24
32
40
48
Question Word Count
Figure 14: The distribution the number of words per question in PhysBench. Questions with length greater than 48 are categorized as 47 for visualization simplicity.
Option Distribution. Figure 15 further elucidates the distribution of word counts, highlighting the diverse patterns of options. The average number of words in the options within PhysBench is 4.36, while the maximum number of words in question reaches 20. It is worth noting that "option" here refers to the text following choice, such as "A. Point A," where the character length is 2.
24.00%
Median 3.00
20.00%
Percentage (%)
16.00%
12.00%
4.00%
0.00%
12
13
15
16
17
18
19
20
Answer Word Count
Figure 15: The distribution of the number of words per question in PhysBench. Options with length greater than 20 are categorized as 20 for visualization simplicity.
Image Resolution. The resolution of most images falls within the 1024-2048 range, accounting for approximately 79.7% of the total images. Only four images in the dataset have a resolution below 256. The minimum resolution is 183, while the maximum is 7201.
Video Resolution. Similarly, the majority of videos have resolution between 1024-2048, covering about 98.6% of the total videos. Only 1.3% of videos have a resolution below 1024. The highest video resolution is 1920, and the lowest 370.
Video Frames. Considering that VLMs typically extract keyframes when processing videes-ggeneraly selecting 6-8 frames-tth the total number of frames in the videos doesn't need to be large. However, this doesn't imply that our videos are limited to 6-8 frames. The frame counts primarily distributed in the ranges of 30-45 frames and 60+ frames, accounting for 59.4% and 35.9% of the total, respectively. The distribution charts for image and video resolution, as well as video frame counts, can be seen in Figure
36

---

Published as a conference paper at ICLR 2025
Image Resclutions
Video Resolutions
Video Frame Counts
1.N%
206
3
2046
D-15
58.4%
Figure 16: The distribution charts for image and video resolution, well as video frame counts. From left to right: the distribution of image resolution, the distribution of video resolution, and the distribution of video frame counts.
D.2 WORD STATICS AND WORD CLOUD
We created word cloud and word frequency chart for our dataset, as shown in Figure 17 It reveals a significant presence of terms related to physical world perception, such as "direction," "camera," "phenomenon," "effects," "relationship." " and "light." Additionally, word frequency chart for the LLaVA-1.5--33 training data, which includes 595K pretraining sam- ples and 665K instruction-tuning samples, as illustrated in Figure 18
generated word cloud and
VILA-I1.5133. In the pretraining data, CC3M (Sharma al. 2018) and COYO (Byeon et al. 2022) 25M primarily consist of image captions, while MMC4 (Zhu al. (2024) 25M includes webpage de- scriptions, none of which significantly contribute to the model's understanding of the physical world. Therefore, we mainly focus the instruction-tun tuning data. The LLaVA-1 1V-1----SSP 1--F-555FF-666K instruction- tuning data has already been used LLaVA-1.5-13B, so it is not considered further. Additionally, FLAN (Wei et al. 2021) consists purely of text data and is thus also excluded from consideration.
The data used for VILA-1.5-13B includes the following:
Captioning: Image Paragraph Captioning (Krause et al. 2017), MSR-VTT (Xu et al.
2016), TextCaps (Sidorov et al. 2020)
Reasoning: CLEVR (Johnson et al. et al. 2021) 2021)
2017), NLVR Suhr et al. 2017). VisualMRC Tanaka
Translation: Multi30k (Elliott 2016)
VQA: Activity Net-vAt Net-QA (Yu
al. 2019). DocVQA (Mathew et al. 2021 GQA Hudson
& Manning, 2019). iVQA (Yang et al. 2021). MSRVTT-QA (Xu et al.
2017 MSVD.
QA (Xu et al. 2017). OCR-VOR (Mishra et al. 2019). ST-VQA (Biten et
2019). Vi-
QuAE (Lerner et al. 2022), VQAv2 (Goyal et al. 2017b), Visual Dialog (Das et al.) 2017)
According to the guidelines from the VILA repository, the aforementioned instruction-tuning data all included in I (Li
al. 2023f). Therefore, we use M analyze the training data for VILA-1.5. The final word frequency statistics and word cloud for the VILA-I.5-13B training data shown Figure 19
PLLaVA-13B, PLLaVA-13B is based on LLaVA-Next (Liu et al. 2024b), with LLaVA-Next in- corporating additional data from ShareGPT-4V (Chen al. 2023a compared to LLaVA-1.5-132
The main enhancement from LLaVA-Next to PLLaVA-13B is the addition of 783k instructional video-to-text tuning data, enabling V--ext to handle video input. Specifically, the training data are sourced from the dataset used in VideoChat2 (Li et al.) (023e). which includes data for var- ious video understanding tasks, such as 27k conversation videos from VideoChat (Li et al., 2023d and Video-ChatGPP (Maaz et al. 2023b), 80k classification task data from Kinetics (Kay et al. 2017) and VthSthV2 (Goyal et al. 701 450k captioned data from Webvid (Bain et al. 2021). YouCook2 (Zhou et al. 20188), Te (Wu al. 2023c), and VideoChat, 117 reasoning data points from NextQA (Xiao et al. 2021b) and CLEVRER (Yi et al., 2019), and 109k annotated question- answering samples from Webvid, TGIF (Li at. 2016), and Ego4D) (Grauman al. 2022
37

---

Published as a conference paper at ICLR 2025
We used ShareGPT-44 and downloaded all the training data from the PLLaVA repository, creating a word cloud and word frequency chart of the training data, as shown in Figure
ball
CoUSe stick
SOUR
ATternti
DOT INTOO DISCED
change
fluida
alteroe
@ffectiplat
tiplate
drop
rinsest
SORIT Se
sater
sequenc
orderpullyy
phenomenon
watter
O
ect
effects
stepSphotncion
phenomena
aame
a
Mick
cont taneer
chang
Figure 17: Word Statics and Word Cloud for PhysBench.
bounding
concise
book
account
204800
desc
on
an
answer
box
features
expll exeeeplanee eep.eeeptannooo
Lanation
share
content
ohrase
THE
100990
56800
summa
reege
letter
Finnoon ion Inter Interpretaan entteprrtation
Figure 18: Word Statics and Word Cloud for LLaVA-1.5-13B Training Data.
400800
shape descriipiiitioos-e
hair Fraffic SAgeet
front man
bus
signtennis
0 glass
left
phone
Clock
BOEELO
table
bed
sidebench Ehingwe
Fight
peer
rerson
pOX
53890
basshall
foogbook
ffalss
eayy
cha 6o
100000
kind people truck wonan
rop
sink
bont
Bird
cat
obj
BCE
Figure 19: Word Statics and Word Cloud for VILA-1.5-13B Training Data.
Starg
descript
de
onssssne
canera
phrase
ex box
referene
events
coor coordinat dinate
man
book
color
person
sen
rennee
P
24033
etteoory
some
preterding
be
bounding collide
a
actionn Sen
yr yrupber
r
ion
hact lett
cat
colly
Buthor
cy
inderr
suchoice ob ject
Vi
Videeo
actions
Figure 20: Word Staties and Word Cloud for PLLaVA-13B Training Data.
38

---

Published as a conference paper at ICLR 2025
that the most frequent words in the training data of LLaVA- 1.5-13B, VILA-115-13B, and PLLaVA-13B are terms like 'description', 'phrase', 'summary' 're- gion', and similar words. Keywords such 'direction' appear much less frequently. We have listed the frequency several key terms from our dataset in the training data of LLaVA-1.5-13B, VII.A-1.5-3B,, and PLLaVA-13B, shown in Figure 21] Although PILLVA--13B includes words like 'collides' and 'camera', they are primarily used to describe phenomena without addressing the underlying physical mechanisms, which may explain why PLLaVA VL-3VB shows substantial im- provement.
Observing Figure 18 19 20
Figure 21: The frequency of common terms in PhysBench within the training data of the LLaVA-
1.5-13B, VILA-115-13B, and PLLaVA-13B models.
E
MORE DETAILS ON THE SETUP
E.
PROMPT FOR LLMS
Video Cases Prompt. The general system prompt for LLMs to generate video or image cases is as follows:
You are an assistant that communicates only in JSON and is an expert in physics. Do not write normal text.
Your response should be in JSON format with the following string fields:
1. 'video description': : A description of the video.
2. 'question": A multiple-choice question based on the video's content.
3. 'ansser": response to the question, with one of the values being 'A', 'B', 'C', 'D'.
For the same 'video description', you may provide multiple question-answer pairs, which should presented in a format.
Important notes: 1. The 'question' should not contain details from the 'video description'. 2. The relationship between each question-answer answer pair is one of juxtaposition, not progression. The following is conversation between the user and the assistant. Ensure that the assistant's 's answers are as rich and informative as possible.
Then we will the below sentence the first turn chat and let the LLMs give the more cases in the latter turns. We replace the examples provided the first turn to enable GPT to generate more diverse responses. Notably, some of the examples generated by GPT do not fully align with the data standards defined Appendix [ use these examples solely as references to collect more diverse data. However, described in Section 3.2, the subsequent data annotation undergoes rigorous five-step manual process to ensure high data quality.
The process of generating VQA templates is similar: questions generated based the captions of the input video or image. However, QA templates are more focused, targeting specific types of relationships, such those related to spatial positioning.
39

---

Published as a conference paper at ICLR 2025
("role": "system", "content": (Contents of the color block above)
{
("role": "user", "content": "give me some cases")
{"role": "assistant", "content":
Video Description: ball thrown horizontally from a cliff
Questions:
1. What causes the ball to fall downward?
A. Air resistance
B. Gravitational force
C. Magnetism
D. Elastic collision
2. Which of the following affects how far the ball travels horizontally?
A. Mass of the ball
B. Initial velocity of the ball
C. Color of the ball
D. Shape of the ball
3. What happens if the ball is thrown with a higher initial velocity?
A. It travels farther horizontally
B. It travels shorter horizontally
C. stops mid-air
D. It moves vertically upward
Answers: [B, B, A]
Video Description: A feather falling in vacuum chamber
Questions:
1. What happens if the vacuum chamber is filled with air?
A. The feather falls faster
B. The feather falls at the same speed
C. The feather floats
D. The feather rises upward
2. What would happen to the action it happened in strong magnetic field?
A. It would faster
B. It would float
C. It would fall slower
D. It would remain unaffected
Answers: [A, D]
("role": "user", "content": "give me some cases")
Caption. video or image caption system prompt is as follows:
You are outstanding video annotator and physicist.
Your task is to provide a detailed description given video based on the user's input video and brief description of the video.
The description should include, but is not limited to, the following aspects:
1. The content of the video.
2. The principles behind the phenomena observed in the video.
3. The properties of certain objects within the video, such estimated size, mass, color, and type.
4. Various spatial information, including relative and absolute positions, sizes, distances, and movements.
Ensure that the descriptions are thorough, precise, and reflect your expertise in physics.
Answer Extraction. The prompt used to instruct GPT-4o is illustrated as follows:
The following sentences contain answers (one of A, B, C, D) and corresponding analysis.
Your role is to find the answer. Please return only of the four letters: A, B, C, D.
The sentences are:
40

---

Published as a conference paper at ICLR 2025
E.2 3D ASSETS
The overview of the 3D assets is shown in Figure 22 After defining the object attribute table, we selected 678 objects and annotated their attributes accordingly. For each object, then adjusted its size and position to ensure that multiple objects (typically 4-5 in our dataset) do not overlap and remain clearly visible. These objects were subsequently used for simulations.
adhesire upp
apple
sstmmy
bud
dish
flying
macket
lendage
bechall
baseet
chinsware
eupbooee
Figure 22: 3D Assets Sample. We have 678 objects and 470 HDR environment total for simula- tion.
E.3 MODEL HYPERPARAMETERS
In our experiments. conducted comparisons with some of the most recent and representative MILMM in the following. In our experiments, we conducted comparisons with some of the most recent and representative MILLMs in the following. We divided the models into three categories: Image VLM, Video VLM, and General VLM, according whether they support only one image input, support a video input (also supports image), or support interlaced video and multiple images.
Following Zhang et al. (2024)). for Image VLMs, we concatenate multiple frames from video (defaulting to 8 frames) into a single image, arranged from left to right and bottom to top, before inputting it into the model. Thus, both Image VLMs and Vidco VLMs were tested question- answer pairs involving only a single image or video, which represent a subset of PhysBench. In contrast, General VLMs, which support interleaved image-text sequences, were tested the full PhysBench test split.
For most open-source models, used the hyperparameter torch dtype torch.float16. However, for some models that only support torch dtype = torch. bfloat we used torch dtype torch. torch.bfloatle b for those cases. Additionally, for other parameter settings, generally followed the configurations provided in the original papers, their code repositories, or the examples from Hugging Face.
E.3.1 IMAGE VLM
The following is a description of the specific models reviewed, as well as the specific parameter configurations, can be found in Table
BLIP-2 (i et al. 2023c employs dual-stage strategy bridge the modality gap fectively, utilizing a lightweight Q-Former pre-trained on 129 million image-text pairs. the first stage, the model initiates the learning of vision-language representations by leveraging frozen image encoder, ViT-g/14 from EVA-CLA CLIP (Fang et al. (023). In the subsequent stage, a frozen LLM, FlanT5 (Chung et al. 2024). is employed to facilitate vision-to-language generative learning. This innovative approach enables effecctiee zero- shot instructed image-to-text generation. The tested version BLIP2-15-x (we call it as BLIP-2 in this paper).
InstructBLIP DDii al. 2024) is derived from the pre-trained BLIP-2 model (Li et al. 2023c), which integrates a ViT-g/14 image encoder, a Vicuna (Chiang et al. 2023). and Q- Former that bridges these two components. During the vision- languae instruction tuning process, only the Q-Former is fine-tuned. utilizing data from 13 distinct visual question- answering datasets. The tested version is IP---- (we call it as InstructBIIIP.
41

---

Published as a conference paper at ICLR 2025
t5-xl in this paper), InstructBLIP-15- -5xxll (we call it as InstructBLIP-t5xxt in this paper), InstructBLIP-vicuna-777 (we call it as InstructBLIP-7B in this paper) and InstructBLIP- vicuna-138 13b (we call it as InstructBLIP-13B in this paper). It is worth noting that both InstructB1IPP (irruutt..P-viccnaa-11 and of questions. For these instances, we disregarded these questions and did not assign any scores.
13b provided no responses to a small part
LLa al. (2023a). LLaVA (Liu al. 2024c establishes connection between the visual encoder ViT-L/14 from CLIP (Radford et al. 2021) and the language decoder LLaMA (Touvron al. 2023) using a lightweight, fully connected (FC) layer. Initially, the system trains this FC layer with 595K image-text pairs, while keeping both the visual encoder and LLM static. Subsequently. LLaVA fine-tunes both the FC layer and the LLM using dataset of 158K instructional vision- vision-language pairs. LLaVa-1.5 (Liu et al. (023a) is an enhanced version of LLaVA, trained on additional datasets. The tested version is LLa and LLa (we call it LLaVA-1.5-7B and LLaVA-1.5-13B in this paper).
Qwen-VL-Chat VL-Chat (Bai et al. 2023b) introduces a series of large-scale vision-language mod- els (LVLMss designed to perceive and understand both text and images. The Qwen-VL. series is built upon the Qwen-LM (Bai al. 2023a) foundation, augmented with visual capabilities through a meticulously designed visual receptor, input-output interface, and a three-stage training pipeline using multilingual multimodal cleaned corpus. These mod- els excel in image description, guestion-answering ing image-caption-box tuples. The series includes Qwen-VL and Owwn-VVL-Che both of which achieve state-of-the-art performance on a broad range of visual-centrie benchmarks and real-world dialog benchmarks, demonstrating their superiority over existing vision- language chatbots. The tested version is Qwen-VL. VL-Chat, as Qwen-VIL VL-Chat have stronger instruction fellow ability than Qwen- VL.
grounding, and text-reading by align-
LLeVA-NeXT (Liu et al. 2024b) is the latest iteration in the LLaVA series (Liu et al. 2024c), building upon the foundation of LLaVA-1.5 (Liu et al. 2023ap. This new model enhances visual reasoning. OCR, and world knowledge capabilities. It increases input im- age resolution to four times more pixels, supporting resolutions up to 672x672, 336x 344, and 1344x336. LLaVA-NeXT features improved visual instruction tuning data mix- ture, further enhancing its visual reasoning and OCR capabilities. Additionally, it sup- ports better visual conversations across various scenarios and applications, demonstrating improved world knowledge and logical reasoning. Despite these enhancements, VA- NeXT maintains the minimalist design and data efficiency of its predecessor, utilizing less than 1M visual instruction tuning samples. The tested versions are LLLVV-NeeTT-mistral and
7b TLLaVAI.6mmistra models being Mistral-7b (Jiang et al. 2023) and Vicuna-7b (Chiang et al., 2023). respec- tively (we call it as LLa LLaVAl.6-mistre and LLa VA1.6-vicua this paper).
(LLaVAL.6-vvcuuna) with the base
InternVL-Chat- V1-5 (Chen et al., 2024c) is an open-source multimodal large language model with enhanced visual understanding through continuous learning vision encoder, dynamic high- nigh-resolution image processing, and hiihtquulity bilingual dataset. The tested version is Intern VL-Chat- Shhht--i-- 11-5-quantable (we call as InternVL-Chat as the GPU memory size restriction of 40G A6000.
Cambrian-1 1 TTong et al., 2024 a family of multimodal large language models (MLLMs) designed with vision-centric approach. This model series addresses the gap between lan- guage models and visual representation learning by thoroughly evaluating various visual representations. Cambrian-1 introduces the Spatial Vision Aggregator (SVA), dynamic and spatially-aware connector that integrates high-resolution vision features with large lan- guage models (LLMs) via cross-attention layers (Dai et al., 2024) while reducing the num- ber of tokens. The tested version is Cambrian- 1-bb (we call it as Cambrian-80 in this paper).
MiniCPM-V Yao et al. 2024). MiniCPM-V2 is a robust multimodal large language model designed for efficient end-side deployment. The model is constructed by inte- grating SigLip-400M (Zhai et al., (203) and MiniCPM-2.4B (Hu et al., (024), connected through perceiver resampler. The latest iteration, MiniCPM-V2.5. further improves upon
42

---

Published as a conference paper at ICLR 2025
its predecessors. Built on SigLip-400M and Llama3-8B BB-Inttruct with a total of 8B parame- ters, MiniCPM-V2255 demonstrates sigmificant performance enhancements over MiniCPM- V2. The most recent and advanced model in the MiniCPM-V series, MiniCPM-V22 V2.6, is built on Sigl.ip-400M and Qwen2-7B (Bai et al.] 2023a), also with a total of 8B pa- rameters. MiniCPM-V2.6 shows substantial performance improvements over MiniCPM- L12 and introduces new capabilities for multi-image and video understanding.
MolmoE (Deitke et al. 2024). MolmoE is a family of open vision-language models devel- oped by the Allen Institute for AI. The Molmo models are trained on PixMo, a dataset of I million highly curated image-text pairs, and exhibit state-of-the-art performance among multimodal models of similar size, while remaining fully open- source. MolmoE-1B is a multimodal
large language model (LLM) with 1.5B active and 7.2B total parameters, based on L..MoE-I--B--A
(Muennighoff et al.) (024). It closely matches the performance of GPT-4V on both academic benchmarks and human evalua- tions, achieving state-of-the-art performance among similarly open multimodal mod- els. Molmo 7B-0. based on OLMo- BB-044 (a preview of the next generation of OLMo models), utilizes OpenAI CLIP as its vision backbone, performing between GPT-4V and GPT-4o on both academic benchmarks and human evaluations. Molmo 7B-D, based on Qwen2-7B and also using OpenAl CLIP as its vision backbone, performs similarly, sit- ting between GPT-4V and GPT-4o in both academic benchmarks and human evaluations. It powers the Molmo demo available at on Qwen2-72B with OpenAI CLIP as the vision backbone, achieves the highest academic benchmark score and ranks second in human evaluations, trailing GPT-4o by only a small margin.
Finally, Molmo 72B, based
Xinyuan-VL (Group 2024). Xinyuan- VL-2B high-performance multimodal large model developed by the Cylingo Group for end-user applications. It is fine-tuned based on Qwen2-VLL VL (Wang et al., 2024c) and trained on over 5 million multimodal data sam- ples, with additional training
small amount of plain text data.
A Vuuila-VL (Gu et al. 2024). The Aquila- VL-2B model is VLM trained using the LLava framework. The Qwen2.5-1.5B (Wang et al.) 2024c) model serves as the LLM, while siglip-so400m-patch 4-384 is employed as the vision tower. The model was trained on our custom- built Infinity-MNN -MM dataset, which consists of approximately 40 million image-text pairs. This dataset combines open-source data collected from the internet with synthetic instruction data generated using open-source VLM models.
DeepSeek-VL (Lu et al. 2024a). DeepSeek- VL is an open-source Vision- Languu-L (VL) model designed for real-worke applications involving vision and language understanding. It demonstrates broad multimodal capabilities, enabling the processing of logical diagrams, web pages, formula recognition, scientific literature, natural images, and embodied intelli- gence in complex scenarios. The models we tested, with sizes of IB and 7B parameters, are respectively sourced from watpseeekka/deeeseeeee-ee v1-1.33-chht and deepseek-ai/deeessed vl-7b-chat.
PaliGemma 2 (Steiner al. 2024) represents significant advancement in vision- language modeling, building upon its predecessor by incorporating the sophisticated Gemma architecture. Following the approach of PaLI-3 (Chen et al.] (023b). the PaliGemma model family utilizes open-source components. specifically combining the SigLIP vision model with Gemma 2 language models (Team et al.) 2024). This multimodal system effectively processes both visual and textual inputs to generate multilingual outputs. The architecture has been optimized achieve superior fine-tuning performance across a comprehensive range of vision-language tasks, including image and short video caption- ing, visual question answering, optical character recognition, object detection, and instance segmentation. The model variants include paligemma2- 10----1-oc-----i-----1 paligemma2-30 ft-docci-448.
Claude (Anthropic) 2024) is a large language model developed by Anthropic with a strong focus safety, interpretability, and alignment. It has been designed to align with human intent and values, using various methodologies like reinforcement learning from human feedback (RLHF) to ensure that the model behaves intended. Claude integrates mecha- nisms to reduce harmful or biased outputs and is optimized for interactive dialogue across range of domains. In this paper, the tested version is Claude-3-opus, Claude-3-sonned
43

---

Published as a conference paper at ICLR 2025
(luude--haiikl Claude-3.5 3.0--33.5.-snttt Except for Claude-3-haiku where images used in their original size, the images for the other three models were resized to 128x 128 due to cost considerations. Notably, both Claude-3-sonnet and Claude-3.5-sonnet demonstrated poor adherence to instructions, often failing to provide direct answers as requested in the prompt, instead offering explanations alongside the options. Therefore, we used GPT-4o to extract the answers for scoring, with the prompt provided in Appendix
E.3.2 VIDEO VLM
The following is a description of the specific models reviewed, well as the specific parameter configurations, can be found in To
Chat-Univi (Jin et al. 2023) is unified vision-language model capable of comprehending and engaging in conversations involving both images and videos through a unified visual representation. Chat- Univi employs set of dynamic visual tokens to uniformly represent images and videos, enabling it to efficiently utilize a limited number of visual tokens to cap- ture the spatial details necessary for images and the comprehensive temporal relationships required for videos. This approach is supported by multiscale representation that allows the model to perceive both high-level semantic concepts and low-level visual details. The tested version is Chat-Univis 7 and Chat-Univi--3B.
Video-LLaVA (Lin et al., 2023a)is a sophisticated multi multi-modal framework designed to empower large language models (LLMs) with the ability to understand both visual and au- ditory content in videos. Unlike previous models that handle either visual or audio signals independently, Video-LLa integrates both to achieve comprehensive video comprehen- sion. The tested version is Video-LLaVA- (we call it as Video-LLaVA in this paper).
PLLaVA (Xu et al. 2024 (Pooling LLaVA) is an advanced video understanding model designed to extend srragebbsee models to video, enabling dense video caption genera- tion. The model employs a simple yet powerful pooling module to bridge image-finetune Vision-Language Models (Vision-LLM) into the video domain, achieving significant per- formance improvements in video captioning tasks. The tested versions are PL-LaVA-7B and PI.LVVA-1BB. Notably, since the model uses num frame a parameter during loading, our approach for processing images involved duplicating the image for the num frames input.
We also evaluated VideoChatGPT (Maaz et 2023a Video-LLaMA (Zhang et al., 2023 and VideoChat2 (Li et al., 2024e), and observed that these models exhibit poor costruction-folloiing capabilities. Despite repeatedly prompting them to provide options, the models consistently re- sponded with descriptions of the video or image content rather than addressing the questions posed. As a result, we did not include specific evaluation metrics for these models in our study.
E.3.3 GENERAL VLM
The following is a description of the specific models reviewed, well as the specific parameter configurations, can be found in Table [1 12
LLaVA-NexT-Interleeyy LLi et al. 2024d) is a multimodal large language model that extends LLaVA's capabilities to handle multi-image, video, and 3D data. By using an in- terleaved data format, it unifies single-image and multi-modal tasks, enabling it to transfer knowledge across different scenarios. The model is trained on the M4-Instt dataset, which includes over million samples spanning multi-image, video, and 3D tasks. The tested versions are inaaeainneerreeee--ee-----bb-- (we call as LLaVA-interleave) and llava- interleave-gween---ddoo
(we call it LLaVA-interleave--po in this paper).
VILA-1.5 (Lin et al. 2023b) is a multimodal visual language model (VLM) designed to handle both multi-image and video understanding tasks. It is pretrained large-scale interleaved image-text data, which enhances its ability to perform tasks such as video rea- soning and in-context learning. The tested versions are VIL3-I15-3B, VILA-I.5-3B-s2, VII.A-1-5-88 1-5-8B and VIL---1.--33R
NVILA (Liu et al. 2024f) is a family of open Vision-Language Models (VLMs) opti- mized for efficient video and multi-image understanding. We enhance VILA's architecture
44

---

Published as a conference paper at ICLR 2025
Model
Generation Setup
BLIP-2
torch.dlype torch.float16. tokens 200
InstruciBLIP-t5xx
torch. sicch_dlye-aa torch.float6,, max_new tokens 200
InstructBLIP-t5xxx
torch.dtype torch.float1,, max tokens = 200
InstructBLIP-7B
torch.dtype tor.....o.coo,,oooomoooooooooo,e max new tokens 200
InstruciBLIP-13BB
torc..flot16. max_new Jokens- 200
LLaVA-1.5-7B
torch.dtype = fo max_new tokens 200, do_sample False
LLaVA-1.5-13B
torch dtype tor.c..
tokens 200. do sample False
torch dtype torchblaao6
L-aVa1.6-mistral
torch_dtype = torch.ftohh....... max max_new_tokens. = 200, do_sample = False
LLaVAl.6-vicuna
torch.dtype torch.float6,, max.new Lokkenns = 200, do do.sample = False
torch, torch_dtype torch.bfloat16, max new tokens 512, num beams 1, do somsam False
IntemVL-Chatl.
torch. torch.float16, max new mokens 512, temperature = 0, num. hemmme = 1, use_cache = True
Cambrian-8B
dtype = www....ootta....mmmoooo context None, sampling False, temperature 0.1, max_new.toker = 10
MiniCPM-V2
= www.toooo.......a. context None, sampling = False, temperature = 0.1,
MiniCPM-V2.5
max_new-token 10
dtype = wor.h.oloat32.... context None, sampling = False, temperature = 0.1,
MiniCPM-V2.6
max new token 10
MolmoE- 1B
dtype torch. stop pop_sttings stop /eedoftext/ |>",
dtype 'auto'. max new tokens = 200
MolmoE- -B-O
dtype torch.float2) stop_string coop_sstings-... stop_strings=" |/doftoxxt |>",
dtype 'auto', max_new_tokeeeeeecemeeeeee new_tokens =20
MolmoE-78 B--no
dtype torch.float32. stop strrings stop wop.stingg==<<<cooofooxtt
endoftoxt |>",
dtype 'auto', waa_nnewtokees. 200
MolmoE-72B
dtype torch.float32 stop strings stop /nnnff==ex<ddofeextt |>".
dtype 'auto', max_new_token.. 200
Xinyuan-VI VI.
max new tokens 128, resize (1024, 1024)
Aquila-VL
do.samplc False, temperature = 0, max maxx_eea_ new_tokens = 4096
BLL-Seek-VL-IB dtype = torch. bfch.bfloat16,
max_new nww.etsstsksss
10. do sassa
use cache=True
Doe-Seck-VL-7B
dtype = torch.bfloat16. max_new new_tokens
10, do_sample=False,
cache=True
Claude-3-opus
max_new_tokens 1000, temperature
Claude-3-sonnet
max max_new.tokens 1000. temperature 0
Claude-3-haiku
max new tokens 1000, temperature 0
Claude-3.5-somee
max_new_tokeme 1000, temperature =0
Table 9: Generating parameters for Image VI.M. Parameters not explicitly stated indicate the use of the model's default system settings.
45

---

Published as a conference paper at ICLR 2025
Model
Generation Setup
torch_dtype torch.float6,, wax.nne..okke. 1024, temperature 0.1, use_cache True, do_sample True
Video-LLaVA
torch.dtype torch. torch.flatt6,, num beams = 1, use.cache True, do damse = False, top-p None, out- put = True, return dict. generate True,
mrik_new_tokenn 10, lemperature 0,
Chat-Univi-7B
=1
torch dtype torch.floate 16, max new tokens = 10, temperature = 0, num heams = 1, use cache = True, do sample = False, top p None, out- scores = True, return dict in generate True, length penalty =
Chat-Univi- Univi-13B
conv mode = conv eval videoqahench, max new tokens = 2556, do sample False
PLLaVA-7B
conv conv_mode conv mwx_new = 256, do da_sample= Falsc
PLLaVA-13B
Table 10: Generating parameters for Video VLM. Parameters not explicitly stated indicate the use of the model's default system settings.
through "scale-then-comprees approach, increasing spatial and temporal resolutions be- fore compressing visual tokens, enabling efficient processing of high-resolution images and long videos. Through systematic optimization across training, fine-tuning, and deployment phases, NVILA achieves comparable or superior performance to leading VLMs while re- ducing training costs by 4.5x, fine-tuning memory by 3.4x, and latency by up to 2.8x. The code and models are publicly available for reproducibility.
Phi-3V (Abdin et al. 2024). Phi-3 is a family of open AI models developed by Microsoft, designed be the most capable and cost-effective small language models (SLMs) avail- able. Phi-3 models outperform other models of the same size and even those in the next size up across various language, reasoning, coding, and math benchmarks. Phi-3V the VLM based on Phi-3. The tested version is Phi-3V-128k (we call it as Phi-3V in this paper).
Phi-3.5V MA 2024). Phi-3.5V, released on November 15, 2024, is advanced version of Phi-3V. This statc-of-the-arr lightweight multimodal model is built on datasets comprising synthetic data and curated, publicly available web content, with focus high- quality, reasoning-rich data in both text and vision. As part of the Phi-3 model family, the multimodal version supports context length of 128K tokens. The model has undergone a comprehensive enhancement process, including supervised fine-tuning and direct prefer- ence optimization, ensure accurate adherence instructions and robust safety measures.
mPLUG-Owl3 (Ye et al., 2024). Here's refined version of your sentence: mPLUG- Ow13 is a state-of-the-art multimodal large language model designed to address the chal- lenges of understanding long image sequences. mPL.U--O13 introduces Hyper Atten- tion, method that enhances the speed of visual sequence processing in multimodal large language models by sixfold, enabling the handling of sequences up to eight times longer. Simultaneously, mPLUGGOW13 maintains exceptional performance across single- image, multi-image, and video tasks. The tested versions are mPLUG-OwI3- 1B (mPLUG- Owl3-1B 1B-2410,4), mPLUG-OW13-2B (mPLUG-OWI3-25 (2T-O-3--B-2201--11 and mPLUG-Ow13. 7B MPPLU--OWS--7--211100
Intern VL2 (Wang et al. 2024e). Intern 2.0 is the latest iteration in the Intern VL se- ries of multimodal large language models. includes range of instruction-tuned models, with parameter sizes ranging from billion to 108 billion. In comparison to state-of tate--oftth-- art open-source multimodal large language models, Intern 2.0 outperforms most open- source alternatives and demonstrates competitive performance that rivals proprietary com- mercial models. Its capabilities include document and chart comprehension, infographics question answering, scene text understanding and OCR tasks, scientific and mathematical wroblem-solving as well as cultural understanding and integrated multimodal processing. Intern VL 2.0 is trained with 8K context window and incorporates training data consist- ing of long texts, multiple images, and videos. This training significantly enhances its abil- ity process and understand these types of inputs, surpassing the capabilities of In
46

---

Published as a conference paper at ICLR 2025
merge-based based approach rather than sequential processing for video data to optimize GPU memory consumption, as demonstrated in Table 28
1.5 (Chen et al. 2023c). For larger model variants, we implement a merge-
Intern 2.5 (Gao et al. 2024b) was released December 9, 2024, representing an advanced iteration in the multimodal large language model (MILM) series. While main- taining the core architecture of InternVI. 2.0, it introduces significant enhancements in training strategies, testing methodologies, and data quality. The model preserves the "ViT- MLP-LLM" paradigm established by its predecessors, Intern VL 1.5 and 2.0. This new version integrates a newly incrementally pre-trained Intern with various pre-trained LLMs, including InternLM 2.5 and Qwen 2.5, utilizing randomly initialized MLP pro- jector. Consistent with previous implementations, Intern 2.5 employs pixel unshuffle operation, reducing visual tokens to one-quarter of their original count, and adopts dy- namic resolution strategy similar to Intern 1.5, processing images in 448x448 pixel tiles. A significant advancement since InternVL 2.0 is the expansion of capabilities to in- clude multi-image and video data processing. For larger model variants, we implement merge-based approach rather than sequential processing for video data to optimize GPU memory consumption, as demonstrated in Table
LLaVA-NeXT- Video (Zhang et al. 2024b) is multimodal large language model designed to excel in video understanding tasks through zero-shot modality transfer. Trained primar- ily on image data, it demonstrates impressive performance on video tasks by leveraging deep learning models with DPO training and Al feedback. The model supports various de- ployment scenarios, from cloud environments to edge devices, making it highly versatile. is part of the broader LLaVA- VA-NeXN suite, focused on advancing visual- visual-language integra- tion. The tested version are -.aVV-- Video 7B- 7B-Qwen and LLLVA VA-NNT- Video-7B- Qwen-dpo, we call them as LLaVA-NV and LLaVA-NV-dpo respectively in this paper.
Mantis CTiag et al. 2024 is multimodal large language model designed for interleaved multi-image tasks. Built on the LLaMA-3 architecture, excels in co-reference, reason- ing, comparison, and temporal understanding. Mantis uses the Mantis-Instruct dataset, containing 721K examples, to train on various multi-image skills. It achieves state-of-the- art performance some interleaved benchmarks, while maintaining strong single- image performance on par with CogVLM (Wang et al.) 2023b). The tested version are Mantis- and Mantis-clip-llamen
Idefics2, Mantis LLaVA, Mantis-siglip-li
GPT (Achiam et al. 2023). GPT-4V is advanced multimodal model that extends GPT- 4's capabilities with integrated vision processing, allowing it to understand and generate text based on visual inputs. GPT--oo is optimized variant designed for better performance in language tasks while maintaining lower computational requirements. GPT-4o-mini is a more lightweight version of GPT-4o, designed for deployment in resource-constrained environments while still providing strong performance in language understanding and gen- eration tasks. The version of GPT-4V used GPT-4-turbo. Notably, due to token length limitations, all images were resized to 512x512 before being input into GPT. Our testing was conducted around mid-August 2024, using the most advanced model available that time. The remaining models were also tested during this period.
Gemíni-1.5 (Team et al. 2023 Gemini- 1.5mini-1.5-Flash 1.5-slssh and Gemini- 1.5-ini-1-5-Pro P.5-Pro are advanced mul- timodal large language models, each designed with distinct capabilities for different perfor- mance needs. Gemini-1.5-Flass emphasizes fast processing and efficient memory usage, making ideal for tasks requiring speed on devices with limited resources. Our testing of Lemini-1.5--laas and Gemini-1.5- P.5-Pro was conducted around mid-August 2024.
E.4 PROMPT FOR VLM TEST
Referring to Liu et al. 2024c), during testing, we appended an end prompt to each inestion-answer pair (i.e., the value corresponding to the "question" key in the Figure The end prompt as follows:
Answer with the option's 's letter from the given choices directly. You can only answer one letter from A, B, C, or D.
47

---

Published as a conference paper at ICLR 2025
Consistent with Zhang et al. (2024a), when asking video-related questions to Image VLMs, prepend the prompt with the following statement:
This is series of images sampled at equal intervals from the beginning the end of a video. Based on the series of images, output the best option for the question.
For Video VLMs, due to their weaker mnstruction-folloinn prompts during testing guide the models in selecting the correct answer rather than simply de- scribing the video content. (Despite this, many models still responded with video descriptions rather than answering the questions, as noted in Appendix E.3.2) For Video LLMs, if the question involves only a single image input, we prepend the following statement:
capabilities, we provided additional
Based on the image, output the best option for the question. You must only output the option.
Add the last line in the prompt:
The best choice option is:
If the input involves only a single video, we prepend the following statement:
This is a series of images sampled at equal intervals from beginning to the end of a video. Based on the series of images, answer the question. Based on the video, output the best option for the question. You must only output the option.
E.5 REFERENCE DATASETS SUMMARY
Our dataset was entirely annotated by humans and underwent two rounds of rigorous filtering and screening. The data sources we utilized, along with their respective uses, are detailed in Table 13
E.6 PROMPT STRATEGIES
In Section 4.1 we explore several prompting strategies:Chain of Thought (CoT) (Kojiaa et al. 2023) with the prompt "Let's think step-by-step!":; Desp-CoT (Wu et al. 2023d). which begins with an image description prompt; and Pure Language Reasoning (PLR), similar to Mathvista (Lu et al. 2024b) The specific prompt content can be found in Appendix [.6 Following the settings Kojima et al. (2022); Chen et al. (2024b), we extract the final generated answer through GPT-4o- mini. In this section, provide the prompts used to implement these three strategies. Note that for Phi-3V:
(1.1) The prompt for CoT (Kojima et al. 2023 is:
Let's think step by step! Start by selecting the correct option's 's letter from the given choices, then provide detailed explanation of your thought process.
(1.2) The prompt for Desp-CoT Coo (Wu et al. 2023d is:
Each image or video is followed by description, which you can refer to. Answer with the option's letter from the given choices directly.
(1.3) The prompt for PLR is:
Each image or video is replaced by description. Answer with the option's ' letter from the given choices directly.
For GPT-4o:
(2.1) The prompt for CoT Kojima et al. 2023 is:
Let's think step by step! Start by selecting the correct option's 's letter from the given choices, then provide detailed explanation of your thought process.
48

---

Published as a conference paper at ICLR 2025
(2.2) The prompt for Desp-CoT (Wu et al. 2023d) is:
Each image or video is followed by description, which you can refer to. Answer with the option's letter from the given choices directly.
(2.3) The prompt for PLR is:
Each image or video is replaced by description. Answer with the option's letter from the given choices directly.
We used the following prompts to generate the corresponding descriptions for each model (for image descriptions, "video" replaced with "image" where applicable):
Give me the description of this video.
E.7 VIPERGPT IMPLEMENTATION
Following the methodology described in ContPhy (Zhng al., 2024b), since the code for its model, ContPro, has not been released, we implemented an oracle neural-symbolic model using ViperGPT (Surs et al.) 2023), which we refer to as ContPhy, and evaluated it on the PhysBench. Similar to ContPhy (Zheng et al., 2024b), we decomposed the drestion-answeriig task into four main modules: video perception, physical simulation, program parser, and symbolic exccution.
Given raw video, the video perception module detects objects and their associated static attributes using the MASK-RCNN detector (He et al., 2017). The physical simulator takes point clouds as input and predicts object dynamics across various scenarios using dynamic prediction models (Li jet al. 2018 Sulsky et al.) 1995). The program parser, powered by a large language model (we use GPT-4o). trans lates the question query into executable Python programs. Based the detected object attributes and predicted dynamics, the symbolic executor runs the programs to derive the answer the question.
Since neuro-symbolic approaches rely on pre-trained domain-specific modules to extract objects' static attributes, physical properties, and dynamic trajectories, this method is suitable for the complex and diverse tasks in PhysBench. For instance, in ContPhy, simulating four types of tasks rroe, cloth, ball, and fluid requires pre-training three models: MASK R-CNN (He al. 2017) based on Detectron2 (Wu et al. 2019), DPI-Net (Li et al. 2018). and Material Point Method (MPM) (Sulsky et al., 1995). Given the variety of tasks in PhysBench, where even individual tasks within certain subcategories can differ significantly, it is impractical to design and call specific module for every task. Therefore, we designed an Oracle model for the QA pairs generated through specific simulations to ensure that the tasks are similar enough to allow the use of consistent logical processing templates following Zheng et al. (002b), while GPT-4o directly provided answers for the remaining questions without invoking additional modules running Python code.
Following the approach in ContPhy, we use ResNet-50 (He et al. 2016) as the backbone for MASK R-CNN densely detect object locations in each frame and associate static attributes such color and material. We use the default config from Detectron2, while the number of classes is different across scenarios. Specifically, the batch size is 16 for GPUs thus each mini-batch has 2 images per GPU. We train the model for about 10k iterations, with a learning rate of 0.01. For image size, we keep the original resolution.
We fine-tune the network using the training set data from all simulations. For scenes involving fluid and bodt-body dynamics, we adopt MPM, while DPI-Net used for other tasks. The training data for these two modules is similar to that of MASK R-CNN, and the overall training procedure is comparable ContPhy. After extracting objects' static attributes, physical properties, and dynamic trajectories, and parsing the natural language query into an executable program, we run the program using the object states as input and produce the predicted answer. Due to the rich diversity and versatile resources of our dataset, it is challenging to ensure that templates perfectly fit all predefined modules. For example, some videos become significantly distorted when rescaled into a square format, and in certain videos. fluids are difficult to extract points due to their similar color to the environment. For the specific training parameters of MPM and DPI-Net, we have kept them consistent with ContPhy.
49

---

Published as a conference paper at ICLR 2025
E.8 HUMAN PERFORMANCE
To evaluate human performance on PhysBench, we recruited 12 graduate students in STEM fields and provided monetary compensation for their participation. Each question was assigned all annotators. To ensure the quality of the results, we followed the methodology of Math (Lu et al.) 2024b) by implementing qualification questions during participant recruitment. These questions tested basic knowledge of physical world concepts, and only those who answered the qualification questions correctly were deemed eligible for the study. Given the large number of questions in PhysBench, the testing was divided into 10 sessions, delivered as online questionnaires with no time constraints for completion. The average score across participants was used as the final measure of human performance.
F MORE EXPERIMENTS RESULTS
F.1 GROUNDINGDINO CONFIGURATION
GroundingDINO has two key parameters: box threshold and text threshold. The box threshold parameter is used to filter predictions based on the confidence level of the detected bounding boxes, ensuring that only boxes with confidence scores above this threshold are retained. On the other hand, the text. threshold parameter filters predictions based on their relevance to the input text prompt, retaining only those that meet or exceed this threshold.
It is important to note that the filtered phrases may contain significant number of duplicates. To address this, we consider a prediction successful only if the set output phrases matches exactly with the set of input phrases, ensuring both completeness and accuracy. The detailed experimental results are presented in Table
F.2 EFFECT OF VISUAL PROMPTING
To enhance the VLM''s responsiveness to visual prompts, we assessed its sensitivity to these prompts. The images used in tests were all 1024x 1024 pixels. We employed two different annotation methods for depth and attribute sub-tasks, as illustrated in Figure 23 drawing on BLINK (Fu et al.) 2024) for reference. For depth, the default color for circles is red. Method (a) follows the ap- proach introduced in the BLINK paper, while the second method aligns exactly with the instances in BLINK-eval. Additionally, the prompts for both methods are largely similar to those in the BLINK- eval examples. Similarly, for attributes, we experimented with two annotation methods, denoted as (c) and (d) in Figure 23 However, since the white text doesn't 't look very clear in the yellow checkbox in the first method, the yellow color was selected.
(a)
b
(c)
(d)
Figure 23: Several different labeling methods. (a) (b) are two methods for labeling depth. (c) (d) are two labeling methods for attributes. Attribute also try (a)(b)
The test images comprise 1000 samples, with uniform circle radius of 20 pixels and font size of 25 pixels. The detailed experimental results are presented in Table [5] For samples (a) and (b), the text prompts require selecting the location or object indicated by the prompt, while samples (c) and
50

---

Published as a conference paper at ICLR 2025
(d) involve identifying bounding box point. For instance, answers for (c) and (d) might be "A. The object enclosed in the red box at point A" using both color and letter identifiers.
Based on the experimental results, we find that the outcomes for (a) and (b) are similar in terms of depth and attribute, though (a) performs slightly better. For attributes, (c) and (d) yield results close to random, likely because the tested models cannot perceive color differences in the bounding boxes, thus failing to answer effectively. However, (a) and (b) significantly improve accuracy. Therefore, we ultimately decided to use the annotation method in (a).
Building on the findings of the previous experiment, we further tested the impact of varying circle sizes using the annotation method, which showed the best model performance. The corresponding results are detailed in Table Additionally, the text size was consistently 5 pixels larger than the center of the circle. Based on the previous experiment, we adopted the annotation method (a) for all subsequent annotations. Therefore, we used method (a) to test depth, with a sample size of 1000 images.
We have tested 6 scales as shown in Figure 24 and the results indicate that for LLaVA- 1.5, larger sizes generally yield better results. For Phi-3V and VILA-1.5, the optimal radius is 30 pixels, although performance does not show significant relationship on either side of 30 pixels. Most models show low sensitivity to size variations, except for LLaVA--155775 To balance performance and aesthetics, we ultimately selected radius size of 30 pixels.
10
20
30
Figure 24: Visual Prompt with Different Size.
F.3 MORE PHYSBENCH RESULTS
The performance of 39 models across 8 ability categories in PhysBench is presented in Table 17 Note that the ability classifications for the physical scene understanding and physics- dynamics categories are identical, though their content differs slightly; these are combined here for clarity. Detailed descriptions of the ability classifications can be found in Appendix C.I while the remaining 36 newly tested models are listed in Appendix K
The details for Ability Type, Physical Object Property, Physical Object Relationships, Physical Scene Understanding, and Physics-based Dynamics can be found in Table [7 Table [8 Table Ta- ble Table 21, respectively.
51

---

beams 1, Published as a conference paper at ICLR 2025
Model
Generation Setup
LLa LLaVA-interleave
torch.dlype torch. max.new tokens 10, do_sample False
LLaVA-interleave-dpo
torch. sich.dlype-aa torch.foatt6,, max new tokens = 10, do vo_sample False
torch.dtype = torch.float1,, max_new tokens 4000, temperature 0.1, num_beams usc_cache False, do do_sample = Falsc, top-p None
VIL.A-I.5-BB
torch dlype = torch.float16. num_beams use_cache False, do_sample = False, top-p None
tokens 4000, temperature 0.1.
VII.A-I.5-BB-22
torch dtype = torch.float16. max new tokens 4000, temperature 0.1. num_beams use_cache False, do_sample = False. top-p None
VII.A-1.5-8B
torch dtype = torch.float1,, beams use cache False, do sample False. top None
tokens 4000, temperature 0.1,
VILAA-1.-13B
torch drype torch.float1,, max new tokens 500, temperature 0, do n0.samplc-- False, length lengthh.enalty repetition repetition.penalty- penalty = (When in error analysis the max new tokens is set to 5000)
Phi-3V
toreh_dtype torch.float16, max_nee tokens 500, temperature do so.sample False, Length_penaile 1, pnpeeitiionpenallyy (When error analysis the max pww_ne___ooe tokens is set to 5000)
Phi-3.5V
LLaVA-NV
torch.dtype torc.. max www.nnew_tkkeee
10
LLVVA-NV-dpo
torch dtype torch.fffoat1.,,..coomooooo max new tokens 10
Mantis- Mantis-Idefies2
torch dtype tww.too.....ho.... torch.hfloat16,
tokens 10, do sample False
torch_dtype torch. max_new_tokens =1, num_beams 1, do_sample False, length penalty
ManVis-LLaVA
torph.dtype torch.bfloat16, max.new.tokenn
beams 1.
Mantis-siglip-Ilame llama3
do do.sample- False
torch.dtype torch.bfloat16, max.new.token do sample False
Mantis-clip-lamma
mPL.UG-OW/3-1B
max_new tokens 100, decode True
mPLUG-OW13-2B
max_new_tokens 100, decode_text True
mPLUG-OW13-7B
max new tokens 100, decode text = True
dtype torch.bfloat1,, max new tokens = 10, do sample False, max num 12
IntemVL2-1B
dtype torch.bfloat16, max_new_tokens = 10, do_samplc False, max_num
IntemBVL2-2B
12
dtype torch.bfloat max_new_tokens = 10, do_samplc False, max_num 12
IntemVL2--4B
dtype torch.bfloat16, nokets 10, do.sample False, max :2
Intern VL2-8B
dtype torch.bfloat16, max new wow.ts 10, do da.samlle False, max num
Intern VL2-26B
dtype torch.bfloat16, max_new_tokens 10, da_sample False, max_num
InterVL2-40R
dtype torch.bfloat16, max_new_tokens 10, do_samplc False, max_num
Intern LL2-76L
max_new_token Lokens 300, temperature 0, seed 42 (When in analysis 2000)
GPT-4o
the max_new meneeeeettoees
GPT-4o-mini
max_new._ooe Lokens 300, temperature seed 42
GPT-4V
max fox_new__toe wokens- 300, temperature 0, seed 42
Gemini-1.5 1.5-flash
stream = True
Gemini-1..5pro
stream = True
Table 11: Generating parameters for General VLM. Parameters not explicitly stated indicate the use of the model's ''s default system settings.
52

---

Published as a conference paper at ICLR 2025
Model
Generation Setup
dlype torch.bfloat16. tokens 10, do sample False,
InterVVL2..-BB
dtype torch.bfloat16, nww._new.tolee 10, do so.se False, max num
IntemVL2..52B
dtype torch.bfloat16, max_new_tokens 10, sampe False, max_num
IntemVL.2.544
dtype torch.bfloat16, max_new.tokens 10, do sample False, max_num
Intem
dtype torch.bfloat16. max new tokens 10, do sample False, max num
Intern V1.2.-222
dtype torch.bfloat16, max new tokens 10, do sample False, max num
Inter VL2.5538
dtype = torch.bfloat16, wwx.new.aeee...m www.newwwooeee tokens 10, do.samplc False, max_num
VL2.57L2..5-8B
=1
NVILA-8B
default
NVILA-I5B
default
NVILA-Lite-88
default
NVILA-Lite-155:
default
Table 12: Generation parameters for more models (continued from Table 11]. All other parameters not specified here use the default model configurations.
Datasets Reference for our PhysBench. 'None' indicates that the dataset does not explic- itly state which license is being used.
Table 13:
Dataset
License
Description of
Unsplash
b5ope: /ww.//nppaaaa..e. com/ License
images
labeling.
Conthyy Zneng at
2024h
CC-BY 4.0
We used his code to genemate come the simulation data. As data, annotated QA.
ChronoMagee
2024
Apache 2.0
DROID Khazalsky
2024
CC-BY 4.0
physics-basee based
As souroe
annotated some
manuplation type.
belonging
As data, annotated some data belonging physics-busee
Ego
manuplattno type.
As source data, we annotated some QA data belonging physics-based dynamics-manpllut
MimicPlay Wang
20236
CC-BY 4.0
type.
nuScenes Ca Caesar al.
2019
CC BY-NC-SA
As source data, we annotated some QA data belonging to spatiull movement type.
Nerwon Wang et
CC-BY
As reference objects.
attribute
FunKPoin Lai et
2021
None
As source data, annotated some QA data belonging physics- based dynsmics-mamuplation type. As source data, we annotated some QA data belonging to physical scene understanding cameenag----ra type.
Photo Tourism Snavely al.
2006
None
Table 14: GroundingDINO Accuracy with different box threshold and threshold. The number of samples tested is 1000.
bos threshokd
threshold
0.1
0.2
0.3
0.4
0.5
0.6
0.7
0.1
0.015
0
0
0.2
0.015
0.075
0.005
0.005
0
0
0.3
0.01
0.03
0.06
0.01
0.005
0
0
0.4
0.005
0.015
4
4
0
0
0.5
0.005
0.005
0
0
0.6
0.005
0
0
0
0
0.7
0.005
0
0
0
0
Table 15: Comparison Visual Prompts for Depth and Attribute Annotation.
Depth
Attribute Prompt (b)
Prompt (a)
Prompt (b)
Prompt (a)
Prompt (c)
Prompi(d)
LLaVA-1.5-7b
27.3
31.5
29.9
25.2
155
21.2
20.2
41.1
38.5
26.0
25.6
Phi-3V-128k
78.2
75.8
41.2
40.1
24.8
25.1
V1LA-1.5-8b
19.9
20.3
27.9
26.1
24.7
53

---

Published as a conference paper at ICLR 2025
Table 16: Effects of different visual prompt circle radius Depth task performance.
10
20
30
40
50
60
LLaVA-1.5-7h
19.2
27.3
27.3
28.4
30.9
31.2
-LaVA-1.5-13b
19.9
21.2
24.9
25.2
25.8
27.9
Phi-3V-128K
74.7
78.2
78.8
76.8
73.1
73.0
VILA--15-8b
20.3
19.9
26.9
25.3
26.7
22.2
Model
Identify Comparison Static
Dynamic Perception Prediction
Judgment Reasoning
Image
InstructBLIP-5-xll
2024
48.30
25.39
36.38
35.76
40.06
34.07
42.74
35.83
InstuctBll -a-BuctBLLIP-S-xxXPP0 ec 2024
$7.63
28.40
33.21
53.64
41.90
32.62
47.58
36.28
InstructLLP--BB al. 2024
25.02
19.31
30.98
24.50
32.21
23.74
29.91
18.28
InstructHLIP-33111
39.49
25.70
32.46
35.33
35.88
27.12
33.62
20.18
BLIP-2
20253
55.97
30.78
36.01
53.64
42.51
33.24
48.15
34.29
LLaVA-T3--B
2023a
47.78
31.26
40.30
45.70
47.40
39.90
45.01
37.38
LLaVA-1.5-138 Lau
52.48
32.70
42.16
42.38
51.17
40.87
43.02
32.65
LLaVAl.6-mistral Liu 20244b
36.79
23.82
23.13
21.19
29.15
15.93
28.21
4.97
112VA1.6vvccuml
21/44
42.38
49.75
43,30
37.13
Oween-VL-Chall Bar 20255
51.44
23.75
44.96
39.74
45.16
37.09
50.14
24.25
InterVVL-Chat 1.3 Libb
2024c
64.78
43.96
74.81
56.29
53.62
41.35
44.16
34.39
Cumbrian-81) Tong
20.52
22.02
33.65
11.68
25.65
Claude-3-opus Amthropp
53.71
32.70
35.82
58.28
43.93
32,90
43,30
27.83
Claode-3-soneel Anthropic
A074
47.78
29.97
36.75
$2.32
38.43
36.47
43.02
30.22
Claude Shaikk- Antiroppc
35.71
27.73
(laude-3.5soonxx(2211n (Antroopi
2024
61.46
34.33
36.19
58.94
43.32
34.00
47.01
25.15
Video VLM
Video-LLa Lin X2028a
47.08
29.15
35.45
35.10
42.71
40.39
43,39
31.16
Chat-Univi--7Ejj 785
25,46
14.54
18.10
31.79
24,16
29.74
31.05
18.29
Shat-Univi-11 383 at 2023
6.10
3.07
12.50
7.95
13.86
9.00
9.69
17.10
PLLaVA-7B Xu 2024
47.95
30.31
31.72
49.67
42.71
39.42
40.46
34.74
PLLaVA-138
50.39
31.60
36.94
43.71
44.44
39.77
41,60
29.17
General VL.M
Interleaved Data
LlaVV-interleavee ei 20744
59.37
37.68
44.21
49.022
47.40
39.07
28.52
32.74
LLaVA-interleav-
2024d
61.73
37.41
40.84
28.77
31.40
VILA--L5-3B
B Lin all 20ASI6
38.01
28.46
32.15
40.52
39.65
36.15
32,99
33.65
VILA-I.5-3B Lan ci muzsh
38.71
29.22
29.12
39.87
39.04
30.97
31.71
34.99
VILA-L.5-8B Lin
39,06
28.97
39.87
37.41
42.48
21.23
30.54
VILA-1.5-136 Lin ci 2014h
50.74
32.90
39.25
47.71
44.24
35.94
31.97
29.44
Phi-3V/Alvinn al 2024
55.01
34.68
36.64
$2.94
47.81
35.06
32.35
32.74
LLaVA-NV Zhang (024bb
29.85
38.56
42.41
38.76
31.63
32.22
ALaVA-NV V-poo Zhang
2024b
47.52
32.22
45.28
40.52
43.53
38.35
32.01
31.74
Mantis-Idefies2 Dning lil
2024
55.36
31.88
41.18
41.83
42.92
36.76
33,25
27.00
Mantis-LLa Jiang
59.81
32.76
29.28
41.18
41.18
36.28
30.43
33.75
sigtis- laama Jiang cl
2024
55.01
32.63
32.72
30.72
42.41
40.84
31.46
34.47
Mantis-llip-llanaa Jiang 2t
52.40
31.60
34.71
36.60
40.16
42.75
31,33
30.68
GPT-4V/Achimm Achiam
GPT-4V
65.13
37.41
44.05
64.71
55.15
34.99
41,43
24.41
GPT-4o Achiam al
20243
71.23
45.73
64.20
71.90
56.37
42.75
43.99
28.63
GPT-4o-mini Achiim
2023
74.46
42.90
41.25
35.04
27.96
Gemini-1.3 1.5 1.5-Ifasi Team
2025
74.89
43.55
51.51
60.78
54.23
39.05
33.76
31.69
Gemini-L.S-pp Team al.
2023
72.28
45.46
63.57
65.36
53.41
37.92
37.08
34.85
PhysBench includes two evaluation dimensions: ability and task. Table presents results categorized by the task-type dimension.
Table 17: Evaluation Results for 39 VLMS Categorized by Ability Dimensions.
54

---

Published as a conference paper at ICLR 2025
Model
Number
Mass
Color
Attribute
Image VL.M
InstructBLIP-45-xl Dai et al.
(2024
47.14
22.18
44.33
31.37
InstructBLIP- xxl Dai et [2024]
52.86
43.00
54.00
33.22
InstructB1IP-77B Dai el al.
(2224
11.23
34.93
35.57
20.66
InstructBI.IP-1311 Dai
2024
29.12
28.67
43.33
30.77
BI.P-2
20123c
51.30
50.85
47.33
34.63
LLaVA-I.3-7B1 Liu et al.
(2023a
38.99
36.86
52.67
35.42
LLaVA-1.5-13B Liu
2025a
41.59
52.22
56.67
35.63
JLaVAl.6-mistee Liu al. 2024bb
34.49
32.42
32.67
26.76
L. .6-vicuna Lau et al. 2024bb
41.07
35.15
61.33
36.41
Qwen- CW-n--
Bai et al.
2023bb
48.35
21.50
51.33
30.45
Intem IntemVI-Chatl... Chen
/202cc
53.73
57.68
71.00
48.12
Cambrian-8B Tong et al.
2024
38.13
14.68
25.67
18.52
Claude-3-opus Anthropic A
2022
45.75
54.95
50.33
35.77
Claude-3-sonnet/ Anthropicc
2024
38.30
44.71
50.00
33.64
Claude-3-haiku Anthropic
2024
40.73
53.24
62.00
38.18
Claudc-3.5-sonnel Anthropic 2024
56.67
58.02
60.33
36.69
Video VLM
Video-LLaVA Lin et al.
2023a)
38.13
28.33
53.00
34.49
Chat-Univi-7B Jin et al. 2023
22.70
22.87
23.00
16.32
Chat-Univi- 13
2023
2.43
3.07
4.00
5.32
PLLaVA-7B et 2024
35.36
48.46
56.33
33.07
PLLaVA-13H Xu ct al.
[2024
34.84
47.44
65.00
35.06
General VLM
Interleaved Data
LLaVA-interleavee
(2024d)
51.30
47.44
65.33
41.66
LLaVA-interdeave--pPa
2024d
55.29
41.30
67.33
42.23
VILA-I15-3B Lin et al. 20233
39.17
29.01
27.00
31.37
VILA-I.5-3B-s2/ Lin
2023b
38.82
30.72
28.00
32.29
VILA-1 Lin ct al. 202.3
39.86
24.23
33.33
32.72
VILA-1.5-13B Lin
2023b
48.01
36.52
40.00
38.47
Phi-3V
2024
43.15
37.88
63.00
40.81
LLVVA-N'ZZamn el al. (2024b)
35.70
38.57
55.67
35.70
LLaVA-NV-dpo Zhang et al. (2024b)
36.40
39.59
54.67
36.34
Mantis-Ideficee Jiang
2024
47.49
32.76
65.33
36.69
Mantis-LLaVA Jiang ct al.
2024
58,41
40.27
61.33
35.98
Mantis-siglipp 1iaiip--llaa11111 Jiang ct al.
2024
45.41
37.88
61.00
38.11
Kantis-clip-llama3 Jiang
2024
42.29
40.96
57.67
36.05
Achiam et al. 2023
52.51
48.81
71.00
43.79
GPT-4o Achiam el al.
2023
61.18
54.27
71.00
52.52
GPT-4o-mini Achiam al.
(2023)
69.67
50.85
72.33
43,29
Gemmini-1.5-flas Team
2023
72.10
54.95
72.33
48.65
Gemini-1 1.5 1.5-pro Team ct al.
(2023
64.99
58.02
75.00
50.04
Table 18: Evaluation Results for 39 VLMs in PhysBench Physical Object Proppetyy Sub-task (the
fifth last column of Table 3)
55

---

Published as a conference paper at ICLR 2025
Model
Size
Location Depth Distance Motion
Image VL.M
InstructBLIIP-55-I Dai 2024
37.50
38.72
30.53
41.67
53,93
InstructBL.IP-55xxx/ Dai el al.
2024
48.21
41.35
27.72
51.67
57.30
InstructBL.IP-7B Dai et al. 2024
33.93
30.92
26.35
20.00
37.08
InstructBLIIP-13111 Dai
2024
26.79
39.10
25.26
43.33
48.31
BLIP-2 Li al. 2023c
58.93
44.74
29.47
46.67
59.55
LLaVA-T.3-7BB Liu et al. 2023a
64.29
43.23
35.09
41.67
51.69
LLaVA-115--33B
20255
57.14
46.24
32.63
45.00
60.67
TT.VAII.--miss
Liu et al. 2024b
26.79
19.55
24.56
10.00
37.08
[J JlaVAI.6-vccuna
Juu cf al. 20244
39.29
51.88
72.98
51.67
59.55
Qwen-VT.-Cat Bai al.
(20.3b
33.93
42.11
45.96
38.33
52.81
IntermVL-Chat Chen
20244
75.00
66.17
76.14
58.33
61.80
Cambrian-8B Tong et al. 2024
14.29
7.89
31.93
11.67
5.62
Claude-3-opus Anthropic
(024
62.50
38.72
31.23
46.67
71.91
Claude-3-sonnet Anthoooo 2024
73.21
42.11
28.77
31.67
62.92
Claude-3-haiku Anthropic
2024
64.29
54.14
47.37
51.67
68.54
Claude-3.5-somme Anthropic 20244
71.43
46.24
25.96
36.67
69.66
Video VLM
Video-LLaVA Lin et al.
(2023a
39.29
40.60
27.72
43.33
50.56
Chat-Univi-771 et al.
202.3
28.57
19.55
17.89
25.00
35.96
Chat-Univi-133B Jm
2023
17.86
16.17
6.32
11.67
15.73
PLLaVA-7B: Xu ct (2024
48.21
34.59
28.42
41.67
60.67
PLLaVA-1385 Xu al.
2024
50.00
42.86
28.42
38.33
57.30
General VLM
Interleaved Data
LLaVA-interleave
al. (2024d
48.21
53.95
69.47
60.00
37.48
LLaVA-interleave-ppo Li
2024d
41.07
53.26
62.46
63.33
36.17
VILA-I.5-3B Lin 202.5b
50.00
40.89
45.61
46.67
28.42
VILA-11553BB Lin et al.
12023b
57.14
37.80
37.19
36.67
26.76
VILA-1.5-8BB Lin et al. 2023b
37.50
22.34
29.47
40.00
31.47
MLLA-1.5-13B Lin
(0236)
64.29
52.92
60.35
53.33
32.57
Phi-3V al. 2024
66.07
53.61
74.39
48.33
26.69
LLVAA Zhang et al. (224bb
32.14
35.40
49.47
45.00
26.00
LLaVA-NV-apo Zhang et al. (2024b)
32.14
35.74
53.68
45.00
50.55
Mantis-Idefics2 Jiang
2024
39.29
58.42
65.61
65.00
32.50
Mantis-LLaVA Jaang of al. 20244
37.50
47.08
31.93
46.67
26.56
Mantis-siglip-llamaa Jaang ct al.
2024
41.07
41.58
27.72
38.33
32.23
Ilamss Jiang ct 2244
55.36
39.86
33.68
43.33
33.89
GPT-4V Achiam et 2023
75.00
59.45
61.40
58.33
38.87
GPT-4o Achiam et al. 2023
85.71
70.45
73.33
83.33
60.58
GPT-4o-mmij Achiam et al.
2023
69.64
51.55
57.19
63.33
39.21
Gemini-1. 1.5-lllshh Team
12023
67.86
70.79
66.32
71.67
44.81
Gemini- 1.5 1.5-pro Team ct al.
2023
66.07
71.48
82.11
68.33
58.30
Table 19: Evaluation Results for 39 VLMs in PhysBench Object 7 Relltionssips Sub-task (the forth last column of Table
56

---

Published as a conference paper at ICLR 2025
Model
Temperature Viewpoint
Light
Image VLM
InstructBI.IP-15-xx Dai al. 2024
70.59
43.26
26.00
30.03
InstructBLLIP-55 Dai et al.
2024
80.88
43.36
40.00
30.30
InstructBLIP-7B Dai et 2024
36.76
22.94
12.00
14.74
SstructBLIP-13BDDD
57.35
21.61
30.61
19.86
BLIP-2
2023c
79.41
40.90
30.00
29.48
L..VAA-.5--BB Lau al al.
2023a
61.76
44.58
24.00
31.76
LLLVA-1.5-13BB
B. Jau al.
202.5
50.00
40.06
38.00
28.03
I.
Liu
2024b
45.59
5.66
40.00
8.01
LLaVA1.6-vicuna Lit et al.
2024b
57.35
44.67
32.00
31.76
Owen-VL-Chat
Bai al. 202.3b
66.18
20.55
20.00
29.66
L et (2024g
72.06
40.43
34.00
31.48
Cambrian-8B Tong et al.
2024
11.76
22.05
20.00
24.66
Claude-3-opus Anthropic
2024
72.06
29.59
86.00
27.93
Claude-3-sonnet Anthroopi
2024
54.41
31.39
36.00
31.30
Claude-3-haiku Anthropic
2024
73.53
26.86
42.00
29.94
Claude-3.5-sonnei Anthropic
22024
75.00
27.80
68.00
24.48
Video VLM
Vidco-LLaVA Lin et
(223aa
67.65
39.40
30.00
26.02
Chat-Univi-7B Jin ct al.
2023
50.00
14,80
20.00
20.75
Chat-Univi-13 Jmn ct al.
(2023
20.59
13.29
16.00
17.93
PLLaVA-7B Xu (2024
60.29
40.81
22.00
31.12
PLLaVA-13B Xu 2024)
66.18
30.63
36.00
30.39
General
VLM + Interleaved Data
LLaVA-interleave Li 2024d
64.71
36.48
41.82
32.85
MaVVA-inerlccav--ppo)
al. 2024d
61.76
35.34
47.27
30.21
VILA-1.5-3B Lin 2023b
54.41
43.26
14.55
26.11
VILA-1.5-3B-s2
2023b
51.47
46.18
10.91
25.48
VILA-1.5-8B Lin et 2023b
44.12
36.29
23.64
25.02
VILA-1.5-13B Lin et al.
2023b
60.29
32.52
14.55
30.21
Phi-3V/Abdin
2024
52.94
38.93
65.45
29.48
LLaVA-NV Zhang ct al. 20244b
52.94
36.29
18.18
31.12
LLaVA-NV-dpo Zhang ct al, 2024bb
Zhang
51.47
35.25
20.00
31.94
Mantis-Idefics2 Jiang ct al. 2024
63.24
27.43
47.27
29.39
Mantis-LLaVA. Jiang
2024
58.82
41.28
23.64
30.57
Mantis-siglip-llama. Drang et al.
2024
51.47
42.41
43.64
30.57
Mantis-clip-llama3 Jiang el al.
2024
52.94
33.93
27.27
30.21
GPT-4V Achiam
2023
88.24
25.07
85.45
22.02
GPT-4o Achiam ci al. 2023
91.18
22.05
83.64
32.67
GPT--o-mmi Acham et al.
2023
82.35
28.56
83.64
27.75
Gemini-1.5-ffal flash Leam et 2023
85.29
33.74
78.18
30.48
Gemini-l.5-pro Team el al.
202.3
85.29
37.32
76.36
31.85
Table 20: Evaluation Results for VLMs in PhysBench Physical Scene Understanding Sub-task (the third last column of Table 3).
57

---

Published as a conference paper at ICLR 2025
Model
Collision Throwing Manipulation Fluid Chemistry Others
Image VLM
InstructBLIP-55. Dai
2024
31.49
34.68
27.89
32.41
62.16
58.16
InstructBLIP-15
et
32.72
31.35
29.90
33.33
52.70
58.84
InstructBL.IP77H5
2024
19.97
27.96
29.94
27.03
38.10
Instruct81.1-133111 Dar
2024
28.26
27.89
35.65
33.78
BLIP-2 al. 2023c
33.64
35.39
30.40
31.17
59.46
58.84
LLaVA-1.5-7BB Liu cl al.
2023a
36.25
37.77
41.21
45.37
59.46
55.10
LLaVA-1.5-1333 -3 Liu et
20225a
33.18
34.92
46.73
50.15
60.81
60.20
LLaVA 1.WA1.6-mmg mistral
et al. 2024b
24.12
11.64
12.81
7.56
62.16
50.34
LLaVAI.6-vicuns Liu
22024b
33.03
35.39
44.97
45.22
52.70
61.90
Qwen-VI.-Chal Bai al. Z0235n
32.57
33.25
37.19
47.84
59.46
59.86
IntemVI--Charl33 Chen
2024c
35.94
39.90
42.71
43.67
55.41
73.47
Cambrian-8B Tong
2024
24.88
39.43
28.89
30.56
36.49
23.47
Claude-3-opus Anthropic
2024
35.79
28.98
25.63
27.78
58.11
69.73
Claude-3-sonnel Anthropic
(2024
38.25
30.64
26.13
35.19
44.59
57.48
Claude-3-haiku Anthropic
2024
39.90
34.42
30.40
67.57
Claude-3.5-someel/ Snn-Soome//nnbopppic
2024
34.25
37.05
27.64
28.09
50.00
71.09
Vidco VLM
Video-I.I Lin al.
2023a
35.48
38.48
29.15
46.30
54.05
53.74
Chat-Univi-7B et 2023
29.80
37.77
5.78
26.39
52.70
40.48
Chat-Univi-13dD Jmn 2023
5.38
12.59
12.81
11.27
20.27
17.01
PLLaVA-7B Xu
2024
33.79
40.86
28.89
41.20
59.46
57.14
PLLaVA-
[2024]
39.43
29.90
39.20
58.11
63.61
General VL.
Interleaved
LLaVA-intericave/ al. (0224d
31.98
41.57
26.30
40.43
45.95
56.40
J.aVA-interleave-dpp al. 2024d
33.63
40.38
28.59
44.60
50.00
54.22
VILA-1.5-3B Lin et al. b
31.07
38.00
30.88
39.35
35.14
45.78
VILA-115-3B-- 32-33BB22111 2 Lin el
(2023b
25.64
35.63
30.76
33.02
40.54
45.78
VILA-115-8BB Lin
202.5b
35.29
37.05
18.58
52.16
51.35
41.42
VILA-115-13B3 Lin
20256
32.43
35.87
30.88
38.73
45.95
46.87
Phi-3V Abdin èt
2024
33.18
35.39
28.95
35.34
59.46
56.68
I. /hang al. 20244h
29.71
41.57
29.35
44.44
59.46
45.90
ILaVA-NV-dpal Zhang al.
2024b
29.41
39.43
30,92
44.44
54.05
46.45
Mantis-Idefics2 Jrang el al. (2024
28.59
38.43
41.89
Mantis-LLaVA Jrang
2024
27.75
40.14
28.11
40.74
43.24
Mantis-siglip-Iinm -lama3
Jiang et 2024
34.09
35.87
25.33
46.14
60.81
50.14
Mantis-clip- -laas-mlaplllm33 Jiang et al.
2024
36,95
41.33
22.92
45.06
$2.70
56.13
GPT-4V/Achian et al. 2023
35.29
25.18
35.22
41.20
70.27
81.20
Achiam et al.
2023
43.74
46.32
35.22
39.20
62.16
86.92
GPT-44-mini Achiam al.
2123
37.41
39.90
25.33
46.76
71.62
78.75
Gemini-1..5ffaas Team
2023
38.86
38.95
24.61
38.43
70.27
77.38
Gemini-1.5-pre Team 2023
34.24
39.43
27.62
39.81
68.92
80.93
Table 21: Evaluation Results for 39 VLMs in PhysBench Physics-based A Dynamics Sub-task (the
second last column of Table? 3)
58

---

Published as a conference paper at ICLR 2025
F.4 PHYSBENCH- VAL RESULTS
Size Format Property TRelationships
Scene 25.00
Dynamics 25.00
Avg
Choice
25.00
Image VLM
InstructS11PP.. IP Dat
2024
48
merge
40.54
47.62
45.95
44,44
44,69
InstructBLIPP.. -PP----- Dar et
2024
12B
48.65
52.38
43.24
49.21
48.60
InstruciBLIP-70 Dai 2024
7B
merge
28.57
30,95
36.36
35
33.14
InstuctBLLIP-1385
13B
merge
30.56
60.00
36.51
37.80
BLIP-2
2025c
12B
merge
51.35
52.38
37.84
46.03
46.93
LLaVA-T.5-7BF Liu 2023a
7B
merge
37.84
54.76
43.24
52.38
48.04
LLaVA-1.5-13BB et 20268
13B
merge
56.76
$2.38
29.73
42.86
45.25
LLaVAI.6-mistrall
20/4bb
7B
merge
35.14
39.68
37.43
LLaVAI.6-vicuma
7B
merge
59,46
42.86
37.84
49.21
47.49
Qwen-VL-Chat Bai
20233
9B
merge
40.54
45.24
40.54
44.44
43.02
IntemVL-Chati: Chen al
21724c
26B
62.16
61.90
54.05
57.14
58.66
Cambrian-8B Tong al 2024
8B
merge
8.11
21.43
21.62
28.57
21.23
Claude-3-opus Anthrupic 2024
merge
45.95
61.90
45.95
57.14
53.63%
Claude-3-xonne Anthropic 20024
merge
43.24
38.10
Claude-3-huiku Anthropic
merge 56.76
$9.52
35.14
49.21
50.28
Claude-3.5
Anthropic
2024
merge
$4.05
69.05
40.54
50.79
53.63
Video VLM
Vidco-LLaVA Lin ct al.
2028h
7B
soq
43.24
33,33
48.65
$2,38
45.25
Phat-Univi-BB/1 7B al 2025
7B
soq
24.32
28.57
29.73
33.35
29.61
Chat-Univi- 13B
2026
13B
seq
21.43
18.92
17.46
PLLaVA-7B Xu 2024
7B
soq
48.65
47.62
32.43
$3.97
46.93
PLLaVA-138 Xu 2022
13B
seq
51.35
50.00
40.54
58.73
$1.40
General VLM Interleaved data
LlaVA-interlevee Li al. (2024d
7B
seq
48.65
59.57
43.24
$1.90
$1.50
LLaVA-interlenve-ppp LTd all 2024d
7B
seq
40.54
48.94
48.65
$0.63
48.00
VIIAA-L.53BB Lin 20266
3B
seq
37.84
43.24
36.71
VILA--15-3B
2b
3B
seq
35,14
$1.06
43.24
36.71
41.00
VILA-1.5-8B Lin 20235
8B
24.32
29.79
43.24
36.71
34.00
VILA-I Lan 20255
13B
seq
32.43
$1.06
37.84
45.57
43.00
seq
51.35
LLaVA-NV Zhang el
2024b
7B
seq
45.95
40.43
32.43
46.15
42.21
LLaVA-NV VV- Sedpoouunggg
2024b
7B
48.65
41.86
29.73
46.15
42.56
Mantis-Idefies2 Jiang ct al
2024
8B
soq
40.54
46.81
43.24
45.57
44.50
Mantis-LLa Uiang al
7B
seq
43.24
45.95
35.44
Muntis-siglip-Iaame Jiang
2024
8B
seq
67.57
40.54
40.51
44.00
Mantis- Mantis-clip-llamaal 1are-llla33 Jiang
2024
8B
$4.05
42.55
35.14
44.30
44.00
GPT-4V Achiam al
2023
62.16
68.09
43.24
67.09
62.00
GPT-4o Achiam ef al
2023
72.97
74.47
54.05
62.03
65.50
GPT-4o-minAAchie Achiam et
2023
soq
64.86
57.45
45.95
63.29
59.00
Gemini-1.5-ffae Team
soq
64.86
54.05
63.29
Gemini-L.5-pro Team
2025
seg
59.46
70.21
56.76
68.35
65.00
Evaluation results for 39 vision-language models in PhysBench-val. Note that the evaluation of General VLMs is based on the data from Video and Image VL.M evaluations, with the addition of interleaved data, meaning that the full test dataset of PhysBench is being assessed. In this context, "seq" refers the sequential input of images after frame selection from videos, while "merge" refers merging video frames into single image.
Table 22:
59

---

Published as a conference paper at ICLR 2025
F.5 EMBODIED TASKS DETAILED DESCRIPTION
To further validate the effectiveness of our data and method, we built simulation platform us- ing MuJoCo (Todorov al. 2012) and the Franka Emika Panda from Menagerie (Zakka et al. 2022). and conducted tests on five embodied tasks. Using the proposed mark-based visual prompt- ing technique with GroundedSAM (Ren et al., 2024) and farthest point sampling (Qi et al. 2017). MOKA (Luu et al. 2024a) converts affordance reasoning into series of visual question-ane answering problems that pre-trained VLMs can solve. The setup matches MOKA, and the general visual setup can be seen in Figure (b)). Our tabletop environment only has one top-down camera, which is the primary camera used MOKA to capture RGBD images. For each task, we report the number of successes out of 10 trials following the setting of et al. (2024a).
(a)
(b)
(c)
(d)
(c)
(f)
(g)
Figure 25: (a) Overview of the simulation platform. (b) Top-down view of the area. (c) Affordance test: Testing whether the robotic arm correctly grasps the object. (d) Force test: Testing whether the robotic arm can properly grasp deformable, fragile, and rigid objects. (e) Color test: Testing
whether the robotic arm can pick up the correct colored object among identical ones. (f) Number
test: Testing whether the robotic arm can grasp a specific number of objects. (g) Tool test: Testing whether the robotic arm can select the correct tool given specific scenario.
We present the specific language instructions given to the VLM in Figure 26 is important to note that, for each task, we only provide single example. For instance, while we tested five objects in the Affordance task-pot, knife, spoon, monitor, and tennis racket- we only used the tennis racket as an example here. For each task, we report the success rate over 10 trials, following the MOKA protocol. "Specifically, the test content and evaluation methods for each task are as follows: (1) Affordance test: This test evaluates whether the robotic arm correctly grasp various objects. Figure (5(c) illustrates the robotic arm successfully grasping a pot. In total, we tested the grasping ability on 10 items, including pot, knife, spoon, spatula, monitor, tennis racket, phone, and others. Specifically, we tested 5 objects-pot, knife, spoon, monitor, and tennis racket-attempping to grasp each object twice. (2) Force Test: This test evaluates the robotic arm's capacity to properly grasp deformable, fragile, and rigid objects. Due to simulation constraints, the evaluation was based on the robotic arm's 'ms output metrics. We tested fragile items (e.g., egg. ripe persimmon), soft items (e.g.. jelly, plastic cup), and rigid objects (e.g.. iron ball), with two attempts per object. It should be noted that the simulation system models all objects as rigid bodies, meaning that breakage during grasping not depicted. Furthermore, while using the MOKA system to control the Panda robotic arm, we were unable to directly manipulate the gripper's size. Instead, we provided the VLM with approximate object dimensions, allowing the VLM to determine the necessary gripping force evaluate success or failure." (3) Color test: This test evaluates whether the robotic arm can accurately pick up the correctly colored object from set of identical items. As shown in Figure e))e), the blocks
identical except for their color, and the task requires selecting the object of the designated color.
We tested five colors blue, pink, brown, green, and orange-ccnducting two trials for each color.
60

---

Published as a conference paper at ICLR 2025
(4) Location test: This test evaluates whether the robotic locations. The goal is to ensure that the robotic arm accurately grasps the required objects based on their positions. Specifically, we tested with three blocks, requiring the arm grasp the middle block (4 attempts); the block farthest from the plate (3 attempts). and the block closest to the plate (3 attempts). (5) Tool test: This test assesses whether the robotic
correctly grasp objects specific
select the appropriate tool for 25j g). the task is: "If you need to cut a watermelon,
a given task. For instance, as shown in Figure[25je) which tool should you grasp?" In this scenario, the robotic arm is expected grasp the fruit knife. In total, we posed 5 questions, with 2 attempts per question, requiring the robotic arm to select and grasp different target tools for each task.
On the other hand, these five tasks require minimal consideration of height (or depth) information, making the evaluation more fundamental. For MOKA's waypoints selected from free space, their height must be explicitly specified for accurate deprojection into 3D space, they are not anchored to any objects. For this reason, typical tabletop manipulation scenarios, MOKA primarily focuses on cases where the waypoints at the same height as the target point.
Affordance
Grasp the tennis racket.
Force
Grasp the egg. Grasp the blue cube.
Color
Location
Grab the block farthest from the plate and move it to the plate.
Tool
Grasp the tools for cutting a watermelon.
Figure 26: The language description of the testing tasks.
We present an overview of our implementation of MOKA in Algorithm I Our experiments aim to enhance the VLM's ability to understand the physical world and validate its impact on downstream embodied agent tasks. Specifically, we employ two methods to improve the VLM: first, fine-tuning the VLM using PhysBench, and second, incorporating the PhysAgent assist during VLM infer- ence. It is worth noting that the five tasks we address are relatively fundamental, unlike the complex multi-action tasks described the MOKA paper, which require hierarchical decomposition from high- to lewelevel actions. In our case, the tasks can be executed directly without such decomposi- tion."
Algorithm MOKA Pipeline
Input: Vision-language Model M, Task instruction text prompt for low-level reasoning Plow and initial observation s
2: Get observation s from the top-down camera
3: Propose keypoint and waypoint candidates and get annotated image f(sz)
4: Query M for low-level motion reasoning, obtain ylow M([prows (s)))
5: Execute Yoow on the robot in simulation
F.6 CORRELATION MAP
Following the approach of Tong et al. (2024): Fang et al. (2024 2025). used the Pearson corre- lation coefficient to construct a relationship matrix. The data used to build this matrix can be found
in Table 23
VQAV
GQA
VisWiz
SQA
TextVQA
POPE
MME
MMBCN
SEED
SEEDI
MMMUsa
MMMU
V.Leh--ench
LLaVA-13-BB
78.5
61.0
50.0
06.5
58.2
55.9
1310.7
61.3
S8.3
61.3
67.0
33.2
31.1
63.4
LLaVA-1.5.13B
80.0
63.3
53.6
71.6
61.3
85.9
1531.3
67.7
63.6
62.4
68.2
36.4
33.6
70.7
InstmecBLL1PP-5
61.1
49.2
60.5
50.1
7888
1210.11
23.7
53.4
58.8
32.9
30.6
hesmecBLLP--338
62.3
49.5
31.4
63.1
50.7
78.9
1212.8
42.0
25.0
55.2
61.7
35.7
33.8
58.2
Qwen-VL-Chall
78.2
57.5
38.9
68.2
61.5
85.6
1487.5
60.6
56.7
58.2
65.4
35,9
32.9
64.1
VILA--.5.3B-42
79.8
61.4
61.3
69.6
63.4
85.3
1431.7
62.8
$2.2
60.0
66.4
32,8
31.3
76.7
VILA--15-8B
80.9
61.9
58.7
79.9
66.3
84.4
1577.0
72.3
66.2
64,2
71.4
36.9
36.0
80.0
VII.A-1.5-1BB
64.3
65.1
72.6
37.9
BLIP-2
41.0
44.6
29
61.0
42.1
85.3
1293.8
44.0
27.0
46.4
49.7
35.4
34.0
56.2
Table 23; The performance of the 10 models used construct the correlation map across 15 other
VLM benchmarks.
61

---

Published as a conference paper at ICLR 2025
The correlation presented in Figure 4a) illustrates the relationships between the four major cate- gories PhysBench and other tasks. Additionally, we provide detailed correlation map between PhysBench and 15 other vision vision-language benchmarks in Figure 27 below.
VOAV2
GGA
ValWzz
8QA
ToutaQA
POPE
MME
MMB
MMB ON
SEED
SEEDI
MMMMU-val
MMMMU
lave banch
MM-
PhyBench
VOAv
0.34
0.08
1.0
GQA
1.0
0.94
0.93
VisW
0.84
0.92
1.0
0.85
0.89
0.58
0.00
0.89
0.84
0.69
0.80
0.15
0.04
0.94
0.95
0.26
SOA
0.74
0.80
0.45
1.0
0.86
0.53
88
0.00
0.87
0.65
0.85
0.80
0.47
0.80
0.91
0.29
TexivQ
0.86
0.84
0.89
0.86
1.0
0.50
00.85
0.89
0.90
0.85
0.90
0.25
0.10
0.87
0.82
0.20
POPE
0.45
0.04
0.38
0.53
0.50
1.0
0.81
0.78
0.76
0.43
0.39
0.20
0.08
0.00
0.55
0.84
0.79
0.80
0.80
0.88
0.85
0.81
0.08
0.000
0.83
0.79
0.41
0.27
0.75
0.61
0.68
0.6
0.89
0.90
0.40
0.25
0.04
0.69
CN
0.84
0.87
0.778
SEED
0.00
SEEDI
0.85
MMMU-val
0.15
0.60
0.88
0.2
MMMU-lest
0.00
0.00
0.00
0.47
0.10
0.000
0.27
0.25
0..2
0.09
(0.08
0.00
1.0
0.11
0.13
0.22
lava bench
0.78
0.88
0.94
0.89
0.87
0.50
0.75
0.84
0.78
0.83
0.88
0.25
0.14
1.0
0.98
0.10
MM-Vet
0.85
0.08
0.96
0.91
0.82
0.50
0.81
0.89
0.85
0.09
0.89
0.34
0.13
0.000
1.0
0.2
0.0
PhyBench
0.20
0.41
0.28
0.29
0.20
0.84
0.69
0.57
0.57
0.23
0.17
0.33
0.22
0.10
0.22
1.0
Figure 27: Correlation map between PhysBench and 15 other vision-language benchmarks.
F.7 PERFORMANCE ON RELATED BENCHMARKS
To further evaluate the contribution of our data and model to understanding the physical world, we conducted tests on three existing benchmarks related to physical-world perception. Notably, these benchmarks focus on specific aspects of physical-world perception, whereas our PhysBench provides more comprehensive and holistic evaluation, as summarized in Table [ Furthermore, since these datasets were not originally designed for VLMs, applied necessary preprocessing to adapt them for our evaluations.
Setup.EmbbSerii (Du et al. 2024) is henchmark designed to evaluate spatial understanding within embodied cnvironments with source image come from MP3D (Chang et al.) 2017). ScanNet (Dai ct al. 2017) and AI2-THOR (Kolve al. 2017). We utilized its benchmark dataset for testing pur- poses. ContPhy (Zheng et al. 2024b) is benchmark aimed at assessing visual models' capabilities in perceiving continuous physical phenomena and properties. It comprises four simulation systems based on Unity3D: Fluid Hourglass, Rope-Pulley System, Cloth Magic Trick, and Ball Playground. Since ContPhy primarily targets visual and physical models, selected 200 items for each of the four categories (split evenly between property-basee and dynamics-based tasks) and translate them into multiple-choice format suitable for VLMs. The question prompts were modified better align with the answering capabilities of VLMs. Physion++ (Tung et al. 2023) evaluates the impact of physical properties such as mass, friction, elasticity, and deformability on physical phenomena. leverages the World simulation platform (Gan et al.) 2020) to generate series of videos, each paired with a corresponding question. The videos consist an inference phase, where arti- ficial systems can identify objects' mechanical properties, followed by prediction phase, where the model must predict whether two specified objects will collide after the video ends. Physion++ is primarily designed for vision models such ResNet (He al. 2016) and VGG (Simonyan & Zisserman, 2014). with answers in the form of binary classification (yes/no), derived by convert- ing model outputs into probabilities. To adapt Physion++ for VLMs, processed the videos and
62

---

Published as a conference paper at ICLR 2025
reformulated the questions into natural language prompts, adding necessary contextual hints. For example, if a video includes a transition phase, we explicitly include a statement like, "The black screen in the video marks transition in objects scenes" in the prompt. Due to the small size of the test set, we combined the train and test sets, resulting in 250 VQA pairs for evaluation. During fine-tuning with PhysBench, we modified the options to an open-ended format. The results, pre- sented in Table [24] are reported in terms of accuracy. All parameter settings and configurations are kept consistent with those outlined in the main text.
Table 24: Performance on Three Related Benchmarks.
ContPhy- ContPhy-Property
ContPhy- ConiPhy-Dynmiics
Physion++
EmbSpatial
Phi-3V
49.25
43.25
68.80
55,91
Phi-3V finetune
64.50
62.75
82.20
66.95
Phi-3V PhysAgent
52.00
44.75
78.40
62.28
Results. As presented in Table 24 leveraging PhysBench data for fine-tuning in zero-shot setting with PhysAgent leads to performance improvements across the benchmarks, particularly in Physion++, where improvements of 19.50% and 9.6% are observed, with fine-tuning achieving the most significant gains. These results highlight the effectiveness of our data and methods in enhancing the capability of Vision-Language Models to comprehend the physical world.
G MORE RELATED WORKS
Vision-Language Models. Vision-Language Models (VLMs) are large language models that in-
tegrate visual modalities, such images and videos, with language knowledge (Wu et al. 2023b
Huang et al. 2024 Bai et al. 2022 Zhan et al. 2024; Dai et al., 2024, Bai et al. 2024b). Notable models like BLIP-2 (Lict al. 2023c) and LLa VA (Liu 2024cc have advanced mmage-capiooning datasets and visual instruction tuning, with LLa VA-Nett further improving single-image perfor- mance at higher computational costs (Liu et al.) 2024b). Subsequent models, such as QWenVLL (Bai cet al. 2023b), Cog VLM (Wang et al.) 2023b). and Yi-VL (Al et al. 2024), have followed a similar architecture LLaVA. As single-image and text interaction technologies continue mature, many recent VLMs (Alayrac et al. 2022 Peng et al.) [023) Pan et al.) [024]] Lin et al. 2023b) are now ca- pable of handling complex visual tasks with interleaved images or videos, enabling VLMs tackle 2024b) and paving the way for interactions with
more sophisticated tasks (Lu et al. 2024b Yu et al.
the real physical world.
Vision-Languaa Benchmarks VLMs (Liu et al. 2024c Achiam et al. 2023 Pan et al. 2024 Yu
al. 2023aa] [how et al.) 2024) ?: Li al. 2024b c) have inherited and advanced many intriguing features from text-only LMs. Benchmarks for VLMs have rapidly emerged to evaluate performance
areas such as image question answering (Ying et al. 2024). in-context response Yuu et al. 2023b),
chart understanding (Li al.) 2024f), and web comprehension (Liu al. 2024d Zhou al. [0233]
Some benchmarks cover spatial relations understanding (Li et al. 2023a), but often overlook the ability to devise complex spatial action plans based on physical world comprehension. Recently, new benchmarks have also emerged that focus on the ability to understand multiple images in long
contexts (Zhang al. [2024a] Kil et al.) 2024 Jiang et al./ [2024] Wu al. [2023 Li et al. 2024e) Ge et al. 2024) and complex realistic environments (Fu et al.) 2024| Bai et al.) (024a)) However, based on answering questions from images, videos, tables, or using
these benchmarks-whene
visual prompts (Fu et al. 2024 Yu et al. 2024a) -ultimately rely on responding to the content of
the given images rather than the true perception of the physical world, thus falling short of achieving
spatial intelligence (Gupta et al. 2021 Yang et al. 2024a)..
Video Benchmarks. With the growing interest in video understanding, the development of bench- marks for VLMs has become increasingly emphasized. In video comprehension, the research com- munity has made significant strides, particularly for short videos. There are specialized bench- marks for temporal perception (Yu et al. 2019: Wu et al.] (2024a), action understanding (Liu et al.) Mangalam et al. 2024), video classification (Kay et al. (017). video reasoning (Xiao et al. a02 Xie et a 2023). video captioning (Miech et al. 2019); Xu et al. 2016), video question-
2024e
answering (Zhou et al.
2024 Li al, 2023g]2024c) long video comprehension (Wu et al. 2024b)
Chandrasegaran al.
2024 video generation (Bansal al. 2024), and interleaved video-text
63

---

Published as a conference paper at ICLR 2025
Table 25: Comparison between PhysBench and other vision-language benchmarks. the for- mat, I, T, V present text, image, and video. Annotated means annotate the existing dataset, like -een-tth--iidd------e Wild)-Detail Liu et al. 2024c). Reasoning means that it requires the VLMs reasoning ability to answer the question.
MSCOCO Karpathy & Fei-Fei (2015). LLaVA Wa;
Dataset
Size
Format
Interleaving
Souree
Dumain
Reasening
VOA
Antonee
Imsge
GOA
22,669,675
Antoaee
32,000
Antounel
45,000
Antarise
Imsge
14,000
Antunised
Imsge
SBBO
19,000
content Image
3000
1-T
Antorined
Image
MMF
1.877
Antarined
Image centere
18,000
Inage Iullucciaaionn detection
TUESS purahl
147
Image
ULVA Lingrall MtMi
60
1+1
Antotuned
Imaie chat
SQA All
1-1
NLVitasmitetn AME
6,867
1-1
1-1
Malt
BLINK qain
1-1
Martis
1:1
Wirero
2,50
1:1
Antonaed
MMML
11,500
1-1
Annotated Web. Textbokk
10,002
V-4-T
Simalaiion Real-woell
Poysical property
question-answering (Wang et al. 2024a). However, these works primarily focus evaluating video content and do not explore the underlying mechanisms of video representation or address true physi- cal world perception. Furthermore, during our experiments, observed significant challenges with current video VS in following instructions and answering questions, as the models frequently output descriptions of the video rather than directly addressing the posed questions.
Interleaved Vision-Langual Benchmarks. VLMs are increasingly processing longer and more complex inputs. Along with this development, several benchmarks with interleaved inputs have emerged Li et al. 2023b Wang et al., 2024b] Meng et al., 2024). For example, SEED-Bench (Li al. 2023a 2024a) focuses on video understanding, BLINK (Fu et al., (024) introduces visual prompts, and NLVR2 (Suhr et al. 2018) differentiates between two images. However, all of these benchmarks still primarily assess content description based on images, without evaluating physical understanding perception abilities. Furthermore, current interleaved benchmarks only involve im- ages and text, while PhysBench is a dataset that interweaves video, image, and text inputs. A detailed comparison with the previously mentioned benchmarks and other vision-language benchmarks can be found in Table 25
Science-related Benchmarks. In addition the benchmarks related to physical world comprehen- sion mentioned Section 2] there are also benchmarks that assess models' understanding through questions. SeiQ (Welbi et al. 2017). ScienceQA LLu et al., 2022), E-EVAL (Hou et al.) 2024). MMLU-STEM (Hendrycks et al., (020)) and C-Eval- STEM (Huang al. 2023b) include some physics-related questions, but these datasets often allow questions to be answered simply by analyzing the provided images, lacking the complexity of rea- soning and computational tasks. JEEBench (Arora et al. 2023) requires multistep reasoning with physics knowledge but is limited in scope and purely text- based. SciBench (Wang et al., 20247), OlympiadBench (He et al., 2024). Math Vista (Lu al., 2024b), and OCWCourses (Lewkowyez et al., 2022) provide college-level physics questions. However, these benchmarks mainly consist of homework and exam-style questions, focusing more on mathematical reasoning (Zheng et al.) 2024a) and general knowledge rather than true physical world comprehension. In contrast, PhysBench is the first systematic and comprehensive designed for understanding the real physical world.
middle university-level physics
benchmark specifically
Table 26: A comparison between PhysBench and other physical understanding benchmarks not in question-answering format.
Progerty
Amtrut
Locatios
Velocis
Tenpersuue
Cunera
Light
Calliãon
Munijulation
Fluid
Intellaaeed
Size
Physees oolWall
2010
X
17,408
In/hyy Risomeeaa Mur
X
0000
200
X
CRAFI Panserri minu
X
47,0000
OaPhy hemmno
PaysRinch
64

---

Published as a conference paper at ICLR 2025
3D Scence VQA. Recently, multi-modal 3D perception has garnered increasing attention due to its connection to the physical world, driving rapid advancements the field. SQA3D (Ma et al. 2023)
highlights the importance of "situations" within contextual understanding. EmbodiedScan (Wang
et al. 2024d) and Scene (Jia et al. (025) expand the scope by collecting more scenes or an- notating additional objects, scaling annotations to the millions. OpenEQA (Majumdar al. 2024
SpatialRGPT-Bench (Cheng et al., 2024). MMScan (Lyu et al.) 2024). and MSNN (Linghu et al. 2024) integrate comprchensive annotations, adapting the task into a VQA format. However, these studies primarily focus geometric relationships, which represent only subset of the broader understanding of the physical world, as discussed in Section Our work prioritizes a more holis- tic evaluation of the physical world perception capabilities of VLMs across four major task cate- gories: Physical Object Properties, Physical Object Relationships, Physical Scene Understanding, and Physics-based Dynamics. Additionally, the spatial components in our dataser differ signifi- cantly from those existing Spatial VQA datasets. While such benchmarks typically rely on 3D point cloud scenes interleaved 2D images from multiple viewpoints, our dataset uses interleaved images to capture physical world dynamics, such as viewpoint rotations and the progression of phys- ical phenomena.
VLMs for Robotic Manipulation. Recently, two main approaches have been proposed for apply- ing Vision-L Languuge Models (VLMs) (Liu et al.,
2024c (Achiam al., 2023 Team al. 2023
Mao et al. 2023a b) to robotic manipulation: {a) directly generating actions (Kmm et al. 2024
Octo
Model Team al. [224 Zawalski al. 2024a Niu al.
2024
Zawalski
(224bb and (b) em-
ploying VLMs as agents (Liu et al. [022aa] Nasiriany et al.
2024
Huang et al. 2023a). Approach (a)
involves directly outputting actions, which requires extensive training for instance, OpenVLA was trained using 64x A1x0 GPUs and produces embodied -erooiifddspccifec actions that necessitate targeted fine-tuning. contrast, approach (b) generates affordance representations through VLMs, which are subsequently converted into actions. This method requires less training and exhibits stronger generalization capabilities. However, suffers from weaker perception capabilities the physical world, leading to performance limitations (Liu et al. 2024a
Mao al. 2024).
H MORE EXAMPLES
We red color right answer. We red indicate the correct answer (correct answer). It is important to note that, to reduce difficulty and facilitate evaluation, we employed multiple-choice format rather than open-ended responses. For space-saving purposes, the example figures in the main text do not display the available options.
H. PHYSICAL OBJECT PROPERTY SUB- -ABS
Question
The color of the original spots on the chameleon's body has not changed during the process of changing its color. What color these spots?
Answer
A. Black B. Green C. White D. Blue
Figure 28: Example for property color. Ability Type is identify.
H.2 PHYSICAL OBJECT RELATIONSHIPS SUB-TASK
H.3 PHYSICAL SCENE UNDERSTANDING SUB-TASK
H.4
PHYSICS-BASHR DYNAMICS SUB-TASE
65

---

Published as a conference paper at ICLR 2025
Question
The weight of the object about the size of ping-pont ball is closest to which of the options?
Answer
A. 5g
B. 50g
C. 500g
D. 5kg
Question
Which fruit the picture is most likely be the heaviest individually?
Answer
A. Watermelon
B. Pear
C. Plum
D. Peach
Question
What can said about mass of brown cube compared to white sphere?
Answer
A. The brown cube is less massive B. The white sphere is less massive
They have the
D. Cannot
Figure 29; Three examples related to property mass, categorized by the following ability types: identification, comparison, and comparison.
66

---

Published as a conference paper at ICLR 2025
Question
How many blueberries there outside the plate?
A nswer
A. 1
B.
C.3 D.4
Question
How many of the books pictured have covers with titles beginning with P?
Answer
A. B.2 C.3 D.4
Question
Following the content the vvideo How many eggs broken?
Answer
A. B.2 C.3 D.4
Question
S 6666
é
éééé
How many balls moving close together without separating?
Answer
A. B. C,4 D.6
Figure 30: Four examples related to property number, ability types are all identification.
67

---

Published as a conference paper at ICLR 2025
Question
the photo, which point with option signifies the object with most sharp?
Answer
A. Point A
B. Point B
C. Point C
D. Point D
Question
Can you tell me which point with option the image points to object that has least
brittle?
Answer
Point
Point
C. Point C
D. Point
Question
Which point with option in the photograph pinpoints the object with the most stiff?
Answer
Point
Point
C. Point C
D. Point
Question
Which point with option in the image marks the object that exhibits the most elastic?
Answer
Point
B. Point B
C. Point C
D. Point D
Question
Which point with option the photograph captures object that has most mallcable?
Answer
A. Point A
B. Point B
C. Point C
D. Point D
Question
Which point with option the image isolates object most soft?
Answer
A. Point A
B. Point B
C. Point C
D. Point D
Figure 31: Six examples of property attributes include sharpness, brittleness, stiffness, clasticity, malleability, and softness.
68

---

Published as a conference paper at ICLR 2025
Question
Moving according to the arrow in the picture, which of the following options are you
most likely to encounter?
Answer
B.
D.
Question
Following the direction indicated by the in the picture, which of the following
options are you most likely encounter?
Answer
A.
B.
D.
MA
Question
What is the
relationship between the speeds of the two objects in the video after they begin to fall?
Answer
Both objects have the same speed,
The egg consistently falls faster than the feather.
C. The feather consistently falls faster than the egg-
D. Initially, the falls faster, but the feather eventually surpasses it.
Figure 32: Three examples for relationships motion, categorized by the following ability types: static, static and dynamic.
Question
Determine which point is nearest the camera:
Answer
A. Point A is nearest
B. Point is nearest C. Point C is nearest
D. Point D is nearest
Figure 33: An example for relationships depth. Ability Type is static.
69

---

Published as a conference paper at ICLR 2025
Question
Find point closest to river:
Answer
A. Point the closest
B. Point B is the closest
C. Point C is the closest
D. Point D the closest
Question
What value distance between each of three small bowls placed side by side
closest to?
Answer
A.3mm
B.3em
C.10cm
D.20cm
Question
What is farthest from the key in the picture?
Answer
Pen
B. Camera
Watch
D. Glasses
Figure 34: Three examples for relationships distance. Ability Type is static, static and dynamic.
Question
What is happening to the size of the ice cube in the video?
Answer
Increasing
B. Decreasing
C. No change
D. Increasing first and then decreasing
Question
Which object is the biggest in volume?
Answer
A. red cube
B. purple cube
C. green cube
D. orange cube
Figure 35; Two examples for relationships size. Ability Type is dynamic and statie,
70

---

Published as a conference paper at ICLR 2025
Question
Among these points, which one is not on the same sunflower as the other points?
Answer
Point
Point
C. Point
D. Point
Question
What the object above the egg?
Answer
Milk
Yogurt
Banana
Egg
Question
(
How does moon pass through sun video?
Answer
A. From the lower left corner to the upper right corner of the picture From the upper left corner to the lower right corner of the picture From the lower right corner the upper left. corner of the picture
From upper right corner to lower left corner of the picture
Question
Which point with the option the image
is corresponds the reference point the
How does moon pass through sun video?
Answer
A. Point A
B. Point B
C. Point C
D. Point D
Figure 36: Four examples for relationships location. Ability Type is static, static, dynamic and static.
71

---

Published as a conference paper at ICLR 2025
Question
Where the light source in picture?
Answer
A. In the clouds the upper right of the screen lower left corner the screen little above the center the screen
D. In the exact center of the screen
Question
Reflecting the events in the video, which of the following alterations to the light source is most likely
result the phenomenon observed?
Answer
A. The color of the light changes from yellow to pink
B. It's just that the light source weaker and the light source position remains the same
C. Move parallel the line between the drumstick and the ballet skirt
D. It's just that the light source stronger and the light source position remains the
Question
From the events the video, which of the listed changes the light source is most likely have resulted the observed phenomenon?
Answer
A. Move parallel to the line between the postcard and the cake
The color of the light changes from orange to blue The color of the light changes from lime yellow to green
The light source moves downward
Question
Taking account phenomena observed in the video, which the following changes light source most likely to have to this result?
Answer
It's just that the light source is stronger and the light position remains the same
It's just that the light source weaker and the light position remains the same
The color of light changes from red to purple
D. The color of the light changes from yellow to blue
Figure 37: Four examples illustrating environmental lighting conditions. The corresponding
ability types are perception, reasoning, reasoning, and reasoning.
72

---

Published as a conference paper at ICLR 2025
Question
How does the brightness in the picture change?
Answer
First gets brighter, then gets darker
First gets darker, then it gets brighter
C. keeps getting darker
D. keeps changing
Figure 38: Examples for scene environmental lighting conditions (Continued). Ability Type is judgement.
Question
What happens the gas pressure inside the bottle before open it?
Answer
A. Increases
B. Decreases
C. Stays the same
D. Varies randomly
Figure 39: Example for scene environmental air conditions and the ability type of it is perception.
Question
What temperature range is typically used for the phenomenon shown in the viden?
Answer
A. Under 100"C B. Approximately 200
C. Nearly 400
D. Over 600CC
Question
W What is the possible cause of the phenomenon in the vidco?
Answer
Increased humidity
Increased temperature
C. Decreased temperature
D. Decreased humidity
Figure 40: Two examples for scene temperature conditions and the ability types of them are all perception.
73

---

Published as a conference paper at ICLR 2025
Question
beginning video, how does the camera's focal length change?
Answer
A. Focus length remains unchanged
B. Focus length increases
Focus length decreases Unknown
Question
Based on the phenomenon in the video, which of the following camera changes can produce the effect in the video?
Answer
A. Move parallel to the line between the motor and the wash basin
B. Move parallel the line between the bunk bed and the wash basin
The camera
upward downward The camera rotates along horizontal axis (left right)
Question
From the video, which of these camera changes could be responsible for the depicted phenomenon?
Answer
A. The camera is farther away from the objects
B. Move parallel to the line between the cupcake and the sponge
The camera is closer the objects
Move parallel the line between the cupcake and blackboard
Question
Based the phenomenon in the video, which of the following changes can produce the effect the video?
Answer
The camera rotates along vertical axis (upside or downside).
The camera is closer to the objects
C. The camera rotates along the horizontal axis (left or right).
D. The camera moves upward or downward
Figure 41: Four examples illustrating scene viewpoint conditions. The corresponding ability types are perception, reasoning, reasoning and reasoning.
74

---

Published as a conference paper at ICLR 2025
Question
Select the option that shows the correct procedure to put on pair of gloves.
image #1:
image #2:
image
Answer
A. A-1-3-2
3-3-2-1 1-1-2-3
C.2-1-3
Question
Which of the following options lists the steps in the correct sequence to put the carrot in the microwave?
image #1:
image #2:
image #3:
image #4:
Answer
2- 3-433
B. B.1-3-2-4
C-2-3-1-4
--2-4-3-1
Question
To poke the watering can, which point most suitable?
Answer
A.
B.
C.
D.
Question
In order to pick up the cup, which of following color points reasonable affordance?
Answer
A.
B. C.
D.
Question
What operation used to transform object from Image
Image
Answer
A. Remove the small ball from the clay block.
Add another hall the clay block.
Air-dry the clay block.
D. Press the small ball into the clay block.
Five examples illustrating dynamics manipulation. The corresponding ability types judgment, judgment, perception, perception and reasoning.
Figure 42:
75

---

Published as a conference paper at ICLR 2025
Question
Following content the video, which option's corresponding picture will happen first?
Answer
A.
B.
C.
D.
Figure 43: Example for dynamics collision and the ability type of it is prediction.
Question
Why are the phenomena happens in the video?
Answer
Because they are lighter than air.
Due to the presence ferric oxide the match heads.
C. Because they are coated with special glue.
D. Due static electricity on the magnet.
Figure 44: Example for dynamics chemistry and the ability type of perception.
Question
What will happen the white ball?
Answer
A. It will drop into the right pit. B. It will not drop into the right pit.
C. Its movement cannot be determined. D. It will drop into the left pit.
Figure 45: Example for dynamics throwing and the ability type is prediction.
76

---

Published as a conference paper at ICLR 2025
Question
Which bottle contains the fastest diffusion rate?
Answer
A. The one on left
B. The one the middle
The one on the right Both are equally
Question
We already know that the liquid poured in is water, so what is the original yellow liquid?
Answer
A. Orange juice
B. Oil
C. Beer
D. Urine
Question
M
Which color object looks the most viscous the video?
Answer
A. Transparent liquid
B. Light yellow color liquid
C. Thick yellow liquid
D. Dark blue liquid
Question
already know that the red particles the picture are liquid particles, which
area of the picture the liquid flow fastest?
Answer
A. All parts of the image at the same speed. the leftmost part of the picture the middle part of the picture
D. the rightmost part of the picture
Figure 46: Four examples illustrating dynamics fluid. The corresponding ability types are percep- tion, reasoning, perception and perception.
77

---

Published as a conference paper at ICLR 2025
Question
Among the listed choices, which one outlines the proper sequence of events in candle burning?
image #1:
image #2:
image #3:
Answer
A. 2-2-3-
B. 1-3
1
D-3-1-2
Question
Which of the following options presents the correct order of occurrences in fruit rotting?
image #1:
image #2:
image #3:
Answer
- B.2-3-1
C.1-3-2
D.1-2-3
Question
The light first passes through the convex lens and then the concave lens. Slide the
concave lens close to the convex lens. Which of the following options will correspond to the phenomenon in the picture?
Answer
B.
C.
D.
Figure 47: Three examples illustrating dynamics others. The corresponding ability types are rea- soning, reasoning and prediction.
78

---

Published as a conference paper at ICLR 2025
1 ERROR STUDY
I.1 DETAILED STATICS
In this section, we present case study analysis of the error types made by GPT-4V, Gemini- 1.5- flash, and Phi-3V across various tasks. The errors are classified into the following five categories. Other less frequent error types are not included this analysis. For the analysis, as mentioned in Section 3.4 we selected 500 samples for each model, but due space limitations, we present only 60 of them here, as shown in Table 27
Perception Error VLMs fail to recognize, classify, or detect the objects or content in images. VLMs are constrained by the representation power visual encoders, and these errors account for the majority of them. See examples in Figure 48) Figure 50 etc.
Reasoning Error VLMs can recognize the text and visual content exactly but make errors in reasoning, leading to incorrect results. See examples in Figure 65] Figure 8T etc.
Lack of Knowledge
VLMs do have specific knowledge, so they finally get a wrong answer.
See examples in Figure 57 Figure 71
Refuse to Answer
VLMs refuse to answer questions and stop answering immediately. See ex-
amples Figure 54
Fail to Follow Instruction VLMs fail to correctly understand instructions and provide erroneous answers. For example, VLMs may not understand the specified conditions in the instruction (see Figure 74).
1.2 MAIN REASON ANALYSIS
Perceptual errors can be classified into two categories: basic perceptual errors and comainn-specife perceptual errors. Basic perceptual errors occur when the model successfully understands the given information but fails to interpret fundamental visual objects correctly. In contrast, domain-specifer perceptual errors arise when the model misinterprets visual inputs due to lack of understanding of specialized conditions. Moreover, GPT-4V often exhibits bias towards textual information, prioritizing text over visual inputs-a a trend highlighted recent studies Cui et al. (2023). A notable example is shown in Figure 68 where the model mistakenly identified the plate as part the background, leading to incorrect depth estimation. This underscores the importance of seeking balanced approach to enhance the model's interpretative capabilities.
Perceptual Errors:
Sometimes, even when the model correctly interprets the text, images, and question, it fails establish rigorous logical chain, leading incorrect conclusions. These types Figures 635 and [8 In the first example,
Reasoning Errors:
of mistakes, known as reasoning errors, are illustrated the model mistakenly assumes that friction can only exist if the machine rotates faster than the object, overlooking the presence of another object, which results in an incorrect answer. In the second example, while the model correctly identifies that the temperature of the wrapped paper is significantly lower than that of the other object, it incorrectly attributes this phenomenon to the flame's 's temperature, leading to an erroneous conclusion.
Lack of Knowledge: Another major cause of errors is the model's lack of relevant knowledge. A notable example is shown in Figure [1] where the model incorrectly assumes that light bending in water is due to refraction. The model clearly does not understand the concept of total internal reflection water, leading to an incorrect conclusion.
The remaining errors, such as those related to textual understanding, refusal to answer, annotation mistakes, and answer extraction issues, account for only a small proportion. However, they remain significant and should not be overlooked.
Other Errors:
I.3 CASE STUDY
79

---

Published as a conference paper at ICLR 2025
Table 27: Table index of case study figures by meta-task with associated enror categories.
Case Figure
Meetn-taok
Subtask
GPT-4a
Gemini-I.5 1.5-flash
Phi-3V
Figure
48
Scene
Light
Perception Error
Perception Error
Perception Error
Figure
49
Dynamics
Collision
Reasoning Error
Perception Error
Reasoning Error
Figure
50
Dynamics
Throwing
Perception Error
Perception Error
Perception Error
Figure
Light
Perception
Perception
Perception
Figure
52
Scene
Viewpoint
Perception Krror
Reasoning Error
Reasoning Error
Figure
53
Scone
Viewpoint
Perception Error
Reasoning Error
Success
Figure
34
Dynamics
Chemistry
Reasoning Error
Refuse Answer
Success
Figure
Dynamics
Manipulation
Perception Error
Perception Error
Perception Error
Figure
56
Dynamies
Manipulation
Perception Error
Perception Error
Perception Error
Figure
57
Property
Attribute
Perception Error
Perception Error
Reasoning Error
Figure
58
Property
Color
Perception Error
Reasoning Error
Perception Error
Figure
Preperty
Mass
Reasouing Error
Success
Perception Error
Figure 60
Property
Mass
Success
Success
Success
Figure 6
Property
Number
Perception Error
Perception Error
Refuse Answer
Figune
Property
Number
Perception Error
Perception Error
Success
Property
Attribute
Lack of Knowledge
Lack of Knowledge
Perception Error
Figure
Relationships
Perception
Perception
Rensoning
Figure
65
Relationships
Motion
Reasoning Error
Reasoning Error
Rensoning Error
Figure
66
Relationships
Depth
Reasoning Frror
Reasoning Error
Success
Relationships
Depth
Success
Success
Success
Figure
Relationships
Perception
Ressoning
Figure
69
Relationships
Distance
Reasoning Error
Reasoning Error
Lack of Knowledge
Figure
70
Relationships
Size
Perception Errur
Reasoning Errur
Reasoning Error
Figure
71
Dynamics
Others
Lack of Knowledge
Lack of Knowledge
Lack of Knowledge
Figure
Relationships
Success
Success
Figure
73
Scene
Viewpoint
Perception Errur
Perception Error
Success
Figure
74
Relationships
Location
Perception Error
Perception Error
Fail to follow instruction
Figure
75
Relationships
Location
Perception Error
Percettion Error
Rensoning Error
Figure
Vicwpoint
Perception Error
Perception Error
Success
Figure
Seene
Temperature
Perception Error
Perception Error
Perception Error
Figure
78
Scene
Temperature
Reasoning Error
Reasoning Error
Reasoning Error
Figure
79
Dynamics
Air
Perception Error
Perception Error
Perception Error
Figure
Dynamics
Air
Reasoning Error
Reasoning
Reasoning
Figure
8
Dynamics
Manipulation
Reasoning Error
Reasoning Error
Reasoning Error
Figure
82
Relationships
Depth
Reasoning Error
Perception Error
Perception Error
Figure
33
Dynamics
Fluid
Perception Error
Perception Error
Perception Error
Figure
Dynamies
Others
Perception Error
Perception Error
Perception Error
Figure
Dynamies
Perception Error
Perception Error
Perception Error
Figure
86
Dynamies
Collision
Perception Error
Perception Error
Perception Error
Figure
87
Scene
Light
Reasoning Error
Perception Error
Success
Figure
Viewpoint
Perception Error
Perception Error
Perception Error
Figure 8V
Relationships
Location
Success
Perception Error
Perception Error
Figure
00
Property
Attribute
Perception Error
Reasoning Error
Perception Error
Figure
0
Dynamics
Manipulation
Perception Error
Success
Perception Error
92
Scene
Viewpoint
Perception Error
Perception Error
Perception Error
Figure
Light
Perception
Perception
Perception
Figure
94
Relationships
Location
Perception Error
Perception Error
Perception Error
Figure
95
Relationships
Location
Perception Frror
Perception Error
Perception Error
96
Dynamies
Others
Reasoning Error
Reasoning Error
Rensoning Error
Figure
Viewpoint
Perception
Reasoning
Success
Figure
9R
Scene
Light
Reasoning Error
Reasoning Error
Rensoning Error
Figure
99
Dynamics
Manipulation
Perception Error
Success)
Reasoning Error
Figure
100
Dynamics
Collision
Perception Error
Perception Error
Success
Figure
Property
Auribute
Perception
Perception
Reasoning
Figure
102
Scene
Light
Success
Reasoning Errur
Success
Figure
103
Scene
Light
Perception Error
Perception Error
Perception Error
Figure
104
Scene
Light
Perception Errur
Perception Error
Reasoning Error
Figure
Light
Success
Perception
Reasoning
Figure
106
Scene
Light
Reasoning Error
Perception Error
Reasoning Error
Figure
107
Secne
Light
Perception Error
Perception Error
Perception Error
80

---

Published as a conference paper at ICLR 2025
Scene: Light
Question: According the video, which alteration the light source is most likely to have triggered the phenomenon? <video
Option:
(A) The color of the light changes from yellow cyan
(B) color the light changes from green to
(C) It's just that the light source weaker and the light source position remains the same
(D) It's just that the light source is stronger and the light source position remains the same
<video
The answer It's that the light
weaker and the position remains the
Harel the ressening:
shadows:
by objects significuntly longer
part
compared the beginning.
2. Relnte shadows to light source: Longer and less defined shadows are indieative of weaker light souree.
their positions drange. This the altention most likely due to the light source itsel,, rather than any changes in the objects being illuminated.
Consider
The
the vidao
Eliminate other options:
Channinn color the light
necessurily lenger defined shadows. A stronger light source (D) would resul: shorter and shurper shudows.
Therefore, logical explanation
weakened, longer less defined shadows.
Choor yoe tust the light souner wakker the tight euzce pourcce remunn sone
Obsenaize of Imaace
The orter impee curs cociover some with foll, hedpped: and wen other atgert on worden cariice
The andono and lightnu cn ln shrecs aanan vonentor latts dirccdos and shape Boogglous das actic inagos.
Andyis Light
inlloaing
inceslty oft
Eirnentee of Coter Chauge Osticne
-piion (Yeliow to Cyraj change light oolor fiver yellow to zye would maat is outioaate ovin cokor rows ofjicts clodoos aut steerved
Optoon
Steilaeyy shonge grezn
cgatficeely,
images
Ultrsaatce of
Sineghering:
Option Di Sturge Light Soureci the light uaree storgee, the inngos would progeniddy Jocoma: trighter. This will staered
progresia drloning aggen
Therrise the moe stary ahorstos to scave the riggend phenomeaos icttor the light scero ieuusker while pocitios serabe on
The phenomenon in the image is the casting of shadows on the wooden floor, which suggests that there light source above the objects. The shadows consistent in shape size, indicating
light source not moving. The most likely alteration the light source that would trigger this phenomenon Therefore, the correct answer is C. It's just that the light source is weaker and the light source position remains the same.
change the light source's intensity.
Figure 48: sample case of physical scene understanding-ligt Back to List of Figures
81

---

Published as a conference paper at ICLR 2025
Dynamics: Collision
Question: What phenomena is result of the explosion?<video 1>
Option:
(A) Diffraction of waves
(B) Interference of waves
(C) Refraction of waves
Reflection of waves
video 1>
The answer is D. Reflection of waves.
Here's the rensoning:
What is a shockwave? shockwaaee is a type of pressure wave that is created by sudden, intense release energy, as explosion.
supersonic speeds. 3. What huppens when a shockwave hils surface? When shuckwave encounlers surface, is reflecied back.
How do shockwaaes travel? Shockwaves travel outward from the
What
video?
visible shockwave emanating explosion.
The shockwave appears bend as it travels across the landscape, which is evidence of reflection.
Therefore, the phenomena seen video reflection shockwave.
Clossr Refeasan of vosaa
Explamtion:
Coveraation of Inager:
The vatior of sleno explaies vecaring feld
The raplainu areesio a lape usoout of sanke and fios kilonol by deskknoeer popeadiis though tess..
Dilirciion
obotado
small
evidawe of savcs bending ansnal dloaells.
Inarkrence or Wasas The occar when tvo more wares evetap and oentine sim non vane putson. The images co met shor any oxntirgping wane
peitens.
dhongo
thyy pas
images show ary
Jullatioe WanD: The whsa bounez bakk ofler biting aoerfirc The imsaco sbow stadovara outvea fram eupkaien and thes wrlleting bact atae lariing goond or other ourtion.
parsoatetyy image you
geneeeed explotice,
sype
tmough
The sludovere liss toe goond and other verfico, it teffeee leck This cridkt foxe (he ciedle palkena und may slackion
bounce teck atter tring the graad.
the attictton dedovsee in dear inbiatice reflccion voxue gianeraroe.
Theerier docke Reflectio
The image shows a large explosion with a significant amount of smoke and dehris being ejected into the air. The smoke is billowing upwards and outwards in all directions, creating cloud-like effect. This visual effect is due the diffraction of waves, which is the bending of waves around obstacles openings. When an explosion occurs, the produced by environment, such as the ground, air, and any obstacles their path. This bending and spreading of the shock waves result in the cloud-like appearance of the smoke and debris. Therefore, the correct answer is A. Diffraction of waves.
can bend spread out as they encounter surrounding
Figure 49: A sample case of physics-bast dynamics collision. Back to List of Figures
82

---

Published as a conference paper at ICLR 2025
Dynamics: Throwing
Question: If we removed the green floating wall and other balls, which pit would the black ball drop into?<video 1>
Option:
(A) The left pit
(B) The right pit
None of the above
(D) Both pits
<video 1>
The answer A. The left pit.
Here the reasoning:
The black ball is sitting on the second platform from the left. black ball.
There white to right
green floating wall the only thing preventing black ball falling the pit. 4. Removing the green wall would remove the obstacle between the black ball and the left pit.
5. Therefore, the black ball would fall into the left pit.
Cholec: B. Theright pit
Reasoning:
Inital Position and Movement block
top positiored above the pit. green floating currently preventing fiom falling directly right pit
Removing the Green Flstting Wall:
romoved, The black ball will straight down due to gravit..
green
longer have its path.
the Black Hall:
black The black ball pusitioned direclly above the right pit.
Without green floating
directly downward.
Conclusion:
Since the black ball directly above the right pat and there are no other obstacles its path, it will fall into the right pit.
Therefore, bicck hall drop the if green floating wall and balls
To determine where the black ball would drop into, need analyze the image and the position of the pits. The black ball located above the left pit and below the right Since the pits are visible from the front, cannot empty. However, based on the position of the black ball, more likely to drop into the pit the left side of the image. Therefore, the answer is A. The black ball would drop into the left pit.
there is barrier
pits are
Figure 50: sample case of physics- based dynamics
throwing. Back to List of Figures
83

---

Published as a conference paper at ICLR 2025
Scene: Light
Question: Observing the phenomena the video, which of the following alterations to the light source most likely to produce such cffect?<video
Option:
(A) It's just that the light source is weaker and the light source position remains the same
(B) It's just that the light souree is stronger and the light source position remains the
(C) The color the light changes from limeyellow blue
(D) The color of the light changes from red to pink
<video
unswer
source posiiion
Herc's the rensoning:
The objects the video cast elongnted shadows, indicating that the light source pasitioned at low angle. The shadows are also relatively faint and indistint, suggesting the light source not strong.
Observation:
Eliminaiio:
B. Stronger light source: stronger light source would produce sharper, more defised shadows. The shadows in the vidzo
blury strong light
Coke changes: While coke changes can affect the pereeived uppearmcce of objects, they wouldn't directly cause the elongated and faint shadows observed the vidco.
The musi likely explmation that the light source is weaker. weaker light source produces intense light. resulting sotter, more bluredd shadows. The low angle of the light source would still contribute the clongated shadow shapos.
Conclusion:
just light strunger light source position
Detailed Explanation:
Observation Shadows:
In the series of images, the shadows of the objects on the ground clearly visible and consistent in shape and position. indicuting position light
shadows change shape,
Intensity of Shadows
The shadows repear well-defineed If the light source were weaker, the shadows woukd be less defined and lighter in appearance.
sugposting light source strung casting shadows..
Color Light:
The images do not show any noticeable change the color of the light. The ground objects maintain a cunsistent color throughour the series images.
the color light had changed (options D), ground.
the objects the strunger light source position
visible shift
Conclusion:
Sines shadows are well-deffned and ihere no change in the color the light, most likely alteration that light source has hecome aroeger while position remains the same.
The most likely explanation for the observed effect is that the light light source position remains same. This is because weaker light source would result in less intense shadows, making them appear more diffuse and less defined. Additionally, if the position of the light source remains the same, the shadows would still be cast the they would be less pronounced
is weaker and the
reduced intensity the light. This would result in the shadows appearing more blurred and less distinct, as seen in the video. Therefore, the correct answer A, It's just that the light source is weaker and the light source position remains the same.
same direction,
Figure 51: A sample case of physical scene understanding light. Back to List of Figures
84

---

Published as a conference paper at ICLR 2025
Scene: Camera
Question: Which of the following camera modifications can explain the effect seen in the video?=vidco 1>
Option:
(A) Move parallel the line between the pudding and blanket
(B) Move parallel to line between the cube and the blanket
(C) The camera moves upward or downward
(D) The camera is farther away from the objects
<video 1>
The the wauwing
dteds is video apear maning ovay
pilko pakling sidik, ds
posperiic
2 Canon Mevnune The mon thety reptaaaionn de the penpactiv duage is tha corirs maving speant or downand Anthe canera eoves speant
de oéjica lolon a uppar shrink sawe open. seaiing the flosks of paiin fatha ovey.
rubatve Dl Mose pantiat so the tae betwees the cato and the taaket Simbir to optio A, this ticmevmmen wouden case ebject appar mnalee
More pualled
padding statee: Mostaa pantlid
wouldd
D. The conore father way foom the oljors Modiag the caner father away world make eijers appcr aniler ter woldee orear the reletiv
clangis
Thexsion, the by stanging the mabie airrca styies
rocifieniza eaphis the cllct
dovavari. This cranes de ilaik depti und parpustre
Choice Mone pasiieel to the Tee betweer deceee and Marker
Explaniio:
Diommstion
Tae easer insao slow oxnaicat change in adazie poulisen Ex déjods (kke Rabakls enbs is the pluc und yellow objozi wit ropant to the tndannaad (he Huild and pellonel.
The
yoloo abpect appar mose hottorally scase tmages while egg plss and boxke rozan relotvvlly statonary.
Andou
The Bankks ycllas olçed covvnn in dieeties dur wagess kationaal skill
The Healet and jillows, which ar the badkgoend, slow vigiifunt veried sovementt indirtin fel the canne see reaving speand devenand.
Desmering th Divertos of Canera Moverant:
Sioc the libels nihe and yellow object we moxing hortnondlly the conore mus muving pasllsl the The oor connec theor ofjicis to Harket TTa berioend rasvrnnt digs wish dhe dradion belnoes das cabe the Halks, sagpiin dei casas is mociag pazlicl daa lnc.
pootist hea betwere
and tte This wontd
ciffomee
eoseeeer, continee
Insoonal ail chavee.
- -peon Cillee vares mevc upeaal doveredd) Thi world soud vattal mosemnnt of oljoro ocuttve bodkgoond. viidi no. observll
-ppoee The comers tuther awyy thee the egencor world cas change wiee otgect, conatior hetoumal
Therefee dieke Move
Maket
The effect seen the video, where the Rubik's Cube appears to be floating mid-air, be explained by the camera moving upward or downward. This is because camera's movement creates perspective that makes the cube appear to be floating, as if it is not supported by any surface. The other options do nol provide the same visual effect. The camera moving parallel to the line between the pudding and the blanket would not create the illusion of floating. and the camera being farther away from the objects would not change the perceived position the cube. Therefore, the correct choice C. The moves upward or downward.
Figure 52: A sample case of physical scene understanding--veexponn.
Back to List of Figures
85

---

Published as a conference paper at ICLR 2025
Scene: Camera
In light of the effects observed in the video, which of the following camera adjustments might achieve the depicted outcome?<video
Question:
Option:
(A) The camera closer the objects
(B) Move parallel to line hetween the cereal and skiboot
(C) Move parallel to line between the cereal cymbal
(D) The camera is farther away from the objects
<video
The anower et The crnara tutter away then the atyers
saplyytng resorng:
Camader camers porticeing change in propeatis parorrty wchoved mosing the oaeea clour futter sayy frmm the some.
optone
unies stjests
apos: cloe tegalen, oganie
ditance betweon
Mose paolled to line betwen the coodl and the synbal Sintler oppies B. ttis would adjes the engle bet th sole the diject.
Condaderr only messiing option, D. conciteer aherrod offirt Moving canero faster awa from objects woeld dhen appoor sundier fatbo ranching vides
gyntul
Durnitdd txplrative:
Ubwenztin on of Movement:
ln seins of fregee, the oljects oppea toskif posiiio rélitive offer. Syeaificllly, the ccxell box, is sysild socrs to daage dekk relative postuen mere egudiiunly compused other starzs
The cneal bos fie oyebid are padicnal opponitc vido of the frans. insao caead box from de keit side fame: tnsodd the right while the genhal movee then the right vide ofe ftom treande laf
Clmerreziy Camera Mexmment
iegios cunase birawing sbyact.
Eadl daangs fhe prilions of fleoysiid.
Elinisaiig Other Optocs
-Irrcce A/ canera close the sticral: Thes would soal in the obiecs araari lage bus would not accoertt chaqge ther rebatve postonn wax ahorned
Oprice booo
betwees and Thi prinnaly the relative positiu lo sccect, apening
cyriitoe change
abarmed
Crnchmore
the plambbl explration the eteerve change relatve postion otrede caners mont pusticl betwoer
sydbal
grobal apeer poótiees
imagss
achieve the depicted outeome,, where the objects appear larger and the background appears smaller, the camera needs be closer to the objects. This because the perspective the image is such that the objects are in the foveground and the objects appear larger
background the decrease in distance, and the backeground will appear smaller due to the incresse distacce. heredoore, the conrect choice
creating a depth. moving
This choice based the principles of penspective and depth of field in photography, where objects cluser to the camera larger, and objects further appear smaller. The other options D) with the desired outcome of making objects oppear lurgrr and background smaller.
Figure 53: A sample case of physical scene weapestaanniin-iivoooii
Back List of Figures
86

---

Published as a conference paper at ICLR 2025
Dynamics: Chemistry
Question: What action displayed in video?-video
Option:
(A) Candle being blown out deliberately
(B) Candle causing wax to overflow
(C) Candle burning with double wick
(D) Candle being used in romantic setting
<video 1>
The answer is C. Candle burning with double wick.
Here's the reasoning:
1. Visual Cues: The video shows tall, cylindrical candle with honeycomb pattern. The flame the top suggests the candle burning.
2. Flame Observation: The flame appears to be unusually tall and wide, indicating a larger- than-normal amount of fuel being consumed.
3. Double Wick Inference: This suggests that the candle
Candle causing overflow
Detailed Explanation:
Burning: series images shows candle burning over time. flame comsistent and does show signs being blown out deliberately (which would typically show sudden change in the flame's behavior).
Observation
noticeable overflow This indicatees that the candle has been burning for while, causing the melt and overflow.
Overflow: In the
images,
left side of the candle.
Double Wick: The images de not show any indication of double wick. The flame is singular and centered, typical for single-wick candle.
4. No Romantic Setting: The background is solid red color with no additioral elements that would suggest romuntic setting (such us dim lighting, additional candles, or romantic devor).
Based on these observations, the correct choice is B. Candle causing wax to overflow.
The image shows single candle with a flame lup, set againat a red background with repeated logus of NATURAL CANDLE The repeated logos suggest that the candle is product being advertised, and the presence of the flame indicates that the candle is currenily burning. The action displayed in the video is not explicitly shown, but based on the context intended purpose, which could be for lightnng ambiance, decoration. most likely action candle being used in romantic setting, candles often associated with romance and intirnate seltings. Therefore, the choice is:
THHE NATURAL
CANDLES'
1969'
image,
reusonable to infer that the candle is being used, likely for its
Candle being usod romantic setting
Figure 54: A sample case of physics-based dynamics
chemistry. Back to List of Figures
87

---

Published as a conference paper at ICLR 2025
Dynamics: Manipulation
Among the listed choices, which one outlines the proper sequence to close the open cabinet doore I><imaaee 2><cimage 3>
Question:
Option: (A) 3-
(B) 3 2-11-11
(C) 2-3
(D) 3
<image 1>
<image 2>
<image 3>
The answer is C, 2-
Here's the reasoning:
Image The robotic fully extended and reaches the cabinet door.
Image 2: The arm starts to close the cabinet door.
Image 3: eabinet door fully closed and the robotic amn retracts back to original position.
Choice: C. 2-
Reasoning Chnin:
Image #1: The cabinet door fully open. This starting point where door its open position.
2. Image #2:
Thee cabinet door partially closed. This indicates that the dooe is the process of being closed but not yet reached fully closed position.
3. Image #3; The cabinet doar fully closed. This is the final position where the door completely shut.
Step-by-Step Explanation:
-Step 1: Start with the cabinet door fully open (Image #1).
-Step
partially closed position (Image #2).
SSepp
Finally, close the doer completely (Image
Thus, the proper sequence to close the open cabinet duor is 1 2-
To determine the proper sequenee to close the open cubinct door, let's analyze the images:
image #1 cabinel door open, image shows the cabinet door closed, image 1/3 shows cabinet door open again.
Based on information,, the correct sequence to close the open cabinet door would be:
the cabinet door (image
Close the cabinct door (image #2)
Open the cabinct door (image #3)
So, A. 1-3-2.
Figure 55: A sample case physics-based dynamics manipulation.
Back to List of Figures
88

---

Published as a conference paper at ICLR 2025
Dynamics: Manipulation
Question: Which of provided options demonstrates accurate steps close the microwave?-viddo 1>
Option:
(A) Remove the blue floating wall and other balls
(B) Remove the green floating wall and other balls
(C) None of the above
(D) Remove both the blue green floating walls
<image 1>
<image 2>
<image 3>
The answer is B. - 3.
The robot arm first moves towards the microwave door (image #1), then presses the door down close it (image #2), and finally, it moves back original position (image #3).
Choice: 1 3-2
Reasoning Chain:
1. Image #1:
The robot arm is approaching the microwave door, which is open. This is the initial step where the robot preparing close the door.
2. Image #3:
The robot arm is now in contact with the microwave door, starting to push it closed. This is the intermediate step where the door is being moved towards the closed position.
3. Image #2:
robot arm is no longer contact This the final step where the door has been successfully closed.
The microwave door fully closed,
Therefore, the correct sequence of steps to close the microwave is 1 3-3-2 2.
The image shows a kitchen with a microwave oven and refrigerator. The correct sequence close the microwave would to first ensure that the door is open, then to close the door, and finally to secure it with the latch. Therefore, the correct sequence is:
1. Open the microwave door. Close the microwave door.
3. Close the latch.
The correct option 1-22--2 33
Figure 56: A sample case of physics- based dynamics- throwing. Back to List of Figures
89

---

Published as a conference paper at ICLR 2025
Property: Attribute
How does the flexibility of the left cloth compare the other cloth?<vidco 1>
Question:
Option:
(A) is flexible
is equally flexible
(C) is harder to bend
(D) is softer
<vidco 1>
flexible
Ileres the reusoming::
plaid. 2. Cloch Movement: Boch cloths are draped over the objects on the sable. The brown cloch drapes in more fluid and uneven manner.
3. Fleability Inference: suggests
fluid and drape brown cloth indicases
bend confoem the shape
objects
4, Comparison The phii cloth, on other hand, drapes in more rigid and uniform manner, implying fluxibiity.
Therefore, based the the cloths infer that the brown cloth is flexible than plaid cluth.
flexible
Step stsp explination:
Intial Observation: first images,
cloths pluced sable with objects underneath
orange, right
2. Cloth Placement: in the thurd image. both clochs are being placed on the objects, The lest cloth (orange) and the right cloth
(checkered) both pinned the
Cloth Behavior: As the clochs are draped over the objeces, we can observe their behavior: In the fourdh image, the left cluth (orange) starts show more pronounced draping and bending around the object compured the right clohh (chockure))
image,
(arange)
flexbbiity conforming more closely shape
object undemmath it.
In the south and seventh images, the left cloth torange) appears more fluid and ticuble, showing more folds and curves compured right cloth (cheokered), which maintains
structured and less deeuble appearance.
Final Observatio: By the last image, the left clots (orango) has clearly shown more flcubilty its ability to drape and confomn to the shape of the object underneath while the right cloth (cheokered) appears stifter and loss flexbble.
these
(orange)
compared right (checkered)
compared to the image and look for any differences the way the cloths are positioned or how they appear to move. The left cloth positioned on the left side of the table, while the other cloth is on the right side. We can observe that the left cloth is not as stretched out as the other cloth, which is more extended towards the right side of the table. This suggests that the left cloth is less flexible than the other cloth. Therefore, the choice is less flexible.
determine flexibility of
cloth, we
analyze
Figure 57: sample case of physical object property-attribute Back to List of Figures
90

---

Published as a conference paper at ICLR 2025
Property: Color
Question: What color does the chameleon turn from blue in the video?<viden 1>
Option:
(A) Red
(B) Grey
(C) Purple
(D) Orange
Syideo 1>
The answer Red.
Here's the explanation:
1. The chameleon starts out vibrant blue color.
The chamelcon then begins change colors.
It transitions to grey color.
4. As the chameleon continues to change, becomes reddish-brown color.
5. It stays that color for the rest of the video.
Thus, the chameleon from blue red.
A. Red
Detailed Explanation:
Initial Color Observation: The first image the chameleon green color. Color Transition: As we move through the series of images, the chameleon's color changes from green to yellow, then to blue.
The fourth and fifth images show the chameleon blue color, color in the sixth image.
Blue Color Observation:
Post-Blue Color Transition:
After the blue color, the chameleon changes brownish
5. Final Color Observation: The last three images show the chameleon in reddish color.
Therefore, the chameleon turns from blue red in the video.
The chameleon in the video turns from blue green. This is evident from the visual information provided the image,
the chameleon shown in different stages changing color. Initially, it blue, and as moves along the branch, gradually changes to green. This color change is a common characteristic of chamelcons, which they use for various purposes such camouflage, communication, and temperature regulation. The chameleon's ability change its color due specialized cells in its skin called chromatophores, which contain pigments that can expand or contract to alter color the chameloon's skin. The process is controlled by the chameleon's nervous system, which responds to changes in light, temperature, and other environmental factors, In this case, the chameleon is likely changing its color blend in with its surroundings to regulate body temperature. Therefore, correct answer green.
Figure 58: sample case of physical object property-color Back to List of Figures
91

---

Published as a conference paper at ICLR 2025
Property: Mass
Question: By proceeding in the direction shown by the arrow the picture, which of the following options are you most likely to come across?<image 1>
Option:
(A) <image 2>
(B) <image 3>
<image
(D) <image
<image 1>
<image 2>
image 3>
iimaee 4>
<imaee $>
The anow points to the right, indicating the path the viewer should take. that the right.
*Looking
imsage shows a buikling right street.
This building the same building shown image D, indicating that D is what the viewer would see by pooceeding the
direction of
Remoning
hridd Irage Analyck:
irmae donn with night indicated by amon.
arosing Soegroed
Dicction of Tiand: The sol sno indiates right han Toon the curcest pestion.
Comperiso sor with Oytions
Oznece
with anace ligborie hralding. This does out crvivement sstaa imnas more roiieniial
right Option Stawe coltiaation tosiltad wad, sistla isiial amage but due slow rige
tiily encommerdd
Optice Storss haalding with patcing ara and has sup, which doos aut munc toe immedies camendinge inmal imagge
Cocluioo:
git image bla gate Opton mast bllee dui turing ridls you will urea shown Optin
To determine the must likely option to come acroes by proceding in the dinection shown by the rod arrow, we meed to amalyaa the image and the aptions provided.
1. The red anow is pointing to the right, indicating direction move foeward. with
lnnes, There are trces, a building with sign rends "CSNGRESS, and conscruction siie with crmee in background.
image
polestrians
4. The option are B, and C, but the fourth option is not visible the image.
Based the visible informsion, we can eliminate option C, not visible in image.
Now, compane
Option A: The image shows a street with multiple lanes, cars, and pedestrians. There is sign thit reads "CONGRESS: " which the red might place Congress.
suggestt direction
Option B: The image shows a street with mahtipe lanes, cars, and pedestrians. There construction site with cranes the might
backgrourd diecction
constuction
Considering the context and the visible elements is the image, the most likely option came acroes by proceeding it the option leads roluted Congress.
direction shown by
Figure 59: sample case of physical object property
mass. property-mass.
Back to List of Figures
92

---

Published as a conference paper at ICLR 2025
Property: Mass
Question: Which fruit in the picture is most likely to be the heaviest individually<<imagee
Option:
(A) Watermelon
(B) Pear
(C) Plum
(D) Peach
<video 1>
Waternelon
Here's the reasoning:
Visual Size: The watermelon noticeably larger than all the other fruits in the picture. This suggests it has greater volume.
Density: While don't know exact densities fruits, watermelons are known be mostly water, making them rolatively dense.
3. Combined: larger volume combined with bigher density leads to greater overall mass, which translates to heavier weight.
Therefore,
the watermelon heaviest fruit the picture.
Watermelon
Reasoning:
Visuul Inspection: By looking al image, watermelon appears significantly larger in size compared fruits.
Typical Weight: Generally, waternelors known he much heavier than pears, plums, and penches. Even small watermelon can weigh severnl pounds, whereas the other fruits typically weigh much less. Density and Volume: Watermelons have high water content and large volume, contributing to their weight. Pears, plums, peaches, less individually.
containing water, much
and thus weigh
Therefore, based on size and typical weight characteristics, the watermelon most likely to be the heaviest individually picture.
To determine which fruit is most likely to be the heaviest individually, we need to consider the average weight each type of fruit.
A. Watermelon: typical watermeton weigh between 20 to 30 pounds.
B. Pear: pear can weigh between 0.25 to 0.5 pounds.
Plumz A plum can weigh between 0.1 to 0.2 pounds.
Peach: peach weigh between
pounds.
Based on the average weights, the watermelon is most likely to be the heaviest fruit individually.
Figure 60: sample case of physical object property-mass. Back to List of Figures
93

---

Published as a conference paper at ICLR 2025
Property: Number
Question: Following the content of the <video>, How many egg yolks in the bowl?<video 1>
Option:
(A)
(B) 2
(C) 3
(D)4
<vidco 1>
The answer D.
Here's the reasoning:
counting: The starts with egg yolks bowl.
2. Adding more:
The person then cracks open and adds 2 more eggs.
3. Total count: Adding the 2 new yolks to the initial 2 gives total of 4 yolks.
Detailed Explanation:
First Image: The is empty.
Second Image: A hand holding an egg over the bowl.
3. Third Image: The egg is being cracked open, and one egg yolk visible in the bowl.
4. Fourth Image: Another egg is being cracked open, and now there two egg yolks in the bowl.
Fifth Image:
egg is being cracked open, and the yolk is about drop into
bowl.
6. Sixth Image:
The third yolk is halfway into the bowl.
Seventh Image: The third egg yolk almost completely in the bowl.
Eighth Image: third egg yolk fully bowl, making total egg yolks.
Therefore, the total number of egg yolks put in the bowl is 3.
The image shows a clear glass bowl with textured surface, containing a liquid that uppears to be mixture of water and egg yolks. The bowl is placed on surface that looks like a white table countertop. There image not provide any information about the number egg yolks were put bowl. Therefore, it not possible to determine the exact number of egg yolks based on the image alone. The correct answer to the question cannot be determined from the image provided.
visible eggs any other objects in the bowl, and the
Figure 61: A sample case of physical object property-number. Back to List of Figures.
94

---

Published as a conference paper at ICLR 2025
Property: Number
Question: How many deer are image 1>
Option:
(A) 2
(B) 4
(C)
(D) 8
<video
The answer is
There are four deer that are fully visible, and two are partially visible. So there are deer.
gc.
Step-by-step explanation:
Observation:
Look closely the image identify the number of deer present. Start from one side of image and count each deer individually. the side, there are two deer visible.
Counting:
- MMvvnng towards the center, there are two more deer.
On the right side, there are two additional deer.
3. Total: Add the number deer counted in each section.
22 (left) 2 (center) (right) deer total.
Therefore, the correct choice is C. 6.
To determine number of deer in image, we need to count each individual deer visible in the scene. Starting from the left side of the image and moving towards the right, we can the following deer:
deer with large antlers grazing on grass.
2. deer with smaller antlers grazing on the grass.
3. deer with large antlers grazing on the grass.
4. deer with smaller antlers grazing the grass.
deer with large antlers grazing on the grass. deer with smaller antlers grazing on the grass.
7. deer with large antlers grazing on the grass.
8. A deer with smaller antlers grazing the grass. deer with large antlers grazing the grass. deer with smaller antlers grazing on grass.
There are total of 10 deer in the image. Therefore, the correct answer is: D. 8
Figure 62: A sample case of physical object property-number. Back to List of Figures
95

---

Published as a conference paper at ICLR 2025
Property: Attribute
Question: Can you tell which point with option in the image points to the object that the least britle?*image
Option:
(A) Point
(B) Point
(C) Point
(D) Point D
<viee 1>
The answer is B. Here's why:
Brittle means easily broken or shatterod.
clothespin, which
enough
B camera lens, made of strong glass and metal, it's designed to withstand impacts and pressure.
4. C is ceramic plate, which brittle and casily broken.
5. D is ceramic mag, also brittlc and casily broken.
Therefore, object pointed by has brittle
Choice: Point
Reassming Chain:
Understanding Britleenes:: Britllness is moterial property that deseribes the tandency of's materia to fracturo or break
without significnt deformation when subjected strees
Analyzing Objects:
Point (Clothespins): Clothsspis ippically made of wood
Both materials be somewhat briills, especially
they
subjecsed besding
Point (Camera Lasj:: Camera lenaes are made of glass and metal. Glass very brimle and cas shatter easily upon impact.
Point (Pltec): Plates usually made of cemmiic glass, both which are britlle materials that break casily if d'dppppe.. mug plastic,
[Mugl: Mugs are
coramic,
less beittle compared to the other materials listed.
Determining
Object:
Among the moterials listed, plastic generally the least brittle
Since mug (Point could made plastic, likely be least britle object the
Therefore, the object at Point D (the mug) the lewt britte among the options proviled
which object
typical usage object.
A ruperelip made of metal and designed holi logethur. is typicully britkk; but bend break subjected
cameni lems made of glass and designod to Reeus light onto sansor. Glass is brittle material and can easily bresk dropped subjected impact.
C. plate made ceramic poseelain and designed hold food. Ceramic and porcelain britke materials break dropped subjected to inpoct.
mode cermiic porcelai is designed hold liguids. Ceraiic and porcelain brittle materials and dropped subjected impact.
Based on this amilsis, the object with leas briike materiaa the puperclip (Option
Figure 63: A sample case of physical object property Back to List of Figures
96

---

Published as a conference paper at ICLR 2025
Relationships: Movement
Question: In the direction pointed by the arrow in the picture, which of the following options are you most likely find?< imagel>
Option:
(A) <image 2>
<image 3>
<image 4>
(D) <image S>
<image 1>
<image 2>
<image 3>
<image 4>
<image $>
The snswer
road. Looking original the road is sidewalk with a few troes building with glass windows, This cunsisentt with imago in choice
The picture points towards the
right side
The other choices
match the description the read the right of
Choice shows the road with the construcion burners und building the distance. does not match description of the
rightt side of the road.
Choice shows close-sp
barrir
walking sidewalt. mutch descrigtion
right
Choice D shows nuid with cans porked on the side. It does not mutch the description of the right side of road.
likely
pointed
Cloce
Isiial linge Azabyii
The lenaal maae chess mad with rottle tomier on the right vido tang heilding oin dettccho awnkanging emarae ne
The move poino staiga closl aloaa toai
dadintive
suffe
The peopodive and digansn of real and haries match diediie inllaaa mone the inkital image
Option Avalysis
Uption
par buildina focurg with troa.
Optiar Auabyii
Optias C aanoo cleccep of pesca walking nou ko copitratios baric.
The irage the nor parrice clerr contraation mad distactive bealding cue the image.
beikdings sides The eevinennn and lyye do maah sicee ietell image
Disona diffecet
walydss Option the lilayy to Sondl diverter poineed by armo is lebtal Image
analyze image step step:
1. The image shows city steee with multiple lines of fffe, including and track.
buildings 3. A person backpuck walking un the sidowalk, there are construcionn crines visible in distance
tres
buckground.
4. The sky clear, and the weather appears to be sunny.
5. There visible
markers indicating specific direction destination.
Rased on the image, it not possitie to detenmine a specifie location or destitation Therefore, the moot licly oprion is:
determine
Figure 64: A sample case of physical object relationships-motiee
Back to List of Figures
97

---

Published as a conference paper at ICLR 2025
Relationships: Movement
Question: Compared the metal object, what is likely the relative speed of the machine's rotation?< video
Option:
(A) The machine rotates slower than the object
(B) The machine rotates at the speed the object
(C) The machine rotates faster than object
(D) The machine does not rotate at
<video 1>
The answer is The machine rotates faster than the object.
Here's the reasoning:
1. Obscrvation:
The motal object being heated by friction.
surfaces against 3. Speed and Friction: The faster the relative speed berween surfaces, the friction generated.
generated
4. Appearance: The heat being generated is enough to melt portion of the metal. This suggests significant
amount of friction.
Conclusion:
object being by friction, friction significant, the muchine rotating faster than the metal object generate this amount friction.
he machise rotates flstier than the object
Datailed Explataiio::
Observation of Images: The series of images shows metal object being processed by machine. The metal object appears rotating, and there noticeable change red
of objoci time, purtiularlly the central where
2. Hear Géneration: The central area the metal object becomes red-bot, indicating that there signifloant friction or hest being ponented
high-sped rotation
snother surfice.
3. Rebrive Motion: For the motal object to best up in this minner, the machine must be rotating st higher speed thom the object liseff, This di ferorrial speal canses friction which generates hest.
4. Visual Cues: he images show the metal object becoming progresively hotter and more deformed in the central area. This suggests that the machine is upplying significant amount of energy to the object, which consisent with the machine rotating faster then the object.
3. Conclusion: Rased on the visual evidance ofte generation and the deformation the metal objecs, reasonable the nocessory friction and hean.
conolude that the machine robsing fiater tham the object
Therefore, the correct choice The machine rviates faster than object.
To determine the relative speed of the machine's rolation compared to the metal object, we need to analyze image. The motal object appears be stationary or moving very slowly, as there is clear motion blur.
the motion of blur indication of rapid movement. The machine. the other hand,
suggesting rotating higher speed. This is because motion blur is typically caused by rapid movement, which in this case is the rotation of the machine. Therefore, the machine's rotation is likely faster metal object's movement. correct choice
machine rotates
object.
Figure 65: sample case of physical object elationships-motiine motion.
Back to List of Figures
98

---

Published as a conference paper at ICLR 2025
Relationships: Depth
Question: In terms of depth relative to the camera, where is the bridge primarily
located?< video 1
Option:
(A) Closer to the camera than the car
(B) Equidistant from the car and the camera
Farther from the than the
(D) Cannot be determined
<video 1=
The answer is C, Farther from the camera than the car.
Here's reasoning:
Identify the key elements:: We the bridge surface.
the bridge and the bridge itself, with the camera positiomed
Focus on perspective: car appears closer to camera because it is the foreground. The bridge extends beyond car towards the borizon.
Apply the concept perspective: In perspective, objects farther away appear smaller and less detailed.
The bridge appears much larger in scale than the car, suggesting it is farther away from the camern. Conclude: Based on the visual cues of size and distance within the frame, we can conclude that the bridge farther from the camera than the car.
C. Farther from the camera than the car
Detailed Explanation:
1. Observation of Images: The series of images shows bridge with car on it. The appears to be in the middle of the bridge. and the bridge structure extends both in front of and behind the car.
Relative Positioning: In the images, the is positioned the bridge, and the bridge extends further the background beyond the car. This indicates that the bridge continues the making the bridge structure farther from the camera than the car.
and cables visible the background. indicating the bridge exlends into the distance. The is closer the camera than these structures.
Depth Perception:
The bridge's
Conclusion: Since the bridge extends beyond the cur and the car positioned on the bridge, the bridge is primarily located farther from the camera than the car.
Therefore, the correct choice is C. Farther from the camera than the car.
To determine the depth of the bridge relative the perspective from which it is taken. The bridge appears to be in the foreground, as it is in sharp focus and there is clear view of the cables and the road surfice. The car, on the other hand, is in the background, as it and appears smaller due to the perspective. clear view, while the bridge is closer Therefore, the bridge primarily
need to analyze the image and the
camera bocause
cables the road surface located closer to the cameri than the
Choice: A. Closer to the camera than the ear
Figure 66: sample case of physical object relationships-dephh, Back to List Figures
99

---

Published as a conference paper at ICLR 2025
Relationships: Depth
Question: Identify the closest point the camera. image 1>
Option:
(A) Point A is closest
(B) Point closest
(C) Point C closest
(D) Point D is closest
<image 1=
The answer is D. Here's why:
Perspective:
concept perspective deternine distance. Objects further away appear
smaller photograph.
2. Object Size: The circles the same size. allowing to directly compare their apparent size in the image.
3. Comparison: Circle D appears larger than the other circles, indicating that is closer to the camera.
Therefore, Point D the closest
D. Point is closest
Reasoning:
Foreground and Background Analysis:
Point D is located on the road, which is in the immediate foreground of the image.
-Points A, B, and C are situnted further back in the field, which in the background.
2. Perspective Depth:
The road in the foreground appears larger and more detailed, indicating closer to the
The field and the fènce posts in the background appear smaller and less detailed, indicating they are further away.
3. Relative Positioning:
Point D is positioned the bottom of the image, which typically represents the closest point the viewer in two-dimensional perspective.
Points A, B, and C are higher up in the image, indicating they are further away from careen..
Based these observations, Point D the closest
sohve this task, we need to analyze the image and determine which point closest to the cumera based on the perspettvve and the size of the points relative to ench other.
Point is located the top the image.
2. Point located the left of the image.
located
ofe image.
4. Point located the bettom the image.
penpoctivve
appesr larger and more detailed,
diat are further away will uppear smaller and less detailed.
the image, Peint appears be largest denailed, while Points oppear smaller and less detailed. Thersfore, Point cleesi point to the cumera.
The Poinr closest
Figure 67: A sample case of physical object relationships-depth. Back to List of Figures
100

---

Published as a conference paper at ICLR 2025
Relationships: Distance
Question: Which object appcars closest to camera in vidoo?evideo
Option:
(A) The metal plate
(B) The string
(C) The yellow ball
(D) The person's hand
vidco 1>
The is C.The yellow ball.
Here is the reasoning chain:
1. The yellow ball is the only object that is dircctly attached the string and swings the video.
2. The string is attached to fixed point above the metal plate. Therefore,
yellow ball closest
camera
the because swinging
of the plate and the person's hand.
4. While the person's hand is closer to the careera at the stari, the yellow ball moves to the very front in the second half of the video.
Therefore, the yellow ball appears closest to the camera in the vidco.
C. The yellow ball
Step-by-stey explanation:
1. Observation of Images: By examining the series imuges, we can see the relative positions of the objects each frume.
2. Depth Perception: Depth perception in images can be inferred by noting which objects overlap others and which objects appear larger smaller to perspective.
3. Relative Positioning:
The metal plate (A) is a large, flat object that serves us background for the other ubjects.
string (R)
appears
scene, attached the top and holding
yellow ball.
The yellow ball (C) is held by the string and appears to be in front of the metal plate.
The person's hand (D) interacting with the yellow and is positioned behind in most frames.
4. Overlapping Objects:
The yellow ball overlaps the metal plate and the string, indicating front of them.
The person's hand behind the yellow most frames, indicating the ball is closer to the camera.
5. Conclusion: Based un the overlapping and relative positioning. the yellow ball (C) appears closest to the camera in the vidco.
To determino which object appears closest to the camera in the video, we need to analyze the image and identify the abject that is most in focus and has the smallest depth of field. The metal plate (A) is in the The string
also in focus but slighly behind the metal plate. The yellow ball (C) in focus but further away than the metal plate. The person's hand (D) focus but even further away than the yellow ball. Therefore, the metal plate appears closest camera.
foroground and sharp focus, indicating
object the
Figure 68: sample case of physical object relationships distanee Back List of Figures
101

---

Published as a conference paper at ICLR 2025
Relationships: Distancc
Question:
What is the expected consequence of doubling the size of the sail?<vidoo
1>
Option:
(A) The cart move backward more quickly
(B) The cart will move forward rapidly
(C) The sail will turn more prominently
(D) No significant change
video 1>
The answer The cart will move foeward rpidly.
Here's
Newtouss Third Law: For every setion, there as equal and opposite reaction.
The Action: The thn pushes air backward
Resetion:
forard.
Salt: larger
ax, increasing
as pushang the cast
Therefire, doubling the size ofe sail will ineresse the force pushing the cart forward, consing move more mpidly.
Eqiandier
Setap:
witl
popelar partizzd how sonads the
Tweesuetng the
popelir
parking
fevard
Roa oftee sue
posticeed
meppetirr
ar pasked by popeller
conting uppoiii dicciinn.
het foree Acalyee
The tase areeaae the penntlrr padting ar bacovond comnemaee by the fine of asting the
Thoso from we inemel sysee torn, papellor, and will and eneel oads other
Deehhing oftte woeld innoau the
fun from perpillr
Hewoeer disw the Soves we vemmal and amart each other out incwaiing the oil cos doe changy n'e out anerul fine iie
deshiig
sigutiiane change
Thartoo,
Noviguttion
To determine expected consequence of doubling the size the sail, need consider the principles physics that govern the operation of sail-powered device. The sail's size is directly related to the area it presents the wind, which in tumn affects the amount force exerted on cart. Doubling the size the sail would speed and direction remain constant, This incressed force would result leading to more rapid movement. Therefore, the most probable outcome of doubling the size of the sail would
the area,
likely increase force exeried
eart, assurning the wind acceleration the
B. The cart will move forward rapidly
This because larger sail would capture wind, translating faster movement.
and thus
Figure 69: A sample of physical object relationships-distance Back to List of Figures
102

---

Published as a conference paper at ICLR 2025
Relationships: Size
How does the size the ping-pong ball compare to the mouth of the hot water bottle?<video 1>
Question:
Option:
(A) ping-pong ball larger
(B) The ping-pong ball is smaller
(C) The ping-pong ball is of similar size
(D) It cannot be determined
<video
ping
pong pong
smaller.
Here's the reasoning:
Observation:
The vidco shows person placing ping-pong ball top mouth of the bettle.
2. Inference: The fact that the ping-pong pong ball can be placed the bottle means it is smaller than the botile's opening.
Conclusion:
The ping-pong ball is smaller mouth hot water bottle.
C. The ping-pong ball is of similar size Step-by-step explanation:
The person is holding water bottle and yellow ping-pong ball. The cap of the bottle has been removed, and the mouth of the bottle visible.
1. Image I:
Image The person is positioning the ping-pong ball near mouth the bottle. preparing to place on top.
3. Image 3: The ping-pong ball is placed on the table, and the bottle is still in the person's hand.
4. Image 4: The person is holding the bottle again, preparing place the fing-pong ball mouth
bottle.
Image 5:
The person is placing the ping-pong ball on mouth the bottle.
6. Image 6: The ping-pong ball now on the mouth of the bottle, and the person is tilting the bottle.
Image 7: The bottle is tilted further, and the ping-pong ball remains the mouth of the
bottle.
Image 8: The bottle is fully tilted, and ping-pong hall is on the mouth of bottle.
From these observations, it is clear that the ping-pone ball fits snugly on the mouth of the bottle without falling in or being too large stay top. This indicates that the size of the ping-pong ball is very similar the size of the mouth of the bottle. Therefore, the correct choice C. The ping-pong is of similar size.
To determine the size comparison berween the ping-pong ball and the mouth of the hot water bottle, need to analyze the image. The ping pong ball appears to be smaller size compared to the mouth of the hot water bottle. This because the mouth of the hot water bottle wide enough to hold the ping-pone pong ball. indicating the botile's mouth larger than ball. Therefore, the correct choice The ping-pony ball is smaller.
Figure 70: A sample case of physical object relationships size. size. Back to List of Figures
103

---

Published as a conference paper at ICLR 2025
Dynamics: Others
Question: What type wave property is predominantly being demonstrated in video?<vidco 1>
Option:
(A) Difiraction
(B) Interference
(C) Refraction
(D) Reflection
<video
insweris C. Refraction.
Hend' why:
Refraction? Redraction banding posses water the bucket). This bending oocurs bucause the speod light chingos as il muves frum one metiium another.
case,
Observing Video: laser clearly changing dircction the brokkt. bends dowowards. This direct visual indication refraction.
Eliminating other Options:
Diffiaction: Diffinction is the spreading of light waves as they pess through narrow opening or around an obstzol:. There's no clear otamppe this the video.
partens. The video show Rellection: Reflection is the bouncing of light off suriace. While there some reflection off the side the bucket, the peimary phenoesenon the bending of the light enters the which refraction.
Interferenee: Interference
overlep, resuitingg
waves inteructig this mamner.
Refraction
The series of images peedominantly demonstrates the property of refiaction. Here the detailed reasoning:
images, d'light traveling through medium (ikeey water) banding uxiis the medium. This bending of light is key indicator of refiection.
Observation Light
Medium Change: Refraction occurs the images. the light insilly traeling through transpueent medium (possibly water) and then exits into the air, causing the light to bend.
paeses
changes speed, cousing
Consistent Bending: consistent handing light path the images aligns behavior light undergaing refraction. The light beam is not spreading out (difEnacion), nor showing portems of consiructive or descructvve interference (interference), also bouncing back from surface (reflection).
Contestual Clues: The setup, including the container and the light boam, suggests an experiment desigsed to show how light behaves when traniiioning betwest difffrentt modin, which classic demonstration of refircttion.
Therefore, the predominunt wave property being demonstrated in the video rufraction.
The video predominantly demonstrates the property of refraction, Refraction the bending of light as passes from one medium another, objects in the image. The light entering the glass angle, causing change direction, which is classic example of refraction. This is common phenomenon that occurs when light passes through transparent materials like glass, where the speed of light changes, causing the light to bend. The other options, such as diffiratioon, interference, video. Diffraction involves the specading light waves around obstacles through small openings. interference is the phenomenon where two more waves superimpose form resultant wave of grenter, lower, or the same amplitude, and reflection is the bouncing back light from a surface, which is not the main focus of this video. Therefore, the correct answer C. Refraction.
see as red light bending around edges
reflection, do not primarily describe observed effect in the
Figure 71: sample case of physics-based dynamics others. Back to List of Figures
104

---

Published as a conference paper at ICLR 2025
Relationships: Size
Question: Compared to the horse drawing, how is the size of the striped paper?-video 1>
Option:
(A) Smaller than the drawing
(B) Exactly the same the drawing
Larger than the drawing
(D) Not relevant
<video 1>
The answer is C. Larger than the drawing.
Here's why:
1. The striped paper covers the cutire horse drawing. This means the striped paper has to be at least as large drawing.
striped paper past the boundaries of the drawing. This indicates the striped paper is larger than the drawing.
striped paper extends beyond borse drawing. The edges
extending
3. Thee striped paper is not perfect fit. The drawing is centered on the striped paper. If it were the same size. the striped paper would exactly the same dimensions as drawing.
Therefore, the striped paper is larger than the horse drawing.
Larger than drowing
Detailed Explmation:
1. Observation of the Imagges:
The first two images show drowing of horse piece of paper.
subsequent images striped paper being placed the drowing.
2. Comparison of Siees:
the fistt
horse drawing visible and occupies the paper. drowing, covers draw ing and extends beyond it.
images, stried poper is placod
3. Edges and Covergg::
striped paper only covers drawing compleeilly beyund edges drawing, indicating that is larger than the drowing inself.
Conclusion:
Since striped paper covers
drawing beyond
striped paper larger
the horse drwwing.
Larger drawing.
To determine the size of the striped paper compared to the horse drawing. we need to analyze the image and the text provided. The text '4, motion illusion" repeated multiple times. which suggests sequence images
vidco. The striped paper appears be larger than the horse drawing, the drawing partially obscured by the striped paper. The striped paper covers the entire drawing. indicating that it is larger in size. Therefore, the cornect answer is:
Larger than the drawing
Figure 72: sample case of physical object relationships-siee size. Back to List of Figures
105

---

Published as a conference paper at ICLR 2025
Scene: Camera
Question: Based on the effects the video, which of these camera changes would result the observed phenomenon? <yideo
Option:
(A) The camera rotates along vertical axis (upside or downside)
(B) The camera moves upward or downward
(C) The camera rotates slong the horizontal axis (left or right)
The is farther away from the objects
<video 1>
The answer camera rolates along the horizontal axis (lell or right).
Here's the reasoning:
1. 1.bbeerve the metion: The objects are moving in way that suggest rotation around horizontal axss The bagel and cerval bowl
indicaring their position relative the camenis viewpoint. The splash appears moving circilir patlem.
2. E Eliminate other options:
A. The camera rosates alung the vertical ands (upside or downsidee;: This woukd cause the objects appear to move up and down, the
objects aypear fiuther mway, change farther away from the objects: This would make objects appear smaller, change their relative positions frame.
cureen mores upwaad downward:
posiions relative to each other the way we sec.
The
3. Cunclusion: The only
that aligns with the observed motion rotation the horizontal axis (lefi camena's perspeetive
righ). would objects appear
Catoo C. The cener miso aloe the Iaxiooral anis (kf right.
Explastion
underased hos
mevererr
Annlucnn Eack Optinn:
Optios (Camera along ventical adtic cana comse upead dewnack, images woelt niting effect Ubyeax would appoar
spondh
lmages
pampactive
world mexan
Optiie C (Canzra rotaio aboux betizenta lf maso lell or perpetivve of wese would heriantaly, Ofteca
waykfappor mid litt ride. aut the angle at which they on straed woeld change mosingg wnw of nestee amend the sone
Optinn D (Cancra fatther away froe the ebjectaa: If vanera sorves father oway. dhe decca would apyce sandko und maee dittas but thii
miatee postiees encrentons
sloeifeing
in mutch desavell pleneesson
offect cnsh
baraneta
stitt the new
horuneta
nostie ofe camera
Exnclusion:
abserol phenssence imago oupainei uanon votiiia shou haniennal an (lal right) Tiis in change destpttion
propectiv
Therstes, the cernct coees
tenooral clett
The observed phenomenen in the video the appearance of floating objects, such the howl of outs and the cinsamon mills, against a plann background. This created by manipulating cumen's position orientation.
To achieve this effect. the camera needs to be positioned at certain distance from the objects and then rotated along the vertiall axis (upeide downside). This rotation
the illusioe objects floating
Thorefore, the correct choice is A. The camera rotates along the vertical axis (upside downsidee).
Figure 73: A sample case of physical scene understanding-riewpoin.
Back to List of Figures
106

---

Published as a conference paper at ICLR 2025
Relationships: Location
Question: Where does the projection of the butterfly appear in the video? <video 1>
Option:
(A) On the ceiling
(B) On the wall behind the bowl
(C) On the floor
(D) On the table
<video 1>
The answer is On the table.
Here is the reasoning:
We see the butterfly hovering within the -haped bowl. 2. Perspective: The bowl is placed table, and we see its reflection on the surface of the table.
1. Visual Perception:
3. Illusion: butterfly appears suspended mid-air inside bowl, which an illusion
created by the pyramid's geometry and way light refracts through it.
4. Actual Location: Although the butterfly appears to be suspended, is actually a projection onto the table surface, which creates the illusion of depth and movement.
Therefore, the projection of the butterfly appears the table.
D. On the table
Detailed Explamation:
1. Observation of the Sctup: The images show transparent, pyramid-like which appears be table. This structure commonly used for creating bolographic projections.
flat surface,
2. Projection Source: base the pyramid is placod a flat, reflective surface, which likely a screen tablet. This typical setup for holographic display where the projected from the below and reflected within the pynamid to create 3D effect.
pyramid structure. The butterfly is not projected onto the cciling, wall, or floor, but rather seen fleating inside the pyramid.
Position Butterfly:
images, butterfly appears
burterfly appears within the pyramid, projection source (the tablet) is the table. Therefore, the projection of the butterfly appears the table.
Conclesion: pyramid placed
the correct choice is the table.
To determine where the projection of the dog appears in the video, we need to analyze the image and the context provided by the question. The image shows dark room with stone wall and triangular object with projection of butterfly The projection not the dog. Therefore, the answer unknown.
Figure 74: A sample case physical object relationships--ocation Back to List of Figures
107

---

Published as a conference paper at ICLR 2025
Relationships: Location
Question:
How docs the moon pass through the sun in the vidco? <vidco 1>
Option:
(A) From the lower left comer to the upper right comerr of the picture
(B) From the upper left corner to the lower right corner of the picture
From the lower right corner the upper left corner of the picture
(D) From the upper right corner the lower comer of the picture
<video
The D. From the upper right the lower left curner of the picture.
Here is the reasoning chain:
1. The video starts with crescent- shaped visible the upper right of the frame.
2. As the video progresses, the creseent shape gels larger, indicating the moon is muving across the sum.
3. The mon continues moving across the sun until the whole sum visible, meaning the moon has passed through
4. Finally, the moon has reached the lower left corner of the frune, completing its journey the sun.
Therefore, we can conclude that moon passed through the sun from the upper right corner to the lower
picture.
Choice: B. From the upper left comner to the lower right corner of the picture
Reasoning:
First Image:
the from the upper left carner. The moo just starting
2. Second Image:
The monn has moved further, covering more sum, from the upper lefl.
Third Image:
The moon continues
covering larger portion of the indicating dowaward and rightward
moverran
4. Fourth Imuge:
moon now covering the towards the center, still moving from upper
right.
Fifh Image:
The moon is now covering the sun from the upper left, with uncovered part moving towards lower right.
6. Sixth Image:
The moon is muving further down and to the right, revealing more of the sun on the upper left side.
7. Seventh Image:
moon is almost completely past the sun, with only small pertion still covered the lower right.
8. Eighth Image:
The moon has almost completely moved past sun, with last coverage on the lower right.
By observing the sequence of images, it is clear that the moen starts covering the sun from the upper left corner and moves towards the lower right corner, making the correct choice 13.
From the upper right comner to the left corner picture
Figure 75: A sample case physical object relationships-locatio Back List of Figures
108

---

Published as a conference paper at ICLR 2025
Scene: Camera
Question: Which of the listed camera changes could cause the effect shown in the video? <video 1>
Option:
(A) The camera rotates along vertical uxis (upside or downside).
(B) The camera moves upward or downward
(C) Move parallel to the line between the headband and the scraper
(D) The camera rotates along the horizontal axis (left right).
<video
1>
The unswer D. The camera notates along the horizontal axis (left right).
Herd's the reasoning
The video shows with objects iniisllly positioned above the frame, graduilly falling into frame. This indicatcs
Obsrrvaion:
that objocts' apparent from within frame.
2. Eliminating Options:
Vertical Axis Rotaton: Rotating
vettically would primarily affect objects' height, their
moversent
frane.
13. Upward Downward Movement: Moving the camera upward or downward would change the objects' distunce from the
cameraa but wouldn't make them uppear move from above into the frame.
Panalel Movement
parallel
headband and senger would only change the objects' frame.
postiion
other, their poettion
Horisontal Axis Rotation: Rotating the camem the horirontal asis (left or right) would objects origitally aligns observed
positianed
gradually uppear
video.
Therefore, the likely
the effect video along the horizontal axis (left righ).
Cheice: camen moves upward or downward
Explimatoo::
Observatio of Object Position::
In the first image, all objects ane aligned horizontally at the top of the frame.
As the progresses, the objects appear image, most objects no langer ascegt positioned lower.
the frums.
semper,
Anulysi
camers were rotating skong the vertical axis (upeide or downside), objects would appear or rocate around a
central point, which not observed here.
the camera moving parallel the line between the headbund and the the relative positions of the objects case.
change harizontally, which
the camera were rocating along the horicuntal axis (lef) right), the objects would appear to rotate around vertical axis,
which also not observed.
Conclusion:
The consistent upward movement of the objects the frame suggests that the comera itseff moving downward. As the camero moves downward the objects appear to move upward the freme. the
Therrffor,, effect
opward downward, specifically downward this
case
The efTect shown the video caused by the camem rotaring slong the veriiall axis (upside or downside). This becuuse the amage apposr be flipedd movement along the vertical asis. other options (mowing porillel the line betwer the headhand and the scraper or rotating along the hocizontal axz) would not cause the sume miroring effect.
the camera's
changes, which result cumenis
Figure 76: A sample case of physical scene understanding-ling light. Back to List of Figures
109

---

Published as a conference paper at ICLR 2025
Scene: Temperature
Question: Which principle is demonstrated by the crystallization of water when the
hand is submerged? <vidco
1>
Option:
(A) Melting Point Depression
(B) Solution Recrystallization
(C) Freezing Point Depression
(D) Boiling Point Elevation
<video
save Frendiy 2ois Depezsias rometing:
Obonation
The vidoo rhoss
whin had
the cyyollloes indlows of
liged to adii whsc ielening
Theenng Feir: The moot part of (1291) Henorvrr the vair thems at serperzaur abore nernll teezing poir.
Ficeiig
aubkaz oxligaiie sukaiion,
schaz. The pleseneson
schver team.
Rixd is Sciaer soeato, haad ocs soline The prnvaco hand aken the properies wates cusing it floor et tempeotao light semaal foesing poin.
Oaher Qptiner
Mitting Troe Depender This refore the keawing of makting peirr wiltt whea soker is adder. This applic No her hocwer wite
obeenss icoiig zut nelina.
Reayuaaliatie:
diontvin uveld
maiiny aystallic upis Wkile opoidlicaion invervll,
lquit added. nelevett cheaned fiezin Darcitos, the prncete toneeatated by the cyralitation of suace vhen the band satemerged is Thoosty Part Depansons
feer Ulavatoon This arime
Froezing Depression
Devaile Explmation:
1. Observation the Images:
The series of images shows hand being submerged in šir of water.
hand remains submerged. the around hand crystallize. forming ice.
2. Understanding Freezing Point Depression:
Frezzing Point Depression a colligtive
that describes the process where the fieering point of liquié this case, additin another substarce solute) by presence abject (such fommation.
3. Application the Scenerio:
When the submorged
Rioeyy localized cooling
crystals
- The presence of the hand in the water cun disrapt the normal freezing precess, causing the water to freeee at cempersture lower than normal freeiing (D'C 327F).
4. Elimination Other Optioss
A. Melling Point Dipression: This refirs to the lowering of the mehtng point of solkd, which is not releszet here as ane mehing.
observing the freezing
than recrystaliizing it D. Doiling Point Elevation: This redens to the increase the boiling point of liquid due to the presence of solute, which celevant the process observed.
Solution Recysallizatio:
dissolving
happening the images.
Therefore, the prisciple demosstrated by the crystalization water whan the hand submerged is Freezing Point Depression.
C. Frozzing Point Depression
Figure 77: A sample case physical scene understanndnng Back List of Figures
110

---

Published as a conference paper at ICLR 2025
Scene: Temperature
Question: Which true about the temperature of the candle flame when compared to the wrapped paper scenario? <yideo
1>
Option:
(A) Higher in the direct contact scenario
(B) Lower in both scenarios
(C) Slightly higher when wrapped
(D) The same in both scenarios
<video 1>
Eigher
Bor for culandian
1. Unketaddrg ig de sourir ore compering the tompertare of axtlle fane when diieetly teadles paper ond whee tee paper warppel aounal glas.
abjert
object
Dina Deeter When dime riverty reches
mundre and affewenty tigher
& Wisyed Secnais: Wisn Do pauc is wraped uroudd plos, the slaw ads es baria. The alao dnsclo saxe oftlle lox. from das flane provellng de pourr mocking the come lags tompention would conset
woapped
Thi hearse the glas the wapped soceati was host skii, abottingg some of the host froes the flone and prewering the from weiling sane figh impernase would dacci cxzad.
Cheor lighhr io diist conactt exntnt
Explamater:
The
cndll fane divetly
The scana act alano candl flane bumnn
vempped sonnd sloo.
Bioeet Scanato
inngos pape cieclly expoad caall: The flane giidly iasilr papo, andina is
Wingeol
In the sourd att tragos the nnaped urcedd glas and tlex cxpesed to caaiie fumxc Tae flane stll lern the bu dee deprecenno
glas atteca the berning procose
pepe trpperoee quddly cradin igstr When nraped sonzal also, slso ank, chorbit wesz leat fan the fazx. This ralacs tangentas:: the coepord din conad conaato.
poer
Conclaaer
keegastee
liglor
intemeda glow)
Therreer the oover cheiee
Higher the direct contor comerio
Toddeemis: de Iaspontav of caalie flaxs vongnaed nasped paxr scumii, ne ued tosoendeer the lea tanafer snl insinion propatio of lcb iateris.
caalle direst will mat
hest five flase tuxsfrel diestly
monck ixfett
the the andie ttans
2. Wraped paper. Whes the cnale vappped is rape. lac Iea fun faam was cisedly tranfored io lie paxx. lastad, the rape uza inmdace. persaiigg box maching the paper This ouges the serperatee cnafie lowr oragoal in paper
able
looo when cadfi
Theecter, ovren choice A Huter the derct conte: screan.
Figure 78: A sample case of physical scene understanding-
Back to List of Figures
111

---

Published as a conference paper at ICLR 2025
Dynamics: Gas
Which option accurately represents the correct sequence of events in gastfication from the choices below? <image 1><immage 2>-image 3>
Question:
Option: (A) 3 3-2-1 2-1
(B)2-1-2 1
(C) 3-3 2D)2-3-1
<image 1>
<image
<image 3>
The correct answer C. 2-
Here's the reasoning:
1. Image Shows a dry whetstone, indicating the starting point. Image Depicts water being added whetstome. This the first step the gasification process. where water introduced.
3. Image 3: Shows the whetstone soaked in water, illustrating the final stage of gasification.
Therefore, sequence accurately represents sleps involved gasification
C-1-3-3
The correct sequence of events in the images is follows:
Image #1: The block and has visible liquid surface.
2. Imzge #2: Some liquid has been applied the block, but not fully covered. There few palihees of liquid.
Image #3; The block now more thoroughly covered with liquid, indicating more liquid been applied time.
The sequence shows the progression from dry block to block with some liquid, and finally to a block with more liquid. Therefore, the currect order (dry block), 2 (some liquid), and (more liquid).
To desermine the cornect sequence of events in gusification, need to analywe the images and the changee they show.
image shows any gasificattion process occurred
it.
imagee
step gasification. silbouetie
now partially by
substance, which lisely the gas penduced during the fest step.
shows the box after secand grsification. white substance surfice the box, indieating
covering the entire
gas been relensed filing
Bused this analysis, the timage
of inomms
placed
2. The goificatrin proeess begins, and the horse silhocette partially obscured by the gas (image #2).
3. The relessed and fill the box (image H3).
Therefore, the cornect umsweris C-1-2-3.
Figure 79; sample case of physics-based based dynamics air. Back List of Figures
112

---

Published as a conference paper at ICLR 2025
Dynamics: Gas
What changes are most likely to occur in the surrounding gas environment when the phenomenon in the video happens? svideo 1>
Question:
Option:
(A) Increased air pressure
(B) Increased humidity
(C) Decreased air pressure
(D) Decreased humidity
*vidoo 1>
The answer D. Decreused humidity. Here's the reasoning:
The video shows the shrinking of lake. This indicates loss of water,, which component of humidity.
Humidity is the amount of moisture in the air. Moisture comes from sources like bodies of water like lakes.
evaporates the atmosphere. This evaperation reduces the of water vapor available in the air near the lake.
the lake shrinks, the
Less water vapor in the air means lower humidity. This leads to drier environment around the shrinking
Therefore, the most likely change in the surroumding enviromment is decrease in humidity.
Choioe: D. Decressed bumality
Explanation:
Obsrrvation of Imagee:
images shows body Over time, the like's water level decreases significuntly. with the lake almust drying up by the end of the series.
lake)
1984 2016.
Understanding P'henomenon::
phanomesin depicted daying up
As the lake dries up, the amount of water maribbll for evaporation decresses.
Impact Humidity:
Humidiy the amcunt water vapee present in air.
Large bodies water contribute the local humidity through the process of exporatiom. atmosphere.
the lake dhes up, there water available
Conclusion:
With less water availbble for ocsperation, the amount water in the air decresses. humidit
Therefoce, the most likely change the surmounding atvironment when the lake drie decrease in humidiay.
The phenemenon in the vidco the formation of large water body, which is likely lake or a reservoir. When happens. the surrounding gas envireement likely to experience docreased air pressure due to the displacement air by the and it forms, it displaces the air above leading to decrease in air pressure. The other options, such incressed air pressure, increased humidity. or decreased humidity, are not directly rolated to the formation of water body. Therefore, most likely change in surrounding gas environment is docreased air pressure. the choice Decreased pressure.
body. This because the water body occupies significant volume space,
Figure 80: A sample case of physics-basee dynamics air. Back to List of Figures
113

---

Published as a conference paper at ICLR 2025
Dynamics: Manipulation
Question: To mix the tongs, which point suitable? <image 1>
Option:
(A) Point A
(B) Point B
(C) Point C
(D) Point D
<image 1>
The answer D. Point D. Here's why:
Purpose: goal
tongs. requires grip Point 13's Grip: Point D located the juneture of the longs, where they form "V" shape. This
twisting turming.
position provides strong grip and allows for leverage.
Other Points:
end of the tong. providing grip making diflicult Point B: the base of the tong. where the metal thinner, making prone bending.
Point C: Is in the middle of the tong. offering doceat grip but not as much leverage as point D.
conclusion, Point offers the combination grip and leverage for twisting and mixing the tongs effectively.
Choice: H. Point B
Ressoning:
Function Tongs Tongs typically used and lift objects. design of tongs suggests they handligg items, pussibly suga
2. Seructure Analysis:
Point This part the laadlee, which suitable for
Point also part hundle, to gripping end designed be bell by
designed mixing. Poiist This the inter part of the gripping end, which not ideal mixing as designed to grip objects.
Poass This the outer part of the gripiing end, which rounded and spoon-ike, making suitable for mixing.
The rounded, spoun like end at Point designed to internct with items being handled. This shape is khoil for mixing us in can sir and cumbine ingredients effectively.
Design Comsideraion:
mixing, rounded ingredients The other points (A. C, and U) are eiher flat or designed for grapping nox for stiring.
through the misture smoothly, ensuring even
Therefore, Point the sutable mixing.
To mix the tongs, the most suitable point would be Point C,
Here's reasening chain:
1. Point A the topmost point of the tongs, which is not ideal for mixing it's the farthest from the handle.
Point B is the bottom point of the tongs, which is also not ideal for mixing as it's the farthest from the handle.
Point the middle point of the tongs, which handle.
suitable for mixing it's the closest
Point D the point where the tongs are bent, which not ideal for mixing as it's not a straight line.
the suitable for mixing the tongs Point C.
Figure 81: A sample of physics-based dynamics manipulation. Back List of Figures
114

---

Published as a conference paper at ICLR 2025
Relationships: Depth
Question:
Identify the closest point to the camera:? svideo 1>
Option:
(A) Point A is closest
(B) Point B is closest
(C) Point C is closest
(D) Point closest
video 1>
The
Hore's why:
Identifying Key Elements:: We need find the closest point camera among points,
and D.
2. Point A's Position: Point is located on the arm the woman closest to the camera.
3. Comparing with Other Points: Point B is woman's clethes, Point is woman's arm further further from the 4. Conclusion: Since Point A the closest the camera. and all other points either clothing or further away on the same arm, Point A the closest point to the camera.
from the camera, and Point
the woman's
R. Point R closest
Reasoning Chain;
Olbsevvation of device the person openating:
Point B on the person's wabt
Point C penson's
2. Relative Positionng::
positional
person.
Poini B un penso''s waist, which clearr the cameraa devioe.
Point C is on the penson's which extended towards the device.
Point
the device, which further away from the cumpured the other points.
3. Depth Porception:
TTee person's body is closer to the camera than the deviee they are openating
the The device (Points and D) is further away from the camera compared to the person's body.
because
4. Conclasion:
Since the person's waist, closest part person cumara, closest point amoog
givm options.
identify closest point camera, located the side of image, while point located the right side of the image. Point located in the center of the image, and point D located on the bottom right comner of the image.
the perspective image. Point
From perspective the side of the the camera and appear be in the background.
image, point A appears to closest point the camera, as is located the foreground, Point point C, and point D all further away from
Therefore, closest point to the camera point
Figure 82: A sample case of physics-baseee based dynamics fluid. Back to List of Figures
115

---

Published as a conference paper at ICLR 2025
Dynamics: Fluid
Question: Rased visual phenomena the video, which alteration the light source is most likely to be the cause? <video 1>
Option:
(A) It's just that light source is weaker and light source position remains same
(B) The color the light changes from purple to yellow
(C) It's just that the light souree is stronger and the light souree position remains the same
(D) The color of the light changes from to limeyellow
video 1>
The answer is Blue conlainer.
Here's the reasoning:
1. The cyan fluid is initially blocked by the purple stick.
purple were removed, cyan fluid flow downwards freely.
There's other obstacle the path cyan fluid, would continue downwards it reaches the blue container.
4. Therefore, the cyan fluid would flow into the blue container.
Choice: Purple container
Explanation:
Initial Setup:
fluids top through sticks that direct the flow different containers,
images setup
2. Observation of Fluid Paths:
The eyan fluid starts the top left flows downwards.
The purple stick positioned such that directs cyan fluid towards green
3. Effect of Removing the Purple Stick:
If the purple stick removed, cyan fluid will no longer be directed towards the green container. the eyan fluid follow the path resistance, directly downwards.
4. New Path of Cyan Fluid:
Without the purple stick, the cyan fluid will flow straight down.
next obstacle its path the orange stick, which will direct cyan towards the right,
5. Final Destination:
After being directed to the right by the orange stick, the eyan fluid will flow into the purple container.
Therefore, if purple stick removed, cyan fluid would flow the purple container.
If purple stick removed. cyan fluid would flow the purple container. This bocause purple stick acting barrier between the and purple containers, preventing the eyan fluid from llwing into the purple vontainer. With the purple stick remavved, there would be no obstruction, allowing cyan fluid to flow into the purple container.
Figure 83: A sample case of physics-basse based dynamics fluid. Back to List of Figures
116

---

Published as a conference paper at ICLR 2025
Dynamics: Others
Question:
What happens to the green laser beam as it passes through the prism?
<video 1>
Option:
(A) bends and splits into spectrum of colors
(B) It reflects off the surface
(C) It remains straight and unchanged
(D) scatters different directions
video 1>
The answer is A. bends and splits into a spectrum of colors.
Here ressening
1. The video shows a green laser beam passing through prism. piece 2. light passes through geism,
that bends light
because differest wavelzngths light trwel differest speeds dirough the glass.
4. This bending of light causes the differmnt wavelengths to spprite, crtating spectrum of cokes.
Therefore, green laser
spiits spectrum as posses through the prisu.
Chokce:
splits spectrum
Explanation:
1. Observation the Images:
The images show grean leser beam passing through priem.
The laser incident prism bends again the prism.
surfsce the prism.
bends
Refraction:
exits prism refraction occurs light passes one medium to another with different refiactive indes, comsing the light change speed and direction.
hending the
The images show measarements of the angles incidence and refraction. which can explained Snell's Low: V. n__ wiie(theta 1)= 2 2isine 2) where n and n 29) nr the reffactive indices the two modis, and Wheta I and refraction, respectively.
Whhta angles
4. Dispersion:
which spliring light constituent coles, This because difTerent wavelinghs light refonct by dilferent seounts.
Although primanly show bending the laser prism typically
Conclasion:
images not explicitly splining into a spectrum of coloss, the barding and the context using prists suggest that dispersion also occurring. Therefore, the exrrect choice that the leser beam bends and splits into spectrum colors.
green boum
passes through prism refraction.
Thms, the conrect answer is A. beads and splits asto spectrum of colors.
When green laser beam posses through prism, undergoes refraction, which is the bending of light passes from one with different refrectiv index than numnal (an imaginary line
mide direction. As the green laser beam enters the prism. it slows down and bends towards perpendicular to the surface at the point of incidence). This bending cmuses the light to spread out into its component colors. spectrum. This phenomenon known
causing light change
Therefors, the correct answer is:
A. It bends and spins into speerum of colers.
Figure 84: sample case of physics-based dynamics others. Back to List of Figures
117

---

Published as a conference paper at ICLR 2025
Dynamics: Others
Question: What is shown happening in the video? <yidco 1>
Option:
(A) Can't tell
(B) Flowers bloom
(C) Flowers grow
(D) Flowers wither
video
The answer is A. Can't tell.
Here's the reasoning:
The video only shows single image of flower. 2. No movement or change: There is no visual indication of growth, blooming, or withering.
1. The video is static:
3. Lack of context: The vidco provides no information about the flower's life cycle the time period captured.
Therefore, based on the provided information, we cannot determine what is happening the video.
B. Flowers bloom
Step-by-step explanation:
1. Obscrvation Initial Images: The first images show flower that is partially open.
The petals are not fully spread out, and the central part of the flower is not fully visible.
2. Progression Over Time: As move through the series of images, the petals of the flower gradually spread out more, and the central part the flower becomes prominent and visible.
3. Final Images: In the last few images, the flower appears to be fully open, with the petals spread wide and the central part fully visible.
4. Conclusion: sequence images shows flower transitioning from partially open state to a fully open state, indicating that the flower is blooming.
Therefore, the correct choice is B. Flowers bloom.
The image shows single yellow daffodil in focus with a blurred background. The daffodil appears be in full bloom with bright yellow center and petals that slightly curled the edges. There possible to determine what is happening in vidco based this still image. Therefore, the answer A. Can't tell.
clear indication of movement or change over time, not
Figure 85: A sample case of physics-based dynamics others. Back to List of Figures
118

---

Published as a conference paper at ICLR 2025
Dynamics: Collision
Question: If the green sphere much heavier, which direction would the red pulley move? <video
Option:
(A) Stationary
(B) Up
(C) Down
Left
video
The mower C. Down.
1ere is reasuning chain:
If sphere much heavier than the sphere, the
will pull down yellw attached
yellow nope
pulley, pulley also be polley by sepamne yellow rope. When the blue pulley goes down, rope will also
pulley comnucted pulled down.
The rod pulkey attached this yellow
will als be pulled down.
Therefore, the red pulley will move down.
Choice: C. Down
Explanation:
Understanding Systeme
The images show system of pulleys with weights attached them.
The red pulley is connected to weight.
The green splore is part different pullcy system but is connected to the same overall structure.
2. Observing the Metion:
the series of images, the red pulley and grey weight remain stationary.
The green sphere and
weight are moving downwards.
3. Eiffect of Increasing the Green Sphere's Weight:
1f the green sphere were much heavier, would exert greater downward foree on its pulley system This increased force would
green sphere downwards rapidly.
4. Impact on the Red Pulley:
The pulley systems intercoemected. increased
heavier green sphere
would create greater tension connecting ropes. This increased temsion would pull the red pulley downwards as well, as the system seeks to bulance the forces.
5. Couclusion:
Given the interconnected nature of the palley systems, much heavier green sphere would result in downward movement pulley the increused the ropes.
Therefore, the red pulley would move down.
Down
Figure 86: A sample of physics-based based dynamics collision. Back to List of Figures
119

---

Published as a conference paper at ICLR 2025
Scene: Light
Question: Considering the occurrences in the video, which light source modification
listed below is most likely responsible for the phenomenon? <video 1>
Option:
(A) The color of the light changes from orange to purple
(B) The color of the light changes from blue to limeyellow
(C) The light source moves downward
(D) Move parallel the line between the cigarette the peach
video 1>
light moves
Here's the ressening:
1. Observe the shndowe Notioe how the shadows of alll objects elongatid and pointing the dinection. This
indicates that
ingle, casting shadows. 2. Compure the shulwws: The shaduw of the cigaretic box is longer than shadow the peach. This means the cigunette box
claser to the light source.
3. Analyze the change: The video shows the shadow of cigaretie becoming shorter. This means the light
4. Doolcee the movement: Since the shadow is shrisking, the light source most be moving downward, tewards the objectt on the suriace.
likely light source modification light source moving downward.
Choice: C. The light source moves downoard
Explmaatoo:
Observation of Shadows:
la the scries asages, the shadows of the objects change position and shape.
Iaitally, the shadows are longer and more stretched out.
As sequence progresses, the shadows Anolysis Shadow Behavior.
shoner
directly baneath objects.
Shudows become shortor and more directly benerth objects when the light source moves closer to the objecrs (i.e, downward) moving parallel the line between the cigarette and the peach, the shadows would shift burisontally
the light
than becoming
Changes in the color the light (options and Rj would not affest the longth and position of the shudows in the manner observed
Conclusion:
observed plenoemenon of shadows becoming shurter and more directly beneuth the objects is consistent with the light
sounce moving dowsward.
Thereforo, most modification respansible for observed changes video that the downward.
To determine most likely vause of the phenomence, let' anayue each optian stop by step:
The color light changos osinge purple: option sugpests unlikely as the coler of the light is not menrioned in the video.
changing color,
The color of light changes from blue limeyellow: This suggests that the changing which light not
also unlikely
video.
The light source moves dowmward: This opeion suggests that the light source moving downward which could cause the option
shadow change shaps. not the most likely cause.
spple notaffeeed by
the and the This option suggests that light moning paraliel to between cigureti the peach. could comse shadow the apple change shape the light source moves, making the most libely canse of the phenamenoon.
parailel the
analysis, aprica 4 soutee moves purallel the line betwem the sigaretie and the peach.
licely
phenomenon.
apple changes shape light
Figure 87: A sample case of physical scene understanding-ligs light. Back to List of Figures
120

---

Published as a conference paper at ICLR 2025
Scene: Camera
Question: Which of the following modifications could produce the visual effects seen in video?
<video 1>
Option:
(A) The camera is closer the objects
Move parallel the line between the jewel and the balletskirt
The camera moves upward downward
(D) Move parallel the line between the jewel and the poncho
video 1>
The answer The camera moves upward or downward.
Here's the reasoning:
1. Observe the visual effect: The objects appear to be getting smaller and further away from the viewer. This
is a classic sign of camera moving away from the scene.
2. Eliminate options:
The
is closer the objects: This would make the objects appear larger, smaller.
B. Move parallel to the line berween the jewel and the ballerskir: This movement would keep the
objects at similar size and perspective, not create the shrinking offect. between jewel the poncha: Similar to option this would cause
Move purallel
the objects shrink.
3. Conclusion: The only remaining possibiiity is that the cumeraa is murving upward or downward, creating
the illusion of objects shrinking and moving fürther away as the camera gnins distance.
B. Move purallel to the line hetween the jewel and the skirt skirt Detailed Explanation:
Observation of Images:
The series of images shows consistent of objects: "WANTED" poster, ballet skirt, poncho, and
jewel.
The relative positions these objocts change slightly from one image to the next.
Analysis of Object Movement:
The "W ANTEDD poster and the ballet skirt appear move slightly to the left in each subsequent image.
The jowel appears to move slightly to the right in each subsequent image.
Deternining Camera Movement:
the camera
moving closer the objects (Choice A), would expect
appear larger
over time, which is not observed.
1f the camera were moving upward downward (Choice C), we would expect a vertical shift
objects' positions, which
observed.
If the camera moving pamallll the line between the jewel and the poncho (Choice would
expect the relative positions of the jewel and the poncho to change more significatll. which not observed.
4. Conclusion:
moving parallel the ballet skirt (Choioe B). This movement would cause the objects the left (the "WANTED" poster and the balket skirt) to appear to move leftward and the object on the right (the jewel) to appear to move rightward, which matches observed changes the images.
observed effect consistent with
jewel and the
The visual effects seen in video, where the objects appear be fleating or hovering in the air, could be produced by the camera moving upward downward. This because when the direction, creates parallax effect, making the objects appear giving the illusion of them floating. The other options, such as moving parallel to the line between the jewel and the poncho or being closer to the objects, would not produce the same effect. Therefore, the currect The camera upward downward.
moves vertical different distances from the camera,
Figure 88: A sample of physical scene understanding-viewya
Back List of Figures
121

---

Published as a conference paper at ICLR 2025
Relationships: Location
Question:
How does the moon pass through the sun in the vidco? <vidco 1>
Option:
(A) From the lower left corner to the upper right comer of the picture
(B) From the upper left corner to the lower right corner of the picture
(C) From the lower right corner to the upper left comner of the picture the lower left comer of the picture
(D) From the upper right
video 1>
The answer is D. From the upper right corner to the lower left corner of the picture.
Here is a breakdoww of the reasoning:
The vidco shows partial solar oclipse. This the passing front
completely covering
2. The initial frame of the video shows the moon partially covering the sun, with crescent-shued portion the sun visible.
crescent indicates that moving towards the lower left.
stapp
entering upper right the and
4. As the video progresses, the crescent of sunlight becomes smaller, indicating that the moon is moving across the sum from right left.
Therefore, the correct answer D: From upper right comee lower of the picture.
Choice: A. From the lower left corner to the right cornrr of the picture
Reasoning:
1. Initial Observation:
The first image shows total solar eclipse, where the moon completely covers the sun, leaving only the
corona visible.
2. Subsequent Images:
In the second imago, small creseent of the sun starts to appear on the lower right side of the moon.
we progress through the images, the crescent becomes larger moves towards upper
left side of the
3. Direction Movement:
The sequence images shows illuminated part sun moving from the lower right upper
left.
This indicates that the moon moving in the opposite direction, from the lower left to the upper right.
Conclusion:
Since the illuminated part of the appears lower and towards the upper left, the momn mast be moving from the lower left to the upper right.
Therefore, correct
the corner upper right corner the picture.
D. From the upper right corner to the lower left corner of the picture
Figure 89: A sample of physical object relationships-locatoon Back to List of Figures
122

---

Published as a conference paper at ICLR 2025
Property: Attribute
Question: Which point with option in the image represents the object that possesses the most brittle? <image 1>
Option:
Point
(B)
(C) Point C
(D) Point D
image 1>
I.
1knrs way.
weliny
*A Auay rued, qpertically ewil for her erpccaty tf made
her demuped
(loot) appore cunrii, is graurity Tkoasor: mitrnely dirpte stope. designed aithard some proseee.
Clandi: trnkk wool for Desislly roissaee lendin, ont shaseely bents
Doside
3 Coedesaa Baol these fasees the one (Poic Al the likey etjon av twiske While de bovd codd be bilkk, each desigt cnd potoria for quiliy
Choice: D. Point B
Reasoning:
1. Liddestanding Britlennes:
Britkeness
peoperty describes the tendency material beeak without significant defonmation relatively litle energy prior
subjected Dritcl
2. Analyzing the Objerts
Point A (Axek:
is typically made metal, which be and durable. Metals med for tooks like they dasigned withstand ampact stress.
Point B (Bewli: The bawl appears to be made of material that could be ceramic or glass. Both cenamics and glass are known
for their trittlmees. They can shaner break easily when dropped ur struck.
Point C (Burne): The hurel litely made of wood metil. Wood generalyy britkk; it hend and absorb impoet to
are designed
britle.
Peint D [Life Jacket): A life jacket is made soft, flexible materials like foom and fabriu. These materials are not brittle; they designed to be buoyast and fleuble.
Ammin the objects, the bow) Point B the most likely to be brittke, Ceramics and glass are chessi examples beittle maserials, as they can break shatter casily under stress
object expresents
Pain unodir sick
bext
New, lefk zndyoe the testinasss each argar
beitle expcidlly
Wiedez tural Wind iateral thas beats, epecialy when tts
4. Lewker log (Prir: Dy Leaker nanual mesital the be bistle expeiilly when i'v ell
otjocs posers
Tiarifer, de unver D. Pois Di loaker buai
Paix: Code wooks banz) oppract fe diidd de zea iaticaac.
Figure 90: A sample case of physical object property-atrribute attribute.
Back to List of Figures
123

---

Published as a conference paper at ICLR 2025
Dynamics: Manipulation
Question: Which of the provided options demonstrates the accurate steps to take the white rope off the black metallic rail? <image 1><imagee 2><immgee 3>
Option: (A) 1-3
(B) 2-33-111
(C) 2-1
(D) 2-3
image 1>
imago
image
3-2-1.
Her's the mosniig
sows rap beis whiw
repe
rpe lueging
complinity anuched
Tharcker. da lockk of iango à laage (sgs hora pallol ofl,, Jon lunge (sgx patiall elft, and lody Imng: trepe Adly staded.
Choise D. D.1-2-3
Ramornay ther:
wod blick meullls
where mellll
Imsu A2;
rope patially ad back indinaoo posco sanoving ope bepas compleady
ny:
The whike supe is compledy of black rndalio rad. This the fini porkia whae Excape Las leze sacoasfally rananal fax theral.
byttee hxphemtio
with (trnae Sunt the wase all baac mutdlic
motilic will
Sup clense sis Complite the proces by dully recaning the white sape from the black rastalic sall
Tha, the sureet soganes images doncnatating the acuunic sispe vikie rope mutdiic
ines stena paovrtdd
rope stacked mps loaedi wooke atrddas, appoas to le par krad piaze
Cjette euguot sopurar -1 tmplice that whine rope to stuched wooden erazen (2), remooed fhee fraly.
Oytice seggnas sequoee 2 This inpllcs tue the white first anached soodes strukee 121, thee the wooke stractao renowal foo the (3)s and sealls, the mee deached fran the 12).
Optice Coaggnas segumne dradied
tepties
(2)s
Optice D arguee the reguraer 1-3-3 3 Thee tmplic thet the ohine is fine wouceet woudin erizum (1), thnn the voodin eneree leneesow tre
the (2), deally rope diacced
Lycs curnaing steakine diodyy atachad sell comos the firs skop Thexties, optins and Cran be diiiinted bocksw they shat with the wootes stuctre bone atacded to ral.
waste
avond weeden
anached Mack mandlic The wooden
seens lngical The socend skg would the moode atrudtrs rall the warl oktuca dhe vikte
which tuthe fin
de sqqaone
Attah whese vosdee
Rertave the wedes starie trem
beauh wiste rego fros de
Figure 91: sample case physics-based dynamics manipulation.
Back to List of Figures
124

---

Published as a conference paper at ICLR 2025
Scene: Camera
Question: Based the phenomenon the video, which of following camera changes can produce the effect in the video? <video 1>
Option:
(A) Move parallel the line between the binder and the safetypin
(B) The camera closer to the objects
(C) Move parallel to the between the ipod and the safetypin
(D) The camera is farther away from the objects
<video 1>
The inowr is The couurs futheer away from dijess
Has the sippttyacc remeesa:
Oberre
sako-llws apcesed shiceo uppr scalle spal jeopeasos.
atgas apear
oppor adhovel
spon
while
falae aay lun Du sloca.
Apply to the vides: Stnce the video dows memeou cttect, the catera man moving Bitter away thee the dtyass
Thouserr te vance change Euit pookees the effer the vileo is cance fuffirr eway fioe fie eljent.
Cloke Moso parffic to the line betwoen the header and sifry pia
Oborraizz Iasao:
The onte of imagee choso binder city earry pes, and ematl spure object on indi-pereeedd cuttirs. The yelow pootiriikk abennoe oppears moving
images.
remus ta the yalow poweteratle aberace the only cianen the change portoon monnzg comatintt thes kett night.
tesker
Deternniza Cazen Mosenaat:
ponder
monomire of the yelore pooder fre let nightt segner than the conera in moving in rivotie fur caneer the milate movion
Didteinn the Optizee
arlyy
mostng horoveslly
aligne
canara
waid seua
perpective chrees, which coserved.
More pasile to the ane betwees the atud and the uarry pe: thew no thot in images in the opnce a intiecat.
D. cenen futher owey from the oljict This world the wout dlange sie peopectie of he ests, which chserved
The oncicoes bett anhoovippe
the yelliwe white
Indicaas tue
beower the tender and urity pre.
Thasfar, the caneet chokee A. Mosic paslicl the lac betwoaa the binka and the safoo pis.
pheooeeoo
optooss prositett
Move guaild tod linc belvees de lande Excaalttte:
If the caner move pondist to the the bensoes the hadde and the antropes, the retate postons of styce will nor change, and the oftict will viitee
the iockear the
offer
The amazn redrree avay diees dhe cbieea
canan ovay bvon ths sbisco, la offeet became saaee proconnec, the piinary aue
If the cancu mono pondll to the lix betwer the ipod docualitpie, dee bccadiiie pootian eljats will aus cleap, and the effest will vidkee
coum atreae
vijeds, any bocoee racc peceeaxed,
prinnay cano offee...
Haaod
Mosc panllci uo lax lize betroam the bunke and the anktogi
ortopii go postiios ejjor,
Figure 92: A sample case of physical scene peteersttanninngiievpoooo
Back List of Figures
125

---

Published as a conference paper at ICLR 2025
Scene: Light
Question: Referring the phenomenon shown in the video, which light change from the options below would most likely cause? <video
Option:
(A) The color of the light changes from limeyellow to pink
(B) It's just that the light
is weaker and the light position remains the (C) It's just that light source stronger the light source position remains the same
(D) Move parallel to the line between the backpack and the cake
<video 1>
The
line between lackpack and cake.
Here's ressoning:
1. Oéserve the shadows: The shadows the vidco elongated and clearly defined, indicuting light source that posiikoned
relatively close to objects.
objects moving als moving. not just changing in intensity or color.
Analyze
changes:
light
moving in direction that parallel to the line connecting 4. Dedoce the light source movement: For the shadows to move this way, the light source must also be moving direction
3. Identdy the shadow movement pattorn: The shadows
backyack and the
parallel to the line connecting the buckpuck and the cake.
Elimminat other options:
Color change wouldh'
"B, A weaker light source would make the shadows less defined, not move them.
C. C.A stronger light souree woulde't change the direction of the shalous.
Therefors, the lisely the shadow changes video sounce moving parallel
backpack and the cake.
Move parallel line Reasoning Chain:
backpack
1. Observation of Shadows:
The series of images shows consistent shadow cast on the ground.
shadow's s shape position change slightly across images.
Analysis of Shadow Movement:
The shadow appears to move horizentally acroes the images.
The objects (backpack, cake, ham, and barrel) remaiin slationary, indicating that the change is due to the light source movement.
Light Source Position:
The shadow's movement suggests that the light source is moving.
The direction of the shadow's movement indicates the direction of the light source's movement.
Direction Light Source Movement:
TThe shadow moves
that suggests the light
moving parallel the line between the
backpack and the cake.
This movement would cause the shadow to shift horizontally as observed in the images.
Elimination
Options:
Optioon A (Colar Change): The images do not any change the color the
Option B (Weaker Light Source): A weaker light source would result in dimmer scene, not change in
shadow position.
Optioo (Stronger Light Source): stronger light source
brighter scene, not change
in shadow position.
Therefore, the most likely cause of the phenomenon shown in the video that the light source is moving paraliel to the line between the backpack and the cake.
The likely
of change light is that light position remains the same. This bocause the shadow of the backpack and the cake appears to be getting larger, indicating that the light source is moving further away from the objects. The other options do not explain the change shadow size.
weaker and the light
Figure 93: A sample of physics-based based dynamics collision. Back List of Figures
126

---

Published as a conference paper at ICLR 2025
Relationships: Location
Question: Relative the movement of the rope, what is the position of the hand
initiating wave? <video
Option:
(A) At the rope's middle point.
(B) At the free end of the rope.
(C) Near the rope's starting end.
(D) At both ends simultancously.
<video 1>
The answer & C. Near the ropess starting and.
Here's breakdown:
Observe the opposite
shown oscillating standing pettera. Standing
Idestify the fioed point: The rope is foxed at the top, indieating that the end attrched to the ceiling is node (4 point no dispiacement).
rope oacillates, creating Anilyae the hands posiiio:: The hand is always near the repe's starting end, not at the middle the fiee end, and it's not si both ends simulaneously.
Recogrisze patter:: The hand holling rope moving up
generating
ahematiin nodes (N) and untinodes (A) along its lenth.
creating standing putlern, respunsible initiating aligns with the choice of being newr the rope's strrinn end.
Therefore,
the
the rope's
C. Near zope's starting
Explanation:
Observation the Images:
The images show exhibriing difTorent harmanic modes (1st, 2nd, 3rd, and 5th hammonic))
hand intiiatnn
vasble image, consistently positioned rupe.
Understading Hammonice
lammanics standing
specific frequencies.
(fundamental frequency) one astmode (A) The 2nd hammonic has two antiodes and three nodes.
nodes (N)
The 3rd liamonic has three nacisedss and four nodes. nodes.
harmonic has five antinodes
3. Posation the Hand:
each image, the hand is moving the cneate these standing wave patteens. ropo,
always
starting
Conclasion:
Since the hand as consitendly the same end of the and not the middle or at both ends, the cocrect answer that the
hand rope's starting and.
Therefore, the comcer choice is
Near the starting end.
position the hand initiating wave, we analyze image of the rope. The rope shown motivn, and
movement
the hund the end the rope. The hand appears the only hand visible and the rope is exlending from it. The rope
to be at the end of the rope, moving from the top left to the bottom right image, indicating that the hand pulling the rope from lop
right. Therefore, the hand initiating the
the
rope.
The vorrect answer is: At both sinultancossly
Figure 94: sample case of physics-basee based dynamics
throwing. Back to List of Figures
127

---

Published as a conference paper at ICLR 2025
Relationships: Location
Question: What position the electromagnet relative to can? <video 1>
Option:
(A) Below the
(B) On top the can
(C) Next to the can
(D) None of the above
<vidco 1>
The answer A. Below the
Here's rensoning
Visual Inspection: In the beginning, machine that is used to launch the can.
the spinning and then sparks coming of the
Inference: We can infor the sparks are related the launching can.
Electromaanet: The sparks coming of the electromagnet.
4. Location: The electromagnet located below the can because the sparks are coming out from below the can.
Therefore, the electromagnet located below the
Next the
Detailed Explanation:
Observation: the image, see can pasitioned inside circular device, which appears to electromagnet. The held horizontally.
Sequence of Events: As we move through the images, the can is shown being ejected from the circular electromagnet responsible for propelling can.
device. This suggests
Final Position: In the last image, the can is seen lying table, and the electromagnet is visible next to it. This confirms that the clectremagnet was positioned horizontally relative to the can.
the final image and herizontally, is clear that the electremagnet was positioned next to the during the process.
Conclusion: Since electromagnet
ejected
The image shows close-up of cylindrical object with metallic surface, which appears be due its shape and the visible text and logo. The electrennanet is positioned around the can, suggesting it is being used hold or manipulate determined with precision without additional context, it. Therefore, the musi accurate choice would be:
can. The exact position of the electromagnet relative can camnot likely eacircling
close proximity
C.Next to the can
The electromranet not below, proximity to the can, which indicated by the visible text and logo on the can and the electromagnet's position around
completely different position from the in close
Figure 95: A sample of physical object relationships-locaaiee
Back to List of Figures
128

---

Published as a conference paper at ICLR 2025
Dynamics: Others
Question: What the primary reason for the insect's behavior shown the video?
<video
Option:
(A) To hunt for food
(B) To minimize light refraction interference
(C) To hide from predators
(D) To communicate other insects
<video
The answer is C. To hide from predaters.
Here's why:
1. The insect's movement is erratic and seemingly random. This is classic predator avoidance strallegy.
2. The insect is camouflaged its surrondinggs Its green color blends with the vegetation, making it barder for predators spot.
3. The insect is in complex enviromment. The branches provide mumerous hiding places and escape routes.
4. The insect's behavior is typical of prey animals. Many insects use a combination of camouflage and rapid, unpredictable movements avoid being caught by predators.
Therefore, the insect's behavior in the video is most likely driven by desize to avoid becoming prey.
To hide foom predators
Detaile Explanation:
1. Obseration the Inseet's Rehawior:
The series of images shows on inseet in intially stationary brnch. Suddenly, the inseet makes rapid movessent,
splesh.
2. Analysis of the Envinnment:
The envireement
where predaters might nutural habiba with branches and defansive mochanism.
spid
indicate sudden nection,
3. Purpose of the Movement:
The spidd movement and splash consisent with hunting bohavior, hypiingyypccllll irvolves stealth and precision
than sudden, conspicurus movements.
bohuvior does sorm relnted to minimizing light refriction interformce,
typically involve such
dramatic action.
There indication of cummunication with other insects, which would licely involve specific repeoted signals
single
4, Comclusion:
plausible for the insect's sudden and splash evade suggesting trying common many insects
predatioss This type
escaping confusing predation.
Therefore, the primary reason for the insectss behavior shown the video hade from prodatons.
The insect's behavior of hending its legs and positioning itself way creales a shadow on the glass surfice likely form of camoutlage. This behavior helps insect blead in making less visible to predntors. The shadow creates visual effect that mimics the texture and shape of the glass, providing a form of protection. This is common survival strategy in the animal kingdom, where many species have food, trying communicatee with other insects, and there indication that trying minimise light re refraction interference. Therefore, the most plausible reason for this hehavior is to hide from predators.
surroundings.
enviromment to advantage. The insect
actively hunting for
Figure 96: sample case of physics-based dynamics fluid. Back to List of Figures
129

---

Published as a conference paper at ICLR 2025
Scene: Camera
Question: Based the visual effects the video, which of the following changes to camera might create this effect? <video
Option:
(A) The rotates along the horizontal axis (left right).
The camera rotates along vertical axis (upside downside).
(C) The camera moves upward or downward
(D) The camera is closer to the objects
*videe 1>
The mower cheeame moves upward or downward.
reasoning:
1. Visua Analysis: The video shows objects appearing to flost and move upwaed. This indicates charge at the caments position relutive to objects.
2. Eliminating Other Optioss:
A. Rotating along the horisontal axis: his would cause objects to move left or right, not upward. would objects appear forward backwand, upward. comera closer to the objocis: This would make objects uppear large, not floet upward.
Rotating the
3. Conclusion: The only optiun that lupicllly explains upward
objects the perspective.
upward.
camera obyects appear upwards
moves upward downward
Exploration:
Initial Observation:
In the first few images, the objects (peanut butter jur, syringe, tripod, etc.) are placed on wuoden surface. progresses, the objects appear upwand nolutiee the wooden surface.
As the
2. Analysis of Object Movement:
The peanut butter jar, syrings, and other itens graduilly rise ahove the woodan surfice. image, objects completely wooden This upward movementt continues until last image, where the objects are still above wuoden suriave.
appeur floating
Movement Considentio::
Horiiontal Rotation (Lefl Right): the camera were rocating hocizuntally, objjccts would appear muve sideways
nihher than upword.
Vertical Rosation (Upside Downside): camera
vertically, the perspective of the objects would changs.
they undommly
-Closer to Dijjcts: If camem were moving closer, the objects wookd appear bnger but would not rise above the surfice
-Upward Downward Movement the camera moves upward, the obyocts would to ris above the surface the
penpective changes.
4. Conclusiae:
consisient movement of objects relutive wooden surface indicates perspective changes.
moving upward.
This creates effect objects rising surface
Therefore, the correct choice
or downward.
The visual effects the video created by the camera rotzting niong the vestical axis (upside or downskde), This is evident appears moving up down, cousing objects oppor creates the illusion the objects moving in circle, which is cummon technique used 360 del-derree videos to give the viewer semse of immenion and movemuni, The comeral' nutasion arund the verticll asis allows the viewer see différent angles of the objocts, making them appeur angles stitahig together create scamless, 360-degree view. gives the viewer the impression of the objects roturing around the camera.
the
The
effect achieved
muliple fiumes movement olong vertical
from different
Figure 97: A sample case of physical scene understanding-vieepoin.
Back to List Figures
130

---

Published as a conference paper at ICLR 2025
Scene: Light
Question: According to the video, which alteration to the light souree most likely to have triggered the phenomenon? <video
1>
Option:
(A) Move parallel to the between the adhesivetape and the corkboard
(B) The color of the light changes from limeyellow to green
(C) The light souree moves downward
(D) The color of the light changes from yellow to white
Svideo 1>
The unsver is The
deveruan.
Hock the roanning
The shulow ctpers pettag
Tie somes Ooe kangr the dlakos
Concunse Since the dadovo are gating loeger the lign scace must curving dowooards.
Mese pandial the betwoa allicin and oekboard Moving the sounee panile the ofjecta would saniiinnty aled shadow iegtt
The cnlor tyee clange lrmyelio to grver oolor light sonor Sinika sptki cake liis alled londs.
shadoo, not leigh
coke change
Choice: C. The light source moves downward
Expianation:
Observatios of Shadows:
in the senes of images, the shacows of the objects (the adhesive tape, the corisboad,, and the object on the iefi) are changing right objects. As the sequener progresses, the shadows become shurter and move cluser the objects.
hitally shudows longer
Analyci Shadow Dehaviur:
and
light
downward
angle
light relative objects changes, caising shadows contract position.
If the light source were moving parallel the line between the adhesive tape and the corkboard (Choice A), the shadoos would shift hurisontally light
change significantly lengh.
Changes the ilumination.
positicn langth
only
Conclusion:
observed phenomenon slaooows becoming moving objects consistent light
source moving downward. Therefore, the most likely alteration the light source that trgggered this phenomencn that the light sounce moved downwward.
This, cornect choice is C. The light source moves downward
ownhout The alveatio light that after
of
Opton A regco dezi the liget mase taevic pandiil to the las ketrecs the adiovie uav the ootdlwedd This would mas rigaifiani daage the shadeun cape detvttoon
iigt
This
sludowb Listatian
Oytooe capaes that the light scunce morex downoer The weult came the duadon to hecees mos dlangrodd and ditoned, as the light scunee in ling: dieecly exetead.
Ihavd on the diannice of the of ades rost lody abvonton vo light anace a clange in angle the light, witch decothed ir oprios C. Theertn nhe vnssr icc. Tho light sonce mokes aevewand
Figure 98: A sample case of physical scene understanding-lit light. Back to List of Figures
131

---

Published as a conference paper at ICLR 2025
Dynamics: Manipulation
Question: To brush (or dust) the toothbrush, which point most suitable? <image
}>
Option:
(A) Point
(B) Point
(C) Point C
(D) Point D
<image
The answer is Point Here is reasoming
1. The goal is to brush the toothbrush, which means clean the bristles.
2. Point is located in the middle of the toothbrush handle, where the bristles are attached.
3. Point A is too high on the toothbrush, and might not reach the bristles.
taothbrush, might 5. Point D is high the toothbrush. and might reach the bristles.
Point
low
bristles.
Therefore, Point the most suitable point to brush the toothbrush.
Choice: A. Point A
Ressoning:
Purpose Drashing Toothbrush:
the brisiles toothbrush, purt cumtact gums likely accummlate debris, toocapaste residue, and bacteri.
Location of Beistle::
are the top toothbruhh, is marked as image
Effectiomess of Cleaning:
any accumulbed debris buctoris ramoved effectivaly. crucial muintzining ural hygicne and ensuring the toothbrush remains effective cleaning teeth.
Clesning briskles
Other Pointe:
require same cleaning peactice to keep the entre toothbeush clean. the peimary focus should be on the bristies.
Points
parts
good
Conclusion:
Therefore, Point most suitable point dust when clenming toothbrush, as directly addresses the part the tootbeushh that needs the must attention for hygieme purposes.
brush dust) the touthbrush, saitble point would Point
Here's the reasoning chain:
soothbruhh standing upright on green
2. The woothbrush has four points labeled A,
3. Peint the of the toothbrush, which not the best point to brush or dust.
1. Point the side of twithrush, whit which also best point
best brush dust,
toothbushh, 6. Point D the side of the soothbrush, which is also not the hest point to brush dust.
point
toothbrush, making
point
Figure 99: A sample case physics-based dynamics
manipulation. Back to List of Figures
132

---

Published as a conference paper at ICLR 2025
Dynamics: Collision
Question: If the red sphere were much heavier, which direction would the purple
hollow pulley rotate? <video
Option:
(A) Anti-clockwise
(B) Clockwise
(C) Stationary
(D) Uncertain
<video 1>
The Auticlockkwse.
Hords the step-by-step explanation:
syseen
Gravity: Pulling the red sphere down.
Tension the purple bel:: This trnsion is crosted by the weight of the red spbere.
"ension the belt.
created the weight grees
sphero and the
(though directly connected purple pulleyy).
weight qphere
Analyne the purple pulley Thepurple is connected to the red sphere via purple belt. Since the rodl sphere is hervier, purple
laggr
right upward
side.
Datemine rotation: apward movessent of purple left side pulley will pulkey upward, it pulls top portion pulley
clokowes.. because with crusing the rotation.
pulley,
Explanation:
Understanding the System:
The system consists of series of pulliys and weights connected by ropes. pupple hollow pulley
question, connected other
2. Current Movement:
Fromn the series of i'mgges observed ihat the red sphere moving upworts while the white sphere moving downsandds current setup sphere being lighter deseend sphere ascend.
ophere, causing spbere
3. Effect Heuvier Red Sphene:
compared This incressed downword force would conse the red sphere to move downwards instcud of opwards.
sphere were hoavier, exert greater
sphere.
Rotation:
As the sphere moves downwardss would pull the rupe connected to the purple hollow pullcy.
The direction: the rope's muvemunt would couse the purple hollow pulley to molate the diretionn that allows the red sphere descend.
3. Determining Retution Direction:
Observing the if the red sphare moves downwardk, the the right side ofehe purple hollow pulley would be pulled down.
downward pull on right would cuose the purple hellow pulley rotate clockwise direction.
Therefore, sphere much pupple bollow pulley would slockwise.
Soaveerr
Figure 100: A sample case of physics-based dynamics collision. Back to List of Figures
133

---

Published as a conference paper at ICLR 2025
Property: Attribute
Question: Reflecting on the phenomena in the video, which of these camera adjustments could cause the observed effect? <video
1>
Option:
(A) Move parallel the between the screwdriver lemonade
(B) The camera is farther away from the objects
(C) The camera is closer to the objects
(D) Move parallel to the line between the screwdriver and the tool
evideo 1>
The Move
line between screwdriver lemonade.
stop byatop ceasoning:
Olberve image: The image shows a set tacis and glass of lemonade on sconingly flat surfice. The perspective on curved or slanted surftee. running paralld
diserted, creating sense of depth where the objects
the image These lines 3. Consider the camera movemsents: The question asks about camera adjustments that could couse this effect.
the
distoriion
notiosblle
appess curved, indiastig change perspective
A. Move parallel to the lnce between the screwdriver and the lemonsdie Moving the camers parallel the line would cause furtler away flom
appear snaller, creating curved lncs effeet.
cimen farther may from objects: This wonld the objects sppear smaller overall, necesemiyy
croose the curved linas effect.
cleer to the objects: This would make the objects knger ovenil, smaller, and would'' create
D. Move parallel to between the screwdriver und the tool: This movement is irreleant the observed disturtion, as
it would nox affect the relative posittuns of objects.
Conclude: Theredors, licely explanation for the observed diturtion Move parallel perspective change the resulting
movement
Cheice: Move pumilel
screwdriver lamonade
Explanation:
Obarrvation of the Imager:
images
set objects, including
background remaining
constamt.
The objects apperr to be moving relative to the camera, but their relative positions to ench other remain unchanged.
Analysi Movement:
chunging size,
moving
away
(clminating choices and C))
The obrects are moving way that suggests a change perspective along the line between the screwdriver and the tool (climinating choice
Directoon
The obrects appear to be moving in direction that is consistent with the camera moving parallel the line between the
serewdrover and the lemonade.
This paraliel movement would objects shift postion trume while maintaining their relative distances and
Conclusion:
The alserved effect consistent with the camen moving perillll to line between the scnowdriver and the lemorade, observed clinging relative positions Therefore, the cornect choice Move porallel to the line between the screwchiver lemonade.
the shift
The observed effect in the viduo caused by the camera wiussment where the comoni movol parallel to the line between the when the
screwdtiver and the toel. This posiiiuns, creating illusion depth
moved puraliel the line between the screwdriver and tool the
"Parlllx "Parallas Shiff in photogrphy. where the uppazent position of an object changes when viewed from differest angles distances. The other upains, such as moving the camera fanther away or claser to the objects, ur moving the comera purallel to the line effect. l'berefore, the
appor
between the screwdivver lemanado,
Figure 101: A sample case of physical object property-attributc
Back to List of Figures
134

---

Published as a conference paper at ICLR 2025
Scene: Light
Question: Referring the phenomenon shown in the video, which light source
change from the options below would most likely the cause? <video
Option:
(A) It's just that the light source is farther away from the objects
(B) It's just that the light source is closer the objects
(C) It's just that the light source is stronger
(D) It's just that the light source weaker
<yideo
just light
Hore the siep-hy-top ressoning:
Observe
significamlly larger elongsted
Understand sladow Somation: Shadows are Sormed whan an objectt Nocka light. The size and shape of the shadow depend on the position of the light source and the object.
Apply lopie: light source is away, shadows be smaller lass defined. light source cleser, stadows would be larger and more defined. If light source stronger, the shadows would be darker.
Conclude: Since the shadows large and elongated, suggests light weak, the spread more larger shadows.
Eliiminte other options: Option A and D are incorect because the shadows are getting arger, not smaller. Option Cis incerrect because
light would darker, decised shadows.
Cheice: A. It's just that the light source is futther away from the objects
Explanatio::
Obser vation of Staddwwx
la the series the shadows the objects tapple, flower, and artichoke) become progressively longer and more defined.
direction
changes, indicuring change position sounce.
Light Source Distance:
When light hits the
further from objects, the shadows by the objects become longer. This because the causing shadows bocoming loegor, which consistent light source moving futther
which
imagos elese prograsioon
awagy.
The intenitty of the light does not appear to change significantl the images. The objects and their surounndings remain sumilarl illminatee throughout the sequance.
1f the light source were simply gentnn stronger wesker, we would expeet to elanges the brightness of the objects and clinage the way.
shadows, the langth the shadows would
Concluioon
The most plansbbl explanation for the observed phenostenon that the light source moving farther away finma the obyects, images, shudows progressively
the shadows lengthen.
Therefore, the cocrect choice A. It's just that the light source farther away froca the objects.
phcremmzoo abore beonee linger more diflieed seen the image This heasse the has novel grater ficanee caiding to vospped uur hoarne los concerned The durtiess also desidifiie roor eiragrall which much light lekin dieect Thavdrre, noot liirly oano this phensncnes lide seace litter avay diject. chaice
Blddy caave changs tisht When tight tather avay oteecss, dadov
lage scare anay from objera
Figure 102: A sample case of physical scene understanding-light.
Back List of Figures
135

---

Published as a conference paper at ICLR 2025
Scene: Light
Question: Taking into account phenomena observed the video, which of the
following changes to the light source is most likely to have led to this result? <video
1>
Option:
(A) It's just the light source stronger and light source position remains same
(B) The color of the light changes from red to cyan
(C) It's just that the light source is weaker and the light source position remains the same
The color of the light changes from white orange
<video }>
The enswer It's jost that the light sourcee is wesker and the light scurce position remains the some.
reasaning:
1. Observe the shadows: The shadows as the video are clongated and faint. This indieates that the light source low and wask.
2. Consider the objects: objects
are not brightly illuminated,
sharp. This reinforces
idea of weak light source.
Eliminate options:
A, Stronger light souroe: A stronger light source would result in sharper shadows and brighter objects, contradictng the observation
change:
change would apprrranc objects, woukdn't explain and elongated
shadows.
D. Color change orange: Simikar option D. color change orange woukint explain the weakened illumination observed the shadows.
Tharefore, the uxet likely explasation that the light source has become weaker, leading the fainer shacows and danmer objects observed the video.
Choice:
I% just that light source is stronger light source position remains same
Explmation:
1. Otservation Shadows:
ln the series of images, the shadows of the ohjects (milk earioe. chicken and other ilems) become progressively darker defined.
The poskion and shape the sharows remain comsistent thnoughoui the imuges.
Light Source Intensity:
suggest light becoming stronger. a booomes stronger, it casa more pronouneed and darker shalows bocause the contrast berween the illuminatiad aress and the shalowed areas incresses.
incressing
Light Source Positio:
The consistancy in the position and shape of the shadows indicates that the position of the light searce has not changed. the would changed
had moved, the direction
Color of'l Light
There noticcable
the of the objects and their shadows exhiba color orange.
shifls suggest change
eyan
Resed on these observations, the most likely explonation for the phenomeno observed in the video that the light source has stronger while stronger and the light source position remains some.
Therefore, the cocrect choice It's just that
It's just light source wesker the light source position remains the same
Te shadows observed
consistent
indicoting that the lght canonn heosming distines,
light souree basome wasker could be due to the light source being moved finther away or the light suunoe issef' becoming dimmer. The position of the shadows relutive the objects und the light remains the same, the light wource position does not change. Therefore, the leading
suggosts
likely explesation
light
Figure 103: A sample case of physical object property
Back to List of Figures
136

---

Published as a conference paper at ICLR 2025
Dynamics: Throwing
Among the listed choices, which one outlines the proper sequence to hang the coat hanger on the cabinet's right door, then open its left door? <image
Question:
<image 2 <image 3>>
Option:
(A) 2
(B) 3-2
(C) 2-1
(D) 2- 1-3
<image 1>
image 2>
image 3>
The answer is A. 1-2
Here the detailed, step-by-slep explanation reasoning chain this choice:
1. Image #Il shows the robot's arm extending towards the right door of the cabinet. This is the initial step in hanging the coal hanger.
2. Image I/2 shows robot's
process hanging 3. Image #3 shows the robo'ss opening the left door the cabinet. This the final step in the sequence.
hanger right doue.
Therefore, the correct order of the images is 1 2 3, which corresponds to the initial position, the hanging action, opening action, respectively.
Choice 1-3-2
Ressoning Chain:
Image
The robor arm hokling the cout honger and positimnd in frent of the open cubinet. This indicates the inkial step where the rubot preparing hang coat cabinet's right dooc.
Image #3:
The hanger hung the cabinet's dooe. The robut holding the indicating the
hanger successfully placed
Image
positioned
suggests now ready
to open the left door of the eabinct.
Therefore, proper soquence
hanger cabinet's right open door Image Al (preporing to hang the coat hanger) Image #3 thanger hung on the right door)
(ready open
door)
Thus, the correct chnice B. 3-
hanger hanging un right left doar, the sequence would he:
image
open. hung hanger
Open
Hang the coar hanger on the right side the left door. Close the left door.
Therefore, the correct sequesco is:
D-2-1-3
Figure 104: sample case of physics-based dynamics throwing. Back to List of Figures
137

---

Published as a conference paper at ICLR 2025
Relationships: Size
Question: Which object the biggest volume? <image
Option:
(A)
(B) red cube
(C) green ball
(D) purple cube
<image
The answer is C.
Here's why:
Although we don't have exact measurements, we can visually assess relative sizes. The green ball is significantly lurger than the cubes, even considering the blue cubc's shadow.
This makes the green ball the biggest volume among the given objects.
Choice: purple cube
Explanation:
several objects: green ball, red cube, a yellow cube, green cube, and purple cube.
1. Visual Inspection: By looking at the image,
2. Relative Size Comparison:
The green ball appears to be smaller in volume compared to the cubes,
Among the cubes, the purple cube noticeably larger than the red, yellow, and green cubes.
3. Conclusion: Based on the visual comparison, the purple eube the largest volume among the objects listed.
Therefore, the biggest object in volume the purple cube (Choice B).
To determine which object has the biggest volume, we need to compare the volumes of the different shapes.
A cube has equal sides, so volume calculated by multiplying the length of one side by
itself twice. The purple cube has sides of length 1, its volume is * 1 1 cubic unit. The yellow cube has sides of length I, its volume also cubic
cubic unit. The red
cube has sides length 1, volume
A sphere has radius, which is the distance from the center to any point on its surface. The green sphere has radius of 1, so its volume is calculated by multiplying the cube of the radius by pi. The volume the green sphere 4/3 pi 1^3
pi cubic units.
Comparing the volumes, we can see that the green sphere has the higgest volume, followed by the purple, yellow, and red cubes, which all have the same volume.
So. the answer green ball.
Figure 105: A sample case of physical object relationships-size, Back to List of Figures
138

---

Published as a conference paper at ICLR 2025
Property: Attributc
Question: In the photograph, which point with option selects the object with the smooth? <vvddoo 1>
Option:
(A) Point A
(B) Point B
(C) Point C
Point D
video
Here's the ressaning:
sumewhat nuugh smmweeryy are typically very smooth, especially when made moterials like nubbeer or plactic.
Point C ring, Rings can be smooth, but they ofter have a slight texture due to the metal muterial they are made from. Point avocado, Avecados have bumpy, textured skin.
Therefore, Point the most likely to be the object with the most smooth texture.
Choise: Point R
Resening Clain:
1. Obsrrvation of Objects:
Point A: stracberry which typically has testural surfice with emall seeds the outside. bell.
round object appears Point C: metalic object, passhly nail or pin, which has a smooth surfice but sharp odges.
Point D: An avorado, which hos rough and humpy skin.
Comparison
Simmwberry (Point A): The surfice not smooth due the
natural texture The surface is unifornly smooth and roundl indicating lack testure.
Pebble/Dall (Point Dj:
Metallic Object (Point While the surface smooth, presence of sharp edges and the ovemil make less
unifornly
Avvocaoo (Poist by: The skin rough and bumpy, making it the losst smooth the abjects.
3. Conclusion:
Amang the objects, other objects.
uniformly smooth,,
inogularitics present the
Therefore, object smoothestt
object
au face, we ecumine object the photograph and asszss texture.
Point A: This is strabbery. Stawwherries have magh testure due to their fibrous outer bneer. suriice, they
white bell. typically
materials
phstic.
Point C: This a matal object, possibly fook Menil abjacts can have smooth rough testure depending on their material and design.
green object, possibly cucumber
mugh
layer.
Bosed on the visual information, Pount B (the white ball) to have the smoothes surfhce the objects in the photograph. T'ereffree,
Figure 106: A sample case of physics-based dynamics fluid. Back to List of Figures
139

---

Published as a conference paper at ICLR 2025
Relationships: Depth
Question: Which point closest to the camera? <video
Option:
(A) Point A lies closest
(B) Point B lies closest
(C) Point C lies closest
Point D lies closest
vidco 1>
The answer is C.
Here's how we got there:
1. The vehicle is closest the camera.
2. Point C the road in front of the vehicle and is the closest point it, making the
closest point the camera.
Point D. A are all in distance behind car, they are further away from the camera.
Therefore, Point C the closest to the camera.
C. Point C lies closest
Reasoning:
1. Point A in the sky, far the background, and is part of the distant horizon and clouds.
2. Point B on the van, which is further down the road and not the closest object to the camera.
3. Point C the dirt road, very close the bottom edge of the image, indicating is
the
4. Point also the dirt road but is further away from camera compared to Point
Therefore, Point C the closest to the camera.
To solve this task, let's analyze the image step by step.
1. The image shows driving dirt road desert-like environment.
There four points labeled A, B, C, and in the image.
side car, point is on the right side of the car, and point is on the ground, slightly behind the car.
Point is located in the sky, point is on
4. The car appears to be closest point B, as it the only point that is directly in line with the
Based on this analysis, the answer Point lies closest to camera.
Figure 107: A sample case of physical object relationships-deye depth. Back to List of Figures
140

---

Published as a conference paper at ICLR 2025
J DISCUSSION AND STATEMENT
J.1 LIMITATION
Portions of our data constructed based on pre-existing datasets, as detailed in Table 13 We have made every effort to ensure that the images included in this paper comply with applicable copyright laws and are appropriately credited. Should you be the copyright holder of any image used in our work and believe that its usage conflicts with your licensing agreements, don't hesitate to get in touch with us directly. We are committed to promptly addressing any legitimate concerns.
Although PhysBench is categorized into 4 major categories and 19 subcategories, making it the first benchmark aimed at evaluating a vision-language model's understanding of the physical world, still does not encompass all aspects of the real physical environment. We invested thousands of hours in data collection, organization, and annotation, following multiple rigorous processes and repeated reviews to ensure data quality. Nevertheless, minor issues may persist in small portion of the dataset.
In the experiments described in Section 3.4 and we utilized GPT-4o-mini to extract answers, which, while currently being one of the most reliable and reproducible methods, may still unavoid- ably introduce hallucinations. Considering the challenges of using LLMs for evaluation, such as hallucinations and knowledge limitations, assessing open-ended formats is particularly difficult (Yu al. 2023b; Ge al., 2024: Li et al., 2023b). Moreover, automatically evaluating the quality of reasoning processes presents additional challenges (Lu et al., (024b). To address these difficul- ties and simplify testing, we adopted multiple-choice format. Nevertheless, PhysBench remains highly challenging, with even the most advanced GPT-4o model achieving less than 50% accuracy. We believe that exploring more complex evaluation formats a crucial direction for future research.
These challenges are not unique to our dataset but are common across various datasets in VLMs. Nevertheless, we believe that the potential benefits outweigh the associated risks, promoting contin- ued advancement and societal progress. To the best of our knowledge, PhysBench and Phys.gent represent the first effort aimed at benchmarking and enhancing VL.Ms for physical world under- standing, marking significant step forward in the field. We are committed to the ongoing refine- ment of our dataset to further improve machine intelligence's comprehension of the physical world, ad advancing embodied AI toward human-level capabilities.
J.2 BOARDER IMPACT
The broader impact of PhysBench and PhysAgent carries both potential benefits and risks upon deployment and release. While some considerations pertain specifically to the nature of the dataset, others reflect broader challenges inherent to instruction-followiii vision-language models (VI.Ms). Below, we outline key risks and corresponding mitigation strategies.
Biases. Phys/ggett may inherit biases from its foundational models, both in vision and language foundation models. These biases can manifest in skewed outcomes or unfair representations, neces- sitating careful evaluation and mitigation efforts.
Anticipated Societal Implications. major societal concern the potential misuse of the dataset and system, including the generation of fabricated content, which may contribute to misinformation, privacy infringements, and other detrimental outcomes. To mitigate these risks, strict adherence to ethical guidelines and ongoing oversight essential.
Environmental Considerations. In alignment with environmental sustainability goals, we commit to publicly releasing the dataset and scripts reduce unnecessary carbon emissions by regenerating similar datasets. Throughout our experiments, we ensure compliance with model and data licensing requirements.
141

---

Published as a conference paper at ICLR 2025
J.3 ETHICS STATEMENT
This study does not raise any ethical concerns, as it exclusively utilizes publicly available pre- existing models, with involvement of subjective evaluations. All research presented in this paper strictly adheres to the ethical guidelines set forth by the ICLR Code of Ethics.
J.4 REPRODUCIBILITY STATEMENT
We have adhered to the standard baseline settings employed by existing evaluation benchmarks or the original testing benchmarks of specific models. All necessary implementation details of our method are provided in Appendix B and F Furthermore. we are committed to releasing the data and code under an open-access license, accompanied by comprchensive instructions to ensure the accu- rate reproduction the primary experimental results presented in this paper. All research conducted complies fully with the ICLR Reproducibility Requirements.
K LATEST RESULTS
The models listed in Table are current of August 2024. Given the rapid evolution of VLMs, we evaluated an additional 36 models in December 2024, as shown in Table 28 A description of these newly added models and their hyperparameters, along with those of the original 39 models, can be found in Appendix [E.3 Due to space limitations, we have not combined the two tables, but consolidated version available on our project page at Our Project Page update the results to reflect the latest advancements in VLMs.
We will continue to
In Figure a(), the largest model size tested 13B. In this experiment, we have updated to larger models to evaluate whether increasing the model size significantly impacts performance on Phys- Bench. Based on the new models added in Table we present Figure [08 below.
The results in Figure 108 that models ranging from IB to larger sizes generally show improve- ments, while the performance between and 13B does not exhibit significant gains, and in some cases, such as with VII.A and PLLaVA, a decrease in performance is observed. This observation constitutes the primary area of interest in Figure B(a). Notably, at larger scales, particularly at 26B and 40B, we observe substantial improvements compared to previous sizes. However, the scalabil- ity patterns in PhysBench differ from traditional VQA benchmarks, which typically demonstrate a strong positive correlation between model size and performance. We selected widely-used bench- marks including TextVQA (Singh et al. 2019). Math Vista eLu al, 2024b), and MMMU (Yue et al.] 2024 for comparison. The relationship between model size and performance these benchmarks is illustrated in Figures [11
112 and T13 which demonstrate this consistent scaling pattern.
In contrast, PhysBench exhibits less pronounced scalability compared to these benchmarks, with performance not always correlating positively with model size, as shown in Figures [08] [109]
and I10 This phenomenon is particularly evident the Scene subcategory of PhysBench, where performance remains relatively stagnant in the 5-20B parameter range, only showing notable im- provements with models exceeding 25B parameters, as demonstrated Figure I09 I09
142

---

Published as a conference paper at ICLR 2025
PhysBench Average Accuracy
DeepSeek-VL
VILA-1.5
26
MiniCPM
LLaVA-1.5
MolmoE
InternLL2
InternVL2.5
10
30
90
70
P Paramefers (B)
Figure 108: The performance of models of different sizes on PhysBench.
40.0
37.6
PhysBerch Sc-mSeeee Average Accuracy
28.0
32.5
DeepSeek-VL
VILA-1.5
MiniCPM
LLaVA-1.5
22.
MolmoE
InternVL2
PLLaVA
InternVL2.5
20.0
Parameters (B)
Figure 109: The performance of models of different sizes on PhysBench-SS Scenee.
PhysBench- Dynamics Averaga Accuracy
DeapSeek-VL
VILA-1.5
MiiiCPM
LLaVA-1.5
34
MolmoE
InterVL2
PLLaVA
InterVL2.5
Parameters (B)
Figure 110: The performance of models of different sizes DeabeemcchDDDamiss...
143

---

Published as a conference paper at ICLR 2025
Text Average Accuracy
80
78
MinICPM MolmoE
InternVL2.5
10
30
60
70
P Parameters (B)
Figure 111: The performance of models of different sizes on TextVQA Singh et al. 2019
MathVista-nin Avarage Accuracy
DeepSock-VL
LLaVA-1.5
MiniCPM
InternLL2
MolmoE
InternVL25
VILA-1.5
(B)
Figure 112: The performance models of different sizes on Math Vista (Lu et 2024b).
MMMMU-ts Average Accuracy
50
DeepSeek-V VL
LLaVA-1.5 1.5 (val)
MinICPM (val)
IntemVL2
MolmoE (val)
IntemVL2.5
VILA-1.5
Parameters (B)
Figure 113: The performance of models of different sizes on MMMU
Yue et al.
2024
144

---

Published as a conference paper at ICLR 2025
Size
Format Propetty
Relationships
Scene
Dynamics
Avg
Image VLM
MolmoE-IB Deitke 2024
IB
merge
43.17
58.06
31.12
41.58
40.63
MolmoE-7B Detke
7B
merge
42.35
63.33
28.86
42.22
40.41
MolmoE-7B-D Dettke 20,2212 2124
7B
merge
49.48
67.78
30.81
47.18
45.18
MolmoE-72B Dettke
72B
merge
55.13
72.08
36.61
46.20
48.67
MinICPM2 Yao
2024
3B
merge
45.62
44.17
27.53
39.92
38,66
MiniCPM2.5 Yao 2024
4B
merge
44.26
47.08
25.68
37.95
37.35
MiniCPM2.6 Yao
20241
8B
merge
51.43
67.92
29.44
45.44
44.89
Xinyuan-VL
Group
2024
2B
merge
47.97
$7.50
33.91
42.22
43.09
Aquila-VI. (Gü
2024
2B
merge
47.04
62.50
30.15
45.49
43.22
DeepSeek- VL.Beekk-VV-1155 [. [.
2024a
IB
merge
38.37
40.28
31.65
38.12
36.58
DeepSeck-
[u al.
20248
7B
merge
43.79
55.97
30.10
38.71
39.46
PaliGemma2-38 (Stener
2024
3B
merge
22.03
23.33
24.75
31.73
25.94
PaliGemma2-10FF Steiner et
2024
10B
merge
35.31
41.53
28.73
39.69
35.40
General VLM Interleaved data
Phi-3.5V [AzureML 2024
4B
seq
45.72
40.15
33.02
39.40
39.75
NVILA-8B Liu at 2024f
8B
seq
55.79
40.29
33.95
43.43
43.82
NVILA-ISB Liu 20241
15B
seq
59.16
42.34
38.78
45.72
46.91
NVII.A-L.tt-SBt
2024f
8B
seq
53.81
39.25
34.62
41.17
42.55
NVILA-LLite- 15B
[Lu
20241
15B
seq
55.44
40.15
38.11
44.38
44,93
mPLUG-OWI3-IBB (Y
2024
IB
38.02
31.54
21.87
33.68
31.68
mPL.UG-OW13-2B Yt al.
2024
2B
40.92
35.11
26.69
35.64
34.87
mPLJUG-OWI3377 et
2024
7B
49.25
45.62
35.90
40.61
42.83
IntemVL2-1BB (Wang
20246
1B
37.05
33.06
22.84
34.92
32.35
VL2e2 Wang
2024c
2B
44.17
35.06
30.54
35.64
36.57
IntemVL2-4B (Wang al
20242
4B
47.12
39.96
30.94
39.76
39.71
IntemVL2-88 8B Wang
2024e
8B
49.05
43.58
27.05
39.47
40.00
IntemV1.2-261 [Wang et al.
21124e
26B
merge
51.92
45.20
37.94
39.34
43.50
IntemV1.2-40B (Wang et 21/242
40B
merge
55.79
50.05
35.86
41.33
45.66
IntemVLL2-76B Wann el
2024e
76B
merge
57.65
52.43
38.07
40.12
46.77
IntemVL.2.5-BB (Gao ci al.
2024bb
1B
seq
44.25
33.30
26.87
38.13
36.15
IntemV1.2.5-2B (Gao al
2024h
2B
49.63
38.15
29.44
38.39
39.22
IntemVL2.554B
(Gao cl al. 2024b
4B
seq
51.03
44.77
31.34
41.79
42.44
IntemVL22.58BB
(Gao al. 2024b
8B
seq
55.87
48.67
29.35
41.20
43.88
IntemVL22.52555 GGo et al 20244
26B
merge
59.08
58.33
36.61
41.79
48.56
IntemVL2.55338 GGo cl al 2024b
38B
merge
58.77
67.51
39.04
45.00
51.94
Intern Vitem/L22.57775 (Gao ct al 2024b
78B
merge
60.32
62.13
37.32
46.11
51.16
o1 (Jacch et 20244
merge
35.27
73.79
40.95
49222
35.11
Evaluation results for latest 36 VLMs. The evaluation of General VLMs is based on the data from Video and Image VLM evaluations, with the addition of interleaved data. "Seq" refers to sequential input frames of videos, while "merge" refers to merging video frames into a single image. Results of original 39 VLMs can be seen in Table
Table 28:
145

---

Published as a conference paper at ICLR 2025
Edentily Comparbon
Perception
Judgment
Reasoning
Arg
Image
VLM
40.53
MolmoB-78
Daike ara 2044
78
54.84
32.49
72.01
37.75
49.85
39.22
45.01
25. 25.20
40.41
MolmoF-7R-1
78
07.04
35.63
74.44
49.67
54.74
43.82
49.00
27.19
45.18
MolmoB-728
Denke sl
2024
72B
44.51
58.94
54.33
43.13
50.71
33.00
MiniCPM2
38
62.41
32.49
45.71
38.41
47.40
36.33
44.86
24.12
38, 66
MiniCPM2.5
MiniCPM2.6
Yāo ataeral
20244
8B
65.82
40.07
72.20
56.95
$2.50
42.03
48.57
26.01
44.89
Xinyuan-VI.
20754
28
63.12
35.77
63.43
41.06
49.34
39.56
45,30
30,86
43.09
Aquila-VL.
et x
2B
62.51
35.02
66.04
$1.66
50.97
41.00
46.72
28.
43.22
DeepSeck
VL-
20244
1B
48.82
29.90
40.49
43.05
41.59
35.78
40.17
30.47
36.58
DeepSeek
PaliGemma2-33 3B
Steiner e1
2024
3B
18.31
25.19
23.69
21.19
24.57
37.29
20.51
25.20
25.94
PaliGenma2--1
26.59
General VLM
Interleaaedd data
Phi-3.5-Vision--ie (AzurcML.
48
35.84
30.64
39.75
NVILA-8B Liu all, 212441
8B
70.71
44.10
38.05
64.71
$5.76
44.18
34,02
30.45
43.82
NVILA-15B
pusoli
15B
75.85
46.28
40.03
64.71
57.70
48.54
33.12
35.42
46.91
NVILA-Lite-S5.
41.77
53.82
41.18
33,12
31.16
42.55
NVILA-Lite-15 1515
15B
68. $8.09
45.60
38.31
$8.17
59.53
45.41
31.46
34.71
44.93
mPLUGG-WW3-1BB
46.56
31,26
25,83
18.53
mPI.IGG-OW
Yereral 2112
2B
54.23
30.65
34.24
41.83
40.77
37. 58
31,46
23.80
34.87
mPLUG-OW13-JB
7B
65,
36.59
44.68
56.21
53.62
41.59
30.69
32,50
42.83
VLternVL.2-1B
Waang 2074c
IntemVV12.2BB
Wany II
21744
2B
N0.77
31.19
34,24
43.79
43.12
35.47
33,25
27. 62
36.57
VLtemVLL2-4B
Wang
IntemV112-8B
Whng d5 al
21744
8B
64.95
36.79
42.28
56.86
$2.19
39.96
32.35
22, 40
40.00
Inter VL.2-
Wang
21244
26B
66.35
40.55
44.02
58.56
$2.31
47.11
38.18
32,18
44.18
IntemVLL-100B
Wang 01
20299
40B
69.83
44.91
49.01
60.13
$5.66
43.50
28.13
32.17
V.227VV1.2766B1
Whlls B0 31
202242
76B
71.23
47.37
51.10
64.05
$2.91
40.16
13.38
34.47
46.77
Intem\V22.5-1B
IntcmnVL2.5-28
Uato
20240
2B
68.35
35.02
37.58
42.48
47.50
39.28
29.92
27.00
39.22
Inter LL2.5-4
4B
66.52
38.84
43.37
60.78
54.43
43.02
30.69
27.91
42.44
LntrrVL2..-8BB
GB0
20240
8B
72.62
43.00
47.39
60.75
$1.58
42.82
31.33
26.09
43.88
Intern LL2.5VL...5-66B
L20
A37440
268
75.94
45.94
58.04
60.78
55.15
40.84
33.89
33.60
48.56
InternVL.2.5222
(Ga0
21
20244b
38B
78.47
46.21
61.17
71.90
60.35
42.61
39.77
34,90
51.16
Jacch
Table 29: Evaluation Results for 36 new VLMs Categorized by Ability Dimensions. PhysBench includes two evaluation dimensions: ability and task. Table 28 presents results categorized by the task-type dimension.
146

---

Published as a conference paper at ICLR 2025
Model
Size Number
Mass
Color
Attribute
Image VLM
MolmoF:1IB Deitke et al.
2024
IB
56.85
23.89
52.67
39.39
MolmoE-7B Deitke et al. 2024
48.01
30.03
56.67
39.46
MolmoE- Deitke et al. [2024
7B
65.16
36.52
63.33
42.73
MolmoE-72B Dettke et al. 2024
72B
63.95
51.19
64.00
50.25
MiniCPM2 Yao
2024
3B
56.52
42.66
66.56
37.26
MiniCPM2.5 Yao
2024
46.26
50.17
45.82
41.80
MiniCPM2.6 Yao ct al.
2024
8B 54.09
58.70
70.23
44.71
Xinyuan-VIL VT. (Group 2024
2B
54.25
47.44
71.67
40.38
Aquila-VL Gu al. 2024
2B
55.63
43.69
64.33
40.38
DeepSeek-VL VL- (Lu et al.
2024a
1B
48.18
32.76
49.33
33.29
DeepSeek-VL-777 VL-7B (Lu et al.
2024a
7B
51.99
39.59
52.33
39,46
PaliGemma2-3B (Stemer
2024
3B
20.80
17.75
17.67
24.34
PaliGemma2- (Stemer et al.
20244
10B
40.03
29.01
45.67
32,51
General
VLM
Interleaved data
Phi-3.5- Vision-Insttiice (AzureML.
2024
49.05
44.37
65.00
40.45
NVILA-8B (Liu et al. 20241
8B
61.35
57.34
75.67
48.90
NVILA-15B [Lu et al. 20241
15B
774.18
55.29
73.67
50.53
NVILA-Lite-SH (Liu al.
2024f
8B
60.14
59.04
75.00
45.42
NVILA-Litt-15BB (Lau cl al.
20241
15B
61.18
55.97
72.33
49.25
mPLUG-OWI3-IB YC cf al. 2024
1B
37.95
44.37
50.33
34.07
mPLUG-OW13-2B
Ye al. 2024
2B 44.19
37.54
61.00
35.84
mPLUG-OW13-7B
al..
7B 57.02
36.52
71.67
43.79
InternVL.--IB Wang el al.
2024e
IB
38.99
34.47
61.67
31.44
IntemVVL--2B
Wang al. 2024g
2B 57.89
34.47
69.33
35.13
IntemBL2--4B Wang al. 2024c
4B
49.39
39.93
71.00
42.51
IntemVL2-8B Wang ci al.
2024c
8B
58.75
36.18
74.67
42.09
IntemVL2-26B Wang ct al.
2024C
26B
57.19
44.71
68.00
47.69
InternVL2--0B
Wang
2024e
40B
64.30
55.29
67.00
49.82
InterVVL2-76B
Wang
2024e
76B
67.76
60.41
64.67
51.24
InternVL.2.5-IB (Gao et al.
2024b
IB
58.58
35.49
71.00
34.49
(Gao ct al. LL2cm2L2.552B
2024b
2B
63.95
42.32
72.67
40.38
LLtcmmV22.544B
2024b
4B
56.15
$2.90
75.00
43.29
Intern
Goao ct al.
2024b
8B
67.94
50.85
75.33
47.69
Intern (I.2.5V1...5-261
[Gao al.
2024b
26B
73.31
57.00
72.67
50.67
VI2ernVL.2.5-38BB
(Cao
2024b
38B
68.11
51.88
73.00
53.16
Gao al.
2024b
38B
5.22
55.63
73.67
52.24
o1
TTech al., 2024
75.22
49.83
74.67
31.31
Table 30: Evaluation Results for 36 new VLMs in PhysBench Physical Object #Property Sub-task (the fifth last column of Table 28
147

---

Published as a conference paper at ICLR 2025
Model
Model Size
Size
Location
Depth Distance Motion
Image VLM
MolmoF- IB Deitke et al.
2024
BB
39.29
51.13
72.28
48.33
50.56
MolmoB-715 Dettke al. 2024
42.86
61.65
75.44
53.33
50.56
MolmoB-7B-DD Deitke Deitke et al. (2024
7B
58.93
60.15
79.30
66.67
59,55
MolmoB-72BB Deitke et al. 2024
72B
76.79
67.29
76.49
68.33
74.16
MiniCPM2 Yao el 2024
3B
50.00
45.49
39.65
46.67
59.55
MiniCPM2.5 Yao cl ml.
20224
66.07
36.84
51.23
43.33
67.42
MiniCPM2.6 Yao ct al.
2024
8B
71.43
67.67
68.77
56.67
73.03
Xinyuan-VL VL Group 2024
2B
55.36
54.14
62.46
50.00
61.80
Aquila-VL (uu
2024
69.64
60.90
64.56
61.67
59.55
DeepSeek- VL-Bee VL-1B
(Lu et al.
2024a
1B
41.07
43.23
31.93
50.00
51.69
DeepSeck-VI--77 (Lu al.
2024a
7B
37.50
51.13
61.40
61.67
59.55
PaliGemma2-3B (Stemne
2024
23.21
21.05
27.02
26.67
13.48
PaliGemma2- 10B (Stemer ct al.
2024
1OB
48.21
45.11
36.14
41.67
50.56
General VLM Interleaved data
Phi-3.5 Iiston-nnstrucct (AzureML)
2024
57.14
59.45
66.67
55.00
30.36
NVILA-8B (Liu et al.
20241
8B
76.79
67.35
70.88
73.33
26.83
NVILA-15B LLu et al.
2024f
15B
76.79
69.07
80.35
75.00
27.59
NVILA-Lite-BB Liu al.
2024f
73.21
61.86
71.23
60.00
27.18
NVILA-Lite-15B Lau cl al
20241
15B
62.50
67.35
73.33
70.00
26.97
mPLUG-OWI3-IBB YC ct al.
2024
1B 46.43
39.52
31.93
35.00
29.46
mPLUG-OWI3-2B (Ye
2024
2B 60.71
41.58
30.88
43.33
34.09
mPLUG-OWI3-7B (Ye et al..
69.64
60.14
69.82
58.33
37.34
IntemVL2-1B (Wang et al.
2024e
IB 44.64
53.26
35.79
50.00
27.66
InternVL2-2B Wang
2024c
2B 78.57
64.95
73.68
63.33
34.30
IntemVLL2-4B Wang al.
2024c
69.64
59.11
75.44
55.00
28.08
VLtcmB (Wang ct al.
2024c
75.00
67.01
77.89
56.67
31.05
VL2-26 (Wagg et al.
2024e
26B
78.57
64.95
73.68
63.33
34.30
InternVLL2-00B Wang
2024e
40B
80.36
64.26
77.54
73.33
40.18
IntemVVL2-76 (Wang et al.
2024e
80.36
67.01
82.81
76.67
41.91
IntemVL22.51BB (Gao ct al.
2024b
1B 48.21
53.26
47.72
50.00
25.73
Intern VL2.522 (Gao ct al.
2024b
2B
48.21
57.73
68.07
50.00
28.08
IntemVL2..544
(Gao ct al.
2024b
75.00
62.54
72.63
55.00
34.92
Intern V (Gao ct al.
2024b
8B
67.86
72.16
78.25
75.00
37.00
Intern 21.2.5-26B (Gao
2024b
26B
75.00
65.64
79.65
76.67
51.66
Interm
Cia
2024b
38B
85.71
68.73
75.79
80.00
64.59
IntemmVL...5788B (iao al.
2024b
80.36
69.42
77.54
86.67
56.43
ol (Jachh et al. 2024
83.93
76.63
78.25
"80.00
71.31
Table 31: Evaluation Results for 36 new VLMs in PhysBench Object RRelationships Sub-task (the
forth last column of Table 28).
148

---

Published as a conference paper at ICLR 2025
Model
Size Temperature Viewpoint
Air
Light
Image VLM
MolmoE- 1B Deitke et al.
2024)
1B
50.00
30.73
26.00
30.2T
MolmoE-7B Deitke
2024)
7B
64.71
21.11
36.00
33.76
Molmoe--7B-DDDeltke el al.
(2024)
7B
79.41
42.70
86.00
26.84
MolmoE-72B Deitke et (2024)
72B
76.79
67.29
76.49
68.33
MiniCPM2 Yao et al. 2024
3B
66.18
20.83
40.00
31.12
MiniCPM2.5 Yao
Yao et al. 2024
4B
48.53
13.85
42.00
35.21
MiniCPM2.6 Yao et al. 2024)
8B
67.65
19.51
42.00
36.21
Xinyuan-VL. Group
2024)
2B
64.71
33.46
48.00
32.21
Aquila-VL (uu al.
2024)
2B
61.76
24.79
28.00
33.48
DeepSeek-VL-111 [Lu al.
2024a
1B
50.00
32.33
32.00
29.75
DeepSeek-VLL VL-7B (Lu al.
2024a
7B
58.82
25.64
28.00
32.85
PaliGemma2-33 Stemer et al.
2024)
3B
17.65
26.20
28.00
23.66
PaliGemma2-108 (Steiner et al.
20224)
10B
48.53
27.14
40.00
28.57
General V VLM Interleaved data
11-3.S-Vision-IIstruee (AzureML
2024
4B
51.47
33.65
50.91
30.85
NVILA-8B (Liu et al.
20241)
8B
72.06
31.29
50.91
33.58
NVILA-15B (Luu et al.
2024f
15B
83.82
38.36
52.73
35.85
NVILA-Lite-8B (iu et
2024f
8B
76.47
31.39
25.45
35.30
NVILA--Lite-15B (Lm et al.
2024f
15B
82.35
38.55
61.82
34.03
mPLUGGOOW1311B Ye et al.
2024
1B
57.35
10.84
50.91
29.48
mPLUG-OW13-2B Ye et al.
2024
2B
63.24
25.35
29.09
25.75
mPLUG-OW13-7B Ye et
2024
7B
76.47
37.51
78.18
30.66
InternLL--IB (Wang et al.
2024e
IB
45.59
18.47
18.18
25.75
InternVL.2-2B (Wang al. 2024e
2B
50.00
28.46
21.82
31.48
InterVL2-44B WWan et al. 2024e
4B
52.94
28.37
54.55
31.39
InternVL2--8B (Wang et al. 2024e
8B
73.53
17.91
36.36
32.67
InternVLL2-6BB (Wang et al. 2024e
26B
76.47
40.25
29.09
33.30
IntemVL2240B [Wang ct al.
2024c
40B
94.12
37.89
58.18
29.39
IntemVV2-77B [Wang et al.
2024c
76B
91.18
39.87
52.73
32.58
InternVL.2.5-1B (Gao et al.
2024b)
1B
52.94
20.17
14.55
32.03
VL.252V (Gao et al.
2024b
2B
58.82
21.58
16.36
35.58
Intern
(Gio et al.
2024b
4B
67.65
28.09
29.09
32.30
IL2.55VL2.5-BB (Gao
2024b)
8B
69.12
18.94
47.27
36.49
(Gao et a
2024b
26B
88.24
34.31
74.55
34.58
Intern VL2.53L2..5338
Gao al.
2024b
38B
88.24
41.47
78.18
32.48
Intern VL2
Gao et al.
2024b
38B
89.71
40.62
90.91
29.30
o1 (Jaech et al. 2024
95.59
47.31
88.82
30.30
Table 32: Evaluation Results for 36 new VLMs in PhysBench Physical Scene Understanding Sub-
task (the third last column of Table 28
149

---

Published as a conference paper at ICLR 2025
Model
Size
Collision Throwing Manipulation Fluid Chemistry Others
Image
VLM
MolmoE- 1B Deiike et al.
2024
IB
32.72
33.02
43.47
44.75
71.62
$7.48
MolmoB-7BB Detike
cl Deilke
2024
7B
28. 26
28.74
36.68
53.55
67.57
67.01
2024
38.56
39.43
39.45
50.15
68.92
72.11
MolmoB-72B
37.02
40.38
30.90
48.30
63.51
80.61
MiniCPM2 Yau 2024
3B
34.82
34,20
34,92
34.88
67.57
67,6 67,69
MiniCPM2.5 (ao Yao
4024
4B
433.44
36.82
27.89
35.03
54,05
60.88
MiniCPM2.6 Yao
AUP-3
8B
35.29
45.37
38.19
43.06
64.86
76.19
Xinyuan- VL (Group
2024
2B
35.94
39.67
38.44
39.81
50.00
64.63
Aquila-VL. (Gil al. 2024
38.10
40.14
42.21
42.75
59.46
75.17
DeepSeek- VT.-1B
2024a
32.10
33.49
38.44
39.35
48.65
$2.04
DeepSeek- VL-7B
DeepSeck-VL-77B (Lu
31.64
35,39
35.43
37.04
54.05
61,90
PaliGemma2-333 Steiner ot al.
2024
3B
37.63
34.68
25.13
34.57
36.49
18.03
PaliGemma2- 108
(Semmer of 2024
10B
37.17
40.62
33.92
35.80
56.76
$4.08
General
VLM
Intericaved data
(AzurcML.)
2024
4B
33.94
38.72
29.19
43.67
59.46
58.58
NVILA-8B (Liu et al. 20241
38.76
45.13
31.24
43.21
70.27
69.75
NVIIA--15B Juue
15B
40.42
43.23
29.31
54.17
77.03
71.93
NVILA-Lite- 8B
2024F
38.31
46.08
28.59
38.43
65.40
NVILA-Lite- 15B 1
20241
15B
41.48
40.38
31.60
45.83
70.27
72.21
mPLJGG-WWI--IB
ela..
2024
1B
29.41
32.07
23.16
42.59
51.35
45.78
mPLUG-OWI1-22B
Yo al.
2024
2B
34.69
35.63
24.85
38.58
56.76
50.14
mPLUG-Owl3-7B Yo al.
2024
37.10
36.34
28.35
43.06
67.57
64.03
IntemVV1.2-B (Wang
31.83
29.22
30.16
41.82
48.65
43.32
Lntem Wane
37.10
32.54
29.43
35.65
50.00
47,96
V12-4V (Wang
20240
4B
35.29
41.09
27.50
47.53
63.51
52.04
V1.288 (Wang el a. 20248
8B
38.46
38.95
29.19
38.43
58.11
61.31
IntemLL2-200 [Wang ct al 2024c
26B
38.91
41.09
26.42
37.
58.11
64.58
IntemVL2-400 (Wang ct al. 2024c
40B
43.14
47.51
20.39
39. 20
67.57
74.39
VL2e77L Wane et
2024e
40.27
42.99
64.86
74.93
Intem LL2.5-1B
Gaa
20246
40.27
37.05
26.42
43.36
52.70
49.86
VL2.5-28
(Ga0 ct al. 2024b
2B
33.48
40.38
28.11
42.13
62.16
55.86
Intern VL255-4B
(Gao el al.
2024b
4B
38.76
42.52
28.83
45.37
60.81
63.49
Intem VL2.5-L2.588
(GB0 ct al. 20240
8B
39.52
36.82
24.25
46.14
67.57
70.03
IntemVL2..52660
[Gao a.
2024b
26B
38.16
43.23
25.69
38.58
67.57
78.75
VL2.5-888
Gao et
2024b
43.89
41.81
78.38
83,92
VL2,5782
20246
38B
42.08
46.32
33.53
37.35
75.68
85.29
01 (Jaech al.
2024
48.27
41.09
337.05
43.52
T..62
8932
Table 33: Evaluation Results for 36 new VLMSs in PhysBench Physics-based A Dynamics Sub-task
(the second last column of Table 28
150


Reviews:
[{"review_id": "1P5ZXUpkV0", "reviewer": "Reviewer_khH4", "summary": "The paper introduces a comprehensive benchmark for evaluating VLMs for physical world understanding. The authors identify four major dimensions comprising of Object Property, Object Relationships, Scene Understanding, and Dynamics and create 100k entries covering these categories. Further, the paper introduces PhysAgent as a solution to aid VLMs in gaining physical world understanding which improves performance for Embodied AI applications as well.", "questions": "- Is the PhysAgent using the DepthAnything, SAM and DINO outputs as input to LLMs or these individual model's outputs are used to get a textual insight which is the sole input to VLMs?\n- Please refer to weakness section. I am open for discussion and changing my score.", "limitations": "", "rating": 8, "confidence": 4, "soundness": 3, "presentation": 4, "contribution": 3, "strengths": "- Paper identifies key issue of missing physical world understanding in VLMs and release a comprehensive benchmark for evaluation and training (Table-1).\n- Fig.4 experiment explains the difference of physical world understanding against the common VQA benchmarks which is very insightful.\n- The method PhysAgent reuses existing foundation models and gains a shows a significant performance boost on the benchmark.\n- The cross-task ability of PhysBench for embodied AI makes the benchmark very useful.", "weaknesses": "- The test set of PhysBench is derived from the similar set of images as train. This ensures that any fine-tuning would result in performance improvement. Did the authors test PhysAgent against out of distribution images that are labeled for physical undersatnding?\n- L443: 'PhysAgent first classifies the question and activaets task-specific prompts'. Does that mean that PhysAgent is dataset dependent? Also, how does it support the claim made at L90 \"PhysAgent retrains the strong generalization abilities of VLMs and their capacity to solve open-ended problems\"? It would be good to see an experiment supporting the L90 claim in the paper.\n- Nit: The appendix should have been kept concise. I appreciate the details but a concise appendix makes it more approachable.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "xZrlCKgSl3", "reviewer": "Reviewer_52XN", "summary": "The authors collect a human-annotated dataset to test physical reasoning in MLMs. They demonstrate that MLMs struggle to reason about physics in images by showing a gap between human and MLM performance. They further collect annotations to analyze why MLMs fail and identify perceptual errors and a lack of knowledge as the main reasons why they fail. Then, they incorporate some specialist vision models to infer depth, objects, etc., to further help the MLMs and show improved performance. They finally show that fine-tuning on their benchmark also helps an object manipulation task in simulation.", "questions": "See weaknesses. \n\nI like the direction of the paper and believe the benchmark would be impactful. However, the details presented are not clear. Hence, I will be willing to increase my score if the authors can clarify my doubts above.", "limitations": "", "rating": 8, "confidence": 4, "soundness": 3, "presentation": 2, "contribution": 3, "strengths": "- The topic is very relevant and interesting since numerous works also show that MLMs lack spatial and physical reasoning and these are crucial skills for embodied AI. \n- Their dataset seems varied and tests a lot of fundamental physical skills. \n- The annotation of why MLMs are failing is interesting. \n- The fact that PhysBench fine-tuning helps an object manipulation task is a good way to show that physical reasoning helps downstream applications.", "weaknesses": "- I am a bit confused about the correlation matrix in Figure 4a.  I am assuming the last 4 rows are PhysBench splits? In that case, the bottom right half seems to indicate that that performance on their benchmark doesn't correlate on the other existing VLM benchmarks (since the left part of the bottom 4 rows are blue). This can imply improving on their benchmark doesn't improve on other vision-language benchmarks? I am also not sure what part of that figure to look at for their claim in Section 3,4 that their benchmark is correlated to MMMU. \n\n- More details on the PhysAgent would have been helpful. What is the base model for PhysAgent - is it an MLM? If so, which one? They outline a 3-step pipeline:\n    1. asking task-specific prompts: How does it learn to ask these questions? Is it tuned to do so or in-context? If so, how is the tuning done? \n    2. Incorporating VLM predictions: How exactly do they incorporate depth, grounding etc? The figure 8 and supplementary makes it seem like a  JSON-like format or a list. Where does the knowledge memory come from? Can the PhysAgent interpret such a format without any fine-tuning?\n    3. How is PhysAgent prompted to do COT? Does it do it zero-shot or, is it taught to do the chain of thought using some training examples? If so, how are the COT training examples made?\n\n- The results in Table 9 are confusing - what the individual columns? They do not match the names with the splits in PhysBench in Table 3. PhysAgent improves on PhysBench, but does it at least retain performance on some other similar benchmarks?", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "cl3NuV4KLt", "reviewer": "Reviewer_xxa5", "summary": "This paper proposes a large-scale benchmark to evaluate VLMs' capability to understand the physical world. The benchmark covers diverse domains such as daily life images and videos, robot manipulation, and autonomous driving. The benchmark takes thousands of hours for data annotation and multi-round quality review, ensuring quality. The extensive experiments test 39 representative VLMs and reveal the limitations of such models' abilities to understand the physical world. To enhance VLMs' physical world understanding abilities, the authors propose a unified approach named PhysAgent to leverage visual foundation models into a chain-of-thought protocol. Since there are existing works that aim at evaluating VLMs' ability in spatial reasoning and physical common senses, this paper claims that the benchmark can enhance the VLMs' abilities on embodied tasks. However, the experiments on embodied tasks are very limited and unfair in demonstrating the role of this benchmark in the era of VLA models.\nIn summary, the paper proposes an important benchmark and proposes a large-scale, diverse, and high-quality benchmark to evaluate VLMs' abilities to understand the physical world. The authors also propose a new framework to enhance VLMs' performance on this benchmark.", "questions": "I appreciate the authors' efforts in developing a benchmark for physical world understanding and evaluating its impact on embodied agents. Recently, several open-source VLA models have emerged, such as OpenVLA [1], Embodied-COT [2], and Octo [3]. Did the authors consider comparing the fine-tuned VLMs with other VLA models on the proposed manipulation tasks? Additionally, is there a way to assess the benchmark’s role in advancing the development of VLA models? Such comparisons and evaluations would further highlight the benchmark’s contribution to the field.\n\n[1] OpenVLA: An Open-Source Vision-Language-Action Model\n\n[2] Robotic Control via Embodied Chain-of-Thought Reasoning\n\n[3] Octo: An Open-Source Generalist Robot Policy", "limitations": "", "rating": 8, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "1. Dataset Scale and Quality: The scale and quality of the dataset are promising. The paper provides thorough details regarding data annotation and quality control processes, which enhance the benchmark's credibility and reliability.\n2. Scalability and Error Analysis: The paper offers valuable insights by analyzing the scalability of the model, data, and framework. Notably, it presents an in-depth examination of error types encountered by commercial VLMs on the benchmark, which is critical for advancing the development of VLMs.\n3. Impact of Fine-tuning: The results demonstrate that fine-tuning on a subset of PhysBench significantly improves performance on embodied tasks compared to the zero-shot performance of the PhysAgent framework, highlighting the value of task-specific adaptation.", "weaknesses": "1. Discussion of Related Work: The paper lacks a comprehensive discussion of related works. The proposed benchmark aims to evaluate and enhance the ability of VLMs to understand the physical world at both the object and scene levels. However, it is crucial to emphasize that understanding 3D spatial relationships at the scene level is a fundamental capability for VLMs. To strengthen the discussion, the authors should compare this work with other 3D spatial VQA benchmarks such as SQA3D [1], OpenEQA [2], EmbodiedScan [3], MMScan [4], and MSQA [5]. These benchmarks offer direct evaluations of VLMs’ understanding of 3D scenes through interleaved 2D or 3D representations, providing valuable context and highlighting gaps or overlaps with the proposed approach.\n2. Comparison of Embodied Tasks: The comparison between zero-shot models (such as existing VLMs and PhysAgent) and fine-tuned models trained on PhysBench raises concerns about fairness. PhysBench, in its current form, is relatively simplistic and may not provide a robust or balanced evaluation. To ensure fair comparisons, the authors should either (1) incorporate more challenging tasks or datasets in PhysBench to match the capabilities of fine-tuned models, or (2) highlight the limitations of comparing zero-shot performance with fine-tuned results to avoid misleading conclusions.\n\n[1] SQA3D: Situated Question Answering in 3D Scenes\n\n[2] OpenEQA: Embodied Question Answering in the Era of Foundation Models\n\n[3] EmbodiedScan: A Holistic Multi-Modal 3D Perception Suite Towards Embodied AI\n\n[4] MMScan: A Multi-Modal 3D Scene Dataset with Hierarchical Grounded Language Annotations\n\n[5] Multi-modal Situated Reasoning in 3D Scenes", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "Rm1WzuhNSO", "reviewer": "Reviewer_eQW6", "summary": "The work proposes PhysBench, which serves as a comprehensive benchmark to evaluate VLM's understanding of physics. PhysBench mainly includes four types of physical questions: Property, Relations, Scene, and Dynamics, and each type has many sub-classes, resulting in 19 different tasks. The work also proposes PhysAgent to improve VLM's ability of physical reasoning.", "questions": "- Are there any reasoning steps included in each question?", "limitations": "", "rating": 8, "confidence": 4, "soundness": 4, "presentation": 2, "contribution": 4, "strengths": "- Compared to all previous works on physical reasoning benchmarks, this dataset is the most comprehensive one to my knowledge. PhysBench includes many different types, which can well evaluate VLM's general physical understanding ability.\n\n- The data sources for videos are various, instead of mainly simulation-based generation. This can further diversify the video data if the dataset has been well-cleaned and filtered. \n\n- Experiments are also extensive, which are conducted across 39 VLMs. The authors also well analyze the reasons why VLMs are still struggling with physical reasoning.\n\n- Further experiments on the transferability of real-world tasks are sound.", "weaknesses": "- The article is quite lengthy, so it is really important to improve its logical flow to enhance readability. Using appropriate formatting elements, such as bold text for key points, can help guide readers through the content and make it easier for them to locate the information they are looking for. It is difficult for me to find my questions when reading this paper. \n\n- There is a lack of consistency in the article's formatting. For example, task names are referred to in multiple ways, and similar tables or images are styled differently across various sections, which may confuse readers. Specifically, Table 9a labels the four main types as Property/Spatial/Envir./Phe., whereas other sections refer to them as Property/Relations/Scene/Dynamics. Are they describing the same thing? Also, styles for Tab 16-20 are different, and what is the last column of Tab 20? The labels above Figure 4 are misaligned with the ticks, and extra ticks appear at the bottom. I believe these ticks should likely be placed on the left side instead. It is better to re-organize the whole paper.\n\n- How do questions design for each video? The paper mentions, \"For videos annotated with physical principles, we generate physics-related questions using both manual design and GPT-4o, following predefined rules.\". But I cannot find further details on it. What are the question templates? How do humans design questions? What is the procedure of question generation by GPT-4o? The authors may comment on more details on this matter. \n\n- How is the human evaluation in Tab 3 (or other possible places) conducted? Also, how do human annotators contribute to the dataset? I can only find the GUI description for this matter. The authors may comment on more details, such as how many people are engaging in each part and so on. \n\n- What is the detailed performance on 19 tasks among PhysAgent and baselines of Fig 9a, since the distribution of the dataset is not balanced.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (Oral)