VideoPhy: Evaluating Physical Commonsense for Video Generation
Authors: Hritik Bansal
Zongyu Lin
Tianyi Xie
Zeshun Zong
Michal Yarom
Yonatan Bitton
Chenfanfu Jiang
Yizhou Sun
Kai-Wei Chang
Aditya Grover
Venue: ICLR 2025 Poster (2025)
Keywords: text-to-video generation
physical commonsense
video-text alignment
generative modeling
video evaluation
Primary area: datasets and benchmarks
TL;DR: We propose VideoPhy, a dataset to assess the physical commonsense in the generated videos from a diverse set of T2V models.

Abstract: Recent advances in internet-scale video data pretraining have led to the development of text-to-video generative models that can create high-quality videos across a broad range of visual concepts, synthesize realistic motions and render complex objects. Hence, these generative models have the potential to become general-purpose simulators of the physical world. However, it is unclear how far we are from this goal with the existing text-to-video generative models. To this end, we present VideoPhy, a benchmark designed to assess whether the generated videos follow physical commonsense for real-world activities (e.g. marbles will roll down when placed on a slanted surface). Specifically, we curate diverse prompts that involve interactions between various material types in the physical world (e.g., solid-solid, solid-fluid, fluid-fluid). We then generate videos conditioned on these captions from diverse state-of-the-art text-to-video generative models, including open models (e.g., CogVideoX) and closed models (e.g., Lumiere, Dream Machine). Our human evaluation reveals that the existing models severely lack the ability to generate videos adhering to the given text prompts, while also lack physical commonsense. Specifically, the best performing model, CogVideoX-5B, generates videos that adhere to the caption and physical laws for 39.6% of the instances. VideoPhy thus highlights that the video generative models are far from accurately simulating the physical world. Finally, we propose an auto-evaluator, VideoCon-Physics, to assess the performance reliably for the newly released models. The code is available here: https://github.com/Hritikbansal/videophy.

Published as a conference paper at ICLR 2025
VIDEOPHY: EVALUATING PHYSICAL COMMONSENSE FOR VIDEO GENERATION
Hritik B
Zongyu Lin
Tianyi Xie
Zeshun Zong'
Michal Yarom
Yonatan Bitton Chenfanfu Jiang'
Yizhou Sun'
Kai-Wei Chang'
Aditya Grover
University of California Los Angeles
Google Research
ABSTRACT
Recent advances in internet-scale video data pretraining have led the development of text-to-video generative models that can create high-quality videos across a broad range of visual concepts, synthesize realistic motions and render complex objects. Hence, these generative models have the potential to become general- purpose simulators of the physical world. However, it is unclear how we are from this goal with the existing text-to-video generative models. To this end, we present VIDEOPHY, a benchmark designed to assess whether the generated videos follow physical commonsense for real-world activities (e.g. marbles will roll down when placed on a slanted surface). Specifically, we curate diverse prompts that involve interactions between various material types in the physical world (e.g., solid-solid, solid-fluid, fluid-fluid). We then generate videos conditioned on these captions from diverse state-of-the-art text-to-video generative models, including open models (e.g., VideoX) and closed models (e.g.. Lumiere, Dream Machine). Our human evaluation reveals that the existing models severely lack the ability to generate videos adhering to the given text prompts, while also lack physical commonsense. Specifically, the best performing model, CogVin CogVideoX-5BB generates videos that adhere to the caption and physical laws for 39.6% of the instances. VIDEOPHY thus highlights that the video generative models are far from accurately simulating the physical world. Finally, propose auto-evaluator, to assess the performance reliably for the newly released models. Code: bttps://qittuut com http://........ti.iaaaaaaaaaeeee
Model Performance on VideoPhy Dataset
Model performance on VIDEOPHY dataset using human evalua- tion. We assess the physical commonsense and semantic adherence to the conditioning caption in the generated videos. We find that Cog CogVide can generate videos that fol- low the caption and physics commonsense for 39.6% of the prompts, while the other mod- els far behind (< 20%). This indicates that the existing models severely lack the abil- ity of being general ceneral-purpose physical world simulators.
Figure 1:
CogMieeX-5B
VideoCrafter
28gMlcoo-28
LaViE
Dream Machine (Luma)
Lumiere-T2i2V
SVD
ZeroScope
Lumicre-TZV
Gen-2
SpenSora
5
100
15
20
25
30
35
Accuracy Based Human Evaluation (%)
INTRODUCTION
Recent advancements pretraining on internet-scale video data 3 122
113
11|
24
have led that can Specifically,
to the development of various text-to-video (T2V) generative models such as Sora 68
generate photo-realistic videos conditioned on text prompt [ [10 22 80 98 15
49
these models generate complex scenes (e.g.. "busy street in Japan' and realistic motions (e.g., 'running', 'pouring') making them amenable for understanding and simulating the physical world.
As humans, develop intuitive understanding of the object interactions through experience

---

Published as a conference paper at ICLR 2025
Prompt: Pouring mik inbo
LaViE
Conversation Violation:
laval the
Prompt: Shovaling sand lifo
Bucket
Newton's First Violation: The sand foon aliled wilhout Exce shovel
Prompt: Wood Stals dowe conal
OpenSora
Solld Constlutive Law Violation:
Rigid onjects (wood) should not deform under small force lead.
Illustration of poor physical commonsense by various T2V generative models. Here, we show that the generated videos can violate diverse range of laws pf physics such as conversation of mass, Newton's first law, and solid constitutive laws. In VIDEOPHY, we curate a wide range of prompts that would be used to assess the physical commonsense of the T2V models.
Figure 2:
with the real-world, without any formal education in physics (also termed as intuitive physics) 27 For instance, we can predict the trajectories of the billiard balls after an application of force. Recent efforts [26] 19.0 have further utilized rext-guided video generation to train agents that can act, plan, and solve goals in the real world. In spite of the strong physical motivations of these works, it remains unclear how well the generated videos from T2V models adhere to the laws of physics.
One might be tempted to assess the physical commonsense of generated videos by comparing them with physical simulations as ground truth. However, this is non-trivial, and no similar approaches have been proposed yet. The main challenges include the lack of mature methods to accurately generate 3D geometries from single-view images or video, which essential for physical simulations. Further, physical simulations usually require precise tuning of material parameters based the expertise of graphics researchers to match real-world dynamics. Recently, some efforts have been
made tune simulation parameters from generated videos (e.g., 40 64 130 (51) Nevertheless, they depend on the physical plausibility of the generated videos themselves, which is again the open question we want to address. Finally, the accurate lighting and rendering are also necessary to convert physically simulated results into images and videos, yet these parameters are also unknown. Most importantly, it should be noted that physical simulations are not equivalent ground truth. They are merely numerical solvers of differential equations that attempt to approximate and describe real-world dynamics based on models proposed by researchers. Prior work such as VBench [42 67 introduced comprehensive benchmark evaluate various qualities of generated videos (e.g.. motion smoothness, background consistency) using existing models, but it does not specifically address the generated videos' adherence to physical laws. Therefore, existing benchmarks and metrics either unreliable or lack coverage for holistic evaluation of the physical commonsense capabilities.
To this end, propose VIDEOPHY, dataset designed to evaluate the adherence of generated videos to physical commonsense real-world scenarios. Specifically, we focus on the intuitive understanding of the behavior and dynamics of various states of matter (solids, fluids) in the physical world [26 For instance, 'water pouring into a glass' will intuitively result in the water level in the glass rising over time. As result, we rely on human perception and experience in the physical world to assess the adherence of the generated videos to physical laws instead of precise dynamical equations, which are harder assess. In Figure 2 we provide qualitative examples illustrate physical commonsense violations in the videos. Our dataset is constructed through three-stage pipeline that involves (a) prompting a large language model 78 to generate candidate captions that depict interactions between diverse states of matter (e.g., solid-solid, solid-fluid. fluid-fluidl (b) human verification of the generated captions, and (c) annotating the complexity in rendering objects synthesizing motions described in the captions based physics simulation.
In total, VIDEOPHY comprises 688 high-quality, human-verine verified captions that will be used to generate videos from T2V models. In addition, the dataset consists cal commonsense of the generated videos. Specifically, we acquire generated videos from twelve diverse T2V models including open models (e.g., OpenSora 80) SVD (33, VideoCrafter2 (22
annotations for physi-
2

---

Published as a conference paper at ICLR 2025
Cog VideiX [23] and closed models (e.g., Pika [8 Lumiere []. 2 229 Dream Machine 22.. Subsequently, we perform human evaluation on the generated videos for semantic adherence to the conditioning text (e.g., do the videos follow the caption?) and physical commonsense (e.g., do the videos follow physical laws intuitively?). Interestingly, we find that the existing T2V generative models severely lack the capability to follow caption accurately and generate videos with physical commonsense, Specifically, the best performing model, Cog VideoX-5B, follows the text and gener- ates physically accurate videos for 39.6% of the instances (55). Our fine-grained analysis reveals that current T2V models are particularly poor at generating physically plausible videos for prompts that require solid-solid interaction (e.g., ball bouncing on the floor, hammer hits nail). In Figure [ we compare the performance (i.e., accurate semantic adherence and physical commonsense) of various T2V generative models on the VIDEOPHY dataset. In addition, we perform a detailed qualitative analysis to study the modes of the failures for different models in detail ($5.2 In particular, we observe that the models often struggle to accurately identify individual objects and comprehend their material properties, which is essential for generating physically plausible dynamics. For instance, an object recognized as a rigid body in the physical world should not deform over time.
Although human evaluation of semantic adherence and physical commonsense is reliable, it is both expensive and difficult to scale. To address this challenge, we introduce VIDEOCON-PHY PHYSICS, an open video-language model designed assess the semantic adherence and physical commonsense of generated videos using user queries grounded in text ($6) Specifically, we finetune VIDFOCON [5), robust semantic adherence evaluator for real videos, on generated videos and human annotations collected as a part of our dataset. Our results demonstrate that VIDEOCON-PHYSS Gemini-Pro Pee---ro-iiiiiee Vision-1.5 [90], showing 9 points improvement in semantic adherence and a 15 points improvement in physical commonsense on unseen prompts. Further, we show that VIDEOCON- PHYSICS generalizes unseen generative models, which established its reliability for evaluating future generative models. Overall, the VIDEOPHY dataset aims to bridge the gap in understanding physical commonsense in generated videos and enables scalable testing for upcoming T2V models.
outperforms
VIDEOPHY DATASET
Our dataset, VIDEOPHY, aims to offer robust evaluation benchmark for physical commonsense in video generative models. Specifically, the dataset is curated with guidelines to cover (a) a wide range of daily activities and objects in the physical world (e.g.. rolling objects, pouring liquid into a glass), (b) physical interactions between various material types (e.g.. solid-solid or solid-fluid interactions), and (c) the perceived complexity of rendering objects and motions under graphic simulation. For instance, ketchup, which follows non-newtonian fluid dynamics [14), is harder to model and simulate than water, which follows newtonian fluid dynamics, using traditional fluid simulators [6). Under the collection guidelines, we curate list of text prompts that will be used for conditioning the text-to-video generative models. Specifically, we follow the 3-stage pipeline to create the dataset.
Category
Difficulty
Example Captions
Easy
Bottle topples off the table. (rigid bodies
Solid-Solid
Hard
Scrubber scrubs a dirty dish. ( commlex contacts
Easy
Water flows down a circular drain. ( (contacts with rigid bodies) )
Solid-Fluid
Hard
A swimmer splashing in the sea water. ( (contacts with high-speed )
Easy
Rain splashing on a pond. ( (mixing of same fluids)
Fluid-Fluid
Hard
Ink spreading in still water. (mixing of different fluids
Table 1: Example captions in the VIDEOPHY dataset. Specifically, we design them to depict the interactions between two states of matter (solid-solid, solid-fluid, fluid-fluid). We further classify the captions easy or hard based on the modeling and simulation complexity in the computer graphics. We highlight the reasoning behind the easy and hard annotations by our expert annotators in the
LLM-Generated Captions (Stage 1). Here, we query large language model, in our case GPT-4 78), to generate a list of 1000 candidate captions depicting real- world dynamics. As the majority of real-world dynamics involve solids fluids, we broadly classify those dynamics into three categories:
3

---

Published as a conference paper at ICLR 2025
Table 2: Statistics of the VIDEOPHY dataset.
Statistic
Number
Total captions
688
Unique actions
138
Total T2V models
12
Total generated videos
11330
Human annotations
36500
Category (Interacting materials)
3
Solid-Solid
289
Solid-Fluid
291
Fluid-Fluid
108
Category (Interaction complexity)
2
Easy
366
Figure 3: Top-20 most frequently occurring verbs (in- ner) and their top-4 direct nouns (outer) in captions.
Hard
322
solid-solid interactions, solid-ftuid interactions, and and fluid-fuud interactions. Specifically, we consider fluid dynamics involving in-viscid and viscous represeeresnnttitvee examples being water and honey, respectively. On the other hand, we find that solids exhibit more diverse constitutive models, including but not limited to rigid bodies, elastic materials, sands, metals, and snow. In total, we prompt GPT-4 to generate 500 candidate captions for solid-solid and solid-fluid interactions, and 200 candidate captions for fluid-fluid interactions. We present the GPT-4 prompts in Appendix G
Human Verification (Stage 2). Since --M--eenertted captions may not adhere to our input query, perform human verification step filter bad generations. Specifically, the authors perform human verification to ensure the quality and relevance of the captions, adhering to these criteria: (1) the caption must be clear and understandable: (2) the caption should avoid excessive complexity, such as overly varied objects or too intricate dynamics; and (3) the captions must accurately reflect the intended interaction categories (e.g., that fluids are mentioned in colid-fluid or fluid-fluid dynamics). Finally, we have 688 captions where 289 captions for solid-solid interactions, 291 for solid- interactions, and 108 for fluid-fluid interactions, respectively. We highlight that our prompts include wide range of material types and physical interactions that are common in both real life and the graphics community. Material types include simple rigid bodies [60), deformable bodics 4T. think shells [23), metal [59]. fracture [115], cream [129]. sand [47) and on. The contact handling is also diverse as is based on the interactions of all aforementioned materials [35] [56 36 [33]. We provide more discussion categorization in Appendix [ We highlight that the data quality is paramount for evaluating foundation models. For instance, Winoground (400 examples)
[105). Visit-Bende (500 examples) [12], LLaVA-Bende (90 examples) (65). and Vibe-Eval (269 examples) [8] are commonly employed to assess vision-language models due to their high-quality despite their limited size. Given that human verification demands significant expert hours and is not scalable within our budget, we prioritize data quality for evaluating T2V models.
Difficulty Annotation (Stage 3). To acquire fine-grained insights into the quality of the video generation, we further annotate our each instance in the dataset with perceived difficulty. Specifically, ask two experienced graphics researchers (senior Ph.D. students in physics-based simulation) to independently classify each caption as easy (0) or hard (1) based on their perception of the complexity simulating the objects and motions in the captions using state-of-the-a physics
engines [56] 25 [18, [32. 87 32]. Subsequently, the disagreements were discussed to reach a unanimous judgment for less than 5% of the instances. The difficulty of a simulation is primarily influenced by the complexity of the model, which varies depending on the type of material. For example, deformable bodies pose a greater modeling challenge than rigid bodies because they change shape under external forces, leading to more complex partial differential equations (PDEs). In contrast, rigid bodies maintain their shape, resulting in simpler models. Another key factor is the numerical difficulty in solving these equations, which increases with the material's 's velocity, especially when high-order terms are involved in the PDEs. As a result, slower-moving materials are generally easier to simulate than faster-moving ones. We note that the level of difficulty is evaluated within
4

---

Published as a conference paper at ICLR 2025
each category (e.g., solid-solid, solid-fluid, fluid-fluid), and cannot be compared across different categories. We present the examples for generated captions in Table
Data Analysis. A fine- aneegrained metadata facilitates comprehensive understanding of the bench- mark. Specifically, we present the main statistics of the VIDEOPHYY dataset in Table Notably, we generate 11330 videos for the prompts in the dataset using diverse range of generative models. In addition, the average caption length is 8.5 words, indicating that most captions are straightforward and do not complicate our analysis with complex phrasing that could be excessively challenging the The dataset includes 138 unique actions grounded in our captions. Figure 3 visualizes the root verbs and direct nouns used in the VIDEOPHY captions, highlighting the diversity of actions and entities. Hence, our dataset encompasses wide range of visual concepts and actions. We perform fine-grained diversity analysis in Appendix
generative models. I
3 EVALUATION
3.1 METRICS
The ability to assess the quality of the generated videos is challenging task. While humans can evaluate videos across various visual dimensions 42 20], we focus primarily on the models' adher- ence to the provided text and the incorporation of physical commonsense. These are key objectives that conditional generative models must maximize. We note that several video characteristics such as object motion, video quality, text adherence, physical commonsense, temporal consistency of subject and object etc. are usually intertwined with each other. It is non-trivial to disentangle their effect when humans make decisions. However, focusing on each aspect at time provides a comprehensive picture of the model capabilities along a specific dimension. In this work, we focus on physical commonsense and semantic adherence. Further, there are diverse ways acquire human judgments such as dense and sparse feedback. While a dense feedback provides detailed information about the model mistakes, it is hard to acquire and miscalibrated (69 [55) Due to the simplicity
of binary judgment and its widespread use in text-to-image generative models (58 [54] we employ binary feedback (0/1) to evaluate the generated videos in this work (more discussion Appendix C Further, experiments will demonstrate that binary feedback effectively highlights differences in the model's quality across various object interactions and levels task complexity.
Semantic Adherence (SA). This metric assesses whether the text caption is semantically grounded in the frames of the generated videos, measuring video-text alignment. Specifically, it assesses if the actions, events, entities, and their relationships are perceived to be correctly depicted in the video frames (e.g., water is flowing into the glass in the generated video for the caption 'water pouring into the glass'). In this work, we annotate the generated videos for semantic adherence, denoted as SA = (0, 1}. Here, SA = 1 indicates that the caption is grounded in the generated video.
Physical Commonsense (PC). This metric evaluates whether the depicted actions, and object's state follow the physics laws in the real-world. For instance, the level of water should increase in the glass as water flows into it, following conversation of mass. In this work, we annotate the physical commonsense of the generated videos, denoted as PC (0, 1}. Here, PC = 1 indicates that the generated movements and interactions align with intuitive physics that humans acquire with their experience in the real-world. As physical commonsense is entirely grounded in the video, it is independent of the semantic adherence capability of the generated video. In this work, we compute the fraction of the videos for which semantic adherence is high (SA 1), physical commonsense is high (PC = 1), and joint performance of these metrics is high (SA = 1, PC = 1).
3.2 HUMAN EVALUATION
We conducted a human evaluation to assess the performance of the generated videos in terms of semantic adherence and physical commonsense using our dataset. The annotations were obtained from a group of qualified Amazon Mechanical Turk (AMT) workers who were provided with the detailed task description (and clarifications) on a shared slack channel. Subsequently, 14 workers who have
We use GPT-4 to enhance and generate longer versions of the original captions. However, we found that most of the T2V models are poor following long/enhanced captions most of the time.
5

---

Published as a conference paper at ICLR 2025
studied high-school physics were chosen perform the annotations after passing qualification test. In this task, annotators were presented with a caption and the corresponding generated video without any information about the generative model. They were asked to provide semantic adherence score (0 or 1) and a physical commonsense score (0 or 1) for each instance. Annotators were instructed to treat semantic adherence and physical commonsense independent metrics and were shown several solved examples by the authors before starting the main annotation task. In some cases, we find that generative models create static scenes instead of video frames with high motion. Here, we ask annotators to judge the physical plausibility of the static scene in the real world (e.g., a static scene of a folded brick does not follow physical commonsense). If the static scenes are noisy (e.g.. unwanted grainy or speckled patterns), we instruct them to consider it as poor physical commonsense. !
The human annotators were not asked to list the violation of the physies laws since it would make the annotations more time-consuming and expensive. Additionally, the current annotations can be performed by annotators experience in the physical world (e.g., workers know that water flows down from a tap, shape of wood log will not change while floating on water) instead of advanced education in physics. A screenshot of the human annotation interface is presented in Appendix H
3.3 AUTOMATIC EVALUATION
While the human evaluation is more accurate for benchmarking, it is time-consuming and expensive to acquire at scale. In addition, we want the model developers with limited resources for human evaluation to use our benchmark. To this end, we design CIDEOCONPHYSSICS for our evaluation dataset. Specifically, we use VIDEOCON, an open video-text language model with 7B parameters, that is trained on real videos for robust semantic adherence evaluation [5]. Specifically, we prompt VIDEOCON to generate text response (Yes/No) conditioned on the multimodal template. We provide details about the templates and score computation using VIDEOCON in Appendix [
reliable auto-rater
Since VIDEOCON is not trained on the generated video distribution or equipped to judge physical commonsense, it is not expected to perform well in our setup in zero-shot manner. Prior work has shown that data-driven approaches can outperform rule-based physics simulators for more complicated systems like weather and climate. Hence, we take a data-driven approach in this work. To this end, we propose VIDEOCON -CCOON, an open-source generative video-text model, that can assess the semantic adherence and physical commonsense of the generated videos. Specifically, finetune VIDEOCON by combining the human annotations acquired for the semantic adherence and physical commonsense tasks over the generated videos. We present the GPT-4V [79] and Gemini- 1.5-Pro-Vishe (90) baselines in Appendix M We assess auto-rater effectiveness by computing the ROC-AUC between humans and its judgments for videos generated with testing prompts.
4
SETUP
Video Generative Models. We evaluate a diverse range of twelve closed and open text-to-video
generative models on VIDEOPHY dataset. The list of the models includes ZeroScope (21), LaVIE [112], VideoCrafter2 (22), OpenSora [80). Cog V2deoX-22 and 5B [123),
(SVD)-
T212V 13). Gen-2 (Runway) 29. Lumiere-T22 T2V, Camiere-T212V (Google) [8) Dream Machine
(Luma AI) 12. and Pika 83 We provide more model and inference details in Appendix and N
Dataset setup. As described earlier, we train VIDEOCON-PHYSEE testing of the generated videos on our dataset ($ 3.3). To facilitate this, we split the prompts in the VIDEOPHY dataset equally into train and test sets. Specifically, we utilize the human annotations on the generated videos for the 344 prompts in the test set for benchmarking, while the human annotations on the generated videos for the 344 prompts in the train set are used for training the automatic evaluation model. We ensure that the distribution of the state of matter (solid-solid, solid-fluid, fluid-fluid) and complexity (casy, hard) is similar in the training and testing.
to enable cheaper and scalable
TThe workers were compensated at rate of $18 per hour.
We note that finetuning separate classifier for semantic adherence and physical commonsense did provide any additional benefits over a single classifier
LIDROCON-PHYSICE trained a multi-task manner.
'While there are various closed models such Sora (68), Kling AI 48) and Genmo 34 we could not get access through their videos due the lack of API support.

---

Published as a conference paper at ICLR 2025
Table 3: Human evaluation results on the VIDEOPHY dataset. report the percentage testing prompts for which the T2V models generate videos that adhere to the conditioning caption and exhibit physical commonsense. We abbreviate semantic adherence SA. and physical commonsense PC. SA, PC indicates the percentage of the instances for which SA=1 and PC=1. Ideally, want the generative models maximize the performance on this metrie. In the first column, highlight the overall performance, and the later columns are dedicated fine-grained performance for the interaction between different states matter the prompts.
Overall (%)
Solid-Sold (%)
Folid-Flmid (%)
Fluid-Fluid (%)
Model
SA, PC
SA
PC
SA, PC
PC
SA,
PC
SA
PC
SA,
PC
SA
PC
Open Models
Cog VidcoX-3B 123
39.6
63.3
53
24.4
50.3
43.3
53.1
76.5
59.3
43.6
61.8
61.8
VidcoCrafter2 22
19.0
48.5
34.6
4.9
31.5
23.8
27.4
57.5
41.8
32.7
69.1
43.6
Cog VidenX-2B
123
18.6
47.2
34.1
12.7
42.9
28.1
21.9
56.1
34.9
25.4
34.5
47.2
LaVIE [112
15.7
48.7
28.0
8.5
37.3
19.0
15.8
52.1
30.8
34,5
69.1
43.6
SVD-T212V 14
11.9
42.4
25.9
27.3
17.1
52.7
32.9
18.2
58.2
34,5
ZeroScope 21
11.9
30.2
32.6
6.3
17.5
22.4
14.4
40.4
37.0
20.0
36.4
47.3
OpenSora 80
4.9
18.0
23.5
1.4
7.7
23.8
7,5
30.1
21.9
7,3
12.7
27.3
Closed Models
Pika 883
19.7
41.1
36.5
13.6
24.8
36.8
16.3
46.5
27.9
44.0
68.0
58.0
Dream Machine
13.6
61.9
21.8
50.0
24.3
23.6
9.0
76.3
11.0
Lumiere-T2111 T212V
8
12.5
48.5
25.0
8.4
37.1
25.2
17.1
59.6
26.0
10.9
49.1
21.8
Lumicre-T2V 8
9.0
38.4
27.9
8.4
26.6
27.3
9.6
47.3
26.0
9.1
45.5
34.5
Gen-2 29
7.6
26.6
27.2
4.0
8.9
37.1
8.1
38.5
18.5
15.1
37.7
26.4
Benchmarking. Here, we generate one video per test prompt for each T2V generative model in our testbed. Subsequently, we ask three human annotators to judge the semantic adherence and physical commonsense of the generated videos. scores from the human annotators. We find that the inter-annotator agreement for semantic adherence and physical commonsense judgment is 75% and 70%, respectively. This indicates that the human annotators find the task of judging physical commonsense more subjective than semantic adherence. In total, we collect 24500 human annotations across the testing prompts and T2V models.
our experiments, we report the majority-voted
Training set for VIDEOCON-PHYSIIN nine T2V models We choose two videos to obtain more data instances for training the automatic ask one human annotator judge the semantic adherence and physical commonsense of the generated videos. In total, we collect 12000 human annotations, half of them for semantic adherence and the other half for physical commonsense. Specifically, we finetune VIDEOCON to maximize the log likelihood of Yes/No conditioned on the multimodal template for semantic adherence and physical commonsense tasks (Appendix TT. We do not collect three annotations per video as it is financially expensive. In total, we spent $3500 on collecting human annotations for benchmarking and training.
Here, we sample videos per training prompt for
evaluation model. Subsequently,
RESULTS
5.1 PERFORMANCE ON VIDEOPHY DATASET
We compare the performance of the T2V generative models on the VIDEOPHY dataset using human evaluation in Table 3 We find that Cog VideoX-SB generates videos that adhere to the caption and follow physics laws (SA 1, PC attributed to its high- gugh-tuality data curation including inclusion detailed captions, and filtering videos with less motion and poor quality. In addition, we find that the rest of the video models achieve a score below 20%. This highlights that the existing video models severely lack the capability to generate videos which follow intuitive physics, and establishes VIDEOPHY as challenging dataset
in 39.6% of the cases. The success of Cog VideoX can be
More specifically, Cog VideoX-5B stands out as the best model for generating videos that demonstrate physical commonsense, achieving performance of 53%, while Cog VodcooX-22 the second best
'Variations in annotations arise from differing tolerance for commonsense violations in imperfect videos. As generative models improve, human annotations will align more closely.
'Since Cog VideoX Dream Machine models were released very recently, they could not included in the training set of the automatic evaluator.
We compared pairwise model predictions using the paired t-test at 95% confidence interval. We find that the difference between VidenX-SS and other vidco models is statistically significant (p<0.0001).
7

---

Published as a conference paper at ICLR 2025
open model at 34.1%. Further, this highlights that scaling the network capacity improves its ability to capture the underlying physical constraints of the interne-sccale video data. In addition, we find that OpenSora performs the worst on the VIDEOPHY dataset, indicating significant potential for the community to improve open-source implementations of Sora. Amongst the closed models, Pika achieves generates videos that achieve positive judgement for semantic adherence and physical commonsense for 19. of the cases, Interestingly, we observe that Dream Machine achieves a high semantic adherence score (61.9%) but poor physical commonsense score (21.8%) which highlights that a optimizing for semantic adherence does not necessarily lead to good physical commonsense.
Variation with the states of matter. We study the variation in the performance of T2V models with the interaction between the diverse states of matter grounded in the captions (e.g., solid-solid) in Table Interestingly, we find that all the existing T2V models perform the worst on the captions that depict interactions between solid materials (e.g.. bottle topples off the table), with the best performing model, VogVideX-SSB achieving 24.4% on accurate semantic adherence and physical commonsense. Furthermore, we observe that Pika achieves the highest performance in the captions that depict interaction between fluid and fluid material types (e.g., rain splashing on a pond). This indicates that the T2V model performance is greatly influenced by the states of matter involved in scene, and highlights that model developers can focus on enhancing semantic adherence and physical commonsense for solid-solid interactions.
Variation with the complexity. We analyze the variation in the video model performance with the complexity in rendering objects or synthesizing interactions grounded in the caption under physical simulation in Appendix Table We find that the semantic adherence and physical commonsense performance of all the video models decreases as the complexity of the captions increases. This indicates that the captions that are harder to simulate physically are also harder to control via conditioning for the vidco generative models. Our analysis thus highlights that the future T2V model development should focus on reducing the gap between the easy and the hard captions from our VIDEOPHY dataset. We provide qualitative generated examples from captions of varying complexity and material states in Appendix V Further, we present results for additional metries in Appendix L
Correlation analysis. To understand the connection between various performance metrics, we examine the correlation between semantic adherence (SA) and physical commonsense (PC) with video quality and motion (Appendix $S). Our empirical results show positive correlation between video quality and both PC and SA, while motion exhibits a negative correlation with PC and SA. This indicates that the video models tend to make more mistakes in the SA and PC when more motions are depicted in them. The closed models (Dream Machine/Pika) contribute the higher end of the video quality while open models (Zeroscope/OpenSora) contribute to the lower end of video quality. While the high quality 'correlated' with the better PC, we note that the absolute performance of the models is quite poor on our benchmark.
5.2 QUALITATIVE ANALYSIS
(a) v. Pika. Water spraying from garden hose onto plants.
(b) v. Dream Machine. The sharp knife severs the fresh loaf of bread.
(c) v. Gen-2, tire rolls through deep puddle, splashing water:
Figure 4: Comparison of CogVideoX-SB with other models. The top row shows the videos generated by Cog VideoX-SB.. (a) For Pika, the water streams on the left and right have drastically different speed. (b) For DM, a part of the bread suddenly changes its shape. (c) For Gen-2, the water droplets remain still in the air.
Here, we provide qualitative analysis of the generated videos to assess the common failure modes.
8

---

Published as a conference paper at ICLR 2025
(a) Duminoes loppling one after another on the table
(b) Leather glove catching hard baseball
Figure 5: Illustration of CogVideoX-5B3s limitations in understanding material properties. Even the best-performing model, Cog Vogiooo--5BB may struggle to correctly capture material properties, leading to unnatural dynamics that do not align with object characteristics. Artifacts the examples: (a) dominoes, which should behave as rigid bodies, show inconsistent changes in geometry and texture over time, (b) the leather glove exhibits unnatural deformations.
Comparison between CogVideoX-SB with other models. We analyze some qualitative examples understand the gap between the best-performing model (Cog Vid and the other models in our testbed. We present some examples in Figure Specifically, we find that SVD-T2I2V is likely underperform in scenes involving vibrant fluid dynamics. Lumiere-T212V and Dream Machine (Luma) perform better than Lumiere- TTV in terms of visual quality, but they lack profound understanding of rigid geometries (e.g. in Figure (()). Further, we notice that Gen-2 sometimes generates static objects in the air with slow camera motion, instead of meaningful physical dynamics (e.g. in Figure [c)). In contrast, Cog VideoX-SB shows decent capability of identifying distinct objects, as deformation from its results seldom mingles multiple objects. Further, it tends to use simpler backgrounds, avoiding complex patterns where flaws are casier to be spotted. Nevertheless, the best-performing model, CogVideoX-SBB may struggle to understand the material properties of the underlying objects, resulting in unnatural or inconsistent deformations, as shown Figure [5] This phenomenon is also observed in results from other video generative models. Our analysis highlights the lack of fine-grained physical commonsense that future research should aim to address.
Failure mode analysis. We present some qual- itative examples to understand the common fail- ure modes the generated video regarding poor physical commonsense. Qualitative examples from various T2V generative models are pro- vided in Figure 13 26 in Appendix [ The common failure modes include (a) Conserva- tion of mass violation: the volume or texture of an object is not consistent over time, (b) New- ton's 's First Law violation: an object changes its velocity in a balanced state without any exter- nal force, (c) Newton's 's Second Law violation: an object violates the conversation of momen- tum, (d) Solid Constitutive Law violation: solids deform in ways that contradict their material properties, e.g., a rigid object deforming over time, (e) Fluid Constitutive Law violation: fluids exhibit unnatural flow motions, and (f) Non-physical penetration: objects unnaturally penetrate each other.
Table 4: Comparison of ROC-AUC for au- tomatic evaluation methods. We find that VIDEOCON-PHYSIC outperforms diverse base- lines, including GPT-4Vision and Gemini-1.5-Pro. for semantic adherence (SA) and physical com- monsense (PC) judgments on the testing prompts.
Method ()Rtoo/(/RRCC-AACCC
SA
PC
Random
50
50
GPT-4- 79
53
53
Gemini-1.5-PP 1..55-Pr-Vising Vision 90
73
58
VIDEOCON (5)
65
54
VIDEOCON-PHYSICE (Ours)
82
73
6 VIDEOCON-PHYSICE AUTOMATIC EVALUATOR FOR VIDEOPHY DATASET
We supplement our dataset with VIDEOCON- PHYSIN, automatic rater for scalable and reliable evaluation of semantic adherence and physical commonsense in the generated videos.
VIDEOCON- COCSSSSS generalizes to unseen prompts. We compare the ROC-AUC of different automatic evaluators with the human predictions on the testing prompts in Table Here, the videos are generated by the models that are used to train the VIDEOCON PHNNIICS model. We find that the VIDEOCON COEEOCS outperforms the zero-shot VIDEOCON by 17 points and 19 points on the semantic adherence and physical commonsense judgment, respectively. This highlights that finetuning with the generated video distribution and human annotations aids in improving the model judgment

---

Published as a conference paper at ICLR 2025
the unseen prompts. Further, notice that the model's agreement higher for semantic adherence as compared to the physical commonsense. This indicates that judging physical commonsense is a harder task than judging semantic adherence for VIDEOCON-PHYSILE the HIPT-4-Vision s judgments are close to random for semantic adherence and physical commonsense on dataset. This implies that faithful evaluations are hard to obtain from the multi-image reasoning capabilities of the GPT-4-Vision in a zero-shot manner. To address this, we test Gemini-Pry Proviiiioo-viiionn 1.5 and find that it achieves good semantic adherence score (73 points), however, it is close to random in physical commonsense evaluation (54 points). This highlights that the existing multimodal foundation models lack the capability to judge physical commonsense.
Interestingly, observe that
VIDEOCON-PHYSIC, generalizes to unseen generative models. To assess performance an unseen video distribution, we train an ab- lated version of VIDEOCON-PHYSICA stricted set of video data. Specifically, we train "IDEOCON-PHYSICS" on human annotations ac- quired from VideoCrafter2, ZeroScope, LaVIE, OpenSora, SVD-T2I2V and Gen-2, and eval- uate it on unseen videos from the remaining T2V models in our testhed generated for the test- ing captions. We present the results in Table We find that VIDEOCON- CELDOCONN-PISSSSS SICS outperforms VIDEOCON by 15 points and 15 points semantic adherence and physical commonsense judgement, respectively. This highlights that VIDEOCON- PHISSISS can judge semantic adherence and physical commonsense T2V generative models are released.
Table 5: Performance of VIDEOCON-PHYSICS on unseen generative model. We train ablated version of VIDEOCON PHEOCON-PHYSICS and find that outperforms the baseline in the semantic adherence (SA) and phys- ical commonsense (PC) judgment averaged over three unseen video models on the testing prompts.
re-
Method
SA
PC
VIDEOCON 5
64
57
TIDEOCON-PHYSSIC (Ours)
79
72
Automatic leaderboard reliably tracks human leaderboard. We create an automatic leaderboard by averaging the semantic adherence and physical commonsense scores of the open and closed video models on the test set. Subsequently, we align these rankings with the human leaderboard based on the joint performance metrics (SA 1, PC = 1). We present the human and automatic leaderboard for the open and closed model Appendix IP We observe that the relative rankings of the models in the automatic leaderboard (Cog VideoX-SB3 VideoCrafier2>>LaVIE Cog VideoX- strongly matches with the relative rankings of the mod- els in the human lcaderboard (Cog VideoXode--BBB DBBB>VidooCraftrr>>CC.
SeI2V=/ZeoScoop>>ppenneet OpenSora) We observe similar trends for the closed models. But, we find that Pika achieves relatively low score the automatic leaderboard, limitation that can be improved by acquiring more data for LIDEOCON-PHYSII the models are similar under both the leaderboards, establishing its reliability for future model development. Further discussion on the usefulness of VIDEOCON-PPIYSICS in Appendix R
Overall, we find that the rankings of most of
Finetuning video models. While VIDEOPHY data is used for model evaluation and building automatic evaluator, we assess whether this dataset can be used to finetune video models Ap- pendix T Post-finetuingg we observe significant decrease in semantic adherence, while physical commonsense remains unchanged. This is likely due to limited training samples, optimization challenges, and the nascency of the video finetuning field. Future work will focus on enhancing physical commonsense in generative models based on these findings.
7 CONCLUSION
In this work, we introduce VIDEOPHY, a first of its kind dataset to assess the physical commonsense in the generated videos. Further, we evaluate a diverse set video models (open and closed models) and found that they significantly lack in the physical commonsense and semantic adherence capabilities. Our dataset unveils that the existing methods are far being general-purpose world simulators. Further, we introduce VIDEOCON- PHYSCCS and scalable evaluation on our dataset. We believe that our work will serve as the comnerstone in studying physical commonsense for video generative modeling.
auto-evaluation model that enables cheap
10

---

Published as a conference paper at ICLR 2025
REPRODUCIBILITY STATEMENT
In this work, we provide a detailed description about the dataset construction in $ Specifically, mention the prompts used for initial caption generation in Appendix G Further, we provide the details about all the video generative models in $ along with the inference details in Appendix[N [N] In addition, provide the details about finetuning VIDEOCON-PUY sICS in Appendix [ Finally, commit to releasing the data, generated videos, and trained VIDEOCON-PIYYSIC in the camera-ready version.
ACKNOWLEDGEMENT
HB is supported in part by AFOSR MURI grant FA9550-22-1-0380. YS was partially supported by NSF 2211557, NSF 2119643, NSF 2303037, NSF 2312501, SRC JUMP 2.0 Center. Amazon Research Awards, and Snapchat Gifts. KC was partially supported by ONR grant N00014-23-1-2780, and U.S. DARPA ECOLE Program No. #HR001 12390060.
REFERENCES
[1] Niket Agarwal, Arslan Ali, Maciej Bala, Yogesh Balaji, Erik Barker, Tiffany Cai, Prithvijit Chattopadhyay, Yongxin Chen, Yin Cui, Yifan Ding, al. Cosmos world foundation model platform for physical ai. arXiv preprint arXiv:2501.03575 2025.
[2] Luma AI. Luma Dream Machine AI Video Generator lumalabs.ai. https://
uma abs het//ddrammmccchhc
2024.
[3] Max Bain, Arsha Nagrani, Güil Varol, and Andrew Zisserman. Frozen in time: A joint video and image encoder for end-to-end retrieval. In IEEE International Conference on Computer Vision, 2021.
[4] Hritik Bansal, Da Yin, Masoud Monajatipoor, and Kai-Wei Chang. How well can text-to- image generative models understand ethical natural language interventions? arXiv preprint arXiv:2210. 15230, 2022.
[5] Hritik Bansal, Yonatan Bitton, Idan Szpektor, Kai- Wei Chang, and Aditya Grover. Videocon: Robust video-language alignment via contrast captions. arXiv preprint 1rXiv:2311.10111, 2023.
[6] Hritik Bansal, Yonatan Bitton, Michal Yarom, Idan Szpektor, Aditya Grover, and Kei-Wei Chang. Tale: Time-aligned captions for multi-scene text-to-video generation. arXiv preprint arXiv:2405.0466 04682, 2024.
[7] Hritik Bansal, Ashima Suvarna, Gantavya Bhatt, Nanyun Peng, Kai-Wei Chang, and Aditya Grover. Comparing bad apples to good oranges: Aligning large language models via joint preference optimization. arXiv preprint arXiv:2404.0055 0004404400030,, 2024.
[8] Omer Bar-Tal, Hila Chefer, Omer Tov, Charles Herrmann, Roni Paiss, Shiran Zada, Ariel Ephrat, Junhwa Hur, Yuanzhen Li, Tomer Michaeli, et al. Lumiere: A space-time diffusion model for video generation. arXiv preprint arXiv:2401. 12945, 2024.
[9] David Baraff. introduction to physically based modeling: rigid body simulation unconstrained rigid body dynamics. SIGGRAPH course notes, 82, 1997.
[10] Christopher Batty, Florence Bertails, and Robert Bridson. A fast variational framework for accurate solid-fluid coupling. ACM Transactions on Graphics (TOG), 2603): 2007.
[11] Yonatan Bisk, Rowan Zellers, Jianfeng Gao, Yejin Choi, al. Piqa: Reasoning about physical commonsense in natural language. In Proceedings of the AAAI conference on artificial intelligence, volume 34, pp. 7432-7439, 2020.
[12] Yonatan Bitton, Hritik Bansal, Jack Hessel, Rulin Shao, Wanrong Zhu, Anas Awadalla, Josh Gardner, Rohan Taori, and Ludwig Schmidt. Visit-bench: A benchmark for vision- language instruction following inspired by real-worle use. arXiv preprint arXiv:2308. 06595, 2023.
11

---

Published as a conference paper at ICLR 2025
[13] Andreas Blattmann, Tim Dockhorn, Sumith Kulal, Daniel Mendelevitch, Maciej Kilian, Do- minik Lorenz, Yam Levi, Zion English. Vikram Voleti, Adam Letts, et al. Stable video diffusion: Scaling latent video diffusion models to large datasets. arXiv preprint arXiv:2311.15121 2023.
[14] Andreas Blattmann, Tim Dockhorn, Sumith Kulal, Daniel Mendelevitch, Maciej Kilian, Do- minik Lorenz, Yam Levi, Zion English, Vikram Voleti, Adam Letts, et al. Stable video diffusion: Scaling latent video diffusion models to large datasets. arXiv preprint arXiv:2311.15122 2023.
[15] Andreas Blattmann, Robin Rombach, Huan Ling, Tim Dockhorn, Seung Wook Kim, Sanja Fidler, and Karsten Kreis. Align your latents: High-resolution video synthesis with latent diffusion models. In Proceedings of the IEEE/CVF Conference on Computer Vision and Putlern Recognition, pp. 22563-22575, 2023.
[16] Robert Bridson. Fluid simulation for computer graphics. AK Peters/CRC Press, 2015.
[17] Tim Brooks, Janne Hellsten, Miika Aittala, Ting-Chun Wang, Timo Aila, Jaakko Lehtinen, Ming-Yo Yu Liu, Alexei Efros, and Tero Karras. Generating long videos of dynamic scenes. Advances in Neural Information Processing Systems, 35:31769-31777 2022.
[18] Tim Brooks, Bill Peebles, Connor Holmes, Will DePue, Yufei Guo, Li Jing, David Schnurr, Joe Taylor, Troy Luhman, Eric Luhman, Clarence Ng. Ricky Wang, and Aditya Ramesh. Video generation models world simulators. 2024. URL https: wtwptttsss//p/ennntettttt /openapeass i com/ research!
video- citeoo-geerraiion mocensoooeees-se wesswwrldd simulators
[19] Jake Bruce, Michael Dennis, Ashley Edwards, Jack Parker-Holder, Yuge Shi, Edward Hughes, Matthew Lai, Aditi Mavalankar, Richie Steigerwald, Chris Apps, et al. Genie: Generative interactive environments. arXiv preprint urXiv:2402.1539) 2024.
[20] Emanuele Bugliarello, H Herman Moraldo, Ruben Villegas, Mohammad Babaeizadeh, Moham- mad Taghi Saffar, Han Zhang, Dumitru Erhan, Vittorio Ferrari, Pieter-Jan Kindermans, and Paul Voigtlaender. Storybench: A multifaceted benchmark for continuous story visualization. Advances in Neural Information Processing Systems, 36, 2024.
[21] cerspense. cerspense/zeroscoope. 576w Hugging Face huggingface.co. seree___2______7 __6 2023.
https
//huggingface..c wtt.eee.co/ceeeeeeeeeeeee http:///eeeeeeeeeeeeeooooooooooo
[22] Haoxin Chen, Yong Zhang, Xiaodong Cun, Menghan Xia, Xintao Wang, Chao Weng. and Ying Shan. Videocrafter2: Overcoming data limitations for high-quality video diffusion models. arXiv preprint arXiv:2401.09047 2024.
[23] Hsiao- Yu Chen, Arnav Sastry, Wim M van Rees, and Etienne Vouga. Physical simulation of environmentally induced thin shell deformation. ACM Transactions on Graphics (TOG), 37 (4):1-13, 2018.
[24] Tsai-Shien Chen, Aliaksandr Siarohin, Willi Menapace, Ekaterina Deyneka, Hsiang- -wi Chao, Byung Eun Jeon, Yuwei Fang. Hsin Ying Lee, Jian Ren, Ming-Hsuan Yang, et al. Panda-70m: Captioning 70m videos with multiple cross-modality teachers. arXiv preprint UrXiv:2402.19473 2024.
[25] Yunuo Chen, Tianyi Xie, Cem Yuksel, Danny Kaufman, Yin Yang, Chenfanfu Jiang, and Minchen Li. Multi-layer thick shells. In ACM SIGGRAPH 2023 Conference Proceedings, pp. 1-9, 2023.
[26] Yilun Du, Sherry Yang, Bo Dai, Hanjun Dai, Ofir Nachum, Josh Tenenbaum, Dale Schuurmans, and Pieter Abbeel. Learning universal policies text-guided video generation. Advances in Neural Information Processing Systems, 36, 2024.
[27] Jiafei Duan, Arijit Dasgupta, Jason Fischer, and Cheston Tan. A survey on machine learning approaches for modelling intuitive physics. arXiv preprint ar arXiv:2202.0648) 06481, 2022.
12

---

Published as a conference paper at ICLR 2025
[28] Patrick Esser, Robin Rombach, and Bjorn Ommer. Taming transformers for high- resouussott image synthesis. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition. pp. 12873-12883, 2021.
[29] Patrick Esser, Johnathan Chiu, Parmida Atighehchian, Jonathan Granskog, and Anastasis Ger- manidis. Structure and content-guided video synthesis with diffusion models. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pp. 7346-7356, 2023.
[30] Patrick Esser, Sumith Kulal, Andreas Blattmann, Rahim Entezari, Jonas Müller, Harry Saini, Yam Levi, Dominik Lorenz, Axel Sauer, Frederic Boesel, et al. Scaling rectified flow transform- ers for high- regouuesolution image synthesis. In Forty-first International Conference on Machine Learning, 2024.
[31] Kawin Ethayarajh, Winnie Xu, Niklas Muennighoff, Dan Jurafsky, and Douwe Kiela. Kto: Model alignment as prospect theoretic optimization. arXiv preprint arXiv:2402.0 400001306,, 2024.
[32] Yu Fang, Ziyin Qu, Minchen Li, Xinxin Zhang, Yixin Zhu, Mridul Aanjaneya, and Chenfanfu Jiang. Iq-mpm: an interface quadrature material point method for non-sticky strongly two-way coupled nonlinear solids and fluids. ACM Transactions on Graphics (TOG), 39(4):51-1, 2020.
[33] Samir Yitzhak Gadre, Gabriel Ilbarco, Alex Fang. Jonathan Hayase, Georgios Smyrnis, Thao Nguyen, Ryan Marten, Mitchell Wortsman, Dhruba Ghosh, Jieyu Zhang, et al. Datacomp: In search of the next generation of multimodal datasets. Advances in Neural Information Processing Systems, 36, 2024.
[34] genmo. Genmo. Create videos and images with AI. genmo.ai. https: http://www.ggngme a17
[35] Jiayuan Gu, Fanbo Xiang, Xuanlin Li, Zhan Ling, Xiqiang Liu, Tongzhou Mu, Yihe Tang, Stone Tao, Xinyue Wei, Yunchao Yao, et al. Maniskil12: A unified benchmark for generalizable manipulation skills. arXiv preprint arXiv:2302.0466
2023.
[36] Xuchen Han, Joseph Masterjohn, and Alejandro Castro. A convex formulation of frictional contact between rigid and deformable bodies. IEEE Robotics and Automation Letters, 2023.
[37] Jonathan Ho, Ajay Jain, and Pieter Abbeel. Denoising diffusion probabilistic models. Advances in neural information processing systems, 33:6840-6835 6851, 2020.
[38] Wenyi Hong. Ming Ding, Wendi Zheng, Xinghan Liu, and Jie Tang. Cogvideo: Large-scale pretraining for text-to-video generation via transformers. arXiv preprint arXiv:2205. 2022.
[39] Edward J Hu, Yelong Shen, Phillip Wallis, Zeyuan Allen-Zhu, Yuanzhi Li, Shean Wang, Lu Wang, and Weizhu Chen. Lora: Low-rank adaptation of large language models. arXiv preprint arXiv:2106.00000
2021.
[40] Tianyu Huang, Yihan Zeng, Hui Li, Wangmeng Zuo, and Rynson WH Lau. Dreamphysics: Learning physical properties of dynamic 3d gaussians with video diffusion priors. arXiv preprint arXiv:2406.0 01476, 2024.
[41] Zhiao Huang, Yuanming Hu, Tao Du, Siyuan Zhou, Hao Su, Joshua Tenenbaum, and Chuang Gan. Plasticinelab: A soft-body manipulation benchmark with differentiable physics. arXiv preprint arXiv:2104.03311, 2021.
[42] Ziqi Huang, Yinan He, Jiashuo Yu, Fan Zhang, Chenyang Si, Yuming Jiang, Yuanhan Zhang, Tianxing Wu, Qingyang Jin, Nattapol Chanpaisit, et al. Vbench: Comprehensive benchmark suite for video generative models. arXiv preprint arXiv:2311.1798) 2023.
hugginnffccEElleDiicrrreeeeettdt EullerDiscreteSSchcclllee - huwgingfaag...cccooo http://eeeeettttteooceeceeneeien
[43]
https
[44] Xuan Ju, Yiming Gao. Zhaoyang Zhang, Ziyang Yuan, Xintao Wang, Ailing Zeng, Yu Xiong, Qiang Xu, and Ying Shan. Miradata: A large-scale video dataset with long durations and structured captions. arXiv preprint arXiv:2407.06335 2024.
13

---

Published as a conference paper at ICLR 2025
[45] Levon Khachatryan, Andranik Movsisyan, Vahram Tadevosyan, Roberto Henschel, Zhangyang Wang, Shant Navasardyan, and Humphrey Shi. Text2video-zero. Text-to-image diffusion mod- els are zero-shot video generators. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pp. 15954-15964 2023.
[46] Diederik P Kingma and Jimmy Ba. Adam: A method for stochastic optimization. arXiv preprint arXiv: /4 12.6900, 2014.
[47] Gergely Klár, Theodore Gast, Andre Pradhana, Chuyuan Fu, Craig Schroeder, Chenfanfu Jiang, and Joseph Teran. Drucker-praged elastoplasticity for sand animation. ACM Transactions on Graphics (TOG), 35(4): 2016.
[48] KlingAl. KLING AI - klingai.com. https://ww/t k Engaing... com/ 2024.
[49] Dan Kondratyuk, Lijun Yu, Xiuye Gu, José Lezama, Jonathan Huang, Rachel Hornung. Hartwig Adam, Hassan Akbari, Yair Alon, Vighnesh Birodkar, et al. Videopoet: A large language model for zero-shot video generation. arXiv preprint arXiv:23 14125, 2023.
[50] Dan Koschier, Jan Bender, Barbara Solenthaler, and Matthias Teschner. Smoothed particle hydrodynamics techniques for the physics based simulation of fluids and solids. arXiv preprint arXiv:2009.0694 2020.
[51] Tengchuan Kou, Xiaohong Liu, Zicheng Zhang, Chunyi Li, Haoning Wu, Xiongkuo Min, Guangtao Zhai, and Ning Liu. Subjective-aligned dateset and metric for text-to-video quality assessment. arXiv preprint arXiv:2403.11956 2024.
[52] LaionAI. GitHub Aaa-ON--/laeshhei--prriioooooo A linear estimator on top of clip to predict the aesthetic quality of pictures github.com. bLtps: com/ LAION- AI/
aesthstic-predictee 2022.
[53] Egor Larionov, Christopher Batty, and Robert Bridson. Variational stokes: a unified pressure- viscosity solver for accurate viscous liquids. ACM Transactions on Graphics (TOG), 36(4):
1- 2017.
[54] Kimin Lee, Hao Liu, Moonkyung Ryu, Olivia Watkins, Yuqing Du, Craig Boutilier, Pieter Abbeel, Mohammad Ghavamzadeh, and Shixiang Shane Gu. Aligning text-to-image models using human feedback. arXiv preprint 2rXiv:2302.12192. 2023.
[55] James R Lewis and Oguzhan Erdinę. User experience rating scales with 7, 11, or 101 points: does it matter? Journal of Usability Studies, 12(2), 2017.
[56] Minchen Li, Zachary Ferguson, Teseo Schneider, Timothy Langlois, Denis Zorin, Daniele Panozzo, Chenfanfu Jiang, and Danny M Kaufman. Incremental potential contact: intersection- and inversion-free, large-deformation dynamics. ACM Trans. Graph., 39(4):49, 2020.
[57] Minchen Li, Danny M Kaufman, and Chenfanfu Jiang. Codimensional incremental potential contact. arXiv preprint arXiv:2012.04457, 2020.
[58] Shufan Li, Konstantinos Kallidromitis, Akash Gokul, Yusuke Kato, and Kazuki Kozuka. Aligning diffusion models by optimizing human utility. arXiv preprint Ttl::2004.44466 2024.
[59] Xuan Li, Minchen Li, and Chenfanfu Jiang. Energetically consistent inelasticity for optimiza- tion time integration. ACM Transactions Graphics (TOG), 41(4):1-16, 2022.
[60] Jacky Liang, Viktor Makoviychuk, Ankur Handa, Nuttapong Chentanez, Miles Macklin, and Dieter Fox. Gpu-accelerated robotic simulation for distributed reinforcement learning. In Conference on Robot Learning. pp. 270-282, PMLR, 2018.
[61] Zhiqiu Lin, Deepak Pathak, Baiqi Li, Jiayao Li, Xide Xia, Graham Neubig, Pengchuan Zhang, and Deva Ramanan. Evaluating text-to-visual generation with image-to-text generation. arXiv preprint arXiv:2404.011 2024.
14

---

Published as a conference paper at ICLR 2025
[62] Zongyu Lin, Wei Liu, Chen Chen, Jiasen Lu, Wenze Hu, Tsu-Jui Fu, Jesse Allardice, Zhengfeng Lai, Liangchen Song, Bowen Zhang, et al. Stiv: Scalable text and image condi- tioned video generation. arXiv preprint arXi::2412.07730, 2024.
[63] Zongyu Lin, Yao Tang, Xingcheng Yao, Da Yin, Ziniu Hu, Yizhou Sun, and Kai-Wei Chang. Qlass: Boosting language agent inference via q-guided stepwise search. arXiv preprint arXiv:2502.02588 2025.
[64] Fangfu Liu, Hanyang Wang, Shunyu Yao, Shengjun Zhang, Jie Zhou, and Yueqi Duan. Physics3d: Learning physical properties of 3d gaussians via video diffusion. arXiv preprint arXiv:2406.0493 043338 2024.
[65] Haotian Liu, Chunyuan Li, Qingyang Wu, and Yong Jae Lee. Visual instruction tuning. Advances in neural information processing systems, 36, 2024.
[66] Shaowei Liu, Zhongzheng Ren, Saurabh Gupta, and Shenlong Wang. Physgen: Rigid-body physics-grounded image-to-video generation.
[67] Yaofang Liu, Xiaodong Cun, Xuebo Liu, Xintao Wang, Yong Zhang, Haoxin Chen, Yang Liu, Tieyong Zeng, Raymond Chan, and Ying Shan. Evalcrafter: Benchmarking and evaluating large video generation models. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 22139-22149, 2024.
[68] Yixin Liu, Kai Zhang, Yuan Li, Zhiling Yan, Chujic Gao, Ruoxi Chen, Zhengqing Yuan, Yue Huang, Hanchi Sun, Jianfeng Gao, al. Sora: A review on background, technology, limitations, and opportunities of large vision models. arXiv preprint arXiv:240..17177 2024.
[69] Luis M Lozano, Eduardo Garcfa-Cueto. and José Muñiz. Effect of the number of response categories on the reliability and validity of rating scales. Methodology, 4(2):73-79, 2008.
[70] Cheng Lu, Yuhao Zhou, Fan Bao, Jianfei Chen, Chongxuan Li, and Jun Zhu. Dpm- solver++: Fast solver for guided sampling of diffusion probabilistic models. arXiv preprint arXiv:2211.01095, 2022.
[71] Mariem Mezghanni, Malika Boulkenafed, Andre Lieutier, and Maks Ovsjanikov. Physically- aware generative network for 3d shape modeling. In Proceedings of the IEEE/CVF Conference Computer Vision and Pattern Recognition, pp. 9330-9341, 2021.
[72] mplugowl. mplug-owl-video. https: video
com
main/mPLUG 1a116-0wwmmppugee
[73] Matthias Müller, Barbara Solenthaler, Richard Keiser, and Markus Gross. Particle-based fluid-fluid interaction. In Proceedings of the 2005 ACM SIGGRAPI//Eungraphic symposium on Computer animation, pp. 237-244, 2005.
[74] Tung Nguyen, Rohan Shah, Hritik Bansal, Troy Arcomano, Romit Maulik, Veerabhadra Kotamarthi, Ian Foster, Sandeep Madireddy, and Aditya Grover. Scaling transformer neu- ral networks for skillful and reliable medium-range weather forecasting. arXiv preprint arXiv:2312.03876 2023.
[75] Junfeng Ni. Yixin Chen, Bohan Jing, Nan Jiang, Bin Wang, Bo Dai, Yixin Zhu, Song-Chun Zhu, and Siyuan Huang. Phyrecon: Physically plausible neural scene reconstruction. arXiv preprint arXiv:2404, 16666, 2024.
[76] Alexander Quinn Nichol and Prafulla Dhariwal. Improved denoising diffusion probabilistic models. In International conference on machine learning, pp. 8162-8171. PMLR, 2021.
[77] James F O'biien, Adam W Bargteil, and Jessica K Hodgins. Graphical modeling and animation of ductile fracture. In Proceedings of the 29th annual conference Computer graphics and interactive techniques, pp. 291-294, 2002.
[78] OpenAL. Gpt-4 technical report. arXiv preprint arXiv:2003.08774 2023a, 2023.
15

---

Published as a conference paper at ICLR 2025
[79] OpenAl. Gpt-4v(ision) system card, 2023b. gpt -v--ystte-ccadd 2023.
http:://ppaada..om//eeesaaoo
[80]
OpenSora. GitHub hpcaitech/OpeeeeSozzz Open-Sora: Democratizing Efficient Video Produc- tion for All github.com.
http://hwww....oooppaaaateeee//p 2024.
[81] Piotr Padlewski, Max Bain, Matthew Henderson, Zhongkai Zhu, Nishant Relan, Hai Pham, Donovan Ong. Kaloyan Aleksiev, Aitor Ormazabal, Samuel Phua, et al. Vibe-eval: A hard evaluation suite for measuring progress of multimodal language models. arXiv preprint arXiv:2405.02287. 2024.
[82] William Peebles and Saining Xie. Scalable diffusion models with transformers, In Proceedings of the IEEE/CVF International Conference on Computer Vision, pp. 4195-4205, 2023.
[83] pika. Pika
aika.art. https://pika..a art/
[84] Luis S Piloto, Ari Weinstein, Peter Battaglia, and Matthew Botvinick. Intuitive physics learning in a deep-learning model inspired by developmental psychology. Nature human behaviour, (9):1257-122) 1267, 2022.
[85] Dustin Podell, Zion English, Kyle Lacey, Andreas Blattmann, Tim Dockhorn, Jonas Müller, Joe Penna, and Robin Rombach. Sdxl: Improving latent diffusion models for high-resolution image synthesis. arXiv preprint arXiv:2307.01952 2023.
[86] Pranav Putta, Edmund Mills, Naman Garg, Sumeet Motwani, Chelsea Finn, Divyansh Garg, and Rafael Rafailov. Agent q: Advanced reasoning and learning for autonomous ai agents. arXiv preprint arXiv:240SS 07 199, 2024.
[87] Ziyin Qu, Minchen Li, Yin Yang. Chenfanfu Jiang, and Fernando De Goes. Power plastics: A hybrid lagrangian/euleriaan solver for mesoscale inclastic flows. ACM Transactions on Graphics (TOG), --1,
[88] Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, et al. Learning transferable visual models from natural language supervision. In International conference on machine learning. pp. 8748-8766 PMLR, 2021.
[89] Rafael Rafailov, Archit Sharma, Eric Mitchell, Christopher D Manning, Stefano Ermon, and Chelsea Finn. Direct preference optimization: Your language model secretly reward model. Advances in Neural Information Processing Systems, 36, 2024.
[90] Machel Reid, Nikolay Savinov, Denis Teplyashin, Dmitry Lepikhin, Timothy Lillicrap. Jean- baptiste Alayrac. Radu Soricut, Angeliki Lazaridou, Orhan Firat, Julian Schrittwieser, et al. Gemini 1.5: Unlocking multimodal understanding across millions of tokens of context. arXiv preprint arXiv:2403.00 055300, 2024.
[91] Robin Rombach, Andreas Blattmann, Dominik Lorenz, Patrick Esser, and Björn Ommer. High-resolution image synthesis with latent diffusion models. Proceedings of the IEEE/VV conference computer vision and pattern recognition, pp- 10684-10695, 2022.
[92] Robin Rombach, Andreas Blattmann, Dominik Lorenz, Patrick Esser, and Bjöm Ommer. High-resolution image synthesis with latent diffusion models. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pp. 10684-10695, 2022.
[93] Chitwan Saharia, William Chan, Saurabh Saxena, Lala Li, Jay Whang. Emily L Denton, Kamyar Ghasemipour, Raphael Gontijo Lopes, Burcu Karagol Ayan, Tim Salimans, et al. Photorealistic text-to-image diffusion models with deep language understanding. Advances in neural information processing systems, 35:36479-36443 2022.
[94] John Schulman, Filip Wolski, Prafulla Dhariwal, Alec Radford, and Oleg Klimov. Proximal policy optimization algorithms. arXiv preprint arXiv: Doxiiv/707.00e /000:0007.0777 2017.
16

---

Published as a conference paper at ICLR 2025
[95] Soshi Shimada, Vladislav Golyanik, Weipeng Xu, and Christian Theobalt. Physcap: Physically plausible monocular 3d motion capture real time. ACM Transactions on Graphics (ToG), 39 (6):1-16, 2020.
[96] Noah Shinn, Federico Cassano, Ashwin Gopinath, Karthik Narasimhan, and Shunyu Yao. Re- flexion: Language agents with verbal reinforcement learning. Advances in Neural Information Processing Systems, 36, 2024.
[97] Eftychios Sifakis and Jemej Barbic. Fem simulation of 3d deformable solids: a practitioner's guide to theory, discretization and model reduction. In Acm siggraph 2012 courses, pp. 1-50. 2012.
[98] Uriel Singer, Adam Polyak, Thomas Hayes, Xi Yin, Jie An, Songyang Zhang, Qiyuan Hu, Harry Yang, Oron Ashual, Oran Gafni, al. Make-a-video, Text-to-video generation without text-video data. arXiv preprint arXiv:2209. 14792, 2022.
[99] Ivan Skorokhodov, Sergey Tulyakov, and Mohamed Elhoseiny. Stylegan-v: A continuous video generator with the price, image quality and perks of stylegan2. In Proceedings of the IEEE/CVF Conference on Computer Vision and Puttern Recognition, pp. 3626-3636, 2022.
[100] Jiaming Song. Chenlin Meng, and Stefano Ermon. Denoising diffusion implicit models. arXiv preprint arXiv:2010.02502, 2020.
[101] Alexey Stomakhin, Craig Schroeder, Lawrence Chai, Joseph Teran, and Andrew Selle. A material point method for snow simulation. ACM Transactions on Graphics (TOG), 32(4): 1-10, 2013.
[102] Keqiang Sun, Junting Pan, Yuying Ge, Hao Li, Haodong Duan, Xiaoshi Wu, Renrui Zhang, Aojun Zhou, Zipeng Qin, Yi Wang, et al. Journeydb: A benchmark for generative image understanding. Advances in Neural Information Processing Systems, 36, 2024.
[103] Tetsuya Takahashi, Tomoyuki Nishita, and Issei Fujishiro. Fast simulation of viscous fluids with elasticity and thermal conductivity using position-based dynamics. Computers & Graphics, 43:21-30, 2014.
[104] Zachary Teed and Jia Deng. Raft: Recurrent all-pairs field transforms for optical flow. In Computer Vision-ECCCV 2020: 16th European Conference, Glasgow, UK, August 23-28, 2020, Proceedings, Part II 16. pp. 402-419; Springer, 2020.
[105) Tristan Thrush, Ryan Jiang, Max Bartolo, Amanpreet Singh, Adina Williams, Douwe Kiela, and Candace Ross. Winoground: Probing vision and language models for visio-linguistic compositionality. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 5238-5248, 2022.
[106] Thomas Unterthiner, Sjoerd Van Steenkiste, Karol Kurach, Raphael Marinier, Marcin Michal- ski, and Sylvain Gelly. Towards accurate generative models of video: new metric & challenges. arXiv preprint arXivv/8/2.0011
2018.
[107] Ruben Villegas, Mohammad Babaeizadeh, Picter-Jan Kindermans, Hernan Moraldo, Han Zhang, Mohammad Taghi Saffar, Santiago Castro, Julius Kunze, and Dumitru Erhan. Phenaki: Variable length video generation from open domain textual descriptions. In International Conference on Learning Representations, 2022.
[108] Bram Wallace, Meihua Dang. Rafacl Rafailov, Lingi Zhou, Aaron Lou, Senthil Purushwalkam, Stefano Ermon, Caiming Xiong, Shafiq Joty, and Nikhil Naik. Diffusion model alignment using direct preference optimization. arXiv preprint arXiv:2311.12900 2023.
[109] Yixin Wan, Arjun Subramonian, Anaelia Ovalle, Zongyu Lin, Ashima Suvarna, Christina Chance, Hritik Bansal, Rebecca Pattichis, and Kai-Wei Chang. Survey of bias in text-to-image generation:
Definition, evaluation, and mitigation. arXiv preprint arXiv:2404.00 404.000300 2024.
[110] Jiuniu Wang, Hangjee Yuan, Dayou Chen, Yingya Zhang, Xiang Wang, and Shiwei Zhang. Modelscope text-to-video technical report. arXiv preprint S3ii:2300806571 2023.
17

---

Published as a conference paper at ICLR 2025
[111] Wenjing Wang, Huan Yang, Zixi Tuo, Huiguo He, Junchen Zhu, Jianlong Fu, and Jiaying Liu. Videofactory: Swap attention in spatiotemporal diffusions for text-to-video generation. arXiv preprint arXiv:2305. 10874, 2023.
[112] Yaohui Wang, Xinyuan Chen, Xin Ma, Shangchen Zhou, Ziqi Huang, Yi Wang. Ceyuan Yang. Yinan He, Jiashuo Yu, Peiqing Yang. et al. Lavie: High-quality video generation with cascaded latent diffusion models. arXiv preprint arXiv:2309. 15/03, 2023.
[113] Yi Wang. Yinan He, Yizhuo Li, Kunchang Li. Jiashuo Yu, Xin Ma, Xinhao Li, Guo Chen, Xinyuan Chen, Yaohui Wang, et al. Internvid: A large-scale video-text dataset for multimodal understanding and generation. arXiv preprint vmXiv:2307.06942 2023.
[114] Mariusz Witezak, Lestaw Juszezak, and Dorota Galkowska. Non-newtonian behaviour of heather honey. Journal of Food Engineering, 104(4):532-537, 2011.
[115] Joshuah Wolper, Yumuo Chen, Minchen Li, Yu Fang, Ziyin Qu, Jiecong Lu, Meggie Cheng, and Chenfanfu Jiang. Anisompm: Animating anisotropic damage mechanics: Supplemental document, ACM Trans. Graph, 39(4), 2020.
[116] Xueqing Wu, Yuheng Ding, Bingxuan Li, Pan Lu, Da Yin, Kai-Wei Chang, and Nanyun Peng. Visco: Benchmarking fine-grained critique and correction towards self-improvement in visual reasoning. arXiv preprint drXiv:2412.02172..2 2024.
[117) Xueqing Wu, Zongyu Lin, Songyan Zhao, Te-Lin Wu, Pan Lu, Nanyun Peng, and Kai-Wei Chang- Vdebugger: Harnessing execution feedback for debugging visual programs. arXiv preprint S.Kiv:2400.134444 2024.
[118] Tianyi Xie, Minchen Li, Yin Yang, and Chenfanfu Jiang. A contact proxy splitting method for lagrangian solid-fluid coupling. ACM Transactions on Graphics (TOG), 42(4):1-14, 2023.
[119] Tianyi Xie, Zeshun Zong, Yuxin Qiu, Xuan Li, Yutao Feng, Yin Yang, and Chenfanfu Jiang. Physgaussian: Physics-integrated 3d gaussians for generative dynamics. arXiv preprint arXiv:2311.12198 2023.
[120] Haike Xu, Zongyu Lin, Jing Zhou, Yanan Zheng, and Zhilin Yang. A universal discriminator for zero-shot generalization. arXiv preprint arXiv:2211. 080999, 2022.
[121] Sirui Xu, Zhengyuan Li, Yu-Xiong Wang, and Liang Yan Gui. Interdiff: Generating 3d human-object interactions with physics-informed diffusion. In Proceedings of IEEE/CVF International Conference on Computer Vision, pp. 14928-14940, 2023.
[122] Hongwei Xue, Tiankai Hang, Yanhong Zeng, Yuchong Sun, Bei Liu, Huan Yang. Jianlong Fu, and Baining Guo. Advancing high-resolution video-language representation with large-scale video transcriptions. In Proceedings of the IEEF/CVF Conference on Computer Vision and Pattern Recognition, pp. 5036-5045, 2022.
[123] Zhuoyi Yang, Jiayan Teng, Wendi Zheng, Ming Ding, Shiyu Huang, Jiazheng Xu, Yuanming Yang, Wenyi Hong, Xiaohan Zhang, Guanyu Feng, al. Cogvideox: Text-to- video diffusion models with an expert transformer. arXiv preprint 24X.:::208000777 2024.
[124] Hanrong Ye, Haotian Zhang, Erik Daxberger, Lin Chen, Zongyu Lin, Yanghao Li, Bowen Zhang, Haoxuan You, Dan Xu, Zhe Gan, et al. Mm-ego: Towards building egocentric multimodal 1lms. arXiv preprint 241.0.2410.07177, 2024.
[125] Qinghao Ye, Haiyang Xu, Guohai Xu, Jiabo Ye, Ming Yan, Yiyang Zhou, Junyang Wang. Anwen Hu, Pengcheng Shi, Yaya Shi, et al. mplug-owl: Modularization empowers large language models with multimodality. arXiv preprint arXiv:2304. 14178, 2023.
[126] Ilker Yildirim, Max H Siegel, Amir A Soltani, Shraman Ray Chaudhuri, and Joshua B Tenenbaum. Perception of 3d shape integrates intuitive physics and analysis-by-ee by-shtesssss.. Nature Human Behaviour, 8(2):320-335, 2024.
18

---

Published as a conference paper at ICLR 2025
[127] Lijun Yu, José Lezama, Nitesh B Gundavarapu, Luca Versari, Kihyuk Sohn, David Minnen, Yong Cheng, Agrim Gupta, Xiuye Gu, Alexander G Hauptmann, et al. Language model beats tokeeier is key visual generation. arXiv preprint arXiv:2310.057..1 2023.
[128] Ye Yuan, Jiaming Song, Umar Iqbal, Arash Vahdat, and Jan Kautz. Physdiff: Physics-guided human motion diffusion model. In Proceedings of the IEEE/CVE International Conference on Computer Vision, pp. 16010-16021, 2023.
[129] Yonghao Yue, Breannan Smith, Christopher Batty, Changxi Zheng, and Eitan Grinspun. Continuum foam: A material point method for shear-dependent flows. ACM Transactions on Graphics (TOG), 34(5): 1-20, 2015.
[130] Tianyuan Zhang, Hong- -ing Yu, Rundi Wu, Brandon Y Feng, Changxi Zheng, Noah Snavely, Jiajun Wu, and William T Freeman. Physdreamer: Physics- based interaction with 3d objects via video generation. arXiv preprint 2rXiv:2404.33022 2024.
[131] Jing Zhou, Zongyu Lin, Yanan Zheng, Jian Li, and Zhilin Yang. Not all tasks are born equal: Understanding zero-shot generalization. The Eleventh International Conference Learning Representations, 2022.
[132] Zeshun Zong, Xuan Li, Minchen Li, Maurizio M Chiaramonte, Wojeiech Matusik, Eitan Grinspun, Kevin Carlberg. Chenfanfu Jiang, and Peter Yichen Chen. Neural stress fields for reduced-order elastoplasticity and fracture. In SIGGRAPH Asia 2023 Conference Papers, pp. 1-11, 2023.
[133] Zeshun Zong, Chenfanfu Jiang, and Xuchen Han. A convex formulation of frictional contact for the material point method and rigid bodies. arXiv preprint arXiv:2403.13783 2024.
19

---

Published as a conference paper at ICLR 2025
A RELATED WORK
Video Generation Models. Recent advancements in video generation models have emerged from two primary architectures: diffusion-based models [29]
and 49 38 107 Among these, diffusion models have garnered significant attention. The model known as SVD [14], built on a Latent Diffusion Model (LDM) 191, proposes three-stage training process for video LDMs: text-to-image pretraining, video pretraining, and video finetuning. Sora 68 represents a state-of-the-art in video generation, utilizing feffussion-transfoore language description processing for video generation. ModelScope [100] is also diffusion-bai based text-to-video model which combines VQGAN (28) a text-encoder, and a denoising UNet. Another diffusion model, VideoCrafter2 [22]. leverages low-quality videos and high-quality videos to generate high-quality videos. LaVIE [12 composed of a base text-to- video model, a temporal interpolation model, and a video super-resolution model, indicating that joint image and video training and temporal self-attention with rotary positional embeddings are key components to boost performance. Given the rapid development of video generation technology, an effective evaluation method for the generated videos becomes crucial. Our paper focuses on evaluating text-to-video generation models for their physical commonsense capabilities.
14
68
15 112
45 62
autoregressive modeling- based approaches [27
architecture with unified training recipes and enhancements in
Evaluating Video Generation Models. To evaluate the quality of a T2V generative model, Fréchet video distance (FVD) traditionally used measure the similarity between real and generated video distributions [166] 20). However, FVD has several limitations for assessing physical commonsense including the requirement for a reference video that difficult obtain for novel scenes, bias towards video quality, and failure to detect unrealistic motions [17 99]. Similarly, CLIPScore [88] measures semantic similarity between generated video frames and the conditioning text shared representation space, making it unsuitable for evaluating physical commonsense in generated videos.
However, there is a growing consensus on the need for more comprehensive metrics to assess the performance of video generation models 42 67 [1 61). V-Bench 42] offers detailed benchmark suite that introduces a hierarchical evaluation protocol, breaking down 'video generation quality' into various granular perspectives. Another framework, EvalCrafter [67]. proposes 17 objective metrics. Despite these advancements, existing methods largely overlook the fundamental aspect physical commonsense. Unlike static images, videos incorporate temporal dimension, embedding physical commonsense information across frames. Our research dives into the measurement of physical commonsense (II) in videos. Additionally, we introduce a VIDEOCON-PHYw analyze specific physical laws that are violated in the generated videos through qualitative analysis.
auto-evaluator and
Physics Modeling. Simulating physical behaviors of solids and fluids has always been an important and popular topic in computer graphics. For solid materials, the simplest physical model is the long-established rigid body simulation [9, where solids are assumed not to deform. Simulation of deformable solids [97]. the other hand, takes into account the strain and stress during deformation. To capture more complicated materials, researchers have been proposing increasingly intricate models for different materials, such as metal (77), sand (47), and snow [10I. In contrast, most of the common fluids [16] in daily life can be broadly categorized as inviscid (50), e.g., water and air, and viscous fluids [103] (3). e.g., honey and oil. Additionally, an orthogonal research direction is to accurately, efficiently, and robustly model contact and interaction between different materials. These
include solid-solid [56] 57 solid-fluid 10 118, and fluid-fluid interactions 73]. Further, recent
advancements in computer vision have started exploring incorporating physics priors into various 3D-aware generation tasks to enhance physical plausibility, such as human animation [[28 95 [21]
and 3D/4D generation [ 119 130]. However, these approaches often depend on high-quality 3D reconstructions from multi- -iew images. Some efforts [66] have also integrated physics- based simulations into video generative models, but the simulations
performed in 2D space due the lack of 3D information, resulting in limited dynamics. In this work, instead of generating, we focus on identifying whether the generated video adheres physical laws.
20

---

Published as a conference paper at ICLR 2025
B DISCUSSION ON CATEGORIZATION
In this work, group various types of solids into a single, unified "solid' category. In theory, all solids can be prescribed as a universal constitutive model governed by the conservation of mass and momentum. Nevertheless, graphics researchers typically model certain materials using simplified constitutive models to lower computational costs. For instance, a rigid body, in reality, is deformable body with very high stiffness. In fact, exact rigidity does not exist in the real world. Similarly, researchers propose other solid constitutive models (e.g. fabrics, granular materials) to simplify computation for some specific material behaviors. Our choice of solid, broad categorization, serves as basis for generalization beyond specific simulation techniques used in graphics research. Our benchmark is designed to capture wide spectrum of physically plausible behaviors, rather than isolating specific graphics sub-domains.
The interactions between solid-solid pairs primarily focus on resolving contact constraints to prevent penetration. In contrast, solid-fluid interactions exhibit more diversity. For example, water can be repelled by an umbrella but can be absorbed by a sponge. Such behaviors, including perme- ability, adhesion, and absorption, are not available in solid-solid interactions. Given the diversity of solid-fluis interactions, we choose to balance the sample counts for solid-solid and solid-fluid interactions to ensure our benchmark does not overemphasize simpler contact-based interactions while underrepresenting more complex and varied solid-fluie dynamics.
Our current design prioritizes simplicity and broad applicability, especially for non-experts who may use the benchmark across various disciplines. Future work can consider more fine-grained categorization to better measure the ability of generative models to handle specific properties (e.g.. plasticity, viscosity) if desired.
C MORE DISCUSSION ON USING BINARY FEEDBACK
We highlight that binary feedback (0/1) is quite popular in aligning generative models such as large language models [1 [20). Further, we observe that binary feedback is much easier to collect at industrial scale by big generative model providers (e.g.. ChatGPT). For instance, we note that the ChatGPT user interface asks binary preference after generating the response to a simple query. Similar extensions exist in the field of text-to- text-to-image generative models [58] 54]. Hence, the binary feedback protocol is quite powerful in studying and improving the generative models.
We highlight that dense feedback system would capture more nuanced mistakes of the video generative models (e.g., completing 8 movements versus 6 movements). However, designing such prompts non-trivial, and evaluating the generated videos in such scenarios is much more challenging, labor-extensive, and expensive (especially with limited academic budgets). Further, we note that the collection of diverse and denser forms of feedback is a crucial future work.
In this work, we do not report posterior physical commonsense performance (PC = given SA = 1) since they can be inferred from the joint and marginal scores. In addition, we note that bad model can easily game the posterior metric. For example, a bad model can generate a vidco which aligns with the prompt for 1 out of 700 prompts in the dataset. Now, assume that this video is also accurate in terms of physical commonsense. Hence, the posterior performance of this model will be 100%. This can be quite misleading for the practitioners.
D LIMITATIONS
In this work, we evaluate the physical commonsense capabilities of T2V generative models. Specifi- cally, we curated the VIDEOPHY dataset, consisting 688 captions. We argue that the captions are comprehensive and high-quality after going through our three-stage data curation pipeline. In the future, it will be pertinent to expand the physical commonsense understanding to more branches of physics, including projective geometry. Also, the recent trend of LLM [86] 63) begins to study the power of process reward modeling, we give up exploring more fine- -re-grained evaluation signals like frame level alignment score of physies due the heavy cost and leave it for future work. It can be interesting to study using fine- -nnaiind language feedback to update video generation model just like the reflection mechanism in L.MM/ VLM 196 117 116). Additionally, test diverse
21

---

Published as a conference paper at ICLR 2025
set of T2V generative models, including both open and closed models. While it is financially and computationally challenging to evaluate an exhaustive list of models, we have aimed to incorporate models with diverse architectures, training datasets, and inference strategies. In the future, it will be important to gain access to and include new high-performance T2V models in our study.
In addition, we perform human annotations using Amazon Mechanical Turkers (AMT), where most of the workers primarily belong to the US and Canada. Hence, the human annotations in this work do not represent the diverse demographies around the globe. As result, our human annotations reflect the perceptual biases of the annotators from Western cultures. In the future, it will be pertinent to assess the impact of diverse groups on our human evaluations. Finally, we acknowledge that text-to-video generative models can perpetrate societal biases in their generated content [009] 4 It is critical that future work quantifies this bias in the generated videos and provides methods for the safe deployment of the models. Also, since our video just covers the domain of physics-specife videos, it interesting to see whether can generalize and evaluate the videos with other properties such as long, egocentric, embodied related [44] 124
etc.
E DATA LICENSING
The VIDEOPHY dataset comprises videos generated by various T2V (Text-to- Video) generative models, detailed in Section[F) The licensing terms for these videos will align with those specified by the respective model owners, as cited in this work. The curated captions and human annotations will be licensed under the MIT License.
F VIDEO GENERATIVE MODELS
110 a latent diffusion- based text-to-video model that adapts the text-to-image generative model 92 for video generation by training high-quality video and image data for enhanced visual quality. Further, we benchmark LaVIE [122, a cascaded video latent diffusion model instead of a single diffusion model. Specifically, the LaVIE model trained with a specialized curated dataset for enhanced visual quality and diversity. In addition, we test VideoCrafter2, latent diffusion T2V model that enhances video generation quality by training on high-quality image-text data 102]. In our study, we also benchmark OpenSora [80]. an open-source effort replicate Sora (18. a high-performant closed latent diffusion model that diffusion transformers (82) for text-to-video generation. Finally, we include Stable VideoDiffusion (SVD) latent diffusion model that can generate high resolution videos conditioned on text or image. Since SVD-I2V (Image-to-Videe
For the open models, we benchmark Zeroscope 21
is publicly available, we utilize that to generate the videos. generate the conditioning images from the captions in the VIDEOPHY dataset. We term the entire pipeline as SVD-T2I2V.
Specifically, we utilize SD-X0 ED-XL-Base-1A 1.0
For the closed models, we include Gen-2 29 a closed latent video diffusion model from Runway. In addition, we include Pika [3 with undisclosed information about the underlying generative model. Specifically, we wrote a custom API to acquire Gen-2 and Pika videos after paying for their monthly subscription for total of $225. Finally, we include two versions of the Lumiere 8 from Google research. Specifically, Lumiere-72V generates a video conditioned on the text, while Lumiere-T212V generates a video conditioned caption using a text-to-iin tottttiiieeeee generative model [93]. Cog VideoX [23] is a most recent open- open-sourced state-of-the-ar video generation models, which uses MMDiT (30 llike architecture, and achieves very good text-to-video alignment and video quality performance.
image, that is in-turn generated with the
G QUERYING GPT-4 FOR PROMPT GENERATION
In this section we discuss the prompt utilized to generate all the prompts including three physical interaction categories: solid-solid, solid-fluid, fluid-fluid for video generation, which is displayed in Table 6 Table 7 and Table g
22

---

Published as a conference paper at ICLR 2025
Develop unique and imaginative captions, each briefly describing the interaction between two different solid materials realistic scene. Each cuption should consist 7-10 words and clearly indicate the solids involived in the action.
Guidelines:
Focus solids used everyday scenarios, avoiding rare seldom-used materials.
Exclude actions "celebrating', "arguing', 'laughing' do not clearly involve physical interaction between materials.
3. Avoid generating static scenes (e.g- 'Lid covers pot to retain heat', 'Stack of paper sits on the desk").
4. Avoid adding participle phrases (e.g.. 'seeteenin it', 'a creamy swirl', 'fizzing energetically) in the caption.
The captions should on the actions require contact forces, or friction fonces. Do focus the actions that require penetration forces.
Format each cuption follows: 'uction': : ACTION, "solid I:: SOLID, 'solid SOLID, 'cuption': CAPTION
Examples Captions (Do Generate Such Captions):
A diamond scratching glass. HW Scratching action that requires penetration
A key scratches the surface of wooden table. #ill Scratching action that requires penetration
Good Examples Of Captions:
A brick presses down on a metal can.
snowball falls ground splits spart.
A small red elastic ball stuck to the wall.
Figure 6: GPT-4 Prompt to Generate Solid-Solid Captions.
Develop unique and imaginative captions, showcasing interaction between solid material with fluid material, for generating video, After crafting the cuption, list the entities that act as solid and fluid in the caption.
Guidelines:
Focus solids and fluids used everyday scenarios, avoiding zare seldom-used materials.
Exclude actions like "celebrating', 'arguing', or 'laughing' that do not clearly involve physical interaction between materials.
3. Avaid actions that execute state change from solid to fluid or vice-versa.
4. Avoid generating static scenes (e.g.. 'Lid covers pot retain heat').
Avoid adding participle phrases (e.g.. 'sweetening 'a creamy swirl', 'fizzing energetically' the caption.
6. The captions should focus on the actions that require contact forces, or friction forces, Do not focus on the actions that require penetration forces.
Format each caption as follows: 'action': : ACTION, 'solid': SOLID, 'fludd': FLUID, 'caption': CAPTION
Bad Examples Of Captions (Do Not Generate Such Captions): Sugar dissolves in water. ## dissolving action will not be visible video Sulfuric acid corroding metal. ## corrosion will not be visible video Water boiling in pot. ## boiling action will not be visible in video
Good Examples Of Captions:
dam break releases a massive floo..
An iron rod falls the water.
A metal spoon stirs the honey a cup.
Figure 7: GPT-4 Prompt to Generate Solid-Fluid Captions.
23

---

Published as a conference paper at ICLR 2025
Develop unique and imaginutive captions, cach briefly describing the interaction between two different fluid materials in realistic scene. Fach cuption should 7-10 words and clearly fluids involied in the
Guidelines:
Focus common fluids used in everyday scenarios, avoiding rare seldom-used materials.
Exclude actions like "celebrating", "arguing', 'laughing' do clearly involve physical interaction between materials.
Avoid generating static scenes (e.g., "Lid covers pot to retain heat"). )
4. Avoid adding participle phrases (e.g.. 'sweetening it', 'a creamy swirl, 'fizzing energetically) in the caption.
S. The captions should focus the actions that require mixing and laying for liquid-liquid interactions. some contact forces between liquid and gas.
6. Format each cuption us follows: "action': ; ACTION, 'fluid 1": FLUID, "Buid 2:: FLUID, 'cuption': CAPTION
Bad Examples Of Captions (Do Generate Such Captions):
Juice solidifies around water in ice trays. e# solidification won't 't be visible in the video Sugar disappears into stitring water. ## dissolving won't be visible video An acid base react neutralize each other, forming water. chemical reactions
visible in the video
Good Examples Captions:
The wind creating ripples across the surface of the lake.
Milk falls into transparent cup of water.
Oil falls transparent cup
Figure 8: GPT-4 Prompt to Generate Fluid-Fluid Captions.
H HUMAN ANNOTATION SCREENSHOT
We display the screenshot of our human annotation system in Figure 9
Answer the following questions based on the caption and the generated video.
Caption: $(caption)
Does the video exhibit Text Adherence (Video-Text Alignment)? No
Does the video follow Physics Laws or Physical Commonsense? (This property is independent of Video-Text Alignment) O
Yes
No
Submit
Figure 9: The screenshot of the human annotation interface.
1 VIDEOCON DETAILS
We prompt VIDEOCON to generate a text response (Yes/No) conditioned on the multimodal template Ti(x:) for semantic adherence and physical commonsense tasks. Formally,
(V,C), Tx(x) (Tro(V),
==
(I)
== PC
where is either semantic adherence to the caption or physical commonsense task, is the condition- ing caption and V is the generated video for the caption C. We provide the multimodal templates
24

---

Published as a conference paper at ICLR 2025
VTs We compute the score from the VIDEOCON model po:
Po (Yes (x)))
88 (Tr()))
(2)
pe Yes (x))) po
where po (x)) is the probability of 'ee'' conditioned Tz))), and {SA,
We present the prompts used for the GPT4V, Cemini--.5-Prov L.5-Pro-Vision, VideoCon basclines, and VIDEOCON-PHYSICE for semantic adherence evaluation in Figure [0 physical commonsense alignment in Figure
Semantic adherence:
Given: V (Video), T (Caption)
Instruction (I): [V] Does this video entail the description [T]?
Response (R): Yes or No
Figure 10: Template used assessing semantic adherence for a generated video.
Physical Commonsense:
Given: V (Video)
Instruction (I): [V] Does this video follow physical laws?
Response (R): Yes or No
Template for assessing physical commonsense. We note that the physical commonsense is independent of the conditioning caption. Hence, it is not present in this template.
Figure 11:
J FINE-GRAINED DIVERSITY ANALYSIS
In this section, we visualize the fine tne-grrined statistics of collections across different physical interac- tion categories (Figure 12 Figure
K RESULTS FOR TASK COMPLEXITY
We compare the performance of various video generative models across different task complexity in
Table 6
L FINE-GRAINED RESULTS
In this section, we report the fine-grained performance of semantic adherence and physical com- monsense scores from all video generation models and compute the scores across different physical interaction categories (solid-solid, solid-fluid and fluid-fluid), as well as difficulty levels (0 and 1).
M AUTOMATIC EVALUATION BASELINES
Similar to 66, we utilize the capability of GPT-4Vision 79 to reason over multiple images
zero-shot manner. Specifically, prompt the GPT-4V model with the caption and 8 video frames
As large video multimodal model, VIDFOcoN predicts token distribution over the entire token vocabulary conditioned the multimodal template. Therefore, po (Yes T((x)) (NN))T(())) is not equal
25

---

Published as a conference paper at ICLR 2025
grais
cower
apple
semp
partitt
hasshall
bud
wood
dip
collecs
polos
Fail
verape
shovel
hii
apple
slice
eqpaeste
claop
claap
ncokfhce
Mack
fervak
wire
strip
ploque
wipe
shredball
sume
wall
dust
pile
Figure 12: Top 20 most frequently occurring verbs (inner circle) and their top direct nouns (outer circle) in our curated captions that consists of interaction between solid-solid states of matter.
cbaerlese
cup
soup
mia
pour
swreop
stir
pot
soak
epill
spread
Soandation
rill
scoeo
apeny
take
car
bayar
batter
drup
stain
layer
gotist
sponge
soap
strokz
swil
oo
Figure 13: Top 20 most frequently occurring verbs (inner circle) and their top 4 direct nouns (outer circle) our curated captions that consists of interaction between solid-fluid states of matter.
sampled uniformly from the generated video. Here, we instruct the model to provide the semantic adherence or and physical commonsense score (0 1). Since GPT-4V does not process videos natively, we assess the automatic evaluation using Gemini-Pro- Poanini-ro-Vision-..555 which can input the caption and the entire generated video. Specifically, we instruct it to provide the semantic adherence (0 or 1) and physical commonsense (0 or 1) of the input video, identical to the GPT-4V analysis. We provide the prompts used in the experiments Figure and
N INFERENCE DETAILS
We add the inference configurations for different video generation models Table
O TRAINING DETAILS FOR VIDEOCON-PHYSICS
To create VIDEOCON-PaR PHYSISS to all the layers of the attention blocks including QKVO, gate, up and down projection matrices. We set the LoRA 32 and 32 and dropout 0.05. The finetuning is performed for epochs using
low-rank adaptation (LoRA) (39 of the VIDEOCON applied
26

---

Published as a conference paper at ICLR 2025
print
water
akcebel
apple
pour
desdovand
pour
dalloee
create
peint
ageay
accept
blow
bnaak
strile
thin
surfae
Biseer
water
oreanor
ripple
lemon
puddle
name
burst
Figure 14: Top 20 most frequently occurring verbs (inner circle) and their top direct nouns (outer circle) in curated captions that consists of interaction between fluid-fluid states of matter.
Table 6: Fine-grained performance
caption complexity using human evaluation. We find that T2V models struggle more on the harder captions than the easier captions both the semantic adherence (SA) and physical commonsense (PC) metrics.
Easy (%)
Hard (%)
Model
SA
PC
SA
PC
Open Models
Cog VideoX-SB
63.8
55.3
62.5
50.3
VideoCrafter2
53.4
38.1
42.6
30.3
Cog VideoX22B
51.1
38.3
42.6
29.0
LaVIE
51.9
31.2
44.8
24.0
SVD-T2I2V
41.8
37.6
43.2
22.6
ZeroScope
32.3
33.9
27.7
31.0
OpenSora
20.1
25.4
5.2
21.3
Closed Model
Pika
45.7
39.9
35.1
32.1
Dream Machine
65.2
29.4
57.8
12.5
Lumiere- 2222
56.6
29.1
38.7
20.0
Lumiere-T2V
38.6
34.9
38.1
19.4
Gen-2
26.6
31.8
26.6
21.6
Adam [46] optimizer with a linear warmup 50 steps followed by linear decay. Similar to [5), we chose the peak learning rate as 1e - 4. We utilized 2 A6000 GPUs with the total batch size of 32. In addition, we finetune our model with 32 frames in the video and the frames are resized to 224 x 224 by image processor. Similar to [72 125), create 32 segments of the video, and sample the middle frame for each segment.
P AUTOMATIC AND HUMAN LEADERBOARD
We compute the physical commonsense and semantic adherence scores for the models on the testing set using
Subsequently, we take their average and create a rankings of the models. We have similar ranking of the models using the joint performance metrics (SA=1, PC=1) from human evaluation. We present the human and automatic leaderboard for the open and closed models in Table I0, Our analysis reveals that the average rank of the models in the automatic leaderboard is 0.66 above or below the expected rank of the model in the human leaderboard. This indicates VIDEOCON. POYSSINS reliable for evaluating the future models our dataset.
27

---

Published as a conference paper at ICLR 2025
Table 7: Fine- aeme-graine performance of T2V models for the interaction between diverse states of matter using human evaluation. Ideally, we want the T2V models to achieve a high score on the SA = and PC / metric while reduce the score the SA=0 and PC=0, SA=/ and PC=0, and SA=0 and PC=/ metrics.
Source
Categury
SA (%)
PC(%)
SA=I and PC=1 (%)
SA=1 und PC=I (%)
SA=0 and PC=1 (%)
SA-00 and PC=0 (%)
Open
Hlai--Flid
618
43.6
182
18.2
18.2
20.0
Cog CogVideXX-tt
Solid Fiuid Solid
17.2
Fludd Fluid
34.5
47.3
25.5
9.1
21.8
43.6
CogVideoX-21 2B
Solid- Solid-Fluid
56.2
34.9
21.9
34.2
13.0
30.8
Solid-Solid
Fluid Fluid
69.1
43.6
34.5
34.5
9.1
21.8
LaVIB
Solid Fluid
52.1
30.8
15.8
36.3
15.1
32.9
Solid-Solid
Fluid-F Flaid
Fluid-Fluid
12.7
27.3
7.3
5.5
20.0
67.3
OpenSom
Solid-Fluid
50.1
21.9
7.5
22.6
14.4
55.5
Solid-Solid
Hluid-Fluid
69.1
43.6
32.7
36.4
10.9
20.0
Videotirafter2
Solid Fluid
57.5
41.8
27.4
30.1
14.4
28.1
Solid Solid
31.5
23.8
26.6
18.9
48.7
Floid-Fluid
58.2
34.5
182
40.0
1644
25.5
SVD-T2ILV
Solid- Flaid
52.7
$2.9
17.1
35.6
15.8
25.5
Solid-Solid
25.9
27.3
4.2
21.7
23.1
51.0
Fluid Faiid
36.4
47.3
20.0
16.4
27.3
36.4
ZenoScupe
Solid Flaid
40.4
37.0
14.4
26.0
22.6
37.0
Solid-Seii Solid-Solid
17.5
22.4
11.2
66.4
Closed Models
Fluid Fluid
76.4
10.9
67.3
21.8
Fluid- Fuiid
37.7
26.4
15.1
22.6
11.3
50.9
Gen-2
Solid-Fuuid
10.4
Floid-Fluid Fluid
45.4
34.5
9.1
36.4
25.5
29.1
Lumiere/T2V
Fluid
26.0
Solid
Floid-Fluid
49.5
21.8
10.9
38.2
10.9
40.0
Lamiore- TV22V
Solid-Fluid
26.0
31.5
Solid
Fluid-Fha
68.0
$8.0
44.0
24.0
14.0
18.0
Solid-Fluid
28

---

Published as a conference paper at ICLR 2025
Table Fine-gree gree-ggaandd performance of T2V models for the complexity of the captions using human evaluation. Ideally, we want the T2V models to achieve a high score on the SA and PC = metric while reduce the score on the SA=0 and PC=0, SA=/ and PC=0, and SA=0 and PC=/ metrics.
Source
Category
(8)
PC(S)
SA=1 and PC=1 (3)
SA=1 and PC=0 (%)
SA:0 and PC=1 (8)
SA=0 and PC=0 (S)
Open Models
Vid
HARD
EASY
63.8
62.6
40.9
38.1
24.5
22.9
14.4
12.3
14.4
12.3
25.2
21.8
CogVidenX-288
EASY
31.1
58.3
20.7
17.6
17.6
31.4
HARD
42.6
29.0
16.1
12.9
12.9
44.5
LaVIE
HARD
EASY
$1.9
44.8
24.0
51.2
11.0
19.6
83.8
11.6
13.0
36.5
42.2
EASY
OpesSon
HARD
13.5
21.3
5.2
10.3
16.1
68.4
VideoCrafter2
EASY
21.2
16.9
29.6
HARD)
SVD-T312V
HARD
EASY
42.0
38.0
16.0
25.0
21.0
37.0
ZeroSoope
HARD
EASY
32.3
27.7
33.9
31.0
13.8
9.7
18.5
18.1
21.3
20.1
47.6
51.0
Closed Models
EASY
19.8
25.1
HARD
37.9
12.5
5.9
52.0
0.6
35.5
EASY
31.8
10.4
21
52.0
HARD
21.6
56.1
Lumiere-T2V
EASY
38.6
34.9
11.1
27.5
23.8
37.6
Lumien-T2122
EASY
56.6
29.1
16.4
40.2
12.7
30.7
HARD
20.0
1.7
49.0
EASY
45.7
39.9
23.7
22.0
16.2
38.2
Pila
HARD
35.1
52.1
14,5
20.6
17.6
47.3
Table 9: Inference details for models in our testbed. Here, NA indicates that the information is not available for the closed models.
Model
Resolution
# of Video Frames
Guidance Scale
Sampling Steps
Noise Scheduler
Open Models
VodeidexX
480 x
7.5
DDPM
ZeroScope
320 576
9
50
DPMSolverMultSSto
VideoCratter2
320 x 512
32
12
50
DDIM [0X]
LaVIE
320 x 512
32
7.5
50
DDPM 371
OpenSora
240 x
7
100
IDDPM 76
SVD-T212V
1024 576
(1,3)
BulerDiserete
Closed Models
Jumiere-TVV
1024 x 1024
80
8
256
NA
Lumicre-T212V
1024 x 1024
80
6
256
NA
Gen-2
720 1280
8.5
100
Dream Machine
720
NA
NA
Pika
640 1088
72
12
NA
NA
Q ADDITIONAL EXPERIMENT: RELATIVE RANKINGS
In this work, we focus on collecting the absolute (0/1) feedback from the human annotators. Here, we aim to understand the effect of changing the feedback acquisition protocol relative rankings for physical commonsense evaluation. Specifically, we ask the three workers to look at two videos simultaneously and pick the one with better physical commonsense. particular, we got 500 pairwise comparisons for 4 video generative models (Cog VidcoX-dSBBBB Pika, Gen2, OpenSora). It costs us $360 to run this human evaluation. Subsequently, we computed the ELO scores these models based on the human annotations. We present the results in Table T2
Interestingly, we find that the relative ranking of these models remains unchanged under both the feedback methods. Specifically, CogVideoX and OpenSora are still the best and worst models the VideoPhy dataset, respectively. We note that the open (usually smaller) video generative models will be penalized for losing close (usually larger) video generative models in the ranking-based
29

---

Published as a conference paper at ICLR 2025
Table 10: Human and Automatic leaderboard for open and closed video generative models. We compute the joint performance metrics (SA = 1, PC = 1) from human evaluation, and average the SA and PC scores from automatic evaluation construct the leaderboard. The models are ranked from best to worst (descending order). We find that the automatic leaderboard reliably tracks the human leaderboard.
Open models
Closed models
Human
VIDEOCON-PHYSSS PHYSICS
Rank diff.
Human
VIDEOCON- IIDOOCON-PHYSICS PHYSICS
Rank diff.
Cog VideoX-5B
Cog VideoX-SBB VideoCrafter2
Pika
Dream Machine
VideoCrafter2
Dream Machine
Lumiere- T2122
Cog VideoX-2B
LaVIE
Lumiere-T2h T212V
SVD-T2I2V
LaVIE
2B
Lumiere-T2V
Pika
SVD-T212V
SVD-T2I2V
0
Gen-2
Gen-2
0
ZeroScope
ZeroScope
OpenSora
OpenSora
Table 11: Automatic Leaderboard Video Generation Models.
#
Open Models
PC
SA
Avg.
1
Videooo
41
49
2
VideoCrafier-2
36
47
42
Closed Models
PC
SA
Avg.
3
LaVIE
35
46
41
Luma Dreamer T2i2V
30
45
4
Mochi
30
42
36
Lamma T2i2V
25
36
5
Video (Base)
29
40
35
Pika
23
34
29
SDXV-T2i2V
33
Gen-2 (Runway)
31
26
29
7
Hunyuan Video
26
38
32
8
ZeroScope
24
35
29
9
Pika
23
34
29
setup. The absolute feedback operates independently across all video generative models, and helps in better contextualizing the capability of the models with similar scales.
R APPLICATIONS OF VIDEOCON-PHYSICS
In this work, we propose VIDEOCON-PIISSEA auto-evaluator that judges the semantic adherence and physical commonsense the generated videos for a given caption. Here, we describe the potential usecases of the model for future work.
The ability to perform model verification on downstream tasks cheaply and reliably is critical. this regard, the model builders can utilize VIDEOCON- -HYSIIS to evaluate their candidate models on the VIDEOPHY dataset at scale. The top candidate models can then be evaluated with the human workers for more accurate evaluation.
Video Generative Model Selection:
Data Filtering:
With the advent of foundation models that trained the internet data, high- quality filtering has emerged as a crucial step in the pipeline [3 [31 62). Here, the data builders can utilize VIDEOCON- PHYN-PSSSCS to filter low-quality video-text data that lacks in semantic adherence and physical commonsense.
Recently, aligning the generative models with human or Al feedback has become pivotal for high-quality generations [94] 89 108 (8). Here, the post-training pipeline of the video generative models can leverage the MODECCON-PLYSLEE model as an reward model that provides feedback to the model generated content. Subsequently, this feedback can be utilized to refine the model for better generations.
Post-training:
CORRELATION WITH VIDEO QUALITY AND MOTION
There are several works that focus on assessing the generated video quality and motions 42 67 Here, aim assess the correlation between the semantic adherence and physical commonsense
30

---

Published as a conference paper at ICLR 2025
Table 12: Results with relative feedback acquisition protocol. We present the ELO score few selected model by asking human annotators to compare the two videas side-by-side, and pick the one that follows physical commonsense more. We also mention that the binary ratings that the videos get under the absolute feedback acquisition protocol. We find that the rankings of the models are identical under both the setups.
Model
ELO
Binary ratings
Cog VogVideoX-5B
1081
53
Pika
1048
36.5
Gen2 1010
27.2
OpenSora
860
23.5
scores and these metrics. Specifically, calculate the video quality using LAION aesthetic classifier [52] and video motion using the RAFT optical flow model [104]. Subsequently, we calculate the pearson correlation between video quality and motion with physical commonsense and semantic adherence. We present the results in Table 13
We find that physical commonsense and semantic adherence correlated positively with the video quality, albeit the correlation is not very strong. In addition, we find that physical commonsense and semantic adherence are negatively correlated with the video motions. This indicates that the video models tend to make more mistakes in the semantic adherence and physical commonsense when more motions are depicted in them. In this work, we consider wide breadth of video generative models open and closed. The closed models (Dream Machine/Gen-2/Pikal contribute to the higher end of the video quality while open models (Zeroscope/OpenSSrra) contribute to the lower end. While the high quality is 'correlated' with the better physical commonsense, we note that the absolute performance of the models is quite poor on our benchmark. For instance, Gen-2 achieves the one of the highest video quality score (5.8 on LAION aesthetics classifier) but has poor semantic adherence and physical commonsense score of 7.6 (Table
Table 13:
Correlation between video quality (Aesthetics) and optical flow (motion) with physical commonsense and semantic adherence over VIDEOPHY dataset.
Metrics
Correlation
Aesthetics-Physicee Commonsense
0.3
Aesthetics-Semante Adherence
0.5
Motion Phhoonll Commonsense
-0.8
Motion-Semantic Adherence
-0.1
T FINETUNING VIDEO MODEL WITH VIDEOPHY DATA
This work is centered around physical commonsense evaluation, and we trained automatic evaluator (ViDEOCON- CCCN--SSSSSSSS using the training set. Here, we assess whether VIDEOPHY training set instances can also be used for finetuning video models. Specifically, we finetune Lumiere-T22 T2222 model on the instances from the training set of VIDEOPITY which achieve score of I on physical commonsense and score of I on semantic adherence. In total, there are 1000 such (video, caption) pairs in the dataset. Post-finetuning. we generate the videos for the test prompts and evaluate them using our automatic evaluator, VIDEOCON- -HCSCCCS.
Table 14: Finetuning Lumiere-T212V with the (video, text) pairs that achieve joint performance score of (i.e., PC=1 1 and SA = 1) in the train set of VIDEOPHY data. While the training set of the VIDEOPHY was primarily collected for training an automatic evaluator, we test whether it can also improve the video generative models itself.
Model
SA
PC
Average
46
25
35
Lumiere-T211V-Finette
36.5
24.6
30.5
31

---

Published as a conference paper at ICLR 2025
We present the results in Table 14 We find that the semantic adherence (video-text alignment) reduces by large margin and physical commonsense remains unchanged after finetuning. This be due to several factors: (a) the number of training samples is not enough, (b) optimization difficulties since the training videos are generated from several generative models (mix of on-policy and off off-policy videos), and (c) vanilla finetuning being bad algorithm for learning from these samples. Since post-training of video generative models is a less explored direction, there can be many ways to improve the generative model's physical commonsense. These results also show that mere finetuning with the samples in the training set of VideoPhy does not lead to large gains in the automatic evaluation on the set. Future work will focus on training better physical commonsense models using the insights provided in our work.
U MORE QUALITATIVE EXAMPLES OF POOR PHYSICAL COMMONSENSE
We present more examples from each generative model where one more physical laws are violated Figure Figure 26
32

---

Published as a conference paper at ICLR 2025
(a) A paddle mixes wet cement in a bucket
(b) A whisk whips cream to perfect fluffy consistency
(c) Hands rub luscious lotion dry skin
(d) The net catches the fast- fast-moving soccer ball
(e) Yogurt merging with strawberry puree
Figure 15:
Unphysical Generated Examples LaVIE. (a) Solid Constitutive Laws Violation: the
metal spoon should not deform; Nonphysical Penetration: the spoon unnaturally passes through the liquid. (b) Solid Constitutive Laws Violation: the whisk exhibits abnormal shape deformation. (c) Solid Constitutive Laws Violation: the two hands show abnormal shape deformation; Nonphysical Penetration:
fingers penetrate each other; Conservation of Mass Violation: the geometry (plus texture) of the two hands inconsistent texture) of the soccer is inconsistent over time; Newton's 's Second Law Violation: the soccer does not fall under gravity. (e) Conservation of Mass Violation: the volume of yogurt in the cup does increase as more yogurt is added.
time. (d) Conservation of Mass Violation: the geometry (plus
V EXAMPLES FROM DIVERSE STATES OF MATTER AND COMPLEXITY
We present few qualitative examples highlighting instances of good physical commonsense and bad physical commonsense in Figure Figure
33

---

Published as a conference paper at ICLR 2025
(a) A blender spins, mixing squeezed juice within it
(b) A teaspoon stirs sugar into a cup of coffee
(c) Hand flipping open book cover
(d) Soap washes grime off dirty hands
(e) Water pouring from watering can onto plants
Figure 16: Unphysical Generated Examples of Gen-2. (a) Conservation of Mass Violation: the
volume of juice in the blender increases over time without new substances being added. (b) Solid Constitutive Laws Violation: the metal spoon should not deform. (c) Conservation of Mass Violation: the volume of the book increases over time; Nonphysical Penetration: the fingers pass through the book. (d) Nonphysical Penetration: fingers penetrate into each other. (e) Newton's 's Second Law Violation: the flowing water appears to be static, ignoring the effect of gravity.
34

---

Published as a conference paper at ICLR 2025
(a) A foot crushing an empty soda can
(b) spinning wheel sprays muddy water
(c) Mustard squirting out of a plastic bottle onto hotdog
(d) Plastic frishee lands on lush grass lawn
(e) Pouring milk into still tea
Figure 17: Unphysical Generated Examples of VideoCrafter2. (a) Newton's Second Law Violation: the metal can deforms without being pressed.
(b) Newton's Second Law Violation: Water splashes while the rolling wheel remains static. (c) Newton's 's Second Law Violation: the bottle floats in the air, ignoring the effect of gravity; Fluid Constitutive Law Violation: the dripping and flowing of mustard unnatural. (d) Conservation of Mass Violation: the geometry (plus texture) of the frisbee is not consistent over time. (e) Conservation of Mass Violation: the total volume milk not increase as more milk is poured into; Nonphysical Penetration: milk penetrates the glass.
glass does
35

---

Published as a conference paper at ICLR 2025
(a) A futuristic hoverboard hovers just above the water
(b) A swimmer splashing in the sea water
(c) Frog leaping from one lilypad to another
(d) Plastic fidget spinner rotating rubber mat
(e) The eraser rubs against the paper, removing pencil marks
Figure 18: Unphysical Generated Examples of ZeroScope. (a) Newton's Second Law Violation: the motion of the hoverboard does not satisfy the momentum equation. (b) Newton's 's Second Law
Violation: the motion of the arm of the swimmer is unnatural. (c) Conservation of Mass Violation:
the geometry (texture) of the frog is inconsistent over time. (d) Newton's 's First Law Violation: the velocity of the fidget spinner changes despite being balanced state. (e) Solid Constitutive Laws Violation: the paper is torn apart without external forces but recovers later.
36

---

Published as a conference paper at ICLR 2025
POOOO
(a) A blender spins, mixing squeezed juice within it
(b) car gliding over road slick with rainwater
(c) A shaker mixes a delightful cocktail at the bar
(d) shiny coin takes a dive into a clear water fountain
(e) A stroller wheels through large puddle
Figure 19: Unphysical Generated Examples of OpenSora. (a) Solid Constitutive Laws Violation: the metal blender should not deform. (b) Newton's Second Law Violation: the car moves backward, violating the momentum equation (c) Solid Constitutive Laws Violation: the metal spoon deforms when stirring the cocktail. (d) Newton's 's First Law Violation: the coin moves on the ground back and forth without horizontal forees. (e) Conservation of Mass Violation: the left rear wheel disappears over time.
37

---

Published as a conference paper at ICLR 2025
(a) A perfume bottle spritzes fragrance into the air
(b) Lemon juice drops splash into water
(c) Loose sneaker swings on dangling foot
(d) Detergent flowing into a bucket of water
(e) The screwdriver tightens the metal screw in the wood
Figure 20: Unphysical Generated Examples of SVD-T212V. (a) Newton's 's Second Law Violation: the perfume spreads back and forth, violating the momentum equation. (b) Newton's 's Second Law Violation: the water drops float in the air, ignoring gravity. (c) Solid Constitutive Laws Violation: the leg exhibits unnatural deformation. (d) Newton's Second Law Violation: the water flows upward into the air without external forces. (e) Solid Constitutive Laws Violation: the screwdriver head deforms unnaturally.
38

---

Published as a conference paper at ICLR 2025
(a) Plastic fidget spinner rotating on rubber mat
THE
(b) Clasping a necklace around neck
(c) A whisk churns heavy cream into whipped cream
(d) A sailboat cuts through the choppy sea waves
(e) A diver plunges headlong into a sparkling pool
Figure 21: Unphysical Generated Examples of Pika. (a) Solid Constitutive Laws Violation: the fidget spinner should not deform. (b) Solid Constitutive Laws Violation: the necklace should not deform. (c) Conservation of Mass Violation: the volume of cream increases over time without additional input. (d) Fluid constitutive Law Violation: unnatural waves on the sea surface. (e) Solid Constitutive Law Violation: one diving shoe splits into two and detaches from the feet.
39

---

Published as a conference paper at ICLR 2025
(a) A spoon stirs a pot of vegetable soup
(b) A whisk spins in the egg mixture, mixing it thoroughly
(c) Coin spins rapidly on a wooden table
(d) Squeezing lemon drops into warm tea
(c) Tea accepts stream of milk
Figure 22: Unphysical Generated Examples of Lumiere-T2Y -2V. (a) Conservation of Mass Violation: the vegetable appears on spoon out of nowhere. (b) Solid Constitutive Laws Violation: the whisk should not deform. (c) Solid Constitutive Laws Violation: the coin splits into two and then merges back into one. (d) Solid Constitutive Laws Violation: the lemon shows an unnatural appearance change; Fluid Constitutive Laws Violation: the lemon juice appears like static glue. (e) Nonphysical Penetration: the tea flows through the cup.
40

---

Published as a conference paper at ICLR 2025
from from the beating stick (a) A drum vibrating
(b) A leaf falls delicately into slow-moving river
(c) A wooden spoon stirring soup in a pot
(d) Hand folds the paper
(e) Skateboard glides on the pavement
Figure 23: Unphysical Generated Examples of Lumiere-T212V. (a) Solid Constitutive Laws Violation: the drum stick head should not deform Newton's Second Law Violation: the leaf floats in the air, ignoring gravity. (c) Conservation of Mass Violation: the vegetable appears on the spoon out of nowhere. (d) Nonphysical Penetration: hands penetrate each other. (e) Solid Constitutive Laws Violation: one leg on the skateboard transforms into a person.
41

---

Published as a conference paper at ICLR 2025
(a) A diver splashing into the pool water
(b) Peeler peels apple
(c) Shoveling sand into bucket
(d) The pickaxe digs into the hard grounds
(e) The rock topples the carefully stacked pile of cans
Figure 24: Unphysical Generated Examples of CogVideoX-22 a) Newton's Second Law Violation: the splash appears without any external force, violating the principle of momentum conservation. (b) Solid Constitutive Law Violation: apple undergoes deformation, which should not occur. (c) Conservation of Mass Violation: the sand's volume changes over time without the addition of new material. (d) Conservation of Mass Violation: the geometry and texture of the pickaxe change inconsistently over time. (c) Solid Constitutive Law Violation: the cans exhibit unnatural deformations.
42

---

Published as a conference paper at ICLR 2025
(a) spinning wheel sprays muddy water
(b) Dominoes toppling one after another on the table
(c) Leather glove catching a hard baseball
(d) Milk blending seamlessly into tea
(e) The net catches the fast-moving soccer ball
Figure 25: Unphysical Generated Examples of Cog CogVideoX-5B. (a) Newton's 's Second Law Violation: the wave dynamics are discontinuous over time, violating the momentum conservation principle. (b) Conservation of Mass Violation: the geometry and texture of the dominoes change inconsistently over time. (c) Solid Constitutive Law Violation: the leather glove exhibits unnatural deformations. (d) Conservation of Mass Violation: the volume of tea remains unchanged despite the addition of milk the glass cup. (e) Solid Constitutive Law Violation: the soccer ball displays unnatural and discontinuous deformations over time.
43

---

Published as a conference paper at ICLR 2025
(a) A brave diver splashes into a pool from great height
(b) Coin flicknn into sparkling, fountain
(c) Metal grinder crushing coffee beans
(d) Oil cascades into vinegar for vinaigrette
(e) Water streams into fresh juice
Figure 26: Unphysical Generated Examples of Dream Machine. (a) Newton's Second Law Violation: the diver floats in mid-air, defying gravity.
(b) Newton's 's Second Law Violation: the coin hovers, disregarding gravitational forces. (c) Conservation of Mass Violation: numerous coffee beans appear spontaneously without source. (d) Nonphysical Penetration: oil and vinegar pass through the glass cup. (e) Conservation of Mass Violation: the juice volume remains unchanged despite the addition of water.
44

---

Published as a conference paper at ICLR 2025
PC=0
PC=1
Easy
(a) Rain creates concentric ripples water.
(c) Rain splashing on a still pond.
(b) Water droplet bouncing on water surface.
(d) Raindrops disturb quiet puddles.
Hard
(4) Juice and fizz shake bottle.
(g) Perfume mist diffusing through the
(1) Yogurt merging with strawberry puree.
(h) Pouring milk into still tea.
Figure 27: Qualitative examples in the fluid-fluid category. Videos in the left column have majority PC score of O, while videos in the right column have a majority PC score of 1. (a) The central ripple does not vanish even in absence of raindrops. (b) The water droplet is floating upwards, defying gravity. (e) The total volume juice is increasing. (f) The color of the yogurt not consistent over time.
45

---

Published as a conference paper at ICLR 2025
PC 0
PC 1
0
6
Easy
The wristwatch knob winds the inner spring tightly.
(c) Cloth banner hanging from wooden twig.
(b) Tying rope to pole.
(d) Wooden swing dangles over the sand in the sandpit.
Hard
(e) Darts lands in the middle of the cork dariboadd.
(g) Metal grinder crushing coffee beans.
(f) Ripping paper sheet by hand.
(h) Shoveling sand into a bucket.
Figure 28: Qualitative examples in the solid-solid category. Videos in the left column have a majority PC score of 0. while videos in the right column have majority PC score of 1. (a) The hands of the clock have illogical motion. (b) One piece of the robe disappears. (e) The geometry and texture of the dart are consistent time. (f) The total volume of the sheet of paper time.
consistent
46

---

Published as a conference paper at ICLR 2025
PC=
PC
Easy
(a) A toy floats on a tranquil neighborhood hood pond.
(c) A leaf floating on the stream water.
(b) Chopsticks dipping a sushi roll into soy sauce.
(d) A small paper boat drifts along a stream.
Hard
(c) Feet kick and splash about in a rain puddle.
(g) A wine bottle pours a red blend into glass.
( Water spraying from a garden hose onto plants.
(h) Foot kicks water in swimming pool.
Figure Qualitative examples in the fluid-fluid category. Videos in the left column have a majority PC score of O, while videos in the right column have majority PC score of 1. (a) The geometry and color the duck head changes over time. (b) One chopstick appears from nowhere. (e) One leg appears from nowhere. (f) The geometry and texture of the leaves not consistent over time.
47


Reviews:
[{"review_id": "yNXQ61AeDA", "reviewer": "Reviewer_tYi4", "summary": "This work targets at building a benchmark that can evaluate the physical commonsense for video generation. Multiple physics-related prompts are first generated and evaluated, serving as the input for different video generators. With human evaluation, it is found that both open-source and closed-source, and find that they significantly lack physical commonsense and semantic adherence capabilities. In order to evaluate in scale, an auto-evaluator is trained.", "questions": "Please see Weaknesses.", "limitations": "", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "The goal of evaluating physical commonsense is quite important for this area. \nThe chosen video generators are comprehensive, including open-source models and closed ones. \nThe presentation is well-organized and easy to follow, with key numbers regarding evaluations.", "weaknesses": "Regarding evaluation, I have several major concerns.\n\n- As mentioned in Sec.3.1, binary feedback (0/1) is used to evaluate semantic adherence and physical commonsense. This discrete value may not reflect and moniter the true capability for different video generators. For example, for a text prompt with 10 physical movements, one generator achieves 8 movements while another is 6. These binary feedback can not tell the gap between two candidates. This example could be too extreme while that could be a weakness of binary value.\n- Besides, I am not sure whether the absolute accuracy of physical achievements is a proper metric.  Especially for Fig.1, I believe the relative scores across different generators (like ELO score) make more sense, which also avoid the weakness of binary feedback\n- Regarding physical commonsense (PC), it really depends on the text following ability of given generators (semantic adherence in this work).  Joint performance may be one alternative for both text and physics evaluation while the posterior probability may be one perspective for physical commonsense alone.\n\nFor auto-evaluator videocon-physics, the fine-tuning details could facilitate the reproduction and transferability from VIDEOCON to other video-language models.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "ZCyaNcatwv", "reviewer": "Reviewer_A9xM", "summary": "This paper presents a benchmark designed to evaluate the physical commonsense of videos generated by text-to-video models. It highlights significant gaps in these models' ability to accurately simulate real-world physics and adhere to caption prompts. It introduces VideoPhy, a dataset consisting of real-world interaction prompts, and VideoCon-Physics, an automatic evaluation pipeline. Evaluation shows that even the best models, such as CogVideoX-5B, only achieve a 39.6% adherence rate to physical laws, emphasizing the need for improvements in video generation models.", "questions": "Please see the weaknesses.", "limitations": "", "rating": 6, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 4, "strengths": "* The paper shifts attention from general visual and semantic quality to the capability of T2V models to simulate real-world physics, addressing a vital aspect of realism in video generation.\n* The paper provides detailed insights into different failure modes, guiding future model improvements and research directions.\n* The automation pipeline, VideoCon-Physics enables scalable assessment of semantic adherence and physical commonsense in generated videos, which can be useful and meaningful to the research communities on T2V generation.", "weaknesses": "* Among the T2V models used for comparison, some still frequently fail to reproduce the scenarios specified by the text prompts. For example, in assessing physical reasoning in a scenario where milk is being poured, one needs to verify whether the milk appropriately fills the cup. However, in practice, these models often fail even to generate a video depicting the act of pouring milk. In such cases, the benchmark may be more influenced by the general video generation capabilities of the models rather than their physical commonsense reasoning abilities, as shown in the similar trends observed between Semantic Adherence and Physical Commonsense scores in Table 4. Therefore, it seems necessary to conduct experiments that assess physical commonsense only on generated videos that have appropriate semantic adherence, ensuring that the evaluation focuses on the models' understanding of physical phenomena rather than their basic ability to generate relevant videos.\n\n* Proposed automating evaluations using a video-to-text model relies on a assumption: the V2T model must have a better understanding of physical phenomena than the T2V model. I remain some concerns about the justification of this assumption, as it is essential for the validity of the automated evaluation method. Without depending on this assumption, it is quite reasonable to suggest that the proposed V2T model can evaluate the T2V model because it has been fine-tuned on data containing these physical phenomena. In this context, I'm curious what it would be if the T2V model is similarly fine-tuned on a portion (training split) of the VideoPhy dataset and then generates videos based on prompts from the test split.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "bfzScBNfM5", "reviewer": "Reviewer_xyoE", "summary": "This paper introduces VideoPHY, a benchmark consisting of 688 captions designed to evaluate text-to-video models on physical commonsense. This work focuses on the real-world activities and interactions, classified by material interactions into three categories: solid-solid, solid-fluid, and fluid-fluid.\n\nThe dataset was initially generated using GPT-4 and then refined through manual filtering, and was annotated with binary (0/1) difficulty levels. Evaluation was based on two binary (0/1) metrics Semantic Adherence and Physical Commonsense, assessed by human or VLM. For the VLM, the authors fine-tuned VideoCon on human annotations, creating VideoCon-Physics.\n\nExperiments comparing current SOTA text-to-video models including both open and closed models, indicate that current models struggle to model physical activities well.", "questions": "see weaknesses above.\n\n1. Given the challenges around categorization raised above, are there plans to expand or refine this benchmark to incorporate a broader range of interaction types or more nuanced physical properties, beyond basic material categories? \n2. Have the authors considered using more detailed metrics to better decouple the concepts?", "limitations": "", "rating": 5, "confidence": 2, "soundness": 2, "presentation": 3, "contribution": 2, "strengths": "1. This paper presents a physical commonsense benchmark that addresses a gap in existing datasets. The field of video generation needs such a benchmark, as t2v models gain popularity partly for their potential as world simulators or physical engines. \n2. The dataset covers a wide range of activities, interactions, and dynamics on various materials as mentioned in line 194/195. The classification of dataset is somehow inspired by graphics field, and the difficulty of simulate those dynamics is also considered, which is a very interesting aspect for constructing the dataset.\n3. Experiments are conducted over most of the current t2v models except for those without API support, providing a comprehensive comparison across a wide range of models.\n4. This work provides insight and contributes to further improving t2v models.", "weaknesses": "1. I am not an expert in graphics or materials, so I am very uncertain about the category definitions and category ratios. I can get the idea of categorizing based on the state of matter, and solid and liquid are the most common. However, in graphics, rigid bodies, soft bodies, particle systems, fabrics, characters and animals are distinct topics that rely on very different physical models, whereas fluid dynamics, such as inviscid and viscous flows, are comparatively less diverse. If the idea of this benchmark is focusing on physical interactions and dynamics, then categorizing by interaction types or physical properties rather than broad material types can be more informative and nuanced. On the category ratios, as stated in line 179-181 as solids involve more physical constitutive models, we might expect more cases of solid-solid interactions than solid-fluid interactions, yet the sample counts are nearly equal (289 and 291, respectively). \n2. Each sample in the training set for VideoCon-Physics was only labeled by one human annotator, while the author also mentioned while annotating benchmark, the agreement of three human annotators is 75% and 70% on SA and PC. This makes the training set less trustworthy.\n3. I am also very skeptical about the motivation of fine-tuning VideoCon for direct evaluation of semantic adherence and physical commonsense. Those two metrics seem to cover a wide range of concepts but only one number between 0 and 1 was given. Directly fine-tuning VideoCon to evaluate especially the physical commonsense without introducing any extra knowledge, reasoning, or explicitly modeling does not make sense to me. The improvement of metrics might result from dataset domain bias, but there is no analysis on that. And from the leader boards of human evaluation and automatic evaluation, the disagreement on models such as Luma Dream Machine and Pika is not negligible. I suggest authors provide more detailed disagreement or evaluation statistics.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "G6U83apVj6", "reviewer": "Reviewer_dBNE", "summary": "The paper presents a benchmark for evaluating physical commonsense for video generation. The benchmark includes 1) high-quality, human-verified captions; 2) automatic evaluation models to evaluate the video generation models; 3) the generated videos using existing methods and human annotations for these videos. Also, the authors reveal some conclusions based on the benchmark results, which could provide insights to the community.", "questions": "Please see the weakness part for my concerns, but it is just a minor one.", "limitations": "", "rating": 8, "confidence": 4, "soundness": 4, "presentation": 4, "contribution": 4, "strengths": "1. The paper is well-written and easy to follow.\n2. The paper presents a high-quality benchmark for evaluating physical commonsense for video generation. Careful data curation for text prompts is employed to ensure the quality of the text prompt.\n3. The authors provide a comprehensive analysis of the text prompt used for evaluation. The category is balanced.\n4. Beyond human evaluation, the authors also provide a model for automatic evaluation by fine-tuning a model for evaluating the physical commonsense of the model. The model eases the use of the benchmark for future research.\n5. The authors also will release the generated videos and human evaluation for future research, which will potentially boost the performance of the automatic evaluation model.", "weaknesses": "1. For the VIDEOCON-PHYSICS model, have the authors conducted a human evaluation for the performance? Since the data used for training this model comes from same human annotators, the trained model may be biased. It would be better if the model can provide a human evaluation for the evaluation results generated by the VIDEOCON-PHYSICS model (just for the check of the correctness of the results provided by the VIDEOCON-PHYSICS model)", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (Poster)