VideoPhy: Evaluating Physical Commonsense for Video Generation

Authors: Hritik Bansal, Zongyu Lin, Tianyi Xie, Zeshun Zong, Michal Yarom, Yonatan Bitton, Chenfanfu Jiang, Yizhou Sun, Kai-Wei Chang, Aditya Grover
Primary area: datasets and benchmarks
Keywords: text-to-video generation, physical commonsense, video-text alignment, generative modeling, video evaluation

Abstract:
Recent advances in internet-scale video data pretraining have led to the development of text-to-video generative models that can create high-quality videos across a broad range of visual concepts, synthesize realistic motions and render complex objects. Hence, these generative models have the potential to become general-purpose simulators of the physical world. However, it is unclear how far we are from this goal with the existing text-to-video generative models. To this end, we present VideoPhy, a benchmark designed to assess whether the generated videos follow physical commonsense for real-world activities (e.g. marbles will roll down when placed on a slanted surface). Specifically, we curate diverse prompts that involve interactions between various material types in the physical world (e.g., solid-solid, solid-fluid, fluid-fluid). We then generate videos conditioned on these captions from diverse state-of-the-art text-to-video generative models, including open models (e.g., CogVideoX) and closed models (e.g., Lumiere, Dream Machine). Our human evaluation reveals that the existing models severely lack the ability to generate videos adhering to the given text prompts, while also lack physical commonsense. Specifically, the best performing model, CogVideoX-5B, generates videos that adhere to the caption and physical laws for 39.6% of the instances. VideoPhy thus highlights that the video generative models are far from accurately simulating the physical world. Finally, we propose an auto-evaluator, VideoCon-Physics, to assess the performance reliably for the newly released models. The code is available here: https://github.com/Hritikbansal/videophy.

Decision: accept (poster)
Reviewer scores: 6, 6, 5, 8

Reviews and discussion:
<thread>
creation date:
1730646240232

thread_type:
official_review

<thread_object>
creator: reviewer_dbne

<content>
summary:
The paper presents a benchmark for evaluating physical commonsense for video generation. The benchmark includes 1) high-quality, human-verified captions; 2) automatic evaluation models to evaluate the video generation models; 3) the generated videos using existing methods and human annotations for these videos. Also, the authors reveal some conclusions based on the benchmark results, which could provide insights to the community.

strengths:
1. The paper is well-written and easy to follow.
2. The paper presents a high-quality benchmark for evaluating physical commonsense for video generation. Careful data curation for text prompts is employed to ensure the quality of the text prompt.
3. The authors provide a comprehensive analysis of the text prompt used for evaluation. The category is balanced.
4. Beyond human evaluation, the authors also provide a model for automatic evaluation by fine-tuning a model for evaluating the physical commonsense of the model. The model eases the use of the benchmark for future research.
5. The authors also will release the generated videos and human evaluation for future research, which will potentially boost the performance of the automatic evaluation model.

weaknesses:
1. For the VIDEOCON-PHYSICS model, have the authors conducted a human evaluation for the performance? Since the data used for training this model comes from same human annotators, the trained model may be biased. It would be better if the model can provide a human evaluation for the evaluation results generated by the VIDEOCON-PHYSICS model (just for the check of the correctness of the results provided by the VIDEOCON-PHYSICS model)

questions:
Please see the weakness part for my concerns, but it is just a minor one.
</content>
</thread_object>

</thread>
<thread>
creation date:
1730675083710

thread_type:
official_review

<thread_object>
creator: reviewer_xyoe

<content>
summary:
This paper introduces VideoPHY, a benchmark consisting of 688 captions designed to evaluate text-to-video models on physical commonsense. This work focuses on the real-world activities and interactions, classified by material interactions into three categories: solid-solid, solid-fluid, and fluid-fluid.

The dataset was initially generated using GPT-4 and then refined through manual filtering, and was annotated with binary (0/1) difficulty levels. Evaluation was based on two binary (0/1) metrics Semantic Adherence and Physical Commonsense, assessed by human or VLM. For the VLM, the authors fine-tuned VideoCon on human annotations, creating VideoCon-Physics.

Experiments comparing current SOTA text-to-video models including both open and closed models, indicate that current models struggle to model physical activities well.

strengths:
1. This paper presents a physical commonsense benchmark that addresses a gap in existing datasets. The field of video generation needs such a benchmark, as t2v models gain popularity partly for their potential as world simulators or physical engines. 
2. The dataset covers a wide range of activities, interactions, and dynamics on various materials as mentioned in line 194/195. The classification of dataset is somehow inspired by graphics field, and the difficulty of simulate those dynamics is also considered, which is a very interesting aspect for constructing the dataset.
3. Experiments are conducted over most of the current t2v models except for those without API support, providing a comprehensive comparison across a wide range of models.
4. This work provides insight and contributes to further improving t2v models.

weaknesses:
1. I am not an expert in graphics or materials, so I am very uncertain about the category definitions and category ratios. I can get the idea of categorizing based on the state of matter, and solid and liquid are the most common. However, in graphics, rigid bodies, soft bodies, particle systems, fabrics, characters and animals are distinct topics that rely on very different physical models, whereas fluid dynamics, such as inviscid and viscous flows, are comparatively less diverse. If the idea of this benchmark is focusing on physical interactions and dynamics, then categorizing by interaction types or physical properties rather than broad material types can be more informative and nuanced. On the category ratios, as stated in line 179-181 as solids involve more physical constitutive models, we might expect more cases of solid-solid interactions than solid-fluid interactions, yet the sample counts are nearly equal (289 and 291, respectively). 
2. Each sample in the training set for VideoCon-Physics was only labeled by one human annotator, while the author also mentioned while annotating benchmark, the agreement of three human annotators is 75% and 70% on SA and PC. This makes the training set less trustworthy.
3. I am also very skeptical about the motivation of fine-tuning VideoCon for direct evaluation of semantic adherence and physical commonsense. Those two metrics seem to cover a wide range of concepts but only one number between 0 and 1 was given. Directly fine-tuning VideoCon to evaluate especially the physical commonsense without introducing any extra knowledge, reasoning, or explicitly modeling does not make sense to me. The improvement of metrics might result from dataset domain bias, but there is no analysis on that. And from the leader boards of human evaluation and automatic evaluation, the disagreement on models such as Luma Dream Machine and Pika is not negligible. I suggest authors provide more detailed disagreement or evaluation statistics.

questions:
see weaknesses above.

1. Given the challenges around categorization raised above, are there plans to expand or refine this benchmark to incorporate a broader range of interaction types or more nuanced physical properties, beyond basic material categories? 
2. Have the authors considered using more detailed metrics to better decouple the concepts?
</content>
</thread_object>

</thread>
<thread>
creation date:
1730696006434

thread_type:
official_review

<thread_object>
creator: reviewer_a9xm

<content>
summary:
This paper presents a benchmark designed to evaluate the physical commonsense of videos generated by text-to-video models. It highlights significant gaps in these models' ability to accurately simulate real-world physics and adhere to caption prompts. It introduces VideoPhy, a dataset consisting of real-world interaction prompts, and VideoCon-Physics, an automatic evaluation pipeline. Evaluation shows that even the best models, such as CogVideoX-5B, only achieve a 39.6% adherence rate to physical laws, emphasizing the need for improvements in video generation models.

strengths:
* The paper shifts attention from general visual and semantic quality to the capability of T2V models to simulate real-world physics, addressing a vital aspect of realism in video generation.
* The paper provides detailed insights into different failure modes, guiding future model improvements and research directions.
* The automation pipeline, VideoCon-Physics enables scalable assessment of semantic adherence and physical commonsense in generated videos, which can be useful and meaningful to the research communities on T2V generation.

weaknesses:
* Among the T2V models used for comparison, some still frequently fail to reproduce the scenarios specified by the text prompts. For example, in assessing physical reasoning in a scenario where milk is being poured, one needs to verify whether the milk appropriately fills the cup. However, in practice, these models often fail even to generate a video depicting the act of pouring milk. In such cases, the benchmark may be more influenced by the general video generation capabilities of the models rather than their physical commonsense reasoning abilities, as shown in the similar trends observed between Semantic Adherence and Physical Commonsense scores in Table 4. Therefore, it seems necessary to conduct experiments that assess physical commonsense only on generated videos that have appropriate semantic adherence, ensuring that the evaluation focuses on the models' understanding of physical phenomena rather than their basic ability to generate relevant videos.

* Proposed automating evaluations using a video-to-text model relies on a assumption: the V2T model must have a better understanding of physical phenomena than the T2V model. I remain some concerns about the justification of this assumption, as it is essential for the validity of the automated evaluation method. Without depending on this assumption, it is quite reasonable to suggest that the proposed V2T model can evaluate the T2V model because it has been fine-tuned on data containing these physical phenomena. In this context, I'm curious what it would be if the T2V model is similarly fine-tuned on a portion (training split) of the VideoPhy dataset and then generates videos based on prompts from the test split.

questions:
Please see the weaknesses.
</content>
</thread_object>

</thread>
<thread>
creation date:
1730704027940

thread_type:
official_review

<thread_object>
creator: reviewer_tyi4

<content>
summary:
This work targets at building a benchmark that can evaluate the physical commonsense for video generation. Multiple physics-related prompts are first generated and evaluated, serving as the input for different video generators. With human evaluation, it is found that both open-source and closed-source, and find that they significantly lack physical commonsense and semantic adherence capabilities. In order to evaluate in scale, an auto-evaluator is trained.

strengths:
The goal of evaluating physical commonsense is quite important for this area. 
The chosen video generators are comprehensive, including open-source models and closed ones. 
The presentation is well-organized and easy to follow, with key numbers regarding evaluations.

weaknesses:
Regarding evaluation, I have several major concerns.

- As mentioned in Sec.3.1, binary feedback (0/1) is used to evaluate semantic adherence and physical commonsense. This discrete value may not reflect and moniter the true capability for different video generators. For example, for a text prompt with 10 physical movements, one generator achieves 8 movements while another is 6. These binary feedback can not tell the gap between two candidates. This example could be too extreme while that could be a weakness of binary value.
- Besides, I am not sure whether the absolute accuracy of physical achievements is a proper metric.  Especially for Fig.1, I believe the relative scores across different generators (like ELO score) make more sense, which also avoid the weakness of binary feedback
- Regarding physical commonsense (PC), it really depends on the text following ability of given generators (semantic adherence in this work).  Joint performance may be one alternative for both text and physics evaluation while the posterior probability may be one perspective for physical commonsense alone.

For auto-evaluator videocon-physics, the fine-tuning details could facilitate the reproduction and transferability from VIDEOCON to other video-language models.

questions:
Please see Weaknesses.
</content>
</thread_object>

</thread>
<thread>
creation date:
1732497587804

thread_type:
official_comment

<thread_object>
creator: authors

<content>
title:
Revised paper update

comment:
We have uploaded the revised version of the paper which addresses most of the comments from the reviewers (highlighted in blue).
</content>
</thread_object>

</thread>


Metareview:
metareview:
The paper addresses an important gap in evaluating physical commonsense in text-to-video (T2V) generation models. The proposed VideoPhy benchmark is insightful, covering diverse physical interactions and revealing significant shortcomings in current models. Reviewers appreciated the comprehensive evaluation, clear presentation, and the automation pipeline VideoCon-Physics. Concerns include the limitations of binary feedback, potential biases in annotations, and the need for more nuanced metrics. The authors' revisions addressed these issues effectively, leading to improved reviewer ratings. Overall, this work is a valuable contribution to the community. The AC recommends acceptance.