Towards World Simulator: Crafting Physical Commonsense-Based Benchmark for Video Generation

Authors: Fanqing Meng, Jiaqi Liao, Xinyu Tan, Wenqi Shao, Quanfeng Lu, Kaipeng Zhang, Yu Cheng, Dianqi Li, Yu Qiao, Ping Luo
Primary area: datasets and benchmarks
Keywords: World Simulator, Physical Commonsense, Video Generation, Evaluation

Abstract:
Text-to-video (T2V) models like Sora have made significant strides in visualizing complex prompts, which is increasingly viewed as a promising path towards constructing the universal world simulator. Cognitive psychologists believe that the foundation for achieving this goal is the ability to understand intuitive physics. However, the capacity of these models to accurately represent intuitive physics remains largely unexplored. To bridge this gap, we introduce PhyGenBench, a comprehensive \textbf{Phy}sics \textbf{Gen}eration \textbf{Ben}chmark designed to evaluate physical commonsense correctness in T2V generation. PhyGenBench comprises 160 carefully crafted prompts across 27 distinct physical laws, spanning four fundamental domains, which could comprehensively assesses models' understanding of physical commonsense. Alongside PhyGenBench, we propose a novel evaluation framework called PhyGenEval. This framework employs a hierarchical evaluation structure utilizing appropriate advanced vision-language models and large language models to assess physical commonsense. Through PhyGenBench and PhyGenEval, we can conduct large-scale automated assessments of T2V models' understanding of physical commonsense, which align closely with human feedback. Our evaluation results and in-depth analysis demonstrate that current models struggle to generate videos that comply with physical commonsense. Moreover, simply scaling up models or employing prompt engineering techniques is insufficient to fully address the challenges presented by PhyGenBench (e.g., dynamic scenarios). We hope this study will inspire the community to prioritize the learning of physical commonsense in these models beyond entertainment applications. We will release the data and codes at https://github.com/PhyGenBench/PhyGenBench

Decision: reject
Reviewer scores: 5, 5, 6, 5, 5

Reviews and discussion:
<thread>
creation date:
1730115336100

thread_type:
official_review

<thread_object>
creator: reviewer_esnr

<content>
summary:
The paper proposes PhyGenBench and PhyGenEval. PhyGenBench is a benchmark with about 160 text prompts used to evaluate models' video generation ability on physics-related text prompts. PhyGenEval is an evaluation framework of PhyGenBench, used to automatically assess the video quality of physics laws, via GPT-prompted questions and VLM perception.

strengths:
- The topic is novel and interesting. Evaluating physics in AI-generated videos is really important. This paper is the first one on this topic as far as I know.

- Experiments show that PhyGenEval is closer to human value.

weaknesses:
- The PhyGenBench is a dataset with 160 text prompts. As a comparison, for the works mentioned in this paper, VideoPhy has 688 prompts with 36.5k human annotations, and DEVIL has more than 800 prompts. Only 160 text prompts may not represent the full complexity of physics law.

- In PhyGenEval, the overall score is set on a four-point scale, but even the top-performing video generation model scores only 0.5 on average. That means the model gets a 0 score in more than half of the test cases. This suggests that the evaluation metric might be overly strict, potentially limiting its effectiveness in distinguishing between models. Such stringent scoring could reduce the benchmark’s ability to accurately reflect model performance differences.

- Since the topic is related to evaluating the physics in generative models, I think it is better to add some discussion on physical reasoning benchmarks in related works, which has been a heated debate topic, such as SuperCLEVR-Physics[1], ContPhy[2], Physion[3] and so on.

[1] Wang, X., Ma, W., Wang, A., Chen, S., Kortylewski, A., & Yuille, A. (2024). Compositional 4D Dynamic Scenes Understanding with Physics Priors for Video Question Answering. ArXiv. https://arxiv.org/abs/2406.00622

[2] Zheng, Z., Yan, X., Chen, Z., Wang, J., Lim, Q. Z., Tenenbaum, J. B., & Gan, C. (2024). ContPhy: Continuum Physical Concept Learning and Reasoning from Videos. ArXiv. https://arxiv.org/abs/2402.06119

[3] Bear, D. M., Wang, E., Mrowca, D., Binder, F. J., Tung, H., Pramod, R. T., Holdaway, C., Tao, S., Smith, K., Sun, F., Kanwisher, N., Tenenbaum, J. B., Yamins, D. L., & Fan, J. E. (2021). Physion: Evaluating Physical Prediction from Vision in Humans and Machines. ArXiv. https://arxiv.org/abs/2106.08261

questions:
See Weakness above.
</content>
</thread_object>

</thread>
<thread>
creation date:
1730602697625

thread_type:
official_review

<thread_object>
creator: reviewer_qell

<content>
summary:
This paper introduces PhyGenBench, a new benchmark for evaluating the physical commonsense capabilities of Text-to-Video (T2V) models, particularly their understanding of intuitive physics. It includes prompts across 27 physical laws within four domains: mechanics, optics, thermal, and material properties. To evaluate performs on this benchmark, the authors propose PhyGenEval, a hierarchical evaluation framework using advanced vision-language models (VLMs) and large language models. Experimental results reveal that current T2V models lack robust physical commonsense, underscoring the gap between these models and true world simulators.

strengths:
- This paper clearly have great novelty. It focus on intuitive physics is unique and addresses an important gap in T2V evaluation.
- PhyGenEval's three tiered framework (key phenomena detection, order verification, and overall naturalness) thoroughly assesses physical realism.
- By getting more attention on the gap in physical commonsense, the benchmark provides great insights on how to improve video generation models to become a real world simulator.

weaknesses:
- The paper includes extensive comparisons to demonstrate PhyGenEval’s effectiveness, suggesting that a two-stage evaluation strategy may align more closely with human judgments for both InternVideo2 and GPT-4-o. Line 965 also notes that alternative open-source models achieve a high correlation coefficient with human evaluations. However, it appears that the main results rely on a specific version of GPT-4-o, which is not explicitly mentioned. As a benchmark, would future users need to evaluate all baselines and methods on updated versions of GPT-4-o to ensure fair comparisons? While the paper suggests that evaluation costs are minimal, I am concerned that this reliance on a specific model version may affect consistency. Have the authors considered using other LVLMs in place of GPT-4-o?
- Certain T2I models may perform poorly on specific prompts. I am not fully convinced that the proposed evaluation method can robustly handle these lower-quality videos.
- The issue of hallucination in large language models (LLMs) does not appear to be addressed in the evaluation protocol, potentially impacting the reliability of the benchmark. It would be beneficial if the authors considered this factor in their assessment framework.

- The author promised more human evaluation results in Appendix C.2 but this result seems under Appendix C.1. The writing seems to be confusing. Also between line 899 and 905, I believe the annotation should be done more rigorously. I am expecting carefully validate results from human annotators or I think the results can be noisy. I think showing the instructions to the human annotators can be particularly helpful.

questions:
- What is "the final score is calculated as 0 according to 4.2" (line 292)? Is the example in Figure receive 0 after this physical commonsense evaluation?

- It seems like the entire evaluation rely on closed sourced LLM: GPT-4o. If in the future, GPT-4o becomes unavailable, how should people compare results?

- some typos such as we pue more detailed (line 410), Appendix C.2 (line 418)
</content>
</thread_object>

</thread>
<thread>
creation date:
1730625853423

thread_type:
official_review

<thread_object>
creator: reviewer_zecl

<content>
summary:
The paper discusses the limitations of current text-to-video (T2V) models like Sora in accurately representing intuitive physics, which is essential for creating a universal world simulator. Understanding intuitive physics is foundational for such a simulator, yet T2V models' performance in this area remains under-explored. To address this gap, the authors introduce PhyGenBench, a Physics Generation Benchmark designed to test T2V models' grasp of physical commonsense and provide a comprehensive assessment of models' understanding of physics.

Also, the paper presents PhyGenEval, a new evaluation framework that uses advanced vision-language and large language models in a hierarchical structure to assess physical commonsense accurately. This dual framework allows for large-scale, automated evaluations aligned with human feedback.

Overall, the paper is well-written and propose a great benchmark for video generation evaluation.

strengths:
The paper is well-written and the contributions are very clear. The proposed benchmark for video generation models focuses on evaluate the physical understanding of video generation models, which is crucial and not well-studied. The evaluation strategy is clear and comprehensive.

weaknesses:
I mainly have two questions about the paper.

1) As PhyGenEval uses VLMs for scoring, I would like to know the effect of different VLMs. For example, GPT-4o for closed-source models and InternVL-2, LLaVA-Video, Oryx, these open-source models that can understand videos. I'm wondering if these models can consistently evaluate the generated videos, which may be an interesting question and I think it should be discussed in the paper to show a more comprehensive understanding of the proposed evaluation pipeline.

2) As the evaluation pipeline needs a lot of retrieval, I'd like to know the success rate of retrieval with GPT-4o. It is crucial for the overall evaluation and I hope the author can provide more details about how to ensure the retrieval is correct.

questions:
As stated in the weakness section.
</content>
</thread_object>

</thread>
<thread>
creation date:
1730676877720

thread_type:
official_review

<thread_object>
creator: reviewer_ppcz

<content>
summary:
In this paper, the authors focus on the evaluation of text-to-video models. To this end, they propose a new benchmark as well as a new evaluation method. Named PhyGenBench, the dataset of prompts evaluate intuitive physics in subcontexts such as mechanics, optics, thermal dynamics, and material properties. Alongside this benchmark is PhyGenEval, an automated eval pipeline where a VLM is combined with GPT-4o to generate evaluative questions and answers. The authors compare PhyGenEval against human evaluations. They also perform some initial experiments with contemporary video models on their proposed benchmark.

strengths:
1. Current generative models of video have serious issues with intuitive physics, and the research community needs a good benchmark to evaluate this capability. The proposed benchmark dataset can serve as a very important dataset for the community.

2. Evaluating intuitive physics can be difficult, and PhyGenEval might be a promising method to automate evaluation without the need for human raters. 

3. The quantitative evaluation of current video models on the benchmark is a strong contribution and shows the need to improve these models in the realm of intuitive physics.

weaknesses:
While PhyGenEval is an interesting approach to automating evaluation on the benchmark, I assert that the approach has some issues and should not be adopted by the community at this moment as a standard evaluation; human raters should be used:

1. The pipeline is not reliable enough, the PCA correlation results are only around .7 - .8

2. The pipeline relies on proprietary models such as GPT-4o and may be difficult to reproduce with open models.

questions:
I believe PhyGenBench is an excellent contribution for the research community, but the PhyGenEval as described is problematic for the reasons listed above. Is it possible that PhyGenEval be described as a potential approach for automated evaluation to be iterated on in subsequent research, with PhyGenBench + human evaluation as the main contribution?
</content>
</thread_object>

</thread>
<thread>
creation date:
1730681800040

thread_type:
official_review

<thread_object>
creator: reviewer_wxrw

<content>
summary:
Although T2V models have shown great progress in generating good media-level content, this paper challenges their capability to become the real world simulator. This paper first proposes a PhyGenBench, 160 T2V prompts composed of several physics categories, then proposes a hierarchical framework to evaluate semantic alignment and physics commonsense alignment. It shows that current models, even ones with large scales, struggle with physical commonsense.

strengths:
(1) This paper handcrafts T2V prompts in a fine-grained way.
(2) This paper provides a carefully designed pipeline to conduct the evaluation of physics commonsense.

weaknesses:
(1) The paper lacks decent novelty in terms of benchmark and evaluator itself. The way it constructs the evaluator heavily relies on several generative models. For example, using GPT4o to do information extraction and create questions sometimes brings about hallucination. Also, using VLMs in different stages can also lead to hallucination. Since it is a complex pipeline composed of different stages, error propagation might happen.

(2) The comparison with other baselines is unfair. The comparisons with other baselines are biased. Although they acknowledge that alternative auto-evaluators lack robustness, they do not demonstrate whether their own auto-evaluator performs effectively on prompts from different benchmarks as part of a generalization analysis. Typically, like concurrent work, these kinds of auto-evaluators are tailored to specific prompt distributions. Basically, the generalization of the reward modeling for world simulators should be enough for another paper.

(3) The number of prompts engaged in this paper are limited, which might be a weak signal for evaluating the video generation models as a world simulator.

questions:
(1) What is the efficiency of using your auto-evaluator? Could you provide an estimation?

(2) Could you provide some error analysis on the bad cases where PhyGenEval is opposite to the human eval? Maybe this can provide some insight on how to further improve the reward modeling of world simulators.
</content>
</thread_object>

</thread>
<thread>
creation date:
1732442630380

thread_type:
official_comment

<thread_object>
creator: authors

<content>
title:
General Response-Part1

comment:
We sincerely thank all reviewers for their valuable feedback and the time they have dedicated to reviewing our work. Your comments have been extremely helpful in guiding our revisions and improving the paper. Notably, We notice that several reviewers have raised overlapping questions. Therefore, we have consolidated these into two key questions that address multiple reviewers' concerns. Below, we provide detailed answers to these two questions first.

Q1: The number of prompts engaged in this paper are limited...

A1: Thank you for your valuable feedback. We have carefully considered your comments and provide detailed responses below:

**1. PhyGenBench covers a broad range of prompts.**  
We design PhyGenBench starting from the most fundamental physical laws. It comprises **160 carefully crafted prompts across 27 distinct physical laws**, spanning **four fundamental domains of physics**. These prompts ensure comprehensive coverage of key physical commonsense principles.

**2. The benchmark focuses on essential and fundamental physical laws.**  
Even with **160 basic prompts**, PhyGenBench effectively exposes significant issues in current models. We focus on **the simplest and most common physical scenarios** (e.g., involving at most two objects). These basic setups already reveal serious limitations in existing models, demonstrating that our benchmark is sufficient for current testing. As models improve, we plan to **expand the benchmark** to include more complex scenarios.

**3. PhyGenBench undergoes rigorous selection and quality control.**  
We carefully refine the benchmark by removing overly complex prompts that current models cannot reasonably depict. We assess whether the T2V-generated videos are **semantically reasonable** to ensure effective evaluation. This process reduces the benchmark to its current size, focusing on scenarios that models can meaningfully generate.

**4. PhyGenBench demonstrates high quality.**  
We support this with both quantitative and qualitative analyses:  

- **Quantitative Analysis**: In Appendix D.2, we analyze the **semantic alignment (SA) scores** of videos generated based on PhyGenBench prompts. SA measures whether video generation models can depict the scenarios described in the prompts. Results show that all tested video generation models achieve high SA scores. For example, Kling achieves **0.85 (machine score)** and **0.89 (human score)**, demonstrating the reliability of PhyGenBench.  
- **Comparison with Other Benchmarks**: **In Table 1 below (Also in Appendix B.2)**, we compare PhyGenBench with benchmarks like VideoPhy in both quantitative and qualitative analyses. Results show that **PhyGenBench prompts achieve an average SA score of 0.80**, significantly outperforming **VideoPhy’s score of 0.63** in human evaluations. This highlights the superior quality of PhyGenBench.

Thank you again for your thoughtful suggestions. If you have any further questions or need additional clarification, please feel free to contact us.

| **Model**    | **Size** | **Videophy (↑)** | **PhyGenBench (↑)** |
| ------------ | -------- | ---------------- | ------------------- |
| CogVideoX    | 5B       | 0.48             | 0.78                |
| Vchitect 2.0 | 2B       | 0.63             | 0.84                |
| Kling        | -        | **0.77**         | **0.89**            |
| **Average**  | -        | 0.63             | 0.80                |

Table 1: Comparison of Semantic alignment scores between PhyGenBench and VideoPhy
</content>
</thread_object>

</thread>
<thread>
creation date:
1732442672631

thread_type:
official_comment

<thread_object>
creator: authors

<content>
title:
General Response-Part2

comment:
Q2: Effects of using Open Models in PhyGenEval (e.g. without GPT-4o)

A2: Thank you for your valuable feedback. We provide detailed responses below:

- **Effectiveness without GPT-4o as the VLM** 
  As shown in **Table 2 below**. We revise **Table 10 in the Appendix** to rename the original PhyGenEval (Open) as **PhyGenEval (Open-S)**, indicating that it uses small-scale open-source models. The results show that even when using only small-scale open-source models, the method achieves a Pearson correlation coefficient of  **0.66**, demonstrating the robustness of the PhyGenEval framework.


- **The questions generated by GPT-4o are part of our benchmark.**
 We need to emphasize that the question generation step using GPT-4o is a part of PhyGenBench. We have demonstrated the high quality of the generated questions in [Response to Reviewer wXRw (Q1)](https://openreview.net/forum?id=6rMHcLWxl4&noteId=dWEq2dmLbu) and both the questions and PhyGenBench have been open-sourced through an [anonymous link](https://github.com/PhyGenBench/PhyGenBench). Therefore, hallucination issues caused by LLMs are not a concern in PhyGenEval.

- **Exploring larger open-source models** 
  We experiment with larger open-source models by replacing the **LLaVA-Interleave-7B** in PhyGenEval (Open-S) with **InternVL-Pro (78B)**, denoting this configuration as **PhyGenEval (Open-L)**. Additionally, we ensemble PhyGenEval (Open-L) with PhyGenEval (Open-S)  , denoting this as **PhyGenEval (Open-Ensemble)**.  We supplement the description of ensemble operations in **Appendix C.2.** 

  Results indicate that compared to small-scale open-source models, the overall alignment coefficient improves from **0.66 to 0.72**, demonstrating that the framework maintains reproducibility even with fully open-source models. We believe that as open-source models continue to advance, their performance in PhyGenEval will improve further. We add this in **Appendix D.3**


- **Alignment with human evaluation improves with VLM advancements**  
  As the capabilities of VLMs used in PhyGenEval improve, we observe an increasing alignment between PhyGenEval and human evaluations. We believe that as open-source models continue to evolve, it will become feasible to use open-source VLMs for the entire workflow, further highlighting the robustness of our method's design.

Thank you again for your valuable suggestions. If you have further questions or require additional clarification, please feel free to contact us.



| Metric                     | Mechanics (ρ ↑) | Optics (ρ ↑) | Thermal (ρ ↑) | Material (ρ ↑) | Overall (ρ ↑) |
| -------------------------- | --------------- | ------------ | ------------- | -------------- | ------------- |
| PhyGenEval (Open-S)        | 0.57            | 0.62         | 0.58          | 0.69           | 0.66          |
| PhyGenEval (Open-L)        | 0.59            | 0.63         | 0.61          | 0.71           | 0.69          |
| PhyGenEval (Open-Ensemble) | 0.62            | 0.65         | 0.64          | 0.73           | 0.72          |
| PhyGenEval (GPT4o)         | 0.63            | 0.57         | 0.68          | 0.77           | 0.71          |
| PhyGenEval (Ensemble)      | **0.75**        | **0.77**     | **0.75**      | **0.84**       | **0.81**      |

Table 2: Comparison of PCA correlation results using different models such as GPT-4o or open-sourced models in PhyGenEval. PhyGenEval (Ensemble) is the result of ensemble of PhyGenEval (Open-S) and PhyGenEval (GPT4o)

[1] ChronoMagic-Bench: A Benchmark for Metamorphic Evaluation of Text-to-Time-lapse Video Generation
</content>
</thread_object>

</thread>
<thread>
creation date:
1732445403192

thread_type:
official_comment

<thread_object>
creator: authors

<content>
title:
Paper Update

comment:
We sincerely thank each reviewer for their valuable suggestions and questions regarding our paper. We have carefully considered these comments and added necessary experiments and clarifications. Under the reviewers' insightful guidance, we believe the quality of our paper has significantly improved. Below, we summarize the main changes, with new modifications in the updated paper highlighted in **red**:

- Revised the caption for **Figure 3** to clarify the PhyGenEval process.
- Updated the caption for **Table 2** to avoid confusion.
- Added more related work in **Appendix A**.
- Supplemented additional computational details in **Appendix C.2**.
- Included an experimental analysis of large-scale open models in **Appendix D.3**.
- Added error case analysis and visualizations for PhyGenEval in **Appendix E**.
- Provided resource consumption details for the method in **Appendix F**.

Thank you again for your constructive feedback. If you have further questions or require additional clarification, please feel free to contact us.
</content>
</thread_object>

</thread>


Metareview:
metareview:
The submission introduces a new benchmark for assessing models' understanding of physical commonsense.  Reviewers were lukewarm about the submission, and all shared concerns about the heavy use of generative AI during benchmark development. Other concerns include insufficient evaluation and dataset scale.  The AC agreed on these issues and encouraged the authors to revise the submission for the next venue.