$PhyWorldBench$: A Comprehensive Evaluation of Physical Realism in Text-to-Video Models

## Abstract

Abstract Video generation models have achieved remarkable progress in creating high-quality, photorealistic content. However, their ability to accurately simulate physical phenomena remains a critical and unresolved challenge. This paper presents PhyWorldBench , a comprehensive benchmark designed to evaluate video generation models based on their adherence to the laws of physics. The benchmark covers multiple levels of physical phenomena, ranging from fundamental principles like object motion and energy conservation to more complex scenarios involving rigid body interactions and human or animal motion. Additionally, we introduce a novel “Anti-Physics” category, where prompts intentionally violate real-world physics, enabling the assessment of whether models can follow such instructions while maintaining logical consistency. Besides large-scale human evaluation, we also design a simple yet effective method that could utilize current MLLM to evaluate the physics realism in a zero-shot fashion.
We evaluate 12 state-of-the-art text-to-video generation models, including five open-source and five proprietary models, with a detailed comparison and analysis. we identify pivotal challenges models face in adhering to real-world physics. Through systematic testing of their outputs across 1,050 curated prompts—spanning fundamental, composite, and anti-physics scenarios—we identify pivotal challenges these models face in adhering to real-world physics. We then rigorously examine their performance on diverse physical phenomena with varying prompt types, deriving targeted recommendations for crafting prompts that enhance fidelity to physical principles.

## 1 Introduction

Figure: Figure 1: Success rates of video generation models on PhyWorldBench. Among open-source models, Wanx demonstrated the highest performance, while Pika achieved the best results among proprietary models with a success rate of 0.262. Despite these advancements, substantial progress remains necessary to refine the capability of these models to accurately simulate the intricate dynamics of the real world.
Refer to caption: x2.png

The field of video generation has made remarkable progress, with models producing visually compelling and often photorealistic outputs. These advances have enabled transformative applications across industries such as entertainment, education, and scientific visualization. However, despite their visual fidelity, do video generation models truly understand the laws of physics in the real world?
To answer this question, we introduce PhyWorldBench, a rigorous benchmark designed to evaluate how well video generation models can simulate real-world physics. As illustrated in PhyWorldBench: A Comprehensive Evaluation of Physical Realism in Text-to-Video Models, PhyWorldBench systematically tests models across multiple levels of physical phenomena, from fundamental concepts like object motion to complex dynamics, including rigid body interactions and human/animal motion. Additionally, we propose a novel Anti-Physics category, where prompts deliberately violate real-world physics. On one hand, this design verifies whether models genuinely understand physical laws—rather than merely reproducing patterns from real-world training data. On the other hand, anti-physics content itself holds practical value in creative applications, where imaginative or otherwise impossible scenarios are beneficial.

We meticulously designed and annotated 1,050 prompts and the standard set for each prompt individually to cover a broad range of physical scenarios. This substantial annotation work ensures that our benchmark is both comprehensive and precise, allowing for a more thorough assessment of video generation models’ capabilities. Furthermore, we present a context-aware-prompt metric using MLLM (OpenAI Team, 2024; Gemini Team, 2024), which directly assesses if the video satisfies the physics standards or not. Such evaluation not only provided an unbiased metric but also significantly reduced the evaluation cost.

To examine the current status of video generation models and provide a detailed analysis, we selected five proprietary models—Sora-Turbo (OpenAI, 2024), Gen-3 (Runway Team, 2024), Kling 1.6 (KlingAI, 2024), Pika 2.0 (Pika Labs Team, 2024), and Luma (Luma AI Team, 2024)—along with seven open-source models: Hunyuan 720p (Kong et al., 2024), Open-Sora 2.0 (Peng et al., 2025), Open-Sora-Plan 1.3 (Lin et al., 2024), CogVideo 14b (Hong et al., 2022), Step-video-T2V (Ma et al., 2025), Wanx-2.1 (WanTeam et al., 2025), and LTX-Video (HaCohen et al., 2024). We generated 12,600 videos to evaluate SOTA models and analyze their ability to simulate real-world physics. We identify challenges, difficult scenarios, and key physics categories while proposing a structured approach to improve physical realism through prompt design.

The insights obtained from this benchmark will contribute to the development of more robust and physically accurate video generation models, addressing both fundamental scientific questions and practical challenges in the field. [Figure 1](https://arxiv.org/html/2507.13428v1#S1.F1) presents the overall benchmark results on PhyWorldBench. Despite recent advancements, models continue to struggle with temporal consistency, realistic motion, and physical plausibility, emphasizing the need for further research to enhance their physics correctness for real-world simulation.

Our work provides a benchmark for video generation models with a focus on physical realism. The key contributions are:

- •
We propose PhyWorldBench, a large-scale, multi-dimensional physics benchmark for evaluating the physics ability of the video generation model.
- •
We conduct an extensive evaluation of twelve state-of-the-art video generation models (five proprietary and seven open-source) with 12,600 generated videos and identified key challenges in simulating real-world physics.
- •
We study the effect of prompt variation on the performance of the video generation model and provided prompt guidelines for generating physics-following videos.

## 2 Related Work

Benchmarks for Text-to-Video Generation. Proprietary video generators (OpenAI, 2024; KlingAI, 2024; Pika Labs Team, 2024; Runway Team, 2024; HailuoAI, 2024; Luma AI Team, 2024) achieve striking quality and temporal coherence but remain opaque. Open-source counterparts (Peng et al., 2025; Lin et al., 2024; Hong et al., 2022; Wang et al., 2023; HaCohen et al., 2024; Kong et al., 2024; Agarwal et al., 2025) enable reproducible research and flexible benchmarking. The development of Text-to-Video (T2V) Generation models has been accelerated by benchmarks that evaluate performance across diverse aspects like quality, realism, and compositionality. VBench (Huang et al., 2024) and EvalCrafter (Liu et al., 2024) provide comprehensive frameworks for assessing video generation models, focusing on metrics such as diversity, temporal coherence, and scalability. Besides, T2V-CompBench (Sun et al., 2024) emphasizes compositionality, ensuring generated videos align with intricate textual prompts, a key challenge in real-world applications.
Recent works emphasize the importance of physical plausibility in video generation. VideoPhy (Bansal et al., 2024) and PhyGenBench (Meng et al., 2024) assess models’ adherence to physical commonsense, while DEVIL (Liao et al., 2024) evaluates dynamic realism, including motion and temporal consistency. Physics-IQ (Motameda et al., 2025) further tests understanding of fundamental principles like fluid dynamics and thermodynamics, highlighting the demand for physics-aware video generation.

We introduce a novel and substantially expanded dataset for evaluating video generation models across a broader range of physical laws. Unlike prior benchmarks, it includes diverse, carefully curated scenarios with both real and deliberately unphysical dynamics. Models perform significantly worse on our benchmark under identical settings, revealing its greater difficulty and realism.

Evaluation Metrics for Text-to-Video Generation.
Traditional text-to-video evaluation metrics like Fréchet Video Distance (FVD) (Unterthiner et al., 2018) and Inception Score (IS) (Salimans et al., 2016) focus on visual quality but fail to assess motion plausibility and adherence to physical laws, especially without reference videos. Recent methods address this gap: VideoScore (He et al., 2024) integrates human feedback, T2V-CompBench (Sun et al., 2024) uses VLMs for spatial and temporal evaluation, and PhyGenBench (Meng et al., 2024) explicitly tests physical commonsense. However, these methods can be computationally intensive or subjective.
We introduce a Yes/No answering metric for efficient, scalable, and objective evaluation of physical commonsense in video generation. Additionally, we propose a simple yet effective strategy leveraging MLLMs like GPT-o1 to assess video quality.

## 3 PhyWorldBench

To evaluate the ability of text-to-video generation models to simulate physical reality, we introduce PhyWorldBench, a comprehensive benchmark spanning a wide range of physical principles and scenarios. Its design is rooted in fundamental physics concepts such as motion, energy conservation, and interaction dynamics, drawing from established literature, including Fundamentals of Physics (Halliday et al., 2013) and Classical Mechanics (Goldstein et al., 2002). Developed with input from experts in physics and video generation.

Figure: Figure 2: Creation Process of PhyWorldBench. The dataset is built through a three-stage pipeline for clarity, consistency, and completeness. First, physics categories and prompts are defined using literature and expert input. Next, GPT-4o, Gemini-1.5-Pro together with human refine prompts for diversity and accuracy. Finally, a curation phase standardizes all prompts, with human-in-the-loop reviews ensuring clarity and eliminating ambiguities.
Refer to caption: x3.png

Figure: Figure 3: Example of Three Prompt Types for a Scenario. This figure illustrates a scenario from the subcategory Linear Motion under the main category Object Motion and Kinematics. The scenario is presented through three levels of prompts: (1) Event Prompt, providing a concise and straightforward event description; (2) Physics-Enhanced Prompt, which builds on the general prompt by incorporating physics-related phenomena while avoiding explicit physical laws; and (3) Detailed Narrative Prompt, enriching the Evnet Prompt with vivid details and contextual elements.
Refer to caption: x4.png

### 3.1 Dataset Creation Pipeline

As shown in [Figure 2](https://arxiv.org/html/2507.13428v1#S3.F2), the creation pipeline consists of three steps, with human-in-the-loop reviews integrated at each step to ensure the benchmark’s accuracy and diversity.

Step 1: Initial Physics Categories Definition. A collaborative process between physics experts and the authors to define the key (main) physics categories to be included in the benchmark.
Our benchmark categorizes three levels:

- •
Fundamental Physics: Covers basic laws such as object motion, energy transfer, and optics.
- •
Composite Physics: Involves real-world phenomena that emerge from multiple interacting principles, such as human motion.
- •
Anti-Physics: Scenarios intentionally designed to violate real-world physics.

The inclusion of anti-physics cases is crucial, as discrepancies in model performance between physically accurate and unphysical scenarios reveal whether a model truly understands physics or merely replicates patterns from its training data, which predominantly follows real-world laws.

We identify broad categories—such as kinematics, rigid body dynamics, fluid behavior, optics, and thermal dynamics—as depicted in the main physics categories in PhyWorldBench: A Comprehensive Evaluation of Physical Realism in Text-to-Video Models. Each category is systematically divided into five subcategories to ensure comprehensive coverage of physics principles from multiple perspectives.

Step 2: Prompt Creation Each subcategory is further expanded into 7 distinct scenarios, with each scenario incorporating three prompt variations: Event Prompt, Physics-enhanced Prompt, and Detailed Narrative Prompt, as illustrated in [Figure 3](https://arxiv.org/html/2507.13428v1#S3.F3). Specifically, we define these three types of prompts as follows:

- •
Event Prompt: A concise and straightforward description of an event. These prompts are designed to assess the model’s ability to generate based on minimal input, focusing on high-level scene comprehension.
- •
Physics-Enhanced Prompt: This type incorporates physics-related phenomena for Event Prompt to enrich the description naturally.
- •
Detailed Prompt: Inspired by how current T2V models generate videos, these prompts are enriched with vivid details and contextual elements, aiming to create more immersive and visually rich outputs. This level of prompt allows us to assess whether providing richer context leads to improved physical accuracy in video generation.

We first use state-of-the-art large language models (LLMs), specifically GPT-4o (OpenAI Team, 2024) and Gemini-1.5-Pro (Gemini Team, 2024) to generate 20 Event Prompts under the predefined category and subcategory. Then we ask humans to select 7 out of them based on diversity and physics representation.
Then human experts meticulously review and validate the generated prompts, providing feedback to iteratively enhance their quality and relevance to the corresponding physics subcategory, ensuring a balanced and representative assessment across various aspects of physical understanding. Furthermore, an ethical review is conducted to ensure that all prompts are fair, unbiased, and free from potentially harmful content. Based on the Event Prompt, human annotators create a Physics-Enhanced Prompt by enriching it with physical consequences. Detailed Narrative Prompt is generated by LLM based on Event Prompt using query in [Table 11](https://arxiv.org/html/2507.13428v1#A9.T11). This rigorous verification process results in a refined set of 1,050 high-quality prompts, systematically organized into main and subcategories.

Figure: Figure 4: Illustration of Our Evaluation Metric and Human Annotations. We demonstrate our evaluation process for assessing the quality of generated videos based on two evaluation criteria: Basic Standards and Key Standards. For Basic Standards, we verify whether the generated video contains the correct number of objects and accurately represents the intended action or event. For Key Standards, we define specific physical phenomena as ground truth and measure if all of these phenomena the generated video satisfies. Both lead to either a score of “0” or “1” for a generated video. Red circles and yellow lines in the figure highlight instances where the generated videos fail to meet the Key Standards.
Refer to caption: x5.png

**Table 1: Statistical Comparison with Existing Physics-Focused T2V Benchmarks. Compared to prior benchmarks, PhyWorldBench offers a significantly larger and more diverse testbed for evaluating the physical commonsense capabilities of T2V models. It provides a broader coverage of physics categories and a greater number of prompts, ensuring a comprehensive assessment of T2V performance.**
| Statistic | VideoPhy | PhyGenBench | Physics-IQ | PhyWorldBench |
| --- | --- | --- | --- | --- |
| Physics Categories | 5 | 27 | 5 | 50 |
| Prompts | 688 | 160 | 396 | 1050 |

Step 3: Standard Creation.
We adopt a Yes/No evaluation metric to assess whether a model can generate videos that accurately align with a given prompt. This metric is chosen for its simplicity and effectiveness and for allowing for an objective assessment while minimizing subjectivity in evaluation. As shown in [Figure 4](https://arxiv.org/html/2507.13428v1#S3.F4), we define two types of evaluation standards: Basic Standards and Key Standards. The Basic Standards specify the essential object(s) that should appear in the video and ensure that the main action or event is depicted. Meanwhile, the Key Standards describe the key physical phenomena that would occur if the event took place in the real world. The annotators are instructed to focus on the natural physical consequences of the action and to be explicit and concise in their annotations. By following these guidelines, we ensure that the evaluation standards remain clear and objective. Following  (Bansal et al., 2024; Meng et al., 2024), we use Semantic Adherence (SA) and Physical Commonsense (PC) to evaluate video performance. SA checks if both “objects” and “Action/Event” align with video, while PC assesses adherence to real-world physics and it checks when all “Key Standards” are satisfied. Annotated as $\text{SA},\text{PC}\in{0,1}$ (1 indicating proper grounding), we report the fraction of videos satisfying a success when $\text{SA}=1$, $\text{PC}=1$, and both jointly.

Figure: Figure 5: Context-aware prompt improves the assessment quality for GPT-4o. In the two examples, prompt (a) explicitly tells the MLLM that the video might contain potential issues, while prompt b does not. We found that the quality assessment is usually more accurate when the context is used. The text is irrelevant to quality is ignored for readability. This phenomenon applied to all our tested models including GPT-4o, GPT-o1, Gemini-2.0-flash, and Qwen-VL-2.0.
Refer to caption: x6.png

### 3.2 Statistics of the dataset

We compare our benchmark with existing physics-focused T2V benchmarks (Bansal et al., 2024; Meng et al., 2024), with a detailed statistical comparison presented in [Table 1](https://arxiv.org/html/2507.13428v1#S3.T1). PhyWorldBench offers the most extensive coverage in terms of the number of physics categories, curated prompts, human annotations, and evaluated T2V videos.

### 3.3 MLLM as Zero Shot Physics Evaluator for Generated Videos

Human evaluation, while flexible, is costly at scale. Existing metrics often lack generalizability and struggle with the diversity in PhyWorldBench. (See [Appendix C](https://arxiv.org/html/2507.13428v1#A3).) MLLMs like GPT-o1 (OpenAI Team, 2024) and Gemini (Gemini Team, 2024) show promise for video evaluation but fall short in reliably assessing physical realism (Bansal et al., 2024; He et al., 2024; Meng et al., 2024). We find that these models tend to rationalize unrealistic content, but their accuracy improves when informed that the video is AI-generated, as illustrated in [Figure 5](https://arxiv.org/html/2507.13428v1#S3.F5). We hypothesize that this improvement stems from MLLMs being primarily trained on real-world videos, leading them to inherently justify observed phenomena unless prompted otherwise. Inspired by this, we propose a simple yet effective method, Context-Aware Prompt (CAP), which explicitly informs the MLLM that the video is generated rather than real. This approach enables the MLLM to provide a more accurate assessment of the video’s physical realism. Critically, CAP does not assume videos are incorrect by default but instead uses a structured chain‑of‑thought prompt—first asking the model to describe the video and then to reason through any potential physics issues—and only labels a video as incorrect if it clearly violates the category‑specific standards. We validate CAP neutrality by measuring false positive rates in physics correct videos and observe only a 0.001 absolute change in semantic adhesion (0.188 to 0.187) and a 0.014 change in physical commonsense (0.172 to 0.186), showing that it does not significantly increase misclassification of correct videos.

Besides, inspired by Chain-of-Thought (Wei et al., 2022), we design a two-step prompting strategy, where we first ask the MLLM to give a thorough description of the video including Object, Event, and Observations, then continue to prompt the MLLM for the final “Yes/No” answer for video quality. Please refer to [Appendix I](https://arxiv.org/html/2507.13428v1#A9) for prompts used in the two steps. We found this leads to consistent improvement in various MLLMs. In [Table 2](https://arxiv.org/html/2507.13428v1#S3.T2), we present the ROC AUC calculations comparing CAP with human results on PhyWorldBench with videos generated by models in [Figure 1](https://arxiv.org/html/2507.13428v1#S1.F1), where a random guess would achieve a score of 50. We evenly sample 8 frames. In this evaluation, CAP is powered by GPT-o1. CAP significantly outperforms other methods, particularly in Physics Commonsense, where it achieves an absolute performance boost of 13.5 compared to the standard GPT-o1. We find CAP also achieves great performance when equipping other proprietary or open-sourced MLLM. [Table 2](https://arxiv.org/html/2507.13428v1#S3.T2) shows the ablation study of various components in CAP. “without CoT” means we directly ask the MLLM to give the answer of the question, and “without Context” means we do not inform MLLM that the video could be of low quality.
Please refer [Appendix C](https://arxiv.org/html/2507.13428v1#A3) for performance of our method on other MLLM and analysis.

**Table 2: ROC-AUC for automatic evaluation methods and ablation on CAP. Rows now list the two metrics—SA (semantic adherence) and PC (physical commonsense)—while columns compare all baselines and ablations.**
| Metric | Qwen-VL-2.0 | Gemini-2.0-Flash | GPT-4o | GPT-o1 | CAP (w/o CoT) | CAP (w/o Context) | CAP |
| --- | --- | --- | --- | --- | --- | --- | --- |
| SA | 72.4 | 74.6 | 72.1 | 75.4 | 76.3 | 77.3 | 80.3 |
| PC | 59.8 | 60.9 | 60.1 | 61.6 | 73.6 | 65.6 | 75.1 |

## 4 Video Generation Results and Analysis

We evaluate both closed-source models including Sora-Turbo (OpenAI, 2024), Gen-3 (Runway Team, 2024), Kling 1.6 (KlingAI, 2024), Pika 2.0 (Pika Labs Team, 2024), Luma (Luma AI Team, 2024), and open-source models including Hunyuan 720p (Kong et al., 2024), Open-Sora 2.0 (Peng et al., 2025), Open-Sora-Plan 1.3 (Lin et al., 2024), CogVideo 14b (Hong et al., 2022), LTX-Video (HaCohen et al., 2024), Step-video-T2V (Ma et al., 2025), Wanx-2.1 (WanTeam et al., 2025). Please check [Appendix D](https://arxiv.org/html/2507.13428v1#A4) for the implementation details. We generate 1050 videos for each model. We conduct a human evaluation of generated videos using Amazon Mechanical Turk. We first introduce model performance and model-wise analysis and then break down the challenge for the video generation model to achieve real-world physics.

### 4.1 Model Performance

**Table 3: We present the human evaluation results of 10 video generation models across 3 physics types. SA denotes semantic adherence, while PC represents physical commonsense. Both indicates satisfaction of both criteria.**
| Model | Fundamental Physics | Composite Physics | Anti-Physics | Overall Performance |  |  |  |  |  |  |  |  |
| --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- |
| SA | PC | Both | SA | PC | Both | SA | PC | Both | SA | PC | Both |  |
| Proprietary Models |  |  |  |  |  |  |  |  |  |  |  |  |
| Sora-Turbo (OpenAI, 2024) | 0.438 | 0.315 | 0.246 | 0.361 | 0.215 | 0.188 | 0.136 | 0.078 | 0.039 | 0.384 | 0.261 | 0.208 |
| Gen-3 (Runway Team, 2024) | 0.320 | 0.228 | 0.161 | 0.258 | 0.142 | 0.103 | 0.108 | 0.029 | 0.020 | 0.280 | 0.183 | 0.130 |
| Kling-1.6 (KlingAI, 2024) | 0.417 | 0.299 | 0.235 | 0.312 | 0.175 | 0.139 | 0.125 | 0.073 | 0.042 | 0.357 | 0.241 | 0.188 |
| Pika 2.0 (Pika Labs Team, 2024) | 0.587 | 0.375 | 0.312 | 0.479 | 0.274 | 0.239 | 0.228 | 0.043 | 0.011 | 0.521 | 0.314 | 0.262 |
| Luma (Luma AI Team, 2024) | 0.464 | 0.259 | 0.220 | 0.328 | 0.199 | 0.160 | 0.056 | 0.000 | 0.000 | 0.385 | 0.218 | 0.184 |
| Open-sourced Models |  |  |  |  |  |  |  |  |  |  |  |  |
| Hunyuan 720p (Kong et al., 2024) | 0.385 | 0.278 | 0.205 | 0.342 | 0.264 | 0.199 | 0.082 | 0.021 | 0.010 | 0.344 | 0.250 | 0.185 |
| Open-Sora 2.0 (Peng et al., 2025) | 0.434 | 0.268 | 0.205 | 0.279 | 0.202 | 0.154 | 0.056 | 0.021 | 0.010 | 0.348 | 0.223 | 0.170 |
| Open-Sora-Plan 1.3 (Lin et al., 2024) | 0.175 | 0.136 | 0.096 | 0.159 | 0.057 | 0.044 | 0.069 | 0.059 | 0.039 | 0.158 | 0.104 | 0.075 |
| CogVideo-5B (Hong et al., 2022) | 0.419 | 0.266 | 0.192 | 0.308 | 0.207 | 0.154 | 0.062 | 0.021 | 0.000 | 0.351 | 0.225 | 0.163 |
| Step-video-T2V (Ma et al., 2025) | 0.353 | 0.245 | 0.189 | 0.333 | 0.251 | 0.191 | 0.085 | 0.021 | 0.020 | 0.320 | 0.224 | 0.173 |
| Wanx-2.1 (WanTeam et al., 2025) | 0.389 | 0.271 | 0.220 | 0.323 | 0.235 | 0.187 | 0.082 | 0.017 | 0.011 | 0.339 | 0.235 | 0.189 |
| LTX-Video (HaCohen et al., 2024) | 0.221 | 0.102 | 0.080 | 0.177 | 0.071 | 0.042 | 0.076 | 0.011 | 0.000 | 0.194 | 0.085 | 0.062 |

[Table 3](https://arxiv.org/html/2507.13428v1#S4.T3) presents a comprehensive evaluation of twelve video generation models. We report the percentage of videos that satisfy both semantic adherence and physical commonsense criteria (Both). The Overall Performance column provides an aggregate score for each model, averaged across 10 physics categories.
Pika 2.0 achieves the best overall performance across all models. In open-sourced models, Hunyuan 720p achieves the best performance on PC, Wanx-2.1 achieves the best performance on Both, and CogVideo achieves the best SA performance. It is worth noting that they even achieve a better performance compared with some of the proprietary. Please check  [Appendix L](https://arxiv.org/html/2507.13428v1#A12) for Leaderboard.

### 4.2 Model Specific Analysis

Our evaluation reveals a trade-off between realism, stylization, and consistency in text-to-video models. Wanx, Step-Video-T2V, Gen-3, Hunyuan, Kling, and Sora produce the most realistic videos, while Pika and Luma add stylized lighting and colors. Open-Sora and LTX-Video struggle with realism, often generating low-fidelity or distorted outputs. Object stability varies—CogVideo, Open-Sora, and LTX-Video frequently produce warped shapes, while Gen-3, Hunyuan, and Kling maintain more stable forms. Motion realism also differs: Kling and Gen-3 generate smooth motion, Open-Sora-Plan and Luma introduce slow-motion effects, and CogVideo and Open-Sora exhibit jittery, erratic movement. For a detailed analysis, see [Appendix J](https://arxiv.org/html/2507.13428v1#A10).

### 4.3 Effect of Physics and Prompt Enhancement

Understanding real-world physics is essential for text-to-video models. Event-based prompts describe actions, while physics-enhanced prompts include natural consequences. A model’s ability to generate realistic videos based on these prompts reflects its real-world understanding. As proprietary models conceal their generation processes, we cannot verify prompt refinement. Instead, we analyze prompt types using open-source models, where prompt integrity can be manually ensured. In [Table 4](https://arxiv.org/html/2507.13428v1#S4.T4), we compare model performance across different prompt types to gauge the effects of physics-aware prompting and refinement. While refinement improves clarity, object detail, and aesthetics, it does not notably enhance physics accuracy. The persistent inconsistencies suggest that these refinements primarily address surface-level quality, leaving deeper physics limitations unresolved. On the other hand, physics-enhanced prompts often yield better results, reflecting the models’ strong prompt-following ability—even though they still struggle with true physical comprehension. Consequently, we propose a straightforward receipt: explicitly integrate physical phenomena into the prompt to nudge the model toward more realistic outcomes.

### 4.4 Anti-Physics Analysis

As shown in [Table 3](https://arxiv.org/html/2507.13428v1#S4.T3), all models show a performance decline from Fundamental to Composite to Anti-Physics categories, reflecting the challenge of generating complex physical phenomena. While they capture basic physics, simulating intricate interactions and “unphysical” scenes remains difficult, highlighting a gap in physics adherence. Our analysis found that failures often stem from models interpreting prompts in a more “reasonable” way rather than attempting and failing. For example, when prompted with “A glass sits on the table, and the wine in the glass has reversed gravity,” the model generates a still image, adhering to real-world physics instead of the prompt. This suggests that models prioritize mimicking training data over true physics understanding.

### 4.5 Failure to Handle Erratic Visual Changes

Certain physical events inherently result in erratic visual changes, such as collisions, breaking, or sudden state transitions. However, current video generation models struggle to accurately depict these scenarios. In the third example of [Figure 9](https://arxiv.org/html/2507.13428v1#A10.F9), a glass cup falls from a table, an event that should cause it to shatter upon impact, leading to a sudden and complex visual transformation. Instead, the model rationalizes the action, generating a video where the glass remains intact and in the same vertical position after falling, completely ignoring the expected breakage and fragmentation. This highlights a fundamental limitation: when an event is supposed to cause an erratic visual change, current AI models fail to reflect the correct physical outcome. Rather than simulating realistic disruptions in structure, motion, or object state, the models tend to default to continuous, smoothed animations, avoiding abrupt transformations that are critical for representing real-world physics.

### 4.6 Breakdown in Physics at Higher Complexity

While many models handle simple, isolated physics well, their performance deteriorates under complex interactions involving multiple forces or constraints. For instance, in billiard collisions, models often fail to preserve momentum and friction, leading to unnatural stopping, incorrect deflections, or interpenetrating objects. Similarly, fluid simulations frequently lack realistic turbulence and surface tension when water interacts with solid surfaces. As demonstrated in [Appendix M](https://arxiv.org/html/2507.13428v1#A13), prompts involving multiple objects introduce significantly higher requirements for semantic adherence. For instance, when tested with the prompt “A feather and a stone are dropped in the air, where both fall, and the stone drops much faster”, all models fail to represent the phenomenon correctly. Beyond failing to adhere to semantics, these errors highlight a broader issue: models struggle to depict the correct physical relationships between interacting objects, often treating them as independent entities rather than systems governed by unified physical laws. This suggests that while models have learned pattern-based approximations of physics, they lack a robust understanding of how multiple physical forces interact dynamically in complex environments.

**Table 4: Physics-following percentage on different types of prompt. Explicitly adding physics usually increases the physics-following ability of video generation model, while a prompt refinement process does not necessary leads to improvement.**
| Model | Event Prompt | Physics-enhanced Prompt | Detailed Prompt |
| --- | --- | --- | --- |
| CogVideo-5B (Hong et al., 2022) | 0.123 | 0.177 | 0.168 |
| Hunyuan 720p (Kong et al., 2024) | 0.159 | 0.198 | 0.155 |
| LTX-Video (HaCohen et al., 2024) | 0.056 | 0.065 | 0.066 |
| Open-Sora-Plan 1.3 (Lin et al., 2024) | 0.062 | 0.067 | 0.063 |
| Open-Sora 2.0 (Peng et al., 2025) | 0.167 | 0.177 | 0.173 |
| Wanx-2.1 (WanTeam et al., 2025) | 0.175 | 0.202 | 0.190 |
| Step-Video-T2V (Ma et al., 2025) | 0.158 | 0.182 | 0.179 |

### 4.7 Cinematic Effects vs. Physical Plausibility

A major challenge in video generation is the tendency for models to prioritize cinematic aesthetics over strict physical realism. Well-performed models like Sora, Gen-3, and Pika often introduce exaggerated motion dynamics, such as objects moving with heightened fluidity or unnatural acceleration, making scenes appear choreographed rather than physically grounded. For example, an object that should fall naturally under gravity might instead descend too smoothly or float unrealistically, breaking expected motion physics. This issue is particularly evident in action sequences, where momentum conservation and force dynamics are distorted for dramatic effect, leading to visually striking but physically implausible results. The first example in [Figure 10](https://arxiv.org/html/2507.13428v1#A10.F10) presented an apple floating in the air, then it suddenly dropped as the camera moved; such stylized rendering, while beneficial for artistic storytelling, poses significant challenges for applications requiring strict physical fidelity.

## 5 Contributions and Future Work

We propose PhyWorldBench, a large-scale, thorough, and multi-dimensional benchmark for evaluating text-to-video generation models across diverse physics categories. It provides insights into which physical phenomena are hardest to simulate and what capabilities current models lack. Additionally, we propose an automated evaluator to measure physical accuracy and a straightforward receipt for designing prompt. Future work will involve continuously evaluating emerging video generation models, updating our benchmark to reflect advancements in the field, and refining our leaderboard and analysis to track progress in physical accuracy and model capabilities.

## Impact Statement

PhyWorldBench establishes a new standard for physics-focused text-to-video generation, significantly elevating the field’s expectations of realism and accuracy. By rigorously assessing models’ capabilities across a comprehensive spectrum of physical phenomena—including motion, energy transfer, and complex interactions—this benchmark not only highlights critical gaps in current systems but also guides researchers toward developing more robust and physics-aware solutions. Moreover, PhyWorldBench acknowledges the ethical implications of physics-driven video synthesis, emphasizing the need for transparency, fairness, and responsible deployment. By mitigating risks such as misinformation, biased simulations, and unintended real-world consequences, it ensures that advancements in this field are aligned with scientific integrity and societal well-being. Ultimately, PhyWorldBench paves the way for advanced applications in education, scientific visualization, and industry, demanding both photorealism and fidelity to real-world physics while upholding ethical standards.

## References

- Agarwal et al. (2025)
Agarwal, N., Ali, A., Bala, M., Balaji, Y., Barker, E., Cai, T., Chattopadhyay, P., Chen, Y., Cui, Y., Ding, Y., et al.
Cosmos world foundation model platform for physical ai.
*arXiv preprint arXiv:2501.03575*, 2025.
- Bansal et al. (2024)
Bansal, H., Lin, Z., Xie, T., Zong, Z., Yarom, M., Bitton, Y., Jiang, C., Sun, Y., Chang, K.-W., and Grover, A.
Videophy: Evaluating physical commonsense for video generation.
*arXiv preprint arXiv:2406.03520*, 2024.
- Gemini Team (2024)
Gemini Team.
Gemini: A family of highly capable multimodal models, 2024.
- Goldstein et al. (2002)
Goldstein, H., Poole, C., Safko, J., and Addison, S. R.
Classical mechanics, 2002.
- Guo et al. (2025)
Guo, X., Huo, J., Shi, Z., Song, Z., Zhang, J., and Zhao, J.
T2vphysbench: A first-principles benchmark for physical consistency in text-to-video generation.
*arXiv preprint arXiv:2505.00337*, 2025.
- HaCohen et al. (2024)
HaCohen, Y., Chiprut, N., Brazowski, B., Shalem, D., Moshe, D., Richardson, E., Levin, E., Shiran, G., Zabari, N., Gordon, O., Panet, P., Weissbuch, S., Kulikov, V., Bitterman, Y., Melumian, Z., and Bibi, O.
Ltx-video: Realtime video latent diffusion.
*arXiv preprint arXiv:2501.00103*, 2024.
- HailuoAI (2024)
HailuoAI.
Hailuo.
[https://hailuoai.video/](https://hailuoai.video/), 2024.
- Halliday et al. (2013)
Halliday, D., Resnick, R., and Walker, J.
*Fundamentals of physics*.
John Wiley & Sons, 2013.
- He et al. (2024)
He, X., Jiang, D., Zhang, G., Ku, M., Soni, A., Siu, S., Chen, H., Chandra, A., Jiang, Z., Arulraj, A., et al.
Videoscore: Building automatic metrics to simulate fine-grained human feedback for video generation.
*arXiv preprint arXiv:2406.15252*, 2024.
- Hong et al. (2022)
Hong, W., Ding, M., Zheng, W., Liu, X., and Tang, J.
Cogvideo: Large-scale pretraining for text-to-video generation via transformers.
*arXiv preprint arXiv:2205.15868*, 2022.
- Huang et al. (2024)
Huang, Z., He, Y., Yu, J., Zhang, F., Si, C., Jiang, Y., Zhang, Y., Wu, T., Jin, Q., Chanpaisit, N., et al.
Vbench: Comprehensive benchmark suite for video generative models.
In *CVPR*, 2024.
- KlingAI (2024)
KlingAI.
KLING.
[https://www.klingai.com/](https://www.klingai.com/), 2024.
- Kong et al. (2024)
Kong, W., Tian, Q., Zhang, Z., Min, R., Dai, Z., Zhou, J., Xiong, J., Li, X., Wu, B., Zhang, J., Wu, K., Lin, Q., Wang, A., Wang, A., Li, C., Huang, D., Yang, F., Tan, H., Wang, H., Song, J., Bai, J., Wu, J., Xue, J., Wang, J., Yuan, J., Wang, K., Liu, M., Li, P., Li, S., Wang, W., Yu, W., Deng, X., Li, Y., Long, Y., Chen, Y., Cui, Y., Peng, Y., Yu, Z., He, Z., Xu, Z., Zhou, Z., Xu, Z., Tao, Y., Lu, Q., Liu, S., Zhou, D., Wang, H., Yang, Y., Wang, D., Liu, Y., Jiang, J., and Zhong, C.
Hunyuanvideo: A systematic framework for large video generative models, 2024.
URL [https://arxiv.org/abs/2412.03603](https://arxiv.org/abs/2412.03603).
- Liao et al. (2024)
Liao, J., Tan, X., Shao, W., Zhang, K., Cheng, Y., Li, D., and Luo, P.
Devil: A comprehensive benchmark for dynamics evaluation in video generation.
*arXiv preprint arXiv:2407.01094*, 2024.
- Lin et al. (2024)
Lin, B., Ge, Y., Cheng, X., Li, Z., Zhu, B., Wang, S., He, X., Ye, Y., Yuan, S., Chen, L., et al.
Open-sora plan: Open-source large video generation model.
*arXiv preprint arXiv:2412.00131*, 2024.
- Liu et al. (2024)
Liu, Y., Cun, X., Liu, X., Wang, X., Zhang, Y., Chen, H., Liu, Y., Zeng, T., Chan, R., and Shan, Y.
Evalcrafter: Benchmarking and evaluating large video generation models.
In *CVPR*, 2024.
- Luma AI Team (2024)
Luma AI Team.
Luma AI.
[https://lumalabs.ai](https://lumalabs.ai), 2024.
Generative AI platform specializing in 3D content and photorealistic modeling.
- Ma et al. (2025)
Ma, G., Huang, H., Yan, K., Chen, L., Duan, N., Yin, S., Wan, C., Ming, R., Song, X., Chen, X., Zhou, Y., Sun, D., Zhou, D., Zhou, J., Tan, K., An, K., Chen, M., Ji, W., Wu, Q., Sun, W., Han, X., Wei, Y., Ge, Z., Li, A., Wang, B., Huang, B., Wang, B., Li, B., Miao, C., Xu, C., Wu, C., Yu, C., Shi, D., Hu, D., Liu, E., Yu, G., Yang, G., Huang, G., Yan, G., Feng, H., Nie, H., Jia, H., Hu, H., Chen, H., Yan, H., Wang, H., Guo, H., Xiong, H., Xiong, H., Gong, J., Wu, J., Wu, J., Wu, J., Yang, J., Liu, J., Li, J., Zhang, J., Guo, J., Lin, J., Li, K., Liu, L., Xia, L., Zhao, L., Tan, L., Huang, L., Shi, L., Li, M., Li, M., Cheng, M., Wang, N., Chen, Q., He, Q., Liang, Q., Sun, Q., Sun, R., Wang, R., Pang, S., Yang, S., Liu, S., Liu, S., Gao, S., Cao, T., Wang, T., Ming, W., He, W., Zhao, X., Zhang, X., Zeng, X., Liu, X., Yang, X., Dai, Y., Yu, Y., Li, Y., Deng, Y., Wang, Y., Wang, Y., Lu, Y., Chen, Y., Luo, Y., Luo, Y., Yin, Y., Feng, Y., Yang, Y., Tang, Z., Zhang, Z., Yang, Z., Jiao, B., Chen, J., Li, J., Zhou,
S., Zhang, X., Zhang, X., Zhu, Y., Shum, H.-Y., and Jiang, D.
Step-video-t2v technical report: The practice, challenges, and future of video foundation model, 2025.
URL [https://arxiv.org/abs/2502.10248](https://arxiv.org/abs/2502.10248).
- Meng et al. (2024)
Meng, F., Liao, J., Tan, X., Shao, W., Lu, Q., Zhang, K., Cheng, Y., Li, D., Qiao, Y., and Luo, P.
Towards world simulator: Crafting physical commonsense-based benchmark for video generation.
*arXiv preprint arXiv:2410.05363*, 2024.
- Motameda et al. (2025)
Motameda, S., Culp, L., Swersky, K., Jain, P., and Geirhos, R.
Physics-iq: Evaluating physical understanding in generative video models.
*arXiv preprint arXiv:2501.09038*, 2025.
- OpenAI (2024)
OpenAI.
Sora: OpenAI’s Multimodal Agent.
[https://openai.com/index/sora/](https://openai.com/index/sora/), 2024.
Accessed: 2024-11-24.
- OpenAI Team (2024)
OpenAI Team.
Gpt-4o.
[https://openai.com/index/hello-gpt-4o/](https://openai.com/index/hello-gpt-4o/), 2024.
- Peng et al. (2025)
Peng, X., Zheng, Z., Shen, C., Young, T., Guo, X., Wang, B., Xu, H., Liu, H., Jiang, M., Li, W., Wang, Y., Ye, A., Ren, G., Ma, Q., Liang, W., Lian, X., Wu, X., Zhong, Y., Li, Z., Gong, C., Lei, G., Cheng, L., Zhang, L., Li, M., Zhang, R., Hu, S., Huang, S., Wang, X., Zhao, Y., Wang, Y., Wei, Z., and You, Y.
Open-sora 2.0: Training a commercial-level video generation model in $200k.
*arXiv preprint arXiv:2503.09642*, 2025.
- Pika Labs Team (2024)
Pika Labs Team.
Pika Labs.
[https://pikalabs.com](https://pikalabs.com), 2024.
Generative AI platform for creating video and visual content.
- Runway Team (2024)
Runway Team.
Runway.
[https://runwayml.com](https://runwayml.com), 2024.
Platform for AI-powered video editing and generative media creation.
- Salimans et al. (2016)
Salimans, T., Goodfellow, I., Zaremba, W., Cheung, V., Radford, A., and Chen, X.
Improved techniques for training gans.
*Advances in neural information processing systems*, 29, 2016.
- Sun et al. (2024)
Sun, K., Huang, K., Liu, X., Wu, Y., Xu, Z., Li, Z., and Liu, X.
T2v-compbench: A comprehensive benchmark for compositional text-to-video generation.
*arXiv preprint arXiv:2407.14505*, 2024.
- Unterthiner et al. (2018)
Unterthiner, T., Van Steenkiste, S., Kurach, K., Marinier, R., Michalski, M., and Gelly, S.
Towards accurate generative models of video: A new metric & challenges.
*arXiv preprint arXiv:1812.01717*, 2018.
- Wang et al. (2023)
Wang, Y., Chen, X., Ma, X., Zhou, S., Huang, Z., Wang, Y., Yang, C., He, Y., Yu, J., Yang, P., et al.
Lavie: High-quality video generation with cascaded latent diffusion models.
*arXiv preprint arXiv:2309.15103*, 2023.
- WanTeam et al. (2025)
WanTeam, Wang, A., Ai, B., Wen, B., Mao, C., Xie, C.-W., Chen, D., Yu, F., Zhao, H., Yang, J., Zeng, J., Wang, J., Zhang, J., Zhou, J., Wang, J., Chen, J., Zhu, K., Zhao, K., Yan, K., Huang, L., Feng, M., Zhang, N., Li, P., Wu, P., Chu, R., Feng, R., Zhang, S., Sun, S., Fang, T., Wang, T., Gui, T., Weng, T., Shen, T., Lin, W., Wang, W., Wang, W., Zhou, W., Wang, W., Shen, W., Yu, W., Shi, X., Huang, X., Xu, X., Kou, Y., Lv, Y., Li, Y., Liu, Y., Wang, Y., Zhang, Y., Huang, Y., Li, Y., Wu, Y., Liu, Y., Pan, Y., Zheng, Y., Hong, Y., Shi, Y., Feng, Y., Jiang, Z., Han, Z., Wu, Z.-F., and Liu, Z.
Wan: Open and advanced large-scale video generative models.
*arXiv preprint arXiv:2503.20314*, 2025.
- Wei et al. (2022)
Wei, J., Wang, X., Schuurmans, D., Bosma, M., brian ichter, Xia, F., Chi, E. H., Le, Q. V., and Zhou, D.
Chain of thought prompting elicits reasoning in large language models.
In Oh, A. H., Agarwal, A., Belgrave, D., and Cho, K. (eds.), *Advances in Neural Information Processing Systems*, 2022.
URL [https://openreview.net/forum?id=_VjQlMeSB_J](https://openreview.net/forum?id=_VjQlMeSB_J).
- Zhang et al. (2025)
Zhang, C., Cherniavskii, D., Zadaianchuk, A., Tragoudaras, A., Vozikis, A., Nijdam, T., Prinzhorn, D. W., Bodracska, M., Sebe, N., and Gavves, E.
Morpheus: Benchmarking physical reasoning of video generative models with real physical experiments.
*arXiv preprint arXiv:2504.02918*, 2025.

## Appendix A Benchmark Details

The benchmark revolves around 10 main physics categories, each divided into 5 subcategories that cover specific physics phenomena, such as object occlusion, center of mass, and circular motion. Each subcategory includes 7 distinct scenarios, with 3 variations of prompts: a general prompt, a physics-enhanced prompt, and a detailed narrative prompt. These variations are designed to provide differing levels of complexity and context for the text-to-video models. The dataset was curated through a three-stage process: first, defining key categories based on fundamental physics literature; second, refining prompts through iterative reviews with large language models and human experts to ensure clarity and alignment with physics principles; and third, conducting detailed curation to create a final set of 1,050 high-quality prompts for a diverse set of physics phenomena. The evaluation utilizes a yes/no framework to assess model adherence to physical laws, categorized into basic standards and key standards, enabling comprehensive performance evaluation. This benchmark establishes a foundation for evaluating models’ ability to generate videos that align with both physical principles and real-world scenarios.

Following  (Bansal et al., 2024) and  (Meng et al., 2024), we use Semantic Adherence (SA) and Physical Commonsense (PC) as metric to evaluate the video performance. SA measures if a caption aligns with video frames, while PC assesses if actions obey real-world physics. Annotated as $\text{SA},\text{PC}\in\{0,1\}$, where 1 indicates proper grounding. We report the fraction of videos satisfying $\text{SA}=1$, $\text{PC}=1$, and both jointly. We create SA and PC based on a thorough review of all prompts and extensive feedback from human annotators, we establish a set of evaluation criteria to systematically assess the generated videos, as illustrated in [Figure 4](https://arxiv.org/html/2507.13428v1#S3.F4). The evaluation is structured around two key standards, each encompassing specific rules:

- •
Semantic Adherence: These criteria evaluate fundamental aspects of video generation to ensure prompt fidelity. The assessment includes:
1.
Whether the video contains the correct number of objects specified in the prompt. (Yes/No)
2.
Whether the depicted event or action is accurately represented. (Yes/No)
- •
Physical Commonsense: This aspect evaluates the model’s capability to capture and represent underlying physical phenomena within the video. The assessment measures the number of distinct physical phenomena accurately showcased in the generated video.

By integrating these two standards, which consist of three core evaluation rules, we define a comprehensive evaluation metric that ensures a balanced and thorough assessment of T2V model performance across multiple dimensions.

## Appendix B Annotation Details

Prompt Annotation
After designing the physics categories, we further prepared prompts within each category. To achieve this, we gathered undergraduate and graduate students majoring in Computer Science for an annotation session. They were instructed to provide high-quality annotations by creating both an Event Prompt and a corresponding Physics-enhanced Prompt that aligned with the primary physics category and sub-category. The Detailed Prompt was generated using GPT-4o with the MLLM prompt (as described in [Table 11](https://arxiv.org/html/2507.13428v1#A9.T11)) and subsequently verified by human annotators. Finally, each prompt underwent a final verification by the authors.

Standard Annotation
Undergraduate and graduate students in Computer Science were also tasked with annotating the objects and events described in the prompts. These annotations were used for SA (Semantic Adherence) evaluation. Additionally, they identified key physics phenomena that should be visually observable in the generated videos, which were used for PC (Physics Commonsense) evaluation. All annotations were reviewed and verified by the authors.

Video Evaluation
For large-scale video evaluation, we utilized Amazon Mechanical Turk. The interface is presented in [Figure 6](https://arxiv.org/html/2507.13428v1#A2.F6) Annotators were asked to assess videos based on three criteria: the existence of objects, the presence of the described event, and the visibility of key physics phenomena, selecting either “Yes” or “No” for each. Each video was evaluated by three independent annotators, with the final decision determined by a majority vote. All annotators were compensated at a rate of $18 USD per hour.

Figure: Figure 6: Amazon Mechanical Turk interface.
Refer to caption: x7.png

## Appendix C Detailed Analysis on Evaluator

While some existing physics metrics have been proposed, none fit PhyWorldBench’s needs due to domain or structural mismatches (Guo et al., 2025; Zhang et al., 2025). For example, PhyGenEval (Meng et al., 2024) assumes a fixed event order, which many of PhyWorldBench’s scenarios do not have, and VideoCon‑Physics (Bansal et al., 2024) is trained specifically on its own VideoPhy benchmark, yielding sub‑optimal performance on PhyWorldBench—so we exclude both to avoid unfair comparisons.

In [Table 6](https://arxiv.org/html/2507.13428v1#A3.T6), we therefore evaluate our CAP across multiple MLLMs, including proprietary GPT‑4o, GPT‑o1, Gemini‑Flash‑2.0, and open‑source Qwen‑VL‑2.0. Crucially, CAP uses a two‑step chain‑of‑thought prompt—first asking the model to describe the video, then to reason about any physics issues—so it does not assume videos are incorrect by default and yields only minimal false‑positive increases (0.001 in Semantic Adherence, 0.014 in Physical Commonsense), confirming its neutrality. Moreover, because it operates solely on the final video and its physics‑based standards, CAP is agnostic to the underlying generative pipeline—whether text‑to‑video, image‑to‑video, video‑to‑video, or hybrid.

We do observe two common failure modes: 1) Aesthetic bias, where visually compelling videos receive high scores despite only moderate physical accuracy (visual appeal and physical plausibility should ideally be assessed independently); and 2) Overconfidence, with the MLLM giving firm judgments in ambiguous or edge‑case scenarios where human annotators often express uncertainty or disagreement.

To further validate our design, we also measure Precision and Recall for both Semantic Adherence (SA) and Physical Commonsense (PC), which is shown in Table [5](https://arxiv.org/html/2507.13428v1#A3.T5).

**Table 5: Precision and Recall for automatic evaluation methods and ablation on CAP, together with ROC‑AUC from the submission.**
| Method | SA | Precision of SA | Recall of SA | PC | Precision of PC | Recall of PC |
| --- | --- | --- | --- | --- | --- | --- |
| Qwen-VL-2.0 | 72.4 | 66.9 | 72.5 | 59.8 | 59.7 | 62.5 |
| Gemini-2.0-Flash | 74.6 | 68.0 | 73.1 | 60.9 | 61.1 | 61.8 |
| GPT-4o | 72.1 | 69.3 | 71.0 | 60.1 | 59.4 | 63.1 |
| GPT-o1 | 75.4 | 71.1 | 73.2 | 61.6 | 63.1 | 62.4 |
| CAP (without CoT) | 76.3 | 71.8 | 79.3 | 73.6 | 70.3 | 74.6 |
| CAP (without Context) | 77.3 | 72.1 | 78.8 | 65.6 | 70.1 | 71.6 |
| CAP | 80.3 | 75.8 | 83.1 | 75.1 | 74.8 | 79.3 |

**Table 6: ROC-AUC Improvement of various MLLM for with our method automatic evaluation methods. Our method consistently achieves great performance on both open-sourced and proprietary models.**
| Method | SA | PC |
| --- | --- | --- |
| Qwen-VL-2.0 | 72.4$\rightarrow$78.6 | 59.8$\rightarrow$74.1 |
| Gemini-2.0-Flash | 74.6$\rightarrow$79.9 | 60.9$\rightarrow$74.0 |
| GPT-4o | 72.1$\rightarrow$78.5 | 60.1$\rightarrow$73.9 |
| GPT-o1 | 75.4$\rightarrow$80.3 | 61.6$\rightarrow$75.1 |

Moreover, even without any fine-tuning, our CAP evaluator achieves 80.3 Semantic Adherence and 75.1 Physical Commonsense on PhyWorldBench, and when this is compared to VideoCon‑Physics’s 82.0 SA and 73.0 PC on the VideoPhy benchmark after fine‑tuning, it highlights CAP’s strong out‑of‑domain performance. Additionally, we conducted fine‑tuning on an 80/20 train/test split of our annotated videos and found that these numbers can be boosted substantially: Qwen‑VL‑2.0 improves from 77.5 → 82.3 SA and 74.5 → 79.1 PC, and GPT‑4o from 78.5 → 84.7 SA and 73.9 → 80.7 PC.

## Appendix D Model Application Details

Open-source generators (code + weights).

- •
Hunyuan (Tencent Hunyuan Community License Agreement) : based on the official 720p repo
[https://github.com/Tencent/HunyuanVideo](https://github.com/Tencent/HunyuanVideo).
- •
LTX-Video (RAIL-M License) : Hugging Face Diffusers pipeline
[https://huggingface.co/docs/diffusers/api/pipelines/ltx_video](https://huggingface.co/docs/diffusers/api/pipelines/ltx_video).
- •
CogVideo v1.5 (Apache-2.0 License) : [https://github.com/THUDM/CogVideo](https://github.com/THUDM/CogVideo).
- •
Open-Sora v2.0 (Apache-2.0 License) : [https://github.com/hpcaitech/Open-Sora](https://github.com/hpcaitech/Open-Sora).
- •
Wanx v2.1 (Apache-2.0 License) : [https://github.com/Wan-Video/Wan2.1](https://github.com/Wan-Video/Wan2.1).
- •
Step-Video-T2V (MIT License) : [https://github.com/stepfun-ai/Step-Video-T2V](https://github.com/stepfun-ai/Step-Video-T2V).
- •
Open-Sora-Plan v1.3 (MIT License) : [https://github.com/PKU-YuanGroup/Open-Sora-Plan](https://github.com/PKU-YuanGroup/Open-Sora-Plan).

Commercial / proprietary generators.

- •
Sora-Turbo, Pika 2.0, Gen-3, Kling 1.6:
accessed via the vendors’ public web interfaces under their standard Terms of Service (no OSS license).
- •
Luma: accessed through the official API.

All third-party code or data are *used* under the above licenses/ToS but are not redistributed in our release.
Our own benchmark assets are released under the MIT License.

## Appendix E Insights into Performance Between Models

Our analysis reveals that both model size and sampling budget significantly impact physics performance. Larger models consistently outperform smaller ones on foundational physics tasks, as shown in Table [7](https://arxiv.org/html/2507.13428v1#A5.T7). For instance, Cosmos 14B outperforms Cosmos 7B across all SA/PC metrics, and CogVideo‑5B beats CogVideo‑1.5B, confirming that scaling yields clear gains in semantic adherence and physical commonsense.

**Table 7: Influence of model scale on physics performance. SA denotes semantic adherence, PC denotes physical commonsense, and Both indicates satisfaction of both criteria. To isolate the effect of size, we compare variants of the same base architecture.**
| Model | Fundamental Physics | Composite Physics | Anti-Physics | Overall Performance |  |  |  |  |  |  |  |  |
| --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- |
| SA | PC | Both | SA | PC | Both | SA | PC | Both | SA | PC | Both |  |
| Cosmos 14B (Agarwal et al., 2025) | 0.374 | 0.268 | 0.212 | 0.331 | 0.243 | 0.182 | 0.091 | 0.069 | 0.020 | 0.333 | 0.241 | 0.184 |
| Cosmos 7B (Agarwal et al., 2025) | 0.354 | 0.243 | 0.201 | 0.339 | 0.221 | 0.185 | 0.088 | 0.054 | 0.020 | 0.323 | 0.218 | 0.178 |
| CogVideo‑5B (Hong et al., 2022) | 0.419 | 0.266 | 0.192 | 0.308 | 0.207 | 0.154 | 0.062 | 0.021 | 0.000 | 0.351 | 0.225 | 0.163 |
| CogVideo‑1.5B (Hong et al., 2022) | 0.382 | 0.247 | 0.175 | 0.246 | 0.199 | 0.148 | 0.073 | 0.021 | 0.010 | 0.310 | 0.210 | 0.150 |

We also observe that generation time correlates with physics accuracy: models with longer sampling schedules (e.g., Hunyuan at 40 min/video or Wanx‑2.1 at 1 hr/video) achieve stronger performance, and simply increasing Open‑Sora’s sampling steps from 50 to 200 yields a significant boost. These findings suggest that, alongside scaling, dedicating more compute to finer‐grained sampling can materially improve adherence to physical laws.

## Appendix F Detailed Performance over All Physics Categories

In the main paper, we presented results across different levels of physics. Here, we provide a detailed breakdown of results for each specific physics category in [Table 8](https://arxiv.org/html/2507.13428v1#A6.T8).

**Table 8: We present the evaluation results of 10 video generation models across 10 physics categories. SA denotes semantic adherence, while PC represents physical commonsense. SA, PC indicates satisfaction of both criteria.**
| Model | Fundamental | Composite Physics | Anti-Physics |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |
| --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- |
| C1 | C2 | C3 | C4 | C5 | C6 | C7 | C8 | C9 | C10 |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |
| SA | PC | Both | SA | PC | Both | SA | PC | Both | SA | PC | Both | SA | PC | Both | SA | PC | Both | SA | PC | Both | SA | PC | Both | SA | PC | Both | SA | PC | b |  |
| Proprietary Models |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |
| Sora-Turbo | 0.552 | 0.429 | 0.362 | 0.279 | 0.144 | 0.135 | 0.337 | 0.163 | 0.115 | 0.524 | 0.476 | 0.371 | 0.578 | 0.500 | 0.363 | 0.356 | 0.178 | 0.129 | 0.136 | 0.019 | 0.019 | 0.408 | 0.306 | 0.276 | 0.540 | 0.320 | 0.270 | 0.136 | 0.078 | 0.039 |
| Gen-3 | 0.448 | 0.371 | 0.276 | 0.223 | 0.097 | 0.097 | 0.250 | 0.125 | 0.077 | 0.366 | 0.436 | 0.267 | 0.444 | 0.212 | 0.152 | 0.188 | 0.129 | 0.099 | 0.107 | 0.000 | 0.000 | 0.275 | 0.235 | 0.167 | 0.394 | 0.192 | 0.141 | 0.108 | 0.029 | 0.020 |
| Kling-1.6 | 0.510 | 0.471 | 0.356 | 0.272 | 0.136 | 0.136 | 0.352 | 0.171 | 0.143 | 0.515 | 0.455 | 0.347 | 0.571 | 0.418 | 0.297 | 0.283 | 0.141 | 0.130 | 0.196 | 0.022 | 0.022 | 0.369 | 0.243 | 0.184 | 0.370 | 0.259 | 0.210 | 0.125 | 0.073 | 0.042 |
| Pika 2.0 | 0.731 | 0.587 | 0.519 | 0.446 | 0.267 | 0.228 | 0.544 | 0.291 | 0.243 | 0.660 | 0.456 | 0.350 | 0.642 | 0.347 | 0.274 | 0.500 | 0.302 | 0.260 | 0.314 | 0.029 | 0.020 | 0.469 | 0.396 | 0.354 | 0.656 | 0.398 | 0.344 | 0.228 | 0.043 | 0.011 |
| Luma | 0.533 | 0.352 | 0.333 | 0.260 | 0.077 | 0.067 | 0.423 | 0.183 | 0.173 | 0.544 | 0.359 | 0.320 | 0.628 | 0.395 | 0.302 | 0.395 | 0.185 | 0.123 | 0.317 | 0.037 | 0.037 | 0.317 | 0.297 | 0.218 | 0.350 | 0.263 | 0.225 | 0.056 | 0.000 | 0.000 |
| Open-sourced Models |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |
| Hunyuan 720p | 0.505 | 0.447 | 0.350 | 0.340 | 0.150 | 0.130 | 0.333 | 0.167 | 0.137 | 0.462 | 0.413 | 0.279 | 0.350 | 0.272 | 0.155 | 0.323 | 0.219 | 0.177 | 0.196 | 0.000 | 0.000 | 0.436 | 0.416 | 0.327 | 0.396 | 0.375 | 0.271 | 0.082 | 0.021 | 0.010 |
| Open-Sora 2.0 | 0.534 | 0.303 | 0.220 | 0.411 | 0.234 | 0.189 | 0.403 | 0.225 | 0.179 | 0.454 | 0.303 | 0.214 | 0.383 | 0.310 | 0.237 | 0.419 | 0.233 | 0.191 | 0.157 | 0.022 | 0.022 | 0.307 | 0.265 | 0.176 | 0.373 | 0.319 | 0.264 | 0.056 | 0.021 | 0.010 |
| Open-Sora-Plan 1.3 | 0.221 | 0.202 | 0.144 | 0.107 | 0.049 | 0.039 | 0.086 | 0.019 | 0.019 | 0.252 | 0.252 | 0.194 | 0.265 | 0.235 | 0.122 | 0.120 | 0.060 | 0.060 | 0.097 | 0.010 | 0.010 | 0.115 | 0.077 | 0.058 | 0.266 | 0.085 | 0.064 | 0.069 | 0.059 | 0.039 |
| CogVideo 5b | 0.476 | 0.272 | 0.223 | 0.333 | 0.125 | 0.115 | 0.438 | 0.198 | 0.156 | 0.470 | 0.500 | 0.350 | 0.495 | 0.326 | 0.179 | 0.304 | 0.174 | 0.130 | 0.143 | 0.010 | 0.000 | 0.367 | 0.306 | 0.235 | 0.413 | 0.304 | 0.228 | 0.062 | 0.021 | 0.000 |
| Step-video-T2V | 0.401 | 0.265 | 0.204 | 0.273 | 0.202 | 0.166 | 0.361 | 0.239 | 0.145 | 0.371 | 0.264 | 0.227 | 0.384 | 0.273 | 0.238 | 0.328 | 0.227 | 0.154 | 0.165 | 0.000 | 0.000 | 0.388 | 0.385 | 0.25107 | 0.446 | 0.368 | 0.31466 | 0.085 | 0.021 | 0.020 |
| Wanx-2.1 | 0.422 | 0.281 | 0.235 | 0.355 | 0.255 | 0.205 | 0.374 | 0.255 | 0.195 | 0.413 | 0.298 | 0.214 | 0.386 | 0.264 | 0.249 | 0.384 | 0.273 | 0.222 | 0.244 | 0.022 | 0.022 | 0.310 | 0.313 | 0.266 | 0.415 | 0.370 | 0.273 | 0.085 | 0.021 | 0.020 |
| LTX-Video | 0.363 | 0.176 | 0.167 | 0.152 | 0.010 | 0.010 | 0.191 | 0.032 | 0.021 | 0.212 | 0.202 | 0.121 | 0.268 | 0.144 | 0.113 | 0.138 | 0.050 | 0.050 | 0.092 | 0.000 | 0.000 | 0.135 | 0.104 | 0.042 | 0.305 | 0.110 | 0.085 | 0.076 | 0.011 | 0.000 |

## Appendix G Limitations

While PhyWorldBench spans a broad set of physical scenarios with its 1,050 prompts across ten categories, it may not fully capture very specialized or highly intricate interactions that occur in some domains or advanced applications. Similarly, our CAP evaluator generally provides balanced feedback, yet it can display a slight preference for videos with particularly polished visual presentation, such as smooth lighting or dynamic camera movement. Future work will focus on expanding the prompt collection to include additional niche phenomena and on enhancing the evaluation workflow to more clearly distinguish visual style from physical correctness.

## Appendix H Code and Data Availability

To ensure full reproducibility, we provide both the generated video assets and all experiment code:

- •
Generated videos and prompts: All videos used in our experiments, along with the corresponding prompt JSON files, are hosted on HuggingFace at [https://huggingface.co/datasets/phyworldbench/phyworldbench](https://huggingface.co/datasets/phyworldbench/phyworldbench). Note that these generated videos are not part of PhyWorldBench, they were instead utilized for our paper experiments. Also, the HuggingFace does not contain the evaluation standards, those are part of the GitHub listed below.
- •
Experimentation code and instructions: The complete implementation of the experiments code as well as detailed setup instructions are available at [https://github.com/ashwin-333/phy-world-bench](https://github.com/ashwin-333/phy-world-bench).
- •
Website: Additionally, this work is available on the NVIDIA Deep Imagination Research website at [https://research.nvidia.com/labs/dir/phyworldbench/](https://research.nvidia.com/labs/dir/phyworldbench/).

## Appendix I Query Used

Here we list the three MLLM queries used in this work, corresponding to the first-stage description prompt ([Table 9](https://arxiv.org/html/2507.13428v1#A9.T9)), the second-stage structured-answer prompt ([Table 10](https://arxiv.org/html/2507.13428v1#A9.T10)), and the detailed-prompt refinement query ([Table 11](https://arxiv.org/html/2507.13428v1#A9.T11)). Note that for proprietary MLLMs such as GPT-o1, GPT-4o and Gemini-2.0-Flash, the model may refuse to evaluate a small fraction of generated videos; the refusal rate is below 3

**Table 9: First prompt for MLLM to generate a detailed description.**
| Suppose you are an expert in judging and evaluating the quality of AI-generated videos. These are<br>frames |
| --- |
| evenly sampled from a generated video from the beginning to the end. This is a generated video from a |
| video model rather than captured from real world,<br>so the video could be low quality, such as fuzzy, |
| inconsistency, especially not following real world physics. Please tell me what is<br>in this video, including |
| what happened and any physics phenomena you observe. Please be sure to include objects in the video, the |
| main event, and any physics phenomena you observe. |

**Table 10: Second prompt for MLLM to generate the final answer**
| Suppose you are an expert in summarization and finding answers. Here is the text description from another |
| --- |
| large language model<br>about an AI generated video: “{previous_response}”.<br>Based on this description, |
| compare the objects and quantities present in the video<br>with the specified object(s): “{object}”. Answer |
| “Yes” if the object(s) could be found in the video, otherwise answer “No”. Also,<br>please check if “{event}” |
| is visually depicted in the video, and answer “Yes” or “No”. Lastly, please check if video satisfies the |
| standards in list: “{physics_phenomenon_list}”, and answer “Yes” or “No” for each standard in the list. |
| Return your evaluation in the following JSON format: |
| “Objects”: “Yes/No”, |
| “Event”: “Yes/No”, |
| “Standard_1”: “Yes/No”, |
| “Standard_2”: “Yes/No”, |
| “…”: “Yes/No” |

**Table 11: MLLM prompt used to get detailed video generation prompts.**
| Refine the sentence: “{prompt}” to contain subject description, action, scene description. (Optional: camera |
| --- |
| language, light and shadow,<br>atmosphere) and conceive some additional actions to make the sentence more |
| dynamic. Make sure it is a fluent sentence, not nonsense. |

## Appendix J Qualitative Analysis on Individual Model

We present here qualitative examples from various models, as shown [Figure 7](https://arxiv.org/html/2507.13428v1#A10.F7), [Figure 8](https://arxiv.org/html/2507.13428v1#A10.F8), [Figure 9](https://arxiv.org/html/2507.13428v1#A10.F9), [Figure 10](https://arxiv.org/html/2507.13428v1#A10.F10), [Figure 11](https://arxiv.org/html/2507.13428v1#A10.F11), [Figure 12](https://arxiv.org/html/2507.13428v1#A10.F12), [Figure 13](https://arxiv.org/html/2507.13428v1#A10.F13), [Figure 14](https://arxiv.org/html/2507.13428v1#A10.F14), [Figure 15](https://arxiv.org/html/2507.13428v1#A10.F15), [Figure 16](https://arxiv.org/html/2507.13428v1#A10.F16), [Figure 17](https://arxiv.org/html/2507.13428v1#A10.F17), [Figure 18](https://arxiv.org/html/2507.13428v1#A10.F18) that demonstrate violations of fundamental physics laws, highlighting their limitations.
Below, we summarize key observations for each model:

- •
CogVideo: Tends to produce distorted objects, such as non-spherical balls that appear dented. Movements are often excessively fast, blurred, erratic, or jittery, sometimes appearing unnaturally sped up.
- •
Gen-3: Generates videos with a cinematic quality, featuring rich lighting and a high level of realism, akin to footage captured with a high-quality camera. Lighting is particularly smooth and well-balanced.
- •
Hunyuan: Produces highly realistic videos, with a quality resembling footage taken by a high-end camera.
- •
Kling: Generates realistic videos, often incorporating dynamic, moving shots that simulate footage captured with a moving camera.
- •
LTX-Video: Produces grainy footage with distorted objects that are difficult to identify. Moving objects often shift shape and struggle to maintain form.
- •
Luma: Tends to generate realistic footage but with overly vibrant or bright colorization. Moving objects sometimes experience distortion or loss of form.
- •
Open-Sora-Plan: Generates relatively realistic videos but with low fidelity, often lacking detail and featuring still objects. Movement can appear overly smoothed or surreal, sometimes resembling slow-motion. The output often has a VHS-like quality.
- •
Open-Sora: Generates videos where objects frequently distort and warp, failing to hold their shape. The outputs often appear less realistic, resembling animation or 3D graphics.
- •
Pika: Produces realistic yet highly stylized videos. Lighting and objects have an unrealistic softness, contributing to a distinctive stylized aesthetic.
- •
Sora: Generates highly realistic videos with cinematic or stylized lighting. Movement is often exaggerated, enhancing the dramatic quality of the footage.
- •
Wanx: Produces visually crisp footage, yet physical interactions are frequently unreliable; rigid bodies may pass through one another, gravity-driven motions appear “floaty,” and impacts are often muted or entirely missing.
- •
Step-Video-T2V: Creates sharp, aesthetically pleasing clips, yet physical reasoning is routinely oversimplified; light fails to refract, flexible objects stay rigid, mass differences do not affect motion, and phenomena such as duplication or buoyancy are either absent or behave contrary to real-world expectations.

Figure: Figure 7: Physics violations observed in videos generated by the Sora-Turbo model. a. Ceiling fan’s blades phase into the ceiling during rotation. b. Bird’s supporting branch phases into nearby leaves, disconnected from the bird’s movement. c. Bottle enters the person’s mouth without spatial adjustment as they react to pepper. d. Person pedals through a bicycle instead of interacting correctly with the pedals. e. Child’s face deforms unnaturally while sliding down a slide.
Refer to caption: x8.png

Figure: Figure 8: Physics Violations Observed in Videos Generated by the Gen-3 Model. a. Buoy remains unnaturally stationary despite visible water movement. b. Child morphs into different positions and exhibits abnormal motion. c. Cyclist pedaling forward moves backward while rounding a curve on a track. d. Rubber ball bounces off the air instead of the wall. e. Child displays abnormal motion, with knees bending unnaturally.
Refer to caption: x9.png

Figure: Figure 9: Physics Violations Observed in Videos Generated by the Kling-1.6 Model. a. Glass with wine remains unaffected by reversed gravity, showing no changes in the liquid’s behavior. b. Small toy airplane and full-sized airplane are depicted as the same size, violating realistic proportions. c. Glass cup glides down instead of falling naturally and slides abnormally after hitting the ground. d. Feather and stone fall at the same speed, inaccurately disregarding differences in air resistance. e. Aluminum can remains intact after being struck by a hammer, failing to collapse as expected.
Refer to caption: x10.png

Figure: Figure 10: Physics Violations Observed in Videos Generated by the Pika 2.0 Model. a. Apple floats in the air, then suddenly falls unnaturally as the camera moves. b. Box moves up without any contact from hands. c. Dog’s face deforms unnaturally along with water motion as it shakes to dry off. d. Plastic bottle resists fully crushing when stepped on. e. Tin foil folds automatically into a person’s hand without proper crumpling motion.
Refer to caption: x11.png

Figure: Figure 11: Physics Violations Observed in Videos Generated by the Luma Model. a. Soccer ball rolls away from the goal in a straight line, showing no natural curve or arc in its path after being kicked. b. Diver exhibits abnormal motion and phases into the water upon jumping from a platform. c. Marble floats unnaturally upward, deviating from expected motion. d. Hands phase into a lump of dough while stretching it out. e. Book exhibits abnormal phasing during its motion.
Refer to caption: x12.png

Figure: Figure 12: Physics Violations Observed in Videos Generated by the Hunyuan 720p Model. a. Glass cup rests on the table instead of phasing through it, failing to follow the anti-physics prompt. b. Two balloons with like charges remain stationary instead of repelling each other, disobeying electromagnetic interaction physics. c. Pendulum swinging between two peaks exhibits irregular and unrealistic motion. d. Steel ball dropping onto a soft clay block does not deform the clay, defying expected material interaction physics. e. Scissors cutting through paper fail to visibly separate the paper, disobeying shear and cutting physics.
Refer to caption: x13.png

Figure: Figure 13: Physics Violations Observed in Videos Generated by the Open Sora Model. a. Train traveling upside down on suspended tracks appears right-side-up, failing to depict the intended anti-physics scenario. b. Flashlight moving across a dark room does not illuminate its surroundings. c. Two billiard balls collide but awkwardly phase into each other instead of bouncing off. d. Backpack loaded off-center remains unnaturally straight, with no tipping or imbalance. e. Bowling ball rolling toward pins exhibits abnormal phasing and unnatural motion.
Refer to caption: x14.png

Figure: Figure 14: Physics Violations Observed in Videos Generated by the Open-Sora-Plan 1.3 Model. a. Lighthouse beam fails to visibly rotate across the ocean, undermining its expected motion. b. Person walking in front of a projector screen casts no shadow on the screen. c. Rolling tire phases into another tire while moving down the street. d. Rock dropped from a cliff glides unnaturally and exhibits abnormal motion and phasing. e. Tightrope walker disobeys gravity with abnormal movement while crossing the wire.
Refer to caption: x15.png

Figure: Figure 15: Physics Violations Observed in Videos Generated by the CogVideo 5b Model. a. Raindrops do not return to the clouds after hitting the ground; instead, they exhibit normal rainfall, failing to demonstrate anti-physics. b. Toy car and real car rolling down ramps are depicted as the same size, violating proportional realism. c. Car driving on an icy road shows no slippage, unrealistically maintaining normal traction. d. Basketball dropped onto the ground displays unnatural bouncing motion. e. Person standing on one leg during yoga exhibits unrealistic human anatomy.
Refer to caption: x16.png

Figure: Figure 16: Physics Violations Observed in Videos Generated by the LTX-Video Model. a. Stationary bicycle appears with a rider, failing to follow the anti-physics prompt of pedaling without one. b. Chair placed on an uneven floor remains stable, inaccurately disregarding the effect of the uneven surface. c. Stone thrown across a river skips the parabolic motion and instead appears directly in the water. d. Rock sitting on the edge of a hill does not fall, disobeying natural gravitational physics and the prompt. e. Wind blowing over sand dunes fails to pick up sand particles, violating expected particle behavior physics.
Refer to caption: x17.png

Figure: Figure 17: Physics Violations Observed in Videos Generated by the Wanx-2.1 Model. a. Basketball dropped from shoulder height rebounds to only a fraction of its release height, indicating excessive inelasticity. b. During the javelin throw, the athlete’s fore-arm visually interpenetrates the javelin shaft, breaking rigid-body constraints. c. Multiple ice cubes clip through one another instead of exhibiting realistic rigid-body collisions and buoyancy. d. The stone ricochets repeatedly on a calm pond surface and never sinks, contrary to expected gravity-driven submersion. e. The aluminum can remains undeformed when struck by the hammer, ignoring momentum transfer and material plasticity.
Refer to caption: x18.png

Figure: Figure 18: Physics Violations Observed in Videos Generated by the Step-Video-T2V Model. a. An object strikes a calm pond, but there is no visible beam of light or refraction as it enters the water. b. A tree branch remains rigid in the wind, never bending or springing back. c. A child and an adult jump on a trampoline yet rebound to identical heights, ignoring mass-dependent dynamics. d. A single coin was expected to duplicate continuously, but no new coins appear. e. A balloon with a hole should deflate downward; instead, it inexplicably rises, contradicting physics.
Refer to caption: x19.png

## Appendix K Further Analysis on Generative Models Struggling with Certain Physical Scenarios

We identify three key factors behind current models’ failures on physics‑intensive scenarios: (1) Training Data Distribution: large web‑scale corpora such as WebVid, Panda70M, or other private web‑crawled datasets are dominated by everyday scenes—our analysis shows this yields strong performance on C1 (Object Motion and Kinematics), C4 (Fluid and Particle Dynamics), C5 (Lighting and Shadows), C8 (Rigid Body Dynamics), and C9 (Human and Animal Motion) but underrepresents abstract or multi‑object interactions; (2) Video Captioning Process: MLLM‑based prompt extraction captures main objects and events but routinely omits fine‑grained physical dynamics; and (3) Model Architecture Limitations: even with detailed prompts, current architectures emphasize short‑term frame coherence without physics priors, leading to physically inconsistent outputs—by contrast, models like TORA explicitly model trajectories for better motion fidelity.

## Appendix L Leaderboard

[Table 12](https://arxiv.org/html/2507.13428v1#A12.T12) presents the leaderboard performance on both human evaluation and our designed CAP. The performance shows a highly consistency and only difference is that for proprietary models, Sora ranked number 2 while Kling ranked number 2 in CAP. Through further analysis, we found that Kling’s more cinematic style can bias CAP toward higher scores despite only moderate physical correctness, whereas our human annotators—who were instructed to focus strictly on physics plausibility—penalize these artifacts, causing Sora to rank higher under human evaluation.

**Table 12: Leaderboard of open-sourced models and proprietary models. We report leaderboard on both human evaluation and CAP.**
| Open models | Proprietary models |  |  |
| --- | --- | --- | --- |
| Human | CAP | Human | CAP |
| Wanx | Wanx | Pika 2.0 | Pika 2.0 |
| Hunyuan | Hunyuan | Sora-Turbo | Kling |
| Step-Video-T2V | Step-Video-T2V | Kling | Sora-Turbo |
| Open-Sora | Open-Sora | Luma | Luma |
| Cogvideo | Cogvideo | Gen-3 | Gen-3 |
| Open-Sora-Plan | Open-Sora-Plan |  |  |
| LTX-video | LTX-video |  |  |

## Appendix M Influence of Multiple Objects

We noticed that generation models often fail when the prompt is more complex, such as when containing multiple objects. We show examples of Sora failing when given prompts with multiple objects in [Figure 19](https://arxiv.org/html/2507.13428v1#A13.F19).

Figure: Figure 19: Sora failed examples on multi-object. We found that generation models tend to fail when the prompt is relatively complex, such as when containing multiple objects.
Refer to caption: x20.png

## Appendix N Dataset Structure

The dataset is organized into three main categories of physics phenomena: Fundamental Physics, Composite Physics, and Anti-Physics. Each category is divided into subcategories, which are further broken down into five sub-subcategories, as summarized below:

### Fundamental Physics

- (a)
Object Motion and Kinematics: Linear Motion, Parabolic Motion, Circular Motion, Rotational Motion, Combined Translational and Rotational Motion
- (b)
Interaction Dynamics: Collisions, Friction and Sliding, Gravity and Free Fall, Electromagnetic Interactions, Tension and Compression
- (c)
Energy Conservation: Potential to Kinetic Energy, Elastic Energy, Energy Loss (Damping), Conservation of Mechanical Energy, Heat Transfer and Phase Changes.
- (d)
Fluid and Particle Dynamics: Water Motion, Smoke and Gas Dynamics, Particle Behavior, Buoyancy and Floating, Viscosity and Flow
- (e)
Lighting and Shadows: Consistency of Illumination, Object Occlusion, Chromatic Aberration and Dispersion, Reflection and Refraction, Shadow Softness and Penumbra
- (f)
Deformations and Elasticity: Material Flexibility, Elastic Rebound, Plastic Deformation, Brittle Fracture, Thermal Expansion and Contraction

### Composite Physics

- (a)
Scale and Proportions: Consistency of Forces Across Scales, Environmental Effects on Different Scales, Proportional Energy Transfer, Structural Integrity Across Scales, Scaling of Mechanical Properties
- (b)
Rigid Body Dynamics: Rotation and Torque, Balance and Stability, Center of Mass, Impact and Deformation, Shear and Cutting
- (c)
Human and Animal Motion: Anatomical Feasibility, Balance and Posture, Locomotion Across Terrains, Reflexive and Involuntary Motion, Coordination and Fine Motor Skills

### Anti-Physics

- (a)
Anti-Physics: Defying Gravity, Energy Creation or Perpetual Motion, Object Phasing and Surreal Interactions, Time Reversal and Alteration, Infinite Duplication and Division

This hierarchical structure ensures coverage of realistic and non-realistic physical scenarios for diverse analysis. Figure [20](https://arxiv.org/html/2507.13428v1#A14.F20) visualises how these ten categories map onto the three physics levels and highlights the numbers of prompts in each branch.

Figure: Figure 20: Dataset Overview. PhyWorldBench contains ten physics categories under three different physics levels.
Refer to caption: x21.png

## Impact Statement

PhyWorldBench establishes a new standard for physics-focused text-to-video generation, significantly elevating the field’s expectations of realism and accuracy. By rigorously assessing models’ capabilities across a comprehensive spectrum of physical phenomena—including motion, energy transfer, and complex interactions—this benchmark not only highlights critical gaps in current systems but also guides researchers toward developing more robust and physics-aware solutions. Moreover, PhyWorldBench acknowledges the ethical implications of physics-driven video synthesis, emphasizing the need for transparency, fairness, and responsible deployment. By mitigating risks such as misinformation, biased simulations, and unintended real-world consequences, it ensures that advancements in this field are aligned with scientific integrity and societal well-being. Ultimately, PhyWorldBench paves the way for advanced applications in education, scientific visualization, and industry, demanding both photorealism and fidelity to real-world physics while upholding ethical standards.

## Appendix O Data Release

We publicly released a comprehensive dataset that includes the core assets of PhyWorldBench, the 1,050 JSON prompt files, the evaluation standards, and the physics categories and subcategories, used to assess physical realism in text-to-video models. We have also attached over 10k of the generated videos that we used for experimentation. The licensing terms for the videos will follow those set by the respective model creators, as referenced in this work, while the curated captions and human annotations will be provided under the MIT License. Additionally, our release will include standardized evaluation protocols, and evaluation scripts to facilitate rigorous assessment. The entire project will be open-sourced, ensuring free access for research and academic purposes.

## Peer Reviews

confidence: 4
contribution: 3
initial_rating: 6
presentation: 3
questions: Questions:   
（1）Questions About CAP Evaluator’s Aesthetic Bias  
You note the Context-Aware Prompt evaluator prefers visually polished videos but provide no quantitative data on this bias. How often does this preference lead to misclassifying physically incorrect yet visually polished videos as physically plausible? Does a visually vivid video with physical violations score higher than a plain but physically correct one? Have you tested if revising CAP prompts to explicitly ignore aesthetics reduces this bias? Clarifying these points will confirm if CAP’s objectivity is compromised and if adjustments can fix it.  
（2）The comparison with the methods of predecessors is not sufficient.  
A comparison between the dataset and some previous related datasets, such as VideoREPA[1], WISA[2], NewtonGen[3], etc.?  
（3）Niche Fundamental Physics Performance
PhyWorldBench covers 10 main physics categories, but your experiments focus on common phenomena and lack breakdowns for niche subcategories like pressure-dependent phase changes or non-uniform heat transfer. Do you have data on model performance in these niche areas? If not, why were they excluded from detailed analysis? Understanding these gaps will help researchers target specific physical principles models struggle with. If the author's reply is strong and reasonable, I will consider increasing my score.  

[1] VideoREPA: Learning Physics for Video Generation through Relational Alignment with Foundation Models  
[2] Wisa: World simulator assistant for physics-aware text-to-video generation  
[3] NEWTONGEN: PHYSICS-CONSISTENT AND CONTROL-LABLE TEXT-TO-VIDEO GENERATION VIA NEURAL NEWTONIAN DYNAMICS
soundness: 2
strengths: Strengths:   
（1）Originality  
It proposes a three-tier (Fundamental, Composite, Anti-Physics) hierarchical structure with 10 main physics categories, going beyond prior benchmarks by using Anti-Physics scenarios to test true physical understanding. The CAP evaluator creatively combines LLMs with domain constraints, separating aesthetics from physics via two-step reasoning to fix traditional evaluators’ inaccuracies. It also offers three prompt variants to test model adaptability, filling gaps of static-prompt benchmarks.  
（2）Quality  
Its benchmark is built via a rigorous three-stage process (literature/expert category definition, LLM-human prompt generation, expert validation) to ensure 1,050 prompts are diverse and accurate. CAP aligns with human evaluation (ROC-AUC: 80.3 for SA, 75.1 for PC) and outperforms baseline LLMs. Experiments cover 12 models (5 proprietary, 7 open-source) with 12,600 videos and systematic analyses to avoid cherry-picking.  
（3）Clarity  
It follows a clear "problem-solution-validation" flow, with the introduction outlining existing benchmark limits and methodology linking components to specific issues. Technical details (CAP’s reasoning, "Yes/No" criteria) are explained in plain language without jargon. Consistent terminology and condensed key findings make it readable for physics and AI researchers.  
（4）Significance  
As a standardized tool, its open benchmark and CAP provide universal metrics for tracking text-to-video physical realism. Prompt design insights (e.g., Physics-Enhanced Prompts boost PC) guide real-world uses like scientific visualization. It shifts evaluation focus to physical correctness, identifies model weaknesses, and reduces educational misinformation risks.
summary: Summary:   
This paper addresses the critical limitation of existing text-to-video (T2V) generation models—their inability to adhere to physical laws despite producing photorealistic content—by proposingPhyWorldBench, a comprehensive benchmark for evaluating physical realism in T2V models.  

Contributions:  
（1）Development of a comprehensive physics benchmark: PhyWorldBench fills the gap of lacking holistic tools to evaluate physical realism in text-to-video models. Its hierarchical structure (three levels, 10 main categories, 50 subcategories) covers diverse physical scenarios—from basic motion and energy conservation to anti-physics scenarios—and includes 1,050 well-curated prompts with variations, enabling systematic testing of models’ physical reasoning capabilities.  
（2）Creation of a zero-shot automatic evaluator: The proposed Context-Aware Prompt method resolves the limitations of traditional evaluation. By guiding large language models to explicitly assess AI-generated videos and use chain-of-thought reasoning, it achieves high accuracy in evaluating physical realism, providing a scalable and objective alternative to costly human evaluation.  
（3）Extensive evaluation of state-of-the-art models: The paper conducts large-scale tests on 12 leading text-to-video models, generating 12,600 videos. This evaluation identifies key challenges models face—such as struggling with complex interactions and prioritizing cinematic aesthetics over physics—and quantifies performance gaps across different physics categories, offering clear directions for future model improvement.
weaknesses: Weaknesses:   
（1）CAP Evaluator’s Aesthetic Bias  
The Context-Aware Prompt (CAP) evaluator favors visually polished videos (e.g., smooth lighting, dynamic camera movement) over physically correct ones, which conflicts with the benchmark’s goal of prioritizing physical plausibility. However, the paper doesn’t quantify this bias or fix it. For example, a visually vivid but gravity-violating floating apple video might get a higher score than a plain yet physically correct one.  
（2）Inadequate Niche Fundamental Physics Analysis  
PhyWorldBench focuses on common physics (e.g., free fall) but ignores niche subcategories of fundamental physics, like pressure-dependent phase changes (water boiling at high altitude) or non-uniform heat transfer (a metal rod heating unevenly). It only reports broad category performance, hiding which specific principles models struggle with.  
（3）No Long-Duration Video Validation  
The paper doesn’t specify video duration or test how performance scales with length. Physical inconsistencies (e.g., trajectory drift) worsen in longer videos, which are needed for real uses (e.g., education). Without this data, the benchmark’s real-world value is limited.  
（4）Lack of Specialized Benchmark Comparisons  
The paper only compares to general physics benchmarks (e.g., VideoPhy) but not specialized ones like Morpheus (real-world experiment focus) or T2VPhysBench (first-principles physics). This hides PhyWorldBench’s unique value.

---

confidence: 4
contribution: 2
initial_rating: 6
presentation: 3
questions: 1. The CAP automated evaluator, while highly accurate in classification, is noted to possess an "aesthetic bias" that causes ranking discrepancies with human judgment; given the non-scalability of human evaluation, what specific plans do the authors have to mitigate or quantify this aesthetic bias in the MLLM, ensuring the CAP tool reliably focuses on physical correctness rather than visual style?
soundness: 3
strengths: 1. The benchmark offers the most comprehensive and systematic coverage of physical phenomena
2. he proposed CAP automated MLLM evaluator significantly improves the accuracy of machine-based physical assessment by using a context-aware prompt, offering a cost-effective and reproducible alternative to expensive human studies.
summary: This paper introduces PhyWorldBench, a comprehensive, large-scale benchmark for evaluating physical realism in Text-to-Video (T2V) models. It features 1050 prompts systematically categorized across 10 physics domains and uniquely includes an "Anti-Physics" category to test genuine physical understanding versus mere pattern imitation from training data. The study employed extensive human evaluation (via Amazon Mechanical Turk) on 12 leading models and proposed a novel automated MLLM evaluation method called CAP (Context-Aware Prompting). Findings show that all current T2V models have very low physical success rates (Pika at 26.2% being the highest), and notably, they tend to revert to realistic physical phenomena when instructed to perform anti-physical actions.
weaknesses: 1. The automated CAP evaluator suffers from an "aesthetic bias," leading to ranking discrepancies with the human gold standard for high-quality, closed-source models. This limits its reliability as a trustworthy, scalable proxy for future research.

2. Many of the investigated physics phenomena (e.g., fluid viscosity, electromagnetic forces) are challenging to assess accurately by eye in a video. This visual complexity makes human scoring error-prone and may also challenge the perceptual accuracy of the MLLM evaluator.

3. The evaluation primarily focuses on whether the final physical state is correct, but it lacks a detailed metric to measure if the physical process is smooth, coherent, and temporally consistent. This overlooks a critical challenge in dynamic T2V modeling.

---

confidence: 3
contribution: 3
initial_rating: 6
presentation: 3
questions: - The paper mentions models sometimes "rationalize" physics violations by generating static scenes. Could you quantify how often this happens across models? This seems like an important failure mode worth analyzing systematically.
- Have you considered releasing a "difficulty rating" for each prompt based on model performance? This could help researchers identify particularly challenging scenarios to focus on.
soundness: 3
strengths: - Important and timely problem. The paper tackles a crucial gap in bridging video generation and physical reasoning. The motivation is clear and aligns well with current research trends.
- Comprehensive experimental design. The evaluation is thorough - 12 models tested, 12,600 videos generated, large-scale human annotation via MTurk. They evaluate across a diverse set of tasks, object falling, rolling, fluid dynamics, etc. 
- Practical automated evaluator. The CAP method achieves 80.3% ROC-AUC for semantic adherence and 75.1% for physical commonsense (Table 2), which is pretty solid for zero-shot evaluation. 
- Clear organization and presentation. The hierarchical organization is logical and makes the benchmark easy to understand and extend. The tables and plots are easy to follow.
summary: This paper introduces PhyWorldBench, a comprehensive benchmark for evaluating physical realism in text-to-video generation models. The benchmark contains 1,050 prompts spanning 10 main physics categories (fundamental, composite, and anti-physics), with each category divided into 5 subcategories and 7 scenarios. The authors evaluate 12 state-of-the-art models (5 proprietary, 7 open-source) and propose CAP (Context-Aware Prompt), a method using MLLMs for automated evaluation. Results show that even the best models (Pika 2.0 achieving 26.2% success rate) struggle significantly with physical realism, particularly with complex interactions and anti-physics scenarios.
weaknesses: - No end-to-end pipeline for new models. The paper doesn’t provide a unified automatic evaluation framework for new video generation models. While CAP is proposed as an automatic evaluator, there's no clear standalone pipeline or released code/API for researchers to evaluate their own models. The paper mentions "we will open-source our codebase" (reproducibility statement) but it's unclear if this includes an easy-to-use evaluation script. For a benchmark paper, providing a simple evaluate interface would greatly increase adoption. The current workflow seems to require manual video generation then evaluation, which is cumbersome. For practical use, an automated submission or leaderboard system would make the benchmark much more usable.
- Metric definitions design lack justification. A few metrics (like the physics consistency score) are described in high-level terms but not fully formalized. It’s hard to know how reproducible they are from the text alone, e.g., whether they use learned physical estimators or ground-truth simulations. The Yes/No binary evaluation (Section 3.1) might be too coarse-grained - a partial physics violation might deserve a score between 0 and 1. The paper acknowledges models often "rationalize" actions rather than fail outright (Section 4.4), suggesting a more nuanced scoring could capture important phenomena.
- Limited real-world coverage. Most of the benchmark focuses on synthetic data (e.g., MuJoCo or Unity scenes). It would be great to see more real videos or robotic interactions to test generalization.
- The scope of the benchmark is somehow overlapped with previous works, e.g., the inclusion of an "anti-physics" category is similar as “counterfactual prompts” in [1]. 

[1] T2VPhysBench: A First-Principles Benchmark for Physical Consistency in Text-to-Video Generation. 2025.

---

confidence: 5
contribution: 1
initial_rating: 4
presentation: 3
questions: See Weakness.
soundness: 2
strengths: 1. Establishing a comprehensive and fine-grained categorization of physical dimensions which is larger than existing benchmarks.
2. The evaluation results are shown to be consistent with human judgments.
summary: This paper proposes a framework for evaluating video generation models in terms of their understanding and generation of physical concepts. The authors establish a comprehensive and fine-grained categorization of physical dimensions, and employ GPT-o1 as the evaluation model to assess ten different models from the perspectives of Semantic Adherence and Physical Commonsense. The evaluation results are shown to be consistent with human judgments.
weaknesses: 1. Apart from introducing a more fine-grained implementation in the dimension categorization, I do not see any genuine innovation in this paper. The authors’ claim regarding Anti-Physics has already been presented and evaluated in paper: Impossible Videos. Another point emphasized by the authors—using prompts of different granularities for the same scenario and concluding that finer-grained prompts lead to better results—has also been thoroughly discussed in paper VBench-2.0.
2. In addition to the lack of novel and valuable conclusions, the evaluation methodology used in this work is much less rigorous compared to previous approaches such as VBench-2.0 and PhyGenBench. Notably, the authors do not consider the impact of camera motion on physical phenomena. For example, in the case of an apple falling, if the camera moves synchronously with the apple, how does the model determine whether the apple is truly falling? The paper merely utilizes a stronger MLLM, GPT-o1, which superficially appears to yield more accurate results. However, the use of a closed-source model introduces a critical flaw: as the model is updated, previous results may become irreproducible, which is a fatal issue for any benchmark.


Decision: Accept (Oral)