Towards World Simulator: Crafting Physical Commonsense-Based Benchmark for Video Generation
Authors: Fanqing Meng
Jiaqi Liao
Xinyu Tan
Quanfeng Lu
Wenqi Shao
Kaipeng Zhang
Yu Cheng
Dianqi Li
Ping Luo
Venue: ICML 2025 poster (2025)
Keywords: Physical Commonsense
Evaluation
World Simulator
Primary area: applications->computer_vision

Abstract: Text-to-video (T2V) models like Sora have made significant strides in visualizing complex prompts, which is increasingly viewed as a promising path towards constructing the universal world simulator. Cognitive psychologists believe that the foundation for achieving this goal is the ability to understand intuitive physics. However, the capacity of these models to accurately represent intuitive physics remains largely unexplored. To bridge this gap, we introduce PhyGenBench, a comprehensive \textbf{Phy}sics \textbf{Gen}eration \textbf{Ben}chmark designed to evaluate physical commonsense correctness in T2V generation. PhyGenBench comprises 160 carefully crafted prompts across 27 distinct physical laws, spanning four fundamental domains, which could comprehensively assesses models' understanding of physical commonsense. Alongside PhyGenBench, we propose a novel evaluation framework called PhyGenEval. This framework employs a hierarchical evaluation structure utilizing appropriate advanced vision-language models and large language models to assess physical commonsense. Through PhyGenBench and PhyGenEval, we can conduct large-scale automated assessments of T2V models' understanding of physical commonsense, which align closely with human feedback. Our evaluation results and in-depth analysis demonstrate that current models struggle to generate videos that comply with physical commonsense. Moreover, simply scaling up models or employing prompt engineering techniques is insufficient to fully address the challenges presented by PhyGenBench (e.g., dynamic scenarios). We hope this study will inspire the community to prioritize the learning of physical commonsense in these models beyond entertainment applications. We will release the data and codes at https://github.com/OpenGVLab/PhyGenBench

Towards World Simulator:
Crafting Physical Commonsense-Based Benchmark for Video Generation
Fanqing Meng" 1 Jiaqi Liao 2 Xinyu Tan
Quanfeng Lu 12 Wenqi Shao 2 Kaipeng Zhang² Yu Cheng
Dianqi Li Ping Luo
Abstract
inspire the community to prioritize the learning of physical commonsense in these models beyond entertainment applications. We release the data and codes at https: //g: thub. wwt./penm/Loabb//////coompp PhyGenBench
Text-to-video (T2V) models like Sora have made significant strides in visualizing complex prompts, which is increasingly viewed as promising path towards constructing the universal world simula- tor. Cognitive psychologists believe that the foun- dation for achieving this goal is the ability to un- derstand intuitive physics. However, the capacity of these models accurately represent intuitive physics remains largely unexplored. To bridge this gap, we introduce PhyGenBench, compre- hensive Physics Generation Benchmark designed to evaluate physical commonsense correctness in T2V generation. PhyGenBench comprises 160 carefully crafted prompts across distinct phys- ical laws, spanning four fundamental domains, which could comprehensively assess models' un- derstanding of physical commonsense. Alongside Bennhh we propose a novel evaluation framework called PhyGenEval. This framework employs a hierarchical evaluation structure utiliz- ing appropriate advanced dision-language mod- els and large language models to assess physical commonsense. Through PhyGenBench and Phy- GenEval, we can conduct large-scale automated assessments of T2V models' understanding of physical commonsense, which aligns closely with human feedback. Our evaluation results and in- depth analysis demonstrate that current models struggle to generate videos that comply with phys- ical commonsense. Moreover, simply scaling up models employing prompt engineering tech- niques is insufficient to fully address the chal- lenges presented by PhyGenBench (e.g., dynamic physical phenomenons). We hope this study will
1. Introduction
Text-to-video (T2V) models such as Sora have made signifi- cant strides in visualizing complex ideas and scenes based on textual input (Yang et al., 2024; Wang et al., 2023). These advancements are increasingly viewed as promising path towards constructing universal simulators of the physical world, which holds immense promise for video generation (Zhu et al., 2024), autonomous driving (Gao et al., 2024), and the development of embodied agents (Mazzaglia et al., 2024). Cognitive psychology posits that intuitive physics, which demonstrated even by human infants (Wood et al., 2024; Battaglia et al., 2013), is essential for achieving this goal. Intuitive physics emphasizes rendered scenes should be visually and interactively natural to humans, rather than adhere to strict physical accuracy. Consequently, path towards developing a world simulator (Xiang et al., 2024), video generation should first be capable of accurately reproducing simple yet fundamental physical phenomenons. However, even state-of-the-art models trained on vast re- sources (Tan et al., 2024) encounter difficulties in correctly generating seemingly trivial physical phenomenons, de- picted in Figure 1, the model fails to understand that the stone should sink in water. This clear pitfall shows sub- stantial gap between current video generation models' and human's 's understanding of basic physics. It reveals how far these models are from being true world simulators.
the
Given this context, it becomes important to assess the extent to which current T2V models can capture intuitive physies in their generated outputs. This requires the development of comprehensive evaluation frameworks that beyond tradi- tional metrics. While numerous Eoxt-to-Video (T2V) evalu- ation benchmarks have emerged (Sun et al., 2024; Huang et al., 2024), they primarily focus on various qualities of generated videos (e.g., motion smoothness, background
Equal contribution 'Shanghai Jiao Tong University "Shanghai AI Laboratory 'The Chinese University of Hong Kong "The University of Hong Kong, Correspondence Wenqi Shao <shaowenqi @ pjlab.org.cn. en>, Ping Luo galno@cs.hku.hke
Proceedings of the ndnd International Conference Machine Learning, Vancouver, Canada. PMLR 267, 2025. Copyright 2025 by the author(s).

---

PhyGenBench
Gravity
Busyancy
Flasticity
Friction
rspril
Mechanics
bottle deebyy the aiotisn, lete surveundin crea
pestly pleced
vianant, alartic bastenball shrown forsefully towords groan) canturty upen insort.
Biu norker used to write, the white the intoraction behween the warker and whitebward sprface.
the
peal Filled
Reflection
Refraction
Diffraction
Tyndall
Optics
plane glidng exen crystil-
sleely interted into gleas crymtal-skar reveding the viead changes sccar es Melting
lange number saap bubbles
may light genereted by aprsjacter particiss.
Sublimation
Boiling
Liquefaction
I
Thermal
A tinclapse coptomer the tranofermction exponnd grachell tempershurs
hinelapse troeffrmmtion of be crear of the aigeficantly
timelapss caphures the traneformation tee is teops? the temperature intter rpper in hanid eminotyent surface.
incrosiing
3o0 dieree Celdus
rapidy moss shove degres Celata
Hardness
Salubility
Dehy property
Flame Reaction
HUn concentrated auffuric add anto place
Material Properties
frogle ags with solid noch surface, whene coliide
clear glaas juice le gestly poured diids
A place of capper ls lonited, enitting shoadly.
styefiourt
Figure 1: Samples of videos generated by Kling or Gen-3 PhyGenBench with different aspects. The results show that current T2V models struggle generate videos that align with physical commonsense (e.g., the lack of a plane's reflection in water in the first video of the second row).
consistency) or spatial relationships, failing to address the critical issue of whether the generated videos adhere to fundamental physical laws. Although some studies have explored the alignment of generated videos with dynamic motions naturalness (Bansal et al., 2024), their benchmarks fail to succinctly capture fundamental physical laws pro- pose sufficiently robust evaluation methods. Therefore, the development of benchmarks and evaluation methodologies specifically tailored to assess intuitive physics generated videos remains a critical yet largely unexplored frontier.
To address these challenges, we propose PhyGenBench and PhyGenEval to automate the evaluation of physical com- monsense understanding capability from T2V models. Phy- GenBench is designed evaluate physical commonsense based on fundamental physical laws in text-to-vides video gen- eration. Inspired by (Halliday et al., 2013), we categorize physical commonsense in the world into four main areas: mechanics, optics, thermal, and material properties. Then principle physical laws and easily observable physical phe- nomenons are identified for each category, resulting in com- prehensive 27 physical laws and 160 validated prompts in con-
There two challenges impeding the evaluation of physi- cal commonsense in T2V models. On one hand, there is lack of benchmarks focused evaluating physical common- sense. This requires selecting semantically simple physical phenomenons that exhibit clear physical phenomena, allow- ing for accurate assessment by either humans or machines. the other hand, there lack of corresponding evalu- ation metries. Traditional metrics like FVD (Unterthiner al., 2018) exhibit limitations in detecting implausible mo- tions (Brooks et al., 2022) and necessitate reference videos, which are often challenging to procure for novel scenes. Recent studies have used video-based VLMs for compre- hensive video evaluation (He et al., 2024b; Sun et al., 2024). However, they often struggle to correctly assess physical commonsense. This limitation stems from their inadequate understanding of physical laws (Jassim et al., 2023) and the fact that these methods are not specifically designed to evaluate physical laws.
the proposed benchmark. Through brainstorming, struct prompts that easily reflect physical laws using sources like textbooks (Harjono et al., 2020). This process results in comprehensive but simple set of prompts reflecting physi- cal commonsense, which are sufficiently clear for evaluation. shown Figure 1, the correctness of physical common- sense in PhyGenBench can be observed through clear phe- nomena (e.g., plane should have reflections in water) On the other hand, benefiting from the simple yet clear physi- cal phenomena in Phy PhyGeenncch prompts, we can propose PhyGenEval, which is novel video evaluation framework for assessing physical commonsense correctness in PhyGen- Bench. PhyGenEval first uses GPT-4ol to analyze physical laws in text, addressing the poor understanding of physical common sense in video-based VLMs. Moreover, consid- ering that previous evaluation metries did not specifically
'The version g8t4
2

---

PhyGenBench
target physical correctness, we propose three-tier hierar- chical evaluation strategy for this aspect, transitioning from image-based to comprehensive video analysis: single image, multiple images, and full video stages. Each stage employs distinct VL.Ms along with custom instructions generated by GPT-4o to form judgments. By combining PhyGenBench and PhyGenEval, we can efficiently evaluate different T2V models' understanding of physical commonsense at scale, producing results highly consistent with human feedback.
aspect, which forms the foundation for realizing a world simulator. To address this gap. we introduce PhyGenBench, benchmark designed to comprehensively measure T2V models' understanding of physical commonsense. We pro- vide more releated works in Appendix A
3. PhyGen
Inspired by (Swartz, 1985). we first define the following terms: "Physical Commonsense.:" Basic intuitive under- standing of how physical objects and actions behave in ev- eryday life; "Physical Laws:" " Universal scientific principles that describe consistent behaviors in nature; "Physical Phe- nomenon:" " Observable events or processes caused by the interaction of physical laws. The purpose of PhyGenBench is to evaluate whether T2V models understand physical com- monsense, while each prompt in PhyGenBench presents a clear physical phenomenon and an underlying physical law.
The contributions of our work are three-fold. i): We pro- posed PhyGenBench, which compasses a wide range of clear physical phenomenons and explicit physical laws. This benchmark can comprehensively measure whether T2V models understand intuitive physies and indirectly assess their gap from world simulator capabilities ii): Along with the benchmark, we propose an automated evaluation frame- work - PhyGenFval, which overcomes the challenges of assessing the correctness of physical commonsense with other metrics and demonstrates high consistency with hu- man feedback on PhyGenBench, enabling users to conduct large-scale automated testing of various T2V models. iii): We conduct extensive evaluations of popular T2V models, even the pess-performnn model, Gen-3, scores only 0.51. This indicates that current models are still far from function- ing as world simulators. Based on our evaluation results, we conduct an in-depth analysis and discover that addressing issues such as dynamics is still challenging through prompt engincering or simply scaling up model. We hope this work inspires the community to focus on the learning of physi- cal commonsense in T2V models, rather than merely using them as tools for entertainment.
Overview. As illustrated in Figure (a), PhyGenBench encompasses four major categories of physical common- sense: "Mechanics", "Optics". "Thermal", and "Material Properties". It incorporates 27 physical phenomena with intrinsic physical laws reflected by the corresponding de- signed 160 prompts:
1. "Mechanics" covers common mechanical laws: gravity, buoyancy, solid pressure, atmospheric pressure, elasticity, friction, and surface tension, with 40 validated prompts. For example, we use "A piece of iron is gently placed on the surface of the water in a tank filled with water" to test T2V model's understanding of Buoyancy, where the iron should sink due to its higher density compared to water.
2. "Optics" " categorizes 6 aspects based on light phenomena: reflection, refraction, scattering, dispersion, interference & diffraation, and trraight-line propagation, yielding 50 prompts. A prompt like "a kite soaring above smooth and tranquil pond" is used to test reflection.
2. Related work
2.1. Benchmarks for text-to-video generation
The rapid advancement of text-to-video (T2V) generation models has necessitated various benchmarks for accurate assessment. Traditional works in video generation, such as FVD (Unterthiner et al., 2018), rely on datasets like UCF- 101 (Soomro, 2012) and Kinetics-400 (Kay et al., 2017), which are limited in scope. Recent benchmarks, includ- ing VBench (Huang et al., 2024) and EvalCrafter (Liu et al., 2024c), aim comprehensively evaluate general video qual- ity across multiple dimensions. In contrast, some studies focus fine-grained evaluation of text-tovvideo (T2V) mod- els from specific aspects. For instance, T2V- T2VcCompBench (Sun et al., 2024) assesses compositional generation capa- bilities, while DEVIL (Liao et al., 2024) evaluates dynamic characteristics of generated videos. Although some research like VideoPhy (Bansal et al., 2024) efforts address the dy- namic motions naturalness of video generation, their bench- marks fail to suceinetly capture fundamental physical laws. Consequently, most existing works overlook this crucial
3. "Thermal" considers phase transitions: Solidification, Melting, Liquefaction, Boiling, deposition, Sublimation, comprising 30 prompts. Inspired by ChronoMuuicBencch (Yuan et al., 2024), the vaporization process is evaluated by the prompt "a timelapse capturing the transformation of water as the temperature rapidly rises above C00"
4. "Material Properties" includes 5 physical properties (color, hardness, solubility, combustibility, and flame reac- tion) and 3 chemical properties (acidity, redox potential, and dehydrating properties), resulting in 40 prompts. We reflect material properties, e.g.. "hardness", through the prompts with expected phenomena, e.g., "an egg being hurled with significant force towards a rock", where the egg should break while the rock remains intact.
Multiple physical laws could be included in a single prompt, which may bring confusion to the evaluation of physical
3

---

PhyGenBench
Physical Knowledge
Physical Laws
160 Prompts
Machanics (25%)
Optics (31%)
place wiite
A cup of water albey poumed out
teto graithy,
bioodoward.
ligid...
kitte swanig frenggal pood
vilmart clsstic Farcefuly throwe the groand
Mott crial Preperties (25%)
Thermal (39%)
STF
>)1>
Oeon by Buaanoo Said freusrt ateophenc frasen bigenoo Soatio sereonn 1o Line Aepegetor. lnteri bepattinn hon, Gatr ane Baactorr Saki
dilictte tragle, sap dyyefoort farce towardc ruppedd apil
The munefommaionn of roun humid over decodes.
Conta bybl bai
(a) The overview of the PhyGenBench
Conceptualization
Prompt Engineering
Quality Control
Prompt
Reviow Griterto
Prompt: cilidee with
Prompt:
Physical
Generated Questions:
Emplicatiin
Carnectness
Physical lose:
2. Simplicity
2. CCmpleteness
Mechania
Optics (50 somples
3. Diversity
3. ..Cnmsponddnce
(40 somples:
3.Completeness
4. Exprressiveness
Prompt Augmentation
Questions Generation
Detailed Prompts
Detailed
Requieeents
A delican, frught sge harled with suld' rekk aurlice ahans celliie
Phystcal Phenomens Detection:
prompt for locating, QUAs based
Material
single inage
Properties (40 somples)
Diversity Enhancement
Physics Order Verification:
(30 somples)
A retriival prompt: Two ideal descriptions based on frone pairs The explict description of whole process
Replace Object
est
Object:
Object!
oq-
clearly Unrealistic Slightly Unreal stic. Almost Reakettic
4: lovel descriptin fianiiiiiinn..... Completely Fontostical;
Overall Naturalesss Evaluntion:
sutdd.
(b) The data construction pipeline of the PhyGenBench
Figure 2: (a) is the overview of the proposed PhyGenBench. (b) is the PhyGenBench data pipeline, which covers four physics categories. We select key physical laws and manually craft initial prompts that reflect the corresponding physical phenomena. GPT-4o adds details and enhances diversity by varying objects. Then, we generate different questions for scoring in each evaluation stage. After manual review, we obtain 160 T2V prompts.
common sense in video generation, even for human annota- tors. To avoid this, we carefully curate prompts to ensure one-to-one correspondence for each physical phenomenon it reflects, with clear physical law inside. By incorporating physical laws from four distinct physical categories, Phy- GenBench offers a thorough assessment of current T2V models' understanding of physical commonsense.
2024). This augmentation process is carefully designed to avoid revealing the expected physical phenomenon. 4) Di- versity Enhancement: Following 12V-CompBench (Sun et al., 2024), we employ GPT-4o to perform object substi- tution on the augmented prompts. This step increases the diversity of the benchmark. 5) Question Generation: Con- sidering that understanding the physical principles implied in prompts increases the evaluation difficulty, following the method of (Singh & Zheng, 2023). For each prompt in Phy- GenBench, we leverage the world knowledge of LLMs to generate different questions for scoring for different eval- uation stage in PhyGenEval that reflect physical correct- ness, These questions are manually reviewed, filtered, and then incorporated into the benchmark. 6) Quality Control: We conduct a thorough review of the prompts, questions and their associated physical laws to ensure accuracy and relevance. Specifically, we ensure that they are clear and accurate. We then randomly use the current T2V model to check if the prompts are simple enough for the model to generate semantically accurate videos. In this process, we focus on the most fundamental physical laws, the most common scenarios, and rigorous quality standards, We find that excessive prompts are unnecessary: even the most basic
Benchmark Construction. As shown in Figure (b). we develop a comprehensive approach to create PhyGenBench. The methodology encompasses five steps: 1) Conceptual- ization: Following (Halliday et al., 2013), We first iden- tify key physical commonsense from four major categories of physics. For each category, we select specific physical laws from textbooks (Harjono et al., 2020), which can widely recognized and can be easily demonstrated through clear, observable physical phenomenon. 2) Prompt En- gineering: For each physical law, we manually craft the initial T2V prompts to clearly depict the underlying phys- ical phenomenon 3) Prompt Augmentation: To enhance the model's 'ss video generation capabilities, we augment the initial T2V prompts by adding additional details, such as more precise descriptions of objects and actions (Yang et al.,

---

PhyGenBench
physical laws and scenarios are sufficient to reveal signif- icant issues in current models and effectively distinguish between different models. For more detailed information can be referred to the Appendix B.
sessments?. Experimental results in Table demonstrate that these methods struggle to generalize the assessment of physical commonsense correctness on PhyGenBench, e.g., VideoScore (He et al., 2024b) has only a spearman correlation of 0.19 on PhyGenBench, which is most cor- related with human assessments except PhyGenEval. We attribute it to the main factor: Directly using video-based VLMs fails comprehend the embedded physical com- monsense (Jassim et al., 2023), current methods not designed with physical commonsense fully understand the physical commonsense in the video, there are three key factors need to solve: i): Physical pro- cesses typically exhibit clear key phenomena depicted by the input prompt (e.g., "the egg breaks upon hitling the rock."). It is necessary to identify these key physical phe- nomena and detect their presence in videos. ii): Physical processes are characterized by causality, manifested in the correct sequence of critical events(e.g.. "The egg touchs the rock first, then breaks.") "). The correct sequence order vali- dates the correctness of physical processes. iii): Physical processes need to possess overall naturalness, which repre- sents the realistic of the overall process. To address these factors, we design a progressive strategy that starts with key physical phenomena, then moves through the sequence of several key phenomena, and finally evaluates the overall naturalness of the entire video. In addition, we use the cus- tomized questions designed for each phase in the section 3 to conduct evaluations with different VLMs at each stage. This hierarchical and refined approach reduces the difficulty compared to existing methods that directly uses VLMs to evaluate physical commonsense, enabling PhyGenEval to achieve results closely aligned with human judgements. In this process, we ensure that GPT-4o's 's API and the infer- ence code of VLMS is configured without randomness, to guarantee the reproducibility of the results.
4. Phy Eval
PhyGenEval aims to assess whether the physical phenomena in the generated videos conform to the corresponding phys- ical laws. To obtain clear judgment, we decompose the evaluation into semantic alignment (SA) and physical com- monsense alignment (PCA). While SA evaluates whether the semantic meaning inferred by the generated video and the input prompt are matched, PCA measures whether the evaluated physical laws are grounded in the videos. For example, for the scene "an egg collides with a stone", SA requires a video containing the egg, the stone, and the colli- sion action. PCA necessitates a video for the whole physical motions in which the egg hits a stone and then breaks, while the stone remains intact. Following (He al., 2024b), we convert both SA and PCA four-point scale, as well as the human ratings.
foundation. To
4.1. Semantic Alignment Evaluation
Directly asking the Vision-Language Modcl(VLM) to align the semantic meaning between videos and imput prompts are difficult, as prompts usually are mixed with semantic entities and physical phenomena, and the intermediate out- comes are subtly implied by the videos. For example, in a prompt like "A timelapse captures the transformation of soup as the temperature rises above 100*C", possible video generation would appear like "The video shows a soup, but there is no transformation of the soup". To ad- first employ GPT-4o to extract object
dress the challenge, and action from the original text prompt, we then utilize GPT-4o to sequentially determine the presence of extracted objects in the video and verify the occurrence of specified actions. This decomposition provides more fine-grained captures and prevents the model from confusing semantic and physical correctness during evaluation. Experimental results demonstrate that our automated evaluation method aligns more closely with human judgment and outperforms previous methods (He et al., 2024b; Sun et al., 2024) in PhyGenBench (Appendix C. In this process, we ensure that GPT-4o's ss API configured without randomness, e.g., setting the temperature to guarantee the reproducibility of the results.
Key Physical Phenomena Detection. This stage aims to detect whether the key physical phenomena occur in the video. Here we define the key phenomena as an observable and distinctive occurrence (e.g., a specific frame) within a physical process that can directly reveal the corresponding physical law, like deformations color changes. During the process of constructing the benchmark, for each input prompt in PhyGenBench, we craft a retrieval prompt pr- and a set of physics-related questions Q. where the retrieval prompt is used to locate the key phenomena frame, and physical-related questions are utilized to check whether the expected physies phenomena are present in the keyframe.
4.2. Physical Commonsense Evaluation
As illustrated in Figure 3 (a), we first obtained both Q and P, by prompting GPT-4o with the input T2V prompt and corresponding physical law. Following (Hessel et al.,
To evaluate physical correctness in the video, we evaluated multiple common evaluation metrics comparing human as-
Anmotators are asked to score the correctness of physical com- monsense in the video. Details refer to Section 5 and Appendix D.
5

---

PhyGenBench
Nuturdiness
Prompti is spriftart farse Swward nigght, solll sarfinn. Hayiical Lasel farceess of stree high and hawdens
Geserated Questions
Ratrivaal Frompt ugg hiss the store
Retiinal Prompti The tostho the vions sightly
PlyGenBench
heaely
Sutement: frame)
Quartion:: The ege doat not breal, the none nemaine unchhsnged
Fnt Framen))
unchanged
daorly Uorea The ear iqputs the
untenned.
Wrong, Statementi The does breah.
Dayytt
Quertion:: the itoee remor unchanged bun the sturters.
Sighthy tem anai semeimpeattto
Reaiste en no namerou
Fromee)
Evaluation in PhyGenEval
Three- Ther
1+2
Key Framnas
1-2
Farns
I+1
1P2
Los Foune
Dueni Ouest Qued
tach hume
Each Sodti-Lat out
QuEn
Rastooo
Aand
Damskanz
Sigle Irage
Singh
Provill Sert 2in
Singh Image
Mulpe Ieages
aldio
(y1ie (PSM(), R)) vUNG,,0) Sutt
.91 (VLM0,, v.) L.)
V/M Orpec P.f)
Physical Phenomera Detection
Plysiic Order Veriiication (asing Quasttion as
Ovecall Nouuaatess Eviluction
Figure 3: An overview of the proposed PhyGenEval. PhyGenEval is divided into three parts: Key Physical Phenomena Detection, Physics Order Verification, and Overall Naturalness Evaluation. Each part uses an appropriate VLM in combination with physical-based customized questions generated by GPT-40. The final score is the combined result of the three parts. For the example in the figure, the three-stage scores are 0, 1 (only Question, is correct), and O. The final score is calculated as O according to Overall Score calculation in Section 4.2.
2021), a keyframe 14 from the video based on the retrieval prompt, where I is the i-th frame in the video. By us- ing the keyframe, we define confidence score of the key phenomena in the video:.
itself after it is broken). we approach this direction by inves- tigating the order correctness from the keyframes (Figure (b)), e.g., the keyframe of the egg hits the stone should be ahead of the keyframe of the broken egg.
Skay Σ max QEQ
(VLM((,,.)) + PT))) 2515442
Similar to the Key Physical Phenomena Detection evalua- tion, we use a retrieval prompt pr and three physical-related questions (4), 92, Qs) in PhyGenBench. p, is used to locate the keyframe (e.g., the moment the egg slightly touches the stone.). While Q1, 92, and q3 are questions check the order correctness from the first frame to the keyframe, from the keyframe to the last frame, and from the first frame to the last frame, respectively. Similarly, we first use CLIPScore to locate the key frame Li, then the order correctness scores of Spefore and Safter are defined as:
where VLM( 9) reflects the presence of physical phenom- ena in 15 for each related question q from Q. (LM(L,,Pr) Pr.) checks whether I matches the retrieval prompt, which en- sures key phenomena occur at the correct frame. Since videos may contain semantic errors, it's also important for determining if key physical phenomena occur (e.g., an egg shouldn't t break in mid-air before hitting a rock). We consider adjacent 5 frames near the keyframe to enhance the robustness. For example, the egg may not be cracked just when it first contacts the stone. We instantiate VL.M- based evaluator V-M(() with VQAScore (Lin et al., 2024), which has been shown promising evaluation results on vi- sual question-answeriig
i-2515i max (VLM 4, qi) + VLM(I;,P.)) Sbefore
Safter max (VLM(1, + VLM(I,,P.)) iS$$4+2
Is assesses the overall physical sequence coherence of the video. The score of answering qa is defined as by Sall = [LM((Io44-2:42) L-1, 93), which evaluates the overall sequence (similar the input video but using man- ually selected key frames). Here employ GPT-4o I.LVV--Interlen (Li et al., 2024) as the VI.M-based evalu- ator VLM(-), as they demonstrate exceptional multi-image comprehension capabilities. The overall score of whole physical order evaluation can be formulated as Sorder = Spefore + Safter + Sall
Physies Order Verification. In this stage, we verify whether key physical phenomena occur in the correct order. The correct physical sequence an ordered series of events in physical process that reflects causality, which repre- sents the necessary prerequisites and temporal order of key physical phenomena. As an example, the egg should first touch the stone and then crack. Considering current models in PhyGenBench generally maintain outcome consistency (Huang et al., 2024) (e.g., the egg would not reassemble
6

---

PhyGenBench
Overall Naturalness Evaluation. This stage aims eval- uate the overall naturalness of the video. we define nat- urainess the dynamic progression that aligns with real- world physical phenomenons (Liao et al., 2024). During the construction of PhyGenBench, we generate 4 different levels of descriptions 9spec for each pair of prompts and their corresponding physical laws, As shown in Figure (c), we require the VLM to score based on p. L, Jepec, and the corresponding video denoted by To:-1. Formally, we define the overall naturalness score as:
Table 1: PCA correlation results with proposed Phy- GenEval in video generation on Phy PhyGenBench. Phy- GenEval is significantly closer to human feedback on Phy- GenBench compared to other metrics.
Metric
Michanies
Optice
Thermal
Matrrial
Overall
PCFP
(41)
(411
viT
,473
+(t)
(97
DEVILL Cluso et al. 2024g
0.36
0.36
0.88
0.53
0.30
0.33
0.27
0.20
0.17
0.38
Mideolity (Bensal et al, 2004)
6.00
-0.03
-0.3
-0.14
0.08
0.08
0.33
0.34
0.03
3.01
Mitoohoooe cite at. 2073bb
6.18
0:30
0.07
0.28
6.11
0.1
611
0.15
0.7
5.19
0.78
6.76
0.77
6.73
0.78
0.81
0.84
Snatural = VV.M(To....P.. 9.pec)
indicating that PhyGenEval serves as effective human- aligned physical commonsense correctness evaluator for PhyGenBench. we provide more details in Appendix D.1.
We implement the VLM-based evaluator VLM(-) using Intern InternVideo2 (Wang et al., 2024) and GPT-40, both of which have promising results in video understanding.
We conduct several case studies to illustrate the differences between various metrics more clearly. As shown in Figure 4, (a) and (f) reveal that VideoScore and DEVIL are prone to misclassifying videos that have smooth and consistent motion but violate fundamental physical laws. Specifically, as for (a), when "an egg exhibits rubber-like elasticity upon impact with a rock instead of breaking," these metrics incor- rectly evaluate it as physically correct. VideoPhy exhibits similar limitations. In (c), it incorrectly assesses "a rock floating on water instead of sinking" " as physically correct. Furthermore, our analysis reveals major flaw in these three methodologies: they cannot incorporate domain-specine physical commonsense. As illustrated in (e), where "the flame from burning copper appears red instead of green," these metrics fail to identify the mistake. This indicates their inability to incorporate domain-specine physical com- monsense. In contrast, PhyGenEval demonstrates a robust integration of physical commonsense and comprehensive video content analysis, resulting in more accurate and phys- ically consistent evaluations in PhyGenBench.
Overall Score. We first discretize Stey, Sorder, and Snatural from the three stages into a 4-point scale, then take their average and apply floor rounding as the final score. For robust purposes, we evaluate Sorder with both GPT4o and Interl--teee and Snatural with both GPT40 and Intern- Video2. The final score is calculated as the ensemble of two methods. Detailed calculation are provided in Appendix C.
5. Experiment
Experiments Setup. We evaluate 8 open-source models, as well as 6 proprietary models Kling (kli, 2024), Pika (Pik, 2023), Sora (Sor, 2024), Vidu (Vid, 2024), Hailuo (Hai, 2024), and Gen-3 (gen, 2024). We compare our pro- posed metric with existing metrics or benchmarks: Video- phy (Bansal et al., 2024), VideoScore (He et al., 2024b) and DEVIL (Liao et al., 2024) More Detailed information is provided in Appendix D.
For human evaluation, we compared the results across 8 ran- domly selected different T2V models. We randomly select 64 prompts from PhyGenBench and generate 64 vidcos for each T2V model. We ask three annotators to provide se- mantic and physical scores for each video". Each annotator will give an integer score of 0-3 for the semantic and phys- ical scores, and the final score is the average of the three scores and rounded up. Finally, we calculate the correlation between the human scores and automatic evaluation scores using Kendall's 7 and Spearman's 's p. we put more detailed information about human evaluation in Appendix D.I.
Quantitative Evaluation. We conduct extensive experi- ments on a wide range of popular video generation models. As illustrated in Table 2, even the best-performing model, Gen-3, only attains a PCA score of 0.51 on PhyGenBench. This indicates that even for prompts containing obvious physical commonsense, current T2V models struggle to generate videos that comply with intuitive physics. It indi- rectly reflects that these models are still far from achieving the world simulator.
Furthermore, we identify the following key observations: 1): Across various categories of physical commonsense, all models consistently demonstrate superior performance in the domain of optics compared to other areas. Notably, Vchi- tect2.0 and Cog VideoX-5b achieve a PCA score in the optics domain comparable to that of closed-soune -oosed-souree models. We posit that this superior performance in the optics domain can be attributed to the abundant and explicit representation of optical knowledge in pre- pre-training datasets, thereby enhane- ing the model's comprehension in this area. 2): Sora and
Human Evaluation. As shown in Table 1, current video generation evaluation metrics largely overlook physical cor- rectness. In contrast, PhyGenEval implements a detailed design for evaluating physical correctness, demonstrating strong correlations with human judgments across all cat- egories. Its overall correlation coefficient reaches 0.81,
2 'Nte that we ask the annotators focus the correctness of the physical phenomena for physical scores.
7

---

PhyGenBench
(6)
(b)
deparit 2014 mack surfect
Antramnn the
Proficilee
Wicetty
Vilofor
DEV3
Prypterbill
Wisotty
Viecodiom
SENVI
Fronge
Team Promia
Hove nettrr
che
upetud
Huarhe
Figure 4:
Different video generation evaluation metric in PhyGenBench. Except for the proposed PhyGenEval, the current methods cannot reasonably assess the correctness of physical commonsense in videos from PhyGenBench.
Table 2:
Evaluation results of PCA with the proposed PhyGenEval in videos generated by several models. The results reveal that all models score very low in physical commonsense accuracy. The scores are normalized to a range of 0-1.
Model
Size
Mechanicx(t)
Opticx(1)
Thermall
Material(t)
Averuge(
CogVideoX (Yang et al,, 2024)
2B
0.38
0.43
0.34
0.39
0.39
CogVideoX (Yang et al., 2024)
5B
0.39
0.55
0.40
0.42
D.45
Open-Sora (Zheng
2024a)
0.37
Lavic (Wang al., 2023)
860M
0.30
0.44
0.38
0.32
0.36
Vehitect 2.0 (Wang ct ol., 2023)
2B
0.41
0.56
0.41
0.37
0.15
Hunyuan (Kong al. 2024)
13B
0.44
0.39
Pyramid Flow (lux) (Jin al., 2024)
2B
0.37
0.50
0.47
0.37
D.43
Pyramid How (sd3) (Jin et 2024)
2B
0.42
0.49
0.36
0.45
0.41
Pikal 0 (Pik, 2023)
0.35
0.56
0.43
0.39
D.44
Gen-3 (gen, 2024)
0.45
Kling (Kli, 2024)
0.45
0.58
0.30
0.40
0.49
Sora (Sor, 2024)
0.50
0.66
0.56
0.16
0.55
(Vid, 2024)
0.48
0.50
Hailuo (Hai, 2024)
0.49
0.67
0.50
0.50
0.55
Hailuo exhibit significantly higher performance compared to other models. Specifically, they demonstrate a robust understanding of material properties, achieving a score of 0.55, which substantially surpasses other models. Kling performs particularly well in thermal, attaining the highest score of 0.50 in this domain. 3): Among open-source mod- els, Hunyuan and CogVideox: 5b perform comparatively well, both exceeding the performance level of Pika. In con- trast, Lavie consistently exhibits lower physical correctness. We provide qualitative analysis in Appendix D.3
Venhancer (He et al., 2024a) aimed to improve general video quality (Huang et al., 2024). More detailed results are in Appendix E.
Rewriting prompt. We aim to explore whether GPT- augmented prompts can address the PhyGenBench chal- lenges. Specifically, we rewrite the original prompts using GPT, adding expected physical outcomes and processes. For example, after "A bottle of juice is slowly poured out in the space station, releasing the liquid into the surrounding area", we add "The liquid forms floating globules, spread- ing out and moving randomly through the air." in the end.
Ablation Study. We conduct a detailed robustness analy- sis, Experimental results demonstrate that the key designs of PhyGenEval are essential, and even when only using open-source models, Phy GvaEval maintains high effeec tiveness. Detailed results provided in Appendix D.4.
As shown in Table 3, we use Cog VideoX and Kling as representative models for open-source and closed-soure, source systems, respectively, to conduct tests. The results indi- cate that prompt rewriting does help the models generate images aligned with physical laws, but it is still far from resolving the issues highlighted by PhyGenBench. Both Cog VideoX 5b and Kling exhibit some growth, but even for Kling, it only achieves a score of 0.56. This demon- strates that current models still severely lack the ability to
6. Diseussion
To explore potential solutions for the challenges posed by PhyGenBench, We focus on widely used and proven- effective methods such as scaling laws (Kaplan et al., 2020), prompt engineering (Fu al., 2024), and some method like
8

---

PhyGenBench
Table 3: Evaluation results of PCA using the proposed PhyGenEval after rewriting prompts The results indicate that although using rewritten prompts leads to some improvement, it is still insufficient to address the challenges highlighted by PhyGenBench. The scores are normalized to a range of 0-1.
Model
Size
Mechanics(1)
Optics(t)
Thermal(t)
Material(
Average(t)
Before Rewriting Prompt
CogVideoX (Yang et al., 2024) Kling
5B
0.39
0.55
0.40
0.42
0.45
0.45
0.58
0.50
0.40
0.49
After Rewriting Prompt
CogVideoX (Yang et al., 2024) Kling
5B
0.39
0.62
0.53
0.52
0.52
0.50
0.64
0.61
0.48
0.56
Table 4: PCA evaluation results with proposed PhyGenEval in videos after VEnhancer. The results indicate that employing VEnhancer fails enhance the model's comprehension of physical commonsense. The scores normalized to range of 0-1.
Model
Size
Mechanics(t)
Optics(t)
Thermal(t)
Material(t)
Average(t)
Vchitect 2.0
2B
0.41
0.56
0.44
0.37
0.45
Vchitect 2.0 (Venhancer)
2B
0.41
0.56
0.42
0.38
0.45
accurately render physical scenes, and this deficiency can- not be easily resolved through simple prompt rewriting. To
benchmarks like VBench that evaluate video quality.
7. Conclusion
illustrate this issue more clearly, as shown in Figure 10, our qualitative analysis shows that rewriting prompts can
In this paper, we explore the gap between current T2V mod- els' understanding of physical commonsense and their role as world simulators. To achieve this, we introduce PhyGen- Bench and PhyGenEval. PhyGenBench a benchmark specifically designed to assess models' understanding of physical commonsense, featuring various physical laws and simple, clear physical phenomenons. Alongside PhyGen- Bench, we propose novel evaluation framework called Phy GenEall to automate the evaluation process. Experi- mental and analytical results show that current T2V models struggle to generate videos that align with physical common- sense, highlighting significant gap from world simulation.
only address simple issues (e.g.. flame color reactions), but remains ineffective for more complex physical processes (e.g., egg breaking, stone sinking).
The robustness of PhyGenBench and PhyGenEval. VEnhancer (He al., 2024a) is generative space-time enhancement framework that improves existing videos by adding spatial details and synthetic motion in the temporal domain. After enhancement by VEnhancer, Vchitect2.0 shows significant improvement on VBench, even surpass- ing Kling. However, VEnhancer only enhances the visual quality of videos (e.g., making them more coherent and clear) without addressing the model's poor understanding of physical commonsense.
Impact Statement
This work contributes to the long-term vision of advancing video generation models toward more physics-consistent world simulators. From an ethical perspective, while strict physical consistency evaluation may reveal current model limitations, it also opens new directions for improving mod- els' physical understanding. data synthesis, and evaluation frameworks. The future societal impact includes enhancing the reliability of video generation technology in scientific simulation, education, and filmmaking.
As shown in Table 4, Vchitect enhanced by VEnhancer still scores similarly to the original version on PhyGenBench. We calculate high Spearman coefficient 0.86 between model scores on PhyGenBench before and after VEnhancer enhancement. This indicates that PhyGenEval primarily fo- cuses on physical correctness and is robust to other factors affecting visual quality. Furthermore, it demonstrates that even if a model can generate videos with better general qual- ity (e.g., ranking higher on VBench), it doesn't necessarily imply a better understanding of physical common sense. This highlights the distinction between PhyGenBench and

---

PhyGenBench
Acknowledgements
Gao, S., Yang, J., Chen, L., Chitta, K., Qiu, Y., Geiger, A., Zhang, J., and Li, H. Vista: A generalizable driving world model with high fidelity and versatile controllabil- ity. arXiv preprint 1433,98, 2024.
This paper is partially supported by the National Key R&D Program of China No.20222D0161000
Halliday, D., Resnick, R., and Walker, J. Fundamentals of physics. John Wiley & Sons, 2013.
References
Pika, 2023. URL htttps:://ww.ppiaaaaaart
Harjono, A., Gunawan, G., Adawiyah, R., and Herayanti, L. An interactive e-book for physics to improve students' conceptual mastery. International Journal of Emerging Technologies in Learning (iJET), 15(5):40-49 2020.
Hailuo, 2024. URL httptttss///haluualll video/
Sora, 2024, URL https://sora.coo
Hailuo, 2024. URL https://www.vidde
He, J., Xue, T., Liu, D., Lin, X., Gao, P., Lin, D., Qiao, Y., Ouyang, W., and Liu, Z. Venhancer: Generative space- time enhancement for video generation. arXiv preprint arXiv:2407.07660 2024a.
Gen-3, 2024. URL https://ruunaamll.ommbbL... introducing-gen Beaiinngggn---aaaae gen-3-alpha/.
Kling, 2024. URL htttps://kking.kuat www.ingg.uuiihuuuccoco
He, X., Jiang, D., Zhang, G., Ku, M., Soni, A., Siu, S., Chen, H., Chandra, A., Jiang, Z.., Aruiraj, A., al. Mantiss- core: Building automatic metrics to simulate fine-grained human feedback for video generation. arXiv preprint 0in:2240..12252, 2024b.
Achiam, J., Adler, S., Agarwal, S., Ahmad, L., Akkaya, I., Aleman, F. L., Almeida, D., Altenschmidt, J., Altman, S., Anadkat, S., al. Gpt-4 technical report. arXiv preprint arXiv:2303.08874, 2023.
Hessel, J., Holtzman, A., Forbes, M., Bras, R. L.,, and Choi, Y. Clipscore: A reference-free evaluation metric for im- age captioning. arXiv preprint arXiv:2104.08718, 2021.
Bansal, H., Lin, Z., Xie, T., Zong, Z., Yarom, M., Bitton, Y., Jiang, C., Sun, Y., Chang, K.-W., and Grover, A. Videophy: Evaluating physical commonsense for video generation. arXiv preprint arXiv:2406.03522 2024.
Huang, Z., He, Y., Yu, J., Zhang, F., Si, C., Jiang, Y., Zhang, Y., Wu, T., Jin, Q., Champaisit, N., et al. Vbench: Compre- hensive benchmark suite for video generative models. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 21807-21818, 2024.
Battaglia, P. W. Hamrick, J. B., and Tenenbaum, J. B. Sim- ulation as an engine of physical scene understanding. Proceedings of the National Academy of Sciences, 110 (45):18327-18332, 2013.
Jassim, S., Holubar, M., Richter, A., WolfT, C., Ohmer, X., and Bruni, E. Grasp: A novel benchmark for eval- uating language grounding and situated physics under- standing in multimodal language models. arXiv preprint arXiv:2311.00 1.09048, 2023.
Bear, D. M., Wang. E., Mrowca, D., Binder, J., Tung, H.-Y. E., Pramod, R., Holdaway, C., Tao, S., Smith, K., Sun, F.-Y., et al. Physion: Evaluating physical prediction from vision in humans and machines. arXiv preprint (mXiv:2106.08261, 2021.
Jin, Y., Sun, Z., Li, N., Xu, K., Jiang, H., Zhuang, N., Huang, Q., Song, Y., Mu, Y., and Lin, Z. Pyramidal flow matching for efficient video generative modeling. arXiv preprint arXiv:2410.05954. 2024.
Brooks, T., Hellsten, J., Aittala, M., Wang, T.-C., Aila, T., Lehtinen, J., Liu, M.-Y., Efros, A., and Karras, T. Generating long videos of dynamic scenes. Advances in Neural Information Processing Systems, 35:31769- 31781, 2022.
Kaplan, J., McCandlish, S., Henighan, T., Brown, B., Chess, B., Child, R., Gray, S., Radford, A., Wu, J., and Amodei, D. Scaling laws for neural language models. arXiv preprint arXiv:2001.0836/ 2020.
Chow*, W., Mao J., Li, B., Seita, D., Guizilini, V. C., and Wang, Y. Physbench: Benchmarking and enhancing vision-language models for physical world understanding. In International Conference on Learning Represemations, 2025.
Kay, W., Carreira, J., Simonyan, K., Zhang, B., Hillier,
C., Vijayanarasimhan, S., Viola, F., Green, T., Back, T., Natsev, P., et al. The kinetics human action video dataset. arXiv preprint 1705.0:700.0699 2017.
Fu. X., He, M., Lu, Y., Wang, W. Y., and Roth, D. Commonsense- 12i challenge: Can text-to-iman genera- tion models understand commonsense? arXiv preprint mrXiv:2406.07544 2024.
Kong, W., Tian, Q., Zhang, Z., Min, R., Dai, Z., Zhou, J., Xiong. J., Li, X., Wu, B., Zhang, J., et al. Hunyuan- video: A systematic framework for large video generative models. arXiv preprint 22x:::4422.00000 2024.
10

---

PhyGenBench
Li, F,, Zhang, R., Zhang, H., Zhang, Y., Li, B., Li, W., Ma, Z.., and Li, C. llava-next-mnerleeae:: Tackling multi- image, video, and 3d in large multimodal models. arXiv preprint arXiv:2407.07895. 2024.
Soomro, K. Ucf101: A dataset of 101 human actions classes from videos in the wild. arXiv preprint arXiv:/212.0400 2012.
Sun, K., Huang, K., Liu, X., Wu, Y., Xu, Z., Li, Z., and Liu, X. T2v-compbench: A comprchensive benchmark for compositional text-to-vider generation. arXiv preprint AxXiv:2407.15503 2024.
Li, Z., Wang. X., Stengel- Eskin, E., Kortylewski, A., Ma, W., Van Durme, B., and Yuille, A. Super-clevr: A vir- tual benchmark to diagnose domain robustness in visual reasoning. In Proceedings of the IEEE/CVF Conference Computer Vision and Pattern Recognition (CVPR), pp. 14963-14973, June 2023.
Swartz, N. The concept of physical law. Cambridge Univer- sity Press, 1985.
Liao, M., Lu, H., Zhang, X., Wan, F., Wang, T., Zhao, Y., Zuo, W., Ye, Q. and Wang, J. Evaluation text-to- video generation models: A dynamies perspective. arXiv preprint ar Ar0iv:2407.00000 240,0010944 2024.
Tan, Z., Yang, X., Qin, L., and Li, H. Vidgen-Im: A large- scale dataset for text-to-video generation. arXiv preprini arXiv:2408.02629 2024.
Unterthiner, T., Van Steenkiste, S., Kurach, K., Marinier, R., Michalski, M., and Gelly, S. Towards accurate generative models of video: A new metric & challenges. arXiv preprint 1rXiv:/8/2.01/
Lin, Z., Pathak, D., Li, B., Li, J., Xia, X., Neubig, G., Zhang, P., and Ramanan, D. Evaluating text-to-visual generation with image-to-text generation. arXiv preprint arXiv:2404.00 02.4..1299 0191,,11111000 2024.
2018.
Wang, J., Ma, A., Cao, K., Zheng, J., Zhang, Z., Feng, J., Liu, S., Ma. Y., Cheng, B., Leng, D., al. Wisa: World simulator assistant for physics-aware text-to-video generation. arXiv preprint arXiv:2503.081533 2025.
Liu, H., Li, C.,, Wu, Q., and Lee, Y.J. Visual instruction tun- ing. Advances in neural information processing systems, 36, 2024a.
Liu, S., Ren, Z., Gupta. S., and Wang, S. Physgen: Rigid- image-to-vided generation. In body European Conference on Computer Vision ECCV, 2024b.
Wang, Y., Chen, X., Ma. X., Zhou, S., Huang, Z.,, Wang, Y., Yang, C., He, Y., Yu, J., Yang, P., et al. Lavie: High- quality video generation with cascaded latent diffusion models. arXiv preprint arXiv:2309. 15103, 2023.
Liu, Y., Cun, X., Liu, X., Wang, X., Zhang. Y., Chen, H., Liu, Y., Zeng, T., Chan, R., and Shan, Y. Evalcrafter: Benchmarking and evaluating large video generation models. In Proceedings of the IEEE/CVF Conference on Computer Vision and Puttern Recognition, pp. 22139- 22149, 2024c.
Wang. Y., Li, K., Li, X., Yu, J., He, Y., Chen, G., Pei, B., Zheng, R., Xu, J., Wang. Z., et al. Internvideo2: Scaling video foundation models for multimodal video understanding. arXiv preprint arXiv:2403.15377, 2024.
Wood, J. N., Ullman, T. D., Wood, B. W., Spelke, E. S., and Wood, S. M. Object permanence in newborn chicks is robust against opposing evidence. arXiv preprint arXiv:2402.1 146002221444.... 2024.
Mazzaglia, P. Verbelen, T., Dhoedt, B., Courville, A., and Rajeswar, S. Multimodal foundation world mod- els for generalist embodied agents. arXiv preprint arXiv:2406. 18043, 2024.
Reid, M., Savinov, N.. Teplyashin. D., Lepikhin, D., Lilli- crap, T., Alayrac, J.-b., Soricut, R., Lazaridou, A., Firat, O., Schrittwieserr, J., et al. Gemini 1.5: Unlocking multi- modal understanding across millions of tokens of context. arXiv preprint arXiv:2403.055380 2024.
Xiang, J., Liu, G., Gu, Y., Gao, Q. Ning, Y., Zha, Y., Feng, Z.. Tao, T., Hao, S., Shi, Y., al. Pandora: Towards general world model with natural language actions and video states. arXiv preprint MxXv:2446.00945. 2024.
Yang, Z., Teng, J., Zheng, W., Ding, M., Huang, S., Xu, J., Yang. Y., Hong, W., Zhang, X., Feng, G., et al. Cogvideox: Text-to-video diffusion models with an ex- pert transformer. arXiv preprint arXiv:2408.06072 2024.
Salimans, T., Goodfellow, I., Zaremba, W., Cheung, V., Radford, A., and Chen, X. Improved techniques for training gans. Advances in neural information processing systems, 29, 2016.
Yuan, S., Huang, J., Xu, Y., Liu, Y., Zhang. S., Shi, Y., Zhu, R., Cheng, X., Luo, J., and Yuan, L. Chronomagic- A benchmark for metamorphic evaluation of text-to-time- lapse video generation. arXiv preprint arXiv:2406. 18522, 2024.
Singh, J. and Zheng, L. Divide, evaluate, and refine: Eval- uating and improving text-to-image alignment with it- erative vqa feedback. Advances in Neural Information Processing Systems, 36:70799-70811 2023.

---

PhyGenBench
Zheng, Z., Peng, X., Yang, T., Shen, C., Li, S., Liu, H. Zhou, Y., Li, T., and You, Y. Open-sora: Democratizing efficient video production for all, March 2024a. URL https://ge thub. cww http:/hccatecccOpe--SSSSSe Sora.
Zheng, Z., Yan, X., Chen, Z., Wang. J., Lim, Q Z. E., Tenen- baum, J. B., and Gan, C. Contphy: Continuum physi- cal concept learning and reasoning from videos. arXiv preprint ar arXiv:2402.066 19, 2024b.
Zhu, Z., Wang, X., Zhao, W., Min, C., Deng, N., Dou, M., Wang, Y., Shi, B., Wang, K., Zhang, C., et al. Is sora world simulator? comprchensive survey on general world models and beyond. arXiv preprint arXiv:2405.03520, 2024.
12

---

PhyGenBench
A. Related work
A.1. Benchmarks For Physical Understanding
Recent benchmarks such as SuperCLEVR-Phyee (Li et al., 2023), ContPhy (Zheng et al., 2024b), and Physion (Bear et al., 2021) have significantly advanced the evaluation of Al's physical reasoning, focusing on understanding and predictive tasks, SuperCLEVR-Phyee R-Phsiies emphasizes reasoning about dynamic properties like velocity and collisions in 4D scenes, ContPhy expands the evaluation scope include diverse physical properties, such underscores the limitations of existing AI models in handling soft-body dynamics. And Physion evaluates models' ability to predict physical phenomena like collisions and motion while benchmarking against human behavior. However, these works primarily target understanding or prediction rather than generative capabilities. In contrast, our work introduces PhyGenBench, a comprehensive benchmark designed to evaluate whether Text-to- Text-to-Video (T2V) models can generate videos that adhere to physical commonsense. Unlike existing benchmarks, our work highlights the generative challenges in intuitive physics, revealing critical gaps in current T2V models and underscoring the need to advance physical commonsense for applications beyond entertainment.
and density, within continuum settings.
A.2. Evaluation metries for text-to-video generation
Conventional approaches to video quality assessment often employ metries such as FVD (Unterthiner et al., 2018) and IS (Salimans al., 2016). However, the detection of unrealistic motions is difficult for them (Brooks al., 2022), and FVD requires reference video that is hard to obtain for novel scenes, making it challenging evaluate the correctness of physical commonsense. Recent studies have explored the use of advanced vision- langunla models (VLMs) as evaluators. For instance, VideoScore (He et al., 2024b) leverages human feedback to train models for video quality assessment, while R2V-ChmpBench (Sun et al., 2024) utilizes powerful models like LLaVA (Liu et al., 2024a) to evaluate the correctness of spatial relationships. Although few works demonstrate improved alignment with human judgments, they fall short in generalizing assessments of physical commonsense correctness. To address this limitation, introduce PhyGenEval, novel method designed to evaluate physical commonsense correctness PhyGenBench. We validate the efficacy of our approach through comprehensive human correlation studies.
B. PhyGenBench Details
B.1. Detailed Overview
fine-grained analysis of the dataset essential for a comprehensive understanding the benchmark. As shown Table 5, PhyGenBench covers major domains in physics, encompassing 27 representative physical laws, which enables it to provide fine-grained evaluation of models' physical capabilities. We evaluate 14 advanced models including Sora, Vidu, and etc. Additionally, our captions encompass totally 165 unique objects and 42 unique actions with an average length of 18.75 75 words.
Table 5: Details of PhyGenBench
comprehensive and
Statistic
Number
Physical Laws
27
Domains
Optics
Mechanics
40
Thermal
30
B.2. Construction
Material Properties
40
we provide additional details about the Questions Generation and Quality Control processes described in Section 3.
Total Captions
160
Total T2V Models
Total Generated Videos
2240
Questions Generation. Considering that a single prompt cannot fully reflect expected physical laws-such placing wooden block on water, where buoyancy should cause to float rather than sink-dires evaluation of videos by VLMs LisguneLanguage Models) using prompts alone is challenging. Research indicates that VLMs struggle to directly comprehend physical laws inherent in videos (Chow* et al., 2025). Therefore, we leverage the extensive world knowledge of LLMs (Large Language Models) to parse prompts and generate physics-infol informed questions for different stages of evaluation (as detailed PhyGenEval).. This approach enables
Unique Objects
165
Unique Actions
42
Average Length of Caption
18.75
13

---

PhyGenBench
VLMs to accurately determine the physical correctness of videos. We have provided the complete prompt and code in our anonymous link. Specifically, for different stage in PhyGenEval, we generate different questions for scoring. In Key Physical Phenomena Detection: At this stage, we detect key frames that contain the expected physical phenomena. For each input prompt in PhyGenBench, we design retrieval prompt p, and a set of physics-related questions Q. The retrieval prompt is used to locate the key frame, while the questions are used to verify whether the expected physical phenomena are present in the identified frame. For example, when parsing the prompt "an egg colliding with a stone," we use GPT-4o with the prompt: "Propose 1-2 questions with answers based on the given prompt and physical laws. Include a retrieval prompt for identifying the key frame, and ensure the question focuses on single image.". In Physics Order Verification: This stage examines whether the sequence of physical events is correct, e-g., whether the egg hits the stone before breaking. We evaluate this by checking the order correctness between key frames (Figure 3 (b)). For example, the frame showing the egg hitting the stone should precede the frame showing the broken egg. Similarly, we prompt GPT-44 to generate a retrieval prompt p, and three physics-related questions (q1, 92, 93). The retrieval prompt identifies the key frame, while q1. 42. and d3 verify the order correctness from the first frame to the key frame, from the key frame to the last frame, and from the first frame to the last frame, respectively. The GPT-4o prompt we use is: "Provide the retrieval prompt and two ideal descriptions based on the selected frame pairs (firs-retrievall and last-retrieval) as well as a complete description of the process." In Overall Naturalness Evaluation: This stage evaluates the overall physical correctness of the dynamic process. For each prompt-physical law pair, we require GPT-4o to generate four levels of descriptions reflecting increasing correctness: Completely Fantastical, Clearly Unrealistic, Slightly Unrealistic, and Almost Realistic. The GPT-44 prompt we use is: "Evaluation Standards: Completely Fantastical: Entirely detached from reality, featuring elements of fantasy or surrealism. Clearly Unrealistic: Contains significant and sustained violations of physical laws, such as implausibly large objects or scenes. Slightly Unrealistic: Features minor or brief distortions, such as unnatural facial expressions or textures that are difficult to notice. Almost Realistic:
No noticeable distortions; fully consistent with reality."
Quality Control. This process is primarily conducted through manual checks. Specifically, we perform detailed evaluations during the benchmark construction process, including verifying the correctness of prompts and their associated physical laws, assessing the semantic simplicity of prompts, and ensuring the accuracy of generated questions. Five senior undergraduate students are recruited, with each question assigned to all five annotators for evaluation.
For each prompt, annotators first verify the correctness of the prompt and its corresponding physical law, cross-checking the information using online resources and tools. Next, for semantic simplicity, we refine the benchmark by removing overly complex prompts that current models cannot reasonably handle. Annotators also assess whether the T2V-generated videos are semantically reasonable to ensure they support effective evaluation. Finally, for the questions generated in Section 3 used for scoring, annotators assess the physical correctness of each GPT-generraed question. For the Overall Naturalness Evaluation stage, the criteria require that each level of description shows progressive improvements in correctness and distinguishability..
Through this process, we optimize the dataset quality based on annotator feedback, producing high-quality dataset that includes prompts, their corresponding physical laws, and the questions used for scoring during evaluation.
B.3. Difference between Videophy and Ours
VIDEOPHY (Bansal al., 2024) comprises 688 curated simple prompts that focus on interactions between three types of physical materials: solid-solid, solid-fluid, and fluid-fluid, but lack annotations of physical laws. The dataset is designed to evaluate a model's understanding of physical commonsense, featuring limited range of physical phenomenons such as rigid body interactions, fluid dynamics, and contact forces. We are better suited than Videophy for evaluating physical commonsense due to two significant differences.
First As shown in Figure 2, PhyCenBench includes 160 carefully crafted prompts across 27 distinct physical laws, spanning four fundamental domains, which comprehensively assess a model's understanding of physical commonsense. While Videophy primarily focuses on interactions between solid-fluid, solid-solid, and fluid-fluid, limiting its coverage and overlooking common physical laws such as phase transitions and basic material properties. What' more, Videophy lacks annotations of physical laws making it hard for VL.M model to evaluate. Second, as shown in Table 6, the average SA score of PhyGenBench (0.80) significantly outperforms that of Videophy (0.63). This indicates that PhyGenBench prompts are well-suited and easy for T2V models to generate high-quality, well-aligned videos, which benefits evaluation of physical correctness. In contrast, as shown in Figure 5, We find that prompts from Videophy pose challenges for T2V models in generating text-aligned and high-quality videos for two main reasons: 1. The prompts lack detail and specificity. For
14

---

PhyGenBench
instance, "A tissue blots a tear from an eye" is overly simplistic (without augmentation). Modern T2V models, such as Cog Video5B (Yang et al., 2024), typically trained with longer and more descriptive captions, which enhance their ability to comprehend and generate content based on prompts. 2. The scenes often complex and unrealistic. For example, "The wristwatch knob winds the inner spring tightl"" describes process involving intricate internal mechanisms that are not visible externally. As a result, is exceedingly difficult for T2V models to generate such scenes accurately.
Kling
Vchitect
CogvideoBb
Text Prompt
Nissue sooking up heardrop,
Prompt
spring tightly.
Text Prompt
tissue blots tear from aye.
Figure 5: Samples of videos generated by Kling, Vchitect, and CogvideoSh in Videophy. All T2V models struggle to achieve proper text alignment and produce high-quality videos, making meaningless evaluate physical correctness in Videophy.
Table 6: Comparison of SA results for video generation between Videophy and PhyGenBench. We randomly select 64 prompts from both Videophy and PhyGenBench, use different T2V models to generate videos, and then ask annotators to score based on our cretiera in Figure 11. The results show that PhyGenBench SA scores significantly outperform Videophy.
Model
Size
Videophy(1)
PhyGenBench (t)
(Yang et al., 2024)
5B
0.48
0.78
Vchitect 2.0
2B
0.63
0.81
Kling
0.77
0.89
Average
0.63
0.80
C. PhyGenEval Details
C.1. Semantic alignment details
To reduce the complexity for VLM models to evaluate sementic correctness of generated videos between prompts, we adopt a two-stage strategy. Initially, we employ GPT-4o to extract objects and actions from the original text prompt. Subsequently, we employ GPT-4o to determine whether the extracted objects are present in the video and verify the occurrence of specified actions. For each video, GPT-4o first assesses the presence of the objects mentioned in the prompt (e.g.. "egg" ") within the video frames. This evaluation is performed according to Question (QI), where GPT-4o assigns a score from 2 based on the completeness of object presence: score of is given if all the objects present, if some of the objects missing, and 0 if none of the objects appear in the video. After determining object presence, GPT-4o moves on to Question 2 (Q2) to check if the specified action (c.g., "pour out") out"") is performed in the video. It assigns a binary score (0 or 1) depending on whether the action is present (1) or absent (0). Finally, these scores are combined to form the overall semantic alignment score. we put more details about other metric baselines in Appendix D.1. In this process, we ensure that GPT-4o's API configured without randomness, e.g,, setting the temperature to 0, guarantee the reproducibility of the results.
15

---

PhyGenBench
C.2. Physical Commonsense alignment details
In this section, we use the same notation in Section 4.2 and provide more detailed description of the calculation and design of the method. In this process, we ensure that GPT-4o's API and the inference code of VLMs is configured without randomness, e.g., setting the temperature to 0, to guarantee the reproducibility of the results.
Key Physical Phenomena Detection. We categorize the T2V prompts into monotonic processes (eg. "melting with increasing temperature") ") and non- non- mee-mooootnicc processes (eg. "an egg hitting a rock") based on the physical phenomena they represent. For prompt with monotonic processes, We only consider using the "Last Frame" as the retrieval prompt, resulting in single question. We can directly calculate VLM(Img,, Q). where the score for the corresponding video of this prompt ranges from 0 to 1. For prompt with non-monotonic processes, we consider both the intermediate key frames and the Last Frame, resulting in two questions. For the intermediate key frames, we calculate (LM Q) - VLM((Ing, Pr), which ranges from 0-2. Consequently, the score range for videos corresponding to this prompt is 0 to 3.
For specific calculatation, we need to calculate VLM Pr) and VLM(I,, q). where Img, is the j-th frame in the video. For VLM pr),, the calculation involves assessing the matching degree between the key frame and the retrieval prompt, which can be directly obtained using the original calculation method in (Lin et al., 2024). For VL.M(T;, q), we follow the computation approach from ChronoMagieBench (Yuan al., 2024), we derive VLM((1;,9) q) by determining the ratio of the VQAScore for the affirmative statement to the combined VQAScores for both the affirmative and negative statements. We perform the calculations of VLM Pr)) and VLM(I,, q) for each key frame within the specified range to obtain the physical correctness score for the problem.
Physics Order Verification. For this stage, which we've primarily introduced in Section 4, we focus key caleu- lation points. The score calculation formula for 41 is Spefore max;-25js (VLM(I,, ,, 91) + VLM(Iy, P.)). Here, VL.((T,, determines if the retrieved key frame satisfies the retrieval prompt,as the physical phenomenon should occur in the keyframe primarily located in Key Phenomena Detection, which is crucial for Key Sequence Verification (e.g. the expected physical phenomenon of egg cracking should occur in the keyframe when the egg hits the stone, rather than other frames when the egg is in the air or else). VT, (1) assesses the correctness of the Key Sequence order in the video. Notably, we calculate VLM([;,,p.) Pr) using VQAScore, yielding a decimal between 0 and 1, while VLM Iy, 91) employs VLM (GPT-4V or LLa VAattennterle for question-answering, scoring or 0 based on the model's Yes No response.
Overall Naturalness Evaluation. Here we mainly explain how to get the score of this part based on the evaluation results under the two-stage strategy described in Section 4. Specifically, we ask the idsod VLM to select the most appropriate option for the video according to the detailed scoring criteria generated by the LLM,, and then we map the options to scores (Completely Fantastical to Almost Realistic corresponds to 0-3 points)
Overall Score. We detail the discretization and calculation process of the scores here. In the stage of key phenomena detection, we categorize the prompts into monotonic and non-monotonic processes based on the physical phenomena they represent. For monotonic processes, the score range is 0-1, which we directly discretize by averaging into integer values from 0-3. Specifically, for monotonic processes with a score range of 0-3, we discretize the scores to [1, 1.5, 2.25]. This is because no points should be awarded if the physical phenomena are incorrect (VLM((T,,P.) even with accurate retrieval. (e.g., The egg hits the stone and does not break)
and (L. Q) 0),
In the stage of key sequence verification, we have three multi- image problems. One point is awarded for each correct answer, resulting in a final integer score from 0-3. Similar to the stage, of key phenomena detection we need to consider both the accuracy of key frame retrieval and the physical question answering. Therefore, we design the following: for Q1, when max;-2sjsi (VLM I, q1) + VLM(I,, p.)) and PL) > 0.5, the question is considered correct. The process for Q2 is similar. For q3, it marked correct when VLM(I,, 4-2-+42 L31.
In the stage of overall naturalness evaluation, we require video-based direct option selection, choosing Completely Fantastical, Clearly Unrealistic, Slightly Unrealistic, and Almost Realistic is scored as 0, 1, 2, and 3 points respectively. Finally, we average all scores and round down to obtain the final score.
For the ensamble operation, in order to reduce the bias caused by using single VLM at a certain stage, we ensemble the results of PvyGenEval using open source models or closed source models. Specifically, we average the two results and round them down.
16

---

PhyGenBench
D. Experiment
D.I. Experiments Setup
T2V model Implementation details. Open-Sora 1.2 (Zheng et al., 2024a) is an open-source project with the goal of reproducing Sora. CogVideo), VideoX 2b (Yang al., 2024) and CogVideoX 5b are large-scale diffusion transformer models for text-to-video generation, incorporating 3D Variational Autoencoder (VAE) for efficient video compression and an expert transformer with Expert Adaptive LayerNorm to improve text-video alignment. LaVie (Wang et al., 2023) is cascaded video latent diffusion model. Vchitect2.0 (Wang et al., 2023), developed by the Shanghai AI Lab, is an advanced video generation model featuring Parallel Transformer architecture to scale up video diffusion models and empower video creation. Hunyuan (Kong et al., 2024) is the largest open-source T2V model. Pyramid Flow a T2V model with autoregressive architecture. Sora (Sor, 2024), Pikal.0 (Pik, 2023), Gen-3 (gen, 2024), Kling (kli, 2024), Vidu (Vid, 2024), and Hailuo (Hai, 2024) are the most popular closed-source models.
Evaluation Metrics details. We compare our proposed PhyGenEval with some evaluation metrics from previous methods like VideoPhy (Bansal al., 2024) and VideoScore (He et al., 2024b). VideoPhy fine-tunes a VLM with the VIDEOPHY dataset proposed by themselves, which includes human feed back about the semantic alignment and dynamic motion correctness about videos. VideoScore is trained on the VIDEOFEEDBACK dataset proposed by themselves, Initialized from the Mantis model. VideoScore provides automatic assessments of video quality based on human scoring criteria. To compare with PhyGenEval on SA and PCA, We only choose the text alignment and fact consistency criteria. Specifically, for the semantic alignment evaluation, we compare the Grid-LLaVA method proposed by T2V-CompBench, which extends the LLaVA (Liu et al., 2024a) model to handle multi-frame inputs by sampling 6 frames uniformly from a video to create an image grid. For the physical commonsense alignment evaluation, we also compare with DEVIL (Liao et al., 2024), which uses Gemini 1.5 Pro (Reid et al., 2024) to assess the overall naturalness of videos and applies the same scoring standard prompt to all videos.
Furthermore, evaluate the effectiveness of our PhyGenEval designs, conduct a large amount of ablation studies and pue more details in Appendix D.4.
Human evaluation details. Here, we provide detailed explanation of the human evaluation described in Section 5. Specifically, we require annotators to score based on the standards outlined in Figure 1, covering both semantic alignment and physical commonsense alignment. For example, as for the video shown in Figure 11, The egg bounces off the rock like a rubber ball, completely violating physical laws like dynamics, the annotator gives a score of 0 for physical commonsense alignment. However, since the video fully includes the egg, the rock, and the collision action, the annotator gives a score of 3 for semantic alignment.
D.2. Quantitative Evaluation
Comparison result about semantic alignment. Here we design a new baseline PhyGenEval (Grid-LLaVA) to illustrate the superiority of the method, which uses the two-stage strategy proposed PhyGenEval from Appendix C.I, but replaces the VLM with Grid-LLaVA proposed in 12V-CompBench (Sun al., 2024). As shown in Table 8, PhyGenEval achieves the highest correlation scores across all categories, demonstrating its effectiveness as human--ligned semantic commonsense correctness evaluator for PhyGenBench. Compared to other methods, PhyGenEval consistently outperforms previous baselines like VideoPhy, VideoScore, and Grid-LLaVA. Specifically, PhyGenEval obtains an overall Kendall's 7 of 0.53 and Spearman's 's p of 0.56, surpassing the Grid-LLaVA (r: 0.35, 0.39). The results clearly show that our PhyGenEval design provides a more accurate and reliable semantic commonsense evaluation in PhyGenBench.
Quantitative result about semantic alignment. As shown in Table 9, nearly all models achieve relatively high SA scores, whether evaluated by machines or humans. This suggests that the scenarios in PhyGenBench are relatively straightforward. making it easier to assess physical commonsense. Among all the models, Kling achieved the highest SA score, with a human evaluation score of 0.89, reflecting its strong instruction understanding and video generation capabilities.
D.3. Qualitative Evaluation.
The different video cases for 4 physical commonsense categories are illustrated in Figure 6. Our main observations are as follows: mechanics, the models struggle to generate simple physically accurate phenomenons. As shown in Figure 6,
17

---

PhyGenBench
Table 7: Details about evaluation models. The table shows duration, FPS, and resolution for each model.
Model
Duration (s)
FPS
Resolution
Open-Sora 1.2 (Zheng et al., 2024a)
4
24
1280 x 720
C 2b
8
720 x 480
V 5b
8
640 x 360
Lavie
512 x 320
Vchitect2.0
8
768 432
Hunyuan
5
24
1280 x 720
Pyramid Flow(flux)
24
8
640 x 384
Pyramid Flow(sd3)
24
8
640 x 384
Pika (Pik, 2023)
24
1280 x 720
Gen-3 (gen, 2024)
11
24
1280 x 768
Kling (kli, 2024)
30
1280 x 720
Vidu (Vid, 2024)
5
30
1280 720
Hailuo (Hai, 2024)
5
30
1280 X 720
Sora (Sor, 2024)
30
1280 x 720
Table 8: SA correlation results with proposed PhyGenEval in video generation. A higher score indicates better performance for category. Bold stands for the best score,
Mechanics
Optics
Thermal
Material
Overall
Metric
r(t)
p(t)
r(t)
p(t)
r(t)
p(t)
r(t)
p(t)
r(t)
p(t)
VideoPhy (Bansal et al., 2024)
0.20
0.25
0.03
0.03
0.20
0.24
0.18
0.22
0.13
0.17
VideoScore (He et al., 2024b)
0.14
0.16
0.13
0.14
0.23
0.02
0.02
0.02
0.05
0.05
Grid-LLaVA (Sun et al., 2024)
0.39
0.43
0.45
0.49
0.30
0.33
0.22
0.26
0.35
0.39
PhyGenEval (Grid-LLaVA)
0.35
0.38
0.46
0.48
0.41
0.44
0.42
0.45
0.42
0.44
PhyGenEval
0.48
0.52
0.64
0.67
0.46
0.49
0.47
0.50
0.53
0.56
all models fail depict the glass ball sinking in water. As for (b), instead showing floating the surface, OpenSora and Gen-3 even produce videos where the ball is suspended. Additionally, the models do not capture special physical phenomenonss, such the state of water in zero gravity, as seen in (a). In optics, the models perform relatively better. (c) and (d) show the models handling reflections of balloons in water and colorful bubbles, though OpenSora and CogVideoX VideoX still produce reflections with noticeable distortions in (d). In thermal, the models fail to generate accurate videos of phase transitions. the melting phenomenon in (e), most models show incorrect results, with Cog VideoX even producing a video where the ice cream increases in size. Similar emrors appear in the sublimation process in (f), with only Gen-3 showing partial understanding. Regarding material properties, (g) shows all models failing recognize that egg should break when hitting rock, with Kling displaying the egg bouncing like rubber ball. For simple chemical reactions, such the black bread experiment in (h), none of the models demonstrate accurate understanding of the expected reaction.
D.4. Ablation study
The Component in PhyGenEval on physical commonsense alignment evaluation. We conduct series of ablation studies to demonstrate the necessity of our method design by examining its correlation with human evaluation results, similar to those described in Section 5. Specifically, we compare: 1) ) The effect of the various stages of PhyGenEval, as proposed in Section 4.2; 2) Performance differences when using various VLMs and their ensembles PhyGenEval, as outlined in Section 4.2. 3) The larger open models in PhyGenEval. Notice that PhyGenEval for physical commnonsense alignment evaluation consists of three stages: key phenomena Detection, key sequence verification, and overall naturalness evaluation.
18

---

PhyGenBench
Table 9: SA evaluation results with proposed Pvy in video generation. Both machine and human evaluations indicate that most models achieve good semantic scores on PhyGenBench. This suggests that the scenarios in PhyGenBench are simple enough to clearly reflect physical phenomena. The scores are normalized range of 0-1.
Model
Size
Mechanics())
Optics(t)
Material(t) (hermal(1)
Average(
CogVideoX (Yang et al., 2024)
2B
0.63
0.67
0.61
0.63
0.64
CogVideoX (Yang et al., 2024)
5B
0.78
0.88
0.78
0.64
0.78
Open-Sora V1.2 (Zheng al., 2024a)
1.IB
0.73
0.85
0.82
0.73
0.79
Lavie (Wang al., 2023)
860M
0.47
0.63
0.73
0.53
0.58
Vchitect 2.0 (Wang al., 2023)
2B
0.92
0.89
0.77
0.74
0.84
Hunyuan (Kong et al., 2024)
13B
0.75
0.83
0.81
0.75
0.79
Pyramid Flow (flux) (Jin et al., 2024)
2B
0.71
0.83
0.75
0.63
0.68
Pyramid Flow (sd3) (Jin al., 2024)
2B
0.71
0.82
0.83
0.56
0.75
Pika (Pik, 2023)
0.63
0.81
0.73
0.69
0.72
Gen-3 (gen, 2024)
0.84
0.93
0.82
0.78
0.85
Kling (kli, 2024)
0.88
0.91
0.87
0.74
0.85
Sora (Sor, 2024)
0.85
0.83
0.86
0.77
0.84
Vidu (Vid, 2024)
0.88
1.00
0.92
0.88
0.92
Hailuo (Hai, 2024)
0.94
0.98
0.86
0.79
0.87
Open-Sere Sero V2.2
CogViddex
Kling
Gen-3
oi of aboely cor reluviin ligat
Pranpe
clea
seed surfice
Bad
ldl
Text Prompe
senese and mramilae bio
large number s0pp dln under mnc.suil sutiigt..
Prunge
Hissgge ciptores the de cree The
agaficant zoone 200 ic.
Ifi Text Prange
Incusfommcionn at
laxiid with die foor firce carfoon ahote calided
upen
Pronge
meetee cocoettered peaned picce brood.
Figure Qualitative comparisons of four categories. Current models perform relatively well in generating optical phenomenons but are weaker in mechanies, thermal, and material properties.
And We denote them as PhyGenEval-S, PhyGenEval-M, and PhyGenEval-V based on the VLM they used.
1) PhyGenEval for physical commnonsense alignment evaluation consists of three stages. We investigate the contribution of each stage to the final performance. Table 11 presents results using one or two stages (employing ensemble strategies
19

---

PhyGenBench
Table 10: Retrieval success rates for some different models.
Model
Force (f)
Light (1)
Heat (1)
Material (f)
Overall (f)
Cogvideo 5B
0.7618
0.9013
0.8046
0.7815
0.8193
Gen3
0.8353
0.9077
0.8627
0.8114
0.8577
Pika
0.7829
0.8777
0.7825
0.7736
0,8107
Lavie
0.7064
0.8328
0.7537
0.7219
0.7596
Vchitect2
0.8078
0.9034
0.8317
0.7668
0.8327
Keling
0.8375
0.9018
0.8319
0.7978
0,8470
Opensora
0,8166
0.8755
0.8528
0.7707
0,8310
Cogvideo 2B
0.7924
0.8255
0.7886
0.7719
0.7971
when multiple VLMs are applicable). We find that optimal performance is achieved only when all three stages are used concurrently, demonstrating the rationale behind PhyGenEval's design.
2) Given potential biases in single models and the costs associated with closed-source models, we offer two PhyGenEval (Li et al., 2024) and Intern Video2
computation methods: using GPT-4o or alternative open-source models (Wang et al., 2024)). Table 12 shows that even using only small scale open-source models achieves a high correlation coefficient of 0.66. Notably, ensembling both methods yields the best results. Considering PhyGenBench 's relatively small size, we find this computational cost acceptable. Therefore we recommend users ensemble these methods.
3) We explore the performance of larger open-source models. Specifically, we replace GPT-4o used in the Physics Order Verification stage of PhyGenEval with Intem (78B), denoted as PhyGenEval (Open-L). Additionally, when we use smaller open-source model like llav-nexx-interleave 7B, denote it as PhyGenEval (Open-S). We denote the original PhyGenEval with closed-source model like GPT-4o PhyGenEval (Closed)
Results show that compared to smaller open-source models, the overall aligmment coefficient with larger open-source models improves from 0.66 to 0.72, indicating that the method remains reproducible even when using exclusively open-source models. We believe that as open-source models continue to advance, they can achieve even better performance within PhyGenEval on PhyGenBench.
The robustness of retrieval operations PhyGenEval In PhyGenEval's Key Physical Phenomena Detection and Physics Order Verification, retrieval operations are required to detect key physical phenomena and their order videos. Since retrieval may not always capture the target frames, incorporate regularization term VLM( (T,, (,Pr into the calculations of Skey and Sorder- VQAScore is used to evaluate the scores of retrieved frames and retrieval prompts, improving the robustness of retrieval operations. (Note: when the retrieval prompt is "Last Frame" or "Middle Frame," the default value is set to 1.)
Additionally, since PhyGenBench intentionally designed with simple and common scenarios, retrieval operations achieve a relatively high success rate. We report the average (M, P,) scores different models Table 10, showing that even Lavie achieves a retrieval score above 0.75, indicating generally high retrieval accuracy.
The Component in GenEval on semantic alignment evaluation. to validate the necessity of our SA evaluation design. Specifically, we compare: 1) VLM Model Selection: We leverage GPT-4o (Achiam et al., 2023) as more robust VLM model for SA evaluation. 2) Effectiveness of our two-stage evaluation method proposed in Appendix C.1
also perform necessary ablation experiments
1) shown Table using GPT-4o PhyGenEval much better than using LLaVA, which achieve a higher Kendall's of 0.53 compared to 0.42, and a higher Spearman's 0.56 versus 0.44. This indicates stronger alignment between GPT-4o's evaluations and human annotations compared to open-source vlm models like Grid-LLaVA (Sun al., 2024), justifying its selection the preferred VLM model in the SA evaluation design. Since PhyGenBench includes limited number of prompts, we believe that the cost of using GPT-4o is acceptable relative to the improvement in performance.
2) To validate the effectiveness of the two-stage strategy, we compare it with the method in BenCChh (Sun et al., 2024), which directly uses Grid-LLaVA to apply the same scoring standard prompt for semantic aligmment evaluation across all videos. For fairness, we also use Grid-LLaVA but implement the two-stage strategy proposed in Appendix C. As shown
20

---

PhyGenBench
Table 11: Comparison of PCA correlation results using each stage in PhyGenEval
Mechanics
Optics
Thermal
Material
Overall
Metric
T(t)
p(t)
T(t)
p(t)
T(t)
p(t)
T(1)
p(t)
T(t)
p(t)
PhyGenEval-S
0.50
0.54
0.43
0.45
0.50
0.54
0.72
0.77
0.56
0.61
PhyGenEval-M
0.46
0.49
0.49
0.53
0.55
0.59
0.53
0.57
0.55
0.60
PhyGenEval-V
0.26
0.30
0.44
0.47
0.33
0.35
0.48
0.52
0.42
0.46
PhyGenEval-SM
0.58
0.61
0.47
0.50
0.58
0.62
0.66
0.70
0.60
0.64
PhyGenEval-SV
0.56
0.59
0.41
0.43
0.58
0.60
0.70
0.74
0.59
0.62
PhyGenEval-MV
0.50
0.53
0.50
0.53
0.53
0.57
0.60
0.64
0.57
0.61
PhyGenEval 0.72
0.75
0.76
0.77
0.73
0.75
0.81
0.84
0.78
0.81
Table 12: Comparison of PCA correlation results using different models such GPT-4o open-sourced models in
PhyGenEval
Mechanics
Optics
Thermal
Material
Overall
Metric
r(t)
p(t)
T(t)
p(t)
T(t)
p(t)
T(t)
p(t)
T(t)
p(t)
PhyGenEval (Open-S)
0.54
0.57
0.59
0.62
0.55
0.58
0.65
0.69
0.62
0.66
PhyGenEval (Open-L)
0.58
0.62
0.63
0.65
0.62
0.64
0.70
0.73
0.67
0.72
PhyGen Ghyaall (Closed)
0.72
0.75
0.76
0.77
0.73
0.75
0.81
0.84
0.78
0.81
in Table 8, PhyGenEEvl-Grid-LLLae outperforms Grid-LLaVA, achieving a higher Kendall's score of 0.42 compared to 0.35, and a higher Spearman's p score of 0.44 versus 0.39. This result demonstrates the effectiveness of our Two Stage Evaluation Method. By decomposing the evaluation into object detection and action detection, we effectively reduces the complexity of the task for VLMs in evaluating the sementic correctness of videos.
64]
Frompt
(b)
Text Frompt
pouned
The quece ctctior relaatt surrearding
corsanticced pouned
Phytiai
Plyrical
craiity
Hemon
Vidaate
VideScorr
DEVIL
Humee
Flysesrval
Videuhyy
Vitesdorre
DRVIL
der Mecirne leye ne ons ande
Freept
Warkate oxptaree
the
surfice winkbok encecaing the sotton servesh The netezack aurfact
blick pa to
and iin gf plo liteet
Phyrtcal Low
Hhyrical
colr trous parale
Hatesn
Videoity
Videodarn
Humaue
PlyGrrba
Vidulfty
ViceaSorre
DEVIL
Figure 7: Visualization of some PhyGenEval error cases.
Use real videos as reference We agreed that using real videos as references might have provided more reference information. However, considering several difficulties:
1. It is challenging to collect real videos for each prompt.
2. Physical processes are diverse, making difficult collect unique real videos.
21

---

PhyGenBench
3. Due differences in frame rates and other factors, video-to-video comparison is also challenging and might require training separate models.
Here, we adopted an alternative approach to incorporate real videos into the PhyGenEval framework.
Specifically, we extracted fifty Video-Caption pairs from WISA (Wang et al., 2025), belonging to mechanics, optics, and thermodynamics categories (WISA did not include physical property categories). We parsed the corresponding video captions into prompt and question formats as in PhyGenBench, and used PhysGenEval for evaluation. The results showed that real videos achieved extremely high scores under PhyGenEval, demonstrating the robustness of the framework.
Mechanics (17 samples)
Optics (17) Thermal (16)
Score
0.93
0.95
0.93
We also tested the performance of open-source models these 50 prompts, using machine scores and machine scores / machine scores of real videos (the latter serving reference scores after error climination), obtaining the following table:
Mechanics
Optics
Thermal
Avg
CogVideo
0.36(0.37)
0.44(0.49)
0.33(0.41)
0.39(..39)
Cog V5g
0.36(0.36)
0.52(0.57)
0,47(0.53)
0.46(0.50)
Opensora V1.2
0.36(0.38)
0.49(0.52)
0.39(0.39)
0.43(0.45)
Lavie
0.23(0.27)
0.42(0.45)
0.36(0.40)
0.36(0.38)
Vchitect 2.0
0.41(0.43)
0.52(0.57)
0.42(0.44)
0.47(0.50)
Hunyuan
0.46(0.49)
0.53(0.55)
0.39(0.40)
0.48(0.51)
Pyramid Flow (Flux)
0.33(0.37)
0.50(0.54)
0.44(0.50)
0.44(0.48)
Pyramid Flow (Sd3)
0.43(0.54)
0.46(0.52)
0.33(0.40)
0.42(0.49)
The Spearman coefficient of model rankings calculated by these two methods is 0.90, indicating that the current evaluation framework can achieve robust results.
E. Discussion
Error case analysis. As shown in Figure 7, we visualize some error cases where both PhyGenEval and competing methods like DEVIL fail correctly identify the physical realism of the videos. These error cases are often caused by confusing but iconic physical phenomena in the videos that do not align with the correct progression of physical processes (e.g., in the erroncous of the "burnt bread" experiment, black coloration appears but does not align with the expected phenomenon), leading to misjudgments. However, even in these cases, PhyGenEval remains closer to human ratings compared to other methods.
The Impact of Scaling on Physical Commonsense in Video Generation. Scaling laws have been extensively validated in video generation models (Kaplan et al., 2020). We investigate their efficacy in addressing the challenges of physical commonsense presented PhyGenBench. As shown in Table 2, Cog Video 5B demonstrates improvements over Cog Video 2B, albeit with limited progress in the Mechanics category. Our qualitative analysis, illustrated in Figure 8, reveals significant advancements in static scenes with CogVideo SB. It accurately captures complex phenomena such as colorful bubbles resulting from interference and diffraction, and CogVic Video 5B generates more realistic boiling simulations compared to its predecessor. However, both models struggle with simple motion dynamics, exemplified by their inability to accurately depict bouncing football. We posit that while scaling up enhances the model's capacity to generate videos that align with physical commonsense for individual objects. it may be insufficient for physical phenomenons involving dynamic physical laws. Addressing these challenges likely requires extensive training on carefully curated synthetic data, as suggested by (Liu et al., 2024b). This approach could potentially bridge the gap in the model's grasp of fundamental physical laws.
rusting of iron. In thermal, despite imperfections,
The robustness of low-quality videos. Although we deliberately select simple scenarios when constructing PhyGenBench to make it easier for models to generate videos that align with prompts while exposing physical errors, certain T2V models
22

---

PhyGenBench
Text Prompt
op of water a sleyy pauee the liui ine the aurronding
AI
Mechanics
Text Prompt
Hhron foccriilly fowords the intraction with serface open impect
Mechanics
Scale
Text Prompt
undir the ouiigh
Optics
Scale
Text Promp?
Incesfommtion owg in raddyy risu abore 100
Thermal
Scale
Text Prompt
traecfemen murice of clean ceverneet owe decodes.
Material Property
up
Figure 8: The qualitative comparison of VideoX 2B and Cog VideoX 5B. The result shows that simply scaling up can solve issues, but dynamic physical phenomenons involving the design of motion patterns remain challenging.
may still perform poorly specific prompts. In this regard, PhyGenEval demonstrates degree of robustness. We select 100 low-quality videos and calculated their average PCA Score, which is 0.14, significantly lower than the average PCA Score in Table 2. Figure 9 provides visualizations of some low-quality videos, most of which fail to meet the prompt requirements, resulting in very low physical scores.
The effect of fine-tuningect of fine-tuning. WISA fine-tunes CogVideo using real text-video pairs and tests it PhyGenBench. The results show that the score increases from 0.41 to 0.43. Although there is some improvement, it is not significant. Therefore, it is still necessary to design fine-tuning methods that explicitly incorporate physical laws, or to use more powerful video generation model backbones.
(o)
geatly
the gestly Filled with eater.
aurfocs bucker Fillet water.
bathibb
Phystol Law
Phystcal Law
PhySenbeal
Buayanty
(a)
Text Proept
(a)
Pronpt
wbroet, elsttic hemia sel is threen forcefully
A. ney of light dhin
digganilyy i seatts oup the plastic
insraction with the serface upon iepoct.
Physical
HrySexEvvl
PhyGenEEel
icity
Light traveli in straight İtc.
Figure 9: Visualization of some low quality videos and their PCA score.
23

---

PhyGenBench
Table 13: Resource consumption of models used in PhyGenEval.
Model
Batch Size
Resources
Times
Memory Utilization Per GPU
GPT-4o((tagee2)
USD
5min
GPT-4o(stage3)
USD 3.1
5min
LLaVA--Nettmmterllave--1
1 x A A100-80KG
2min
20408Mi8
VQAScore
1 x A100-80GB
10min
72726MiB
Video
1 x A100-80GB
1min
7766MiB
F. Computaional Resources
Although our method involves different stages, remains straightforward. Table 13 provides summary of the resource consumption, showing that the entire evaluation process be completed quickly and at low cost.
24

---

PhyGenBench
Expected Phenomenon
green fiom reoction
Text Prompt
Rewritten Prompt
piece copper igrited, enitting unique steadlly
A piecs of gnited, emifting vind and flams burns steadlly The casper heats undergoing combustion, which results the emision light and preduces o green color flone
Expected Phenomenon
while the stone intact becouss stone harder thon ogg
Text Prompt
Rewritten Prompt
fragle agnificent rugeed
delicete, frogile, hurled with signficnnt force ruggn, soldd rock surfoce, where colidde upon impact. Upon callstio,, the shatters, breaking spart. Fragments of the egg shell scatter and the inner contenta spill out
Rewrite
Expected Phenomenon
stone sink to the
Text Prompt
Rearitten
A stone a gently plased the aurface of pool filled with water.
gantly placed trans-tionng water descends.
paol filled soth water. stone
Rewrite
Expected Phenomenon
applx shrisk. sadic
Text
Rearitten
A timelasse captures the resetion es concentrated sulfuric acid poured arte sice of apple
asple. shrisk, corbenize, black, acidic
Rewrite
Figure 10: The qualitative comparison of effects before and after using rewritten prompts. The results indicate that rewriting prompts addresses only a few basic issues (such as flame color reactions), while the majority of problems remain unsolved.
25

---

PhyGenBench
Text Prompt
A delicate, fragile egg hurled with significant force towards a rugged, solid rock surface, where collides upon impact.
Physical Law
Stone is harder than egg, the egg should crack
Semantic Evaluation Criterion
0: None of the objects involved in the interaction are present.
1: Some of the objects involved in the interaction are missing.
2: All the objects involved in the interaction are present, but the interaction is not presented.
3: All the objects involved the interaction are present, and the interaction is presented.
Score
Semantic Fully Violation
Semantic Fully Adherence 3
Physical Evaluation Criterion
O:
Entirely Unrealistic: Complete Departure from Physical Laws The video presents situations entirely disconnected from physical common sense, with clearly impossible phenomena occurring throughout.
Significantly Unrealistic: Significant Violation of Physical Laws The video shows large-scale or prolonged deviations from general physical laws, such as scenarios that contradict basic principles of energy, or matter behavior.
1:
Slightly Unrealistic: Minor Inconsistencies with Physical Laws The video includes small or brief moments where physical laws are slightly distorted, but the overall scene mostly adheres to common physical processes.
2:
3: Almost Realistic: Near-Perfect Adherence to Physical Laws The video exhibits no obvious violations of physical laws, closely matching real- world physical processes in every aspect.
Score
Entirely Unrealistic
Almost Realistic
1
2
3
Figure 11: Detailed diagram the human evaluation process. We ask the annotators to score the semantic alignment and physical commonsense alignment of the video according to the scoring criteria in the figure.
26


Reviews:
[{"review_id": "JEvkbhHnLo", "reviewer": "Reviewer_QPju", "summary": "This paper introduces PhyGenBench, a benchmark assessing physical commonsense correctness in Text-to-Video (T2V) models, and PhyGenEval, an automated evaluation framework. PhyGenBench includes 160 prompts covering 27 physical laws across mechanics, optics, thermal, and material properties, ensuring a comprehensive assessment. PhyGenEval evaluates key physical phenomena, causal order, and overall naturalness using Vision-Language Models (VLMs) and Large Language Models (LLMs). The study shows that scaling models and prompt engineering alone are insufficient, emphasizing the need for better physics-aware video generation. PhyGenBench and PhyGenEval provide a scalable and structured evaluation framework, encouraging advancements in realistic world simulation.\n\n\n## update after rebuttal\nThis is an important and timely contribution to the community. The authors propose a benchmark and evaluation methods for video generation. I am inclined to recommend acceptance of the work.", "questions": "", "limitations": "", "rating": 4, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for your valuable insights, which are fundamental to strengthening our paper. We handle all questions in their given order and will incorporate those details in the revision.\n\nQ1: more ablation studies on different training techniques.\n\nA1: We randomly sampled 1200 Video-Prompt pairs from WISA and perform lora(r=128) fine-tuning on CogVideoX 5B. The results shown below indicated that it does not solve the problem of model understanding of physical laws. This may be due to:\n\n- The training set is too small, not covering enough domains.\n- The base model's capabilities are insufficient, making it difficult to generalize.\n- More explicit injection of physical laws is needed, such as using synthetic videos.\n\n| Model | Mechanics | Optics | Thermal | Material | Avg |\n| --- | --- | --- | --- | --- | --- |\n| CogVideoX 5B | 0.39 | 0.55 | 0.40 | 0.42 | 0.449 |\n| CogVideoX 5B （FT） | 0.38 | 0.58 | 0.41 | 0.40 | 0.453 |\n\nQ2: its applicability to unseen, real-world scenarios remains unclear.\n\nA2: We first discussed the effectiveness of PhyGenEval on newly added prompts. Specifically, we extracted 50 prompts from WISA and applied 8 open-source models to generate videos. For each model, we extracted 20 videos and perform human evaluation. We calculated the human alignment coefficient(spearman) with machine scores in the table below, PhyGenEval has a certain generalization ability beyond PhyGenBench prompts.\n\n| Mechanics | Optics | Thermal | Avg |\n| --- | --- | --- | --- |\n| 0.71 | 0.76 | 0.76 | 0.74 |\n\nNext, we acknowledged that using real videos as references would have provided more reference signals, but this also presented several challenges: collecting real videos for each prompt was difficult, physical processes were diverse making definitive examples hard to find, and comparing videos with different frame rates required additional model training. Therefore, we adopted an alternative approach to incorporate real physical videos into PhyGenEval. Specifically, we extracted 50 Video-Caption pairs from WISA, belonging to mechanics, optics, and thermodynamics categories. We parsed the corresponding video captions into the prompt and question format used in PhyGenBench and evaluated them using PhysGenEval. The results showed that real videos achieved extremely high physical alignment scores under PhyGenEval, demonstrating the robustness of the framework.\n\n| Mechanics(17 samples) | Optics(17) | Thermal(16) |\n| --- | --- | --- |\n| 0.93 | 0.95 | 0.93 |\n\nWe tested the performance of 8 models on these prompts, using machine scores and machine scores / real video machine scores, with the latter serving as a reference score to eliminate errors. The results were as follows. The spearman coefficient of model rankings calculated by these two methods is **0.90**, indicating that PhyGenEval can achieve robust results.\n\n|  | Mechanics | Optics | Thermal | Avg. |\n| --- | --- | --- | --- | --- |\n| CogVideoX2B | 0.36\\0.37 | 0.44\\0.49 | 0.33\\0.41 | 0.39\\0.39 |\n| CogVideoX5B | 0.36\\0.36 | 0.52\\0.57 | 0.47\\0.53 | 0.46\\0.50 |\n| OpensoraV1.2 | 0.36\\0.38 | 0.49\\0.52 | 0.39\\0.39 | 0.43\\0.45 |\n| Lavie | 0.23\\0.27 | 0.42\\0.45 | 0.36\\0.40 | 0.36\\0.38 |\n| Vchitect2.0 | 0.41\\0.43 | 0.52\\0.57 | 0.42\\0.44 | 0.47\\0.50 |\n| Hunyuan | 0.46\\0.49 | 0.53\\0.55 | 0.39\\0.40 | 0.48\\0.51 |\n| Pyramid Flow(flux) | 0.33\\0.37 | 0.50\\0.54 | 0.44\\0.50 | 0.44\\0.48 |\n| Pyramid Flow(sd3) | 0.43\\0.54 | 0.46\\0.52 | 0.33\\0.40 | 0.42\\0.49 |\n\nQ3: The evaluation method depends on CLIPScore, its accuracy is not well-verified.\n\nA3: First, we considered the possibility of inaccurate retrieval when designing our method. For instance, the calculation of $S_{key}$ at line 300, which includes $VLM(I_j,P_r)$, was specifically added as a regularization term to account for retrieval inaccuracies (as explained in line 304). Additionally, we pointed out that since we deliberately control the simplicity of scenes in PhyGenBench and require T2V models to semantically match the prompts, the success rate of CLIPScore retrieval aws relatively high. We provided an analysis of this in Appendix D.4 under \"The robustness of retrieval operations.\"\n\nQ4: the framework does not explicitly assess motion coherence\n\nA4: We considered motion coherence to be a general video quality, but here we mainly focused on the correctness of physical laws. We tested the Temporal Quality - Motion Smoothness metric from VBench.  However, As shown in the table below, we found that the correlation coefficient between motion smoothness scores and human ratings of physical correctness is only 0.013, which made it challenging to use the metric as a distinguishing factor for supporting physics-based evaluation.\n\n| Min | Max | Avg | Spearman | Kendall |\n| --- | --- | --- | --- | --- |\n| 0.971 | 0.995 | 0.985 | 0.013 | 0.017 |\n\nWe're truly grateful for your suggestions, and please let us know if any concerns arise.", "claims_and_evidence": "Most of claims are supported by author's evidence.\nThe author conducted experiments to support the core findings: 1. current T2V models struggle with physical commonsense is well-supported by experimental results from 14 models, with Gen-3 achieving only 0.51 in physical commonsense accuracy. 2. The hierarchical evaluation strategy in PhyGenEval such as Key Physical Phenomena Detection, Physics Order Verification, and Overall Naturalness is clearly explained and validated through correlation with human ratings. This provides convincing evidence that it is a more effective metric than existing alternatives like VideoScore and VideoPhy. \nHowever, the claim that scaling models and prompt engineering alone cannot resolve physical commonsense issues is partially supported, more ablation studies on different training techniques (e.g., incorporating physics-based priors) could provide deeper insights.", "methods_and_evaluation_criteria": "The proposed method and criteria make sense to me. The benchmark PhyGenBench overs 160 prompts across 27 physical laws, ensuring a diverse and structured evaluation of mechanics, optics, thermal, and material properties. It contains three-tiered evaluation: key physical phenomena detection, physics order verification, and overall naturalness, breaks down physical correctness into measurable components. Besides, the performance aligns with human evaluations.\nWhile physics order verification evaluates causality, there is no explicit check for temporal smoothness in video sequences (e.g., abrupt frame transitions violating motion continuity). Additionally, it primarily relies on key frame detection using CLIPScore. However, the accuracy of CLIPScore in reliably identifying key frames and detecting physical phenomena remains unclear, warranting further validation.", "theoretical_claims": "There is no theoretical claims.", "experimental_designs_or_analyses": "The experiment designs and analyses are sufficent.", "relation_to_broader_scientific_literature": "The paper builds upon prior research in text-to-video (T2V) generation, physical commonsense reasoning, and evaluation benchmarks for generative models, while addressing critical gaps in these areas. Previous benchmarks for text-to-video (T2V) models (e.g., VBench, EvalCrafter) primarily evaluate motion smoothness, spatial consistency, and overall video quality, but they do not assess physical correctness, a gap that PhyGenBench aims to fill. While prior works like Physion and ContPhy focus on physical reasoning in vision-language models for prediction tasks, they do not evaluate generative capabilities, whereas PhyGenBench assesses whether T2V models can generate physically plausible videos rather than just recognizing or predicting physical events. Additionally, existing VLM-based evaluation methods (e.g., VideoScore, VideoPhy) struggle to detect violations of physical laws, but PhyGenEval introduces a hierarchical evaluation framework that explicitly verifies key physical phenomena, causal order, and overall naturalness, making it more aligned with real-world physics principles. By introducing PhyGenBench and PhyGenEval, the paper advances the scientific understanding of physics-aware video generation, providing a scalable and automated method for evaluating physical commonsense in generative models.", "essential_references_not_discussed": "One relevant technical report that assesses the physical understanding of video generation models and could be discussed is \"How Far is Video Generation from World Model? – A Physical Law Perspective.\" This work examines the extent to which T2V models adhere to fundamental physical laws, providing additional context for evaluating physical commonsense in generative models.", "other_strengths_and_weaknesses": "Overall, I think this is an important work for the community. My comments about the strengths and weaknesses can be seen as follows:\nStrengths:\n1. The paper addresses a crucial gap in text-to-video (T2V) generation by introducing a benchmark that evaluates physical commonsense, an aspect largely overlooked in existing works. The combination of a structured benchmark (PhyGenBench) and a hierarchical evaluation framework (PhyGenEval) provides a novel and scalable approach to assessing physical correctness in generative models.\n2. The evaluation includes 14 T2V models, comparing their physical commonsense accuracy (PCA) scores, and correlates automated assessments with human evaluations.\n3.By emphasizing intuitive physics in generative AI, the work could influence future developments in video synthesis, robotics simulation, and AI-driven scientific visualization, expanding the application of T2V models beyond entertainment.\n\nWeaknesses:\n1. While PhyGenBench covers a diverse range of physical laws, its applicability to unseen, real-world scenarios remains unclear. Further validation on dynamically generated prompts or real-world physics-based tasks would improve its robustness.\n2. The evaluation method depends on CLIPScore to locate key frames, but its accuracy in reliably detecting physical phenomena is not well-validated, which may introduce errors in assessment.\n3. While physics order verification ensures correct event sequences, the framework does not explicitly assess motion coherence, which is critical for realistic video generation. Integrating temporal smoothness metrics could strengthen the evaluation.", "other_comments_or_suggestions": "My comments can be found in the above 'Other Strengths And Weaknesses' section.", "questions_for_authors": "My questions can be found in the above 'Other Strengths And Weaknesses' section.", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "gXxwMsHdNl", "reviewer": "Reviewer_bpjp", "summary": "The paper introduces PhyGenBench, a benchmark specifically designed to assess text-to-video (T2V) models on their ability to generate physically plausible videos grounded in intuitive physics. It comprises 160 carefully constructed prompts covering 27 distinct physical laws across four fundamental domains: mechanics, optics, thermal phenomena, and material properties. Additionally, the authors propose PhyGenEval, a novel hierarchical evaluation framework leveraging vision-language models (VLMs) and GPT-4o to measure semantic alignment and physical commonsense alignment in generated videos. Their extensive evaluation reveals that existing T2V models significantly underperform in accurately generating physically correct phenomena, indicating substantial room for improvement toward genuine world simulators.", "questions": "", "limitations": "", "rating": 4, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We're grateful for your suggestions, which plays a critical role in elevating our paper's quality. We tackled all questions one by one and will incorporate those details in the revision.\n\nQ1: Can the PhyGenEval framework be easily adapted to new physical phenomena\n\nA1: We sampled 50 prompts from WISA and applied 8 open-source models to generate videos. For each model, we sampled 20 videos and conduct human evaluation. We then calculated the human alignment coefficient with the model scores, resulting in the table below. As can be seen, PhyGenEval demonstrates certain generalization capabilities beyond the prompts in PhyGenBench.\n\n| Mechanics | Optics | Thermal | Avg. |\n| --------- | ------ | ------- | ---- |\n| 0.71      | 0.76   | 0.76    | 0.74 |\n\nQ2: Have you explored incorporating simulation-generated videos as references in your benchmark\n\nA2: We agreed that using real videos as references might have provided more reference information. However, considering several difficulties:  1. It is challenging to collect real videos for each prompt 2. Physical processes are diverse, making it difficult to collect unique real videos 3. Due to differences in frame rates and other factors, video-to-video comparison is also challenging and might require training separate models. Here, we adopted an alternative approach to incorporate real videos into the PhyGenEval framework.\n\nSpecifically, we extracted fifty Video-Caption pairs from WISA, belonging to mechanics, optics, and thermodynamics categories (WISA did not include physical property categories). We parsed the corresponding video captions into prompt and question formats as in PhyGenBench, and used PhysGenEval for evaluation. The results showed that real videos achieved extremely high scores under PhyGenEval, demonstrating the robustness of the framework.\n\n| Mechanics(17 samples) | Optics(17) | Thermal(16) |\n| --------------------- | ---------- | ----------- |\n| 0.93                  | 0.95       | 0.93        |\n\nWe also tested the performance of open-source models on these 50 prompts, using machine scores and machine scores / machine scores of real videos (the latter serving as reference scores after error elimination), obtaining the following table:\n\n|                      | Mechanics  | Optics     | Thermal    | Avg        |\n| -------------------- | ---------- | ---------- | ---------- | ---------- |\n| CogVideoX2B          | 0.36(0.37) | 0.44(0.49) | 0.33(0.41) | 0.39(0.39) |\n| CogVideoX5B          | 0.36(0.36) | 0.52(0.57) | 0.47(0.53) | 0.46(0.50) |\n| Opensora V1.2        | 0.36(0.38) | 0.49(0.52) | 0.39(0.39) | 0.43(0.45) |\n| Lavie                | 0.23(0.27) | 0.42(0.45) | 0.36(0.40) | 0.36(0.38) |\n| Vchitect 2.0         | 0.41(0.43) | 0.52(0.57) | 0.42(0.44) | 0.47(0.50) |\n| Hunyuan              | 0.46(0.49) | 0.53(0.55) | 0.39(0.40) | 0.48(0.51) |\n| Pyramid Flow（Flux） | 0.33(0.37) | 0.50(0.54) | 0.44(0.50) | 0.44(0.48) |\n| Pyramid Flow（Sd3）  | 0.43(0.54) | 0.46(0.52) | 0.33(0.40) | 0.42(0.49) |\n\nThe Spearman coefficient of model rankings calculated by these two methods is **0.90**, indicating that the current evaluation framework can achieve robust results.\n\nQ3: Would it be feasible to automatically generate prompts using generative models or RL\n\nA3: 1. We currently used LLMs (e.g., GPT-4o) in the Diverse Enhancement and question generation steps of prompt construction to expand prompts and parse physical laws. Afterward, we conducted detailed manual reviews to control the quality of PhyGenBench. In the future, we will further explore automated approaches to generate high-quality prompts, reducing human effort in this process. \n\n2.We will consider using the automated prompt construction method mentioned above to generate a training set, then use various T2V models to generate videos and conduct human scoring to provide reward signals. Subsequently, we can train T2V models using processes like DPO. Alternatively, we can use human scoring to train reward models and implement RLHF algorithms like PPO to optimize the models' physical understanding capabilities.\n\nQ4: physics-based video synthesis or evaluation could strengthen the context provided...\n\nA4: We will discuss these directions you mentioned in the related work section of the next version of our paper. For example, PhysGen[1] simulating videos of rigid body motion; PhysMotion[2] simulating real I2V scenarios, etc.\n\nYour suggestions are greatly valued, and please let us know if you need any clarification.\n\n[1] PhysGen: Rigid-Body Physics-Grounded Image-to-Video Generation\n\n[2] PhysMotion: Physics-Grounded Dynamics From a Single Image", "claims_and_evidence": "The claims made in the submission are clearly supported by convincing evidence. The authors provide comprehensive evaluations, demonstrating that current T2V models fail to capture intuitive physical laws robustly. Experimental results are thoroughly presented, including comparisons across various state-of-the-art models, clearly highlighting their shortcomings in physical correctness.", "methods_and_evaluation_criteria": "The claims made in the submission are clearly supported by convincing evidence. The authors provide comprehensive evaluations, demonstrating that current T2V models fail to capture intuitive physical laws robustly. Experimental results are thoroughly presented, including comparisons across various state-of-the-art models, clearly highlighting their shortcomings in physical correctness.", "theoretical_claims": "The paper does not primarily rely on theoretical proofs, so this section is not applicable.", "experimental_designs_or_analyses": "The experimental designs and analyses are sound. The authors clearly document their evaluation methodology, including human assessments, which validate their automated metric's high alignment with human judgments.", "relation_to_broader_scientific_literature": "This paper makes a good contribution by addressing a gap in current benchmarks, which primarily focus on visual quality or semantic alignment, by explicitly assessing intuitive physics in T2V models. The authors position their work effectively relative to existing benchmarks like VideoPhy, VideoScore, and DEVIL, highlighting the unique aspects and strengths of their proposed methods.", "essential_references_not_discussed": "The authors have discussed relevant literature. However, explicitly referencing prior efforts in physics-based video synthesis or evaluation could strengthen the context provided.", "other_strengths_and_weaknesses": "NO", "other_comments_or_suggestions": "NO", "questions_for_authors": "Can the PhyGenEval framework be easily adapted to new physical phenomena, or does this require extensive manual recalibration?\n\nHave you explored incorporating simulation-generated videos as references in your benchmark, and could this improve the robustness of your evaluations?\n\nWould it be feasible to automatically generate prompts using generative models or reinforcement learning to improve scalability and diversity further?", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "2pcSZKdG7E", "reviewer": "Reviewer_AK75", "summary": "The paper introduces a benchmark designed to assess the extent to which generative video models internalize physical laws. The authors construct a dataset comprising 160 prompts that incorporate 27 physical phenomena. Additionally, they propose a method leveraging large vision-language models to automatically evaluate the physical correctness of the generated videos. They evaluate 8 open-source models and 6 proprietary models.", "questions": "", "limitations": "", "rating": 3, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Many thanks for your feedback, which is vital to improving our paper's standard. We respond to all inquiries in sequence and will incorporate those details in the revision.\n\nQ1: The evaluation is done using only 160 which might not be significant enough...\n\nA1: Following your suggestion, we randomly selected 100 prompts from the 160 prompts and repeated this 10 times. We calculated the Spearman coefficient and Kendall correlation coefficient between the model's ranking and the ranking using all 160 prompts. The results were shown in the table below, reflecting the similarity of the model rankings and indicated that a low variance existed within the current prompts set.\n\n| Spearman | Kendall |\n| -------- | ------- |\n| 0.87     | 0.82    |\n\nOur current 160 prompts focuses on the most basic physical laws and scenarios, which should be sufficient to expose problems in the models. As T2V models develop, we will gradually expand to include new physical laws and design more complex scenarios that reflect physical principles.", "claims_and_evidence": "They claim that scaling and prompt engineering alone do not significantly improve physical commonsense.\nThey claim that there automated evaluation method is suited to evaluate physical commonsense, which is confirmed by the correlation with human evaluations and by the fact that the evaluation is robust to changes that affect visual quality without affecting the physical correctness.", "methods_and_evaluation_criteria": "The authors create there own benchmark and evaluation method that seems robust to changes that affect visual quality without affecting the physical correctness.", "theoretical_claims": "N/A", "experimental_designs_or_analyses": "Each video was evaluated by 3 independent annotators.", "relation_to_broader_scientific_literature": "Evaluating the adherence of large models to physical laws is a very important direction, given the wide adoption of these models in the industry.", "essential_references_not_discussed": "N/A", "other_strengths_and_weaknesses": "Strength: The paper is clear. The score decomposition used in the automatized evaluation process makes a lot of sense. \nWeakness: The evaluation is done using only 160 which might not be significant enough. Would it be possible to perform the evaluation on 1000 prompts and show that the score ordering does not change? Or alternatively, sample 10 sets of 100 prompts among the 160 and compute the variance. I am happy to raise my score if the authors can show that the computed score is low-variance.", "other_comments_or_suggestions": "N/A", "questions_for_authors": "N/A", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "QKTfYkDakm", "reviewer": "Reviewer_uagC", "summary": "The paper introduces PhyGenBench, a benchmark designed to evaluate whether Text-to-Video models accurately adhere to fundamental physical laws. The study aims to assess how well these models can simulate intuitive physics, which is considered essential for developing a general world simulator. To systematically evaluate T2V models, the paper proposes a hierarchical evaluation method, PhyGenEval, which assesses the physical commonsense correctness of generated videos. The study evaluates state-of-the-art T2V models.", "questions": "", "limitations": "", "rating": 3, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We appreciate your suggestions, which are essential for enhancing the paper. We address all questions sequentially and will incorporate those details in the revisions.\n\nQ1:Comparing generated videos with real physics to eval.\n\nA1:We considered using real videos as references to eval but encountered some scalability challenges. Collecting gt videos for each prompt was difficult, as physical processes varied greatly across scenarios, making a single deterministic video insufficient as a golden reference. Additionally, evaluating semantic consistency across videos with different frame rates and scenarios remained an unresolved challenge.\n\nInstead, we incorporated real world physical videos into PhyGenEval as references. We sampled 50 real videos along with detailed captions from WISA[1]. After parsing these captions into the PhyGenBench format and evaluting the real videos, our evaluation showed they achieved extremely high physical alignment scores under PhyGenEval, serving a strong reference baselines for other models to compare. And demonstrated the robustness of PhyGenEval for real physical scenarios.\n\n| Mechanics(17 samples) | Optics(17) | Thermal(16) |\n| -- | -- | -- |\n| 0.93 | 0.95 | 0.93 |\n\nWe further tested the performance of 8 models on these prompts, using real physical video scores as a normalization score to for more realistic comparison (e.g., model generated videos score / real physical videos score).\n\n|   | Mechanics | Optics |Thermal | Avg. |\n| -- | -- | -- | -- | -- |\n| CogVideoX2B | 0.36\\0.37 | 0.44\\0.49 | 0.33\\0.41 | 0.39\\0.39 |\n| CogVideoX5B  | 0.36\\0.36 | 0.52\\0.57 | 0.47\\0.53 | 0.46\\0.50 |\n| OpensoraV1.2 | 0.36\\0.38 | 0.49\\0.52 | 0.39\\0.39 | 0.43\\0.45 |\n| Lavie  | 0.23\\0.27 | 0.42\\0.45 | 0.36\\0.40 | 0.36\\0.38 |\n| Vchitect2.0 | 0.41\\0.43 | 0.52\\0.57 | 0.42\\0.44 | 0.47\\0.50 |\n| Hunyuan | 0.46\\0.49 | 0.53\\0.55 | 0.39\\0.40 | 0.48\\0.51 |\n| Pyramid Flow(flux) | 0.33\\0.37 | 0.50\\0.54 | 0.44\\0.50 | 0.44\\0.48 |\n| Pyramid Flow(sd3) | 0.43\\0.54 | 0.46\\0.52 | 0.33\\0.40 | 0.42\\0.49 |\n\nThe Spearman of model rankings calculated by these two methods was **0.90**, indicating the robustness of PhyGenEval. And this could be further improved by incorporating real physical video scores. We will include this in the revision.\n\nQ2:How do you ensure that these models correctly assess physical realism.\n\nA2:Compared with VLMs, LLMs were trained on real physics reasoning datasets, demonstrating better physics understanding capabilities. Thus, we addressed VLM evaluation limitations by using LLMs to understand physical laws during PhyGenBench construction, reducing physics comprehension difficulty. In PhyGenEval, we also incorporated priors from physical laws and evaluated basic principles step by step(e.g.,The egg breaks after it hits the stone), lowering the evaluation difficulty. As demonstrated in Table 10, PhyGenEval significantly exceeded the direct evaluation capabilities of GPT-4o.\n\nQ3:Some physical laws require additional context.\n\nA3:As shown in Figure 2b, we addressed exactly this issue through a prompt augmentation stage. For example, we transformed \"egg collides stone\" into \"fragile egg was hurled with significant force towards solid rock,\" to eliminated potential ambiguities. Furthermore, Quality Control check for prompt completeness also verified it. (line 209).\n\nQ4:Expect PhyGenBench to generalize.\n\nA4:Currently, we focused on basic physical laws that effectively revealed limitations in existing T2V models. As simulation engines and T2V models became more powerful, we hoped to generate real reference videos for each prompt in the future, and trained video-to-video scoring models through them.\n\nQ5:FT on physics-specific dataset.\n\nA5:We randomly selected 1200 Video-Text pairs from WISA and performed lora(r=128) fine-tuning on CogVideoX 5B. The results indicated that this did not solve the problem. We believed this might due to: The training set was too small to cover enough domains; The base model was not strong enough, making it difficult to generalize; More explicit injection of physical laws was needed, such as training with synthetic videos.\n\n| Model | Mech. | Opt. | The. | Mat. | Avg.  |\n| -- | -- | -- | -- | -- | -- |\n| CogVideoX 5B | 0.39  | 0.55 | 0.40 | 0.42 | 0.449 |\n| +FT | 0.38  | 0.58 | 0.41 | 0.40 | 0.453 |\n\nQ6:Error analysis.\n\nA6:We provided some error cases in line 1094. Besides, We collected 50 videos where machine scores differ from human. The statistical information is:\n\n|Type | Per. | Avg Diff. (0-3)|\n| -- | -- | -- |\n| Higher | 90%  | 1 |\n| Lower  | 10%  | 1  |\n\nWe defined 3 core error cases: spatial, semantic, and temporal understanding errors. The results show that most are due to temporal understanding, which we will improve it in the future.\n\n| Sem. | Spa. | Tem. |\n| ---- | ---- | ---- |\n| 28%| 12%| 64% |\n\nOthers:We have already cited PhysGen(line 1112).We will update InterDyn in the revision.\n\n[1] WISA: World Simulator Assistant for Physics-Aware Text-to-Video Generation", "claims_and_evidence": "The claims made in the submission are generally supported by clear and well-structured evidence, but some areas may require further validation or refinement. \n\nThe use of Vision-Language Models and GPT-4o can introduce potential biases. The paper does not provide a robust error analysis of PhyGenEval’s failure cases.\n\nThe paper positions PhyGenBench as a step toward general world simulation. However, no discussion is provided on how improving physics modeling would integrate into broader world simulation goals. How does this work compare to embodied AI efforts in world modeling?", "methods_and_evaluation_criteria": "Yes, the proposed methods and evaluation criteria, including PhyGenBench and PhyGenEval, generally make sense for evaluating the physical commonsense capabilities of T2V models.\n\nThe use of GPT-4o and other VLMs raises concerns about potential biases and failure cases (e.g., if the VLM itself has poor physical commonsense). \n\nThe paper argues that scaling models alone does not solve physical reasoning issues, but does not test whether fine-tuning on physics-specific datasets could help.\n\nThe paper positions PhyGenBench as a step toward general world simulation, but does not explicitly compare its approach to other physics-focused AI evaluations (e.g., embodied AI or reinforcement learning environments).", "theoretical_claims": "The paper primarily focuses on benchmarking and evaluation of T2V models rather than presenting formal theoretical claims with rigorous proofs.", "experimental_designs_or_analyses": "Yes, I analyzed the soundness and validity of the experimental design and analysis in the paper. Overall, the experiments are well-structured and provide valuable insights.\n\nThe paper’s three-level evaluation framework (PhyGenEval) effectively assesses physical commonsense in T2V models. By breaking down evaluation into Key Physical Phenomena Detection, Physics Order Verification, and Overall Naturalness, it ensures a systematic, structured analysis aligned with human judgment.\n\nThe wide-ranging model evaluation ensures fair comparisons and highlights that even top models struggle with intuitive physics, reinforcing the gap between AI-generated videos and real-world simulation.\n\nLimitations:\n\nThe study evaluates T2V models only against text-based descriptions, rather than comparing outputs with real-world physics simulations. This could limit the reliability of the benchmark, as models may generate visually plausible but physically incorrect videos that still align with textual prompts. Incorporating physics simulation datasets as a reference could provide a more objective evaluation of physical correctness in generated videos.\n\nSome physical laws require more context than what is provided in the text-based prompts. For example, a prompt about an object floating or sinking in water may not specify material density, leading to ambiguous interpretations. Ensuring that prompts contain clear and complete physical conditions would improve the accuracy of assessments.\n\nA key concern is that PhyGenEval relies on GPT-4o and Vision-Language Models for scoring. If GPT-4o has flaws in its understanding of physics, it could introduce biases into the evaluation. The study does not analyze cases where PhyGenEval disagrees with human evaluators, making it difficult to assess its true reliability. Conducting an error analysis could help identify potential weaknesses in the evaluation framework.\n\nAdditionally, while the study claims that scaling models alone does not improve physical reasoning, it does not test whether fine-tuning on physics-specific datasets could lead to significant improvements.", "relation_to_broader_scientific_literature": "The paper builds on advancements in T2V generation, which has rapidly improved in terms of visual quality, motion coherence, and scene complexity. Models like Sora, Gen-3, and Pika can generate high-resolution videos from text prompts, but they lack an understanding of physical commonsense. Prior research in video generation has largely focused on aesthetic quality and temporal consistency, rather than ensuring that generated videos follow real-world physics. This paper addresses that gap by introducing PhyGenBench, a benchmark that explicitly evaluates whether T2V models generate videos that align with fundamental physical laws.\n\nTraditional metrics like Fréchet Video Distance (FVD) assess video quality but fail to measure physical correctness. The paper introduces PhyGenEval, a three-tier evaluation framework that combines Vision-Language Models and GPT-4o to assess physical correctness in generated videos. This approach moves beyond simple perceptual metrics and provides a structured way to measure how well AI models understand physics.", "essential_references_not_discussed": "The paper claims that current T2V models fail at intuitive physics but does not cite prior work on physics-based video generation, where researchers have attempted to incorporate physical constraints into generative models.\n\nPhysGen: Rigid-Body Physics-Grounded Image-to-Video Generation\nInterDyn: Controllable Interactive Dynamics with Video Diffusion Models", "other_strengths_and_weaknesses": "Strengths: \n\nOne of the paper's key strengths is its originality in addressing a critical gap in T2V evaluation. While prior research has focused on visual quality, motion coherence, and spatial relationships, this study is one of the first to systematically assess physical commonsense adherence in generated videos.\n\nThe three-tier evaluation structure in PhyGenEval is another significant contribution. Unlike traditional metrics like Fréchet Video Distance (FVD) and VideoScore, which primarily measure perceptual fidelity, the proposed framework evaluates physical commonsense in the generated videos.\n\nWeaknesses:\n\nPhyGenBench evaluates T2V models only against text-based descriptions, rather than comparing them to real-world physics simulations. This raises concerns about whether the benchmark truly reflects real-world physics fidelity.\n\nWhile the study argues that PhyGenEval aligns well with human judgment, it does not analyze failure cases where GPT-4o misinterprets physics. Since LLMs and VLMs may not have a strong grasp of causality or dynamics, their scoring could introduce systematic biases.", "other_comments_or_suggestions": "Typos:\n\nL1075: \"we effectively reduces\" -> \"we effectively reduce\"\n\n\"sementic\" --> \"semantic\"", "questions_for_authors": "The paper evaluates T2V models against text-based descriptions rather than real-world physics simulations or recorded videos. Have you considered comparing generated videos with real physics-based datasets? If not, how do you justify this omission, given that real-world comparisons would provide a more objective evaluation?\n\nThe evaluation framework relies on GPT-4o and Vision-Language Models. How do you ensure that these models correctly assess physical realism, given that VLMs are not explicitly trained for physics verification?\n\nSome physical laws require additional context that may be missing from text-based prompts. How do you ensure that prompts are unambiguous and do not introduce interpretation biases?\n\nHow well do you expect PhyGenBench to generalize as future T2V models improve? Will the benchmark need updating as models become more sophisticated, or is it designed to remain relevant long-term?", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Meta-review: This paper proposes a new benchmark for video models as "world simulators", quantifying some recent claims that T2V models could be important for embodied AGI research. The paper proposes a hierarchical approach which tackles multiple dimensions of the problem and makes it clear as to which models work where. This work could become an important benchmark in a rapidly expanding field and lays the bedrock for more rigorous evaluation. All reviewers recommended accept and authors addressed feedback constructively.
Decision: Accept (poster)