Title: WISA: World simulator assistant for physics-aware text-to-video generation
Venue: NeurIPS 2025
Accepted: yes
Keywords: Video Generation

Diffusion Model

Video Dataset
Abstract: Recent advances in text-to-video (T2V) generation, exemplified by models such as Sora and Kling, have demonstrated strong potential for constructing world simulators. However, existing T2V models still struggle to understand abstract physical principles and to generate videos that faithfully obey physical laws. This limitation stems primarily from the lack of explicit physical guidance, caused by a significant gap between high-level physical concepts and the generative capabilities of current models. To address this challenge, we propose the **W**orld **S**imulator **A**ssistant (**WISA**), a novel framework designed to systematically decompose and integrate physical principles into T2V models. Specifically, WISA decomposes physical knowledge into three hierarchical levels: textual physical descriptions, qualitative physical categories, and quantitative physical properties. It then incorporates several carefully designed modules—such as Mixture-of-Physical-Experts Attention (MoPA) and a Physical Classifier—to effectively encode these attributes and enhance the model’s adherence to physical laws during generation. In addition, most existing video datasets feature only weak or implicit representations of physical phenomena, limiting their utility for learning explicit physical principles. To bridge this gap, we present **WISA-80K**, a new dataset comprising 80,000 human-curated videos that depict 17 fundamental physical laws across three core domains of physics: dynamics, thermodynamics, and optics. Experimental results show that WISA substantially improves the alignment of T2V models (such as CogVideoX and Wan2.1) with real-world physical laws, achieving notable gains on the VideoPhy benchmark. Our data, code, and models are available in the [Project Page](https://wisav1.github.io/WISA/).

WISA: World Simulator Assistant for Physics-Aware Text-to-Video Generation

Abstract

Recent advances in text-to-video (T2V) generation, exemplified by models such as Sora and Kling, have demonstrated strong potential for constructing world simulators. However, existing T2V models still struggle to understand abstract physical principles and to generate videos that faithfully obey physical laws. This limitation stems primarily from the lack of explicit physical guidance, caused by a significant gap between high-level physical concepts and the generative capabilities of current models. To address this challenge, we propose the W orld S imulator A ssistant ( WISA ), a novel framework designed to systematically decompose and integrate physical principles into T2V models. Specifically, WISA decomposes physical knowledge into three hierarchical levels: textual physical descriptions, qualitative physical categories, and quantitative physical properties. It then incor-

porates several carefully designed modules-such as Mixture-of-Physical-Experts Attention (MoPA) and a Physical Classifier-to effectively encode these attributes and enhance the model's adherence to physical laws during generation. In addition, most existing video datasets feature only weak or implicit representations of physical phenomena, limiting their utility for learning explicit physical principles. To bridge this gap, we present WISA-80K , a new dataset comprising 80,000 human-curated videos that depict 17 fundamental physical laws across three core domains of physics: dynamics, thermodynamics, and optics. Experimental results show that WISA substantially improves the alignment of T2V models (such as CogVideoX and Wan2.1) with real-world physical laws, achieving notable gains on the VideoPhy benchmark. Our data, code, and models are available in the https://wisav1.github.io/WISA/.

1 Introduction

Many recent studies (e.g., Cosmos [1], Kling [14], Step-Video-T2V [21], Sora [26], and CogVideoX [41]) have endeavored to develop robust text-to-video (T2V) models for building world simulators [39, 6, 43]. While these models are capable of generating highly realistic and text-consistent videos, leveraging the scale of their data and architectures, they still face challenges in understanding abstract physical principles and producing videos that fully align with real-world physical laws [3, 23].

The substantial gap between abstract physical laws and their visual manifestations presents a significant challenge for injecting physical guidance into T2V models. Physical principles or laws are often conveyed through abstract natural language, reflecting the underlying operational logic of the real world. In contrast, generative models map textual descriptions directly to the visual appearance of objects, including their color and shape. There is a complex logical reasoning process between physical principles and the visual physical phenomena they give rise to. However, generative models, which are trained to map learned data distributions, struggle to extract appropriate physical information from a single textual instruction and translate it into a physically consistent visual representation for a specific scenario. This challenge becomes even more pronounced in video generation, where the strict temporal order of physical events must be preserved.

To this end, we propose the W orld S imulator A ssistant ( WISA ), which decomposes abstract physical principles into multiple categories of physical information and integrates them into T2V models to enable physics-aware generation. Specifically, it decomposes physical principles into three levels: textual physics descriptions, qualitative physics categories, and quantitative physical properties, and designs appropriate tailored injection methods for each type of information. The Textual Physical Description outlines the physical principles relevant to the scene, the resulting physical phenomena, and their specific visual manifestations. WISA incorporates this information by concatenating it with the caption before feeding it into the text encoder. The Qualitative Physics Categories indicate the types of physical phenomena that may be present in a scene. Following the focus of existing physical T2V benchmarks (e.g., VideoPhy and PhyGenBench), WISA targets 17 representative phenomena commonly encountered in video generation tasks. These span three major branches of physics (i.e., dynamics, thermodynamics, and optics) and include examples such as collision (dynamics), refraction (optics), and melting (thermodynamics). Recognizing that different physical phenomena require distinct physical features, WISA proposes M ixture-ofP hysical-Experts A ttention ( MoPA ), inspired by MoE [30] and MoH [12]. MoPA assigns expert attention heads to individual physics categories, activating only the relevant experts during generation to specialize in modeling the associated phenomena. When a scene involves multiple physical phenomena, MoPA dynamically activates multiple expert heads, allowing the model to effectively capture and synthesize complex physical interactions. Quantitative Physics Properties refer to numerical physical attributes that directly influence the physical process, such as density, duration, and temperature. WISA encodes these properties as physical embeddings and injects them into the model via AdaLN [27]. In addition, WISA employs a Physical Classifier, which is designed to recognize qualitative physics categories and assist in perceiving physical properties.

However, extracting the above physical information and subsequently understanding physical principles from general scene video in existing datasets [25, 36] is a suboptimal approach for T2V models. Firstly, general scene videos often feature the interweaving of multiple physical phenomena. Individual physical phenomena are not prominently visualized, which makes it difficult to accurately extract physical information and establish a precise connection between the physical data and its

corresponding visual manifestation. Secondly, in these datasets, only a few videos distinctly highlight specific physical phenomena as representative examples, while most videos treat physical phenomena as secondary elements. For instance, in the Figure 2, the flow of water is a secondary element. Despite having physical information guidance, the T2V models are unable to perceive the physical principles of fluid motion from this type of data.

To address these challenges, we collect and construct WISA-80K , a dataset containing 80,000 videos that represent 17 physical phenomena across three major branches of physics as shown in Figure 1, designed as a data assistant for world simulators. Specifically, based on the previously defined physics categories, we manually collect videos that clearly exhibit obvious physical phenomena corresponding to each category (e.g., as shown in the lower part of Figure 2). We then apply shot boundary detection, aesthetic

quality filtering, and video captioning to the raw videos. Subsequently, we leverage GPT-4o mini to extract and decompose the physical information from the video captions into textual physics descriptions, qualitative physics categories, and quantitative physics properties for WISA.

Our contributions can be summarized as follows:

We propose a physical principle decoupling method, bridging the gap between physical laws and generative modeling. In this method, physical principles are represented as structured physical information, encompassing textual physical descriptions, qualitative physics categories, and quantitative physical properties.

We present the World Simulator Assistant (WISA), which guides T2V models to efficiently learn specific physical phenomena based on structured physical information, through specialized designs such as Mixture-of-Physical-Experts Attention (MoPA) and Physical Classifier.

We manually collect 80,000 video clips that clearly showcase physical phenomena, creating the first large-scale physics video dataset, WISA-80K. It broadly covers common physical phenomena observed in the real world, encompassing 17 types of physical events (e.g., Collision, Melting, and Reflection) across three major branches of physics.

Quantitative and qualitative experimental results demonstrate WISA and WISA-80K can greatly assist basic T2V models in producing videos that better align with real-world physical laws, while introducing only a 3.5% increase in parameter count and 5% inference time.

2 Related Work

Text-to-Video Generation Early text-to-video (T2V) generation research [10, 9, 33, 7, 8, 34, 4] primarily extend image generation models [5, 18, 17, 28, 19, 22] with temporal capabilities to enable video generation. These methods often suffered from limited realism and restricted motion dynamics. The powerful 3D spatio-temporal modeling and scalability of Diffusion Transformers [27, 15] have greatly advanced the development of visual generation models. Enabled by Diffusion Transformers, a series of recent T2V works (including OpenSora [42], Cosmos [1], Sora [26], CogVideoX [41], HunyuanVideo[x], Kling [14], Wan2.1 [31], and Step-Video-T2V [21]) significantly improve the realism and motion quality of video generation by scaling up model parameters and training data. These works are widely considered as a promising pathway towards building a World Simulator. However, they still struggle to generate videos that fully comply with real-world physical laws as they essentially fit the data distribution [13] from general-scene datasets such as Koala-36M [36] and OpenVid [25], where physical laws are not explicitly reflected and physical phenomena are not prominently presented (e.g., in the upper part of Figure 2). In contrast, our carefully curated WISA-80K dataset prioritizes the explicit presentation of typical physical phenomena as the primary criterion for video collection as presented in Figure 1. And it provides detailed and structured

physical information annotations, making it a valuable data assistant for enhancing the physical consistency of video generation.

Physical-aware Video Generation Recently, researchers [24, 3, 23, 16, 2, 20, 40, 37] have increasingly focused on improving and evaluating the physical consistency of generated videos. On the one hand, Videophy [3] and PhyGenBench [23] build test samples that reflect various physical laws, and they evaluate how well generated videos follow real-world physical laws by either training physics classification models with manual annotations or using question-answering methods based on Vision-Language models [38]. Physics-IQ[24] establishes a high-quality image-to-video benchmark designed to evaluate the ability of I2V models to generate physically consistent video sequences based on an initial state and textual instructions. On the other hand, DANO [16], MotionCraft [2], and PhysGen [20] parse objects from images and estimate their rigid motion in a differentiable manner by considering physical properties such as mass, inertia, friction, and rotation. Based on these estimations, they animate the images into videos. However, these methods are restricted to fixed physical categories (e.g., rigid motion) and static scenarios that involve only object motion, which hinders their generalizability. PhyT2V [40] leverages large language models and vision-language models to extract physical inconsistency information from generated videos. Based on the extracted physical feedback, it iteratively refines the textual description over multiple rounds, improving video generation quality. Although this approach offers generality, it introduces significant inference overhead and fails to enhance the generative model's ability to encode physical knowledge. In this paper, WISA incorporates structured physical information into the generative model, enhancing its physical perception and enabling it to handle various physical phenomena more effectively.

3 WISA-80K

3.1 Data Collection and Annotation

Physical Laws Definition: In previous physics evaluation benchmarks [3, 23], the physical phenomena emphasized in video generation tasks have primarily focused on three fundamental branches of physics: Dynamics , Thermodynamics , and Optics . Therefore, in this paper, we select 17 representative physical phenomena from these three core domains, excluding specialized cases such as electromagnetic phenomena.

Dynamics : We consider six common dynamic phenomena: Collision , Rigid Body Motion , Elastic Motion , Liquid Motion , Gas Motion , and Deformation . For instance, the swinging of a pendulum serves as an example of Rigid Body Motion .

Thermodynamics : We select six common thermodynamic phenomena: Melting , Solidification , Vaporization , Liquefaction , Explosion , and Combustion . For example, a time-lapse of melting ice cream illustrates the Melting phenomenon.

Optics : We define five common optical phenomena: Reflection , Refraction , Scattering , Interference and Diffraction , and Unnatural Light Sources .

We did not include certain physical phenomena (e.g., sublimation, condensation) due to their infrequent occurrence in real-world scenarios and the associated difficulties in collecting sufficient high-quality data. Subsequently, for each selected physical phenomenon, we manually collected videos from the Internet that clearly demonstrate the corresponding behavior, without relying on any existing video datasets for filtering or selection. During the collection process, we exclude videos with overlaid text or significant visual blur to ensure clarity and quality. As a result, we curate a dataset comprising approximately 40,000 videos.

Pre-processing and Caption: We use PySceneDetect [29] to split the raw videos into individual scene clips, followed by filtering based on aesthetic scores. This process yields approximately 80,000 high-quality clips. Then, we utilize Qwen2.5-VL [35] to generate video captions using the following prompt: { Please describe the content of this video in as much detail as possible, including the objects, scenery, animals, and camera movements within the video. } The caption length is limited to 256 tokens.

Qualitative physics categories

Dynamics

:

collision, deformation, liquid motion

Thermodynamic

:

explosion

Optics

: (20)

no obvious phenomenon

Camera Movement

: (22)

no

Change of subjects

:

liquid objects appearance,

object decomposition and splitting, mixing of

multiple objects, object disappearance

•

The

most

important

physical

principle

is

the

conservation

of

momentum.

when

the

balloon

bursts, the water inside is rapidly expelled in all

directions, ·

· ·

, leading to a dynamic splash effect.

•

The phenomenon of fluid dynamics is crucial, as

it

dictates

how the water droplets

disperse and

interact with the surrounding air and surfaces.

Textual physical description

Text Description

with Qwen2-VL

Physical Description

with GPT-4o mini

The video depicts

a

person

holding

a

purple

water balloon in their hand. The individual is

positioned outdoors, with a natural background.

Suddenly, a stream of water is directed towards

the balloon, causing it to burst. The water from

the

balloon

sprays

out

in

all

directions,

creating a splash and a burst of water droplets.

Text Prompt

Quantitative physical properties

•

Density: water 1 (

)

•

Time: 0 to 3 (

)

•

Temperature: 15 to 25 (

)

Video Data

Collection

Physical Class Definition

Dynamics:

(1) collision

(2) rigid body motion

Thermodynamic:

(8) melting

(9) Solidification

Optics:

(15) reflection

(16) refraction

Camera Movement:

Change of subjects

：

Manually

Search

Manually

Evaluation

Scene detection and

striping

Aesthetics screening

Q

1

1

1

C↑-

S

88

园

目

3.2 Physical Information Decompose

We believe that simple video captions are not sufficient to clearly represent the physical information and related physical phenomena in a video. As shown in the Figure 3, we further constructed structured physical annotations to analyze the physical information from multiple dimensions. Specifically, we decompose the physical information into: textual physical descriptions , qualitative physics categories , and quantitative physical properties .

Textual physical descriptions : Provide a detailed explanation of the physical principles to be considered and the resulting intuitive physical phenomena, while supplementing the missing physical information in the prompt. For instance, the prompt " an antique clock swings " corresponds to the textual physical description: " ... the amplitude of the swing gradually decreases ... ".

Qualitative physics categories : These indicate the types of physical phenomena involved in a video. Although each video is collected based on a specific physical phenomenon, it may still encompass multiple types. Therefore, for each video, we identify the presence of dynamics-related, optics-related, and thermodynamics-related phenomena to effectively handle cases where multiple physical effects are coupled. Additionally, three categories of anomalies (i.e., No obvious dynamic phenomenon , No obvious thermodynamic phenomenon , and No obvious optical phenomenon ) are introduced to account for scenarios that do not involve dynamics, thermodynamics, or optical phenomena. Furthermore, nine categories of visual phenomena are introduced, two of which pertain to whether the shot exhibits motion, while the remaining seven correspond to changes in the state of moving entities (i.e., Object decomposition and splitting , Mixing of multiple objects ... For detailed explanations, please refer to Supplementary Material A.5 . In total, there are 29 qualitative physics categories.

Quantitative physical properties : Three physical attributes related to multiple physical phenomena are annotated, namely the density of primary motion physics, the time range during which the physical phenomenon occurs, and the temperature range during which the physical phenomenon occurs.

Due to the significant computational overhead and cost associated with video multi-modal models, the annotation of the above physical information is carried out using GPT-4o mini based on the caption. Specifically, we conduct five rounds of annotation to label qualitative physical phenomenon categories (i.e., dynamics, thermodynamics, optics, motion, the state of objects), and three rounds to annotate quantitative physical attributes (i.e., Density , Time and Temperature ). Detailed annotation prompts and examples are provided in the Supplementary Material A.7 and A.8 .

To verify the reliability of the automatic annotations, we conducted a manual evaluation on a randomly sampled subset of 500 videos from our dataset. Our evaluation focused on the three types of AIgenerated annotations: 1) Textual Physical Descriptions: We measured human rater satisfaction (i.e., does the description accurately reflect the video's physics?). Result: 95% satisfaction rate. 2) Qualitative Physics Categories: We compared the AI-generated labels against the ground-truth labels assigned during initial video collection. Result: 76% accuracy. 3) Quantitative Physical Properties: We again measured human rater satisfaction (i.e., are the estimated density/time/temperature values plausible?). Result: 86% satisfaction rate. While some label noise is inherent in any large-scale,

T5

Physical

Module

Physical

Classifier

Scale, Shift

Physical Module

Gate

Quantitative

Physical

Properties

•

Density (

𝑔/𝑐𝑚 3

)

•

Time (

𝑠

)

•

Temperature (

℃

)

Scientific

Notation

Conversion

Timestep

PQ embedding

Linear

Linear

Head 1

Head 1

Linear

Head 2

Head 2

Linear

Head n-1

Head n-1

Linear

Head n

Head n

Mixture-of-Physical-Experts Attention

Physics-Guided Random Gate

Output

Input

T5

Physics Class

Guidance

Random

Perturbations

World Simulator Assistant (WISA)

Linear

Qualitative physics categories

Dynamics

: (2)

Rigid body motion

Thermodynamic

: (14)

no obvious phenomenon

Optics

: (20)

no obvious phenomenon

Camera Movement

: (22)

no

Change of subjects

: (29)

no

Output

Projection

LoRA

MM-DiT

Block

(2)

(29)

(21)

Physics Class

Guidance

𝐿𝐵𝐶𝐸

Visual noised token

Physical token

Addition

Concatenate

Prompt token

1.

The  most  important  physical  principle  to

consider is the conservation of energy, ·

· ·

2.

As  the  pendulum  swings,  ·

· ·

,  but  due  to

damping  (e.g.,  air  resistance  and  internal

friction),

the

amplitude

of

the

swing

gradually  decreases  over  time  while  the

frequency remains constant.

Textual physical description

The pendulum of an antique clock swings

back and forth inside its  casing, the metal

rod

and

bob

moving

in

a

stable

and

rhythmic motion.

Text Prompt

Physical class token

automatically annotated dataset, these results demonstrate that the overall quality of WISA-DATA80K's annotations is high. The data is sufficiently reliable to provide a strong learning signal, as evidenced by the performance gains in experiments. More analysis of WISA-80K please refer to the Supplementary Material A.6 and A.9 .

4 Method

4.1 Overview

Given textual physical descriptions, qualitative physical categories, and quantitative physical properties, we design the WISA framework to efficiently incorporate these conditions into existing T2V models (i.e., CogVideoX [41] or Wan2.1 [31]). To facilitate the learning of physical knowledge while preserving the model's original capabilities with limited video data, we design three distinct condition injection methods tailored to each of the three categories of physical information, as illustrated in Figure 4. Specifically, for the textual physical descriptions, we concatenate them with the video caption and leverage the generative model's inherent semantic understanding to generate visual phenomena described in text (Such as "amplitude of the swing gradually decreases over time" in Figure 4). For qualitative and quantitative physical conditions, WISA introduces the Physical Module. In this module, we propose a Mixture-of-Physical-Experts Attention (MoPA), which assigns expert heads to each physics category to model category-specific features. Quantitative physical quantities are encoded as physical embeddings and then integrated into the denoising feature within the module using AdaLN. Additionally, we introduce a qualitative Physical Classifier to help the model understand the physical conditions. Due to the significant computational and parameter cost introduced by MoPA, only one physical module is inserted after all the Diffusion transformer blocks to accelerate training and reduce the overall burden. Detailed explanations and elaborations of the Physical Module and Physical Classifier are provided in Sec. 4.2 and Sec. 4.3.

4.2 Physical Module

Most videos from real-world scenes involve the coupling of multiple physical phenomena. Even when decomposed into distinct physical categories in WISA-80K, it remains challenging for T2V models to comprehend the abstract qualitative physical categories and accurately model specific types of physical phenomena. To address this challenge, we propose a Mixture-of-Physical-Experts Attention within the Physical Module. Inspired by MoH [12], this mechanism assigns each head in the multi-head self-attention to a specific class of physical phenomena and activates the output of the relevant head only when the corresponding phenomenon is present. This approach treats each head as an expert in its domain, enabling it to independently model the properties of a particular physical phenomenon. In the presence of coupled physical phenomena, multiple corresponding expert heads are activated to effectively model the interactions among them.

Specifically, qualitative physical categories are encoded as P c ∈ R C , where C denotes the number of defined physical phenomena (i.e., 29). Here, P i c = 0 indicates that the corresponding category is not activated, and P i c = 1 indicates that the corresponding category is activated, with i being the category index. Physical categories cannot be absolutely correct and may contain noise, such as incorrect activations or suppressions. To mitigate the impact of these noises on training, we employ a random perturbation operation, where the positions with P i c = 1 are set to 0, and the positions with P i c = 0 are set to 1.0 with a certain probability (i.e., 0.2), resulting ˆ P c . After the multi-head self-attention operation, the denoising feature F h ∈ R N × d × h (where h presents the number of head and h = C , and d denotes head dimension) will interact with ˆ P c to activate and suppress the experts corresponding to different physical phenomena. The feature dimension is then restored through concatenation and a linear layer. The mathematical representation of this process is as follows:

where Random denotes random perturbations operation, MHSA represents multi-head self-attention, and ⊙ denotes element-wise multiplication.

Due to the large variations in the time and temperature spans of different physical phenomena, we first represent the temperature and time in quantitative information using scientific notation, with coefficients and exponents. These values P p ∈ R n are mapped through a linear layer, concatenated with the timestep embedding T e ∈ R t , and injected by AdaLN. The mathematical representation of this process is as follows:

Generative models often consist of multiple transformer blocks with large feature dimensions, inserting the Physical Module after every block would lead to an explosion in both parameters and computational complexity. Therefore, we insert the Physical Module only after the final transformer block, achieving efficient physical information guidance while mitigating the aforementioned issues.

4.3 Physical Classifier

To guide the generative model in understanding abstract physical categories and modeling physical properties, we introduce a Physical Classifier after the Physical Module to predict qualitative physical categories. We introduce a learnable embedding vector, which we call the [PHYSICS_TOKEN]. This token is concatenated with the noisy visual tokens and text prompt tokens and is processed by the entire MM-DiT and MoPA architecture. The final hidden state of the [PHYSICS_TOKEN] F c ∈ R C . is fed into a simple MLP classification head. This head outputs logits corresponding to the 29 qualitative physical categories, performing a multi-label classification task over the categories defined in our work. This output is used only to compute the multi-label binary cross-entropy loss for training.

where C is the number of physical categories, and f c ∈ R C represents the predicted probabilities, which are obtained by passing F c through the sigmoid function. For each video, the model predicts which of these phenomena are present. During inference, the output from the classifier head is entirely discarded and has no influence on the video generation process. Its sole purpose is to serve as an auxiliary training signal, guiding the model to better learn and represent physical concepts.

To balance the introduced classification loss L pc and the diffusion loss L diffusion , we adopt the following loss function to optimize the physics-aware generative model.

where λ is balance coefficient.

5 Experiments

Training Setting: We select the current representative open-source T2V model, CogVideoX-5B and Wan2.1-14B, as the base T2V models to validate the effectiveness of WISA. More training detail, please refer to Supplementary Material A.3 .

Evaluation: We select VideoCon-Physics from Videophy [3] to evaluate the physical law consistency (PC) and semantic coherence (SA) of the generated videos. We use 160 carefully crafted prompts from PhyGenBench [23] and 344 prompts from Videophy, designed to reflect various physical principles, for testing. Following VideoCon-Physics 2 , we compute SA and PC by averaging the predicted results. Additionally, we adopt the Inception Score (IS) to evaluate the perceptual quality of generated videos and employ CLIP similarity (CLIPSIM) [11] to measure text-video alignment. More evaluation detail, please refer to Supplementary Material A.3 .

5.1 Quantitative comparison

We select five general text-to-video generation models (i.e., VideoCrafter2, OpenSora, HunyuanVideo, CogVideoX-5B and Cosmos-Diffusion-7B) and PhyT2V, a method specifically designed to enhance physical properties, for quantitative comparison, as shown in Table 1.

VideoPhy : WISAachieves state-of-the-art performance on both SA and PC metrics while maintaining high efficiency. Compared to the CogVideoX-5B, CogVideoX-5B-WISA improves SA and PC scores by 0.05 and 0.04, respectively, demonstrating that our proposed method significantly enhances the realism of generated videos. PhyT2V improves its performance by iteratively analyzing physical errors in generated video captions and adjusting the input prompts based on feedback from VideoConPhysics scores. However, its cumbersome pipeline, which involves multiple rounds of Tarsier-34B [32] inference for video generation, introduces extremely long inference time-approximately 9 times longer than the original generation model. Cosmos exhibits poor performance due to the disordered physical processes and inconsistent temporal sequences. Furthermore, Wan2.1-14B-WISA also achieves performance improvements on Wan2.1, showing advantages in metrics such as IS and CLIPSIM. However, since Wan2.1 is not among the nine generative models used to construct the training data for VideoCon-Physics (whereas CogVideoX is included), it shows certain disadvantages in SA and PC. Despite this, Wan2.1 demonstrates superior video quality, achieving better performance in the IS.

PhyGenBench : We also evaluate WISA using prompts from PhyGenBench, observing significant improvements on both CogVidoX and Wan2.1, which demonstrates the generalizability of WISA.

5.2 Qualitative comparison

We further provide a qualitative comparison with existing methods to demonstrate the advantages of WISA. As shown in the Figure 5, for the example of the rope supports a wooden swing, CogVideoX-

The rope supports a wooden swing

CogVideoX-5B

Cosmos

PhyT2V

CogVideo-5B

-WISA

An apple falls into a vat of cider, sending up a spray

Explosion : A firework bursts in mid-air, releasing a brilliant flash of light …. Fragments of colored paper scatter wildly, spiraling away like confetti ...

Melting : An ice cube melts in a warm environment … causes tiny droplets to form on its surface. these droplets gradually merge into small streams, sliding down the edges.

Reflection : A calm lake mirrors a single swan gliding gracefully across its surface … distorting its reflection ... that echoes the motion with each gentle stroke of its wings.

Interference and diffraction : A small soap bubble floats gently in the air. The surface of the bubble … causes light to interfere, creating beautiful rainbow patterns …

Rigid Body Motion : An antique clock's pendulum hangs inside its ornate wooden casing, with the camera focusing on its rhythmic motion. The metal rod remains …

Elastic motion : A bright green rubber band stretches as fingers pull it taut, revealing its thin ... demonstrating the elasticity of materials through clear deformation.

Rigid Body Motion : A solitary lantern swings gently in the night, casting flickering shadows on the cobblestone … creating dancing shapes that morph and twist …

Elastic motion : A vibrant red rubber ball bounces steadily on a smooth, sunlit sidewalk. each impact compresses the rubber, then releases, sending it soaring back into the air.

5B-WISA generates a video where the rope suspending the wooden seat swinging back and forth in accordance with physical laws. In contrast, CogVideoX-5B produces unstable swing motion; PhyT2V fails to generate the swinging behavior of the wooden seat; and Cosmos generates a physically inconsistent scene where the rope breaks while the wooden seat remains horizontally suspended. In the example on the right, WISA successfully simulates the process of an apple falling into water: the water surface remains calm before the apple enters, splashes form as the apple impacts the water, and the apple experiences buoyant force after submersion. However, CogVideoX-5B generates chaotic water and apple movements, PhyT2V omits the falling process, and Cosmos mistakenly generates two apples at the end. Additional videos generated by CogVideoX-5B-WISA, demonstrating various physical phenomena, are also presented in the Figure 6. All aforementioned videos, along with comparisons on Wan2.1, are provided in the Project Page.

5.3 Ablation Study

We conduct ablation studies on VideoPhy using VideoCon-Physics to verify the effectiveness of key components in our method, as shown in the Table 2. The baseline is CogVideoX-5B. As expected, removing MoPA results in a performance drop due to the absence of qualitative physical information as guidance. Similarly, the inclusion of the Physical Classifier aids the generative model in perceiving and modeling physical properties, thereby enhancing both semantic relevance and consistency with physical laws. Notably, the evaluation model VideoCon-Physics [3] is trained on samples generated

by nine different T2V models, leading to a distribution shift when compared to the real-world videos in WISA-80K. Consequently, relying solely on LoRA yields only limited improvement.

To further investigate the impact of clearly-defined physical phenomena data versus general scene data on physical perception, we fine-tune LoRA on 80,000 videos from an open-source video dataset. This results in only a slight performance decline, indicating that the physically grounded videos in WISA-80K provide substantial value for modeling physical properties.

5.4 User Preference

The physical consistency of generated videos is abstract and difficult to quantify directly. Therefore, we conduct a human evaluation to assess the effectiveness of WISA. Specifically, we selected three representative models for comparison. The evaluation considered two aspects: semantic consistency and physical alignment. Each candidate model is ranked in both aspects, receiving a score based on its ranking: 3 points for first place, 2 points for second, and 0 points for last. We collected preference re-

200

400

600

800

Semantic Consistency

Physical Alignment

User Preference

CogvideoX-5B

Cosmos

PhyT2V

WISA

sults from 100 participants. As shown in the Figure 7 demonstrates that WISA achieves a significant advantage in physical alignment, while also maintaining strong semantic consistency.

5.5 Attention Map Analysis

We further conduct a visual analysis of the Mixture-of-Physical-Experts attention maps, aiming to investigate whether different physical experts focus on the regions corresponding to distinct physical phenomena. As shown in the Figure 8, the rigid body motion expert perfectly focuses on the swing region, while the non-dynamics expert attends to the static background with no apparent motion. This demonstrates that the MoPA effectively models and captures the corresponding physical attributes.

6 Conclusion

In this paper, we present WISA framework, which decomposes physical principles into structured physical information, including textual physical descriptions, qualitative physical categories, and quantitative physical properties. To help T2V models learn these physical aspects effectively, WISA incorporates two key components: the Mixture-of-Physical-Experts Attention and the Physical Classifier. Building on this, we construct WISA-80K, a dataset containing 80,000 video clips that cover 17 physical phenomena across three fundamental categories of physics, providing a high-quality data foundation. Experimental results show that WISA and WISA-80K can effectively help produce videos that

Generated video

Attention map of

'rigid body motion'

expert

Attention map of 'no obvious dynamic phenomenon' expert

better align with real-world physical laws, while the additional computational overhead is under 5%. We hope that WISA can provide valuable insights into the research on building powerful world simulators. We further discuss the limitation of this paper in the Supplementary Material A.2.

References

Niket Agarwal, Arslan Ali, Maciej Bala, Yogesh Balaji, Erik Barker, Tiffany Cai, Prithvijit Chattopadhyay, Yongxin Chen, Yin Cui, Yifan Ding, et al. Cosmos world foundation model platform for physical ai. arXiv preprint arXiv:2501.03575 , 2025.

Luca Savant Aira, Antonio Montanaro, Emanuele Aiello, Diego Valsesia, and Enrico Magli. Motioncraft: Physics-based zero-shot video generation. arXiv preprint arXiv:2405.13557 , 2024.

Hritik Bansal, Zongyu Lin, Tianyi Xie, Zeshun Zong, Michal Yarom, Yonatan Bitton, Chenfanfu Jiang, Yizhou Sun, Kai-Wei Chang, and Aditya Grover. Videophy: Evaluating physical commonsense for video generation. arXiv preprint arXiv:2406.03520 , 2024.

Andreas Blattmann, Tim Dockhorn, Sumith Kulal, Daniel Mendelevitch, Maciej Kilian, Dominik Lorenz, Yam Levi, Zion English, Vikram Voleti, Adam Letts, et al. Stable video diffusion: Scaling latent video diffusion models to large datasets. arXiv preprint arXiv:2311.15127 , 2023.

Andreas Blattmann, Robin Rombach, Huan Ling, Tim Dockhorn, Seung Wook Kim, Sanja Fidler, and Karsten Kreis. Align your latents: High-resolution video synthesis with latent diffusion models. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition , pages 22563-22575, 2023.

Tim Brooks, Bill Peebles, Connor Holmes, Will DePue, Yufei Guo, Li Jing, David Schnurr, Joe Taylor, Troy Luhman, Eric Luhman, Clarence Ng, Ricky Wang, and Aditya Ramesh. Video generation models as world simulators. 2024.

Haoxin Chen, Yong Zhang, Xiaodong Cun, Menghan Xia, Xintao Wang, Chao Weng, and Ying Shan. Videocrafter2: Overcoming data limitations for high-quality video diffusion models. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition , pages 7310-7320, 2024.

Jiasong Feng, Ao Ma, Jing Wang, Bo Cheng, Xiaodan Liang, Dawei Leng, and Yuhui Yin. Fancyvideo: Towards dynamic and consistent video generation via cross-frame textual guidance. arXiv preprint arXiv:2408.08189 , 2024.

Rohit Girdhar, Mannat Singh, Andrew Brown, Quentin Duval, Samaneh Azadi, Sai Saketh Rambhatla, Akbar Shah, Xi Yin, Devi Parikh, and Ishan Misra. Factorizing text-to-video generation by explicit image conditioning. In European Conference on Computer Vision , pages 205-224. Springer, 2024.

Yuwei Guo, Ceyuan Yang, Anyi Rao, Yaohui Wang, Yu Qiao, Dahua Lin, and Bo Dai. Animatediff: Animate your personalized text-to-image diffusion models without specific tuning. arXiv preprint arXiv:2307.04725 , 2023.

Jack Hessel, Ari Holtzman, Maxwell Forbes, Ronan Le Bras, and Yejin Choi. Clipscore: A reference-free evaluation metric for image captioning. arXiv preprint arXiv:2104.08718 , 2021.

Peng Jin, Bo Zhu, Li Yuan, and Shuicheng Yan. Moh: Multi-head attention as mixture-of-head attention. arXiv preprint arXiv:2410.11842 , 2024.

Bingyi Kang, Yang Yue, Rui Lu, Zhijie Lin, Yang Zhao, Kaixin Wang, Gao Huang, and Jiashi Feng. How far is video generation from world model: A physical law perspective. arXiv preprint arXiv:2411.02385 , 2024.

Kuaishou. Kling. https://klingai.kuaishou.com/ , 2024. Accessed: 2024-09-03.

Black Forest Labs. Flux. https://github.com/black-forest-labs/flux , 2024.

Simon Le Cleac'h, Hong-Xing Yu, Michelle Guo, Taylor Howell, Ruohan Gao, Jiajun Wu, Zachary Manchester, and Mac Schwager. Differentiable physics simulation of dynamicsaugmented neural objects. IEEE Robotics and Automation Letters , 8(5):2780-2787, 2023.

Zhimin Li, Jianwei Zhang, Qin Lin, Jiangfeng Xiong, Yanxin Long, Xinchi Deng, Yingfang Zhang, Xingchao Liu, Minbin Huang, Zedong Xiao, et al. Hunyuan-dit: A powerful multi-resolution diffusion transformer with fine-grained chinese understanding. arXiv preprint arXiv:2405.08748 , 2024.

Yaron Lipman, Ricky TQ Chen, Heli Ben-Hamu, Maximilian Nickel, and Matt Le. Flow matching for generative modeling. arXiv preprint arXiv:2210.02747 , 2022.

Shanyuan Liu, Dawei Leng, and Yuhui Yin. Bridge diffusion model: bridge non-english language-native text-to-image diffusion model with english communities. arXiv preprint arXiv:2309.00952 , 2023.

Shaowei Liu, Zhongzheng Ren, Saurabh Gupta, and Shenlong Wang. Physgen: Rigid-body physics-grounded image-to-video generation. In European Conference on Computer Vision , pages 360-378. Springer, 2024.

Guoqing Ma, Haoyang Huang, Kun Yan, Liangyu Chen, Nan Duan, Shengming Yin, Changyi Wan, Ranchen Ming, Xiaoniu Song, Xing Chen, et al. Step-video-t2v technical report: The practice, challenges, and future of video foundation model. arXiv preprint arXiv:2502.10248 , 2025.

Yuhang Ma, Shanyuan Liu, Ao Ma, Xiaoyu Wu, Dawei Leng, and Yuhui Yin. Hico: Hierarchical controllable diffusion model for layout-to-image generation. Advances in Neural Information Processing Systems , 37:128886-128910, 2024.

Fanqing Meng, Jiaqi Liao, Xinyu Tan, Wenqi Shao, Quanfeng Lu, Kaipeng Zhang, Yu Cheng, Dianqi Li, Yu Qiao, and Ping Luo. Towards world simulator: Crafting physical commonsensebased benchmark for video generation. arXiv preprint arXiv:2410.05363 , 2024.

Saman Motamed, Laura Culp, Kevin Swersky, Priyank Jaini, and Robert Geirhos. Do generative video models understand physical principles? arXiv preprint arXiv:2501.09038 , 2025.

Kepan Nan, Rui Xie, Penghao Zhou, Tiehan Fan, Zhenheng Yang, Zhijie Chen, Xiang Li, Jian Yang, and Ying Tai. Openvid-1m: A large-scale high-quality dataset for text-to-video generation. arXiv preprint arXiv:2407.02371 , 2024.

OpenAI. Sora. https://openai.com/ , 2024. Accessed: 2024-09-03.

William Peebles and Saining Xie. Scalable diffusion models with transformers. In Proceedings of the IEEE/CVF International Conference on Computer Vision , pages 4195-4205, 2023.

Dustin Podell, Zion English, Kyle Lacey, Andreas Blattmann, Tim Dockhorn, Jonas Müller, Joe Penna, and Robin Rombach. Sdxl: Improving latent diffusion models for high-resolution image synthesis. arXiv preprint arXiv:2307.01952 , 2023.

PySceneDetect Developers. Pyscenedetect. https://www.scenedetect.com/ .

Carlos Riquelme, Joan Puigcerver, Basil Mustafa, Maxim Neumann, Rodolphe Jenatton, André Susano Pinto, Daniel Keysers, and Neil Houlsby. Scaling vision with sparse mixture of experts. Advances in Neural Information Processing Systems , 34:8583-8595, 2021.

Wan Team. Wan: Open and advanced large-scale video generative models. 2025.

Jiawei Wang, Liping Yuan, Yuchen Zhang, and Haomiao Sun. Tarsier: Recipes for training and evaluating large video description models. arXiv preprint arXiv:2407.00634 , 2024.

Jing Wang, Ao Ma, Jiasong Feng, Dawei Leng, Yuhui Yin, and Xiaodan Liang. Pt-t2i/v: An efficient proxy-tokenized diffusion transformer for text-to-image/video-task. In The Thirteenth International Conference on Learning Representations .

Jiuniu Wang, Hangjie Yuan, Dayou Chen, Yingya Zhang, Xiang Wang, and Shiwei Zhang. Modelscope text-to-video technical report. arXiv preprint arXiv:2308.06571 , 2023.

Peng Wang, Shuai Bai, Sinan Tan, Shijie Wang, Zhihao Fan, Jinze Bai, Keqin Chen, Xuejing Liu, Jialin Wang, Wenbin Ge, et al. Qwen2-vl: Enhancing vision-language model's perception of the world at any resolution. arXiv preprint arXiv:2409.12191 , 2024.

Qiuheng Wang, Yukai Shi, Jiarong Ou, Rui Chen, Ke Lin, Jiahao Wang, Boyuan Jiang, Haotian Yang, Mingwu Zheng, Xin Tao, et al. Koala-36m: A large-scale video dataset improving consistency between fine-grained conditions and video content. arXiv preprint arXiv:2410.08260 , 2024.

Yaohui Wang, Xinyuan Chen, Xin Ma, Shangchen Zhou, Ziqi Huang, Yi Wang, Ceyuan Yang, Yinan He, Jiashuo Yu, Peiqing Yang, et al. Lavie: High-quality video generation with cascaded latent diffusion models. International Journal of Computer Vision , pages 1-20, 2024.

Yi Wang, Kunchang Li, Xinhao Li, Jiashuo Yu, Yinan He, Guo Chen, Baoqi Pei, Rongkun Zheng, Zun Wang, Yansong Shi, et al. Internvideo2: Scaling foundation models for multimodal video understanding. In European Conference on Computer Vision , pages 396-416. Springer, 2024.

Jiannan Xiang, Guangyi Liu, Yi Gu, Qiyue Gao, Yuting Ning, Yuheng Zha, Zeyu Feng, Tianhua Tao, Shibo Hao, Yemin Shi, et al. Pandora: Towards general world model with natural language actions and video states. arXiv preprint arXiv:2406.09455 , 2024.

Qiyao Xue, Xiangyu Yin, Boyuan Yang, and Wei Gao. Phyt2v: Llm-guided iterative selfrefinement for physics-grounded text-to-video generation. arXiv preprint arXiv:2412.00596 , 2024.

Zhuoyi Yang, Jiayan Teng, Wendi Zheng, Ming Ding, Shiyu Huang, Jiazheng Xu, Yuanming Yang, Wenyi Hong, Xiaohan Zhang, Guanyu Feng, et al. Cogvideox: Text-to-video diffusion models with an expert transformer. arXiv preprint arXiv:2408.06072 , 2024.

Zangwei Zheng, Xiangyu Peng, Tianji Yang, Chenhui Shen, Shenggui Li, Hongxin Liu, Yukun Zhou, Tianyi Li, and Yang You. Open-sora: Democratizing efficient video production for all, March 2024.

Zheng Zhu, Xiaofeng Wang, Wangbo Zhao, Chen Min, Nianchen Deng, Min Dou, Yuqi Wang, Botian Shi, Kai Wang, Chi Zhang, et al. Is sora a world simulator? a comprehensive survey on general world models and beyond. arXiv preprint arXiv:2405.03520 , 2024.

NeurIPS Paper Checklist

1. Claims

Question: Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?

Answer: [Yes]

Justification: The abstract and introduction clearly articulate the main contributions of the paper, including the proposed WISA and WISA-80K (3), the design of the Physical Module (4.2), and the overall performance of WISA in experiments (5). These claims are well supported by the methods and experimental results presented in the main paper.

Guidelines:

The answer NA means that the abstract and introduction do not include the claims made in the paper.

The abstract and/or introduction should clearly state the claims made, including the contributions made in the paper and important assumptions and limitations. A No or NA answer to this question will not be perceived well by the reviewers.

The claims made should match theoretical and experimental results, and reflect how much the results can be expected to generalize to other settings.

It is fine to include aspirational goals as motivation as long as it is clear that these goals are not attainable by the paper.

2. Limitations

Question: Does the paper discuss the limitations of the work performed by the authors?

Answer: [Yes]

Justification: The limitations of the proposed method have been clearly discussed in the paper (A.2).

Guidelines:

The answer NA means that the paper has no limitation, while the answer No means that the paper has limitations, but those are not discussed in the paper.

The authors are encouraged to create a separate "Limitations" section in their paper.

The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (e.g., independence assumptions, noiseless settings, model well-specification, asymptotic approximations only holding locally). The authors should reflect on how these assumptions might be violated in practice and what the implications would be.

The authors should reflect on the scope of the claims made, e.g., if the approach was only tested on a few datasets or with a few runs. In general, empirical results often depend on implicit assumptions, which should be articulated.

The authors should reflect on the factors that influence the performance of the approach. For example, a facial recognition algorithm may perform poorly when image resolution is low or images are taken in low lighting. Or a speech-to-text system might not be used reliably to provide closed captions for online lectures because it fails to handle technical jargon.

The authors should discuss the computational efficiency of the proposed algorithms and how they scale with dataset size.

If applicable, the authors should discuss possible limitations of their approach to address problems of privacy and fairness.

While the authors might fear that complete honesty about limitations might be used by reviewers as grounds for rejection, a worse outcome might be that reviewers discover limitations that aren't acknowledged in the paper. The authors should use their best judgment and recognize that individual actions in favor of transparency play an important role in developing norms that preserve the integrity of the community. Reviewers will be specifically instructed to not penalize honesty concerning limitations.

3. Theory assumptions and proofs

Question: For each theoretical result, does the paper provide the full set of assumptions and a complete (and correct) proof?

Answer: [NA]

Justification: This paper does not include theoretical results.

Guidelines:

The answer NA means that the paper does not include theoretical results.

All the theorems, formulas, and proofs in the paper should be numbered and crossreferenced.

All assumptions should be clearly stated or referenced in the statement of any theorems.

The proofs can either appear in the main paper or the supplemental material, but if they appear in the supplemental material, the authors are encouraged to provide a short proof sketch to provide intuition.

Inversely, any informal proof provided in the core of the paper should be complemented by formal proofs provided in appendix or supplemental material.

Theorems and Lemmas that the proof relies upon should be properly referenced.

4. Experimental result reproducibility

Question: Does the paper fully disclose all the information needed to reproduce the main experimental results of the paper to the extent that it affects the main claims and/or conclusions of the paper (regardless of whether the code and data are provided or not)?

Answer: [Yes]

Justification: The paper provides all necessary implementation details, including model architecture, training settings, hyperparameters, and dataset preprocessing steps. We will release the full code and trained models upon publication.

Guidelines:

The answer NA means that the paper does not include experiments.

If the paper includes experiments, a No answer to this question will not be perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data are provided or not.

If the contribution is a dataset and/or model, the authors should describe the steps taken to make their results reproducible or verifiable.

Depending on the contribution, reproducibility can be accomplished in various ways. For example, if the contribution is a novel architecture, describing the architecture fully might suffice, or if the contribution is a specific model and empirical evaluation, it may be necessary to either make it possible for others to replicate the model with the same dataset, or provide access to the model. In general. releasing code and data is often one good way to accomplish this, but reproducibility can also be provided via detailed instructions for how to replicate the results, access to a hosted model (e.g., in the case of a large language model), releasing of a model checkpoint, or other means that are appropriate to the research performed.

While NeurIPS does not require releasing code, the conference does require all submissions to provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution. For example

(a) If the contribution is primarily a new algorithm, the paper should make it clear how to reproduce that algorithm.

(b) If the contribution is primarily a new model architecture, the paper should describe the architecture clearly and fully.

(c) If the contribution is a new model (e.g., a large language model), then there should either be a way to access this model for reproducing the results or a way to reproduce the model (e.g., with an open-source dataset or instructions for how to construct the dataset).

(d) We recognize that reproducibility may be tricky in some cases, in which case authors are welcome to describe the particular way they provide for reproducibility. In the case of closed-source models, it may be that access to the model is limited in some way (e.g., to registered users), but it should be possible for other researchers to have some path to reproducing or verifying the results.

5. Open access to data and code

Question: Does the paper provide open access to the data and code, with sufficient instructions to faithfully reproduce the main experimental results, as described in supplemental material?

Answer: [No]

Justification: To preserve anonymity during the review process, we have not released the code and data yet. However, we commit to open-sourcing both the code and data, along with detailed instructions for reproducing the experiments, upon acceptance.

Guidelines:

The answer NA means that paper does not include experiments requiring code.

Please see the NeurIPS code and data submission guidelines ( https://nips.cc/ public/guides/CodeSubmissionPolicy ) for more details.

While we encourage the release of code and data, we understand that this might not be possible, so 'No' is an acceptable answer. Papers cannot be rejected simply for not including code, unless this is central to the contribution (e.g., for a new open-source benchmark).

The instructions should contain the exact command and environment needed to run to reproduce the results. See the NeurIPS code and data submission guidelines ( https: //nips.cc/public/guides/CodeSubmissionPolicy ) for more details.

The authors should provide instructions on data access and preparation, including how to access the raw data, preprocessed data, intermediate data, and generated data, etc.

The authors should provide scripts to reproduce all experimental results for the new proposed method and baselines. If only a subset of experiments are reproducible, they should state which ones are omitted from the script and why.

At submission time, to preserve anonymity, the authors should release anonymized versions (if applicable).

Providing as much information as possible in supplemental material (appended to the paper) is recommended, but including URLs to data and code is permitted.

6. Experimental setting/details

Question: Does the paper specify all the training and test details (e.g., data splits, hyperparameters, how they were chosen, type of optimizer, etc.) necessary to understand the results?

Answer: [Yes]

Justification: We provide all relevant experimental details in Supplementary Material A.3. These details ensure the clarity and reproducibility of our experimental results.

Guidelines:

The answer NA means that the paper does not include experiments.

The experimental setting should be presented in the core of the paper to a level of detail that is necessary to appreciate the results and make sense of them.

The full details can be provided either with the code, in appendix, or as supplemental material.

7. Experiment statistical significance

Question: Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?

Answer: [No]

Justification: While we do not report explicit error bars or confidence intervals, we follow the standard evaluation protocols from prior related work to ensure fair comparisons.

Guidelines:

The answer NA means that the paper does not include experiments.

The authors should answer "Yes" if the results are accompanied by error bars, confidence intervals, or statistical significance tests, at least for the experiments that support the main claims of the paper.

The factors of variability that the error bars are capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, or overall run with given experimental conditions).

The method for calculating the error bars should be explained (closed form formula, call to a library function, bootstrap, etc.)

The assumptions made should be given (e.g., Normally distributed errors).

It should be clear whether the error bar is the standard deviation or the standard error of the mean.

It is OK to report 1-sigma error bars, but one should state it. The authors should preferably report a 2-sigma error bar than state that they have a 96% CI, if the hypothesis of Normality of errors is not verified.

For asymmetric distributions, the authors should be careful not to show in tables or figures symmetric error bars that would yield results that are out of range (e.g. negative error rates).

If error bars are reported in tables or plots, The authors should explain in the text how they were calculated and reference the corresponding figures or tables in the text.

8. Experiments compute resources

Question: For each experiment, does the paper provide sufficient information on the computer resources (type of compute workers, memory, time of execution) needed to reproduce the experiments?

Answer: [Yes]

Justification: We provide detailed information about the compute resources used for our experiments, including the type of GPUs and other relevant training settings ( A.3). These details are included to support reproducibility.

Guidelines:

The answer NA means that the paper does not include experiments.

The paper should indicate the type of compute workers CPU or GPU, internal cluster, or cloud provider, including relevant memory and storage.

The paper should provide the amount of compute required for each of the individual experimental runs as well as estimate the total compute.

The paper should disclose whether the full research project required more compute than the experiments reported in the paper (e.g., preliminary or failed experiments that didn't make it into the paper).

9. Code of ethics

Question: Does the research conducted in the paper conform, in every respect, with the NeurIPS Code of Ethics https://neurips.cc/public/EthicsGuidelines ?

Answer: [Yes]

Justification: The research conducted in this paper fully adheres to the NeurIPS Code of Ethics. We have ensured that all ethical guidelines regarding data usage, privacy, and research integrity have been strictly followed.

Guidelines:

The answer NA means that the authors have not reviewed the NeurIPS Code of Ethics.

If the authors answer No, they should explain the special circumstances that require a deviation from the Code of Ethics.

The authors should make sure to preserve anonymity (e.g., if there is a special consideration due to laws or regulations in their jurisdiction).

10. Broader impacts

Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?

Answer: [No]

Justification: The paper does not explicitly discuss societal impacts. However, since the work focuses on text-to-video generation, we acknowledge potential misuse, such as generating misleading or harmful content. We plan to include safeguards and responsible usage guidelines when releasing models to mitigate such risks.

Guidelines:

The answer NA means that there is no societal impact of the work performed.

If the authors answer NA or No, they should explain why their work has no societal impact or why the paper does not address societal impact.

Examples of negative societal impacts include potential malicious or unintended uses (e.g., disinformation, generating fake profiles, surveillance), fairness considerations (e.g., deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations.

The conference expects that many papers will be foundational research and not tied to particular applications, let alone deployments. However, if there is a direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that an improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point out that a generic algorithm for optimizing neural networks could enable people to train models that generate Deepfakes faster.

The authors should consider possible harms that could arise when the technology is being used as intended and functioning correctly, harms that could arise when the technology is being used as intended but gives incorrect results, and harms following from (intentional or unintentional) misuse of the technology.

If there are negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition to attacks, mechanisms for monitoring misuse, mechanisms to monitor how a system learns from feedback over time, improving the efficiency and accessibility of ML).

11. Safeguards

Question: Does the paper describe safeguards that have been put in place for responsible release of data or models that have a high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)?

Answer: [Yes]

Justification: Since our work focuses on text-to-video generation, we recognize its potential for misuse, such as in generating misleading or fake images. To mitigate this, we intend to release our model under a research-only license with clear usage restrictions. The dataset used is public and curated to avoid harmful content.

Guidelines:

The answer NA means that the paper poses no such risks.

Released models that have a high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere to usage guidelines or restrictions to access the model or implementing safety filters.

Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images.

We recognize that providing effective safeguards is challenging, and many papers do not require this, but we encourage authors to take this into account and make a best faith effort.

12. Licenses for existing assets

Question: Are the creators or original owners of assets (e.g., code, data, models), used in the paper, properly credited and are the license and terms of use explicitly mentioned and properly respected?

Answer: [Yes]

Justification: We use several publicly available datasets and pretrained models (e.g., CogVideoX, Wan2.1) in our experiments. All datasets and models are properly cited in the main text and/or appendix. We ensured that their licenses are respected, and no data under restrictive or ambiguous licenses was used.

Guidelines:

The answer NA means that the paper does not use existing assets.

The authors should cite the original paper that produced the code package or dataset.

The authors should state which version of the asset is used and, if possible, include a URL.

The name of the license (e.g., CC-BY 4.0) should be included for each asset.

For scraped data from a particular source (e.g., website), the copyright and terms of service of that source should be provided.

If assets are released, the license, copyright information, and terms of use in the package should be provided. For popular datasets, paperswithcode.com/datasets has curated licenses for some datasets. Their licensing guide can help determine the license of a dataset.

For existing datasets that are re-packaged, both the original license and the license of the derived asset (if it has changed) should be provided.

If this information is not available online, the authors are encouraged to reach out to the asset's creators.

13. New assets

Question: Are new assets introduced in the paper well documented and is the documentation provided alongside the assets?

Answer: [No]

Justification: While we plan to release the code and model after the review process, at the time of submission no new assets are publicly released. Upon release, we will ensure that comprehensive documentation, licensing terms, and usage guidelines are provided in accordance with the NeurIPS guidelines.

Guidelines:

The answer NA means that the paper does not release new assets.

Researchers should communicate the details of the dataset/code/model as part of their submissions via structured templates. This includes details about training, license, limitations, etc.

The paper should discuss whether and how consent was obtained from people whose asset is used.

At submission time, remember to anonymize your assets (if applicable). You can either create an anonymized URL or include an anonymized zip file.

14. Crowdsourcing and research with human subjects

Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given to participants and screenshots, if applicable, as well as details about compensation (if any)?

Answer: [NA]

Justification: The paper does not involve any crowdsourcing or research with human subjects.

Guidelines:

The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.

Including this information in the supplemental material is fine, but if the main contribution of the paper involves human subjects, then as much detail as possible should be included in the main paper.

According to the NeurIPS Code of Ethics, workers involved in data collection, curation, or other labor should be paid at least the minimum wage in the country of the data collector.

15. Institutional review board (IRB) approvals or equivalent for research with human subjects

Question: Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or an equivalent approval/review based on the requirements of your country or institution) were obtained?

Answer: [NA]

Justification: The paper does not involve crowdsourcing or research with human subjects, so IRB approval is not applicable.

Guidelines:

The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.

Depending on the country in which research is conducted, IRB approval (or equivalent) may be required for any human subjects research. If you obtained IRB approval, you should clearly state this in the paper.

We recognize that the procedures for this may vary significantly between institutions and locations, and we expect authors to adhere to the NeurIPS Code of Ethics and the guidelines for their institution.

For initial submissions, do not include any information that would break anonymity (if applicable), such as the institution conducting the review.

16. Declaration of LLM usage

Question: Does the paper describe the usage of LLMs if it is an important, original, or non-standard component of the core methods in this research? Note that if the LLM is used only for writing, editing, or formatting purposes and does not impact the core methodology, scientific rigorousness, or originality of the research, declaration is not required.

Answer: [NA]

Justification: LLMs were only used for writing and editing assistance. They were not involved in any part of the core methodology, experimental design, or scientific contribution.

Guidelines:

The answer NA means that the core method development in this research does not involve LLMs as any important, original, or non-standard components.

Please refer to our LLM policy ( https://neurips.cc/Conferences/2025/LLM ) for what should or should not be described.

A Technical Appendices and Supplementary Material

A.1 Broader Impacts and Ethical Considerations

The development of powerful generative models, particularly those capable of creating realistic video content, carries a significant responsibility. The work presented in this paper, while aimed at advancing scientific understanding and technical capability in physics-aware video generation, is not exempt from potential societal impacts and ethical challenges. This section transparently discusses these issues and outlines the steps we are taking to mitigate potential harms.

Potential for Misuse and Societal Impact: We recognize the dual-use nature of our work. The same technology that can be used for creative applications, scientific simulation, or special effects could also be exploited to create compelling but fabricated content for malicious purposes. The enhanced physical realism of videos generated by a model like WISA could amplify the verisimilitude of synthetic media, increasing its potential to be used for misinformation, disinformation (e.g., "deepfakes"), or propaganda. Such content could erode public trust, be used as false evidence, or create realistic depictions of accidents or violence to incite fear.

Dataset Ethics and Release Strategy: The creation and distribution of any large-scale dataset require careful consideration of privacy, copyright, and consent. The WISA-80K dataset is constructed using video clips from publicly available channels on YouTube.

Copyright and Terms of Service: To respect the rights of content creators and to comply with platform Terms of Service, we have adopted a metadata-only release strategy. We will not host or distribute any video files, clips, or raw pixel data. The released dataset will contain only the public YouTube video IDs, the relevant start and end timestamps of the physical phenomena, and our corresponding physical annotations. This is a standard and widely accepted practice in the research community (e.g., AudioSet) that enables reproducible research while avoiding copyright infringement.

Responsible Data Access: To further ensure responsible use, the WISA-80K metadata will not be available for direct public download. Instead, we will implement a gated access mechanism. Researchers wishing to use the dataset must submit a request outlining their institutional affiliation and research purpose. They will be required to agree to a Data Usage Agreement (DUA), which will stipulate that: (1) the dataset is to be used for non-commercial research purposes only; (2) users are responsible for their own adherence to YouTube's Terms of Service when accessing the original videos; and (3) the metadata and any derived video content may not be redistributed.

Content Creator Rights: We will provide a clear and accessible opt-out mechanism on our future project page. Any content creator can request the removal of their video's ID from our dataset at any time, and we are committed to promptly honoring all such requests.

Annotation Process: Quality and Bias: The physical annotations in WISA-80K were generated using a large language model (GPT-4o-mini). We acknowledge that this automated process may introduce biases or factual inaccuracies ("hallucinations").

Human Validation: To quantify the quality and reliability of these annotations, we conducted a human validation study on a randomly sampled subset of 500 videos. The results (detailed in Appendix X) indicate a high degree of quality, with 95% satisfaction for textual descriptions and 86% satisfaction for the plausibility of quantitative estimates. The accuracy for qualitative category classification was 76% against human-assigned labels.

Dataset Positioning: These results confirm that while some label noise is inherent, the annotations provide a strong and reliable learning signal. Nonetheless, WISA-80K should be understood as a large-scale, weakly-supervised dataset rather than a gold-standard, error-free resource. We encourage future work to further refine and build upon these annotations.

Mitigation Strategies for the Generative Model: To address the risks of misuse associated with the generative model itself, we commit to the following safeguards for any future public release:

Visible Watermarking: All video outputs generated by our released model will be programmatically embedded with a clear and persistent visible watermark to identify them as synthetic.

Responsible AI License: We plan to release the model under a Responsible AI License (e.g., a CreativeML Open RAIL-M license). Such licenses contractually prohibit users from employing the model for malicious, deceptive, illegal, or unethical purposes, including the generation of harmful misinformation.

A.2 Limitation

Although our approach significantly improves the ability of existing T2V models to generate videos that align with real-world physical laws, it still has the following limitations: 1) Limited physical categories : We collect 80,000 videos in WISA-80K, covering 17 types of physical phenomena. However, due to constraints in time and manpower, the dataset does not include all physical phenomena encountered in the real world, such as corrosion or vacuum environments. 2) Limited physical information guidance : WISA primarily provides high-level semantic guidance and lacks detailed constraints at the physical mechanism level (e.g., energy conservation, Newton's laws). However, introducing more detailed physical principle constraints currently requires modeling object motion based on image or 3D information, which suffers from poor generalization and can only handle limited categories and scenarios. How to incorporate physical principle constraints into text-to-video generation while maintaining generalization remains an area worth further research. 3) Expanding WISA to Unseen Classes : We conducted additional qualitative evaluations on unseen physical categories such as corrosion and electromagnetism, as shown in Figure 9. As expected, both our WISA-enhanced model and the base model struggle to generate physically plausible videos for these categories. This limitation primarily stems from the absence of relevant concepts and visual examples in the WISA-80K training set, underscoring the current challenge of generalizing to entirely out-of-distribution physical phenomena. However, we argue that WISA's modular design, particularly the Mixture-of-Physical-Experts Attention (MoPA) mechanism, makes it inherently more scalable than monolithic architectures. When introducing a new physical category, traditional fine-tuning methods (e.g., applying LoRA on the base model) typically require retraining large portions of the network to assimilate the new knowledge. In contrast, MoPA allows for a more targeted and efficient expansion: we can simply add and train a new expert head dedicated to the novel phenomenon while keeping the existing experts largely frozen, thus preserving prior knowledge and facilitating incremental learning.

Appleslowlyoxidizesandcorrodesintheair

Touchthedoorhandleinwintertodischargestaticelectricity.

Baseline

Wan2.1-WISA

Further work: In future research, we plan to explore physically consistent generation for the image-to-video (I2V) task. In principle, WISA can be applied to I2V as well. The main distinction between I2V and text-to-video (T2V) lies in the input: I2V models receive a clean latent code for the first frame, whereas the rest of the input text and model architecture remain the same. With explicit initial frame information, the I2V model primarily learns to infer subsequent states, generating intermediate and final frames that follow plausible physical processes.

Compared with T2V, I2V is particularly relevant in embodied intelligence and robotics, where agents often observe a concrete initial state and must predict or plan the outcomes of actions in a physically consistent way. While T2V remains a more challenging task-requiring the model to imagine the entire physical process from scratch based solely on abstract text-the I2V task provides a complementary and practically important setting to study and improve physical consistency in video generation. We aim to extend WISA to I2V in future work, leveraging its modular design to generate videos that maintain accurate physical dynamics starting from a known initial state.

A.3 Training and Evaluation Detail

Training: We choose two representative open-source T2V models-CogVideoX-5B and Wan2.114B-as the base models to validate the effectiveness of the proposed WISA. WISA is trained on our constructed WISA-80K dataset for 8,000 steps, using a learning rate of 2e-5 and a batch size of 16. For CogVideoX-5B, the video resolution is set to 480×720 with 49 frames per video, while for Wan2.1-14B, the resolution is 480×832 with 81 frames. We adopt LoRA with a rank of 128 and an alpha of 16. During training, only the physical module, physical classifier, and LoRA parameters are updated, resulting in a total of 187 million learnable parameters for CogVideoX-5B and 587 million for Wan2.1-14B. All experiments are conducted on 8 A100 GPUs, each equipped with 80 GB of memory.

Evaluation: VideoCon-Physics was trained by collecting videos generated from nine different models, which were manually annotated for adherence to real-world physical laws and strong semantic consistency. Using this data, a vision-language model (VLM) was fine-tuned to serve as a reward model. During inference, the generated video and corresponding text prompt are fed into VideoCon-Physics, which outputs scores ranging from 0 to 1 for both metrics.

A.4 Inference without Annotation

For any given user prompt, we use a large language model (GPT-4o) with a set of predefined instructions to generate the required physical annotations (textual description, qualitative categories, quantitative properties). Crucially, this process uses only the input text prompt, with no access to any visual information, thus preventing any unfair information leakage. The cost is also minimal (approx. 2000 tokens per prompt). The detailed instructions are provided in the Figure 12, Figure 13, and Figure 14.

A.5 The Definition of Physical Categories

We define a total of 29 qualitative physical categories, organized into 5 major classes. The physical categories within each class, along with their corresponding category IDs, are listed as follows:

Dynamics : 1. Collision , 2. Rigid Body Motion , 3. Elastic Motion , 4. Liquid Motion , 5. Gas Motion , 6. Deformation , and 7. No obvious dynamic phenomenon

Thermodynamics : 8. Melting , 9. Solidification , 10. Vaporization , 11. Liquefaction , 12. Explosion , 13. Combustion and 14. No obvious thermodynamic phenomenon

Optics : 15. Reflection , 16. Refraction , 17. Scattering , 18. Interference and Diffraction , 19. Unnatural Light Sources , and 20. No obvious optical phenomenon

Camera motion : 21. Yes , 22. No

The state of object : 23. Liquids Objects Appearance , 24. Solid Objects Appearance , 25. Gas Objects Appearance , 26. Object decomposition and splitting , 27. Mixing of Multiple Objects , 28. Object Disappearance and 29. No Change

Specifically, Liquids objects appearance : new liquids appear from the camera over time and due to external forces, such as water squeezed out of a towel. Solid objects appearance : new solids appear from the camera over time and due to external forces, such as Chemical reaction that produces precipitates, or cars drive in from outside the camera. Gas objects appearance : new gas appears from the camera over time and due to external forces. Object decomposition and splitting : Over time and under the action of external forces, an object is broken into multiple sub-parts: such as fruits and vegetables being cut in half. Mixing of multiple objects : Over time and with the action of external forces, two objects of the same state mix together, such as two solutions mixing. Object disappearance : As time passes and external forces act, objects disappear from the camera. No change : No change in the state of the object

A.6 Dataset Property Analysis

We visualize the distribution of different physics categories and video frame counts in WISA80K, as shown in the paper Figure 1. Dynamics frequently occur in daily life, accounting for the

Consistency

Inconsistency

Optic

77%

23%

Thermodynamic

64%

36%

Dynamic

82%

18%

0

20

40

60

80

100

largest proportion at 47%. Optics and thermodynamics, which typically require specific temperature or environmental conditions, account for 29% and 24%, respectively. The proportions of each subcategory are shown in the outer ring of the Figure Based on the labels of the manually collected videos, we evaluate the accuracy of the qualitative physical category annotations. The results are shown in the Figure 10, where the accuracy for dynamics, optics, and thermodynamics reaches 84%, 71%, and 64%, respectively, with an overall accuracy of 75%.

A.7 More Examples and Annotation

Following the proposed physical information annotation pipeline, we construct the WISA-80K dataset. Several example videos and their corresponding annotations are shown in the Figure 11. This pipeline enables accurate and detailed annotation of physical information, ensuring that each video is comprehensively labeled with its relevant physical properties and phenomena.

A.8 Annotation Prompts

The detailed prompt used for physical information annotation is illustrated in the Figure 12, Figure 13, and Figure 14.

A.9 Word Cloud

We conducted a word frequency analysis on the textual physical description in the dataset and generated the word cloud shown in Figure 15. To filter out irrelevant words, we retained only nouns and selected them based on their frequency, from highest to lowest. Notably, physical terms such as 'motion,' 'phenomenon,' and 'light' appear more frequently, highlighting the strong physical relevance of the dataset.

A.10 Discussion of Quantitative Evaluation

During the quantitative evaluation, we observe several misjudgments in VideoCon-Physics, as shown in the Figure 16. Specifically, WISA generates a physically plausible process where the object enters the water first, followed by the splash, aligning well with real-world physical laws. However, this sample only receives a low score of 0.08 from VideoCon-Physics. We further conduct a simple test using Qwen2.5-VL for evaluation, and the model also struggles to distinguish the correct or incorrect sequence of physical events. These findings show the limitations of existing video-based physics evaluation metrics, indicating that future research into more reliable physical property assessments for videos is necessary.

Captions

: The video depicts a close-up view of a mechanical device in

operation. The device appears to be a type of press or hydraulic machine,

characterized by its metallic components and the presence of a yellow and

black striped cylinder at the top. The cylinder has a warning label, indicating

caution or danger, which is typical for industrial machinery. The machine is in

the process of compressing or forming a metal component, as evidenced by

the movement of the upper part of the device, which is descending and

exerting pressure on the lower part. The lower part of the machine consists of

a circular base and a cylindrical component that is being compressed. The

base has a handle attached to it, which is likely used for manual operation or

adjustment. The metal components show signs of wear and use, with visible

scratches and marks, suggesting that this is a well-used piece of equipment.

The background is blurred, focusing attention on the machine, but it appears

to be an industrial setting, possibly a workshop or a factory floor.

Textual physical description

: when generating a video of the described

mechanical device, it is crucial to consider the principle of force and pressure,

particularly how hydraulic systems operate based on pascal's principle, which

states that pressure applied to a confined fluid is transmitted undiminished in

all directions. additionally, attention should be paid to the physical

phenomenon of deformation, as the metal component being compressed will

experience changes in shape and possibly yield stress, reflecting the material's

properties under load.

Qualitative physics categories

:{

Dynamics: collision, deformation,

Thermodynamics: no obvious thermodynamic phenomenon,

Optics: no obvious optical phenomenon,

Camera motion: no,

The state of object: 1. no change 2. object decomposition and splitting (if

the metal component is being cut or broken) 3. solid objects appearance (if

new metal components are formed during the process),

}

Quantitative physical properties

: {

Density: mechanical device: 7.5 to 8.0 g/cm³

cylindrical component: 7.5

to 8.0 g/cm³

,

Time: 0 to 5 seconds,

Temperature: 20 to 100 degrees celsius,

}

Captions

: The video begins with a serene forest scene, showing a dirt path

winding through a dense area of trees. The trees are tall and green, indicating

a healthy forest environment. The path is flanked by the trunks of these trees,

and the ground is covered with a layer of fallen leaves and small plants. The

sky is not visible, suggesting that the camera is focused on the ground level.

As the video progresses, there is a sudden and dramatic change in the scene.

A large explosion occurs, sending a massive cloud of smoke and debris into

the air. The smoke is thick and billows upwards, obscuring the view of the

forest and the path. The explosion creates a bright flash of light, which is

visible even through the smoke. The force of the explosion is so intense that it

appears to shake the camera, causing it to vibrate slightly. The explosion is

the focal point of the video, and it dominates the scene. The smoke and debris

are the only visible elements.

Textual physical description

: when generating a video of an explosion in a

forest scene, it's crucial to consider the principles of conservation of

momentum and energy, as well as the behavior of gases and smoke in

response to rapid changes in pressure. the explosion should realistically

demonstrate how the shockwave propagates through the air, causing nearby

objects to react (e.g., trees swaying or debris being displaced), and how the

smoke rises and expands due to the hot gases produced, following the laws of

fluid dynamics.

Qualitative physics categories

:{

Dynamics: collision, gas motion, deformation,

Thermodynamics: explosion,

Optics: scattering, unnatural light source,

Camera motion: yes,

The state of object: 1. gas objects appearance 2. object decomposition and

splitting 3. object disappearance,

}

Quantitative physical properties

: {

Density: smoke: 0.001 to 0.01 g/cm³

debris: 1 to 2.5 g/cm³

,

Time: occur rapidly after the explosion. the main physical phenomena,

including the explosion and the subsequent rise of smoke and debris, would

typically take place within a very short time frame. \n\nbased on the

description, the explosion itself and the immediate effects would likely occur

within: 0 to 5 seconds.,

Temperature: 500 to 1000 degrees celsius,

}

"model": 'GPT

-4o-

mini'

content: "You are

a physics expert

. Now you want to help the generative model generate videos based on

text

descriptions.

You

need

to

provide

some physics

knowledge

to

make the generated

videos more

consistent with the laws of physics. What is the most important physical principle and the special physical

phenomena to be aware of when generating a video based on the following sentence? Please explain briefly

with one or two sentence."

content: caption

content: "You are a physicist with expertise in material properties and fluid mechanics. Your task is to

assist a generative model in creating videos that align with the laws of physics based on text descriptions.

Carefully analyze the text and provide a single estimated density value or range for each main moving

object described. Express the density value or range for each object in the unit g/cm³. Ensure that each

object is listed only once, with its corresponding density value or range. If no density information can be

estimated for an object, omit it from the output. Present the results in the format: Object: XXX to XXX

g/cm³ (for a range) or Object: XXX g/cm³ (for a single value), separated by newlines for each object.

'

content: "You are a physicist with expertise in dynamics and time-related physical processes. Your task is

to

assist

a

generative

model

in

creating

videos

that

align

with

the

laws

of

physics

based

on

text

descriptions.

Carefully

analyze

the

text

and estimate

the

time

range

during

which the

main

physical

phenomena occur. Only output the time range in the format XXX to XXX seconds."

content: "You are a physicist with expertise in thermodynamics and temperature-related phenomena. Your

task is to assist a generative model in creating videos that align with the laws of physics based on text

descriptions.

Carefully

analyze

the

text

and

estimate

the

temperature

range

during

which

the

main

physical phenomena occur, expressed in degrees Celsius (e.g., '100 to 200'). Only output the temperature

"role": 'system'

"role": "user"

content: caption

content: caption

content: caption

Textual physical description

Quantitative Physical Properties

Qualitative physics categories

"model": 'GPT

-4o-

mini'

content:

"You

are

a

physicist

with

expertise

in

classical and

modern dynamics.

Carefully

analyze

the

following text and determine which of the following dynamic phenomena are most likely represented in the

described scene. Please base your judgment on the principles of motion, force interactions, and material

behavior. Select one or more options from the list below and provide your answer by outputting only the

corresponding names, separated by commas (if multiple apply):\

Collision (e.g., objects impacting and exchanging momentum)

Rigid body motion (e.g., rotation or translation without deformation)

Elastic motion (e.g., oscillations, vibrations, or stretching and compressing of elastic materials)

Liquid motion (e.g., flow of liquids or interactions with liquids)

Gas motion (e.g., expansion, compression, or flow of gases)

Deformation (e.g., structural deformation under applied pressure)

No obvious dynamic phenomenon."

content: caption

content: "You are a physicist with expertise in thermodynamics. Your task is to carefully analyze the

following

text

and

determine

which

thermodynamic

phenomena

are

most

likely

represented

in

the

described scene. Use the principles of energy transfer, phase transitions, and heat-related processes to

guide your judgment. Below are detailed explanations of the thermodynamic phenomena to help you make

an accurate assessment:\

Melting: A solid turns into a liquid due to heat being absorbed. Examples include ice melting into water or

metal melting in a furnace.

Solidification: A liquid turns into a solid as heat is removed. Examples include water freezing into ice or

molten metal solidifying when cooled.

Vaporization: A liquid transforms into a gas through boiling or evaporation. Examples include water boiling

into steam or alcohol evaporating into vapor.

Liquefaction: A gas transforms into a liquid due to cooling or pressure increase. Examples include water

vapor condensing into liquid or liquefied natural gas formation.

Explosion: A sudden and rapid release of energy causes a violent expansion. Examples include chemical

explosions, gas detonations, or bursting pressurized containers.

Combustion: An exothermic chemical reaction, typically involving fuel and oxygen, resulting in heat and

light. Examples include burning wood, gasoline, or natural gas.

No obvious thermodynamic phenomenon: The described scene does not exhibit any distinct thermodynamic

process.

Carefully evaluate the text and select one or more phenomena from the above list based on the scene's

description. Provide your answer by outputting only the corresponding names (e.g., "Melting, Combustion")

separated

by commas if multiple

phenomena apply.

If

none

apply,

output

"No obvious thermodynamic

phenomenon."

"role": 'system'

"role": "user"

content: caption

Qualitative physics categories

"model": 'GPT

-4o-

mini'

content: You are a physicist with expertise in optics. Your task is to analyze the following text carefully

and determine which of the listed optical phenomena are most likely represented based solely on the clear

and explicit descriptions provided in the text. Base your judgment strictly on principles of light behavior,

wave properties, and interactions with matter. Avoid making assumptions about phenomena not clearly

implied

or

described.

Select

one

or

more

options

from

the

list

below

and

provide

your

answer

by

outputting only the corresponding names, separated by commas (if multiple apply):\

Reflection (e.g., light bouncing off a surface)

Refraction (e.g., light bending as it passes through a medium)

Scattering (e.g., light dispersed in various directions, such as in fog or smoke)

Interference and diffraction (e.g., light waves overlapping, forming patterns or bending around obstacles)

Unnatural light source (e.g., artificial or unexpected light emissions, such as lasers or LEDs)

No obvious optical phenomenon.

content: caption

content:

"You

are

a

cinematographer

and

physics

expert.

Your

task

is

to

analyze

the

provided

text

description and determine whether the camera is in motion within the scene. Consider factors such as

panning,

tilting, tracking, or any other type of camera movement. If the text implies camera motion,

output 'Yes' If the text indicates a stationary camera or lacks information about camera motion, output

'No' Only output 'Yes' or

'No'

."

"role": 'system'

"role": "user"

content: caption

content: "You are an ordinary residents. Please provide the possible phenomena's name in the following 7

types

(i.e.,

liquids

objects

appearance;

solid

objects

appearance;

object

decomposition

and

splitting;

mixing of multiple objects, object disappearance, no change) without any explanation.

The types of phenomena:\

liquids objects appearance: new liquids appear from the camera over time and due to external forces, such

as water squeezed out of a towel.

solid objects appearance: new solids appear from the camera over time and due to external forces, such

as Chemical reaction produces precipitate or car drives in from outside the camera.

Gas objects appearance: new gas appear from the camera over time and due to external forces.

Object decomposition

and splitting:

Over time and under the action of external forces, an

object

is

broken into multiple sub-parts: such as fruits and vegetables being cut in half.

Mixing of multiple objects: Over time and with the action of external forces, two objects of the same

state mix together, such as two solutions mixing.

Object disappearance: As time passes and external forces act, objects disappear from the camera.

No change: No change in the state of the object."

content: Which of the above phenomena are most likely to occur in the text description: {caption}

pieces

E

emission

distortions

br

symmetry

potential descent

directions

Tor

coffee

torque

manner

body

expa

interacts

waveprocess

resistance

reaction

impact visibility

W.material

pinb!

spinb!!

optics

illumination

inertia

propagation

andscape

flame

ch

ground

change

forces

C

relation

trajec

ectory

clouds

lift

surface

fuel

ball

colors

mat

wayS

flames?

dynamics

combustion

glass

description

p

attention

properties

pres

balance

oxy

/gen

splash

focus

mass

intensity

e

law

timeanos

account

fireworks

thermodynamics

deformation

raindrops

perspective

waves

transfer

patterns

concept

mist

energy

notion

3

tree

world

smoke

man

respo

ripples

Wi

hd

strt

river

per

illia

abil

rson

int

convection

P

ate

fish

balls

hs

gase

lights

reflections

rocks

rrents

sen

moveme

night

waterfall

es

rayleig

ngths

experience velocitysubstance

flickering

textures

sunlight

shadows

soap

laws

toy

lake

momentum

point

moves

objectbuoyancy

fire

glow

Mofoliage

oosunrise

mixing

transitiontransitions

chemical

gravity

color

acceleration

steam

interference

realism

positions

density

incidence

rategrassgemstones

difference

sand medium

speed

leadde

position

diffusion

portrayal

bubble

ay

structure

hues

conditions

interplay

scatter

base

table

spheres

beads

scenery

diffraction

plate

height

production

branches bowl

components performance

conversion

iridescent

stream

stillness explosions

CogVideoX-5B

WISA

0.05 <

0.5:

Violating the laws of physics

Violating the laws of physics

0.08 <

0.5:

Violating the laws of physics

Consistent with physical laws

0.29 <

0.5:

Violating the laws of physics

Consistent with physical laws

0.78 >

0.5:

Consistent with physical laws

Consistent with physical laws

三

Peer reviews:
{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper considers the problem of physics-obeying t2v generation. Three main contributions:\n1. *(Modeling improvement)* the paper proposes \"WISA\", which is a collection of modeling changes (e.g. trainable adapters) that bake physics-obeying inductive biases into pretrained video generative models through fine-tuning\n2. *(Dataset improvement)* the paper proposes a video dataset, \"WISA-80k\", with videos of various physical phenomena with detailed annotations\n3. *(Combining modeling improvement with dataset improvement)* the authors find that training CogVideoX and Wan2.1 on the WISA-80k dataset, using the WISA framework, improves physics-obeying video generation performance\n\nSTRENGTHS AND WEAKNESSES:\nStrengths:\n1. The teaser figure is very informative and clear. It makes it very clear what the dataset contribution is.\n2. The motivation in the introduction is written+explained very well and is easy to follow.\n3. In Section 3.2, the paper makes good use of examples to make clear what is being constructed.\n4. The paper considers a very important problem (making video generative models better obey physical laws) and it proposes a new labeled dataset that will likely be useful for the community for building better such models.\n5. The description of section 4.2, the physical model, was very easy to understand.\n6. The WISA framework improves performance for CogVideoX and Wan2.1. \n7. The attention map analysis for the swing is very strong, I love that section and that figure.\n\nWeaknesses:\n1. The naming of things is confusing and unintuitive at times. For example, referring to the proposed framework of modules as a \"Simulator Assistant\" doesn't sound quite right to me, because the proposal involves network modules that must be integrated into the base video model and trained from scratch. Additionally, referring to the dataset as \"WISA-80k\", next to the framework of modules which is also titled \"WISA\", seems misleading; these are two completely different things--one is a new dataset, and other is a neural network. It seems like a non-standard/confusing way of naming things.\n2. I think it's very important to have a discussion of related works in the body of the paper, but the current paper has left all of the related works discussion to the appendix. I might suggest moving some other content to the appendix, moving some related works discussion to the body of the paper... related work is crucial to have in the part of the paper that most people will read\n3. I had a hard time understanding Section 4.3, the Physical Classifier. Can you please explain in more detail A) what is the input to this model at train/inference, B) what is the output to this model at train/inference, and C) what it is \"classifying\". \n4. It is unclear from the experiment design which of the three constituent parts of the WISA model adapters/additions are responsible for the improved performance (see Questions for more details)\n5. In Section 5.2, the qualitative comparison, the paper discusses how the WISA-based model does a better job erasing pencil marks. After watching the video, I disagree; it looks like the pencil marks disappear before the eraser even touches them, and they also disappear all at once. I might suggest highlighting this as a failure case, and removing the screenshots from the paper, because the screenshots and descriptions misleadingly imply that this is a success case. (Also a minor typo--line 265, it says \"CogVideoX-5B fails to generate any pencil marks\", when I think it should say \"fails to erase any pencil marks\"). In contrast, the apple example is excellent, that one should be front and center.\n6. I might suggest referring to the models as \"CogVideoX-5B-WISA\" and \"Wan2.1-WISA\" in the paper. (This is already done in Table 1, which is great but should be done in the text as well, in my opinion). It it very important to know which is the base model. (If the paper only had one base model, it wouldn't matter as much... )\n7. Overall, the paper is vague about some important experimental details (see Questions below)\n\nQUESTIONS:\n1. The paper says (approx line 126) *\"we manually collected videos from the Internet that clearly demonstrate the corresponding behavior, without relying on any existing video datasets for filtering or selection\"*. Where on the internet were the videos gathered from? Can you please provide more details about this data gathering procedure.\n2. As I understand it, WISA includes three different ways of baking physical inductive biases into the model: 1) appending \"textual physical descriptions\" to the end of the text prompt; 2) using a Mixture of Physical Experts Attention head to encode qualitative physical attributes; 3) using AdaLN to encode numerical physical attributes into the model. My first question is: is that understanding correct? And my second question is: to what extent can each of the three of these be used without the other two, and still improve performance? E.g. perhaps you could append the textual physical descriptions, but not do either of 2 or 3, and that would give the same performance. Did you conduct any ablations on these sub-parts? How do we know that we can't achieve the same (or better) performance than the complicated WISA framework just with the test-time LLM-guided prompt-engineering? See line 138: \"we believe that simple video captions are not sufficient.\" This is a major hypothesis, and doesn't appear to be tested. A clear answer to this could increase my evaluation score.\n3. How accurate were the automatically generated annotations (see lines 161-167) for the WISA-80k dataset? Was there any effort to evaluate how accurate they are? (For example, I've seen prior works do things like, randomly sample a small subset of videos, have a user study to measure how accurate each aspect of the labels are, and use that as a proxy for the reliability of the overall dataset). Or is the hypothesis that these are likely noisy, but it doesn't matter, because it still improves performance? A clear answer to this could increase my evaluation score. For example, Appendix A.11 talks about limitations of AI-generated evaluation metrics (which is true). But the labels from WISA-80k are themselves potentially very noisy / AI generated. \n4. It is well-established that Wan 2.1 has better video generation quality than CogVideoX. Does this WISA technique have diminishing returns when applied to a base model which already has better physics? Is this reflected somewhere in the metrics? Additionally, do the authors have any explanation/intuitions for why the Wan2.1 scores aren't substantially higher than the CogVideoX scores?\n5. Why are there missing entries in Table 1?\n6. Why is the PhyT2V in the same section in Table 1 as CogVideoX? Is it built using the same base model? Line 251 implies it's built on top of Tarsier-34B, a different video generation model.\n7. To what extent did cherry picking of results happen before the evaluation pipeline / comparison study? E.g. did you choose the first seed, or the best seed for both models, or the best seed only for the WISA model, etc. Please provide more info about this.\n8. What is the \"open source video dataset\" mentioned in lines 289-290?\n9. Can you please provide more info about the User Preference study (section 5.4)? E.g. how many videos, which videos. \n10. In Section A.5 the paper mentions how many learnable params there are for CogVideo, but not for Wan2.1, how many are there?\n\nLIMITATIONS:\nYes", "TITLE": "Official Review", "RECOMMENDATION": 4, "REVIEWER_CONFIDENCE": 4, "CLARITY": 3, "IMPACT": 2, "ORIGINALITY": 3, "DATE": "2025-06-28"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nCurrent text-to-video models cannot produce videos that follow physical laws faithfully, degrading the realism significantly. Therefore, the paper proposes to curate high-quality video dataset, WISA-80k, which collects 80k video clips ranging from 17 different physical phenomena. After captioning with Qwen2.5-VL, three aspects of physical knowledge are decomposed with GPT-4o, including textual physical description, qualitative physical categories, and quantitative physical properties. To incorporate the physical knowledge into text-to-video generation process, the paper proposes a physical module, leveraging mixture-of-experts self-attention mechanism to handle different physical laws. Textual physical description and quantitative physical properties are encoded and integrated in the model input. Qualitative physical classifier predicts the physical category and is jointly trained with diffusion loss. The experiments show that WISA outperforms baseline methods (Tab. 1, Fig. 7), and ablation study is performed in Tab. 2.\n\nSTRENGTHS AND WEAKNESSES:\n## Strengths\n- The paper curates a high-quality video dataset, WISA-80k, covering 17 different physical phenomena across three major categories: dynamic, thermaldynamic, and optic. The dataset provides detailed annotations, including textual physical description, qualitative physical categories, and quantitative physical properties. This dataset could facilitate future research in video generation models and physical simulation. \n- The physical phenomena are decomposed into three aspects and integrated into the video generation pipeline. Specifically, a mixture-of-experts multi-head self-attention mechanism is used in the physical module, attending to different physical effects with a specific head, enhancing the physical plausibility of generated videos. The ablation study in Tab. 2 validates the design choices. \n\n## Weaknesses \n- It is unclear how WISA model handle input prompts without detailed annotation, such as textual physical description, qualitative physical categories, and quantitative physical properties. According to the training pipeline, this information is required and integrated into the framework, but in-the-wild prompt data do not have such detailed annotation. The paper should provide more details about how the model obtains such information for inference. Otherwise, the comparison with baseline methods might be unfair. \n- The proposed method is only used for text-to-video generation, and the paper does not discuss or evaluate the model's capability for image-to-video generation, which is possible to evaluate with the curated dataset.\n- Details are missing. For example, the details of computing physical consistency, semantic coherence are not included.\n- Missing discussion with important prior works. \"Physics-IQ: Benchmarking physical understanding in generative video models (2025)\" is another benchmark for analyzing physical plausibility of video generation models, but it is not discussed in the paper.\n\nQUESTIONS:\n- How does the proposed method handle input prompts without detailed annotation?\n- How could the model and curated dataset be used for image-to-video generation?\n- How to compute physical consistency, semantic coherence?\n- Please include and discuss \"Physics-IQ: Benchmarking physical understanding in generative video models (2025)\"\n\nLIMITATIONS:\nyes", "TITLE": "Official Review", "RECOMMENDATION": 5, "REVIEWER_CONFIDENCE": 4, "CLARITY": 3, "IMPACT": 3, "ORIGINALITY": 3, "DATE": "2025-06-28"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper addresses a critical and intriguing topic: physics-aware text-to-video (T2V) generation. The contributions are twofold. First, the authors propose a dataset named WISA-80K, which covers 17 types of physical phenomena. Second, they introduce a new framework designed to better integrate physical laws into T2V models. Experimental results validate the effectiveness of their approach.\n\nSTRENGTHS AND WEAKNESSES:\nStrengths:\n1. I appreciate the exploration of physics-aware T2V generation, which is both challenging and important. The analysis of how to integrate physical principles into T2V models, along with the proposed benchmark dataset WISA-80K, is valuable to the community.\n\n2. The proposed attention mechanism, MoPA, is novel. It also appears sound, as I have seen similar (not identical) designs in other works.\n\n3. The effort put into data annotation is commendable and appreciated.\n\n4. The paper is well-written and easy to follow.\n\n\nWeaknesses:\n\n1. The evaluation metrics used focus primarily on text-video consistency and visual quality, but they largely overlook how well the generated videos adhere to physical principles (beyond user studies).\n - Could the authors design specific metrics to assess physical plausibility?\n - A feasible approach might be to evaluate physical correctness in toy scenarios. For example, generate a video where a ball undergoes free fall motion, and measure how well it follows expected physical behavior.\n\n2. It would be beneficial to include video comparison results in the supplementary materials. This would help reviewers and area chairs better assess the effectiveness of the proposed method.\n\n3. How physical plausibility is ensured remains unclear. I have the following concerns and questions:\n - Between MoPA and the physics classifier, which contributes more to enhancing physics awareness? The paper lacks an ablation study on this aspect.\n - The core component—the physics classifier—is intended to guide the T2V model to understand physical information. However, the implementation details are vague, making it difficult for other researchers to reproduce the work.\n - Have you compared the proposed physics classifier with DPO-related methods? Which approach is more effective in guiding the model to understand and follow physical principles?\n\nQUESTIONS:\nAs stated in the Strengths and Weaknesses, my main concerns are as follows:\n\n1. The need for more rigorous evaluation specifically targeting physical plausibility (related to Weakness 1).\n\n2. A more detailed ablation study comparing the contributions of MoPA and the physics classifier (related to Weakness 3).\n\n3. A clearer and more thorough description of the physics classifier, including implementation details (related to Weakness 3).\n\n4. A comparison or discussion of the proposed physics classifier with DPO-based methods, particularly regarding their effectiveness in guiding models to learn physical reasoning (related to Weakness 3).\n\nLIMITATIONS:\nyes", "TITLE": "Official Review", "RECOMMENDATION": 4, "REVIEWER_CONFIDENCE": 4, "CLARITY": 3, "IMPACT": 3, "ORIGINALITY": 3, "DATE": "2025-07-05"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper proposes WISA, a framework that injects explicit physics knowledge into text-to-video diffusion models. WISA encodes physics as structured text, qualitative categories, and quantitative attributes, and is trained on WISA-80K, an 80 K-video dataset rich in physical phenomena. Two key modules drive the approach:\n1. Mixture-of-Physical-Experts Attention (MoPA), where each attention head specializes in a distinct physical phenomenon;\n2. A multi-label physical classifier that supervises the model with BCE loss to focus on relevant attributes.\nOn the VideoPhy and PhyGenBench benchmarks, WISA achieves high scores for both physical and semantic consistency, incurring only ~5 % extra inference latency. Ablations confirm each component’s contribution.\n\nSTRENGTHS AND WEAKNESSES:\nStrength:\n1. Clear problem focus & motivation, and physics representation.  Tackles the well-known physics-violation issue in T2V models with an intuitive three-level representation (textual, categorical, numerical cues).\n2. Propose an open-source new dataset. Releases a well-annotated dataset that will benefit the community.\n3. Achieves strong scores on VideoPhy and PhyGenBench; ablation studies and human evaluation reinforce the gains.\n4. Provides visualization results. Qualitative examples and attention maps make the improvements easy to see and interpret.\n\nWeaknesses:\n1. Scalability: Adding a new physical category requires (i) manually defining the label, (ii) collecting extra data, and (iii) re-training the MoPA heads. This pipeline limits WISA’s ability to scale to larger, more diverse physics datasets.\n2. How does WISA behave on unseen physical categories? Should provide some failure cases. They would be beneficial for a more comprehensive analysis.\n3. MoPA is inserted only in the final layer; does this leave early representations under-constrained? If MoPA modules are sparsely inserted into earlier layers, does performance improve?\n4. What is the model’s classification accuracy on physical categories during evaluation?\n5. Missing some important references [1][2][3][4].\n\nDatasets:\n\n[1] ChronoMagic-Bench: A Benchmark for Metamorphic Evaluation of Text-to-Time-lapse Video Generation, NeurIPS 2024.\n\nMethods:\n\n[2]  PhysGen: Rigid-Body Physics-Grounded Image-to-Video Generation, ECCV 2024\n\n[3] MagicTime: Time-lapse Video Generation Models as Metamorphic Simulators, TPAMI 2025\n\n[4] Synthetic Video Enhances Physical Fidelity in Video Synthesis, Arxiv 2025\n\nQUESTIONS:\n1. Does performance drop noticeably on unseen physical categories?\n2. Should provide representative failure cases.\n3. What is the model’s classification accuracy on physical categories during evaluation?\n4. Is there an easy way to extend the pretrained WISA model when new physical categories are introduced?\n\nLIMITATIONS:\nYes", "TITLE": "Official Review", "RECOMMENDATION": 4, "REVIEWER_CONFIDENCE": 3, "CLARITY": 3, "IMPACT": 4, "ORIGINALITY": 3, "DATE": "2025-07-11"}