How Far Is Video Generation from World Model: A Physical Law Perspective
Authors: Bingyi Kang
Yang Yue
Rui Lu
Zhijie Lin
Yang Zhao
Kaixin Wang
Gao Huang
Jiashi Feng
Venue: ICML 2025 poster (2025)
Keywords: video generation
diffusion model
world model
Primary area: deep_learning->foundation_models

Abstract: Scaling video generation models is believed to be promising in building world models that adhere to fundamental physical laws. However, whether these models can discover physical laws purely from vision can be questioned. 
A world model learning the true law should give predictions robust to nuances and correctly extrapolate on unseen scenarios.
In this work, we evaluate across three key scenarios: in-distribution, out-of-distribution, and combinatorial generalization.
We developed a 2D simulation testbed for object movement and collisions to generate videos deterministically governed by one or more classical mechanics laws.
We focus on the scaling behavior of training
diffusion-based video generation models to predict object movements based on initial frames.
Our scaling experiments show perfect generalization within the distribution, measurable scaling behavior for combinatorial generalization, but failure in out-of-distribution scenarios.
Further experiments reveal two key insights about the generalization mechanisms of these models: (1) the models fail to abstract general physical rules and instead exhibit "case-based" generalization behavior, i.e., mimicking the closest training example; (2) when generalizing to new cases, models are observed to prioritize different factors when referencing training data: color $>$ size $>$ velocity $>$ shape.
Our study suggests that scaling alone is insufficient for video generation models to uncover fundamental physical laws.

How Far is Video Generation from World Model: A Physical Law Perspective
Bingyi Kang" Yang Yue" 12
Rui Lu Zhijie Lin Yang Zhao' Kaixin Wang² Gao Huang? Jiashi Feng'
* Eqqual Contribution (in alphabetical order)
Abstract
1. Introduction
Scaling video generation models is believed be promising in building world models that ad- here to fundamental physical laws. However, whether these models can discover physical laws purely from vision can be questioned. A world model learning the true law should give predic- tions robust to nuances and correctly extrapolate on unseen scenarios. In this work, we evaluate across three key scenarios: in-distribution, out- of-distribution, and combinatorial generalization. We developed 2D simulation testbed for object movement and collisions to generate videos de- terministically governed by one or more classical mechanics laws. We focus on the scaling behav- ior of training diffusion-based video generation models to predict object movements based ini- tial frames. Our scaling experiments show perfect generalization within the distribution, measurable scaling behavior for combinatorial generalization, but failure in out-of-distribution scenarios. Fur- ther experiments reveal two key insights about the generalization mechanisms of these models: (1) the models fail to abstract general physical rules and instead exhibit "casc-based" generaliza- tion behavior, i.e., mimicking the closest train- ing example; (2) when generalizing to new cases, models are observed to prioritize different factors when referencing training data: color > size > velocity > shape. Our study suggests that scaling alone is insufficient for video generation models to uncover fundamental physical laws.
Foundation models (Bommasani et al., 2021) have emerged remarkable capabilities by scaling the model and data to an unprecedented scale (Brown, 2020; Kaplan et al., 2020). As an example, video generation (Brooks et al., 2024; Yang et al., 2024) not only generates high-fidelity and surreal videos, but has also sparked a new surge of interest in the construction of world models (Yang et al., 2023). This topic is receiving great attention from robotics (Yang et al., 2023; Yue et al., 2024) and autonomous driving (Hu et al., 2023) due to the ability to generate realistic data and accu- rate simulations. These models are required to comprehend fundamental physical laws to produce data that extend be- yond the training corpus and guarantee precise simulation. However, it remains open question whether video genera- tion can discover such rules merely by observing vidcos. We aim to provide a systematic study to answer this question by specifically focusing on scaling up the data and models.
It is challenging to determine whether a vidco model has learned a law instead of simply memorizing the data. Since the internal knowledge of the model is inaccessible, we can only infer the understanding of the model by exam- ining its predictions on unseen scenarios, i.e., its general- ization ability. In this paper, we propose categorization (Figure 1) for a comprehensive evaluation based on the re- lationship between training and test data. In-distribution (ID) generalization assumes that training and testing data are independent and identically distributed (i.i.d.). Out-of- distribution (OOD) generalization, on the other hand, refers to the model's performance testing data that come from different distribution than the training data, particularly when latent parameters fall outside the range seen during training. Human-level physical reasoning can easily extrap- olate OOD and predict physical processes without having encountered the exact same scenario before. Additionally, also examine special OOD capacity called combina- torial generalization, which assesses whether model can combine two distinct concepts in novel way, a trait often
BByteDance Research *Tsinghua University TTechnion. Corre- spondence to: Bingyi Kang <bingykang @ @emaill.mmm........... Yang Yue <le-y22@ maiilsssinghua.ccu.mm
Proceedings of the nnnd International Conference Machine Learning, Vancouver, Canada. PMLR 267, 2025. Copyright 20255 by the author(s).

---

How Far is Video Generation from World Model: A Physical Law Perspective
3
23
83
83
23
83
43
8
8
53
33
83
33
23
33
23
8
8
53
23
233
x
83
8
33
in-distribution
out-of-distribution
combinatorial
Figure 1: Categorization of generalization patterns. denotes training data. X denotes testing data.
considered essential for foundation models in the advance- ment toward artificial general intelligence (AGI) (Du & Kaelbling, 2024).
from 67% to 10%. This suggests that scaling is critical for improving combinatorial generalization.
Our empirical analysis reveals two intriguing properties of the generalization mechanism in video generation models. First, these models can be easily biased by "deceptive" ex- amples from the training set, leading them to generalize in a "case-based manner under certain conditions. This phenomenon, also observed in large language models (Hu et al., 2024), describes a model's tendency to reference sim- ilar training cases when solving new tasks. For instance, consider video model trained data of a high-speed ball moving in uniform linear motion. If data augmentation is performed by horizontally flipping the videos, thereby intro- ducing -verse-directton motion, the model may generate a scenario where a low-speed ball reverses direction after the initial frames, even though this behavior is not physically correct. Second, we explore how different data attributes compete during the generalization process. For example, if the training data for uniform motion consists of red balls and blue squares, the model may transform red square into ball immediately after the conditioning frames. This behav- ior suggests that the model prioritizes color over shape. Our pairwise analysis reveals the following prioritization hierar- chy: color >> size velocity > shape. This ranking could explain why current video generation models often struggle with maintaining object consistency. We hope these find- ings provide valuable insights for future research in video generation and world models.
Moreover, real-world videos typically contain complex, non- rigid objects and motions, which present significant chal- lenges for quantitative evaluation and even human validation. The rich textures and appearances of such videos can act as confounding factors, distracting the model from focus- ing on the underlying physics. To mitigate these issues, we specifically focus on classical mechanics and develop 2D simulator with objects represented by simple geometric shapes. Each video depicts the motion or collision of these 2D objects, governed entirely by one two fundamental physical laws given the initial frames. This simulator allows us to generate large-scale datasets to support the scaling of video generation models. Additionally, we have developed tool to infer internal states (e.g., the position and size) of each object in the generated video from pixels. This enables us establish quantitative evaluation metries for physical law discovery.
We begin by investigating how scaling video generation models affects ID and OOD generalization. We select three fundamental physical laws for simulation: uniform linear motion of ball, perfectly elastic collision between two balls, and parabolic motion of ball. We scale the dataset from 30K to 3 million examples and increase the video dif- fusion model's parameters from 22M to 310M. Consistently, we observe that the model achieves a near-perfect ID gener- alization across all tasks. However, the OOD generalization error does not improve with increased data and model size, revealing the limitations of scaling video generation models in handling OOD data. For combinatorial generalization, design environment that involves multiple objects un- dergoing free fall and collisions to study their interactions. Each time, four of the eight objects are selected to create a video. In total, 70 combinations (Ct) are possible. We use 60 of them for training and 10 for testing. We train models by varying the number of training data from 600K to 6M. We manually evaluate the generated test samples by labeling them "abnormal" if the video looks physi- cally implausible. The results demonstrate that scaling the data substantially reduces the percentage of abnormal cases
2. Video Generation for Phyiscal Law Discovery
2.1. Problem Definition
In this section, we aim to establish the framework and define the concept of physical laws discovery in the context of video generation. In classical physics, laws are articulated through mathematical equations that predict future state and dynamics from initial conditions. In the realm of video- based observations, each frame represents a moment in time, and the prediction of physical laws corresponds to generating future frames conditioned on past states.
2

---

How Far is Video Generation from World Model: A Physical Law Perspective
Consider physical procedure which involves several latent Z _ RR* each represent- ing a certain physical parameter such as velocity position. By classical mechanics, these latent variables will evolve by a differential equation & = F(z). In the discrete ver- sion, if the time gap between two consecutive frames is 6, then we have 1 2 2, + (2t)). Denote the ren- dering function as R(-) : Z I B3xHxW which renders the state of the world into an image of shape H x W with RGB channels. Consider video V = +1,, 12......... J1,) consisting of I. frames that follows the classical mechanics dynamics. The physical coherence requires that there ex- ists a series of latent variables which satisfy the following requirement: 1) 2 = 2, + FF()),, I,..., L- 1. 2) 4. R(z,). t = L, ...... L. We train video generation model p parametried by 0. where pe (4,, ..., It) charac- terizes its understanding of video frames. We can predict the subsequent frames by sampling from po po .,,...,Ae) Lo) based on initial frames' condition. The variable c usually takes the value of 1 or 3 depending on the tasks. Therefore, physical coherence loss can be simply defined as -log MoggPo(Ccto)) (o Iz ,,,..., I.). measures how likely the predicted value will cater to real-world development. The model must understand the underlying physical pro- cess to accurately forecast subsequent frames, which we can quantatively evaluate whether video generation models correctly discover and simulate the physical laws.
spacetime patches, as transformer tokens. Notably, self- attention is applied to the entire spatio-temporal sequence of video tokens, without distinguishing between spatial and temporal dimensions. For positional embedding, a 3D vari- ant of RoPE (Su et al., 2024) is adopted. As stated in Sec.2.1, 1, our video model is conditioned on the first c frames. The e-frame video is zero- zero-padded to the same length as the full physical video. We also introduce a binary mask "video" by setting the value of the first frames to 1, indicating that these frames are condition inputs. The noise, condition and mask videos are concatenated along the channel dimension to form the final input to the model.
variables z (21, -2.-... . 2/)
2.3. On the Verification of Learned Laws
Suppose we have a video generation model learned as de- scribed above. How do we determine whether the underly- ing physical law has been discovered? A well-established law describes the behavior of the natural world, egg., how objects move and interact. Therefore, a video model in- corporating true physical laws should be able to withstand experimental verification, producing reasonable predictions under any circumstances, which demonstrates the model's generalization ability. To comprehensively evaluate this, we consider the following generalization categorization (see Figure 1) within the scope of this paper: 1) In-distribution (ID) generalization describes the setting where training data and testing data are from the same distribution. In our case, both training and testing data follow the same law and are located in the same domain. 2) A human who has learned a physical law can easily extrapolate to scenarios that have never been observed before. This ability is referred to as dittof-distribution (OOD) generalization. Although it sounds challenging, this evaluation necessary as in- dicates whether model can learn principled rules from data. 3) Moreover, there is situation between ID) and OOD, which has more practical value. We call this combinatorial generalization, representing scenarios where every "con- cept" or object has been observed during training, but not all of their combinations. It tests a model's ability to com- bine relevant information from past experiences in novel ways. A similar concept has been explored in LLMs (Rive- land & Pouget, 2024), which showed that models can excel at linguistic-instructin tasks by recombining previously leareed components, without task-specific experience.
2.2. Video Generation Model
In this paper, we focus exclusively on video generation models, leaving discussions on other modeling methods for future work. Following Sora (Brooks et al., 2024), we adopt the Variational Auto-Encoder (VAE) and DiT archi- tectures for video generation. The VAE compresses videos into latent representations both spatially and temporally, while the DiT models the denoising process. This approach shows strong scalability and achieves promising results in generating high-quality videos.
VAE Model. We employ a (2+1)D-VAN VAE to project videos into a latent space. Starting with the SD1.5 VAE structure, we extend it into a spatiotemporal autoencoder using 3D blocks (Yu et al., 2023b). All parameters of the (2+1)D- VAE are pretrained on high-quality image and video data to maintain strong appearance modeling while enabling motion modeling. More details are provided in Appendix B.2. In this paper, we fax the pretrained VAE encoder and use it as a video compressor. The results in Appendix B.3 confirm the VAE's ability to accurately encode and decode the physical event videos. This allows to focus solely on training the diffusion model to learn the physical laws.
3. In-Distribution and Out-of-Distribution Generalization
In this section, we study the correlation between ID/OOD generalization and model or data scaling, focusing on video generation models. We use deterministic tasks governed by basic kinematic equations, as they allow clear definitions of ID/OOD and straightforward quantitative error evaluation.
Diffusion model. Given the compressed latent represen- tation of the VAE model, we flatten it into a sequence of
3

---

How Far is Video Generation from World Model: A Physical Law Perspective
uni motion
collision
parabola
steps using 32 Nvidia A GPUs with a batch size of 256, which was sufficient for convergence, since model trained for 300K steps achieves similar performance. We keep the pre-trained VAE fixed. Each video consists of 32 frames with a resolution of 128x 128. We also experimented with a 256x256 resolution, which yielded a similar generalization error but significantly slowed down the training process.
Table I: Details of DiT model sizes.
Model
Layers
Hidden size
Heads
#Param
DiT-S
12
384
6
22.5M
DīT-B
12
768
89.5M
Figure 2: Downsampled vidco visualization. The arrow indicates the progression of time.
DiT-L
24
1024
310.0M
DIT-XL
28
1152
16
456.0M
Evaluation metries. We observed that the learned models are able to generate balls with consistent shapes. To obtain the center positions of the i-th ball in the generated videos, zi, we use a heuristic algorithm based on the mean of col- ored pixels, distinguishing the balls by color. To ensure the correctness of zj, we exclude frames with part of a ball out of view, yielding valid frames T. For collision scenar- ios, only frames after the collision are considered. We then compute the velocity of each ball, vy, at each moment by differentiating their positions. The error for a video is de- fined as: e = NiTi Σi= Eter 1 - off] where of is the calculated velocity at time t, of is the ground-truit velocity in the simulator, N is the number of balls and [T] is the number of valid frames.
3.1. Fundamental Physical Scenarios
Specifically, we consider three physical scenarios. 1) Uni- form Linear Motion: A colored ball moves horizontally with constant velocity. This is used to illustrate the law of Intertia. 2) Perfectly Elastic Collision: Two balls of different sizes and speeds move horizontally toward each other and collide. The underlying physical law is the con- servation of energy and momentum. 3) Parabolic Motion: ball with an initial horizontal velocity falls due to grav- ity. This represents Newton's 's second law of motion. Each motion is determined by its initial frames. See Figure 2 for visualizations.
Training data generation. We use Box2D to simulate kinematic states for various scenarios and render them as videos, with each scenario having 2-4 degrees of freedom (DoF), such as the balls' initial velocity and mass. An in-distribution range is defined for each DoF. We generate training datasets of 30K, 300K, and 3M videos by uniformly sampling a high-dimensional grid within these ranges. All balls have the same density, so their mass is inferred from their size. Gravitational acceleration is constant in parabolic motion for consistency. The initial ball positions are initial- ized randomly within the visible range. More details are provided in Appendix C.I.
Baseline. We calculate the error between the ground truth velocity and the parsed values of the ground truth video, referred to as Groundtruth (GT). This represents the system error- error-caused by parsing video into velocity- and defines the minimum error a model can achieve.
3.2. Perfect ID and Failed OOD Generalization
For in-distribution (ID) generalization in Figure 3, increas- ing the model size (DiT-S to DiT-L) or the amount of data (30K to 3M) consistently decreases the velocity error across the three tasks, strongly evidencing the importance of scal- ing for ID generalization. Take the uniform motion task as example: the DÎT-S model has velocity error of 0.022 with 30K data, while DiT-L achieves an error of 0.012 with 3M data, very close to the error of 0.010 obtained with ground truth video.
Test data generation. We evaluate the trained model using both ID and OOD data. For ID evaluation, we sample from the same grid used during training, ensuring that no specific data point is part of the training set. OOD evaluation videos are generated with initial radius and velocity values outside the training range. There are various types of OOD setting, e.g. veloeity/radius-olyy or both OOD.
However, the results differ significantly for out-of- distribution (OOD) predictions. First, OOD velocity errors are an order of magnitude higher than ID errors in all set- tings. For example, the OOD error for the DiT-L model in uniform motion with 3M data is 0.427, while the ID error is just 0.012. Second, scaling up the training data and model size has little or negative impact on reducing this prediction error. The variation in velocity error is highly random as data or model size changes, e.g., the error for DiT-B on uni-
Models. For each scenario, train models of varying sizes from scratch, as shown in Table 1. This ensures that the results are not influenced by uncontrollable pretraining data. The first three frames are provided as conditioning, which is sufficient to infer the velocity of the balls and predict the subsequent frames. The diffusion model is trained for 100K
4

---

How Far is Video Generation from World Model: A Physical Law Perspective
Uniform Motion
Collision
Parabola
0.45
0.3
0.4
GT
0.4
0.25
0.35
ID:
Velocity Error
0.35
0.2
DIT-S
0.3
DIT-B
0.3
0.15
DIT-L
OOD: :
0.06
0.03
0.02
0.03
0.02
0.04
DIT-S
0.01
0.01
0.02
DIT-B
DIT-L
30k
300k
3m
30k
300k
3m
30k
300K
3m
Figure 3: The error in the velocity of balls hetween the ground truth state in the simulator and the values parsed from the generated video by the diffusion model, given the first frames.
form motion is 0.433, 0.328 and 0.358, with data amounts of 30K, 300K and 3M. We also trained DiT-XL. on the uni- form motion 3M dataset but observed no improvement in OOD generalization. As result, we did not carry out DiT- XL training in other scenarios. These findings suggest the inability of scaling to perform reasonably in OOD scenarios. The sharp difference between the ID and OOD settings mo- tivates us to study the generalization mechanism of video generation in Section 5.2.
4. Combinatorial Generalization
In Section 3, video generation models failed to reason in the OOD scenarios. This is understandable deriving pre- cise physical laws from data difficult for both humans and models. For example, took scientists centuries to formulate Newton's three laws of motion. However, even child can intuitively predict outcomes in everyday situations by combining elements from past experiences. This ability to combine known information to predict new scenarios is called combinatorial generalization. Now, we evaluate the combinatorial abilities of diffusion-based video models.
Figure 4: Downsampled videos. The black objects are fixed and others are dynamic.
unique templates. See Figure 4 for examples. Each ob- ject has a distinct visual appearance and color, allowing the model to infer object type and physical attributes by learning associations from training data. Since all objects start at rest with zero initial velocity, we provide the first frame so that the model can infer their attributes and use this information to predict future events.
4.1. Combinatorial Physical Scenarios
We selected the PHYRE simulator (Bakhtin et al., 2019) as our testbed a 2D environment involves multiple objects that fall and then collide with each other, forming complex physical interactions. It features various types of objects, including balls, jars, bars, and walls, which can be either fixed or dynamic. This enables complex interactions such as collisions, parabolic trajectories, rotations, and friction to occur simultaneously within a video. Despite this complex- ity, the underlying physical laws are deterministic, allowing the model to learn the laws and predict unseen scenarios.
Each template was initialized with random sizes and posi- tions for four objects, generating 100K videos cover range of possible scenarios. To explore the combinatorial ability and scaling effects of the model, we structured the training data at three levels: minimal set of 6 templates (0.6M videos) that includes all types of interaction of two objects between the eight types of objects, and larger sets with 30 and 60 templates (3M/6M videos), with the set of 60 templates almost covering the entire template space. The minimal training set places the greatest demand on the model's '' ability for compositional generalization.
Training Data. There are eight types of objects are con- sidered, including two dynamic gray balls, group of fixed black balls, fixed black bar, dynamic bar, a group of dy- namic standing bars, dynamic jar and a dynamic standing stick. Each task contains one red ball and four randomly chonsen objects from the eight types, resulting in C3 = 70
Test Data. For each training template, we reserve a small set of videos to create the in-template evaluation set. Ad- ditionally, 10 unused templates are reserved for the our- of-template evaluation set to assess the model's ' ability to
5

---

How Far is Video Generation from World Model: A Physical Law Perspective
generalize to new combinations not seen during training.
the in-template testing set. This can be explained by the fact that each training example in the 6-template set is exposed ten times more frequently than those in the 60-template set, allowing it to better fit the in-template tasks associated with template 6. Furthermore, we conducted an additional ex- periment using a DiT-B model on the full 60 templates to verify the importance of model scaling. As expected, the abnormal rate increases 24%. These results suggest that both model capacity and coverage of the combination space are crucial for combinatorial generalization. This insight implies that scaling laws for video generation should focus on increasing combination diversity, rather than merely scal- ing up data volume. Some generated videos can be found in Figure 21, Figure 22, Figure 23, and Figure 24.
Models. The first frame is used as the conditioning for video generation since the initial objects are static, We found that smaller models such as DiT-S struggled with complex videos, so we mainly used DiT-B and DIT-XI.. All models were trained for 1000K gradient steps on 64 Nvidia A100 GPUs with a batch size of 256, ensuring convergence. To better capture the complexity of physical events, we increased the resolution to 256x256 with 32 frames.
Evaluation Metrics. In the ID/OOD setting, scenes are simple (e.g., 1-2 colored balls on plain background), en- abling reliable position and velocity estimation via pixel averaging and frame differencing. In contrast, the combina- torial setting includes many irregularly shaped and similarly colored objects, making pixel assignments ambiguous and position estimation unreliable. We also tested methods such as Hough circle detection and pretrained object detectors, but they resulted in significant errors. Given these chal- lenges, we excluded velocity as a metric.
5. Deeper Analysis
In this section, we aim to investigate the generalization mechanism of a video generation model, through systemic experimental designs. Based on the findings, we try to identify certain patterns in combinatorial generalization that might be helpful in harnessing or prompting the models.
Instead, we rely on a combination of objective video fidelity metrics and human evaluation. We use several metries to as- sess the fidelity of the generated videos compared to ground truth. Frechet Video Distance (FVD) (Unterthiner et al., 2018 8) calculates the feature distances between generated and real videos using features from Inflated-3D ConvNets (I3D) pretrained on Kinetics-400 (Carreira & Zisserman, 2017). SSIM and PSNR (Wang et al., 2004) are pixel-level metrics: SSIM evaluates brightness, contrast, and structural similarity, while PSNR measures the ratio between peak signal and mean squared error, both averaged across frames. LPIPS (Zhang et al., 2018) gauges the perceptual similarity between image patches. Since pixel-level metrics may not fully capture physical plausibility, we also include human evaluations, reporting the abnormal ratio of videos gener- ated that violate physical laws assessed by humans. This combined approach offers a comprehensive evaluation for physical correctness in the complex setting.
5.1. Understanding Generalization from Interpolation and Extrapolation
The generalization ability of a model originates from its interpolation and extrapolation ability (Xu et al., 2020; Balestriero et al., 2021). In this section, we design ex- periments explore the limits of these abilities for a video generation model. We design datasets which delibrately leave out some latent values, i.e. velocity. After training, we test model's 's prediction on both seen and unseen scenarios. We mainly focus on uniform motion and collision processes.
We create a series of training sets, where certain range of velocity is absent. Each set contains 200K videos to ensure faimess. As shown in Figure 5 (1)-(2), with a large gap in the training set, the model tends to gen- erate videos where the velocity is high or low to resemble the training data when initial frames show middle-range velocities. We find that the video generation model's OOD accuracy closely related to the size of the gap, as seen in Figure 5 (3), when the gap is reduced, the model cor- rectly interpolates for most of the OOD data. Moreover, as shown in Figure 5 (4) and (5), when subset of the miss- ing range is reintroduced (without increasing the amount of data), the model exhibits stronger interpolation abilities. In total, In Section 3, the evaluation is strictly OOD extrapola- tion. In this setting, simply increasing the amount of data or model size does not significantly improve performance due to the difficulty of truly grasping the physical law and true extrapolation. In contrast, Figure 5 reveals transition from extrapolation to interpolation. As the gap between the adjacent training regions narrows, model performance on
Uniform Motion.
4.2. Scaling Law Observed for Combinatorial Generalization
requires higher resolution, much more training iterations, and larger model sizes to perform well on this task due to increased complexity. Consequently, we are unable to conduct a comprehensive sweep of all data and model size combinations as in Section 3. Therefore, we start with the largest model, DiT-XL, to study data scaling behavior for combinatorial generalization. As shown in Table 2, when the number of templates increases from 6 to 60, all metrics improve on the out-of-template testing sets. Notably, the abnormal rate for human evaluation significantly reduces from 67% to 10% Conversely, the model trained with 6 tem- plates achieves the best SSIM, PSNR, and LPIPS scores on
6

---

How Far is Video Generation from World Model: A Physical Law Perspective
Table 2: Combinatorial generalization results. The results are presented in the format of (in-template resulty
{out-of-template result).
Model
#Templates
FVD (L)
SSIM (T)
PSNR (1)
LPIPS (L)
Abnormal (L)
DiT-XL
6
18.2/22.1
0.973 / /.943
32.8 25.5
0.028 / 0.082
3% 67%
DIT-XL.
30
19.5
0.973 / 0.950
32.7
0.028/ / 0.065
3% 18%
DIT-XI
60
17.6/ 18.7
0.972 / 0.951
32.4/27.3
0.030/ / 0.062
2% / 10%
DiT-B
60
18.4/214
0.967 / 0.949
30.9 27.0
0.035 / 0.066
3% / 24%
the test set improves, reflecting the model's stronger ability to interpolate.
ward the high-speed range. In contrast, the Set-2 model occasionally produces videos with negative velocities, as highlighted by the green circle. For instance, low-speed ball moving from left to right may suddenly reverse di- rection after its condition frames. This could occur since the model identifies reversed training videos as the clos- est match for low-speed balls. This distinction between the two models suggests that the video generation model is influenced by "'eceptive" examples in the training data. Rather than abstracting universal rules, the model appears to rely on memorization and case-based imitation for OOD generalization.
Collision. It involves multiple variables and is more chal- lenging since the model has to learn a two-dimensional nonlinear function. Specifically, we exclude one or more square regions from the training set of initial velocities for two balls and then assess the velocity prediction error after the collision. For each velocity point, we sample a grid of radius parameters generate multiple video cases and compute the average error. As shown in Figure 6 (1)-(2,, the video generation model's extrapolation error induces an intringuing discrepancy among OOD points: For the OOD velocity combinations that lie within the convex hull of the training set, i.e,, the internal red squares in the yellow region, the model generalizes well. However, the model experiences large errors when the latent values lie in the exterior space of the convex hull of the training set.
5.3. How Does Diffusion Model Retrieve Data?
We aim to investigate the ways a video model performs case
close training examples for a given test input. We use uniform linear motion for this study. Specifically, we compare four attributes, i.e., color, shape, size, and velocity, in a pairwise manner. Through comparisons, we seek to determine the model's 's preference for relying on specific attributes in case matching.
5.2. Memorization or Generalization
Previous work (Hu et al., 2024) indicates that LLMs rely on memorization, reproducing training cases during infer- ence instead of learning the underlying rules for tasks like addition arithmetic. In this section, we investigate whether video generation models display similar behavior, memo- rizing data rather than understanding physical laws, which limits their generalization to unseen data.
Every attribute has two disjoint sets of values. For each pair of attributes, there are four types of combinations. We use two combinations for training and the remaining two for testing. For example, we compare color and shape in Figure 8 (1). Videos of red balls and blue squares with the same range of size and velocity are used for training. At test time, a blue ball changes shape into square immediately after the condition frames, while a red square transforms into ball. We observed no exceptions on 1,400 test cases, showing that the model prioritizes color over shape for case matching. A similar trend is observed in the comparisons of size vs. shape and velocity vs. shape, as illustrated in Figure 8 (2)-(3), indicating that shape is the least prioritized attribute. This suggests that diffusion-based video models inherently favor other attributes over shape, which may ex- plain why current open-set video generation models usually struggle with shape preservation.
We train our model on uniform motion videos with velocities
Sei-1 w.o. flip
Set-2 w. flip
4.0
Velocity generated video
2.5
[2.5, 4.0). using the first three frames as in-
put conditions. Two
0.0
training sets are used: Ser-/ only contains balls moving from left to right, while Ser-2 includes movement in both directions, by us- ing horizontal flipping at training time. In evaluation, we focus on low-speed balls (v E [1.0, 2.5),, which were not present in the training data. As shown in Figure 7, the Set-/ model generates videos with only positive velocities, biased to-
Law teetts
Iest date
Tain Región
-2.5
-..0
1.0
2.5
4.0
1.0
2.5
4.0
Velocity of conditioned frames
Figure 7: motion illustrating memorization.
The example of uniform
The other three attribute pairs are presented in Figure 9. For the continuous attributes of velocity and size, we selected two distinct values the extreme boundaries create bi- nary comparisons: 1.0 and 4.0 for velocity, and 0.7 and 1.4 for size. For the comparison of color vs. size, the train-
7

---

How Far is Video Generation from World Model: A Physical Law Perspective
Velocity of conditioned frames v.8. Velocity of generated video
4.0
Reger
2.5
1.0
1.0
2.5
4.0
1.0
2.5
4.0
1.0
2.5
4.0
1.0
2.5
4.0
1.0
2.5
4.0
Law of Inertia
Test data
Train Region
Figure 5: Uniform motion video generation. Models are trained on datasets missing middle velocity range. example. in first figure, training velocities cover [1.0, 1.25] and [3.75, 4.0), excluding the middle range. When evaluated with velocity condition the missing range [1.25, 3.75], the generated velocity tends to shift away from the initial condition, breaking the Law of Inertia.
Drsinin
92
Largy
Figure 6: Collision video generation. Models trained on the yellow region and evaluated data points in both yellow (ID) and red (OOD) regions. When the OOD range is surrounded by the training region, the OOD generalization error remains relatively small and comparablc to the ID error.
Fredi 07 Phol GT
Velucity
Omiiing
Figure 9: Uniform motion. (1) Color v.s. size, (2) Color v.s. velocity, (3) Size velocity. The change in velocity is evident when comparing the ground truth with the predicted vidcos.
stew
Stope
stape
can actually enable conceptually-comet combinable video gen- cration? In this section, we idenfify three foundamental combinatorial patterns through experimental design.
presstessiio www.eeawww..gggggooeooonooo wwg.ogoootteedddnn (aaee))
Attribute composition. As shown in Figure 14 (1)-(2, tain attribute pairs-suche velocity and size, color and size exhibit some degree of combinatorial generalization.
Figure 8: Uniform motion. (1) Color shape. (2) Size shape, (3) Velocity v.. shape. The arrow = signifies that the generated videos shift from their specified conditions to resemble similar training cases. For example, in the first figure, the model trained videos of blue balls and red squares. When conditioned with blue ball, shown in the bottom, transforms into blue square, i.e., mimicking training case by color.
Spatial composition. As given by Figure 11 (left side) in the Appendix, the training data contain two distinct types of physical events. One type involves a blue square moving horizontally with constant velocity while red ball remains stationary. In contrast, the other type depicts red ball moving toward and then bouncing off a wall, while the blue square remains stationary. At test time, when the red ball and the blue square are moving simultaneously. the model is able to correctly generate the movement of both balls.
ing consists of: (1) red, small balls and (2) blue, large balls. During evaluation, across 202 evaluation cases, the size was consistently transformed while the color remained fixed, clearly demonstrating that the color > size. Similar observations are evident in Figure 9 (2) and (3), leading to conclusion that color > velocity and size > velocity. Based on the above analysis, we establish the prioritization order color size velocity shape.
Temporal combination. As illustrated on the right side of Figure 11, when the training data includes distinct physi- cal events-har featuring two balls colliding without bounc- ing and the other half showing red ball bouncing off wall-the model learns to combine these events temporally. Consequently, when the balls collide near the wall, the model accurately predicts the collision and the blue ball will rebound off the wall with unchanged velocity.
5.4. Complex Combinatorial Generalization
In Section 4, we show that scaling the data coverage can boost combinatorial generalization. But what kind of data
8

---

How Far is Video Generation from World Model: A Physical Law Perspective
Therefore, the video generation model can identify basic physical events in the training set and combine them across attributes, time, and space to generate videos featuring com- plex chains of physical events.
aim learn models that can accurately predict how an environment evolves after some actions taken. Previ- ously, they often operated in an abstracted space and were used in reinforcement learning (Sutton, 2018) to enable planning (Silver et al., 2016; Schrittwieser et al., 2020) or facilitate policy learning through virtual interactions (Hafner et al., 2019; 2020; Yue et al., 2023). With the advancement of generative models, world models can now directly work with visual observations by employing general frame- work of conditioned video generation. For example, in autonomous driving (Hu et al., 2023; Jia et al., 2023; Gao et al., 2024; Zheng et al., 2024), the condition is the driver's s operations, while in robot world models (Yang et al., 2023; Black et al., 2023; 1xw, 2024), the condition is often the control signals. Genie (Bruce et al., 2024) instead recovers the conditions from video games in an unsupervised way. In our physical law discovery setting, it does not require per- step action/condition since the physical event is determined by the underlying laws once an initial state is specified. See more related works in Appendix A.
5.5. Is Video Sufficient for Complete Physics Modeling?
For video generation model to function as a world model, the visual representation must provide sufficient informa- tion for complete physics modeling. In our experiments, we found that visual ambiguity leads to significant inaccuracies in fine-grained physics modeling. For example, in Figure 10, is difficult to determine if ball can pass through a gap based on vision alone when the size difference at the pixel level, leading to visually plausible but incorrect results. Sim- ilarly, visual ambiguity in the horizontal position of ball relative to a block can result in different outcomes. These findings suggest that relying solely on visual representations, may be inadequate for accurate physics modeling.
7. Conclusion and Discussion
Conclusion. Video generation is believed to be a promising way toward scalable world models. However, its capability to learn physical laws from visual observations has not yet been verified. We conducted the first systematic study in this area by examining its generalization performance in three typical scenarios: in-distribution, out-of-distribune (OOD), and combinatorial generalization. The findings in- dicate that scaling alone cannot address the OOD problem, although it does enhance performance in other scenarios. Our in-depth analysis suggests that the generalization of the video model is based more on referencing similar training examples than learning universal rules. We observed prior- itization order of the color >> size >> velocity > shape in this behavior. In conclusion, our study suggests that naive scaling is insufficient to discover physical laws.
Figure 10: First row: Ground truth; second row: generated video. Ambiguities visual representation result inaccuracies fine- grained physics modeling.
6. Related Works
Video generation. Open-set video generation is mainly based on diffusion models (Rombach et al., 2022; Ho et al., 2022b;a; He et al., 2024) or auto-regressive models (Yu et al., 2023a;b; Kondratyuk et al., 2023). These models often re- quire a pre-trained image or video VAE (Kingma, 2013; Van Den Oord al., 2017) for data compression improve computational efficiency. Some approaches leverage pre- trained rext-to-image (T2I) models for zero-shot (Khacha- tryan et al., 2023; Zhang et al., 2023b) or few-shot (Wu et al., 2023) video generation. Furthermore, Image-to- Video (12V) generation (Zeng et al., 2024; Girdhar et al., 2023; Zhang et al., 2023a) shows that video quality improves substan- tially when conditioned an image. The Diffusion Trans- former (DiT) (Peebles & Xie, 2023) demonstrates better scaling behavior than U-Net (Ronneberger et al., 2015) for T2I generation. Sora (Brooks et al., 2024) leverages the DIT architecture to directly operate on space-time patches of latent video and image codes. Our model follows Sora's 's architecture and conceptually aligns with 12V generation, relying on image(s) for conditioning instead of text prompts. There are several previous work evaluating the physical un- derstanding of large video models (Bansal et al., 2024; Sun et al., 2024; Liao et al., 2024) and benchmarks evaluating large video models more broadly (Huang et al., 2024; Liu et al., 2024).
Disceusion. The insights from our study have practical im- plications for world scenarios. Real-world video data is inherently vast, continuous, and high-dimensional, mak- ing it much harder to comprehensively cover compared to pure language data, For instance, in real-world robotics, factors such as object velocity, joint configurations, camera perspectives, background noise, and task goals vary across continuous and combinatorially large space. Our results suggest a path forward for addressing these challenges- in- creasing the combinatorial diversity of training data leads to significantly better performance in physical video modeling. Moreover, our findings imply that model scaling is most effective when supported by a sufficiently diverse training distribution. This has direct implications for fututre work to design both datasets and models that can generalize robustly in complex, real-world environments.
World model. World models (Ha & Schmidhuber, 2018)
9

---

How Far is Video Generation from World Model: A Physical Law Perspective
Impact Statement
Bommasani, R., Hudson, D. A., Adeli, E., Altman, R., Arora, S., Arx, S., Bernstein, M. S., Bohg, J., Bosse- lut, A., Brunskill, E., et al. On the opportunities and risks of foundation models. arXiv preprint arXiv:2108.07258, 2021.
This paper presents work whose goal to advance the field of Machine Learning. There are many potential societal consequences of our work, none of which feel must be specifically highlighted here.
Bouman, K. I.., Xiao, B., Battaglia, P., and Freeman, W. T. Estimating the material properties of fabric from video. In Proceedings of the IEEE international conference on computer vision, pp. 1984-1991, 2013. 15
References
Ix world model. 2024. URL https:ttt///cc.ct /ww.. 1x.
Brooks, T., Peebles, B., Holmes, C., DePue, W. Guo, Y., Jing. L., Schnurr, D., Taylor, J., Luhman, T., Luhman, E., Ng. C., Wang, R., and Ramesh, A. Video generation models as world simulators. 2024. URL https://
Allen, K. R., Smith, K. A., and Tenenbaum, J. B. Rapid trial- and-error learning with simulation supports flexible tool use and physical reasoning. Proceedings of the National Academy of Sciences, 117(47):9332-229
2020. 15
opena i com/ aosom/reearchh 3, 9, 16, 22
Ates, T., Atesoglu, M. S., Yigit, C., Kesen, Kobas, M., Erdem, E., Erdem, A., Goksun, T., and Yuret, D. Craft: A benchmark for causal reasoning about forces and inter- actions. arXiv preprint arXiv:2012.04293, 2020. 15
Brown, T. B. Language models are few-shot learners. arXiv preprint arXiv:2005. /4/65, 2020.
Bruce, J., Dennis, M. D., Edwards, A., Parker-Holder, J.,
Shi, Y., Hughes, E., Lai, M., Mavalankar, A., Steiger- wald, R., Apps, C., et al. Genie: Generative interactive environments. In Forty-first International Conference on Machine Learning, 2024. 9
Bakhtin, A., van der Maaten, L., Johnson, J., Gustafson, L., and Girshick, R. Phyre: A new benchmark for physical reasoning. Advances in Neural Information Processing Systems, 32, 2019. 5, 15
Bycon, M. Park, B., Kim, H., Lee, S., Back, W., and Kim, S. Coyo-700m: Image-text pair dataset.
Balestriero, R., Pesenti, J., and LeCun, Y. Learning in high dimension always amounts to extrapolation. arXiv preprint urXiv:2110.0944 2021. 6
//tpt//ggtt//////// com/kakaobrain/ coyo-datasets 2022. 16
Cao, Q.. Wang. Li, X., Chen, Y., Ma, C., and Yang. X. Teaching video diffusion model with latent physical phe- nomenon knowledge. arXiv preprint 2rX 2024. 15
Bansal, H., Lin, Z., Xie, T., Zong, Z., Yarom, M., Bitton, Y., Jiang, C., Sun, Y., Chang, K.-W,, and Grover, A.
Videophy: Evaluating physical commonsense for video generation. arXiv preprint arXiv:2406.03520, 2024. 9
Carreira, and Zisserman, A. Quo vadis, action recogni- tion? model and the kinetics dataset. In proceed- ings of the IEEE Conference Computer Vision and Pattern Recognition, pp. 6299-6308, 2017. 6
Beaumont, R. and Schuhmann, C. Laion- sttps://qithut com/
aesthetics vl. (atOOO-AA/aaionndaaaassaa waw.aaattteaaaaaaaaeeeeeeaeeeaee /bl ob/main/
laion- aesthetic....ee.... md, 2022. 16
de Silva, B. M., Higdon, M., Brunton, S. L.,, and Kutz, J. N. Discovery of physics from data: Universal laws and discrepancies. Frontiers in artificial intelligence, 3:25, 2020. 15
Bell, S., Upchurch, P., Snavely, N., and Bala, K. Mate- rial recognition in the wild with the materials in context database. In Proceedings of the IEEE conference computer vision and pattern recognition, pp. 3479-3487, 2015. 15
Diederik, P. K. and Ba, J. Adam: A method for stochastic optimization. ICLR, 2015. 16
Black, K., Nakamoto, M., Atreya, P., Walke, H., Finn, C., Kumar, A., and Levine, S. Zero-shot robotic manipulation with pretrained image-editing diffusion models. arXiv preprint ar arXiv:2310.1066
Du, Y. and Kaelbling, L. Compositional generative model- ing: single model is not all you need. arXiv preprint arXiv:2402.00 103, 2024.
2023. 9
Blattmann, A., Dockhorn, T., Kulal, S., Mendelevitch, D., Kilian, M., Lorenz, D., Levi, Y., English, Z., Voleti, V., Letts, A., et al. Stable video diffusion: Scaling latent video diffusion models to large datasets. arXiv preprint arXiv:23/1.15122 2023. 17
Gadre, S. Y., Iharcoo, G., Fang, A., Hayase, J., Smyrnis, G., Nguyen, T.. Marten, R., Wortsman, M., Ghosh, D., Zhang, J., Orgad, E.,, Entezari, R., Daras, G., Pratt, S., Ramanu- jan, V., Bitton, Y., Marathe, K., Mussmann, S., Vencu, R., Cherti, M., Krishna, R., Koh, P. W., Saukh, O., Ratner,
10

---

How Far is Video Generation from World Model: A Physical Law Perspective
A. J., Song, S., Hajishirzi, H., Farhadi, A., Beaumont, R., Oh, S., Dimakis, A. G., Jitsev, J., Carmon, Y., Shankar, V., and Schmidt, L. Datacomp: In search of the next gener- ation of multimodal datasets. ArXiv, abs/2304.14108, 2023. URL https: /tpiiipp...../ htt.a/ap...sssmmmm..e.eeemmeeeme semanticccooolr.  htg/CorrusID: :ttiuuII::283582888
Ho, J., Salimans, T.,, Gritsenko, A., Chan, W., Norouzi, M., and Fleet, J. Video diffusion models. Advances in Neural Information Processing Systems, 35:8633-8646, 2022b. 9
Hong, W., Ding, M., Zheng, W., Liu, X., and Tang, J. Cogvideo: Large-scale pretraining for text-to-video gener- ation via transformers. arXiv preprint arXiv:2205. 15868, 2022. 17
16
Gao, S., Yang. J., Chen, L., Chitta, K., Qiu, Y., Geiger, A., Zhang, J., and Li, H. Vista: A generalizable driving world model with high fidelity and versatile controllabil- ity. arxiv preprint arXiv:2405. 17398, 2024.
Hu, A., Russell, L., Yeo, H., Murez, Z., Fedoseev, G., Kendall, A., Shotton, J., and Corrado, G. Gaia-l: A generative world model for autonomous driving. arXiv preprint arXiv:2309. 70800, 2023. 1,9
Girdhar, R., Gustafson, L., Adcock, A., and van der Maaten, L. Forward prediction for physical reasoning. arXiv preprint 2006v 10734, 2020. 15
Hu, Y., Tang, X., Yang, H., and Zhang, M. Case-based or rule-based: How do transformers do the math? ICML, 2024. 7,2.2
Girdhar, R., Singh, M., Brown, A., Duval, Q., Azadi, S., Rambhatla, S. S., Shah, A., Yin, X., Parikh, D., and Misra, I. Emu video: Factorizing text-to-video gener- ation by explicit image conditioning. arXiv preprint urXiv:2311.10700 2023. 9
Huang, Z., He, Y., Yu, Zhang, F,, Si, C., Jiang, Y., Zhang, Y., Wu, T., Jin, Q., Champaisit, N., et al. Vbench: Compre- hensive benchmark suite for video generative models. In Proceedings of the IEEE/CVF Conference on Computer Vision and Puttern Recognition, pp. 21807-21818, 2024.
Groth, O., Fuchs, F. B., Posner, I., and Vedaldi, A. Shapes- tacks: Learning vision-based physical intuition for gen- eralised object stacking. In Proceedings of the european conference on computer vision (eccv), pp. 702-717, 2018. 15
Isola, P., Zhu, J.-Y., Zhou, T., and Efros, A. A. Image- to-image translation with conditional adversarial net- works. 2017 IEEE Conference on Computer Vi- sion and Pattern Recognition (CVPR), pp. 5967-5976. 2016. URL https: /aapi. www.ppp...aaaa.................e www.nticschooeoe..e..... org CorposID:
Ha, D. and Schmidhuber, J. Recurrent world models facil- itate policy evolution. Advances in neural information processing systems, 31, 2018. 9
16
Hafner, D., Lillicrap, T., Ba, J., and Norouzi, M. Dream to control: Learning behaviors by latent imagination. arXiv preprint arXiv: 1tt.19192........ 2019. 9
Jia, F., Mao, W., Liu, Y., Zhao, Y., Wen, Y., Zhang, C., Zhang, X., and Wang. T. Adriver-i: A general world model for autonomous driving. arXiv preprint arXiv:231 1. 131449 2023. 9
Hafner, D., Lillicrap, T., Norouzi, M., and Ba, Mas- tering atari with discrete world models. arXiv preprint arXiv:2010.02/ 2020. 9
Kaplan, J., McCandlish, S., Henighan, T., Brown, T. B., Chess, B., Child, R., Gray, S., Radford, A., Wu, J., and Amodei, D. Scaling laws for neural language models. arXiv preprint arXiv:2001 000e::2000100361,, 2020.
Hafner, D., Pasukonis, J., Ba, J., and Lillicrap, T. Mastering diverse domains through world models. arXiv preprint
mrXiv:2301.0410. 2023. 19
Khachatryan, L., Movsisyan, A., Tadevosyan, V., Henschel, R., Wang. Z., Navasardyan, S., and Shi, H. Text2video- zero: Text-to-image diffusion models are zero-shot video generators. In Proceedings of the IEEE/CVF Interna- tional Conference on Computer Vision, pp. 15954-15980 2023.
He, C., Shen, Y., Fang, C., Xiao, F. Tang, L., Zhang, Y., Zuo. W., Guo, Z., and Li, X. Diffusion models in low- level vision: A survey. arXiv preprint arXiv:2406. 1138, 2024. 9
Ho, J., Jain, A., and Abbeel, P. Denoising diffusion proba- bilistic models. Advances in neural information process- ing systems, 33:684006851,, 2020. 15
Kingma, D. P. Auto-encoding variational bayes. arXiv preprint arXiv: 14, 2013. 9
Ho, J., Chan, W. Saharia, C., Whang, J., Gao, R., Gritsenko, A., Kingma, D. P., Poole, B., Norouzi, M., Fleet, D. J., et al. Imagen video: High definition video generation with diffusion models. arXiv preprint 2rx0022210002303,, 2022a.
Kondratyuk, D., Yu, L., Gu, X., Lezama, J., Huang, J., Hornung, R., Adam, H., Akbari, H., Alon, Y., Birodkar, V., et al. Videopoet: A large language model for zero-shot video generation. arXiv preprint arXiv: 14125, 2023.

---

How Far is Video Generation from World Model: A Physical Law Perspective
Liao, M., Ye, Q.. Zuo, W., Wan, F,, Wang. T., Zhao, Y., Wang, J., Zhang. X., al. Evaluation of text-to-video generation models: A dynamics perspective. Advances in Neural Information Processing Systems. 37:109790- 109816, 2024. 9
Schrittwieser, J., Antonoglou, I., Hubert, T., Simonyan, K., Sifre, I.., Schmitt, S., Guez, A., Lockhart, E., Hassabis, D., Graepel, T., et al. Mastering atari, go, chess and shogi by planning with learned model. Nature, 588(7839): 604-609, 2020. 9
Lin, S., Liu, B., J., and Yang. X. Common diffusion noise schedules and sample steps are flawed. In Proceedings of the IEEE/CVF winter conference applications of computer vision, pp. 5404-5411 5401, 2024. 16
Silver, D., Huang, A., Maddison, J., Guez, A., Sifre, L., Van Den Driessche, G., Schrittwieser, J., Antonoglou, I., Panneershelvam, V., Lanctot, M., et al. Mastering the game of go with deep neural networks and tree search. nature, 529(7587)444-4935 2016.
Liu, Y., Cun, X., Liu, X., Wang, X., Zhang, Y., Chen, H., Liu, Y., Zeng, T., Chan, R., and Shan, Y. Evalcrafter: Benchmarking and evaluating large video generation models. In Proceedings of the IEEE/CVF Conference Computer Vision and Pattern Recognition, pp. 22139- 22149, 2024.
Song. Y., Sohl-Dickstein, J., Kingma, P., Kumar, A., Er- mon, S., and Poole, B. Score-based generative modeling through stochastic differential equations. arXiv preprint arXiv:2011.13450 2020. 15
Su, J., Ahmed, M., Lu, Y., Pan. S., Bo, W., and Liu, Y. Roformer: Enhanced transformer with rotary position embedding. Neurocomputing, 568: 168:127063, 2024.
Loshchilov, I. Decoupled weight decay regularization. arXiv preprint arXiv:1711.05100 2017. 16
Sun, K., Huang, K., Liu, X., Wu, Y., Xu, Z., Li, Z., and Liu, X. T2v-compbench: A comprehensive benchmark for compositional text-to-video generation. arXiv preprint arXiv:2407. 1.XYv:2407.14555, 2024.
Melnik, A., Schiewer, R., Lange, M.. Muresanu, A., Sacidi, M., Garg, A., and Ritter, H. Benchmarks for physical reasoning ai. arXiv preprint arXiv:2312.10728, 2023. 15
Peebles, W. and Xie, S. Scalable diffusion models with transformers. In Proceedings of the IEEE/CVF Interna- tional Conference on Computer Vision, pp. 4195-4205, 2023. 9, 16
Sutton, R. S. Reinforcement learning: An introduction.
Bradford Book, 2018.
Unterthiner, T., Van Steenkiste, S., Kurach, K., Marinier, R., Michalski, M., and Gelly, S. Towards accurate generative models of video: A new metric & challenges. arXiv preprint
Riveland, R. and Pouget, A. Natural language instructions induce compositional generalization in networks of neu- rons. Nature Neuroscience, 27(5):988-99) 2024. 3, 19
2018. 6
Van Den Oord, A., Vinyals, O., al. Neural discrete rep- resentation learning. Advances in neural information processing systems, 30, 2017. 9
Rombach, R., Blattmann, A., Lorenz, D., Esser, P., and Ommer, B. High-resolution image synthesis with latent diffuuson models. In Proceedings of the IEEE/CVF con- ference computer vision and pattern recognition, pp. 10684-10695, 2022. 9
Wang. W., Yang, H., Tuo, Z., He. H., Zhu, J., Fu, J. and Liu, J. Swap attention in spa- tiotemporal diffusions for text-to-video generation. 2023. URL https: /taps:://ai... semanticscholes org/ NN:
Ronneberger, O., Fischer, P., and Brox, T. U-net: Con- volutional networks for biomedical image segmenta- tion. In Medical image computing and cosputer-assisted intervention 2015: 18th international confer- ence, Munich, Germany, October 5-9, 2015, proceedings, part III 18, pp. 234-241. Springer, 2015. 9
16
Wang. X., Zhang, X., Zhu, Y., Guo. Y., Yuan, X., Xiang, I.., Wang, Z., Ding, G., Brady, D., Dai, Q., and Fang, L. Panda: A gigapixel-level human-centric video dataset. In 2020 IEEE/CVF Conference Computer Vision and Pattern Recognition (CVPR), pp. 3265-3275, 2020. doi: 10.1 10.1009CC24442000200000000 230.0002266000000003333.... 16
Salimans, and Ho. J. Progressive distillation for fast sampling of diffusion models. arXiv preprint arXiv:2202.005// 00000.000552... 2022. 16
Wang, Z., Bovik, A. C., Sheikh, H. R., and Simoncelli, E. P. Image quality assessment: from error visibility to struc- tural similarity. IEEE transactions on image processing, 13(4):600-61) 612, 2004. 6
Schott, L., Von Kügelgen, J., Träuble, F., Gehler, P., Rus- sell, C., Bethge, M., Schölkopf, B., Locatello, F., and Brendel, W. Visual representation learning does not gen- eralize strongly within the same domain. arXiv preprint arXiv:2107.0822/ 2021. 22
Weitnauer, E., Goldstone, R. L., and Ritter, H. Perception and simulation during concept learning. Psychological Review, 2023. 15
12

---

How Far is Video Generation from World Model: A Physical Law Perspective
Wu, J., Lim, J. J., Zhang, H., Tenenbaum, B., and Free- man, W. T. Physics 101: Learning physical object prop- erties from unlabeled videos. In BMVC, volume 2, pp. 7, 2016. 15
Yue, Y., Kang, B., Xu, Z., Huang, G., and Yan, S. Value- consistent representation learning for data-efficient rein- forcement learning. In Proceedings of the AAAI Confer- ence on Artificial Intelligence, 2023. 9
Wu, J. Z., Gc, Y., Wang, X., Lei, S. W., Gu, Y., Shi, Y., Hsu, W., Shan, Y., Qie, X., and Shou, M. Z. Tune-a- video: One-shot tuning of image diffusion models for text-to-video generation. In Proceedings the IEEF/CVF International Conference on Computer Vision, pp. 7623- 7633, 2023. 9
Yue, Y., Wang, Y., Kang, B., Han, Y., Wang. S., Song, S., Feng, J., and Huang, G. Deer-vla: Dynamic inference of multimodal large language models for efficient robot execution. Advances in Neural Information Processing Systems, 2024.
Zeng, Y., Wei, G., Zheng, J., Zou, J., Wei, Y., Zhang, Y., and Li. H. Make pixels dance: High-dynamic vidco generation. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 8850- 8860, 2024.
Xu, K., Zhang, M., Li, Du, S. S., Kawarabayashi, K.-i., and Jegelka, S. How neural networks extrapolate: From feedforward to graph neural networks. arXiv preprint arXiv:2009. / 1848, 2020. 6
Zhang, R., Isola, P., Efros, A. A., Shechtman, E., and Wang, O. The unreasonable effectiveness of deep features as perceptual metric. In Proceedings of the IEEE conference on computer vision and pattern recognition, pp. 586-595, 2018.
Xue, C., Pinto, V., Gamage, C., Nikonova, E., Zhang, P., and Renz, J. Phy-q as measure for physical reasoning intelligence. Nature Machine Intelligence, 5(1):83-93, 2023. 15
Xue, T., Chen, B., Wu, J., Wei, D., and Freeman, W. T. Video enhancement with task-oriented flow. International Journal of Computer Vision, pp. 1-20, 2017. URL sttp:::://p...mmattiii.ee otg/CorrusID::04111
Zhang, S., Wang, J., Zhang, Y., Zhao, K., Yuan, H., Qin, Z., Wang, X., Zhao, D., and Zhou, J. 12vgen-xl: High-quality image-to-vider synthesis via cascaded diffusion models. arXiv preprint arXiv:2311.04145, 2023a. 9
16
Zhang, Y. Wei, Y., Jiang, D., Zhang, X., Zuo. W., and Tian, Q. Controlvideo: Training feaming-free controllable text-to-video generation. arXiv preprint arXiv:2305. /3077, 2023b. 9
Yang, J., Gao, S., Qiu, Y., Chen, L., Li, T., Dai, B., Chitta, K., Wu, P., Zeng, J., Luo, P., et al. Generalized predictive model for autonomous driving. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 14662-14672, 2024.
Zheng, W., Song, R., Guo, X., and Chen, L. Genad: Gen- erative end-to-end autonomous driving. arXiv preprint arXhv:240.111502 2024.
Yang, M., Du, Y., Ghasemipour, K., Tompson, J., Schuur- mans, D., and Abbeel, P. Learning interactive real-world simulators. arXiv preprint arXiv:2310.061 14, 2023. 1,
Yi, K., Gan, C., Li, Y., Kohli, P., Wu, J., Torralba, A., and Tenenbaum, J. B. Clevrer: Collision events for video representation and reasoning. arXiv preprint arXiv:1910.01442. 2019. 15
Yu, L.,, Cheng, Y., Sohn, K., Lezama, J., Zhang, H., Chang, H., Hauptmann, A. G., Yang, M.-H., Hao, Y.. Essa, I. et al. Magvit: Masked generative video transformer. In Proceedings of the IEEF/CVF Conference on Computer Vision and Pattern Recognition, pp. 10459-10469, 2023a.
Yu, L., Lezama, J., Gundavarapu, N. B., Versari, L., Sohn, K., Minnen, D., Cheng, Y., Gupta, A., Gu, X., Haupt- mann, A. G., et al. Language model beats diffusion- tokenizer is key to visual generation. arXiv preprint arXiv:23 2023b. 3, 9, 16
13

---

How Far is Video Generation from World Model: A Physical Law Perspective
Appendix Contents
More related works
15
Latent Video Diffusion Model
15
B.1
Diffusion preliminaries
15
B.2
VAE Architecture and Pretrain
16
B.3 VAE Reconstruction...
16
B.4 DiT Implementation Details
16
Detailed Experimental setup
17
C.I Fundamental Physical Scenarios Data
17
C.2 Combinatorial Experiments Evaluation Setup
17
D Experiments SOTA Video Generation Models
17
D.1 Is the Prioritization Relevant to VAE?
17
D.2 Can pretrained models learn physical laws?
18
More Experiments and Discussions
19
E.1 Can Language and Numerics Aid in Learning Physical Laws?
19
E.2 Continuous Experiment for Pairwise Comparison
19
E.3 Principle Behind Data Retrieval in the Diffusion Model
20
E,4 Failure Cases in Combinatorial Generalization
21
Comparison with ID/OOD Generalization Works
22
Visualization
23
14

---

How Far is Video Generation from World Model: A Physical Law Perspective
Spatial combinatorial generalization
Temporal combinatorial generalization
train data
test
train data
test
uniform motion only
only bounce
ground truth
collision only
bounce only
ground truth
prediction
prediction
Figure Spatial temporal combinatorial generalization. The subsets of training set contain disjoint physical events. However, the trained model can combine these two types of events across spatial and temporal dimensions.
A. More related works
Physical reasoning. It refers to the ability to understand and predict the way objects will interact under certain conditions according physical laws. (Melnik al., 2023) categorize physical reasoning tasks into passive and interactive tasks. Passive tasks often require the AI predict certain properties of objects, e.g., materials (Bell et al., 2015; Bouman et al., 2013), physical parameters (Wu al., 2016), stability of physical system (Groth al., 2018), 1),, or involve question-answeringg for the agent to recognize conceptual differences (Weitnauer et al., 2023; Yi al., 2019), or describe physical scenario (Ates et al., 2020). For interactive tasks, AI is required to control some objects in the environment to complete certain tasks based on physical commonsense, e.g., solving classical mechanics puzzles (Bakhtin et al., 2019), flying a bird to reach target position (Xue et al., 2023), and using tool (Allen et al., 2020). Two works closely related to ours are Girdhar et al. (2020) and Silva al. (2020). Girdhar et al. (2020) introduce a forward prediction model to aid physical reasoning address what learned in the prediction model. de Silva al. (2020) attempt discover universal physical laws from data with abstracted internal states and human expertise introduced in the dynamic model design. Some works also aim to incorporate physical knowledge into video generation models through manually designed modules (Cao al., 2024). In contrast, we focus on recovering physical laws from raw observation without any human priors, akin to newborn baby.
do not
B. Latent Video Diffusion Model
B.1. Diffusion preliminaries
Let p(z) be real data distribution. Diffusion models (Ho et al., 2020; Song et al., 2020) learn the data distribution by denoising samples from a noise distribution step-by-step. In this paper, we use the Gaussian diffusion models, where the video V is progressively corrupted by gaussian noise ~N I) during the forward process, denoted by
(I) where Ct, 3r are the time- dependenn noise scheduler. We use the original DDPM formulation (Ho et al., 2020), where 0t = Vtt. 3 VI - Yts 7t. monotonically decreasing scheduler from 0. The diffusion models are trained to
V = a. B,E,
15

---

How Far is Video Generation from World Model: A Physical Law Perspective
reverse the forward corruptions, denoted by
(lyy-po(Ve [|ly - Po c, co, c,
where the target y can be the corrupted noise E, the original video V, or the velocity between data and noise. Following (Sal- imans & Ho, 2022), we use the velocity prediction train the video diffusion models, denoted by
y 7/
VTV.
(3)
Also, to eliminate the training and inference gap and ensure a zero signal-to-noise ratio at the final timestep, following (Lin et al., 2024), we set "/t to 1 when t = 1.
B.2. VAE Architecture and Pretrain
We commence with the structure of the SD1.5-VAF, retaining the majority of the original 2D convolution, group normaliza- tion, and attention mechanisms on the spatial dimensions. To inflate this structure into spatial-tempord auto-encoder, we convert the final few 2D downsample blocks of the encoder and the initial few 2D upsample blocks of the decoder into 3D ones, and employ multiple extra ID layers to enhance temporal modeling. For all the downsample blocks where temporal downsampling is required, we replace all the original 2D downsample layers with re-initialized causal 3D downsample layers by adding causal paddings to the head of the frame sequence (Yu et al., 2023b) and introduce an additional causal ID convolution layer after the original ResNetBlock. As for the decoder part, all the 2D Dearest-Interpolee substituted by a 2D convolution layer and a channel-to-space transformation, which behave precisely the same as Pearest-Interpolatiin operations before the first training step. For the discriminator part, we inherit the structure of the original 2D PatchGAN (Isola et al., 2016) discriminator used by SDI.5 and design a 3D PatchGAN discriminator based on the 2D version. Different from the generator module, we train the group of discriminators from scratch for the consideration of stability. Subsequently, all parameters of the (2+1)D-Ve VAE D--VEE are jointly trained with high-quality image and video data to preserve the capability of appearance modeling and to enable motion modeling. For the image dataset, filter data samples from LAION-Aesthetics (Beaumont & Schuhmann, 2022), COYO (Byeon et al., 2022) and DataComp (Gadre et al., 2023) with high aestheties and clarity to form a high-quality subset. As for the video dataset, we collect a high-quality subset from Vimeo-90K (Xue et al., 2017), Panda- 70M (Wang et al., 2020) and HDVG (Wang et al., 2023). In the training process, we train the entire structure for 1M steps and only the random resized crop and random horizontal flip are applied in the data augmentation process.
operations specifically initialized to
B.3. VAE Reconstruction
In this paper, we wefix the pretrained VAE encoder and use it as a video compressor. To verify the VAE's ability to accurately encode and decode the physical event videos, we evaluate its reconstruction performance. Specifically, we use the VAE to encode and decode (i.e., reconstruct) the ground truth videos and calculate the reconstruction error, érecon We then compare this error the ground truth error, Egt, as shown in Table 3. The results show that Green is very close to egs and both are an order of magnitude lower than the OOD error, illustrated in Figure 3. It confirms the pretrained VAE's ability to accurately encode and decode the physical event videos used in this paper.
Table 3: Comparison of errors for ground truth videos and VAE reconstruction videos.
Scenario
Ground Truth Error
VAE Reconstruction Error
Uniform Motion
0.0099
0.0105
Collision
0.0117
0,0131
Parabola
0.0210
0.0212
B.4. DiT Implementation Details
Following Sora (Brooks et al., 2024), we directly use the DiT architecture (Peebles & Xie, 2023) as the denoising model. The only modification is that we employ a 3D variant of RoPE to better fit the video data. Throughout the paper, we use maximum learning rate of x 10-4 with cosine decay, a bach size of 256. All models are trained with the AdamW (Diederik & Ba, 2015; Loshchilov, 2017) optimizer, with 81 = 0.9, B2 = 0.999 and the weight decay weight is equal to 0.01. Horizontal flipping is the only data augmentation not specified.
16

---

How Far is Video Generation from World Model: A Physical Law Perspective
C. Detailed Experimental setup
C.1. Fundamental Physical Scenarios Data
For the Box2D simulator, we initialize the world as a 10 x 10 grid, with a timestep of 0.1 seconds, resulting in a total time span of 3.2 seconds (32 frames). For all scenarios, we set the radius r E [0.7., 1.5] and velocity E [1, 4] as in-distribution (in-dist) ranges. Out-of-distribution (OOD) ranges are defined as E [0.3, 0.6] U [1.5, and u [0. 0.8] u [4.5, 6.0).
Collision Scenario:
The four degrees of freedom (DoFs) are the masses of the two balls and their initial velocities, fully determining the collision outcomes. We generate 3k, 30k, and 3M training samples by sampling grid points from the 4-dimensional in-dist joint space of radii and velocities. For in-dist evaluation, we randomly sample about 2k points from the grid, ensuring they are not part of the training set. For OOD evaluation, we sample from the OOD ranges, generating approximately 4.8k samples across six OOD levels: (1) only r1 OOD, (2) only V OOD, (3) both r1 and r2 OOD, (4) both Vi and 22 OOD, (5) ri and 2 OOD, and (6) r1. U1 r2, and 122 OOD. Additionally, for collisions, we ensure that all collisions occur after the 4th frame in each video, allowing the initial velocities of both balls to be inferred from the conditioned frames.
Uniform and Parabolic Motion: The two DoF' are the ball's mass and initial velocity. We generate 3k, 30k, and 3M training samples by sampling from the 2-dimensional in-dist joint space of radius and velocity. For in-dist evaluation, we sample approximately 1.05k for uniform motion and 1.1k for parabolic motion. For OOD evaluation, we generate about 2.4k (uniform motion) and 2.5k (parabolic motion) samples across three OOD levels: (1) only r OOD, (2) only v OOD, and (3) both r and v OOD.
C.2. Combinatorial Experiments Evaluation Setup
To evaluate the generated videos, we performed human evaluations to determine the abnormal ratio, defined as the proportion of videos that violate physical laws as assessed by human intuition. We utilized 60 templates for training and reserved 10 unused templates for evaluation. For the in-template evaluation set, we sampled 2 videos from each of the 60 training templates, forming total of 120 videos (excluded from training). For the out-of-template evaluation set, we sampled 10 videos from each of the 10 reserved templates, forming total of 100 videos. We recruited 10 human evaluators to assess the videos. Each evaluator independently reviewed all vidcos and determined whether the physical processes depicted adhered to the physical laws. We average the abnormal rates over all the evaluators and report the result. Each evaluator received the following instruction: "For each video shown to you, all objects start with free fall. You should judge whether the physical processes in the video obey physical laws based on your intuition. Select 'Yes' if the process adheres to physical laws and 'No' otherwise. Please make your judgment based on your first impression".
D. Experiments on SOTA Video Generation Models
In this section, we explore how the findings in this paper apply to open-source pretrained models, such as Cog Video (Hong et al., 2022) and Stable Video Diffusion (SVD) (Blattmann et al., 2023).
D.I. Is the Prioritization Relevant to VAE?
To evaluate how much the prioritization order color > size > velocity > shape is influenced by the explicit instantiation of the video VAE used in our work, we perform experiments with alternative architectures. Specifically, we replace the VAE from our primary experiments with the VAE from the recently released CoGVideo and rerun the tests. By freezing the CoGVideo VAE and training only the diffusion component, we find that the prioritization order remains consistent.
As detailed in Section 5.3, we designed experiments to compare the prioritization of velocity and shape. Videos of balls with low velocities and squares with high velocities were used for training. During testing, we evaluated scenarios where ball with high velocity transformed into a square immediately after the conditioned frames, and square with low velocity transitioned into a ball. Across 1,400 test cases, no exceptions were observed. The results are identical to those presented in Figure 8 (3), confirming that the prioritization order velocity > shape holds. For velocity and size, as shown in Figure 12 (1), the model consistently maintained the initial size and velocity for most test cases, even beyond the training distribution. However, slight preference for size over velocity was observed, particularly when radius and velocity values were at extreme ends (top-left of the plot). These results confirm that size holds higher priority than velocity. Finally, for the prioritization between color and velocity, we trained the model on high-speed blue balls and low-speed red balls. During
17

---

How Far is Video Generation from World Model: A Physical Law Perspective
testing, high-speed red balls appeared much slower than their conditioned velocity, while balls in the test set changed their color. As illustrated in Figure 12 (2), this demonstrates that color is prioritized over velocity. In summary, our experiments with the CoGVideo VAE confirm the robustness of the prioritization order color > size >> velocity > shape, demonstrating that these findings generalize across different VAE pretrain.
Condition
trainset
Error
Law of Inertia
red ball
blue ball
red trainset
blue trainset
4.0
1.40
""A
AM
Radius of generated video
3.5
14
47
1.22
3.0
2.5
... ...
1.05
2.0
1.5
0.87
1.0
0.8
1.0
1.2
1.4
0.70
0.87
1.05
1.22
1.40
Radius
Radius conditioned frames
Figure 12: Prioritization experiments with CogVideo' VAE (1) Velocity vs. size. (2) Color velocity.
D.2. Can pretrained models learn physical laws?
The primary goal of this paper is to answer the fundamental question: "Can video generation model learn physical laws?" Existing pretrained models are built on large-scale datasets sourced from publicly available internet content. However, the extent to which these datasets contain videos about physical laws or structured physical interactions remains unknown. This lack of transparency leads to critical concern - contamination of evaluation data: There is a risk that pretrained models may have been exposed to data similar to our evaluation experiments during training. This potential overlap undermines the validity of assessing their ability learn physical reasoning. To address these concerns, we chose to train model from scratch, which provides complete control over the training data. This approach allows to precisely ensure that the results stem from the model's actual learning process rather than potential pretraining data overlap contamination. This methodology is supported by recent research into the mechanisms of language models, where training relatively small models from scratch has proven effective in isolating and analyzing reasoning processes more rigorously.
To assess whether the inability to learn physical laws is specific to our setup or extends to open-source pretrained models, we conducted experiments using the Stable Video Diffusion (SVD) model on a representative uniform motion task. We fine-tuned SVD for 300k steps, following the same training setup as DiT-B in our experiments. The results, summarized in Table 4, reveal several key observations. First, SVD's VAE demonstrates very small reconstruction error, indicating that its VAE component effectively preserves kinematic information. However, the diffusion component of SVD, despite pretraining, performs worse than DiT-B in both in-distribution (ID) and outto--distributiin (OOD) predictions. We attribute this to the considerable gap between the pretraining dataset (internet videos) and the simplified kinematic test scenarios used in this evaluation. Notably, the OOD error for SVD is an order of magnitude larger than its ID error, a trend consistent with DiT-B. This finding reinforces the robustness of our conclusion that pretraining alone does not address the inability video generation models to generalize OOD scenarios, a limitation that persists across architectures, including pretrained models like SVD.
Table 4: Results of finetuning SVD model.
Model
ID Error
OOD Error
GT
0.0099
0.0104
DiT-B (from scratch, ours)
0.0138
0,3583
BED-VAE-Rcoom
0.0103
0.0107
0.0505
0.9081
18

---

How Far is Video Generation from World Model: A Physical Law Perspective
E. More Experiments and Discussions
E.1. Can Language and Numerics Aid in Learning Physical Laws?
As discussed in Section 3, video generation based solely on image frames fails to learn physical laws, showing significant prediction errors in OOD scenarios, despite containing all the necessary information. reinforcement learning, numerical values (e.g,, states) are often used as conditions for world models (Hafner al., 2023), and language representations have shown generalization capabilities in LLMs (Riveland & Pouget, 2024). This raises the question: can additional multimodal inputs, such numerics and text, improve video prediction and capture physical laws? We experimented with collision scenarios and DiT-B models, adding two variants: one conditioned on vision and numerics, and the other on vision and text. For numeric conditioning, we map the state vectors to embeddings and add layer-wise features to video tokens. For text, we converted initial physical states into natural language deseriptions, obtained text embeddings using a T5 encoder, and then added a cross-attention layer to aggregate textual representations for video tokens.
Collision w. DiT-B
GT
0.55
ID:
Velocity Error
vision
0.35
vision+numeric
0.15
vision+text
OOD:
0.03
vision
0.02
vision+numeric
0.01
0
vision+text
30k
300k
3m
Figure 13: Comparison of different modal conditions for video generation.
As shown in Figure 13, for in-distribution generalization, adding numeric and text conditions resulted in prediction errors comparable to using vision alone. This suggests that visual frames already contain sufficient information for accurate predictions, and the additional numeric or text data do not provide further benefits. However, in OOD scenarios, the vision- plus-numerics condition exhibited slightly higher errors, while the vision- Aasion-plus-languaee condition showed significantly higher errors. We hypothesize that the embeddings of language tokens and numerics may cause the model to overfit to specific patterns present in the training data, thereby impairing its ability to generalize to unseen OOD scenarios. Additionally, language tokens are discrete and exhibit greater variability compared to continuous numeric embeddings, making the model more susceptible to overfitting when using language inputs, accounting for the higher OOD errors in the vision-plus-languaa condition compared to the vision-plus-numeric, condition.
E.2. Continuous Experiment for Pairwise Comparison
We also conducted attribute comparison experiments using continuous values for velocity and size to quantify these prioritizations. For velocity size, the combinatorial generalization performance is surprisingly good. The model effectively maintains the initial size and velocity for most test cases beyond the training distribution. However, slight preference for size over velocity is noted, particularly with extreme radius and velocity values (top left and bottom right in Figure 14 (1)). In Figure 14 (2), color can be combined with size most of the time. Conversely, for color vs. velocity in Figure/ 14 (3), high-speed blue balls and low-speed red balls are used for training. At test time, low-speed blue balls appear much faster than their conditioned velocity. No ball in the test set changes its color, indicating that color is prioritized over velocity. The above analysis is consistent with the conclusion drawn from the binary attribute comparisons: color > size* > velocity > shape.
To confirm that the shift observed in Figure 14 is due in fact to attribute prioritization and not to general behavior, we performed control experiments corresponding to the main experiments shown in panels (2) and (3) of Figure 14. These experiments were designed without an attribute conflict (that is, only the red color was used). As shown in Figure 15, even extreme cases, the predictions remained accurate without any change. This result indicates that the drift observed
19

---

How Far is Video Generation from World Model: A Physical Law Perspective
Condition
trainset
Errr
Law Inertia
red
ball
blue ball
red trainset
blue trainset
4.0
/1
Radius of generated video
1.40
Velocity of generated video
4.0
3.5
/
3.0
1.22
3.25
Velocity
2.5
1.05
2.0
0.87
1.75
1.5
1.0
1.0
1.2
1.4
0.70
0.87
1.05
1.22
1.40
1.0
1.75
2.5
3.25
4.0
Radius
Radius of conditioned frames
Velocity of conditioned trames
Figure 14: Uniform motion. (1) Velocity v.s. size: The arrow + indicates the direction generated videos shifting from their initial conditions. (2) Color v.s. size: Models are trained with small red balls and large blue balls, and evaluated on reversed color-size pair conditions. All generated videos retain the initial color but show slight size shifts from the original. (3) Color velocity: Models are trained with low-speed red balls and high-speed blue balls, and evaluated reversed color-velocity pair conditions. All generated videos retain initial color but show large velocity shifts from the original.
in Figure 14 arises specifically from the attribute conflict and the presence of attribute prioritization.
Law of Inertia
red trainset
red ball
1.40
Radius of generated video
Velocity generated video
4.0
1.22
3.25
1.05
2.5
0.87
1.75
0.70
0.87
1.05
1.22
1.40
1.0
1.75
2.5
3.25
4.0
Radius of conditioned frames
Velocity of conditioned frames
Figure 15: Control experiment for pairwise comparison. These experiments were designed without attribute conflict (i.c., only the red color).
E.3. Principle Behind Data Retrieval in the Diffusion Model
As discussed Section 5.3, the diffusion model appears to rely more on memorization and case-based imitation, rather than abstracting universal rules for OOD generalization. The model exhibits preference for specific attributes during case matching, with prioritization order of color size > velocity shape. In this section, aim explore the reasoning behind this prioritization.
Since the diffusion model is trained by minimizing the loss associated with predicting VAE latent, we hypothesize that the prioritization may be related to the distance in VAE latent space (though we use pixel space here for clearer illustration) between the test conditions and the training set. Intuitively, when comparing color and shape as in Figure (I), a change in shape from ball rectangle results in minor variation of the pixels, primarily at the corners. In contrast, color change from blue red causes more significant pixel difference. Thus, the model tends preserve color while allowing the shape to vary. From the perspective of pixel variation, the prioritization of color >> size > velocity shape can be explained by the extent of pixel change associated with each attribute. Changes in color typically result in large pixel variations because it affects nearly every pixel on its surface. In contrast, changes in size modify the number of pixels, but do not drastically alter the individual pixels' values. Velocity affects pixel positions over time, leading to moderate variation as the object shifts, while shape changes often involve only localized pixel adjustments, such as at edges or corners Therefore, the model prioritizes color because causes the most significant pixel changes, while shape changes are less impactful terms of
20

---

How Far is Video Generation from World Model: A Physical Law Perspective
pixel variation.
To further validate this hypothesis, designed variant experiment comparing color and shape, shown in Figure 16, In this case, we use blue ball and red ring. For the ring to transform into the ball without changing color, it would need to remove the ring's external color, turning it into blank space, and then fill the internal blank space with the ball's color, resulting in significant pixel variation. Interestingly, in this scenario, unlike the previous experiments shown Figure 14 (I), the prioritization of color j. shape does not hold. The red ring can transform into either red ball or blue ring, as demonstrated by the examples. This observation suggests that the model's prioritization may indeed depend on the complexity of the pixel transformations required for each attribute change. Future work could explore more precise measurements of these variations in pixel VAE latent space to better understand the model's training data retrieval process.
color
Straining
evaluation
blue
red
O
shape
time
Figure 16: Uniform motion. Color vs. shape. The shapes ball and a ring. Transforming from ring ball leads to large pixel
variation.
E.4. Failure Cases in Combinatorial Generalization
Temporal combinatorial generalization
train data
failure case
collision only
bounce only w.o. red bull
ground truth
prediction
Figure 17: Failure in combinatorial generalization. Note that the bounce in the training do include the red ball.
21

---

How Far is Video Generation from World Model: A Physical Law Perspective
In Figure 11, present successful cases of spatial and temporal combinatorial generalization, where the trained model combine these two types of events in spatial and temporal dimensions generate videos under unseen conditions. However, the model does not always succeed in performing such compositions, and here illustrate some failure cases. As shown in Figure 17, when the training set lacks a red ball in bounce event, model sometimes struggles. In the -oder-ggeerated video, the red ball sometimes disappears after a collision. This likely stems from the model's data retrieval mechanism: since the training set does not include a red ball a collision scenario, when collision happens, the model retrieves similar training cases without allows the diffusion model to generate novel videos by composing spatial and temporal segments from the training set, its reliance data retrieval limits its effectiveness. As result, the model may produce unrealistic outcomes by retrieving and combining segments without understanding the underlying rules.
ball, causing it to vanish post-collision.
summary, while combinatorial generalization
10"
Loss
10
0K
50K
100K
150K
200K
300K
Steps
Figure 18: Training loss Equation (2) curve of DĪT-L model trained on 3M collision videos.
F. Comparison with ID/OOD Generalization Works
ID/OOD generalization (Schott et al., 2021) in machine learning has been extensively discussed foundational works. These discussions highlight the fundamental challenges of OOD generalization in machine learning. However, when applied to specific settings with unique data forms, distributions, and model inductive biases, it exhibits nuanced behaviors that remain scientifically significant. For instance, similar debates have emerged in the study of large language models (LLMs), where considerable attention has been devoted to determining whether models learn underlying rules rely case-based memorization for performing arithmetic (Hu et al., 2024). These insights have spurred advancements in rule-based reasoning approaches within the LLM community. Similarly, our work investigates how video generation models handle generalization within their own domain.
Our findings go beyond the simplistic observation that ID scenarios succeed while OOD scenarios fail. Instead, we provide deeper insights and discoveries specific to video generation:
1. Addressing the Debate on Learning Physical Laws in Video Generation Models: Through systematic experiments, we provide clear answer whether physical laws can be learned by scaling video generation models. This has been a topic of debate in the video generation community. For instance, OpenAl's Sora Technical Report (Brooks al., 2024) suggests that video generation models learn rules and act as world simulators. implying they might generalize universal laws. Our findings challenge this assumption, showing that current video generation models fail to learn physical laws as universal rules.
2. Scaling Guidance for Combinatorial Generalization: We demonstrate the importance of combinatorial generalization and identify scaling laws for improving generalization in video generation models. Our findings highlight that increasing the diversity of combinations the training data more effective for achieving realistic physics than merely scaling the data volume or model size. This offers actionable guidance for building video generation models that better capture physical realism.
22

---

How Far is Video Generation from World Model: A Physical Law Perspective
3. Revealing the Generalization Mechanism and Understanding the Boundaries of Video Generation Models: We uncover how video generation models generalize, primarily relying referencing similar training examples rather than learning underlying universal principles. This provides deeper understanding of their limitations and biases in representing physical phenomena.
These findings transcend traditional ID/OOD analyses, offering a more detailed understanding of the limitations and potential of video generation models.
G. Visualization
uniform motion
collision
parabola
Figure 19: The visualization of in-distribution evaluation cases with very small prediction errors.
23

---

How Far is Video Generation from World Model: A Physical Law Perspective
uniform motion
collision
parabola
Figure visualization out-of--isttribution evaluation cases with large prediction errors.
w7
Figure 21: The visualization of out-of-template evaluation cases that appear plausible and adhere to physical laws, generated by DiT-XI. trained on 6M data (60 templates). Zoom in for details. Notably, the first four cases generated by the model are nearly identical to the ground truth. In some cases, such the rightmost example, the generated video seems physically plausible but differs from the ground truth due visual ambiguity, discussed Section 5.5.
24

---

How Far is Video Generation from World Model: A Physical Law Perspective
The visualization of oet-oattemplate evaluation cases that appear abnormal and violate physical laws, generated by DIT-XL. trained on 6M data (60 templates). Abnormalities are highlighted with red dotted circles. Case I: A grey ball suddenly disappears. Case 2: The rigid-body bar breaks in several intermediate frames during contact with the ball, then recovers after contact. Case 3: The rigid-body jar fails to maintain its shape when interacting with the bar in several intermediate frames. Case 4: The rigid-body bar breaks several intermediate frames during contact with the standing sticker. Most of the abnormal cases we observed involve object disappearance or shape inconsistencies, which can be explained by the case matching preference discussed in Section 5.3.
Figure 22:
25

---

How Far is Video Generation from World Model: A Physical Law Perspective
(1)
(2)
(3)
(4)
Figure 23: More visualization of out-of-template evaluation cases (1) that appear abnormal and violate physical laws. Case 5: The gray ball passes through a rigid-budy bar. Case A cross stick first falls the ground, then violates gravity by returning standing position. Case 7: A small gray ball disappears in 3rd frame and reappears in different location the 5th frame, resembling form "teleportation", Case 8: One end a rigid-body body bar remains suspended in mid-air, violating expected gravitational behavior.
26

---

How Far is Video Generation from World Model: A Physical Law Perspective
(5)
(6)
(7)
(8)
Figure 24: More visualization of out-of-template evaluation cases (2) that appear abnormal and violate physical laws. Case 9: Two balls. one large and one small. collide and then exchange positions in physically implausible manner. Case 10: A small jar remains floating in midair, defying gravity. Case 11: rigid-body bar appears to split into separate pieces. Case 12: A short rigid-body bar visible in earlier frames but suddenly disappears in the 4th frame. 27


Reviews:
[{"review_id": "ttSdttX9Nd", "reviewer": "Reviewer_WMWe", "summary": "This paper evaluates if scaling video generation models enables learning physical laws. Using a 2D simulator for object motion governed by classical mechanics, experiments reveal: (1) near-perfect in-distribution (ID) generalization with scaling, (2) failure in out-of-distribution (OOD) scenarios despite scaling, and (3) improved combinatorial generalization via increased data diversity. Models exhibit \"case-based\" generalization, prioritizing attributes (color > size > velocity > shape) rather than abstract rules. Key contributions include systematic scaling analysis and insights into model biases.", "questions": "", "limitations": "", "rating": 4, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thanks for supporting the acceptance of our work. Please find our responses to your questions below.\n\n`1. pixel-level metrics (SSIM/PSNR) may not fully capture physical plausibility; over-reliance on human evaluation for combinatorial cases`\n\nThank you for your comment. We agree that pixel-level metrics alone are insufficient to fully capture physical plausibility. However, we relied on a combination of objective video fidelity metrics and human evaluation:\n\n1. **FVD, SSIM, PSNR, and LPIPS measure fidelity of generated videos to ground-truth videos** governed by physical laws. While not exlicitly designed for physical correctness, they reflect plausibility to some extent by measuring consistency and realism with the groundtruth —an approach consistent with prior work like Genie [1].\n\n2. As you pointed out, fidelity alone may not fully represent physical plausibility. To address this, we also conducted a human evaluation, where each evaluator was specifically instructed to **focus on assessing violations of physical laws**.\n\nWe believe this combined approach offers a comprehensive evaluation for physical correctness in the complex setting. \n\n[1] Bruce, Jake, et al. \"Genie: Generative interactive environments.\" ICML 2024, Best paper.\n\n`2. fixed VAE usage limits exploration of end-to-end training benefits`\n\nThanks for your question. Here we explain why we chose to fix the VAE and why doing so does not limit the performance of the diffusion model in our setting:\n\n1. **Training stability**: During diffusion training, the VAE is used to define the training objective. Updating the VAE during diffusion training makes the latent space unstable, slowing convergence. Hence, widely used architectures like Stable Diffusion 3, DALL·E 3, and Hunyuan Video pretrain and fix the VAE.\n\n2. **The VAE is not a bottleneck in our setup**. As shown in Appendix A.3.2,  we validate that VAE reconstructions of the input videos exhibit minimal error, indicating that most inaccuracies arise from the diffusion model rather than the VAE. This was also positively noted by Reviewer hEH6.\n\nWe hope this explanation clarifies our rationale and addresses your concerns about fixing the VAE.\n\n`3. The prioritization hierarchy (color > size > velocity > shape) is tested only in controlled scenarios;  testing only three physical scenarios (linear motion, collision, parabola) limits generalizability; Limited scenario diversity (2D, synthetic data); broader validation (e.g., real-world textures) is needed to confirm universality;`\n\nThank you for your valuable insights. We use simplified synthetic scenarios for the following reasons:\n\n1. **Abundant and Controllable Data**: Synthetic settings enable large-scale, controlled data generation, allowing systematic study of specific physical principles. **Defining settings like ID/OOD or combinatorial generalization is challenging in real-world datasets**.\n2. **Isolated Physical Laws**: Each synthetic scenario is governed by a single, well-defined kinematic law. In contrast, real-world videos often involve multiple entangled factors (e.g., unknown friction, unobservable forces), making it hard to attribute behavior to specific laws.\n\n3. **Measurable Physical Quantities**: In our controlled setup, physical quantities like velocity and mass can be reliably extracted from video frames. In real-world scenarios, such values are often unobservable, making it hard to verify whether generated videos obey physical laws.\n\nBy simplifying the rendering process, we isolate core challenges in learning physical dynamics, making our experiments quantitatively tractable and our findings interpretable.\n\nHowever, we agree that broader validation with realistic data is important for future work. This would require great effort in collecting and curating controllable real-world data, and developing new metrics for evaluating physical consistency.\nWe appreciate your suggestions and welcome further discussion.\n\n`4. minimal discussion of real-world applicability.`\n\nOur work focuses on scientific insights, particularly regarding the underlying mechanisms of generalization in physical video modeling. The insights from the paper can inform such real-world scenarios:\n\nReal-world video data is vast, continuous, and high-dimensional, making it more difficult to fully cover than pure language. For example, In real-world robotics, variables such as object velocity, joint configurations, camera angles, noisy backgrounds, and task goals vary across a continuous and combinatorially large space.\n\n**Our paper contributes actionable insights to this challenge**: We show that **scaling combinatorial diversity** in the training data—rather than simply increasing dataset size—is significantly more effective for improving physical video modeling. And it also implies that Model Scaling is Effective When Supported by Diversity.\n\nWe will include this discussion in the revision.\n\n`Summary`\n\nWe hope our responses improved your confidence in our paper.", "claims_and_evidence": "Claims are largely supported. OOD failure is evidenced by consistent high errors across scaling levels. Case-based generalization is validated through experiments with flipped training data and attribute conflicts. However, the prioritization hierarchy (color > size > velocity > shape) is tested only in controlled scenarios; broader validation (e.g., real-world textures) is needed to confirm universality.", "methods_and_evaluation_criteria": "The 2D simulator effectively isolates physical variables, and velocity error metrics align with the goal of assessing physical law adherence. Human evaluations for combinatorial cases add robustness. However, pixel-level metrics (SSIM/PSNR) may not fully capture physical plausibility, and fixed VAE usage limits exploration of end-to-end training benefits.", "theoretical_claims": "No formal theoretical claims are made. The framework for evaluating generalization (ID/OOD/combinatorial) is conceptual but well-defined. The empirical focus aligns with the paper’s goals.", "experimental_designs_or_analyses": "Scaling experiments (model/data size) are rigorous, and controlled attribute comparisons clarify prioritization. However, testing only three physical scenarios (linear motion, collision, parabola) limits generalizability.", "relation_to_broader_scientific_literature": "Connects strongly to prior work on world models and LLM memorization. Challenges assumptions in video generation (e.g., Sora’s physical reasoning claims) by demonstrating OOD limitations. Advances understanding of scaling’s role in combinatorial generalization, aligning with trends in foundation models.", "essential_references_not_discussed": "N/A", "other_strengths_and_weaknesses": "Novel insights into case-based generalization; actionable scaling guidelines for combinatorial diversity; clear challenge to prevailing narratives about video models as world simulators. Weaknesses: Limited scenario diversity (2D, synthetic data); over-reliance on human evaluation for combinatorial cases; minimal discussion of real-world applicability.", "other_comments_or_suggestions": "N/A", "questions_for_authors": "N/A", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "KjiY2xEwvO", "reviewer": "Reviewer_PdyP", "summary": "The authors create a benchmark to evaluate the physical understanding of large video models at scale. Specifically, they measure the generalization performance of the model under variations of physically meaningful quantities such as color, shape, size and velocity.", "questions": "", "limitations": "", "rating": 4, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for your thorough review and for supporting the acceptance of our work.\n\n\nWe appreciate you pointing out these relevant works on evaluating the physical understanding of large video models. We agree that incorporating them will help position our contributions within the broader landscape of physical evaluation in video generation research. In the revised version of the paper, we will include a discussion of these works—Videophy, T2V-CompBench, Devil, VBench, and EvalCrafter—in the Related Work section under the “Video Generation” subsection.", "claims_and_evidence": "They claim that the model does not integrate fundamental physical laws which they demonstrate by showing that scaling the dataset size and parameter number enables the model to generalize inside the training distribution but not outside. They claim that the model has a priority order when generalizing color > size > velocity > shape which they demonstrate by switching attribute between training and testing.", "methods_and_evaluation_criteria": "The author build there own benchmark. The benchmark is a class of 2D physics environments of multiple objects colliding and parameterized by shape, color, size and velocity which enables to isolate the evaluation of the physics understanding.", "theoretical_claims": "N/A", "experimental_designs_or_analyses": "They use a significant number of 100 to 1000 test cases per experiment.", "relation_to_broader_scientific_literature": "Evaluating the adherence of large models to physical laws is a very important direction, given the wide adoption of these models in the industry.", "essential_references_not_discussed": "There are several previous work evaluating the physical understanding of large video models that are not cited: \n1) Videophy: Evaluating physical commonsense for video generation\n2) T2V-CompBench: A Comprehensive Benchmark for Compositional Text-to-video Generation\n3) Devil: A comprehensive benchmark for dynamics evaluation in video generation\nFurthermore, benchmark evaluating large video models more broadly:\n1) Vbench: Comprehensive benchmark suite for video generative models.\n2) Evalcrafter: Benchmarking and evaluating large video generation models", "other_strengths_and_weaknesses": "Strength: the paper makes an effort to investigate why they observe such results which is unlike many benchmark paper on physical accuracy of video generative models", "other_comments_or_suggestions": "N/A", "questions_for_authors": "N/A", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "myLeuXj6Hu", "reviewer": "Reviewer_6Hod", "summary": "This paper explores whether scaling video generation models enables them to learn physical laws. It first provides a thorough problem definition and then evaluates video generation models under three scenarios: in-distribution, out-of-distribution, and combinatorial generalization. The authors develop a 2D simulation testbed that simulates three fundamental physical principles. Through comprehensive experiments on simulated data, they conclude that scaling video generation models alone is insufficient for effective world modeling.", "questions": "", "limitations": "", "rating": 3, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "`1. The prioritization conclusion is drawn from single uniform linear motion, rasing concerns about its generalizability to other physical contexts.`\n\nThank you for your thoughtful comment. We selected the uniform linear motion scenario as it provides **a highly representative and clean setting**—velocity is easy to measure, and with only one object, the pixel regions corresponding to attributes such as size, color, and speed are not affected by interference from other objects\n\n To verify the robustness of our conclusion, we also ran experiments on **parabolic motion and collisions**. **The factor prioritization remained consistent**, supporting the generalizability of our findings. We will include these additional results in the revision.\n\n`2. a rationale for selecting eight types of objects`\n\nThank you for your question. To evaluate combinatorial generalization,  we define a \"combination\" as the physical interaction between different object types. The Phyre simulator we use provides a total of eight distinct object types that enable a wide range of physical interactions. We include **all** of them to create as complex a combinatorial space as possible for evaluating combinatorial generalization. We appreciate your suggestion and will clarify this rationale in the revision.\n\n`3. in Combinatorial Generalization, the evaluation **excludes velocity as a metric** and instead relies on FVD, SSIM, PSNR, and LPIPS. However, these are image/video metrics and do not guarantee physical correctness.`\n\nThank you for your valuable comment. Please allow us to clarify why velocity was excluded as a metric in the Combinatorial Generalization setting, unlike in the ID/OOD setting.\n\nIn the ID/OOD setting, scenes are simple (e.g., 1–2 colored balls on a plain background), enabling reliable position and velocity estimation via pixel averaging and frame differencing (line 203). In contrast, the Combinatorial setting includes many irregularly shaped and similarly colored objects, making pixel assignments ambiguous and position estimation unreliable. We also tested methods such as Hough circle detection and pretrained object detectors, but they resulted in significant errors.\n\nGiven these challenges, we excluded velocity as a metric and instead relied on a combination of objective video fidelity metrics and human evaluation:\n\n1. **FVD, SSIM, PSNR, and LPIPS** measure fidelity of generated videos to ground-truth videos ones governed by physical laws. While not exlicitly designed for physical correctness, they reflect plausibility to some extent by measuring consistency and realism with the groundtruth —an approach consistent with prior work like Genie [1].\n\n2. We also conducted **human evaluation**, where each evaluator was specifically instructed to **focus on assessing violations of physical laws**.\nWe believe this combined approach offers a comprehensive evaluation for physical correctness in the complex setting. We will clarify this in the revised paper.\n\n[1] Bruce, Jake, et al. \"Genie: Generative interactive environments.\" ICML 2024, Best paper.\n\n`4. How should the discrepancy between Section 3.2 and Figure 3 be interpreted?`\n\nThank you for your insightful question. **The difference arises from the type of generalization being evaluated**.\n\nIn Section 3.2, the evaluation is **strictly OOD extrapolation**. For example, if the training velocity range is [1.0,4.0], then the test set contains velocities outside that range, such as [0.0,0.8]∪[4.5,6.0]. In this setting, simply increasing the amount of data or model size does not significantly improve performance due to the difficulty of truly grasping the physical law and true extrapolation.\n\nIn contrast, Figure 3 reveals **a transition from extrapolation to interpolation**, where the test set lies between two disjoint training subsets, more like an interpolation scenario. As the gap between the two training regions narrows, model performance on the test set improves, reflecting the model's stronger ability to interpolate.\n\nWe will clarify this distinction in the revision.\n\n`5. no supplementary material`\n\nWe have well-organized code and datasets and will make them public once the paper is officially published to support future research.\n\n`6. With only the first frame as input, how does the model infer object properties and predict future events?`\n\nIn the Combinatorial Generalization setting, each object has a **distinct visual appearance and color**, allowing the model to** infer object type and physical attributes by learning associations** from training data.\n\n As all objects **start static with zero velocity**, the model can use these inferred attributes to predict future events.\n\n`7. Minor Issues`\n\nThank you for pointing out these minor issues. We have corrected them in the revision.\n\n`Summary`\n\nWe hope our responses have addressed your concerns and strengthened confidence in our paper.", "claims_and_evidence": "The central claim of this paper is that scaling video generation models is insufficient to uncover physical laws. This claim is well-supported by extensive experimental validation. \n\nA secondary claim suggests that video models prioritize different factors when referencing training data. However, this conclusion—particularly the ranking of these factors—is drawn from a single scenario, uniform linear motion, which raises concerns about its generalizability to other physical contexts.", "methods_and_evaluation_criteria": "The paper introduces three physical scenarios to assess the model’s ability to infer physical laws. This approach is a reasonable simplification, as the selected rules are fundamental and representative of broader physical principles.", "theoretical_claims": "To the best of my knowledge, the theoretical claims appear to be correct.", "experimental_designs_or_analyses": "1. In the Combinatorial Generalization setting, the paper selects eight types of objects but does not provide a rationale for this choice.\n\n2. Additionally, in Combinatorial Generalization, the evaluation excludes velocity as a metric and instead relies on FVD, SSIM, PSNR, and LPIPS. However, these are image/video metrics and do not guarantee physical correctness.\n\n3. As mentioned in Section 3.2, the model fails to improve performance as data or parameters scale up. However, Figure 3 shows that as the training region expands, accuracy improves for OOD data. How should this discrepancy be interpreted?", "relation_to_broader_scientific_literature": "This paper is related to machine learning, particularly in video generation and world modeling.", "essential_references_not_discussed": "None.", "other_strengths_and_weaknesses": "This paper focuses on an important problem—whether scaling video generation models can enable world modeling. I believe the insights provided will be beneficial to the broader research community.", "other_comments_or_suggestions": "Minor Issues:\n- Incorrect characters in Line80 (\"color ¿ size ¿ velocity ¿\")\n- Some sections lack textual content, e.g., A.4.1, A.4.2, and A.4.4.", "questions_for_authors": "Line 258: It is mentioned that only the first frame is used as a condition. How does the model infer the physical attributes of objects and accurately predict subsequent events?", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "VKl2zK39ow", "reviewer": "Reviewer_hEH6", "summary": "This paper investigates whether state-of-the-art video generative models can learn fundamental physical laws from purely visual data. Inspired by the vision of video models as “world simulators” (e.g. OpenAI’s Sora​), the authors conduct a systematic study using a controlled 2D physics environment. They construct a simulation testbed (based on Box2D) where simple geometric objects move and collide according to known classical mechanics laws (uniform motion, elastic collision, parabolic motion). Using this testbed, they generate large training datasets (up to 3 million videos) and train diffusion-based video generation models (with a VAE + transformer architecture similar to Sora​) to predict future frames from initial conditions. The goal is to evaluate if these models, given enough data and scale, can infer and obey the underlying physical laws without explicit supervision.", "questions": "", "limitations": "", "rating": 3, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for your thorough review and for appreciating our method design. Please find our responses to your questions below.\n\n`1. More qualitative examples of failure cases in OOD or combinatorial scenarios`\n\nThank you for the helpful suggestion. In Appendix A.8, we have included failure cases with side-by-side comparisons in both the OOD (Figure 20) and the Combinatorial Generalization setting (Figure 22). \n\nFor OOD setting, Figure 20 already illustrates all representative failure modes, as failure in this setting primarily involves inaccuracies in velocity prediction.\n\nFor Combinatorial setting, We are now providing additional 8 failure cases via the anonymous links below. (In accordance with the ICML rebuttal policy, we include images only, not videos.) These examples will be added into the revision. We will also include many demo **videos** on our project webpage once the paper is published.\n\nhttps://ibb.co/Pv5qgSfL\n\nhttps://ibb.co/F9G5nSX\n\n`2. discussion of alternative approaches such as PINN, symbolic regression of physical laws, or structured simulation engines`\n\nWe appreciate your suggestion to expand the discussion to compare video diffusion model (VDMs) with alternative approaches for physics modeling and to highlight what may be missing in purely data-driven models.\n\n1. **Physical Consistency and Explicit Inductive Biases**: Current VDMs lack explicit representations of physical laws and instead rely on statistical correlations learned from data. As our findings show, this can result in failure cases under conditions such as unseen object velocities or mass values. In contrast, approaches like PINN, symbolic regression, and structured simulation engines encode or recover governing equations, offering stronger guarantees of physical consistency and better extrapolation to unseen velocity and mass values [1, 2].\n\n2. Structured methods often lack **visual fidelity and scalability**. For example, PINNs are typically tailored to a single equation and require retraining when parameters or initial conditions change [3]. Most existing work also focuses on small-scale, low-dimensional problems, limiting applicability to realistic video generation. In contrast, VDMs generate high-fidelity visuals and scale more effectively across diverse scenarios.\n\n3. **Complementarity, Not Competition**:\n These observations point toward a promising direction: combining the strengths of both methods. For instance, physics engines or PINNs could be used to predict future physical states, while VDMs handle rendering and visual synthesis. Such hybrid systems could preserve both physical accuracy and visual realism.\n\nWe will incorporate the discussions into the revision.\n\n[1]  PINN: A deep learning framework for solving forward and inverse problems involving nonlinear partial differential equations.\n\n[2] Genesis: A Universal and Generative Physics Engine for Robotics and Beyond\n\n [3] Learning the solution operator of parametric partial differential equations with physics-informed DeepOnets.\n\n`3. the conclusion that the VDMs do not model OOD physical scenarios is too obvious, and isn't the whole point of scaling is to model more things to be ID?`\n\nThank you for raising this important point. While the failure of current VDMs to model OOD physics may seem expected, our paper provides deeper insight:\n\n1. **Generalization-by-memorization mechanism**:  Despite optimism that scaling VDMs enables generalization to complex, unseen scenarios [4,5], our experiments show that VDMs often rely on retrieving patterns from similar training examples rather than learning underlying physical principles. This generalization-by-memorization mechanism, not been clearly articulated prior to our work, underscores the limitations of VDMs and the need for structural priors and inductive biases in physical modeling.\n\n2. **The Limits of Turning OOD into ID — and Actionable Insights**: While we agree that the goal of scaling is to absorb more variation into the in-distribution regime, **real-world video data is vast, continuous, and high-dimensional, making it more difficult to fully cover than pure language**. For example, in robotics, variables such as object velocity, joint configurations, camera angles, noisy backgrounds, and task goals vary across a continuous and combinatorially large space.\n\nOur paper contributes actionable insights to this challenge: We demonstrate that **scaling combinatorial diversity** in the training data—rather than simply increasing dataset size—is significantly more effective for improving physical video modeling. \n\nWe hope this helps clarify the intuition and contributions of our work.\n\n[4] OpenAI. Sora Technical Report: Video Generation Models as World Simulators.\n\n[5] X.AI. 1X World Model. https://www.1x.tech/discover/1x-world-model\n\n`Summary`\n\nWe hope our responses have addressed your concerns. If you have any further questions, please feel free to reach out.", "claims_and_evidence": "The authors claim that current video generation models fail to infer universal physical laws and instead generalize by referencing similar training cases. This is supported by experiments showing that, while models can perfectly generalize within the training distribution, they break down on novel scenarios. The structured scaling analysis provides convincing evidence: even large diffusion models trained on massive data cannot correctly predict physics in unseen setups, underscoring that scaling alone is insufficient. The claim is well-supported by quantitative metrics (e.g. high error in predicting object velocity in OOD tests) and qualitative observations of physically implausible generated motions.", "methods_and_evaluation_criteria": "The methods and evaluation criteria for the problem are well-chosen. The 2D simulator approach is an excellent strategic decision to tackle an otherwise intractable evaluation problem​. The use of a strong diffusion model ensures the study tests the frontier of what’s possible. The experiments are structured to answer specific questions (ID vs OOD vs combinatorial generalization), and the metrics directly measure success on those terms. I also appreciate that the authors validated their VAE wasn’t a bottleneck – they report in the appendix that VAE reconstructions of videos have minimal error​, so any mistakes are from the diffusion model learning, not from lossy compression. This attention to detail in evaluation bolsters confidence. Overall, the methods are appropriate and quite thorough for this study.", "theoretical_claims": "This work is primarily empirical.", "experimental_designs_or_analyses": "The experimental design is well-structured. The authors systematically scale up training data size and model capacity to test how generalization improves (or doesn’t) with scale. They evaluate model performance on: (1) the training distribution, (2) held-out but similar (in-distribution) scenarios, (3) genuinely novel (OOD) scenarios that involve new object properties or dynamics, and (4) combinatorial cases that mix seen components in new ways. This comprehensive coverage provides a strong empirical foundation to analyze generalization. The results show perfect in-distribution generalization (models can interpolate within the training range) and some combinatorial generalization (performance improves gradually with scale for mixtures of seen concepts). Crucially, they demonstrate complete failure in true OOD generalization, even for the largest models.", "relation_to_broader_scientific_literature": "This paper builds upon prior work in video generation and world models, engaging with the question of whether large generative models can learn physics without explicit supervision. It connects to a growing literature on using foundation models as simulators for real-world processes, referencing studies that scale video models (like OpenAI’s Sora) and those investigating physical common sense in AI. By demonstrating the limits of current diffusion models, the paper contributes to ongoing discussions about the necessity of structure and inductive biases in AI for learning physics. A point that could strengthen the literature comparison is a deeper discussion of alternative approaches such as physics-informed neural networks, symbolic regression of physical laws, or structured simulation engines. Contrasting the diffusion model’s performance with these could highlight what is missing (e.g., explicit enforcement of Newtonian mechanics or object permanence) and emphasize that certain insights from physics-based learning might be needed in conjunction with data-driven models.", "essential_references_not_discussed": "N/A", "other_strengths_and_weaknesses": "While I like the paper overall, I don't know how I feel about the paper's overall intuition. Of course, videos models cannot model physics OOD, but what we want is that scaling could make a large-scale pretrained video model to be a sufficient approximation of a real-world projection. The authors primarily conduct experiments by training on smaller domains physics informed data (with some further tests on CogVid and SVD), and conclude the VDMs do not model OOD physical scenarios --- well, I personally just think that is too obvious, and isn't the whole point of scaling is to model more things to be ID? Nevertheless, I think the paper could be accepted, but will not object if it is rejected.", "other_comments_or_suggestions": "N/A", "questions_for_authors": "N/A", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Meta-review: This paper evaluates whether scaling video generation models enables learning physical laws. Using a 2D simulator for object motion governed by classical mechanics, experiments reveal many interesting conclusions. I recommend accepting this paper. However, the authors should modify this paper carefully according to the given comments.
Decision: Accept (poster)