PISA Experiments: Exploring Physics Post-Training for Video Diffusion Models by Watching Stuff Drop
Authors: Chenyu Li
Oscar Michel
Xichen Pan
Sainan Liu
Mike Roberts
Saining Xie
Venue: ICML 2025 poster (2025)
Keywords: video generation models
intuitive physics
world model
Primary area: applications->computer_vision
TL;DR: This paper explores physics post-training for video diffusion models and release a novel benchmark for evaluating video diffusion model's physical understanding ability.

Abstract: Large-scale pre-trained video generation models excel in content creation but are not reliable as physically accurate world simulators out of the box. This work studies the process of post-training these models for accurate world modeling through the lens of the simple, yet fundamental, physics task of modeling object freefall. We show state-of-the-art video generation models struggle with this basic task, despite their visually impressive outputs. To remedy this problem, we find that fine-tuning on a relatively small amount of simulated videos is effective in inducing the dropping behavior in the model, and we can further improve results through a novel reward modeling procedure we introduce. Our study also reveals key limitations of post-training in generalization and distribution modeling. Additionally, we release a benchmark for this task that may serve as a useful diagnostic tool for tracking physical accuracy in large-scale video generative model development. Code is available at this repository: https://github.com/vision-x-nyu/pisa-experiments.

PISA Experiments:
Exploring Physics Post-Training for Video Diffusion Models by Watching Stuff Drop
Chenyu Li Oscar Michel Xichen Pan
Sainan Liu?
Mike Roberts?
2 Saining Xie
Abstract
line of research that seeks to evolve these models from con- tent creators to world simulators for embodied agents (Yang et al., 2023; 2024b; Agarwal et al., 2025). However, accu- rate world modeling is considerably more challenging than creative content creation because looking "good enough" is not sufficient: generated pixels must faithfully represent a world state evolving in accordance with the laws of physics and visual perspective.
Large-scale pre-trained video generation mod- els excel in content creation but are not reliable as physically accurate world simulators out of the box. This work studies the process of post- training these models for accurate world modeling through the lens of the simple, yet fundamental, physics task of modeling object freefall. We show state-of-the the-hhe-rtttt video generation models struggle with this basic task, despite their visually impres- sive outputs. To remedy this problem, we find that fine-tuning on a relatively small amount of simulated videos is effective in inducing the drop- ping behavior in the model, and can further improve results through novel reward model- ing procedure we introduce. Our study also re- veals key limitations of post-training in gener- alization and distribution modeling. Addition- ally, we release a benchmark for this task that may serve as a useful diagnostic tool for track- ing physical accuracy in large-scale video gen- erative model development. Code is available at this repository: https: / / thub. com/ vision- x nyyuppisa wtwpppss-eepperiniieeeeeeea
We find that although the generations of state-of-ther -he- models are impressive visually, these models still struggle to generate results that are accurate physically, even though these models are pretrained on internet-scale video data demonstrating a wide variety of complex physical interac- tions. The failure to ground and align visual generations to the laws of physics suggests that pretraining is not enough and post-training stage is needed. Much like how pre- trained Large Language Models (LLMs) need to be adapted through post-training before they can be useful conversa- tional assistants, pretrained video generative models ought to be adapted through post-training before they can be de- ployed as physically accurate world simulators.
In this work, we rigorously examine the post-training pro- cess of video generation models by focusing on the simple yet fundamental physics task of modeling object freefall, which we find is highly challenging for state-of the mod- els. Specifically, we study an image-to-vides video! (I2V) sce- nario where the goal is generate a video of an object falling and potentially colliding with other objects on the ground, starting from initial image of the object sus- pended in midair. We chose to study this single task, rather than general physics ability as a whole, because its simplic- ity allows us to conduct controlled experiments that yield insights into the strengths and limitations of the post-training process, which we believe will become an increasingly im- portant component of research in generative world modeling. Additionally, the simplicity of the dropping task allows it to be implemented in simulation which is desirable because it allows us to easily test the properties of dataset scaling, gives us access to ground truth annotations for evaluation, and gives us the ability to precisely manipulate the simulation
1. Introduction
Over the past year, video generation models have advanced significantly, inspiring visions of a future where these mod- els could serve as realistic world models (Craik, 1967; Le- Cun, 2022; Hafner et al., 2019; 2023; Ha & Schmidhuber, 2018). State-of-the-art video generation models models ex- hibit impressive results in content creation (OpenAI, 2024: Kuaishou, 2024; Luma, 2024; Runway, 2024) and are al- ready being used in advertising and filmmaking (Runway, 2025; NBC, 2025). These advancements have sparked
Equaa contribution, alphabetical order. 'New York Uni- versity "Intel Labs. Correspondence to: Saining Xie <sain-
Proceedings of the 42nd International Conference Machine Learning. Vancouver, Canada. PMLR 267, 2025. Copyright 2025 by the author(s).
'We discuss our decision to formulate this task in the image-to- video setting instead of the video-to-vine video setting in Appendix A.

---

PISA Experiments: Exploring Physics Post- Most-Training for Video Diffusion Models by Watching Stuff Drop
Figure 1. Our PISA (Physics-Infommed Informed Simulation and Alignment) evaluation framework includes new video dataset, where objects dropped in variety of real-world (left) and synthetic (right) scenes. For visualization purposes, we depict object motion by overlaying multiple video frames each image shown above. Our real-world videos enable evaluate the physical accuracy of generated video output, and our synthetic vidcos enable us to improve accuracy through the use of post-training alignment methods.
environment for controlled experimentation.
outside the training distribution, such objects dropping from unseen depths or heights. Additionally, while our post- trained model generates object motion that is 3D-consistent and physically accurate, we observe misalignment between the generated and ground truth dropping time distribution.
Named after Galilco's 's famous dropping experiment, we in- troduce the PISA (Physics-Infommn Simulation and Align- ment) framework for studying physics post-training in the context of the dropping task. PISA includes new real and simulated video datasets, as shown in Figure containing diverse set of dropping scenarios. PISA also includes set of specifc metrics that focus measuring physical accuracy. Our real- world videos and metrics enable us to evaluate the physical accuracy of generated video output, and our synthetic videos enable us to improve accuracy through a post-training process we introduce.
These findings indicate that post-training likely to be essential component of future world modeling systems. The challenges identify in this relatively simple task are likely to persist when modeling more sophisticated physical phenomena. By introducing the PISA framework and bench- mark, we provide a useful diagnostic tool for researchers to test whether models are on the path to acquiring gen- eral physical abilities, as well as identify key limitations that researchers should be aware of when integrating new capabilities into their models through post-training.
Our study reveals that current state-of-the-ar video genera- tive models struggle significantly with the task of physically accurate object dropping. Generated objects frequently ex- hibit impossible behaviors, such as floating midair, defying gravity, or failing to preserve realistic trajectories during freefall. However, we find that simple fine-tuning can remarkably effective: fine-tuning an open-source model on small dataset of just a few thousand samples enables it to vastly outperform state-of-the-a art models in physical accu- racy. We further observe that pretrained models are critical for success; models initialized randomly, without leverag- ing pretraining on large-scale video datasets, fail to achieve comparable results. We also introduce a novel framework for reward modeling that yields further improvement. We demonstrate that our reward learning system is highly flexi- ble in that different reward functions be chosen to target different axes of physical improvement.
2. Related Work
Modeling Intuitive Physics. Intuitive physics refers to the innate or learned human capacity to make quick and accu- rate judgments about the physical properties and behaviors of objects in the world, such as their motion, stability, in- teractions. This ability, present even in infancy (Spelke et al., 1992; Baillargeon, 2004; Battaglia et al., 2013), is crucial for navigating and understanding everyday life. Replicat- ing intuitive physics is foundational step toward creating systems that can interact effectively and safely in dynamic, real-world environments (Lake al., 2017). Gravity, as component of intuitive physics, plays a pivotal role in both domains. It is one of the most universal and observable physical forces, shaping our expectations about object mo- tion, stability, and interaction (Hamrick et al., 2016; Ullman
Our analysis also reveals key limitations. First, we see that model performance degrades when tasked with scenarios
2

---

PISA Experiments: Exploring Physics Post- fost-Training for Video Diffusion Models by Watching Stuff Drop
al., 2017). Many studies in cognitive science (Battaglia al., 2013) and Al (Wu et al., 2015; Bear al., 2021) have relied on physics engines to evaluate and model intuitive physics. Our work uses the Kubric engine (Greff et al., 2022) to generate training videos.
Capturing slow- motion videos 120 FPS in 720p
"Invisible" hanging wire
Video Generation Models as World Simulators. Video generation has long been an intriguing topic in computer vision, particularly in the context of predicting future frames (Srivastava al., 2015; Xue et al., 2016). More cently, as large-scale generative models have become promi- nent, Yang et al. explored how a wide range of real-worid dynamics and deccision-makinn processes can be expressed in terms of video modeling (Yang et al., 2024b; 2023). The introduction of the Sora model (OpenAI, 2024) marked leap in the quality of generated videos and ignited interest in leveraging such models "world simulators." Over the past year, numerous video generation models have emerged, some open-source (Zheng et al., 2024; Yang et al., 2024c; Jin et al., 2024; Agarwal et al., 2025) and others commercially available (Kuaishou, 2024; Luma, 2024; Runway, 2024; OpenAI, 2024). Related our work, Kang et al. (Kang al., 2024) study the extent to which video generation mod- els learn generalizable laws of physics when trained 2D data from synthetic environment.
being dropped
Figure 2. The setup for collecting real-world videos.
3.1. Task Definition Assumptions
Our task can summarized as follows: given an image of an object suspended in midair, generate video of the object falling and colliding with the ground and potentially other objects. Since video is an incomplete partial observation of the 4D world, we make a number of assumptions to constrain the task space. These assumptions are crucial for ensuring that our metrics are reliable signals for physical accuracy, since they are only approximations of task success computed from a single ground truth and generated video.
Evaluating Video Generation Models. Traditional image- based metrics for generative modeling, such Fréchet in- ception distance (FID) (Heusel et al., 2017) or inception score (IS) (Salimans et al., 2016), can be incorporated into the video domain, either by applying them on a frame- by-frame basis by developing video-specific versions, such as Fréchet video distance (FVD) (Unterthiner et al., 2018). Going beyond distribution matching measures, sev- eral benchmarks have developed suites of metrics that aim to better evaluate the semantic or visual quality of generated videos. For example, V-Bench (Huang et al., 2024) offers a more granular evaluation by measuring video quality across multiple dimensions, such as with respect to subject consis- tency spatial relationships. In physics, some works, such as VideoPhy (Bansal et al., 2024) and PhyGenBench (Meng et al., 2024), evaluate in the T2V setting by utilizing multi- modal large language models (MLLM) to generate a VQA- based score. More recently, Cosmos (Agarwal et al., 2025) and Physics-IQ (Motamed et al., 2025), evaluate physics in the image-to-video and video-to-video settings.
Specifically, we assume that the falling object is completely still in the initial frame, that only the force of gravity is acting on the object while it falls, and that the camera does not move. The first two assumptions are necessary for the image-to-video setting. Since we do not provide multiple frames as input, otherwise impossible to establish the initial velocity or acceleration of the falling object without these assumptions. The last assumption necessary as our metrics derive from the motion of segmentation masks, which would be affected in the presence of camera motion.
3.2. Real World Data
6.2
Oten
3. PisaBench
6.15
Table
feequeecy
Offcc
Our benchmark, PisaBench, examines the ability of video generative models to produce accurate physical phenomena by focusing on a straightforward dropping task.
0.10
6:05
Kishen
Sope
6:0
Stain
Number of Objects
Ent
00
Figure 3. Statistics of the real- wol-world data: (a) number of objects in cach vidco, (b) the proportions of different scenes in the videos.
3

---

PISA Experiments: Exploring Physics Post- Tost-Training for Video Diffusion Models by Watching Stuff Drop
Simulation
ATTT
Real World
Caption "A Black helmet falls."
Figure 4. Examples of various objects included in our dataset. For simulation, utilize the GSO dataset (Downs et al., 2022), while for the real-world dataset, we curate our own set of common household objects.
Figure 5. Example of annotations in real-world data. For segmen- tation masks, we manually annotate first frame and utilize SAM 2 to produce segmentation masks across frames. For captions, we annotate "(object description} falls." for all video segments.
Real World Videos. We collect a set of 361 real-world videos demonstrating the dropping task for evaluation. As is shown in Figure 4. the dataset includes diverse set of objects with different shapes and sizes, captured across various settings such as offices, kitchens, parks, and more (see Figure 3). Each video begins with an object suspended by an invisible wire in the first frame, which is necessary to enforce the assumption that objects are stationary at the start of the video. This assumption is required in our image- to-video setting: otherwise, the initial velocity of an object is ambiguous. We cut the video clips to begin as soon as the wire is released. We record the videos in slow wow-motion...... at 120 frames per second (fps) with cellphone cameras mounted on tripods to eliminate camera motion. An example of our video collection setup show Figure Additional details on our collection system are provided in Appendix H.
rics compare frames from the ground-truth video with the generated video. Further details about the metrics, including their formulas and our resampling procedure for accounting for differences in fps, is described in Appendix B.
Trajectory L.2. For each frame in both the generated video and ground truth, we calculate the centroid of the masked re- gion. After doing this, we compute the average Lg distance between the centroids of corresponding frames.
Chamfer Distance (CD). To assess the shape fidelity of objects, we calculate the Chamfer Distance (CD) between the mask regions of the generated video and ground truth.
Intersection over Union (loU), We use the Intersection over Union (IoU) metric evaluate object permanence. The loU measures objects' degree of overlap between the generated video and ground truth.
Simulated Test Videos. Since our post-training process uses dataset of simulated videos, we also create simula- tion test-set of 60 videos for understanding sim2real transfer. We create two splits of 30 videos each: one featuring ob- jects and backgrounds seen during training, and the other featuring unseen objects and backgrounds. See Section 4.1 for details on how our simulated data is created.
3.4. Evaluation Results
We evaluate 4 open models including CogVideoX-SB- 12V(Yang al., 2024c), DynamiCrafter(Xing et al., 2023), Pyramid-Flow(Lin et al., 2024), and Open-Sora-V1. et al., 2024), as well as 4 proprietary models including Sora (OpenAI, 2024), (ling-V(KKuaishhn
Annotations. As is shown in Figure 5, annotate each video with caption and segmentation masks estimated from the SAM 2 (Ravi et al., 2024) video segmentation model. We create a descriptive caption for each object in the format of "[object description] falls." " This caption is used to provide context to the task when text input supported.
2024), Kling- Vl.5(Kuaishou, 2024), and Runway Gen3 (Runway, 2024). We also evaluate OpenSora post-trained through the pro- cesses of Supervised Fine-Tuning (PSFT) and Object Re- ward Optimization (ORO); see Section 4 for details.
The results of running the baseline models on the benchmark indicate a consistent failure to generate physically accurate dropping behavior, despite the visual realism of their gen- erated frames. Qualitatively, we see common failure cases in Figure 6, such as implausible object deformations, float-
3.3. Metrics
We propose three metrics to assess the accuracy of trajecto- ries, shape fidelity, and object permanence. Each of our met-
4

---

PISA Experiments: Exploring Physics Post- Most-irainnn for Video Diffusion Models by Watching Stuff Drop
Kinng V15
Open-Sora
Open
GRO (Seg)
Open-Sora (Flow]
Open-Sora (Depth
Figure Qualitative comparison of results on real test set (row 1-2). simulated seen test set (row 3-4) and simulated unseen test set (row 5-6). We present the results of popular open-source and commercially available models alongside those of models fine-tuned through our method, Existing models often struggle to generate videos depicting objects falling, whereas our PSFT method effectively introduces knowledge of free- fall into the model. ORO enables the model to more accurately learn object motion and shape.
ing, hallucination of new objects, and unrealistic special effects. We further visualize a random subset of generated trajectories on the left of Figure 8. In many cases, the object remains completely static, and sometimes the object even moves upward. When downward motion present, it is often slow or contains unrealistic horizontal movement.
troduce variability, we randomly sample the camera height between 0.4 and 0.6 meters and position objects between I and 3 meters away from the camera, which corresponds to the distributions observed in the real-world dataset. More information about the dataset can be found in Appendix K.
4.2. Physics Supervised Fine- Fine-Tuning (PSFT).
4. Physics Post-Traie Tost-Training
030
We present a post-training process to address the limitations of current models described in Section 3.4. We utilize sim- ulated videos that demonstrate realistic dropping behavior. Our approach for post-training is inspired by the two-stage pipeline consisting of supervised fine-tuning followed by reward modeling commonly used in LLMs. We find that our pipeline improves performance on both real and simu- lated evaluations, with greater gains observed in simulation. This is due to the sim-to-real gap, though our approach still shows substantial gains in transferring to real-world data.
precsahing
Nio pre-suiiiggg
8-
0.2
0.50
Steps
Steps [bl
(a)
Cherder
6:0
4.1. Simulated Adaptation Data
0.6
Numbbrr of Samprie
Ccl
668
The first stage of our approach involves supervised fine- tuning. We use Kubric (Greff et al., 2022), a simulation and rendering engine designed for scalable video generation, to create simulated videos of objects dropping and colliding with other objects on the ground. Each video consists of 1 1-6 dropping objects onto (possibly empty) pile of up to 4 objects underneath them. The videos are 2 seconds long, consisting of 32 frames at 16 fps. The objects are sourced from the Google Scanned Objects (GSO) dataset (Downs et al., 2022), which provides true-to-scale 3D models created scans across diverse categories (examples shown in Figure 4). The camera remains stationary in each video and is oriented parallel to the ground plane. To in-
Figure 7. Plots (a). (b), and (c) demonstrate that metrics tend to improve with further training and that leveraging a pre-trained video diffusion model enhances performance compared to random initialization. In plot (d), the size of the training dataset varies in each training run (each consisting of 5k steps). With only 5k samples, we can achieve optimal results.
We use the pretrained Open-Sora v1.2 (Zheng al., 2024) model as our base model and fine-tune it on our simulated video dataset. We employ Open-Sora v1.2's rectified flow training objective without modification (Liu et al., 2022). Each fine- tuning experiment is conducted with a batch size of 128 and learning rate of 1e-4 on two 80GB NVIDIA
from
5

---

PISA Experiments: Exploring Physics Post- Tost-Training for Video Difflsson Models by Watching Stuff Drop
Real
Sim (Seen)
Sim (Unseen)
Method
L2 (L)
CD (J)
IoU (1)
L2 (L)
CD ())
IoU (1)
L2 ())
CD (L)
IoU (1)
Proprietary
Sora (OpenAI, 2024)
0.174
0.488
0.065
0.149
0,446
0.040
0.140
0.419
0.031
Kling-V1 (Kuaishou, 2024)
0.157
0,425
0.056
0.142
0.415
0.032
0.145
0.437
0.028
Kling- V1.5 (Kuaishou, 2024)
0.155
0,424
0.058
0.137
0.396
0.033
0.132
0.405
0.029
Runway Gen3 (Runway, 2024)
0.187
0.526
0.042
0.170
0.509
0.040
0.149
0.460
0.038
CogVioooX-SB-DD (Yang et al., 2024c)
0.138
0.366
0.080
0.112
0.315
0.020
0.101
0.290
0.020
Open
DynamiCrafter (Xing al., 2023)
0.187
0.504
0.021
0.157
0.485
0.039
0.136
0.430
0.033
Pyramid-Flow (Jin et al., 2024)
0.175
0,485
0.062
0.126
0.352
0.059
0.130
0.381
0.048
Open-Sora (Zheng et al., 2024)
0.175
0.502
0.069
0.139
0.409
0.036
0.130
0.368
0.034
Open-Sora + PSFT (base)
0.076
0,188
0.139
0.036
0.088
0.165
0.028
0.058
0.129
Ours
base + ORO (Seg)
0.075
0.183
0.142
0.033
0.076
0.170
0.032
0.063
0.145
base ORO (Flow)
0.067
0.164
0.136
0.026
0.062
0.122
0.022
0.045
0.071
base ORO (Depth)
0.067
0.159
0.129
0.031
0.072
0.124
0.022
0.046
0.096
Table 1. PisaBench Evaluation Results. This table compares the performance of four proprietary models, four open models, and the models worldw and simulated test set which is decomposed into seen and unseen object splits,
fine-tuned with PSFT and PSFT + ORO Across all metrics, our PSFT models outperform all other baselines, including proprietary models like Sora. Reward modeling further enhances results, with segmentation rewards improving the shape-based IoU metric and optical rewards and depth rewards enhancing the motion-based L2 and CD metrics. This suggests that rewards can be flexibly adjusted to target specific aspects of performance.
A100 GPUs As shown in Figure 6, fine-tuning with this data alone is sufficient to induce realistic dropping behavior in the model. Quantitatively, our PSFT model substantially improves on both our simulated and real-world benchmark, as shown in Table Dataset size. We conduct an ablation study on the number of training samples to understand the amount of data required for optimal performance on our benchmark. We create random subsets from 500 to 20,000 samples and train our model for 5,000 gradient steps on each subset. Notably, as shown in Figure 7, only 5,000 samples are needed to achieve optimal results. Effect of pre- training. Additionally, we investigate the impact of Open- Sora's 's pre-training on adaptation. We randomly initialize the Open-Sora's 's denoising network while keeping the pre- trained initialization of the compressor network and train the model on dataset of 5k training samples. As shown in Figure& 8, the learned knowledge from Open-Sora's pre- training plays a critical role in our task.
14/36 are static trajectories
Figure 8. On the left, we plot random trajectories from the baseline models in Table 1. On the right, we show random trajectories from our fine-tuned model. The baseline trajectories exhibit unrealistic behavior, and most of them stay completely static. On the right, we see the trajectories consistently falling downward with collision and rolling bchavior being modeled after the point of contact.
motion and shape more closely align with the ground truth. We follow the VADER framework from (Prabhudesai et al., 2024) and introduce three reward models. The differences between approach and VADER include: (1) our reward model utilizes both generated videos and ground truth in- stead of generated videos and conditioning. (2) gradients propagate through all denoising time steps in fine-tuning. Consequently, the VADER objective is modified as follows:
Overall, using PSFT on only 5k samples is sufficient to push Open-Sora's 's performance past all other evaluated models, including state-of-the-ar commercial video generators, by wide margin. This is made possible by leveraging the knowledge from the sufficiently pre-trained base model.
4.3. Object Reward Optimization (ORO)
J(0) B(cor))-D. [R mo)]
(1)
In the second stage, we propose Object Reward Optimiza- tion (ORO) to use reward gradients to guide the video gen- eration model toward generating videos where the object's
where D is the ground truth dataset, po(.) is a given video diffusion model, 266- Zy E B lxxxx are generated video and ground truth, and e E Hx the initial image.
6

---

PISA Experiments: Exploring Physics Post- Tost-Training for Video Difflssin Models by Watching Stuff Drop
Segmentation Reward. We utilize SAM 2 (Ravi et al., 2024) generate segmentation masks across frames for generated videos. We define segmentation reward as the IoU between the dropping object's mask in generated video and the mask from the groud truth simulated segmentation.
From Equation (2), we that the random variables that af- feet object motion are Z (depth) and Y (height) (the camera focal length f is fixed). Thus, we are interested in testing generalization on unseen values of Y and Z.
We create a simulated test set in which a single object is dropped from varying depths and heights, using objects and backgrounds unseen during training. We uniformly sam- plc depth and height values (in meters) from the Cartesian product of the ranges [1, 5] and [0.5, 2.5], respectively. The camera height is fixed at 0.5m. and depth-height pairs out- side the camera viewing frustum are discarded. A sample is in-distribution (ID) its dropping depth and height both fall in the range [1, 3] and [0.5, 1.5].
Optical Flow Reward. We utilize RAFT (Teed & Deng, 2020) to generate generated video's optical flow Vie and ground truth's optical flow Vet, We define the optical flow reward R( zu) [ V
Depth Reward. We utilize Deyth-Anything- V2 (Yang et al., 2024a) to generate generated video's 's depth map D and ground truth's 's depth map D We define the optical flow reward as R(z): ro) = - D De
Since we have access to the ground truth dropping time in simulation, we also employ a dropping time error, a metric describe in Appendix B. Our analysis in Table 2 shows that performance degrades for aut-of-distribution scenarios.
Details on implementation can be found in Appendix C.
We begin from the checkpoint of the first stage, which is trained on 5,000 samples trained over 5,000 gradient steps. We then fine-tune the model with ORO on the simulated dataset, using a batch size and two 80GB NVIDIA 1 GPUs for each fine- toning experiment. We set a learning rate of 1c-6 -6 for segmentation reward and depth reward and 1e-5 for optical flow.
Since depth and height are the main physical quantities that affect falling dynamics, this finding indicates that our model may struggle to learn a fully generalizable law that accounts for the interaction of perspective and gravity.
As shown in Table 1, incorporating ORO in reward modeling further improves performance. Additionally, each reward function enhances the aspect of physicality that aligns with its intended purpose-sed segmentattnntttioonnnnn rewards improve shape accuracy, while flow rewards and depth rewards improve motion accuracy. This demonstrates the process is both modular and interpretable.
Setting
L2 (L)
Chamfer (L)
IOU (t) Time Error (+)
ID OOD
0.036
0.088
0.155
0.091
0.044
0.143
0.049
0.187
Table 2. Results of our metrics on in-distribution (ID) and out-of- distribution (OOD) depth-height combinations. The values used for depth range from 1-5m (ID range [I,3]) and height values range from 0.5 (ID range [0.5, 1.5).
5. Assessing Learned Physical Behavior
5.2. Distributional Analysis
Having introduced our post-training approaches in Section 4, we probe into the model's understanding of the interaction between gravity and perspective- -th two laws that deter- mine the dynamics of our videos. We first test the learned physical behavior of our model can generalize dropping heights and depths beyond its training distribution. Then, we study the ability of the model to learn the probability distribution induced by the uncertainty of perspective.
Camera View
Side View
Camera
5.1. Generalization to Unseen Depths and Heights
Figure 9. Demonstration of ambiguity in 2D perspective projec- tions. Each of the three clouds appears the exact same in the camera's 's image. The right side shows how we perform scale and translation augmentation generate deliherately ambiguous data.
Depth and height are the main factors that affect the dy- namics of falling object in our videos. We can see this by combining the laws of gravity with perspective under our camera assumptions to model the object's image coordi- nate as a function of time (further details on our coordinate system are described in Appendix D):
The evolution of physical system is not uniquely deter- mined by single initial image, since the lossy uncertainty of perspective induces a distribution of possible outcomes as shown in Figure 9. An ideal video world model should (1) output videos that are faithful to the evolution of some plau- sible world state and (2) provide accurate coverage across
vt) = f {(r0
Z 2 gt²
Yo
1
(2)

---

PISA Experiments: Exploring Physics Post- Post-Training for Video Diffusion Models by Watching Stuff Drop
the entire distribution of the world that is possible from its conditioning signal. In this section, examine these two facets by studying p(tly): the distribution of dropping times possible from an object at coordinate in the image plane. To do this, we create a simulated dataset that has a much wider distribution y) than our PSFT dataset. See Appendix for more details on its construction.
the model's learned conditional distribution p(4ly). We create 50 different initial images with differing values of y, generate 128 different videos from each, and estimate the dropping time in each video. Using the laws of gravity, the laws of perspective, and the assumption of uniform depth sampling in our dataset, we can analytically derive the probability p((|y) as
p(tly) { (Zuax Zmin 19
{
6min StS Umax
(3)
otherwise
Height
where ,3 is constant that depends on J. y and the camera height. The derivation is given in Appendix E.
30 15
5
10 15
10
15
We then measure goodness-of fit for each of the 50 experi- ments using the Kolmogonv--mmirov (KS) test (Massey Jr, 1951). The null hypothesis of the KS test that the two distributions being compared are equal, and we consider p-values less than 0.05 as evidence of misalignment, Since our measured times have limited precision and can only take 32 distinct values- to estimating the contact frame-we approximate the ground truth p(f]y) using a Monte Carlo method. We sample 1000 values from the ground truth distri- bution and then quantized them into 32 bins corresponding to their frame, which we use as ground truth observations in the KS test. We find that in all 50/50 cases, the p-value from the test is less than 0.05, which provides evidence that the model does not learn the correct distribution of dropping times. We visualize the misalignment between the empirical CDF of the model's in Figure 11.
Figure 10. Examples of model trajectories lifted to 3D. The blue line represents the height of the camera ray passing through the bottom of the dropping object as function of depth. The set of possible dropping trajectories given depth depicted in gray. The lifted trajectory of the model is depicted in green.
1.0
0.8
0.8
0-44
0.4 0.6
0.0
1.0
0.0
0.2
0.3
0.0
0.4
0.6
0.8
In summary, while our model's trajectories show promis- ing tendencies ground themselves plausible 3D world states, the range of possible outputs from the model does not align with the ground truth distribution.
Figure H1. Visualizing p(tly) misalignment for different images. Green shows the ground-truth CDF, orange is the 32-frame quan- tized version, and blue is the empirical CDF of 128 different samples of dropping times from the model.
Testing (1): 3D faithfulness of trajectories.
6. Conclusion
After training our model on this new dataset, we test whether its trajectories are consistent with a valid 3D world state, We first obtain an estimated dropping time from generated videos using the procedure described in Section 5.1. Us- ing knowledge of the camera position, focal length, sensor width, and y. we can obtain an implied depth and height of the trajectory. We can then back-project the video trajectory to 3D and analyze whether they constitute physically accu- rate trajectories. We give further details about this process in Appendix G. As show in in Figure 10, we find that our model's lifted trajectories consistently align with the 3D trajectory at the height and depth implied by its dropping time, giving evidence that the model's visual outputs are faithful to some plausible real-works state.
This work studics post-training as an avenue for adapting adapting pre-trained video generator into world models. We introduce post-training strategy that is highly effective in aligning our model. Our work raises interesting insights into the learned distributions of generative models. Qualitatively, large scale image or video generative models appear to excel at generating likely samples from the data distribution, but this alone does not imply that they match the data distri- bution well in its entirety. As long as a model able to generate likely samples, global distributional misalignment is not necessarily problem for content creation. However, this problem becomes critical for world models, where align- ment across the entire distribution is necessary for faithful world simulation. The insights revealed by our study, made possible by our constrained and tractable setting, indicate that although post- training improves per-sample accuracy, general distributional alignment remains unsolved.
Testing (2): distributional alignment.
Going beyond the level of individual trajectories, we study
8

---

PISA Experiments: Exploring Physics Post- Tost-Training for Video Diffusion Models by Watching Stuff Drop
Impact Statement
Craik, K. J. W. The nature of explanation, volume 445, CUP Archive, 1967.
This paper aims to explore physics post-training for video diffusion models. We release benchmark for image-to- video task, focusing on a simple yet fundamental scenario: object free-fall. Building on powerful video generation model Open-Sora(Zheng et al., 2024), we introduce Physics Supervised Fine-tuning (PSFT) and Object Reward Opti- mization (ORO) to effectively introduce and enhance drop- ping behavior in the model. There are many potential soci- etal consequences of our work, none which we feel must be specifically highlighted here.
Downs, L., Francis, A., Koenig, N., Kinman, B., Hickman, R., Reymann, K., McHugh, T. B., and Vanhoucke, V. Google scanned objects: A high-quality dataset of 3d scanned household items. In 2022 International Confer- ence on Robotics and Automation (ICRA), pp. 2553-2560. IEEE, 2022.
Greff, K., Belletti, F., Beyer, I.,, Doersch, C., Du, Y., Duck- worth, D., Fleet, D. J., Gnanapragasam, D., Golemo, F., Herrmann, C., et al. Kubric: A scalable dataset generator. Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pp.
Acknowledgment
2022.
We thank Boyang Zheng, Srivats Poddar, Ellis Brown, Shengbang Tong, Shusheng Yang, Jihan Yang, Daohan Lu, Anjali Gupta and Ziteng Wang for their help with data collec- tion. We thank Jiraphon Yenphraphai for valuable assistance in setting up our simulation code. We thank Runway and Kling AI for providing API credit. SX also acknowledges support from Intel AI SRS, Korean AI Research Hub, Open Path AI Foundation, Amazon Research Award, Google TRC program, and NSF Award IIS-2443404.
Ha, D. and Schmidhuber, J. Recurrent world models facil- itate policy evolution. Advances in neural information processing systems, 31, 2018.
Hafner, D., Lillicrap. T., Ba, J., and Norouzi, M. Dream to control: Learning behaviors by latent imagination. arXiv preprint arXiv: 1912.0162..13 2019.
Hafner, D., Pasukonis, J., Ba, J., and Lillicrap, T. Mastering diverse domains through world models. arXiv preprint arXiv:2301.04104. 2023.
References
Hamrick, B., Battaglia, W., Griffiths, T. L., and Tenen- baum, J. B. Inferring mass complex scenes by mental simulation. Cognition, 157:61-76, 2016.
Agarwal, N., Ali, A., Bala, M., Balaji, Y., Barker, E.,, Cai, T., Chattopadhyay, P., Chen, Y., Cui, Y., Ding, Y., et al. Cosmos world foundation model platform for physical AL. arXiv preprint arXiv:2501.05: 255.055500000.555 2025.
Heusel, M., Ramsauer, H., Unterthiner, T., Nessler, B., and Hochreiter. S. Gans trained by a two time-scale update rule converge to a local nash equilibrium. Advances in neural information processing systems, 30, 2017.
Baillargeon, R. Infants' physical world. Current directions in psychological science, 13(3):89 94, 2004.
Bansal, H., Lin, Z., Xie, T., Zong, Z., Yarom, M., Bitton, Y., Jiang, C., Sun, Y., Chang, K.-W., and Grover, A. Videophy: Evaluating physical commonsense for video generation. arXiv preprint arXiv:2406.0333 2024.
Huang, Z., He, Y., Yu, J., Zhang, F., Si, C., Jiang, Y., Zhang, Y., Wu, T., Jin, Q., Chanpaisit, N., et al. Vbench: Compre- hensive benchmark suite for video generative models. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 21807-21815 2024.
Battaglia, P. W., Hamrick, J. B., and Tenenbaum, J. B. Sim- ulation as an engine of physical scene understanding. Proceedings of the National Academy of Sciences, 110 (45):18327-18332. 2013.
Jin, Y., Sun, Z., Li, N., Xu, K., Jiang, H., Zhuang, N., Huang, Q. Song, Y., Mu, Y., and Lin, Z. Pyramidal flow matching for efficient video generative modeling. arXiv preprint arXiv:2410.055944 2024.
Bear, D. M., Wang, E., Mrowca, D., Binder, F. J., Tung. H.-Y. F., Pramod, R., Holdaway, C., Tao, S., Smith, K., Sun, F.-Y, et al. Physion: Evaluating physical prediction from vision in humans and machines. arXiv preprint arXiv:2 2021.
Kang, B., Yue, Y., Lu, R., Lin, Z., Zhao, Y., Wang, K., Huang, G., and Feng, J. How far is video generation from world model: A physical law perspective. arXiv preprint arXiv:2411.02335 2024.
Kuaishou. Kling, 2024.
URL https://kling: kuai shou. com. Accessed: 2024.
Community, B. O. Blender- - a 3d modelling and rendering package, 2018. URL http: www. blender. org.
Lake, B. M., Ullman, T. D., Tenenbaum, J. B., and Gersh- man, S. J. Building machines that learn and think like people. Behavioral and brain sciences, 40:e253, 2017.
Coumans, E. et al. Bullet physics engine. Open Source Software: htpp://ulletphysicc.. org, 13))844 2010.

---

PISA Experiments: Exploring Physics Post- Tost-Training for Video Diffusion Models by Watching Stuff Drop
LeCun, Y. A path towards autonomous machine intelligence version 0.9. 2, 2022-06-27. Open Review, 62(1):1-62, 2022.
Srivastava, N., Mansimov, E., and Salakhudinov, R. Unsu- pervised learning of video representations using Istms. In International conference machine learning, pp. 843- 852. PMLR, 2015.
Liu, X., Gong, C., and Liu, Q. Flow straight and fast: Learning to generate and transfer data with rectified flow. arXiv preprint ar (XXi:222990000000 2022.
Teed, Z. and Deng, J. Raft: Recurrent all-pairs field trans- forms for optical flow. In Computer Vision-ECCV 2020: 16th European Conference, Glasgow, UK, August 23-28, 2020, Proceedings, Part II 16, pp. 402-419. Springer, 2020.
Luma.
Dream machine, 2024. lumalabs ai darddrraaaaacii. Accessed: 2024.
URL https://
Massey Jr, F. The kolmogorov-smimoo test for goodness of fit. Journal of the American statistical Association, (253):68-3 78, 1951.
Ullman, T. D., Spelke, E., Battaglia, P., and Tenenbaum, J. B. Mind games: Game engines as an architecture for intuitive physics. Trends in cognitive sciences, 21(9): 649-665, 2017.
Meng, F., Liao, J., Tan, X., Shao, W., Lu, Q., Zhang, K., Cheng, Y., Li, D., Qiao, Y., and Luo, P. To- wards world simulator: Crafting physical commonsense- based benchmark for video generation. arXiv preprint arXiv:2410.05333, 2024.
Unterthiner, T., Van Steenkiste, S., Kurach, K., Marinier, R., Michalski, M., and Gelly, S. Towards accurate generative models of video: A new metric & challenges. arXiv preprint 1rXiv:1812.0177
2018.
Motamed, S., Culp. L., Swersky, K., Jaini, P., and Geirhos, R. Do generative video models learn physical principles from watching videos? arXiv preprint arXiv:2501.09038, 2025.
Wu, J., Yildirim, I., Lim, J. J., Freeman, B., and Tenenbaum, J. Galileo: Perceiving physical object properties by inte- grating a physics engine with deep learning. Advances in neural information processing systems, 28, 2015.
NBC. Coca-cola causes controversy with ai-made ad, 2025. Accessed: 2025-01-17.
Xing, J., Xia, M., Zhang, Y., Chen, H., Yu, W., Liu, H.,
Wang. X., Wong, T.-T., and Shan, Dynamicrafter: Ani- mating open-domain images with video diffusion priors. arXiv preprint arXiv:2310. 12.0..2..00 12190, 2023.
OpenAL. Sora, 2024. URL wwwph:tpp/////raae com. Ac- cessed: 2024.
Xue, T., Wu, J., Bouman, K., and Freeman, B. Visual dynamies: Probabilistic future frame synthesis via cross convolutional networks. Advances in neural information processing systems, 29, 2016.
Prabhudesai, M., Mendonca, R., Qin, Z., Fragkiadaki, K., and Pathak, D. Video diffusion alignment via reward gradients. arXiv preprint arXhv:2407.0873: 2024.
Ravi, N., Gabeur, V., HU, Y.TT,, Hu, R., Ryali, C., Ma, T., Khedr, H., Rädle, R., Rolland, C.. Gustafson, L., Mintun, E,, Pan, J., Alwala, K. V., Carion, N., Wu, C.-Y., Girshick, R. Dollár, P., and Feichtenhofer, C. Sam 2: Segment anything in images and videos. arXiv preprint arXiv:2408.00 00714, 2024. URL https:// arxiv. org/ abs /2008/2440 00714.
Yang, L.., Kang, B., Huang, Z., Zhao, Z., Xu, X., Feng, J., and Zhao, H. Depth anything v2. arXiv:2406.094: 2024a.
Yang, M., Du, Y., Ghasemipour, K., Tompson, J., Schuur- mans, D., and Abbeel, P. Learning interactive real-world simulators. arXiv preprint arXiv:23 /0.06/ 14, 2023.
Runway, Gen-3 alpha, 2024. URL https: WRKLhtttss//ruuuaynn hoop://eeaarrchtttttoocccaannneo 3alpha. Accessed: 2024.
Yang, S., Walker, J.,
J., Du, Y., Bruce, J., Barreto, A., Abbeel, P., and Schuurmans, Video as the new language for real-world decision making. arXiv preprint 172 2024b.
Runway, AIFF 2025: AI Film Festival, 2025. URL https: //aiff. runwayml com/. Accessed: 2025-01-17.
Yang, Z., Teng, J., Zheng, W., Ding, M., Huang, S., Xu, J., Yang. Y., Hong, W., Zhang, X., Feng, G., et al. Cogvideox: Text-to-video diffusion models with an expert transformer. arXiv preprint 26Xiv:248..0607.. 2024c.
Salimans, T., Goodfellow, I., Zaremba, W., Cheung, V., Radford, A., and Chen, X. Improved techniques for training gans. Advances in neural information processing systems, 29, 2016.
Zheng, Z., Peng. X., Yang, T., Shen C., Li, S., Liu, H., Zhou, Y., Li, T., and You, Y. Open-sora: Democratizing efficient video production for all, March 2024. URL https: //github.
Spelke, E. S., Breinlinger, K., Macomber, J., and Jacobson, K. Origins of knowledge. Psychological review, 99(4): 605, 1992.
10

---

PISA Experiments: Exploring Physics Post- most-Trainin for Video Diffusion Models by Watching Stuff Drop
A. Discussion of Image-to-Vite Video setting.
We note that our choice single-image input, as opposed to multi-frame input, comes with some trade-offs. choose the image-to-videe setting because it is widely supported among many different models, allowing us to make effective comparisons across the current state-of-the-arr However, only conditioning on single frame introduces significant ambiguity. Due the loss of information caused by projecting the 3D world through perspective, it may not be possible to directly infer the size of the object its height. In practice, we find our metrics are still reliable signals of task success, but we still study the problem of ambiguity more extensively in Section 5.2.
B. Metric details.
We propose three metrics to assess the accuracy of trajectories, shape fidelity, and object permanence. Each of our metries compare frames from the ground-truth video with the generated video. Because different models operate different fps, perform fps alignment part of our evaluation process. To perform fps alignment, generated videos to the ground truth using fgen and fg.. where fgen and fgi are the fps of generated video and ground truth respectively. For i-th frame in the generated video, we find the corresponding aligned frame index j in the ground truth video:
map each frame index of the
Sgen fg
j = round
Through fps alignment, we downsample the ground truth video to match the frame number of the generated video. We denote the downsampled ground truth as { and the generated video as {I N in the generated video.
Ji=1- where N is the number of frames
Trajectory L2. For each frame in both the generated video and ground truth, we calculate the centroid of the masked region. We then compute L2 distance between the centroids of corresponding frames:
c C
where C C.¹² E ? are the centroids of the dropping object in the i-th frame of generated video and the ground truth respectively. Chamfer Distance (CD), To assess the shape fidelity of objects, we calculate the Chamfer Distance (CD) between the mask regions of the generated video and ground truth:
CD
min (p
alle +
1
Σ
min l|g
pll2
N
P
gEQ.
Q
PEP
pEP.
9EQ.
where P. [p) and Q; truth respectively.
{qg
[Q.
are the sets of mask points in the i-th frame of the generated video and ground
Intersection over Union (IoU). We the Intersection over Union (loU) metric to evaluate object permanence. IoU measures objects' degree of overlap between the generated video and ground truth. This is formulated as follows:
N
M
n M
IoU =
(6)
N
M
U
where a M {0, Hxx are binary segmentation masks of the falling object in the i-th frame of the generated and ground truth videos respectively.
Time error. When testing on videos generated in simulation, we can provide a timing error. From the dropping height Yn of the ground truth video, which we have access to from the simulator, we can derive 'drop = / Yog. We then obtain a dropping time from the model's output by estimating the frame of impact the first frame F whose centroid velocity in the y direction is negative. If tdoop occurs in between F and F 1. then we define the time error Eume zero. Otherwise, define the time error as
Fiie = min (
F
fdrop )
fdrop
(7)
fps
fps

---

PISA Experiments: Exploring Physics Post- Post-Training for Video Diffusion Models by Watching Stuff Drop
C. ORO implementation details.
In our setting, we not cut the gradient after step like VADER. The gradient Vall(a6, zo) backpropagates through all diffusion timesteps and update the model weights 8:
OR(a),
Or
Vo(R(16) 20))
Σ
Or.
00
where T is the total diffusion timesteps.
Segmentation Reward. We utilize SAM 2 (Ravi et al., 2024) to generate segmentation masks across frames for generated video:
M SAM-2(xzg)
(9)
where Me denotes the masks of the falling object in the generated video. We obtain ground truth masks M using Kubric (Greff' et al., 2022). To avoid non-differentiable reward, we use Sigmoid to normalize mask logits of generated video instead of converting them to binary masks. We use loU between M and M as reward function:
R( zo) = IM M
(10)
Maximizing objective 1 is equivalent to minimizing the following objective:
[(] -.ee.c)- le) T M
(11)
This objective constrains the position and shape of the generated object in the video, encouraging a greater intersection with the object region in the ground truth video. The model learns to generate more accurate object positions and shapes through training with this objective.
Optical Flow Reward. We utilize RAFT (Teed & Deng, 2020) to generate optical flow for both generated videos and ground truth:
Vgen (AAF((2)))
(12)
/ (AFF
where /gen Ve denote the optical flows of generated videos and ground truth. We define the reward as follows:
R(*o mo) - V Va
(13)
Maximizing objective 1 is equivalent to minimizing the following objective:
((0) = E (a 1 1
(14)
This objective constrains the motion of the generated object in the video. The model learns to generate more accurate physical motion through training with this objective.
Depth Reward. We utilize Depth-Anything-vV- V2 (Yang et al., 2024a) to generate optical depth maps for both generated videos and ground truth:
gen Depth-Anytthng--V((ee
(15)
D Depth-Anything-V226
where Dgen D denote the depth maps of generated videos and ground truth. We define the reward follows:
R( =n) Dem D
(16)
Maximizing objective 1 is equivalent to minimizing the following objective:
J(O)= = (.ee.e)>.Peg-pottee D D
(17)
This objective constrains the 3d motion of the generated object in the video. The model learns to generate more accurate 3d physical motion through training with this objective.
12

---

PISA Experiments: Exploring Physics Post- fost-Training for Video Diffusion Models by Watching Stuff Drop
D. Coordinate system
give a visualization the coordinate system used this paper Figure 12. To compute y, we first leverage a segmentation map and find pixel row index that just below the object. Once this row index is found, can easily be computed from the camera position, camera sensor size, and image resolution, We note that because our camera assumed to be in perspective with the XY plane, ignore X and (not shown in figure) in analyses in Section 5.1 and Section 5.2.
Object
Object
image plane
y
camera center
Camera view
Side view
Z
Figure A visualization the coordinate system used in this paper (not scale). The image plane height of the object is denoted y. its actual height in 3D as Y, and its depth Z. The camera focal length is denoted as f.
E. Derivation of p(tly)
In our dataset construction, assume uniform distribution for Z, where Z U(Zmins Zmax), where Zmin 2 and Zmax 18. As shown in Figure 12, the dropping height Y is a linear function of Z, i.e Y 3Z for the slope that can be computed from y. J. the sensor size, and the camera height. This means we can solve for dropping time as 2 (2 + 3Z). Applying the transformation rule for probability density yields
Imin Smax
p(tlv)
(Zmav 2min B
(18)
otherwise
where tmin 2 + 3Zmim) and tmax 12 + 3.max.. Plugging in Zmin 2 and Zmax 18 yields Equation (3).
F. Ambiguous dataset
We introduce new dataset for distributional analysis that broadens p(t|y), in contrast to the PSFT dataset, which prioritizes realism and has a narrower distribution due to limited object depth variability. To create dataset with p(tly) that is sufficiently diverse for meaningful analysis, we first set up the initial scenes as before, but then apply an augmentation where a new depth values is sampled uniformly from [2, 18] and the object scaled and translated such that appears the same the original image, as shown Figure 9. For simplicity, we limit our scenes single dropping object with no other objects on the ground. We also disable shadows, preventing the model from using them as cues infer depth and height. Our dataset contains 5k samples consisting of k unique initial scenes each containing 5 different trajectories produced by the augmentation.
G. Lifting trajectories to 3D
To lift trajectories to 3D, we first estimate froop as described in Section 5.1. Using SAM2 to estimate object masks in the generated video, we can obtain trajectory of the bottom the object which we denote as you 01...., VN where
13

---

PISA Experiments: Exploring Physics Post- Post-Training for Video Difflsson Models by Watching Stuff Drop
N trop x fps. From Edrop, we can solve for an implied depth Z = 2gt
We then compute the lifted 3D trajectory as
B
+ y/ - BZ
H. PisaBench Details
In this section, we discuss the details of data collection pipeline and annotations. We present more examples of real-world videos and corresponding annotations in Figure 13.
H.1. Data Collection Pipeline
Collecting Real World Videos. We enlist approximately 15 volunteers participate in the data collection process. We hand out tripod, tape, and invisible wire for each volunteer. To ensure the quality, diversity, and minimize the ambiguity introduced by the environments, volunteers are provided with detailed guidelines. The key points of the data collection guidelines are shown in Table 3.
Raw videos processing. For the collected raw videos, we cut each video into multiple clips and crop their sizes. For each video clip, we annotate its starting position in the original long video and ensure that the duration of each segment does not exceed 12 seconds. Regarding the sizes of the videos, we manually crop each video an aspect ratio of 1: 1, ensuring that the falling objects remain fully visible within the frame during the cropping process. The processing interface is shown in Figure 14.
H.2. Annotation Details
We present annotation details Figure 15. For video captions, present the word cloud figure in (a). For segmentation masks, annotate all objects in the first frame using positive and negative points, which are then propagated across frames using the SAM 2 (Ravi et al., 2024) model to produce segmentation masks for all objects throughout the video. The annotation interface is shown in (b).
In addition to providing the annotated caption "(object description] falls.", we also add information to inform off-the-shele models of the task's s context as much as possible. To further enhance task comprehension, we append an additional description "A video that conforms the laws of physics." We also employ negative prompts "no camera motion" and "no slow-motion" to ensure environmental stability and impose constraints the generated videos. These prompts explicitly instruct the models to avoid including camera motion or any non-real-time object motion, thereby maintaining consistency with realdworld physics.
I. Inference Details
We present the inference configurations of each closed or open model we evaluate in Table For models that do not support generating videos with 1:1 aspect ratio, pad initial frames with black borders the resolution supported by these models, and finally remove the black borders from the generated videos.
J. More Qualitative Examples
We present more qualitative examples in Figure 16 Figure 22. Although in some showcases, models can roughly predict the downward trend, models still struggle to predict plausible shape and motion. The defects the models can be mainly attributed to the following aspects:
Trajectory correctness: in most videos, models fail to predict even the basic falling trajectory of objects, as shown in Figure 19 (a), despite this being highly intuitive for humans. Even in cases where the falling trajectory roughly correctly predicted, the models still struggle to accurately predict subsequent events, such as collisions, as illustrated in Figure 16 (f).
Object consistency: in many generated videos, object consistency is poor. Models struggle to infer the appearance of objects from multiple viewpoints a physically plausible manner, resulting in unnatural appearances, as shown in Figure 16 (a). Additionally, models perform poorly in maintaining object permanence, causing objects to appear blurry, as illustrated in Figure 20 (f). Furthermore, models sometimes introduce new objects into the video, as depicted in
14

---

PISA Experiments: Exploring Physics Post- Post-Training for Video Diffusion Models by Watching Stuff Drop
Figure 20 (e).
Scene consistency: models struggle to maintain scene consistency, leading to abrupt transitions in many videos. These sudden changes make videos appear unnatural, as shown in Figure 18 (f).
K. Simulated Adaption Details
We use the Kubric (Greff et al., 2022) simulation and rendering engine for creating our simulated vidcos. Kubric uses PyBullet (Coumans et al., 2010) for running physics simulations and Blender (Community, 2018) for rendering. We set the simulation rate to 240 steps per second and render 2-second videos at 16 fps, resulting in 32 frames per video. Each scene consists of objects from the Google Scanned Objects (GSO) dataset (Downs et al., 2022) and uses environmental lighting from HDRI maps provided by Kubric. We use 930 objects and 458 HDRI maps for training and 103 objects and 51 HDRI maps for testing.
For each video, randomly choose 1-6 objects to drop. These objects placed height uniformly sampled from 0.5m to 1.5m. Below each of these objects, possibly empty pile of up to 4 objects spawns beneath to create collisions. The objects are placed in spawn region of size 2m x 2m.
The camera is initially positioned Im behind this region, with its height varying uniformly between 0.4m and 0.6m. Once all objects are placed, the camera moves back in random increments until all objects are visible within the camera frame. The camera uses focal length of 35 mm, a sensor width of 32mm, and an aspect ratio of x
L. Limitations
In this work, we collect and manually annotate dataset of 361 real-world videos and design three spatial metrics to evaluate the performance of state-of-the-ar image-to-video (I2V) models in fundamental physical scenario: free fall. Our metrics focus solcly on spatial positional relationships, excluding object appearance attributes such as color. To enable more fine-grained evaluations of appearance characteristics, we aim to develop metrics based on Multimodal Large Language Models (MLLMs) or pixel-level analysis in future work.
Furthermore, propose the PSFT and ORO methods to fine-tune the Open-Sora model (Zheng et al., 2024), improving its ability to generate physically plausible videos. Despite these improvements, certain limitations remain, specifically, the generation of blurry objects in some videos. We hope to address these challenges in future research by refining both the dataset and the fine-tuning strategies, aiming to produce videos that better maintain object visuals.
15

---

PISA Experiments: Exploring Physics Post- most-Training for Video Diffusion Models by Watching Stuff Drop
(a) A white paper roll falls.
(c) A black bottle falls...
(b) A transparent bottle falls.
(d) A white bottle falls.
Figure Examples world videos and annotations. We present video frames first row and mask annotations in the second
row.
16

---

PISA Experiments: Exploring Physics Post- Post-Training for Video Diffusion Models by Watching Stuff Drop
Aspect
Requirements
Camera
The camera must be stabilized using tripod.
The dropping object should remain visible throughout the entire fall.
The trajectory of the object should be sufficiently centered in the frame.
Ensure the slow-motion setting is configured 120 fps.
Avoid a completely top-down perspective; the frame should include both the floor and the wall for spatial context.
It is acceptable to record one long video containing multiple drops the same location.
Objects
Most objects should be rigid and non-deformable.
A limited number of flexible or deformable objects may included, as such data is also valuable.
Dropping Procedure
Secure the object with wire using tape, ensuring stability. Multiple tapings may be necessary for proper stabilization.
Visibility of the wire the video is acceptable.
No body parts should appear in the frame. If this is challenging, consider having partner monitor the camera or use screen-sharing software to view the camera feed on laptop for uninterrupted framing.
Record videos in horizontal orientation to simplify cropping and to help keep the frame free of unnecessary elements.
Use short wire to enhance object stability.
The object should remain stationary before being dropped.
Scene Composition
Make the scenes dynamic and engaging. Include interactions with other objects, such as collisions or objects tipping over. Static objects should serve as active elements rather than mere background props.
Avoid filming in classroom or laboratory environments.
Include a variety of dropping heights.
Film in different environments, ensuring at least one setting is outside your apartment.
Minimize human shadows in the frame whenever possible.
Ensure good lighting and maintain strong contrast between the objects and the back- ground.
Table Key points of real world videos collection guideline. We have detailed requirements for eamera, objects, dropping procedure and scene composition to ensure the quality, diversity and minimize ambiguity introduced by environments.
17

---

PISA Experiments: Exploring Physics Post-Traininn Training for Video Diffusion Models by Watching Stuff Drop
01.00
604 milinee
Opentedde Cnp
(a)
(b)
Figure Video processing interface. (a) we annotate starting positions in the original long videos and clip them into multiple clips less than 12 seconds. (h) We drag the cropping box crop the video size to aspect ratio of 1:1.
pet Poller orange tubo sransparent bottle woll black eraser
scotch
lue
bottiel
C
Geleceed Dolect
whl
inpaste
box
V9
ectcaie
plastic
bottle
water
box
box
packing
black
Cola
bottle
Chied
paper
eer
r
cup
anggee
e
Uiyad
bat packing
der
rreen
ry
yellow
#rapping paper
pott
red
wr
apping
paper
le
e
package
roll
white bottle
990
pink
cup
boxtaee
roll
loca
cola
pale yellow
e
white
black
shoe
paper (a)
grey
(b)
Figure 15. Annotation details of real world videos. (a) Word cloud of objects in video captions. Our videos contain a variety of daily life objects. (b) Interface for annotating positive and negative points in the first frame. Red and blue dots indicate positive and negative points respectively. We annotate all objects the midair and ground.
Model
Resolution
Number of Frames
FPS
Guidance Scale
Sampling Steps
Noise Scheduler
Sora
720 x 720
150
30
Closed
Kling-V1.5
960 x 960
150
30
1.0
Kling-VI
960 x 960
150
30
1.0
Runway Gen3
1280 768
156
30
CogVideoX-5B-- -5B-12
720 x 480
48
8
6.0
50
DDIM
Open
DynamiCrafter
512 x 320
90
30
0.7
50
DDIM
Pyramid-Flow
1280 x 768
120
24
4.0
BulerDiscrete
Open-Sore Sora
512 x 512
90
30
7.0
30
RFLOW
Table 4. Inference details for models we evaluate, where "_" indicates the information is available.
18

---

PISA Experiments: Exploring Physics Post- most-Training for Video Diffusion Models by Watching Stuff Drop
(a) A brown bottle falls.
(b) A grey bottle falls.
(c) A grey paper cup falls.
(d) A paper cup falls.
(e) A white bottle falls.
(f) A white box falls.
Figure 16. Qualitative examples of Kling- V1 (Kuaishou, 2024). In (a) (b) (c) (f) objects have tendency to fall. (b) (c) roughly consistent with the laws of physics. In (a) (f), the shape of the object does not match the first frame. In (d), the paper cup is suspended in midair. In (e), new object is introduced. In (e), the model fails correctly predict the collision that occurs after the white box falls and the chain of events that follows.
19

---

PISA Experiments: Exploring Physics Post- mostnnrainnn for Video Diffusion Models by Watching Stuff Drop
(a) A black and grey glove falls.
(b) A black bottle falls.
(c) A blue and white box falls.
(d) A brown bottle falls.
(e) A Coca-Cola can falls.
TANK
TANK
TANK
TANK
TANK
TANK
TANK
(f) A pink box falls.
Figure Qualitative examples Runway Gen3 (Runway. 2024). (b) (e), objects have a tendency to fall. (a) (e) (0, new objects are introduced. In (b) (d), the shape of the object does not match the first frame. In (c), the box is suspended in midair.
20

---

PISA Experiments: Exploring Physics Post- mostnnrainnn for Video Diffusion Models by Watching Stuff Drop
(a) A black bottle falls.
(b) A black helmet falls.
(c) A paper box falls.
(d) A white bottle falls.
(e) A grey paper cup falls.
(f) A white box falls.
Figure 18. Qualitative examples of VideoX--BBBBB (Yang et al., 2024c). In (a) (0) objects have a tendency to fall. However, in all the videos, there violations of physics. In (a) (b), the objects divided into parts. In (c) (d) (e). the shape of the object does not match the first frame. In (c), the trajectory is not vertical fall. In (f), scene changes suddenly, which does not match the first frame.
21

---

PISA Experiments: Exploring Physics Post- cost-nrainin for Video Diffusion Models by Watching Stuff Drop
(a) A black box falls.
(b) A card holder falls.
(c) A white bottle falls.
(d) A white box falls.
(e) An orange and white box falls.
(f) A shoe falls.
Figure 19. Qualitative examples of DynamiCrafter (Xing et al., 2023). In all the videos, objects do not have a tendency to fall, suspended
in the midair.
22

---

PISA Experiments: Exploring Physics Post- mostnnrainnn for Video Diffusion Models by Watching Stuff Drop
(a) A black bottle falls.
(b) A green and white box falls.
(c) A grey bottle falls.
(d) An orange tube falls.
(e) A white bottle falls.
(f) A plastic box falls.
Figure Qualitative examples Pyramid-Flow (Jin et al., 2024). In (b) (d) (e), objects have a tendency to fall. (a) (b) (e) (f), new objects are introduced. In (c), scene changes, which does not match the first frame.. In (d), the tube hecomes blurry.
23

---

PISA Experiments: Exploring Physics Post- fost-nraining for Video Diffusion Models by Watching Stuff Drop
(a) A bottle full of water falls.
(b) A brown bottle falls.
(c) A grey paper cup falls.
(d) A paper box falls.
(e) A white bottle falls.
(f) A white box falls.
Figure Qualitative examples of Open-Sora (Zheng et al., 2024). all the videos, objects not have a tendency fall, suspended
the midair. In (b) (d), scene changes suddenly, which does not match the first frame. In (e), new object is introduced.
24

---

PISA Experiments: Exploring Physics Post- mostnnrainnn for Video Diffusion Models by Watching Stuff Drop
(a) A brown bottle falls.
(b) A grey eraser falls.
(c) A grey paper cup falls.
(d) A transparent bottle falls.
(e) A red wrapping paper falls.
(f) A white bottle falls.
Figure 22. Qualitative examples our method (Open-Sora PSFT + ORO). videos, objects have a tendency However.
the consistency of objects is still insufficient. In some frames, objects hecome blurry. Objects sometimes disappear after collision.
25


Reviews:
[{"review_id": "j0gsZvuE27", "reviewer": "Reviewer_7wqN", "summary": "This work finds the generations of SOTA video generation models are visually impressive but are physically inaccurate. This work rigorously examines the post-training process of video generation models by focusing on the simple yet fundamental physics task of modeling object freefall which is highly challenging for state-of-the-art models. \n\nThey find fine-tuning a small amount of simulated videos can effectively improve the physical consistency. They further introduce two reward models for reward gradient training. The study also reveals key limitations of post-training in generalization and distribution modeling. Their released benchmark can also serve as a useful diagnostic tool for measuring the emergence of accurate physics modeling in video generative models.\n\nOverall, the proposed benchmark is quite valuable for evaluating physics modeling, but the technical contribution is lacking and claims are not quite solid.", "questions": "", "limitations": "", "rating": 2, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We thank the reviewer for taking the time to review our work. We are extremely grateful for the feedback given, and below we address the main concerns raised. \n\n> **Q1:** Marginal improvement from ORO.\n\nOn simulated data, ORO yields substantial gains. As the goal of our work is to study the process of post-training in a rigorous and controlled manner, we chose simulation as the primary domain for both training and evaluation. Even though we do not directly address the sim2real gap in this work, our method does also show generalizability to real data, and importantly, it maintains its outperformance of all other state-of-the-art commercial models. \n\nTo see if we could push real world performance even further, we ran an experiment using depth reward. The depth reward improves performance on the real world dataset by a large margin in L2 (11% improvement) and Chamfer distance (15% improvement). Please see [here](https://anonymous.4open.science/r/ICML-7650-Rebuttal-1645/1_depth_reward.md) for the full results.\n\nWe also found that ORO significantly improves the results in the phase of trajectory *after* collision with the ground, which is the most complex part to model. Please see our response to hCaK for more details.\n\n> **Q2:** Weakness of OpenSora.\n\nWe agree that it is important to make sure the claims made for OpenSora hold for other models. We applied the PSFT procedure to Pyramid-Flow[1], a more recent and performant open video model than OpenSora. On our real test set, Pyramid-Flow outperforms the baselines.  \n\n| Model               | L2 ⬇️ | CD ⬇️  | IoU ⬆️ |\n| ------------------- | ----------------- | ----------------- | ---------------- |\n| Pyramid-Flow + PSFT | 0.081         | 0.194         | 0.121        |\n\nTraining curves and more evaluation tables can be found [here](https://anonymous.4open.science/r/ICML-7650-Rebuttal-1645/2_pyramid_flow.md). \n\n> **Q3:** OOD evaluation.\n\nWe evaluated our data on challenging and OOD settings in both real and simulated data. These videos include scenarios such as objects sliding down ramps, falling into containers, or domino-like setups. A summary of our relative improvement over the base OpenSora model is shown below. \n\n| Scenarios             | L2     | CD     | IoU     |\n| --------------------- | ------ | ------ | ------- |\n| Domino (Real)         | 47% | 54% | 42%  |\n| Ramp (Real)           | 41% | 47% | 18%  |\n| Stairs (Real)         | 25% | 20% | 103% |\n| Ramp (Simulated)      | 87% | 90% | 55%  |\n| Container (Simulated) | 75% | 67% | 3.7%   |\n\nPlease see [here](https://anonymous.4open.science/r/ICML-7650-Rebuttal-1645/3_ood.md) for more information about the dataset construction and a full table breakdown across settings and comparisons with baselines. \n\n> **Q4:** Reward method is specialized.\n\nWe disagree that our reward modeling framework is specialized. In fact, it is highly general, since it only requires dense annotation maps, such as segmentation, depth or flow, to be provided for the generated and ground truth video. \n\nAccuracy on physics tasks besides dropping is an important problem though, and concurrent work has shown evidence that ORO could be effective in more general settings. VideoJam [2] uses optical flow supervision in a similar manner to ORO and finds that it dramatically improves motion accuracy in video diffusion models, including in modeling human motion. \n\n> **Q5:** Potential for under-training on larger datasets.\n\nWe continued to finetune our model trained on 20k data samples for 5k more steps (batch size of 128). As can be seen in the figure [here](https://anonymous.4open.science/r/ICML-7650-Rebuttal-1645/4_20k_curve.md), our metrics do not consistently improve further as a result of doing this. \n\n> **Q6:** Concern about resolution.\n\nDue to computational limitations, we were not able to train models at resolution higher than 256. However, the video architecture of OpenSora supports zero shot evaluation at 512 resolution. Overall, our metrics slightly degrade, though performance could be improved with finetuning at 512. A summary on real data is shown below and the full table can be found [here](https://anonymous.4open.science/r/ICML-7650-Rebuttal-1645/5_512_resolution.md).\n\n| Output Resolution | L2 (⬇️) | CD (⬇️) | IoU (⬆️) |\n| ----------------- | ----------------- | ----------------- | ---------------- |\n| 256x256    | 0.153             | 0.432             | 0.069            |\n| 512x512    | 0.175             | 0.502             | 0.069            |\n\n\nPlease let us know if you have any further questions. If our response has sufficiently addressed the concerns you have mentioned, we kindly ask that you raise your score. Thank you very much again for taking the time to review our work! \n\nReferences\n\n[1] Pyramidal Flow Matching for Efficient Video Generative Modeling\n\n[2] VideoJAM: Joint Appearance-Motion Representations for Enhanced Motion Generation in Video Models", "claims_and_evidence": "The claims are good overall. \n\nA few issues:\n- The improvement from ORO is quite marginal. For example, ORO increases IoU scores from 0.139 to 0.142. It's uncertain if the proposed reward models are useful or not.\n- The authors conclude by experimenting on a single model Open-Sora, which is quite weak nowadays. It's unknown if the observations can transfer to other models.\n- The paper doesn't evaluate if the learned physics can transferred to more OOD settings, i.e. dramatically different scenes (e.g. indoor, oudoor) and objects (e.g. cat, human etc.).", "methods_and_evaluation_criteria": "-  The proposed methods (PSFT, ORO) make sense and can improve the physical consistency.\n- While the proposed reward models make sense, it's quite specialized and may not work for general cases. For example, does these reward models work for human actions?", "theoretical_claims": "No theoretical claims.", "experimental_designs_or_analyses": "Overall, the experimental design and analyses are solid and convincing. \n\nA few issues:\n- When ablation on dataset size, all models are trained for 5k steps, which may be under-trained for large dataset size.\n- The fine-tuning video resolution is unspecified in the paper. From supplementary material, the models are trained on 256p, which may be too small for many objects in the proposed benchmark. How does the model perform on higher resolution, e.g. 512p?", "relation_to_broader_scientific_literature": "N/A", "essential_references_not_discussed": "N/A", "other_strengths_and_weaknesses": "N/A", "other_comments_or_suggestions": "N/A", "questions_for_authors": "See issues in previous sections.", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "WtogdQIoX7", "reviewer": "Reviewer_ocR6", "summary": "**Main Findings:**\n\n- This paper addresses the physics-based task of modeling object freefall in video diffusion models, specifically formulated as follows: given an initial image of an object suspended midair, the goal is to generate a video depicting the object realistically falling, colliding with the ground, and potentially interacting with other objects.\n\n- A new evaluation framework called PISA (Physics-Informed Simulation and Alignment) is proposed, including a video dataset. Results reveal that current state-of-the-art video generation models significantly struggle with accurately performing this fundamental physics task.\n\n**Main Algorithmic/Conceptual Ideas:**\n\n- The paper introduces a post-training method aimed at enhancing video generation models through Physics-Supervised Fine-Tuning (PSFT) and Object Reward Optimization (ORO).\n\n**Main Results:**\n\n- Evaluations using the proposed PISA framework demonstrate that existing SOTA video generation models have limited capabilities in accurately generating object freefall, highlighting weaknesses in their physical modeling abilities.\n\n- While the proposed PSFT and ORO methods significantly enhance model performance on the benchmark task, their generalization to out-of-distribution (OOD) scenarios remains limited.", "questions": "", "limitations": "", "rating": 3, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We thank the reviewer for taking the time to review our work. We are extremely grateful for the feedback given, and below we address the main concerns raised. \n\n> **Q1:** Limited scope of the paper.\n\nOur goal in this paper is not to create a generalist state-of-the-art physics video model, but rather to treat the post-training process itself as the primary object of study. As shown in the paper, leading commercial models are unable to reliably simulate some of the most basic aspects of physics. As such, we believe that post-training is an overlooked aspect of video modeling that will soon become a mainstream research area, especially in world models/simulators for embodied AI applications. \n\nThe simplicity of our task was deliberately chosen because it can serve as a probing mechanism for understanding aspects of the post-training process that would otherwise be overlooked in a more general setting. For example, we use the fact that our task can be analytically descibed with the laws of gravity and perspective to conduct the analysis in Sec. 5. We believe the distribution matching limitations are an important finding because if they are present in a setting as simple as ours, then the problem is likely to persist in more complex settings that are of interest to world model researchers. Hence this contribution is a highly valuable insight for future research looking to leverage video models as planners or simulators in embodied applications. \n\n> **Q2:** Bias in post-training.\n\nThe problem of models overfitting/memorizing statistical patterns in the training data is ubiquitous across all areas of deep learning, even in the post-training of leading LLMs [1]. Our paper does not study solutions to this broader generalization problem, and the extent to which it is possible to solve is not clear either. In this sense, some of the results presented in this paper, such as those in section 5.1, are not surprising. We hope that you would agree with our view that presenting these results is scientifically important, even if they are somewhat negative and unsurprising. \n\n> **Q3:** OOD generalization.\n\nWe evaluated our data on challenging and OOD settings in both real and simulated data. These videos include scenarios such as objects sliding down ramps, falling into containers, or domino-like setups. A summary of our relative improvement over the base OpenSora model is shown below. \n\n| Scenarios             | L2     | CD     | IoU     |\n| --------------------- | ------ | ------ | ------- |\n| Domino (Real)         | 47% | 54% | 42%  |\n| Ramp (Real)           | 41% | 47% | 18%  |\n| Stairs (Real)         | 25% | 20% | 103% |\n| Ramp (Simulated)      | 87% | 90% | 55%  |\n| Container (Simulated) | 75% | 67% | 3.7%   |\n\nPlease see [here](https://anonymous.4open.science/r/ICML-7650-Rebuttal-1645/3_ood.md) for more information about the dataset construction and a full table breakdown across settings and comparisons with baselines.  \n\n> **Q4:** Degradation of original capabilities.\n\nWe evaluated our model on VBench to understand the effect that our post-training process has on the model's original capabilities. Overall, there is a degradation in aesthetic quality and image quality, likely stemming from the limited realism of our simulated videos. The degradations could potentially be mitigated by adding aesthetic samples into post-training, which has been shown to be effective for both image [2] and video [3] models. Aside from these degradations, the performance on the other metric categories either improves or remains intact. Please see [here](https://anonymous.4open.science/r/ICML-7650-Rebuttal-1645/6_vbench.md) for a full breakdown.\n\n> **Q5**: Concern about resolution.\n\nDue to computational limitations, we were not able to train models at resolution higher than 256. However, the video architecture of OpenSora supports zero shot evaluation at 512 resolution. Overall, our metrics slightly degrade, though performance could be improved with finetuning at 512. A summary on real data is shown below and the full table can be found [here](https://anonymous.4open.science/r/ICML-7650-Rebuttal-1645/5_512_resolution.md).\n\n| Output Resolution | L2 (⬇️) | CD (⬇️) | IoU (⬆️) |\n| ----------------- | ----------------- | ----------------- | ---------------- |\n| $256\\times256$    | 0.153             | 0.432             | 0.069            |\n| $512\\times512$    | 0.175             | 0.502             | 0.069            |\n\n\nPlease let us know if you have any further questions. If our response has sufficiently addressed the concerns you have mentioned, we kindly ask that you raise your score. Thank you very much again for taking the time to review our work!\n\nReferences\n\n[1] Embers of autoregression show how large language models are shaped by the problem they are trained to solve\n\n[2] Emu: Enhancing Image Generation Models Using Photogenic Needles in a Haystack\n\n[3] Stable Video Diffusion: Scaling Latent Video Diffusion Models to Large Datasets", "claims_and_evidence": "Yes, it is clear and convincing evidence.", "methods_and_evaluation_criteria": "Yes, the proposed methods and evaluation make sense for the target problem.", "theoretical_claims": "Not applicable. No theoretical claim is proposed.", "experimental_designs_or_analyses": "Yes, I have checked.", "relation_to_broader_scientific_literature": "Evaluating video generation models' capabilities in physical modeling is crucial for developing effective world models. Addressing this research problem is particularly significant for advancements in embodied AI.", "essential_references_not_discussed": "No, the related work is solid.", "other_strengths_and_weaknesses": "**Strengths:**\n\n- The proposed PISA (Physics-Informed Simulation and Alignment) evaluation framework is well-motivated and thoughtfully designed. Experimental results clearly highlight current limitations in the physical modeling capabilities of state-of-the-art video generation models.\n\n- The introduced PSFT and ORO methods demonstrate significant performance improvements on the PISA benchmark. The limitations in generalizing to out-of-distribution (OOD) scenarios are explicitly and effectively discussed.\n\n**Weaknesses:**\n\n- The scope of this paper is somewhat limited. Evaluating only object freefall addresses just a narrow aspect of the physical modeling capabilities of video generation models. The assessment would be more comprehensive and convincing if additional physical scenarios, such as collisions or diverse movement interactions (like in CLEVR dataset), were included.\n\n- Post-training specifically on object freefall scenarios introduces a bias. Although PSFT and ORO effectively enhance performance on the PISA benchmark, the restricted variability in direction and speed during object freefall could allow for dataset-specific optimization or \"hacking.\" This limitation is further corroborated by experimental evidence.\n\n- The improvement methods, while effective, still demonstrate limited generalization to out-of-distribution (OOD) scenarios. This limitation should be addressed by introducing greater variability and complexity in training conditions.", "other_comments_or_suggestions": "No additional comments.", "questions_for_authors": "- If the video generation models become biased toward object freefall after post-training and lose their original capabilities? How might their performance be affected in scenarios involving collisions or movements in different directions (e.g., similar to CLEVR)? Additionally, how do these models perform on standard video generation benchmarks such as V-Bench?\n\n- Regarding the GIF demonstrations provided in the supplementary materials, evaluating models based on higher-resolution object freefall experiments could yield more robust results. Lower-resolution demonstrations might limit the models' ability to accurately generalize semantic categories and effectively reason about object interactions, thereby posing additional challenges for evaluation.", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "5AWVXNKnqe", "reviewer": "Reviewer_hCaK", "summary": "The paper argues that the large-scale video generative modeling has enabled creative content generation but the accurate world modeling is missing. This is due to the complexities in the physical laws and perspectives that underpin the real-world videos. To solve this problem, the paper proposes to use targeted post-training. Specifically, the paper studies the potential of post-training in image-to-video generative models for freefalling objects under gravity. \n\nThe paper studies free falling because it is simple to simulate and evaluate to gain insights into the post-training stage. The experimental results suggest that: (a) the existing video generative models are quite bad at physically accurate object dropping, (b) simple finetuning of the video model on 1000s of examples fixes this problem significantly, (c) they consider further RL-tuning with multiple reward models and target multiple axes for physical improvement. \n\nThe paper also finds that finetuning struggles to generalize beyond the unseen depths and heights, and its struggle with trajectory distribution and dropping-time distribution. Overall, the paper performs insightful data collection and post-training that can serve as a useful data point for the community. I do have several comments on the work’s limitations and potential suggestions.", "questions": "", "limitations": "", "rating": 4, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We thank the reviewer for taking the time to review our work. We are extremely grateful for the feedback given, and below we address the main concerns raised. \n\n> **Q1:** Lack of human evaluation.\n\nThis is a great point. While our work focused on quantitative evaluation, human evaluation is important too. To address this limitation, we asked four volunteers (none of them authors) to rank preferences in generation between our PSFT+ORO model and Sora. We showed 30 videos, 10 from our sim seen test set, 10 from our sim unseen test set, and 10 from our real world test set. For each video, we asked the anotators to select a preference for each of the three following questions:\n\n1. Which video does better at following basic physics?\n2. Which video has better visual realism?\n3. Which video does a better job at preserving the state/identity of objects, i.e. not hallucinating or introducing implausible state changes?\n\nOverall, our model is prefered to Sora 90% of the time in physical accuracy, 56% of the time in visual realism, and 68% of the time in preserving object identity. The full results are shown [here](https://anonymous.4open.science/r/ICML-7650-Rebuttal-1645/7_human_eval.md).\n\n> **Q2:** Evaluating before and after impact.\n\nWe estimated the contact frame using the method described in Appendix B of the paper and ran our benchmark. Overall, we find that ORO most improves the results *after* the point of contact. This indicates that ORO is strongest at improving the most difficult aspects of physics modeling. The full set of evaluation tables can be found [here](https://anonymous.4open.science/r/ICML-7650-Rebuttal-1645/8_impact.md). \n\n> **Q3:** OOD generalization.\n\nWe evaluated our data on challenging and OOD settings in both real and simulated data. These videos include scenarios such as objects sliding down ramps, falling into containers, or domino-like setups. A summary of our relative improvement over the base OpenSora model is shown below. \n\n| Scenarios             | L2     | CD     | IoU     |\n| --------------------- | ------ | ------ | ------- |\n| Domino (Real)         | 47% | 54% | 42%  |\n| Ramp (Real)           | 41% | 47% | 18%  |\n| Stairs (Real)         | 25% | 20% | 103% |\n| Ramp (Simulated)      | 87% | 90% | 55%  |\n| Container (Simulated) | 75% | 67% | 3.7%   |\n\nPlease see [here](https://anonymous.4open.science/r/ICML-7650-Rebuttal-1645/3_ood.md) for more information about the dataset construction and a full table breakdown across settings and comparisons with baselines. \n\n> **Q4:** Attribution for Physics-IQ.\n\nThank you for pointing this out. We did cite this paper in our related work section of our submission, though we did not point out the similarity between our concurrently developed metrics. We will definitely add an acknowledgement of this.\n\nPlease let us know if you have any further questions or concerns. We are grateful that you feel our paper should be accepted, and if our response has sufficiently addressed the concerns you have mentioned, we would appreciate it if you consider raising your score further. Thank you very much again for taking the time to review our work!", "claims_and_evidence": "Yes", "methods_and_evaluation_criteria": "Yes", "theoretical_claims": "NA", "experimental_designs_or_analyses": "Yes", "relation_to_broader_scientific_literature": "Mentioned in the summary, and strengths.", "essential_references_not_discussed": "NA", "other_strengths_and_weaknesses": "**Strengths**  \n- The authors collect 361 real-world videos to assess model performance in real environments, which is crucial for quantifying the sim-to-real gap—an aspect missing from prior work [1].  \n- The entire framework is well-designed for the object dropping phenomenon -- having simulated videos for training the models and assessing them at different heights and depths for generalization study.\n- The paper also goes beyond mere supervised finetuning and shows the ability to improve performance with further RL-tuning.\n\n**Weaknesses**  \n- The study lacks a reliability measure with human evaluation; it is unclear whether humans prefer the finetuned model’s outputs over the generalist model’s.  \n- Figure 7 shows that the object dropping under gravity can be roughly broken down into two parts: straightline motion before impact and collision/rolling motion after impact. That figure suggests that the finetuned model has learned about behavior before motion pretty well but it is not clear about the later part. A good way to test this would be to breakdown the Table 1 finetuned model numbers into before impact and after impact.\n- The study does not establish a strong conclusion on whether post-training improves world modeling in video models. The generalization capability appears limited, and experiments focus only on object falling—a relatively simple physical phenomenon. There is uncertainty about how the results extend to other physical behaviors, given the difficulty in acquiring simulated data for more complex scenarios.  \n\n[1] Section 5.3.2 in https://arxiv.org/abs/2501.03575v1\n[2] https://arxiv.org/abs/2406.03520\n[3] https://arxiv.org/abs/2501.09038", "other_comments_or_suggestions": "- The evaluation metrics closely resemble those in the Physics-IQ paper [3], and proper attribution should be provided.", "questions_for_authors": "Mentioned in the strengths and weaknesses", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {"ethical_review_concerns": "NA"}}, {"review_id": "zCv5PdBKqx", "reviewer": "Reviewer_WLEt", "summary": "Current large-scale pre-trained video generation models excel in content creation but are not suitable as physically accurate world simulators out of the box. Therefore, this paper introduces the PISA framework, providing diagnostic tools for assessing the physical modeling capabilities of video generation models. Experimental validation highlights the crucial role of post-training (fine-tuning and reward modeling) in enhancing physical accuracy.", "questions": "", "limitations": "", "rating": 3, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We thank the reviewer for taking the time to review our work. We are extremely grateful for the feedback given, and below we address the main concerns raised. \n\n> **Q1:** Lack of human evaluation.\n\nThis is a great point. While our work focused on quantitative evaluation, human evaluation is important too. To address this limitation, we asked four volunteers (none of them authors) to rank preferences in generation between our PSFT+ORO model and Sora. We showed 30 videos, 10 from our sim seen test set, 10 from our sim unseen test set, and 10 from our real world test set. For each video, we asked the anotators to select a preference for each of the three following questions:\n\n1. Which video does better at following basic physics?\n2. Which video has better visual realism?\n3. Which video does a better job at preserving the state/identity of objects, i.e. not hallucinating or introducing implausible state changes?\n\nOverall, our model is prefered to Sora 90% of the time in physical accuracy, 56% of the time in visual realism, and 68% of the time in preserving object identity. The full results are shown [here](https://anonymous.4open.science/r/ICML-7650-Rebuttal-1645/7_human_eval.md).\n\n\n> **Q2:** Concern about distribution of dropping times.\n\nWe believe that the distribution matching failure is an important insight for researchers developing world models. A goal of this paper is to make researchers aware of the strengths and weaknesses when post-training video models that struggle with basic physics. As such, we hope the reviewer agrees that negative results like this are important scientific findings that should be shared, even if they are limitations.\n\n> **Q3:** Struggle to fundamentally resolve the lack of physical realism.\n\nOur benchmark and human evaluation indicates our model to be more physically realistic than all other baselines, including frontier closed source models like Sora, on our dropping task. Our goal is not to solve physics as a whole, but rather to shed light on the post-training process that in our view will become an increasingly critical part of the world modeling stack. The strengths and limitations of this post-training process presented in our paper are valuable insights for future research.\n\n> **Q4:** Generalization issues.\n\nWe evaluated our data on challenging and OOD settings in both real and simulated data. These videos include scenarios such as objects sliding down ramps, falling into containers, or domino-like setups. A summary of our relative improvement over the base OpenSora model is shown below. \n\n| Scenarios             | L2     | CD     | IoU     |\n| --------------------- | ------ | ------ | ------- |\n| Domino (Real)         | 47% | 54% | 42%  |\n| Ramp (Real)           | 41% | 47% | 18%  |\n| Stairs (Real)         | 25% | 20% | 103% |\n| Ramp (Simulated)      | 87% | 90% | 55%  |\n| Container (Simulated) | 75% | 67% | 3.7%   |\n\nPlease see [here](https://anonymous.4open.science/r/ICML-7650-Rebuttal-1645/3_ood.md) for more information about the dataset construction and a full table breakdown across settings and comparisons with baselines.", "claims_and_evidence": "The claims are supported by clear evidence.", "methods_and_evaluation_criteria": "The methods and evaluation criteria make sense for the problem.", "theoretical_claims": "The proofs are correct.", "experimental_designs_or_analyses": "The authors designed various metrics to comprehensively reflect the degree of fit between the generated falling trajectories and the real trajectories. However, there are still some issues that require further research, such as the lack of an assessment of human subjective perception of physical realism. I believe this is crucial for the practical application of generated videos. It's difficult to intuitively reflect the degree of this realism with only the quantitative metrics provided in the paper.", "relation_to_broader_scientific_literature": "The methods and benchmarks proposed in this paper are significant for further investigating the key deficiencies in current video generation models.", "essential_references_not_discussed": "N/A", "other_strengths_and_weaknesses": "The methods and benchmarks proposed in this paper are significant for enhancing the physical realism of generated videos. The authors' experiments found that while post-training can effectively improve physical realism, there are certain limitations, such as the incorrect distribution of dropping times, which has not been thoroughly discussed. Moreover, although the problem addressed in this paper is important and the proposed methods and benchmarks are valuable, I believe that this post-training approach struggles to fundamentally resolve the lack of physical realism. The authors' experiments also revealed generalization issues with post-training.", "other_comments_or_suggestions": "I have no further suggestions.", "questions_for_authors": "I have no questions.", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Meta-review: 1x weak reject, 2x weak accept, 1x accept.
This submission introduces a diagnostic benchmark for assessing physical consistency in video generative models by focusing on object freefall, and it proposes post-training techniques—Physics-Supervised Fine-Tuning (PSFT) and Object Reward Optimization (ORO)—to improve physical realism.
The reviewers agree on the (1) value of the novel benchmark for revealing the physical limitations in current video generation models, (2) effectiveness of post‑training in mitigating major physical inaccuracies, and (3) comprehensive experimental design covering real and simulated data, supported further by additional human evaluation in the rebuttal.
They also concur on the (1) narrow scope limited solely to object freefall, (2) marginal improvements in some metrics (e.g. minor IoU gains), and (3) unresolved concerns regarding broad generalization and potential degradation of the base model’s overall performance.
Although Reviewer 7wqN remains less convinced despite the detailed author rebuttal, the strong follow-up responses provided by the authors—especially addressing limited scope, evaluation in diverse scenarios, and training detail clarifications—in combination with the more positive stances from the other reviewers, lead the AC to lean towards acceptance of this submission.
Decision: Accept (poster)