PhysCtrl: Generative Physics for Controllable and Physics-Grounded Video Generation
Authors: Chen Wang
Chuhao Chen
Yiming Huang
Zhiyang Dou
Yuan Liu
Jiatao Gu
Lingjie Liu
Venue: NeurIPS 2025 poster ()
Keywords: Physics simulation
video generation
diffusion models
Primary area: deep_learning

Abstract: Existing video generation models excel at producing photo-realistic videos from text or images, but often lack physical plausibility and 3D controllability. To overcome these limitations, we introduce PhysCtrl, a novel framework for physics-grounded image-to-video generation with physical parameters and force control. At its core is a generative physics network that learns the distribution of physical dynamics across four materials (elastic, sand, plasticine, and rigid) via a diffusion model conditioned on physics parameters and applied forces. We represent physical dynamics as 3D point trajectories and train on a large-scale synthetic dataset of 550K animations generated by physics simulators. We enhance the diffusion model with a novel spatiotemporal attention block that emulates particle interactions and incorporates physics-based constraints during training to enforce physical plausibility. Experiments show that PhysCtrl generates realistic, physics-grounded motion trajectories which, when used to drive image-to-video models, yield high-fidelity, controllable videos that outperform existing methods in both visual quality and physical plausibility. Our code, model and data will be made publicly available upon publication.

PhysCtrl: Generative Physics for Controllable and Physics-Grounded Video Generation
Chen Wang'", Chuhao Chen' Yiming Huang', Zhiyang Dou?
Yuan Liu", Jiatao Gu², Lingjie Liu'
University of Pennsylvania, PMIT, *HKUST "equal contribution
{chenw30, chuhaoc, chuhaoc,,mhuaneeee ywhuung9,,,,,. , jgu32, lii.lninniii.. liu]-seas upenn edu frankdou@mit edu; yuanlyOust.ir https://cwchenwee gi titb io physctrt
Input Image & Force
Generated Trajectories Video
Fankira
10
Figure 1: We propose PhysCtrl, novel framework for -rsiise- image-to- video generation with physical material and force control. PhysCtrl supports generating physics-plausibie motion trajectories across multiple materials as control signals (second row), and allows controls over physics parameters (e.g.. Young's Modulus of elastic material (third row)) and force (last row). Note that in the bottom three rows, overlaid trajectories and frames use lighter hues for earlier time steps and darker hues for later ones.
Abstract
Existing video generation models excel producing photo-realistic videos from text images, but often lack physical plausibility and 3D controllability. To overcome these limitations, introduce PhysCtrl, novel framework for physics- grounded image-to-videe video generation with physical parameters and force control. core a generative physics network that learns the distribution of physical dynamics across four materials (elastic, sand, plasticine, and rigid) via diffusion model conditioned on physies parameters and applied forces. We represent physical dynamics as 3D point trajectories and train large-scale synthetic dataset of 550K animations generated by physics simulators. enhance the diffusion model with novel spatiotemporal attention block that emulates particle interactions and incorporates physics-basee constraints during training to enforce physical plausibility. Experiments show that PhysCtrl generates realistic, physics-groundee motion trajectories which, when used to drive image-to-vine models, yield high- fidelity, controllable videos that outperform existing methods in both visual quality and physical plausibility.
39th Conference on Neural Information Processing Systems (NcurIPS 2025).

---

Introduction
Video generation has emerged as a transformative technology, powering applications in gaming [7. 80, 14], animation [10, 92, 26], autonomous driving [84,86], digital avatars [30, 101], robotics [49]. Modern video generative models [62, 92, 94, 4] can produce photo-realistic videos from text or single images. However, they often lack physical plausibility, controllability over dynamic physical behaviors and high fidelity, because they are trained on massive 2D videos in a pure data-driven manner [2, 3].
To achieve
video generation, incorporating inductive biases of physical dynamics
is crucial. Driven by this, recent works have combined physies simulators [37, 1, 56] with neural representations (e.g., Gaussian splats) to simulate rigid or non-rigid dynamics and render them into videos [91, 38, 51, 76] under scene-specific settings. While physics simulators based Newtonian mechanics can model the dynamics of diverse real-world gystems-including soft/rigid bodies, fluids, and gases [37, 59, 56], they suffer from high computational cost, sensitivity to hyperparameters (e.g., simulation substeps, grid size), numerical instabilities, and tradc-off's between generality and accuracy, As a result, when directly using physics simulator for video generation, people have to tune several hyperparameters and might need to switch simulators with regard to object material (e.g. MPM for elastic and rigid body simulators for rigid). It might also lack robustness and suffer from slow speed (especially for inverse problems).
To address these issues, we propose PhysCtrl, framework for physics-grounded image-to-video generation with explicit control over physical parameters and external forces. A key component of framework is generative physics network, diffusion-based model that learns the distribution of physical dynamics. It works on various material types, requires minimal user input and supports fast forward and backward. Conditioned on physical parameters and applied forces, it predicts physical dynamics that serve as control signals for pretrained video generative models [24]. In our design, we address two fundamental questions to achieve robust, efficient, and generalizable physics priors for controllable video generation:
1. What is appropriate representation of physical dynamics for providing control in video models? We seek a representation that enables efficient control of video models while generalizing across wide range of materials. Very recent work controllable video generation [21, 24] has shown that video models can synthesize rich and coherent content from only sparse and explicit point controls. Mcanwhile, point clouds offer greater flexibility and generalization for modeling different materials than other explicit representations, such as meshes or voxel grids, making them more suitable for learning-based generative physics networks. Considering these two aspects, we propose to represent physical dynamics as 3D point trajectories, enabling compact motion encoding and seamless integration with video generative models while supporting diverse material types.
2. How to embed generative physics priors across various materials into a network? High-quality and diverse data essential for learning the distribution of physical dynamics (i.e., generative physics). We therefore collect a large-scale synthetic dataset of 550K object animations across four material types (elastic, sand, plasticine, and rigid), capturing complex, pwysics-groundeeee dynamics via physics simulators. Using this dataset, we design a diffusion model to generate plysics-plausibe 3D motion trajectories conditioned on physical conditions. Inspired by particle dynamics [37], where particles interact with neighbors to determine their next state, we introduce novel spatiotemporal attention block in the diffusion model to emulate these interactions: it first aggregates spatial influences from neighboring points and then predicts each point's 's trajectory over time. Finally, to embed explicit physical knowledge directly into the network, we incorporate physics-basee based constraints during training, ensuring that the generated motions are
We conduct comprehensive evaluations of our method, demonstrating our model can produce physics- plausible motion trajectories. We further show that the generated trajectories can be used as the input for majjecory-conditione video model for image-to-vides generation, outperforming existing video generative models in both visual fidelity and physics plausibility. Our key contributions are:
We introduce PhysCtrl, a novel and scalable framework that represents physics dynamics as 3D point trajectories over time, enabling physics-grounder image-to-video generation with explicit control over physical parameters and external forces.

---

We develop a diffusion based point trajectory generative model equipped with spatiotemporal attention mechanism and physics- based constraints, efficiently learning generative physical dynamics across four material types.
We collect a large-scale synthetic dataset of 550K object animations, spanning elastic, sand, plasticine, and rigid materials, using physics simulators. We will release this dataset support future research in physical dynamics learning.
We demonstrate the effectiveness of PhysCtrl in generating realistie, physics-grounded dynamics and achieve high-quality image-to-video generation results given user-specified physics parame- ters and external forces.
2 Related Work
Neural Physical Dynamics Traditionally, physical dynamics are solved with numerical methods such as finite element method (FEM) [102), position-based dynamics (PBD) [60, 55J, material point method (MPM) [37], smoothed-particle hydrodynamics (SPH) [17, 63, 43] and mass-spring systems [52]. Physical Informed Neural Networks (PINNs) [64] use neural networks to approximate the solution of partial differential equations and incorporate physics constraints in the loss functions. Combined with neural fields [58], PINNs achieve success in domains like fluids [13, 85] but are limited in per-scene optimization setting. Concurrent work, ElastoGen [19], replaces part of the physics simulation with neural networks for faster inference, but relies on a voxel representation, supports only elastic materials, and requires full 3D model as input. Graph Neural Networks (GNNs) have emerged as an effective tool for modeling particle interactions with diverse material types [69, 93, 70, 95]. However, such approaches typically rely on next-step predictions for modeling dynamics, making them susceptible to drift and error accumulation over time. In contrast, our method represents objects as flexible point clouds and leverages a spatio-temporal trajectory diffusion model to robustly capture the dynamics of diverse materials in unified framework.
Controllable Video Generative Models Video generative models are trained on massive text-video paired datasets and achieve high-quality video generation [29, 4, 41, 11, 94]. Existing works have shown that additional control signals can be injected into pretrained models for controllable video generation, such as camera movement [25, 20], human pose [30], and point movement [21, 24, 5]. However, these models lack an understanding of physical laws and thus generate outputs that are often not physically plausible. Furthermore, they cannot support explicit physics control. Our work focuses on generating physics-groundes dynamics that can be used as physics control signal for video models.
Video Generation Existing methods leverage physics simulators to produce videos. One approach reconstructs neural representations from multi-view images, applies simulation on these representations, and then renders the results into video. For example, PhysGaussian [91], Spring-Gaus [99], and Vid2Sim [9] integrate MPM, spring-mass systems, and LBS-based simulation [59] into 3D Gaussians for simulation and rendering. VR-GS [38] applies physics-aware Gaussian Splatting in VR/MR for real-time, intuitive 3D interaction and physics-based editing. PhysDreamer [97] distills motions from video models to estimate physics parameters. These methods are scene-specific and require high-quality 3D reconstruction to achieve good results. Re- cently, researchers started to combine physics simulators with video generative models. PhysGen [51], PhysGen [8] and PhysMotion [76] generate videos of 2D rigid body dynamics or deformable dy- namies. These methods rely on physics simulators generate dynamics and coarse texture and only use video models for texture refinement. PhysAnimator [90] combines physical simulators and a sketch-guided video diffusion model for animations. Compared with methods that rely on physics simulators, our method embeds physics priors into diffusion model, which avoids manual hyperparameter tuning and improves numerical stability for dynamics prediction. The predicted dynamics can be used as guidance for video generative models to synthesize physics-grounded and controllable videos. Concurrent works WonderPlay [48] and Force Prompting [22] also investigate using force as the condition signal for video generation.
4D Dynamics Parametric models have been widely used to represent category-specific deformable shapes, such as SMPL and SMAL [54, 103] for human and animal bodies, FLAME [47] for faces, MANO [67] for hands. Recent advances in 4D dynamics have been exploring to capture object dynamics of arbitrary topologies [61, 57, 77, 45, 77, 12] with Neural-ODE and coordinate-MLPs.
3

---

With the success of diffusion models [28, 72, 73, 74] on high-quality generation on several modalities, including text [23]. image [66, 68], audio [42, 44, 33], video [29, 27] and 3D [50, 71, 96, 53], researchers have started to learn the distribution of object dynamics with diffusion models [18, 6, 98, 88]. VecSonsVeeSets [6] introduced a 4D representation with latent vector sets, and trained conditional diffusion model for dynamic reconstruction from sparse point cloud sequences. DNF [98] leverages a dictionary-basee generation. However, these methods are only trained on datasets with limited number of shapes that contain only human and animal motions, while our method focuses on learning physics- physics-grounded dynamics, which contain large variety of dynamic phenomena. We also more flexible point representation that is better suited for downstream tasks.
neural field to learn a compact motion space for unconditional 4D
grounded
3 Preliminary
We generate ground-truth point trajectories for training our generative physies network (also referred to as Sphysics-groundee trajectory generative model") on data synthesized by physics simulators, including MPM and rigid body simulators. Here we review the basics of MPM, which form the basis for our physics-awand aware constraint in Section 4.
Material Point Method Material Point Method (MPM) [75, 65, 40, 37, 31, 91] simulates the deformation of discrete material particles under the assumption of continuum mechanics, where the transformation of each particle from the material space to the world space is defined by deformation mapping 6(X, t). and the associated deformation gradient F = V (X, f) measures the local deformation of the material such as rotation and stretch. The evolution of @ at time is governed by the conservation of mass and momentum, which can be formulated as
Dv Dt
Up Dt
+ fext
-V-
(1)
where p. v and fext denote the density, the velocity field and the per-unit volume external force respectively. The Cauchy stress Fe oF derived from the deformation gradient F and physics parameters (e.g. Young's 's modulus and Poisson's 's ratio v) related to specific constitutive models. Based on Equation (1), MPM associates particles with background grids in the simulation, performing a particle-to-gric (P2G) and grid-to- particle (G2P) transfer loop. For stepping to + 1, the P2G transfer can be formulated
8y (F)F and the energy density function (F) are
mi (v 1+ v() = Σ vo aT OF (F)) (x,)
(2)
At
where p and represent attributes for particle and grid. 1 is the initial particle volume and (x())) the B-spline kernel defined on i-th grid evaluated at x Grid mass m = Σp N₂ (xxx)))m and grid momentum m = Ex (xxx).)mp(( + C, - xf,)) are obtained according to the standard APIC [36], where C is the affine matrix. The G2P transfer can be formulated as:
C (Ar)² Σ
x
F++1 = AI+ Σ v Nx ))F
(3)
Afterwards, vp and xp are updated as vot1 E, (((xt) ++ and x x Aiv
4 Method
Given a monocular image, our method generates physics-grounde videos with the control signals of physics parameters and external forces. The core part of our method is conditional diffusion point cloud trajectories (Section 4.1) with physics parameters
model to generate and external forces conditioning. To enable that, illustrated in Figure we first lift the input image into 3D points (Section 4.2). Once we obtain the generated trajectories, we leverage them as the condition pre-trained video models for image-to-video synthesis (Section 4.2).
4.1 Physics-Groundee Generative Dynamics
Our goal is to learn the distribution of physical dynamics across various materials - termed generative dynamics using a diffusion- based model, thereby avoiding the high cost, hyperparameter sensitivity,

---

Point Cloud Lifting
Trajectory Generation
Video Generation
Object
Plyllis Aurewterr
Mape
Sagnarndd Object
Gradied Falen
Segment
Embed
Mult: Vier
trago
Input Image
Nevdl View Genecation
Physics-Grounded Grounded Trajectory Generative Model
Pre- tre-trained Video
Generative Model
Ourppnt Video
Trainable Models
Pdin
Frazen Modeels
Point Cloud
Trojectory
Recon.
User Input
Figure 2: An overview of PhysCtrl Given a single image, we first lift the object in that image into 3D points. We then generate physics-grounded motion trajectories conditioned on physics parameters and external force with diffusion model, which are then used as strong physics-grone grounded guidance for image-to-vidde generation.
numerical instabilities, and generality-accuracy trade-off's of classical simulators. We select point clouds as our representation because they flexibly model diverse materials and suffice to control pretrained video models. Specifically, each object is represented by 2048 points in practice; we predict their trajectories over time and use them as control signals for video synthesis. We use 2048 points for guiding video model because prior work [24] show that it can achieve similar results with more points. Also, works on 4D reconstruction and generation [34, 46, 97] demonstrated that real-world motion can be represented with sparse number of basis or control points.
4.1.1 Problem Setting
Given an object, represented as 3D point cloud with N points Po = {x" E R and its physics param- eters (E, v). our trajectory generative model generates its dynamics given an initial force. Specifically, the dy- namics of the object is represented by the position of each point future F timesteps P = pI.F = {P*) p=
Phyaic Conditiona
Nulay Clouds
Xg
Nobann
AdstayerNoom
AdalayerNorm
(xx)) f Denote the force, drag point and bound- ary condition (floor height) f R D R3, and E -... Thus, the goal of PhysCtrl is to predict P under the condition e = (Po. f. D. {E, v/. h. [mat]]. Here, we use an additional [mat] token to denote different materials. In this paper, we cover four different materials: elastic, plasticine, sand, and rigid. Notably, because of our flex- ible point cloud representation, the model is not limited to these four categories and can be readily extended to other materials, such fluids, given sufficient computational resources.
Spatial Multi-Head Attention
AdalayerNorm
AdalayerNorm
+
Pointwise Feed Forward
AdalayerNorm
Temporal Multi-Head Attention
X
Neet Blck
We train our trajectory generative model on data from physics
Utear
[37] and rigid-body solver. Simulator hyperparameters (c.g., substeps, grid size) intro- duce variability that our model, conditioning only on core physics parameters, does not capture directly. To account for this uncertainty, we employ diffusion model to learn the conditional distribution p(Plc). Our method can also be extended to learning physics from more simulation methods since it requires only sampled points.
Xg
x
Denoised
Figure 3: Our trajectory generation architec- ture which consists of spatial attention and temporal attention in each block.
4.1.2 Physics-grounded Trajectory Generative model
Prior trajectory generative models for human motion synthesis [79, 100] typically project all point positions into single latent space, applying attention to only temporal correlations. This approach is inadequate for our setting (see Figure 1), as overlooks crucial spatial relationships. While naive 4D attention across both space and time can model spatio-temporal correlations in physics simulation
5

---

data, it is suboptimal in terms of quality and efficiency due to the combinatorial explosion of spatial points across time steps. Instead, since we aim model point cloud trajectories with a one-to-one point correspondence across frames, we introduce an efficient attention mechanism tailored for physics simulation data, which first applies spatial attention followed by temporal attention. This design reduces the computational complexity and, more importantly, reflects the underlying process of physics simulation: first integrating information from neighboring points, then propagating forward in time dimension.
Specifically, given noisy point cloud sequences, we apply point embedding and project it to latent dimensions, add sinusoidal positional embeddings in both space and time and predict its trajectory offset with our denoising network D. The core of network D is diffusion transformer consisting of a set of spatial-temporal attention blocks as shown in Figure 3. Each block contains two attention layers: spatial attention and temporal attention.
Spatial attention learns the correlation of each point with other points in the same frame with self- attention. To inject physical conditioning into the attention layer, we first map them into additional tokens using MLPs: cond = MLPphhs ([: D; {E, 2) , h, [mat]] E IR. Then, we concatenate them with point positions along the sequence dimension. Motivated by CogVideoX) [94], we apply the adaptive layer norm to positional tokens and physical tokens separately to facilitate the alignment between the two spaces:
p = SelfAttn (AdaLN( cond))) Vf E []
(4)
Temporal attention mainly aggregates information of the same point across all timesteps for temporal consistency. We also apply attention to the input point cloud Po for better trajectory learning.
T = SelfAutn (TT)),
Vp E [1,N]
(5)
where Тр [xx
x
x
x (F+1)xd
4.1.3
Training Losses
We train a standard diffusion model in which we add Gaussian noise c of different levels to the entire point cloud sequence: P. = a,P + d,C and then feed the noisy point cloud sequence into the denoising network DD. We use the -redictpre formulation of diffusion models: P = D(P,, 4, c).
Diffusion Loss We use MSE loss between the predicted and ground truth signal given noise samples:
Lan PPp-
Velocity Loss We regulate the velocity across two frames, similar to that used in MDM [79]:
F-1 [(p/+1 1(p/+1-pl) 110+1-p11, -p') - -p/11-
Evel
Σ
F
Physics Loss To enable the model to learn plysics-plausble motion trajectories, we introduce physics-based supervision as regularization to enforce physical plausibility for the elastic, plasticine and sand material from MPM. Specifically, constrain the position and velocity of the predicted points to adhere to the deformation gradient update (Equation (3)) across frames:
P-2 N
Sphys N(F - 2)
1
F f+
= I+ 0(x))=
IT Σ of VN(x x (8)
f=1
where Ff+11 and F/P are the ground-truth deformation gradient between adjacent frames and X/ E ps is the predicted position. To obtain approximation of grid velocity v in Equation (8), perform one P2G and G2P step (Equation (2)) at each frame in training. This can be formulated as
Ep No (5.(s ( (8 + +c((x, - x(,))
++
Ep (x/.)mp

---

where C is also from ground ground-truth and 4/-1_ = (x/+2 x2)((AT)) Note that we ignore the stress term and next-frame point velocity + because yields accurate approximation when the frame interval AT is much larger than the substep interval At for MPM simulation.
Boundary Loss To enforee the boundary condition of the ground, we add a penetration loss, preventing the points from passing through the surface:
Lavorr
(max(h- 0))
(10)
N f=1p=1
Overall, our training loss is: = Lain + Avel + Aphys + Aloooe Lowor-
4.2 Physics-grounded Image-to-
Mnddeo Video Generation
Starting with single image of 3D scene with objects, we first segment out [39] the objects and generate novel view images for each object. We then feed both the novel views and the segmented image into a multiview Gaussian reconstruction model [78] and extract a point cloud for the input objects. For input with floor conditions, we support user input to select the floor region and use VGGT [83] to reconstruct the 3D scene. Then we align the coordinate system of VGGT and the 3D points of the object and obtain the height of the floor using principal component analysis. We then use our trajectory generative model to generate the dynamics of object points. The generated 3D point trajectories are then projected to the image space of the input camera viewpoint to obtain the motion trajectories of each pixel. The projected pixel trajectories can be directly used as conditioning signals for a pre-trained video generative model to produce the final video. Specifically, we use DaS [24] as the video model. It takes "tracking video" as condition, which is the projected 3D point trajectories of 2D grid anchor points at the first frame. For each anchor point, we associate it with the nearest 3D object point. Then, project the 3D point trajectories into 2D and get the final tracking video.
"The chair was gently lifted apseaeds on the left seot back in smooth, controlled motion, as if gentle upwards force dragging it the back. backgrourd remains wichatnged"
Auturistic spiked UFO havers above glawing clowds at sunser. it rotates counter clockeiss and farmnsy with steody motion."
Dropkeything
Warz.1
CogWicco
Ours
Figure 4: Qualitative comparison between our method and existing video generation methods.
Input Image & Force
Generated Tracks and Video
Figure 5: PhysCtrl generates videos of the same object under different physics parameters and forces.
7

---

Experiments
5.1 Evaluation on Image-to-Vides Video Generation
Baselines We compare PhysCtrl with state-of-the-arr controllable video generative models, including Wan2.1-12V-144 [82], CogVideoX [94], DragAnything [89], ObjCtrl-2.5D [87]. The first two methods support amage-to-vides generation with text prompts. We use ChatGPT-40 to generate text prompts based on the direction of the object movement. The last two achieve controllable video generation with user-specified single-point trajectories. We use the trajectories of the drag point generated by our model to prompt the model.
Quantitative Evaluation Since we are the first method to inject physics prior into a video model, we utilize GPT-40 evaluate three aspects of 12 generated videos in a 5-Likert score inspired by VideoPhy [2]: (1) Semantic Adherence (SA): how well the content and motion in the video match the description in the text prompt, especially the alignment with the force direction and position; (2) Physical commonsense (PC): whether the object's motion follows intuitive, physically plausible dynamics given the applied force direction and position;
(3) Video Quality (VQ): overall visual and temporal quality of the video. Results in Table show that method achieve the best results across all baselines. Results of user study can be found in the supplemental.
The qualitative results between method and baselines be found in Figure 4. CogVideoX-5B [94] and Wan2.1 [82] have strong generation ability and partly follow the text prompts. However, they only use text prompts conditions and lack precise control, thus, they cannot produce motions that fully reflect physics conditions. For example, the chair in Figure doesn't move according to the force direction. Anythnng [89] uses purely 21) trajectories and cannot distinguish between camera motion and object motion, thus sometimes generating camera motions while objects remain static. More importantly, both Drattnht [89] and ObjCtrl2.5D [87] only use coarse trajectory as a condition and struggle to generate more complex motions, e.g, the UFO case in Figure that contains both rotations and depth change. In comparison, PhysCtrl produces physics-plausible
Qualitative Evaluation
physics- plausible videos that follow the given forces by generating physics-grounded 3D trajectories as strong conditional signal to guide the superior generation capability of pretrained video generative models for video synthesis.
Table I: Results of video evaluation.
Table 2: Quantitative comparison trajectory
generation.
SAT
PCt
VQf
Method
vloUf
CDJ
Corr
DragAnything [89]
2.9
2.8
2.8
ObjCtri [87]
1.5
1.3
1.4
M2V [6]
24.92%
0.2160
0.1064
Wan2.1 [82]
3.8
3.7
3.6
MDM [79]
53.78%
0.0159
0.0240
VogVideoX [94]
3.2
3.2
3.1
Ours
77.59%
0.0028
0.0015
Ours
4.5
4.5
4.3
Results on Varying Physical Conditions Since our trajectory generative model is conditioned on external forces and physics parameters, we can generate videos of the same object under varying conditions. As shown in Figure 5, we can change the Young's modulus in elastic material to produce results with different deformations given the same force. The direction and amplitude of the force can also be adjusted to match the user's desired motion. We found that Poisson's 's ratio y has negligible influence on the generated trajectories, similar to the findings in PhysDreamer [97].
5.2 Evaluation on Generative Dynamics
Baselines We compare our approach with existing methods that focus on generative dynamics, including Motion2VecSet, [6] and MDM [79]. Motion2VecSets is method for reconstructing sparse point cloud sequences; eliminate the sparse point cloud condition and introduce physics conditions instead. MDM is primarily aimed at human motion generation, so we substitute human joints with point clouds and incorporate physics conditions as additional tokens. For computation efficiency, we trained all baselines and ablations on our elastic subset of 160K objects that contains complex deformations for metrics comparison.
Evaluation Metries Following [45, 6]. we adopt volume Intersection over Union (vloU), Chamfer Distance (CD) and Li-distance error for evaluation. vloU measures the overlap between predicted and ground truth point clouds, CD measures the averaged per-point pairwise nearest neighbor distance
8

---

between two point clouds, Ly-distance is the Euclidean distance between two corresponding point clouds. Each metric is calculated at each timestep separately and averaged across all frames.
Input Points
Input Points
122
MDNY
MDM
Oura
Ours
Reference
Reference
Figure 6: Qualitative results: Compared to baselines, our method enables high-quality and coherent generation of motion sequences from physics conditions and closely matches the reference.
Table 3: Ablation study on trajectory generative model.
Table 4: Ablation study on using traditional sim-
ulator and our model for video generation.
Method
vloL.f
CDJ
Cor+
Method
SAT
PC1
VQf
w/o spatial attention
33.76%
0.2348
0.1163
Traditional (2048 points)
4.3
4.3
4.4
temporal attention
33.63%
0.0480
0.0507
wlo physics loss
76.30%
0.0030
0.0016
Traditional (8 192 points)
4.3
4.3
4.2
Ours
77.59%
0.0028
0.0015
Ours (2048 points)
4.5
4.5
4.3
Results Table shows the quantitative comparison of method with other baselines. Our method demonstrates the best performance over all metrics on testing set. The qualitative comparison be found in Figure 1. Our model achieves trajectories. Motion2vecsets struggles to generate time-coherent motions because our experiments, there is sparse point cloud condition in their original setting. M2V struggles to generate coherent motions in our experiments. There are two potential reasons for this. Firstly, their model is originally designed for point cloud completion, but in our setting, there is sparse point cloud condition. Prior work [98] also found that M2V does not work well in this situation. Secondly, their deformation latent is encoded frame-by-free -f-mre without temporal interaction. MDM generate consistent motion sequences, but fails to capture detailed deformations because all points in frame are projected into single latent. The superiority of our method is based our spatial-temporal attention block, which leverages explicit per-point correspondence.
and consistent generation of motion
5.3 Ablation Study
The qualitative and quantitative results of the ablation study for trajectory generation can be found in Figure and Table 3. Our physics loss improved all the metrics and makes the results our trajectory generation close the ground truth. The physics loss aligns the updated deformation gradient with the ground truth and constrains the predicted positions. Although without physics loss, our model achieve good results, it can be further improved with physical guidance regularization.
Table presents the ablation study for video generation. Results show that using our trajectory generation model for video generation is on par with using traditional simulator. Also, results also show that using more points didn't bring a performance gain for video generation.

---

Cloud
Traj. physics losa)
(Ours)
(Reference)
Point Cloud
Tral. (w/o
Traj. (Oun)
Traj, (Reference)
Figure 7: Comparison of using physics loss trajectory generation. Here we show the final point position and tracks for points. With physics loss, the results are more closely aligned with the reference (simulated by MPM).
Point Cloud
Traj, (Ours)
Traj. (Reforence)
Input Image & Force
Trajectory
Last Video Frome
Figure 8: Failure cases.
Discussion
Failure Cases As shown in Figure our model contains three components: point cloud lifting. trajectory generation and video generation. (1) Failure due to point cloud lifting is extremely rare. Single image-to-3D produces reasonable geometry overall and unlikely to yield severely distorted or implausible shapes. While minor artifacts (e.g. geometry not perfectly smooth or noisy points on the surface) may occur, they have minimal impact on our results. We achieve such robustness to geometric variations because trajectory generation model the diverse Objaverse dataset and applying data augmentation of surface noise. (2) Failure cases our trajectory generation model: model cannot handle thin structures well and sometimes might fail to accurately capture complex internal deformations. (3) Failure cases due to video generation: The video model cannot fully follow the trajectory when it conflicts with the prior ofthe video model. For example, when the user input for the object material very stiff, but it appears soft according to RGB information. The video model might also hallucinate unexpected content in the occluded region, e.g, for animal, it might generate five legs. See Figure 8 for example failure cases.
Extension to Multiple Objects MPM achieves multi-object interaction by representing scene dynamics point movement. Our model also predicts point trajectories, so it is also inherently capable of multi-object interaction. We did a preliminary experiment on multi-objects on a simplied setting: we create dataset that has an object dragged towards a cube and colliding with the cube from different angles and distances. We trained our model it and achieved 93.70% vIOU on the held out testing set.
7 Conclusion and Limitations
In this paper, we introduce PhysCtrl, novel framework for -rrsunde video generation with physics parameters and force control. We design a diffusion model with spatial-temporal attention blocks and physics-based supervision to effectively and efficiently learn complex physical deformations directly on point cloud sequences. The generated motion trajectories can be used as a strong conditional signal for pre-trained video generative models. Our experiments demonstrate that PhysCtrl can generate physics- physics-grounded grounded dynamics and enable high-quality image-to-video generation results conditioned external forces and physics parameters.
Our approach mostly focuses on single-object dynamics for four material types and does not cover all possible materials. We only do initial study on multiple objects and more complex phenomena should be investigated, such as intricate boundary conditions. Future work includes addressing these limitations and extending PhysCtrl to more diverse and complex physies phenomena in the real world.
10

---

References
[1] Pymunk (2023), https:
[2] Bansal, H., Lin, Z., Xie, T., Zong, Z., Yarom, M., Bitton, Y., Jiang, C., Sun, Y., Chang, K.W.,
Grover, A.: Videophy: Evaluating physical commonsense for video generation. arXiv preprint arXiv:2406.03522 (2024)
[3] Bansal, H., Peng. C., Bitton, Y.. Goldenberg, R., Grover, A., Chang, K.W.: Videophy-2: A challenging action-centri physical commonsense evaluation in video generation. arXiv preprint arXiv:2503. 06800 (2025)
[4] Blattmann, A., Dockhorn, T., Kulal, S.. Mendelevitch, D., Kilian, M., Lorenz. D., Levi, Y..
English, Z., Voleti, V., Letts, A., et al.: Stable video diffusion: Scaling latent video diffusion models large datasets. arXiv preprint arXiv:2311.151 15112 (2023)
[5] Burgert, R., Xu, Xian, W., Pilarski, O., Clausen, P., He, M., Ma, L., Deng, Y., Li, L.. vidco diffusion models using real-time warped noise. In: Proceedings of the Computer Vision and Pattern Recognition Conference. pp. 13-23 (2025)
Mousavi, M., et al.: Go-with-the-fllow
[6] Cao, W. Luo, C., Zhang, B., Nießner, M., Tang, J.: Motion2vecsets: 4d latent vector set diffusion for non-rigid shape reconstruction and tracking. In: CVPR. pp.
(2024) [7] Che, H., He, X., Liu, Q.. Jin, C.,, Chen, H.: Gamegen-x: Interactive open-world game video generation. arXiv preprint arXiv:241 0101:2411.00700 1.00769 (2024)
[8] Chen, B., Jiang, H., Liu, S., Gupta, S., Li, Y., Zhao, H., Wang, S.: Physgen3d: Crafting miniature interactive world from single image. arXiv preprint arXiv:2503. (2025)
[9] Chen, C.. Dou, Z., Wang. C.,, Huang, Y., Chen, A., Feng, Q.. Gu, J., Liu, L.: Vid2sim: Generalizable, video-based reconstruction of appearance, geometry and physics for mesh-free simulation. IEEE Conference on Computer Vision and Pattern Recognition (CVPR) (2025)
[10] Chen, H., Xia, M., He, Y., Zhang, Y., Cun, X., Yang, S., Xing, J., Liu, Y., Chen, Q., Wang, X., et al.: Videocrafter1: Open diffusion models for high-quality video generation. arXiv preprint arXiv:2310.19112 (2023)
[11] Chen, H., Zhang, Y., Cun, X., Xia, M., Wang, X., Weng, C., Shan, Y.: Videocrafter2:
Overcoming data limitations for high-quality video diffusion models. In: CVPR. pp. 7310- 7320 (2024)
[12] Chen, R.T., Rubanova, Y., Bettencourt, J., Duvenaud, D.K.: Neural ordinary differential equations. NeurIPS 31 (2018)
[13] Chu, M., Liu, L., Zheng, Q., Franz, E., Seidel, H.P., Theobalt, C., Zayer, R.: Physics informed neural fields for smoke reconstruction with sparse data. ACM TOG 41(4), 1-14 (2022)
[14] Decart, E., Campbell, S., McIntyre, Q., Chen, X., Quevedo, J.: Oasis: universe in transformer (2024)
[15] Deitke, M., Liu, R., Wallingford, M., Ngo, H., Michel, O., Kusupati, A., Fan, A., Laforte, C., Voleti, V.. Gadre, S.Y., al.: Objaverse-xl: A universe of 10m+ 3d objects. Advances in Neural Information Processing Systems 36, 35799-3588 (2023)
[16] Deitke, M., Schwenk, D., Salvador, J., Weihs, L., Michel, O., VanderBilt, E., Schmidt, L., Ehsani, K., Kembhavi, A., Farhadi, A.: Objaverse: A universe of annotated 3d objects. In:
CVPR. pp. 13142-13153 (2023)
[17] Desbrun, M.: Smoothed particles: A new paradigm for animating highly deformable bodies.
Computer Animation and Simulatioon/Springer Vienna (1996)
[18] Erkoç, Z., Ma, F. Shan, Q. Nießner, M., Dai, A.: Hyperdiffusion: Generating implicit neural fields with weight-space diffusion. In: ICCV. pp. 13300-14310 (2023)
[19] Feng. Y., Shang, Y., Feng, X., Lan, L., Zhe, S., Shao, T., Wu, H., Zhou, K., Su, H., Jiang, C., et al.: Elastogen: 4d generative elastodynamics. arXiv preprint arXiv:2405.1 150.2405510066 (2024)
[20] Fu, X., Liu, X., Wang, X., Peng. S., Xia, M., Shi, X., Yuan, Z., Wan, P., Zhang, D., Lin, D.: 3dtrajmaster: Mastering 3d trajectory for multi-entity motion video generation. arXiv preprint 172.792412.07759 (2024)

---

[21] Geng, D.. Herrmann, C., Hur, J., Cole, F., Zhang, S., Pfaff, T., Lopez-Guevara,, T., Doersch, C., Aytar, Y., Rubinstein, M., et al.: Motion prompting: Controlling video generation with motion trajectories. arXiv preprint mwXiv:2412.027000 (2024)
[22] Gillman, N., Herrmann, C., Freeman, M., Aggarwal, D., Luo, E., Sun, D.. Sun, C.: Force
prompting: Video generation models can learn and generalize physics-based control signals.
arXiv preprint Xiv:22500 19386 (2025)
[23] Gong, S., Li, M., Feng, J., Wu, Z., Kong, L.: Diffuseq: Sequence sequence text generation with diffusion models. ICLR (2023)
[24] Gu, Z., Yan, R., Lu, J., Li, P., Dou, Z., Si, C., Dong, Z., Liu, Q. Lin, C., Liu, Z., et al.: Diffusion as shader: 3d-aware video diffusion for versatile video generation control. arXiv preprint wiXi:2550100844 (2025)
[25] He, H., Xu, Y., Guo, Y., Wetzstein, G., Dai, B., Li, H., Yang. C.: Cameractrl: Enabling camera control for text-to-video generation. arXiv preprint arXiv:2404.02I1 (2024)
[26] He, Y., Yang. T., Zhang. Y., Shan, Y., Chen, Q.: Latent video diffusion models for figh-fideiity long video generation. arXiv preprint arXiv:2211 1.13221 (2022)
[27] Ho, J., Chan, W., Saharia, C., Whang, J., Gao, R., Gritsenko, A., Kingma, D.P., Poole, B., Norouzi, M., Fleet, D.J., et al.: Imagen video: High definition video generation with diffusion models. arXiv preprint (..i:221000023:: (2022)
[28] Ho, J., Jain, A., Abbeel, P.: Denoising diffusion probabilistic models. NeurlPS 33, 6840-6851 (2020)
[29] Ho, J., Salimans, T., Gritsenko, A.A., Chan, W., Norouzi, M., Fleet, D.J.; Video diffusion models. In: ICLR Workshop Deep Generative Models for Highly Structured Data (2022), attps://openreve ew net/ /
[30] Hu, L.: Animate anyone: Consistent and controllable image-to-video synthesis for character animation. In: CVPR. pp. 8153-8163 (2024)
[31] Hu, Y., Fang. Y., Ge, Z., Qu, Z., Zhu, Y., Pradhana, A., Jiang, C.: moving least squares material point method with displacement discontinuity and two-way rigid body coupling. ACM Transactions on Graphics (TOG) 37(4), 1-14 (2018)
[32] Hu, Y., Li, T.M., Anderson, L., Ragan-Kelley, J., Durand, F.: Taichi: language for high- performance computation on spatially sparse data structures. ACM Transactions on Graphics
(TOG) 38(6), 1-16 (2019)
[33] Huang, R., Huang, J., Yang, D., Ren, Y., Liu, L., Li, M., Ye, Z., Liu, J., Yin, X., Zhao, Z.: Make-an-atee aaudio:: Text-to-audio generation with prompt-enhances diffusion models. In: International Conference on Machine Learning, pp. 13916-13932. PMLR (2023)
[34] Huang, Y.H., Sun, Y.T., Yang. Z., Lyu, X., Cao, Y.P.P Qi, X.: Sc-gs: Sparse-controlled
gaussian splatting for editable dynamic scenes. In: Proceedings of the IEEE/CVF conference on computer vision and pattern recognition. pp. 4220-4230 (2024)
[35] Jiang, C., Gast, T., Teran, J.: Anisotropic elastoplasticity for cloth, knit and hair frictional contact. ACM Transactions on Graphics (TOG) 36(4), 1-14 (2017)
[36] Jiang, C., Schroeder, C., Selle, A., Teran, J., Stomakhin, A.: The affine particle-in-cell method. ACM Transactions on Graphics (TOG) 34(4), 1-10 (2015)
[37] Jiang, C.,, Schroeder, C., Teran, J., Stomakhin, A., Selle, A.: The material point method for
simulating continuum materials. In: Aem siggraph 2016 courses, pp. 1-52 (2016)
[38] Jiang, Y., Yu, C., Xie, T., Li, X., Feng. Y.. Wang, H., Li. M.. Lau, H., Gao, F., Yang. Y., et al.: Vr-gs: A physical dynamics-aware interactive gaussian splatting system in virtual reality. In:
ACM SIGGRAPH 2024 Conference Papers. pp. 1-1 (2024)
[39] Kirillov, A., Mintun, E., Ravi, N., Mao, H., Rolland, C., Gustafson, L., Xiao, T., Whitchcad,
S., Berg, A.C., Lo, W.Y., et al.: Segment anything. In: ICCV. pp. 4015-4026 (2023)
[40] Klár, G., Gast, T., Pradhana, A., Fu, C., Schroeder, C., Jiang, C., Teran, J.: Drucker-prager elastoplasticity for sand animation. ACM Transactions on Graphies (TOG) 35(4), 1-12 (2016)
[41] Kong, W., Tian, Q., Zhang, Z., Min, R., Dai, Z.,, Zhou, J., Xiong, J., Li, X., Wu, B., Zhang, J., et al.: Hunyuanvideo: systematic framework for large video generative models. arXiv preprint arXiv:2412.03603 (2024)
12

---

[42] Kong, Z., Ping, W., Huang, J., Zhao, K., Catanzaro, B.: Diffwave: A versatile diffusion model for audio synthesis. arXiv preprint arXiv:2009.0976: (2020)
[43] Kugelstadt, T., Bender, J., Fernander--eemánddz, J.A., Jeske, S.R., Löschner, F., Longva, A.: Fast corotated clastic sph solids with implicit zero-energy mode control. Proceedings of the ACM on Computer Graphics and Interactive Techniques 4(3), 1-21 (2021)
[44] Lan, Z., Hao, Y., Zhao, M.: Guiding audio editing with audio language model. arXiv preprint
arXiv:2509.21625 (2025)
[45] Lei, J., Daniilidis, K.: Cadex: Learning canonical deformation coordinate space for dynamic
surface representation via neural homeomorphism. In: CVPR. pp. 6624 0624-6634 (2022)
[46] Lei, Weng, Y., Harley, A.W., Guibas, L., Daniilidis, K.; Mosca: Dynamic gaussian fusion from casual videos via 4d motion scaffolds. In: Proceedings the Computer Vision and Pattern Recognition Conference. pp. 6165-6177 (2025)
[47] Li. T., Bolkart, T., Black, M.J., Li, H., Romero, J.: Learning model of facial shape and expression from 4d scans. ACM TOG 36(6), 194-1 (2017)
[48] Li, Z., Yu, H.X., Liu, W., Yang, Y.. Herrmann, C., Wetzstein, G., Wu, J.: Wonderplay: Dynamic 3d scene generation from a single image and actions. arXiv preprint arXiv:2505 18151 (2025)
[49] Liang, J., Liu, R., Ozguroglu, E.,, Sudhakar, S., Dave, A., Tokmakov, P., Song, S., Vondrick, C.: Dreamitate: Real-world visuomotor policy learning via video generation. arXiv preprint arXiv:2406. I 16862 (2024)
[50] Liu, R., Wu, R., Van Hoorick, B., Tokmakov, P., Zakharov, S., Vondrick, C.: Zero-I-to-3: sero-shot one image to 3d object. In: ICCV. pp. 9298 9309 (2023)
[51] Liu, S., Ren, Z., Gupta, S., Wang, S.: Physgen: Rigid-body generation. In: ECCV. pp. 360-378. Springer (2024)
image-to-video
[52] Liu, T., Bargteil, A.W., O O'Brien, J.F., Kavan, L.: Fast simulation of mass-spring systems. ACM TOG 32(6). 1-7 (2013)
[53] Long, X., Guo, Y.C., Lin, C., Liu, Y., Dou, Z.,, Liu, I.., Ma, Y., Zhang, S.H., Habermann, M., Theobalt, C., et al.: Wonder3d: Single image 3d using cross-domain diffusion. In: CVPR. pp. 9970-9980 (2024)
[54] Loper, M., Mahmood, N., Romero, J., Pons-Moll, G., Black, MJJ.: Smpl: A skinned multi- person linear model. In: Seminal Graphics Papers: Pushing the Boundaries, Volume 2, pp. 851-866 (2023)
[55]
Macklin, M., Müller, M.: Position based fluids. ACM Transactions on Graphics (TOG) 32((). 1-12 (2013)
[56] Macklin, M., Müller, M., Chentanez, N.: Xpbd: position-based based simulation of compliant
constrained dynamics. In: Proceedings of the 9th International Conference on Motion in Games. pp. 49-54 (2016)
[57] Mescheder, L., Occhsle, M., Niemeyer, M., Nowozin, S., Geiger, A.: Occupancy networks: Learning 3d reconstruction in function space. In: CVPR. pp. 4460-4470 (2019)
[58] Mildenhall, B., Srinivasan, P., Tancik, M., Barron, J., Ramamoorthi, R., Ng, R.: Nerf: Repre- senting scenes as neural radiance fields for view synthesis. In: ECCV (2020)
[59] Modi, V., Sharp, N., Perel, O., Sueda, S., Levin, D.I.: Simplicits: Mesh-free, geometry-agnostic
elastic simulation. ACM TOG (3(4), 1-11 (2024)
[60] Müller, M., Heidelberger, B., Hennix, M., Ratcliff, J.: Position based dynamics. Journal of
Visual Communication and Image Representation 18(2), 109-118 (2007)
[61] Niemeyer, M., Mescheder, L., Oechsle, M., Geiger, A.: Occupancy flow: 4d reconstruction by learning particle dynamics. In: ICCV. pp. 5379-5389 (2019)
[62] OpenAl: (2024), /ttpp:://polt
com/ index/sora
[63] Peer, A., Gissler, C., Band, S., Teschner, M.: An implicit sph formulation for incompressible linearly elastic solids. In: Computer Graphics Forum. vol. 37, pp- 135-148. Wiley Online Library (2018)
13

---

[64] Raissi, M., Perdikaris, P., Karniadakis, G.E.; Physics -nffroeed neural networks: A deep learning framework for solving forward and inverse problems involving nonlinear partial differential equations. Journal of Computational physics 378, 686-707 (2019)
[65] Ram, D., Gast, T., Jiang, C.. Schroeder, C., Stomakhin, A., Teran, J., Kavehpour, P.: A material point method for viscoelastic fluids, foams and sponges. In: Proceedings of the 14th ACM SIGGGRAAH//angrrahhiss Symposium on Computer Animation. pp. 157-163 (2015)
[66] Rombach, R., Blattmann, A., Lorenz, D., Esser, P., Ommer, B.: High-resolution image synthesis with latent diffusion models. In: CVPR. pp. 10684-10695 (2022)
[67]
Romero, J., Tzionas, D., Black, M.J.: Embodied hands: Modeling and capturing hands and bodies together. ACM TOG 36(6) (2017)
[68] Saharia, C., Chan, W., Saxena, S., Li, L.,, Whang, J., Denton, E.L.,, Ghasemipour, K., Gon- tijo Lopes, R., Karagol Ayan, B., Salimans, T., et al.: Photorealistic text-to-image diffusion models with deep language understanding. NeurlPS 35, (6479-36494 (2022)
[69] Sanchez-Gonzalez, A., Godwin, J., Pfaff, T., Ying, R., Leskovec, J., Battaglia, P.: Learning
simulate complex physics with graph networks. In: International conference on machine learning. pp. 8459-8468. PMLR (2020)
[70] Shi, H., Xu, H., Huang, Z., Li, Y., Wu, J.: Robocraft: Learning to see, simulate, and shape
elasto-plastic objects in 3d with graph networks. The International Journal of Robotics Research 43).), 533-549 (2024)
[71] Shue, J.R., Chan, E.R., Po, R., Ankner, Z., Wu, J., Wetzstein, G.: 3d neural field generation using triplane diffusion. In: CVPR. pp. 20875-20886 (2023)
[72] Song, J., Meng, C., Ermon, S.: Denoising diffusion implicit models (2020)
[73] Song, Y., Ermon, S.: Generative modeling by estimating gradients of the data distribution.
NeurlPS 32 (2019)
[74] Song, Y., Sohl-Dicksttin, J., Kingma, D.P., Kumar, A., Ermon, S., Poole, B.: Score-based generative modeling through stochastic differential equations. In: ICLR (2020)
[75] Stomakhin, A., Schroeder, C., Chai, L.,, Teran, J., Selle, A.: A material point method for snow
simulation. ACM Transactions on Graphies (TOG) 32(4), 1-10 (2013)
[76] Tan, X., Jiang, Y., Li, X., Zong, Z., Xie, T., Yang, Y., Jiang, C.: Physmotion: Physics-grounded (2024)
dynamics from a single image. arXiv preprint arXiv:241
[77] Tang, J., Xu, D., Jia, K., Zhang, L.: Learning parallel dense correspondence from spatio- temporal descriptors for efficient and robust 4d reconstruction. In: CVPR. pp. 6022-6031 (2021)
[78] Tang, J., Chen, Z., Chen, X., Wang, T., Zeng, G., Liu, Z.: Lgm: Large multi-view gaussian model for high-resolution 3d content creation. In: ECCV. pp. 1-18. Springer (2024)
[79] Tevet, G., Raab, S., Gordon, B., Shafir, Y., Cohen-or, D., Bermano, A.H.: Human motion diffusion model. In: ICCV (2023)
[80] Valevski, D., Leviathan, Y., Arar, M., Fruchter, S.: Diffusion models are real-time game engines. arXiv preprint arXiv:2408. 14837 (2024)
[81] Voleti, V., Yao, C.H., Boss, M., Letts, A., Pankratz, D., Tochilkin, D., Laforte, C., Rombach, R. Jampani, V.: Sv3d: Novel multi-view synthesis and 3d generation from a single image using latent video diffusion. In: ECCV. pp. 439-457. Springer (2024)
[82] Wang, A., Ai, B., Wen, B., Mao, C., Xic, C.W., Chen, D., Yu, F., Zhao, H., Yang. J., Zeng, J., Wang. J., Zhang, J., Zhou, J., Wang. J., Chen, J., Zhu, K., Zhao, K., Yan, K., Huang, L., Feng, M., Zhang, N., Li, P., Wu, P., Chu, R., Feng. R., Zhang, S., Sun, S., Fang, T., Wang. T., Gui, T., Weng, T., Shen, T., Lin, W., Wang. W., Wang. W., Zhou, W., Wang. W., Shen, W., Yu, W., Shi, X., Huang, X., Xu, X., Kou, Y., Lv, Y., Li, Y., Liu, Y., Wang. Y., Zhang, Y., Huang, Y., Li, Y., Wu, Y., Liu, Y., Pan, Y., Zheng, Y., Hong, Y., Shi, Y., Feng, Y., Jiang, Z., Han, Z., Wu, Z.F. Liu, Z.: Wan: Open and advanced large-scale video generative models. arXiv preprint (2025)
arXiv:2503.200
[83] Wang. J., Chen, M., Karaev, N., Vedaldi, A., Rupprecht, C., Novotny, D.: Vggt: Visual geome- try grounded transformer. In: Proceedings of the Computer Vision and Pattern Recognition Conference. pp. 5294-5306 (2025)
14

---

[84] Wang, X., Zhu, Z., Huang, G., Chen, X., Zhu, J., Lu, J.: Drivedreamer: Towards real-world- drive world models for autonomous driving. In: ECCV. pp. 55-72. Springer (2024)
[85] Wang, Y. Tang, S., Chu, M.: Physics-informed learning of characteristic trajectories for smoke
reconstruction. In: ACM SIGGRAPH 2024 Conference Papers. pp. 1-11 (2024)
[86] Wang, Y., He, J., Fan, L., Li, H., Chen, Y., Zhang, Z.: Driving into the future: Multiview visual forecasting and planning with world model for autonomous driving. In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. pp. 14749-14759 (2024)
[87] Wang, Z., Lan, Y. Zhou, S., Loy, C.C.: Objctrl-2.5 d: Training-fre freinng-freeeee object control with camera poses, arXiv preprint arXiv:2412.07721 (2024)
[88] Wu, R., Gao, R., Poole, B., Trevithick, A., Zheng, C., Barron, J.T., Holynski, A.: Cat4d: Create anything in 4d with multi-view video diffusion models. arXiv preprint ar11v:2411.18613 (2024)
[89] Wu, W., Li, Z., Gu, Y. Zhao, R., He, Y., Zhang, DJ.,, Shou, M.Z., Li, Y., Gao, T., Zhang, D.: Draganything: Motion control for anything using entity representation. In: ECCV. pp. 331-348. Springer (2024)
[90] Xie, T., Zhao, Y.. Jiang, Y., Jiang, C.: Physanimator: Physics-guide girided generative cartoon animation. arXiv preprint arXiv:2501.16550 (2025)
[91] Xie, T., Zong, Z.. Qiu, Y.. Li, X., Feng, Y.. Yang, Y., Jiang C.: Physgaussiam: Physics- integrated 3d gaussians for generative dynamics. In: CVPR. pp. 4389-4398 (2024)
[92] Xing, J., Xia, M., Zhang, Y., Chen, H., Yu, W., Liu, H., Liu, G., Wang, X., Shan, Y., Wong, T.T.: Dynamicrafter: Animating open-domain images with video diffusion priors. In: ECCV. pp. 399-417. Springer (2024)
[93] Yang, C., Gao, W., Wu, D., Wang, C.: Learning to simulate unseen physical systems with graph neural networks. arXiv preprint arXiv:2201. 1976 (2022)
[94] Yang, Z., Teng, J., Zheng. W., Ding, M., Huang, S., Xu, J., Yang. Y., Hong, W., Zhang, X., Feng, G., et al.: Cogvideox: Text-to-video diffusion models with an expert transformer, arXiv preprint arXiv:2408.0600
(2024)
[95] Zhang, K., Li, B., Hauser, K., Li, Y.: Adaptigraph: Material-adaptive graph- based neural dynamics for robotic manipulation. In: Proceedings of Robotics: Science and Systems (RSS) (2024)
[96] Zhang, L., Wang, Z., Zhang, Q. Qiu, Q., Pang, A., Jiang, H., Yang, W., Xu, L., Yu, J.: Clay: A controllable large-scale generative model for creating high-quality 3d assets. ACM TOG 43(4), 1-20 (2024)
[97] Zhang. T., Yu, H.X., Wu, R., Feng, B.Y Zheng, C., Snavely, N., Wu, J., Freeman, W.T.: Physdreamer: Physics-based interaction with 3d objects via video generation. In: ECCV. pp. 588-406. Springer (2024)
[98] Zhang, X., Li, N., Dai, A.: Dnf: Unconditional 4d generation with dictionary- neural fields. arXiv preprint arXiv:2412.05161 (2024)
[99] Zhong, L., Yu, H.X., Wu, J., Li. Y.: Reconstruction and simulation of elastic objects with spring-mass 3d gaussians. In: ECCV. pp. 407-423. Springer (2024)
[100] Zhou, W., Dou, 7., Cao, Z., Liao, 7.,, Wang, J., Wang, W., Liu, Y., Komura, T., Wang, W., Liu, L.: Emdm: Efficient motion diffusion model for fast and high-quality motion generation. In: ECCV. pp. 18-38. Springer (2024)
[101] Zhu, S. Chen, J.L., Dai, Z.. Dong. Z., Xu, Y., Cao, X., Yao, Y., Zhu, H., Zhu, S.: Champ: Controllable and consistent human image animation with 3d parametric guidance. In: ECCV.
pp. 145-162. Springer (2024)
[102] Zienkiewicz, O.C., Taylor, R.L., Nithiarasu, P., Zhu, J.: The finite element method, vol. 3. Elsevier (1977)
[103] Zuffi, S., Kanazawa, A., Jacobs, D.W., Black, M.I.: 3d menagerie: Modeling the shape and pose of animals. In: CVPR. pp. 6365-6373 (2017)
15

---

The supplementary material covers the following sections: Implementation Details Appendix A, User study Appendix B, Physics Parameter Estimation Appendix C, Results Appendix D, Societal Impacts Appendix F,, Data and Model Safeguards Appendix F. We also encourage readers to refer to our supplementary videos for demonstrations of animatable results..
A Implementation Details
Dataset. To make our model handle diverse objects and motion trajectories, we generate data using physics simulation using high-quality 3D objects selected from ObjaverseXL [16, 15]. We simulate animations for each object with the MPM simulator [37] as the ground-truth. We use fixed number of simulated points N 2048 (uniformly sampled on the faces of the mesh) and frames F 24 align with our model's ' input. For data augmentation, we randomly rotate the object around y-axis and add noise equ ~ (O, 0.01²) to each sampled initial point. Our whole dataset contains 550K objects, including 150K elastic objects of different drag force directions, 100K objects of gravity across elastic, sand, plasticine and rigid respectively. For the simulated animation of varying drag force, we randomly sample a constant force f, a drag point D E Po and physics parameters E E [10*, 10"], magnitude between 0.02G' and 0.3G in total (G the gravity of the whole object) and is only applicd to points close to the drag point D.
[0.05, 0.45]. The force has an outward direction of the object surface and
Training For metric comparison and ablation, we train our base model on the 150K elastic subset that contains different force and physical parameters with 6 layers and 256 latent size on 8 NVIDIA L40 GPUs with 48GB GPU memory for 60K iterations with a total batch size of 32, which takes about 30 hours. We randomly leave out 100 animations from this dataset as the test set and keep the remaining ones for training. We train large model of different materials with 12 layers and a 512 latent size on all 550K data with the same iterations and batch size, which takes about 80 hours. We use Adam W optimizer with betas (0.9, 0.999) and learning rate of 1e-4 with cosine schedule and a warmup of 100 steps. We clip the gradient with the maximum norm of 1.0 and train with 6 precision. We use a DDIM scheduler for sampling. For 25 diffusion steps. it takes Is and 3s for the base and large model. For 4 diffusion steps, takes 0.13s and 0.48s for the base and large model. We found that 4 steps can already achieve great results due to the low uncertainty of the model.
Image-to-3D Pipeline We use SAM [39] to segment the object in the input image and run SV3D [81] to generate 20 novel-view images of that object with orbit camera poses, from which we pick three images with azimuth (90%, 180 270%) relative to the input and send them together with the input into LGM [78] for 3D Gaussian reconstruction. We then convert the 3D Gaussians to plain point cloud and sample N points using farthest point sampling (FPS) for trajectory generation.
GPT-4o Evaluation We prompt GPT-4o with the following prompt use it for evaluation (Results might vary with GPT updates):
You are tasked with evaluating the quality of image-to-vides generation produced by model.
For each test case, you will be given: 1. A text prompt describing a single object and a force applied to it. The force's position and direction are visualized as red arrow in the input image. 2. An input image of the object. 3. Five sets of 10 evenly spaced frames-each set corresponds to video generated by a different model from the same input.
Please evaluate this video based the following three criteria using 5-point Likert scale (1 poor, 5 excellent)
Semantic Adherence: How well the content and motion in the video match the description in the text prompt, especially the alignment with the force direction and position. Note that the video should starts with the input image.
Physical Commonsense: Whether the object's motion follows intuitive, physically plausible dynamics given the applied force direction and position.
16

---

Table 5: Results of user study.
Ours
Cog VideoX
Wan
DragAnything
ObjCtrl2.5D
Physies Plausibility
81.0 %
5.5% 6.2%
10.2% 18.3%
1.2%
2.1%
Video Quality 66.0% %
4.5%
5.0%
Video Quality: The overall visual and temporal quality of the video (note that static nearly-statcc sequences less preferred). Provide your evaluation for each video strictly in the following one-line format:
Video i, Semantic Adherence score, Physical Commonsense score, Video Quality score
B
User Study
We conducted a user study to evaluate the physics plausibility and overall quality of the videos generated by model and other baselines. The study consisted of 12 questions, each including an input image with the force location and direction marked on the image, a text prompt describing the image and applied force, and generated video results produced by five different methods. The users are asked to carefully observe the videos and evaluate them from two aspects: (1) Physics plausibility: select the one that best matches the force direction (red arrow) and corresponding text prompt. The force and text prompt are assumed to match each other. (2) Overall Video quality: Select the one that has the best visual and temporal quality.
We received a total of 35 responses (35 x 12) and computed the percentage times each method was selected the best-performing video for each question. The results are summarized in Table 5, showing the preference rates for each method. The findings indicate that our model consistently outperforms baseline methods in terms of both physics plausibility and video quality. Although Wan received the second-best video quality, some of these high-quality videos suffer from low physics plausibility.
C Physies Parameter Estimation
Our trained trajectory generation model learns the conditional distribution of physically plausible motion trajectories, so it can also be used for inverse problems, i.e, to estimate the condition c given ground truth trajectories P. The intuition is that a that is closer to the ground truth will introduce less discrepancy between the denoised trajectories and ground truth trajectories. To this end, define energy function that measures how well the model can denoise noisy version of P under that condition:
(11) During optimization, the denoiser D frozen and only e is optimizable. We add random noise the ground truth trajectory and feed it into the trained network to denoise. The gradient of the energy function will be backpropagated to optimize e.
E(c) E.-11,T D(Prit,c)1
We simulate 15 trajectories for clastic materials to test our physics parameter estimation pipeline. We compare method with differentiable MPM [32], which needs to accumulate gradients over hundreds of substeps for one backward pass (costing more than 3min compared to Is for ours). Table 6 shows that our method only takes about minutes while achieving relatively good results, which also demonstrates that our trained diffusion model captures physics-plausible motion trajectories.
D
More Results
More results of our method and baseline comparisons can be found in Figure 9, We strongly encourage the readers to look at our video for better comparison, as isolated frames cannot fully represent the physical dynamics well.
17

---

Table 6: Mean Absolute Error (MAE) of Young's Modulus on physics parameter estimation.
Method
Runtime (min.)
MAE of (
Ours
2
0.506
Diff. MPM (5 iters)
20
0,439
Diff. MPM (15 iters)
60
0.394
E
Societal Impacts
Positive Impacts Our method integrates physically grounded simulation signals into video generative models, offering new avenues for controllable and physically plausible video synthesis. These can support people from amateurs to filmmakers and designers in rapidly prototyping ideas with accurate physical behavior, democratizing access to high-fidelity visual tools.
Negatíve Impacts High-fidelity generative models, especially when conditioned physical signals, may misused for creating deceptive content such as realistic yet fabricated disaster footage or physically plausible fake videos. This poses risks for misinformation and erosion of public trust. Although our approach enhances physical plausibility, it is important note that the generated outputs are not real-world occurrences.
F
Data and Model Safeguards
Given the dual-use nature of video generation models, we recognize that our pretrained model could be misused to generate deceptive, physically plausible videos for misinformation. As such, will implement appropriate safeguards to support controlled access when we release our model, including: (1) requiring users agree to usage guidelines and restrictions, (2) distributing the model under research-only license, (3) investigating automatic safety filters that can flag potentially harmful uses. These steps aim reduce the risk of malicious unintended applications while still supporting reproducible research.
Our training data consists exclusively of synthetic point cloud trajectories representing object motion under simulated physics. These datasets contain no images, videos, or human-related content, and thus should pose risk of visual misinformation, privacy violations, unsafe content. All point clouds are generated in simulation environments and contain only geometric and physical information about object movement.
18

---

"A saft striped blorkket liee plat on the ground then slowly lifts upwards if an spoard force dragging the middlee"
orange starfich rotating cleckeiss fowards the ziht creating smooth natural nation"
chyccr2.59 DrazAnythngg
Wonz1
Copvidook
Ours
"A fighttr jet flies at high speed the sky sircraft"
pair wireles headphanes rests
upsard lifting the
kefore lifting into the ais there insisible force to its bandls"
Drapkoythire
objore.ss 3D
Wonz1
CogVideot
Oura
"the penguin is fully litted upodtds and flent into the air with a netural metion, as if there a force applied onto its left
"the giraffe is dragged upwards fowand left and flasting inte the aix with natural mation, there apper force."
Ob/CH2.50 5D Drapinycting
Winz1
CogMddok
Ours
Figure 9: More qualitative comparison between our method and baselines.
19

---

NeurIPS Paper Checklist
The checklist is designed to encourage best practices for responsible machine learning research, addressing issues of reproducibility, transparency, research ethics, and societal impact. Do not remove the checklist: The papers not including the checklist will be desk rejected. The checklist should follow the references and follow the (optional) supplemental material. The checklist does NOT count towards the page limit.
Please read the checklist guidelines carefully for information on how to answer these questions. For each question in the checklist:
You should answer [Yes], [No], or [NA].
[NA] means either that the question is Not Applicable for that particular paper or the relevant information is Not Available.
Please provide short (1-2 sentence) justifiicton right after your answer (even for NA).
The checklist answers are an integral part of your paper submission. They are visible to the reviewers, area chairs, senior area chairs, and ethics reviewers. You will be asked to also include (after eventual revisions) with the final version of your paper, and its final version will be published with the paper.
The reviewers of your paper will be asked to use the checklist as one of the factors in their evaluation.
While "[Yes] " is generally preferable to "[No] ", it is perfectly acceptable to answer "[No] provided proper justification is given (e.g., "error bars are not reported because it would be too computationally expensive" or "we were unable to find the license for the dataset we used"). In general, answering "[No] or "[NA] is not grounds for rejection. While the questions are phrased in binary way, we acknowledge that the true answer often more nuanced, so please just use your best judgment and write justification to elaborate. All supporting evidence can appear either in the main paper or the supplemental material, provided in appendix. If you answer [Yes] to a question, in the justification please point to the section(s) where related material for the question can be found.
IMPORTANT, please:
Delete this instruction block, but keep the section heading "NeurIPS Paper Checklist",
Keep the checklist subsection headings, questions/answers and guidelines below.
Do not modify the questions and only use the provided macros for your answers.
1. Claims
Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?
Question:
Answer: [Yes]
Justification: Our claims are validated by our quantitative and qualitative results in the experimental results section.
Guidelines:
The answer NA means that the abstract and introduction do not include the claims made the paper.
The abstract and/or introduction should clearly state the claims made, including the contributions made the paper and important assumptions and limitations. A No or
NA answer to this question will not be perceived well by the reviewers.
The claims made should match theoretical and experimental results, and reflect how much the results can be expected to generalize to other settings.
It is fine to include aspirational goals as motivation as long as is clear that these goals are not attained by the paper.
2. Limitations
Question: Does the paper discuss the limitations of the work performed by the authors? Answer: [Yes]
20

---

Justification:
We provided our limitations in our last section.
Guidelines:
The answer NA means that the paper has no limitation while the answer No means that the paper has limitations, but those are not discussed in the paper.
The authors are encouraged to create separate "Limitations" section in their paper.
The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (e.g., independence assumptions, noiseless settings, model well-specification. asymptotic approxinations only holding locally). The authors should reflect how these assumptions might be violated in practice and what the implications would be.
The authors should reflect on the scope of the claims made, e.g., if the approach was only tested on a few datasets or with few runs. In general, empirical results often depend on implicit assumptions, which should be articulated.
The authors should reflect the factors that influence the performance of the approach. For example, facial recognition algorithm may perform poorly when image resolution low images are taken in low lighting. Or speech-to-text system might be used reliably to provide closed captions for online lectures because it fails to handle technical jargon.
The authors should discuss the computational efficiency the proposed algorithms and how they scale with dataset size.
If applicable, the authors should discuss possible limitations of their approach to address problems of privacy and fairness.
While the authors might fear that complete honesty about limitations might be used by reviewers as grounds for rejection, a worse outcome might be that reviewers discover limitations that aren't 't acknowledged in the paper. The authors should use their best judgment and recognize that individual actions in favor of transparency play an impor- tant role in developing norms that preserve the integrity of the community. Reviewers will be specifically instructed to not penalize honesty concerning limitations.
3. Theory assumptions and proof'
Question: For each theoretical result, does the paper provide the full set of assumptions and complete (and correct) proof?
Answer:
[NA]
Justification: Our paper does not include theoretical results.
Guidelines:
The answer NA means that the paper does not include theoretical results.
All the theorems, formulas, and proofs in the paper should be numbered and cross- referenced.
All assumptions should be clearly stated or referenced the statement of any theorems.
The proofs can either appear in the main paper or the supplemental material, but if they appear in the supplemental material, the authors are encouraged provide short proof sketch to provide intuition.
Inversely, any informal proof provided in the core of the paper should be complemented by formal proofs provided in appendix or supplemental material.
Theorems and Lemmas that the proof relies upon should be properly referenced.
4. Experimental result reproducibility
Question: Does the paper fully disclose all the information needed to reproduce the main ex- perimental results of the paper to the extent that affects the main claims and/or conclusions of the paper (regardless whether the code and data are provided or not)?
Answer: [Yes]
Justification: We include the necessary details in the supplemental and will release the training code and checkpoints for reproducibility.
Guidelines:
21

---

The answer NA means that the paper does not include experiments.
If the paper includes experiments, a No answer to this question will not be perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data are provided or not.
If the contribution is a dataset and/or model, the authors should describe the steps taken to make their results reproducible or verifiable.
Depending on the contribution, reproducibility can be accomplished in various ways. For example, if the contribution is novel architecture, describing the architecture fully might suffice, or if the contribution is a specific model and empirical evaluation, it may be necessary to either make it possible for others to replicate the model with the same dataset, or provide access to the model. In general. releasing code and data is often one good way accomplish this, but reproducibility instructions for how to replicate the results, access to a hosted model (e.g.. in the case of large language model), releasing of a model checkpoint, other means that appropriate to the research performed.
also be provided via detailed
While NeurlPS does not require releasing code, the conference does require all submis- sions to provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution. For example
(a) the contribution is primarily a new algorithm, the paper should make clear how to reproduce that algorithm. (b) If the contribution is primarily new model architecture, the paper should describe the architecture clearly and fully.
(c) If the contribution is a new model (e.g., a large language model), then there should either be way to access this model for reproducing the results or way to reproduce the model (e.g., with an open-source dataset instructions for how to construct the dataset).
(d) We recognize that reproducibility may be tricky in some cases, in which case authors are welcome deseribe the particular way they provide for reproducibility. In the case of closed-source models, it may be that access to the model is limited in some way (e.g., to registered users), but it should be possible for other researchers to have some path to reproducing or verifying the results.
5. Open access to data and code
Question: Does the paper provide open access the data and code, with sufficient instruc- tions to faithfully reproduce the main experimental results, as described in supplemental material?
Answer: [No]
Justification: We don't provide the code during submission. We will open-source the code, model and checkpoints after acceptance.
Guidelines:
The answer NA means that paper does not include experiments requiring code.
Please see the NeurIPS code and data submission guidelines (https://nips. cc/
public/guide wwwlcc//ggi/eseee /guides/CoeeSummii SwwoodeeSumisiiioooooooo
for more details.
While we encourage the release of code and data, we understand that this might not be possible, "No" an acceptable answer. Papers cannot be rejected simply for not including code, unless this is central to the contribution (e.g., for new open-source benchmark).
The instructions should contain the exact command and environment needed to run to reproduce the results. See the NeurlPS code and data submission guidelines (https: guigguidssssse (CodeSubmiii wwt.SSuumsssiinn.... aasionoooccy) for more details.
/ /nips.cc/publli
The authors should provide instructions on data access and preparation, including how to access the raw data, preprocessed data, intermediate data, and generated data, etc.
The authors should provide scripts to reproduce all experimental results for the new proposed method and baselines. only subset of experiments are reproducible, they should state which ones are omitted from the script and why.
22

---

At submission time, to preserve anonymity, the authors should release anonymized versions (if applicable).
Providing as much information as possible in supplemental material (appended to the paper) recommended, but including URLs to data and code is permitted.
6. Experimental setting/details
Question: Does the paper specify all the training and test details (e.g., data splits, hyper- parameters, how they were chosen, type of optimizer, etc.) necessary to understand the results?
Answer: [Yes]
Justification: We provide the creation of datasets, the training, and testing details in the paper and supplemental.
Guidelines:
The answer NA means that the paper does include experiments.
The experimental setting should be presented the core the paper to level of detail that is necessary to appreciate the results and make sense of them.
The full details can be provided either with the code, in appendix, supplemental material.
7. Experiment statistical significance
Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?
Question:
Answer: [No]
Justification: Our experiments are computationally expensive (8x Nvidia L40 for nearly two days per experiment) to be run multiple times for error bars.
Guidelines:
The answer NA means that the paper does include experiments.
The authors should answer "Yes" if the results are accompanied by error bars, confi- dence intervals, statistical significance tests, least for the experiments that support the main claims of the paper.
The factors of variability that the error bars are capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, overall
run with given experimental conditions).
The method for calculating the error bars should be explained (closed form formula, call to a library function, bootstrap, etc.)
The assumptions made should be given (e.g., Normally distributed errors).
It should be clear whether the error bar is the standard deviation the standard error of the mean.
is OK to report 1-sigma error bars, but one should state it. The authors should preferably report a 2-sigma error bar than state that they have 96% CI, if the hypothesis of Normality of errors is not verified.
For asymmetric distributions, the authors should be careful not to show in tables or figures symmetric error bars that would yield results that are out range (e.g. negative error rates).
If error bars reported in tables plots, The authors should explain in the text how they were calculated and reference the corresponding figures or tables in the text.
8. Experiments compute resources
For each experiment, does the paper provide sufficient information on the com- puter resources (type of compute workers, memory, time of execution) needed to reproduce the experiments?
Question:
Answer: [Yes]
We report the number of GPUs and running time for each experiment in the supplemental.
Justification:
23

---

Guidelines:
The answer NA means that the paper does not include experiments.
The paper should indicate the type of compute workers CPU or GPU, internal cluster, or cloud provider, including relevant memory and storage.
The paper should provide the amount compute required for each of the individual experimental rums well estimate the total compute.
The paper should disclose whether the full research project required more compute than the experiments reported in the paper (e.g., preliminary or failed experiments that didn't make into the paper).
9. Code of ethies
Question: Does the research conducted in the paper conform, in every respect, with the NeurIPS Code of Ethics https: / /ttp///eerr//////ee cc cc/public/ httpppuul///hiiiie c/Ethic/tiicec csQuidelinnene htt.li/c/ehccSuieeelniineee ines?
Answer: [Yes]
Justification: Our paper conforms the NeurIPS code of ethics in every respect.
Guidelines:
The answer NA means that the authors have not reviewed the NeurIPS Code of Ethics.
If the authors answer No, they should explain the special cireumstances that require deviation from the Code of Ethics.
The authors should make sure to preserve anonymity (e.g.. if there is a special consid- eration due to laws or regulations their jurisdiction).
10. Broader impacts
Question: Does the paper discuss both potential positive socictal impacts and negative societal impacts of the work performed?
Answer: [Yes]
Justification:
We discussed the impacts in the supplementary material.
Guidelines:
The answer NA means that there is no societal impact of the work performed.
If the authors answer NA or No, they should explain why their work has societal impact or why the paper does not address societal impact.
Examples of negative societal impacts include potential malicious unintended uses (c.g., disinformation, generating fake profiles, surveillance), fairness considerations (e.g., deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations.
The conference expects that many papers will be foundational research and not tied to particular applications, let alone deployments. However, if there is direct path to any negative applications, the authors should point it out. For example, it is legitimate to point that improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point out that a generic algorithm for optimizing neural networks could enable people train models that generate Deepfakes faster.
The authors should consider possible harms that could arise when the technology is being used as intended and functioning correctly, harms that could arise when the technology is being used as intended but gives incorrect results, and harms following from (intentional or unintentional) misuse of the technology.
If there are negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition attacks, mechanisms for monitoring misuse, mechanisms to monitor how a system learns from feedback over time, improving the efficiency and accessibility of ML).
11. Safeguards
Question: Does the paper describe safeguards that have been put in place for responsible release of data models that have a high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)?
24

---

Answer: [Yes]
Justification:
We provide the safeguards in the supplemental.
Guidelines:
The answer NA means that the paper poses no such risks.
Released models that have a high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example requiring that users adhere to usage guidelines or restrictions to access the model or implementing safety filters.
Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images.
We recognize that providing effective safeguards is challenging. and many papers do not require this, but we encourage authors to take this into account and make best faith effort.
12. Licenses for existing assets
Are the creators or original owners of assets (e.g.. code, data, models), used in the paper. properly credited and are the license and terms of use explicitly mentioned and properly respected?
Question:
Answer: [Yes]
Justification:
We cite the original papers when using their code, data or models.
Guidelines:
The answer NA means that the paper does use existing assets.
The authors should cite the original paper that produced the code package or dataset.
The authors should state which version of the asset used and, possible, include URL.
The name of the license (e.g., CC-BY 4.0) should be included for each asset.
For scraped data from a particular source (e.g.. website), the copyright and terms of service of that source should be provided.
assets are released, the license, copyright information, and terms of use in the package should be provided. For popular datasets, paperswi thcode com datasets has curated licenses for some datasets. Their licensing guide can help determine the license of dataset.
For existing datasets that re-packaged, both the original license and the license of the derived asset (if it has changed) should be provided.
If this information is not available online, the authors are encouraged to reach out to the asset's creators.
13. New assets
Question: Are new assets introduced in the paper well documented and is the documentation provided alongside the assets?
Answer: [No]
Justification: We don't release the datasets in the submission. We will provide all the code for reproducing the dataset and the dataset itself after acceptance.
Guidelines:
The answer NA means that the paper does not release new assets.
Researchers should communicate the details the dataset/code/model as part of their
submissions via structured templates. This includes details about training, license, limitations, etc.
The paper should discuss whether and how consent was obtained from people whose asset used.
At submission time, remember to anonymize your assets (if applicable). You can either create anonymized URL or include anonymized zip file.
14. Crowdsourcing and research with human subjects
25

---

Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given to participants and screenshots, if applicable, as well as details about compensation (if any)?
Answer: [NA]
Justification: Our paper does not involve crowdsourcing nor research with human subjects.
Guidelines:
The answer NA means that the paper does not involve crowdsourcing research with human subjects.
Including this information in the supplemental material is fine, but if the main contribu- tion of the paper involves human subjects, then as much detail as possible should be included in the main paper.
According to the NeurIPS Code of Ethies, workers involved in data collection, curation, or other labor should be paid at least the minimum wage in the country of the data collector.
15. Institutional review board (IRB) approvals or equivalent for research with human subjects
Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or equivalent approval/review based the requirements of your country or institution) were obtained?
Question:
Answer: [NA]
Justification: Our paper does not involve crowdsourcing nor research with human subjects.
Guidelines:
The answer NA means that the paper does not involve crowdsourcing research with human subjects.
Depending on the country in which research is conducted, IRB approval (or equivalent)
may be required for any human subjects research. If you obtained IRB approval, you should clearly state this in the paper.
We recognize that the procedures for this may vary significantly between institutions and locations, and we expect authors to adhere to the NeurIPS Code of Ethics and the guidelines for their institution.
For initial submissions, do not include any information that would break anonymity (if applicable), such as the institution conducting the review.
16. Declaration of LLM usage
Question: Does the paper describe the usage of LLMs if it is an important, original, or non-standard component of the core methods in this research? Note that if the LLM is used only for writing, editing, formatting purposes and does not impact the core methodology, scientific rigorousness, or originality of the research, declaration is not required.
Answer: [Yes]
Justification: We use the reasoning ability of LLM to assess the quality of our video generation and described it in the paper.
Guidelines:
The answer NA means that the core method development in this research does not involve LLMs any important, original, non-standard components.
Please refer to our L.LM policy (https : /ttpas:///nuuiiie htt///eerrpp.//// het.e//aeeeeeeee///0000000 for what should or should not be described.
26


Reviews:
[{"review_id": "9Yt5FGzEJC", "reviewer": "Reviewer_vXqE", "summary": "PhysCtrl is a physics-grounded framework for controllable video generation, addressing existing models' lack of physical plausibility and 3D control. It features a diffusion-based generative physics network that learns physical dynamics for four materials, represented as 3D point trajectories. These trajectories act as control signals for video models, enabling explicit control over physics parameters and forces. Key contributions include its scalable framework, a diffusion model with spatiotemporal attention and physics-based constraints, and a large 550K synthetic dataset.", "questions": "1. How do the high costs and instabilities of physics simulators (for training data generation) impact the learned model's generalization or physical fidelity?\n2. Does the fixed 2048-point representation limit expressive power or fidelity when handling complex geometries or fine deformations?\n3. What are the specific limitations of the single-image 3D conversion step, and how do initial reconstruction errors affect subsequent physics simulation and video generation?", "limitations": "yes", "rating": 3, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 2, "significance": 2, "originality": 3, "strengths_and_weaknesses": "PhysCtrl offers significant advantages for physics-grounded, controllable video generation. Its core is a diffusion-based generative physics network that learns 3D point trajectories for four material types. This approach circumvents the high computational cost and instabilities of classical physics simulators for inference, making it efficient. The framework incorporates a spatiotemporal attention mechanism and physics-based constraints, and is trained on a large 550K synthetic dataset. Experiments show it outperforms existing methods in visual quality and physical plausibility.\nHowever, PhysCtrl has limitations. It currently focuses on single-object dynamics and only four material types, unable to model complex phenomena like multi-object coupling or intricate boundary conditions. While its inference is efficient, the training data relies on computationally expensive and potentially unstable physics simulators. Moreover, its experimental setup is highly resource-intensive (e.g., 8x Nvidia L40 for nearly two days per experiment), which prevented reporting error bars.", "was_revised": true, "final_justification": "I have reviewed the authors’ rebuttal and will keep my score unchanged.", "rebuttal": "Thank you for your insightful comments, which are crucial to improving the quality of our work. The following are detailed responses to your concerns.\n\n> Material types and modelling complex phenomena\n\nOur four material types (elastic, sand, rigid, plasticine) already cover a wide range of objects. **As acknowledged in the limitations section, modeling multi-object coupling or complex boundary conditions is beyond the scope of this work**. To the best of our knowledge, no existing neural network approach currently addresses this challenge effectively. However, since it was requested and to assess our framework’s potential in this setting, we conducted a preliminary experiment, where we created a small dataset that has an object dragged towards a cube and colliding with a cube from different angles and distances. We trained our model on it and achieved 93.70% vIOU, demonstrating the potential of our framework for multiple objects. Nevertheless, fully addressing this problem is nontrivial and future work could explore incorporating stronger physics-informed losses to better handle complex deformations, as well as developing scalable and modular architectures that can flexibly support different numbers of interacting objects.\n\n> Computational efficiency\n\nCompared to current AI models, the experimental setup of our trajectory generation model is not that resource-intensive. Our base/large model only needs 8L40 48G GPUs for training 30/80 hours, which is accessible in most academic settings. Our trajectory generation model also require less resources than our baselines for image-to-video generation: DragAnything needs A100s (training time not available) to directly finetune a much heavier video model (SVD); Wan2.1 and CogVideoX require at least hundreds of GPUs for training.\n\n> How do the high costs and instabilities of physics simulators (for training data generation) impact the learned model's generalization or physical fidelity?\n\nThe high costs and instabilities are minor issues and have little effect on our model’s generalization and physical fidelity. Although the simulator runs slower than our model, we can utilize multiple low-end GPUs (e.g. 2080Ti) and generate data efficiently. The instability is mostly due to numerical issues, we filter them out by checking out of boundary values and NaN values. Our final dataset is high-quality, large-scale and contains diverse categories and geometries, which enables our trained model to be generalizable and stable. We will release the dataset.\n\n> Does the fixed 2048-point representation limit expressive power or fidelity when handling complex geometries or fine deformations?\n\nPrior work on 4D generation and reconstruction (PhysDreamer [1], SC-GS [2], MoSca [3]) has shown that **real-world motion can be effectively captured using sparse basis or control points**. Also, our choice of 2048 points for the foreground object is motivated by Table 3 in Diffusion as Shader (DaS), where the authors show that using 4900 points (including background static points) achieves similar results with more points (8100). From our observations, our method, trained on the diverse Objaverse dataset, performs reliably using 2048 sparse points, and we have not observed notable issues in fidelity or expressiveness. To further validate this, we tested using simulated trajectories of different numbers of points as the input for the video model. We follow our paper’s protocol to use GPT for evaluation and present the results in the table below. There is indeed minimal difference (GPT itself might also have some errors) when using different numbers of points to drive the video. \n\nWe agree that sparse points may not model extremely fine deformations. However, in such cases, the video model itself might also not be able to generate those delicate details. To the best of our knowledge, no existing methods for physics-grounded video generation are capable of handling such fine-grained deformations effectively.  We will discuss this challenge for all the existing methods (including our work) in the limitation and leave this for future work.\n\n|                                                | SA &uarr; | PC &uarr;  | VQ &uarr;  |\n|------------------------------------------------|------|------|------|\n| Ours (2048 points, 4900 tracks)                             | 4.50 | 4.50 | 4.33 |\n| Use Traditional simulator (2048 points, 4900 tracks)  | 4.33 | 4.33 | 4.41 |\n| Use Traditional simulator (8192 points, 19600 tracks) | 4.33 | 4.33 | 4.17 |\n\n[1] PhysDreamer: Physics-Based Interaction with 3D Objects via Video Generation, Zhang et al., ECCV 2024\n[2] SC-GS: Sparse-Controlled Gaussian Splatting for Editable Dynamic Scenes, Huang et al., CVPR 2024\n[3] MoSca: Dynamic Gaussian Fusion from Casual Videos via 4D Motion Scaffolds, Lei et al., CVPR 2025\n\n> Limitations of single image to 3D and its errors on subsequent steps\n\nThe single-image-to-3D step produces reasonable geometry overall and is very unlikely to yield severely distorted or implausible shapes. While minor artifacts such as surface noise may occur, they have minimal impact on subsequent steps. We achieve such robustness to geometric variations because we trained the trajectory generation network on the diverse Objaverse dataset and applied surface noise augmentation during data generation. Moreover, the predicted point trajectories are used as motion signals to guide the video model, and the video model will generate fine visual details with this guidance."}, {"review_id": "JKI7dZM8oc", "reviewer": "Reviewer_yGS5", "summary": "The paper introduces a method for an image-to-video generation with controllable physics. In a first step, the existing methods are used to lift the image to a 3D point-cloud. Given a force vector, a novel diffusion-based physics generation model estimates potential point trajectories. In the last step the generated trajectories a used to guide an existing image-to-video generation process.", "questions": "1. What are the advantages of a learned generative physics model over an actual simulator? \n2. What are common failure cases of the method?\n3. How did you settle on using $N=2048$ points for the point-cloud representation? This seems like a central design decision which is not further discussed in the paper. Especially for fine-grained sand simulation this seems like a surprisingly low value.\n\nI am open to increase my score should the authors be able to show the benefit of generative dynamics over traditional simulation for the purpose of video generation and provide some insights into common failure cases.", "limitations": "Yes", "rating": 5, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 3, "significance": 3, "originality": 3, "strengths_and_weaknesses": "**Strengths**\n\n1. **Qualitative results:** The shown qualitative results are of high quality and the generative object motions/deformation seem physically plausible\n2. **Supplementary videos:** The supplementary video is well made and summarises the core parts of the method and showcases some good examples. The additional demo video on how to use the method is a nice cherry on top.\n3. **Related work section:** The related works sections covers a wide range of different research directions while highlighting their shortcomings for the posed problem.\n4. **Clarity**: The paper is well written and gives a clear and easy to follow introduction into the method and the experimental evaluation. The included figures sufficiently support the understanding of the method.\n\n**Weaknesses**\n1. **Why generative physics for video generation? (Also Question 1):** Physics generation and video generation are separate steps in the pipeline and the model requires explicit specification of the physics parameters so there is no direct need for using a generative method for the physics. Since the generative physics model seems to be the core contribution of the paper, it would be relevant to see how the full pipeline would perform if the learned model was replaced with a standard simulator. Alternatively, the authors could provide a more in-depth discussion on the necessity of using a learned simulator. (Also see Question 1)\n2. **Failure cases:** The paper does not show any failure cases which would be relevant to give an actual impression of the models limitations. For example it would be relevant to see where the generative physics network breaks physical plausibility or controllability.\n\n**Misc**\n- Tab.1 seemingly contains a faulty looking value for DragAnything", "was_revised": true, "final_justification": "After the rebuttal I decided to raise my score from \"Borderline Accept\" to \"Accept\" and to raise my confidence to a \"confident\". Especially the answers regarding the advantages of a learned dynamics model over traditional simulators are convincing and defused my main concern regarding the paper (W1/Q1). Furthermore, all my questions have been sufficiently answerd (Q1-Q3).", "rebuttal": "Thanks for your insightful questions, which is crucial for improving the quality of our work. The following are detailed responses to your concerns.\n\n> Advantages of a learned generative physics model over an actual simulator.\n\nUsing neural simulator has several benefits: (1) **More robust**: we use a traditional simulator to generate training data by tuning their parameters and filtering failure cases. The filtered training data helps us train a robust neural simulator, even for cases where classical simulators struggle. For example, we found that when using MPM to simulate a waterdrop falling onto the ground from different heights, there are around 10% failure cases, mostly due to numerical instabilities, while a trained neural network is always stable; (2) **Easy-to-use**: our model requires minimal input from user, including only force direction and material. In contrast, MPM requires many other hyperparameters, including grid size, damping scale for grid velocity, frame dt and substep dt etc; (3) **Unifying different simulators and generalization to different materials**: we unified four materials in one unified network by creating data from both MPM and a rigid body simulator. If using an actual simulator, users would need to switch simulators for different materials. Moreover, due to our unified point representation, users do not need meshes to simulate rigid objects, while standard rigid body simulator requires mesh as input; (4) **Inference speed**: our base model and large model costs 1s and 3s for forward diffusion sampling, respectively, while MPM requires more than 4s. For the backward step, MPM requires 3min, while our method only needs 0.1s (no diffusion). Fast backward speed is especially beneficial for inverse problems.\n\n> Results of using a traditional simulator.\n\nWe use the point trajectories from traditional simulators to generate videos on our testing set and use GPT for evaluation, the results are shown in the table below. Using our model is even slightly better than using the traditional simulator in certain metrics. The visual results mostly look similar. We will update these results in the paper. As such, using our model not only achieves great video generation results on successful cases of MPM, but also has the 4 benefits mentioned above.\n\n|                                                | SA &uarr; | PC &uarr;  | VQ &uarr;  |\n|------------------------------------------------|------|------|------|\n| Ours (2048 points, 4900 tracks)                             | 4.50 | 4.50 | 4.33 |\n| Use Traditional simulator (2048 points, 4900 tracks)  | 4.33 | 4.33 | 4.41 |\n| Use Traditional simulator (8192 points, 19600 tracks) | 4.33 | 4.33 | 4.17 |\n\n> Common failure cases of the method.\n\n(1) **Failure cases of our diffusion model**: we compared our results with MPM simulation on the Objaverse testing set. We found that cases with low vIoU typically involve thin structures or scenarios where our model predicts global motion but fails to accurately capture internal deformations. (2) **Failure cases due to video model**: we found in some cases, especially when the trajectories might conflict with the prior of video model, DaS cannot fully follow our trajectories. For example, when the user input for the object material is very stiff, but it appears soft according to RGB information. Then the resulting video might still be stiff. The video model might also hallucinate some unexpected content in the occluded region. For example, for an animal, it might generate five legs. (3) Failure cases due to image-to-3D is very rare.\n\nWe will add these discussions and visualization of failure cases in the revision.\n\n> How did you settle on using N=2048 points for the point-cloud representation?\n\nWe choose N=2048 considering that: (1) Prior works on 4D generation and reconstruction (PhysDreamer [1], SC-GS [2], MoSCA [3]) demonstrated that **real-world motions can be represented with a sparse number of basis or control points;** (2) The point trajectory is only providing a physics deformation signal for the video model and details will be generated by the video model itself. Our choice of 2048 points for the foreground object is further motivated by Table 3 in Diffusion as Shader (DaS), where the authors show that using 4900 points (including background static points) achieves similar results with more points (8100). In the Table above, we tested on simulated trajectories of more points as the input for the video model, results also show that using more points didn’t bring significant difference in performance.\n\n[1] PhysDreamer: Physics-Based Interaction with 3D Objects via Video Generation, Zhang et al., ECCV 2024\n[2] SC-GS: Sparse-Controlled Gaussian Splatting for Editable Dynamic Scenes, Huang et al., CVPR 2024\n[3] MoSca: Dynamic Gaussian Fusion from Casual Videos via 4D Motion Scaffolds, Lei et al., CVPR 2025\n\n> Misc: Tab.1 seemingly contains a faulty looking value for DragAnything\n\nThanks for pointing this out. The number should be 2.8."}, {"review_id": "C8ZkeuIBFa", "reviewer": "Reviewer_F6p4", "summary": "This paper presents PhysCtrl, a framework for generating physics-grounded videos from a single image with explicit control over material properties and external forces. The pipeline first segments the target object, reconstructs its 3D point cloud using multi-view synthesis, and then predicts its motion using a learned trajectory generation model trained on simulated dynamics from the Material Point Method (MPM). These trajectories are projected and used to guide a pretrained video diffusion model to generate physically plausible videos", "questions": "As similar pipelines have been proposed in prior work, where the object is first lifted to 3D, then simulated, and finally rendered, the overall novelty of the proposed framework may be limited. Furthermore, the rendering module directly adopts an existing video diffusion model without specific adaptations or modifications. Therefore, the main contribution appears to lie in the trajectory generation module. \n\nHowever, I would like the authors to clarify the necessity of training a neural trajectory predictor instead of directly using MPM, which should yield similar results for these relatively simple object animation cases. As mentioned above, users still need to manually specify inputs such as boundary conditions and external forces, which raises questions about the practical advantages of the proposed model over traditional simulation. \n\nAdditionally, since the synthetic dataset seems to include only single-object dynamics, it is unclear whether the method would generalize to multi-object interactions with collisions.\n\nWhile I understand it’s not possible to compare against all baseline methods, it would be nice to include a comparison with this SOTA method [1], which also addresses motion-controllable video generation.\n\n[1] Burgert R, Xu Y, Xian W, et al. Go-with-the-flow: Motion-controllable video diffusion models using real-time warped noise. CVPR 2025", "limitations": "yes", "rating": 4, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 3, "significance": 2, "originality": 2, "strengths_and_weaknesses": "Strength:\n\n1. The paper is well written and easy to follow\n2. The proposed pipeline is technically sound and comprehensive\n3. Expreiments are thorough and a demo video is provided to demonstrate the usage of of the proposed method.\n4. The results of the proposed method look impressive, showing significant improvement over the baseline methods\n\nWeakness:\n\n1. The necessity of learning a trajectory generation model is not fully justified. Since the model takes inputs similar to MPM (e.g., material properties and external forces), it's unclear what advantage it offers over directly using simulation, especially for small-scale scenes where MPM is efficient\n2. It is unclear how the model accounts for external elements like the ground plane if only the foreground object is segmented. For example, in Figure 1, how does the model determine that the penguin is supposed to collide with the table while the plane should stay in the air when rotating?\n3. The approach seems limited to single-object dynamics. No multi-object interactions or collisions are demonstrated, and it appears the dataset does not include such scenarios.\n\nMinor: \n\nSome recent relevant papers could also be cited for completeness:\n\n[1] Burgert R, Xu Y, Xian W, et al. Go-with-the-flow: Motion-controllable video diffusion models using real-time warped noise. CVPR 2025\n\n[2] Xie T, Zhao Y, Jiang Y, et al. PhysAnimator: Physics-Guided Generative Cartoon Animation. CVPR 2025", "was_revised": true, "final_justification": "After reading the rebuttal, I am raising my score to 4. I acknowledge the advantages of the neural simulator over traditional simulators in single-object dynamics, particularly in terms of speed and robustness.", "rebuttal": "Thank you for your insightful comments and questions, which is crucial to improving the quality of our work. The following are detailed responses to your concerns.\n \n> The model takes inputs similar to MPM and the necessity of training a neural trajectory predictor is not clear.\n\nWe would like to first clarify that **our user inputs are less than MPM**. We only require minimal information: force and material information, while MPM requires many other hyperparameters: grid size, damping scale for grid velocity, frame dt and substep dt etc. Besides this, using our model has several other benefits: (1) **More robust**: we use a traditional simulator to generate training data by tuning their parameters and filtering failure cases. The filtered training data helps us train a robust neural simulator, even for cases where classical simulators struggle. For example, we found that when using MPM to simulate a waterdrop falling onto the ground from different heights, there are around 10% failure cases (due to numerical instabilities), while a trained neural network is more stable; (2) **Unify different simulators and generalization to different materials**: we unified four materials in one unified network by creating data from both MPM and a rigid body simulator. If using an actual simulator, users would need to switch simulators for different materials. Moreover, due to our unified point representation, users do not need meshes to simulate rigid objects, while **standard rigid body simulator requires mesh as input**; (3) **Inference speed**: our base model and large model costs 1s and 3s for forward diffusion sampling, while MPM requires more than 4s. For one backward step, MPM requires 3min, while our method only needs 0.1s (no diffusion).\n\n> It is unclear how the model accounts for external elements like the ground plane if only the foreground object is segmented. For example, in Figure 1, how does the model determine that the penguin is supposed to collide with the table while the plane should stay in the air when rotating?\n\n**Our model takes floor condition as an additional condition (Line 172 of the paper) and the floor condition is calculated automatically from the input**. The model will automatically determine whether the points will collide with the boundary because the generated point trajectory is conditioned on the boundary. To calculate the floor condition, we use grounded segmentation or user input to select the floor and use VGGT to obtain the 3D points of the whole input image (including the floor and object), as our image-to-3D only reconstructs the foreground object. Then we align the coordinate systems of VGGT and our  image-to-3D using the correspondences of the foreground object. Finally, we obtain the value of the floor using principal component analysis based on the transformed ground plane points. If there is no floor detected, the value will be set to a placeholder. \n\nIn Figure 1, we would like to clarify that the penguin won’t collide with the table because of the upward force. The plane case doesn’t contain a floor.\n\n> It is unclear whether the method would generalize to multi-object interactions.\n\nAs acknowledged in the limitations section, modeling multi-object coupling or complex boundary conditions is beyond the scope of this work. To the best of our knowledge, no existing neural network approach currently addresses this challenge effectively. We did a preliminary experiment by creating a small dataset that has an object dragged towards a cube and colliding with the cube from different angles and distances. We trained our model on it and achieved 93.70% vIOU on the held out testing set. This preliminary experiment demonstrates the potential of our framework for multiple objects. Nevertheless, fully addressing this problem remains nontrivial and future work could explore incorporating stronger physics-informed losses to better handle complex deformations, as well as developing scalable and modular architectures that can flexibly support different numbers of interacting objects.\n\n> Comparison with Go-with-the-flow\n\nWe run Go-with-the-Flow using the same test set and test protocol as in our paper. Following the official Github Repo, we use the cut-and-drag tool for motion control and then perform video generation. We observed that it supports only limited motion control, restricted to translation, rotation, and scaling of the original object, which hinders its ability to model complex deformations in elastic objects or morphological changes in materials like sand. Furthermore, it lacks support for 3D motion, such as the inward rotation in our UFO test case. These constraints result in Go-with-the-Flow performing worse than our method:\n\n|                                                | SA &uarr; | PC &uarr;  | VQ &uarr;  |\n|--------------------------------------------|------|------|------|\n| Ours                                        | 4.50 | 4.50 | 4.33 |\n| Go-with-the-flow                      | 3.58 |3.66| 3.58 |\n\n> Relevant papers.\n\nThanks for pointing them out. We will add them in the revision."}, {"review_id": "mPkxGJTOBN", "reviewer": "Reviewer_eLwH", "summary": "This paper proposes a way to do physics-controllable i2v generation. As I understand it, the main contribution of this paper is a novel way of generating point trajectories for objects in response to user-provided physics parameters and applied forces. They propose a diffusion model (which they call a \"generative physics network\") which inputs physics conditions and a point cloud, and outputs the position of each point in future timesteps. I.e. they propose an auxiliary model for \"approximate\" physics simulation, as a representation bottleneck. They train this model using a synthetic dataset of size 550k generated by physics simulators. Finally, they feed this point cloud into some else's pretrained video diffusion model (Diffusion as Shader) which accepts pixel trajectories as conditioning. The end-to-end result is a (force+image+text)2video generation model.", "questions": "1. Can you please explain exactly how you used the Diffusion as Shader baseline?\n2. The description in lines 49-59 of the diffusion model (\"... it first aggregates spatial influences from neighboring points...\") sounds very related to graph neural networks, did you explore that connection at all?\n3. It seems to me like the physics loss is the most complicated part of the paper (there's half a page of math there) and it seems the least important to the downstream performance? In Table 3, the row without physics loss is barely worse than the other rows, whereas the spatial attention and temporal attention are clearly crucial. Is this true? If this is the case, I would strongly recommend moving the Material Point Method section in the paper to the appendix, so the reader can black box the loss function, and move brief summaries of the experiment details from the appendix (e.g. construction of synthetic dataset, design of human study) into the body of the paper. \n4. What were some failure cases of the model? And were the failure cases more because an initial poor 3D point cloud was extracted? or because the diffusion model wasn't able to model it well? or because, given point cloud trajectories, the Diffusion as Shader wasn't able to accept them as conditioning in a good way?\n5. More generally, were certain classes of demos more or less likely to succeed and fail? E.g. Diffusion as Shader is built on top of CogVideoX... did the video prior make certain classes of demos more or less likely to succeed? Or more generally, what is the significance of the choice of the video prior for the base video model? \n6. In line 262, should the sentence read: \"... cannot produce motions that fully reflect *physical conditioning*\"? Or am I misunderstanding the claim you're trying to make in that sentence?", "limitations": "(Yes)", "rating": 5, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 4, "clarity": 3, "significance": 3, "originality": 3, "strengths_and_weaknesses": "Overall, I think this paper is excellent, I intend to advocate strongly for this paper to be accepted.\n\nStrengths:\n1. This paper approaches a very timely modeling problem (physics-conditioned video generation) with a novel and interesting technique that, notably, doesn't involve using any physics simulator at inference (although it does make use of 3D priors at inference, as a representational bottleneck). It makes good use of building on top of prior works while also proposing a new architecture (the force-conditioned point cloud diffusion model) which can be used modularly, for many different downstream tasks. \n2. I think that the choice of baselines is very effective. Two text-based (Wan 2.1, CogVideoX), and two single-point trajectory-conditioned (DragAnything, ObjCtlr-2.5D). Comparing to them clearly makes the point that PhysCtrl is superior.\n3. Likewise, the evaluation metrics for the diffusion model are very clear.\n4. I thought the writing and motivations were very clear, and the paper was a pleasure to read. I especially thought that the explanation of the Problem Setting (4.1.1) was explained very clearly. More generally, I think the organization of the paper is excellent (modulo a few potential suggestions for improvement, see below)\n\nWeaknesses:\n1. I thought some experimental details were swept under the rug, which made some of the takeaways unclear (see Questions section).\n2. Some of the phrasing was a tad vague and misleading. E.g. lines 60-61: \"we further show that the generated trajectories can drive pretrained video models for synthesizing physically plausible i2v generation\". By the time I got to the end of the paper I understand what was meant by \"drive\" (namely, plugging in the generated trajectories into someone else's trajectory-conditioned model) but I think this should be made more specific up front, or else it's not clear what is this paper's specific contribution. Additionally, I think that the claim on line 251 (\"we are the first method to inject physics prior into a video model\") is false/misleading; there are plenty of works that do physics-based video generation, including papers cited by the present work (e.g. Physdreamer). Is there perhaps a different way to word that phrase that more precisely describes the novelty proposed by this paper?\n3. (Minor weakness, more subjective) it would be nice if in the beginning of the Method section, when it discusses how the method works, and how the subsequent subsections correspond, for those to be in agreement/chronological order during training/inference (i.e. not going from 4.2 to 4.1 to 4.2. This just made it a tad harder for me to follow the logic on the first read)", "was_revised": true, "final_justification": "I already had a high opinion of the paper before the rebuttal period, due to the novelty + utility of the task (force-conditioned video generation), and the high quality of the output videos. I also thought that the experiment design was reasonable and strong. My questions to the authors were mainly about experiment design, and those have all been addressed.", "rebuttal": "Thanks for your insightful comments for improving our work. We appreciate your positive feedback and your constructive suggestions, which has been a source of great encouragement for us. The following are detailed responses to your concerns.\n\n> Some of the phrasing was a tad vague and misleading.\n\nWe will make it specific in the beginning that we use the generated trajectories as input of a trajectory-conditioned model.\n\n> The claim on line 251 (\"we are the first method to inject physics prior into a video model\") is false/misleading.\n\nOur model is conceptually different from PhysDreamer from a high level. **PhysDreamer exploits physics knowledge from pretrained video models**. It estimates physical properties of a trained 3DGS using the motion priors from video models with differentiable MPM. Differently, **our model is injecting physical priors into video models** by conditioning it on our generated motion trajectories. Moreover, we show in the supplemental that our model is also able to estimate physical properties given known motion trajectories. There are concurrent works, such as WonderPlay [1], Force prompting [2] (both appear after our submission), that attempt to improve the physics plausibility of video models. We will discuss these works and adjust our tones in the revision.\n\n[1] WonderPlay: Dynamic 3D Scene Generation from a Single Image and Actions, Arxiv, May 2025\n[2] Force Prompting: Video Generation Models Can Learn and Generalize Physics-based Control Signals, Arxiv, May 2025\n\n> Beginning of Method Subsection\n\nThanks for your advice. We will adjust the beginning of the method section to a more logical order.\n\n> How to use Diffusion as Shader.\n\nDiffusion as Shader (DaS) takes a **tracking video** as the input of the conditioning branch of ControlNet. Each frame of the tracking video is the projected 3D point trajectories of **2D anchor points (fixed grid positions for all videos)** from the first frame. Our model generates the 3D point trajectories of the object, so we need to convert it to the tracking video of DaS for inference. Specifically, for each 2D anchor point in the first frame, we associate it with the nearest 3D point in the object. Then, we can project the 3D point trajectories into 2D and generate the tracking video frame by frame. We will add these details to the next version.\n\n> Explore GNN.\n\nWe chose a Transformer-based architecture due to its strong representational capacity and general effectiveness across a range of tasks. While we did not exactly test a GNN, we experimented with a variant of the Transformer that has a similar spirit of GNN, that is, each point only attends to its k-nearest neighboring points to model local relationship, which is similar to the local aggregation behavior of GNNs. We found that this approach doesn’t work well, particularly has severe error accumulation. This supports our decision to apply attention across all points, which may allow the model to better capture long-range dependencies and complex interactions.\n\n> It seems to me like the physics loss is the most complicated part of the paper (there's half a page of math there) and it seems the least important to the downstream performance? In Table 3, the row without physics loss is barely worse than the other rows, whereas the spatial attention and temporal attention are clearly crucial. Is this true? If this is the case, I would strongly recommend moving the Material Point Method section in the paper to the appendix, so the reader can black box the loss function, and move brief summaries of the experiment details from the appendix (e.g. construction of synthetic dataset, design of human study) into the body of the paper.\n\nThe physical loss helps the model to generate more physics plausible trajectories. In Figure 7, we show four examples that using physical loss helps generate more physical plausible trajectories. For these cases, if there is no physics loss, generating videos with these trajectories will not look consistent with the input force. We appreciate your suggestion and will move some experiment details into the main paper.\n\n> Discussion of failure cases due to poor 3D point cloud extraction / our diffusion model / Diffusion as Shader\n\nClassification of failure cases: (1) **Failure due to initial poor image-to-3D reconstruction is very rare**. Single image-to-3D produces reasonable geometry overall and is unlikely to yield severely distorted or implausible shapes (note that we do not need appearance information). While minor artifacts (e.g. geometry not perfectly smooth or noise points on the surface) may occur, they have minimal impact on our results. We achieve such robustness to geometric variations because we trained the trajectory generation network on the diverse Objaverse dataset and applied surface noise augmentation during data generation. Moreover, the video model only needs point trajectories as motion signals and will generate the details. (2) **Failure cases of our diffusion model**: we compared our results with MPM simulation on the Objaverse testing set. We found that cases with low vIoU typically involve thin structures or scenarios where our model predicts global motion but fails to accurately capture internal deformations. (3) **Failure cases due to video model**: we found in some cases, especially when the trajectories might conflict with the prior of video model, DaS cannot fully follow our trajectories. For example, when the user input for the object material is very stiff, but it appears soft according to RGB information. Then the resulting video might still be stiff. The video model might also hallucinate some unexpected content in the occluded region. For example, for an animal, it might generate five legs.\n\nWe will add these discussions and visualization of failure cases in the revision.\n\n> Did the video prior make certain classes of demos more or less likely to succeed? Or more generally, what is the significance of the choice of the video prior for the base video model?\n\nAs discussed above, when the trajectories conflict with the prior of the video model, DaS might not fully follow our trajectories. We selected CogVideoX, as it was the strongest available model at the time of our submission. We think that using more powerful video models (especially those that have better world knowledge) will generally bring better results, e.g. generate more plausible details for occluded parts.\n\n> Line 264.\n\nThanks for pointing this out. We agree that using “physical conditioning” is more accurate."}]

Author rebuttal:

Meta-review: The submission received divergent ratings, including 1 Borderline reject, 1 Borderline accept, and 2 Accept. Three reviewers were actively engaged with the discussion and gave positive scores. Reviewer vXqE, who gave the only negative score, expressed concerns about the generalization. The authors and the AC tried to engage Reviewer vXqE in the discussion. Reviewer vXqE acknowledged that the rebuttal addressed the concerns well, but without changing the rating.

After checking the review, the rebuttal, the manuscript, and the discussion, the AC concurs with the other three reviewers' assessment and recommends acceptance. The work has a decent contribution to video generation and deserves sharing with the community.
Decision: Accept (poster)