NewtonGen: Physics-consistent and Controllable Text-to-Video Generation via Neural Newtonian Dynamics
Authors: Yu Yuan
Xijun Wang
Tharindu Wickremasinghe
Zeeshan Nadir
Bole Ma
Stanley H. Chan
Venue: ICLR 2026 Poster (2026)
Keywords: Generative Models
Video Generation
Primary area: generative models
TL;DR: NewtonGen leverages Neural Newtonian Dynamics to learn general latent dynamics, enabling physically aware and controllable Text-to-Video generation.

Abstract: A primary bottleneck in large-scale text-to-video generation today is physical consistency and controllability. Despite recent advances, state-of-the-art models often produce unrealistic motions, such as objects falling upward, or abrupt changes in velocity and direction. Moreover, these models lack precise parameter control, struggling to generate physically consistent dynamics under different initial conditions. We argue that this fundamental limitation stems from current models learning motion distributions solely from appearance, while lacking an understanding of the underlying dynamics. In this work, we propose NewtonGen, a framework that integrates data-driven synthesis with learnable physical principles. At its core lies trainable Neural Newtonian Dynamics (NND), which can model and predict a variety of Newtonian motions, thereby injecting latent dynamical constraints into the video generation process. By jointly leveraging data priors and dynamical guidance, NewtonGen enables physically consistent video synthesis with precise parameter control.  All data and code are available at https://github.com/pandayuanyu/NewtonGen.

NEWTONGEN: PHYSICS-CONSISTENT AND CONTROL- LABLE TEXT-TO-VIEE VIDEO GENERATION VIA NEURAL NEWTONIAN DYNAMICS
Yu Yuan Purdue University
Xijun Wang
Tharindu Wickremasinghe
Purdue University
Purdue University
Zeeshan Nadir Samsung Research America
Bole Ma
Stanley H. Chan
Purdue University
Purdue University
Newton Gen
Lincar
Kinematics
Kinematies
Deformation
Motioa
Size Clanging
Hybrid Motion
(a) Selected Dynamics Supported by NewtonGen
condineca penancecre (lecatio, spard)
condtoral pereeakm spard)
Divere
Concaar Metic
Parameter Controllability Offered NewtonGen
Figure 1: NewtonGen generates physically-consiseen videos from text prompts, with diverse dynamic perception (a), and precise parameter control (b).
ABSTRACT
A primary bottleneck in large-scale text-to-video generation today is physical consistency and controllability. Despite recent advances, state-of-the-arr mod- els often produce unrealistic motions, such as objects falling upward, or abrupt changes in velocity and direction. Moreover, these models lack precise parame- ter control, struggling generate physically consistent dynamics under different initial conditions. argue that this fundamental limitation stems from current models learning motion distributions solely from appearance, while lacking understanding of the underlying dynamics. this work, we propose Newton- Gen, framework that integrates data-driven synthesis with learnable physical principles. At its core lies trainable Neural Newtonian Dynamics (NND), which model and predict variety of Newtonian motions, thereby injecting latent dynamical constraints into the video generation process. By jointly leveraging data priors and dynamical guidance, NewtonGen enables physically consistent video synthesis with precise parameter control. All data and code are available at
INTRODUCTION
Since the breakthrough of probabilistic diffusion models in the early 2020's (See, e.g., Ho et al. (2020): [Song et al. (2021); Ramesh et al. (2021): Rombach et al. (2022)). foundational vision models

---

have created unprecedented opportunities for digital content generation. While contemporary video generators can synthesize visually appealing frames (Ho et al. 2022 OpenAl 2024b Hong et al. 2023
Peebles & Xie, 2023 Kong al. 2024 Yang et al. 20256). they struggle produce dynamic sequences that adhere to physically plausible motion. For instance, many videos generated by these methods violate basic physical laws such as objects falling upward, or abruptly changing velocity and
direction BBansal et al.)
2024 2025 Zhang et al. 2025a Li et al. 2025b; Duan et al. [0255] Motamed al. 2025 Gu al. 2025). The goal of this paper
provide a solution to these issues.
The failures in the above situations, according to some literature, can potentially be remedied
2020). However, recent researches such as Kang et al. (2025); Li et al.] (2025a); Chefer et al. (2025); Lin et al. (2025); Bansal et al. (2024) 2025): Yuan et al.
by scaling laws (Kaplan al.
(2025a) consistently point to deeper reason that current models only learn the distribution of visual
appearances. They lack an understanding of the underlying physical laws. Existing frameworks typically treat videos as spatio-temporal tokens and optimize the likelihood at the pixel level. During inference, the models mainly rely on memorization and imitation, making it difficult to generalize to out-of-distribution scenarios (Kang et al.) 2025). To bridge this gap, we argue that we need to explicitly incorporate physical laws into the learning process. This is not only crucial step for video generation, but also essential for connecting generative AI with the physical world.
In this paper, introduce NewtonGen, novel framework that integrates a data-driven, pre- pre-trained video generator with physics-informed, Neural Newtonian Dynamics (NND). In NND, we introduce neural ordinary differential equation (neural ODE) model to learn and predict the Newtonian motion from physics-clean data. By learning the dynamics of motion and manipulating its initial physical states, we can predict physics-consistent trajectories, orientations, and shapes. Subsequently, a motion-controlled video generator produces diverse and realistic videos by conditioning on both the predicted states and scene prompts. In summary, the contribution of this paper is twofold:
1. We propose NewtonGen, a physics-consisent and controllable text-to-video framework that explicitly incorporates dynamics into the generation process, allowing for interpretable, white-box control over generated motion.
2. We introduce Neural Newtonian Dynamics (NND), which models different dynamics via unified neural ordinary differential equations (ODEs). NND can efficiently learn the latent dynamics from a small amount of physics-clean data.
We conducted extensive experiments, showing that NewtonGen achieves physical consistency and controllability across various dynamics, as illustrated in Figure. [ and outperforms other baselines.
RELATED WORK
2.1
VIDEO GENERATION MODELS
The emergence of diffusion models (Ho al. 2020 Song et al. 2021) ) has greatly enhanced the
ability of generative models to produce visually realistic images (Ramesh et al. 2021 Rombach et al.,
2022 SD2). In video generation, models learn the distribution of real-world motion from large-scale
datasets (OpenAl 2024b Blattmann et al. 2023 Hong et al. 2023 Yang et al. 2025b; Kong et al.) 2024 NVIDIA 2025; Wan et al. 2025) The method DiT proposed by Peebles & Xie (2023)
introduces transformer architectures into diffusion models, further enhancing their scalability (Kaplan
al. 2020) for video generation tasks. Following this trend, representative video generation models
(e.g., Sora OpenAleeennn (2024b)) aim to leverage extensive video data to evolve toward general-purpose
world simulators.
However, current video generation models still lack an understanding of real-world physics. Studies show that increasing data or model size does not help them learn the physical rules behind video
content (Kang et al. 2025 Liu et al. 2025 Motamed et al. 2025 Lin et al. 2025). As a result, these models often produce videos that look realistic but contain physically incorrect dynamics when applied to out-oof-dstrrbuut
cases (Kang al. 2025 Bansal et al.) 2025 2024 Meng et al. 2025 Zhang et al., 2025al Li et al. 2025at Gu et al., 2025 Chefer et al. 2025). The main reason is that
they focus on appearance-leva motion rather than the underlying dynamics.
2

---

2.2 PHYSICS-AWARE GENERATION
To address the challenge of physical plausibility in video generation, recent research efforts incorpo- rate explicit physical priors into generative pipelines. Based on the stage and approaches of injecting physical knowledge, these methods can be broadly categorized into three types:
Generation then Physical Simulation. These methods first generate static 3D models or images conditioned on textual or visual inputs using generative models. Subsequently, physical simulation techniques such as Material Point Method (MPM) (Stomakhin et al., 2013) is applied to animate these static outputs into dynamic 3D scenes or videos (Lin et al. 2024 Xie et al., 20224] Tan et al. 2024 Zhang et al., 2024; Hsu et al., 2024). Although the post hoc physics-based rendering process explicit and controllable, it demands significantly more manual effort. These methods can be
summarized in the following Equation:
Video Generation
(G)((I
(I)
Physical Simulation
where P denotes the physical simulation, G denotes the video generator parameterized by network weight th. I is the input conditional prompt or image, V is the video we want.
Physical Simulation then Generation. Approaches in this category (Yuan et al. 2023 Liu et al. 2024 (avvatt Aira et al. 2024 Chen et al. 2025 Xie et al. 2025 Li 2025d) first apply physical simulation to conditionally specified images to generate pla plausible dynamic behaviors. The simulated dynamics are then utilized as conditional inputs for video generation models. For instance, PhysGen (Liu et al. 2024) segments dynamic objects from input images, simulates their motion according to Newtonian mechanics, and then refines the rendering by conditioning a video generation model on both simulated object positions and static backgrounds. However, generative models themselves lack any inherent physical reasoning or simulation capability: users must predefine the physical simulation parameters and rules for each scenario, and these settings cannot readily generalize to other contexts or different physical laws. These approaches can be summarized as:
Gy
Generation with Learned Physics Priors. As illustrated in Equation 3 these methods leverage
physical priors extracted from large-scale pretrained models to guide the generative process directly (Li ct al. 2024 Lv et al.] 2024 Xu et al. 2024 Yang et al. |
et al. [225]] Xue et al.
2025
Cao et al.] 2024; Wang et al. [025;] Yuan et al. 20256 Zhang et al. 2025b] Chefer et al.
2025
Zhang et al. [025cc] Feng et al. [2025 Yang al. 2025a For example, PhyT2V T2V (Xue et al.
(OpenA))2024a) model (VLM) (Wang al. (024a) as physics-consistency evaluators, performing multiple rounds of self-refinement to generate videos with improved physical plausibility. The main limitation of this line of work lies in the implicit assumption that existing models are capable of physical reasoning. In practice, however, these large-scale models, much like conventional video generation models, derive their so-called "physical understanding" purely from data fitting, and thus struggle when faced with physically challenging out-of-distribun however, our physical prior driven by both explicit physies models and physics-cleat data, which gives it stronger conditional controllability and better out-of-distributie generalization.
2025)
employs a large language model (LLM) ChatGPT (OpenAl, 2024a) and a rision-languue
scenarios. Our method broadly fits within this paradigm;
V G (T)
where @ is the learned physical parameters.
2.3 LEARN PHYSICS FROM VIDEOS
Leveraging the spatiotemporal information in videos, methods such as Wu et al. (2015): Watters ct al. (2017); ); Wu et al. (2017); : Belbute-Peree et al. (2018); Raissi et al. (2019); Chari et al. (201
Greydanus et al. (2019); Lutter al. (2019) Zhong & Leonard (2020): Jaques al. (2020): Le Guen
& Thome (2020) : Hotherr et al. (2023): ) Garrido et al. (2025): : Garcia et al. (2025); Deng et al. (2025)
Li et al. (2025c) estimate the parameters of known governing equation, which in turn enables tasks such as future- future-frame prediction and physical reasoning. These methods usually adopt an encoder-
decoder structure. Each frame is encoded into latent physical state using models like variational
3

---

2019). The latent state is then processed by physies engine and decoded back to reconstruct the frame for training. Most of these approaches are designed for single type of simple dynamical system. They are difficult to generalize to different systems within a single framework.
autoencoder (VAE) (Kigmma & Welling 2013
Our Neural Newtonian Dynamics (NND) is partly inspired by the aforementioned methods. We adopt an encoder-only architecture integrated with physics-informed general neural ordinary differential equations (ODEs) explicitly capture diverse dynamics from videos.
3
PRELIMINARY CONCEPTS
3.1 INCORPORATING PHYSICAL DYNAMICS INTO DATA-DRIVEN VIDEO GENERATION
Existing video generation models (OpenAl 2024b Hong et al. 2023 Kong et al. 2024 Yang et al.
20256 Blattmann al. 2023) are mostly data driven, relying on large-scale video datasets without physical annotations. W While they achieve good performance within training domains, they often fail in out-of-distribution scenarios by violating basic physical laws (Chefer et al. 2025] Kang et al.) 2025). In contrast, physies-driven dynamics methods explicitly incorporate governing constraints, yielding better physical plausibility and cut-of-distribution generalization (Champion et al., 2019). To combine the strengths of both, propose incorporating physical dynamics into data-driven video generation. This hybrid paradigm leverages the low-bias learning capacity of data-driven models while injecting lightweight dynamics priors to enforce consistency with fundamental laws, thereby achieving improved generalization and physically coherent video synthesis.
3.2 MODELING THE DYNAMICS GENERAL PIYSSICS- -INOOREEDD NEURAL ODE
To understand how NewtonGen works, we first ask: what is the best way to describe Newtonian motion? Physics textbooks tell us that if we are given the initial position, initial velocity, acceleration and mass, we can predict the trajectory of how the object moves in space and time. In mathematics, this is done through ordinary differential equations (ODEs). Based on this intuition, we consider a second-order system governed by autonomous ODEs with explicit time-varying external forces. We constrain the ODEs to the second order, because most common physical motions in daily life (e.g.. flying balls) can generally be described by second-order dynamics. Even more complex motions and parce-dimensione scenes, the dynamics can still be effectively characterized by second-order formulations over relatively short time intervals with sufficiently dense anchor points.
To handle a wide range of video generation tasks, we require the ODE framework capable of accommodating diverse dynamics. This raises the following question: how can we construct universal ODE framework that can describe various types of motion? To this end, we introduce two key design principles:
1. Latent Physical States. We define 9-dimensional latent physical state vector Z [x. y, Ugg By. 0, w, s, a). Here, x, y represent the position, and Uz. by represent velocity of the object's s center of mass. 0, encode the object's rotation or rotation about pivot point. s, are the object's s shortest and longest dimensions, and is projected area. This formulation allows other complex behaviors. 3D motion effect can also be equivalently realized through the combination of position and size control.
physical states to capture translation, rotation, deformation, and
2. Linear Physics-Informin Neural ODEs with a Residual MLP. Different motions follow inherently different dynamical laws: for instance, free-fall can be described by a simple linear ODE, while a damped pendulum or other unknown motion cannot. To address this, we combine linear physics-informed neural ODEs with residual multilayer perceptron (MLP) as illustrated Equation and Figure. aa.. The linear ODEs capture the dominant linear dynamics, while the residual MLP models nonlinear and unknown components, enabling the system to flexibly approximate a wide range of physical behaviors.
0z + b +c MLP(Z)
4

---

where is element of the 9-dimensional latent physical state vector Z, and az. bu, C,, d2 learnalle parameters of the linear ODE. We can use multiple ODEs to predict future physical states in compact autonomous form:
Z, Zo + Func(Z (a(() dr,
where Fune(Zi (Zr) represents the collection of all individual dz/d ODEs, and Zo Z(to) is the known initial physical state at time to.
4 METHODOLOGY
Problem Definition. We study generating videos of foreground objects whose motion obey classical mechanics, with controllable physical parameters, from prompts describing the scene and initial conditions.
Overall Framework. As shown in Figure. NewtonGen consists of two main stages. As illustrated in Figure. b), in the first stage, we train the proposed Neural Newtonian Dynamics (NND) on small set of physics-clean data learn the underlying motion dynamics and parameters. In the second stage shown in Figure. c[c,, we use the learned dynamics to predict future physical states from arbitrary initial conditions specified by the user via text prompts inference. and feed these predictions, together with the scene prompt, into motion-controlled text-to-vider generation model produce the final video.
(a) Neural Newtonian Dynamics (NND)
(b) Stage I: NND Training Pipeline
by ta Eu
General Physics-Informed Neural ODK
NND
Stamp:
Linear Neural ODE for
Linear Neural ODE
Linear
Video
Neural
Zo
ODE
ODE
Physical
Z
Solver odeint
Encoder
yx My]
[0 ar]
[s]
Zg
+
Eg
Residual MLP
for non-linear and unknown dynamics
Physics-Clean
Physical [x,y, Vy. s.
Vidcos
Pbyssod Paramatann
(c) Stage Physsics-Cnttrolaabe Inference Pipeline
World size: 18
Object size: 0.5
Object Necrest Shape: Circle
World
-Initial position: (12, 4)
Settings
Angular velocity rad/s # (w)
fax
Motion-Controlled T2V Models
Learned NND
Warped
Structured Noisc
tiny moon rotates along perfect orculer arbit ground lorge planet...
Scerc Prompt:
Predicted
Rendered
Physical States
Optical Flow
(Latent Variable)
Figure 2: The overall framework NewtonGen. a) Neural Newtonian Dynamics (NND) employs thysics-infomme linear neural ODEs combined with an MLP to build general dynamics learning framework suitable for diverse motions. We train NND on physies-clean dataset capture the underlying dynamics. c) Using the learned NND together with a data-driven motion-controlled model, we generate physically plausible and controllable videos.
4.1 NEURAL NEWTONIAN DYNAMICS
As discussed in Subsection 3.2 our Neural Newtonian Dynamics (NND) aims to construct a unified model capable of capturing wide range of dynamical behaviors. Its core composed of physics- informed general neural ordinary differential equations (Neural ODEs) (Chen et al. 2018). As demonstrated in Figure. a)) physics-iffrrmed linear neural ODEs employed model the
5

---

underlying linear dynamics, while residual three-layer MLP captures nonlinear and unknown components of the dynamics. With this design, the learnable neural ODEs can represent more complex or real-world dynamics. Given initial physical states Zo and a time stamp t, the ODE solver odeint (Chen et al. 2018) can be used to predict the object's future physical states Z,
4.2 TRAINING FOR NEURAL NEWTONIAN DYNAMICS
Overall Training Pipeline. Figure. b illustrates that, for training Neural Newtonian Dynamics (NND), we adopt an encoder-only architecture. This design does not require decoding back to images, and optimizes solely in the latent physical space, significantly reducing computational cost. Specifically, Video Physical Encoder Ephys compresses each video frame into its corresponding physical state. The initial state Zo and the sequence of frame time stamps (t1, 22; f3....) ) are fed into
NND, which predicts (2,, Z2. Z.....). The loss is then computed between the predicted states and the states (Z1, Zg, Zz....) extracted by the Encoder Ephys
T
Loss T
Σ 1=1
Ephys (I, - -NNDw (Ephys tTo),
(6)
Z,
Z.
where T denotes the number of sampled time stamps, I, is the video frame at time I, and k represents the learnable parameters of the ODEs.
Training Data. To enable Neural Newtonian Dynamics to learn accurate and effective representations of physical dynamics, we require "physics-clean" video data. That is, the motion in the videos should be prominent and monotonic, with motion blur excessive noise in each frame, and minimal color, texture, or background distractions. However, to our knowledge, such high-quality datasets of physical dynamics are still lacking. To address this, developed a Python- bbandd physics data simulator that can render videos with precise timestamps for different world settings, initial conditions, and types of dynamics. More details are shown in the Supplementary Material.
Video Physical Encoder. To extract physical state labels from videos, we first apply the visual segmentation foundation model SAM2 (Ravi al. 2025) obtain masks for the dynamic regions in each frame. From the extracted masks, we extract the centroid, area, long/short axes, and orientation of the foreground mask via morphological analysis, and compute velocities from inter-frame differences. Finally, these attributes are uniformly quantized to form the physical states Z.
4.3 INFERENCE FOR PHYSICAL-CONTROLII
TEXT-TO- VIDEO GENERATION
As illustrated in Figure. Cc), during inference, we decouple physical dynamics reasoning from video generation. Physical dynamics reasoning focuses modeling and predicting the motion of dynamic objects, while video generation leverages rich scene understanding and generation capabilities to render detailed and flexible visual content.
We adopt GGo-wih-the -ho-Flow (Burgert et al., 2025) as our base video generation model, which achieves motion control through structured noise (Chang al. 2024 By warping the independently initialized Gaussian noise of each frame according to the input optical flow, temporal correlations emerge between the initial noise of consecutive frames, leading more effective motion control. Other motion-controlles video generation models (Yin et al.] 2023 Wang et al.) 2024b Zhhng al. 2025d) typically encode trajectories or bounding boxes through ControlNet (Zhang al.
2023 or additional encoders and inject the features into the base video generators. However, these approaches often struggle with handling deformations, rotations, or more complex motions. We choose Go-with-the-Flow for its generality and effectiveness.
To effectively transfer the physical knowledge from our NND to the video generation model, a multi-step procedure is required. First, based on the user's physical prompts, we parse the initial physical state Zo and future time stamps. Zo and the frame timestamps are fed into the trained NND to obtain the corresponding physical states for all future frames. Next, using the world setting information parsed from the physical prompts (e.g., scene dimensions, object size, and the closest simple geometric shape of the object), compute approximate pixel-level optical flow for each frame based on the predicted physical states. These flows are then temporally and spatially downsampled match the resolution of the video generator's latent space, resulting in structured

---

optical flow sequence. Finally, combining the user's scene prompts, video frames are sampled to produce the final videos.
EXPERIMENTS
In this section, we evaluate the applications of our framework for physically and control- lable video generation. Subsection 5.1 presents implementation details, Subsection 5.2] compares NewtonGen with other baselines, and Subsection 5.3 discusses the results of ablation study.
5.1 IMPLEMENTATION DETAILS
Supported Motion Types. In NewtonGen, we evaluate 12 distinct types of motion: uniform motion, acceleration, deceleration, parabolic motion, 3D motion, slope sliding, circular motion, rotation, parabolic motion with rotation, damped oscillation, size changing, and deformation. These categories cover the most common fundamental motion patterns encountered in everyday scenarios. The tested velocity magnitudes are mostly within the range of 0-15 m/s, while the duration of the generated motions is typically concentrated within 1-2 seconds.
Training Details for NND. We optimize the NND with the AdamW optimizer (initial learning rate I x 10-3) and CosineAnnellingL. scheduler (Loshchilov & Hutter 2017). For each type of motion, collect 100 physical videos with different initial conditions from the physics simulator mentioned in Subsection 4.2 as training data. The model is trained with a batch size of 64 for a total of 20,000 epochs, which requires about hours on a single NVIDIA A100 80 GB GPU.
Metries. Assessing the physical consistency of different video generation models is hampered by the absence of a shared ground truth and by the fact that each synthetic sequence is defined in its own coordinate frame and scale and time. Consequently, single, unified physical evaluation metric cannot be directly applied. In this work, we are inspired by the Physical Invariance Score (PIS) (Zhang et al., 2025a), which evaluates physical plausibility by checking whether a motion preserves its expected invariants C. For example, in parabolic motion and the horizontal velocity Up should be constant. We use SAM2 (Ravi al. (025) segment the object in each frame and obtain its centroid and shape features. Velocities are estimated from
centroid differences. The Physical Invariance Score for quantity C is defined as the relative standard deviation of C over time:
PIS (1 + C. ((Ccl + +c)
where C' denotes one of the quantities introduced above (i.e., the horizontal velocity, the vertical acceleration, or the angular speed). e = x 10 -5- added to the denominator to prevent division by zero. The PIS score is bounded in [0, 1], with value of indicating that the evaluated physical quantity remains perfectly invariant.
We also adopt the background consistency (BC) and motion smoothness (MS) metrics from VBench (Huang et al., 2024)) to assess scene consistency and motion quality.
5.2 COMPARISONS WITH OTHER METHODS
General Comparisons. We compare our method with five baselines: SORA OpenAi 2024b), Veo3 (Goole 2025). Cog Vog (Yang et al. 2025b), Wan2.2 (Wan et al. 2025) and PhyT2V (Xuc alal. 2005). These baselines represent the current state-of-the-ard in both closed-source and open- source video generation models, as well as physics- based generation methods. We standardize the video generation settings across all methods ensure maximum fairness in comparison. We collected 24 prompts for each motion type to assess the physical generation capabilities of all methods.
In Figure. 3 the video sequences generated by NewtonGen exhibit the highest degree of physical consistency across all 12 motion types. The motions display smooth and realistic trajectories without abrupt changes in direction or speed, realistic 3D movement effects (with object scale gradually increasing as distance decreases), physically plausible self-rotation (objects preserve shape with uniform angular velocity), smooth deformations (edges stretch or shrink progressively), and natural size variations (c.g., balloon diameter increases over time but at decelerating rate). Table] T further shows that our model achieves significantly higher physical consistency scores than competing methods across different motion categories. Notably, some motion types do not admit perfect physical
7

---

invariants; in these cases, still compute quantities such angular velocity and compare them against reference simulation videos under the same conditions.
Baselines
NewtonGen
Uniform Motion: "A bowling ball moving steadily down polished wooden lone..."
"A black off-road accelerating straight sandy ferrain..."
WanzZ Ciggidee
Deceleration: 'A silver sports car moves in straight line on a countryside road of dusk.."
Parabolic Mation: "A basketball thrown angle towords hoop..."
Phy Pyyzzv
Movement: bowling ball moving from the far end polished wooden lane: toward the camera...
Sora
Slope Sliding: 'A smartphone sliding quickly down glessy inclined table surface..."
Yea
Cireular Mation: small planet moves smoothly around bright star..."
Cogg VideVe --8
Rotation:" "A wooden dowel rotating gently tiled kitchen..."
Wanz.2
paintbrush angle, rotating while falling..."
PhyTzv
Damped Oseillation: "A small decorative bell hanging from fine chain..."
Size Changing: 'A white balloon inflating cozy living room..."
Wanz.2
Deformation: 'A piece soft dough is evenly flattened workbench..."
Figure 3: Visual comparisons of different text-to-video generation methods across diverse physical dynamics, where our method consistently shows strong physical consistency and controllability (such control the shape of dough).
Parameter Controllability Comparisons. In Figure. we demonstrate NewtonGen's 's ability to perceive physical parameters. Unlike other models, our method faithfully reflects world settings, object properties, and initial conditions, with trajectories and velocities that better follow physical laws (third row). More cases provided in the Appendix.
8

---

Table 1: Quantitative comparison with different methods. The reference scores are computed on the simulated videos, and the detailed definition of PIS for each type of motion is provided in the Appendix. BC and MS denote background consistency and motion smoothness, respectively. with values in parentheses indicating the standard deviation across videos. We highlight the best and second-best results for each metric.
Methous
Morion Types
Metricst
Reference
Soca
Von1
CogVidooX-SR
Wan222
PhyT22
Chirs
PIS-c
0.5972
0.6549(0022)
(.978400.006)
(L5592/0.007)
0.5349(0014)
0.98500UL
Uniforn Morion
BC
1,0000
0.957300.00)
(.009))0002)
0.9634(0.018)
0.96830.027)
0,9612(0.15))
0,969440 020)
MS
1,0000
0.9926/0.03)
0.95520.001)
0.9905(0.005)
0.9339
0.9576(0.615)
0.996200 (033
PIS-a;
0.34299
0,3437(02255)
3000
0.545600.033)
0.30770.261
0
(155
Acceleration (Uniform)
BC
1,0000
0,9495(0.011) l)
0.9373000015)
0.951820.057)
0.9665000188)
0 0,9636(0.221)
0,974840 01
MS
1,0000
0,9852/0.01)
0.901X(00004
0.9876600008)
0.9508(00015)
0.9622(0010)
0.991860 009)
PIS dz
0.3872
0.6162(00072)
0.6175001022
0.4988000014
0.470500325)
0.516700023)
(6,9))
Deceleration (Uniferm)
BC
1,0000
0.9494(0.006)
0.295(0.090)
0.9623000177
(0.12
0.9622/0.022)
0.974400 012)
MS
1,0000
0.96
0.9933/0.000)
(19797(00124)
0.57
0.9534(0.014)
0.994700 000))
PlS.
0.9388
(.9095(0.014)
0.9443/00022)
0.7392(0.007)
0.7747(0.120 120)
0.657100.199 199)
0,980000 002)
Parabolic Motion
PIS dv
0.9487
0.572300266)
0.7663/0.139)
0.423000025)
0.55710.993)
0.356700799)
0.81890000.9
BC
1,0000
0,9:30(0.023)
0.951400.023)
(1933000.1300
0.9002/0.025)
0.9436(0.046)
0.90XE((0014)
MS
1,0000
0,9915(0.004)
0.994820.000 (02)
0.9856(0.009)
0.990300007 007)
0, 0,9844(0.01)
0.995740 001)
PtS A
0,7388
0,501300.005)
(3952/0.005)
0.302600.005)
0.45830.005)
0 0.2911(0.007)
0.667240.000 005)
3D Motion
PIS-c,
0,5086
(.8481
(20100)
0.6900000003
0.8384(0.015)
0.65ID(0.002)
0.9377 (07)
BC
1,0000
0.9:26/0.017)
0.94100000222
0.962100.019)
0.977200005)
0. 0.9629(0.016)
0.9672(0.018)
MS
1,0000
0,9934(0.003)
0.9944.0.000)
0.994520.0033
0 0,9943/00002 002)
0.9858(0.012)
0 0.995440.00 005)
PlS-.,
0.5741
0.4951(0113)
(.6181(01157)
0L353330. 160)
0.31080.4221
0,3570/0 354)
0.651220 041)
Slupe Sliding
PiS-o,
0.9148
014)16(0.212)
0.3815(0.092)
0.475100.023
0.4292(0.669) 0.3967
0.544000.033)
BC
1,0000
0.9667/0.013 3)
0.9631000166
0.955620.00 024)
0.963((.011)
0.9568/0.023) 022)
0.978700 010)
MS
1,0000
0.991900.005)
0.905800.000
0.5003/0.0006)
0.9112000055
0.9629/0.014)
0.9971 0011
0.9933
0.839300010)
0.893300.007)
0.7726:0.026)
0.16720.006)
(.039)
09788800018)
Ciroular Motion
RC
1.0000
0.9684(0.012)
0.9842(0.013)
0.9745(0006)
0.9677(0.027)
0.981240
Ms
1,0000
0.9949900011
0.99600.001)
0.99999000000
0.99
0.907400.002)
0.995000
PiS-
0.96366
0.4267(01099)
0.5285/04466
(16596/0.023)
0.3425/0.172)
0.7842
Rotation (Uniform)
BC
1,0000
0.9543(0.08))
0.96900.018)
0.9897(0.025)
0.962000010)
0.9575(0.028)
0,U70000 (08)
MS
1,0000
0,990000006)
0.99:2/0.000)
0.9795/0.029)
0.9909/0.007)
0.9578(0.006)
0.995880 (002)
PIS-4:
0.5990
0,5797(0111))
0.702X001997
0AS58001155)
U. (.7099(0.039)
0.944600 (UR)
PIS
0,9657
0.4903(2.581)
0.560
(20.11201177
0.4331(1 982)
0.2879(0.046)
0.56 14/0 028)
Para, wi Rocation
PIS-
0.9829
0.6522/0 556)
0.9019/0.110
0.338000.199 1999
0,3474/03344 334)
0.411990 105)
0,928900 O29)
BC
1,0000
0.9532/0.016)
0.9883/00088)
0.9567(0.018)
(.607)
026675(0.000)
0.979660 009)
MS
1,0000
0.988900005)
0.995300.001
0198-1(000115)
(.9921(0.003)
0.99M990 0011
PIS-oy
0.9402
0:4018/0 364)
0.3516(0.452)
0.3083(0.055)
0.3494/0.399 395)
(0.642)
0,524000 01
Dimped Oscillation
DC
1.0000
0.9738/0.013)
0.9666/00011)
0.9699000007
0.971500.014
0.9708(0.010)
0,974220.012)
MS
0.990000014)
0.905800.001
0.9853/0.005)
0.9919/0.019 0099)
0.9667(0.005)
0.99680
A,
0.3501
0.2840(0000)
0.4167(0.006)
0.5774000007)
0.1972/0.022
0,401000011)
0.636220.010 010)
Size Clanging
BC
1,0000
0.9507(0.025)
0.9545(0.017)
0.963600.019)
0.9735000188)
0.9666(0.022)
0,968960001 01015)
MS
1,0000
0.9916(0.003
0.9955(0.002)
0.992620.007)
0.9889(0.005
0.9925/0.004)
0.995500 (02)
PIS 21
0.92447
0.3626/0.04)
0.3466000017)
0.255000.002)
0.3515000433
0.300100.003
0.54
Deformation
BC
1.0000
0.9553(0.009)
0.9058/0.052)
0.946200.018)
0.9347(0042)
0.9211000110)
0.9475.0 025)
MS
1,0000
0.9943(01004)
0.99200.006)
0.9935/0.009)
0.5903000009
0.9567(0.01)
0.9157000013
Table 2: Quantitative results of ablation study. We compute the normalized absolute error between the predicted and ground -ruth physical states across all time steps within the test batch.
Monions
Uni
Abc
Dec
Pam
3DMe
Slope
Cire
Rota
ParaRota
Osei
Size
Def
Ablations
Normalizedl
Absolutc Errr
0.0174
0.0069
0.0193
0.0831
0.3358
0.0382
0.1239
Our-hata0
0.0632
0.0260
0.0184
0.0284
0.1079
0.0935
0.1246
0.0739
0.0273
0.10415
0.2527
0.0555
Dar-data100 100
0.0142
0.0034
0.3042
0.0324
0.0255
0.0058
0.00664
0.0425
0.1193
0.0357
Ourditt500
0.0195
0,0051
0.0072
0,0040
0.0192
0,0307
0.0196
0,0049
0.0063
0.0694
0.1379
0.0290
5.3 ABLATION STUDY
Our ablation study focuses on the effects of the MLP in Neural Newtonian Dynamics (NND), the training data scale and real-world video training. As shown in Table adding the MLP significantly improves NND's performance nonlinear dynamics and noisy data. Increasing the training dataset size does not lead to notable gains, indicating that NND can accurately infer the underlying system dynamics from relatively small number of physically clean samples.
9

---

NewtonGen
Sora
lekid buste lendfged
Pa-SA symd
Localon Frame Stred
Couao STA
Poare
Proma
Figure NewtonGen generates videos that can accurately reflect user-specified initial physical parameters, including object position, velocity, angle, shape and size.
Videoo/Masks
Dynamics Videos
(C) Generated Videos by NewtonGen
Figure 5: Feasibility of training NewtonGen(NND) on real-world videos
Real- world video training. To evaluate NewtonGen real-world data, select real-world videos from the PISABench (Li al. 2025a) (example frames are shown in Fig. S(a). We train NND on these fall videos. As shown in Fig. ((b), NND effectively learn the underlying falling dynamics, even though the videos contain motion blur. Using the dynamics learned from these real videos, generate physically plausible falling motions in scenes,
NewtonGen then generalize as illustrated in Fig. c[.. For this real-world case, the PIS-E, and PIS-G., scores are 0,8485 and 0.6008, which lower than our results on the simulated dataset (0.9803 and 0.8189). This shows that collecting real dynamic scenes is feasible but time-consuming and requires careful setup (e.g.. arranging scenes and measuring physical scales), and the data quality is often lower than that of clean simulations. Our simulated physics-clean data provides faster and cheaper way to obtain high-quality training data.
CONCLUSION
In this paper, we introduce NewtonGen, a physics-consisten and controllable text-to-video generation framework. NewtonGen integrates Neural Newtonian Dynamics (NND) module, which learns latent dynamics for diverse motions from a small set of physically accurate examples and predicts future physical states. We validate NewtonGen tasks, demonstrating its physical consistency and parameter controllability. NewtonGen holds the potential to narrow the gap between current generative models and the real physical world.
twelve different dynamic video generation
Limitations. While our framework effectively models and predicts the dynamics of most common motions, it is based continuous dynamics. This means that NewtonGen can be less effective for handling multi-object interactions (e.g., collisions or coalescence). We expect that future work incorporating event-based or discrete neural architectures will address these limitations.
10

---

7 ETHICS STATEMENT AND REPRODUCIBILITY STATEMENT
Ethics Statement. This model is designed to generate high-quality content and educational videos; however, when misused without labels and watermarks, it can produce fake videos and lead to the spread of misinformation.
Reproducibility Statement. All data, code and model weights will made publicly available. In our model evaluation, we fix the random seed and provide the test prompts and generated videos in the supplementary materials and appendix. In addition, we include more detailed explanations of the evaluation metries in the Appendix.
ACKNOWLEDGMENTS
This work supported. in part. by the United States National Science Foundation under the grants 2133032, 205, and research award from Samsung Research America.
REFERENCES
Stable Diffusion. http:////gtttt...oo/////aaaaaaaa
Hritik Bansal, Zongyu Lin, Tianyi Xie, Zeshun Zong, Michal Yarom, Yonatan Bitton, Chenfanfu Jiang, Yizhou Sun, Kai-Wei Chang, and Aditya Grover. VideoPhy: Evaluating physical commonsense for video generation. arXiv preprint arXiv:2406.03520. 2024.
Hritik Bansal, Clark Peng, Yonatan Bitton, Roman Goldenberg, Aditya Grover, and Kai-We Chang VideoPhy-2: A challenging action-centrie physical commonsense evaluation in video generation. arXiv preprini arXiv:2503.068m0 2025.
Filipe de Avila Belbute-Peres Kevin Smith, Kelsey Allen, and Josh Tenenbaum. End-to-end differentiable physies for learning and control. In Advances in Neural Information Processing Systems, 2018.
Andreas Blattmann, Tim Dockhorn, Sumith Kulal, Daniel Mendelevitch, Maciej Kilian, Dominik Lorenz, Yam Levi, Zion English, Vikram Voleti, Adam Letts, Varun Jampani, and Robin Rombach. Stable video diffusion: Scaling latent video diffusion models to large datasets. arXiv preprint arXiv:2311.15127, 2023.
Ryan Burgert, Yuancheng Xu, Wenqi Xian, Oliver Pilarski, Pascal Clausen, Mingming He, Ma, Yitong Deng, Lingxiao Li, Mohsen Mousavi, Michael Ryoo, Paul Debevec, and Ning Yu. Go-with- the-flow: Motion- cattoo--controllable video diffusion models using real-tine warped noise. In IEEE/CVF Conference on Computer Vision and Patlern Recognition, 2025.
Qinglong Cao, Ding Wang, Xirui Li. Yuntian Chen, Chao Ma, and Xiaokang Yang. Teaching video diffusion model with latent physical phenomenon knowledge. arXiv preprint 2..i::211..13444 2024.
Kathleen Champion, Bethany Lusch, J. Nathan Kutz, and Steven I.. Brunton. Data-driven discovery of coordinates and governing equations. Proceedings of the National Academy of Sciences, 116 (45):22445-22451, 2019.
Pascal Chang, Jingwei Tang, Markus Gross, and Vinicius C. Azevedo. How i warped your noise: temporally-correlatee noise prior for diffusion models. In International Conference on Learning Representations, 2024.
Pradyumna Chari, Chinmay Talegaonkar, Yunhao Ba, and Achuta Kadambi. Visual physics: Discov- ering physical laws from videos. arXiv preprint arXiv:1911.1189) 2019.
Hila Chefer, Uriel Singer, Amit Zohar, Yuval Kirstain, Adam Polyak, Yaniv Taigman, Lior Wolf, and Shelly Sheynin. Videojam: Joint appearance motion representations for enhanced motion generation in video models. In International Conference on Machine Learning, 2025.
11

---

Boyuan Chen, Hanxiao Jiang, Shaowei Liu, Saurabh Gupta, Yunzhu Li, Hao Zhao, and Shenlong Wang. Physgen3d: Crafting a miniature interactive world from a single image. In JEEE/CVF Conference Computer Vision and Pattern Recognition, 2025.
Ricky T. Q. Chen, Yulia Rubanova, Jesse Bettencourt, and David Duvenaud. Neural ordinary differential equations. In Advances in Neural Information Processing Systems, pp. 6572-6583, 2018.
Congyue Deng, Brandon Y. Feng, Cecilia Garraffo, Alan Garbarz, Robin Walters, William T. Freeman, Leonidas Guibas, and Kaiming He. Denoising hamiltonian network for physical reasoning. arXiv preprint 0rXiv:2503.0759, 2025.
Haoyi Duan, MooggXXng Yu, Sirui Chen, Li Fei-Fei, and Jiajun Wu. WorldScore: A unified evaluation benchmark for world generation. arXiv preprint orXiv2254.00983, 2025.
Tao Feng. Xianbing Zhao, Zhenhua Chen, Tien Tsin Wong, Hamid Rezatofighi, Gholamreza Haffari, and Lizhen Qu. -hooudd-grouuned motion forecasting equation discovery for guuiddd image-to-video generation. arXiv preprint arXiv:2507.06830, 2025.
Alejandro Castañeda Garcia, Jan Gemert, Daan Brinks, and Nergis Tömen. Learning physics from video: Unsupervised physical parameter estimation for continuous dynamical systems. In IEEE/CVF Conference on Computer Vision and Pattern Recognition, 2025.
Quentin Garrido, Nicolas Ballas, Mahmoud Assran, Adrien Bardes, Laurent Najman, Michael Rabbat, Emmanuel Dupoux, and Yann LeCun. Intuitive physics understanding emerges from self-supervised pretraining on natural videos. arXiv preprint 27X1::2500.089977, 2025.
Google. Veo 3: Our state-of-the-art video generation model. https:://issuuioo.ggr google
com/modeess/eoo-o 2025.
Samuel Greydanus, Misko Dzamba, and Jason Yosinski. Hamiltonian neural networks. In Advances in Neural Information Processing Systems, 2019.
Jing Gu, Xian Liu, Yu Zeng, Ashwin Nagarajan, Fangriu Zhu, Daniel Hong, Yue Fan, Qianqi Yan. Kaiwen Zhou, Ming- Yu Liu, and Xin Eric Wang. Phyworldbench: A comprehensive evaluation of physical realism in text-to-video models. arXiv preprint arXiv:2507.13428, 2025.
Jonathan Ho, Ajay Jain, and Pieter Abheel. Denoising diffusion probabilistic models. In Advances in Neural Information Processing Systems, pp. 6840-6851, 2020.
Jonathan Ho, Tim Salimans, Alexey Gritsenko, William Chan, Mohammad Norouzi, and David Fleet. Video diffusion models. In Advances in Neural Information Processing Systems, volume 35, pp. 8633-8646, 2022.
Florian Hofherr, Lukas Koestler, Florian Bernard, and Daniel Cremers. Neural implicit representations for physical parameter inference from single video. In IEEE/CVF Winter Conference on Applications of Computer Vision, 2023.
Wenyi Hong, Ming Ding, Wendi Zheng, Xinghan Liu, and Jie Tang. Cogvideo: Large-scale pretraining for text-to-video generation via transformers. In International Conference on Learning Representations, 2023.
Hao-Ya Yu Hsu, Zhi-Hao Lin, Albert Zhai, Hongchi Xia, and Shenlong Wang. Autovfx: Physically realistic video editing from natural language instructions. arXiv preprint arXiv:2411.023300 2024.
Ziqi Huang, Yinan He, Jiashuo Yu, Fan Zhang, Chenyang Si, Yuming Jiang, Yuanhan Zhang. Tianxing Wu, Qingyang Jin, Nattapol Chanpaisit, Yaohui Wang, Xinyuan Chen, Limin Wang, Dahua Lin, Yu Qiao, and Ziwei Liu. VBench: Comprehensive benchmark suite for video generative models. In JEEE/CVF Conference on Computer Vision and Pattern Recognition, 2024.
Miguel Jaques, Michael Burke, and Timothy Hospedales. mhysics-as-inverse - Unsupervised physical parameter estimation from video. In International Conference on Learning Representa- fions, 2020.
12

---

Bingyi Kang, Yang Yue, Rui Lu, Zhijie Lin, Yang Zhao, Kaixin Wang, Gao Huang, and Jiashi Feng. How far is video generation from world model: A physical law perspective. In International Conference on Machine Learning, 2025.
Jared Kaplan, Sam McCandlish, Tom Henighan, Tom B. Brown, Benjamin Chess, Rewon Child, Scott Gray, Alec Radford, Jeffrey Wu, and Dario Amodei. Scaling laws for neural language models. arXiv preprint arXiv: 2001.08361, 2020.
Diederik P. Kingma and Max Welling. Auto-encoding variational bayes. arXiv:1312.61 14, 2013.
arXiv preprint
Diederik P. Kingma and Max Welling. An introduction to variational autoencoders. arXiv preprint arXiv:1906.02691, 2019.
Weijie Kong, Qi Tian, Zijian Zhang, Rox Min, Zuozhuo Dai, Jin Zhou, Jiangfeng Xiong, Xin Li, Bo Wu, Jianwei Zhang, et al. Hunyuanvideo: A systematic framework for large video generative models. arXiv preprint arXiv:24/2.03603 2024.
Vincent Le Guen and Nicolas Thome. Disentangling physical dynamics from unknown factors for unsupervised video prediction. In IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 1147--1448 2020.
Chenyu Li, Oscar Michel, Xichen Pan, Sainan Liu, Mike Roberts, and Saining Xie. Pisa experiments: Exploring physies post-training for video diffusion models by watching stuff drop. International Conference Machine Learning, 2025a.
Dacheng Li, Yunhao Fang, Yukang Chen, Shuo Yang. Shiyi Cao, Justin Wong. Michael Luo, Xiaolong Wang, Hongxu Yin, Joseph E. Gonzalez, Ion Stoica, Song Han, and Yao Lu. WorncModelBenchh Judging video generation models world models. arXiv preprint arXiv:2502.20664, 2025b.
Shiqian Li, Ruihong Shen, Chi Zhang, and Yixin Zhu. Neural force field: Learning generalized physical representation from a few examples. arXiv preprint arXiv:2502.08800 0888777 2025c.
Zhengqi Li, Richard Tucker, Noah Snavely, and Aleksander Holynski. Generative image dynamics. In IEEE/CVF Conference on Computer Vision and Pattern Recognition, 2024.
Zizhang Li, Hlog-XXigg Yu, Wei Liu, Yin Yang. Charles Herrmann, Gordon Wetzstein, and Jiajun Wu. Wonderplay: Dynamic 3d scene generation from single image and actions. In International Conference on Computer Vision, 2025d.
Jiajing Lin, Zhenzhong Wang, Shu Jiang, Yongjie Hou, and Min Jiang. Phys4dgen: A physics-driven framework for controllable and efficient 4d content generation from a single image. arXiv preprint arXiv:2411.16800 2024.
Minghui Lin, Xiang Wang. Yishan Wang, Shu Wang, Fengqi Dai, Pengxiang Ding, Cunxiang Wang, Zhengrong Zuo, Nong Sang, Siteng Huang, and Donglin Wang. Exploring the evolution of physics cognition in video generation: A survey, arXiv preprint arXiv:2503.21765, 2025.
Daochang Liu, Junyu Zhang, Anh-Dung Dinh, Eunbyung Park, Shichao Zhang, and Chang Xu. Generative physical ai in vision: A survey, 2025.
Shaowei Liu, Zhongzheng Ren, Saurabh Gupta, and Shenlong Wang. Physgen: Rigid-body physics- grounded image-to-video generation. In European Conference on Computer Vision, 2024.
Ilya Loshchilov and Frank Hutter. Sgdr: Stochastic gradient descent with warm restarts. In Interna- tional Conference on Learning Representations, 2017.
Michael Lutter, Christian Ritter, and Jan Peters. Deep lagrangian networks: Using physics as model prior for deep learning. In International Conference on Learning Representations, 2019.
Jiaxi Lv, Yi Huang Huang, Mingfu Yan, Jiancheng Huang, Jianzhuang Liu, Yifan Liu Liu, Yafei Wen, Xiaoxin Chen, and Shifeng Chen. Gpt4motion: Scripting physical motions in text-to-video generation via dlender-orienter gpt planning. In IEEE/CVF Conference on Computer Vision and Pattern Recognition, 2024.
13

---

Fanqing Meng, Jiaqi Liao, Xinyu Tan, Wenqi Shao, Quanfeng Lu, Kaipeng Zhang, Cheng Yu, Dianqi Li, Yu Qiao, and Ping Luo. Towards world simulator: Crafting physical commonsense based benchmark for video generation. In International Conference on Machine Learning, 2025.
Saman Motamed, Laura Culp, Kevin Swersky, Priyank Jaini Jaini, and Robert Geirhos. Do generative video models understand physical principles?, 2025.
NVIDIA. Cosmos world foundation model platform for physical ai. arXiv preprint 303725003555 2025.
OpenAl. Introducing openai ol ol-preview. 2024a.
ttps:://peeaai.com/nneee
OpenAl. Video generation models as world simulators. https://openai.com/ /ndex
wtw.ddddssmmaaasr///omoo 2024b.
Karran Pandey, Matheus Gadelha, Yannick Hold-Geoffroy, Karan Singh, Niloy J. Mitra, and Paul Guerrero. Motion modes: What could happen next? In /EEE/CVF Conference on Computer Vision and Pattern Recognition, 2025.
William Peebles and Saining Xie. Scalable diffusion models with transformers. In International Conference Computer Vision, 2023.
Maziar Raissi, Paris Perdikaris, and George Karniadakis. Physics-informed neural networks: A deep learning framework for solving forward and inverse problems involving nonlinear partial differential equations. Journal of Computational Physics. 378:686-707, 2019.
Aditya Ramesh, Mikhail Pavlov, Gabriel Goh, Scott Gray, Chelsea Voss, Alec Radford, Mark Chen, and Ilya Sutskever. Zero-shot text-to-image generation. In International Conference Machine Learning, 2021.
Nikhila Ravi, Valentin Gabeur, Yuan-Ting Hu, Ronghang Hu, Chaitanya Ryali, Tengyu Ma, Haitham Khedr, Roman Rädle, Chloe Rolland, Laura Gustafson, Eric Mintun, Junting Pan, Kalyan Vasudev Alwala, Nicolas Carion, Chao Yuan Wu, Ross Girshick, Piotr Dollár, and Christoph Feichtenhofer. Sam 2: Segment anything in images and videos. International Conference on Learning Representations, 2025.
Robin Rombach, Andreas Blattmann, Dominik Lorenz, Patrick Esser, and Björn Ommer. High- resolution image synthesis with latent diffusion models. In IEEE/CVE Conference on Computer Vision and Puttern Recognition, pp. 10674-10685, 2022.
Luca Savant Aira, Antonio Montanaro, Emanuele Aiello. Diego Valsesia, and Enrico Magli. Motion- craft: Physics-based zero-shot video generation. In Advances in Neural Information Processing Systems, 2024.
Yang Song, Jascha Sohl-Dickstein, Diederik P Kingma, Abhishek Kumar, Stefano Ermon, and Ben Poole. Score-based generative modeling through stochastic differential equations. In International Conference on Learning Representations, 2021. URL /tttps://openrevieew net/ www./roruuww..t/ooouuuu
1d=PxTIG12RRHS
Alexey Stomakhin, Craig Schroeder, Lawrence Chai, Joseph Teran, and Andrew Selle. A material point method for snow simulation. ACM Transactions Graphics, 2013.
Xiyang Tan, Ying Jiang, Xuan Li, Zeshun Zong Tianyi Xie, Yin Yang, and Chenfanfu Jiang. Physmotion: Physics-grounded dynamics from single image. arXiv preprint 24Xiv:2411.17188) 2024.
Team Wan, Ang Wang, Baole Ai, Bin Wen, Chaojie Mao, Chen-Wei Xie, Di Chen, Feiwu Yu, Haiming Zhao, Jianxiao Yang, Jianyuan Zeng. Jiayu Wang, Jingfeng Zhang, Jingren Zhou, Jinkai Wang, Jixuan Chen, Kai Zhu, Kang Zhao, Keyu Yan, Lianghua Huang. Mengyang Feng, Ningyi Zhang, Pandeng Li, Pingyu Wu, Ruihang Chu, Ruili Feng. Shiwei Zhang, Siyang Sun. Tao Fang, Tianxing Wang. Tianyi Gui, Tingyu Weng. Tong Shen, Wei Lin, Wei Wang, Wei Wang. Wenmeng Zhou, Wente Wang, Wenting Shen, Wenyuan Yu, Xianzhong Shi, Xiaoming Huang, Xin Xu, Yan
14

---

Kou, Yangyu Lv, Yifei Li, Yijing Liu, Yiming Wang. Yingya Zhang, Yitong Huang, Yong Li, You Wu, Yu Liu, Yulin Pan. Yun Zheng, Yuntao Hong, Yupeng Shi, Yutong Feng, Zeyinzi Jiang, Zhen Han, Zhi-Fan Wu, and Ziyu Liu. Wan: Open and advanced large-scale video generative models. arXiv preprint arXiv:2503 250::223332334 2025.
Jiawei Wang, Liping Yuan, Yuchen Zhang, and Haomiao Sun. Tarsier: Recipes for training and evaluating large video description models. arXiv preprint wrXiv:2477.00633 2024a.
Jing Wang. Ao Ma, Ke Cao, Jun Zheng, Zhanjic Zhang, Jiasong Feng, Shanyuan Liu, Yuhang Ma. Bo Cheng, Dawei Leng, Yuhui Yin, and Xiaodan Liang. WISA: World simulator assistant for physics-aware text-to-video generation. arXiv preprint 2r02v22502.08153 2025.
Zhouxia Wang, Ziyang Yuan, Xintao Wang, Yaowei Li, Tianshui Chen, Menghan Xia, Ping Luo, and Ying Shan. Motionctrl: A unified and flexible motion controller for video generation. In ACM SIGGRAPH 2024 Conference Papers, pp. 1-11, 2024b.
Nicholas Watters, Daniel Zoran, Theophane Weber, Peter Battaglia, Razvan Pascanu, and Andrea Tacchetti. Visual interaction networks: Learning a physics simulator from video. In Advances in Neural Information Processing Systems, 2017.
Jiajun Wu, liker Yildirim, Joseph J. Lim, William T. Freeman, and Joshua B. Tenenbaum. Galileo: Perceiving physical object properties by integrating a physics engine with deep learning. In Advances in Neural Information Processing Systems, 2015.
Jiajun Wu, Erika Lu, Pushmeet Kohli, Bill Freeman, and Josh Tenenbaum. Learning to see physics via visual de-animation. In Advances in Neural Information Processing Systems, 2017.
Tianyi Xie, Zeshun Zong, Yuxing Qiu, Xuan Li, Yutao Feng, Yin Yang, and Chenfanfu Jiang. Physgaussian: Physics-integrated 3d gaussians for generative dynamics. In JEEE/CVF Conference on Computer Vision and Pattern Recognition, 2024.
Tianyi Xie, Yiwei Zhao, Ying Jiang, and Chenfanfu Jiang. Physanimator: Mhysics-guides generative cartoon animation. In IEEE/CVF Conference on Computer Vision and Pattern Recognition, 2025.
Tianshuo Xu, Zhifei Chen, Leyi Wu, Hao Lu, Yuying Chen, Lihui Jiang, Bingbing Liu, and Yingcong Chen. Motion dreamer: Realizing physically coherent video generation through scene-aware motion reasoning. arXiv preprint arXiv:2412.005:: 2024.
Qiyao Xue, Xiangyu Yin, Boyuan Yang, and Wei Gao. PhyT2V: Llm-guided iterative self-refinement for physics-grounded grounded text-to-video generation. In IEEF/CVF Conference on Compuler Vision and Puttern Recognition, 2025.
Xindi Yang. Baolu Li, Yiming Zhang. Zhenfei Yin, Lei Bai, Liqian Ma, Zhiyong Wang, Jianfei Cai, Tien-Tsin Wong, Huchuan Lu, and Xu Jia. Vlipp: Towards physically plausible video generation with vision and language informed physical prior. In International Conference Computer Vision, 2025a.
Zhuoyi Yang. Jiayan Teng. Wendi Zheng, Ming Ding, Shiyu Huang, Jiazheng Xu, Yuanming Yang, Wenyi Hong, Xiaohan Zhang, Guanyu Feng, et al. Cogvideox: Text-to-video diffusion models with an expert transformer. In International Conference on Learning Representaatinns 2025b.
Shengming Yin, Chenfei Wu, Jian Liang, Jie Shi, Houqiang Li, Gong Ming, and Nan Duan. Dragnuwa: Fine-grained control in video generation by integrating text, image, and trajectory. arXiv preprint arXiv:2308. 08089999..00 2023.
Jianhao Yuan, Fabio Pizzati, Francesco Pinto, Lars Kunze, Ivan Laptev, Paul Newman, Philip Torr, and Daniele De Martini. Likephys: Evaluating intuitive physics understanding in video diffusion models via likelihood preference. arXiv preprint 2rXiv:2510.1111 15112 2025a.
Ye Yuan, Jiaming Song. Umar Iqbal, Arash Vahdat, and Jan Kautz. Physdiff: moysscs--uidee human motion diffusion model. In International Conference Computer Vision, 2023.
15

---

Yu Yuan, Xijun Wang. Yichen Sheng, Prateek Chennuri, Xingguang Zhang, and Stanley Chan. Generative photography: Scene-consistem camera control for realistic text-to- eext-to-image synthesis. IEEE/CVF Conference Computer Vision and Pattern Recognition, 2025b.
Chenyu Zhang, Daniil Cherniavskii, Andrii Zadaianchuk, Antonios Tragoudaras, Antonios Vozikis, Thijmen Nijdam, Derck W. E. Prinzhorn, Mark Bodracska, Nicu Sebe, and Efstratios Gavves. Morpheus: Benchmarking physical reasoning of video generative models with real physical experiments. arXiv preprint arXiv: 2025a.
Ke Zhang, Cihan Xiao, Yiqun Mei, Jiacong Xu, and Vishal M. Patel. Think before you diffuse: Lims-guided physics-aware video generation. arXiv preprint arXiv:2505.2165: 2025b.
Lvmin Zhang, Anyi Rao, and Maneesh Agrawala. Adding conditional control to text-to-image diffusion models. In JEEE International Conference Computer Vision, 2023.
Tianyuan Zhang, Hong-Xing Xing Yu, Rundi Wu, Brandon Feng, Changxi Zheng, Noah Snavely, Jiajun Wu, and William T. Freeman. PhysDreamer: Physics-based interaction with 3d objects via video generation. European Conference on Computer Vision, 2024.
Xiangdong Zhang, Jiaqi Liao, Shaofeng Zhang, Fanqing Meng, Xiangpeng Wan, Junchi Yan, and Yu Cheng. Vidcorepa: Learning physics for video generation through relational alignment with foundation models. arXiv preprint arXiv:2505. 23656, 2025c.
Zhenghao Zhang, Junchao
Junchao Liao, Menghao Li, Zuozhuo Dai, Bingxue Qiu, Siyu Zhu, Long Qin, and Weizhi Wang. Tora: Trajectory-orientedd diffusion transformer for video generation. In IEFF/CVF Conference on Computer Vision and Pattern Recognition, 2025d.
Yaofeng Desmond Zhong and Naomi Ehrich Leonard. Unsupervised learning of lagrangian dynamics from images for prediction and control. Advances Neural Information Processing Systems, 2020.
16

---

Appendix
A APPENDIX INTRODUCTION
This appendix provides additional discussions and details on the plysics-clean video simulator (Section [), Neural Newtonian Dynamics network design and prediction accuracy analysis (Section C. evaluation details (Section D). more visual results (Section ED, and a Q & A section (Section F)
To illustrate the continuity and effects of physical coherence and controllability, we recommend that readers view the Videos included in the Project Page httpss//wwaanppcccccco////vttttt
B MORE DETAILS OF THE DATA SIMULATOR
For each type of motion, we construct a physics-clean dataset for training the NND model. Our simulator is built upon physical principles and renders videos with time stamps. The simulator supports multi-parameter control, including initial position, velocity, orientation, angular veloc- ity, world settings (world size, friction coefficient, acceleration/decellratio coefficient, damping coefficient, pivot point), and object properties (size, shape). Representative samples are shown Figure. [6] while the complete simulator code and additional video examples are provided at http://hhwbb.ooomomaaaaaayyyyy/e
0.00
0.16
0.31
0.46
0.61
0.76
0.91
Parabolic Motion
Parabolic Motion with Rotation
Damped Oscillation
3D Motion
Figure 6: Sample physics-clean videos generated by our simulator.
C MORE DETAILS OF NEURAL NEWTONIAN DYNAMICS
C.I NEURAL NEWTONIAN DYNAMICS NETWORK
In Algorithm|] I we present the detailed architecture of the Neural Newtonian Dynamics network. We model the most salient dynamics using a physics-driven linear Neural ODEs, and augment it with learnalle ML. to capture nonlinear and unknown dynamics.
17

---

Algorithm Neural Newtonian Dynamics Network Architecture
Require: Initial physical state Zu [at y, Uzi Vy: 0, W, s, L, a), time stamps tu...., tr
Ensure: Future Latent physical states (())
1: Define learnable parameters:
Oz: bz, Cx), (ay. by: Cy) for linear 2nd-order dynamics of (z, y)
(,) for linearized pendulum or circular motion (6, w)
(Ks, 93), (av, Bt), (oaa for Ist-order dynamics of (s, ()
Residual scale
2: Define residual MLP: ResMLP: IR IR (initialized to 0)
3: for each time do
4: Split Z [r, M/ Uz, vy, w., 8, a,
5: Compute linear dynamics:
ar lin ax + Us +
a
lin
ayy + by Cy
d0 / dt
tn dl - L)0 Fw
ds lin dl = a, + Baz allnn /dl and + 3r, da /dl 0aal + Bo
6: Compute residual correction:
[a any du ds dl da tanh(ResMLP(ZZ)
7: Update derivatives:
dz
Lin [Ugy Uy Lin + dx By
+ a d0/dt, dt, dn + dutes ddlin + ds dilm + dites da dd
dt
8: end for
9: Integrate ODE by odeint obtain Ztt) over to,...,
C.2 ACCURACY OF NEURAL NEWTONIAN DYNAMICS PREDICTIONS
Figure. to Figure. 18 show the predictions of the trained Neural Newtonian Dynamics (NND) model for each type of motion. Given the initial physical state Zo, the model's predicted physical states closely follow the ground truth over time.
Pred
Pred
Pred
5
0.0
0.2
0.4
0.5
0.0
0.2
04
0.0
0.8
0.0
0,2
044
0.6
0,5
vy
thets
omoga
Pred
Pred
Pred
0.0
0.0
0.0
Pred
Pred
Pred
68
80
Figure 7: Comparison of NND predictions and ground truth for uniform motion.
18

---

VX
Pred
Pred
Prod
0.9
0.0
4y
thete
omega
Pred
Pred
Pred
7.0
7.0
2.0
1.
10-
10
0.0
0.0
0.0
Figure 8: Comparison of NND predictions and ground truth for acceleration.
VX
15-
Pred
15-
Pred
Pred
0.0
0.0
0.0
vy
theta
omega
Prod
Pred
Pred
1.8
1.
1.0
50
80
6.0
Figure 9: Comparison of NND predictions and ground truth for deceleration.
Pred
Pred
Pred
0.0
0.2
0.4
04
0.8
0.0
2.2
0.4
0.5
6.6
0.0
0.4
0.5
thata
omega
Pred
Pred
Pred
-10
-2-
0.0
1.5
Pred
155
Pred
Pred
1.0
10
3.0
Figure 10: Comparison of NND predictions and ground truth for parabolic motion.
19

---

Pred
Pred
Prod
vy
theta
omega
Pred
Pred
0.2
0.0
0.2
Figure 11: Comparison of NND predictions and ground truth for 3D motion.
10.0
10.0
10:0
Pred
Pred
7.1
Pred
0.2
thata
omaga
Pred
Pred
Pred
2.0
2.0
2.2
1.0
1.0
1.2
0.0
0.0
0.0
Figure 12: Comparison of NND predictions and ground truth for slope sliding.
Pred
Pred
Pred
0.0
0.2
04
0.6
0.5
0.2
02
044
0.6
0.8
0.0
9.2
0.4
0.6
0,8
vy
theta
amego
Pred
Pred
Pred
2-
Pred
Pred
Pred
Figure 13: Comparison of NND predictions and ground truth for circular motion.
20

---

28.0
20.5
Prod
Pred
Pred
2.5
0.0
theta
omegga
Pred
Pred
7.0
10-
7.4
GT
1.0
1.0
0.5
0.2
0.2
04
0.0
Figure 14: Comparison of NND predictions and ground truth for rotation.
15-
Fred
15
Pred
Pred
theta
omega
Pred
Prod
Prod
-15
2.0
15-
10.
1.0
3.0
Figure 15: Comparison of NND predictions and ground truth for parabolic motion with rotation.
Pred
Pred
Pred
0.0
0.2
04
0.6
0.5
0.2
02
044
0.6
0.8
0.0
9.2
0.4
0.6
0.5
vy
theta
amego
Pred
Pred
Pred
Pred
Pred
Pred
Figure 16: Comparison of NND predictions and ground truth for damped oscillation.
21

---

Pred
10
Pred
Pred
vy
theta
omega
Pred
Pred
Pred
0.2
2-
0.0
0.2
Figure 17: Comparison of NND predictions and ground truth for size changing.
25-
Pred
10
Pred
Pred
vy
theta
amaggo
Pred
Pred
Pred
02
1-
0.0
Figure 18: Comparison of NND predictions and ground truth for deformation.
22

---

D EVALUATION DETAILS
D.1 PHYSICAL INVARIANCE SCORE
The Physical Invariance Score (PIS) as described in equation indicates whether a certain quantity C' remains invariant over time. If the laws of physics are replicated perfectly, C' remains constant, and C.-1 0 = PIS + 1. For each type of motion, suitable C' should be selected.
Uniform Motion: An object prompted to travel horizontally uniform velocity in each scene. Therefore, we select the horizontal velocity
the invariant feature.
Under these motions, we check the object obeys the law of accelerating (or decelerating) at constant rate. The guidance parameters and prompts specify horizontal motion. Therefore, we set C' = ag for acceleration, and C -a for deceleration.
Uniform Acceleration and Deceleration:
Parabolic Motion: Under this motion, there is no horizontal acceleration. Therefore is expected to be constant. Additionally, the vertical acceleration dy due to gravity should be constant.
The prompt guides object travel towards the observer, creating the effect of increasing object dimensions, while also having 2D motion. We approximate this effect as having a constant vertical velocity Vyy. and constant increment rate in the long-axis of the object Ál.
3D Motion:
Slope Sliding: An object is prompted to slide down a constant slope. Assuming negligible effects from friction,
expect accelerations ary ay to be constant.
Circular Motion: Objects are guided to orbit in circular path, we assume the angular velocity about the orbital center is constant.
When objects are prompted to "spin" or "rotate about their axes", we assume that the brief duration of the video, that they rotate in constant angular velocity w.
Rotation:
Parabolic Motion with Rotation: Videos under this category should describe a superposition of a projectile motion under gravity, and a rotation about the object's axis. Therefore the metrics used in these two motions (Uz; Gy. w) are used for C.
Damped Oscillation is simulated through various instances of pendulums, hinged at the top. We assume small angles (0) for the stride. This leads to the vertical force varying with cos(O), and we assume be constant. Thereby we C ay.
Size Changing:
We prompt videos where it's natural to increase an object's overall size, while maintaining it's aspect (e.g. inflating balloon). Assuming a constant rate of inflation,
C' = Ar; the rate of increasing the radius of the object.
Objects under this category should expand. stretch, or spread-out over time. The aspect ratios may change. (e.g., the spread of thick viscous liquid). We assume that the object increases its dimensions a constant rate, and track this rate along it's long axis Al.
Deformation:
After selecting C' for motion type, C.. C₁ is calculated for every video. This lends to PIS score per video. The final score reported in table aahlewshow the median PIS score after generating 12 different videos for each motion. In our case, temporal derivatives are formed from successive frames with A/ 1/(FPS value), then mapped physical units using a constant of 0.00625 meters per pixel. Each video is preprocessed with 5-frame moving-average filter reduce noise in derivative estimates.
Some feature assumptions are idealizations that may not hold in the real world. Accordingly, report Reference PIS computed directly from the guidance mask used to drive the video generation. For example, in 3D motion. Al need not be constant, expecting C, 0 = PIS 1 is unrealistic. The mask- based PIS instead, serves as a practical upperbound- the score that would be achieved if the generator perfectly followed the guidance mask (which itself has C, 0).
23

---

Table 3: Samples of Testing Prompts.
Motion
Testing Prompts
Uniform Motion
A small metal cube sliding steadily along smooth laboratory bench, reflections visible on the surface, scattered tools in the background, captured from fixed side camera.
A red rubber ball rolling constant speed on a polished wooden floor, pulled by thin string, with scattered papers and books in the background, observed from fixed side camera.
Acceleration
A red sedan accelerating in a straight line on a clean highway, the road flat and clear, with only a palc sky and distant horizon in the background, captured from fixed roadside camera.
A black off-road SUV accelerating in straight line on sandy terrain. with continuous sand dunes in the background, a few white clouds in the sky, sunlight slanting, kicking up fine sand particles,
viewed from a stationary side-angle camera.
Deceleration
A yellow bus decelerates in straight line in front of traffic light city street, with pedestrians crossing nearby, and the wet road reflecting the sky, captured by fixed side-view camera.
A red coach brakes and decelerates in a straight line on a highway, with road signs and streetlights nearhy and the city skyline visible in the distance, captured by fixed dide-viww camera.
Parabolic Motion
A golf ball is hit at an angle with initial speed. The camera captures its parabolic trajectory from the side. The scene takes place on a sunny golf course with manicured fairways, sand bunkers, and distant trees, adding depth and realism.
A volleyball is served at an angle, captured from the side by stationary camera. The scene is set on outdoor beach volleyball court, with sand texture, net, and distant palm trees in view.
3D Motion
A fighter jet accelerates slowly from the distance along the runway towards the camera, hangars and runway lights visible in the background, captured from a fixed oblique side camera.
A cardboard box slides from the distance along warehouse floor towards the camera, shelves and crates visible in the background, captured from a fixed oblique side camera.
Slope Sliding
A hardcover book accelerating down a carpeted inclined board a classroom, chalkboard and desks in the background, captured from a fixed side camera parallel to the ramp.
A small metal cube sliding down laboratory ramp, shiny reflections on its surface, scattered tools and wires in the background, captured from a fixed side camera parallel to the ramp.
Circular Motion
A tiny moon orbits a gas giant along smooth, circular path. The top-down view shows the consistent motion without motion trails.
A comet with a glowing tail orbits distant star along a stable circular path. A top-down perspective emphasizes the symmetrical orbit and the stationary central star.
Rotation
A metal rod spinning on a concrete floor, faint scratches and dust visible, captured from a fixed top-down camera.
A wooden dowel rotating gently tiled kitchen floor, soft shadows from ceiling lights, viewed from a stationary overhead camera.
Parabola +Rotation
A pen is thrown at an angle, rotating as it falls. Captured from a side camera, the notebook and desk provide background details and depth.
A thin cylindrical rod gently tossed, rotating along its long axis, fixed side camera, realistic reflections, ground shadows visible, subtle motion blur.
Damped Oscillation
A small decorative hell hanging from a fine chain. The fixed camera captures realistic material and shadows.
A realistic pendulum with a spherical bob swinging from fixed pivot. The fixed camera captures the entire motion.
Size Changing
A red helium balloon gradually inflating in a sunny park, children playing in the background, trees casting soft shadows, captured from a stationary side camera.
A transparent water balloon expanding in laboratory, scientific instruments and glassware around, bright fluorescent lights overhead, captured from a fixed top-down camera.
Deformation
A long strip of yogurt slowly spreads into smooth layer, captured by fixed overhead camera.
A long strip of jelly gradually deforms and flattens on plate, captured by a fixed overhead camera.
24

---

E MORE VISUAL RESULTS
E.
MORE GENERAL COMPARISON RESULTS
From Figure. T9 to Figure. 30
provide additional visual results and comparisons with other
methods.
blue glass marble moving steady speed along marble countertop, reflections and nearby kitchen utensils visible, captured from fixed side camera.
Sora
Veo3
CogVideoX-5B
Wan2.2
Phy
Ours
Uniform Motion
Figure 19: Visual comparisons uniform motion.
dark gray sports accelerating in straight line on empty road at dusk, wet road reflecting light, orange sunset casting colors on the car, slight wind rustling the grass along the roadside, viewed from stationary side-angle camera.
Sora
Vea3
C5gVideoX-5B
Wan2.2
PhysT2V
Ours
Acceleration
Figure 20: Visual comparisons acceleration.
25

---

A red bowling ball rolls in a straight line on wooden lane and gradually decelerates, slowing down before reaching the pins, with the polished lane surface reflecting light, captured by a fixed side-view camera.
Sora
Veo3
CogVideoX-5B
Wan2.2
PhyT2V
Ours
Deceleration
Figure 21: Visual comparisons on deceleration.
A baseball thrown at an angle with an initial speed. The camera captures its flight from the side, rising and then descending. The scene is set on baseball field, with dirt infield and green outfield grass, and stadium seats faintly visible in the background.
Sora
Veo3
CogVideoX-5B
Wan2.2
PhyT2V
Parabolic Motion
Figure 22: Visual comparisons parabolic motion.
26

---

cardboard box slides from the distance along warehouse floor towards the camera, shelves and crates visible in the background, captured from fixed oblique side camera.
Sora
Veo.
CogVideoX-SB
Wan2.2
PhyT2V
3D Motion
Figure 23: Visual comparisons on 3D motion.
A ceramic mug accelerating down wooden inclined board, kitchen tiles and shelves in the background, natural daylight streaming through window, captured from fixed side comera paralle! to the romp.
Sora
Veo3
CogVideoX-SB
Wan2.2
PhyT2V
Slope Sliding
Figure 24: Visual comparisons slope sliding.
27

---

comet with glowing tail orbits distant along stable circular path, top- down perspective emphasizes symmetrical orbit and the stationary central star.
Sora
Vea3
Cog VBg
Wan2.2
PhyT2V
Ours
Circular Motion
Figure 25: Visual comparisons on circular motion.
A metal rod spinning concrete floor, faint scratches and dust visible, captured from fixed top-down camera.
Sora
Veo3
VogVideoX-5B
Wan2.2
PhyT2V
Ours
Rotation
Figure 26: Visual comparisons rotation.
28

---

A paintbrush is thrown at ongle, rotating while folling. Coptured from side comera, the artist's desk with palette and papers background.
Sora
Ve03
CogVideoX-SB
Wan2.2
PhyT2V
Ours
Parabolic Motion with Rotation
Figure 27: Visual comparisons on parabolic motion with rotation.
small decorative bell hanging from a fine chain. The fixed comera coptures realistic material and shadows.
Soru
Vea3
CogVideoX-5B
Wan2.2
PhyT2V
Ours
m
M
Damped Oscillation
Figure 28: Visual comparisons damped oscillation.
29

---

metallic silver balloon gradually expanding in festive hall, fairy lights and streamers in the background, colorful reflections on the floor, captured from stationary overhead camera.
Veo3
CogVideoX-SB
Wan2.2
PhyT2V
Ours
Size Changing
Figure 29; Visual comparisons on size changing.
piece of soft dough evenly flattened on workbench, captured by a fixed overhead comera.
Sora
Vea3
CogVideoX-SB
Wan2.2
PhyT2V
Ours
Deformation
Figure 30: Visual comparisons on deformation.
30

---

E.2
MORE PARAMETER CONTROLLABILITY COMPARISON RESULTS
Figure.
31 and Figure.
32
illustrate the physical parameter control capability of NewtonGen.
coconut
soccer
orange
tennis
football
Same Initial Location [x-2.0, y-1.0].. Initial Speed [v.-5.0, -055
Figure 31: Given the same initial physical states but different scene descriptions, NewtonGen can generate diverse scenes with consistent motion.
Initial locution [x-2.0, 110.0) Initial speed 55,5,, 112)
Size [0.4]
Initial location yx9.00 Initial speed [o, -.00, n, -0.0]
Size [0.6]
A golf ball is hit at an angle with an initial speed. The comera captures its parabolic trajectory from side. The takes place a sunny golf course with manicured fairways, sand bunkers, and distant trees, adding depth and realism.
Initial location [x-2.0, 100.0 Initial speed [5,-5.0, 0088
Size [0.3]
Initial Xx-8.0, y-800
Initial speed 10.00.0 v,
Size [0.6]
baseball thrown an angle an speed. camera captures its flight from the side, rising ond then descending. The scene set baseball field, with dirt infield and green outfield grass, and stadium seats faintly visible the background.
Figure 32: Given different initial physical states but the same scene description, NewtonGen can generate the corresponding motions.
31

---

F QUESTIONS AND ANSWERS
Why is a NND (neural ODE) necessary to model/forecast Newtonian motion, and why not train a simple neural network predict the coefficients of parabola (for the parabolic trajectory)?
Question 1:
Answer 1: Our NND learns the underlying dynamics behind different systems, rather than merely fitting simple kinematics (trajectories) from data. They also provide unified framework capable of representing diverse types of dynamics.
For some motions, the underlying physical dynamics equations are already known, so why do we still need neural networks learn dynamics?
Question 2:
Answer 2: Many complex real-world motions are difficult capture with simple physical formulas. For example, when rotation, parabolic motion, and even deformation occur simultaneously, it is challenging for humans to explicitly formulate the underlying physical laws. In contrast, our ODE model directly learns the dynamics from video data.
Question 3: Does your physical control model compromise the generative model's original physical effects or performance (e.g., shadows)?
Answer Empirically, we have not observed any degradation in physical plausibility, such as shadow dynamics, after applying control. Our framework is training-free in the second stage, it injects physically consistent optical flow as a control condition only during inference, which preserves the model's 's original capabilities.
Question 4: Can NewtonGen (NND) handle video generation tasks involving collisions, rebounds, or explosions?
Answer Currently, NewtonGen (NND) does not support such cases, as it designed for continuous dynamics. These tasks would require additional event-based ODEs or hard-coded implementations.
Question 5: Can NewtonGen generate the motions of multiple objects' motion in a video?
Answer 5: Yes. NND can independently predict the physical states of multiple objects and then feed them into the notion-controlling video generator. The main bottleneck for video quality lies in the latter.
Question 6:
Why choose -Go-with-the-Fii Flow" instead of other motion control models the base model for the second-stage stage video generation?
Answer 6: Other models often control motion through trajectories or bounding boxes, which makes it difficult for them to handle tasks involving deformation rotation. In contrast, Go-with-the-Flow based on optical flow control and thus has the potential to address such challenges.
Question 7: NND fast during training and inference? Answer 7: Yes. NND is trained in the latent space rather than directly on videos, and its learnable parameters are concentrated in lightweight three-layer MLP. As a result, inference can achieve real-time or faster speeds.
32


Reviews:
[{"review_id": "aCYz5nBgZf", "reviewer": "Reviewer_tuJM", "summary": "The paper presents a hybrid generative framework that combines physical dynamics with data driven video generation. Its main contribution is Neural Newtonian Dynamics (NND), a learnable ODE based module that predicts latent physical states and enforces physical constraints during text to video synthesis. This method seeks to close the gap between visual realism and physical consistency in generative video models.", "questions": "1. A direct comparison with a traditional physical simulation following the same pipeline would be valuable. Currently, it is unclear where the NND demonstrates a distinct advantage, is it in computational efficiency, scalability to data (which physics simulations don't have), or improved fidelity compared to directly employing a physics engine? Considering that the NND itself is initially trained on physical simulations, clarifying where it provides genuine value beyond simulation would help better position the contribution.", "limitations": "", "rating": 6, "confidence": 3, "soundness": 2, "presentation": 3, "contribution": 3, "strengths": "1. The conceptual direction of embedding explicit dynamical models into video generation is strong and timely and the overall structure of the framework is clean and modular, separating physical dynamics reasoning from visual synthesis. \n\n2. The experimental evaluation is broad and convincing for their claims, with meaningful metrics and visualizations (even though generated videos are slightly on the short side). Comparisons with “Physical simulation then Generation” methods would add context, the current analysis supports the claims.", "weaknesses": "1. The physical state encoder relies heavily on SAM2 and OpenCV to detect and extract geometric and kinematic properties from frames. While suitable for simple rigid-body scenarios, it is unclear how this mechanism could extend to richer physical properties such as elasticity, reflection, lighting, wave propagation, breaking, or liquid dynamics, leaving aside the potential extension of NND to multi-object scenarios. The current setup appears limited to tracking motion and deformation rather than modeling material or environmental physics, which constrains the framework’s generalization to more complex phenomena.\n\n2. The mapping from physical prompts to initial physical states $Z_0$ is not well-detailed. It remains unclear how the system estimates or derives physical parameters like initial position, velocity, or angular speed from the prompt, and how sensitive the final generation is to inaccuracies in these estimates.\n\n3. As indicated in Section 2, \"Physical Simulation then Generation\", \"... generative models themselves lack any inherent physical reasoing or simulation capability: users must predefine the physical simulation parameters and rules for each scenario, and these setings cannot readily generalize to other context or different physical laws\". I also fail to see how NND generalizes in this case, which I believe is also addmited in Appendix F, Question 4.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "9vVulWQOT8", "reviewer": "Reviewer_r8si", "summary": "This paper presents NewtonGen, a framework aimed at solving the lack of physical consistency and controllability in text-to-video (T2V) generation. The authors propose a framework that integrates a data-driven T2V model with learnable physical principles. Its core is the Neural Newtonian Dynamics (NND) module, a physics-informed neural ODE that models a 9-dimensional latent physical state (e.g., position, velocity, rotation, size). The NND is first trained on \"physics-clean\" synthetic data to learn these dynamics. During inference, this trained NND predicts a sequence of physical states from user-defined initial conditions, which are then converted to optical flow to guide a motion-controlled T2V generator. Experiments across 12 motion types demonstrate quantitative improvements in physical consistency and precise parameter control. The reviewer regards NewtonGen as a novel and well-motivated two-stage framework for injecting physical realism and controllability into video generation. The core contribution, the NND module, is noted as being cleverly designed, as it combines linear ODEs with a residual MLP to learn complex dynamics from synthetic data efficiently. The reviewer acknowledges that the framework shows significant quantitative improvements on the Physical Invariance Score (PIS) metric and possesses excellent qualitative controllability. However, the reviewer raises concerns about the framework's reliance on synthetic \"physics-clean\" data, which poses significant questions about the sim-to-real gap. Furthermore, the reviewer points out that the method is explicitly limited to continuous dynamics, excluding collisions or discrete interactions, which narrows the scope of its \"Newtonian\" claims", "questions": "1. The authors should explicitly state in the main paper that the framework's current application is generation from parameters, not prediction from real video. The \"continuous dynamics only\" limitation should also be candidly discussed in the main text, with potential extensions for handling discrete events like collisions.\n\n2. Add a \"Perfect Oracle\" Baseline: It is suggested to add a \"perfect simulator\" baseline to Table 1, where the ground-truth simulator optical flow is fed directly into the generator. This would quantify the performance gap between the learned NND and a \"perfect\" physics oracle.\n\n3. The authors should report standard deviations for the PIS scores in Table 1, rather than only the median, to provide a clearer measure of generation stability.", "limitations": "", "rating": 6, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 4, "strengths": "1. The NND module is cleverly designed, combining physics-informed linear neural ODEs with a residual MLP. This allows it to flexibly learn a wide range of dynamical systems, from simple to complex, with high data efficiency.\n\n2. The framework achieves exceptional physical consistency across multiple motion types. In quantitative evaluations using the \"Physical Invariance Score\" (PIS) metric, NewtonGen's results significantly outperform all SOTA baselines and are very close to the simulated ground truth.\n\n3. The model achieves precise control over physical parameters, addressing a key weakness in existing T2V models. Experiments demonstrate it can generate motions that accurately correspond to user-specified initial physical states (e.g., position, velocity, size).", "weaknesses": "1. Sim-to-Real Gap: The NND is trained exclusively on synthetic \"physics-clean\" data. The paper does not demonstrate or discuss how the framework would handle real-world, noisy videos, suggesting its current application may be limited to generation from scratch rather than editing or predicting existing real-world videos.\n\n2. Limited Scope of Dynamics: The framework explicitly excludes discrete events such as collisions, rebounds, or multi-object interactions.\n\n3. Approximation of 3D Motion: The paper claims to handle 3D motion, but the method is \"equivalently realized through the combination of position and size control\". This appears to be a 2.5D approximation (i.e., objects scaling larger as they move closer) rather than a true 3D representation capable of handling complex perspective rotations.\n\n4. Misaligned Baseline Comparison: The paper compares NND against general-purpose T2V models like Sora and Veo3. These baselines were not optimized for fine-grained physical control, making the comparison somewhat misaligned.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "9jX0T4kXxD", "reviewer": "Reviewer_UcEn", "summary": "This paper addresses the core bottlenecks of physical consistency and parameter controllability in state-of-the-art text-to-video (T2V) generation. Existing models rely on appearance-level motion memorization without understanding underlying physical laws, leading to unrealistic motions (e.g., upward-falling objects) and poor generalization. To solve this, the authors propose NewtonGen, a hybrid framework integrating data-driven T2V generation with physics-informed Neural Newtonian Dynamics (NND). NND leverages unified neural ordinary differential equations (ODEs) combined with a residual MLP to learn latent dynamics from small-scale \"physics-clean\" data (generated via a custom simulator). It models 9-dimensional latent physical states (position, velocity, rotation, size, etc.) to predict physics-consistent trajectories. During inference, NND’s state predictions and scene prompts guide a motion-controlled T2V model (Go-with-the-Flow) to generate videos.", "questions": "Please follow weakness.", "limitations": "", "rating": 8, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "1. Directly addresses two critical limitations of state-of-the-art text-to-video (T2V) models—physical inconsistency (e.g., upward-falling objects, abrupt velocity changes) and lack of parameter controllability—which scaling alone cannot resolve. This fills a key research gap, as current models only learn motion distributions from visual appearance rather than underlying physical laws.\n\n2. NND’s design (physics-informed linear neural ODEs + residual MLP) unifies diverse Newtonian motions (12 types tested, including uniform velocity, parabolic motion, rotation, deformation, and 3D motion) into a single framework. It efficiently learns latent dynamics from a small amount of \"physics-clean\" data, avoiding over-reliance on large-scale datasets.\n\n3. Compares against 5 representative baselines (closed-source: Sora, Veo3; open-source: CogVideoX-5B, Wan2.2; physics-based: PhysT2V) across 12 motion types, ensuring fairness by standardizing generation settings.\n\n4. Separates physical dynamics reasoning (NND, for state prediction) from video generation (motion-controlled model like Go-with-the-Flow). This design retains the visual quality of existing T2V models while injecting physical constraints and enabling easy integration with other T2V frameworks.", "weaknesses": "1. Relies on continuous ODEs and thus cannot handle discrete physical events (e.g., collisions, rebounds, explosions). This restricts its applicability to real-world scenarios involving interactive or abrupt physical interactions.\n2.  NND is trained exclusively on data from a custom Python physics simulator (designed to avoid motion blur, noise, or background distractions). Performance on real-world videos (with clutter, occlusions, or motion blur) is unproven, as no experiments on real datasets are reported.\n3. While NewtonGen outperforms baselines on physical consistency (PIS), it does not quantitatively evaluate visual quality (e.g., texture realism, shadow dynamics, background coherence) against competitors. This makes it unclear if physical consistency comes at the cost of visual fidelity.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "6JQjC4RL9Q", "reviewer": "Reviewer_CpA9", "summary": "The paper presents a video generation pipeline, that consists on two models: 1) single-object motion generator; and 2) (frozen) foundational video diffusion model. The motion generator is parametrized to take the initial physical state of the object (position, rotation, size, velocity, etc) and predict the future ones. It is trained on a small synthetic dataset. The predicted states are then converted to an optical flow, and the video generation backbone is conditioned to generate the video based on it using the go-with-the-flow noise tweaking method. The pipeline is converted with existing open and closed source baselines and shown to generate physically more plausible . The evaluation is performed using the Physical Invariance Score.", "questions": "- It is not well described how the object attributes are computed using the \"morphological analysis and OpenCV-based tools\" (L295). Would be good to include more details for completeness in the paper (without requiring a reader to study the code).\n- Type on L161: \"equation, This\"\n- \"most common physical motions in daily life (e.g., flying balls)\": I disagree with this statement, and believe that the most common physical motion in daily life are humans moving and talking.", "limitations": "", "rating": 2, "confidence": 4, "soundness": 2, "presentation": 3, "contribution": 2, "strengths": "- The paper targets a very important research problem, directing more community's focus towards it.\n- The writing and exposition is very clear, and also provides an interesting overview of the past literature\n- The paper releases the source code, which is a good step toward ensuring reproducibility and transparency.\n- The method compares to top-notch closed-source video generation baselines which makes the results more trustworthy. I even checked the Veo3 generation quality on the prompt \"a baseball ball flying from left to right\", and confirm that it fails to understand the prompt even for such a simple scenario in ~50% of the cases.", "weaknesses": "- I believe the approach is overly simplistic and inherently unscalable. It would not help to generate complex realistic motions (e.g., a person juggling jellies, a cup breaking against a wall, a squirrel jumping on a thin tree branch, human dancing, etc.), which is the core physical challenge. In its current form, it is just a way to generate simple optical flow for a single object motion. Even single-object deformations (e.g., see example with the dough) are not generated realistically. The authors should clearly explain in detail how the approach could be scaled to very difficult motion cases described above for which it feels infeasible to 1) construct the pre-training dataset, and 2) create the initial physical state. I think that for simple rigid motion, it's not that modern video generators cannot render it, they just fail to understand the prompt.\n- If one treats the paper not as a general-purpose pipeline which is improving physics in video generators, but as an \"optical flow conditioning\" paper, then there is not much novelty in it. There are dozens of work that condition video models on motion, and what the current pipeline is doing is automating optical flow generation (in an overly simplistic way) instead of asking a human to provide manual input. But this could be automated in a simpler ways without the need to train a model (e.g., by asking LLMs to generate the trajectories, either directly or writing a trajectory generation code).", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (Poster)