Force Prompting: Video Generation Models Can Learn And Generalize Physics-based Control Signals
Authors: Nate Gillman
Charles Herrmann
Michael Freeman
Daksh Aggarwal
Evan Luo
Deqing Sun
Chen Sun
Venue: NeurIPS 2025 poster ()
Keywords: controllable video generation
physical forces
synthetic data
Primary area: deep_learning
TL;DR: Video generalization models can learn physics-based controls from synthetic data and generalize this control across diverse geometries, settings, and materials.

Abstract: Recent advances in video generation models have sparked interest in world models capable of simulating realistic environments.
While navigation has been well-explored, physically meaningful interactions that mimic real-world forces remain largely understudied. 
In this work, we investigate using physical forces as a control signal for video generation and propose force prompts which enable users to interact with images through both localized point forces, such as poking a plant, and global wind force fields, such as wind blowing on fabric. We demonstrate that these force prompts can enable videos to respond realistically to physical control signals by leveraging the physical prior in the original pretrained model, without using any 3D asset or physics simulator at inference. The primary challenge of force prompting is the difficulty in obtaining high quality paired force-video training data, both in the real world due to the difficulty of obtaining force signals, and in synthetic data due to limitations in the visual quality and domain diversity of physics simulators. Our key finding is that video generation models can *generalize* remarkably well when adapted to follow physical force conditioning from videos synthesized by Blender, even with limited demonstrations of few objects (e.g., flying flags, rolling balls, etc.). Our method can generate videos which simulate forces across diverse geometries, settings, and materials. We also try to understand the source of this generalization and perform ablations on the training data that reveal two key elements: visual diversity and the use of specific text keywords during training. Our approach is trained on only around 15k training examples for a single day on four A100 GPUs, and outperforms existing methods on force adherence and physics realism, bringing world models closer to real-world physics interactions. All datasets, code, and model weights will be open-sourced. Video examples can be found at https://sites.google.com/view/force-prompting-neurips2025

Force Prompting: Video Generation Models Can Learn and Generalize Physics-based Control Signals
Nate Gillman Brown University
Charles Herrmann' Google DeepMind
Michael Freeman Brown University
Daksh Aggarwal Brown University
Evan Luo Brown University
Deqing Sun Google DeepMind
Chen Sun Brown University
Abstract
Recent advances in video generation models have sparked interest world models capable of simulating realistic environments. While navigation has been well- explored, physically meaningful interactions that mimic real-world forces remain largely understudied. In this work, we investigate using physical forces control signal for video generation and propose force prompts which enable users interact with images through both localized point forces, such as poking plant, and global wind force fields, such as wind blowing on fabric. We demonstrate that these force prompts can enable videos to respond realistically to physical control signals by leveraging the visual and motion prior the original pretrained model, without using any 3D asset physics simulator at inference. The primary challenge force prompting is the difficulty in obtaining high quality paired vvido training data, both in the real world due to the difficulty of obtaining force signals, and in synthetic data due to limitations in the visual quality and domain diversity physics simulators. Our key finding is that video generation models can generalize remarkably well when adapted to follow physical force conditioning from videos synthesized by Blender, even with limited demonstrations of few objects (e.g., flying flags, rolling balls). Our method can generate videos which simulate forces across diverse geometries, settings, and materials. We also try understand the source of this generalization and perform ablations on the training data that reveal key elements: visual diversity and the use of specific text keywords during training. Our approach is trained on only around 15k training examples for single day on four A100 GPUs, and outperforms existing methods on force adherence and physics realism, bringing world models closer to real-world physics interactions. We release datasets, code, model weights, and interactive video demos our project page, https://force-prmmiingggg wyromptingggithbie
1 Introduction
Humans develop an intuitive understanding of how objects respond to forces since infancy (Wilkening and Cacchione 2010: Ullman et al.) 2017): a gentle poke causes a plant to sway, while a breeze creates rippling patterns across fabric. Do video generation models, which encode powerful visual and motion priors through internet-scale pretraining, possess similar level of intuitive physics understanding? And if so, how to elicit their capabilities to interact with force inputs? A positive answer to these questions would provide more flexible and expressive interface for video content creation, enable interactive video generation with user input (e.g., generating a video game), and eventually lead to intuitive world model for intelligent agents to plan and make decisions with.
"Equal advising. Correspondence to: nate gillman brown.cdu chensun @ brown.cdu 39th Conference on Neural Information Processing Systems (NeurIPS 2025).

---

Train a force conditioned video model with limited synthetic paired data
Global Forces (Wind)
Local Forces (Poke)
Input: Force, Image, Text
Output: Video
loput: Force, Image, Text
Output: Video
2) Video generation model generalizes force conditioning
Generalizes Different Settings Materials
Generalizes Different Objects and Geometries
Generalizes Different Affordances
Hints Mass Understanding
Figure I: Force prompting allows users to apply either global or local forces to objects an image and then generate the resultant video. Despite being trained a limited set of synthetie videos (15k for global force and 23k for local force), we observe significant generalization to different settings, materials, objects, geometries, affordances, and some initial hints at mass understanding. Trajectory visualization or alpha overlay are incorporated to better illustrate movement for some examples.
We introduce Force Prompting, step towards incorporating force-based control (direction and magnitude) into video generation models. We explore two distinct categories of force prompts: local foree prompts, such as instantaneous pokes or pulls applied to specific regions, and global force prompts, such as sustained directional wind that affects the entire scene uniformly. Crucially, as manually collecting foree annotations from natural videos is both costly and difficult, we instead leverage physics simulators (e.g.. Blender) to hand-craft perfectly annotated training data. With our data creation pipeline, we specify collection objects along with the force conditions, and simulate the resulting dynamics to obtain the paired training videos. We hypothesize that such sim2real generalization fcasible because state-of-the-a video generation models already encode strong priors about visual dynamics, and our paired force-video data serves the role of eliciting their understanding of the physics- based control signals.
We implement Force Prompting by introducing additional force control as local or global vector fields on a video generation model (Yang et al., (2024) conditioned on initial frame and text. We also curate an evaluation benchmark of diverse objects and motion types to evaluate global and local force prompts. As illustrated Figure our main finding that despite the synthetic visual appearance and few objects (flying flags and rolling balls) in our training data, video generation models can indeed learn to execute fine-grained force prompts, and exhibit surprisingly strong generalization behavior across diverse settings, object shapes and materials, geometry, and affordances. Through

---

extensive human evaluations, we demonstrate that Force Prompting exhibits superior adherence to physical instruction while maintaining realistic motion and visual quality, when compared to text- conditioned baselines. This validates our hypothesis that synthetic data can teach video generation models intuitive physics and control without damaging their video priors. We further show that simply extrapolating the future by treating forces as local trajectories is insufficient, and our approach significantly outperforms the state-of-the-ar in trajectory-control 2024). Notably, Force Prompting can be trained in approximately a single day on four NVIDIA A1O0 GPUs. We also try to understand the cause of this strong generalization and perform a careful ablation the training data. We find two elements that appear important to generalization: visual diversity in the training data with respect to the control signal and the usage of certain text keywords at training time, which appear to help elicit the understanding of force control signals.
video generation (Geng et al.)
In summary, our main contributions are as follows:
1. We introduce physical forces as conditioning signals for video generation through two models: one for localized point forces and another for global wind forces.
2. We find that video models can execute precise force prompts with broad generalization to different settings, objects, geometries, and affordances despite minimal training data (15K videos) and modest computational resources (one day on four A A100 GPUs). We also attempt to understand the source of this generalization and perform careful ablations on the training data, finding two key elements: visual diversity with respect to the control signal, as well as the usage of text keywords at training time, which appear to help elicit understanding of force control signals.
3. We show that our force-conditioned model has some degree of mass understanding, where the same force can cause lighter object to move farther than heavier one.
We release all datasets, code, and models on our project page, https://fooeeerromttnngggiiioiio
2 Related Works
Video generation: In the last several years, video generation models have made rapid progress in visual quality and realistic dynamics (Singer et al. 2022, Ho et al. 2022, Blatmann et al. 2023
Girdhar et al. 2023 [Ba- Tal et al. 2024 Brooks et al. 2024). ). particular, Sora (Brooks et al. 2024
was one of the first video generation models to demonstrate truly compelling diverse real- world physical phenomena and directly advocated for the future use of using video generators as simulators for the physical world. In the last half year, significant progress has been made by open source models such as CogVine Vigeo ( (Yang et al. 2024) and Wan 2.1 (Wang et al., 2025), even approaching the quality of closed-souree models. While these models act strong video priors, they primarily use text and images as input and lack precise control over general actions or other physical inputs.
Controllable video generation: As video models have rapidly progressed, so too has the accom- panying field of controllability for these models with the majority of work in this domain focusing on either camera control (He et al. 2024 Zheng et al. 2024; Sun et al. (0244) or various paradigms
of motion control (Ynn et al. 2023: Chen al., [023 Wang et al. 2024 Shi et al. 2024, Niu et al.,
2024| Wu et al.] 2024 Geng et al. [2024]] Li et al. [022a]] Namekata et al. 2024 Zhang et al.) 2024b), such as ttag-based, trajectory- based, and optical flow-baser based techniques. Many of the existing motion control models Yin et al. (2023): Chen et al. (2023); Zhang et al. (2024b) require the complete pre-specified trajectory, specifying the location of the pixel on every generated frame. This reliance on full temporal information makes it difficult to use these models for simulation or prediction tasks.
Motion Prompting (Geng et al., 2024), concurrent work, uses spatio-temporally sparse trajectories as a conditioning signal, enabling users to specify motion over a few frames for video extrapolation. While this might superficially resemble force control, crucial distinctions exist. First, global phe- nomena like wind or fluid dynamics are naturally expressed as forces but are difficult or impossible to represent with trajectories. Second, applied forces fundamentally depend on an object's 's mass or material properties- a dependency absent when specifying motion or location (e.g., identical forces induce greater displacement in lighter objects). Third, specifying an object's location across a few frames is not equivalent to an applied force; the same observed motion could result from numerous alternative causes, such as camera movement or internal object changes. We compare to Motion Prompting and demonstrate significantly better adherence to the conditioning force.
3

---

Interactive world models: Paralleling the interest in video generation models, interactive world models (Ha and Schmidhuber 2018) have gained significant attention. Despite extensive research in this area, investigations have predominantly concentrated on video game environments Valevski et al. (2024); Che et al. (2024); [Bruce et al. (2024). While a few contemporary studies have begun exploring real-world applications (e.g., |arr et al. (2024); [Agarwal et al. (2025), ) none explores interactions besides camera control or text. In contrast, our work focuses on interaction through physical forces.
Physical simulators and hybrid approaches: Early work (Davis et al. 2015a b) on generating video based on intuitive forces extracts modal bases of vibrating objects in 21) image space; these works, as well as their modern adaptations (Li et al. 2024b), represented motion as a series of vibrations with different frequencies and intensities, which works well for vibration-like motions but struggles to represent many types of motion, such as linear motion. This led to an alternative research direction
explicitly incorporating physics solvers (Chen et al. 2022, Zhong et al. [2024] Le Cleac' h et al., 2023 Xie et al. 2024 Zhang et al.) [024a] Huang et al. |024]] Liu et al. 2024a]] Lin et al. [024]] et al.
2024p. However, almost all these techniques require the 3D geometry of the scenes. Recent work has focused on combining both physics simulators and generative models, trying to get the best of both worlds: accurate dynamies from the simulator and better appearance from generative models. For example, PhysGen (Liu et al.] 2024b) uses a rigid-body physics solver to model object collisions and then renders these scenes through video generator, and PhysMotion (Tan et al. 2024 uses combination of 3D physics solver and video generation model. However, due to their usage of physics simulators, they are limited in the types of dynamics they can model. In contrast, we explore using the video generation as simulator and do not use physics simulator at inference time. We mention some concurrent works as well: (Li et al., 2025a) also explores the use of simulated videos to finetune generative models. but their focus is on modeling object freefall as opposed to learning physics-basee based control: [i et al. (2025b) explores action-coms coniiiinodd video generation, but their model requires the use of a physies simulator inference time; and Wang* et al. (2025) explores force-conditioned video generation, but their framework requires learning a 3D point cloud trajectory model from synthetic data, and then passing that 4D temporal volume into point cloud-conditioned video generation model.
3 Method: Force Prompting
The goal of Force Prompting is to enable users to interact with images through physical forces. To this end, we explore two distinct force prompts paradigms: a global model that allows users to animate an entire scene with directional wind forces, and local model that enables precise interaction through localized point forces applied to specific objects within the image. Our video generation method takes as input a triple (r, 6, x). where T is the text prompt. & E Rexhxw is the initial frame with height width and channels, and is the physics control signal which represents the force being applied: for the wind force model, this is simply a force vector (magnitude, angle) E IR and for the point force model, this is force vector (magnitude, angle) E IR² along with pixel coordinates (I, v) E R specifying where to apply the force. The goal is to generate a video v E / xexhxx While we train the global force and local force models with different synthetic datasets and encode the force inputs differently for each, both models share identical architectures and training procedures.
3.1 Synthetic training data
To construct our global wind force dataset, we use physics simulator to generate videos of flags waving in the wind. And we construct our local point force dataset in two parts: for the first part, we use a physics simulator to generate videos of a ball rolling across the ground; and for the second part, we use a model (Zhang et al.] (024a)) which integrates 3D Gaussians and a physics simulator to generate videos of plant being poked. We provide more detail below.
Global force dataset: We use Blender construct a dataset of flags waving in response varying wind conditions. In order to generate diverse dataset, we randomize multiple parameters for each video: flag quantity (Unif(1,.... 64]), flag color (from a set of 100), flag positions, camera placement, HDRIs (High Dynamic Range Images), which are 360-degree panoramic images used for lighting and background purposes (selected from 50 options on Polyhaven), wind direction in [0, 360), and wind speed in [0, 1). where O corresponds to no wind, and 1 corresponds to very strong wind. Each video captures the flags' transitions from stationary to wind-affected state, Our training dataset has 15k videos.

---

Frame
Frame
very Frame 10
Frame 40
Magnitude
Prompt: "A cozy, waven swing gently sways back and forth under the shade of a palm tree"
High Magnitude
Same Prompt: "A cozy, swing gently sways back forth under the shade of palm tree"
Figure 2: Visualizing the point force control signal. The magnitude of applied force is proportional to the gaussian blob's 's velocity in the control signal, producing proportionally stronger impulses. Stronger forces (bottom) generate faster-mon moving blobs and correspondingly larger physical responses than gentler forces (top). Note, red line added at the same location in each image for visualization. In our method, we enable the force prompt to dictate the object's trajectory, deliberately excluding such specifics from the text prompt.
The first scenario in our dataset comprises 12k videos of balls, with one of them rolling in response to being pushed by an unseen point-wise force (the force actor is nor rendered), and the other balls remaining stationary. We generate these videos using Blender with randomized parameters: ball quantity [2, 4}), ball textures (soccer balls using Polyhaven mesh [p- = 2/3], or bowling balls modeled as smooth spheres [p = 1/3]). ball colors (from a set of 108), ball positions, camera position, ground textures (from 42 Polyhaven options), target ball selection, force angle in [0, 360), and force magnitude in [0,1]. We assign the bowling ball to be four times the mass of the soccer ball with the goal of teaching the model mass-based dynamics. The second scenario (lIk videos) utilizes PhysDreamer (Zhang et al. (002a)). a generativ--simllan hybrid, and features videos of a carnation swaying back and forth in response to being poked by an unseen force. We generate these videos with randomized camera position, contact points, force angles, and magnitudes. We use mixed dataset with the goal of teaching the model that a point force can result in both simple linear motion, and complex oscillatory dynamics. depending on what type of object the force is applied to. In both scenarios. the force magnitude corresponds gentle poke, and the force magnitude 1 corresponds to a much stronger poke.
Local force dataset:
For both datasets, project forces from 3D space parameters. This transformation maps force vectors and object positions from the physical world coordinate system screen coordinates, allowing us model forces within the image frame. We generate detailed text prompts using the GPT-4o API, creating a unique descriptions for each HRDI background and ground texture, plus single shared prompt for all PhysDreamer carnation videos.
the 2D pixel plane using the camera's
3.2 Local and Global Force Prompts
As the wind force is applied globally, and the point force is applied locally, propose two different force encoding strategies.
Encoding strategy, global force: The wind force control signal is parameterized by a force F E [0, 1] and an angle [0, 360). The goal is to develop a tensor representation for the physics prompt æ. which we denote by से E / xx/xexxxr Here, f 49 is the number of frames, c 3 the number of color channels, and = 480 and w = 720 are the height and width of the generated video. We define the first channel of मे to be 1 + 2 F' [-1,, to be sin f. This defines smooth map Jo. 1] x [0, 360) xxexxx which encodes the angle and magnitude of the wind force field.
1-1,11 1]. the second channel be 0, and the third angle
The point force control signal л specifies a localized force, so it parameterized by the pixel coordinates (xr, y) E (0..... w - 1} x (0...., - 1} in addition to the force magnitude F' E [0, 1] and angle 0 [0, 360). At high level, define the tensor
Encoding strategy, local force:
5

---

Control Over Location, Magnitude, and Direction of Force
Local Force Prompt Generalizes Diverse Physical Applications
a) Linear motion
b) Oscillatory motion
Complex motion
d) Multiple forces
Figure 3: Qualitative results for the Local Force (Poke) model. Top section: For local forces, the control signal can specify both the location, magnitude, and direction of the force. Bottom section: despite the limited training data, the model generalizes to different types of motion. We add blue lines visualize time-lapse of some objects' movements.
representation IR xexxxxx for the control signal π to be a sequence of frames where a Gaussian blob starts at the pixel location (z, y), and then moves in the direction 0 at constant velocity, for
total distance affinely proportional to the force F. Full mathematical details Appendix A..]] This defines a continuous map (O....,
1} x {0..... h 1}} x [0, 1] [0, 360) - R xxex ix which encodes the coordinates where the force applied, as well as the point force magnitude and angle, into tensor representation न. We present visual example of this in Figure 2 In the case of the local force, we note that the displacement the Gaussian blob is nonzero when the force is F 0, as our training dataset convention is that F 0 indicates small force.
We note that force values across the ball rolling and plant poking training videos are not calibrated to any absolute physical scale. Instead, they follow intuitive relative physics where smaller force values (approaching F 0) correspond to gentle pokes resulting in minimal initial displacement, while larger force values produce stronger pokes with correspondingly greater initial displacement. We also wish highlight that our force prompting models are fundamentally different from video generative models with trajectory-base control such as (Zhang et al.) 2024b) Geng et al., (2244). This because the gaussian blob which serves as force indicator for the point force model is generally far away from the pixels that it affects, as demonstrated in the complex oscillatory motion of the swaying flower in Figure 2 Similarly, the wind force control signal under-specifies which points must move to which locations, as that control signal is global and causal.

---

Point Force Model
Linear Motion
Oscillatory Motion
Complex Motion
Force
Real.
Visual
Force
Real.
Visual
Force
Real.
Visual
Adh.
Physics
Qual.
Adh.
Physics
Qual.
Adh.
Physics
Qual.
Text-only, zero-shot
72%
50%
48%
67%
48%
52%
73%
48%
49%
Text-only, fine-tuned
79%
53%
52%
62%
52%
58%
74%
55%
54%
Motion Prompting
91%
93%
100%
89%
76%
99%
86%
76%
98%
Global Force Model
Tethered Motion
Acrodynamic Motion
Fluid Dynamics
Force
Real.
Visual
Force
Real.
Visual
Force
Real.
Visual Qual.
Adh.
Physics
Qual.
Adh.
Physics
Qual.
Adh.
Physics
Text-only, zero-shot
91%
50%
54%
97%
48%
47%
84%
53%
47%
Text-only. fine-tuned
62%
48%
47%
57%
70%
50%
71%
58%
49%
Motion Prompting 93%
82%
100%
90%
75%
100%
90%
80%
95%
Table Comparison to baselines. Top: Local point force model. Bottom: Global wind force model. We present % win rates of our method against baselines in 2AFC human study results (i.e. values above 50% indicate a preference for Force Prompting) for force adherence, realistic physics, and visual quality. We find that none of the other methods provide consistent adherence to the input force.
3.3 Architecture and Training
We build the force prompting models on top of Cog VideoX-SB--22 (Yang et al., 2024). a video generative model which accepts text and initial frame as conditional inputs. This model generates 49-frame videos at 8-fps. In order to integrate force prompt conditioning, we add a ControlNet (hang et al. 2023) which inputs a physics control prompt π. processing through downscaling. encoding. and temporal compression before combining with hidden states via zero convolution. The ControlNet clones the first six transformer layers and fine-tunes them while keeping the base model's transformer layers frozen. We base our implementation on (Karachev and Xu (2025) with modifications to adhere more closely to the original ControlNet design. We train the models on a four 80 GB A100 GPU cluster for 5000 training steps, which takes approximately one day. Training uses an instantaneous batch size per device of 1, with two gradient accumulation steps, for an effective batch size of 8. Full hyperparameter details are listed in Appendix A.
4 Quantitative and Qualitative Results
We propose a benchmark dataset for both force prompting models using images that we curate from Pexels| We conduct a 2AFC human study (N = 10) using Prolific comparing our force prompting model against three baselines on these benchmark datasets.
The first baseline text-only, zero-shot, which uses the original VideoX model and describes the intended force with string and appending it to the end of the original text prompt. Two example prompt string suffixes are "the apple is moved very forcefully, upwards and the left", and "the wind is medium strength, blowing right". The second baseline is text-only. fine-tuned, which has the same ControlNet architecture as our force prompting model, but with zero-tensor control signals, as well as force suffixes added to the end of the text prompts during training. Our third baseline is Motion Prompting (Geng et al. 2024), built on Lumiere (Bar- Tal al. 2024) (run by the authors). It is the only nack-conditioned model that accepts temporally sparse tracks as conditioning signal. We simulate force prompt's 's impulse by tracing push paths from target objects for the first frames. While the model is meant to accept temporally sparse trajectories, 3 frames of trajectory is out of domain for the intended use case of Motion Prompting.
Baseline models:
Human study for local force benchmark: We create a benchmark by curating 63 images from Pexels demonstrating three categories of physical interactions: 1) linear movement patterns (toy car, toy train on straight track, hot air balloon); 2) oscillatory movement patterns (windmill, pendulums, ornament, and swing); and 3) complex movement patterns (toy train on circular track, various plants including ivy, apple tree, and flowers). Table prresents human evaluation results for point forces, showing that despite training only on ball rolling (linear) and plant poking (complex) scenarios, our force prompting model demonstrates strong generalization across all motion categories. We note that this model successfully handles multiple forces "zero-shot" during inference, despite only being trained to handle single force, as seen in Figure 3 and detailed in Appendix B.
7

---

Control Over Direction of Forec
Global Force Prompt Generalizes to Diverse Settings and Affordances
a) Tethered motion
Acrodynamics motion
<) Fluid Dynamics
Figure 4: Qualitative results for the Global Force (Wind) model. Top: from the starting image, different directions for the force result in different videos. Bottom: while the model was only trained on flags, it can generalize to many different settings producing different types of motion.
Human study for global force benchmark: We create benchmark by curating 41 images from Pexels which demonstrate three different types of physical properties. The first is lethered motion (hair, cloth, clothing on person, paper lantern attached to hook). The second is aerodynamic motion (bubbles, falling leaves, inflatable tube in pool, floating litter, confetti). And the third is fluid dynamics (fog, smoke, snow, steam). Table [ we present human evaluation results for the global wind force model. Note that the base VideoX model is good at generating videos for all three motion categories (tethered, aerodynamic, fluid dynamic). However, our training data only has tethered motion (flags waving on a flagpole). We observe that the global wind control model trained only on labeled videos with tethered motion results in model with generalized control over aerodynamic motion and fluid motion as well. We visualize some of these generalization patterns in Figure [
The point force model, trained data from a single carnation, demonstrates remarkable generalization to other plants, as we illustrate in Figures] and 3 To evaluate this generalization quantitatively, we compare our approach against PhysDreamer (Zhang et al. 2024a). which employs 3D assets and an integrated physics simulator. Using their benchmark dataset of plant species, our results in Table show that the point force model successfully
Human study comparing PhysDreamer:
generalizes to various roses, tulips, and alocasia without specific training on these plants. While do claim replace physics-based simulation approaches, our purely neural method offers exceptional generalizability and produces responses that align with "intuitive physics," effectively conveying plausible physical interactions human evaluators. We also also conduct an extended qualitative comparison with 8 other physics simulation models; see Appendix C.2 for more details.
Ablation Studies
5.1 Ablation Study #1: Composition of Synthetic Dataset
How do synthetic dataset design choices affect model generalization? In this section we analyze the impact of dataset diversity on force modeling tasks. Sample results are illustrated in Figure 5 more in depth results are in Figure 7 (Appendix), and additional videos are the project webpage.

---

Ablation Studies: Importance of Strategic Diversity in Synthetic Training Data
Global wind force model ablation: training videos contain only 1 flag
Global wind force model ablation: training videos contain only background
Local point force model ablation: training videos contain only 1 ball, no distractors
Model overfits to waving flag, resudting in physically implausible arm motion
Degradation in background visual quality, control signal fails to generalize to steum control
Force prompt doesn' localize correctly, causing the wrong hot air balloon move
Figure 5: Results from our ablation studies on synthetic dataset design choices. Left: when the global wind force model is trained on a dataset with only one flag, it overfits, causing the woman's arm to wave unnaturally like fabric. Middle: when trained with a single background, the global force model has significantly degraded overall visual quality. Right: when trained without distractor objects, the point force model cannot properly localize motion, applying forces indiscriminately rather than to the intended target.
Alocacia
Carnation
Rose (Orange)
Rose (Red)
Rose (White)
Tulip
Mcan
Motion Realism
40%
50%
50%
60%
40%
50%
48.33%
Visual Quality 20%
40%
50%
40%
20%
50%
36.67%
Force Adherence 60%
70%
50%
50%
50%
70%
58.33%
Table 2: Comparison to PhysDreamer. Values represent the percentage of evaluators preferring the Force Prompting model over PhysDreamer, an approach that uses physies simulation during generation. Values above 50% indicate preference for our force prompting model. The results show that Force Prompting outperforms PhysDreamer on force adherence and achieves comparable performance on motion realism, while PhysDreamer maintains an advantage in visual quality.
Point force training dataset ablation: For the localized point force task, we conduct ablation study by removing "distractor balls" from scenes, leaving only a single ball affected by the point force. Our results show that the presence distractor balls significantly improves force localization. Without them, the model exhibits undesirable behaviors: when poking one hot air balloon, all balloons move slightly; when poking rose in a glass vase, both the rose and vase move together, failing to isolate the force application. Visuals are in Figure as well as the project webpage.
For the global wind force task, we evaluate two diversity factors: flag quantity and background variety. We find that training with single background leads to models that follow force physics but frequently fail to differentiate between foreground and background, reducing visual quality. Similarly, when restricting scenes to contain only one flag ... 64)), the model successfully models cloth mechanics but fails to generalize to other materials. In these cases, smoke from campfires remains unaffected by wind, and confetti either doesn't 't respond stays unnaturally suspended. We also observe that bubbles don't t respond to wind, while human limbs incorrectly billow like cloth. These failures indicate that insufficient scene diversity causes the model to overfit to stationary backgrounds and limited material interactions. These findings are illustrated in Figure 7as as well as the project webpage. Additionally, we trained unified model to learn both point force prompts and wind force prompts. We found that this results in more dynamic backgrounds, but has slightly less robust point force control. Additional details are in Appendix C.3
Global force training dataset ablation:
instead of variable number
5.2 Ablation Study #2: Text Prompt Specificity
How does specificity of the text prompt affect model outputs? In this ablation study, we investigate how material descriptions in text prompts affect model generalization through a x 2 grid search ablation study. We train and test our wind model with and without wind-related keywords (wind/breeze/blow) Our results in Figure and the project webpage show that omitting these keywords during training significantly increases failure cases in our benchmark dataset remains static, lanterns collapse unexpectedly, and steam appears without cause. contrast, models trained with wind-specific termi- nology demonstrate superior generalization to diverse wind scenarios. Interestingly, the presence of

---

Same force results in different motion based on object's inferred mass
a) single cube vs. stack of cubes
b) single book stack of books
c) succer ball vs. bowling ball
Seccer ball
Bosiligg hall
8.00
0.00
0799
Forte Applicd Object
Figure 6: Mass understanding: We find that the model has some degree of understanding of mass, in that the same force applied to two objects with different masses will result in different amounts of motion. We demonstrate this qualitatively this result is consistent across range of force magnitudes. See additional examples in the project webpage.
(a) and (b) and quantitatively in (c), showing that
these keywords during inference has less impact than during training, though using wind terminology generally produces more robust results.
6
Mass Understanding
This section examines the model's capacity for mass understanding. which define to be the ability to recognize that objects with different apparent masses should respond distinctively to the same applied force. A model with robust mass understanding would demonstrate physically intuitive behaviors: a book sliding further than a stack of books when pushed with equal force, or a wooden ornament swinging more freely than its identical metal counterpart under the same impulse. We focus our quantitative analysis on the ball-rolling scenario, as it allows for objective measurement using automatic object detection. Then, we focus our qualitative analysis on other scenarios which present greater challenges for obtaining reliable metrics at scale, such as the swinging ornament.
To quantitatively assess the model's mass under- standing, we design an experiment to measure whether soccer balls roll farther than bowling balls when subjected to identical forces. We generate initial condition images across four ground surfaces (dirt, grass, stone, and wood), with three color variations each for both bowling balls and soccer balls. Additional experiment details are in Appendix A.2 Results presented in Figure 6 confirm two key physical principles: the distance traveled increases linearly with applied force for both ball types, and soccer balls consistently travel farther than bowling balls across all force magnitudes, demonstrating the model's intuitive understanding of mass-dependent physics in this scenario.
Force-Mass Relationship Quantitative Study:
Force-Mass Relationship Qualitative Study: We evaluate mass understanding across four bench- mark tasks featuring geometrically identical objects with different implied masses. Our test scenarios includes ornaments (wooden versus cast iron), laundry baskets (empty versus filled with clothes), book stacks (one, two, or three books), and cube stacks (single versus double cube). To ensure experimental control, utilize the where only the implied mass differs between conditions. Figure 66 presents some of these results, with demonstrating that lighter objects consistently travel farther when subjected identical forces. This pattern remains robust across four random seeds. This behavior suggests an emergent understanding of mass-dependent physics in our prreeppromptd model. Other results are in the project webpage. Additionally, we find that the mass understanding behavior persists in the zero-shot multiple objects setting. We include these experimental details Appendix B.2
API generate initial frames with variations
7 Conclusion
We introduce Force Prompting, enabling users to interact with generative video models through physically meaningful controls including localized point forces and global wind effects. Our approach demonstrates that video generation models can successfully learn to respond to force-baseds condition- ing from limited synthetic training data, generalizing remarkably well to diverse objects, materials, and scenarios without requiring physics simulators at inference time. These results suggest promis- ing direction for developing intuitive world models that respond natural physical interactions, with potential applications in both creative content generation and embodied Al planning.
10

---

Acknowledgments
We would like to thank Bill Freeman, Miki Rubinstein, Junyi Zhang, Junhwa Hur, Noah Fischer, Saining Xie, Calvin Luo, Shijie Wang, Koven Yu, Tian Yun, and Zilai Zeng for useful discussions. We would also like to thank the anonymous NeurIPS reviewers. This project was partially supported by Samsung. Our research was conducted using computational resources at the Center for Computation and Visualization at Brown University.
References
Niket Agarwal, Arslan Ali, Maciej Bala, Yogesh Balaji, Erik Barker, Tiffany Cai, Prithvijit Chat- topadhyay, Yongxin Chen, Yin Cui, Yifan Ding, et al. 2025, Cosmos world foundation model platform for physical ai. arXiv preprint arXiv:250.033575.
Luca Savant Aira, Antonio Montanaro, Emanuele Aiello, Diego Valsesia, and Enrico Magli. 2024. Motioncraft: Physics- pessics-aased zero-shot video generation. arXiv preprint arXiv:2405. 135557
Amir Bar, Gaoyue Zhou, Danny Tran, Trevor Darrell, and Yann LeCun. 2024. Navigation world models. arXiv preprint urXi::2412.03572.
Omer Bar-Tal, Hila Chefer, Omer Tov, Charles Herrmann, Roni Paiss, Shiran Zada, Ariel Ephrat, Junhwa Hur, Guanghui Liu, Amit Raj, al. 2024. Lumiere: A space-time diffusion model for video generation. SIGGRAPH Asia 2024 Conference Papers, pages 1-11.
Andreas Blattmann, Tim Dockhorn, Sumith Kulal, Daniel Mendelevitch, Maciej Kilian, Dominik Lorenz, Yam Levi, Zion English, Vikram Voleti, Adam Letts, et al. 2023. Stable video diffusion: Scaling latent video diffusion models to large datasets. arXiv preprint arXiv2331.115127.
Tim Brooks, Bill Peebles, Connor Homes, Will DePue, Yufei Guo, Li Jing, David Schnurr, Joe Taylor, Troy Luhman, Eric Luhman, Clarence Ng, Ricky Wang, and Aditya Ramesh. 2024. Video generation models as world simulators)
Jake Bruce, Michael D Dennis, Ashley Edwards, Jack Parker-Holder, Yuge Shi, Edward Hughes, Matthew Lai, Aditi Mavalankar, Richie Steigerwald, Chris Apps, et al. 2024. Genie: Generative interactive environments. Forty-first International Conference on Machine Learning.
Haoxuan Che, Xuanhua He, Quande Liu, Cheng Jin, and Hao Chen. 2024. Gamegen-x: Interactive open-world game video generation. arXiv preprint arXiv:2411.00769.
Boyuan Chen, Hanxiao Jiang. Shaowei Liu, Saurabh Gupta, Yunzhu Li, Hao Zhao, and Shenlong Wang. 2025. Physgen3d: Crafting miniature interactive world from single image. CVPR.
Hsiao-yu Chen, Edith Tretschk, Tuur Stuyck, Petr Kadlecek, Ladislav Kavan, Etienne Vouga, and Christoph Lassner. 2022. Virtual elastic objects. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition.
Tsai-Shien Chen, Chieh Hubert Lin, Hung- Yu Tseng, Tsung- Yi Lin, and Ming-Hsus Yang. 2023. Motion-conditioned diffusion model for controllable video synthesis. arXiv preprint arXiv:2304. 14404.
Abe Davis, Katherine L Bouman, Justin G Chen, Michael Rubinstein, Fredo Durand, and William Freeman. 2015a. Visual vibrometry: Estimating material properties from small motion in video. In Proceedings of the ieee conference on computer vision and pattern recognition.
Abe Davis, Justin G Chen, and Frédo Durand. 2015b. Image-space modal bases for plausible manipulation of objects in video. ACM Transactions on Graphics (TOG).
Daniel Geng, Charles Herrmann, Junhwa Hur, Forrester Cole, Serena Zhang, Tobias Pfaff, Tatiana GupezaGuevara, Carl Doersch, Yusuf Aytar, Michael Rubinstein, et al. 2024. Motion prompting: Controlling video generation with motion trajectories. arXiv preprint 20200024...02700
Rohit Girdhar, Mannat Singh, Andrew Brown, Quentin Duval, Samaneh Azadi, Sai Saketh Rambhatla, Akbar Shah, Xi Yin, Devi Parikh, and Ishan Misra. 2023. Emu video: Factorizing text-to-video generation by explicit image conditioning. arXiv preprint arXiv:2311.1077
11

---

David Ha and Jürgen Schmidhuber. 2018. World models. arXiv preprint arXiv: 1803.00200.
Hao He, Yinghao Xu, Yuwei Guo, Gordon Wetzstein, Bo Dai, Hongsheng Li, and Ceyuan Yang. 2024. Cameractrl: Enabling camera control for text-to- video generation
Jonathan Ho, William Chan, Chitwan Saharia, Jay Whang, Ruiqi Gao, Alexey Gritsenko, Diederik P Kingma, Ben Poole, Mohammad Norouzi, David Fleet, et al. 2022. Imagen video: High definition video generation with diffusion models. arXiv preprint arXiv:22 20X00:2200.02333.
Tianyu Huang, Haoze Zhang, Yihan Zeng, Zhilu Zhang, Hui Li, Wangmeng Zuo, and Rynson WH Lau. 2024. Dreamphysics: Learning physical properties of dynamic 3d gaussians with video diffusion priors. arXiv preprint 27Xiv:2406.01476.
Denis Karachev and Yuancheng Xu. 2025. Cogvideox controlnet extention. https:///ithub. com/ TheDenk/
Simon Le Cleac'h, h, Hong-Xing Yu, Michelle Guo, Taylor Howell, Ruohan Gao, Jiajun Wu, Zachary Manchester, and Mac Schwager. 2023. Differentiable physies simulation of aynaetes-augmented neural objects. IEEE Robotics and Automation Letters
Chenyu Li, Oscar Michel, Xichen Pan, Sainan Liu, Mike Roberts, and Saining Xie. 2025a. Pisa experiments: Exploring physics post-training for video diffusion models by watching stuff drop. arXiv preprint urXiv:2503.0955)
Yaowei Li, Xintao Wang. Zhaoyang Zhang, Zhouxia Wang. Ziyang Yuan, Liangbin Xie, Yuexian Zou, and Ying Shan. 2024a. Image conductor: Precision control for interactive video synthesis. arXiv preprint 2406v:2406.13332
Zhengqi Li, Richard Tucker, Noah Snavely, and Aleksander Holynski. 2024b. Generative image dynamies. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recogni- tion.
Zizhang Li, Hong-Xing Yu, Wei Liu, Yin Yang. Charles Herrmann, Gordon Wetzstein, and Jiajun Wu. 2025b. Wonderplay: Dynamic 3d scene generation from single image and actions. arXiv preprint arXiv:2505.1811
Jiajing Lin, Zhenzhong Wang, Shu Jiang, Yongjie Hou, and Min Jiang. 2024. Phys4dgen: physics- driven framework for controllable and efficient 4d content generation from single image. arXiv preprint arXiv:2411. 1. 16800.
Fangfu Liu, Hanyang Wang, Shunyu Yao, Shengjun Zhang, Jie Zhou, and Yueqi Duan. 2024a. Physics3d: Learning physical properties of 3d gaussians via video diffusion. arXiv preprint 2403:334006
Shaowei Liu, Zhongzheng Ren, Saurabh Gupta, and Shenlong Wang. 2024b. Physgen: Rigid-body physics-grounded image-to-video generation. In ECCV.
Ilya Loshchilov and Frank Hutter. 2019. Decoupled weight decay regularization In International Conference on Learning Representations.
Koichi Namekata, Sherwin Bahmani, Ziyi Wu, Yash Kant, Igor Gilitschenski, and David B Lin- dell. 2024. Sg-i2v: Self-guided trajectory control in image-to-video generation. arXiv preprint arXiv:2411.04989,
Muyao Niu, Xiaodong Cun, Xintao Wang, Yong Zhang, Ying Shan, and Yinqiang Zheng. 2024. Mofa- video: Controllable image animation via generative motion field adaptions in frozen image-to-video diffusion model. arXiv preprint 2rXiv:2405.20222
Shaoqing Ren, Kaiming He, Ross Girshick, and Jian Sun. 2015. Faster r-cnn: Towards real-time object detection with region proposal networks. Advances in neural information processing systems, 28.
12

---

Xiaoyu Shi, Zhaoyang Huang, Yun Wang, Weikang Bian, Dasong Li, Yi Zhang, Manyuan Zhang. Ka Chun Cheung, Simon See, Hongwei Qin, et al. 2024. Motion-i2v: Consistent and controllable image-to-video generation with explicit motion modeling. In ACM SIGGRAPH 2024 Conference Papers, pages 1-11.
Uriel Singer, Adam Polyak, Thomas Hayes, Xi Yin, Jie An, Songyang Zhang, Qiyuan Hu, Harry Yang, Oron Ashual, Oran Gafni, et al. 2022. Make-a-video: Text-to-video generation without text-video data. arXiv preprint 2rXiv:2209.14773
Wengiang Sun, Shuo Chen, Fangfu Liu, Zilong Chen, Yueqi Duan, Jun Zhang, and Yikai Wang. 2024. Dimensionx: Create any 3d and 4d scenes from a single image with controllable video diffusion. arXiv preprint arXiv:2411.0492
Xiyang Tan, Ying Jiang, Xuan Li, Zeshun Zong, Tianyi Xie, Yin Yang, and Chenfanfu Jiang. 2024. Physmotion: ttlysic--grunnder dynamics from single image. arXiv preprint arXiv:2411.17188
Tomer D Ullman, Elizabeth Spelke, Peter Battaglia, and Joshua B Tenenbaum. 2017. Mind games: Game engines as an architecture for intuitive physics. Trends in cognifive sciences, 2199)6649-665
Dani Valevski, Yaniv Leviathan, Moab Arar, and Shlomi Fruchter. 2024. Diffusion models are real-time game engines. arXiv preprint arXiv:2408. 14837.
Ang Wang. Baole Ai, Bin Wen, Chaojie Mao, Chen Wei Xie, Di Chen, Feiwu Yu, Haiming Zhao, Jianxiao Yang, Jianyuan Zeng, Jiayu Wang, Jingfeng Zhang, Jingren Zhou, Jinkai Wang. Jixuan Chen, Kai Zhu, Kang Zhao, Keyu Yan, Lianghua Huang, Mengyang Feng, Ningyi Zhang, Pandeng Li, Pingyu Wu, Ruihang Chu, Ruili Feng, Shiwei Zhang. Siyang Sun, Tao Fang, Tianxing Wang, Tianyi Gui, Tingyu Weng, Tong Shen, Wei Lin, Wei Wang. Wei Wang, Wenmeng Zhou, Wente Wang, Wenting Shen, Wenyuan Yu, Xianzhong Shi, Xiaoming Huang, Xin Xu, Yan Kou, Yangyu Lv. Yifei Li, Yijing Liu, Yiming Wang, Yingya Zhang, Yitong Huang, Yong Li, You Wu, Yu Liu, Yulin Pan, Yun Zheng, Yuntao Hong, Yupeng Shi, Yutong Feng, Zeyinzi Jiang, Zhen Han, Zhi-Fan Wu, and Ziyu Liu. 2025. Wan: Open and advanced large-scale scale video generative models. arXiv preprint 2503:22000.0314
Chen Wang*, Chuhao Chen*, Yiming Huang, Zhiyang Dou, Yuan Liu, Jiatao Gu, and Lingjie Liu. 2025. Physetrl: Generative physics for controllable and physics-grounded video generation. In NeurIPS.
Zhouxia Wang, Ziyang Yuan, Xintao Wang, Yaowei Li, Tianshui Chen, Menghan Xia, Ping Luo, and Ying Shan. 2024. Motionetrl: A unified and flexible motion controller for video generation. In ACM SIGGRAPH 2024 Conference Papers.
Friedrich Wilkening and Trix Cacchione. 2010. Children's ' intuitive physics. The Wiley- wlackell handbook of childhood cognitive development, pages 473-496-
Weijia Wu, Zhuang Li, Yuchao Gu, Rui Zhao, Yefei He, David Junhao Zhang, Mike Zheng Shou, Yan Li, Tingting Gao, and Di Zhang. 2024. Draganything: Motion control for anything using entity representation. In European Conference on Computer Vision.
Tianyi Xie, Zeshun Zong, Yuxing Qiu, Xuan Li, Yutao Feng, Yin Yang, and Chenfanfu Jiang. 2024. Physgaussian: Physics-integrated 3d gaussians for generative dynamics. In CVPR.
Zhuoyi Yang, Jiayan Teng, Wendi Zheng, Ming Ding, Shiyu Huang, Jiazheng Xu, Yuanming Yang, Wenyi Hong. Xiaohan Zhang, Guanyu Feng, et al. 2024. Cogvideox: Text-to-video diffusion models with an expert transformer. arXiv preprint ar arXiv:2408.0600
Shengming Yin, Chenfei Wu, Jian Liang, Jie Shi, Houqiang Li, Gong Ming, and Nan Duan. 2023. Dragnuwa: Fine-grained control in video generation by integrating text, image, and trajectory. arXiv preprint arXiv:2308.08m 08089.
Lvmin Zhang, Anyi Rao, and Maneesh Agrawala. 2023. Adding conditional control to text-to-image diffusion models.
13

---

Tianyuan Zhang, Hong-Xing Yu, Rundi Wu, Brandon Y Feng, Changxi Zheng, Noah Snavely, Jiajun Wu, and William T Freeman. 2024a. Physdreamer: Physics-based based interaction with 3d objects via video generation. In ECCV.
Zhenghao Zhang, Junchao Liao, Menghao Li, Zuozhuo Dai, Bingxue Qiu, Siyu Zhu, Long Qin, and Weizhi Wang. 2024b. Tora: Trajentory-orienteed diffusion transformer for video generation. arXiv preprint arXiv:2407.21705.
Guangcong Zheng, Teng Li, Rui Jiang, Yehao Lu, Tao Wu, and Xi Li. 2024. Cami2v: Camera- controlled image-to-vides video diffusion model. arXiv preprint arXiv:2410.1590
Licheng Zhong, Hong-Xinn Xing Yu. Jiajun Wu, and Yunzhu Li. 2024. Reconstruction and simulation of elastic objects with spring-mass 3d gaussians. In European Conference on Computer Vision.
NeurIPS Paper Checklist
1. Claims
Question: Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?
Answer: [Yes]
Justification: We claim that video generation models trained with force prompting can generalize to out-of-domain scenarios, where out-of-domain refers to conditioning images beyond our synthetic dataset. We believe that our validation benchmark, which is best effort proxy to test physics-informed generation, paired with our human evaluation study is extensive and diverse enough to support this claim.
Guidelines:
The answer NA means that the abstract and introduction do not include the claims made in the paper.
The abstract and/or introduction should clearly state the claims made, including the contributions made in the paper and important assumptions and limitations. A No or
NA answer to this question will not perceived well by the reviewers.
The claims made should match theoretical and experimental results, and reflect how much the results can be expected to generalize other settings.
is fine to include aspirational goals as motivation as long as is clear that these goals are not attained by the paper.
2. Limitations
Question:
Does the paper discuss the limitations of the work performed by the authors?
Answer: [Yes]
Justification: We enumerate specific failure modes of approach in dedicated section on failures and limitations. We also highlight specific ablations in our synthetic data construction and text-prompt construction that highlight the limitations and failures that remedied on the path to our final reported results.
Guidelines:
The answer NA means that the paper has no limitation while the answer No means that the paper has limitations, but those are not discussed in the paper.
The authors are encouraged to create a separate "Limitations" section their paper.
The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (e.g., independence assumptions, noiseless settings, model well well-specification, asymptotic approximations only holding locally). The authors should reflect on how these assumptions might be violated in practice and what the implications would be.
14

---

The authors should reflect on the scope of the claims made, e.g., if the approach was only tested on few datasets or with few runs. In general, empirical results often depend on implicit assumptions, which should be articulated.
The authors should reflect the factors that influence the performance of the approach. For example, facial recognition algorithm may perform poorly when image resolution is low images are taken in low lighting. Or speech-to-text system might not be used reliably to provide closed captions for online lectures because it fails to handle technical jargon.
The authors should discuss the computational efficiency of the proposed algorithms and how they scale with dataset size.
If applicable, the authors should discuss possible limitations of their approach to address problems of privacy and fairness.
While the authors might fear that complete honesty about limitations might be used by reviewers as grounds for rejection, worse outcome might be that reviewers discover limitations that aren't 't acknowledged in the paper. The authors should use their best judgment and recognize that individual actions in favor of transparency play an impor- tant role in developing norms that preserve the integrity of the community, Reviewers will be specifically instructed not penalize honesty concerning limitations.
3. Theory assumptions and proofs
Question: For each theoretical result, does the paper provide the full set of assumptions and a complete (and correct) proof?
Answer: [NA]
Justification:
We do not present theoretical results in this work.
Guidelines:
The answer NA means that the paper does not include theoretical results.
All the theorems, formulas, and proofs in the paper should be numbered and cross- referenced.
All assumptions should be clearly stated or referenced the statement of any theorems.
The proofs can either appear in the main paper or the supplemental material, but they appear in the supplemental material, the authors encouraged provide short proof sketch to provide intuition.
Inversely, any informal proof provided in the core of the paper should be complemented by formal proofs provided in appendix or supplemental material.
Theorems and Lemmas that the proof relies upon should properly referenced.
4. Experimental result reproducibility
Does the paper fully disclose all the information needed to reproduce the main ex- perimental results the paper to the extent that affects the main claims and/or conclusions of the paper (regardless of whether the code and data are provided not)?
Question:
Answer: [Yes]
We release our dataset generation code, evaluation benchmark dataset, model training evaluation code, and detailed instructions for how reproduce our results therein. We also provide model checkpoints, which may be used to produce exactly the videos used in our human evaluation.
Justification:
Guidelines:
The answer NA means that the paper does not include experiments.
If the paper includes experiments, a No answer to this question will not be perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data provided or not.
If the contribution is dataset and/or model, the authors should describe the steps taken to make their results reproducible or verifiable.
Depending on the contribution, reproducibility can be accomplished in various ways. For example, if the contribution is a novel architecture, describing the architecture fully might suffice, or if the contribution is specific model and empirical evaluation, it may
15

---

be necessary to either make it possible for others to replicate the model with the same dataset, or provide access the model. In general. releasing code and data is often one good way to accomplish this, but reproducibility can also be provided via detailed instructions for how to replicate the results, access to a hosted model (e.g., in the case of large language model), releasing of a model checkpoint, or other means that are appropriate to the research performed.
While NeurIPS does not require releasing code, the conference does require all submis- sions to provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution. For example
(a) If the contribution is primarily a new algorithm, the paper should make it clear how to reproduce that algorithm.
(b) If the contribution is primarily new model architecture, the paper should describe the architecture clearly and fully.
(c) If the contribution is a new model (e.g., a large language model), then there should either be way access this model for reproducing the results or way to reproduce the model (e.g., with an open-source dataset or instructions for how to construct the dataset).
(d) We recognize that reproducibility may be tricky in some cases, in which authors are welcome to describe the particular way they provide for reproducibility. In the case of closed-source models, it may be that access to the model is limited in some way (e.g., to registered users), but it should be possible for other researchers have some path reproducing or verifying the results.
5. Open access to data and code
Question: Does the paper provide open access to the data and code, with sufficient instruc- tions to faithfully reproduce the main experimental results, as described in supplemental material?
Answer: [Yes]
Justification: We provide all artifacts required to reproduce our work, including code, datasets, survey forms, and precise instruction.
Guidelines:
The answer NA means that paper does not include experiments requiring code.
Please see the NeurIPS code and data submission guidelines (https://nips... cc/
jublic/guidess httoli/c/uiieee//deebbbbbsssssss solicooiiiioyo)) for more details.
While we encourage the release of code and data, we understand that this might not be possible, so "No" is an acceptable answer. Papers cannot be rejected simply for not including code, unless this is central to the contribution (e.g., for new open-source benchmark).
The instructions should contain the exact command and environment needed to run to reproduce the results. See the NeurlPS code and data submission guidelines (https for more details.
7 7/nips. cc/ httcc//bbllcggguuueeeeeee wwwpc//ggueeeeeeee wwwooddeuuiisssoononoo
The authors should provide instructions data access and preparation, including how to access the raw data, preprocessed data, intermediate data, and generated data, etc.
The authors should provide scripts to reproduce all experimental results for the new proposed method and baselines. only subset of experiments are reproducible, they should state which ones are omitted from the script and why.
At submission time, to preserve anonymity, the authors should release anonymized versions (if applicable).
Providing as much information as possible in supplemental material (appended to the paper) recommended, but including URLs to data and code permitted.
6. Experimental setting/details
Question:
Does the paper specify all the training and test details (c.g., data splits, hyper- parameters, how they were chosen, type of optimizer, etc.) necessary to understand the results?
Answer: [Yes]
16

---

We release the entire data generation pipeline, the data itself, and training code. For our work, we do not optimize hyperparameters that is, the settings within the code release are those that produced the checkpoints we distribute. Guidelines:
Justification:
The answer NA means that the paper does not include experiments.
The experimental setting should be presented in the core the paper to a level of detail that is necessary appreciate the results and make sense of them.
The full details can be provided either with the code, in appendix, or as supplemental material.
7. Experiment statistical significance
Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?
Question:
Answer: [Yes]
Justification: We provide error bars for our quantitative mass understanding study, and clearly describe that the variability comes from distinct surfaces, color, and random seeds.
Guidelines:
The answer NA means that the paper does not include experiments.
The authors should answer "Yes" if the results are accompanied by error bars, confi- dence intervals, or statistical significance tests, at least for the experiments that support the main claims of the paper.
The factors of variability that the error bars are capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, or overall
run with given experimental conditions).
The method for calculating the error bars should be explained (closed form formula, call to a library function, bootstrap, etc.)
The assumptions made should be given (e.g.. Normally distributed errors).
should be clear whether the error bar is the standard deviation the standard error of the mean.
It is OK to report 1-sigma error bars, but one should state it. The authors should preferably report a 2-sigma error bar than state that they have 96% CI, if the hypothesis of Normality of errors is not verified.
For asymmetric distributions, the authors should be careful figures symmetric error bars that would yield results that are out of range (e.g. negative error rates).
show in tables or
If error bars are reported in tables or plots, The authors should explain in the text how they were calculated and reference the corresponding figures or tables in the text.
8. Experiments compute resources
For each experiment, does the paper provide sufficient information on the com- puter resources (type of compute workers, memory, time of execution) needed reproduce the experiments?
Question:
Answer: [Yes]
Justification: We describe the exact GPU models and quantity for our experiments in the abstract. We also describe roughly how long it takes to produce our results on said hardware.
Guidelines:
The answer NA means that the paper does not include experiments.
The paper should indicate the type of compute workers CPU or GPU, internal cluster, or cloud provider, including relevant memory and storage.
The paper should provide the amount of compute required for each of the individual experimental runs as well as estimate the total compute.
The paper should disclose whether the full research project required more compute than the experiments reported the paper (e.g., preliminary or failed experiments that didn't 't make into the paper).
17

---

9. Code of ethics
Question: Does the research conducted the paper conform, in every respect, with the NeurlPS Code of Ethics
Answer: [Yes]
Justification: We aim uphold fairness collaborators, research participants, and provide ample attribution to their respective authors. Our synthetic dataset does not depict humans, and any humans depicted in our evaluation benchmark or paper figures are from open-access and royalty free images where persons have given explicit consent to appear. Guidelines:
The answer NA means that the authors have not reviewed the NeurIPS Code of Ethics.
If the authors answer No, they should explain the special circumstances that require deviation from the Code of Ethics.
The authors should make sure to preserve anonymity (e.g.. there a special consid- eration due to laws or regulations in their jurisdiction).
10. Broader impacts
Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?
Answer: [Yes]
Justification: In the appendix, we describe scope that we consider appropriate for our work and emphasize the risks in using our method for precise simulation. We believe the potential for positive impact is high, provided that it is clear that our work is substitute for precise physics simulators.
Guidelines:
The answer NA means that there is no societal impact of the work performed.
If the authors answer NA or No, they should explain why their work has no societal impact why the paper does not address societal impact.
Examples of negative societal impacts include potential malicious unintended
(e.g., disinformation, generating fake profiles, surveillance), fairness considerations (e.g., deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations.
The conference expects that many papers will be foundational research and not tied partieular applications, let alone deployments. However, if there is direct path to any negative applications, the authors should point it out. For example, it is legitimate point out that an improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point out that a generic algorithm for optimizing neural networks could enable people train models that generate Deepfakes faster.
The authors should consider possible harms that could arise when the technology is being used as intended and functioning correctly, harms that could arise when the technology is being used as intended but gives incorrect results, and harms following from (intentional or unintentional) misuse of the technology.
If there are negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition to attacks, mechanisms for monitoring misuse, mechanisms to monitor how system learns from feedback over time, improving the efficiency and accessibility of ML).
11. Safeguards
Does the paper describe safeguards that have been put in place for responsible release of data or models that have a high risk for misuse (e.g., pretrained language models, image generators, scraped datasets)?
Question:
Answer: [NA]
Justification: Our data are not obtained by internet scraping: they are designed by hand selected manually. While our work builds upon video generation models, which do have high potential for misuse, we inherit the safety precautions present the models we fine-tune.
18

---

Guidelines:
The answer NA means that the paper poses no such risks.
Released models that have high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere to usage guidelines or restrictions to access the model or implementing safety filters.
Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images.
recognize that providing effective safeguards challenging, and many papers do not require this, but we encourage authors to take this into account and make best faith effort.
12. Licenses for existing assets
Are the creators or original owners of assets (e.g., code, data, models), used in the paper, properly credited and are the license and terms of use explicitly mentioned and properly respected?
Question:
Answer: [Yes]
Justification: We have audited and reviewed respective licenses and user agreements of all our work's 's dependencies. We seek to give proper and thorough attribution to all authors of code, publications, and software the best of our knowledge. We dcliberately chose Blender as our physics simulator because of its permissive license.
Guidelines:
The answer NA means that the paper does use existing assets.
The authors should cite the original paper that produced the code package or dataset.
The authors should state which version of the asset is used and, if possible, include URL.
The name of the license (e.g., CC-BY 4.0) should be included for each asset.
For scraped data from a particular source (e.g., website), the copyright and terms of service of that source should be provided.
assets are released, the license, copyright information, and terms use in package should be provided. For popular datasets, paperswi wwwode.tho.....m. com datasets has curated licenses for some datasets. Their licensing guide can help determine the license of dataset.
For existing datasets that re-packaged, both the original license and the license of the derived asset (if it has changed) should be provided.
If this information is not available online, the authors are encouraged to reach out to the asset's s creators.
13. New assets
Question: Are assets introduced in the paper well documented and is the documentation provided alongside the assets?
Answer: [Yes]
Justification: Our evaluation benchmark and training data are distributed along with our code files. For each sample, note the image, parameters used in its force information, a text-prompt, and any scenery-related specifics used in its generation.
Guidelines:
The answer NA means that the paper does release new assets.
Researchers should communicate the details the dataset/code/moodll as part their submissions via structured templates. This includes details about training, license, limitations, etc.
The paper should discuss whether and how consent was obtained from people whose asset used.
At submission time, remember to anonymize your assets (if applicable). You can either create anonymized URL or include an anonymized zip file.
19

---

14. Crowdsourcing and research with human subjects
Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given participants and screenshots, if applicable, as well as details about compensation (if any)?
Answer: [Yes]
Justification: The appendix contains example questions and screenshots from our human studies.
Guidelines:
The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.
Including this information in the supplemental material is fine, but if the main contribu- tion of the paper involves human subjects, then as much detail as possible should be included in the main paper.
According to the NeurIPS Code of Ethics, workers involved data collection, curation, or other labor should be paid at least the minimum wage in the country of the data collector.
15. Institutional review board (IRB) approvals or equivalent for research with human subjects
Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or an equivalent approval/review based on the requirements of your country or institution) were obtained?
Question:
Answer: [No]
Justification: We do not have human subjects in our work, though we collect responses from survey participants. Our survey asks respondents to answer opinion-based questions about the videos we present; our videos free from flashing colors or sensitive topics that may be harmful there are no risks to the participant.
Guidelines:
The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.
Depending on the country in which research is conducted, IRB approval (or equivalent)
may be required for any human subjects research. If you obtained IRB approval, you should clearly state this in the paper.
We recognize that the procedures for this may vary significantly between institutions and locations, and we expect authors to adhere the NeurIPS Code of Ethics and the guidelines for their institution.
For initial submissions, do not include any information that would break anonymity (if applicable), such as the institution conducting the review.
Declaration LLM usage
Question: Does the paper describe the usage of LLMs if it is an important, original, or non-standard component of the core methods in this research? Note that if the LLM is used only for writing, editing, formatting purposes and does not impact the core methodology, scientific rigorousness, or originality of the research, declaration is not required.
Answer: [NA]
Justification: we use L.MSs for prompt upscaling, but this is standard technique used by CogVideoX and other generative models to automatically obtain detailed image descriptions.
Guidelines:
The answer NA means that the core method development in this research does not involve L.LMs any important, original, non-standard components.
Please refer to our LLM for what should or should not be described.
20

---

A Implementation details
A.1 Training hyperparameters
We train for 5000 steps with effective batch size of 8 (using gradient accumulation over two steps) and saved model checkpoints every five hundred steps. We used bf16 mixed precision with tf32 support and initialized from the THUDM/COO -eigViieeX-Sb-----
pretrained weights. ControlNet (Zhang (ct al. 2023) was initialized from the first six transformer layers, for all three input channels, using downscaling factor of eight, and unit weight coefficient. We use the AdamW optimizer (Loshchilov and Hutter 2019) (initial learning rate x 10-5, 31 = 0.9, A2 = 0.95, maximum gradient norm 1.0) under a cosine-with-e wettrrestarts learning- schedule (one cycle, 250 warm-up steps). We set our random seed to 42. Training vidcos contained up to 49 frames at 8 fps.
There are total of 42 transformer blocks that we may use to initialize our ControlNet. We chose the first six due to memory constraints. With more compute resources, one could potentially achieve higher quality control. We expose the number of transformer blocks to use in the ControlNet as command line argument for convenience in our training code release.
A.2 Additional details: mass understanding quantitative study
Each ball is subjected to forces of magnitude 0.125 n. where n (1,...,8). with 10 videos generated per force value using different random seeds. To ensure experimental control, we utilize the GPT-Image-1 1 API to generate first frames and their variations, maintaining consistent initial ball positions and shapes across conditions. We automatically detect ball position using a Faster R-CNN model with ResNet-50-FPN backbone (Ren et al. 2015), tracking the sports ball class (ID 37). We compute the distance traveled as the Euclidean distance in pixel space between detected bounding box centers in the first and last frames.
A.3 Additional details: encoding strategy, point force
For the first frame, we set the pixel values equal to O everywhere except for a Gaussian blob of radius 20 centered at (xr y), which gets value of 1; and the final frame of the control signal, the blob will have moved a distance of (3 + F) pixels. This ensures that when the force is minimized F = 0, the total displacement w/8, and when the force is maximized F = 1, the total displacement is w//2.
A.4 Additional details: comparison with PhysDreamer
We took six flower demo videos from the PhysDreamer teaser videos (Alocacia, Carnation, Orange Rose, Red Rose, White Rose, Tulip) and took a screenshot of still frame from the video that did not have any force annotation prompts (i.e. the same force vector from the original demo) into the Force Prompting model. We presented these videos side by side in the human study, which we served using Qualtrics. Both videos for given scene were shown to the human annotator simultancously, with left/right randomization.
it. We passed these six frames, as well as the six equivalent force
B
Additional details: zero-shot multiple forces
B.1 Multiple forces for multiple objects, benchmark
Our experiments confirm that the model successfully handles multiple simultaneous forces without requiring retraining. This capability emerges zero-shot by simply adding multiple Gaussian blobs to the control signal videos- one for each applied force. We include these videos in the project webpage. The 6 scenes we tested are:
An image with Christmas ornaments
An image with 2 toy
An image with 2 vases, each with flower in it
An image with 2 roses leaning diagonally, growing out of the ground
21

---

An image with 2 dandelions growing in field
An image with 2 apples on separate branches of a tree
We generated these images using the GPT-Image-1 API. To test each image, we identified two directions where it would be reasonable poke each object; for example, the dandelions can each be poked to the left and to the right. Then for each image, we used 4 different multiple-force force prompts, representing the 2 x 2 different ways of combining the two different directions; for example, we can poke both dandelions to the left, poke both to the right, poke one to the right and the other to the left, then poke one to the left and the other to the right. We computed one video for each, using the same seed for all of them, with no cherry-picking. We found that 5/6 of the videos in this multi-poke benchmark had perfect force adherence, and one of them had nearly perfect force adherence. The only failure case was in the two apples scene; when the left apple is poked to the right and the right apple is poked to the left, they move as they were both poked to the right.
B.2 Multiple forces for multiple objects, mass understanding
We used the GPT--mage-1 1 API to construct images where two objects of different apparent masses are present. The first image contains an empty laundry basket on the left side of the frame, and a full laundry basket on the right side of the frame. The second image contains book on the left side of the frame, and a stack of books on the right side of the frame. We passed both sets of images into the multi-poke Force Prompting inference pipeline and instructed the model to poke both sets of objects towards the middle of the frame with the same force magnitude. Across 8 different force magnitudes (0.125 i, (1.-..,8) 8)) found that the lighter object moved much further, with the heavier item barely moving at all. We include videos on our project page.
C Additional qualitative results
C.1 Failures and Limitations
Figure iilusttrates and categorizes failure cases of Force Prompting. We observe model correlation issues for example, in hair-blowing scenarios, faces sometimes reorient based on wind direction, likely reflecting patterns training data where hair typically blows backward. Our method is fundamentally constrained by the underlying video prior's physical understanding: we focus on controlling existing physical capabilities rather than improving the model's physics comprehension. We defer to other works that specifically aim to enhance physical accuracy in generative models, while noting that our approach benefits from efficiently leveraging the scaling properties of the base model.
C.2 Extended comparison with physics simulation models
To demonstrate the point force model's versatility, we curate a benchmark using first-frame images
from prominent physic--in-the-lop papers: PhysDreamer (Zhang et al., (024a)) DreamPhysics
Huang et al. (2024), MotionCraft (Aira et al. 2024), PhysGaussian (Xie et al. 2024), PhysGen (Liu
et al. 2024b). PhysGen3D (Chen al. (025); ). Physies3D (Liu et al. | (024a)) and PhysMotion (Tan
et al. 2024). We apply our force prompting approach to these diverse scenarios, including poking plants (alocasia, ficus, bouquet of flowers), moving vehicles (boat in water, toy cars), and household objects (rocking horse). Our video results (see project webpage) illustrate that our purely neural method can handle the same visual scenarios almost as effectively as approaches requiring explicit physics simulation at inference time. These qualitative results are in line with our findings in Table 2
C.3 Training a unified model
We also train unified model to learn both point force prompts and wind force prompts. We run inference on this joint model using our point force benchmark, well as our wind force benchmark. Our findings:
More dynamic backgrounds: in many of the videos, the background moves more dynamically in natural way. For example, for the apple tree, the surrounding leaves move more naturally
22

---

Ablation Studies: Importance of Synthetic Training Data Diversity
Global wind force model ablation: training videos contain only flag
Model overfis vaving flog, resulting physically impiausible arm motion
Control signal juils generulize smoke control
Global wind force model ablation: training videos contain only background
Degradation background visual quality. and control fails to generalize to steam control
Degradation background visual quality
Local point force model ablation: training videos contain only ball, no distractor balls
Forcz prompt doems 't localiee correctly, resulting is physically implausible train movement
Force prompt doesn "tlocalize correctly. causting wrong balloon to move
Figure 7: Results from our ablation studies on synthetic dataset design choices. Top: when the global wind force model is trained dataset with only one flag, it overfits, causing the woman's 's arm to wave unnaturally like fabric and failing to generalize to fluid dynamics scenarios such as smoke. Middle: when trained with single background, the global force model fails differentiate between foreground and background elements, significantly degrading overall visual quality. Bottom: when trained without distractor objects, the point force model cannot properly localize motion, applying forces indiscriminately rather than to the intended target.
after the apple is poked; in the video where kid is sitting in the middle of toy train track, the kid is moving more naturally while the train is moving around the track; and in the video with falling leaves in the forest with a woman sitting on chair in the background, the woman in the chair moves more while the leaves are being blown.
Slightly less robust point force control: some of the point force videos (e.g. the blueberry bush), the control signal is not respected.
designed this experiment by sourcing 50% training data and control signals each batch from the synthetic point force dataset, and the other 50% from the synthetic wind force dataset. We trained using the same architecture and number of training steps as the original model.
C.4 Scaling the dataset size
We trained from scratch a wind force model which uses half as many synthetic flag waving videos. The only variable that we changed was the dataset size; everything else (including the number of training steps and learning rate scheduler) remained the same. For this model, we found some additional failure cases. One failure case: for the image where woman holds sheet on the beach,
23

---

gentle wisp of steum curls upward from the open spout of kettle, swirling gracefully in the warm air Ablation Studies: Importance of Using Force-Related Keywords
Using "wind/breeze/bloww keywords during training im improves model performance, regardless of whether we use the keywords during inference
Avoiding "wind/breeze/blow" keywords during training hurts model performance, regardless of whether the keywords during inference
Using "wind/breed fhind/breeez/blow keywords during inference
Using "wind/breeze/loow keywords during inference
gentle wisp of curls upward from the open spout of kettle, swirling gracefully in the breeze
gentle wisp of steum curls upward from the open spout of a kettle, swirling gracefully in the warm brezze
Not using "wind/breezzebloww keywords during inference
Not using "wind/brecx//blow keywords during inference
gentle wisp of curls upward from the open spout of kettle, swirling gracefully in the warm air
Figure 8: Results from our ablation studies on text prompt specificity. this ablation study, investigate how material descriptions in text prompts affect model generalization through 2 x 2 grid search ablation study. We train and test our wind model with and without wind-related keywords (wind/breeze/blow) Our results demonstrate that omitting these keywords during training significantly increases failure cases in our benchmark dataset. For example, steam is conjured out of thin air instead of being blown correctly. In contrast, models trained with wind-specific terminology demonstrated superior generalization to diverse wind scenarios.
the model hallucinates bedsheet in the background, indicating that the model has likely memorized the "waving flag" pattern from the training dataset and is injecting into the output video. A second failure case: the confetti's response to the wind isn't as convincing. For example, some of the confetti will blow in the wind's direction but some will stay stationary. This indicates that the model hasn't t generalized properly, perhaps because less training data.
D
Additional emergent phenomena
D.1
Case study #1: Does the model enforce physical affordances?
The Force Prompting models demonstrate a surprising capability to respect object-specific movement constraints. For example, when train on a circular track is poked forward, it follows the curved trajectory of the track rather than continuing in a straight line-a behavior similarly observed with windmills respecting their rotational axis. We also note interesting emergent behaviors with multi part objects: poking the lead car of toy train forward sometimes pulls the entire train along, while other times only the first car moves; conversely, backward forces consistently push the entire train as unit.
D.2 Case study #2: Does the model understand atomicity of objects?
We evaluate the local force model's sensitivity to the specific pixel chosen as the application point for localized forces. Results demonstrate consistent object movement regardless of which part of object receives the force. For example, whether poking a train's engine, middle car, caboose, the entire object responds appropriately the applied force. This suggests the model has developed holistic understanding of object wholeness rather than simply responding to pixel-level manipulations.

---

Limitations of Force Prompting Method
Failure Case #1: The Physics is Out-of- oft-of-DDmmin For the Base Model
The egg rolls in the prompted direction, but the base video model has difficulty rolling non- spherical objects, so the egg appears to float
The kite blown in the prompted direction, the base video model difficulty generating physically plausible video kite dragging person
Failure Case #2: The Base Vidco Model's Prior Competes with the Force Prompt
The rocking chair moves in the prompted direction, but the base video model has trouble distinguishing between foreground and background objects
confetti moves prompted direction, bud base video model forces the scene conjure extra confetti
Figure 9: Analysis failure cases. We illustrate and categorize failure cases of Force Prompting. The top row shows scenarios where the generated physical motion is out-of-domain for the base Cog VideoX model, leading to partial adherence to the force prompt. The bottom row depicts failures in visual fidelity physical realism when the video prior conflicts with the force prompt's s intent. More examples are available on our project webpage.
D.3
Case study #3: Does the model preserve cinematic effects of the original image?
The model demonstrates ability maintain the original image's stylistic and cinematic properties throughout generated sequences. For example, when animating a toy car from an image with depth-of-field effect, the model preserves the background blur the moves, ensuring visual continuity with the source image's aesthetic. This suggests the model not only understands physical motion but also respects the artistic intent and visual language of the input image. See the project page for videos.
E Impact Statement
This work may be used to enhance the physical plausibility and controllability video generation models. Applications include video content creation with fine-grained output control for physics- based forces and second-order solutions that leverage enhanced intuitive physics such as motion planning and world simulation. We urge the community to think critically about the potential risks of our work, specifically the modeling of physical phenomena. Our work is not substitute for precise physical simulation, rather we focus on what we have described as "intuitive physics", i.e. motion that is visually plausible humans. Indeed, there are many unintuitive physical phenomena in the world where precise and specialist-level simulation is required. We emphasize that our work is unsuitable for use cases requiring high fidelity and precise simulation, including but not limited to materials science, architecture, mechanical engineering, and civil engineering.
25

---

Example 1. Please watch both videos. Which video more accurately shows effect of wind blowing to the right?
Video more accurately shows wind blowing right since dress blown right, while in Video the dress not blown at all.
Figure 10: demonstration question from one of our surveys. Participants are shown an example question with response along with the reasoning for that response.
F Survey Details and Instructions
We sourced participants from Prolific, compensating responders $12/hr. Our surveys specify the number of questions and expected time limit. For example, we present the following participants at the start of the survey:
Thank you for taking this survey! It should take less than 25 minutes.
There are 208 questions total. You should aim to spend around 7-8 seconds per question. You will be shown two videos, and you must choose which video more accurately shows the effect of the wind blowing in the direction indicated by the question. Please read each question carefully.
There are hidden vigilance questions, so please make sure you answer to the best of your ability. We will be rejecting extremely poor quality responses.
At the end of the survey, there is place to put your Prolific ID so we can confirm you' 've taken the survey. Please respond only once to this survey (you may have done a similar survey the past day, that fine) and thank you for your time!
Please do not spend more than 25 minutes on this survey! We don't It want to waste your time :)
We then present participants with example questions and what we consider an appropriate response along with our reasoning. screen shot of an example can be found in Figure I0 We then present participants with questions following the example for them to answer. They may select their preference from two videos by selecting the radio button underneath their selection, which is depicted in Figure II
26

---

Q4 of 208. Please watch both videos. Which video more accurately shows the effect of wind blowing to the left?
Figure 11: question from one of our surveys. Participants are shown two videos side to side, with radio buttons beneath that they may make selection of which better adheres to the question. The videos play automatically and simultaneously.
27

---

Figure 12: Samples from our synthetic training datasets. Top (ball) and middle (flower) are timelapses from our point force training dataset; bottom (flag) are timelapses from the global force training dataset. Our key finding is that video generation models can generalize well when adapted to follow physical force conditioning from videos synthesized by Blender, even with limited demonstrations on few ojects.
28


Reviews:
[{"review_id": "65ngGwBV0l", "reviewer": "Reviewer_fCVZ", "summary": "This article introduces physical forces as novel conditioning signals for video generation models. The core insight is that pretrained video models possess latent intuitive physics priors that can be elicited via fine-tuning on limited synthetic data. There are several innovations. Physics-based control encoding, Sim2Real generalization and emergent mass understanding. Human evaluations demonstrate superior force adherence and realism over text-conditioned baselines and trajectory-based methods.", "questions": "Please refer to weakness.", "limitations": "Please refer to weakness.", "rating": 4, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 3, "significance": 2, "originality": 2, "strengths_and_weaknesses": "Strength\n\n1. The authors carried out a novel and interesting task, and it achieved certain results. Additionally, they provide codes in supplementary materials.\n\n2. The experimental results are very abundant, with a wide variety of objects, and the experimental data and network parameters descriptions are also very detailed.\n\n3. Human evaluation shown in Table 1 reflects the effectiveness of proposed model.\n\nWeakness\n\n1. In some cases, such as in the example of a scene where the wind blows, usually only the target object moves while the background remains stationary. This does not conform to the principles of physics.\n\n2. Figure 3 is hard to follow. What do the blue lines in the last two columns mean?\n\n3. The presented model has poor performance on real physics, such as fluid dynamics. As shown in Figure 4, the diffusion of smoke affected by the wind is very unrealistic. The smoke emerging from the cigarette end cannot be straight, and the inertia is not so significant.", "was_revised": true, "final_justification": "The rebuttal addressed my concerns, I will raise the rating to borderline accept.", "rebuttal": "Thank you for the thoughtful feedback and the recognition of our \"novel and interesting tasks\" with \"abundant\" experimental results. We consider the **primary contribution of our paper to be:** the definition of the task (adding physical forces as a conditioning signal for video generation models) and the initial demonstration that this task is not only feasible but yields interesting and potentially promising results. In particular, we show that despite limited training data, the model can generalize the physical conditioning signals to many different images and scenarios.\n\nWe agree with your assessment that our model, based on CogVideoX, sometimes produces video outputs that deviate from real-world physics, especially in complex scenarios like fluid dynamics. However, we believe our approach remains highly promising due to the significant advancements in video generation quality over the last two years. State-of-the-art models, such as Sora and Veo 3, have achieved incredible motion and physics realism, approaching photo-realism.\nBecause these cutting-edge models are proprietary, we opted to use an open-source model that provided a reasonable balance between quality and computational feasibility. We expect that applying our proposed techniques to a more capable, state-of-the-art base model would yield substantially better results. This is partially supported by our current findings: for certain scenarios like wind blowing on cloth and leaves, or objects oscillating after being poked, the CogVideoX-based model already produces more physically accurate results. We are optimistic that as the underlying video generation models continue to improve, our method's performance across a wider range of physical scenarios will also improve significantly.\n\nBelow we provide detailed answers to your questions.\n\n---\n\n## Q1: sometimes the target moves while the background remains stationary\nWe agree that the video generation model can violate physical principles, especially for smaller objects. This is a known limitation of earlier video generation models and an active area of research (e.g. motion-based losses) for video generation base models.\nTo improve it in our setting, we adapted our training data during post-training. As shown in Appendix Figure 7, we found that adding more flags and increasing background diversity significantly mitigates these errors. While not completely solved, our experiments suggest that enhancing the quality and diversity of our synthetic training data is a promising direction for future work.\n\n---\n\n## Q2: clarifying the blue lines in Figure 3\n\nThe blue lines visualize a time-lapse of the object's movement, with each line segment connecting its position from one frame to the next. For the flower, the sequence unfolds like this:\n\n- *Frame 1:* A single blue dot marks the initial position of the point force target.\n- *Frame 2:* A blue arc shows the flower's path as it moves downwards from its previous location.\n- *Frame 3:* Another blue arc traces the flower's movement as it arcs back upwards.\n\nWe will update the figure's caption in the revised version to more clearly illustrate this time-lapse.\n\n---\n\n## Q3: unrealistic videos with fluid dynamics\n\nAs previously discussed, we acknowledge that some samples demonstrate a limited understanding of physics. However, we believe this paper introduces a novel task and approach whose performance will improve significantly as the underlying video generation models advance. Our method's effectiveness is inherently tied to the physics quality of its base model. For example, while our wind force model performs well with objects like cloth and leaves, its performance is limited in scenarios where the base model itself struggles with physics realism. \n\nWe provide evidence for this relationship in our human study. As shown in **Table 1**, the physics quality of our Force Prompting method is directly correlated with the physics quality of the base model. This suggests that with a more capable foundation model, our approach can achieve substantially better results."}, {"review_id": "g8EYbCDMi1", "reviewer": "Reviewer_ugp2", "summary": "The paper presents a study on learning video generation models from generalized physics-based control signals. Unlike previous force-controlled video generation methods that rely on integrated physical simulators, this work adopts a fully data-driven approach, enabling controllable interaction with images through force inputs. The authors introduced two types of force models: localized point forces and global wind force fields. To train their model, they constructed a dataset using diverse physical simulators from multiple sources. Extensive experiments demonstrated the model’s strong generalization capabilities in video generation.", "questions": "1. Line 145–146: What exactly does “HDRIs” refer to? Does it stand for High Dynamic Range Images? A brief explanation would be helpful for clarity.\n2. In Table 1, why is PhysDreamer not included as a baseline for comparison?\n3. In Table 2, the authors compare their method with PhysDreamer, but the explanation of the comparison is somewhat lacking. Since generalization is a key strength of this work, more evidence and clarification on how the comparisons were conducted would be valuable. Were the evaluation metrics and testing scenarios consistent?\n4. To be honest, I had difficulty fully understanding the meaning of localized point forces and global force fields. Could the authors clarify why these terms were chosen? Are there more standard or physics-friendly terms that could be used to describe them?", "limitations": "1. The paper relies too heavily on existing paired force-video datasets and does not take advantage of larger force-enabled but unlabeled (unsupervised) datasets.\n2. While the authors mention alternative approaches involving physical simulator integration, it’s unclear why comparisons with other existing methods such as PhysGen are not included. Could the authors elaborate on this choice?", "rating": 5, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 3, "significance": 3, "originality": 3, "strengths_and_weaknesses": "Strengths\n\n1. The paper proposes a fully data-driven approach that enables controllable interaction with visual content through force-based inputs.\n2. A diverse dataset is constructed using multiple physical simulators, enhancing the robustness and variety of training data.\n3. Extensive experiments convincingly demonstrate the model’s strong generalization ability in video generation tasks.\n\nWeaknesses\n\n1. The paper relies too heavily on existing paired force-video datasets and does not take advantage of larger force-enabled but unlabeled (unsupervised) datasets.\n2. Line 145–146: What exactly does “HDRIs” refer to? Does it stand for High Dynamic Range Images? A brief explanation would be helpful for clarity.\n3. In the construction of the local force dataset, the authors mention simulating a rolling ball in response to unseen point-wise forces using Blender. This raises a question: why was Blender chosen over other platforms like Unity or more advanced physics simulators?\n4. A small portion of the dataset is collected using PhysDreamer. Given the limitations of existing simulators, will the proposed method face bottlenecks related to data diversity or label quality? How do the authors ensure high-quality, task-relevant force annotations for such specific downstream applications?\n5. In Table 1, why is PhysDreamer not included as a baseline for comparison?\n6. In Table 2, the authors compare their method with PhysDreamer, but the explanation of the comparison is somewhat lacking. Since generalization is a key strength of this work, more evidence and clarification on how the comparisons were conducted would be valuable. Were the evaluation metrics and testing scenarios consistent?\n7. While the authors mention alternative approaches involving physical simulator integration, it’s unclear why comparisons with other existing methods such as PhysGen are not included. Could the authors elaborate on this choice?\n8. Additionally, many of the referenced figures or supplementary materials in the paper do not hyperlink correctly when clicked, which affects the ease of review.", "was_revised": true, "final_justification": "After carefully reading the authors’ rebuttal and follow-up clarifications, I am now more confident in the contributions of this work. The authors have thoroughly addressed my major concerns, especially those related to dataset construction, baseline comparisons, and the rationale behind the choice of simulation tools.\n\nI suggest the authors include in their revised manuscript the reasoning provided on why PhysDreamer and PhysGen were not ideal baselines for certain evaluations. I also understand that WonderPlay was released after the NeurIPS submission deadline, but it would be helpful to briefly acknowledge it in the revised version for completeness.\n\nOne area I found a bit unclear in the original submission was the choice of terminology—specifically, “localized point force” and “global force fields.” It would improve clarity if the paper could offer a more explicit definition and justification for these terms earlier on.\n\nDespite these minor issues, I believe the novelty, feasibility, and potential impact of this paper clearly outweigh the concerns. Therefore, I raise my rating to 5: Accept.", "rebuttal": "Thank you for the thoughtful feedback. Below we provide detailed answers to your questions.\n\n---\n\n## Q1: the paper relies too heavily on labeled force-video datasets, rather than unlabeled datasets\n\nWe consider the **primary contribution of our paper to be:** the definition of the task (adding physical forces as a conditioning signal for video generation models) and the initial demonstration that this task is not only feasible but yields interesting and potentially promising results. While there may be other different approaches or datasets that we could have used to train the model, we believe that this shows that there are many potentially interesting and exciting follow ups to our proposed task.\n\nWe would also note that:\n1. Our method is enabled by the robust motion priors present in large, **unlabeled** (with respect to physical forces) video datasets used to train the base model. This allows us to effectively guide the model with a limited amount of synthetic force-video data.\n\n2. To the best of our knowledge, there is **no** existing fine-grained force-video dataset. One of our contributions is to provide such a dataset, and the best practices on how to curate it. Our data is synthesized with physics simulators, thus can be scaled up if necessary, without any manual annotations.\n3. We agree that using unlabeled video data to train force conditioned models (e.g. by inferring the forces from those videos with a machine learning algorithm) is a promising approach to generalize the force conditioning to even more diverse scenarios. However, to the best of our knowledge such algorithms are still nascent. \n\n---\n\n## Q2: defining HDRI\n\nHDRI stands for “High Dynamic Range Image”. It is a media format that is useful when you need a background image for composited scenes, as it stores a much wider range of luminance values than regular images. We’ll add more information about this to the next draft, thanks for the suggestion.\n\n---\n\n## Q3: why was Blender chosen over other platforms or more advanced physics simulators?\n\nWhile we agree that advanced engines like Unity and Unreal Engine 5 could have been beneficial, their licensing terms explicitly prohibit their use for training generative AI models. We therefore chose Blender for its highly permissive license. We also believe that using a more advanced simulation engine would indeed improve the performance of our method.\n\n---\n\n## Q4: does the method face bottlenecks relating to data diversity or label quality?\n\nWe investigated the impact of data diversity on model performance, and concluded that: \n\n1. More diverse synthetic data (more background, more flags) indeed helps generalization.\n2. Our method is nonetheless robust and generalizes to diverse scenarios. \n\nThese results of these ablation studies are summarized in Section 5, and detailed figures are in Figure 7 in the appendix.\n\n---\n\n## Q5: why was PhysDreamer not included as a baseline for comparison in Table 1?\n\nUnlike our method, **PhysDreamer** is not a general-purpose model; it must be fit to each individual scene. To use it, you first need to learn a 3D Gaussian representation of the scene and estimate physical properties like mass, Young’s modulus, and Poisson's ratio for each object. This approach is not suitable for many objects in our benchmark, such as a train moving on a track, where for example, Young's modulus does not make sense as a modeling choice.\n\nIn contrast, our **Force Prompting** method is designed for a general task. To apply it, you simply upload an image of the scene, and our neural architecture intuitively models the object's physics.\n\nWe did, however, conduct a direct comparison against PhysDreamer in **Table 2**, where we ran our Force Prompting model on six of PhysDreamer's flower demos.\n\n---\n\n## Q6: more details about comparison with PhysDreamer\n\nWe believe that the evaluation metrics and testing scenarios were consistent. We will detail here the experimental design choices that we made to ensure a reasonable comparison between PhysDreamer and Force Prompting. We’ll add these details to the next draft of the paper.\n\n* We took 6 flower demo videos from the PhysDreamer teasers (Alocacia, Carnation, Orange Rose, Red Rose, White Rose, Tulip) and screenshotted a still frame from the video that did not have any force annotation on it. \n* We passed these six frames, as well as the 6 equivalent force prompts (i.e. the same force vector from the original demo) into the Force Prompting model.\n* We presented these videos side by side in the human study, which we served using Qualtrics. Both videos for a given scene were shown to the human annotator simultaneously, with left/right randomization.\n\nThe purpose of comparing Force Prompting to PhysDreamer was to demonstrate that Force Prompting can successfully generalize to various plants (Alocacia, Orange Rose, Red Rose, White Rose, Tulip), without training on them. In contrast, the PhysDreamer method requires a learned 3D Gaussian representation, as well as estimates for the mass, Young’s modulus, and Poisson ratio for the object. To be clear, we’re not claiming that Force Prompting outperforms PhysDreamer on any sort of motion/force/video quality metrics. Rather, we conducted that human study just to demonstrate that our Force Prompting method, which doesn't use any physics simulator or 3D assets at inference time, is *somewhat comparable* to PhysDreamer, which does require those inductive biases and explicit representations.\n\n---\n\n## Q7: why was PhysGen not included as a baseline for comparison?\n\nPhysGen is specifically designed for rigid body simulation. As a result it would not work on approximately 78% of the scenes in our benchmark (e.g. the complex and oscillatory motions– flowers, ornaments, apple trees). As for the remaining 22% of the scenes in our benchmark (e.g. rolling toy cars), setting up PhysGen to run inference is non-trivial, requiring:\n\n1. A “perception” data preprocessing step, which involves extracting a foreground segmentation mask, synthesizing an inpainted background image, extracting normal, shading, and albedo maps, and physics properties of foreground objects\n2. Specifying the simulation parameters (force on each object, gravity, initial velocity and acceleration)\n\nIn contrast, with our Force Prompting model you only need to provide the first frame and a force vector. We wanted to include a comparison of our benchmark against another method, but to the best of our knowledge, the only system that enables force control in similarly diverse settings is WonderPlay (which integrates physical simulators, unlike our work), which was made public after the NeurIPS deadline. \n\n---\n\n## Q8: hyperlinks broken\n\nWe’re sorry about that! We have identified a fix for this and it will be solved in the next draft of the paper.\n\n---\n\n## Q9: clarify the choice of terminology “localized point force” and “global force fields”\n\nWe categorize simulator-based forces into two distinct types: localized forces that are applied to a specific object in a scene (e.g. pushes and pulls), and global forces that affect the entire scene uniformly (e.g. wind). We built one model for each case. We termed the former a “localized point force model”, and the latter a “global wind force”. We think that if we clarify this at the start of the paper that it would make the definitions more clear."}, {"review_id": "ndQOHMBIgo", "reviewer": "Reviewer_Pchz", "summary": "The paper proposes a controllable image-to-video generation model conditioned on input forces, which include a point-force control model and global wind force model. To train the model, the author collected synthetic datasets using blender and found a limited amount of synthetic data is enough for training. The trained model also demonstrate good generalization across motion types and settings. User study shows the effectiveness of proposed method.", "questions": "1. **Interactive motion**\n\nI wonder could such controllable model applied to model the interaction between the objects which applied force on and rest objects in the scene. For example, in the ball in the scene, can the model handle the ball collision when applying force on one ball towards the other. \n\n\n2. **Unified model**\n\nI am curious if possible to train a unified model that could handle both forces. The unified model is more helpful as current each model seems a little overfitting to certain scenarios. If trained in a unified way, will the model performance degrade and will the generalization ability disappear. \n\n3. **Multi-objects**\n\nIn all existing demos shown in the paper, there is only one single object being affected by the force. I understand the author only collect a dataset with single objects, but the proposed control signal should have the ability to model multi objects at once (like simultaneously move multi balls), otherwise the apply setting is little weak, just like physgaussian, I think video generation models should have stronger capacity.", "limitations": "yes", "rating": 4, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 4, "significance": 3, "originality": 3, "strengths_and_weaknesses": "**Strengths**:\n\n1. **Simple idea, good results**\n\nThe idea in the paper is simple but effective, and the visual results generated by the trained model are great. \n\n2. **Synthetic dataset**\n\nThe model is fine-tuned using synthetic data, and the trained model could generalize to different real settings. It looks promising and provides good insight to the video generation community. \n\n3. **Good experiments and ablation**\n\nI appreciate the abundant experiments and ablation in the paper, and those summarized findings are interesting and helpful.\n\n**Weakness:**\n\n1. **Insufficient metrics**\n\nAll quantitative evaluation used in the paper are human user study, which is quite subjective. I would like to see some objective metrics. As the training using synthetic dataset, a straight way is to compare the generated result and simulated results (like moving ball) and evaluate the material influence, and mass understanding in a more general and rigorous way. Otherwise it could be hard to tell the results are occasion for certain examples or indeed some common patterns behind it. \n\n2. **Missing qualitative comparison to different methods**\n\nI didn't find the video results and qualitative comparison in B.2 and attached supp as claimed in the paper. The comparison to baseline cogvideox is necessary as it could show the improvement of introducing such explicit control signal. \n\n3. **Limited setting**\n\nFrom current results, it seems the model could only model one kind of force for each type of input, and only moving one object with simple motion (no interaction with the scene), the setting is quite constrained under my view.", "was_revised": true, "final_justification": "Thanks the author for the detailed rebuttal. I think the rebuttal experiments are convincing and solid. The limitation of the work is the motion is relatively simple without object-object interactions, but the model is still interesting and will benefit the community. Another issue is the lack of quantitative comparison, hope in the final version could see more comparison results to different method on more examples and rebuttal experiment settings.", "rebuttal": "Thank you for the thoughtful feedback. Below we provide detailed answers to your questions.\n\n---\n\n## Q1: can the model be extended to handle multiple forces at once?\n\n**Additional experiments confirm that the model successfully handles multiple simultaneous forces without requiring retraining.** This capability emerges zero-shot by simply adding multiple Gaussian blobs to the control signal videos—one for each applied force. While submission constraints prevent us from updating the supplemental materials or sharing new videos at this time, we can provide detailed descriptions of these results and will make the videos available in the next version of our project website.\n\nThe 6 scenes we tested are:\n\n* An image with 2 christmas tree ornaments\n* An image with 2 toy cars\n* An image with 2 vases, each with a flower in it\n* An image with 2 roses leaning diagonally, growing out of the ground\n* An image with 2 dandelions growing in a field\n* An image with 2 apples on separate branches of a tree\n\nTo test each image, we identified two directions where it would be reasonable to poke each object (e.g. the dandelions can each be poked to the left and to the right). Then for each image, we used 4 different “multi-poke” force prompts, representing the 2x2 different ways of combining the two different directions (e.g. poke both dandelions to the left, poke both to the right, poke one to the right and the other to the left, then poke one to the left and the other to the right). We computed one video for each, using the same seed for all of them, with no cherry-picking allowed. We then computed the “multi-poke force adherence” to be the fraction of the 8 pokes which were respected.\n\n**We found that 5 / 6 of the videos in this multi-poke benchmark had perfect force adherence, and one of them had nearly perfect force adherence.** The only failure case: in the 2 apples scene, when the left apple is poked to the right and the right apple is poked to the left, they move as if they were both poked to the right. \n\nWe want to emphasize that this worked zero-shot: no additional training to the model was required. In fact, we were prepared to train an additional model using synthetic videos of multiple balls being poked simultaneously, but decided to check if the desired behavior would work zero-shot instead. We report the results in the table below. \n\n|               Scene description        | 2 ornaments | 2 toy cars | 2 flowers in vases | 2 roses in nature | 2 dandelions | 2 apples |\n| :-------------------- | :---------- | :--------- | :----------------- | :---------------- | :----------- | :------- |\n| **Multi-poke force adherence** | 100%        | 100%       | 100%               | 100%              | 100%         | 87.5%    |\n\nAdditionally, **we have conducted new experiments and concluded that the mass understanding behavior indeed persists in the zero-shot multiple objects setting.** In more detail: we have used the GPT-Image-1 API to construct images where two objects of different apparent masses are present. The first image contains an empty laundry basket on the left side of the frame, and a full laundry basket on the right side of the frame. The second image contains a book on the left side of the frame, and a stack of books on the right side of the frame. We passed both sets of images into the multi-poke Force Prompting inference pipeline and instructed the model to poke both sets of objects towards the middle of the frame with the same force magnitude. Across 8 different force magnitudes $0.125*i,$ for  $i=1,\\dots,8$, we found that the lighter object moved much further, with the heavier item barely moving at all. \n\nWe’ll include the experiment details in the next draft of the paper, and we’ll include the videos in the next draft of the project page.\n\n---\n\n## Q2: can a unified model be trained?\n\n**Additional experiments confirm that we can train a unified model to learn both point force prompts and wind force prompts.** We have trained a unified model which loads both datasets at once and trains on them jointly. We ran inference on this joint model using our point force benchmark, as well as our wind force benchmark. Our findings:\n\n* *More dynamic backgrounds:* in many of the videos, the background moves more dynamically in a natural way. For example, in the apple tree, the surrounding leaves move more naturally after the apple is poked; in the video where a kid is sitting in the middle of a toy train track, the kid is moving more naturally while the train is moving around the track; in the video with falling leaves in the forest with a woman sitting on a chair in the background, the woman in the chair moves more while the leaves are being blown.\n* *Slightly less robust point force control:* on some of the point force videos (e.g. the blueberry bush), the control signal is not respected.\n\nIn more detail: we designed this experiment by sourcing 50% of the training data and control signals in each batch from the synthetic point force dataset, and the other 50% from the synthetic wind force dataset. We trained using the same architecture and number of training steps as the original model. We will include this ablation study in the next draft of the paper.\n\n---\n\n## Q3: request for more objective metrics\n\nWe included objective metrics (not human study metrics) to evaluate how general the mass understanding behavior is in Figure 5. That graph demonstrates that (light) soccer balls roll much further than (heavy) bowling balls, and that this phenomenon is robust across ball color, ground material, and force magnitude. We copy the numbers here as well:\n\n| Force magnitude  | 0.125  | 0.250  | 0.375  | 0.500  | 0.625  | 0.750   | 0.875   | 1.000 |\n| :-------------- | :----- | :----- | :----- | :----- | :----- | :------ | :------ | :------ |\n| Distance traveled (bowling ball) | 37.75  | 48.49  | 61.86  | 73.07  | 88.06  | 100.01  | 115.83  | 135.38  |\n| Distance traveled (soccer ball) | 62.83  | 76.21  | 91.66  | 107.72 | 129.55 | 157.94  | 178.22  | 200.92  |\n\nDistances in this table are measured in pixels.\n\n---\n\n## Q4: missing qualitative comparison as promised in B.2\n\n**Table 1** demonstrates that CogVideoX+Force Prompting indeed has better adherence to the force prompt than the base CogVideoX model. Specifically, for the point force model, Force Prompting is preferred over CogVideoX more than 67% of the time, and for the wind force model, Force Prompting is preferred more than 84% of the time. These metrics are in the “Text-only, zero-shot” row; we’ll make it more clear in the next version of the paper that these come from the CogVideoX baseline. Additionally, in the next version of the project webpage, we’ll include qualitative examples for the base CogVideoX model. \n\nNot including the comparison to prior works as promised in B.2 was an oversight on our part, we apologize. We’re unfortunately not allowed to update the supplemental, but we can describe the videos to you, and we will release them in the next draft of the website (per neurips policy, all of our promises will be made public for the community’s inspection). The 8 papers and scenes we considered are:\n\n* PhysDreamer (ECCV 2024), poking an alocacia plant\n* DreamPhysics (AAAI 2025), poking a houseplant\n* MotionCraft (NeurIPS 2024), poking a ship in the water\n* PhysGaussian (CVPR 2024), poking a bouquet of flowers\n* PhysGen (ECCV 2024), poking a toy car\n* PhysGen3D (CVPR 2025), poking a toy car made of felt\n* PhysMotion, poking a toy rocking horse\n* Physics3D, poking a bouquet of flowers\n\nFor each of those scenes, we take a screenshot of the first frame from the respective paper’s demo, and pass it through the Force Prompting model. We obtain a resulting video comparable to corresponding video from each of those sites, but without the need for any explicit 3D assets or physics simulation at inference time."}, {"review_id": "QLvQ8NAet5", "reviewer": "Reviewer_dEDd", "summary": "Force Prompting is a video generation framework that conditions a pretrained video generation model (CogVideoX) on explicit physical force signals (both global and localized forces) without requiring a physics engine at inference time. With ControlNet style finetuning, the CogVideoX-based video model can take in forces as conditions. The finetuning datasets are synthetically generated, using Blender scenes as well as an existing physics-based method (PhysDreamer). Through extensive human studies and automatic evaluations, the paper demonstrates that Force Prompting not only faithfully adheres to the specified forces but also generalizes beyond its limited synthetic training set to diverse real-world images, materials, and motions.", "questions": "How does performance improve (or plateau) if you scale up (or down) the synthetic dataset by 2×?\nHow confident are you about the mass understanding behavior? I think it cannot be conclusively determined until the method is extended to handle multiple objects, and then we can see how the model behaves when two objects of different mass are present in the same video.", "limitations": "Yes", "rating": 5, "confidence": 5, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 4, "clarity": 4, "significance": 4, "originality": 3, "strengths_and_weaknesses": "Strengths:\n- Strong sim2real generalization despite limited training data.\n- The methodology is sound, integrating a state-of-the-art video model with a lightweight ControlNet for physics conditioning.\n- Experiments span multiple benchmarks, include ablations on data design and prompt specificity.\n- Overall significance in demonstrating that one can achieve comparable visual effect to prior physics-engine-based works with a simple synthetic dataset and moderate finetuning costs (1 day of training with four A100 GPUs)\n\nWeakness:\n- Due to the nature of the method (based on video generation without modeling explicitly in 3D space), physical control (e.g. direction of forces) is not guaranteed to be adhered by the generated content by design.\n- The method is limited by the capacity of the video generation model, which might lead to suboptimal performance on out-of-domain content.\n- The length of the generated video is capped by the base model (CogVideoX).", "was_revised": true, "final_justification": "I am happy with the response provided by the authors, and I will maintain the original rating to accept.", "rebuttal": "Thank you for the thoughtful feedback. Below we provide detailed answers to your questions.\n\n---\n\n## Q1: can the method be extended to handle multiple objects?\n\n**Additional experiments confirm that the model successfully handles multiple simultaneous forces without requiring retraining.** This capability emerges zero-shot by simply adding multiple Gaussian blobs to the control signal videos—one for each applied force. While submission constraints prevent us from updating the supplemental materials or sharing new videos at this time, we can provide detailed descriptions of these results and will make the videos available in the next version of our project website.\n\nThe 6 scenes we tested are:\n\n* An image with 2 christmas tree ornaments\n* An image with 2 toy cars\n* An image with 2 vases, each with a flower in it\n* An image with 2 roses leaning diagonally, growing out of the ground\n* An image with 2 dandelions growing in a field\n* An image with 2 apples on separate branches of a tree\n\nTo test each image, we identified two directions where it would be reasonable to poke each object (e.g. the dandelions can each be poked to the left and to the right). Then for each image, we used 4 different “multi-poke” force prompts, representing the 2x2 different ways of combining the two different directions (e.g. poke both dandelions to the left, poke both to the right, poke one to the right and the other to the left, then poke one to the left and the other to the right). We computed one video for each, using the same seed for all of them, with no cherry-picking allowed. We then computed the “multi-poke force adherence” to be the fraction of the 8 pokes which were respected.\n\n**We found that 5 / 6 of the videos in this multi-poke benchmark had perfect force adherence, and one of them had nearly perfect force adherence.** The only failure case: in the 2 apples scene, when the left apple is poked to the right and the right apple is poked to the left, they move as if they were both poked to the right. \n\nWe want to emphasize that this worked zero-shot: no additional training to the model was required. In fact, we were prepared to train an additional model using synthetic videos of multiple balls being poked simultaneously, but decided to check if the desired behavior would work zero-shot instead. We report the results in the table below. \n\n|               Scene description        | 2 ornaments | 2 toy cars | 2 flowers in vases | 2 roses in nature | 2 dandelions | 2 apples |\n| :-------------------- | :---------- | :--------- | :----------------- | :---------------- | :----------- | :------- |\n| **Multi-poke force adherence** | 100%        | 100%       | 100%               | 100%              | 100%         | 87.5%    |\n\nWe’ll include the experiment details in the next draft of the paper, and we’ll include the videos in the next draft of the project page.\n\n---\n\n## Q2: does mass understanding behavior persist in the multiple objects setting?\n\n**We have conducted new experiments and concluded that the mass understanding behavior indeed persists in the zero-shot multiple objects setting.** In more detail: we have used the GPT-Image-1 API to construct images where two objects of different apparent masses are present. The first image contains an empty laundry basket on the left side of the frame, and a full laundry basket on the right side of the frame. The second image contains a book on the left side of the frame, and a stack of books on the right side of the frame. We passed both sets of images into the multi-poke Force Prompting inference pipeline and instructed the model to poke both sets of objects towards the middle of the frame with the same force magnitude. Across 8 different force magnitudes $0.125*i,$ for  $i=1,\\dots,8$, we found that the lighter object moved much further, with the heavier item barely moving at all. We will include the results on the next draft of the project page, as well as on the website.\n\n---\n\n## Q3: how does performance change if we scale up or down the synthetic training dataset?\n\n**We have conducted additional ablation studies, and we found that model robustness gets worse when you scale down the synthetic training dataset.** In more detail: we trained from scratch another wind force model. This new model uses half as many synthetic flag waving videos as the dataset that we used in the paper. The only variable that we changed was the dataset size; everything else (including the number of training steps and learning rate scheduler) remained the same.\n\nFor this 0.5x dataset model, we found some additional failure cases. *One failure case:* for the image where a woman holds a sheet on the beach, the model hallucinates a bedsheet in the background, indicating that the model has likely memorized the “waving flag” pattern from the training dataset and is injecting it into the output video. *A second failure case:* the confetti’s response to the wind isn’t as convincing, e.g. some of the confetti will blow in the wind’s direction but some will stay stationary. This indicates that the model hasn’t generalized properly, perhaps because of less training data. Unfortunately we’re not allowed to share videos during the rebuttal period, but we will include these ablation experiments in the next draft of the paper. \n\n---\n\n## Q4: method is limited by capacity of video gen model, similarly with length of its outputs\n\nThis is a good point. Our perspective is that this is a feature, rather than a bug. Our results in Table 1 indicate that Force Prompting videos are as physically realistic as the videos from the base model. This implies that as video models continue to improve, training a Force Prompting ControlNet on top of them will result in more physics capabilities. We hope that our results will become even more useful when applied to the larger, more capable video models of the future."}]

Author rebuttal:

Meta-review: This paper proposes a novel image-to-video generation framework that leverages physical forces as conditioning signals. The method demonstrates realistic physics under wind and point-force conditioning and, notably, achieves strong generalization through careful conditioning design and fine-tuning with a small amount of simulation data. Post-rebuttal, all reviewers recognized the solid results, emergent properties, and potential impact of this work in advancing physically grounded video generation. The AC concurs and recommends acceptance.
Decision: Accept (poster)