Title: Force Prompting: Video Generation Models Can Learn And Generalize Physics-based Control Signals
Venue: NeurIPS 2025
Accepted: yes
Keywords: controllable video generation

physical forces

synthetic data
Abstract: Recent advances in video generation models have sparked interest in world models capable of simulating realistic environments.
While navigation has been well-explored, physically meaningful interactions that mimic real-world forces remain largely understudied. 
In this work, we investigate using physical forces as a control signal for video generation and propose force prompts which enable users to interact with images through both localized point forces, such as poking a plant, and global wind force fields, such as wind blowing on fabric. We demonstrate that these force prompts can enable videos to respond realistically to physical control signals by leveraging the physical prior in the original pretrained model, without using any 3D asset or physics simulator at inference. The primary challenge of force prompting is the difficulty in obtaining high quality paired force-video training data, both in the real world due to the difficulty of obtaining force signals, and in synthetic data due to limitations in the visual quality and domain diversity of physics simulators. Our key finding is that video generation models can *generalize* remarkably well when adapted to follow physical force conditioning from videos synthesized by Blender, even with limited demonstrations of few objects (e.g., flying flags, rolling balls, etc.). Our method can generate videos which simulate forces across diverse geometries, settings, and materials. We also try to understand the source of this generalization and perform ablations on the training data that reveal two key elements: visual diversity and the use of specific text keywords during training. Our approach is trained on only around 15k training examples for a single day on four A100 GPUs, and outperforms existing methods on force adherence and physics realism, bringing world models closer to real-world physics interactions. All datasets, code, and model weights will be open-sourced. Video examples can be found at https://sites.google.com/view/force-prompting-neurips2025


Peer reviews:
{"IS_META_REVIEW": false, "comments": "SUMMARY:\nForce Prompting is a video generation framework that conditions a pretrained video generation model (CogVideoX) on explicit physical force signals (both global and localized forces) without requiring a physics engine at inference time. With ControlNet style finetuning, the CogVideoX-based video model can take in forces as conditions. The finetuning datasets are synthetically generated, using Blender scenes as well as an existing physics-based method (PhysDreamer). Through extensive human studies and automatic evaluations, the paper demonstrates that Force Prompting not only faithfully adheres to the specified forces but also generalizes beyond its limited synthetic training set to diverse real-world images, materials, and motions.\n\nSTRENGTHS AND WEAKNESSES:\nStrengths:\n- Strong sim2real generalization despite limited training data.\n- The methodology is sound, integrating a state-of-the-art video model with a lightweight ControlNet for physics conditioning.\n- Experiments span multiple benchmarks, include ablations on data design and prompt specificity.\n- Overall significance in demonstrating that one can achieve comparable visual effect to prior physics-engine-based works with a simple synthetic dataset and moderate finetuning costs (1 day of training with four A100 GPUs)\n\nWeakness:\n- Due to the nature of the method (based on video generation without modeling explicitly in 3D space), physical control (e.g. direction of forces) is not guaranteed to be adhered by the generated content by design.\n- The method is limited by the capacity of the video generation model, which might lead to suboptimal performance on out-of-domain content.\n- The length of the generated video is capped by the base model (CogVideoX).\n\nQUESTIONS:\nHow does performance improve (or plateau) if you scale up (or down) the synthetic dataset by 2×?\nHow confident are you about the mass understanding behavior? I think it cannot be conclusively determined until the method is extended to handle multiple objects, and then we can see how the model behaves when two objects of different mass are present in the same video.\n\nLIMITATIONS:\nYes", "TITLE": "Official Review", "RECOMMENDATION": 5, "REVIEWER_CONFIDENCE": 5, "CLARITY": 4, "IMPACT": 4, "ORIGINALITY": 3, "DATE": "2025-07-01"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe paper proposes a controllable image-to-video generation model conditioned on input forces, which include a point-force control model and global wind force model. To train the model, the author collected synthetic datasets using blender and found a limited amount of synthetic data is enough for training. The trained model also demonstrate good generalization across motion types and settings. User study shows the effectiveness of proposed method.\n\nSTRENGTHS AND WEAKNESSES:\n**Strengths**:\n\n1. **Simple idea, good results**\n\nThe idea in the paper is simple but effective, and the visual results generated by the trained model are great. \n\n2. **Synthetic dataset**\n\nThe model is fine-tuned using synthetic data, and the trained model could generalize to different real settings. It looks promising and provides good insight to the video generation community. \n\n3. **Good experiments and ablation**\n\nI appreciate the abundant experiments and ablation in the paper, and those summarized findings are interesting and helpful.\n\n**Weakness:**\n\n1. **Insufficient metrics**\n\nAll quantitative evaluation used in the paper are human user study, which is quite subjective. I would like to see some objective metrics. As the training using synthetic dataset, a straight way is to compare the generated result and simulated results (like moving ball) and evaluate the material influence, and mass understanding in a more general and rigorous way. Otherwise it could be hard to tell the results are occasion for certain examples or indeed some common patterns behind it. \n\n2. **Missing qualitative comparison to different methods**\n\nI didn't find the video results and qualitative comparison in B.2 and attached supp as claimed in the paper. The comparison to baseline cogvideox is necessary as it could show the improvement of introducing such explicit control signal. \n\n3. **Limited setting**\n\nFrom current results, it seems the model could only model one kind of force for each type of input, and only moving one object with simple motion (no interaction with the scene), the setting is quite constrained under my view.\n\nQUESTIONS:\n1. **Interactive motion**\n\nI wonder could such controllable model applied to model the interaction between the objects which applied force on and rest objects in the scene. For example, in the ball in the scene, can the model handle the ball collision when applying force on one ball towards the other. \n\n\n2. **Unified model**\n\nI am curious if possible to train a unified model that could handle both forces. The unified model is more helpful as current each model seems a little overfitting to certain scenarios. If trained in a unified way, will the model performance degrade and will the generalization ability disappear. \n\n3. **Multi-objects**\n\nIn all existing demos shown in the paper, there is only one single object being affected by the force. I understand the author only collect a dataset with single objects, but the proposed control signal should have the ability to model multi objects at once (like simultaneously move multi balls), otherwise the apply setting is little weak, just like physgaussian, I think video generation models should have stronger capacity.\n\nLIMITATIONS:\nyes", "TITLE": "Official Review", "RECOMMENDATION": 4, "REVIEWER_CONFIDENCE": 4, "CLARITY": 4, "IMPACT": 3, "ORIGINALITY": 3, "DATE": "2025-07-02"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe paper presents a study on learning video generation models from generalized physics-based control signals. Unlike previous force-controlled video generation methods that rely on integrated physical simulators, this work adopts a fully data-driven approach, enabling controllable interaction with images through force inputs. The authors introduced two types of force models: localized point forces and global wind force fields. To train their model, they constructed a dataset using diverse physical simulators from multiple sources. Extensive experiments demonstrated the model’s strong generalization capabilities in video generation.\n\nSTRENGTHS AND WEAKNESSES:\nStrengths\n\n1. The paper proposes a fully data-driven approach that enables controllable interaction with visual content through force-based inputs.\n2. A diverse dataset is constructed using multiple physical simulators, enhancing the robustness and variety of training data.\n3. Extensive experiments convincingly demonstrate the model’s strong generalization ability in video generation tasks.\n\nWeaknesses\n\n1. The paper relies too heavily on existing paired force-video datasets and does not take advantage of larger force-enabled but unlabeled (unsupervised) datasets.\n2. Line 145–146: What exactly does “HDRIs” refer to? Does it stand for High Dynamic Range Images? A brief explanation would be helpful for clarity.\n3. In the construction of the local force dataset, the authors mention simulating a rolling ball in response to unseen point-wise forces using Blender. This raises a question: why was Blender chosen over other platforms like Unity or more advanced physics simulators?\n4. A small portion of the dataset is collected using PhysDreamer. Given the limitations of existing simulators, will the proposed method face bottlenecks related to data diversity or label quality? How do the authors ensure high-quality, task-relevant force annotations for such specific downstream applications?\n5. In Table 1, why is PhysDreamer not included as a baseline for comparison?\n6. In Table 2, the authors compare their method with PhysDreamer, but the explanation of the comparison is somewhat lacking. Since generalization is a key strength of this work, more evidence and clarification on how the comparisons were conducted would be valuable. Were the evaluation metrics and testing scenarios consistent?\n7. While the authors mention alternative approaches involving physical simulator integration, it’s unclear why comparisons with other existing methods such as PhysGen are not included. Could the authors elaborate on this choice?\n8. Additionally, many of the referenced figures or supplementary materials in the paper do not hyperlink correctly when clicked, which affects the ease of review.\n\nQUESTIONS:\n1. Line 145–146: What exactly does “HDRIs” refer to? Does it stand for High Dynamic Range Images? A brief explanation would be helpful for clarity.\n2. In Table 1, why is PhysDreamer not included as a baseline for comparison?\n3. In Table 2, the authors compare their method with PhysDreamer, but the explanation of the comparison is somewhat lacking. Since generalization is a key strength of this work, more evidence and clarification on how the comparisons were conducted would be valuable. Were the evaluation metrics and testing scenarios consistent?\n4. To be honest, I had difficulty fully understanding the meaning of localized point forces and global force fields. Could the authors clarify why these terms were chosen? Are there more standard or physics-friendly terms that could be used to describe them?\n\nLIMITATIONS:\n1. The paper relies too heavily on existing paired force-video datasets and does not take advantage of larger force-enabled but unlabeled (unsupervised) datasets.\n2. While the authors mention alternative approaches involving physical simulator integration, it’s unclear why comparisons with other existing methods such as PhysGen are not included. Could the authors elaborate on this choice?", "TITLE": "Official Review", "RECOMMENDATION": 5, "REVIEWER_CONFIDENCE": 4, "CLARITY": 3, "IMPACT": 3, "ORIGINALITY": 3, "DATE": "2025-07-03"}

comments: SUMMARY:
This article introduces physical forces as novel conditioning signals for video generation models. The core insight is that pretrained video models possess latent intuitive physics priors that can be elicited via fine-tuning on limited synthetic data. There are several innovations. Physics-based control encoding, Sim2Real generalization and emergent mass understanding. Human evaluations demonstrate superior force adherence and realism over text-conditioned baselines and trajectory-based methods.

STRENGTHS AND WEAKNESSES:
Strength

1. The authors carried out a novel and interesting task, and it achieved certain results. Additionally, they provide codes in supplementary materials.

2. The experimental results are very abundant, with a wide variety of objects, and the experimental data and network parameters descriptions are also very detailed.

3. Human evaluation shown in Table 1 reflects the effectiveness of proposed model.

Weakness

1. In some cases, such as in the example of a scene where the wind blows, usually only the target object moves while the background remains stationary. This does not conform to the principles of physics.

2. Figure 3 is hard to follow. What do the blue lines in the last two columns mean?

3. The presented model has poor performance on real physics, such as fluid dynamics. As shown in Figure 4, the diffusion of smoke affected by the wind is very unrealistic. The smoke emerging from the cigarette end cannot be straight, and the inertia is not so significant.

QUESTIONS:
Please refer to weakness.

LIMITATIONS:
Please refer to weakness.
TITLE: Official Review
CLARITY: 3
IMPACT: 2
ORIGINALITY: 2
DATE: 2025-07-05