Force Prompting: Video Generation Models Can Learn and Generalize
  Physics-based Control Signals
Authors: name: Nate Gillman, name: Charles Herrmann, name: Michael Freeman, name: Daksh Aggarwal, name: Evan Luo, name: Deqing Sun, name: Chen Sun

Recent advances in video generation models have sparked interest in world
models capable of simulating realistic environments. While navigation has been
well-explored, physically meaningful interactions that mimic real-world forces
remain largely understudied. In this work, we investigate using physical forces
as a control signal for video generation and propose force prompts which enable
users to interact with images through both localized point forces, such as
poking a plant, and global wind force fields, such as wind blowing on fabric.
We demonstrate that these force prompts can enable videos to respond
realistically to physical control signals by leveraging the visual and motion
prior in the original pretrained model, without using any 3D asset or physics
simulator at inference. The primary challenge of force prompting is the
difficulty in obtaining high quality paired force-video training data, both in
the real world due to the difficulty of obtaining force signals, and in
synthetic data due to limitations in the visual quality and domain diversity of
physics simulators. Our key finding is that video generation models can
generalize remarkably well when adapted to follow physical force conditioning
from videos synthesized by Blender, even with limited demonstrations of few
objects. Our method can generate videos which simulate forces across diverse
geometries, settings, and materials. We also try to understand the source of
this generalization and perform ablations that reveal two key elements: visual
diversity and the use of specific text keywords during training. Our approach
is trained on only around 15k training examples for a single day on four A100
GPUs, and outperforms existing methods on force adherence and physics realism,
bringing world models closer to real-world physics interactions. We release all
datasets, code, weights, and interactive video demos at our project page.