Robot Learning from a Physical World Model
Authors: name: Jiageng Mao, name: Sicheng He, name: Hao-Ning Wu, name: Yang You, name: Shuyang Sun, name: Zhicheng Wang, name: Yanan Bao, name: Huizhong Chen, name: Leonidas Guibas, name: Vitor Guizilini, name: Howard Zhou, name: Yue Wang
Organization: Deepmind

We introduce PhysWorld, a framework that enables robot learning from video
generation through physical world modeling. Recent video generation models can
synthesize photorealistic visual demonstrations from language commands and
images, offering a powerful yet underexplored source of training signals for
robotics. However, directly retargeting pixel motions from generated videos to
robots neglects physics, often resulting in inaccurate manipulations. PhysWorld
addresses this limitation by coupling video generation with physical world
reconstruction. Given a single image and a task command, our method generates
task-conditioned videos and reconstructs the underlying physical world from the
videos, and the generated video motions are grounded into physically accurate
actions through object-centric residual reinforcement learning with the
physical world model. This synergy transforms implicit visual guidance into
physically executable robotic trajectories, eliminating the need for real robot
data collection and enabling zero-shot generalizable robotic manipulation.
Experiments on diverse real-world tasks demonstrate that PhysWorld
substantially improves manipulation accuracy compared to previous approaches.
Visit https://pointscoder.github.io/PhysWorld_Web/{the project webpage}
for details.