PhysDreamer: Physics-Based Interaction with 3D Objects via Video
  Generation
Authors: name: Tianyuan Zhang; name: Hong-Xing Yu; name: Rundi Wu; name: Brandon Y. Feng; name: Changxi Zheng; name: Noah Snavely; name: Jiajun Wu; name: William T. Freeman
Published: 2024-04-19 17:41:05

Abstract: Realistic object interactions are crucial for creating immersive virtual
experiences, yet synthesizing realistic 3D object dynamics in response to novel
interactions remains a significant challenge. Unlike unconditional or
text-conditioned dynamics generation, action-conditioned dynamics requires
perceiving the physical material properties of objects and grounding the 3D
motion prediction on these properties, such as object stiffness. However,
estimating physical material properties is an open problem due to the lack of
material ground-truth data, as measuring these properties for real objects is
highly difficult. We present PhysDreamer, a physics-based approach that endows
static 3D objects with interactive dynamics by leveraging the object dynamics
priors learned by video generation models. By distilling these priors,
PhysDreamer enables the synthesis of realistic object responses to novel
interactions, such as external forces or agent manipulations. We demonstrate
our approach on diverse examples of elastic objects and evaluate the realism of
the synthesized interactions through a user study. PhysDreamer takes a step
towards more engaging and realistic virtual experiences by enabling static 3D
objects to dynamically respond to interactive stimuli in a physically plausible
manner. See our project page at https://physdreamer.github.io/.