Towards Physical Understanding in Video Generation: A 3D Point
  Regularization Approach
Authors: name: Yunuo Chen; name: Junli Cao; name: Anil Kag; name: Vidit Goel; name: Sergei Korolev; name: Chenfanfu Jiang; name: Sergey Tulyakov; name: Jian Ren
Published: 2025-02-05 21:49:06

Abstract: We present a novel video generation framework that integrates 3-dimensional
geometry and dynamic awareness. To achieve this, we augment 2D videos with 3D
point trajectories and align them in pixel space. The resulting 3D-aware video
dataset, PointVid, is then used to fine-tune a latent diffusion model, enabling
it to track 2D objects with 3D Cartesian coordinates. Building on this, we
regularize the shape and motion of objects in the video to eliminate undesired
artifacts, \eg, nonphysical deformation. Consequently, we enhance the quality
of generated RGB videos and alleviate common issues like object morphing, which
are prevalent in current video models due to a lack of shape awareness. With
our 3D augmentation and regularization, our model is capable of handling
contact-rich scenarios such as task-oriented videos. These videos involve
complex interactions of solids, where 3D information is essential for
perceiving deformation and contact. Furthermore, our model improves the overall
quality of video generation by promoting the 3D consistency of moving objects
and reducing abrupt changes in shape and motion.