Shape-for-Motion: Precise and Consistent Video Editing with 3D Proxy
Authors: name: Yuhao Liu, name: Tengfei Wang, name: Fang Liu, name: Zhenwei Wang, name: Rynson W. H. Lau

Recent advances in deep generative modeling have unlocked unprecedented
opportunities for video synthesis. In real-world applications, however, users
often seek tools to faithfully realize their creative editing intentions with
precise and consistent control. Despite the progress achieved by existing
methods, ensuring fine-grained alignment with user intentions remains an open
and challenging problem. In this work, we present Shape-for-Motion, a novel
framework that incorporates a 3D proxy for precise and consistent video
editing. Shape-for-Motion achieves this by converting the target object in the
input video to a time-consistent mesh, i.e., a 3D proxy, allowing edits to be
performed directly on the proxy and then inferred back to the video frames. To
simplify the editing process, we design a novel Dual-Propagation Strategy that
allows users to perform edits on the 3D mesh of a single frame, and the edits
are then automatically propagated to the 3D meshes of the other frames. The 3D
meshes for different frames are further projected onto the 2D space to produce
the edited geometry and texture renderings, which serve as inputs to a
decoupled video diffusion model for generating edited results. Our framework
supports various precise and physically-consistent manipulations across the
video frames, including pose editing, rotation, scaling, translation, texture
modification, and object composition. Our approach marks a key step toward
high-quality, controllable video editing workflows. Extensive experiments
demonstrate the superiority and effectiveness of our approach. Project page:
https://shapeformotion.github.io/