Sketch3DVE: Sketch-based 3D-Aware Scene Video Editing
Authors: name: Feng-Lin Liu, name: Shi-Yang Li, name: Yan-Pei Cao, name: Hongbo Fu, name: Lin Gao

Recent video editing methods achieve attractive results in style transfer or
appearance modification. However, editing the structural content of 3D scenes
in videos remains challenging, particularly when dealing with significant
viewpoint changes, such as large camera rotations or zooms. Key challenges
include generating novel view content that remains consistent with the original
video, preserving unedited regions, and translating sparse 2D inputs into
realistic 3D video outputs. To address these issues, we propose Sketch3DVE, a
sketch-based 3D-aware video editing method to enable detailed local
manipulation of videos with significant viewpoint changes. To solve the
challenge posed by sparse inputs, we employ image editing methods to generate
edited results for the first frame, which are then propagated to the remaining
frames of the video. We utilize sketching as an interaction tool for precise
geometry control, while other mask-based image editing methods are also
supported. To handle viewpoint changes, we perform a detailed analysis and
manipulation of the 3D information in the video. Specifically, we utilize a
dense stereo method to estimate a point cloud and the camera parameters of the
input video. We then propose a point cloud editing approach that uses depth
maps to represent the 3D geometry of newly edited components, aligning them
effectively with the original 3D scene. To seamlessly merge the newly edited
content with the original video while preserving the features of unedited
regions, we introduce a 3D-aware mask propagation strategy and employ a video
diffusion model to produce realistic edited videos. Extensive experiments
demonstrate the superiority of Sketch3DVE in video editing. Homepage and code:
http://http://geometrylearning.com/Sketch3DVE/