3D Gaussian Editing with A Single Image
Authors: name: Guan Luo; name: Tian-Xing Xu; name: Ying-Tian Liu; name: Xiao-Xiong Fan; name: Fang-Lue Zhang; name: Song-Hai Zhang
Published: 2024-08-14 13:17:42

Abstract: The modeling and manipulation of 3D scenes captured from the real world are
pivotal in various applications, attracting growing research interest. While
previous works on editing have achieved interesting results through
manipulating 3D meshes, they often require accurately reconstructed meshes to
perform editing, which limits their application in 3D content generation. To
address this gap, we introduce a novel single-image-driven 3D scene editing
approach based on 3D Gaussian Splatting, enabling intuitive manipulation via
directly editing the content on a 2D image plane. Our method learns to optimize
the 3D Gaussians to align with an edited version of the image rendered from a
user-specified viewpoint of the original scene. To capture long-range object
deformation, we introduce positional loss into the optimization process of 3D
Gaussian Splatting and enable gradient propagation through reparameterization.
To handle occluded 3D Gaussians when rendering from the specified viewpoint, we
build an anchor-based structure and employ a coarse-to-fine optimization
strategy capable of handling long-range deformation while maintaining
structural stability. Furthermore, we design a novel masking strategy to
adaptively identify non-rigid deformation regions for fine-scale modeling.
Extensive experiments show the effectiveness of our method in handling
geometric details, long-range, and non-rigid deformation, demonstrating
superior editing flexibility and quality compared to previous approaches.