Towards Realistic Example-based Modeling via 3D Gaussian Stitching
Authors: name: Xinyu Gao; name: Ziyi Yang; name: Bingchen Gong; name: Xiaoguang Han; name: Sipeng Yang; name: Xiaogang Jin
Published: 2024-08-28 11:13:27

Abstract: Using parts of existing models to rebuild new models, commonly termed as
example-based modeling, is a classical methodology in the realm of computer
graphics. Previous works mostly focus on shape composition, making them very
hard to use for realistic composition of 3D objects captured from real-world
scenes. This leads to combining multiple NeRFs into a single 3D scene to
achieve seamless appearance blending. However, the current SeamlessNeRF method
struggles to achieve interactive editing and harmonious stitching for
real-world scenes due to its gradient-based strategy and grid-based
representation. To this end, we present an example-based modeling method that
combines multiple Gaussian fields in a point-based representation using
sample-guided synthesis. Specifically, as for composition, we create a GUI to
segment and transform multiple fields in real time, easily obtaining a
semantically meaningful composition of models represented by 3D Gaussian
Splatting (3DGS). For texture blending, due to the discrete and irregular
nature of 3DGS, straightforwardly applying gradient propagation as SeamlssNeRF
is not supported. Thus, a novel sampling-based cloning method is proposed to
harmonize the blending while preserving the original rich texture and content.
Our workflow consists of three steps: 1) real-time segmentation and
transformation of a Gaussian model using a well-tailored GUI, 2) KNN analysis
to identify boundary points in the intersecting area between the source and
target models, and 3) two-phase optimization of the target model using
sampling-based cloning and gradient constraints. Extensive experimental results
validate that our approach significantly outperforms previous works in terms of
realistic synthesis, demonstrating its practicality. More demos are available
at https://ingra14m.github.io/gs_stitching_website.