StdGEN: Semantic-Decomposed 3D Character Generation from Single Images
Authors: name: Yuze He; name: Yanning Zhou; name: Wang Zhao; name: Zhongkai Wu; name: Kaiwen Xiao; name: Wei Yang; name: Yong-Jin Liu; name: Xiao Han
Published: 2024-11-08 17:54:18

Abstract: We present StdGEN, an innovative pipeline for generating semantically
decomposed high-quality 3D characters from single images, enabling broad
applications in virtual reality, gaming, and filmmaking, etc. Unlike previous
methods which struggle with limited decomposability, unsatisfactory quality,
and long optimization times, StdGEN features decomposability, effectiveness and
efficiency; i.e., it generates intricately detailed 3D characters with
separated semantic components such as the body, clothes, and hair, in three
minutes. At the core of StdGEN is our proposed Semantic-aware Large
Reconstruction Model (S-LRM), a transformer-based generalizable model that
jointly reconstructs geometry, color and semantics from multi-view images in a
feed-forward manner. A differentiable multi-layer semantic surface extraction
scheme is introduced to acquire meshes from hybrid implicit fields
reconstructed by our S-LRM. Additionally, a specialized efficient multi-view
diffusion model and an iterative multi-layer surface refinement module are
integrated into the pipeline to facilitate high-quality, decomposable 3D
character generation. Extensive experiments demonstrate our state-of-the-art
performance in 3D anime character generation, surpassing existing baselines by
a significant margin in geometry, texture and decomposability. StdGEN offers
ready-to-use semantic-decomposed 3D characters and enables flexible
customization for a wide range of applications. Project page:
https://stdgen.github.io