En3D: An Enhanced Generative Model for Sculpting 3D Humans from 2D
  Synthetic Data
Authors: name: Yifang Men; name: Biwen Lei; name: Yuan Yao; name: Miaomiao Cui; name: Zhouhui Lian; name: Xuansong Xie
Published: 2024-01-02 12:06:31

Abstract: We present En3D, an enhanced generative scheme for sculpting high-quality 3D
human avatars. Unlike previous works that rely on scarce 3D datasets or limited
2D collections with imbalanced viewing angles and imprecise pose priors, our
approach aims to develop a zero-shot 3D generative scheme capable of producing
visually realistic, geometrically accurate and content-wise diverse 3D humans
without relying on pre-existing 3D or 2D assets. To address this challenge, we
introduce a meticulously crafted workflow that implements accurate physical
modeling to learn the enhanced 3D generative model from synthetic 2D data.
During inference, we integrate optimization modules to bridge the gap between
realistic appearances and coarse 3D shapes. Specifically, En3D comprises three
modules: a 3D generator that accurately models generalizable 3D humans with
realistic appearance from synthesized balanced, diverse, and structured human
images; a geometry sculptor that enhances shape quality using multi-view normal
constraints for intricate human anatomy; and a texturing module that
disentangles explicit texture maps with fidelity and editability, leveraging
semantical UV partitioning and a differentiable rasterizer. Experimental
results show that our approach significantly outperforms prior works in terms
of image quality, geometry accuracy and content diversity. We also showcase the
applicability of our generated avatars for animation and editing, as well as
the scalability of our approach for content-style free adaptation.