SyncHuman: Synchronizing 2D and 3D Generative Models for Single-view
  Human Reconstruction
Authors: name: Wenyue Chen, name: Peng Li, name: Wangguandong Zheng, name: Chengfeng Zhao, name: Mengfei Li, name: Yaolong Zhu, name: Zhiyang Dou, name: Ronggang Wang, name: Yuan Liu

Photorealistic 3D full-body human reconstruction from a single image is a
critical yet challenging task for applications in films and video games due to
inherent ambiguities and severe self-occlusions. While recent approaches
leverage SMPL estimation and SMPL-conditioned image generative models to
hallucinate novel views, they suffer from inaccurate 3D priors estimated from
SMPL meshes and have difficulty in handling difficult human poses and
reconstructing fine details. In this paper, we propose SyncHuman, a novel
framework that combines 2D multiview generative model and 3D native generative
model for the first time, enabling high-quality clothed human mesh
reconstruction from single-view images even under challenging human poses.
Multiview generative model excels at capturing fine 2D details but struggles
with structural consistency, whereas 3D native generative model generates
coarse yet structurally consistent 3D shapes. By integrating the complementary
strengths of these two approaches, we develop a more effective generation
framework. Specifically, we first jointly fine-tune the multiview generative
model and the 3D native generative model with proposed pixel-aligned 2D-3D
synchronization attention to produce geometrically aligned 3D shapes and 2D
multiview images. To further improve details, we introduce a feature injection
mechanism that lifts fine details from 2D multiview images onto the aligned 3D
shapes, enabling accurate and high-fidelity reconstruction. Extensive
experiments demonstrate that SyncHuman achieves robust and photo-realistic 3D
human reconstruction, even for images with challenging poses. Our method
outperforms baseline methods in geometric accuracy and visual fidelity,
demonstrating a promising direction for future 3D generation models.