Chupa: Carving 3D Clothed Humans from Skinned Shape Priors using 2D
  Diffusion Probabilistic Models
Authors: name: Byungjun Kim; name: Patrick Kwon; name: Kwangho Lee; name: Myunggi Lee; name: Sookwan Han; name: Daesik Kim; name: Hanbyul Joo
Published: 2023-05-19 17:59:18

Abstract: We propose a 3D generation pipeline that uses diffusion models to generate
realistic human digital avatars. Due to the wide variety of human identities,
poses, and stochastic details, the generation of 3D human meshes has been a
challenging problem. To address this, we decompose the problem into 2D normal
map generation and normal map-based 3D reconstruction. Specifically, we first
simultaneously generate realistic normal maps for the front and backside of a
clothed human, dubbed dual normal maps, using a pose-conditional diffusion
model. For 3D reconstruction, we ``carve'' the prior SMPL-X mesh to a detailed
3D mesh according to the normal maps through mesh optimization. To further
enhance the high-frequency details, we present a diffusion resampling scheme on
both body and facial regions, thus encouraging the generation of realistic
digital avatars. We also seamlessly incorporate a recent text-to-image
diffusion model to support text-based human identity control. Our method,
namely, Chupa, is capable of generating realistic 3D clothed humans with better
perceptual quality and identity variety.