RodinHD: High-Fidelity 3D Avatar Generation with Diffusion Models
Authors: name: Bowen Zhang; name: Yiji Cheng; name: Chunyu Wang; name: Ting Zhang; name: Jiaolong Yang; name: Yansong Tang; name: Feng Zhao; name: Dong Chen; name: Baining Guo
Published: 2024-07-09 15:14:45

Abstract: We present RodinHD, which can generate high-fidelity 3D avatars from a
portrait image. Existing methods fail to capture intricate details such as
hairstyles which we tackle in this paper. We first identify an overlooked
problem of catastrophic forgetting that arises when fitting triplanes
sequentially on many avatars, caused by the MLP decoder sharing scheme. To
overcome this issue, we raise a novel data scheduling strategy and a weight
consolidation regularization term, which improves the decoder's capability of
rendering sharper details. Additionally, we optimize the guiding effect of the
portrait image by computing a finer-grained hierarchical representation that
captures rich 2D texture cues, and injecting them to the 3D diffusion model at
multiple layers via cross-attention. When trained on 46K avatars with a noise
schedule optimized for triplanes, the resulting model can generate 3D avatars
with notably better details than previous methods and can generalize to
in-the-wild portrait input.