SlowGuess/NeurIPS_2023_OCR huggingface dataset:
Choice of diffusion model. Our method is not tied to a particular diffusion model. To demonstrate this, we substitute Imagen [55] with the open-source Stable Diffusion model [52]. Stable Diffusion is a latent diffusion model, so at each training iteration we use our NeRF to render an RGB image, pass it through the latent encoder, and then apply the SDS loss on the latent embeddings. In Figure 6, we show qualitative results of our method trained using Stable Diffusion. Overall we observe that we are able to generate high quality avatars, regardless of the choice of the diffusion model.
Figure 6: Generation results using Stable Diffusion.
4.2 Comparison with the state of the art
Qualitative Evaluations. In Figure 7, we show a qualitative comparison of our method with DreamFusion. DreamFusion suffers from limited control over generation. Even though it was prompted to generate the full body of the subject, very often it produces a 3D model of the upper body, or the head. At the same time it cannot properly disentangle the human subject from other objects in the scene, resulting in 3D models that contain parts of the environment. More importantly though, it very often produces unpleasant visual artifacts, such as non-realistic body proportions, missing or multiple limbs, as well as degenerate geometry that can be attributed to viewpoint overfitting. Our method is able to overcome these issues by utilizing a strong 3D prior on human body geometry. For more comparisons we refer the readers to the Supplementary Material.
Figure 8 shows a comparison between DreamHuman and AvATARCLIP. We can see that our method is able to generate significantly better geometry and texture quality. The geometry of the reconstructed avatars with AvATARCLIP is very close to the underlying body model geometry, with only minor modifications. As a result, it cannot handle loose-fitting clothing, dresses, and accessories like hats. The model textures from AvATARCLIP also have significant artifacts and do not match the realism and overall quality of DreamHuman in all examples we tried.
Quantitative evaluation using CLIP. Following common practice, we also use CLIP to evaluate the alignment of the rendered 3D models with the input text prompts. We use a total of 160 prompts with descriptions of people. The results are shown in Table 1. We can see that our method consistently outperforms DreamFusion. We also include comparisons with AvatarCLIP. However, note that AvatarCLIP is trained using CLIP as supervision, meaning that the CLIP-based metrics are biased
A professional boxer
A policewoman
Figure 7: Comparison with DreamFusion. For each example we show the rendered 3D model as well as the corresponding surface normals. Both methods were asked to reconstruct the full body of the subject by prepending A DSLR full body photo to the prompt.
Figure 8: Comparison with AvatarCLIP. We compare DreamHuman with AvatarCLIP [18]. From left to right we used the following prompts: astronaut, construction manager, firefighter, gardener, pilot, police officer, robot, senior citizen, soldier, teenager, warrior, witch, wizard. Notice that our method generates much more realistic texture and geometry. All illustrations are in default A-Pose.
favorably towards AvatarCLIP. This means that a direct comparison to other methods is potentially unfair.
Table 1: Evaluation of the rendered 3D models using CLIP. We report the R-Precision as well as whether the true caption is in the top 3 and 5 highest-scoring captions.
Method	R-Precision ↑	Top-3 ↑	Top-5 ↑
DreamFusion [43]	0.775	0.888	0.925
AvatarCLIP [18]	0.855	0.962	0.981
Ours	0.838	0.931	0.956
User study. We conducted a user study to assess the quality of our 3D avatar generation pipeline. We used 20 text prompts from the AvatarCLIP website, selected from the General Description category. We ran our method on those 20 text prompts and rendered the final results in the rest pose, from the front and from the side. We did the same for the precomputed meshes that the AvatarCLIP authors provide on their website. We then asked the users to rate the two methods on (a) the perceived agreement between the renderings and the input text, and (b) the perceived visual quality of the generated avatar. The ratings were on a scale from 1-5, with 1 meaning Very Bad and 5 Very Good. The results in Table 2 show that the raters consistently preferred our method over AvatarCLIP.
4.3 Generation variability.
Here we investigate whether there is variation in our generations for the same prompt, but with different random seeds. As originally observed in DreamFusion [43], there is limited variation in the generations, probably because the SDS loss is mode-seeking, so it tends to latch on to specific modes of the distribution. Our results demonstrate some level of variability, as seen on Figure 9.
Table 2: User study on 3D avatar generation quality.
Method	Agreement with text ↑	Visual quality ↑
AvatarCLIP [18]	2.96	2.49
Ours	4.45	4.16
A man wearing a camel coat and a hat
coat and a hat
A man wearing a Christmas sweater
A clown
Figure 9: Generations using different random seeds. For each example we show the generated 3D avatar using 2 different random seeds as initialization.
A woman wearing a long dress
A cowboy
An Asian woman wearing a life jacket
ing a life jacket
We presented *DreamHuman*, a novel method for generating 3D human avatars from text. Our method leverages statistical 3D human body models and recent advances in 3D modelling and text-to-3D generation to create animatable 3D human avatars, without any paired text-to-3D supervision. We illustrated that our method can generate photorealistic results, with detailed geometry and outperforms the state of the art by a large margin.
Limitations and future work. Since our model is trained without any 3D data, it sometimes draws fine details like wrinkles using the albedo map instead of creating them based on geometry. Future work can address this by leveraging 3D data to resolve some of the reconstruction ambiguities. Additionally, the model sometimes cannot properly disentangle albedo from shading, resulting in baked reflections and shadows. Current computational constraints from the diffusion models prevent us from scaling the method to very high resolution textures and geometric detail like hair. Finally, the realism of clothing animation can benefit from a video model.
Broader Impact. While our method does not use any additional training data, it relies in part on text-to-image diffusion models which have been pre-trained on large-scale datasets containing sometimes insufficiently curated images and captions [7] (N.B. the level of effective automation to guarantee the removal of undesired content is considerable for most models we use in this paper). Also, text-to-image generators use LLMs for the text encoder, pre-trained on uncurated internet-scale data. Although our method uses statistical 3D human body shape models learnt using highly curated and diverse data to remove bias, ultimately our generation process may be vulnerable to some bias in its dependencies.
The goal of our method is to generate 3D models of people, which has the potential to be misused in connection with fake media. However, it is important to highlight that our rendered 3D human models are typically less realistic than their 2D-generated counterparts. Regardless, in practical settings, safeguards should be used to prevent abuse, such as filtering the input text prompts and detecting any unsafe content in the model renderings.
Our method has the ability to generate people with diverse body shapes, appearance, skin color and clothing. This can enable the generation of diverse large-scale synthetic 3D datasets for human-related tasks, and in turn may support training models with fairer outcomes across different groups.
DreamHuman can potentially augment the work of artists and other creative professionals. It could be used as a complementary tool to boost productivity. It also has the potential to democratize 3D content creation that currently requires specialized knowledge and expensive proprietary software.
[58] Jascha Sohl-Dickstein, Eric Weiss, Niru Maheswaranathan, and Surya Ganguli. Deep unsupervised learning using nonequilibrium thermodynamics. In International Conference on Machine Learning, pages 2256-2265. PMLR, 2015.
[59] Guy Tevet, Brian Gordon, Amir Hertz, Amit H Bermano, and Daniel Cohen-Or. Motionclip: Exposing human motion generation to clip space. In Computer Vision-ECCV 2022: 17th European Conference, Tel Aviv, Israel, October 23–27, 2022, Proceedings, Part XXII, pages 358–374. Springer, 2022.
[60] Guy Tevet, Sigal Raab, Brian Gordon, Yonatan Shafir, Amit H Bermano, and Daniel Cohen-Or. Human motion diffusion model. *ICLR*, 2023.
[61] Edgar Tretschk, Ayush Tewari, Vladislav Golyanik, Michael Zollhöfer, Christoph Lassner, and Christian Theobalt. Non-rigid neural radiance fields: Reconstruction and novel view synthesis of a dynamic scene from monocular video. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pages 12959-12970, 2021.
[62] Dor Verbin, Peter Hedman, Ben Mildenhall, Todd Zickler, Jonathan T. Barron, and Pratul P. Srinivasan. Ref-NeRF: Structured view-dependent appearance for neural radiance fields. CVPR, 2022.
[63] Chung-Yi Weng, Brian Curless, Pratul P. Srinivasan, Jonathan T. Barron, and Ira Kemelmacher-Shlizerman. HumanNeRF: Free-viewpoint rendering of moving people from monocular video. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pages 16210-16220, June 2022.
[64] Hongyi Xu, Eduard Gabriel Bazavan, Andrei Zanfir, William T Freeman, Rahul Sukthankar, and Cristian Sminchisescu. Ghum & ghuml: Generative 3d human shape and articulated pose models. In CVPR, 2020.
[65] Hongyi Xu, Thiemo Alldieck, and Cristian Sminchisescu. H-nerf: Neural radiance fields for rendering and temporal reconstruction of humans in motion. Advances in Neural Information Processing Systems, 34: 14955–14966, 2021.
[66] Yuelang Xu, Hongwen Zhang, Lizhen Wang, Xiaochen Zhao, Huang Han, Qi Guojun, and Yebin Liu. Latentavatar: Learning latent expression code for expressive neural head avatar. In ACM SIGGRAPH 2023 Conference Proceedings, 2023.
[67] Jiahui Yu, Yuanzhong Xu, Jing Yu Koh, Thang Luong, Gunjan Baid, Zirui Wang, Vijay Vasudevan, Alexander Ku, Yinfei Yang, Burcu Karagol Ayan, et al. Scaling autoregressive models for content-rich text-to-image generation. arXiv preprint arXiv:2206.10789, 2022.
[68] Andrei Zanfir, Eduard Gabriel Bazavan, Hongyi Xu, William T Freeman, Rahul Sukthankar, and Cristian Sminchisescu. Weakly supervised 3d human pose and shape reconstruction with normalizing flows. In Computer Vision-ECCV 2020: 16th European Conference, Glasgow, UK, August 23-28, 2020, Proceedings, Part VI 16, pages 465-481. Springer, 2020.
[69] Fuqiang Zhao, Wei Yang, Jiakai Zhang, Pei Lin, Yingliang Zhang, Jingyi Yu, and Lan Xu. Humannerf: Efficiently generated human radiance field from sparse inputs. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pages 7743-7753, June 2022.