DreamHuman: Animatable 3D Avatars from Text
Authors: Nikos Kolotouros
Thiemo Alldieck
Andrei Zanfir
Eduard Gabriel Bazavan
Mihai Fieraru
Cristian Sminchisescu
Venue: NeurIPS 2023 spotlight (2023)
Keywords: text to 3d; 3d avatars
TL;DR: Generate Controllable 3D Avatars from Textual Descriptions

Abstract: We present \emph{DreamHuman}, a method to generate realistic animatable 3D human avatar models entirely from textual descriptions. Recent text-to-3D methods have made considerable strides in generation, but are still lacking in important aspects. Control and often spatial resolution remain limited, existing methods produce fixed rather than 3D human models that can be placed in different poses (i.e. re-posable or animatable), and anthropometric consistency for complex structures like people remains a challenge. \emph{DreamHuman} connects large text-to-image synthesis models, neural radiance fields, and statistical human body models in a novel optimization framework. This makes it possible to generate dynamic 3D human avatars with high-quality textures and learnt per-instance rigid and non rigid geometric deformations. We demonstrate that our method is capable to generate a wide variety of animatable, realistic 3D human models from text. These have diverse appearance, clothing, skin tones and body shapes, and outperform both generic text-to-3D approaches and previous text-based 3D avatar generators in visual fidelity.

DreamHuman: Animatable 3D Avatars from Text
Nikos Kolotouros
Thiemo Alldieck
Andrei Zanfir
Eduard Gabriel Bazavan
Mihai Fieraru
Cristian Sminchisescu
Google Research
(kolotouros, alldieck, andreiz, egaaaaann fieraru, Ltw.attiiiicccccceeeaooggggooeee
Abstract
We present DreamFuuann, a method to generate realistic animatable 3D human avatar models solely from textual descriptions. Recent text-to-3I) methods have made considerable strides generation, but are still lacking in important aspects. Control and often spatial resolution remain limited, existing methods produce fixed rather than animated 3D human models, and anthropometric consistency for complex structures like people remains challenge. Dreamlluman connects large text-to-image synthesis models, neural radiance fields, and statistical human body models novel modeling and optimization framework. This makes possible to generate dynamic 3D human avatars with high-quality textures and learned, instance-specific, surface deformations. We demonstrate that our method capable to generate wide variety of animatable, realistic 3D human models from text. Our 3D models have diverse appearance, clothing, skin tones and body shapes, and significantly outperform both generic text-to-3D approaches and previous text-based 3D avatar generators in visual fidelity.
Introduction
The remarkable progress Large Language Models [46, 8] has sparked considerable interest in generating wide variety of media modalities from text. There has been significant progress in text-to-image [49, 50, 52, 67, 10, 34), text-to-speech [37, 41], text-to-music [2, 19] and text-to-3I) few. Key the success of some of the popular generative image
[22, 43] generation, to methods conditioned on text has been diffusion models [52, 50, 55]. Recent works have shown these text-to-image models can be combined with differentiable neural 3D scene representations [5] and optimized to generate realistic 3D models solely from textual descriptions [22, 43J.
Controllable generation of photorealistic 3D human models has been in the focus of the research community for long time. This is also the goal of our work; we want to generate realistic, animatable 3D humans given only textual descriptions. Our method goes beyond static text-to-3D generation methods, because we learn a dynamic, articulated 3D model that can be placed different poses, without additional training fine-tuning. We capitalize the recent progress in text-to- 3D generation [43], neural radiance fields [31, 5] and human body modelling [64, 3] to produce 3D human models with realistic appearance and high-quality geometry. We achieve this without using any supervised text-to-3D data, or any image conditioning. We generate photorealistic and animatable 3d human models by relying only text, can be seen in Figure I and Figure As impressive as reneral-purposs. 3D generation methods [43] are, we argue these are suboptimal for 3D human synthesis, due limited control over generation which often results in undesirable visual artifacts such as unrealistic body proportions, missing limbs, or the wrong number of fingers. Such inconsistencies can be partially attributed to known problems of text-to-image networks, but become even more apparent when considering the arguably more difficult problem of 3D generation. Besides enabling animation capabilities, we show that geometric and kinematic human priors can resolve anthropometric consistency problems in an effective way. Our proposed method, coined
37th Conference on Neural Information Processing Systems (NcurIPS 2023).

---

dreadlocks
blonde wORaD wearing
Figure 1:
Example of 3D models synthesized and posed by our method. Dreamlluman can produce an animatable 3D avatar given only textual description of a human's appearance. test time, avatar can be reposed based on set of 3D poses motion, without additional refinement.
DreamHuman, can become a powerful tool for professional artists and 3D animators and can automate complex parts of the design process, with potentially transformative effects in industries such as gaming, special effects, as well as film and content creation.
Our main contributions are:
We present a novel method to generate 3D human models that can be placed in variety of poses, with realistic clothing deformations, given only a single textual description, and by training without any supervised text-to-3D data.
Our models incorporate 3D human body priors that are necessary for regularizing the generation and re- re-posing of the resulting avatar, by using multiple losses to ensure the quality of human structure, appearance, and deformation.
We improve the quality of the generation by means semantic zooming with refining prompts to add detail in perceptually important body regions, such as the face and the hands.
2
Related Work
There considerable work related diffusion models [58] and their applications image generation [17, 35, 11, 52, 50, 55, 54] or image editing [24, 53, 16, 32]. Our focus is on text-to-3D [22, 43, 47] and more specifically on realistic 3D human generation conditioned on text prompts. the following subsections we revisit some of the relevant work related to our goals.
Text-to-3D generation. CLIP-Forge [56] combines CLIP [45] text-image embeddings with learned 3D shape prior to generate 3D objects without any labeled text-to-3D pairs. DreamFields [22]
optimizes NeRF model given text prompt using guidance from CLIP [45]. CLIP-Mesh [25] also
uses CLIP, but substitutes NeRF with meshes as its underlying 3D representation. DreamFusion [43]
builds on top DreamFields and uses supervision from diffusion based sea--to-mmage-mone modell [54].
Latent-NeRF [30] uses similar strategy with DreamFusion, but optimizes a NeRF that operates in the space of a Latent Diffusion model [52]. TEXTure [S1] takes as input both a text prompt and a target mesh and optimizes the texture map agree with the input prompt. Magic3D [28] uses 2-stage strategy that combines Neural Radiance Fields with meshes for high resolution 3D generation. Unlike our method, all mentioned works produce static 3D scene given text prompt. When queried with human related prompts, results often exhibit artifacts like missing face details, unrealistic geometric proportions, partial body generation, incorrect number of body parts like legs or fingers. We generate accurate and anthropomorphically consistent results by incorporating 3D human priors in the loop.
Text-to-DD human generation. Several methods [40, 60, 4, 26, 15] learn to generate 3D human motions from text by leveraging text-to-MoCap datasets. MotionCLIP [59] learns to generate 3D human motions without using any paired text-to-motion data by leveraging CLIP as supervision. However, all these methods output 3D human motions in the form of 3D coordinates or human body model parameters [29] and not have the capability to generate photorealistic results. AvatarCI.IP [18] learns NeRF in the rest pose of SMPL [29] which is then converted back to mesh using

---

A Buddhist monk
Asian man wearing a navy suif
weartng short Jean cropped top
wearing wedding dress
A with blond hoir brown leather jocket
young wearing burtlenekk
A pregnant person of color
A thin Manathn runner
A man wearing C'hristmas sweater
A senior Black person wearing polo shirt
A Karate muster wearing black
A bodybuilder wearing a tanktop
clown
chef dressed white A plus model wearing pyjamas
A Black female surgeon
An Indian bride in traditional dress
woman wearing ski clothes
woman in gym clothes
A farmer
Sponish flamenco dancer
person diving suit
Black person in military uniforn
wearing homber jacket
track and field athlete
A person dressed at the Venice Carmival
A man wearing a hoodie
Figure 2: 3D human avatars generated using our method given text prompts. We render each example in random pose from viewpoints, along with corresponding surface normal maps.
3

---

C( 9) C La + Lm + Lp Lo Lads
Model Input
Prompt woman wearing
Semuntic Zoom
dress
Full body photo of (p)
Pose B
imGHUM Six
NeRF f( v.0.d.s.0.
Albedo e
Renderer
Headshot of (p)
Diffusion Model
Posed space Canonical space
Spherical
Photo of right arm of (p)
Harmonics
Density
Figure Overview of DreamIHuman. Given text prompt, such as woman wearing dress, we generate a realistic, animatable 3D avatar whose appearance and body shape match the textual description. A key component our pipeline is a deformable and cose-conditioned NeRF model learned and constrained using imGHUM [3], an implicit statistical 3D human pose and shape model. At each training step, we synthesize our avatar based on randomly sampled poses and render from random viewpoints. The optimisation of the avatar structure is guided by the Score Distillation Sampling loss [43] powered by a text-to-image generation model [54]. We rely on imGHUM [3] to add pose control and inject anthropomorphic priors in the avatar optimisation process. We also use several other normal, mask and orientation- -based losses order to ensure coherent synthesis. NeRF, body shape, and spherical harmonics illumination parameters (in red) are optimised.
marching cubes. However the reposing procedure depends on fixed skinning weights that limit the overall realism of the animation. In contrast, our method learns per-instance pose-specific geometric deformations that result in significantly more realistic clothing appearance. Concurrent work AvatarCraft [23] produces avatars with geometry very much tied the underlying SMPL [29] model and thus cannot model loose-fitting clothing. Another concurrent work, DreamAvatar [9], suffers from the fact that to be retrained every time for new pose, which makes computationally prohibitive to repose.
Deformable Neural Radiance Fields. Several methods attempt learn Deformable NeRFs
model dynamic content [38, 44, 61, 39, 57]. There has also been work on representing articulated human bodies [65, 36, 63, 69, 20, 66, 27]. The method more closely related ours is H-NeRF [65], which combines implicit human body models with NeRFs. Compared to H-NeRF, our method uses simpler approach where enforce consistency directly in 3D and not via renderings of different density fields. Also, while H-NeRF that uses videos for supervision, our only input is text, and we use are not constrained by the poses and viewpoints present on the video. Thus our method can generalize better in variety of different poses and camera viewpoints.
3 Methodology
3.1 Architecture
We rely on Neural Radiance Fields (NeRF) [31] to represent our 3D scene as a continuous function of its spatial coordinates [33]. We use a multi-layer perceptron (MLP) that maps each spatial point x E R to tuple (c, of RGB color and density values. To render a scene using NeRF, one needs to cast rays from the camera center passing through the image pixels and then compute the expected color C along each ray. In practice, this is done by sampling points on the ray and then approximating the rendering integral [5]
= Σ W.C;, w; = a; [[ 11--0 az). a; = exp (-T, ||x41 - -x(1).
(1)
jsi
While NeRF provides a general purpose scene representation, we aim to regularize the optimised geometry and appearance using human structural priors. To that effect, we use imGHUM [3], which the implicit version of the GHUM [64] body model, and thus compatible with neural scene representations. Given pose 0 and shape / parameters, imGHUM predicts semantic signed distance

---

function S(x, 0 (8) that maps 3D spatial point x to tuple (d,s) containing the signed distance d of the point from the body surface together with semantic correspondence code s E R that associates x with the nearest surface point on the body.
Our model architecture uses mip-NeRF 360 [5] for the NeRF backhone. An overview can be seen in Figure 3. Specifically, we modify each of the MLPs in the standard NeRF model in order to operate in the imGHUM [3] semantic signed distance space instead of the standard 3D coordinates. Given a 3D point x E IS and pose and shape parameters 0 and [3 respectively, we first encode it with imGHUM [3] into the 4D semantic descriptor (d,s) S(x, 0 (). We can then learn a NeRF f in this semantic space
(c, = f s)..
(2)
where & represents the trainable weights for the NeRF module. Similarly with DreamFusion [43], c models the albedo of the surface at the corresponding point, and use this together with the learnt
geometry to produce shaded renderings.
By learning a NeRF in the semantic signed distance space of a human body model, we learn a representation that can generalize to different human poses and body shapes. This is because the local geometry and color are generally preserved in (d, s) for different shape and pose parameters. One can think of the process similarly to learning a NeRF for the template pose and then warping to new shapes and poses by leveraging the 3D correspondences from the body model [42, 6]. However, animating the model in different poses is challenging. Clothing deformations work reasonably only for tight-fitting clothing or for accessories that are usually moving rigidly with the body, such as hats and glasses. For this reason, we propose to augment and modulate the NeRF input with pose and shape parameters, thus giving it the capability to model non-rigid pose and surface dependent effects beyond the body shape itself. By doing so, the model can learn per-instance, pose-dependent deformations of the clothing surface, on top of what the imGHUM model can represent. Thus, our NeRF input becomes
(c, = f( d. s, 3)
To make sure the NeRF model conforms the underlying body geometry we propose calculate the final density as the maximum of the density T computed by the NeRF MLP and the density proxy f(d) = aa(-ad) computed from imGHUM based on the signed distance value. In the previous equation, a is the sigmoid function and a positive constant that controls the sharpness of the density field. Effectively
smooth scaled indicator function, with 7 x a inside the body and f 0 outside. In this way we avoid undesirable artifacts, such as the model removing limbs or fine structure like fingers, unless the prompt indicates so.
Shading and rendering model. We found that a diffuse reflectance model [43] does not produce very realistic renderings of the human appearance, with results that often look cartoon-like. Hence we rely on a spherical harmonics lighting model [48] and preserve the first 9 components. During NeRF training. we additionally optimize for the spherical harmonics coefficients (i.e. h R 1=x10), However, by using just the optimized coefficients can lead to inadequate albedo-shading disentanglement and occasionally some geometric regions may never get highlights. Empirically, we found that sampling random coefficients a fraction of the time during training produces better results.
Semantic zoom. One limitation in using text-to- text-to-image diffusion model for supervision is its 64 64 pixels input resolution. As a result textures are often blurry and the geometry lacks fine details. One way around this would be the use of super-resolution diffusion models, e.g. the 64 x 64 256 256. However these make rendering very expensive as memory requirements increase by a factor of 16. By using a human body model with attached semantics like imGHUM to control the NeRF, we benefit from direct correspondences between the 3D space occupancy and the human body parts. We can then very easily infer the location of important body parts such as the head, hands, etc. for any given pose. Therefore, during optimization we propose to use this information to zoom in on different parts of the body, thus increasing the effective model resolution. This can leverage both detail implicit in the image diffusion model used, and structure in the imGHUM human body prior. Instead of rendering a 64 x 64 image of the whole body, we render instead 64 x 64 image of the head and some body parts where fine details are important. In total, we define 6 semantic regions: head, upper body, lower body, midsection, left arm, right arm. We also modify the text prompt accordingly, in order to explicitly encode this information in the text. In contrast to AvatarCLIP that only zooms-in on the face, zooming-in on all body parts results in much crisper textures and geometric detail throughout. For more information please check our Supplementary Material.
5

---

3.2 Loss functions
imGHUM density loss. To enforce that the estimated avatar follows the underlying body shape geometry, we add an L1 loss between the NeRF density and a density proxy computed from imGHUM. This loss encourages sparse modifications in the body geometry and is necessary to preserve important geometric details on the body. The density loss is defined as
4 ||| -112-
Predicted normal loss. Following Ref-NeRF [62], we modify the MLP to also predict the surface normal vector n' at each spatial location and then add loss between the predicted normals and the normals n obtained from the gradient of the density field. The normal loss is
Ln Σ Wt nn
(5)
In our case, this loss serves two purposes: it acts as a smoothness loss on the surface normals and also helps learning the pose-dependent deformations. Regarding the first part, noticed that for clothing such as skirts or dresses with uniform dark texture, the resulting surface normals are often very noisy, resulting in sub-optimal shading results. Naturally, the predicted surface normals are smoother that the density normals because of the spectral bias of MLPs and hence this loss acts as surface regularizer. More importantly though, the auxiliary task of predicting the surface normals encourages the MLP to use the pose conditioning information during optimization. The pose-dependent density deformations are sparse and subtle since a considerable part of the work is usually handled decently by imGHUM. Hence, it is easy for the MLP to ignore the conditioning on the pose parameters because it has a small overall impact on the loss. Note, however, that pose conditioning is necessary order to predict the correct surface normals. If not used, then the predicted normal vector at particular point on the surface, e.g. on the arm, will be always the same, regardless of the limb orientation, because it only depends on the canonical coordinates (d,s).
Foreground mask loss. The above density loss forces the NeRF to respect the underlying body geometry and disentangles the subject from the background. However, we noticed that in some cases this can result in making the clothing hair translucent. To prevent it, we add loss on the rendered mask M that encourages it to be binary. The loss is defined as
H
W
Lm
min (log M(x, log(1 M(x
(6)
HW
1=1
where ((x,,)==X Wi, i.e. the sum of the rendering weights for the ray through pixel (x,
Diffusion Models and Score Distillation Sampling. Diffusion models are a class of generative models that learn to produce samples from a target distribution by iteratively denoising samples coming from a tractable base distribution. They consist of fixed forward process that gradually transforms a sample from the data distribution to Gaussian noise and a learnable reverse process that approximates the inverse of the forward process.
To generate images from the data distribution given an NeRF with parameters T, [43] proposed to use Score Distillation Sampling. This involves optimizing approximation of the diffusion model training loss. The gradient of the Score Distillation Sampling loss with respect to the NeRF is defined
Ou
L.ds Ee-210011, wc (t)(t) (É(Zz) y, t)
E)
04
where e is the injected noise, the noisy rendered image and epsilon the noise prediction from the diffusion model. We use the SDS loss [43, 54] to supervise the 3D generation given the actively modified semantic-zoom prompts.
Additional losses. We the orientation loss C. from Ref-NeRF [62] that penalizes 'back-facing' normals for points along the ray that are visible, as well as the loss on the proposal weights C, in mip-NeRF360 [5].
Our full loss function then becomes
L = Lade + Lo + Cp + Lm + Ln + C,
(8)
6

---

wearing striped shirt and while linen
An African dressed in traditional clothes
Figure 4: Importance of semantic zoom. For each example, the left image shows the generated avatar with semantic zoom, whereas the right image an avatars generated without Notice how the semantic zoom allows us to reconstruct sharper, higher-quality textures.
3.3 Optimization
Body pose sampling. Previous methods like H-NeRF [65] and Human-NeRF [63] have limited generalization capabilities because they only trained poses and viewpoints that are present in an input video. Our method on the other hand does not have such constraints. At cach optimization step, sample random pose from distribution [68] trained 3D motion capture [1, 21, 12, 14, 13] and use this to pose imGHUM. Sampling different poses is necessary for learning the dependency of the surface geometry the model shape and pose parameters. At the same time it helps disentangle the generated avatar from objects in the background. Without the pose randomization strategy often times there is not sufficient disentanglement of the avatar geometry from the background and the final geometry includes additional objects such as the ground floor, or even the shadow of the person around the legs
Other details. We optimize the NeRF and the imGHUM shape parameters & instead of randomly sampling shape parameters. This is because the body shape is often explicitly or implicitly described in the caption. We generate one avatar with an underlying body shape given all constraints coming from the text prompt and the related losses. Similarly with DreamFusion, camera positions spherical coordinates and then augment the input prompt with view-dependent conditioning based on the azimuth and elevation. We also randomly select the radius from the origin as well as the focal length of the camera. For additional details please see our Supplementary Material.
randomly sample
Experiments
In this section illustrate the effectiveness of proposed method. We show how the individual proposed components help, and how we compare to recent state-of-the-art methods. Figure 2 shows wide variety of generated 3d human models in different poses, shapes, skin tones and body compositions. Due to space constraints, additional results are available in the Supplementary Material.
illustrate diverse body
4.1 Ablation Study
Semantic zoom. Figure 4, we show the importance our semantic zoom strategy. Notice how method is able to generate much higher-quality textures, both for the body and the face.
Pose-dependent deformations. In Figure 5, we show examples of how we can learn realistic garment deformations. In the example of the ballerina, one can see that the skirt deforms more naturally when legs move. On the other hand the baseline without non-rigid deformations struggles to capture the skirt geometry and exhibits floating artifacts around the legs. Similar observations can be made for the man wearing shorts. We hypothesize that our model can infer this because the text-to-image generator has been trained lots of images of people wearing clothes in different poses. Therefore,

---

with pose-coreetiees
Wiltour posercomeccies
Win
Without cornective
bollerino
mun wenring o shirt, sunglosses
Figure 5: Importance of pose-dependent deformations and pose sampling in the NeRF model, f d, 0, (3). Our non-rigid pose-dependent deformations enable more realistic clothing when reposing the avatar. For each of the two example prompts we show two generated avatars, with and without pose-correctives. Notice how the skirt and the shorts move more naturally when reposing the avatar.
our model, although not using video, or relying on text-to-video diffusion loss, can leverage general knowledge on how clothing drapes.
Choice of diffusion model. Our method is not tied particular diffusion model. To demonstrate this, we substitute Imagen [55] with the open-source Stable Diffusion model [52]. Stable Diffusion is latent diffusion model, at each training iteration through the latent encoder, and then apply the SDS loss on the latent embeddings. In Figure 6, we show qualitative results of method trained using Stable Diffusion. Overall observe that we able to generate high quality avatars, regardless of the choice of the diffusion model.
NeRF render an RGB image, pass
elderly person weuring waistcouf
wearing aweuter
woman dressed shi clothes
Figure Generation results using Stable Diffusion.
4.2 Comparison with the state of the art
Qualitative Evaluations. In Figure 7, we show a qualitative comparison of our method with DreamFusion. DreamFusion suffers from limited control generation. Even though it was prompted to generate the full body of the subject, very often it produces 3D model the upper body, the head. At the same time it cannot properly disentangle the human subject from other objects in the scene, resulting in 3D models that contain parts of the environment. More importantly though, very often produces unpleasant visual artifacts, such as non-realistic body proportions, missing multiple limbs, as well as degenerate geometry that can be attributed to viewpoint overfitting. Our method is able to overcome these issues by utilizing strong 3D prior human body geometry. For more comparisons we refer the readers to the Supplementary Material.
Figure 8 shows comparison between DreamHumun and AvatarCI.IP. We can see that our method is able to generate significantly better geometry and texture quality. The geometry of the reconstructed avatars with AvatarCLIP is very close the underlying body model geometry, with only minor modifications. As a result, it cannot handle loose- loose-fitting clothing, dresses, and accessories like hats. The model textures from AvatarCLIP also have significant artifacts and do not match the realism and overall quality of DreamHuman in all examples we tried.
Quantitative evaluation using CLIP. Following common practice, we also use CLIP to evaluate the alignment of the rendered 3D models with the input text prompts. We use a total of 160 prompts with descriptions of people. The results are shown in Table 1. We can see that our method consistently outperforms DreamFusion. We also include comparisons with AvatarCI.IP. However, note that AvatarCLIP is trained using CLIP as supervision, meaning that the CLIP-based metrics are biased

---

DreamFusion
Ours
DreamFusion
Ours
Woong
Makgle deadece
31 Auulippe orma
Mindag aren
professional boser
A polieewoman
Figure 7: Comparison with DreamFusion. For each example show the rendered 3D model as well as the corresponding surface normals. Both methods were asked to reconstruct the full body of the subject by prepending A DSLR full body photo the prompt.
AvatarCLIP
Ours
Figure 8: Comparison with AvatarCLIP We compare DreamHuman with AvatarCLIP [18]. From left to right we used the following prompts: astronaut, construction manager, firefighter, gardener, pilot, police officer, robot, senior citizen, soldier, teenager, warrior, witch, wizard. Notice that our method generates much more realistic texture and geometry. All illustrations are in default A-Pose.
favorably towards AvatarCLIP, This means that direct comparison other methods is potentially unfair.
Table Evaluation of the rendered 3D models using CLIP, We report the R- R-Precision as well as whether the true caption is in the top 3 and 5 highest-scoring captions.
Method
R-Precision f
Top-3 f
Top-5 f
DreamFusion [43]
0.775
0,888
0.925
AvatarCLIP [18]
0.855
0.962
0.981
Ours
0.838
0.931
0.956
User study. We conducted user study assess the quality of 3D avatar generation pipeline. We used 20 text prompts from the AvatarCLII website, selected from the General Description category. We our method those 20 text prompts and rendered the final results the rest pose, from the front and from the side. We did the same for the precomputed meshes that the AvatarCLIP authors provide on their website. We then asked the agreement between the renderings and the input text. and (b) the perceived visual quality of the generated avatar. The ratings were on scale from 1-5, with 1 meaning Very Bad and Very Good. The results in Table 2 show that the raters consistently preferred our method over AvatarCLIP.
rate the two methods on (a) the perceived
4.3 Generation variability.
Here we investigate whether there is variation in our generations for the same prompt, but with different random seeds. As originally observed in DreamFusion [43], there is limited variation in the generations, probably because the SDS loss mode-seeking, so tends latch on to specific modes of the distribution. Our results demonstrate some level of variability, seen on Figure 9,

---

Table 2: User study on 3D avatar generation quality.
Method
Agreement with text
Visual quality 1
AvatarCLIP [18]
2.96 4.45
2.49
Ours
4.16
A mon wearing a Christmas sweeter
A woman wearing a long dress
mus wearing a comel coat hat
An Asian wearing life jacket Figure 9: Generations using different random seeds. For each example show the generated 3D avatar using different random seeds as initialization.
clown
comboy
5 Conclusion
We presented DreamHuman, a novel method for generating 3D human avatars from text. Our method leverages statistical 3D human body models and recent advances in 3D modelling and text-to-3D generation to create animatable 3D human avatars, without any paired text-to-3D supervision. We illustrated that our method can generate photorealistic results, with detailed geometry and outperforms the state of the art by a large margin.
Limitations and future work. Since our model is trained without any 3D data, sometimes draws fine details like wrinkles using the albedo map instead of creating them based on geometry. Future work can address this by leveraging 3D data to resolve some of the reconstruction ambiguities. Additionally, the model sometimes cannot properly disentangle albedo from shading, resulting in baked reflections and shadows. Current computational constraints from the diffusion models prevent us from scaling the method to very high resolution textures and geometric detail like hair. Finally, the realism of clothing animation can benefit from video model.
Broader Impact. While method does not any additional training data, it relies in part on text-to-image diffusion models which have been pre-trained on large-scale datasets containing sometimes insufficiently curated images and captions [7] (N.B. the level of effective automation to guarantee the removal of undesired content is considerable for most models we use in this paper). Also, text-to-image generators use LLMs for the text encoder, pre-trained on uncurated internet-scall data. Although our method uses statistical 3D human body shape models learnt using highly curated and diverse data to remove bias, ultimately our generation process may be vulnerable some bias in its dependencies.
The goal of our method is generate 3D models of people, which has the potential to be misused in connection with fake media. However, is important to highlight that our rendered 3D human models are typically less realistic than their DD-generated counterparts. Regardless, in practical settings, safeguards should be used to prevent abuse, such as filtering the input text prompts and detecting any unsafe content in the model renderings.
Our method has the ability to generate people with diverse body shapes, appearance, skin color and clothing. This can enable the generation of diverse large-scale synthetic 3D datasets for human-related tasks, and in turn may support training models with fairer outcomes across different groups.
Dreaml H Human can potentially augment the work of artists and other creative professionals. It could be used complementary tool boost productivity. It also has the potential democratize 3D content creation that currently requires specialized knowledge and expensive proprictary software.
10

---

References
[1] http://nocap... ca. cmu. edu/.
[2] Andrea Agostinelli, Timo I Denk, Zalán Borsos, Jesse Engel, Mauro Verzetti, Antoine Caillon, Qingqing Huang, Aren Jansen, Adam Roberts, Marco Tagliasacchi, et al. Musiclm: Generating music from text. arXiv preprint 2rx1v2331.11335. 2023.
[3] Thiemo Alldieck, Hongyi Xu, and Cristian Sminchisescu. imghum: Implicit generative models of 3d human shape and articulated pose. In Proceedings the IEEE/CVF International Conference on Computer Vision, pages 1461-5470, 2021.
[4] Nikos Athanasiou, Mathis Petrovich, Michael J. Black, and Gül Varol. Teach: Temporal action composi- tions for 3d humans. In International Conference on 3D Vision (3DV), September 2022.
[5] Jonathan T. Barron, Ben Mildenhall, Dor Verbin, Pratul P. Srinivasan, and Peter Hedman. Mip-nerf 360: Unbounded anti-aliased neural radiance fields. CVPR, 2022.
[6] Eduard Gabriel Bazavan, Andrei Zanfir, Mihai Zanfir, William Freeman, Rahul Sukthankar, and Cristian Sminchisescu. Hspace: Synthetic parametric humans animated in complex environments, 2021.
[7] Abeba Birhane, Vinay Uday Prabhu, and Emmanuel Kahembwe, Multimodal datasets: misogyny, pornog- raphy, and malignant stereotypes. arXiv preprint arXiv:21 2021.
[8] Tom Brown, Benjamin Mann, Nick Ryder, Melanie Subhiah, Jared D Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, Language models are few-shot leamnrr.. Advances neural information processing systems, 33:1 177-1999,,,, 2020.
[9] Yukang Cao, Yan-Pe Cao, Kai Han, Ying Shan, and Kwan- Yoe K Wong. Dreamavatar: Text-and-shape guided 3d human avatar generation via diffusion models. arXiv preprint arXiv:2344.009 2023.
[10] Huiwen Chang. Han Zhang. Jarred Barber, AJ Maschinot, Jose Lezama, Lu Jiang, Ming- Hiugg-suann Yang. Kevin Murphy, William T Freeman, Michael Ruhinstein, et al. Muse: Text-to-image generation via masked generative transformers. arXiv preprint arXiv:2301.00704. 2023.
[11] Prafulla Dhariwal and Alexander Nichol. Diffusion models beat gans on image synthesis. Advances in Neural Information Processing Systems, 34:8780-8799 8888800-8442277 2021.
[12] Mihai Fieraru, Mihai Zanfir, Elisabeta Oneata, Alin-Ionut Popa, Vlad Olaru, and Cristian Sminchisescu. Three-dimensional reconstruction of human interactions. In The IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), June 2020.
[13] Mihai Fieraru, Mihai Zanfir, Elisabcta Oneata, Alin-Ionut Popa, Vlad Olaru, and Cristian Sminchisescu. Learning complex 3d human self-contact. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 35, pages 1343-1351, 2021.
[14] Mihai Ficraru, Mihai Zanfir, Silviu Criittiariiin Pirica, Vlad Olaru, and Cristian Sminchisescu. Aifit: Automatic human- interpretable feedback models fitness training. In The IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), June 2021.
[15] Chuan Guo, Shihao Zou, Xinxin Z Sen Wang, Wei Ji, Xingyu Li, and Li Cheng. Generating diverse and natural 3d human motions from text. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pages 5152-5161, June 2022.
[16] Amir Hertz, Ron Mokady, Jay Tenenbaum, Kfir Aberman, Yacl Pritch, and Daniel Cohen- Or. Prompt-to- prompt image editing with cross attention control. 2022.
[17] Jonathan Ho, Ajay Jain, and Pieter Abbeel. Denoising diffusion probabilistic models. Advances in Neural Information Processing Systems. 33:6840 6851, 2020.
[18] Fangzhou Hong, Mingyuan Zhang, Liang Pan, Zhongang Cai, Lei Yang, and Ziwei Liu. Avatarelip: Zero-shot text-driven generation and animation of 3d avatars. ACM Transactions on Graphics (TOG), 41 (4):1-19, 2022.
[19] Qingqing Huang, Daniel S Park, Tao Wang, Timo Denk, Andy Ly. Nanxin Chen, Zhengdong Zhang, Zhishuai Zhang, Jiahui Yu, Christian Frank. et al. Noise2music: Text-conditioned music generation with diffusion models. arXiv preprint 2rX1v.200..33117.000 2023.

---

[20] Mustafa Işık, Martin Rünz, Markos Georgopoulos, Taras Khakhulin, Jonathan Starck, Lourdes Agapito, and Matthias Nießuer. Humanrf: High-fideliig neural radiance fields for humans in motion. ACM Transactions on Gruphics (TOG), 12( 2023. doi: 10.1145/3592415. URL https://doi org/ 1145/ 3592415.
[21] Catalin Ionescu. Dragos Papava, Vlad Olaru, and Cristian Sminchisescu. Human3.6m Large scale datasets and predictive methods for 3d human sensing in natural environments. IEEE Transactions on Pattern Analysis and Machine Intelligence, 26(7):1325-139) jul 2014.
[22] Ajay Jain, Ben Mildenhall, Jonathan T. Barron, Pieter Abbeel, and Ben Poole. Zero-shot text-guided object generation with dream fields. 2022.
[23] Ruixiang Jiang, Can Wang. Jingbo Zhang, Menglei Chai, Mingming Hc, Dongdong Chen, and Jing Liao. Avatarcraft: Transforming text into neural human avatars with parameterized shape and pose control. arXiv preprint arXi::2303. 17606, 2023.
[24] Bahjat Kawar, Shiran Zada, Oran Lang, Omer Tov, Huiwen Chang, Tali Dekel, Inbar Mosseri, and Michal Trani. Imagic: Text-based real image editing with diffusion models, 2023.
[25] Nasir Mohammad Khalid, Tianhao Xie, Eugene Belilovsky, and Popa Tiheriu. Clip-mesh: Generating textured meshes from text using pretrained image-text models. SIGGRAPH Asia 2022 Conference Papers, December 2022.
[26] Jihoon Kim, Jiseob Kim, and Sungjoon Choi. Flame: Free-form language-based motion synthesis & editing. AAAI, 2023.
[27] Zhe Li, Zerong Zheng, Yuxiao Liu, Boyao Zhou, and Yebin Liu. Posevocab: Learning joint- sint-sttuuctued pose embeddings for human avatar modeling. In ACM SIGGRAPH Conference Proceedings, 2023.
[28] Chen-Hsuan Lin, Jun Gao, Luming Tang. Towaki Takikawa, Xiaohui Zeng, Xun Huang, Karsten Kreis, Sanja Fidler, Ming- Yu Liu, and Tsung Yi Lin. Magic3d: High-resolution text-to-3d content creation. arXiv preprint arXiv:2211 10440, 2022.
[29] Matthew Loper, Naureen Mahmood, Javier Romero, Gerard Pons-Moll, and Michael Black. Smpl: skinned multi-person linear model. ACM transactions on graphics (TOG), 34(6):1-16, 2015.
[30] Gal Metzer, Elad Richardson, Or Patashnik, Raja Giryes, and Daniel Cohen-Or. Latent-nert for shape- guided generation of 3d shapes and textures. arXiv preprint wxXiv:221..07600 2022.
[31] Ben Mildenhall, Pratul P. Srinivasan, Matthew Tancik, Jonathan T. Barron, Ravi Ramamoorthi, and Ren
Ng. Nerf: Representing scenes as neural radiance fields for view synthesis. In ECCV, 2020.
[32] Ron Mokady, Amir Hertz, Kfir Aberman, Yacl Pritch, and Daniel Cohen-Or. Null-text inversion for editing real images using guided diffusion models, 2022.
[33] Alexander Mordvintsev. Nicola Pezzotti, Ludwig Schubert, and Chris Olah. Differentiable image parame- terizations. Distill, 32)) 2018
[34] Alex Nichol, Prafulla Dhariwal, Aditya Ramesh, Pranav Shyam, Pamela Mishkin, Bob McGrew, llya Sutskever, and Mark Chen. Glide: Towards photorealistic image generation and editing with text-guided diffusion models. arXiv preprint arXiv:21 2021.
[35] Alexander Quinn Nichol and Prafulla Dhariwal. Improwed denoising di diffusion probabilistic models. In International Conference on Machine Learning, pages 8162-81181111 8171. PMLR, 2021.
[36] Atsuhiro Noguchi, Xiao Sun, Stephen Lin, and Tatsuya Harada. Neural articulated radiance field. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pages 5762-5772, 2021.
[37] Aaron Oord, Yazhe Li, Igor Babuschkin, Karen Simonyan, Oriol Vinyals, Koray Kavukcuoglu, George Driessche, Edward Lockhart, Luis Cobo, Florian Slimberg, et al. Parallel wavenet: Fast high-fidelity speech synthesis. In International conference on machine learning, pages 3918-3926. PMLR, 2018
[38] Keunhong Park, Utkarsh Sinha. Jonathan Barron, Sofien Bouaziz. Dan Goldman. Steven M Seitz, and Ricardo Martin- Bruulla. Nerfies: Deformable neural radiance fields. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pages 5865-5874, 2021.
[39] Keunhong Park, Utkarsh Sinha, Peter Hedman, Jonathan T. Barron, Sofien Bouaziz, Dan B Goldman, Ricardo Martin Bruaala. and Steven M. Seitz. Hypernerf: A dibber-dimenioonll representation for topologically varying neural radiance fields. ACM Trans. Graph., 40(6), dee 2021.
12

---

[40] Mathis Petrovich, Michael J. Black, and Gül Varol. TEMOS: Generating diverse human motions from textual descriptions. In European Conference on Computer Vision (ECCV), 2022.
[41] Wei Ping, Kainan Peng, and Jitong Chen. Clarinet: Parallel wave generation in end-to-end text-to-speech. arXiv preprint arXiv:/807.000 00221,7 2018.
[42] Gerard Pons-Moll, Sergi Pujades, Sonny Hu, and Michael Black. Clothcap: Seamless 4d clothing capture and retargeting ACM Transactions on Graphics, (Proc. SIGGRAPH). 36(4), 2017. URL http: //dx.doi.coo olt.:.oo..r/////...
org/ 10. 1145/3072955 3073711. Two first authors contributed equally.
[43] Ben Poole, Ajay Jain, Jonathan T. Barron, and Ben Mildenhall. Dreamfusion: Text-to-3d using 2d diffusion. ICLR, 2023.
[44] Albert Pumarola, Enric Corona, Gerard Mols,Moll, and Francese Moreno-Noguer, D-nerf: Neural radiance fields for dynamic scenes. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 10318-10327, 2021.
[45] Alec Radford, Jong Wook Kim. Chris Hallacy. Aditya Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, al. Learning transferable visual models from natural language supervision. In International conference on machine learning, pages 7748-8763. PMLR, 2021.
[46] Colin Raffel, Noam Shazeer. Adam Roberts, Katherine Lee, Sharan Narang, Michael Matena, Yanqi Zhou, Wei Li, and Peter J. Liu. Exploring the limits of transfer learning with a unified text-to txxtet transformer. Journal of Machine Learning Research, 21(44)):1667 2020. URI. http:// ///////mmrr org/ htt/appprr/v// 20-074. htmml.
[47] Amit Raj. Srinivas Kaza, Ben Poole, Michael Niemeyer, Nataniel Ruiz, Ben Mildenhall, Shiran Zada, Kfir Aberman, Michael Rubinstein, Jonathan Barron, Yuanzhen Li, and Varun Jampani. Dreambooth3d: Subject-driven text-to-3d generation, 2023.
[48] Ravi Ramamoorthi and Pat Hanrahan. An efficient representation for irradiance environment maps. In Proceedings of the 28th annual conference on Computer graphics and inteructive techniquex, pages 497-500, 2001.
[49] Aditya Ramesh, Mikhail Pavlov, Gabriel Goh, Scott Gray, Chelsea Voss, Alec Radford, Mark Chen, and Ilya Sutskever. Zero-shot text-to-image generation. In International Conference on Machine Learning, pages 8821-8831, 2021. 2021.
[50] Aditya Ramesh, Prafulla Dhariwal, Alex Nichol, Casey Chu, and Mark Chen. Hierarchical text-conditional image generation with clip latents. arXiv preprint arXiv:2204.06122, 2022.
[51] Flad Richardson, Gal Metzer, Yuval Alaluf, Raja Giryes, and Daniel Cohen-Or. Texture: Text-guided texturing of 3d shapes. SIGGRAPH, 2023.
[52] Robin Rombach, Andreas Blattmann, Dominik Lorenz, Patrick Esser, and Bjöm Ommer. High-resolution image synthesis with latent diffusiin models. In Proceedings of the IEEE/CV Conference Computer Vision and Pattern Recognition, pages 10684-10695, 2022.
[53] Nataniel Ruiz, Yuanzhen Li, Varun Jampani, Yael Pritch, Michael Rubinstein, and Kfir Aberman. Dream- booth: Fine tuning text-to-image diffusion models for subject-driven generation. 2022.
[54] Chitwan Saharia, William Chan, Saurabh Saxena, Lala Li, Jay Whang, Emily I. Denton, Kamyar Ghasemipour, Raphael Gontijo Lopes, Burcu Karagol Ayan, Tim Salimans, et al. Photorealistic text-to- image diffusion models with deep language understanding. Advances in Neural Information Processing Systems,
2022.
[55] Chitwan Saharia, Jonathan Ho, William Chan, Tim Salimans, David J Fleet, and Mohammad Norouzi. Image super-resolution via iterative refinement. IEEE Transactions on Pattern Analysis and Machine Intelligence, 2022.
[56] Aditya Sanghi, Hang Chu, Joseph G Lambourne, Ye Wang, Chin-Yi Cheng, Marco Fumero, and Ka- mal Rahimi Malekshan. Clip-forge: Towards zero-shot text-to-shape generation. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 18603-18613, 2022.
[57] Uriel Singer, Shelly Sheynin, Adam Polyak, Oron Ashual, lurii Makarov, Filippos Kokkinos, Naman Goyal, Andrea Vedaldi, Devi Parikh, Justin Johnson, et al. Text-to-4d dynamic scene generation. arXiv preprint arXiv: 2023.
13

---

[58] Jascha Sohl-Dickstein, Eric Weiss, Niru Maheswaranathan, and Surya Ganguli. Deep unsupervised learning using nonequilibrium thernodynamiis.. In International Conference on Machine Learning, pages 2256-2265. PMLR, 2015.
[59] Guy Tevet, Brian Gordon, Amir Hertz, Amit H Bermano, and Daniel Cohen-Or. Motionclip: Exposing human motion generation to clip space. Computer Vision-ECCV' 2022: 17th European Conference, Tel Avir, Israel, October 23-27, 2022, Proceedings. Purt XXII, pages 358-374 Springer, 2022.
[60] Guy Tevet, Sigal Raab, Brian Gordon, Yonatan Shafir, Amit H Bermano, and Daniel Cohen-Or. Human motion diffusion model. ICLR, 2023.
[61] Edgar Tretschk, Ayush Tewari. Vladislav Golyanik, Michael Zollhöfer, Christoph Lassner, and Christian Theobalt. Non-rigid neural radiance fields: Reconstruction and novel view synthesis of a dynamic scene from monocular video. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pages
2021.
[62] Dor Verbin. Peter Hedman. Ben Mildenhall, Todd Zickler, Jonathan T. Barron. and Pratul P. Srinivasan. Ref-NeRF: Structured view-dependent appearance for neural radiance fields. CVPR, 2022.
[63] Chung- Yi Weng, Brian Curless, Pratul P. Srinivasan, Jonathan T. Barron, and Ira Kemelmacher-Shlizzr HumanNeRF: Pree-viewpoint rendering of moving people from monocular video. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pages 16210 16220, June 2022.
[64] Hongyi Xu, Eduard Gabriel Bazavan, Andrei Zanfir, William T Freeman, Rahul Sukthankar, and Cristian Sminchisescu. Ghum ghuml: Generative 3d human shape and articulated pose models, In CVPR, 2020.
[65] Hongyi Xu, Thiemo Alldieck, and Cristian Sminchisescu. H-nerf: Neural radiance fields rendering and temporal reconstruction of humans in motion. Advances in Neural Information Processing Systems, 34: 14966, 2021.
[66] Yuclang Xu, Hongwen Zhang, Lizhen Wang. Xiaochen Zhao, Huang Han, Qi Guojun, and Yebin Liu. Latentavatar: Learning latent expression code for expressive neural head avatar. In ACM SIGGRAPH 2023 Conference Proceedings, 2023.
[67] Jiahui Yu, Yuanzhong Xu, Jing Yu Koh, Thang Luong, Gunjan Baid, Zirui Wang, Vijay Vasudevan, Alexander Ku, Yinfei Yang, Bureu Karagol Ayan, al. Scaling autoregressive models for content-rich 2tt.:22200.0888 2022.
text-to-image generation. arXiv preprint
[68] Andrei 7anfir, Eduard Gabriel Bazavan, Hongyi Xu, William T Freeman, Rahul Sukthankar, and Cristian Sminchisescu. Weakly supervised 3d human pose and shape reconstruction with normalizing flows. In Computer Aision-ECCV 2020: 16th European Conference, Glavgow, UK. August 23-28, 2020, Proceedings. Purt VI 16, pages 465-481- Springer. 2020.
[69] Fuqiang Zhao, Wei Yang. Jiakai Zhang, Pei Lin, Yingliang Zhang, Jingyi Yu, and Lan Xu, Humannerf: Efficiently generated human radiance field from sparse inputs. In Proceedings of the IEEE/CVE Conference on Computer Vision and Pattern Recognition (CVPR), pages 7743-7753, June 2022.
14


Reviews:
[{"review_id": "uXWsGq99uA", "reviewer": "Reviewer_KMW9", "summary": "This paper proposes a method to generate high-quality and animatable 3D human from textual input.", "questions": "- What is the rendering resolution at inference time?\n\n- How are the query locations sampled? Are they sampled in free space or bounded box?\n\n- Which method was used to perform rendering with spherical harmonics lighting model? \n\n- How are the camera parameters (both extrinsic and intrinsic) set? Since there are no known camera parameters for the images generated with Diffusion models, I am wondering how the authors set the parameters to render images.\n\n- Is there a reason behind choosing mip-NeRF360 as the backbone? Would using basic NeRF degrade the performance?\n\n- What is s (the output of imGHUM) exactly? Is it an index of the nearest vertex on the body?\n\n- Although the lighting result is much better than the previous work (Dream Fusion), still it looks unnatural. What is the reason behind this and how it can be improved?", "limitations": "- The idea is interesting and the results are great. However, there is a concern regarding the reproducibility.", "rating": 6, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "- The result is good, showing clear improvement compared to the previous text-to-3D method.\n\n- The method can be learned without 3D GT.\n\n- The ablation study is thoroughly done.", "weaknesses": "- What is the run-time at inference?\n\n- Density loss: I think this can work when the suject is wearing a tight clothing. However, wouldn't this confuse the network when the subject is wearing a clothing with large deformatation (i.e., more gap is present between the body model and the actual geometry)? \n\n- What is w_i of equation 5?\n\n- What is the proposal weights L_p in L195?\n\n- It is written that 4 renderings are supervised for a single training step (supplementary material L7). Does this mean that 4 randomly selected semantic parts (zoomed-in semantic parts) of the same pose are rendered? Or does this mean that single semantic part is rendered from different camera pose and body pose? Also, would rendering and supervising predictions less than four (e.g., when using single GPU with less memory) degrade the performance? Discussion on the running environment and performance would be helpful for the readers.\n\n- Lack of implementation details, making it difficult to reproduce. Also, no plan on supporting the reproducibility has been presented. ", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We thank the reviewer for the valuable feedback and will address all raised questions.\n\n**Weaknesses**\n- *Runtime*: The rendering time for a 512x512 image is 2.6 seconds, whereas for 256x256 it is 0.67 seconds. These timings were measured on a TPU v3 chip with 8 cores.\n- *Density loss*: The purpose of this loss is to act as a regularizer and encourage the model to preserve important details such as facial features as we show in Fig. 4 of the Supplementary Material. Our results in Figure 2 of the main paper show that we can handle loose clothing such as dresses, something that previous work like AvatarCLIP cannot.\n- W_i are the weights of the samples along a ray as defined in Eq (1). The formulation we use is the same as in Eq (10) of Ref-NeRF [56].\n- *Proposal weights*: We use mip-NeRF 360 as our NeRF backbone, which consists of 2 MLPs, the proposal MLP and the NeRF MLP. The proposal weights are the rendering weights as defined in Eq (1) but for the proposal MLP. L_p as proposed in mip-NeRF 360 is used to supervise the proposal MLP and penalizes the proposal weights when they underestimate the distribution of the NeRF MLP. Please refer to Section 3 of mip-NeRF 360 for a more detailed discussion.\n- *Training details and environment*:\nFor each view we randomly select camera poses, semantic parts and body poses. The selection probabilities for the semantic parts are listed in Table 1 of the Supplementary Material. We combine each view with a different semantic part and a different pose. We include a discussion of the running environment in Section 1 of the Supplementary Material. Our network can be trained on TPUs or GPUs and with 16GB of memory we can fit 1 view per device.\nThe model still has acceptable performance when trained with 2 views per step but it needs more iterations to converge. With a smaller number of views, we noticed that it’s harder for our network to learn the dependency of f on $\\theta$ (Eq 3), probably because of the higher variance in the gradients, similarly to training a neural network with a small batch size. With 1 view and all other hyperparameters unmodified the  optimization becomes unstable. In Figure 1 of the rebuttal PDF we show examples trained with 2 and 4 views. \n- *Implementation details*: Thank you for pointing this out. We have included the hyperparameters and training strategy in the main paper and the supplementary material. We will expand this section in the supplementary material and include a more detailed algorithm section. As we show in Figure 4 of the rebuttal PDF, our method is generic enough and works with other open-source diffusion models such as Stable Diffusion.\n\n**Questions**\n- *Render resolution*: Since our model is NeRF-based and we trained it with camera randomizations, we are not tied to a particular resolution. In the paper figures we used a resolution of 512x512 pixels. In the Supplementary Material videos we used 256x256 due to document size constraints.\n- *Query location sampling*: We assume that the scene resides inside a unit sphere centered around the origin and we use an additional scaling factor to make sure the human fits in the unit sphere. \n- *SH*: We use a simple Spherical Harmonics Diffuse Lighting model. Given the 9 light source coefficients $c_j$, for each pixel in the image we compute the dot product between the 9-element vector of the learnable light coefficients and the vector of the corresponding spherical harmonics basis functions $h_j$ and then we use this to compute the final shaded color given the albedo. More specifically, in our formulation, the shaded pixel value $s_i$ at pixel $i$ with albedo $\\alpha_i$ and unit surface normal $(x_i, y_i, z_i)$ is $s_i = \\alpha_i \\cdot \\left(\\sum_{j=0}^8 c_j h_j(x_i, y_i, z_i)\\right)$. More details can be found in Spherical Harmonics Lighting: The Gritty Details (https://3dvar.com/Green2003Spherical.pdf). We will include all these details in the Supplementary Material.\n- *Camera parameters*: We use a reasonable distance from the subject to ensure that we don’t have severe perspective distortion effects. For example, for the full body shots the camera distance is between 2.5 and 4 meters away from the subject and the camera zoom is chosen such that the person takes up a large portion of the image and is fully visible in most frames.\n- *NeRF backbone*: mip-NeRF 360 has been shown to achieve better results than NeRF and is computationally cheaper because it needs fewer samples per ray than the standard NeRF. However any other competitive Nerf method can be used instead.\n- *imGHUM*: For a given pose θ and a point (x,y,z) in the 3D space, imGHUM returns the distance d of the point from the surface along with a semantic code s that associates it with the closest point on the surface of the posed GHUM model. If e.g. the closest point on the mesh is the tip of the index finger, then s is the 3D position of the tip of the index finger in the GHUM template mesh. The semantic code s is not an explicit vertex id, but rather a continuous surface mapping. We kindly refer the to the imGHUM paper for a more detailed explanation.\n- *Color quality*: It has been reported in previous works (e.g. DreamFusion) that the use of the SDS loss tends to produce saturated colors, and we have observed similar effects. Besides our proposed spherical harmonics estimation strategy, one way to improve the appearance could be to augment the training with an additional loss that attempts to match the statistics of our rendered images to those of natural images. \n"}, {"review_id": "MZOvSF35kr", "reviewer": "Reviewer_urhH", "summary": "This work proposes a method for text-driven human avatar generation. It combines animatable human nerf and diffusion model to implement avatar generation and animation. This work produces photorealistic avatars with high-quality details by incorporating spherical harmonics lighting model and semantic zoom. Extensive experiments demonstrate SOTA performance and the effectiveness of each design in the proposed framework.", "questions": "1. How to determine the rendering camera poses for each part in the semantic zoom?\n2. Which diffusion model is used in this work? Is this diffusion model finetuned on human body images?\n", "limitations": "Yes, addressed.", "rating": 8, "confidence": 5, "soundness": 4, "presentation": 3, "contribution": 3, "strengths": "1. This is the first diffusion-based work that successfully produces photorealistic animatable 3D human avatars.\n2. This work shows temporally consistent animation results. I believe this can open up more application possibilities for optimization-based avatar generation methods.\n3. The incorporation of spherical harmonics lighting model can alleviate the long-standing issue of unrealistic over-saturated color for text-driven 3D object generation.\n4. The semantic zoom loss is simple yet effective to improve the quality for detail regions such as face, arm, and hand.\n", "weaknesses": "1. The imGHUM is designed for the whole body, it contains parameters for hand and facial expressions. But there is no result for the animation of facial expressions and hand poses. It could be better if the authors can provide animation results to show the controllability of these details. I think this will also help prove the necessity of semantic zoom loss.\n2. I believe this work uses a more powerful diffusion model, and imGHUM is not completely open source. These two points will limit access to the proposed model. Is there any plan to release an online demo or interface for users?\n3. Although the authors mentioned the training strategy in Sup Mat, there is no clear description of the computation cost of the proposed method.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We thank the reviewer for the valuable feedback and will address all raised questions.\n\n**Weaknesses**\n1. *Hands and facial expressions*: Thanks for the very good suggestion. For the rebuttal we have produced examples of renderings with varying facial expressions (Figure 2) and hand poses (Figure 3) and we will include them in the final version. Note that this is similar to varying shape coefficients (Figure 1 of the Supplementary Material), as we did not train with varying hand poses or expressions. Nevertheless, our model is able to generalize by leveraging the capabilities of imGHUM. \n2. *Different diffusion models*: We use the same model as DreamFusion to ensure a fair comparison. Notice however, that our model is not necessarily more powerful than other public domain models like e.g Stable Diffusion. We included some example results by replacing Imagen with Stable Diffusion in Figure 4 of the rebuttal PDF. The performance of the 2 variants is perceptually very similar, thus showing that our model is agnostic to the architecture of the underlying text-to-image diffusion model. Regarding the availability of imGHUM, it is distributed under an academic license. We will strive to make a demo available.\n3. *Computation cost*: Our model is trained for 50K iterations and this takes about 6 hours on a TPU v4 machine, which is similar to other competing methods (1.5 hours for DreamFusion for static avatars,, 4-10 hours for AvatarCLIP). After training we can repose and render our model very fast, and takes 2.6 seconds for rendering a 512x512 image, and 0.67 seconds for rendering a 256x256 image.\n\n**Questions**\n1. *Semantic zoom*: Given a particular body pose and shape we can get the 3D joint locations for the body part. Given these we place the camera at a reasonable distance from the body part to minimize perspective distortion effects. We also compute a range of focal lengths that ensures that the body part occupies a large enough portion of the image.\n2. *Diffusion model*: We use Imagen, a general-purpose text-to-image diffusion model. We did not finetune on human body images. As we show in Figure 4 of the rebuttal PDF, our method works with other diffusion models such as the open-source Stable Diffusion."}, {"review_id": "WfP3iEEuRN", "reviewer": "Reviewer_hMhh", "summary": "This paper presents a method to generate animatable 3D human avatars from text. The pipeline is similar to DreamFusion and is built upon the Nerf representation and diffusion model. However, a key difference is that an imGHUM body model is introduced as prior, which allows for the construction of a deformable Nerf representation and a 3D animatable human model. This design not only enables animation capabilities but also effectively addresses the anthropometric consistency issues. In addition, a semantic zooming loss is proposed to refine details in body regions like the face and hands, resulting in a more photo-realistic overall quality.\nQuantitative and qualitative comparisons are conducted with state-of-the-art baselines such as DreamFusion and AvatarCLIP. The extensive results demonstrate that the proposed method outperforms previous approaches across all metrics. The visual quality and geometry detail are particularly impressive. Furthermore, the study includes an analysis of different components to highlight their importance within the framework.\n", "questions": "- Is there any quantitative result for more direct evaluation regarding the view consistency and pose dependency of the generated 3D avatar? \n\n", "limitations": "- As shown in Figure 7, artifacts can be found in the soldier avatar generated by DreamHuman. The generated avatar has more than two hands, and the method struggles with generating correct accessories. This can be really hard for the current model design since it's built on imGHUM body model.", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "- The paper is well-written and easy to follow.\n- The overall results are impressive, especially the appearance and the geometry details of the generated 3D human avatar.\n- Although each component can be seen in previous works like DreamFusion and AvatarCLIP, this work did a good job on putting all losses and modules together properly and achieving promising 3D human avatar modeling.\n- Extensive ablation experiments are conducted to show the importance of proposed components. \n- The proposed semantic zooming loss is interesting and effective, which largely improves the visual quality and helps to generate sharper, higher-quality textures.\n", "weaknesses": "- It’s not clear to me how to decide the shape parameters for the imGHUM model during optimization. If it is optimized together with the Nerf model, will it introduce additional training costs? \n- The overall computation cost is not clearly listed and compared. It would be better to report the optimization time and inference time for a single model/text prompt for the proposed method and baselines.\n- It would be better to show more quantitative results for the semantic zooming loss since it's one of the key contributions of this work.\n", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We thank the reviewer for the valuable feedback and will address all raised questions.\n\n**Weaknesses**\n- *Shape parameters*: The shape parameters are treated as additional parameters of the overall model during optimization and we compute the gradient of the loss with respect to them as if they were trainable weights of the neural network. This does not introduce any significant computational cost (only 10 additional latent shape parameters), whereas the NeRF MLPs have over 1 million parameters.\n- *Computation cost*: Training our method for 50K iterations takes around 6 hours per example on a TPU v4 machine. After training we can render the subject in an arbitrary pose at 512x512 resolution in 2.6 seconds, and at 256x256 in 0.67 seconds. The rendering time is similar with DreamFusion, however DreamFusion trains only for 15K iterations which takes about 1.5 hours. We need a larger number of iterations until convergence because we train a dynamic and not a static avatar. AvatarCLIP takes up to 10 hours to train on a single GPU with 32GB of memory. These are all typical times for training NeRF-based architectures.\n- *Semantic zoom*: We would like to thank the reviewer for their suggestion. We will add more results in the supplementary material.\n\n**Questions**\n- To the best of our knowledge there are no quantitative metrics to assess the view consistency and the quality of the pose-dependent deformations. To evaluate the quality of text-based 3D generations most methods rely on imperfect 2D metrics, such as the CLIP R-Precision score. However these are not very good for measuring the quality of the 3D geometry or the view-consistency of the generations, as they only measure the agreement of the rendering with the text prompt.\nWe train a NeRF with no view-dependent effects, so our model is by design viewpoint consistent. The pose consistency relies on pose-sensitive deformations learned from data.\n\n**Limitations**\n- Indeed this is an interesting failure case of our method. In fact after inspecting the result we found that although the extra hand is painted on the military uniform, no extra geometry is created. In practice we observed that text-to-image models tend to exhibit certain pose biases, i.e. when prompted to generate an image of a soldier, most generated images include soldiers holding a weapon. In the vast majority of cases, randomization during training helps mitigate this issue, but in rare cases errors like this may occur. In fact in Figure 6 of the rebuttal PDF we show that if we run again the optimization with a different random seed we get a correct result this time.\n"}, {"review_id": "z7tEmHIeFD", "reviewer": "Reviewer_FnWG", "summary": "This is a paper focusing on generating 3D animatable full-body human avatar from text using pretrained 2D diffusion model and Score Distillation Sampling (SDS).\nThe proposed approach differs from the existing approach in that, instead of directly representing a canonical space using surface template e.g. SMPL, it 1) uses the implicit 3D human model to establish the correspondence and condition the canonical representation on the pose parameter, 2) adopts a per-part optimization, and 3) uses a physics based shading formulation and jointly optimizes the environment lighting. These three technical novelty intent to achieve more plausible deformation for loose clothing, better detail reconstruction for faces and hands, and more realistic colors, respectively.\nThe comparisons with AvatarClip and DreamFusion demonstrate the advantages of the proposed method.", "questions": "It would be great to address my comments above. In particular, I look forward to seeing in the rebuttal:\n\n1. Comparison with AvatarClip\n\n2. Clarification about shading and visualization of the learned albedo\n\n3. Some visual examples of some avatars with loose clothes in walking motion, shown from the frontal view.\n\n4. It would be great to include a user study on the visual quality. \n", "limitations": "Yes", "rating": 6, "confidence": 4, "soundness": 2, "presentation": 3, "contribution": 3, "strengths": "- substantially better visual quality compared to AvatarClip and DreamFusion, although the former only with limited evidence.\n\n- the use of imGHUM seems to improve the diversity of clothing.\n\n- the part-based optimization visibly improves the visual quality of the generation.\n", "weaknesses": "- Even though the following papers can be considered concurrent, they should be mentioned in the related work. \nCao, Yukang, et al. \"Dreamavatar: Text-and-shape guided 3d human avatar generation via diffusion models.\" arXiv preprint arXiv:2304.00916 (2023).\nJiang, Ruixiang, et al. \"AvatarCraft: Transforming Text into Neural Human Avatars with Parameterized Shape and Pose Control.\" arXiv preprint arXiv:2303.17606 (2023).\n\n- The comparison with AvatarClip is not sufficient. In the supplemental material and Tab 1, results of AvatarClip should be included.\n\n- Since the NeRF in canonical space is pose-dependent, it is theoretically more prone to overfitting. How does the method perform for unseen poses?\n\n- The fact that the shape parameter $\\beta$ can vary is not well explained.\n\n- The benefit of the shading and optimization of the SH for environment light is not elaborated sufficiently. What is the albedo before shading? How exactly do you model the irradiance (include rendering equation). The training trick with randomly perturbed SH coefficients is not well motivated. If the goal is to improve disentanglement, why not use some smoothness regularization?\n\n- The animations are shown in a fixed view. It's hard to judge whether loose clothing such as dresses and jackets deform as claimed from a fixed viewing angle.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We thank the reviewer for the valuable feedback and will address all raised questions.\n\n**Weaknesses**\n- *Concurrent works*: Thanks for the suggestion. We will cite and discuss the suggested references in the final version of the paper. We would like to highlight how our method compares to these 2 approaches.\n  1. AvatarCraft proposes to learn an animatable 3D avatar, but the avatar is very much tied to the underlying SMPL model and does not include instance-specific pose-dependent non-rigid effects. As a result, the geometry of the generated avatars is very close to the underlying body geometry and cannot model loose-fitting clothing, accessories like hats or dresses (see Fig. 5 and 6 of the paper). DreamHuman however is able to handle these cases much more effectively, as shown in our qualitative results.\n  2. DreamAvatar can better capture geometric variations beyond the underlying SMPL body model, but has the major disadvantage that it needs to be retrained every time for a new pose which makes it computationally very expensive. In contrast, our model, once trained, can be easily reposed at no extra cost. Additionally, avatars produced using DreamAvatar exhibit geometric and texture artifacts.\n  3. Overall, by inspecting the quality of the generated avatars, one can see that our method is able to create significantly more realistic avatars, with better texture and geometry, and can be reposed at test time at no major additional cost.\n\n- *AvatarCLIP comparisons*: We did not include comparisons with AvatarCLIP on Table 1 because it was trained using CLIP as supervision, so the CLIP metrics would be biased towards methods that optimize CLIP similarity losses. AvatarCLIP explicitly optimizes the CLIP similarity score. Even if a CLIP model with a different architecture is used for training and testing (e.g. ViT B16 vs ViT B32), all these models were still trained on the same data with the same strategy. Similar observations have been made in Table 1 of DreamFusion in their comparison with the previous work DreamFields. Nevertheless, as requested we run AvatarCLIP using the publicly available implementation on the same 160 prompt set and report the numbers: R-Precision=0.855, Top-3=0.962, Top-5=0.981. We will include these in the final version, and also include additional AvatarCLIP results. We also conducted a user study to assess the visual quality as requested (see below).\n\n- *Pose generalization*: The results shown in the paper (Fig. 1) and in the animations are on unseen poses. From these we can conclude that our model generalizes well on novel poses. We did not observe overfitting because the 3d pose (MoCap) datasets on which the prior was trained are large-scale and diverse.\n\n- *Shape parameter variation*: We optimize the shape parameters as free variables as the shape is part of the identity (in contrast to the pose). The body shape could be constrained by elements of the prompt and we want to make sure that the shape is consistent with those elements in the prompted textual description. For example, the text prompt “a bodybuilder wearing a white shirt” implies that the generated person should have a muscular build. We will add this discussion to the paper.\n\n- *Spherical Harmonics*: For exact details about the Spherical Harmonics model we use please refer to our answer to reviewer KMW9. The primary goal of randomizing SH coefficients is to support the geometry learning (similar to Dreamfusion’s randomized light direction). Randomizing SH coefficients in some cases additionally helps to decouple albedo and lighting, e.g. a dark shirt can be obtained with the absence of light or with dark color. When randomizing the light, the former is no longer possible. However, albedo estimation was not the primary goal and we would like to investigate further in future work. Thank you for the suggestion.\n\n- Animations: We plan to release more high-resolution animation results from varying viewpoints in the final version to better showcase how our method handles clothing deformations. All animations in the supplementary material were rendered from the same viewpoint for consistency reasons. We added an unrolled animation for a skirt from a front view in Figure 7 of the rebuttal, as well as a video animation for a dress in the link (see response to AC as per the conference’s guidelines for submitting videos).\n\n**Questions**\n1. We added a CLIP-based comparison with AvatarCLIP as well as a user study as mentioned previously in our response.\n2. We added renderings of the albedo in Figure 5 of the rebuttal PDF. Overall the albedo looks plausible. For the shading please refer to our answer in the previous question.\n3. We added a rolled-out animation in the rebuttal PDF (Fig. 7)  and also included videos in the response to AC section. We will include more results like this in the final version.\n4. Following the reviewer's suggestion we conducted a user study to assess the quality of our results. We used 20 text prompts from the AvatarCLIP website from the “General Description” category. We run our method on those 20 text prompts and rendered the final results in the rest pose from the front and the side. We did the same for the precomputed meshes that the AvatarCLIP authors provide in their website. We then asked the users to rate the two methods on (a) the perceived agreement between the renderings and the input text, and (b) the perceived visual quality of the generated avatar. The ratings were on a scale from 1-5 with 1 meaning “very bad” and 5 “very good”. Our method achieved an average rating of 4.45 for “Agreement with text” and 4.16 for “Visual Quality”. AvatarCLIP scored 2.96 on “Agreement with text” and 2.49 on “Visual Quality”. In terms of the scale of the study, the user study we carried out is slightly larger than the one in AvatarCLIP (Ours: 20 images with 25 raters, AvatarCLIP: 8 images with 22 raters). We will include the full list of text prompts used in the Supplementary Material.\n\n"}]

Author rebuttal:
We would like to thank the reviewers for their valuable feedback.

- Reviewer KMW9 states that our method shows clear improvements over previous methods without the need for 3D ground truth data, and that we have a comprehensive ablation study.
- Reviewer urhH notes that our method produces animatable photorealistic 3D avatars, with temporally consistent animation results, and appreciates our novel components that improve the overall reconstruction quality.
- Reviewer hMhh highlights that our 3D generation results are impressive, appreciates our extensive ablation study and notes that the novel components of our method contribute in achieving higher-quality results.
- Reviewer FnWG praises the substantially better visual quality compared to previous methods and the importance of our semantic prompting.

We will address the questions raised by each reviewer separately under each review. The summary of the rebuttal is as follows:
- We performed extra evaluations against AvatarCLIP and conducted a user study on the visual quality of the results.
- We demonstrated that our method also works with open-source diffusion models such as StableDiffusion, thus addressing potential concerns about the reproducibility.
- We added extra clarifying qualitative results.
We will include all these results in the final version of the paper.

Meta-review: This paper aims at the task of generating an animatable human avatar given a text prompt as input. The technical contributions include: a) building on top of an implicit 3D human pose and shape model (i.e., imGHUM) which provides not only geometry but also deformation of each query point. b) adopting a semantic zoom training strategy that enhances details on small body parts (e.g., face and hands) and c) using a physics-based shading formulation that produces more realistic texture. Experimental results and ablation studies demonstrate the effectiveness of the proposed modules and its superior performance against existing methods (e.g., DreamFusion, AvatarCLIP).

The strengths of the paper are:
- All reviewers agree that the results of the paper are impressive, including photo-realistic synthesis, detailed geometry and temporally consistent animation.
- Thorough ablation studies to verify the contribution of proposed modules (Reviewer hMhh, KMW9)
- The semantic zoom training strategy and physics-based shading module are both simple yet effective (Reviewer urhH, hMhh, FnWG)

Pre-rebuttal, the reviewers mentioned the following common concerns with other clarification questions:
- Inference time. (Reviewer KMW9, urhH, hMhh)
- Reproductivity. (Reviewer KMW9)
- Missing comparisons and ablation studies. (Reviewer FnWG)

After rebuttal, reviewers acknowledged that all concerns are addressed except reproductivity. After reading the paper, reviews and rebuttals, I agree with the reviewers that this work provides valuable technique and insight to the emerging text-to-3D community, especially its significant performance compared to existing works. Thus I recommend for acceptance. Meanwhile, I urge the authors to release their code to facilitate future research.
Decision: Accept (spotlight)