DreamWaltz: Make a Scene with Complex 3D Animatable Avatars
Authors: Yukun Huang
Jianan Wang
Ailing Zeng
He CAO
Xianbiao Qi
Yukai Shi
Zheng-Jun Zha
Lei Zhang
Venue: NeurIPS 2023 poster (2023)
Keywords: Avatar Generation
3D Content Creation
NeRF
Diffusion Model
TL;DR: We propose DreamWaltz, a framework for high-quality animatable 3D avatar generation from texts.

Abstract: We present DreamWaltz, a novel framework for generating and animating complex 3D avatars given text guidance and parametric human body prior. While recent methods have shown encouraging results for text-to-3D generation of common objects, creating high-quality and animatable 3D avatars remains challenging. To create high-quality 3D avatars, DreamWaltz proposes 3D-consistent occlusion-aware Score Distillation Sampling (SDS) to optimize implicit neural representations with canonical poses. It provides view-aligned supervision via 3D-aware skeleton conditioning which enables complex avatar generation without artifacts and multiple faces. For animation, our method learns an animatable 3D avatar representation from abundant image priors of diffusion model conditioned on various poses, which could animate complex non-rigged avatars given arbitrary poses without retraining. Extensive evaluations demonstrate that DreamWaltz is an effective and robust approach for creating 3D avatars that can take on complex shapes and appearances as well as novel poses for animation. The proposed framework further enables the creation of complex scenes with diverse compositions, including avatar-avatar, avatar-object and avatar-scene interactions. See https://dreamwaltz3d.github.io/ for more vivid 3D avatar and animation results.

Dream Waltz: Make a Scene with Complex 3D Animatable Avatars
Yukun Huang 12++, Jianan Wang" +1, Ailing Zeng', He Cao', Xianbiao Qi', Yukai Shi', Zheng-Jun Zha', Lei Zhang'
International Digital Economy Academy (IDEA) University of Science and Technology of China
(a) Princess Elsa in different poses.
(b) Uzumaki Naruto in dance
(c) Waody dances with Storntrooper:
(d) Waody sits on: (Left) a chair; (Right) chair made of cheese.
Figure I: PreamWaltz is cext-to-D-avvate generation framework, which can (a. b) create complex
3D animatable avatars from texts, (c, d) ready for 3D scene composition with diverse interactions.
Abstract
We present Dream Wrltz, novel framework for generating and animating complex 3D avatars given text guidance and parametric human body prior. While recent methods have shown encouraging results for text-to-3D generation of common objects, creating high-quality and animatable 3D avatars remains challenging. To create high-quality 3D avatars, Dream Waltz proposes 3D-consistent occlusion- aware Score Distillation Sampling (SDS) to optimize implicit neural representations with canonical poses. provides view-aligned supervision via 3D-aware skeleton conditioning which enables complex avatar generation without artifacts and multi- ple faces. For animation, from abundant image priors of diffusion model conditioned on various poses, which could animate complex non-rigged avatars given arbitrary poses without retraining. Extensive evaluations demonstrate that Dream Waltz is an effective and robust approach for creating 3D avatars that can take on complex shapes and appearances as well as novel poses for animation. The proposed framework further enables the creation of complex scenes with diverse compositions, including waata-avvtta avatar-object and avatar-scens interactions. See http://dreammall223..gg github.. for more vivid 3D avatar and animation results.
method learns an animatable 3D avatar representation
Introduction
The creation and animation of 3D digital avatars are essential for various applications, including film and cartoon production, video game design, and immersive media such AR and VR. However,
Equual contribution.
Work done during an internship IDEA.
*Corresponding author.
37th Conference on Neural Information Processing Systems (NcurIPS 2023).

---

traditional techniques for constructing such intricate 3D models are costly and time- conssminn requiring thousands of hours from skilled artists with extensive aesthetics and 3D modeling knowledge. In this work, we seek solution for 3D avatar generation that satisfies the following desiderata: (1) easily controllable over avatar properties through textual descriptions; (2) capable of producing high-quality and diverse 3D avatars with complex shapes and appearances; (3) the generated avatars should be ready for animation and scene composition with diverse interactions.
The advancement of deep learning methods has enabled promising methods which can reconstruct 3D human models from monocular images [36, 45] or videos [44, 14, 46, 41, 12, 30]. Nonetheless, these methods rely heavily on the strong visual priors from image/video and human body geometry, making them unsuitable for generating creative avatars that can take on complex and imaginative shapes or appearances. Recently, integrating 2D generative models into 3D modeling [31, 18, 10] has gained significant attention to make 3D digitization more accessible, reducing the dependency on extensive 3D datasets. However, creating animatable 3D avatars remains challenging: Firstly, avatars often require intricate and complex details for their appearance (e.g., loose cloth, diverse hair, and different accessories); secondly, avatars have articulated structures where each body part is able to assume various poses coordinated and constrained way; and thirdly, avatar changes shape and texture details such as creases when assuming different poses, making animation for complex avatars extremely challenging. As a result, while DreamFusion [31] and subsequent methods [18, 5] have demonstrated impressive results on text- text-guided creation of stationary everyday objects, they lack the proper constraints to enforce consistent 3D avatar structures and appearances, which presents significant challenges to producing intricate shapes, appearances and poses for 3D avatars, let alone for animation.
In this paper, we present Dream Waltz, a framework for generating high-quality 3D digital avatars from text prompts utilizing human body prior of shapes and poses, ready for animation and compo- sition with diverse avatar-avatar, avatar-object and avatar-scene interactions. BreamWallz employs a trainable Neural Radiance Field (NeRF) as the 3D avatar representation, a pre-trained text-and- skeleton-conditional diffusion model [48] for shape and appearance supervision, and SMPL mod- els [2] for extracting 3D-aware posed-skeletonn with 3D-consistent SDS, which resolves the view disparity between the diffusion model's supervision and NeRF's rendering. By training an animatable NeRF with diffusion supervision conditioned on human pose prior,
Our method enables high-quality avatar generation
deform the generated non-rigged avatar arbitrary poses for realistic test-time animation without retraining.
The key contributions of DreamWalt lie in four main aspects:
We propose novel text-to-avatar generation framework named Dream Waltz, which is capable of creating animatable 3D avatars with complex shapes and appearances.
For avatar creation, we propose SMPL-guided 3DD-consistent Score Distillation Sampling strategy with occlusion culling, enabling the generation of high-quality avatars, e.g., avoiding the Janus (multi-face) problem and limb ambiguity.
We propose to learn an animatable NeRF representation from diffusion model and human pose prior, which enables the animation of complex avatars. Once trained, we can animate the created avatar with any pose sequence without retraining.
Experiments show that Dream Waltz is effective in creating high-quality and animatable avatars, ready for scene composition with diverse interactions across avatars and objects.
2 Related Work
Text-guided image generation. Recently, there have been significant advancements in text-to- image models such as GLIDE [26], unCLIP [33], Imagen [35], and Stable Diffusion [34], which enable the generation of highly realistic and imaginative images based on text prompts. These generative capabilities have been made possible by advancements in modeling, such as diffusion models [6, 39, 27], and the availability of large-scale web data containing billions of image-text pairs [37, 38, 4]. These datasets encompass wide range of general objects, with significant varia- tions in color, texture, and camera viewpoints, providing pre-trained models with a comprehensive understanding of general objects and enabling the synthesis of high-quality and diverse objects.

---

AvatarCLIP AvatarCraft
DreamAvatar
DreamWalds (Ours) Complex/ Animatable/
ComplexX Animatable/
Complex/ Animatable
Figure 2: Comparison of text-driven 3D avatar generation methods, including: AvatarCLIP [8], AvatarCraft [13]. DreamAvatar [3] and Dream Waltz (Ours). AvatarCLIP and AvatarCraft assume strong SMPL constraints, which makes straightforward for the generated avatars to align with SMPL for animation. But due to the constraints, the avatars cannot take complex shapes and appearances; With weak SMPL constraints, DreamAvatar struggles with wrong avatar geometry and requires retraining for each pose adjustment. Different from existing methods, Dream Waltz enables complex and animatable 3D avatar generation benefiting from the proposed SMPL-guided 3D-consistent SDS and deformation learning from human pose prior.
Furthermore, recent works [48, 9, 15] have explored incorporating additional conditioning, such as depth maps and human skeleton poses, generate images with more precise control.
Text-guided 3D generation. Dream Fields [11] and CLIPmesh [23] were groundbreaking in their utilization of CLIP [32] optimize an underlying 3D representation, aligning its 2D renderings with user-specified text prompts, without necessitating costly 3D training data. However, this approach tends result in less realistic 3D models since CLIP only provides discriminative supervision for high-level semantics. In contrast, recent works have demonstrated remarkable text-to-3D generation results by employing powerful text-to-image diffusion models robust 2D prior for optimizing trainable NeRF with Score Distillation Sampling (SDS) [31, 18, 5, 10]. Nonetheless, the produced geometry and texture are static, blurry and usually lack intricate details necessary for avatar creation.
Text-guided 3D avatar generation. Avatar-CLIP [8] starts with initializing 3D human geometry shape VAE network and subsequently employs CLIP [32] for shape sculpting and texture generation. To animate the generated 3D avatar, they propose CLIP- CLIP-guided reference-bic based motion synthesis method. However, this approach tends produce less realistic and oversimplified 3D models due to the limited guidance provided by CLIP, which primarily focuses on high-level semantic diserimination. Concurrent to our work, DreamAvatar [3] and AvatarCraft [13] both utilize pre-trained text-to-image diffusion models and SMPL models as shape prior for avatar generation.
While DreamAvatar focuses on producing static posed-3D avatars which are incapable of animation, AvatarCraft generates higher-quality avatars via carse-to-fine and multi-box training and enables animation via local transformation between the template mesh and the target mesh based on SMPL models. We summarize the key differences between our work and related works in Fig. 2.
3 Method
3.1 Preliminary
Text-to-3D generation. Recent methods [31, 18, 21] have shown encouraging results text-to-3D generation of common objects by integrating three essential components:
(I) Neural Radiance Fields (NeRF) [25, 1, 22] is commonly adopted as the 3D representation for text-to-3D generation [40, 18]. parameterized by trainable MLP. For rendering, batch of rays r(k) + kd are sampled based on the camera position o and direction d on per-pixel basis. The MI.P takes r(k) as input and predicts density 7 and color c. The volume rendering integral is then approximated using numerical quadrature to yield the final color of the rendered pixel:
No
C.(r)
Σ
Ω (1 -epeepppppp)))))
3

---

where No is the number of sampled points on ray, Ω2 - exp( Exp(-Ej=i TJoÿ) transmittance, and 6 is the distance between adjacent sample points.
) is the accumulated
(2) Diffussion models [7, 27] which have been pre-trained extensive image-fext datasets [33, 35, 40] provide a robust image prior for supervising text-to-3D generation. Diffusion models learn to estimate the denoising score Vx log Pdata (x) by adding noise to clean data x p(x) (forward process) and learning to reverse the added noise (backward process). Noising the data distribution to
isotropic Gaussian is performed in T timesteps, with pre-defined noising schedule ar E (0, 1) and
ae = IT==1 ag. according to:
ūxX + vI- ā,e, where e ~ (O,
In the training process, the diffusion models learn to estimate the noise by
L = (x,e~~W(...) [|Eg (ze, 1) - -11
Once trained. one can estimate x from noisy input and the corresponding noise prediction.
(3) Score Distillation Sampling (SDS) [31, 18, 21] is technique introduced by DreamFusion [31] and extensively employed to distill knowledge from pre-tran pre-trained diffusion model c into differentiable 3D representation. For NeRF model parameterized by 0, its rendering x can be obtained by g()) where g a differentiable renderer. SDS calculates the gradients of NeRF parameters 0 by,
On, θx
Vo x) Er,e (e(t)(E -
(1)
0x 00
where (t) a weighting function that depends on the timestep I and denotes the given text prompt.
SMPL [19] is 3D parametric human body model with vertex-based linear deformation model, which decomposes body deformation into identity-related and pose-related shape deformation. contains N 6, 890 vertices and K = 24 keypoints. Benefiting from its efficient and expressive human motion representation ability. SMPL has been widely used in human motion-driver tasks [8, 47, 20). SMPL parameters include a 3D body joint rotation & E IR xx333 3, body shape 3 E I0 and 3D global scale and translation R
Formally, constructing a rest pose E)3,E) involves combining the mean template shape T from the canonical space, the chape-dependent deformations B3) R3N and the pose-dependent deformations BC( E IRNN to relieve artifacts in a standard linear blend skinning (LBS) [24] by,
T(B x) = T (s( + Bp(E).
To map the SMPL parameters 3,5 to a triangulated mesh, function M is adopted to combine the rest pose mesh ((B. E). the corresponding keypoint positions T(3) E 3R3,, pose &, and set of blend weights W R N via a LBS function as,
M(B.E) ((I(B.
To obtain the corresponding vertex under the observation pose vo. affine deformation (E, (s) with skinning weight Wi is used to transform the ki keypoint j& from the canonical pose to the observation pose as,
V
(2)
3.2 DreamWaltz: A Text-to-Avatar Generation Framework
3.2.1 Creating a Canonical Avatar
DreamWaltz employs a trainable NeRF as the 3D avatar representation. It leverages SMPL prior in two ways: (1) initializing NeRF; (2) extracting 3D-aware and occlusion-aware posed-skeletons to condition ControlNet [48] for 3D-consistent Score Distillation Sampling. Fig 3 (a) illustrates our method how to create canonical avatar.

---

Stage I: Canonical Avatar Creation
Canonical SMPL
Skeleton
Occlusion C Culling
Skeleton
Keypaint Extroction
Random Camera View
Right Ear Noos
3D-t--2D Projection
Cavisilel
3D Consistancy
(visible)
Lsds
ControlNet (frozen)
Feature
Feature
o
Queey
Decade
MLP
Hash Look-up
Rendering
Rendered
"Noruto Urumaki"
bservation
Feature Grid
Network
Image
Noise
Text Prompt
Canonical Avatar Representation
3D-consistent Score Distillation Sampling
(a) Canonical Avatar Creation. We randomly sample camera viewpoint to render both the 3D avatar and skeleton image with occlusion culling.
SMPL model. The extracted SMPL 3D keypoints are projected The skeleton image is by construction 3D-consistent with the rendered avatar. We then utilize ControlNet [48) conditioned on the text prompt and the skeleton image instead of the original SD [34].
Point-Vertex
Distance
Position
Random Sampling
Keypoint
Eacoding
MLP
Wd
Closest Vertex Query
Human Pose Prior
Posed SMPL
Skeleton
Inverse-LBB
Density Weighting
Arbitrary Pose-Guided Conditioning
Feature
Lods
3D 3D-consistent Score Distillation Sampling
Decode
Hash Feature Grid
Rendering
Stage II: Diffusion-G uided Animatable Avatar L Learning
iservvaion
Network
Heeneeed
Image
Animatable Avatar Representation
(b) Animatable Avatar Learning. We randomly sample viable poses from off-the-shelr VPoser [29] for SMPL model condition ControlNet [48] and to learn generalizable density weighting function refine vertex-hases pose transformation, enabling animation of complex avatars.
Figure 3: Illustration our framework for canonical and animatable avatar creation. (a) shows how create canonical avatar from text with 3D>-consistent occlusion-awar Score Distillation Sampling, and (b) demonstrates how to further learn an animatable avatar with sampled human pose prior.
SMPL-guided initialization. To speed up the NeRF optimization and to provide a reasonable initial input for retrieving informative supervision from the diffusson model, we pre-train NeRF based on an SMPL mesh. The SMPL model could be in the canonical pose as adopted in our method to avoid self-occlusion silhouette image Za of the SMPL model given randomly sampled viewpoint and minimize the MSE loss between the NeRF-rendered image and the silhouette image wan. Note that our NeRF renders images in the latent space Stable Diffusion [34], so is necessary to use the VAE-based image encoder to transform silhouette images into the latent space for the loss calculation. We empirically found that SMPL-guided NeRF initialization significantly improves the geometry and the convergence speed for avatar generation.
in any chosen pose for posed-avatar creation [3]. Specifically, we render the
3D-consistent Score Distillation Sampling. Vanilla SDS [40, 43, 18] as previously introduced in Sec. 3.1 utilizes riewddependent prompt augmentations such as "front view of ..." for diffusion model provide crucial 3D niew-consistent supervision. However, this prompting strategy cannot guarantee precise view consistency, leaving the disparity between the viewpoint of the diffusion model's 's supervision image and NeRF's rendering image unresolved. Such inconsistency causes quality issues for 3D generation, such as blurriness and the Janus (multi-face) problem. Inspired by recent works controllable image generation [48, 15), we propose utilize additional 3D-aware conditioning images to improve SDS for 3D-consistent NeRF optimization. Specifically, additional

---

conditioning image e is injected to Eq. for SDS gradient computation:
Oze dx
VoLsDs (0, x) = Er,e (xx)(
c) Ox a0
(3)
where conditioning image can be one or a combination of skeletons, depth maps, normal maps and etc. In practice, we choose skeletons as the conditional image type because they provide minimal image structure priors and enable complex avatar generation. In order to acquire 3D-consistent supervision, the conditioning image's viewpoint should be in syne with NeRF's rendering viewpoint. To achieve this for avatar generation, we use human SMPL models to produce conditioning images.
Occlusion culling. The introduction of 3w-awere conditional images enhance the 3D consistency in the SDS optimization process. However, the effectiveness is constrained by the adopted diffusion model [48] its interpretation of the conditional images. As shown in Fig. 8 (a), we provide back-view skeleton map as the conditional image to ControlNet [48] and perform text-to-image generation, but a frontal face still appears in the generated image. Such defects bring problems such as multiple faces and unclear facial features to 3D avatar generation. To this end, we propose to use occlusion culling algorithms [28] in computational graphies detect whether facial keypoints are visible from the given viewpoint and subsequently remove them from the skeleton map if considered invisible. Body keypoints remain unaltered because they reside in the SMPL mesh, and it difficult to determine whether they are occluded without introducing new priors.
3.2.2 Learning an Animatable Avatar
Fig. (b) illustrates our framework for generating animatable 3D avatars. In the training process, we randomly sample SMPI. models of viable poses from VPoser [29] to condition ControlNet [48] and to learn generalizable density weighting function to refine vertex based pose transformation, enabling animation of complex avatars. At test time, DreamWaltz is capable of creating animation based on arbitrary motion sequences without requiring further pose-by-pose retraining.
SMPL-guided avatar articulation. Referring to Sec. 3.1, SMPL defines vertex transformation from observation space to canonical space according Eq. 2. In this work, we use SMPL-guided transformation to achieve NeRF-represented avatar articulation. More concretely, for each sampled point p on a NeRF ray, we find its closest vertex V. based on a posed SMPL mesh. We then utilize the transformation matrix skel of Ve to project to the canonical space feature grid for feature querying and subsequent volume rendering. When p is close to Ve. the calculated articulation approximately correct. However, for non-skin-tight complex avatars, p may be far away from any mesh vertex, resulting in erroneous coordinate transformation causing quality issues such as extra limbs and artifacts. To avoid such problems, we further introduce novel density weighting mechanism.
Density weighting network. We propose density weighting mechanism to suppress color con- tribution from erroneously transformed point p. effectively alleviating undesirable artifacts. To achieve this, we train generalizable density weighting network MLPDwN. More concretely, we project sampled point p and its closest vertex V. to the canonical space via the transformation matrix Takt, embed these two coordinates with the positional embedding function PE(() and then use the concatenated embeddings as inputs to MLPpwN. The process can be defined as,
d' (LLPPN(PE((Tatt.) p) + PE(T vo)).
We then compute density weights ua according to the distance d between p and ve, and d':
wa Sigmoid(-(d 1--dd') da)/a).
(5)
where a is preset parameter. Finally, the density ö of sampled point p is re-weighted to 8 wa for subsequent volume rendering in NeRF.
Sampled human pose prior. To enable animating generated avatars with arbitrary motion sequences, we need to make sure that the density weighting network MLPpww is generalizable arbitrary poses. To achieve this, we utilize VPoser [29] as a human pose prior, which is a variational autoencoder that learns a latent representation of human pose. During training, we randomly sample SMPL pose parameters& & from VPoser to construct the corresponding posed meshes. We utilize the mesh to (1) extract skeleton maps as conditioning images for 3D-consistent SDS; (2) to serve as mesh guidance for learning animatable avatar representation. This strategy aligns avatar articulation learning with

---

View
View
Gintoki
Link in Zelda
Mulan
Woody Toy Story
Doctor Who
Joker
Retro astronaut with bulky suit
Kratos
Spiderman
Figure 4: Qualitative results from two views of Dream Waltz. Given text prompts, it can generate high-quality 3D avatars with complex geometry and texture.
SDS supervision, ensuring that MLPDWN could learn generalizable density weighting function from diverse poses. We also observe that SDS with diverse pose conditioning could further improve the visual quality of created avatar, c.g., sharper appearance.
3.3 Making a Scene with Animatable 3D Avatars
Dream Waltz provides an effective solution to animatable 3D avatar generation from text, readily applicable make a scene with diverse interactions. For example, different animatable avatars could be rendered in the same scene, achieving avatar-avatar animation. In Fig. 1 (c), we can make the avatar "Woody" dance with "Stormtrooper". The animation process does not require any retraining. However, the constructed scene might exhibit unruliness, artifacts, interpenetration issues because the naive composite rendering does not take into account the interactions between different components.
Although not our main contribution, we explore the refinement of compositional scene representation with the proposed 3D-consistent SDS. Firstly, employ Dream Waltz and Latent-NeRF [21] to generate avatars and objects of the desired scene, respectively. Then, we manually set the motions of avatars and objects, and adopt composite rendering [13] to obtain scene image. For scene refinement, we further provide text prompt of the desired scene for ControlNet, and use animatable avatar learning of Dream Waltz fine-tune the whole scene representation. Benefiting from diffusion model's 's scene-related image priors, the visual quality of the generated dynamic 3D scene could be further improved. More results and discussions are provided in Appendix B.4.
Experiment
We validate the effectiveness of our proposed framework for avatar generation and animation. In Sec. 4.1, we evaluate avatar generation with extensive text prompts for both qualitative comparisons and user studies. In Sec. 4.2, demonstrate avatar animation given novel motion sequences. We present ablation analysis in Sec, 4.3 and illustrate that our framework can be further applied make complex scenes with diverse interactions in Sec. 4.4.
Implementation details. FreamWaltz is implemented in PyTorch and be trained and evaluated on single NVIDIA 3090 GPU. For the canonical avatar creation stage, we train the avatar representation for 30,000 iterations, which takes about an hour. For the animatable avatar learning stage. the avatar representation and the introduced density weighting network are further trained for 50,000 iterations. Inference takes less than 3 seconds per rendering frame. Note that the two stages can be combined for joint training, but we decouple them for training efficiency.
4.1 Evaluation of Canonical Avatars
High-quality avatar generation. We show 3D avatars created by Dream Waltz diverse text prompts in Fig. 4. The geometry and texture quality are consistently high across different viewpoints for different text prompts. See Appendix A. for more discussions and results.

---

Comparison with SOTA methods. We compare with existing SDS-based methods for complex (non- -oon-kkintiigh) avatar generation. Latent-NeRF [21] and SJC [43] are general text-to-3D models. AvatarCraft [13] is not included for comparison because it cannot generate complex avatars. Further- more, AvatarCraft utilizes a coarse-to- and body-face separated training to improve generation quality, which is orthogonal to our method. DveamAvatar [3] is the most relevant to our method, utilizing SMPL human prior without overly constraining avatar complexity. It is evident from Fig. that our method consistently achieves higher quality in geometry and texture.
OuR
DreamAvaler
Laieni NeRF
sJC
Mobile Suit Gundam
CAO CAO
DragonBall
Figure 5: Qualitative comparisons for complex avatar generation. Text inputs are listed below.
User studies. We conduct user studies to evaluate the quality of avatar generation against existing text-to-3D generation methods: Latent-NeRF [21]. SJC [43] and DreamAvatar [3]. We use the 25 text prompts DreamA- vatar released and their showcase models for comparison. We asked 12 volunteers to score 1 (worst) to 4 (best) in terms of (1) avatar geometry and (2) avatar texture. We do not measure text alignment the avatar prompts are well-known characters universally respected by all the competing models. As shown in Fig. 6, the raters favor avatars generated by DrttmWaltz. for both better geometry and texture quality. Specifically, DreamWalte outperforms the best competitor Avaarr by score 0.67 on geom- etry and by score 1.01 texture.
4.0
Latent-NoRF
User Studies
3.5
DrsamAvattr
Ours
3.0
2.5
2.0
1.5
1.0
0.5
3.05
3:72
1.19
1.73
2.74
0.0
Geometry
Toxture
User preference study. Dream Waltz obtains higher scores in both geometry and texture.
Figure 6:
4.2 Evaluation of Animatable Avatars
We demonstrate the efficacy of our animation learning method with animation results two motion sequences as shown Fig. 7: row (a) displays motion sequences skeletons as animation inputs; our framework directly applies the motion sequences to our generated avatars "Flynn Rider" and "Woody" without any retraining. We render corresponding normal and RGB sequences (b) and (c). In (d), we provide free- viewpiitn rendering of a chosen sequence frame. Since we essentially infer a posed 3D avatar for each motion frame, the renderings are by construction 3D-consistent and can creatively use the 3D model for novel-view video generation.

---

(b) Nammall (o) Skokkon
Malion
(c) RGB
Fine View
(ci) RGB
Flynn Rider
Woody in Toy Story
Figure 7: Animation results by applying skeleton motion sequences to animatable avatars generated
by DreamWaltz. For each pose frame we could effortlessly render 3D-consistent fice-viewpoints.
4.3 Ablation Studies
To evaluate the design choices of DreamWaltz, ablate the effectiveness of our proposed occlusion culling and animation learning.
Effectiveness of occlusion culling. Occlusion culling is crucial for resolving view ambiguity, both for and generation, as shown in Fig. (a) and Fig. (b), respectively. Limited by the view-aware capability, ControlNet fails to generate the back-view image of character even with view-dependent text and skeleton prompts, as shown in Fig. (a). The introduction of OC eliminates the ambiguity of skeleton conditions and helps ControlNet to generate correct views. Similar effects can be observed in text-to-3D generation, as shown in Fig. (b).
w/O OC
w/ OC
w/o Occlusion Culling
w/ Ooclusion Culling
(a) ControlNet: "backviewof Mulan"
(b) Text-to-33 tonttto-DDD-o0 "Miku"
Figure 8: Ablation study on occlusion culling (OC), Occlusion culling refines the skeleton condition
image by removing occluded human keypoints, which helps (a) ControlNet [48] to correctly generate character back view, (b) text-to-3D model to resolve the multi-face problem.
Effectiveness of animation learning. We compare our animation learning strategy with other animation methods, including: vanilla Inverse-LBS (Baseline) and AvatarCraft [13]. From Fig. 9, evident that our animation learning method significantly more effective than Inverse-LBS and AvatarCraft [13]. Note that AvatarCraft cannot faithfully animate complex features, e.g. Woody's 's hat.
Inverse-L.Bs
AvatarCraft
DreamWaltz (Ours)
Figure 9: Comparison of different animation strategies, including: Inverse-LBS (Baseline), Avatar-
Craft [13]. and DreamWall (Ours). Our method learns to animate avatars from pose-conditiones image priors, therefore able to animate complex characters with significantly higher quality.

---

4.4 Further Analysis and Application
Shape control via SMPL parameter G. Dream Waltz allows shape adjustment for controllable 3D avatar generation. As shown in Fig. 10, the body proportions of the generated avatars can be changed by simply specifying different SMPL parameter 3.
Short "Liunel Mesai"
Tall "Michuel Jordun"
Tall "Lionel Messi"
Fat "Michael Jordan"
Figure 10: Our method supports adjusting the body proportions of avatars by adjusting the SMPL parameter For example, athlete avatars of different heights and fatness be generated.
Creative and diverse avatar generation. Dream Waltz achieves stable 3D avatar generation via 31D-consistent SDS, enabling creative and diverse 3D avatar creation with low failure rates. Given imaginative text prompts and different random seeds, method successfully generates various high-quality avatars, e.g., "a doctor wearing Woody's hat", as shown in Fig. 11.
0
sood=1
sood=2
sood=33
So0d=0
sood=
Sood=2
sood=3
doctor wearing Woody's hat
Taylor Swift in Snow W hite costume
Figure
Creative avatar creation with different random seeds and initial noises.
Scene composition with diverse interactions. Benefiting from DreamWaltz could compose animatable avatars and other 3D assets into the same scene. We give few examples to highlight the potential for enabling diverse interactions: (1) avatar-avatar interaction as shown in Fig. (c) depicting "Woody" dancing with "Stromtrooperr: and (2) avatar-scene interaction as shown in Fig. (d) with "Woody" sitting on different chairs. The interactions can be freely composed make dynamic scene. More results are provided in Appendix A.3.
5 Discussion and Conclusion
Limitation. Although DreamWall can generate SOTA high-quality complex avatars from textual descriptions, the visual quality can be significantly improved with higher resolution training higher time and computation cost. The quality face and hand texture can further improved through dedicated optimization of close-up views well adopting stronger SMPLX instead of SMPL.
Societal Impact. Given our utilization of Stable Diffusion (SD) as the 2D generative prior, our model could potentially inherit societal biases present within the vast and loosely curated web content harnessed for SD training. We strongly encourage the usage to be transparent, ethical, non-offensive and conscious avoidance of generating proprietary characters.
Conclusion. We propose Dream Waltz, novel learning framework for avatar creation and animation with text and human shape/pose prior. For high-quality 3D avatar creation, we propose to leverage human priors with SMPL-guided initialization, further optimized with 3D-consistent occlusion- aware Score Distillation Sampling conditioned on 3D-aware skeletons. Our method learns an animatable NeRF representation that could retarget the generated avatar to any pose without retraining, enabling realistic animation with arbitrary motion sequences. Extensive experiments show that Dream Waltz effective and robust with state-of-the- the-art avatar generation and animation. Benefiting from Dream Waltz, we could unleash our imagination and make diverse scenes with avatars.
10

---

References
[1] Jonathan T Barron, Ben Mildenhall, Matthew Tancik, Peter Hedman, Ricardo Martin- Baralaa and Pratul P Srinivasan. Mip-NeRF: A Multiscale Representation for Anti-Aliasing Neural Radiance Fields. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pages 5855-5864, 2021.
[2] Federica Bogo, Angjoo Kanazawa, Christoph Lassner, Peter Gehler, Javier Romero, and Michael J Black. Keep it SMPL: Automatic estimation of 3D human pose and shape from single image. In Computer Vision-ECCV 2016: 14th European Conference, Amsterdam, The Netherlands, October 11-14, 2016, Proceedings, Part V 14, pages 561-578. Springer, 2016.
[3] Yukang Cao, Yan-Pei Cao, Kai Han, Ying Shan, and Kwan- Yee K Wong. DreamAvatar: Text-and-Shape Guided 3D Human Avatar Generation via Diffusion Models. arXiv preprint arXiv:23304.0 00916, 2023.
[4] Soravit Changpinyo, Piyush Sharma, Nan Ding, and Radu Soricut. Conceptual 12M: Pushing Web-Scale Image-Text Pre- Pre-Trrining To Recognize Long-Tail Tail Visual Concepts. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 3558-3568,
2021.
[5] Rui Chen, Yongwei Chen, Ningxin Jiao, and Kui Jia, Fantasia3D: Disentangling Geometry and Appearance for High-quality Text-to-3D Content Creation. arXiv preprint arXiv:2303.1387.. 2023.
[6] Prafulla Dhariwal and Alexander Nichol. Diffusion Models Beat GANs on Image Synthesis. Advances in Neural Information Processing Systems, 34:8780-8794, 2021.
[7] Jonathan Ho, Ajay Jain, and Picter Abbeel. Denoising Diffusion Probabilistic Models. Advances in Neural Information Processing Systems, 33:6840-6851, 2020.
[8] Fangzhou Hong, Mingyuan Zhang, Liang Pan. Zhongang Cai, Lei Yang, and Ziwei Liu. Avatar- CLIP: Zero-Shot Text-Driven Generation and Animation of 3D Avatars. ACM Transactions on Graphics (TOG), 41(4):1-19, 2022.
[9] Lianghua Huang, Di Chen, Yu Liu, Yujun Shen, Deli Zhao, and Jingren Zhou. Com- poser: Creative and controllable image synthesis with composable conditions. arXiv preprint arXiv:2302.09778, 2023.
[10] Yukun Huang, Jianan Wang, Yukai Shi, Xianbiao Qi, Zheng-Jun Zha, and Lei Zhang. Dream- Time: An Improved Optimization Strategy for Text-to-3D Content Creation. arXiv preprint arXiv:2306. /242,, 2023.
[11] Ajay Jain, Ben Mildenhall, Jonathan T Barron, Pieter Abbeel, and Ben Poole. Zero-Shot Text- Guided Object Generation With Dream Fields. In Proceedings of the IEEF/CVF Conference on Computer Vision and Pattern Recognition, pages 867-876, 2022.
[12] Boyi Jiang, Yang Hong, Hujun Bao, and Juyong Zhang. Selfrecon: Self reconstruction your digital avatar from monocular video. In Proceedings of the IEEE/CVF Conference Computer Vision and Pattern Recognition, pages 5615, 2022.
[13] Ruixiang Jiang, Can Wang, Jingbo Zhang. Menglei Chai, Mingming He, Dongdong Chen, and Jing Liao. AvatarCraft: Transforming Text into Neural Human Avatars with Parameterized Shape and Pose Control. arXiv preprint arXiv:2303.1 17666, 2023.
[14] Wei Jiang, Kwang Moo Yi, Golnoosh Samei, Oncel Tuzel, and Anurag Ranjan. Neuman: Neural human radiance field from a single video. In Computer Vision-ECCV 2022: 17th European Conference, Tel Aviy, Israel, October 23-27, 2022, Proceedings, Part XXXII, pages 402-418. Springer, 2022.
[15] Xuan Ju, Ailing Zeng, Chenchen Zhao, Jianan Wang, Lei Zhang, and Qiang Xu. HumanSD: Native Ekeleton-Guided Diffusion Model for Human Image Generation. arXiv preprint 2xX4v:230..04269, 2023.

---

[16] Diederik P Kingma and Jimmy Ba. Adam: A Method for Stochastic Optimization. In Interna- tional Conference on Learning Representations. 2015.
[17] Ruilong Li, Shan Yang, David A. Ross, and Angjoo Kanazawa. Learn to Dance with + Music Conditioned 3D Dance Generation, 2021.
[18] Chen-Hsuan Lin, Jun Gao, Luming Tang, Towaki Takikawa, Xiaohui Zeng, Xun Huang, Karsten Kreis, Sanja Fidler, Ming- Yu Liu, and Tsung-Yi Lin. Magic3D: High-Resolutee Text-to-3D Content Creation. arXiv preprint arXiv:2211.10440 2022.
[19] Matthew Loper, Naureen Mahmood, Javier Romero, Gerard Pons-Moll, and Michael J Black. SMPL: a skinned multi-person linear mode. ACM transactions on graphics (TOG), 34(6):1-11, 2015.
[20] Naureen Mahmood, Nima Ghorbani, Nikolaus F Troje, Gerard Pons-Moll, and Michael J Black. AMASS: Archive of motion capture as surface shapes. In Proceedings of the IEEE/CVF international conference on computer vision, pages 5442-5451, 2019.
[21] Gal Metzer, Elad Richardson, Or Patashnik, Raja Giryes, and Daniel Cohen-Or Or. Latent-NeRF for Shape-Guided Generation of 3D Shapes and Textures. arXiv preprint arXiv:2211.07600, 2022.
[22] Ben Mildenhall, Pratul Srinivasan, Matthew Tancik, Jonathan T Barron, Ravi Ramamoorthi, and Ren Ng. NeRF: Representing Scenes as Neural Radiance Fields for View Synthesis. Communications of the ACM, 65(1)999-106, 2021.
[23] Nasir Mohammad Khalid, Tianhao Xie, Eugene Belilovsky, and Tiberiu Popa. CLIP-Mesh: Generating textured meshes from text using pretrained image-text models. In SIGGRAPH Asia 2022 Conference Papers. pages 1-8, 2022.
[24] Alex Mohr and Michael Gleicher. Building efficient, accurate character skins from examples. ACM Transactions Graphics (TOG), 22(3):562-555 -58, 2003.
[25] Thomas Müller, Alex Evans, Christoph Schied, and Alexander Keller. Instant Neural Graphics Primitives with Multiresolution Hash Encoding. ACM Transactions on Graphics (ToG), 41(4):1-15, 2022.
[26] Alex Nichol, Prafulla Dhariwal, Aditya Ramesh, Pranav Shyam, Pamela Mishkin, Bob McGrew, Ilya Sutskever, and Mark Chen. GLIDE: Towards Photorealistic Image Generation and Editing with Text-Guided Diffusion Models. arXiv preprint arXiv:2 12. 10741, 2021.
[27] Alexander Quinn Nichol and Prafulla Dhariwal. Improved Denoising Diffusion Probabilistic Models. In International Conference Machine Learning, pages 8162-8171. PMLR, 2021.
[28] Ioannis Pantazopoulos and Spyros Tzafestas. Occlusion Culling Algorithms: A Comprehensive Survey. Journal of Intelligent and Robotic Systems, 35: 123-156, 2002.
[29] Georgios Pavlakos, Vasilcios Choutas, Nima Ghorbani, Timo Bolkart, Ahmed AA Osman, Dimitrios Tzionas, and Michael J Black. Expressive body capture: 3d hands, face, and body from single image. In Proceedings of the IEEF/CVF conference on computer vision and pattern recognition, pages 10975-109888 2019.
[30] Sida Peng, Junting Dong, Qianqian Wang, Shangzhan Zhang, Qing Shuai, Xiaowei Zhou, and Hujun Bao. Animatable neural radiance fields for modeling dynamic human bodies. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pages 14 14323, 2021.
[31] Ben Poole. Ajay Jain, Jonathan T Barron, and Ben Mildenhall. DreamFusion: Text-to-3D using 2D Diffusion. arXiv preprint arXiv:2209. /4988, 2022.
[32] Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, et al. Learning Transferable Visual Models From Natural Language Supervision. International Conference on Machine Learning, pages PMLR, 2021.
12

---

[33] Aditya Ramesh, Prafulla Dhariwal, Alex Nichol, Casey Chu, and Mark Chen. Hierarchical Text-Conditional Image Generation with CLIP Latents. arXiv preprint arXiv:2204.06125, 2022.
[34] Robin Rombach, Andreas Blattmann, Dominik Lorenz, Patrick Esser, and Bjöm Ommer. High- Resolution Image Synthesis with Latent Diffusion Models. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 10684-10695, 2022.
[35] Chitwan Saharia, William Chan, Saurabh Saxena, Lala Li, Jay Whang, Emily Denton, Seyed Kamyar Seyed Ghasemipour, Burcu Karagol Ayan, S Sara Mahdavi, Rapha Gontijo Lopes, et al. Photorealistic Text-to-lmag Image Diffusion Models with Deep Language Understanding. arXiv preprint arXiv:2205. 1487, 2022.
[36] Shunsuke Saito, Zeng Huang, Ryota Natsume, Shigeo Morishima, Angjoo Kanazawa, and Hao Li. Pifu: Pixel-aligned implicit function for high- nigh-resolution clothed human digitization, In Proceedings of the IEEE/CVF international conference on computer vision, pages 2304-24 2214,, 2019.
[37] Christoph Schuhmann, Romain Beaumont, Richard Vencu, Cade Gordon, Ross Wightman, Mehdi Cherti, Theo Coombes, Aarush Katta, Clayton Mullis, Mitchell Wortsman, et al. LAION- SB: An open large-scale dataset for training next generation image-text models. arXiv preprint arXiv:2210.08402. 2022.
[38] Piyush Sharma, Nan Ding, Sebastian Goodman, and Radu Soricut, Conceptual Captions: A Cleaned, Hypernymed, Image Alt-text Dataset For Automatic Image Captioning. In Proceedings of the 56th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 2556-2565, 2018.
[39] Jiaming Song, Chenlin Meng, and Stefano Ermon. Denoising Diffusion Implicit Models. In International Conference on Learning Representations, 2021.
[40] Jiaxiang Tang. Stable-dreamfussinn: http::/ggiaaa..oooaaaaav////eeee
Text-to-3d with stable-diffusion, 2022.
[41] Gusi Te, Xiu Li, Xiao Li, Jinglu Wang, Wei Hu, and Yan Lu. Neural Capture of Animatable 3D Human from Monocular Video. In Computer Vision-ECCV 2022: 17th European Conference, Tel Aviv, Israel, October 23-27, 2022, Proceedings, Part VI, pages 275-291. Springer, 2022.
[42] Timo Von Marcard, Roberto Henschel, Michael J Black, Bodo Rosenhahn, and Gerard Pons- Moll. Recovering Accurate 3D Human Pose in The Wild Using IMUs and Moving Camera. In Proceedings of the European conference on computer vision (ECCV), pages 601-617, 2018.
[43] Haochen Wang, Xiaodan Du, Jiahao Li, Raymond A. Yeh, and Greg Shakhnarovich. Score Jacobian Chaining: Lifting Pretrained 2D Diffusion Models for 3D Generation. arXiv preprint arXiv:2212.0077 2022.
[44] Chung- Yi Weng, Brian Curless, Pratul P Srinivasan, Jonathan T Barron, and Ira Kemelmacher- Shlizerman. Humannerf: Free-viewpoint rendering of moving people from monocular video. In Proceedings of the IEEE/CVF Conference Computer Vision and Pattern Recognition, pages 16210-16220, 2022.
[45] Yuliang Xiu, Jinlong Yang, Xu Cao, Dimitrios Tzionas, and Michael J Black. ECON: Explicit Clothed humans Obtained from Normals. arXiv preprint arXiv:2212.07422 2022.
[46] Zhengming Yu, Wei Cheng, Xian Liu, Wayne Wu, and Kwan- Yee Lin. MonoHuman: Animat- able Human Neural Field from Monocular Video. arXiv preprint arXiv:2304. 020001, 2023.
[47] Ailing Zeng, Xuan Ju, Lei Yang, Ruiyuan Gao, Xizhou Zhu, Bo Dai, and Qiang Xu. DeciWatch: Simple Baseline for 10x Efficient 2D and 3D Pose Estimation. In Computer Vision-ECCV 2022: 17th European Conference, Tel Aviv, Israel, October 23-27, 2022, Proceedings, Part V, pages 607-624, Springer, 2022.
[48] Lvmin Zhang and Maneesh Agrawala. Adding Conditional Control to Eoxt-to-Image Diffusion Models. arXiv preprint arXiv:2302.05543 2023.
13

---

Appendix
In this supplementary material, Sec. A presents more qualitative results of Dream Waltz on avatar creation, animation and interaction. Sec. B gives more analysis on the design choices of our method. Section C provides more comprehensive implementation details.
A More Qualitative Results
We provide more results of our proposed method, including more generated avatars in Sec. A.I, more animated avatar sequences in Sec. A.2, and more demonstrations on diverse interactions in Sec. A.3.
A.1 Avatar Creation
We provide more text-to-3D avatar generations in Fig. 12 with wide range of text prompts including celebrities, popular cartoon/movie characters and text descriptions. Note that Dream Waltz is capable of generating diverse avatars. For instance, we can produce avatars with human-realisen like "Tiger Woods", avatars wearing intricate clothing such as "Napoleon" and "Marie Antoinette", and avatars tailored to user ser-ppoovded characteristics like "Blue fairy with wings", among others.
appearance
A.2 Avatar Animation
We provide more animation results six characters shown in Fig. 13. Please refer the project page at https://dreamwalzz3d, githuu..o for more animation sequences.
A.3 Diverse Interaction
We provide more results of diverse interactions in Fig. 14, including: avatar-object, avatar-scene, and avatar-avatar interactions. Please refer to the videos on project page at http::/dddammaallz22giibbb.. for the full sequences.
B
More Analysis
B.1 Visualization of SDS Gradients
We visualize the SDS supervision gradients (Z: y, <e| for NeRF renderings in Fig. 15 (a) and the denoised images derived from noise predictions eg (zz; (2, y. f) in Fig. 15 (b). These visualizations are based on the text prompt y of "superman" and f. of 980, while additional conditioning of depth and skeleton is provided in the second and third row, respectively, rendering viewpoint of NeRF. It is evident that depth and skeleton images offer more informative optimization gradients compared to text alone. However, depth images heavily rely on the SMPL prior, leading to gradients that conform tightly to the avatar's 's skin, resulting in the disappearance of superman's s cape. On the other hand, skeleton images as adopted by Dream provide both informative and flexible supervision, accurately capturing the avatar's shape, pose, and intricate details such as the cape.
accordance with the current
B.2 Effects of Random-Pie Optimization on Avatar Quality
In Fig. 16, we present visualizations of the avatars obtained at various stages, all depicted in canonical pose. In Stage I, static avatar generated by optimizing its 3D representation with the canonical pose. In Stage II, the system undergoes training on randomly sampled human poses to facilitate animation learning. Although not our primary objective, this design enables the generated avatar to further refine its appearance with different poses. As result, minor adjustments are observed, typically leading to sharper details. However, some cases, the geometry may become more simplified, aligning more closely with the SMPL prior.
14

---

(al View
D View
Tayfor
Abraham Uncoin
View
View
Napolsan
Baseball payer
Dustor
Super Marie
Monkay Lulfy
Ca View
(0) View 2
Blue fairy with wings
Snow White, Dianey
Belle, Disney
Rapunzel
Peter Pan
Arid Daney
Merida, Disney
Da) View
(0) Viww
Luke Skywalker
Ther
Desdpool
Batman
Green Anoow
Harley Quin
Tods
Pb View
Superman
Dootor Strngg
Hulk
Assassin Albir
Jack Marston
Lincel Musi
Michael Jondan
Figure 12: Text-to-3D avatars generated with Dream Waltz, each displayed for two views.
15

---

Flynn Rider
Joker
Mobile Suit
Thanos
Gintoki
Mulan
Figure 13: Avatar animations on six characters.
(a) Avatar-object interaction. Stormtrooper holding lightsaber and dancing.
(b) Avatar-scene interaction. Woody sitting on a chair and applauding.
Avatar-avatar interaction. Woody and Stormtrooper playing basketball.
Figure 14: Avatar animations with diverse interactions.
Stable Diffusion
depth
CoutrolNet
skeletton
(a) SDS gradients
(b) One-step denoised images
Figure 15: Visualization of the SDS gradients (a) and the corresponding denoised images (b), given are conditioned on additional depth and
the text prompt "superman". The second and third skeleton images, respectively, indicated in the upper left corner of each visualization. It is clear that the skeleton image as adopted by DreamWaltz provides more informative supervision compared to text alone. Skeleton conditioning is also less restrictive than depth conditioning, successfully avoiding the disappearance of superman's cape.
16

---

After Stage
After Stage II
Flynn Rider
Woody
Mobile Suit Gundam
Joker
Mulan
Optimus Prime
Figure 16: Visualization of canonical avatars obtained at different stages. The optimization at Stage slightly changes the shape and appearance of avatars, resulting sharper details (e.g., Woody's hat) but sometimes more simplified geometry (e.g.. Flynn's 's clothes).
B.3 Single-stage Training vs. Two-stage Training
In Fig. 17, we present a qualitative comparison between single-stage training (Stage II only) and two-stage training (Stage I + Stage II) approaches using our proposed framework, specifically for an avatar animation (e.g., on dance motion sequence). When applied to different characters, the single-stage strategy may result in problematic body topology and noticeable artifacts. In contrast, the two-stage strategy effectively mitigates these issues, leading to improved visual quality. When employing single-stage strategy with end-to-end training, the model is required to simultaneously learn the generation avatar geometry and appearance, along with animation. This introduces complex optimization dynamics, leading to potential slower optimization and sub-optimal results.
(a) Stage Stage
Only
(b) Stage
(d) Stage Only (c) Stage Stage
Figure 17: Qualitative comparisons of single-stage training (Stage only) with two-stage training (Stage I + Stage II) on dance motion sequence, both based on our proposed framework. For different characters, the single-stage strategy may suffer incorrect body topology and severe artifacts, shown in (b). In contrast, the two-stage strategy can relieve these issues (from (b) to (a)) with better visual quality (from (d) (c)). The one-stage strategy tends be subjected slow optimization speed and sub-optimal results.
17

---

B.4 Effects of Joint Optimization for Scene Composition
Benefiting from DreamWaltz, in scenes with interactions. One approach would be to simply render different animatable avatars together in scene. However, such composition is susceptible to issues such artifacts and unnatural interactions. To further improve the scene quality, we could fine-tune for each specific scene. As shown in Fig. 18, fine-tuning brings noticeable improvements, such enhancements Woody's hat and boots, as well as more realistic "hands bumping" interactions.
create diverse animatable avatars that are prepared to engage
w/o Soene Refinement
Scene Refinement
Text Prompt for Scene Refinement: "Woody in Toy Story playing baskethall with Stormtrooper"
Figure 18: To further enhance the visual quality complex scene generation involving multiple avatars and interactions, scene refinement (i.e. fine-tuning with our proposed 3D-consistent SDS) can be applied to eliminate artifacts. As depicted the frames marked with blue boxes, fine-tuning brings noticeable improvements, such as enhancing the appearance of Woody's 's boots and achieving more realistic "hands bumping" effects.
B.5 Realistic Animation with Pose-dependent Changes
To further demonstrate our animation performance, we provide animation results complex character "Elsa" (with long hair and skirt) using our animation method, in comparison with animating extracted mesh with the commercial application Mixamo. As demonstrated Fig. 19, with our method, Elsa's skirt and hair exhibit significantly more natural displacements and movements (as highlighted in yellow and red, respectively) as pose changes.
Canonical "Elsa"
Animatable "Elsa"
Extracted mesh "Elsa" animated by Mixamo
Figure Animation results of complex avatar (Elsa with long hair and skirt). Our animation method could achieve high-quality realistic animation with pose-dependent changes (hair displacement and skirt movements as highlighted in yellow and red respectively). while animation with rigged mesh fails with hair stuck to left arm and unrealistic skirt.
18

---

C More Implementation Details
Diffuusinn Guidance. We adopt ControlNet [48] with Stable-Diffusion v1.5 [34] as the backbone to provide 2D supervision. Specifically, we utilize the score distillation sampling (SDS) technique introduced by DreamFusion [31] to obtain the back-propagation gradients 3D avatar representation. During training. we randomly sample the timestep from a uniform distribution of [20, 980], and the classifier- guidance scale is set to 50.0. The weight term w(() of SDS loss set to 1.0, and we normalize the SDS gradients to stabilize the optimization process. The conditioning scale for ControlNet is set to 1.0 by default.
The proposed Dream Waltz utilizes two types of conditioning: text prompt and skeleton image. The text prompt given by the user to provide avatar description. View-dependent text augmentation from DreamFusion [31] is also used:
"front view of..." camm [0 |00
"backside view of..." Geam [180 270"]
"side view of..."
otherwise,
where camm denotes the azimuthal angle of camera position. The skeleton image is exported from the 3D SMPL mesh, where the rendering view is required to be consistent with the rendering view of NeRF for training.
NeRF Rendering. We adopt Instant-NGP [25] as the implicit avatar representation. The ray marching accelcration based on occupancy grid is disabled for dynamic scene rendering. The 3D avatar representation renders "latent images" in the latent space of 66646444 following Latent- NeRF [21], where the "latent images" can be decoded into RGB images of 1512x512x3 by the VAE decoder of Stable Diffusion [34]. During training, the camera positions are randomly sampled in spherical coordinates, where the radius, azimuthal angle, and polar angle of camera position are sampled from [1.0, 2.0), [0, 360] and [60, 120), respectively.
Optimization. Throughout the entire training process, rate of le-3, and batch size is set to 1. For the canonical avatar creation stage, we train the avatar representation for 30,000 iterations, which takes about hour single NVIDIA 3090 GPU. For the animatable avatar learning stage, the avatar representation and the introduced density weighting network are further trained for 50,000 iterations. Inference takes less than 3 seconds per rendering frame. We further fine-tune the hybrid avatar representations for 30,000 more steps for scenarios with multiple avatars and complex interactions.
Adam [16] optimizer with learning
Dataset. To create animation demonstrations, we utilize SMPL-format motion sequences from the 3DPW [42] and AIST+ [17] datasets to animate avatars. SMPL-format motion sequences extracted from in-the-wild videos are also used.
19


Reviews:
[{"review_id": "mD0GrKMXtg", "reviewer": "Reviewer_kXf3", "summary": "This work proposes a method for text-driven human avatar generation. It combines animatable human nerf and diffusion model to implement avatar generation and animation. Extensive experiments demonstrate that its performance outperforms existing works. Also, this work supports avatar-avatar, avatar-object, and avatar-scene interactions.", "questions": "How to implement avatar-object and avatar-scene interaction? Please introduce more details for these two applications.", "limitations": "Not addressed. There is no discussion of broader societal impacts.", "rating": 4, "confidence": 5, "soundness": 3, "presentation": 2, "contribution": 3, "strengths": "1. This is among the first diffusion-based works that can generate animatable 3D avatars, which also support the interactions between avatars and scenes/objects.\n2. The occlusion culling method is well-motivated and it can address the multi-face issue effectively together with the carefully selected text prompt for viewpoints.\n3. Experiments show SOTA performance for text-driven avatar generation and animation.", "weaknesses": "1. This work is a little bit overclaimed (line 49-51) because there exist several prior works which can generate animatable 3D avatars with complex shapes and appearances, such as [1,2].\n2. The authors claim that DreamWaltz is able to make a scene with diverse interactions across avatars, objects, and scenes. However, it is difficult for me to evaluate whether this point is technically challenging because there is no specific design or module to enable these interactions in the proposed framework. Please explain more about this point because it is the main contribution of this work as it is mentioned in the title.\n3. The proposed framework is similar to AvatarCraft, the only difference is the SMPL inverse skinning and ControlNet.Please explain more about the differences between DreamWaltz and AvatarCraft.\n4. It is confusing to classify AvatarCraft as not animatable (Table 1) because AvatarCraft can also deform the canonical avatar to different target poses in the observation space. I believe by using SMPL, DreamWaltz can only change the pose parameters to implement avatar animation, which is quite similar to AvatarCraft.\n5. The authors might use the term 'generalizable NeRF' carefully because when we use 'generalizable NeRF', it usually refers to the NeRF model that is not overfitted to one 3D scene but can generalize to any inputs instead. It is highly recommended to think about this term again to avoid confusion or ambiguity because the NeRF model which can be deformed to any target pose is usually named as deformable/animatable instead of generalizable.\n6. I think this work is technically solid and has good performance, but there exists large improvement room for the current manuscript.\n\n[1] EVA3D: Compositional 3D Human Generation from 2D Image Collections.\n[2] Avatargen: a 3d generative model for animatable human avatars.\n", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "The authors are grateful for the detailed and in-depth feedback from the Reviewer. We have substantially revised the manuscript as suggested by the reviewer. Below we address the mentioned concerns separately.\n\n### **Q: Overclaiming in Line 49-51: “for the first time capable of generating avatars with complex shapes and appearance.”**\n\n**A:** We apologize for the misleading statement in Line 49-51: “for the first time capable of generating avatars with complex shapes and appearance, ready for high-quality animation and interaction”. Our work focuses on the challenging text-to-avatar generation using pre-trained vision-language models, different from EVA3D and AvatarGen which targets a specific avatar domain using fashion images. Existing works of AvatarCLIP, AvatarCraft, and DreamAvatar are more comparable. Compared to these works, our work can create both complex (non-SMPL-like) and animatable avatars from text, which is why we claim \"for the first time\". We appreciate the reviewer's comment and will revise the statement in Line 49-51.\n\n### **Q: More explanation about making a scene with diverse interactions.**\n\n**A:** Making a scene with diverse interactions is challenging because it requires a deep understanding of the intricate interplay between body poses, objects, shape, and proximity. These interactions are hard to model by hand, and a recent work [1] later than ours proposes to learn human interaction priors from large image collections. Our work similarly employs image priors (where social interactions are abundant) from a pretrained diffusion model to enable more realistic scene interactions. Specifically, we use the proposed 3D-consistent SDS for composite scene NeRF fine-tuning. The scene-specific SDS gradients from the diffusion model can enhance the visual quality of the scene involving avatars and interactions, for example, make “hands bumping” effects more realistic, as shown in Fig. 5 of the supplementary material.\n\n[1] Generative Proxemics: A Prior for 3D Social Interaction from Images. Arxiv 2023.\n\n### **Q: Differences between DreamWaltz and AvatarCraft.**\n\n**A:** AvatarCraft and our DreamWaltz are both two-stage approaches, where a canonical avatar is first created and then animated, but the details and results are significantly different. \n\nThe biggest disagreement is whether animation is learnable. AvatarCraft only borrows the inverse-LBS from SMPL to animate the implicit field, with a hard-coded mask function to filter out the points far from the template mesh. Also based on inverse-LBS, our DreamWaltz allows the implicit field and mask function to be further learned under the supervision of rich image prior: the controlnet supervision from any pose condition. This improvement is not trivial, as it gets rid of the over-reliance on the SMPL mesh topology, allowing us to animate avatars with more complex (non-SMPL) shapes (as shown in Fig. 7(c) of our paper), which is crucial for imaginative text-driven avatar creation.\n\nBesides animation, other differences lie in canonical avatar creation and scene making. Thanks to our proposed 3D-consistent SDS loss, we could use a more concise pipeline to create canonical avatar without coarse-to-fine and multi-bbox training, achieving faster training speed (only 46.2% training time of AvatarCraft, using one NVIDIA A100 GPU) and comparable visual quality. \n\nWe also explore scene making with interactions across avatars and objects (which is not discussed by AvatarCraft), finding that scene-specific fine-tuning with our proposed 3D consistent SDS can further eliminate artifacts and make interactions more realistic.\n\n### **Q: Why classify AvatarCraft as not animatable in Table 1?**\n\n**A:** We gently remind the reviewer that we classify AvatarCraft as animatable in Table 1, where DreamAvatar is classified as not animatable. Although DreamAvatar allows pose control, the target pose needs to be pre-determined because it requires retraining for each pose control and thus is impractical for animation due to the inefficiency (takes two hours for a novel pose) and inconsistency (the appearance of the avatar in different poses may be changed, as shown in Fig. 1 of the rebuttal pdf). In fact, DreamAvatar doesn't claim to be able to animate avatars, nor does it provide animation results. For the above reasons, we believe DreamAvatar can be safely classified as not animatable.\n\n### **Q: Using SMPL, DreamWaltz is quite similar to AvatarCraft.**\n\n**A:** Yes, we admit that DreamWaltz can change the pose parameters to implement rough avatar animation, which is quite similar to AvatarCraft. However, DreamWaltz can animate complex avatars like “Woody with cowboy hat”, while AvatarCraft can only animate SMPL-like avatars such as “Woody” with a bald head, as shown in Fig. 7(c) of our paper.\n\n### **Q: Concern about the term “generalizable NeRF”.**\n\n**A:** We sincerely appreciate the reviewer's suggestion and will change the term \"generalizable NeRF\" to \"deformable NeRF\" in the revision.\n\n### **Q: There exists large improvement room for the current manuscript.**\n\n**A:** We thank the reviewer for the positive comments on the soundness and performance of our work. The current manuscript will be carefully polished. DreamWaltz solves the incompatibility between complex structure (non-SMPL topology) and animation in the existing text-driven avatar creation works, and explores making scenes with diverse interactions. We sincerely hope that our responses could address the reviewer’s concerns.\n\n### **Q: How to implement avatar-object and avatar-scene interaction?**\n\n**A:** Please refer to Q4 in the global rebuttal for these implementation details.\n\n### **Q: Missing Limitations and Societal Impacts.**\n\n**A:** We have supplemented the discussion of limitations and societal impacts in the global response."}, {"review_id": "ifMqTmo9wS", "reviewer": "Reviewer_1xa2", "summary": "The work proposes a method for generating 3D skeleton animatable characters by distilling a latent diffusion model. It uses Control Net to add additional key point map conditioning to the diffusion process, improving the granularity of pose control. It uses DreamFusion to distill a NeRF model given a text prompt and the skeleton conditioning in a fixed A-pose. The model is then refined by sampling additional poses from the SIMPL model used as a skeleton prior. The model is shown to be able to generate animatable models from diverse text prompts that does not require re-training for new novel poses. ", "questions": "- L22. what is the resolution of the model for rendering at 3s.\n- L228. Does joint training affect generalization performance? A comparison would be useful here. \n- How consistent is quality of results? Results showing reconstructions using different initial noise values would demonstrate diversity of generations. \n- L272. Is inverse LBS just the proposed method without the additional MLP for d'? Clarification would be useful. \n- Eqn (5). The motivation of this formula is not apparent from the text. Why would Sigmoid(d') not work similarly?", "limitations": "Authors have not addressed limitations or societal impact. \nIt is suggested that the authors outline weaknesses mentioned above and suggest remediation strategies as future work.\nSocial impact follows prior works such as DreamFusion.\n", "rating": 5, "confidence": 4, "soundness": 3, "presentation": 2, "contribution": 3, "strengths": "The method proposes a stable training regime, starting with retraining using a SIMPLE mask, followed by single pose and fine-tuning on multiples poses. The addition of depth culling also helps remove artifacts seen in prior methods. ", "weaknesses": "- The work's main claim of animatability needs further evidence. The results show animation only for non-complex characters; no characters with long skirts or hair are animated, only reconstructed. For the samples that are shown, the animation quality is not a big departure from LBS deformation. Furthermore, there is a lack of diversity in aspects of pose that are not included in the skeleton model, for example hands are blurring and biased to a specific pose. \n- The ability of the method to capture interactions appears limited to blending in 3D, where complex interactions between parts are not modeled, presumably due to the independent training. For example, the hands are not clasped in the Waltz Fig. 3\n- The generated models exhibit unnatural body proportions due to the mismatch between the skeleton conditioning and the text conditioning. For example, Michael Jordan (a basketball player) and Lionel Messi (a football player) seem to have the same body proportions in Fig 1. of the Supplementary. \n", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "The authors are grateful for the reviewer's valuable feedback and insightful questions. We are encouraged by your support for this work! Below we address the concerns separately.\n\n### **Q: Further evidence of animatability is needed. No animation results of complex characters (with long skirts or hair) are shown.**\n\n**A:** We show an animation result of Mulan (long hair) in Fig. 2 of the supplementary material. We further provide more animation results in Fig. 5 of the rebuttal pdf, animating Elsa with a long skirt and hair. As shown in Fig. 7(c) of our paper, our learnable animation method is significantly more effective than baseline methods Inverse-LBS and AvatarCraft. Intuitively, it makes sense that our animation method learned from diffusion image priors is better than the baseline methods that only rely on nearest neighbor vertex query and SMPL’s inverse-LBS, because SMPL only describes the naked human body and is not suitable for complex shapes.\n\n### **Q: Lack of pose diversity in the skeleton model.**\n\n**A:** It is true that the pose of the skeleton model lacks diversity and may cause blurring and artifacts. But by upgrading the skeleton model to SMPLX and ControlNet to v1.1, these shortcomings can be alleviated.\n\n### **Q: The ability of the method to capture interactions appears limited to blending in 3D.**\n\n**A:** In addition to independent training and blending in 3D, we also explore scene-specific fine-tuning, which puts multiple avatars in the same scene for joint training with the proposed 3D-consistent SDS loss (where image priors would “correct” unrealistic interactions), demonstrating improvements in visual quality as shown in Fig. 5 of the supplementary material pdf.\n\n### **Q: Unnatural body proportions.**\n\n**A:** The body proportions can be freely controlled via specifying SMPL model parameters. We provide results of using different SMPL shape parameters to control body proportions in Fig. 3 of the rebuttal pdf.  \n\n### **Q: What is the resolution of the model for rendering at 3s.**\n\n**A:** For animation rendering, our 3D avatar representation renders 128x128 latents. Then, the latents are decoded by the VAE decoder of Stable-Diffusion to obtain 1024x1024 rgb images. The whole rendering and decoding process takes less than 3 seconds. The speed bottleneck is mainly in nearest neighbor vertex queries of ray sampling points for inverse-LBS. We use CPUs for these computations due to the huge memory requirement. Improving rendering speed remains for future work where we could consider writing faster GPU operators and more efficient nearest neighbor querying.\n\n### **Q: Does joint training affect generalization performance?**\n\n**A:** In fact, we provide a comparison of two-stage training (Stage I + II) and joint training (Stage II only) in Fig. 7 of the supplementary material pdf. The results show that joint training does hurt generalization performance, leading to more artifacts and “phantom limbs”. More discussions are given in sec 2.4 of the supplementary material.\n\n### **Q: How consistent is the quality of results when using different initial noise values?**\n\n**A:** We provide these results in Fig. 4 of the rebuttal pdf. The quality of avatars with different initial noises are consistently good.\n\n### **Q: Is inverse LBS in L272 just the proposed method without the additional MLP for d'?**\n\n**A:** Inverse-LBS in L272 is the proposed method without additional MLP for d’ and NeRF fine-tuning in Stage II. We will add this clarification in the revision.\n\n### **Q: The motivation of Eqn (5) is missing. Why would Sigmoid(d') not work similarly?**\n\n**A:** Sorry for the missing motivation of Eqn (5). We derive this formula from the mask function ${\\eta(\\boldsymbol{p})}$ in the AvatarCraft paper. This function aims to set the density of the points far from the target mesh surface to zero, formulated as:\n\n$$\n\\eta(\\boldsymbol{p})=\n\\begin{cases}\n0, \\text{if } d(\\boldsymbol{p}) > \\delta,\\\\\\\\\n1, \\text{if } d(\\boldsymbol{p}) \\le \\delta,\n\\end{cases}\n$$\n\nwhere $d(\\cdot)$ is the distance function between the ray sampling point $\\boldsymbol{p}$ and its nearest neighbor vertex, $\\delta$ is a constant threshold. Such a constant threshold is not suitable for avatars with complex shapes, so our work use the MLP-predicted value $d’$ (i.e., Eqn (4) in our paper) to replace the threshold $\\delta$, resulting in:\n\n$$\nw_d:=\\eta(\\boldsymbol{p})=\n\\begin{cases}\n0, \\text{if } d(\\boldsymbol{p}) - d' > 0,\\\\\\\\\n1, \\text{if } d(\\boldsymbol{p}) - d' \\le 0.\n\\end{cases}\n$$\n\nFinally, we introduce a sigmoid function with preset parameter $a$ to make it smooth and differentiable, obtaining Eqn (5):\n\n$$ w_d = \\text{Sigmoid}(-(d-d')/a). $$\n\nAn implementation of $ \\text{Sigmoid}(d') $ might also work similarly but lacks intuition.\n\n### **Q: Missing Limitations and Societal Impacts.**\n\n**A:** We thank the reviewer for the valuable suggestions. We have provided the discussion of limitations and societal impacts in the global response."}, {"review_id": "TsITQt38uh", "reviewer": "Reviewer_DdUA", "summary": "This paper proposes a new method for text-to-3D avatar generation. The proposed pipeline has two stages. The first stage generates a static avatar while the second stage learns the deformation properties of the avatar for animation. The authors propose 3D-consistent occlusion-aware score distillation sampling which seems to improve the generation quality over previous methods with standard score distillation sampling. The paper further includes the results of animation and interaction of the generated avatars.", "questions": "1. Why is DreamAvatar not regarded as non-animatable?\n\n2. How does the learned pose-dependent shape and texture change look in the canonical space?\n\n3. What causes the disappearing leg artifacts in the video?", "limitations": "There is no limitation discussed in the paper.", "rating": 4, "confidence": 4, "soundness": 2, "presentation": 3, "contribution": 2, "strengths": "**Method:** The proposed 3D-consistent Score Distillation Sampling and occlusion culling is reasonable and technically sound. They seem to be effective in the provided ablation study.\n\n**Experiment:** The comparison with Sota is fair and complete to the best of my knowledge. The generated static avatars are in good quality.\n\n**Presentation:**. The paper is well-structured and written. I find it easy to follow and understand.", "weaknesses": "1. I have some doubts regarding the relationship to the existing work in Table 1. DreamAvatar is marked as non-animatable, however, the original DreamAvatar paper does show reposing results in Fig 4 in their paper. This seems to contradict the claim in this paper.  And regarding the interaction between avatars/objects, shouldn't this be possible with all methods as it is basically composited volume rendering if I understand correctly?\n\n3. In L34, the authors mention that realistic animation involves changing texture and shape in different poses. However, in the qualitative results in the videos, I did not see such pose-dependent changes but only articulation with LBS. It would be helpful to visualize the pose-dependent changes in the canonical space to understand if the model indeed learns the pose-dependent effects. \n\n4. In video 00:00-00:01, on the leftmost sequence, Woody’s upper legs disappear when they are crossing. Why do such artifacts happen?\n\nI am on the negative side currently. The main contribution of this paper w.r.t previous works seems to be making 3D text-to-avatars \"animatable\" and \"interactive\" for the first time (L49, table 1). However, regarding \"animatable\", previous work DreamAvatar does demonstrate avatars in new poses, and it's unclear why DreamAvatar is not \"animatable\". Also, the animation quality in this paper is still not satisfactory - there are artifacts (legs disappearing) and no obvious pose-dependent effects. Regarding \"interactive\", it seems that this is achieved simply by rendering two avatars in a composite way, which is straightforward and can be done with all previous methods. I hope the authors to clarify the contributions of this paper w.r.t previous methods.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for the thoughtful feedback and valuable questions! Below we address the questions and concerns separately.\n\n### **Q: Why is DreamAvatar regarded as non-animatable?**\n\n**A:** Although DreamAvatar can repose the avatar via **retraining**, it is impractical for animation due to the inefficiency (it takes two hours for a novel pose) and inconsistency (the appearance of the avatar in different poses may be changed, as shown in  Fig. 1 of the rebuttal pdf). In fact, DreamAvatar doesn't claim to be able to animate avatars, nor does it provide animation results. For the above reasons, we believe DreamAvatar can be safely marked as not animatable.\n\n### **Q: Interaction between avatars/objects should be possible with all methods as it is basically composited volume rendering.**\n\n**A:** Regarding the interaction between avatars/objects, yes, naive interactions should be possible for all methods via composited volume rendering, and we will modify some ambiguous expressions such as Table 1 and Line 262. Different from existing works, we explore the 2D image prior (where social interactions are abundant) from diffusion model for improving scene interaction. Specifically, we find that scene-specific fine-tuning with our proposed 3D consistent SDS can further eliminate artifacts and make interactions more realistic, as shown in Fig. 5 in the supplementary material.\n\n### **Q: How does the learned pose-dependent shape and texture change look in the canonical space?**\n\n**A:** We provide a visualization of pose-dependent changes in Fig. 5 of the rebuttal pdf, where character Elsa's skirt and hair are changed with pose changes. Unfortunately, these changes cannot be displayed in canonical space because our animation operations are irreversible.\n\n### **Q: What causes the disappearing leg artifacts in the video?**\n\n**A:** The disappearing leg artifacts are caused by wrong predictions of the density weighting network, which makes the masking area around the legs part too large. Due to the instability of SDS optimization, the density weighting network is difficult to converge to an optimal state for the Woody case. But this situation is not common, most characters' upper legs don't disappear when they are crossing: e.g. Fig. 1 (a) of the main paper, Fig. 2 of the supplementary material. \n\n### **Q: Contributions of this paper w.r.t previous methods.**\n\n**A:** The main contribution of this paper is making text-driven 3D avatars “complex” and “animatable” for the first time. Previous text-to-avatar works AvatarCLIP and AvatarCraft are animatable, but the avatars generated by them are required to be highly similar to SMPL templates and oversimplified in shape. DreamAvatar is not animatable because it requires retraining for each pose control. For animation, despite rare artifacts, our method can animate various complex avatars without manual rigging and pose-dependent effects are provided in rebuttal pdf Fig. 5. For making scenes with interactions, our work not only renders two avatars in a composite way, but also proposes a scene-specific fine-tuning with our proposed 3D-consistent SDS.\n\n### **Q: Missing Limitations.**\n\n**A:** We have supplemented the discussion of limitations in global response."}, {"review_id": "dbgYZ82jJo", "reviewer": "Reviewer_rngx", "summary": "The paper presents an approach for creating animatable avatars from text prompts. It builds on DreamFusion and makes it articulated by incorporating articulated NeRF and SMPL body model. It also replaces the vanilla text-to-image model (StableDiffusion) with ControlNet to introduce 3D consistent SDS loss. The performance is evaluated using a user study where the proposed method is shown to outperform existing methods. ", "questions": "### Implementation Details:\n- Would be nice to provide further implementation details. What is the batch size, learning rate, guidance value, etc.?\n- How is the background learned during training, what are light settings, etc? \n\n### Ablation Studies:\n- What is the impact of the density-weighted network? Existing methods for neural avatars (AniSDF, HumanNeRF, etc.) use different variants to accommodate non-rigid deformations. How does the performance compare with those? \n- What happens if random body poses from humans prior are not used during training? Would the DWN module still work? Since it operates in the canonical space, why random body poses are required in the first place? ", "limitations": "The paper does not discuss the limitations of the proposed approach. Overall, the visual quality of the generated avatars is quite limited and there are severe artifacts in the animations (e.g., on the arms of Woody in support). The time required to generate an avatar is also a limitation of the proposed method and should be discussed. The qualitative results are mostly provided for the cartoonish characters which are well known. It would have been nice to see more creative avatars e.g., a doctor with a Woody's Hat, etc. ", "rating": 4, "confidence": 5, "soundness": 3, "presentation": 3, "contribution": 2, "strengths": "- The paper addresses the challenging problem of creating neural avatars from text prompts. \n- The paper demonstrates that DreamFusion can be extended to articulated humans. \n- The proposed method is intuitive and makes sense, though it is not entirely novel. ", "weaknesses": "### Novelty \n- The main limitation of the paper is the lack of novelty. The paper builds on DreamFusion and then adopts existing methods for articulated NeRF. It replaces the vanilla stableDiffussion model with ControlNet for 3D consistent SDS loss. Overall, the proposed approach is a straightforward combination of existing methods and it is hard for me to identify any novel technical contribution of the paper.  \n\n### Baselines \n- A simple baseline is missing. Similar to AvatarClip, we can extract a static mesh using MarchingCube and then rig it with SMPL. This will make it animatable. Since the paper's main contribution is animatable avatars, I believe having this baseline is important to properly validate the contributions of the paper. \n\n### Visual quality \n- The quality of the generated avatars is also limited even though they are all well-known characters. \n\n", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for the detailed feedback and questions! Below we address the questions and concerns separately.\n\n### **Q: The paper is a straightforward combination of existing methods and lacks novelty.**\n\n**A:** Our work addresses two important problems in text-dirven avatar generation:\n\n**i)** SDS-based methods struggle to provide view-consistent supervision for avatar creation. Our work proposes 3D-consistent SDS with occlusion culling by utilizing SMPL and ControlNet, effectively increasing generation quality and avoiding multiple faces.\n\n**ii)** Images contain rich information of human poses and interactions. We distill this information from diffusion model to avatar through arbitrary pose conditioned SDS, reducing the dependence on SMPL and allowing avatars with complex shapes to be animated.\n\nBesides, compared to existing text-to-avatar works AvatarCLIP, AvatarCraft, and DreamAvatar, our work achieves the best visual quality and for the first time allows animation of complex avatars.\n\n### **Q: A mesh-based animation baseline is missing.**\n\n**A:** The solution based on mesh extraction and SMPL animation have two disadvantages.\n\n**i)** There is a serious loss of fidelity in the conversion from the implicit field to mesh.\n\n**ii)** SMPL only describes the naked human body and is not suitable for animating complex shapes, which is why the avatars created by AvatarCLIP are all SMPL-like shapes.\n\nFor comparison, we further provide animated results by mesh-based method and ours in Fig. 5 of the rebuttal pdf. Even with the professional tool Mixamo (requires manual rigging), the mesh-based animated results are still inferior to ours.\n\n### **Q: The quality of the generated avatars is limited.**\n\n**A:** Our results already outperform existing text-driven sota works with much faster generation speed (46% and 50% training time of AvatarCraft and DreamAvatar). Both reviewer kXf3 and DdUA gave positive comments on our experimental results, such as: “Experiments show SOTA performance for text-driven avatar generation and animation,” “The comparison with Sota is fair and complete to the best of my knowledge. The generated static avatars are in good quality.” The quality could be further improved via mesh refinement or zoom-in training but won’t be our contribution.\n\n### **Q: Would be nice to provide further implementation details.**\n\n**A:** We gently remind the reviewer that these implementation details are already provided in Sec. 3 of the supplementary material. Throughout the entire training process, we set the batch size to 1, set the classifier-free guidance scale to 50, and use the AdamW optimizer with a learning rate of 1e-3.\n\n### **Q: How is the background learned during training, what are light settings?**\n\n**A:** We use a two-layer MLP network as background NeRF to model the background which is common in DreamFusion implementations. We encode the ray direction using frequency encoding and utilize it as an input to the background NeRF, which predicts background color solely based on the ray direction. During training, the background NeRF is also optimized by SDS gradients, with the learning rate of 1e-3.\n\nThe lighting modeling remains as future work (also not yet considered in related work of DreamAvatar and AvatarCraft) since our work focuses on text-to-avatar generation.\n\n### **Q: What is the impact of the density-weighted network?**\n\n**A:** The density-weighted network mitigates animation artifacts due to inaccurate warping by setting the density of the points far from the target mesh surface to zero.\n\n### **Q: How is the performance of the density-weighted network compared to different variants for non-rigid deformations?**\n\n**A:** We did try various modules for non-rigid deformations, such as the non-rigid motion module from HumanNeRF, but found that the network failed to learn the correct deformations (severe artifacts appeared). We think the reason is that learning deformations from arbitrary pose-conditioned ControlNet is much more difficult than from videos.\n\n### **Q: Why are random body poses required during training? Would the DWN module still work without random body pose?**\n \n**A:** The DWN module will not work if random body poses are not used during training. The inputs of DWN involve the ray sampling point $p$ and its nearest neighbor vertex $v_c$, while the nearest vertex $v_c$ is related to current body pose. If the body pose is fixed, DWN will degenerate to a constant value, similar to the animation method of AvatarCraft which won’t be able to animate complex avatars.\n\n### **Q: The paper has limitations in visual quality, time cost, and avatar creativity.**\n\n**A:** Although not fully satisfactory, our method outperforms previous text-driven avatar creation works in terms of visual quality, time cost, and avatar creativity.\n\n**a) Visual quality.** For static avatar creation, our results achieve sota visual quality compared to previous works on text-driven avatar creation. For animation, the severe artifacts of the Woody case are actually not common, and we provide more animation results in Fig. 2 and Fig. 5 of the rebuttal pdf.\n\n**b) Time cost.** To generate a canonical avatar, our work takes about one hour on one NVIDIA A100 GPU, significantly outperforming existing text-driven avatar creation works. For comparison, AvatarCLIP takes 5 hours on one NVIDIA A100 GPU, AvatarCraft takes 2.2 hours on one NVIDIA A100 GPU, DreamAvatar takes about 2 hours on one NVIDIA RTX 2080Ti GPU.\n\n**c) Creative characters.** We mainly provide cartoon characters for a fair comparison with previous text-driven avatar creation works. Many photorealistic avatars of celebrities are provided in Fig. 1 of the supplementary material pdf, including “Taylor Swift”, “Albert Einstein”, “Emma Stone”, “Linoel Messi”, etc. Following the reviewer's suggestion, we provide more creative avatars in Fig. 4 of the rebuttal pdf, such as “a doctor with Woody's Hat” and “Taylor Swift in Snow White costume”."}]

Author rebuttal:
We sincerely thank all reviewers and ACs for their time and efforts. Below we provide the responses to some frequently asked questions and main concerns, as well as the discussions of limitations and societal impacts.

### **Q1: Contribution of our work w.r.t previous methods.**
**A:** Previous methods for text-driven avatar generation include AvatarCLIP [1], AvatarCraft [2], and DreamAvatar [3]. Compared to them, we achieve both complex (non-SMPL-like) and animatable avatar generation for the first time. Besides, we explore how to make scenes with diverse interactions, where our framework can be used for scene-specific fine-tuning to further improve visual quality and interaction realism with 2D prior where social interactions are abundant .

### **Q2: Motivation of our animation learning.**
**A:** Text-driven avatar generation is zero-shot and cannot obtain animation information from video data, so existing works such as AvatarCLIP and AvatarCraft rely on SMPL's inverse skinning for animation. However, SMPL only describes naked human bodies and is not suitable for animating avatars with complex appearances. The core idea of our animation method is to distill pose-dependent appearance knowledge from the pretrained diffusion model for deformation learning, which effectively compensates for the deficiency of SMPL-based animation.

### **Q3: Why is DreamAvatar classified as non-animatable?**

**A:** DreamAvatar allows posed-avatar generation, however this requires retraining (takes about 2 hours) for each new target pose and cannot guarantee a consistent appearance (as shown in Fig. 1 of the rebuttal pdf), making DreamAvatar non-animatable.

### **Q4: How to implement avatar-object and avatar-scene interaction?**

**A:** Implementation details of these two interactions are given as follows. We will add these details in the revision.

**Avatar-object interaction.** The whole training process of avatar-object interaction is actually the same as pure avatar generation, except that the text prompt needs to add object descriptions. For example, we change “Lara Croft” to “Lara Croft with weapons” and change “Kobe Bryant” to “Kobe Bryant with basketball”, obtaining the results in Fig. 1(c) of our paper. In fact, our method can further achieve avatar-object animation, as shown in Fig. 2 of the rebuttal pdf.
 
**Avatar-scene interaction.** First, we learn the animatable avatar NeRF representation (e.g., “Woody”) via Stage I + II of DreamWaltz, and train the static scene NeRF representation (e.g., “a chair made of cheese”) via Latent-NeRF. Then, both the avatar and the static scene NeRFs are aligned manually and can be rendered by the same camera. We introduce an extra fine-tuning stage as mentioned in sec 3.2.3, utilizing the proposed 3D-consistent SDS loss to fine-tune the hybrid avatar-scene NeRFs and the introduced density weighting network. Here the SDS loss is scene-specific, since we condition the ControlNet on the scene-specific textual description (e.g., “Woody sitting in a chair made of cheese and applauding”) and skeleton images (e.g., random frames from the “sitting and clapping” motion sequence). It takes 30,000 iterations for the extra fine-tuning stage, as described in L78-L79 in the supplementary material pdf.

### **Limitations**
Although DreamWaltz can generate SOTA high-quality complex avatars from textual descriptions, the visual quality can be significantly improved with higher resolution training at higher time and computation cost. The quality of face and hand texture can be further improved through dedicated optimization of close-up views as well as adopting SMPLX instead of SMPL for ControlNet conditioning. 

The body portions can be freely controlled via specifying SMPL model parameters, for future work we could train a SMPL shape generative model to facilitate the textual control of SMPL body shapes.

Animation relies on distilling 2D prior depicting randomly selected poses into NeRF representations and the introduced NeRF re-weighting module. Extended training durations tend to yield heightened animation quality and fewer artifacts. However, determining the optimal training duration a-priori remains challenging. To address this issue, we intend to investigate metrics conducive to quantifying training convergence.

Following the optimization-based approach akin to DreamFusion, the generation of an avatar necessitates a dedicated optimization procedure for each textual input, consuming approximately an hour. In the interest of expediting the avatar creation process for arbitrary text inputs, we are poised to investigate techniques such as modulation to foster generalization, thereby enabling swifter avatar acquisition.

### **Societal Impacts**
Societal impact follows prior 3D generative works such as DreamFusion [4] and AvatarCraft. 

Given our utilization of Stable Diffusion (SD) as the 2D generative prior, our model could potentially inherit societal biases present within the vast and loosely curated web content harnessed for SD training. We strongly encourage the usage to be transparent, ethical, non-offensive and a conscious avoidance of generating proprietary characters.

It is important to acknowledge that generative models such as ours may have implications for the displacement of creative workers through automation.  Nevertheless, DreamWaltz is intended to be a tool to liberate designers and animators from laborious and repetitive work, thereby to focus on intellectual creativity and to enhance accessibility.

### **Reference**

[1] AvatarCLIP: Zero-Shot Text-Driven Generation and Animation of 3D Avatars. SIGGRAPH 2022.

[2] AvatarCraft: Transforming Text into Neural Human Avatars with Parameterized Shape and Pose Control. ICCV 2023.

[3] DreamAvatar: Text-and-Shape Guided 3D Human Avatar Generation via Diffusion Models. Arxiv 2023.

[4] DreamFusion: Text-to-3D using 2D Diffusion. ICLR 2023.

Meta-review: This paper proposes a method for creating 3D human avatars from text prompts using guidance from a pre-trained diffusion model. The main claimed contributions of this work are (a) improvements in visual quality (b) ability to create posed characters and (c) the ability to create characters with objects and in scenes.

The reviewers' scores hover around borderline (3BR, 1BA).

The first main legitimate concern that multiple reviewers raised was the fact that the claimed contribution around being able to generate scenes with multiple interacting characters or objects and characters interacting with each other, the technical details of this part are not well-described in the paper. The authors described it in much detail in their response to the reviewers. However, the approach that they used to create these scenes is simplistic -- render two characters/an object and a character separately first with their individual text prompts, rigidly align them and then fine tune their Nerfs together with a text prompt. For characters holding objects they used text prompts of the type "[X] holding [Y]". So, in essence, there is nothing unique about the proposed method that enables this capability. Hence, the AC does not regard this a compelling contribution of this work and strongly recommends that the authors to significantly tone down this claimed contribution in the final manuscript and simply show it as an additional application/capability in the results section.

The second major concern of the reviewers' is around comparisons to the works AvatarCraft [2], and DreamAvatar [3], both in terms of conceptual novelty and the quality of the results. As per the NeurIPS guidelines, the authors are not required to compare to the concurrent works of AvatarCraft [2], and DreamAvatar [3]. Hence, the AC and SAC disregarded them while making the final decision. 

Without AvatarCraft [2], and DreamAvatar [3], in the picture, the proposed work shows a clears advancement over the state-of-the-art prior methods in terms of quality, functionality (models articulation) without requiring optimization for each pose.

Hence, the final decision is to accept this work. Congratulations!

The authors are strongly advised to make changes to the original manuscript that they've promised to the reviewers, add clear details of how "scenes" with multiple avatar and/or objects are created and tone down this claimed contribution.
Decision: Accept (poster)