SlowGuess/NeurIPS_2023_OCR huggingface dataset:
Although not our main contribution, we explore the refinement of compositional scene representation with the proposed 3D-consistent SDS. Firstly, we employ DreamWaltz and Latent-NeRF [21] to generate avatars and objects of the desired scene, respectively. Then, we manually set the motions of avatars and objects, and adopt composite rendering [13] to obtain a scene image. For scene refinement, we further provide a text prompt of the desired scene for ControlNet, and use animatable avatar learning of DreamWaltz to fine-tune the whole scene representation. Benefiting from diffusion model's scene-related image priors, the visual quality of the generated dynamic 3D scene could be further improved. More results and discussions are provided in Appendix B.4.
We validate the effectiveness of our proposed framework for avatar generation and animation. In Sec. 4.1, we evaluate avatar generation with extensive text prompts for both qualitative comparisons and user studies. In Sec. 4.2, we demonstrate avatar animation given novel motion sequences. We present ablation analysis in Sec. 4.3 and illustrate that our framework can be further applied to make complex scenes with diverse interactions in Sec. 4.4.
Implementation details. DreamWaltz is implemented in PyTorch and can be trained and evaluated on a single NVIDIA 3090 GPU. For the canonical avatar creation stage, we train the avatar representation for 30,000 iterations, which takes about an hour. For the animatable avatar learning stage, the avatar representation and the introduced density weighting network are further trained for 50,000 iterations. Inference takes less than 3 seconds per rendering frame. Note that the two stages can be combined for joint training, but we decouple them for training efficiency.
4.1 Evaluation of Canonical Avatars
High-quality avatar generation. We show 3D avatars created by DreamWaltz on diverse text prompts in Fig. 4. The geometry and texture quality are consistently high across different viewpoints for different text prompts. See Appendix A.1 for more discussions and results.
Comparison with SOTA methods. We compare with existing SDS-based methods for complex (non-skin-tight) avatar generation. Latent-NeRF [21] and SJC [43] are general text-to-3D models. AvatarCraft [13] is not included for comparison because it cannot generate complex avatars. Furthermore, AvatarCraft utilizes a coarse-to-fine and body-face separated training to improve generation quality, which is orthogonal to our method. DreamAvatar [3] is the most relevant to our method, utilizing SMPL human prior without overly constraining avatar complexity. It is evident from Fig. 5 that our method consistently achieves higher quality in geometry and texture.
Figure 5: Qualitative comparisons for complex avatar generation. Text inputs are listed below.
User studies. We conduct user studies to evaluate the quality of avatar generation against existing text-to-3D generation methods: Latent-NeRF [21], SJC [43] and DreamAvatar [3]. We use the 25 text prompts DreamAvatar released and their showcase models for comparison. We asked 12 volunteers to score 1 (worst) to 4 (best) in terms of (1) avatar geometry and (2) avatar texture. We do not measure text alignment as the avatar prompts are well-known characters universally respected by all the competing models. As shown in Fig. 6, the raters favor avatars generated by DreamWaltz for both better geometry and texture quality. Specifically, DreamWaltz outperforms the best competitor DreamAvatar by score 0.67 on geometry and by score 1.01 on texture.
Figure 6: User preference study. DreamWaltz obtains higher scores in both geometry and texture.
4.2 Evaluation of Animatable Avatars
We demonstrate the efficacy of our animation learning method with animation results on two motion sequences as shown in Fig. 7: row (a) displays motion sequences in skeletons as animation inputs; our framework directly applies the motion sequences to our generated avatars "Flynn Rider" and "Woody" without any retraining. We render the corresponding normal and RGB sequences in (b) and (c). In (d), we provide free-viewpoint rendering of a chosen sequence frame. Since we essentially infer a posed 3D avatar for each motion frame, the renderings are by construction 3D-consistent and we can creatively use the 3D model for novel-view video generation.
Flynn Rider
Woody in Toy Story
Figure 7: Animation results by applying skeleton motion sequences to animatable avatars generated by DreamWaltz. For each pose frame we could effortlessly render 3D-consistent free-viewpoints.
To evaluate the design choices of DreamWaltz, we ablate on the effectiveness of our proposed occlusion culling and animation learning.
Effectiveness of occlusion culling. Occlusion culling is crucial for resolving view ambiguity, both for 2D and 3D generation, as shown in Fig. 8 (a) and Fig. 8 (b), respectively. Limited by the view-aware capability, ControlNet fails to generate the back-view image of a character even with view-dependent text and skeleton prompts, as shown in Fig. 8 (a). The introduction of OC eliminates the ambiguity of skeleton conditions and helps ControlNet to generate correct views. Similar effects can be observed in text-to-3D generation, as shown in Fig. 8 (b).
(a) ControlNet: "backview of Mulan"
(b) Text-to-3D: "Miku"
Figure 8: Ablation study on occlusion culling (OC). Occlusion culling refines the skeleton condition image by removing occluded human keypoints, which helps (a) ControlNet [48] to correctly generate character back view, (b) text-to-3D model to resolve the multi-face problem.
Iiku"
Effectiveness of animation learning. We compare our animation learning strategy with other animation methods, including: vanilla Inverse-LBS (Baseline) and AvatarCraft [13]. From Fig. 9, it is evident that our animation learning method is significantly more effective than Inverse-LBS and AvatarCraft [13]. Note that AvatarCraft cannot faithfully animate complex features, e.g. Woody's hat.
Inverse-LBS
Figure 9: Comparison of different animation strategies, including: Inverse-LBS (Baseline), Avatar-Craft [13], and DreamWaltz (Ours). Our method learns to animate avatars from pose-conditioned image priors, therefore able to animate complex characters with significantly higher quality.
DreamWaltz (Ours)
4.4 Further Analysis and Application
Shape control via SMPL parameter  $\beta$ . DreamWaltz allows shape adjustment for controllable 3D avatar generation. As shown in Fig. 10, the body proportions of the generated avatars can be changed by simply specifying different SMPL parameter  $\beta$ .
Short "Lionel Messi"
Tall "Michael Jordan"
Figure 10: Our method supports adjusting the body proportions of avatars by adjusting the SMPL parameter  $\beta$ . For example, athlete avatars of different heights and fatness can be generated.
Tall "Lionel Messi"
Fat "Michael Jordan"
Creative and diverse avatar generation. DreamWaltz achieves stable 3D avatar generation via 3D-consistent SDS, enabling creative and diverse 3D avatar creation with low failure rates. Given imaginative text prompts and different random seeds, our method successfully generates various high-quality avatars, e.g., "a doctor wearing Woody's hat", as shown in Fig. 11.
seed=0
seed=1
seed=2
seed=3
a doctor wearing Woody's hat
Figure 11: Creative avatar creation with different random seeds and initial noises.
seed=0
seed=1
seed=2
seed=3
Taylor Swift in Snow White costume
Scene composition with diverse interactions. Benefiting from DreamWaltz, we could compose animatable avatars and other 3D assets into the same scene. We give a few examples to highlight the potential for enabling diverse interactions: (1) avatar-avatar interaction as shown in Fig. 1 (c) depicting "Woody" dancing with "Stromtrooper"; and (2) avatar-scene interaction as shown in Fig. 1 (d) with "Woody" sitting on different chairs. The interactions can be freely composed to make a dynamic scene. More results are provided in Appendix A.3.
Limitation. Although DreamWaltz can generate SOTA high-quality complex avatars from textual descriptions, the visual quality can be significantly improved with higher resolution training at higher time and computation cost. The quality of face and hand texture can be further improved through dedicated optimization of close-up views as well as adopting stronger SMPLX instead of SMPL.
Societal Impact. Given our utilization of Stable Diffusion (SD) as the 2D generative prior, our model could potentially inherit societal biases present within the vast and loosely curated web content harnessed for SD training. We strongly encourage the usage to be transparent, ethical, non-offensive and a conscious avoidance of generating proprietary characters.
Conclusion. We propose DreamWaltz, a novel learning framework for avatar creation and animation with text and human shape/pose prior. For high-quality 3D avatar creation, we propose to leverage human priors with SMPL-guided initialization, further optimized with 3D-consistent occlusion-aware Score Distillation Sampling conditioned on 3D-aware skeletons. Our method learns an animatable NeRF representation that could retarget the generated avatar to any pose without retraining, enabling realistic animation with arbitrary motion sequences. Extensive experiments show that DreamWaltz is effective and robust with state-of-the-art avatar generation and animation. Benefiting from DreamWaltz, we could unleash our imagination and make diverse scenes with avatars.
In this supplementary material, Sec. A presents more qualitative results of DreamWaltz on avatar creation, animation and interaction. Sec. B gives more analysis on the design choices of our method. Section C provides more comprehensive implementation details.
A More Qualitative Results
We provide more results of our proposed method, including more generated avatars in Sec. A.1, more animated avatar sequences in Sec. A.2, and more demonstrations on diverse interactions in Sec. A.3.
A.1Avatar Creation
We provide more text-to-3D avatar generations in Fig. 12 with a wide range of text prompts including celebrities, popular cartoon/movie characters and text descriptions. Note that DreamWaltz is capable of generating diverse avatars. For instance, we can produce avatars with a human-realistic appearance like "Tiger Woods", avatars wearing intricate clothing such as "Napoleon" and "Marie Antoinette", and avatars tailored to user-provided characteristics like "Blue fairy with wings", among others.
A.2Avatar Animation
We provide more animation results on six characters as shown in Fig. 13. Please refer to the project page at https://dreamwaltz3d.github.io/ for more animation sequences.
A.3 Diverse Interaction
We provide more results of diverse interactions in Fig. 14, including: avatar-object, avatar-scene, and avatar-avatar interactions. Please refer to the videos on project page at https://dreamwaltz3d.github.io/ for the full sequences.
B.1 Visualization of SDS Gradients
We visualize the SDS supervision gradients  $\| \epsilon_{\phi}(\mathbf{z}_t; y, t) - \epsilon \|$  for NeRF renderings in Fig. 15 (a) and the denoised images derived from noise predictions  $\epsilon_{\phi}(\mathbf{z}_t; y, t)$  in Fig. 15 (b). These visualizations are based on the text prompt  $y$  of "superman" and  $t$  of 980, while additional conditioning of depth and skeleton is provided in the second and third row, respectively, in accordance with the current rendering viewpoint of NeRF. It is evident that depth and skeleton images offer more informative optimization gradients compared to text alone. However, depth images heavily rely on the SMPL prior, leading to gradients that conform tightly to the avatar's skin, resulting in the disappearance of superman's cape. On the other hand, skeleton images as adopted by DreamWaltz provide both informative and flexible supervision, accurately capturing the avatar's shape, pose, and intricate details such as the cape.
B.2 Effects of Random-Pose Optimization on Avatar Quality
In Fig. 16, we present visualizations of the avatars obtained at various stages, all depicted in a canonical pose. In Stage I, a static avatar is generated by optimizing its 3D representation with the canonical pose. In Stage II, the system undergoes training on randomly sampled human poses to facilitate animation learning. Although not our primary objective, this design enables the generated avatar to further refine its appearance with different poses. As a result, minor adjustments are observed, typically leading to sharper details. However, in some cases, the geometry may become more simplified, aligning more closely with the SMPL prior.
Figure 12: Text-to-3D avatars generated with DreamWaltz, each displayed for two views.
Figure 13:Avatar animations on six characters.
(b)Avatar-scene interaction. Woody sitting on a chair and applauding.
(c) Avatar-avatar interaction. Woody and Stormtrooper playing basketball.
Figure 14:Avatar animations with diverse interactions.
Figure 15: Visualization of the SDS gradients (a) and the corresponding denoised images (b), given the text prompt "superman". The second and third rows are conditioned on additional depth and skeleton images, respectively, as indicated in the upper left corner of each visualization. It is clear that the skeleton image as adopted by DreamWaltz provides more informative supervision compared to text alone. Skeleton conditioning is also less restrictive than depth conditioning, successfully avoiding the disappearance of superman's cape.
After Stage I
After Stage II
Figure 16: Visualization of canonical avatars obtained at different stages. The optimization at Stage II slightly changes the shape and appearance of avatars, resulting in sharper details (e.g., Woody's hat) but sometimes more simplified geometry (e.g., Flynn's clothes).
B.3 Single-stage Training vs. Two-stage Training
In Fig. 17, we present a qualitative comparison between single-stage training (Stage II only) and two-stage training (Stage I + Stage II) approaches using our proposed framework, specifically for an avatar animation (e.g., on a dance motion sequence). When applied to different characters, the single-stage strategy may result in problematic body topology and noticeable artifacts. In contrast, the two-stage strategy effectively mitigates these issues, leading to improved visual quality. When employing a single-stage strategy with end-to-end training, the model is required to simultaneously learn the generation of avatar geometry and appearance, along with animation. This introduces complex optimization dynamics, leading to potential slower optimization and sub-optimal results.
Figure 17: Qualitative comparisons of single-stage training (Stage II only) with two-stage training (Stage I + Stage II) on a dance motion sequence, both based on our proposed framework. For different characters, the single-stage strategy may suffer incorrect body topology and severe artifacts, as shown in (b). In contrast, the two-stage strategy can relieve these issues (from (b) to (a)) with better visual quality (from (d) to (c)). The one-stage strategy tends to be subjected to slow optimization speed and sub-optimal results.
B.4 Effects of Joint Optimization for Scene Composition
Benefiting from DreamWaltz, we can create diverse animatable avatars that are prepared to engage in scenes with interactions. One approach would be to simply render different animatable avatars together in a scene. However, such composition is susceptible to issues such as artifacts and unnatural interactions. To further improve the scene quality, we could fine-tune for each specific scene. As shown in Fig. 18, fine-tuning brings noticeable improvements, such as enhancements to Woody's hat and boots, as well as more realistic "hands bumping" interactions.
B.5 Realistic Animation with Pose-dependent Changes
To further demonstrate our animation performance, we provide animation results of complex character "Elsa" (with long hair and skirt) using our animation method, in comparison with animating extracted mesh with the commercial application Mixamo. As demonstrated in Fig. 19, with our method, Elsa's skirt and hair exhibit significantly more natural displacements and movements (as highlighted in yellow and red, respectively) as pose changes.
Canonical "Elsa"
Animatable "Elsa"
Extracted mesh "Elsa" animated by Mixamo
Diffusion Guidance. We adopt ControlNet [48] with Stable-Diffusion v1.5 [34] as the backbone to provide 2D supervision. Specifically, we utilize the score distillation sampling (SDS) technique introduced by DreamFusion [31] to obtain the back-propagation gradients of 3D avatar representation. During training, we randomly sample the timestep from a uniform distribution of [20, 980], and the classifier-free guidance scale is set to 50.0. The weight term  $w(t)$  of SDS loss is set to 1.0, and we normalize the SDS gradients to stabilize the optimization process. The conditioning scale for ControlNet is set to 1.0 by default.
The proposed DreamWaltz utilizes two types of conditioning: text prompt and skeleton image. The text prompt is given by the user to provide avatar description. View-dependent text augmentation from DreamFusion [31] is also used:
$$
\left\{ \begin{array}{l l} \text {＂ f r o n t v i e w o f ＂} & \theta_ {\mathrm {c a m}} \in [ 0 ^ {\circ}, 9 0 ^ {\circ} ] \\ \text {＂ b a c k s i d e v i e w o f ＂} & \theta_ {\mathrm {c a m}} \in [ 1 8 0 ^ {\circ}, 2 7 0 ^ {\circ} ] \\ \text {＂ s i d e v i e w o f ＂} & \text {o t h e r w i s e ,} \end{array} \right.
$$
where  $\theta_{\mathrm{cam}}$  denotes the azimuthal angle of camera position. The skeleton image is exported from the 3D SMPL mesh, where the rendering view is required to be consistent with the rendering view of NeRF for training.
NeRF Rendering. We adopt Instant-NGP [25] as the implicit avatar representation. The ray marching acceleration based on occupancy grid is disabled for dynamic scene rendering. The 3D avatar representation renders "latent images" in the latent space of  $\mathbb{R}^{64\times 64\times 4}$  following Latent-NeRF [21], where the "latent images" can be decoded into RGB images of  $\mathbb{R}^{512\times 512\times 3}$  by the VAE decoder of Stable Diffusion [34]. During training, the camera positions are randomly sampled in spherical coordinates, where the radius, azimuthal angle, and polar angle of camera position are sampled from [1.0, 2.0], [0, 360] and [60, 120], respectively.
**Optimization.** Throughout the entire training process, we use Adam [16] optimizer with a learning rate of 1e-3, and batch size is set to 1. For the canonical avatar creation stage, we train the avatar representation for 30,000 iterations, which takes about an hour on a single NVIDIA 3090 GPU. For the animatable avatar learning stage, the avatar representation and the introduced density weighting network are further trained for 50,000 iterations. Inference takes less than 3 seconds per rendering frame. We further fine-tune the hybrid avatar representations for 30,000 more steps for scenarios with multiple avatars and complex interactions.
Dataset. To create animation demonstrations, we utilize SMPL-format motion sequences from the 3DPW [42] and AIST++ [17] datasets to animate avatars. SMPL-format motion sequences extracted from in-the-wild videos are also used.