SlowGuess/NeurIPS_2023_OCR huggingface dataset:
Yukun Huang $^{1,2*†}$ , Jianan Wang $^{1*‡}$ , Ailing Zeng $^{1}$ , He Cao $^{1}$ , Xianbiao Qi $^{1}$ , Yukai Shi $^{1}$ , Zheng-Jun Zha $^{2}$ , Lei Zhang $^{1}$
(a) Princess Elsa in different poses.
(b) Uzumaki Naruto in dance moves.
(c) Woody dances with Stormtrooper.
(d) Woody sits on: (Left) a chair; (Right) a chair made of cheese.
Figure 1: DreamWaltz is a text-to-3D-avatar generation framework, which can (a, b) create complex 3D animatable avatars from texts, (c, d) ready for 3D scene composition with diverse interactions.
We present DreamWaltz, a novel framework for generating and animating complex 3D avatars given text guidance and parametric human body prior. While recent methods have shown encouraging results for text-to-3D generation of common objects, creating high-quality and animatable 3D avatars remains challenging. To create high-quality 3D avatars, DreamWaltz proposes 3D-consistent occlusion-aware Score Distillation Sampling (SDS) to optimize implicit neural representations with canonical poses. It provides view-aligned supervision via 3D-aware skeleton conditioning which enables complex avatar generation without artifacts and multiple faces. For animation, our method learns an animatable 3D avatar representation from abundant image priors of diffusion model conditioned on various poses, which could animate complex non-rigged avatars given arbitrary poses without retraining. Extensive evaluations demonstrate that DreamWaltz is an effective and robust approach for creating 3D avatars that can take on complex shapes and appearances as well as novel poses for animation. The proposed framework further enables the creation of complex scenes with diverse compositions, including avatar-avatar, avatar-object and avatar-scene interactions. See https://dreamwaltz3d.github.io/ for more vivid 3D avatar and animation results.
The creation and animation of 3D digital avatars are essential for various applications, including film and cartoon production, video game design, and immersive media such as AR and VR. However,
*Equal contribution.
$^{\dagger}$ Work done during an internship at IDEA.
$\ddagger$  Corresponding author.
traditional techniques for constructing such intricate 3D models are costly and time-consuming, requiring thousands of hours from skilled artists with extensive aesthetics and 3D modeling knowledge. In this work, we seek a solution for 3D avatar generation that satisfies the following desiderata: (1) easily controllable over avatar properties through textual descriptions; (2) capable of producing high-quality and diverse 3D avatars with complex shapes and appearances; (3) the generated avatars should be ready for animation and scene composition with diverse interactions.
The advancement of deep learning methods has enabled promising methods which can reconstruct 3D human models from monocular images [36, 45] or videos [44, 14, 46, 41, 12, 30]. Nonetheless, these methods rely heavily on the strong visual priors from image/video and human body geometry, making them unsuitable for generating creative avatars that can take on complex and imaginative shapes or appearances. Recently, integrating 2D generative models into 3D modeling [31, 18, 10] has gained significant attention to make 3D digitization more accessible, reducing the dependency on extensive 3D datasets. However, creating animatable 3D avatars remains challenging: Firstly, avatars often require intricate and complex details for their appearance (e.g., loose cloth, diverse hair, and different accessories); secondly, avatars have articulated structures where each body part is able to assume various poses in a coordinated and constrained way; and thirdly, avatar changes shape and texture details such as creases when assuming different poses, making animation for complex avatars extremely challenging. As a result, while DreamFusion [31] and subsequent methods [18, 5] have demonstrated impressive results on text-guided creation of stationary everyday objects, they lack the proper constraints to enforce consistent 3D avatar structures and appearances, which presents significant challenges to producing intricate shapes, appearances and poses for 3D avatars, let alone for animation.
In this paper, we present DreamWaltz, a framework for generating high-quality 3D digital avatars from text prompts utilizing human body prior of shapes and poses, ready for animation and composition with diverse avatar-avatar, avatar-object and avatar-scene interactions. DreamWaltz employs a trainable Neural Radiance Field (NeRF) as the 3D avatar representation, a pre-trained text-and-skeleton-conditional diffusion model [48] for shape and appearance supervision, and SMPL models [2] for extracting 3D-aware posed-skeletons. Our method enables high-quality avatar generation with 3D-consistent SDS, which resolves the view disparity between the diffusion model's supervision and NeRF's rendering. By training an animatable NeRF with diffusion supervision conditioned on human pose prior, we can deform the generated non-rigged avatar to arbitrary poses for realistic test-time animation without retraining.
The key contributions of DreamWaltz lie in four main aspects:
- We propose a novel text-to-avatar generation framework named DreamWaltz, which is capable of creating animatable 3D avatars with complex shapes and appearances.
- For avatar creation, we propose a SMPL-guided 3D-consistent Score Distillation Sampling strategy with occlusion culling, enabling the generation of high-quality avatars, e.g., avoiding the Janus (multi-face) problem and limb ambiguity.
- We propose to learn an animatable NeRF representation from diffusion model and human pose prior, which enables the animation of complex avatars. Once trained, we can animate the created avatar with any pose sequence without retraining.
- Experiments show that DreamWaltz is effective in creating high-quality and animatable avatars, ready for scene composition with diverse interactions across avatars and objects.
Text-guided image generation. Recently, there have been significant advancements in text-to-image models such as GLIDE [26], unCLIP [33], Imagen [35], and Stable Diffusion [34], which enable the generation of highly realistic and imaginative images based on text prompts. These generative capabilities have been made possible by advancements in modeling, such as diffusion models [6, 39, 27], and the availability of large-scale web data containing billions of image-text pairs [37, 38, 4]. These datasets encompass a wide range of general objects, with significant variations in color, texture, and camera viewpoints, providing pre-trained models with a comprehensive understanding of general objects and enabling the synthesis of high-quality and diverse objects.
AvatarCLIP
ComplexX Animatable
AvatarCraft
DreamAvatar
Complex√ AnimatableX
Figure 2: Comparison of text-driven 3D avatar generation methods, including: AvatarCLIP [8], AvatarCraft [13], DreamAvatar [3] and DreamWaltz (Ours). AvatarCLIP and AvatarCraft assume strong SMPL constraints, which makes it straightforward for the generated avatars to align with SMPL for animation. But due to the constraints, the avatars cannot take on complex shapes and appearances; With weak SMPL constraints, DreamAvatar struggles with wrong avatar geometry and requires retraining for each pose adjustment. Different from existing methods, DreamWaltz enables complex and animatable 3D avatar generation benefiting from the proposed SMPL-guided 3D-consistent SDS and deformation learning from human pose prior.
DreamWaltz (Ours)
Complex√ Animatable√
Furthermore, recent works [48, 9, 15] have explored incorporating additional conditioning, such as depth maps and human skeleton poses, to generate images with more precise control.
Text-guided 3D generation. Dream Fields [11] and CLIPmesh [23] were groundbreaking in their utilization of CLIP [32] to optimize an underlying 3D representation, aligning its 2D renderings with user-specified text prompts, without necessitating costly 3D training data. However, this approach tends to result in less realistic 3D models since CLIP only provides discriminative supervision for high-level semantics. In contrast, recent works have demonstrated remarkable text-to-3D generation results by employing powerful text-to-image diffusion models as a robust 2D prior for optimizing a trainable NeRF with Score Distillation Sampling (SDS) [31, 18, 5, 10]. Nonetheless, the produced geometry and texture are static, blurry and usually lack intricate details necessary for avatar creation.
Text-guided 3D avatar generation. Avatar-CLIP [8] starts with initializing 3D human geometry via a shape VAE network and subsequently employs CLIP [32] for shape sculpting and texture generation. To animate the generated 3D avatar, they propose a CLIP-guided reference-based motion synthesis method. However, this approach tends to produce less realistic and oversimplified 3D models due to the limited guidance provided by CLIP, which primarily focuses on high-level semantic discrimination. Concurrent to our work, DreamAvatar [3] and AvatarCraft [13] both utilize pre-trained text-to-image diffusion models and SMPL models as shape prior for avatar generation. While DreamAvatar focuses on producing static posed-3D avatars which are incapable of animation, AvatarCraft generates higher-quality avatars via coarse-to-fine and multi-box training and enables animation via local transformation between the template mesh and the target mesh based on SMPL models. We summarize the key differences between our work and related works in Fig. 2.
Text-to-3D generation. Recent methods [31, 18, 21] have shown encouraging results on text-to-3D generation of common objects by integrating three essential components:
(1) Neural Radiance Fields (NeRF) [25, 1, 22] is commonly adopted as the 3D representation for text-to-3D generation [40, 18], parameterized by a trainable MLP. For rendering, a batch of rays  $\mathbf{r}(k) = \mathbf{o} + k\mathbf{d}$  are sampled based on the camera position  $\mathbf{o}$  and direction  $\mathbf{d}$  on a per-pixel basis. The MLP takes  $\mathbf{r}(k)$  as input and predicts density  $\tau$  and color  $c$ . The volume rendering integral is then approximated using numerical quadrature to yield the final color of the rendered pixel:
$$
\hat {C} _ {c} (\mathbf {r}) = \sum_ {i = 1} ^ {N _ {c}} \Omega_ {i} \cdot (1 - \exp (- \tau_ {i} \delta_ {i})) c _ {i},
$$
where  $N_{c}$  is the number of sampled points on a ray,  $\Omega_{i} = \exp (-\sum_{j = 1}^{i - 1}\tau_{j}\delta_{j})$  is the accumulated transmittance, and  $\delta_{i}$  is the distance between adjacent sample points.
(2) Diffusion models [7, 27] which have been pre-trained on extensive image-text datasets [33, 35, 40] provide a robust image prior for supervising text-to-3D generation. Diffusion models learn to estimate the denoising score  $\nabla_{\mathbf{x}}\log p_{\mathrm{data}}(\mathbf{x})$  by adding noise to clean data  $\mathbf{x}\sim p(\mathbf{x})$  (forward process) and learning to reverse the added noise (backward process). Noising the data distribution to isotropic Gaussian is performed in  $T$  timesteps, with a pre-defined noisng schedule  $\alpha_{t}\in (0,1)$  and  $\bar{\alpha}_{t}\coloneqq \prod_{s = 1}^{t}\alpha_{s}$ , according to:
$$
\mathbf {z} _ {t} = \sqrt {\bar {\alpha} _ {t}} \mathbf {x} + \sqrt {1 - \bar {\alpha} _ {t}} \epsilon , \text {w h e r e} \epsilon \sim \mathcal {N} (\mathbf {0}, \mathbf {I}).
$$
In the training process, the diffusion models learn to estimate the noise by
$$
\mathcal {L} _ {t} = \mathbb {E} _ {\mathbf {x}, \epsilon \sim \mathcal {N} (\mathbf {0}, \mathbf {I})} \left[ \| \epsilon_ {\phi} (\mathbf {z} _ {t}, t) - \epsilon \| _ {2} ^ {2} \right].
$$
Once trained, one can estimate  $\mathbf{x}$  from noisy input and the corresponding noise prediction.
(3) Score Distillation Sampling (SDS) [31, 18, 21] is a technique introduced by DreamFusion [31] and extensively employed to distill knowledge from a pre-trained diffusion model  $\epsilon_{\phi}$  into a differentiable 3D representation. For a NeRF model parameterized by  $\theta$ , its rendering  $\mathbf{x}$  can be obtained by  $\mathbf{x} = g(\theta)$  where  $g$  is a differentiable renderer. SDS calculates the gradients of NeRF parameters  $\theta$  by,
$$
\nabla_ {\boldsymbol {\theta}} \mathcal {L} _ {\mathrm {S D S}} (\phi , \mathbf {x}) = \mathbb {E} _ {t, \boldsymbol {\epsilon}} \left[ w (t) \left(\epsilon_ {\phi} \left(\mathbf {x} _ {t}; y, t\right) - \boldsymbol {\epsilon}\right) \frac {\partial \mathbf {z} _ {t}}{\partial \mathbf {x}} \frac {\partial \mathbf {x}}{\partial \boldsymbol {\theta}} \right], \tag {1}
$$
where  $w(t)$  is a weighting function that depends on the timestep  $t$  and  $y$  denotes the given text prompt.
SMPL [19] is a 3D parametric human body model with a vertex-based linear deformation model, which decomposes body deformation into identity-related and pose-related shape deformation. It contains  $N = 6,890$  vertices and  $K = 24$  keypoints. Benefiting from its efficient and expressive human motion representation ability, SMPL has been widely used in human motion-driven tasks [8, 47, 20]. SMPL parameters include a 3D body joint rotation  $\xi \in \mathbb{R}^{K\times 3}$ , a body shape  $\beta \in \mathbb{R}^{10}$ , and a 3D global scale and translation  $t\in \mathbb{R}^3$ .
Formally, constructing a rest pose  $T(\beta, \xi)$  involves combining the mean template shape  $\bar{T}$  from the canonical space, the shape-dependent deformations  $B_{S}(\beta) \in \mathbb{R}^{3N}$ , and the pose-dependent deformations  $B_{P}(\xi) \in \mathbb{R}^{3N}$  to relieve artifacts in a standard linear blend skinning (LBS) [24] by,
$$
T (\beta , \xi) = \bar {T} + B _ {S} (\beta) + B _ {P} (\xi).
$$
To map the SMPL parameters  $\beta, \xi$  to a triangulated mesh, a function  $M$  is adopted to combine the rest pose mesh  $T(\beta, \xi)$ , the corresponding keypoint positions  $\mathcal{J}(\beta) \in \mathbb{R}^{3K}$ , pose  $\xi$ , and a set of blend weights  $\mathcal{W} \in \mathbb{R}^{N \times K}$  via a LBS function as,
$$
M (\beta , \xi) = \mathrm {W} (T (\beta , \xi), \mathcal {J} (\beta), \xi , \mathcal {W}).
$$
To obtain the corresponding vertex under the observation pose  $\mathbf{v}_o$ , an affine deformation  $\mathcal{G}_k(\xi, j_k)$  with skinning weight  $w_k$  is used to transform the  $k_{th}$  keypoint  $j_k$  from the canonical pose to the observation pose as,
$$
\mathbf {v} _ {o} = \sum_ {k = 1} ^ {K} w _ {k} \mathcal {G} _ {k} (\xi , j _ {k}). \tag {2}
$$
3.2 DreamWaltz: A Text-to-Avatar Generation Framework
3.2.1 Creating a Canonical Avatar
DreamWaltz employs a trainable NeRF as the 3D avatar representation. It leverages SMPL prior in two ways: (1) initializing NeRF; (2) extracting 3D-aware and occlusion-aware posed-skeletons to condition ControlNet [48] for 3D-consistent Score Distillation Sampling. Fig 3 (a) illustrates our method on how to create a canonical avatar.
(a) Canonical Avatar Creation. We randomly sample a camera viewpoint to render both the 3D avatar and an SMPL model. The extracted SMPL 3D keypoints are projected to a 2D skeleton image with occlusion culling. The skeleton image is by construction 3D-consistent with the rendered avatar. We then utilize ControlNet [48] conditioned on the text prompt and the skeleton image instead of the original SD [34].
(b) Animatable Avatar Learning. We randomly sample viable poses from off-the-shelf VPoser [29] for SMPL model to condition ControlNet [48] and to learn a generalizable density weighting function to refine vertex-based pose transformation, enabling animation of complex avatars.
Figure 3: Illustration of our framework for canonical and animatable avatar creation. (a) shows how to create a canonical avatar from text with 3D-consistent occlusion-aware Score Distillation Sampling, and (b) demonstrates how to further learn an animatable avatar with sampled human pose prior.
SMPL-guided initialization. To speed up the NeRF optimization and to provide a reasonable initial input for retrieving informative supervision from the diffusion model, we pre-train NeRF based on an SMPL mesh. The SMPL model could be in the canonical pose as adopted in our method to avoid self-occlusion or in any chosen pose for posed-ayar creation [3]. Specifically, we render the silhouette image  $x_{\mathrm{sil}}$  of the SMPL model given a randomly sampled viewpoint and minimize the MSE loss between the NeRF-rendered image  $x$  and the silhouette image  $x_{\mathrm{sil}}$ . Note that our NeRF renders images in the latent space of Stable Diffusion [34], so it is necessary to use the VAE-based image encoder to transform silhouette images into the latent space for the loss calculation. We empirically found that SMPL-guided NeRF initialization significantly improves the geometry and the convergence speed for avatar generation.
3D-consistent Score Distillation Sampling. Vanilla SDS [40, 43, 18] as previously introduced in Sec. 3.1 utilizes view-dependent prompt augmentations such as "front view of ..." for diffusion model to provide crucial 3D view-consistent supervision. However, this prompting strategy cannot guarantee precise view consistency, leaving the disparity between the viewpoint of the diffusion model's supervision image and NeRF's rendering image unresolved. Such inconsistency causes quality issues for 3D generation, such as blurriness and the Janus (multi-face) problem. Inspired by recent works of controllable image generation [48, 15], we propose to utilize additional 3D-aware conditioning images to improve SDS for 3D-consistent NeRF optimization. Specifically, additional
conditioning image  $c$  is injected to Eq. 1 for SDS gradient computation:
$$
\nabla_ {\boldsymbol {\theta}} \mathcal {L} _ {\mathrm {S D S}} (\phi , \mathbf {x}) = \mathbb {E} _ {t, \epsilon} \left[ w (t) \left(\epsilon_ {\phi} \left(\mathbf {x} _ {t}; y, t, c\right) - \epsilon\right) \frac {\partial \mathbf {z} _ {t}}{\partial \mathbf {x}} \frac {\partial \mathbf {x}}{\partial \boldsymbol {\theta}} \right], \tag {3}
$$
where conditioning image  $c$  can be one or a combination of skeletons, depth maps, normal maps and etc. In practice, we choose skeletons as the conditional image type because they provide minimal image structure priors and enable complex avatar generation. In order to acquire 3D-consistent supervision, the conditioning image's viewpoint should be in sync with NeRF's rendering viewpoint. To achieve this for avatar generation, we use human SMPL models to produce conditioning images.
Occlusion culling. The introduction of 3D-aware conditional images can enhance the 3D consistency in the SDS optimization process. However, the effectiveness is constrained by the adopted diffusion model [48] on its interpretation of the conditional images. As shown in Fig. 8 (a), we provide a back-view skeleton map as the conditional image to ControlNet [48] and perform text-to-image generation, but a frontal face still appears in the generated image. Such defects bring problems such as multiple faces and unclear facial features to 3D avatar generation. To this end, we propose to use occlusion culling algorithms [28] in computational graphics to detect whether facial keypoints are visible from the given viewpoint and subsequently remove them from the skeleton map if considered invisible. Body keypoints remain unaltered because they reside in the SMPL mesh, and it is difficult to determine whether they are occluded without introducing new priors.
3.2.2 Learning an Animatable Avatar
Fig. 3 (b) illustrates our framework for generating animatable 3D avatars. In the training process, we randomly sample SMPL models of viable poses from VPoser [29] to condition ControlNet [48] and to learn a generalizable density weighting function to refine vertex-based pose transformation, enabling animation of complex avatars. At test time, DreamWaltz is capable of creating an animation based on arbitrary motion sequences without requiring further pose-by-pose retraining.
SMPL-guided avatar articulation. Referring to Sec. 3.1, SMPL defines a vertex transformation from observation space to canonical space according to Eq. 2. In this work, we use SMPL-guided transformation to achieve NeRF-represented avatar articulation. More concretely, for each sampled point  $\mathbf{p}$  on a NeRF ray, we find its closest vertex  $\mathbf{v}_c$  based on a posed SMPL mesh. We then utilize the transformation matrix  $\mathbf{T}_{\mathrm{skel}}$  of  $\mathbf{v}_c$  to project  $\mathbf{p}$  to the canonical space feature grid for feature querying and subsequent volume rendering. When  $\mathbf{p}$  is close to  $\mathbf{v}_c$ , the calculated articulation is approximately correct. However, for non-skin-tight complex avatars,  $\mathbf{p}$  may be far away from any mesh vertex, resulting in erroneous coordinate transformation causing quality issues such as extra limbs and artifacts. To avoid such problems, we further introduce a novel density weighting mechanism.
Density weighting network. We propose a density weighting mechanism to suppress color contribution from erroneously transformed point  $\mathbf{p}$ , effectively alleviating undesirable artifacts. To achieve this, we train a generalizable density weighting network  $\mathrm{MLP}_{\mathrm{DWN}}$ . More concretely, we project sampled point  $\mathbf{p}$  and its closest vertex  $\mathbf{v}_c$  to the canonical space via the transformation matrix  $\mathbf{T}_{\mathrm{skel}}$ , embed these two coordinates with the positional embedding function  $\mathrm{PE}(\cdot)$  and then use the concatenated embeddings as inputs to  $\mathrm{MLP}_{\mathrm{DWN}}$ . The process can be defined as,
$$
d ^ {\prime} = \mathrm {M L P} _ {\mathrm {D W N}} \left(\mathrm {P E} \left(\mathbf {T} _ {\mathrm {s k e l}} \cdot \mathbf {p}\right) \oplus \mathrm {P E} \left(\mathbf {T} _ {\mathrm {s k e l}} \cdot \mathbf {v} _ {c}\right)\right). \tag {4}
$$
We then compute density weights  $w_{d}$  according to the distance  $d$  between  $\mathbf{p}$  and  $\mathbf{v}_c$ , and  $d'$ :
$$
w _ {d} = \operatorname {S i g m o i d} \left(- \left(d - d ^ {\prime}\right) / a\right). \tag {5}
$$
where  $a$  is a preset parameter. Finally, the density  $\delta$  of sampled point  $\mathbf{p}$  is re-weighted to  $\delta \cdot w_{d}$  for subsequent volume rendering in NeRF.
Sampled human pose prior. To enable animating generated avatars with arbitrary motion sequences, we need to make sure that the density weighting network  $\mathrm{MLP}_{\mathrm{DWN}}$  is generalizable to arbitrary poses. To achieve this, we utilize VPoser [29] as a human pose prior, which is a variational autoencoder that learns a latent representation of human pose. During training, we randomly sample SMPL pose parameters  $\xi$  from VPoser to construct the corresponding posed meshes. We utilize the mesh to (1) extract skeleton maps as conditioning images for 3D-consistent SDS; (2) to serve as mesh guidance for learning animatable avatar representation. This strategy aligns avatar articulation learning with
Figure 4: Qualitative results from two views of DreamWaltz. Given text prompts, it can generate high-quality 3D avatars with complex geometry and texture.
SDS supervision, ensuring that  $\mathrm{MLP}_{\mathrm{DWN}}$  could learn a generalizable density weighting function from diverse poses. We also observe that SDS with diverse pose conditioning could further improve the visual quality of created avatar, e.g., sharper appearance.
3.3 Making a Scene with Animatable 3D Avatars
DreamWaltz provides an effective solution to animatable 3D avatar generation from text, readily applicable to make a scene with diverse interactions. For example, different animatable avatars could be rendered in the same scene, achieving avatar-avatar animation. In Fig. 1 (c), we can make the avatar "Woody" dance with "Stormtrooper". The animation process does not require any retraining. However, the constructed scene might exhibit unruliness, artifacts, or interpenetration issues because the naive composite rendering does not take into account the interactions between different components.