SlowGuess/NeurIPS_2023_OCR huggingface dataset:
Xiao Han $^{1,4*}$   
Jiankang Deng $^{3}$
Yukang Cao $^{2*}$  Yi-Zhe Song $^{1,4}$
Kai Han $^{2}$   
Tao Xiang $^{1,4\dagger}$
Xiatian Zhu $^{1,5}$   
Kwan-Yee K. Wong $^{2\dagger}$
<sup>1</sup>University of Surrey <sup>2</sup>The University of Hong Kong <sup>3</sup>Imperial College London <sup>4</sup>iFlyTek-Surrey Joint Research Centre on AI <sup>5</sup>Surrey Institute for People-Centred AI
Recently, text-guided 3D generative methods have made remarkable advancements in producing high-quality textures and geometry, capitalizing on the proliferation of large vision-language and image diffusion models. However, existing methods still struggle to create high-fidelity 3D head avatars in two aspects: (1) They rely mostly on a pre-trained text-to-image diffusion model whilst missing the necessary 3D awareness and head priors. This makes them prone to inconsistency and geometric distortions in the generated avatars. (2) They fall short in fine-grained editing. This is primarily due to the inherited limitations from the pre-trained 2D image diffusion models, which become more pronounced when it comes to 3D head avatars. In this work, we address these challenges by introducing a versatile coarse-to-fine pipeline dubbed HeadSculpt for crafting (i.e., generating and editing) 3D head avatars from textual prompts. Specifically, we first equip the diffusion model with 3D awareness by leveraging landmark-based control and a learned textual embedding representing the back view appearance of heads, enabling 3D-consistent head avatar generations. We further propose a novel identity-aware editing score distillation strategy to optimize a textured mesh with a high-resolution differentiable rendering technique. This enables identity preservation while following the editing instruction. We showcase HeadSculpt's superior fidelity and editing capabilities through comprehensive experiments and comparisons with existing methods.
Modeling 3D head avatars underpins a wide range of emerging applications (e.g., digital telepresence, game character creation, and AR/VR). Historically, the creation of intricate and detailed 3D head avatars demanded considerable time and expertise in art and engineering. With the advent of deep learning, existing works [87, 28, 33, 72, 8, 38, 15] have shown promising results on the reconstruction of 3D human heads from monocular images or videos. However, these methods remain restricted to head appearance contained in their training data which is often limited in size, resulting in the inability to generalize to new appearance beyond the training data. This constraint calls for the need of more flexible and generalizable methods for 3D head modeling.
Recently, vision-language models (e.g., CLIP [55]) and diffusion models (e.g., Stable Diffusion [69, 61, 59]) have attracted increasing interest. These progresses have led to the emergence of text-to-3D generative models [34, 62, 44, 27] which create 3D content in a self-supervised manner. Notably, DreamFusion [54] introduces a score distillation sampling (SDS) strategy that leverages a pre-trained image diffusion model to compute the noise-level loss from the textual description, unlocking the potential to optimize differentiable 3D scenes (e.g., neural radiance field [45], tetrahedral mesh [66], texture [58, 9], or point cloud [50]) with 2D diffusion prior only. Subsequent research efforts [43, 6,
*Equal contributions † Corresponding authors
‡ Webpage: https://brandonhan.uk/HeadSculpt
Figure 1: Examples of generation and editing results obtained using the proposed HeadSculpt. It enables the creation and fine-grained editing of high-quality head avatars, featuring intricate geometry and texture, for any type of head avatar using simple descriptions or instructions. Symbols indicate the following prompt prefixes: * “a head of [text]” and † “a DSLR portrait of [text]”. The captions in gray are the prompt suffixes while the blue ones are the editing instructions.
65, 79, 42, 75, 40, 56, 76] improve and extend DreamFusion from various perspectives (e.g., higher resolution [39] and better geometry [10]).
Considering the flexibility and versatility of natural languages, one might think that these SDS-based text-to-3D generative methods would be sufficient for generating diverse 3D avatars. However, it is noted that existing methods have two major drawbacks (see Fig. 6): (1) Inconsistency and geometric distortions: The 2D diffusion models used in these methods lack 3D awareness particularly regarding camera pose; without any remedy, existing text-to-3D methods inherited this limitation, leading to the multi-face "Janus" problem in the generated head avatars. (2) Fine-grained editing limitations: Although previous methods propose to edit 3D models by naively fine-tuning trained models with modified prompts [54, 39], we find that this approach is prone to biased outcomes, such as identity loss or inadequate editing. This problem arises from two causes: (a) inherent bias in prompt-based editing in image diffusion models, and (b) challenges with inconsistent gradient back-propagation at separate iterations when using SDS calculated from a vanilla image diffusion model.
In this paper, we introduce a new head-athlete-focused text-to-3D method, dubbed HeadSculpt, that supports high-fidelity generation and fine-grained editing. Our method comprises two novel
components: (1) Prior-driven score distillation: We first arm the pre-trained image diffusion model with 3D awareness by integrating a landmark-based ControlNet [84]. Specifically, we adopt the parametric 3D head model, FLAME [38], as a prior to obtain a 2D landmark map [41, 31], which serves as an additional condition for the diffusion model, ensuring the consistency of generated head avatars across different views. Further, to remedy the front-view bias in the pre-trained diffusion model, we utilize an improved view-dependent prompt through textual inversion [17], by learning a specialized <back-view> token to emphasize back views of heads and capture their unique visual details. (2) Identity-aware editing score distillation (IESD): To address the challenges of fine-grained editing for head avatars, we introduce a novel method called IESD. It blends two scores, one for editing and the other for identity preservation, both predicted by a ControlNet-based implementation of InstructPix2Pix [5]. This approach maintains a controlled editing direction that respects both the original identity and the editing instructions. To further improve the fidelity of our method, we integrate these two novel components into a coarse-to-fine pipeline [39], utilizing NeRF [48] as the low-resolution coarse model and DMTET [66] as the high-resolution fine model. As demonstrated in Fig. 1, our method can generate high-fidelity human-like and non-human-like head avatars while enabling fine-grained editing, including local changes, shape/texture modifications, and style transfers.
Text-to-2D generation. In recent years, groundbreaking vision-language technologies such as CLIP [55] and diffusion models [25, 13, 59, 68] have led to significant advancements in text-to-2D content generation [61, 57, 1, 69, 70]. Trained on extensive 2D multimodal datasets [63, 64], they are empowered with the capability to "dream" from the prompt. Follow-up works endeavor to efficiently control the generated results [84, 85, 47], extend the diffusion model to video sequence [67, 3], accomplish image or video editing [23, 32, 81, 5, 77, 14, 22], enhance the performance for personalized subjects [60, 17], etc. Although significant progress has been made in generating 2D content from text, carefully crafting the prompt is crucial, and obtaining the desired outcome often requires multiple attempts. The inherent randomness remains a challenge, especially for editing tasks.
Text-to-3D generation. Advancements in text-to-2D generation have paved the way for text-to-3D techniques. Early efforts [82, 27, 44, 62, 34, 29, 11] propose to optimize the 3D neural radiance field (NeRF) or vertex-based meshes by employing the CLIP language model. However, these models encounter difficulties in generating expressive 3D content, primarily because of the limitations of CLIP in comprehending natural language. Fortunately, the development of image diffusion models [69, 1] has led to the emergence of DreamFusion [54]. It proposes Score Distillation Sampling (SDS) based on a pre-trained 2D diffusion prior [61], showcasing promising generation results. Subsequent works [37] have endeavored to improve DreamFusion from various aspects: Magic3D [39] proposes a coarse-to-fine pipeline for high-resolution generations; Latent-NeRF [43] includes shape guidance for more robust generation on the latent space [59]; DreamAvatar [6] leverages SMPL [4] to generate 3D human full-body avatars under controllable shapes and poses; Guide3D [7] explores the usage of multi-view generated images to create 3D human avatars; Fantasia3D [10] disentangles the geometry and texture training with DMTET [66] and PBR texture [49] as their 3D representation; 3DFuse [65] integrates depth control and semantic code sampling to stabilize the generation process. Despite notable progress, current text-to-3D generative models still face challenges in producing view-consistent 3D content, especially for intricate head avatars. This is primarily due to the absence of 3D awareness in text-to-2D diffusion models. Additionally, to the best of our knowledge, there is currently no approach that specifically focuses on editing the generated 3D content, especially addressing the intricate fine-grained editing needs of head avatars.
3D head modeling and creation. Statistical mesh-based models, such as FLAME [38, 15], enable the reconstruction of 3D head models from images. However, they struggle to capture fine details like hair and wrinkles. To overcome this issue, recent approaches [8, 71, 72, 51] employ Generative Adversarial Networks (GANs) [46, 20, 30] to train 3D-aware networks on 2D head datasets and produce 3D-consistent images through latent code manipulation. Furthermore, neural implicit methods [87, 16, 28, 88] introduce implicit and subject-oriented head models based on neural rendering fields [45, 48, 2]. Recently, text-to-3D generative methods have gained traction, generating high-quality 3D head avatars from natural language using vision-language models [55, 69]. Typically, T2P [85] predicts bone-driven parameters of head avatars via a game engine under the CLIP guidance [55]. Rodin [80] proposes a roll-out diffusion network to perform 3D-aware diffusion. DreamFace [83] employs a
a DSLR portrait of Saul Goodman
turn him into a clown
Figure 2: Overall architecture of HeadSculpt. We craft high-resolution 3D head avatars in a coarse-to-fine manner. (a) We optimize neural field representations for the coarse model. (b) We refine or edit the model using the extracted 3D mesh and apply identity-aware editing score distillation if editing is the target. (c) The core of our pipeline is the prior-driven score distillation, which incorporates landmark control, enhanced view-dependent prompts, and an InstructPix2Pix branch.
selection strategy in the CLIP embedding space to generate coarse geometry and uses SDS [54] to optimize UV-texture. Despite producing promising results, all these methods require a large amount of data for supervised training and struggle to generalize well to non-human-like avatars. In contrast, our approach relies solely on pre-trained text-to-2D models, generalizes well to out-of-domain avatars, and is capable of performing fine-grained editing tasks.
HeadSculpt is a 3D-aware text-to-3D approach that utilizes a pre-trained text-to-2D Stable Diffusion model [69, 59] to generate high-resolution head avatars and perform fine-grained editing tasks. As illustrated in Fig. 2, the generation pipeline has two stages: coarse generation via the neural radiance field (NeRF) [48] and refinement/editing using tetrahedron mesh (DMTET) [66]. Next, we will first introduce the preliminaries that form the basis of our method in Sec. 3.1. We will then discuss the key components of our approach in Sec. 3.2 and Sec. 3.3, including (1) the prior-driven score distillation process via landmark-based ControlNet [84] and textual inversion [17], and (2) identity-aware editing score distillation accomplished in the fine stage using the ControlNet-based InstructPix2Pix [5].
Score distillation sampling. Recently, DreamFusion [54] proposed score distillation sampling (SDS) to self-optimize a text-consistent neural radiance field (NeRF) based on the pre-trained text-to-2D diffusion model [61]. Due to the unavailability of the Imagen model [61] used by DreamFusion, we employ the latent diffusion model in [59] instead. Specifically, given a latent feature  $z$  encoded from an image  $x$ , SDS introduces random noise  $\epsilon$  to  $z$  to create a noisy latent variable  $z_{t}$  and then uses a pre-trained denoising function  $\epsilon_{\phi}(z_{t};y,t)$  to predict the added noise. The SDS loss is defined as the difference between predicted and added noise and its gradient is given by
$$
\nabla_ {\theta} \mathcal {L} _ {\mathrm {S D S}} (\phi , g (\theta)) = \mathbb {E} _ {t, \epsilon \sim \mathcal {N} (0, 1)} \left[ w (t) \left(\epsilon_ {\phi} \left(z _ {t}; y, t\right) - \epsilon\right) \frac {\partial z}{\partial x} \frac {\partial x}{\partial \theta} \right], \tag {1}
$$
where  $y$  is the text embedding,  $w(t)$  weights the loss from noise level  $t$ . With the expressive text-to-2D diffusion model and self-supervised SDS loss, we can back-propagate the gradients to optimize an implicit 3D scene  $g(\theta)$ , eliminating the need for an expensive 3D dataset.
3D scene optimization. HeadSculpt explores the potential of two different 3D differentiable representations as the optimization basis for crafting 3D head avatars. Specifically, we employ NeRF [48] in the coarse stage due to its greater flexibility in geometry deformation, while utilizing DMTET [66] in the fine stage for efficient high-resolution optimization.
(1) 3D prior-based NeRF. DreamAvatar [6] recently proposed a density-residual setup to enhance the robustness and controllability of the generated 3D NeRF. Given a point  $\mathbf{x}$  inside the 3D volume, we can derive its density and color value based on a prior-based density field  $\bar{\sigma}$ :
$$
F (\mathbf {x}, \bar {\sigma}) = F _ {\theta} (\gamma (\mathbf {x})) + (\bar {\sigma} (\mathbf {x}), \mathbf {0}) \mapsto (\sigma , \mathbf {c}), \tag {2}
$$
where  $\gamma (\cdot)$  denotes a hash-grid frequency encoder [48], and  $\sigma$  and  $\mathbf{c}$  are the density and RGB color respectively. We can derive  $\bar{\sigma}$  from the signed distance  $d(\mathbf{x})$  of a given 3D shape prior (e.g., a canonical FLAME model [38] by default in our implementation):
$$
\bar {\sigma} (\mathbf {x}) = \max  \left(0, \text {s o f t p l u s} ^ {- 1} (\tau (\mathbf {x}))\right), \tau (\mathbf {x}) = \frac {1}{a} \operatorname {s i g m o i d} (- d (\mathbf {x}) / a), \text {w h e r e} a = 0. 0 0 5. \tag {3}
$$
To obtain a 2D RGB image from the implicit volume defined above, we employ a volume rendering technique that involves casting a ray  $\mathbf{r}$  from the 2D pixel location into the 3D scene, sampling points  $\pmb{\mu}_i$  along the ray, and calculating their density and color value using  $F$  in Eq. (2):
$$
C (\mathbf {r}) = \sum_ {i} W _ {i} \mathbf {c} _ {i}, \quad W _ {i} = \alpha_ {i} \prod_ {j <   i} (1 - \alpha_ {j}), \quad \alpha_ {i} = 1 - e ^ {\left(- \sigma_ {i} \left\| \boldsymbol {\mu} _ {i} - \boldsymbol {\mu} _ {i + 1} \right\|\right)}. \tag {4}
$$
(2) DMTET. It discretizes a deformable tetrahedral grid  $(V_{T}, T)$ , where  $V_{T}$  denotes the vertices within grid  $T$  [19, 66], to model the 3D space. Every vertex  $\mathbf{v}_{i} \in V_{T} \subset \mathbb{R}^{3}$  possesses a signed distance value  $s_{i} \in \mathbb{R}$ , along with a position offset  $\Delta \mathbf{v}_{i} \in \mathbb{R}^{3}$  of the vertex relative to its initial canonical coordinates. Subsequently, the underlying mesh can be extracted based on  $s_{i}$  with the differentiable marching tetrahedra algorithm. In addition to the geometry, we adopt the Magic3D approach [39] to construct a neural color field. This involves re-utilizing the MLP trained in the coarse NeRF stage to predict the RGB color value for each 3D point. During optimization, we render this textured surface mesh into high-resolution images using a differentiable rasterizer [36, 49].
3.2 3D-Prior-driven score distillation
Existing text-to-3D methods with SDS [54] assume that maximizing the likelihood of images rendered from various viewpoints of a scene model  $g(\cdot)$  is equivalent to maximizing the overall likelihood of  $g(\cdot)$ . This assumption can result in inconsistencies and geometric distortions [54, 65]. A notable issue is the "Janus problem" characterized by multiple faces on a single object (see Fig. 6). There are two possible causes: (1) the randomness of the diffusion model which can cause inconsistencies among different views, and (2) the lack of 3D awareness in controlling the generation process, causing the model to struggle in determining the front view, back view, etc. To address these issues in generating head avatars, we integrate 3D head priors into the diffusion model.
Landmark-based ControlNet. In Section 3.1, we explain our adoption of FLAME [38] as the density guidance for our NeRF. Nevertheless, this guidance by itself is insufficient to have a direct impact on the SDS loss. What is missing is a link between the NeRF and the diffusion model, incorporating the same head priors. Such a link is key to improving the view consistency of the generated head avatars. To achieve this objective, as illustrated in Fig. 2, we propose the incorporation of 2D landmark maps as an additional condition for the diffusion model using ControlNet [84]. Specifically, we employ a ControlNet  $\mathcal{C}$  trained on a large-scale 2D face dataset [86, 12], using facial landmarks rendered from MediaPipe [41, 31] as ground-truth data. When given a randomly sampled camera pose  $\pi$ , we first project the vertices of the FLAME model onto the image. Following that, we select and render some of these vertices into a landmark map  $\mathcal{P}_{\pi}$  based on some predefined vertex indexes. The landmark map will be fed into ControlNet and its output features are added to the intermediate features within the diffusion U-Net. The gradient of our SDS loss can be re-written as
$$
\nabla_ {\theta} \mathcal {L} _ {\mathrm {S D S}} (\phi , g (\theta)) = \mathbb {E} _ {t, \epsilon \sim \mathcal {N} (0, 1), \pi} \left[ w (t) \left(\epsilon_ {\phi} \left(z _ {t}; y, t, \mathcal {C} \left(\mathcal {P} _ {\pi}\right)\right) - \epsilon\right) \frac {\partial z}{\partial x} \frac {\partial x}{\partial \theta} \right]. \tag {5}
$$
Enhanced view-dependent prompt via textual inversion. Although the landmark-based ControlNet can inject 3D awareness into the pre-trained diffusion models, it struggles to maintain back view head consistency. This is expected as the 2D image dataset used for training mostly contains only front or side face views. Consequently, when applied directly to back views, the model introduces ambiguity as front and back 3D landmark views can appear similar, as shown in Fig. 8. To address this issue, we propose a simple yet effective method. Our method is inspired by previous works [54, 65, 39] which found it beneficial to append view-dependent text (e.g., "front view", "side view" or "back
view") to the provided input text based on the azimuth angle of the randomly sampled camera. We extend this idea by learning a special token <back-view> to replace the plain text "back view" in order to emphasize the rear appearance of heads. This is based on the assumption that a pre-trained Stable Diffusion does has the ability to "imagine" the back view of a head - it has seen some during training. The main problem is that a generic text embedding of "back view" is inadequate in telling the model what appearance it entails. A better embedding for "back view" is thus required. To this end, we first randomly download 34 images of the back view of human heads, without revealing any personal identities, to construct a tiny dataset  $\mathcal{D}$ , and then we optimize the special token  $v$  (i.e., <back-view>) to better fit the collected images, similar to the textual inversion [17]:
$$
v _ {*} = \underset {v} {\arg \min } \mathbb {E} _ {t, \epsilon \sim \mathcal {N} (0, 1), z \sim \mathcal {D}} \left[ \| \epsilon - \epsilon_ {\phi} \left(z _ {t}; v, t\right) \| _ {2} ^ {2} \right], \tag {6}
$$
which is achieved by employing the same training scheme as the original diffusion model, while keeping  $\epsilon_{\phi}$  fixed. This constitutes a reconstruction task, which we anticipate will encourage the learned embedding to capture the fine visual details of the back views of human heads. Notably, as we do not update the weights of  $\epsilon_{\phi}$ , it stays compatible with the landmark-based ControlNet.
3.3 Identity-aware editing score distillation
After generating avatars, editing them to fulfill particular requirements poses an additional challenge. Previous works [54, 39] have shown promising editing results by fine-tuning a trained scene model with a new target prompt. However, when applied to head avatars, these methods often suffer from identity loss or inadequate appearance modifications (see Fig. 10). This problem stems from the inherent constraint of the SDS loss, where the 3D models often sacrifice prominent features to preserve view consistency. Substituting Stable Diffusion with InstructPix2Pix [5, 21] might seem like a simple solution, but it also faces difficulties in maintaining facial identity during editing based only on instructions, as it lacks a well-defined anchor point.
To this end, we propose identity-aware editing score distillation (IESD) to regulate the editing direction by blending two predicted scores, i.e., one for editing instruction and another for the original description. Rather than using the original InstructPix2Pix [5], we employ a ControlNet-based InstructPix2Pix  $\mathcal{I}$  [84] trained on the same dataset, ensuring compatibility with our landmark-based ControlNet  $\mathcal{C}$  and the learned <back-view> token. Formally, given an initial textual prompt  $y$  describing the avatar to be edited and an editing instruction  $\hat{y}$ , we first input them separately into the same diffusion model equipped with two ControlNets,  $\mathcal{I}$  and  $\mathcal{C}$ . This allows us to obtain two predicted noises, which are then combined using a predefined hyper-parameter  $\omega_{e}$  like classifier-free diffusion guidance (CFG) [26]:
$$
\begin{array}{l} \nabla_ {\theta} \mathcal {L} _ {\mathrm {I E S D}} (\phi , g (\theta)) = \mathbb {E} _ {t, \epsilon \sim \mathcal {N} (0, 1), \pi} \left[ w (t) \left(\underbrace {\hat {\epsilon} _ {\phi} \left(z _ {t} ; y , \hat {y} , t , \mathcal {C} \left(\mathcal {P} _ {\pi}\right) , \mathcal {I} \left(\mathcal {M} _ {\pi}\right)\right)} - \epsilon\right) \frac {\partial z}{\partial x} \frac {\partial x}{\partial \theta} \right], \tag {7} \\ \omega_ {e} \epsilon_ {\phi} (z _ {t}; \hat {y}, t, \mathcal {C} (\mathcal {P} _ {\pi}), \mathcal {I} (\mathcal {M} _ {\pi})) + (1 - \omega_ {e}) \epsilon_ {\phi} (z _ {t}; y, t, \mathcal {C} (\mathcal {P} _ {\pi}), \mathcal {I} (\mathcal {M} _ {\pi})) \\ \end{array}
$$
where  $\mathcal{P}_{\pi}$  and  $\mathcal{M}_{\pi}$  represent the 2D landmark maps and the reference images rendered in the coarse stage, both being obtained under the sampled camera pose  $\pi$ . The parameter  $\omega_{e}$  governs a trade-off between the original appearance and the desired editing, which defaults to 0.6 in our experiments.
We will now assess the efficacy of our HeadSculpt across different scenarios, while also conducting a comparative analysis against state-of-the-art text-to-3D generation pipelines.