SlowGuess/NeurIPS_2023_OCR huggingface dataset:
Implementation details. HeadSculpt builds upon Stable-DreamFusion [73] and Huggingface Diffusers [78, 53]. We utilize version 1.5 of Stable Diffusion [69] and version 1.1 of ControlNet [84, 12] in our implementation. In the coarse stage, we optimize our 3D model at  $64 \times 64$  grid resolution, while using  $512 \times 512$  grid resolution for the fine stage (refinement or editing). Typically, each text prompt requires approximately 7,000 iterations for the coarse stage and 5,000 iterations for the fine stage. It takes around 1 hour for each stage on a single Tesla V100 GPU with a default batch size of 4. We use Adam [35] optimizer with a fixed learning rate of 0.001. Additional implementation details can be found in the supplementary material.
Shape 1
Shape 2
Shape 3
Figure 3: Generation results with various shapes. The first row shows three randomly sampled FLAME models, while the second row presents our generated results (incl. normals) using these FLAME models as initialization. All results are under the same text prompt.
$\ddagger$  sad
Figure 4: More specific editing results.  $\ddagger$  Instruction prefix: make his expression as [text].
‡ disgusted
make him bald
give him a beard
give him a sunglass
Baseline methods for generation evaluation. We compare the generation results with five baselines: DreamFusion [73], Latent-NeRF [43], 3DFuse [65] (improved version of SJC [79]), Fantasy3D [10], and DreamFace [83]. We do not directly compare with DreamAvatar [6] as it involves deformation fields for full-body-related tasks.
Baseline methods for editing evaluation. We assess IESD's efficacy for fine-grained 3D head avatar editing by comparing it with various alternatives since no dedicated method exists for this: (B1) One-step optimization on the coarse stage without initialization; (B2) Initialized from the coarse stage, followed by optimization of another coarse stage with an altered description; (B3) Initialized from the coarse stage, followed by optimization of a new fine stage with an altered description; (B4) Initialized from the coarse stage, followed by optimization of a new fine stage with an instruction based on the vanilla InstructPix2Pix [5]; (B5) Ours without edit scale (i.e.,  $\omega_{e} = 1$ ). Notably, B2 represents the editing method proposed in DreamFusion [54], while B3 has a similar performance as Magic3D [39], which employs a three-stage editing process (i.e., Coarse + Coarse + Fine).
4.1 Qualitative evaluations
Head avatar generation with various prompts. In Fig. 1, we show a diverse array of 3D head avatars generated by our HeadSculpt, consistently demonstrating high-quality geometry and texture across various viewpoints. Our method's versatility is emphasized by its ability to create an assortment of avatars, including humans (both celebrities and ordinary individuals) as well as non-human characters like superheroes, comic/game characters, paintings, and more.
Head avatar generation with different shapes. HeadSculpt leverages shape-guided NeRF initialization and landmark-guided diffusion priors. This allows controlling geometry by varying the FLAME shape used for initialization. To demonstrate adjustability, Fig. 3 presents examples generated from diverse FLAME shapes. The results fit closely to the shape guidance, highlighting HeadSculpt's capacity for geometric variation when provided different initial shapes.
Head avatar editing with various instructions. As illustrated in Fig. 1 and Fig. 4, HeadSculpt's adaptability is also showcased through its ability to perform fine-grained editing, such as local changes (e.g., adding accessories, changing hairstyles, or altering expressions), shape and texture modifications, and style transfers.
Head avatar editing with different edit scales. In Fig. 5, we demonstrate the effectiveness of IESD with different  $\omega_{e}$  values, highlighting its ability to control editing influence on the reference identity.
Figure 5: Impact of the edit scale  $\omega_{e}$  in IESD. It balances the preservation of the initial appearance and the extent of the desired editing, making the editing process more controllable and flexible.
Figure 6: Comparison with existing text-to-3D methods. Unlike other methods that struggle or fail to generate reasonable results, our approach consistently achieves high-quality geometry and texture, yielding superior results. *Non-official implementation. † Generated from the online website demo.
Comparison with existing methods on generation results. We provide qualitative comparisons with existing methods in Fig. 6. We employ the same FLAME model for Latent-NeRF [43] to compute their sketch-guided loss and for Fantasy3D [74] as the initial geometry. The following observations can be made: (1) All baselines tend to be more unstable during training than ours, often resulting in diverged training processes; (2) Latent-NeRF occasionally produces plausible results due to its use of the shape prior, but its textures are inferior to ours since optimization occurs solely in the latent space; (3) Despite 3DFuse's depth control to mitigate the Janus problem, it still struggles to generate 3D consistent head avatars; (4) While Fantasy3D can generate a mesh-based 3D avatar, its geometry is heavily distorted, as its disentangled geometry optimization might be insufficient for highly detailed head avatars; (5) Although DreamFace generates realistic human face textures, it falls short in generating (i) complete heads, (ii) intricate geometry, (iii) non-human-like appearance, and (iv) composite accessories. In comparison, our method consistently yields superior results in both geometry and texture with much better consistency for the given prompt. More comparisons can be found in the supplementary material.
Figure 7: User study. Numbers are averaged over 42 responses.
Generation	Dream Fusion	Latent-NeRF	3DFuse	Fantasia3D	Ours
CLIP-R [52]	95.83	87.50	70.83	62.50	100.00
CLIP-S [24]	26.06	26.30	23.41	23.26	29.52
Editing	B3	B4	B5	Ours
CLIP-DS [18]	16.62	8.76	14.03	16.84
Table 1: Objective evaluation with CLIP-based metrics. All numbers are calculated with CLIP-L/14.
Figure 8: Analysis of prior-driven score distillation.
Figure 9: Failure cases.
4.2 Quantitative evaluations
User studies. We conducted user studies comparing with four baselines [73, 74, 65, 43]. 42 volunteers ranked them from 1 (worst) to 5 (best) individually based on three dimensions: (1) consistency with the text, (2) texture quality, and (3) geometry quality. The results, shown in Fig. 7, indicate that our method achieved the highest rank in all three aspects by large margins.
CLIP-based metrics. Following DreamFusion [54], (1) We calculate the CLIP R-Precision (CLIP-R) [52] and CLIP-Score (CLIP-S) [24] metrics, which evaluate the correlation between the generated images and the input texts, for all methods using 30 text prompts. As indicated in Tab. 1, our approach significantly outperforms the competing methods according to both metrics. This outcome provides additional evidence for the subjective superiority observed in the user studies and qualitative results. (2) We employ the CLIP Directional Similarity (CLIP-DS) [5, 18], to evaluate the editing performance. This metric measures the alignment between changes in text captions and corresponding image modifications. Specifically, we encode a pair of images (the original and edited 3D models rendered from a specific viewpoint) along with a pair of text prompts describing the original and edited subjects, e.g., "a DSLR portrait of Saul Goodman" and "a DSLR portrait of Saul Goodman dressed like a clown". We compare our approach against B3, B4, and B5 by evaluating 10 edited examples. The results, presented in Tab. 1, highlight the superiority of our editing framework according to this metric, indicating improved editing fidelity and identity preservation compared to alternatives.
4.3 Further analysis
Effectiveness of prior-driven score distillation. In Fig. 8, we conduct ablation studies to examine the impact of the proposed landmark control and textual inversion priors in our method. We demonstrate this on the coarse stage because the refinement and editing results heavily depend on this stage. The findings show that landmark control is essential for generating spatially consistent head avatars. Without it, the optimized 3D avatar faces challenges in maintaining consistent facial views, particularly for non-human-like characters. Moreover, textual inversion is shown to be another vital component in mitigating the Janus problem, specifically for the back view, as landmarks cannot exert control on the rear view. Overall, the combination of both components enables HeadSculpt to produce view-consistent avatars with high-quality geometry.
Figure 10: Analysis of identity-aware editing score distillation.
Effectiveness of IESD. In Fig. 10, we present two common biased editing scenarios produced by the baseline methods: insufficient editing and loss of identity. With Stable Diffusion, specific terms like "Saul Goodman" and "skull" exert a more substantial influence on the text embeddings compared to other terms, such as "older" and "Vincent van Gogh". B1, B2, and B3, all based on vanilla Stable Diffusion, inherit such bias in their generated 3D avatars. Although B4 does not show such bias, it faces two other issues: (1) the Janus problem reemerges due to incompatibility between vanilla InstructPix2Pix and the proposed prior-driven score distillation; (2) it struggles to maintain facial identity during editing based solely on instructions, lacking a well-defined anchor point. In contrast, B5 employs ControlNet-based InstructPix2Pix [84] with the proposed prior score distillation, resulting in more view-consistent editing. Additionally, our IESD further uses the proposed edit scale to merge two predicted scores, leading to better identity preservation and more effective editing. This approach allows our method to overcome the limitations faced by the alternative solutions, producing high-quality 3D avatars with improved fine-grained editing results.
Limitations and failure cases. While setting a new state-of-the-art, we acknowledge HeadSculpt has limitations, as the failure cases in Fig. 9 demonstrate: (1) non-deformable results hinder further extensions and applications in audio or video-driven problems; (2) generated textures are highly saturated and less realistic, especially for characters with highly detailed appearances (e.g., Freddy Krueger); (3) some inherited biases from Stable Diffusion [69] still remain, such as inaccurate and stereotypical appearances of Asian characters (e.g., Sun Wukong and Japanese Geisha); and (4) limitations inherited from InstructPix2Pix [5], such as the inability to perform large spatial manipulations (e.g., remove his nose).
We have introduced HeadSculpt, a novel pipeline for generating high-resolution 3D human avatars and performing identity-aware editing tasks through text. We proposed to utilize a prior-driven score distillation that combines a landmark-based ControlNet and view-dependent textual inversion to address the Janus problem. We also introduced identity-aware editing score distillation that preserves both the original identity information and the editing instruction. Extensive evaluations demonstrated that our HeadSculpt produces high-fidelity results under various scenarios, outperforming state-of-the-art methods significantly.
Societal impact. The advancements in geometry and texture generation for human head avatars could be deployed in many AR/VR use cases but also raises concerns about their potential malicious use. We encourage responsible research and application, fostering open and transparent practices.
Acknowledgment. This work is partially supported by Hong Kong Research Grant Council - Early Career Scheme (Grant No. 27208022) and HKU Seed Fund for Basic Research. We also thank the anonymous reviewers for their constructive suggestions.
[86] Yinglin Zheng, Hao Yang, Ting Zhang, Jianmin Bao, Dongdong Chen, Yangyu Huang, Lu Yuan, Dong Chen, Ming Zeng, and Fang Wen. General facial representation learning in a visual-linguistic manner. In CVPR, 2022. 5
[87] Yufeng Zheng, Victoria Fernández Abrevaya, Marcel C. Bühler, Xu Chen, Michael J. Black, and Otmar Hilliges. I M Avator: Implicit morphable head avatars from videos. In CVPR, 2022. 1, 3
[88] Wojciech Zielonka, Timo Bolkart, and Justus Thies. Instant volumetric head avatars. In CVPR, 2023. 3
Table 2: Hyper-parameters of HeadSculpt.
Camera setting	θ range	(20, 110)
Radius range	(1.0, 1.5)
FoV range	(30, 50)
Render setting	Resolution for coarse	(64, 64)
Resolution for fine	(512, 512)
Max num steps sampled per ray	1024
Iter interval to update extra status	16
Diffusion setting	Guidance scale	100
t range	(0.02, 0.98)
ω(t)	√αt(1 - αt)
Training setting	#Iterations for coarse	70k
#Iterations for fine	50k
Batch size	4
LR of grid encoder	1e-3
LR of NeRF MLP	1e-3
LR of si and Δvi in DMTET	1e-2
LR scheduler	constant
Warmup iterations	20k
Optimizer	Adam (0.9, 0.99)
Weight decay	0
Precision	fp16
Hardware	GPU	1 × Tesla V100 (32GB)
Training duration	1h (coarse) + 1h (fine)
A.1 Details about 3D scene models
In the coarse stage, we make use of the grid frequency encoder  $\gamma (\cdot)$  from the publicly available Stable DreamFusion [73]. This encoder maps the input  $\mathbf{x}\in \mathbb{R}^3$  to a higher-frequency dimension, yielding  $\gamma (\mathbf{x})\in \mathbb{R}^{32}$ . The MLP within our NeRF model consists of three layers with dimensions [32, 64, 64,  $3 + 1 + 3]$ . Here, the output channels '3', '1', and '3' represent the predicted normals, density value, and RGB colors, respectively. In the fine stage, we directly optimize the signed distance value  $s_i\in \mathbb{R}$ , along with a position offset  $\Delta \mathbf{v}_i\in \mathbb{R}^3$  for each vertex  $\mathbf{v}_i$ . We found that fitting  $s_i$  and  $\mathbf{v}_i$  into MLP, as done by Fantasia3D [74], often leads to diverged training.
To ensure easy reproducibility, we have included all the hyperparameters used in our experiments in Tab 2. The other hyper-parameters are set to be the default of Stable-DreamFusion [73].
A.2 Details about textual inversion
In the main paper, we discussed the collection of a tiny dataset consisting of 34 images depicting the back view of heads. This dataset was used to train a special token, <back-view>, to address the ambiguity associated with the back view of landmarks. The images in the dataset were selected to encompass a diverse range of gender, color, age, and other characteristics. A few samples from the dataset are shown in Fig. 11. While our simple selection strategy has proven effective in our specific case, we believe that a more refined collection process could further enhance the controllability of the learned <back-view> token. We use the default training recipe provided by HuggingFace Diffusers<sup>2</sup>, which took us 1 hour on a single Tesla V100 GPU.
$^{2}$ https://github.com/huggingface/diffusers/blob/main/examples/textual_inversion
Figure 11: Samples of the tiny dataset collected for learning <back-view> token.
B Further analysis
B.1 Effectiveness of textual inversion on 2D generation
To show the effectiveness of the learned <back-view> token, we conduct an analysis of its control capabilities in the context of 2D generation results. Specifically, we compare two generation results using Stable Diffusion [69], with both experiments sharing the same random seed. One experiment has the plain text prompt appended with the plain phrase "back view," while the other experiment utilizes the learned special token <back-view> in the prompt. We present a selection of randomly generated results in Fig. 12. The observations indicate that the <back-view> token effectively influences the pose of the generated heads towards the back, resulting in a distinct appearance. Remarkably, the <back-view> token demonstrates a notable generalization ability, as evidenced by the Batman case, despite not having been trained specifically on back views of Batman in the textual inversion process.
B.2 Inherent bias in 2D diffusion models
In our main paper, we discussed the motivation behind our proposed identity-aware editing score distillation (IESD), which can be attributed to two key factors. Firstly, the limitations of prompt-based editing [54, 39] are due to the inherent bias present in Stable Diffusion (SD). Secondly, while InstructPix2Pix (IP2P) [5] offers a solution by employing instruction-based editing to mitigate bias, it often results in identity loss. To further illustrate this phenomenon, we showcase the biased 2D outputs of SD and ControlNet-based IP2P in Fig. 13. Modified descriptions and instructions are utilized in these respective methods to facilitate the editing process and achieve the desired results. The results provide clear evidence of the following: (1) SD generates biased outcomes, with a tendency to underweight the "older" aspect and overweight the "skull" aspect in the modified description; (2) IP2P demonstrates the ability to edit the image successfully, but it faces challenges in preserving the identity of the avatar.
The aforementioned inherent biases are amplified in the domain of 3D generation (refer to Fig. 10 in the main paper) due to the optimization process guided by SDS loss, which tends to prioritize view consistency at the expense of sacrificing prominent features. To address this issue, our proposed IESD approach combines two types of scores: one for editing and the other for identity preservation. This allows us to strike a balance between preserving the initial appearance and achieving the desired editing outcome.
Figure 12: Analysis of the learned <back-view> on 2D image generation. For each pair of images, we present two 2D images generated with the same random seed, where the left image is conditioned on the plain text "back view" and the right image is conditioned on the <back-view> token.
Figure 13: Analysis of the inherent bias in 2D diffusion models. For each case, we display several 2D outputs of SD and IP2P, utilizing modified descriptions and instructions, respectively, with reference images from our coarse-stage NeRF model to facilitate the editing process.
Figure 14: Additional comparisons with existing methods on generation (Part 1). *Non-official.
C Additional qualitative comparisons
C.1 Comparison with existing methods on generation results
We provide more qualitative comparisons with four baseline methods [73, 43, 65, 74] in Fig. 14 and Fig. 15. These results serve to reinforce the claims made in Sec. 4.1 of the main paper, providing further evidence of the superior performance of our HeadSculpt in generating high-fidelity head avatars. These results showcase the ability of our method to capture intricate details, realistic textures, and overall visual quality, solidifying its position as a state-of-the-art solution in this task.
Notably, to provide a more immersive and comprehensive understanding of our results, we include multiple outcomes of our HeadSculpt in the form of  $360^{\circ}$  rotating videos. These videos can be accessed on https://brandonhan.uk/HeadSculpt, enabling viewers to observe the generated avatars from various angles and perspectives.
Figure 15: Additional comparisons with existing methods on generation (Part 2). *Non-official.
Figure 16: Comparisons with existing methods on editing.*Non-official.
C.2 Comparison with existing methods on editing results
Since the absence of alternative methods specifically designed for editing, we conduct additional evaluations of the editing results generated by existing methods by modifying the text prompts. Fig. 16 illustrates that bias in editing is a pervasive issue encountered by all the baselines. This bias stems from the shared SDS guidance function, which is based on a diffusion prior, despite the variations in representation and optimization methods employed by these approaches. Instead, IESD enables the guidance function to incorporate information from two complementary sources: (1) the original image gradient, which preserves identity, and (2) the editing gradient, which captures desired modifications. By considering both terms, our approach grants more explicit and direct control over the editing process compared to the conventional guidance derived solely from the input.
Figure 17: Results across random seeds (0, 1, 2).*Non-official.
C.3 Comparison with existing methods on stability
We observe that all baselines tend to have diverged training processes as they do not integrate 3D prior to the diffusion model. Taking two shape-guided prior methods (i.e., Latent-NeRF [43] and Fantasia3D [10]) as examples, we compare their generation results and ours across different random seeds. We conduct comparisons under the same default hyper-parameters and present the results in Fig. 17. We notice that prior methods need to try several runs to get the best generation while ours can achieve consistent results among different runs. Our method is thus featured with stable training, without the need for cherry-picking over many runs.