HeadSculpt: Crafting 3D Head Avatars with Text
Authors: Xiao Han
Yukang Cao
Kai Han
Xiatian Zhu
Jiankang Deng
Yi-Zhe Song
Tao Xiang
Kwan-Yee K. Wong
Venue: NeurIPS 2023 poster (2023)
Keywords: 3D generative model
head avatar
diffusion models
neural rendering
TL;DR: a versatile pipeline for generating and editing 3D head avatars with textual prompts

Abstract: Recently, text-guided 3D generative methods have made remarkable advancements in producing high-quality textures and geometry, capitalizing on the proliferation of large vision-language and image diffusion models. 
However, existing methods still struggle to create high-fidelity 3D head avatars in two aspects: 
(1) They rely mostly on a pre-trained text-to-image diffusion model whilst missing the necessary 3D awareness and head priors. 
This makes them prone to inconsistency and geometric distortions in the generated avatars. 
(2) They fall short in fine-grained editing. This is primarily due to the inherited limitations from the pre-trained 2D image diffusion models, which become more pronounced when it comes to 3D head avatars. 
In this work, we address these challenges by introducing a versatile coarse-to-fine pipeline dubbed HeadSculpt for crafting (i.e., generating and editing) 3D head avatars from textual prompts. 
Specifically, we first equip the diffusion model with 3D awareness by leveraging landmark-based control and a learned textual embedding representing the back view appearance of heads, enabling 3D-consistent head avatar generations. 
We further propose a novel identity-aware editing score distillation strategy to optimize a textured mesh with a high-resolution differentiable rendering technique. 
This enables identity preservation while following the editing instruction.
We showcase HeadSculpt's superior fidelity and editing capabilities through comprehensive experiments and comparisons with existing methods.

HeadSculpt: Crafting 3D Head Avatars with Text
Xiao Han
Yukang Cao
Kai Han Tao Xiang!.41
Xiatian Zhu Yean-Yee K. Wonga!
Jiankang Deng"
Yi-Zhe Song 1.4
'University of Surrey "iFlyTek-Sume Surrey Joint Research Centre on AI
The University of Hong Kong
IImperial College London SSurrey Institute for Soope-Ceentee Al
Abstract
Recently, text-guided 3D generative methods have made remarkable advancements in producing high-quality textures and geometry, capitalizing on the proliferation of large vision-language and image diffusion models. However, existing methods still struggle to create high-fidelity 3D head avatars in two aspects: (I) They rely mostly a pre-trained text-to-image diffusion model whilst missing the necessary 3D awareness and head priors. This makes them prone to inconsistency and geometric distortions in the generated avatars. (2) They fall short in fine-graines editing. This is primarily due to the inherited limitations from the pre-trained 2D image diffusion models, which become more pronounced when it comes 3D head avatars. In this work, we address these challenges by introducing a versatile coarse-to-fie pipeline dubbed HeadSculpt for crafting (i.e., generating and editing) 3D head avatars from textual prompts. Specifically, we first equip the diffusion model with 3D awareness by leveraging landmark-based control and learned textual embedding representing the back view appearance of heads, enabling 3D-consistent head avatar generations. We further propose a novel identity-aware editing score distillation strategy to optimize a textured mesh with high- high-resollutio differentiable rendering technique. This enables identity preservation while following the editing instruction. We showcase HeadSculpt's 's superior fidelity and editing capabilities through comprehensive experiments and comparisons with existing methods. 1
Introduction
Modeling 3D head avatars underpins wide range of emerging applications (e.g., digital telepresence, game character creation, and AR/VR). Historically, the creation of intricate and detailed 3D head avatars demanded considerable time and expertise in art and engineering. With the advent of deep learning, existing works [87, 28, 33, 72, 8, 38, 15] have shown promising results on the reconstruction of 3D human heads from monocular images videos. However, these methods remain restricted to head appearance contained in their training data which is often limited in size, resulting in the inability to generalize new appearance beyond the training data. This constraint calls for the need of more flexible and generalizable methods for
3D head modeling.
Recently, langun-language models (e.g., CLIP [55]) and diffusion models (e.g.. Stable Diffusion [69, 59]) have attracted increasing interest. These progresses have led to the emergence of text-to-3D generative models [34, 62, 44, 27] which create 3D content in self-supervised manner. Notably, DreamFusion [54] introduces a score distillation sampling (SDS) strategy that leverages a pre-trained image diffusion model to compute the noise-level loss from the textual description, unlocking the potential to optimize differentiable 3D scenes (e.g., neural radiance field [45], tetrahedral mesh [66], texture [58, 9]. point cloud [50]) with 2D diffusion prior only. Subsequent research efforts [43,
Equul contributions
Corresponding authors
Webpage: httpe://brandonhan, uk/ wwt..kk//eeeeeee tt//HeadSculpt
37th Conference on Neural Information Processing Systems (NcurIPS 2023).

---

"ron Mon
Terracolia Army
Napoleen Bonaparte
Obana with boseball cup
make him Joker
like the
turn hine into Pixur style
Batmon
Simpson
Simpsoos
of War
Taylor Swift
masqueride mask
make her sculpture
Black Panther Marsel
"I am Groot
Vincent van Gogh
'Audrey Hepòurn
make her color-restored
make her claymation
Thoofface DC
Narato Utumaki
LLo Tolstoy
Genalt in The Witcher
smiling
turn him into Minecraft
Doctor Strange
HHuk
boy with facial patatinn
Cassarr in Rise of the Planet of Apes
swimmer with goggle
make it carved out of waod
Figure I: Examples of generation and editing results obtained using the proposed HeadSculpt.
enables the creation and fine-grained editing of high-quality head avatars, featuring intricate geometry and texture, for any type of head avatar using simple descriptions instructions. Symbols indicate the following prompt prefixes: * "a head of [text]" and i "a DSLR portrait of [text]]" The captions in gray are the prompt suffixes while the blue ones the editing instructions.
65, 79, 42, 75, 40, 56, 76] improve and extend DreamFusion from various perspectives (e.g., higher resolution [39] and better geometry [10].
Considering the flexibility and versatility of natural languages, one might think that these SDS-based text-to-3D generative methods would be sufficient for generating diverse 3D avatars. However, noted that existing methods have two major drawbacks (see Fig. 6): (I) Inconsistency and geometric distortions: The 2D diffusion models used in these methods lack 3D awareness particularly regarding camera pose: without any remedy, existing text-to-3D methods inherited this limitation, leading to the multi-face "Janus" " problem the generated head avatars. (2) Fine-grained editing limitations: Although previous methods propose to edit 3D models by naively fine-tuning trained models with modified prompts [54, 39], we find that this approach is prone to biased outcomes, such as identity loss inadequate editing. This problem arises from two causes: (a) inherent bias in prompt- based editing in image diffusion models, and (b) challenges with inconsistent gradient back-propagation at separate iterations when using SDS calculated from vanilla image diffusion model.
based
In this paper, we introduce new foucuudd text-to-3D method, dubbed HeadSculpt, that supports high-fidelig generation and fine-graie fine-grained editing. Our method comprises two novel

---

components: (1) Prior-driven score distillation: We first arm the pre pre-trained image diffusion model with 3D awareness by integrating a landmark-based ControlNet [84]. Specifically, we adopt the parametric 3D head model, FLAAEE [38], prior to obtain a 2D landmark map [41, 31], which serves as an additional condition for the diffusion model, ensuring the consistency of generated head avatars across different views. Further, to remedy the front-view bias in the pre-trained diffusion model, we utilize an improved view-dependent prompt through textual inversion [17], by learning a specialized <back-view token to emphasize back views of heads and capture their unique visual details. (2) Identity-aware editing score distillation (IESD): To address the challenges of fine-grained editing for head avatars, introduce novel method called IESD. It blends two scores, one for editing and the other for identity preservation, both predicted by a ControlNet- -bsed implementation of InstructPix2Pii [5]. This approach maintains controlled editing direction that respects both the original identity and the editing instructions. To further improve the fidelity of our method, we integrate these novel components into coarse-to-fine pipeline [39], utilizing NeRF [48] as the low-resolution coarse model and DMTET [66] as the high-resolution fine model. As demonstrated in Fig. 1, our method can generate high-fidelity human-like and non-human-line like head avatars while enabling fine-grained editing, including local changes, shape/texture modifications, and style transfers.
2 Related work
Text-to-2D generation. In recent years, groundbreaking vision- vision-language technologies such as CLIP [55] and diffusion models [25, 13, 59, 68] have led to significant advancements in text-to-2D content generation [61, 57, 69, 70]. Trained
extensive 2D multimodal datasets [63, 64], they are empowered with the capability to "dream" from the prompt. Follow-up works endeavor to efficiently control the generated results [84, 85, 47], extend the diffusion model to video sequence [67, 3]. accomplish image or video editing [23, 32, 81, 5, 77, 14, 22), enhance the performance for personalized subjects [60, 17], etc. Although significant progress has been made in generating 2D content from text, carefully crafting the prompt is crucial, and obtaining the desired outcome often requires multiple attempts. The inherent randomness remains challenge, especially for editing tasks.
Text-to-3D generation. Advancements in lext-to-2D generation have paved the way for text-to-3D techniques. Early efforts [82, 27, 44, 62, 34, 29, 11] propose to optimize the 3D neural radiance field (NeRF) or vertex-based meshes by employing the CLIP language model. However, these models encounter difficulties in generating expressive 3D content, primarily because of the limitations of CLIP in comprehending natural language. Fortunately, the development of image diffusion models [69, 1] has led to the emergence of DreamFusion [54]. It proposes Score Distillation Sampling (SDS) based on a pre-trained 2D diffusion prior [61], showcasing promising generation results. Subsequent works [37] have endeavored improve DreamFusion from various aspects: Magic3D [39] proposes a coarse-to-fine pipeline for high-resolution generations; Latent-NeRF [43] includes shape guidance for more robust generation on the latent space [59]; DreamAvatar [6] leverages SMPL [4] to generate 3D human full-body avatars under controllable shapes and poses; Guide3D [7] explores the usage of multi-view generated images to create 3D human avatars; Fantasia3D [10] disentangles the geometry and texture training with DMTET [66] and PBR texture [49] as their 3D representation; 3DFuse [65] integrates depth control and semantic code sampling to stabilize the generation process. Despite notable progress, current text-to-3D generative models still face challenges in producing view-consistent 3D content, especially for intricate head avatars. This is primarily due to the absence of 3D awareness in text-to-2D diffusion models. Additionally, to the best of our knowledge, there currently no approach that specifically focuses on editing the generated 3D content, especially addressing the intricate fine-grained editing needs of head avatars.
3D head modeling and creation. Statistical mesh- models, such as FLAME [38, 15], enable the reconstruction of 3D head models from images. However, they struggle to capture fine details like hair and wrinkles. To overcome this issue, recent approaches [8, 71, 72, 51] employ Generative Adversarial Networks (GANs) [46, 20, 30] to train 3D-aware networks on 2D head datasets and produce 3D- consistent images through latent code manipulation. Furthermore, neural implicit methods [87, 16, 28, 88] introduce implicit and subject-oriente
head models based on neural rendering fields [45, 48, 2]. Recently, text-to-3D generative methods have gained traction, generating high-quality 3D head avatars from natural language using lasion-language models [55, 69]. Typically. T2P [85] predicts bone-driven parameters of head avatars via game engine under the CLIP guidance [55]. Rodin [80] proposes roll-out diffusion network to perform 3D-aware diffusion. DreamFace [83] employs
3

---

(a) Coarse Stage
(e) Prior-driven Score Distiliation (PSD)
Lundmark
Decription
Jasrste: <baskstewo
High-quality
Enharced vinar
experdent promppt
Render [iowree]
Model
Lons
Landdmmank
Contnoliet
FLAME NaRt
DSLR portrait of Goodman
(b) Fine Stage
(DNIECT)
Projected
Projected Landmark
Verg
Lanent (Stadee Diffaioo)
Reralerool bmaage
Encwder
PSD
(ligreea)
(lighrooa)
Parder
Por baseed
Meah (DMTо))
Trained Courne Nothr
Instrucshicali (Editing Only)
Refinement
Identity-aware Editing Score Distillation
Figure Overall architecture HeadSculpt. We craft high-resolution 3D head avatars coarse- to-fine manner. (a) We optimize neural field representations for the coarse model. (b) We refine or edit the model using the extracted 3D mesh and apply identity-aware editing score distillation if editing is the target. (c) The core of our pipeline is the prior-driven score distillation, which incorporates landmark control, enhanced view-dependent prompts, and an InstructPix2Pix branch.
selection strategy in the CLIP embedding space to generate coarse geometry and uses SDS [54] to optimize UV-texture. Despite producing promising results, all these methods require a large amount of data for supervised training and struggle to generalize well to non-human-like avatars. In contrast, our approach relies solely on pre-trained text-to-2) 2D models, generalizes well to out-of-domain avatars, and is capable of performing fine-grained editing tasks.
3 Methodology
HcadSculpt is a 3D-aware text-to-3D approach that utilizes a pre-trained text-to-2D Stable Diffusion model [69, 59] to generate high-resolution head avatars and perform fine-grained editing tasks. As illustrated in Fig. 2, the generation pipeline has two stages: coarse generation via the neural radiance field (NeRF) [48] and refinement/edittng using tetrahedron mesh (DMTET) [66]. Next, we will first introduce the preliminaries that form the basis of our method in Sec. 3.1. We will then discuss the key components of our approach in Sec. 3.2 and Sec, 3.3, including (1) the prior-driven score distillation process via landmark-based ControlNet [84] and textual inversion [17], and (2) identity-aware editing score distillation accomplished in the fine stage using the ControlNet-baut InstruetPix2Pix [5].
3.1 Preliminaries
Score distillation sampling. Recently, DreamFusion [54] proposed score distillation sampling (SDS) to self-optimize a text-consistent neural radiance field (NeRF) based the pre-trained text-to-2D diffusion model [61]. Due to the unavailability of the Imagen model [61] used by DreamFusion, we employ the latent diffusion model in [59] instead. Specifically, given latent feature encoded from an image v, SDS introduces random noise to to create a noisy latent variable and then uses a pre-trained denoising function e (=.:y, y, f) predict the added noise. The SDS loss is defined the difference between predicted and added noise and its gradient is given by
Oz dx Ox 00
V VoLsD ( (()) Er,c~N(0.1) wo(t (Eo (26:8.) c)
(1)
where y is the text embedding, w(t) weights the loss from noise level With the expressive text-to- 2D diffusion model and self sulf-uupeee SDS loss, we can back-propagate the gradients to optimize an implicit 3D scene s()), eliminating the need for expensive 3D dataset.
3D scene optimization. HeadSculpt explores the potential of two different 3D differentiable repre- sentations as the optimization basis for crafting 3D head avatars. Specifically, we employ NeRF [48] the coarse stage due to greater flexibility geometry deformation, while utilizing DMTET [66] in the fine stage for efficient high-resolution optimization.

---

3D prior-based NeRF. DreamAvatar [6] recently proposed robustness and controllability of the generated 3D NeRF. Given a point x inside the 3D volume, we can derive its density and color value based on prior-based density field Ø:
residual setup to enhance the
F(x, 5) = Fe (x(x)) + (x(),, 0) + (a,c),
(2)
where y(-) denotes hash-grid frequency encoder [48], and and are the density and RGB color respectively. We can derive ō from the signed distance d(x) of a given 3D shape prior (e.g., a canonical FLAME model [38] by default in our implementation):
1 a
(x) = max (0, softplus Taiinssstrrr (T(xx)) -(x) sigmoid((d(()) where a = 0.005.
To obtain a 2D RGB image from the implicit volume defined above, we employ volume rendering technique that involves casting a ray r from the 2D pixel location into the 3D scene, sampling points pe along the ray, and calculating their density and color value using F in Eq. (2):
c(r) = Σ W,C,,
W₁ = 0. II az): J
ag = 1 cl -
(2) DMTET. It discretizes deformable tetrahedral grid (Vr, T). where Vi denotes the vertices within grid T' [19, 66], to model the 3D space. Every vertex V, E Vr R possesses a signed distance value E I. along with a position offset Áv, E R2 of the vertex relative to its initial canonical coordinates. Subsequently. the underlying mesh can be extracted based on 8; with the differentiable marching tetrahedra algorithm. In addition to the geometry, we adopt the Magic3I approach [39] to construct a neural color field. This involves re-utilizing the MLP trained in the coarse NeRF stage to predict the RGB color value for each 3D point. During optimization, we render this textured surface mesh into high-resolution images using a differentiable rasterizer [36, 49].
3.2 3D-Prior-driven score distillation
Existing text-to-3D methods with SDS [54] assume that maximizing the likelihood of images rendered from various viewpoints of a scene model g()) is equivalent to maximizing the overall likelihood of g(-). This assumption can result in inconsistencies and geometric distortions [54, 65]. A notable issue the "Janus problem" characterized by multiple faces on a single object (see Fig. 6). There are two possible causes: (1) the randomness of the diffusion model which can cause inconsistencies among different views, and (2) the lack of 3D awareness in controlling the generation process, causing the model to struggle in determining the front view, back view, etc. To address these issues in generating head avatars, we integrate 3D head priors into the diffusion model.
Landmark-based ControlNet. In Section 3.1, we explain our adoption of FLAMEE [38] as the density guidance for our NeRF Nevertheless, this guidance by itself is insufficient to have a direct impact on the SDS loss. What is missing is a link between the NeRF and the diffusion model, incorporating the same head priors. Such link is key to improving the view consistency of the generated head avatars. To achieve this objective, as illustrated in Fig. 2, we propose the incorporation of 2D landmark maps as an additional condition for the diffusion model using ControlNet [84]. Specifically, we employ a ControlNet C trained on large-scale 2D face dataset [86, 12], using facial landmarks rendered from MediaPipe [41, 31] as ground-truth data. When given a randomly sampled camera pose m, we first project the vertices of the FLAME model onto the image. Following that, we select and render some of these vertices into a landmark map Pa based on some predefined vertex indexes. The landmark map will be fed into ControlNet and its output features are added to the intermediate features within the diffusion U-Net. The gradient of our SDS loss can be re-written as
0z Or VoLsns(O, g(0) (0,.)) vo()) (eo (26 y, (,C(,)) -c) Ör 0e
Enhanced view-dependent prompt via textual inversion. Although the landmark-based ControlNet can inject 3D awareness into the pre-trained diffusion models, it struggles to maintain back view head consistency. This is expected as the 2D image dataset used for training mostly contains only front or side face views. Consequently, when applied directly to back views, the model introduces ambiguity as front and back 3D landmark views can appear similar, as shown in Fig. 8. To address this issue, we propose a simple yet effective method. Our method is inspired by previous works [54, 65, 39] which found it beneficial to append view-dependent text (e.g., "front view", "side view" or "back
5

---

view") to the provided input text based on the azimuth angle of the randomly sampled camera. We extend this idea by learning a special token <back-view> to replace the plain text "back view" in order to emphasize the rear appearance of heads. This is based on the assumption that pre-trained Stable Diffusion does has the ability to "imagine" the back view of a head it has seen some during training. The main problem is that generic text embedding of "back view" is inadequate in telling the model what appearance it entails. A better embedding for "back view" is thus required. To this end, we first randomly download 34 images of the back view of human heads, without revealing any personal identities, to construct a tiny dataset D, and then we optimize the special token v (i.e., <back- (back-vie>>) to better fit the collected images, similar to the textual inversion [17]:
Ux arg min B1,,,N(0.1). [|ee cs (26 v. (2),,) t)][]
(6)
which is achieved by employing the same training scheme as the original diffusion model, while keeping Eo fixed. This constitutes reconstruction task, which we anticipate will encourage the learned embedding to capture the fine visual details of the back views of human heads. Notably, as we do not update the weights of Eg. it stays compatible with the aandmark-basee ControlNet.
3.3 ldentity-aware editing score distillation
After generating avatars, editing them to fulfill particular requirements poses an additional challenge. Previous works [54, 39] have shown promising editing results by fine-tuning a trained scene model with new target prompt. However, when applied to head avatars, these methods often suffer from identity loss or inadequate appearance modifications (see Fig. 10). This problem stems from the inherent constraint of the SDS loss, where the 3D models often sacrifice prominent features to preserve view consistency. Substituting Stable Diffusion with InstruetPix2PPx [5, 21] might seem like a simple solution, but it also faces difficulties in maintaining facial identity during editing based only on instructions, as it lacks a well-defined anchor point.
To this end, we propose identity-aware editing score distillation (IESD) to regulate the editing direction by blending two predicted scores, i.e., one for editing instruction and another for the original description. Rather than using the original InstructPix2Pix [5], we employ a ControlNet-hass InstrucdPix2Pix I [84] trained on the same dataset, ensuring compatibility with our landmark- based ControlNet C and the learned <back-vise describing the avatar to be edited and an editing instruction 9. we first input them separately into the same diffusion model equipped with two ControlNets, I and C. This allows us to obtain two predicted noises, which are then combined using a predefined hyper-paramene parameter we like classifier-free diffusion guidance (CFG) [26]:
token. Formally, given an initial textual prompt
V VeLiesD(e.g(0)) E,,.,,(,,11.. w(t) (ég b.s. -c)
8z Ōx
Ox 00
(7)
WeEp (ze) U. t, C,C(P,,),, C(P()),.( + (1-we)ed (20) v, t, ((P:),
where Par and M2 represent the 2D landmark maps and the reference images rendered in the coarse stage, both being obtained under the sampled camera pose π. The parameter we governs a trade-off between the original appearance and the desired editing, which defaults 0.6 in our experiments.
4 Experiments
We will now assess the efficacy of our HcadSculpt across different scenarios, while also conducting a comparative analysis against statc-of-the-art text-to-3D generation pipelines.
Implementation details. HeadSculpt builds upon Diffusers [78, 53]. We utilize version 1.5 of Stable Diffusion [69] and version 1.1 of ControlNet [84, 12] our implementation. In the coarse stage, we optimize our 3D model at 64 64 grid resolution, while using 512 x 512 grid resolution for the fine stage (refinement or editing). Typically, each text prompt requires approximately 7, 000 iterations for the coarse stage and 5, 000 iterations for the fine stage. It takes around 1 hour for each stage on a single Tesla V100 GPU with a default batch size of 4. We use Adam [35] optimizer with a fixed learning rate of 0.001. Additional implementation details can be found in the supplementary material.
[73] and Huggingface

---

Shape
Shape
Shape
DSLR portrait of Saul Goodman
Figure 3: Generation results with various shapes. The first row shows three randomly sampled FLAME models, while the second row presents generated results (incl. normals) using these FLAME models as initialization. All results are under the same text prompt.
surprised
disgusted
bold give beard
give sunglass
Figure 4: More specific editing results. $ Instruction prefix: make his expression as [text].
Baseline methods for generation evaluation. We compare the generation results with five baselines:
DreamFusion [73]. Latent-NeRF [43], 3DFuse [65] (improved version of SJC (791), Fantasia3D [10],
and DreamFace [83]. We not directly compare with DreamAvatar [6] as involves deformation
fields for full-body-related tasks.
Baseline methods for editing evaluation. We assess IESD's efficacy for fine-grained 3D head avatar editing by comparing it with various alternatives since no dedicated method exists for this: (B1)
the coarse stage without initialization; (B2) Initialized from the coarse stage, followed by optimization of another coarse stage with an altered description; (B3) Initialized from the coarse stage, followed by optimization of new fine stage with an altered description; (B4) Initialized from the coarse stage, followed by optimization of new fine stage with an instruction based the vanilla InstructPix2Pix [5]; (B5) Ours without edit scale (i.e., We 1). Notably, B2 represents the editing method proposed in DreamFusion [54], while B3 has similar performance as Magic3D [39], which employs three-stage editing process (i.e., Coarse Coarse Fine).
One-step optimization
4.1 Qualitative evaluations
Head avatar generation with various prompts. In Fig. 1, we show a diverse array of 3D head avatars generated by our HeadSculpt, consistently demonstrating high-quality geometry and texture across various viewpoints. Our method's versatility is emphasized by its ability to create assortment of avatars, including humans (both celebrities and ordinary individuals) as well as non-human characters like superheroes, comic/game characters, paintings, and more.
Head avatar generation with different shapes. HeadSculpt leverages shape-guided NeRF initializa- tion and mundmark-guided diffusion priors. This allows controlling geometry by varying the FLAME shape used for initialization. To demonstrate adjustability, Fig. 3 presents examples generated from diverse FLAME shapes. The results fit closely to the shape guidance, highlighting HeadSculpt's capacity for geometric variation when provided different initial shapes.
Head avatar editing with various instructions. As illustrated in Fig. and Fig. 4, HeadSculpt's adaptability is also showeased through its ability perform fine- ine-grained editing, such local changes (e.g., adding accessories, changing hairstyles, or altering expressions), shape and texture modifications, and style transfers.
Head avatar editing with different edit scales. In Fig. 5, we demonstrate the effectiveness IESD with different We values, highlighting its ability to control editing influence the reference identity.
7

---

We We We We
We
Saul Goodman
kim: into clown
Figure 5: Impact of the edit scale we IESD. It balances the preservation of the initial appearance and the extent of the desired editing, making the editing process more controllable and flexible.
DreamFusion* [75]
Latent-NeRF [4S]
3DFuse [65]
Fantasin3D
DreamFace
[83]
HeadSculpt (Ours)
DSLR portraii of Salvador Dali
head Storntrooper
DSLR portrait of female soldler, wearing helmet
DSLR portrait of young black iady with short hair, wearing a headppone
Figure 6: Comparison with existing text-to-3D methods. Unlike other methods that struggle fail to generate reasonable results, our approach consistently achieves high-quality geometry and texture, yielding superior results. * *Non-official implementation. Generated from the online website demo.
Comparison with existing methods generation results. We provide qualitative comparisons with existing methods in Fig. 6. We employ the same FLAME model for Latent-NeRF [43] to compute their sketch-guided loss and for Fantasia3D) [74] the initial geometry. The following observations can he made: (1) All baselines tend to be more unstable during training than ours, often resulting diverged training processes; (2) Latent-NeRF occasionally produces plausible results due to its use of the shape prior, but its textures are inferior to ours since optimization occurs solely in the latent space; (3) Despite 3DFuse's 's depth control mitigate the Janus problem, still struggles to generate 3D consistent head avatars; (4) While Fantasia3D can generate a mesh- 3D avatar, its geometry is heavily distorted, its disentangled geometry optimization might be insufficient for highly detailed head avatars; (5) Although DreamFace generates realistic human face textures, it falls short in generating (i) complete heads, (ii) intricate geometry, (iii) non-human-like appearance, and (iv) composite accessories. In comparison, our method consistently yields superior results in both geometry and texture with much better consistency for the given prompt. More comparisons can be found in the supplementary material.

---

DreamFusion
Dream Fusson
Latent-
NeRF
Fantasia3D
LaRent-NeRF
CLIPAR
100.00
3DFuse
23.26
Fantasia3D
HeadSculpt
Editing
B3
B4
185
Ours
(Ours)
Geometry Quality
CLIP-DS (18)
16.62
8,76
14.03
16.84
Consistency
Quality
Figure 7: User study. Numbers aver- aged over 42 responses.
Table 1: Objective evaluation with CLIP-based met- rics. All numbers are calculated with CLIP-L/14.
HeadSeulpt (Ours)
w/o Landmark Ctrl
HeadSculpt (Ours)
w/o Textual Inversion
head of Woonyy the Toy Story
of wearing a Wurkong
Freddy Krueger
head of Bumblebee in Transformers
head Mario in Mario Franchise
Japanese Geisha
remove his nose
Figure 8: Analysis of prior-driven score distillation.
Figure 9: Failure cases.
4.2 Quantitative evaluations
User studies. We conducted user studies comparing with four baselines [73, 74, 65, 43]. 42 volunteers ranked them from I (worst) to (best) individually based on three dimensions: (1) consistency with the text, (2) texture quality, and (3) geometry quality. The results, shown in Fig. 7, indicate that our method achieved the highest rank in all three aspects by large margins.
CLIP-based metrics. Following DreamFusion [54]. (1) We calculate the CLIP R- R-Precision (CLIP- R) [52] and CLIP-Score (CLIP-S) [24] metrics, which evaluate the correlation between the generated images and the input texts, for all methods using 30 text prompts. As indicated in Tab. 1, our approach significantly outperforms the competing methods according to both metrics. This outcome provides additional evidence for the subjective superiority observed in the user studies and qualitative results. (2) We employ the CLIP Directional Similarity (CLIP-DS) [5, 18], to evaluate the editing performance. This metric measures the alignment between changes in text captions and corresponding image modifications. Specifically, we encode pair images (the original and edited 3D models rendered from specific viewpoint) along with pair of text prompts describing the original and edited subjects, e.g., "a DSLR portrait of Saul Goodman" and "a DSLR portrait Saul Goodman dressed like clown". We compare approach against B3, B4, and B5 by evaluating 10 edited examples. The results, presented Tab. 1, highlight the superiority of our editing framework according to this metric, indicating improved editing fidelity and identity preservation compared to alternatives.
4.3 Further analysis
Effectiveness of prior-driven score distillation. In Fig. 8, we conduct ablation studies to examine the impact of the proposed landmark control and textual inversion priors our method. We demonstrate this on the coarse stage because the refinement and editing results heavily depend on this stage. The findings show that landmark control essential for generating spatially consistent head avatars. Without it, the optimized 3D avatar faces challenges in maintaining consistent facial views, particularly for con-human-like characters. Moreover, textual inversion shown to be another vital component in mitigating the Janus problem, specifically for the back view, as landmarks cannot exert control on the rear view. Overall, the combination of both components enables HeadSculpt produce view-consistent avatars with high-quality geometry.

---

One-stage
we
HeadSculpt
Modified description: DSLR portrait of Hooldrr) Saud Goodman
Instruction: make him older
Modilied description: portitt skull of Vincent van Gogh
Instruction: turn his face into a skall
Figure 10: Analysis of identity-aware editing score distillation.
Effectiveness of IESD. Fig. 10, we present two common biased editing scenarios produced by the baseline methods: insufficient editing and loss of identity. With Stable Diffusion, specific terms like "Saul Goodman" and "skull" exert a more substantial influence on the text embeddings compared other terms, such "older" and "Vincent van Gogh". B1, B2, and B3, all based vanilla Stable Diffusion, inherit such bias in their generated 3D avatars. Although B4 does not show such bias, faces two other issues: (1) the Janus problem reemerges due to incompatibility between vanilla InstructPix2Pix and the proposed prior-driven score distillation; (2) it struggles to maintain facial identity during editing based solely on instructions, lacking well-define anchor point. In contrast, B5 employs ControlNet base InstructPix2Pix [84] with the proposed prior score distillation, resulting more view-consistent editing. Additionally, our IESD further uses the proposed edit scale to merge two predicted scores, leading to better identity preservation and more effective editing. This approach allows our method to overcome the limitations faced by the alternative solutions, producing high-quality 3D avatars with improved fine-grained editing results.
acknowledge HeadSculpt has limitations, as the failure cases in Fig. 9 demonstrate: (1) non-deformable results hinder further extensions and applications in audio or video-driven problems; (2) generated textures are highly saturated and less realistic, especially for characters with highly detailed appearances (e.g., Freddy Krueger); (3) some inherited biases from Stable Diffusion [69] still remain, such as inaccurate and stereotypical appearances of Asian characters (e.g., Sun Wukong and Japanese Geisha); and (4) limitations inherited from InstructPix2Pix [5], such as the inability to perform large spatial manipulations (e.g., remove his nose).
Limitations and f'ilure cases. While setting new Kate-of-the-art
5 Conclusions
We have introduced HeadSculpt, a novel pipeline for generating high-resolution 3D human avatars and performing identity-aware editing tasks through text. We proposed to utilize prior-driven score distillation that combines a landmark- based ControlNet and view-dependent textual inversion to address the Janus problem. We also introduced identity-awart editing score distillation that preserves both the original identity information and the editing instruction. Extensive evaluations demonstrated that HeadSculpt produces high-fidelity results under various scenarios, outperforming state-of- the-art methods significantly.
The advancements in geometry and texture generation for human head avatars could be deployed in many AR/VR use cases but also raises concerns about their potential malicious use. We encourage responsible research and application, fostering open and transparent practices.
Societal impact.
Acknowledgment. This work is partially supported by Hong Kong Research Grant Council Early Career Scheme (Grant No. 27208022) and HKU Seed Fund for Basic Research, We also thank the anonymous reviewers for their constructive suggestions.
10

---

References
[1] Yogesh Balaji, Scungjun Nah, Xun Huang, Arash Vahdat, Jiaming Song, Karsten Kreis, Miika Aittala, Timo Aila, Samuli Laine, Bryan Catanzaro, al. edifh: Text-to-image diffusion models with ensemble of expert denoisers. arXiv preprint arXiv:2211.011
2022. 3
[2] Jonathan T Barron, Ben Mildenhall, Dor Verbin, Pratul Srinivasan, and Peter Hedman. Mip-nerf 360: Unbounded anti-aliased neural radiance fields. In CVPR, 2022.
[3] Andreas Blattmann, Robin Rombach, Huan Ling, Tim Dockhor, Scung Wook Kim, Sanja Fidler, and Karsten Kreis, Align your latents: High-resolution video synthesis with latent diffusion models. In CVPR, 2023. 3
[4] Fedorica Bogo, Angjoo Kanazawa, Christoph Lassner, Peter Gehler, Javier Romero, and Michael Black. Keep smpl: Automatic estimation of 3d human pose and shape from single image. In ECCV, 2016.
[5] Tim Brooks, Aleksander Holynski, and Alexei A Efros. Instractpix2pix: Learning follow image editing instructions. CVPR, 2023. 7, 9, 10, 17
[6] Yukang Cao, Yan-Pei Cao, Kai Han, Ying Shan, and Kwan- Yee K Wong. Dreamavatar: Text-and-shan shape guided 3d human avatar generation via diffusion models. arXiv preprint arXiv:2304.009) 2023. 1, 3, 5,
[7] Yukang Cao, Yan-Pei Cao, Kai Han, Ying Shan, and Kwan- Yee K Wong. Guide3d: Create 3d avatars from text and image guidance. arXiv preprint aiv::22300. 09705, 2023. 3
[8] Frie R. Chan, Connor 7. Lin, Matthew A. Chan, Koki Nagano, Boxiao Pan, Shalini De Mello, Orazio Gallo, Leonidas Guibas, Jonathan Tremblay, Sameh Khamis, Tero Karras, and Gordon Wetzstein. Efficient geometry-aware 3D generative adversarial networks. CVPR, 2022. 1.
[9] Dave Zhenyu Chen, Yawar Siddiqui, Hsin- Ying Lee, Sergey Tulyakov, and Matthias NieBner. Text2tex: Text-driven texture synthesis via diffusion models. ICCV, 2023. 1
[10] Rui Chen, Yongwei Chen, Ningxin Jiao, and Kui Jia. Fantasia3d; Disentangling geometry and appearance for high-quality text-to-3d content creation. ICCV, 2023. 2, 7,
[11] Yongwei Chen, Rui Chen, Jiabao Lei, Yabin Zhang, and Kui Jia. Tango: Text-driven photorealistic and robust 3d stylization via lighting decomposition. In NeurlPS, 2023. 3
[12] CrucibleAL.
ControlNetMeddiaPippeaaa.
http::///unggnngaacc..o//rouuuu
2023. 5,
[13] Prafulla Dhariwal and Alexander Nichol. Diffusion models beat gans on image synthesis. In NeurlPS, 2020. 3
[14] Patrick Esser, Johnathan Chiu, Parmida Atighehchian, Jonathan Granskog, and Anastasis Germanidis,
Structure and content-guided video synthesis with diffusion models. arXiv preprint arXiv:2302.03011 2023. 3
[15] Yao Feng, Haiwen Feng, Michael J Black, and Timo Bolkart. Learning an animatable detailed 3d face model from in-the- wild images. ACM Transactions Graphics (TOG), 2021. 3.3
[16] Guy Gafni, Justus Thies, Michael Zollhofer, and Matthias Nießner. Dynamic neural radiance fields for monocular 4d facial avatar reconstruction. In CVPR, 2021. 3
[17] Rinon Gal, Yuval Alaluf, Yuval Atzmon, Or Patashnik, Amit H Bermano, Gal Chechik, and Daniel Cohen-Or. An image is worth one word: Personalizing text-to-image generation using textual inversion. In ICLR, 2023. 3. 4.6
[18] Rinon Gal, Or Patashnik, Haggai Maron, Amit H Bermano, Gal Chechik, and Daniel Cohen-Or Or. Stylegan- nada: Clip-guided domain adaptation of image generators. ACM Transactions Graphics (TOG), 2022.
[19] Jun Gao, Wenzheng Chen, Tommy Xiang, Alec Jacobson, Morgan McGuire, and Sanja Fidler. Learning deformable tetrahedral meshes for 3d reconstruction. In NeurlPS, 2020. 5
[20] Ian Goodfellow. Jean Pouget-Abadie, Mehdi Mirza, Bing Xu, David Warde-Farley, Sherjil Ozair, Aaron Courville, and Yoshua Bengio. Generative adversarial networks. Communications of the ACM, 2020. 3
[21] Ayaan Haque, Matthew Tancik, Alexei A Efros, Aleksander Holynski, and Angjoo Kanazawa. Instruct- nerf2nerf: Editing 3d scenes with instructions. In ICCV, 2023. 6

---

[22] Amir Hertz, Kfir Aberman, and Daniel Cohen-Or. Delta denoising score. arXiv preprint arXI1:2304.000900 2023. 3
[23] Amir Hertz, Ron Mokady, Jay Tenenhaum, Kfir Aberman, Yael Pritch, and Daniel Coh Prompt-to- prompt image editing with cross attention control. arXiv preprint arXiv:2208.0/6666 2022. 3
[24] Jack Hessel, Ari Holtzman, Maxwell Forbes, Ronan Le Bras, and Yejin Choi. Clipscore: A reference-free evaluation metric for image captioning. In EMNLP, 2021. 9
[25] Jonathan Ho, Ajay Jain, and Pieter Abbeel. Denoising diffusson probabilistic models. In NewrlPS, 2020. 3
[26] Jonathan Ho and Tim Salimans. Classifier-free diffusion guidanee. In NeurlPS Workshop, 2021. 6
[27] Fangzhou Hong. Mingyuan Zhang, Liang Pan. Zhongang Cai, Lei Yang, and Ziwei Liu. Avatarclip: zero-shot text-driven generation and animation of 3d avatars. ACM Transactions Graphics (TOG), 2022. 3
[28] Yang Hong, Bo Peng. Haiyao Xiao, Ligang Liu, and Juyong Zhang. Headnerf: A real-time nerf-based parametric head model. CVPR, 2022. 1,3
[29] Ajay Jain, Ben Mildenhall, Jonathan T Barron, Picter Abbeel, and Ben Poole. Zero-shot text-guided object generation with dream fields. In CVPR, 2022.
[30] Tero Karras, Samuli Laine, and Timo Aila. A style-baser generator architecture for generative adversarial networks. In CVPR, 2019.
[31] Yury Kartynnik, Artsiom Ablavatski, Ivan Grishchenko, and Matthias Grundmann. Real-time facial surface geometry from monocular video on mobile gpus. In CVPR workshopx, 2019. 3,5
[32] Bahjat Kawar, Shiran Zada, Oran Lang, Omer Tov, Huiwen Chang, Tali Dekel, Inbar Mosseri, and Michal Trani. Imagic: Text-based real image editing with diffusion models. In CVPR, 2023. 3
[33] Taras Khakhulin, Vanessa Sklyarova, Victor Lempitsky, and Zakharov, Realistic one-shot mesh-based head avatars. In ECCV, 2022.
[34] Nasir Mohammad Khalid, Tianhao Xie, Eugene Belilovsky, and Popa Tiberiu. Clip-mesh: Generating textured meshes from text using pretrained image-text models. In SIGGRAPH Asia, 2022. 1.3
[35] Diederik Kingma and Jimmy Ba. Adam: A method for stochastic optimization. In ICLR, 2015.
[36] Samuli Laine, Janne Hellsten, Tero Karras, Ycongho Scol, Jaakko Lehtinen, and Timo Aila. Modular primitives for high-performance differentiable rendcring. ACM Transactions on Graphics (TOG), 2020. 5
[37] Chenghao Li, Chaoning Zhang, Atish Waghwase, Lik- Hang Lee, Francois Rameau, Yang Yang. Sung- Ho Bae, and Choong Seon Hong. Generative ai meets 3d: A survey on text-to-3d in aige era. arXiv preprint arXiv:2305.00 20111633... 2023. 3
[38] Tianye Li, Timo Bolkart, Michael Black. Hao Li, and Javier Romero. Learning a model of facial shape and expression from 4d scans. ACM Transactions on Graphics (TOG), 2017. 1,
[39] Chen-Hsuan Lin, Jun Gao, Luming Tang, Towaki Takikawa, Xiaohui Zeng, Xun Huang, Karsten Kreis, Sanja Fidler, Ming-Yo Yu Liu, and Tsung- Yi Lin. Magic3d: High-resolution text-to-3d content creation. In CVPR, 2023. 2, 3, 5, 6, 7, 17
[40] Ruoshi Liu, Rundi Wu, Basile Van Hoorick, Pavel Tokmakov, Sergey Zakharov, and Carl Vondrick. 1-to-1-to-3: Zero-shot one image to 3d object. In ICCV, 2023. 2
[41] Camillo Lugaresi. Jiuqiang Tang, Hadon Nash, Chris McClanahan, Esha Uboweja, Michael Hays, Fan Zhang, Chuo-Ling Chang, Ming Yong. Juhyun Lee, et al. Mediapipe: A framework for perceiving and processing reality. In CVPR workshops, 2019. 3,5
[42] Luke Melas-Kyriazi, Iro Laina, Christian Rupprecht, and Andrea Vedaldi. Realfusion: 360deg reconstruc- tion of any object from a single image. In CVPR, 2023. 2
[43] Gal Metzer, Elad Richardson, Or Patashnik, Raja Giryes, and Daniel Cohen-Or. Latent-nerf for shape- guided generation of 3d shapes and textures. In CVPR, 2023. 3, 7, 8, 9, 19, 20, 21, 22
[44] Oscar Michel, Roj Bar-On, Richard Liu, Sagie Benaim, and Rana Hanocka. Texi2mesh: Text-driver neural stylization for meshes. In CVPR, 2022. 1,3
12

---

[45] Ben Mildenhall, Pratul P Srinivasan, Matthew Tancik, Jonathan T Barron, Ravi Ramamoorthi, and Ren Ng. Nerf: Representing scenes neural radiance fields for view synthesis. In ECCV, 2020. 1,3
[46] Mehdi Mirza and Simon Osindero. Conditional generative adversarial nets. arXiv preprint arXiv: 2014. 3
[47] Chong Mou, Xintao Wang. Liangbin Xie, Jian Zhang, Zhongang Qi. Ying Shan, and Xiaohu Qie. T2i- adapter: Learning adapters to dig out more controllable ability for text-to-image diffusion models. arXiv preprint arXiv:2302.000 0845503...8.,..... 2023. 3
[48] Thomas Müller, Alex Evans, Christoph Schied, and Alexander Keller. Instant neural graphics primitives with multiresolution hash encoding. ACM Transactions on Graphics (TOG), 2022. 3, 4,4
[49] Jacob Munkberg, Jon Hasselgren, Tianchang Shen, Jun Gao, Wenzheng Chen, Alex Evans, Thomas Müller, and Sanja Fidler. Extracting triangular 3d models, materials, and lighting from images. CVPR, 2022. 3,
[50] Alex Nichol, Heewoo Jun, Prafulla Dhariwal, Pamela Mishkin, and Mark Chen. Point-e: A system for generating 3d point clouds from complex prompts. arXiv preprint orXiv:2212.08720 2022.
[51] Roy Or-EI, Xuan Luo, Mengyi Shan, Eli Shechtman, Jeong Joon Park, and Ira Cemelmacher-Shlizemmee Stylesdf: High-resolution 3d-consistent image and geometry generation. In CVPR, 2022. 3
[52] Dong Huk Park, Samaneh Azadi, Xihui Liu, Trevor Darrell, and Anna Rohrbach. Benchmark for compositional text-to-image synthesis. In NeurlPS Datasets and Benchmarks Track (Round 1), 2021. 9
[53] Adam Paszke, Sam Gross, Francisco Massa, Adam Lerer, James Bradbury, Gregory Chanan, Trevor Killeen, Zeming Lin, Natalia Gimelshein, Luca Antiga, et al. Pytorch: An imperative style, high-performance deep learning library. In NeurIPS, 2019. 6
[54] Ben Poole, Ajay Jain, Jonathan T. Barron, and Ben Mildenhall. Dreamfusion: Text-to-3d using 2d diffusion. In ICLR, 2022. 1, 2, 4, 6, 7, 17
[55] Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, et al. Learning transferable visual models from natural language supervision. In /CML, 2021. 1,3
[56] Amit Raj. Srinivas Kaza, Ben Poole, Michael Niemeyer, Nataniel Ruiz, Ben Mildenhall, Shiran Zada, Kfir Aberman, Michael Rubinstein, Jonathan Barron, et al. Dreambooth3d: Subject-driven text-to-3d generation. arXiv preprint arXiv: arXiv:2303. 13508, 2023. 2
[57] Aditya Ramesh, Prafulla Dhariwal, Alex Nichol, Casey Chu. and Mark Chen. Hierarchical text-conditional image generation with clip latents. arXiv preprint ar 2rXiv:2204.06122 2022. 3
[58] Elad Richardson, Gal Metzer, Yuval Alaluf, Raja Giryes, and Daniel Cohen-Or. Texture: Text- Text-guided texturing of 3d shapes. arXiv preprint wrXiv:2302.0172.1 2023.
[59] Robin Rombach, Andreas Blattmann, Dominik Lorenz, Patrick Esser, and Bjöm Ommer. High-resolution image synthesis with latent diffusion models. In CVPR, 2022. 1,
[60]
Nataniel Ruiz, Yuanzhen Li, Varun Jampani, Yael Pritch, Michael Rubinstein, and Kfir Aberman. Dream- booth: Fine tuning text-to-image diffusion models for subject-driven generation. In CVPR, 2023. 3
[61] Chitwan Saharia, William Chan, Saurabh Saxena, Lala Li, Jay Whang, Emily Denton, Seyed Kamyar Seyed Ghasemipour, Burcu Karagol Ayan, S Sara Mahdavi, Rapha Gontijo Lopes, et al. Photorealistic text-to- image diffusion models with deep language understanding. arXiv preprint arXiv:2205. 1487, 2022. 3,
[62] Aditya Sanghi, Hang Chu, Joseph G Lambourne, Ye Wang, Chin-Yi Cheng, Marco Fumero, and Ka- mal Rahimi Malekshan. Clip-forge: Towards zero-shot text-to-shape generation. In CVPR, 2022. 1,
[63] Christoph Schuhmann, Romain Beaumont, Richard Vencu, Cade Gordon, Ross Wightman, Mehdi Cherti, Theo Coombes, Aarush Katta, Clayton Mullis, Mitchell Wortsman, et al. Laion-Sb: An open large-scale dataset for training next generation image-text models. In NeurIPS, 2022. 3
[64] Christoph Schuhmann, Richard Vencu, Romain Beaumont, Robert Kaezmarezyk, Clayton Mullis, Aarush Katta, Theo Coombes. Jenia Jitsev, and Aran Komatsuzaki. Laion-400m: Open dataset of clip- dip-filtered 400 million image-text text pairs, arXiv preprint arx0v:21110021114, 2021. 3
13

---

[65] Junyoung Seo, Wooseok Jang, Min-Seop Kwak, Jaehoon Ko, Hyeonsu Kim, Junho Kim, Jin-Hwa Kim. Jiyoung Lee, and Seungryong Kim. Let 2d diffuusion model know 3d-consistency for robust text-to-3d generation. arXiv preprint urXiv:2303.07937. 2023. 2, 7, 8, 9, 19,
[66] Tianchang Shen, Jun Gao, Kangxue Yin, Ming- Yu Liu, and Sanja Fidler. Deep marching tetrahedra: a hybrid representation for high-resolution 3d shape synthesis. In NeurIPS, 2021. 1, 1,3,4, 4,
[67] Uriel Singer, Adam Polyak, Thomas Hayes, Xi Yin, Jie An, Songyang Zhang, Qiyuan Hu, Harry Yang, Oron Ashual, Oran Gafni, et al. Make-a-video: Texl-to-video generation without text-video data. In ICLR, 2023. 3
[68] Jiaming Song. Chenlin Meng, and Stefano Ermon. Denoising diffusion implicit models. In ICLR, 2021. 3
[69] Stability.AI. Stable diffuuson. 2022. 1, 3,3,4 4, 6, 10, 17
[70] Stability.AI. Stability Al releases Deeplloyd IF, powerful text-to-image model that smartly integrate text into images.
ai/blog/ ai/blog/deeppppfff.eoooooo heepfloyd-ae -eett--fffft-tt-ee- to- image-modeeellle model, 2023. 3
[71] Jingxiang Sun, Xuan Wang, Yichun Shi, Lizhen Wang, Jue Wang. and Yebin Liu. Ide-3d: Interactive disentangled editing for high-resolution 3d-aware portrait synthesis. ACM Transactions on Graphics (TOG), 2022. 3
[72] Jingxiang Sun, Xuan Wang, Lizhen Wang, Xiaoyu Li, Yong Zhang, Hongwen Zhang, and Yebin Liu. Next3d: Generative neural texture rasterization for 3d-aware head avatars. In CVPR, 2023. 1.3
[73] Jiaxiang Tang. cabbledrraamfusinn Text-to-3d with stable-
crable-dreamtie drw.mfusionnn 2022. 6, 7, 8, 9, 16, 19, 20, 21
[74] Jaaxian Tang. uanffsiaadd.nooffiiial. https://github. http///aaaaseeey//ttttaaaaaaaaa unoffiiciiii
2023. 8, 16, 19, 20, 21, 22
[75] Junshu Tang, Tengfei Wang, Bo Zhang, Ting Zhang, Ran Yi, Lizhuang Ma, and Dong Chen. Make-it-3d: High-fidelity 3d creation from a single image with diffusion prior. ICCV, 2023. 2
[76] Christina Tsalicoglou, Fabian Manhardt, Alessio Tonioni, Michael Niemeyer, and Federico Tombari. Textmesh: Generation of realistic 3d meshes from text prompts. arXiv preprint arXiv:2304. 12439, 2023. 2
[77] Dani Valevski, Matan Kalman, Yossi Matias, and Yaniv Leviathan. Unitune: Text-driven image editing by fine tuning an image generation model on single image. arXiv preprint arXiv:22 2022.
[78] Patrick von Platen, Suraj Patil, Anton Lozhkov, Pedro Cuenca, Nathan I Lambert, Kashif Rasul, Mishig Davaadorj. and Thomas Wolf. Diffusers: State-of-the-an diffusion models. https://gitee com/
2022.
[79] Haochen Wang, Xiaodan Du, Jiahao Li, Raymond A Ych, and Greg Shakhnarovich. Score jacobian chaining: Lifting pretrained 2d diffusion models for 3d generation. CVPR, 2023. 2,
[80] Tengfei Wang, Bo Zhang, Ting Zhang, Shuyang Gu, Jianmin Bao, Tadas Baltrusaitis, Jingjing Shen, Dong Chen, Fang Wen, Qifeng Chen, et al. Rodin: A generative model for sculpting 3d digital avatars using diffusion. In CVPR, 2023. 3
[81] Jay Zhangjie Wa, Yixiao Ge, Xintao Wang. Stan Weixian Lei, Yuchao Gu, Wynne Hsu, Ying Shan, Xiaohu Qie, and Mike Zheng Shou. Tune-a-video: One-shot tuning of image diffusion models for text-to-video generation. In ICCV, 2023. 3
[82] Jiale Xu, Xintao Wang, Weihao Cheng, Yan-Pei Cao, Ying Shan, Xiaohu Qie, and Shenghua Gao. Dream3d: Zero-shot text-to-3d synthesis using 3d shape prior and text-to-image diffusion models. In CVPR, 2023.
[83] Longwen Zhang, Qiwei Qiu, Hongyang Lin, Qixuan Zhang, Cheng Shi, Wei Yang. Ye Shi, Sibei Yang. Lan Xu, and Jingyi Yu. Dreamface: Progressive generation of animatable 3d faces under text guidance. arXiv preprint 23xiv:2304.03117, 2023. 3, 7,8
[84] Lvmin Zhang and Maneesh Agrawala. Adding conditional control to text-to-image difflusion models. arXiv preprint arXii:2302.05544 2023. 3, 5, 6, 10
[85] Rui Zhao, Wei Li, Zhipeng Hu, Lincheng Li, Zhengxia Zou, Zhenwei Shi, and Changjie Fan. Zero-shot ear-tt--arrmmeter translation for game character auto-creation. In CVPR, 2023. 3
14

---

[86] Yinglin Zheng. Hao Yang. Ting Zhang. Jianmin Bao, Dongdong Chen, Yangyu Huang, Lu Yuan, Dong Chen, Ming Zeng, and Fang Wen. General facial representation learning visual-linguiite manner. In CVPR, 2022.
[87] Yufeng Zheng. Victoria Fernández Abrevaya, Marcel C. Bühler, Xu Chen, Michael J. Black, and Otmar Hilliges. 1M Avatar: Implicit morphable head avatars from videos. In CVPR, 2022. 1,
[88] Wojeiech Zielonka, Timo Bolkart, and Justus Thies. Instant volumetric head avatars, In CVPR, 2023. 3
15

---

Table Hyper-paramelers HeadSculpt.
range
(20, 110)
Camera setting
Radius range
(1.0, 1.5)
FoV range
(30, 50)
Resolution coarse Resolution for fine
(64, 64)
(512, 512)
Render setting
Max num steps sampled per ray Tter interval to update extra status
1024
16
Guidance scale range w(f)
100
Diffusion setting
(0.02, 0.98)
Var(1 ox)
#Tterations for coarse
70k
#lterations for fine Batch size
50k
LR of grid encoder
lc-3
of NeRF MLP
Training setting
LR of s, and Av; in DMTET
le-2
LR scheduler
constant
Warmup iterations
20k
Optimizer
Adam (0.9, 0.99)
Weight decay
Precision
fp16
GPU
x Tesla V100 (32GB)
Hardware
Training duration
Ih (coarse) Ih (fine)
A Implementation details
A.1 Details about 3D scene models
In the coarse stage, we make use grid frequency encoder 7(-) from the publicly available Stable DreamFusion [73]. This encoder maps the input R3 to ((x) E R.2 The MLP within our NeRF model consists three layers with dimensions [32, 64, 64, 3+1+3J. Here, the output channels '3', '1', and '3' represent the predicted normals, density value, and RGB colors, respectively. In the fine stage, we directly optimize the signed distance value 8; E R, along with position offset Av; R for each vertex v,. We found that fitting and into MLP, as done by Fantasia3D [74], often leads to diverged training.
dimension, yielding
To ensure easy reproducibility, have included all the hyperparameters used in our experiments in Tab 2. The other hyper-parameters are set to be the default of GableeDreemmuusonn [73].
A.2 Details about textual inversion
In the main paper, we discussed the collection of tiny dataset consisting of 34 images depicting the back view of heads. This dataset was used train special token, <back-view>, to address the ambiguity associated with the back view of landmarks. The images in the dataset were selected to encompass diverse range of gender, color, age, and other characteristics. dataset are shown in Fig. 11. While our simple selection strategy has proven effective in our specific case, we believe that a more refined collection process could further enhance the controllability of the learned <back-viet token. We use the default training recipe provided by HuggingFace Diffusers 2, which took us hour on a single Tesla V100 GPU.
few samples from the
16

---

Figure 11: Samples of the tiny dataset collected for learning <back-view> token.
B Further analysis
B.1 Effectiveness of textual inversion on 2D generation
To show the effectiveness of the learned <back-vies> view> token, we conduct an analysis of its control capabilities in the context 2D generation results. Specifically, we compare two generation results using Stable Diffusion [69], with both experiments sharing the same random seed. One experiment has the plain text prompt appended with the plain phrase "back view," while the other experiment utilizes the learned special token <back-view> in the prompt. We present selection of randomly generated results in Fig. 12. The observations indicate that the <back-view> token effectively influences the pose of the generated heads towards the back, resulting in distinct appearance. Remarkably, the <back-views token demonstrates a notable generalization ability, as evidenced by the Batman case, despite not having been trained specifically on back views of Batman in the textual inversion process.
B.2 Inherent bias 2D diffusion models
In our main paper, we discussed the motivation behind our proposed identity editing score distillation (IESD), which can be attributed to two key factors. Firstly, the limitations of prompt- based editing [54, 39] are due to the inherent bias present in Stable Diffusion (SD). Secondly, while InstructPix2Pix (IP2P) [S] offers a solution by employing instruction-ban editing to mitigate bias, often results in identity loss. To further illustrate this phenomenon, we showcase the biased 2D outputs of SD and ControlNet-based IP2P in Fig. 13. Modified descriptions and instructions are utilized in these respective methods facilitate the editing process and achieve the desired results. The results provide clear evidence of the following: (1) SD generates biased outcomes, with a tendency to underweight the "older" aspect and overweight the "skull" aspect in the modified description; (2) IP2P demonstrates the ability to edit the image successfully, but it faces challenges in preserving the identity of the avatar.
The aforementioned inherent biases are amplified in the domain of 3D generation (refer to Fig. 10 in the main paper) due to the optimization process guided by SDS loss, which tends to prioritize view consistency at the expense of sacrificing prominent features. To address this issue, our proposed IESD approach combines two types of scores: one for editing and the other for identity preservation. This allows us to strike a balance between preserving the initial appearance and achieving the desired editing outcome.
17

---

"back view"
<back-Viest
"back view"
w/ <back-vae
w/ "back view"
<back-viex
seed: 413
seed: 16772
seed: 40805
IBED OPTRD
a DSLR portrait of Obama
seed:
DSLR portroit of Hillory Clinton seed: 19656
seed: 2367
seed: 62156
O Opa
DSLR portritt of boy with facial painting
seed: $3236
seed: 62424
seed: 72649
a DSLR portnoit of Batman
Figure 12: Analysis of the learned <back-vin -ve>>>>i 2D image generation. For each pair of images, we present two 2D images generated with the same random seed, where the left image is conditioned on the plain text "back view" and the right image conditioned on the Öback-view> token.
Map
Reference Image
InstructPix2Pix
seed: 19056
sced: 72854
seed: 50233
seed: 64136
Modified description: DSLR portrait of [[oldor] Soul Goodman
Instruction: make him older
seed:
seed:
88141
Modified description: PSStR-p skull of Vincent van Gogh
Instruction: turn his face into skall
Figure 13: Analysis of the inherent bias 2D diffusion models. For each case, we display several 2D outputs of SD and IP2P, utilizing modified descriptions and instructions, respectively, with reference images from our coarse-stage NeRF model to facilitate the editing process.
18

---

DreumFusion' [3
[43]
3DFuse [65]
Fantasia3D [74]
HeadSculpt
DSLR portrait of Batman
DSLR portrait Black. Panther Marvel
a DSLR portrait of Two-face in DC
DSLR portrait Strange
a head of Terracotto Army
Figure 14: Additional comparisons with existing methods generation (Part 1). *Non-official.
C Additional qualitative comparisons
C.1 Comparison with existing methods on generation results
We provide more qualitative comparisons with four baseline methods [73, 43, 65, 74] in Fig. 14 and Fig. 15. These results serve to reinforee the claims made in Sec. 4.1 of the main paper. providing further evidence of the superior performance of our HeadSculpt in generating high-fidelity head avatars. These results showease the ability of our method capture intricate details, realistic textures, and overall visual quality, solidifying its position
state-of-the-an solution in this task.
Notably, provide a more immersive and comprehensive understanding of our results, we include multiple outcomes of our HeadSculpt in the form of 360 rotating videos. These videos can be accessed on https: //bbaadonnnnaannn.nm uk Ht/HeddSSuupt,, enabling viewers observe the generated avatars from various angles and perspectives.
19

---

DreamFusion' [7]
Latent-NeRF [43]
3DFuse [65]
Fantasia3D [74]
HeadSculpt (Ours)
head Simpson Simpsons
a head of Naruto Uzumaki
DSLR portrait Napoleon Bonaparte
a DSLR portrait of Leo Tolstoy
DSLR portrait Audrey Hepburn
13
DSLR portrait of Obama with a bavehall cap
DSLR portrait Taylor Swift
Figure 15: Additional comparisons with existing methods on generation (Part 2). *Non-official.
20

---

DreamFusion' [73]
[43]
[74]
HeadSculpt (Ours)
DSLR portrait of Saul Goodman
DSLR portruit +folder) Saul Goodman
make him older
DSLR portrait of Vincent van Gogh
ASLR-poriraii skull Vincent Gogh
skull
Figure 16: Comparisons with existing methods on editing."Non-olld -Non-official.
C.2 Comparison with existing methods on editing results
Since the absence of alternative methods specifically designed for editing, we conduct additional evaluations of the editing results generated by existing methods by modifying the text prompts. Fig. 16 illustrates that bias in editing is pervasive issue encountered by all the baselines. This bias stems from the shared SDS guidance function, which based on diffusion prior, despite the variations in representation and optimization methods employed by these approaches. Instead, IESD enables the guidance function to incorporate information from two complementary sources: (1) the original image gradient, which preserves identity, and (2) the editing gradient, which captures desired modifications. By considering both terms, our approach grants more explicit and direct control over the editing process compared to the conventional guidance derived solely from the input.
21

---

[43]
Fantasia3D [74]
HeadSculpt
a head of ant-man in Marvel
Figure 17: Results across random seeds (0, 2). Nono-offiii
C,3
Comparison with existing methods stability
We observe that all baselines tend to have diverged training processes as they not integrate 3D prior to the diffusion model. Taking two shape-guided prior methods (i.e., Latent-NeRF [43] and Fantasia3D [10]) as examples, we compare their generation results and ours across different random seeds. We conduct comparisons under the same default dyper-parameteer and present the results in Fig. 17. We notice that prior methods need to try several runs to get the best generation while ours can achieve consistent results among different runs. Our method is thus featured with stable training, without the need for therry-pickingg over many runs.
22


Reviews:
[{"review_id": "c6LouKKvdL", "reviewer": "Reviewer_ooKH", "summary": "Authors proposed a solution for generating human heads based on text description. The technology is based on popularised concepts of control signal for pre-trained diffusion model and static 3D model to guarantee consistent head. Moreover, authors presents the concept of the “back” of the head with a textual inversion. The qualitative and quantitative evaluation shows the superiority of the method over existing competitors.\n", "questions": "Most of the following questions can improve the manuscript:\n- Why do you use  DMTet for discretization? The motivation is missing at line 144+\n- In the paragraph near L155 you are elaborating the source of the existing issues. Do you think that main problem is an ambiguity and with rendering each representation for uniformly sample cameras will help to disambiguate the gradients during denoising? \n- How does the back-view concept is so important if you introduce the landmark based controlNet and the FLAME structure? Can you elaborate why it is not enough to use just the full head with landmarks? The Figure 6 slightly contradicts original motivation of the textual inversion concept for the “back”.\n- Could you explain in more details the limitation from InstructPix2Pix (L305)?\n", "limitations": "- All faces only in a neutral position that is not always correlate with the textual description (e.g. neutral clown) \n- The overall image quality is biased to the unrealistic colours\n- Most of the ideas will work only for the presented setup of the heads", "rating": 7, "confidence": 4, "soundness": 4, "presentation": 4, "contribution": 3, "strengths": "- there are plenty of interesting ideas how to do specific solution including domain knowledge: 1) statistic 3d model (e.g. FLAME), 2) improved score distillation with additional special token, 3) editing directions\n- Authors demonstrate a wide set of experiments to evaluate their method including small use study\n", "weaknesses": "There is a huge limitation of the method compared to the baselines\n- The user study can be biased - small set of people, without description of the task and based on the figure 4 it is extremely easy to select the introduced method.\n- Images are not photorealistic, furthermore the results have an disdvantage from the volumetric rendering (small noise is still visible on all counters)\n- Results are biased in many directions: huge emotion bias despite the neutral emotions (Figure 1), colour issue for the shadows and unrealistic light baked to the texture.\n", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "# 2. Response to reviewer `#ooKH`\nWe thank reviewer `#ooKH` for recognizing that our solution is \"interesting\" and our qualitative and quantitative evaluation \"shows superiority\".\nWe address the proposed questions as follows - **we regularly refer to the general response above and the provided one-page pdf with figures:**\n\n## 2.1. Details about the user study\nWe conducted the user studies in the form of a questionnaire supported by Google Forms. Each volunteer will be presented with 20 randomly selected generated results with rotating videos, with the description to be:\n\n```You will be presented with 20 sets of results with each generated by 5 text-to-3D methods. Your task is to evaluate and rank the results based on three dimensions: consistency with the provided prompt, texture quality, and geometry quality. For each set of results, please assign a score of 5 to the best candidate and 1 to the worst, with intermediate scores for the others.```\n\n### 2.1.1. Updated results with additional volunteers\n\n>The user study can be biased - small set of people\n\nTo reduce the potential bias in the user study, we have now expanded the evaluation to include additionally 22 volunteers, bringing the total number of participants to 42. The updated user study statistics are presented in Fig. 8,  indicating that our method consistently achieves the highest ranks.\n\n### 2.1.2. Additional quantitaive evaluations\n\nTo present more comprehensive evaluations, we have also now conducted a quantitative evaluation using CLIP-R and CLIP-Score as the objective metrics. The results, provide further quantitative evidence that supports the subjective user study's findings. Please refer to Sec. 1.4 of the above general response for more details.\n\n## 2.2. Noises on the textures\n  \nPlease refer to Sec. 1.3 of the above general response.\n\n## 2.3. Biased results in emotion and color\n  \nPlease refer to Sec. 1.2 of the above general response.\n\n## 2.4. Motivation of DMTet for discretization\n   \nWe opted for a mesh-based approach to achieve enhanced resolution in the fine stage. Yet, when explicit meshes are directly derived from implicit representations like NeRF using marching cubes, they may yield lower-quality surfaces, especially for the boundary. Introducing DMTet, a differentiable mesh representation, into the optimization pipeline offers the potential to refine the geometry extracted from NeRF. We will elaborate this consideration in the forthcoming revised edition.\n\n## 2.5. Source of the existing issues\n\nThe main reason for the Janus problem is the absence of 3D prior in the diffusion model, since it is trained on 2D images without camera pose conditioning.\nBetter sampling camera poses might solve this problem to some extent, but a relatively large batch size is needed to sample as many poses as possible to better alleviate the ambiguity (e.g., Fantasia3D uses 24 as batch size over 8 Nvidia RTX 3090 GPUs to guarantee a satisfactory result). \nInstead, we focus on solving the core reasons for the Janus problem by proposing to integrate the 3D head prior to the pre-trained diffusion model via the projected facial landmarks. This proposed solution offers improved training efficiency and is more user-friendly.\nWe will further explain this in the forthcoming revised edition.\n\n## 2.6. Importance of $\\texttt{\\<back-view\\>}$ concept\n\nThanks for this question. Because the pre-trained diffusion model only supports 2D information conditions, we must project 3D landmarks to a 2D landmark map. However, the projection brings ambiguity over front and back views. Concretely, since the 2D image dataset used for training landmark ControlNet mostly contains only front or side face views, the model tends to generate a front-view face (i.e., a dataset bias), given such an ambiguous 2D landmark map. The proposed $\\texttt{\\<back-view\\>}$ concept is designed exactly to resolve this bias. \n\n>  Can you elaborate why it is not enough to use just the full head with landmarks?\n\nUsing full-head landmarks and considering their self-occlusion might be a possible solution, but this would require more robust full-head landmark registration (hard to collect), along with the need for retraining the ControlNet on extra data. Instead, as a cheaper solution, we leverage available facial landmarks and ControlNet capabilities in this work. We will explore more elegant landmark usage in future work.\n\n>The Figure 6 slightly contradicts the original motivation of the textual inversion concept for the “back”.\n\nFor the comment regarding Fig. 6, we intend to show that landmark control is relatively more important than the $\\texttt{\\<back-view\\>}$ concept.\nWithout landmarks, it will be hard to distinguish different views and thus tend to generate always biased front views even with the help of $\\texttt{\\<back-view\\>}$ concept.\nIn summary, landmarks should play a more central role in conveying 3D pose information to the 2D diffusion process, whilst the $\\texttt{\\<back-view\\>}$ concept is also indispensable for alleviating the ambiguity brought by the projection process. We will further clarify this in the revised manuscript. \n\n## 2.7. Limitation from InstructPix2Pix\n\nAs mentioned in the original paper, InstructPix2Pix is limited by the visual quality of the Prompt2Prompt-generated dataset. Specifically, it struggles with viewpoint changes, makes excessive image alterations, fails to isolate objects in cases, and cannot easily reorganize or swap objects. These limitations carry on our pipeline since we directly use InstructPix2Pix as is.\nWe showcase an unsatisfactory editing example in Fig. 14. Importantly, our framework can easily incorporate any improvements made in the InstructPix2Pix and its follow-up works, e.g., MagicBrush[1].\n\n[1] MagicBrush: A Manually Annotated Dataset for Instruction-Guided Image Editing."}, {"review_id": "xw9OsmAJaK", "reviewer": "Reviewer_ztub", "summary": "The paper proposes a novel method for performing text-to-3D generation for specifically human (or humanoid) heads. These generated heads can be further edited and refined using more fine-grained detailed text prompts while still preserving the identity of the generated asset. This is accomplished with two main contributions: using a landmark-aware score distillation loss in order to ensure that the generated heads roughly align with a template head mesh (FLAME), and introducing a loss function which balances text-based editing with a new description with the original description used to generate the asset. The generated results fix a number of artifacts with existing text-to-3D generation methods specifically for heads. For example, the generated heads do not suffer from the multi-face problems, and the strong prior given by a parametric head model ensures that the generated results are geometrically correct. Additionally, the results can be edited while still maintaining the identity of the original generated results. This is shown in comparison to a number of baseline methods, which are outperformed.\n\nAFTER REBUTTAL:\n\nI have read the rebuttal and appreciate the detailed response. I appreciate the additional quantitative evaluations, as I believe this was a limitation of the method that has been addressed. However, I still think the quality is a bit limited, but this is the case for most text-to-3D methods. The additional comparisons weakness has been addressed as well. However, I don't see justification for the the more stable training - I would suggest removing this claim without verifying it significantly, while Fig 11 is nice, I am not sure this is enough to make an entire claim based on.", "questions": "I do not have additional questions brought up (see weaknesses for evaluations questions). The paper is exceptionally clear in describing the method and contributions.", "limitations": "The paper adequately addresses the limitations of the proposed method. The first limitation, which mentions that the results are “non-deformable” is insightful as this was a question which is natural to ask considering that the FLAME template is a deformable model for heads. Additionally, the generated results still are not completely photorealistic, such as those representations which have been fit from captured data. Some additional failure cases of the method would be interesting to see in order to understand better what the limitations are of the method and which piece would be the biggest bottleneck for someone who wanted to integrate this work into their application.", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 4, "contribution": 3, "strengths": "In my opinion, the strengths of the paper are that:\n1. The presentation of the paper is extremely high quality. The introduction and related work sections are comprehensive, motivate the problem well, and clearly delineate the contributions provided in the paper. The methods section is very clearly described and is simple to follow for those familiar with the field of generative 3D. I view this as very important because it is much easier to glean information from the paper, and future work in text-to-3D (potentially for avatar generation) would be much more likely to build off of this method.\n2. The contributions are clearly stated, and to the best of my knowledge they are novel, and they seem like they could be relatively significant. \n    - The idea of integrating ControlNet together with a score distillation loss makes intuitive sense for gaining better control over generated 3D assets, and I have not, to the best of my knowledge, seen it proposed elsewhere. Maintaining control over the structures which are generated from text-to-3D, including ensuring they have higher quality and less noisy geometry, is a very important problem in text-to-3D generation and I view this solution as potentially having a large impact on those working in this field.\n    - While balancing losses between an editing prompt and original prompt seems simple, it does not seem to be used in other text-to-3D methods and seems like it could be useful for text-based editing of already fitted representations in other applications.\n3. I find the ablation study to be high quality, and ablate all of the parts of the method which are relevant: introducing ControlNet into the SDS loss, the representation chosen and coarse-to-fine optimization, and the hyperparameters for balancing the editing losses. These are all clearly ablated, along with other smaller contributions such as textual inversion for generating the back of the head.", "weaknesses": "In my opinion, there is only one main weakness with the paper, which is that the evaluations and comparisons to baseline methods do not seem complete.\n1. There are no quantitative comparisons. While the user study is insightful, methods like DreamFusion proposed the CLIP-R metric. Is there a reason why this was not used in this paper? Does the metric do a poor job at quantifying what it is trying to measure? Additionally, some quantitative results for evaluating the editing quality would be useful: CLIP-R (or something like it, if CLIP-R is not useful) can be used to ensure that the identity is still preserved despite editing with an additional prompt (and perhaps also show that the rendered images are pushed closer to a desired fine-grained editing prompt).\n2. The baseline methods are not evaluated entirely fairly. For example, I don’t see any of the baseline methods evaluated for editing quality. What happens if one of the generated examples (ex: those from Fig 4) go through a fine-grained text-based edit? How much will the identity be degraded or the edit not be done sufficiently? I think this is extremely important to show because without it, I don’t know what is the state-of-the-art that is currently being improved on and thus don’t know if the proposed method is even better.\n3. It is mentioned that the optimization of the proposed method is “more stable” (L244-245) than baseline methods. However, this claim is not justified anywhere. This would be an extremely important contribution for text-to-3D (where optimization is notoriously brittle), so some quantification of this would be very impactful if it could be shown. For example, ranges of hyperparameters trained with which still lead to a good solution, or error bars on generated results, or failure cases.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "# 3. Response to reviewer `#ztub`\nWe really appreciate the feedback provided by reviewer `#ztub`. \nThanks for finding our presentation clear and our contributions novel and regarding our method as a fundamental basis that future work could \"build upon\".\nBelow we address the proposed concerns about  evaluations and comparisons  - **we regularly refer to the general response above and the provided one-page pdf with figures:**\n\n## 3.1. More quantitative evaluation\nPlease refer to Sec. 1.4 of the above general response.\n\n## 3.2. Baseline methods evaluated for editing quality\nAs suggested, we have now provided several editing results produced by the baselines via prompt modification. As shown in Fig. 13, bias in editing is a common problem that all baselines suffer. This is because although variation in the representation and optimization methods, these methods share the same guidance function (i.e., $\\mathcal{L}_{\\mathrm{SDS}}$) of diffusion prior, where the bias comes from. In this paper, we propose IESD, which allows the guidance function to incorporate information from two complementary sources: 1) the original image gradient that preserves identity and 2) the editing gradient that captures desired modifications. By factoring both terms, our IESD enables more explicit and direct control over the editing process compared to the conventional guidance derived from the input alone. \nWe will further clarify this in the revised version.\n\n## 3.3. Stability comparisons\n\n> It is mentioned that the optimization of the proposed method is “more stable” (L244-245) than baseline methods. However, this claim is not justified anywhere.\n\nWe observed that all baselines tend to have diverged training processes as they do not integrate 3D prior to the diffusion model. Taking two shape-guided prior methods (i.e., Latent-NeRF and Fantasia3D) as examples, we compare their generation results and ours across different random seeds. We conduct comparisons under the same default hyper-parameters and present the results in Fig. 11. We can observe that prior methods need to try several runs to get the best generation while ours can achieve consistent results among different runs. Our method is thus featured with stable training, without the need for cherry-picking over many runs.\n\n> For example, ranges of hyperparameters trained with which still lead to a good solution\n\nAll the experiments, for all different methods, presented in the submitted manuscript and rebuttal were conducted under the same default hyper-parameters.\n\n## 3.4. Failure cases and bottlenecks\n\nThanks. We have now given several failure cases in Fig. 14. As discussed in the limitation part of the manuscript, our method is not perfect, for example, 1) it inherits the bias from the diffusion model, e.g., it cannot correctly generate characters in the eastern culture like \"Sun Wukong\"; 2) it can not handle highly-detailed textures due to the mode-seeking property of $\\mathcal{L}_{\\mathrm{SDS}}$, e.g., \"Freddy Krueger\".\n\n> which mentions that the results are “non-deformable” is insightful as this was a question which is natural to ask considering that the FLAME template is a deformable model for heads.\n\nPlease refer to Sec. 1.1 of the above general response.\n\n> the generated results still are not completely photorealistic\n\nPlease refer to Sec. 1.2 of the above general response."}, {"review_id": "55ZTB0kB4O", "reviewer": "Reviewer_thuE", "summary": "Existing text-driven 3D generative models could have many problems, such as geometric artifacts (e.g., Janus problem),  and visual inconsistency.  This work focuses on 3D head avatar generation, which utilizes FLAME model to incorporate human geometric priors into generation, hence resolving those problems in 3D generation.  The proposed framework utilizes a coarse-to-fine framework, and it first generates a coarse face via the neural radiance field (NeRF) and then performs refinement/editing using tetrahedron mesh (DMTet).  As a result, the proposed method could generate diverse 3D human faces, and it also enables identify-aware editing with the help of both ControlNet and InstructPix2Pix.", "questions": "Besides the questions listed in the weakness, I would like to ask:\n(1) Is the generated head avatar capable of performing various expressions?  Or is it possible to perform different expressions by using simple post processes?\n\n(2) Is there a clear definition of the so-called identity-awareness of the editing process? Merely blending styles appears to be an unsatisfactory solution.\n\n(3) Is it possible to generate/edit human head avatars with different shapes?", "limitations": "Yes.", "rating": 6, "confidence": 5, "soundness": 3, "presentation": 3, "contribution": 2, "strengths": "(1) This work proposes a text-guided 3D generation, specifically for human heads.  It could successfully generate various human heads by using or modifying many existing methods, such as FLAME, ControlNet, Instruct-Pix2Pix, and DMTet.\n\n(2) This method also studies identity-aware editing by defining a trade-off between the original appearance and the desired editing.", "weaknesses": "The proposed text-guided 3D generation method focuses on human heads, but it seems like combining 3D head priors with the existing general 3D generation or style-editing methods.  A few problems associated with 3D human heads have not been deeply studied, and please refer to the following detailed weaknesses:\n\n(1) This method effectively addresses the limitations encountered in current text-guided 3D generation approaches through the integration of 3D head priors with the FLAME model. By leveraging the FLAME model for 3D head generation, it is expected that this method could facilitate intricate expression variations.  However, the current generated head avatars seem to be static only, and the manuscript did not provide many expressions except for the neutral expression.  Enabling expressions is important for 3D head avatars, and a few examples could be found in the FLAME model, ICT-FaceKit, and the very relevant work DreamFace.\n\n(2) The identity-aware editing has not been well-justified.  Based on my understanding, the current method simply mixes the style of the original appearance and the desired editing to generate the facial texture based on the ControlNet-based InstructPix2Pix.  I am wondering if it is possible to do a few specific editing of human heads, such as changing the hairstyle, wearing different glasses, and perhaps a few other geometric adjustments (e.g., getting fatter or getting thinner).", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "# 4. Response to reviewer `#thuE`\nWe are grateful the reviewer `#thuE` took the time to thoroughly review our work and found that our method \"effectively addresses the limitations encountered in current text-guided 3D generation approaches\".\nWe provide our responses regarding the raised concerns as follows  - **we regularly refer to the general response above and the provided one-page pdf with figures:**\n\n## 4.1. Expression variations\n\n> the current generated head avatars seem to be static only, and the manuscript did not provide many expressions except for the neutral \n\nWe acknowledge our current method does not enable direct animation, as discussed in the limitations. This is the sacrifice of increasing generalization ability, where we didn't optimize in the pre-trained parametric space of FLAME while only using it as the density initialization of NeRF. However, expression edits are possible using IESD, as Fig. 9 shows. Despite lacking direct animation, our edited expressions demonstrate a reasonably consistent appearance and identity.\nPlease refer to Sec. 1.1 of the above general response for more details.\n\n## 4.2. Specific editing\n\n> I am wondering if it is possible to do a few specific editing of human heads, such as changing the hairstyle, wearing different glasses\n\nIn Fig. 10, we have now provided several additional editing results as suggested, including hairstyle change, beard change, and adding sunglass. It is evident that our method can achieve satisfactory editing results.\n\n>  and perhaps a few other geometric adjustments (e.g., getting fatter or getting thinner)\n\n> Is it possible to generate/edit human head avatars with different shapes?\n\nWe found it is generally difficult to do geometric adjustments via editing since it is challenging and ambiguous to describe a desired geometry via text prompts alone. However, thanks to our NeRF initialization from the FLAME model and corresponding landmark control in the diffusion prior, our method supports geometric variation in generation by changing the FLAME model. We only used the canonical FLAME model by default in the submitted manuscript. To demonstrate our method's ability for geometric adjustment, we have further provided several examples with different FLAME models as the initialization, as shown in Fig. 12. We will add this result in the revision.\n\n## 4.3. Definition of IESD\n\n> Is there a clear definition of the so-called identity awareness of the editing process?\n\nWe define identity awareness as the ability to make desired modifications to an avatar's appearance based on editing instructions while preserving key facial features and attributes that represent the avatar's core identity and preventing undesired changes unrelated to the edit.\nAs introduced in L202-213, the proposed IESD is a variant of SDS, characterized by blending two different scores for noise prediction. \n\n> Based on my understanding, the current method simply mixes the style of the original appearance and the desired editing to generate the facial texture based on the ControlNet-based InstructPix2Pix.\n\nWe note that it is more than a style mix of the original appearance and the desired editing, because the gradient predicted by InstructPix2Pix for editing is dominant in the desired editing area, so it's capable of local region editing, as shown in Fig. 10."}, {"review_id": "RN8DKG951d", "reviewer": "Reviewer_Gcqm", "summary": "This work proposes a text-to-avatar creation pipeline, building upon dreamfusion and magic3d. To alleviate the geometric ambiguity, the authors replace the vanilla stable diffusion with a landmark-conditioned diffusion model finetuned with controlnet. They also use textual inversion to obtain specific token for back view. Extensive experiments show that the proposed method achieves SOTA performance.", "questions": "- The method used DMTet in the second stage, which is a mesh-based representation. But the mesh has difficulties in representing hair, beards, and so on. How do you solve these parts?\n- How do you train landmark-conditioned controlnet? training data?", "limitations": "Authors have claimed the limitations.", "rating": 6, "confidence": 5, "soundness": 3, "presentation": 3, "contribution": 2, "strengths": "- The generation results are impressive and compelling. As shown in Fig. 1, the method can generate avatars in different domains including photo-realistic, anime.\n- The writing is easy to follow.", "weaknesses": "- The generated avatars lack details and suffer some artifacts. The results in Fig. 1 suffer obvious noises especially when zoomed in. Hair and clothing lack details.\n- How about the generation diversity? Given the same text prompt, can the model generate multiple avatars that align with the text?\n- One key aspect of avatars is animation. How could the generated avatars be driven?\n- Fig. 2 is a bit confusing for me. It would be better to clean it.\n- As in Fig. 3, the editing performs well for frontal view, but fails on side and back views.\n- Are results in the main paper cherry-picked? Could you show some randomly sampled results? and more visualization for the geometry?", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "# 5. Response to reviewer `#Gcqm`\nWe thank reviewer `#Gcqm` for agreeing with our motivation and acknowledging the results as \"impressive and compelling\". \nBelow we respond to the doubts put forward by the reviewer - **we regularly refer to the general response above and the provided one-page pdf with figures:**\n\n## 5.1. Noises and artifacts in generated avatars\n\nPlease refer to Sec. 1.3 of the above general response.\n\n## 5.2. Generation diversity\n\nIndeed, like existing text-to-3d methods (e.g., DreamFusion), our method also does not yield large amounts of diversity across random seeds (as shown in Fig. 11). This is likely due to the mode-seeking property of $\\mathcal{L}_{\\mathrm{SDS}}$ combined with that at high noise levels, the\nsmoothed densities may not have many distinct modes. Understanding the interplay between guidance strength, diversity, and loss functions remains an important open direction for future research.\n\n## 5.3. Avatar animation\n\nPlease refer to Sec. 1.1 of the above general response.\n\n## 5.4. Cleaning Fig. 2\nThanks for pointing this out. We will reorganize the layout and clean it in the revised version.\n\n## 5.5. Editing performs badly on side and back views\n\nThis is because the front view of a head represents almost the whole information for editing, the editing results of side and back views sometimes are inferior to those of front views. However, we respectfully argue that this is not always the case, e.g., most of the editing results as shown in Fig. 1 are similarly satisfactory among all views.\n\n## 5.6. Randomly sampled results\n\n> Are results in the main paper cherry-picked? Could you show some randomly sampled results?\n\nThe results shown in the submitted manuscript are not cherry-picked. Please also check the newly added results across random seeds in Fig. 11, where we don't find obvious geometry and appearance differences among different runs. \n\n> and more visualization for the geometry?\n\nFor geometry visualization, we have now provided additional normal-rendered images in Fig. 12. Please find more examples in the supplementary file. \n\n## 5.7. Difficulties of mesh in representing hair and beards\n\nIn this work, we didn't design specific modules to handle the issue of hair and beard (out of our current focus and scope). We also note that, compared with pure NeRF-based methods, using mesh as a second-stage representation can offer sharper and more structural hair and beard, since the initialization from NeRF's density can be further trained to improve at a much higher resolution. To further improve the quality of hair and beard, one possible solution for future work is building high-quality hair/beard templates, which can be disentangled from the whole optimization process. \n\n## 5.8. Details about landmark-conditioned ControlNet\n\nAs mentioned in L167, instead of training ControlNet by ourselves, we took an off-the-shelf version[1] trained on LAION-Face[2] dataset, including 50M diverse face images and the corresponding face landmarks predicted by Google MediaPipe.\n\n[1] ControlNetMediaPipeFace. https://huggingface.co/CrucibleAI/ControlNetMediaPipeFace.\n\n[2] General facial representation learning in a visual-linguistic manner.\n"}]

Author rebuttal:
We deeply appreciate the reviewers' thoughtful feedback recognizing the presentation, novelty, and performance of our method.
The reviewers suggested additional experiments and illustrations to highlight strengths, clarify limitations, and illustrate future directions. We are pleased to have conducted all of these valuable recommended experiments, as outlined in the general and per-reviewer responses. Next, we would like to first provide some general responses to address common questions. **To avoid any confusion with the figures in the original submission, we have numbered the new figures in the attached document starting from Fig. 8 onwards.**

# 1. General response

## 1.1. Avatar expressions [Reviewers `#ztub`, `#thuE`, `#Gcqm`]
In its current form, our method does not support direct animation of the generated avatars, despite enabling barely satisfactory facial expression changes through editing (as shown in Fig. 9). This limitation arises because we initialize the NeRF density using only the FLAME model, without preserving the mesh structure and point correspondence. Our motivation for this design choice is to maximize the generalization capability across the entire pipeline. In principle, we could employ any head parametric model (e.g. NPHM[1]) as the 3D representation and optimize an avatar within its parametric space (i.e. shape and expression parameters). However, by constraining the optimization to a pre-trained parametric space in this manner, it would restrict the ability to generalize to out-of-distribution data (e.g. non-human-like avatars). We believe there is an inherent trade-off between generalization ability and structural consistency that warrants further investigation. As future work, we will explore solutions for the best compromise between these two competing objectives.

## 1.2. Biased results [Reviewers `#ooKH`, `#ztub`]
Indeed, the current pipeline exhibits biases in aspects like color, appearance, and emotion inherited from two sources: 1) the diffusion prior, which tends to memorize training data during generation[2]; and 2) the mode-seeking nature of the SDS loss, which requires large CFG values to boost fidelity at the cost of high saturation and unrealistic colors. As demonstrated in the manuscript, these biases plague all existing text-to-3D methods but are not specific to our model, though IESD can partially mitigate them during editing. By our knowledge and experience, some concurrent works might be able to alleviate these biases: a stronger diffusion model[4], image reconstruction loss[5], and more advanced score distillation[6].

## 1.3. Noises and artifacts in generated avatars [Reviewers `#ooKH`, `#Gcqm`]
We do acknowledge that the current results are far from perfect even though our results have already outperformed prior alternatives by a large margin. This is still an unsolved problem with a heavy need for further investigation and innovation. More specifically, this is due to the challenging mode-seeking nature of this zero-shot task, where real albedo textures are not available for model optimization. Compared with NeRF-based methods rendered at a much smaller $64\times 64$ resolution, the noises in our setting become more apparent because we are using mesh-based representation at an $8\times$ higher resolution at the fine stage. We hypothesize that further fine-tuning the diffusion prior on a large dataset with manually collected albedo maps[3] could help mitigate these issues, which we leave for future work. While not yet perfect, we believe our method demonstrates strong zero-shot performance and establishes a promising direction for high-fidelity generative avatar modeling without albedo supervision. We argue this already makes a significant contribution to the community. 

## 1.4. More quantitative evaluation [Reviewers `#ooKH`, `#ztub`]

### 1.4.1 Generation evaluation
We appreciate the insightful suggestion. Following the suggestion, we computed CLIP-R (CLIP-L/14) and CLIP-Score metrics for all methods using the 30 text prompts from the generative process. As shown in Tab. 1, our approach substantially outperforms the competitors on both metrics. This aligns with and further substantiates the subjective superiority demonstrated in the user study. We will include this evaluation in the revised version.

### 1.4.2 Editing evaluation
We didn't provide a quantitative evaluation for editing because we consider editing to be fundamentally a subjective task and also there is no standard protocol to measure its performance. Regardless, as suggested, we have now adopted CLIP Directional Score as described in InstructPix2Pix and StyleGAN-NADA to measure the editing performance.
It measures how much the change in text captions agrees with the change in the images. Concretely, for the directional score, we encode a pair of images (the original and edited 3D models, rendered at a given viewpoint), as well as a pair of text prompts that describe the original and edited scenes, e.g., "a DSLR portrait of Saul Goodman" and "a  DSLR portrait of Saul Goodman dressed like a clown". We compare our method with B3-B5 mentioned in Fig. 7 due to the absence of existing methods, with the scores calculated on 10 editing results shown in Tab. 1. Please note that this metric might also inherit the bias from CLIP. We will keep finding better evaluation metrics and improve this evaluation in the future.

[1] Learning Neural Parametric Head Models.

[2] Extracting Training Data from Diffusion Models.

[3] Dreamface: Progressive generation of animatable 3d faces under text guidance.

[4] DeepFloyd IF. https://stability.ai/blog/deepfloyd-if-text-to-image-model.

[5] TextMesh: Generation of Realistic 3D Meshes From Text Prompts.

[6] ProlificDreamer: High-Fidelity and Diverse Text-to-3D Generation with Variational Score Distillation.


Meta-review: All the reviewers are quite positive about this work. The rebuttal has addressed the concerns raised by the reviewer. This paper has presented a new approach to 3D avatar generation from pretrained text-to-image generation models. The generated 3D avatars are impressive, and the presented ideas are interesting and inspiring. The authors are recommended to incorporate experimental results in the rebuttal to the main paper or the supplement. 

The authors are also recommended to take the suggested actions provided by the ethics reviewer.
Decision: Accept (poster)