ID-to-3D: Expressive ID-guided 3D Heads via Score Distillation Sampling
Authors: Francesca Babiloni
Alexandros Lattas
Jiankang Deng
Stefanos Zafeiriou
Venue: NeurIPS 2024 poster (2024)
Keywords: Score Distillation Sampling
3D Heads
ID-conditioning
ID-to-3D
Primary area: machine_vision

Abstract: We propose ID-to-3D, a method to generate identity- and text-guided 3D human heads with disentangled expressions, starting from even a single casually captured ‘in-the-wild’ image of a subject. The foundation of our approach is anchored in compositionality, alongside the use of task-specific 2D diffusion models as priors for optimization. First, we extend a foundational model with a lightweight expression-aware and ID-aware architecture, and create 2D priors for geometric and texture generation, via fine-tuning only 0.2% of its available training parameters. Then, we jointly leverage a neural parametric representation for the expression of each subject and a multi-stage generation of highly detailed geometry and albedo texture. This combination of strong face identity embeddings and our neural representation enables accurate reconstruction of not only facial features but also accessories and hair, and can be meshed to provide render-ready assets for gaming and telepresence. Our results achieve an unprecedented level of id-consistent and high-quality texture and geometry generation, generalizing to a ‘world’ of unseen 3D identities, without relying on large 3D captured datasets of human assets.

ID-to-3D: Expressive DDguuded 3D Heads via Score Distillation Sampling
Francesca Babiloni, Alexandros Lattas, Jiankang Deng, Stefanos Zafeiriou Imperial College London, UK
https://ddookd..iiiub..i
Abstract
We propose ID-to-3D, method to generate identity- and text-guided 3D human heads with disentangled expressions, starting from even single casually captured in-the-wild image of a subject. The foundation of our approach is anchored in compositionality, alongside the use of task-specific 2D diffusion models as priors for optimization. First, we extend foundational model with lightweight expression-awe aware and ID-aware architecture, and create 2D priors for geometry and texture generation, via fine-tuning only 0.2% of available training parameters. Then, we jointly leverage a neural parametric representation for the expressions of each subject and a multi-stage generation of highly detailed geometry and albedo texture. This combination of strong face identity embeddings and neural representation enables accurate reconstruction of not only facial features but also accessories and hair and can be meshed to provide render-ready assets for gaming and telepresence. Our results achieve an unprecedented level of identity-consiseent and high-quality texture and geometry generation, generalizing to "world" of unseen 3D identities, without relying on large 3D captured datasets human assets.
Figure 1: ID-to-3D 3D leverages identity conditioning and score distillation sampling on large diffusion models, achieving high-quality 3D human asset generation from "in-the-wild" images, without training on large scanned datasets. From left to right: a) renderings, b) input images, c) normals.
Introduction
The remarkable ability of humans to discern facial characteristics and emotional cues in others makes the development of high-quality 3D head avatars a challenging yet foundational task for diverse array of emerging applications, including digital telepresence, game character generation, and the creation of virtual and augmented reality experiences. However, the acquisition of such 3D human assets remains daunting task, that requires either manual work typically performed by graphic artists, expensive and laborious scanning. High-quality 3D facial scanning, achieved initially
38th Conference on Neural Information Processing Systems (NcurIPS 2024).

---

% DELR foce pontrnal"
Sqyeas
Yeup 94
Yexp
505
9/
L
Pc
Yeex
3/3
Keep
I00 ...0D0
Yia
100 0300
Bontbe
SIG, 2
8F.
%
Generation
Generotion
Kep
K
Training
Inference
Figure 2: (Left) Overall pipeline. ID-to-3D generates expressive 3D head avatars via ArcFace yia and textual Yee conditioning. uses as prior geometry-oriented Φy and albedo oriented @a pretrained models. Training) The training phase uses SDS to optimize 3D geometry Vgg. texture vaa and a set of expressions latent codes kesp- It also leverages random lighting and random expression conditioning Bexp- Inference) At deployment time, ID-to-3D extracts high-quality identity- expressive 3D meshes. (Right) ID-consistent expressive 3D heads generated by our method. ID-to-3D creates 3D assets that support relighting, ID-consistent editing, and physical simulation.
by hardware with controlled polarized illumination 15 51
has been simplified tremendously
using simpler devices, color-space space methods, and inverse rendering [25] 39 3]. Nevertheless, the requirements for expert knowledge, hardware, and computational time do not allow wide adoption or mass usage.
Consequently, statistical modeling and the rise of deep learning investigated techniques reconstruct 3D face assets from casually captured images of subject in arbitrary poses, lighting conditions, and occlusions (referred to as "in-the-wild"). 3D Morphable Models (3DMMs) and Generative Adversarial Networks (GANs) can be used to model facial geometry 4 19
56 24 while GANs and diffusion models have achieved state-of-the-ard modeling and reconstruction of facial appearance 40
22
92 60
However, common denominator in all the above is the requirement for vast datasets of scanned facial shapes and appearances, in order to avoid limited generalization, ethnicity under-repressentaion and oversmoothed geometries, requiring up to 10, 000 scans for 3DMMs training [6] facial appearance modeling [22]. Moreover, the requirement for aligned data in 3D, and also in UV texture maps, limits the utilized area to the facial region, and introduces registration errors and additional costs. Our proposed method bypasses these significant issues by utilizing large 2D generative models, pre-trained on vast and casy-to-acquire 2D images, and using only small set of rasterized 3D data for finetuning, while achieving state-of-the-art 3D head generation.
More recently, the of Score Distillation Sampling (SDS) (3 and large- scale diffusion models
explored the automatic generation of 3D content thanks to text 85
10,
45
31
92
37
26 or
79 80) Despite promising results, leveraging 2D priors to generate (1) Large text-to-image foundational models are usually trained to generate realistic RGB images, where geometry, texture, and lighting cannot be individually segregated. This narrow "rendering knowledge" in the 2D guidance compromises the 3D fidelity, texture quality, and consistency of the generated assets, leading unrealistic geometries and distorted textures with recognizable artifacts such as Janus problems, incorrect proportions, oversaturated albedo, and mismatching texture and geometry details. (2) It is difficult precisely
image prompts [66, 91 64
realistic 3D head avatars remains challenging:
control facial attributes solely using typical prompting methods [13 the gramularity to single out the specificity of subject's identity and facial expression that might be lengthy and complex to convey in natural language. Moreover, methods that leverage image prompts lack the capability to capture features that represent the facial identities of a subject independently of contextual scene information (i.e., ID embeddings). The lack of reliable control over identity and facial expression prevents the personalization of 3D head avatars, resulting in limited range of possible output expressions custom attributes without incurring identity drifting.
59
88 84 Textual prompts lack
pose, expression,

---

In this work, we present ID-to-3D, a new method to generate expressive and identity-consistery consistent 3D human heads using a text prompt and small set of 1-5 casually captured. in- un-the-wild, images of a subject. ID-to-3D creates a variety of separated yet ID>consistent expressions in single optimization, leveraging as priors compositionality and task-specific 2D) diffusion models. To ensure consistency, the identity of each subject is encoded via facial embedding features and, to encourage expressivity, emotions are disentangled via novel ID-specific neural parametric representation. The generation of each 3D asset leverages score distillation sampling and two-stage pipeline to create shape details, texture features, and materials. During optimization, the guidance is given by a foundational model variant, working as 2D
extended into a) erentity-aware b) task-aware and e) expression-awe prior for either geometry or texture generation. Each stage guidance is trained only once for all the potential identities, with lightweight fine- tne-tuning strategy changing only 0.2% of all the available training parameters.
Overall, we present the following contributions: (1) we introduce the first method for areface-conditionel generation of 3D head assets using SDS. (2) Novel ID- ID-Conditioned Expressive Model: our model creates identity-conditione expressive repre- sentation for each subject, enabling the generation of up to 13 unique and ID-consistent expressions captured by latent codes and associated with a set of 3D assets with separate geometric, albedo, and material information. (3) Novel
3D Head Asset Generation:
and text-to-DD-Aleet Models: we present a novel approach to creating ID D-conditioned and popression-conditionn
eox-to-imaage models capable of generating realistically plausible normals and albedo images from a small set of 3D assets. Exten- sive experiments confirm that our method outperforms text-based and image-based SDS baselines by producing relight-able 3D head assets with unprecedented geometric details, superior texture quality, and able to exhibit wide variety of expressions. Fig. displays 3D heads generated with II>-to-3D.
2 Related Work
3D Human Generation and Reconstruction. Human modelling and reconstruction in the 3D domain is typically based on 3D Morphable Models (3DMMs) 6 43 78, which model variations in human shape and appearance using PCA. The advent of deep learning enabled very expressive reconstructions [1 68 but also extended this line of research beyond linear spaces. Neural Parametric Head models 24 explored the use of signed distance function and deformation field to generate expressive geometries. Generative Adversarial Networks have also been proposed to generate [4 21 and reconstruct faces from "in-the-wild" images (22 50 40). Diffusion models showed impressive results in modeling skin textures (92 60 especially when paired with large-scale, high-quality datasets of real scans.
Text-to-3D Human Generation. The creation of 3D human assets via text conditioning has seen significant progress, building on the foundations laid by advances in text-to-2D generation (55] 69 72. 67]. Initial attempts [86, 29 54 73 55 32, M utilized the CLIP language model to optimize implicit or explicit 3D representations. The seminal work of DreamFusion (63 introduced the Score Distillation Sampling (SDS) loss, which uses pre-trained 2D diffusion as prior for 3D generation.
This work led to a revolution in text-to-3D generation [53 57 27 85
45
48
47 [2] and text-to- 44 and Headevolver [2
52
3D human generation 9
76 53
26
49 DreamAvatar
g TADA
43 to create 3D human avatars with controllable shapes and poses. F [0 separated geometry and texture training, employing DMTET (77) and PBR texture (58) for the 3D representation. HumanNorm (31 introduced the idea of training normal diffusion model to guide high-quality geometry generation. Despite promising results, these models face challenges in generating highly detailed and expressive 3D heads, due to the inherent limitations of natural language conditioning.
build upon the use of a template 66
Personalized Generation with Diffusion Models. Controllable generation is crucial to develop widely applicable generative models. Work in this domain includes the costumization of GANs [74]
33
34 and research dedicated to steer the generation of large-scale foundation models with additional control signals (2 70
35
36
30 93. 88 71I. The use of ID embeddings as an alternative to text
prompts showed promising results in 2D generation 62 87 84 59, but remains underexplored in
the creation of 3D human avatars with SDS. Related research supplements traditional text prompts in
SDS pipelines with image-based prompts. DreamBooth3D [66] and Avatarbooth [91] proposed to
train an ad hoe model to use as a guide in the generation of 3D objects or avatars. Magic [64]
incorporate 3D and 2D priors in SDS generation. DreamCraft3D [79] proposed hierarchical 3D
3

---

content pipeline to generate textured 3D meshes from a single unposed image. Despite encouraging results, the creation of high-quality head assets with these methods remains challenging. due the difficulty of extracting appropriate facial features using only naive image prompting.
3 ID-to-3D
We propose a novel method for identity-driven human head generation, which utilizes a pre-trained 2D model to distill expressive head avatars with high geometric details and high- feghfiidelity textures, avoiding the need for large-scale training on 3D datasets. As illustrated in Figure 2 starting from subject's identity embeddings, a two-stage SDS pipeline. After convergence, the learned 3D representation can be used to create ID-driven expressive heads, that ready be used in common rendering engines.
method trains
of latent expression representations using
3.1 3D Head Optimization Objective
Score Distillation Sampling generation pipeline optimizes a 3D representation using a pre-trained 2D diffusson model & as guidance. The pipeline optimization objective is to align the distribution of 3D asset renderings with the target distribution p(xo) (iexa). created by the 2D diffusion model conditioned on an input text Beex.. Given the distribution of renderings under various camera conditions o² Xx (hex)) = I qa (XolViext Urext c),(c)dc, the optimization objective reads:
min DKL (g" (Xol p(xxoleex))
The target distribution p(xolyee) is typically estimated by a foundational text-to-image diffusion model that approximates the distribution of natural RGB images [69] (i.e., trained over large and uncurated datasets such as LAION-5I 5B (75).. Despite its indisputable success in creating a variety of assets, using the above objective and guidance to generate detailed 3D heads remains a complicated task. First, this target distribution might drift significantly from the distribution of natural heads rendered in realistic light and camera conditions. Second, a general one-shot guidance model does not have explicit ways differentiate texture and geometric characteristics, which makes the creation of fight-independent texture and high-quality geometry extremely challenging. Third, the naive use of text prompts limits the control over the generated head assets, since textual prompts cannot easily or exhaustively capture facial and expression features. In our pipeline's generation, we decompose the Obj. into two smaller and more controllable objectives:
min D DKL( (9 (2 c, Yuess yexps Bal) p cc, Vhext yeep- Pia))
8
geometry generation objective
+ DKI(a on (zg c, yhext; Mexps Via) p(zūle, thext, Deepp Mia)).
tearture generation objective
In the above equation, Og represents the parameterization of the 3D geometry, 2 denotes normal maps, f₂ denotes the parameterization of the 3D textures and zi albedo textures. Conditioning is introduced in the form of textual prompt Viex,, identity condition Ya. and expression condition Meep- The letter denotes the lighting condition the rendered image. The target distributions for geometry and texture generation refer to the ideal distribution of head-specific normal and texture maps, which are practice estimated via geometry-orienter and albedo-oriented models guided by face-specific conditioning.
3.2 2D Guidance
To initiate the 3D head reconstruction, we start from the development of 2D priors capable of accurately separating texture and geometric details while, at the same time, consistently representing the facial characteristic of subject under various expressions, conveying different emotional states. The difficulty of this task lies in its nuanced nature, exacerbated by the lack of large-scale 3D human sean datasets, which makes the capture detailed face features and the generalization to new identities particularly difficult when training from scratch even when naively fine-tuning from large-scale model. To solve this challenge, we propose to explicitly model geometry and appearance domains, identity conditioning yia and expression conditioning yexp. achieving modular separation

---

of otherwise entangled -together information. To overcome the need for a large-scale dataset, we leverage small dataset of human heads with different expressions (NHPM) [24], a pre-trained stable diffusion model (SD) [69]. and selective fine-tuning strategy that affects only minimal number of parameters, needed to accommodate these new conditionings. practice, we use rasterized normals as a 2D proxy for geometric information and rasterized albedo as representation of appearance information. We treat the shift from the natural image distribution towards normal maps and albedo textures as "style-transfer" tasks, aiming to leave the content of the SD features unchanged while modifying their simiaariy information. We use Low-Rank adaptation matrices (LoRA) [00 to adjust Query Q, Key K and Value V features of the self-attention to work in the adjusted normal and albedo domains. The normal- -ddated self attentonn equation becomes the following:
Z Qu( K", V*), Q" X X K" XW +XWK V XW V +XW
(3)
while the albedo-and -daated seff-attention can be read by changing the superscript a. As identity representation we select the identity embeddings Yia, from a state-of-the-a art face recognition network 16 94), a compact vector of facial features extracted from "in-the-wild" images of subject. As expression conditioning, we use Yesp CLIP embeddings (65] for the textual descriptor of the 23 FACS coded expressions proposed in FaceWare 77.. To ensure control over the generated head and face representation during deployment, we treat the integration of identity and expression information in the baseline architecture as "multimodal conditioning", by including their contribution in the SD cross-attention layers via Id-dteteer [88] strategy:
ZČa= Au( Q, Ktext Vlext) + Aia Au Q, Kid Vid) + Aeap Att(Q, Kexp, Vexp)
In the above equation, Aid and Aeep control the contribution of identity and expression conditioning. The Q=XWe term represents the Query extracted from SD features, ==* and ==* w the Key and Values extracted from the expression embedding, and K' and vi==y,WW! the Key and Values extracted from the identity embedding. We train only our additional parameters, leaving the rest of the model frozen, targeting 0.2% of the overall trainable parameters. We separately optimize a 2D prior for geometry oy and textures 6a. using image-conditioninn pairs created from the renders of the NHPM dataset under various camera poses The training objective for the geometric 2D prior follows the same training objective as a traditional SD model [69]
Exiplle
Ezzy y.w. llc Co,, (z. f, C, Viexs Vid, Yesp) 1|
(5)
while the analogous training objective for the 2D texture prior can be derived by changing both superscript n and subscript 9 to a. Note that in Equation [] Yesp indicates CLIIP embeddings for the textual descriptor of the renders (i.e., camera view and subject's attributes) and does not convey face-identity, z, indicates zū noised at timestep t. Examples of generated prompt-to-imagee can be seen in Figure 2 as well as additional materials.
3.3 Geometry Generation
We represent an -ppetity geometry as a neural parametric head model composed of a deep marching tetrahedra (DMTET (77)) representation, additionally coupled with a set of identity- dependent facial expression latent codes. Our lightweight geometric representation produces highly detailed geometry and a broad range of expressions without notable identity drift, as shown by our experiments. Compared to explicit semplate-based approaches [92 44 43), it has the flexibility to dynamically modulate the local resolution of the mesh to capture high-frequency geometrical details. In other words, adjusts the mesh resolution of specific regions of the face to adapt to the given subject and expression.
The DMTET geometry representation uses a deformable tetrahedral grid I' and network V. to generate a 3D asset [77]. We extend DMTET to learn multiple expressions at the same time. We model each facial expression with a learnable latent code kapp E Rd and design the network J k as a Transformer [8I] parameterized by by learnable parameters, that processes both the deformable grid and the expression information. During the training phase, we randomly select one of the potential expressions, estimate the signed distance function (SDF) for the underlying head, and use a differentiable marching tetrahedra layer to convert the implicit representation into the explicit surface mesh for that ID and expression, compelling the V model to learn diverse set of expressions that are consistent with the identity at hand. We supervise optimization through an
5

---

Dream Face
Human Norm
Magic 123
Dream Face
Human Norm
Magic
TADA
DC3D
Ours
TADA*
DC3D
Ours
123
Craig Robinson
Lucy Liu
Donald Trump
Kanye West
Florence Pugh
Freddy Mercury
Figure 3: Qualitative results for text-to-3D (*) and image-to-3D methods. Methods are evaluated
under the same text prompt and rendering conditions. DreamCraft3 is reported as DC3D. Geometry displayed via normal maps camera coordinates. Using only small set of 5 images as condition- ing, ID-to-3D achieves high geometric quality and realistic textures.
SDS loss 63 computed using the rasterized geometry model og 2D prior. We optimize the 3D representation B, as follows:
VLsDS(0,) = Be,r. (t))(EE, (z,", Yid Vexp, Ytext, 6) - Og(0,, c)
;, [Kepp vol
(6)
00
In the above equation, g00, c) represents the normals of the rendered image, created using the differentiable render g and the camera pose c, and k is the randomly sampled expression code.
Lastly, combine the SDS loss with Laplacian regularizer, encourage smooth surfaces.
3.4 Texture Generation
Given a trained geometry model oy. we model the texture appearance 0.. To ensure ID-aligned expression generation, follow an analogous parameterization for the expressions in the texture domain. We represent an identity-specific appearance as neural parametric head model composed of pseudo-albedo prediction network Vr, coupled with a of ID-dependent facial expression latent codes. We instantiate each expression as a learnable latent code keap E Rderp and model V,, (k as Transformer trained to predict the spatially- varyingg reflectance in UV-map representation [89], texture. We use an
namely the albedo, roughness and specularity, for each ID and expression-specee off-the-shelf physically-bast -ased renderer (38]. Reflectance disentanglement is an ill-posed problem. and in the absence of prior data, we use camera and illumination randomization as regularization constraint 18 41 each iteration, we sample random environment illumination maps, augmented with random Y-axis rotations. During training, we randomize the sampling of the latent expression and deploy the albedo model Φ. SDS loss optimize 0.:
(LsDs((.) = Ec,t, (())(EE. (z) Yid Yexp: Ytext, t) - Og(O., c,
;0, = [ke vol. (7)
00,

---

Texture Geometry
63
10
252
159
HursunNoem
Dreamt-ace
02
188
145
TADA
Desum SD
Gonmetry
TADA
44
215
118
Magic123
6
HumanNorm"
164
Dessui see
Fantneidd))
Texure
DreamCraft3D
Magic123
79
205
162
165
TADA
Ours
153
85
Miggee22
0.0 0.2
0.4 0.6
0.5
1.0
DrsumCruftaD
Table 1: Frechet Inception Distance measures the geomet- ric and texture quality for the generated 3D heads. "*" indi- cates text-to-3D SDS pipelines.
Figure 4: (Left) Identity Similarity Distribution between "in-the-wild" images and renderings of 3D heads. (Right) Comparative Preference Survey on texture quality (outside) and geometry quality (in). We report % of preferences.
where g(a, c, 1) represents the pseudo-albesd maps created using a differentiable render and a sample camera pose and lighting condition.
4 Experiments
We assess the efficacy of ID-to-3D as a specialized method for ID-driven expressive human face generation in different scenarios and report comparative analysis against state-of-the-art text-to-3I) and image-to-3D generation pipelines. Further analysis, implementation details and discussion can be found in additional material.
4.1 Identity Generation
We benchmark ID-to-3D against several state-of-the-art methods in the domain of SDS-based 3D
face asset generation. Specifically, we consider Fantasia3D [10], three recent and similar methods
specialized text-to-human avatar generation (i.e., Human-Norm (31], TADA [44] DreamFace (92)
and two methods that leverage both text and images to create 3D assets (i.e., Magic123 [64].
DreamCraft3D (79). In order to compare together text-to-3D and image-to-3D methods, we select a test benchmark of 40 celebrity names, suggested by ChatGPT, covering actors, sports, and media
personalities. We automatically download 25 images for each identity from BingImages [: 5 images to use input and 20 to textual prompt "a DSLR face portrait of..." and the same input images.
references for our comparisons. For all methods, we use the same
Qualitative Comparisons. Results of existing methods are reported in Figure 3 under the same lighting and rendering conditions. The 3D assets created by ID-to-3D show realistic texture, sharp fine-grained details, and ID fidelity, capturing facial characteristics of the input identity without relying heavily on often ambiguous text prompts or naively lifting 2D images in the 3D domain.
Identity Similarity Distribution. We perform quantitative evaluation on all the evaluated models using similarity ID. We assess the ID fidelity of the 3D assets using the CosFace similarity metric 833 40]. We center and align the 3D objects and collect renders in wide range of camera positions (i.e. elevation [-15 +15 and rotation [- 65 +655]). We measure the cosine similarity between the ID features of each render and set of 20 in- in-the-widd wiid images of the same identity, used as reference. We report the distribution of the identity similarity for each method in Figure 4 (Left). Note that the variance of the distribution correlates with the ID consistency of the 3D object across viewpoints. Despite being able to generate realistic skin textures, FreamFace cannot create hair or eyes by design, resulting in the lowest average similarity score. DreamCraft3D and Magic125 123 leverage the input image to achieve a photorcalistic front-facing camera render, but struggle to create 3D consistent heads, resulting in distributions associated with the highest variance. As clearly visible from Figure 4 ID-to-33 3D is capable of creating realistic and consistent 3D heads, reporting the lowest variance and the highest similarity score.
Quantitative Comparisons:
Quantitative Comparisons: FID and User-Study. The evaluation of the generated 3D geometries and textures is performed using the Frechet Inception Distance (FID) metric 28). For texture quality, the FID calculated between the renderings and the images from Stable Diffusion V1.5 [69]. For
geometry quality, FID is determined by comparing normal maps with those extracted from the NHPM test set (24) To further substantiate our analysis, we conduct two user preference surveys, comparing our method with the strongest baselines for the generation of texture and geometry. We compare
7

---

(a) Eyes closed
(b) Brow raised
(c) Brow lowerer
(d) Cheeks puffed
(c) Grin
(f) Dimpler
(g) Lip roll
(h) Squeeze
(i) Eyes closed
(j) Brow raised
(k) Brow lowerer
(I) Cheeks puffed
(m) Grin
(n) Dimpler
(o) Lip roll
(p) Squeeze
Figure 5: ID-to-3D expression diversity. Renderings and normal maps in camera coordinates
taken for identities: Will Smith, Anya Taylor Joy, and Kanye West. Our method achieves fine-grained geometry carving and high-quality texture generation, realistically reproducing various skin tones.
the best 2 performing text-to-3D and image- -o- methods against ID-to-3Din the user survey, as this enabled us to gather more responses, As shown in Table Aable[an and Figure 4 (Right), FID metrics and user evaluations report aligned results. Our model achieves the lowest FID scores and the highest user preference in both geometry and texture generation, showcasing together the overall stronger performance of ID-to-3Das a human- -peeffic geometry and texture generator.
Expressive ID-conditioned Generation
ID-to-3D is specifically designed to create complex, uncommon, and subtle expressions with level of details not previously achievable using existing SDS methods. In Figure 5 we showcase the unique ability of our method to generate wide range of expressions that remain identifiable and yet identity-consisten.
Quantitative Analysis: Identity Similarity Distribution Across Expressions. We test the ID consistency across views and expressions by reporting the cosine similarity between the ID features of reference render (i.e., neutral expression and front-facing camera) and the ID) features captured across the remaining expressions and view points. Figure 6 (Right) shows the distribution of ID similarity computed for set of 10 subjects. ID-to-3D consistently produces 3D assets with high ID) similarity and low variance.
Quantitative Analysis: Expression Variety Visualization. A core characteristic of ID-to-3D is its robust ability produce a wide range of unique and vivid expressions. In this section, we provide visualization of this expression diversity. We select subset of 13 expressions and 10 subjects, extracting for each 3D head set of renders in range of 9 camera poses. Then we extract the ID features for each render and project them into 2D points using t-SNE. As visible in Figure 6 (Middle), plotting these points clearly shows the heterogeneity of the generated expressions and identities. Firstly. renders associated with the same identity are grouped in distinct clusters. Furthermore, within each cluster, we observe how there a local variation of IDD features as they adapt their response to different expressions.
4.3 ID-conditioned Text-based Customization
The unique characteristics of our model allow for the customization objects without identity or expression drift. Figure 7 and (Right) showease how variety of alterations can be imposed on the

---

Kanye
Lacy Liu
Zendayya
Skarggard
Evanna
B. Claes
Lip Roll
Dumpler
Wr Smiith
Liu
Angry
Brow Raised
A. Jay
K. West
Lipe Down
Alexander Skarsgard
Sqqueeze
Brow Lowerer
Choclas Puffed
Eyes Closed
Chocks pullid
Mouth Stretch
Dicapler
Griz
Lip Roll
Neutral Open
Eyes Closed
Nose Down
Figure 6: Expressions analysis. ID-to-3D creates a variety of expressions with robust ID consistency. (Left) Visualizations of different expressions for identities (i.e. Bill Gates, Alexander Skarsgard). (Middle) Expression diversity. t-SNE plot visualizing the ID embeddings computed considering different camera poses, expressions, and subjects. Different identities and expressions are clustered separately. (Right) Identity similarity distributions between neutral-pose and remaining expres- sions. Rendered with 2
male
maie
M..dde bair
brown halr
Cumberbatch
Benedict
Smitth
Will
Trump
Kardashian
Kim
Luey Liu
... fomale ...as female
-rreonn hair
gr.ennn hair
Figure 7: Mlenttty-Consister Editing, (Left) De-aged 3D heads generated using different identity conditioning and the textual prompt: "...as a cute baby". Normal maps are displayed in world coordinates photorealistic renderings. (Right) Geometry and texture editing with text prompts. ID-to-3D edits appearance and geometric features in an ID-consistent manner.
object's geometry, given the appropriate text conditioning while maintaining the subject's general physiognomy. In particular, ID-to-3D displays aligned geometry and appearance even after editing, preserving the ability to convey different expressions (e.g. from "eyes closed" to "brow lowerer", from "neutral" to "squeeze") even after facial hair textures have been altered via textual prompt.
Figure provides evidence on the text-guided editing capability of our model for richer and more complex textual prompts. In particular, we showcase text prompts associated with various hairstyles, head accessories, and face shape changes driven by different ethnic backgrounds. Note that our method can interpret and exploit text-based inputs not addressed in previous works (e.g. id-driven changes in "aging", "gender", "heritage"). Even when using the exact same text prompt (e.g. "A woman with African-Americes heritage, ... 70's hairstyle.") consistent assets that simultaneously align with the text and retain the characteristic facial features of the input ID. Our approach is able to address practical scenarios and opens new avenues for expressive text-guided editing of 3D assets.
model generates unique identity-
Conclusion and Ethical Considerations
Limitations. Despite setting new state-of-the-art, we acknowledge ID-to-3D limitations: (1) The generalization capacity is constrained by the employed face embedding network [17], the pretrained

---

Inputl
Input] woman Africon- American herilage, curly voluminous dark hair. 70's hairstyle.
Input2 woman African- American heritage, curly voluminous dark hair. 70's hairstyle.
Input3 man blonde head. fair skin tone, curly voluminous hair puff style.
Ingurl face portrait of Dragonborn with grey stone dragon hurness, wearing short red hair and strong grey horns made of roch.
Input2 A forest Draganborn Tieffing with green dragon horns mude of foliage. wearing blonde hair green robe.
Input3 A Caucasian man with bald huirless head, blue eyes blue shirt.
Input2
Inputl A face portroit of a Japanese woman wearing delicate pink hair makeup
Input2 Japanese woman with light skin tone and dark elegant hair. Long straight hairstyle.
Input3 An Indian wise king wearing a crown with mighty beard.
Input3
Figure 8: Identity-Consistee Consistent Editing with Rich Textual Prompts. ID-to-3D generates ID- consistent assets that reflect both subtle and significant changes in geometry and appearance described in detailed textual input. Normal maps displayed in world coordinates next photorealistic renderings and input prompts.
large-scale diffusion models 69 (8], and the finetuning of the 2D guidance models on a dataset of relatively small size, which might result in inaccurate representation biases and biases due dataset imbalances; (2) the texture-guidance model results might come short of photorealism. The generated textures mimie the albedos from the [24] dataset, which contains expression-rich data but only low-resolution UV diffuse albedo maps; (3) the lack of specific optimization for physically bounded textures and geometries might occasionally produce unnatural exaggerated facial characteristics, hence we call them pseudo-albedo, despite producing render-ready assets; (4) the computational resources needed for PBR rendering hinder potential applications in video-driven problems.
Societal Impact. Technological advancements in automatic 3D human generation have various beneficial applications, but also raise important ethical considerations about representation and potential misuses. We advocate for responsible research and take the following steps to mitigate unauthentic reconstructions: (1) we restrict training data the facial region only. (2) we advocate for the replacement of text- based prompts with larger ID embeddings that are sometimes lacking (90] but entail significantly less bias than methods trained only on celebrity datasets and text [65]
Conclusion. In this work, we present ID-to-3D, novel method for expressive 3D head asset generation from one or more face images. Our method deploys a novel human parametric expression model in tandem with specialized geometry and albedo guidance, not only to create intricately detailed head avatars with realistic textures but also to achieve strikingly ID-consistent results across wide range of expressions, setting new benchmark in comparison to existing SDS techniques. Without having to rely on 3D captured datasets that are expensive to collect and typically biased, and without being constrained on specific geometry template, our method can be employed by broad range of subjects, with different features such as skin tone and hairstyle.
Acknowledgements. S. Zafeiriou and part of the research was funded by the EPSRC Fellowship DEFORM (EP//010003//), EPSRC Project GNOMON (EP/X01 130011)34/) and Turing Al Fellowship (EP/Z534699/1).
10

---

References
[1] https//www.hingg wowinnn.oo/mmags//eeeedd
[2] Marmoset toolbag. luttps://marmoset co/toolbag/ Accessed: 2022-05-20.
Dejan Azinovié, Olivier Maury, Christophe Hery, Matthias Nießner, and Justus Thics. High-res facial appearance capture from polarized smartphone images. In Proceedings of the IEEE/CVF Conference on Computer Vision and Puttern Recognition, pages 16836-16846, 2023.
[4] Volker Blanz and Thomas Vetter. A morphable model for the synthesis of 3d faces. In SIGGRAPH 199, 1999.
[5] Federica Bogo, Angjoo Kanazawa, Christoph Lassner, Peter Gehler, Javier Romero, and Michael J Black. Keep it smpl: Automatic estimation of 3d human pose and shape from a single image. In Computer Vision-ECCV 2016: 14th European Conference, Amsterdam, The Netherlands, October 11-14, 2016, Proceedings, Purt V 14, pages 561-578. Springer, 2016.
[6] James Booth, Anastasios Roussos, Stefanos Zafciriou, Allan Ponniah, and David Dunaway, A 3d morphable model learnt from 10,000 faces. Proceedings of the IEEE conference on computer vision and pattern recognition, pages 5543-5552, 2016.
[7] Chen Cao, Yanlin Weng, Shun Zhou, Yiying Tong, and Kun Zhou. Facewarehouse: A 3d facial expression database for visual computing. IEEE Transactions on Visualization and Computer Graphics, 20(3):413-
425, 2013.
[8] Yukang Cao. Yan-Pei Cao. Kai Han. Ying Shan, and Kwan- Yee K Wong. Dreamavatar: Text-and-shape
guided 3d human avatar generation via diffusion models. arXiv preprint arXiv:2304. 30040091666....01 2023.
[9] Yukang Cao. Van-Pei Cao, Kai Han, Ying Shan, and Kwan- Yee K Wong. Guide3d: Create 3d avatars from
text and image guidance. arXiv preprint arXiv:2308. 09705, 2023.
[10] Rui Chen, Yongwei Chen, Ningxin Jiao, and Kui Jia. Fantasia3d: Disentongling geometry and appearance for high-quality text-to-3d content creation. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pages
2023.
[11] Yongwei Chen, Rui Chen, Jiabao Lei, Yabin Zhang, and Kui Jia, Tango: Text-driven photorealistic and robust 3d stylization via lighting decomposition. Advances in Neural Information Processing Systems, 35:30923-30936, 2022.
[12] Yen-Chi Cheng, Hsin- Ying Lee, Sergey Tulyakov, Alexander G Schwing, and Liang-Yas Gui. Sdfusion: Multimodal 3d shape completion, reconstruction, and generation. In Proceedings of the IEEF/CVF Conference on Computer Vision and Pattern Recognition, pages 4456-4465, 2023.
[13] Siying Cui, Jiankang Deng, Jia Guo, Xiang An, Yongle Zhao, Xinyu Wei, and Ziyong Feng. Idadapter: Learning mixed features for tuning-free personalization of text-to-image models. arXiv preprint arXiv:2403. 13535, 2024,
[14] Radek Dan&ček, Michael J Black, and Timo Bolkart. Emoca: Emotion driven monocular face capture and animation. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 20311-20322 2022.
[15] Paul Debevec. Tim Hawkins, Chris Tchou, Haarm- Pieter Duiker, Westley Sarokin, and Mark Sagar. Acquiring the reflectance field of human face. In Proceedings of the 27th annual conference on Computer graphics and interactive techniques, pages 145-156, 2000.
[16] Jiankang Deng, Jia Guo, Evangelos Ververas, Irene Kotsia, and Stefanos Zafeirio.. Retinaface: Single-shot
nulti-level face localisation in the wild. In CVPR, 2020.
[17] Jiankang Deng. Jia Guo, Niannan Xue, and Stefanos 7afeirion. Areface: Additive angular margin loss for deep face recognition. Proceedings of the IEEF/CVF conference on computer vision and pattern recognition, pages 4690-4699, 2019.
[18] Valentin Deschaintre, Miika Aittala, Fredo Durand, George Drettakis, and Adrien Bousseau. Single-image svbrdf capture with a rendering aware deep network. ACM Transactions on Graphics (ToG), (15: 2018.
[19] Bernhard Egger, William AP Smith, Ayush Tewari, Stefanie Wuhrer, Michael Zollhoefer, Thabo Beeler, Florian Bernard, Timo Bolkart, Adam Kortylewski, Sami Romdhani, et al. 3d morphable face mastls-past, present, and future. ACM Transactions on Graphics (ToG), 39(5).1-38, 2020.
[20] Rinon Gal, Yuval Alaluf, Yuval Atzmon, Or Patashnik, Amit Bermano, Gal Chechik, and Daniel Cohen-Or. An image is worth one word: Personalizing text-to-image generation using textual inversion.
arXiv preprint arXiv:2208.00 01618, 2022.
[21] Baris Gecer, Alexandros Lattas, Stylianos Ploumpis, Jiankang Deng, Athanasios Papaioannou, Stylianos Moschoglou, and Stefanos Zafeiriou. Synthesizing coupled 3d face modalities by trunk-brance generative adversarial networks. In European conference computer vision, pages 415-433, Springer, 2020.
[22] Baris Gecer, Stylianos Ploumpis. Irene Kotsia, and Stefanos 7afeiriou. Ganfit: Generative adversarial network fitting for high fidelity 3d face reconstruction. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition. pages 1155-1164, 2019.
[23] Abhijeet Ghosh, Graham Fyffe, Borom Tunwattanapong. Jay Busch, Xueming Yu, and Paul Debevec. Multiview face capture using polarized spherical gradient illumination. In Proceedings of the 2011 SIGGRAPH Asia Conference, pages 1-10, 2011.
[24] Simon Giebenhain, Tobias Kirschstein, Markos Georgopoulos, Martin Rünz, Lourdes Agapito, and Matthias Nießiner. Learning neural parametric head models. In Proceedings of the IEEE/CVF Conference Computer Vision and Pattern Recognition, pages 21003 220022, 2023.
11

---

[25] Paulo Gotardo, Jérémy Riviere, Derek Bradley, Abhijeet Ghosh. and Thabo Beeler. Practical dynamic facial appearance modeling and acquisition. ACM Transactions on Graphics (ToG), 37(6):1-13, 2018.
[26] Xiao Han, Yukang Cao, Kai Han, Xiatian Zhu, Jiankang Deng, Yi-Zhe Song, Tao Xiang. and Kwan- Yee K Wong. Headsculpt: Crafting 3d head avatars with text. Advances Neural Information Processing Systems,
36, 2024.
[27] Ayaan Haque, Matthew Tancik, Alexei A Efros, Aleksander Holynski, and Angjoo Kanazawa. Instruct- nerf2nerf: Editing 3d scenes with instructions. In Proceedings of the IEEE/CVF International Conference
on Computer Vision, pages 19740-19750, 2023.
[28] Martin Heusel, Hubert Ramsauer, Thomas Unterthiner, Bernhard Nessler, and Sepp Hochreiter. Gans trained by two time-scale update rule converge to a local nash equilibrium. Advances in neural information
processing systems, 30, 2017.
[29] Fangzhou Hong, Mingyuan Zhang, Liang Pan. Zhongang Cai, Lei Yang, and Ziwei Liu. Avatarclip:
Zero-shot text-driven generation and animation of 3d avatars. arXiv preprint arXiv:2205. 08535, 2022.
[30] Edward J Hu, Yelong Shen, Phillip Wallis, Zeyuan Allen-Zhu, Yuanzhi Li, Shean Wang. Lu Wang, and Weizhu Chen. Lora: Low-rank adaptation large language models. arXiv preprint urXiv:2106.096S)
2021.
[31] Xin Huang. Ruizhi Shao, Qi Zhang, Hongwen Zhang, Ying Feng, Yebin Liu, and Qing Wang. Humannorm: Learning normal diffusion model for high-quality and realistic 3d human generation. arXiv preprint
arXiv:2310.01-000 2023.
[32] Ajay Jain, Ben Mildenhall. Jonathan T Barron, Pieter Abbeel, and Ben Poole. Zero-shot text-guided object generation with dream fields. In Proceedings of the IEEF/CVF conference on computer vision and pattern
recognition, pages 867-876, 2022.
[33] Minguk Kang, Jun-Yan Zhu, Richard Zhang. Jaesik Park, Eli Shechtman, Sylvain Paris, and Taesung Park. Scaling up gans for text-to-image synthesis. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 00124-10144 2023.
[34] Tero Karras, Miik Aittala, Samuli Laine, Erik Härkönen, Janne Hellsten, Jaakko Lehtinen, and Timo
Aila. Alias-free generative adversarial networks. Advances in newral information processing systems, 34:852-865 263, 2021.
[35] Tero Karras, Samuli Laine, and Timo Aila. A style-based generator architecture for generative adversarial networks. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pages 4401-4410, 2019.
[36] Tero Karras, Samuli Laine, Miika Aittla, Janne Hellsten, Jaakko Lehtinen, and Timo Aila. Analyzing and improving the image quality stylegan. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pages 11118119, 2020.
[37] Nikos Kolotouros, Thiemo Alldieck, Andrei Zanfir, Eduard Bazavan, Mihai Fieraru, and Cristian Smin- chisescu. Drcamhuman: Animatable 3d avatars from text. Advances in Neural Information Processing
Systems. 36. 2024.
[38] Samuli Laine, Janne Hellsten, Tero Karras, Ycongho Scol, Jaakko Lehtinen, and Timo Aila. Modular primitives for high- perhroerformnccc differentiable rendering, ACM Transactions on Graphics (ToG), 39(6):-
14, 2020.
[39] Alexandros Lattas, Yiming Lin, Jayanth Kannan. Ekin Ozturk, Luca Filipi. Giuseppe Claudio Guamnera,
Gaurav Chawla, and Abhijeet Ghosh. Practical and scalable desktop-based based high-quality facial capture. In European Conference on Computer Vision, pages 522-537. Springer, 2022.
[40] Alexandros Lattas, Stylianos Moschoglou, Stylianos Ploumpis, Baris Geeer, Jiankang Deng, and Stefanos Zafeiriou. Fitme: Deep photorealistic 3d morphable model avatars. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 8629-8640. 2023.
[41] Alexandros Lattas, Stylianos Moschoglou. Stylianos Ploumpis. Baris Gecer, Abhiject Ghosh, and Stefanos Zafeiriou. Avatarme+:: Facial shape and brdf inference with photorealistic rendering -awae gans. IEEE Transactions on Pattern Analysis and Machine Intelligence, (211))9269992285 2021.
[42] Ruilong Li, Karl Bladin, Yajie Zhao, Chinmay Chinara, Owen Ingraham. Pengda Xiang, Xinglei Ren, Pratusha Prasad, Bipin Kishore, Jun Xing, et al. Learning formation of physically- face attributes. In Proceedings of the IEEE/CVF conference on computer vision and paltern recognition, pages 3410-3419, 2020.
[43] Tianye Li, Timo Bolkart, Michael Black, Hao Li, and Javier Romero. Learning a model of facial shape
and expression from 4d scans. ACM Trans. Graph. 36(6):194-1, 2017.
[44] Tingting Liao, Hongwei Yi. Yuliang Xiu, Jiaxaing Tang, Yangyi Huang. Justus Thies, and Michael J Black. Tada! text animatable digital avatars. arXiv preprint 2r38. 10899, 2023.
[45] Chen-Hsuan Lin, Jun Gao, Luming Tang. Towaki Takikawa, Xiaohui Zeng. Xun Huang, Karsten Kreis, Sanja Fidler, Ming-Yo Yu Liu, and Tsung- Yi Lin. Magic3d: High-resolution text-to-3d content creation. In Proceedings of the JEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 300-309, 2023.
[46] Hongyu Liu, Xuan Wang. Ziyu Wan, Yujun Shen, Yihing Song. Jing Liao, and Qifeng Chen. Headartist: Text-conditioned 3d head generation with self score distillation. arXiv preprint 2rXiv:23/2.07539 2023.
[47] Minghua Liu, Chao Xu, Haian Jin, Linghao Chen, Mukund Varma T, Zexiang Xu, and Hao Su. One-2- 3-45: Any single image to 3d mesh in 45 seconds without per-shape optimization. Advances in Neural Information Processing Systems, 36, 2024.
12

---

[48]
Ruoshi Liu, Rundi Wu, Basile Van Hoorick, Pavel Tokmakov, Sergey Zakharov, and Carl Vondrick. Zero- 1-to-3: Zero-shot one image to 3d object. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pages 9298-9309. 2023.
[49] Xian Liu, Xiaohang Zhan, Jiaxiang Tang. Ying Shan, Gang Zeng, Dahua Lin, Xihui Liu, and Ziwei Liu. Hu- mangaussian: Text-driven 3d human generation with gaussian splatting. arXiv preprint arXiv:231 177661, 2023.
[50] Huiwen Luo, Koki Nagano, Han- Wei Kung. Qingguo Xu, Zejian Wang. Lingyu Wei, Liwen Hu, and Hao
Li. Normalized avatar synthesis using stylegan and perceptual refinement. In Proceedings of the IEEF/CVF Conference on Computer Vision and Pattern Recognition, pages 16662-11672, 2021.
[51] Wan-Chun Ma. Tim Hawkins, Pieter Peers, Charles-Felix Chabert, Malte Weiss, Paul E Debevec, et al. Rapid acquisition of specular and diffuse normal maps from polarized spherical gradient illumination. Rendering Techniques, 9(10): 2007.
[52] Luke Melas- Melia-Kyriazii, Iro I Laina, Christian Rupprecht, and Andrea Vedaldi. Realfusion: 360deg reconstruc- tion of any object from a single image. In Proceedings of the IEFF//VF conference on computer vision and pattern recognition, pages 8446-8455, 2023.
[53] Gal Metzer, Elad Richardson, Or Patashnik, Raja Giryes, and Daniel Cohen-Or. Latent-nerf for shape- guided generation of 3d shapes and textures. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pottern Recognition, pages 16663,1673,, 2023,
[54] Oscar Michel, Roi Bar-On, Richard Liu, Sagie Benaim, and Rana Hanocka. Text2mesh: Text-driven neural stylization for meshes. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. pages -35022 2022.
[55] Nasir Mohammad Khalid, Tianhao Xie, Eugene Belilovsky, and Tiberiu Popa. Clip-mesh: Generating textured meshes from text using pretrained image-text models. In SIGGRAPH Asia 2022 conference
papers, pages 1-8, 2022.
[56] Stylianos Moschoglou. Stylianos Ploumpis, Mihalis A Nicolaou, Athanasios Papaioannou, and Stefanos Zafeiriou. 3dfacegan: Adversarial nets for 3d face representation, generation, and translation. International
Journal of Computer Vision, 128:2534 1,515,,5555,, 2020.
[57] Norman Müller. Yawar Siddiqui. Lorenzo Porzi. Samuel Rota Bulo. Peter Kontschieder, and Matthias NieBner. Diffrf: Rendering- guided 3d radiance field diffusion. In Proceedings of the IEEE/CVF Conference
on Computer Vision and Puttern Recognition, pages 4328-4333 2023.
[58] Jacob Munkberg, Jon Hasselgren, Tianchang
Jun Gao, Wenzheng Chen, Alex Evans, Thomas Müller, and Sanja Fidler. Extracting triangular 3d models, materials, and lighting from images. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 8280-8290, 2022.
[59] Foivos Paraperas Papantoniou, Alexandros Lattas, Stylianos Moschoglou, Jiankang Deng. Bernhard Kainz, and Stefanos Zafeiriou. Are2face: A foundation model of human faces. arXiv preprint arXiv:2403. 11641, 2024.
[60] Foivos Paraperas Papantoniou. Alexandros Lattas, Stylianos Moschoglou, and Stefanos Zafeiriou. Re- lightify: Relightable 3d faces from single image via diffusion models. In Proceedings of the IEEE/CVF International Conference on Computer Vision. pages 8806-8817, 2023.
[61] Georgios Pavlakos, Vasileios Choutas, Nima Ghorbani, Timo Bolkart, Ahmed AA Osman, Dimitrios Tzionas, and Michael Black, Expressive body capture: 3d hands, face, and body from a single image. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pages 10975-10985,
2019.
[62] Xu Peng, Junwei Zhu, Boyuan Jiang, Ying Tai, Donghao Luo, Jiangning Zhang, Wei Lin, Taisong Jin, Chengjie Wang, and Rongrong Ji. Portraitbooth: A versatile portrait model for fast identity- preserved personalization. arXiv preprint 1oXiv:2312.063355
2023. Ben Poole, Ajay Jain, Jonathan T Barron, and Ben Mildenhall. Dreamfusion: Text-to-3d using 2d diffusion.
[63]
In ICLR, 2023.
[64] Guocheng Qian, Jinjie Mai, Abdullah Hamdi, Jian Ren, Aliaksandr Siarohin, Bing Li, Hsin-Ying Lee, Ivan Skorokhodov, Peter Wonka, Sergey Tulyakov, al. Magic1 One image to high-quality 3d object generation using both 2d and 3d diffusion priors. arXiv preprint arXiv:2306. /7843, 2023.
[65] Alec Radford, Jong Wook Kim, Chris Hallacy. Aditya Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, et al. Learning transfcrable visual models from natural language supervision. In International conference on machine learning, pages 8748 8763. PMLR,
2021.
[66] Amit Raj. Srinivas Kaza, Ben Poole, Michael Niemeyer. Nataniel Ruiz. Ben Mildenhall, Shiran Zada, Klir Aberman, Michael Rubinstein. Jonathan Barron, et al. Dreambooth3d: Subject-driven text-to-3d generation. 2023,
In Proceedings of the IEEE/CVF International Conference on Computer Vision, pages Aditya Ramesh, Prafulla Dhariwal, Alex Nichol, Casey Chu, and Mark Chen. Hierarchical text-conditional image generation with clip latents. arXiv preprint arXiv:2204.061
[67]
1(2):3, 2022.
[68] George Retsinas, Panagiotis P Filntisis, Radek Danecek, Victoria F Abrevaya, Anastasios Roussos, Timo Bolkart, and Petros Maragos. 3d facial expressions through analysis- bessss-bb--euurlle neural- arXiv preprint arXiv:2404.0m 04104, 2024.
[69] Robin Rombach, Andreas Blattmann, Dominik Lorenz, Patrick Esser, and Björ Ommer. High-resolution image synthesis with latent diffusion models. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pages 10684-10695, 2022.
13

---

[70]
Nataniel Ruiz. Yuanzhen Li, Varun Jampani, Yael Pritch, Michael Rubinstein, and Kfir Aberman. Dream- booth: Fine tuning text-to-image diffusion models for subject-driven generation. In Proceedings of the IEEE/CVF Conference on Computer Vision and Puttern Recognition, pages 22500-22510, 2023.
[71] Nataniel Ruiz, Yuanzhen Li, Varun Jampani, Wei Wei, Tingho Hou, Yael Pritch, Neal Wadhwa, Michael Rubinstein, and Kfir Aberman. Hyperdreambooth: Hypernetworks for fast personalization of text-to-image models. arXiv preprint arXiv:2.307.0653
2023.
[72] Chitwan Saharia, William Chan, Saurabh Saxena, Lala Li, Jay Whang, Emily L Denton, Kamyar Ghasemipour, Raphael Gontijo Lopes, Burcu Karagol Ayan, Tim Salimans, et al. Photorcalistic text-to- image diffusion models with deep language understanding. Advances in neural information processing systems, 35:36479-36644
2022.
[73] Aditya Sanghi, Hang Chu, Joseph G. Lambourne, Ye Wang, Chin-Yi Cheng. Marco Fumero, and Ka- mal Rahimi Malekshan. Clip-forge: Towards zero-shot text-to-shape generation. Proceedings of the
IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pages 18603- 18613, June 2022.
[74] Axel Sauer, Tero Karras, Samuli Laine, Andreas Geiger, and Timo Aila. Stylegan-t: Unlocking the power of gans for fast large-scale text-to-image synthesis. In International conference machine learning, pages 30105-30118, 18. PMLR, 2023.
[75] Christoph Schuhmann, Romain Beaumont, Richard Vencu, Cade Gordon, Ross Wightman, Mehdi Cherti, Theo Coombes, Aarush Katta, Clayton Mullis, Mitchell Wortsman, et al. Laion-5b: An open large-scale dataset for training next generation image-text models. Advances in Neural Information Processing Systems, 35:25278-25294, 2022.
[76] Junyoung Seo, Wooseok Jang. Min-Seop Kwak, Hyeonsu Kim, Jachoon Ko, Junho Kim, Jin-Hwa Kim, Jiyoung Lee, and Seungryong Kim. Let 2d diffusion model know 3d-consistency for robust text-to-3d generation. arXiv preprint 2xXvv22303.079:
2023.
[77] Tianchang Shen, Jun Gav. Kangxue Yin. Ming- Yu Liu, and Sanja Fidler. Deep marching tetrahedra: a hybrid representation for high-resolution 3d shape synthesis. Advances in Neural Information Processing Systems, 34:6087 6101, 2021.
[78] William AP Smith, Alassane Seck, Hannah Dee, Bernard Tiddeman, Joshua B Tenenbaum, and Bernhard Egger, A morphable face alhedo model. In Proceedings of the IEEF/CVF Conference on Computer Vision and Pottern Recognition, pages 501 150000, 2020.
[79] Jingxiang Sun, Bo Zhang, Ruizhi Shao, Lizhen Wang, Wen Liu, Zhenda Xie, and Yebin Liu. Dreameraft3d: Hicrarchical 3d generation with bootstrapped diffusion prior. arXiv preprint arXiv:23 10..1188 2023. [80] Junshu Tang, Tengfei Wang, Bo Zhang. Ting Zhang, Ran Yi, Lizhuang Ma, and Dong Chen. Make-it-3d: High-fidelit 3d creation from a single image with diffusion prior. In Proceedings of the IEEE/CVF International Conference Computer Vision. pages 29922, 2023.
[81] Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Lukasz Kaiser, and Illia Polosukhin. Attention all you need. Advances neural information processing systems,
30, 2017.
[82] Duotun Wang, Hengyu Meng, Zeyu Cai, Zhijing Shao, Qianxi Liu, Lin Wang, Mingming Fan, Ying Shan, Xiaohang Zhan, and Zeyu Wang. Hcadevolver: Text to head avatars via locally learnable mesh deformation. arXiv preprint arXiv:2403, 09326, 2024.
[83] Hao Wang, Yitong Wang. Zheng Zhou, Xing Ji, Dihong Gong, Jingehao Zhou, Zhifeng Li, and Wei Liu. Cosface: Large margin cosine loss for deep face recognition. In Proceedings of the IEEF Conference on
Computer Vision and Pattern Recognition (CVPR), June 2018
[84] Qixun Wang. Xu Bai, Haofan Wang. Zekui Qin, and Anthony Chen. Instantid: Zero-shot identity- preserving generation in seconds. arXiv preprint arXiv:2401.075/
2024.
[85] Zhengyi Wang, Cheng Lu, Yikai Wang, Fan Bao, Chongxuan Li, Hang Su, and Jun Zhu. Prolificdreamer: gighliielitty and diverse text-to-3d generation with variational score distillation. Advances Neural
Information Processing Systems, volume 36, 2023.
[86] Jalle Xu, Xintao Wang, Weihao Cheng, Yan-Pei Cao, Ying Shan, Xiaohu Qie, and Shenghua Gao. Dream3d: zeooshoo text-to-3d synthesis using 3d shape prior and text-to-image diffusion models. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 80908-20913 18, 2023.
[87] Yuxuan Yan, Chi Zhang, Rui Wang, Yichao Zhou, Gege Zhang, Pei Cheng, Gang Yu, and Bin Fu. 2023.
Facestudio: Put your face everywhere in seconds. arXiv preprint #xxv:2312..0666
[88] Hu Ye, Jun Zhang. Sibo Liu, Xiao Han, and Wei Yang. Ip-adapter: Text compatible image prompt adapter for text-to-image diffusion models, arXiv preprins arXiv:2308.0677 06721, 2023.
[89] Jonathan Young. xatlas. https: : h/t::tgttttt thub. com/ cwwp:pcc///cccccccco......ooccct saccyy/xatii ax Accessed: 2022-05-20.
[90] Seyma Yucer. Furkan Teklas, Noura AI Moubayed, and Toby P Breckon. Measuring hidden bias within face recognition via racial phenotypes. In Proceedings of the IEEE/CVF Winter Conference on Applications of Computer Vision, pages 995-1004, 2022.
[91] Yifei Zeng, Yuanxun Lu, Xinya Ji, Yao Yao. Hao Zhu, and Xun Cao. Avatarbooth: High-quality and customizable 3d human avatar generation. arXiv preprint arXiv:2306.09988
2023.
[92] Longwen Zhang, Qiwei Qiu, Hongyang Lin, Qixuan Zhang, Cheng Shi, Wei Yang. Ye Shi, Sibei Yang. Lan Xu, and Jingyi Yu. Dreamface: Progressive generation of animatable 3d faces under text guidance. arXiv preprint 2334:2300.03/11 2023.
[93] Lvmin Zhang, Anyi Rao, and Maneesh Agrawala. Adding conditional control to text-to-image diffusion models. In Proceedings of the IEEE/CVF International Conference Computer Vision, pages 3836-3847,
14

---

2023. [94] Zheng Zhu, Guan Huang, Jiankang Deng, Yun Ye, Junjie Huang, Xinze Chen, Jiagang Zhu, Tian Yang. Dalong Du. Jiwen Lu, and Jie Zhou. Webface260m: A benchmark for million-saale deep face recognition. TPAMI, 2022.
15

---

1. Claims
Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?
Question:
Answer: [Yes]
Justification: The claims are listed in both the abstract and introduction, reflect the paper's method and are justified by the experimental section.
2. Limitations
Question: Does the paper discuss the limitations the work performed by the authors? Answer: [Yes]
Justification: The limitations listed in the limitations section.
3. Theory Assumptions and Proofs
Question: For each theoretical result, does the paper provide the full set of assumptions and complete (and correct) proof?
Answer: [NA]
Justification: [NA]
4. Experimental Result Reproducibility
Question: Does the paper fully disclose all the information needed reproduce the main perimental results the paper to the extent that affects the main claims and/or conclusions of the paper (regardless whether the code and data are provided or not)?
Answer: [Yes]
Justification: We base our method on public code, report our technical details in the supple- mental materials, and will be releasing our code.
5. Open access to data and code
Question: Does the paper provide open access to the data and code, with sufficient instruc- tions faithfully reproduce the main experimental results, as described in supplemental material?
Answer: [Yes]
Justification: We use already publicly available data, and will be open- open-sourcing our code.
6. Experimental Setting/Details
Question: Does the paper specify all the training and test details (c.g., data splits, hyper- parameters, how they were chosen, type of optimizer, etc.) necessary to understand the results?
Answer: [Yes]
Justification: We include all the above details in the supplemental materials.
7. Experiment Statistical Significance
Question: Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?
Answer: [No]
Justification: Error bars are not reported because it would be too computationally expensive.
8. Experiments Compute Resources
Question: For each experiment, does the paper provide sufficient information on the com- puter resources (type of compute workers, memory, time of execution) needed to reproduce the experiments?
Answer: [Yes]
Justification: We have included the above details in the supplemental materials.
9. Code Of Ethics
Question: Does the research conducted the paper conform, in every respect, with the
NeurlPS Code of Ethics
16

---

Answer: [Yes]
Justification:
10. Broader Impacts
Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?
Answer: Yes
Justification:
We discuss both in the Conclusion and Ethical Considerations section.
11. Safeguards
Question: Does the paper describe safeguards that have been put in place for responsible release of data or models that have a high risk for misuse (c.g., pretrained language models, image generators, scraped datasets)?
Answer: [Yes]
Justification: We discuss both in the Conclusion and Ethical Considerations section.
12. Licenses for existing assets
Are the creators or original owners of assets (e.g.. code, data, models), used in the paper, properly credited and are the license and terms of use explicitly mentioned and properly respected?
Question:
Answer: [Yes]
Justification:
13. New Assets
Question: Are new assets introduced in the paper well documented and is the documentation provided alongside the assets?
Answer: [NA]
Justification: the paper does not release new assets.
14. Crowdsourcing and Research with Human Subjects
Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given to participants and screenshots, if applicable, as well as details about compensation (if any)?
Answer: [NA]
Justification:
15. Institutional Review Board (IRB) Approvals or Equivalent for Research with Human Subjects
Question: Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or equivalent approval/review based the requirements of your country or institution) were obtained?
Answer: [NA]
Justification:
17


Reviews:
[{"review_id": "TY93c6CqDe", "reviewer": "Reviewer_HnyJ", "summary": "**Summary\nThis paper focuses on the task of 3D head generation. Specifically, the authors first extend a traditional diffusion model to a text-to-normal version and a text-to-albedo version with ID-aware and expression-aware cross-attention layers. Then, with the trained diffusion models, the authors optimize a neural parametric head model with a score distillation sampling loss. Extensive experiments demonstrate that the proposed method outperforms existing text-to-3D and image-to-3D methods in terms of 3D head generation. However, I have some concerns about this paper. My detailed comments are as follows.", "questions": "Please refer to the weakness section.", "limitations": "Yes.", "rating": 5, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "**Positive points\n1.\tThe authors introduce the first method for arcface-conditioned generation of 3D heads with score distillation sampling loss.\n2.\tThe proposed method can also achieve ID-conditioned text-based 3D head editing (e.g., age editing, changing hair color and gender).", "weaknesses": "1.\tAlthough the proposed method generates a similar geometry to the input identity, the synthesized texture appears much unrealistic. What might be the cause of this phenomenon? Additionally, some implicit 3D representations, like NeRF [A-C] and 3DGS [D-F], can model high-fidelity surfaces for 3D heads. Why do the authors choose DMTET over these representations? It would be better if the authors could provide more discussion about the above questions.\n2.\tThe authors use five images as identity references for each 3D head. How is this optimal number determined? What is the relationship between identity similarity and the number of reference images? Quantitative results in terms of this should be provided.\n3.\tThere are some misalignments between Table 1, Figure 3, and Figure 4. For example, Fantasia3D is included only in Table 1 but does not appear in Figure 3 or the right column of Figure 4..\n4.\tIn the original paper of Fantasia3D [G], the proposed method can only synthesize 3D assets given a text prompt as input. How do the authors adjust this methods to generate 3D heads when conditioned on specific identity? \n5.\tFor each 3D head asset, the authors extract identity features from multiple RGB images. How are these features combined? Are they added or concatenated? It would be helpful to provide more details about this operation.\n**Minor issues\n1.\tOn page 7, line 263, there is a missing space between “ID-to-3D” and “as”.\n**Reference\n[A] NeRF: Representing Scenes as Neural Radiance Fields for View Synthesis. ECCV 2020.\n[B] Implicit and Disentangled Face Lighting Representation Leveraging Generative Prior in Neural Radiance Fields. TOG 2023.\n[C] Geometry-enhanced Novel View Synthesis from Single-View Images. CVPR 2024\n[D] 3D Gaussian Splatting for Real-Time Radiance Field Rendering. SIGGRAPH 2023.\n[E] Photorealistic Head Avatars with Rigged 3D Gaussians. CVPR 2024.\n[F] Relightable Gaussian Codec Avatars. CVPR 2024.\n[G] Disentangling Geometry and Appearance for High-quality Text-to-3D Content Creation. ICCV 2023.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "# 2 Response to reviewer `HnyJ`\nWe thank the reviewer for recognizing the novelty of our method and appreciating the experimental section. Below we respond to the doubts put forward by the reviewer. **We regularly refer to the general response above and the one-page pdf provided with the figures.** \n## 2.1 Details about the method.\nWe provide the additional requested clarifications: \n### 2.1.1 Cause of unrealistic texture.\nPlease refer to Sec.1.1 of the above general response.\n### 2.1.2 Choice of 3D representation.\nPlease refer to Sec.1.2 of the above general response.\n\n## 2.2 Relationship between identity similarity and number of reference images.\nWe analyze the relationship between identity similarity and the number of reference images in Fig.B. We consider 40 identities, 25 in-the-wild images for each subject, and extract for each image its identity embedding (ArcFace) [1]. For each identity, we consider the center of the distribution as representative of its facial features. We report the similarity between the center of the distribution and the mean ArcFace created with a subset of N number of reference images.\nThe plot shows the averaged trend for 40 identities (blue line) together with its standard deviation (light blue).\nThe trend reaches a plateau after 20 images, while 5 images is enough to reach an identity similarity of more than 0.95 for all the IDs considered. In our experiments, we selected 20 images to use as references for our comparisons and kept 5 images to use as input to our method, ensuring a good trade-off between identity similarity retention and practicality.  \n\n## 2.3 Misalignments in Figures 3,4 and Table 1.\nWe clarify the misalignments:\n- We perform a quantitative evaluation on all the evaluated models using similarity ID (Fig.4 left) FID on geometry and texture (Table 1).\n- We compare the best 2 performing text-to-3D and image-to-3D methods against ID-to-3D in the user survey (Fig.4 right), as this enabled us to gather more responses.\n- We omit the worst-performing method (Fantasia3D) in the qualitative comparisons (Fig.3) due to space constraints. \nWe included a qualitative comparison against Fantasia3D in Fig.F. We will amend the revised version of the paper to include comparisons and clarification.\n\n## 2.4 Creation of Fantasia3D assets.\nStable Diffusion textual embeddings struggle to represent a specific identity but have knowledge of named celebrities included in the training data, which can be generated by prompting \"name + surname\". To compare ID-to-3D and text-to-3D methods, we create a dataset of celebrity names suggested by ChatGPT and use the textual prompt \"A DSLR portrait of [name + surname]\" to create Fantasia3D / TADA / Human-Norm / DreamFace 3D assets. \n\n## 2.5 Identity features concatenation.\nFor each image, we extract the ArcFace identity embedding after cropping and centering. The identity embeddings are then concatenated and processed by a shallow 2-layer MLP to match the dimension of the text features in the pretrained diffusion model. This representation serves as identity conditioning for the geometry-oriented and albedo-oriented 2D diffusion models.\n\n[1] Arcface: Additive angular margin loss for deep face recognition. CVPR 2019", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "yLRQBp7WvG", "reviewer": "Reviewer_vm8B", "summary": "This work proposes a new approach for the generation of 3D human heads, which enables guidance with identity, facial expressions, and text descriptions. The approach is structured around two principal components: 1) the authors fine-tune a previously established text-to-image diffusion model through LORA on a specialized dataset of 3D human head models, to obtain the 2D guidance with separated texture and geometric details. 2) the method executes the generation of geometry and texture in separate stages, utilizing the SDS loss to optimize the process. It considers specific designs, such as the learnable latent code of facial expression, to enrich the geometry and texture details. It obtains better performance than several existing works.", "questions": "There are a few questions that should be addressed in the rebuttal, please see paper weakness for more information.\n\n******** after rebuttal\nThanks for providing additional experiments, that addressed some of the concerns. I raised my rating to borderline accept, mainly for its contribution of providing a relatively complete method for simultaneously controlling facial ID, expressions, and characteristics.\n\nYet, the generated head, especially the texture, is still low in quality. Although this may be influenced by the dataset used, it is still a limitation of the method as there is currently no better dataset available (per the author's rebuttal), and it is also unlikely to construct a higher-quality dataset. Is there any other possible ways to further improve the image quality?", "limitations": "yes", "rating": 5, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "1. The proposed method is reasonable. The finetuning of the diffusion model on a specific 3D head dataset provides better guidance for geometry and texture.\n2. It supports various types of conditional inputs, including identity, expression, and text description, thereby enriching its versatility in application.\n3. It obtains better performance than several existing works.", "weaknesses": "1. The generated head is of low visual quality. We are aware that text-to-image diffusion models are capable of producing very high-resolution images. However, the learned facial textures (resolution, clarity) showcased in this study are relatively poor. In comparison, other single-image 3D reconstruction methods, such as 3D GAN inversion combined with technologies like NeRF, can achieve very high visual quality. The text guided 3D portrait generation method, Portrait3D (siggraph 24), is also of high visual quality. \n2. The innovativeness of the method is modest. Fine-tuning diffusion models on specific datasets and using SDS as a supervisory loss for 3D modeling are both fairly common practices. The methodological innovation in this study seems insufficient.\n3. The expressions and text-guided editing scenarios demonstrated in the experimental section are quite basic (such as \"eyes closed,\" \"brow lowerer,\" \"de-aged\"), which limits their practicality. It is suggested to showcase more practical editing effects, such as changes in hairstyle or face shape, richer text-based guidance, to better understand its editing performance.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "# 3 Response to reviewer `vm8B`\nWe thank the reviewer for the feedback. We thoroughly clarify and respond to all the concerns raised.  **We regularly refer to the general response above and the one-page pdf provided with the figures.**\n## 3.1 Discussion and comparisons on the quality of the generated textures.\nWe address the 3 comparisons asked by the reviewer in the following:\n\n### 3.1.1 Comparison with Stable-Diffusion text-to-image models. \nIn Fig.H, we provide comparisons of our specialized 2D models against two well-established Stable-Diffusion text-to-image models usually used in SDS pipelines [7,8].\nAs visible, these Stable-Diffusion models have three main shortcomings that limit their applicability to the task at hand (i.e. creation of ID-driven 3D heads with expression control):\n- __Low ID retention.__ The models struggle to consistently create outputs of specific identities since they rely only on textual prompts. \n- __Low expressivity.__ The use of natural language to enforce expression conditioning is ineffective. The models overlook expression-related prompts to boost photorealism.\n- __Inconsistent Lighting.__ The models generate a wide range of lighting conditions, to enhance photorealism and artistic effect. This complicates the separation of lighting and albedo contributions when creating renderable-ready assets with SDS. \n\nPlease refer to Sec.1.1 of the above general response for a more in-depth discussion on the impact of the NPHM dataset bias on texture quality.\n\n### 3.1.2 Comparison with 3D-aware image synthesis methods.\nPlease refer to Sec.1.2 of the above general response for a description of the advantages of our 3D representation against single-image 3D reconstruction methods that use 3D GAN inversion combined with NeRF.\n\n### 3.1.3 Comparison with Portrait3D (siggraph 24).\nWe acknowledge this concurrent work (as per NeurIPS guidelines) published on 19 July 2024 in the ACM Transactions on Graphics (TOG), Volume 43, Issue 4. We will include and discuss it in the main manuscript.\n\n## 3.2 Limited Innovation.\nAs noted by `G3ux`, `HnyJ` and `fD9E`, our approach introduces significant advancements in 3D head generation, addressing key challenges in the field. In particular, we present to the reviewer 2 overlooked contributions, further substantiating the innovativeness of our model:\n\n- __Production of editable identity-driven 3D heads with expression control via SDS.__ To the best of the author's and other reviewers' knowledge, we propose the first SDS model producing identity-driven, editable, and highly detailed 3D heads with expression control from in-the-wild images of subjects without the need for large-scale 3D datasets, which allow us to set a new SoTA for human heads generation via SDS.\n\n- __Design of a neural parametric expression model compatible with SDS pipelines.__ This methodological innovation allows ID-to-3D to disentangle expressions from identity in 3D assets, a challenge in SDS research [1,2]. The control over facial expressions ease editing while ensuring consistent identity preservation across diverse 3D models.\n\n## 3.3 Expression and text-guided editing capability.\nWe address the 2 concerns raises below:\n### 3.3.1 Expression Conditioning.\n> The expressions [...] demonstrated in the experimental section are quite basic.\n\nOur expression controls can convey extreme expressions (e.g. ’squeeze’ and ’cheeks puffed’), as well as handle subtle changes from the neutral reference (e.g. ’dimpler’ and ’lip roll’) with unprecedented levels of geometric detail and expression-conditioned wrinkles (Fig.5 and 6 of the main paper).\\\nAs highlighted by `G3ux` and `fD9E`, our method goes beyond basic scenarios and shows promising results compared to relevant literature:\n1) The capability to generate 3D models with disentangled expressions has been a challenge in previous SDS research [1, 2]\n\n2) Managing the complexity of facial dynamics while ensuring identity consistency in generated 3D heads is a well-known problem in literature, with specialized methods like [3, 4, 5, 6] all struggling to create expression-driven wrinkles.\n\n### 3.3.2 Text-Guided Editing.\n> The [...] text-guided editing scenarios demonstrated in the experimental section are quite basic.\n\nAs requested, we provide evidence on text-guided editing capability of our model beyond basic scenarios. Fig.A presents the text-guided editing scenarios suggested by the reviewer. In particular, we showcase:\n- 11 unique rich text prompts associated with 6 hairstyles (Fig.A1-8, A10, A11).\n- 2 head accessories (Fig.A3, A4, A9).\n-  5 face shape changes driven by 3 different ethnicities (Fig.A1, A2, A5, A6, A9).\n\nNote that :\n-  Our method can interpret and exploit text-based inputs not addressed in previous works (e.g. id-driven changes in 'aging', 'gender', 'heritage'). \n\n-  Even when using the exact same text prompt (Fig A1, A2) our model generates unique identity-consistent assets that simultaneously align with the text and retain the characteristic facial features of the input ID.    \n\nOur approach is able to address practical scenarios and opens new avenues for expressive text-guided editing of 3D assets.\n\n[1] HumanNorm. CVPR 2024.\\\n[2] HeadSculpt. NeurIPS 2023.\\\n[3] Learning neural parametric head models. CVPR 2023.\\\n[4] DECA: Detailed Expression Capture and Animation. SIGGRAPH 2021.\\\n[5] SMIRK: 3D Facial Expressions through Analysis-by-Neural-Synthesis. CVPR 2024.\\\n[6] GANHead: Towards Generative Animatable Neural Head Avatars. CVPR 2023.\\\n[7] Stable Diffusion 2.1. [https://huggingface.co/stabilityai/stable-diffusion-2-1](https://huggingface.co/stabilityai/stable-diffusion-2-1).\\\n[8] DreamLike-Photoreal 2.0. [https://huggingface.co/dreamlike-art/dreamlike-photoreal-2.0](https://huggingface.co/dreamlike-art/dreamlike-photoreal-2.0).", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "6BrdzoitXl", "reviewer": "Reviewer_G3ux", "summary": "The paper presents a novel technique for creating 3D human head models from a single real-world image, guided by identity and text. This method is based on compositionality and uses task-specific 2D diffusion models as optimization priors. The authors extend a base model and fine-tune only a small training parameters to create 2D priors for geometry and texture generation. Additionally, the method utilizes a neural parametric representation for expressions, allowing the creation of highly detailed geometry and albedo textures.", "questions": "* Discussing the diversity of the dataset during training.\n* Discussing the robustness of id embeddings\n* Correcting citations.\n* Explaining why fine-tune using a geometry diffusion model with style-transfer instead of another geometry diffusion model.", "limitations": "The authors acknowledged the limitations and deliberated on the possible negative impacts of the suggested technology on society.", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "1. The method demonstrates significant innovation in the field of 3D head generation, particularly in generating high-quality models without the need for large-scale 3D datasets.\n2. The capability to generate 3D models with disentangled expressions is a notable advancement, as this has been a challenge in previous research.", "weaknesses": "1. The paper should discuess dataset diversity. Does the training dataset used in this study possess enough diversity to prevent potential biases?\n2. There's a lack of analysis on the robustness of ID embeddings. Are these identity embeddings robust enough to accurately depict identity features across various expressions and poses?\n3. Some citations are missing, specifically at line 191 where template-based approaches require references.\n4. Is it logical for geometry diffusion to be fine-tuned using style-transfer methods? It could be reasonable if a pre-trained standard SD model like RichDreamer[1] is utilized.\n5. Some results aren't convincing; for instance, in the last row of Fig9 within supplementary materials, reconstruction seems to lose eye-catching hair details.\n6. Supplementary materials have been placed in a separate zip file instead of being attached to the main paper, which might breach some submission rules.\n\n[1] RichDreamer: A Generalizable Normal-Depth Diffusion Model for Detail Richness in Text-to-3D", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "# 4 Response to reviewer `G3ux`\nWe appreciate the reviewer's thorough feedback and the definition of our work as a \"significant innovation in the field of 3D head generation\". We provide all the requested clarification and experiments below. **We regularly refer to the general response above and the one-page pdf provided with the figures.**\n\n## 4.1 Discussion on dataset diversity.\nPlease refer to Sec.1.3 of the above general response.\n\n## 4.2 Analysis of the robustness of ID embeddings.\nWe provide analyses on the robustness of the ID embeddings w.r.t. expressions and camera poses in Fig.C and Fig.D, which we will include in our work.\\\nWe use our training dataset (NPHM dataset). For each identity and expression, we collect renders for 9 different camera rotation angles [-60°, -45°, -30°, -15°, 0°, 15°, 30°, 45°, 60°] and extract their identity embedding (ArcFace [1]). Examples of renders are visible in Fig.G. We use a neutral pose with a 0° rotation angle as a reference.\\\nNote that, per definition, a similarity score above 0.5 signifies the same identity [1].\n- **The impact of expressions on ID embeddings (Fig.C)** is isolated by computing the cosine similarity between the neutral reference and all the remaining expressions captured with a rotation angle of 0°.   \n- **The impact of the camera pose on ID embeddings (Fig.D)** is isolated by computing the cosine similarity between the neutral reference and the neutral expression captured with all the 9 possible rotation angles. \n  \n\nAs visible from Fig.C and D, ArcFace reliably captures identity features across various expressions and poses, showcasing robust behavior even for extreme expressions (e.g. \"Squeeze\" Avg-SimID: 0.62) and substantial camera rotation (e.g. -60° Avg-SimID: 0.7).\n\n\n## 4.3 Missing references.\nWe added references to line 191 [2,3,4]\n\n## 4.4 Initialization of the 2D models.\nID-to-3D uses two ID-driven and task-specific guidance models during geometry and texture generation.\\\nWe create both geometry-oriented and texture-oriented models starting from the same pre-trained weights. This design choice avoids the introduction of two different initialization biases and has the goal of favoring ID consistency between the two finetuned representations (Fig.7 Additional, Fig.H rows 1 and 2).  \n\n## 4.5 Artifacts.\nDespite avoiding any Janus artifacts or major misalignments, failure cases of our ID-to-3D might involve slight misalignment between texture and geometry (e.g. Fig.9 last row in additional material).  We suggest that this is due to the lack of specific optimization for physically bounded textures and geometries and plan to improve on this portion of the pipeline in future work.\n\n[1] Arcface: Additive angular margin loss for deep face recognition. CVPR 2019\\\n[2] DreamFace. SIGGRAPH 2023\\\n[3] TADA. 3DV 2024\\\n[4] FLAME: Learning a model of facial shape and expression from 4D scans. SIGGRAPH 2017", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "IwNdJNXDhQ", "reviewer": "Reviewer_fD9E", "summary": "The paper introduces a novel method for generating 3D human heads guided by both identity and textual descriptions. The proposed method employs face image embeddings and textual descriptions to optimize a neural representation for each subject. By leveraging task-specific 2D diffusion models as priors and a neural parametric representation for expressions, the method achieves high-quality results while requiring minimal training. Extensive experiments demonstrate the proposed method could generate higher-quality 3D facial assets than state-of-the-art methods.", "questions": "See the weaknesses section.", "limitations": "Limitations are discussed in the paper.", "rating": 7, "confidence": 5, "soundness": 4, "presentation": 3, "contribution": 3, "strengths": "1. The proposed method is novel and promising in the field of 3D facial asset generation. The use of task-specific 2D diffusion models as priors for 3D head generation is a novel technique that reduces the dependency on large 3D datasets. The method incorporates a neural parametric representation to disentangle expressions from the identity, which is a reasonable way to manage the complexity of facial dynamics and ensure identity consistency in generated 3D heads. \n\n2. Experiments evaluation is strong and demonstrates the superiority of the proposed method over existing methods. The quantitative analysis and visualization are impressive. The visual results also clearly show the advantage over state-of-the-art methods. Detailed metrics and visualization highlight the method's ability to produce high-quality, identity-consistent 3D models, showcasing significant improvements in terms of details and textures. The evaluation covers various aspects such as generalization to new identities, handling of different ethnicities, and avoidance of oversmoothing, providing a thorough validation of the method's effectiveness. \n\n3. The generated 3D models are of high quality, with detailed textures and geometry. The method ensures that the identity of the input image is well-preserved in the 3D output. The versatility of the method is also demonstrated, which provides practical utility beyond simple model generation.", "weaknesses": "1. The generated 3D head models, while detailed, sometimes appear exaggerated and more like caricatures than realistic human faces. This diminishes the method's applicability in scenarios where photorealism is critical. Addressing this issue would involve refining the model to balance between preserving identity and achieving realistic facial features. \n\n2. The performance and quality of the method on significantly larger and more complex datasets are not extensively discussed. Evaluating and demonstrating the scalability of the approach would enhance its credibility and applicability. Discussing potential limitations when scaling up and providing strategies to handle larger datasets would be valuable additions. It would be interesting to discuss or show what could be further improved if larger datasets are incorporated. \n\n3. The writing can be improved for better clarity and accessibility. For instance, it should be explicitly described that each identity requires a training stage. This would help readers understand the necessity and implications of the training process. Additionally, simplifying and clarifying the description of the multi-stage pipeline would make the methodology more accessible to other researchers and practitioners.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "# 5 Response to reviewer `fD9E` \nWe thank the reviewer `fD9E` for recognizing our work as \"novel and promising\", acknowledging its \"superiority [...] over existing methods\" and appreciating its \"versatility\". Below we address the proposed concerns about the limitations of our method and its potential extensions.  **We regularly refer to the general response above and the one-page pdf provided with the figures**\n\n## 5.1 Bias towards exaggerated features.\nIndeed, the current pipeline exhibits a bias towards exaggerated features. We identify 3 main causes of this phenomenon:\n\n1. **Dataset size**. We train our 2D guidance model on the NPHM dataset, which is characterized by a relatively small number of subjects. Consequently, it may under-represent the wide spectrum of human head shapes and textures, resulting in output that may appear cartoonish. We provide a breakdown of dataset diversity in Fig.E and a more detailed comment in Sec.1.3 of the general response.\n\n2. **Dataset texture quality**. While the NPHM dataset excels in expressive details, it lacks high-quality texture scans. As a result, the generated 3D model textures may appear unnatural in some cases. Examples of NPHM renders and our generated 2D textures can be seen in Fig.G and H respectively, and a more detailed discussion can be found in Sec.1.1 of the above general response.\n\n3. **Geometry generation pipeline**. Our method employs a sequence of random camera poses to guide the geometry generation process. However, this can introduce point-of-view biases, leading to certain portions of the 3D model being overemphasized. \n\nConsidering the above observations, we theorize that using a larger-scale dataset could address the issue by improving the photorealism of the generated 3D head models, and recognize that it represents one of the immediate next steps to further advance our work.\n\n## 5.2 Discussion on scaling up to larger and more complex datasets. \nScaling up our SDS pipeline to exploit larger or more complex datasets is an interesting problem, as long as such datasets are available.\\\nWe identify 2 potential next steps:\n- **Extending the current NPHM dataset** by acquiring more scans to overcome its distribution bias as discussed in the reply above. This extension would not entail any adjustments to the method but would improve its generalization capabilities.\n- **Leverage more complex data types** by combining datasets with temporal signals (e.g. video) and very high-quality textures (e.g. lightstage data). Merging different data types into our pipeline would require method adjustments. One solution could be to employ a proxy representation, such as 3DMMs or a UV texture template, to allow for easy segmentation of semantics (i.e. hair/face subdivisions, etc.), that could be easier to link with large-scale SD textual embeddings. In this direction, we identify the recently released AVA256 ( 13th June 2024) [1] as a good candidate dataset to use in future work.\n\n## 5.3 Improve the writing  for better clarity and accessibility.\nWe appreciate and agree with the reviewer's recommendations to improve the manuscript. We will revise and update the text as suggested to improve clarity.\n\n[1] [https://about.meta.com/realitylabs/codecavatars/ava256](https://about.meta.com/realitylabs/codecavatars/ava256)", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:
We sincerely appreciate the reviewers’ insightful feedback, which acknowledges the novelty of our method (G3ux, HnyJ, fD9E) and the quality of our experimental results (G3ux, HnyJ, fD9E, vm8B). The reviewers recommended additional experiments and visualizations to emphasize strengths, address limitations, and highlight future improvements. We have diligently conducted all the suggested experiments, as detailed in our responses to both general and reviewer-specific comments. **To prevent confusion with the original submission’s figures, we have labeled the new figures in the attached document using alphabetical letters.**

# 1 General Response
## 1.1 Generation of unrealistic textures. [ Reviewers `HnyJ`, `vm8B`, `fD9E` ]
ID-to-3D creates the textures of the expressive 3D heads with 2D guidance specialized in pseudo-albedo generation. This 2D model reliably generates ID-driven images under consistent and diffuse lighting conditions (Fig.H row 1), while also allowing direct control over the subject's expression -- two features absent in previous text-to-image SD models [1, 2]  (Fig.H rows 3 and 4).
We train our model using the NPHM dataset [3], which is rich in expression data and geometric quality. \
Nevertheless, we acknowledge 2 of its main shortcomings that affect the synthesized texture quality:
- **Dataset bias.** Due to its limited size, it is inherently prone to biases, especially when compared to standard datasets used for SD training (e.g. Lion 5B [4], Lion-Face 20M [5]).
- **Dataset texture details.** The dataset provides scans with only UV diffuse albedo maps, which exhibit a relatively low level of texture photorealism (as visible from its renders in Fig.G).

Despite these drawbacks, NPHM remains the best choice of its type in the public domain, and given higher-quality data, our method could greatly benefit.
*The generated 3D textures resemble the NPHM suboptimal 2D albedos encountered during the training of the texture-guidance model. Nevertheless, this limitation is substantially outweighed by the capability to create images and 3D assets with control over the subject's expression and identity.*

## 1.2 Alternative 3D representations. [ Reviewers `HnyJ`, `vm8B` ]
In our 3D representation, we use DMTET for geometry generation and a Transformer predicting spatially varying reflectance during texture generation. This 3D representation offers 2 key advantages:

- **Combines benefits from implicit and explicit 3D representations** [6]. This allows the generation of high-frequency details and expression-dependent wrinkles while directly producing render-ready 3D heads (i.e. textured 3D meshes). 
- **Disentangles geometry and appearance generation.** It allows a separate optimization of geometry and appearance, ensuring flexibility in editing and mitigating the generation of artifacts.

Compared to other methods:
- **3D-aware image synthesis methods** [7,8,9,10] focus on generating novel 2D views of input images by using 3D GAN inversion combined with NeRF-based representation. They generally struggle to handle side or back views effectively and do not produce textured meshes that can be used as-is in downstream applications. We consider this line of work orthogonal to ours.

- **NeRF-based methods** [11] jointly optimize density and RGB color for a given scene. Similarly, **3DGS-based methods** [12, 13, 14] optimize at once explicit 3D gaussians and their spherical harmonics for appearance. Although they achieve high visual quality when trained on pixel-perfect views of a scene, when used in SDS pipelines they both a) struggle to perform effective surface recovery of high-frequency details and b) train intertwined geometry and appearance representation that cannot be easily manipulated.

We agree that other representations also offer interesting possibilities for future research.
In particular, 3DGS shows very good results in human 3D reconstruction from lightstage data [14] or high-quality videos [13]. How to extend the 3DGS representation to achieve photorealistic generation from in-the-wild images using SDS is an open but interesting line of research.

## 1.3 Dataset bias. [ Reviewers `G3ux`, `fD9E` ]
We appreciate the insightful questions and agree with the importance of discussing dataset diversity and minimizing potential biases. 
We provide a detailed breakdown of the NPHM dataset [3] in Fig.E. During the 2D guidance training, we use gender, age, ethnicity, and hairstyle as textual prompts to guide generation. As a result, our method generates diverse identities, ethnicities, and ages (Fig.A, Figures of the main paper). We are aware and acknowledge the limitations inherent in a small-sized dataset, such as the underrepresentation of minorities, and we revise the main manuscript to add clarifications and highlight this critical issue. 

[1] Stable Diffusion 2.1. https://huggingface.co/stabilityai/stable-diffusion-2-1.

[2] DreamLike-Photoreal 2.0. https://huggingface.co/dreamlike-art/dreamlike-photoreal-2.0.

[3] Learning neural parametric head models. CVPR 2023.

[4] LAION-5B: An open large-scale dataset for training next generation image-text models. NeurIPS 2022.

[5] General Facial Representation Learning in a Visual-Linguistic Manner. CVPR 2022. 

[6] Deep Marching Tetrahedra. NeurIPS 2021. 

[7] Implicit and Disentangled Face Lighting Representation Leveraging Generative Prior in Neural Radiance Fields. TOG 2023.

[8] Geometry-enhanced novel view synthesis from single view images. CVPR 2024.

[9] Efficient Geometry-aware 3D Generative Adversarial Networks. CVPR 2022.

[10] PanoHead: Geometry-Aware 3D Full-Head Synthesis in 360°. CVPR 2023.

[11] NeRF: Representing Scenes as Neural Radiance Fields for View Synthesis. ECCV 2020.

[12] 3D Gaussian Splatting for Real-Time Radiance Field Rendering. SIGGRAPH 2023. 

[13] Photorealistic Head Avatars with Rigged 3D Gaussians. CVPR 2024.

[14] Relightable Gaussian Codec Avatars. CVPR 2024.

Meta-review: This paper presents a method for ID and expression preserving 3D head synthesis using pre-trained diffusion models. To accomplish this task, the authors propose to finetune a 2D diffusion model into a normal and an albedo prediction models using a smaller dataset using NPHM with ID and expression controls added in. The authors propose to them use these diffusion models to learn a DMTET style-mesh and texture representation for heads, which can be relit and controlled for expressions. The results show improvements in terms of geometric quality and expression control versus the existing SOTA approaches using pre-trained diffusion models and SDS loss.

Reviewers noted the low texture quality and cartoonish appearance of the generated results as a limitation, which the authors acknowledge as stemming from the diffuse albedo of the models in NPHM. Other reviewers also noted the lack of comparisons to recent Nerf and Gaussian splatting based approaches, which are more photorealistic in quality, which the authors correctly pointed out as not being relightable. Notwithstanding these limitations, the approach does present the best in class accuracy, and is a step forward in advancing the generation of ID and expression controlled 3D faces from diffusion models. Hence, all reviewers recommend accepting this paper. The AC concurs and recommends acceptance.
Decision: Accept (poster)