Title: ID-to-3D: Expressive ID-guided 3D Heads via Score Distillation Sampling
Venue: NeurIPS 2024
Accepted: yes
Keywords: Score Distillation Sampling

3D Heads

ID-conditioning

ID-to-3D
Abstract: We propose ID-to-3D, a method to generate identity- and text-guided 3D human heads with disentangled expressions, starting from even a single casually captured ‘in-the-wild’ image of a subject. The foundation of our approach is anchored in compositionality, alongside the use of task-specific 2D diffusion models as priors for optimization. First, we extend a foundational model with a lightweight expression-aware and ID-aware architecture, and create 2D priors for geometric and texture generation, via fine-tuning only 0.2% of its available training parameters. Then, we jointly leverage a neural parametric representation for the expression of each subject and a multi-stage generation of highly detailed geometry and albedo texture. This combination of strong face identity embeddings and our neural representation enables accurate reconstruction of not only facial features but also accessories and hair, and can be meshed to provide render-ready assets for gaming and telepresence. Our results achieve an unprecedented level of id-consistent and high-quality texture and geometry generation, generalizing to a ‘world’ of unseen 3D identities, without relying on large 3D captured datasets of human assets.

ID-to-3D: Expressive ID-guided 3D Heads via Score Distillation Sampling

Abstract

We propose ID-to-3D, a method to generate identity- and text-guided 3D human heads with disentangled expressions, starting from even a single casually captured in-the-wild image of a subject. The foundation of our approach is anchored in compositionality, alongside the use of task-specific 2D diffusion models as priors for optimization. First, we extend a foundational model with a lightweight expression-aware and ID-aware architecture, and create 2D priors for geometry and texture generation, via fine-tuning only 0.2% of its available training parameters. Then, we jointly leverage a neural parametric representation for the expressions of each subject and a multi-stage generation of highly detailed geometry and albedo texture. This combination of strong face identity embeddings and our neural representation enables accurate reconstruction of not only facial features but also accessories and hair and can be meshed to provide render-ready assets for gaming and telepresence. Our results achieve an unprecedented level of identity-consistent and high-quality texture and geometry generation, generalizing to a 'world' of unseen 3D identities, without relying on large 3D captured datasets of human assets.

1 Introduction

The remarkable ability of humans to discern facial characteristics and emotional cues in others makes the development of high-quality 3D head avatars a challenging yet foundational task for a diverse array of emerging applications, including digital telepresence, game character generation, and the creation of virtual and augmented reality experiences. However, the acquisition of such 3D human assets remains a daunting task, that requires either manual work typically performed by graphic artists, or expensive and laborious scanning. High-quality 3D facial scanning, achieved initially

"A

DSLR

face

azaanbs,=dxa

portrait."

exp

Pg

Pa

exp="eyesclosed”

ext

G

exp="browraised”

0000...000

000...0000

g(c, 0g)

g(c,e,

exp="dimpler

Geometry

Texture

Generation

Generation

umopasou=dxa

0g

Oa

exp

Training

Inference

E x pr e s s i on  a n d  c o l

our  e d i t i ng

Rel i gh t i n g

D

e-agi n g

by hardware with controlled polarized illumination [15, 51, 23], has been simplified tremendously using simpler devices, color-space methods, and inverse rendering [25, 39, 3]. Nevertheless, the requirements for expert knowledge, hardware, and computational time do not allow wide adoption or mass usage.

Consequently, statistical modeling and the rise of deep learning investigated techniques to reconstruct 3D face assets from casually captured images of a subject in arbitrary poses, lighting conditions, and occlusions (referred to as 'in-the-wild'). 3D Morphable Models (3DMMs) and Generative Adversarial Networks (GANs) can be used to model facial geometry [4, 19, 56, 24], while GANs and diffusion models have achieved state-of-the-art modeling and reconstruction of facial appearance [40, 22, 92, 60].

However, a common denominator in all the above is the requirement for vast datasets of scanned facial shapes and appearances, in order to avoid limited generalization, ethnicity under-representation, and oversmoothed geometries, requiring up to 10 , 000 scans for 3DMMs training [6] or facial appearance modeling [22]. Moreover, the requirement for aligned data in 3D, and also in UV texture maps, limits the utilized area to the facial region, and introduces registration errors and additional costs. Our proposed method bypasses these significant issues by utilizing large 2D generative models, pre-trained on vast and easy-to-acquire 2D images, and using only a small set of rasterized 3D data for finetuning, while achieving state-of-the-art 3D head generation.

More recently, the use of Score Distillation Sampling (SDS) [63] and large-scale diffusion models explored the automatic generation of 3D content thanks to text [85, 10, 45, 44, 31, 92, 37, 26] or image prompts [66, 91, 64, 79, 80]. Despite promising results, leveraging 2D priors to generate realistic 3D head avatars remains challenging: (1) Large text-to-image foundational models are usually trained to generate realistic RGB images, where geometry, texture, and lighting cannot be individually segregated. This narrow 'rendering knowledge" in the 2D guidance compromises the 3D fidelity, texture quality, and consistency of the generated assets, leading to unrealistic geometries and distorted textures with recognizable artifacts such as Janus problems, incorrect proportions, oversaturated albedo, and mismatching texture and geometry details. (2) It is difficult to precisely control facial attributes solely using typical prompting methods [13, 59, 88, 84]. Textual prompts lack the granularity to single out the specificity of a subject's identity and facial expression that might be lengthy and complex to convey in natural language. Moreover, methods that leverage image prompts lack the capability to capture features that represent the facial identities of a subject independently of pose, expression, or contextual scene information (i.e., ID embeddings). The lack of reliable control over identity and facial expression prevents the personalization of 3D head avatars, resulting in a limited range of possible output expressions or custom attributes without incurring identity drifting.

In this work, we present ID-to-3D, a new method to generate expressive and identity-consistent 3D human heads using a text prompt and a small set of 1-5 casually captured, in-the-wild, images of a subject. ID-to-3D creates a variety of separated yet ID-consistent expressions in a single optimization, leveraging as priors compositionality and task-specific 2D diffusion models. To ensure consistency, the identity of each subject is encoded via facial embedding features and, to encourage expressivity, emotions are disentangled via a novel ID-specific neural parametric representation. The generation of each 3D asset leverages score distillation sampling and a two-stage pipeline to create shape details, texture features, and materials. During optimization, the guidance is given by a foundational model extended into an a) identity-aware, b) task-aware and c) expression-aware variant, working as 2D prior for either geometry or texture generation. Each stage guidance is trained only once for all the potential identities, with a lightweight fine-tuning strategy changing only 0 . 2% of all the available training parameters.

Overall, we present the following contributions: (1) ArcFace-Conditioned 3D Head Asset Generation : we introduce the first method for arcface-conditioned generation of 3D head assets using SDS. (2) Novel ID-Conditioned Expressive Model : our model creates an identity-conditioned expressive representation for each subject, enabling the generation of up to 13 unique and ID-consistent expressions captured by latent codes and associated with a set of 3D assets with separate geometric, albedo, and material information. (3) Novel Text-to-2D-Normals and Text-to-2D-Albedo Models: we present a novel approach to creating ID-conditioned and expression-conditioned text-to-image models capable of generating realistically plausible normals and albedo images from a small set of 3D assets. Extensive experiments confirm that our method outperforms text-based and image-based SDS baselines by producing relight-able 3D head assets with unprecedented geometric details, superior texture quality, and able to exhibit a wide variety of expressions. Fig. 1 displays 3D heads generated with ID-to-3D.

2 Related Work

3D Human Generation and Reconstruction. Human modelling and reconstruction in the 3D domain is typically based on 3D Morphable Models (3DMMs) [4, 6, 43, 78], which model variations in human shape and appearance using PCA. The advent of deep learning enabled very expressive reconstructions [14, 68], but also extended this line of research beyond linear spaces. Neural Parametric Head models [24] explored the use of signed distance function and deformation field to generate expressive geometries. Generative Adversarial Networks have also been proposed to generate [42, 21] and reconstruct faces from 'in-the-wild' images [22, 50, 40]. Diffusion models showed impressive results in modeling skin textures [92, 60] especially when paired with large-scale, high-quality datasets of real scans.

Text-to-3D Human Generation. The creation of 3D human assets via text conditioning has seen significant progress, building on the foundations laid by advances in text-to-2D generation [65, 69, 72, 67]. Initial attempts [86, 29, 54, 73, 55, 32, 11] utilized the CLIP language model to optimize implicit or explicit 3D representations. The seminal work of DreamFusion [63] introduced the Score Distillation Sampling (SDS) loss, which uses a pre-trained 2D diffusion as prior for 3D generation. This work led to a revolution in text-to-3D generation [53, 57, 27, 85, 45, 48, 52, 47, 12] and text-to3D human generation [9, 76, 45, 53, 26, 46, 49]. DreamAvatar [8], TADA [44] and Headevolver [82] build upon the use of a template [61, 43, 5] to create 3D human avatars with controllable shapes and poses. Fantasia3D [10] separated geometry and texture training, employing DMTET [77] and PBR texture [58] for the 3D representation. HumanNorm [31] introduced the idea of training a normal diffusion model to guide high-quality geometry generation. Despite promising results, these models face challenges in generating highly detailed and expressive 3D heads, due to the inherent limitations of natural language conditioning.

Personalized Generation with Diffusion Models. Controllable generation is crucial to develop widely applicable generative models. Work in this domain includes the costumization of GANs [74, 33, 35, 36, 34] and research dedicated to steer the generation of large-scale foundation models with additional control signals [20, 70, 30, 93, 88, 71]. The use of ID embeddings as an alternative to text prompts showed promising results in 2D generation [62, 87, 84, 59], but remains underexplored in the creation of 3D human avatars with SDS. Related research supplements traditional text prompts in SDS pipelines with image-based prompts. DreamBooth3D [66] and Avatarbooth [91] proposed to train an ad hoc model to use as a guide in the generation of 3D objects or avatars. Magic123 [64] incorporate 3D and 2D priors in SDS generation. DreamCraft3D [79] proposed a hierarchical 3D

content pipeline to generate textured 3D meshes from a single unposed image. Despite encouraging results, the creation of high-quality head assets with these methods remains challenging, due to the difficulty of extracting appropriate facial features using only naive image prompting.

3 ID-to-3D

We propose a novel method for identity-driven human head generation, which utilizes a pre-trained 2D model to distill expressive head avatars with high geometric details and high-fidelity textures, avoiding the need for large-scale training on 3D datasets. As illustrated in Figure 2, starting from a subject's identity embeddings, our method trains a set of latent expression representations using a two-stage SDS pipeline. After convergence, the learned 3D representation can be used to create ID-driven expressive heads, that are ready to be used in common rendering engines.

3.1 3D Head Optimization Objective

AScore Distillation Sampling generation pipeline optimizes a 3D representation θ using a pre-trained 2D diffusion model ϕ as guidance. The pipeline optimization objective is to align the distribution of 3D asset renderings with the target distribution p ( x 0 | y text ) , created by the 2D diffusion model conditioned on an input text y text. Given the distribution of renderings under various camera conditions q θ ( x 0 | y text ) = ∫ q θ ( x 0 | y text , c ) p ( c ) d c , the optimization objective reads:

The target distribution p ( x 0 | y text ) is typically estimated by a foundational text-to-image diffusion model that approximates the distribution of natural RGB images [69] (i.e., trained over large and uncurated datasets such as LAION-5B [75]). Despite its indisputable success in creating a variety of assets, using the above objective and guidance to generate detailed 3D heads remains a complicated task. First, this target distribution might drift significantly from the distribution of natural heads rendered in realistic light and camera conditions. Second, a general one-shot guidance model does not have explicit ways to differentiate texture and geometric characteristics, which makes the creation of light-independent texture and high-quality geometry extremely challenging. Third, the naive use of text prompts limits the control over the generated head assets, since textual prompts cannot easily or exhaustively capture facial and expression features. In our pipeline's generation, we decompose the Obj. 1 into two smaller and more controllable objectives:

In the above equation, θ g represents the parameterization of the 3D geometry, z n 0 denotes normal maps, θ a denotes the parameterization of the 3D textures and z a 0 albedo textures. Conditioning is introduced in the form of textual prompt y text, identity condition y id, and expression condition y exp . The letter l denotes the lighting condition of the rendered image. The target distributions for geometry and texture generation refer to the ideal distribution of head-specific normal and texture maps, which are in practice estimated via geometry-oriented and albedo-oriented models guided by face-specific conditioning.

3.2 2D Guidance

To initiate the 3D head reconstruction, we start from the development of 2D priors capable of accurately separating texture and geometric details while, at the same time, consistently representing the facial characteristic of a subject under various expressions, conveying different emotional states. The difficulty of this task lies in its nuanced nature, exacerbated by the lack of large-scale 3D human scan datasets, which makes the capture of detailed face features and the generalization to new identities particularly difficult when training from scratch or even when naively fine-tuning from a large-scale model. To solve this challenge, we propose to explicitly model geometry and appearance domains, identity conditioning y id and expression conditioning y exp, achieving a modular separation

of otherwise entangled-together information. To overcome the need for a large-scale dataset, we leverage a small dataset of human heads with different expressions (NHPM) [24], a pre-trained stable diffusion model (SD) [69], and a selective fine-tuning strategy that affects only a minimal number of parameters, needed to accommodate these new conditionings. In practice, we use rasterized normals as a 2D proxy for geometric information and rasterized albedo as a representation of appearance information. We treat the shift from the natural image distribution towards normal maps and albedo textures as 'style-transfer' tasks, aiming to leave the content of the SD features unchanged while modifying their self-similarity information. We use Low-Rank adaptation matrices (LoRA) [30] to adjust Query Q , Key K and Value V features of the self-attention to work in the adjusted normal and albedo domains. The normal-adapted self-attention equation becomes the following:

while the albedo-adapted self-attention can be read by changing the superscript n to a . As identity representation we select the identity embeddings y id, from a state-of-the-art face recognition network [17, 16, 94], a compact vector of facial features extracted from 'in-the-wild' images of a subject. As expression conditioning, we use y exp CLIP embeddings [65] for the textual descriptor of the 23 FACS coded expressions proposed in FaceWareHouse [7]. To ensure control over the generated head and face representation during deployment, we treat the integration of identity and expression information in the baseline architecture as 'multimodal conditioning', by including their contribution in the SD cross-attention layers via IP-Adapter [88] strategy:

In the above equation, λ id and λ exp control the contribution of identity and expression conditioning. The Q = XW Q term represents the Query extracted from SD features, K exp = y exp W exp K and V exp = y exp W exp V the Key and Values extracted from the expression embedding, and K id = y id W id K and V id = y id W id V the Key and Values extracted from the identity embedding. We train only our additional parameters, leaving the rest of the model frozen, targeting 0 . 2% of the overall trainable parameters. We separately optimize a 2D prior for geometry ϕ g and textures ϕ a , using image-conditioning pairs created from the renders of the NHPM dataset under various camera poses c . The training objective for the geometric 2D prior follows the same training objective as a traditional SD model [69]:

while the analogous training objective for the 2D texture prior can be derived by changing both superscript n and subscript g to a . Note that in Equation 5, y exp indicates CLIP embeddings for the textual descriptor of the renders (i.e., camera view and subject's attributes) and does not convey face-identity. z n t indicates z n 0 noised at timestep t . Examples of generated prompt-to-images can be seen in Figure 2 as well as additional materials.

3.3 Geometry Generation

We represent an identity-specific geometry as a neural parametric head model composed of a deep marching tetrahedra (DMTET [77]) representation, additionally coupled with a set of identitydependent facial expression latent codes. Our lightweight geometric representation produces highly detailed geometry and a broad range of expressions without notable identity drift, as shown by our experiments. Compared to explicit template-based approaches [92, 44, 43], it has the flexibility to dynamically modulate the local resolution of the mesh to capture high-frequency geometrical details. In other words, it adjusts the mesh resolution of specific regions of the face to adapt to the given subject and expression.

The DMTET geometry representation uses a deformable tetrahedral grid Γ and a network Ψ g to generate a 3D asset [77]. We extend DMTET to learn multiple expressions at the same time. We model each facial expression with a learnable latent code k n exp ∈ R d exp and design the network Ψ g (Γ , k n exp ) as a Transformer [81], parameterized by ψ g learnable parameters, that processes both the deformable grid and the expression information. During the training phase, we randomly select one of the potential expressions, estimate the signed distance function (SDF) for the underlying head, and use a differentiable marching tetrahedra layer to convert the implicit representation into the explicit surface mesh for that ID and expression, compelling the Ψ g model to learn a diverse set of expressions that are consistent with the identity at hand. We supervise optimization through an

Dream Human

1Magic

Dream Human Magic

TADA*

DC3D

Ours

TADA*

DC3D

Ours

Face*

Norm*

123

Face* Norm*

123

SDS loss [63], computed using the rasterized geometry model ϕ g as 2D prior. We optimize the 3D representation θ g as follows:

In the above equation, g ( θ g , c ) represents the normals of the rendered image, created using the differentiable render g and the camera pose c , and k n exp is the randomly sampled expression code. Lastly, we combine the SDS loss with a Laplacian regularizer, to encourage smooth surfaces.

3.4 Texture Generation

Given a trained geometry model θ g , we model the texture appearance θ a . To ensure ID-aligned expression generation, we follow an analogous parameterization for the expressions in the texture domain. We represent an identity-specific appearance as a neural parametric head model composed of a pseudo-albedo prediction network Ψ a , coupled with a set of ID-dependent facial expression latent codes. We instantiate each expression as a learnable latent code k a exp ∈ R d exp and model Ψ a ( k a exp ) as a Transformer trained to predict the spatially-varying reflectance in a UV-map representation [89], namely the albedo, roughness and specularity, for each ID and expression-specific texture. We use an off-the-shelf physically-based renderer [38]. Reflectance disentanglement is an ill-posed problem, and in the absence of prior data, we use camera and illumination randomization as a regularization constraint [18, 41]. On each iteration, we sample random environment illumination maps, augmented with random Y-axis rotations. During training, we randomize the sampling of the latent expression and deploy the albedo model ϕ a SDS loss to optimize θ a :

81

Ours

83

HumanNorm

DreamCraft3D

14

12

TADA

Geometry

7

Magic123

5

6

DreamFace

Texture

10

Fantasia3D

Ours

TADA

HumanNorm

Magic123

0.0

0.2

0.4

0.6

0.8

1.0

DreamCraft3D

where g ( θ a , c , l ) represents the pseudo-albedo maps created using a differentiable render g and a sample camera pose and lighting condition.

4 Experiments

We assess the efficacy of ID-to-3D as a specialized method for ID-driven expressive human face generation in different scenarios and report comparative analysis against state-of-the-art text-to-3D and image-to-3D generation pipelines. Further analysis, implementation details and discussion can be found in additional material.

4.1 Identity Generation

We benchmark ID-to-3D against several state-of-the-art methods in the domain of SDS-based 3D face asset generation. Specifically, we consider Fantasia3D [10], three recent and similar methods specialized in text-to-human avatar generation (i.e., Human-Norm [31], TADA [44], DreamFace [92]) and two methods that leverage both text and images to create 3D assets (i.e., Magic123 [64], DreamCraft3D [79]). In order to compare together text-to-3D and image-to-3D methods, we select a test benchmark of 40 celebrity names, suggested by ChatGPT, covering actors, sports, and media personalities. We automatically download 25 images for each identity from BingImages [1]: 5 images to use as input and 20 to use as references for our comparisons. For all methods, we use the same textual prompt 'a DSLR face portrait of...' and the same input images.

Qualitative Comparisons. Results of existing methods are reported in Figure 3 under the same lighting and rendering conditions. The 3D assets created by ID-to-3D show realistic texture, sharp fine-grained details, and ID fidelity, capturing facial characteristics of the input identity without relying heavily on often ambiguous text prompts or naively lifting 2D images in the 3D domain.

Quantitative Comparisons: Identity Similarity Distribution. We perform a quantitative evaluation on all the evaluated models using similarity ID. We assess the ID fidelity of the 3D assets using the CosFace similarity metric [83, 40]. We center and align the 3D objects and collect renders in a wide range of camera positions (i.e. elevation [ -15 ◦ , +15 ◦ ] and rotation [ -65 ◦ , +65 ◦ ]). We measure the cosine similarity between the ID features of each render and a set of 20 in-the-wild images of the same identity, used as reference. We report the distribution of the identity similarity for each method in Figure 4 (Left). Note that the variance of the distribution correlates with the ID consistency of the 3D object across viewpoints. Despite being able to generate realistic skin textures, DreamFace cannot create hair or eyes by design, resulting in the lowest average similarity score. DreamCraft3D and Magic123 leverage the input image to achieve a photorealistic front-facing camera render, but struggle to create 3D consistent heads, resulting in distributions associated with the highest variance. As clearly visible from Figure 4, ID-to-3D is capable of creating realistic and consistent 3D heads, reporting the lowest variance and the highest similarity score.

Quantitative Comparisons: FID and User-Study. The evaluation of the generated 3D geometries and textures is performed using the Frechet Inception Distance (FID) metric [28]. For texture quality, the FID is calculated between the renderings and the images from Stable Diffusion V1.5 [69]. For geometry quality, FID is determined by comparing normal maps with those extracted from the NHPM test set [24]. To further substantiate our analysis, we conduct two user preference surveys, comparing our method with the 4 strongest baselines for the generation of texture and geometry. We compare

(a)Eyes closed

(b)Brow raised

(c）Brow lowerer

(d) Cheeks puffed

(g) Lip roll

(h) Squeeze

(i) Eyes closed

(j） Brow raised

(k）Brow lowerer

(l) Cheeks puffed

(n) Dimpler

(o) Lip roll

(p) Squeeze

the best 2 performing text-to-3D and image-to-3D methods against ID-to-3Din the user survey, as this enabled us to gather more responses. As shown in Table 1 and Figure 4 (Right), FID metrics and user evaluations report aligned results. Our model achieves the lowest FID scores and the highest user preference in both geometry and texture generation, showcasing together the overall stronger performance of ID-to-3Das a human-specific geometry and texture generator.

4.2 Expressive ID-conditioned Generation

ID-to-3D is specifically designed to create complex, uncommon, and subtle expressions with a level of details not previously achievable using existing SDS methods. In Figure 5 we showcase the unique ability of our method to generate a wide range of expressions that remain identifiable and yet identity-consistent.

Quantitative Analysis: Identity Similarity Distribution Across Expressions. We test the ID consistency across views and expressions by reporting the cosine similarity between the ID features of a reference render (i.e., neutral expression and front-facing camera) and the ID features captured across the remaining expressions and view points. Figure 6 (Right) shows the distribution of ID similarity computed for a set of 10 subjects. ID-to-3D consistently produces 3D assets with high ID similarity and low variance.

Quantitative Analysis: Expression Variety Visualization. A core characteristic of ID-to-3D is its robust ability to produce a wide range of unique and vivid expressions. In this section, we provide a visualization of this expression diversity. We select a subset of 13 expressions and 10 subjects, extracting for each 3D head a set of renders in a range of 9 camera poses. Then we extract the ID features for each render and project them into 2D points using t-SNE. As visible in Figure 6 (Middle), plotting these points clearly shows the heterogeneity of the generated expressions and identities. Firstly, renders associated with the same identity are grouped in distinct clusters. Furthermore, within each cluster, we observe how there is a local variation of ID features as they adapt their response to different expressions.

4.3 ID-conditioned Text-based Customization

The unique characteristics of our model allow for the customization of 3D objects without identity or expression drift. Figure 7 and 2 (Right) showcase how a variety of alterations can be imposed on the

J.Bezos

Zendaya

A.Skarsgard

C.Evans

B.Gates

Lip Roll

W.Smith

L.Liu

A.T.Joy

Brow Raised

Angry

K.West

Squeeze

Eyes Closed

Cheeks puffed

Lip Roll

Neutral Open

Eyes Closed

Nose Down

0.5

0.8

1.0

BrowLowerer

Squeeze

...asamale

...asamale

...blondehair...brownhair

Will

...asafemale...asafemale

...green hair...greenhair

Cumberbatch

Smith

Trump

Kardashian

object's geometry, given the appropriate text conditioning while maintaining the subject's general physiognomy. In particular, ID-to-3D displays aligned geometry and appearance even after editing, preserving the ability to convey different expressions (e.g. from 'eyes closed' to 'brow lowerer', from 'neutral' to 'squeeze') even after facial hair textures have been altered via textual prompt.

Figure 8 provides evidence on the text-guided editing capability of our model for richer and more complex textual prompts. In particular, we showcase text prompts associated with various hairstyles, head accessories, and face shape changes driven by different ethnic backgrounds. Note that our method can interpret and exploit text-based inputs not addressed in previous works (e.g. id-driven changes in 'aging', 'gender', 'heritage'). Even when using the exact same text prompt (e.g. 'A woman with African-American heritage, ... 70's hairstyle.') our model generates unique identityconsistent assets that simultaneously align with the text and retain the characteristic facial features of the input ID. Our approach is able to address practical scenarios and opens new avenues for expressive text-guided editing of 3D assets.

5 Conclusion and Ethical Considerations

Limitations. Despite setting a new state-of-the-art, we acknowledge ID-to-3D limitations: (1) The generalization capacity is constrained by the employed face embedding network [17], the pretrained

large-scale diffusion models [69, 88], and the finetuning of the 2D guidance models on a dataset of relatively small size, which might result in inaccurate representation biases and biases due to dataset imbalances; (2) the texture-guidance model results might come short of photorealism. The generated textures mimic the albedos from the [24] dataset, which contains expression-rich data but only low-resolution UV diffuse albedo maps; (3) the lack of specific optimization for physically bounded textures and geometries might occasionally produce unnatural exaggerated facial characteristics, hence we call them pseudo-albedo, despite producing render-ready assets; (4) the computational resources needed for PBR rendering hinder potential applications in video-driven problems.

Societal Impact. Technological advancements in automatic 3D human generation have various beneficial applications, but also raise important ethical considerations about representation and potential misuses. We advocate for responsible research and take the following steps to mitigate unauthentic reconstructions: (1) we restrict training data to the facial region only. (2) we advocate for the replacement of text-based prompts with larger ID embeddings that are sometimes lacking [90] but entail significantly less bias than methods trained only on celebrity datasets and text [65].

Conclusion. In this work, we present ID-to-3D , a novel method for expressive 3D head asset generation from one or more face images. Our method deploys a novel human parametric expression model in tandem with specialized geometry and albedo guidance, not only to create intricately detailed head avatars with realistic textures but also to achieve strikingly ID-consistent results across a wide range of expressions, setting a new benchmark in comparison to existing SDS techniques. Without having to rely on 3D captured datasets that are expensive to collect and typically biased, and without being constrained on a specific geometry template, our method can be employed by a broad range of subjects, with different features such as skin tone and hairstyle.

Acknowledgements. S. Zafeiriou and part of the research was funded by the EPSRC Fellowship DEFORM (EP/S010203/1), EPSRC Project GNOMON (EP/X011364/1) and Turing AI Fellowship (EP/Z534699/1).

References

https://www.bing.com/images/feed.

Marmoset toolbag. https://marmoset.co/toolbag/ . Accessed: 2022-05-20.

Dejan Azinovi´ c, Olivier Maury, Christophe Hery, Matthias Nießner, and Justus Thies. High-res facial appearance capture from polarized smartphone images. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition , pages 16836-16846, 2023.

Volker Blanz and Thomas Vetter. A morphable model for the synthesis of 3d faces. In SIGGRAPH '99 , 1999.

Federica Bogo, Angjoo Kanazawa, Christoph Lassner, Peter Gehler, Javier Romero, and Michael J Black. Keep it smpl: Automatic estimation of 3d human pose and shape from a single image. In Computer Vision-ECCV 2016: 14th European Conference, Amsterdam, The Netherlands, October 11-14, 2016, Proceedings, Part V 14 , pages 561-578. Springer, 2016.

James Booth, Anastasios Roussos, Stefanos Zafeiriou, Allan Ponniah, and David Dunaway. A 3d morphable model learnt from 10,000 faces. In Proceedings of the IEEE conference on computer vision and pattern recognition , pages 5543-5552, 2016.

Chen Cao, Yanlin Weng, Shun Zhou, Yiying Tong, and Kun Zhou. Facewarehouse: A 3d facial expression database for visual computing. IEEE Transactions on Visualization and Computer Graphics , 20(3):413425, 2013.

Yukang Cao, Yan-Pei Cao, Kai Han, Ying Shan, and Kwan-Yee K Wong. Dreamavatar: Text-and-shape guided 3d human avatar generation via diffusion models. arXiv preprint arXiv:2304.00916 , 2023.

Yukang Cao, Yan-Pei Cao, Kai Han, Ying Shan, and Kwan-Yee K Wong. Guide3d: Create 3d avatars from text and image guidance. arXiv preprint arXiv:2308.09705 , 2023.

Rui Chen, Yongwei Chen, Ningxin Jiao, and Kui Jia. Fantasia3d: Disentangling geometry and appearance for high-quality text-to-3d content creation. In Proceedings of the IEEE/CVF International Conference on Computer Vision , pages 22246-22256, 2023.

Yongwei Chen, Rui Chen, Jiabao Lei, Yabin Zhang, and Kui Jia. Tango: Text-driven photorealistic and robust 3d stylization via lighting decomposition. Advances in Neural Information Processing Systems , 35:30923-30936, 2022.

Yen-Chi Cheng, Hsin-Ying Lee, Sergey Tulyakov, Alexander G Schwing, and Liang-Yan Gui. Sdfusion: Multimodal 3d shape completion, reconstruction, and generation. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition , pages 4456-4465, 2023.

Siying Cui, Jiankang Deng, Jia Guo, Xiang An, Yongle Zhao, Xinyu Wei, and Ziyong Feng. Idadapter: Learning mixed features for tuning-free personalization of text-to-image models. arXiv preprint arXiv:2403.13535 , 2024.

Radek Danˇ eˇ cek, Michael J Black, and Timo Bolkart. Emoca: Emotion driven monocular face capture and animation. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition , pages 20311-20322, 2022.

Paul Debevec, Tim Hawkins, Chris Tchou, Haarm-Pieter Duiker, Westley Sarokin, and Mark Sagar. Acquiring the reflectance field of a human face. In Proceedings of the 27th annual conference on Computer graphics and interactive techniques , pages 145-156, 2000.

Jiankang Deng, Jia Guo, Evangelos Ververas, Irene Kotsia, and Stefanos Zafeiriou. Retinaface: Single-shot multi-level face localisation in the wild. In CVPR , 2020.

Jiankang Deng, Jia Guo, Niannan Xue, and Stefanos Zafeiriou. Arcface: Additive angular margin loss for deep face recognition. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition , pages 4690-4699, 2019.

Valentin Deschaintre, Miika Aittala, Fredo Durand, George Drettakis, and Adrien Bousseau. Single-image svbrdf capture with a rendering-aware deep network. ACM Transactions on Graphics (ToG) , 37(4):1-15, 2018.

Bernhard Egger, William AP Smith, Ayush Tewari, Stefanie Wuhrer, Michael Zollhoefer, Thabo Beeler, Florian Bernard, Timo Bolkart, Adam Kortylewski, Sami Romdhani, et al. 3d morphable face models-past, present, and future. ACM Transactions on Graphics (ToG) , 39(5):1-38, 2020.

Rinon Gal, Yuval Alaluf, Yuval Atzmon, Or Patashnik, Amit H Bermano, Gal Chechik, and Daniel Cohen-Or. An image is worth one word: Personalizing text-to-image generation using textual inversion. arXiv preprint arXiv:2208.01618 , 2022.

Baris Gecer, Alexandros Lattas, Stylianos Ploumpis, Jiankang Deng, Athanasios Papaioannou, Stylianos Moschoglou, and Stefanos Zafeiriou. Synthesizing coupled 3d face modalities by trunk-branch generative adversarial networks. In European conference on computer vision , pages 415-433. Springer, 2020.

Baris Gecer, Stylianos Ploumpis, Irene Kotsia, and Stefanos Zafeiriou. Ganfit: Generative adversarial network fitting for high fidelity 3d face reconstruction. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition , pages 1155-1164, 2019.

Abhijeet Ghosh, Graham Fyffe, Borom Tunwattanapong, Jay Busch, Xueming Yu, and Paul Debevec. Multiview face capture using polarized spherical gradient illumination. In Proceedings of the 2011 SIGGRAPH Asia Conference , pages 1-10, 2011.

Simon Giebenhain, Tobias Kirschstein, Markos Georgopoulos, Martin Rünz, Lourdes Agapito, and Matthias Nießner. Learning neural parametric head models. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition , pages 21003-21012, 2023.

Paulo Gotardo, Jérémy Riviere, Derek Bradley, Abhijeet Ghosh, and Thabo Beeler. Practical dynamic facial appearance modeling and acquisition. ACM Transactions on Graphics (ToG) , 37(6):1-13, 2018.

Xiao Han, Yukang Cao, Kai Han, Xiatian Zhu, Jiankang Deng, Yi-Zhe Song, Tao Xiang, and Kwan-Yee K Wong. Headsculpt: Crafting 3d head avatars with text. Advances in Neural Information Processing Systems , 36, 2024.

Ayaan Haque, Matthew Tancik, Alexei A Efros, Aleksander Holynski, and Angjoo Kanazawa. Instructnerf2nerf: Editing 3d scenes with instructions. In Proceedings of the IEEE/CVF International Conference on Computer Vision , pages 19740-19750, 2023.

Martin Heusel, Hubert Ramsauer, Thomas Unterthiner, Bernhard Nessler, and Sepp Hochreiter. Gans trained by a two time-scale update rule converge to a local nash equilibrium. Advances in neural information processing systems , 30, 2017.

Fangzhou Hong, Mingyuan Zhang, Liang Pan, Zhongang Cai, Lei Yang, and Ziwei Liu. Avatarclip: Zero-shot text-driven generation and animation of 3d avatars. arXiv preprint arXiv:2205.08535 , 2022.

Edward J Hu, Yelong Shen, Phillip Wallis, Zeyuan Allen-Zhu, Yuanzhi Li, Shean Wang, Lu Wang, and Weizhu Chen. Lora: Low-rank adaptation of large language models. arXiv preprint arXiv:2106.09685 , 2021.

Xin Huang, Ruizhi Shao, Qi Zhang, Hongwen Zhang, Ying Feng, Yebin Liu, and Qing Wang. Humannorm: Learning normal diffusion model for high-quality and realistic 3d human generation. arXiv preprint arXiv:2310.01406 , 2023.

Ajay Jain, Ben Mildenhall, Jonathan T Barron, Pieter Abbeel, and Ben Poole. Zero-shot text-guided object generation with dream fields. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition , pages 867-876, 2022.

Minguk Kang, Jun-Yan Zhu, Richard Zhang, Jaesik Park, Eli Shechtman, Sylvain Paris, and Taesung Park. Scaling up gans for text-to-image synthesis. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition , pages 10124-10134, 2023.

Tero Karras, Miika Aittala, Samuli Laine, Erik Härkönen, Janne Hellsten, Jaakko Lehtinen, and Timo Aila. Alias-free generative adversarial networks. Advances in neural information processing systems , 34:852-863, 2021.

Tero Karras, Samuli Laine, and Timo Aila. A style-based generator architecture for generative adversarial networks. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition , pages 4401-4410, 2019.

Tero Karras, Samuli Laine, Miika Aittala, Janne Hellsten, Jaakko Lehtinen, and Timo Aila. Analyzing and improving the image quality of stylegan. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition , pages 8110-8119, 2020.

Nikos Kolotouros, Thiemo Alldieck, Andrei Zanfir, Eduard Bazavan, Mihai Fieraru, and Cristian Sminchisescu. Dreamhuman: Animatable 3d avatars from text. Advances in Neural Information Processing Systems , 36, 2024.

Samuli Laine, Janne Hellsten, Tero Karras, Yeongho Seol, Jaakko Lehtinen, and Timo Aila. Modular primitives for high-performance differentiable rendering. ACM Transactions on Graphics (ToG) , 39(6):114, 2020.

Alexandros Lattas, Yiming Lin, Jayanth Kannan, Ekin Ozturk, Luca Filipi, Giuseppe Claudio Guarnera, Gaurav Chawla, and Abhijeet Ghosh. Practical and scalable desktop-based high-quality facial capture. In European Conference on Computer Vision , pages 522-537. Springer, 2022.

Alexandros Lattas, Stylianos Moschoglou, Stylianos Ploumpis, Baris Gecer, Jiankang Deng, and Stefanos Zafeiriou. Fitme: Deep photorealistic 3d morphable model avatars. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition , pages 8629-8640, 2023.

Alexandros Lattas, Stylianos Moschoglou, Stylianos Ploumpis, Baris Gecer, Abhijeet Ghosh, and Stefanos Zafeiriou. Avatarme++: Facial shape and brdf inference with photorealistic rendering-aware gans. IEEE Transactions on Pattern Analysis and Machine Intelligence , 44(12):9269-9284, 2021.

Ruilong Li, Karl Bladin, Yajie Zhao, Chinmay Chinara, Owen Ingraham, Pengda Xiang, Xinglei Ren, Pratusha Prasad, Bipin Kishore, Jun Xing, et al. Learning formation of physically-based face attributes. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition , pages 3410-3419, 2020.

Tianye Li, Timo Bolkart, Michael J Black, Hao Li, and Javier Romero. Learning a model of facial shape and expression from 4d scans. ACM Trans. Graph. , 36(6):194-1, 2017.

Tingting Liao, Hongwei Yi, Yuliang Xiu, Jiaxaing Tang, Yangyi Huang, Justus Thies, and Michael J Black. Tada! text to animatable digital avatars. arXiv preprint arXiv:2308.10899 , 2023.

Chen-Hsuan Lin, Jun Gao, Luming Tang, Towaki Takikawa, Xiaohui Zeng, Xun Huang, Karsten Kreis, Sanja Fidler, Ming-Yu Liu, and Tsung-Yi Lin. Magic3d: High-resolution text-to-3d content creation. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition , pages 300-309, 2023.

Hongyu Liu, Xuan Wang, Ziyu Wan, Yujun Shen, Yibing Song, Jing Liao, and Qifeng Chen. Headartist: Text-conditioned 3d head generation with self score distillation. arXiv preprint arXiv:2312.07539 , 2023.

Minghua Liu, Chao Xu, Haian Jin, Linghao Chen, Mukund Varma T, Zexiang Xu, and Hao Su. One-23-45: Any single image to 3d mesh in 45 seconds without per-shape optimization. Advances in Neural Information Processing Systems , 36, 2024.

Ruoshi Liu, Rundi Wu, Basile Van Hoorick, Pavel Tokmakov, Sergey Zakharov, and Carl Vondrick. Zero1-to-3: Zero-shot one image to 3d object. In Proceedings of the IEEE/CVF International Conference on Computer Vision , pages 9298-9309, 2023.

Xian Liu, Xiaohang Zhan, Jiaxiang Tang, Ying Shan, Gang Zeng, Dahua Lin, Xihui Liu, and Ziwei Liu. Humangaussian: Text-driven 3d human generation with gaussian splatting. arXiv preprint arXiv:2311.17061 , 2023.

Huiwen Luo, Koki Nagano, Han-Wei Kung, Qingguo Xu, Zejian Wang, Lingyu Wei, Liwen Hu, and Hao Li. Normalized avatar synthesis using stylegan and perceptual refinement. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition , pages 11662-11672, 2021.

Wan-Chun Ma, Tim Hawkins, Pieter Peers, Charles-Felix Chabert, Malte Weiss, Paul E Debevec, et al. Rapid acquisition of specular and diffuse normal maps from polarized spherical gradient illumination. Rendering Techniques , 9(10):2, 2007.

Luke Melas-Kyriazi, Iro Laina, Christian Rupprecht, and Andrea Vedaldi. Realfusion: 360deg reconstruction of any object from a single image. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition , pages 8446-8455, 2023.

Gal Metzer, Elad Richardson, Or Patashnik, Raja Giryes, and Daniel Cohen-Or. Latent-nerf for shapeguided generation of 3d shapes and textures. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition , pages 12663-12673, 2023.

Oscar Michel, Roi Bar-On, Richard Liu, Sagie Benaim, and Rana Hanocka. Text2mesh: Text-driven neural stylization for meshes. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition , pages 13492-13502, 2022.

Nasir Mohammad Khalid, Tianhao Xie, Eugene Belilovsky, and Tiberiu Popa. Clip-mesh: Generating textured meshes from text using pretrained image-text models. In SIGGRAPH Asia 2022 conference papers , pages 1-8, 2022.

Stylianos Moschoglou, Stylianos Ploumpis, Mihalis A Nicolaou, Athanasios Papaioannou, and Stefanos Zafeiriou. 3dfacegan: Adversarial nets for 3d face representation, generation, and translation. International Journal of Computer Vision , 128:2534-2551, 2020.

Norman Müller, Yawar Siddiqui, Lorenzo Porzi, Samuel Rota Bulo, Peter Kontschieder, and Matthias Nießner. Diffrf: Rendering-guided 3d radiance field diffusion. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition , pages 4328-4338, 2023.

Jacob Munkberg, Jon Hasselgren, Tianchang Shen, Jun Gao, Wenzheng Chen, Alex Evans, Thomas Müller, and Sanja Fidler. Extracting triangular 3d models, materials, and lighting from images. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition , pages 8280-8290, 2022.

Foivos Paraperas Papantoniou, Alexandros Lattas, Stylianos Moschoglou, Jiankang Deng, Bernhard Kainz, and Stefanos Zafeiriou. Arc2face: A foundation model of human faces. arXiv preprint arXiv:2403.11641 , 2024.

Foivos Paraperas Papantoniou, Alexandros Lattas, Stylianos Moschoglou, and Stefanos Zafeiriou. Relightify: Relightable 3d faces from a single image via diffusion models. In Proceedings of the IEEE/CVF International Conference on Computer Vision , pages 8806-8817, 2023.

Georgios Pavlakos, Vasileios Choutas, Nima Ghorbani, Timo Bolkart, Ahmed AA Osman, Dimitrios Tzionas, and Michael J Black. Expressive body capture: 3d hands, face, and body from a single image. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition , pages 10975-10985, 2019.

Xu Peng, Junwei Zhu, Boyuan Jiang, Ying Tai, Donghao Luo, Jiangning Zhang, Wei Lin, Taisong Jin, Chengjie Wang, and Rongrong Ji. Portraitbooth: A versatile portrait model for fast identity-preserved personalization. arXiv preprint arXiv:2312.06354 , 2023.

Ben Poole, Ajay Jain, Jonathan T Barron, and Ben Mildenhall. Dreamfusion: Text-to-3d using 2d diffusion. In ICLR , 2023.

Guocheng Qian, Jinjie Mai, Abdullah Hamdi, Jian Ren, Aliaksandr Siarohin, Bing Li, Hsin-Ying Lee, Ivan Skorokhodov, Peter Wonka, Sergey Tulyakov, et al. Magic123: One image to high-quality 3d object generation using both 2d and 3d diffusion priors. arXiv preprint arXiv:2306.17843 , 2023.

Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, et al. Learning transferable visual models from natural language supervision. In International conference on machine learning , pages 8748-8763. PMLR, 2021.

Amit Raj, Srinivas Kaza, Ben Poole, Michael Niemeyer, Nataniel Ruiz, Ben Mildenhall, Shiran Zada, Kfir Aberman, Michael Rubinstein, Jonathan Barron, et al. Dreambooth3d: Subject-driven text-to-3d generation. In Proceedings of the IEEE/CVF International Conference on Computer Vision , pages 2349-2359, 2023.

Aditya Ramesh, Prafulla Dhariwal, Alex Nichol, Casey Chu, and Mark Chen. Hierarchical text-conditional image generation with clip latents. arXiv preprint arXiv:2204.06125 , 1(2):3, 2022.

George Retsinas, Panagiotis P Filntisis, Radek Danecek, Victoria F Abrevaya, Anastasios Roussos, Timo Bolkart, and Petros Maragos. 3d facial expressions through analysis-by-neural-synthesis. arXiv preprint arXiv:2404.04104 , 2024.

Robin Rombach, Andreas Blattmann, Dominik Lorenz, Patrick Esser, and Björn Ommer. High-resolution image synthesis with latent diffusion models. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition , pages 10684-10695, 2022.

Nataniel Ruiz, Yuanzhen Li, Varun Jampani, Yael Pritch, Michael Rubinstein, and Kfir Aberman. Dreambooth: Fine tuning text-to-image diffusion models for subject-driven generation. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition , pages 22500-22510, 2023.

Nataniel Ruiz, Yuanzhen Li, Varun Jampani, Wei Wei, Tingbo Hou, Yael Pritch, Neal Wadhwa, Michael Rubinstein, and Kfir Aberman. Hyperdreambooth: Hypernetworks for fast personalization of text-to-image models. arXiv preprint arXiv:2307.06949 , 2023.

Chitwan Saharia, William Chan, Saurabh Saxena, Lala Li, Jay Whang, Emily L Denton, Kamyar Ghasemipour, Raphael Gontijo Lopes, Burcu Karagol Ayan, Tim Salimans, et al. Photorealistic text-toimage diffusion models with deep language understanding. Advances in neural information processing systems , 35:36479-36494, 2022.

Aditya Sanghi, Hang Chu, Joseph G. Lambourne, Ye Wang, Chin-Yi Cheng, Marco Fumero, and Kamal Rahimi Malekshan. Clip-forge: Towards zero-shot text-to-shape generation. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) , pages 18603-18613, June 2022.

Axel Sauer, Tero Karras, Samuli Laine, Andreas Geiger, and Timo Aila. Stylegan-t: Unlocking the power of gans for fast large-scale text-to-image synthesis. In International conference on machine learning , pages 30105-30118. PMLR, 2023.

Christoph Schuhmann, Romain Beaumont, Richard Vencu, Cade Gordon, Ross Wightman, Mehdi Cherti, Theo Coombes, Aarush Katta, Clayton Mullis, Mitchell Wortsman, et al. Laion-5b: An open large-scale dataset for training next generation image-text models. Advances in Neural Information Processing Systems , 35:25278-25294, 2022.

Junyoung Seo, Wooseok Jang, Min-Seop Kwak, Hyeonsu Kim, Jaehoon Ko, Junho Kim, Jin-Hwa Kim, Jiyoung Lee, and Seungryong Kim. Let 2d diffusion model know 3d-consistency for robust text-to-3d generation. arXiv preprint arXiv:2303.07937 , 2023.

Tianchang Shen, Jun Gao, Kangxue Yin, Ming-Yu Liu, and Sanja Fidler. Deep marching tetrahedra: a hybrid representation for high-resolution 3d shape synthesis. Advances in Neural Information Processing Systems , 34:6087-6101, 2021.

William AP Smith, Alassane Seck, Hannah Dee, Bernard Tiddeman, Joshua B Tenenbaum, and Bernhard Egger. A morphable face albedo model. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition , pages 5011-5020, 2020.

Jingxiang Sun, Bo Zhang, Ruizhi Shao, Lizhen Wang, Wen Liu, Zhenda Xie, and Yebin Liu. Dreamcraft3d: Hierarchical 3d generation with bootstrapped diffusion prior. arXiv preprint arXiv:2310.16818 , 2023.

Junshu Tang, Tengfei Wang, Bo Zhang, Ting Zhang, Ran Yi, Lizhuang Ma, and Dong Chen. Make-it-3d: High-fidelity 3d creation from a single image with diffusion prior. In Proceedings of the IEEE/CVF International Conference on Computer Vision , pages 22819-22829, 2023.

Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Łukasz Kaiser, and Illia Polosukhin. Attention is all you need. Advances in neural information processing systems , 30, 2017.

Duotun Wang, Hengyu Meng, Zeyu Cai, Zhijing Shao, Qianxi Liu, Lin Wang, Mingming Fan, Ying Shan, Xiaohang Zhan, and Zeyu Wang. Headevolver: Text to head avatars via locally learnable mesh deformation. arXiv preprint arXiv:2403.09326 , 2024.

Hao Wang, Yitong Wang, Zheng Zhou, Xing Ji, Dihong Gong, Jingchao Zhou, Zhifeng Li, and Wei Liu. Cosface: Large margin cosine loss for deep face recognition. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR) , June 2018.

Qixun Wang, Xu Bai, Haofan Wang, Zekui Qin, and Anthony Chen. Instantid: Zero-shot identity-preserving generation in seconds. arXiv preprint arXiv:2401.07519 , 2024.

Zhengyi Wang, Cheng Lu, Yikai Wang, Fan Bao, Chongxuan Li, Hang Su, and Jun Zhu. Prolificdreamer: High-fidelity and diverse text-to-3d generation with variational score distillation. In Advances in Neural Information Processing Systems , volume 36, 2023.

Jiale Xu, Xintao Wang, Weihao Cheng, Yan-Pei Cao, Ying Shan, Xiaohu Qie, and Shenghua Gao. Dream3d: Zero-shot text-to-3d synthesis using 3d shape prior and text-to-image diffusion models. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition , pages 20908-20918, 2023.

Yuxuan Yan, Chi Zhang, Rui Wang, Yichao Zhou, Gege Zhang, Pei Cheng, Gang Yu, and Bin Fu. Facestudio: Put your face everywhere in seconds. arXiv preprint arXiv:2312.02663 , 2023.

Hu Ye, Jun Zhang, Sibo Liu, Xiao Han, and Wei Yang. Ip-adapter: Text compatible image prompt adapter for text-to-image diffusion models. arXiv preprint arXiv:2308.06721 , 2023.

Jonathan Young. xatlas. https://github.com/jpcy/xatlas . Accessed: 2022-05-20.

Seyma Yucer, Furkan Tektas, Noura Al Moubayed, and Toby P Breckon. Measuring hidden bias within face recognition via racial phenotypes. In Proceedings of the IEEE/CVF Winter Conference on Applications of Computer Vision , pages 995-1004, 2022.

Yifei Zeng, Yuanxun Lu, Xinya Ji, Yao Yao, Hao Zhu, and Xun Cao. Avatarbooth: High-quality and customizable 3d human avatar generation. arXiv preprint arXiv:2306.09864 , 2023.

Longwen Zhang, Qiwei Qiu, Hongyang Lin, Qixuan Zhang, Cheng Shi, Wei Yang, Ye Shi, Sibei Yang, Lan Xu, and Jingyi Yu. Dreamface: Progressive generation of animatable 3d faces under text guidance. arXiv preprint arXiv:2304.03117 , 2023.

Lvmin Zhang, Anyi Rao, and Maneesh Agrawala. Adding conditional control to text-to-image diffusion models. In Proceedings of the IEEE/CVF International Conference on Computer Vision , pages 3836-3847,

2023.

Zheng Zhu, Guan Huang, Jiankang Deng, Yun Ye, Junjie Huang, Xinze Chen, Jiagang Zhu, Tian Yang, Dalong Du, Jiwen Lu, and Jie Zhou. Webface260m: A benchmark for million-scale deep face recognition. TPAMI , 2022.

1. Claims

Question: Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?

Answer: [Yes]

Justification: The claims are listed in both the abstract and introduction, reflect the paper's method and are justified by the experimental section.

2. Limitations

Question: Does the paper discuss the limitations of the work performed by the authors?

Answer: [Yes]

Justification: The limitations are listed in the limitations section.

3. Theory Assumptions and Proofs

Question: For each theoretical result, does the paper provide the full set of assumptions and a complete (and correct) proof?

Answer: [NA]

Justification: [NA]

4. Experimental Result Reproducibility

Question: Does the paper fully disclose all the information needed to reproduce the main experimental results of the paper to the extent that it affects the main claims and/or conclusions of the paper (regardless of whether the code and data are provided or not)?

Answer: [Yes]

Justification: We base our method on public code, report our technical details in the supplemental materials, and will be releasing our code.

5. Open access to data and code

Question: Does the paper provide open access to the data and code, with sufficient instructions to faithfully reproduce the main experimental results, as described in supplemental material?

Answer: [Yes]

Justification: We use already publicly available data, and will be open-sourcing our code.

6. Experimental Setting/Details

Question: Does the paper specify all the training and test details (e.g., data splits, hyperparameters, how they were chosen, type of optimizer, etc.) necessary to understand the results?

Answer: [Yes]

Justification: We include all the above details in the supplemental materials.

7. Experiment Statistical Significance

Question: Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?

Answer: [No]

Justification: Error bars are not reported because it would be too computationally expensive.

8. Experiments Compute Resources

Question: For each experiment, does the paper provide sufficient information on the computer resources (type of compute workers, memory, time of execution) needed to reproduce the experiments?

Answer: [Yes]

Justification: We have included the above details in the supplemental materials.

9. Code Of Ethics

Question: Does the research conducted in the paper conform, in every respect, with the NeurIPS Code of Ethics https://neurips.cc/public/EthicsGuidelines ?

Answer: [Yes]

Justification:

10. Broader Impacts

Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?

Answer: Yes

Justification: We discuss both in the Conclusion and Ethical Considerations section.

11. Safeguards

Question: Does the paper describe safeguards that have been put in place for responsible release of data or models that have a high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)?

Answer: [Yes]

Justification: We discuss both in the Conclusion and Ethical Considerations section.

12. Licenses for existing assets

Question: Are the creators or original owners of assets (e.g., code, data, models), used in the paper, properly credited and are the license and terms of use explicitly mentioned and properly respected?

Answer: [Yes]

Justification:

13. New Assets

Question: Are new assets introduced in the paper well documented and is the documentation provided alongside the assets?

Answer: [NA]

Justification: the paper does not release new assets.

14. Crowdsourcing and Research with Human Subjects

Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given to participants and screenshots, if applicable, as well as details about compensation (if any)?

Answer: [NA]

Justification:

15. Institutional Review Board (IRB) Approvals or Equivalent for Research with Human Subjects

Question: Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or an equivalent approval/review based on the requirements of your country or institution) were obtained?

Answer: [NA]

Justification:

Peer reviews:
{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe paper introduces a novel method for generating 3D human heads guided by both identity and textual descriptions. The proposed method employs face image embeddings and textual descriptions to optimize a neural representation for each subject. By leveraging task-specific 2D diffusion models as priors and a neural parametric representation for expressions, the method achieves high-quality results while requiring minimal training. Extensive experiments demonstrate the proposed method could generate higher-quality 3D facial assets than state-of-the-art methods.\n\nSTRENGTHS:\n1. The proposed method is novel and promising in the field of 3D facial asset generation. The use of task-specific 2D diffusion models as priors for 3D head generation is a novel technique that reduces the dependency on large 3D datasets. The method incorporates a neural parametric representation to disentangle expressions from the identity, which is a reasonable way to manage the complexity of facial dynamics and ensure identity consistency in generated 3D heads. \n\n2. Experiments evaluation is strong and demonstrates the superiority of the proposed method over existing methods. The quantitative analysis and visualization are impressive. The visual results also clearly show the advantage over state-of-the-art methods. Detailed metrics and visualization highlight the method's ability to produce high-quality, identity-consistent 3D models, showcasing significant improvements in terms of details and textures. The evaluation covers various aspects such as generalization to new identities, handling of different ethnicities, and avoidance of oversmoothing, providing a thorough validation of the method's effectiveness. \n\n3. The generated 3D models are of high quality, with detailed textures and geometry. The method ensures that the identity of the input image is well-preserved in the 3D output. The versatility of the method is also demonstrated, which provides practical utility beyond simple model generation.\n\nWEAKNESSES:\n1. The generated 3D head models, while detailed, sometimes appear exaggerated and more like caricatures than realistic human faces. This diminishes the method's applicability in scenarios where photorealism is critical. Addressing this issue would involve refining the model to balance between preserving identity and achieving realistic facial features. \n\n2. The performance and quality of the method on significantly larger and more complex datasets are not extensively discussed. Evaluating and demonstrating the scalability of the approach would enhance its credibility and applicability. Discussing potential limitations when scaling up and providing strategies to handle larger datasets would be valuable additions. It would be interesting to discuss or show what could be further improved if larger datasets are incorporated. \n\n3. The writing can be improved for better clarity and accessibility. For instance, it should be explicitly described that each identity requires a training stage. This would help readers understand the necessity and implications of the training process. Additionally, simplifying and clarifying the description of the multi-stage pipeline would make the methodology more accessible to other researchers and practitioners.\n\nQUESTIONS:\nSee the weaknesses section.\n\nLIMITATIONS:\nLimitations are discussed in the paper.", "TITLE": "Official Review", "RECOMMENDATION": 7, "REVIEWER_CONFIDENCE": 5, "SOUNDNESS_CORRECTNESS": 4, "DATE": "2024-06-16"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe paper presents a novel technique for creating 3D human head models from a single real-world image, guided by identity and text. This method is based on compositionality and uses task-specific 2D diffusion models as optimization priors. The authors extend a base model and fine-tune only a small training parameters to create 2D priors for geometry and texture generation. Additionally, the method utilizes a neural parametric representation for expressions, allowing the creation of highly detailed geometry and albedo textures.\n\nSTRENGTHS:\n1. The method demonstrates significant innovation in the field of 3D head generation, particularly in generating high-quality models without the need for large-scale 3D datasets.\n2. The capability to generate 3D models with disentangled expressions is a notable advancement, as this has been a challenge in previous research.\n\nWEAKNESSES:\n1. The paper should discuess dataset diversity. Does the training dataset used in this study possess enough diversity to prevent potential biases?\n2. There's a lack of analysis on the robustness of ID embeddings. Are these identity embeddings robust enough to accurately depict identity features across various expressions and poses?\n3. Some citations are missing, specifically at line 191 where template-based approaches require references.\n4. Is it logical for geometry diffusion to be fine-tuned using style-transfer methods? It could be reasonable if a pre-trained standard SD model like RichDreamer[1] is utilized.\n5. Some results aren't convincing; for instance, in the last row of Fig9 within supplementary materials, reconstruction seems to lose eye-catching hair details.\n6. Supplementary materials have been placed in a separate zip file instead of being attached to the main paper, which might breach some submission rules.\n\n[1] RichDreamer: A Generalizable Normal-Depth Diffusion Model for Detail Richness in Text-to-3D\n\nQUESTIONS:\n* Discussing the diversity of the dataset during training.\n* Discussing the robustness of id embeddings\n* Correcting citations.\n* Explaining why fine-tune using a geometry diffusion model with style-transfer instead of another geometry diffusion model.\n\nLIMITATIONS:\nThe authors acknowledged the limitations and deliberated on the possible negative impacts of the suggested technology on society.", "TITLE": "Official Review", "RECOMMENDATION": 6, "REVIEWER_CONFIDENCE": 4, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2024-07-10"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis work proposes a new approach for the generation of 3D human heads, which enables guidance with identity, facial expressions, and text descriptions. The approach is structured around two principal components: 1) the authors fine-tune a previously established text-to-image diffusion model through LORA on a specialized dataset of 3D human head models, to obtain the 2D guidance with separated texture and geometric details. 2) the method executes the generation of geometry and texture in separate stages, utilizing the SDS loss to optimize the process. It considers specific designs, such as the learnable latent code of facial expression, to enrich the geometry and texture details. It obtains better performance than several existing works.\n\nSTRENGTHS:\n1. The proposed method is reasonable. The finetuning of the diffusion model on a specific 3D head dataset provides better guidance for geometry and texture.\n2. It supports various types of conditional inputs, including identity, expression, and text description, thereby enriching its versatility in application.\n3. It obtains better performance than several existing works.\n\nWEAKNESSES:\n1. The generated head is of low visual quality. We are aware that text-to-image diffusion models are capable of producing very high-resolution images. However, the learned facial textures (resolution, clarity) showcased in this study are relatively poor. In comparison, other single-image 3D reconstruction methods, such as 3D GAN inversion combined with technologies like NeRF, can achieve very high visual quality. The text guided 3D portrait generation method, Portrait3D (siggraph 24), is also of high visual quality. \n2. The innovativeness of the method is modest. Fine-tuning diffusion models on specific datasets and using SDS as a supervisory loss for 3D modeling are both fairly common practices. The methodological innovation in this study seems insufficient.\n3. The expressions and text-guided editing scenarios demonstrated in the experimental section are quite basic (such as \"eyes closed,\" \"brow lowerer,\" \"de-aged\"), which limits their practicality. It is suggested to showcase more practical editing effects, such as changes in hairstyle or face shape, richer text-based guidance, to better understand its editing performance.\n\nQUESTIONS:\nThere are a few questions that should be addressed in the rebuttal, please see paper weakness for more information.\n\n******** after rebuttal\nThanks for providing additional experiments, that addressed some of the concerns. I raised my rating to borderline accept, mainly for its contribution of providing a relatively complete method for simultaneously controlling facial ID, expressions, and characteristics.\n\nYet, the generated head, especially the texture, is still low in quality. Although this may be influenced by the dataset used, it is still a limitation of the method as there is currently no better dataset available (per the author's rebuttal), and it is also unlikely to construct a higher-quality dataset. Is there any other possible ways to further improve the image quality?\n\nLIMITATIONS:\nyes", "TITLE": "Official Review", "RECOMMENDATION": 5, "REVIEWER_CONFIDENCE": 3, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2024-07-11"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\n**Summary\nThis paper focuses on the task of 3D head generation. Specifically, the authors first extend a traditional diffusion model to a text-to-normal version and a text-to-albedo version with ID-aware and expression-aware cross-attention layers. Then, with the trained diffusion models, the authors optimize a neural parametric head model with a score distillation sampling loss. Extensive experiments demonstrate that the proposed method outperforms existing text-to-3D and image-to-3D methods in terms of 3D head generation. However, I have some concerns about this paper. My detailed comments are as follows.\n\nSTRENGTHS:\n**Positive points\n1.\tThe authors introduce the first method for arcface-conditioned generation of 3D heads with score distillation sampling loss.\n2.\tThe proposed method can also achieve ID-conditioned text-based 3D head editing (e.g., age editing, changing hair color and gender).\n\nWEAKNESSES:\n1.\tAlthough the proposed method generates a similar geometry to the input identity, the synthesized texture appears much unrealistic. What might be the cause of this phenomenon? Additionally, some implicit 3D representations, like NeRF [A-C] and 3DGS [D-F], can model high-fidelity surfaces for 3D heads. Why do the authors choose DMTET over these representations? It would be better if the authors could provide more discussion about the above questions.\n2.\tThe authors use five images as identity references for each 3D head. How is this optimal number determined? What is the relationship between identity similarity and the number of reference images? Quantitative results in terms of this should be provided.\n3.\tThere are some misalignments between Table 1, Figure 3, and Figure 4. For example, Fantasia3D is included only in Table 1 but does not appear in Figure 3 or the right column of Figure 4..\n4.\tIn the original paper of Fantasia3D [G], the proposed method can only synthesize 3D assets given a text prompt as input. How do the authors adjust this methods to generate 3D heads when conditioned on specific identity? \n5.\tFor each 3D head asset, the authors extract identity features from multiple RGB images. How are these features combined? Are they added or concatenated? It would be helpful to provide more details about this operation.\n**Minor issues\n1.\tOn page 7, line 263, there is a missing space between “ID-to-3D” and “as”.\n**Reference\n[A] NeRF: Representing Scenes as Neural Radiance Fields for View Synthesis. ECCV 2020.\n[B] Implicit and Disentangled Face Lighting Representation Leveraging Generative Prior in Neural Radiance Fields. TOG 2023.\n[C] Geometry-enhanced Novel View Synthesis from Single-View Images. CVPR 2024\n[D] 3D Gaussian Splatting for Real-Time Radiance Field Rendering. SIGGRAPH 2023.\n[E] Photorealistic Head Avatars with Rigged 3D Gaussians. CVPR 2024.\n[F] Relightable Gaussian Codec Avatars. CVPR 2024.\n[G] Disentangling Geometry and Appearance for High-quality Text-to-3D Content Creation. ICCV 2023.\n\nQUESTIONS:\nPlease refer to the weakness section.\n\nLIMITATIONS:\nYes.", "TITLE": "Official Review", "RECOMMENDATION": 5, "REVIEWER_CONFIDENCE": 4, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2024-07-12"}