FaceComposer: A Unified Model for Versatile Facial Content Creation
Authors: Jiayu Wang
Kang Zhao
Yifeng Ma
Shiwei Zhang
Yingya Zhang
Yujun Shen
Deli Zhao
Jingren Zhou
Venue: NeurIPS 2023 poster (2023)
Keywords: diffusion model; talking face generation; face generation

Abstract: This work presents FaceComposer, a unified generative model that accomplishes a variety of facial content creation tasks, including text-conditioned face synthesis, text-guided face editing, face animation etc. Based on the latent diffusion framework, FaceComposer follows the paradigm of compositional generation and employs diverse face-specific conditions, e.g., Identity Feature and Projected Normalized Coordinate Code, to release the model creativity at all possible. To support text control and animation, we clean up some existing face image datasets and collect around 500 hours of talking-face videos, forming a high-quality large-scale multi-modal face database. A temporal self-attention module is incorporated into the U-Net structure, which allows learning the denoising process on the mixture of images and videos. Extensive experiments suggest that our approach not only achieves comparable or even better performance than state-of-the-arts on each single task, but also facilitates some combined tasks with one-time forward, demonstrating its potential in serving as a foundation generative model in face domain. We further develop an interface such that users can enjoy our one-step service to create, edit, and animate their own characters. Code, dataset, model, and interface will be made publicly available.

Face Compose:::::: A Unified Model for Versatile Facial Content Creation
Jiayu Wang"), Kang Zhao"!
Yifeng Ma Shiwei Zhang', Yingya Zhang',
Yujun Shen Deli Zhao', Jingren Zhou
Alibaba Group
TTinggha University
Ant Group
(wyn vjy, zbeokang... 23, mayifeng...e.... zhangj In. www. wew.yingya.ckk yingya.ayy ayy.jingren. (tlou//olibbbbiic.coo
(ehenyujun0302) phaodeli)0gmaaa coo
Abstract
This work presents FaceComposer, unified generative model that accomplishes a variety of facial content creation tasks, including text-conditioned face synthesis, text-guided face editing, face animation etc. Based the latent diffusion framework, FaceComposer follows the paradigm of compositional generation and employs diverse face-specific conditions, e.g., Identity Feature and Projected Normalized Coordinate Code, to release the model creativity at all possible. To support text control and animation, clean up some existing face image datasets and collect around 500 hours of talking- face videos, forming high-quality large- scale multi-modal face database. temporal self-attention module is incorporated into the U-Net structure, which allows learning the denoising process on the mixture images and videos. Extensive experiments suggest that our approach not only achieves comparable or even better performance than state-of-the-arts on each single task, but also facilitates some combined tasks with one-time forward, demonstrating its potential in serving as foundation generative model in face domain. We further develop an interface such that users can enjoy our one-step service to create, edit, and animate their own characters. Code, dataset, model, and interface will made publicly available.
Introduction
Due to the rapid development of generative models, such as diffusion models (DMs) [10, 15, 47], VAEs [21], GANs [6] and flow models [5] in the computer vision area, automatic content creation has recently received an increasing amount of attention for its real-world applications. Benefiting from these generative models, facial content creation as critical part has recently achieved impressive progress and simultaneously shows great application potential, egg. virtual digital human, artistic creation, and intelligent customer service.
Existing face generative models [20, 23, 39, 61] are usually developed as highly customized systems, meaning that model can only handle one task. However, this design poses two significant challenges: 1) hard to accomplish complex tasks, such as integrating face creating, editing and then animating the generated face in single step: redundant consumption of memory and computation. For example, one needs to train and save a number of models to build a multi- nulti-functional system, and perform complicated inference processes. The challenges could inevitably limit its further applications and development.
To tackle these problems, propose compositional FaceComposer in this work, unified model that is capable of simultancously tackling versatile facial tasks, including face generation, face editing,
Equal contribution.
37th Conference on Neural Information Processing Systems (NcurIPS 2023).

---

face animation, and their combinations. Specifically, we decompose a given face into multi-level representative factors, e.g., Identity Features, Projected Normalized Coordinate Code (PNCC) [64] and Text2Face (T2F) Embeddings, and then train powerful latent diffusion model conditioned on them to compose the input face. This design provides the model with exceptional controllability over facial content creation, enabling a seamless process from generating face to utilizing it. In particular, we have additionally incorporated temporal self-attention to enable joint learning from both images and videos simultaneously. To optimize FaceComposer, we finally gather high-quality large-scale multi-modal face dataset, including 1.1 million face images from pre-existing datasets and 500 hours of meticulously cleaned talking face videos.
Extensive quantitative and qualitative results demonstrate that FaceComposer achieves exceptional performance in various facial content creation tasks. FaceComposer surpasses previous state-of- the-art methods in face generation, face editing, and audio-driven face animation in terms of the most widely-used evaluation metrics. Furthermore, we have devised intricate yet imaginative tasks to showcase the advantages of our diverse condition composition.
2 Related work
Face generation. The goal of face generation [34, 53, 56, 57] is to generate photo-realistic face images. Among them, StyleGAN series [17-19] boost the generation quality by introducing controllable and interpretable latent space. After that, some variants [51, 63, 42, 41, 52] are proposed for further quality improvement. One of them, TediGAN [51], maps text into the StyleGAN latent space to make the text-guided face generation. Another example is LAFITE [63], which presents a language-free training framework with a conditional StyleGAN generator. Recently, diffusion models [10, 32, 33, 38, 37] become more and more popular in image synthesis due to their strong generative abilities, We find fine-tuning a pre-trained diffusion model will bring comparable or even superior performance the GAN paradigm.
Face editing. Face editing [20, 23, 25, 27, 54] aims to manipulate the face images guided by text descriptions masks. Similar to face generation, StyleGAN- methods [51, 29, 49, 65] show remarkable performance in face editing. Besides TediGAN, StyleClip [29] combines representation ability of CLIP [31] and generative power of StyleGAN optimize the editing direction. In contrast to the above methods, ColIDiff [13] trains the diffusion model with multi-modal conditions (text + mask), in order make them complementary. It is noted that TediGAN and CollDiff both support face generation and editing, which differ from ours in the following aspects: 1) TediGAN is essentially designed as bespoke system, i.e., it can only complete one task (generation or editing) in one model inference. 2) CollDiff uses multiple diffusion models (one model for one condition), leading to low efficiency in training and inference stage. 3) FaceComposer only use one diffusion model in unified framework, and can finish different facial creation combinations in one model inference.
Face animation. Face animation [39, 36, 50, 55, 35, 30, 61, 62, 26, 44] intends to make the target face move according to a driving video or audio. Video-driven methods [39, 36, 50, 55, 35] focus on modeling the motion relationships between source and target face. For example, FOMM [39] decouples appearance and motion and introduces keypoints representation to support complex motions. PIRenderer [36] employs 3DMM to better control the face motions. Compared with keypoints and 3DMMs, the PNCC we used contains more dense and intuitive facial structure information, making it easier to be learned by the model. Audio-driven methods [30, 61, 62, 26, 44] pay more attention to the lip synchronization (lip-sync) between the audio and the mouth of target face. Makel(Talk [62] disentangles the content and speaker information control lip motions. By learning from powerful lip-sync discriminator, Wav2lip [30] improves lip-syne accuracy. PC-AVS [61] generates pose- controllable talking faces by modularizing audio-visual representations. However, all of them only consider intra- frame information for reconstruction, ignoring inter-frame relationship, which is fixed by adding a temporal attention module in our FaceComposer.
3 FaceComposer
Recent days have witnessed the powerful generative ability of Latent Diffusion Models (LDMs) [37], which is consequently equipped as the backbone of our FaceComposer. We model various facial content creations as mullppe-condition-diie denoising process. Fig. I shows the overview of

---

Training Process
Loss
Frames
Image
Aecface
PNCC
Sketch
Identity Feature
T2F Embedding
Diverse Conditions
Face Generation
N
Guidance
D
Face Animation
DOW mrpp
Inference Process
Figure 1: The framework of FaceComposer, which takes f frames and five face-related conditions as input, uses LDMs to predict the noise added in the latent space. We can combine diverse conditions finish face pnneraationncdiinn/aiimattii or their combinations. For example, the green A conditions for face generation, yellow D for face editing, and red $ for face animation.
FaceComposer. In the unified framework, the inputs of different tasks are expressed diverse conditions, including Mask, PNCC, Sketch, Identity Feature and T2F Embedding. We achieve versatile facial content creations through condition combinations (Sec. 3.2). Different from the standard LDMs that is only designed for image generation, FaceComposer supports both static and dynamic content creations, i.e. our dataset contains both images and videos. So introduce temporary attention module into LDMs for the two modalities joint training (Sec. 3.3 and Sec. 3.4).
3.1 Preliminaries
We denote the input f frames as R where H and W are height and width of input frames (we set W 256 in experiments).
Latent diffusion model. To save the computational resources of DMs, we follow LDMs to encode frames into latent space: Zn E(xo)) / X with pre-trained image encoder E, where Ctatent means the dimension of latent space, / and w are set to 32 in practice. And in the end of denoised process, the final žo will be mapped into pixel space with image decoder D Tp = D(z)) E IR xxxXXX In the latent space, the diffusion model zg can be parameterized to predict the added noise:
Esimple Ez,,,t.. [||e 2o 20(220 o.c, C)
where C denotes the condition, & E [I.. T], NN 1) the random Gaussian noise, a, and de are two scalars related to f. We freeze E' and D, and start from pre-trained LDMs.
Compositional generation pipeline. As a pioneering work of compositional generation, Composer [12] decomposes an image into eight conditions to improve the controllability of image synthesis, inspiring us to treat the inputs of different facial content creations as multiple conditions, i.e., C in Eq. is condition set. And we adopt the same guidance directions as [12]: eo L0(2,, C) wzo C2) + ( 61), where is the guidance weight, and C2 two subsets of C, respectively.
3.2 Diverse face-specific conditions
Condition decomposition We convert different inputs into the following five face-specific conditions.
Mask: Mask used to force FaceComposer to generate or edit face certain region. Based on the nine parsing areas of face [22]. we randomly mask one or all of them. When >>> 1, different frames mask the same region. Before taken condition, masks are also mapped into latent space through E.

---

Table 1:
Versatile creations based on condition compositions. M. S. PNCCs, ID and T2F are short for Mask, Sketch, PNCC sequence, Identity Feature, and T2F Embedding, respectively.
Single Creation
Versatile Creations
Task
Conditions
Task
Conditions
1
T2F
PNCCs+12F
face generation
? S
face generatiion-amimation
PNCCs+ID
M+T2F
ID+M
face editing
M+S
face generation+ecditng
ID+T2F
i
M+T2F+PNCCs
i ID+T2F+PNCCs
face animation
face genertiion-ediing-anniaatoi
PNCC: PNCC [64] represents the geometric information of the face, a pre-defined PNCC sequence effectively guide facial animation generation. We use FLAME Fitting [24] to extract the PNCC of all frames. Different from the mask, the distribution of PNCC differs significantly from the original frame, so we add a trainable module Ep to encode the PNCC.
Sketch: Sketch describes the contours of different parts of the face (c.g., face shape, eye size, mouth location), it contains local details with low semantics. We adopt the same method as [12] extract the sketches. Similar to PNCC, we input sketch information into a trainable E, module to obtain the condition.
Identity Feature: Identity Feature indicates the identity attribute, excluding trivial information (such as hair color, texture, expression). It can direct the model to generate a face with the specified ID. We use ArcFace [3] to get the Identity Feature.
Text2Face (T2F) Embedding has two functions: 1) complementing Identity Feature with detailed information; 2) enabling control. Specifically, during the training stage, T2F Embedding is extracted from reference image with Face Clip [60] to feed facial details into FaceComposer. In the denoising process, besides the reference image, we can also obtain T2F Embedding from text prompt with an extra prior model (similar to DALL-E [33]).
T2F Embedddngg:
Conditioning mechanisms Considering Mask, PNCC and Sketch represent the spatial local information of frames, they are all extracted from Zn and concatenated with z, in the channel dimension. In contrast, Identity Feature and T2F Embedding define the global semantic information, hence we get them from the reference image, and add the projected Identity Feature into time embedding, serve T2F Embedding as key and value for cross attention module of ze (corresponding to Spatial Att in Fig. 1). Note that, except for PNCC, which needs to be fitted in advance, all left conditions are extracted on-the-fly. We adopt a similar condition training strategy to Composer: setting 0.5 dropout probability for each condition, 0.1 to drop all conditions, and 0. to reserve them all.
Condition composition As we mentioned above, we can support versatile tasks: face generation, face editing, face animation and their combinations by combining our diverse conditions.
We list some representative creations Tab. Taking face animation as an example, we use the PNCC sequence (predicted by audio). Mask (masking the mouth region of the target face), and T2F Embedding (providing texture information from reference image) to generate Talking Head. It is worth mentioning that FaceComposer one-time forward by conditioning on Identity Feature (guiding face generation). T2F Embedding (extracted from editing prompt) and PNCC sequence (obtained from reference video). More details are shown Sec. 4.3.
finish face generation + editing + animation with
3.3 Temporal self-attention module
In order to create static and dynamic contents simultaneously, we prepare a multi-modal database, consisting of both images and vidcos. And we argue that joint image-video training is important

---

TediGAN
CollDiff
LAFITE
Ours
Hy tax lngeo lae
Lac Pereasss Las racuth Moct bair. high Charlhhonns, carry Sle in lipricd,
lox lms hand.
she ko mshel hanvy nakinp pi
The soemer he She lparce Sheie ancting.
So. Ieceee-- ralonay entling and atructive
Helte hage teeak youag
He ball and lig
The vocsa lea dightty bgg liss, high chealhoral bage wauiing
heary
avd flce. stractive.
Eie is young
Figure 2: The qualitative results of face generation.
for FaceComposer, considering face image will align facial content and text description, while video will link spatial and temporal information of face sequences.
Inspired by [40, 11 we introduce temporary self-attention (TSA) layer into LDMs (corresponding to Temporal Att in Fig. 1), and select half batch samples from images (i.e. f = 1 for 0.5 and the other half from videos (i.e. f 5 for left 0.5B) to make the joint strategy fully benefit each other within each batch (see ablation study in Sec. 4.4), where is the total batch size. Assuming the input of TSA is in the shape of B ff x Cinter h w', it will be arranged to { h x w') x f x Cinter before entered into TSA, where Cinter, h², w are the intermediate channel dimension and feature map size. When f TSA degrades to identity transformation.
3.4 Multi-modal face database
To empower FaceComposer with image and video generation capabilities, we construct high-quality large-scale multi-modal face database comprising 1.1 million face images with text annotations and approximately 500 hours-long talking face videos.
Image data. To construct the image part of our database, we carefully clean up LAION-Face [60] and merge the cleaned dataset with CelebA-HQ [16] and FFHQ [17]. We clean up LAION-Face using two approaches. Firstly, We use CLIP [31] to filter out the image-text pairs whose text descriptions do not match the images. Specifically, for each image-text pair, we compute the cosine similarity between CLIP features extracted from the image and the text and filter out the pair if the similarity is lower than predefined threshold. Secondly, we use an off-the-shelf face detector [4] to detect faces in images and filter out images with no faces detected. Finally, we obtain the cleaned LAION-Face dataset. It contains million face images with corresponding text captions.
Video data. To construct the video part of our database, we collect talking face videos from Youtube, BBC television, and some other web data. We manually clean collected videos ensure high video quality and audio-visual coherence. Our collected talking face dataset includes more than 500 hours 720P~1080P videos with the audio track. We will release the dataset when the paper made public. For more details, please refer to Supplementary Material.
Experiments
4.1 Experimental setup
Implementation details. During the training, our model starts from a pre-trained LDMs', and is further trained on our multi-modal face database through a joint training mechanism. To enable text
5

---

TediGAN
StyleCLIP
Ours
TediGAN
StyleCLIP
Ours
Ours
Source (a)
Black Hair
With Sunglasses
Comic Style
Figure 3: The qualitative results of face editing. (a) The source images. (b) The images whose attributes are edited through text descriptions by different methods. (c) The stylized results generated by FaceComposer. FaceComposer is able to edit the style of images using text (the left column) or a style reference image (the right column).
control, we trained a 1B parameter prior model for projecting captions to T2F Embeddings. For the LDMs, we pretrain it with 1M steps on the full multi modal dataset using only T2F Embeddings as the condition, and then finetune the model for 200K steps with conditions enabled. The prior model is trained for LM steps on the image dataset.
Evaluation tasks. We evaluate FaceComposer on face generation, face animation and face editing tasks, which respectively using the Multi-Modal CelebA-HQ [51]. HDTF [59] + MEAD-Ncutral subset of MEAD [45] that only contains the neutral facial expression videos). and the randomly selected images from both CelebA and Non-CelebA datasets with randomly chosen descriptions.
Evaluation metrics. For face generation task, adopt Fréchet inception distance (FID) [9] to measure the image quality. Since FID cannot reflect whether the generated images are well conditioned on the given captions, choose to adopt R-precision [53] as another evaluation metric. For face editing task, we compute the identity similarities (IDS) between the input faces and the edited faces measure the identity consistency. For face animation task, adopt the Landmark Distance around mouths (M-LMD) [1], the Landmark Distance on the whole face (F-LMD), the structural similarity (SSIM) [48], the Cumulative Probability of Blur Detection (CPBD) [28], and the confidence scores of SyncNet [2] (Synccont) as evaluation metries. To facilitate a more equitable comparison between methods generating only the mouth region (visual dubbing methods) and those generating the entire face (one shot methods), we computed the SSIM and CPBD for the mouth region, denoted as SSIM-M and CPBD-M, respectively. For more details about the experimental settings, please refer to Supplementary Material.
4.2 Comparisons single task
In this subsection, demonstrate the performance of FaceComposer on different tasks, including static content creation, e.g., face generation and editing, and dynamic content creation task, egg., face animation. Note that for the face animation task, Audio2PNCC model to generate PNCCs with lip motions controlled by input audio.
audio to drive face motions. We train an
Face generation. We compare our method with TediGAN [51], LAFITE [63] and ColIDiff [13]. We generate 30, 000 images for each method using randomly selected unseen text descriptions. Tab. shows that our method achieves the best results in terms of both FID and R- R-precision. Such results

---

Mouth GT
AVCT
MakeitTalk
EAMM
Wav2Lip
StyleTalk
SadTalker
PC -C-
StyleSync
Ours
Figure 4: Qualitative results of face animation. It can be seen that FaceComposer not only achieves accurate lip-sync but also produces high-fidelity results in the mouth area.
Table 2: Results of face generation.
Table 3: Results of face editing.
Method
FID
R-precision(991)
Accuracyt
Realismi
Method
IDSt
Accuracyt
Realismt
TediGAN [S1]
107.14
44.96
2.80
3.55
TediGAN [51]
0.62
2.92
2.09
CollDif [13]
98.76
67.41
3.43
2.96
StyleClip [29]
0.75
4.27
2.87
LAFTTE [63]
12.54
81.94
4.07
3.88
Ours 11.34
86.63
4.86
4.67
Ours
0.94
4.58
4.47
demonstrate the effectiveness of massive training data. The qualitative comparison is shown in Fig. 2. We can observe that other methods can generate text-relevant results. However, their results lack some attributes contained in the input captions. On the other hand, our results achieve higher fidelity while maintaining text-visual coherence.
We also evaluate the Accuracy and Realism through user study, in which the users are asked to judge which image is more realistic, and more coherent with the given captions. We randomly sampling 30 images with the same text conditions and collect more than 20 surveys from different people with various backgrounds. Compared with the state-of-the-arts FaceComposer achieves better accuracy and realism values as shown in Tab. 2, which proves that methods can generate images with the highest quality and text-relevance.
Face editing. To evaluate the face editing performance of FaceComposer, we compare our method with TediGAN [51] and StyleClip [29]. Results are shown in Tab. 3, and we can see that our method achieves better results in all metrics (accuracy and realism come from user study, following the same scheme in Tab. 2), which proves that our methods can generate images with the highest quality and text-relevance. We show some visual results in Fig. 3. As can see, our methods can perform face editing more precisely, and the achieved results are the most photorealistic and most coherent with the given texts.
Face animation. We compare our method with state-of-the-ar methods including MakelrTalk [62], Wav2Lip [30]. PC-AVS [61]. AVCT [46]. EAMM [14], StyleTalk [26]. SadTalker [58], and StyleSyne [7]. The samples of the compared methods are generated either with their released codes or with the help of their authors. As shown in Tab. 4, FaceComposer achieves the best performance in most metrics. The strong performance SSIM, SSIM-M. CPBD, and CPBD-M metrics indicates the high quality of videos generated by FaceComposer. Given that Wav2Lip employs SyncNet as a discriminator during its training phase, reasonably attains the highest confidence score from SyncNet, surpassing even the ground truth. Numerous prior arts [7, 58, 26, 8, 43) have reported that Wav2L.pp, despite attaining high SyneNet scores, does not fare well in qualitative evaluations (e.g., user studies) of lip-sync, attributed to the production of blurry results and, occasionally, exaggerated
7

---

Table 4: Results of face animation HDTF and MEAD-Neutral
HDTF MEAD-Neutral
Methods
SSIM
CPBD1
F-LMD|
M-LMD4
Sync
SSIM-M1
CPBD-M
MakeltTalk [62]
0.63/0.74
0.19/0.10
4,10/3.88
4.22/5.41
3.07/2.02
0.62/0.69
0,15/0.06
Wav2Lip [30]
0.75/0.79
0.18/0.12
2.01/2.38
2.54/2.95
5,27/3.99
0.68/0.78
0,08/0.03
PC-AVS [61]
0.51/0.51
0.23/0.07
3.64/4.76
3.52/3991
4.16/3.09
0.60/0.67
0.10/0.05
AVCT [46]
0.73/0.77
0.17/0.10
2.85/2.68
3,53/4.46
3.81/2.56
0.70/0.73
0.16/0.08
FAMM [14]
0.59/0.41
0.08/0.08
4.16/7.39
4.19/5.03
2.30/1.40
0.60/0.71
0.13/0.05
StyleTalk [26]
0.78/0.79
0.23/0.12
2.10/2.35
2.40/2.80
4.17/3.05
0.76/0.80
0.16/0.09
SadTalker [58]
0.61/0.73
0.21/0.12
3.98/3.67
3.46/4.09
4.05/2.62
0.61/0.69
0.15/0.10
StyleSync [7]
0.77/0.80
0.21/0.12
1.93/2.22
2.36/2.76
4.21/3.10
0.76/0.80
0.17/0.10
Ground Truth
1/1
0.23/0.20
0/0
0/0
4.52/3.57
1/1
0.21/0.12
FaceComposer
0.78/0.84
0.27/0.14
1.84/2.16
2.25/2.70
4.27/3.12
0.78/0.83
0.18/0.10
Table 5: User study results of different methods on HDTF and MEAD-Neutral for the face animation.
LS, VQ, OR stand for user study metries LipSync, VideoQuality and OverallRealness, respectively.
HDTF MEAD-Neutral
Methods
Lst
VQ1
OR
MakeltTalk
1.71/2.20
1,87/2.88
1.44/1.74
Wav2Lip
2.93/3.33
1.02/1.10
1.10/1.12
PC-AVS
2.88/3.20
1.97/2.46
1.73/1.98
AVCT
2.04/2.76
2.60/2.66
2.46/2.62
EAMM
1.90/2.58
1.30/1.78
1,62/1.94
Style'Talk
3.10/3.60
3,08/3.00
2.44/2.82
SadTalker
3,22/3.68
2.82/2.92
1.97/2.42
Ground Truth
4.34/4.56
4.06/4.26
4.22/4.40
Compoerr
3.53/3.96
3.38/3.50
2.93/3.73
lip motions. Our method's confidence score aligns most closely with the ground truth, and our method's M-LMD scores are the best. This indicates that FaceComposer achieves precise lip-sync. We provide more analysis regarding quantitative evaluation in the Supplementary Material.
Fig. 4 shows the qualitative results, can be seen that MakeltTalk, AVCT, and EAMM fail to produce accurate mouth movements, while FaceComposer achieves accurate lip-sync. Although PC-AVS and Wav2Lip are competitive with FaceComposer terms of lip-syne, they can only generate blurry results. FaceComposer produces high-fidelity results in the mouth area. Style'Talk often produces overly smooth lip movements. The lip-sync SadTalker is slightly inferior to our method. StyleSync occasionally yields jittery and exaggerated motions.
We conduct user study involving 20 participants. Each participant is requested to assign scores to five videos sampled from the test dataset for each method, evaluating them on three aspects: (1) Lip sync accuracy; (2) Video quality; (3) Overall Realness. The scoring employs the Mean Opinion Scores (MOS) rating protocol (scoring from 1 to 5, larger is better). Tab. shows the results. Our method achieves the best all aspects, which demonstrates the effectiveness our method.
4.3 Advantage of unified framework
In this subsection, demonstrate the compositional generation capabilities, which depicted in Tab. 1, of FaceComposer with one-time forward. Here we provide two examples of versatile creations. In practical application, one customize different conditions to meet specific requirements. More results will be provided in Supplementary Material.
Face generation+-aiimation FaceComposer has the ability of generating desired face with random T2F Embedding and animating with one-time forward. We show six examples in Fig. to demonstrate such an ability. As we can see, the generated results in each row share almost the same mouth shape as that of the driving PNCCs. Meanwhile, each generated result consistently adheres to the given T2F Embeddings.

---

Botman
Wooden Mask
Cartoon Man
Anime Girl
Attructive
3D Robot
Man
T2F
T2F
PNCCS
PNCCs
Figure 5: Face generation and animation results. FaceComposer supports generating desired faces with T2F Embedding and animating faces with PNCCs simultaneously in one-time forward.
Besides, FaceComposer incorporate additional identity conditions to animate given face in a prescribed style. Some results are shown in Fig. 7. is clear that each animated frame has the same mouth shape and expression that of the corresponding PNCCs, while maintaining its original identity and face shape.
Face enmeration-ddiiin--aniie
4.4 Ablation study
To isolate the contributions our designs, we conduct the ablation studies from three aspects. Note that the Audio2PNCC model is identical in all variants and we only compare the ability of these variants to convert PNCCs to videos.
Image/Video dataset impact. We compare the following three variants: (1) only train FaceComposer using videos (w/o Image), (2) only train FaceComposer using im- ages (w/o Video), (3) our full model (Full), for face animation on HDTF dataset Tab. 6. Full achieves the best performance in most metries, which demonstrates the effective- ness of joint training on image and video data. We observe that w/o Video attains the highest score CPBD metric. This may be due to two reasons. First, high-fidelity face images from CelebHQ and FFHQ are incorporated into our image data. Second, although video dataset in high definition, the motion in the vidcos may blur certain frames, which may lower the model's s score in the CPBD metric.
Table 6: Ablation results of face animation on HDTF
Method
SSIM1
CPBD
F-LMD M-LMDD
w/o Image
0.78
0.27
1.88
2.46
wo Video
0.77
0.29
1.94
2.40
Full
0.78
0.27
1.84
2.25
Fig. 6 shows the qualitative ablation results. w/o Image fails to generate realistic texture in the mouth area, i.e. the teeth and the tongue. This may imply that high-resolution image data are crucial to learning to generate high-quality textures. w/o Video fails to generate mouth shapes corresponding to input PNCCs. This may stem from the fact that the image data only contains images with a limited number of mouth shapes. On the other hand, Full generates high-fidelij results in the mouth area, produces accurate mouth shape, and attains satisfactory frame consistency. This indicates the necessity constructing and learning from a multi-modal face database.
Ground Truth
w/o Video
w/o Image
Full
Figure 6: Qualitative ablation results.
Condition numbers. We investigate the effect different numbers of conditions on FaceComposer in Tab. 7. Considering face ceneraaion//diiinn/aannaioo are the fundamental tasks, we take T2F; Mask and PNCC three basic conditions (i.e., FaceComposer w/o (ID+S)), add Sketch condition in FaceComposer w/o ID. and keeps all five conditions in FaceComposer. It can be seen that

---

3D Green Model
Painting
Drawing
Anime Student
Cyberpusk
Curtoon
ID T2F
ID+T22
PNCCs
PNCCs
Figure 7: Face generation-ediing-aannimiii results. FaceComposer is able to generate face images with specified identities, edit image styles, and animate face images simultaneously in one-time forward.
Table 7: Ablation results with different conditions or data scales for face generation, editing, and animation.
Face Generation R-precision($$)) (
Face Editing IDS
Face Animation
Variants
FIDL
SSIM
CPBD1
F-LMD!
M-LMDJ
FomeComposer
11.34
86.63
0.94
0.78
0.27
1.84
2.25
FaceComposer w/o
11.40
86.28
0.94
0.78
0.26
1.89
2.28
FaceComposer w/o (ID+S)
11.27
86.91
0.93
0.78
0.26
1.88
2.28
FaceComposer-
11.60
86.85
0.93
0.78
0.26
1.91
2.37
FaceComposers with different number of conditions keep stable performance, no matter face generation, editing animation task. This is reasonable, bacause the training dataset is fixed, when the number of conditions increases, no additional information introduced for specific task.
Dataset scale. To demonstrate the impact of dataset scale on generation quality, we show an ablation study in Tab. 7. Considering the state-of-the-art methods generally have dozens of hours of training data, we reduce the training data of FaceComposer to the similar scale for a fair comparison. Specifically, we randomly sampled 10 hours of video and 4.5W images from our original dataset to train FaceComposer (denoted as FaceComposer-). From Tab. 7, it can be observed that FaceComposer- inferior to FaceComposer due the decrease of data information, but it is still better than other state-of-the-arss in Tabs. 2 to 4.
Conclusion
This paper introduces a unified framework (named as FaceComposer) for versatile facial content creation, i.e., 1) generating faces from scratch, 2) adjusting portion of the generated ones, and 3) making the generated target move. demonstrates a complete creative process from static (image) dynamic (video) content. FaceComposer employs LDM with multiple conditions handle all above facial creation tasks. We evaluate its performance in various tasks and demonstrate our versatile creative ability different combined tasks. Demo interface, Social impact and Ethic statement will be moved to Supplementary Material due to limited space.
References
[1] L. Chen, R. K. Maddox, Z. Duan, and C. Xu. Hierarchical cross-modal talking face generation with dynamic pixel-wise loss. In IEEE Conf. Comput. Vis. Pattern Recog., 2019.
10

---

[2] J. S. Chung and A. Zisserman. Out of time: Automated lip syne in the wild. In Asian Conf. Compud. Vis., 2017.
[3] J. Deng. J. Guo, N. Xue, and S. Zafeiriou. Areface: Additive angular margin loss for deep face recognition. In IEEE Conf. Comput. Vis. Pattern Recog. 2019.
[4] Deng, Guo. Ververas, Kotsia, and S. Zafeiriou. Retinaface: Single-shot multi-level face localisation in the wild, In IEEE Conf. Comput. Vis. Pattern Recog. 2020.
[5] L. Dinh, D. Krueger, and Y. Bengio. Nice: Non-linear independent components estimation. arXiv preprint arXiv: IMXiv:1410.8516 2014.
[6] I. Goodfellow, J. Pouget-Abadig M. Mirza, B. Xu, D. Warde-Farley, S. Ozair, A. Courville, and Y. Bengio. Generative adversarial networks. Communications of the ACM, 2020.
[7] J. Guan, Z. Zhang. H. Zhou, T. Hu, K. Wang. D. He, H. Feng. J. Liu, E. Ding, Z. Liu, al. Stylesyne: High-fidelity generalized and personalized lip syne in style-based generator. In IEEE Conf. Comput. Vis. Pattern Recog., 2023.
[8] S. Gururami, A. Mallya, T--C. Wang, R. Valle, and M.-Y. Liu. Spacex: Speech-driven portrait animation with controllable expression. arXiv preprint arXiv:2211.09809 2022.
[9] M. Heusel, H. Ramsauer, T. Unterthiner, B. Nessler, and S. Hochreiter. Gans trained by a two time-scale update rule converge to local nash equilibrium. In Adv. Neural Inform. Process. Syst., 2017.
[10] J. Ho, A. Jain, and P. Abbeel. Denoising diffusion probabilistic models. In Adv. Neural Inform. Process. Syst., 2020.
[11] J. Ho, T. Salimans, A. Gritsenko, W. Chan, M. Norouzi, and D. Fleet. Video diffusion models. arXiv:2204,000 033558 2022.
[12] L. Huang, D. Chen, Y. Liu, Y. Shen, Zhao, and J. Zhou. Composer: Creative and controllable image synthesis with composable conditions. In Int. Conf. Mach. Learn., 2023.
[13] Z. Huang, K. C. Chan, Y. Jiang, and Z. Liu. Collaborative diffusion for multi-modal face generation and editing. In IEEE Conf. Comput. Vis. Pattern Recog., 2023,
[14] X. Ji, H. Zhou, K. Wang, Q Wu, W. Wu, F. Xu, and X. Cao. Famm: One-shot emotional talking face via mudio-based emotion-awar motion model. In SIGGRAPH, 2022.
[15] J. Karras, A. Holynski, T.-C. Wang, and I. with stable diffusion. In Int. Conf. Comput. Vis., 2023.
Dreampose: Fashion vidco synthesis
[16] T. Karras, T. Aila, S. Laine, and Lehtinen. Progressive growing of gans for improved quality, stability, and variation. In Int. Conf. Learn. Represent., 2018.
[17] T. Karras, S. Laine, and T. Aila. A style-based generator architecture for generative adversarial networks. In IEEE Conf. Comput. Vis. Pattern Recog., 2019.
[18] T. Karras, S. Laine, M. Aittala, J. Hellsten, I Lehtinen, and T. Aila. Analyzing and improving the image quality of stylegan. In IEEE Conf. Comput. Vis. Pattern Recog., 2020.
[19] T. Karras, M. Aittala, S. Laine, E. Härkönen, J. Hellsten, J. Lehtinen, and T. Aila. Alias-free generative adversarial networks. In Adv. Neural Inform. Process. Syst., 2021.
[20] G. Kim, T. Kwon, and C. Ye. Diffusionclip: Text-guided diffusson models for robust image manipulation. In IEEE Conf. Comput. Vis. Pattern Recog., 2022.
[21] D. Kingma and M. Welling. Auto-encoding variational bayes. arXiv preprint arXiv: 2013.
[22] C.-H. Lee, Liu. L. Wu, Luo. Maskgan: Towards diverse and interactive facial image manipulation. In IEEE Conf. Comput. Vis. Pattern Recog., 2020.
[23] B. Li, X. Qi. T. Lukasiewicz, and P. H. Torr. Manigan: Text-guided image manipulation. In /EEE Conf. Comput. Vis. Pattern Recog., 2020.
[24] T. Li. T. Bolkart, J. Black, H. Li, and J. Romero. Learning model of facial shape and expression from 4d scans. ACM Trans. Graph., 2017.

---

[25] Y. Liu, M. De Nadai. D. Cai, H. Li. X. Alameda-Pineda N. Sebe, and B. Lepri. Describe what to change:
A text-guided unsupervised image-to-image translation approach. In ACM Int. Conf. Multimedia, 2020.
[26] Y. Ma, S. Wang. Z. Hu, C. Fan, T. Lv, Y. Ding. Z. Deng, and X. Yu. Styletalk: One-shot talking head generation with controllable speaking styles. In Assoc. Adv, Artif. Intell., 2023.
[27] S. Nam, Y. Kim, and S. Kim. Text-adaptive generative adversarial networks: Manipulating images with natural language. In Adv. Neural Inform. Process. Syst., 2018.
[28] N. D. Narvekar and L. J. Karam. A no no-reference perceptual image sharpness metric based on a cumulative prohability of blur detection. In Int. Qual. of Multimedia Exp. Worksh,, 2009.
[29] O. Patashnik, Z. Wu, E. Shechtman, D. Cohen-Or, and D. Lischinski. Styleclip: Text-driven manipulation of stylegan imagery. In Int. Conf. Comput. Vis., 2021.
[30] K. Prajwal, R. Mukhopadhyay, V. P. Namboodiri, and C. Jawahar. A lip syne expert is all you need for speech to lip generation in the wild. In ACM Int. Conf. Multimedia, 2020.
[31] A. Radford. J. W. Kim, C. Hallacy, A. Ramesh, G. Goh, S. Agarwal, G. Sastry, A. Askell, P. Mishkin, Clark, et al. Learning transferable visual models from natural language supervision. In Int. Conf. Mach. Learn., 2021.
[32] A. Ramesh, M. Pavlov, G. Goh, S. Gray, C. Voss, A. Radford, M. Chen, and I. Sutskever. Zero-shot text-to-image generation. In Int. Conf. Mach. Learn., 2021.
[33] A. Ramesh, P. Dhariwal, A. Nichol, C. Chu, and M. Chen. Hicrarchical text-conditional image generation with clip latents. arXiv preprint arXiv:2204.06122 2022.
[34] S. Reed, Z. Akata, X. Yan, L. Logeswaran, B. Schiele, and H. Lee. Generative adversarial text to image synthesis. In Int. Conf. Mach. Learn, 2016.
[35] Y. Ren, G. Li. S. Liu, and T. H. Li. Deep spatial transformation for pose-guided person image generation and animation. IEEE Trans. Image Process., 2020.
[36] Y. Ren, G. Li, Chen, H. Li, and Liu, Pirenderer: Controllable portrait image generation via semantic neural rendering. In Int. Conf. Comput. Vis., 2021.
[37] R. Rombach, A. Blattmann, D. Lorenz, P. Esser, and B. Ommer. High-resolution image synthesis with latent diffusion models. In IEEE Conf. Comput. Vis. Pattern Recog., 2022.
[38] C. Saharia, W. Chan, S. Saxena, I. Li, J. Whang, E. L. Denton, K. Ghasemipour, R. Gontijo Lopes, B. Karagol Ayan, T. Salimans, et al. Photorealistic text-to-image diffusion models with deep language understanding. In Adv. Neural Inform. Process. Syst., 2022.
[39] A. Siarohin, S. Lathuilière, S. Tulyakov, E. Ricci, and N. Sebe. First order motion model for image animation. Adv Neural Inform. Process. Syst., 2019.
[40] U. Singer, A. Polyak, T. Hayes, X. Yin, An. S. Zhang, Q. Hu, H. Yang, O. Ashual, O. Gafni, et al. Make-a-video: Text-to-video generation without text-video data. In Int. Conf. Learn. Represent. 2022.
[41] D. Stap, M. Bleeker, S. Ibrahimi, and M. Ter Hoeve. Conditional image generation and manipulation for user-specified content. arXiv preprint arXiv:2005. 200,00990,, 2020.
[42] J. Sun, Q. Deng, Q. Li, M. Sun, M. Ren, and Z. Sun. Anyface: Free-style text-to-face synthesis and manipulation. In IEFE Conf. Comput. Vis. Pattern Recog., 2022.
[43] D. Wang, Y. Deng, Z. Yin, H.-Y. Shum, and B. Wang. Progressive disentangled representation learning for fine-grained controllable talking head synthesis. In IEEE Conf. Comput. Vis. Pattern Recog., 2023.
[44] J. Wang, K. Zhao, S. Zhang, Y. Zhang, Y. Shen, D. Zhao, and J. Zhou. Lipformer: High-fidelity and generalizable talking face generation with pre-learned facial codebook. In IEFE Conf. Comput. Vis. Pattern Recog,, 2023.
[45] K. Wang, Q. Wu, L. Song, Z. Yang. W. Wu, C. Qian, R. He. Y. Qiao, and C. C. Loy. Mead: A large-scale audio-visual dataset for emotional face generation. In Eur. Conf. Comput. Vis., 2020.
[46] S. Wang, L. Li, Y. Ding, and X. Yu. One-shot talking face generation from single- speaker audio-visus correlation learning. In Assoc. Adv. Artif. Intell., 2022.
12

---

[47] T. Wang, T. Zhang. B. Zhang, H. Ouyang, D. Chen, Q. Chen, and F. Wen. Pretraining is all you need for image-to-image translation. arXiv preprint arXiv:2205. 12952, 2022.
[48] Z. Wang, A. C. Bovik, H. R. Sheikh, and E. P. Simoncelli. Image quality assessment: From error visibility to structural similarity. IEEE Trans. Image Process., 2004.
[49] T. Wei, D. Chen, W. Zhou, J. Liao, Z. Tan, L. Yuan, W. Zhang, and N. Yu. Hairclip: Design your hair by text and reference image. In IEEE Conf. Comput. Vis. Pattern Recog., 2022.
[50] O. Wiles, A. Koepke, and A. Zisserman. X2face: A network for controlling face generation using images, audio, and pose codes. In Eur. Conf. Compul. Vix.,0188. 2018.
[51] W. Xia, Y. Yang, J.-H. Xue, and B. Wu. Tedigan: Text-guided diverse face image generation and manipulation. In IEEE Conf. Comput. Vis, Pattern Recog. 2021.
[52] W. Xia, Y. Yang. J.-H. Xue, and B. Wu. Towards open- world text-guided face image generation and manipulation. arXiv preprint arXiv:2104 0890090,,...111 2021.
[53] T. Xu, P. Zhang, Q. Huang, H. Zhang, Z. Gan, X. Huang, and X. He. Attngan: Fine-grained text to image generation with attentional generative adversarial networks. In IEEE Conf. Comput. Vis. Pattern Recog., 2018.
[54] Z. Xu, T. Lin, H. Tang, F. Li, D. He, N. Sehe, R. Timofte, L. Van Gool, and E. Ding. Predict, prevent, and evaluate: Disentangled text-driven image manipulation empowered by pre-trained vision-language model. In IEEE Conf. Comput. Vis. Pattern Recog,, 2022.
[55] E. Zakharov, A. Ivakhnenko, A. Shysheya, and V. Lempitsky. Fast bi-layer neural synthesis of one-shot realistic head avatars. In Eur. Conf. Comput. Vis., 2020.
[56] H. Zhang. T. Xu, H. Li, S. Zhang, X. Wang, X. Huang, and D. N. Metaxas. Stackgan: Text to photo-realistic image synthesis with stacked generative adversarial networks. In Int. Conf. Comput. Vis., 2017.
[57] H. Zhang, Xu, H. Li, S. Zhang. X. Wang, X. Huang, and D. N. Metaxas. Stackgan++: Realistic image synthesis with stacked generative adversarial networks. IEEE Trans. Pattern Anal. Mach. Intell., 2018.
[58] W. Zhang, X. Cun, X. Wang, Y. Zhang, X. Shen, Y. Guo, Y. Shan, and F. Wang. Sadtalker: Learning realistic 3d motion coefficients for stylized audio-driven single image talking face animation. In IEEE Conf. Comput. Vis. Pattern Recog., 2023.
[59] 7. Zhang, L. Li, Y. Ding, and C. Fan. Flow-guided one-shot talking face generation with a high-resolution audio-visual dataset. In IEEE Conf. Comput. Vis. Pattern Recog., 2021.
[60] Y. Zheng, H. Yang, T. Zhang, J. Bao, D. Chen, Y. Huang, L. Yuan, D. Chen, M. Zeng, and F. Wen. General facial representation learning in visual-linguisti manner. In IELE Conf. Comput. Vis. Pattern Recog.. 2022.
[61] H. Zhou, Y. Sun, W. WI, C. C. Loy, X. Wang, and 7. Liu. Pose-controllable talking face generation by implicitly modularized audio-visual representation. In IEEE Conf. Comput. Vis. Puttern Recog., 2021.
[62] Y. Zhou, X. Han, E. Shechtman, J. Echevarria, E, Kalogerakis, and D. Li. Makeittalk: Speaker-aware head animation. ACM Trans. Graph., 2020.
[63] Y. Zhou, R. Zhang, C. Chen, C. Li, C. Tensmeyer, T. Yu, J. Gu, J. Xu, and T. Sun. Towards language-free training for text-to-image generation. In IEEE Conf. Comput. Vis. Pattern Recog, 2022.
[64] X. Zhu, 7. Lei, X. Liu, H. Shi, and S. 7. Li. Face alignment across large poses: A 3d solution. In IEFE Conf. Comput. Vis. Pattern Recog,, 2016.
[65] Y. Zhu, H. Liu, Y. Song, Z. Yuan, X. Han, C. Yuan, Q Chen, and J. Wang. One model to edit them all: Free-form text-driven image manipulation with semantic modulations. In Ady. Neural Inform. Process. Syst., 2022.
13


Reviews:
[{"review_id": "U5DYZC1t6P", "reviewer": "Reviewer_tZ4V", "summary": "The paper proposes a unified framework for facial generative models that allows text/spatial/audio condition facial editing tasks. The results are reasonable and comparable to prior works. Based on the stable diffusion prior, this model can generalize well to different style domains.", "questions": "The paper would benefit from a discussion and comparison with recent related works. This would help address my concerns about missing comparisons and discussions with recent related works.", "limitations": "The social impact has been discussed.", "rating": 5, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "- The paper presents a unified framework that allows different face editing settings including face stylization, audio-driven animation, attribution editing, etc. The simultaneous style transfer and facial animation provides a one-stage solution to stylized facial animation, which avoids error accumulation and computation waste. More importantly, based on the SD prior, this model can generalize arbitrary style in the wild, such as anime, oil painting etc.\n\n- The technical contribution is moderate as most of the modules come from prior works. However, the whole system aims at addressing several face generative tasks in one model which is beneficial for many applications.\n\n- The paper organization and writing are easy to follow.", "weaknesses": "- The paper would benefit from an analysis of the pros and cons of jointly training with different conditions using text and experiment results. This is one key to support the effectiveness of the unified framework design.\n\n- Some face generation+editing results in Figure A5 look odd. For example, the mask inpainted face has a different skin color compared to the neck in (a).\n\n- While MakeItTalk, Wav2Lip and PC-AVS are not SOTA methods of face animation, recent related works such as styleTalk, styletalk, AVCT etc. should be discussed and compared if available.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for your comments!\n\n$\\textbf{1. Pros and cons of different conditions.}$\n\nIt is agreed that the analysis of the pros and cons of jointly training with different conditions is important for our unified framework design, we list them below and will add them in the final version.\n \n$\\textbf{Pros:}$ Training with multi-conditions, FaceComposer can support different tasks (e.g. face generation, face editing and face animation) and enjoy diverse controllabilities (e.g. accomplishing combined tasks among face creating, editing and animating with one-time forward) **with a unified model**. And Table R4 (upper half) shows that the performance of FaceComposer keeps stable when changing the number of conditions on a fixed dataset.  \n$\\textbf{Cons:}$ Due to the general design of FaceComposer for a variety of tasks, we need more training data to make different tasks perform well, which inevitably increases the cost of training. With a limited scale of training data, the performance of FaceComposer may degrade on some tasks, as illustrated in Table R4 (bottom half). We thus collect a high-quality large-scale multi-modal face database to alleviate such a problem.\n\n$\\textbf{2. Problems in Figure A5.}$\n\nThank you for pointing it out. In Figure A5(a), FaceComposer needs to fill the mask region in target face with the Identity Feature from source face. We admit this is a challenging setting since the Identity Feature only provides identity attribute, without facial skin or texture information. The skin color mismatch issue can be alleviated by masking both face and neck, and the visualization of this task could be affected by the difference between source and target face, such as age, gender. We show more results in Figure R2.\n\n$\\textbf{3. More SOTA of face animation.}$\n\nThanks for your reminder, we add more test sets, metrics and SOTA methods in Table R1. As we can see, FaceComposer achieves the best results, demonstrating the effectiveness of our design on face animation. StyleTalk is second to ours, because we share the same audio2PNCC design, but the representations of PNCC and generator of StyleTalk are inferior to FaceComposer. SadTalker adopts similar 3D motion coefficients regression, performing closely to StyleTalk. AVCT and EAMM are worse than the former three methods, since AVCT trains the model on one-identity dataset and the condition design of EAMM can not make full use of multi-identity dataset."}, {"review_id": "HWX7WD61GU", "reviewer": "Reviewer_BQjN", "summary": "This paper presents a facial content generation framework named FaceComposer, which is based on a Latent Diffusion Model (LDM). The primary aim of this framework is to facilitate text-conditioned face synthesis/editing and animation. The conditions employed in this model encompass a variety of aspects, including mask, PNCC (Projected Normalized Coordinate Code), sketch, identity feature, and Text2Face embedding. To ensure the generation of dynamic content, the authors introduce a temporal self-attention module within the LDM during the training phase. The experimental results affirm the superiority of this framework, showcasing enhanced synthesis quality in both static and dynamic settings. This novel approach thus provides a robust solution to the challenges of face synthesis and animation.", "questions": "1. As highlighted in the perceived weaknesses, the comparison settings for the different methods do not appear to be identical. To provide a fair and comprehensive evaluation, it would be beneficial to include some ablation studies to investigate the effect of dataset scale on generation quality. This additional layer of analysis would contribute significantly to the robustness and validity of the study's findings.", "limitations": "Certainly, the paper could be further enriched by discussing the social implications of the collected dataset. The acquisition of such an extensive dataset, with over 500 hours of talking face videos, has broad implications that could be relevant to many fields. It would be better to include some social impact discussion on it.", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "1. The concept underlying this paper is both straightforward and efficacious, offering a user-friendly yet potent solution to the problems at hand.\n\n2. The application of the Projected Normalized Coordinate Code (PNCC) as a condition in the diffusion model, particularly in the context of face animation, presents a unique and stimulating approach to this field of study.\n\n3. The collection of a new dataset comprising more than 500 hours of talking face videos is an important contribution. This sizable dataset is likely to have significant utility in the further exploration and development of this area.", "weaknesses": "1. Comparative Methods: Including the results from StyleTalk [20] in the comparison would be beneficial given its shared design principle of audio2PNCC for talking face generation. This comparison could provide a more comprehensive overview of how the proposed method performs against closely related approaches.\n\n2. Dataset Details: As Table 4 indicates, the release model of all comparison methods isn't trained on HDTF. This makes it difficult to ascertain whether the performance gap results from the FaceComposer itself or the additional training data sourced from Youtube, BBC, and so on. Conducting more ablation studies on the influence of the dataset would provide more clarity in this regard.\n\n3. Missing References: It appears that there are some relevant references missing from the current list. \n\n[a] DreamPose: Fashion Image-to-Video Synthesis via Stable Diffusion\n\n[b] Pretraining is all you need for image-to-image translation", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for your comments!\n\n$\\textbf{1. Comparative methods.}$\n\nThanks. Table R1 displays the comparison results between FaceComposer and StyleTalk. It can be observed FaceComposer has slightly better performance than StyleTalk, stemming from two aspects: 1) FaceComposer employ FLAME to represent the geometric information of the face, which is more dense and expressive than the BFM used by StyleTalk [1]; 2) As the foundation model of FaceComposer, Latent Diffusion Models has greater potential in generation than the PIRenderer of StyleTalk.\n\n$\\textbf{2. Dataset details and dataset scale.}$\n\nFirstly, considering FaceComposer may benefit from the same distribution between testing and training sets (both from HDTF) in Table 4, while other methods are not trained on HDTF, we add a test set MEAD-Neutral to remove the performance bias introduced in the testing stage, as shown in Table R1.\n\nSecondly, we agree with you that including some ablation studies of dataset scale will provide a comprehensive evaluation, so we verify the effect of dataset scale on face animation task in Table R4 (bottom half), other tasks are on the progress. We can observe that: 1) FaceComposer- is still better than other SOTA methods, proving the superiority of FaceComposer itself; 2) FaceComposer performs better than FaceComposer-, demonstrating that the large-scale dataset is essential for our FaceComposer, so we collect a large-scale high-quality training data to support different facial content creation tasks.\n\n$\\textbf{3. Missing references.}$\n\nThanks. We will add and discuss them in the final version.\n\n$\\textbf{4. Impact of the collected dataset.}$\n\nThanks. Our dataset will facilitate numerous research areas. The large data volume and high video quality of our dataset are well suited for various image/video generation tasks, such as unconditional face generation, face reenactment, and face swapping. Our dataset also has potential for 3D applications, such as 3D face generation. As a high-quality audio-visual dataset, our dataset facilitates the research in audio-visual speech recognition, speech separation, and audio-driven face animation.\nOur dataset can be used for deepfakes, which will have a negative effect. However, our dataset can also be leveraged in forgery detection tasks to prevent such concerns. We filter out harmful content from our dataset to prevent it from being used for malicious purposes. We will do our utmost to regulate the application and acquisition of our dataset to avoid potential misuse.\nWe will include the discussion in the final version. Thanks.\n\n[1] Learning a model of facial shape and expression from 4D scans, SIGGRAPH Asia 2017"}, {"review_id": "GF5azIlir0", "reviewer": "Reviewer_BHq7", "summary": "The paper presents an all-in-one pipeline that can perform face generation, editing and animation and can be driven by multiple signals such as audio, text and sketches. The proposed model is based on a Latent Diffusion Model (LDM) and works by decomposing images faces into several representations capturing identity (ArcFace embeddings), geometry (PNCCs), face shape (sketch embeddings) and text description (T2F embeddings). These embeddings are then used as conditions and guidance for the LDM, which is taken pre-trained and fine-tuned on a large dataset containing both images and videos. The model is able to train simultaneously with images and videos thanks to temporal attention modules inserted in the U-Net architecture. The model is evaluated on face generation, editing and animation and compared to SoTA methods for each task. Facial animation is performed using an audio-to-PNCC model, which predicts the 50 expression parameters and 3 jaw pose parameters from Wav2Vec embeddings.", "questions": "The initial clips in the supplementary material seem to be of a much higher quality than the ones that follow. The authors should clarify if the masks are used for the initial videos. This is will help to put the performance of the model in perspective and determine if it should be judged as a video dubbing or a one-shot speech-driven facial animation method.\n\nHow does the temporal attention used in this paper differ from that used in Make-A-Video[*] or Video Diffusion Models[**]\n\n[*] Singer et.al. \"Make-A-Video: Text-to-Video Generation without Text-Video Data\"\n[**] Ho et. al. \"Video Diffusion Models\"", "limitations": "The authors discuss the limitations of the method in the supplementary material. They also point out some negative applications of methods such as the one proposed in the paper.", "rating": 5, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 1, "strengths": "The proposed framework is indeed flexible and can perform tasks that would normally require multiple modules. The advantage of a unified model apart from the simplicity is the computational efficiency (i.e. low latency), since it can perform several steps in parallel without stringing together multiple models.\n\nThe authors have collected a curated, extensive dataset of images and videos. They make sure that the dataset only contains high quality frames and highly correlated text-image pairs. The dataset will be made public and will undoubtedly be very useful for researchers in the field.\n\nThe authors include an ablation study showing the effect of training on only videos or only images and share their hypotheses into why the addition of video data results in slightly blurrier frames.\n\nThe lip movements appear to be synchronised with the audio and this is also visible in the SyncNet and LMD metrics.", "weaknesses": "The proposed framework does not present significant methodological novelty and it is not clear what the key contributions are besides the collection of multiple conditions and pretrained models to drive a unified framework. Temporal attention has been used before in [*] and [**] and the conditioning mechanism and guidance used in the paper are standard practice for most diffusion models. Make-a-Video [*] also is trained on both image and video data and has a similar approach but is missing from the references. The audio-to-PNCC network is based on StyleTalk but is only described in the supplementary material.\n\nFrom the supplementary video the performance of the model seems to degrade a lot when the masks are not used (identity feature + PNCCs). This setting is much more challenging and the proposed model seems to struggle to preserve the identity or produce natural head motion. Furthermore the visual quality also seems to degrade. The authors also do not mention if the results of Table 4 are measured when using masks or not. If masks are used then this is an unfair comparison against other methods such as PC-AVS that are truly one-shot and do not simply in-paint the mouth. This would also explain why the SSIM and CPBD are better.\n\nThe authors only have an ablation study examining how the use of images and videos during training affects the performance of the model. They have not performed ablation studies to explore how the performance improves with the addition of each condition. \n\nThe description of the user interface of the tool is not of particular interest for machine learning research. The authors should consider removing this section from the main paper and using the space to add experiments such as ablation studies or to describe some of the components in more detail.\n\n[*] Singer et.al. \"Make-A-Video: Text-to-Video Generation without Text-Video Data\"\n[**] Ho et. al. \"Video Diffusion Models\"", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for your comments!\n\n$\\textbf{1. Novelty and contributions.}$\n\nWe would like to clarify our primary contribution, which is the unified generative framework with various means of controllability for versatile facial content creation. Our design enjoys some merits in both training and inference stages. For training, we only need to train and save one model for most tasks, reducing memory and computation cost. For inference, multi-condition-driven scheme enables us to accomplish combined tasks in one-time forward, improving inference flexibility and efficiency. We also collect a high-quality multi-modal face database to ensure the success of FaceComposer, and we hope such a database would be beneficial to further studies in this field. Besides, thanks for the reminder of the missing references: Make-A-Video and Video Diffusion Models. We will add more discussion in the revision.\n\n$\\textbf{2. Mask influence.}$\n\nFirstly, we agree with you that the setting, i.e. Identity Feature (IF) + PNCCs, is challenging since IF only indicates the identity attribute and PNCCs only represent facial geometric information, both of which do not contain detailed information, like texture, skin or hair color. \n\nSecondly, FaceComposer in Table 4 and initial clips in supplementary material have mask, and we show the mask shape in Figure R1. We agree with you that there may exist an unfair comparison between video dubbing (like Facecomposer, Wav2Lip) and one-shot talking head (like PC-AVS), although this setting is very **common** in many talking head practices [1][2][3]. Considering that both categories of methods need to generate the mouth, we add evaluation metrics only for the mouth region (SSIM-M and CPBD-M, besides M-LMD) to reduce the influence of mask, as shown in Table R1. It can be seen our SSIM-M and CPBD-M are still the best, proving the effectiveness of FaceComposer in non-mask areas.\n\nThirdly, we want to claim the mask is not the key to the generation quality of FaceComposer, and we justify it by given more videos (please see Rebuttal_Video.mp4 submitted to AC), including results of IF + PNCCs and other condition combinations without mask. The high generation quality could be attributed to the foundation generative model (LDM) and the high-quality training data.\n\n[1] A lip sync expert is all you need for speech to lip generation in the wild, ACMMM 2020   \n[2] Masked Lip-Sync Prediction by Audio-Visual Contextual Exploitation in Transformers, SIGGRAPH Asia 2022  \n[3] StyleSync: High-Fidelity Generalized and Personalized Lip Sync in Style-based Generator, CVPR 2023\n\n$\\textbf{3. Ablation studies of condition impact.}$\n\nFollowing your suggestion, we conduct ablation studies in Table R4 (upper half) to explore the condition impact. We want to explain the performance is stable, not improved, with the addition of each condition, because we only increased the number of conditions decomposed from the training data, not the training data itself. Taking the face generation in Table R4 (upper half) as example, adding extra conditions (e.g. sketch, IF) into baseline will not bring additional benefits, since they only use the T2F Embedding as condition in the test phase. In contrast, increasing the size of dataset will introduce new data information, which can improve the performance as we show in Table R4 (bottom half).\n\n$\\textbf{4. User interface.}$\n\nThanks for the suggestion. We will remove the user interface to Supplementary Material to save more spaces for experiments.\n\n$\\textbf{5. Difference of temporal attention.}$\n\nThe temporal attention in FaceComposer has no difference with that in Make-A-Video/Video Diffusion Models. Describing this module is to make our paper self-contained. We will add the missing references to avoid misunderstanding."}, {"review_id": "z5vuKrvHWz", "reviewer": "Reviewer_UEp9", "summary": "The advancement of generative models has significant progress in automatic facial content creation. However, current models pose challenges due to their high customization and inefficiencies. To address this, a unified model called FaceComposer is proposed.\n\n- The model leverages images, videos, multi-modal face datasets. \n- Better performance than SOTA. \n- The user-friendly interface of FaceComposer enables easy face generation, editing, and animation.\n", "questions": "The framework is able to deal with various tasks by using different forms of datasets. I would like to know how much data and computation cost we need for each task in order to make the proposed method work. ", "limitations": "- Excellent method supported by well designed experiments. \n- Maybe adding human eval can make the work better. \n", "rating": 7, "confidence": 4, "soundness": 4, "presentation": 4, "contribution": 4, "strengths": "- Well written text with nice figures and numerical results. \n- Promising results supported by numerical results. \n- Proposed method works in many styles\n- Unified framework enables the proposed method to work on multiple tasks. \n", "weaknesses": "- The interface is confusing. \n- Maybe adding human eval can make the work better. \n\n", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for your comments!\n\n$\\textbf{1. Confusing interface.}$\n\nWe apologize for the confusion caused by the user interface. Here, we will provide a detailed explanation of the demo page in Figure A6. The five rectangular boxes in the first row represent five condition inputs: Mask, PNCC, Sketch, Identity Feature (IF) and T2F Embedding (T2F) from left to right. FaceComposer supports condition combinations, so you can check \"nothing\" in the boxes for the conditions you don't need, and drop image/video to the corresponding boxes or click that boxes to upload them for the conditions you want. For Mask, we will provide nine parsing choices to help you mask the uploaded image or video. For PNCC, one can drop an audio to the box, then the audio2PNCC module will be called to extract the PNCC sequence. For Sketch, we will automatically extract the sketch for the uploaded image or video. For IF, only reference image is needed, ArcFace model will be called to get the IF. For T2F, you can upload the reference image, where T2F is obtained from Face Clip model, or use the input box for \"Prompt / Negative Prompt\" below to make the T2F extracted from the text. The negative prompt will be put into $c_1$, and prompt is in $c_2$ (the explanation of $c_1$ and $c_2$ can be found in line 107~109 of the main paper). Below the input box of the prompt, we show five parameters: (1) Guidance Scale is the guidance weight we used in guidance directions; (2) Num samples mean how many samples you want to generate; (3) Seed is the random seed used in random process in FaceComposer, like noise generation; (4) Steps mean the number of DDIM steps in inference process; (5) Image resolution, just as its name implies. We hope these words clear your confusion and we will consider refine the interface in the final version to make it easy to follow.\n\n$\\textbf{2. Human eval.}$\n\nThanks for your suggestion. Table R2 and R3 demonstrate the user studies we add for face generation and face animation, we skip face editing task since its user study has been shown in Table 3 of the main paper. Participants are asked to score twenty images sampled from testset for each SOTA in Table R3 and ten videos in Table R2, where Accuracy and Realism keep the same meaning as we claim in line 195~196 of the main paper, LipSync is the sync degree between lip movements and speech content, OveralRealness tells which video is more natural and realistic, VideoQuality depicts the visual quality of generated video. It can be observed that FaceComposer achieves the best results in terms of both face generation and face animation.\n\n$\\textbf{3. Data and computation cost.}$\n\nWe apologize for the misunderstanding caused by our unclear description. We only have one training data: 1.1M images + 500 hours videos, and do not use different datasets for different tasks. We decompose this multi-modal database into five face-specific conditions to train a unified model. Then in the inference phase, we can make different combinations of these conditions to achieve different tasks, as we list in Table 1 of the main paper. We spend approximately four days training FaceComposer with eight NIVIDA A100 GPUs, based on a pre-trained Latent Diffusion Models."}]

Author rebuttal:
To all reviewers:

We thank all reviewers for their efforts in reviewing our paper and appreciate their valuable comments. We will address their individual concerns in the rebuttals per review. Here, we list some concerns in common. If not specified, Table(Figure) \*/A\*/R\* represent the corresponding table(figure) in main paper/Supplementary Material/pdf file in the rebuttal, respectively. 

$\textbf{1. More experiments of face animation in Table 4.}$

For face animation, more experimental comparisons are shown in Table R1. (1) We add four SOTA methods, including StyleTalk, AVCT[1], EAMM[2] and SadTalker[3]. It can be seen FaceComposer outperforms all of them. Note that despite sharing the design principle of audio2PNCC, FaceComposer is superior to StyleTalk, since the FLAME we use to represent PNCC is more expressive than the BFM that StyleTalk adopts [4], and the Latent Diffusion Models in FaceComposer has more potential in terms of generation than the PIRenderer in StyleTalk. (2) Considering the audio2PNCC of FaceComposer is trained on HDTF, we add another testset (MEAD-Neutral, a subset from MEAD with only neutral expression) to demonstrate our superiority across different data distributions. (3) Since some methods use mask (e.g. FaceComposer, Wav2Lip) and some use one-shot (e.g. PC-AVS, MakeItTalk, StyleTalk), in order to reach a fair comparison, we add two new metrics (SSIM-M and CPBD-M) besides M-LMD, to only evaluate the mouth region quality. SSIM-M and CPBD-M stand for SSIM and CPBD calculated in the mouth area, respectively.

$\textbf{2. Ablation study of condition numbers.}$

We add the ablation studies in Table R4 (upper half) to investigate the effect of different numbers of conditions on FaceComposer. Considering face generation/editing/animation are the basic tasks, we take FaceComposer with three conditions (T2F Embedding, Mask, PNCC) as baseline, "baseline + Sketch" means baseline with Sketch condition, "baseline + Sketch + Identity Feature" is equal to FaceComposer with all five conditions. It can be seen that Facecomposers with different number of conditions keep stable performance, no matter in face generation, editing or animation task. We argue this is reasonable, bacause the training dataset is fixed, when the number of conditions increases, no additional information is introduced for a specific task. Note that adding conditions does not increase the training set, but increases the number of conditions that are decomposed from the training set.

$\textbf{3. Ablation study of dataset scale.}$

To demonstrate the impact of dataset scale on generation quality, we show an ablation study on face animation in Table R4 (bottom half). More experiments on face generation/editing are currently being conducted and will be included in the final version. Considering the SOTA methods generally have dozens of hours of training data, we reduce the training data of FaceComposer to the similar scale for a fair comparison. Specifically, we randomly sampled 10 hours of video and 4.5W images from our original dataset to train a FaceComposer (denoted as FaceComposer-). From Table 4 (bottom half), it can be observed that FaceComposer- is inferior to FaceComposer due to the decrease of data information, but it is still better than other SOTA. Besides the performance gap caused by small dataset, we would like to clarify FaceComposer is a unified generative model that requires a large-scale training data to benefit a variety of facial content creation tasks. Therefore, we collect a large-scale multi-modal face database and will make it public under the constraints of ethical review. 

[1] One-shot Talking Face Generation from Single-speaker Audio-Visual Correlation Learning, AAAI 2022  
[2] EAMM: One-Shot Emotional Talking Face via Audio-Based Emotion-Aware Motion Model, SIGGRAPH 2022  
[3] SadTalker: Learning Realistic 3D Motion Coefficients for Stylized Audio-Driven Single Image Talking Face Animation, CVPR 2023  
[4] Learning a model of facial shape and expression from 4D scans, SIGGRAPH Asia 2017

Meta-review: The paper received 2 Borderline Accept, 1 Weak Accept, and 1 Accept decision. The paper addresses several face-generative tasks in one model, which benefits many applications. The collection of a new dataset comprising more than 500 hours of talking face videos is also an important contribution. Although the original manuscript had several analyses missing, the authors presented them well in the rebuttal. Ethics Reviewers raise some concerns; the authors partially addressed them, and the paper can be released with proper care.

The ACs decided to accept the paper to NeurIPS. The authors should include the extra analyses presented in the rebuttal in the final paper version. The authors also should follow the recommendations from Ethics Reviewers.
Decision: Accept (poster)