SlowGuess/NeurIPS_2023_OCR huggingface dataset:
Jiayu Wang\*, Kang Zhao\*, Yifeng Ma\*, Shiwei Zhang\*, Yingya Zhang\*, Yujun Shen\*, Deli Zhao\*, Jingren Zhou
1Alibaba Group 2Tsinghua University 3Ant Group
{wangjiayu.wjy, zhaokang.zk, mayifeng.myf, zhangjin.zsw, yingya.zyy, jingren.zhou}@alibaba-inc.com
{shenyujun0302,zhaodeli}@gmail.com
This work presents FaceComposer, a unified generative model that accomplishes a variety of facial content creation tasks, including text-conditioned face synthesis, text-guided face editing, face animation etc. Based on the latent diffusion framework, FaceComposer follows the paradigm of compositional generation and employs diverse face-specific conditions, e.g., Identity Feature and Projected Normalized Coordinate Code, to release the model creativity at all possible. To support text control and animation, we clean up some existing face image datasets and collect around 500 hours of talking-face videos, forming a high-quality large-scale multi-modal face database. A temporal self-attention module is incorporated into the U-Net structure, which allows learning the denoising process on the mixture of images and videos. Extensive experiments suggest that our approach not only achieves comparable or even better performance than state-of-the-arts on each single task, but also facilitates some combined tasks with one-time forward, demonstrating its potential in serving as a foundation generative model in face domain. We further develop an interface such that users can enjoy our one-step service to create, edit, and animate their own characters. Code, dataset, model, and interface will be made publicly available.
Due to the rapid development of generative models, such as diffusion models (DMs) [10, 15, 47], VAEs [21], GANs [6] and flow models [5] in the computer vision area, automatic content creation has recently received an increasing amount of attention for its real-world applications. Benefiting from these generative models, facial content creation as a critical part has recently achieved impressive progress and simultaneously shows great application potential, e.g., virtual digital human, artistic creation, and intelligent customer service.
Existing face generative models [20, 23, 39, 61] are usually developed as highly customized systems, meaning that one model can only handle one task. However, this design poses two significant challenges: 1) hard to accomplish complex tasks, such as integrating face creating, editing and then animating the generated face in a single step; 2) redundant consumption of memory and computation. For example, one needs to train and save a number of models to build a multi-functional system, and perform complicated inference processes. The challenges could inevitably limit its further applications and development.
To tackle these problems, we propose compositional FaceComposer in this work, a unified model that is capable of simultaneously tackling versatile facial tasks, including face generation, face editing,
face animation, and their combinations. Specifically, we decompose a given face into multi-level representative factors, e.g., Identity Features, Projected Normalized Coordinate Code (PNCC) [64] and Text2Face (T2F) Embeddings, and then train a powerful latent diffusion model conditioned on them to compose the input face. This design provides the model with exceptional controllability over facial content creation, enabling a seamless process from generating a face to utilizing it. In particular, we have additionally incorporated temporal self-attention to enable joint learning from both images and videos simultaneously. To optimize FaceComposer, we finally gather a high-quality large-scale multi-modal face dataset, including 1.1 million face images from pre-existing datasets and 500 hours of meticulously cleaned talking face videos.
Extensive quantitative and qualitative results demonstrate that FaceComposer achieves exceptional performance in various facial content creation tasks. FaceComposer surpasses previous state-of-the-art methods in face generation, face editing, and audio-driven face animation in terms of the most widely-used evaluation metrics. Furthermore, we have devised intricate yet imaginative tasks to showcase the advantages of our diverse condition composition.
Face generation. The goal of face generation [34, 53, 56, 57] is to generate photo-realistic face images. Among them, StyleGAN series [17-19] boost the generation quality by introducing controllable and interpretable latent space. After that, some variants [51, 63, 42, 41, 52] are proposed for further quality improvement. One of them, TediGAN [51], maps text into the StyleGAN latent space to make the text-guided face generation. Another example is LAFITE [63], which presents a language-free training framework with a conditional StyleGAN generator. Recently, diffusion models [10, 32, 33, 38, 37] become more and more popular in image synthesis area due to their strong generative abilities. We find fine-tuning a pre-trained diffusion model will bring comparable or even superior performance to the GAN paradigm.
Face editing. Face editing [20, 23, 25, 27, 54] aims to manipulate the face images guided by text descriptions or masks. Similar to face generation, StyleGAN-based methods [51, 29, 49, 65] show remarkable performance in face editing. Besides TediGAN, StyleClip [29] combines representation ability of CLIP [31] and generative power of StyleGAN to optimize the editing direction. In contrast to the above methods, CollDiff [13] trains the diffusion model with multi-modal conditions (text + mask), in order to make them complementary. It is noted that TediGAN and CollDiff both support face generation and editing, which differ from ours in the following aspects: 1) TediGAN is essentially designed as a bespoke system, i.e., it can only complete one task (generation or editing) in one model inference. 2) CollDiff uses multiple diffusion models (one model for one condition), leading to low efficiency in training and inference stage. 3) FaceComposer only use one diffusion model in a unified framework, and can finish different facial creation combinations in one model inference.
Face animation. Face animation [39, 36, 50, 55, 35, 30, 61, 62, 26, 44] intends to make the target face move according to a driving video or audio. Video-driven methods [39, 36, 50, 55, 35] focus on modeling the motion relationships between source and target face. For example, FOMM [39] decouples appearance and motion and introduces keypoints representation to support complex motions. PIRenderer [36] employs 3DMM to better control the face motions. Compared with keypoints and 3DMMs, the PNCC we used contains more dense and intuitive facial structure information, making it easier to be learned by the model. Audio-driven methods [30, 61, 62, 26, 44] pay more attention to the lip synchronization (lip-sync) between the audio and the mouth of target face. MakeItTalk [62] disentangles the content and speaker information to control lip motions. By learning from a powerful lip-sync discriminator, Wav2lip [30] improves lip-sync accuracy. PC-AVS [61] generates pose-controllable talking faces by modularizing audio-visual representations. However, all of them only consider intra-frame information for reconstruction, ignoring inter-frame relationship, which is fixed by adding a temporal attention module in our FaceComposer.
3 FaceComposer
Recent days have witnessed the powerful generative ability of Latent Diffusion Models (LDMs) [37], which is consequently equipped as the backbone of our FaceComposer. We model various facial content creations as a multiple-condition-driven denoising process. Fig. 1 shows the overview of
Figure 1: The framework of FaceComposer, which takes  $f$  frames and five face-related conditions as input, uses LDMs to predict the noise added in the latent space. We can combine diverse conditions to finish face generation/editing/animation or their combinations. For example, the green  $\triangle$  conditions are for face generation, yellow  $\square$  for face editing, and red  $\star$  for face animation.
FaceComposer. In the unified framework, the inputs of different tasks are expressed as diverse conditions, including Mask, PNCC, Sketch, Identity Feature and T2F Embedding. We achieve versatile facial content creations through condition combinations (Sec. 3.2). Different from the standard LDMs that is only designed for image generation, FaceComposer supports both static and dynamic content creations, i.e. our dataset contains both images and videos. So we introduce a temporary attention module into LDMs for the two modalities joint training (Sec. 3.3 and Sec. 3.4).
We denote the input  $f$  frames as  $\pmb{x}_0\in \mathbb{R}^{f\times 3\times H\times W}$ , where  $H$  and  $W$  are height and width of input frames (we set  $H = W = 256$  in experiments).
Latent diffusion model. To save the computational resources of DMs, we follow LDMs to encode frames into latent space:  $\pmb{z}_0 = E(\pmb{x}_0) \in \mathbb{R}^{f \times C_{\text{latent}} \times h \times w}$  with a pre-trained image encoder  $E$ , where  $C_{\text{latent}}$  means the dimension of latent space,  $h$  and  $w$  are set to 32 in practice. And in the end of denoised process, the final  $\widetilde{\pmb{z}}_0$  will be mapped into pixel space with image decoder  $D: \widetilde{\pmb{x}}_0 = D(\widetilde{\pmb{z}}_0) \in \mathbb{R}^{f \times 3 \times H \times W}$ . In the latent space, the diffusion model  $\pmb{z}_{\theta}$  can be parameterized to predict the added noise:
$$
\mathcal {L} _ {\text {s i m p l e}} = \mathbb {E} _ {\boldsymbol {z}, \epsilon , t, C} \left[ \| \epsilon - \boldsymbol {z} _ {\theta} \left(a _ {t} \boldsymbol {z} _ {0} + \sigma_ {t} \epsilon , C\right) \| _ {2} ^ {2} \right], \tag {1}
$$
where  $C$  denotes the condition,  $t \in \{1, \dots, T\}$ ,  $\epsilon \in \mathcal{N}(0, 1)$  is the random Gaussian noise,  $a_t$  and  $\sigma_t$  are two scalars related to  $t$ . We freeze  $E$  and  $D$ , and start from a pre-trained LDMs.
Compositional generation pipeline. As a pioneering work of compositional generation,Composer [12] decomposes an image into eight conditions to improve the controllability of image synthesis, inspiring us to treat the inputs of different facial content creations as multiple conditions, i.e.,  $C$  in Eq.1 is a condition set. And we adopt the same guidance directions as [12]:  $\hat{z}_{\theta}(\boldsymbol {z}_t,\boldsymbol {C}) = \omega \boldsymbol{z}_{\theta}(\boldsymbol {z}_t,\boldsymbol {c}_2) + (1 - \omega)\boldsymbol{z}_{\theta}(\boldsymbol {z}_t,\boldsymbol {c}_1)$ , where  $\omega$  is the guidance weight,  $c_{1}$  and  $c_{2}$  are two subsets of  $C$ , respectively.
3.2 Diverse face-specific conditions
Condition decomposition We convert different inputs into the following five face-specific conditions.
Mask: Mask is used to force FaceComposer to generate or edit a face in a certain region. Based on the nine parsing areas of face [22], we randomly mask one or all of them. When  $f > 1$ , different frames mask the same region. Before taken as a condition, masks are also mapped into latent space through  $E$ .
Table 1: Versatile creations based on condition compositions.  $M, S, {PNCCs},{ID}$  and  ${T2F}$  are short for Mask, Sketch, PNCC sequence, Identity Feature, and T2F Embedding, respectively.
Single Creation	Versatile Creations
Task	Conditions	Task	Conditions
face generation	① T2F	face generation+animation	① PNCCs+T2F
② S	② PNCCs+ID
③ ...	③ ...
face editing	① M+T2F	face generation+editing	① ID+M
② M+S	② ID+T2F
③ ...	③ ...
face animation	① M+T2F+PNCCs	face generation+editing+animation	① ID+T2F+PNCCs
② ...	② ...
PNCC: PNCC [64] represents the geometric information of the face, a pre-defined PNCC sequence can effectively guide facial animation generation. We use FLAME Fitting [24] to extract the PNCC of all frames. Different from the mask, the distribution of PNCC differs significantly from the original frame, so we add a trainable module  $E_{p}$  to encode the PNCC.
Sketch: Sketch describes the contours of different parts of the face (e.g., face shape, eye size, mouth location), it contains local details with low semantics. We adopt the same method as [12] to extract the sketches. Similar to PNCC, we input sketch information into a trainable  $E_{s}$  module to obtain the condition.
Identity Feature: Identity Feature indicates the identity attribute, excluding trivial information (such as hair color, texture, expression). It can direct the model to generate a face with the specified ID. We use ArcFace [3] to get the Identity Feature.
T2F Embedding: Text2Face (T2F) Embedding has two functions: 1) complementing Identity Feature with detailed information; 2) enabling text control. Specifically, during the training stage, T2F Embedding is extracted from reference image with Face Clip [60] to feed facial details into FaceComposer. In the denoising process, besides the reference image, we can also obtain T2F Embedding from text prompt with an extra prior model (similar to DALL-E 2 [33]).
Conditioning mechanisms Considering Mask, PNCC and Sketch represent the spatial local information of frames, they are all extracted from  $\pmb{x}_0$  and concatenated with  $\pmb{z}_t$  in the channel dimension. In contrast, Identity Feature and T2F Embedding define the global semantic information, hence we get them from the reference image, and add the projected Identity Feature into time embedding, serve T2F Embedding as key and value for cross attention module of  $\pmb{z}_{\theta}$  (corresponding to Spatial Att in Fig. 1). Note that, except for PNCC, which needs to be fitted in advance, all left conditions are extracted on-the-fly. We adopt a similar condition training strategy to Composer: setting 0.5 dropout probability for each condition, 0.1 to drop all conditions, and 0.1 to reserve them all.
Condition composition As we mentioned above, we can support versatile tasks: face generation, face editing, face animation and their combinations by combining our diverse conditions.
We list some representative creations in Tab. 1. Taking face animation as an example, we use the PNCC sequence (predicted by audio), Mask (masking the mouth region of the target face), and T2F Embedding (providing texture information from reference image) to generate Talking Head. It is worth mentioning that FaceComposer can finish face generation + editing + animation with a one-time forward by conditioning on Identity Feature (guiding face generation), T2F Embedding (extracted from editing prompt) and PNCC sequence (obtained from reference video). More details are shown in Sec. 4.3.
3.3 Temporal self-attention module
In order to create static and dynamic contents simultaneously, we prepare a multi-modal database, consisting of both images and videos. And we argue that joint image-video training is important
Figure 2: The qualitative results of face generation.
for our FaceComposer, considering face image will align facial content and text description, while video will link spatial and temporal information of face sequences.
Inspired by [40, 11], we introduce a temporary self-attention (TSA) layer into LDMs (corresponding to Temporal Att in Fig. 1), and select half batch samples from images (i.e.  $f = 1$  for  $0.5B$ ) and the other half from videos (i.e.  $f = 5$  for left  $0.5B$ ) to make the joint strategy fully benefit each other within each batch (see ablation study in Sec. 4.4), where  $B$  is the total batch size. Assuming the input of TSA is in the shape of  $B \times f \times C_{inter} \times h' \times w'$ , it will be arranged to  $\{B \times h' \times w'\} \times f \times C_{inter}$  before entered into TSA, where  $C_{inter}, h', w'$  are the intermediate channel dimension and feature map size. When  $f = 1$ , TSA degrades to an identity transformation.
3.4 Multi-modal face database
To empower FaceComposer with image and video generation capabilities, we construct a high-quality large-scale multi-modal face database comprising 1.1 million face images with text annotations and approximately 500 hours-long talking face videos.
Image data. To construct the image part of our database, we carefully clean up LAION-Face [60] and merge the cleaned dataset with CelebA-HQ [16] and FFHQ [17]. We clean up LAION-Face using two approaches. Firstly, We use CLIP [31] to filter out the image-text pairs whose text descriptions do not match the images. Specifically, for each image-text pair, we compute the cosine similarity between CLIP features extracted from the image and the text and filter out the pair if the similarity is lower than a predefined threshold. Secondly, we use an off-the-shelf face detector [4] to detect faces in images and filter out images with no faces detected. Finally, we obtain the cleaned LAION-Face dataset. It contains 1 million face images with corresponding text captions.
Video data. To construct the video part of our database, we collect talking face videos from Youtube, BBC television, and some other web data. We manually clean collected videos to ensure high video quality and audio-visual coherence. Our collected talking face dataset includes more than 500 hours  $720\mathrm{P}\sim 1080\mathrm{P}$  videos with the audio track. We will release the dataset when the paper is made public. For more details, please refer to Supplementary Material.
Implementation details. During the training, our model starts from a pre-trained LDMs*, and is further trained on our multi-modal face database through a joint training mechanism. To enable text
*https://github.com/Stability-AI/stablediffusion
Figure 3: The qualitative results of face editing. (a) The source images. (b) The images whose attributes are edited through text descriptions by different methods. (c) The stylized results generated by FaceComposer. FaceComposer is able to edit the style of images using text (the left column) or a style reference image (the right column).
control, we trained a  $1B$  parameter prior model for projecting captions to T2F Embeddings. For the LDMs, we pretrain it with  $1M$  steps on the full multi-modal dataset using only T2F Embeddings as the condition, and then finetune the model for 200K steps with all conditions enabled. The prior model is trained for  $1M$  steps on the image dataset.
Evaluation tasks. We evaluate FaceComposer on face generation, face animation and face editing tasks, which respectively using the Multi-Modal CelebA-HQ [51], HDTF [59] + MEAD-Neutral (a subset of MEAD [45] that only contains the neutral facial expression videos), and the randomly selected images from both CelebA and Non-CelebA datasets with randomly chosen descriptions.
Evaluation metrics. For face generation task, we adopt Fréchet inception distance (FID) [9] to measure the image quality. Since FID cannot reflect whether the generated images are well conditioned on the given captions, we choose to adopt R-precision [53] as another evaluation metric. For face editing task, we compute the identity similarities (IDS) between the input faces and the edited faces to measure the identity consistency. For face animation task, we adopt the Landmark Distance around mouths (M-LMD) [1], the Landmark Distance on the whole face (F-LMD), the structural similarity (SSIM) [48], the Cumulative Probability of Blur Detection (CPBD) [28], and the confidence scores of SyncNet [2] ( $\mathrm{Sync}_{\mathrm{conf}}$ ) as evaluation metrics. To facilitate a more equitable comparison between methods generating only the mouth region (visual dubbing methods) and those generating the entire face (one shot methods), we computed the SSIM and CPBD for the mouth region, denoted as SSIM-M and CPBD-M, respectively. For more details about the experimental settings, please refer to Supplementary Material.
4.2 Comparisons on single task
In this subsection, we demonstrate the performance of FaceComposer on different tasks, including static content creation, e.g., face generation and editing, and dynamic content creation task, e.g., face animation. Note that for the face animation task, we use audio to drive face motions. We train an Audio2PNCC model to generate PNCCs with lip motions controlled by input audio.
Face generation. We compare our method with TediGAN [51], LAFITE [63] and CollDiff [13]. We generate 30,000 images for each method using randomly selected unseen text descriptions. Tab. 2 shows that our method achieves the best results in terms of both FID and R-precision. Such results
Figure 4: Qualitative results of face animation. It can be seen that FaceComposer not only achieves accurate lip-sync but also produces high-fidelity results in the mouth area.
Table 2: Results of face generation.
Method	FID↓	R-precision(%)↑	Accuracy↑	Realism↑
TediGAN [51]	107.14	44.96	2.80	3.55
CollDiff [13]	98.76	67.41	3.43	2.96
LAFITE [63]	12.54	81.94	4.07	3.88
Ours	11.34	86.63	4.86	4.67
Table 3: Results of face editing.
Method	IDS↑	Accuracy↑	Realism↑
TediGAN [51]	0.62	2.92	2.09
StyleClip [29]	0.75	4.27	2.87
Ours	0.94	4.58	4.47
demonstrate the effectiveness of massive training data. The qualitative comparison is shown in Fig. 2. We can observe that other methods can generate text-relevant results. However, their results lack some attributes contained in the input captions. On the other hand, our results achieve higher fidelity while maintaining text-visual coherence.
We also evaluate the Accuracy and Realism through a user study, in which the users are asked to judge which image is more realistic, and more coherent with the given captions. We randomly sampling 30 images with the same text conditions and collect more than 20 surveys from different people with various backgrounds. Compared with the state-of-the-arts, FaceComposer achieves better accuracy and realism values as shown in Tab. 2, which proves that our methods can generate images with the highest quality and text-relevance.
Face editing. To evaluate the face editing performance of FaceComposer, we compare our method with TediGAN [51] and StyleClip [29]. Results are shown in Tab. 3, and we can see that our method achieves better results in all metrics (accuracy and realism come from user study, following the same scheme in Tab. 2), which proves that our methods can generate images with the highest quality and text-relevance. We show some visual results in Fig. 3. As we can see, our methods can perform face editing more precisely, and the achieved results are the most photorealistic and most coherent with the given texts.
Face animation. We compare our method with state-of-the-art methods including MakeItTalk [62], Wav2Lip [30], PC-AVS [61], AVCT [46], EAMM [14], StyleTalk [26], SadTalker [58], and StyleSync [7]. The samples of the compared methods are generated either with their released codes or with the help of their authors. As shown in Tab. 4, FaceComposer achieves the best performance in most metrics. The strong performance in SSIM, SSIM-M, CPBD, and CPBD-M metrics indicates the high quality of videos generated by FaceComposer. Given that Wav2Lip employs SyncNet as a discriminator during its training phase, it reasonably attains the highest confidence score from SyncNet, surpassing even the ground truth. Numerous prior arts [7, 58, 26, 8, 43] have reported that Wav2Lip, despite attaining high SyncNet scores, does not fare well in qualitative evaluations (e.g., user studies) of lip-sync, attributed to the production of blurry results and, occasionally, exaggerated