SlowGuess/NeurIPS_2023_OCR huggingface dataset:
Table 4: Results of face animation on HDTF and MEAD-Neutral.
Methods	HDTF / MEAD-Neutral
SSIM↑	CPBD↑	F-LMD↓	M-LMD↓	Syncconf↑	SSIM-M↑	CPBD-M↑
MakeItTalk [62]	0.63/0.74	0.19/0.10	4.10/3.88	4.22/5.41	3.07/2.02	0.62/0.69	0.15/0.06
Wav2Lip [30]	0.75/0.79	0.18/0.12	2.01/2.38	2.54/2.95	5.27/3.99	0.68/0.78	0.08/0.03
PC-AVS [61]	0.51/0.51	0.23/0.07	3.64/4.76	3.52/3.91	4.16/3.09	0.60/0.67	0.10/0.05
AVCT [46]	0.73/0.77	0.17/0.10	2.85/2.68	3.53/4.46	3.81/2.56	0.70/0.73	0.16/0.08
EAMM [14]	0.59/0.41	0.08/0.08	4.16/7.39	4.19/5.03	2.30/1.40	0.60/0.71	0.13/0.05
StyleTalk [26]	0.78/0.79	0.23/0.12	2.10/2.35	2.40/2.80	4.17/3.05	0.76/0.80	0.16/0.09
SadTalker [58]	0.61/0.73	0.21/0.12	3.98/3.67	3.46/4.09	4.05/2.62	0.61/0.69	0.15/0.10
StyleSync [7]	0.77/0.80	0.21/0.12	1.93/2.22	2.36/2.76	4.21/3.10	0.76/0.80	0.17/0.10
Ground Truth	1/1	0.23/0.20	0/0	0/0	4.52/3.57	1/1	0.21/0.12
FaceComposer	0.78/0.84	0.27/0.14	1.84/2.16	2.25/2.70	4.27/3.12	0.78/0.83	0.18/0.10
Table 5: User study results of different methods on HDTF and MEAD-Neutral for the face animation. LS, VQ, OR stand for user study metrics LipSync, VideoQuality and OverallRealness, respectively.
Methods	HDTF / MEAD-Neutral
LS↑	VQ↑	OR↑
MakeItTalk	1.71/2.20	1.87/2.38	1.44/1.74
Wav2Lip	2.93/3.33	1.02/1.10	1.10/1.12
PC-AVS	2.88/3.20	1.97/2.46	1.73/1.98
AVCT	2.04/2.76	2.60/2.66	2.46/2.62
EAMM	1.90/2.58	1.30/1.78	1.62/1.94
StyleTalk	3.10/3.60	3.08/3.00	2.44/2.82
SadTalker	3.22/3.68	2.82/2.92	1.97/2.42
Ground Truth	4.34/4.56	4.06/4.26	4.22/4.40
FaceComposer	3.53/3.96	3.38/3.50	2.93/3.73
lip motions. Our method's confidence score aligns most closely with the ground truth, and our method's M-LMD scores are the best. This indicates that FaceComposer achieves precise lip-sync. We provide more analysis regarding quantitative evaluation in the Supplementary Material.
Fig. 4 shows the qualitative results. It can be seen that MakeItTalk, AVCT, and EAMM fail to produce accurate mouth movements, while FaceComposer achieves accurate lip-sync. Although PC-AVS and Wav2Lip are competitive with FaceComposer in terms of lip-sync, they can only generate blurry results. FaceComposer produces high-fidelity results in the mouth area. StyleTalk often produces overly smooth lip movements. The lip-sync of SadTalker is slightly inferior to our method. StyleSync occasionally yields jittery and exaggerated motions.
We conduct a user study involving 20 participants. Each participant is requested to assign scores to five videos sampled from the test dataset for each method, evaluating them on three aspects: (1) Lip sync accuracy; (2) Video quality; (3) Overall Realness. The scoring employs the Mean Opinion Scores (MOS) rating protocol (scoring from 1 to 5, larger is better). Tab. 5 shows the results. Our method achieves the best in all aspects, which demonstrates the effectiveness of our method.
4.3 Advantage of unified framework
In this subsection, we demonstrate the compositional generation capabilities, which are depicted in Tab. 1, of FaceComposer with one-time forward. Here we provide two examples of versatile creations. In practical application, one can customize different conditions to meet specific requirements. More results will be provided in Supplementary Material.
Face generation+animation. FaceComposer has the ability of generating a desired face with a random T2F Embedding and animating it with a one-time forward. We show six examples in Fig. 5 to demonstrate such an ability. As we can see, the generated results in each row share almost the same mouth shape as that of the driving PNCCs. Meanwhile, each generated result consistently adheres to the given T2F Embeddings.
Figure 5: Face generation and animation results. FaceComposer supports generating desired faces with T2F Embedding and animating faces with PNCCs simultaneously in a one-time forward.
Face generation+editing+animation. Besides, FaceComposer can incorporate additional identity conditions to animate a given face in a prescribed style. Some results are shown in Fig. 7. It is clear that each animated frame has the same mouth shape and expression as that of the corresponding PNCCs, while maintaining its original identity and face shape.
To isolate the contributions of our designs, we conduct the ablation studies from three aspects. Note that the Audio2PNCC model is identical in all variants and we only compare the ability of these variants to convert PNCCs to videos.
Image/Video dataset impact. We compare the following three variants: (1) only train FaceComposer using videos ( $w/o$  Image), (2) only train FaceComposer using images ( $w/o$  Video), (3) our full model (Full), for face animation on HDTF dataset
in Tab. 6. Full achieves the best performance in most metrics, which demonstrates the effectiveness of joint training on image and video data. We observe that w/o Video attains the highest score in CPBD metric. This may be due to two reasons. First, high-fidelity face images from CelebHQ and FFHQ are incorporated into our image data. Second, although our video dataset is in high definition, the motion in the videos may blur certain frames, which may lower the model's score in the CPBD metric.
Fig. 6 shows the qualitative ablation results.  $w/o$  Image fails to generate realistic texture in the mouth area, i.e. the teeth and the tongue. This may imply that high-resolution image data are crucial to learning to generate high-quality textures.  $w/o$  Video fails to generate mouth shapes corresponding to input PNCCs. This
Table 6: Ablation results of face animation on HDTF
Method	SSIM↑	CPBD↑	F-LMD↓	M-LMD↓
w/o Image	0.78	0.27	1.88	2.46
w/o Video	0.77	0.29	1.94	2.40
Full	0.78	0.27	1.84	2.25
w/o Video
w/o Image
Figure 6: Qualitative ablation results.
may stem from the fact that the image data only contains images with a limited number of mouth shapes. On the other hand, Full generates high-fidelity results in the mouth area, produces accurate mouth shape, and attains satisfactory frame consistency. This indicates the necessity of constructing and learning from a multi-modal face database.
Condition numbers. We investigate the effect of different numbers of conditions on FaceComposer in Tab. 7. Considering face generation/editing/animation are the fundamental tasks, we take T2F, Mask and PNCC as three basic conditions (i.e., FaceComposer  $w / o$  ( $\mathbf{ID} + \mathbf{S}$ ), add Sketch condition in FaceComposer  $w / o$  ID, and keeps all five conditions in FaceComposer. It can be seen that
Figure 7: Face generation+editing+animation results. FaceComposer is able to generate face images with specified identities, edit the image styles, and animate face images simultaneously in a one-time forward.
Table 7: Ablation results with different conditions or data scales for face generation, editing, and animation.
Variants	Face Generation	Face Editing	Face Animation
FID↓	R-precision(%)↑	IDS↑	SSIM↑	CPBD↑	F-LMD↓	M-LMD↓
FaceComposer	11.34	86.63	0.94	0.78	0.27	1.84	2.25
FaceComposer w/o ID	11.40	86.28	0.94	0.78	0.26	1.89	2.28
FaceComposer w/o (ID+S)	11.27	86.91	0.93	0.78	0.26	1.88	2.28
FaceComposer-	11.60	86.85	0.93	0.78	0.26	1.91	2.37
FaceComposers with different number of conditions keep stable performance, no matter in face generation, editing or animation task. This is reasonable, because the training dataset is fixed, when the number of conditions increases, no additional information is introduced for a specific task.
Dataset scale. To demonstrate the impact of dataset scale on generation quality, we show an ablation study in Tab. 7. Considering the state-of-the-art methods generally have dozens of hours of training data, we reduce the training data of FaceComposer to the similar scale for a fair comparison. Specifically, we randomly sampled 10 hours of video and 4.5W images from our original dataset to train a FaceComposer (denoted as FaceComposer-). From Tab. 7, it can be observed that FaceComposer- is inferior to FaceComposer due to the decrease of data information, but it is still better than other state-of-the-arts in Tabs. 2 to 4.
This paper introduces a unified framework (named as FaceComposer) for versatile facial content creation, i.e., 1) generating faces from scratch, 2) adjusting portion of the generated ones, and 3) making the generated target move. It demonstrates a complete creative process from static (image) to dynamic (video) content. FaceComposer employs a LDM with multiple conditions to handle all above facial creation tasks. We evaluate its performance in various tasks and demonstrate our versatile creative ability in different combined tasks. Demo interface, Social impact and Ethic statement will be moved to Supplementary Material due to limited space.
[1] L. Chen, R. K. Maddox, Z. Duan, and C. Xu. Hierarchical cross-modal talking face generation with dynamic pixel-wise loss. In IEEE Conf. Comput. Vis. Pattern Recog., 2019.