Feature-Driven Talking Face Generation with StyleGAN2
Authors: Tao Zhang
Kai Tang
Weiwu Zhang
Kazushige Ouchi
Venue: Submitted to ICLR 2023 (2023)
Keywords: Talking face
GAN
Feature Selection

Abstract: In this work, we wish to use a face image that generate a more natural and real face talking animation video. This is not an easy task because face appearance variation and semantics of speech are coupled together when tacking face have a micro movement. Audio features sometimes contain information about expressions, but they are not accurate enough. So a single audio feature cannot fully represent the movement of the face. For the above reason, we want to use different features to generate talking faces. The StyleGan series show good performance in the direction of image processing, and can perform the style migration task of portraits very well at the same time. We find that StyleGan can be used as a talking face generator. At the same time, we also encode and extract non-identity features and non-lip features, and try to find the subtle relationship between the features and the talking face. We also use the evaluation and ablation study to measure the quality of the generated videos and examine whether our approach is effective and feasible.

Under review as a conference paper at ICLR 2023
FEATURE-DRIVEN TALKING FACE GENERATION WITH STYLEGAN2
Anonymous authors Paper under double-blind review
ABSTRACT
In this work, wish face image that generate more natural and real face talking animation video. This is not an easy task because face appearance variation and semantics of speech are coupled together when tacking face have micro movement. Audio features sometimes contain information about expressions, but they are not accurate enough. So a single audio feature cannot fully represent the movement of the face. For the above reason, we want to use different features to generate talking faces. The StyleGan series show good performance in the direction of image processing, and can perform the style migration task of portraits very well at the same time. We find that StyleGan can be used as a talking face generator. At the same time, we also encode and extract non-identity features and non-lip features, and try to find the subtle relationship between the features and the talking face. We also use the evaluation and ablation study measure the quality of the generated videos and examine whether our approach is effective and feasible.
INTRODUCTION
Using the characteristics of multimedia to realize the interaction between virtual characters and users one of the applications of Al technology. Audio is often easier obtain compared with video. The task of using audio and an image of face to generate video animation has recently been carried out by more and more researchers to realize and explore. Solving the task essential to achieve wide range of practical applications, such as virtual character interaction, copying videos in other languages, video of a conference or role-playing game, and so on.
Graphics-based face animation generation methods often require a completely original video se- quence as input (Liu & Ostermann 2011) (Garrido et al. 2015) (Suwajanakorn et al. 2017). Fried al. (2019) proposed proposed new method to edit the conversation header video based its transcript to produce a real output video. However, a retimed background video is required as input in their method. It takes about hour of video produce the best quality results. There is also model that takes into account the movement of the face and uses landmarks to drive it (Wang et al. 2019) (Za- kharov 2019) (Gu et al., 2020). There have also been many ways generate facial animations through audio drivers in recent times (Jamaludin et al. 2019). Prajwal et al. (2020) use the pre- trained lip-syne model and add it the system that generates facial animation obtain the effect of lip synchronization.
In numan-to-human communication, speech sounds inevitably involve lip movements. That is, the speaker's 's lip movements and speech are closely related. In speech recognition and speaker recog- nition, the most commonly used speech features are Mel-scale Frequency Cepstral Coefficients (MFCC). Utilize MFCC to obtain speech-related features and find the relationship with the features of lip movement when a person speaks. However, it is not possible to obtain all the information about face motion with audio alone. To make the face more natural, additional features are needed to make the generated face have more realistic features.
possible using speech to generate mouth animation.
There are many ways to generate talking faces driven by speech, we propose a method using Style- Gan2 generate animation. At the same time, considering lin-lii-eelated features, try to extract some features except identity features and lip features. Combine audio features with facial features to hopefully get more realistic facial animation.

---

Under review as a conference paper at ICLR 2023
-20
160
202
Figure I: The left parameter dimension map and the right parameter amplitude map after piece of audio passes through MFCC.
The rest of the paper is organized as follows: we survey the recent developments in this field in Section 2. In Section 3, we introduce our approach. In Section 4, training details are shown. After these, we evaluate our method. Finally, conclude our work in section 6.
2 RELATED WORK
In this section, we briefly review some related works talking face generation.
2.1 GENERATIVE ADVERSARIAL NETWORK (GAN)
Generative adversarial network (GAN) is framework that is composited of a generator network and a discriminator network. can utilized to train a generative model. The generator tries to fool the discriminator maximally. On the other hand, the discriminator also tries to discriminate the generated samples from the true ones as much as possible. By using this manner, both the generator and discriminator's performance can be improved in the end. GAN methods have wildly utilized for many computer vision tasks, for example, image synthesis (Radford et al. 2016). image super- resolution (Ledig et al. 2017). and image style transfer (Zhu al. (017)) In recent years, some methods proposed to improve the original GAN from different perspectives. Such as Conditional GAN (CGAN) (Mirza Osindero, 2014). the InfoGAN (Chen et al. 2016), and the CycleGAN (Zhu et al., 2017). The GAN methods also can be utilized in the data enhancement. For example, we can use GAN generate different action images of people train an action recognition model.
2.2 TALKING FACE GENERATION
Synthesizing high-fidelity audio-driven facial video sequences is an important and challenging prob- lem in many applications like digital humans, chatting robots, and virtual video conferences.
For long time, the topic of synthesizing realistic speech video from audio with image in- put is very charming for researchers. In the beginning, people adopted the model-based approach. These methods need establish the relationship between audio semantics and lip movement. Such as phoneme mapping (Fisher 1968 and anatomical actions (Edwards et al. 2016).. Because the suitable for large-scale use.
establishment of this relationship is quite difficult,
In recent years, with the development of GAN technology, the research of constrained talking face generation from speech began to flourish. Kumar al. (2017) attempted to generate key points syn- chronized with audio by using delay LSTM (Graves & Schmidhuber) 2005). They learn mapping between the input audio and the corresponding lip landmarks of Barack Obama. [Suwajanakor et al. (017) also proposed method named "teeth proxy" for improving the visual quality of teeth gen- eration. But these methods can only train specific people who have a lot of video data. Fried et al. (2019) proposed that the video of a single speaker be edited seamlessly by adding deleting phrases in the speech. Unfortunately, to accomplish this task they still need at least one hour's data for each speaker.
2

---

Under review as a conference paper at ICLR 2023
Image 256x256
Encoder
StyleGAN2
Encoder
VGG
Combine
StylGAN2
Moc Encoder
Figure 2: Schematic diagram of end-to-end generative network model.
Subsequently, Chung al. (207))tried to the encoder-decoder CNN model to learn the cor- responding relationship between the original audio and video. Combined with RNN and GAN (Goodfellow et al., 2014), Dalalifar al. (2018) used the LSTM network to create lip markers in audio input and conditional GAN (CGAN) to generate the result face image based on specific set of lip markers. Compared with CGAN, Vougioukas al. (0088) proposed temporary GAN (Saito iet al. 2017) to improve the synthesis quality. However, the above method is still only suitable for human faces synthetic dataset.
Recently, people pay more attention to the synthesis technology of face for any person's conversa- tion. Chen et al. (2018) considered the correlation between speech and lip movement when generat- ing multiple lip images. Researchers use optical flow to better express information between frames. The optical flow not only represents the current shape information, but also represents the previous time information.
Front photos usually have identity and language information. Assuming this, Zhou et al. (2019) used an antagonistic learning method to separate different types of information from an image the process of image generation. This kind of non-entangled representation convenient, that is, audio and video can be used the source of voice information. Therefore, when applying the generated network, we can not only output features, but also express them more clearly.
To find the high-level correlation between audio and video, Zheng al. (2018) proposed tual information approximation to approximate the mutual information between modes. Chen et al. (219)) applied landmarks and action attention generating talking faces. The author further pro- poses temporal consistency of dynamic pixel loss. Wiles et al. Wiles et al.) (2018) proposed a self-monitoring framework called x2face for learning emhedded features and generating target fa- cial motion. As long as the embedded features are learned, it can generate video from any input.
APPROACH
We propose an end-to-end architecture for taking face synthesis as shown figure3. The generative network is encoder-decoder structure. The system consists of generator using StyleGan2 and discriminator which evaluates the generated sequence. We adopted a modified Zhou et al, (2019) as the backbone.
3

---

Under review as a conference paper at ICLR 2023
Figure 3: Images generated using upsampling have "small holes".
3.1 IDENTITY ENCODER
We use Identity Encoder E, to get the identity information from the still frame. A U-net architecture is used with skip connections between the Identity encoder and the Decoder. The features at the five scales in the downsampling process are used as the output of the Encoder E,
3.2 VIDEO ENCODER
When encoding the video sequence, We want to be able obtain lip motion features in the face, blink features of eyes, and motion features of facial contours. We modified FAN (Bulat Tz- imiropoulos (017) to get features. The resulting lip features used establish relationships with audio features, and the eye and contour features are used to influence facial movements.
3.3 AUDIO ENCODER
The Mel-scale frequency cepstral coefficients the audio are used as input, and the features in the audio are obtained using E,,
3.4 DECODER
We first tried upsampling (Zhou et al. 2019) as our generator. However, when evaluating the generated results, the situation shown in the figure appears. There will be watermark-like features such as "small holes" the image. The StyleGan series perform well in the direction of image processing and can perform the style migration task of portraits very well the same time. Using StyleGan2 can solve the problem of "small holes". We spliced together the audio feature, the top identity feature, and the face feature, as the styles of StyleGAN2.Take Take the ID encoder multi-scale features from small to large as the identity style. Also inside the generator, we spliced the features of the up up-sampling part of the generator and the features of the same scale in the down- womn-sampling stage of the identity feature together.
3.5 DISCRIMINATOR
classification discriminator is used to extract the speech features in the ID encoder and make them misclassified, thus removing the speech-related information. A discriminator is also used to optimize the sequence. The generated and ground truth images are sent to a multiscale discriminator D with Np layers.
4

---

Under review as a conference paper at ICLR 2023
axtio
Fale
Grannd
Figure 4: Audio-driven talking face generation
TRAINING
4.1 LIP SYNCHRONIZATION
Regarding the synchronization of the mouth and audio when performing Video Encoder, extract the mouth features of the talking face. The normalized L2 loss is used to calculate the contrast loss between the audio feature and the lip feature, and an unnormalized L2 contrast loss between the audio feature and the lip feature is added on this basis. At the same time, a discriminator Dayne added for the lip feature consistency adversarial of video and speech, and Lgn is used as the loss.
Layne min max
Σ (Er. (log Syyae (Fa) EF, [log(1 D Dsyne F.)))
E. Dayne
4.2 TALKING FACE GENERATION.
Combining identity feature f: video feature f. and audio feature far our system can generate frame using the styleGan2. The newly generated frame be expressed as G(fa, for fi)- The generation results can be expressed as
G( F.,, F.)
where fik) is the identifying feature of the random keth frame, which acts identity guidance.
VGG19 reconstruction loss is used to help capture the face movement. Our overall loss function consists of VGG19 reconstruction loss and temporal GAN loss, where discriminator D seg takes the generated sequence G(Fa, F,, F,) as input. These two terms can be formulated as follows:
N
Lugg
[VGG((()) - F, F.)
Ggan
min max
E a [log D,, (Ia))) + EF.F.A [log(1 D,, F., F.))))
The overall learning objective for the whole system formulated as follows:
C Agan + Augg + AsynoLaync
Besides. the identity prentity-preeerving module of the network is trained on a subset the MS-Celeb-IM dataset. (Guo al. 2016).

---

Under review as a conference paper at ICLR 2023
Refermoc
Cround ToomeTIndh
Pake:
Figure 5: Video-driven talking face generation
5 EXPERIMENTS
Both audio and video can drive the generation of facial animations, and in order to evaluate the respective effects, our evaluation divided into two parts.
5.1 DATASETS
Our model is trained and evaluated on the LRW dataset. Each sample of LRW is 1-second video. It intercepts the head movement of the speaker when they speak a certain word. We also use the CREMA-D dataset as the test set. Each of the actors speaks 12 sentences and contains different emotions and intensities of the speakers.
5.2 METRICS
We use PSNR, SSIM Wang et al. 2004). CarD(Narvekar & Karam 2009 evaluation indicators evaluate the quality of our generated videos. For the LRW dataset, audio-driven and video-driven generated videos are evaluated. CREMA-D mainly evaluates the video-driven part.See Table
Figure 6: An audio-driven fake video and real video evaluation comparison.
Table 1: The quantitative results on LRW and CREMA-D.
Metries LRW(audio)
LRW(video)
CREMA-D(video)
PSNR
29.893
32.406
27.777
SSIM
0.912
0.911
0.534
CPBD
0.110
0.077
0.242
For the generated result, in the video-driven generated video, the eyes will blink like the ground truth, and the face will also turn like the ground truth. Audio-driven generated videos are mostly lip movements.

---

Under review as a conference paper at ICLR 2023
Table 2: Qualitative results of each method on the LRW dataset
Method
PSNR
SSIM
CPBD
SDA
27.100
0.818
0.268
Way2lip
0.862
0.152
DAVS
29.9
0.73
AVTG
20.107
0.810
0.102
Ours(audio)
29.893
0.912
0.110
Ours(video)
32.406
0.911
0.077
We also want to evaluate the effect of our generated video by comparing it with some existing mod- els. Here we choose several classic talking face models for comparison, namely SDA (Vougioukas) ct al. 2020), Wav2lip (Prajwal et al. 2020), DAVS (Zhou et al. 2019), and AVTG (Chen et al, 2019 For SDA, visually, the actual picture is relatively blurry. Wav2lip does a better job of sync- ing the audio, but the facial movements are all static. DAVS has a "hole" situation. Our model outperforms these models on some metries and falls short on others.See Table 2
5.3 BLATION EXPERIMENT
We also did some simple ablation experiments. The first is to compare the use of L₁ and VGG for reconstruction loss. Using VGG produces better textures and details than L1.
Figure 7: From left to right: ground truth, image generated with L₁ image generated with VGG.
At the same time, we also remove synchronization discriminator to see the audio-driven gener- ated image and the movement of the mouth. A small dataset of 39 classes is used here. It can be seen that the synchronization of mouth movements is weakened.
Figure 8: The effect of the synchronization discriminator: generative images and ground truth com- parisons.

---

Under review as a conference paper at ICLR 2023
We implemented the method using Pytorch. The batch size is set to be 8 with le-4 learning rate and trained on 4 RTX3090.
CONCLUSION AND FUTURE WORK
In this work, we use an unsupervised GAN network extract features from facial animations. Therefore, the talking face can be generated from both the voice features and the facial features. Audio-driven video can mainly generate mouth movements, while from video-driven taking face, there are not only mouth movements, but also eye and face turning movements. We evaluate the generated results, which are better than some classic models from the past. At the same time, we also show that using VGG loss can improve the quality of generated images, and the synchronous discriminator also has its role.
At present, in the video generated by the above method, when the head rotates greatly, the boundary effect not very good. Next, we will further study the non-lip-related feature extraction method to make the generated image more realistic.
REFERENCES
Adrian Bulat and Georgios Tzimiropoulos. How far we from solving the 2d & 3d face alignment problem? (and a dataset of 230,000 3d facial landmarks). arXiv preprint arXiv:170..00 1et..07333,2 2017. ICCV 2017.
Lele Chen, Zhiheng Li, Ross K. Maddox, Zhiyao Duan, and Chenliang Xu. Lip movements gener- ation at glance. arXiv preprint arXiv: 10404, 2018.
Lele Chen, Ross K. Maddox, Zhiyao Duan, and Chenliang Xu. Hierarchical cross-modal talking face generationwith dynamic pixel-wise loss. arXiv preprint urXiv:/905, 1905.03820, 2019. The IEEE Conference on Computer Vision and Pattern Recognition (CVPR).
Xi Chen, Yan Duan, Rein Houthooft, John Schulman, Ilya Sutskever, and Pieter Abbeel. Infogan: interpretable representation learning by information maximizing generative adversarial nets. In Proceedings of the 30th International Conference Neural Information Processing Systems, pp- 1180-2188, 2016.
Joon Son Chung, Amir Jamaludin, and Andrew Zisserman. You said that? arXiv preprint 170500770 2017.
Pif Edwards, Chris Landreth, Eugene Fiume, and Karan Singh. Jali: animator-centric viseme model for expressive lip synchronization. ACM Transactions on Graphics, 35(4):1-11, 2016.
Cletus G. Fisher. Confusions among visually perceived consonants. Journal of speech and hearing research, 1144)769-884, 1968.
Ohad Fried, Ayush Tewari, Michael Zollhöfer, Adam Finkelstein, Eli Shechtman, Dan B Goldman, Kyle Genova, Zeyu Jin, Christian Theobalt, and Maneesh Agrawala. Text-based editing of talk- inghead video. ACM Transactions on Graphics (TOG). 38(4):1-14 2019.
Pablo Garrido, Levi Valgaerts, H Sarmadi, Ingmar Steiner, Kiran Varanasi, Patrick Pérez, and Chris- tian Theobalt. Vdub: Modifying face video of actors for plausible visual alignment to dubbed audio track. Computer graphics forum, Wiley Online Library, 34:1 14:199-204, 2015.
Ian Goodfellow, Jean Pouget-Abadie, Mehdi Mirza, Bing Xu, David Warde-Farley, Sherjil Ozair, Aaron Courville, and Yoshua Bengio. Generative adversarial nets. In Advances in Neural Infor- mation Processing Systems 27 (NIPS 2014), pp. 2672-2680, 2014.
Alex Graves and Jürgen Schmidhuber. Framewise phoneme classification with bidirectional Istm and other neural network architectures. Neural Networks. 18(5-6):622-61, 2005.
Kuangxiao Gu, Yuqian Zhou, and Thomas Huang. Finet: Landmark driven fetching and learning network for faithful talking facial animation synthesis. arXiv preprint 19Xv:199109224 2020. AAAI 2020.
8

---

Under review as a conference paper at ICLR 2023
Yandong Guo, Lei Zhang, Yuxiao Hu, Xiaodong He, and Jianfeng Gao. Ms-celeb- Im: A dataset and benchmark for large-scale face recognition. arXiv preprint arXiv:1607.08221, 2016.
Seyed Ali Jalalifar, Hosein Hasani, and Hamid Aghajan. Speech-driven facial reenactment using conditional generative adversarial networks. arXiv preprint arXiv: 103.00180,,, 2018. ECCV 2018.
Amir Jamaludin, Joon Son Chung, and Andrew Zisserman. You said that?: Synthesising talking faces from audio. International Journal of Computer Vision, 127(11-12)
2019.
Rithesh Kumar, Jose Sotelo, Kundan Kumar, Alexandre de Brebisson, and Yoshua Bengio, Oba- manet: Photo-realistic lip-sync from text. arXiv preprint arXiv:1801.01442. 2017.
Christian Ledig. Lucas Theis, Ferenc Huszar, Andrew Cunningham Jose Caballero, Alejandro Acosta, Andrew Aitken, Alykhan Tejani, Johannes Totz, Zehan Wang, and Wenzhe Shi. Photo- realistic single image super-resolution using a generative adversarial network. In 2017 IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pp. 105-114, IEEE, 2017.
Kang Liu and Joern Ostermann. Realistic facial expression synthesis for image-based basee talking head. In 2011 IEEE International Conference on Multimedia and Expo, pp. 1-6. IEEE, 2011.
Mehdi Mirza and Simon Osindero. Conditional generative adversarial nets. arXiv preprint arXiv: 17844 2014.
Niranjan D. Narvekar and Lina J. Karam. A no-reference perceptual image sharpness metrie based on a cumulative probability of blur detection. In 2009 International Workshop on Quality of Multimedia Experience. IEER, 2009.
K R Prajwal, Rudrabha Mukhopadhyay. Vinay P. Namboodiri, and C. V. Jawahar. A lip syne expert is all you need for speech to lip generation in the wild. In Proceedings of the 28th ACM International Conference on Multimedia, 2020.
Alec Radford, Luke Metz, and Soumith Chintala. Unsupervised representation learning with deep convolutional generative adversarial networks. In ICLR2016, 2016.
Masaki Saito, Eiichi Matsumoto, and Shunta Saito. Temporal generative adversarial nets with sin- gular value clipping. In 2017 IEEE International Conference Computer Vision (ICCV), pp- 2849-2858, 2017.
Supasorn Suwajanakom, Steven Maxwell Seitz, and Ira Samnelmacher-Shlizemmeee Synthesizing obama: learning lip sync from audio. ACM Transactions on Graphics (TOG), 36(4):1-13, 2017.
Konstantinos Vougioukas, Stavros Petridis, and Maja Pantic. End-to-end speech-driven facial ani- mation with temporal gans. arXiv preprint arXiv: 000..0005.3333 2018.
Konstantinos Vougioukas, Stavros Petridis, and Maja Pantic. Realistic speech-drive facial anima- tion with gans. International Journal of Computer Vision volume, 128: 188:198-1444 2020.
Tingchun Wang, Mingyu Liu, Andrew Tao, Guilin Liu, Jan Kautz, and Bryan Catanzaro. Few- shot video-to-video synthesis. In Proceedings of the 33rd International Conference on Neural Information Processing Systems, pp. 5013-5024, 2019.
Zhou Wang, A.C. Bovik, H.R. Sheikh, and E.P. Simoncelli. Image quality assessment: from error visibility to structural similarity. IEEE Transactions on Image Processing, 13(4):600-612 2004.
Olivia Wiles, A. Sophia Koepke, and Andrew Zisserman. X2face: A network for controlling face generation by using images, audio, and pose codes. arXiv preprint arXiv 10550, 2018. European Conference on Computer Vision.
Egor Zakharov, Aliaksandra Shysheya, Egor Burkov, and Victor Lempitsky. Few-shot adversar- ial learning of realistic neural talking head models. In Proceedings of the IEEE International Conference on Computer Vision, pp. 9459-9468. IEEE, 2019.
Aihua Zheng, Huaibo Huang, Ran He, and Hao Zhu. High-resolution talking face generation via mutual information approximation. arXiv preprint 18/2 2018.

---

Under review as a conference paper at ICLR 2023
Hang Zhou, Yu Liu, Ziwei Liu, Ping Luo, and Xiaogang Wang. Talking face generation by ad- versarially disentangled audio-visual representation. In Proceedings of the Thirty-This AAAI Conference Artificial Intelligence and Thirty-First Innovative Applications of Artificial Intelli- gence Conference and Ninth AAAI Symposium on Educational Advances in Artificial Intelligence pp. 9299-9306, 2019.
Jun Yan Zhu, Taesung Park, Phillip Isola, and Alexei A. Efros. Unpaired image-to- translation using cycle-consistenta adversarial networks. In 2017 IEEE International Conference on Computer Vision (ICCV), pp. 2242-2251. IEEF, 2017.
10


Reviews:
[{"review_id": "k4XgXfwXXNe", "reviewer": "Reviewer_FwS3", "summary": "", "questions": "", "limitations": "", "rating": 1, "confidence": 5, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "This paper proposed to use StyleGAN2 as the generator to synthesize talking head videos, taking as input the ID feature and the video or audio feature. The consistency of video and audio features is constrained by an adversarial loss. The experiments are conducted on the LRW and CREAM-D datasets.", "summary_of_the_review": "For the above reasons, this paper is entirely below the acceptance threshold.", "strength_and_weaknesses": "Strength:\n1. The reason for using StyleGAN2 as the generator is discussed with an illustration.\n\nWeaknesses:\n1. The language fails to convey the meaning. Please revise the paper with the help of English professionals.\n2. The cited state-of-the-arts (SOTAs) are out-of-date. Please cite and make a comparison with more recent papers, such as [1][2][3][4].\n3. The method is proposed unclearly. For example, why are there two StyleGAN2 in Figure 2? Moreover, the Decoder mentioned in section 3.1 is missing in Figure 2.\n4. The novelty of the proposed method has not been discussed in the paper. \n5. The generated results should be compared with SOTAs visually. \n6. Minor issues:\n    a) Mcc Encoder in Figure 2 should be MFCC Encoder.\n    b) The second line in section 3.4, \"... as shown in the figure appears\" should be Figure 3.\n    c) To be formal, I suggest the author use \"adversarial\" rather than \"antagonistic\" and use \"concatenate\" rather than \"splice\".\n\n[1] Wu, Haozhe, et al. \"Imitating arbitrary talking style for realistic audio-driven talking face synthesis.\" Proceedings of the 29th ACM International Conference on Multimedia. 2021.   \n[2] Zhou, Hang, et al. \"Pose-controllable talking face generation by implicitly modularized audio-visual representation.\" Proceedings of the IEEE/CVF conference on computer vision and pattern recognition. 2021.   \n[3] Ye, Zipeng, et al. \"Audio-driven talking face video generation with dynamic convolution kernels.\" IEEE Transactions on Multimedia (2022).   \n[4] Ji, Xinya, et al. \"EAMM: One-Shot Emotional Talking Face via Audio-Based Emotion-Aware Motion Model.\" SIGGRAPH, 2022.", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "2: Several of the paper’s claims are incorrect or not well-supported.", "technical_novelty_and_significance": "1: The contributions are neither significant nor novel.", "empirical_novelty_and_significance": "1: The contributions are neither significant nor novel."}, "extra_text": {}}, {"review_id": "g8Y8rw2mYIz", "reviewer": "Reviewer_34UQ", "summary": "", "questions": "", "limitations": "", "rating": 1, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "This paper describes an approach for generating video of a talking face from a reference image using either speech as input, or using a video sequence of another person as input.  The architecture uses a separate encoder each for learning a representation of the acoustic features, the lip motion, and the identity.  These learned representations are then concatenated and used as input to StyleGAN2 models for generating the output sequence.", "summary_of_the_review": "The approach taken in the paper seems to be a reasonable one.  Speech-driven talking faces have seen success elsewhere, as has StyleGAN2 for generating content.  However, with so many gaps in the information provided, it is impossible to gauge the quality of the work.  Likewise, the choice of objective metrics makes it difficult to understand how effective is the model at producing visual speech as the measures relate more to image quality.", "strength_and_weaknesses": "+The work compares against a number of baseline approaches.\n\n+StyleGAN models have shown to work well for multiple problems.\n\n-The paper lacks a lot of detail that would be required to reproduce the method.  Information about the architectures, the feature processing, and so on is all missing.\n\n-It is difficult to evaluate the quality of the output from the way the results are presented.  There are some objective measures included, but these relate more to the image quality and not the visual speech content captured in the video.\n", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "1: The main claims of the paper are incorrect or not at all supported by theory or empirical results.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "1: The contributions are neither significant nor novel."}, "extra_text": {}}, {"review_id": "VxvLJAGDatW", "reviewer": "Reviewer_MS16", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "This paper addresses the problem of generating a talking face animation video driven by audio (or video), conditioned on a target face appearance. The proposed approach consists of a StyleGan2-like generator conditioned on output from identity, video and audio encoders. ", "summary_of_the_review": "Recommend to Reject.", "strength_and_weaknesses": "Pros:\n- An interesting approach for generating talking faces\nCons:\n- The paper requires proof reading\n- Limited novelty: most of the components of the proposed solution have already been published\n- Limited evaluation and ablation analysis\n\n\n", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "2: Several of the paper’s claims are incorrect or not well-supported.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}, {"review_id": "P2Qltn_gqL", "reviewer": "Reviewer_gYGv", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "This work is for talking animation video generation. The authors propose a generative pipeline via different feature extraction like ID, Lip and audio. Also, the authors did some experiments and ablation study for evaluation.", "summary_of_the_review": "This paper presents a fact that using combined audio features with facial feature could improve facial animation. \nHowever, there is not any novel design or insight about this statement. \nUsing StyleGAN2 to generate image is not a contribution. \nStyleGAN2 might not be applicable for video generation due to lack of speech details.", "strength_and_weaknesses": "The idea of using combine audio and image feature for generation task is interesting, ", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "2: Several of the paper’s claims are incorrect or not well-supported.", "technical_novelty_and_significance": "1: The contributions are neither significant nor novel.", "empirical_novelty_and_significance": "Not applicable"}, "extra_text": {}}]

Author rebuttal:

Decision: Reject