GeneFace: Generalized and High-Fidelity Audio-Driven 3D Talking Face Synthesis
Authors: Zhenhui Ye
Ziyue Jiang
Yi Ren
Jinglin Liu
Jinzheng He
Zhou Zhao
Venue: ICLR 2023 poster (2023)
Keywords: Talking Face Generation
Neural Radiance Field

Abstract: Generating photo-realistic video portraits with arbitrary speech audio is a crucial problem in film-making and virtual reality. Recently, several works explore the usage of neural radiance field (NeRF) in this task to improve 3D realness and image fidelity. However, the generalizability of previous NeRF-based methods is limited by the small scale of training data. In this work, we propose GeneFace, a generalized and high-fidelity NeRF-based talking face generation method, which can generate natural results corresponding to various out-of-domain audio. Specifically, we learn a variational motion generator on a large lip-reading corpus, and introduce a domain adaptative post-net to calibrate the result. Moreover, we learn a NeRF-based renderer conditioned on the predicted motion. A head-aware torso-NeRF is proposed to eliminate the head-torso separation problem. Extensive experiments show that our method achieves more generalized and high-fidelity talking face generation compared to previous methods. Video samples and source code are available at https://geneface.github.io .

Published as a conference paper at ICLR 2023
GENEFACE: GENERALIZED AND HIGH-FIDELITY BUDIO-DRIVER 3D TALKING FACE SYNTHESIS
Zhenhui ;e!; Ziyue Jiang'; Yi Ren", Jinglin Liu', JinZheng He', Zhou Zhao it
School of Computer Science and Technology, Zhejiang University {zhenhuiye, 1angz i yue, jinglinliu, jinzhenghe, zhaozhou ju. edu. cn
"ByteDance
ren. yiibytedance.com com
ABSTRACT
Generating photo-realistic video portraits with arbitrary speech audio is crucial problem in film-making and virtual reality. Recently, several works explore the usage of neural radiance field (NeRF) in this task to improve 3D realness and im- age fidelity. However, the generalizability out-of-domain audio is limited by the small scale of training data. In this work, we propose GeneFace, a generalized and high-fidelity NeRF-based talking face generation method, which can generate natural results corresponding to various out-of-domain audio. Specifically, we learn a variational motion generator on a large lip-reading corpus, and introduce domain adaptative post-net to calibrate the result. Moreover, we learn a NeRF-based renderer conditioned on the pre- dicted facial motion. A head-aware torso-NeRF is proposed to eliminate the head- torso separation problem. Extensive experiments show that our method achieves more generalized and high-fidelity talking face generation compared previous methods
previous NeRF-based methods
INTRODUCTION
Audio-driven face video synthesis is an important and challenging problem with several applications such as digital humans, virtual reality (VR), and online meetings. Over the past few years, the com- munity has exploited Generative Adversarial Networks (GAN) as the neural renderer and promoted the frontier from only predicting the lip movement (Prajwal et al. 2020 Chen et al. (019) to gener- ating the whole face (Zhou al. 2021 [u al.) 2021). However, GAN-based renderers suffer from several limitations such as unstable training, mode collapse, difficulty in modelling delicate details (Suwajanakorn et al., 2017 Thies et al. 22020, and fixed static head pose (Pham et al., 2017 Taylor et 201 Cudeiro al. 2019). Recently, Neural Radiance Field (NeRF) (Mildenhall et al. 2020) has been explored in talking face generation. Compared with GAN-based rendering techniques, NeRF renderers could preserve more details and provide better 3D naturalness since it models continuous 3D scene in the hidden space.
Recent NeRF-based works (Guo al. 2021 Liu et al. 2022 Yao al. 2022 manage learn an end-to-end audio-driven talking face system with only few-minutes-long video. However, the current end-to-end framework is faced with two challenges. 1) The first challenge is the weak gen- cralizability due to the small scale of training data, which only consists of about thousands-many audio-image pairs. This deficiency of training data makes the trained model not robust to out-of- domain (OOD) audio in many applications (such as cross tross-lingual (Guo et al. [222 Liu et al. 2022) or singing voice). 2) The second challenge is the so-called "mean face" problem. Note that the au- dio to its corresponding facial motion is one-to- one-to-many mapping, which means the same audio input may have several correct motion patterns. Learning such mapping with regression
model
Authors contribute equally this work.
Corresponding author
Video samples and source code available httph::/ggnefaace..c.p. www.aaa................. thub. i

---

Published as a conference paper at ICLR 2023
leads over-smoothing and blurry results (Ren et al. 2021); specifically, for some complicated audio with several potential outputs. it tends to generate an image with half-opened and blurry mouth, which leads to unsatisfying image quality and bad lip-synchronization. To summarize, the current NeRF-bbased methods are challenged with the weak generalizability problem due to the lack of audio-to-motion training data and the "mean face" results due the one-to-many mapping.
In this work, we develop a talking face generation system called GeneFace to address these two challenges. To handle the weak generalizability problem, we devise an audio-to-motion model to predict the 3D facial landmark given the input audio. We utilize hundreds of hours of audio-motion pairs from a large-scale lip reading dataset (Afouras et al., 2018) to learn a robust mapping. As model, we adopt a variational
for the "mean face" problem, instead of using the regression- auto-encoder (VAE) with a flow-based prior as the architecture of the audio-to-motion model, which helps generate accurate and expressive facial motions. However, due to the domain shift between the generated landmarks (in the multi-speaker domain) and the training set of NeRF (in the target person domain), we found that the NeRF-based renderer fails to generate high-fidelity frames given the predicted landmarks. Therefore, a domain adaptation process is proposed to rig the predicted landmarks into the target person's ' distribution. To summarize, system consists of three stages:
Audio-to-motion, We present a variational motion generator to generate accurate and expressive facial landmark given the input audio.
2 Motion domain adaptation. To overcome the domain shift, we propose semi-supervised ad- versarial training pipeline to train a domain adaptative post-net, which refines the predicted 3D landmark from the multi-speaker domain into the target person domain.
3 Motion-to-image- We design a NeRF-based renderer to render high-fidelity frames conditioned on the predicted 3D landmark.
The main contributions of this paper are summarized as follows:
We present stage framework that enables the NeRF-based talking face system to enjoy the large-scale lip-reading corpus and achieve high generalizability to various OOD audio. We propose an adversarial domain adaptation pipeline to bridge the domain gap between the large corpus and the target person video.
We are the first work that analyzes the "mean face" problem induced by the one-to- tonyy audio- to-motion mapping in the talking face generation task. To handle this problem, we design varia- tional motion generator to generate accurate facial landmarks with rich details and expressiveness.
Experiments show that our GeneFace outperforms other tate-of-thhear GAN- and NeRF- based baselines from the perspective of objective and subjective metrics.
2 RELATED WORK
Our approach is a 3D talking face system that utilizes a generative model to predict the 3DMM- based motion representation given the driving audio and employs neural radiance field to render the corresponding images of a human head. It is related to recent approaches to audio-driven talking head generation and scene representation networks for the human portrait.
Audio-driven Talking Head Generation With the progress of audio synthesis technology (Ye
al.] 2022; Huang al. 2022a blc) and generative models(Zhang et al., 2022 2023), generating
Talking faces in line with input audio has attracted much attention of the computer vision community.
Earlier works focus on synthesizing the lip motions on static facial image (Jamaludin et al. 2019
Tony Ezzat & Poggio 2002 Vougioukas et al. 2220)). Then the frontier is promoted to synthesize the
full head (Yu et al. 2020) Zhou et al. [219 2020). However, free pose control not feasible in these methods due to the lack of 3D modeling. With the development of 3D face reconstruction techniques
(Deng et al., 2019), many works explore extracting 3D Morphable Model (3DMM) (Paysan et al.
2009 from the monocular video to represent the facial movement (Tero Karras & Lehtinen 2017 Yi et al. 2020) in the talking face system, which is named as model-based methods. With 3DMM, affine model of facial expression and identity
coarse 3D face mesh M can be represented
code:
M M Bigi Beape
2

---

Published as a conference paper at ICLR 2023
b Draing Audio
HuBLRT Teatures
x-N(0.1)
A based
Prior
Erhanced
Latert
Wavefett like Decoder
Generated
Gone
BN
RoLLU
Refined Landmark
Rendered
Head-Nee Neff
Torso-Nuff NARF
Variational Mation Generator
Domain Adaptative Post-net
3DMM NeRF Renderer
Frame
Figure I: The inference process of GeneFace. BN denotes batch normalization.
where M is the average face shape; Bud and Bexp are the PCA bases of identity and expression; i and e are known as identity and expression codes.
By modeling the 3D geometry with 3DMM, the model-based works manage manipulate the head pose and facial movement. However, 3DMM could only define a coarse 3D mesh of the human head, and delicate details (such as hair, wrinkle, teeth, etc.) are ignored. It raises challenges for GAN-based methods to obtain realistic results. Recent advances in neural rendering have created prospect: instead of refining the geometry of 3DMM adding more personalized attributes as auxiliary conditions for GAN-based renderers, we could leave these delicate details to be modeled implicitly by the hidden space of the neural radiance field.
Neural Radiance Field for Rendering Face The recent proposed neural radiance field (NeRF)
(Mildenhall et al., 2020) Kellnhofer et al. 2021 Pumarola et al. 2021 Sitzzann et al., 2019) has attracted much attention in the human portrait rendering field since could render high-fidelity
images with rich details such as hair and wrinkles. For instance, Siizmaann et al.) (2019) presents a compositional NeRF for generating each part of the upper body. Gafni et (2021) and Pumarola cit--eepresiioncoonitioooo
et al. (202T) propose
dynamic NeRFs for modeling the dynamics of human face. Chan al.) (2022) proposes hybrid aaplcit-implics tri-plane representation to achieve fast and geometry- aware human face rendering. HHng et al. (2022) proposes a real-time
NeRF-based parametric head model.
Several works have also applied NeRF in the audio-driven talking face generation task. Zhang et al. (202T) devises an implicit pose code to modularize audio-visual representations. [uuo al. (2021) first presents an end-to-end audio-driven NeRF to generate face images conditioned Deepspeech
(Hannun al., 2014) audio features. Recently, Liu et al. (2022) proposed a semantic-aware dynamic ray sampling module to improve the sample efficiency and design a torso deformation module to stabilize the large-scale non-rigid torso motions. Yao et al. (2022) introduces two disentangled representations (eye and mouth) provide improved conditions for NeRF. To achieve few-shot training, Shen et al. (2022) conditions the face radiance field on 2D reference images to learn the face prior, thus greatly reducing the required data scale (tens of seconds of video) and improve the convergence speed (about 40k iterations). However, all of the previous NeRF-based work focuses on better image quality reducing the training cost, while the generalizability out-of-domain audio is relatively an oversight.
Our GeneFace could be regarded as bridging the advantages of the aforementioned two types of works. Compared with previous 3DMM- methods, our work could enjoy good 3D naturalness and high image quality brought by the NeRF-based renderer. Compared with previous end-to-end NeRe- methods, we improve the generalizabity to out-of-domain audio via introducing a gen- erative audio-to- motionn model trained on a large lip reading corpus.
3 GENEFACE
In this section, we introduce proposed GeneFace. As shown in Fig. I GeneFace is composed of three parts: 1) variational motion generator that transforms HuBERT features (Hsu et al. 2T.0 into 3D facial landmarks; 2) a post-net to refine the generated motion into the target person domain; 3) a NeRF- renderer to synthesize figh-fideiity frames. We describe the designs and the training process of these three parts in detail in the following subsections.
3

---

Published as a conference paper at ICLR 2023
HUBERT Festures
Audio
Pretrained SyncNet
Large Video Corpus
Flow based
Deep 30 Recon
MaveNett-1Be
Encoder
Decoder
Generated
Landmark
3DMM Landmmark
Variational Motion Generator
Figure 2: The structure of variational motion generator. Dashed arrows means the process only performed during training: and only the dashed rectangle part is used during inference.
3.1 VARIATIONAL MOTION GENERATOR
To achieve expressive and diverse 3D head motion generation, we introduce a variational auto- encoder (VAE) to perform a generative and expressive audio-to-motion transform, namely the vari- ational motion generator, as shown in Fig.
Audio and motion representation To better extract the semantic information, we utilize Hu- BERT, state-of-the-art ASR model, to obtain audio features from the input wave and use it as the condition of the variational motion generator. As for the motion representation, to represent detailed facial movement in Euclidean space, select 68 key points from the reconstructed 3D head mesh and use their position as the action representations. Specifically,
(2) where LM3D E 5 M and M the 3DMM mesh and mean mesh defined in Equation (1), the index of the key landmark the mesh. In this paper, we name this action representation 3D landmarks for abbreviation.
LMsD = {(M - M E f}.
Dilated convolutional encoder and decoder Inspired by WaveNet, to better extract features from the audio sequence and construct long-term temporal relationships in the output sample, we design the encoder and decoder as fully convolutional networks where the convolutional layers have incre- mentally increased dilation factors that allow its receptive field grow exponentially with depth. In contrast to previous works, which typically divide the input audio sequence into sliding windows obtain smooth result, we manage to synthesize the whole sequence of arbitrary length within single forward. To further improve the temporal stability of the predicted landmark sequence, Gaussian filter is performed eliminate tiny fluctuations in the result.
We also notice that the gaussian prior of vanilla VAE limits the performance of our 3D landmark sequence generation process from two prospectives: 1) the datapoint of each time index is independent of each other, which induces noise to the sequence generation task where there a solid temporal correlation among frames. 2) optimizing VAE prior push the posterior distribution towards the mean, limiting diversity and hurting the generative power. To this end, following Ren et al. 2021), we utilize normalizing flow to provide a complex and time-related distribution as the prior distribution of the VAE. Please refer to Appendix A. A.
Flow-based Prior
for more details.
Training Process Due to the introduction of prior flow, the closed-form ELBO is not feasible, hence we the Monte-Carlo ELBO loss (Ren et al.] 2021) train the VAE model. Besides, inspired by Prajwal et al. 2020), we independently train a sync-expert Deyne that measures the possibility that the input audio and 3D landmarks in-sync, whose training process can be found in Appendix A.2. The trained sync- syne-expert is then utilized to guide the training of VAE. To summarize, the training loss of variational motion generator (VG) follows:
Eva(áh 0, - (a)()) [log Po (4|2, a)) (I-(gg ()) (i ( [log D (i))e(() (3)
where Φ, 0,e denote the model parameters of the encoder. decoder and the prior, respectively. c denotes the condition features of VAE. The ground truth and predicted 3D landmarks are represented by 1 and i, respectively.
4

---

Published as a conference paper at ICLR 2023
Presrained Synehet
Peatures
BEAA
Large Video Corpus
Variational Motion Generator
Domair Adaptive Post-net
Rahnad jastrek
MLP- base- Dnc.
Target Person Video
soWill Lundmark
Figure The training process of Domain Adaptative Post-net.
3.2 DOMAIN ADAPTIVE POST-NET
As we train the variational motion generator on a large multi-speaker dataset, the model can general- ize well with various audio inputs. However, the scale of the target person video is relatively tiny (about 4-5 minutes) compared with the multi-speaker lip reading dataset (about hundreds of hours), there exists domain shift between the predicted 3D landmarks and the target person domain. As consequence, the NeRF-based renderer cannot generalize well with the predicted landmark, which results in blurry unrealistic rendered images. To this end, A naive solution is fine-tune the variational generator in the target person dataset. The challenge is that we generally only have short personalized video, and the generalizability of the model may be lost after the fine-tuning.
Under such circumstances,, we design a semi-supervised adversarial training pipeline to perform domain adaptation. be specific, learn a post-net refine the VAE-predicted 3D landmarks into the personalized domain. We consider two requirements for this mapping: 1) it should preserve the temporal consistency and lip-synchronization of the input sequence; 2) it should correctly map each frame into the target person's s domain. To fulfill the first point, we utilize ID CNN as the structure of post-net and adopt the sync-cxpert to supervise the lip-synchromizattion; for the second point, we jointly train MLP-structured frame-level discriminator that measures the identity similarity of each landmark frame to the target person. The detailed structure of the post-net and discriminator can be found in Appendix A.3
The training process of post-net is shown in Fig During training, the MLP discriminator tries to distinguish between the ground truth landmark extracted from the target person's video and the refined samples (l) generated from the large-scale dataset. We use the LSGAN loss to update the discriminator:
Training Process
(D( = (Ds 0) (e=sr))((((()))) 1)
(4)
where and 8 are the parameters of the post-net PN and discriminator D. l' is the ground truth 3DMM landmark of the target person dataset, and i the 3D landmarks refined by the post-net.
As for the training of post-net, the post-net competes with the discriminator while being guided by the pre-trained sync-expert to maintain lip synchronization. Besides, utilize the target person dataset to provide a weak supervised signal to help the adversarial training. Specifically, we extract the audio c' of the target person video for VAE to predict the landmarks !' ~ po (4) c') and en- courage the refined landmarks (N)( to approximate the ground truth expression l'. Finally, the training loss post-net is:
LPx(w) (D)
1)31 E ( (i)
(5)
+E +E ((PN))))))))))))))))))))))))))))
3.3 NERF-BASER BASED RENDERER
We obtain robust and diverse audio-to-motion mapping through the variational motion generator and pest-net. Next, we design base-based renderer to render high-fidelin frames conditioned on the predicted 3D landmarks.
5

---

Published as a conference paper at ICLR 2023
Target Person Video
Head Pose
DD Recon
NeRl
Sandand
Landmark
3DMM Landmart
3DMM Ne5t Bunderer...
Figure 4: The training process of NeRF-based renderer.
3D landmark--onnittooned NeRF Inspired by Guo et al. (202T), we present conditional NeRF to represent the dynamic talking head. Apart from viewing direction and 3D location æ, the 3D landmarks / will act as the condition to manipulate the color and geometry of the implicitly represented head. Specifically, the implicit function F can be formulated as follows:
F9 : d,)) (c)
(6)
where e and denote the color and density the radiance field. To improve the continuity between adjacent frames, we use the 3D landmarks from the three neighboring frames to represent the facial shape, i.e., e 3x3200 We notice that some facial landmarks only change small range, which numerically raises challenges for NeRF to learn the high-frequency image details. Therefore, we normalize the input 3D landmarks point-wisely, which is necessary to achieve better visual quality.
Following the setting of volume rendering, to render each pixel, we emit a camera ray r(() L-d in the radiance field, with camera center o, viewing direction d. The final color C is calculated by aggregating the color along the ray:
((r,4:)) = I' co(((()) (o((()()).
co(r(t),1, T(t)
where t,, and t, is the near bound and far bound of ray r; cy and dg are the output of the implicit function Fo. T(1) is the accumulated transmittance along the ray from i, to I, which is defined as:
T(t) exp(- S." ao(r(T))dr)
Head-aware Torso-NeRF To better model the head and torso movement, we train two NeRFs to render the head and torso parts, respectively. As shown in Fig. 4 we first train a head-NeRF to render the head part, then train a torso-NeRF to render the torso part with the rendering image the head-NeRF as background. Following Guo et al.) (022), we assume the torso part is in canonical space and provide the head pose h to torso-NeRF as signal to infer the torso movement. The torso-NeRF implicitly learns to expect the location of the rendered head, then rigid the torso from canonical space to render a natural result.
However, this cooperation between head-NeRF and torso-NeRF is fragile since the torso-NeRF cannot observe the head-NeRF's actual output. Consequently, several recent works report that the torso-NeRF produces head-torso separation artifacts (Liu al. 2022 Yao al. 2022) when the head pose is relatively large. Based on the analysis above, we propose to provide the torso-NeRF with a perception of the rendering result of the head-NeRF. Specifically, we the output color Chead of the head-NeRF as pixel-wise condition of the torso-NeRF. The torso's implicit function Fiorao expressed as:
Fiorso: (I, Chead: do, II,1) - (c,
where do is view direction in the canonical space, II 3x34 is the head pose that composed of rotation matrix and a transform vector.
Training Process We extract 3D landmarks from the video frames and use these landmark-iman image pairs to train our NeRF-based renderer. The optimization target of head-NeRF and torso-NeRF is reduce the photo-metric reconstruction error between rendered and ground-truth images. Specifi- cally, the loss function can be formulated as:
LNERF (0) = Σ [[Co(r:)) -
(10)
PER

---

Published as a conference paper at ICLR 2023
where R is the set of camera rays, Cg is the color of the ground image.
4 EXPERIMENTS
4.1 DATASET PREPARATION AND PREPROCESSING
Dataset preparation. Our method aims to synthesize high-fidelity talking face images with great generalizability to out-domain audio. To learn robust audio-to-motion mapping, a large-scale lip- reading corpus is needed. Hence we use LRS3-TED (Afouras et al. 2018) to train our variational generator and post-net Additionally, a certain person's 's speaking video of a few minutes in length with an audio track is needed to learn NeRF-based person portrait renderer. To be specific, in order to compare with the state-of-the-ar methods, we utilize the data set of et al. (2021) and et al. 2021), which consist of 5 videos of an average length of 6,000 frames in 25 fps.
Data preprocessing. As for the audio track, we downsample the speech wave into the sampling rate of 16000 and process it with a pretrained HuBERT model. For the video frames of LRS3 and the target person videos, we resample them into 25 fps and use the toolkit proposed by Deng et al. (2119) to extract the head pose and 3D landmarks. As for the target person videos, they are cropped into 51 112x512 12 and each frame is processed with the help of an automatic parsing method (Lee et al.,
2020 for segmenting the head and torso part and extracting a clean background.
4.2 EXPERIMENTAL SETTINGS
Comparison baselines. We compare our GeneFace with several remarkable works: 1) Wav2Lip (Prajaal et al. 2020), which pretrain a sync-expert to improve the lip-synchronization performance; 2) MakelrTalk (Zhou et al. 2020), which also utilize 3D landmark the action representation; 3) PC-AVS (Zhou et 1022). which first modularize the audio-visual representation. 4) LiveSpeech- Portriat (Lu et al. 2021b. which achieves photorealistic results at over 30fps; 5) AD-NeRF (Guo et al. 2021). which first utilize NeRF to achieve talking head generation. For Wav2Lip, PC-AVS, and Makelr'Talk, the LRS3-TED dataset is used train the model, and a reference clip of the target person video is used during the inference stage; for LSP, both of LRS3-TED dataset and the target person video is used to train the model; for the NeRF-based method, AD-NeRF, only the target person video is used to train an end-to-end audio-to-image renderer.
Implementation Details. We train the GeneFace on NVIDIA RTX 3090 GPU, and the detailed training hyper-parameters of the variational generator, post-net, and NeRe-baseed are listed in Ap- pendix B For variational generator and post-net, it takes about 40k and 12k steps to converge (about 12 hours). For the NeRF-based renderer, we train each model for 800k iterations (400k for head and 400k for the torso, respectively), which takes about 72 hours.
4.3 QUANTITATIVE EVALUATION
Evaluation Metrics We employ the FID score (Heusel et al. 2017 to measure image quality. We utilize the landmark distance (LMD) (Chen al. 2018) and synenet confidence score (Prajwal al. 2020) to evaluate lip synchronization. Furthermore, to evaluate the generalizability. we additionally test all methods with set of out-of-domain (OOD) audio, which consists of cross-lingual, cross- gender, and singing voice audios.
The results are shown in Table We have the following observations. (1) Our GeneFace achieves good iipsynchrronization with high generalizability. Since Wav2Lip is jointly trained with SyncNet, it achieves the highest sync score that is higher than the ground truth video. Our method performs best in LMD and achieves better sync score than other baselines. When tested with out-of-domain audios, while the sync-score of person-specific methods (LSP and AD- NeRF) significantly drops, GeneFace maintains good performance. (2) Our GeneFace achieves the best visual quality. We observe that one-shot methods (Wav2Lip, MakeltTalk, and PC-AVS) perform
Evaluation Results
'we select samples of good quality in the IRS3-TED dataset, the selected subset contains 19,775 short. videos from 3,231 speakers and is about 120 hours-long.

---

Published as a conference paper at ICLR 2023
Method
FID 4
LMD4
Syne
FID(OOD) J Sync(OOD)
Wav2Lip
71.40
3.988
9.212
68,05
9.645
MakeirTalk
57.96
4.848
4.981
53.33
4.933
PC-AVS
96.81
5.812
6.239
98.31
6.156
LSP
29.30
4,589
6.119
35.21
4.320
AD-NeRF
27.52
4.199
4.894
35.69
4.225
Ground Truth
0.00
0,000
8.733
N/A
N/A
GeneFace (ours)
22.88
3,933
6.987
27.38
6.212
Table 1: Quantitative evaluation with different methods. Best results are in bold.
Audio
/w/
Tf
Juu!
20
/um/
20
/um/
GeneFace
Figure 5: The comparison of generated key frame results. We show the phonetic symbol of the key frame and the corresponding synthesized talking heads of AD-NeRF and GeneFaae. We mark the head-torso separation artifact, blurry mouth, un-syne results with brown, blue, and red arrow, respectively. Please zoom in for better visualization. More qualitative comparisons can be found in demo video.
poorly on FID due to low image fidelity. Since we use 3D landmarks as the condition of the NeRF renderer, it address the mean face problem and leads to better lip syncronization and visual quality than AD-NeRF.
4.4 QUALITATIVE EVALUATION
To compare the generated results of each method, show the keyframes of two clips in Fig.5) Due to space limitations, we only compare our GeneFace with AD-NeRF here and provide full results with all baselines in Appendix C.I We observe that although both methods manage to generate high-fidelity results, GeneFace solves several problems that AD-NeRF has: 1) head-torso separation (brown arrow) due to the separate generation pipeline of head and torso part; 2) blurry mouth images due to the one-to-many audio-to- mapping; 3) unsynchronized lip due to the weak generalizability.
User Study We conduct user studies test the quality of audio-driven portraits. Specifically, sample audio clips from English, Chinese, and German for all methods to generate the videos, and then involve 20 attendees for studies. We adopt the Mean Opinion score (MOS) rating protocol for evaluation, which is sealed from 1 to The attendees are required to rate the videos based on three aspects: (1) lip-syne accuracy; (2) video realness; (3) image quality.
We compute the average score for each method, and the results are shown in Table We have the following observations: 1) Our GeneFace achieves comparatively high lip-syne accuracy with Wav2Lip (Prajwal et al.) (2000) since both them learn a generalized audio-to-motion mapping on large dataset with guidance from a sync-expert. As for the video realness and image quality, the Person-specific methods (LSP, AD-NeRF, and GeneFace) outperform one-shot methods (Wav2Lip, MakeltTalk, and PC-AVS). Although LSP has slightly better image quality than GeneFace, our method achieves the highest video realness and lip-syne accuracy score.
4.5
ABLATION STUDY
In this section, we perform ablation study to prove the necessity of each component in GeneFace.
8

---

Published as a conference paper at ICLR 2023
Methods
Wav2Lip
MakelfTalk
PC-AVS
LSP
AD-NeRF
GeneFace (ours)
Lip-sync Accuracy
3.77:10.25
2.86110.3
3.111:0.30
3,65-0.20
3.05.10.26
3.82
Image Quality
3.38
2.84+0.20
2.73+:0.25
3.92-0.13
3.44 3.31
3.87=0.16
Video Realness
3.274:0.6
2.32+0.30
2.46:t.228
3.62=0.24
3.87=00166
Table 2: User study with different methods. The error bars are 95% confidence interval.
Setting
FIDI
LMD|
Synct
FID(OOD),
Sync(OOD)
GeneFace
22.88
3.933
6.987
27.38
6.212
w/o prior flow
24.71
4.063
6,404
29.55
5.831
w/o syne-expert
24.02
4.151
5.972
30.77
5.549
w/o post-net
30.26
4.532
5.085
35,58
5.248
w. fine-tune
25.75
4.227
6.875
29.30
5.966
w/o head-aware
26.34
3.948
6.899
28.89
6.167
Table 3: Ablation study results. The ablation settings are described in Sec. 4.5
Varaiational motion generator We test two settings on the variational motion generator: (1) w/o prior flow, where we replace the flow-based prior with a gaussian prior. The results are shown in Table (line 2), where the Sync score drops by a relatively large margin. This observation suggests that the temporal enhanced latent variable contributes to the stability of the predicted landmark se- quence. (2) w/o sync-expert (line 3), where the variational motion generator is no longer supervised by pretrained sync-expert. We observe that it leads to a significant degradation in Sync score.
Domain adaptative post-net In the setting w/o post-net, we remove the domain adaptative post- net, the results shown in Table 3 (line 4). It can be seen that directly using the 3D landmarks predicted by the variational motion generator leads to a significant performance drop in FID and Sync scores. To further investigate the efficacy of post-net, we utilize T-SNE visualize the land- marks of different domains in Fig. I0 The visualization results prove that there exists significant domain gap between the LRS3 dataset and the target person video, and our post-net successfully rigs the predicted landmarks from the LRS3 domain into the target person domain. We also try to replace the post-net with directly fine-tuning on the target person video (line 5), although it achieves competitive sync score in-domain audios, its performance in OOD audio is worse.
Head-aware torSo-NeRF In the w/o head-ware setting, we remove head image condition of the (line 6). Due to the unawareness of the head's 's location,
torso-NeRF. The results shown in Table (line the head-torso separation occurs occasionally, which results a drop in the FID score.
5 CONCLUSION
In this paper, we propose GeneFace for talking face generation, which aims to solve the weak gen- eralizability and mean face problem faced by previous besed methods. A variational motion generator is proposed to construct a generic audio-to-motion mapping based on a large corpus. We then introduce domain adaptative post-net with an adversarial training pipeline rig the predicted motion representation into the target person domain. Moreover, head-aware torso-NeRF is present address the head-torso separation issue. Extensive experiments show that our method achieves more generalized and high-fidelity talking face generation compared to previous methods. Due to space limitations, we discuss the limitations and future work in Appendix D
ACKNOWLEDGMENT
This work was supported in part by the National Natural Science Foundation of China Grant No. 62222211, Zhejiang Electric Power Scienne and Technology Project Yiwise.
and

---

Published as a conference paper at ICLR 2023
ETHICS STATEMENT
GeneFace improves the lip synchronization and expressiveness of the synthesized talking head video. With the development of talking face generation techniques, much easier for people to synthesize fake videos of arbitrary persons. In most situations, they utilize these techniques to facilitate the movie and entertainment industry and reduce the bandwidth of video streaming by sending audio signals only. However, the talking face generation techniques can be misused. As it is more difficult for people to distinguish synthesized videos, the algorithm may utilized to spread fake information obtain illegal profits. Potential solutions like digital face forensics methods to detect deepfakes must be considered. We also plan to include restrictions in the open-source license of the GeneFace project prevent "deepfake" -related abuse. We hope the public is aware of the potential risks of misusing new techniques.
REFERENCES
Triantafyllos Afouras, Joon Son Chung, and Andrew Zisserman. Lrs3-ted: large-scale dataset for visual speech recognition. arXiv preprint 18Xiv:180..000 00096, 2018.
Eric R. Chan, Connor Z. Lin, Matthew A. Chan, Koki Nagano, Boxiao Pan, Shalini De Mello, Orazio Gallo, Leonidas Guibas, Jonathan Tremblay, Sameh Khamis, Tero Karras, and Gordon Wetzstein. Efficient geometry-aware 3d generative adversarial networks. In CVPR, pp. 16123- 16133, June 2022.
Lele Chen, Zhiheng Li, Ross K Maddox, Zhiyao Duan, and Chenliang Xu. Lip movements genera- tion glance. ECCV, pp. 520-535, 2018
Lele Chen, Ross K Maddox, Zhiyao Duan, and Chenliang Xu. Hierarchical cross-modal talking face generation with dynamic pixel-wise loss. In CVPR, pp. 7832-7841, 2019.
Daniel Cudeiro, Timo Bolkart, Cassidy Laidlaw, Anurag Ranjan, and Michael J Black. Capture, learning, and synthesis of 3d speaking styles. In CVPR, pp. 10101-10111, 2019.
Yu Deng, Jiaolong Yang, Sicheng Xu, Dong Chen, Yunde Jia, and Xin Tong. Accurate 3d face reconstruction with weakly-supervise learning: From single image image set. CVPRW, 2019.
Guy Gafni, Justus Thies, Michael Zollhofer, and Matthias Niessner. Dynamic neural radiance fields for monocular 4d facial avatar reconstruction. In CVPR, pp. 8649-8658, June 2021.
Yudong Guo, Keyu Chen, Sen Liang, Yong-Jin Liu, Hujun Bao, and Juyong Zhang. Ad-nerf: Audio driven neural radiance fields for talking head synthesis. In ICCV, pp. 5784-5794, 2021.
Awni Hannun, Carl Case, Jared Casper, Bryan Catanzaro, Greg Diamos, Erich Elsen, Ryan Prenger, Sanjeev Satheesh, Shubho Sengupta, Adam Coates, et al. Deep speech: Scaling up end-to-end speech recognition. arXiv preprint 14Xi::4412.5567 2014.
Martin Heusel, Hubert Ramsauer, Thomas Unterthiner, Bernhard Nessler, and Sepp Hochreiter. Gans trained by a two time-scale update rule converge local nash equilibrium. Advances in neural information processing systems, 30, 2017.
Yang Hong. Bo Peng, Haiyao Xiao, Ligang Liu, and Juyong Zhang. Headnerf: A real-time nerf- based parametric head model. In CVPR, pp. 20374-20384, June 2022.
Wei-Ning Hsu, Benjamin Bolte, Yao-Hung Hubert Tsai, Kushal Lakhotia, Ruslan Salakhutdinov, and Abdelrahman Mohamed. Hubert: Self-supervised speech representation learning by masked prediction of hidden units. JEEE/ACM Transactions Audio, Speech, and Language Processing, 29:3451-3460, 2021.
Rongjie Huang, Max WY Lam, Jun Wang, Dan Su, Dong Yu, Yi Ren, and Zhou Zhao. Fast- diff: A fast conditional diffusion model for high-quality speech synthesis. arXiv preprint arXiv:2204.0992: 09934, 2022a.
10

---

Published as a conference paper at ICLR 2023
Rongjie Huang, Yi Ren, Jinglin Liu, Chenye Cui, and Zhou Zhao. Generspeech: Towards style transfer for generalizable out-of-domain text-to-speech. In NIPS, 2022b.
Rongjie Huang, Zhou Zhao, Huadai Liu, Jinglin Liu, Chenye Cui, and Yi Ren. Prodiff: Progressive fast diffusion model for high-quality text-to-speech. In Proceedings the 30th ACM International Conference on Mullimedia, pp. 2595-2605, 2022c.
Amir Jamaludin, Joon Son Chung, and Andrew Zisserman. You said that?: Synthesising talking faces from audio. International Journal of Computer Vision, 127(11:1767-1779, 2019.
Petr Kellnhofer, Lars C Jebe, Andrew Jones, Ryan Spicer, Kari Pulli, and Gordon Wetzstein. Neu- ral lumigraph rendering. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 4287-4297, 2021.
Cheng-Han Lee, Ziwei Liu, Lingyun Wu, and Ping Luo. Maskgan: Towards diverse and interactive facial image manipulation. In CVPR, pp. 5549-5558, 2020.
Xian Liu, Yinghao Xu, Qianyi Wu, Hang Zhou, Wayne Wu, and Bolei Zhou. Semantic-awar implicit neural audio-driven video portrait generation. arXiv preprint arXiv:2201.07786, 2022.
Yuanxun Lu, Jinxiang Chai, and Xun Cao. Live speech portraits: real-time photorealistic talking- head animation. ACM Transactions on Graphics, 40(6) 2021.
Ben Mildenhall, Pratul P Srinivasan, Matthew Tancik, Jonathan T Barron, Ravi Ramamoorthi, and Ren Ng. Nerf: Representing scenes as neural radiance fields for view synthesis. In ECCV, pp- 405-421. Springer, 2020.
Pascal Paysan, Reinhard Knothe, Brian Amberg, Sami Romdhani, and Thomas Vetter. A 3d face model for pose and illumination invariant face recognition. In 2009 sixth IEEE international conference on advanced video and signal based surveillance, pp. 296-301. Icce, 2009.
Hai X Pham, Samuel Cheung, and Vladimir Pavlovic. Speech-driven 3d facial animation with implicit emotional awareness: a deep learning approach. In CVPRW, pp. 80-88, 2017.
KR Prajwal, Rudrabha Mukhopadhyay, Vinay Namboodiri, and CV Jawahar. A lip sync expert is all you need for speech lip generation in the wild. In ACM MM, pp. 484 492, 2020.
Albert Pumarola, Enric Corona, Gerard Pons-Moll, and Francesc Moreno-Noguer. D-nerf: Neural radiance fields for dynamic scenes. In CVPR, pp. 10318-10327, 2021.
Yi Ren, Jinglin Liu, and Zhou Zhao. Portaspeech: Portable and high-quality generative text-to- speech. volume 34, pp. 13963-13974, 2021.
Shuai Shen, Wanhua Li, and Zheng Zhu. Learning dynamic facial radiance fields for few-shot talking head synthesis. In ECCV, pp. 682, October 2022.
Vincent Sitzmann, Michael Zollhöfer, and Gordon Wetzstein. Scene representation networks: Con- tinuous 3d-structure-awwre neural scene representations. volume 32, 2019.
Supasorn Suwajanakom,, Steven Seitz, and Ira Saaetmaachhr-Shizeemmaa Synthesizing obama: learning lip sync from audio. ACM Transactions Gruphics (ToG), 36(4):1-13, 2017.
Sarah Taylor, Taehwan Kim, Yisong Yue, Moshe Mahler, James Krahe, Anastasio Garcia Rodriguez, Jessica Hodgins, and Iain Matthews. deep learning approach for generalized speech animation. ACM Transactions on Graphics (TOG), 36(4):1-11, 2017.
Samuli Laine Antti Herva Tero Karras, Timo Aila and Jaakko Lehtinen. Audio-driven facial ani- mation by joint endto-end learning of pose and emotion. ACM Transactions on Graphics, 36(4), 2017.
Justus Thies, Mohamed Elgharib, Ayush Tewari, Christian Theobalt, and Matthias Nießner. Neural voice puppetry: Audio-driven facial reenactment. In ECCV, pp. 716-731. Springer, 2020.
Gadi Geiger Tony Ezzat and Tomaso Poggio. Trainable videorealistic speech animation. ACM Transactions Graphics, 21(3), 2002.
11

---

Published as a conference paper at ICLR 2023
Konstantinos Vougioukas, Stavros Petridis, and Maja Pantic. Realistic speech-driven facial anima- tion with gans. International Journal of Computer Vision, 128(5):1398-144 1413, 2020.
Shunyu Yao, RuiZhe Zhong, Yichao Yan, Guangtao Zhai, and Xiaokang Yang. Dfa-nerf: Person- alized talking head generation via disentangled face attributes neural rendering. arXiv preprint arXiv:2201.00791, 2022.
Zhenhui Ye, Zhou Zhao, Yi Ren, and Fei Wu. Syntaspeech: Syntax-aware generative adversarial text-to-speech. arXiv preprint arXiv:2204. /172 2022.
Ran Yi, Zipeng Ye, Juyong Zhang, Hujun Bao, and Yong-Jin Liu. Audio-driven talking face video generation with learning-basee personalized head pose. arXiv preprint arXiv:2002.10137 2020.
Lingyun Yu, Jun Yu, Mengyan Li, and Qiang Ling. Multimodal inputs driven talking face gener- ation with Technology, 31(1):203-211 2020.
dependency. IEEE Transactions on Circuits and Systems for Video
Zhimeng Zhang, Lincheng Li, Yu Ding, and Changjie Fan. Flow-guided one-shot talking face generation with high-resolution audio-visual dataset. In CVPR, pp. 3661-3670, 2021.
Zijian Zhang, Zhou Zhao, and Zhijie Lin. Unsupervised representation learning from pre-trained diffusion probabilistic models. In NIPS, 2022.
Zijian Zhang, Zhou Zhao, Jun Yu, and Qi Tian. Shiftddpms: Exploring conditional diffusion models by shifting diffusion trajectories. arXiv preprint arXiv:2302.02377. 2023.
Hang Zhou, Yu Liu, Ziwei Liu, Ping Luo, and Xiaogang Wang. Talking face generation by adver- sarially disentangled audio-visual representation. In AAAI, volume 33, pp. 2299-9306, 2019.
Hang Zhou, Yasheng Sun, Wayne Wu, Chen Change Loy, Xiaogang Wang. and Ziwei Liu. Pose- controllable talking face generation by implicitly modularized audio-visual representation. In CVPR, pp. 4176-4186, 2021.
Yang Zhou, Xintong Han, Eli Shechtman, Jose Echevarria, Evangelos Kalogerakis, and Dingzeyu Li. Makelttalk: speaker-aware talking-head animation. ACM Transactions on Graphics (TOG),
39(6):115,, 2020.
A DETAILS OF MODELS
A.1 VARIATIONAL MOTION GENERATOR
Following PortaSpeech, our variational motion generator consists of an encoder, decoder, and a flow-based prior model. The encoder, as shown in Fig. 6aa is composed of a ID-convolution followed by ReLU activation and layer normalization, and a non-causal WaveNet. The decoder. as shown in Fig. 6b consists of a non-causal WaveNet and ID transposed convolution followed by ReLU and layer normalization. The prior model, as shown in Fig. [6c| normalizing flow, which is composed of ID-convolution coupling layer and a channel-wiss flip operation. HuBERT features are utilized as the audio condition of these three modules.
A.2 SYNC-EXPERT
Our sync-expert inputs window of Ti consecutive 3D) landmark frames and an audio feature clip of size Ta x D. where Ti and Ta are the lengths of the video and audio clip respectively, and D the dimension of HuBERT features. The sync-expert is trained to discriminate whether the input audio and landmarks are synchronized. It consists of a landmark encoder and an audio encoder, as shown in Fig. 7 both of which are comprised of stack of ID-convolutions followed by batch normal- ization and ReLU. We use cosine-similane Specifically. compute cosine-similane for the landmark embedding I and audio embedding to represent the probability that the input audio-landmark pair is synchronized. The training loss of sync-expert be represented
with binary cross-entropy loss to train the sync-expert.
a
Lsyne = CE max |a
(11)
1111,,))
12

---

Published as a conference paper at ICLR 2023
z~ ()
landmark
2p
Zg
Non Causal
TransposedConv1D
lubert-
WavNet
ReLU +Layer Norm
Flip
ConvlD + ReLU
hubert
Non Causal
hubert
Conv1D Coupling Layer
+Layer Norm
WavNet
landmark
20,(
(a) Encoder
(b) Decoder
(c) Flow-based Prior
Figure 6: The structure of encoder, decoder, and flow-based prior variational motion generator.
Pos/Neg?
cosine similarity
ReLU
ReLU
X
Conv1D+BN
Conv1D+BN
Aubert clip
landmark clip
Figure 7: The structure of sync-expert.
refined landmark
Conv1D + ReLU
+Batch Norm
coarse landmark
Figure 8: The structure of pest-net.
A.3 DOMAIN ADAPTATIVE POST-NET AND DISCRIMINATOR
The domain adaptative post-net, shown in Fig. is composed of stack of residual ID- convolution followed by ReLU and batch normalization. The discriminator is a MLP composed of stack of fully connected layers followed with ReLU and dropout.
B DETAILED EXPERIMENTAL SETTINGS
B. MODEL CONFIGURATIONS
We list the hyper-parameters of GeneFace in Tab.
13

---

Published as a conference paper at ICLR 2023
Table Hyper-parameter list
Hlype-paramette
GeneFace
Encoder Layers
8
Decoder Layers
4
Encoder/Decoder Conv ID Kernel
5
Eneoder/Decoder Conv Channel Size
192
Variational Motion Generator
Latent Size
16
Prior Flow Layers
Prior Flow Conv ID Kernel
3
Prior Flow Conv ID Channel Size
64
Sync-expert Layers
14
Sync-expert Channel Size
512
Post-net Layers
8
Post-net ConviD Kernel
3
Post-net ConvID Channel Size
256
Post-net and Discriminator
Discrimnator Layers
5
Discrimnator Linear Hidden Size
256
Discrimmator Dropout Rate
0,25
Head/Torso-Netee Layers
11
Head/Torso- NeRF Hidden Size
256
NeRF-based Renderer
Landmark/Head Color Encoder Layers
3
Landmark/ Color Encoder Hidden Size
128
Sctting
L2 error on 3D landmark,
LMD
GeneFace (VAE + Flow landmark NeRF)
0.0371
3.933
vanilla VAE landmark NeRF
0.0385 0.0424
4.063
Regression Model landmark NeRF
4.305
AD-NeRF
N/A
4.199
Table 5: Ablation study on 3D Landmark L.2 error.
C ADDITIONAL EXPERIMENTS
C. QUALITATIVE RESULTS WITH BASELINES
To compare the generated results of each method, we show the keyframes of one in-domain audio clip in Fig 9 We have the following observations: Wav2Lip achieves competitive lip-sync perfor- mance yet generates blurry mouth results; 2) MakeltTalk and PC-AVS fail to preserve the speaker's identity, leading to unrealistic generated results; 3) I.SP generates unnatural lip movement during the transition phase of different syllables. Please see our supplementary video for better visualization.
C.2 EVALUATION 3D LANDMARK L2 ERROR
To evaluate the contribution of the variational generator to the quality the predicted landmark, we adopt L2 eиror the predicted 3D landmarks the metric. We compare our vairiaitonal generator (VAE+Flow) against vanilla VAE and a simple regression model trained with MSE loss. The results listed in Table It can be seen that removing the prior flow using regression-based model leads to a performance drop.
C.3 T-SNE VISUALIZATION FOR DOMAIN ADAPTATION
To further investigate the efficacy of post-net, we utilize T-SNE to visualize the landmarks dif- ferent domains in Fig. 10. The visualization results prove that there exists significant domain gap between the LRS3 dataset and the target person video, and our post-net successfully rigs the predicted landmarks from the LRS3 domain into the target person domain.
14

---

Published as a conference paper at ICLR 2023
Audio
/w
D/
GT
Wav2Lip
MakeltTalk
PC-AVS
LSP
AD-NeRF
GeneFace
Figure 9: The comparison of generated key frame results. We show the phonetic symbol of the key frame and the corresponding synthesized talking heads of all baselines. Please zoom in for better visualization. More qualitative comparisons can be found in demo video.
D LIMITATIONS AND FUTURE WORK
There are mainly two limitations of the proposed approach. Firstly, we found the landmark sequence generated by variational motion generator and post-net occasionally has tiny fluctuations, which results in some artifacts such as shaking hairs, etc. Currently, we utilize a heuristic post-processing method (Gaussian filter) to alleviate this problem. We will explore better modeling the temporal information in the network architecture to further improve the stability. Secondly, the current NeRF- based renderer is majorly based on the setting of vanilla NeRF, which results in long training and inference time.
In the future, we will explore three directions: (1) Most importantly, we will try to enhance the performance of the NeRF backend by combining recent progress in accelerated and weight NeRF (2) We will try to utilize GeneFace to achieve style and emotion controllable talking face generation. (3) We will improve the image fidelity and lip-sync quality of GeneFace when driven by singing voices.
15

---

Published as a conference paper at ICLR 2023
Figure 10: The T-SNE visualization of 3DMM landmarks in different datasets. The green and blue points denote the ground truth landmarks in LRS3 dataset and the target person video; The red and yellow points represent the predicted landmarks without/with the domain adaptation.
16


Reviews:
[{"review_id": "lsAf2KiWZbM", "reviewer": "Reviewer_6Jxc", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "This paper focuses on how to solve the generalization problem of realistic talking face video generation tasks with a landmark refinement module. They proposed GeneFace, a model that introduces additional 3DMM landmark refinement for a target person, thus transferring knowledge learned from a large corpus to out-of-domain data. By utilizing the joint supervision of a lip-syncing expert and an identity discriminator, the model adjusts the generalized 3DMM landmark prediction to match the face of the target person. In this way, the 3DMM prediction can match the distribution of the data used to train the person-specific video renderer. In addition, the paper introduces a head-aware torso-NeRF to address the head-torso separation issue due to dramatic head movements. The generalized and specialized parts of the model are separately trained on a large lip-reading dataset and two smaller lip-reading datasets, and it can achieve good experimental results without fine-tuning, proving its effectiveness.", "summary_of_the_review": "The paper has good innovation, has done some original work, and the proposed method is concise and has good reproducibility. But the training details of the baseline model are vague, which makes me skeptical about the result of the baseline comparison part. ", "strength_and_weaknesses": "Strength：\n1. The paper proposed a novel method to generalize the knowledge learned from the large talking face dataset to out-of-domain data, thus improving few-shot training performance without having to change parameters learned from a large corpus.\n2. This paper has done a lot of ablation experiments, and the experimental details shown in Table 3 can enable readers to understand the improvement brought by each method and enhance the reproducibility of the method.\n3. There is a detailed description regarding the structure chart of each component and specific hyperparameters of the method in this paper. \nWeakness:\n1. There is a little description of the dataset used to train the baseline models, e.g., whether the target person videos are used during training. This information is critical for a fair comparison. ", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}, {"review_id": "bQGFVqvlNUQ", "reviewer": "Reviewer_rq8u", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "The authors address the problem of synthesizing realistic talking faces from audio. They build upon a VAE-style encodings of 3D facial landmarks that are created from HuBERT features extracted from the audio signals. Instead of a vanilla VAE, the authors argue that a flow-based (non-Gaussian) prior is beneficial for better landmark generation. Conditioned on the predicted facial landmarks, a NeRF-style renderer then synthesizes an animated head and torso.", "summary_of_the_review": "The paper is of minor novelty and the main technical contribution lacks in-depth evaluation. The presented qualitative results are not convincing, i.e. they do not seem to be better than existing approaches to me.\n\n**Edit after rebuttal:**\n\nThe authors provided a strong and convincing rebuttal. They were able to address the major points that, in my view, put the paper below acceptance threshold. The authors could demonstrate that with appropriate post-processing, their method outperforms AD-NeRF, They also provided further evaluation that closes the gap the paper had before.\n\nI consider this a substantial improvement to the original paper and therefore will upgrade my rating.", "strength_and_weaknesses": "**Novelty**\n\nThe approach is of minor novelty. Audio-driven talking head synthesis based on NeRF has already been proposed in AD-NeRF and the 3D facial landmark generation is a HuBERT conditioned VAE plus flow modeling. While this specific architecture has not been used before, the paper does not provide significant novelty beyond modified architectures and details in the training.\n\n**Technical Details**\n\nThe technical presentation of the paper seems sound and is easy to follow. Details on the model are explained in the appendix, such that I believe the method can be reproduced from the information given in the paper.\n\n**Empirical Evaluation**\n\n(a) Qualitative results (video linked in the paper): The output of the approach produces significant artifacts such as hair that is constantly changing it's size/volume, unstable and wobbly face reconstruction, and significant artifacts in the neck area. I do not agree that this is an improvement over MakeItTalk, PC-AVS, LSP, or AD-NeRF.\n\n(b) User study. The user study has a small sample size (5 clips from 3 languages, 10 attendees) and given this small sample size, the differences in MOS scores compared to existing work seems not statistically significant. I'd further advise to include the standard deviation to the mean opinion scores, which will provide a confidence on the ratings.\n\n(c) Quantitative Evaluation. All metrics are neural feature maps. These alone can be delusive, a metric comparison would be more convincing. One option would be to compare the l2 error on facial landmarks to evaluate accuracy of lip and general face animation.\n\n(d) In-depth analysis. One major contribution of this paper is the variational motion generator for the generation of 3D facial landmarks. However, this contribution is lacking empirical analysis. Fig. 6 provides some insights on the domain adaptation, but a proper evaluation of the quality of the produced landmarks is not given. It would be more convincing if the authors could demonstrate clear and consistent improvements in 3D landmark generation by their variational motion model compared to a simple regression baseline, a simple VAE baseline, and landmarks from other approaches. Unfortunately, neither a quantitative evaluation (l2 error on predicted landmarks) nor a qualitative evaluation is provided.", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "4: All of the claims and statements are well-supported and correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}, {"review_id": "aMVeQ6MPds", "reviewer": "Reviewer_LMgs", "summary": "", "questions": "", "limitations": "", "rating": 5, "confidence": 5, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "This paper proposes GeneFace, a generalized and high-fidelity NeRF-based talking face generation method to generate natural results corresponding to various out-of-domain audio. This paper introduces a variaitional motion generator on a large lip-reading corpus, a domain adaptative post-net to calibrate the result and NeRFbased renderer conditioned on the predicted motion. The proposed methods achieve good performances on the public datasets.", "summary_of_the_review": "This paper introduces a generalized and high-fidelity NeRF-based talking face generation method for 3D talking head generation. The proposed methods consist of  a variaitional motion generator on a large lip-reading corpus, a domain adaptative post-net to calibrate the result and NeRFbased renderer. The writing and completion of this paper is good. However, this paper is not good enough due to the novelty and experimental weaknesses. The novelty of the proposed methods are limited and the experiments cannot support the motivations of the proposed methods very well.", "strength_and_weaknesses": "Strength: 1) The proposed methods achieve remarkable and comparable performances with the previous works;\n\n2) The domain adaptive post-net is interesting for the talking head generation tasks among the different targets. It may help enhance the generalization ability of the generated landmarks for the various faces.\n\nWeaknesses: 1) The novelty is limited. The landmark generation and nerf-based rendering networks are widely adopted for 3D talking head generation. The domain generalization is not firstly introduced for the various faces and audios. The speciality and priority of the proposed methods  need to be highlighted in the paper.\n\n2) The experiments cannot support the motivations and methods very well. The proposed methods aim to enhance the generalization abality for various faces. Comparing with the previous methods on the OOD conditions, the advantages of the proposed methods are not obvious.  Especially the comparisons of these methods on OOD faces need to be analyzed to highlight the effectiveness of the proposed domain adaptive post net. \n", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}, {"review_id": "Y2g4oh-yTE1", "reviewer": "Reviewer_vzEy", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "This paper proposed an end-to-end NeRF-based method, for talking face generation. It was trying to solve 2 problems: 1) weak generalizability due to small scale of training data, 2) “mean face” result: bad image quality and bad lip-synchronization. This paper proposed 3 parts: 1) Variational Motion Generator used to generate landmarks from audio, 2) Domain Adaptative Post-Net for landmark refinement, and 3) NeRF based Renderer for final frame generation. It produces good talking face generation results.", "summary_of_the_review": "The paper is overall well written and much of it is well described. It combines previous works and proposes an new approach to generating 3D talking face. It produces better results than previous NeRF-based work. I recommend this work.", "strength_and_weaknesses": "Strength:\nThis paper has well-organized structure, and it is clear in logicality. This work has well-designed network structure and proposed structure, which can actually solve proposed problems. This work combines previous works and gives an new approach to generating talking face in another way, which moves the NeRF-based method field forward a bit.\n \nWeaknesses:\nSome evaluation metric do not outperform other methods, this work could design more evaluation metrics to evaluate the results.\nThe generated face in the demo video is moving around, maybe consider temporal information and design into network structure.\nThe lip-synchronization is not as good as wav2lip, the synchronization can be improved.", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "4: All of the claims and statements are well-supported and correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}, {"review_id": "Cah3jsqtSD", "reviewer": "Reviewer_Qesv", "summary": "", "questions": "", "limitations": "", "rating": 8, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "This paper presents a method on audio driven 3D portrait. A three-stage pipeline is given. The first stage trains Variational Motion Generator to predict landmarks based on audio signal. The second stage uses a post-net to compensate for the domain shift between the predicted 3D landmarks and the target person domain. The third stage adopts NeRF for high fidelity rendering conditioned on landmarks. When dealing with torso-NeRF, a pixel-wise condition is added to NeRF function to reduce head-torso separation artifacts. \n\nAccording to the description in these three stages. The Variational Motion Generator is a generic model. Both the Domain Adaptative Post-net and NeRF renderer adopt subject-specific training to model personalized attributes.\n\nThe method could solve one-to-many mapping problem faced by generic NeRF-based talking face systems. And it could solve head-torso separation problem in rendering.", "summary_of_the_review": "This paper proposes a high-fidelity audio driven 3D talking face system. A novel generalized audio to landmarks module is given. And a subject-specific NeRF module is used to for high fidelity modeling which is an incremental modification on AD-NeRF architecture.\nThe title didn’t completely suit for the content and reference is not enough. Also limitations and future work should be added.", "strength_and_weaknesses": "strengths:\nThe method could solve one-to-many mapping problem faced by generic NeRF-based talking face systems. And it could solve head-torso separation problem in rendering.\n\nweaknesses:\n1) The title use ‘generalized’ to describe this system. It seems this word implies that no subject-specific training is needed. However according to the paper, both the Domain Adaptative Post-net and NeRF renderer adopt subject-specific training to model personalized attributes and need days to converge. I recommend changing this title to avoid misleading.\n2) This paper adopts NeRF for high fidelity rendering. This paper should add discussion on NeRF based head techniques such as NerFACE, HeadNeRF and EG3D, but not restricted to NeRF for talking face. One paper is left: Learning Dynamic Facial Radiance Fields for Few-Shot Talking Head Synthesis (ECCV 2022)\n3) This paper lacks discussion on limitations and future work.\n4) In Fig 1. Variational Motion Generator’s ‘Encoder’ should be ‘Decoder’. In Equ (8), ‘-’ is left in the computing of accumulated transmittance.", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}]

Author rebuttal:

Decision: Accept: poster