GAIA: Zero-shot Talking Avatar Generation
Authors: Tianyu He
Junliang Guo
Runyi Yu
Yuchi Wang
jialiang zhu
Kaikai An
Leyi Li
Xu Tan
Chunyu Wang
Han Hu
HsiangTao Wu
sheng zhao
Jiang Bian
Venue: ICLR 2024 poster (2024)
Keywords: Talking Avatar Generation
Video Generation
Disentanglement
Diffusion Models
Primary area: generative models

Abstract: Zero-shot talking avatar generation aims at synthesizing natural talking videos from speech and a single portrait image. Previous methods have relied on domain-specific heuristics such as warping-based motion representation and 3D Morphable Models, which limit the naturalness and diversity of the generated avatars. In this work, we introduce GAIA (Generative AI for Avatar), which eliminates the domain priors in talking avatar generation. In light of the observation that the speech only drives the motion of the avatar while the appearance of the avatar and the background typically remain the same throughout the entire video, we divide our approach into two stages: 1) disentangling each frame into motion and appearance representations; 2) generating motion sequences conditioned on the speech and reference portrait image. We collect a large-scale high-quality talking avatar dataset and train the model on it with different scales (up to 2B parameters). Experimental results verify the superiority, scalability, and flexibility of GAIA as 1) the resulting model beats previous baseline models in terms of naturalness, diversity, lip-sync quality, and visual quality; 2) the framework is scalable since larger models yield better results; 3) it is general and enables different applications like controllable talking avatar generation and text-instructed avatar generation.

Published as a conference paper at ICLR 2024
GAIA: ZERO-SHOT TALKING AVATAR GENERATION
Tianyu He Junliang Guo', Runyi Yu², Yuchi Wang", Jialiang Zhu, Kaikai An, Leyi Li Xu Tan', Chunyu Wang, Han Hu, HsiangTao Wu, Sheng Zhao, Jiang Bian
Microsoft
ht http:://mccroooff..ittbbbb
ABSTRACT
Zero-shot talking avatar generation aims synthesizing natural talking videos from speech and a single portrait image. Previous methods have relied on domain- specific heuristics such as warping-based motion representation and 3D Morphable Models, which limit the naturalness and diversity of the generated avatars. In this work, we introduce GAIA (Generative AI for Avatar), which eliminates the domain priors in talking avatar generation. In light of the observation that the speech only drives the motion of the avatar while the appearance of the avatar and the background typically remain the same throughout the entire video, we divide our approach into two stages: 1) disentangling each frame into motion and appearance representations; 2) generating motion sequences conditioned on the speech and reference portrait image. We collect a large-scale high-quality talking avatar dataset and train the model on it with different scales (up to 2B parameters). Experimental results verify the superiority, scalability, and flexibility of GAIA 1) the resulting model beats previous baseline models in terms of naturalness, diversity, lip-sync quality, and visual quality; 2) the framework is scalable since larger models yield better results; 3) is general and enables different applications like controllable talking avatar generation and text-instructed avatar generation.
INTRODUCTION
Talking avatar generation aims at synthesizing natural videos from speech, where the generated mouth shapes, expressions, and head poses should be in line with the speech content. Previous studies achieve high-quality results by imposing avatar-specific training (i.e., training adapting a specific
model for each avatar) (Thies et al.) 2020 Tang et al., 2022; Du et al. 2023 Guo et al. 2021). or by leveraging template video during inference (Prajwal et 2020 Zhou et al. 2021 [hhnn et al. 2023; Zhong et al., 2023). More recently, significant efforts have been dEdicated to designing and
improving zero- zero-shot shot talking avatar generation (Zhou et al. 2020: Wang et al., 2021a| Zhang et al.
2023b Wang et al. 2023; Yu et al. 2022; Gururani et al. 2022 Styputkowski et al. 2002)), l.c.,
only a single portrait image the target avatar available to indicate the appearance of the target avatar. However, these methods relax the difficulty of the task by involving domain priors such as warping-based motion representation (Siarohin et al. 2019; Wang et al.] 2021b). 3D Morphable Models (3DMMs) (Blanz& & Vetter 1999), etc. Although effective, the introduction of such heuristics hinders direct learning from data distribution and may lead to unnatural results and limited diversity.
In contrast, in this work, we introduce GAIA (Generative AI for Avatar), which eliminates the domain priors in talking avatar generation. GAIA reveals two key insights: 1) the speech only drives the motion of the avatar, while the background and the appearance of the avatar typically remain the same throughout the entire video. Motivated by this, we disentangle the motion and appearance for each frame, where the appearance is shared between frames and the motion is unique to each frame. To predict motion from speech, we encode motion sequence into motion latent sequence and
"Equal contribution.
1 Corresponning author; Xu Tan miuta@mmcrosoff.comm)

---

Published as a conference paper at ICLR 2024
predict the latent with diffusion model conditioned on the input speech; 2) there exists enormous diversities in expressions and head poses when an individual is speaking the given content, which calls for large-scale and diverse dataset. Therefore, collect high-quality talking avatar dataset that consists of 16K unique speakers with diverse ages, genders, skin types, and talking styles, to make the generation results natural and diverse.
More specifically, to disentangle the motion and appearance, we train a Variational AutoEncoder (VAE) consisting of two encoders (i.e., motion encoder and an appearance encoder) and one decoder. During training, the input of the motion encoder is the facial landmarks (Wood et al. 2021) of the current frame, while the input of the appearance encoder is a frame that is randomly sampled within the current video clip. Based on the outputs of the two encoders, the decoder is optimized to reconstruct the current frame. After we obtain the well-trained VAF, we have the motion latent (i.e., the output of the motion encoder) for all the training data. Then, we train a diffusion model to predict the motion latent sequence conditioned on the speech and one randomly sampled frame within the video clip, which provides appearance information to the generation process. During inference, given the reference portrait image of the target avatar, the diffusion model takes it and an input speech sequence the condition, and generates the motion latent sequence that is in line with the speech content. The generated motion latent sequence and the reference portrait image are then leveraged to synthesize the talking video output using the decoder of the VAE.
For the collected dataset, to enable the desired information can be learned from data, we propose several automated filtration policies to ensure the quality of the training data. We train both the VAE and the diffusion model on the filtered data. From the experimental results, we have three key conclusions: 1) GAIA is able to conduct zero-shot talking avatar generation with superior performance on naturalness, diversity, lip-sync quality, and visual quality. It surpasses all the baseline methods significantly according to our subjective evaluation; 2) we train the model with different scales, varying from 150M to 2B. The results demonstrate that the framework is scalable since larger models yield better results; 3) GATA is general and flexible framework that enables different applications including controllable talking avatar generation and text-instructed avatar generation.
RELATED WORKS
Speech-driven talking avatar generation enables synthesizing talking videos in syne with the input speech content. Early methods have been proposed to train or adapt specific model for each avatar with a focus on overall realness (Thies et al. 2020 Lu et al. 2021). natural head poses (Zhou et al., 2021). high lip-syne quality (Lahiri et al. 2021) and emotional expression (Ji et al. (2021).
Despite sigmificant advances made by these methods, the costs are high due to the avatar-specific training. This motivates zero-shot talking avatar generation, where only one portrait image of the target avatar is given. However, animating a single portrait image is not easy due to the limited information we have. MakeltTalk (Zhou et al. (220)) handled this by first predicting 3D landmark displacements from the speech input, then the predicted landmarks are transferred to warping-based motion representation (Siarohin et al., 2019), which is employed to warp the reference image to the desired expression and pose. Burkov et al. 2020) achieved pose-identity disentanglement, but needs additional fine-tur fine-tuning for the unseen identities. More recently, SadTalker (Zhang et al.) 2023b) leveraged 3DMMs as an intermediate representation between the speech and the video, and proposed two modules to predict the expression coefficients of 3DMMs and head poses respectively. In general, the current solutions relax the difficulty of the task by involving domain priors like warping- based transformation (Zhou et al. 2020 Wang et al. 202 2022 Liu et al.) 2022 Drobyshev et al. 2022 Gururani et al. 2022). ) 3DMMs (Ren et al. 2021 Zhang et al. (02132023b) etc. Although the introduction of these heuristics makes the modeling easier, they ievitabll hinder the end-to-end learning from data distribution, leading to unnatural results and limited diversity. PC-AVS (Zhou et al. 2021) and PD-FGC (Wang et al. (023) similarly introduced identity space and non-identity space by leveraging the identity labels. The authors employed contrastive learning to align the non-identity space and speech content space. Our method differs in three ways: 1) they need additional driving video. Instead, we generate the entire motion from the speech at the same time and also provide the option to control the head pose; 2) they use contrastive learning to align speech and visual motion. In contrast, we leverage diffusion models to predict motion from the speech; 3) our method does not need additional identity labels. As verified in experiments, our method results in natural and consistent motion, and flexible control for talking avatar generation.
2

---

Published as a conference paper at ICLR 2024
Motion Latent
Appearance Latent
Decoder
Diffusson Model
Appearance Encoder
Motion Encoder
Pose Predictor
Speech Encoder
Stage 1: Motion Appearance Disentanglement
Stage Speech-to-Motion Generation
Figure 1: Method overview. GAIA consists of a VAE (the orange modules) and diffusion model (the blue and green modules). The VAE is firstly trained to encode each video frame into disentangled representation (i.e., motion and appearance representation) and reconstruct the original frame from the disentangled representation. Then the diffusion model is optimized to generate motion sequences conditioned on the speech sequences and random frame within the video clip. During inference, the diffusion model takes an input speech sequence and the reference portrait image as the condition and yields the motion sequence, which decoded the video by leveraging the decoder of the VAE.
3 DATA COLLECTION AND FILTERING
A data-driven model is naturally scalable for large datasets, but it also requires high-quality data as it learns from data distribution. We con- struct our dataset from diverse sources. For high-quality public datasets, we collect High- Definition Talking Face Dataset (HDTF) (Zhang et al. 2021) and Casual Conversation datasets v1&v2 (CC v1&v2) (Hazirbas et al. 2021 Por- gali et al.) 2023) which contain thousands of Identities (IDs) with a diverse set of ages, gen- ders, and apparent skin types. In addition to these three datasets, we also collect a large-scale internal talking avatar dataset which consists of 7K hours of videos and 8K unique speaker IDs, to make the resulting model scalable and unbiased.
Table 1: Statistics of the collected dataset.
Raw
Filtered
Datasets
#IDs
#Hours
#IDs
#Hours
HDTF
362
16
359
14
CC vl
3, 3,011
750
2,957
330
CC v2 Internal
5, 5676
440
4, 646
183
8, 0,007
7,000
8, 0.00
642
Total
16, 9417
8,206
15, 969
169
The overview of the dataset statistics demonstrated in Tab. []]
However, the raw videos are surrounded by noisy cases that are harmful to the model training, such as non-speaking clips and rapid head moves. To enable the desired information can learned from data, we develop several automated filtration policies to improve the quality of the training data: 1) to make the lip motion visible, the frontal orientation of the avatar should toward the camera; 2) to ensure the stability, the facial movement in video clip should be smooth without rapid shaking; 3) to filter out corner cases where the lip movements and speech are not aligned, the frames that the avatar wear masks keep silent should be removed. Please refer Appendix [A.] for more details. After filtration, we find that a majority of raw videos are dropped, which is necessary for the training of data-driven model according to preliminary experimental results, where the video quality generated by models trained on raw videos falls behind the one trained on filtered data.
4 MODEL
4.1
MODEL OVERVIEW
The zero-shot scenario that generates talking video of an unseen speaker with one portrait image and a speech clip requires two key capabilities the model: 1) the disentangled representation of appearance and motion from the image, the former should be consistent while the latter dynamic in the generated video; 2) generate the motion representation conditioned on the speech in each timestamp. Correspondingly, AutoEncoder (VAE) (Kingma & Welling 2014 that extracts image representations and a diffusion model for speech-to-motion generation.
shown in Fig. we propose models including Variational
3

---

Published as a conference paper at ICLR 2024
Problem Definition Given one portrait image and sequence of speech clip [$1, sN] the model aims to generate a talking video clip [21,..,, Øy] which is lip-syncing with speech s and appearance consistent with image
4.2 MOTION AND APPEARANCE DISENTANGLEMENT
Given frame of talking video we would like to encode its motion representation which will serve as the generation target of the diffusion model. Therefore, it is crucial to disentangle the motion and appearance representation from We propose VAE that consists of two encoders, i.., motion EM and appearance encoder A and one decoder D. We then use the appearance information from the i-th frame and the motion information from the j-th frame to reconstruct the j-th frame by the VAE, in order to prevent the leakage of the appearance information in reconstruction. In this way, as the i- and j-th frames from one video clip contain the same appearance but different motion information, i.e., the same person talking different words, the VAE model will learn to first extract the pure appearance feature from the i-th frame, and then combine it with the pure motion feature of the j-th frame to reconstruct the original j-th frame. The individuals of the i- and j-th frame can be flexibly chosen for both self- recontrccntttti and cross-reenactment settings.
Motion and Appearance Encoder Specifically, denote the raw RGB image of as xt R RBx and its landmark as x E IR #xxx3 x3 which is predicted by an external tool (Wood et al. 2021). The landmark is supposed to only contain the locations of key facial features such as the mouth, while the raw image provides other appearance information including identity and background. Given two frames zr() and z(j) from one video clip, the model takes (i) and xn' (z) as imputs to the appearance and motion encoder respectively, and produces their latent representations:
(i) (A() =² = Ex ()),
where =i) E Rh*"x"" and 2mh =²)(j) RA" x="x3, Note that in practice use smaller size of hm than h as landmarks usually contain less information which easier to encode. The two latent representations are then projected the same size and concatenated together to reconstruct z" (j) by
the decoder:
(²()) = (i),
The two encoders EA and Er share similar model architectures except for the downsampling factors, and z"h (j) is first up-sampled to the same size z (j) followed by concatenation and projection and then served as the input to the decoder.
Training We train the VAE model in an adversarial manner learn perceptually rich representations following previous works (Esser et al. 2021 Rombach et al.) 2022). In addition to the perceptual L. reconstruction loss (Zhang et al. 2018) T.ree .r) and the KI.-penalty (x((2) the latent towards standard normal distribution (Kingma & Welling 2014), we introduce discriminator faie to distinguish between the real frame x and the generated f::
Lais (I, .)) = log [xis(x) + log(1 - [dis())).
(3)
Then the total loss function of training the VAE can be written as:
LV
min max Lrrel En, Exs) Eaa(x; EMd Laislx; Sais)). EA.EM.D Jeie
4.3 SPEECH-TO-MOTIO GENERATION
Once the VAE is trained, we are able to obtain a motion latent sequence z E NRA" xw"" an appearance latent sequence za R xx xw" for each video clip. We also have its corresponding speech feature 2 Nx' extracted by wav2vec 2.0 (Bacvski et al. 2020).. We leverage diffusion model with Conformer (Gulati et al., 2020) backbone S to predict the motion latent sequence zt conditioned on the paired speech feature zx and one reference frame z(i). The speech feature gives the driving information and the reference frame provides identity-relater information like facial contour, the shape of eyes, etc.
Since the speech feature 2" comes from a fixed feature extractor (Baevski et al. 2020). to adapt it to our model, process it with lightweight speech encoder A before feeding it into the diffusion
4

---

Published as a conference paper at ICLR 2024
model. Given that the diffusion model predicts the motion latent sequence, thus the motion latent (i) of the reference frame (t) as the condition. which is obtained by the pre-trained motion encoder Exs. During training, the reference frame is randomly sampled within the video clip. Following previous practice (Du et al., 2023), we generate a pseudo-sentence for data augmentation by sampling subsequence with random starting point and random length for each training pair.
Diffussio Model Our goal is to construct forward diffusion process and a reverse diffusion process that has tractable form to generate data samples. The forward diffusion gradually perturbs data samples z0" into Gaussian noise with infinite time steps. Then in the reverse diffusion, with the learned score function, the model is able to generate desired data samples 2 from Gaussian noise in an iterative denoising process. Formally, the forward diffusion can be modeled as the following stochastic differential equation (SDE) (Song et al. 2021):
d2 = 2 3(2," dt + 3, dwy, E [0,1],
(5)
where noise schedule 3g is non-negative function, W, is the standard Wiener process (i.e., Brownian motion). According to previous literature (Song et al. (202T) the reverse diffusion that transforms the Gaussian noise to the data sample can therefore be written as:
Vlog di v Be dü,, t E [0, 1], 1 "t -
dæy"
(6)
where Wj is the reverse-time Wiener process, is the probability density function of 2!"
In addition, /Song et al. (2021) have shown that there is an ordinary differential equation (ODE) for the reverse diffusion:
1 (2 Vlogg((="")) dt.
d=
Given the above formulation, train neural network S to estimate the gradient of the log-density of noisy data sample V log Pt (zf")) As a result, we can model p(ag") by sampling 2 ~ (0, and then numerically solving either Equ. oor Equ. 7
Conditioning In addition to the noised data sample, our diffusion model processes additional conditional information: the noise time step the speech feature 2", and reference motion latent (i) coming from the same clip. Following previous successes (Ho et al. 2020 Rombach et al. 2022). the noise time step is projected to an embedding and then directly added to the input of each Conformer block. For the speech feature, since it should be aligned with the output, we add it to the hidden feature of each Conformer block an element-wise manner. For the reference motion latent, we employ a cross-attention layer (Vaswani et al. 2017 Rombach et al. (022)) for each Conformer block, in which the hidden sequence in the Conformer layer acts as the query and the reference motion latent acts the key and value.
Pose-controllable Generation Predicting motion latent from the speech is a one-to-many mapping problem since there are multiple plausible head poses when speaking a sentence. To alleviate this ill-posed issue, we propose to incorporate pose information during training (Du et al. [2023] Tang et al. 2022). To achieve this, we extract the head poses rº N (pitch, yaw, and roll) using an open-source tool and add the extracted poses to the output of speech encoder AA through learned linear layer. By complementing the prediction with the head poses, the model puts more focus on generating realistic facial expressions, mouth shapes, etc.
To enable flexible generation during inference (i.e., one can use either the appointed head poses or the predicted one to control the generated talking video), we also train pose predictor P to estimate the head poses according to the speech. The pose predictor P consists of several convolutional layers and is optimized by the mean square error between the extracted head poses x" and the estimated one ??.
Training
We jointly train the models S, A and P with the following loss function:
Las Egg |||
1
112 + Lmsse (x², .2P).
(8)
where the first term is the data loss, 20m = S(x7" t, 2-, nm (i), x²). and the second item is the loss for head pose prediction.
https:://itthbb.com wttp:////ow//mmw.....ooooo
5

---

Published as a conference paper at ICLR 2024
Table 2: Quantitative comparisons of the GATA VAE model with previous video-driven baselines.
Self-Reconstruction
Cross-Rennactment
Methods
FID
LPIPS!
PSNR*
AKDL
MSI
FID
AKD|
MSIT
FOMM
23.843
0.196
22.669
2.160
0.839
45.951
3.404
0.838
HcadGAN
21.499
0.278
18.555
2.990
0.835
90.746
5.964
0.788
face-vid2vid
18.601
0.184
23.681
2.195
0.813
28.093
3.630
0.853
GAIA (Ours)
15.730
0.167
23.942
1.442
0.856
15.200
2.003
1.102
Table 3: Quantitative comparisons of the GAIA framework with previous speech-driven methods. The subjective evaluation is rated at five grades (1-5) in terms of overall naturalness (Nat.), lip-sync quality (Lip.), motion jittering (Jit.), visual quality (Vis.), and motion diversity (Mot... Note that, the Sync-D score for real video is 8.548, which is close to ours.
Subjective Evaluation
Objective Evaluation
Methods
Nat.f
Lip.t
Jit.
Vis.t
Mot.t
Syne-DL
MSI
FIDI
MakelrTalk
2.148
2.161
1.739
2.789
2.571
9.932
1.140
28.894
Audio2Hlead
2.355
3.278
2.014
2.494
3.298
8.508
0.635
28.607
SadTalker
2.884
4.012
4.020
3.569
2.625
8.606
1.165
22.322
GAIA (Ours)
4.362
4.332
4.345
4.320
4.243
8.528
1.181
22.924
EXPERIMENTS
Benefitting from the disentanglement between motion and appearance, GAIA enables two common scenarios: the video-driven generation which aims to generate results with the appearance from a reference image and the motion from a driving video, and the speech-driven generation where the motion is predicted from speech clip. The video-driven generation evaluates the VAE, while the speech-driven one evaluates the whole GAIA system. We compare GAIA with state-of-the-ae methods for the two scenarios in Sec. 5.2 and further make detailed analyses in Sec. [5.3 to understand the model better. To verify the scalability GAIA, we evaluate it at different scales in Sec. 5.3 i.c., from 150M to 2B model parameters in total. Due to the flexibility of our architecture, we also enable extended applications like text-instructed avatar generation, cose-controllable and fully controllable talking avatar generation (i.e., the mouth region is synced with the speech, while the rest of facial attributes can be controlled by the given talking video), which we demonstrate in Sec. 5.4
5.1 EXPERIMENTAL SETUPS
Datasets We train our model on the union of the datasets described in Sec. 3 and we randomly sample 100 videos from them as the validation set. For the test set, to climinate the potential overlap and evaluate the generality of our model, create out-domain test set by choosing 500 videos from TalkingHead- IKH (Wang et al. 2021b) dataset. We test all basclines on the same set.
Implementation Details We adjust the VAE and the diffusion model to different scales by changing the hidden size and the number of layers in each block, resulting in VAE of 80M, 700M, 1.7B parameters and diffusion model of 180M, 600M, 1.2B parameters. Refer to Appendix B for the details of model architecture and training strategies.
Evaluation We utilize various metries including subjective and objective ones to provide thorough evaluation of the proposed framework. Subjective metrics: we conduct user studies to evaluate the lip-syne quality, visual quality, and head pose naturalness of the generated videos. 20 experienced users are invited to participate. We adopt MOS (Mean Opinion Score) as our metric. We present one video at a time and ask the participants to rate the presented video at five grades (1-5) in terms of overall naturalness, lip-sync quality, motion jittering, visual quality, and motion diversity respectively. Objective metrics: we adopt various objective metrics to evaluate the visual and motion quality of generation results. For visual quality, we report FID (Heusel et al. 2017) and LPIPS (Zhang et al.) 2018) for perceptual similarity, and PSNR to measure the pixel-level mean squared error (MSE) between the ground truth and the reconstruction of the VAE. In addition, detect the landmarks of

---

Published as a conference paper at ICLR 2024
Ref. Image
video
recommend
quip
guest
Ref: Imagee
why
acoess
under
Makeetaall Roal
Figure 2: Qualitative comparison with the state-of-the-ar speech- methods. It shows that GAIA achieves higher naturalness, lip-sync quality, visual quality and motion diversity. In contrast, the baselines tend to highly rely on the reference image (Ref. Image) therefore making generation with slight head motions (e.g., most of the baselines generate results with closed eyes when the eyes of the reference image are closed) or inaccurate lip synchronization.
ground truth and reconstructed images and report the Average keypoint distance (AKD) (Wang et al. 2021b) between them, evaluate the motion quality of VAE reconstructions. Motion Stability Index (MSI) (Ling et al., 20222) which measures the motion stability of results is also reported. Following previous works (Thies al.) 2020) Tang et al., 2022). we adopt Syne-D (SyneNet Distance) to measure the lip-sync quality via SyncNet (Chung & Zisserman 2016
5.2 RESULTS
We compare the proposed GAIA model with state-of-the-e baselines in this section. Our model is general and can be applied to two common settings: the video-driven generation which aims to generate results with the appearance from reference image and the motion from a driving video, and the speech-driven generation where the motion will be predicted from speech clip. The video-driven generation evaluates the VAE, while the speech-driven one evaluates the whole GAIA system.
5.2.
VIDEO-DRIVEN RESULTS
We consider two different settings of the video-dríven talking avatar generation including self- reconstruction and cross-reenactment, depending whether the individual of the appearance frame is consistent with the driving motion frames. Details of the two settings are provided in Appendix B.2 We compare with three strong baselines including FOMM (Siarohin et al. (2019), HeadGAN (Doukas et al. 2021) and face-vid2vid (Wang et al., 2021b). which are all equipped with feature warping, commonly utilized prior technique in talking video generation. The results are shown in Tab. 2 The VAE of GAIA achieves consistent improvements over previous video-drives baselines, especially the cross-reenactment settings, illustrating our model successfully disentangles the appearance and motion representation. Note that as a part of the data-driven framework, we try to make the VAE as simple as possible, and climinate some commonly used external components such as a face recognition model (Deng et al. 2020) that provides identity-preservinn preserving losses.
5.2.2 SPEECH-DRIVED RESULTS
The speech-driven talking avatar generation is enabled by predicting motion from the speech instead of the driving video. We provide both quantitative and qualitative comparisons with MakeltTalk (Zhou) et al. 2020), Audio2Head (Wang et al.) 2021a), and SadTalker (Zhang et al., 2023b) in Tab. 3 andd Fig. It can be observed that GAIA surpasses all the baselines by large margin in terms of subjective evaluation. More specifically, as shown in Fig. [ the baselines tend to make generation with high dependence the reference image, if the reference image is given with closed
7

---

Published as a conference paper at ICLR 2024
Table 4: Scaling the VAE of GAIA. "#Params." and "#Hours" indicate the number of parame- ters and the size of the training dataset.
Table 5: Scaling the diffusion model of GAIA. We use the VAE model of 700M parameters for all experiments.
#Params. VAE
#Hours
FIDJ
#Params. Diffusion
#Hours
Sync-DL
80M
0.5K
18.353
180M
0.1K
9.145
80M
1K
17.486
180M
1K
8.913
700M
1K
15.730
600M
1K
8.603
1.7B
15.886
1.2B
1K
8.528
eyes or unusual head poses. In contrast, GAIA is robust to various reference images and generates results with higher naturalness, lip-sync quality, visual quality and motion diversity. For the objective evaluation in Tab. the best MSI score demonstrates that GAIA generates videos with great motion stability. The Sync-D score of 8.528, which is close to the one of real video (8.548), illustrates that the generated videos have great lip synchronization. We obtain comparable FID score to the baselines, which might be affected by the diverse head poses as we find that the model trained without diffusion realizes better FID score in Tab. 6
5.3 ABLATION STUDIES
5.3.1 ABLATION STUDIES ON SCALING
We change the scale of the model parameters as well as the training dataset to show the scalable of GAIA. For the model, we change the scales of VAE and Diffusion separately to study their influence on the framework. For the training set, we use the whole set with 1K hours or the subset of
The results listed in Tab. and Tab. 5 and size both benefit the proposed GALA framework. For the VAE model, the results are tested with the selftreconttt setting, which tends converge when the model larger than 700M. For the sake efficiency, we utilize the 700M VAE model in our main experiments. As for the diffusion model, we still realize better results when the model grows up to 1.2B parameters.
find that scaling up the parameters and data
5.3.2 ABLATION STUDIES ON PROPOSED TECHNIQUES
We study the proposed techniques de- tail: 1) we encode each frame to the latent without disentanglement, and uti- lize the diffusion model to predict the la- tent (w/o disentanglement); 2) we gener- ate the motion latent without making the condition on the head pose (w/o head pose); 3) we the Conformer to pre- dict the motion latent directly without the diffusion process (w/o diffusion); 4)
Table 6: Ablation studies on the proposed techniques.
Methods
Sync-D4
MSI1
FIDL
GAIA (700M + 180M)
8.913
1.132
24.242
who disentanglement
12.680
1.423
140.009
w/o head pose
9.134
1.208
23.648
w/o diffusion
9.817
1.486
21.049
w. landmark prediction
9.331
1.038
27.022
we synthesize the coordinates of the landmarks, instead of the latent representation (w. landmark prediction). All experiments are conducted based on the 700M VAE model and the 180M diffusion model. As shown in Tab. 66, which demonstrates that: 1) the model without disentanglement fails to generate effective results; 2) the model trained without head pose or diffusion process yields inferior performance; 3) predicting landmarks, instead of the motion latent like ours, degrades the performance in all aspects. This illustrates that encoding motion into latent representation helps the learning of motion generation.
5.4 CONTROLLABLE GENERATION
Pose-controllable Talking Avatar Generation As introduced in Sec. 4.3 in addition predicting the head pose from the speech, we also enable the model with pose-controllable generation. We implement it by replacing the estimated head pose with either handcrafted pose the one extracted from another video, which is demonstrated in Fig. 3(a) Refer to Appendix D for more details.
Fully Controllable Talking Avatar Generation Due to the controllability of the inverse diffusion process, we can control the arbitrary facial attributes by editing the landmarks during generation.
8

---

Published as a conference paper at ICLR 2024
Image
Red. Motion
Vidoo
Image
Video2
AARI
(a) Pose-controllable video generation.
(b) Fully controllable video generation.
Ref. Image
Ref. Image
Suppetoo Civenaa
Turns boodkt
Ned
(c) Text-driven video generation.
Figure 3: Examples of controllable and text-driven video generation. Due to the flexibility of our framework, 1) we enable gulti-granullriity motion control over the generated video. 2) we realize text-instructed video generation. See Sec. 5.4 for the details.
Specifically, we train diffusion model to synthesize the coordinates of the facial landmarks. The landmarks that we want to edit are fixed to reference coordinates. Then we leave the model to generate the rest. In Fig. 3(b] we show the results fully controllable generation, i.e., the mouth and jaw are synced with the speech, while the rest of the facial attributes are controlled by the reference motion. Refer to Appendix D for more details.
Text-driven Video Generation In general, the diffusion model is motion generator conditioned on speech, where the condition can be altered to other modalities flexibly. To show the generality of our framework, we consider textual instructions as the condition of the diffusion model, and enable the text-to-video generation (Fig. 3(c). Refer to Appendix E.2 for more details.
5.5 DISCUSSION
Different from previous works that employ warping-based motion representation (Wang al. 202 Drobyshev et al. (022),, pre-defined 3DMM coefficients (Zhang et al.] (003b)). we propose to eliminate these heuristics and generate the full motion latent the same time. The framework discloses three insights: 1) the complete disentanglement between the motion and the appearance the key to achieving zero-shot talking avatar generation; 2) handling one-to-many mapping with the diffusion model and learning full motion from real data distribution result in natural and diverse generations; 3) less dependence on heuristics and labels makes the method general and scalable.
6 CONCLUSION
We present GAIA, data-driven framework for zero-shot talking avatar generation which consists of two modules: a variational autoencoder that disentangles and encodes the motion and appearance representations, and a diffusion model to predict the motion latent conditioned on the input speech. We collect a large-scale dataset and propose several filtering policies to enable the successful training of the framework end-to-end, The GAIA framework is general and scalable, which can provide natural and diverse results in zero-shot talking avatar generation, well being flexibly adapted to other applications including controllable talking avatar generation and text-driven video generation.
Limitations and Future Works Our work still has limitations. For example, we leverage a pre- trained landmark extractor and a head pose extractor, which may hinder the end-to-end learning of the models. We leave the fully end-to-end learning (c.g., disentangle motion and appearance without the help of landmarks) future work.

---

Published as a conference paper at ICLR 2024
Responsible AI Considerations GAIA is intended for advancing AI/M research talking avatar generation. We encourage users to use the model responsibly and to adhere to the Microsoft Responsible AI Principles P We discourage users from using the method to generate intentionally deceptive or untrue content or for inauthentic activities. To prevent misuse, adding watermarks is common way and has been widely studied in both research and industry works (Ramesh et al.) (0022 Saharia et al., (022)). On the other hand, as an AIGC model, the generation results of our model can be utilized to construct artificial datasets and train discriminative models.
REFERENCES
Alexei Baevski, Yuhao Zhou, Abdelrahman Mohamed, and Michael Auli. wav2vee 2.0: A framework for self- supervieed learning of speech representations. Advances in Neural Information Processing Systems (NeurIPS), 33:12449-12444 2020.
Volker Blanz and Thomas Vetter. A morphable model for the synthesis of 3d faces. In Proceedings of the 26th annual conference on Computer graphics and interactive techniques, pp. 187-194, 1999.
Egor Burkov, Igor Pasechnik, Artur Grigorev, and Victor Lempitsky. Neural head reenactment with latent pose descriptors. In Proceedings of the IEEE/CVF conference on computer vision and pultern recognition, pp. 13786-13795, 2020.
J Chung, A Nagrani, and Zisserman. Voxceleb2: Deep speaker recognition. Interspeech, 2018.
Joon Son Chung and Andrew Zisserman. Out of time: automated lip sync in the wild. Asian conference on computer vision, pp. 251-263. Springer, 2016.
Alexandre Defossez, Gabriel Synnaeve, and Yossi Adi. Real time speech enhancement in the waveform domain. In Interspeech, pp. 3291-3292 2020.
Jiankang Deng, Jia Guo, Evangelos Ververas, Irene Kotsia, and Stefanos Zafeiriou. Retinaface: Single-shot multi-level face localisation in the wild. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pp. 5203-5212, 2020.
Michail Christos Doukas, Stefanos Zafeiriou, and Viktoriaa Sharmanska. Headgan: One-shot neural head synthesis and editing. Proceedings of the IEEE/CVF International Conference on Computer Vision (ICCV), pp- 14398-14407, 2021.
Nikita Drobyshev, Jenya Chelishev, Taras Khakhulin, Aleksei Ivakhnenko, Victor Lempitsky, and Egor Zakharov. Megaportraits: One-shot megapixel neural head avatars. In Proceedings of the 30th ACM International Conference on Multimedia, pp. 2663 2671, 2022.
Chenpeng Du, Qi Chen, Tianyu He, Xu Tan, Xie Chen, Kai Yu, Sheng Zhao, and Jiang Bian. Dae-talker: High fidelity speech-driven talking face generation with diffusion autoencoder. In Proceedings of the 31st ACM International Conference on Multimedia, 2023.
Patrick Esser, Robin Rombach, and Bjorn Ommer. Taming transformers for high-resolution im- age synthesis. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pp. 12873-12883 2021.
Anmol Gulati, James Qin, Chunn--heng Chiu, Niki Parmar, Yu Zhang, Jiahui Yu, Wei Han, Shibo Wang, Zhengdong Zhang, Yonghui Wu, et al. Conformer: Convolution-augmente transformer for speech recognition. Interspeech, 2020.
Yudong Guo, Keyu Chen, Sen Liang, Yong-Jin Liu, Hujun Bao, and Juyong Zhang. Ad-nerf: Audio driven neural radiance fields for talking head synthesis. In Proceedings of the IEEE/CVF International Conference Computer Vision (ICCV) pp. 5784-5794, 2021.
Siddharth Gururani, Arun Mallya, Ting-Chun Wang, Rafael Valle, and Ming- Yu Liu. Spacex: Speech-driven portrait animation with controllable expression. arXiv preprint 1..9992 2022.
10

---

Published as a conference paper at ICLR 2024
Caner Hazirbas, Joanna Bitton, Brian Dolhansky, Jacqueline Pan, Albert Gordo, and Cristian Canton Ferrer. Towards measuring faimess in ai: the casual conversations dataset. IEEE Transactions on Biometrics. Behavior, and Identity Science, 4(3):324-332. 2021.
Martin Heusel, Hubert Ramsauer, Thomas Unterthiner, Bernhard Nessler, and Sepp Hochreiter. Gans trained by a two time-scale update rule converge to a local nash equilibrium. Advances Neural Information Processing Systems (NeurIPS), 30, 2017.
Jonathan Ho, Ajay Jain, and Pieter Abbeel. Denoising diffusion probabilistic models. Advances in Neural Information Processing Systems (NeurIPS), 33:684006885 2020.
Xinya Ji, Hang Zhou, Kaisiyuan Wang. Wayne Wu, Chen Change Loy, Xun Cao, and Feng Xu. Audio-driven emotional video portraits. In Proceedings of the IEEE/CVF Conference on Computer Vision and Putlern Recognition (CVPR), pp. 14080-14089, 2021.
Diederik P Kingma and Jimmy Ba. Adam: A method for stochastic optimization. In International Conference on Learning Representations (ICLR), 2015.
Diederik Kingma and Max Welling. Auto-encoding variational bayes. In International Conference on Learning Representations (ICIR), 2014.
Avisek Lahiri, Vivek Kwatra, Christian Frueh, John Lewis, and Chris Bregler. Lipsync3d: Data- efficient learning of personalized 3d talking faces from video using pose and lighting normalization. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pp. 2755-2755755 7756,2764, 2021.
Jun Ling, Xu Tan, Liyang Chen, Runnan Li, Yuchao Zhang, Sheng Zhao, and Li Song. Stable- face: Analyzing and improving motion stability for talking face generation. arXiv preprint arXiv:2208. 13717, 2022.
Xian Liu, Qianyi Wu, Hang Zhou, Yuanqi Du, Wayne Wu, Dahua Lin, and Ziwei Liu. Audio- driven co-speech gesture video generation. Advances in Neural Information Processing Systems (NeurIPS), 35:21386-21399, 2022.
Yuanxun Lu, Jinxiang Chai, and Xun Cao. Live speech portraits: real-time photorealistic talking-head animation. ACM Transactions on Graphics (TOG), 40(6): 2021.
Bilal Porgali, Vitor Albiero, Jordan Ryda, Cristian Canton Ferrer, and Caner Hazirbas. The casual conversations v2 dataset. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pp. 10 10-17, 2023.
KR Prajwal, Rudrabha Mukhopadhyay, Vinay P Namboodiri, and CV Jawahar. A lip syne expert is all you need for speech to lip generation in the wild. In Proceedings of the 28th ACM International Conference on Multimedia, pp. 484-492, 2020.
Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, et al. Learning transferable visual models from natural language supervision. In International Conference Machine Learning (ICML), pp. 3748-8763. PMLR, 2021.
Aditya Ramesh, Prafulla Dhariwal, Alex Nichol, Casey Chu, and Mark Chen. Hierarchical text- conditional image generation with clip latents. arXiv preprint arXiv:2204.06125, 2022.
Yurui Ren, Ge Li, Yuanqi Chen, Thomas H Li, and Shan Liu. Pirenderer: Controllable portrait image generation via semantic neural rendering. In Proceedings of the IEEE/CVF International Conference on Computer Vision (ICCV), pp. 13759- 13768, 2021.
Robin Rombach, Andreas Blattmann, Dominik Lorenz, Patrick Esser, and Björn Ommer. High- resolution image synthesis with latent diffusion models. In Proceedings of the IEEE/CVF Confer- ence on Computer Vision and Pattern Recognition (CVPR), pp. 10684-10695, 2022.
Chitwan Saharia, William Chan, Saurabh Saxena, Lala Li, Jay Whang, Emily L Denton, Kamyar Ghasemipour, Raphael Gontijo Lopes, Burcu Karagol Ayan, Tim Salimans, et al. Photorealistic text-to-image diffusion models with deep language understanding. Advances in Neural Information Processing Systems (NeurIPS), 35:36479-36494, 2022.

---

Published as a conference paper at ICLR 2024
Shuai Shen, Wenliang Zhao, Zibin Meng, Wanhua Li, Zheng Zhu, Jie Zhou, and Jiwen Lu. Difftalk: Crafting diffusion models for generalized audio-driven portraits animation. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pp. 1982-1991, 2023.
Aliaksandr Siarohin, Stéphane Lathuilière, Sergey Tulyakov, Elisa Ricci, and Nieu Sebe. First order motion model for image animation. Advances in Neural Information Processing Systems (NeurIPS), 32, 2019.
Yang Song, Jascha Sohl-Dickstein, Diederik Kingma, Abhishek Kumar, Stefano Ermon, and Ben Poole. Score-based generative modeling through stochastic differential equations. In International Conference on Learning Representations (ICLR), 2021.
Michat Stypulkowski, Konstantinos Vougioukas, Sen He, Maciej Zieba, Stavros Petridis, and Maja Pantic. Diffused heads: Diffusion models beat gans on talking- face generation. arXiv preprint arXIv:2301.03396 2023.
Anni Tang, Tianyu He, Xu Tan, Jun Ling, Runnan Li, Sheng Zhao, Li Song, and Jiang Bian. Memories are one-to-many mapping alleviators in talking face generation. arXiv preprint arXiv:22/2.55005 2022.
Justus Thies, Mohamed Elgharib, Ayush Tewari, Christian Theobalt, and Matthias Nießner. Neural voice puppetry: Audio-driven facial reenactment. In European Conference on Computer Vision (ECCV), pp. 716-731. Springer, 2020.
Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Lukasz Kaiser, and Illia Polosukhin. Attention is all you need. Advances in Neural Information Processing Systems (NeurIPS), 30, 2017.
Duomin Wang, Yu Deng, Zixin Yin, Heung- Yeung Shum, and Baoyuan Wang. Progressive disentan- gled representation learning for fine-graine controllable talking head synthesis. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pp. 17979-17989, 2023.
S Wang, L Li, Y Ding, C Fan, and X Yu. Audio2head: Audio-driven one-shot talking-head generation with natural head motion. In International Joint Conference on Artificial Intelligence. IJCAI, 2021a.
Suzhen Wang, Lincheng Li, Yu Ding, and Xin Yu. One-shot talking face generation from single- speaker audio-visual correlation learning. In Proceedings of the AAAI Conference on Artificial 2022.
Intelligence, volume 36, pp.
Ting-Chun Wang, Arun Mallya, and Ming-Yo Yu Liu. One-shot free-view neural talking-head synthesis for video conferencing. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pp. 10039-10049, 2021b.
Errlll Wood, Tadas Baltrušaitis, Charlie Hewitt, Sebastian Dziadzio, Thomas J Cashman, and Jamie Shotton. Fake it till you make it: face amalysis in the wild using synthetic data alone. In Proceedings of the IEEE/CVF International Conference on Computer Vision (ICCV), pp. 368 3681-369,, 2021.
Zhentao Yu, Zixin Yin, Deyu Zhou, Duomin Wang, Finn Wong, and Baoyuan Wang. Talking head generation with probabilistic audio-to-visual diffusion priors. arXiv preprint arXiv:2212.04228, 2022.
Bowen Zhang, Chenyang Qi, Pan Zhang, Bo Zhang, HsiangTao Wu, Dong Chen, Qifeng Chen, Yong Wang, and Fang Wen. Metaportrait: cenntity-preserving talking head generation with fast personalized adaptation. In Proceedings of the IEEF/CVF Conference on Computer Vision and Puttern Recognition (CVPR), pp. 22096-22105, 2023a.
Richard Zhang, Phillip Isola, Alexei A Efros, Eli Shechtman, and Oliver Wang. The unreasonable effectiveness of deep features as a perceptual metric. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pp. 386-595, 2018.
12

---

Published as a conference paper at ICLR 2024
Wenxuan Zhang, Xiaodong Cun, Xuan Wang, Yong Zhang, Xi Shen, Yu Guo, Ying Shan, and Fei Wang. Sadtalker: Learning realistic 3d motion coefficients for stylized audio-driven single image talking face animation. In Proceedings of the IEEE/CVF Conference Computer Vision and Pattern Recognition (CVPR), pp. 8652-8661, 2023b.
Zhimeng Zhang, Lincheng Li, Yu Ding, and Changjie Fan. Flow-guided one-shot talking face generation with high-resolution audio-visual dataset. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pp. 3661-3670, 2021.
Weizhi Zhong, Chaowei Fang, Yinqi Cai, Pengxu Wei, Gangming Zhao, Liang Lin, and Guanbin Li. Identity-preseringg talking face generation with landmark and appearance priors. Proceedings of the IEEE/CVF Conference Computer Vision and Pattern Recognition (CVPR), pp. 9729-9738. 2023.
Hang Zhou, Yasheng Sun, Wayne Wu, Chen Change Loy, Xiaogang Wang, and Ziwei Liu. Pose- controllable talking face generation by implicitly modularized audio-visual representation. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pp. 4176 4186, 2021.
Yang Zhou, Xintong Han, Eli Shechtman, Jose Echevarria, Evangelos Kalogerakis, and Dingzeyu Li. Makelttalk: speaker-aware talking-head animation. ACM Transactions On Graphics (TOG), 39(6): 1-15, 2020.
13

---

Published as a conference paper at ICLR 2024
A DATA ENGINEERING
A.1 DATA FILTRATION
As introduced in Sec. 3 we collected large-scale talking avatar dataset which consists of 8.2K hours of videos and 16.9K unique speaker IDs. However, the raw videos are surrounded by noisy cases that are harmful to the model training, such as non-speaking clips and rapid head moves. To enable the desired information can be learned from data, we develop several automated filtration policies to improve the quality of the training data.
To accurately learn the motion of the lip of the individual, it should be clearly visible by the model. Therefore, we maintain the frontal orientation of the individual toward the camera consistent in video clip, and filter out frames with large deflections where the lips may be incomplete. Specifically, we calculate the clockwise angles formed by the positions of both eye corners in relation to the tip of the nose, using the tip of the nose the horizontal reference line. Ideally, the angle should measure 180 degrees, for which we establish a range around it. Frames that fall outside this range will be dropped.
To ensure the quality of the generation results, the facial movement in video clip should be smooth without rapid shaking. Therefore, we monitor the face positions between adjacent frames and ensure that there is no significant displacement in continuous timestamps. We calculate the movement of the key point and face rectangle detected by an open-source detector and limit the difference between two adjacent frames to pre-defined threshold. In addition. we crop frames to place the talking head at the center to make its position consistent across different videos.
To filter out corner cases where the lip movements and speech are not aligned, we detect and filter the frames where individuals are wearing masks or not speaking.
It is worth noting that the data requirements for the VAE and the diffusion model are different because the VAE model does not need to deal with the alignment between the speech and image, therefore we use loose threshold for the filtration policies for training the VAE model.
We execute the filtration policies frame- trame-by-framm by-frame for all raw videos, and retain the video segments with consecutive satisfactory frames longer than three seconds. The statistics of the filtered dataset are listed in Tab. [ We can find that majority of raw videos are dropped, which is necessary for the training of data-driven model according our preliminary experimental results, where the video quality generated by models trained on raw videos falls behind the one trained on filtered data.
A.2 SPEECH PROCESSING
For each obtained video clip, we extract its speech and normalize the speech to proper amplitude range. To reduce the background noise, we also apply a denoiser (Defossez et al., 2020) for each normalized speech clip. Since deep acoustic features have been found to be superior to traditional acoustic features like MFCC and mel-spectrogram (Baevski et al.] 2020), following previous practice (Du et al. (023), we leverage pre-trained wav2vec 2.0 (Baevski et al. 2020 to extract the speech feature from the speech.
B EXPERIMENTAL SETTINGS
B. IMPLEMENTATION DETAILS
The VAE consists of traditional convolutional residual blocks, with downsampling factors and 16 for appearance and motion encoder respectively. By changing the hidden size and number of layers in a block, we can control the size of the VAE model, and result small (80M parameters, dhidden 128, Rlayer 2), base (700M parameters, dnidden = 256, Ptayer = 4), and large (1.7B parameters, dhidden = 512, aaayer = 8) settings. The learning rate is set to 4.5 x e and keeps constant during training. We use Conformer (Gulati et al.) 2020) the backbone of the diffusion model. Similarly, we adjust the hidden size and the number of layers to obtain the speech- meotconoomotion
httpp::///gg com davi ski sking/
14

---

Published as a conference paper at ICLR 2024
Table 7: Quantitative comparisons of the GAIA small 11 VAE model (80M parameters) trained the VoxCeleb2 (Chung et al., 2018) dataset with previous video-driven baselines.
Seff-Reconstrruttoon
Cross-Reenactmet
Methods
FID4
LPIPS
PSNRt
AKD
MSI1
FIDL
AKD
MSI
FOMM
23.843
0.196
22.669
2.160
0.839
45.951
3.404
0.838
HcadGAN
21.499
0.278
18.555
2.990
0.835
90.746
5.964
0.788
face-vid2vid
18.604
0.184
23.681
2.195
0.813
28.093
3.630
0.853
GAIA (VoxCeleb2)
16.099
0.173
22.896
1.434
1.083
27.643
2.968
1.035
models with different scales: small (180M parameters, dhadden = 512, Mager = 6), base (600M parameters, dhidden 1280, Rlaee 12), and large (1.2B parameters, dhidden 2048, Rayer 12). The learning rate starts from 1.0 x e- and follows the inverse square root schedule. For both the VAE and the diffusion model, we adopt Adam (Kingma & Ba 2015) optimizer and train our models on 16 V100 GPUs. We use the resolution of 256 x 256 for all the settings.
B.2 SETTINGS FOR VIDEO-DRIVE EXPERIMENTS
For the reofssscontttuttion sctting, we choose the first frame of each video as the input to the appearance encoder, and the others driving frames whose landmarks extracted and fed the motion encoder. We test on all frames in the test set.
For the cross-reenaetment setting, follow previous works (Zhang et al. 2023a and randomly sample one frame from other videos as the appearance. To eliminate the effects of randomness, we rounds for each driving video. We generate 100 frames in each round for each video.
B.3 SETTINGS FOR SPEECH-DRIVEN DRIVEN EXPERIMENTS
During training, we randomly sample training pairs with length N from 125 to 250 to augment the training set for the speech-to-motion model (Du al.) (2023). For each video, use the first frame of each video as the reference image for all the methods.
C MORE EXPERIMENTAL RESULTS
Due the limited space in the main paper, we provide more experimental results for both video-driven and speech-driven settings in this section.
C.1 MORE VIDEO-DRIVEN RESULTS
In addition training our model on the dataset we proposed, we also train the small model (80M parameters) on the VoxCeleb2 (Chung et al. 2018) dataset which is utilized by previous baselines such as HeadGAN (Doukas et al.) (021) and face-vid2vis (Wang et al., (021bb to provide fair comparisons with them. The results are listed in Tab. 7 When trained the same dataset, GAIA still outperforms previous baselines on most metrics, showing the effectiveness of our model.
We provide qualitative results of video-driven self-reconstruction and cross- reenactment, and compare them with FOMM (Siarohin et al. 2019) and face-vid2vid (Wang et al., (2221)) in Fig. 4 For the self-reconstruction task which is relatively simple, both baselines and our model can achieve good results, while our model recovers more fine-grained details such as wrinkles and skin textures.
For the cross-reenactment setting, cross cross-identity reenactment in other words, model clearly outperforms baselines by dealing well with motion disentanglement and appearance reconstruction simultaneously.
15

---

Published as a conference paper at ICLR 2024
Source
Driving
FOMM
face-vid2vid
GAIA (Ours)
Figure 4: Qualitative examples of vidco-driven self-reconstrucciion (first row) and cross- reenactment (second and last rows) results from baselines and our GAIA VAE model.
Table 8: Quantitative comparisons of the GAIA framework with previous speech-driven methods. The subjective evaluation is rated at five grades (1-5) terms of overall naturalness (Nat.), lip-sync quality (Lip.), motion jittering (Jit.), visual quality (Vis.), and motion diversity (Mot.). the Syne-D score for real video is 8.548, which is close to ours. PD-FGC depends on extra driving videos to provide pose, expression and eye motions. We use the real (ground-truth) video as its driving video.
Subjective Evaluation
Objective Evaluation
Methods
Nat.
Lip-t
Jit_
Vis.
Mot.1
Sync-D4
MST
FIDJ
MakeliTalk
2.148
2.161
1.739
2.789
2.571
9.932
1.140
28.894
Audio2Head
2.355
3.278
2.014
2.494
3.298
8.508
0.635
28.607
PC-AVS
2.797
3.843
3.546
3.452
2.091
8.341
0.677
59.464
SadTalker
2.884
4.012
4.020
3.569
2.625
8.606
1.165
22.322
PD-FGC*
3.283
3.893
1.905
3.417
4.512
8.573
0.478
58.943
GAIA (Ours)
4.362
4.332
4.345
4.320
4.243
8.528t
1.181
22.924
C.2 MORE SPEECH-DRIVEN RESULTS
We provide full quantitative comparisons with MakeltTalk (Zhou et al. 2020). Audio2Head (Wang al. 2021a), a). PC-AVS (Zhou al.) (0021. SadTalker (Zhang al. 2023b). and PD-FGC (Wang et al. 2023) in Tab. g It can be observed that GAIA surpasses all the baselines by large margin in terms of subjective evaluation. The best MSI score demonstrates that GAIA generates videos with great motion stability. The Syne-D score of 8.528, which is close to the one of real video (8.548), illustrates that the generated videos have great lip synchronization.
We give more ablation studies for the proposed techniques in Tab. 9 All experiments are conducted based on the 700M VAE model and the 180M diffusion model. First, we study the conditioning mechanism for the speech-to- model generation: 1) we directly add both the speech feature and the reference motion latent z (i) to each block of the Conformer layer (Spe. Add. Ref. Add.); 2) in each cross-altention layer (Vaswani al. 2017 Rombach et al. 2022). the hidden sequence in the Conformer layer acts as the query, and both the speech feature and the reference motion latent 2nh used the key and value (Spe. Att. & Ref. Att.); the speech feature is used the key and value in each block (Spe. Att. & Ref. Add.). We also replace the Conformer backbone in the diffusion model
layer, while the reference motion latent zn (i) is directly added each
16

---

Published as a conference paper at ICLR 2024
Table More ablation studies on the proposed techniques. See Sec. C.2
for details.
Methods
Sync-D4
MSI
FID4
GAIA (700M 180M)
8.913
1.132
24.242
Spe. Add. & Ref. Add.
8.989
1.125
28.189
Spe. Att. & Ref. Att.
10.231
1.139
28.718
Spe. Att. & Ref. Add.
10.180
1.015
24.021
Transformerr
9.312
0.623
25.385
Ref. Puse
Ref. Image
Video 1
Video
Vidio
Figure 5: Examples of poseccontrollable talking avatar generation. We extract the head poses from the reference video (first row), and use it to control the generation of different identities. Note that in this demonstration, we only control the head poses, while the lip motion and facial expression are generated according the given speech, instead of the reference video.
with the Transformer Vaswani al. (017) (w/ Transformer). From the table, we can observe that, adding the speech feature to the Conformer block and cross-attending to the reference motion latent (GAIA) achieves the best performance in terms of all three metrics. We also conclude that replacing the Conformer with the Transformer leads to significant motion jittering as the MSI score drops lot.
D CONTROLLABLE TALKING AVATAR GENERATION
D.1 POSF-CONTROLLABLE TALKING AVATAR GENERATION
As introduced in Sec. 4.3 in addition to predicting the head pose from the speech, we also enable the model with costrolltoll generation. We implement it by replacing the estimated head pose with either a handcrafted design pose or one extracted from another video. In detail, Fig. (3a) is achieved by feeding the fixed pitch, yaw, and roll of head poses to the speech-to-motion model during generation. We also demonstrate the results of making generation with the head poses extracted from reference video in Fig. 5 It can be observed that GAIA can generate results that head poses consistent with the given one, while the lip motion is in line with the speech content.
D.2 FULLY CONTROLLABLE TALKING AVATAR GENERATION
Due to the controllability of the inverse diffuuson process, we can control the arbitrary facial attributes by editing the landmarks during generation. Specifically, we train a diffusion model to synthesize the coordinates of the facial landmarks. The landmarks that we want to edit are fixed to the given coordinates. Then we leave the model to generate the rest. This enables more flexible and fine- ane-grained control over the generated videos. In particular, we provide the examples in Fig, [b6] where all non-lip motion is aligned with the reference one, and the lip motion is in line with the speech content.
17

---

Published as a conference paper at ICLR 2024
E TEXT-INSTRUCTED AVATAR GENERATION
E.1 EXPERIMENTAL DETAILS
In general, the diffusion model a motion generator conditioned on speech, where the condition can be altered to other modalities flexibly. To show the generality of our framework, we consider textual motion instructions the condition of the diffusion model, to enable the text-instructed generation. Specifically, when provided with a single reference portrait image, the generation should follow textual instructions such as "please smile" or "turn your head left" to generate a video clip with the character performing the desired action.
We extract parallel data with text instructions and action videos from our dataset. We leverage the CC vI dataset (Hazirbas et al. (021)) which contains data with off-screen instructional speeches and action videos of the participant. We then extract the instructional text and match it with the corresponding video clips of each action with the timestamp annotations. As result, the text-instructed training set comprises of 28.8 hours videos and 24K textual instructional examples. We also modify the architecture of the diffusion model by substituting the speech feature with the textual semantic representations encoded by a pre-trained CLIP (Radford et al. 2021) text encoder.
E.2 RESULTS
To evaluate performance of the text-instructed generation, we randomly select 10 portraits that do not appear in the training set. For each of them, we provide 10 distinct textual instructions. Given the subjective nature of this task, we recruit volunteers with relevant professional knowledge to rate the generation results between 0 5 from three different perspectives: accuracy of instruction following, video quality, and identity preservation.
The three scores over the generated videos 4.21, 4.41, and 4.64 respectively, showing that the text-instructed model demonstrates strong abilities to generate actions that align with instructions, and the generated videos exhibit outstanding quality being natural and fluent. The text-instructed extension demonstrates the strong generality of the proposed GAIA framework.
18


Reviews:
[{"review_id": "nhkezJyr7A", "reviewer": "Reviewer_1S6P", "summary": "The authors propose to use a generative latent diffusion model to address the problem of talking head synthesis from audio and a single photo. The pipeline consists of a variational autoencoder that encodes the video frames into appearance and motion latent representations and a diffusion model that is trained to predict the pre-trained motion latent from audio and pose conditioning. The authors also propose to use a data filtering approach to remove the noisy samples from the dataset to achieve high-quality results. The experimental evaluation is quite extensive and includes audio, head pose, and text-driven examples.\n\nUPD: The rebuttal has addressed my concerns, and the results look genuinely impressive.", "questions": "- Please address the concerns mentioned in the weaknesses\n- Could the authors clarify if they plan to release the filtered dataset and the pre-trained models?", "limitations": "", "rating": 8, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "- Impressive quality of the results in terms of both lipsync and visual quality\n- The model's design is straightforward yet evidently effective\n- The paper is well-written and the evaluation is pretty extensive", "weaknesses": "- Missing evaluation of disentanglement between appearance and pose latent codes, i.e., cross-reenactment with the motion codes extracted from the image of a different identity.\n- Missing discussion of the related works, such as [1, 2], that explored the concept of pose-identity disentanglement for talking head synthesis before this work.\n- As far as I can tell, the proposed method and the baselines were trained on different datasets. The resulting comparison evaluates the proposed framework _and_ the dataset at the same time. A comparison should include the experiments where base methods are trained on the same dataset, and the proposed method is trained on unprocessed datasets used in previous works.\n- Comparison of the inference time between the compared methods is not provided. I would also argue that some baselines, such as SadTalker, can be substantially improved, given the computational budget of the proposed method that runs the diffusion model for every time step. Ex., with the fine-tuning of the model given the source frame.\n\n[1] Burkov et al., \"Neural Head Reenactment with Latent Pose Descriptors\", CVPR 2020\n[2] Drobyshev et al., \"MegaPortraits: One-shot Megapixel Neural Head Avatars\", ACMMM 2022", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "3NP8TiZs8K", "reviewer": "Reviewer_ft55", "summary": "The paper proposes an audio-driven talking head synthesis model named GAIA that is an end-to-end trainable data-driven solution. The model has two main stages 1. disentanglement of motion and appearance with VAE 2. speech-to-motion generation based on diffusion model. Also, the paper proposes a new talking head dataset with 8.2K hours of video and 16.9K unique speakers.", "questions": "1. In section 3, what does 'we collect High Definition Talking Face Dataset (HDTF) (Zhang et al., 2021) and Casual Conversation datasets v1&v2 (CC v1&v2) (Hazirbas et al., 2021; Porgali et al., 2023)' and 'we also collect a large-scale internal talking avatar dataset named AVENA' mean? Does it mean you collect those datasets you use their sample in your dataset or you use their samples in your training but they are not in your dataset? From the supplementary material, my understanding is you collect AVENA and use samples from other datasets (HDTF, CC v1, and v2) in the training of your model. Could you please elaborate and make it clear?\n\n2. Why do you have a discussion section at the end of Section 4 Model? I think it makes more sense in/after experiments. So, you can consider reorganizing the manuscript to have a better flow and complete discussion.\n\n3. 3. I am not sure the model can be named as zero-shot as it requires one shot for unseen faces. So, could you elaborate on the following '... generates a talking video of an unseen speaker with one portrait image ...'?\n\n3. Ethical consideration is left in Appendix F. However, for this study ethical consideration is important. So, you might consider putting it into the main manuscript to give necessary importance if possible.", "limitations": "", "rating": 6, "confidence": 5, "soundness": 3, "presentation": 3, "contribution": 2, "strengths": "The manuscript proposes a new dataset.\n\nThe writing is supported by equations and well-drawn figures that make the explanation clear.\n\nAlthough the experiments with existing models are not enough (see weaknesses), the ablation study is rich and increases the overall quality.", "weaknesses": "The gap/ limitations of 3DMM-based models are found and addressed well by proposing an end-to-end trainable model. I am not sure it is novel enough as the other end-to-end trainable talking face synthesis models are not discussed enough.\n\nThe experiments are limited, especially comparison with end-to-end trainable models not provided. I suggest enriching the benchmarking with other existing models such as  PC-AVS and PD-FGC as they are also end-to-end trainable models. \n\nAlthough the writing quality is decent, it is hard to follow as it refers to other sections frequently and other issues (see Questions 1 and 2).", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "wUn0m5sjQC", "reviewer": "Reviewer_Lkke", "summary": "This paper proposes a data driven approach for generation of 2D avatars. The method disentangles motion and appearance and uses a diffusion model to allow motion generation conditioned on pose and speech data.", "questions": "1) will the dataset be shared as part of this submission?\n2) It would be interesting to see an ablation on training data size to assess whether there are benefits from scaling data further.\n3) How does randomness from the diffusion model affect generations?", "limitations": "", "rating": 6, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "1) The method is conceptually simple and sensible.\n2) It is shown to scale well in terms of model size and as a self-supervised method can utilize readily available training data at scale.\n2) Method requires very few pretrained components.\n3) Evaluation includes user study which is always good for addressing output quality.\n4) Method is highly flexible and allows a high degree of control from pose, facial attributes and text.", "weaknesses": "1) A comparison with https://arxiv.org/pdf/2012.08261.pdf for video driven is critically missing as a recently proposed SOTA method. In their paper they show improvements compared to face-vid2vid and FOMM which are used as baselines here and they provide a pretrained checkpoint.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "X340xyYxq6", "reviewer": "Reviewer_tNS3", "summary": "This work aims to generate talking avatars with two separate modules: 1. first disentangle motion and appearance; 2 then generate head motions in accordance with speech.  As the proposed method does not utilize 3DMMs, the proposed data-driven method is promising to generate talking avatars with better diversity and naturalness.  The reported experiments show better quantitative results and better visual quality.", "questions": "1. is it possible to drive cartoon characters (humanoid or non-humanoid)?\n\n2. will the proposed method and the established dataset be publically available?", "limitations": "", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "1. This work establishes a large-scale talking avatar dataset for data-driven talking avatar generation.\n\n2. The proposed data-driven method could achieve taking avatar generation with superior performance on naturalness, diversity, lip-sync quality, and visual quality.  \n\n3. The proposed method is scalable, and the authors conduct experiments that show the larger model is employed, the better performance could be achieved.\n\n4. The authors show that the proposed method could support many related applications, such as controllable talking avatar generation and text-driven video generation.", "weaknesses": "The authors did not provide a large number of generation examples. I hope it is possible to see more visual results.\nFor example, \n1. long videos for the generated talking avatars;\n\n2. different reference video/frame but the same driving video;\n\n3. different driving video but the same reference video/frame.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (poster)