VASA-1: Lifelike Audio-Driven Talking Faces Generated in Real Time
Authors: Sicheng Xu
Guojun Chen
Yu-Xiao Guo
Jiaolong Yang
Chong Li
Zhenyu Zang
Yizhong Zhang
Xin Tong
Baining Guo
Venue: NeurIPS 2024 oral (2024)
Keywords: talking face; face video generation;
Primary area: generative_models

Abstract: We introduce VASA, a framework for generating lifelike talking faces with appealing visual affective skills (VAS) given a single static image and a speech audio clip. Our premiere model, VASA-1, is capable of not only generating lip movements that are exquisitely synchronized with the audio, but also producing a large spectrum of facial nuances and natural head motions that contribute to the perception of authenticity and liveliness. 
The core innovations include a diffusion-based holistic facial dynamics and head movement generation model that works in a face latent space, and the development of such an expressive and disentangled face latent space using videos.
Through extensive experiments including evaluation on a set of new metrics, we show that our method significantly outperforms previous methods along various dimensions comprehensively. Our method delivers high video quality with realistic facial and head dynamics and also supports the online generation of 512$\times$512 videos at up to 40 FPS with negligible starting latency.
It paves the way for real-time engagements with lifelike avatars that emulate human conversational behaviors.

VASA-1: Lifelike Audio-Driven Talking Faces Generated in Real Time
Sicheng Xu" Microsoft Research Asia sichngxuemi crwsoft. com
Guojun Chen" Microsoft Research Asia guoch@microsofe
Yu-Xiao Guo" Microsoft Research Asia songuuuiicooooftto com
Jiaolong Yang"*
Chong Li
Zhenyu Zang Microsoft Research Asia
Microsoft Research Asia j aiaoyan@micc croooooof..rooo.....mmmooooooo.e crosoft. com
Microsoft Research Asia chol@micnp ammicrostt.ccmmmmm com
cennyuzangmmicrooofff... com
Yizhong Zhang
Xin Tong
Baining Guo
Microsoft Research Asia y yizzhan@microo crosoft. com
Microsoft Research Asia xtonggmicrosoft..cm com
Microsoft Research Asia bainguo@micren www.oftt.ioooooooomoommmmoccooce com
Abstract
We introduce VASA, framework for generating lifelike talking faces with ap- pealing visual affective skills (VAS) given a single static image and a speech audio clip. Our premiere model, VASA-I, is capable not only generating lip movements that are exquisitely synchronized with the audio, but also produc- ing a large spectrum of facial nuances and natural head motions that contribute to the perception of authenticity and liveliness. The core innovations include a diffusion- holistic facial dynamics and head movement generation model that works in face latent space, and the development of such expressive and disentangled face latent space using videos. Through extensive experiments including evaluation set of new metries, show that our method signifi- cantly outperforms previous methods along various dimensions comprehensively. Our method delivers high video quality with realistic facial and head dynam- ics and also supports the online generation of 512x512 videos at up to 40 FPS with negligible starting latency. It paves the way for real-time engagements with lifelike avatars that emulate human conversational behaviors. Project webpage: https:: wttp:://wwwmmccoo...t...mmmmm www.sofotoo...mommmmmmmm com/ www./nn htt.:////...////eeeeeeeppp/////a
Introduction
In the realm of multimedia and communication, the human face is not just visage but dynamic canvas, where every subtle movement and expression can articulate emotions, convey unspoken messages, and foster empathetic connections. The emergence of Al-generated talking faces offers a window into a future where technology amplifies the richness of human-humaa and human-Al interactions. Such technology holds the promise of enriching digital communication [64 33), increasing accessibility for those with communicative impairments [I, transforming education methods with interactive AI tutoring [8, and providing therapeutic support and social interaction in healthcare (3133
As one step towards achieving such capabilities, our work introduces VASA-1, new method that can produce audio-generated talking faces with a high level of realism and liveliness. Given a static
Equal contributions. t; Corresponding author. See the contribution statement section for contributions.
38th Conference on Neural Information Processing Systems (NcurIPS 2024).

---

Audio clip +
|| |+||
VASA.1
Single image
Control signals
Figure Given a single portrait image, speech audio clip, and optionally set of other control signals, approach produces high-quality lifelike talking face video of 512x 512 resolution at up to 40 FPS. The method is generic and robust, and the generated talking faces can faithfully mimic human facial expressions and head movements, reaching high level of realism and liveliness. (All the photorealistic portrait images in this paper are virtual, non-existing identities generated by 30 5). See our project page for the generated video samples with audios.)
face image of an arbitrary individual, alongside a speech audio clip from any person, our approach is capable generating hyperr-ealisttc talking face video efficiently. This video not only features lip movements that are meticulously synchronized with the audio input but also exhibits wide range of natural, human-like facial dynamics and head movements.
Creating talking faces from audio has attracted significant attention in recent years with numerous 63 61
approaches proposed 777 39 75 51
74 36 26). However, existing techniques are still far from achieving the authenticity of natural talking faces. Current research has predominantly focused on the precision creation of expressive facial dynamics and the subtle nuances of lifelike facial behavior remain largely neglected. This results in generated faces that seem rigid and unconvincing. Additionally, natural head movements also play a vital role in enhancing the perception of realism. Although recent studies have attempted to simulate realistic head motions the generated animations and the genuine human movement patterns.
25
62
lip synchronization with promising accuracy obtained [39, 61. The
71 74, there remains a sizable gap between
Another important factor is the efficiency of generation, which plays a pivotal role in real-time applications such as live communication. While image and video diffusion techniques have brought remarkable advancements
as well as the broader video generation field (6, 99, the substantial computation demands have limited their practicality for interactive systems. A critical need exists for optimized algorithms that can bridge the gap between high-quality video synthesis and the low-latency requirements of real-time applications.
talking face generation 22
49 551
Given the limitations of existing methods, this work develops efficient yet powerful audio- conditioned generative model that works in the latent space of head and facial movements. Different from prior works, we train a Diffusion Transformer model on the latent space of holistic facial dynamics as well as head movements. We consider all possible facial dynamics including lip motion, (non-lip) expression, eye gaze and blinking, among others model its probabilistic distribution in unified manner. By contrast, existing methods often apply separate models for different factors, even with interleaved regressive and generative formulations for them [62] 76 71 60 (74]. Our holistic facial dynamics modeling, together with the jointly learned head motion patterns, leads to the generation of a diverse array of lifelike and emotive talking behaviors. Furthermore, we incorporate a set of optional conditioning signals such main gaze direction, head distance, and emotion offset into the learning process. This makes the generative modeling of complex distribution more tractable and increases the generation controllability.
single latent variable and

---

To achieve our goal, another challenge lies in constructing the latent space for the aforementioned holistic facial dynamics and gathering the data for the diffusion model training. Beyond facial and head movements, a human face image contains other factors such as identity and appearance. In this work, we seek to build a proper latent space for human face using a large volume of face videos. Our aim is for the face latent space to possess both total state of disentanglemen between facial dynamics and other factors, as well as a high degree of expressiveness to model rich facial appearance details and dynamic nuances. We base our method the 3D-aided representation [64] [9] which was proven to be expressive, and equip it with a collection of newly-designed loss functions critical to effective disentanglement. Without the new designs face generation, especially the liveliness with nuanced emotions. Trained on face videos in an self-supervised weakly-superrised manner, our encoder can produce well-disentangled factors including 3D appearance, identity, head pose and holistic facial dynamics, and the decoder can generate high quality faces following the given latent codes.
never reach a high quality of talking
VASA-1 has collectively advanced the realism of lip-audio synchronization, facial dynamics, and head movement to new heights. Coupled with high image generation quality and efficient running speed, we achieved real-time talking faces that are realistic and lifelike. Through detailed evaluations, we show that our method significantly outperforms existing methods on set of metrics, including novel dita-n metric called Contrastive Audio and Pose Pretraining (CAPP) for measuring the audio-pose alignment and pose variation intensity score that is related to the vividness of head motion. We believe VASA-1 brings us closer to a future where digital Al avatars can engage with us in ways that are as natural and intuitive as interactions with real humans, demonstrating appealing visual affective skills for more dynamic and empathetic information exchange.
2 Related Work
Disentangled face representation learning. The representation of facial images through disen- tangled variables has been extensively studied by previous works. Some methods utilize sparse keypoints 44 72 or 3D face models [4 22 73 to explicitly characterize facial dynamics and other properties, but these can suffer from issues such as inaccurate reconstructions or limited expressive capabilities. There are also many works dedicated to learning disentangled representations within a latent space. A common approach involves separating faces into identity and non- con-idttttty compo-
nents, then recombining them across different frames, either in a 2D [1 76 34 77, 37 60 (54] or 3D context [64 8 The main challenge faced by these methods is the effective disentangiemen of various factors while still achieving expressive representations of all static and dynamic facial attributes, which is addressed in this work.
Audio-driven talking face generation. Talking face video generation from audio inputs has been a long- standing task in computer vision and graphics. Early works have focused on synthesizing only the lips, achieved by mapping audio signals directly to lip movements while leaving other facial attributes unchanged [2] 39 77,
[3. More recent efforts have expanded the scope include a broader array of facial expressions and head movements derived from audio inputs. For instance, the method of [74] separates the generation targets into different categories, including lip-only 3DMM coefficients, eye blinks, and head poses. (71) proposed to decompose lip and non-lip features on the top of the expression latent from (76). Both 774 and [71] regress lip-related representations directly from audio features and model other attributes in a probabilistic manner. In contrast to these approaches, our method generates comprehensive facial dynamics and head poses from audio along with other control signals. This approach differs from the trend of further disentanglement, seeking instead to create more holistic and integrated outputs.
have led significant employed the adversar- have leveraged diffusion or ruto-regressive models to capture diverse video distributions. Recently, several works concurrent to us (55 65) have adapted video diffusion techniques to audio-driven talking face generation, achieving promising results despite the slow training and inference speeds. In contrast, our method is able to not only generating high-quality results but also achieve real-time efficiency - a metric crucial to officiincy-demandee applications such as live communication.
Video generation. Recent advances in generative models 10
27
48
progress in video generation. Earlier video generation approaches
59
36
46
ial learning [24] framework, while more recent methods (69
7
23
32
3

---

3 Method
Overall framework. As illustrated in Fig. D our method takes single face image, optional control signals, and a speech audio clip to produce a realistic talking face video. Instead of generating video frames directly, we generate holistic facial dynamics and head motion in the latent space conditioned on audio and other signals. To achieve this, we start by constructing a face latent space and training the face encoder and decoder. An expressive and disentangled face latent learning framework crafted and trained on real-life face videos. Then we train a simple yet powerful Diffusion Transformer to model the motion distribution and generate the motion latent codes in the test time given audio and other conditions.
3.1 Expressive and Disentangled Face Latent Space Construction
Given a corpus of unlabeled talking face videos, we aim to build latent space for human face with high degrees of disentanglement and expressiveness. The disentanglement enables effective generative modeling of the human head and holistic facial behaviors on massive videos, irrespective of the subject identities. It also enables disentangled factor control of the output which is desirable in many applications. Existing methods fall short of either expressiveness [ 42 60) or disentan- glement [64] 19, 73 or both. The expressiveness of facial appearance and dynamic movements, on the other hand, ensures that the decoder output high quality videos with rich facial details and the latent generator is able to capture nuanced facial dynamics.
To achieve this, we base our model on the 3D-aid face reenactment framework from 64 19 The 3D appearance feature volume can better characterize the appearance details in 3D compared to 2D feature maps. The explicit 3D feature warping is also powerful in modeling 3D head and facial movements. Specifically, we decompose facial image into canonical 3D appearance volume Vapp, an identity code 2 3D head pose g and facial dynamics code zdyn, Each of them extracted from a face image by an independent encoder, except that Vapp is constructed by first extracting a posed 3D volume followed by rigid and non-rigid 3D warping to the canonical volume, as done in [1]. A single decoder D takes these latent variables as input and reconstructs the face image, where similar warping fields in the inverse direction are first applied to Vepp to get the posed appearance volume. Readers are referred to 19 for more details of this architecture.
To learn the disentangled latent space, the core idea is to construct image reconstruction loss by swapping latent variables between different images in videos. Our basic loss functions are adapted from 19.. However, we idenrified the poor disentanglement between facial dynamics and head pose using the original losses. The disentanglement between identity and motions is also imperfect. Therefore, we introduce several additional losses crucial to achieve our goal. Inspired by (37], we add pairwise head pose and facial dynamics transfer loss to improve their disentanglement, Let I, and I, be two frames randomly sampled from the same vidco. We extract their latent variables using the encoders, and transfer I, 's head pose onto I, as I Ii,epar D(V .zzz pose .z) Zdy) and I, lss facial motion onto I as L D(V 2, ,2, pose ,2,"m) ). The discrepancy loss loonsist between
L,s and i 1, is subsequently minimized. To reinforce the disentanglement between identity and motions, we add a face identity similarity loss lerass id for the cross-identity pose and facial motion transfer results. Let I, and L be the video frames of two different subjects, we can transfer the motions of I onto I, and obtain I
5,2y youe dyn D(Vapp zid 2ose Zz dyn). Then, a cosine similarity
16 extracted from I, and i a.xtor = is applied. As we'll show in the experiments, our new loss function deigns are crucial to achieve an effective factor disentanglement and facilitate the high-quality, lifelike talking face generation.
loss between the deep face identity features
3.2 Holistic Facial Dynamics Generation with Diffusion Transformer
Given the constructed face latent space and trained encoders, we can extract the facial dynamics and head movements from real-life talking face videos and train a generative model. Crucially, we consider identity-agnostic holistic facial dynamics generation (HFDG), where our learned latent codes represent all facial movements such as lip motion, (non-lip) expression, and eye gaze and blinking. This is in contrast to existing methods that apply separate models for different factors 60 74). Furthermore, previous
with interleaved regression and generative formulations [62] 76 71

---

audio featurs
Transformer blocks
*N
encoda
Eaudio
App, volume
other conditonc
single
oudio featurr
Prev.
encode
Embs
desojs
Add nolte
conditions
generoted video framns
motion labents
diflusion precess
nolse
Tframes
Training pipaline for motion latent diffusion
Test pipaline for audio to video generation
Figure Our holistic facial dynamics and head pose generation framework with diffusion transformer.
methods often train on limited number of identities 74
68 21 and cannot model the wide range
of motion patterns of different humans, especially given an expressive motion latent space.
In this work, we utilize diffusion models for audio-conditioned HFDG and train on massive talking face videos from large number of identities. In particular, we apply transformer architecture [58] 38
52
for our sequence generation task. Figure showws an overview of our HFDG framework.
Formally, a motion sequence extracted from a video clip is defined as X = {[z! Z; ""
1,... W. Given its accompanying audio clip a, we extract the synchronized audio features [poudio], 1 for which we use a pretrained feature extractor Wav2Vec2 [3]
Diffusion formulation. Diffusion models define two Markov chains 27 48 the forward chain
progressively adds Gaussian noise to the target data, while the reverse chain iteratively restores the raw signal from noise. Following the denoising score matching objective [48] define the simplified loss function as
E1-,1 X2, (x)
H(X', t, C)I
where denotes the time step, X = X is the raw motion latent sequence, and X' is the noisy inputs generated by the diffusion forward process ( NX -I-BBX-- 3,I). H. is our transformer network which predicts the raw signal itself instead of noise. C is the condition signal, be described next.
Conditioning signals. The primary condition signal for our audio-driven motion generation task is the audio feature sequence A. We also incorporate several additional signals, which not only make the generative modeling more tractable but also increase the generation controllability.
Specifically, consider the main eye gaze direction g, head-to head-to-cament to-cemera distance d, and emotion offset e. The main gaze direction, g = (8, ø), is defined by a vector in spherical coordinates. specifies the focused direction of the generated talking face. We extract g for the training video clips using 22 on each frame followed by a simple histogram-based clustering algorithm. The head distance d is normalized scalar controlling the distance between the face and the virtual camera, which affects the face scale in the generated face video. We obtain this scale label for the training videos using [17). The emotion offset modulates the depicted emotion the talking face. Note that emotion is often intrinsically linked to and can be largely inferred from audio; hence, e serves only as a global offiset added to enhance or moderately alter the emotion when required. It is not designed to achieve total emotion shift during inference or produce emotions incongruent with the input audio. In practice, we use the averaged emotion coefficients extracted by as our emotion signal.
In order to achieve a seamless transition between adjacent windows, we incorporate the last K frames of the audio feature and generated motions from the previous window the condition of the current one. To summarize, our input condition can be denoted as C = =x Pre, ; A, g. d, e). All conditions concatenated with noise along the temporal dimension as the input the transformer.
Classifier-free guidance (CFG) (28). In the training stage, we randomly drop each of the input conditions. During inference, apply
x (1+
Σ Xe) H(X', C) Σn do H(X', t, Cla=e)
5

---

where de is the CFG scale for condition c. Cl denotes that the condition c is replaced with 0. During training,
drop probability of 0.1 for each condition except for X and pre for which we use 0.5. This is to ensure the model can well handle the first window with no preceding audio and motions (i.e., set to (f). We also randomly drop the last few frames of A to ensure robust motion generation for audio sequences shorter than the window length.
3.3 Talking Face Video Generation
At inference time, given arbitrary face image and audio clip, we first extract the 3D appear- ance volume Vap and identity code zd using our trained face encoders. Then, we extract the audio features, split them into segments of length W, and generate the head and facial motion sequences (X [[z p z; [ one by one in a Hiding-window manner using our trained diffusion transformer H. The final video can be generated subsequently using our trained decoder.
4 Experiments
Implementation details. For face latent space learning, we use the public VoxCeleb2 dataset from
[4] which contains talking face videos from about 6K subjects. We reprocess the dataset and discard the clips with multiple individuals and those of low quality using the method of [50]. For motion latent generation, we use an 8-layer transformer encoder with an embedding dim 512 and head number as our diffusion network. The model is trained on VoxCeleb2 [14] and another high-resolution talk video dataset collected by us, which contains about 3.5K subjects. our default setup, the model uses forward-facing main gaze condition, an average head distance of all training videos, and an empty emotion offset condition. The CFG parameters are set to AA 0.5 and Ag = 1.0, and 50 sampling steps are used. Our face latent model takes around 7 days of training on a 4 NVIDIA RTX A6000 GPUs workstation, and the diffusion transformer takes around days. The total data used for training comprises approximately 500K clips, each lasting between 2 to 10 seconds. The parameter counts of our 3D-aided face latent model and diffusion transformer model are about 200M and 29M respectively.
Evaluation benchmarks. We evaluate method using two datasets. The first is subset of VoxCeleb2 [4]. We randomly selected 46 subjects from the test split of VoxCeleb2 and randomly sampled 10 video clips for each subject, resulting in total of 460 clips. These video clips are about 5~115 seconds long (80% are less than 10 seconds), with most of the content being interviews and news reports. To further evaluate of vocal variations, we further collected 32 one-minute clips of 17 individuals. These videos are predominantly sourced from online coaching sessions and educational lectures and the talking styles are considerably more diverse than VoxCeleh2. We refer to this dataset as OneMin-32.
method under long speech generation with wider range
Inference speed. Our method generates video frames of 512x512 size at 45fps in the offline batch processing mode, and can support up 40fps in the online streaming mode with preceding latency of only 170ms, evaluated on a desktop PC with single NVIDIA RTX 4090 GPU.
4.1 Quantitative Evaluation
Evaluation metries. We use the following metrics for quantitative evaluation of our generated lip movement, head pose and overall video quality, including a new data-driven audio-pose synchroniza- tion metric trained in way similar to CLIP 44
Audio-lip synchronization. We use pretrained audio-lip synchronization network, i.e., Sync- Net (15), to assess the alignment of the input audio with the generated lip movements in videos. Specifically, we compute the confidence score and feature distance as Sc and Sp respectively. Higher Sc and lower Sp indicate better audio-lip synchronization quality in general.
Audio-pose aligument. Measuring the alignment between the generated head poses and input audio is not trivial and there are no well-established metrics. A few recent studies [74] 52 employed the Beat Align Score (45) to evaluate audio-pose alignment. However, this metric not optimal because the concept of a "beat" in the context of natural speech and human head motion is ambiguous. this work, we introduce a new data-driven metric called Contrastive Audio and Pose Pretraining (CAPP) score. Inspired by CLIP [40), we jointly train pose

---

Table I: Quantitative comparison with previous methods on two benchmarks.
VoxCeleb2
OneMin-32
Sc 1
Sn 4
CAPP1
AP
Sc t
Sp 4
CAPPT
AP
FVD25 +
MakelrTalk
4.176
13.513
-0.051
0.210
-0.123
14.340
0.002
0.190
304.83
Audio2Head
6.172
8,470
0.246
0.260
5,992
8.211
0.205
0.239
209.77
SadTalker
5.843
8.813
0.441
0.275
5,501
8,850
0,383
0.252
214.51
Ours
8.841
6.312
0.468
0.304
7.957
6.635
0.465
0.316
105.88
Ours 110% dates
8.818
6,298
0.457
0,229
7,990
6.645
0.441
0.229
147.401
Real video
7.640
7.189
0.588
0,505
7.192
7.254
0.559
0.405
29.25
sequence encoder and an audio sequence encoder and predict whether the input pose sequence and audio are paired. The audio encoder is initialized from pretrained Wav2Vec2 network B and the pose encoder is randomly initialized 6-layer transformer network. The input window size is 3 seconds. Our CAPP model is trained on 2K hours of real-life audio and pose sequences, and demonstrates robust capability assess the degree of synchronization between audio inputs and generate poses (see Sec. 4.3)
Pose variation intensity. We further define a pose variation intensity score A which is the average of the pose angle differences between adjacent frames. Averaged over all generated frames, AP provides an indication of the overall head motion intensity generated by a method.
Video quality. Following previous video generation works 669 46], we use the Fréchet Video Distance (FVD) [57] to evaluate the generated video quality. We compute the FVD metric using sequences of 25 consecutive frames, at resolution of 224x224.
Compared methods. We compare our method with there existing audio-driven talking face genera- tion methods: MakelfTalk 777,, Audio2Head 62 and SadTalker 77].
Main results. For each audio input, we generate a single video for deterministic approaches, i.e., MakeItTalk and Audio2Head. For SadTalker and our method, we sample three videos for each audio and average the computed metrics. Since different pose representations are used by these methods, we re-extract the head poses from the generated frames to compute the pose-related metrics (i.e.. CAPP and AP). For the FVD metric, we use 2K 25-frame video clips of both the real videos and generated ones. For reference purpose, we also report the evaluated metrics of real videos.
Table I presents the results on the VoxCeleb2 and OneMin-32 benchmarks. Note that we did not evaluate the FVD on VoxCeleb2 as its video quality is varied and often low. On both benchmarks, our method achieves the best results among all methods on all evaluated metrics. In terms of audio-lip synchronization scores (Sc and Sp), our method outperforms all others by wide margin. Note that our method yields better scores than real videos, which is due to effect of the audio CFG (see Sec. 4.3). Our generated poses are better aligned with the audios especially on the OneMin-32 benchmark, as reflected by the CAPP scores. The head movements also exhibit the highest intensity according to AP, although there's 's still a gap to the intensity of real videos. Our FVD score significantly lower than others, demonstrating the much higher video quality and realism of our results.
4.2 Qualitative Evaluation
Visual results. Figure I presents
representative audio-driven talking face generation results of our method. Visually inspected, our method can generate high-quality video frames with vivid facial emotions. Moreover, it can generate human-like conversational behaviors, including sporadic shifts in eye gaze during speech and contemplation, as well as the natural and variable rhythm of eye blinking, among other nuances. We highly recommend that readers view our video results in the supplementary material to fully perceive the capabilities and outpul quality of our method.
Generation controllability. Figure shows our generated results under different control signals including main eye gaze, head distance, and emotion offset. Our model can well interpret these signals and produce talking face results that closely adhere to these specified parameters.
Disentanglement of face latents. Figure A. shows that when applying the same motion latent sequences onto different subjects, our method effectively maintains both the distinct facial movements and the unique facial identities. This indicates the efficacy of our method in disentangling identity and motion. Figure A.2 further illustrates the effective disentanglement between head pose and facial
7

---

Gaze Control
Distance Control
Emotion Control
Figure 3: Generated talking faces under different control signals. Top results under different main gaze direction condition (forward-facils facindfaainn leftwards, rightwards, and upwards, respectively). Middle row: results under different head distances (from far near). Bottom row: results under different emotion offset (neutral, happy, angry and surprised, respectively).
dynamics. By holding one aspect constant and changing the other, the resulting images faithfully reflect the intended head and facial motions without interference.
-uito--dist generation. Our method exhibits the capability to handle photo and audio inputs that fall outside the training distribution, such artistic photos, singing audio clips, and non-English speech, as illustrated in Figure A.3
Comparison with other methods. Some visual examples from different methods are presented in Figure A.5 A.6 Our method outperforms the others in terms of the precise audio-lip synchronization and delivers much more vivid and natural facial dynamics and head movements.
4.3 Analysis and Ablation Study
CAPP metric. We analyze the effectiveness of our proposed CAPP metric in measuring the alignment between audio and head pose.
First, we study its sensitivity to temporal shifting by manually introducing frame offsets to ground-truth audio-pose pairs. We extract 3-second clip segments from the VoxCeleb2 test split, yielding approximately 2.1K audio-pose pairs. The average CAPP score for these pairs 0.608, as shown in Table Manual frame shifts lead a rapid decline in CAPP scores, approaching This indicates a robust correlation between CAPP scores and nudio-head pose alignment.
Table 2: CAPP under frame shifting
0
=1
#2
+3
0.608
0.462
0.206
0.069
0.082
for shifts larger than two frames.
We further investigate the effect of head movement intensity on CAPP by manually scaling the pose dif- ferences between consecutive frames using various factors. Tsole that altering movement inten- sity negatively impacts the CAPP scores, demonstrating CAPP can assess the alignment of audio and pose in terms of their intensity. However, this sensitivity to intensity appears less pronounced than that temporal misalignment.
Table 3: CAPP under pose variation scaling
x0.2
x0.5
x1.0
x 1.5
x3.0
0.368
0.584
0.608
0.587
0.505

---

Table 4: Ablation study of the audio and main gaze CFG scales as well as the sampling steps. Eg
denotes the average angular error of main gaze directions and & the average head distance error.
Sc 1
Sp
CAPPT
AP
FVD25
Ee,
E,
A 0.0, Ag
7.087
7.391
0.414
0.291
117.425
5.730
0.004
AA = 0.0, A 1.0
7.134
7.345
0.421
0.290
116.547
5.329
0.004
AA = 0.0, AS 2.0
7.108
7.386
0.414
0.298
117.784
5.064
0.005
ÀA = 0.5, A = 1.0
7,957
6.635
0.465
0.316
105,884
5.253
0.005
AA 1.0, AS
8.218
6.437
0.474
0.342
104.886
5.333
0.005
AA 2.0, ÀR 1.
8,295
6.397
0.455
0.395
104,293
5.531
0.005
AA = 0.5, Ag = 1.0 (steps= 10)
8,293
6,363
0,523
0.243
117.060
5,469
0.006
Real video
7.192
7.254
0.559
0.405
29.244
Dyn. Source
Target
Ours w/o consist
Ours
Figure 4: Ablation study on loss function Leonsist for disentangled latent space learning. We generate the results by only transferring the facial dynamies from source to target with head pose unchanged. leonsist is crucial for decoupling subtle yet important facial dynamics from head pose.
CFG seales for diffusion model. The CFG strategy 28 diffusion models can attain a trade-off between sample quality and diversity. Here evaluate the choice of the CFG scales for the audio and main gaze conditions (i.e., AA and >g in Eq.2) in our model.
As shown in Table as we increase the value of Ag. the accuracy of gaze control improves. Increasing the audio CFG scale to XA 0.5 significantly enhances the performance of lip-audio alignment (Se and Sp), pose-audio alignment (CAPP), and pose variation intensity (AP). With positive audio CFG, the lip-audio alignment scores even surpass those evaluated real videos (the results without audio CFG, i.e., AA = 0, were slightly worse than or comparable to them). Moreover, the FVD score shows a slight drop which indicates slightly better video quality.
Further increasing AA marginally improves lip-audio synchronization and reduces FVD25, but at the cost of slightly degrading audio-pose synchronization and gaze controllability. addition, observa- tions from the generated videos indicate that higher A significantly amplifies mouth movements for strong vocals and causes head pose jitter during rapid speech. For balanced performance and overall generation quality, we set AA = 0.5 and Xg 1.0 as our standard configuration.
We also evaluated the influence of sampling steps performance. Table illustrates that decreasing the steps from 50 to 10 improves audio-lip and audio-pose alignment while compromising pose variation intensity and overall video quality. This step reduction could accelerate the inference process by a factor of 5 for this latent motion generation module.
Training data scale. To validate the data scale influence and compare our model with previous methods similar scales, we trained diffusion model using only 10% of the data (i.e., 50K clips). As shown in Table I the model trained on this reduced dataset demonstrates comparable audio-lip

---

and audio-pose synchronization to the full-dataset model, although the FVD and Ap metrics are not as good. Nonetheless, it still significantly outperforms previous methods across all metries assessing synchronization, motion intensity, and video quality. This indicates that our approach remains highly effective even with much less data, and that increasing the dataset size enhances motion diversity.
we introduce new losses comeesst and leroes_id to improve the disentanglement of facial dynamics, head pose, and face identity. To validate the effectiveness of lommiist we transfer only facial dynamics from the source image to the target while keeping the target's pose unchanged. Figure 4 shows that without leonsist, the latent model may struggle to replicate subtle facial dynamics such as side glances and lip asymmetries which are oftentimes coupled with head poses (e.g., skewed mouth may coincide with a tilted head, and the gaze direction usually aligns with the head's pose). Decoupling these subtle yet important dynamics are challenging without explicit constraints from loomeist-
Losses for latent space learning. As described in Sec. 3.1
We also evaluate the face identity loss leross_id for cross identity driving during training. We use all 108 subjects from the VoxCeleb2 test set for evaluation. For each subject, we chose the image that is closest to frontal view from the first frames of all its clips serve as the target image. Then we randomly selected 50 clips of other subjects as source videos, which leads to a total of 5,400 reesatetenatttent clips. We calculate the facial identity preservation score by averaging the facial identity feature cosine similarity over all generated frames of all subjects. With the introduced face identity loss feross_idt _id this identity preservation score of our results improved from 0.72 to 0.80.
5 Conclusion
In summary, our work presents an audio-driven talking face generation model that stands out for its efficient generation of realistic lip synchronization, vivid facial expressions, and naturalistic head movements from single image and audio input. It significantly outperforms existing methods in delivering video quality and performance efficiency, demonstrating promising visual affective skills in the generated face videos. The technical cornerstone is an innovative holistic facial dynamics and head movement generation model that works in an expressive and disentangled face latent space.
The adaanementss made by VASA-1 1 have the potential to reshape -hmman and human-Al inter- actions across various domains, including communication, education, and healthcare. The integration of controllable conditioning signals further enhances the model's adaptability for personalized user experiences.
There are still several limitations with our method. Currently, it processes human regions only up to the torso. Extending the full upper body could offer additional capabilities. While utilizing 3D latent representations, the absence of a more explicit 3D face model such as [66 67 may result in artifacts like texture sticking due to the neural rendering. Additionally, our approach does not account for non-rigid elements like hair and clothing, which could be addressed with a stronger video prior. In the future, we also plan to incorporate more diverse talking styles and emotions to improve expressiveness and control.
Contribution statement
Sicheng Xu, Guojun Chen, Yu-Xiao Guo were the core contributors to the implementation, training, and experimentation of various algorithm modules, well the data processing and management. Jiaolong Yang initiated the project idea, led the project, designed the overall framework, and pro- vided detailed technical advice to each component. Chong Li, Zhengyu Zang and Yizhong Zhang contributed to enhancing the system quality, conducting evaluations, and demonstrating results. Xin Tong provided technical advice throughout the project and helped with project coordination. Baining Guo offered strategic research direction guidance, scientific advising, and other project supports. Paper written by Jiaolong Yang and Sicheng Xu.
Acknowledgments
We would like to thank our colleagues Zheng Zhang, Zhirong Wu, Shujie Liu, Dong Chen, Xu Tan, Yu Deng, Lidong Zhou, and others for the valuable discussions and insightful suggestions for our project.
10

---

References
[1] http::/wwww.pnowwwii.ooo////eeee
ai siii-deiiee--i---e av-avaiar--aaae to- peope disabilities htti:liieee.322222666..... h.mll 2024. [Online; accessed 20-May-20244.
[2] Ahmed A Abdelrahman, Thorsten Hempel, Aly Khalifa, Ayoub Al-Hamadi, and Laslo Dinges. net: Fine sine-grained gaze estimation in unconstrained environments. In 2023 8th International Conference on Frontiers of Signal Processing (ICFSP). pages 98 102. IEEE, 2023.
[3] Alexei Baevski, Yuhao Zhou, Abdelrahman Mohamed, and Michael Auli. wav2vee 2.0: A framework for self-supervised learning of speech representations. Advances in Neural Information Processing Systems,
33:12449- 12460, 2020.
[4] Omer Tar- Hila Chefer, Omer Tov, Charles Hermmann Roni Paiss, Shiran Zada, Ariel Ephrat, Junhwa
Hur. Yuanzhen Li. Tomer Michaeli, et al. Lumiere: A space-time diffusion model for video generation. arXiv preprint arXiv:2401. 12945, 2024.
[5] James Betker, Gabriel Goh, Li Jing. Tim Brooks, Jianfeng Wang, Linjie Li, Long Ouyang, Juntang Zhuang. Joyce Lee, Yufei Guo, et al. Improving image generation with better captions. https://cdn. openai. com/papers/tall-ce p.dd.... 2(3):8, 2023.
[6] Andreas Blattmann, Tim Dockhorn, Sumith Kulal, Daniel Mendelevitch, Maciej Kilian, Dominik Lorenz, Yam Levi, Zion English, Vikram Voleti, Adam Letts, et al. Stable video diffusion: Scaling latent video diffusion models to large datasets, arXiv preprint arXiv:2311.15177 2023.
[7] Andreas Blattmann, Robin Rombach, Huan Ling, Tim Dockhorn, Seung Wook Kim, Sanja Fidler, and Karsten Kreis. Align your latents: fiih-resoluutinn video synthesis with latent diffusion models. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 22563-
22575, 2023.
[8] Aras Bozkurt, Xiao Junhong. Sarah Lambert, Angelica Pazurek, Helen Crompton, Suzan Koseoglu, Robert Farrow, Melissa Bond, Chrissi Nerantzi, Sarah Honeychurch, al. Speculative futures chatgpt and generative artificial intelligence (ai): A collective reflection from the educational landscape. Asian Journal of Distance Education, 18(1): -13,533-33,, 2023.
[9] Tim Brooks, Bill Peebles, Connor Holmes, Will DePue, Yufei Guo, Li Jing, David Schnurr, Joe Taylor, Troy Luhman, Eric Luhman, Clarence Ng, Ricky Wang, and Aditya Ramesh. Video generation models as world simulators. 2024.
[10] Tom Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared D Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, al. Language models are few-shot learners. Advances in Neural Information Processing Systems, 33: 13:1877-1901 2020.
[11] Egor Burkov, 1gor Pasechnik, Artur Grigorev, and Victor Lempitsky. Neural head reenactment with latent pose descriptors. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 13786-13795 2020.
[12] Lele Chen, Zhiheng Li, Ross K Maddox, Zhiyao Duan, and Chenliang Xu. Lip movements generation glance. European Conference on Computer Vision, pages 520-535, 2018.
[13] Kun Cheng, Xiaodong Cun, Yong Zhang, Menghan Xia, Fei Yin, Mingrui Zhu, Xuan Wang. Jue Wang. and Nannan Wang. Videoretalking: Audio-based lip synchronization for talking head video editing in the wild. In SIGGRAPH Asia 2022, pages 1-9, 2022.
[14] Joon Son Chung. Arsha Nagrani, and Andrew Zisserman. Voxceleb2: Deep speaker recognition. arXiv
preprint arXiv: /006. 05622,, 2018.
[15] Joon Son Chung and Andrew Zisserman, Out of time: automated lip sync in the wild. In Asian Conference Computer Vision Workshops, pages 251-263. Springer, 2017.
[16] Jiankang Deng. Jia Guo, Niannan Xue, and Stefanos Zafeiriou. Arcface: Additive angular margin loss for deep face recogmition. In Proceedings of the JEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 4690-4699, 2019.
[17] Yu Deng, Jiaolong Yang. Sicheng Xu, Dong Chen, Yunde Jia, and Xin Tong. Accurate 3d face reconstruc- tion with weakly-supervised learning: From single image to image set. In Proceedings of the IEEF/CVF Conference on Computer Vision and Pattern Recognition Workshops, pages 0-0, 2019.
[18] Nikita Drobyshev, Antoni Bigata Casademunt, Konstantinos Vougioukas, Zoe Landgraf, Stavros Petridis, and Maja Pantic. Emoportraits: Emotion- multimodal shet head avatars. arXiv preprint arXiv:2404. /91 2024.
[19] Nikita Drobyshev, Jenya Chelishev, Taras Khakhulin, Aleksei Ivakhnenko, Victor Lempitsky, and Egor Zakharov. Megaportraits: One-shot megapixel neural head avatars. In Proceedings of the 30th ACM International Conference on Multimedia, pages 2663-2671, 2022.
[20] Chenpeng Du Qi Chen, Tianyu He, Xu Tan, Xie Chen, Kai Yu, Sheng Zhao, and Jiang Bian. Dac-talker: High fidelity speech- driven talking face generation with diffusion autoencoder. In Proceedings of the ACM International Conference on Multimedia, pages 4281-4289 2023.
[21] Yingruo Fan, Zhaojiang Lin, Jun Saito, Weaping Wang. and Taku Komura. Faceformer: Speech-driver 3d facial animation with transformers. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 18770-1888 18780, 2022.

---

[22]
Yue Gao, Yuan Zhou. Jinglu Wang. Xiao Li, Xiang Ming, and Yan Lu. High-fidelity and freely controllable talking head video generation. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 5609-5619, 2023.
[23]
Rohit Girdhar, Mannat Singh, Andrew Brown, Quentin Duval, Samanch Azadi, Sai Saketh Rambhatla, Akbar Shah, Xi Yin, Devi Parikh, and Ishan Misra. Emu video: Factorizing text-to-video generation by explicit image conditioning. arXiv preprint arXiv:2311. 10709, 2023.
[24] lan Goodfellow, Jean Pouget-Abadie, Mehdi Mirza, Bing Xu, David Warde-Farley, Sherjil Ozair, Aaron Courville, and Yoshua Bengio. Generative adversarial nets, Advances in Neural Information Processing
Systems, 27, 2014.
[25] Yudong Guo, Keyu Chen, Sen Liang, Yong-Jin Liu, Hujun Bao, and Juyong Zhang. Ad-nerf: Audio driven neural radiance fields for talking head synthesis. In /EEE/CVF International Conference on Computer Vision, pages 5784-5794, 2021.
[26] Tianyu He, Junliang Guo, Runyi Yu, Yuchi Wang, Jialiang Zhu, Kaikai An, Leyi Li, Xu Tan, Chunyu Wang, Han Hu, et al. Gaia: Zero-shot talking avatar generation. In International Conference on Learning Representations, 2024.
[27] Jonathan Ho, Ajay Jain, and Pieter Abheel. Denoising diffusion probabilistic models. Advances in neural information proessing systems, 33:6840-6851, 2020.
[28] Jonathan Ho and Tim Salimans. Classifier-free diffusion guidance. arXiv preprint arXiv:2207.12500 2022.
[29] Esperanza Johnson, Ramón Hervás, Carlos Gutiérrez López de la Franca, Tania Mondéjar, Sergio F Ochoa, and Jesús Favela. Assessing empathy and managing emotions through interactions with an affective avatar. Health informatics journal, 24(2):182-19)
2018.
[30] Tero Karras, Samuli Laine, Miika Aittlaa, Janne Hellsten, Jaakko Lehtinen, and Timo Aila. Analyzing and improving the image quality of stylegan. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 8110-8 8119, 2020.
[31] Greg Kessler. Technology and the future of language teaching. Foreign Language Annals, 51(1):205-216 2018.
[32] Dan Kondratyuk, Lijun Yu, Xiuye Gu, José Lezama, Jonathan Huang, Rachel Hornung, Hartwig Adam, Hassan Akbari, Yair Alon, Vighnesh Birodkar, et al. Videopoet: A large language model for zero-shot video generation. arXiv preprint 2312::3312.11222
2023.
[33] Julian Leff, Geoffrey Williams, Mark Huckvale, Maurice Arbuthnot, and Alex P Leff. Avatar therapy for persecutory auditory hallucinations: What is it and how does it work? Psychosis, 6(2): 176, 2014.
[34] Borong Liang, Yan Pan, Zhizhi Guo, Hang Zhou, Zhibin Hong, Xiaoguang Han, Junyu Han, Jingtuo Liu, Errui Ding, and Jingdong Wang. Expressive talking head generation with granular audio-visual control. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 3387-3396. 2022.
[35] Shugao Ma, Tomas Simon, Jason Saragih, Dawei Wang. Yuecheng Li, Fernando De La Torre and Yaser Sheikh. Pixel codec avatars. In IEEE/CVP Conference on Computer Vision and Pattern Recognition, pages 64-73, 2021.
[36] Yifeng Ma, Suzhen Wang, Zhipeng Hu, Changjie Fan, Tangjie Lv, Yu Ding, Zhidong Deng, and Xin Yu. Styletalk: One-shot talking head generation with controllable speaking styles. In AAAI Conference on Artificial Intelligence, pages arXiv-2301, 2023.
[37] Youxin Pang, Yong Zhang, Weize Quan. Yanbo Fan, Xiaodong Cun, Ying Shan, and Dong-ming Yan. Dpe: Disentanglement of pose and expression for general video portrait editing. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 427-436, 2023.
[38] William Pecbles and Saining Xie. Scalable diffusion models with transformers. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pages 4195-4205, 2023.
[39] KR Prajwal. Rudrabha Mukhopadhyay, Vinay P Namboodiri, and CV Jawahar. A lip sync expert is all you need for speech to lip generation in the wild. ACM International Conference on Multimedia, pages
484-492, 2020.
[40] Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, et al. Learning transferable visual models from natural language supervision. In International Conference on Machine Learning, pages 8748-8763. PMLR, 2021.
[41] Imogen C Rehhm, Emily Foenander, Klaire Wallace, Jo-Anne M Abbott, Michael Kyrios, and Neil Thomas. What role can avatars play in e-mental health interventions? exploring new models of clicnt-therapist interaction. Frontiers in Psychiatry, 7:186, 2016.
[42] Yurui Ren, Ge Li, Yuanqi Chen, Thomas H Li, and Shan Liu. PIRenderer: Controllable portrait image generation via semantic neural rendering. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pages 13759-13768. 2021.
[43] Andrey V Savchenko. Hsemotion: High-speed emotion recognition library. Software Impacts, 14: 14:100433, 2022.
12

---

[44] Aliaksandr Siarohin, Stéphane Lathuilière, Sergey Tulyakov, Elisa Ricci, and Nicu Sebe. First order motion model for image animation. In Advances in Neural Information Processing Systems, 2019.
[45] Li Siyao, Weijiang Yu, Tianpei Gu, Chunze Lin, Quan Wang, Chen Qian, Chen Change Loy, and Ziwci Liu. Bailando: 3d dance generation by actor-critic gpt with choreographic memory. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 1050-11059, 2022.
[46] Ivan Skorokhodov, Sergey Tulyakov, and Mohamed Elhoseiny. Stylegan-v: A continuous video generator with the price, image quality and perks of stylegan2. In Proceedings of the IEEE/CVF Conference on Computer Vision and Puttern Recognition, pages 3626-3636. 2022.
[47]
Jiaming Song. Chenlin Meng, and Stefano Ermon. Denoising diffusion implicit models. arXiv preprint
arXiv:2010.02.02,, 2020.
[48] Yang Song. Jascha
Diederik P Kingma. Abhishek Kumar, Stefano Ermon, and Ben Poole. Score-based generative modeling through stochastic differential equations. arXiv preprint
arXiv:2011.1345. 2020.
[49] Michal Styputkowski, Konstantinos Vougioukas, Sen He, Maciej Zigba, Stavros Petridis, and Maja Pantic. Diffused heads: Diffusion models heat gans on talking-face generation. In Proceedings of the IEFF/CVF
Winter Conference on Applications of Computer Vision. pages 5091-5100 2024.
[50] Shaolin Su. Qingsen Yan, Yu Zhu, Cheng Zhang, Xin Ge. Jinqiu Sun, and Yanning Zhang. Blindly assess image quality in the wild guided by self-adaptive hyper network. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 3667-3676, 2020.
[51] Yasheng Sun, Hang Zhou, Ziwei Liu, and Hideki Koike. Speech2talking-facee Inferring and driving face with synchronized audio-visual representation. International Joint Conference on Artificial Intelligncee volume 2. page 4, 2021.
[52] Zhiyao Sun, Tian Lv, Sheng Ye, Matthieu Gaetan Lin, Jenny Sheng. Yu-Hui Wen, Minjing Yu, and Yong-jin Liu. Diffposetalk: Speech-driven stylistic facial animation and head pose generation via diffusion models. arXiv preprint 23 2023.
[53] Supasorn Suwajanakom, Steven M Seitz, and Ira Keemelnacher-hhizzzrmann Synthesizing obama: learning lip syne from audio. ACM Transactions on Graphics, 36(4):1-13, 2017.
[54] Shuai Tan, Bin Ji, Mengxiao Bi, and Ye Pan. Edtalk: Efficient disentanglement for emotional talking head synthesis. arXiv preprint arXiv:2404.00 204424444,, 2024.
[55] Linrui Tian, Qi Wang. Bang Zhang, and Liefeng Bo. Emo: Emote portrait alive-generating expressive
portrait videos with audio2video diffusion model under weak conditions. arXiv preprint 240.::20.2.2485.. 2024.
[56] Sergey Tulyakov, Ming- Yu Liu, Xiaodong Yang, and Jan Kautz. Mocogan: Decomposing motion and content for video generation. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pages 1526-1535, 2018.
[S7] Thomas Unterthiner, Sjoerd van Steenkiste, Karol Kurach, Raphaël Marinier, Marcin Michalski, and Sylvain Gelly. Fvd: A new metric for video generation. 2019.
[58] Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Lukasz Kaiser, and Illia Polosukhin. Attention is all you need. Advances in Neural Information Processing Systems.
30, 2017.
[59] Carl Vondrick, Hamed Pirsiavash, and Antonio Torralba. Generating videos with scene dynamics. Advances in Neural Information Processing Systems, 29, 2016.
[60] Duomin Wang, Yu Deng, Zixin Yin, Heung-Yeung Shum, and Baoyuan Wang. Progressive disentangled representation learning for fine- funeggrieeed controllable talking head synthesis. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 17979-17989. 2023.
[61] Jiayu Wang. Kang Zhao, Shiwei Zhang, Yingya Zhang. Yujun Shen, Deli Zhao, and Jingren Zhou. Lipformer: High-fidelity and generalizable talking face generation with a pre-leamedd facial codchook. In JEEF/CVF Conference on Computer Vision and Pattern Recognition, pages 13844-13853, 2023.
[62] Suzhen Wang. Lincheng Li. Yu Ding, Changjie Fan, and Xin Yu. Audio2head: Audio-driven one- shot talking-head generation with natural head motion. In International Joint Conference on Artificial Intelligence, 2021.
[63] Suzhen Wang, I Lincheng L.,, Yu Ding, and Xin Yu. One-shot talking face generation from single-speaker audio-visual correlation learning. In AAAI Conference on Artificial Intelligence, volume 36, pages 2531- - 2539, 2022.
[64] Ting- Chun Wang, Arun Mallya, and Ming- Yu Liu. One-shot view neural talking- head synthesis for video conferencing. In IEEE/CVF Conference on Computer Vision and Puttern Recognition, pages 10039-10049, 2021.
Ting-Chun
[65] Huawei Wei, Zejum Yang, and Zhisheng Wang. Aniportrait: Audio-driven synthesis of photorealistic portrait animation. arXiv preprint 24X3 17694, 2024.
[66] Yue Wu, Yu Deng. Jiaolong Yang. Fangyun Wei, Qifeng Chen, and Xin Tong. Anifacegan: Animatable ad-aw face image generation for video avatars. Advances in Neural Information Processing Systems, 1836688-36201 2022.
13

---

[67] Yue Wu, Sicheng Xu, Jianfeng Xiang, Fangyun Wei, Qifeng Chen, Jiaolong Yang. and Xin Tong. Anipor- traitgan: Animatable 3d portrait generation from 2d image collections. In SIGGRAPH Asia 2023, pages 1-9, 2023.
[68] Jinbo Xing, Menghan Xia, Yuechen Zhang, Xiaodong Cun, Jue Wang, and Tien- Wong. Codetalker: Speech-driven 3d facial animation with discrete motion prior. In Proceedings the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 12780-12790, 2023.
[69] Wilson Yan, Yunzhi Zhang, Pieter Abbeel. and Aravind Srinivas. Videogpt: Video generation using vq-vae and transformers. arXiv preprint arXiv:210k J0157, 2021.
[70] Fei Yin. Yong Zhang, Xiaodong Cun, Mingdeng Cao. Yanbo Fan. Xuan Wang, Qingyan Bai, Baoyuan Wu, Jue Wang, and Yujiu Yang. Styleheat: One-shot high-resolution editable talking face generation via pre- pre-trained stylegan. In European Conference Computer Vision, pages 85-101, 2022.
[71] Zhentao Yu, Zixin Yin, Deyu Zhou, Duomin Wang. Finn Wong, and Baoyuan Wang. Talking head generation with probabilistic audio-to-visual diffusion priors. In Proceedings of the IEEE/CVF International Conference Computer Vision, pages 7645-7655. 2023.
[72] Egor Zakharov, Aleksci Ivakhnenko, Aliaksandra Shysheya, and Victor Lempitsky. Fast bi-layer neural synthesis of onc-shot realistic head avatars, In European Conference on Computer Vision, pages 524-540, 2020.
[73] Bowen Zhang, Chenyang Qi, Pan Zhang, Bo Zhang, Hsiang Tao Wu, Dong Chen, Qifeng Chen, Yong Wang, and Fang Wen. Metaportrait: preettrinr talking head generation with fast personalized adaptation. In Proceedings of the IEEE/CVF Conference Computer Vision and Pattern Recognition, pages 22096-22105, 2023.
[74] Wenxuan Zhang, Xiaodong Cun, Xuan Wang, Yong Zhang, Xi Shen, Yu Guo, Ying Shan, and Fei Wang. Sadtalker: Learning realistic 3d motion coefficients for stylized audio-driven single image talking face animation. In IEEE/CVF Conference Computer Vision and Pattern Recognition, pages 8652-8661, 2023.
[75] Zhimeng Zhang, Lincheng Li, Yu Ding, and Changjie Fan. Flow-guided one-shot talking face generation with high-resolution audio-visual dataset. In IEEE/CVF Conference Computer Vision and Puttern Recognition, pages 3661-3670, 2021.
[76] Hang Zhou, Yasheng Sun, Wayne Wu, Chen Change Loy. Xiaogang Wang, and Ziwei Liu. Pose-controllable talking face generation by implicitly modularized audio-visual representation. In Proceedings of the IEEE/CVF Conference on computer Vision and Pattern Recognition, pages 4176 4186, 2021.
[77] Yang Zhou, Xintong Han, Eli Shechtman, Jose Echevarria, Evangelos Kalogerakis, and Dingzeyu Li. Makelttalk: speaker- talking- head animation. ACM Transactions On Graphics (TOG), 39(6):1115
2020.
14

---

A Societal Impacts and Responsible AI Considerations
Our research focuses on generating audio-driven visual affective skills for virtual Al avatars, aiming for positive applications. It not intended create content that is used mislead or deceive. However, like other related content generation techniques, it could still potentially be misused for impersonating humans. We opposed to any behavior that creates misleading harmful contents of real persons. Currently, the videos generated by this method still contain identifiable artifacts, and the numerical study in Section 4 shows that there's still gap achieve the authenticity of real videos. Furthermore, we have trained a neural network based detector to distinguish real videos and those generated by our VASA-1, and the detector shows a 97.8% accuracy for this task.
While acknowledging the possibility of misuse, it's imperative to recognize the substantial positive potential of our technique. The benefits ranging from enhancing educational equity, improving ac- cessibility for individuals with communication challenges, and offering companionship or therapeutic support to those in need underscore the importance of our research and other related explorations. We are dedicated to developing AI responsibly, with the goal of advancing human well-being.
To combat potential misuse of our technique and other related ones and provide necessary safeguards, we are also working on applying our method for advancing face media forgery detection. Specifically, are training generic face forgery detection models that incorporate our generated talking face videos as part of the training data. Our preliminary exploration shows that using our method to generate training data can lead to obvious improvement of generality for the forgery detection models, and we'll ll keep the community updated on new progresses.
B
More Qualitative Evaluation, Comparison and Ablation Study
See Figure A. A.2
A.6
15

---

Figure A.1: Disentanglement between identity and motion. In these examples, the same generated head and facial motion sequences applied onto three different face images.
Figure A.2: Disentanglement between head pose and facial dynamics. From top bottom: the raw generated sequence, applying generated poses with fixed initial facial dynamics, and applying generated facial dynamics with fixed initial head pose and pre-defined spinning poses, respectively.
16

---

Figure A.3: Generation results with -iitti-iittiitttion images hhon-poooorealistic) and audios (singing audios for the first two and non- won-English speech for the last row). Our method still generate high quality videos well-aligned with the audios, although it was not trained on such data variations. See the supplementary video with audio for better illustration of these results.
MakettTalk
Audio2Head
Sadaakeer
Ours
Figure A.4: Generation results from different methods with the input audio segment uttering "push ups". See our supplementary vidco for better illustration and comparison.
17

---

Makektall
Audio2Head
SadTalker
Ours
D
Figure A.5: Generation results from different methods with the input audio segment uttering "excruciating". See supplementary video for better illustration and comparison.
MakeltTalk
Audio2Head
SadTalker
Ours
Figure A.6: Generation results from different methods with the input audio segment uttering "what?". See our supplementary video for a better illustration and comparison.
18

---

Makeitaalk
Audio2Head
SadTalker
Ours
Figure A.7: Generation results from different methods with the input audio segment uttering "lots of questions". See supplementary video for better illustration and comparison.
19

---

NeurIPS Paper Checklist
1. Claims
Question: Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?
Answer: [Yes]
Justification: Yes, the claims match the paper's contributions and scope.
Guidelines:
The answer NA means that the abstract and introduction do not include the claims made in the paper.
The abstract and/or introduction should clearly state the claims made, including the contributions made in the paper and important assumptions and limitations. A No or
NA answer to this question will not perceived well by the reviewers.
The claims made should match theoretical and experimental results, and reflect how be expected to generalize other settings.
much the results
is fine to include aspirational goals as motivation as long is clear that these goals are not attained by the paper.
2. Limitations
Question: Does the paper discuss the limitations of the work performed by the authors?
Answer: [Yes]
Justification:
We discuss the limitations in the Section 5
Guidelines:
The answer NA means that the paper has no limitation while the answer No means that the paper has limitations, but those not discussed in the paper.
The authors are encouraged to create separate "Limitations" section in their paper.
The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (e.g., independence assumptions, noiseless settings, model well-specification. asymptotic approximations only holding locally). The authors should reflect how these assumptions might be violated in practice and what the implications would be.
The authors should reflect on the scope of the claims made, e.g., if the approach was only tested on few datasets or with few runs. In general, empirical results often depend on implicit assumptions, which should be articulated.
The authors should reflect the factors that influence the performance of the approach. For example, facial recognition algorithm may perform poorly when image resolution low images are taken in low lighting. Or speech-to-text system might be used reliably to provide closed captions for online lectures because it fails to handle technical jargon.
The authors should discuss the computational efficiency of the proposed algorithms and how they scale with dataset size.
If applicable, the authors should discuss possible limitations of their approach address problems of privacy and faimess.
While the authors might fear that complete honesty about limitations might be used by reviewers as grounds for rejection, a worse outcome might be that reviewers discover limitations that aren't 't acknowledged in the paper. The authors should use their best judgment and recognize that individual actions in favor of transparency play an impor- tant role in developing norms that preserve the integrity of the community. Reviewers will be specifically instructed to not penalize honesty concerning limitations.
3. Theory Assumptions and Proofs
Question:
For each theoretical result, does the paper provide the full set of assumptions and complete (and correct) proof?
Answer: [NA]
20

---

Justification:
This paper does not include theoretical results.
Guidelines:
The answer NA means that the paper does not include theoretical results.
All the theorems, formulas, and proofs in the paper should be numbered and cross- referenced.
All assumptions should be clearly stated referenced in the statement of any theorems.
The proofs can either appear in the main paper or the supplemental material, but they appear in the supplemental material, the authors are encouraged to provide short proof sketch provide intuition.
Inversely, any informal proof provided in the core of the paper should complemented by formal proofs provided in appendix or supplemental material.
Theorems and Lemmas that the proof relies upon should be properly referenced.
4. Experimental Result Reproducibility
Question: Does the paper fully disclose all the information needed to reproduce the main ex- perimental results of the paper to the extent that affects the main claims and/or conclusions of the paper (regardless whether the code and data are provided or not)?
Answer: [Yes].
Justification: Yes, the paper has provided the necessary information.
Guidelines:
The answer NA means that the paper does not include experiments.
If the paper includes experiments, a No answer to this question will not be perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data are provided or not.
If the contribution is dataset and/or model, the authors should describe the steps taken to make their results reproducible or verifiable.
Depending on the contribution, reproducibility can be accomplished in various ways. For example, if the contribution is a novel architecture, describing the architecture fully might suffice, or if the contribution is a specific model and empirical evaluation, it may be necessary to either make possible for others to replicate the model with the same dataset, or provide access to the model. In general. releasing code and data is often one good way accomplish this, but reproducibility can also be provided via detailed instructions for how to replicate the results, access to a hosted model (e.g.. in the case of large language model), releasing of a model checkpoint, or other means that are appropriate to the research performed.
While NeurIPS does not require releasing code, the conference does require all submis- sions to provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution. For example
(a) the contribution is primarily a new algorithm, the paper should make clear how to reproduce that algorithm.
(b) the contribution is primarily a new model architecture, the paper should describe the architccture clearly and fully.
(c) If the contribution is new model (e.g., a large language model), then there should either way to access this model for reproducing the results or way to reproduce the model (e.g., with an open-source dataset instructions for how to construct the dataset).
(d) We recognize that reproducibility may be tricky in some cases, in which case authors are welcome to describe the particular way they provide for reprodueibility. the case of closed-source models, may be that access to the model is limited in some way (e.g., to registered users), but it should be possible for other researchers have some path to reproducing or verifying the results.
5. Open access to data and code
Question: Does the paper provide open access to the data and code, with sufficient instruc- tions to faithfully reproduce the main experimental results, as described in supplemental material?
21

---

Answer: [No]
Justification: Based on the RAI considerations, we will not release our code or data in case of potential misuse, discussed in Section A
Guidelines:
The answer NA means that paper does not include experiments requiring code.
Please see the NewrIPS code and data submission guidelines (https://nips.cm cc/pu
blic/ Belcc/guies///e CoweSoomiessiiieeeoo ssinissssoyyyyyy for more details.
While we encourage the release of code and data, we understand that this might not be possible, so "No" is an acceptable answer. Papers cannot be rejected simply for not including code, unless this is central to the contribution (e.g., for new open-source benchmark).
The instructions should contain the exact command and environment needed to to reproduce the results. See the NeurlPS code and data submission guidelines (https
wttp://iiggii/////ee/eee http:/uuddde/CoeSuubbbb ssionPolicy) for more details.
The authors should provide instructions on data access and preparation, including how to access the raw data, preprocessed data, intermediate data, and generated data, etc.
The authors should provide scripts to reproduce all experimental results for the new proposed method and baselines. only subset of experiments are reproducible, they should state which ones are omitted from the script and why.
At submission time, preserve anonymity, the authors should release anonymized versions (if applicable).
Providing as much information as possible in supplemental material (appended to the paper) recommended, but including URLs to data and code permitted.
6. Experimental Setting/Details
Question: Does the paper specify all the training and test details (e.g., data splits, hyper- parameters, how they were chosen, type of optimizer, etc.) necessary to understand the results?
Answer: [Yes]
Justification:
We included the experimental setting and details in Section
Guidelines:
The answer NA means that the paper does not include experiments.
The experimental setting should be presented in the core of the paper to level of detail that is necessary to appreciate the results and make sense of them.
The full details can be provided either with the code, in appendix, or as supplemental material.
7. Experiment Statistical Significance
Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?
Question:
Answer: [Yes]
Justification: Our method was evaluated on datasets with sufficient data samples and the results are statistically meaningful; see Section 4 for details.
Guidelines:
The answer NA means that the paper does not include experiments.
The authors should answer "Yes" if the results are accompanied by error bars, confi- dence intervals, or statistical significance tests, at least for the experiments that support the main claims of the paper.
The factors of variability that the error bars capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, or overall with given experimental conditions).
The method for calculating the error bars should be explained (closed form formula, call to a library function, bootstrap, etc.)
The assumptions made should be given (e.g., Normally distributed errors).
22

---

It should be clear whether the error bar is the standard deviation or the standard error of the mean.
is OK to report 1-sigma error bars, but one should state it. The authors should preferably report a 2-sigma error bar than state that they have 96% CI, if the hypothesis of Normality of errors is not verified.
For asymmetric distributions, the authors should be careful not to show in tables or figures symmetric error bars that would yield results that are out range (e.g. negative enror rates).
If error bars reported in tables plots, The authors should explain in the text how they were calculated and reference the corresponding figures or tables in the text.
8. Experiments Compute Resources
For each experiment, does the paper provide sufficient information on the com- puter resources (type of compute workers, memory, time of execution) needed reproduce the experiments?
Question:
Answer: [Yes]
Justification:
We reported the compute resources in Section R
Guidelines:
The answer NA means that the paper does not include experiments.
The paper should indicate the type of compute workers CPU or GPU, internal cluster, or cloud provider, including relevant memory and storage.
The paper should provide the amount compute required for each of the individual experimental runs well estimate the total compute.
The paper should disclose whether the full research project required more compute than the experiments reported in the paper (e.g., preliminary failed experiments that didn't make into the paper).
9. Code Of Ethics
Question: Does the research conducted the paper conform, in every respect, with the NeurlPS Code of Ethics http:p/p/..aarrr...co//////////a
Answer:[Yes)
Justification:
The research presented in the paper adheres to all aspects of the NeurIPS Code
Ethics.
Guidelines:
The answer NA means that the authors have not reviewed the NeurIPS Code of Ethics.
If the authors answer No, they should explain the special circumstances that require deviation from the Code of Ethics.
The authors should make sure to preserve anonymity (e.g., there a special consid- eration due to laws or regulations in their jurisdiction).
10. Broader Impacts
Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?
Answer:
[Yes]
Justification: discussed the positive and negative societal impacts in Section Guidelines:
The answer NA means that there no societal impact of the work performed.
If the authors answer NA or No, they should explain why their work has no societal impact why the paper does not address societal impact.
Examples of negative societal impacts include potential malicious or unintended uses (e.g., disinformation, generating fake profiles, surveillance), fairness considerations (e.g., deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations.
23

---

The conference expects that many papers will be foundational research and not tied to particular applications, let alone deployments. However, if there is direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that an improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point out that a generic algorithm for optimizing neural networks could enable people to train models that generate Deepfakes faster.
The authors should consider possible harms that could arise when the technology is being used as intended and functioning correctly, harms that could arise when the technology being used as intended but gives incorrect results, and harms following from (intentional or unintentional) misuse of the technology.
If there are negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition attacks, mechanisms for monitoring misuse, mechanisms to monitor how system learns from feedback over time, improving the efficiency and accessibility ML).
11. Safeguards
Does the paper describe safeguards that have been put in place for responsible release of data or models that have a high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)?
Question:
Answer: [Yes]
Justification: Currently we have no plan to release the model or data to avoid potential misuse. We also discussed the development of safeguards in Section
Guidelines:
The answer NA means that the paper poses such risks.
Released models that have high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere usage guidelines or restrictions to access the model or implementing safety filters.
Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images.
We recognize that providing effective safeguards is challenging, and many papers do require this, but we encourage authors to take this into account and make best faith effort.
12. Licenses for existing assets
Question: Are the creators or original owners of assets (e.g.. code, data, models), used in the paper, properly credited and are the license and terms of use explicitly mentioned and properly respected?
Answer: [Yes]
Justification:
We cited the paper for the model/dataset we used in our paper.
Guidelines:
The answer NA means that the paper does not use existing assets.
The authors should cite the original paper that produced the code package dataset.
The authors should state which version of the asset is used and, if possible, include URL.
The name of the license (c.g., CC-BY 4.0) should be included for each asset.
For scraped data from a particular source (e.g., website), the copyright and terms of service of that source should be provided.
If assets are released, the license, copyright information, and terms of use in the package should be provided. For popular datasets, paperswithcome com/datase aam/aatsees has curated licenses for some datasets. Their licensing guide can help determine the license of dataset.
For existing datasets that are re-packaged, both the original license and the license of the derived asset (if it has changed) should be provided.

---

If this information is not available online, the authors are encouraged to reach out to the asset's 's creators.
13. New Assets
Question: Are new assets introduced in the paper well documented and is the documentation provided alongside the assets?
Answer: [NA]
Justification: We will not release assets.
Guidelines:
The answer NA means that the paper does not release new assets,
Researchers should communicate the details of the dataset/code/moodl as part of their submissions via structured templates. This includes details about training, license, limitations, etc.
The paper should discuss whether and how consent was obtained from people whose asset is used.
At submission time, remember to anonymize your assets (if applicable). You can either create an anonymized URL or include an anonymized zip file.
14. Crowdsourcing and Research with Human Subjects
Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given to participants and screenshots, if applicable, as well details about compensation (if any)?
Answer: [NA]
Justification:
This paper does not involve crowdsourcing nor research with human subjects.
Guidelines:
The answer NA means that the paper does not involve crowdsoureing nor research with human subjects.
Including this information in the supplemental material is fine, but if the main contribu- tion of the paper involves human subjects, then as much detail as possible should be included in the main paper.
According the NeurIPS Code of Ethies, workers involved in data collection, curation, or other labor should be paid at least the minimum wage in the country of the data collector.
15. Institutional Review Board (IRB) Approvals or Equivalent for Research with Human Subjects
Question: Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or an equivalent approval/review based on the requirements of your country or institution) were obtained?
Answer: [NA]
Justification: This paper does not involve crowdsourcing nor research with human subjects.
Guidelines:
The answer NA means that the paper does not involve crowdsourcing research with human subjects.
Depending on the country in which research is conducted, IRB approval (or equivalent) may be required for any human subjects research. If you obtained IRB approval, you should clearly state this in the paper.
We recognize that the procedures for this may vary significantly between institutions and locations, and we expect authors to adhere to the NeurlPS Code of Ethics and the guidelines for their institution.
For initial submissions, do not include any information that would break anonymity (if applicable), such as the institution conducting the review.
25


Reviews:
[{"review_id": "puQ3mvhhBd", "reviewer": "Reviewer_WJD9", "summary": "This paper aims to effectively and efficiently generate high-fidelity audio-driven talking head videos. To improve performance and efficiency, the authors have designed a Diffusion Transformer model within the latent space of motion signals, encompassing facial dynamics and head movements. Additionally, they propose a data-driven metric named Contrastive Audio and Pose Pretraining.", "questions": "- At inference time, how does the model generate condition signals like the main eye gaze direction and head-to-camera distance, given that the driving signal is only audio?\n- This does not decrease the novelty of this work. However, a quantitative comparison between VASA-1 and EMO would be quite interesting. Given that EMO's code is unavailable, leveraging the image and audio from their officially provided videos for comparison is encouraged.", "limitations": "Yes.", "rating": 7, "confidence": 4, "soundness": 4, "presentation": 4, "contribution": 3, "strengths": "- The paper applies the diffusion model to the task of generating audio-driven talking head videos, innovatively defining the diffusion model within the latent features of motion rather than those of the image, which is quite interesting.\n- The paper is well-written and easy to follow, with detailed experiments that convincingly demonstrate the effectiveness of the proposed method.", "weaknesses": "The primary concern is the paper's contribution, as the realism and liveliness of the generated videos could be attributed to the performance of MegaPortraits. MegaPortraits' encoders effectively learn latent motion and appearance representations, supported by robust 3D warping generators and an image generator that ensures high-quality outputs. VASA-1, in a way, learns to generate latent motion representations akin to those in MegaPortraits through audio inputs. Despite this dependency, the method performs well overall. Therefore, my overall assessment leans towards accepting it, albeit with some reservations.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "**Response to W1 (contributions):** Thank you for the comment. In response to your question on our technical contributions and relationship with MegaPortraits, we'd like to emphasize two aspects. **First**, our motivation in the first place is to model the human face conversational behavior (facial dynamics and head movement) *holistically using a diffusion model*, in a latent space that is agnostic to ID and appearance. This is our core innovation and, to the best of our knowledge, no previous methods have done this (it differs from the trends of further factor disentanglement and direct image generation; see our discussion in Lines 47-55, 103-111). **Second**, in pursuit of the aforementioned goal and building a disentangled latent space, we did find the 3D-aided representation to be promising especially in terms of expressiveness and hence chose to leverage them. However, they can NOT meet our requirement of effective disentanglement. We made some insightful and provably critical modifications (Line 149-160, 344-350, and Figure A.8), without which we can never reach the current generation quality, esp. the liveliness with nuanced emotions. We perhaps have underemphasized the importance and contribution of such modifications, and will revise our presentation in the revision. \n\nApart from the two main technical contributions, our paper also offers other ones such as the design of face-factor-conditioned diffusion training, the CAPP model for filling the missing piece of pose-audio alignment evaluation, etc., which are also novel and valuable to the community.\n\n**Response to Q1:** The extra condition signals such as the main eye gaze direction and head-to-camera distance are optional and they are provided by users. If not given, we can either set them to some default parameters  (e.g., a forward-looking eye gaze and the average head-to-camera distance of the training data; see Line 220-222), or just leave them blank for unconditional generation.\n\n**Response to Q2:** Thank you for the suggestion. We have run our method on the images and audios from EMO's official website as per your suggestion. Some visual results including animations can be found in the one-page PDF provided on this page (we are unable to provide video links per the rebuttal policy). EMO is a concurrent work which we mentioned and discussed in the related work section. It uses an image diffusion model based on StableDiffusion to generate talking face videos, which is a significantly different technique. EMO can generate high-quality videos but suffers from heavy computation and slow generation speed compared to ours.  \n\nAs shown in Figure I of the attached PDF,  our method works consistently well on EMO's demonstrated cases and delivers vivid talking head videos. It is obvious that EMO has smaller head motion compared to ours, perhaps due to the constraint of the face region mask used by it. Also, EMO seems less robust than ours in some cases, with artifacts – such as abrupt expression change, inaccurate lip sync, and subtle texture detail flickering – occasionally appear upon close inspection (note that their reported average lip-sync score is significantly lower than ours). On the other hand, however, EMO's video quality is slightly higher than ours in terms of sharpness, owing to their use of the large and powerful image generation foundation model.\n\n---\n\nWe hope we have addressed your questions. If not, it would be great to let us further know your concerns during discussion.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "jY9V0mUZuX", "reviewer": "Reviewer_wMXj", "summary": "This paper introduces a two-stage talking head method that can generate impressive talking faces. It includes 1) A diffusion-based model to generate implicit facial dynamics and head movements from audio and additional conditions. 2) A modified 3D-aided face reenactment decoder for generating faces from latent space. This method delivers high video quality with fast inference speed", "questions": "1. How does the number of layers in the 8-layer transformer encoder affect the results in the paper?\n2. Will the proposed CAPP in the paper be open-sourced?\n3. Does the 3D-aided face reenactment part of the method use distillation to speed up? How can the inference speed of MegaPortraits be accelerated?\n4. What are the parameter counts for each stage of the model?", "limitations": "The supplementary materials of the paper include relevant discussions.", "rating": 7, "confidence": 4, "soundness": 4, "presentation": 1, "contribution": 3, "strengths": "1. Although many works utilize diffusion models to map audio to intermediate facial features, VASA-V1 demonstrates excellent engineering and generation capabilities, achieving appealing results.\n2. This method surpasses real-time speed at a resolution of 512x512, with fast startup cost and ID switching speed, leaving an impressive effect.\n3. The method outperforms existing comparative methods in terms of visual effects and numerical results for video realism and audio-visual synchronization.", "weaknesses": "1. The 3D-aided face reenactment framework stage should be crucial for the overall method. However, some descriptions are too brief and vague, making them hard to follow.\n2. The paper's explanation of the fusion method for condition signals in the Diffusion Transformer is confusing and needs more specific details.\n3. The comparison methods in the paper lack implementation details. Considering the different scales of training data for various methods, are the comparison results in the table fair?", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "**Response to W1:** Thank you for the comment. We will add more details of the 3D-aided face latent model into our main paper and appendix. Our network architecture follows MegaPortraits [18] where details can be found. To achieve disentanglement, we modified the loss functions and incorporated the cross-transfer consistency loss $L_{consist}$ and ID-disentanglement loss $l_{cross-id}$. (Line 149-160). These loss modifications are critical (Line 344-350 and Figure A.8), without which we can never reach the current generation quality, esp. the liveliness with nuanced emotions. Again, we'll add more details and further improve the clarity as per your suggestion.\n\n**Response to W2:** Different conditional signals are directly concatenated with noise along the temporal dimension as the input to the transformer.\n\n**Response to W3:** First, please note that different methods in the literature may have used different data for training. We are unable to compare with these methods using exactly the same data and have simply followed the practice of running the trained models for comparison.\n\nIn terms of training data scale, our model is trained on ~500k clips (2-10 seconds each). To validate the data scale influence and compare it with previous methods at similar scales, we additionally trained a model using only 10% of the data (i.e., 50k clips). As shown in Table I in the attached PDF, the model trained with 10% of the data achieves comparable audio-lip and audio-pose synchronization to the full-dataset model, though the FVD and $\\Delta$p metrics are not as good. This shows that our method performs well even with much less data, and more data enhances the motion diversity.\n\nRegarding the compared methods, Audio2Head used >70k data clips for training but clearly underperformed compared to our model with 50k clips. MakeItTalk and SadTalker used very small subsets of Voxceleb for training, but there's no clear evidence that increasing their data would improve their performance significantly or even bring any positive consequence - we explain the reasons as follows. MakeItTalk uses an LSTM to map audio features to landmark offsets deterministically, which may struggle with modeling complex data distributions and one-to-many mappings as training data increases. SadTalker assigns a style code for each identity to generate head poses, but more data will introduce more diverse head motion patterns for the same identity, which a shallow VAE with a condition code might not be able to effectively model. Our model with 10% data still significantly outperforms these methods in all metrics assessing synchronization, motion intensity, and video quality.\n\n**Response to Q1:** We set the transformer layer number to 8 as we found it produces good results while enabling the whole algorithm to run in real time on a consumer-grade GPU. We didn't explore more layers or larger model size because real-time efficiency is a key factor we want to achieve. We presume that a larger model size will further improve the performance because our current model is still small, and we'll further explore this in our future work.\n\n**Response to Q2:** Yes, we will soon release the CAPP model, which we believe fills the missing piece of audio-pose alignment metric in talking face generation research and will be valuable to the community.\n\n**Response to Q3:** No, we did not use distillation or any other strategies to speed-up the 3D-aided face encoder and decoder. These models are naturally small and run very fast. Note we only need to run the encoder once so essentially only the decoder needs to run for generating each video frame.\n\n**Response to Q4:** The parameter counts of our 3D-aided face latent model and diffusion transformer model are about 200M and 29M respectively.\n\n---\n\nWe hope we have addressed your questions. If not, it would be great to let us further know your concerns during discussion.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "c8ZtI9Buqz", "reviewer": "Reviewer_1diB", "summary": "The paper presents a method for generating highly realistic talking head avatars that combines the diversity of facial expressions with the real-time generation speed. It provides a practical and commercially valuable approach to the field of talking head generation.", "questions": "1. Will this project be open-sourced? As it could actively promote progress in the field of talking head generation. If it is not open-sourced, it is suggested to provide more implementation details.\n2. Why is there no comparison with the recent EMO[1] method, for which there are already corresponding implementations in the open-source community?\n\n[1] Tian L, Wang Q, Zhang B, et al. Emo: Emote portrait alive-generating expressive portrait videos with audio2video diffusion model under weak conditions[J]. arXiv preprint arXiv:2402.17485, 2024.", "limitations": "1. Although VASA-1 has made significant progress in generating realistic facial dynamics and head movements, the paper mentions that it currently only handles the portrait area up to the torso and does not extend to the full body. The coordination of full-body movements is necessary to achieve more natural and realistic virtual characters.\n\n2. The paper mentions that, despite using a 3D latent representation, a more explicit 3D facial model is not employed, which may lead to some artifacts caused by neural rendering, such as texture adhesion issues.", "rating": 8, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "1. The overall structure of the paper is very clear and coherent, with a well-defined problem statement.\n2. By decoupling the information in dynamic faces, better control over the expressiveness of the generated faces can be achieved, meeting the needs of the users.\n3. The visual presentation of the video is excellent and leaves a lasting impression.", "weaknesses": "1. The paper does not explain why the proposed method can achieve real-time generation; the use of a diffusion transformer structure might actually lead to a decrease in speed.\n2. There are some unclear configurations in the implementation section of the method, such as the scale of the video used for training.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "**Response to W1:** We achieve real-time efficiency because of our framework design, i.e. (diffusion-based) motion generation in latent space + (CNN-based) decoding in image space. Our diffusion transformer works in the *latent space* and is small (only 29M parameters), so it runs very fast. The CNN image decoder is also small (55M parameters) and runs efficiently. (Note that we only need to run the face *encoder* once for generating a video and thus its time can be neglected or counted into the starting latency.) \n\nWe simply evaluated the running efficiency with our whole method naively deployed in PyTorch without any special speed-up strategy. We believe there's still room for improvement with sophisticated implementation optimization.\n\n**Response to W2:** Regarding the scale of  video data, we trained our model on approximately 500k clips (2-10 seconds each). In the attached one-page PDF, we also provide an additional ablation study for training data scale. As shown in Table I,  the model trained with 10% of the data achieves comparable audio-lip and audio-pose synchronization to the full-data model, though the FVD and $\\Delta$p metrics are not as good. This shows that our method performs well even with much less data, and more data enhances the motion diversity.\n\n**Response to Q1:** We'll try our best to release the source code of our project in the future. However, we hope you understand that due to significant concerns regarding the potential risks, particularly those related to deepfake and fraud, we (as well as the community) do need to be very cautious with releasing a powerful model. In fact, due to RAI considerations, our team faced great difficulties in getting approved by our organization for open source, unlike any other projects we did before. While we are seeking for the possibility of open source, we'll also add more implementation details such as those suggested by the reviewers into the revised paper. Also note that we will soon release the CAPP model, which we believe fills the missing piece of audio-pose alignment metric in talking face generation research and will be valuable to the community.\n\n**Response to Q2:** EMO is a concurrent work (published Feb 27th on arXiv) at the time of our submission (May 22th) and there's no public implementation, so we did not compare with it. However, we did mention it with some discussions in our paper (Line 41-44, 115-118). EMO uses an image diffusion model based on StableDiffusion to generate talking face videos, which is a significantly different technique. It can generate high-quality videos but suffers from heavy computation and slow generation speed compared to ours. \n\nIn the attached one-page PDF, we provide our results including animations on some samples from EMO's official website (we are unable to provide video links per the rebuttal policy). As shown in Figure I, our method works consistently well on EMO's demonstrated cases and delivers vivid talking head videos. It is obvious that EMO has smaller head motion compared to ours, perhaps due to the constraint of the face region mask used by it. Also, EMO seems less robust than ours in some cases, with artifacts – such as abrupt expression change, inaccurate lip sync, and subtle texture detail flickering – occasionally appears upon close inspection (note that their reported average lip-sync score is significantly lower than ours). On the other hand, however, EMO's video quality is slightly higher than ours in terms of sharpness, owing to their use of the large and powerful image generation foundation model.\n\n**Response to Limitations:** Thank you for the comments. We plan to handle upper-body/full-body and explore more explicit 3D representations in our future work (both projects are on-going).\n\n---\n\nWe hope we have addressed your questions. If not, it would be great to let us further know your concerns during discussion.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "5ufDeqnwyr", "reviewer": "Reviewer_xKzZ", "summary": "The paper present VASA, a new method for talking head video generation. The method is build as a diffusion model using Transformer. To increase the performance of the model and allow more control on the generated video the authors decided to use the representation from [1] and learn to disentangle its components. With this the model can control the gaze, expression and camera position. The method achieve impressive qualitative and outperforms the methods they compare with.\n\n\n\n[1] Nikita Drobyshev, Jenya Chelishev, Taras Khakhulin, Aleksei Ivakhnenko, Victor Lempitsky, and Egor Zakharov. Megaportraits: One-shot megapixel neural head avatars. In Proceedings of the 30th ACM International Conference on Multimedia, pages 2663–2671, 2022", "questions": "How do the authors deal with the resolution difference between methods when computing FVD ? \n\nWith CAPP score are the head motions related to speech semantics measured ? (e.g head shake when saying no.)\nto finish", "limitations": "The authors address the limitation in the appendix.", "rating": 7, "confidence": 5, "soundness": 3, "presentation": 4, "contribution": 2, "strengths": "The qualitative results are very impressive and probably better than the state-of-the-art.\nThe idea of disentangling the existing representation of [1] is nice.\nThe model is very fast at inference on costumer grade GPU.\nThe paper is well written and easy to understand.\n\n\n\n\n[1] Nikita Drobyshev, Jenya Chelishev, Taras Khakhulin, Aleksei Ivakhnenko, Victor Lempitsky, and Egor Zakharov. Megaportraits: One-shot megapixel neural head avatars. In Proceedings of the 30th ACM International Conference on Multimedia, pages 2663–2671, 2022", "weaknesses": "My main issue with the paper is that despite the work being presented as reproducible in the checklist I feel that many too many details are missing to actually reproduce the method or the experiments :\n- The new proposed CAPP score lack details for reproducibility e.g  training procedure. Sharing this model would also be  of interest to the community (and do not raise ethical concern which could justify keeping the code private). As of now it is difficult to know if the metric is actually sound. \n- It is unclear if the entire voxceleb dataset is used and how many clips remain after preprocessing ?\n- Lack of details of the new OneMin-32 dataset : size, type of videos, resolution, origins of the videos...\n- The paper says that the model is trained on \"4 NVIDIA RTX A6000 GPUs\" and that the model \"train on massive talking face videos from a large number of identities\". How long does the training take and how much data is actually used. Because if all of VoxCeleb + the new unreleased dataset are used the training could be very long. More details are required here.\n- It is not clear how the condition are used ion the network. Are they simply concatenated to the motion latent or used in cross Attention inside the Transformer\n-  It is not entirely clear if the architecture of [2] used out of the box to obtain the facial latent or if it was modified for the disentanglement.\n\nAssuming that the dataset used is very large (>10e6 samples) is the comparison against the other methods that use 50k-100k samples for training fair. An ablation with a training on a dataset of that scale would have been interesting. Without it the impressive qualitative results of the method could simply be due to the huge amount of data.\n\nThe novelty is limited the paper mostly reuse existing modules and innovation is mainly in the disentanglement.\n\nThe comparison against sota is limited, the most recent method, SadTalker, is from 2022.\n\nThe FVD score on VoxCeleb should have been shown anyway, other method from the literature present it.\n\nThe method apparently used [1] for gaze direction estimation. However [72] appears to be a method to classify gaze between different modes (fixed, quick motion...). Was the method modified to obtain gaze direction g that is used in the paper ?\n\nThe ablation only present results on the gaze and audio condition. It would have been interesting to also see the effect of the expression condition.\n\n\n\n\n[1] Raimondas Zemblys, Diederick C Niehorster, and Kenneth Holmqvist. gazenet: End-to-end eye-movement event detection with deep neural networks. Behavior research methods, 51:840–864, 2019.\n\n[2] Nikita Drobyshev, Jenya Chelishev, Taras Khakhulin, Aleksei Ivakhnenko, Victor Lempitsky, and Egor Zakharov. Megaportraits: One-shot megapixel neural head avatars. In Proceedings of the 30th ACM International Conference on Multimedia, pages 2663–2671, 2022.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "**Response to W1 (CAPP model sharing):** We will soon release the CAPP model, which we believe fills the missing piece of audio-pose alignment metric in talking face generation research and will be valuable to the community.\n\n**Response to W2 (Voxceleb training data):** We used the entire training set of Voxceleb2 for our model training. After filtering out invalid or low-quality videos, we ended up with approximately 170K training clips.\n\n**Response to W3 (OneMin-32 data details):** It contains 32 one-minute video clips of 17 subjects (Line 229-232). They are mostly educational lectures and coaching sessions sourced from online video sharing platforms. The resolution is 512x512.\n\n**Response to W4 (training time):** Our face latent model (encoder and decoder) takes around 7 days' training on a 4xA6000 workstation, and the diffusion transformer takes ~3 days. The total data used for training is ~500k clips (about 2-10 seconds each).\n\n**Response to W5 (condition usage):**  Yes, the condition signals are simply concatenated with noise along the temporal dimension as the input to the transformer. We'll clarify this in the paper.\n\n**Response to W6 (change to [2]):**  Our latent model architecture is same to [2]. We did not change the architecture but modified the training loss functions (Line 149-160) which are critical to achieve disentanglement (Figure A.8 and Line 344-350).\n\n**Response to W7 (data scale ablation):**  As mentioned above, our data size is ~500K (not *\">10e6\"*). In the attached one-page PDF, we add a data size ablation study and comparison with other methods, as per the reviewer's suggestion. We trained a model using only 10% of the data (i.e., 50k clips). As shown in Table I, this model achieve comparable audio-lip and audio-pose synchronization to the full-data model, though the FVD and $\\Delta$p metrics are not as good. This shows that our method performs well even with much less data, and more data enhances the motion diversity. Moreover, it still significantly outperforms other methods in all metrics assessing synchronization, motion intensity, and video quality.\n\n**Response to W8 (novelty):**  *First*, our motivation in the first place is, to model the human conversational behavior (facial dynamics and head movement ) *holistically using diffusion model* in a latent space that is agnostic to ID and appearance. This is our core innovation and, to the best of our knowledge, no previous methods have done this (it differs from the trend of further factor disentanglement and direct image generation; see our discussion in Line 47-55, 103-111). *Second*, in pursuit of the aforementioned goal, we did find the 3D-aided representation to be promising especially in terms of expressiveness and hence chose to leverage them. However, they can NOT meet our requirement of effective disentanglement. We made some insightful and provably critical modifications (Line 149-160, 344-350, and Figure A.8), without which we can never reach the current generation quality, esp. the liveliness with nuanced emotions. We perhaps have underemphasized the importance and contribution of such modifications, and will revise our presentation in the revision. \n\nApart from the two main contributions, our paper also offers other ones such as the design of face-factor-conditioned diffusion training, the CAPP model for filling the missing piece of pose-audio alignment metric, etc., which are also novel and valuable to the community.\n\n**Response to W9 (limited comparison to sota?):** To the best of our knowledge, there are no other published methods which can generate both audio-driven head poses and facial movements with single images. We mentioned some concurrent unpublished works in our paper, and have added the visual comparison with a concurrent work EMO in the one-page PDF. We'd appreciate it if the reviewer can point out some specific papers that we should compare with. \n\nRegarding the added comparison with EMO, we provide our results on some samples from EMO's official website (we are unable to provide video links per the rebuttal policy). As shown in Figure I, our method works consistently well and delivers vivid talking head videos. It is obvious that EMO has smaller head motion compared to us. Also, EMO seems less robust than ours in some cases, with artifacts – such as abrupt expression change, inaccurate lip sync, and subtle texture detail flickering – occasionally appear upon close inspection (note that their reported average lip-sync score is significantly lower than ours). On the other hand, however, EMO's video quality is slightly higher than ours in terms of sharpness, owing to their use of the large and powerful image generation foundation model.\n\n**Response to W10 (FVD on Voxceleb2) :** As shown in Table I of the attached PDF, we provide the FVD scores of different methods on Voxceleb2. However, it should be noted that the video quality of Voxceleb2 varies widely and is often low (see Figure II of the PDF). Hence the FVD score may not accurately reflect the true generation quality, as mentioned in our paper.\n\n**Response to W11 (gaze estimation) :**  Thanks for your careful reading. We found that we inadvertently cited the wrong paper: we actually used L2CS-Net [a] to extract gaze direction. Will fix this error in our revised paper.\n\n[a] Ahmed A.Abdelrahman, Thorsten Hempel, Aly Khalifa and Ayoub Al-Hamadi, L2CS-Net: Fine-Grained Gaze Estimation in Unconstrained Environments, 2022\n\n**Response to Q1:** The results of different methods are resized to the same resolution (224x224) for FVD evaluation.\n\n**Response to Q2:** This is an interesting question. For now we did not conduct in-depth analysis on whether the CAPP score captures the semantic relationship between speech and pose. We'll further explore this in our future work and thank you for the suggestion.\n\n---\n\nWe hope we have addressed your questions. If not, it would be great to let us further know your concerns during discussion.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:
We sincerely thank all reviewers for the valuable comments and suggestions. We are encouraged by the reviewer's acknowledgment that our paper:  *"innovatively defining the diffusion model within... which is quite interesting"*; *"convincingly demonstrate the effectiveness.."* (Reviewer WJD9); *"demonstrates excellent engineering and generation capabilities"*, *"leaving an impressive effect"* (Reviewer wMXj); *"visual presentation is excellent and leaves a lasting impression"* *"has made significant progress ..."* (Reviewer 1diB); *"qualitative results are very impressive"* (Reviewer xKzZ).

We'd like to reiterate our novelty and contributions here:

- We propose *diffusion-based holistic human face conversational behavior modelling* (facial dynamics and head movement), in a latent space that is agnostic to ID and appearance. This is our core innovation and, to the best of our knowledge, no previous methods have done this. It is a new approach which *differs from the recent trends of further facial factor disentanglement and direct image generation* (see our discussion in Lines 47-55, 103-111, 115-118).
- We build a highly disentangled latent space to achieve the aforementioned goal. Although we leveraged existing 3D-aided representation and models due to their high expressiveness, they can NOT meet our requirement of effective disentanglement. We made some insightful and provably critical modifications (Line 149-160, 344-350, and Figure A.8), without which we can never reach the current generation quality, esp. the liveliness with nuanced emotions. 
- We offer a few other supporting contributions including a controllable diffusion framework that enables flexible control of different face properties, and a new data-driven metric CAPP score for evaluating the alignment between audio and head pose.
- We advance audio-driven talking face generation to a new level of realism and liveliness not achieved before. Our work marks the dawn of real-time lifelike avatars which have the potential to reshape human-human and human-AI interactions across broad application domains.

We address each reviewer's questions and concerns under their respective reviews.

The attached one-page PDF contains the following figure and table contents:

- Visual comparison with EMO on EMO's official videos (Figure I)
- Sampled images from VoxCeleb2 to demonstrate the varied video quality and explain why we did not evaluate the FVD on it (Figure II)
- Training data scale ablation of our method and the requested FVD score on VoxCeleb2 (Table I)

Meta-review: Claimed contributions
 - Diffusion-based learning of the latent space for the human face and head movements, with the backbone of MegaPortraits by specially-designed losses to disentangle the face and head movement and to disentangle the identity and motions
 - Demonstration of the above with audio-driven human face video generation from a single image in real-time

Strengths
 - Superb results, both quantitatively and qualitatively
 - Novel insights into human face understanding with the proposed disentanglement of the face and head movements
 - New metric "CAPP" to evaluate the alignment of audio and head poses
 - Real-time performance with a compact latent diffusion model

Weaknesses
 - Largely depends on MegaPortraits

Even with its dependence on MegaPortraits, the reviewers and I all agree that the paper is insightful and impactful to human face video generation and human face understanding, thus recommending acceptance as an oral
Decision: Accept (oral)