VASA-3D: Lifelike Audio-Driven Gaussian Head Avatars from a Single Image
Authors: Sicheng Xu
Guojun Chen
Jiaolong Yang
Yizhong Zhang
Yu Deng
Stephen Lin
Baining Guo
Venue: NeurIPS 2025 poster ()
Keywords: 3DGS
Avatars
3D Talking Head
Single-shot
Primary area: applications

Abstract: We propose VASA-3D, an audio-driven, single-shot 3D head avatar generator. This research tackles two major challenges: capturing the subtle expression details present in real human faces, and reconstructing an intricate 3D head avatar from a single portrait image. To accurately model expression details, VASA-3D leverages the motion latent of VASA-1, a method that yields exceptional realism and vividness in 2D talking heads. A critical element of our work is translating this motion latent to 3D, which is accomplished by devising a 3D head model that is conditioned on the motion latent. Customization of this model to a single image is achieved through an optimization framework that employs numerous video frames of the reference head synthesized from the input image. The optimization takes various training losses robust to artifacts and limited pose coverage in the generated training data. Our experiment shows that VASA-3D produces realistic 3D talking heads that cannot be achieved by prior art, and it supports the online generation of 512x512 free-viewpoint videos at up to 75 FPS, facilitating more immersive engagements with lifelike 3D avatars.

VASA-3D: Lifelike Audio-Driven Gaussian Head Avatars from a Single Image
Sicheng Xu, Guojun Chen; Jiaolong Yang.
Yu Deng, Yizhong Zhang, Stephen Lin, Baining Guo
Microsoft Research Asia
Frichengxu,guoch jiaoyan, yizzhan, derpehaanndnnyy,stttce stevelin, baingu)) crosoft.com
Single videos
VASA-3D model
Audio
Iapthaad Control signals
Given single portrait image, approach produces animatable 3D Gaussian head that can be driven with any speech audio clip to generate lifelike, free-viewpoint talking face videos in real time. It adapts the powerful motion latent space of VASA-1 []] to 3D, and leverages the high realism of VASA-1 in 2D video generation to train the 3D head model. (Note: all the portrait
Figure 1:
images in this document virtual, non-existing identities generated by 22 3 See supplemental material for generated video samples with audio.)
Abstract
We propose VASA-3D, an audio-driven, single-shot 3D head avatar generator. This research tackles two major challenges: capturing the subtle expression details present in real human faces, and reconstructing an intricate 3D head avatar from a single portrait image. To accurately model expression details, VASA-3D leverages the motion latent of VASA-1 [ a method that yields exceptional realism and vividness in 2D talking heads. A critical element of work is translating this motion latent to 3D, which is accomplished by devising 3D head model that is conditioned on the motion latent. Customization of this model single image achieved through an optimization framework that employs numerous video frames the reference head synthesized from the input image. The optimization takes various training losses robust to artifacts and limited pose coverage in the generated training data. Our experiment shows that VASA-3D produces realistic 3D talking heads that cannot be achieved by prior art, and it supports the online generation of 512x512 free-viee niewpoitt videos up to 75 FPS, facilitating more immersive engagements with lifelike 3D avatars.
"Equal contributions. Corresponding author.
39th Conference on Neural Information Processing Systems (NcurIPS 2025).

---

Introduction
Advances in generating 3D head avatars are revolutionizing digital interaction, effectively bridging the gap between physical presence and virtual engagement. Vivid representations of human faces serve enhance various applications, ranging from virtual reality and gaming to remote education and online meetings. By conveying realistic facial expressions and movements, 3D head avatars can foster deeper sense of connection within virtual environments, making interactions more personal and engaging, and significantly improving user experience and immersion.
Most recent research 3D head avatars utilize parametric head representations derived from 3D [8] 19, 20]. The model's shape and motion personalized to image or video data of reference face, and then the avatar animation is driven by an audio or video track of what the avatar will say. Despite the impressive developments to date, significant challenges still remain. One major issue with current methods is that their output often lacks the nuanced motion and subtle expressions of real human faces, resulting in less visually compelling facial dynamics, Additionally, a vast majority of existing methods [4] 2 5 I0 I
scans (4 5 6 7 8 10 IT 12 B 14
T5 16 17
12 13 22
16 17 23 18] require video or multiview data of the reference face for avatar modeling,
limiting their utility.
In this paper, we present VASA-3D, an audio-driven 3D head avatar generator that transforms a single portrait image into a lifelike 3D talking head, synchronized with any speech audio input. The head avatar is modeled with 31) Gaussian splatting [24] 17 which ensures multiview consistency and facilitates real-time audio-driven animation and free-view rendering. Notably, the model captures and conveys dynamic expression details with a degree of realism that markedly exceeds current fatte-of-the-are techniques.
We observe that the expression terms of parametric head representations used for 3D head avatars, such as 3DMM (25 26 and FLAME 27 are modeled on 3D scans of just few hundred subjects. To model more diverse and detailed facial dynamics at minimal acquisition cost, VASA-3D instead takes advantage of 2D head videos, which are abundant online. Specifically, it employs the motion latent of VASA-1 [., which has been trained on data from 9.5K subjects, to capture rich facial dynamics. This motion latent, though learned on 2D data, encodes implicit 3D structure, and a key contribution of our work is in translating it to a 3D avatar. We accomplish this by first mapping it to the parameters of FLAME head model, which 3D Gaussians are bound as done in (17). Although FLAME's expression parameters are modeled on just hundreds of 4D face captures, VASA-3D addresses this limitation by next predicting dense, freeform Gaussian deformations that are conditioned on the motion latent, thereby enabling the generation of more expressive 3D dynamic heads.
Our latent motion controlled Gaussian avatar offers great potential for 3D talking head synthesis, but customizing it to just a single portrait image poses a challenging problem. Existing methods that personalize a head avatar representation using single image [8 14 15 19
20, (8 encode facial expression via a parametric head model, thus limiting expressiveness. Our solution is to utilize a pretrained portrait video generation model, namely VASA-1 M1 to transform the reference image into a collection of frames with varied facial expressions and head poses, and then fit the avatar to these frames. As there exist limitations with this synthetic data, well as overfitting issues due to dense deformation, we have developed an optimization framework with various training losses designed to robustly train the model despite these problems.
VASA-3D represents a significant step forward in 3D head avatar synthesis, capitalizing on 2D head videos to enrich its model of facial dynamics and allowing customization of this advanced model using only a single portrait image. The effectiveness and realism of VASA-3D is validated through various experiments, where it demonstrates clear superiority over recent techniques. By creating lifelike avatars that accurately reflect human expressions and facial motions, this approach paves the way for more immersive and engaging virtual experiences.
2 Related Work
3D Face and Head Representations. common representation for 3D head avatars is parametric mesh-based models such as 3DMM (25) and FLAME 27 These models are built on a collection of
3D head scans, from which the principal components of shape with respect to identity and expression used as bases for modeling head and face geometry. Though compact and efficient, these
2

---

parametric models provide low-fidelity mesh representations and limited detail for facial expressions, whose bases are derived from scanned data of only hundreds of subjects.
An alternative approach based on neural radiance fields (NeRFs) [29] does not explicitly represent geometry but instead stores the radiance field of head in neural network. With these radiance values, head appearance at novel views can be synthesized by volumetric rendering. This approach
has led to many works that can generate highly photorealistic head avatars 44 9 [3]. However, NeRF-based methods often require multiview images or video of the reference head, thus restricting their usage. Moreover, rendering speeds for NeRF-based models typically do not reach the levels needed for real-timm timl- applications.
Real-time performance with high rendering quality has been achieved through representations based on 3D Gaussians 17
30), whose positions, orientations, and densities are optimized for the reference head. By accounting for the visibility of each Gaussian and rendering only those that can be seen, real-time performance is attainable [24]. Rigging 31) Gaussians to a parametric head model allows them be dynamically controlled through parameter manipulation. Our work utilizes this representation but controls face and head motion using VASA-1 motion latents, for which residuals these Gaussians are incorporated to precisely model the subtle expression details that these latents capture.
18
20
23
3D Head Reconstruction. 3D head avatars can be reconstructed from a reference head using multi-view correspondences [7 18). For greater practical convenience, much attention has focused on one-shot methods that require only single head image. These methods either rely parametric head model as a strong prior & (0 or predict a volumetric [9] or tri-plane [14 [3] [19] representation for NeRF rendering. In this work, propose to leverage the considerable recent advance in 2D talking face generation [ to synthesize close approximations of additional views of the input face, providing further data for training.
collection of frames synthesized in our method may resemble monocular video input, which is [1 12 I3 [6]. However, our
used in several works for 3D head avatar reconstruction 4 5 training data differs significantly from monocular video in two respects. One is that a broad range of head poses and facial expressions can be synthesized with our approach, much beyond than what can reasonably be captured in a video of a reference head. The other is that the images generated by VASA-1 [ lack temporal texture consistency, which creates problems when using common training losses based on pixel-wise comparisons. We overcome this issue through judicious selection of losses that are robust to such artifacts.
6 I0
Head Avatar Animation. Head avatars are typically modeled in a way that they can driven using parameters of parametric models like 3DMM and FLAME 4 5
6
7
g
10 9
12
1
14 15
16 17 18 20). Their reliance on parametric models for animation encoding and control limits the expressiveness of faces. In contrast, our method drives animation using VASA-1 motion latents, which provide a richer expression representation learned from an abundance of 2D head videos. Though the 3D Gaussian splats that represent head shape in our work rigged to a FLAME model, we incorporate residuals conditioned on the motion latents, giving expression control to these latents.
3 Method
Our VASA-3D framework, illustrated in Fig. 2 is built on two main ideas: adapting the VASA- 1 motion latent to 3D, and leveraging the high realism of VASA-1 ① in 2D talking head video generation to facilitate single-shot customization of the 3D head model. We train VASA-3D models with carefully-designed losses that enhance visual quality while avoiding issues that may arise from the synthesized videos.
3.1 VASA-3D Model
3D Gaussian Representation. Our VASA-3D model is based on 3D Gaussians [24 equipped with Gaussian deformation fields driven by the VASA-1 motion latent. The 3D head is represented as a set of 3D Gaussians g = (g. (14; Ti, 8;, Ci, ox)) J each with position μ., rotation scale color c, and opacity a. To ease learning of Gaussian parameters and deformation fields, we make use of priors from existing 3D head parametric models. Specifically, use 3D Gaussians bound to
3

---

feece faiie
Appaxrance
deformation
deformation
Ail lake Kast
Agp
Single Fruree
Istore
Book geomiams
AIB
Denolee
Dhscadee
Auciic
23*
ZPP
Sysce
Surc
Motiee
deformatioon
VAS deformation
Synthetic Data Generation
VASA-3D Model and Training
the VASA-1 model to generate a collection of synthetic talking face videos as well as their corresponding motion latents, which are used to train VASA-3D model. The driving sources for these videos can be in-the-wild audios and/or face videos. Our VASA-3D model represented by deformable 3D Gaussians attached FLAME mesh. Two deformation fields are applied the Gaussians, mesh and another modulated by VASA motion latents. After training, VASA-3D model can be driven with VASA motion latents generated from audios or videos real time.
Figure 2: Overview of VASA-3D. Given single portrait image,
based the FLAME
FLAME model [31] a representation introduced in GaussianAvatars Unlike in and other previous works, animations will be driven by the VASA-1 latent.
We decompose the deformation into two parts: a Base Deformation, and another deformation we call VAS Deformation. The former is driven by the FLAME parametric model change the geometric properties of the Gaussians including position, rotation, and scale. The latter derives the fine-grained geometric and color variations, which crucial for expressing the motion nuances captured in VASA-1 and improving the rendering quality.
Base Deformation. Given motion latent x dyd pose] produced by the VASA-1 diffusion model, we first map them to FLAME parameters using two MLPs. The first MLP M converts the facial dynamics code z dyn to the expression-related FLAME parameters ===p = (sb, gave 0jaw) which includes the expression PCA coefficients, eye pose, and jaw pose, respectively. The second MLP MM uses pose to predict the FLAME pose parameters e² n@neck galobal t) including neck rotation, global rotation, and global translation. These two mappings can written as:
euup t (zdem) pose MP(2200s)
Both M and M have three fully connected layers, with 256 hidden units per layer followed by a ReLU activation function. Additionally, a shape coefficient shape is jointly optimized during training and fixed during inference.
Given these parameters, the FLAME mesh will be rigged accordingly, which drives the changes of (1,, Fú, St) for the Gaussians g; attached to the mesh triangles. Additional details of this process can be found in
VAS Deformation. We further learn dense Gaussian deformation fields for our 3D head model, modulated by VASA-1 motion latents. Two MLPs are introduced to predict the deformations of Gaussians in the face and neck region, respectively. The first MLP Dr takes Gaussians gi in FLAME's facial region face well as the VASA-1 facial dynamics latent dyyn as input and predicts the full transformation Âg; = (Ap,, Ar, As; Ac, Aos). We also feed the FLAME expression parameters exp the MLP that it is for the FLAAME neck region Qneck- It takes the Gaussians, VASA pose pose and FLAME pose parameters pose input predict the residuals. The VAS deformations be expressed as:
of the current base expression. The second MI.P responsible
Agies//cc D' (g., dym EeP))
Agjen,.cck <- DD (B) pose gose)
These two MLPs share the same architecture as M and MP except for the different input and output dimensions. For input Gaussian positions μa, we apply sinusoidal positional encoding with L = 4 29

---

Animation and Rendering, Once trained. VASA-3D models can be animated using VASA-1 motion latents. The driving sources can be either audios or videos. For audio input, we use VASA-I's diffusion transformer to generate motion latents. For videos, we use VASA-1 motion encoders for latent code extraction. The animation frames are efficiently rendered with Gaussian Splatting and the whole animation and rendering pipeline can run in real time on a commodity GPU.
3.2 Synthetic Training Data Generation
We leverage VASA-1 to generate video frames with diverse set of poses and expressions from the given single image. To achieve this, one can either use real speech audios and/or face videos for data generation. For example, in most of our experiments, we randomly sample up 10 hours of video clips from the VoxCeleb2 dataset [32 to render the training data. We extract the VASA-1 motion latent for each frame and use the VASA-1 decoder to drive the portrait image and synthesize the corresponding frames. The paired motion latent and video frame data will be used for VASA-3D model training, which we present next.
3.3 Robustified Model Training
We train our models in an end-to-end manner where all the trainable modules, including Gaussian parameters and the ML.ss for deformation, are trained together from scratch.
Challenges. Our synthesized training data and the dense free-form deformation in our 3DGS- -based head model give rise to several challenges for training.
Unlike real videos, inconsistency of temporal texture and facial shape exists among the synthesized frames.
Large viewing angles are often missing from the training data, leading to difficulties in shape reconstruction.
The inclusion of residuals for the Gaussians can lead to overfitting to the training video frames.
We apply the following losses to train the model effectively without succumbing to these issues.
Reconstruction Losses. We use combination of the structural similarity index measure (SSIM)
and the L, color difference as the photometric loss between the generated image and the ground truth image:
Lyecon = LasssmLssiin + (1 Assim L1.
(5)
Perceptual Losses. As temporal texture inconsistency reduce the efficacy of the photometric loss, we rely on perception-level losses that measure visual quality but are robust to temporal inconsistency among the training frames. Specifically, we apply the Learned Perceptual Image Patch Similarity (LPIPS) loss (33] with pretrained VGG network (34). To further improve realism, we add multi- scale patch discriminators and apply an adversarial loss for training. We employ three discriminators with different input image scales and trained them together with our model. The perceptual loss functions can be written as:
Lpere-
Aipips Lipips + Lodu-
(6)
SDS Loss. Since the synthesized video data generally covers limited range of poses, we apply the SDS loss [35] to minimize visual artifacts in side views and to enlarge the range of valid viewing angles. Specifically, we render model from random viewing angles for which we apply the SDS loss Lede- Random views are uniformly sampled from azimuth angles in the range 180 180"] and elevation angles in the range 2-2.55 22.5"). We use StableDiffusion v2.1 [36] as the diffuusin model in our SDS loss with classifier- free guidance factor 10.0 and gradient scale 0.001. The text prompt is 'huann portrait, realistic photography, , by DSLR camera'.
Though the dense VAS deformations help capturing detailed expressions and improving image quality, they also require careful design of regularization to avoid overfitting to each frame's training data. In light of this, we compute Trecon, L.per and Teyds for the rendered images of the Gaussians both after base deformation and after VAS deformation, denoted as g' and g", respectively. This
5

---

20
Setting
PSNR
L14
SSIM
LPIPSL
Set
Spl
27
26
Basic
25.74
0.0228 0.8544
0.0768
6.6347
8.1265
Average
25
+VAS deform.
27.19
0.0195
0.8654 0.0695 6.9636
7.9050
24
+Liads
27.23
0.0195
0.8653 0.0707
6.9581
7.9191
Training data size
23
- Smin -
10min -e- 20min + 30min
+cconsss
27.33
0.0192
0.8672 0.0706
6.9429
7,9221
1h
-6
- 10h
+Lcas
26.62
0.0209
0.8472 0.0657 6,9145 7,9422
22
50K 100K 150K 200K 250K 300K 350K 400K
Training lteration
GT
7,1517 7.7770
Figure 3: Effects of dataset size and train- ing iterations
Table 1: Quantitative results ablation studies
approach aims for G' to effectively capture facial features shared across frames to the extent possible through multi-frame consistency, while g" focuses on modeling the residual facial details of different frames.
Render Consistency Loss. Although the SDS loss helps to eliminate artifacts in profile regions, we found that it also tends to smooth out the details across all regions in our case. This issue is pronounced for G" because the Gaussian residuals are learned for each frame. Such flexibility makes G" susceptible to the side effect of the SDS loss, especially for regions not well captured by the current view (hence less constrained by the ground-truth reference image). In contrast, g' are less prone to this issue, as these Gaussians are optimized to jointly fit the multi-frame data with different poses and thus are less affected by Leds. Therefore, we design a loss to regularize G" with G'. Specifically, for each training iteration render additional pair of images with G' and G" respectively, under a new view angle significantly different from the current training view. We apply render consistency loss Leansist between these two images lg(g") and ('("")
Leonsist LPIPS (g"), stop_sgrroeee grad (g')),
where the noo-gradient operator prevents G' from being negatively affected.
In practice, the new view randomly sampled with the azimuth angle the range [-35%; -55%] or [35", 555) and elevation angle in the range [-15%, 15%.. Of the two azimuth ranges, we select the one that is farther away from the view of the training frame.
Sharpening Loss. To further increase the sharpness of the rendered results, we optionally apply (CAS) filter 37 to the model's rendered images and use them to further train the model. Specifically, we apply the CAS filter to the rendered image and then apply the LPIPS loss between the sharpened image and the original image. The CAS loss Lees is applied at the end of the training process as lightweight finetuning step.
a contras--daapiiv--haapeiii
The overall loss function is combination of the aforementioned losses:
L Lesim + L1 + Lipips Lado+
(8)
Lgds + Leonsiet + Leas + Lothers-
where the loss weights are omitted for brevity, and Lothers stands for other loss functions introduced in [17] such as the Gaussian position and scale losses (see (17]). More details about our losses and their implementation can be found in the supplementary material.
4
Experiments
4.1 Analysis and Ablation Study
We conduct experiments analyze our method well as the design choices. In the following experiments, we use ten portraits, including five males and five females, generated by StyleGAN2 2 to train models. We train the models using 4 NVIDIA A100 40G GPUs and batch size of 4. A 512x512 resolution is used for both the training data and VASA-3D rendering throughout this paper.
Inference Speed. Given an audio clip as input, the animation and 512 x 512 video frame rendering of our VASA-3D model can run at 75fps with a preceding latency of only 65ms, evaluated on single NVIDIA RTX 4090 GPU. A real-time demo is provided in the supplementary videos.

---

GT Reference
w/o Lops conder
Base Deform.
Ly
VAS Deform.
Figure 4: Left: VAS deformation not only improves image quality (see also Table 1 but also captures facial nuances subtle yet critical for expressing emotions. Right: The SDS loss eliminates artifacts in profile regions while the render consistency loss enhances the details smoothed out by the SDS loss.
w/o Leas
Leas
w/o Las
Leas
Figure 5: The CAS loss improves the overall rendering sharpness.
4.1.1 Dataset Size and Training Time
We first analyze the influence of dataset size (i.e., length of training videos synthesized by VASA-1) and training time (in iterations) on result quality. For each image, we use VASA-1 to render eight training datasets of different sizes, i.e., Smin, 10min, 20min, 30min, 1h, 2h, 5h, 10h, using VASA-1 latents extracted from random video clips in VoxCeleb2 [32]. We evaluate the models at varying training iteration numbers (up to 400K) on our test set, which are VASA-1 generated videos of 3min for each image.
Figure 3 shows how the average PSNR scores across the portraits the test set improves as the dataset size and training iterations increase. It is observed that the improvements almost plateau after the dataset size reaches 2 hours and after the number of iterations exceeds 200K. Therefore, we set the total number of iterations to 200K by default in the following experiments. We also trained our model with 20K iterations and compare it against other baselines in some experiments below. Since the training data size does not affect training time, we simply set it to 10 hours. For each portrait, 10-hour dataset can be rendered in less than hour on NVIDIA 1100 40G GPUs. Training with 20K/200K iterations takes about 1.8/18 hours for each model.
7

---

Figure 6: Example frames from the generation results of VASA-3D. The first row shows the frontal view and the second row presents side views of the same frames.
FID
Sct
Sp
ID Simt
VASA-1
5.24
8.142
6.9237
0.8154
VASA-3D
7.45
8.121
6,9300
0.7874
Table 2: Audio-driven generation comparison with VASA-1, the results of which represent our upper- bound performance training data is generated by it. VASA-3D has only subtle performance gap to VASA-1, yet providing true-3D, freeview-renderabb talking heads not achieved by VASA-1.
Some examples of our results generated by the default setting are presented in Fig Our method is shown to generate high-quality 3D head renderings with accurate audio-lip sync, vivid facial expressions, and lively head motions. Video results provided in the supplementary materials, where readers can more fully examine the quality of VASA-3D generation.
4.1.2 Effect of VAS Deformation
We further train different variants of our method with the same training and test setup, and the quantitative results are presented in Table where the evaluation metrics include PSNR, LI error, SSIM, LPIPS, and lip-audio synchronization score. For the lip-sync score, we use SyncNet 38] to assess the alignment confidence score Sc and feature distance Sp.
Thole the effect of our VAS Deformation, compared a basic setting with Base Deformation only. All the image quality and lip-sync metrics are significantly improved, demonstrating the importance of the VASA-latent-driven Gaussian deformation. Some visual comparisons are presented in Fig. The image quality is clearly improved by VAS deformation. More importantly, the facial expressions including subtle facial details follow the ground-truth reference frame more closely, displaying the expressive talking features with nuanced facial details that are modeled by VASA-1.
4.1.3 Effects of Different Losses
Fig. 4 exxibits improvements with the SDS loss and render consistency loss. Due to the limited pose coverage of our training data, artifacts can be clearly observed for the results without SDS loss under side views rendered at 45" azimuth angles. The SDS loss provides additional regularization to the model and eliminates the artifacts in side views. However, also tends to smooth out details. Adding the render consistency loss improves the results by enhancing the rendering details. Fig. shows results from training with the CAS loss, where the images are further sharpened.
Table I shows numerical results with different losses. The full method without the CAS loss (i.e., the fourth row) yields the best image quality in terms of the PSNR, LI and SSIM metrics, whereas the method with the CAS loss has the best perceptual score measured by LPIPS due to the enhanced image sharpness. The lip-sync scores remain stable with different loss functions incorporated.
4.2 Audio-Driven Generation and Comparisons
In this experiment, we construct our training datasets by collecting five random audio clips from the web (two males and three females), each with total length of 25 minute For each audio clip, we
OOr model can use either video frames audios for training. Here we use audios since some compared methods require continuous videos and cannot utilize the 10-hour dataset in Sec. 4.] containing short video clips.

---

Sct
Sp4
ID Simt
Video Qualityf
US Overall Preferencef
ER-NeRF
5,921
8.7788
0.7732
1.82
1.08%
GeneFace
5.922
9.6066
0.7857
1.73
0.72%
Mimic Talk
5.270
10,9368
0.7748
2.23
3,58%
TalkingGaussian
6.701
8.1061
0.7971
2.38
0.72%
VASA-3D
8.121
6,9300
0.7874
4.29
93.91%
VASA-3D (20k iter)
7,980
7.0020
Table 3: Comparison with audio-driven 3D talking head methods that trained videos. Note that all methods except ours do not generate head pose. "US" denotes User Study (see text for details). VASA-3D (20k iter) denotes our model trained with only 1/10 of default iteration number.
apply VASA-1 to drive a StyleGAN2 portrait image to generate the synthetic video data. We only use the first 20-minutes to train the models, and the remaining 5-minutes used as the test set.
Comparison with VASA-1. We compare our method to VASA-1, our training data generator, to check the quality difference of the generated talking videos. Table Sable?ppr presents the frame FID (39). LipSync (38 confidence score Sc and feature distance Sp, and the facial identity similarity between test video frames and driving portrait images, calculated with ArcFace [40]. The performance gap between VASA-3D and VASA-1 is small.
Comparison with Previous 3D Talking Head Avatar Methods. To our knowledge, no existing method deals with the same task as ours - iee., single photo expressive, fully-animatable 3D head avatar driven by audio making the comparison difficult. Most audio-driven 3D talking avatars are trained on long videos, and they typically do not generate full head dynamics such head pose.
Still, to facilitate comparison with state-of-the-ar techniques for reference purposes, we consider the following methods: ER-NERF 41], GeneFace [42), MimicTalk [16], and TalkingGaussian (23). We employ the same video data as in our VASA-3D to train these methods and use the audios from the test set to generate videos. Table 3 shows the evaluation results of different methods. Our model surpasses the other methods on the LipSync metrics by a wide margin. Its identity similarity score is marginally lower than TalkingGaussian (23 and better than others. We further conduct user studies to evaluate the rendered video quality and the overall realism of the audio-driven results. 15 participants were invited to assess: 1) the visual quality of the rendered talking head videos (audio muted), with ratings from 1 to 5, and 2) the user preference of the results from the five compared methods, judged by overall realism. Our visual quality rating was significantly higher than the other methods and the users chose the VASA-3D as the best one for 93.91% % of the presented cases.
More comparisons. We further compare our method with related works on video-driven 3D talking head animation (i.e., the face reenactment task). Note that face reenactment is not focus our work; the goal here is to further compare VASA-3D's rendered video quality as well as its expressiveness on facial expression against prior art. Details of this experiment can be found in the suppl. material.
4.3 Generation with Additional Control Signal
Inheriting the capabilities of VASA-1, our VASA-3D can take additional control signals besides an audio clip, such as eye gaze direction, head distance, and emotion offset. Fig. ass well as our supplementary video present typical results with emotion offset control, where the generated 3D talking heads closely adhere to different emotion offsets and exhibit emotive talking styles.
4.4 Artistic Image Experiments
We also tested our method on artistic-style portrait images, with some examples shown in Fig. and Fig. g Our method can effectively handle such artistic images and produce convincing 3D videos.
5
Conclusion
VASA-3D offers unparalleled realism for audio-driven 3D head avatars by presenting way to leverage the extensive expression data present in online 2D head videos. With meticulously designed architecture and training scheme, our model can be easily customized using only single

---

Neutral
Anger
Happiness
Sadness
Surprise
Figure 7: Audio-driven generation results with additional control signal of emotion offset. The results are generated with the same audio clip. See the accompanying video for animated results with audio.
Figure 8: Results artistic-style images. See our videos for animated results with audio.
portrait image. We believe approach paves the way for more immersive and engaging virtual experiences with 3D head avatars.
Limitations and Future Work. Our method still has several limitations. Limited by the viewing angles of the synthetic training videos, it does not model the back of heads. This issue could potentially be resolved through 3D inpainting, since the back of head mostly rigid. Similar to VASA-I, our method does not handle dynamic elements such as accessories. Extending VASA-3D to include the upper body is another interesting direction we will explore in future.
Societal Impacts and Responsible AI Considerations
Our research aims to support positive applications of virtual AI avatars and is not intended for creating mislcading or deceptive content. However, like other related techniques, VASA-3D could potentially be misused in generating the likeness of real person. Throughout the development of VASA-3D, responsible AI considerations were factored into all stages. To safeguard against such harm, we are training face forgery detection models that incorporate training data. Though VASA-3D produces visually realistic results, we have found that they are easily distinguishable from authentic videos by these models and improve the models' generalizability.
models' outputs part of the
While recognizing the potential misuse, it important to acknowledge the substantial positive impact that our research technique could eventually have. We are currently examining potential benefits, such as its application an Al coworker and AI tutor, which can enhance latent intelligence accessibility for knowledge workers and learners. These applications highlight the significance of this research and other related investigations. We are committed to developing AI responsibly, with the goal of advancing human well-being.
10

---

References
[1] Sicheng Xu, Guojun Chen, Yu-Xiao Guo, Jiaolong Yang, Chong Ii, Zhenyu Zang, Yizhong Zhang, Xin Tong, and Baining Guo. Vasa-1: Lifelike audio-driven talking faces generated in real time, arXiv preprint urXiv:2404. 10667, 2024.
[2] Tero Karras, Samuli Laine, Miika Aittala, Janne Hellsten, Jaakko Lehtinen, and Timo Aila. Analyzing and improving the image quality of StyleGAN. In Proc. CVPR, 2020.
[3] James Betker, Gabriel Goh, Li Jing, Tim Brooks, Jianfeng Wang, Linjie Li, Long Ouyang. Juntang Zhuang. Joyce Lee, Yufei Guo, et al. Improving image generation with better captions. htts://cd.. openai. -atppppprrsdalle---pp... 2(3):, 2023.
[4] Guy Gafni, Justus Thies, Michael Zollhofer, and Matthias Nießner. Dynamic neural radiance fields for monocular 4d facial avatar reconstruction. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 8649-8658, 2021.
[5] Philip-Wiliaam William Grassal, Malte Prinzler, Titus Leistner, Carsten Rother, Matthias Nießner, and Justus Thies. Neural head avatars from monocular rgh videos. In Proceedings of the IEEF/CVF conference on computer vision and pattern recognition, pages 18653-18664, 2022.
[6] Yufeng Zheng, Victoria Fernández Abrevaya, Marcel C Bühler, Xu Chen, Michael J Black, and Otmar Hilliges. Im avatar: Implicit morphable head avatars from videos. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pages 13545-135555 2022.
[7] Xuan Gao, Chenglai Zhong, Jun Xiang, Yang Hong. Yudong Guo, and Juyong Zhang. Reconstructing personalized semantic facial nerf models from monocular video. ACM Transactions on Graphics (TOG), 11-6) 2022.
[8] Taras Khakhulin, Vanessa Skiyarova, Victor Lempitsky, and Egor Zakharov. Realistic one-shot mesh-based head avatars. In ECCV, 2022.
[9] Yang Hong. Bo Peng, Haiyao Xiao, Ligang Liu, and Juyong Zhang, Headnerf: real-time nerf-based parametric head model. In CVPR, 2022.
[10] Yufeng Zheng, Wang Yifan, Gordon Wetzstein, Michael Black, and Otmar Hilliges. Pointavatar: Deformable point-based head avatars from videos. In Proceedings of the IEEE/CVF conference on computer vision and patlem recognition, pages 21057-21067, 2023.
[11] Wojeiech Zielonka, Timo Bolkart, and Justus Thies. Instant volumetric head avatars. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pages 4584-4584, 2023.
[12] Yuelang Xu, Lizhen Wang, Xiaochen Zhao, Hongwen Zhang, and Yebin Liu. Avatarmav: Fast 3d head avatar reconstruction using motion-aware neural voxels. In ACM SIGGRAPH 2023 Conference Proceedings, pages 10 2023.
[13] Xiaochen Zhao, Lizhen Wang. Jingxiang Sun, Hongwen Zhang, Jinli Suo, and Yebin Liu. Havatar: High- fidelity head avatar via facial model conditioned neural radiance field, ACM Transactions on Graphics. 43(1):1-16 16, 2023.
[14] Weichuang Li, Longhao Zhang, Dong Wang, Bin Zhao, Mulin Chen Zhigang Wang, Bang Zhang, Zhongjian Wang, Liefeng Bo, and Xuclong Li. One-shot iighfiiclity talking-head synthesis with deformable neural radiance field. In CVPR, pages 17969-17978. 2023.
[15] Xucting Li, Shalini De Mello, Sifci Liu, Koki Nagano, Umar Iqbal, and Jan Kautz. Generalizable onc-shot neural head avatar. In NewrIPS, 2023.
[16] Zhenhui Ye, Tianyun Zhong, Yi Ren, Ziyue Jiang, Jiawei Huang, Rongjie Huang, Jinglin Liu, Jinzheng He, Chen Zhang, Zehan Wang. et al. Mimictalk: Mimicking personalized and expressive 3d talking face in minutes. arXiv preprint arXiv:2410.0673: 2024.
[17] Shenhan Qian, Tobias Kirschstein, Liam Schoneveld, Davide Davoli, Simon Giebenhain, and Matthias Nießner. Gaussianavatars: Photorealistic head avatars with rigged gaussians. In Proceedings of the IEEE/CVF Conference on Computer Vision and Puttern Recognition, pages 20299-20309, 2024.
[18] Yuelang Xu, Bengwang Chen, Zhe Li, Hongwen Zhang, Lizhen Wang. Zerong Zheng, and Yebin Liu. Gaussian head avatar: Ultra fidelltyy head avatar via dynamic gaussians, In Proc. CVPR, 2024.
11

---

[19] Zhenhui Ye. Tianyun Zhong, Yi Ren, Jiaqi Yang. Weichuang Li. Jiawei Huang, Ziyue Jiang, Jinzheng He. Rongjie Huang. Jinglin Liu, al. Realdd-portrit: One-shot realistic 3d talking portrait synthesis. arXiv preprint #xXx:2440.08500 2024.
[20] Xuangeng Chu and Tatsuya Harada. Generalizable and animatable gaussian head avatar. In The Thirty- eighth Annual Conference on Neural Information Processing Systems, 2024.
[21] Yudong Guo, Keyu Chen, Sen Liang, Yong-Jin Liu, Hujun Bao, and Juyong Zhang. Ad-nerf: Audio driven neural radiance fields for talking head synthesis. In Proceedings of the IEEE/CVF international conference on computer vision, pages 5784 5794, 2021.
[22] Jiahe Li, Jiawei Zhang, Xiao Bai, Jun Zhou, and Lin Gu. Efficient region-aware neural radiance fields for high-fidelity talking portrait synthesis. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pages 7568 7578, 2023.
[23] Jiahe Li, Jiawei Zhang, Xiao Bai, Jin Zheng, Xin Ning, Jun Zhou, and Lin Gu. Talkinggaussian: Structure- persistent 3d talking head synthesis gaussian splatting. arXiv preprint arXiv:2404./5260 2024.
[24] Bernhard Kerbl, Georgios Kopanas, Thomas Leimkühler. and George Drettakis. 3d gaussian splatting for real-time radiance field rendering. ACM Trans. Graph., 42(():139-1 2023.
[25] V. Blanz and T. Vetter. A morphable model for the synthesis of 3d faces. In ACM SIGGRAPH, pages 187-194, 1999.
[26] B. Amberg, R. Knothe, and T. Vetter. Expression invariant face recognition with a morphable model. In International Conference on Automatic Face and Gesture Recognition, 2008.
[27] Tianye Li, Timo Bolkart, Michael Black, Hao Li, and Javier Romern, Learning a model of facial shape and expression from 4d scans. ACM Trans. Graph., 36(6):194-1, 2017.
[28] Xuangeng Chu, Yu Li, Ailing Zeng, Tianyu Yang, Lijian Lin. Yunfei Liu, and Tatsuya Harada. GPAvatar: Generalizable and precise head avatar from image(s). The Twelfth International Conference on Learning Representations, 2024.
[29] Ben Mildenhall, Pratul Srinivasan, Matthew Tancik, Jonathan Barron, Ravi Ramamoorthi, and Ren Ng. Nerf: Representing seenes as neural radiance fields for view synthesis. Communications of the ACM, 65(1):99 106, 2021.
[30] Liangxiao Hu, Hongwen Zhang, Yuxiang Zhang, Boyao Zhou, Boning Liu, Shengping Zhang, and Liqiang Nie. Gaussianavatar: Towards realistic human avatar modeling from a single video via animatable 3d gaussians. In Proceedings of the IEEE/CVF conference computer vision and pattern recognition, pages 634 644-644,220 2024.
[31] Tianye Li, Timo Bolkart, Michael. J. Black, Hao Li, and Javier Romero. Learning model of facial shape and expression from 4D scans. ACM Transactions Graphics, (Proc. SIGGRAPH Asia), (6(6):194:1 194:17, 2017.
[32] Joon Son Chung, Arsha Nagrani, and Andrew Zisserman. Voxceleb2: Deep speaker recognition. In Interspeech 20/8. ISCA, September 2018.
[33] Richard Zhang, Phillip Isola, Alexei A Efros, Eli Shechtman, and Oliver Wang. The unreasonable effectiveness of deep features as perceptual metric. In CVPR, 2018.
[34] Karen Simonyan and Andrew Zisserman. Very deep convolutional networks for large-scale image recogni- tion. In International Conference on Learning Representations, 2015.
[35] Ben Poole, Ajay Jain, Jonathan Barron, and Ben Mildenhall. Dreamfusion: Text-to-3d using 2d diffusion. arXiv preprint 22Xiv:2209.1499809 2022.
[36] Robin Rombach, Andreas Blattmann, Dominik Lorenz, Patrick Esser, and Bjöm Ommer. High-resolution image synthesis with latent diffusion models. Proceedings of the IEEF/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pages 10684-10695, June 2022.
[37] AMD. Fidelityfx contrast adaptive sharpening. 2019.
[38] J. S. Chung and A. Zisserman. Out of time: automated lip syne in the wild. In Workshop Multi-view Lip-reading, ACCV, 2016.
[39] Maximilian Seitzer. pytorch-fid: FID Score for PyTorch.
https://ggthuu..ca/rsiiitas
pytorch- August 2020. Version 0.3.0.
12

---

[40] Jiankang Deng. Jia Guo, Niannan Xue, and Stefanos Zafeiriou. Areface: Additive angular margin loss for deep face recognition. In Proceedings of the IEEE/CVF conference computer vision and puttern recognition, pages 4690-4699, 2019.
[41] Jiahe Li, Jiawei Zhang, Xiao Bai. Jun Zhou, and Lin Gu. Efficient region-aware neural radiance fields high-fidelity talking portrait synthesis. Proceedings of IEEE/CVF International Conference on Computer Vision (ICCV). pages 7568 7578, October 2023.
[42] Zhenhui Ye, Ziyue Jiang. Ren, Jinglin Liu, Jinzheng He, and Zhou Zhao. Geneface: Generalized and high-fidelity audio-driven 3d talking face synthesis. arXiv preprint arXiv:2301.13330 2023.
[43] Hao Zhu, Wayne Wu, Wentao Zhu, Liming Jiang, Siwei Tang. Zhang, Ziwei Liu, and Chen Change Loy. CelebV-IHQ: A large-scale video facial attributes dataset. In ECCV, 2022.
[44] Phong Tran, Egor Zakharov, Long-Ninn -hat Ho, Anh Tuan Tran, Liwen Hu, and Hao Voodoo 3d: Volumetric portrait disentanglement for one-shot 3d head reenactment. Proceedings of the IEEE/CVF Conference on Computer Vision and Puttern Recognition, 2024.
[45] Yu Deng, Duomin Wang, and Baoyuan Wang. Portrait4--2:: Pseudo multi-view data creates better 4d head synthesizer. arXiv preprint 1355: 2024.
13

---

NeurIPS Paper Checklist
1. Claims
Question: Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?
Answer: [Yes]
Justification: Yes, the claims match the paper's contributions and scope. See Section |
Guidelines:
The answer NA means that the abstract and introduction do not include the claims made in the paper.
The abstract and/or introduction should clearly state the claims made, including the contributions made in the paper and important assumptions and limitations. A No or
NA answer to this question will not be perceived well by the reviewers.
The claims made should match theoretical and experimental results, and reflect how much the results can be expected to generalize to other settings.
It is fine to include aspirational goals as motivation long is clear that these goals are not attained by the paper.
2. Limitations
Question: Does the paper discuss the limitations of the work performed by the authors? Answer: [Yes]
We discuss the limitations of our approach the appendix included in the supplementary material.
Justification:
Guidelines:
The answer NA means that the paper has no limitation while the answer No means that the paper has limitations, but those are not discussed in the paper.
The authors are encouraged to create a separate "Limitations" section their paper.
The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (e.g., independence assumptions, noiseless settings, model well-specification, asymptotic approximations only holding locally). The authors should reflect how these assumptions might be violated in practice and what the implications would be.
The authors should reflect on the scope of the claims made, e.g., if the approach was only tested on few datasets or with few runs. In general, empirical results often depend on implicit assumptions, which should be articulated.
The authors should reflect the factors that influence the performance of the approach. For example, facial recognition algorithm may perform poorly when image resolution is low images are taken in low lighting. Or speech-to-text system might not be used reliably to provide closed captions for online lectures because it fails to handle technical jargon.
The authors should discuss the computational efficiency of the proposed algorithms and how they scale with dataset size.
If applicable, the authors should discuss possible limitations of their approach to address problems of privacy and fairness.
While the authors might fear that complete honesty about limitations might used by reviewers as grounds for rejection, worse outcome might be that reviewers discover limitations that aren't 't acknowledged judgment and recognize that individual actions in favor of transparency play impor- tant role in developing norms that preserve the integrity of the community. Reviewers will be specifically instructed to not penalize honesty concerning limitations.
the paper. The authors should use their best
3. Theory assumptions and proofs
Question: For each theoretical result, does the paper provide the full set of assumptions and complete (and correct) proof?
Answer: [NA]
14

---

Justification:
This paper does not include theoretical results
Guidelines:
The answer NA means that the paper does not include theoretical results.
All the theorems, formulas, and proofs in the paper should be numbered and cross- referenced.
All assumptions should be clearly stated or referenced the statement of any theorems.
The proofs can either appear the main paper or the supplemental material, but they appear in the supplemental material, the authors are encouraged to provide short proof sketch to provide intuition.
Inversely, any informal proof provided in the core of the paper should be complemented by formal proofs provided in appendix supplemental material.
Theorems and Lemmas that the proof relies upon should be properly referenced.
4. Experimental result reproducibility
Does the paper fully disclose all the information needed to reproduce the main ex- perimental results of the paper to the extent that it affects the main claims and/or conclusions of the paper (regardless of whether the code and data are provided or not)?
Question:
Answer: [Yes]
The paper has provided the necessary information. Section presents the details of our training data construction, the network architecture, and all the training losses.
Justification:
Guidelines:
The answer NA means that the paper does include experiments.
If the paper includes experiments, No answer to this question will not be perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data are provided or not.
If the contribution is dataset and/or model, the authors should describe the steps taken to make their results reproducible or verifiable.
Depending on the contribution, reproducibility can be accomplished in various ways. For example, if the contribution is novel architecture, describing the architecture fully might suffice, or if the contribution is a specific model and empirical evaluation, it may be necessary to either make possible for others to replicate the model with the same dataset, or provide access to the model. In general. releasing code and data is often one good way accomplish this, but reproducibility can also be provided via detailed instructions for how to replicate the results, access to a hosted model (e.g.. in the case of large language model), releasing of a model checkpoint, or other means that are appropriate to the research performed.
While NeurIPS does not require releasing code, the conference does require all submis- sions to provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution. For example
(a) If the contribution is primarily a new algorithm, the paper should make it clear how reproduce that algorithm. (b) If the contribution is primarily
model architecture, the paper should describe the architecture clearly and fully.
(c) If the contribution is a new model (e.g., a large language model), then there should either be way to access this model for reproducing the results or way to reproduce the model (e.g., with an open-source dataset instructions for how to construct the dataset).
(d) We recognize that reproducibility may be tricky in some cases, in which case authors are welcome deseribe the particular way they provide for reproducibility, In the case of closed-source models, it may be that access to the model is limited in some way (e.g., registered users), but it should be possible for other researchers to have some path to reproducing or verifying the results.
5. Open access to data and code
Question: Does the paper provide open access to the data and code, with sufficient instruc- tions to faithfully reproduce the main experimental results, as described in supplemental material?
15

---

Answer: [Yes] for data and partial code
Justification: We can share our training data since they are synthesized videos, and part of the code for reproduction and comparison. However, due responsible Al considerations, we are not able to release our full code to prevent potential misuse such as deepfake for fraud. See our discussion in the paper.
Guidelines:
The answer NA means that paper does not include experiments requiring code.
Please see the NeurlPS code and data submission guidelines (https://nips.co cc/
for more details.
While we encourage the release of code and data, we understand that this might be possible, so "No" is an acceptable answer. Papers cannot be rejected simply for not including code, unless this is central to the contribution (e.g., for new open-source benchmark).
The instructions should contain the exact command and environment needed to run reproduce the results. See the NeurIPS code and data submission guidelines (https /CodeSubmeei "piips.c/ppulli//guu Sotpa///ggideeeeeCeuuuuubbiii ssionoolicy for more details.
The authors should provide instructions on data access and preparation, including how to access the raw data, preprocessed data, intermediate data, and generated data, etc.
The authors should provide scripts to reproduce all experimental results for the new proposed method and baselines. If only subset of experiments are reproducible, they should state which ones are omitted from the script and why.
At submission time, to preserve anonymity, the authors should release anonymized versions (if applicable).
Providing as much information as possible supplemental material (appended the paper) is recommended, but including URLs to data and code is permitted.
6. Experimental setting/details
Question: Does the paper specify all the training and test details (e.g., data splits, hyper- parameters, how they were chosen, type of optimizer, etc.) necessary to understand the results?
Answer: [Yes]
Justification:
We included the experimental setting and details in Section 4
Guidelines:
The answer NA means that the paper does not include experiments.
The experimental setting should be presented in the core of the paper to level of detail that is necessary to appreciate the results and make sense of them.
The full details can be provided either with the code, in appendix, as supplemental material.
7. Experiment statistical significance
Question: Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?
Answer: [Yes]
Justification: Our method was evaluated on datasets with sufficient data samples and the results are statistically meaningful; see Section for details.
Guidelines:
The answer NA means that the paper does not include experiments.
The authors should answer "Yes" if the results are accompanied by error bars, confi- dence intervals, statistical significance tests, least for the experiments that support the main claims of the paper.
The factors of variability that the error bars are capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, or overall with given experimental conditions).
16

---

The method for calculating the error bars should be explained (closed form formula, call a library function, bootstrap, etc.)
The assumptions made should be given (e.g., Normally distributed errors).
It should be clear whether the error bar is the standard deviation the standard error of the mean.
It is OK to report 1-sigma error bars, but one should state it. The authors should preferably report a 2-sigma error bar than state that they have 96% CI, if the hypothesis of Normality of errors is not verified.
For asymmetric distributions, the authors should be careful figures symmetric error bars that would yield results that are out of range (e.g. negative rates).
show in tables or
If error bars are reported in tables or plots, The authors should explain the text how they were calculated and reference the corresponding figures tables in the text.
8. Experiments compute resources
For each experiment, does the paper provide sufficient information on the puter resources (type of compute workers, memory, time of execution) needed to reproduce the experiments?
Question:
Answer: [Yes]
Justification:
We reported the compute resources in Section 4
Guidelines:
The answer NA means that the paper does not include experiments.
The paper should indicate the type of compute workers CPU or GPU, internal cluster, or cloud provider, including relevant memory and storage.
The paper should provide the amount of compute required for each of the individual experimental runs as well estimate the total compute.
The paper should disclose whether the full research project required more compute than the experiments reported in the paper (e.g., preliminary failed experiments that didn't 't make it into the paper).
9. Code of ethics
Question: Does the research conducted in the paper conform, in every respect, with the NeurIPS Code of Ethics
Answer: [Yes]
Justification: The research presented in the paper adheres to all aspects of the NeurIPS Code of Ethics.
Guidelines:
The answer NA means that the authors have reviewed the NeurIPS Code of Ethics.
If the authors answer No, they should explain the special circumstances that require deviation from the Code of Ethics.
The authors should make sure to preserve anonymity (e.g., there a special consid- eration due to laws or regulations in their jurisdiction).
10. Broader impacts
Question: Does the paper discuss both potential positive socictal impacts and negative societal impacts of the work performed?
Answer: [Yes]
Justification:
We discussed the positive and negative societal impacts in supplementary.
Guidelines:
The answer NA means that there is no societal impact of the work performed.
If the authors answer NA or No, they should explain why their work has no societal impact why the paper does not address societal impact.
17

---

Examples of negative societal impacts include potential malicious unintended uses (e.g., disinformation, generating fake profiles, surveillance), faireess considerations (e.g., deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations.
The conference expects that many papers will be foundational research and not tied to particular applications, let alone deployments. However, if there is direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point out that generic algorithm for optimizing neural networks could enable people to train models that generate Deepfakes faster.
The authors should consider possible harms that could arise when the technology is being used intended and functioning correctly, harms that could arise when the technology is being used as intended but gives incorrect results, and harms following from (intentional unintentional) misuse of the technology.
If there are negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition to attacks, mechanisms for monitoring misuse, mechanisms monitor how system learns from feedback over time, improving the efficiency and accessibility of ML).
11. Safeguards
Does the paper deseribe safeguards that have been put in place for responsible release of data or models that have a high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)?
Question:
Answer: [Yes]
Justification: Yes, we have described safeguards; see the appendix for discussions.
Guidelines:
The answer NA means that the paper poses no such risks.
Released models that have high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere to usage guidelines or restrictions to access the model or implementing safety filters.
Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images.
We recognize that providing effective safeguards is challenging, and many papers do not require this, but we encourage authors to take this into account and make best faith effort.
12. Licenses for existing assets
Are the creators or original owners of assets (e.g., code, data, models), used in the paper, properly credited and are the license and terms of use explicitly mentioned and properly respected?
Question:
Answer: [Yes]
Justification: We cited the paper for the model/dataset we used in our paper.
Guidelines:
The answer NA means that the paper does use existing assets.
The authors should cite the original paper that produced the code package dataset.
The authors should state which version of the asset used and, possible, include URL.
The name of the license (e.g., CC-BY 4.0) should be included for each asset.
For scraped data from a particular source (e.g., website). the copyright and terms of service of that source should be provided.
If assets are released, the license, copyright information, and terms of use in package should be provided. For popular datasets, paperswithcone com/ datasets has curated licenses for some datasets. Their licensing guide can help determine the license of dataset.
18

---

For existing datasets that are re-packaged, both the original license and the license of the derived asset (if has changed) should be provided.
this information is not available online, the authors are encouraged to reach out to the asset's s creators.
13. New assets
Question: Are new assets introduced in the paper well documented and is the documentation provided alongside the assets?
Answer: [NA]
Justification:
We have assets to release.
Guidelines:
The answer NA means that the paper does not release new assets.
Researchers should communicate the details of the datasettcoodemmodl as part of their submissions via structured templates. This includes details about training, license, limitations, etc.
The paper should discuss whether and how consent was obtained from people whose asset is used.
At submission time, remember anonymize your assets (if applicable). You can either create an anonymized URL or include an anonymized zip file.
14. Crowdsourcing and research with human subjects
Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given to participants and screenshots, if applicable, as well details about compensation (if any)?
Answer: [Yes]
Justification:
See the supplementary document for user study details including screen-
shots.
Guidelines:
The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.
Including this information in the supplemental material is fine, but if the main contribu- tion of the paper involves human subjects, then as much detail as possible should be included the main paper.
According to the NeurIPS Code Ethics, workers involved in data collection, curation, or other labor should be paid at least the minimum wage in the country of the data collector.
15. Institutional review board (IRB) approvals or equivalent for research with human subjects
Question: Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or an equivalent approval/review based on the requirements of your country or institution) were obtained?
Answer: [NA]
Justification: This paper does involve crowdsourcing research with human subjects.
Guidelines:
The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.
Depending on the country in which research is conducted, IRB approval (or equivalent) may be required for any human subjects research. If you obtained IRB approval, you should clearly state this in the paper.
We recognize that the procedures for this may vary significantly between institutions and locations, and we expect authors to adhere the NeurIPS Code of Ethics and the guidelines for their institution.
19

---

For initial submissions, do not include any information that would break anonymity (if applicable), such as the institution conducting the review.
16. Declaration of LLM usage
Question: Does the paper describe the usage of LLMs if it is important, original, or non-standard component of the core methods in this research? Note that the LLM is used only for writing, editing, formatting purposes and does not impact the core methodology, scientific rigorousness, or originality of the research, declaration is not required.
Answer: [NA]
Justification: This research does involve LLMs in any aspects.
Guidelines:
The answer NA means that the core method development in this research does not involve LLMs as any important, original, non standaadd components.
Please refer to our LLM policy for what should or should not be described.
20

---

A More Training Details
For the SDS loss Lids. we apply it every 10 iterations due to its high computation cost. We apply Luds with time step E in normalized range of Emin = 0.02 to maa 0.98, with fmar decaying by 0.98 every 2,000 iterations. The CAS loss Leas is applied after 200K iterations, and the model is fine-tuned for an additional 20K iterations with Leas and other losses.
In all our experiments, the loss weights are set as Assim 0.1, Atpips 1.0, Xade 0.001, Asds = 1.0, Aconsist = 0.01, and Apes 10.0. These weights are empirically chosen without careful tuning.
As mentioned in Sec. 4.1 of the main paper, our models are trained for 200K iterations by default, excluding the CAS loss finetuning iterations. Gaussian densification and pruning start at the 10K iterations, with intervals of 2K iterations. We stop this process after 100K iterations or when the total number of Gaussians exceeds 200,000.
B More Experimental Results
Comparison with 3D Talking Head Avatar Methods. Sec 4.2 of the main paper compared our method against some 3D talking head avatar models, with numerical results provided including user studies. Fig A presents some examples. Our method is shown to generate renderings with accurate audio-lip syne, vivid facial expressions, and lively head motions, surpassing the capabilities of existing 3D talking head avatar methods.
3D head
Fig. B shows the screenshot of our user study interface. To assess the visual quality of the rendered videos, we asked the participants to assign satisfaction scores from 1 to 5. Videos were presented one at time, with the play order of different methods randomized for each test case. We asked the participants to provide their own judgment of satisfaction when watching a talking avatar on screen. Note that individual satisfaction levels may vary; however, the averaged scores provide a fair basis for comparison as each participant rated results from all methods. To evaluate user preferences for overall realism, we display the results of all compared methods side by side and ask the participants to select the one that looks the most realistic to them. Method names remained anonymous and their orders are randomly shuffled for each test case.
Comparison with 3D Face Reenactment Methods. As mentioned in Sec 4.2 we further compare our method with related works on video-driven 3D talking head animation, Le., the face reenactment task. Note that face reenactment is not the focus of our work; the goal here is to further check our video quality and its expressiveness on facial expression.
Specifically, we collect 26 portraits, each with I-minute high-quality talking videos, from the CelebVHQ [K] dataset. For each portrait, we randomly selected one frame from the video and use the VASA-1 decoder to render 10 hours of training frames from it, with VASA-1 latents extracted from random VoxCeleb2 video clips. With the collected 1-minute real talking videos as test sets, we compared our model with the following video-driven 3D head avatar methods: GAGAvatar 20), GPAvatar 28). Pearri (19), Voodoo3D 44 and Portrait4D-v2 (45).
Table A shows the averaged results of these methods with different metrics. Our model outperforms all the other methods under all the metrics.
PSNR PSNR
L11
SSIM
LPIPSJ
Sct
Sp.
GAGAvatar
25.74
30.53
0.0257
0,8695
0,0829
5.502
8.693
GPAvatar
24.91
29.41
0.0288
0.8583
0.1016
4.785
9.256
Peatttrrttrait
23.78
28.04
0.0338
0.8481
0.1091
4.971
9.179
Voodoo3D
23.43
28.39
0.0343
0.8380
0.1209
4.307
9.500
Portrait4D-v2
23.19
27.55
0.0356
0.8325
0.0946
5.823
8,530
VASA-3D
26.21
31.11
0.0255
0.8741
0.0760
6.453
7.996
GT
6.673
7,802
VASA-1
25.93
31.01
0.0261
0.8544
0.0809
6.302
8.061
Table A: Comparison with video-driven 3D face reenactment methods.
21

---

Reference
ER-NeRF
GeneFace
MimicTalk
TalkingGaussian
Ours
"diferennt
"need"
"popular"
"only"
"when"
"line"
"deal"
'ace
Figure A: Visual examples of audio-driven 3D talking head generation. Note: all methods except ours do not produce head pose, so we apply the pose sequences in training data for them. Bext viewed with zoom; supplementary video for comprehensive comparisons.
22

---

Video Quality User Study
User Preference User Study
B
D
E
LSC
Figure B: The user study interface for result comparison with existing audio-driven 3D talking head avatar methods. Left: To assess the visual quality of the rendered videos, we asked the participants assign satisfaction scores from 5. Videos were presented one time, with the play order of different methods randomized for each test case. We asked the participants to provide their own judgment of satisfaction when watching talking avatar on screen. Note that individual satisfaction levels may vary; however, the averaged scores provide a fair basis for comparison as each participant rated results from all methods.. Right: To evaluate user preferences for overall realism, display the results of all compared methods side by side and ask the participants to select the one that looks the most realistic to them. Method names remained anonymous and their orders are randomly shuffled for each test case.
23


Reviews:
[{"review_id": "6IwJfF7Ifk", "reviewer": "Reviewer_9EUc", "summary": "The paper introduces VASA-3D, a method for generating lifelike, audio-driven 3D head avatars from a single portrait image. The key innovation lies in leveraging the motion latent space of VASA-1 to animate a 3D Gaussian splatting model. By synthesizing diverse facial expressions and poses via VASA-1, the method trains a 3D head model that supports free-viewpoint rendering and real-time animation. A robust training framework with tailored losses addresses challenges like temporal inconsistency and limited pose coverage in synthetic data. Experiments demonstrate superior realism and performance compared to existing methods, with 512×512 videos rendered at 75 FPS.", "questions": "1. How does VASA-3D handle identity preservation when synthesizing expressions not present in the input image?\n2. What are the trade-offs between using FLAME as a base model and alternative parametric representations?\n3. Can the method generalize to non-frontal input images or low-resolution portraits?\n4. How sensitive is the training to the choice of VASA-1 as the motion latent source?", "limitations": "1. Synthetic Data Dependency: Relies on VASA-1 for training data, which may not capture all real-world variations.\n2. High Resource Requirements: Time and computational demands for training may restrict accessibility.\n3. Artistic Image Support: While tested on artistic portraits, real-world applicability remains unexplored.", "rating": 5, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 4, "clarity": 4, "significance": 4, "originality": 4, "strengths_and_weaknesses": "Strengths:\n1. Single-Image Customization: Achieves high-quality 3D head avatars from a single image, overcoming the need for multiview data.\n2. Integration of 2D and 3D: Effectively translates 2D motion latent into 3D deformations, enabling expressive animations.\n3. Real-Time Performance: Supports free-viewpoint rendering at 75 FPS, making it practical for immersive applications.\n4. Strong Empirical Results: Outperforms state-of-the-art methods in metrics like FID, lip-sync accuracy, and user studies.\n\nWeaknesses:\n1. Reliance on VASA-1: Synthetic data quality depends on VASA-1, which may inherit its limitations.\n2. Computational Cost: Requires extensive synthetic data (10 hours per portrait) and 18-hour training, limiting scalability.\n3. Limited Generalization: Tested primarily on StyleGAN2-generated portraits; performance on real-world images is unverified.\n4. Code Unavailability: VASA-1 is not open sourced", "was_revised": true, "final_justification": "I appreciate the authors' detailed response. Most of my concerns got addressed fairly. I keep the final score.", "rebuttal": "Thanks for your recognition of our work. We appreciate your positive feedback and address your questions as follows.\n \n \n**W1 & L1:** Thank you for the comment. Indeed, using synthetic data created by VASA-1 (or other talking head video generation method) may introduce distribution gaps to real data. We'll add more related discussions in the limitation section. Nevertheless, we found that our training data enables VASA-3D to generate vivid and realistic talking heads that exhibit clear superiority over other techniques. Also note that VASA-3D could also leverage future advances in 2D talking head generation for improved training.\n \n**W2 & L2:**\nSince training is done offline, efficiency was not our main focus. However, we show in the following that strong results can still be achieved with much less computation.\n \nTraining data size and generation speed:\n \nAlthough 10 hours of training video is the default setting, it is shown in Fig. 3 that similar results could be obtained with just 2 or 5 hours of data. Note that training data generation is fast: <1 hour for 10 hours of video (Line 244), or ~10 min for 2 hours of data.\n \nFurthermore, our comparisons with other methods in Sec. 4.3 used only *20 minutes* of audio-generated video data for training (because some of the methods require continuous video and cannot utilize the collection of video clips in our 10-hour data). The results show that VASA-3D trained with 20-min data still generates highly realistic 3D talking heads, surpassing the other methods by a wide margin across metrics evaluating lip-sync, video quality, etc. (see \"5. Methd_Comparisons.mp4\" and Table 3). The 20-min data can be generated in <2min.\n \nTraining time:\n \nWe wish to point out that 18 hours of training time is reasonable for 3D model optimization on videos. For reference, we report the training time of different methods in Table 3 of the main paper. For the compared methods, we stop the training when convergence was observed on our training data. Note that we kept their original implementation of running on a single GPU unchanged.\n \nWe did not carefully optimize the training framework or tune the hyperparameters for efficiency considerations. But the training time can be reduced with more GPUs or implementation optimizations (e.g., less frequent SDS loss computation which is the most costly part).  For the comparison below, we found that our model trained with 20K iterations (1.8 hours) already significantly outperforms other methods and nearly reaches the performance of the model trained with 200K iterations.\n \n| |$S_C\\uparrow$  |$S_D\\downarrow$ | FID $\\downarrow$ |\n| ---- |---- |---- |---- |\n|  ER-NeRF (26h, 1 GPU) |  5.921 |8.7788 | 18.36 |\n|  GeneFace (25h, 1 GPU) |  5.922 | 9.6066 | 26.33 |\n|  MimicTalk (32h, 1 GPU) | 5.270 |10.9368 | 20.85 |\n|  TalkingGaussian (3h, 1 GPU) |  6.701 | 8.1061 | 23.62 |\n|  Ours-20K (1.8h, 4GPU)  | 7.980 |7.0020 | 10.35\n|  Ours-200K (18h, 4GPU) |  8.121 | 6.9300 | 7.45 |\n \nIn summary, the training cost is reasonable, and significant computational reductions (down to 20 minutes of training video and 1.8 hours of training) still lead to high quality results. We intend to study efficiency improvements in future work.\n \n**W3 & L3:** There were experiments on real-world images, and the performance was equally good to StyleGAN2-generated portraits. Please see Line 36, Table I, and Fig. V in the supplementary pdf, and the video results in \"2.Main_Results.mp4\" (02:03-02:59).\n \n**W4**: We plan to release the code and some processed synthetic training data of VASA-3D for research purposes, which will facilitate checking the details of our VASA-3D implementations and exactly reproducing some typical VASA-3D avatars and the driving sequences shown in our paper. \n \n**Q1:** Identity preservation under novel expressions is primarily handled by VASA-1. The synthetic training videos generated by VASA-1 effectively preserve identity with novel head poses and expressions, as demonstrated in their paper.\n \n**Q2:** FLAME is a well-established 3D morphable model that provides a parametric representation of human faces, and it's widely used in previous 3D Gaussian avatar methods such as [a,b,c]. We followed these works to use FLAME and did not try alternative parametric models. Other parametric models are applicable as well and we'll explore the trade-offs in our future work.\n \n \n**Q3:** Yes, VASA-3D can generalize to non-frontal input images (with yaw angles up to ~45 degrees) and low-resolution portraits.\n \n \n**Q4:** The choice of VASA-1 as the motion latent source is based on its proven ability to generate expressive talking videos with diverse head poses and expressions, which are essential for supervising our 3D Gaussian model training. Although we have not explicitly tested other motion latent sources, our method does not depend on any specific architecture of VASA-1 or any specific properties of VASA-1's latent space.\n \n \n[a] S Qian et al. GaussianAvatars: Photorealistic Head Avatars with Rigged 3D Gaussians\n \n[b] PW Grassal et al. Neural Head Avatars from Monocular RGB Videos\n \n[c] Y Zheng et al. I M Avatar: Implicit Morphable Head Avatars from Videos\n \n---\n\nThank you once again and please let us know if you have any further questions."}, {"review_id": "aiVtNPG0XN", "reviewer": "Reviewer_trR7", "summary": "This paper presents a method to create a human head avatar from a single image that can be driven by audio. Given an input image, it uses a 2D diffusion model, VASA-1, to generate video frames to form the training dataset, which is then used to optimize a 3DMM-driven 3DGS avatar model, GaussianAvatars. With the aid of the powerful motion space of VASA-1, this method can capture a wider range of facial expressions and body movements, which better align with the audio input. The authors also explore several regularization techniques to enhance visual details without harming 3D consistency. Solid experiments and sufficient visual results are provided to exhibit the effectiveness of this method.", "questions": "- Given the existence of two separate networks for the face and the neck, how can one tell if a Gaussian splat belongs to the face or the neck?", "limitations": "yes", "rating": 5, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 4, "clarity": 4, "significance": 3, "originality": 3, "strengths_and_weaknesses": "# Strengths\n1. This is a well-written paper with a strong motivation and impressive execution. The visual results, especially the video demos, are very convincing - the rendered avatars show great 3D consistency, sharp details, and expressive facial movements that align well with the audio.\n1. The method is concise and self-contained - each component appears for a clear reason and is consolidated with sufficient experiments. The long list of loss terms seems verbose, but they are actually all necessary, as shown in Table 1 and Figure 4&5.\n1. It is very nice to see the rendering of \"Base Deform\" and \"+VAS Deform\" side-by-side in Figure 4, which clearly shows the advantage of using the broader motion space of VASA-1.\n1. The ablation study of dataset size and number of iterations shown in Figure 3 is a good example of scaling up in an effective way.\n\n# Weaknesses\n1. The efficiency and  latency of this method during inference is not reported. Does it support realtime audio-driven synthesis given a trained avatar?", "was_revised": true, "final_justification": "3DGS avatars often take many views and frames to become robust to viewing points and animations, while simply plugging in the SDS loss leads to artifacts in unseen views. This method combines the generative ability of VASA-1 with the 3D consistency of 3DGS, producing impressive visual and quantitative results. Given that the concerns of myself and other reviewers are resolved, I am glad to still vote for acceptance.", "rebuttal": "Thank you for the positive feedback on our work. We appreciate your recognition of the contribution and effectiveness of our method. We address your questions as follows.\n \n**W1:** Given an audio clip and a trained VASA-3D model, our method can generate rendered frames at **75 fps** (i.e., ~13ms for one frame) with a **65ms** latency. The speed and latency were measured on one RTX 4090 GPU; see Line 282 of the main paper. So yes, **it supports real-time audio-driven synthesis**. A recorded real-time demo is also provided in the supplementary video (\"1.Realtime_Demo.mp4\").\n \n \n**Q1:** The face and neck regions are pre-defined on the FLAME template mesh. Since 3D Gaussians are bound to the triangles of the deformed FLAME mesh, the regions they belong to can be easily derived.\n \n---\n \nKindly let us know if you have any further questions."}, {"review_id": "s1X4nRo9zQ", "reviewer": "Reviewer_iSKP", "summary": "This paper builds upon VASA-1 and 3D Gaussian Splatting (3DGS) to extend the task of talking head generation into the 3D domain. By leveraging VASA-1 to render a large number of video frames from a single input image, the authors train a 3DGS-based model. Furthermore, they utilize the motion latent from VASA-1 to generate realistic 3D talking head videos with relatively fast inference speed.", "questions": "1. Will the authors release the code and models introduced in this paper?\n2. Since VASA-1 already supports some degree of head pose and camera movement control to produce pseudo-3D effects, how does the proposed method compare against VASA-1 under such settings?", "limitations": "yes", "rating": 4, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 3, "significance": 2, "originality": 2, "strengths_and_weaknesses": "**Strengths:**\n\n1. The proposed method leverages VASA-1’s motion latent to enhance the expressiveness of facial motions in the 3DGS framework.\n2. The paper presents several impressive demo videos that are visually convincing and contain relatively few artifacts under various head poses and expressions.\n3. The proposed base deformation and VASA deformation modules are well-motivated and experimentally validated.\n\n**Weaknesses:**\n\n1. If I have not missed any key details, the proposed approach seems to heavily rely on VASA-1 to generate long video sequences from a single input image to train an avatar-specific 3D model. This significantly limits the method’s applicability, generalization ability, and efficiency in terms of storage and scalability.\n2. While the use of motion latents from VASA-1 is claimed to be novel, it resembles prior approaches that condition 3DGS on dynamic face maps (e.g., SEGA[1], ExAvatar[2]) to compensate for the limited expressiveness of 3DMM-based representations. Additionally, the strong dependency on the non-open-source VASA-1 makes the method difficult to reproduce or benchmark.\n3. Compared to some recent 3DGS-based avatar generation methods such as HRAvatar[3] and RGBAvatar[4], this method appears to require significantly more training video data.\n4. From an implementation and benchmarking perspective, the paper overlooks a potentially fairer baseline: using an end-to-end diffusion-based talking head generation model[5,6]  to synthesize a training video from a single image, followed by training a 3DGS-based avatar model [3,4] from that video for comparison.\n5. Although the authors introduce an SDS loss to mitigate artifacts from VASA-1 outputs (e.g., misalignments and large-pose distortions), this solution substantially increases the training cost. Moreover, Table 1 shows only marginal quantitative gains from this additional loss.\n\n[1] SEGA: Drivable 3D Gaussian Head Avatar from a Single Image\n\n[2] Expressive Whole-Body 3D Gaussian Avatar\n\n[3] High-Quality and Relightable Gaussian Head Avatar\n\n[4] RGBAvatar: Reduced Gaussian Blendshapes for Online Modeling of Head Avatars\n\n[5] HunyuanPortrait: Implicit Condition Control for Enhanced Portrait Animation\n\n[6] Hallo: Hierarchical Audio-Driven Visual Synthesis for Portrait Image Animation", "was_revised": true, "final_justification": "I appreciate the authors' response. While the method has clear limitations, it demonstrates strong performance, and some of my initial concerns have been alleviated. Accordingly, I am inclined to moderately increase my score.", "rebuttal": "Thank you for the constructive feedback, and we address the questions as follows.\n\n**W1:** Thank you for the comment. We’d like to clarify that the video sequences used for model training do not necessarily need to be long. In fact, the model can be trained on only a collection of individual images (no videos) or a collection of short video clips (5-10 seconds long as done with random VoxCeleb2 samples in Sec. 4.1). Although we set the default training video length to 10 hours, it is shown in Fig. 3 that using only 2 hours leads to just a marginal performance drop, and we also used only 20-minute audio-generated continuous training video to compare with existing methods in Sec 4.3 (some of the compared methods require continuous video data due to their data preprocessing design and training loss implementation). Under these tested settings, the computational requirements are reduced significantly without much sacrifice in quality.\n\nWe are unsure if this satisfactorily addresses your concern on reasons that may limit the \"applicability, generalization ability, and efficiency\". Please let us know if it doesn't or you have any further questions.\n\n**W2 & Q1:** We thank the reviewer for pointing out the related works SEGA [1] (a concurrent and arXiv work, April 2025) and ExAvatar [2], which we will cite in our revised paper. However, with due respect, we find significant differences between these works and ours, in terms of their motivations and solutions to enhance the expressiveness of 3D parametric models. \n\nThe key motivation of our work is that the parameter space of existing 3D parametric models is intrinsically limited and cannot capture nuanced motion and subtle expressions of real human faces when talking (See Line 4-5, Line 25-33, Line 40-42). So we advocate *the use of the powerful latent space learned on massive 2D talking face videos to drive 3D Gaussian avatars*. To the best of our knowledge, this is a novel technical contribution not explored by previous works, and we have demonstrated its clear advantage (Table 1, Fig. 5, etc.).  \n\nPlease note that both [1] and [2] still rely on FLAME's expression parameters to drive their 3D models. Even though their dynamic offsets are learned from the subject’s data, their raw driving signals still stem from the FLAME expression space. These offsets lead to results that more closely adhere to the subject’s appearance, but they do little to enhance the expression space to encompass subtle expressions and nuanced motions. In contrast, our 3D model is driven by learned latent codes (from VASA-1), not FLAME expression codes.\n\nRegarding open-sourcing, we plan to release the code and some synthetic training data of VASA-3D for research purposes, which will facilitate checking the details of our VASA-3D implementations and exactly reproducing some typical VASA-3D avatars and the driving sequences shown in our paper. \n\n**W3:** While more training data was used in our method (minimum 20-minute videos containing 30K frames tested our paper) than HRAvatar [3] or RGBAvatar [4] (typically 2K-5K frames), there are key differences in task setup and data assumptions. These two methods use carefully captured monocular videos with exaggerated expressions and diverse poses to improve data coverage. Moreover, their training and testing data often come from highly similar distributions, enabling efficient data use.\n\nIn contrast, we simply take random real-world talking videos or audios to generate our training data, which represent an easier and more scalable data capture setup than these methods. We also use random real-world audios for testing. Under such situation, it is natural to use more training data to cover a wide motion distribution including nuanced facial expressions.\n\nAdditionally, our method must handle the inconsistencies in VASA-1-generated synthetic videos, further increasing the complexity. This makes our setting different and more challenging than these 3DGS avatar works.\n\n**W4:** We thank the reviewer for suggesting this potential baseline approach. We are unable to add this baseline in a short time window. However, we wish to make a few arguments:\n\n- Once again, our key motivation is to build expressive 3D head avatars animated with powerful latent spaces beyond 3DMM parameters. We have demonstrated the limitations of 3DMM parameters in expressing nuanced motions (Table 1, Fig. 5, etc.). As far as we can see, both [3,4] still use FLAME parameters to drive 3D facial expression, similar to the base model we used, i.e., GaussianAvatars. In fact, we view [3,4] as technical advancements orthogonal to ours and they can be applied as our base model in lieu of GaussianAvatars.\n\n- Using a large, diffusion-based talking head generation model such as [5,6] to generate data will incur significantly increased computation cost compared to using VASA-1 (e.g., <1 fps generation using [6]). Moreover, using such data still does not resolve the aforementioned challenge.\n\n- It's not clear to us how to adapt such baselines to the *audio-driven* scenarios. The task we handle in this work is audio-driven 3D talking face generation, with the target application being virtual, autonomous conversational agents. We provide more discussions related to audio-driven and video-driven methods at the end of this response.\n\n**W5:** The main goal of the SDS loss is to mitigate visual artifacts in side views (Line 189). Since the enhanced regions only occupy small image areas in side views, their improvement is not well reflected by the numerical values computed on all images with both frontal and side views. However, the visual improvements can be clearly observed (Fig. 4). We will add explanations in the revision, and we thank the reviewer for this comment. \n\n**Q1:** Please see above.\n\n**Q2:** There are multiple advantages of VASA-3D compared to VASA-1 in terms of 3D-related capabilities:\n- There's no true camera viewpoint control in VASA-1. The head pose control therein primarily steers the head angle with shoulder and torso regions largely unchanged. The camera distance control was approximated with 2D head region size as conditioning signal. In contrast, VASA-3D supports true 3D camera control and free-view rendering. \n\n- VASA-3D can be naturally integrated into 3D environment (VR/AR/Games) as it's inherently a 3D model with 3DGS rendering.\n\n- Last but not least, even when rendered as 2D videos, VASA-3D's results exhibit strong texture consistency. In contrast, VASA-1 shows noticeable texture sticking artifacts under head rotations and large facial motions, especially around regions with high-frequency textures such as hair and beard.\n\nWe will add more related discussions in our revised paper, and we thank the reviewer for raising this question.\n\n---\n\n**Motivation, Contribution, and Relationship to Video-Driven Methods:** \n\nHere we'd like to rephrase the motivation and main contribution of our paper. First, we observed that existing *audio-driven* 3D talking head methods cannot achieve highly realistic and vivid talking head results. The quality is far worse than the 2D talking face counterpart. After examination, we found that most existing methods apply 3D parametric models, in particular, FLAME, for expression control, following a generic \"audio->FLAME expression->deformed 3D model\" paradigm. However, existing 3D parametric models like FLAME are constructed by registering 3D scans – typically from just a few hundred subjects and without sophisticated expression design – and then applying PCA. They have intrinsic limitations in handling nuanced motion and subtle expressions of real human faces when talking and expressing emotions. So we propose to use a latent space learned on massive 2D face videos to drive 3D avatars, an attempt not made by any previous methods. We also propose a framework to map the learned latents to 3D Gaussian variations (Sec. 3.2) and a set of carefully designed losses to handle the training on synthesized data and addressing the challenges they incur (Line 170).\n\nWe have demonstrated the clear advantage of our method compared to using FLAME parameters. We also achieved high-quality and naturalistic audio-driven 3D talking head animation results from a single image, significantly pushing the state-of-the-art quality in this domain and minimizing the performance gap to audio-driven 2D talking head methods.\n\nWe also would like to re-emphasize that the task we handle in this work is *audio-driven talking face* generation. We acknowledge that there is rapid progress in 3D avatar reconstruction and video-based reenactment, such as those works pointed out by the reviewer. However, please note that audio-driven and video-driven 3D head avatar generation are often treated as distinct problems. As we mentioned earlier, those works can potentially be applied as our base models. In fact, we believe that *adapting these works to the audio-driven scenario is not trivial and achieving high realism and vividness in the talking scenario introduces challenges not fully addressed by these works*. Taking our base model GaussianAvatars as an example,  their video results on expression transfer (with FLAME expression codes) look convincing (see demos on their website). However, when it comes to *audio-synchronized motion behaviors and the expression of facial nuances subtle yet critical for expressing emotions*, FLAME's expression space is insufficient, as shown in our paper.\n\nWe are open to any discussion and please let us know if you have any further comments."}, {"review_id": "8imz0M4hRk", "reviewer": "Reviewer_mvUV", "summary": "This paper proposes VASA-3D, an audio-driven framework for generating expressive 3D head avatars from a single image. It builds on the motion latent of VASA-1 and maps it to a 3D head model through an optimization process using self-synthesized multi-view data. The method achieves realistic 3D talking heads and supports real-time free-viewpoint rendering at 512×512 resolution.", "questions": "1. As VASA-1 has not been open-sourced to date, and this work is essentially a 3D extension built upon it, it raises the question of whether the authors plan to release the code and models for VASA-3D. Open-sourcing this work would be particularly important for reproducibility and for promoting further research in the field.\n2. The paper adopts the motion latent from VASA-1 to drive the 3DGS representation. However, it is unclear why this latent space is preferred over directly using FLAME parameters for motion control, especially given that the base deformation in the proposed method is essentially obtained by mapping the VASA-1 latent to FLAME parameters via MLPs. The authors should clarify what advantages the VASA-1 latent offers compared to FLAME parameters.\n3. In the VAS deformation module, the model learns residuals for all 3DGS parameters. As discussed in the paper, this may lead to overfitting to the synthesized training data. Would it be possible to improve generalization by only learning animation-relevant parameters such as position and rotation, while keeping other attributes like color, scale, and opacity fixed? \n4. The training data for this method is generated using VASA-1 to produce over 10 hours of video. It is unclear whether pose control is applied during this process to ensure diverse viewpoints. Without explicit pose variation, audio-driven generation may result in overly redundant, front-facing video frames. The authors are encouraged to provide a more detailed description of the video generation pipeline.\n5. Does the FLAME model used in this work include teeth geometry? Additionally, the paper should clarify how the 3D Gaussian Splatting (3DGS) representation is bound or aligned with the FLAME mesh. It would also be helpful to specify the total number of Gaussians used for rendering the final avatar.", "limitations": "yes", "rating": 4, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 3, "significance": 2, "originality": 2, "strengths_and_weaknesses": "### Strengths\n1. VASA-3D effectively bridges 2D and 3D head modeling by leveraging the motion latent of high-quality 2D talking heads (VASA-1) and translating it into 3D space.\n2. The paper presents experiments demonstrating that VASA-3D outperforms recent methods in both visual quality and motion realism.\n\n### Weaknesses\n1. VASA-3D is a speech-driven 3D avatar generation method built upon 3D Gaussian Splatting (3DGS), which plays a central role in the framework, including learning 3DGS parameters from a reference image and audio. However, the paper lacks a comprehensive discussion of prior work on 3DGS and its applications in human avatar modeling. Such as GaussianAvatars and GAGAvatar (mentioned in the paper) are not sufficiently reviewed in the related work section. \n2. Although the paper focuses on audio-driven 3D head avatar generation and claims that few existing methods address this specific setting, it is worth noting that VASA-1 can also be driven by video. There are several recent methods capable of generating video-driven 3D avatars, such as FlashAvatar[1] and GBS[2]. However, the paper does not include experimental comparisons with these relevant approaches. Including such comparisons would provide a more complete evaluation and help clarify the strengths and limitations of the proposed method in the broader context of 3D avatar generation.\n3. This work is largely built upon the foundation of VASA-1. Its main contribution lies in constructing a prediction model that maps VASA-1’s motion latent to 3DGS parameters, and optimizing it through a set of loss functions. However, both the MLP-based prediction model and the loss formulations are commonly used in prior work, limiting the novelty of the proposed method.\n4. There are some typos in the paper. For example, in line 212, the symbol following “two images” is inconsistent with the one used in Equation (7). \n\n[1]. Jun Xiang, Xuan Gao, Yudong Guo, and Juyong Zhang. Flashavatar: High-fidelity head avatar with efficient Gaussian embedding. In The IEEE Conference on Computer Vision and Pattern Recognition (CVPR), 2024\n[2].Shengjie Ma, Yanlin Weng, Tianjia Shao, and Kun Zhou. 3d Gaussian blendshapes for head avatar animation. In ACM SIGGRAPH 2024 Conference Papers, pages 1–10, 2024", "was_revised": true, "final_justification": "The authors have addressed my concerns, and I will raise my rating accordingly.", "rebuttal": "Thank you for the valuable comments and suggestions. Before we address the reviewer's questions, we'd like to summarize the motivation and main contribution of our paper below.\n\n**Motivation, Contribution, and Relationship to Video-Driven Methods:** \n\nIn our work, we observed that existing audio-driven 3D talking head methods cannot achieve highly realistic and vivid talking head results. The quality is far worse than the 2D talking face counterpart. After examination, we found that most existing methods apply 3D parametric models, in particular, FLAME, for expression control, following a generic \"audio->FLAME expression->deformed 3D model\" paradigm. However, existing 3D parametric models like FLAME are constructed by registering 3D scans – typically from just a few hundred subjects and without sophisticated expression design – and then applying PCA. They have intrinsic limitations in handling nuanced motion and subtle expressions of real human faces when talking and expressing emotions. So we propose to use a latent space learned on massive 2D face videos to drive 3D avatars, an attempt not made by any previous methods. We also propose a framework to map the learned latents to 3D Gaussian variations (Sec. 3.2) and a set of carefully designed losses to handle the training on synthesized data and addressing the challenges they incur (Line 170).\n\nWe have demonstrated the clear advantage of our method compared to using FLAME parameters. We also achieved high-quality and naturalistic audio-driven 3D talking head animation results from a single image, significantly pushing the state-of-the-art quality in this domain and minimizing the performance gap to audio-driven 2D talking head methods.\n\nWe also would like to re-emphasize that the task we handle in this work is *audio-driven talking face* generation. We acknowledge that there is rapid progress in 3D avatar reconstruction and video-based reenactment, such as those works pointed out by the reviewer. However, please note that audio-driven and video-driven 3D head avatar generation are often treated as distinct problems. Adapting video-driven methods to the audio-driven scenario is not trivial and achieving high realism and vividness in the talking scenario introduces challenges not fully addressed by these methods. Taking our base model GaussianAvatars [17] as an example, their video results on expression transfer (with FLAME expression codes) look convincing (see demos on their website). However, when it comes to *audio-synchronized motion behaviors and the expression of facial nuances subtle yet critical for expressing emotions*, FLAME's expression space is insufficient, as shown in our paper. In fact, these 3D avatar reconstruction and video-based animation methods represent technical advancements orthogonal to ours and they can be applied as our base model, similar to GaussianAvatars.\n\n---\n\nWe now respond to the reviewer's comments below\n\n**W1:** Due to the sheer number of papers on this topic, we opted to discuss the previous work based on the properties they share, rather than discuss each of them individually. For example, the 3D Gaussian representations used by GaussianAvatars and GAGAvatar are mentioned in Line 81-83 of the Related Work. For 3D head reconstruction, the multi-view correspondence used by GaussianAvatars is mentioned in Line 89-90, and the one-shot approach by GAGAvatar using a parametric head model as a strong prior is described in Line 90-92. For head animation, it is explained in Line 104-108 that GaussianAvatars and GAGAvatar utilize parametric models for animation encoding and control, while the animations of VASA-3D are instead driven by the VASA-1 motion latent.\n \nWe feel that presenting the related work in this holistic manner allows us to position our method more clearly within the global context of the recent literature. However, to further address the reviewer’s concern, we could add the following at Line 106: *“For example, models based on 3D Gaussian splats such as GaussianAvatar [17] and GAGAvatar [20] control facial motion by modulating the expression parameters of the FLAME and 3DMM models, respectively, on which the splats are bound.”* Please let us know if this would be an acceptable solution.\n\n**W2:**  Our paper indeed focuses on *audio-driven* 3D head avatar generation as the reviewer noted. As we mentioned earlier, audio-driven and video-driven 3D head avatar generation are often treated as distinct problems. FlashAvatar and GBS are both video-driven 3D head avatar models, and they both use FLAME's expression parameters to drive the avatars, unlike ours that proposes using a latent space learned from massive 2D videos. These works are orthogonal to ours and can be potentially applied as our base model as well, similar to GaussianAvatars.\n\n**W3:** We wish to emphasize that the novelty of this work lies mostly in its new approach for driving animation of a 3D head avatar, rather than in the technical details for implementing this idea. Nevertheless, there also exist important technical innovations, as discussed below.\n\nRegarding our model framework, our instantiation is based on GaussianAvatars and we introduced a new module with two region-wise MLPs (neck and face) conditioned on VASA latent and head pose to regress additional Gaussian variations.\n \nRegarding our loss functions, they are designed to handle the unique challenges we had when training on the synthesized videos: temporal inconsistency, view angle limitation, and model overfitting (see Line 170-177). To our knowledge, no prior method has dealt with the same problem as ours. Although some loss terms might have been seen in different papers, our devised combination is a well-crafted and effective way to overcome the special challenges presented by our training data. Our novel-view render-consistency loss $L_{consist}$ with a $stop$_$gradient$ operator is a new one specially designed to regularize our final Gaussian output and mitigate the side effect brought by the SDS loss.\n\n**W4:** Thanks for pointing out the typos. We will correct them in the revised paper.\n\n**Q1:** Thank you for the suggestion regarding open source. We plan to release the code and some synthetic training data of VASA-3D for research purposes, which will facilitate checking the details of our VASA-3D implementations and exactly reproducing some typical VASA-3D avatars and the driving sequences shown in our paper.\n\n**Q2:** As we discussed above, our key motivation is that FLAME parameters are insufficient for capturing nuanced motion and subtle expressions of real human faces. This limitation is especially pronounced for the talking scenario where humans are sensitive to emotional expression and demand a high degree of motion realism. The latent space of VASA-1 is learned on a large volume of face videos to capture rich facial dynamics in a disentangled manner. The 2D talking face generation results of the VASA-1 also shows the capability of VASA-1 latent to represent nuanced motion and diverse talking behaviors. In light of this, we propose to apply it for 3D Gaussian head motion control and demonstrated its advantage in this work (Table 1, Fig. 3, etc.).\n\nDirectly regressing dense 3D motions from VASA-1 latent vectors would pose significant challenges in learning. Therefore, we decouple the deformation to a base deformation guided by FLAME and a dense, free-from deformation (VAS deformation) guided by VASA-1 latent. This way, we properly leveraged the priors in FLAME to ease the learning but go beyond it to use VASA-1 latent as the ultimate driving signal.\n\n**Q3:** We tried this alternative approach on one case where we only predict position and rotation offsets in VAS deformation while keeping color, scale, and opacity fixed. A quality drop is observed as shown in the table below. And we did not observe obvious improvements related to generalization.\n\n| |PSNR$\\uparrow$ | L1$\\downarrow$ |SSIM$\\uparrow$ | LPIPS$\\downarrow$| $S_C\\uparrow$| $S_D\\downarrow$|\n| ---- |---- |---- |---- |---- |---- |---- |\n|  Pos-rot-only |  25.99 |0.0207 |0.8846 | 0.0665 | 6.918|7.926|\n|  Full |  26.54 | 0.0195 |0.8874 | 0.0645 |7.033  |7.884|\n\n**Q4:** Our training data can be generated using either audios or face videos. The 10-hour training data used to conduct ablation study and validate algorithm designs was generated using face videos  (Line 234). Specifically, we used video clips randomly sampled from VoxCeleb2 where each clip is typically of 5-10 seconds. The pose variation in the generated data is large. We also used audios to generate training data in Sec. 4.2 and 4.3 (Line 270, 292), where we use 20-minute continuous audios to generate training videos (some of the compared methods require continuous video for training and cannot utilize the collection of video clips in our 10-hour data). For this case, we did not apply additional pose augmentation for simplicity and the results look satisfactory. We conjecture that VASA-1-generated pose contains adequate variations and our loss designs are effective to facilitate the reconstruction with limited angles. Applying pose variations is nevertheless an excellent idea, and we thank the reviewer for this suggestion!\n\n**Q5:** Yes, the FLAME model we used includes teeth geometry. Each 3D Gaussian is bound to a triangle of the FLAME mesh, following GaussianAvatars (Line 123-125).\n\n---\n\nPlease let us know if you have any further questions."}]

Author rebuttal:

Meta-review: (a, b) Precise alignment of audio and visuals in realistic avatars is an unsolved problem, although there is progress. This work approaches optimizing the rendering in VASA latent space with audio/speech driven physics based guidance (via gaussian splat smoothing). This conditioning of video gen on audio is important finding. The explicit parameterization of facial expressions in the splats is also key. The paper is written well; and the video demos are compelling (clearly the authors put in rigor into the work). 

(c) There are some related works and baseline that were missing; as were pointed out by the reviewers. Authors addressed these in the rebuttal. Its not clear how real-time and generalizable this can be in scenarios where audio/speech inputs are noisy (a potential next work, this does not diminish this paper's contributions). The work is directly building on VASA-1, and so, that also limits the overall generalization capacity. 

(d) Impact in single image/view driven synthesis of long form video. Shows ways to address realism of facial expressions in Audio visual context generation. Beating state of the art. Clarity of presentation and rigor in evaluations.

(e) Bulk of the reviewer questions were re: missing related work, claims about novelty (which were addressed), corrections in paper presentations, and 1-2 potential missing baselines (which do not change the overall story arc). These all help clarify the work but they are mostly procedural. No technical flaws are highlighted.
Decision: Accept (poster)