Title: VASA-3D: Lifelike Audio-Driven Gaussian Head Avatars from a Single Image
Venue: NeurIPS 2025
Accepted: yes
Keywords: 3DGS

Avatars

3D Talking Head

Single-shot
Abstract: We propose VASA-3D, an audio-driven, single-shot 3D head avatar generator. This research tackles two major challenges: capturing the subtle expression details present in real human faces, and reconstructing an intricate 3D head avatar from a single portrait image. To accurately model expression details, VASA-3D leverages the motion latent of VASA-1, a method that yields exceptional realism and vividness in 2D talking heads. A critical element of our work is translating this motion latent to 3D, which is accomplished by devising a 3D head model that is conditioned on the motion latent. Customization of this model to a single image is achieved through an optimization framework that employs numerous video frames of the reference head synthesized from the input image. The optimization takes various training losses robust to artifacts and limited pose coverage in the generated training data. Our experiment shows that VASA-3D produces realistic 3D talking heads that cannot be achieved by prior art, and it supports the online generation of 512x512 free-viewpoint videos at up to 75 FPS, facilitating more immersive engagements with lifelike 3D avatars.


Peer reviews:
{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper proposes VASA-3D, an audio-driven framework for generating expressive 3D head avatars from a single image. It builds on the motion latent of VASA-1 and maps it to a 3D head model through an optimization process using self-synthesized multi-view data. The method achieves realistic 3D talking heads and supports real-time free-viewpoint rendering at 512×512 resolution.\n\nSTRENGTHS AND WEAKNESSES:\n### Strengths\n1. VASA-3D effectively bridges 2D and 3D head modeling by leveraging the motion latent of high-quality 2D talking heads (VASA-1) and translating it into 3D space.\n2. The paper presents experiments demonstrating that VASA-3D outperforms recent methods in both visual quality and motion realism.\n\n### Weaknesses\n1. VASA-3D is a speech-driven 3D avatar generation method built upon 3D Gaussian Splatting (3DGS), which plays a central role in the framework, including learning 3DGS parameters from a reference image and audio. However, the paper lacks a comprehensive discussion of prior work on 3DGS and its applications in human avatar modeling. Such as GaussianAvatars and GAGAvatar (mentioned in the paper) are not sufficiently reviewed in the related work section. \n2. Although the paper focuses on audio-driven 3D head avatar generation and claims that few existing methods address this specific setting, it is worth noting that VASA-1 can also be driven by video. There are several recent methods capable of generating video-driven 3D avatars, such as FlashAvatar[1] and GBS[2]. However, the paper does not include experimental comparisons with these relevant approaches. Including such comparisons would provide a more complete evaluation and help clarify the strengths and limitations of the proposed method in the broader context of 3D avatar generation.\n3. This work is largely built upon the foundation of VASA-1. Its main contribution lies in constructing a prediction model that maps VASA-1’s motion latent to 3DGS parameters, and optimizing it through a set of loss functions. However, both the MLP-based prediction model and the loss formulations are commonly used in prior work, limiting the novelty of the proposed method.\n4. There are some typos in the paper. For example, in line 212, the symbol following “two images” is inconsistent with the one used in Equation (7). \n\n[1]. Jun Xiang, Xuan Gao, Yudong Guo, and Juyong Zhang. Flashavatar: High-fidelity head avatar with efficient Gaussian embedding. In The IEEE Conference on Computer Vision and Pattern Recognition (CVPR), 2024\n[2].Shengjie Ma, Yanlin Weng, Tianjia Shao, and Kun Zhou. 3d Gaussian blendshapes for head avatar animation. In ACM SIGGRAPH 2024 Conference Papers, pages 1–10, 2024\n\nQUESTIONS:\n1. As VASA-1 has not been open-sourced to date, and this work is essentially a 3D extension built upon it, it raises the question of whether the authors plan to release the code and models for VASA-3D. Open-sourcing this work would be particularly important for reproducibility and for promoting further research in the field.\n2. The paper adopts the motion latent from VASA-1 to drive the 3DGS representation. However, it is unclear why this latent space is preferred over directly using FLAME parameters for motion control, especially given that the base deformation in the proposed method is essentially obtained by mapping the VASA-1 latent to FLAME parameters via MLPs. The authors should clarify what advantages the VASA-1 latent offers compared to FLAME parameters.\n3. In the VAS deformation module, the model learns residuals for all 3DGS parameters. As discussed in the paper, this may lead to overfitting to the synthesized training data. Would it be possible to improve generalization by only learning animation-relevant parameters such as position and rotation, while keeping other attributes like color, scale, and opacity fixed? \n4. The training data for this method is generated using VASA-1 to produce over 10 hours of video. It is unclear whether pose control is applied during this process to ensure diverse viewpoints. Without explicit pose variation, audio-driven generation may result in overly redundant, front-facing video frames. The authors are encouraged to provide a more detailed description of the video generation pipeline.\n5. Does the FLAME model used in this work include teeth geometry? Additionally, the paper should clarify how the 3D Gaussian Splatting (3DGS) representation is bound or aligned with the FLAME mesh. It would also be helpful to specify the total number of Gaussians used for rendering the final avatar.\n\nLIMITATIONS:\nyes", "TITLE": "Official Review", "RECOMMENDATION": 4, "REVIEWER_CONFIDENCE": 3, "CLARITY": 3, "IMPACT": 2, "ORIGINALITY": 2, "DATE": "2025-07-01"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper builds upon VASA-1 and 3D Gaussian Splatting (3DGS) to extend the task of talking head generation into the 3D domain. By leveraging VASA-1 to render a large number of video frames from a single input image, the authors train a 3DGS-based model. Furthermore, they utilize the motion latent from VASA-1 to generate realistic 3D talking head videos with relatively fast inference speed.\n\nSTRENGTHS AND WEAKNESSES:\n**Strengths:**\n\n1. The proposed method leverages VASA-1’s motion latent to enhance the expressiveness of facial motions in the 3DGS framework.\n2. The paper presents several impressive demo videos that are visually convincing and contain relatively few artifacts under various head poses and expressions.\n3. The proposed base deformation and VASA deformation modules are well-motivated and experimentally validated.\n\n**Weaknesses:**\n\n1. If I have not missed any key details, the proposed approach seems to heavily rely on VASA-1 to generate long video sequences from a single input image to train an avatar-specific 3D model. This significantly limits the method’s applicability, generalization ability, and efficiency in terms of storage and scalability.\n2. While the use of motion latents from VASA-1 is claimed to be novel, it resembles prior approaches that condition 3DGS on dynamic face maps (e.g., SEGA[1], ExAvatar[2]) to compensate for the limited expressiveness of 3DMM-based representations. Additionally, the strong dependency on the non-open-source VASA-1 makes the method difficult to reproduce or benchmark.\n3. Compared to some recent 3DGS-based avatar generation methods such as HRAvatar[3] and RGBAvatar[4], this method appears to require significantly more training video data.\n4. From an implementation and benchmarking perspective, the paper overlooks a potentially fairer baseline: using an end-to-end diffusion-based talking head generation model[5,6]  to synthesize a training video from a single image, followed by training a 3DGS-based avatar model [3,4] from that video for comparison.\n5. Although the authors introduce an SDS loss to mitigate artifacts from VASA-1 outputs (e.g., misalignments and large-pose distortions), this solution substantially increases the training cost. Moreover, Table 1 shows only marginal quantitative gains from this additional loss.\n\n[1] SEGA: Drivable 3D Gaussian Head Avatar from a Single Image\n\n[2] Expressive Whole-Body 3D Gaussian Avatar\n\n[3] High-Quality and Relightable Gaussian Head Avatar\n\n[4] RGBAvatar: Reduced Gaussian Blendshapes for Online Modeling of Head Avatars\n\n[5] HunyuanPortrait: Implicit Condition Control for Enhanced Portrait Animation\n\n[6] Hallo: Hierarchical Audio-Driven Visual Synthesis for Portrait Image Animation\n\nQUESTIONS:\n1. Will the authors release the code and models introduced in this paper?\n2. Since VASA-1 already supports some degree of head pose and camera movement control to produce pseudo-3D effects, how does the proposed method compare against VASA-1 under such settings?\n\nLIMITATIONS:\nyes", "TITLE": "Official Review", "RECOMMENDATION": 4, "REVIEWER_CONFIDENCE": 4, "CLARITY": 3, "IMPACT": 2, "ORIGINALITY": 2, "DATE": "2025-07-02"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper presents a method to create a human head avatar from a single image that can be driven by audio. Given an input image, it uses a 2D diffusion model, VASA-1, to generate video frames to form the training dataset, which is then used to optimize a 3DMM-driven 3DGS avatar model, GaussianAvatars. With the aid of the powerful motion space of VASA-1, this method can capture a wider range of facial expressions and body movements, which better align with the audio input. The authors also explore several regularization techniques to enhance visual details without harming 3D consistency. Solid experiments and sufficient visual results are provided to exhibit the effectiveness of this method.\n\nSTRENGTHS AND WEAKNESSES:\n# Strengths\n1. This is a well-written paper with a strong motivation and impressive execution. The visual results, especially the video demos, are very convincing - the rendered avatars show great 3D consistency, sharp details, and expressive facial movements that align well with the audio.\n1. The method is concise and self-contained - each component appears for a clear reason and is consolidated with sufficient experiments. The long list of loss terms seems verbose, but they are actually all necessary, as shown in Table 1 and Figure 4&5.\n1. It is very nice to see the rendering of \"Base Deform\" and \"+VAS Deform\" side-by-side in Figure 4, which clearly shows the advantage of using the broader motion space of VASA-1.\n1. The ablation study of dataset size and number of iterations shown in Figure 3 is a good example of scaling up in an effective way.\n\n# Weaknesses\n1. The efficiency and  latency of this method during inference is not reported. Does it support realtime audio-driven synthesis given a trained avatar?\n\nQUESTIONS:\n- Given the existence of two separate networks for the face and the neck, how can one tell if a Gaussian splat belongs to the face or the neck?\n\nLIMITATIONS:\nyes", "TITLE": "Official Review", "RECOMMENDATION": 5, "REVIEWER_CONFIDENCE": 4, "CLARITY": 4, "IMPACT": 3, "ORIGINALITY": 3, "DATE": "2025-07-02"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe paper introduces VASA-3D, a method for generating lifelike, audio-driven 3D head avatars from a single portrait image. The key innovation lies in leveraging the motion latent space of VASA-1 to animate a 3D Gaussian splatting model. By synthesizing diverse facial expressions and poses via VASA-1, the method trains a 3D head model that supports free-viewpoint rendering and real-time animation. A robust training framework with tailored losses addresses challenges like temporal inconsistency and limited pose coverage in synthetic data. Experiments demonstrate superior realism and performance compared to existing methods, with 512×512 videos rendered at 75 FPS.\n\nSTRENGTHS AND WEAKNESSES:\nStrengths:\n1. Single-Image Customization: Achieves high-quality 3D head avatars from a single image, overcoming the need for multiview data.\n2. Integration of 2D and 3D: Effectively translates 2D motion latent into 3D deformations, enabling expressive animations.\n3. Real-Time Performance: Supports free-viewpoint rendering at 75 FPS, making it practical for immersive applications.\n4. Strong Empirical Results: Outperforms state-of-the-art methods in metrics like FID, lip-sync accuracy, and user studies.\n\nWeaknesses:\n1. Reliance on VASA-1: Synthetic data quality depends on VASA-1, which may inherit its limitations.\n2. Computational Cost: Requires extensive synthetic data (10 hours per portrait) and 18-hour training, limiting scalability.\n3. Limited Generalization: Tested primarily on StyleGAN2-generated portraits; performance on real-world images is unverified.\n4. Code Unavailability: VASA-1 is not open sourced\n\nQUESTIONS:\n1. How does VASA-3D handle identity preservation when synthesizing expressions not present in the input image?\n2. What are the trade-offs between using FLAME as a base model and alternative parametric representations?\n3. Can the method generalize to non-frontal input images or low-resolution portraits?\n4. How sensitive is the training to the choice of VASA-1 as the motion latent source?\n\nLIMITATIONS:\n1. Synthetic Data Dependency: Relies on VASA-1 for training data, which may not capture all real-world variations.\n2. High Resource Requirements: Time and computational demands for training may restrict accessibility.\n3. Artistic Image Support: While tested on artistic portraits, real-world applicability remains unexplored.", "TITLE": "Official Review", "RECOMMENDATION": 5, "REVIEWER_CONFIDENCE": 3, "CLARITY": 4, "IMPACT": 4, "ORIGINALITY": 4, "DATE": "2025-07-13"}