GaussianTalker: Speaker-specific Talking Head Synthesis via 3D Gaussian Splatting

Venue: acmmm_org_ACMMM_2024_Conference
Published: 2024-04-09 11:08:14

Abstract: Recent works on audio-driven talking head synthesis using Neural Radiance Fields (NeRF) have achieved impressive results. However, due to inadequate pose and expression control caused by NeRF implicit representation, these methods still have some limitations, such as unsynchronized or unnatural lip movements, and visual jitter and artifacts. In this paper, we propose GaussianTalker, a novel method for audio-driven talking head synthesis based on 3D Gaussian Splatting. With the explicit representation property of 3D Gaussians, intuitive control of the facial motion is achieved by binding Gaussians to 3D facial models. GaussianTalker consists of two modules, Speaker-specific Motion Translator and Dynamic Gaussian Renderer. Speaker-specific Motion Translator achieves accurate lip movements specific to the target speaker through universalized audio feature extraction and customized lip motion generation. Dynamic Gaussian Renderer introduces Speaker-specific BlendShapes to enhance facial detail representation via a latent pose, delivering stable and realistic rendered videos. Extensive experimental results suggest that GaussianTalker outperforms existing state-of-the-art methods in talking head synthesis, delivering precise lip synchronization and exceptional visual quality. Our method achieves rendering speeds of 130 FPS on NVIDIA RTX4090 GPU, significantly exceeding the threshold for real-time rendering performance, and can potentially be deployed on other hardware platforms.

Main paper: 
 introduction: 
 Audio-driven talking head synthesis has attracted significant attention in various fields, including digital avatars, virtual reality, interactive entertainment, and online meetings [32, 44, 47] . This task aims to synthesize a video in which the target speaker's lip movements are in sync with the given audio input.In existing research, talking head synthesis approaches are primarily categorized into 2D-based and 3D-based approaches. Initially, most 2D-based approaches depended on Generative Adversarial Networks [16] or image-to-image translation [19] to synthesize talking heads in sync with audio. However, the absence of a unified facial model resulted in shortcomings in the identity preservation and pose control of the synthesized videos. More recently, research has been conducted to apply the Neural Radiance Fields (NeRF) [31] to this task. NeRF is a method that models continuous 3D scenes using implicit functions. Compared to 2D-based approaches, NeRFbased approaches can synthesize more lifelike talking head videos by effectively leveraging 3D facial modeling. Nevertheless, these approaches often encounter issues such as unsynchronized or unnatural lip movements, visual jitters, and sporadic artifacts, primarily because NeRF's implicit definition entangles static facial geometry with dynamic motion, complicating the control over facial poses and expressions.Recently, 3D Gaussian Splatting (3D GS) [23] has shown impressive achievements in 3D scene reconstruction. 3D GS employs 3D Gaussians as discrete geometric primitives, offering an explicit 3D scene representation and optimizing for real-time rendering performance. In comparison to NeRF, 3D GS not only significantly enhances rendering efficiency and visual quality, but also its paradigm based on 3D Gaussians is easier to control, making it potentially possible to flexibly and intuitively control facial movements. An intuitive strategy is to drive a Gaussian point cloud for facial motion by using parametric 3D facial models [6, 27, 34] . These models typically provide a comprehensive parameter space that allows for the control of facial attributes, such as shape, pose, and expression. By binding the Gaussians to the geometric topology of the model, dynamic talking heads can be generated by synchronizing the displacement of the bound Gaussians with changes in the facial attribute parameters.Natural lip movements and realistic visual effects are crucial for talking head synthesis. However, the deviation of 3D facial models from a specific speaker's face makes it challenging to synthesize lifelike videos using such a basic binding strategy. Specifically, there are two main challenges: 1) Distribution inconsistencies between the parameters driving the 3D Gaussians and the actual parameters can lead to deviations in point positions. This distribution inconsistency specifically refers to differences in speaking styles. For example, some individuals may speak with a small open mouth, while the driving signal is a wide open mouth. Lip movements that do not align with the speaker's talking style present unnatural effects and even cause lip artifacts. 2) 3D facial models are limited in modeling complex faces. They can only capture macroscopic muscle movements but not fine details such as wrinkles and teeth. This inherent limitation hinders the 3D Gaussians' accurate representation of a specific face, and the loss of facial details will also lead to visual flicker and artifacts.For natural lip movements and realistic visual effects, bridging the subtle discrepancies between the 3D facial model and the specific speaker's face is critical. Firstly, audio typically carries speaker identity information. it is necessary to decouple the driving audio from the original speaker's identity information and ensure that the synthesized lip movements closely match the target speaker's style. Secondly, intuitive control of Gaussian attributes helps break through the representation limitations of facial models. With a comprehensive set of blendshapes, these Gaussian attributes can be fine-tuned to capture facial details, refining the facial representation and further fitting a specific speaker.In this work, we propose GaussianTalker, a framework designed to generate highly natural and realistic talking head videos, adaptable to multiple languages and various timbres of audio input. For dynamic reconstruction, 3D Gaussians are bound to the geometric topology of the parameterized FLAME [27] , driving the Gaussians through the deformation of the FLAME mesh, ensuring accurate facial animation. To address the challenge of unnatural lip movements caused by inconsistent distributions, we propose a Speaker-specific Motion Translator. Employing timbre transformation-based contrastive learning for feature decoupling, a speaker-agnostic audio feature is extracted and merged with the target identity embedding to generate facial poses and expressions that closely align with the target speaker. To confront the challenge posed by unrealistic visual effects caused by the inherent limitations of 3D facial models, the Dynamic Gaussian Renderer introduces Speaker-specific BlendShapes designed to predict the target speaker's latent pose representation and refine facial details through the computation of detail-enhancing Gaussian attributes. It is important to note that our framework is not only capable of achieving speeds well beyond real-time rendering thresholds on contemporary GPUs, but it is also expected to be deployed on other hardware platforms, paving the way for its extensive application across multiple platforms.The main contributions of this paper are summarized as follows:• A novel framework for audio-driven talking head synthesis that utilizes 3D Gaussian Splatting bound to FLAME, which generates lifelike rendered videos by associating data from different modalities with specific speakers. • A Speaker-specific Motion Translator decouples identity and employs personalized embedding for natural lip movements, while a Dynamic Gaussian Renderer refines Gaussian attributes through latent pose to ensure realistic visual effects.GaussianTalker surpasses state-of-the-art methods in lipsync and image quality, with its high rendering speed underscoring multi-hardware platform application potential. 
related work 2.1 audio-driven facial animation: 
 Most early methods of audio-driven facial animation required establishing a mapping relationship between phonemes and visemes. These methods [3, 4] often overlook individual differences and are overly complex. The emergence of large-scale high-definition video datasets [1, 8, 12, 61] , coupled with advancements in deep learning, has propelled the domain of learning-based audio-driven facial animation to the forefront of research. Initial efforts utilized 2D approaches [10, 14, 20, 36, 49, 52, 59, 63, 64] , employing Generative Adversarial Networks [16] or image-to-image translation [19] , to generate facial animations. However, these approaches fell short of accurately reproducing a speaker's face due to lack of 3D facial modeling, leading to shortcomings in identity preservation and pose controllability. The adoption of 3D Morphable Models (3DMM) [2, 41, 66] has spurred research into audio-driven facial animation using 3DMM. These approaches [22, 30, 46, 51, 58] , leveraging 3D modeling, are capable of rendering a more lifelike talking style compared to 2D approaches. Nevertheless, intermediate transformations through 3DMM may lead to information loss [2] . Moreover, prior universal 3D audio-driven animation methods [35, 39] generated vertex predictions with biases toward specific speakers, resulting in animations that fail to express personal styles.Our method utilizes a parametric 3D facial model to represent the human face, generating facial motion representation from the driving audio. By identity decoupling and personalized embedding, we accommodate the unique styles of different speakers.Recent studies have adopted Neural Radiance Fields (NeRF) [31] for 3D human head modeling and rendering. NeRF encodes the scene into a continuous volumetric field using a neural network, which enables high-quality 3D rendering. Following its successful application in dynamic scenes [15, 33, 37] , NeRF has been naturally extended to talking head synthesis tasks [7, 17, 26, 29, 43, 45, [55] [56] [57] . Some studies have explored the implementation of talking face video generation in an end-to-end way [17, 29, 55] . Subsequent studies have realized several improvements in rendering efficiency [26, 45] , few-shot synthesis [28, 43] , and lip shape generalization [7, 56, 57] .With the advent of 3D Gaussian Splatting (3D GS) [23] , considerable potential has been shown in enhancing both rendering efficiency and quality. Recent work [40, 42, 65] has implemented dynamic head reconstruction based on 3D GS. GaussianHead [50] employs learnable Gaussian diffusion for detailed head generation to accurately reproduce dynamic facial details. MonoGaussian-Avatar [11] utilizes 3D Gaussian representation and deformation fields for learning explicit avatars from monocular portrait videos.Despite these approaches' effectiveness, audio-driven talking head synthesis faces a significant challenge: cross-modal facial parameter generation from audio fails to precisely capture a specific person's facial nuances. To our knowledge, we first apply 3D GS to this task and overcome the above problem. By Speaker-specific BlendShapes, our method facilitates the synthesis of more lifelike videos, charting a new course in the field of audio-driven highfidelity talking head synthesis. 
method 3.1 overview: 
 In this section, we provide a detailed description of the architectural components of our proposed framework, GaussianTalker. Gaus-sianTalker employs the parametric FLAME model [27] to serve as a bridge between facial animation and rendering. As shown in Figure 2 , the overall framework consists of two main modules: 1) Speaker-Specific Motion Translator, which converts audio signals into speaker-specific FLAME parameters sequence for facial animation control; 2) Dynamic Gaussian Renderer, which utilizes FLAME to drive 3D Gaussians and renders dynamic talking head in real-time. The following subsections detail the design and training mechanisms of each module.Given the diversity of audio inputs, we recognize that model generalization is challenging when solely dependent on video data from specific individuals. Consequently, we train the module using a multilingual, multi-individual dataset to improve its adaptability to diverse audio inputs. However, due to substantial variations in individual speaking styles, the distribution of FLAME parameters generated by the module might diverge from those of the target speaker, potentially compromising the rendered videos' realism. To address this issue, we develop a Universal Audio Encoder that decouples identity information from content information, and a Customized Motion Decoder that integrates personalization features. Additionally, we refer to SelfTalk [35] and introduce a lipreading constraint mechanism based on self-supervision to further refine the synchronization of lip movements. This module's detailed architecture and workflow are illustrated in Figure 2(b) . 𝐴 !"#$%& 𝐴 '%( 𝐴 ")*,, 𝐴 ")*,- Universal Audio Encoder 𝑓 !"#$%& 𝑓 . = 𝑓 '%( 𝑓 , = 𝑓 ")*,, 𝑓 -= 𝑓 ")*,- … …speaker identity information and content information, which are usually tightly coupled. This indicates that in addition to conveying the content of speech, the audio also carries attributes unique to the speaker, such as pitch and timbre. In the task of audio feature extraction, our goal is to capture speech content while excluding information related to speaker identity. Therefore, we start at the audio encoding stage to achieve effective decoupling of identity and content information. We extract audio features using the audio encoder of a pre-trained Wav2Vec 2.0-based multilingual ASR model [54] . As shown in Figure 3 , we segment a speaker's audio into 𝑘 + 1 segments, and then randomly select a segment as the anchor audio 𝐴 𝑎𝑛𝑐ℎ𝑜𝑟 = 𝐴 𝑐1,𝑡 1 . Next, we perform timbre conversion [38] on the anchor audio to obtain positive samples 𝐴 𝑝𝑜𝑠 = 𝐴 𝑐1,𝑡 2 , while the remaining audio segments are used as negative samples. The subscripts 𝑐 𝑖 , 𝑡 𝑖 denote the content and timbre of the audio respectively. By encoding the audio, we get the respective feature, whose formulas are as follows:EQUATIONwhere 𝒇 𝑡 denotes the audio feature, 𝐸 𝑢𝑛𝑖 and 𝜃 𝑢𝑛𝑖 denote the audio encoder and its learnable parameters, and 𝐴 𝑡 denotes the audio.Positive samples are consistent in content with the anchor audio, and we anticipate them to generate similar audio features. For negative sample features, we aim for maximal dissimilarity from the anchor audio feature to achieve de-identification of audio encoding. To achieve this goal, we use the InfoNCE loss [48] for optimization. To maintain the original capability of the encoder, we use 𝐸 𝑎𝑠𝑟 and 𝐷 𝑎𝑠𝑟 , encoder and decoder of the pre-trained Wav2Vec 2.0 model [54] to guide the optimization:EQUATIONEQUATIONwhere 𝜏 denotes the temperature.We introduce an identity embedding to generate motions that capture the speaker's talking style, such as the difference in the degree of mouth opening when different speakers talk. The Identity embedding takes personal identity information as input and generates the personalized feature consistent with the audio feature dimensions. Next, the personalized feature is fused with the audio feature to serve as the input of the Customized Motion Decoder. The decoder based on a transformerbased structure, outputs the FLAME parameters sequence directly:EQUATIONwhere Ŷ1:𝑇 = ( ŷ1 , ..., ŷ𝑇 ) denotes the generated FLAME parameters sequence, 𝐷 𝑚 denotes the motion decoder, 𝐸 𝑖𝑑 denotes the identity embedding, 𝑝 denotes the speaker's identity, and 𝜃 𝑖𝑑 , 𝜃 𝑚 denote the learnable parameters of the corresponding component, respectively.The training objectives of Speaker-specific Motion Translator include three components: reconstruction, lip motion smoothness, and potential consistency. Reconstruction. To ensure the generated motion sequence precisely aligns with the ground truth data, we calculate the mean square error of the ground truth 𝒀 1:𝑇 = (𝒚 1 , ..., 𝒚 𝑇 ) and the generated FLAME parameters sequence Ŷ1:𝑇 , as well as the mean square error of the corresponding vertices of the FLAME mesh 𝑽 1:𝑇 = (𝒗 1 , ..., 𝒗 𝑇 ) and V1:𝑇 = ( v1 , ..., v𝑇 ):EQUATIONwhere 𝑁 denotes the dimension of FLAME parameters, 𝐾 denotes the number of FLAME mesh vertices, 𝑇 denotes the total number of video frames.Lip motion smoothness. To avoid the potential motion jitter problem that results from relying solely on reconstruction loss, we enhance lip motion smoothness by calculating parameter changes across frames:EQUATION)Potential consistency. During training, the generated motion sequence is passed to the lip-reading encoder to extract lip-reading feature. To ensure coherence between the audio-text feature and lip-reading feature, we compute their mean square error:EQUATIONwhere 𝑀 𝐿 denotes the index of lip region, 𝐸 𝑙𝑖𝑝 denotes the lip encoder, and 𝐷 denotes the feature dimension.The final loss function is formulated as follows:EQUATION3D GS [23] is a rasterization technique designed for the real-time rendering of photorealistic scenes. It utilizes a collection of 3D Gaussians for modeling, where each Gaussian in the model is characterized by the following learnable parameters: position 𝒖 ∈ R 3 , indicating the center of the Gaussian; rotation 𝒓 ∈ R 3×3 , defining its orientation; scaling factor 𝒔 ∈ R 3 , controlling its size; opacity 𝜶 ∈ R, determining its visibility; and spherical harmonics coefficients 𝜿 ∈ R 3× (𝑘+1) 2 , which determine the RGB color 𝒄 through a k-order spherical harmonic function. This set of parameters for each Gaussian is denoted as G = {𝒖, 𝒓, 𝒔, 𝜶 , 𝜿 }. During the rendering process, the color 𝐶 of each pixel is determined by aggregating the contributions of all overlapping Gaussians, as described by the following equation:EQUATIONWe aim to employ explicit 3D Gaussians to represent facial expressions, but static 3D Gaussians fall short of capturing dynamic changes in expressions. Consequently, we devised a Dynamic Gaussian Renderer that anchors the Gaussians to the FLAME triangles. This approach leverages FLAME's BlendShapes and Skin Weights to control the deformation of the Gaussians. We also incorporate some Speaker-specific Blendshapes to enhance geometric and textural detail in facial rendering and introduce an Inpainting Generator to seamlessly integrate the rendered face with the original image. The complete pipeline is presented in Figure 2(c ).Given a set of FLAME parameters, 𝜷 for shape, 𝜺 for expression, and 𝝍 for pose changes, the vertices of the FLAME model in the global space, denoted as 𝒗, are determined by the following motion rules:EQUATIONwhere 𝒗 𝑏𝑎𝑠𝑒 represents the vertices of the FLAME mesh in canonical space. 𝐿𝐵𝑆 signifies the standard Linear Blend Skinning function with weights 𝑊 and the function 𝐽 stands for the joint regressor based on the pose 𝝍, both as defined in the FLAME model. The 𝐵𝑆 operation denotes a linear blend shaping process that creates facial animations by combining blendshapes in accordance with FLAME parameters, weighted by the blendshape weights 𝑊 𝑏𝑠 . For a triangle T of the FLAME mesh, which is defined by its vertices {𝒗 0 , 𝒗 1 , 𝒗 2 } and its edges {𝒆 01 = 𝒗 1 -𝒗 0 , 𝒆 02 = 𝒗 2 -𝒗 0 , 𝒆 12 = 𝒗 2 -𝒗 1 }, we establish a local coordinate system based on T . The system's origin, 𝑃, representing T 's position in global space, is determined by a barycentric combination of the triangle's vertices. The system's orientation in global space is captured by the rotation matrix 𝑅, and the scale factor 𝑆 reflects the size of T . These components are delineated as follows:EQUATIONEQUATIONEQUATIONwhere 𝜼 = (𝜂 0 , 𝜂 1 , 𝜂 2 ) is the learnable barycentric coefficients, and the orthogonal unit vectors [𝒏 0 ; 𝒏 1 ; 𝒏 2 ] are calculated as follows:EQUATIONThen a Gaussian is associated within this local space, denoted as G = {ū, r, s, 𝜶 , 𝜿, T , 𝜼}, enabling G to track the movements of T . Specifically, the attributes ū, r, s of G are defined relative to the local coordinate system, rather than the global space. We initialize its position ū at the origin, rotation r as the identity matrix, and scale s as a unit vector. When rendering, we transform the G into global space by:EQUATIONwhere 𝒖, 𝒓, 𝒔 denotes the global position, orientation, scale of G.During optimization, we employ an adaptive density control strategy similar to the one described by 3D GS [23] , to dynamically add or remove Gaussians based on the view space positional gradient and the opacity of each Gaussian. Specifically, when a Gaussian G 𝑖 is anchored to a triangle T 𝑖 of the FLAME mesh, all its derivatives generated by cloning or splitting from G 𝑖 , denoted as (G 𝑖,1 , G 𝑖,2 , G 𝑖,3 , ...), inherit the same local coordinate system that we established based on T 𝑖 .We have addressed the challenge of capturing expressive facial motion at a coarse level by binding Gaussians to the triangles of the FLAME mesh. However, since FLAME primarily encodes only coarse facial deformations, it remains a challenge to reproduce speaker-specific facial geometry and texture details, such as teeth and wrinkles. To overcome this limitation, we introduce a set of learnable BlendShape (BS) weights that are designed to capture and delineate these nuanced facial features. Initially, we define a Multilayer Perceptron (MLP), designated as 𝑊 𝛾 , which accepts the pose parameters 𝝍 from the FLAME and generates a latent pose representation 𝜸 as follows:EQUATIONTo refine geometrical details, specialized BS weights, 𝑊 𝑝𝑜𝑠 for position and 𝑊 𝑟𝑜𝑡 for rotation, are employed to update the corresponding attributes of Gaussian G within the local space:EQUATIONThese revised local attributes, ū′ and r ′ , are then converted to global space when rendering, as explained in Equation 15. Similarly, for texture refinement, we apply compensation to 𝜿 0 , the zerothorder coefficient of 𝜿 in G, which governs the base color calculated by zeroth-order spherical harmonics function. This adjustment is achieved through BS weights designated as 𝑊 𝑐𝑜𝑙𝑜𝑟 :EQUATIONWith the incorporation of Speaker-specific BlendShapes for facial detailing, G is represented as G = {ū ′ , r ′ , s, 𝜶 , 𝜿 ′ 0 , 𝜿 𝑟𝑒𝑠𝑡 , T , 𝜼}, where 𝜿 𝑟𝑒𝑠𝑡 denotes the higher-order(1 to k-th) coefficients of 𝜿.By incorporating Dynamic Deformation and Speaker-specific BlendShapes, we achieve a consistent alignment of the head pose with the original face image, which enables a stable reintegration of the rendered face into the original video frame. Nonetheless, using an alternative audio source may lead to discrepancies in lip shapes between the rendered and original video frames. These discrepancies are particularly pronounced around the facial contour and chin, resulting in visual inconsistencies. To address these issues and enhance the integrity of the composite image, we introduce an Inpainting Generator, denoted as 𝐹 . It is designed to seamlessly fill in these mismatches and improve the visual continuity of the final frames. The blending process is described by the following equation: EQUATIONEQUATION)Gaussian Attributes. We aim to ensure that during optimization, the positions of the Gaussians faithfully align with the associated FLAME triangles and their sizes remain within reasonable bounds to avoid visual jitter and artifacts. To achieve this, we apply constraints on the Gaussians' position and scale:EQUATION)where 𝜖 𝑝 and 𝜖 𝑠 serve as predefined thresholds that set the maximum allowable positions for ū′ and scales for s, respectively.Gaussian Semantic Category. The dynamic deformation aims for the deformed Gaussians to approximate the associated FLAME mesh. Ideally, Gaussians bound to specific triangles, like the lips, should exactly match the corresponding region's color and position in the rendered image. However, it's a challenge to prevent Gaussians from shifting to implausible positions during optimization, resulting in jittering artifacts. To address this issue, we introduce a Gaussian semantic loss L 𝑠𝑒𝑔 . We render an auxiliary semantic segmentation map 𝑀 𝑎𝑢𝑥 using the Gaussian Renderer, where instead of utilizing the learned color 𝒄 𝑖 in Equation 9, we assign a fixed color 𝒄 𝑓 𝑖𝑥 to each Gaussian point based on its parent triangle's category; for example, assigning red to facial points and yellow to lip points:EQUATIONThen, we compute the mean square error between 𝑀 𝑔𝑎𝑢 and 𝑀 𝑔𝑡 obtained by face parsing [62] :EQUATIONConsequently, the final loss function is formulated as follows:EQUATION* Wav2Lip is jointly trained with SyncNet, and LSE-C and LSE-D are its optimization objectives. As a result, it obtains better scores than the ground truth. [8] and 100 samples of English HDTF videos [61] for joint training. During the data preprocessing phase, we begin by extracting audio and video frames from the source video. Subsequently, facial keypoints are extracted and smoothed [5, 25] , then employ EMOCA [13] to obtain FLAME parameters sequence. Finally, we apply the face parsing technique [62] , which segments each video frame into distinct facial regions.4.1.2 Compared baselines. We compare our method with several representative methods: 1) Wav2Lip [36] : Sync-expert supervised learning enhances lip sync accuracy; 2) AD-NeRF [17] : NeRF synthesizes audio-driven videos end-to-end; 3) RAD-NeRF [45] : Discrete learnable mesh boosts rendering efficiency; 4) ER-NeRF [26] : Three-plane hash representation enables high-quality rendering; 5) GeneFace++ [56] : Pitch-aware audio-to-motion module and landmark LLE method ensure stable rendering. 
implementation details.: 
 The official FLAME model lacks vertices and triangles for teeth, so we manually attach it with 262 vertices and 546 faces to depict the teeth and inner mouth regions. These newly added elements are not assigned independent BS and LBS weights but rather are designed to move in sync with the lip area of the original FLAME model. For GaussianTalker's training, we employ the Adam Optimizer [24] across all modules. The Speaker-specific Motion Translator is trained for 100,000 iterations, with the batch size set to 1. This training takes about 6 hours, using a learning rate of 1 × 10 -4 . Furthermore, we train the Dynamic Gaussian Renderer for 50,000 iterations using a batch size of 8, and this training lasts about 4 hours. While optimizing the Gaussians' attributes, we fine-tune the FLAME parameters within the dataset to ensure they more accurately reflect the face of each frame. All experiments are performed on a single NVIDIA RTX4090 GPU.In quantitative evaluation, we focus on both synthesized quality of the head and accuracy of lip movements.Our comparisons are divided into two settings: 1) Self-driven setting, where each speaker's own audio is used to drive the corresponding reference frame. 2) Cross-driven setting, where we extract two extra audio clips from HDTF [61] , named Testset A and Testset B, to drive each speaker. For each generated result, we rescaled frames into the same size for a fair comparison.We employ Peak Signal-to-Noise Ratio (PSNR) and Structural Similarity Index Measure (SSIM) for assessing overall image quality. To assess image details, we utilize Learned Perceptual Image Patch Similarity (LPIPS) [60] . To quantify the similarity between the generated video and the ground truth, we employ Fréchet Inception Distance (FID) [18] . For assessing audiolip synchronization, we utilize Landmark Distance (LMD) [9] to gauge the extent of synchronization with the ground truth, and complementarily evaluate lip movement accuracy through Lip Sync Error Confidence (LSE-C) and Lip Sync Error Distance (LSE-D) [36] .. The results are presented in Table 1 . Our results indicate that, in comparison with leading audio-driven talking head video synthesis methods, our method outperforms others, achieving the highest scores across all image quality assessment metrics. Owing to measures implemented to reduce jitter and artifacts, the videos produced by our method more accurately resemble the actual scenes. Regarding lip synchronization, although Wav2Lip excels in the LSE-C and LSE-D metrics owing to its joint training with SyncNet, our method still surpasses others in these metrics. Additionally, our method demonstrates superior performance in the LMD metric, suggesting a closer and more accurate alignment with actual lip movements. We also compared the generalization ability of each method using FID score and LSE-C and LSE-D metrics with out-of-distribution (OOD) audio inputs, detailed in Table 2 . The results show that our We also tested the rendering speed. With an NVIDIA RTX4090 GPU and data preloaded into memory, a sequence of video frames can be output at 130 FPS. The output video has the same resolution as the original video. This far exceeds the performance of other methods. In addition, we developed a rendering pipeline for Gaussians with MNN [21] and OpenGL [53] and subsequently deployed GaussianTalker. On devices with the Apple M1 chip, it achieved rendering speeds of 36 FPS, demonstrating its adaptability across various platforms.To more effectively assess image quality and lip synchronization, we present a comparative analysis of our method with others in Figure 4 . Compared to our method, Wav2Lip primarily concentrates on lip synchronization yet suffers low image quality, particularly around the mouth area. AD-NeRF and RAD-NeRF cannot naturally perform the blinking process, as evidenced by artifacts appearing in eye closure frames. Moreover, these two methods suffer from head-torso separation when driving cross-identity audio. ER-NeRF demonstrates noticeable jittering in both the head and torso during speech, which severely impacts the realism of the video. GeneFace++ shows limited expressiveness and lip movement variety, failing to perform actions like staring or frowning convincingly.Most NeRF-based approaches train the head and torso independently, which can compromise the torso's rendering quality in scenes with extensive torso movement. Furthermore, rendering can introduce artifacts stemming from the incomplete separation of the speaker from the background during preprocessing. In contrast, our approach not only delivers superior image clarity but also achieves enhanced lip synchronization performance, particularly with cross-identity and cross-gender audio inputs, showcasing robust generalization capabilities. Please see our supplementary for better visualization.In this section, we conducted ablation studies in a self-driven setting to validate the effectiveness of each component. Quantitative assessment metrics included PSNR, LPIPS, and LMD. The results are shown in Table 3 and Figure 5. 4.4.1 Universal Audio Encoder. We investigate the impact of audio decoupling in the audio encoder, which excludes identity information from the original audio and extracts audio feature that contains only content information. Removing audio decoupling decreases all metrics, with the LMD metric suffering the most, indicating poorer lip synchronization. Figure 5 (a) also illustrates the artifact issues in some of the lip shapes. 4.4.2 Speaker-specific BlendShapes. We explore the impact of eliminating the Speaker-specific BlendShapes, which play a crucial role 4.4.4 Gaussian Semantic Loss. We investigate the effect of ablating Gaussian semantic loss, which clarifies the binding relationship between 3D Gaussians and FLAME and further normalizes the motion of Gaussians. Without the Gaussian semantic loss, the speaker will jitter slightly, leading to a reduction in image quality metrics. 
conclusion: 
 In this work, we propose GaussianTalker, a novel framework for audio-driven talking head synthesis via 3D Gaussian Splatting integrated with the FLAME model. GaussianTalker associates multimodal data with specific speakers, reducing potential identity bias between audio, 3D mesh, and video. The Speaker-specific FLAME Translator employs identity decoupling and personalized embedding to achieve synchronized and natural lip movements, while the Dynamic Gaussian Renderer refines Gaussian attributes through a latent pose for stable and realistic rendering. Extensive experiments showed that GaussianTalker outperforms state-of-the-art performance in talking head synthesis, while achieving ultra-high rendering speed that significantly surpasses other methods. We believe that this innovative approach will encourage future studies to develop more fluid and lifelike character expressions and movements. By leveraging advanced Gaussian models and generative techniques, the animation of characters will go far beyond mere lip-syncing, capturing a broader range of character dynamics.  
 
 Appendix:  


Decision: The paper described a talking head video generation method which combines gaussian splatting with a parametric 3DMM. The Reviewers seem to have all a positive feeling about this paper, with one reviewer that changed mind after rebuttal and suggest weak rejection.

The paper seems to have several positive aspects, the most important being:
- Combining gaussian splatting with a parametric 3DMM is deemed novel and providing a "fresh perspective" for adding controllability.
- Results are good, surpassing the state of the art both in terms of quality and generation speed.
- Lots of potential practical applications.

On the other hand, some concerns are raised. Most of them were clarified in the rebuttal. The most critical issues are:
- The method only renders the facial region, while other frame details are taken from the ground-truth. This leads to artifacts, mostly in the neck region. 
- Given the above, some metrics could be biased.

Overall, the AC believes that the paper is valuable, and  the rebuttal was clear in solving reviewers' concerns. It provided additional evidence in support. Also, the rationale behind the choice of rendering only the facial part is reasonable i.e. audio features are totally uncorrelated to, for example, hair or background. Using those to generate such visual details is counterproductive. This seems a clever observation.

While there are diverging opinions (3 BA and 1 WR), the AC thinks positive aspects overcome negative ones. There still might be an issue with the metrics as their are computed: the AC agrees with BYRs that computing SSIM, LPIPS and FID in this setting is not fair with respect to previous methods that generate larger part of the images, yet it also shows that the proposed solution works better than rendering the entire head, given the reasoning above. The lip-sync metric instead is robust to that and supports the paper claim.

Ultimately, the AC thinks that the paper is worth acceptance, given that such experimental detail is specified in the camera ready and perhaps made also clear in the tables.

Reviews: review: paper_summary: paper_summary: This paper presents a talking head method based on dynamic 3DGS. The method introduces a speaker-specific motion translator that can generate precise facial motion parameters consistent with the target speaker's style based on the input audio. Moreover, the authors propose a dynamic Gaussian renderer that combines FLAME with 3D Gaussian point rendering, achieving realistic facial rendering and high frame rate real-time generation.
strength_weakness: strength_weakness: 1. This papaer realizes dynamic Gaussians by binding FLAME with 3D Gaussian Splatting, achieving real-time driving effects at 130 fps, surpassing previous methods based on NeRF，high FPS real-time rendering has certain application value..
2.The Speaker-specific Motion Translator presented in this work offers a fresh perspective by generating driving coefficients tailored to the target speaker.
3.The proposed method surpasses the compared methods in lip synchronization, image quality, and rendering speed.
limitations: limitations: 1.Is this paper the first talking head method based on 3DGS? If not, please provide a comparison with other 3DGS-based methods.
2.The proposed method segments the face and other head regions, only driving the facial area. It does not enable control over head pose. Although an Inpainting Generator is used to stitch the face, head, and background together, it may still introduce certain distortions.
paper_summary: paper_summary: The paper proposes a framework for audio-driven talking head synthesis using 3D Gaussian splatting coupled to FLAME, which generates lifelike rendered videos by associating data from different modalities with specific speakers.
strength_weakness: strength_weakness: 1. The paper proposes a framework for audio-driven talking head synthesis using 3D Gaussian splatting coupled to FLAME, which generates lifelike rendered videos by associating data from different modalities with specific speakers.
2. A speaker-specific motion translator decouples identity and uses personalized embedding for natural lip movements, while a dynamic Gaussian renderer refines Gaussian attributes through latent pose to ensure realistic visual effects.
limitations: limitations: 1. While the method proposed in this paper effectively addresses the common problem of jitter, it has a significant limitation in that it only generates the central facial region, using ground truth for the remaining parts. In contrast, other methods generate the entire head, which makes the comparison unfair, especially for metrics such as PSNR, SSIM, and LPIPS. In addition, the lip synchronization metrics, LSE-C and LSE-D, do not seem to reach state-of-the-art levels.
2. Since this paper uses a 3D approach and Flame for reconstruction, the authors should provide synthesized results of the head from different views.
3. The innovation in this paper seems to be incremental, leaning towards the use of existing models. Flame and 3DGS are used for rendering. And 3DGS has already been cited in the face domain, as seen in works like GaussianHead and MonoGaussianAvatar [1]. While this is acceptable, the authors should discuss the differences between the 3DGS used for rendering and GaussianHead and MonoGaussianAvatar.
[1] Yufan Chen, Lizhen Wang, Qijing Li, Hongjiang Xiao, Shengping Zhang, Hongxun Yao, and Yebin Liu. 2023. Monogaussianavatar: Monocular gaussian point-based head avatar. arXiv preprint arXiv:2312.04558 (2023).
paper_summary: paper_summary: The paper presents GaussianTalker, a novel framework for audio-driven talking head synthesis that leverages 3D Gaussian Splatting in conjunction with the FLAME model. GaussianTalker decouples identity from motion using a speaker-specific motion translator module to employ personalized embedding for natural lip movements. GaussianTalker achieves real-time rendering performance (130FPS on an Nvidia RTX 4090 GPU) while surpassing state-of-the-art methods in lip-sync and image quality.
strength_weakness: strength_weakness: 1. The method proposes to decouple the driving audio from the speaker's identity information, ensuring that the synthesized lip movements closely match the target speaker's style. This enables GaussianTalker to reproduce realistic talking head videos.
2. GaussianTalker achieves real-time rendering speed while capturing fine motion details of the talking face. The method shows superior ability in both self-driven and cross-driven tasks compared with the state-of-the-art methods.
3. The paper detailedly introduces a novel strategy that binds the 3D Gaussian to the FLAME mesh to enable realistic animation. They propose a Gaussian semantic loss $\mathcal{L}_{seg}$ to guarantee the color of each Gaussian conforms to its parent triangle's category.
limitations: limitations: 1. The paper assigns a fixed color to each Gaussian point based on its parent triangle's category, while in the original 3DGS[1], the pixel's color is determined by aggregating the contributions of all overlapping Gaussians, which may have different colors. An adequate analysis is necessary to validate the effectiveness and correctness of this modification.
2. While the hyper-parameter settings are mentioned in the supplementary materials, the paper does not include a discussion on how these settings were chosen or their impact on the network’s performance. A detailed examination of the hyper-parameter settings is crucial for understanding and replicating the training process effectively. 
3. The method separates the facial part from the torso and the background and renders them independently. Notably, artifacts are observed in the neck area of the generated videos. This raises concerns about potential quality degradation in scenarios with dynamic backgrounds or significant head movements. It would be beneficial for the paper to address these issues and explore potential solutions.

References:

[1] Bernhard Kerbl, Georgios Kopanas, Thomas Leimkühler, and George Drettakis. 2023. 3D Gaussian Splatting for Real-Time Radiance Field Rendering. (2023)
paper_summary: paper_summary: The paper introduces a novel method for generating talking head animations driven by audio inputs. This method employs 3D Gaussian Splatting, which significantly improves the accuracy and realism of facial animations, particularly enhancing lip synchronization and visual quality over previous techniques. The system is built around two main components: the Speaker-specific Motion Translator and the Dynamic Gaussian Renderer. The Motion Translator customizes facial movements to match the unique characteristics of the speaker's voice and style, while the Renderer uses speaker-specific blend shapes to add fine details to the face, thus enhancing the fidelity of the generated videos. GaussianTalker achieves impressive performance metrics, notably in rendering speeds suitable for real-time applications, and demonstrates substantial improvements in both quantitative and qualitative assessments over state-of-the-art methods. The study highlights potential applications in virtual reality, digital avatars, and other multimedia platforms, pushing forward the realism and personalization of digital human representations.
strength_weakness: strength_weakness: 1. The paper introduces an innovative use of 3D Gaussian Splatting for audio-driven talking head synthesis. This novel approach improves upon the traditional methods by enabling finer control over facial expressions and movements through a parametric facial model (FLAME).
2. GaussianTalker significantly enhances the realism and synchronization of lip movements with the audio input, which are crucial for creating believable talking head videos. Additionally, the method achieves high rendering speeds, making it suitable for real-time applications, a notable improvement over existing state-of-the-art methods.
limitations: limitations: 1. Previous NeRF-based methods[1][2] have rendered the entire head (including hair), but this method only renders the facial region, why is it designed this way?
2. The author mentions in line 821 that the inference speed reaches 140FPS after preloading the data into memory, but as far as I know, the inference process of the source code of RAD-NeRF and ER-NeRF includes a data load process, and I'm wondering if that's not a fair comparison in Table 1?
3. The experimental section lacks a user study, which is an important part of the assessment.
4. Experiments need to be compared with additional 2D-based methods to demonstrate validity, such as [3][4].
5. I know it is not compulsory, but publishing the code will greatly benefit the community and ease the difficulty of understanding.

[1] Ye Z, Jiang Z, Ren Y, et al. GeneFace: Generalized and High-Fidelity Audio-Driven 3D Talking Face Synthesis[C]//The Eleventh International Conference on Learning Representations. 2022.
[2] Li J, Zhang J, Bai X, et al. Efficient region-aware neural radiance fields for high-fidelity talking portrait synthesis[C]//Proceedings of the IEEE/CVF International Conference on Computer Vision. 2023: 7568-7578.
[3] Zhang Z, Hu Z, Deng W, et al. Dinet: Deformation inpainting network for realistic face visually dubbing on high resolution video[C]//Proceedings of the AAAI Conference on Artificial Intelligence. 2023, 37(3): 3543-3551.
[4] Zhong W, Fang C, Cai Y, et al. Identity-preserving talking face generation with landmark and appearance priors[C]//Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. 2023: 9729-9738.