MimicTalk: Mimicking a personalized and expressive 3D talking face in minutes

Authors: zhenhui  ye
tianyun  zhong
yi  ren
ziyue  jiang
jiawei  huang
rongjie  huang
jinglin  liu
jinzheng  he
chen  zhang
zehan  wang
xize  chen
xiang  yin
zhou  zhao
Venue: NeurIPS_cc_2024_Conference
Published: 2024-05-14 12:23:45

Abstract: Talking face generation (TFG) aims to animate a target identity's face to create realistic talking videos. Personalized TFG is a variant that emphasizes the perceptual identity similarity of the synthesized result (from the perspective of appearance and talking style). While previous works typically solve this problem by learning an individual neural radiance field (NeRF) for each identity to implicitly store its static and dynamic information, we find it inefficient and non-generalized due to the per-identity-per-training framework and the limited training data. To this end, we propose MimicTalk, the first attempt that exploits the rich knowledge from a NeRF-based person-agnostic generic model for improving the efficiency and robustness of personalized TFG. To be specific, (1) we first come up with a person-agnostic 3D TFG model as the base model and propose to adapt it into a specific identity; (2) we propose a static-dynamic-hybrid adaptation pipeline to help the model learn the personalized static appearance and facial dynamic features; (3) To generate the facial motion of the personalized talking style, we propose an in-context stylized audio-to-motion model that mimics the implicit talking style provided in the reference video without information loss by an explicit style representation. The adaptation process to an unseen identity can be performed in 15 minutes, which is 47 times faster than previous person-dependent methods. Experiments show that our MimicTalk surpasses previous baselines regarding video quality, efficiency, and expressiveness. Video samples are available at https://mimictalk.github.io .

Main paper: 
 introduction: 
 Audio-driven talking face generation (TFG) (Prajwal et al., 2020; Hong et al., 2022; Tian et al., 2024; Xu et al., 2024) is a cross-modal task that leverages multi-modal knowledge from speech, vision, and computer graphics to animate a target identity's face given arbitrary driving audio, aiming to create lifelike talking videos or interactive avatars. Personalized TFG (Suwajanakorn et al., 2017; Thies et al., 2020a; Guo et al., 2021; Lu et al., 2021b ) is a variant of TFG with several real-world applications, such as video conferencing and audio-visual chatbots, in which we emphasize that the generated results should have excellent perceptual similarity to a specific individual (from the perspective of both visual quality and expressiveness). In this setting, a video clip of the target identity (from several seconds to minutes) is provided as a detailed reference for the target person's personalized attributes, such as geometry shape (Mildenhall et al., 2021; Kerbl et al., 2023) and talking style (Wu et al., 2021; Ma et al., 2023; Tan et al., 2023) . To meet the quality requirement of high perceptual identity similarity, the community has focused on identity-dependent methods (Tang et al., 2022; Li et al., 2023a; Xu et al., 2023) , in which an individual model is trained from scratch for each target person's video. The primary reason for the prevailing of these identity-dependent approaches is that during the training process, the person-specific model can implicitly memorize nuanced personalized details of the target person's video, such as their speaking style and subtle expressions, which are hard to represent explicitly with hand-crafted conditions. As a result, these approaches could train small-scale person-dependent models, achieving high identity similarity and expressive results.However, the identity-dependent methods (Tang et al., 2022; Li et al., 2023a) face two well-known challenges: (1) the first is weak generalizability. The limited training data scale in the per-identityper-training restricts the generalizability to out-of-domain (OOD) conditions during inference. For example, lip-sync performance may degrade when driven by OOD audio (from a different language or speaker), and the rendering may fail when synthesizing OOD expressions (such as large yaw).(2) the second is low training and sample efficiency. Since the identity-dependent model needs to be trained from scratch on the target person video and no prior knowledge can be enjoyed, the training process can take more than several hours (low training efficiency), and it often requires over 1-minute length of training data to achieve reasonable lip-sync results (low sample efficiency). By contrast, another line of works, the identity-agnostic (or, say, one-shot) TFG methods (Zhou et al., 2020; Wang et al., 2021; Zhao and Zhang, 2022; Li et al., 2023b) , which incorporate various identities' video during training and only use one source image during inference, can achieve good generalizability to OOD conditions since various audio/expressions have been seen during the training. However, due to the limited information in the single input image, the one-shot methods cannot exploit the rich samples of the target identity to imitate its personalized attributes. Therefore, it is promising to bridge the gap between person-agnostic one-shot methods and person-dependent small-scale models to achieve expressive, generalized, and efficient personalized TFG.Based on the observation above, our key motivation is to dig out the prior knowledge of pre-trained person-agnostic TFG models (Li et al., 2024; Chu et al., 2024; Ye et al., 2024) to better mimic the personalized attributes of target identities. We propose MimicTalk, a high-quality and expressive personalized TFG framework, which utilizes the generalizability power of a large generic TFG model to improve the efficiency and robustness of TFG and devises a talking style-controllable motion generator to obtain results of high expressiveness and personalized characteristics. Specifically, (1) our method starts with a 3D person-agnostic generic model. This is due to the finding that the widely used warping-based methods cannot well handle large movements, and state-of-theart person-dependent methods have turned to Neural Radiance Field (NeRF) or Gaussian Splatting, which are advanced neural rendering techniques with strong 3D prior, to achieve video consistency and geometric realism. Inspired by this, we resort to a recent NeRF-based one-shot TFG method to provide a 3D-aware base model in our framework. (2) Given a pre-trained one-shot model, we design a static-dynamic (SD)-hybrid adaptation pipeline for an efficient and stable finetuning process. Specifically, we propose a tri-plane inversion method to learn a personalized 3D face representation to store the static texture details; we also first introduce low-rank adaptation (LoRA) in TFG to adapt the dynamic characteristics of the target speaker, which are difficult to model explicitly. (3) We note that generating motion of personalized talking styles is vital for perceptual reality in audiodriven TFG. Although finetuning the motion generation module on the target individual dataset can allow the network to implicitly memorize the target talking style, it leads to high computation costs and training instability. To address this, we propose an in-context-style audio-to-motion (ICS-A2M) model capable of mimicking the in-context zero-shot talking style. The ICS-A2M model is facilitated by a newly proposed training paradigm named audio-guided motion-infilling, which encourages the model to denoise the partially masked motion track by exploiting the implicit talking style revealed in the unmasked motion track. We also incorporate conditional flow-matching into the co-speech motion generation task, achieving state-of-the-art lip-sync and style-mimicking accuracy.The contributions of the paper are summarized as follows:• We are the first work that considers utilizing 3D person-agnostic models for personalized TFG. We propose an SD-hybrid pipeline for efficient and high-quality adaptation to learning the static and dynamic characteristics of the target speakers.• We propose the ICS-A2M model, which achieves high lip-sync quality and in-context talking style mimicking audio-driven TFG.• Our MimicTalk only requires a few seconds long reference video as the training data and several minutes for training. Experiments show that our MimicTalk surpasses previous person-dependent baselines in terms of both expressiveness and video quality while achieving 47x times faster convergence. 
related work: 
 Our work is mainly about personalized and expressive talking face generation. We discuss the related works in the field of talking face generation and expressive co-speech facial motion generation, respectively.Based on different real-world applications, talking face generation (TFG) methods have been majorly divided into two settings: identity-agnostic and identity-dependent. The identity-agnostic methods focus on the one-shot scenario: the model is provided with only one image and aims to animate it to create a video. These approaches hence train a large-scale generic model with a large amount of identity video data so that it can generalize to unseen photos during the inference stage. On the other hand, identity-dependent methods focus on achieving better video quality for a specific speaker: typically using a segment of the target user's video as training data and expecting the model to mimic the personalized features of that speaker, known as personalized TFG. These two lines of work have evolved independently over the years and have developed significantly different methodologies. (1) As for the person-agnostic methods, the earliest works (Chung et al., 2017; Prajwal et al., 2020; Wang et al., 2022) typically adopt a pixel-to-pixel framework (Isola et al., 2017) or generative adversarial network (GAN) setting to generate the result, which results in training instability and bad visual quality. Then, the majority of prior works (Averbuch-Elor et al., 2017; Ren et al., 2021; Wang et al., 2021; Hong et al., 2022; Zhao and Zhang, 2022; Hong and Xu, 2023; Liang et al., 2024; Jiang et al., 2024b) resort to a dense warping field (Siarohin et al., 2019) to deform the pixels of the source image given the 3D-aware keypoints extracted from the driving resource. The warping-based method achieves better image fidelity, yet due to a lack of 3D prior knowledge, it occasionally produces warping and distortion artifacts. Recently, to handle these artifacts, some work (Zeng et al., 2022; Sun et al., 2023; Li et al., 2023b,c; Ye et al., 2024) propose one-shot NeRF-based methods by learning to reconstruct a 3D face representation (tri-plane) (Chan et al., 2022) from the source image.(2) As for the person-dependent setting (Thies et al., 2020b; Yi et al., 2020; Lu et al., 2021a; Ye et al., 2022) , which trains an individual model on a specific video of the target identity, the recent works (Guo et al., 2021; Yao et al., 2022; Tang et al., 2022; Ye et al., 2023; Li et al., 2023a) are mainly based on NeRF for its high image fidelity and realistic 3D modeling. Despite the NeRF-based person-specific method achieving the best video quality and identity similarity among all TFG methods, it typically requires hours of training, and its generalizability to driving conditions is hampered by limited training data. To our knowledge, Mimictalk is the first work that considers bridging the gap between NeRF-based person-agnostic and person-dependent methods to improve the task of personalized TFG.The recent advance in neural rendering has significantly improved the image quality, temporal consistency, and stability of the synthesized video, making expressiveness the next focus of the TFG community. The critical challenge in achieving high expressiveness is to generate high-quality facial motion from audio content. This requires the generated motion sequence to not only synchronize with the audio track but also reflect a consistent and expressive talking style similar to the target speaker. Some studies implicitly model this process in a deterministic and end-to-end audio-toimage model (Prajwal et al., 2020; Guo et al., 2021) . For better controllability and audio-lip synchronization, other works propose explicitly modeling the audio-to-motion mapping using external generative models (Thies et al., 2020b; Ye et al., 2023) . However, these studies do not explicitly model the speaker's talking style. To address this, Wu et al. (2021) and Ma et al. (2023) developed a style vector extracted from arbitrary motion sequences to achieve explicit talking style control. EMMN (Tan et al., 2023) disentangles expression styles and lip motion, constructing a memory bank to produce lip-synced videos with vivid expressions. A concurrent work, VASA-1 (Xu et al., 2024) , proposes to add previous audio/motion latent as the input of the latent diffusion model to keep temporal consistency. We can see that most previous methods rely on intermediates to represent the talking style Jiang et al. (2024a) , risking information loss. In contrast, our method first achieves in-context-learning (ICL) talking style control, better preserving the target identity's talking style. 
mimictalk: 
 As shown in Fig. 1 , MimicTalk is a personalized and expressive 3D talking face generation framework, in which the personalized renderer inherits the rich facial knowledge from a person-agnostic generic model (discussed in Sec. 3.1) via a static-dynamic (SD)-hybrid adaptation pipeline (in Sec. 3.2). In Sec. 3.3, we propose an in-context stylized audio-to-motion (ICS-A2M) model to generate personalized facial motion, which is necessary to achieve high expressiveness in the generated video. We describe the design and training process in the following sections. Due to space limitations, we provide technical details in Appendix B.The previous personalized TFG works trained a separate model on the target speaker to memorize personalized information of the target person, in which NeRF is typically used as the underlying technology to better store the target speaker's geometry, texture, and other information. We aim to leverage the prior knowledge of pre-trained person-agnostic TFG models to achieve greater generalizability and efficiency than previous person-dependent methods. To this end, we resort to recent one-shot NeRF-based TFG works (Li et al., 2023b (Li et al., , 2024;; Ye et al., 2024; Chu et al., 2024) to construct a person-agnostic TFG model. Specifically, We build our base model on the basis of Real3D-Portrait (Ye et al., 2024) with its official implementation1 . As shown in Fig. 2 , the first step is reconstructing a canonical 3D face (formulated as tri-plane representation (Chan et al., 2022) ) from the target person's source image I src :EQUATIONwhere FaceRecon is an image-to-3D face reconstruction model based on SegFormer (Xie et al., 2021) that transforms the input image into a tri-plane representation Chan et al. (2022) . Then a light-weight SegFormer-based motion adapter could control the facial expression in the 3D face, and a volume renderer of Mip-NeRF (Barron et al., 2021) could render the dynamic talking face of arbitrary head pose by controlling the camera:EQUATIONwhere PNCC src and PNCC tgt are projected normalized coordinate codes (Li et al., 2023b) , which is rasterized from the 3DMM face (Paysan et al., 2009) is the driving camera to control the head pose. I raw is the volume-rendered low-resolution image, which is further processed by a super-resolution module to generate the final high-resolution result:EQUATIONWe provide detailed network structure of the FaceRecon, MotionAdapter, VolumeRenderer, and SuperResolution of person-agnostic renderer in Fig. 5 of Appendix B. For more details about the one-shot person-agnostic base model, please refer to (Ye et al., 2024) .With the pre-trained person-agnostic renderer introduced in Sec. 3.1, we could synthesize talking face videos for unseen identities without any adaptation. However, there exists a significant identity similarity gap between the un-tuned renderer and previous person-dependent methods, which is reflected in two aspects: (1) The static similarity, which measures whether the generated frame has same texture (e.g., wrinkles, teeth, and hair) or geometry details as the target identity;(2) The dynamic similarity, which describes the relationship between the input motion condition and the facial muscle/torso movement in the output facial image. To be more intuitive, trained on the large-scale talking face dataset with various speakers, our person-agnostic model learns a statistically averaged motion-to-image mapping for face animation, which produces talking person videos that are semantically correct yet lack personalized characteristics. By contrast, the previous person-dependent methods, by overfitting the model on one target identity, inherently learn the personalized motionto-image mapping in the model. Based on the observation above, we propose an efficient staticdynamic-hybrid (SD-Hybrid) adaptation pipeline to achieve good static/dynamic identity similarity, which is shown in Fig. 2 .Tri-Plane Inversion for Static Similarity. The canonical 3D face representation P cano , which is extracted from the source image by the pre-trained 3D face reconstruction model (Ye et al., 2024) , stores all static properties of the target identity (i.e., geometry and texture information). We find that the information loss in this feed-forward image-to-3D transform is the primary reason for the inferior static similarity in our method. To this end, inspired by previous GAN-inversion methods (Roich et al., 2021) , we propose a tri-plane inversion technique, which regards the canonical 3D face representation as a learnable parameter and optimizes it to maximize the static identity similarity. To be specific, as shown in Fig. 2 , when adapting our model to a specific identity given a video clip, we initialize the learnable tri-plane with the first frame's image-to-3D prediction, then optimize the tri-plane alongside other parameters.Talking style prompt Adaptation Process. As shown in Fig. 2 , with the SD-hybrid design, the rendering process of the predicted image can be expressed as:EQUATION) where P * cano is the learnable 3D face representation, which is initialized from the first-frame prediction by the pre-trained 3D face reconstruction model. SR denotes the super-resolution module, θ * 1 , θ * 2 , θ * 3 are the learnable LoRA parameters injected in the volume renderer, motion adapter, and the super-resolution module, respectively.During adaptation, a short video clip of a specific identity is utilized as the training data, and only the personalized tri-plane and LoRAs are updated. The training loss of the SD-hybrid adaptation is:EQUATION)where L 1 , L LPIPS , L ID , denotes L1 loss, LPIPS loss by VGG16 (Simonyan and Zisserman, 2014) , and identity loss by VGGFace (Cao et al., 2018) , respectively. We set the learning rate to 0.001, λ LPIPS = 0.2, λ ID = 0.1. Thanks to the static-dynamic-hybrid design, our method achieves good identity similarity while enjoying a fast and low-memory-cost adaptation process compared to existing person-dependent methods, illustrated in Table 1 . Besides, we demonstrate that our SD-Hybrid pipeline achieves good training and sample efficiency in Fig. 4 .In the above sections, we have proposed a unified framework for motion-conditioned talking face generation. Then, we present in-context stylized audio-to-motion (ICS-A2M) model to generate personalized facial motion for audio-driven scenarios.Audio-guided Motion Filling Task Inspired by the success of in-context learning methods in large language models (Liu et al., 2023 ) and text-to-speech synthesis (Le et al., 2023) , we devise an audio-guided motion infilling task. We visualize the audio-guided motion-infilling task's detailed training and inference pipeline in Fig. 7 of Appendix B.3. Specifically, the audio-motion pairs are temporally aligned and channel-wise concatenated and processed by the audio-to-motion model.During training, we randomly masked several segments in the motion track and trained the model with the motion reconstruction error on the masked segments. Since the model is provided with surrounding unmasked motion and the complete audio track, it learns to exploit the talking style in the motion context to predict the masked co-speech motion more accurately. During inference, as shown in Fig. 3 (a), we could concatenate the reference audio-motion pair as the talking style prompt, arbitrary driving audio as the condition, and a noisy placeholder for predicted motion as the input of the model. This way, the model could predict the audio-synchronized facial motion with the talking style provided in the style prompt.Audio-to-Motion Flow Matching Model We first introduce an advanced generative model, flow matching (Lipman et al., 2023) , to generate expressive facial motions. The model is parameterized by a transformer θ and predicts a flow field ϕ that stores the velocity v t = dxt dt of the data points x t pointing towards the target distribution. Similar to diffusion models, the time step t is increasing from 0 to 1 during the inference process, i.e., we have prior data points x 0 ∼ N (0, 1) and the target data points x 1 ∼ q(x), where q(x) denotes the ground truth data distribution. Due to space limitation, we provide detailed preliminaries of flow matching in Appendix B.3.2. To be intuitive, we visualize the forward process of our flow-matching-based audio-to-motion model in Fig. 3 (a) . The network input is the same as what we defined in the previous paragraph (a concatenation of the style prompt, audio condition, and noisy motion x t ). The model's output is the velocity v t of the noisy motion x t . And we could train the model with the conditional flow matching (CFM) objective:EQUATION)where v t (x, a, s; θ)is the predicted velocity by flow matching model of parameter θ, a and s is th audio condition and style prompt, and u t (x|x 1 ) is the ground truth velocity of the optimal transport (OT) path (Le et al., 2023) given the target motion x 1 and the noisy motion x t , which is exactly the unit vector pointing from x t to x 1 .Apart from the flow matching objective, we find it necessary to add a lip-sync loss on the denoised sample x1 to generate accurate and expressive facial motion. Specifically, we first solve the ODE problem in Eq. 9 to obtain the predicted sample x1 using the predicted velocity v t (x, a, s; θ), then feed it into a pre-trained audio-expression SyncNet (Ye et al., 2023) to obtain a discriminative sync loss L sync that measure the synchronization between the input audio and the predicted expression. Therefore, the training loss of the ICS-A2M model is:EQUATION)where λ sync = 0.05. During inference, as visualized in Fig. 3(b ), now that the A2M model could predict the velocity field of the data point, we could iteratively push the data from the Gaussian distribution to the target distribution by solving the ordinary differential equation in Eq. 9 of Appendix B.3.2 and finally obtain the predicted motion x1 to drive the personalized renderer in Sec. 3.2.Classifier-Free Guidance for Enhancing Style Mimicking Classifier-free guidance (CFG) is a popular inference method that manipulates the condition strength during the sampling process of conditional diffusion models. While CFG has become a standard technique in text-to-image Rombach et al. ( 2021) and other weak-conditioned generation tasks, we found it also helps enhance the style mimicking quality of our stylized audio-to-motion model. Specifically, we mix the two network predictions with/without style prompt to construct the CFG velocity v CFG t :EQUATION)where w is the CFG scale and we empirically set w = 2. v(x, a, s is the network prediction with both audio condition and style prompt as the input, and v(x, a, 0) is the predicted velocity with zero style prompt. We obtain the pre-trained person-agnostic renderer from the official implementation of Ye et al. (2024) . For the SD-Hybrid adaptation, we trained the model on 1 Nvidia A100 GPU, with a batch size of 1 and total iterations of 2,000, requiring about 8 GB of GPU memory and 0.26 hours. Regarding the ICS-A2M model, we trained it on 4 Nvidia A100 GPUs, with a batch size of 20,000 mel frames per GPU. The flow-matching-based ICS-A2M model was trained for 500,000 iterations, taking 80 hours. We provide full experiment details in Appendix C.Data Preparation. To evaluate the personalized renderers, we tested on ten 3-minute-long target person videos by (Tang et al., 2022) and (Ye et al., 2023) . To train the ICS-A2M model, we use a large-scale lip-reading dataset, voxceleb2 (Chung et al., 2018) , which consists of about 2,000 hours videos from 6,112 celebrities.Compared Baselines. We compare our method with three person-dependent methods: (1) RAD-NeRF (Tang et al., 2022) , (2) GeneFace (Ye et al., 2023) , and (3) ER-NeRF (Li et al., 2023a) . We also compare with a style-oriented TFG method that considers controlling the talking style, (4) StyleTalk (Ma et al., 2023) . We discuss the characteristics of all test methods in Appendix A. 
quantitative evaluation: 
 We use CSIM to measure identity preservation, PSNR, FID to measure the image quality, and AED (Deng et al., 2019) and SyncNet confidence (Chung and Zisserman, 2017) to measure audio-lip synchronization. The results are shown in Table 1 . We have the following observations: (1) Thanks to the powerful flow matching model and the in-context-style-mimicking ability, our method achieves the best lip accuracy (AED) and perceptual lip-sync quality (SyncNet confidence); (2) Our SDhybrid adaptive renderer shows better visual quality than the person-specific baselines.(3) Thanks to the efficiency of the LoRA-based adaptation process, our method requires significantly less training time to adapt to a new identity within 2,000 iterations and 15 minutes (47x faster than RAD-NeRF).It also requires a low GPU memory usage (8.239 GB) for adaptation.We provide demo videos at https://mimictalk.github.io. We also adopted several case studies to demonstrate better performance. Specifically, (1) our SD-Hybrid adaptation has better train- Talking Style-Coherent Motion Prediction by ICS-A2M Model Given a short reference video as the talking style prompt, our ICS-A2M model could accurately mimic its talking style (such as smiling or duck mouth). We provide a demo video at https://mimictalk.github.io/static/ videos/demo_ics_a2m.mp4 for better demonstration. We also conducted a comparative mean score opinion (CMOS) test between our method and StyleTalk (Ma et al., 2023) to qualitatively evaluate talking style accuracy. As shown in Table 3 , our method achieves better talking style controllability and identity similarity. Please refer to Appendix C.3 for detailed user study settings.We conducted the Mean Opinion Score (MOS) test to evaluate the perceptual quality of generated samples, scaled from 1 to 5. Following Chen et al. (2020) , the attendees are required to rate the ICS-A2M Model. We also analyze three settings in the ICS-A2M: (1) replace the flow matching model with a deterministic transformer, as shown in line 2 of Table 5 , which leads to worse motion reconstruction quality and worse sync score;(2) replace the in-context style control with a handcrafted style vector in (Wu et al., 2021) or learn a style encoder as in StyleTalk Ma et al. (2023) , as shown in line 3 and line 4, which leads to worse motion reconstruction quality, proving that the ICL talking style mimicking could prevent information loss caused by compressing the style into a global encoding;(3) remove the sync loss during training, as shown in line 5, which leads to significantly worse perceptual lip-sync performance. 
conclusion: 
 In this paper, we propose MimicTalk, an efficient and expressive personalized talking face generation framework. We first come up with the idea of adapting a pre-trained 3D person-agnostic model to personalized datasets to inherit its generalizability and achieve fast training. The SD-Hybrid adaptation pipeline helps the generic model learn the target person's static and dynamic features, leading to better identity similarity than previous person-dependent baselines. Besides, the proposed ICS-A2M model is the first facial motion generator that enables in-context talking style control, which helps produce expressive facial motion in the generated video. Due to space limitations, we provide impact statements in Sec. E and discuss limitations and future works in Appendix D. Question: Does the paper provide open access to the data and code, with sufficient instructions to faithfully reproduce the main experimental results, as described in supplemental material?Answer: [Yes] Justification: We have provided model details and training details to improve reproducibility. We will also release the code after the review phase.Guidelines:• The answer NA means that paper does not include experiments requiring code. Guidelines:• The answer NA means that there is no societal impact of the work performed.• If the authors answer NA or No, they should explain why their work has no societal impact or why the paper does not address societal impact. • Examples of negative societal impacts include potential malicious or unintended uses (e.g., disinformation, generating fake profiles, surveillance), fairness considerations (e.g., deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations. • The conference expects that many papers will be foundational research and not tied to particular applications, let alone deployments. However, if there is a direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that an improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point out that a generic algorithm for optimizing neural networks could enable people to train models that generate Deepfakes faster. • The authors should consider possible harms that could arise when the technology is being used as intended and functioning correctly, harms that could arise when the technology is being used as intended but gives incorrect results, and harms following from (intentional or unintentional) misuse of the technology. • If there are negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition to attacks, mechanisms for monitoring misuse, mechanisms to monitor how a system learns from feedback over time, improving the efficiency and accessibility of ML). 
safeguards: 
 Question: Does the paper describe safeguards that have been put in place for responsible release of data or models that have a high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)? Answer: [Yes] Justification: Please see Appendix E. 
 
 Appendix: As for the person-dependent talking face renderer, all previous methods like RAD-NeRF (Tang et al., 2022) , GeneFace (Ye et al., 2023) , and ER-NeRF (Li et al., 2023a) follow a per-identity-pertraining paradigm, which means they have to learn a model from scratch for an unseen identity, which is time-consuming and the generalizability of the model is limited due to the small scale of training data. By contrast, our method is the first work to adapt a pre-trained generic one-shot 3D talking face model into the target identity. Thanks to using LoRA and the tri-plane inversion, our method could converge in 2,000 iterations (47 times faster than RAD-NeRF) and enjoy better generalizability, sample efficiency, and training efficiency due to the generic backbone.As for the audio-to-motion model in the audio-driven TFG, most previous works like SadTalker (Zhang et al., 2023) and StyleTalk (Ma et al., 2023) utilize a deterministic mapping to model the audio-to-motion transform, and cannot achieve talking style control. By contrast, our method first introduces flow-matching for the audio-to-motion task and achieves in-context talking style control.We provide detailed network structure of the person-agnostic renderer in Figure 5 .We visualize a detailed process that plugs the LoRA into our person-agnostic renderer in Fig. 6 . For each pretrained kernel or weight of shape c in × c out , the LoRA Matrix A and LoRA Matrix B could represent a weight matrix of shape c in × c out and rank r, where r << min(c in , c out ) and we set r = 4 in our setting. During training, the pre-trained weights of the backbone are fixed, and only the LoRA matrices are updated.In Fig. 7 , we present a visualization of the training and inference process for the Audio-Guided Motion Infilling task. Paired audio-motion samples are required for this task, which can be easily extracted from talking face datasets and used as training data. During training, we randomly mask several segments in the motion track and encourage the model to reconstruct them based on the complete audio track and the unmasked motion context. This training approach enables the model to learn to mimic the talking style provided in the context. During inference, there are two main usage scenarios. Firstly, we can guide the model to mimic a specific talking style by providing an audio-motion pair of the target person as the talking style prompt. This setting allows for in-context talking style control over the generated facial motions. Alternatively, we also support audio-only motion sampling. Without a style prompt, the model will generate semantically correct facial motions with a randomly sampled talking style.In this section, we introduce preliminaries of flow matching. Conditional flow matching (CFM) is a variant of continuous normalizing flows (Chen et al., 2018) , which is a class of generative models for modeling an unknown data distribution q(x), where x denotes the data point in the distribution. The CFM method aims to predict a time-dependent flow field that represents the velocity u t = dxt dt of the data point x t at a continuous timestep t, where the data point x 0 ∼ p(x) starts from a simple prior distribution p(x) (e.g., Gaussian distribution) at the timestep t = 0, and is pushed towards the target distribution q(x) by the velocity field as the timesteps finally approach t = 1. This process can be formulated as an ordinary differential equation (ODE) as follows:EQUATIONwhere u t is the ground truth velocity for x t and is expected to be approximated by the CFM model. v t is the predicted velocity by the CFM model with the parameter θ. Once the u t is obtained, we can train the model with a simple flow-matching objective:EQUATION)Once the training is done, we can obtain v t (x t ; θ) ≈ dxt dt , and solve the ODE in Eq. 9 to obtain the predicted sample x 1 . Now, we consider the specific formulation of the ground truth velocity u t . We adopt the optimal transportation (OT) path proposed in (Lipman et al., 2023) to obtain the ground truth velocity. Specifically, given the target data point x 1 and the current data point x t , the most efficient way is to go with a straight line, i.e., u t = (x 1 -x t )/(1 -t), where (1 -t) is a normalization term to make the ground truth velocity a unit vector. We choose the OT path for its simplicity and prior evidence (Lipman et al., 2023; Le et al., 2023) that it outperforms other alternatives (such as diffusion paths (Ho et al., 2020; Song et al., 2021) ).In the following sections, we illustrate the model configuration and training details of our Mim-icTalk.We provide detailed hyper-parameter settings about the model configuration in Table 6 . We use the pre-trained one-shot person-agnostic renderer provided in the official implementation3 of (Ye et al., 2024) .For the SD-Hybrid adaptation, we trained the model on 1 Nvidia A100 GPU, with a batch size of 1, requiring about 8 GB of GPU memory. Surprisingly, our method achieved better results than existing person-specific baselines in just 2,000 iterations, which took about 0.26 hours and was 47 times faster than RAD-NeRF.Regarding the ICS-A2M model, we trained it on 4 Nvidia A100 GPUs, with a batch size of 20,000 mel frames per GPU. The flow-matching-based ICS-A2M model was trained for 500,000 iterations, taking 80 hours.As for mean score opinion (MOS) test in Table 2 , we select 5 audio clips and 10 trained identities (as used in by Tang et al. (2022) and Ye et al. (2023) ) to construct 50 talking portrait video samples for each method. Each video has been rated by 20 participants. We perform the identity similarity, visual quality, and lip-synchronization MOS evaluations. For MOS, each tester is asked to evaluate the subjective score of a video on a 1-5 Likert scale. For identity similarity, we tell the participants to "only focus on the similarity between the identity in the source image and the video"; for visual quality, we tell the participants to "focus on the overall visual quality, including the image fidelity and smooth transition between adjacent frames"; as for lip synchronization, we tell the participants to "only focus on the semantic-level audio-lip synchronization, and ignores the visual quality".As for comparative mean score (CMOS) test in Table 3 , we first trained 10 person-specific renderers on 10 identities' ten-second-long videos. We randomly select 5 out-of-domain audio clips for driving each renderer. So there are 50 result videos for each setting. We include 20 participants in the user study. Each tester is asked to evaluate the subjective score of two paired videos on a -3 +3 Likert scale(e.g., the first video is constantly 0.0, and the second video is +3 means the tester strongly prefers the second video). To examine the aspect of (lip-sync, pose-sync, expressiveness), we tell the participants to "only focus on the (lip-sync, pose-sync, expressiveness), and ignore the other two factors.".In this section, we discuss the limitations of the proposed method and how we plan to handle them in future work. Firstly, in this paper, our main focus is on the face segment. By contrast, the rigid modeling of the hair and torso segment are relatively naive and occasionally produce artifacts. We plan to adopt conditional video diffusion models like (Hu et al., 2023) to enhance the naturalness of the hair and torso segments. Secondly, we can consider more conditions, such as eyeball movement and hand gestures. Finally, the inference speed (15 FPS on 1 A100) can be improved by introducing more efficient network structures like Gaussian Splatting.In this section, we discuss the ethical impacts that might be brought by the rapidly developing talking face generation technology and our measures to address these concerns.MimicTalk facilitates efficient and expressive personalized talking face synthesis. With the development of talking face generation techniques, it is much easier to synthesize talking human portrait videos. Under appropriate usage, this technique could facilitate real-world applications like virtual idols and customer service, improving the user experience and making human life more convenient. However, the talking face generation method can be misused in deepfake-related usages, raising ethical concerns. We are highly motivated to handle these misusage problems. To this end, we plan to include several restrictions in the license of MimicTalk. Specifically,• We will add visible watermarks to the video synthesized by MimicTalk so that the public can easily tell the fakeness of the synthesized video. • The synthesized videos should only be used in educational or other legal usages (like online courses), and any abuse will take responsibility by tracking the method we come up with in the next point. • We will also inject an invisible watermark into the synthesized video to store the information of the video maker so that the video maker has to account for the potential risk raised by the synthesized video.• The answer NA means that the paper does not include experiments.• If the paper includes experiments, a No answer to this question will not be perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data are provided or not. • If the contribution is a dataset and/or model, the authors should describe the steps taken to make their results reproducible or verifiable. • Depending on the contribution, reproducibility can be accomplished in various ways.For example, if the contribution is a novel architecture, describing the architecture fully might suffice, or if the contribution is a specific model and empirical evaluation, it may be necessary to either make it possible for others to replicate the model with the same dataset, or provide access to the model. In general. releasing code and data is often one good way to accomplish this, but reproducibility can also be provided via detailed instructions for how to replicate the results, access to a hosted model (e.g., in the case of a large language model), releasing of a model checkpoint, or other means that are appropriate to the research performed. • While NeurIPS does not require releasing code, the conference does require all submissions to provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution. (d) We recognize that reproducibility may be tricky in some cases, in which case authors are welcome to describe the particular way they provide for reproducibility.In the case of closed-source models, it may be that access to the model is limited in some way (e.g., to registered users), but it should be possible for other researchers to have some path to reproducing or verifying the results.https://github.com/yerfor/Real3DPortraitWe release the source code at https://mimictalk.github.io.https://github.com/yerfor/Real3DPortrait/ 


Decision: The paper introduces a new approach to improve the efficiency and robustness of personalized talking face generation. The paper receives unanimous accept from all reviewers. The authors are suggested to further revise according to the comments of the reviewers in the final version.

Reviews: review: paper_summary: paper_summary: This paper targets to tackle efficient 3D realistic talking face customization. Rather than learning an individual neural radiance field for each identity, this work exploits a person-agnostic model to improve the efficiency and robustness of personalized talking face generation. A static-dynamic-hybrid adaptation pipeline is proposed to adapt a person agnostic base model to a specific identity. To achieve style control, an in-context audio-to-motion model is devised to mimic the talking style of reference video.
strength_weakness: strength_weakness: 1. According to the attached supplementary video, the improvement seems very marginal or even hard to notice. Also, only three personal identities are showcased. It will be more persuasive if more identities are demonstrated. I am concerned whether the PNCC representation offers sufficient information or not. At the same time, the simultaneous optimization of a learnable reconstructed face and LoRA looks weird, making it confusing about which component is really functioning. 

2. Authors claim this approach is a fast adaptation method that significantly surpasses the previous person-dependent approaches. In the comparison approaches, all the 3 Nerf based methods do not aim to improve the efficiency. Is there any related work that also handles the efficiency of generation or this is the first work. If not, authors should also include relevant approaches in this paper. Specifically, there might be other works targeted to efficiently fitting to a specific identity rather than adaptation. Will it be better to also take them into consideration instead of making a strong claim of 47 times speed faster.

3. The information in the workflow of Fig.2 is minimal. The representation of the NerF field uses a cube, which might have a clearer description. Meanwhile, each component of the devised network is drawn as a block diagram, which might require clearer description on the specific architecture details (If not here, the appendix). Similarly, for the inference pipeline, authors spent a lot of effort to illustrate the flow-matching training paradigm, which makes this figure a little bit messy.
questions: questions: Please refer the weakness section.
limitations: limitations: Authors adequately addressed the limitations.
paper_summary: paper_summary: This MimicTalk work aims to bridge the gap between the person-agnostic one-shot TFG setting and the person-dependent small-scale TFG setting. A carefully designed static-dynamic-hybrid adaptation pipeline is proposed to achieve expressive, generalized, and efficient personalized TFG. Furthermore, an in-context stylized audio-to-motion model is introduced to mimic the talking style from the reference video. Experiments demonstrate the superiority of MimicTalk compared to previous methods.
strength_weakness: strength_weakness: There are several losses used in the pipeline. How to choose these loss weights and how these loss weights affect the performance could be further discussed.
questions: questions: The loss weight choice could be discussed.
limitations: limitations: The limitations have been discussed.
paper_summary: paper_summary: The paper presents MimicTalk, an approach to improve the efficiency and robustness of personalized talking face generation. Instead of using separate NeRFs for each identity, MimicTalk adapts a person-agnostic NeRF-based model for specific individuals. They also propose an in-context stylized audio-to-motion (ICS-A2M) model to create facial movement that imitates the target person's speaking style. The adaptation process converges quickly for unseen identities. Experiments show that MimicTalk outperforms previous baselines in terms of video quality, efficiency, and expressiveness.
strength_weakness: strength_weakness: 1.  The head pose of the talking face video is not generated, for a talking face video, the sync of the head pose and audio is very important.
2. The facial motion is represented by PNCC, which limits the expressiveness of the generated results, e.g. detailed emotions, eyeball movements.
questions: questions: 1.  The $L_{sync}$ provided in **GeneFace** is used to evaluate the synchronization between audio and 68 sparse landmarks. In this paper, the authors call it audio-expression synchronization loss, which is not adequate. Sparse landmarks cannot reflect the richness and expressiveness of facial motions. It seems the judgement of the motion expressiveness (FID/user study) is missing in this paper.
2.  The results of talking style control are not sufficient. Only two examples are given. More qualitative/quantitative evaluations of the style controllability are needed.
3. Since this is about 3D talking face generation. It would be better to see some novel-view results.
limitations: limitations: The authors have discussed the limitations and broader impacts in their appendix. The authors should also discuss the limitations as I list in the **Weaknesses** section.
paper_summary: paper_summary: The paper presents a novel approach namely MimicTalk to personalized talking face generation. Unlike previous methods that rely on individual neural radiance fields (NeRF) for each identity, the authors propose a more efficient and generalized framework using a person-agnostic generic model. MimicTalk introduces a static-dynamic-hybrid adaptation pipeline and an in-context stylized audio-to-motion model. The proposed method aims to achieve high-quality, expressive results while significantly reducing training time. Experimental results suggest that MimicTalk surpasses previous baselines in video quality, efficiency, and expressiveness.
strength_weakness: strength_weakness: 1. The paper claims superior performance over existing methods, but the experimental results presented are not sufficiently comprehensive or detailed to fully substantiate these claims. Although some video results are provided in the URL, the results are not convincing enough. In particular, I cannot see the advantages of the proposed method in the style-control results. There are also not enough examples for comparison to state-of-the-art approaches. 

2. The experiments are limited in scope, primarily focusing on a narrow set of scenarios and datasets. The evaluation lacks a diverse range of conditions such as varying lighting, occlusions, and different levels of head movements, which are crucial for real-world applicability. More comprehensive testing across varied and challenging scenarios would strengthen the validity of the claims.
questions: questions: See weaknesses section.
limitations: limitations: The paper addressed the limitations in the appendices.

Comments: title: Dear Reviewer
Dear Reviewer
Hoping that our response could address your concern
Hoping that our response could address your concern
Thanks for your acknowledgment of our rebuttal
Final Rating
Dear AC and Reviewers,
Author Response to Reviewer iWSN
Thanks for the detailed rebuttal
Author Response to Reviewer TKxi (Part 2/2)
Author Response to Reviewer TKxi (Part 1/2)
Missing details for rebuttal response
Rebuttal by Authors
comment: Dear Reviewer dHKC.

As the discussion period is closing in several hours, we would like to know if there are any additional questions. We are glad to answer them.

Again, we sincerely appreciate your insightful review and positive recommendation for our paper.
Dear Reviewer fWzj.

As the discussion period is closing in several hours, we would like to know your feedback on our rebuttal and if there are any additional questions. We are glad to answer them.
Dear Reviewer dHKC,

Thank you again for your time and effort in reviewing our work! We would appreciate it if you can let us know if our response has addressed your concern. As the end of the rebuttal phase is approaching, we look forward to hearing from you and remain at your disposal for any further clarification that you might require.
Dear Reviewer fWzj,

Thank you again for your time and effort in reviewing our work! We would appreciate it if you can let us know if our response has addressed your concern. As the end of the rebuttal phase is approaching, we look forward to hearing from you and remain at your disposal for any further clarification that you might require.
Dear Reviewer TKxi,

Thanks for your engaged discussion and acknowledgment of our rebuttal. We'd like to especially thank you for the three suggestions (details of user study, head pose prediction experiments, and clarifying the contribution of style mimicking). We have integrated the latest discussion and additional experiments into the revised manuscript, which we found has improved the soundness and completeness of the paper.

Again, thank you for your expertise and positive feedback!
After reading your response, most of my questions have been addressed. However, I believe that compared to the submitted version, many changes are needed. For example, the user studies, the experiments on pose generation, and the claims regarding the contribution of style mimicking should be revised. I have mixed feelings about this. I improve the rating, but I am more inclined to suggest resubmitting to another conference to refine these issues. The final decision can be left to the area chair.
Thanks again for your great efforts and valuable comments.

We have carefully addressed the main concerns and provided detailed responses to each reviewer. We hope you might find the responses satisfactory. As the end of the rebuttal phase is approaching, we would be grateful if we could hear your feedback regarding our answers to the reviews. We will be very happy to clarify any remaining points (if any).
Dear Reviewer iWSN,

Thanks for your acknowledgment of our rebuttal. We will add the discussion on the design of the multi-objective loss functions of our model in the Method Section. We will also add the additional experiments on OOD poses and the limitation discussion of PNCC in the revised manuscript.

Again, we'd like to thank you for your precious time, expertise, and deep understanding of our work.
I have read other reviews and author response. My concers have been addressed. The additional experiments on OOD poses and the limitation discussion of PNCC (raised by Reviewer TKxi) are interesting points to explore. I suggest including these discussions and results in the final version.
> Q3: I checked Rebuttal Demo 4 for more style prompts. The results aren't good. A larger CFG sometimes enhances the style's intensity, but it also adds more artifacts. These results don't seem to support the claim of style mimicking.
- We have provided an additional **Post-Rebuttal Demo 1**. Please kindly refer to our demo page for a quick view.
- Thanks for your feedback. Firstly we want to **clarify the claim of style mimicking**. We acknowledge that the claim of style mimicking in the initial manuscript is somewhat misleading. Note that the task of this paper is personalized talking face generation (TFG), in which we have a short training video clip of the target identity, and we aim to train a TFG system that mimics not only its visual attributes but also its talking style. Therefore, in personalized TFG, we only need to run our model with the in-domain style prompt of the target identity (**in-domain style prompt** denotes using the training video clip as the style prompt). Actually, in Lines 74-78 of the original manuscript, we claim that the initial intention of proposing the ICS-A2M model is to let the model in-context (i.e., without the need for finetune) mimic the target identity's talking style. By contrast, as for supporting out-of-domain style prompts (**OOD style prompt** denotes cross-identity videos that have critical expressions like duck face that are unseen in the training video of the target identity), we regard it as an interesting feature of our method, rather than the key problem to be solved in this paper. As shown in **Post-Rebuttal Demo 1**, using the target identity's in-domain style prompt, our method could produce expressive results with good identity similarity and style similarity, and the performance is robust to a large CFG scale (e.g, cfg_scale=4). This result demonstrates that our method achieves the goal of personalized TFG (i.e., mimicking the target identity's visual attributes and talking style). We feel sorry for causing the misunderstanding, and **we will revise the claim of style mimicking** in the original manuscript by adopting the above discussion into the Introduction Part. We will emphasize that the major goal of the proposed ICS-A2M model is to better mimic the talking style of the target identity to achieve personalized TFG, and **we will introduce OOD talking style mimicking as an interesting feature** of our model in the Experiment Part. 
- Then we want to discuss **why the OOD style mimicking is not very robust**. In the Rebuttal Demo 4, we used various OOD style prompts to drive the target identities (e.g., Obama never played a duck face in the training video clip, but we set Trump's duck face video clip as the style prompt of the Obama renderer). Considering the scarce of motion-image pair in the ten-second-long training video, it is hard to render OOD extreme expressions without artifacts (e.g., for a model trained on a video that the speaker never says "oh", it is difficult to synthesize the speaker saying "oh" with good image quality). To analyze the reason for the artifact, we visualize the facial motion generated by our ICS-A2M model at various CFG scales and notice that the **ICS-A2M model could faithfully produce lip-sync facial motion of OOD talking styles at all tested CFG scales**. Now that the audio-to-motion stage works well, we suspect that the reason behind the visual artifact is that, when increasing the CFG scale, the generated facial motion is getting more similar to the OOD style prompt. However, the facial motion pattern in the OOD style prompt is quite different from the training motion condition of our person-specific renderer, hence resulting in visual artifacts.
- To **improve the robustness of OOD style mimicking**, we found two possible solutions: **1)** the first is to **tune the CFG scale** as a hyper-parameter. For instance, in Rebuttal Demo 4, while CFG=4 occasionally leads to artifacts, CFG=2 is a stable choice that balances the style similarity and visual quality. **2)** the second direction is to improve the generalizability of the motion-conditioned renderer. We could adopt **data augmentation** to synthesize images of diverse facial motions through recent advanced face editing methods, which can improve OOD performance, and hence could facilitate more stable OOD style mimicking. Due to time limitations, we leave this as future work.

Finally, we'd like to thank you for your precious time and valuable comments, which have improved the soundness and clarity of this manuscript. We will be very happy to clarify any remaining points (if any).
Dear Reviewer yTCz,

Thanks for your fast response! Sorry for the late reply since we were doing the relevant additional experiments. We hope our response fully resolves your remaining concerns. 

> Q1:  It would be better to model the audio-to-motion and audio-to-pose together. Please include some evaluations of the audio-to-pose in your revised version.
- Thanks for your feedback. In the initial rebuttal, we train a new audio-to-pose model from scratch due to its simplicity. Following your suggestion, we tried to **jointly model the audio-to-pose and audio-to-face motion together** . Specifically, we load the pre-trained audio-to-motion model and change the output dimension of its final layer: from the 64 dimensions (3DMM expression) to 70 (64+6) dimensions, in which the  6 additional dimensions denote the predicted head pose, including Euler angle (3 dim) and head translation (3 dim). We finetuned the audio-to-motion/pose model for 50,000 steps until reaching its convergence. 
- We then perform a **quantitative evaluation to evaluate audio-lip/pose synchronization and expressiveness** in different settings of the audio-to-motion/pose model. Specifically, we finetune 10 person-specific renderers on 10 ten-second-long training videos. Then we use the different settings of the generic audio-to-motion/pose model to predict the face motion or head pose. To be coherent with real-world applications, we randomly select 5 out-of-domain audio clips for driving each renderer. So there are 50 result videos for each setting. Since there is no ground truth for the result videos, we turn to user study for evaluation. We included 20 attendees and asked them to evaluate the comparative mean opinion score between each setting with the results of our original version. The attendees are required to rate in terms of audio-lip-sync, audio-pose-sync, and overall expressiveness. Please refer to **Q2 & A2 for more details of the user study**. The results are shown as follows. The error bars are 95\% confidence interval.
| settings      | CMOS audio-lip-sync | CMOS audio-pose-sync     | CMOS-expressive  |
| :---        |    :----:   |             :----:   |      :----: |
| Audio-to-Motion + poses extracted from other videos （original version）      | 0.000       | 0.000   | 0.000   |
| Audio-to-Motion + Audio-to-Pose, separately (initial rebuttal version)   | $0.215 \pm 0.293$        | $1.341 \pm 0.284$      |  $0.647 \pm 0.251$   |
| Audio-to-Motion/Pose, jointly (current version)  | $0.468 \pm 0.235$        | $1.653 \pm 0.316$      |  $0.952 \pm 0.301$   |

- In line 1 and line 2, we can see that predicting the pose from audio greatly improves the audio-pose-sync performance, and in line 3 we can see that jointly predicting the face motion and head pose could further improve the performance from audio-lip-sync, audio-pose-sync, and expressiveness. We suppose that it is because the head pose strictly aligns with face motion, and there is a high correlation between them (facial motion can be a strong hint for predicting the head pose and vice versa). Therefore, joint modeling of motion and pose could improve the model's ability to better predict them. We found the additional modeling of head pose greatly improves the naturalness of our method, and will add the above discussion in the revised manuscript. Thanks for your insightful suggestion!

> Q2: Could you provide more details about the user study? e.g., how many individuals participated and how many cases were evaluated?

A2: In the additional user study during the rebuttal, we follow the main setting as we did in the paper. Specifically, we first trained 10 person-specific renderers on 10 identities' ten-second-long videos. We randomly select 5 out-of-domain audio clips for driving each renderer. So there are **50 result videos for each setting**. We include **20 participants in the user study**. The biggest difference is that we test the **Comparative Mean Opinion Score (CMOS)** instead of the Mean Opinion Score (MOS) in the original paper to better evaluate the comparative performance between different settings. For CMOS, each tester is asked to evaluate the subjective score of two paired videos on a -3~+3 Likert scale(e.g., the first video is constantly 0.0, and the second video is +3 means the tester strongly prefers the second video). To examine the aspect of (lip-sync, pose-sync, expressiveness), we tell the participants to *"only focus on the (lip-sync, pose-sync, expressiveness), and ignore the other two factors."*

Due to space limitations, please refer to the next comment for the **reply to Q3**.
Thank you for your response. However, I still have some questions and concerns as follows:

1. Why do you model audio-to-pose and audio-to-face motion separately? Since head pose should align with face motion, it would be better to model them together. I understand the rebuttal time is limited, but if you want to claim to handle head pose generation, you should include some evaluations in your revised version.

2. Could you provide more details about the user study? e.g., how many individuals participated and how many cases were evaluated?

3. I checked **Rebuttal Demo 4** for more style prompts. The results aren't good. A larger CFG sometimes enhances the style's intensity, but it also adds more artifacts. These results don't seem to support the claim of style mimicking.
# Author Response to Reviewer fWzj (Part 2/2)

> Q5: The representation of the NeRF field needs a clearer description. Meanwhile, each component of the devised network is drawn as a block diagram, which might require a clearer description of the specific architectural details.
- A5: Thanks for your suggestion. We use tri-plane representation to represent the 3D NeRF field. We have replaced the cube with a tri-plane image and added a label description to improve the clarity of Figure 2 in the original paper. Please refer to **Figure 1 in the attached one-page PDF** for details. As for the specific architectural details of each block diagram, we provide an additional figure that plots the architectural details of each component in our generic model. Please refer to **Figure 2 in the attached one-page PDF** for details. Actually, in the original paper, we omitted the structure details because we want to imply that our SD-Hybrid method can be applied to an arbitrary one-shot person-agnostic model, not only the model we used in the paper. However, we acknowledge that it is necessary to provide network details to make it easier for readers to understand the role of each component in the model. We will add Figure 2 in the attached one-page PDF in Appendix B.1 of the revised manuscript.

> Q6: For the inference pipeline (Fig 3 in the original paper), the authors spent a lot of effort to illustrate the flow-matching training paradigm, which makes this figure a little bit messy.
- Thanks for pointing it out. In Figure 3 of the original paper, we introduced the model input and the sample processes of the ICS-A2M model, respectively. The intention is to highlight two novelty points of our ICS-A2M model: The first is the masked input that concatenates the style prompt with the audio condition, which is the key contribution that enables in-context talking style control; The other is the flow-matching sample process that predicts velocity. Note that we are the first to use flow-matching for modeling audio-to-motion mapping. However, we acknowledge that this diagram is a bit messy. We provide a revised version of this figure to improve clarity. Please refer to **Figure 3 in the attached one-page PDF**.

# Summary
In summary, following the given comments, we have performed several experiments and revised the manuscript from several aspects, which we find has enhanced the soundness and clarity of the paper. Again, we would like to appreciate the reviewer’s valuable review. We sincerely hope the reviewer will reconsider their rating in light of the rebuttal.