MimicTalk: Mimicking a personalized and expressive 3D talking face in minutes
Authors: Zhenhui Ye
Tianyun Zhong
Yi Ren
Ziyue Jiang
Jiawei Huang
Rongjie Huang
Jinglin Liu
Jinzheng He
Chen Zhang
Zehan Wang
Xize Cheng
Xiang Yin
Zhou Zhao
Venue: NeurIPS 2024 poster (2024)
Keywords: Talking Face Generation; Personalization; Talking Style Control
Primary area: machine_vision
TL;DR: We propose MimicTalk, a personalized talking face generation method that can adapted to a target identity in few minutes and mimicking its talking style.

Abstract: Talking face generation (TFG) aims to animate a target identity's face to create realistic talking videos. Personalized TFG is a variant that emphasizes the perceptual identity similarity of the synthesized result (from the perspective of appearance and talking style). While previous works typically solve this problem by learning an individual neural radiance field (NeRF) for each identity to implicitly store its static and dynamic information, we find it inefficient and non-generalized due to the per-identity-per-training framework and the limited training data. To this end, we propose MimicTalk, the first attempt that exploits the rich knowledge from a NeRF-based person-agnostic generic model for improving the efficiency and robustness of personalized TFG. To be specific, (1) we first come up with a person-agnostic 3D TFG model as the base model and propose to adapt it into a specific identity; (2) we propose a static-dynamic-hybrid adaptation pipeline to help the model learn the personalized static appearance and facial dynamic features; (3) To generate the facial motion of the personalized talking style, we propose an in-context stylized audio-to-motion model that mimics the implicit talking style provided in the reference video without information loss by an explicit style representation. The adaptation process to an unseen identity can be performed in 15 minutes, which is 47 times faster than previous person-dependent methods. Experiments show that our MimicTalk surpasses previous baselines regarding video quality, efficiency, and expressiveness. Video samples are available at https://mimictalk.github.io .

MimicTalk: Mimicking a personalized and expressive 3D talking face in minutes
Zhenhui Ye 112
Tianyun Zhong 111.2 Yi Ren
Ziyue Jiang
112
Jiawei Huang i 1,2 Zehan Wang"
Rongjic Huang"
Jinglin liu Jinzheng He
Chen Zhang
Xize Chen 1 Xiang Yin 2 Zhou Zhao *1
'Zhejiang University, 2ByteDance {zhenhuiye, thaozhou] (zj...zjeeeeeee edu.. cn,
{ren.yt yi, yinxiang....... stephen] @bytedanceecccoooo
Abstract
Talking face generation (TFG) aims to animate a target identity's face to create realistic talking videos. Personalized TFG is variant that emphasizes the per- ceptual identity similarity of the synthesized result (from the perspective of ap- pearance and talking style). While previous works typically solve this problem by learning an individual neural radiance field (NeRF) for each identity to implicitly store its static and dynamic information, find it inefficient and non-generalized -er----e--ee pertttiinng framework and the limited training data. To
due to the this end, propose MimicTTalk, the first attempt that exploits the rich knowledge from a NeRF-based person- aagnosi generic model for improving the efficiency and robustness of personalized TFG. To be specific, (I) first come up with person-agnostic 3D TFG model as the base model and propose to adapt into a specific identity; (2) we propose statie-dynamic-hyybrid adaptation pipeline help the model learn the personalized static appearance and facial dynamic fea- tures; (3) To generate the facial motion of the personalized talking style, we pro- pose an in-context stylized audio-to-motion model that mimics the implicit talking style provided in the reference video without information loss by an explicit style representation. The adaptation process to an unseen identity can be performed in minutes, which is 47 times faster than previous person-dependert methods. Ex- periments show that our MimicTTakk surpasses previous baselines regarding video quality, efficiency, and expressiveness. Source code and video samples are avail- able at
cet.ttaa...gtttu....... github.io.
1 Introduction
Audio-driven talking face generation (TFG) (t M12I1 Hong et al, 2022 Than aaze:
Xi (024) is cross-modal task that leverages multi-modal knowledge from speech, vision, and computer graphics to animate a target identity's face given arbitrary driving audio, aiming to create lifelike talking videos interactive avatars. Personalized TFG (Suwajanakom
all 2017 Thies al 20208; Guo et all 2021: Lu all 20216) is a variant of TFG with several real-world applications, such video conferencing and audio-visual chatbots, in which emphasize that the generated results should have excellent perceptual similarity to specific individual (from the perspective of both visual quality and expressiveness). In this setting, video clip of the target identity (from several seconds to minutes) is provided as a detailed reference for the target person's
Equal contribution. Interns at ByteDance Corresponding author.
38th Conference on Neural Information Processing Systems (NcurIPS 2024).

---

personalized attributes, such as geometry shape (Mildenhal et all 2OZI Kerbl et all
2OZE) and
talking style (Wu 202T: aal 2023; Tan all, 2023). To meet the quality requirement of
methods (Tang etall 2022; all 20ZKS: et 2023), which an individual model is trained from scratch for each target person's video. The primary reason for the prevailing of these dleniity-dependen approaches is that during the training process, the person-specific model can implicitly memorize nuanced personalized details of the target person's video, such as their speaking style and subtle expressions, which are hard to represent explicitly with hand-crafted conditions. As a result, these approaches could train small-scale person-dependenn models, achieving high identity similarity and expressive results.
high perceptual identity similarity, the community has focused on identity-dependee
However, the dlentity-dependen methods (Tang et al, 2022: L1 et all 20233) face two well-known challenges: (1) the first weak generalizability. The limited training data scale in the per-identity- per-training restricts the generalizability to out-of-doman
(OOD) conditions during inference. For example, lip-syne performance may degrade when driven by OOD audio (from different language or speaker), and the rendering may fail when synthesizing OOD expressions (such as large yaw). (2) the second is low training and sample efficiency. Since the identity-dependen model needs to be trained from scratch on the target person video and no prior knowledge can be enjoyed, the training process can take more than several hours (low training efficiency), and it often requires over 1-minute length of training data to achieve reasonable lip-sync results (low sample efficiency). By contrast, another line of works, the identity-agnostic (or, say, one-shot) TFG methods (Zhou etall 2020 Wang et al, MO2I: Zhao and Zhang. 2022: et all 20Z30), which incorporate vari- ous identities' video during training and only use one source image during inference, can achieve good generalizability to OOD conditions since various audio/expressions have been seen during the training. However, due to the limited information in the single input image, the one-shot methods cannot exploit the rich samples of the target identity to imitate its personalized attributes. Therefore, it is promising to bridge the gap between person-agnostic one-shot methods and person-dependent small-scale models to achieve expressive, generalized, and efficient personalized TFG.
Based on the observation above, our key motivation is to dig out the prior knowledge of pre-trained person-agnostic TFG models (La 20244 Chu et al, 20ZE; Ye cla 2122)) to better mimic the personalized attributes of target identities. We propose MimicTalle a high-quality and expres- sive personalized TFG framework, which utilizes the generalizability power of large generic TFG model to improve the efficiency and robustness of TFG and devises a talking style-controllable motion generator to obtain results of high expressiveness and personalized characteristics. Specif- ically, (I) our method starts with a 3D person-agnostic generic model. This is due to the finding that the widely used warping-based methods cannot well handle large movements, and state-of-the art person-dependent methods have turned to Neural Radiance Field (NeRF) or Gaussian Splatting, which are advanced neural rendering techniques with strong 3D prior, to achieve video consistency and geometric realism. Inspired by this, we resort to recent NeRF- one-shot TFG method to provide a 3D-aware base model in our framework. (2) Given pre-trained one-shot model, we de- sign a static-dynamic (SD)-hybrid adaptation pipeline for an efficient and stable finetuning process. Specifically, we propose a tri-plane inversion method to learn a personalized 3D face representation to store the static texture details; we also first introduce low-rank adaptation (LoRA) in TFG to adapt the dynamic characteristics of the target speaker, which are difficult to model explicitly. (3) We note that generating motion of personalized talking styles is vital for perceptual reality in audio- driven TFG. Although finetuning the motion generation module on the target individual dataset can allow the network to implicitly memorize the target talking style, it leads to high computation costs and training instability. To address this, we propose an it-eontext-style audio-to-motin (ICS- A2M) model capable of mimicking the in-context zero-shot talking style. The ICS-A2M model is facilitated by a newly proposed training paradigm named audio-guided motion- iitiionnniillinn. which en- courages the model to denoise the partially masked motion track by exploiting the implicit talking style revealed in the unmasked motion track. We also incorporate conditional flo-mmatching into the co-speech motion generation task, achieving state-of-the-a art lip-sync and style- meriikiiiii accuracy.
The contributions of the paper are summarized as follows:
We are the first work that considers utilizing 3D person-agnostic models for personalized TFG. We propose an SD-hybrid pipeline for efficient and high-quality adaptation to learn- ing the static and dynamic characteristies of the target speakers.

---

We propose the ICS-A2M model, which achieves high lip-syne quality and in-context talk- ing style mimicking audio-driven TFG.
Our MimicTalk only requires a few seconds long reference video as the training data and several minutes for training. Experiments show that our MimicTalle surpasses previous person-dependent baselines terms of both expressiveness and video quality while achiev- ing 47x times faster convergence.
2
Related Work
Our work is mainly about personalized and expressive talking face generation. We discuss the related works in the field of talking face generation and expressive co-speech facial motion generation, respectively.
2.1 Talking Face Generation
Based on different real-world applications, talking face generation (TFG) methods have been ma- jorly divided into two settings: identity-agnostic and identit--deenddnt. The identity-agnostic meth- ods focus on the one-shot scenario: the model is provided with only one image and aims to animate it to create video. These approaches hence train a large-scale generic model with large amount of identity video data so that it can generalize to unseen photos during the inference stage. On the other hand,
methods focus on achieving better video quality for a specific speaker: typ- ically using a segment of the target user's video as training data and expecting the model to mimic the personalized features of that speaker, known as personalized TFG. These two lines of work have evolved independently over the years and have developed significantly different methodologies (1) As for the person-agnostic methods, the earliest works (Chung all, 2007; Prajwal et all 202; Wang et all 20222) typically adopt a pixel-to-pixed framework (Isola el all (011) or generative ad- versarial network (GAN) setting generate the result, which results in training instability and bad visual quality. Then, the majority of prior works (Averbuch- et all, 2017; Ren el all, 2021; Wang
202T: Hong et aL, 2072 Zhao and Zhang, 2022: Hong and Xu. 207K% Liang et al,, 4074
Jiang et all, ()244) resort to a dense warping field (Starohin et all 2013) to deform the pixels of the source image given the 3D-aware keypoints extracted from the driving resource. The warping- based method achieves better image fidelity, yet due to a lack of 3D prior knowledge. it occasionally pro- duces warping and distortion artifacts. Recently, to handle these artifacts, some work (Zeng et all 2022: Sun_etal. all 2023; Li all 20233; Ye et all, ((224) propose one-shot NeRF-based methods by learning to reconstruct a 3D face representation (tri-plane) (Chan et all, 2022) from the source image. (2) As for the person-dependent setting (Thies_et el all 20206; Y ci all 2020 Lu all 20218: Ye eal 2022), which trains an individual model on a specific video of the target identity, the recent works (Guo all 2021: Yao et all 2022: [ang et all R022; YY et all 2023 L all 20243) are mainly based on NeRF for its high image fidelity and realistic 3D modeling. Despite the NeRF-based person- sprscffc method achieving the best video quality and identity similarity among all TFG methods, typically requires hours training, and its generalizability to driving conditions is hampered by limited training data. To our knowledge, Mimictalk is the first work that considers bridging the gap between NeRe-based person-agnostic and person-dependent methods to improve the task of personalized TFG.
2.2 Expressive Facial Motion Generation
The recent advance neural rendering has significantly improved the image quality, temporal con- sistency, and stability of the synthesized video, making expressiveness the next focus of the TFG community. The critical challenge in achieving high expressiveness is to generate high-quality facial motion from audio content. This requires the generated motion sequence to not only synchronize with the audio track but also reflect a consistent and expressive talking style similar to the target speaker. Some studies implicitly model this process in a deterministic and end-to-end audio-to- image model (Prajwal et al], 2020; Guo et all, 2021). For better controllability and audio-lip syn- chronization, other works propose explicitly modeling the audio-to-motion mapping using external generative models (Thies et all 20206; Ye et 2023). However, these studies do not explicitly model the speaker's talking style. To address this, Wu all (2021) and Ma et all (2023) developed style vector extracted from arbitrary motion sequences to achieve explicit talking style control.
3

---

:
Person-specifi
Provide tolking style
3D face representation
Expressive results the torget person
Adapted the torget person
Optional: talking style reference
SD- SD-hybrid Adapted Talking Renderer
Provide content
In-Context Stylized Audio-to Mudio-to-Motion
Driving audio
In-context tolking
Stylized expression
style mimicking
Figure I: The inference process of MimicTalk. We use an in-context stylized audio-to-motion model to produce expressive facial motion mimicking the talking style of a reference video. Then, a per- sonalized renderer could render high-quality talking face videos that mimic the static and dynamic visual attributes of the target identity.
EMMN Man all (IZK) disentangles expression styles and lip motion, constructing a memory
bank to produce lip-synced vidcos with vivid expressions. concurrent work, VASA-1 (Euctaal 2044), proposes to add previous audio/motion latent the input of the latent diffusion model to keep can see that most previous methods rely on intermediates to represent the method first achieves in-contett-learning (ICL) talking style control, better preserving the target identity's 's talking style.
temporal consistency.
talking style Diang al. (MO2ZE), risking information loss. In contrast,
MimicTalk
As shown in Fig. II MimicTalk is personalized and expressive 3D talking face generation frame- work, in which the personalized renderer inherits the rich facial knowledge from person-agnostic generic model (discussed in Sec. [Т]I) via static-dynamic (SD)-hybrid adaptation pipeline (in Sec. (.2). In Sec. 6.% we propose an in-context stylized audio-to-motion (ICS-A2M) model to generate personalized facial motion, which necessary achieve high expressiveness in the generated video. We describe the design and training process in the following sections. Due to space limitations, we provide technical details in Appendix E
3.1 NeRF-hased Person-Agnostic Renderer
The previous personalized TFG works trained a separate model on the target speaker to memorize personalized information of the target person, in which NeRF is typically used as the underlying technology to better store the target speaker's geometry, texture, and other information. We aim to leverage the prior knowledge of pre-trained person-agnostic TFG models to achieve greater gener- alizability and efficiency than previous person-dependent methods. To this end, we resort to recent one-shot NeRF-based TFG works (Lietall МлΣКΕ 2024; Ye 2024; Chu et all 20ZE to con- struct a person-agnostic TFG model. Specifically, We build our base model on the basis of Real3D- Portrait (Ye eeeeeall all (OZE)) with its official implementation m, As shown in Fig. E the first step is reconstructing canonical 3D face (formulated as tri-plane representation (Chan all 2022)) from the target person's 's source image Lgc:
Peapo FaceRecon(Lac),
where FaceRecon is an image-to-3D face reconstruction model based on SegFormer (Xie all 2021) that transforms the input image into a tri-plane representation Chan et all (2022). Then liight SegFormer- based motion adapter could control the facial expression in the 3D face, and volume renderer of Mip-NeRF (Barron all 202T) could render the dynamic talking face of arbitrary head pose by controlling the camera:
I VolumeRenderer( cano Mot MotionndaaterrPPPCCC. PNN pN camgg),
where PNCC and PNCC are projected normalized coordinate codes (Lretal 20236), which is rasterized from the 3DMM face (Paysan all 2009) of the source and target image, and camar

---

Prediction by one-shot Image-to-30 Face model
Leornable tri-piane for specific identity
Expression tripplane
Tri-plane converted the target expression
*/c
Porometer frozen
Parameter leammbbl
LoRAs plugged in
LoRAs
Target frame
Expression Tri-plane
Tri-plone rendered
Motion * Adapter
L1 loss
LPIPS loss
Source / target
loss
expression (PNCC)
Learnoble Tri-plane
GAN loss
of sttaretpersor
One-shot */s
initialize
Volume Renderer
Super- * Resolution
Image-to-3D Reconstruction
One-shot
Low-resolution
Predicted frame
The first frame of target person
Predicted
results
Forward once
Tri-plane
Figure 2: The training process the personalized TFG renderer via the static -laticdynamiic -SD)-hybrid adaptation pipeline. We adopt pretrained one-shot person-agnostic 3D TFG model the back- bone, then fine-tune a person-dependent 3D face representation to memorize the static geometry and texture details. We also inject LoRA units into the backbone to learn the personalized dynamic features.
is the driving camera to control the head pose. Law is the volume-rendered low-resolution image, which is further processed by super-resolution module generate the final high-resolution result:
Iped = SuperRessoltionn(Tan
We provide detailed network structure of the FaceRecon, MotionAdapter, VolumeRenderer, and SuperResolution of person-agnostic renderer in Fig. of Appendix (B] For more details about the one-shot person-agnostie base model, please refer to etal 2(24)..
3.2
Hefie-Dynaami-HHybrri Identity Adaptation
With the pre-trained person-agnostic renderer introduced in Sec. B. we could synthesize talking face videos for unseen identities without any adaptation. However, there exists a significant identity similarity gap between the un-tuned renderer and previous person-dependent methods, which is re- flected in two aspects: (1) The static similarity, which measures whether the generated frame has same texture (e.g., wrinkles, teeth, and hair) or geometry details as the target identity; (2) The dy- namic similarity, which describes the relationship between the input motion condition and the facial muscle/torso movement the output facial image. To be more intuitive, trained on the large-scale talking face dataset with various speakers, our person-agnostic model learns a statistically averaged motion-to-ine -t-imagee mapping for face animation, which produces talking person videos that are se- mantically correct yet lack personalized characteristics. By contrast, the previous person-dependent methods, by overfitting the model on one target identity, inherently learn the personalized motion- to-image mapping in the model. Based the observation above, we propose efficient static- dynamic-hyrrid (SD-Hybrid) adaptation pipeline to achieve good static/dynamic identity similarity, which is shown in Fig. DI
Tri-Plane Inversion for Static Similarity. The canonical 3D face representation P Peano, which is extracted from the source image by the pre-trained 3D face reconstruction model ct 2024)), stores all static properties of the target identity (i.e., geometry and texture information). We find that the information loss in this feed-forward image-to-3D transform is the primary reason for the inferior static similarity in our method. To this end, inspired by previous (Roich et all 2021). we propose tri-plane inversion technique, which regards the canonical 3D face representation as learnable parameter and optimizes to maximize static identity similarity. To be specific, shown in Fig. El when adapting our model to specific identity given video clip, we initialize the learnable tri-plane with the first frame's image-to-3D prediction, then optimize the tri tri-plane alongside other parameters.
methods
5

---

Audio Condition [B.T2. T2. Cal
Driving Audio H
ODE Solver
Reference video
Iteratively Denoise
Velocity of Noisy Motion dt
dxx
[B,T2. Cm]
EALO
X
Transformer
Network Input [B,T; + C2.Co Gm]
Ta Talking style prompt [B, T, Ca Cm]
Noisy motion Xy: [B, T2, Cm]
Time Embed.
Figure 3: The process of in-context stylized motion prediction. For the training process please refer to Fig. M
Injecting LoRAs for Dynamic Similarity. As for dynamic similarity, since the person-agnostic model learns averaged motion-to-image mapping in the multi-speaker talking face dataset, we need to adapt the generic model specifically to the target person's video to learn its personalized facial dynamics. A naive solution is directly fine-tune the whole model the last few layers the target person's video. However, considering the large model capacity and the small data scale of the target person's video. it faces several challenges, such high GPU memory footprints, training instability, and catastrophic forgetting. To this end, we resort to low-rank adaptation (LoRA) (EI
et all 202I), which was initially proposed for efficient adaptation of language models and recently extended to computer vision applications such as text-to-image synthesis (Rombach et al, 2021). As shown in Fig. of Appendix BA LoRAs be conveniently plugged into our person-agnostic model by injecting a low-rank learnable matrix to every linear layer and convolution kernel. All pre- trained parameters the person-agnostie model are fixed, and only the LoRAs are updated during training.
Adaptation Process.
As shown in Fig. E. with the SD-hybrid design, the rendering process of the predicted image can be expressed as:
camar) I pred SR (VolumeeennereerPPe.. PoatreeerrrPcttttttoooooAAnadde PNNC PNCC 01), camary; 05); 0g),
(4)
where Piano is the learnable 3D face representation, which is initialized from the first-frame pre- diction by the pre-trained 3D face reconstruction model. SR denotes the super-resolution module, 07, 05 05 are the learnable LoRA parameters injected the volume renderer, motion adapter, and the super-resolution module, respectively.
During adaptation, short video clip of specific identity is utilized as the training data, and only the personalized tri-plane and LoRAs are updated. The training loss of the SD-hybrid adaptation is:
LSD-Hybrid L1 + ALPIPS LLPIP + Anp LID.
(5)
where L1, LIPPPS, LID, denotes LI loss, LPIPS loss by VGG16 (Simonyan and Zisserman, 201 4), and identity loss by VGGFace (Cao all 201S), respectively. We set the learning
0.001,
AlPiPS = 0.2, Åι 0.1. Thanks to the static- mattic-dynamic-hybrit design, our method achieves good identity similarity while enjoying fast and mow-meeory-cost adaptation process compared exist- ing person-deepe deepnneen methods, illustrated in Table MI Besides, we demonstrate that our SD- SD-Hybrid pipeline achieves good training and sample efficiency Fig.
3.3 In-Context Stylized Audio-to-Motion
In the above sections, we have proposed unified framework for motion-conditioned talking face generation. Then, we present in-context stylized audio-to-motion (ICS-A2M) model to generate personalized facial motion for audio-driven scenarios.

---

Audio-guided Motion Filling Task Inspired by the success of in-context learning methods in large language models aLl 2023) and text-to-speech synthesis et 2023). we devise an audio-guided motion infilling task. We visualize the audio-guided motion-infilling task's detailed training and inference pipeline Fig. D of Appendix B. Specifically, the audio-motion pairs are temporally aligned and channel-wise concatenated and processed by the audio-to-motion model. During training, we randomly masked several segments in the motion track and trained the model with the motion reconstruction error on the masked segments. Since the model is provided with surrounding unmasked motion and the complete audio track, it learns to exploit the talking style in the motion context to predict the masked co-speech motion more accurately. During inference, as shown in Fig. G (a), we could concatenate the reference audio-motion pair as the talking style prompt, arbitrary driving audio the condition, and a noisy placeholder for predicted motion the input of the model. This way, the model could predict the audio-synchronized facial motion with the talking style provided in the style prompt.
Audio-to-Motion Flow Matching Model We first introduce an advanced generative model, flow matching (Lipman et all, (023), to generate expressive facial motions. The model is parameterized by transformer A and predicts a flow field ó that stores the velocity V. d: X₁ pointing towards the target distribution. Similar to diffusion models, the time step t is increasing from 0 to during the inference process, i.c., we have prior data points xa (N, 1) and the target data points X1 ~ q(x), where g(x) denotes the ground truth data distribution. Due to space limitation, we provide detailed preliminaries of flow matching in Appendix B3222 To be intuitive, we visualize the forward process of our now-matching-bart based audio-to-motion model in Fig. B (a). The network input is the same as what we defined in the previous paragraph (a concatenation of the style prompt, audio condition, and noisy motion xt). The model's output is the velocity v, of the noisy motion xs. And we could train the model with the conditional flow matching (CFM) objective:
dx, of the data points
CCFM Er..(r)) In (x/x;) vx a, s; 0)
(6)
where (z a, s; i) the predicted velocity by flow matching model of parameter 8, a and s is th audio condition and style prompt, and u, (x(x1) is the ground truth velocity of the optimal transport (OT) path (Le all (023) given the target motion x1 and the noisy motion X,, which is exactly the unit vector pointing from x, to X1.
Apart from the flow matching objective, we find it necessary to add a lip-syne loss on the denoised sample &1 to generate accurate and expressive facial motion. Specifically, we first solve the ODE problem in Eq. 0 to obtain the predicted sample X using the predicted velocity vx a, 8; 0), then feed it into a pre-trained loss Layee that measure the synchronization between the input audio and the predicted expression. Therefore, the training loss of the ICS-A2M model is:
SyncNet (Ye et al 2(ZE) to obtain discriminative sync
LICS-A2M LCFM + Asyne Leyne-
where Asyne = 0.05. During inference, as visualized in Fig. (bb),, now that the A2M model could predict the velocity field of the data point, we could iteratively push the data from the Gaussian dis- tribution to the target distribution by solving the ordinary differential equation in Eq. off Appendix BE3 and finally obtain the predicted motion á; to drive the personalized renderer in Sec. 3.2
Classifier-Free Guidance for Enhancing Style Mimicking Classifier-free guidance (CFG) is popular inference method that manipulates the condition strength during the sampling process of conditional diffusion models. While CFG has become a standard technique in text-to-image Kome bach et all (2021) and other ceak-coondiionned generation tasks, we found it also helps enhance the style mimicking quality of our stylized audio-to-motion model. Specifically, we mix the two network predictions with/without style prompt to construct the CFG velocity vir..
CPG = v(x. + w (v(x, a, s) v(x. a, 0)
(8)
where is the CFG scale and we empirically set w 2. v(x, a, is the network prediction with both audio condition and style prompt as the input, and v(x, a, 0) is the predicted velocity with zero style prompt.
7

---

Table 1: Quantitative results of all methods. Time. and Mem. denote training time and GPU memory
used for adaptation on 1 GPU. StyleTalk a one-shot method, so the Time. and Mem, are
Methods
CSIM
PSNR1
FID
AED
Sync.
Time (h))
Mem. (GB)J
RAD-NeRF (Tang et all (0222)
0.825
30.85
33.51
0.122
4.916
13.22
5,432
GeneFace efel 2025)
0.819
30.44
34.16
0.115
6,480
16.57
7.981
ER-NeRF LLeeaa
0.834
31.45
30.38
0.113
5.631
3.77
8.676
StyleTalk (Mal 2036
0,671
27.39
45.25
0.106
7.173
/
Mimic'Tale (ours)
0.837
31.72
29.94
0.098
8.072
0.26
8.239
Table 2: MOS score of different methods. The error bars are 95% confidence interval.
Methods
RAD-NeRF
GeneFace
ER-NeRF
StyleTall
MimicTTakk
ID. Similarity
3.96::.29
3.78
4.06+:..24
3.27=0.38
4.15:10.20
Visual Quality
3.98=0024
3.87
4.10:t0.22
3.46=10.35
4.22:10.20
Lip Syne.
3.65+0.27
3.93+0.22
3.82+:024
4.01=0022
4.13:1022
Experiment
4.1 Experimental Setup
Implementation Details. We obtain the pre-trained person-agnostic renderer from the official implementation of all (2024). For the SD-Hybrid adaptation, we trained the model on I Nvidia A100 GPU, with a batch size of 1 and total iterations of 2,000, requiring about 8 GB of GPU memory and 0.26 hours. Regarding the (CS-A2M model, we trained it Nvidia A A100 GPUs, with a batch size of 20,000 mel frames per GPU. The fow-matching. based ICS-A2M model was trained for 500,000 iterations, taking 80 hours. We provide full experiment details Appendix
Data Preparation. To evaluate the personalized renderers, tested ten 3-minute-long target person videos by (Tang et al, 2022) and ((Ye ct all, 2023). To train the ICS-A2M model, we use a large-scale lip-reading dataset, voxceleb2 (Chung et all 20T8), which consists of about 2,000 hours videos from 6, 12 celebrities.
Compared Baselines. We compare our method with three person-dependent methods: (1) RAD- NeRF (Tang et all 2022), (2) GeneFace ell m023), and (3) ER-NeRF (Lretal (1244) We also compare with style-oriented TFG method that considers controlling the talking style, (4) StyleTalk (Ma al. 2023). We discuss the characteristics of all test methods in Appendix AT
4.2 Quantitative Evaluation
We use CSIM measure identity preservation, PSNR, FID to measure the image quality, and AED (Deng et all 2019) and SyncNet confidence (Chung and Zissermary 2017) to measure audio-lip syn- chronization. The results are shown in Table M. We have the following observations: (1) Thanks to the powerful flow matching model and the wwcttext--tyle--miieee the best lip accuracy (AED) and perceptual lip-syne quality (SyneNet confidence); (2) Our SD- hybrid adaptive renderer shows better visual quality than the person-specific baselines. (3) Thanks to the efficiency of the LoR A-based adaptation process, our method requires significantly less training time to adapt to new identity within 2,000 iterations and 15 minutes (47x faster than RAD-NeRF). also requires a low GPU memory usage (8.239 GB) for adaptation.
ability, our method achieves
4.3
Qualitative Evaluation
4.3.1
Case Study
We provide demo videos https://mimiictalkk..ibtt git.l...itttuub.. 1o. We also adopted several case stud- ies to demonstrate better performance. Specifically, (1) our SD-Hybrid adaptation has better train-
We release the source code https://iimictalk github.

---

CSIM
CSIM
3.88
0.50
3.86
3.0
20 56 100200500 1k dk ék 10k
15s
180s
Adebtation Steds
Trainina Data Scales
CSIM results at different iterations
(b) CSIM results different data scales
Figure 4: Training/data efficiency of SD-Hybrid adaptation: CSIM results at different iterations and data scales. The baseline RAD-NeRF uses 180-second-long training samples and is updated for 250,000 iterations.
Table 3: CMOS results on the style controllability and identity similarity of MimicTalk and StyleTalkk, CMOS score ranges from -3 to +3. Error bars are 95% confidence intervals.
Methods
CMOS-style-comtrolf
CMOS-identiit--imilarrgg
#1. MimicTalk (ours)
0.549 + 0.225
1.735 0.362
#2. StyleTalk (Maetal 20233
0.000
0.000
ing/sample efficiency than previous person-specity methods; (2) our ICS-A2M model predicts styl- ized facial motion.
Training/ Sample Efficiency of SD-Hybrid Adaptation To evaluate the training and sample effi- ciency of our SD-hybrid adaptation, we perform a case study of a talking video of President Obama provided by [ang et al, (2022). For training efficiency, as shown in Fig. Ha)), adapt the model a 180-second-long clip as the training data and use the lasting 10-second clip as the validation set. Our SD-hybrid adaptation enjoys fast convergence and good performance compared the person- specific baseline. As for the sample efficiency, we visualize the results of CSIM at different scales of training data in Fig. ().
be observed that the CSIM score improves as the amount of training data increases. Additionally, our method achieves comparable performance to the person- person-specific baseline, which was trained with 180 seconds of data, using only mnethhir of the data (60 seconds).
Talking Style-Coherent Motion Prediction by ICS-A2M Model Given short reference video as the talking style prompt, our ICS-A2M model could accurately mimic its talking style (such as smiling duck mouth). We provide a demo video httpa://mimictaat gi trithubb.o. static/ videos//ded /emo ics a2m. a2m.mp4 for better demonstration. We also conducted comparative mean score opinion (CMOS) test between our method and StyleTalk (Ma (I23) to qualitatively evaluate talking style accuracy. As shown in Table B. our method achieves better talking style controllability and identity similarity. Please refer Appendix C< for detailed user study settings.
4.3.2 User study
We conducted the Mean Opinion Score (MOS) test evaluate the perceptual quality of generated samples, scaled from 1 to 5, Following Chen el all (2020), the attendees are required to rate the
Table Ablation studies on different settings in the SD-hybrid adptation.
Settings
CSIM
PSNR
FIDJ
AED
APDJ
#1. Ours Hyb-id
0.837
31.72
29.94
0.098
0.028
#2. Ours Tri fti-plane Inv.
0.823
30.65
33.28
0.102
0.030
#3. Ours LoRAs
0.805
29.95
36.56
0.108
0.033

---

Table 5: Ablation studies on different settings in ICS-A2M. L.2.ammmrrk denotes the L2 reconstruction error on the 68 3D landmarks, and Lyyns denotes eepresexppsssion synchronization contrastive loss provided by (Chung and Zisserman, 7) and eYe all, 2023)
Settings
L2d
Lsyne
#1. Ours (ICS-A2M)
0.026
0.423
42. Ours w.o. Flow Matching
0.034
0.466
#3. Ours w. style vec. (Huang and Belongie, 2017)
0.031
0.429
#4. Ours w. style (Ma 2023)
0.029
0.428
#5. Ours w.o. syne
0.028
0.536
videos from three aspects: (1) identity similarity. (2) visual quality, and (3) lip-sync. Detailed settings are in Appendix ox The results are shown in Table D. We have the following observa- tions: (I) our method achieves the best lip-synchronization and identity similarity visual quality compared to SOTA person-dependent methods (RAD-NeRF, GeneFace, and ER-NeRF). The MOS results demonstrate the effectiveness of the proposed MimidTalk framework.
4.3.3 Ablation study
SD-Hybrid Adaptation. We tested two settings in the SD-Hybrid adaptation: (1) Not performing tri-plane inversion to finetune a personalized tri-plane and (2) not injecting LoRAs into the model. As shown in line 2 and line 3 of Table EL utilizing both techniques achieves the best identity similarity (CSIM), visual quality (FID), and geometry accuracy (AED and APD).
ICS-A2M Model. We also analyze three settings in the ICS-A2M, (1) replace the flow matching model with a deterministic transformer, as shown in line 2 of Table EL which leads to worse motion reconstruction quality and worse sync score; (2) replace the in-context style control with hand- crafted style vector in ell ((2lI) or learn a style encoder as in StyleTaak etll (2025), as shown in line 3 and line 4, which leads to worse motion reconstruction quality, proving that the ICL talking style mimicking could prevent information loss caused by compressing the style into a global encoding; (3) remove the sync loss during training, as shown in line 5, which leads to significantly worse perceptual lip-sync performance.
5 Conclusion
In this paper, we propose MimicTalk,, an efficient and expressive personalized talking face generation framework. We first come up with the idea of adapting pre-trained 3D person-agnostic model to personalized datasets to inherit its generalizability and achieve fast training. The SD-Hi HD-Hybrid adaptation pipeline helps the generic model learn the target person's 's static and dynamic features, leading to better identity similarity than previous person-dependent baselines. Besides, the proposed ICS-A2M model is the first facial motion generator that enables in-context talking style control, which helps produce expressive facial motion in the generated video. Due to space limitations, we provide impact statements in Sec. E and discuss limitations and future works Appendix D.
6 Acknowledgments
This work was supported in part by the National Natural Science Foundation of China under Grant No. 62222211 and National Natural Science Foundation of China under Grant No.62072397.
References
Hadar Averbuch-Elor, Daniel Cohen-Or, Johannes Kopf, and Michael F Cohen. 2017. Bringing portraits life. ACM transactions on graphics (TOG).
Jonathan T Barron, Ben Mildenhall, Matthew Tancik, Peter Hedman, Ricardo Martin-Bruall and Pratul P Srinivasan. 2021. Mip-nerf: multiscale representation for anti-aliasing neural radiance fields. In ICCV, pages 5855-5864.
10

---

Qiong Cao, Li Shen, Weidi Xie, Omkar M Parkhi, and Andrew Zisserman. 2018. Vggface2: A dataset for recognising faces across pose and age. In 2018 13th IEEE international conference on automatic face & gesture recognition ( (FG 2018), pages 67-74. IEEF.
Eric R. Chan, Connor Z. Lin, Matthew A. Chan, Koki Nagano, Boxiao Pan, Shalini De Mello, Orazio Gallo, Leonidas Guibas, Jonathan Tremblay, Sameh Khamis, Tero Karras, and Gordon Wetzstein. 2022. Efficient geometry-aware 3D generative adversarial networks. In CVPR.
Lele Chen, Guofeng Cui, Ziyi Kou, Haitian Zheng, and Chenliang Xu. 2020. What comprises good talking-head video generation?: A survey and benchmark. arXiv preprint w0Xiv:2005.0320..
Ricky TQ Chen, Yulia Rubanova, Jesse Bettencourt, and David K Duvenaud. 2018. Neural ordinary differential equations.
Xuangeng Chu, Yu Li, Ailing Zeng. Tianyu Yang. Lijian Lin, Yunfei Liu, and Tatsuya Harada. 2024.
Gpavatar: Generalizable and precise head avatar from image(s).
Joon Son Chung. Amir Jamaludin, and Andrew Zisserman. 2017. You said that? arXiv preprint arXiv:1705.02966.
Joon Son Chung, Arsha Nagrani, and Andrew Zisserman. 2018. Voxceleb2: Deep speaker recogni- tion. arXiv preprint arXiv: arXiv:/80..000 056205622.
Joon Son Chung and Andrew Zisserman. 2017. Out of time: automated lip syne in the wild. In ACCV 2016 International Workshops, pages 251-263. Springer.
Yu Deng, Jiaolong Yang, Sicheng Xu, Dong Chen, Yunde Jia, and Xin Tong. 2019. Accurate 3d face reconstruction with weakly-supervised learning: From single image to image set. In IEEE Computer Vision and Pattern Recognition Workshops.
Yudong Guo, Keyu Chen, Sen Liang, Yong-Jin Liu, Hujun Bao, and Juyong Zhang. 2021, Ad-nerf: Audio driven neural radiance fields for talking head synthesis. In ICCV.
Jonathan Ho, Ajay Jain, and Pieter Abbeel. 2020. Denoising diffusson probabilistic models.
Fa- Ta-Ting Hong and Dan Xu, 2023. Implicit identity representation conditioned memory compensa- tion network for talking head video generation. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pages 23062-23072.
Fa-Ting Hong, Longhao Zhang. Li Shen, and Dan Xu. 2022. Depth-aware generative adversarial network for talking head video generation. In CVPR.
Edward J Hu, Yelong Shen, Phillip Wallis, Zeyuan Allen-Zhu, Yuanzhi Li, Shean Wang. Lu Wang, and Weizhu Chen. 2021. Lora: Low-rank adaptation of large language models. arXiv preprint 11Xi::200600685..
Li Hu, Xin Gao, Peng Zhang, Ke Sun, Bang Zhang, and Liefeng Bo. 2023. Animate anyone: Consistent and controllable image-to-video synthesis for character animation. arXiv preprint arXiv:2311.17117
Xun Huang and Serge Belongie. 2017. Arbitrary style transfer in real-time with adaptive instance normalization. In ICCV, pages 1501-1510.
Phillip Isola, Jun-Yan Zhu, Tinghui Zhou, and Alexei A Efros. 2017. Image-to-image translation with conditional adversarial networks. In CVPR.
Jianwen Jiang, Chao Liang, Jiaqi Yang, Gaojie Lin, Tianyun Zhong, and Yanbo Zheng. 2024a. Loopy: Taming audio-driven portrait avatar with long-term motion dependency. arXiv preprint arx1v:2409.02634.
Jianwen Jiang, Gaojie Lin, Zhengkun Rong Chao Liang, Yongming Zhu, Jiaqi Yang. and Tianyun Zhong. 2024b. Mobileportrait: Real-time one-shot neural head avatars on mobile devices. arXiv preprint ar 0r71:2240..0511.
11

---

Bernhard Kerbl, Georgios Kopanas, Thomas Leimkühler, and George Drettakis. 2023. 3d gaussian splatting for real-time radiance field rendering. ACM Transactions on Graphies, 42(4):1-14.
Matthew Le, Apoorv Vyas, Bowen Shi, Brian Karrer, Leda Sari, Rashel Moritz, Mary Williamson, Vimal Manohar, Yossi Adi, Jay Mahadeokar, al. 2023. Voicebox: Text-guided multilingual universal speech generation at scale. arXiv preprint arXiv:2306.
Jiahe Li, Jiawei Zhang, Xiao Bai, Jun Zhou, and Lin Gu. 2023a. Efficient region-aware neural radiance fields for high-fidelity talking portrait synthesis. In ICCV.
Weichuang Li, Longhao Zhang, Dong Wang. Bin Zhao, Zhigang Wang. Mulin Chen, Bang Zhang, Zhongjian Wang, Liefeng Bo, and Xuelong Li. 2023b. One-shot high-fidelity talking-head syn- thesis with deformable neural radiance field. In CVPR.
Xueting Li, Shalini De Mello, Sifei Liu, Koki Nagano, Umar Iqhal, and Jan Kautz. 2023c. General- izable one-shot neural head avatar. arXiv preprint arXiv:2306.08760
Xueting Li, Shalini De Mello, Sifei Liu, Koki Nagano, Umar Iqbal, and Jan Kautz. 2024. General- izable one-shot 3d neural head avatar. NeurlPS, 36.
Chao Liang, Jianwen Jiang, Tianyun Zhong, Gaojie Lin, Zhengkun Rong, Jiaqi Yang, and Yongming Zhu. 2024. Superior and pragmatic talking face generation with teacher-student framework. arXiv preprint 4rX
Yaron Lipman, Ricky TQ Chen, Heli Ben-Hamu, Maximilian Nickel, and Matt Le. 2023. Flow matching for generative modeling. In ICLR.
Yiheng Liu, Tianle Han, Siyuan Ma. Jiayue Zhang, Yuanyuan Yang. Jiaming Tian, Hao He, An- tong Li, Mengshen He, Zhengliang Liu, et al. 2023, Summary of chatgpt-related research and perspective towards the future of large language models. Meta-Radioloye page 100017.
Yuanxun Lu, Jinxiang Chai, and Xun Cao. 2021a. Live speech portraits: Real-time photoistic talking- head animation. ACM Transactions on Graphics.
Yuanxun Lu, Jinxiang Chai, and Xun Cao. 2021b. Live speech portraits: real-time photorealistic talking-heen head animation. ACM Transactions on Graphics (TOG), 40(6):1-17.
Yifeng Ma, Suzhen Wang, Zhipeng Hu, Changjie Fan, Tangjie Lv, Yu Ding, Zhidong Deng, and Xin Yu. 2023. Styletalk: One-shot talking head generation with controllable speaking styles. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 37, pages 1896-1904.
Ben Mildenhall, Pratul P Srinivasan, Matthew Tancik, Jonathan T Barron, Ravi Ramamoorthi, and Ren Ng. 2021. Nerf: Representing scenes neural radiance fields for view synthesis. Communi- cations of the ACM, 65(1)999-106.
P. Paysan, R. Knothe, B. Amberg, S. Romdhani, and T. Vetter. 2009. A 3d face model for pose and illumination invariant face recognition. Proceedings of the 6th IEEE International Conference on Advanced Video and Signal based Surveillance (AVSS) for Security, Safety and Monitoring in Smart Environments.
KR Prajwal, Rudrabha Mukhopadhyay, Vinay P Namboodiri, and CV Jawahar. 2020. A lip sync expert all you need for speech lip generation in the wild. In ACM MM.
Yurui Ren, Ge Li, Yuanqi Chen, Thomas H Li, and Shan Liu, 2021. Pirenderer: Controllable portrait image generation via semantic neural rendering. In ICCV.
Daniel Roich, Ron Mokady, Amit H Bermano, and Daniel Cohen-Or. 2021. Pivotal tuning for latent-based editing of real images. ACM Trans. Graph.
Robin Rombach, Andreas Blattmann, Dominik Lorenz, Patrick Esser, and Bjöm Ommer. 2021. High-resolution image synthesis with latent diffusion models,
Aliaksandr Siarohin, Stéphane Lathuilière, Sergey Tulyakov, Elisa Ricci, and Nieu Sebe. 2019. First order motion model for image animation.
12

---

Karen Simonyan and Andrew Zisserman. 2014. Very deep convolutional networks for large-scale image recognition. arXiv preprint arXiv:1409.1553
Yang Song. Jascha Sohl-Dickstein, Diederik P Kingma, Abhishek Kumar, Stefano Ermon, and Ben Poole. 2021. Score-based generative modeling through stochastic differential equations. In ICLR.
Jingxiang Sun, Xuan Wang, Lizhen Wang, Xiaoyu Li, Yong Zhang, Hongwen Zhang, and Yebin Liu. 2023. Next3d: Generative neural texture rasterization for 3d-aware head avatars. In Proceedings of the IEEE/CVF Conference Computer Vision and Pattern Recognition, pages 20991-21002.
Supasorn Suwajanakorn, Steven M Seitz, and Ira Saaeelmacchr-Shlizemene 2017. Synthesizing obama: learning lip syne from audio. ACM Transactions on Graphics (ToG), 36(4):1-13.
Shuai Tan, Bin Ji, and Ye Pan. 2023. Emmn: Emotional motion memory network for audio-driven emotional talking face generation. In ICCV, pages 22146-22155
Jiaxiang Tang, Kaisiyuan Wang, Hang Zhou, Xiaokang Chen, Dongliang He, Tianshu Hu, Jingtuo Liu, Gang Zeng, and Jingdong Wang. 2022. Real-time neural radiance talking portrait synthesis via audio-spatial decomposition. arXiv preprint arXiv:22 1.12368.
Justus Thies, Mohamed Elgharib, Ayush Tewari, Christian Theobalt, and Matthias Nießner. 2020a. Neural voice puppetry: Audio-driven facial reenactment. In Computer Aision-ECCY 2020: 16th European Conference, Glasgow, UK, August 23-28, 2020, Proceedings, Part XVI 16, pages 716- 731. Springer.
Justus Thies, Mohamed Elgharib, Ayush Tewari, Christian Theobalt, and Matthias Nießner. 2020b. Neural voice puppetry: Audio-driven facial reenactment. In ECCV 2020.
Linrui Tian, Qi Wang, Bang Zhang, and Liefeng Bo. 2024. Emo: Emote portrait dlive-generating ex- pressive portrait videos with audio2video diffusion model under weak conditions. arXiv preprint arXiv:2402. 17485.
Ting-Chun Wang, Arun Mallya, and Ming- Yu Liu. 2021. One-shot free-view neural talking-head synthesis for video conferencing. In CVPR.
Yaohui Wang, Di Yang, Francois Bremond, and Antitza Dantcheva. 2022. Latent image animator: Learning to animate images via latent space navigation. arXiv preprint arXiv:2203. 090443.
Haozhe Wu, Jia Jia, Haoyu Wang, Yishun Dou, Chao Duan, and Qingshan Deng. 2021. Imitating arbitrary talking style for realistic audio-driven talking face synthesis. In ACM MM.
Enze Xie, Wenhai Wang, Zhiding Yu, Anima Anandkumar, Jose M Alvarez, and Ping Luo. 2021. Segformer: Simple and efficient design for semantic segmentation with transformers. In NIPS.
Sicheng Xu, Guojun Chen, Yu-Xiao Guo, Jiaolong Yang, Chong Li, Zhenyu Zang, Yizhong Zhang, Xin Tong, and Baining Guo. 2024. Vasa-l: Lifelike audio-driven talking faces generated in real time. arXiv preprini arXiv:2404 2066677704.0667.
Yuelang Xu, Benwang Chen, Zhe Li, Hongwen Zhang, Lizhen Wang, Zerong Zheng, and Yebin Liu. 2023. Gaussian head avatar: Ultra high-fidelity head avatar via dynamic gaussians. arXiv preprint arXiv:2312.03029,
Shunyu Yao, RuiZhe Zhong, Yichao Yan, Guangtao Zhai and Xiaokang Yang. 2022. Dfa-nerf: Personalized talking head generation via disentangled face attributes neural rendering. arXiv preprint arXiv:2201.000 207991....
Zhenhui Ye, Ziyue Jiang, Yi Ren, Jinglin Liu, JinZheng He, and Zhou Zhao. 2023. Geneface: Generalized and high-fidelity audio-driven 3d talking face synthesis. In ICLR.
Zhenhui Ye, Tianyun Zhong, Yi Ren, Jiaqi Yang, Weichuang Li, Jiangwei Huang, Ziyue Jiang, Jinzheng He, Rongjie Huang, Jinglin Liu, Chen Zhang, Xiang Yin, Zejun Ma, and Zhou Zhao. 2024. Real3d-portrait, One-shot realistic 3d talking portrait synthesis.
13

---

Zipeng Ye, Mengfei Xia, Ran Yi, Juyong Zhang, Yu-Kun Lai, Xuwei Huang, Guoxin Zhang, and Yong-jin Liu. 2022. Audio-driven talking face video generation with dynamic convolution kernels. IEEF Transactions on Multimedia.
Ran Yi, Zipeng Ye, Juyong Zhang, Hujun Bao, and Yong-Jin Liu. 2020. Audio-driven talking face video generation with learning-basee bbased personalized head pose. arXiv preprint arXiv:2002.10133
Bohan Zeng, Boyu Liu, Hong Li, Xuhui Liu, Jianzhuang Liu, Dapeng Chen, Wei Peng, and Baochang Zhang. 2022. Fnevr: Neural volume rendering for face animation. In NIPS.
Wenxuan Zhang, Xiaodong Cun, Xuan Wang. Yong Zhang, Xi Shen, Yu Guo, Ying Shan, and Fei Wang. 2023. Sadtalker: Learning realistic 3d motion coefficients for stylized audio-driven single image talking face animation. In CVPR.
Jian Zhao and Hui Zhang. 2022. Thin-plate spline motion model for image animation. In CVPR.
Yang Zhou, Xintong Han, Eli Shechtman, Jose Echevarria, Evangelos Kalogerakis, and Dingzeyu Li. 2020. Makelttalk: speaker-aware talking-head animation. ACM Transactions On Graphics
1-5)
14

---

Eipression Th srat
Output feature map
rendered Image
Super-essoltioon image
Split
LayrerNorm Unnea
by
um Rendering
st
Upsample2X
Layerfoom Attenton
ReLU)
RelU)
ConviD Leoky ReeL
Interpolate Concal
Leaky ReLU
6: Sample
Coarse line sampling
Ray Sumpler
coreect
Duerfop Patchimbed
Upsample 2X
Upsample 2X
SegFormer
Tri-plane converted target expression
Head pose
Input map
Source / PACE
Volume rendered 128
pizw si o +3
Leamable
Triplane
(a) SegFormer-bsedd Motion dapter
SegFormer Block Motion Adaptor
(c) Volume Renderer
Super Resolution Module
Figure The detailed network structure of the person-agnoste renderer.
A Comparison between Different Methods
As for the drrson-dependent talking face renderer, all previous methods like RAD-NeRF (Tang etall 2022), GeneFace (Ye 2023), and ER-NeRF (Lletall 2025) follow per-identity -per- training paradigm, which means they have to learn a model from scratch for an unseen identity, which time- time-consuming and the generalizability of the model is limited due to the small scale of training data. By contrast, our method is the first work to adapt a pre-trained generic one-shot 3D talking face model into the target identity. Thanks using LoRA and the tri-plane inversion, our method could converge in 2,000 iterations (47 times faster than RAD-NeRF) and enjoy better generalizability, sample efficiency, and training efficiency due to the generic backbone.
As for the audio-to-motion model in the audio-driven TFG, most previous works like SadTalker (Zhang et al, 2023) and StyleTalk (Ma al 2023) utilize a deterministic mapping to model the Audio-to-motion transform, and cannot achieve talking style control. By contrast, our method first introduces flow-mate Now-matching for the audio-to-mon -omooioo task and achieves in-context talking style control.
B
Model Details
B.1
Network Structure of the Person-Agnostic Renderer
We provide detailed network structure of the person-agnostic renderer in Figure 5
B.2 Details on LoRAs for SD-Hybrid Adaptation
visualize a detailed process that plugs the LoRA into our person-agnostic renderer Fig. For each pretrained kernel or weight of shape Cin. X Cout, the LoRA Matrix A and LoRA Matrix B could represent a weight matrix of shape Can X Cont and rank r, where r << min(Cin: Cout) and we set 4 in our setting. During training, the pre-trat pre-trained weights of the backbone are fixed, and only the LoRA matrices are updated.
B.3
Details on In-Context Stylized Audio-to-Motion Model
B.3.1
Audio-Guided Motion Infilling Task
In Fig. we present a visualization of the training and inference process for the Audio-Guided Motion Infilling task. Paired audio-motion samples are required for this task, which can be easily extracted from talking face datasets and used as training data. During training, we randomly mask several segments in the motion track and encourage the model to reconstruct them based on the complete audio track and the unmasked motion context. This training approach enables the model to learn to mimic the talking style provided in the context.
15

---

Conv Layer
A
weights
Layer
Cout
LaRA Matrix
A
Pretrained Kernels Weights
+
r
LoRA Matrix
IR
XCout
Ci
weights
Adapted weights
Figure 6: The process that plugs LoRAs into convolutional/incar layers of the person aggostic renderer.
Training Process: Denoising Masked Motion
Inference Usagel: Audio2Mation with In-context talking style
Inference Usage2: Audio-only Sampling
Audio (HUBERT)
Raw Data
Driving Audio
Driving Audio
Motion (3DMM)
Aligned audio-motion data
Audio Audio-Motion Motion Pairs
Condition
Condition
Model Input
Unmasked audio with randomly masked motion
Talking style prompt Nolsy placeholder predicted motion
Noisy placeholder predicted motion
Sampled motion #1
Model Output
Reconstructed motion
Stylized predicted motion
Sampled motion #2
Figure 7: The training process and inference usage of the Audio-Guided Motion Infilling Task.
During inference, there are two main usage scenarios. Firstly, we can guide the model to mimic specific talking style by providing audio-motion pair of the target person as the talking style prompt. This setting allows for in-context talking style control over the generated facial motions. Alternatively, also support audio-only motion sampling. Without style prompt, the model will generate semantically correct facial motions with a randomly sampled talking style.
B.3.2 Preliminaries on Flow Matching
In this section, we introduce preliminaries of flow matching. Conditional flow matching (CFM) variant of continuous normalizing flows (Chen et all 2018). which is a class of generative models for modeling an unknown data distribution q(x), where æ denotes the data point the distribution. The CFM method aims to predict a time-dependent flow field that represents the velocity u, of of the data point æ; at continuous timestep I, where the data point æg p(x)) starts from simple prior distribution p(x) (e.g., Gaussian distribution) at the timestep f. 0. and pushed towards the target distribution q(x) by the velocity field the timesteps finally approach 1. This process
drs
16

---

can be formulated as an ordinary differential equation (ODE) as follows:
dxt ur (21, 0), 8.L.00 ~ p(r), e [0, di
where uy is the ground truth velocity for zy and is expected to be approximated by the CFM model. Ur is the predicted velocity by the CFM model with the parameter 0. Once the uy is obtained, we can train the model with simple flow-matching objective:
CCFM =
Ee|lu
ve (2 0)1
(10)
Once the training is done, obtain uy (Ire; 0) dus and solve the ODE in Eq. B obtain the predicted sample 211
Now, consider the specific formulation of the ground truth velocity uy. We adopt the optimal transportation (OT) path proposed in (Lipman et al, 2023) to obtain the ground truth velocity. Specif- ically, given the target data point 1 and the current data point Zt, the most efficient way is to go with a straight line, i.e., Ug = (x1 21 t), where (1 - t) is a normalization term to make the ground truth velocity a unit vector. We choose the OT path for its simplicity and prior evidence (Lipman et all 2023; Le et all 2023) that it outperforms other alternatives (such as diffusion paths
(Ho et all 2020: Song al. 202T)).
C Experiment Details
In the following sections, we illustrate the model configuration and training details of our Mim- TaTk..
C.1 Model Configuration
We provide detailed hyper-paramede settings about the model configuration in Table G
Table Model Configuration
pepeer-prraeter
Value
LoRA rank
2
SD -D-HHbrid Adaptation
Learnable Tri-plane shape
256
32 x
Transtormer Attention Hidden Size
1024
Transformer Norm Type
LayerNorm
Transformer Attention Layers
16
Transformer MLPP Layers per Block
Transformer Altention Head Hidden Size
64
ICS-A2M Model
Transformer Attention Heads
8
Flow Matching Final sigma
0.
Flow Matching ODE method
midpoint 5
Flow Matching- ODE infer steps
C.2 Training Details
We use the pre-trained one-shot person person-agnoste -gnoosic renderer provided in the official implementation of (Ye et all 2024).
For the SD-Hybrid adaptation, trained the model 1 Nvidia A100 GPU, with batch size of 1, requiring about 8 GB of GPU memory. Surprisingly, our method achieved better results than existing person-specific baselines in just 2,000 iterations, which took about 0.26 hours and was 47 times faster than RAD-NeRF.
Regarding the ICS-A2M model, we trained it on 4 Nvidia A100 GPUs, with batch size of 20,000 mel frames per GPU. The www-matching-bast based ICS-A2M model was trained for 500,000 iterations, taking 80 hours.
17

---

C.3 Detailed User Study Sefting
As for mean score opinion (MOS) test in Table 2 we select 5 audio clips and 10 trained identities (as used in by Tang et all (2022) and et all (2023)) to construct 50 talking portrait video samples for each method. Each video has been rated by 20 participants. We perform the identity similarity, visual quality, and lip-synchronization MOS evaluations. For MOS, each tester is asked to evaluate the subjective score of a video on 1-5 Likert scale. For identity similarity, we tell the participants to "only focus on the similarity between the identity in the source image and the video"; ; for visual quality, we tell the participants to "focus on the overall visual quality, including the image fidelity and smooth transition between adjacent frames"; as for lip synchronization, we tell the participants to "only focus on the semantic-level audio-lip lip synchronization, and ignores the visual quality".
As for comparative mean score (CMOS) test in Table E, we first trained 10 person cesson--pecific renderers on 10 identities' each renderer. So there are 50 result videos for each setting. We include 20 participants in the user study. Each tester is asked to evaluate the subjective score two paired videos on a -3 +3 Likert scale(e.g., the first video is constantly 0.0, and the second video is +3 means the tester strongly prefers the second video). To examine the aspect of (lip-syne, pose-syne, expressiveness), we tell the participants to focus on the (lip-sync, pose-sync, expressiveness), and ignore the other two factors.". ".
second-long videos. We randomly select 5 out-of-domain audio clips for driving
D Limitations and Future Work
In this section, we diseuss the limitations of the proposed method and how we plan handle them in future work. Firstly, in this paper, our main focus is on the face segment. By contrast, the rigid modeling of the hair and torso segment relatively naive and occasionally produce artifacts. We plan to adopt conditional video diffusion models like eHu 20233 to enhance the naturalness of the hair and torso segments. Secondly, we consider more conditions, such as eyeball movement and hand gestures. Finally, the inference speed (15 FPS on 1 A100) can be improved by introducing more efficient network structures like Gaussian Splatting.
E Broader Impacts
In this section, we discuss the ethical impacts that might be brought by the rapidly developing talking face generation technology and our measures to address these concerns.
MimicTalk facilitates efficient and expressive personalized talking face synthesis. With the devel- opment of talking face generation techniques, it is much easier synthesize talking human portrait videos. Under appropriate usage, this technique could facilitate real-world applications like virtual idols and customer service, improving the user experience and making human life more convenient. However, the talking face generation method can be misused in seepaake-rllttd usages, raising ethical concerns. We are highly motivated to handle these misusage problems. To this end, we plan to include several restrictions in the license of MimicTalk. Specifically,
We will add visible watermarks to the video synthesized by MimicTalk so that the public can easily tell the fakeness of the synthesized video.
The synthesized videos should only be used in educational or other legal usages (like online courses), and any abuse will take responsibility by tracking the method we come up with in the next point.
We will also inject an invisible watermark into the synthesized video to store the informa- tion of the video maker that the video maker has account for the potential risk raised by the synthesized video.
18

---

NeurIPS Paper Checklist
The checklist designed to encourage best practices for responsible machine learning research, ad- dressing issues of reproducibility, transparency. research ethics, and societal impact. Do not remove the checklist: The papers not including the checklist will be desk rejected. The checklist should follow the references and follow the (optional) supplemental material. The checklist does NOT count towards the page limit.
Please read the checklist guidelines carefully for information on how to answer these questions. For each question in the checklist:
You should answer [Yes], [No] or [NA].
[NA] means either that the question is Not Applicable for that particular paper or the relevant information is Not Available.
Please provide short (12 sentence) justification right after your answer (even for NA).
The checklist answers are an integral part of your paper submission. They are visible to the reviewers, area chairs, senior area chairs, and ethics reviewers. You will be asked to also include (after eventual revisions) with the final version of your paper, and its final version will be published with the paper.
The reviewers of your paper will be asked to use the checklist as one of the factors in their evalu- is perfectly acceptable answer "[No] provided proper justification is given (e.g.. "error bars are not reported because it would be too computationally expensive" or "we were unable to find the license for the dataset we used"). In general, answering "[No] or "[NA] is not grounds for rejection. While the questions are phrased binary way, we acknowledge that the true amswer is often more nuanced, so please just use your best judgment and write justification to elaborate. All supporting evidence can appear either in the main paper or the supplemental material, provided in appendix. If you answer [Yes] to question, in the justification please point to the section(s) where related material for the question can be found.
ation. While "[Yes] generally preferable to "[No]
IMPORTANT, please:
Delete this instruction block, but keep the section heading "NeurIPS paper checklist",
Keep the checklist subsection headings, questions/answers and guidelines below.
Do not modify the questions and only the provided macros for your answers.
1. Claims
Question: Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?
Answer: [Yes]
Justification: Our contributions include: (1) first propose to exploit generic model for per- sonalized 3D talking face generation; (2) SD-hybrid adaptation pipleine for fast and ef- fective identity adaptation; (3) ICS-A2M model for expressive motion generation.
Guidelines:
The answer NA means that the abstract and introduction do not include the claims made the paper.
The abstract and/or introduction should clearly state the claims made, including the contributions made in the paper and important assumptions and limitations. A No
NA answer to this question will not perceived well by the reviewers.
The claims made should match theoretical and experimental results, and reflect how much the results
be expected to generalize other settings.
fine to include aspirational goals motivation as long it is clear that these goals are not attained by the paper.
2. Limitations
Question: Does the paper discuss the limitations of the work performed by the authors? Answer: [Yes]
19

---

Justification:
See Appendix D.
Guidelines:
The answer NA means that the paper has no limitation while the answer No means that the paper has limitations, but those are not discussed in the paper.
The authors are encouraged to create separate "Limitations" section in their paper.
The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (e.g., independence assumptions, noiseless settings, model well-specification, asymptotic approximations only holding locally). The au- thors should reflect on how these assumptions might be violated in practice and what the implications would be.
The authors should reflect on the scope of the claims made, e.g., if the approach was only tested on few datasets or with few runs. In general, empirical results often depend on implicit assumptions, which should be articulated.
The authors should reflect on the factors that influence the performance of the ap- proach. For example, facial recognition algorithm may perform poorly when image resolution is low or images are taken in low lighting. Or a speech-to-tex system might not be used reliably to provide closed captions for online lectures because fails to handle technical jargon.
The authors should discuss the computational efficiency of the proposed algorithms and how they scale with dataset size.
If applicable, the authors should discuss possible limitations of their approach ad- dress problems of privacy and faireess.
While the authors might fear that complete honesty about limitations might be used by reviewers as grounds for rejection, worse outcome might be that reviewers discover limitations that aren't 't acknowledged in the paper. The authors should use their best judgment and recognize that individual actions in favor of transparency play an impor- tant role in developing norms that preserve the integrity of the community. Reviewers will be specifically instructed to not penalize honesty concerning limitations.
3. Theory Assumptions and Proofs
Question: For each theoretical result, does the paper provide the full set of assumptions and a complete (and correct) proof?
Answer: [NA]
Justification:
This paper is application paper. No theoretical assumption is made.
Guidelines:
The answer NA means that the paper does not include theoretical results.
All the theorems, formulas, and proofs in the paper should be numbered and cross- referenced.
All assumptions should be clearly stated or referenced in the statement of any theo- rems.
The proofs can either appear in the main paper or the supplemental material, but if they appear in the supplemental material, the authors are encouraged provide short proof sketch to provide intuition.
Inversely, any informal proof provided in the core of the paper should be comple- mented by formal proofs provided in appendix or supplemental material.
Theorems and Lemmas that the proof relies upon should be properly referenced.
4. Experimental Result Reproducibility
Question: Does the paper fully disclose all the information needed to reproduce the main experimental results of the paper to the extent that it affects the main claims and/or conclu- sions of the paper (regardless of whether the code and data are provided not)?
Answer: [Yes]
Justification: We have provided model details and training details to improve reproducibil- ity. We will also release the code after the review phase.
Guidelines:
20

---

The answer NA means that the paper does not include experiments.
If the paper includes experiments, a No answer this question will not be perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data are provided or not.
If the contribution is a dataset and/or model, the authors should describe the steps taken make their results reproducible verifiable.
Depending on the contribution, reproducibility can be accomplished in various ways. For example, if the contribution is novel architecture, describing the architecture fully might suffice, or the contribution a specific model and empirical evaluation, it may be necessary to either make it possible for others to replicate the model with the same dataset, or provide access to the model. In general. releasing code and data is often one good way to accomplish this, but reproducibility can also be provided via detailed instructions for how to replicate the results, access to a hosted model (e.g., in the case of a large language modell, releasing of model checkpoint, other means that are appropriate to the research performed.
While NeurIPS does not require releasing code, the conference does require all sub- missions to provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution. For example
(a) the contribution primarily new algorithm, the paper should make it clear how to reproduce that algorithm.
(b) If the contribution is primarily new model architecture, the paper should describe the architecture clearly and fully.
(c) the contribution is a new model (e.g., a large language model), then there should either be way to access this model for reproducing the results or way to re- produce the model (e.g., with an open-source dataset or instructions for how to construct the dataset).
(d) We recognize that reproducibility may be tricky in some cases, in which case au- thors are welcome to describe the particular way they provide for reproducibility. In the case of closed-source models, it may be that access to the model is limited in some way (e.g., to registered users), but it should possible for other researchers to have some path to reproducing or verifying the results.
5. Open access to data and code
Question: Does the paper provide open access the data and code, with sufficient instruc- tions to faithfully reproduce the main experimental results, as described in supplemental material?
Answer: [Yes]
Justification: We have provided model details and training details to improve reproducibil- ity. We will also release the code after the review phase.
Guidelines:
The answer NA means that paper does not include experiments requiring code.
Please see the NeurIPS code and data submission guidelines (https://nips.c cc/
for more details.
While we encourage the release of code and data, we understand that this might not be possible, so No an acceptable answer. Papers cannot be rejected simply for not including code, unless this is central to the contribution (e.g., for a new open-source benchmark).
The instructions should contain the exact command and environment needed to run to reproduce the results. See the NeurIPS code and data submission guidelines (https:
//mips.cc/publi///uuii
ssiionolicyy for more details.
The authors should provide instructions on data access and preparation, including how to access the raw data, preprocessed data, intermediate data, and generated data, etc.
The authors should provide scripts to reproduce all experimental results for the new proposed method and baselines. only subset of experiments are reproducible, they should state which ones are omitted from the script and why.
21

---

At submission time, to preserve anonymity, the authors should release anonymized versions (if applicable).
Providing as much information as possible in supplemental material (appended to the paper) recommended, but including URLs to data and code is permitted.
6. Experimental Setting/Details
Question: Does the paper specify all the training and test details (e.g., data splits, hyper- parameters, how they were chosen, type of optimizer, etc.) necessary to understand the results?
Answer: [Yes]
Justification: We have provided details experimental setting and details in the paper and appendix.
Guidelines:
The answer NA means that the paper does not include experiments.
The experimental setting should be presented in the core the paper to level of detail that is necessary appreciate the results and make sense of them.
The full details can be provided either with the code, in appendix, as supplemental material.
7. Experiment Statistical Significance
Question: Does the paper report error bars suitably and correctly defined or other appropri- ate information about the statistical significance of the experiments?
Answer: [Yes]
Justification:
We use 95% confidence in user study.
Guidelines:
The answer NA means that the paper does not include experiments.
The authors should answer "Yes" if the results are accompanied by error bars, confi- dence intervals, or statistical significance tests, at least for the experiments that support the main claims of the paper.
The factors of variability that the error bars are capturing should clearly stated (for example, train/test split, initialization, random drawing of some parameter, or overall run with given experimental conditions).
The method for calculating the error bars should be explained (closed form formula,
call a library function, bootstrap, etc.)
The assumptions made should be given (e.g.. Normally distributed errors).
should be clear whether the error bar is the standard deviation or the standard error of the mean.
is OK report 1-sigma error bars, but should state it. The authors should prefer- ably report a 2-sigma error bar than state that they have a 96% CI, if the hypothesis of Normality of errors not verified.
For asymmetric distributions, the authors should careful not to show in tables or figures symmetric error bars that would yield results that are out of range (e.g. negative error rates).
If error bars are reported in tables or plots, The authors should explain the text how they were calculated and reference the corresponding figures or tables in the text.
8. Experiments Compute Resources
Question:
For each experiment, does the paper provide sufficient information on the com- puter resources (type of compute workers, memory, time of execution) needed reproduce the experiments?
Answer: [Yes]
Justification:
We have introduced the compute resources used during the experiments.
Guidelines:
The answer NA means that the paper does not include experiments.
22

---

The paper should indicate the type of compute workers CPU or GPU, internal cluster, or cloud provider, including relevant memory and storage.
The paper should provide the amount compute required for each of the individual experimental runs well estimate the total compute.
The paper should disclose whether the full research project required more compute than the experiments reported in the paper (c.g.. preliminary or failed experiments that didn't 't make it into the paper).
9. Code Of Ethics
Question: Does the research conducted in the paper conform, in every respect, with the NeurlPS Code of Ethics
Answer: [Yes]
Justification:
The research conform to the NeurlPS code of Ethics,
Guidelines:
The answer NA means that the authors have not reviewed the NeurIPS Code of Ethics.
If the authors answer No, they should explain the special circumstances that require deviation from the Code of Ethics.
The authors should make sure to preserve anonymity (e.g.. there a special consid- eration due to laws or regulations in their jurisdiction).
10. Broader Impacts
Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?
Answer: [Yes]
Justification: Please see Appendix F
Guidelines:
The answer NA means that there no societal impact of the work performed.
If the authors answer NA or No, they should explain why their work has no societal impact why the paper does not address societal impact.
Examples of negative societal impacts include potential malicious unintended uses (e.g., disinformation, generating fake profiles, surveillance), fairness considerations (e.g., deployment of technologies that could make decisions that unfairly impact spe- cific groups), privacy considerations, and security considerations.
The conference expects that many papers will be foundational research and not tied to particular applications, let alone deployments. However, if there is a direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that an improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point out that a generic algorithm for optimizing neural networks could enable people to train models that generate Deepfakes faster.
The authors should consider possible harms that could arise when the technology is being used as intended and functioning correctly, harms that could arise when the technology being used as intended but gives incorrect results, and harms following from (intentional unintentional) misuse of the technology.
If there are negative societal impacts, the authors could also discuss possible mitiga- tion strategies (e.g., gated release of models, providing defenses addition to attacks, mechanisms for monitoring misuse, mechanisms to monitor how system learns from feedback over time, improving the efficiency and accessibility ML).
11. Safeguards
Does the paper deseribe safeguards that have been put in place for responsible release of data or models that have high risk for misuse (c.g., pretrained language models, image generators, or scraped datasets)?
Question:
Answer: [Yes]
Justification: Please Appendix E
23

---

Guidelines:
The answer NA means that the paper poses no such risks.
Released models that have high risk for misuse or dual-use should be released with necessary safeguards allow for controlled use of the model, for example by re- quiring that users adhere to usage guidelines or restrictions to access the model or implementing safety filters.
Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images.
We recognize that providing effective safeguards is challenging, and many papers do not require this, but we encourage authors to take this into account and make best faith effort.
12. Licenses for existing assets
Are the creators or original owners of assets (e.g., code, data, models), used in the paper, properly credited and the license and terms of use explicitly mentioned and properly respected?
Question:
Answer: [Yes]
Justification: The research conform the licenses for existing assets.
Guidelines:
The answer NA means that the paper does not use existing assets.
The authors should cite the original paper that produced the code package or dataset.
The authors should state which version of the asset is used and, if possible, include URL.
The name of the license (e.g., CC-BY 4.0) should be included for each asset.
For scraped data from a particular source (e.g., website), the copyright and terms of service of that source should be provided.
assets are released, the license, copyright information, and terms use in the pack- age should be provided. For popular datasets, paper thcode com com/dataseta has curated licenses for some datasets. Their licensing guide can help determine the li- cense of dataset.
For existing datasets that re-packaged, both the original license and the license of the derived asset (if has changed) should be provided.
If this information is not available online, the authors are encouraged to reach out to the asset's 's creators.
13. New Assets
Question: Are new assets introduced in the paper well documented and is the documenta- tion provided alongside the assets?
Answer: [Yes]
Justification:
We will well documented the source code that will be released after the re-
view.
Guidelines:
The answer NA means that the paper does not release new assets.
Researchers should communicate the details of the dataset/code/model part of their submissions via structured templates. This includes details about training, license, limitations, etc.
The paper should discuss whether and how consent was obtained from people whose asset is used.
At submission time, remember to anonymize your assets (if applicable). You can either create an anonymized URL or include an anonymized zip file.
14. Crowdsourcing and Research with Human Subjects
Question: For crowdsourcing experiments and research with human subjects, does the pa- per include the full text of instructions given to participants and screenshots, if applicable, well as details about compensation (if any)?

---

Answer: [Yes]
Justification: See Appendix OK!
Guidelines:
The answer NA means that the paper does involve crowdsoureing nor research with human subjects.
Including this information in the supplemental material is fine, but if the main contri- bution of the paper involves human subjects, then as much detail as possible should be included in the main paper.
According the NeurIPS Code of Ethics, workers involved in data collection, tion, or other labor should be paid at least the minimum wage the country of the data collector.
15. Institutional Review Board (IRB) Approvals or Equivalent for Research with Human Subjects
Question: Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or an equivalent approval/review based on the requirements of your country or institution) were obtained?
Answer: [NA]
Justification: There is no risks in mean opinion score test.
Guidelines:
The answer NA means that the paper does involve crowdsoureing nor research with human subjects.
Depending on the country in which research is conducted, IRB approval (or equiva- lent) may be required for any human subjects research. you obtained IRB approval, you should clearly state this in the paper.
We recognize that the procedures for this may vary significantly between institutions and locations, and we expect authors to adhere to the NeurlPS Code of Ethics and the guidelines for their institution.
For initial submissions, do include any information that would break anonymity (if applicable), such as the institution conducting the review.
25


Reviews:
[{"review_id": "3exOljYZBI", "reviewer": "Reviewer_fWzj", "summary": "This paper targets to tackle efficient 3D realistic talking face customization. Rather than learning an individual neural radiance field for each identity, this work exploits a person-agnostic model to improve the efficiency and robustness of personalized talking face generation. A static-dynamic-hybrid adaptation pipeline is proposed to adapt a person agnostic base model to a specific identity. To achieve style control, an in-context audio-to-motion model is devised to mimic the talking style of reference video.", "questions": "Please refer the weakness section.", "limitations": "Authors adequately addressed the limitations.", "rating": 5, "confidence": 3, "soundness": 2, "presentation": 2, "contribution": 2, "strengths": "1. Overall, I like the idea of first learning a generic base talking face model, on top of which the model is adapted to a personalized identity. \n\n2. The design of adapting the neural volume through LoRA with PNCC input is intuitive and interesting.  \n\n3. To generate personalized facial motion, an in-context style mimicking audio-driven module is proposed to inject the dynamic motion style.", "weaknesses": "1. According to the attached supplementary video, the improvement seems very marginal or even hard to notice. Also, only three personal identities are showcased. It will be more persuasive if more identities are demonstrated. I am concerned whether the PNCC representation offers sufficient information or not. At the same time, the simultaneous optimization of a learnable reconstructed face and LoRA looks weird, making it confusing about which component is really functioning. \n\n2. Authors claim this approach is a fast adaptation method that significantly surpasses the previous person-dependent approaches. In the comparison approaches, all the 3 Nerf based methods do not aim to improve the efficiency. Is there any related work that also handles the efficiency of generation or this is the first work. If not, authors should also include relevant approaches in this paper. Specifically, there might be other works targeted to efficiently fitting to a specific identity rather than adaptation. Will it be better to also take them into consideration instead of making a strong claim of 47 times speed faster.\n\n3. The information in the workflow of Fig.2 is minimal. The representation of the NerF field uses a cube, which might have a clearer description. Meanwhile, each component of the devised network is drawn as a block diagram, which might require clearer description on the specific architecture details (If not here, the appendix). Similarly, for the inference pipeline, authors spent a lot of effort to illustrate the flow-matching training paradigm, which makes this figure a little bit messy.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "# Author Response to Reviewer fWzj (Part 1/2)\n\nWe thank the reviewer for the constructive feedback and the positive remarks on our proposed \"generic-model-to-adaptation\" framework. We acknowledge that your concerns are mainly about qualitative results and some technical designs, and hope our response resolves your concerns fully.\n\n> Q1: The improvement seems very marginal or even hard to notice. Only three personal identities are showcased. It will be more persuasive if more identities are demonstrated.\n- A1: Thanks for your helpful feedback. To better compare the performance, we additionally tested 9 additional identities (from an AAAI 2024 Kaggle competition of talking face generation) with diverse languages and appearances and critical conditions (such as large head pose or long hair). Please kindly refer to **Rebuttal Demo 1** on our demo page (please refer to the paper for the URL) and the results prove that our method has better data efficiency and lip-sync quality than the most competitive baseline ER-NeRF (ICCV 2023). Besides, in the **Rebuttal Demo 3** on our demo page, we show that our method could well handle the OOD pose while the baseline cannot. We suspect the prior knowledge in our generic backbone is the reason for our method's generalizability to OOD poses.\n\n> Q2: I am concerned whether the PNCC representation offers sufficient information or not.\n- A2: PNCC is a rasterized image obtained by projecting the 3DMM mesh of the target identity. It possesses fine-grained geometry information of the human head of the target expression. So we suspect the PNCC's amount of information is equal to (or slightly less than) the 3DMM mesh. There are several previous works that also adopt PNCC and achieve subtle expression control. We also provide an additional demo video (**Rebuttal Demo 4 on our demo page**), in which we show that using PNCC representation we can control various talking styles.\n- However, we acknowledge that the PNCC (or other explicit motion representations like 3DMM expression code or facial landmark) makes it hard to represent subtle movements like eyeball movement, which is a common problem for all talking face methods that use explicit motion representations. We plan to explore implicit representation such as audio-to-latent to improve the expressiveness of the model. We will add the above discussion to the limitation and future work section in the revised version. Thanks for your insightful suggestion!\n\n> Q3:  The simultaneous optimization of a learnable reconstructed face and LoRA looks weird, making it confusing about which component is really functioning.\n- A3: Thanks for your feedback. As shown in **line 2 and line 3 of Table 3 in the original paper**, only optimizing either the learnable reconstructed face or LoRA leads to a significant performance drop. Actually, the simultaneous optimization of these two components is the reason for the name of our adaptation pipeline \"static-dynamic-hybrid adaptation\" in Section 3.3. Specifically, learning the reconstructed face (i.e., the triplane representation) is to optimize the target identity's static attributes (such as geometry shape or appearance), and optimizing the LoRA in the model backbone is to learn the personalized dynamic attributes (such as how the facial muscle moves when doing a specific expression). **Empirically**, we find only optimizing the learnable reconstructed face could preserve the static details like teeth and hair of the target identity, but the talking avatar lacks personalized dynamic attributes of the target person (e.g., Theresa May has deep wrinkles when smiling). On the other hand, only optimizing the LoRA could produce videos with personalized facial subtle motions, but the static details are missing (e.g., high-frequency in the hair or teeth region). Therefore, we find a joint optimization of the learnable reconstructed face and LoRA could simultaneously learn the static and dynamic attributes of the target identity.\n- We found our original discussion on the design of SD-hybrid adaptation is insufficient and may cause confusion. We will add the discussion above in Section 3.3 of the revised version. Thanks for your helpful comment!\n\n> Q4: Is there any related work that also handles the efficiency of generation or this is the first work. If not, the authors should also include relevant approaches in this paper. Specifically, there might be other works targeted at efficiently fitting a specific identity rather than adaptation. Will it be better to also take them into consideration instead of making a strong claim of 47 times speed faster?\n- A4: Thanks for pointing this out! We notice that there has been training acceleration from the earliest NeRF-based AD-NeRF (ICCV 2021, taking 40 hours for training an identity) to RAD-NeRF (arxiv 2022, taking 10 hours) to the recent ER-NERF (ICCV 2023, taking 4 hours). However, this improvement of efficient fitting is caused by the improvement of model structure (from the AD-NeRF's MLP-based vanilla NeRF to the RAD-NeRF's grid-based instant-NGP, to the ER-NeRF's attention-enhanced instant-NGP). To our knowledge, we are the first work that improves training efficiency by proposing a new training paradigm (fast adapting from a pre-trained person-agnostic model), which is orthogonal to previous works that improve the network structure. By exploiting the rich prior knowledge from the generic model backbone, our method achieves not only faster convergence but also better video quality and generalizability. We hope the proposed \"generic-model-to-adaptation\" training paradigm could pave the way for a new generation of NeRF-based works. We will add the discussion above in the related works section of the revised version, which we find could better highlight the novelty of this paper.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "PBXIxfN5KG", "reviewer": "Reviewer_iWSN", "summary": "This MimicTalk work aims to bridge the gap between the person-agnostic one-shot TFG setting and the person-dependent small-scale TFG setting. A carefully designed static-dynamic-hybrid adaptation pipeline is proposed to achieve expressive, generalized, and efficient personalized TFG. Furthermore, an in-context stylized audio-to-motion model is introduced to mimic the talking style from the reference video. Experiments demonstrate the superiority of MimicTalk compared to previous methods.", "questions": "The loss weight choice could be discussed.", "limitations": "The limitations have been discussed.", "rating": 7, "confidence": 4, "soundness": 4, "presentation": 3, "contribution": 3, "strengths": "The introduction part presents a good observation of identity-dependent and identity-agnostic methods. To overcome their limitations, a static-dynamic-hybrid adaptation pipeline is introduced to first build an initial 3D face mode from a pretrained one and then finetune it with small-scale person-specific video data. To make the finetune process more efficient and stable, the LoRA technique is adopted. The design motivation is clear and intuitive.\n\nTo achieve audio-to-motion generation, an In-Context Stylized Audio-to-Motion module is introduced built upon flow matching, and trained via infilling mask task.\n\nThe illustrated video results and quantitative results show the effectiveness and efficiency of the introduced pipeline.\n\nThe paper is well-organized and easy to follow. The illustration and implementation details are well demonstrated.", "weaknesses": "There are several losses used in the pipeline. How to choose these loss weights and how these loss weights affect the performance could be further discussed.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We are grateful for your positive review and valuable comments, and we hope our response fully resolves your concerns.\n\n> Q1: There are several losses used in the pipeline. How to choose these loss weights and how these loss weights affect the performance could be further discussed.\n\n- A1: Thanks for your advice. Since the training objective of the renderer and the ICS-A2M consists of multiple terms, we acknowledge that there should be more discussion on choosing the appropriate loss weights to achieve a good overall quality. Specifically, \n- 1\\) as shown in Equation (4) of the original paper, the loss of the renderer is: \n\n$$\\mathcal{L}_\\text{renderer} = \\mathcal{L}_1 + \\lambda _{LPIPS}  \\cdot \\mathcal{L} _\\text{LPIPS}  + \\lambda _{ID} \\cdot \\mathcal{L} _\\text{ID} .$$ \n- Where $\\mathcal{L}_ 1$ is the L1 loss between the predicted frame and the GT frame and is the main objective. However, using L1 loss alone is known to cause over-smoothing in the generated sample, hence we additionally adopt LPIPS loss to improve the high-frequency texture details in the generated result. However, LPIPS loss is insensitive to tiny spatial offsets, hence we take LPIPS as an auxiliary loss to improve image quality and set a relatively small loss weight $\\lambda _\\text{LPIPS}=0.2$. Identity loss $\\mathcal{L} _{ID}$ is similar to LPIPS loss, except that the loss is empowered by VGGFace, which focuses more on face similarity. We find that a large identity loss causes training instability, hence setting a small loss weight $\\lambda _\\text{ID}=0.1$.\n\n- 2\\) As for the ICS-A2M model, as shown in Equation (6) of the original paper, the loss is:\n\n $$\\mathcal{L} _\\text{ICS-A2M} = \\mathcal{L} _\\text{CFM} + \\lambda _\\text{sync}\\cdot\\mathcal{L} _\\text{sync},$$\n\n- where L_CFM is the conditional flow matching (CFM) objective and L_sync is a discriminative loss that measures the synchronization between the input audio and the predicted expression. We found sync loss is necessary to improve lip-sync quality as otherwise the generated motion tends to be over-smoothed. However, a large weight of sync loss (e.g.) makes the generated sample over-flicking, and we found $\\lambda_\\text{sync}=0.05$ is a good choice to balance the temporal stability and lip-sync accuracy of the generated sample.\n- We plan to add the above discussion in Appendix B. of the revised manuscript, which we found could help the reader to understand the design of the proposed multi-objective training loss.\n\n# Summary\nFollowing your comments, we did an in-depth analysis of the design of the proposed method, which we found has improved the clarity of the paper. Again, we thank the reviewer for the insightful review and \"Accept\" recommendation for our paper.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "3YoJgdsYaO", "reviewer": "Reviewer_TKxi", "summary": "The paper presents MimicTalk, an approach to improve the efficiency and robustness of personalized talking face generation. Instead of using separate NeRFs for each identity, MimicTalk adapts a person-agnostic NeRF-based model for specific individuals. They also propose an in-context stylized audio-to-motion (ICS-A2M) model to create facial movement that imitates the target person's speaking style. The adaptation process converges quickly for unseen identities. Experiments show that MimicTalk outperforms previous baselines in terms of video quality, efficiency, and expressiveness.", "questions": "1.  The $L_{sync}$ provided in **GeneFace** is used to evaluate the synchronization between audio and 68 sparse landmarks. In this paper, the authors call it audio-expression synchronization loss, which is not adequate. Sparse landmarks cannot reflect the richness and expressiveness of facial motions. It seems the judgement of the motion expressiveness (FID/user study) is missing in this paper.\n2.  The results of talking style control are not sufficient. Only two examples are given. More qualitative/quantitative evaluations of the style controllability are needed.\n3. Since this is about 3D talking face generation. It would be better to see some novel-view results.", "limitations": "The authors have discussed the limitations and broader impacts in their appendix. The authors should also discuss the limitations as I list in the **Weaknesses** section.", "rating": 5, "confidence": 5, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "1.  This paper can achieve fast adaption from generic to person-specific model, and the overall reconstruction quality seems good compared to other methods.\n2. Several proposed improvement also has been proved as effective.", "weaknesses": "1.  The head pose of the talking face video is not generated, for a talking face video, the sync of the head pose and audio is very important.\n2. The facial motion is represented by PNCC, which limits the expressiveness of the generated results, e.g. detailed emotions, eyeball movements.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We thank the reviewer for the constructive feedback and the positive remarks on our Soundness and Contribution. We acknowledge that your concerns are mainly about the experiment of this paper, and hope our response resolves your concerns fully.\n\n> Q1: The head pose of the talking face video is not generated. \n- A1: Thanks for pointing this out! In the original paper, we didn't consider predicting the head pose since we needed to calculate error-based metrics such as average pose distance (APD). We acknowledge that the sync of the head pose and audio is important for talking head generation. Therefore, we additionally train an **audio-to-pose model**, which follows the main structure of the ICS-A2M model proposed in the original paper. Please kindly refer to the **Rebuttal Demo Video 2** on our demo page, whose URL is given in the original paper (due to the rebuttal policy). The Rebuttal Demo 2 shows that our model can produce novel head poses that are coherent with the input audio. We will add the audio-to-pose model to the revised manuscript. Thanks for your helpful comment!\n\n> Q2: The motion representation PNCC may limit the expressiveness of the generated results, such as detailed emotions or eyeball movements.\n- A2: We acknowledge that the PNCC cannot represent subtle emotion and eyeball movement. This is a common problem for all talking face methods that use explicit motion representations (e.g. 3DMM exp code, landmark, PNCC). In the future, we will explore implicit representation such as audio-to-latent to improve the expressiveness of the model. We will add the above discussion to the limitation section in the revised version. Thanks for your insightful suggestion!\n\n> Q3: Sparse landmark-based $L_{sync}$ metric cannot reflect the richness and expressiveness of facial motions. It seems the judgment of motion expressiveness (FID/user study) is missing in this paper.\n- A3: We acknowledge that sparse landmarks (used in GeneFace) or the 3DMM expression code (used in this paper) can cause information loss in measuring the expressiveness of facial motion, so the audio-expression sync loss $L_{sync}$ is not a good choice for evaluation metrics in Table 5 of the original paper. We found it non-trivial to design an objective metric for audio-lip expressiveness. Following your suggestion, we turned to user study. Specifically, we adopt the Comparative Mean Opinion Score (CMOS) as the metric of lip-sync (CMOS-sync) and expressiveness (CMOS-expressive). Please kindly refer to **Table 1 in the attached one-page PDF** for details. We can see that our default setting performs best and the other 4 settings or the baseline ER-NeRF lead to lower CMOS scores.\n\n> Q4:  The results of talking style control are not sufficient. More qualitative/quantitative evaluations of the style controllability are needed.\n- A4: Thanks for your helpful feedback. We have made amendments to the results of talking style control. Specifically, 1) we provide an additional demo video (please kindly refer to **Rebuttal Demo 4** on the demo page) with more style control examples to help the reader make a qualitative comparison. 2) We additionally perform user studies (the Comparative Mean Opinion Score test) to evaluate the style control ability of our method and the baseline StyleTalk. Please refer to **Table 2 in the attached one-page PDF** for details. Both qualitative/quantitative evaluations show that our method performs better in talking style control.\n\n> Q5:  It would be better to see some novel-view results.\n- A5: Thanks for your suggestion! We provide an additional demo video (please kindly refer to **Rebuttal Demo 3** on the demo page), in which we drive our method and the baseline with several out-of-domain poses. We found that our method can well handle the OOD poses and generate high-quality results while the baseline cannot. Possibly, the prior knowledge in our generic backbone is the reason for the generalizability of OOD poses.\n\n# Summary\nIn summary, following the given comments, we have performed in-depth analysis and several experiments, which we find has enhanced the soundness of the paper and proved the performance improvement of the proposed method. Again, we would like to appreciate the reviewer’s valuable review. We sincerely hope the reviewer will reconsider their rating in light of the rebuttal.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "tntHaZc57W", "reviewer": "Reviewer_dHKC", "summary": "The paper presents a novel approach namely MimicTalk to personalized talking face generation. Unlike previous methods that rely on individual neural radiance fields (NeRF) for each identity, the authors propose a more efficient and generalized framework using a person-agnostic generic model. MimicTalk introduces a static-dynamic-hybrid adaptation pipeline and an in-context stylized audio-to-motion model. The proposed method aims to achieve high-quality, expressive results while significantly reducing training time. Experimental results suggest that MimicTalk surpasses previous baselines in video quality, efficiency, and expressiveness.", "questions": "See weaknesses section.", "limitations": "The paper addressed the limitations in the appendices.", "rating": 5, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "1. The paper introduces a novel hybrid approach that combines static and dynamic features for personalized talking face generation. The proposed in-context stylized audio-to-motion model enhances the expressiveness of generated videos by mimicking the talking style from reference videos. The method is reasonable and interesting, and could probably bring some insights to the community. \n\n2. The proposed method significantly reduces the training time required for generating personalized talking faces. \n\n3. By leveraging a person-agnostic model, the paper demonstrates improved robustness and efficiency in handling various identities and out-of-domain conditions. \n\n4. The authors promised to release the code.", "weaknesses": "1. The paper claims superior performance over existing methods, but the experimental results presented are not sufficiently comprehensive or detailed to fully substantiate these claims. Although some video results are provided in the URL, the results are not convincing enough. In particular, I cannot see the advantages of the proposed method in the style-control results. There are also not enough examples for comparison to state-of-the-art approaches. \n\n2. The experiments are limited in scope, primarily focusing on a narrow set of scenarios and datasets. The evaluation lacks a diverse range of conditions such as varying lighting, occlusions, and different levels of head movements, which are crucial for real-world applicability. More comprehensive testing across varied and challenging scenarios would strengthen the validity of the claims.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We are grateful for your positive review and valuable comments, and we hope our response fully resolves your concerns.\n\n> Q1: The experimental results presented are not sufficiently comprehensive or detailed to fully substantiate these claims. Although some video results are provided in the URL, the results are not convincing enough. In particular, I cannot see the advantages of the proposed method in the style-control results.\n- A1: Thanks for your helpful feedback. We have enriched the results of talking style control. Specifically, 1) we tried different classifier-free guidance (CFG) scales in the sampling process of the ICS-A2M model to further improve the talking style control. We provide a new demo video (please kindly refer to **Rebuttal Demo 4** on the demo page) with 6 additional style control examples to help the reader make a qualitative comparison. 2) We also perform user studies to evaluate the style control ability of our method and the baseline StyleTalk. Please refer to **Table 2 in the attached one-page PDF** for details. Both qualitative/quantitative evaluations show that our method performs better in talking style control.\n\n> Q2: There are also not enough examples for comparison to state-of-the-art approaches.\n- A2: We acknowledge that making comparisons on more identities with diverse languages and appearances could better prove the generalizability of our method. To this end, we provide an additional demo video (please kindly refer to **Rebuttal Demo 1** on the demo page) with 9 additional identities and the results prove that our method has better data efficiency and lip-sync quality than the most competitive baseline ER-NeRF (ICCV 2023).\n\n> Q3: The experiments are limited in scope. The evaluation lacks a diverse range of conditions such as different levels of head movements, which are crucial for real-world applicability. \n- A3: Thanks for your insightful comment! We admit that the limited evaluation scope is a common problem in the field of NeRF-based talking face generation. And we have performed an additional experiment that drives our method and the baseline with out-of-domain (OOD) head movements.  We provide an additional demo video (please kindly refer to **Rebuttal Demo 3** on the demo page), and the results show that our method could well handle the OOD head movement while the baseline cannot. We suspect the prior knowledge in our generic backbone is the reason for our method's generalizability to OOD poses. We will add this experiment to the revised manuscript. Besides, as for other critical situations, as can be seen **Rebuttal Demo 1** on our demo page, our method is more robust than the baseline for difficult training videos (such as identities with long hair or sided faces). We also plan to benchmark our method on more critical datasets such as varying lighting and occlusions in the future. \n\n# Summary\nFollowing your comments, we have performed several additional experiments and found the experiment results have been more comprehensive and convincing. Again, we thank the reviewer for the insightful review and positive recommendation for our paper.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:
# General Response
We would like to thank the reviewers for their constructive reviews! Following the comments and suggestions of reviews, we have performed additional experiments and revised the manuscript. We have also uploaded **4 new demo videos on the demo page** (Please kindly refer to our original paper for the URL due to the rebuttal policy). Here we summarize the revision as follows:
- As suggested by Reviewer fWzj and dHKC, we provide a demo video (**Rebuttal Demo 1** on the demo page) that compares our method with the baseline of 9 additional identities, and the results prove that our method has better data efficiency and lip-sync quality.
- To predict head pose from audio, as suggested by Reviewer TKxi, we additionally train an audio-to-pose model, which follows the main structure of the ICS-A2M model proposed in the original paper. As shown in **Rebuttal Demo 2** on the demo page, our model can produce novel head poses that are coherent with the input audio.
- As suggested by Reviewer TKxi and dHKC, we provide a demo video (**Rebuttal Demo 3** on the demo page) that compares our method with the baseline when driven by various OOD head poses, and the results show that our method could well handle the OOD pose while the baseline cannot.
- As suggested by Reviewer dHKC and fWzj, we provide a demo video (**Rebuttal Demo 4** on the demo page) that tests the talking style mimicking ability of our method. By tuning the classifier-free guidance (CFG) scale during the sampling process of the ICS-A2M model, we further improve the style similarity for our flow-matching-based model. The results show that our method could handle various style references well (6 prompts in the video).
- As suggested by Reviewer iWSN and fWzj, we have enriched technical content such as how to make tradeoffs among multiple losses, discussion on previous works that also explore training-efficient NeRF-based talking face generation, and improved the clarity of Figure 2/3 in the original paper. Please refer to the **attached one-page PDF** for details.

Thanks again for the reviewers' great efforts and valuable comments, which have improved the soundness of the manuscript. We have carefully addressed the main concerns and provided detailed responses to each reviewer. We hope you will find the responses satisfactory. We would be grateful if we could hear your feedback regarding our answers to the reviews.

Meta-review: The paper introduces a new approach to improve the efficiency and robustness of personalized talking face generation. The paper receives unanimous accept from all reviewers. The authors are suggested to further revise according to the comments of the reviewers in the final version.
Decision: Accept (poster)