Title: MimicTalk: Mimicking a personalized and expressive 3D talking face in minutes
Venue: NeurIPS 2024
Accepted: yes
Keywords: Talking Face Generation; Personalization; Talking Style Control
Abstract: Talking face generation (TFG) aims to animate a target identity's face to create realistic talking videos. Personalized TFG is a variant that emphasizes the perceptual identity similarity of the synthesized result (from the perspective of appearance and talking style). While previous works typically solve this problem by learning an individual neural radiance field (NeRF) for each identity to implicitly store its static and dynamic information, we find it inefficient and non-generalized due to the per-identity-per-training framework and the limited training data. To this end, we propose MimicTalk, the first attempt that exploits the rich knowledge from a NeRF-based person-agnostic generic model for improving the efficiency and robustness of personalized TFG. To be specific, (1) we first come up with a person-agnostic 3D TFG model as the base model and propose to adapt it into a specific identity; (2) we propose a static-dynamic-hybrid adaptation pipeline to help the model learn the personalized static appearance and facial dynamic features; (3) To generate the facial motion of the personalized talking style, we propose an in-context stylized audio-to-motion model that mimics the implicit talking style provided in the reference video without information loss by an explicit style representation. The adaptation process to an unseen identity can be performed in 15 minutes, which is 47 times faster than previous person-dependent methods. Experiments show that our MimicTalk surpasses previous baselines regarding video quality, efficiency, and expressiveness. Video samples are available at https://mimictalk.github.io .


Peer reviews:
{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe paper presents a novel approach namely MimicTalk to personalized talking face generation. Unlike previous methods that rely on individual neural radiance fields (NeRF) for each identity, the authors propose a more efficient and generalized framework using a person-agnostic generic model. MimicTalk introduces a static-dynamic-hybrid adaptation pipeline and an in-context stylized audio-to-motion model. The proposed method aims to achieve high-quality, expressive results while significantly reducing training time. Experimental results suggest that MimicTalk surpasses previous baselines in video quality, efficiency, and expressiveness.\n\nSTRENGTHS:\n1. The paper introduces a novel hybrid approach that combines static and dynamic features for personalized talking face generation. The proposed in-context stylized audio-to-motion model enhances the expressiveness of generated videos by mimicking the talking style from reference videos. The method is reasonable and interesting, and could probably bring some insights to the community. \n\n2. The proposed method significantly reduces the training time required for generating personalized talking faces. \n\n3. By leveraging a person-agnostic model, the paper demonstrates improved robustness and efficiency in handling various identities and out-of-domain conditions. \n\n4. The authors promised to release the code.\n\nWEAKNESSES:\n1. The paper claims superior performance over existing methods, but the experimental results presented are not sufficiently comprehensive or detailed to fully substantiate these claims. Although some video results are provided in the URL, the results are not convincing enough. In particular, I cannot see the advantages of the proposed method in the style-control results. There are also not enough examples for comparison to state-of-the-art approaches. \n\n2. The experiments are limited in scope, primarily focusing on a narrow set of scenarios and datasets. The evaluation lacks a diverse range of conditions such as varying lighting, occlusions, and different levels of head movements, which are crucial for real-world applicability. More comprehensive testing across varied and challenging scenarios would strengthen the validity of the claims.\n\nQUESTIONS:\nSee weaknesses section.\n\nLIMITATIONS:\nThe paper addressed the limitations in the appendices.", "TITLE": "Official Review", "RECOMMENDATION": 5, "REVIEWER_CONFIDENCE": 3, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2024-06-20"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe paper presents MimicTalk, an approach to improve the efficiency and robustness of personalized talking face generation. Instead of using separate NeRFs for each identity, MimicTalk adapts a person-agnostic NeRF-based model for specific individuals. They also propose an in-context stylized audio-to-motion (ICS-A2M) model to create facial movement that imitates the target person's speaking style. The adaptation process converges quickly for unseen identities. Experiments show that MimicTalk outperforms previous baselines in terms of video quality, efficiency, and expressiveness.\n\nSTRENGTHS:\n1.  This paper can achieve fast adaption from generic to person-specific model, and the overall reconstruction quality seems good compared to other methods.\n2. Several proposed improvement also has been proved as effective.\n\nWEAKNESSES:\n1.  The head pose of the talking face video is not generated, for a talking face video, the sync of the head pose and audio is very important.\n2. The facial motion is represented by PNCC, which limits the expressiveness of the generated results, e.g. detailed emotions, eyeball movements.\n\nQUESTIONS:\n1.  The $L_{sync}$ provided in **GeneFace** is used to evaluate the synchronization between audio and 68 sparse landmarks. In this paper, the authors call it audio-expression synchronization loss, which is not adequate. Sparse landmarks cannot reflect the richness and expressiveness of facial motions. It seems the judgement of the motion expressiveness (FID/user study) is missing in this paper.\n2.  The results of talking style control are not sufficient. Only two examples are given. More qualitative/quantitative evaluations of the style controllability are needed.\n3. Since this is about 3D talking face generation. It would be better to see some novel-view results.\n\nLIMITATIONS:\nThe authors have discussed the limitations and broader impacts in their appendix. The authors should also discuss the limitations as I list in the **Weaknesses** section.", "TITLE": "Official Review", "RECOMMENDATION": 5, "REVIEWER_CONFIDENCE": 5, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2024-07-01"}

comments: SUMMARY:
This MimicTalk work aims to bridge the gap between the person-agnostic one-shot TFG setting and the person-dependent small-scale TFG setting. A carefully designed static-dynamic-hybrid adaptation pipeline is proposed to achieve expressive, generalized, and efficient personalized TFG. Furthermore, an in-context stylized audio-to-motion model is introduced to mimic the talking style from the reference video. Experiments demonstrate the superiority of MimicTalk compared to previous methods.

STRENGTHS:
The introduction part presents a good observation of identity-dependent and identity-agnostic methods. To overcome their limitations, a static-dynamic-hybrid adaptation pipeline is introduced to first build an initial 3D face mode from a pretrained one and then finetune it with small-scale person-specific video data. To make the finetune process more efficient and stable, the LoRA technique is adopted. The design motivation is clear and intuitive.

To achieve audio-to-motion generation, an In-Context Stylized Audio-to-Motion module is introduced built upon flow matching, and trained via infilling mask task.

The illustrated video results and quantitative results show the effectiveness and efficiency of the introduced pipeline.

The paper is well-organized and easy to follow. The illustration and implementation details are well demonstrated.

WEAKNESSES:
There are several losses used in the pipeline. How to choose these loss weights and how these loss weights affect the performance could be further discussed.

QUESTIONS:
The loss weight choice could be discussed.

LIMITATIONS:
The limitations have been discussed.
TITLE: Official Review
SOUNDNESS_CORRECTNESS: 4
DATE: 2024-07-06

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper targets to tackle efficient 3D realistic talking face customization. Rather than learning an individual neural radiance field for each identity, this work exploits a person-agnostic model to improve the efficiency and robustness of personalized talking face generation. A static-dynamic-hybrid adaptation pipeline is proposed to adapt a person agnostic base model to a specific identity. To achieve style control, an in-context audio-to-motion model is devised to mimic the talking style of reference video.\n\nSTRENGTHS:\n1. Overall, I like the idea of first learning a generic base talking face model, on top of which the model is adapted to a personalized identity. \n\n2. The design of adapting the neural volume through LoRA with PNCC input is intuitive and interesting.  \n\n3. To generate personalized facial motion, an in-context style mimicking audio-driven module is proposed to inject the dynamic motion style.\n\nWEAKNESSES:\n1. According to the attached supplementary video, the improvement seems very marginal or even hard to notice. Also, only three personal identities are showcased. It will be more persuasive if more identities are demonstrated. I am concerned whether the PNCC representation offers sufficient information or not. At the same time, the simultaneous optimization of a learnable reconstructed face and LoRA looks weird, making it confusing about which component is really functioning. \n\n2. Authors claim this approach is a fast adaptation method that significantly surpasses the previous person-dependent approaches. In the comparison approaches, all the 3 Nerf based methods do not aim to improve the efficiency. Is there any related work that also handles the efficiency of generation or this is the first work. If not, authors should also include relevant approaches in this paper. Specifically, there might be other works targeted to efficiently fitting to a specific identity rather than adaptation. Will it be better to also take them into consideration instead of making a strong claim of 47 times speed faster.\n\n3. The information in the workflow of Fig.2 is minimal. The representation of the NerF field uses a cube, which might have a clearer description. Meanwhile, each component of the devised network is drawn as a block diagram, which might require clearer description on the specific architecture details (If not here, the appendix). Similarly, for the inference pipeline, authors spent a lot of effort to illustrate the flow-matching training paradigm, which makes this figure a little bit messy.\n\nQUESTIONS:\nPlease refer the weakness section.\n\nLIMITATIONS:\nAuthors adequately addressed the limitations.", "TITLE": "Official Review", "RECOMMENDATION": 5, "REVIEWER_CONFIDENCE": 3, "SOUNDNESS_CORRECTNESS": 2, "DATE": "2024-07-14"}