Venue: ICLR 2023 Conference

Reviews:

final_score: rating: 6: marginally above the acceptance threshold
review_content: summary_of_the_paper: This paper proposed an end-to-end NeRF-based method, for talking face generation. It was trying to solve 2 problems: 1) weak generalizability due to small scale of training data, 2) “mean face” result: bad image quality and bad lip-synchronization. This paper proposed 3 parts: 1) Variational Motion Generator used to generate landmarks from audio, 2) Domain Adaptative Post-Net for landmark refinement, and 3) NeRF based Renderer for final frame generation. It produces good talking face generation results.
strength_and_weaknesses: Strength:
This paper has well-organized structure, and it is clear in logicality. This work has well-designed network structure and proposed structure, which can actually solve proposed problems. This work combines previous works and gives an new approach to generating talking face in another way, which moves the NeRF-based method field forward a bit.
 
Weaknesses:
Some evaluation metric do not outperform other methods, this work could design more evaluation metrics to evaluate the results.
The generated face in the demo video is moving around, maybe consider temporal information and design into network structure.
The lip-synchronization is not as good as wav2lip, the synchronization can be improved.
clarity,_quality,_novelty_and_reproducibility: This paper has well-organized structure, and it is clear in logicality. This work proposed three sub-network and they can solve problems. It is reproducible work according to the designed network structure and experiment results.
summary_of_the_review: The paper is overall well written and much of it is well described. It combines previous works and proposes an new approach to generating 3D talking face. It produces better results than previous NeRF-based work. I recommend this work.
review_title: null

final_score: rating: 6: marginally above the acceptance threshold
review_content: summary_of_the_paper: The authors address the problem of synthesizing realistic talking faces from audio. They build upon a VAE-style encodings of 3D facial landmarks that are created from HuBERT features extracted from the audio signals. Instead of a vanilla VAE, the authors argue that a flow-based (non-Gaussian) prior is beneficial for better landmark generation. Conditioned on the predicted facial landmarks, a NeRF-style renderer then synthesizes an animated head and torso.
strength_and_weaknesses: **Novelty**

The approach is of minor novelty. Audio-driven talking head synthesis based on NeRF has already been proposed in AD-NeRF and the 3D facial landmark generation is a HuBERT conditioned VAE plus flow modeling. While this specific architecture has not been used before, the paper does not provide significant novelty beyond modified architectures and details in the training.

**Technical Details**

The technical presentation of the paper seems sound and is easy to follow. Details on the model are explained in the appendix, such that I believe the method can be reproduced from the information given in the paper.

**Empirical Evaluation**

(a) Qualitative results (video linked in the paper): The output of the approach produces significant artifacts such as hair that is constantly changing it's size/volume, unstable and wobbly face reconstruction, and significant artifacts in the neck area. I do not agree that this is an improvement over MakeItTalk, PC-AVS, LSP, or AD-NeRF.

(b) User study. The user study has a small sample size (5 clips from 3 languages, 10 attendees) and given this small sample size, the differences in MOS scores compared to existing work seems not statistically significant. I'd further advise to include the standard deviation to the mean opinion scores, which will provide a confidence on the ratings.

(c) Quantitative Evaluation. All metrics are neural feature maps. These alone can be delusive, a metric comparison would be more convincing. One option would be to compare the l2 error on facial landmarks to evaluate accuracy of lip and general face animation.

(d) In-depth analysis. One major contribution of this paper is the variational motion generator for the generation of 3D facial landmarks. However, this contribution is lacking empirical analysis. Fig. 6 provides some insights on the domain adaptation, but a proper evaluation of the quality of the produced landmarks is not given. It would be more convincing if the authors could demonstrate clear and consistent improvements in 3D landmark generation by their variational motion model compared to a simple regression baseline, a simple VAE baseline, and landmarks from other approaches. Unfortunately, neither a quantitative evaluation (l2 error on predicted landmarks) nor a qualitative evaluation is provided.
clarity,_quality,_novelty_and_reproducibility: see comments above.
summary_of_the_review: The paper is of minor novelty and the main technical contribution lacks in-depth evaluation. The presented qualitative results are not convincing, i.e. they do not seem to be better than existing approaches to me.

**Edit after rebuttal:**

The authors provided a strong and convincing rebuttal. They were able to address the major points that, in my view, put the paper below acceptance threshold. The authors could demonstrate that with appropriate post-processing, their method outperforms AD-NeRF, They also provided further evaluation that closes the gap the paper had before.

I consider this a substantial improvement to the original paper and therefore will upgrade my rating.
review_title: null

final_score: rating: 8: accept, good paper
review_content: summary_of_the_paper: This paper presents a method on audio driven 3D portrait. A three-stage pipeline is given. The first stage trains Variational Motion Generator to predict landmarks based on audio signal. The second stage uses a post-net to compensate for the domain shift between the predicted 3D landmarks and the target person domain. The third stage adopts NeRF for high fidelity rendering conditioned on landmarks. When dealing with torso-NeRF, a pixel-wise condition is added to NeRF function to reduce head-torso separation artifacts. 

According to the description in these three stages. The Variational Motion Generator is a generic model. Both the Domain Adaptative Post-net and NeRF renderer adopt subject-specific training to model personalized attributes.

The method could solve one-to-many mapping problem faced by generic NeRF-based talking face systems. And it could solve head-torso separation problem in rendering.
strength_and_weaknesses: strengths:
The method could solve one-to-many mapping problem faced by generic NeRF-based talking face systems. And it could solve head-torso separation problem in rendering.

weaknesses:
1) The title use ‘generalized’ to describe this system. It seems this word implies that no subject-specific training is needed. However according to the paper, both the Domain Adaptative Post-net and NeRF renderer adopt subject-specific training to model personalized attributes and need days to converge. I recommend changing this title to avoid misleading.
2) This paper adopts NeRF for high fidelity rendering. This paper should add discussion on NeRF based head techniques such as NerFACE, HeadNeRF and EG3D, but not restricted to NeRF for talking face. One paper is left: Learning Dynamic Facial Radiance Fields for Few-Shot Talking Head Synthesis (ECCV 2022)
3) This paper lacks discussion on limitations and future work.
4) In Fig 1. Variational Motion Generator’s ‘Encoder’ should be ‘Decoder’. In Equ (8), ‘-’ is left in the computing of accumulated transmittance.
clarity,_quality,_novelty_and_reproducibility: Although some small mistakes occur in this paper. The writing of the paper is well organized. It is easy for readers to follow. I think this paper is reasonable and reproducible.
The Variational Motion Generator, Domain Adaptative Post-net and the torso-NeRF conditioned on pixel-wise colors are novel in NeRF based talking face system.
summary_of_the_review: This paper proposes a high-fidelity audio driven 3D talking face system. A novel generalized audio to landmarks module is given. And a subject-specific NeRF module is used to for high fidelity modeling which is an incremental modification on AD-NeRF architecture.
The title didn’t completely suit for the content and reference is not enough. Also limitations and future work should be added.
review_title: null

final_score: rating: 6: marginally above the acceptance threshold
review_content: summary_of_the_paper: This paper focuses on how to solve the generalization problem of realistic talking face video generation tasks with a landmark refinement module. They proposed GeneFace, a model that introduces additional 3DMM landmark refinement for a target person, thus transferring knowledge learned from a large corpus to out-of-domain data. By utilizing the joint supervision of a lip-syncing expert and an identity discriminator, the model adjusts the generalized 3DMM landmark prediction to match the face of the target person. In this way, the 3DMM prediction can match the distribution of the data used to train the person-specific video renderer. In addition, the paper introduces a head-aware torso-NeRF to address the head-torso separation issue due to dramatic head movements. The generalized and specialized parts of the model are separately trained on a large lip-reading dataset and two smaller lip-reading datasets, and it can achieve good experimental results without fine-tuning, proving its effectiveness.
strength_and_weaknesses: Strength：
1. The paper proposed a novel method to generalize the knowledge learned from the large talking face dataset to out-of-domain data, thus improving few-shot training performance without having to change parameters learned from a large corpus.
2. This paper has done a lot of ablation experiments, and the experimental details shown in Table 3 can enable readers to understand the improvement brought by each method and enhance the reproducibility of the method.
3. There is a detailed description regarding the structure chart of each component and specific hyperparameters of the method in this paper. 
Weakness:
1. There is a little description of the dataset used to train the baseline models, e.g., whether the target person videos are used during training. This information is critical for a fair comparison. 
clarity,_quality,_novelty_and_reproducibility: Clarity: The paper is clear in the description of the method and the description of the performance metrics.
Quality: Overall quality is above average. The article proposes a framework with some innovations. The experiments are focused on the main issue proposed by this paper. 
Novelty: The paper has significant innovation. The domain adaptive post-net is relatively novel, and for the first time, a method other than fine-tuning is proposed for domain transfer in talking face generation.
Reproducibility: The paper has high reproducibility. The description of the method is detailed, and there are detailed experimental details.
summary_of_the_review: The paper has good innovation, has done some original work, and the proposed method is concise and has good reproducibility. But the training details of the baseline model are vague, which makes me skeptical about the result of the baseline comparison part.
review_title: null

final_score: rating: 5: marginally below the acceptance threshold
review_content: summary_of_the_paper: This paper proposes GeneFace, a generalized and high-fidelity NeRF-based talking face generation method to generate natural results corresponding to various out-of-domain audio. This paper introduces a variaitional motion generator on a large lip-reading corpus, a domain adaptative post-net to calibrate the result and NeRFbased renderer conditioned on the predicted motion. The proposed methods achieve good performances on the public datasets.
strength_and_weaknesses: Strength: 1) The proposed methods achieve remarkable and comparable performances with the previous works;

2) The domain adaptive post-net is interesting for the talking head generation tasks among the different targets. It may help enhance the generalization ability of the generated landmarks for the various faces.

Weaknesses: 1) The novelty is limited. The landmark generation and nerf-based rendering networks are widely adopted for 3D talking head generation. The domain generalization is not firstly introduced for the various faces and audios. The speciality and priority of the proposed methods  need to be highlighted in the paper.

2) The experiments cannot support the motivations and methods very well. The proposed methods aim to enhance the generalization abality for various faces. Comparing with the previous methods on the OOD conditions, the advantages of the proposed methods are not obvious.  Especially the comparisons of these methods on OOD faces need to be analyzed to highlight the effectiveness of the proposed domain adaptive post net. 

clarity,_quality,_novelty_and_reproducibility: Clarity: The clarity of this paper is good. The pipeline of the proposed methods is clear and the illustrations are pretty helpful for the readers although there are still lots of typos in the paper.

Quality: The completion of this paper is still good. However the experiments cannot support the motivations and methods very well. The analysis of the proposed methods need to be improved.

Novelty: The novelty is limited. The landmark generation and nerf-based rendering networks are widely adopted for 3D talking head generation. The domain adaptive post-net is interesting to enhance the generalization for the different faces. But lots of works adopt the differences of the target faces and average-face which also helps to enhance the generalization of the methods.

Reproducibility: The clarify of the proposed methods are clear. But the authors did not provide the key code of the method. It is possible to reproduce the performances of the proposed methods but it may be very challenging due to the three complex networks.

summary_of_the_review: This paper introduces a generalized and high-fidelity NeRF-based talking face generation method for 3D talking head generation. The proposed methods consist of  a variaitional motion generator on a large lip-reading corpus, a domain adaptative post-net to calibrate the result and NeRFbased renderer. The writing and completion of this paper is good. However, this paper is not good enough due to the novelty and experimental weaknesses. The novelty of the proposed methods are limited and the experiments cannot support the motivations of the proposed methods very well.
review_title: null

Meta-review: metareview:_summary,_strengths_and_weaknesses: Summary:
This paper presents a NeRF-based method GeneFace for generating realistic talking face from audio. The authors introduce a VAE encoder and a domain-adaptive postnet for mapping audio to 3D landmarks, and for improving generalization to out-of-domain audio.
Experiments demonstrate the effectiveness of the proposed method for generating high-fidelity talking face comparing with previous methods.

Strengths:
- The proposed DNN architecture is reasonable and addresses two issues with previous NeRF-based method -- OOD audio generalization and "mean face" problem.
- The main innovation is the domain adaptive postnet, a method other than fine-tuning for domain transfer in talking face generation.
- The DNN model is described in detail, and the results could be reproducible.
- The paper is well-written and easy to follow.


Weaknesses:
- The domain adaptive transform associated with posenet might reduce the quality of Lip-synchronization while improving image fidelity.
- There is a temporal instability problem with GeneFace and might introduce artifacts such as shaking hairs.
- The training and inference of NeRF-based renderer are time consuming.

justification_for_why_not_higher_score: The main reasons are due to the weakness listed in the summary, especially the domain adaptive transform associated with posenet might reduce the quality of Lip-synchronization while improving image fidelity, and the training and inference of NeRF-based renderer are time consuming.
justification_for_why_not_lower_score: The proposed DNN architecture is reasonable and addresses two issues with previous NeRF-based method -- OOD audio generalization and "mean face" problem. 
Experiments demonstrate the effectiveness of the proposed method for generating high-fidelity talking face comparing with previous methods.
Decision: Accept: poster