Venue: NeurIPS 2022 Conference

Reviews:

final_score: rating: 5: Borderline accept: Technically solid paper where reasons to accept outweigh reasons to reject, e.g., limited evaluation. Please use sparingly.
review_content: summary: This paper presented a method for learning an animatable 3D-aware generative model for face images. The method is built on top of the 3D-aware generative model GRAM and introduces 3DMM-guided imitation learning to disentangle the latent space corresponding to identity and expression variations. The method is trained on an image dataset and compared with a few previous baselines, DiscoFaceGAN, CONIFG, etc. The method is quantitatively evaluated with FID and KID, where the performance is comparable to but generally not better than previous approaches. The method also performs qualitative comparisons on mostly randomly sampled images where the authors showed better view consistency than previous work.
strengths_and_weaknesses: The paper has the following strengths:

+ The topic is highly interesting. Controlling the trendy 3D-aware generators is of significant interest in the research communities of machine learning, computer vision, and graphics. Achieving high-quality control of 3D-aware generators is also desirable for many applications.

+ The idea is interesting and has its novelty. DiscoFaceGAN has leveraged imitation learning to learn to sample 2D faces from a 3DMM fashioned latent space. This work adopted a similar idea to the 3D-aware GANs and introduced methods to guide/regularize the 3D representations.

+ Results are encouraging. Thanks to 3D-aware GANs, the proposed method resulted in higher quality view consistency, which is generally not easy to achieve with a 2D GAN architecture.

I'm generally satisfied with the technical components and the novelty of this paper. My main concerns are regarding the quality of results and the evaluation, detailed as follows:

- In achieving facial animation, the paper works well around the mouth region but seems to fail at the eye region: the video-driven animation cannot animate motions around the eyes, such as blinks. This indicates no blinking eyes have been learned in either the 3D deformation modeling or the appearance modeling, or a combination of them. One can blame that the training data has few closed-eye images, but would data augmentation of closed-eye images resolve this problem in the 3D GAN scenario? Could 3DMM guidance be sufficient to model eye animation? 

- The DiscoFaceGRAM is a baseline proposed in this method but not well explained. How does it different from the proposed approach? It looks like the DiscoFaceGRAM generated images have artifacts unseen from either DiscoFaceGAN or GRAM, Where are they coming from? Is it due to the architectural design or the training process?

- Quantitative results seem to be not better than previous approaches.

- Almost all comparisons are done with random samples across different approaches, which is not easy to interpret. The best way might be to project images into the latent space and compare the results of editing so there is a common reference. On the other hand, the video results showed some projection-then-edit animations where we can see more artifacts? How were these images/videos produced? It looks like the approach might have limitations dealing with reconstruction/protection compared to 2D StyleGANs. Perhaps more discussion is needed here too.




questions: Please discussion aforementioned questions in the rebuttal.
limitations: I believe there need to be more discussions around the limitations of the method in terms of technical details and applications, as mentioned before. The negative societal impact is adequately discussed in my opinion.
review_title: null

final_score: rating: 5: Borderline accept: Technically solid paper where reasons to accept outweigh reasons to reject, e.g., limited evaluation. Please use sparingly.
review_content: summary: This paper proposes to generate 3DMM controllable 3D-aware faces. They design a generator that produces two 3D fields, namely a template radiance field and an expression-driven 3D deformation field. The expression-driven 3D deformation field is learned with the guidance of 3DMM. Experiments show that this method can produce high-quality 3D consistent controllable faces with delicate control.
strengths_and_weaknesses: ++ The demo video of this paper is impressive from the perspective of controllable 3d-aware generative models.

++ The design of one template radiance field and a deformation field is interesting and reasonable.

++ The paper is well-written.
questions: 1. The authors write in the "Face editing and animation with GANs" section that ", these methods do not offer a generative modeling of face identities thus are difficult to generate virtual subjects". More discussions could be made. Why not compare with them by first generating virtual samples and then controlling the results with 2D GANs? Or as the authors have shown inversion plus editing results. The comparisons can also be made.

2. Why would the 3D landmark losses affect the FID? As described,  it is computed on 5K randomly synthesized images. And why replacing the 3D landmark loss in Eq. 8 with the 2D landmark regression loss would greatly increase the FID score? 
limitations: 1. The whole story in the introduction and title seems to be overclaiming. 

Words such as "Realistic Video Avatars" and “high-quality animatable video avatars” have appeared in the title and abstract. From the perspective of Realistic Video Avatars, the results can barely meet the need. As all samples are aligned at the same position, the results are far from realistic.

Moreover, if the authors intend to show results in a reenactment setting, it is better that they can show comparisons with results generated in a [GRAM + Face Vid2vid/PIRenderer] manner, and ask users to provide studies on the realness and quality of the results.

The reviewer's suggestion is to tone down the story and change the title to something like "Realistic Animatable 3D-Aware Face Image Generation with XXX (Explicitly Deformable Radiance Manifolds)"

2. The comparisons and discussions are not sufficient.

a) The authors write that "Perhaps the most relevant work to ours is DiscoFaceGAN". This might be leading readers to believe that only comparing to DiscoFaceGAN is sufficient. Actually, StyleRig is also very relevant to the task of disentangled generative modeling and basically shares the same setting as this paper. The authors should discuss StyleRig (Although they also suffer from texture stitching) and try making a comparison with them.

b) Please also refer to the question part.

c) The numerical comparisons are limited. Why not use the metrics in DiscoFaceGAN on disentanglement for evaluation?

3.  No audio is provided in the supplementary video, thus the reviewer cannot tell whether the lip movements are consistent with the source. Also, there are no 3D results shown, making the 3D consistent claim not strong enough.
review_title: null

final_score: rating: 5: Borderline accept: Technically solid paper where reasons to accept outweigh reasons to reject, e.g., limited evaluation. Please use sparingly.
review_content: summary: This paper proposes a novel pipeline for animatable 3D-aware face image generation. They first propose a template radiance field to obtain the rough face geometry, then an expression-driven deformation field is proposed to animate the facial images based on the change of 3DMM parameters. Qualitative results verify the effectiveness of the proposed pipeline.
strengths_and_weaknesses: # Strengths

1. The proposed method makes sense to me. It is natural to predict the non-rigid facial movement by inferring the per-point deformation based on 3DMM parameters. The presented qualitative results are mostly visibly plausible.

2. The paper is well-written with thorough related works.

# Weakness

1. In Eq. 7, the chamfer distance is computed between the canonical 3DMM shape (reconstructed by the input exp and id param) and the predicted shape (inferred from the depth map as indicated by authors). How do the authors guarantee that the two shapes are at the same resolution, i.e., they are comparable for CD calculation. Besides, since the 3DMM only contain meaningful information at the facial regions, how do the authors deal with the neighbor regions like hairs?

2. I am curious about how do the authors deal with the wearings, like the glasses, ear rings, etc. Such small items could not be inferred from the 3DMM prior, so how to guarantee the visual quality and 3D shape of them?

3. Missing some relevant comparison baselines. Some important baselines are not compared, including but not limited to [1, 2, 3, 4]

[1] Pie: Portrait image embedding for semantic control, Tewari et al., TOG 2020.

[2] Stylerig: Rigging stylegan for 3d control over portrait images, Tewari et al., CVPR 2020.

[3] Designing an encoder for stylegan image manipulation, Tov et al., SIGGRAPH 2021.

[4] Headnerf: A real-time nerf-based parametric head model, Hong et al., CVPR 2022.
questions: The questions are elaborated in the weakness section, see above.
limitations: They are all properly discussed in the appendix.
review_title: null

final_score: rating: 4: Borderline reject: Technically solid paper where reasons to reject, e.g., limited evaluation, outweigh reasons to accept, e.g., good evaluation. Please use sparingly.
review_content: summary: This paper proposed a framework for face image manipulation. It attempts to achieve fine-grained control over attributes and facial expressions by better preserving 3D consistency. The key contributions include a template implicit field and a 3D deformation field. The reported experimental results show that the proposed method can produce high-quality animatable video avatars with good texture consistency.
strengths_and_weaknesses: Strengths: 
+ The concept of 3D-aware face generation is a sensible idea. Ensuring 3D consistency for face image manipulation has shown high texture consistency across poses and expressions.
Weaknesses:
- It applied 3DMM expression parameters as a prior for disentanglement of latent variables. Although the experiments in this paper only focuses on 2D face image editing, from the title to the abstract, it easily confuses the reader that they addressed 3D face editing task - e.g., animation usually implies the generation of a 3D avatar from a given face.  The last sentence of the abstract about "strong visual 3D consistency" is misleading because there is no 3D reconstruction and editing contribution in this paper indeed.  
- Regarding the 2D face image generation task, the idea of combining image rendering and 3DMM guided face editing is not novel. Also, from the experimental result view, they are not impressive to argue their contributions. From Figure 3 to Figure 4, their improvements are marginal compared to DiscofaceGAN.  
- More importantly, the comparisons are not convincing with only two out of dated methods.  Some missing methods will need to be considered for comparison in the experiment:
TransEditor: Transformer-Based Dual-Space GAN for Highly Controllable Facial Editing 
InterFaceGAN: Interpreting the Disentangled Face Representation Learned by GANs
questions: 1. What is the motivation behind AniFaceGAN? In introduction, there is not enough discussion about the motivation. From line 37 to line 38, the transition is not smooth. What weakness does NeRF have and how the proposed AniFaceGAN can do better? If NeRF is already "producing realistic images with strong 3D consistency", why do authors bother inventing a new method?
2. What does Fig. 2 try to express? I had thought this is the overall framework but apparently, it is at most incomplete. For face image manipulation, we expect both input and output are face images and the underlying 3D model serves as prior or regularization terms. The training framework on the right and the network structure on the left do not help each other. It is simply too difficult to decipher how this figure is related to the task of face image manipulation - e.g., what is the difference between the two radiance manifolds? how come the arrow points from target to template (not the other way around)?
3. Regarding Figs. 3-4, why do authors use different face identities in visual comparison? The same question applies to the video in supplementary materials. When different faces are used to compare different methods, it is difficult to see the difference. 
4. About the comparison with DiscoFaceGAN, there are a few strong claims - e.g., "DiscoFaceGAN shows better image quality in terms of FID and KID, but sacrifices the 3D-consistency" (line 218) and "DiscoFaceGAN suffers from inconsistency for different expressions and poses" (caption of Fig. 5). I am afraid I cannot see this weakness from either Fig. 5 or supplementary video. Can authors provide further clarification? Note that something obvious to the authors might not be that apparent to others.
5. How about the computational complexity of AniFaceGAN? There is little discussion about the computational cost. 
limitations: I am curious about the performance of AniFaceGAN in some extreme situations - e.g., extreme pose or lighting conditions. Meanwhile, all demo video are very short - it is difficult to evaluate the quality of an animation for such short duration. I am wondering if the authors have video demo of longer period - say >3 seconds.
review_title: null

Meta-review: metareview: The paper addresses an interesting topic and advances the state of the art. The authors have responded sufficiently to the criticisms of the reviewers including the one reviewer that was recommending rejection. The authors are encouraged to incorporate the clarifications and additional results in the final version.
confidence: Certain
Decision: Accept