Venue: NeurIPS 2022 Conference

Reviews:

final_score: rating: 7: Accept: Technically solid paper, with high impact on at least one sub-area, or moderate-to-high impact on more than one areas, with good-to-excellent evaluation, resources, reproducibility, and no unaddressed ethical considerations.
review_content: summary: The paper attacks the standard but hard problem of starting with a single-view 2D face shot and creating a 3D face model that can take expression and pose as inputs. Their approach is to meld pi-GAN with 3D Morphable Modeling (3DMM) of priors (based on PCA basis faces) and Generative Occupancy Field (GOF).  I believe that they are also proposing the 3D Landmark Loss for this application, and a Volume Warping Loss which is a nice conservation type prior.  
strengths_and_weaknesses: Strengths: 
- The proposals seem totally reasonable: they bring together solid components and good ideas. 
- The comparisons to state-of-the-art seem fine, and I appreciated the additional results in the Supplemental
- They show large gains to controllability
- The ablation study was useful to see 
- I found the paper easy to read and mostly well-written

Weaknesses: 
- This is pretty close to HeadNeRF but with the extreme case of just 1 input view, I would have appreciated more discussion of that
- While it's nice their loss function in (9) balances many goals, they end up with 6 (5 free) hyperparameters lambda to set
- Metrics: it would be nice to see a really proper human rater paired comparisons survey done (but yes I see the qualitative results in the Supplemental)
- They limit the generated images to 64 x 64 during training (and then use another Neural Renderer to map it to 512 x 512... this seemed a bit iffy), and then Table 1 suggests their method without the extra Neural Renderer step). I am not sure I grokked the full ramifications and comparison issues due to this.
questions: -  "we assume that the 3DMM coefficients of all the training images follow a normal distribution".  Surprised that you don't need a conditional covariance matrix for that, does it just not matter much?

- I may have missed it, but I didn't see where or how you chose the different values for lambda's in (9)? 

- I did not understand this sentence "This allows us to incorporate a 3DMM parameter reconstruction loss that ensures the
generated face images to commit to the input condition."
limitations: Authors note the obvious problem in this field of creating fakes. In fact, it is hard to conceive of when you would want to do this other than trying to fake people out, but while I think this is not a really important problem to work on, I recognize that in general 3D from 2D is a popular set of problems and do not think the paper should be rejected for its choice of problem.
review_title: null

final_score: rating: 4: Borderline reject: Technically solid paper where reasons to reject, e.g., limited evaluation, outweigh reasons to accept, e.g., good evaluation. Please use sparingly.
review_content: summary: This paper aims the task of controllable 3D face synthesis based on a set of single-view real-world face images. The authors propose a solution based on the framework of pi-GAN and the widely used parametric 3D Morphable Model (3DMM). The key idea is that the conditional Generative Occupancy Field is employed to explicitly guide the volume sampling procedure of the 3DMM mesh. Meanwhile, for fine-grained 3D shape control, the 3D landmark loss and a volume warping loss is used. In the experiments, both qualitative evaluation and quantitative evaluation show the effectiveness of the proposed method. 
strengths_and_weaknesses: Strengths: The idea of combing pi-GAN and GOF as well as 3DMM for the task of controllable 3D face synthesis shows a certain novelty. 

Weaknesses: The novelty of the proposed method is limited. It can not be regarded as a new framework since all the related methods of pi-GAN, GOF and 3DMM are well-known. Meanwhile, the experimental results are not sufficient. The comparison of the proposed method with the Baseline method is missed in Table 1 and Table 2. Moreover, the presentation of the whole paper is not good. For example, in section 3.2, “where the shape, expression, pose and other factors of a face are modeled by a set of PCA bases and coefficients z = (z_shape, z_exp, z_tex, z_else)”, here, z_tex denotes the factor of pose or texture? 

questions: 1.  In section 3.2, “where the shape, expression, pose and other factors of a face are modeled by a set of PCA bases and coefficients z = (z_shape, z_exp, z_tex, z_else)”, here, z_tex denotes the factor of pose or texture? 

2. Please give the comparsion results of the propsoed method with the baseline model mentioned in the beginning of section 3.2. 
limitations: 1. The novolty of this paper is limited. 
2. The presentation of the whole paper is not smooth and well structured. 
3. The experimental results are not sufficient.
review_title: null

final_score: rating: 5: Borderline accept: Technically solid paper where reasons to accept outweigh reasons to reject, e.g., limited evaluation. Please use sparingly.
review_content: summary: This paper proposed a new NeRF-based conditional 3D face synthesis framework, which enables the control over head poses and expressions using a parametric 3D face model (3DMM). 

The main contribution of this model is a conditional Generative Occupancy Field (cGOF) that effectively enforces the shape of the generated face to commit to the 3DMM model mesh. Besides that, novel loss terms such as 3D landmark loss and volume warping loss are proposed in the system. 

The authors trained the model on the CelebA dataset and conducted both quantitative and qualitative comparisons to other methods. The results and comparisons showed that the proposed method outperforms the state-of-the-art methods. Meanwhile, the ablation study was also done to show the effectiveness of each proposed component.

strengths_and_weaknesses: Strengths:

1. This paper proposed a new NeRF-based 3D face synthesis framework, which allows controlling the large head pose change and expression changes of frontal face images.

2. The core contribution is the usage of the conditional Generative Occupancy Field (cGOF) model. Other terms such as 3D landmark loss and volume warping loss terms all contributed to the effectiveness of the proposed framework.
 
3. This paper is well structured and easy to follow. The authors clearly explained each term and did ablation study to show the impact of each proposed component.

4. The authors further did quantitative and qualitative evaluations on the proposed method and some state-of-the-art methods such as PIE, StyleRig, DiscoFaceGAN, E2E, HeadNeRF, etc, showing that the proposed method outperforms. 

Weaknesses:
1. I'm a little concerned that the originality of this proposed work is not enough. For example, the 3DMM conditioning idea is similar to the work in [8] and the cGOF idea is inspired from [53]. It would be great to show that how the way the authors model the 3DMM conditioning in a normal distribution outperforms the VAE model in [8].

2. Some technical details are not very clear to me. Please see the following questions. 

questions: 1. For the 3DMM conditioning, the authors mentioned that they assume the 3DMM coefficients of all the training images follow a normal distribution, while it's not very clear what are the representations of the expression components. Is it also a PCA model or FACs based model? The authors cited [37], which didn't mention about expression components. 

2. For the 3D landmark loss, the authors mentioned that they didn't consider the facial contour landmarks due to the impact of occlusion. However, in extreme head poses such as 90 degree left/right view, part of the inner facial landmarks are not visible as well. It's not clear how the authors handle these landmarks and be able to generate plausible results for side view head poses. 

3. It would also be great if the authors could include the inference time of the proposed framework. 
limitations: The authors addressed the limitations of the work that the proposed method couldn't handle precise texture and lighting control over the generated face. And also the model relies on 3DMM model, which could not generate high-fidelity shape details such as wrinkles and hair.

For societal impact, the authors mentioned that the work might be used to generate fake facial images. Defensive algorithms on recognizing these images should be developed.
review_title: null

Meta-review: metareview: Paper attacks a hard problem and brings together state-of-the-art ideas to demonstrate substantial wins.  Many good points were raised by the reviewers, and we ask the authors to carefully read through the feedback and address what they can for the final version. 
confidence: Certain
Decision: Accept