Venue: NeurIPS 2022 Conference

Reviews:

final_score: rating: 5: Borderline accept: Technically solid paper where reasons to accept outweigh reasons to reject, e.g., limited evaluation. Please use sparingly.
review_content: summary: This paper proposed a unified framework FNeVR for face animation, which combines 2D motion warping and 3D volume rendering.  Specifically, a 3D Face Volume Rendering module is designed for facial details rendering. Moreover, a pose editor is also incorporated to change the facial pose. The experimental results on popular benchmarks verify the superiority of FNeVR. 
strengths_and_weaknesses: The manuscript is well written and clearly clarifies the objective and main idea in the research field. The proposed framework first applies to neural volume rendering for face animation and achieves better performance than SOTA methods.
questions: (1) It is better to conduct the ablation study to investigate the influence of two hyperparameters in formula (13). 

(2) From the presentations (Figure 4(b)) for the cross-identity reenactment experiment, the authors only study the rendering performance when the people in source images and driving videos belong to the same gender. Do other human attributes influence the performance of FNeVR, such as gender, age, haircut, and beard?  

(3) In section 4.4, the pose editor is capable of facial pose editing with a slight Euler angle. Can LPE generate reliable results when a larger Euler angle is given? Of course, the given Euler angle is reasonable for human face rotations.  

limitations: In the cross-identity reenactment experiment, more experiments could be included to fully demonstrate the validity of the proposed framework, including cross-gender and cross-age reenactment experiments.
review_title: null

final_score: rating: 5: Borderline accept: Technically solid paper where reasons to accept outweigh reasons to reject, e.g., limited evaluation. Please use sparingly.
review_content: summary: This paper proposes to combine First-order motion, face vid2vid, and face volume rendering into one framework. The prior 3D knowledge from 3DMM is used to help the volume rendering procedure. The SPADE decoder from the open-sourced Face vid2vid is also used for the final prediction of the method. Experiments show that the method improves FOMM and Face vid2vid quantitatively.
strengths_and_weaknesses: ++ The idea of involving a simple orthogonal volume rendering for face generation is actually interesting and reasonable.

++ The way of formulating the volume rendering is nice.

++ The 3D prior from 3DMM is nicely involved without interfering with the inference procedure.

++ The quantitative results outperform many previous methods.

questions: 1. Why exactly is this paper's Orthogonal Adaptive Ray-Sampling better than the hierarchical (two-stage) volume sampling. What is the computational burden difference between the Orthogonal Adaptive Ray-Sampling and NeRF's rendering? And where is the adaptive part?

2. The authors concatenate the volume rendering feature map $F_r$ and the warped feature $F_w$ for the final prediction. What is the performance if the wrapped feature is not sent into the SPADE decoder? Is $F_w$ involved in FOMM+FVR? Is $I_m$ the results of FOMM+FVR?

3. The reviewer assumes that the Flops and memory consumption difference is led by the 3D landmark encoding procedure of Face vid2vid. Thus I am curious about the memory consumption of the models in the ablation studies (FOMM, FOMM + SPADE, FOMM+FVR). According to the paper, FNeVR would have a larger model size than these methods. Particularly, FOMM + SPADE is a combination of open-sourced code, which is the true baseline of the proposed method.

Actually, the reviewer encourages the authors to replace the SPADE decoder with a more lightweight one and adopts a variant of FOMM+FVR as the final result of this paper. As can be seen in Table 4, no obvious gain can be achieved by involving the SPADE decoder. Adding it involves more computation and makes the method less elegant.

4. The ablation study without the loss function (4) is not given. To my understanding, this is very crucial in integrating 3D information.

limitations: I have mixed feelings about this paper. Involving a simple volume rendering into FOMM like reenactment pipeline is intuitive and nice. The whole formulation, including involving 3DMM as weak prior to density estimation is not entirely novel but reasonable. Overall, I think the novelty of this paper is sufficient.

-- However, the video results of this paper are not impressive enough. Only a few videos are involved, which are not sufficient for reviewers to evaluate. No obvious improvements can be seen given the video results. Even re-training FOMM again might lead to the difference. The rotation results are also of poor quality. More importantly, no video comparisons with face vid2vid (S) are given. This is the most important weakness of this paper. 

For now, I am a bit leaning towards rejection, but I will have a look at the rebuttal for the final decision. Why are the results of Face vid2vid not given? The authors should prepare a nicer video for future use of all kinds.

-- The lack of certain analysis and ablations as stated in the Questions part. Moreover, this paper should also consider comparing with other reenact works such as MARR and PIRenderer. 

-- No ethical issues are discussed. As a matter of fact, FOMM itself has certain issues.

-- Certain descriptions in this paper are confusing. For example:

a) In the abstract the authors write "this is the first work to formulate the neural volume rendering of face animation with a new architecture design". To the best of the reviewer's knowledge, every work that involves neural volume rendering into face animation would somehow propose a new architecture. It is not sure what the authors are claiming.

In the introduction, the authors claim "this is the first work of neural volume rendering for face animation", which is inappropriate. NeRF has been applied to face animation in methods such as NerFACE and ADNeRF.

b) The authors use "Model-free methods" to indicate methods such as First-order Motion Model. The "Model" here might lead to misunderstandings as the authors even write "Model-free models". It is actually "human-designed intermediate structural representation" free. 

-- Certain papers mentioned in the review may not be cited. Please check the references.
review_title: null

final_score: rating: 6: Weak Accept: Technically solid, moderate-to-high impact paper, with no major concerns with respect to evaluation, resources, reproducibility, ethical considerations.
review_content: summary: This paper proposes a Face Neural Volume Rendering network for identity-preserving and photo-realistic face animation, which unifies the 2D motion warping and 3D volume rendering. Specifically, on top of 2d motion warping, it leverages the 3d face reconstruction for shape information and proposes a face volume rendering module to enhance facial details. It shows good quality and performance on several talking-head benchmarks. 
strengths_and_weaknesses: - It well integrates 2d motion warping field, 3d face reconstruction and 3d volume rendering into a unified framework and shows the effect of each module with an ablation study. However, I think it would be good if the paper could provide more insights into the motivation to the design of each module. For example, the volume rendering is still typically expensive and why here it opts for volume rendering vs rasterization of neural texture (similar to Deferred Neural Rendering). 

- Extensive experiments and evaluations. Results are quantitatively and qualitatively good. It also provides analysis why the method is inferior to SOTA on some metrics (CSIM)

- The paper is well presented. 


questions: - It will be good to provide speed evaluation at inference, compared to other SOTA methods. 

- Is N_\sigma equal to 1? It is a bit confusing. 

- There are multiple papers that use single-pass uniform sampling when volume rendering feature maps or adaptively sample based on the geometric prior (3d face reconstruction in this case). it does not seem to be a novelty as claimed by the paper. Please clarify the contribution and do a comparison against single-stage uniform sampling or geometric-prior guided sampling. 

 
limitations: - As stated above, for the major contribution of the paper FVR, I do not see a strong design motivation to it given its computational cost. 

- It will be good to show failure examples as well.
review_title: null

Meta-review: metareview: After rebuttal all reviewers recommend acceptance. The authors are encouraged to follow the reviewer suggestions on improving the final paper.
confidence: Certain
Decision: Accept