Venue: NeurIPS 2022 Conference

Reviews:

final_score: rating: 6: Weak Accept: Technically solid, moderate-to-high impact paper, with no major concerns with respect to evaluation, resources, reproducibility, ethical considerations.
review_content: summary: This paper proposes Generative Neural Articulated Radiance Fields, named GNARF, serving as the 3D representation of 3D-aware generator to handle the datasets with more deformation, such as human-body datasets. The non-rigid motion is represented by the surface deformation derived from the source and target meshes, which helps the tri-plane focus on learning the canonical feature volume. The experiments give a comprehensive study of GNARF on several human datasets, including ATISS++, SURREAL, as well as FFHQ. The experimental results demonstrate the GNARF can achieve better image quality and editability compared with several baselines.
strengths_and_weaknesses: This paper aims to generate 3D humans from a posed 2D image collection, which is rarely explored in the literature. Compared to prior arts of 3D-GANs, the paper pays more attention to the non-rigid human bodies instead of the rigid objects like faces and cars. It is valuable to solve this challenging problem. Although this task is very difficult, the synthesized results are not very good. The details of the human body, such as face and clothing,  are of a low quality. Besides, I also have some confusion about the proposed techniques, which will be presented in the following section. 
questions: 1. As shown in this paper, SF is derived from the target and canonical template meshes. Does it mean that each batch also needs to sample the target SMPL models besides the latent code? If I am right, it is not fair to compare with the EG-3D baseline w.o. any 3D mesh information. 
2. It is not convincing that the authors remove the background to stabilize the training process. Why not use a separate background model? StyleNeRF has demonstrated that it can work even on the CompCars dataset with large background variations.
3. Since the tri-plane is required to learn the canonical feature volume, why is the latent code still injected into the tri-plane generator network? Why not only add the stochastics to the shallow MLP because the canonical pose across different instances is similar. 
limitations: My concerns about this paper is the synthesis quality. Although it has improved on the baseline, it still has a large gap to real-world applications like avatar animation. 
Besides, the mouth of the synthesized faces in Fig.6  and the demo video are not realistic. I guess that the Flame model does not model the teeth. Does it mean that the GNARF is very dependent on the template meshes? When transferred to a new dataset like LSUN cats which does not have any ground truth template meshes, how does GNARF perform? 
I think this paper is interesting, and I tend to accept this paper (raise the scores) if the authors can address my concerns.
review_title: null

final_score: rating: 7: Accept: Technically solid paper, with high impact on at least one sub-area, or moderate-to-high impact on more than one areas, with good-to-excellent evaluation, resources, reproducibility, and no unaddressed ethical considerations.
review_content: summary: This paper proposes a technique called GNARF for controllable 3D aware generation of human bodies and faces with different articulations and expressions. The authors build on the prior EG3D generative model and extend it along two dimensions -- to entire 3D bodies and for more explicit control and disentangling of base identity faces and their expressions. The authors' main idea is to introduce an explicit surface deformation module into the generative model, which deforms a canonical triplane representation. The authors parametrize this surface deformation via parametric mesh SMPL or FLAME models. They compare their surface deformation approach against MVC and blend-skinning and also their generative models for body and face against SOTA existing generative models and show improvements in quality.
strengths_and_weaknesses: - Originality: The work is original along many dimensions. It extends 3D-aware generative models to body poses, which has not been considered in prior work (other than in [115], which is concurrent work). It is also novel in introducing the explicit surface deformation model  into the EG3D framework, making it more controllable. However, it builds heavily on EG3D. Nevertheless, in my opinion this work introduces sufficient novelty over the existing works to warrant publication.

- Quality: The work is technically sound; the method appropriate and its steps logical; and most claims (a few exceptions noted below) are well supported via experiments. To the authors' credit they have also gone above and beyond the submission requirements and compared against several unpublished concurrent works [115, 127] and shown the superior performance of their approach. The authors have discussed the limitations of their method honestly and in much detail.

- Clarity: The material is clearly presented and easily understandable. Many implementation details are included in the supplementary material and the authors have promised to release their code upon acceptance.

- Significance: 
- The dominant approach for modeling humans (bodies and faces) and animals has been via parametric mesh models for several decades now. The advent of 3D aware GANS is heralding in a new era of GAN-based generative models, which have the promise to provide much higher images quality and photo-realism, while not being limited only to the surfaces (face and body) modeled by the mesh models. 3D aware-GAN models can also model hair, eyeglasses and clothing details beyond meshes. However, the question of how best to model body articulation/surface deformation caused by expressions/body articulation into the context of 3D-aware GANS is an important open one. This work provides an important step towards addressing this latter problem. That said the proposed solution's falling back on parametric mesh models to model the surface deformation is perhaps not the most elegant solution, as the authors acknowledge themselves as well.
- The authors' comparisons of the different types of 3D mesh-based deformations (MVC, blend-skinning and surface-based) is also quite interesting and insightful.
questions: From Table 2, it is evident that the baseline EG3D (no warping) approach has FID scores nearly as good as or better than GNARF, meaning that it is able to generate highly realistic posed images. However in line 244 the authors claim that for EG3D (no warping) poor results for reposed images are produced "since it is difficult to accurately estimate the SMPL mesh from the generated images". If the image quality of the posed images is good as per the FID scores, what limits good SMPL mesh fitting, and reposing of the EG3D generated images?

How do the authors propose to handle cases such as subject wearing long skirts or dresses, which are likely to not fit well to the two-legged tight-fit model of SMPL?
limitations: The authors have adequately and honestly discussed the limitations of their proposed method.
review_title: null

final_score: rating: 7: Accept: Technically solid paper, with high impact on at least one sub-area, or moderate-to-high impact on more than one areas, with good-to-excellent evaluation, resources, reproducibility, and no unaddressed ethical considerations.
review_content: summary: This paper proposes a method for generative neural articulated radiance fields. The main strategy is to generate radiance fields in a canonical pose and warp them using an explicit deformation field into a desired body pose or facial expression. The main technical contribution of this work is to combine the recently proposed tri-plane feature volume representation with an explicit feature volume deformation which is guided by a template shape. High-quality results on human body and faces are demonstrated in the paper.
strengths_and_weaknesses: This paper proposes a practical 3D-aware GAN framework for the generation of editable radiance fields of human bodies. The main strength of this paper is to combine several existing strategies like tri-plane feature volume representation, StyleGAN2 generator, feature volume deformation, neural volume rendering, image super-resolution, etc. Although each component has been proposed before, integration them together to generate satisfying results is not easy and could be used for related applications.

On the other side, the weakness is the lack of technical contribution as each component is borrowed from other papers. Meanwhile, some component can be replaced with more suitable strategy. For example, based on our implementation and the experimental results reported in the paper, tri-plane representation still has limited representation ability, which can be replaced by other representations to achieve better performance.
questions: I don't have questions.
limitations: As mentioned in the related work, HeadNeRF is related to this work, and the authors point out that HeadNeRF needs to acquire training images of the same person performing various expressions in different lighting conditions. However, HeadNeRF can explicitly control properties like identity and expressions. To my knowledge, the proposed method can not achieve disentangled representation.
review_title: null

Meta-review: metareview: The reviewers all recognize the quality of the work, particularly technical soundness and quality of the experimental setting and there is a clear consensus for acceptance. I ask the authors to address the reviewers concerns, particularly clear up any confusion in the manuscript and better analysis of the synthesis results.  

confidence: Certain
Decision: Accept