Venue: ICLR 2022 Conference

Reviews:

final_score: rating: 10: strong accept, should be highlighted at the conference
review_content: summary_of_the_paper: In this manuscript, authors proposed a novel 3D-aware generative model for photo-realistic high resolution image synthesis. The proposed method combines Neural Radiance Fields (NerF) and StyleGAN, and tackles the challenges of efficiency, multi-view consistency and rendering quality. Specifically, authors propose to use NeRF to render low resolution images before feeding them into 2D StyleGAN network to upsample the feature maps in order to obtain images with high resolution and therefore the proposed model can render images at interactive rates. Extensive experiments on numerous datasets demonstrate superior performance than prior state of art 3D generative view synthesis approaches and largely close the gap between 2D and 3D view synthesis methods.
main_review: Strength:
(1) The proposed StyleNerF Model can synthesize high resolution, photo-realistic images with strong multi-view consistency, while achieving interactive rendering rate. Furthermore, it allows controls of camera poses and style attributes. 
(2) To tackle the problem of efficiency, authors first use NeRF to volume render low resolution feature maps, and then feed into stylegan 2D synthesis blocks to synthesize high resolution images. To avoid multi-view inconsistencies, authors proposed to use NeRF path regularization along with other design choices.
(3) Extensive experiments on several benchmarks as well as ablation study validate the effectiveness of proposed component and demonstrate state of the art results on 3D generative view synthesis.

Weakness:
(1) The proposed method remove viewing direction from the NeRF to avoid ambiguity learned from single images, which is fine, but strictly speaking it’s no longer a “radiance field”, but instead a “irradiance fields”.
(2) In Figure 3, authors need to add clarification for each notation (such as FG,BG, BLK) to make reader easier for readers to understand.
(3) From the demo video, I saw a “gaze following” phenomena, where the eye looking direction always follows the camera path, is there any explanation and way to fix this problem?
(4) I am curious if the proposed method can also synthesize 3D scenes from unstructured photos. For example, StyleGAN is able to train on photos of nature landscapes to generate beautiful 2D images. I am wondering what would happen if the proposed model was train on such data.

summary_of_the_review: Based on the strength and weakness I mentioned above, I strongly recommend this paper to be accepted because its novelty on closing the gap between 3D GAN and 2D GAN problem and this is the first 3D generative view synthesis work that can render high resolution images with interactive speed while preserving very high rendering quality. Its proposed component can be very useful to push the boundaries of this area in the future.
review_title: null

final_score: rating: 6: marginally above the acceptance threshold
review_content: summary_of_the_paper: The paper presented StyleNeRF, a 3D-aware generative model for high-resolution image synthesis with high multi-view consistency. 
StyleNeRF integrates the neural radiance field (NeRF) into a style-based generator to improve rendering efficiency and 3D consistency. 
It performs volume rendering only to produce a low-resolution feature map, and progressively applies upsampling in 2D.
It also presents a new upsampling module and a new regularization loss to enforce 3D consistency. 

main_review: Strengths:
+ The proposed method is able to synthesize high-resolution images at interactive rates while preserving 3D consistency at high quality. 
+ It enables control of camera poses and different levels of styles. 
+ It also supports new tasks such as style mixing, inversion, and simple semantic edits.

Weaknesses:
- The paper lacks novelty. The method is mostly a simple combination of Pi-GAN which uses a StyleGAN-alike MLP for NeRF, and GIRAFFE which uses volume rendering to generate low-resolution features that are sequentially upsampled by CNN. Thus, the contribution of this work seems not particularly strong. 
- The approximation relationship in Eq. 5 is not quite obvious. Please provide more detailed proof and explanations for it.
- In Figure 2, it is not clear how the internal representations are visualized? What do different colors mean? In addition, the mentioned artifacts can hardly be noticed in the final image output. 
- The upsampler design is not quite clear. It is hard to understand how such a design solves the “chessboard”, “texture sticking” and "bubble" artifacts.
- What is $\varphi_\theta$ in Eq. 7?
- For evaluation, in Figure 4, the results of the baselines, e.g pi-GAN, are surprisingly bad. This is not consistent with the results in the original paper. This requires some explanations.
- In Figure 3, what is $K_{fg}$ and $K_{bg}$, and how to obtain them?
summary_of_the_review: This is a borderline paper.  It achieves very interesting results. However, it has quite a few issues in terms of novelty and evaluation, and many algorithm designs and technical details are not well explained.
review_title: null

final_score: rating: 6: marginally above the acceptance threshold
review_content: summary_of_the_paper: This paper proposes a 3D-aware generative model for photo-realistic high-resolution image synthesis. The core of this work is a set of tricks that are employed to speed up the generation and enforce the multi-view consistency while trying to synthesize higher resolution images in 2D with a CNN-based renderer.
main_review: With the great success that has been achieved by StyleGAN in 2D, it is natural to extend the key idea behind StyleGAN to 3D for a 3D-aware generative model. It is clearly shown in the paper that this work has strengths including:
1. a very good empirical effort to extend StyleGAN to 3D.
2. the proposed method produces SOTA results, the generated images are impressive.
3. the proposed method is fast and supports high-resolution renderings.

Meanwhile, there are also several weaknesses:
1. lack of technical novelty, the proposed combines several existing techniques that are well explored in the GAN field.
2. the key contribution that makes this work conceptually different may be unclear. While it is true that all the tricks as a whole deliver impressive results, it is not easy to identify the unique contribution in this work.
3. 3D-aware generative models have a focus on multi-view consistency, in contrast to 2D GANs. However, the multi-view consistency (particularly at fine levels) is compromised when resorting to the CNN-based renderer to produce the final image. The proposed pipeline is a good workaround but this compromise is absolutely nonnegligible, it does not fundamentally solve the key issue existing in this 3D-aware generative method. This deviates from the original course of the 3D-aware generative model.

More that need clarification:
1. what is the geometric meaning of the equation (5)? Is it that only the feature at the distance of the depth value is hit for the subsequent rendering? If this is the case, the rendering process still has to evaluate the density of all the samples along the ray for obtaining the depth value. I am not sure about the gain of this approximation in terms of improving computation efficiency. Would appreciate it if this can be explained.
2. how general is the upsampler design in this work? It seems to work very well in the paper. Would be exciting to see if this design choice can also be used in other GANs to improve the synthesis and thus show the generalizability of this design choice.
3. this work uses separate networks for the background and the foreground (based on NeRF++), I am wondering if this is critical, as this design is not employed in other baseline methods, in which the generated images have their bg and fg twisted together.
4. it is a bit confusing that GRAF and pi-GAN perform so much worse than in their original paper. The datasets used in this paper are different from theirs, comparisons on their datasets may be a better option to rule out those irrelevant factors (e.g., hyper-parameters) and thus offer a fairly understandable comparison.
5. in Table 1, I understand that FID and KID do not penalize the 2D GAN for having no multi-view consistency, but still, there should be proper highlights in this table... We can have a metric for quantitatively measuring the multi-view consistency (e.g., like in GRAF, measuring the multi-view reconstruction quality of generated multi-view images of the same 3D), on which the proposed method would dominate.

Others:
1. section 3.2, around equation (5), some of the Phi may have incorrect superscripts, causing some troubles in following this part.
2. section 3.4: "an mapping network" -> "a mapping network".
3. it may be better to not expand the equation (2) and not expose many g, the g functions are not used elsewhere, so it may only make it lengthy and a bit more unreadable.
4. section 4.2: "GARF" -> "GRAF".
summary_of_the_review: Overall, I appreciate very much the engineering effort behind this work. Although the lack of significant conceptual novelty, it has truly brought up the 3D-aware generative model onto a new level with great quality. Please check the concerns expressed above for not giving a higher rating.
review_title: null

final_score: rating: 8: accept, good paper
review_content: summary_of_the_paper: This paper proposes StyleNeRF which combines NeRF and a style-based generator to improve rendering efficiency and 3D consistency in high-resolution image synthesis. In this paper, (1) The NeRF is used to produce a low-resolution feature map and upsample it progressively to high resolution. (2) Several designs are proposed to improve 3D consistency, including a desirable upsampler, a novel regularization term. (3) A progressive training strategy is adopted to significantly improves the stability of learning the real geometry.
main_review: Strengths:
(1) StyleNeRF gives better qualitative and quantitative results than the baseline in high resolution situations, which is particularly evident in the fewer artifacts. Also, this method adopts rendering at interactive rates. (2) The paper solves the problem of high-resolution synthesis that cannot be solved by traditional NeRF-based methods. (3) The proposed pioneering approach of fusing 2D GAN tricks and NeRF-based tricks is a meaningful and groundbreaking work.

Weaknesses:
(1) There are some typos. In Figure 3. The 'r' in 'right' should be 'R'. The 'an mapping network' should be 'a mapping network' in the third row of Section 3.4. The pi-GAN and π-GAN(in Baseline details of Appendix C) should be written in the same way. (2) Figure 3 is quite confusing, the blocks in figure should be mentioned in paper, like FG and BLK. (3) It is suggested to conduct more quantitative experiments about different upsampling operators to show the strengths of the proposed method. (4) There is an interesting phenomenon where the pupil position in the human eyes of the synthesized images (Figure 6 and 15)  are different so that the human eyes seem to toward the direction of the camera observation, while the pupil position in the human eyes of the synthesized images (Figure 12 and 13)  are nearly same. Could this be explained?

summary_of_the_review:  I think the contributions of this paper far outweigh its shortcomings and this paper can be accepted.
review_title: null

Meta-review: An interesting paper on combining NerFs with StyleGAN to get high-quality and high-resolution 3d aware generative models. The results are very good visually and also allow interactive speeds.  The technique is natural and concurrent papers are proposing variations

The reviewers identified a few limitations including that the nerf does not have a viewing direction and also seems limited to aligned objects with a common structure, like faces. Still the results are very interesting and suitable for publication.
Decision: Accept (Poster)