Venue: NeurIPS 2021 Conference

Reviews:

final_score: rating: 7: Good paper, accept
review_content: summary: The paper proposes GOF, a method to improve the geometry and quality of generative methods that use a neural radiance field representation. The core idea is to modify the network's output to predict occupancy which can be used to find the surface of the rendered object. Once the surface is known, the network can be sampled in a region around the surface that shrinks during training. This accomplishes two goals, it consolidates the geometry to a surface and requires fewer samples during inference. When compared to existing approaches, the depths and normals are cleaner and the image quality metrics are marginally better.
main_review: I’ve listed my primary concerns below. Overall the paper is well written and the method is simple and straightforward. For these reasons I think this paper is marginally above the acceptance threshold.

a) The paper never actually demonstrates why the recovered representation is better than existing methods (other than showing a depth map). It is noted that downstream tasks like relighting could be improved, but is not demonstrated. A good portion of the paper also discusses the ability to reduce the number of samples taken during inference which could help with “applications on mobile devices”. It would have been nice to see quantitative values demonstrating the speed up of the proposed method. As the paper currently stands, the quality of the image outputs look fairly similar to pi-GAN. I think this paper would be much stronger if advantages on downstream tasks or a speedup compared to baselines were demonstrated.

b) How much can we attribute the improved depth and normals on the use of occupancy compared to the included priors? It would have been nice to see qualitative examples of pi-GAN or GRAF trained with the extra priors. I am also slightly concerned about the quantitative evaluation. Since a different CNN is trained for each baseline, it seems that baselines with larger image FID will be further out of distribution when testing the CNN on the ground truth test images. How do you know that errors in the predicted test depths are a result of bad depths from the baseline versus a distribution mismatch on the CNN input?

c) GOF focuses the evaluation on forward facing faces. It is argued that the smaller camera baseline causes the compared methods to “diffuse” the geometry. GRAF and pi-GAN also test on a 360 scene of synthetic cars and appear to recover fairly good geometry. For scenes with larger baselines is GOF still necessary or do existing approaches work well enough?

d) The conversion of the network to output alpha instead of density forgoes the variable dictating the distance between samples (delta). While the samples are uniform, the distance does change during training because the sampling interval shrinks. Is this something that was considered and is it an issue for this model?

NIT:
LN 177 “a” -> “at”
LN 222 “ms = ? times”
limitations_and_societal_impact: This method is more restrictive in the types of scenes it can represent as it assumes each ray intersects with a single surface. This should be listed in the limitations.
review_title: null

final_score: rating: 7: Good paper, accept
review_content: summary: The paper suggests a new generative model for 3D-aware image synthesis by generating radiance fields with an underlying 3D occupancy representation. This choice of representations allows the generative model to learn surfaces, which are rendered to view consistent images from arbitrary poses using volume rendering. During training, the rendering is performed using samples from a gradually shrinking sampling region around the surface, which encourages the surface to be more compact. The paper demonstrates the method's ability to synthesize images with 3D consistency and further presents their matching shape and normal.
main_review: (A) Those are the main strengths and novelty I find in the paper:
1. The 3D surface representation results learned from unposed images are pretty impressive, based on the fact that they are learned in an unsupervised manner.
2. The paper is well written and has a good flow. The related works section presents the method's origin story and binds the reader to the relevant area. Also, in the method section, the authors give the background for the neural radiance field naturally and easily.
3. In the case of representing object surfaces, the motivation for more concentrated volume densities is evident. Therefore, the use of occupancy representations, which could inherently ensure deterministic surfaces, is good and intuitive for the task of generating compact surfaces (like the presented faces and cats). 
4. I appreciate the presented measure of the weights' variance, as it indicates the method compactness compared to the other baselines.

(B) Following are the weaknesses I find in the presented method and the concerns which I would like to be addressed by the authors:
1. The need for compact surfaces is clear for objects surfaces (like faces and cats), but it is not entirely trivial for me why the previous works are more prone to predict diffuse surfaces. Namely, why directly training equation 2 (which is the optimization that the basslines GRAF and pi-GAN follows) fails to maintain the surface compactness? The authors suggest that it may arise from the shape-color ambiguity, yet I see this ambiguity as another existing problem for other geometrical defects that can exist even with compact surfaces. Hence, my biggest concern that arises is the method's key contribution compare to its basslines. There are two different aspects of the presented method compared to its baseline: the 3D representation and the sampling procedure. My question is, which one of those is "responsible" for the surface compactness? Is it the sampling procedure around the surface that is shrinking and facilitating the weights' concentration, or is it the underlying surface representation? I would like if the authors can suggest an adequate ablation study to answer this concern or give a better intuition for why the baselines tend to diffuse densities. 
2. The authors present the evaluation of the generated images as done in standard GAN's image models, but I find the generated surfaces' evaluation to be insufficient. Since the paper's central claim is better modeling of the 3D representation, the evaluation should support that claim. I find the geometric evaluation to rely on the correctness of the CNN to predict depth maps. Nonetheless, if I understand correctly, the depth maps from the presented method, and the two other baselines are taken as the expected depth, not according to some level set as presented in the figures. A better evaluation would be to show that the generated geometry preserves similar geometry information as in the data set (e.g., mean curvature, mean geodesic distance of random points, faces are and so on). I suggest the authors present other possible evaluations for the underlying geometry or give a better motivation for their proposed evaluation.
3. I believe the authors have an error in equation 9 (the opacity regularization) where each element should be multiplied by its probability, namely: p*log(p)+(1-p)log(1-p). This will give the entropy of the binary probability alpha, and minimizing this regularization aims to drive each sample on the ray to either 1 or 0. If there is no error here, I would like to get an explanation or intuition regarding this formula.
4. Regarding the presented ablation study: it is not clear from the text what is the corresponding experiment for each image in figure 6. Moreover, the ablation should check the contribution of each component (regularization prior) separately. As for what I understand, the authors include results of either with and without both priors. I would appreciate it if the authors can supply an ablation of removing one regularization at a time, to determine the effect of each regularization.

(C) Some other minor questions and suggestions to the authors:
1. I wonder why it seems there is a problem with modeling the eyes geometry (more noticeable in the CelebA). I suspect it relates to the fact the light field is a bit more complicated to present, but I would like to hear the author's opinion.
2. How did the authors extract the shape (shown in the figures) from the pi-GAN and GRAF methods? Namely, how the level set was chosen?
3. Line 150 says that the density varies between -10 to 50, but the physical density should be positive.
4. I suggest the authors present the concrete differences for the other baselines (pi-GAN and GRAF) in the context of the radiance field modeling and sampling procedure. It would help to understand the authors' contribution and novelty compared to existing baselines.
limitations_and_societal_impact: The authors did address the main limitations of the method and its potential social impacts. However, those are mentioned in the supplementary only. I suggest the authors move them or add a short discussion in the main paper on the basics limitation and societal impact. Lastly, I appreciate the author's comment regarding the potential environmental effects.
review_title: null

final_score: rating: 7: Good paper, accept
review_content: summary: This paper proposes a new framework for generative radiance fields, along the lines of GRAF and pi-GAN, but the authors focus on generating cleaner surfaces using an approach that is closely related to UNISURF. This method makes sense, and the generated shapes look a lot cleaner and less noisy than those generated by pi-GAN and more expressive than GRAF. 
main_review: Paper strengths:
+ generative 3D-aware methods are very interesting and challenging, the proposed framework introduces a surface-centric approach which is new
+ the qualitative results, especially the surfaces (i.e., the focus of this work), look really good - a significant improvement over previous work

Paper weaknesses:

- some missing related work:
-> Wang et al., "NeuS: Learning Neural Implicit Surfaces by Volume Rendering for Multi-view Reconstruction" as a concurrent method to UNISURF, which can be optionally cited
-> Yariv et al., "Volume Rendering of Neural Implicit Surfaces" as a concurrent method to UNISURF, which can be optionally cited
-> Sitzmann et al., "Scene Representation Networks", NeurIPS 2019 as one of the very first 2D supervised neural implicit representations
-> Kellnhofer et al., "Neural Lumigraph Rendering", CVPR 2021 as one of the more recent and SOTA surface-based neural rendering frameworks

- method: the proposed method makes sense, but seems like a relatively simply combination of GRAF and UNISURF; the manuscript would very much benefit from a specific and detailed discussion on what's similar and different between the two

- quantitative evaluation: in table 1, GOF outperforms pi-GAN only marginally; that may be okay, because the focus here is on the surface; however, I was surprised when I realized that the FID scores reported here for pi-GAN on CelebA and Cats are 15.8 and 17.6, respectively, whereas they are listed as 14.7 and 16.8 in that paper, which are both lower than the proposed method

- qualitative results: the cats generated by FOG seem considerably smoother and less realistic than those of pi-GAN

Overall, my assessment is that this submission takes some of the important ideas from UNISURF and applies them to a different problem setting, i.e. that of unconditional image generation. It seems very interesting that the very naive UNISURF-like progressive shrinking technique works in a GAN framework; I had my doubts that such an approach would remain stable once the updates became confined to the local region. Therefore, despite this submission being strongly influenced by the core ideas of UNISURF, I think it merits publication.
limitations_and_societal_impact: The authors discuss and evaluate their limitations.
review_title: null

final_score: rating: 7: Good paper, accept
review_content: summary: The authors propose a novel formulation of the 3D GAN problem using hybrid volume-surface approach. The method leads to a more accurate shape reconstruction allowing for more efficient rendering. The reconstructed image quality is also comparable or better.
main_review: [Originality] 

The method closely follows standard 3D GAN pipeline but introduces novel volume-surface formulation of the representation. The authors explain motivation for the technique and demonstrate its benefits.

[Quality]

The technique is demonstrated adequately and the evaluation is sufficient.

[Clarity]

The exposition is clear. All necessary details for reimplementation appear to be present. The network architecture in the supplemental could be described more visually for intuitive comparison to other methods.

[Significance]

The paper falls within the scope of image/shape GAN methods and shares their potential application scope.

[Questions, Comments and Concerns]

1) The Eq. 4 replaces exponential of sum (of sigmas) by effectively a product of exponentials (as the alphas are computed by sigmoids). These two formulations are mathematically very close. Would the original NeRF formulation (Eq. 2) still be applicable if the exp(-sigma) would be used for mapping into the desired [0, 1] range before thresholding? In another words, is the key innovation the adaptive sampling around the surface alone or is the alpha formulation also necessary?

2) The surface sampling pattern of GOF appears more efficient than that of GRAF. However, the training was still limited to 128x128 px resolution as in pi-GAN [6]. Why is that? Should this be listed in limitations (same as for previous work)?

3) In Sup Fig. 3, the authors demonstrate interpolation in the latent space. However, is the method suitable also for inverse rendering, that is finding a fitting latent code for given output 2D image? Does the surface manifold representation provide robust gradient for such tasks?

4) Minor comments:
- L110: "ensure the compactness of learned object surfaces." - What does this refer to? Size of the trained model? Area of the extracted surface manifold?
- Eq.2 for T_i misses z in the argument
- Figure 3 is very low-res to the point that it is hard to see differences.
- Figure 6 - please label which prior is ablated in which panel.
- The computational resources were promised in the appendix but I did not find them in the supplement.

[Conclusion]

Overall, the methods main contribution is its formulation of the volume/surface representation. The application to GAN problem does not seem to require any particularly unique features when compared to the multiview supervision as demonstrated by the concurrent work [10]. Still, it is a completely novel technique and demonstrating its efficiency in GAN setup is perfectly valid since it generally tends to be harder to train. The method produces superior surface surfaces when compared to previous volume-only based GAN techniques. The image quality is also quantitatively higher although qualitatively the method does trade-off the sampling noise artifacts to sharp boundary artifacts in some images so the visual preference may be highly subjective and vary on case-by-case basis. Weighing all these factors, I lean positive but the authors should correct their limitation discussion by moving it to the main paper and potentially adding new points.

limitations_and_societal_impact: [Societal impact]

The method could be utilized in fast low-cost content production for 3D applications including AR/VR.


[Limitations]

Some of the results trade off blur and noise for sharp edge artifacts that are subjectively more disturbing (see the eye region in the last cat example in the video). Same in Fig 5.

The authors discuss the limitations in the supplemental which does not seem right. The discussion should clearly be in the main paper as it is essential to the message of the paper. The authors admit shape distortions in some difficult areas. The authors also mention "stripe" artifacts in surface based renderings. I am not sure if this is the same artifact that I observed above but I note that the sharp-edge artifacts are equally present in both the surface and volume renders (see Fig. 5).

The authors could also discuss the resolution as a potential limitation and note that the quality of the generated images is nowhere near to what 2D GANs such as StyleGAN2 can produce (which is true for all exisiting 3D GANs).
review_title: null

Meta-review: The paper received reviews from four expert reviewers in the community. All reviewers agree that the proposed hybrid volume-surface approach is novel in the context of 3D generative modeling. In the initial review, there were some concerns regarding the qualitative evaluation (reviewer mS1W), results on synthetic cars (reviewer mS1W), and evaluation on geometry (reviewer 3D7c), ablation on the regularization prior (reviewer 3D7c). After the rebuttal, the reviewers were convinced by the additional results as well as clarification. The AC reads the reviews, rebuttal, and also agrees that all the concerns raised by the reviewers have been properly addressed with the additional explanation and experimental results. The AC thus recommends to accept and encourages the authors to revise and strengthen the paper by incorporating the suggested changes and additional evaluation. 
Decision: Accept (Poster)