Venue: NeurIPS 2022 Conference

Reviews:

final_score: rating: 7: Accept: Technically solid paper, with high impact on at least one sub-area, or moderate-to-high impact on more than one areas, with good-to-excellent evaluation, resources, reproducibility, and no unaddressed ethical considerations.
review_content: summary: The paper introduces a 3D-aware image generation network that usess a Plenoxels-style voxel grid [1] as its scene representation. The complete network architecture consists of a sparse 3D CNN generator with novel upsampling layers that ensure sparsity of the upsampled densities. The full resolution voxel grid has size $128^3$ and stores RGB+density values for the occupied voxels. 

In a parallel branch, there is a 2D GAN (following StyleGAN v2) that generates the scene background.

The sparse voxel grid is rendered at the required output resolutoin using conventional volume rendering techniques. The foreground is post-processed by a shallow (two hidden layers) refinement CNN, whose output is then composited on the the background image. 

At training time, the resulting image is passed to a discriminator based on StyleGAN2. The authors describe several regularizing losses, including a novel sparsity term on the voxel densities.

The authors report both quantitative and qualitative comparisons to state-of-the-art 3d-aware GANs and demonstrate results that demonstrate a high degree of geometric consistency across viewpoints. 

[1] Yu et al. Plenoxels: Radiance Fields without Neural Networks. 2022
strengths_and_weaknesses: This is a strong paper with an original technical contribution in the sparse CNN generator and the use of a sparse RGB-density voxel grid as the representation for a 3D-aware GAN. The voxel visualizations in the paper, as well as the animated sequences in the supplemental video, demonstrate the geometric plausibility of the generated representations.

The paper is very well written and the discussion of related works is thorough. The comparisons to related work are quite comprehensive, and a key ablation study is included.

A particularly exciting aspect of the method is the very fast rendering times (after a slower initial generation of the voxel grid), which could allow this technique to run at interactive framerates on low-powered platforms.


--

Even though the results exhibit high geometric consistency, the visual quality of the generated still frames does not quite match that of some other methods. This limitation is acknowledged in the paper.

My main critique is around the description of viewpoint dependence. Plenoxels stores spherical harmonic coefficients in the voxels, while the proposed method seems to store RGB values without directional dependence (Eq. 3.). This is a quite significant departure from other NeRF-inspired representations, and should be made more explicit in the discussion of the method (Section 3.1).

It is somewhat surprising that the foreground generator is conditioned on the camera pose $\xi$, since the volumetric grid can be rendered from arbitrary viewpoints $\xi'$, even those that it was not conditioned on. Could the authors clarify for which results in the paper $\xi = \xi'$ ? Would it be a viable ablation to compare two animated sequences, one with $\xi = \xi'$ at each frame, and one where the generator was invoked only once?

questions:  - See previous section for questions about voxel & generator view dependence.
 - What is the speedup gained from using sparse convolutions at grid sizes > $32^3$?
limitations: The authors are clear about the limitations of their work. Societal impacts are adequately discussed.
review_title: null

final_score: rating: 5: Borderline accept: Technically solid paper where reasons to accept outweigh reasons to reject, e.g., limited evaluation. Please use sparingly.
review_content: summary: Authors propose an approach to generate high quality images from a latent code that are multi-view consistent. The intuition is to replace the MLP that is used to parameterized radiance fields, with a hybrid approach that relies on sparse 3D convolutions to speed up the rendering time. Additionally, they propose a way to disentangle the foreground and background.
strengths_and_weaknesses: ++ Relevant problem for the community. Generating high quality renderings with an adversarial approach is an unsolved problem despite the recent progress

++ Clarity: the paper is overall clear, authors put lots of effort to describe every details of the approach. They sufficiently discuss limitations and ethical concerns.


-- Novelty, Contributions and Experiments. Whereas the general idea is interesting, the first thing that comes to mind once reading this paper is: why is this approach better than EG3D [4]? Indeed, they tackle the same problem and I would argue that the results of EG3D are actually better. Quantitative comparisons that author report seem to be aligned with this, and qualitative experiments are not showed (I would encourage authors to report those). According to authors, the main difference between their approach is explained at l142-l143, which is "In contrast to recent works, we do not use a coordinate-based MLP..". While this is true, it is not demonstrated that the proposed method is actually better. 
questions: The main question is around novelty and experiments/comparisons. I'd like authors to convince me that this paper introduces some different capability compared to [4] or that they solve the problem better. In the current form, I do not think the paper is doing a good job at convincing the reader that this method is pushing the state of art.

---
Post rebuttal: Authors made the fair point that [4] was concurrent work and there was no code release at the time of the submission. Hence I am leaning towards acceptance.
limitations: Yes, discussed.
review_title: null

final_score: rating: 6: Weak Accept: Technically solid, moderate-to-high impact paper, with no major concerns with respect to evaluation, resources, reproducibility, ethical considerations.
review_content: summary: The paper proposes a novel 3D GAN architecture for rendering of 3D consistent objects supervised by collection of single-view images. The key innovation is the introduction of sparse voxel grid as a more computationally efficient representation when compared to the most commonly used MLPs. The authors describe how sparsity is enforced and used to reduce memory constraints of their approach. As a result, they can render medium-sized images from their voxel grid directly without need to explicit upsampling. The validation shows the method produces better images at lower computation cost than previous work and it is in some respects competitive with more recent methods published in parallel.
strengths_and_weaknesses: 
[Originality] 

1) The method combines an established concept of NeRF-based 3D-GAN with previously described sparse voxel convolutions. The authors further describe several key tricks that are required for successful and efficient training. Many of them are inspired by similar work, but at least the loss L_D seems unique to this publication. Overall, the technical contribution is incremental but, nevertheless, it leads to a significant improvement upon SOTA in the target application (here I ignore the parallel work).


[Clarity]

2) The exposition is very clear, easy to read and I have not discovered many typos or grammar issues.

3) I find problematic the way how parallel work is discussed. GRAM, StyleNERF, and EG3D are all parallel work so they do not count against contributions of this paper. I appreciate that the authors tacked the task of discussing their properties and that they included them in their result tables. However, I am not entirely sure all conclusions that they have made are fully substantiated. For instance, L264 seem to suggest that EG3D is less view consistent than the proposed method but no such effect is shown. Both methods use internal 128px representation for the radiance field and therefore, any further processing increasing the output image resolution is in principle similar for both. The authors also argue, that the advantage of their method is the possibility of faster rendering after the first frame (Table 3). However, the same could be said about EG3D as it also could precompute the StyleGAN representation and then just sample the features the same way the new paper does. In general, I find these over-extended assumptions about parallel work both not properly validated and also unnecessary. As these work are not counted as a previous work, there is no need to prove that the new method is in all or any respects better. The authors should rather focus on properties of their own work and show why it is excellent. Since the general architecture and performance appears quite similar, focusing on the possibilities of the explicit voxel representation seem like a clear goal.

[Validation]

4) The method is evaluated on common datasets and the results are clearly better than previous work and comparable to parallel work. The results are limited to 256px resolution with is also comparable to recent methods.

5) There are no quantitative experiments done to validate the claim of better geometry and 3D consistency (L74). The authors could also attempt to measure depth and pose accuracy as done in previous work [4]. Since there is still a small CNN used for the image refinement it does not seem obvious that the method is perfectly view consistent.

6) The extracted shapes appear quite noisy (Fig. 5). 

7) The authors only ablate the loss L_DV and only in terms of sparsity. Neither of the losses that are used for regularization (there are 4 in total) is evaluated in terms of end-to-end performance impact.

8) The training is extremely costly even compared to MLP-based NeRF GANs (L224).


[Significance]

9) The paper tackles a relevant problem of 3D representation generation and it does take a path distinct from other work. While it is not clear whether the method actually is clearly better than some of the parallel work, it does clearly beat the previous baselines. The voxel representation may have practical benefits that have not been mentioned in the paper. E.g. I can imagine that they would be easier to edit in a 3D modeler. However, no such application has been shown.


[Conclusion]

10) While I see some issues with the exposition, the paper is of a high quality overall. The discussion of the performance and relation to other work can be addressed. The bigger concern is the small scope of the validation. While the most critical image synthesis quality is directly evaluated on common datasets, this leads to a total of three numbers reported in Table 2. The 3D consistency, design ablation (with exception of Table 1) and the shape quality are not directly validated in a quantifiable way. That being said, I do not suspect such experiments to significantly change the narrative of the paper so I am still inclined towards acceptance, especially if the authors can improve upon these aspects in the final version.
questions: 11) If voxels are pruned based on visibility in "a" rendered view (L167), cannot they be later missing after occlusion changes due to a new viewpoint at test time?

limitations: 12) The authors have mentioned the performance may drop is scene is more complex and pruning is not very effective. Other limitations I suspect is the lack of geometric smoothness in extracted shapes and the high training cost.
review_title: null

Meta-review: metareview: It is valuable now to introduce this technical idea, even if the results do not quite match existing methods.  Future work building on this idea may well do so, and it would impede the progress of the subfield to demand both the new idea and SOTA results.  

The rebuttal takes on extra work building on the reviewers' suggestions, which gives confidence that the final paper will be of high quality.

At the same time, the primary criterion for oral presentation is importance to the community as a whole, so the relatively narrow scope (3D computer vision) would really require more world-leading results, and possibly demonstrations of a wider set of applications in order to alert practitioners in adjacent subfields.


confidence: Certain
Decision: Accept