Venue: NeurIPS 2022 Conference

Reviews:

final_score: rating: 7: Accept: Technically solid paper, with high impact on at least one sub-area, or moderate-to-high impact on more than one areas, with good-to-excellent evaluation, resources, reproducibility, and no unaddressed ethical considerations.
review_content: summary: This paper introduces new techniques in training NeRF-based GANs. The proposed techniques addresses the patch-based trianing issue, which blocked privious works to directly generate nerfs that renders into high quality resolution images. As a result, previous works lacks in training time, in addition to lack of view consistency because of image-based upsampling. The proposed method utilizes a hyper-network to generate filters for the patch discriminator under different resolutions, and utilizes an annealed beta distribution for samping the random scale for patch discrimination. Both disign choices are well founded and validated by the quality of the result. 
strengths_and_weaknesses: Strengths:
Originality:
This paper proposes new solutions to patch-based discriminators in training NeRF-based GANs. The proposed techniques are simple to implement and effective, as demonstrated by the results. In addition, the authors validate the quality on new datasets(megascan plants&food), which further demonstrated the geometric quality of the generated nerf.

Significance:
I think this paper would be of significance to communities interested in generative methods centered around NeRF. Though it can be said that the proposed technique is simple, but it is more a merit than a drawback. I think those solid gadgets are keys to make things work better and better.

Quality:
This paper conducts enough experiments to support the claim, and justifications of the design choices make sense. I find the evidence in the paper convincing and the webpage of results are quite representative.

Clarity:
This paper is well written and easy to follow. Readers should be able to reimplement the method based on the inforamtion provided.

questions: 1. Are all baseline results generated using view-independent NeRF as well?
2. As suggested by the paper, using the beta distribution with the improved annealing improves the stability and convergence speed. Is that also the case for the hyper-network as well? Is the method robust to train given the hyper-generated patch discriminator?



limitations: The authors addressed the limitations and potential negative impact adequately.
review_title: null

final_score: rating: 5: Borderline accept: Technically solid paper where reasons to accept outweigh reasons to reject, e.g., limited evaluation. Please use sparingly.
review_content: summary: The paper proposes a non-upsampler-based 3D-aware generator. To train the non-upsampler-based generator, the paper presents a patch-based discriminator. In contrast to previous patch-based schemes, the author improves patch-based discriminators in two ways. First, they adopt location and scale to modulate the discriminator (in particular, the feature maps).  Second, they modify the patch sampling strategy based on an annealed beta distribution to stabilize training and accelerate the convergence. Furthermore, they introduce two plants and food datasets, in order to evaluate 3D-aware generators. 
strengths_and_weaknesses: Strengths:

The paper is well-written, and the experiments are comprehensive.

The patch-based discriminator studied in the paper is important for differentiating high-resolution images and would be a good contribution to the community.

Weaknesses：

1. The paper's main contribution is to propose a new patch-based discriminator, which is particularly useful when the generator can only render partial pixels. I would like to know whether the generator synthesizes all images or only a subset of pixels during training. Since the generator adopts a tri-plane representation, it is memory efficient. If the generator is already able to synthesize all pixels without running out of memory, it seems unnecessary to adopt a patch-based discriminator. 

2. I think patch-based discriminators are valuable. A more challenging setting can demonstrate the method's effectiveness, namely training pi-GAN on high-resolution images. Since pi-GAN is memory-intensive, rendering all pixels for high-resolution images is impossible during training. Maybe this setting is more suitable for demonstrating the efficacy of the patch-based discriminator. 

3. If possible, the authors should provide the results of the full-resolution discriminator. Since the full-resolution discriminator can see the global image, its performance can be used as an upper bound for the reference of patch-based discriminators.

4. In equation 4, a simple baseline is $lerp[1, r/R, 1 - min(t/T, 1)]$. If possible, the author should compare this setting to the proposed beta strategy. 

5. Line 204, how to use pose supervision for the discriminator? The author can briefly introduce the details. 



questions: Please see the weaknesses.
limitations: Yes, they did.
review_title: null

final_score: rating: 6: Weak Accept: Technically solid, moderate-to-high impact paper, with no major concerns with respect to evaluation, resources, reproducibility, ethical considerations.
review_content: summary: This paper proposes two important techniques to train 3D generative models from 2D supervision.  

Existing works adopt nerf to render 3D radiance fields into 2D images with a large memory cost. As a result, they fail to train the discriminator with high resolution and more often render 2D images with low resolution and generate high-res images by 2D CNN up-sampling. However, it brings strong view-inconsistent problems. 

This paper explores patched-based discriminator. By rendering small patches, the generator can directly synthesize high-res images without any upsampling tricks. To improve the quality of patch discriminator, the paper proposes 1) a location- and scale-aware discriminator and 2) a beta distribution patch sampling strategy, both showing good improvement. It also produces quite nice and view consistent results and outperforms the baselines.

strengths_and_weaknesses: Strength:
The patch-based solution to solve the view-inconsistency problem of nerf-based generative models is reasonable and the two proposed techniques are quite simple but seem to work well.
The final FID scores are better than all the baselines with much less training time needed, and the results looks quite nice!

Weakness:
It is very weird why the strategy cannot be applied in 2D cases, which raises concerns of generalization.

questions: I have several questions about the implementation details.

1)	As for the rendering equation, did the authors adopt the original ray-based rendering equation (in original nerf paper), or light cone based one (in mip-nerf?) If I didn’t understand wrong, it seems the authors choose the former. In such a case, would it produce anti-aliasing problems when rendering a 64x64 patch with the scale to be 1(let’s say the original image is 256x256)? Adopting the mip-nerf seems to be the straightforward solution. Could the authors discuss this part?

2)	How do you make real patch data for training the discriminator?  Are you cropping the real images with specific scales, then resizing them to be 64x46? I wonder do you also consider the antialiasing problem here?

3)	What’s the relationship between patch size and final image size? E.g., In table2 it seems for 512x512 image size, 64 patch size seems to be the best. Does it mean that for 256x256 image size we should change to 32 patch size? 

4)	It is good to see that the proposed generators can generate up to 512x512 images. I wonder what's maximum. Can it be scaled to 1024x1024? 

5)	For face and cat, the camera poses are limited to the front views.  I wonder can you do large view changes, e.g., rendering side faces, or rendering from overhead? I want to know whether the radiance field will remain view-consistent for unseen training views. On the other side, I recognize that if the training images cover 360 degrees of the object, then the radiance field doesn’t have such issues.

limitations: I appreciate that the authors honestly report that the proposed techniques don’t work in 2D cases. However, I treat it as a limitation of the paper since it undermines the generalization of the techniques.
review_title: null

final_score: rating: 5: Borderline accept: Technically solid paper where reasons to accept outweigh reasons to reject, e.g., limited evaluation. Please use sparingly.
review_content: summary: The authors propose a framework to generate 3D images from latent code. Authors analyze in details the drawbacks of current methods, based on upsampling strategy that could affect multi-view consistency or rely on low resolution rendering. They came up with a patch based strategy similar to GRAF to improve the performance. The idea is to condition the discriminator with a hypernetwork to better make sense of the output of the convolutional filter at multiple scales. Additionally, they propose a beta distribution when sampling the scale, improving the overall convergence
strengths_and_weaknesses: +++ Relevance. The work is super relevant for the community, with a lot of possible applications.

+++ Clarity. The paper is very well written and well motivated. Authors describe in details all the aspects of 3D generators, explaining their drawbacks and how to mitigate those. In particular, I really appreciated in the supplementary material the attempts at 2D generation and all the failing experiments the authors reported. This could be very useful for the community as GAN methods usually rely on a substantial amount of trial and errors before getting to the right combination of architecture and loss functions.

--- Novelty and Experiments. Whereas this paper has many good aspects and I'd like to see it published, I have a few doubts regarding the overall novelty and the missing comparisons.

In particular, the method "borrows" multiple aspects from EG3D and GRAF, but comparisons are very limited. Indeed EG3D outperforms this method substantially in terms of FID, authors hint that the upsampler leads to a better FID score but worse multiview consistency (l240), but they do not provide any example of this.

Similarly, the method borrows the patch-based, multi-scale discriminator idea from GRAF, improving with the conditioning from a hypernetwork, but I am not sure there is a direct comparison with GRAF?

There are no examples of inferred geometry. Ideally authors would also show the inferred geometry and qualitatively compare with EG3D to prove that the method leads to better multi-view consistency.

questions: I'd like the authors to comment on the limitations I described in the Strength and Weakness section. In particular, why one could not simply rely on the EG3D triplane representation + GRAF patch based discriminator to achieve the same results proposed here. Is there any experiment showing that it's not sufficient?

Also, why one should prefer this method instead of EG3D? Seems to obtain better quantitative metrics (and qualitative metrics are not shown).
limitations: Yes, Limitations and Ethical concerns are described in the supplementary material
review_title: null

Meta-review: metareview: The reviewers found the method simple and effective and considered it a contribution of interest to the community. Claims are well supported by experiments and design choices have been validated. The paper is well written. Furthermore, the authors provided highly detailed responses to all questions by reviewers, which creates confidence that reviewers' remarks will be addressed in the final paper.
confidence: Certain
Decision: Accept