Venue: NeurIPS 2021 Conference

Reviews:

final_score: rating: 6: Marginally above the acceptance threshold
review_content: summary: 
This paper proposes a GAN capable of generating a relightable radiance field when trained on an unlabeled dataset of face images. The relightable radiance field generator is conditioned on the 3D location and a latent code and outputs a volume density field (à la NeRF) and an "albedo" field (I strongly advise against the use of this word; see below) that allows explicit control of the light location. The training is conducted in the GAN style, where the training signal comes from only a discriminator loss. The intuition is that the generator is better off generating a meaningful 3D volume that, when rendered from a random camera view, generates a photorealistic face image that falls into the training data distribution, than generating some intricately-designed volume that gives faces of different identities when viewed from different angles.

The authors demonstrate that the model is able to learn 3D shapes in an unsupervised fashion from just 2D images. The work is mainly compared against pi-GAN, which is shown to suffer from what the authors call "color-shape ambiguities." For efficient rendering, the authors also propose an auxiliary network that predicts the surface location given the latent code and a viewing direction, such that the network can sample around the predicted surface to avoid expensive sampling of unoccupied space.
main_review: The paper explores a cute idea: by additionally modeling lighting, one learns better 3D shapes in an unsupervised way from unstructured 2D images. This gain seems free by exploiting the lighting variation that already exists in the datasets.

The model design is sensible and simple (in a positive sense). The observation that bad 3D shapes do not affect view synthesis in approaches that do not model lighting like pi-GAN is interesting. The shape results look convincing. The insight that the "local shading" alternative underperforms the adopted shading scene is appreciated. The paper is well-written and easy to follow.

Now the drawbacks:

I strongly advise against calling the generated radiance field the "albedo field." Although the authors clarify in the text that this "albedo" depends on the viewing direction and lighting and hence is not the conventional albedo, this makes the paper really hard to read since any vision and graphics person is familiar with the concept that albedo does not depend on viewing directions or lighting. I had to keep reminding myself of the fact that this "albedo" is special. If the properties contradict the two fundamental properties of real albedo, why use it? Maybe something like "pre-cosine radiance" or "unmodulated/demodulated radiance"?

Related to this albedo complaint, the authors applied Lambertian shading to the "albedo" to allow explicit control of lighting. A simpler alternative is directly conditioning the albedo field generation on the light direction also, eliminating the need for a post-hoc Lambertian shading process. While I can see this alternative may not provide direct control over lighting, it warrants an ablation study. If this simpler alternative ends up working better, the authors can consider naming the field something like "light transport field"? See https://arxiv.org/pdf/2008.03806.pdf and https://arxiv.org/pdf/1911.11530.pdf. The current post-hoc Lambertian shading is a bit awkward, without a solid reason for its existence.

In terms of result quality, I'm concerned on the diversity of what the GAN is able to generate. The results shown seem to be suffering from mode collapse. This concern can be easily addressed by showing a video where we vary the latent code and in the meanwhile change the viewpoint and light direction. Then one would be able to judge if the generated results are diverse or not. Not asking for new experiments, but would love to see this in the revision or a future version of this paper.
limitations_and_societal_impact: Yes.
review_title: null

final_score: rating: 7: Good paper, accept
review_content: summary: This paper aims to improve the density distribution of generative radiance field models by generating relitable reflectance fields and using different lightings to shade them during training. The core idea is that artifacts will arise when the generated density distribution ("shape") is unnatural and is getting shaded by different lightings, which should be detected and resolved by a discriminator. Experiments verify that this shading regularization can yield naturally looking shapes induced by the generated radiance fields.
main_review: ---------------UPDATED AFTER BURETTAL

I feel the reponses have addressed my concerns. I recommend acceptance. My suggestion is to add the explanation for view-dependence (to address dataset biases, in addition to non-Lambertian shading) to the main paper.


---------------ORIGINAL REVIEW

Strengths:
+ Simple and interesting idea to improve generative radiance fields.
+ Novel in the context of generative radiance fields.
+ Clearly written.
+ Straightforward implementation with good qualitative results that achieve the goal.

Weakness:
- My major concern is on the unconstrained nature of the "albedo" fields. It is dependent on direction and lighting, making it unconstrained, and possibly unstable, to regularize the density distribution. The "color-shape" ambiguity is still not addressed. In the proposed formulation, the ambiguity can still be baked into the dependency on direction and lighting, i.e., instead of optimizing a good shape, the model might choose to optimize a sophisticated but working A(r,z,d,mu) function. For example, if a generated face shape has no nose, to have a plausible appearance that is multi-view consistent, the model can either (1) generate a nose in the shape and generate a constant albedo in A(z), or (2) keep the "no nose face" and hack the A(r,z,d,mu) function by giving high albedo to left nose when light source is on the left, and low albedo when light source is on the right. This is not seen in the shown images but this might happen to some extent, or might happen to some random seed. Showing results averaged over multiple runs could alleviate this concern and I might increase my rating.

- I understand that the flexible A() function is used also to compensate the simplified lighting and reflectance model. But as for the dependence of view angle, why not use a simple specular part in the BRDF? Just like the test in Figure 8c.
limitations_and_societal_impact: The authors have discussed the mentioned issues to some extent, but not fully addressed them. Please see above comments.
review_title: null

final_score: rating: 7: Good paper, accept
review_content: summary: This paper proposes a new method to train generative radiance fields that can achieve more accurate 3D shape reconstruction. The key idea is that instead of directly synthesizing the images from different viewpoints through volume ray tracing, which may suffer from color-shape ambiguity, it synthesizes albedo and normal and then renders the image with different lighting conditions. This new method requires the synthesized albedo and normal to be able to render realistic images under various lighting conditions, providing an extra regularization for shape reconstruction. To further improve the rendering speed, it uses an extra 2D CNN to predict a depth map from latent code and camera poses so that it only needs to sample near the surface in volume ray tracing. Experiments show that the proposed method greatly improves shape reconstruction with similar image synthesis quality compared to state-of-the-arts.
main_review: I will list the strengths of the paper and some minor suggestions that I believe may help improve the paper. My current evaluation of the paper is very positive. Authors may prioritize answering other reviewers' questions in the rebuttal. 

Strengths:

1. The major contribution of the paper is clear. It proposes the multi-lighting constraint to help regularize the 3D shape reconstruction when training generative radiance fields, which is intuitively reasonable practically effective. In my opinion, this is an interesting idea and may inspire future research in the related fields.

2.  The paper is clearly written, well-organized, and easy to follow. The citations is complete and appropriate. 

3. The experiments are comprehensive and convincing. It clearly supports the major contribution of the paper that the depth prediction quality can be improved by the multi-lighting constraint. Moreover, ablation studies also demonstrate that all minor design improvements such as depth prediction to reduce sampling number and lighting parameters as inputs can further improve the image synthesis quality. 

4. Authors clearly summarize the limitations of the current method, which may help researchers to build new frameworks based on current results. 

Minor suggestions:

1. When first reading the paper, I was confused by how GAN inversion is done in practice. I later found the details in the supplementary but authors can consider adding a reminder pointing to the supplementary in the main paper. 

2. When discussing the intuition of adding the multi-lighting constraint, authors can consider connecting it to photometric stereo, which can achieve accurate normal reconstruction from images taken under different lighting conditions. This may make the argument more convincing and easy to understand. 

3. One way to improve the rendering quality is that instead of using sampled lighting conditions, authors can use captured real HDR environment maps. This may minimize the domain gap and render more realistic results. It may be even more useful if authors hope to reconstruct specular highlights in the future. 

4. One minor thing to check is if both the lighting and albedo are in the sRGB space or linear RGB space. If they are not in the color space, authors may consider using gamma correction to align them. 
limitations_and_societal_impact: Authors mention in the discussion paragraph that one limitation is that the current method does not handle spatially varying material. I cannot see any other obvious limitations that need to be mentioned in the paper. I cannot see any potential negative societal impact either.
review_title: null

final_score: rating: 4: Ok but not good enough - rejection
review_content: summary: The paper proposes a novel generative model for neural radiance field, where the key idea is to synthetically add shading effects based on random lighting configurations. Shading will make images unrealistic when the geometry is corrupted. Therefore, the system gets trained to generate accurate shapes in its implicit radiance field representation. The system makes convincing results both qualitatively and quantitatively.
main_review: The idea of the paper is very simple and intuitive (adding lighting effects and utilizing discriminator to discover accurate geometry). I am not familiar with the complete literature but if no papers have shown this idea before, this idea could be an interesting one for the computer vision community but not for the machine learning community. I am curious if other reviewers spot any existing works with similar ideas. Given extensive GAN literature, there may exist similar ideas somewhere.

Empirical evaluations (both qualitative and quantiative) look convincing, where experimental results also support the claims qualitively and quantitatively. However, they are shown only for human faces.

I am stuck on the overall recommendation but am leaning towards negative. The first reason is the merit/impact to the machine learning community. The strength of this paper is the idea which is for the computer vision community. The technical contribution is weak, as there is not much to learn from the paper. The second reason is that results are only presented for faces. This is very critical. We already have very good approaches that turn single image into 3D face models. This paper will not bring any impacts. The idea of this paper will really shine if this idea works on more variety of object categories. With those reasons, I will vote for rejection.
limitations_and_societal_impact: I did not find texts addressing the limitations and societal impact. The check-seat has a comment saying that "see the supplementary document" on this point, but I do not see relevanat texts.  Deep fakes are relevant societal impac.
review_title: null

Meta-review: The submission initially received mixed reviews. After the rebuttal, all reviewers became positive (though reviewer YNf7 didn't update the score).  The AC agrees with the reviewers that the proposed idea is neat and well-executed. The authors are strongly encouraged to address the remaining concerns in the camera-ready version. This includes adding the results in response to reviewers TVTi, mEVj, and YNf7, addressing the concern on view dependence, among others.
Decision: Accept (Poster)