Venue: NeurIPS 2021 Conference

Reviews:

final_score: rating: 6: Marginally above the acceptance threshold
review_content: summary: This paper proposes a method for joint shape, BRDF and illumination estimation using a NeRF-like pipeline. Compared with NeRF, the proposed method can also solve for illuminations, and it  outperforms NeRD on both quality and efficiency due to a novel network query-based pre-integrated lighting approach.
main_review: ### Strengths:

1. The idea of network query-based pre-integrated lighting is novel, and combining it with smooth manifold auto-encoder (SMAE) in a differentiable  pipeline is non-trivial.
   
2. The proposed Neural-PIL achieves the state-of-the-art in both quality and efficiency.


### Weaknesses:

1. The flowchart in Fig. 3 can be improved, e.g., I did not see Light-SMAE in this figure. Moreover, the training strategy in L251-257 seems complex, e.g., why freeze BRDF-SMAE’s decoder only, any experimental support? It would be good if the authors can elaborate more on this in rebuttal.
   
2. The SMAE ablation study (in supplementary) provides nice visual comparisons, however it lacks quantitative comparisons. In Fig. A1, for example, although Neural-PIL shows finer details than SGs, its color looks blueish (also in Fig. 6), therefore it's hard to tell whether the PSNR/SSIM are better. I'd like to hear some explanations on this in rebuttal.

limitations_and_societal_impact: See Weaknesses above.
review_title: null

final_score: rating: 6: Marginally above the acceptance threshold
review_content: summary: The paper presents a method to reconstruct geometry and BRDF from a set 
of multi-view images take under various illuminations. It learns a low 
dimensional manifold for illuminations and BRDFs and use these data priors 
to regularize the estimation. It also applies a network to predict the 
integral of light for efficient rendering under environment maps. It 
shows results on both synthetic and real scenes. 

main_review: 
### Strengths:
1. Using networks to predict the integral of lighting is novel and 
reduces the computation cost for rendering under complex lighting conditions.

2. Using auto-encoders to learn a data manifold for BRDFs and lighting is interesting 
and helps regularize the prediction.  

### Weakness: 

1. The results seem to be incomplete. The paper says it has results on    
4 real scenes, however, only results on the cape scene are showed. 
Moreover, for the cape scene, only several images from different views are shown.
It is desired to show more video results on more diverse scenes under different illumination conditions 
and viewpoints, and include comparisons to baselines. Similarly
I cannot find the view synthesis results on Ship, Chair, and Lego scenes.  
From the current results, I cannot judge whether the proposed method can 
generate convincing results on real scenes. 

2. In terms of comparison to NeRF and NeRD in Figure 7, it is confusing to me 
why the proposed method would outperform NeRF even for view synthesis tasks. 
In fact from the video, I think the results of NeRF has more details while the 
proposed method generates blurry renderings. It is desired to show quantitative 
and qualitative results on the NeRF dataset to further validate this claim. 

3. In the novel view and relighting results in the video, ground truth should 
be included. The current results are not fully convincing, for example, there 
are clear artifacts on the globe scene. 

4. It is not clear to me whether the proposed method requires images under 
varying illuminations. Requiring images under different environment 
lightings will make the acquisition very difficult and make the method impractical.   
How does the performance change with images under fixed and varying illuminations?

5. In terms the comparison on lighting, while the paper claims Neural-PIL has 
better performance, from the results in Figure 6 (second row) and the results in 
supp (Figure A1), I think the Neural-PIL illuminations often suffer from incorrect 
colors and highlight positions. Therefore it's not convincing to me that Neural-PIL 
is definitely better. To better show this, a quantitative comparison should be included.

6. The paper uses an auto-encoder to regularize the material estimation. While the paper
does ablation on different loss terms in training the auto-encoder, a more important 
and intuitive comparison is how does the full pipeline work with and without using 
the auto-encoder? How does the method work by directly predicting the BRDF coefficients
without the auto-encoder? 

7. Another important component to validate is the accuracy of the PIL network. It is not 
certain to me that such a network can be trained effectively without direct supervision.
A comparison between the predicted integrated lighting and the GT integral on synthetic 
data would be helpful.  

Overall while I like some components and design of the paper, I think the paper needs 
significant improvement in terms of visual quality and experiments to further validate the 
effectiveness of the method. 
limitations_and_societal_impact: Please see main review.
review_title: null

final_score: rating: 6: Marginally above the acceptance threshold
review_content: summary: This paper proposes to learn a neural implicit representation from multiple images under different illumination that can separate complex SVBRDFs and lighting, which can be used for applications such as view synthesis and relighting. There are two major contributions: one is two learned manifolds to constrain BRDF and lighting parameters that can better preserve high-frequency information, the other is an MLP based network called Neural-PIL that can render intensity through a single forward pass based on the new lighting and BRDF representation. Experiments show that the learned lighting manifold can better recover details of the environment map compared to traditional spherical Gaussian representation, and the proposed method can achieve more accurate view synthesis and relighting results compared to the baseline NeRD. 
main_review: I will list the strengths of the paper and my questions. When listing my questions, I will separate them into major questions that may change my evaluation and minor questions. Authors can focus on major questions in the rebuttal.

Strengths:
1. The idea of using learned manifolds to constrain lighting and BRDF reconstruction is interesting. Given the ill-posed nature of inverse rendering under an unconstrained environment, such constraints are necessary to achieve satisfactory reconstruction results. Besides, the new lighting manifold is shown to better preserve high-frequency lighting compared to spherical Gaussian representation but with much fewer parameters. This new representation may inspire future research too. 

2. It is great to use a pre-filtered environment map to reduce the rendering cost. While this method is popular in the graphic community, it is the first time I see people use that to help to solve inverse rendering problem. 

Major questions:

1. The writing of the paper may need to be improved. Many important details are probably missing. 

a. I have a hard time understanding how the Neural-IPL is trained. Line 250 and 251 do not mention explicitly that Neural-IPL is trained first and fixed through the rendering process. Only when I check the supplementary material, did I understand this part. Authors can consider moving more training details from the supplementary material to the main paper to make the main paper easier to follow. 

b. Line 182: It is unclear to me how the termination point is computed. I guess it is probably done by computing the weighted average based on the transmittance but more details should be included in the paper. 

c. Table A1 in the supplementary only summarizes the network structure of Light-SMAE. It will be good to have similar summaries for other networks. 

d. Line 192: It has been established before that we will only use $w_r$ as an input to render the color, which causes it to be very confusing to say Neural-IPL takes $w_i$ as input here. This is explained later in Line 197 but the whole paragraph may still be confusing to read. We can just say the input is $w_r$. In addition, the $w_i$ in Figure 3(b) can be changed to $w_r$. 

e. It may not be explained in the paper if the illumination of the input image changes, how we should handle lighting code $z_l$. Do we always have different lighting codes for every input image or that is only true when the illumination changes?

2. The experimental results can probably be more convincing. 

a. There is only 1 real example shown in the paper. And for this real example, it is difficult to observe specular highlights. There is no reconstructed environment map for real examples shown in the paper. 

b.  Table b shows that the proposed method achieves more realistic view synthesis results compared to NeRF on their synthetic dataset (Ship, Chair, Lego). Line 312 mentions that it is because the proposed method has more realistic view-dependent effects. It will be very interesting to show these examples in the paper, especially because the reconstructions of NeRF are already quite impressive. This can be very compelling results to show the advantages. 

Other issues:

1. Line 188 to 191 says that Monte Carlo sampling or numerical accumulation is not practical. But if the environment map resolution is only 128 x 256, the computational cost of directly computing the integral is probably lower than the MLP. The main argument for Neural-IPL is probably that $z^l$ provides a good prior, not the computational cost.  

2. It will be interesting to show a comparison of view synthesis and relighting results with and without the BRDF embedding. Since BRDF embedding only reduces the dimension from 7 to 4, I am curious how significantly the results can be improved. 

3.  The normal map in Figure A3 of the supplementary material is wrong. It probably shows the normal map in the world coordinate system instead of in the camera coordinate system. 

Minor questions: 

1. Line 290: are shows in Figure 6 -> are shown. 

While I like the idea of using learned manifold to constrain lighting and BRDF reconstruction, especially the lighting reconstruction, I think the current experiments and writing may have space for improvement. Therefore, I keep my review to be slightly negative. I would like to see  authors' rebuttal and update my review. 


After rebuttal: authors rebuttal solved my questions and I upgraded my review to be positive. Please refer to the rebuttal discussion for details. Thanks authors for addressing my questions.
limitations_and_societal_impact: Authors have listed the limitations quite completely in section 5. Maybe one more limitation to add is that the proposed method may completely rely on the geometry prediction from the coarse network and cannot optimize the geometry jointly with BRDF and lighting. The reason is that the current method only computes color at the surface points. 

I cannot see any potential negative societal impact.
review_title: null

final_score: rating: 6: Marginally above the acceptance threshold
review_content: summary: The method suggests to learn a 3D model of shape, reflectance and light from a set of 2D images. They use a forward model that combines the three factors and invert it by optimizing its parameters to explain the 2D images. The key is in the specific parametrization of the three factors.

The first factor is a classic implicit model of shape. Rays are cast from the views of the 2D input images to find 3D depth. They define it as expected opacity-weighted depth, which might have its problem on (soft) silhouette edges. Mode finding, via Iterated Re-weighted least squares would be still fast and maybe more correct, not resulting in depth values floating in the air.  This said, this part is common.

The reflectance is parametrized as a spatial field of 4D latent codes. Each latent code itself is an embedding of the 7D Cook-Torrence parameters over the MERL database. The paper says the 7D codes are "not constrained enough", but no evidence for this is given. The embedding is pre-trained using a modern smooth auto-encoder.

The illumination is parametrized in a similar way: a smooth auto-encoding of a collection of environment maps into a low-dimensional code.

The work dwells a bit too much on how this is a graat AE'ing, but it is just shopped previous work, so it should maybe be phrased more neutral. 

Now the key innovation is how the two latter factors combine. The direct way is to solve an integral where the BRDF slice is convolved with the illumination. in SH this is a dot product, but at the expense of angular details. Instead, the suggestion is to pick up the realtime-rendering idea of pre-filtered illumination. This allows to get the integral estimate in constant time, after pre-processing the envmap. Now instead of preprocessing the envmap (which would be possible, there is only one illumination per scene and pre-process = pyramidal blur = constant time), authors suggest to use a network that directly returns pre-filtered results. For this idea to be fully appreciated, one would need to look at the two ablations chosen by those familiar with game graphics: pixel basis (MIP maps) and MC directly. This would need to be compared against an NN doing this pre-filtering job. It is a cool idea, but does it do anything in the end? Comparison is only made against SH, which is for the birds to those learned in graphics. The pixel basis ablation is: take the envmap code, decode into pixel basis, MIP, use it as Karris explains with the 7D shading model parameters. The MC baseline is, take envmap code, decode into pixel basis, MC it with the 7D shading model parameters as in any path tracer (eventually, importance-sample). Clearly MC noise hurts human eyes, but does it also hurt SGD convergence? We remember Noise2Noise and other instances were training on unbiased noisy data in the grand scheme of SGD make no difference.

Instead of those ablations, focused on what is proposed, the paper compares to an Arxiv paper and a published alternative by Li et al. Comparison to Li et al is okay, we need to use something, done. Comparison to NeRD is more questionable. NeRD is very similar to this work. It more appears this paper is a fix to NeRD than a true baseline. Most agree, Arxiv papers cannot be held AGAINST a submission. Now authors might agree that not every success over an Arxiv paper indicates something. Otherwise, Arxiv turns into a "strawman repository" and everything is even more confusing then it already is these days.

For results, one would now expect to see something specular. As Fig. 6 shows, this is the particular strength of this approach. Alas, there was none. Results are very small, they are the same data s in NeRD. For relighting, this reviewer is very unsure what to compare this to? The Graphics approach is usually to make a photo under the novel light condition and then see how much the AI could predict it. Here we only see an image.
main_review: See above. This reviewer preferred to merge description of the work and critique in one field in this case.

This reviewer is unsure about what to recommend. As an *increment* over NeRD, it will not pass. It does not analyze the benefits of pre-integration nor does it show results (shiny reflections) where this really contributes. As a *replacement* for NeRD, this is a unique system that deserves publication. There is concurrent work this method appears to be promising to outperform, it is simpler to code and pre-integration not yet understood enough in Computer Vision probably. Authors would need to clarify what is the position, replacement or increment?

If suggestion is a replacement then the writing would need to change and instead of shooting down a NeRD baseline from the "straw man repo Arxiv", the text needs to be reworked and NeRD has to be "dissolved" into a "strong ablation".
limitations_and_societal_impact: No idea
review_title: null

Meta-review: All reviewers were somewhat positive  on the paper after an extensive discussion.

They all agree on the novelty of the neural pre-integrated illumination network (Neural-PIL) as well as the advantages of using smooth manifold auto-encoders (SMAE). 

Authors are strongly encouraged to include the new results they provided during rebuttal in the final version of the paper and to highlight their  method on real images.

Given the current interest in neural rendering, even though the visual improvements over the state of the art are somewhat subtle,  the Metareviewer thinks that the novel contributions of this paper should be present at NeurIPS.
Decision: Accept (Poster)