Venue: NeurIPS 2022 Conference

Reviews:

final_score: rating: 7: Accept: Technically solid paper, with high impact on at least one sub-area, or moderate-to-high impact on more than one areas, with good-to-excellent evaluation, resources, reproducibility, and no unaddressed ethical considerations.
review_content: summary: The paper proposes a neural representation based on spherical neural fields 
for natural illuminations that are compact and rotation equivariant. 
It proposes a method based on variational auto-encoders to learn such a 
representation. The learned model can act as a statistical prior for natural 
illuminations. The experiment results show that the proposed neural
representation can better represent natural illuminations than traditional 
spherical lighting representations such as spherical harmonics and spherical
gaussian. The paper also shows that the learned model can be incorporated in
inverse renderings tasks as illumination priors and leads to better lighting
estimation. 

strengths_and_weaknesses: Strengths

1. The paper proposes a novel solution to an important problem. The idea of 
using spherical neural fields to represent natural illuminations is inspiring,
and the design of the input to make the represent rotation equivariant is 
also technically sound. The paper also proposes to apply variational
auto-encoders to learn such a model, which also makes sense to me.

2. The paper performs a thorough evaluation against baseline methods to show the
advantages of the proposed representation. It also performs different ablation
studies to validate the capability of the representation and different design
choices.   

3. Such a representation has great potential in many applications, especially in 
inverse rendering tasks. The paper does experiments which show that the learned
priors help estimate more accurate lighting.

Weakness:

1. The paper evaluates the performance of the model with different latent code
size. I am also wondering how the performance changes with different sizes of 
the spherical neural field network. This is an important factor because ideally,
we would like the network to be as small as possible so that the cost of
evaluating such a network would be minimal, especially when using it in inverse
rendering tasks.

2. In Table 1, the performance of SH drops when D increase from 147 to 300. What is
the reason for it?

3. In the third row of Figure 4, why there will be reddish colors that do not
exist in the two source environment maps?

4. In the inverse rendering tasks, the paper only compares to spherical
harmonics. It's also worth adding comparisons to spherical gaussians as they are also 
commonly used in inverse rendering tasks as in [60].

5. When fitting the environment maps in Figure 3, it seems that the performance 
of the proposed method does not improve much with the increase of the coding
size, while SG and SH can better approximate the shape of the lighting. What 
are the limiting factors that prevent the method from further reconstructing more
accurate environment maps? How such a problem can be tackled?


Overall, I like the idea of the paper, and the evaluations are also thorough. I
believe that such an illumination prior can be useful in many tasks such as
inverse rendering and is worth introducing to the community. 

questions: See above.
limitations: The limitations are well discussed.
review_title: null

final_score: rating: 3: Reject: For instance, a paper with technical flaws, weak evaluation, inadequate reproducibility and incompletely addressed ethical considerations.
review_content: summary: This paper presents the neural illumination model based on the variational auto-decoder. By introducing the rotational invariance to the latent variable, the model can represent plausible environment map that could be useful for the inverse rendering task. The evaluation showed that the proposed model has better representation ability with a compact latent code. 
strengths_and_weaknesses: Strengths:

- The attempt to represent light sources in a generative model is very interesting. Adding rotational invariance to the latent space in a generative model is also an interesting attempt.
- The authors properly addressed potential limitations in the method.
- The description of the proposed method is clear and almost no ambiguity exists.

Weakness:

This research consists of two elements: defining the lighting with a generative model and making the model rotationally invariant (around the gravity axis). Therefore, I would like to address concerns about each of these factors. 

- As for the first part, instead of directly optimizing the pointwise function, learning a latent space (i.e. generative model) for a target domain from external data in advance and optimizing its latent variables to generate an instance that satisfies specific criteria have already been done in many computer vision tasks such as CNN-SLAM and StyleNeRF. Therefore, essentially using a similar strategy to generate an environment map doesn’t seem to be a very novel idea. Although an embedded lighting space can indeed represent a wide range of environment maps with fewer parameters, the spaces generated in this way are generally limited to the low-to-mid-frequency range as was mentioned in the paper. Nevertheless, there is little effort to express high-frequency components, as has been done in recent generative models such as StyleGAN. Furthermore, I feel that claiming a contribution for the learnt latent space as “natural illumination prior” is somewhat of an overclaim, as it is true of all data-driven models including other than generative models.

- As for the rotational invariance, I recognize the advantage of being able to learn a latent space with a small number of data, including overlap due to rotation, but I feel that the negative effects of this constraint are not small. As authors have properly identified this in the paper, in real inverse rendering applications, it is very unusual that the camera is parallel to the ground, and if we want to use the environment map generated by the proposed method, we need to compensate for the gravity direction for the coordinate system transformation. But not all digital cameras unfortunately have a gyro sensor. In addition, the realization of rotational invariance is almost entirely based on existing methods, except for the transfer of coordinates to angle representation, so there seems to be little novelty in itself though it seems novel that this invariance is plugged in the generative mocel.

- Due to the lack of real experimental results about the inverse rendering task, it is not clear how useful this model in the real applications (e.g., how the reconstruction accuracies of other attributes such as shape and reflectance improve). In addition to synthetic examples as presented in the paper, there should have been results showing the superiority of the proposed method in real experiments.

- Authors claimed that the O(n) trick proposed in [15] can resolve the scalability issue but have not been verified in the paper. The paper also didn’t show the comparison between SO(2) and SO(3) invariance.

- There are too many mistakes in references.
questions: - Please explain more clearly what new ideas in the paper could be useful in the community. 
- Please clarify more how to represent the high-frequency lighting information with the proposed method. Also, please provide more theoretical evidence of how useful individual components in the proposed method (i.e. latent representation and rotational invariance) are in inverse rendering in practice.
- If there is a misunderstanding, please correct me on the concerns pointed in weakness.

limitations: The authors describe limitations and negative societal impact.
review_title: null

final_score: rating: 6: Weak Accept: Technically solid, moderate-to-high impact paper, with no major concerns with respect to evaluation, resources, reproducibility, ethical considerations.
review_content: summary: The paper proposes a generative model for illumination (spherical incident radiance fields of incoming light from sources at infinite distance, not interacting with the scene). It uses a vector-neuron-based network (encoding the illumination function in MLPs with SIREN-activations), which provides SO(3)-equivariance (of which the SO(2)-part is used/relevant in practice) which is trained as a variational "auto-decoder", a variant of a VAE that does not require the encoding part (which is harder to implement in an equivariant way). The model is trained on a set of HDR environment maps and can later be used for inference task. Applications demonstrated include inverse rendering of objects with local illumination models.

The main contribution seems to be the design of a network representing a statistical illumination prior that is also equivariant, i.e., can learn lighting scenarios under various rotations without (cumbersome) data augmentation. According to the paper (I am not familiar enough with the literature to confirm this), only few pieces of prior work have considered priors on illumination conditions but had less sophisticated solutions in terms of representation and handling large dynamic range.
strengths_and_weaknesses: **Strength:**
- The paper proposes a sophisticated system with many complex parts that are carefully tuned to fit together. In particular, maintaining equivariance at all stages involves some non-straightforward and recent techniques.
- Handling SO(3)-invariance is hard (and group-theoretic "brute-force" solutions based on Wigner-Matrices easily become conceptually very challenging), and the proposed approach appears to navigate around the difficulties well, providing a practical solution, while still remaining expressive and accurate.
- The method apparently does not have fixed resolution limits that linear representations would be prone to (even if just serving as output layer of a more complex network). Functional complexity is still limited by scaling issues (Gram matrix).
- Results are good, surpassing linear SH and non-linear SG base-lines numerically. Visual results are also plausible/convincing.
- The paper is very well written.

**Weaknesses:**
- The paper is mostly a "systems" approach to solve a specific application problem. While very well executed, the broader impact (methodological novelty) to machine learning is limited (although non-negligible, as generative models on spherical domains might be a relevant class of problems). One could debate whether a graphics or vision venue would be a better fit for this topic.
- Quantitative evaluations are limited to SH/SG base-line methods; however, it is probably difficult to find better comparisons due to limited prior work and complexity of the overall setting.
- Despite good writing, the compositional nature of the contribution requires studying the background literature (in particular, vector neurons and auto-decoders) in depth to get a full picture. Again, this is probably hard to avoid.
- The contribution is to some extend incremental (but solid, nonetheless).

questions: If used in an inverse-rendering scenario (say, reconstruction from multi-view photography), what would be required (in addition) to employ the method)? As far as I could see, occlusion and interreflection effects are not modeled (which is fine, but it might be interesting to understand if the method is directly applicable to more challenging tasks in practice).
limitations: I have not found major limitations that are not discussed adequately. The discussion of broader social impact also appears adequate to me.
review_title: null

final_score: rating: 6: Weak Accept: Technically solid, moderate-to-high impact paper, with no major concerns with respect to evaluation, resources, reproducibility, ethical considerations.
review_content: summary: This paper proposes a generative model for natural illumination using a neural field representation based on a variational auto-decoder. The key idea is that natural illumination is highly structured (e.g., lighting comes from above), which has a prior that can be learned and represented well by a generative model. Also, natural illumination generally has a geometric symmetry that a rotation with respect to the vertical axis (i.e., gravity axis) is equally likely, which can be used to restrict the possible illumination space. The proposed method is based on this property and is rotation-equivariant by adding a rotation-invariant transformation on the input direction and the feature vector before the network. The method is demonstrated by extensive evaluations both qualitatively and quantitatively.
strengths_and_weaknesses: ### Strengths
- This paper is well written. It is a good primer for understanding the structure of natural illuminations and their representation including Spherical Harmonic (SH), Spherical Gaussian (SG). The motivation of rotation invariance and equivariance is well presented, and the core idea of SO(2) equivariance is described intuitively and clearly.
- Promising results. The proposed method shows promising results both qualitatively and quantitatively as shown in Table 1 and Figures 3-5. Figure 3 shows that the proposed method RENI captures high-frequency detail such as the primary light source (i.e., sun) more accurately compared to SH and SG where the sun is more blurry. The interpolation and inpainting results in Figures 3 and 4 are also promising and interesting.
- Extensive evaluation. The paper provides extensive evaluations including interpolation, inpainting, and inverse rendering, as well as the comparison to other methods (e.g., SH, SG). The experiments are also done with different dimensions and comparisons are conducted accordingly with various latent dimensions.

### Weaknesses
- Ablation study is missing. While the paper provides extensive evaluations of a range of tasks, an ablation study is missing. A comparison to the baseline model without rotation-equivariance would help show the efficiency of the proposed method in restricting the space of natural illuminations.
- Some preliminaries can be explained more in detail. As the proposed method has some overlap with Vector Neurons (e.g., invariant layer), adding preliminaries regarding Vector Neurons would help readers better understand the proposed method.
- Reference format. Although it is not directly related to the proposed method, the reference section should be fixed. Most references miss the name of the conference or journal, and the names of authors are replaced by “et al”, which should be fixed to the full list of the authors.
questions: - Figure 1 shows the result with 3 x 20 latent code, but all the results used in Section 4 (Evaluation) has different latent dimensions (i.e., N = 9, 36, 49, 100). Is there a special reason why Figure 1 has a different dimension? Also, Figure 3 does not show the case of N = 100, which is in the quantitative comparison in Table 1.
- What is the resolution of the environment maps that are used in the training? Do all the images have the same resolution? If not, how is it sampled in the training?
limitations: The limitations and impact are well described including the memory footprint of the gram matrix, and the possible misalignment of the y-axis in the image. Broader impact and possible bias in the illumination data (i.e., captured in Europe) are also addressed properly.
review_title: null

Meta-review: metareview: The paper introduces a rotation-equivariant conditional spherical neural fields for illumination priors.
Reviewers mostly like the novelty of the proposed approach, its fit for the considered task of illumination priors, technical soundness and experimental evaluation that is thorough and shows merits of the approach. The rebuttals to the reviewers also were thorough and addressed reviewers concerns well. All in all this is a conceptually and experimentally solid and interesting paper that merits publication at NeurIPS.
confidence: Less certain
Decision: Accept