Rotation-Equivariant Conditional Spherical Neural Fields for Learning a Natural Illumination Prior
Authors: James A D Gardner
Bernhard Egger
William A P Smith
Venue: NeurIPS 2022 Accept (2022)
Keywords: Neural Fields
Equivariance
High Dynamic Range
Environment Maps
Illumination Prior
Inverse Rendering

Abstract: Inverse rendering is an ill-posed problem. Previous work has sought to resolve this by focussing on priors for object or scene shape or appearance. In this work, we instead focus on a prior for natural illuminations. Current methods rely on spherical harmonic lighting or other generic representations and, at best, a simplistic prior on the parameters. We propose a conditional neural field representation based on a variational auto-decoder with a SIREN network and, extending Vector Neurons, build equivariance directly into the network. Using this, we develop a rotation-equivariant, high dynamic range (HDR) neural illumination model that is compact and able to express complex, high-frequency features of natural environment maps. Training our model on a curated dataset of 1.6K HDR environment maps of natural scenes, we compare it against traditional representations, demonstrate its applicability for an inverse rendering task and show environment map completion from partial observations.

Rotation-Equivariant Conditional Spherical Neural Fields for Learning a Natural Illumination Prior
James A. D. Gardner Department of Computer Science
Bernhard Egger Cognitive Computer Vision Lab
University of York York, United Kingdom
Erlangen, Germany bernhard. eegerrd.gggeerffrr
james. www.ggarrrdrrreeecmmmm ac.uk
William A. P. Smith
Department of Computer Science
University of York York, United Kingdom
william smith@york.o ac. uk
Abstract
Inverse rendering is an ill-posed problem. Previous work has sought to resolve this by focussing on priors for object or scene shape or appearance, In this work, we instead focus prior for natural illuminations. Current methods rely on spherical harmonic lighting or other generic representations and, at best, simplistic prior the parameters. We propose conditional neural field representation based on variational auto-decoder with a SIREN network and, extending Vector Neurons, build equivariance directly into the network. Using this, develop rotation- equivariant, high dynamic range (HDR) neural illumination model that is compact and able to express complex, high-frequency features of natural environment maps. Training our model on a curated dataset of 1.6K HDR environment maps natural scenes, we compare it against traditional representations, demonstrate its applicability for an inverse rendering task and show environment map completion from partial observations. A PyTo Torch implementation, our dataset and trained models can be found at jadgardner,,ggee bhtegarddeeggihuuioo//RRI.
1 Introduction
Compact but expressive lighting representations play an essential role in graphics, enabling realistic
lighting effects at real-time frame rates (38 (3 54 34 [2] and in computer vision enabling scene relighting [59] 58), face relighting [55] 43 17 42] and object insertion (56) 47 31). Real-world illu- mination highly complex and variable. with very high dynamic range, and therefore inherently challenging to represent. However, real-world illumination does contain statistical regularities [16]. particularly for outdoor, naturally lit scenes. For example, lighting usually comes predominantly from above, often with the strongest illumination coming from primary light sources in a few directions. Also, the sky and sunlight produce only limited range of colours. In addition, illumination environ- ments have a canonical up direction (vertical axis aligns with gravity) but arbitrary horizontal rotation (any rotation about the vertical is equally likely). These regularities and geometric symmetries can significantly restrict the space of possible illuminations to constrain inverse problems or enable the synthesis realistic lighting.
36th Conference on Neural Information Processing Systems (NcurIPS 2022).

---

ve:(Z) ~ N00 IaN)
SO(2) Rotation Latent Code
Around Vertical Y-Axis
2
Figure I: On the left, we visualize environment maps derived from random latent samples of RENI, natural illumination prior, as well as the average illumination in the centre. RENI is rotation- cquivariant to rotations of the latent codes around the vertical y-axis (right). Plots are shown for 3 x 20 latent code at two rotations, 160 degrees apart, and the resulting output of the RENI network equally rotated.
Lighting Representations An illumination environment is spherical signal. A relatively small set of alternatives are used for their representation within vision and graphics. A widely used representation in graphics is an environment map [39 38, 533 [54], which is a regularly sampled 2D image representing a flattening of the sphere. usually via an equirectangular projection. However, the projection introduces distortions leading to irregular sampling on the sphere, it is not compact, introduces boundaries and provides no constraint. Nevertheless, environment map representations have been used in inverse settings where every pixel the map is optimised independently (4T).
Spherical harmonic (SH) lighting 3 38 is compact lighting representation commonly used in
real-time computer graphics [38
22
46 and inverse rendering (59 53 31 40 17). While SHs can be
used to represent the illumination environment directly, more commonly, they represent pre-integrated lighting, i.e. the illumination environment convolved with a bidirectional reflectance distribution function (BRDF). When the BRDF is low frequency. as it is for Lambertian diffuse reflectance, then the convolution is also low frequency making the approximation with SHs very accurate (3.
53 61 SGs represent lighting environment collection of Gaussian lobes the sphere, each of which has 6 degrees of freedom (three for RGB amplitude, two for spherical direction and one for sharpness). While this allows the reconstruction localised high- figh-frreuency features, still requires many lobes to approximate complex illumination environments. (31) compares SH and SG for object re-lighting, finding SG able to recover higher frequency lighting using a similar number of parameters as SH, though both still required large number of parameters to approximate ground truth.
An alternative, growing in popularity, the Spherical Gaussian (SG) representation 54
Both SHs and SGs are rotation equivariant. A rotation of the illumination environment corresponds directly to a rotation of the SH basis or the SG lobe directions. Equivalently, they can represent any rotation of given environment with equal accuracy. However, they provide prior over the space of possible illuminations. SGs or SHs can represent any colour of light coming from any direction.
Illumination Priors When humans solve inverse rendering tasks, they rely strong priors over the space of possible illuminations. For example, they resolve convex/concave shading ambiguities with the feghting-from-abbee priors have been almost completely ignored in computer vision, with the vast majority of inverse rendering techniques allowing arbitrary illumination within their chosen representation space.
assumption 26 52 Given this, it is surprising that statistical illumination
There are a small number of exceptions. Both Egger et al. [17 and Yu and Smith (59 learn a linear statistical model with Gaussian prior in the space of SH coefficients. While providing a useful constraint to avoid unrealistic illumination environments, this approach inherits the weakness of SHs in being unable to reproduce high-frequu -igh-frequency lighting effects while also losing the rotation

---

Rotation-Equivarian Conditional Spherical Neural Field
Invariant Transformation
d'
X
Figure 2: We propose to represent a space of spherical signals via a rotation-equivariann conditional spherical neural field. The signal direction be queried by evaluation of the network and rotating the Vector Neuron conditioning latent code Z, corresponds to rotating the spherical signal.
cquivariance. Yu and Smith [59] seek to overcome this by rotation augmentation at training time, but this brute force approach makes no guarantee of rotation equivariance. Sztrajman al. [49] separate environment maps into HDR and LDR components. Using CNN-based auto-encoder for estimations of LDR components of lighting alongside low dimensional SG model for the HDR lighting provided by the sun. They too require data augmentation in the form of rotations, can make no equivariance guarantees and, due the low dimensionality of the SG model, struggle represent environments with multiple HDR light sources.
Neural Fields Neural fields 57 have provided impressive results in a range of applications
including representations of objects and scenes [44 33
12 23 36 50), in inverse rendering 61] and robotics (30 11 35]. The work most closely related to ours is Neural-PIL. direction query vector and auto-decoder architecture. However, they do not have rotation-equivarianc and apply natural light prior. They also model pre-integrated lighting, conditioning the latter layers of their FiLM SIREN on a material roughness parameter.
7
48
8 They use FiLM SIREN similar that proposed in Pi-GAN (10), and like us
Rotation Invariance/Eqivariinne Two important symmetries in computer vision are invariance and equivariance to the rotation group (9]. Some works attempt to achieve these properties via data augmentation [59 37, 49 which still results missed cases for continuous rotations. solution alleviating the need for extensive data augmentation is the Vector Neuron (15), which offers a framework for designing SO(3)-En ((g)-Eqqiva networks via latent matrix representation rather than latent vector. This allows direct mapping of rotations applied to the network's 's input to its output, resulting in all possible rotations being explicitly represented via rotations of the latent codes.
Contribution We desire representation of natural illumination environments that offers the following features. Generative: A generative model that captures the statistical regularities of natural illumination with a well-behaved latent space within which we can optimise solve inverse problems. Compact: Reduces the dimensionality of inverse problems while preserving high-frequency lighting effects that are important for non-Lan non-Lambertian appearance. Rotation Aaquiion-Eqivariannt Respect the canonical orientation, i.e. any rotation of an environment about the vertical should be equally likely and equally well represented. Statistical Prior: Provides a prior to regularise inverse problems, or that can be sampled from for synthesis, only generating plausible illumination environments. IIDR: Correctly handle HDR quantities essential for realistic rendering and the reproduction of natural light.
We introduce RENI A Rotation-Equivariaat Natural Illumination model. Our key contributions are:
An extension of Vector Neurons to a rotation-equivariant neural field representation for spherical images, optionally restricted to rotations about the vertical axis.
A variational autodecoder architecture for generative model of spherical signals.

---

The first natural, outdoor HDR illumination model.
Evaluated in inverse rendering task showing significant performance improvements other lighting representations.
We choose to model scene radiance, i.e. environment lighting, directly as opposed to pre-integrated lighting with particular BRDF. This makes our model more general since it can be used with arbitrary BRDFs at inference time or even for tasks other than rendering, such as to constrain shape from specular flow.
Rotation-Rquivariant Conditional Spherical Neural Fields
We wish to construct a generative model of spherical signals that is rotation equivariant with respect to the latent representation of the signal. That is to say, a rotation of the latent representation corresponds to rotation of spherical signal and a signal can be reconstructed with exactly the same accuracy in any rotation. We begin by proposing variant of Vector Neurons representation of spherical signals.
[5] for SO(3) equivariant
To achieve this, we use an ordered list of 3D vectors for our latent representation. Our model takes the form of conditional spherical neural field, f S R3xN - R such that f(d. Z) computes the value of the signal represented by latent code Z E R N in direction d E RR, with ||d] = 1. For colour images, M and outputs an RGB colour. By using spherical neural field, we are agnostic to how the signals are sampled on the sphere. We can generate any sampling simply by choosing the grid of directions as appropriate. We also avoid boundary effects since our domain continuous.
We construct the neural field such that is invariant a rotation of both d and Z simultancously (i.e. f(Rd, RZ) = f(d, Z) with R E SO(3)). This entails that the neural field is equivariant with respect to rotation of Z only (i.e. rotating Z corresponds to rotating the spherical signal such that f(d. RZ) fRR d. Z)).
Equivariant Transformation Key to our approach is a transformation of the inputs to the neural field, (d, Z), such that they are rotation invariant. These divide into two parts: d' the directional input to the spherical neural field and 2. Z' the latent code on which the neural field is conditioned.
The direction in which we wish to evaluate the spherical neural field must be encoded relative the latent code in the particular rotation in which we encounter it. This is satisfied by using the inner product (d, Z), i.e. the matrix-vector product d' = Z d E R.V. Unlike Vector Neurons, our input is a direction not a position. Hence, the rotation invariant feature ||d]] conveys no information and we do not use it.
For the latent code, we follow the Vector Neurons invariant layer: Z' VN-In However, we use the full Gram matrix Z Z since we expect the dimensionality of the latent space, N, remain moderate for our data. If this O(N2) size becomes problematic, we can use the same scalable solution used by Vector Neurons [15).
Since our neural field is equivariant we do not need to augment our training data over the space of rotations. Observing a spherical signal once means we can reconstruct it with the same accuracy in any rotation.
Variational Auto-decoder We train our conditional spherical neural field as a decoder-only archi- tecture, i.c. an auto-decoder (36). This means that we optimise the network weights simultaneously with the latent codes for each training sample. This avoids the need to design rotation-equii -quiiiii-euvvrriant encoder while, for inverse tasks, only the decoder is needed so we avoid the redundancy of also training an encoder. However, training with no regularisation on the learnt latent space does not lead space that smooth or that follows known distribution. This means it cannot be sampled from, does not produce meaningful interpolations and provides no prior for inverse problems. For this reason, we use variational auto-decoder (60) architecure, Each training sample is represented by a mean, nt E IR&N, and standard deviation, o; E R3N,, that provide the parameters of normal distri- bution from which the flattened latent code for that training sample drawn: vee(Z.) ~ N(p, E.). where Σ, diag(o?) (o. o.ax) is the diagonal covariance matrix. Using the reparameterisation

---

trick (28), we can generate a latent code vec(Z,) = μ4 + o; @ e where the noise is sampled as NO IaN). During training, we optimise μ4; and o, for each training sample and use the Kullback-Leibb divergence as a loss to regularise the distribution of each latent code toward the standard normal distribution:
CKLD ΣΣ (1 log(o pi.j
- :.g))
where K is the number of training samples.
3 RENI: A Statistical Model of Natural Illumination
We describe how construct statistical model of natural illumination environments restricted version of a rotation-equee -ataiion-quuvariaant conditional spherical neural field. We call our model RENI (Rotation-Equiie Equivariant Natural Illumination). In contrast to the full (eeuqivrrrrrr formulation in Section we propose a variant with only (2) equivariance and explain in detail our training data, losses and implementation.
2
SO(2) Equivariance Natural environments have canonical "up" direction (defined by gravity) but arbitrary rotation about this vertical axis. For this reason, we do not want arbitrary SO(3) rotation equivariance. This would have the undesirable effect of permitting unnatural environment orientations (such as with the sky at the bottom), providing a less useful prior when solving inverse problems. So, we propose restricted transformation of the neural field inputs that invariant only rotations, Ry. about the vertical (y) axis.
In order to construct the invariant features, we use two selection matrices: S.z selects the components that are orthogonal the axis of rotation (i.e. the æ and 2 components) and are thus affected by a y-axis rotation, Sy selects the unaffected y component:
0 0
S
Sy
=
0]
The directional part of the invariant input now contains three components: d' (s (S2; d. s. Z), || The first is the invariant component d. The second encodes relative to Z in the z-2 plane. The third measures the norm of d projected into the x-2 plane which unchanged by rotations about y.
The conditioning part of the invariant input now contains two components: VN-In(Z) = Z' Z, (S22 Z) T Szz The first is simply the invariant component of each column of Z. The second is the invariant transformation (Gram matrix) of latent vectors projected into the r-2 plane.
Reconstruction Loss and HDR HDR imaging is essential for realistic rendering and enables the accurate representation of the full dynamic range of natural light. Therefore our model must learn an HDR representation of natural illumination. Computing reconstruction loss in linear HDR space is dominated by large values and leads to poor reconstruction similar to (18), we train our network to output log (HDR) values and compute the reconstruction loss log space. We further normalise the log (HDR) values to the range [-1, 1, 1] by scaling and shifting using the maximum and minimum values across the whole training set.
most of the environment. Therefore,
Each training sample comprises P pairs of directions and corresponding normalised log (gDR)) RGB colours that we store in the matrices D, [d.,.... d(p] E g and C, = [C.1...., C.P. E 3 respectively. RENI is agnostic to the resolution and sampling of the spherical signal. The neural field can be queried for any direction. In practice, our dataset contains spherical images in an equirectangular sampling containing P = 2H² pixels where H the height of the images. Since equirectangular images are irregularly sampled, we weight the mean squared reconstruction loss by sine the polar angle, 0(d):
Creson
sin sin()(d,))) -c,,l?
(2)
i=1
5

---

Neural Field Our neural field, f, is implemented as a SIREN an MLP with a periodic activation function. This architecture has proven highly effective representing wide range of complex natural signals. In order to condition the SIREN on the invariant latent code Z', we use ceaitiiiiinnn---connttteeenee
[44] [7], i.e. we simply input both d' and Z' to the network. We also tested MLoo-onnditioeed SIREN [TO) but could not get significant performance improvements and, perhaps due to the small size of our dataset, FiLM conditioning resulted a non-smooth latent space that could drastically affect performance when fitting to unseen images.
We have curated a dataset of 1,694 HDR equírectangular images of outdoor, natural illumination environment obtained with either CCO 1.0 Universal Public Domain Dedication license or with written permission to redistribute a low-resolution version of their dataset (25) 4 5IJ. All images were then checked to ensure they did not contain any personally identifiable information or offensive content and any images that contained predominantly unnatural light sources removed. 21 images were also selected and held back for optimising only the latent codes at test time, resulting in a training dataset of 1,673 HDR images.
Training Data
14,
19 27]
Training
We use Adam 29 to optimise the sum of the reconstruction and KLD losses:
B
Train Lreeon + D CKLD
where 3 is a hyperparameter to weight the KLD loss and D = 3N is the dimensionality of the latent space, used normalise for the choice of N. We randomly initialise the mean latent code for each image, p4; from a standard normal distribution. In order to ensure the positivity of the variances, of we optimise log(o?;;) which we initialise randomly from normal distribution with mean -5 and variance 1. We use all the pixels from single training image as a mini-bated batch and use batch size 1.
In order to speed up training, we employ a progressive strategy. We start by training with low resolution (H 16) images and double resolution every 800 epochs until we reach H = 128 resolution for total of 2,400 epochs. This allows the network to quickly learn low-frequency features, gradually obtaining higher frequency details later the training. Unlike with a convolutional architecture, the ppatially-continue resolution training using a single network. We simply upsample the grid of directions, D., and increase the resolution of the target equirectangular images match this resolution.
nature of the neural field means we can implement multi-
We used Weights and Biases (6) for experiment tracking, visualisations and saper-parameter grid search. This yielded the best performance using variational auto- decode,, SIREN with 5 layers each with 128 hidden features. We use an exponentially decaying learning rate starting 10-5 and decreasing to 10-7 by the end of training and weight the KL.D loss B = 10-4 to be of similar magnitude to the reconstruction loss. Training RENI with latent code dimension of 27, took around 15 hours on a local Nvidia A40 48GB GPU.
Model Fitting At test time, the SIREN network is held static, and only latent codes are optimised in order to fit to an unseen image. We initialise the latent code to zeros, corresponding to the mean environment map (see Figure T] left). This provides an unbiased initialisation the absence of any prior information about the environment. We found performance on test images was improved by including a cosine similarity loss LCosine on the RGB colour vectors (32) weighted using the same sin (f))) weight as used in Lecon- We also include a prior loss on the latent vector Lptior = / 12. Our test time loss is therefore:
Les = Lreeu + PLCusine + L.rior-
(4)
We again use multi-resolluion training scheme, fitting with the same resolutions and number of epochs as used during training. A hyper-parameter grid search found the best performance when using
= 10- 10-7, and an exponentially decaying learning rate starting at 10-2 and decreasing to 10-4
A PyTorch implementation, our dataset and trained models can be found at jadgard
net.github.io/RENI

---

Evaluation
Generalisation We begin by evaluating the generalisation performance of RENI when approximating unseen environments. We compare against SH and SG and explore how generalisation performance varies as a function of latent code dimension. We con- sider RGB spherical harmonics of order 2.
Table 1: The mean PSNR when fitting to the test set for increasing latent dimensions. Where an exact di- mensionality comparison is not possible for SG the dimensionality used is shown in brackets.
D
RENI
SH
SG
5, 6 and 9 equal to latent code dimension D = x N for N = 9, 36, 49, 100. Since SG requires dimensionality that is multi- ple 6, where D is not a multiple of 6, we bias in SGs favour by using the next multi- ple, ie. [D/6] 6. We used an open-source implementation of per-pixel environment map fitting provided by [31 fit SG models. As shown in Figure 3 RENI can capture higher frequency detail than both SH and SG, is less dominated by high-value pixels and can reproduce accurate HDR values. A comparison of the mean PSNR
27
17.02
12.80
16.04
(N 30)
108
19.58
15.89
18.44
147
19.97
16.44
19.26
(N 150)
300
20.47
17.07
20.02
across the test set images for increasing latent code dimensionality is shown in Table |
Ground Trush
RENT
S11
5G
RCN
S11
10
109
108
10-1
147
Ground Truits
10a
RENI
RENI
10
27
10-1
108
10-2
147
Figure 3: Generalisation to images with latent code dimensions, D 3NV for N 9,36,44 49 and for SH of equal dimensionality (orders 2, 5, and 6). SG results are with dimensionality 30, 108, 150. Heat maps with log-scale colour bars for ground truth and RENI are also shown.
To test the impact of restricting RENI's equivariance to SO(2) we ran an ablation of models with SO(3), SO(2) and without equivariance three sizes of latent code dimension D. For the model without equivariance, we augmented the dataset with rotations of the images at increments of 0.785rad for a training dataset size of 13384 images. The SO(2) case performs best for all latent code sizes, and both the S0(2) and SO(3) outperform the model trained purely using augmentation whilst using significantly less data. Results shown in Table
an ablation of model sizes. When using smaller network, reconstruction quality suffers due to the representational power of the network being reduced. Whereas the larger networks over-fit on the training data and optimising latent codes to fit unseen images becomes more challenging, perhaps due to the small size of the dataset.

---

Table Mean PSNR on the test set for mod- els with varying levels of equivariance.
Table Mean PSNR on the test set for differ- ent network sizes and latent dimensionality.
D
None
SO(2)
SO(3)
Layers
D = 27
D 108
D = 147
27
11.32
17.02
14.00
16.25
18.29
18.57
108
15.85
19.58
18.27
17.02
19.58
19.97
147
14.64
19.97
17.45
16.38
18.13
18.15
Latent Space Interpolation As shown in Figure linear interpolations between codes result in smooth transitions between images and plausible natural environments for all intermediate latent codes showing how RENI encodes meaningful internal representation of natural illumination.
Figure 4: Interpolation results for RENI with latent code dimension of D 108. Rows and show interpolations between random latent codes, and row shows an interpolation between two training images with the ground -trthhhtthhhu images shown.
Environment Completion Any picture of natural scene contains cues about the surrounding environment that the scene was captured in, such as likely sun locations and possible environmental content. If RENI is only provided with small portion of the complete environment map in its loss at test time, RENI can hallucinate plausible completions of the environment. As shown in Figure RENI makes sensible estimations about the possible colours and shapes of land and sky and often predicts quite accurate sun locations despite the sun being outside the image crop.
Inverse Rendering To test the performance RENI in an inverse rendering pipeline, we imple- mented normalised Blinn-Phoon environment map shader in PyTorch3D, enabling fully differen- tiable rendering. We render 3D object with fixed geometry, pose, camera and material parameters such that only lighting in the scene is unknown. Optimising only latent codes, we minimise the same losses as used in LTests without the sin()(d) weighting needed equirectangular images, between rendering using a ground truth environment map and one using the output of RENI. This was tested for incremental increases weighting the Blinn- Phong specular term (K2). from K, to K, 1.0 in steps of 0.2. A normalisation factor (n + 2)/(4m(2 e( = ))) [21], was applicd to K2 to get a steady transition from diffuse to specular. We achieved the best performance using we
10 ? 10-4 and exponentially decaying learning rate starting 10 and decreasing to 10 over 2,400 epochs. We use an environment map resolution of H 64 and render the object with resolution of 128 In Figure 6 we compare against SH environment maps computed in closed form using linear least squares for increasing specularity. As expected, SH performs well at producing accurate renders of diffuse objects. However, the environment maps produced unnatural, and as the specular term increases in weight, SH degrades in performance significantly compared to RENI.
Non-convexity of Reconstruction Error in Latent Space RENI is rotation equivariant. However, this does not necessarily mean that optimising to fit rotated image will yield rotated version of the latent code resulting from fitting an unrotated version of the image. This means that the loss landscape of our reconstruction losses is not convex. To verify this, we fit RENI to version of the test set containing pairs of unseen images rotated 180 degrees apart. Initialising at the mean environment and optimising resulted in pairs of latent codes Z, Zz. Ideally, this would result in each latent code being explained via y-axis rotation the other. To test this we minimised M for Iz, M Zzll2 and obtained rotation matrix R that minimises [M R| p. We then calculate the

---

Ground Truth
RENI Full
Masked Ground Truth
REN! Inpainting
Figure 5: Col. 2 shows the output of optimising latent codes full ground truth (Col. 1). When trained on a masked ground-truth (Col. 3) RENI predicts plausible continuations for the environment and makes accurate estimations of sun locations (Col. 4). Results from D 108 model.
relative error between RZ₁ and Zz:
RZ;
ZZll
E
IZ F
For model of D = 27, this resulted in F < 2.0% for 18 of the 21 image pairs, demonstrating that both latent codes can largely be explained as simple rotation of the other. However, for the remaining three images, the eтгоr higher, suggesting there is redundancy in the latent space, i.e. there are multiple possible explanations for a single image. Better latent space regularisation, tuning model dimensionality and larger dataset might help resolve this.
5 Discussion and Conclusion
We introduced rotation-squivariann spherical neural fields and used them to create RENI, natural illumination prior. Demonstrating how random samples from RENI always produce plausible illumination maps and RENI's usefulness for environment completion and inverse rendering. There are many exciting avenues for future research, for example, implementing RENI in larger inverse rendering pipelines where could simple drop-in replacement for SH and using RENI for LDR to HDR image reconstruction. Furthermore, our spherical neural field is the initial step towards Generative Adversarial Network (20) (GAN) for spherical signals and could used as the generator alongside Spherical CNN [4 for the discriminator in GAN framework.

---

lunsisg Speciscly
Grasd Sual Tentorend May
Figure 6: Reconstruction results in inverse rendering task. The specular Blinn-Phong Phong term K₃ increases from left to right in steps of 0.2. Both RENI and SH have a dimensionality of D 27. SH performs well in the purely diffuse case, but the resulting environment map reconstruction is unnatural, and as K2 increases RENI performs significantly better.
Limitations Using the Gram matrix for the invariant representation. which is O limits the size of the latent code you can realistically use. This could be addressed through the use of the invariant layer proposed by [15] reduce this (n)) The very highest frequency detail is still not resolved even when using larger latent code sizes; scaling up with larger dataset and network size is likely to address this. Because RENI has prior for natural illumination, unlike SH, RENI's performance decreases when fit to indoor scenes. Interestingly, RENI still generalises to indoor scenes quite well; see supplementary material for examples of RENI fitting indoor scenes.
Human vision has complex interactions between illumination, geometry and texture priors. For exam- ple, the Hollow Face Illusion 266 arises from face geometry priors overriding the lighting from above illumination prior; while in the Bas relief ambiguity (4, geometric priors cause incorrect lighting estimation. Our model discounts these interactions, learning an illumination prior independently but not its interactions with other cues.
We strictly allow only SO(2) equivariance. However, considering typical camera coordinate systems, the up axis will sometimes not align with gravity when the camera is pointed up or down. For inverse problems, this would mean the gravity vector would need to be explicitly estimated (an accelerometer would resolve this). Alternatively, we could build our model with full SO(3) equivariance but then learn prior over the space of camera poses relative gravity.
Broader Impact The illumination prior presented here has the potential to improve the performance of inverse rendering pipelines, of which there are both positive and negative downstream cases that should be considered; for example, the generation of highly realistic 3D models of a person's 's likeness without their consent. Advances in inverse and neural rendering could potentially lead to employment loss in the generation of 3D assets. However, we are optimistic that the democratisation of 3D model generation will be a value-generating technology for the large majority. The illumination prior might be biased since most of the available and used HDR images were captured in Europe.
Acknowledgments and Disclosure of Funding James Gardner was supported by the EPSRC Centre for Doctoral Training in Intelligent Games & Games Intelligence (IGGI) (EP/S022325/1). We would like to thank the attendees of Dagstuhl Seminar, 22121 3D Morphable Models and Beyond, for their valuable insights and discussions around this work.
10

---

References
[1] Whitemagus 3D. HDRIs Whitemagus 3D Models. en-US. URL: http www
whi whitemagus3dmode.. com
[2] Matan Atzmon and Yaron Lipman. "SAL: Sign Agnostic Learning of Shapes from Raw Data".
In: IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR). 2020.
[3] R. Basri and D.W. Jacobs. "Lambertian reflectance and linear subspaces". In: IEEE Transac- tions on Pattern Analysis and Machine Intelligence 25.2 (2003), pp. 218-233,
[4] Peter N. Belhumeur, David J. Kriegman, and Alan L. Yuille. "The Bas-Relief Ambiguity". In: International Journal of Computer Vision 35.1 (Nov. 1999), pp. 33-44.
[5] Sai Bi, Zexiang Xu, Pratul Srinivasan, Ben Mildenhall, Kalyan Sunkavalli, Miloš Hašan, Yannick Hold-Geoffroy, David Kriegman, and Ravi Ramamoorthi. "Neural Reflectance Fields for Appearance Acquisition". In: arXiv. 2020.
[6] Lukas Biewald. Experiment Tracking with Weights and Biases. 2020. URL: attps://ve wandb. com/
[7] Mark Boss, Raphael Braun, Varun Jampani, Jonathan T. Barron, Ce Liu, and Hendrik P. A. Lensch. "NeRD: Neural Reflectance Decomposition from Image Collections". In: IEEE
International Conference on Computer Vision (ICCV). 2021.
[8] Mark Boss, Varun Jampani, Raphael Braun, Ce Liu, Jonathan T. Barron, and Hendrik P. A. Lensch. "Neural-PIL: Neural Pre-Integrated Lighting for Reflectance Decomposition". In: Advances in Neural Information Processing Systems (NeurIPS). 2021.
[9] Michael M. Bronstein, Joan Bruna, Taco Cohen, and Petar Veličković. Geometric Deep Learning: Grids, Groups, Graphs, Geodesics, and Gauges. 2021.
[10] Eric Chan, Marco Monteiro, Petr Kellnhofer, Jiajun Wu, and Gordon Wetzstein. "pi-GAN: Periodic Implicit Generative Adversarial Networks for 3l>-Aware Image Synthesis". In: 202/
IEEE/CVF Conference on Computer Vision and Puttern Recognition (CVPR). 2021.
[11] Boyuan Chen, Robert Kwiatkowski, Carl Vondrick, and Hod Lipson. "Full-Body Visual Self-Modeling of Robot Morphologies" In: Science Robotics 7.68 (2021).
[12] Zhiqin Chen and Hao Zhang. "Learning Implicit Fields for Generative Shape Modeling". In: 2019 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR). 2019.
[13] Julian Chibane, Aymen Mir, and Gerard Pons-Moll. "Neural Unsigned Distance Fields for Im- plicit Function Learning". In: Advances Neural Information Processing Systems (NeurIPS). 2020.
[14] Taco S. Cohen, Mario Geiger, Jonas Köhler, and Max Welling. "Spherical CNNs". In: Interna- tional Conference on Learning Representations. 2018.
[15] Congyue Deng, Or Litany, Yueqi Duan, Adrien Poulenard, Andrea Tagliasacchi, and Leonidas Guibas. "Vector Neurons: A General Framework for Equiirrqunaa Networks". In: IEEE International Conference on Computer Vision (ICCV). Apr. 2021.
[16] Ron O. Dror, Alan S. Willsky, and Edward H. Adelson. "Statistical characterization of real- world illumination". In: Journal of Vision 4.9 (Sept. 2004). pp. 11-11.
[17] Bernhard Egger, Sandro Schönborn, Andreas Schneider, Adam Kortylewski, Andreas Morel- Forster, Clemens Blumer, and Thomas Vetter. "Occlusion-Awwe 3D Morphable Models and an Illumination Prior for Face Image Analysis". ". In: International Journal of Computer Vision 126 (2018), pp. 1269-1287
[18] Gabriel Eilertsen, Joel Kronander, Gyorgy Denes, Rafall K. Mantiuk, and Jonas Unger. "HDR Image Reconstruction from Single Exposure Using Deep CNNs In: ACM Trans. Graph. 36.6 (Nov. 2017). Place: New York, NY, USA Publisher: Association for Computing Machinery.
[19] GiantCowFilms. HDRIs GiantCow en-US. URL.: https: / hatee://ggannnnn /ggantcooffffo ilms. com/ category/háris/
[20] Ian Goodfellow, Jean Pouget-Abadie, Mehdi Mirza, Bing Xu, David Warde-Farley, Sherjil Ozair, Aaron Courville, and Yoshua Bengio. "Generative Adversarial Nets". In: Advances in Neural Information Processing Systems. Ed. by Z. Ghahramani, M. Welling, C. Cortes, N. Lawrence, and K. Q Weinberger. Vol. 27. Curran Associates, Inc., 2014.
[21] Yoshiharu Gotanda, Naty Hoffman, Adam Martinez, and Ben Snow. "Physically- -assed Shad- ing in Film and Game Production". In: ACM SIGGRAPH 2010 Courses. SIGGRAPH 12.
Association for Computing Machinery, 2010

---

[22] Robin Green. "Spherical harmonic lighting: The gritty details". In: Archives of the game developers conference. Vol. 56. 2003, p.
[23] Amos Gropp, Lior Yariv, Niv Haim, Matan Atzmon, and Yaron Lipman. "Implicit Geometric Regularization for Learning Shapes". In: Proceedings of Machine Learning and Systems. 2020.
[24] Poly Haven. HDRIs Poly Haven. URT.: htttss::/ppolhhavve..cmm cww.weiinccoo////iii..aa..
[25] hdrmaps. HDRIs harmaps. en-US. URL: http::::/ds:::dmpaapptaa com/
[26] Harold Hill and Vicki Bruce. "Independent Effects of Lighting, Orientation, and Stereopsis on the Hollow-Face Illusion". In: Perception 22.8 (1993), pp. 887-897.
[27] iHDRi. IDRIS iIIDRi. URR: http:///wwww.ahhh....oooooottiaa ottt:/orro--kiieeoooooooo/o
[28] Diederik P Kingma and Max Welling. "Auto-Encoding Variational Bayes". In: International
Conference on Learning Representations (ICLR). 2013. DOT: :tt::10..8850// 0...5055//AXII 1312. .614
[29] Diederik P. Kingma and Jimmy Ba. "Adam: A Method for Stochastic Optimization". In:
International Conference on Learning Representations (ICLR) (2015).
[30] Yunzhu Li, Shuang Li, Vincent Sitzmann, Pulkit Agrawal, and Antonio Torralba. "3D Neural Scene Representations for Visuomotor Control". In: Conference on Robot Learning (CoRL). 2021.
[31] Zhengqin Li. Mohammad Shafiei, Ravi Ramamoorthi, Kalyan Sunkavalli, and Manmohan Chandraker. "Inverse rendering for complex indoor scenes: Shape, spatially lighting and svbrdf from single image". In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. 2020, pp. 2475-2484.
[32] Demetris Marnerides, Thomas Bashford Mashford-RRgers. Jonathan Hatchett, and Kurt Debattista. "ExpandNet: A Deep Convolutional Neural Network for High Dynamic Range Expansion from Low Dynamic Range Content". In: Computer Graphics Forum 37 (2018).
[33] Lars Mescheder, Michael Oechsle, Michael Niemeyer, Sebastian Nowozin, and Andreas Geiger. "Occupancy Networks: Learning 3D Reconstruction Function Space". In: Proceedings IEEE Conf. Computer Vision and Pattern Recognition (CVPR). 2019.
[34] Ren Ng, Ravi Ramamoorthi, and Pat Hanrahan. "All-Frequency Shadows Using Non-Linear Wavelet Lighting Approximation". In: ACM SIGGRAPH 2003 Papers. SIGGRAPH '03. Association for Computing Machinery, 2003, pp. 376-381.
[35] Joseph Ortiz,, Alexander Clegg, Jing Dong, Edgar Sucar, David Novotny, Michael Zollhoe- fer, and Mustafa Mukadam. "ISDF: Real-Time Neural Signed Distance Fields for Robot Perception". en. In: Robotics: Science and Systems (May 2022).
[36] Jeong Joon Park, Peter Florence, Julian Straub, Richard Newcombe, and Steven Lovegrove. "DeepSDF: Learning Continuous Signed Distance Functions for Shape Representation" In: Computer Vision & Putlern Recognition (CVPR). 2019.
[37] Charles R Qi, Hao Su, Kaichun Mo, and Leonidas J Guibas. "Pointnet: Deep learning on point sets for 3d classification and segmentation" " In: Proceedings of the IEEE conference on computer vision and pattern recognition. 2017, pp. 652-660.
[38] Ravi Ramamoorthi and Pat Hanrahan. " "An efficient representation for irradiance enviromment maps". In: Proceedings of the 28th annual conference on Computer graphics and interactive techniques. SIGGRAPH '01. Association for Computing Machinery, Aug. 2001, pp. 497-500.
[39] Ravi Ramamoorthi and Pat Hanrahan. "Frequency Space Environment Map Rendering". In: Proceedings of the 29th Annual Conference on Computer Graphics and Interactive Techniques. SIGGRAPH '02. Association for Computing Machinery, 2002, pp. 517-526.
[40] Viktor Rudnev, Mohamed Elgharib, William Smith, Lingjie Liu, Vladislav Golyanik, and Christian Theobalt. "NeRF for Outdoor Scene Relighting". . In: European Conference Computer Vision (ECCV). 2022.
[41] Soumyadip Sengupta, Jinwei Gu, Kihwan Kim, Guilin Liu, David W. Jacobs, and Jan Kautz. "Neural Inverse Rendering of an Indoor Scene from a Single Image". In: International Confer- ence on Computer Vision (ICCV). 2019.
[42] Soumyadip Sengupta, Angjoo Kanazawa, Carlos D. Castillo, and David Jacobs. "SfSNet: Learning Shape, Reflectance and Illuminance of Faces in the Wild". In: Computer Vision and Pattern Regognition (CVPR). 2018
[43] Zhixin Shu, Ersin Yumer, Sunil Hadap, Kalyan Sunkavalli, Eli Shechtman, and Dimitris Samaras. Neural Face Editing with Intrinsic Image Disentangling. 2017.
12

---

[44] Vincent Sitzmann, Julien Martel, Alexander Bergman, David Lindell, and Gordon Wetzstein. "Implicit neural representations with periodic activation functions", In: Adv. Neural Inf. Process. Syst. 33 (2020). pp. 7462-7473.
[45] HDRI Skies. HDRIs HDRI Skies. en-US. URL: htt.ttttt:///iiisiiieeette skies.oossiissso
[46] Peter-Pike Sloan, Jan Kautz, and John Snyder. "Precomputed Radiance Transfer for Real-Time
Rendering in Dynamic, Low-Frequency Lighting Environments". en. In: ACM Transactions on Graphics 21.3 (2002), p. 10.
[47]
Shuran Song and Thomas Funkhouser. "Neural Illumination: Lighting Prediction for Indoor Environments". In: IEEE Conference on Computer Vision and Pultern Recognition. 2019.
[48] Pratul P. Srinivasan, Boyang Deng, Xiuming Zhang, Matthew Tancik, Ben Mildenhall, and Jonathan T. Barron. "NeRV: Neural Reflectance and Visibility Fields for Relighting and View Synthesis". In: JEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR). 2020.
[49] Alejandro Sztrajman, Alexandros Neophytou, Tim Weyrich, and Eric Sommerlade. "High- Dynamie-Range Lighting Estimation From Face Portraits". In: 2020 International Conference on 3D Vision (3DV). 2020, pp. 355-363.
[50] Matthew Tancik, Vincent Casser, Xinchen Yan, Sabeek Pradhan, Ben Mildenhall, Pratul P. Srinivasan, Jonathan T. Barron, and Henrik Kretzschmar. "Block-NeRF: Scalable Large Scene Neural View Synthesis". In: arXiv. 2022.
[51]
textures. HDRIs textures. URL: lhttps://wwt textures. com/
[52]
Rhiannon Thomas, Marko Nardini, and Denis Mareschal. "Interactions between "light-froue above" and convexity priors in visual development". In: Journal of Vision 10.8 (July 2010), pp. 6-6.
[53] Yu-Ting Tsai and Zen-Chung Shih. "All-Fiequency Precomputed Radiance Transfer using Spherical Radial Basis Functions and Clustered Tensor Approximation" In: Association for Computing Machinery 25.3 (2006). 10.
[54] Jiaping Wang, Peiran Ren, Minmin Gong, John Snyder, and Baining Guo. "All-Frequency Ren- dering of Dynamic, Paatially-Varyingg Reflectance". In: Association for Computing Machinery 28.5 (2009), p. 10.
[55] Yang Wang, Lei Zhang, Zicheng Liu, Gang Hua, Zhen Wen, Zhengyou Zhang, and Dimitris Samaras. "Face Relighting from a Single Image under Arbitrary Unknown Lighting Condi- tions". In: JEEE transactions on pattern analysis and machine intelligence 31 (Nov. 2009),
pp. 1968-84.
[56) Zian Wang, Jonah Philion, Sanja Fidler, and Jan Kautz. "Leearning Indoor Inverse Rendering with 3D Spatially- -arying Lighting". In: Proceedings of International Conference on Computer Vision (ICCV). 2021.
[57] Yiheng Xie, Towaki Takikawa, Shunsuke Saito, Or Litany, Shiqin Yan, Numair Khan, Federico Tombari, James Tompkin, Vincent Sitzmann, and Srinath Sridhar. "Neural Fields in Visual Computing and Beyond". In: Computer Graphics Forum (Farographies 2022) (Nov. 2021). [58] Ye Yu, Abhimetra Meka, Mohamed Elgharib, Hans-Peter Seidel, Christian Theobalt, and Will Smith. "Self-supervised Outdoor Scene Relighting". In: European Conference on Computer Vision (ECCV). 2020.
[59] Ye Yu and William A. P. Smith. "Outdoor inverse rendering from a single image using multi- view self-supervision". In: IEEE Transactions on Pattern Analysis and Machine Intelligence. 2021.
[60] Amir Zadeh, Yao-Chong Lim, Paul Pu Liang, and Louis-Philippe Morency. "Variational Auto-Decoder: A Method for Neural Generative Modeling from Incomplete Data". In: arXiv. 2021.
[61] Kai Zhang, Fujun Luan, Qianqian Wang, Kavita Bala, and Noah Snavely. "PhySG: Inverse Rendering with Spherical Gaussians for Physics-basee Material Editing and Relighting". In: The IEEE/CVF Conference on Computer Vision and Puttern Recognition (CVPR). 2021.
13

---

Checklist
The checklist follows the references. Please read the checklist guidelines carefully for information on how to answer these questions. For each question, change the default [TODO] to [Yes], [No], or [N/A] You are strongly encouraged to include a justification your answer, either by referencing the appropriate section of your paper or providing brief inline description. For example:
Did you include the license to the code and datasets? [Yes] See Section 3
Did you include the license to the code and datasets? [No] The code and the data are proprietary.
Did you include the license the code and datasets? [N/A]
Please do not modify the questions and only use the provided macros for your answers. Note that the Checklist section does not count towards the page limit. In your paper, please delete this instructions block and only keep the Checklist section heading above along with the questions/answers below.
1. For all authors...
(a) Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope? [Yes]
(b) Did you describe the limitations of your work? [Yes] See Section S
(c) Did you discuss any potential negative societal impacts of your work? [Yes] See Section Broader Impact]
(d) Have you read the ethics review guidelines and ensured that your paper conforms to them? [Yes]
2. If you are including theoretical results...
(a) Did you state the full set of assumptions of all theoretical results? [N/A]
(b) Did you include complete proofs of all theoretical results? [N/A]
3. If you ran experiments..
(a) Did you include the code, data, and instructions needed to reproduce the main exper- imental results (either in the supplemental material as URL)? [Yes] Please supplementary material for the code which includes test set images and a README explaining how to setup and run a trained model. This is not yet release ready and will be refactored prior to being publicly released.
(b) Did you specify all the training details (e.g., data splits, hyperparameters, how they were chosen)? [Yes] See page 6, paragraph 'Training Details' and page 8 paragraph Inverse Rendering'.
(c) Did you report error bars (e.g.. with respect the random seed after running experi- ments multiple times)? [N/A] No error bars due to deterministic results.
(d) Did you include the total amount of compute and the type of resources used (e.g.. type of GPUs, internal cluster, or cloud provider)? [Yes] See page 6 paragraph 'Training Details'.
4. If you are using existing assets code, data, models) or curating/releasing new assets...
(a) If your work uses existing assets, did you cite the creators? [Yes] See page
paragraph
"Trinnin Data" and page 7 paragraph 4 "eneralissatoon
(b) Did you mention the license of the assets? [Yes] See page 6 paragraph Training
Data"
(c) Did you include any new assets either in the supplemental material or as a URL? [N/A] No original assets were created for this research.
(d) Did you discuss whether and how consent was obtained from people whose data you're 're using/curating? [Yes] See p paragraph "rainiig Data"
(e) Did you discuss whether the data you are using/curating contains personally identifiable information or offensive content? [Yes] See page/6] paragraph ""raaphhh"TTeee Trriining Data'
5. If you used crowdsourcing conducted research with human subjects...
14

---

(a) Did you include the full text of instructions given to participants and screenshots, if applicable? [N/A] No crowdsourcing human subjects were used in this research. (b) Did you describe any potential participant risks, with links to Institutional Review Board (IRB) approvals, if applicable? [N/A] No crowdsourcing human subjects were used this research.
(c) Did you include the estimated hourly wage paid to participants and the total amount spent on participant compensation? [N/A] No crowdsourcing or human subjects were used in this research.
15


Reviews:
[{"review_id": "gijX96DtHC2", "reviewer": "Reviewer_VopV", "summary": "The paper proposes a generative model for illumination (spherical incident radiance fields of incoming light from sources at infinite distance, not interacting with the scene). It uses a vector-neuron-based network (encoding the illumination function in MLPs with SIREN-activations), which provides SO(3)-equivariance (of which the SO(2)-part is used/relevant in practice) which is trained as a variational \"auto-decoder\", a variant of a VAE that does not require the encoding part (which is harder to implement in an equivariant way). The model is trained on a set of HDR environment maps and can later be used for inference task. Applications demonstrated include inverse rendering of objects with local illumination models.\n\nThe main contribution seems to be the design of a network representing a statistical illumination prior that is also equivariant, i.e., can learn lighting scenarios under various rotations without (cumbersome) data augmentation. According to the paper (I am not familiar enough with the literature to confirm this), only few pieces of prior work have considered priors on illumination conditions but had less sophisticated solutions in terms of representation and handling large dynamic range.", "questions": "If used in an inverse-rendering scenario (say, reconstruction from multi-view photography), what would be required (in addition) to employ the method)? As far as I could see, occlusion and interreflection effects are not modeled (which is fine, but it might be interesting to understand if the method is directly applicable to more challenging tasks in practice).", "limitations": "I have not found major limitations that are not discussed adequately. The discussion of broader social impact also appears adequate to me.", "rating": 6, "confidence": 2, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "**Strength:**\n- The paper proposes a sophisticated system with many complex parts that are carefully tuned to fit together. In particular, maintaining equivariance at all stages involves some non-straightforward and recent techniques.\n- Handling SO(3)-invariance is hard (and group-theoretic \"brute-force\" solutions based on Wigner-Matrices easily become conceptually very challenging), and the proposed approach appears to navigate around the difficulties well, providing a practical solution, while still remaining expressive and accurate.\n- The method apparently does not have fixed resolution limits that linear representations would be prone to (even if just serving as output layer of a more complex network). Functional complexity is still limited by scaling issues (Gram matrix).\n- Results are good, surpassing linear SH and non-linear SG base-lines numerically. Visual results are also plausible/convincing.\n- The paper is very well written.\n\n**Weaknesses:**\n- The paper is mostly a \"systems\" approach to solve a specific application problem. While very well executed, the broader impact (methodological novelty) to machine learning is limited (although non-negligible, as generative models on spherical domains might be a relevant class of problems). One could debate whether a graphics or vision venue would be a better fit for this topic.\n- Quantitative evaluations are limited to SH/SG base-line methods; however, it is probably difficult to find better comparisons due to limited prior work and complexity of the overall setting.\n- Despite good writing, the compositional nature of the contribution requires studying the background literature (in particular, vector neurons and auto-decoders) in depth to get a full picture. Again, this is probably hard to avoid.\n- The contribution is to some extend incremental (but solid, nonetheless).\n", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"ethics_review_area": ""}, "extra_text": {}}, {"review_id": "XwS44e70ky", "reviewer": "Reviewer_cfSy", "summary": "This paper presents the neural illumination model based on the variational auto-decoder. By introducing the rotational invariance to the latent variable, the model can represent plausible environment map that could be useful for the inverse rendering task. The evaluation showed that the proposed model has better representation ability with a compact latent code. ", "questions": "- Please explain more clearly what new ideas in the paper could be useful in the community. \n- Please clarify more how to represent the high-frequency lighting information with the proposed method. Also, please provide more theoretical evidence of how useful individual components in the proposed method (i.e. latent representation and rotational invariance) are in inverse rendering in practice.\n- If there is a misunderstanding, please correct me on the concerns pointed in weakness.\n", "limitations": "The authors describe limitations and negative societal impact.", "rating": 3, "confidence": 4, "soundness": 2, "presentation": 3, "contribution": 2, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Strengths:\n\n- The attempt to represent light sources in a generative model is very interesting. Adding rotational invariance to the latent space in a generative model is also an interesting attempt.\n- The authors properly addressed potential limitations in the method.\n- The description of the proposed method is clear and almost no ambiguity exists.\n\nWeakness:\n\nThis research consists of two elements: defining the lighting with a generative model and making the model rotationally invariant (around the gravity axis). Therefore, I would like to address concerns about each of these factors. \n\n- As for the first part, instead of directly optimizing the pointwise function, learning a latent space (i.e. generative model) for a target domain from external data in advance and optimizing its latent variables to generate an instance that satisfies specific criteria have already been done in many computer vision tasks such as CNN-SLAM and StyleNeRF. Therefore, essentially using a similar strategy to generate an environment map doesn’t seem to be a very novel idea. Although an embedded lighting space can indeed represent a wide range of environment maps with fewer parameters, the spaces generated in this way are generally limited to the low-to-mid-frequency range as was mentioned in the paper. Nevertheless, there is little effort to express high-frequency components, as has been done in recent generative models such as StyleGAN. Furthermore, I feel that claiming a contribution for the learnt latent space as “natural illumination prior” is somewhat of an overclaim, as it is true of all data-driven models including other than generative models.\n\n- As for the rotational invariance, I recognize the advantage of being able to learn a latent space with a small number of data, including overlap due to rotation, but I feel that the negative effects of this constraint are not small. As authors have properly identified this in the paper, in real inverse rendering applications, it is very unusual that the camera is parallel to the ground, and if we want to use the environment map generated by the proposed method, we need to compensate for the gravity direction for the coordinate system transformation. But not all digital cameras unfortunately have a gyro sensor. In addition, the realization of rotational invariance is almost entirely based on existing methods, except for the transfer of coordinates to angle representation, so there seems to be little novelty in itself though it seems novel that this invariance is plugged in the generative mocel.\n\n- Due to the lack of real experimental results about the inverse rendering task, it is not clear how useful this model in the real applications (e.g., how the reconstruction accuracies of other attributes such as shape and reflectance improve). In addition to synthetic examples as presented in the paper, there should have been results showing the superiority of the proposed method in real experiments.\n\n- Authors claimed that the O(n) trick proposed in [15] can resolve the scalability issue but have not been verified in the paper. The paper also didn’t show the comparison between SO(2) and SO(3) invariance.\n\n- There are too many mistakes in references.", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"ethics_review_area": ""}, "extra_text": {}}, {"review_id": "KK88f4KUEtB", "reviewer": "Reviewer_1WWp", "summary": "The paper proposes a neural representation based on spherical neural fields \nfor natural illuminations that are compact and rotation equivariant. \nIt proposes a method based on variational auto-encoders to learn such a \nrepresentation. The learned model can act as a statistical prior for natural \nilluminations. The experiment results show that the proposed neural\nrepresentation can better represent natural illuminations than traditional \nspherical lighting representations such as spherical harmonics and spherical\ngaussian. The paper also shows that the learned model can be incorporated in\ninverse renderings tasks as illumination priors and leads to better lighting\nestimation. \n", "questions": "See above.", "limitations": "The limitations are well discussed.", "rating": 7, "confidence": 5, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Strengths\n\n1. The paper proposes a novel solution to an important problem. The idea of \nusing spherical neural fields to represent natural illuminations is inspiring,\nand the design of the input to make the represent rotation equivariant is \nalso technically sound. The paper also proposes to apply variational\nauto-encoders to learn such a model, which also makes sense to me.\n\n2. The paper performs a thorough evaluation against baseline methods to show the\nadvantages of the proposed representation. It also performs different ablation\nstudies to validate the capability of the representation and different design\nchoices.   \n\n3. Such a representation has great potential in many applications, especially in \ninverse rendering tasks. The paper does experiments which show that the learned\npriors help estimate more accurate lighting.\n\nWeakness:\n\n1. The paper evaluates the performance of the model with different latent code\nsize. I am also wondering how the performance changes with different sizes of \nthe spherical neural field network. This is an important factor because ideally,\nwe would like the network to be as small as possible so that the cost of\nevaluating such a network would be minimal, especially when using it in inverse\nrendering tasks.\n\n2. In Table 1, the performance of SH drops when D increase from 147 to 300. What is\nthe reason for it?\n\n3. In the third row of Figure 4, why there will be reddish colors that do not\nexist in the two source environment maps?\n\n4. In the inverse rendering tasks, the paper only compares to spherical\nharmonics. It's also worth adding comparisons to spherical gaussians as they are also \ncommonly used in inverse rendering tasks as in [60].\n\n5. When fitting the environment maps in Figure 3, it seems that the performance \nof the proposed method does not improve much with the increase of the coding\nsize, while SG and SH can better approximate the shape of the lighting. What \nare the limiting factors that prevent the method from further reconstructing more\naccurate environment maps? How such a problem can be tackled?\n\n\nOverall, I like the idea of the paper, and the evaluations are also thorough. I\nbelieve that such an illumination prior can be useful in many tasks such as\ninverse rendering and is worth introducing to the community. \n", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"ethics_review_area": ""}, "extra_text": {}}, {"review_id": "0vnYhBVD64m", "reviewer": "Reviewer_PVj2", "summary": "This paper proposes a generative model for natural illumination using a neural field representation based on a variational auto-decoder. The key idea is that natural illumination is highly structured (e.g., lighting comes from above), which has a prior that can be learned and represented well by a generative model. Also, natural illumination generally has a geometric symmetry that a rotation with respect to the vertical axis (i.e., gravity axis) is equally likely, which can be used to restrict the possible illumination space. The proposed method is based on this property and is rotation-equivariant by adding a rotation-invariant transformation on the input direction and the feature vector before the network. The method is demonstrated by extensive evaluations both qualitatively and quantitatively.", "questions": "- Figure 1 shows the result with 3 x 20 latent code, but all the results used in Section 4 (Evaluation) has different latent dimensions (i.e., N = 9, 36, 49, 100). Is there a special reason why Figure 1 has a different dimension? Also, Figure 3 does not show the case of N = 100, which is in the quantitative comparison in Table 1.\n- What is the resolution of the environment maps that are used in the training? Do all the images have the same resolution? If not, how is it sampled in the training?", "limitations": "The limitations and impact are well described including the memory footprint of the gram matrix, and the possible misalignment of the y-axis in the image. Broader impact and possible bias in the illumination data (i.e., captured in Europe) are also addressed properly.", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 2, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "### Strengths\n- This paper is well written. It is a good primer for understanding the structure of natural illuminations and their representation including Spherical Harmonic (SH), Spherical Gaussian (SG). The motivation of rotation invariance and equivariance is well presented, and the core idea of SO(2) equivariance is described intuitively and clearly.\n- Promising results. The proposed method shows promising results both qualitatively and quantitatively as shown in Table 1 and Figures 3-5. Figure 3 shows that the proposed method RENI captures high-frequency detail such as the primary light source (i.e., sun) more accurately compared to SH and SG where the sun is more blurry. The interpolation and inpainting results in Figures 3 and 4 are also promising and interesting.\n- Extensive evaluation. The paper provides extensive evaluations including interpolation, inpainting, and inverse rendering, as well as the comparison to other methods (e.g., SH, SG). The experiments are also done with different dimensions and comparisons are conducted accordingly with various latent dimensions.\n\n### Weaknesses\n- Ablation study is missing. While the paper provides extensive evaluations of a range of tasks, an ablation study is missing. A comparison to the baseline model without rotation-equivariance would help show the efficiency of the proposed method in restricting the space of natural illuminations.\n- Some preliminaries can be explained more in detail. As the proposed method has some overlap with Vector Neurons (e.g., invariant layer), adding preliminaries regarding Vector Neurons would help readers better understand the proposed method.\n- Reference format. Although it is not directly related to the proposed method, the reference section should be fixed. Most references miss the name of the conference or journal, and the names of authors are replaced by “et al”, which should be fixed to the full list of the authors.", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"ethics_review_area": ""}, "extra_text": {}}]

Author rebuttal:

Decision: Accept