Neural-PIL: Neural Pre-Integrated Lighting for Reflectance Decomposition
Authors: Mark Boss
Varun Jampani
Raphael Braun
Ce Liu
Jonathan T. Barron
Hendrik Lensch
Venue: NeurIPS 2021 Poster (2021)
Keywords: Inverse Rendering
Coordinate-based MLP
BRDF Estimation
Illumination Estimation
NeRF
Inverse Graphics

Abstract: Decomposing a scene into its shape, reflectance and illumination is a fundamental problem in computer vision and graphics. Neural approaches such as NeRF have achieved remarkable success in view synthesis, but do not explicitly perform decomposition and instead operate exclusively on radiance (the product of reflectance and illumination). Extensions to NeRF, such as NeRD, can perform decomposition but struggle to accurately recover detailed illumination, thereby significantly limiting realism. We propose a novel reflectance decomposition network that can estimate shape, BRDF, and per-image illumination given a set of object images captured under varying illumination. Our key technique is a novel illumination integration network called Neural-PIL that replaces a costly illumination integral operation in the rendering with a simple network query. In addition, we also learn deep low-dimensional priors on BRDF and illumination representations using novel smooth manifold auto-encoders. Our decompositions can result in considerably better BRDF and light estimates enabling more accurate novel view-synthesis and relighting compared to prior art. Project page: https://markboss.me/publication/2021-neural-pil/

Neural-PIL: Neural Pre-Integrated Lighting for Reflectance Decomposition
Mark Boss University of Tübingen
Varun Jampani Google Research
Raphael Braun University of Tübingen
Ce Liu" Microsoft Azure AI
Jonathan T. Barron
Hendrik P. A. Lenseh University of Tübingen
Google Research
Abstract
Decomposing a scene into its shape, reflectance and illumination is fundamen- tal problem in computer vision and graphics. Neural approaches such as NeRF have achieved remarkable success in view synthesis, but not explicitly per- form decomposition and instead operate exclusively reflectance and illumination). Extensions to NeRF, such as NeRD, can perform decomposition but struggle to accurately recover detailed illumination, thereby significantly limiting realism. We propose a novel reflectance decomposition network that estimate shape, BRDF and per-image illumination given set object images captured under varying illumination. Our key technique is a novel illumination integration network called Neural-PIL that replaces costly illumination integral operation in the rendering with a simple network query. In addition, also learn deep low-dimensional priors on BRDF and illumination representations using novel smooth manifold auto-encoders. Our decompositions can result considerably better BRDF and light estimates enabling more ac- curate novel view-synthesis and relighting compared to prior art. Project page: www.mm/liaaaiii......mmmmmm 2021 -nerral-uurlppp.
radiance (the product of
Introduction
Inverse rendering is the task of decomposing a scene into its underlying physical properties, such geometry and materials. Recovering these properties is useful for several vision and graphics
applications such as view synthesis [10 [1 51 57). relighting
4 10 11
23
24
38
51 [8],
and object insertion 7 21 (38. In this work, we aim recover the 3D shape and spatially-
varying bidirectional reflectance distribution function (SVBRDF) of an object imaged under different
illumination conditions, shown in Fig. [ Estimating shape, illumination, and SVBRDF from 2D images is a highly ill-posed problem, as an observed pixel may appear dark either due to a dark surface material, or due to the incident light at that surface being reduced absent.
Our approach follows the recent success of coordinate-bast scene representation networks 14
43
in representing 3D scenes for high-quality view-synthesis 44 44]] These models decompose the scene into models of shape and radiance (emitted light), thereby enabling view synthesis. However, performing complete inverse rendering requires that radiance is decomposed further, into illumination and material appearances (SVBRDF) 66 [1] 63 A key component in learning these neural SVBRDF decomposition networks is the differentiable rendering [1]00 63 that generates images and gradients for the estimated lighting and SVBRDF parameters. These methods leverage traditional rendering techniques within modern deep learning frameworks enable Work done while Google.
44
46
49
35th Conference on Neural Information Processing Systems (NcurIPS 2021).

---

Specular
illum.
Examplary Training Imuges
Roughness
Normal
Novel View
Swap Illumination
Figure 1: Problem setting. Our neural-PIL. based technique decomposes images observed under unknown illumination into high-quality BRDF, shape and illuminations. This allows then synthesize novel views (targets shown in insets) and perform relighting or illumination transfer.
backpropagation. This is often expensive, rendering requires computing integrals over the incoming light at each 3D location in the scene. As a remedy, recent works [TT 63 approximate the incident light by spherical Gaussians (SG), thereby accelerating the illumination integration. However, these SG representations lack the capacity required to model or recover the shape and material properties of highly reflective objects images in complex natural environments.
In this work, we aim to replace the costly illumination integration step within these rendering approaches with learned network. Inspired by the real-time graphics literature on image-based lighting [28), we propose a novel pre-integrated lighting (PIL. network that converts the illumination integration process used in rendering into simple network query. Our neural-PP -IL takes input a latent vector representation for the environment map, the surface roughness, and an incident ray direction, and from them predicts integrated illumination estimate. This query-basel approach for light integration results in efficient rendering and thereby simplifies and accelerates rendering and optimization. This neural light representation is also significantly more expressive than the more commonly used SG representation, thereby enabling fidelityy renderings. The architecture of our neural-PIL illustrates this illumination pre-integration for different surface roughness levels.
conditional multi-layer perceptrons (MLP) with FiLM layers [12 Fig.
In addition, we also present smooth manifold auto-encoder (SMAE), based on interpolating auto-encoders [5], that can learn effective low-dimensional representations of light and BRDFs. This learned low-dimensional space serves strong regularizer or prior for constraining the solution space of BRDFs and illumination. These constraints are critical, due to the ill-posedness of problem setting. The smoothness of this manifold enables stable and effective gradient- based optimization of BRDF and light parameters. The neural-PIL, light-SMAE, and BRDF-SMAE networks pre-trat pre-trained (BRDFs). We integrate these component networks into our decomposition framework, in which we optimize 3D neural volume with shape and SVBRDF while also optimizing per-image illuminations.
dataset with high-qualit environment maps (illumination) and materials
We perform an empirical analysis on synthetic datasets, along with qualitative and quantitative visual results real-world datasets. We demonstrate that decomposition network using our neural-PIL can estimate more accurate shape and material properties compared to prior art. The 3D assets with material properties produced by our model can be used to generate high-quality relighting and view-synthesis results with finer details compared to existing approaches.
2 Related Work
Coordinate-basse MLPs allow spatial information be stored within the weights of neural network, thereby allowing the retrieval of information solely by querying coordinates [14] 46) (52) These methods have been combined with neural volume rendering 440 enable photorealistic results on novel view synthesis, well well-exemplified by NeRF [4]. In NeRF, a coordinate- based model is used model field of volumetric density and color, and renderings are produced by ray- merahhnggg through that neural volume. Though NeRF is capable of photorealistic renderings, it has many limitations
that have been explored by recent work, such as: no relighting capabilities 6 [1
63 long 39 60), extraction of 3D geometry and 64 This work addresses some of these challenges that
training times
39 (44) long inference times
25
materials [11, and generalization 12
enables relighting, extracts conventional 3D geometry and material estimate, and enables real-time

---

Roughness 0.01
Roughness 0.5
Roughness 1.0
Figure 2: Pre-integrated lighting. As the roughness of the material increases, the reflected radiance depends on a larger region the environment map. Brute-fort integrals over the environment map are expensive, hence propose porddintte-based MLP that is trained directly output the integrated illumination values conditioned on the surface roughness and view direction.
rendering (as our 3D assets are compatible with existing real-time rendering engines). The concurrent works of NeRD [T]. NeRV [51] and PhySG [63] are most clostly related to ours. These methods decomposes the scene into shape and analytical SVBRDF parameters. However, NeRV requires known illumination, and NeRD and PhySG employ a spherical Gaussian (SG) model, which is not capable of modeling detailed illumination patterns.
BRDF estimation is a challenging research problem that aims to estimate the appearance of physical material. For the highest accuracy results, measurements are performed under controlled laboratory conditions with known view and light positions 3 9 32 33
but this does not allow for the on-site capture of materials. Casual capture methods aim to solve this constraint by only requiring a camera, and sometimes a known light source. Often, machine learning techniques are leveraged to reduce the ambiguity through the use of data-drive priors and large datasets of BRDFs. Additional constraints from planar surfaces viewed under camera flash illumination
considered 20 estimation. This casual 30,
for single-shot [6 26 36 477, few-shot M or multi-shot [2] 17
18
setup be extended to estimating the BRDF and shape of objects 6 7
10
45
47 62 or
scenes [38 [88. Most of these methods are based on known active illumination.
limited number assumed to be responsible for the majority of
of light sources most often single illumination a scene. Relying on only natural, uncontrolled illumination adds several additional challenges due the drastically increased ambiguity across shape, illumination and BRDFs. Often these challenges are reduced by keeping the specular albedo cee-spaialll--vayiiie or by removing Other approaches require temporal traces and limit the casual capture
it entirely [35 59 63
setup 19 56
Illumination estimation from a single image is an inherently challenging problem. The task inherently linked to BRDF estimation, illumination affects appearance and is only indirectly observable from its interactions with surface materials. These two tasks are often solved in con- junction, sometimes by decomposing single object into shape, reflectance, and a global set of spherical Gaussians (SGs) [T] 63). Chen et al. [13 leverage a deep prior of environment maps with homogeneous materials, using invertible neural BRDF model. Li et al. (38] decompose entire scene into a simplified BRDF model with hemispherical SGs per point in the scene. The image of the environment in the background may be incorporated into prediction, shifting the problem to completion of the HDR environment map from sparse observations 221 50 55]. We not only learn deep prior but rendering aware network which capable of integrating the environment
illumination for specific surface roughness enabling rendering the entire hemisphere of incoming light with single evaluation.
Method
Given an image collection of an object captured under varying illumination conditions and camera viewpoints, we aim to jointly estimate the object's ss 3D) shape and spatially-varying BRDF, well as the illumination conditions of each image. Our input consists of a set of q images with s pixels each: C3 E g=x3;
{I.....g) along with per-pixel masks M, E {0, 1]xx1 indicating which pixels belong to the object. Our goal is to learn neural 3D volume V where, at each point x E R3 we estimate the BRDF parameters for the Cook-Torrance model [[5] E IR7 (diffuse b, E R3 specular b, E R roughness b, E IR), unit-length surface normal n E R3 and optical density a E IR. In addition, we also estimate latent vectors representing per-image illumination z! e 128 This problem statement corresponds to the practical application of recovering 3D model of some
3

---

real-world object (for e.g., statue or landmark) that has been photographed by different people at different times.
3.1
Preliminaries
Brief overview of neural radiance fields (NeRF). Our method is based on NeRF 44 which creates a neural volume for novel view synthesis using two Multi- Multi-Layer- learns coarse representation, which samples the 3D volume in a fixed sampling pattern, and the second MLP uses this knowledge to sample the volume in high-density areas at a finer resolution. The output of the MLP is view-dependent color e RR3 and optical density E IR for each given 3DD location a E TR3 and view direction d E TR3 In order to render the output color è E R for a camera ray r(t)
(MLPs). The first MLP
td, with ray origin o E R and view direction d, we approximate (via numerical quadrature) the integral ē(r) = f (()))( di with T(I) = exp(- - St. (()) di), using the near and far bounds of the ray in and t respectively [44]
Image formation and image-based lighting. NeRF directly models view-dependent color è each 3D location. Thus, a simple image formation process that integrates the color information along camera rays is sufficient to render images. In contrast, we want explicitly estimate an object material decomposition at each 3D location. We therefore must use a more explicit rendering formulation that relates image formation estimates the radiance L. R33 at along the outgoing view direction W. E R (w. -d): (ax, w.) = و far, w,, wo; b) (2x, (w,, n.) dw,, where fr is the BRDF evaluation, L, R incoming light, and w, E R is the incoming light direction. Using this single bounce rendering formulation, and ignoring exposure variation and tone-mapping, L., is equivalent to NeRF's color d
BRDFs and illumination. The rendering equation [27
The formulation can be split into diffuse and specular components:
Lo(x, wo) = ba
(ir,,w;aee [x, (w) n.) dwis + / fx, wi Wai by, by Li(x, wi wi)(w: n) dw's (1)
π J
diffuse
specular
where Sa now only represents the specular portion of the BRDF evaluation. Following Karis et al. (28), several parts of this integration can be pre-computed (9]
Tol wo)== (c)
(ba/n)L,(mn)
by (F n) Bolws n, b,) + Bi(w, n, (n,b))) TT. br)
(2)
diffise
specular
The illumination is now pre- integrated as: [,w,,, b,) [ w,,,,, L, w:)dw, which only depends on the mirrored view direction w, (which subsumes the surface normal n)) and the roughness by., where D describes the microfacet distribution (53) This light pre-integration is illustrated in Fig. Note that the same pre-integrated [, b,)) is queried twice: the diffuse part captures the entire hemisphere and therefore is parameterized by the surface normal n E R& and a diffuse roughness of 1. The specular part looks up [w,,, b,) for the reflected view direction w, E R and the specular roughness by. While the pre-integration already considers the microfacet distribution D, one must also account for shadowing, masking, and the Fresnel term. As shown in Karis et al. (28), these remaining parts can be pre-computed into two 2D lookup textures (LUT) Bo and B, indexed by (wo n)) and the roughness by. These are combined with the Fresnel term at normal incidence Folw., n.) (1 w. h)s with h ||M, + woll.
This pre-integration approach replaces the complex integration during shading with a set of simple additions and multiplications. We have integrated the core idea of this approach into efficient differentiable neural rendering framework, which allows for the optimization of geometry, BRDF, and illumination simultaneously via standard backpropagation. We further reduce the computational complexity by mimicking the pre-integration of [w,,,, by)) with simple query through our Neural- PIL that operates directly on a neural representation the illumination, as we will now explain.
3.2 Decomposition with Neural-PII.
Fig- shows the neural decomposition architecture which closely follows the architectures of NeR
4K and NeRD [T], but with some key differences. The coarse network learns a view and
illuminationn-deeedeen color whereas the fine network decomposes the scene into BRDF parameters.

---

(a) Coarse network.
(b) Decompasition network.
Position
Va,
m
Main Netorrk
Linesc
Donsity 0
Density a
Normal nt
Main Network
Linex
Reld
Lincar
Linear
Linex
DRDI SMAE
Linar
Linoe
Rendon
Position in
Color c
View Direction
Conditional Network
View Direction We
Reflect
Illmiiation Embedding
Light Direction
Linear
Illumination Embedding
Neural-PII.
Figure 3: Decomposition with Neural-PIL architecture. (a) Similar to NeRF-W our coarse network uses a latent illumination estimate to predict view-dependent color and density. (b) Pre-trained networks restrict the possible BRDF representation (BRDF-SMAE) and the incident lighting (PIL) to lower-dimensional spaces. A single evaluation of Neural-PIL returns pre-filtered illumination cone according to surface roughness. Using that, the BRDF estimate, and a surface normal (the unit-norm gradient of our density estimate), we render the shaded color c.
Coarse network. Like in NeRF [44), the aim of the coarse network is to obtain rough point density that helps in finer sampling for the following decomposition network. As illustrated in Figure Ga the coarse network takes 3D location æ, view direction w. and illumination embedding z' as input and predicts point density a and color e at æ. In contrast to NeRF, which estimates view-conditioned colors, we estimate both view and illumination conditioned colors, as our input images can be captured under varying illumination. Refer to the supplement for architecture details.
Decomposition network. The decomposition network estimates density a and BRDF embedding z E Id at each 3D location x in the implicit volume. As illustrated in Figure 3b. the conditional network in the coarse network is replaced by explicit rendering in the decomposition network. There are two key innovations in the decomposition network: 1) Use a novel pre-integrated light (PIL) network that results in efficient rendering while also representing the illumination with high fidelity. 2) We learn smooth oww-dimesiional manifolds to represent illumination and BRDF parameters, which serve as strong priors. We will now explain our rendering process, the Neural-PIL, and the smooth manifold auto-encoders (SMAE).
Rendering process. For rendering, we use the rendering formulation in Equation 2 We estimate normal at by computing the gradient Vo, of the density w.r.t. the input position (by passing gradients through the decomposition network). We also convert the BRDF embedding x" into BRDF parameters with our BRDF-SMAF. Unlike NeRF [44). which integrates sample colors along the camera ray, we first compute the expected termination of each ray (similar to depth map) with the sample densities along the camera ray, and then do the rendering only that point along each camera ray. At the ray termination positions, compute the integrated illumination L, [,,,, b,) and use Equation for rendering with the estimated BRDF and normal n.
The integration of the incoming light traditionally approximated by Monte Carlo sampling, in which illumination contributions from many directions are numerically integrated. The computation of the pre-integrated Li(w,, b,)) also involves either this costly numerical accumulation, or convolution performed on the complete environment map - though neither approach is practical within differential rendering engine. We therefore learn network that performs this light pre- integration, thereby converting the costly integral computation into a simple network query. The architecture of the Neural-PIL is visualized in Fig, 4 The Neural-PIL takes input the illumination embedding z', the incoming light direction w, and the roughness b, at point, and directly predicts the pre-integrated light b,),,, The aim of the Neural-PIL to first decode the illumination along the incoming mirror direction w, from the given embedding z' and then mimic the light pre-integration process for the surface roughness by. Following this general intuition, the Neural-PII. takes W, as input, and we condition the first few layers of the network with illumination 2 and condition later layer with roughness b,. The first few layers intended to decode all required
Neural-PII..
5

---

Roughness
Embedding
Diroction
Reconstruction Loss Lr
b,
Emhedding
Reconstruction
FiLM- SIREN
Input p
P
Cyclic Loss Co 2
Zn
Ple
Mapping Network
Adversarial Loss C.
Smoothness Loss L.
Linear LLyer
Linear Interpolation
Light L.
Figure 4: Neural-PIL. A coordinate- MLP returns the pre-integrated radiance for the query direction, where roughness determines the inte- gration footprint.
Figure 5: Smooth manifold auto-encoder. By imposing specific losses on interpolations be- tween input samples, our Smooth Manifold Au- toencoder encourages smooth embedding space.
illumination information for the given direction, and the later layers are intended to perform light integration conditioned on the material roughness.
For the Neural-PIL design, we leverage pi-GAN-like [2 architecture with FiLM-SIREN layers. Each FILM-SIREN layer [12] takes the modulating parameters (a scalar A and two vectors ,3 and y) to modulate the output y of the earlier linear layer followed by sine computation follows: ((y) = sin(Aor @ y + 3)), where @ denotes a Hadamard product. In our Neural-PII. we employ two mapping networks to predict modulating parameters(7) (3) for the FiLM-SIREN layers. The first mapping network generates the modulating parameters for the first layers from the illumination embedding z", while the second one for the penultimate layer from the given roughness b,.
In addition to converting a costly light integration process in the rendering into a simple network query, our Neural-PPi PILL has another key advantage. State-of- -fate-of-theeart the-art differentiable rendering frameworks [I] that work with BRDFs use either spherical Gaussian (SG) or spherical harmonic (SH) light representations, which both suffer from lack of fine details. Although one could represent fine illumination details with a large number of SG or SH bands, this parameter increase would also make the rendering prohibitively slow with high memory costs. In contrast, Neural-PIL. is an MLP that directly produces pre-integrated light required for the rendering. Our experiments also demonstrate that our Neural-PIL can represent finer details in illumination compared to SG representation (Sec. Fig. 6). See the supplementary for more Neural-PIL architecture details.
22 38 63
Smooth manifold auto-encoder (SMAE). Since jointly estimating 3D shape, materials and lighting a highly underconstrained problem, in order to converge to plausible solutions, must regularize optimization towards likely illuminations and BRDFs. For this, we learn lww-dimensional smooth manifolds that capture the data distribution of BRDFs and illuminations. In addition to acting as strong priors, optimizing on smooth manifolds (as opposed to directly optimizing on standard BRDF space, which need not be smooth) allows for more effective gradient-based optimization of reflectance decomposition for a given scene. Fig. iilllstrates our SMAE that we use to learn separate low-dimensional manifolds represent BRDF and illumination embeddings. Specifically, Interpolating Autoencoders 5 with several additional loss functions. The encoder network E takes input p (either BRDF or light environment map) and generates the latent embedding z, which is then passed onto the decoder network G that generates an input reconstruction p'. We then randomly sample two latent vectors from the mini-batch: z, and z;; followed by sampling m E N linearly interpolated embeddings that are uniformly spaced between za and z5: {x n 1, 2, . m. We pass each of these interpolated latents z/, through the decoder G and the encoder E to obtain &'. and 2/, respectively. Using the four losses depicted in Fig. 5he encoder and decoder networks are trained jointly. One is the standard reconstruction loss L. between input p and reconstruction p'. In addition, we add a discriminator network on P', and use the standard adversarial loss La used LSGAN 41], which ensures that the interpolated latent vectors can generate plausible data. We enforce bijective mapping of the encoder and the decoder with a cyclic loss L., which is the L2lloss between the interpolated latents {z(, ) and their re-estimated counterparts (#)). Lastly, to ensure that the leamned embedding space is smooth, we impose smoothness loss C, the gradient of decoder
use

---

G' w.r.t the interpolating scalar value a: L, 1 1/m (VaG((z) The total loss to train SMAE a combination of the losses: C + + X2L. Azhz-
Despite being only 7-dimensional, the space of the use is too unconstrained for task, and imposes strong correlations between the diffuse and specular terms of real-world materials. We therefore train BRDF-SMAE with an MLP encoder and decoder that maps these 7D parameters into 4D latent embeddings z" E E using a dataset of real-world BRDF material collections [T]. Similarly, real-world illuminations exhibit significant statistical regularities: lights are more likely to be tinted blue or yellow, and brighter light is more likely to coming from above than below. To capture this regularity, we train Light-SMAE with CNN encoder and decoder on a dataset of 320 environment maps from [61). We then map the 128 x 256 2D environment maps onto a 128 dimensional smooth latent space, x' E RR 128 We provide more network and training details for BRDF-SMAE and Light-SMAE. the supplementary.
BRDF representation 15 that
Training. Since we have several networks in our decomposition learning pipeline, we will briefly explain the overall training protocol here with more details in the supplementary. We first train Light-SMAE and BRDF-SMAE with dataset of environment maps and BRDFs respectively. The Neural-PIL. network is then trained with the manifold created by the frozen Light-SAME encoder. This separation is mainly done ease the memory requirements of training both networks jointly. With the frozen BRDF'SMAE's decoder in the decomposition network and with Neural-PIL in the rendering step, we jointly optimize both the coarse network and the decomposition network for a given set of scene images. For stability, we only optimize the illumination embedding 2 via decomposition network and we do not backpropagate the loss signal onto illumination in the coarse network. More training details can be found in the supplement.
Experiments
We evaluate approach w.r.t different baselines on the aspects of BRDF and light estimation, view synthesis, and relighting.
Ground Truth MC Illumination SGs Illumination PIL Illumination
Roughness Roughness 0..2
Baselines. The closest work to ours is NeRD [ which forms our primary com- parison across different evaluations. To our knowledge, there exists no other published work that tackles the same problem of esti- mating shape, illumination and BRDF from images of varying illumination. For view synthesis, we also compare with NeRF [44. For BRDF evaluations,
128x256
168 Parameters
128 Parameters
Figure 6: Neural-PIL vs. spherical Gaussian (SG) vs. Monte-Carlo (MC) integration renderings. With known geometry and reflectance, we optimize using MC integration for the direct illumination, SGs as well as our latent illumination via Neural-PII. This figure shows final renderings with the optimized light parameters, while the recovered illumination is shown in the insets. Despite Neural-P1 having fewer param- eters, is able to recover more detailed environment maps and thereby produce accurate renderings.
also compare with Li et al. [37] which does BRDF decom- position from single image. In addition, we combine Li el al. (37) with NeRF [44] to create baseline that is closer our problem setting.
Datasets. To enable the comparisons with NeRD (I]. we use the publicly released dataset used in [TT] which provides syn- thetic (Chair, Globe, Car) and 4 real-world scenes. Two real-world datasets consist of multi-view captures with fixed, unknown illumination (Cape), relatively varying illumination (Head) and two others where the illumination varies with each image (Gnome, MotherChild), In addition, we also present view synthesis results on datasets (Ship, Chair, Lego) used in NeRF [4].
Fildelity of NeLL Since Neural-PIL forms the key component of our decomposition frame- work, we evaluate its learned light representation against a more commonly used spherical Gaussians (SG) representation. Additionally, we add a baseline which directly optimizes environment map using Monte-Carlo (MC) integration. We render a simple metallie sphere using an unseen environ- ment as shown in Fig. 6 with two different roughness levels 0.2 and 0.5. Assuming known roughness
7

---

View Relight
Diffuse
Specular
Roughness
Normal
Environment Render
GT
Re-Render
View
Cape
NeRD
Glabe
Ous
(a) BRDF and light estimation.
(b) View synthesis and relighting.
GT
NeRF
NeRD
Ours
GT
NeRF
NeRD
Ours
GT
NeRF
NeRD
Ours
(c) View synthesis with NeRF, NeRD and our approach.
Figure 7: Visual comparisons. (a) Our model produces more accurate BRDF and illumination estimates, which results in more faithful rendering results. (b) To evaluate view synthesis and relighting we keep the camera and light fixed (col 2), then move the camera (col 3), and then adjust the lighting (col 4). (c) Even when using a single illumination (the problem setting used by NeRF) our method produces shape estimates with fewer artifacts and more detail than both NeRF or NeRD.
and shape, we optimize for SG illumination using the SG-based differentiable rendering used in NeRD [I]]. Similarly, we optimize the latent illumination representation using our Neural-PIL-based renderer. For the MC baseline, we leverage BRDF importance sampling, which based on the surface roughness describes how the rays would likely scatter. Here, we cast 128 samples-per-pin
(spp) based on the BRDF towards the environment map with resolution of 128 x 256. The resulting estimated MC, SG illumination and Neural-PIL. illuminations are shows Fig. 6 Compared to the SG illumination model with 24 lobes and 168 parameters, our recovered illumination vector z' with only 128 dimensions captures more details, especially the high-frequency light panels. This leads to significantly reduced rendering error for both roughness values even though the illumi- nation prediction more ambiguous for rougher materials. While the MC integration could easily recover detailed highlights, the remaining areas not recovered well. Besides the improved quality, Neural-PII. based rendering is also much faster. Rendering million samples with our Neural- -PL network takes just 1.86 ms compared to 210 ms rendering with 24 SGs. Table shows average PSNR on rendered spheres with more visual results similar to Fig. 6 in the supplements. Our method outperforms both baselines in reconstruction quality.
Ablation study. To showcase the effectiveness of our novel additions, we perform an ablation of the BRDF-SMAE. Table shows the influence of the BRDF-SMAE on material estimation. These are the PSNR values on the 3 synthetic scenes under varying illumination. It is clear from the table that, especially in estimating the specular parameter, using BRDF-SMAE, improves the results drastically. As this parameter is also tied to the diffuse color a degradation in performance is expected. The roughness parameter - even though it is uncorrelated to diffuse and specular - also improved most likely due to improved color parameters. For the ablation of Neural PIL network, one can refer to NeRD IT] as a baseline that neither uses BRDF-SMAE nor Neural PIL. PSNR metrics in Table 3a shows that our method can result in better decomposition compared to [Л].
BRDF evaluations. Following the results in NeRD [1]. Table 3a shows the BRDF estimation metrics for different techniques computed on the scenes Globe, Car and Chair. When compared with NeRD, our approach resulted in better diffuse and roughness parameters. Only the prediction of the specular parameter worse compared parameterization, which can reduce some ambiguity but also limits the space of expressible materials. visual comparison is shown in Fig. 7 demonstrraiinn clear visual improvements w.r.t. [37. One can observe higher frequency details in the environment map using our approach compared to NeRD and the final renderings also show that our result closer to GT rendering (top-right). Refer to the supplementary material for more visual results.
NeRD. This may be due NeRD's basecolor-metallic
8

---

Roughness
MC
SGs
Neural- PIL (Ours)
Parameter
w/o BRDF SMAE
Ours
34,88
31.57
38.76
Diffuse 11.87
20.22
Specular 9.24
16.84
0.5
35.14
28.98
35.28
Roughness 16.51
24.82
Table 1: Better illumination estimates with Neural-PTL. Average PSNR with 6 rendered spheres shows that Neural-PIL achieves better PSNR over the spherical Gaussian (SG) and Monte- Carlo integration (MC) baselines. More accurate illuminations also enables improved BRDF decom- position and relighting.
Table 2: Ablation study. Average PSNR of BRDF estimation on synthetic scens under varying illumination demonstrates the posi- tive influence of using the BRDF-SMAE to constrain the BRDF parameter space.
(a) BRDF decomposition
(b) View synthesis
(c) View synthesis and relighting
PSNRT
[14
[1)
Ours
Synthetic
Real-World
Synthetic
Real. World
Diffuse
1.06
1.15
18.24
20.22
Method PSNRT
SSIMF PSNRT
SSIMT
Method PSNRT SSIM
PSNRT SSIMT
Specular
25.70
16.84
NeRF
34.24
0.97
23.34
0.85
NeRF
21.05
0.89
20.11
0.87
Rouginess
17.18
17.28
15.00
24.82
NeRD
30.07
0.95
23.86
0.88
NeRD
27.96
25.81
0.95
Ours
30.08
0.95
23.95
0.90
Ours
29.24
0.96
26.23
0.95
Table 3: Comparisons with baselines. (a) A comparison against methods for BRDF decomposition under unknown illuminations, where we see that our model performs well (consistent with our improved relighting performance). (b) An evaluation of view-synthesis (without relighting) under single illumination, where our model performs well despite this not being our primary task. (c) Here, input images taken under different illumination conditions, are required. Our model outperforms both baselines by a significant margin: NeRF (which is not intended to address this task) but also NeRD (which targets this same problem statement).
joint relighting and view synthesis
View synthesis and relighting. On the datasets with fixed illumination (Cape, NeRF-Ship. NeRF- Chair, NeRF-Lego), we directly compare our renderings with existing novel view synthesis techniques (both NeRF 44 and NeRD [1] here). Table TableBBjshows Shoshows novel view evaluation metrics on these datasets with fixed illumination. Results show that our results are better than NeRD showing the improved capture of view-dependent effects. NeRF still outperforms NeRD and our method in the synthetic fixed illumination setting, but outperformed on the real-world fixed illumination dataset. However, the fixed illumination might in general limit the decomposition capabilities, as shadows do appear always at the same surface locations and therefore might not be correctly disentangled from the BRDF.
On datasets with varying illumination across images (Gnome, MotherChild, Chair, Car, Globe, Head), we need to do both view synthesis and relighting to generate novel unseen test views. Table [3c displays the results these datasets. NeRF 44 can not do relighting and is included as weak baseline. The results are significantly better than NeRD and shows that our method can more faithfully estimate the underlying parameters resulting in better relighting under novel illumination conditions.
Fig. shoss a couple of results with view synthesis and relighting. The renderings demonstrate realistic view synthesis including re-lighting. Fig. 7clshows novel view synthesis comparison with NeRF and NeRD on Cape scene captured with fixed illumination. Despite NeRF being a strong baseline it could not recover the complete surface due to the reflectiveness. On the other hand, our view synthesis results are more close to the GT on unseen views compared to both NeRF and NeRD.
5 Conclusion
We presented novel reflectance decomposition technique that can estimate shape, per-image illumination and BRDF from images captured in unknown and varying illuminations. The key innovation is the neural-PIL network that can replace costly light integration during rendering with simple network query resulting in a fast and practical differentiable rendering with high-fidelity illumination. In addition, we propose novel learning techniques with SMAES that can learn effective low-dimensional smooth manifolds for both BRDF and light representations. Experiments on both synthetic and real-world scenes demonstrate superior decomposition results along with better novel view synthesis and relighting in comparison to prior art

---

Limitations. While our techniques make significant strides in the areas of differentiable rendering as well as shape and material decomposition, several challenges still remain in this complex problem setting. Our approach can not handle inter-reflections. Concurrent works such as NeRV [51] are capable of handling inter-reflections and shadowing but only for known illumination. Due to the large ambiguity between the interplay of all effects, solving everything jointly is an extremely challenging problem. Another limitation of our method is that we can not guarantee to converge to the correct underlying BRDF, reflectance and illumination. Our loss is only photometric and, therefore, we find one solution which explains all input images, e,g,, adding a new input image might converge to a different representation, new effects visible. Also, while neural network is capable of producing higher frequency illumination with fewer parameters compared to standard representations such SGs, mirror like reflections are still not possible and therefore can limit the reconstruction quality when mirror-like surfaces are present in the scene.
Broader impact. As is generally the case in machine learning, biases in the data used during training may result in biases in the learned model. Our pre-trained networks for BRDFs and incident illumination serve as priors on materials and lighting conditions, and so any bias the training data used for pre-training those models may result in bias in our estimations of materials and illumination. the presented techniques were applied to human subjects (which we do not do here) the performance of the model might vary function of the subject's skin color for skewed training distributions.
The purpose of our model to better enable the creation of highly accurate 3D models from photographs, which could then be used as visual effects in film or television, or in video games. Currently, the creation or acquisition of 3D assets largely the domain of specialized CGI artists. Improved tools for automating this task may lower the barrier to entry into these careers, which may be seen as harming job opportunities for artists already working in this area. Despite this, we are hopeful that the commoditization of tools for 3D model acquisition will have net positive impact by allowing a wider range of people to automatically construct high-fidelity 3D models from their image collections.
Acknowledgments and Disclosure of Funding
This work has been funded by the Deutsche Forschungsemmeinschatt (DFG, German Research Foun- dation) under Germany's 's Excellence Strategy - EXC number 2064/1 Project number 390727645 and SFB 1233, TP 02- Project number 276693517. It was supported by the German Federal Ministry of Education and Research (BMBF): Tübingen AI Center, FKZ: 011S18039A.
References
[1] Miika Aittala, Timo Aila, and Jaakko Lehtinen. Reflectance modeling by neural texture synthesis. In ACM Transactions on Graphics (ToG), 2018.
[2] Rachel Albert, Dorian Yao Chan, Dan B. Goldman, and James F. O'Brian. Approximate SvBRDF estimation from mobile phone video. In Eurographies Symposium on Rendering. 2018.
[3] Louis-Philippp Asselin, Denis Laurendeau, and Fran-François Lalonde. Deep SVBRDF estima- tion on real materials. In International Conference on 3D Vision (3DV), 2020.
[4] Jonathan T. Barron and Jitendra Malik. Shape, illumination, and reflectance from shading. In IEEE Transactions on Pattern Analysis and Machine Intelligence (PAMI), 2015.
[5] David Berthelot, Colin Raffel, Aurko Roy, and Ian Goodfellow. Understanding and improving interpolation in autoencoders via adversarial regularizer. International Conference on Learning Representations (ICLR), 2019.
[6] Sai Bi. Zexiang Xu, Pratul Srinivasan. Ben Mildenhall. Kalyan Sunkavalli, Miloš Hašan, Yannick Hold Good-Geeoffro, David Kriegman, and Ravi Ramamoorthi. Neural reflectance fields for appearance acquisition. ArXiv e-prints, 2020.
[7] Sai Bi, Zexiang Xu, Kalyan Sunkavalli, Miloš Hašan, Yannick Hold-Geoffroy, David Kriegman, and Ravi Ramamoorthi, Deep reflectance volumes: Relightable reconstructions from multi-view photometric images. In European Conference on Computer Vision (ECCV), 2020.
10

---

[8] Sai Bi, Zexiang Xu, Kalyan Sunkavalli, David Kriegman, and Ravi Ramamoorthi. Deep 3d capture: Geometry and reflectance from sparse multi-view images. In IEEE Conference on Computer Vision and Pattern Recognition (CVPR), 2020.
[9] Mark Boss, Fabian Groh, Sebastian Herholz, and Hendrik A. Lensch. Deep Dual Loss BRDF Parameter Estimation. In Workshop on Material Appearance Modeling, 2018.
[10] Mark Boss, Varun Jampani, Kihwan Kim, Hendrik P.A. Lensch, and Jan Kautz. Two-shot spatially-varying BRDF and shape estimation. In IEEE Conference on Computer Vision and Puttern Recognition (CVPR), 2020.
[11] Mark Boss, Raphael Braun, Varun Jampani, Jonathan T. Barron, Ce Liu, and Hendrik P.A. Lensch. NeRD: Neural reflectance decomposition from image collections. In IEEE International Conference Computer Vision (ICCV), 2021.
[12] Eric Chan, Marco Monteiro, Petr Kellnhofer, Jiajun Wu, and Gordon Wetzstein. pi-GAN: Periodic implicit generative adversarial networks for 3D-aware image synthesis. In IEEE Conference on Computer Vision and Pattern Recognition (CVPR), 2021.
[13] Zhe Chen, Shohei Nobuhara, and Ko Nishino. Invertible neural BRDF for object inverse rendering. In European Conference on Computer Vision (ECCV), 2020.
[14] Zhiqin Chen and Hao Zhang. Learning implicit fields for generative shape modeling. IEEE Conference on Computer Vision and Pattern Recognition (CVPR), 2019.
[15] Robert L. Cook and Kenneth E. Torrance. A reflectance model for computer graphics. ACM Transactions on Graphics (ToG), 1982.
[16] Valentin Deschaintre, Miika Aitalla, Fredo Durand, George Drettakis, and Adrien Bousseau. Single-image SVBRDF capture with a rendering deep network. In ACM Transactions on Graphics (ToG), 2018.
[17] Valentin Deschaintre, Miika Aitalla, Fredo Durand, George Drettakis, and Adrien Bousseau. Flexible SVBRDF capture with multi-image deep network. In Eurographics Symposium Rendering, 2019.
[18] Valentin Deschaintre, George Drettakis, and Adrien Bousseau. Guided fine-tuning for large- scale material transfer. In Eurographics Symposium on Rendering, 2020.
[19] Yue Dong, Guojun Chen, Pieter Peers, Jianwen Zhang, and Xin Tong. Appearance-from--ooiis Recovering spatially varying surface reflectance under unknown lighting. ACM Transactions on Graphics (SIGGRAPII ASIA), 2014.
[20] Duan Gao, Xiao Li, Yue Dong, Pieter Peers, and Xin Tong. Deep inverse rendering for high- resolution SVBRDF estimation from an arbitrary number of images. In ACM Transactions on Graphics (SIGGRAPH), 2019.
[21] Marc-André Gardner, Kalyan Sunkavalli, Ersin Yumer, Xiaohui Shen, Emiliano Gambaretto, Christian Gagné, and Jean- -rannFrnçois Lalonde. Learning to predict indoor illumination from single image. ACM Transactions on Graphics (ToG), 2017.
[22] Marc-Andre Gardner, Yannick Hold-Geoffro, Kalyan Sunkavalli, Christian Gagne, and Jean- Francois Lalonde. Deep parametric indoor lighting estimation. In IEEE International Confer- ence on Computer Vision (ICCV), 2019.
[23] Dan B. Goldman, Brian Curless, Aaron Hertzmann, and Steven M. Seitz. Shape and spatially- varying BRDFs from photometric stereo. IEEE Transactions Pattern Analysis and Machine Intelligence (PAMI), 2009.
[24] Tom Haber, Christian Fuchs, Phillipe Bekaer, Hans-Peter Seidel, Michael Goesele, and Hendrik A. Lensch. Relighting objects from image collections. In IEEE Conference on Computer Vision and Pattern Recognition (CVPR), 2009.

---

[25] Peter Hedman, Pratul P. Srinivasan, Ben Mildenhall, Jonathan T. Barron, and Paul Debevec. Baking neural radiance fields for real-time view synthesis. In IEEE International Conference on Computer Vision (ICCV), 2021.
[26] Philipp Henzler, Valentin Deschaintre, Niloy J Mitra, and Tobias Ritschel. Generative modelling of BRDF textures from flash images. ACM Transactions on Graphics (SIGGRAPH ASIA), 2021.
[27] James T. Kajiya. The rendering equation. In ACM Transactions on Graphics (SIGGRAPH), 1986.
[28] Brian Karis. Real shading in unreal engine Technical report, Epic Games, 2013.
[29] Jan Kautz, Pere-Pau Vázquez Alcocer, Wolfgang Heidrich, and Hate- Seidel. A unified approach to prefiltered environment maps. Eurographics Symposium on Rendering, 2000.
[30] Berk Kaya, Suryansh Kumar, Carlos Oliveira, Vittorio Ferrari, and Luc Van Gool. Uncalibrated neural inverse rendering for photometric stereo of general surfaces. In IEEE International Conference Computer Vision (ICCV), 2021.
[31] Petr Kellnhofer, Lars Jebe, Andrew Jones, Ryan Spicer, Kari Pulli, and Gordon Wetzstein. Neural lumigraph rendering. In IEEE Conference on Computer Vision and Pattern Recognition (CVPR), 2021.
[32] Jason Lawrence, Szymon Rusinkiewicz, and Ravi Ramamoorthi, Efficient BRDF importance sampling using factored representation. ACM Transactions on Graphics (ToG), 2004.
[33] Hendrik P. A. Lensch, Jan Kautz, Michael Goscle, and Hans-Peter Seidel. Image-based reconstruction of spatially varying materials. In Eurographics Conference on Rendering, 2001.
[34] Hendrik P.A. Lensch, Jochen Lang, M. Sa Asla, and Hans-Peter Seidel. Planned sampling of spatially varying BRDFs. In Computer Graphies Forum, 2003.
[35] Xiao Li, Yue Dong, Pieter Peers, and Xin Tong. Modeling surface appearance from a single photograph using self augmeneddd convolutional neural networks. In ACM Transactions on Graphics (ToG), 2017.
[36] Zhengqin Li, Kalyan Sunkavalli, and Manmohan Chandraker. Materials for masses: SVBRDF acquisition with a single mobile phone image. In European Conference on Computer Vision (ECCV), 2018
[37] Zhengqin Li, Zexiang Xu, Ravi Ramamoorthi, Kalyan Sunkavalli, and Manmohan Chandraker. Learning to reconstruct shape and eatiaaly-vvryiin reflectance from a single image. In ACM Transactions on Graphics (SIGGRAPH ASIA), 2018.
[38] Zhengqin Li, Mohammad Shafici, Ravi Ramamoorthi, Kalyan Sunkavalli, and Manmohan Chandraker. Inverse rendering for complex indoor scenes: Shape, spatially-varying lighting and SVBRDF from single image. In IEEE Conference on Computer Vision and Pattern Recognition (CVPR), 2020.
[39] Lingjie Liu, Jiatao Gu, Kyaw Zaw Lin, Tat-Seng Chua, and Christian Theobalt. Neural sparse voxel fields. In Advances in Neural Information Processing Systems (NeurIPS), 2020.
[40] Stephen Lombardi, Tomas Simon, Jason Saragih, Gabriel Schwartz, Andreas Lehrmann, and Yaser Sheikh. Neural volumes: Learning dynamic renderable volumes from images. ACM Transactions on Graphics (ToG), 2019.
[41] Xudong Mao, Qing Li, Haoran Xie, Raymond Y.K. Lau, Zhen Wang, and Stephen Paul Smolley. Least squares generative ad adversarial networks. In IEEE International Conference on Computer Vision (ICCV). 2017.
[42] Ricardo Bartin-Brualla, Noha Radwan, Mehdi S. M. Sajjadi, Jonathan T. Barron, Alexey Doso- vitskiy, and Daniel Duckworth. NeRF in the Wild: Neural Radiance Fields for Unconstrained Photo Collections. In IEEE Conference on Computer Vision and Pattern Recognition (CVPR), 2021.
12

---

[43] Lars Mescheder, Michael Oechsle. Michael Niemeyer, Sebastian Nowozin, and Andreas Geiger. Occupancy networks: Learning 3d reconstruction function space. In IEEE Conference on Computer Vision and Pattern Recognition (CVPR), 2019.
[44] Ben Mildenhall, Pratul Srinivasan, Matthew Taneik, Jonathan T. Barron, Ravi Ramamoorthi, and Ren Ng. NeRF: Representing scenes as neural radiance fields for view synthesis. In European Conference on Computer Vision (ECCV), 2020.
[45] Giljoo Nam, Diego Gutierrez, and Min H. Kim. Practical SVBRDF acquisition of 3d objects with unstructured flash photography. In ACM Transactions on Graphics (SIGGRAPH ASIA), 2018.
[46] Jeong Joon Park, Peter Florence, Julian Straub, Richard Newcombe, and Steven Lovegrove. Deepsdf: Learning continuous signed distance functions for shape representation. In IEEE Conference on Computer Vision and Pattern Recognition (CVPR), June 2019.
[47] Shen Sang and Manmohan Chandraker. Single-shot neural relighting and SVBRDF estimation. In European Conference on Computer Vision (ECCV), 2020.
[48] Soumyadip Sengupta, Jinwei Gu, Kihwan Kim, Guilin Liu, David W. Jacobs, and Jan Kautz. Neural inverse rendering of an indoor scene from a single image. In IEEE International Conference Computer Vision (ICCV), 2019.
[49] Vincent Sitzmann, Julien N.P. Martel, Alexander W. Bergman, David B. Lindell, and Gordon Wetzstein. Implicit neural representations with periodic activation functions. In Advances in Neural Information Processing Systems (NeurIPS), 2020.
[50] Shuran Song and Thomas Funkhouser. Neural illumination: Lighting prediction for indoor environments. IEEE Conference on Computer Vision and Pattern Recognition (CVPR), 2019.
[51] Pratul P. Srinivasan, Boyang Deng, Xiuming Zhang, Matthew Tancik, Ben Mildenhall, and Jonathan T. Barron. NeRV: Neural reflectance and visibility fields for relighting and view synthesis. In IEEE Conference on Computer Vision and Pattern Recognition (CVPR), 2021.
[52] Matthew Tancik, Pratul P. Srinivasan, Ben Mildenhall, Sara Fridovich- Keil, Nithin Raghavan, Utkarsh Singhal, Ravi Ramamoorthi, Jonathan T. Barron, and Ren Ng. Fourier features let networks learn high frequency functions in low dimensional domains. Advances in Neural Information Processing Systems (NeurIPS), 2020.
[53] Bruce Walter, Stephen R. Marschner, Hongsong Li, and Kenneth E. Torrance. Microfacet models for refraction through rough surfaces. In Eurographics Symposium on Rendering, 2007.
[54] Qianqian Wang, Zhicheng Wang, Kyle Genova, Pratul Srinivasan, Howard Zhou, Jonathan T. Barron, Ricardo Martin-Brualla Noah Snavely, and Thomas Funkhouser. Ibrnet: Learning multi-view maage- rendering. In IEEE Conference on Computer Vision and Pattern Recognition (CVPR), 2021.
[55] Henrique Weber, Prévost. Donald, and Jean-François Lalonde. Learning to estimate indoor lighting from 3d objects. In International Conference on 3D Vision (3DV), 2018
[56] Rui Xia, Yue Dong, Pieter Peers, and Xin Tong. Recovering shape and spatially- surface reflectance under unknown illumination. ACM Transactions on Graphics (SIGGRAPH ASIA), 2016.
[57] Zexiang Xu, Sai Bi, Kalyan Sunkavalli, Sunil Hadap, Hao Su, and Ravi Ramamoorthi. Deep view synthesis from sparse photometric images. ACM Transactions on Graphics (ToG), 2019.
[58] Zexiang Xu et Deep image-based relighting from optimal sparse samples. ACM Tronsactions on Graphics (ToG), 2018.
[59] Wenjie Ye, Xiao Li, Yue Dong, Pieter Peers, and Xin Tong. Single image surface appearance modeling with self-augmented cnns and inexact supervision. Computer Graphics Forum, 2018.
13

---

[60] Alex Yu, Ruilong Li, Matthew Tancik, Hao Li, Ren Ng, and Angjoo Kanazawa. PlenOctrees for real-time rendering of neural radiance fields. In IEEE International Conference on Computer Vision (ICCV), 2021.
[61] Greg Zaal. Hdri haven, 2019. https///hrrihavenccom/
[62] Jianzhao Zhang, Guojun Chen, Yue Dong, Jian Shi, Bob Zhang, and Enhua Wu. Deep inverse rendering for practical object appearance scan with uncalibrated illumination. Advances in Computer Graphics, 2020.
[63] Kai Zhang, Fujun Luan, Qianqian Wang, Kavita Bala, and Noah Snavely. PhySG: Inverse rendering with spherical Gaussians for physics-based material editing and relighting. In IEEE Conference on Computer Vision and Pattern Recognition (CVPR), 2021.
[64] Yuxuan Zhang, Wenzheng Chen, Huan Ling, Jun Gao, Yinan Zhang, Antonio Torralba, and Sanja Fidler. Image GANs meet differentiable rendering for inverse graphics and interpretable 3d neural rendering. In International Conference on Learning Representations (ICLR), 2021.
14


Reviews:
[{"review_id": "tHyXaowcci", "reviewer": "Reviewer_rECe", "summary": "This paper proposes a method for joint shape, BRDF and illumination estimation using a NeRF-like pipeline. Compared with NeRF, the proposed method can also solve for illuminations, and it  outperforms NeRD on both quality and efficiency due to a novel network query-based pre-integrated lighting approach.", "questions": "", "limitations": "", "rating": 6, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "### Strengths:\n\n1. The idea of network query-based pre-integrated lighting is novel, and combining it with smooth manifold auto-encoder (SMAE) in a differentiable  pipeline is non-trivial.\n   \n2. The proposed Neural-PIL achieves the state-of-the-art in both quality and efficiency.\n\n\n### Weaknesses:\n\n1. The flowchart in Fig. 3 can be improved, e.g., I did not see Light-SMAE in this figure. Moreover, the training strategy in L251-257 seems complex, e.g., why freeze BRDF-SMAE’s decoder only, any experimental support? It would be good if the authors can elaborate more on this in rebuttal.\n   \n2. The SMAE ablation study (in supplementary) provides nice visual comparisons, however it lacks quantitative comparisons. In Fig. A1, for example, although Neural-PIL shows finer details than SGs, its color looks blueish (also in Fig. 6), therefore it's hard to tell whether the PSNR/SSIM are better. I'd like to hear some explanations on this in rebuttal.\n", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "See Weaknesses above.", "review_text": "", "extra_scores": {"needs_ethics_review": "No", "ethics_review_area": "", "time_spent_reviewing": "4"}, "extra_text": {}}, {"review_id": "lAaar049iK4", "reviewer": "Reviewer_pHT7", "summary": "The paper presents a method to reconstruct geometry and BRDF from a set \nof multi-view images take under various illuminations. It learns a low \ndimensional manifold for illuminations and BRDFs and use these data priors \nto regularize the estimation. It also applies a network to predict the \nintegral of light for efficient rendering under environment maps. It \nshows results on both synthetic and real scenes. \n", "questions": "", "limitations": "", "rating": 6, "confidence": 5, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "\n### Strengths:\n1. Using networks to predict the integral of lighting is novel and \nreduces the computation cost for rendering under complex lighting conditions.\n\n2. Using auto-encoders to learn a data manifold for BRDFs and lighting is interesting \nand helps regularize the prediction.  \n\n### Weakness: \n\n1. The results seem to be incomplete. The paper says it has results on    \n4 real scenes, however, only results on the cape scene are showed. \nMoreover, for the cape scene, only several images from different views are shown.\nIt is desired to show more video results on more diverse scenes under different illumination conditions \nand viewpoints, and include comparisons to baselines. Similarly\nI cannot find the view synthesis results on Ship, Chair, and Lego scenes.  \nFrom the current results, I cannot judge whether the proposed method can \ngenerate convincing results on real scenes. \n\n2. In terms of comparison to NeRF and NeRD in Figure 7, it is confusing to me \nwhy the proposed method would outperform NeRF even for view synthesis tasks. \nIn fact from the video, I think the results of NeRF has more details while the \nproposed method generates blurry renderings. It is desired to show quantitative \nand qualitative results on the NeRF dataset to further validate this claim. \n\n3. In the novel view and relighting results in the video, ground truth should \nbe included. The current results are not fully convincing, for example, there \nare clear artifacts on the globe scene. \n\n4. It is not clear to me whether the proposed method requires images under \nvarying illuminations. Requiring images under different environment \nlightings will make the acquisition very difficult and make the method impractical.   \nHow does the performance change with images under fixed and varying illuminations?\n\n5. In terms the comparison on lighting, while the paper claims Neural-PIL has \nbetter performance, from the results in Figure 6 (second row) and the results in \nsupp (Figure A1), I think the Neural-PIL illuminations often suffer from incorrect \ncolors and highlight positions. Therefore it's not convincing to me that Neural-PIL \nis definitely better. To better show this, a quantitative comparison should be included.\n\n6. The paper uses an auto-encoder to regularize the material estimation. While the paper\ndoes ablation on different loss terms in training the auto-encoder, a more important \nand intuitive comparison is how does the full pipeline work with and without using \nthe auto-encoder? How does the method work by directly predicting the BRDF coefficients\nwithout the auto-encoder? \n\n7. Another important component to validate is the accuracy of the PIL network. It is not \ncertain to me that such a network can be trained effectively without direct supervision.\nA comparison between the predicted integrated lighting and the GT integral on synthetic \ndata would be helpful.  \n\nOverall while I like some components and design of the paper, I think the paper needs \nsignificant improvement in terms of visual quality and experiments to further validate the \neffectiveness of the method. ", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "Please see main review.", "review_text": "", "extra_scores": {"needs_ethics_review": "No", "ethics_review_area": "", "time_spent_reviewing": "2"}, "extra_text": {}}, {"review_id": "bwVgq6jFMO9", "reviewer": "Reviewer_6Uz6", "summary": "The method suggests to learn a 3D model of shape, reflectance and light from a set of 2D images. They use a forward model that combines the three factors and invert it by optimizing its parameters to explain the 2D images. The key is in the specific parametrization of the three factors.\n\nThe first factor is a classic implicit model of shape. Rays are cast from the views of the 2D input images to find 3D depth. They define it as expected opacity-weighted depth, which might have its problem on (soft) silhouette edges. Mode finding, via Iterated Re-weighted least squares would be still fast and maybe more correct, not resulting in depth values floating in the air.  This said, this part is common.\n\nThe reflectance is parametrized as a spatial field of 4D latent codes. Each latent code itself is an embedding of the 7D Cook-Torrence parameters over the MERL database. The paper says the 7D codes are \"not constrained enough\", but no evidence for this is given. The embedding is pre-trained using a modern smooth auto-encoder.\n\nThe illumination is parametrized in a similar way: a smooth auto-encoding of a collection of environment maps into a low-dimensional code.\n\nThe work dwells a bit too much on how this is a graat AE'ing, but it is just shopped previous work, so it should maybe be phrased more neutral. \n\nNow the key innovation is how the two latter factors combine. The direct way is to solve an integral where the BRDF slice is convolved with the illumination. in SH this is a dot product, but at the expense of angular details. Instead, the suggestion is to pick up the realtime-rendering idea of pre-filtered illumination. This allows to get the integral estimate in constant time, after pre-processing the envmap. Now instead of preprocessing the envmap (which would be possible, there is only one illumination per scene and pre-process = pyramidal blur = constant time), authors suggest to use a network that directly returns pre-filtered results. For this idea to be fully appreciated, one would need to look at the two ablations chosen by those familiar with game graphics: pixel basis (MIP maps) and MC directly. This would need to be compared against an NN doing this pre-filtering job. It is a cool idea, but does it do anything in the end? Comparison is only made against SH, which is for the birds to those learned in graphics. The pixel basis ablation is: take the envmap code, decode into pixel basis, MIP, use it as Karris explains with the 7D shading model parameters. The MC baseline is, take envmap code, decode into pixel basis, MC it with the 7D shading model parameters as in any path tracer (eventually, importance-sample). Clearly MC noise hurts human eyes, but does it also hurt SGD convergence? We remember Noise2Noise and other instances were training on unbiased noisy data in the grand scheme of SGD make no difference.\n\nInstead of those ablations, focused on what is proposed, the paper compares to an Arxiv paper and a published alternative by Li et al. Comparison to Li et al is okay, we need to use something, done. Comparison to NeRD is more questionable. NeRD is very similar to this work. It more appears this paper is a fix to NeRD than a true baseline. Most agree, Arxiv papers cannot be held AGAINST a submission. Now authors might agree that not every success over an Arxiv paper indicates something. Otherwise, Arxiv turns into a \"strawman repository\" and everything is even more confusing then it already is these days.\n\nFor results, one would now expect to see something specular. As Fig. 6 shows, this is the particular strength of this approach. Alas, there was none. Results are very small, they are the same data s in NeRD. For relighting, this reviewer is very unsure what to compare this to? The Graphics approach is usually to make a photo under the novel light condition and then see how much the AI could predict it. Here we only see an image.", "questions": "", "limitations": "", "rating": 6, "confidence": 5, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "See above. This reviewer preferred to merge description of the work and critique in one field in this case.\n\nThis reviewer is unsure about what to recommend. As an *increment* over NeRD, it will not pass. It does not analyze the benefits of pre-integration nor does it show results (shiny reflections) where this really contributes. As a *replacement* for NeRD, this is a unique system that deserves publication. There is concurrent work this method appears to be promising to outperform, it is simpler to code and pre-integration not yet understood enough in Computer Vision probably. Authors would need to clarify what is the position, replacement or increment?\n\nIf suggestion is a replacement then the writing would need to change and instead of shooting down a NeRD baseline from the \"straw man repo Arxiv\", the text needs to be reworked and NeRD has to be \"dissolved\" into a \"strong ablation\".", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "No idea", "review_text": "", "extra_scores": {"ethical_concerns": "No idea", "needs_ethics_review": "No", "ethics_review_area": ["I don’t know"], "time_spent_reviewing": "3 h"}, "extra_text": {}}, {"review_id": "FYzWGEOrJ1F", "reviewer": "Reviewer_rZtL", "summary": "This paper proposes to learn a neural implicit representation from multiple images under different illumination that can separate complex SVBRDFs and lighting, which can be used for applications such as view synthesis and relighting. There are two major contributions: one is two learned manifolds to constrain BRDF and lighting parameters that can better preserve high-frequency information, the other is an MLP based network called Neural-PIL that can render intensity through a single forward pass based on the new lighting and BRDF representation. Experiments show that the learned lighting manifold can better recover details of the environment map compared to traditional spherical Gaussian representation, and the proposed method can achieve more accurate view synthesis and relighting results compared to the baseline NeRD. ", "questions": "", "limitations": "", "rating": 6, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "I will list the strengths of the paper and my questions. When listing my questions, I will separate them into major questions that may change my evaluation and minor questions. Authors can focus on major questions in the rebuttal.\n\nStrengths:\n1. The idea of using learned manifolds to constrain lighting and BRDF reconstruction is interesting. Given the ill-posed nature of inverse rendering under an unconstrained environment, such constraints are necessary to achieve satisfactory reconstruction results. Besides, the new lighting manifold is shown to better preserve high-frequency lighting compared to spherical Gaussian representation but with much fewer parameters. This new representation may inspire future research too. \n\n2. It is great to use a pre-filtered environment map to reduce the rendering cost. While this method is popular in the graphic community, it is the first time I see people use that to help to solve inverse rendering problem. \n\nMajor questions:\n\n1. The writing of the paper may need to be improved. Many important details are probably missing. \n\na. I have a hard time understanding how the Neural-IPL is trained. Line 250 and 251 do not mention explicitly that Neural-IPL is trained first and fixed through the rendering process. Only when I check the supplementary material, did I understand this part. Authors can consider moving more training details from the supplementary material to the main paper to make the main paper easier to follow. \n\nb. Line 182: It is unclear to me how the termination point is computed. I guess it is probably done by computing the weighted average based on the transmittance but more details should be included in the paper. \n\nc. Table A1 in the supplementary only summarizes the network structure of Light-SMAE. It will be good to have similar summaries for other networks. \n\nd. Line 192: It has been established before that we will only use $w_r$ as an input to render the color, which causes it to be very confusing to say Neural-IPL takes $w_i$ as input here. This is explained later in Line 197 but the whole paragraph may still be confusing to read. We can just say the input is $w_r$. In addition, the $w_i$ in Figure 3(b) can be changed to $w_r$. \n\ne. It may not be explained in the paper if the illumination of the input image changes, how we should handle lighting code $z_l$. Do we always have different lighting codes for every input image or that is only true when the illumination changes?\n\n2. The experimental results can probably be more convincing. \n\na. There is only 1 real example shown in the paper. And for this real example, it is difficult to observe specular highlights. There is no reconstructed environment map for real examples shown in the paper. \n\nb.  Table b shows that the proposed method achieves more realistic view synthesis results compared to NeRF on their synthetic dataset (Ship, Chair, Lego). Line 312 mentions that it is because the proposed method has more realistic view-dependent effects. It will be very interesting to show these examples in the paper, especially because the reconstructions of NeRF are already quite impressive. This can be very compelling results to show the advantages. \n\nOther issues:\n\n1. Line 188 to 191 says that Monte Carlo sampling or numerical accumulation is not practical. But if the environment map resolution is only 128 x 256, the computational cost of directly computing the integral is probably lower than the MLP. The main argument for Neural-IPL is probably that $z^l$ provides a good prior, not the computational cost.  \n\n2. It will be interesting to show a comparison of view synthesis and relighting results with and without the BRDF embedding. Since BRDF embedding only reduces the dimension from 7 to 4, I am curious how significantly the results can be improved. \n\n3.  The normal map in Figure A3 of the supplementary material is wrong. It probably shows the normal map in the world coordinate system instead of in the camera coordinate system. \n\nMinor questions: \n\n1. Line 290: are shows in Figure 6 -> are shown. \n\nWhile I like the idea of using learned manifold to constrain lighting and BRDF reconstruction, especially the lighting reconstruction, I think the current experiments and writing may have space for improvement. Therefore, I keep my review to be slightly negative. I would like to see  authors' rebuttal and update my review. \n\n\nAfter rebuttal: authors rebuttal solved my questions and I upgraded my review to be positive. Please refer to the rebuttal discussion for details. Thanks authors for addressing my questions.", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "Authors have listed the limitations quite completely in section 5. Maybe one more limitation to add is that the proposed method may completely rely on the geometry prediction from the coarse network and cannot optimize the geometry jointly with BRDF and lighting. The reason is that the current method only computes color at the surface points. \n\nI cannot see any potential negative societal impact. ", "review_text": "", "extra_scores": {"ethical_concerns": "I cannot see any ethical concerns. ", "needs_ethics_review": "No", "ethics_review_area": "", "time_spent_reviewing": "5"}, "extra_text": {}}]

Author rebuttal:

Meta-review: All reviewers were somewhat positive  on the paper after an extensive discussion.

They all agree on the novelty of the neural pre-integrated illumination network (Neural-PIL) as well as the advantages of using smooth manifold auto-encoders (SMAE). 

Authors are strongly encouraged to include the new results they provided during rebuttal in the final version of the paper and to highlight their  method on real images.

Given the current interest in neural rendering, even though the visual improvements over the state of the art are somewhat subtle,  the Metareviewer thinks that the novel contributions of this paper should be present at NeurIPS.
Decision: Accept (Poster)