Neural-PIL: Neural Pre-Integrated Lighting for Reflectance Decomposition

1 Introduction . Inverse rendering is the task of decomposing a scene into its underlying physical properties , such as geometry and materials . Recovering these properties is useful for several vision and graphics applications such as view synthesis [ 10 , 11 , 51 , 57 ] , relighting [ 4 , 10 , 11 , 23 , 24 , 38 , 51 , 58 ] , and object insertion [ 7 , 21 , 38 ] . In this work , we aim to recover the 3D shape and spatiallyvarying bidirectional reflectance distribution function ( SVBRDF ) of an object imaged under different illumination conditions , as shown in Fig . 1 . Estimating shape , illumination , and SVBRDF from 2D images is a highly ill-posed problem , as an observed pixel may appear dark either due to a dark surface material , or due to the incident light at that surface being reduced or absent . Our approach follows the recent success of coordinate-based scene representation networks [ 14 , 40 , 43 , 44 , 46 , 49 ] in representing 3D scenes for high-quality view-synthesis [ 44 , 49 ] . These models decompose the scene into models of shape and radiance ( emitted light ) , thereby enabling view synthesis . However , performing complete inverse rendering requires that radiance is decomposed further , into illumination and material appearances ( SVBRDF ) [ 6 , 11 , 51 , 63 ] . A key component in learning these neural SVBRDF decomposition networks is the differentiable rendering [ 10 , 11 , 63 ] that generates images and gradients for the estimated lighting and SVBRDF parameters . These methods leverage traditional rendering techniques within modern deep learning frameworks to enable ∗Work done while at Google . 35th Conference on Neural Information Processing Systems ( NeurIPS 2021 ) . backpropagation . This is often expensive , as rendering requires computing integrals over the incoming light at each 3D location in the scene . As a remedy , recent works [ 11 , 63 ] approximate the incident light by spherical Gaussians ( SG ) , thereby accelerating the illumination integration . However , these SG representations lack the capacity required to model or recover the shape and material properties of highly reflective objects or images in complex natural environments . In this work , we aim to replace the costly illumination integration step within these rendering approaches with a learned network . Inspired by the real-time graphics literature on image-based lighting [ 28 ] , we propose a novel pre-integrated lighting ( PIL ) network that converts the illumination integration process used in rendering into a simple network query . Our neural-PIL takes as input a latent vector representation for the environment map , the surface roughness , and an incident ray direction , and from them predicts an integrated illumination estimate . This query-based approach for light integration results in efficient rendering and thereby simplifies and accelerates rendering and optimization . This neural light representation is also significantly more expressive than the more commonly used SG representation , thereby enabling higher-fidelity renderings . The architecture of our neural-PIL uses conditional multi-layer perceptrons ( MLP ) with FiLM layers [ 12 ] . Fig . 2 illustrates this illumination pre-integration for different surface roughness levels . In addition , we also present a smooth manifold auto-encoder ( SMAE ) , based on interpolating auto-encoders [ 5 ] , that can learn effective low-dimensional representations of light and BRDFs . This learned low-dimensional space serves as a strong regularizer or prior for constraining the solution space of BRDFs and illumination . These constraints are critical , due to the ill-posedness of our problem setting . The smoothness of this manifold enables stable and effective gradientbased optimization of BRDF and light parameters . The neural-PIL , light-SMAE , and BRDF-SMAE networks are pre-trained on a dataset with high-quality environment maps ( illumination ) and materials ( BRDFs ) . We integrate these component networks into our decomposition framework , in which we optimize a 3D neural volume with shape and SVBRDF while also optimizing per-image illuminations . We perform an empirical analysis on synthetic datasets , along with qualitative and quantitative visual results on real-world datasets . We demonstrate that our decomposition network using our neural-PIL can estimate more accurate shape and material properties compared to prior art . The 3D assets with material properties produced by our model can be used to generate high-quality relighting and view-synthesis results with finer details compared to existing approaches . 2 Related Work . Coordinate-based MLPs allow spatial information to be stored within the weights of a neural network , thereby allowing the retrieval of information solely by querying coordinates [ 14 , 43 , 46 , 52 ] . These methods have been combined with neural volume rendering [ 40 ] to enable photorealistic results on novel view synthesis , well-exemplified by NeRF [ 44 ] . In NeRF , a coordinate-based model is used to model a field of volumetric density and color , and renderings are produced by ray-marching through that neural volume . Though NeRF is capable of photorealistic renderings , it has many limitations that have been explored by recent work , such as : no relighting capabilities [ 6 , 11 , 42 , 51 , 63 ] , long training times [ 39 , 54 ] , long inference times [ 11 , 25 , 31 , 39 , 60 ] , extraction of 3D geometry and materials [ 11 ] , and generalization [ 12 , 54 , 64 ] . This work addresses some of these challenges that enables relighting , extracts a conventional 3D geometry and material estimate , and enables real-time rendering ( as our 3D assets are compatible with existing real-time rendering engines ) . The concurrent works of NeRD [ 11 ] , NeRV [ 51 ] and PhySG [ 63 ] are most clostly related to ours . These methods decomposes the scene into shape and analytical SVBRDF parameters . However , NeRV requires known illumination , and NeRD and PhySG employ a spherical Gaussian ( SG ) model , which is not capable of modeling detailed illumination patterns . BRDF estimation is a challenging research problem that aims to estimate the appearance of a physical material . For the highest accuracy results , measurements are performed under controlled laboratory conditions with known view and light positions [ 3 , 9 , 32 , 33 , 34 ] , but this does not allow for the on-site capture of materials . Casual capture methods aim to solve this constraint by only requiring a camera , and sometimes a known light source . Often , machine learning techniques are leveraged to reduce the ambiguity through the use of data-drive priors and large datasets of BRDFs . Additional constraints from planar surfaces viewed under camera flash illumination are considered for single-shot [ 1 , 16 , 26 , 36 , 47 ] , few-shot [ 1 ] or multi-shot [ 2 , 9 , 17 , 18 , 20 ] estimation . This casual setup can be extended to estimating the BRDF and shape of objects [ 6 , 7 , 8 , 10 , 30 , 45 , 47 , 62 ] or scenes [ 38 , 48 ] . Most of these methods are based on known active illumination . A limited number of light sources — most often a single one — are assumed to be responsible for the majority of illumination in a scene . Relying on only natural , uncontrolled illumination adds several additional challenges due to the drastically increased ambiguity across shape , illumination and BRDFs . Often these challenges are reduced by keeping the specular albedo non-spatially-varying , or by removing it entirely [ 35 , 59 , 63 ] . Other approaches require temporal traces and limit the casual capture setup [ 19 , 56 ] . Illumination estimation from a single image is an inherently challenging problem . The task is inherently linked to BRDF estimation , as illumination affects appearance and is only indirectly observable from its interactions with surface materials . These two tasks are often solved in conjunction , sometimes by decomposing a single object into shape , reflectance , and a global set of spherical Gaussians ( SGs ) [ 11 , 63 ] . Chen et al . [ 13 ] leverage a deep prior of environment maps with homogeneous materials , using an invertible neural BRDF model . Li et al . [ 38 ] decompose an entire scene into a simplified BRDF model with hemispherical SGs per point in the scene . The image of the environment in the background may be incorporated into prediction , shifting the problem to completion of the HDR environment map from sparse observations [ 21 , 50 , 55 ] . We not only learn a deep prior but a rendering aware network which is capable of integrating the environment illumination for a specific surface roughness enabling rendering the entire hemisphere of incoming light with a single evaluation . 3 Method . Given an image collection of an object captured under varying illumination conditions and camera viewpoints , we aim to jointly estimate the object ’ s 3D shape and spatially-varying BRDF , as well as the illumination conditions of each image . Our input consists of a set of q images with s pixels each : Cj ∈ Rs×3 ; j ∈ { 1 , . . . , q } along with per-pixel masks Mj ∈ { 0 , 1 } s×1 indicating which pixels belong to the object . Our goal is to learn a neural 3D volume V where , at each point x ∈ R3 , we estimate the BRDF parameters for the Cook-Torrance model [ 15 ] b ∈ R7 ( diffuse bd ∈ R3 , specular bs ∈ R3 , roughness br ∈ R ) , unit-length surface normal n ∈ R3 and optical density σ ∈ R. In addition , we also estimate latent vectors representing per-image illumination zl ∈ R128 . This problem statement corresponds to the practical application of recovering a 3D model of some real-world object ( for e.g. , a statue or landmark ) that has been photographed by different people at different times . 3.1 Preliminaries . Brief overview of neural radiance fields ( NeRF ) . Our method is based on NeRF [ 44 ] which creates a neural volume for novel view synthesis using two Multi-Layer-Perceptrons ( MLPs ) . The first MLP learns a coarse representation , which samples the 3D volume in a fixed sampling pattern , and the second MLP uses this knowledge to sample the volume in high-density areas at a finer resolution . The output of the MLP is a view-dependent color c ∈ R3 and optical density σ ∈ R for each given 3D location x ∈ R3 and view direction d ∈ R3 . In order to render the output color ĉ ∈ R3 for a camera ray r ( t ) = o + td , with ray origin o ∈ R3 and view direction d , we approximate ( via numerical quadrature ) the integral ĉ ( r ) = ∫ tf tn T ( t ) σ ( t ) c ( t ) dt with T ( t ) = exp ( − ∫ t tn σ ( t ) dt ) , using the near and far bounds of the ray tn and tf respectively [ 44 ] . Image formation and image-based lighting . NeRF directly models view-dependent color ĉ at each 3D location . Thus , a simple image formation process that integrates the color information along camera rays is sufficient to render images . In contrast , we want to explicitly estimate an object material decomposition at each 3D location . We therefore must use a more explicit rendering formulation that relates image formation to BRDFs and illumination . The rendering equation [ 27 ] estimates the radiance Lo ∈ R3 at x along the outgoing view direction ωo ∈ R3 ( ωo = −d ) : Lo ( x , ωo ) = ∫ Ω fr ( x , ωi , ωo ; b ) Li ( x , ωi ) ( ωi · n ) dωi , where fr is the BRDF evaluation , Li ∈ R3 is incoming light , and ωi ∈ R3 is the incoming light direction . Using this single bounce rendering formulation , and ignoring exposure variation and tone-mapping , Lo is equivalent to NeRF ’ s color ĉ . The formulation can be split into diffuse and specular components : Lo ( x , ωo ) = bd π ∫ Ω Li ( x , ωi ) ( ωi · n ) dωi︸ ︷︷ ︸ diffuse + ∫ Ω fs ( x , ωi , ωo ; bs , br ) Li ( x , ωi ) ( ωi · n ) dωi︸ ︷︷ ︸ specular ( 1 ) where fs now only represents the specular portion of the BRDF evaluation . Following Karis et al . [ 28 ] , several parts of this integration can be pre-computed [ 29 ] : Lo ( x , ωo ) ≈ ( bd/π ) L̃i ( n , 1 ) ︸ ︷︷ ︸ diffuse + bs ( F0 ( ωo , n ) B0 ( ωo · n , br ) +B1 ( ωo · n , br ) ) L̃i ( ωr , br ) ︸ ︷︷ ︸ specular ( 2 ) The illumination is now pre-integrated as : L̃i ( ωr , br ) = ∫ Ω D ( br , ωi , ωr ) Li ( x , ωi ) dωi which only depends on the mirrored view direction ωr ( which subsumes the surface normal n ) and the roughness br , where D describes the microfacet distribution [ 53 ] . This light pre-integration is illustrated in Fig . 2 . Note that the same pre-integrated L̃i ( ωr , br ) is queried twice : the diffuse part captures the entire hemisphere and therefore is parameterized by the surface normal n ∈ R3 and a diffuse roughness of 1 . The specular part looks up L̃i ( ωr , br ) for the reflected view direction ωr ∈ R3 and the specular roughness br . While the pre-integration already considers the microfacet distribution D , one must also account for shadowing , masking , and the Fresnel term . As shown in Karis et al . [ 28 ] , these remaining parts can be pre-computed into two 2D lookup textures ( LUT ) B0 and B1 indexed by ( ωo · n ) and the roughness br . These are combined with the Fresnel term at normal incidence F0 ( ωo , n ) = ( 1− ωo · h ) 5 with h = ‖ωi + ωo‖ . This pre-integration approach replaces the complex integration during shading with a set of simple additions and multiplications . We have integrated the core idea of this approach into an efficient differentiable neural rendering framework , which allows for the optimization of geometry , BRDF , and illumination simultaneously via standard backpropagation . We further reduce the computational complexity by mimicking the pre-integration of L̃i ( ωr , br ) with a simple query through our NeuralPIL that operates directly on a neural representation of the illumination , as we will now explain .

Summary: The method suggests to learn a 3D model of shape, reflectance and light from a set of 2D images. They use a forward model that combines the three factors and invert it by optimizing its parameters to explain the 2D images. The key is in the specific parametrization of the three factors.  The first factor is a classic implicit model of shape. Rays are cast from the views of the 2D input images to find 3D depth. They define it as expected opacity-weighted depth, which might have its problem on (soft) silhouette edges. Mode finding, via Iterated Re-weighted least squares would be still fast and maybe more correct, not resulting in depth values floating in the air.  This said, this part is common.  The reflectance is parametrized as a spatial field of 4D latent codes. Each latent code itself is an embedding of the 7D Cook-Torrence parameters over the MERL database. The paper says the 7D codes are "not constrained enough", but no evidence for this is given. The embedding is pre-trained using a modern smooth auto-encoder.  The illumination is parametrized in a similar way: a smooth auto-encoding of a collection of environment maps into a low-dimensional code.  The work dwells a bit too much on how this is a graat AE'ing, but it is just shopped previous work, so it should maybe be phrased more neutral.   Now the key innovation is how the two latter factors combine. The direct way is to solve an integral where the BRDF slice is convolved with the illumination. in SH this is a dot product, but at the expense of angular details. Instead, the suggestion is to pick up the realtime-rendering idea of pre-filtered illumination. This allows to get the integral estimate in constant time, after pre-processing the envmap. Now instead of preprocessing the envmap (which would be possible, there is only one illumination per scene and pre-process = pyramidal blur = constant time), authors suggest to use a network that directly returns pre-filtered results. For this idea to be fully appreciated, one would need to look at the two ablations chosen by those familiar with game graphics: pixel basis (MIP maps) and MC directly. This would need to be compared against an NN doing this pre-filtering job. It is a cool idea, but does it do anything in the end? Comparison is only made against SH, which is for the birds to those learned in graphics. The pixel basis ablation is: take the envmap code, decode into pixel basis, MIP, use it as Karris explains with the 7D shading model parameters. The MC baseline is, take envmap code, decode into pixel basis, MC it with the 7D shading model parameters as in any path tracer (eventually, importance-sample). Clearly MC noise hurts human eyes, but does it also hurt SGD convergence? We remember Noise2Noise and other instances were training on unbiased noisy data in the grand scheme of SGD make no difference.  Instead of those ablations, focused on what is proposed, the paper compares to an Arxiv paper and a published alternative by Li et al. Comparison to Li et al is okay, we need to use something, done. Comparison to NeRD is more questionable. NeRD is very similar to this work. It more appears this paper is a fix to NeRD than a true baseline. Most agree, Arxiv papers cannot be held AGAINST a submission. Now authors might agree that not every success over an Arxiv paper indicates something. Otherwise, Arxiv turns into a "strawman repository" and everything is even more confusing then it already is these days.  For results, one would now expect to see something specular. As Fig. 6 shows, this is the particular strength of this approach. Alas, there was none. Results are very small, they are the same data s in NeRD. For relighting, this reviewer is very unsure what to compare this to? The Graphics approach is usually to make a photo under the novel light condition and then see how much the AI could predict it. Here we only see an image.