Neural-PIL: Neural Pre-Integrated Lighting for Reflectance Decomposition

1 Introduction . Inverse rendering is the task of decomposing a scene into its underlying physical properties , such as geometry and materials . Recovering these properties is useful for several vision and graphics applications such as view synthesis [ 10 , 11 , 51 , 57 ] , relighting [ 4 , 10 , 11 , 23 , 24 , 38 , 51 , 58 ] , and object insertion [ 7 , 21 , 38 ] . In this work , we aim to recover the 3D shape and spatiallyvarying bidirectional reflectance distribution function ( SVBRDF ) of an object imaged under different illumination conditions , as shown in Fig . 1 . Estimating shape , illumination , and SVBRDF from 2D images is a highly ill-posed problem , as an observed pixel may appear dark either due to a dark surface material , or due to the incident light at that surface being reduced or absent . Our approach follows the recent success of coordinate-based scene representation networks [ 14 , 40 , 43 , 44 , 46 , 49 ] in representing 3D scenes for high-quality view-synthesis [ 44 , 49 ] . These models decompose the scene into models of shape and radiance ( emitted light ) , thereby enabling view synthesis . However , performing complete inverse rendering requires that radiance is decomposed further , into illumination and material appearances ( SVBRDF ) [ 6 , 11 , 51 , 63 ] . A key component in learning these neural SVBRDF decomposition networks is the differentiable rendering [ 10 , 11 , 63 ] that generates images and gradients for the estimated lighting and SVBRDF parameters . These methods leverage traditional rendering techniques within modern deep learning frameworks to enable ∗Work done while at Google . 35th Conference on Neural Information Processing Systems ( NeurIPS 2021 ) . backpropagation . This is often expensive , as rendering requires computing integrals over the incoming light at each 3D location in the scene . As a remedy , recent works [ 11 , 63 ] approximate the incident light by spherical Gaussians ( SG ) , thereby accelerating the illumination integration . However , these SG representations lack the capacity required to model or recover the shape and material properties of highly reflective objects or images in complex natural environments . In this work , we aim to replace the costly illumination integration step within these rendering approaches with a learned network . Inspired by the real-time graphics literature on image-based lighting [ 28 ] , we propose a novel pre-integrated lighting ( PIL ) network that converts the illumination integration process used in rendering into a simple network query . Our neural-PIL takes as input a latent vector representation for the environment map , the surface roughness , and an incident ray direction , and from them predicts an integrated illumination estimate . This query-based approach for light integration results in efficient rendering and thereby simplifies and accelerates rendering and optimization . This neural light representation is also significantly more expressive than the more commonly used SG representation , thereby enabling higher-fidelity renderings . The architecture of our neural-PIL uses conditional multi-layer perceptrons ( MLP ) with FiLM layers [ 12 ] . Fig . 2 illustrates this illumination pre-integration for different surface roughness levels . In addition , we also present a smooth manifold auto-encoder ( SMAE ) , based on interpolating auto-encoders [ 5 ] , that can learn effective low-dimensional representations of light and BRDFs . This learned low-dimensional space serves as a strong regularizer or prior for constraining the solution space of BRDFs and illumination . These constraints are critical , due to the ill-posedness of our problem setting . The smoothness of this manifold enables stable and effective gradientbased optimization of BRDF and light parameters . The neural-PIL , light-SMAE , and BRDF-SMAE networks are pre-trained on a dataset with high-quality environment maps ( illumination ) and materials ( BRDFs ) . We integrate these component networks into our decomposition framework , in which we optimize a 3D neural volume with shape and SVBRDF while also optimizing per-image illuminations . We perform an empirical analysis on synthetic datasets , along with qualitative and quantitative visual results on real-world datasets . We demonstrate that our decomposition network using our neural-PIL can estimate more accurate shape and material properties compared to prior art . The 3D assets with material properties produced by our model can be used to generate high-quality relighting and view-synthesis results with finer details compared to existing approaches . 2 Related Work . Coordinate-based MLPs allow spatial information to be stored within the weights of a neural network , thereby allowing the retrieval of information solely by querying coordinates [ 14 , 43 , 46 , 52 ] . These methods have been combined with neural volume rendering [ 40 ] to enable photorealistic results on novel view synthesis , well-exemplified by NeRF [ 44 ] . In NeRF , a coordinate-based model is used to model a field of volumetric density and color , and renderings are produced by ray-marching through that neural volume . Though NeRF is capable of photorealistic renderings , it has many limitations that have been explored by recent work , such as : no relighting capabilities [ 6 , 11 , 42 , 51 , 63 ] , long training times [ 39 , 54 ] , long inference times [ 11 , 25 , 31 , 39 , 60 ] , extraction of 3D geometry and materials [ 11 ] , and generalization [ 12 , 54 , 64 ] . This work addresses some of these challenges that enables relighting , extracts a conventional 3D geometry and material estimate , and enables real-time rendering ( as our 3D assets are compatible with existing real-time rendering engines ) . The concurrent works of NeRD [ 11 ] , NeRV [ 51 ] and PhySG [ 63 ] are most clostly related to ours . These methods decomposes the scene into shape and analytical SVBRDF parameters . However , NeRV requires known illumination , and NeRD and PhySG employ a spherical Gaussian ( SG ) model , which is not capable of modeling detailed illumination patterns . BRDF estimation is a challenging research problem that aims to estimate the appearance of a physical material . For the highest accuracy results , measurements are performed under controlled laboratory conditions with known view and light positions [ 3 , 9 , 32 , 33 , 34 ] , but this does not allow for the on-site capture of materials . Casual capture methods aim to solve this constraint by only requiring a camera , and sometimes a known light source . Often , machine learning techniques are leveraged to reduce the ambiguity through the use of data-drive priors and large datasets of BRDFs . Additional constraints from planar surfaces viewed under camera flash illumination are considered for single-shot [ 1 , 16 , 26 , 36 , 47 ] , few-shot [ 1 ] or multi-shot [ 2 , 9 , 17 , 18 , 20 ] estimation . This casual setup can be extended to estimating the BRDF and shape of objects [ 6 , 7 , 8 , 10 , 30 , 45 , 47 , 62 ] or scenes [ 38 , 48 ] . Most of these methods are based on known active illumination . A limited number of light sources — most often a single one — are assumed to be responsible for the majority of illumination in a scene . Relying on only natural , uncontrolled illumination adds several additional challenges due to the drastically increased ambiguity across shape , illumination and BRDFs . Often these challenges are reduced by keeping the specular albedo non-spatially-varying , or by removing it entirely [ 35 , 59 , 63 ] . Other approaches require temporal traces and limit the casual capture setup [ 19 , 56 ] . Illumination estimation from a single image is an inherently challenging problem . The task is inherently linked to BRDF estimation , as illumination affects appearance and is only indirectly observable from its interactions with surface materials . These two tasks are often solved in conjunction , sometimes by decomposing a single object into shape , reflectance , and a global set of spherical Gaussians ( SGs ) [ 11 , 63 ] . Chen et al . [ 13 ] leverage a deep prior of environment maps with homogeneous materials , using an invertible neural BRDF model . Li et al . [ 38 ] decompose an entire scene into a simplified BRDF model with hemispherical SGs per point in the scene . The image of the environment in the background may be incorporated into prediction , shifting the problem to completion of the HDR environment map from sparse observations [ 21 , 50 , 55 ] . We not only learn a deep prior but a rendering aware network which is capable of integrating the environment illumination for a specific surface roughness enabling rendering the entire hemisphere of incoming light with a single evaluation . 3 Method . Given an image collection of an object captured under varying illumination conditions and camera viewpoints , we aim to jointly estimate the object ’ s 3D shape and spatially-varying BRDF , as well as the illumination conditions of each image . Our input consists of a set of q images with s pixels each : Cj ∈ Rs×3 ; j ∈ { 1 , . . . , q } along with per-pixel masks Mj ∈ { 0 , 1 } s×1 indicating which pixels belong to the object . Our goal is to learn a neural 3D volume V where , at each point x ∈ R3 , we estimate the BRDF parameters for the Cook-Torrance model [ 15 ] b ∈ R7 ( diffuse bd ∈ R3 , specular bs ∈ R3 , roughness br ∈ R ) , unit-length surface normal n ∈ R3 and optical density σ ∈ R. In addition , we also estimate latent vectors representing per-image illumination zl ∈ R128 . This problem statement corresponds to the practical application of recovering a 3D model of some real-world object ( for e.g. , a statue or landmark ) that has been photographed by different people at different times . 3.1 Preliminaries . Brief overview of neural radiance fields ( NeRF ) . Our method is based on NeRF [ 44 ] which creates a neural volume for novel view synthesis using two Multi-Layer-Perceptrons ( MLPs ) . The first MLP learns a coarse representation , which samples the 3D volume in a fixed sampling pattern , and the second MLP uses this knowledge to sample the volume in high-density areas at a finer resolution . The output of the MLP is a view-dependent color c ∈ R3 and optical density σ ∈ R for each given 3D location x ∈ R3 and view direction d ∈ R3 . In order to render the output color ĉ ∈ R3 for a camera ray r ( t ) = o + td , with ray origin o ∈ R3 and view direction d , we approximate ( via numerical quadrature ) the integral ĉ ( r ) = ∫ tf tn T ( t ) σ ( t ) c ( t ) dt with T ( t ) = exp ( − ∫ t tn σ ( t ) dt ) , using the near and far bounds of the ray tn and tf respectively [ 44 ] . Image formation and image-based lighting . NeRF directly models view-dependent color ĉ at each 3D location . Thus , a simple image formation process that integrates the color information along camera rays is sufficient to render images . In contrast , we want to explicitly estimate an object material decomposition at each 3D location . We therefore must use a more explicit rendering formulation that relates image formation to BRDFs and illumination . The rendering equation [ 27 ] estimates the radiance Lo ∈ R3 at x along the outgoing view direction ωo ∈ R3 ( ωo = −d ) : Lo ( x , ωo ) = ∫ Ω fr ( x , ωi , ωo ; b ) Li ( x , ωi ) ( ωi · n ) dωi , where fr is the BRDF evaluation , Li ∈ R3 is incoming light , and ωi ∈ R3 is the incoming light direction . Using this single bounce rendering formulation , and ignoring exposure variation and tone-mapping , Lo is equivalent to NeRF ’ s color ĉ . The formulation can be split into diffuse and specular components : Lo ( x , ωo ) = bd π ∫ Ω Li ( x , ωi ) ( ωi · n ) dωi︸ ︷︷ ︸ diffuse + ∫ Ω fs ( x , ωi , ωo ; bs , br ) Li ( x , ωi ) ( ωi · n ) dωi︸ ︷︷ ︸ specular ( 1 ) where fs now only represents the specular portion of the BRDF evaluation . Following Karis et al . [ 28 ] , several parts of this integration can be pre-computed [ 29 ] : Lo ( x , ωo ) ≈ ( bd/π ) L̃i ( n , 1 ) ︸ ︷︷ ︸ diffuse + bs ( F0 ( ωo , n ) B0 ( ωo · n , br ) +B1 ( ωo · n , br ) ) L̃i ( ωr , br ) ︸ ︷︷ ︸ specular ( 2 ) The illumination is now pre-integrated as : L̃i ( ωr , br ) = ∫ Ω D ( br , ωi , ωr ) Li ( x , ωi ) dωi which only depends on the mirrored view direction ωr ( which subsumes the surface normal n ) and the roughness br , where D describes the microfacet distribution [ 53 ] . This light pre-integration is illustrated in Fig . 2 . Note that the same pre-integrated L̃i ( ωr , br ) is queried twice : the diffuse part captures the entire hemisphere and therefore is parameterized by the surface normal n ∈ R3 and a diffuse roughness of 1 . The specular part looks up L̃i ( ωr , br ) for the reflected view direction ωr ∈ R3 and the specular roughness br . While the pre-integration already considers the microfacet distribution D , one must also account for shadowing , masking , and the Fresnel term . As shown in Karis et al . [ 28 ] , these remaining parts can be pre-computed into two 2D lookup textures ( LUT ) B0 and B1 indexed by ( ωo · n ) and the roughness br . These are combined with the Fresnel term at normal incidence F0 ( ωo , n ) = ( 1− ωo · h ) 5 with h = ‖ωi + ωo‖ . This pre-integration approach replaces the complex integration during shading with a set of simple additions and multiplications . We have integrated the core idea of this approach into an efficient differentiable neural rendering framework , which allows for the optimization of geometry , BRDF , and illumination simultaneously via standard backpropagation . We further reduce the computational complexity by mimicking the pre-integration of L̃i ( ωr , br ) with a simple query through our NeuralPIL that operates directly on a neural representation of the illumination , as we will now explain .

Summary: The paper presents a method to reconstruct geometry and BRDF from a set  of multi-view images take under various illuminations. It learns a low  dimensional manifold for illuminations and BRDFs and use these data priors  to regularize the estimation. It also applies a network to predict the  integral of light for efficient rendering under environment maps. It  shows results on both synthetic and real scenes.