Neural-PIL: Neural Pre-Integrated Lighting for Reflectance Decomposition

1 Introduction . Inverse rendering is the task of decomposing a scene into its underlying physical properties , such as geometry and materials . Recovering these properties is useful for several vision and graphics applications such as view synthesis [ 10 , 11 , 51 , 57 ] , relighting [ 4 , 10 , 11 , 23 , 24 , 38 , 51 , 58 ] , and object insertion [ 7 , 21 , 38 ] . In this work , we aim to recover the 3D shape and spatiallyvarying bidirectional reflectance distribution function ( SVBRDF ) of an object imaged under different illumination conditions , as shown in Fig . 1 . Estimating shape , illumination , and SVBRDF from 2D images is a highly ill-posed problem , as an observed pixel may appear dark either due to a dark surface material , or due to the incident light at that surface being reduced or absent . Our approach follows the recent success of coordinate-based scene representation networks [ 14 , 40 , 43 , 44 , 46 , 49 ] in representing 3D scenes for high-quality view-synthesis [ 44 , 49 ] . These models decompose the scene into models of shape and radiance ( emitted light ) , thereby enabling view synthesis . However , performing complete inverse rendering requires that radiance is decomposed further , into illumination and material appearances ( SVBRDF ) [ 6 , 11 , 51 , 63 ] . A key component in learning these neural SVBRDF decomposition networks is the differentiable rendering [ 10 , 11 , 63 ] that generates images and gradients for the estimated lighting and SVBRDF parameters . These methods leverage traditional rendering techniques within modern deep learning frameworks to enable ∗Work done while at Google . 35th Conference on Neural Information Processing Systems ( NeurIPS 2021 ) . backpropagation . This is often expensive , as rendering requires computing integrals over the incoming light at each 3D location in the scene . As a remedy , recent works [ 11 , 63 ] approximate the incident light by spherical Gaussians ( SG ) , thereby accelerating the illumination integration . However , these SG representations lack the capacity required to model or recover the shape and material properties of highly reflective objects or images in complex natural environments . In this work , we aim to replace the costly illumination integration step within these rendering approaches with a learned network . Inspired by the real-time graphics literature on image-based lighting [ 28 ] , we propose a novel pre-integrated lighting ( PIL ) network that converts the illumination integration process used in rendering into a simple network query . Our neural-PIL takes as input a latent vector representation for the environment map , the surface roughness , and an incident ray direction , and from them predicts an integrated illumination estimate . This query-based approach for light integration results in efficient rendering and thereby simplifies and accelerates rendering and optimization . This neural light representation is also significantly more expressive than the more commonly used SG representation , thereby enabling higher-fidelity renderings . The architecture of our neural-PIL uses conditional multi-layer perceptrons ( MLP ) with FiLM layers [ 12 ] . Fig . 2 illustrates this illumination pre-integration for different surface roughness levels . In addition , we also present a smooth manifold auto-encoder ( SMAE ) , based on interpolating auto-encoders [ 5 ] , that can learn effective low-dimensional representations of light and BRDFs . This learned low-dimensional space serves as a strong regularizer or prior for constraining the solution space of BRDFs and illumination . These constraints are critical , due to the ill-posedness of our problem setting . The smoothness of this manifold enables stable and effective gradientbased optimization of BRDF and light parameters . The neural-PIL , light-SMAE , and BRDF-SMAE networks are pre-trained on a dataset with high-quality environment maps ( illumination ) and materials ( BRDFs ) . We integrate these component networks into our decomposition framework , in which we optimize a 3D neural volume with shape and SVBRDF while also optimizing per-image illuminations . We perform an empirical analysis on synthetic datasets , along with qualitative and quantitative visual results on real-world datasets . We demonstrate that our decomposition network using our neural-PIL can estimate more accurate shape and material properties compared to prior art . The 3D assets with material properties produced by our model can be used to generate high-quality relighting and view-synthesis results with finer details compared to existing approaches . 2 Related Work . Coordinate-based MLPs allow spatial information to be stored within the weights of a neural network , thereby allowing the retrieval of information solely by querying coordinates [ 14 , 43 , 46 , 52 ] . These methods have been combined with neural volume rendering [ 40 ] to enable photorealistic results on novel view synthesis , well-exemplified by NeRF [ 44 ] . In NeRF , a coordinate-based model is used to model a field of volumetric density and color , and renderings are produced by ray-marching through that neural volume . Though NeRF is capable of photorealistic renderings , it has many limitations that have been explored by recent work , such as : no relighting capabilities [ 6 , 11 , 42 , 51 , 63 ] , long training times [ 39 , 54 ] , long inference times [ 11 , 25 , 31 , 39 , 60 ] , extraction of 3D geometry and materials [ 11 ] , and generalization [ 12 , 54 , 64 ] . This work addresses some of these challenges that enables relighting , extracts a conventional 3D geometry and material estimate , and enables real-time rendering ( as our 3D assets are compatible with existing real-time rendering engines ) . The concurrent works of NeRD [ 11 ] , NeRV [ 51 ] and PhySG [ 63 ] are most clostly related to ours . These methods decomposes the scene into shape and analytical SVBRDF parameters . However , NeRV requires known illumination , and NeRD and PhySG employ a spherical Gaussian ( SG ) model , which is not capable of modeling detailed illumination patterns . BRDF estimation is a challenging research problem that aims to estimate the appearance of a physical material . For the highest accuracy results , measurements are performed under controlled laboratory conditions with known view and light positions [ 3 , 9 , 32 , 33 , 34 ] , but this does not allow for the on-site capture of materials . Casual capture methods aim to solve this constraint by only requiring a camera , and sometimes a known light source . Often , machine learning techniques are leveraged to reduce the ambiguity through the use of data-drive priors and large datasets of BRDFs . Additional constraints from planar surfaces viewed under camera flash illumination are considered for single-shot [ 1 , 16 , 26 , 36 , 47 ] , few-shot [ 1 ] or multi-shot [ 2 , 9 , 17 , 18 , 20 ] estimation . This casual setup can be extended to estimating the BRDF and shape of objects [ 6 , 7 , 8 , 10 , 30 , 45 , 47 , 62 ] or scenes [ 38 , 48 ] . Most of these methods are based on known active illumination . A limited number of light sources — most often a single one — are assumed to be responsible for the majority of illumination in a scene . Relying on only natural , uncontrolled illumination adds several additional challenges due to the drastically increased ambiguity across shape , illumination and BRDFs . Often these challenges are reduced by keeping the specular albedo non-spatially-varying , or by removing it entirely [ 35 , 59 , 63 ] . Other approaches require temporal traces and limit the casual capture setup [ 19 , 56 ] . Illumination estimation from a single image is an inherently challenging problem . The task is inherently linked to BRDF estimation , as illumination affects appearance and is only indirectly observable from its interactions with surface materials . These two tasks are often solved in conjunction , sometimes by decomposing a single object into shape , reflectance , and a global set of spherical Gaussians ( SGs ) [ 11 , 63 ] . Chen et al . [ 13 ] leverage a deep prior of environment maps with homogeneous materials , using an invertible neural BRDF model . Li et al . [ 38 ] decompose an entire scene into a simplified BRDF model with hemispherical SGs per point in the scene . The image of the environment in the background may be incorporated into prediction , shifting the problem to completion of the HDR environment map from sparse observations [ 21 , 50 , 55 ] . We not only learn a deep prior but a rendering aware network which is capable of integrating the environment illumination for a specific surface roughness enabling rendering the entire hemisphere of incoming light with a single evaluation . 3 Method . Given an image collection of an object captured under varying illumination conditions and camera viewpoints , we aim to jointly estimate the object ’ s 3D shape and spatially-varying BRDF , as well as the illumination conditions of each image . Our input consists of a set of q images with s pixels each : Cj ∈ Rs×3 ; j ∈ { 1 , . . . , q } along with per-pixel masks Mj ∈ { 0 , 1 } s×1 indicating which pixels belong to the object . Our goal is to learn a neural 3D volume V where , at each point x ∈ R3 , we estimate the BRDF parameters for the Cook-Torrance model [ 15 ] b ∈ R7 ( diffuse bd ∈ R3 , specular bs ∈ R3 , roughness br ∈ R ) , unit-length surface normal n ∈ R3 and optical density σ ∈ R. In addition , we also estimate latent vectors representing per-image illumination zl ∈ R128 . This problem statement corresponds to the practical application of recovering a 3D model of some real-world object ( for e.g. , a statue or landmark ) that has been photographed by different people at different times . 3.1 Preliminaries . Brief overview of neural radiance fields ( NeRF ) . Our method is based on NeRF [ 44 ] which creates a neural volume for novel view synthesis using two Multi-Layer-Perceptrons ( MLPs ) . The first MLP learns a coarse representation , which samples the 3D volume in a fixed sampling pattern , and the second MLP uses this knowledge to sample the volume in high-density areas at a finer resolution . The output of the MLP is a view-dependent color c ∈ R3 and optical density σ ∈ R for each given 3D location x ∈ R3 and view direction d ∈ R3 . In order to render the output color ĉ ∈ R3 for a camera ray r ( t ) = o + td , with ray origin o ∈ R3 and view direction d , we approximate ( via numerical quadrature ) the integral ĉ ( r ) = ∫ tf tn T ( t ) σ ( t ) c ( t ) dt with T ( t ) = exp ( − ∫ t tn σ ( t ) dt ) , using the near and far bounds of the ray tn and tf respectively [ 44 ] . Image formation and image-based lighting . NeRF directly models view-dependent color ĉ at each 3D location . Thus , a simple image formation process that integrates the color information along camera rays is sufficient to render images . In contrast , we want to explicitly estimate an object material decomposition at each 3D location . We therefore must use a more explicit rendering formulation that relates image formation to BRDFs and illumination . The rendering equation [ 27 ] estimates the radiance Lo ∈ R3 at x along the outgoing view direction ωo ∈ R3 ( ωo = −d ) : Lo ( x , ωo ) = ∫ Ω fr ( x , ωi , ωo ; b ) Li ( x , ωi ) ( ωi · n ) dωi , where fr is the BRDF evaluation , Li ∈ R3 is incoming light , and ωi ∈ R3 is the incoming light direction . Using this single bounce rendering formulation , and ignoring exposure variation and tone-mapping , Lo is equivalent to NeRF ’ s color ĉ . The formulation can be split into diffuse and specular components : Lo ( x , ωo ) = bd π ∫ Ω Li ( x , ωi ) ( ωi · n ) dωi︸ ︷︷ ︸ diffuse + ∫ Ω fs ( x , ωi , ωo ; bs , br ) Li ( x , ωi ) ( ωi · n ) dωi︸ ︷︷ ︸ specular ( 1 ) where fs now only represents the specular portion of the BRDF evaluation . Following Karis et al . [ 28 ] , several parts of this integration can be pre-computed [ 29 ] : Lo ( x , ωo ) ≈ ( bd/π ) L̃i ( n , 1 ) ︸ ︷︷ ︸ diffuse + bs ( F0 ( ωo , n ) B0 ( ωo · n , br ) +B1 ( ωo · n , br ) ) L̃i ( ωr , br ) ︸ ︷︷ ︸ specular ( 2 ) The illumination is now pre-integrated as : L̃i ( ωr , br ) = ∫ Ω D ( br , ωi , ωr ) Li ( x , ωi ) dωi which only depends on the mirrored view direction ωr ( which subsumes the surface normal n ) and the roughness br , where D describes the microfacet distribution [ 53 ] . This light pre-integration is illustrated in Fig . 2 . Note that the same pre-integrated L̃i ( ωr , br ) is queried twice : the diffuse part captures the entire hemisphere and therefore is parameterized by the surface normal n ∈ R3 and a diffuse roughness of 1 . The specular part looks up L̃i ( ωr , br ) for the reflected view direction ωr ∈ R3 and the specular roughness br . While the pre-integration already considers the microfacet distribution D , one must also account for shadowing , masking , and the Fresnel term . As shown in Karis et al . [ 28 ] , these remaining parts can be pre-computed into two 2D lookup textures ( LUT ) B0 and B1 indexed by ( ωo · n ) and the roughness br . These are combined with the Fresnel term at normal incidence F0 ( ωo , n ) = ( 1− ωo · h ) 5 with h = ‖ωi + ωo‖ . This pre-integration approach replaces the complex integration during shading with a set of simple additions and multiplications . We have integrated the core idea of this approach into an efficient differentiable neural rendering framework , which allows for the optimization of geometry , BRDF , and illumination simultaneously via standard backpropagation . We further reduce the computational complexity by mimicking the pre-integration of L̃i ( ωr , br ) with a simple query through our NeuralPIL that operates directly on a neural representation of the illumination , as we will now explain .

Summary: This paper proposes to learn a neural implicit representation from multiple images under different illumination that can separate complex SVBRDFs and lighting, which can be used for applications such as view synthesis and relighting. There are two major contributions: one is two learned manifolds to constrain BRDF and lighting parameters that can better preserve high-frequency information, the other is an MLP based network called Neural-PIL that can render intensity through a single forward pass based on the new lighting and BRDF representation. Experiments show that the learned lighting manifold can better recover details of the environment map compared to traditional spherical Gaussian representation, and the proposed method can achieve more accurate view synthesis and relighting results compared to the baseline NeRD.