Light Sampling Field and BRDF Representation for Physically-based Neural Rendering
Authors: Jing Yang
Hanyuan Xiao
Wenbin Teng
Yunxuan Cai
Yajie Zhao
Venue: ICLR 2023 poster (2023)
Keywords: Neural Rendering

Abstract: Physically-based rendering (PBR) is key for immersive rendering effects used widely in the industry to showcase detailed realistic scenes from computer graphics assets. A well-known caveat is that producing the same is computationally heavy and relies on complex capture devices. Inspired by the success in quality and efficiency of recent volumetric neural rendering, we want to develop a physically-based neural shader to eliminate device dependency and significantly boost performance. However, no existing lighting and material models in the current neural rendering approaches can accurately represent the comprehensive lighting models and BRDFs properties required by the PBR process. Thus, this paper proposes a novel lighting representation that models direct and indirect light locally through a light sampling strategy in a learned light sampling field. We also propose BRDF models to separately represent surface/subsurface scattering details to enable complex objects such as translucent material (i.e., skin, jade). We then implement our proposed representations with an end-to-end physically-based neural face skin shader, which takes a standard face asset (i.e., geometry, albedo map, and normal map) and an HDRI for illumination as inputs and generates a photo-realistic rendering as output. Extensive experiments showcase the quality and efficiency of our PBR face skin shader, indicating the effectiveness of our proposed lighting and material representations. 

Published as a conference paper at ICLR 2023
LIGHT SAMPLING FIELD AND BRDF REPRESENTA- TION FOR PHYSICALLY-BASED NEURAL RENDERING
Jing Yang', ; Hanyuan Xiao ; Wenbin Teng, Yunxuan Cai, Yajie Zhao
Institute for Creative Technologies
University of Southern California
(jyang, hxiao, wteng, ycai, zhao)eict usc.ct..uuc.uu
ABSTRACT
Physically-based rendering (PBR) is key for immersive rendering effects used widely in the industry to showcase detailed realistic scenes from computer graph- ics assets. A well-known caveat is that producing the same is computationally heavy and relies on complex capture devices. Inspired by the success in qual- ity and efficiency of recent volumetric neural rendering, we want to develop a physically-based neural shader to eliminate device dependency and significantly boost performance. However, no existing lighting and material models in the current neural rendering approaches can accurately represent the comprehensive lighting models and BRDFs properties required by the PBR process. Thus, this paper proposes a novel lighting representation that models direct and indirect light locally through a light sampling strategy in learned light sampling field. We also propose BRDF models to separately represent surface/subsurfat scattering de- tails enable complex objects such translucent material (i.e., skin, jade). We then implement our proposed representations with an end-to-end physically- neural face skin shader, which takes a standard face asset (i.c., geometry, albedo map, and normal map) and an HDRI for illumination as inputs and generates a photo-realistic rendering as output. Extensive experiments showcase the quality and efficiency of our PBR face skin shader, indicating the effectiveness of our proposed lighting and material representations.
INTRODUCTION
Physically-band hased rendering (PBR) provides a shading and rendering method to accurately represent how light interacts with objects virtual 3D scenes. Whether working with a real-time rendering system in computer graphics or film production, employing PBR process will facilitate the creation of images that look like they exist in the real world for a more immersive experience. Industrial PBR pipelines take the guesswork out of authoring surface attributes like transparency since their method- ology and algorithms are based on physically accurate formulae and resemble real-world materials. This process relies on onerous artist tuning and high computational power in long production ey- cle. In recent years, academia has shown incredible success using differentiable neural rendering in extensive tasks such as view synthesis (Mildenhall al.) 2020). inverse rendering (Zhang al.) 2021a), and geometry inference (Liu et 20019). Driven by the efficiency of neural rendering, natural next step would be to marry neural rendering and PBR pipelines. However, none of the ex- isting neural rendering representations supports the accuracy, expressiveness, and quality mandated by the industrial PBR process.
A PBR workflow models both specular reflections, which refers to light reflected off the surface, and diffusion or subsurface scattering, which describes the effects of light absorbed or scattered in- ternally. Pioneering works of differentiable neural shaders such as Softras (L.iu et al.) 2019) adopted the Lambertian model as BRDF representation, which only models the diffusion effects and results in low-quality rendering. NeRF (Mildenball et al. 2020) proposed novel radiance field repre- sentation for realistic view-synthesis under an abit-b lighting transport assumption without explicitly modeling BRDFs or lighting, and hence is limited to a fixed static scene with scope for relighting. In follow-up work, NeRV (Srinivasan et al., 2020) took one more step by explicitly modeling directional light, albedo, and visibility maps make the fixed scene relightable. The indi- rect illumination was achieved by ray tracing under the assumption of one bounce of incoming light.
* Equal contributions. We would like to thank Marcel Ramos and Chinmay Chinara at Vision and Graphics Lab (VGL.) for their valuable help data preparation and paper writing.

---

Published as a conference paper at ICLR 2023
However, this lighting model is computationally very heavy for real-world environment illumination when more than one incoming directional lights exist. To address this problem, NeRD (Boss et al. 2020) and PhySG (Zhang et al. 2021a) employ low-cost global environment illumination mod- eling method using spherical gaussian (SG) to extract parameters from HDRIs. Neural-PII. (Boss) et al., (02T) further proposed pre-trained light encoding network for more detailed global illu- mination representation. However, it is still a global illumination representation assuming the same value for the entire scene, which is not true in the real world, where illumination subjected to shadows and indirect illumination bouncing off of objects in different locations in the scene. Thus it's still an approximation but not an accurate representation of the environmental illumination. Re- garding material (BRDF) modeling, all the current works adopt the basic rendering parameters (such as albedo, roughness, and metalness) defined in the rendering software when preparing the synthetic training data. However, they will fail in modeling intricate real-world objects such as participating media (c.g., smoke, fog) and translucent material (organics, skins, jade), where high scattering and subsurface scattering cannot be ignored. Such objects require more effort and hence attract more interest in research in their traditional PBR process.
In this work, we aim to design accurate, efficient lighting/ illumination and BRDF representations to enable the neural PBR process, which will support hugh-quality and photo-realiste rendering in fast and lightweight manner. To achieve this goal, we propose a novel lighting representation a Light Sampling Field to model both the direct and indirect illumination from HDRI environment maps. Our Light Sampling Field representation faithfully captures the direct illumination (incom- ing from light sources) and indirect illumination (summary of all indirect incoming lighting from surroundings) given an arbitrary sampling location in continuous field. Accordingly, we propose BRDF representations in the format of surface specular, surface diffuse, and subsurface scattering for modeling real-world object material. This paper mainly evaluates the proposed representations with a novel volumetric neural physically- extensive high-quality database, including real captured ground truth images as well as synthetic im- ages for illumination augmentation. We also introduce novel way of integrating surface normals into volumetric rendering for higher fidelity. Coupled with proposed lighting and BRDFs models, our light transport module delivers pore-level realism in both and underneath-surface appearance unprecedentedly. Experiments show that our Light Sampling Field is robust enough to learn illumi- nation by local geometry. Such effect usually can only be modeled by ray tracing. Therefore, our method compromises neither efficiency nor quality with the Light Sampling Field when compared to ray tracing.
shader for human facial skin. We trained with an
The main contributions of this paper are as follows: 1) A novel volumetric lighting representation that accurately encodes the direct and indirect illumination positionally and dynamically given an environment map. Our local representation enables efficient modeling of complicated shading ef- fects such as inter-reflectance in neural rendering for the first time as far as we are aware. 2) A BRDF measurement representation that supports the PBR process by modeling specular, diffuse, and sub- surface scattering separately. 3) A novel and lightweight neural PBR face shader that takes facial skin assets and environment maps (HDRIs) as input and efficiently renders photo-realistic, high- fidelity, and accurate images comparable to industrial traditional PBR pipelines such as Maya. Our face shader is trained with image database consisting of extensive identities and illuminations. Once trained, our models will extract lighting models and BRDFs from input assets, which works well for novel subjects/ illumination maps. Experiments show that our PBR face shader signifi- cantly outperforms the sate-of-theearr neural face rendering approaches with regard to quality and accuracy, which indicates the effecctivnnes of the proposed lighting and material representations.
RELATED WORK
Volumetric Neural Rendering Volumetric rendering models the light interactions with volume densities of absorbing, glowing, reflecting, and scattering materials (Max] 1995). A neural volu- metric shader trains a model from a set of images and queries rendered novel images. The recent state-of-the--ar was summarized in survey (Tewari et al. (022). In addition, Zhang et al. (2019) introduced the radiance field as a differentiable theory of radiative transfer. Neural Radiance Field (NeRF) (Mildenhall et al., 2020) further described scenes as differentiable neural representation and the following raycasting integrated color in terms of the transmittance factor, volume density, and the voxel diffuse color. Extensions to NeRF were developed for better image encoding (Yu et al. 2020
ray marching (Bi et al. 2020a),, network efficiency (Lombardi et al. 2021; Yariy et al.] 2020). real-
istic shading (Suhail et al. 2022 and volumetric radiative decomposition (Bi et al. 2020b Rebain
2

---

Published as a conference paper at ICLR 2023
et al. 2020; Zhang et al., 2021c Verbin et al. 2021). In partieular, NeRV (Srinivasan et al. 2020 NeRD (Boss et al. 2020; 2021) decomposed the reconstructed volume into geometry. SVBRDF BRD. and illumination given set of images under varying lighting conditions. RNR (Chen et al. 2020b) assumed the environmental illumination as distant light and is able to decompose the scene into an albedo map with 10-order spherical harmonics (SH) of incoming directions.
Portrait and Face Relighting Early single-image relighting techniques utilize CNN-based image translation (Nabbach et al. 2017 Thies et al. 2019). Due the lack of 3D models, image trans- lation approaches cannot recover surface materials or represent realistic high-fidelity, details, thus neural volumetric relighting approaches are widely adopted recently. Ma et al. (2021) proposed a lightweight representation to decode only the visible pixels during rendering. [i et al. (2021) The face relighting utilized strong priors. /hhou al. (2019) fitted a 3D face model to the input image and obtained refined normal to help achieve relighting. Chen et al. (2020a) relit the image by ing spherical harmonics lighting on a predicted 3D face. Hou et al. (2022) introduced a shadow mask estimation module to achieve novel face relighting with geometrically consistent shadows. With high-quality volumetric capture in lightstage (Debevec et al. 2000) to obtain training data, this trend achieved the following: regression of one-light-at-al light-at-a-tie time (OLAT) image for relighting (Meka et al. 2019), encoding the feature tensors for a Phong shading into UV space and relighting using an HDRI map (Meka et al. 2020), neural renderer that can predict the non-diffuse residuals (Zhang et al. 2021b). Bi et al. (202l) proposed neural networks to learn relighting implicitly but lacked modeling both surface and subsurface reflectance properties following physical light transport. Sim- ilar to our approach most, [Sun et al. (2021) inferred both light transport and density, and enabled relighting and view synthesis from sparse set of input images.
3
METHODS
3.1
PRELIMINARIES
Welertn
Gl
IN
38
Figure 1: Skin Scattering Model. The incident light L; scattered from the skin consists of an oil layer specular reflection (a) or pen- etrates one or more scattering layers (b, c, d) at some point under- neath the oil layer. The difference in melanin distribution or color results in the diversity of skin appearance. In humans, the distribu- tion and color of melanin is the primary determinant of skin appear- ance. Darker and denser melanin in the epidermis leads to darker skin color and vice versa.
Hemogishiee
layer
Ipiderma
We use the rendering equation Kajiya 1986) to estimate the radiance L at a 3D point x with the outgoing direction wo: L(x, Wo) E. f(x, wo: L, (1: where f(x, Wor wi) the BRDF representation and L, L; [x, wx) measures the radiance of incident light with direction wi-
Generally, the incident light could be categorized as direct vs. indirect light. Fig. illustrates an example of direct light (path (a). (b)) and indirect light (path (c)) in human face skin, where subsurface scattering happens in the deeper dermis layer, causing the nearby area receive indirect light. Therefore, the rendering formulation can be split into separate components with direct and indirect lighting:
L(x, wi) = / fo(x, wa WI) (w) (w )dw; + fxs war Lid (x, (w, ddw, (1)
w.ED+
w.Ef
where far Saa represent the BRDF evaluation of surface and subsurface, respectively. Following the render equation, we design lightweight physically-basee rendering method by learning different BRDF representations and modeling the direct and indirect lights, as we will introduce in the next few sections.
3.2 ILLUMINATION AND BIDIRECTIONAL REFLECTANCE DISTRIBUTION LEARNING
We propose a lighting model and material model to construct the Light Sampling Field and estimate the BRDF representations, as will be introduced in the next several sections.
3.2. LIGHTING MODEL
Considering the interaction between light and different skin layers, we propose a novel light model- ing method using HDRIs that decomposes lighting into direct illumination and indirect illumination.
3

---

Published as a conference paper at ICLR 2023
For direct illumination, We use light importance sampling simulate external light sources, such as light bulbs. And we implement ray tracing for the specular reflectance effect. For indirect illumina- tion, we introduce Light Sampling Field that models location- -wwtre illumination using SH. This learned local light-probe models subsurface scattering and inter-reflectance effects.
HDRI Image
Uniform Spherical Sampling
Importance Sampling
(b) Light Sampling Field Visualization
(a) Importance Light Sampling.
Figure 2: Lighting Model. (al We use importance sampling to sample heavily on pixels with high
intensity. (b) We use various densities of sampling bring attention locations with data as in the
Light Sampling Field.
Importance Light Sampling for Direct Illumination. Direct radiance comes from the HDRI map directly. In order to compute the contribution of pixels in HDRI to different points our radiance field, ment map onto a sphere. Each pixel on the sphere is regarded as distant directional light source. Hence, direct lighting is identical at all locations in the radiance field. Such representation preserves specular reflection usually achieved by ray tracing methods, We take two steps to construct the rep- resentation. Firstly, we uniformly sample a point grid of size point is light source candidate. Secondly, we apply our importance light sampling method to filter valid candidates by two thresholds: 1) an intensity threshold that clips the intensity of outliers with extreme values; 2) an importance threshold that filters out outliers in textureless regions. Fig. 2a illustrates this process.
SkyDome represent direct illumination by projecting the HDRI environ-
800 on the sphere, where each
Position
Specular Strength
BRDF
Position
Light Sampling Indirect Field
Lighting
V
fss
C
Lid
View Direction
Scattoring
View
Direction
MLP
n
Sass
6
Outputs
Direct Light
albedo P
fad
Embedding
Fincodig
(a) Material Network
(b) Light Sampling Field Network
Figure 3: Material and Light Sampling Field Network. Our material network takes in 3D po- sition and view direction inputs and predicts the specular strength and skin scattering parameters of our BRDF model. (b) The Light Sampling Field network applies similar differentiable network but with the light sampling created by our importance light sampling technique another input to predict SH coefficients.
Light Sampling Field for Indirect Illumination. Indirect illumination models the incoming lights reflected emitted from surrounding objects, which is achieved by ray tracing with the assumption of limited bounce times in the traditional PBR pipeline. Inspired by the volumetric lightmaps used in Unreal Engine (Karis & Games 2013). which stores precomputed lighting in sampled points and use them for interpolation at runtime for modeling indirect lighting of dynamic and moving objects, We adopt a continuous Light Sampling Field for accurately modeling the il- lumination variation in different scene positions. We use Spherical Harmonics (SH) to model the total incoming lights at each sampled location separately. We compute the local SH by multiplying fixed Laplace's SH basis with predicted SH parameters. Specifically, each color channel (RGB). Therefore, we acquire 3 (color channels) x 4 (basis) = 12-D vector as local SH representation. We downsample the HDRI map to 100 150 resolution and project it to sphere. Each pixel on the map is considered an input lighting source. We use the direction and color of each pixel the lighting embedding to feed into light field sampling network for inference of
use SH of degree = 1 for
4

---

Published as a conference paper at ICLR 2023
coefficients of local SH. We visualize our Light Sampling Field with selected discrete sample points in Fig. 2b
Learning Light Sampling Field. We design network (Fig. 3b) to predict the spherical Harmon- ics coefficient C of a continuous light field. The inputs of this network are the lighting embedding 2', the positional encoding of 3D location x, and view direction w. Conditioned on the lighting rep- resentations, the network succeeds in predicting the accurate and location-aware lighting. Fig. 6bb evaluates our lighting model.
3.2.2 MATERIAL MODEL
The choice of reflectance parameters usually relies artists' tuning and therefore requires high computational power and results in a long production cycle, but sometimes not very ideal. To tackle the problem, propose lightweight material network (Fig. (a) estimate the BRDF parameters including specular strength E R and skin scattering 7 E I through learnable parameters. The parameters are crucial represent the reflectance property of both surface and subsurface. Together with input albedo p. we can construct a comprehensive BRDF to model surface reflection and sub- surface scattering. It consists of surface specular component fas surface diffuse component J'dd. and subsurface scattering component fees. Refer to Sec. 3.3 for a detailed light transport and Fig for an evaluation of modeling subsurface scattering.
3.3 LIGHT TRANSPORT
Light transport defines a light path from the luminaire to the receiver. We introduce the light trans- port that connects our lighting model and material model in Fig. 4 We also detail the rendering equation in this section to match the light transport along the light path.
Sormal
8300:
W
Brastai
Saning le Necional
M
Geraur
Lightoo
Laiined
Benderag
Light Transport. Starting from sampling lights from HDRI and 3D points close to geometry along the light path, feed encoded locations. view direction to Material network and Light Sampling Field network (with extra direct illumination code). The networks output material and indirect illumination code. Our PBR equation generates radiance each sample based material model and light transport in the volume and further composites all the radiance on the light path to the receiver.
Figure 4:
Volume Casting. Our lighting model includes direct illumination and indirect illumination. Light transports from explicit lights and casts along the light path. We, therefore, define the light transport along the light path, p. in the volume by the following:
L(X, = (f) (p(t)) L (p(t), wo) dt
(2)
where L(*, w.) the total radiance at & along the light path direction wa. (xx) the volume density, which is converted from the input geometry. r(x) exp(- So (p(()))dt) is the visibility that indicates whether the location is visible the light path, where p(t) represents the 3D location on the light path along w. from & and defined as p(t) = x two.
We use explicit geometry construct the intersection coordinates, Xg. between any arbitrary light path and the input geometry. For any point along the light path w, define the density the following central Gaussian distribution
reliable density field. More specifically, we locate
5

---

Published as a conference paper at ICLR 2023
function: (xx as exp (xdc(xx))((2))))) where (x(x) is the distance between and the intersection Xo. Gt and å are two scalars that determine the magnitude and standard deviation of the gaussian distribution.
Material Scattering. The light transport between the scene objects and light sources is charac- terized by the rendering equation. Eqn. introduce the rendering covering direct illumination and indirect illumination. Also, classified by reflection location, the reflected radiance T(x w.) has two components: surface reflectance and subsurface volume scattering. To have comprehensive light transport representation, we further develop the equation with the dissection of light as well as the specialized BSSRDF components:
L(x, / wi
/ fsa(x, wor Li wi)l n nx we
fxo wo, wi) Wi) (x,w))) nx
surface specular reflectance with direct light
surface diffuse reflectance with direct light
+ f
Sana(X, (x, Wo: w;) (x, wi) wi nx ddw;
WiEN
subsurtace scattering with indireet light
here (x, wi) and 1,d(xx wi) are the incoming direct and indirect radiance from direction w; at point x, respectively. fes. fed. and fees are the different counterparts of light transport parameterized by material representations. We show the complete light transport in Eqn. Aconsists of our light and material representation.
L(x, wo) = 7xx- I
[x wi) Hillloo-((a R(w,, nx) do, +
P&
w.Ef LX wi)l ny dw,
Ef.
surface spocular reflectance with direct light
surface diffuse reflectance with direct light
Px + Txx
T. (x, wi))k ny Idws;
(4)
WIER
subsurface scatlering with indirect light where 7x and 7ix are the predicted specular strength and scattering from material network at x. R(w,, nx) denotes the reflection direction of w, at the surface with normal ny and e denotes the specular exponent. Also, pE and P are surface and subsurface albedo sampled from the input albedo map correspondingly with geometry.
IMPLEMENTATION DETAILS
In order to construct the density field a, we a, and ô to be 10 and 0.5, respectively. We further compare the rendering results of other values and visualize them in the Appendix. In the constructed radiance field, to sample rays, we draw 1024 random rays per batch. Along each ray, we sample 64 points for the shading model. The low-frequency location of 3D points and direction of rays are transformed to high-frequency input via positional encoding and directional encoding respec- tively (Mildenhall et al., 2020). The length of encoded position and view direction is 37 and 63 respectively in the material network and the Light Sampling Field network. Also, importance light sampling takes 800 light samples e E from the HDRI input for direct lighting. We further down- sample the input HDRI and embedded all pixels as a light embedding z E 15x15000 to feed into the Light Sampling Field network. We networks. For the material network, encoded sample locations are fed in the first layer of the MLP, while the encoded view direction is later fed in layer 4. The output of material MLP for each queried 3D point is specular strength E IR, and scattering 1) E Re The Light Sampling Field network has similar network structure but also has direct light embedding coded spherical coefficients CT E IR for indirect lighting. During light transport, we obtain weighted value for each ray based on distribution among the sampled points along the ray. After introducing values from pre-processed albedo and normal maps, the value of each component on rays is gathered and visualized as an image with pixels representing their intensities following our rendering Eqn. Finally, the rendered RGB values are constrained with ground truth by an MSE loss. our application, MLP modules can converge in 50, 000 iterations (2.6 hour) on a single Tesla V V100, with decent results the same level of detail as reference images.
8-layer MLP with 256 neurons in each layer for both
layer 4 as input and outputs en-

---

Published as a conference paper at ICLR 2023
5
EXPERIMENTS AND EVALUATION
5.1 TRAINING DATASET
Our training dataset composed of a synthetic image dataset and a Lightstage-scanned image dataset. In synthetic dataset, we used a professionally-tuned Maya face shader to render 40-view colored images under all combinations between 21 face assets and 101 HDRI+86 OLAT illumi- nation. Lightstage-see expressions under white illumination. We carefully selected subjects in both dataset preparation to cover diverse ages, skin colors, and gender. Further details can be found in Appendix.
dataset consists of 16-view captured colored images of 48 subjects in 27
5.2 EVALUATION AND ANALYSIS
Lambettin
Specular
Figure 5: Material Modeling. Under HDRI illumination in column (a), we show the appearance of (b) Lambertian BRDF, (c) Cook-Torrance BRDF, and (d) (including direct diffuse, direct specular, and indirect diffuse). We adjust the intensity of each component for visualization purposes.
(a) Appearance at three different locations under the same illumination
NA
NA
HDRI
SH
SG
Neuni-PL
Ours
9G
Neurs-PI
Ours
(b) Lighting Model Evaluation.
Figure 6: Light Evaluation. @ap Given HDRI illumination, we capture the appearance of mirror ball and grey ball in the Light Stage at different locations. (b) We rendered a uniform white ball and grid of such balls with different lighting models: SH (degree I 1), SG, Neural-PIL, and ours. We separated diffuse lighting and specular lighting by turning off the corresponding component in the rendering equation.

---

Published as a conference paper at ICLR 2023
Material Model Evaluation. We conduct comparison experiment with only surface scattering (BRDF) in Fig. 5 which presents two BRDF materials (middle two columns) and our proposed material with layer-by-layer decomposition (right four columns). From the comparison, Cook- Torrance BRDF has a more specular effect than Lambertian yet saves rubber-alike appearance from the uncanny valley. Apart from surface scattering. our method also predicts subsurface scattering to achieve a vivid look around the nose tip and ears by depicting red blood cells and vessel color.
Lighting Model Evaluation. We show our captured real data under a real fixed HDRI illumination in Fig. 6a Fig. 6b illustrates uniform white ball illumination within scene rendered by SH lighting (degree I = 1), Spherical Gaus- sian (SG) Neural-PIL (Boss et al., 2021) and our method, respectively. Compared with other models, our proposed method delivers the highest fidelity of illumination with the widest spectrum of light well as lighting field sensi- tive to lighting distribution. We further provide an extensive ablation study to validate our light sampling for modeling direct illumination in Fig. 15
Evaluation of Light and Material Modeling in indirect illumination. We evaluate our light and material compo- nents Fig. 7 In (c), we use pre-calculated SH of degree I == 1 to model the global illumination and albedo dif- fuse scattering to render the face, resulting in a face im- age with strong shadows and an unnatural appearance. In (b), we introduce learned subsurface scattering in material but still use the same pre-calculated SH of degree / = as global illumination, which results in color shift and arti- facts. In (c), we further introduce local SH and infer a light sampling field to replace the pre-calculated SH. Together with the full spectrum of material layers, we achieve real- istic rendering effects. In particular, we demonstrate inter- reflection effects the zoom-in box. The shadow soft- ened by modeling the scattering and positional illumination.
Ib
Figure 7: Evaluation of light and material modeling. (a) Rendering results using proposed materials and location-aware local SH as lighting model. (b) Rendering results us- ing proposed materials with global SH. (c) Rendering results with global SH and without using the subsurface scattering layer the material. We also demonstrate the inter-reflection effects in the zoom-in boxes. Com- pare to (c) and (b), (a) achieves soft shadows.
Inverse Rendering. Our method can also achieve high- fidelity inverse rendering with multi-view images (under various illumination) instead of geometry and texture maps as input. To make this possible, we additionally imple- mented MLP to predict a density field. We present our results under novel illumination in Fig. g
Figure 8: Inverse Rendering, (a) An example input of a set of multi-viw im- ages. (b) Reference view under the in- put illumination. (c) Inverse rendering results in two novel environment illumi- nations.
(6)
5.3 QUALITATIVE RESULTS
Rendering on General Object Assets. We picked orange and meat in addition to face subjects to show that our method is gen- eralizable on diverse objects with multi-layered structures Fig. 9 Through testing different organic materials, tent sharpness and realistic appearance, especially in accomplishing specular reflection orange and accurate soft shadow meat.
show consis-
Maya Comparison. We compare our renderings with Maya, Figure 9: General objects.
industry-level rendering engine, under HDRI or OLAT (one-light-at----ime illumination and present zoom-in pore-level details in Fig. 10 The zoom-in inspections show comparable even better
8

---

Published as a conference paper at ICLR 2023
HDRI
(b) OLAT
Figure 10: Qualitative comparison between ours and Maya renderings with zoom-in under (a) HDRI and (b) OLAT. The left images and green crops in each pair are Maya renderings. The right images and yellow crops in each pair are ours. Our result achieves a realistic appearance of skin texture in addition accurate diffuse and ambient illumination effects.
rendering results in their sharpness illumination. Skin wrinkles and forehead specularity particularly rich and sharp from the proposed method. At testing time, with the same rendering assets and queries (2500 2500 in resolution), 800 x 800 in resolution. Under OLAT settings, our method casts hard shadows and soft shadows as accurately Maya. More comparisons and qualitative results can be found Fig. and Fig. [8
method requires the training images with only
Qualitative Comparison. In Fig. I6 2021), NeLF (Sun et al. 2021), SIPR (unn et al., 2019), Neural-PIL (Boss et al. 2021), and our methods under HDRI and OLAT illuminations. We present more testing performance of our trained- once models on other novel subjects in other datasets in Fig 19
compare the rendering results of FRF (Tewar et al.,
5.4 QUANTITATIVE RESULTS
We evaluated PSNR, SSIM, and LPIPS quantitative measurements (Table with Maya rendering as benchmark. Specifically, LPIPS evaluates the perception on the VGG-19 network. netwrkk. Ours outper- forms other baseline methods in three metrics. Moreover, We not provide the SSIM score of NeLF due to the slight view difference.
Baseline PSNR SSIM f
FRF
SIPR
NeLF
Neural-PII.
Ours
29.83
33.43
33.80
33.22
36.69
0.4489
0.8507
N/A
0.9153
0.9250
LPIPS (VGG) 4
0.6017
0.1265
0.7538
0.1082
0.0664
Table 1: Facial rendering metrics at 800 x 800 resolution.
CONCLUSION
We demonstrate that the prior neural rendering representation for physically-based rendering fails to accurately model environment lighting or capture subsurface details. In this work, we propose a differentiable light sampling field network that models dynamic illumination and indirect lighting in a lightweight manner. In addition, we propose a flexible material network that models subsurface scattering for complicated materials such the human face. Experiments both synthetic and real- world datasets demonstrate that our light sampling field and material network collectively improve the rendering quality under complicated illumination compared with prior works. In the future, we will focus on modeling more complicated materials such as translucent materials and participated media. We will also collect datasets based on general objects and apply them for extensive tasks such as inverse rendering.

---

Published as a conference paper at ICLR 2023
7 ACKNOWLEDGMENT
This research is sponsored by the U.S. Army Research Laboratory (ARL) under contract number W91 INF-14-D-0005. Army Research Office also sponsored this research under Cooperative Agree- ment Number W911NF-20-2-0055
We would also would like to acknowledge Sony Corporation of America R&D Center, US Lab for their support. Statements and opinions expressed, and content included, do not necessarily reflect the position or the policy of the Government, and no official endorsement should be inferred. Further, the views and conclusions contained in this document are those of the authors and should not be interpreted as representing the official policies, either expressed or implied, of the Army Research Office or the U.S. Government. The U.S. Govern- ment is authorized to reproduce and distribute reprints for Govermment purposes notwithstanding any copyright notation herein.
REFERENCES
Sai Bi, Zexiang Xu, Pratul Srinivasan, Ben Mildenhall, Kalyan Sunkavalli, Miloš Hašan, Yannick Hold-Geoffroy, David Kriegman, and Ravi Ramamoorthi. Neural reflectance fields for appearance acquisition. arXiv preprint arXiv:2008. 208.02000...8244 2020a.
Sai Bi, Zexiang Xu, Kalyan Sunkavalli, Miloš Hašan, Yannick Hold-Geoffroy, David Kriegman, and Ravi Ramamoorthi. Deep reflectance volumes: Relightable reconstructions from multi-view photometrie images. arXiv preprint ND0:20007.0992. 2020b.
Sai Bi, Stephen Lombardi, Shunsuke Saito, Tomas Simon, Shih-En Wei, Kevyn Mcphail, Ravi Ra- mamoorthi, Yaser Sheikh, and Jason Saragih. Deep relightable appearance models for animatable faces. ACM Transactions on Graphics (TOG), 40(4):1-15, 2021.
Mark Boss, Raphael Braun, Varun Jampani, Jonathan T Barron, Ce Liu, and Hendrik Lensch. Nerd: Neural reflectance decomposition from image collections. CoRR, 2020.
Mark Boss, Varun Jampani, Raphael Braun, Ce Liu, Jonathan Barron, and Hendrik Lensch. Neural- pil: Neural ireeinterrated lighting for reflectance decomposition. Advances in Neural Information Processing Systems, 34, 2021.
Yajing Chen, Fanzi Wu, Zeyu Wang, Yibing Song, Yonggen Ling, and Linchao Bao. Self-supervised learning of detailed 3d face reconstruction. IEEE Transactions Image Processing, 29:8696- 8705, 2020a.
Zhang Chen, Anpei Chen, Guli Zhang, Chengyuan Wang, Yu Ji, Kiriakos N Kutulakos, and Jingyi Yu. A neural rendering framework for free- free-viewpoint relighting. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 5599-5610, 2020b.
Paul Debevec. The light stages and their applications to photoreal digital actors. SIGGRAPH Asia, 2(4):1-6, 2012.
Paul Debevec, Tim Hawkins, Chris Tchou, Haarm-Pieter Duiker, Westley Sarokin, and Mark Sagar. Acquiring the reflectance field of human face. In Proceedings of the 27th annual conference on Computer graphics and interactive techniques, pp. 156, 2000.
Abhijeet Ghosh, Graham Fyffe, Borom Tunwattanapong, Jay Busch, Xueming Yu, and Paul De- bevec. Multiview face capture using polarized spherical gradient illumination. In Proceedings of the 2011 SIGGRAPH Asia Conference, pp. 1-10, 2011.
Andrew Hou, Michel Sarkis, Ning Bi, Yiying Tong, and Xiaoming Liu. Face relighting with geo- metrically consistent shadows. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 4217-4226, 2022.
James T. Kajiya. The rendering equation. SIGGRAPH Comput. Graph., 2044):143-150, aug 1986. ISSN 0097-8930. doi: 10.1145/15886. 15902. URL /ttp://///1...0g00.111111555//10 15 15902
Brian Karis and Epic Games. Real shading in unreal engine 4. Proc. Physically Based Shading Theory Practice, 4:3, 2013.
10

---

Published as a conference paper at ICLR 2023
Shichen Liu, Tianye Li, Weikai Chen, and Hao Li. Soft rasterizer: A differentiable renderer for image-based 3d reasoning. The IEEE International Conference on Computer Vision (ICCV), Oct 2019.
Stephen Lombardi, Tomas Simon, Gabriel Schwartz, Michael Zollhoefer, Yaser Sheikh, and Ja- son Saragih. Mixture of volumetric primitives for efficient neural rendering. arXiv preprint 103.1:2003.01954 2021.
Shugao Ma, Tomas Simon, Jason Saragih, Dawei Wang, Yuecheng Li, Fernando De La Tore, and Yaser Sheikh. Pixel codec avatars. In Proceedings of the IEEF/CVF Conference on Computer Vision and Pattern Recognition, pp. 64-73, 2021.
Wan-Chun Ma, Tim Hawkins, Pieter Peers, Charles-Feil Chabert, Malte Weiss, Paul E. Dehevec, et al. Rapid acquisition of specular and diffuse normal maps from polarized spherical gradient illumination. Rendering Techniques, 2007(9):10, 2007.
Nelson Max. Optical models for direct volume rendering. IEEE Transactions Visualization and Computer Graphics, 12):999-08,, 1995.
Abhimitra Meka, Christian Haene, Rohit Pandey, Michael Zollhöfer, Sean Fanello, Graham Fyffe, Adarsh Kowdle, Xueming Yu, Jay Busch, Jason Dourgarian, et al. Deep reflectance fields: high- quality facial reflectance field inference from color gradient illumination. ACM Transactions on Graphics (TOG), 38(4):1-12, 2019.
Abhimitra Meka, Rohit Pandey, Christian Häne, Sergio Orts-Escolano, Peter Barnum, Philip David- Son, Daniel Erickson, Yinda Zhang, Jonathan Taylor, Sofien Bouaziz, et al. Deep relightable textures: volumetric performance capture with neural rendering. ACM Transactions on Graphics (TOG), 39(6):1-21, 2020.
Ben Mildenhall, Pratul P Srinivasan, Matthew Tancik, Jonathan T Barron, Ravi Ramamoorthi, and Ren Ng. Nerf: Representing scenes as neural radiance fields for view synthesis. In European Conference on Computer Vision, pp. 405-4211 Springer, 2020.
Oliver Nalbach, Elena Arabadzhiyska, Dushyant Mehta, H-P Seidel, and Tobias Ritschel. Deep shading: convolutional neural networks for screen space shading. In Computer graphics forum, volume 36, pp. 65-78. Wiley Online Library, 2017.
Daniel Rebain, Wei Jiang, Soroosh Yazdani, Ke Li, Kwang Moo Yi, and Andrea Tagliasacchi. Derf: Decomposed radiance fields. arXiv preprint w:Xiv:2011.124440 2020.
Pratul P Srinivasan, Boyang Deng, Xiuming Zhang, Matthew Tancik, Ben Mildenhall, and Jonathan T Barron. Nerv: Neural reflectance and visibility fields for relighting and view syn- thesis. arXiv preprint arXiv:2012.0392. 2020.
Mohammed Suhail, Carlos Esteves, Leonid Sigal, and Ameesh Makadia. Light field neural render- ing. In Proceedings of the IEEE/CVF Conference Computer Vision and Pattern Recognition, pp. 8269-8279, 2022.
Tiancheng Sun, Jonathan T Barron, Yun-Ta Tsai, Zexiang Xu, Xueming Yu, Graham Fyffe, Christoph Rhemann, Jay Busch, Paul E Debevec, and Ravi Ramamoorthi. Single image portrait relighting. ACM Trans. Graph., 38(4):79-1, 2019.
Tiancheng Sun, Kai-En Lin, Sai Bi, Zexiang Xu, and Ravi Ramamoorthi. Nelf: Neural light- transport field for portrait view synthesis and relighting. arXiv preprint arXiv:2107.1235. 1021.
Ayush Tewari, Ohad Fried, Justus Thies, Vincent Sitzmann, Stephen Lombardi, Kalyan Sunkavalli, Ricardo Bruaala. Tomas Simon, Jason Saragih, Matthias Nießner, et al. State of the art on neural rendering. In Computer Graphics Forum, volume 39, pp. 701-727. Wiley Online Library, 2020.
Ayush Tewari, Tae-Hyun Oh, Tim Weyrich, Bernd Bickel, Hans-Peter Seidel, Hanspeter Pfister, Wojeiech Matusik, Mohamed Elgharib, Christian Theobalt. et al. Monocular reconstruction of neural face reflectance fields. Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 4791 4890, 2021.

---

Published as a conference paper at ICLR 2023
Justus Thies, Michael Zollhöfer, and Matthias Nießner. Deferred neural rendering: Image synthesis using neural textures. ACM Transactions on Graphics (TOG), 38(4):1-12, 2019.
Dor Verbin, Peter Hedman, Ben Mildenhall, Todd Zickler, Jonathan T Barron, and Pratul P Srini- vasan. Ref-nerf: Structured view-dependent appearance for neural radiance fields. arXiv preprint arXiv:2112.03907, 2021.
Ting-Chun Wang, Ming- Yu Liu, Jun- Yan Zhu, Andrew Tao, Jan Kautz, and Bryan Catanzaro. High- resolution image synthesis and semantic manipulation with conditional gans. In Proceedings of the IEEE Conference Computer Vision and Puttern Recognition, 2018.
Lior Yariv, Yoni Kasten, Dror Moran, Meirav Galun, Matan Atzmon, Basri Ronen, and Yaron Lip- man. Multiview neural surface reconstruction by disentangling geometry and appearance. Ad- vances in Neural Information Processing Systems, 33, 2020.
Alex Yu, Vickie Ye, Matthew Tancik, and Angjoo Kanazawa. pixelnerf: Neural radiance fields from one or few images. arXiv preprint 2rXiv:2012.02190, 2020.
Cheng Zhang. Lifan Wu, Changxi Zheng, loannis Gkioulekas, Ravi Ramamoorthi, and Shuang Zhao. A differential theory of radiative transfer. ACM Transactions on Graphics (TOG), 38(6):
1-16, 2019.
Kai Zhang, Fujum Luan, Qianqian Wang, Kavita Bala, and Noah Snavely. Physg: Inverse rendering with spherical gaussians for physics-based material editing and relighting. In Proceedings of the IEEF/CVF Conference Computer Vision and Pattern Recognition, pp. 5453-5462, 2021a.
Xiuming Zhang, Sean Fanello, Yun-Ta Tsai, Tiancheng Sun, Tianfan Xue, Rohit Pandey, Sergio Orts Osco-asc,,,no, Philip Davidson, Christoph Rhemann, Paul Debevec, et al. Neural light transport for relighting and view synthesis. ACM Trunsactions on Graphics (TOG), 40(1):1-17, 2021b.
Xiuming Zhang, Pratul Srinivasan, Boyang Deng, Paul Debevec, William Freeman, and Jonathan T Barron. Nerfactor: Neural factorization of shape and reflectance under an unknown illumination. arXiv preprint arXiv:2106.01977 2021c
Hao Zhou, Sunil Hadap, Kalyan Sunkavalli, and David W Jacobs. Deep single-image portrait re- lighting. In Proceedings of the IEEE/CVF International Conference Computer Vision, pp. 7194-7202,22 2019.
12

---

Published as a conference paper at ICLR 2023
A
DATASETS
40
33
20
35
49
62
61
47
32
19
12
21
36
50
63
62
73
60
18
22
64
68
17
65
76
B5
39
45
16
5
13
42
57
81
17
B2
71
44
29
10
28
43
38
66
77
84
26
24
40
Figure 11: OLAT Mapping. (a) Area labels for OLAT mapping: (b) Spherical mapping; (c) OLAT mapping and renderings.
a
(b)
Figure 12: Training Dataset. (a) Light Stage-scanned real image dataset, (b) Maya-rendered syn-
thetic image dataset
A.1 LIGIT STAGE
Face Capture. We use the Light Stage (Debevec 2012) capture data for training purposes. The Light Stage features controllable lights and cameras, allowing us capture multiview images using polarized spherical gradient illumination (Ma al.) 2007 Ghosh et al.) 201 By decomposing the specular and diffuse surface reflections from the captured images, diffuse albedo and high-frr figh-frequency normals. We further fit the reconstructed 3D face to our template mesh for consistent UV space textures.
generate image space
simulate OLAT illumination, we approximate directional light source by using an area light source positioned on sphere. We then convert this area light into a High Dynamic Range Image (HDRI) map using equirectangular mapping. We illustrate the segmentations and corresponding OLAT mapping in Fig.
OLAT Mapping.
A.2 TRAINING DATASET
Our training dataset consists of 1) a Light Stage-scanned multi-view colored image dataset under white illumination shown in Fig. 12 (a), 2) synthetic multi-vien colored image dataset rendered via a professionally-tuaet Maya face shader shown Fig. (b). In the following sections, we introduce data composition and settings for synthetic data rendering and lightstage scanning.
13

---

Published as a conference paper at ICLR 2023
Synthetic Image Dataset Our input rendering assets to Maya renderer 101 HDRI environment maps, 86 OLAT environment maps, and 21 face assets: the HDRI data covers various illuminations from outdoor open areas to small indoor areas; OLAT environment map cover 86 different regions in SkyDome as directional light; 21 subjects cover a variety of skin color, age, and gender. Each face asset consists of coarse mesh, albedo map, and high-frequency normal map. We rendered 40 fixed-view RGBA-space images under all combinations of illumination and face assets. In total, acquired 37, 160 images in 800 800-pixel resolution for our synthetic image dataset.
Lightetage-scanned Real Image Dataset Synthetic data is insufficient to train a model with out- put rendering close the real world. We utilized lightstage additionally capture multi-view images for 48 subjects under uniform white illumination. Besides the diverse skin colors, gender, and age of subjects, the lightstage-scanee
dataset consists 27 expressions of each subject. We set up 16 fixed cameras covering different viewpoints of the frontal view. In total, we acquired 20, 736 images 3008 x 4096-pixel resolution for our real image dataset.
B MORE RESULTS AND ANALYSIS
8 6- 0.2
0.25
0.5
1.0
2,0
Gy
Density Field Construction. We show how a, and affect the constructed density field in Fig. 13 Larger å results in more evenly dis- tributed density samples along the light path. The constructed density field hence presents a coarse boundary around the imput geome- try. Therefore, the rendered results tend to have blurry appearance. However, smaller 5 approaches Dirac delta distribution, where only samples close to the intersections have valid density values. The rendered results thus have black stripes. Larger a alleviates the false density construction and results in accu- rate density values around the input geometry.
10.0
Figure 13: Desnity Field Construction. We present a comparison of hyperparameters (a, and 5) in density field construction.
High-frequency Surface Normal. To inves- tigate the effectiveness of high-frequency sur- face normal as input, we compare results by re- placing geghffreqeecyy input normal maps with low-frequency one extracted from coarse geom- ctry (Fig. 14). Given other input assets unchanged, rendered results with low-frequency normal maps show the incapability of extracting high- figh-freyeency information from input images. Therefore, gigh-frequeny normal maps are crucial in achieving pore-level sharpness even though training data contains high-frequency details images.
Ablation Study on Light Sampling We val- idate the Importance Light Sampling for sim- ulating various external light sources with an extensive ablation study in Fig. 15 Under the same illumination conditions, the Importance Light Sampling can generate soft and appropri- ate diffuse reflection while preserving the accu- rate lighting distribution from the input HDRI. It ensures that the rendered images maintain the high degree of realism and fidelity to the orig- inal lighting conditions. In contrast, Uniform Spherical Sampling, while capable of repre- senting the lighting environment with the same number sampled lights, tends to produce hard shadows and may result in less detailed and over- exposed images.
Teguiny
ESTul
Figure 14: High-frequency Surface Normal
14

---

Published as a conference paper at ICLR 2023
(8)
(b)
Diffuse
Specular
Rendering
Diffuse
Speciier
Rendering
Figure 15: Ablation study on the impact of different light sampling techniques. We showcase the layer-by-layer decomposition of the direct lighting model using (a) Importance Light Sampling, and (b) Uniform Spherical Sampling. We adjust the intensity of each component equally for visualization purposes.
Input
Output
Novel Subjects
Novel Viewpoint
Methods
Known
Directional Light
Views
HDRI
Lighting
SIPR
Single-view
SH
FRF
Single-view
OLAT
4
Neural-PIL.
Single-view
OLAT
4
NeLF
Multi-view
Ours
Multi-view
HDRI
Table 2: Input, output, and functionality of our baselines. Multi-view indicates that we first convert geometry and textures multiview representation utilizing the ray-mesh intersection. Our method is capable of rendering novel subjects from novel viewpoints using either directional light or HDRI as queried illumination, without compromising on quality.
Qualitative Comparison. In Fig- T6 we compare the rendering results of FRE, NeLF, SIPR, Neural-PIL,, and our methods under HDRI and OLAT (one-light-att---imme)) illuminations, as well as using multi-view images as input instead geometry. To be more concrete, we identify different methods' input, output, and functionality in Table 2 Our method stands out for its ability to pro- duce clear hard shadows resulting from full occlusion by face geometry, and soft shadows caused by indirect illumination. In contrast, Neural-PIL. and NeLF do not model directional light and were not trained on OLAT data, so we compare them only under HDRI illumination. SIPR an image- based relighting method that models the scene in 2D, and cannot be queried from novel viewpoints. FRE, Neural-Pik PIL, and NeLF, on the other hand, models in 3D. Neural-PII inherits a per-scene-per train manner NeRF and, thus is not generalizable different subjects. In addition, provide rendering results from Maya, top-notch industrial renderer, as a reference for comparison.
More Qualitative Results. Fig. 18 presents comprehensive collection of qualitative results. Each row showcases the rendering inputs, including the geometry, albedo map, and normal map, followed by six rendered images. The first three show different facial expressions under all-white illumination, while the last three display neutral expressions under different lighting conditions. By utilizing our photo-realistic neural renderer, we are able to render images at any resolution without compromising quality.
Rendering Speed. In addition to the rendering results, also compare the rendering speed in Table With the same specification of output and environment, our method is able to achieve up to 47-49 times faster with engineering acceleration (e.g. multi-thread processing)
15

---

Published as a conference paper at ICLR 2023
Hort
CUAT2
GLAT3
GLAT
FRC
NeLF
8PP
Nors
Mova
Ouro
Ouea
SIFR
Majo
Ou
HDRI illumination
(b) OLAT illumination
Figure 16: Qualitative comparison with baseline methods. Under HDRI illumination, our rendering results achieve the highest sharpness and less plastic/rubher-like appearance, where Ours takes geometry and textures as input and Ours * takes multiview inputs; (b) OLAT illumination simulates strong directional lighting. While our robust direct light modeling enables hard shadow casting, our indirect light modeling realistically softens shadow at the boundary.
Resolution 800
800
1440 x 1440
2880 2880
5760
5760
Maya
130s
370s
1857s
7848s
Ours
3s
8s
31s
158s
Table 3: Facial rendering speed benchmark on different resolution in seconds
Testing Other Datasets. To demonstrate that high-quality training data does not determine the robustness of the method, we evaluated our trained model over other available resources in Fig. [9 We converted three datasets match our input follows,
Triplegangers. We used FaceX 3DMM to fit and align geometries; albedo map directly transferred from source; normal map was inferred using (Wang et al. (2018).
3D Scan Store. used 3DMM to fit and align geometries; albedo and normal maps were provided and not further processed.
FaceScape. Geometries and albedo maps were provided and not further processed. The normal map was unavailable and not used.
Our method delivers promising fidelity on all three testing datasets. First, our method is capable of adapting different mesh topologies. For example, meshes in the Triplegangers dataset has denser vertices in the front face than behind while meshes in FaceScape have more uniform density, but trained-once model performs equally well in both testing datasets. Second, method does not sacrifice fidelity when input normal is unavailable during testing. Thanks to explicit geometry volume and robust model, pixels in the albedo map are precisely mapped onto the surface and therefore, yield no noise in rendering.
Finally, we acquired and tested pre-trained model over the Generated Photos dataset containing only fow-frequency albedo and normal map in Fig, 20 The dataset is generated by some anonymous method with only single-view online image as input. Our method not only outputs clear silhouette but also shows realistic pre-level detail when high-frequency input is unavailable.
16

---

Published as a conference paper at ICLR 2023
HDRI
(b) OLAT
Figure 17: We show more qualitative comparisons of our method and Maya under OLAT, with zoom-in on the images.
(a) HDRI and (bp
17

---

Published as a conference paper at ICLR 2023
Figure 18: Qualitative results. In each row, We present our input (col. 1), three novel view dering with neutral expression (col. 2), two other expressions under white illumination (col. 3, 4), and three frontal view rendering with neutral expression under different HDRI illumination (col. 5. 6, 7). Geometry, albedo map and normal map in UV space are list in the first column from top to bottom.
18

---

Published as a conference paper at ICLR 2023
Sauppe pm
Bardrng
SAE
Figure 19: Testing performance of trained-once models other open source datasets. To demonstrate fidelity and generalization our method, we further test our pre-trained model on Triplegangers, FaceScape and 3D Store datasets. Besides accurate shading, our rendering illustrates sharp detail in pore, wrinkle and hair.
19

---

Published as a conference paper at ICLR 2023
Figure 20: Testing performance of our trained-once models on GeneratedPhotoo dataset (in-the- wild). Our model demonstrates high fidelity when accurate scans of albedo and normal map unavailable in this test.
20


Reviews:
[{"review_id": "rw83opEDZDK", "reviewer": "Reviewer_rwHM", "summary": "", "questions": "", "limitations": "", "rating": 8, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "This paper proposed a volumetric lighting representation that aims to model the indirect illumination given an illuminations map via a coordinate-based MLP. The proposed lighting representation achieves the state-of-the-art rendering quality on human faces, while being orders of magnitudes faster than the rendering engine. \nIn addition, the paper proposes a material network that explicitly models subsurface scattering on human face. \nBoth synthetic and real experiments demonstrate the render quality of the method outperforms prior works.", "summary_of_the_review": "Overall, the paper proposed a novel approach for volumetric lighting representation. Experiments show that the volumetric lighting representation achieves state-of-the-art performance on face rendering and is 47-49 times faster than the industry-level rendering engine.\nBut, some assumptions and details are not clearly stated in the paper. Please address the questions and concerns in the weaknesses part. ", "strength_and_weaknesses": "Strength\n\n- The proposed light sampling field is very novel and interesting to me. Method-wise, it is the first to model all the spatial-varying in-direct illumination in a coordinate-based MLP network. Speed-wise, it outputs the SH for fast and efficient rendering. Performance-wise, it generalizes well on novel subjects and illumination maps.\n\nWeaknesses\n\n- How is the material network generalized to different people? The input to material-net is only 3D position and ray direction, there is no person-specific information input to the network. So how can the material network generalize to different people with different skin properties? \nBesides, the skin properties may change spatially even for the same person with different expressions. \n- From the material network, the paper assumes that the specular strength and skin scattering are spatially identical across different subjects. If it is the case, the paper should mention this assumption in the paper. And I also feel that this assumption is a bit unrealistic. \n- In eq(2), what is the $\\omega$ stand for? View direction or light direction? It is very confusing that the notation is inconsistent between eq(2) and eq(3),(4).\n- How is the 3D position x defined? It seems that it is defined in a head-centric coordinates system. Please provide details of it. \n- In Figure 6, 2rd row, last 3 columns, it seems that the direct diffuse and indirect diffuse are somehow inconsistent with the 1st row. Is it a mistake? If not, can the author explain more? Why the direct diffuse here look more like indirect diffuse?", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "4: The contributions are significant, and do not exist in prior works.", "empirical_novelty_and_significance": "4: The contributions are significant, and do not exist in prior works."}, "extra_text": {}}, {"review_id": "oVxia7FLmh6", "reviewer": "Reviewer_oe43", "summary": "", "questions": "", "limitations": "", "rating": 8, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "The paper presents a physical-based neural rendering network. The network codes lights and BRDF using 2 subnet-light sampling field network and material network. Then they are feed to construct light transport to generate radiance.", "summary_of_the_review": "Overall good paper and I vote for acceptance initially.", "strength_and_weaknesses": "S:\nThe paper proposes to model light into direct and indirect lighting which takes local lighting such as inter-reflection into consideration.\n\nInstead of only modelling surface BRDF, the proposed method also takes sub-surface into consideration which also enables it to generate more realistic results.\n\nThe proposed network is lightweight and achieves much better results compared with existing methods\n\nW:\nI have no major objection to the paper\n\nMinor:\nPage 4, Figure 3: (a) The Light Sampling Field network applies  (b) The Light Sampling Field network applies\n", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "4: All of the claims and statements are well-supported and correct.", "technical_novelty_and_significance": "4: The contributions are significant, and do not exist in prior works.", "empirical_novelty_and_significance": "Not applicable"}, "extra_text": {}}, {"review_id": "PqNAtucHTm", "reviewer": "Reviewer_M78E", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "This paper proposes a physically-based neural shader that has the potential to model several challenging light transport effects realistically and efficiently. Authors designed several new modules to solve this challenging problem, including a neural lighting representation that separates direct and indirect illumination, a neural material model that handles surface reflection and subsurface scattering. Combined with neural volume rendering, authors show through experiments that their new neural shader can render transculent materials realistically, such as human skin. ", "summary_of_the_review": "This paper solves a challenging and important problem: how to design general neural renderer that can efficiently render complex materials, i.e. translucent materials realistically. The paper showed some interesting results on human face rendering and can significantly accelerate industrial standard software. However, I believe the current version may have many issues to be addressed. Many design choices in method section may be confusing and not reasonable. Experiment part may need to be re-designed to emphasize the major advantages of the proposed method. Therefore, I currently vote for rejecting this paper. Authors can consider solving my questions in the weaknesses section in the rebuttal. I will be happy to change my rating if authors can convince me. Thanks a lot!", "strength_and_weaknesses": "Strength of the paper:\n1. The paper solves an important and challenging problem: building a neural renderer that can render complex translucent materials efficiently under environment map illumination. Some design choices may inspire future research, such as explicitly separating surface reflection and scattering. \n2. Experiments show that the proposed neural shader can render high-quality human face images with much less time compared to an industrial standard path tracer. Authors also show that the proposed method may have the potential to be extended to handle general objects. \n\nWeaknesses of the paper:\nI list my major questions towards the paper below, which I hope authors can address in the rebuttal. \n\n1. While I am very excited after reading the abstract and introduction, I find the method section quite confusing. I may suggest some necessary details are missing and some design choices may not be reasonable. \n\n    a. Page 4 -- important light sampling: It is quite standard in computer graphics to do important sampling for the environment map in an unbiased way, why not just follow the standard method. In addition, the proposed sampling strategy seems to be problematic: why ignoring the textureless regions? Suppose the environment map is a cloudy sky, does that mean we won't have no samples for the sky region at all? In addition, why thresholding the intensity instead of important sampling. Won't that cause the method to miss important ambient lighting? \n\n    b. Figure 3 -- material sampling field network: It is unexpected to encode ray direction omega when predicting material parameters. Doesn't that mean the material is angularly varying? It doesn't seem to be physically correct. \n\n    c. Page 4, Figure 3 -- light sampling field: Similarly, if the output of the light sampling field is spherical harmonics, I assume it should be a radiance field covering lighting emitted in every direction. Why do we encode the ray direction as an input to the network? The SH should only be decided by the 3D location. In addition, how many orders of SH coefficients are used here? \n\n    d. Page 4 -- indirect illumination: The term indirection illumination is not accurately used here. Indirect illumination includes both interreflection as well as light coming inside skin. Here authors clearly only model light coming through skin but not interreflection. This should be made clear in the paper. \n\n    f. Page 5 -- dynamic objects: It is not accurate to claim that the method can handle dynamic object. Here the precomputed light lobe can only partially handle \"moving\" objects. If the object has some non-rigid transformations, the precomputed light lobe won't be able to handle the changing shadows and interreflection, etc. This should be explained clearly in the paper. \n\n    g. Page 6 -- volume casting: Why proposing new volume casting scheme instead of following prior works, such as NeuS? In addition, the current volume casting method can be problematic as it uses distance between x and the intersection point to define density function, which means the same point on different rays may have very different behavior. I am not sure if that's desirable. \n\n    h. Page 6, Eq. (4) -- material scattering: This part may lack important details: how to get e, \\rou_x^s and \\rou_x^ss for real data?\n\n2. The focus of the experiments might be wrong. If I understand the method correctly, here authors use ground truth lighting and some material parameters to train the neural shader. Therefore, the proposed method should not be compared with prior neural reconstruction methods such as Neural-PIL and NeLF as the inputs are very different: they do not have GT material parameters and lighting as inputs. Instead, it should be compared with standard rendering pipeline to show that the proposed method can render realistic images with much less time consumption. I believe experiment in supplementary C.1 is the most important one and should be moved into the main paper with detailed discussion. \n\n3. Further questions with respect to the experiments:\n\n    a. Page 6 -- high-frequency surface normal: how to compute the high-frequency normal maps for training. \n \n    b. Page 6 -- OLAT: the method is designed for using environment map lighting as input, how to handle OLAT iinput? \n\n    c. Page 7 -- Light sampling field: this part is very confusing. The output of indirect light field is also SH. Why will the pre-calculated SH lead to worse results. In addition, I am curious if the implementation is correct in FIgure 5 (b), because I believe SH may result in missing specular highlight but should not cause global shift of the color. \n\n    d. Page 7 -- Lighting model evaluation: This experiment may not be very meaningful as prior method PhySG and Neural-PIL design their method to jointly reconstruct environment while the proposed method has GT environment map as input. It is expected that sampling GT environment map should lead to more accurate lighting. \n\nIn the following, I also list my minor questions, which will not significantly influence my final ratings towards the paper. Authors can choose whether to address or not in the rebuttal. \n\n1. Related work: when discussing portrait and face relighting, the series of works on Codec Avatar are missing, such as Ma et al. Pixel Codec Avatar and Bi et al. Deep Relightable Appearance Models for Animatable Faces. These works should be cited and discussed in related works. ", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "2: Several of the paper’s claims are incorrect or not well-supported.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}, {"review_id": "8ByAO2Dt-p", "reviewer": "Reviewer_NfuH", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "This paper proposed a novel human-face neural rendering representation for physically-based rendering. The authors first give a volumetric lighting representation that accurately encodes the direct and indirect illumination positionally and dynamically gives an environment map. In addition, they also proposed a BRDF measurement representation that supports the PBR process by modeling specular, diffuse, and subsurface scattering for the human face. Finally, based on the above representations, the authors proposed a novel and lightweight neural PBR face shader that takes facial skin assets and environment maps (HDRIs) as input and efficiently renders photo-realistic, high-fidelity, and accurate images. The experiments show that the proposed PBR face shader significantly outperforms the state-of-the-art neural face rendering approaches.", "summary_of_the_review": "Based on the novelty and reliable experiments, I suggest this paper be weakly accepted.", "strength_and_weaknesses": "Strengths\n+ There are some innovations in the method\n1. It is an incremental innovation to represent previous discrete precomputed lighting points as a continuous MLP because many physical attributes have been modeled using MLPs since the publication of NeRF. (Eqn. 4).\n2. It is novel to model the face's oil layer, epidermis, and dermis using a composited BRDF.\n+ Relatively reliable experiments\n1. Thorough experiments and evaluations on different modalities. The effects of different sampling strategies and rendering models are verified in 4.2 and 4.3, respectively.\n2. As shown in Fig.10 and Tab.1, the method in this paper outperforms SOTAs in both qualitative and quantitative indicators, especially the rendering of illumination and shadows on the skin surface.\n3. A large number of experiments are also given in the supplementary material to demonstrate the robustness of the proposed method under different parameters, such as illumination, resolution, etc.\n+ Well written\n\nWeaknesses\n- Some parameters and method details are not clear\n1. What is the ‘lighting embedding $z^l$’, and how to obtain it? It is suggested to give more details of the 'lighting embeddings $z^l$' in the first paragraph of page 5.\n2. How to get the local SH representation in the paragraph of ‘ Light Sampling Field for Indirect Illumination.’? The authors should give an equation or a more detailed explanation.\n3. The introduction of the proposed network structure, such as Material and Light Sampling Field Network, should be more detailed.\n- What is the purpose of the ‘High-frequency Surface Normal ’experiment in the first paragraph of section 4.2? It is difficult to connect this experiment with the paper's key contributions.\n- Eyes are indispensable components of the human face. However, none of the experiments provides results of them. Maybe the method may fail in these areas. It is also suggested to provide results or analysis here.\n", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "4: All of the claims and statements are well-supported and correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}]

Author rebuttal:

Decision: Accept: poster