Learning to Decouple the Lights for 3D Face Texture Modeling

Authors: tianxin  huang
zhenyu  zhang
ying  tai
gim hee lee
Venue: NeurIPS_cc_2024_Conference
Published: 2024-05-07 16:32:16

Abstract: Existing research has made impressive strides in reconstructing human facial shapes and textures from images with well-illuminated faces and minimal external occlusions. 
Nevertheless, it remains challenging to recover accurate facial textures from scenarios with complicated illumination affected by external occlusions, \eg a face that is partially obscured by items such as a hat. 
Existing works based on the assumption of single and uniform illumination cannot correctly process these data.
In this work, we introduce a novel approach to model 3D facial textures under such unnatural illumination. Instead of assuming single illumination, our framework learns to imitate the unnatural illumination as a composition of multiple separate light conditions combined with learned neural representations, named Light Decoupling.
According to experiments on both single images and video sequences, we demonstrate the effectiveness of our approach in modeling facial textures under challenging illumination affected by occlusions.

Main paper: 
 introduction: 
 Recently, 3D face reconstruction has made significant progress [4, 12, 13, 2, 30, 22] with the rapid development of digital human and meta-universe technologies. These techniques have become increasingly proficient in recovering details of face shapes and textures from single images or video sequences. Their performances have been particularly commendable on well-illuminated faces.Despite the advancements, the real world presents more complex scenarios. As shown in the blue and red rectangled regions of input images in Fig. 1 , self occlusions from facial parts such as the nose, or external occlusions such as hats or hairs introduce illumination changes and produce shadows in certain regions. Although recent works [16, 2, 21, 23, 22] replace the linear models [6, 4] with non-linear GAN-inversion-based textured models to greatly enhance the quality of textures and complete occluded facial areas, they still rely on the assumption that the environment illumination is single and uniform. As the illumination affected by self and external occlusions is unnatural, it may deviate drastically from the single and uniform illumination assumption, which would bring unexpected influence to the textures modelled with existing methods.As shown in Fig. 1-a , methods [16, 10, 2] based on the diffuse-only texture and mesh render [10] bake both shadows caused by self occlusion of the nose and external occlusion of the hat onto the texture, where the re-rendered output also appears unrealistic and lacks authenticity. Recent works proposed by Dib et al. [12, 13, 11] combine local reflectance texture model incorporating diffuse, specular, and roughness albedos with ray-tracing render [26] to implicitly model influence of the self occlusion. As shown in Fig. 1 -b, these methods can create more realistic rendered output and avoid the shadows of self occlusion on the recovered texture in the blue rectangle, while the influence from external occlusion in the red rectangle remains unresolved. [12] , while optimizing with ray-tracing render. (c) Our method learns neural representations to decouple the original illumination into multiple light conditions, where the influence from external occlusions can be modeled as one of the conditions. White and black regions in the masks denote 1 and 0, respectively.To mitigate the above-mentioned issues, we propose a face texture modeling framework for faces under complicated and unnatural illumination affected by external occlusions. Given the limited physical information about external occlusions presented in a single facial image or video sequence, accurately modeling how occlusion impacts illumination becomes nearly unattainable. Regarding this challenge, we propose to use multiple separate light conditions to imitate the local illumination of different facial areas after being affected by external occlusions. As shown in Fig. 1-c , our method can model the influence from external occlusion as one of the light conditions and eliminate its effect on the recovered texture as highlighted in the red rectangle, where the rendered output is also more realistic and closer to the input image especially on the forehead and eyes. Specifically, we use a spatial-temporal neural representation to predict masks for different light conditions, which are used to combine rendered results under multiple light conditions into final output. The number of light masks is adpatively adjusted during optimization. As external occlusions may directly cover parts of the face, we also introduce another neural representation to provide a continuous prediction for the available face region. Furthermore, our approach incorporates realistic constraints by introducing priors from the statistical model and pre-trained face perceptual models to ensure our extracted textures construct lifelike human faces.Our contribution in this work can be summarized as:• We present a new face texture modeling framework by learning to decouple the environmental illumination into multiple light conditions with neural representations; • We propose realistic constraints to help improve the authenticity of texture by introducing human face priors; • Extensive experiments on images and videos confirm that our method delivers more realistic renderings and improved facial textures compared to existing approaches under challenging illumination affected by occlusions.2 Related Works 
3d morphable model: 
 The 3D Morphable Model (3DMM) [4, 15, 32, 6] is a widely employed linear statistical framework used for modeling the geometry and texture of faces. It is constructed based on aligned face images using Principal Component Analysis (PCA). Within the 3DMM framework, the coordinates and colors of facial mesh vertices are computed through a linear combination of numerical parameters. These parameters can be estimated to enable face reconstruction through optimization-based fitting [4, 6] or learning-based approaches [10, 30, 43, 42] , with the goal of minimizing the disparities between the rendered and original face images. Traditional 3DMMs, although effective, are limited by their linear PCA basis. Recent advancements [16, 21, 23, 44] have introduced non-linear basis by incorporating pre-trained mesh or texture decoder networks. In these methods, latent codes for these decoders are optimized to align with the original face images. This approach significantly enhances the robustness of the model when faced with issues such as occlusions and missing textures. However, such methods demand access to large, well-preprocessed datasets for effective pre-training.Texture and illumination modeling plays a pivotal role in 3D face reconstruction, directly impacting the color rendering in generated images. Accurate modeling for 3D face textures and environmental illumination is vital for subsequent applications such as face relighting [36, 18, 31] or animation [38, 30] . To achieve high-resolution rendering, recent approaches [9, 21, 16, 3] usually opt for UV mapping to model facial texture, as opposed to the earlier methods that focused on vertex colors [32] .The UV map can either be initialized with a PCA basis [15] or estimated through the use of a pre-trained non-linear texture decoder [22, 2, 16] . There are two commonly used models to estimate human skin reflectance for the texture modeling framework: the Lambertian model [1] and the Blinn-Phong model [5] . The Lambertian model is more computationally efficient, while the Blinn-Phong model produces more realistic rendering results by accounting for specular attributes in textures.As capturing environmental illumination directly can be challenging, most existing methods assume it as primarily uniform Spherical Harmonics [33, 34] . This approach, however, tends to "bake" shadows cast by facial characters or external occlusions into the textures. Although method [45] introduce implicit texture modeling to achieve better face rendering performances, it requires multi-view images under uniform illumination for optimization, which is not appropriate for face images affected by occlusions. 2D shadow removal methods [41, 17, 28] try to eliminate the influence of shadows directly with networks. These methods have higher inference efficiency, while the size of their training set may greatly affect their performances. Notably, Dib et al. [12, 13, 11] have introduced techniques that implicitly model self-shadows through ray-tracing, demonstrating exceptional performance on faces with severe self occlusions. Despite these advancements, existing methods usually encounter challenges in handling shadows caused by external occlusions such as hats or hair, tending to take the shadows as part of textures.To address these issues, we propose a novel framework to recover clear textures under challenging illumination affected by external occlusions. Our approach achieves this by decoupling the unnatural affected illumination into multiple light conditions using learned neural representations.3 Our MethodologyQuestion: Does the paper discuss the limitations of the work performed by the authors?Answer: [Yes]Justification: We have discussed about that in Sec. A.1.Guidelines:• The answer NA means that the paper has no limitation while the answer No means that the paper has limitations, but those are not discussed in the paper. • The authors are encouraged to create a separate "Limitations" section in their paper.• The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (e.g., independence assumptions, noiseless settings, model well-specification, asymptotic approximations only holding locally). The authors should reflect on how these assumptions might be violated in practice and what the implications would be. • The authors should reflect on the scope of the claims made, e.g., if the approach was only tested on a few datasets or with a few runs. In general, empirical results often depend on implicit assumptions, which should be articulated. • The authors should reflect on the factors that influence the performance of the approach.For example, a facial recognition algorithm may perform poorly when image resolution is low or images are taken in low lighting. Or a speech-to-text system might not be used reliably to provide closed captions for online lectures because it fails to handle technical jargon. • The authors should discuss the computational efficiency of the proposed algorithms and how they scale with dataset size. • If applicable, the authors should discuss possible limitations of their approach to address problems of privacy and fairness. • While the authors might fear that complete honesty about limitations might be used by reviewers as grounds for rejection, a worse outcome might be that reviewers discover limitations that aren't acknowledged in the paper. The authors should use their best judgment and recognize that individual actions in favor of transparency play an important role in developing norms that preserve the integrity of the community. Reviewers will be specifically instructed to not penalize honesty concerning limitations. 
problem definition: 
 This work focus on the texture modeling problem of human faces under challenging environment illumination affected by external occlusions. Given an input image or video sequences I in taken from affected illuminations, our task is to recover clear and accurate texture T from I in and ensure that T can synthesize results I out close to I in .As illustrated in Fig. 2 , instead of making the assumption that the face is exposed to a single and uniform illumination, we propose to model n possible separate light conditions γ 1 ∼ γ n in the environment, where the masks for possible regions M N are predicted with a neural representation f (•). Effective masks M L and rendered faces I Rs are selected from original M N and rendered I Rn during optimization with the Adaptive Condition Estimation (ACE) strategy. The final rendered face under multiple light conditions is I R = I Rs ⊙ M L , which is merged with the input images to construct the outputI out = I R ⊙ M o + I in ⊙ (1 -M o ).The face shape is modeled with statistical coefficients α, β, and p. Following [12] , they are mainly optimized with landmark loss:EQUATIONwhere q in is 2D key points detected from I in . |q in | is the number of points in q in . q out denotes projected 3D key points on 2D plane. The color-related variables such as textures and lights are mainly optimized with photometric loss:EQUATIONwhere |I in | is defined as the number of pixels of input image I in . More details are presented below.Light Condition Initialization. Following [12, 13] using B = 9 bands Spherical Harmonics (SH) and ray-tracing rendering to model the illumination under self occlusions, we use n separate SH to model n possible light conditions. The coefficients are then simply initialized asγ i = 2 • i n I -1 for γ i ∈ γ = {γ 1 , .., γ n },where I is a all-one matrix with the same shape as the SH coefficient. Please note that we use each SH to imitate the local illumination after being affected by the external occlusion, instead of the global natural illumination. This means that we do not need to consider physical influence of occlusion in each SH. Instead, we optimize each SH independently to directly imitate the illumination in different face regions. Neural Representations for Face Segment. To decouple the illumination into multiple light conditions, we design a pair of spatial-temporal continuous neural representations to segment the face into regions for different light conditions. As illustrated in Fig. 2 , spatial positions x, y and temporal position t of pixels are normalized and embedded into a coordinates system. t is decided by the number of frames in the input image/video sequence. For the single image reconstruction, t is set as a constant 0, where it would be i/k for the i th frame of a k frames video sequences. For the convenience of learning, (x, y) are normalized into [-1.0, 1.0] and t is normalized into [0.0, 1.0].A Multi-Layer Perceptron (MLP) f (•) is then introduced to predict the probability of assignment to each light condition. Given the render mask M R in the ray-tracing-assisted rendering, the light masks can be obtained as M N = M R ⊙ f (x, y, t). Effective masks M L are selected from M N with Adaptive Condition Estimation (ACE) to provide the final segment for different light conditions.Similarly, we use another MLP g(•) to predict the probability that each pixel belongs to the face regions to avoid the influence from direct occlusion such as hat or hair. The Face mask is thus given by: M o = M R ⊙ g(x, y, t). A pre-trained semantic segmentation network [27] is introduced for distillation to g(•), while the probability for labels of all face components such as eyes, mouth, or nose are added together to construct a classifier h(•) to predict the association of a pixel at x, y, t to the face. The distillation loss L seg can be simply defined as:EQUATIONNote that g(•) is co-optimized together with the L pho and L seg to further distinguish some occlusions hard to be fitted with the 3DMM statistical model.As the complexity of illumination is mutative in different surroundings, we design a strategy to estimate the number of light conditions existing in this environment during optimization. Specifically, light masks with larger area than a pre-defined threshold ϵ in M N are preserved in M L , while smaller ones are dropped with corresponding light conditions and not further optimized in later iterations. Given the number of light masks M N same as light condition n, thenM N = {M i N } n i=1 , M L = {M i L } n L i=1 = {M i N | 1 |Iin| m∈M i N m > ϵ, i <= n}. n L <=n is the number of preserved light conditions. m denotes the pixel in M i N . We introduce an regularization L area to to encourage the area concentration on fewer masks:EQUATIONwhere |I in | is the number of pixels in the mask.To ensure each light condition is mainly contained in one mask of M L , we introduce a binary regularization L bin to ensure that the predicted probability for each light condition tends to 0 or 1:EQUATIONwhere m ∈ M i L denotes each pixel value in i th light mask M L . n L is the number of masks in M L . Note that L area is different from L bin as it pushes the mask values M i N far from the mean of all masks, while L bin pushes mask values away from mean of different positions in the same mask M i L . ACE is executed at a specific iteration iter 0 to select M L from M N . We use L area to encourage area concentration before iter 0 , while using L bin to push the mask to 0 or 1 after iter 0 .To ensure the reconstructed texture is reasonable and lifelike, we propose global prior constraint L GP , local prior constraint L LP , and human prior constraint L HP by introducing both priors from 3DMM statistical model and the pre-trained perceptual model. Global Prior Constraint. The global prior constraint is used to ensure the consistency of overall hue between optimized texture T and initialized texture T 0 from the 3DMM statistical model calculated with δ. As the colors mainly come from diffuse albedo in T , we estimate the overall hue of T 0 with a K-means algorithm on its diffuse albedo T 0 D and get a 4 × 4 color matrices C. Given the diffuse albedo of Texture T as T D , this term is given by:EQUATIONLocal Prior Constraint. The local prior constraint is used to ensure the local smoothness of optimized texture T by constraining its local variation to be similar to T 0 . Let us define the diffuse, specular, and roughness albedos of texture T as T D , T S , T R , respectively. We set the local variations L 2 = ω 0 L pho + ω 1 L lan + ω 2 L seg + ω 3 L area 12:else if n = iter 0 then L 2 = ω 0 L pho + ω 1 L lan + ω 2 L seg + ω 4 L bin 17: end if 18:Let θ f , θ g be the parameters of f (•) and g(•). L 3 = ω 0 L pho + ω 5 L GP + ω 6 L LP + ω 7 L HP 26: Update coefficients with L 3 : ∇ T,γ,θ f L 3 . 27: end for of T D , T S , T R as N D , N S , N R ,EQUATIONHuman Prior Constraint. Human prior constraint is introduced to enhance the optimization of texture T in Stage 3 with a face recognition network FaceNet [35] pre-trained on large dataset such as VGGFace2 [7] or Casia-Webface [39] . The recognition model pre-trained on VGGFace2 and Casia-webface tends to classify the input images to 8,631 and 10,575 identities with different genders, ethnicity, etc. In this work, we propose a human prior constraint by maximizing the probability that the rendered face is recognized on one of the identities by FaceNet. Defining the FaceNet as f r (•), the human prior constraint can then be written as:EQUATIONI Rs are rendered under multiple light conditions. L HP constrains the texutures to be more reasonable.The pipeline of the entire training process of our proposed framework is presented in Alg. 1. As illustrated in Fig. 2 , the training of our proposed method consists of 3 stages, which is similar to NextFace [12] . In Stage 1, the expression β, shape α, and pose p coefficients are optimized to construct the basic face shape. In Stage 2, all coefficients, and the neural representations f (•) and g(•) are optimized together to reconstruct the face with statistical model. In Stage 3, the texture T is directly optimized without coefficient δ, while γ and f (•) are optimized with small learning rates. ω 0 ∼ ω 7 are pre-defined weights. We use Adam optimizer [19] for optimization. NextFace* denotes results optimized within regions selected with face parsing [27] . We do not have textures for CPEM [30] or D3DFR [10] as they predict vertex colors instead of uv textures.4 ExperimentsQuestion: For each theoretical result, does the paper provide the full set of assumptions and a complete (and correct) proof?Answer Guidelines:• The answer NA means that the paper does not include experiments.• The experimental setting should be presented in the core of the paper to a level of detail that is necessary to appreciate the results and make sense of them. • The full details can be provided either with the code, in appendix, or as supplemental material. 
dataset and implementation details: 
 We follow NextFace [12] for the ray-tracing rendering and multi-stage organization of optimization pipeline. Detailed hyper-parameter settings can be found in our supplementary. For our experiments, we utilize two datasets: Voxceleb2 [8] and CelebAMask-HQ [29, 24] . VoxCeleb2 [8] is a diverse dataset encompassing numerous videos collected from interviews, movies and videos, where the same person may have multiple separate videos. CelebAMask-HQ [29, 24] is a large scale face image dataset with fine attributes annotation and high resolution, widely used in face editing and generation.Evaluation. We construct a collection of evaluation data with challenging illumination affected by external occlusions for comparative analysis. This collection includes 38 pairs of single images, 24 pairs of videos from Voxceleb2 [8] with 256 × 256 resolution, and 62 single images from CelebAMask-HQ [29] with 512 × 512 resolution. Each pair consists of source images affected by external occlusions and target images without occlusion, both from the same person, where each video is sampled to 8 frames for sequence-based comparisons.To quantitatively assess the quality of recovered textures, face textures extracted from the occluded source images are leveraged to synthesize the unoccluded target images. Specifically, we optimize source and target images separately following Sec. 3. Then, keeping the face shape, pose, and illumination invariant, we replace the target image's texture with the source's and re-render it to the synthetic result. This allows us to measure texture quality by quantifying the differences between the synthesized target images and the original target images. We also introduce differences between original and reconstructed source images within facial regions acquired by face parsing [27] as an assistant metric to see if textures restore source images well.Our quantitative comparison employs three metrics: PSNR, SSIM, and perceptual error LPIPS [40] of AlexNet [20] . State-of-the-art methods D3DFR [10] , CPEM [30] , NextFace [12] , and RGB Fitting methods proposed in FFHQ-UV [2] are introduced for comparison. For D3DFR, we adopt its enhanced PyTorch version. As NextFace [12] does not distinguish the face and external occlusion when optimizing the texture, we introduce NextFace* in subsequent comparison by adding face parsing [27] to select the face region during optimization. I out = M o ⊙ I R + (1 -M o ) ⊙ I in , respectively.Evaluation on Single Images. We conduct an evaluation about the performance of our method in collected single images sourced from the Voxceleb2 dataset [8] . As shown in Fig. 3 , our approach demonstrates a marked improvement in recovering clearer textures from the original images taken under challenging illumination affected by external occlusions. Our method also surprisingly recovers clear textures under strong colorful lights, which may benefit from the realistic constraints to keep textures reasonable. In contrast, existing methods tend to incorporate these occlusions, shadows, or colorful lights directly into the texture, resulting in less satisfactory outcomes.Furthermore, our methods achieves superior results on the synthesis of target images and comparable performances on the reconstruction of source images as shown in Table 1 . It confirms our method consistently recovers both accurate and clear textures. Although NextFace [12] performs a little better on PSNR and SSIM of source images, it performs the worst on synthetic target images as it actually severely over-fits source images. As shown in Fig. 3 , it bakes shadows and occlusions to the textures, which confirms it is not appropriate for faces affected by external occlusions.Evaluation on Video Sequences. We conduct an evaluation on video sequences collected from Voxceleb2 dataset [8] to further substantiate the efficacy of our approach. To adapt our framework to video sequences, we share texture, illumination and shape coefficients across all frames during optimization. As D3DFR [10] and FFHQ-UV [2] do not provide support for sequences, we recurrently apply them to each single image. Quantitative comparison in Table 2 demonstrates that our method constantly performs superior to existing methods on texture modeling from continuous video sequences. Please check our Supplementary for corresponding qualitative results.We extend our comparisons on CelebAMask-HQ [29] . As this dataset lacks multiple images from the same identity, we focus on evaluating the performance based on the recovered textures and the reconstructed results of source images. The outcomes of our assessment are visually depicted in Fig. 4 . We observe that our method continues to perform well in the task of recovering clear textures from challenging input. Additionally, our reconstructed results exhibit a higher degree of realism on the reconstructed results when compared to the outcomes produced by other methods.To validate the performances of our method more sufficiently, we further evaluate our method on the single image dataset proposed by [41] , which includes 100 images affected by manually created external shadows, as well as corresponding ground truths. The quantitative and qualitative comparisons are presented in Table 5 and Fig. 7 . We can observe that our method still outperforms other methods under faces with diverse shadows. [41] . As [41] provides corresponding ground truths of shadow affected images, we directly use such ground truths as the target images.Ablation Study on Losses. In this section, we analyze the impact of the proposed losses: L GP , L LP , and L HP by comparing the rendered results under relative brighter light conditions. As illustrated in Fig. 5 , it becomes evident that the inclusion of the L GP loss plays a significant role in eliminating the abnormal colors on the texture, where L LP loss effectively remove large artifacts obviously different from human faces. L HP can further reduce slight unreasonable defects with priors from the pre-trained recognition model [35] . Quantitative comparisons in Table 3 also confirm that each proposed loss contributes to the final performances.Ablation Study on Neural Representations. To ascertain the significance of the neural representations employed, we perform an analysis where we eliminate both the neural representations: f (•) and g(•) from the light decoupling pipeline. As depicted in Fig. 6 , the neural representation for the decoupling of light conditions f (•) notably contributes in the removal of shadows artifacts from the results. Furthermore, the neural representation g(•) displays its efficacy in minimizing existing irregularities in the synthetic results, which achieves smoother and cleaner results. We also present corresponding quantitative comparisons in Table 4 , which demonstrates that both f (•) and g(•) have obvious influence on the final performances. 
conclusion: 
 In this paper, we present a novel framework dedicated to recover clear facial textures from images taken under challenging illumination affected by external occlusion. Our approach makes use of neural representations to decouple the original illumination into multiple separate light conditions across various facial regions. Then the affected complicated illumination can be modelled with the combination of different light conditions. Furthermore, we introduce well-established human face priors through the realistic constraints to enhance the realism of our results. According to experiments on single images and video sequences, our method consistently surpasses existing techniques to recover clearer and more accurate textures from faces under affected unnatural illumination. 
experiment statistical significance: 
 Question: Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?Answer: [No] Justification: We follow existing related works for the setting of error bar.Guidelines:• The answer NA means that the paper does not include experiments.• The authors should answer "Yes" if the results are accompanied by error bars, confidence intervals, or statistical significance tests, at least for the experiments that support the main claims of the paper. • The factors of variability that the error bars are capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, or overall run with given experimental conditions).• The method for calculating the error bars should be explained (closed form formula, call to a library function, bootstrap, etc.) • The assumptions made should be given (e.g., Normally distributed errors). • It should be clear whether the error bar is the standard deviation or the standard error of the mean. • If the authors answer No, they should explain the special circumstances that require a deviation from the Code of Ethics. • The authors should make sure to preserve anonymity (e.g., if there is a special consideration due to laws or regulations in their jurisdiction). 10. Broader Impacts Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed? Answer: [Yes] Justification: Similar as existing human face reconstruction works, our method, focusing on recovering more realistic face textures under shadows from self and external occlusions, may also lead to the ethical concerns about privacy, consent, and the spread of misinformation.Strengthening the management of the actual facial data collected may alleviate this situation. Guidelines:• The answer NA means that there is no societal impact of the work performed.• If the authors answer NA or No, they should explain why their work has no societal impact or why the paper does not address societal impact.• Examples of negative societal impacts include potential malicious or unintended uses (e.g., disinformation, generating fake profiles, surveillance), fairness considerations (e.g., deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations. • The conference expects that many papers will be foundational research and not tied to particular applications, let alone deployments. However, if there is a direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that an improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point out that a generic algorithm for optimizing neural networks could enable people to train models that generate Deepfakes faster. • The authors should consider possible harms that could arise when the technology is being used as intended and functioning correctly, harms that could arise when the technology is being used as intended but gives incorrect results, and harms following from (intentional or unintentional) misuse of the technology. • If there are negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition to attacks, mechanisms for monitoring misuse, mechanisms to monitor how a system learns from feedback over time, improving the efficiency and accessibility of ML). 
safeguards: 
 Question: Does the paper describe safeguards that have been put in place for responsible release of data or models that have a high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)? Answer: [NA] Justification: The paper does not have such risks as we only use public datasets. Guidelines:• The answer NA means that the paper poses no such risks.• Released models that have a high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere to usage guidelines or restrictions to access the model or implementing safety filters. • Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images. • We recognize that providing effective safeguards is challenging, and many papers do not require this, but we encourage authors to take this into account and make a best faith effort. 12. Licenses for existing assets Question: Are the creators or original owners of assets (e.g., code, data, models), used in the paper, properly credited and are the license and terms of use explicitly mentioned and properly respected? Answer: [Yes] Justification: We have cited them in the references. Guidelines:• The answer NA means that the paper does not use existing assets.• The authors should cite the original paper that produced the code package or dataset.• The authors should state which version of the asset is used and, if possible, include a URL. • The name of the license (e.g., CC-BY 4.0) should be included for each asset.• For scraped data from a particular source (e.g., website), the copyright and terms of service of that source should be provided. • If assets are released, the license, copyright information, and terms of use in the package should be provided. For popular datasets, paperswithcode.com/datasets has curated licenses for some datasets. Their licensing guide can help determine the license of a dataset. 
 
 Appendix: We use AlbedoMM [37] to initialize the face texture and optimize neural representations. Limited by the capacity of AlbedoMM, the initialized face texture may not be accurate enough for subsequent optimization in some occasions. We can explore to resolve this problem by combining our method with non-linear texture modeling methods such as [22, 2] in our future work. Furthermore, the optimization of multiple light conditions makes our method slower than other single illumination methods [12, 2] . It costs 340s for a 256 × 256 image on 2080ti, which is slower than 240s of FFHQ-UV [2] but still affordable.In Table 6 , we present details about the hyper-parameters mentioned in Sec. 3. Although it seems that there are multiple parameters, the setting is robust for both images and video sequences from adopted datasets. We conduct all experiments on a Nvidia 2080ti GPU with a 2.9Ghz i5-9400 CPU.It costs 340s for a 256 × 256 image. Although the former mentioned existing 3D face texture modeling methods cannot process facial shadows from external occlusions directly, there are some 2D shadow removal methods [41, 17, 28] trying to eliminate the shadows directly from the images. Therefore, another alternative simple baseline is to pre-process the image with 2D shadow-removal networks before texture modeling.In this section, we introduce the most recent method [17] with a pre-trained model to pre-process the images before feeding them to compared methods mentioned in Sec. 4.1. The quantitative and qualitative comparison are presented in Table 7 , Table 8 and Fig. 11 , respectively. Pre-processing with 2D shadow removal methods indeed improves the performances of baselines, while our method still outperforms them. From qualitative results, we observe that the shadow The red and black rectangles mark shadow-affected regions and detailed textures, respectively. g(•) will weaken both shadows and details from textures when reducing w 2 to loose its constraint. removal model cannot fully remove the external shadows in cases where the external shadows cover relatively large regions. Although modifying the shadow removal model to be more powerful may further improve performances, it goes beyond the range of this work. We can explore it in the future.Besides the former mentioned shadow removal baselines, learning-based deocclusion methods [25, 14] can remove external occlusions from faces by predicting the occluded regions. Such operations also have the potential to deal with external shadows by directly treating the shadow regions as occlusions. In this section, we conduct a brief comparison with the most recent open sourced deocclusion method [25] . The quanlitative and qualitative results are presented in Table 9 and Fig. 12 , respectively. The metrics are evaluated on the target images mentioned in Sec. 4.1. We can observe that the method [25] lose some facial details such as beards. The reason is that the deocclusion method [25] actually divides occlusion regions by the distances between input images and the reconstructed results from 3D Morphable Model (3DMM), which will also regard details hard to describe with 3DMM as occlusions. Our methods treat the shadows as different illumination instead of occlusions, which can preserve facial details in these regions in subsequent optimization.A.5 Discussion about the number of Sphere Harmonics (SH) bands.In this work, we follow NextFace [1] to use 9-bands SH to model the local illumination, which can actually capture quite fine details. However, SH with more bands would have stronger ability for the modeling of illumination. To verify if the external shadows can be directly modelled with more bands SH, we also provide the quantitative results of our method modelled with only one single global SH in 9, 12, 15, 18 bands in Table 10 , where we remove our light decoupling framework. We observe that increasing the number of bands in a single global SH yields quite limited improvements. A possible reason is that the external occluded shadows on human faces represent drastic illumination changes in relatively small areas, which may not be appropriately modeled as a single global SH during optimization.A.6 Discussion about the number of lighting conditions n.In this work, we initialize n different lighting conditions to imitate the illumination affected by external occlusion at the beginning, where some of them would be removed by ACE in subsequent processing. To explore the influence of different numbers of n, we conduct a ablation study for the number of n used for initialization of lighting conditions in Table 11 . Single images from VoxCeleb2 are used for evaluation. We observe that n = 3 produces sub-optimal results, likely because the number of lighting condition candidates is insufficient to model images with complex illuminations. In contrast, n = 5, 7, 9 yield good and similar results as there are enough initial lighting conditions, and any redundant ones are removed. The result for n = 5 is slightly better. While introducing larger n and further adjusting the hyper-parameters might improve performance, it would also increase the optimization burden.Please note that the rendred faces used to calculate Human Prior Constraint (HP) are calculated with I Rs . As illustrated in Fig. 2 , I Rs includes rendered faces under multiple different lighting conditions. We cannot use perceptual loss here because we do not have **corresponding ground truth images under the multiple decoupled lighting conditions**, where HP does not need such ground truths. Nonetheless, the perceptual loss can be indeed applied between the final rendered result I out and input image I in . We present a comparison between such perceptual loss implementation and the HP in Table 12 . We can see that HP still has better performances, which can confirm it provides more effective constraints for the textures through rendered faces under multiple lighting conditions.A.8 Discussion about g(•).Both f (•) and g(•) have impacts to remove artifacts on the recovered textures. To confirm that they have principled differences, we implement a discussion about the effect of g(•). As presented in the third column of Fig. 9 , g(•) cannot fully avoid the influence of external shadows on the faces. As presented in Sec. 3.3, L seg is introduced to constrain g(•) to ensure that g(•) predicts relatively accurate face regions. L seg is weighted by w 2 in optimization as defined in Alg. 3. Although the shadow effect can decrease when we reduce w 2 to allow g(•) to filter out more face regions, the detailed textures are equally weaken and may be fully removed as shown in the last row of Fig. 9 . It confirms that f (•) is still essential for this framework.A.9 Analysis about L bin and L area in ACE.In ACE mentioned in Sec 3.3, two regularization constraints L area and L bin are introduced. As the blue rectangle input face of Fig. 10 is not affected by any external occlusion, its illumination can be modelled with only one light condition. L area helps remove redundant light conditions. However, only using L area may create decoupled light conditions obviously different from the observation of input image, as illustrated in the red rectangle regions of Fig. 10 . Adding L bin can ensure the predicted masks M L to be nearly binarized, which makes the decoupled light conditions consistent as the human observation of the input image.A.10 Differences between Stage 2 and Stage 3.Illustrated in Fig. 2 , our approach composes of both Stage 2 and Stage 3 to optimize the face texture.In this section, we conduct comparisons to demonstrate differences between textures from Stage 2 and Stage 3. The results are presented in Fig. 13 . We can observe that textures from AlbedoMM [37] We present an ablation study to confirm the effect of g(•) against direct segmented mask from face parsing [27] to predict M o . The circled parts of Fig. 14 show that the parsed masks may be inappropriate due to the limitation of generalizability. We can see in the first row of Fig. 14 that the rendered result I R may have rough edges and artifact colors from the occlusion, while our method can avoid this problem by refining the parsed mask with g(•). Moreover, some parts may be missing on the parsed mask. As shown in the second and third rows of Fig. 14 , artifacts show up in these unconstrained regions. Our method can complete these missing regions for more reasonable results. A.12 Discussion about the failure cases.Except the efficiency problem mentioned in Sec. A.1, our primary limitation is the initialization with AlbedoMM. As shown in Fig. 15 , for faces with many high-frequency details, such as wrinkles, our method may lose these details during reconstruction. Replacing AlbedoMM with more powerful face representations could address this issue. We plan to explore this further in future work. Reconstructed SourceIn this section, we present three representative examples from each sequence in Fig. 17 . It is evident that our method performs the best in generating synthetic results close to the target sequences, exhibiting a high degree of realism. In contrast, other methods still produce less convincing outcomes due to negative effects from external occlusions. More results on images sourced from Voxceleb2 [8] and CelebAMask-HQ [24] are presented in Fig. 18 and Fig. 19 . Our method still performs better. Please refer to the attached video for more results on sequences from Voxceleb2 [8] . Guidelines:• The answer NA means that the abstract and introduction do not include the claims made in the paper. • The abstract and/or introduction should clearly state the claims made, including the contributions made in the paper and important assumptions and limitations. A No or NA answer to this question will not be perceived well by the reviewers. • The claims made should match theoretical and experimental results, and reflect how much the results can be expected to generalize to other settings. • It is fine to include aspirational goals as motivation as long as it is clear that these goals are not attained by the paper. 


Decision: After the rebuttal and extensive reviewer-author discussion, all four reviewers proposed acceptance, thus, the AC recommends acceptance. The AC strongly encourages the authors to include the missing comparisons, citations, and the rest of the rebuttal/discussion material in the final version of the paper.

Reviews: review: paper_summary: paper_summary: The paper presents a method for reconstructing 3D face textures from monocular images in the presence of occlusions, both self-occlusions and occlusions by other scene elements such as hats. The paper identifies a key limitation in existing works -- they all get impacted by the lighting changes introduced by an occluder, as they assume globally consistent lighting. The paper proposes to model the scene using a combination of different lighting for the different regions of the face. The different regions are reconstructed without direct supervision. Additionally, a mask is used to separate out the occluder pixels. Modeling multiple light sources enables the method to model appearance effects due to occluders and thus enables higher quality reconstructions than the state of the art.
strength_weakness: strength_weakness: The main weakness is the lack of discussion and comparisons with existing works, and the lack of stronger baselines. 

The paper does not mention existing research on dealing with occlusions. Here are a couple:
- Robust Model-based Face Reconstruction through Weakly-Supervised Outlier Segmentation [CVPR23]
- Occlusion-aware 3D Morphable Models and an Illumination Prior for Face Image Analysis [IJCV18]
Both these papers jointly perform 3D reconstruction and solve for occluders. They might not use texture optimization and instead perform per-vertex color optimization, but their techniques should be trivially extended for texture optimization. Comparisons to these baselines makes it impossible to evaluate this submission. 

In addition, there are other strong baselines that were not considered. Generative models, such as those introduced by "Face De-occlusion using 3D Morphable Model and Generative Adversarial Network" [ICCV19] can remove occlusions from portrait images. Can the output of those methods directly be used with NextFace for high-quality reconstructions? 

The other weakness relates to the exposition. The technical details of the paper were not easy to read. I did not understand how many number of masks are used, what 't' refers to in L128, how many such 't' are used. How are all the hyperparameters, such as 'n', n_L', m, epsilon, etc., chosen? I did not find a discussion. The paper is not reproducible in its current state.
questions: questions: Please refer to weaknesses.
limitations: limitations: Yes
paper_summary: paper_summary: This work tackles the problem of external shadows in single image face reconstruction. Specifically, when the input image contains foreign shadows, this often affects the quality of the estimated facial texture, as the external shadows often become baked into the texture or leave behind undesirable artifacts in the shadow region. The paper proposes a comprehensive solution to this problem, including a way to decouple the face image into the result of multiple lights as well as an Adaptive Condition Estimation (ACE) strategy to select which lights are present in the image. The paper further proposes multiple human face priors, such as a global prior to ensure the face texture hue is consistent with the initialization from a 3DMM model and a local prior to ensure that the smoothness of the face texture is similar to the initialization. Experiments demonstrate that the method is able to improve rendering performance on in-the-wild face images by rendering both source images with external shadows and target images without external shadows. The major improvements on target images without external shadows shows that the method is able to produce accurate facial textures that are minimally impacted by the presence of these shadows. Qualitative ablations are also provided to further aid in evaluating the method.
strength_weakness: strength_weakness: For the ablation studies, it would be much more convincing to have tables with quantitative results demonstrating the margin of improvement of each component. Simply picking a few favorable qualitative examples is easy and not convincing. 

In almost all examples in this work, the foreign (external) shadows involved are caused by hats. What about other types of foreign shadows caused by tree leaves, pens, paper, hands, etc? How are the resulting face textures in these situations? It would be nice to compare with the baselines on some images with more diverse foreign shadows. If it is difficult to find such images in the wild, you can find a small test set of 100 or so images from the paper "Portrait Shadow Manipulation (SIGGRAPH 2020)" with diverse foreign shadow shapes. 

Some important citations are missing from this work, especially in the face relighting and portrait shadow removal domains. There are several face relighting methods that involve intrinsic decomposition of faces and illumination modeling, some of which involve ray tracing to handle self shadows: 

1. SfSNet : Learning Shape, Reflectance and Illuminance of Faces in the Wild (CVPR 2018)
2. Face Relighting with Geometrically Consistent Shadows (CVPR 2022)
3. Learning Physics-guided Face Relighting under Directional Light (CVPR 2020)

In addition, the shadow removal domain is highly relevant to this work since a simple solution to this problem would be to perform a preprocessing step to remove the foreign (external) shadows from the image first using a portrait shadow removal method before performing face reconstruction. These methods should be cited and discussed in the paper and the authors should verify that their method outperforms this simple baseline: 

1. Portrait Shadow Manipulation (SIGGRAPH 2020)
2. Unsupervised Portrait Shadow Removal via Generative Priors (MM 2021)
3. Blind Removal of Facial Foreign Shadows (BMVC 2022)
questions: questions: Please see the experiments I suggested above, especially with regard to quantitative evaluations for ablations, evaluations on images with more diverse foreign (external) shadows, and comparing with the simple baseline of running a foreign shadow removal method on the images beforehand. I will be carefully reviewing the rebuttal as well as the opinions of the other reviewers to decide if I would like to change my rating. Please also factor in missing citations, as the two areas I mentioned are very relevant to this work.
limitations: limitations: The authors have at least made an attempt to include a limitations section, although more analysis on failure cases would be helpful. For example, does the method fail when the external shadow covers most of the face? There is no section on potential negative societal impact, although there are certainly some potential concerns with face reconstruction methods, as with any face modeling work. The potential to edit the face and produce fake content is always present.
paper_summary: paper_summary: This paper proposes a method to recover face albedo by disentangling the input image into an albedo and shading maps (called "light conditions"). The shading maps are generated by a network which are combined with the albedo texture (generated from AlbedoMM) that is rendered under a set of $n$ lighting conditions represented using spherical harmonics. The networks for predicting the shading maps, spherical harmonics and the texture are optimized through a photometric loss along with regularizations. In order to generate meaningful masks, a binarization constraint is added on the generated masks along with an area constraint. While the final renders are significantly better than prior work, two crucial comparisons are missing i.e against the Haar-wavelet basis (https://graphics.stanford.edu/papers/allfreqmat/) and the Albedo from AlbedoMM
strength_weakness: strength_weakness: 1) The authors have not compared against optimizing in haar-wavelets (https://graphics.stanford.edu/papers/allfreqmat/) which are designed specifically to model sharp illumination dependent effects. Without this comparison, it is hard to asses the improvements the proposed model offers over classical representations. I understand that such an optimization may be compute intensive, but it is necessary.

2) There are no results shown of the initial AlbedoMM texture the texture map is initialized from. AlbedoMM already gives a relatively uniform texture map that is free from lighting artifacts, it is unclear what additional benefits texture optimization yields.
questions: questions: How many bands of SH are optimized? Did you investigate optimizing higher number of bands instead of generating the masks (again, as done in: http://graphics.stanford.edu/papers/allfreq/allfreq.pdf
limitations: limitations: Yes.
paper_summary: paper_summary: The paper presents a new face texture modeling framework by learning to decouple environmental illumination into multiple light conditions with neural representations. The proposed method can reconstruct a cleaner texture compared to previous work that use a single environment map. The proposed method is neat and reasonable. The paper is well-written.
strength_weakness: strength_weakness: * Line 119 says the n lighting coefficients are initialized differently. Can you provide some insights behind these design choices? Is the method sensitive to the initialization of lighting?

* In the proposed Human Prior Constraint, the rendered face is encouraged to be similar to a specific face in the FaceNet. Why not use the common perceptual loss as D3DFR?

* The texture of D3DFR in Figure 3 is missing. I know it reconstructs the BFM texture, which is not a diffuse albedo map. But I think it should also be provided. In addition, the FFHQ-UV's texture is presented in UV space while NextFace and the proposed methods' texture are presented in image space, I suggest presenting them all in UV space or all in image space.
questions: questions: See Weaknesses.
limitations: limitations: The limitations are discussed.

Comments: title: Response for Reviewer SyuP
Response to Reviewer SyuP
comment: Thank you for your comment! We are happy that our response can address your problem. We will add the former mentioned discussions to the revised version.
Thanks, I now understand the distinction between ignoring the outliers in [1] and modeling them in this submission. I will raise my score.
Thank you very much for your response! We are encouraged that our rebuttal can address your concerns. We will release our source codes after we clean up them.
Thanks for your rebuttal, it addressed my main concerns. I will keep my score. Do you plan to release the fitting code? I think it is a great contribution to the community.
Thank you for your response! We apologize for the confusion regarding the qualitative results.
We agree that the work "[1] Robust Model-based Face Reconstruction through Weakly-Supervised Outlier Segmentation [CVPR23]" can avoid shadow effects on the textures. But **please note that it is different from our method and limited by texture details.** [1] predicts a binary mask to remove these shadow regions, while our method reconstructs them with different illuminations. 

The details, e.g. beards, may be difficult to be captured by [1] because it fully relies on a **pure linear parametric 3D Morphable Model (3DMM)** to model the face textures and distinguish the occlusion regions. As claimed in Sec.3.1 of [1], it models face textures with 3DMM parameters predicted by networks.
Although 3DMM does not have explicit shadows on its textures as shown in the rebuttal PDF, it is hard to construct details, e.g. mentioned beards, as claimed in “Parametric Face Models”, Sec. 2 of [2]. Therefore, even lowering the weight of the regularizer in its Eq.8 may not be able to capture the details due to the limitation of the 3DMM model itself. It is also mentioned in the Limitation, Sec. 4.5 of [1] that a face model capable of depicting facial details, makeup, and beard is required. While incorporating such a model could potentially address this issue, doing so requires substantial effort and falls outside the scope of our paper, which we leave for future work.

[2] Self-supervised Multi-level Face Model Learning for Monocular Reconstruction at over 250 Hz

In our method, although we also initialize face textures using a parametric AlbedoMM in Stage 2, our direct texture optimization in Stage 3 allows us to add fine details to the textures, as shown in Figure 10 of the appendix in our paper. This can be achieved because we model the shadows with illumination to avoid their influence on textures. Directly optimizing the textures in [1] may not achieve this. As explained in Sec. 3.2 of [1], it treats regions which cannot be well described with 3DMM under a global illumination, e.g. the beard regions, as occlusions and learns to predict a mask to remove them. Since these regions are removed, they cannot be further used to refine the 3DMM textures through optimization.

Since we are not allowed to include more qualitative results in this comment, we will provide additional visualizations of the predicted masks from [1] in the revised version to support our claim. Furthermore, the quantitative comparisons in Q1 also confirm the effectiveness of our method, which will be discussed in greater detail in the revised version.

Thank you once again for your feedback. Please feel free to reach out if you have any further questions.
Thanks for the response, and for clarifying that some papers I mentioned were focussed only on occlusions rather than shadows. However, Robust Model-based Face Reconstruction through Weakly-Supervised Outlier Segmentation [CVPR23] deals with outliers generally, including shadows? The qualitative results in the pdf seem to show that their results are indeed robust to shadows. Their results do not capture details such as beards very well, but that is an orthogonal issue to robustness to shadows? I suspect these details can be captured by lowering the weight of the regularizer in Eq.8 of their paper (https://arxiv.org/pdf/2106.09614). Can you comment on this?
Thank you once again for your valuable time and constructive suggestions! We are inspired by your recognition of our method and our responses.  We will add all the mentioned comparisons and discussions to our final version.
Thanks for your response! I've raised my score to Weak Accept. Please remember to include these new comparisons in the final version to deliver a more convincing story as well as citing the relighting and shadow removal methods I mentioned previously to have a more comprehensive discussion of related methods.
Thank you for your feedback. In the previous rebuttal, we presented results from 15 representative pairs due to time and hardware constraints. We have now extended our evaluation in Q3 to all 100 data pairs below.

|  |CPEM|D3DFR|NextFace| NextFace* | FFHQ-UV | Ours
|--|--|--|--|--|--|--|
|PSNR $\uparrow$ | 23.31| 25.45 |22.16| 22.24 | 23.89 | **28.78**
|SSIM $\uparrow$ |0.83 |0.86 | 0.84  | 0.84 | 0.88 | **0.91**
|LPIPS $\downarrow$ | 9.93 |9.18 | 9.58 | 9.64 | 7.96 | **7.33**

We can see that our method consistently outperforms the others, further validating its effectiveness.
Thank you for your rebuttal! Most of my concerns are addressed here. However, for evaluating in Q3, can we use all 100 images in that dataset? This would be more comprehensive and convincing to me than only evaluating on 15 of them.