Learning to Decouple the Lights for 3D Face Texture Modeling
Authors: Tianxin Huang
Zhenyu Zhang
Ying Tai
Gim Hee Lee
Venue: NeurIPS 2024 poster (2024)
Keywords: Face Texture
Light Decoupling
Neural Representation
Primary area: machine_vision
TL;DR: Learning to recover face textures under illumination affected by external occlusions

Abstract: Existing research has made impressive strides in reconstructing human facial shapes and textures from images with well-illuminated faces and minimal external occlusions. 
Nevertheless, it remains challenging to recover accurate facial textures from scenarios with complicated illumination affected by external occlusions, \eg a face that is partially obscured by items such as a hat. 
Existing works based on the assumption of single and uniform illumination cannot correctly process these data.
In this work, we introduce a novel approach to model 3D facial textures under such unnatural illumination. Instead of assuming single illumination, our framework learns to imitate the unnatural illumination as a composition of multiple separate light conditions combined with learned neural representations, named Light Decoupling.
According to experiments on both single images and video sequences, we demonstrate the effectiveness of our approach in modeling facial textures under challenging illumination affected by occlusions.

Learning to Decouple the Lights for 3D Face Texture Modeling
Tianxin Huang'
Ying Tai School of Computing, National University of Singapore
Zhenyu Zhang?
Gim Hee Lee'
2 Nanjing University
huangtxenus edu. gimhee lee@nus. edu.sg
Abstract
Existing research has made impressive strides in reconstructing human facial shapes and textures from images with well-lllmminted faces and minimal external occlusions. Nevertheless, it remains challenging to recover accurate facial textures from scenarios with complicated illumination affected by external occlusions, e.g. face that partially obseured by items such a hat. Existing works based on the ussumption of single and uniform illumination cannot correctly process these data. In this work, we introduce novel approach model 3D facial textures under such unnatural illumination. Instead of assuming single illumination, our framework learns to imitate the unnatural illumination separate light conditions combined with learned neural representations, named Light Decoupling. According to experiments on both single images and video sequences, we demonstrate the effectiveness of our approach in modeling facial textures under challenging illumination affected by occlusions. Our codes would be open sourced at
composition of multiple
Introduction
30 22 with the rapid development of digital human and meta-universe technologies. These techniques have become increasingly proficient in recovering details of face shapes and textures from single images or video sequences. Their performances have been particularly commendable on well-illuminated faces.
Recently, 3D face reconstruction has made significant progress 44, 13
Despite the advancements, the real world presents more complex scenarios. shown in the blue and red rectangled regions of input images in Fig. I self occlusions from facial parts such as the nose, or external occlusions such as hats or hairs introduce illumination changes and produce shadows in certain regions. Although recent works [6 2 21 non-linear GAN-inversion-baan based textured models to greatly enhance the quality of textures and complete occluded facial areas, they still rely the assumption that the environment illumination single and uniform. As the illumination affected by self and external occlusions unnatural, may deviate drastically from the single and uniform illumination assumption, which would bring unexpected influence to the textures modelled with existing methods.
23 22
replace the linear models 6 4 with
As shown in F methods [6 10 based on the diffuse-only texture and mesh render [TO] bake both shadows caused by self occlusion of the nose and external occlusion of the hat onto the texture, where the re-rendered output also appears unrealistic and lacks authenticity. Recent works proposed by Dib et al. [[2] 13 [T]] combine local reflectance texture model incorporating diffuse, specular, and roughness albedos with ray-tracing render (26 implicitly model influence of the self occlusion. As shown in Fig. b these methods can create more realistic rendered output and avoid the shadows of self occlusion on the recovered texture in the blue rectangle, while the influence from external occlusion in the red rectangle remains unresolved.
38th Conference on Neural Information Processing Systems (NcurIPS 2024).

---

Input
Texture
Illumination
Output
Mesh render
(a)
Texture
Output
Ray tracing render
Optinize
(b)
Brighter
Texture
Element wise muttiply
render
Output
Optimize
Neural Representation
Figure 1: Blue and red rectangles mark regions affected by self and external occlusions, respectively, (a) Texture modeling with diffuse-only texture map. (b) Texture modeling based on diffuse, specular, and roughness albedos from local reflectance model [I2]. while optimizing with ray ray-tracing render. (c) Our method learns neural representations to decouple the original illumination into multiple light conditions, where the influence from external occlusions be modeled White and black regions the masks denote and O, respectively.
of the conditions.
To mitigate the above-mentioned issues, we propose a face texture modeling framework for faces under complicated and unnatural illumination affected by external occlusions. Given the limited physical information about external occlusions presented accurately modeling how occlusion impacts illumination becomes nearly unattainable. Regarding this challenge, we propose to use multiple separate light conditions to imitate the local illumination of differrnt facial areas after being affected by external occlusions. As shown in Fig. [c. our method can model the influence from external occlusion as one of the light conditions and eliminate its effect the recovered texture highlighted in the red rectangle, where the rendered output is also more realistic and closer to the input image especially on the forehead and eyes. Specifically, we use a spatial-temporal neural representation to predict masks for different light conditions, which used to combine rendered results under multiple light conditions into final output. The number of light masks is adpatively adjusted during optimization. As external occlusions may directly cover parts of the face, we also introduce another neural representation to provide a continuous prediction for the available face region. Furthermore, our approach incorporates realistic constraints by introducing priors from the statistical model and pre-trained face perceptual models to ensure our extracted textures construct lifelike human faces.
single facial image or video sequence,
Our contribution in this work can be summarized as:
We present face texture modeling framework by learning to decouple the environmen- tal illumination into multiple light conditions with neural representations;
We propose realistic constraints help improve the authenticity of texture by introducing human face priors;
Extensive experiments on images and videos confirm that our method delivers more realistic renderings and improved facial textures compared to existing approaches under challenging illumination affected by occlusions.
Related Works
2.1
3D Morphable Model
The 3D Morphable Model (3DMM) 4 15
is a widely employed linear statistical framework used for modeling the geometry and texture of faces. It is constructed based aligned face images

---

using Principal Component Analysis (PCA). Within the 3DMM framework, the coordinates and colors of facial mesh vertices are computed through linear combination of numerical parameters. These parameters can be estimated to enable face reconstruction through optimization- based fitring 44 G or
43 42], with the goal of minimizing the disparities between the rendered and original face images. Traditional 3DMMs, although effective, are limited by their linear PCA basis. Recent advancements [16 pre-trained mesh or texture decoder networks. In these methods, latent codes for these decoders are optimized to align with the original face images. This approach significantly enhances the robustness of the model when faced with issues such occlusions and missing textures. However, such methods demand access to large, well- preeproepeee datasets for effective pre-training.
learning-basee approaches [00 30
21
23 44] have introduced non-linear basis by incorporating
2.2 Face Texture and Illumination Modeling
Texture and illumination modeling plays a pivotal role in 3D face reconstruction, directly impacting the color rendering in generated images. Accurate modeling for 3D face textures and environmental
illumination is vital for subsequent applications such as face relighting [36 18 31
or animation [38
30. To achieve high-resolution rendering, recent approaches mapping to model facial texture, as opposed to the earlier methods that focused on vertex colors (32). The UV map can either be initialized with a PCA basis [15] or estimated through the use of a pre-trained non-linear texture decoder [2 2 [6]. There are two commonly used models to estimate human skin reflectance for the texture modeling framework: the Lambertian model [[] and the Blinn- Phong model [ The Lambertian model is more computationally efficient, while the Blinn-Phong model produces more realistic rendering results by accounting for specular attributes textures.
21 16
3 usually opt for UV
As capturing environmental illumination directly can be challenging, most existing methods assume it as primarily uniform Spherical Harmonics [33 34] This approach, however, tends to "bake" shadows cast by facial characters or external occlusions into the textures, Although method [45] introduce implicit texture modeling to achieve better face rendering performances, requires multi-view images under uniform illumination for optimization, which is not appropriate for face images affected by occlusions. 2D shadow removal methods [4 17 try to eliminate the influence of shadows directly with networks. These methods have higher inference efficiency, while the size of their training set may greatly affect their performances. Notably, Dib el al. [12] 12,11 1I] have introduced techniques that implicitly model self-shadows through ray-tracing, demonstrating exceptional performance on faces with severe self occlusions. Despite these advancements, existing methods usually encounter challenges in handling shadows caused by external occlusions such hats hair, tending take the shadows as part of textures.
To address these issues, we propose novel framework to recover clear textures under challenging illumination affected by external occlusions. Our approach achieves this by decoupling the unnatural affected illumination into multiple light conditions using learned neural representations.
Our Methodology
3.1 Problem Definition
This work focus on the texture modeling problem of human faces under challenging environment illumination affected by external occlusions. Given an input image or video sequences Ji,, taken from affected illuminations, our task is to recover clear and accurate texture T from Tin and ensure that T can synthesize results Tuu close to Jin-
3.2 Overall Illustration
As illustrated in Fig. 2, instead of making the assumption that the face is exposed to single and uniform illumination, we propose to model possible separate light conditions 7 Yu in the environment, where the masks for possible regions My are predicted with neural representation f(.). Effective masks Mr. and rendered faces /s are selected from original MN and rendered / Tn during optimization with the Adaptive Condition Estimation (ACE) strategy. The final rendered face under multiple light conditions Tn Σ TR C Mr, which merged with the input images to construct the output Lout IR @ M, + Iin @ (1 Mo). The face shape is modeled with statistical
3

---

3D Morphable Model (3DMM)
Hrighter
Light Decoupling
MR
Ray-tracing render
Difluse
Specular
Roughness
Texture T
JR
A
c- Idmmcit
+
O
+
at.5
B- Expression
Condition
Pose
O Multiph
M
Lian
Lights
M
M,
In
86)
+Im M.)
pho
T.To
Stage Optimize a.B.p
Stage 2: Optimize a. a,B.á, v, f(), R()
Stage 3: Optimize y, 7()
Figure 2: Illustration of our framework. The pipeline is proposed recover texture T and 3DMM statistical coefficients ax 3, 5, from the input image Iin- The statistical coefficient is used to initialized T. Render mask MM. and Faces IRa under light conditions 7 = {71 7n are acquired through ray-tracing rendering. f(.) and g(-) are neural representations predicting light masks MN and facial region mask M,. ACE is introduced select effective masks Mr and rendered faces IR. Jss are combined into IR with Mr, where J is merged with surroundings in Lin with M, to construct output image Lout- Lpho and Lian are photometric loss and landmark loss, respectively.
coefficients a, 8, and p. Following [12], they are mainly optimized with landmark loss:
Lion
||gout
Qin 12.
[ginl
where Oin 2D key points detected from Iin. 1 is the number of points in Oi lout denotes projected 3D key points on 2D plane. The color-related variables such as textures and lights are mainly optimized with photometric loss:
1
Lpho
Hout linll,
(2)
Lin
where Ima is defined as the number of pixels of input image Iin More details are presented below.
3.3 Light Decoupling
using B = 9 bands Spherical Harmonics (SH) and ray-tracing rendering model the illumination under self occlusions, model n possible light conditions. The coefficients are then simply initialized as 7/8 = 2 21-1 for " 7 = "?, where is all-one matrix with the Please note that we use each SH to imitate the local illumination after being affected by the external occlusion, instead of the global natural illumination. This means that we do not need to consider physical influence of occlusion in each SH. Instead, we optimize each SH independently to directly imitate the illumination in different face regions.
Light Condition Initialization. Following 12 13
separate SH to
shape the SH coefficient.
Neural Representations for Face Segment. To decouple the illumination into multiple light conditions, we design pair of ppatial-temporae continuous neural representations segment the face into regions for different light conditions. As illustrated in Fig. [ spatial positions æ, and temporal position / of pixels are normalized and embedded into coordinates system. is decided by the number of frames in the input image/video sequence. For the single image reconstruction, is set as a constant where would be i/k for the ith frame of a frames video sequences. For the convenience of learning, (z, y) are normalized into 1.0, 1.0] and t is normalized into [0.0, 1.0].

---

A Multi-Layer Perceptron (MLP) f(.) is then introduced to predict the probability of assignment to each light condition. Given the render mask R in the ray-tracing aaagiiiiin-- rendering, the light assisted masks can be obtained Mx = MR @ f" y. t). Effective masks Mr. are selected from MN with Adaptive Condition Estimation (ACE) to provide the final segment for different light conditions.
Similarly, we use another MLP g(-) to predict the probability that each pixel belongs to the face regions to avoid the influence from direct occlusion such as hat hair. The Face mask is thus given by: M, = MR O g y, t). A pre-trained semantic segmentation network [27] is introduced for distillation g(+), while the probability for labels of all face components such as eyes, mouth, or nose are added together to construct classifier h(-) to predict the association of pixel at æ, y, to the face. The distillation loss Lseg can be simply defined as:
1 Leeg - 19((, v. 1) h(x, y. 22
Iin
Note that g(-) is co-optimized together with the Lyho and Leeg to further distinguish some occlusions hard be fitted with the 3DMM statistical model.
Adaptive Condition Estimation (ACE). As the complexity of illumination is mutative in different surroundings, we design strategy to estimate the number of light conditions existing in this environment during optimization. Specifically, light masks with larger area than pre-defined threshold e in My are preserved in Mr, while smaller ones are dropped with corresponding light conditions and not further optimized in later iterations. Given the number of light masks MN same as light condition n, then MN = {Mý Y Mr = {Mi 1 = [My] [fin| Σmemy, m > G, <= n).
n. <= n is the number of preserved light conditions. m denotes the pixel in M We introduce an regularization Larea to to encourage the area concentration on fewer masks:
M
Larea
M
1.
where |Tm] is the number of pixels in the mask.
To ensure each light condition is mainly contained in one mask of Mr, we introduce a binary regularization Luin to ensure that the predicted probability for each light condition tends to 0 or 1:
Emeat
Σ
IIM
Lbin
1,
(5)
Lini
where E Mt denotes each pixel value in 7th light mask Mr.- n.y is the number of masks in Mr..
Note that Larea is different from Loin as it pushes the mask values My far from the mean of all masks, while Loin pushes mask values away from mean of different positions the same mask Mi. ACE is executed at a specific iteration itero to select Mr from My. We use Larea to encourage area concentration before ilero, while using Loin to push the mask to or after ilero-
3.4 Realistic Constraints
To ensure the reconstructed texture is reasonable and lifelike, we propose global prior constraint LGP, local prior constraint LLP, and human prior constraint LuP by introducing both priors from 3DMM statistical model and the pre-trained perceptual model.
Global Prior Constraint. The global prior constraint is used to ensure the consistency of overall hue between optimized texture T and initialized texture 7 from the 3DMM statistical model calculated with 8. As the colors mainly come from diffuse albedo in T, we estimate the overall hue of T with K- K-means algorithm its diffuse albedo Tß and get x 4 color matrices C. Given the diffuse albedo of Texture T as TD. this term is given by:
LGP
1
Σ
min ITD cll2- CEC
(6)
Local Prior Constraint. The local prior constraint is used to ensure the local smoothness of optimized texture T by constraining its local variation to be similar to T Let us define the diffuse, specular, and roughness albedos of texture T as Tp. Ts. TR. respectively. We set the local variations
5

---

Algorithm Training Process
1: Set the number of itero iters
2: Stage
3: for n== 1 to iter; do
4: Calculate the Loss of Stage 1 L1 Lian
5: Update coefficients with L1: V Vo,8.pL1.
6: end for
7: Stage 2:
8: for n = itery do
9: if' n <, itero then
10:
Calculate the Loss of Stage 2 on MN and LRn:
L2 = woLopho + wi + wylsse + wa Laraa else if n = itero then
12:
Select Mr and Ins from MN and Inn with ACE; 14: else
Calculate the Loss of Stage 2 Mr. and InS: I.2 = Tphopho + Laan + w2 Disg + wa Ltin
16:
end if
18: Let f 0g be the parameters of f()) and g()).
19: Update coefficients with L2: Va,8.prr
20: end for
21: Stage
22: Extract texture T from 8 separate variable.
23: for n== 1 to itery do
24: Calculate the Loss of Stage
25: La = wp Ipho + wa LGp + wgl.t. + wywLnn Lup
26: Update coefficients with Lzy: VT..0, Iz..
27:
end for
of Dp Ts. Ta as ND. Ns. NR. which are computed with the 5 x 5 neighbors around each pixel, ND Tp Neighbor(Tp)) ND. No. No are local variations calculated from To, The local prior constraint LLP is then defined as:
LLP
(IND - Ngll + ||Ns N2I + |R - NRRl).
Tial
Human Prior Constraint. Human prior constraint is introduced to enhance the optimization of texture T Stage with face recognition network FaceNet [35] pre-trained on large dataset such as VGGFace2 [7] or Sebbaee [39]. The recognition model pre-trained on VGGFace2 and Casia-webface tends classify the input images 8,631 and 10,575 identities with different genders, ethnicity, etc. In this work, we propose a human prior constraint by maximizing the probability that the rendered face is recognized on one of the identities by FaceNet. Defining the FaceNet f-(,), the human prior constraint can then be written as:
LHP Σ min ||
8||2-
(8)
(EFA(TR))
Rs are rendered under multiple light conditions. LHP constrains the texutures to be more reasonable.
3.5 Training Pipeline
The pipeline of the entire training process of our proposed framework is presented in Alg. I As illustrated in Fig. 2 the training of our proposed method consists of 3 stages, which is similar to NextFace [12. In Stage 1, the expression 88 shape a, and pose coefficients are optimized to ʃ(.) and g(-) are optimized together to reconstruct the face with statistical model. In Stage 3, the texture 7 is directly optimized without coefficient 6, while y and f()) are optimized with small learning rates.
construct the basic face shape. In Stage 2, all coefficients, and the neural representations
wi wy pre-defined weights. We use Adam optimizer [9 for optimization.

---

Source
CPEM
D3DFR
NextFace NextFace* FFHQ-UV
Ours
Target
Image
Texture
Image
Texture
Image
Texture
Figure 3: Comparison Voxceleb2 images. The diffuse albedo is visualized the texture because contains most of the color information. Textures from source images are used to synthesize the target images. NextFace* denotes results optimized within regions selected with face parsing [27]. We do not have textures for CPEM 30 or D3DFR 10 as they predict vertex colors instead of uv textures.
4 Experiments
4.1 Dataset and Implementation Details
We follow NextFace [2] for the ray-tracing rendering and multi-stage organization of optimization pipeline. Detailed hyper-parameter settings can be found in our supplementary. For our experiments, utilize two datasets: Voxceleb2 (8) and CelebAMask-HH
(29 24] VoxCeleb2 [8] is diverse
dataset encompassing numerous videos collected from interviews, movies and videos, where the same person may have multiple separate videos. CelebAMask-HI 29) 24 is large scale face image dataset with fine attributes annotation and high resolution, widely used in face editing and generation.
Evaluation. We construct collection of evaluation data with challenging illumination affected by external occlusions for comparative analysis. This collection includes 38 pairs of single images, 24 pairs of videos from Voxceleb2 [8] with 256 x 256 resolution, and 62 single images from CelebAMask-HO 22 with 512 x 512 resolution. Each pair consists of source images affected by external occlusions and target images without occlusion, both from the same person, where each video is sampled to 8 frames for sequence- based comparisons.
To quantitatively assess the quality of recovered textures, face textures extracted from the occluded source images are leveraged to synthesize the unoccluded target images. Specifically, we optimize source and target images separately following Sec. 3 Then, keeping the face shape, pose, and illumination invariant, we replace the target image's texture with the source's and re-render it to the synthetic result. This allows us to measure texture quality by quantifying the differences between the synthesized target images and the original target images. We also introduce differences between original and reconstructed source images within facial regions acquired by face parsing [27 as an assistant metric to see if textures restore source images well.
Our quantitative comparison employs three metrics: PSNR, SSIM, and perceptual error LPIPS [40] of AlexNet [20]. State-of-thh -frtt--attttttt methods D3DFR [1O], CPEM [30], NextFace [[2]. and RGB are introduced for comparison. For D3DFR, we adopt its
Fitting methods proposed in FFHQ-UV
enhanced PyTorc version. As NextFace [2] does not distinguish the face and external occlusion when optimizing the texture, we introduce NextFace* subsequent comparison by adding face parsing 27 to select the face region during optimization.
7

---

Source
CPEM
D3DFR
NextFace NextFace FFHQ-UV
Ours
Ours+
L
L
LA
LA
li
Image
Texture
Image
Texture
Image
Texture
Figure 4: Comparison results the CelebAMask-HO dataset. Ours and Ours+ denote our rendered
results In directly overlapped onto original images, and results combined with environments: lout M. o In (1 Mo) @ Iin respectively.
Table 1: Quantitative comparison on sin- gle images from Voxceleb2. Source and Target denote differences evaluation on re- constructed source images and synthetic tar- get images. LPIPS is multiplied with 10². Underline and bold mark the suboptimal and optimal results, respectively.
Table Quantitative comparison on video sequences from Voxceleb2. Source and Tar- get denote differences evaluation on recon- structed source sequences and synthetic target sequences.
NextFace NextPace"
FFHQ-UV
CPEM 133DHR NextiFace NextFace
PHRQ-UV
Ours
PSNR
27.69
24.55
30.60
30.65
29.70
30.67
Sonree SSIM
0.87
0.87
0.92
0.92
0,92
0.92
PSNR
27.06
24.26
32.76
28.73
32.85
LPIPS
9.21
9.92
8.17
8.45
7.51
7.62
Source LPIPS
0.93
PSNR
24.33
26.20
24.15
24.32
24.35
29.15
Target SSIM
0.87
0.90
0.87
0.87
0.91
0.91
PSNR
24.70
24.21
25.03
29.22
LPPSS
9.58
8.13
10.27
9.91
7.66
6.92
Targer
SSIM
0.91
0.36
LPIPS
9.43
7.26
10.52
10.02
7.19
6.36
4.2 Evaluation on Voxceleb2
Evaluation on Single Images. We conduct an evaluation about the performance of our method in collected single images sourced from the Voxceleb2 dataset [8]. As shown in Fig. 3 our approach demonstrates marked improvement in recovering clearer textures from the original images taken under challenging illumination affected by external occlusions. Our method also surprisingly recovers clear textures under strong colorful lights. which may benefit from the realistic constraints to keep textures reasonable. contrast, existing methods tend to incorporate these occlusions, shadows, or colorful lights directly into the texture, resulting in less satisfactory outcomes.
Furthermore, our methods achieves superior results on the synthesis target images and comparable performances on the reconstruction of source images as shown in Table [ It confirms our method consistently recovers both accurate and clear textures. Although NextFace [22 performs little better on PSNR and SSIM of source images, it performs the worst synthetic target images as it actually severely over-fits source images. As shown in Fig. 3 it bakes shadows and occlusions to the textures, which confirms it is not appropriate for faces affected by external occlusions.
Evaluation on Video Sequences. We conduct an evaluation on video sequences collected from Voxceleb2 dataset 8 to further substantiate the efficacy of our approach. To adapt framework

---

Source
NA
Light
Occlusion
Target
Figure 6: Ablation study for the neural represen- tations. NA means remove both f(-) and g(-). while + Light and + Occlusion denote adding f()) and g(-), respectively.
Ablation study for losses. GP, LP, and HP denote LGP. LLP, LHP, respectively, while NA means to remove all of them.
Figure 5:
Table 3: Quantitative ablation study for pro- posed Losses GP, LP, and HP.
Table 4: Quantitative ablation study for f(-), g()).
Metries NA + Light (f(+)) Occlusion (g(-))
Metrics
NA
GP
GP+LP GP+LP+HP (ours)
PSNR 25.19
27.52
29.22
PSNR
27.20 28.81
28.82
29.22
SSIM 0.87
0.89
0.91
SSIM 0.89
0.90
LPIPS J. 9.16
7.75
6.36
LPIPS
8.34 6.78
6.49
6.36
to video sequences, we share texture, illumination and shape coefficients across all frames during optimization. As D3DFR (10) and FFHQ-UV [2] do provide support for sequences, recur- rently apply them to each single image. Quantitative comparison "Table[demosstraeee that our method constantly performs superior existing methods texture modeling from continuous video sequences. Please check our Supplementary for corresponding qualitative results.
4.3 Evaluation on CelebAMask--HO
We extend our comparisons on CelebAMask-HH HQ 29 As this dataset lacks multiple images from the same identity, we focus on evaluating the performance based on recovered textures and reconstructed results of source images. The outcomes of our assessment are visually depicted in Fig. + We observe that our method continues perform well in the task of recovering clear textures from challenging input. Additionally, reconstructed results exhibit higher degree of realism the reconstructed results when compared to the outcomes produced by other methods.
4.4 Evaluation on Images with Diverse Shadows 41
To validate the performances of our method more sufficiently, we further evaluate our method the single image dataset proposed by [41], which includes 100 images affected by manually created external shadows, as well as corresponding ground truths. The quantitative and qualitative comparisons are presented in Table 5 and Fig. 7 We can observe that our method still outperforms other methods under faces with diverse shadows.
Table 5: Quantitative comparisons images with diverse shadows.
Metrics |CPEM D3DFR NextFace NextFace FFHQ-UV Ours
PSNR
24.75
23.02
32.10
31.92
26.93
32.13
SSIM
0.83
0.83
0.94
0.94
0.89
0.94
LPIPS
10.99
12.76
5.26
5.34
8.71
6.29
PSNR
23.34
25.02
21.83
21.93
24.00
28.97
SSIM
0.84
0.87
0.84
0.84
0.88
0.92
LPIPS
9.31
9.11
9.48
9.69
7.82
7.00

---

Source
CPEM
D3DFR
NextFace NextFace*
FFHQ-UV
Ours
Target
Image
Texture
Image
Texture
Figure 7: Comparison on images with diverse shadows 41 As provides corresponding ground truths of shadow affected images, we directly use such ground truths as the target images.
4.5 Ablation Study
Ablation Study on Losses. In this section, we analyze the impact of the proposed losses: LGp. LLP. and LиP by comparing the rendered results under relative brighter light conditions. As illustrated in Fig. it becomes evident that the inclusion of the LGP loss plays significant role in eliminating the abnormal colors on the texture, where LLP loss effectively remove large artifacts obviously different from human faces. LиP further reduce slight unreasonable defects with priors from the pre-trained recognition model [35]. Quantitative comparisons in Table 3 also confirm that each proposed loss contributes the final performances.
Ablation Study on Neural Representations. To ascertain the significance of the neural repre- sentations employed, we perform analysis where eliminate both the neural representations: f()) and g(-) from the light decoupling pipeline. As depicted in Fig. 6 the neural representation for the decoupling of light conditions f-) notably contributes in the removal of shadows artifacts from the results. Furthermore, the neural representation g(-) displays efficacy in minimizing existing irregularities in the synthetic results, which achieves smoother and cleaner results. We also present corresponding quantitative comparisons Table which demonstrates that both f(-) and g(-) have obvious influence the final performances.
5
Conclusion
In this paper, we present novel framework dedicated to recover clear facial textures from images taken under challenging illumination affected by external occlusion. Our approach makes use of neural representations decouple the original illumination into multiple separate light conditions across various facial regions. Then the affected complicated illumination can be modelled with the combination of different light conditions. Furthermore, introduce well-established human face priors through the realistic constraints to enhance the realism of our results. According to experiments single images and video sequences, method consistently surpasses existing techniques to recover clearer and more accurate textures from faces under affected unnatural illumination.
Acknowledgement
This research project supported by the National Research Foundation (NRF) Singapore, under its NRF-Investigaaoosship Programme (Award ID. NRF-NRF109-0008),
References
Edward Angel. nteractive computer graphics: towndown approach with shader-based opengl. 2011.
10

---

[2] Haoran Bai, Di Kang, Haoxian Zhang, Jinshan Pan, and Linchao Bao. Flhq-uv: Normalized facial uv-texture dataset for 3d face reconstruction. In Proceedings the IEEE/CVF Conference Computer Vision and Pattern Recognition. pages 362-371, 2023.
[3] Linchao Bao, Xiangkai Lin, Yajing Chen, Haoxian Zhang, Sheng Wang, Xuefei Zhe, Di Kang, Haozhi Huang. Xinwei Jiang, Jue Wang. et al. ACM Transactions on Graphics (TOG), 41(1):1-21, 2021.
3d digital buman head creation from rgb-d selfies.
[4] Volker Blanz and Thomas Vetter. A morphable model for the synthesis of 3d faces. In Seminal Graphics Pupers: Pushing the Boundaries, Volume 2, pages 157-164, 2023.
[5] James F Blinn. Models of light reflection for computer synthesized pictures. In Proceedings of the 4th annual conference on Computer graphics and interactive techniques, pages 192-198, 1977.
[6] James Booth, Anastasios Roussos, Stefanos Zafeiriou, Allan Ponniah, and David Dunaway. A 3d morphable model Iearnt from 10,000 faces. In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 5543-5552, 2016.
[7] Qiong Cao, Li Shen, Weidi Xie, Omkar M Parkhi, and Andrew Zisserman. Vggface2: A dataset for recognising faces across pose and age. In 2018 13th TEEE international conference on automatic face & gesture recognition (FG 2018), pages 67-74. IEEE, 2018
[8] Joon Son Chung, Arsha Nagrani, and Andrew Zisserman. Voxceleh2: Deep speaker recognition. arXiv preprint arXiv: 05622, 2018.
[9] Wei-Chieh Chung, Jian-Kai Zhu, I-Chao Shen, Yu-Ting Wu, and Yung- Yu Chuang. Stylefaceuv: A 3d face map generator for view-consistent face image synthesis. 2022.
[10] Yu Deng, Jiaolong Yang, Sicheng Xu, Dong Chen, Yunde Jia, and Xin Tong. Accurate 3d face reconstruc- tion with weakly-supervised learning: From single image image set. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition workshops, pages 0-0, 2019.
[11] Abdallah Dib, Junghyun Ahn, Cedric Thebault, Philippe-Horee Henri Gosselin, and Louis Chevallier. S212: Self- supervisd high fidelity face reconstruction from monocular image. arXiv preprint 22370:2203.07732. 2022.
[12] Abdallah Dib, Gaurav Bharaj. Junghyun Ahn, Cédric Thébault, Philippe Gosselin, Marco Romeo, and Louis Chevallier. Practical face reconstruction via differentiable ray tracing. Computer Graphics Forum, volume 40, pages 153-164. Wiley Online Library, 2021.
[13] Abdallah Dib, Cedric Thebault, Junghyun Ahn, Philippe- Henri Gosselin, Christian Theobalt, and Louis Chevallier. Towards high fidelity monocular face reconstruction with rich reflectance using self-supervised learning and ray tracing. In Proceedings of the IEEE/CVF International Conference Computer Vision, pages 12819-12829, 2021
[14] Bernhard Egger, Sandro Schönbom, Andreas Schneider, Adam Kortylewski, Andreas Morcl-Forster Clemens Blumer, and Thomas Vetter. Occlusion-awan 3d morphable models and an illumination prior for face image analysis. International Journal of Computer Vision, 1011116998229 2018.
[15] Bernhard Egger, William AP Smith, Ayush Tewari, Stefanie Wuhrer, Michael Zollhoefer, Thabo Beeler, Florian Bernard, Timo Bolkart, Adam Kortylewski, Sami Romdhani, et al. 3d morphable face models-pan past, present, and future. ACM Transactions on Graphics (ToG), 39(5):1-38, 2020.
[16] Baris Gecer, Stylianos Ploumpis, Irene Kotsia, and Stefanos Zafeiriou. Ganfit: Generative adversarial network fitting for high fidelity 3d face reconstruction. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pages 1155-1164 2019.
[17] Yingqing He, Yazhou Xing. Tianjia Zhang, and Qifeng Chen. Unsupervised portrait shadow removal via generative priors. In Proceedings of the 29th ACM International Conference on Multimedia, pages 236-244, 2021.
[18] Andrew Hou, Michel Sarkis, Ning Bi, Yiying Tong, and Xiaoming Liu. Face relighting with geometri- cally consistent shadows. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pages 4217-4226, 2022.
[19] Diederik P. Kingma and Jimmy Ba. Adam: A method for stochastic optimization. arXiv preprint M4Xiv:1422.6000 2014.

---

[20] Alex Krizhevsky, Ilya Sutskever, and Geoffrey E Hinton. Imagenet classification with deep convolutional neural networks. Advances in neural information processing systems, 25, 2012.
[21] Alexandros Lattas, Stylianos Moschoglou, Baris Gecer, Stylianos Ploumpis, Vasileios Triantafyllou, Abhijeet Ghosh, and Stefanos Zafeiriou. Avatarme: Realistically renderable 3d facial reconstruction" mn-the-wild". In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pages 760-769, 769, 2020.
[22] Alexandros Lattas, Stylianos Moschoglou, Stylianos Ploumpis, Baris Gecer, Jiankang Deng, and Stefanos 7afeiriou. Fitme: Deep photorealistic 3d morphable model avatars. In Proceedings of the IEEF/CVF Conference on Computer Vision and Pattern Recognition, pages 8629-8640, 2023.
[23] Alexandros Lattas, Stylianos Moschoglou, Stylianos Ploumpis, Baris Gecer, Abhiject Ghosh, and Stefanos Zafeiriou. Avatarme+4: Facial shape and brdf inference with photorealistic rendering-aware gans. IEEE Transactions on Puttern Analysis and Machine Intelligence, 44(12):929-99284 2021.
[24] Cheng-Han Lee, Ziwei Liu, Lingyun Wu, and Ping Luo. Maskgan: Towards diverse and interactive facial image manipulation. In IEEE Conference on Computer Vision and Pattern Recognition (CVPR), 2020.
[25] Chunlu Li, Andreas Morel-Forster, Thomas Vetter, Bernhard Egger, and Adam Kortylewski. Robust model-based face reconstruction through weakly- outlier segmentation. In Proceedings of the IEEE/CVF Conference on Computer Vision and Puttern Recognition, pages 372-381, 2023.
[26] Tzu-Mao Li, Miik Aittala, Frédo Durand, and Jaakko Lehtinen. Differentiable monte carlo ray tracing through edge sampling. ACM Trans. Graph. (Proc. SIGGRAPH Asia), 37(6):222:1 22222222111 11, 2018
[27] Jinpeng Lin, Hao Yang. Dong Chen, Ming Zeng. Fang Wen, and Lu Yuan. Face parsing with roi tanh- warping. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 5654-5663, 2019.
[28] Yaojie Liu, Andrew Z Hou, Xinyu Huang, Liu Ren, and Xiaoming Liu. Blind removal of facial foreign shadows. In BMVC, page 88, 2022.
[29] Ziwei Liu, Ping Luo, Xiaogang Wang, and Xiaoou Tang. Large-scale celebfaces attributes (celeba) dataset. Retrieved August, 15(2018):11, 2018.
[30] Langyuan Mo, Haokun Li, Chaoyang Zou, Yubing Zhang, Ming Yang, Yihong Yang, and Mingkui Tan. Towards accurate facial motion retargeting with iddnttty-consisten and expression-exclusee constraints. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 36. pages 1981-1989, 2022.
[31] Thomas Nestmeyer, Jean- François Lalonde, lain Matthews, and Andreas Lehrmann. Learning physics- guided face relighting under directional light. In Proceedings of the IEEE/CVF Conference Computer Vision and Pottern Recognition, pages 5124-5133, 2020.
[32] Pascal Paysan, Reinhard Knothe, Brian Amberg, Sami Romdhani. and Thomas Vetter. 3d face model for pose and illumination invariant face recognition. In 2009 sixth IEEE international conference on advanced video and signal based surveillance, pages 296-301. Icce, 2009.
[33] Ravi Ramamoorthi and Pat Hanrahan. An efficient representation for irradiance environment maps. In Proceedings of the 28th annual conference on Computer graphics and interactive techniques, pages 597-500,22 2001.
[34] Ravi Ramamoorthi and Pat Hanrahan. A pinnal-n framework for inverse rendering. In Proceedings of the 28th annual conference Computer gruphics and interactive techniques, pages 117-128, 2001.
[35] Florian Schroff, Dmitry Kalenichenko, and James Philbin. Facenet: A unified embedding for face recognition and clustering. Proceedings of the IEEE conference on computer vision and pattern recognition, pages 815-823, 2015.
[36] Soumyadip Sengupta, Angjoo Kanazawa, Carlos D Castillo, and David W Jacobs. Sfsnet: Learning shape, reflectance and illuminance of facesin the wild'. In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 6296 6305, 2018
[37] William AP Smith, Alassane Seck, Hannah Dee, Bernard Tiddeman, Joshua B Tenenbaum, and Bernhard Egger. A morphable face albedo model. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 5011-5020 2020.
12

---

[38] Chao Xu, Yang Liu, Jiazheng Xing, Weida Wang, Mingze Sun. Jun Dan, Tianxin Huang, Siyuan Li. Zhi-Qi Cheng, Ying Tai, et al. Facechain-imagineid Freely crafting high-fidelity diverse talking faces from disentangled audio. Proceedings of the IEEE/CVF Conference on Computer Vision and Pattenn Recognition, pages 1292-1302 2024.
[39] Dong Yi, Zhen Lei, Shengeai Liao, Stan Li. Learning face representation from scratch. arXiv preprint arxiv:1411.7922 2014.
[40] Richard Zhang, Phillip Isola, Alexci Efros, Eli Shechtman, and Oliver Wang. The unreasonable effectiveness of deep features as a perceptual metric. Proceedings of the IEEE conference on computer vision and pattern recognition, pages 586 595, 2018.
[41] Xuaner Zhang, Jonathan Barron, Yun-Ta Ta Tsai, Rohit Pandey, Xiuming Zhang, Ren Ng. and David Jacobs. Portrait shadow manipulation. ACM Transactions on Graphics (TOG), 39(4):78-1, 1, 2020.
[42] Zhenyu Zhang, Renwang Chen, Weijian Cao, Ying Tai, and Chengjie Wang. Learning neural proto-face field for disentangled 3d face modeling in the wild. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pages 382-393, June 2023.
[43] Zhenyu Zhang, Yanhao Ge, Ying Tai, Xiaoming Huang. Chengjie Wang, Hao Tang, Dongjin Huang. and Zhifeng Xie. Learning restore 3d face from in-the-wild degraded images. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pages 4237 4247, June 2022.
[44] Mingwu Zheng, Hongyu Yang, Huang, and Liming Chen. Imface: nonlinear 3d morphable face model with implicit neural representations. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 20343-20352, 2022.
[45] Mingwu Zheng, Haiyu Zhang, Hongyu Yang, and Huang. Neuface: Realistic neural face rendering from multi-view images. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 16868-16877, 2023.
13

---

A Appendix Supplemmentry Material
A.1 Limitation
We use AlbedoMM [37 to initialize the face texture and optimize neural representations, Limited by the capacity of AlbedoMM, the initialized face texture may be accurate enough for subsequent optimization in some occasions. We can explore to resolve this problem by combining our method with non-linear texture modeling methods such as 22 2 in our future work. Furthermore, the optimization of multiple light conditions makes our method slower than other single illumination methods [[2 2] costs 340s for 256 256 image on 2080ti, which slower than 240s of FFHQ-UV 2 but still affordable.
A.2 Details of Parameter Settings
In T we present details about the hyper-pane seper-parameters mentioned in Sec. 3 Although seems that there are multiple parameters, the setting is robust for both images and video sequences from adopted datasets. We conduct all experiments on a Nvidia 2080ti GPU with 2.9Ghz i5-9400 CPU. It costs 340s for 256 x 256 image.
x, t
16
64
64
MN
f())
SoftMax
x,y,
16 --- 64
64
M
g
Sigmoid
Leaky Relu
Figure 8: Detailed design of f(-) and 9(-).
Table 6: Hyper-paramette settings, n is the number of initial light conditions presented in Fig.
Parameter
~ Landmark itera ~ iters
2c3, 1e-3, 1.5c2, 0.5, 25, 2c3, 2.0, 1.0 Mediapipe
100, 2000, 400, 200
0.17
n
A.3 Comparisons against the combination of 2D Shadow Removal and Texture Modeling
Although the former mentioned existing 3D face texture modeling methods cannot process facial shadows from external occlusions directly, there are some 2D shadow removal methods 41 17 28
trying to eliminate the shadows directly from the images. Therefore, another alternative simple networks before texture modeling.
baseline is to pre-process the image with 2D In this section, introduce the most recent method [17] with pre-trained model to pre-process the images before feeding them to compared methods mentioned in Sec. 4.I The quantitative and qualitative comparison presented in Table Table and Fig. respectively.
Table 7: Comparisons against basclines with 2D shadow-removal pre-processing on
CPEM D&DFR NextFace NextFace FFHQ-UV
Ours
PSNR
25.56
25.60
25.24
25.47
26.19
28.97
SSIM
0.86
0.88
0.87
0.87
0.90
0.92
LPIPS
8.56
8.68
6.37
6.41
7.02
7.00
Pre-processing with shadow removal methods indeed improves the performances of baselines, while our method still outperforms them. From qualitative results, we observe that the shadow
14

---

g(.
g(.) + f()
Source
W2 = 50
W2 = 100
W2 = 150
(W2 = 150)
GS
Figu
Figure 9: Discussion about the effect g(-). W2 is the weight to constrain g(.). defined Alg.
The red and black rectangles mark chadow-affected regions and detailed textures, respectively. g( will weaken both shadows and details from textures when reducing W2 to loose constraint.
Input
NA
Larea
Larea + Lbi
IRs:
IRs
Figure 10: Ablation study for Larea and Loin in ACE. NA denotes removing both Lorea and Liin- Larea can remove redundant light conditions as shown by the blue rectangle, while Lbin ensures the light condition shown in the red rectangle region is consistent as observation of the input. Mr and It. are predicted masks and rendered faces defined in Fig. 2 respectively.
removal model cannot fully remove the external shadows in cases where the external shadows cover relatively large regions. Although modifying the shadow removal model to be more powerful may further improve performances, it goes beyond the range of this work. We can explore it in the future.
A.4 Comparisons with Deocclusion methods
Besides the former mentioned shadow removal baselines, learning- based deocclusion methods 25 14] can remove external occlusions from faces by predicting the occluded regions. Such operations also have the potential to deal with external shadows by directly treating the shadow regions as occlusions. In this section, we conduct a brief comparison with the most recent open sourced deocclusion method [25]. The quanlitative and qualitative results are presented in Table and
Fig. 12 respectively. The metrics are evaluated on the target images mentioned in Sec. 4. We can observe that the method (25) lose some facial details such as beards. The reason that the dcocclusion method (25) actually divides occlusion regions by the distances between input images
15

---

Table 8: Comparisons against basclines with 2D shadow-removal pre-processing on imgees from Voxceleb2.
CPEM D3DFR NextFace NextFace* FFHQ-UV Ours
PSNR 1
24.84
26.78
23.77
24.51
25.35
29.22
SSIM f
0.87
0.90
0.85
0.86
0.91
0.91
LPIPS
10.20
7.93
10.47
9.64
7.62
6.36
Shadow Removed
Source
CPEM
D3DFR
NextFace NextFace FFHQ-UV
Ours
Target
Image
Texture
Image
Texture
Figure 11: Qualitative Comparisons against baselines with 2D shadow-removal pre-processing.
and the reconstructed results from 3D Morphable Model (3DMM), which will also regard details hard to describe with 3DMM as occlusions. Our methods treat the shadows as different illumination instead of occlusions, which can preserve facial details in these regions in subsequent optimization.
A.5 Discussion about the number of Sphere Harmonics (SH) bands.
In this work, follow NextFace [1] actually capture quite fine details. However, SH with more bands would have stronger ability for the modeling of illumination. To verify if the external shadows can be directly modelled with more bands SH, we also provide the quantitative results of our method modelled with only one single global SH in 9, 12, 15, 18 bands in Table TO] where remove light decoupling framework. We observe that increasing the number of bands in single global SH yields quite limited improvements, A possible reason is that the external occluded shadows on human faces represent drastic illumination changes in relatively small areas, which may not be appropriately modeled as single global SH during optimization.
9-bands SH to model the local illumination, which can
A.6 Discussion about the number of lighting conditions n1
In this work, initialize different lighting conditions to imitate the illumination affected by external occlusion at the begimning, where some of them would be removed by ACE in subsequent processing. To explore the influence of different numbers of conduct ablation study for the number of n used for initialization of lighting conditions in Table Single images from VoxCeleb2 used for evaluation. We observe that number of lighting condition candidates is insufficient to model images with complex illuminations. In contrast, 5, 7,9 9 yield good and similar results there enough initial lighting conditions, and any redundant ones are removed. The result for n = is slightly better. While introducing larger and further adjusting the parame-arseeters might improve performance, it would also increase the optimization burden.
3 produces sub-optimal results, likely because the
A.7 Differences between Human Prior Constraint and Perceptual Loss
Please note that the rendred faces used calculate Human Prior Constraint (HP) are calculated with I I As illustrated in Fig.2, 2, [ includes rendered faces under multiple different lighting conditions.
16

---

Table 9: Comparisons against basclines with Deocclusion methods on Voxceleb2.
Single Images
Sequences
Methods
Deocelu
Ours
Deocclu Ours
PSNR
25.13
29.22
25.03
29.15
SSIM f
0.88
0.91
0.88
0.91
LPIPS
14.09
6.36
14.18
6.92
Source
De-occlu
Ours
Target
Image
Bu
Bu
Texture
Image
Texture
Figure 12: Qualitative Comparisons with the Deocclusion method 25
We cannot use perceptual loss here because we do not have *#corresponding ground truth images under the multiple decoupled lighting conditions** where HP does not need such ground truths. Nonetheless, the perceptual loss can be indeed applied between the final rendered result Loue and input image Iin- We present a comparison between such perceptual loss implementation and the HP in Table 12] We can that HP still has better performances, which can confirm it provides more effective constraints for the textures through rendered faces under multiple lighting conditions.
A.8 Discussion about g(-).
Both f(-) and g(-) have impacts to remove artifacts on the recovered textures. To confirm that they have principled differences, we implement discussion about the effect of g(-). As presented in the third column of Fig. 9 g(-) cannot fully avoid the influence of external shadows on the faces. As presented Sec. 3.3, Eseg introduced to constrain g(-) to ensure that g(-) predicts relatively accurate face regions. Lseg is weighted by in optimization defined in Alg. 3 Although the
shadow effect can decrease when we reduce w2 allow g(-) to filter out more face regions, the
detailed textures are equally weaken and may be fully removed as shown in the last row of Fig. 9 It confirms that f(-) is still essential for this framework.
A.9 Analysis about Luin and Larea in ACE.
In ACE mentioned in Sec 3.3 two regularization constraints Laree and Lbin are introduced. As the blue rectangle input face of Fig. i0 not affected by any external occlusion, its illumination can be modelled with only one light condition. Larea helps remove redundant light conditions. However, only using Tarrea may create decoupled light conditions obviously different from the observation of input image, as illustrated in the red rectangle regions of Fig. 10 Adding Loin ensure the
17

---

Table 10: Ablation study for the number of SH bands.
15
Ours
PSNR
25.87
25.26
25.27
25.34
29.22
SSIM
0.87
0.87
0.87
0.87
0.91
LPIPS
9.16 9.23
9.22
9.10
6.36
Table 11: Ablation study for the number of lighting conditions n.
(we use)
9
PSNR
28.49
29.22
29.14
29.16
SSIM
0.90
0.91
0.91
0.91
LPIPS
6.37
6.36
6.37
6.39
predicted masks Mr be nearly binarized, which makes the decoupled light conditions consistent as the human observation of the input image.
A.10 Differences between Stage and Stage 3.
Illustrated in Fig. 2 our approach composes of both Stage 2 and Stage optimize the face texture. In this section. we conduct comparisons to demonstrate differences between textures from Stage and Stage The results are presented in Fig. 13
We can observe that textures from AlbedoMM [37
Stage
Stage
Source
Reconstructed
Texture
Reconstructed
Texture
Figure 13:
Differences between Stage 2 and Stage 3. In Stage 3, the texture is refined with details from the source image, such as the beard, to render more realistic reconstructed image.
in Stage 2 are quite over-smoothed and lack of details. With Stage 3, details in source images are added to textures, which can reconstruct more realistic results than Stage 2.
A.11 Visualization about Mo
We present an ablation study to confirm the effect of g(-) against direct segmented mask from face parsing [27] to predict Mg. The circled parts of Fig. 14 show that the parsed masks may be inappropriate due to the limitation of generalizability. We can see in the first row of Fig. 1h4t that the rendered result IR may have rough edges and artifact colors from occlusion, while our method can avoid this problem by refining the parsed mask with g(-). Moreover, some parts may be missing on the parsed mask. As shown in the second and third rows Fig. [4] artifacts show up these unconstrained regions, Our method can complete these missing regions for more reasonable results.
18

---

Table 12: Comparison between HP and Perceptual loss.
Perceptual Loss 10 (ours)
PSNR SSIM
28.72
29.22
0.91 0.90
LPIPS
6.46
6.36
w/O 8()
W/ g(:)
Input
M
M
1R
Figure 14: Ablation study for the usage of neural representation g(-). W/O g(-) and W/ g(-) denote using parsed mask (27] and using 9(-), respectively. Mo and In are the mask and rendered results, as mentioned in Sec. 3
A.12 Discussion about the failure cases.
Except the efficiency problem mentioned in Sec. A.I our primary limitation is the initialization with AlbedoMM. As shown in Fig. 15 for faces with many high-frequency details, such as wrinkles, our method may lose these details during reconstruction. Replacing AlbedoMM with more powerful face representations could address this issue. We plan explore this further in future work.
Source
Reconstructed
Source
Reconstructed
Figure 15: Some failure cases.
A.13 Effect Adaptive Condition Estimation
As described in Sec. 3.3 the Adaptive Condition Estimation (ACE) is proposed to select effective Mr. and TRs from the initialized Mx and Tnn- remove ACE, we use initialized MN and T Tr we can that the rendered results Lout are almost
as M2 and Rs directly. As shown in Fig. 16
19

---

Input
W/o ACE
W/
lm
M, (up) and (down)
loue
My(up) and (n:(d))))
lou
Figure 16: Ablation study for ACE. W/O ACE and W/ ACE denote removing ACE by using M and Rn as Mr and TR.. and using ACE to select M,, and 7 Rs.. respectively.
Source
CPEM
D3DFR
NextFace NextFace FFHQ-UV
Ours
Ours+
GT
Figure 17: Comparison results on the video sequences from Voxceleb2. Ours and Ourst denote our rendered results I directly overlapped onto original images, and results combined with environments: lout = M, 3 IR + (1 M,) @ lin, respectively. The symbols are defined following Sec. 3
the same, while the results optimized without ACE contain multiple redundant and inaccurate light conditions Mt. confirms that ACE can help remove these unnecessary light conditions and keep effective ones. With ACE, our method can decouple the original illumination affected by occlusions into light conditions more consistent with actual observations of input images.
A.14 More Visualized Results
In this section, we present three representative examples from each sequence in Fig. 17 It is evident that our method performs the best in generating synthetic results close the target sequences, exhibiting high degree of realism. In contrast, other methods still produce less convincing outcomes due to negative effects from external occlusions. More results on images sourced from Voxceleb2 and CelebAMMak--Hi HQ 24 are presented in Fig. [8 and Fig. 19 Our method still performs better. Please refer to the attached video for more results on sequences from Voxceleb2 g
20

---

Source
CPEM
D3DFR
NextFace
NextFace* FFHQ-UV
Ours
Target
VINE WEARD
Figure 18: More results on single images from Voxceleh2 8 We can see that our method can synthesize more accurate target images based on textures from the source images, which validate the quality of our acquired textures.
21

---

Input
CPEM
D3DFR
NextFace
NextFace*
FFHQ-UV
Ours
Figure 19: More reconstructed images/textures results on CelebAMask- HQ 24 We can see that our method can reconstruct more realistic results, with clear textures without shadow effects.
22

---

NeurIPS Paper Checklist
1. Claims
Question: Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?
Answer: [Yes]
Justification:
We have discussed them in the page 2.
Guidelines:
The answer NA means that the abstract and introduction do not include the claims made in the paper.
The abstract and/or introduction should clearly state the claims made, including the contributions made in the paper and important assumptions and limitations. A No or
NA answer to this question will not perceived well by the reviewers.
The claims made should match theoretical and experimental results, and reflect how much the results
be expected to generalize other settings.
is fine to include aspirational goals as motivation as long is clear that these goals are not attained by the paper.
2. Limitations
Question: Does the paper discuss the limitations of the work performed by the authors?
Answer: [Yes]
Justification:
We have discussed about that in Sec. A.I
Guidelines:
The answer NA means that the paper has no limitation while the answer No means that the paper has limitations, but those not discussed in the paper.
The authors are encouraged to create separate "Limitations" section in their paper.
The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (e.g., independence assumptions, noiseless settings, model well-specification. asymptotic approximations only holding locally). The authors should reflect how these assumptions might be violated in practice and what the implications would be.
The authors should reflect on the scope of the claims made, e.g., if the approach was only tested on few datasets or with few runs. In general, empirical results often depend on implicit assumptions, which should be articulated.
The authors should reflect the factors that influence the performance of the approach. For example, facial recognition algorithm may perform poorly when image resolution low images are taken in low lighting. Or speech-to-text system might be used reliably to provide closed captions for online lectures because it fails to handle technical jargon.
The authors should discuss the computational efficiency of the proposed algorithms and how they scale with dataset size.
If applicable, the authors should discuss possible limitations of their approach address problems of privacy and faimess.
While the authors might fear that complete honesty about limitations might be used by reviewers as grounds for rejection, a worse outcome might be that reviewers discover limitations that aren't 't acknowledged in the paper. The authors should use their best judgment and recognize that individual actions in favor of transparency play an impor- tant role in developing norms that preserve the integrity of the community. Reviewers will be specifically instructed to not penalize honesty concerning limitations.
3. Theory Assumptions and Proofs
Question:
For each theoretical result, does the paper provide the full set of assumptions and complete (and correct) proof?
Answer: [NA]
23

---

Justification: We do not have theoretical contributions in this work, where our contributions are validated with experiments.
Guidelines:
The answer NA means that the paper does not include theoretical results.
All the theorems, formulas, and proofs in the paper should be numbered and cross- referenced.
All assumptions should be clearly stated or referenced the statement of any theorems.
The proofs can either appear in the main paper or the supplemental material, but if they appear in the supplemental material, the authors are encouraged to provide short proof sketch provide intuition.
Inversely, any informal proof provided in the core of the paper should be complemented by formal proofs provided in appendix or supplemental material.
Theorems and Lemmas that the proof relies upon should be properly referenced.
4. Experimental Result Reproducibility
Does the paper fully disclose all the information needed to reproduce the main ex- perimental results of the paper to the extent that it affects the main claims and/or conclusions of the paper (regardless of whether the code and data are provided or not)?
Question:
Answer: [Yes]
Justification: All the hyper-parameters and network organizations are provided in the supplementary. We would release our codes in http:/giiwiioionoaaainnannnaa//a
Guidelines:
The answer NA means that the paper does not include experiments.
If the paper includes experiments, a No answer to this question will not be perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data are provided or not.
If the contribution is dataset and/or model, the authors should describe the steps taken make their results reproducible or verifiable.
Depending on the contribution, reproducibility can be accomplished in various ways. For example, if the contribution is novel architecture, describing the architecture fully might suffice, or if the contribution a specific model and empirical evaluation, it may be necessary either make it possible for others to replicate the model with the same dataset, or provide access to the model. In general. releasing code and data is often one good way accomplish this, but reproducibility can also be provided via detailed instructions for how to replicate the results, access to a hosted model (e.g., in the case of large language model), releasing of a model checkpoint, or other means that appropriate to the research performed.
While NeurlPS does not require releasing code, the conference does require all submis- sions to provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution. For example
(a) the contribution is primarily a new algorithm, the paper should make clear how reproduce that algorithm. (b) If the contribution is primarily new model architecture, the paper should describe the architecture clearly and fully.
(c) the contribution is a new model (e.g., a large language model), then there should either be way to access this model for reproducing the results or way to reproduce the model (e.g., with an open-source dataset instructions for how to construct the dataset).
(d) We recognize that reproducibility may be tricky in some cases, in which case authors are welcome deseribe the particular way they provide for reproducibility. In the case of closed-source models, it may be that access to the model is limited in some way (e.g., to registered users), but it should possible for other researchers to have some path to reproducing or verifying the results.
5. Open access to data and code
24

---

Does the paper provide open access to the data and code, with sufficient instruc- tions faithfully reproduce the main experimental results, as described in supplemental material?
Question:
Answer: [Yes]
Justification:
codes and data http://ggihub.com/Taaiiiiii www.aiinnnnnnn/Deaae......o.
Our
would
be
released
in
Guidelines:
The answer NA means that paper does not include experiments requiring code.
Please see the NeurIPS code and data submission guidelines (https://nips.. cc/
ollicy for more details.
While encourage the release of code and data, understand that this might not be possible, so "No" an acceptable answer. Papers cannot be rejected simply for not including code, unless this is central to the contribution (e.g., for new open-source benchmark).
The instructions should contain the exact command and environment needed to run to reproduce the results. See the NeurlPS code and data submission guidelines (https
//nips.ccpublle pwwppppguuuiieeeeep Suci:///uuiieee/deeuuuubbiii
(CodeSubiisss wwtauubmmsssooni ssiiiissicccyy for more details.
The authors should provide instructions data access and preparation, including how to access the raw data, preprocessed data, intermediate data, and generated data, etc.
The authors should provide scripts to reproduce all experimental results for the new proposed method and baselines. only subset of experiments are reproducible, they should state which ones are omitted from the script and why.
At submission time, preserve anonymity, the authors should release anonymized versions (if applicable).
Providing as much information as possible in supplemental material (appended to the paper) recommended, but including URLs to data and code permitted.
6. Experimental Setting/Details
Question: Does the paper specify all the training and test details (e.g., data splits, hyper- parameters, how they were chosen, type of optimizer, etc.) necessary to understand the results?
Answer: [Yes]
Justification: Training, test details described in Sec. 4.1 and Sec. A.2
Guidelines:
The answer NA means that the paper does not include experiments.
The experimental setting should be presented in the core of the paper level of detail that is necessary to appreciate the results and make sense of them.
The full details can be provided either with the code, in appendix, as supplemental material.
7. Experiment Statistical Significance
Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?
Question:
Answer:
[No]
Justification:
We follow existing related works for the setting of error bar.
Guidelines:
The answer NA means that the paper does not include experiments.
The authors should answer "Yes" if the results are accompanied by error bars, confi-
dence intervals, statistical significance tests, least for the experiments that support the main claims of the paper.
The factors of variability that the error bars are capturing should be clearly stated (for
example, train/test split, initialization, random drawing of some parameter, or overall with given experimental conditions).
25

---

The method for calculating the error bars should be explained (closed form formula, call to a library function, bootstrap, etc.)
The assumptions made should be given (e.g., Normally distributed errors).
It should be clear whether the error bar is the standard deviation or the standard error of the mean.
OK to report 1-sigma error bars, but one should state it. The authors should preferably report a 2-sigma error bar than state that they have 96% CI, if the hypothesis of Normality of errors is not verified.
For asymmetric distributions, the authors should be careful not to show in tables or figures symmetric error bars that would yield results that out of range (e.g. negative error rates).
If error bars are reported in tables or plots, The authors should explain in the text how they were calculated and reference the corresponding figures or tables in the text.
8. Experiments Compute Resources
For each experiment, does the paper provide sufficient information on the com- puter resources (type of compute workers, memory, time execution) needed to reproduce the experiments?
Question:
Answer: [Yes]
Justification:
We have provided that in the supplementary.
Guidelines:
The answer NA means that the paper does not include experiments.
The paper should indicate the type of compute workers CPU or GPU, internal cluster, or cloud provider, including relevant memory and storage.
The paper should provide the amount of compute required for each of the individual experimental runs as well as estimate the total compute.
The paper should disclose whether the full research project required more compute than the experiments reported in the paper (e.g., preliminary failed experiments that didn't make it into the paper).
9. Code Of Ethics
Does the research conducted in the paper conform, in every respect, with the NeurIPS Code of Ethics
Question:
Answer: [Yes]
Justification:
We have reviewed that and claim we conform with the NeurlPS Code of Ethics.
Guidelines:
The answer NA means that the authors have not reviewed the NeurIPS Code of Ethics.
If the authors answer No, they should explain the special cireumstances that require deviation from the Code of Ethics.
The authors should make sure to preserve anonymity (e.g.. if there is special consid- eration due to laws or regulations their jurisdiction).
10. Broader Impacts
Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?
Answer: [Yes]
Justification: Similar existing human face reconstruction works, method, focusing recovering more realistic face textures under shadows from self and external occlusions, may also lead the ethical concerns about privacy, consent, and the spread of misinformation. Strengthening the management of the actual facial data collected may alleviate this situation. Guidelines:
The answer NA means that there no societal impact of the work performed.
If the authors answer NA or No, they should explain why their work has no societal impact why the paper does not address societal impact.
26

---

Examples of negative societal impacts include potential malicious unintended uses (e.g., disinformation, generating fake profiles, surveillance), faireess considerations (e.g., deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations.
The conference expects that many papers will be foundational research and not tied to particular applications, let alone deployments. However, if there is direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point out that generic algorithm for optimizing neural networks could enable people to train models that generate Deepfakes faster.
The authors should consider possible harms that could arise when the technology is being used intended and functioning correctly, harms that could arise when the technology is being used as intended but gives incorrect results, and harms following from (intentional unintentional) misuse of the technology.
If there are negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition to attacks, mechanisms for monitoring misuse, mechanisms monitor how system learns from feedback over time, improving the efficiency and accessibility of ML).
11. Safeguards
Does the paper deseribe safeguards that have been put in place for responsible release of data or models that have a high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)?
Question:
Answer: [NA]
Justification:
The paper does not have such risks as we only use public datasets.
Guidelines:
The answer NA means that the paper poses no such risks.
Released models that have high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere to usage guidelines or restrictions to access the model or implementing safety filters.
Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images.
We recognize that providing effective safeguards is challenging, and many papers do not require this, but we encourage authors to take this into account and make best faith effort.
12. Licenses for existing assets
Are the creators or original owners of assets (e.g., code, data, models), used in the paper, properly credited and are the license and terms of use explicitly mentioned and properly respected?
Question:
Answer: [Yes]
Justification:
We have cited them in the references.
Guidelines:
The answer NA means that the paper does use existing assets.
The authors should cite the original paper that produced the code package dataset.
The authors should state which version of the asset used and, possible, include URL.
The name of the license (e.g., CC-BY 4.0) should be included for each asset.
For scraped data from a particular source (e.g., website). the copyright and terms of service of that source should be provided.
If assets are released, the license, copyright information, and terms use in package should be provided. For popular datasets, paperswithcode com/ datasets has curated licenses for some datasets. Their licensing guide can help determine the license of dataset.
27

---

For existing datasets that are re-packaged, both the original license and the license of the derived asset (if has changed) should be provided.
If this information is available online, the authors are encouraged to reach out to the asset's s creators.
13. New Assets
Question: Are new assets introduced in the paper well documented and is the documentation provided alongside the assets?
Answer: [NA]
Justification: We use and cite existing datasets in this work. Other assets including codes/models would be released in
Guidelines:
The answer NA means that the paper does release assets.
Researchers should communicate the details of the dataset/code/model as part of their submissions via structured templates. This includes details about training, license, limitations, etc.
The paper should discuss whether and how consent obtained from people whose asset is used.
At submission time, remember to anonymize your assets (if applicable). You can either create anonymized URL or include an anonymized zip file.
14. Crowdsourcing and Research with Human Subjects
Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given to participants and sereenshots, if applicable, as well details about compensation (if any)?
Answer: [NA]
Justification:
We do include such experiments.
Guidelines:
The answer NA means that the paper does not involve crowdsoureing nor research with human subjects.
Including this information in the supplemental material is fine, but if the main contribu- tion of the paper involves human subjects, then as much detail as possible should be included in the main paper.
According to the NeurlPS Code of Ethics, workers involved in data collection, curation, other labor should be paid at least the minimum wage in the country of the data collector.
15. Institutional Review Board (IRB) Approvals or Equivalent for Research with Human Subjects
Question: Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or an equivalent approval/review based on the requirements of your country or institution) were obtained?
Answer: [NA]
Justification: All our experiments about human faces are based existing public datasets.
Guidelines:
The answer NA means that the paper does involve crowdsourcing nor research with human subjects.
Depending on the country in which research is conducted, IRB approval (or equivalent) may be required for any human subjects research. If you obtained IRB approval, you should clearly state this in the paper.
We recognize that the procedures for this may vary significantly between institutions and locations, and expect authors to adhere to the NeurlPS Code of Ethics and the guidelines for their institution.
For initial submissions, do not include any information that would break anonymity (if' applicable), such the institution conducting the review.
28


Reviews:
[{"review_id": "H4Ugo8C1QI", "reviewer": "Reviewer_WqKW", "summary": "The paper presents a method for reconstructing 3D face textures from monocular images in the presence of occlusions, both self-occlusions and occlusions by other scene elements such as hats. The paper identifies a key limitation in existing works -- they all get impacted by the lighting changes introduced by an occluder, as they assume globally consistent lighting. The paper proposes to model the scene using a combination of different lighting for the different regions of the face. The different regions are reconstructed without direct supervision. Additionally, a mask is used to separate out the occluder pixels. Modeling multiple light sources enables the method to model appearance effects due to occluders and thus enables higher quality reconstructions than the state of the art.", "questions": "Please refer to weaknesses.", "limitations": "Yes", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 2, "contribution": 2, "strengths": "The paper is very well motivated. It identifies a key problem and motivates the solution very well. \nThe method is novel and simple. \nThe results are impressive and outperform the chosen baselines. The reconstructed masks fit very well to the shadow boundaries", "weaknesses": "The main weakness is the lack of discussion and comparisons with existing works, and the lack of stronger baselines. \n\nThe paper does not mention existing research on dealing with occlusions. Here are a couple:\n- Robust Model-based Face Reconstruction through Weakly-Supervised Outlier Segmentation [CVPR23]\n- Occlusion-aware 3D Morphable Models and an Illumination Prior for Face Image Analysis [IJCV18]\nBoth these papers jointly perform 3D reconstruction and solve for occluders. They might not use texture optimization and instead perform per-vertex color optimization, but their techniques should be trivially extended for texture optimization. Comparisons to these baselines makes it impossible to evaluate this submission. \n\nIn addition, there are other strong baselines that were not considered. Generative models, such as those introduced by \"Face De-occlusion using 3D Morphable Model and Generative Adversarial Network\" [ICCV19] can remove occlusions from portrait images. Can the output of those methods directly be used with NextFace for high-quality reconstructions? \n\nThe other weakness relates to the exposition. The technical details of the paper were not easy to read. I did not understand how many number of masks are used, what 't' refers to in L128, how many such 't' are used. How are all the hyperparameters, such as 'n', n_L', m, epsilon, etc., chosen? I did not find a discussion. The paper is not reproducible in its current state.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "**Q1: Lack of discussion and comparisons with 3D face reconstruction works dealing with de-occlusion, such as [1, 2].**\n\n[1] Robust Model-based Face Reconstruction through Weakly-Supervised Outlier Segmentation (CVPR'23)\n\n[2] Occlusion-aware 3D Morphable Models and an Illumination Prior for Face Image Analysis (IJCV'18)\n\n**A1:** Thank for your reminding. References [1] and [2] propose different methods to predict masks to remove external occlusions, where they optimize 3D Morphable Model(3DMM) according to the remained regions to reconstruct de-occluded faces. We will discuss these methods in more detail in the revised version.\n\nHowever, de-occlusion focuses on removing external occlusions, while **the shadows are not just occlusions**. A shadow region is actually the face under another illumination, which may include useful texture information. Simply treating shadowed regions as occlusions and removing them using de-occlusion methods may lead to the loss of face texture information in these areas. Our method, which integrates multiple lighting conditions, is more suitable for recovering textures from faces with shadows.\n\nHere, we present a comparison with the recent open-sourced de-occlusion method [1] by extracting its 3D textures and evaluating them using the texture quality metrics as mentioned in Section 4.1 of the main paper. The quantitative results, evaluated on single images from VoxCeleb2, are presented below.\n|  | Deocclu[1] | Ours |\n|--|--|--|\n|PSNR $\\uparrow$ | 25.13 |**29.22** |\n|SSIM $\\uparrow$ | 0.88 | **0.91** |\n|LPIPS $\\downarrow$ | 14.09 | **6.36** |\n\nWe also conduct comparisons on Video sequences from VoxCeleb2:\n|  | Deocclu[1] | Ours |\n|--|--|--|\n|PSNR $\\uparrow$  | 25.03 |**29.15** |\n|SSIM $\\uparrow$  | 0.88 | **0.91** |\n|LPIPS $\\downarrow$ | 14.18 | **6.92** |\n\nOur method demonstrates superior performances. We also provide qualitative comparisons in Fig. 18 of the rebuttal PDF. Our method constructs more realistic results. For example, results from [1] miss details such as beards, while our method preserves them.\n\n**Q2: Can we use the de-occluded results of generative de-occlusion model such as [3] on NextFace for high quality reconstruction?**\n\n[3] Face De-occlusion using 3D Morphable Model and Generative Adversarial Network (ICCV'19)\n\n**A2:** Reference [3] trains a generative model to produce de-occluded face images from input and initial 3DMM synthesis images. We will discuss this in more detail in the revised version.\n\nHowever, [3] does not provide its source code, making a direct comparison unavailable at this time. Additionally, as stated in Sec. 5.1 of [3], the model is trained on synthetic data constructed by layering common non-face objects onto faces. This approach focuses on learning how to remove these objects, without any consideration about shadows.\n\nAs we mainly discuss about **the influence of shadows from self and external occlusions** in this work, a closer idea is using 2D shadow removal methods mentioned by **Q3-Reviewer SyuP**. Such works desire to introduce generative model to recover high quality images free from shadows. \n\nWe conducted experiments using the output of the shadow removal method [4] as input to CPEM, D3DFR, NextFace, NextFace*, and FFHQ-UV to evaluate their ability to acquire accurate textures without shadows. The quantitative comparisons are presented in **Q3-Reviewer SyuP**, and we also provide qualitative comparisons in Fig. 17 of the rebuttal PDF.\n\nOur results show that introducing the output of such generative models to the texture modeling baselines still yields inferior results compared to our method.\n\n[4] Unsupervised Portrait Shadow Removal via Generative Priors (MM 2021)\n\n**Q3: The technical details of the paper were not easy to read. I did not understand how many number of masks are used, what 't' refers to in L128, how many such 't' are used. How are all the hyper-parameters, such as 'n', n_L', m, epsilon, etc., chosen?**\n\n**A3:** Sorry about the confusing parts. The number of masks are actually changing according to the illumination complexity of the image. As illustrated in Line 117~124 of the paper, we initialize $n$ separate lighting conditions, where $f(\\cdot)$ predicts one mask for each Lighting condition. The $n$ masks make up $M_N$ in Fig.2. Then, the parameters of $n$ lighting conditions and $f(\\cdot)$ are optimized to construct the face image, where some lighting conditions not existing in the face image will get smaller mask areas as illustrated in Fig.2 of the paper. \n\nThen, in ACE, we drop such lighting conditions and masks with small mask areas, as explained in Line 142~147 of the paper. In this way, $n_L$ masks in $M_N$ are remained, which make up $M_L$. We can see that $n_L$ is a adaptive number less than $n$. \n\nWe select $n=5$ in this work. It means there would be 5 masks in $M_N$ at the beginning. Where $n_L<=n$ would be decided according to the subsequent optimization. Corresponding discussions could be found in **Q1-Reviewer NbDK**.\n\n$m$ mentioned in Line 142 to 157 is not a hyper-parameter, it represents the mask value between 0 ~ 1 at a pixel position, e.g., $m=0$ at the black regions of the masks.\n$\\epsilon$ is a threshold to filter out the masks with quite small areas. We set it as 0.17 in this work.\n\n$t$ is decided by the number of frames in the input image/video sequence. For the single image reconstruction, $t$ is set as a fixed 0 value, where it would be $i/k$ for the $i_{th}$ frame of a $k$ frames video sequences. In this work, we mainly conduct comparisons on 8-frame sequences as claimed in Line 203, page 7 of the paper.\nWe will add more details about the mentioned contents in the revised version.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "t0m25FL1fY", "reviewer": "Reviewer_SyuP", "summary": "This work tackles the problem of external shadows in single image face reconstruction. Specifically, when the input image contains foreign shadows, this often affects the quality of the estimated facial texture, as the external shadows often become baked into the texture or leave behind undesirable artifacts in the shadow region. The paper proposes a comprehensive solution to this problem, including a way to decouple the face image into the result of multiple lights as well as an Adaptive Condition Estimation (ACE) strategy to select which lights are present in the image. The paper further proposes multiple human face priors, such as a global prior to ensure the face texture hue is consistent with the initialization from a 3DMM model and a local prior to ensure that the smoothness of the face texture is similar to the initialization. Experiments demonstrate that the method is able to improve rendering performance on in-the-wild face images by rendering both source images with external shadows and target images without external shadows. The major improvements on target images without external shadows shows that the method is able to produce accurate facial textures that are minimally impacted by the presence of these shadows. Qualitative ablations are also provided to further aid in evaluating the method.", "questions": "Please see the experiments I suggested above, especially with regard to quantitative evaluations for ablations, evaluations on images with more diverse foreign (external) shadows, and comparing with the simple baseline of running a foreign shadow removal method on the images beforehand. I will be carefully reviewing the rebuttal as well as the opinions of the other reviewers to decide if I would like to change my rating. Please also factor in missing citations, as the two areas I mentioned are very relevant to this work.", "limitations": "The authors have at least made an attempt to include a limitations section, although more analysis on failure cases would be helpful. For example, does the method fail when the external shadow covers most of the face? There is no section on potential negative societal impact, although there are certainly some potential concerns with face reconstruction methods, as with any face modeling work. The potential to edit the face and produce fake content is always present.", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 2, "contribution": 3, "strengths": "The improvement in rendering performance on target images (w/o external shadows) clearly shows that the method is able to minimize the impact of external shadows from the source image. The qualitative results further support this conclusion, as the facial textures do not contain external shadow artifacts. \n\nQualitative ablations are available to help assess the impact of each technical contribution in this work. \n\nThis work is sufficiently novel since it enables more accurate face reconstruction under the condition of external occlusions from the source image, which is a heavily understudied problem. It also goes beyond traditional work in this area since it considers the scenario that there could be more than one light in the image and proposes a method to estimate the set of lights illuminating the face.", "weaknesses": "For the ablation studies, it would be much more convincing to have tables with quantitative results demonstrating the margin of improvement of each component. Simply picking a few favorable qualitative examples is easy and not convincing. \n\nIn almost all examples in this work, the foreign (external) shadows involved are caused by hats. What about other types of foreign shadows caused by tree leaves, pens, paper, hands, etc? How are the resulting face textures in these situations? It would be nice to compare with the baselines on some images with more diverse foreign shadows. If it is difficult to find such images in the wild, you can find a small test set of 100 or so images from the paper \"Portrait Shadow Manipulation (SIGGRAPH 2020)\" with diverse foreign shadow shapes. \n\nSome important citations are missing from this work, especially in the face relighting and portrait shadow removal domains. There are several face relighting methods that involve intrinsic decomposition of faces and illumination modeling, some of which involve ray tracing to handle self shadows: \n\n1. SfSNet : Learning Shape, Reflectance and Illuminance of Faces in the Wild (CVPR 2018)\n2. Face Relighting with Geometrically Consistent Shadows (CVPR 2022)\n3. Learning Physics-guided Face Relighting under Directional Light (CVPR 2020)\n\nIn addition, the shadow removal domain is highly relevant to this work since a simple solution to this problem would be to perform a preprocessing step to remove the foreign (external) shadows from the image first using a portrait shadow removal method before performing face reconstruction. These methods should be cited and discussed in the paper and the authors should verify that their method outperforms this simple baseline: \n\n1. Portrait Shadow Manipulation (SIGGRAPH 2020)\n2. Unsupervised Portrait Shadow Removal via Generative Priors (MM 2021)\n3. Blind Removal of Facial Foreign Shadows (BMVC 2022)", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "**Q1: Quantitative evaluations for ablations.**\n\n**A1:** Thank you for your suggestion. Here, we provide the quantitative ablation study of our proposed components below, following the same setting as Sec. 4.4 of the main paper. GP, LP, and HP denote our proposed constraints $L_{GP}$, $L_{LP}$, and $L_{HP}$ mentioned in Sec. 3.4, where Light and Occlusion denote the $f(\\cdot)$ for combining multiple lighting conditions, and $g(\\cdot)$ to remove the direct external occlusion, as claimed in Sec. 3.3, respectively. \n\nHere is the ablation study for constraints:\n\n|  | NA | GP | GP+LP|GP+LP+HP (ours)|\n|--|--|--|--|--|\n|PSNR $\\uparrow$ |27.20|28.81| 28.82 |29.22 |\n|SSIM $\\uparrow$ |0.88|0.89| 0.90 | 0.91 |\n|LPIPS $\\downarrow$ |8.34|6.78| 6.49 | 6.36 |\n\nHere is the ablation study for the proposed $f(\\cdot)$ and $g(\\cdot)$.\n\n|  | NA | + Light ($f(\\cdot)$) | + Occlusion ($g(\\cdot)$) \n|--|--|--|--|\n|PSNR $\\uparrow$ | 25.19|27.52 |29.22\n|SSIM $\\uparrow$ |0.87 |0.89 | 0.91\n|LPIPS $\\downarrow$ | 9.16|7.75 | 6.36\n\nWe can see that each component has contribution to the final performances.\n\n**Q2: Missing citations and Discussions about relighting works [1, 2, 3].**\n\n[1]SfSNet : Learning Shape, Reflectance and Illuminance of Faces in the Wild (CVPR 2018)\n\n[2]Face Relighting with Geometrically Consistent Shadows (CVPR 2022)\n\n[3]Learning Physics-guided Face Relighting under Directional Light (CVPR 2020)\n\n**A2:** [1] proposes a framework for decomposing face attributes from in-the-wild images through physics-based rendering, while [3] predicts residuals to enhance diffuse relighting performance. [2] introduces ray-tracing to model geometrically consistent self-occlusion shadows for relighting. However, these methods primarily focus on self-occlusion shadows rather than external shadows. Due to character limitations, we will discuss these approaches in more detail in the revised version.\n\n**Q3: Evaluations on images with more diverse foreign (external) shadows.**\n\n[4] Portrait Shadow Manipulation (SIGGRAPH 2020)\n\n**A3:** Thank you for your suggestion, we introduce 15 pairs of data (30 images) from [4] for evaluation under more diverse external shadows. The quantitative results are presented below, where some qualitative results are presented in Fig. 16 of the rebuttal PDF. We can see that our method still outperforms other methods under faces with diverse shadows.\n|  |CPEM|D3DFR|NextFace| NextFace* | FFHQ-UV | Ours\n|--|--|--|--|--|--|--|\n|PSNR $\\uparrow$ | 23.34| 25.02 |21.83| 21.93 | 24.00 | **28.97**\n|SSIM $\\uparrow$ |0.84 |0.87 | 0.84  | 0.84 | 0.88 | **0.92**\n|LPIPS $\\downarrow$ | 9.31|9.11 | 9.48 | 9.69 | 7.82 | **7.00**\n\n**Q4: Comparing with the simple baseline of running a foreign shadow removal method such as [4, 5, 6] on the images beforehand.**\n\n[5] Unsupervised Portrait Shadow Removal via Generative Priors (MM 2021)\n\n[6] Blind Removal of Facial Foreign Shadows (BMVC 2022)\n\n**A4:** It is an interesting idea. To validate its effectiveness, we introduce the shadow removal work [5] to pre-process the face images with shadows because it is the most recent shadow-removal work with a pre-trained model.\nHere are the quantitative results evaluated on data from [4].\n|  |CPEM|D3DFR|NextFace| NextFace* | FFHQ-UV | Ours\n|--|--|--|--|--|--|--|\n|PSNR $\\uparrow$ | 25.56| 25.60 |23.49| 25.47 | 26.19 | **28.97**\n|SSIM $\\uparrow$ |0.86 |0.88 | 0.86  | 0.87 | 0.90 | **0.92**\n|LPIPS $\\downarrow$ |8.56|8.68 | 9.09 | 6.41 | 7.02 | **7.00**\n\nHere are quantitative the results evaluated on single images from VoxCeleb2.\n\n|  |CPEM|D3DFR|NextFace| NextFace* | FFHQ-UV | Ours\n|--|--|--|--|--|--|--|\n|PSNR $\\uparrow$ | 24.84| 26.78 |23.77| 24.51 | 25.35 | **29.22**\n|SSIM $\\uparrow$ |0.87 |0.90 | 0.85  | 0.86 | 0.91 | **0.91**\n|LPIPS $\\downarrow$ |10.20|7.93 | 10.47 | 9.64 | 7.62 | **6.36**\n\nWe also provide qualitative comparisons in Fig. 17 of the rebuttal PDF. The pre-processing with the shadow removal method indeed improves the performances of baselines, while our method still outperforms them. \n\nFrom the qualitative results, we observe that the shadow removal model cannot fully remove the external shadows in cases where the external shadows cover relatively large regions. Although modifying the shadow removal model to be more powerful may further improve the final performances, it goes beyond the range of this work. We can explore it in the future.\n\n**Q5: More analysis on failure cases. For example, does the method fail when the external shadow covers most of the face?**\n\n**A5:** As shown in Fig. 16 of the rebuttal PDF, our method performs well even under external shadows covering most of the face. We present visualizations of our inferior cases in Fig. 19 of the rebuttal PDF. As mentioned in the Limitations section of the main paper, our primary limitation is the initialization with AlbedoMM. For faces with many high-frequency details, such as wrinkles, our method may lose these details during reconstruction. Replacing AlbedoMM with more powerful face representations could address this issue. We plan to explore this further in future work.\n\n**Q6: There is no section on potential negative societal impact.**\n\n**A6:** Sorry for the lack of such discussion. Similar as existing human face reconstruction works, our method, focusing on recovering more realistic face textures under shadows from self and external occlusions, may also lead to the ethical concerns about privacy, consent, and the spread of misinformation. We will discuss about it carefully in the revised version.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "w0EKZ8Ed4J", "reviewer": "Reviewer_W5JR", "summary": "This paper proposes a method to recover face albedo by disentangling the input image into an albedo and shading maps (called \"light conditions\"). The shading maps are generated by a network which are combined with the albedo texture (generated from AlbedoMM) that is rendered under a set of $n$ lighting conditions represented using spherical harmonics. The networks for predicting the shading maps, spherical harmonics and the texture are optimized through a photometric loss along with regularizations. In order to generate meaningful masks, a binarization constraint is added on the generated masks along with an area constraint. While the final renders are significantly better than prior work, two crucial comparisons are missing i.e against the Haar-wavelet basis (https://graphics.stanford.edu/papers/allfreqmat/) and the Albedo from AlbedoMM", "questions": "How many bands of SH are optimized? Did you investigate optimizing higher number of bands instead of generating the masks (again, as done in: http://graphics.stanford.edu/papers/allfreq/allfreq.pdf", "limitations": "Yes.", "rating": 6, "confidence": 3, "soundness": 2, "presentation": 3, "contribution": 3, "strengths": "1) From my perspective, the central contribution of this paper is a somewhat 'neural' spherical harmonics (SH) representation, where the 'neural' masks help model sharp illumination effects such as shadows which SH fails to capture. This is certainly an interesting direction to explore\n\n2) The paper is well written\n\n3) Quantitative and qualitative results are better than prior work (especially in albedo). However, some crucial comparisons are missing.", "weaknesses": "1) The authors have not compared against optimizing in haar-wavelets (https://graphics.stanford.edu/papers/allfreqmat/) which are designed specifically to model sharp illumination dependent effects. Without this comparison, it is hard to asses the improvements the proposed model offers over classical representations. I understand that such an optimization may be compute intensive, but it is necessary.\n\n2) There are no results shown of the initial AlbedoMM texture the texture map is initialized from. AlbedoMM already gives a relatively uniform texture map that is free from lighting artifacts, it is unclear what additional benefits texture optimization yields.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "**Q1: The authors have not compared against optimizing in haar-wavelets[1] which are designed specifically to model sharp illumination dependent effects.**\n\n[1] Triple Product Wavelet Integrals for All-Frequency Relighting\n\n**A1:** Thank you for your suggestion. Please note that our contribution lies not in proposing a universal neural illumination model for all scenes, but in decoupling lights to aid in recovering more accurate textures in 3D face reconstruction.\n\nReference [1] offers a solution for modeling environment illumination using haar wavelets for relighting. However, [1] focuses on efficient rendering, with all geometrical and texture information pre-processed and fixed. This differs significantly from the 3D face reconstruction setting, where geometrical, texture, and illumination parameters are co-optimized, making illumination optimization more challenging.\n\nTo our knowledge, haar-wavelet modeling of illumination has not yet been applied to 3D face reconstruction, and no open-source codes are available. To validate this idea, we create a baseline by replacing the environment map modeling in NextFace and NextFace* using Spherical Harmonics (SH) with haar wavelets.\n\nThe optimization of wavelets is considerably slower than SH, making it difficult to conduct quantitative comparisons on the full datasets. Therefore, we conducted a comparison on a subset of 5 pairs of single images from VoxCeleb2. The quantitative comparisons are presented below:\n\n|  | NextFace | NextFace(haar) | NextFace* | NextFace*(haar) |Ours\n|--|--|--|--|--|--|\n|PSNR $\\uparrow$ | 21.82 | 21.84 | 22.12 | 22.12 | **26.66** |\n|SSIM $\\uparrow$ | 0.83| 0.84 | 0.84 | 0.84 | **0.90** |\n|LPIPS $\\downarrow$ | 11.12| 11.37 | 10.51 | 11.30 | **6.29**|\n\nWe observe that modeling global illumination with haar wavelets has limited improvements over the original SH on PSNR for our face reconstruction task. Although performance might be enhanced by modifying the wavelets or increasing the size of the environment map, such modifications are beyond the scope of this paper and would incur significant time costs. Optimizing such a 64 × 64 environment map from wavelets takes approximately 28 minutes on a 256 × 256 image, whereas our method takes about 340 seconds. Therefore, our method remains more efficient and effective at this time.\n    \n**Q2: There are no results shown of the initial AlbedoMM texture the texture map is initialized from. AlbedoMM already gives a relatively uniform texture map that is free from lighting artifacts, it is unclear what additional benefits texture optimization yields.**\n\n**A2:** As demonstrated in Fig. 2 and Alg. 1 of the main paper, we use AlbedoMM to initialize the textures in Stage 2, while the textures are directly optimized in Stage 3. Therefore, the textures from Stage 2 are precisely the AlbedoMM textures.\n\nIn Section A.4 of the appendix, on page 12, we have provided a comparison between the AlbedoMM textures (Stage 2) and the final textures (Stage 3). As shown in Fig. 10 of the appendix, the texture optimization adds more details, such as beards, to the AlbedoMM textures, resulting in a more realistic reconstruction.\n\n**Q3: How many bands of SH are optimized? Did you investigate optimizing higher number of bands instead of generating the masks again?**\n\n**A3:** In this work, we follow NextFace [1] by using 9-band spherical harmonics (SH) to model local illumination. Additionally, we provide quantitative results of our method modeled with a single global SH using 9, 12, 15, and 18 bands, by removing the $f(\\cdot)$ and $g(\\cdot)$.\n\n| B | 9 | 12 | 15|18|Ours\n|--|--|--|--|--|--|\n|PSNR $\\uparrow$ |25.19 | 25.26| 25.27 | 25.34 |**29.22**\n|SSIM $\\uparrow$ | 0.87 | 0.87 | 0.87 | 0.87 |**0.91**\n|LPIPS $\\downarrow$ | 9.16 | 9.23 | 9.22 | 9.10 |**6.36**\n\nWe observe that increasing the number of bands in a single global SH yields quite limited improvements. A possible reason is that the external occluded shadows on human faces represent drastic illumination changes in relatively small areas, which may not be appropriately modeled as a single global SH during optimization.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "PsSHaDXZzH", "reviewer": "Reviewer_NbDK", "summary": "The paper presents a new face texture modeling framework by learning to decouple environmental illumination into multiple light conditions with neural representations. The proposed method can reconstruct a cleaner texture compared to previous work that use a single environment map. The proposed method is neat and reasonable. The paper is well-written.", "questions": "See Weaknesses.", "limitations": "The limitations are discussed.", "rating": 7, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "* The first method to tackle the challenging but important problem of face reconstruction which is neglected by most existing works.\n* The method is novel and neat. After reading the introduction and going through Fig.1, I believe the proposed method can well solve the problem.\n* The paper is well-written and easy to follow.", "weaknesses": "* Line 119 says the n lighting coefficients are initialized differently. Can you provide some insights behind these design choices? Is the method sensitive to the initialization of lighting?\n\n* In the proposed Human Prior Constraint, the rendered face is encouraged to be similar to a specific face in the FaceNet. Why not use the common perceptual loss as D3DFR?\n\n* The texture of D3DFR in Figure 3 is missing. I know it reconstructs the BFM texture, which is not a diffuse albedo map. But I think it should also be provided. In addition, the FFHQ-UV's texture is presented in UV space while NextFace and the proposed methods' texture are presented in image space, I suggest presenting them all in UV space or all in image space.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "**Q1: Line 119 says the n lighting coefficients are initialized differently. Can you provide some insights behind these design choices? Is the method sensitive to the initialization of lighting?**\n\n**A1:** Sure. The lighting coefficients are just simply initialized as a uniform distribution between -1 to 1 as illustrated in Line 119, Sec. 3.3 of the paper. Such operation can provide $n$ initialized local illuminations ranging from dark to light. During the later optimization of Stage 2, the $n$ groups of lighting coefficients are optimized towards actual illumination. Coefficients that deviate significantly from the existing illumination in the images will have masks with smaller areas, as shown in the visualized $M_N$ of Fig.2 of the main paper. These coefficients with smaller masks are deemed less important and are removed according to a predefined threshold $\\epsilon$ in ACE.\n\nHere, we also provide a ablation study for the number of $n$ used for initialization of lighting conditions below. Single images from VoxCeleb2 are used for evaluation.\n\n| n | 3 | 5 (we use) | 7  | 9 |\n|--|--|--|--|--|\n|PSNR $\\uparrow$ | 28.49 |29.22 |29.14 | 29.16|\n|SSIM $\\uparrow$ | 0.90 | 0.91 |0.91 | 0.91|\n|LPIPS $\\downarrow$ | 6.37 | 6.36 | 6.37 | 6.39 |\n\nWe observe that $n=3$ produces sub-optimal results, likely because the number of lighting condition candidates is insufficient to model images with complex illuminations. In contrast, $n=3, 5, 9$ yield good and similar results as there are enough initial lighting conditions, and any redundant ones are removed. The result for $n=5$ is slightly better. While introducing larger $n$ and further adjusting the hyper-parameters might improve performance, it would also increase the optimization burden. Therefore, we use $n=5$ in this work.\n\n**Q2: In the proposed Human Prior Constraint, the rendered face is encouraged to be similar to a specific face in the FaceNet. Why not use the common perceptual loss as D3DFR?**\n\n**A2:** Please note that the rendered faces used to calculate the Human Prior Constraint (HP) are $I_{Rs}$. As illustrated in Fig. 2 of the main paper, $I_{Rs}$ includes rendered faces under multiple predicted lighting conditions. We cannot apply perceptual loss to $I_{Rs}$ because we do not have corresponding ground truth images under the multiple decoupled lighting conditions. However, HP does not require such ground truths. Perceptual loss can indeed be applied between the final rendered result $I_{out}$ and input image $I_{in}$.\nBelow, we present a comparison between the perceptual loss implementation and the HP we proposed. We can see that HP still has better performances, which can confirm it provides more effective constraints for the textures through rendered faces under multiple lighting conditions.\n\n|  | Perceptual Loss | HP(ours) |\n|--|--|--|\n|PSNR $\\uparrow$ | 28.72 |**29.22** |\n|SSIM $\\uparrow$ | 0.90 | **0.91** |\n|LPIPS $\\downarrow$ | 6.46 | **6.36** |\n\n**Q3: The texture of D3DFR in Figure 3 is missing. In addition, the FFHQ-UV's texture is presented in UV space while NextFace and the proposed methods' texture are presented in image space, I suggest presenting them all in UV space or all in image space.**\n\n**A3:** Thank you for your suggestion. We agree that the texture presentation space needs to be consistent. To achieve this, we transform all UV-textures to the image space. This allows textures from D3DFR and CPEM to be visualized as colored faces, even if they do not have UV-textures. Fig. 16 and 17 in the rebuttal PDF are provided in this space. We will also update the texture images in the main paper accordingly in the revised version.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:
We thank all reviewers for your valuable comments. We are inspired that all reviewers recognize the sufficient motivation and good performance of our method. We apologize for any confusion caused by missing comparisons, citations, unclear definitions, and other issues. In this rebuttal, we make reply to each concern carefully, point by point.
All quantitative comparisons are based on the texture quality metric for target images as defined in Line 198-203 of the main paper. We also present qualitative results in the attached rebuttal PDF. We hope our response addresses your concerns well. Please feel free to comment if you have any further questions or suggestions about this work.

Meta-review: After the rebuttal and extensive reviewer-author discussion, all four reviewers proposed acceptance, thus, the AC recommends acceptance. The AC strongly encourages the authors to include the missing comparisons, citations, and the rest of the rebuttal/discussion material in the final version of the paper.
Decision: Accept (poster)