EUROGRAPHICS 2023 / K. Myszkowski and M. Nießner(Guest Editors)

Volume 42 (2023), Number 2

Makeup Extraction of 3D Representation
via Illumination-Aware Image Decomposition(Supplementary Material)
Xingchao Yang,1,2

Takafumi Taketomi,11 CyberAgent, AI Lab, Japanand Yoshihiro Kanamori22 University of Tsukuba, Japan

In this supplementary material, we provide more detailed example results of our method. We also present the figures without adjusting thecontrast of the specular reconstruction mentioned in the main paper.
The example results are presented from Fig. 1 to Fig. 8. The first rows show the input images and the disentangled coarse materials ofreconstructed 3D faces. The second rows show the fully reconstructed 3D faces and the disentangled coarse materials in the UV space. The
third rows demonstrate the refined facial materials and extracted bare skin and makeup. The final rows show the fully reconstructed renderings
and the overlaid rendering images (from left to right: bare skin only, bare skin plus makeup, bare skin multiplied by diffuse shading, and plus
specular reconstruction).

© 2023 The Author(s)Computer Graphics Forum © 2023 The Eurographics Association and John
Wiley & Sons Ltd. Published by John Wiley & Sons Ltd.

Yang et al. / Makeup Extraction of 3D Representation via Illumination-Aware Image Decomposition

Input image 𝐼𝑖𝑛

Diff. albedo 𝐷𝑐

Diff. shading 𝐷𝑐𝑆

Diff. recons. 𝑅𝑐𝑑

Spec. albedo 𝑆𝑐

Spec. shading 𝑆𝑐𝑆Spec. recons. 𝑅𝑐𝑠

Recons. image 𝐼𝑟𝑒𝑐

Diff. albedo 𝐷𝑐

Diff. shading 𝐷𝑐𝑆

Diff. recons. 𝑅𝑐𝑑

Spec. albedo 𝑆𝑐

Spec. shading 𝑆𝑐𝑆Spec. recons. 𝑅𝑐𝑠

Unwrapped texture 𝑇𝑚

Completed texture 𝑇𝑓

Diff. albedo 𝐷𝑓𝑏
Bare skin 𝐷𝑚𝑚Makeup (1 − 𝐴) ⊙ 𝐷𝑚

Diff. shading 𝐷𝑓𝑆Spec. recons. 𝑅𝑓𝑠

Rendered makeup face

Bare skin

MakeupDiffuseSpecularFigure 1: Outputs for each step and overlaid rendering images.

© 2023 The Author(s)Computer Graphics Forum © 2023 The Eurographics Association and John Wiley & Sons Ltd.

Yang et al. / Makeup Extraction of 3D Representation via Illumination-Aware Image Decomposition

Input image 𝐼𝑖𝑛

Diff. albedo 𝐷𝑐

Diff. shading 𝐷𝑐𝑆

Diff. recons. 𝑅𝑐𝑑

Spec. albedo 𝑆𝑐

Spec. shading 𝑆𝑐𝑆Spec. recons. 𝑅𝑐𝑠

Recons. image 𝐼𝑟𝑒𝑐

Diff. albedo 𝐷𝑐

Diff. shading 𝐷𝑐𝑆

Diff. recons. 𝑅𝑐𝑑

Spec. albedo 𝑆𝑐

Spec. shading 𝑆𝑐𝑆Spec. recons. 𝑅𝑐𝑠

Unwrapped texture 𝑇𝑚

Completed texture 𝑇𝑓

Diff. albedo 𝐷𝑓𝑏
Bare skin 𝐷𝑚𝑚Makeup (1 − 𝐴) ⊙ 𝐷𝑚

Diff. shading 𝐷𝑓𝑆Spec. recons. 𝑅𝑓𝑠

Rendered makeup face

Bare skin

MakeupDiffuseSpecularFigure 2: Outputs for each step and overlaid rendering images.

© 2023 The Author(s)Computer Graphics Forum © 2023 The Eurographics Association and John Wiley & Sons Ltd.

Yang et al. / Makeup Extraction of 3D Representation via Illumination-Aware Image Decomposition

Input image 𝐼𝑖𝑛

Diff. albedo 𝐷𝑐

Diff. shading 𝐷𝑐𝑆

Diff. recons. 𝑅𝑐𝑑

Spec. albedo 𝑆𝑐

Spec. shading 𝑆𝑐𝑆Spec. recons. 𝑅𝑐𝑠

Recons. image 𝐼𝑟𝑒𝑐

Diff. albedo 𝐷𝑐

Diff. shading 𝐷𝑐𝑆

Diff. recons. 𝑅𝑐𝑑

Spec. albedo 𝑆𝑐

Spec. shading 𝑆𝑐𝑆Spec. recons. 𝑅𝑐𝑠

Unwrapped texture 𝑇𝑚

Completed texture 𝑇𝑓

Diff. albedo 𝐷𝑓𝑏
Bare skin 𝐷𝑚𝑚
Makeup (1 − 𝐴) ⊙ 𝐷𝑚

Diff. shading 𝐷𝑓𝑆Spec. recons. 𝑅𝑓𝑠

Rendered makeup face

Bare skinMakeup

DiffuseSpecularFigure 3: Outputs for each step and overlaid rendering images.

© 2023 The Author(s)Computer Graphics Forum © 2023 The Eurographics Association and John Wiley & Sons Ltd.

Yang et al. / Makeup Extraction of 3D Representation via Illumination-Aware Image Decomposition

Input image 𝐼𝑖𝑛

Diff. albedo 𝐷𝑐

Diff. shading 𝐷𝑐𝑆

Diff. recons. 𝑅𝑐𝑑

Spec. albedo 𝑆𝑐

Spec. shading 𝑆𝑐𝑆Spec. recons. 𝑅𝑐𝑠

Recons. image 𝐼𝑟𝑒𝑐

Diff. albedo 𝐷𝑐

Diff. shading 𝐷𝑐𝑆

Diff. recons. 𝑅𝑐𝑑

Spec. albedo 𝑆𝑐

Spec. shading 𝑆𝑐𝑆Spec. recons. 𝑅𝑐𝑠

Unwrapped texture 𝑇𝑚

Completed texture 𝑇𝑓

Diff. albedo 𝐷𝑓𝑏Bare skin 𝐷𝑚𝑚Makeup (1 − 𝐴) ⊙ 𝐷𝑚

Diff. shading 𝐷𝑓𝑆Spec. recons. 𝑅𝑓𝑠

Rendered makeup face

Bare skin

MakeupDiffuseSpecularFigure 4: Outputs for each step and overlaid rendering images.

© 2023 The Author(s)Computer Graphics Forum © 2023 The Eurographics Association and John Wiley & Sons Ltd.

Yang et al. / Makeup Extraction of 3D Representation via Illumination-Aware Image Decomposition

Input image 𝐼𝑖𝑛

Diff. albedo 𝐷𝑐

Diff. shading 𝐷𝑐𝑆

Diff. recons. 𝑅𝑐𝑑

Spec. albedo 𝑆𝑐

Spec. shading 𝑆𝑐𝑆Spec. recons. 𝑅𝑐𝑠

Recons. image 𝐼𝑟𝑒𝑐

Diff. albedo 𝐷𝑐

Diff. shading 𝐷𝑐𝑆

Diff. recons. 𝑅𝑐𝑑

Spec. albedo 𝑆𝑐

Spec. shading 𝑆𝑐𝑆Spec. recons. 𝑅𝑐𝑠

Unwrapped texture 𝑇𝑚

Completed texture 𝑇𝑓

Diff. albedo 𝐷𝑓𝑏Bare skin 𝐷𝑚𝑚Makeup (1 − 𝐴) ⊙ 𝐷𝑚

Diff. shading 𝐷𝑓𝑆Spec. recons. 𝑅𝑓𝑠

Rendered makeup face

Bare skin

MakeupDiffuseSpecularFigure 5: Outputs for each step and overlaid rendering images.

© 2023 The Author(s)Computer Graphics Forum © 2023 The Eurographics Association and John Wiley & Sons Ltd.

Yang et al. / Makeup Extraction of 3D Representation via Illumination-Aware Image Decomposition

Input image 𝐼𝑖𝑛

Diff. albedo 𝐷𝑐

Diff. shading 𝐷𝑐𝑆

Diff. recons. 𝑅𝑐𝑑

Spec. albedo 𝑆𝑐

Spec. shading 𝑆𝑐𝑆Spec. recons. 𝑅𝑐𝑠

Recons. image 𝐼𝑟𝑒𝑐

Diff. albedo 𝐷𝑐

Diff. shading 𝐷𝑐𝑆

Diff. recons. 𝑅𝑐𝑑

Spec. albedo 𝑆𝑐

Spec. shading 𝑆𝑐𝑆Spec. recons. 𝑅𝑐𝑠

Unwrapped texture 𝑇𝑚

Completed texture 𝑇𝑓

Diff. albedo 𝐷𝑓𝑏
Bare skin 𝐷𝑚𝑚Makeup (1 − 𝐴) ⊙ 𝐷𝑚

Diff. shading 𝐷𝑓𝑆Spec. recons. 𝑅𝑓𝑠

Rendered makeup face

Bare skin

MakeupDiffuseSpecularFigure 6: Outputs for each step and overlaid rendering images.

© 2023 The Author(s)Computer Graphics Forum © 2023 The Eurographics Association and John Wiley & Sons Ltd.

Yang et al. / Makeup Extraction of 3D Representation via Illumination-Aware Image Decomposition

Input image 𝐼𝑖𝑛

Diff. albedo 𝐷𝑐

Diff. shading 𝐷𝑐𝑆

Diff. recons. 𝑅𝑐𝑑

Spec. albedo 𝑆𝑐

Spec. shading 𝑆𝑐𝑆Spec. recons. 𝑅𝑐𝑠

Recons. image 𝐼𝑟𝑒𝑐

Diff. albedo 𝐷𝑐

Diff. shading 𝐷𝑐𝑆

Diff. recons. 𝑅𝑐𝑑

Spec. albedo 𝑆𝑐

Spec. shading 𝑆𝑐𝑆Spec. recons. 𝑅𝑐𝑠

Unwrapped texture 𝑇𝑚

Completed texture 𝑇𝑓

Diff. albedo 𝐷𝑓𝑏Bare skin 𝐷𝑚𝑚Makeup (1 − 𝐴) ⊙ 𝐷𝑚Diff. shading 𝐷𝑓𝑆Spec. recons. 𝑅𝑓𝑠

Rendered makeup face

Bare skin

MakeupDiffuseSpecularFigure 7: Outputs for each step and overlaid rendering images.

© 2023 The Author(s)Computer Graphics Forum © 2023 The Eurographics Association and John Wiley & Sons Ltd.

Yang et al. / Makeup Extraction of 3D Representation via Illumination-Aware Image Decomposition

Input image 𝐼𝑖𝑛

Diff. albedo 𝐷𝑐

Diff. shading 𝐷𝑐𝑆

Diff. recons. 𝑅𝑐𝑑

Spec. albedo 𝑆𝑐

Spec. shading 𝑆𝑐𝑆Spec. recons. 𝑅𝑐𝑠

Recons. image 𝐼𝑟𝑒𝑐

Diff. albedo 𝐷𝑐

Diff. shading 𝐷𝑐𝑆

Diff. recons. 𝑅𝑐𝑑

Spec. albedo 𝑆𝑐

Spec. shading 𝑆𝑐𝑆Spec. recons. 𝑅𝑐𝑠

Unwrapped texture 𝑇𝑚

Completed texture 𝑇𝑓

Diff. albedo 𝐷𝑓𝑏
Bare skin 𝐷𝑚𝑚Makeup (1 − 𝐴) ⊙ 𝐷𝑚

Diff. shading 𝐷𝑓𝑆Spec. recons. 𝑅𝑓𝑠

Rendered makeup face

Bare skin

MakeupDiffuseSpecularFigure 8: Outputs for each step and overlaid rendering images© 2023 The Author(s)Computer Graphics Forum © 2023 The Eurographics Association and John Wiley & Sons Ltd.

Yang et al. / Makeup Extraction of 3D Representation via Illumination-Aware Image Decomposition

Rendering and result overlay

UV texture

2D portrait

Input

Unwrapped and completed

Bare skin

MakeupDiff. shading

Spec. recons.

Figure 9: Makeup-aware facial inverse rendering and component-wise reconstruction. The top row displays a makeup portrait input andoverlaid rendering images (from left to right: bare skin only, bare skin plus makeup, bare skin multiplied by diffuse shading, and plus specular
reconstruction) whereas the bottom row shows disentangled materials in the UV space

Step 1. Coarse Reconstruction (Sec. 3.1)

Step 2. UV Completion and Refinement (Sec. 3.2)

Step 3. Extraction (Sec. 3.3)

𝑬

Input image 𝐼𝑖𝑛

Diff. albedo 𝐃𝑐

Diff. shading 𝐃𝑐𝑆

Diff. recons. 𝐑𝑑𝑐

𝑭𝑹𝑵

Unwrapped 𝐓𝑢

Diff. shading 𝐃𝑓𝑆

Diff. albedo 𝐃𝑓

DSD-GANSpec. albedo 𝐒𝑐

Spec. shading 𝐒𝑐𝑆Spec. recons. 𝐑𝑠𝑐

Bare skin 𝐃𝑏𝑚

Completed 𝐓𝑓

recons. 𝐼𝑟𝑒𝑐

3D face recons.

UV completion
Geometry 𝐆𝑐

Normal 𝐍𝑐

Recons. texture 𝐑 𝑐Coarse facial materials of 3DMM

Spec. recons. 𝐑𝑓𝑠

Alpha matte 𝐀

Normal 𝐍𝑓

Refined facial materials
Makeup 𝐃𝑚𝑚 Alpha matte (1 − 𝐀)

Makeup extraction

Figure 10: Overview of our framework. Given a makeup portrait, we extract an illumination-independent bare skin and makeup in the UV
space via the following three steps; First, we reconstruct a 3D face to estimate the coarse facial materials using a 3D face reconstruction
network FRN. Second, we refine the coarse facial materials, which might have missing pixels due to occlusion. We apply an inpainting
network DSD-GAN and then apply optimization. Finally, we extract a bare skin, makeup, and an alpha matte from the refined diffuse albedo
using makeup extraction network E.

© 2023 The Author(s)Computer Graphics Forum © 2023 The Eurographics Association and John Wiley & Sons Ltd.

Yang et al. / Makeup Extraction of 3D Representation via Illumination-Aware Image Decompositionℒ𝑡𝑣
initializeℒ𝐷Diff. albedo 𝐃𝑐

Diff. recons. 𝐑𝑓𝑑

Diff. albedo 𝐃𝑓ℒ𝑡𝑣
initializeℒ𝑁

Normal 𝐍𝑐
SH lighting 𝐋𝑠ℎ𝑐

Normal 𝐍𝑓
initializeℒ𝑠ℎℒ𝑟𝑒𝑐𝑜𝑛𝑠

Lambert
Shading

SH lighting 𝐋𝑓𝑠ℎ

ℒ𝑣𝑔𝑔

Diff. shading 𝐃𝑓𝑆

Full recons. texture 𝐑𝑓

Completed texture 𝐓𝑓ℒ𝑡𝑣
initializeℒ𝑅𝑠

Spec. recons. 𝐑𝑠𝑐

Spec. recons. 𝐑𝑓𝑠

Figure 11: Facial material optimization module for Step 2 We optimize the coarse facial materials {Dc , Nc , Rsc } and SH lighting Lcsh so thatthe full reconstruction R f resembles the completed texture T f .The refined diffuse albedo D f , normal N f , and specular reconstruction Rsf arethe outputs. ⊕ and ⊗ denote the per-pixel addition and multiplication, respectively.

© 2023 The Author(s)Computer Graphics Forum © 2023 The Eurographics Association and John Wiley & Sons Ltd.

Yang et al. / Makeup Extraction of 3D Representation via Illumination-Aware Image Decomposition

NeutralExpression

Pose

Illumination(a) Input(b) facial inverse rendering(3D face recons. result, Diff. albedo, Diff. shading)(Spec. shading, Spec. albedo, Spec. recons.)(c) Completion(d) Final output(Diff. shading, Bare skin)(Spec. recons, Makeup)(e) Recons.

Figure 12: Intermediate outputs for each step. Left to right and top to bottom: (a) input makeup portraits, (b) facial inverse rendering (diffusealbedo, diffuse shading, specular shading, specular albedo, and specular reconstruction), (c) unwrapped and completed textures, (d) final
refined facial materials (diffuse shading, bare skin, specular reconstruction, and makeup), and (e) rendered bare skin face and renderedmakeup face using textures from (d).

© 2023 The Author(s)Computer Graphics Forum © 2023 The Eurographics Association and John Wiley & Sons Ltd.

Yang et al. / Makeup Extraction of 3D Representation via Illumination-Aware Image Decomposition(a) Input (b) Rendering (c) Bare skin (d) Makeup(e) Diff.

(f) Spec.

Figure 13: Final outputs of our framework. (a) Input makeup portraits, and (b) fully reconstructed renderings and corresponding textures.
Columns from (c) to (f) are similar to those of Fig. 9.

© 2023 The Author(s)Computer Graphics Forum © 2023 The Eurographics Association and John Wiley & Sons Ltd.

Yang et al. / Makeup Extraction of 3D Representation via Illumination-Aware Image Decomposition(a) Input(b) Completion(c) Diff. albedo(d) Diff. shading(e) Spec. recons.

Figure 14: Outputs of decomposed refined facial materials in complex illumination conditions. Left to right: (a) input makeup portraits, (b)completed UV textures, (c) diffuse albedo, (d) diffuse shading, and (e) specular reconstruction.

© 2023 The Author(s)Computer Graphics Forum © 2023 The Eurographics Association and John Wiley & Sons Ltd.

Yang et al. / Makeup Extraction of 3D Representation via Illumination-Aware Image Decomposition(a) Input(b) Diff. albedo(c) Diff. shading(d) Diff. recons.

(e) Spec. albedo(f) Spec. shading(g) Spec. recons.

Figure 15: Qualitative ablation study for skin tone adjustment. For each identity, the upper and lower rows display the results before and after
the skin tone adjustment, respectively. Left to right: (a) input face images, (b) diffuse albedo, (c) diffuse shading, (d) diffuse reconstruction,(e) specular albedo, (f) specular shading, and (g) specular reconstruction.

© 2023 The Author(s)Computer Graphics Forum © 2023 The Eurographics Association and John Wiley & Sons Ltd.