Relighting Humans: Occlusion-Aware Inverse Rendering for Full-Body
Human Images
YOSHIHIRO KANAMORI, University of Tsukuba
YUKI ENDO, University of Tsukuba & Toyohashi University of Technology

Input image

Light transport

New light

New light

Light

AlbedoRelit shading

Relit image

Relit shading

Relit image

Fig. 1. Given a single human image and its mask, our method infers a light transport map (i.e., coefficient vectors of second-order spherical harmonics that
encode light occlusion), light, and albedo map using convolutional neural networks, and allows fast relighting of the human figure under different illuminations
with self-shadows. Input image courtesy of Jose Martinez.
ACM Reference Format:
Yoshihiro Kanamori and Yuki Endo. 2018. Relighting Humans: OcclusionAware Inverse Rendering for Full-Body Human Images. ACM Trans. Graph.
37, 6, Article 270 (November 2018), 11 pages. https://doi.org/10.1145/3272127.
3275104

Relighting of human images has various applications in image synthesis.
For relighting, we must infer albedo, shape, and illumination from a humanportrait. Previous techniques rely on human faces for this inference, basedon spherical harmonics (SH) lighting. However, because they often ignore
light occlusion, inferred shapes are biased and relit images are unnaturallybright particularly at hollowed regions such as armpits, crotches, or garment
wrinkles. This paper introduces the first attempt to infer light occlusion in the
SH formulation directly. Based on supervised learning using convolutional
neural networks (CNNs), we infer not only an albedo map, illumination but
also a light transport map that encodes occlusion as nine SH coefficients per
pixel. The main difficulty in this inference is the lack of training datasetscompared to unlimited variations of human portraits. Surprisingly, geometric
information including occlusion can be inferred plausibly even with a small
dataset of synthesized human figures, by carefully preparing the dataset sothat the CNNs can exploit the data coherency. Our method accomplishes
more realistic relighting than the occlusion-ignored formulation.

1

CCS Concepts: • Computing methodologies → Rendering; Image-based
rendering;
Additional Key Words and Phrases: inverse rendering, light transport, convolutional neural network
Authors’ addresses: Yoshihiro Kanamori, University of Tsukuba, kanamori@cs.tsukuba.
ac.jp; Yuki Endo, University of Tsukuba & Toyohashi University of Technology, endo@
val.cs.tut.ac.jp.
© 2018 Copyright held by the owner/author(s). Publication rights licensed to the
Association for Computing Machinery.
This is the author’s version of the work. It is posted here for your personal use. Not forredistribution. The definitive Version of Record was published in ACM Transactions on
Graphics, https://doi.org/10.1145/3272127.3275104.

INTRODUCTION

Relighting of human images has various applications in image synthesis such as stylized shading of portraits [Chai et al. 2015; Shuet al. 2017a] or cut & paste of human image clips [Xue et al. 2012].
For physically-based relighting of a human portrait, we must infer
reflectance, shape, and illumination from the single image. Previous
techniques obtain the cues of albedo and shape from human faces
via fitting of morphable 3D face models [Blanz and Vetter 1999]
or inference based on convolutional neural networks (CNNs) [Sengupta et al. 2018], and infer illumination on the basis of sphericalharmonics (SH) lighting [Basri and Jacobs 2003; Ramamoorthi and
Hanrahan 2001].
The SH-based lighting yields an elegant analytical formulation of
shading from surface normals and illumination if we ignore the lightocclusion; we can calculate per-pixel SH bases from normals, andthen illumination is obtained in the form of SH coefficients using
least squares [Kemelmacher-Shlizerman and Basri 2011]. However,as is well known in the realtime-rendering literature, rendered images without light occlusion lacks photorealism because hollowed
regions become unnaturally bright, although they ought to be occluded, compared to other regions. An approximate solution is to

ACM Transactions on Graphics, Vol. 37, No. 6, Article 270. Publication date: November 2018.

270:2 •Kanamori and Endodarken the hollowed regions by multiplying scalar values dependingon occlusion, i.e., ambient occlusion [Zhukov et al. 1998]. A moreelegant solution is to encode light occlusion and cosine decay as
SH coefficients, which we refer to as a light transport vector, andformulate lighting calculation as a dot product of the light transport
vector and SH coefficients of illumination [Sloan et al. 2002]. Unfortunately, calculating occlusion requires the geometry to be inferred,and is quite computationally expensive due to visibility sampling at
each surface point.
In this paper, we introduce the first attempt to infer not only
diffuse albedo but also a light transport vector for each pixel from amasked full-body human image, which is accomplished by supervised learning using CNNs, with a ground-truth training dataset
synthesized from scanned 3D human figures. The main problem
in making this inference possible is the lack of a training dataset,
considering the unlimited variations of human portraits regarding
poses, genders, builds, and garments. To the best of our knowledge,
there is only a single publicly-available dataset of scanned 3D human figures [Zhang et al. 2017], but it lacks variations (i.e., only fiveindividuals with one or two outfits each). We additionally purchasedcommercial datasets of clothed 3D human figures, amounting to
only a few hundreds of models. Surprisingly, by carefully selecting
standing figures and aligning them in the training images, CNNs canlearn plausible light transport vectors, which can capture occlusions
at armpits, crotches or garment wrinkles, even from such a small
dataset. This result implies that CNNs can learn geometric information including occlusion from the silhouettes of human figures, i.e.,binary masks to some extent, which is a similar conclusion drawn
from the recent work inferring normal maps only from silhouettelines [Lun et al. 2017].
Thanks to the inferred light transport maps, we can relight humanportraits quite efficiently just by calculating dot products of lighttransport vectors and SH coefficients of light, followed by channelwise multiplication of inferred albedo maps. The inference of albedo
and light transport maps is fast (0.43 sec. for each 1024 × 1024
image), and our inferred albedo and light transport vectors havesufficient quality for plausible relighting of human images, as shownin Figure 1.

2

RELATED WORK

For single-image physically-based relighting, we must solve inverse
rendering, i.e., estimation of shape, reflectance, and illuminationfrom a single image, which is a highly ill-posed problem. Classicalmethods relax it by assuming that some of the three components
are known, or use prior knowledge of the target in order to estimate the remaining components. Recent methods adopt data-driven
approaches that exploit statistics of the three components in thetarget domain.
Classical inverse rendering. The earliest technique is shape-fromshading [Horn 1989], which estimates shape from the shading in
an input image with known illumination. While methods in theearly years assume simple illumination models such as point, directional, or area light sources, recent ones adopt environmental
illumination represented with second-order SH [Johnson and Adelson 2011]. Also with known shape (e.g., convex shape [Chandraker
ACM Transactions on Graphics, Vol. 37, No. 6, Article 270. Publication date: November 2018.

and Ramamoorthi 2011], occluding contour [Lopez-Moreno et al.
2013], or approximate geometry [Kholgade et al. 2014]), one canestimate reflectance and illumination. Another mainstream in thisliterature is intrinsic images [Barrow and Tenenbaum 1978; Bonneel
et al. 2017], which decomposes an input image into shading (i.e.,
the product of shape and illumination) and reflectance based on the
Retinex theory [Land and McCann 1971]. With this decomposition,
we can change the color or texture while retaining the shading.
However, for relighting, we must further decompose the shading
into shape and illumination.
Data-driven approaches. Data-driven approaches are commonly
adopted in recent techniques for, e.g., outdoor/indoor illumination
estimation [Gardner et al. 2017; Hold-Geoffroy et al. 2017], estimation of specular reflectance and illumination [Georgoulis et al. 2018;
Oxholm and Nishino 2012] as well as intrinsic images [Baslamisli
et al. 2018; Bell et al. 2014; Narihira et al. 2015; Shi et al. 2017]. As ageneralization of both shape-from-shading and intrinsic images, Barron and Malik [2015] factored single input images of general objects
into shape, diffuse reflectance, and SH illumination, via optimization
with statistical priors.
Face inverse rendering. Simultaneous inference similar to Barronand Malik’s work has been actively studied in the inverse renderingof human faces since the seminal work of the 3D morphable model(3DMM) [Blanz and Vetter 1999]. The 3DMM is a statistical modelof albedo and shape of human faces and serves as a strong priorfor face inverse rendering via geometric fitting to the target face
image. While the illumination model used in the original 3DMM
paper [Blanz and Vetter 1999] was directional light, currently thestandard choice is again second-order SH. Due to the increase oflarge-scale publicly-available face datasets, many learning-based
methods with [Tewari et al. 2017] and without [Sengupta et al.
2018; Shu et al. 2017b] 3DMMs have been proposed for face inverse
rendering.
Our work also adopts second-order SH illumination, but tackles
inverse rendering of not only faces but also full bodies including
garments. Full body images contain face regions, and thus existing
techniques for faces can be applied to infer illumination. However,
one concern is that most of the existing techniques assume thatlight occlusion is ignorable; this assumption might be valid for faces
because most faces are approximately convex except for the vicinity
of noses, but it does not hold true for concave regions in the human
body, e.g., armpits, a crotch, or a neck under a chin, that should
receive less light due to self-shadowing. Consequently, such concave
regions become unnaturally bright if we ignore the light occlusion.
For better relighting, we learn light occlusion for SH-based shading.
Schneider et al. [2017] also proposed to account for light occlusion
in SH-based face inverse rendering to better handle face wrinkles.
They extended a 3DMM [Paysan et al. 2009] so that not only albedo
and shape but also per-vertex light transport vectors can be reconstructed via multilinear regression. However, light transport vectors
are available only in the face region.
Apart from the SH-based formulation, Yamaguchi et al. [2018]
inferred a base mesh and high-quality textures of a face from asingle image. Without considering lighting formulation, they infer
textures for photorealistic rendering of faces using regression with

Relighting Humans • 270:3
an adversarial loss. Their method relies on plenty of high-quality
measured data, which are unfortunately not available in general forhuman bodies.
Other human-oriented techniques. Traditionally, human wholebody relighting has been performed based on measurement undercontrolled setups with multiple lights and cameras [Debevec et al.
2000; Li et al. 2013]. In monocular settings, RGB video cameras arealso used for capturing faces with multiple temporal frames, e.g.,[Garrido et al. 2013]. Here we focus on single-image techniques. If
the human target figure is almost naked, we can obtain a reasonableshape cue for inverse rendering by fitting statistical 3D body models [Anguelov et al. 2005; Balan et al. 2007] after segmenting out thefigure mask [Guan et al. 2009]. However, this is generally not applicable to human figures wearing garments. There are also techniquesthat can estimate garment shapes from single images [Danerek et al.
2017; Zhou et al. 2013]. Our method is versatile and can capture
garment wrinkles plausibly from various human portraits.
CNN-based techniques for material inference. Recent methods caninfer materials [Aittala et al. 2016; Li et al. 2017] of objects using
CNNs from a single image of flat-surface objects. Innamorati etal. [2017] proposed an interesting approach that decomposes aninput image into multiple components for manual photo retouching. They account for light occlusion in the form of ambient occlusion and decompose the shading component into six directions
based on non-negative first-order SH bases. With this formulation,
photo-retouch artists can emulate relighting by manually increasing/decreasing directional shading components. Inspired by theirwork, we will compare our method with the conventional SH formulation plus ambient occlusion in Section 7.

3

SPHERICAL HARMONICS (SH) LIGHTING

In this section, we briefly review spherical harmonics (SH) lightingwith and without consideration of light occlusion.

3.1

SH Lighting without Occlusions

SH are orthonormal basis functions defined on the spherical domain,and known as advantageous for capturing low-frequency signals in
the rendering community. It is shown that just nine SH bases (i.e.,
basis functions up to second order) can capture up to 99.22% of theirradiance on a convex surface [Basri and Jacobs 2003].
Let us review the mathematical formulation [Ramamoorthi and
Hanrahan 2001]. If we ignore light occlusion and interreflection, theirradiance E(n) can be calculated with an integral of arbitrary incoming radiance L(ωi ) over the hemispherical domain Ω(n) definedby a unit normal vector n∫
E(n) =
L(ωi ) max(n · ωi , 0) dωi .
(1)
Ω(n)

We omit the dependency on surface position for simplicity. Ramamoorthi and Hanrahan projected the spherical signals of theincoming illumination distribution L(ωi ) and the cosine decay termmax(n · ωi , 0) to SH. Using elevation and azimuth angles θ , ϕ toparameterize a unit direction vector ω = (θ, ϕ), these signals are

expanded as
L(θ, ϕ) =ÕLl,m Yl,m (θ, ϕ),

(2)

Õ

(3)

l,m

A(θ ) = max(cos θ, 0) =Al Yl,0 (θ ),lwhere Yl,m are SH with l ≥ 0, −l ≤ m ≤ l, and m ≤ 2. Ll,mand Al are coefficients for the illumination and cosine decay term,
respectively. A(θ ) does not depend on the azimuth angle ϕ. The
integral in Equation (1) is now rewritten asÕE(θ, ϕ) =Âl Ll,m Yl,m (θ, ϕ),
(4)
l,m

q

4π
2l +1 Al . Here Yl,m can be represented as polynomialsof coordinates of a unit normal n = (x, y, z)T . If we rewrite thewhere Âl =coefficients {Ll,m } as a vector L and the basis functions {Âl Yl,m }as a vector Ŷ, E is calculated as a dot product
E = ŶT L.

3.2

(5)

SH Lighting with Occlusions

Although the above formulation is elegant, the critical problem isthat light occlusion is ignored. Concave regions should receive lesslight due to self-shadowing, and thus should be darker than otherconvex regions. To account for light occlusion in Equation (1), thevisibility term V (ωi ) should be added in the integrand∫E(n) =
L(ωi ) V (ωi ) max(n · ωi , 0) dωi .
(6)
Ω(n)

V (ωi ) returns zero if the light in the incoming direction ωi is occluded and one otherwise. Unfortunately, V (ωi ) does not have any
analytical form in general, and one must sample visibility by casting
many shadow rays at each surface point, which is quite computationally expensive.
Sloan et al. [2002] proposed to precompute the visibility term
together with the cosine decay term, and project the compound
spherical signal onto SH in order to enable efficient dot-productcalculation (similar to Equation (5)) during real-time rendering
E = TT L,(7)

where T is a vector that encodes SH coefficients of the compound
spherical signal of the visibility term and the cosine decay term.
They also proposed to handle glossy reflection and approximate interreflection. This technique is well-known as precomputed radiancetransfer (PRT), which has been studied and extended extensively inthe real-time rendering literature.
Hereafter we refer to T as a light transport vector and a ninechannel image containing per-pixel light transport vectors as a lighttransport map.

4

OUR LOSS FUNCTIONSIn this section, we define the loss functions to infer albedo and lighttransport maps using our CNNs based on the SH formulation.
For training and testing, we prepare a synthetic human image
dataset DH and an illumination dataset DL (see Section 6 for thedetails). The synthetic human image dataset DH contains a set of
ACM Transactions on Graphics, Vol. 37, No. 6, Article 270. Publication date: November 2018.

270:4 •Kanamori and Endo

Albedo decoder
Skip-connection
Input data

Body image I
Binary mask M
Ground truth (GT)

Albedo map ΛLight ΠTransport map ψ

ResNetblock

Inferred data~
Albedo map Λ

Deconv.

Mult

Encoder(Conv.)
Skip-connection

Conv.

Concatenate

ResNet
block

~

Mult

Light Π~Body image I~
Transport map ψ

Deconv.

Transport decoder
15 types of loss functions

Fig. 2. Our network architecture. The input image (multiplied with the binary mask) is fed to the encoder, and the output feature vector is then fed to thedecoders of albedo, light transport, and light (Section 5). We calculate 15 types of loss functions by considering the total variations (TVs) for albedo and lighttransport maps as well as the combinations of inferred outputs and their ground-truth (GT) (Section 4).

a binary mask Mcj ∈ {0, 1} N ×c (where N is the number of pixels,
c is the number of channels, and j = 1, 2, . . . , |DH |), albedo mapΛj ∈ RN ×3 , and light transport map Ψj ∈ RN ×9 for each 3D humanmodel. The illumination dataset DL contains SH illumination coefficients for RGB channels Πk ∈ R9×3 , where k = 1, 2, . . . , |DL |. Notethat we multiply the binary mask Mcj to the ground-truth data andnetwork outputs (e.g., M3j ∗ Λj or M9j ∗ Ψj , where ∗ denotes elementwise multiplication) so that we can ignore out-of-mask pixels. In
the following explanation, we omit the element-wise multiplicationof the binary mask for simplicity.
We use a CNN architecture for inferring light, albedo, and lighttransport maps (see Section 5 for the network models). The inputof the CNN is a masked, RGB full-body human image Ij,k = Λj ∗(Ψj Πk ). Let Λ̃j,k = Λ̃(Ij,k ; Θλ ) ∈ RN ×3 be the CNN output for analbedo map, Ψ̃j,k = Ψ̃(Ij,k ; Θψ ) ∈ RN ×9 the CNN output for a lighttransport map, and Π̃j,k = Π̃(Ij,k ; Θπ ) ∈ RN ×9 the CNN output
for illumination. Regarding notations, we use tildes (˜) to indicate
inferred outputs, and denote f˜ (x; y) to indicate that x is the input
and y is the parameter of network f˜ . We optimize these network
parameters Θλ , Θψ , and Θπ via regression.
Our CNN architecture has a similar design to SfSNet [Sengupta
et al. 2018], which infers light, albedo, and normal maps for faces
simultaneously. The loss functions used in SfSNet are L1 losses forthe inferred albedo map, normal map (from which a light transport
map without light occlusion can be calculated analytically), light,
and the reconstructed image using the three components. We also
use similar four loss functions, but we do not infer normal maps but
infer light transport maps directly. Namely, we use L1 losses for Λ̃j,k ,Ψ̃j,k , Π̃j,k , and the reconstructed image Ĩj,k = Λ̃j,k ∗ (Ψ̃j,k Π̃j,k ).
Furthermore, we also use the following L1 losses:
TV losses: L1 total variation (TV) losses both for albedo Λ̃j,k
and light transport maps Ψ̃j,k ,Shading losses: Three patterns of combination of inferred/GTdata to compute a shading map, i.e., Ψ̃j,k Πk , Ψj Π̃j,k , andΨ̃j,k Π̃j,k ,ACM Transactions on Graphics, Vol. 37, No. 6, Article 270. Publication date: November 2018.

Reconstruction losses: Six patterns of combination to reconstruct an input image, i.e., Λj ∗ (Ψj Π̃j,k ), Λj ∗ (Ψ̃j,k Πk ),Λj ∗ (Ψ̃j,k Π̃j,k ), Λ̃j,k ∗ (Ψj Πk ), Λ̃j,k ∗ (Ψj Π̃j,k ), and Λ̃j,k ∗(Ψ̃j,k Πk ).
In total, we use 15 L1 losses. All weights are set to one.
To consider the benefit of the 15 losses, let us take the shading
losses, i.e., the three losses for a shading map, as an example. For themultiplication of a light transport map and a light, there are threecombinations, namely, GT * inferred (i.e., Ψj Π̃j,k ), inferred * GT (i.e.,Ψ̃j,k Πk ), and inferred * inferred (i.e., Ψ̃j,k Π̃j,k ), where GT means
ground truth. If GT is involved, GT works as a weighting matrix forthe inferred output, which enforces the output to lie on a solution
manifold in the high-dimensional space. If both are inferred outputs,
the loss becomes a soft constraint for the intermediate output, i.e.,
the inferred shading map. For the choice of loss functions, we chose
formulae that do not introduce bias, except for the TV losses. In
this way, involving as many formulae that do not introduce bias
as possible as losses is a quite general technique, and would be
beneficial to other problems. We show an ablation study with andwithout these losses in Section 7.1.

5

NETWORK MODELS

Figure 2 illustrates our encoder-decoder network. As mentioned,
our network is similar to that of SfSNet, except that ours has much
more parameters. Our encoder has six convolutional layers whose
output channels are { 64, 128, 256, 512, 512, 512 } and the stride is two.
The encoded features are then fed to the decoders for albedo, lighttransport, and light. The decoders for albedo and light transportmaps have almost the same architecture, except that the numbers of
output channels are different (i.e., nine for light transport and three
for albedo). Each decoder has a residual block (consisting of two
convolutional layers with 512 channels) and six deconvolutional
layers (output channels are { 512, 512, 256, 128, 64, 9 or 3 } andthe stride is also two). The encoder and decoders are connectedusing skip-connections. For the light decoder, the outputs of the encoder and decoders for albedo and light transport are concatenatedRelighting Humans • 270:5
and fed to four convolutional layers, which yield a 27-dimensional
vector. While SfSNet uses average pooling layers, ours consists of(de-)convolutional layers only. Each (de-)convolutional layer (except
for the first and final layers) is followed by batch normalization and(leaky) ReLU. The first three deconvolutional layers of each decoder
are followed by dropout with probability 0.5.

6

DATASET GENERATIONAs explained in Section 4, we prepared a synthetic human image
dataset and an illumination dataset. Here we explain the details.
Synthetic human image dataset. Our synthetic human image dataset
consists of a binary mask, albedo map, normal map, and light transport map, created by rendering each scanned 3D human figure usinga hardware-accelerated renderer. The scanned 3D human figures
were obtained from two resources; one is the publicly-available
BUFF dataset [Zhang et al. 2017], and the other is commercial websites. The BUFF dataset contains 9,613 standing 3D figures, but lacks
variations for our purpose. Namely, it only includes five individuals
with one or two outfits each and time-varying poses, and thus subsequent 3D models of the same individual in the same outfit are almost
identical. To avoid biasing the training dataset, we manually picked
74 representative models from the BUFF dataset. The commercial
data were purchased from different websites and amount to 271
models. We randomly split the models, 345 in total, into 276 training
data and 69 test data. Figure 3 shows some examples of our training
data. Note that some albedo maps contain self-shadows because
shading was not completely removed during the scanning process.
When creating the dataset, we carefully aligned 3D models sothat our CNNs can exploit the geometric regularity of our small
dataset. Namely, we rendered front-facing figures in the middle ofsquare images while aligning them so that they have almost thesame vertical size in the images with vertical paddings at the top
and bottom of a fixed size (5% of image heights). Regarding poses,
we only used standing figures and removed sitting ones from ourtraining/test datasets. The image resolution is 1024 × 1024 pixels.
No data augmentation is employed for the human image dataset.
Illumination dataset. For our illumination dataset, we used theLaval Indoor HDR dataset [Gardner et al. 2017] containing 2,144 environment maps in panoramic HDR format. We first converted theminto diffuse SH coefficients and calculated a reference brightnessof each environment map using Equation (5) with a front-facing
normal n = (0, 0, 1)T . We omitted dark environment maps if thereference brightness is lower than 0.2, and scaled the brightness ofother environment maps so that reference brightness lies within[0.7, 0.9]. To obtain further variations, we rotated each data 35 times
by 10 degrees around the vertical axis. We then reduced the redundancy using k-means clustering and manually removed unusual
illuminations (e.g., too bright lights, back-lights, and lights causing
too strong contrasts in shadings). Finally, from the remaining 50
illuminations, we randomly picked 40 illuminations for training and
10 for testing. Figure 4 shows some examples of our training data.

Fig. 3. Examples from our synthetic human image dataset. For each humanfigure, an albedo map, binary mask, normal map, and light transport map(visualized as a shading map) are displayed. Note that some albedo maps
contain self-shadows due to scanning inadequacy. Each image is trimmed.

Fig. 4. Examples from our illumination dataset. SH illumination coefficientsare visualized as shading maps of a sphere and the top-left model in Figure 3.

7

EXPERIMENTS

We implemented our CNN models using Python and the chainer
library, and ran our code on a PC with NVIDIA GeForce GTX 1080
Ti GPUs. We used Adam as an optimizer with a fixed learning rate
0.0002 and batch size 1. The computation times for one epoch oftraining on a single GPU were about three hours with our CNN
models. We used the synthetic images of 1024 × 1024 pixels fortraining in our results. Our CNN models, as well as other models
for comparisons, were trained up to 60 epochs. For relighting, weused Debevec’s environment maps [2004], namely, kitchen_probe
for Figures 1, 7, and 8 and grace_probe for Figure 1. The input
photographs in our results were downloaded from Unsplash1 . Specifically, we selected high-quality free-license images of single humanfigures, generated their binary masks automatically using Adobe
Photoshop with manual correction, applied trimming and uniform
scaling, and then added paddings to make them 1024 × 1024 pixels.
1 https://unsplash.com/

ACM Transactions on Graphics, Vol. 37, No. 6, Article 270. Publication date: November 2018.

270:6 •Kanamori and Endo

Table 1. RMSE and SSIM for the inferred results using each method. The light transport maps of SfSNet and SfSNet-AO were computed from corresponding
normal maps analytically. Best values are highlighted in boldface.

SfSNetSfSNet-AO
Ours (min)Ours (full)Shading
0.299
0.293
0.237
0.219

RMSE within binary masks
Transport Normal
AO
Light
0.526
0.346
N/A
0.207
0.529
0.347
0.083 0.207
0.406
N/A
N/A
0.205
0.393
N/A
N/A 0.199

Albedo
0.135
0.131
0.131
0.129

Shading
0.884
0.890
0.909
0.927

SSIM within bounding boxes of masks
Transport Normal
AO
Light
0.755
0.776
N/A
0.446
0.749
0.772
0.946 0.475
0.777
N/A
N/A
0.473
0.781
N/A
N/A 0.500

Albedo
0.954
0.955
0.953
0.943

Table 2. RMSE and SSIM for an ablation study for our 15-losses. Best values are highlighted in boldface.

W/o TVW/o shading
W/o reconstruction

InputsSfSNetSfSNet-AO Ours (min) Ours (full)Shading
0.226
0.227
0.224

GT

RMSE
Transport Light
0.391
0.202
0.398
0.201
0.394
0.198

SfSNetSfSNet-AO

Albedo
0.126
0.132
0.144

GT

Shading
0.923
0.922
0.925

SfSNet-AO

GT

SSIM
Transport Light
0.784
0.471
0.781
0.496
0.782
0.503

SfSNetAlbedo
0.956
0.940
0.907

SfSNet-AO Ours (min) Ours (full)GT

Fig. 5. Comparisons with synthetic data. From left to right, input images, shading maps, normal maps, ambient occlusion maps, light maps (i.e., lightinformation visualized by shading a sphere), and albedo maps. The light maps are in the same order as shading/albedo maps, from top to bottom. The redovals in inferred shading maps highlight differences between SfSNet-AO and “Ours (full).”

7.1

Comparisons of Inference

To clarify the advantage of our method, we compared it with threealternative methods. The first one is SfSNet [Sengupta et al. 2018],
ACM Transactions on Graphics, Vol. 37, No. 6, Article 270. Publication date: November 2018.

Relighting Humans • 270:7

InputsSfSNetSfSNet-AO

Ours (min)Ours (full)

SfSNetSfSNet-AO

SfSNet-AO

SfSNetSfSNet-AO

Ours (min)Ours (full)Fig. 6. Comparisons with real photographs. The red ovals in inferred shading maps highlight differences between SfSNet-AO and “Ours (full).” Input images
courtesy of Guillaume Bolduc, George Gvasalia, Jacob Postuma, and Kat Garcia.

but the network architecture is not the original one for small 128 ×
128 images but much richer one defined in Section 5. In this case, adecoder of SfSNet outputs three-channel normal maps, instead ofnine-channel light transport maps. The second method is SfSNet
plus ambient occlusion (hereafter we call it SfSNet-AO). A singlechannel ambient occlusion is inferred by an additional decoder
branch. The third method is our network with four losses only,
similar to SfSNet. We refer to the 4-loss version as “Ours (min)” andthe 15-loss version as “Ours (full).” Comparisons between “SfSNet”
and “Ours (min)” reveal the impact of considering light occlusion
whereas those between “Ours (min)” and “Ours (full)” demonstrate
the effectiveness of the full loss.
Figures 5 and 6 show the results of qualitative comparisons usingsynthetic test data and photographs, respectively. The red ovals in
inferred shading maps highlight differences between SfSNet-AO
and “Ours (full).” The first row of Figure 5 indicates that all methods
suffer from separating textures from shading maps. The shading

maps of SfSNet often seem like flat bas-reliefs because light occlusion
is ignored. In the first and fourth rows, SfSNet-AO estimates thedepth gaps between jackets and shirts smaller than the actual gap.
Such biased estimate in shading maps often yields unnaturallydarkened albedo maps. Comparing our two variants, i.e., “Ours(min)” and “Ours (full)”, the latter yields sharper shading maps than
the former. Also in Figure 6, we can see the similar tendency withreal photographs.
For quantitative comparison, Table 1 summarizes the RMSE andSSIM of each component. To reduce the effects of out-of-mask-pixels,
we calculate RMSEs within binary masks whereas we calculate
SSIMs within the bounding boxes of binary masks. The table showsthat “Ours (full)” is consistently better than other alternatives except
for “Albedo SSIM”. The reason why “Albedo SSIM” of “Ours (full)”is lower than others is that “Ours(full)” better cancels the baked-in
shadings (see Section 6) in GT albedos and thus its output albedos
become more dissimilar to “GT.” Table 2 further reveals the impacts
ACM Transactions on Graphics, Vol. 37, No. 6, Article 270. Publication date: November 2018.

270:8 •Kanamori and Endo

SfSNetSfSNet-AO

Ours (full)

GT (SH rendering)

Fig. 7. Relighting comparison using synthetic data. The red ovals highlight differences between SfSNet-AO and “Ours (full).”of the TV losses, shading losses, and reconstruction losses. We cansee the tendency that overall the accuracies are lower than those of“Ours (full)” in Table 1. Note that light transport and albedo maps of“W/o TV” are slightly better than those of “Ours (full).” This result isreasonable because TV losses enforce smoothing, i.e., add biases, tothe inferred outputs in compensation for generalization capability.

7.2

Relighting and Light Transfer

Figures 7 and 8 show the results of relighting with inferred albedo
and light transport maps, given synthetic test images and real photographs, respectively. Comparisons with path-traced reference images as well as movies are available in the supplemental material.
By inferring illuminations in two human portraits, we can transfer the inferred illumination to each other. Figure 9 shows the resultsof light transfer with synthetic human images. The inferred illuminations have colors slightly different from the ground-truth, but thepatterns of the illuminations are similar. The relit human images
are therefore similar to the ground-truth.

ACM Transactions on Graphics, Vol. 37, No. 6, Article 270. Publication date: November 2018.

8

DISCUSSIONS

Silhouettes as priors. Some existing methods [Barron and Malik
2015; Lun et al. 2017] on shape-from-shading suggested that object
silhouettes serve as shape priors. In the recent work of the CNNbased shape inference from 2D silhouettes [Lun et al. 2017], the sizeof the training dataset is around ten thousand. Compared to thissize, it was a surprise that we can infer plausible albedo and shadingfrom only a few hundreds of training data. To confirm how much thesilhouettes help our inference, we inferred light transport maps only
from the binary masks. For this inference, we used the CNN model
for light transport maps and used only those loss functions related
to light transport maps. Figure 10 shows the resultant shading maps
and corresponding ground-truth. Surprisingly, we can observe therough concave shapes under the chin and the flat shapes of the instep.
This result implies that our CNN models also learned a strong shapeprior from silhouettes thanks to the regularity of our small training
dataset.

Relighting Humans • 270:9

SfSNetSfSNet-AO

Ours (full)Fig. 8. Relighting comparison using real photographs. The red ovals highlight differences between SfSNet-AO and “Ours (full).” Input images courtesyof George Gvasalia and Jacob Postuma.

GT shading

Masks

Inferred shading

GT shading

Masks

Inferred shading

Fig. 10. Shape from mask. For each human figure, a ground-truth shading
map, mask, and inferred shading map are displayed. These results imply
that our network can learn a strong shape prior from silhouettes.

Inferred light

Inferred light

GT light

GT light

W/o fine-tuning

Input image

Input image

Relit result

Relit GT

Relit result

Relit GT

Fig. 9. Light transfer. By performing inference for two images and swapping
inferred lights, we can transfer the inferred light to each other.

Fully-tuned
Input image

Self-supervised learning. Recent methods for intrinsic decomposition or image disentanglement, e.g., [Sengupta et al. 2018], often employ self-supervised learning to fine-tune networks that aretrained with synthetic data; only the single loss for the differences
between input images and products of inferred outputs is considered, and the network is trained using unlabeled real photographs.
We fine-tuned the model of “Ours (full)” with and without fixing
the network parameters of the decoders. However, in both cases,
the inferred outputs collapsed (see Figure 11); the light transportmaps lost details, the corresponding shading maps bleached, andthe albedo maps got close to the input images. This is probably
because our light transport maps have much larger degrees of freedom (i.e., nine dimensions per pixel) than normal maps inferred in[Sengupta et al. 2018], and thus are more difficult to fine-tune under
the unconstrained setting in self-supervised learning. We thus did
not adopt self-supervised learning in other results. The details of theexperimental settings are available in the supplemental material.
Sitting poses. To evaluate the ability of our network for handlingvarious poses, we fed synthetic human images in sitting poses,

W/o fine-tuning

Fully-tuned

W/o fine-tuning

Fully-tuned

W/o fine-tuning

Encoder-tuned

W/o fine-tuning

Encoder-tuned
Input image

W/o fine-tuning

Encoder-tuned

Fig. 11. Comparison to self-supervised learning. By fine-tuning network
parameters using only unlabeled photographs with (i.e., encoder-tuned;lower row) and without (i.e., fully-tuned; upper row) fixing decoders, theoutputs collapsed; the shading maps bleached and the albedo maps got
close to the input images. Input images courtesy of Philip Martin and Ali
Morshedlou.

which were not included in training or test data. Figure 12 shows
the results. The inferred outputs are unexpectedly well compared to
the ground-truth, which is probably because our training dataset is
ACM Transactions on Graphics, Vol. 37, No. 6, Article 270. Publication date: November 2018.

270:10• Kanamori and Endomaps are inferred accordingly. The large differences in appearance
are then encoded in the inferred albedo maps so that the productsof the three components become similar to the input images. A
straightforward solution is to enrich the training light dataset using,e.g., the environment maps used in [Endo et al. 2017], so that goodnearest neighbors can be found for various inputs.

Inferred light

GT light

8.1
Inferred light

GT light
Input images

Inferred shading

GT shading

Inferred albedo

GT albedo

Fig. 12. Inferred results with sitting poses. Our network can handle sitting
poses unexpectedly well even though it is not trained with them.

Inferred light

Limitations

Here we explain the limitations of our method. Although our method
is based on a better formulation of SH-based lighting, i.e., with consideration of light occlusion, it is still a crude approximation oflighting calculation. First of all, we only handle diffuse albedo. This
limitation mainly stems from our dataset; most of the commercial
data do not have specular components, even though SH representation can naturally handle specular components as demonstrated
in the original PRT paper. Adding artificial specular components
to our training dataset, as done in [Innamorati et al. 2017], seems
inappropriate in our case because human skin and clothes shouldhave different reflectance. Material assignment with semantic segmentation for hundreds of meshes is ideal but can be a challenging
project by itself. As our work is the first attempt regarding both fullbody relighting and SH-based light occlusion learning/inference, we
believe this limitation is acceptable to encourage follow-up studies.
Also, while we used second-order SH for representing light occlusion, Sloan et al. [2002] suggested to use higher-order bases because
occlusion causes high-frequency signals. As is often the case withlearning-based methods, our method might fail with conditionsquite dissimilar to the training dataset, e.g., harsh illuminations, as
demonstrated in Figure 13.

GT light

9Inferred light

GT light
Input images Inferred shading GT shading

Inferred albedo GT albedo

Fig. 13. Failure examples with unusual lights. Our network suffers frominferring lights quite different from the training data.

sufficiently rich for the network to learn shapes of body parts suchas arms and legs.
Unusual lights. We also evaluated the ability for handling various
lights, as shown in Figure 13. Unfortunately, our network could notplausibly infer lights that were quite different from those in our training data. Our network seems to reconstruct nearest-neighbor lightsthat can be found in the training dataset, and the light transport
ACM Transactions on Graphics, Vol. 37, No. 6, Article 270. Publication date: November 2018.

CONCLUSIONS AND FUTURE WORKIn this paper, we have paved the way to occlusion-aware relightingfrom single-view human images and accompanying inference using
CNNs. Inspired by the seminal work of the precomputed radiancetransfer [Sloan et al. 2002], we employed SH-based lighting, i.e.,
dot-product calculation of second-order spherical harmonics (SH)coefficient vectors of illumination and occlusion (i.e., light transfer
vectors), and trained our models using our synthetic ground-truth
dataset. Plausible inference of albedo and light transport maps werepossible probably because of our small yet geometrically-aligned human image dataset. By considering light occlusion, inferred albedo
and shading maps (i.e., the product of a light transport map andillumination) as well as relighting results are more plausible thanthose obtained by using previous techniques without considering
light occlusion.
One obvious direction of future work is to extend our first attempt to more physically-accurate inverse rendering, based on theformulations extensively studied in the literature of precomputed radiance transfer. For example, other basis functions such as wavelets
or spherical Gaussians might be beneficial to handle high-frequency
shadows or illumination. A quite important future work would be
to build a publicly-available, high-quality 3D human models, whichis crucial to develop this human-oriented research.

Relighting Humans •ACKNOWLEDGMENTS
The authors would like to thank ZOZO Technologies, Inc. for generous financial support throughout this project, without which thiswork was not possible. The authors would also like to thank theanonymous referees for their constructive comments, and Ms. Sina
Kitz for proof-reading the final version of this paper. For our accompanying video, input images courtesy of Kat Garcia, Kinga Cichewicz, George Gvasalia, and Jacob Postuma.

REFERENCES
Miika Aittala, Timo Aila, and Jaakko Lehtinen. 2016. Reflectance modeling by neural
texture synthesis. ACM Trans. Graph. 35, 4 (2016), 65:1–65:13.
Dragomir Anguelov, Praveen Srinivasan, Daphne Koller, Sebastian Thrun, Jim Rodgers,
and James Davis. 2005. SCAPE: shape completion and animation of people. ACM
Trans. Graph. 24, 3 (2005), 408–416.
Alexandru O. Balan, Leonid Sigal, Michael J. Black, James E. Davis, and Horst W.
Haussecker. 2007. Detailed Human Shape and Pose from Images. In 2007 IEEE
Computer Society Conference on Computer Vision and Pattern Recognition (CVPR
2007).
Jonathan T. Barron and Jitendra Malik. 2015. Shape, Illumination, and Reflectance from
Shading. IEEE Trans. Pattern Anal. Mach. Intell. 37, 8 (2015), 1670–1687.
H. G. Barrow and J. M. Tenenbaum. 1978. Recovering intrinsic scene characteristics
from images. Comp. Vis. Sys. (1978).
Anil S. Baslamisli, Hoang-An Le, and Theo Gevers. 2018. CNN based Learning using
Reflection and Retinex Models for Intrinsic Image Decomposition. In Proceedings ofthe IEEE Conference on Computer Vision and Pattern Recognition (CVPR 2018).
R. Basri and D. W. Jacobs. 2003. Lambertian reflectance and linear subspaces. IEEE
Transactions on Pattern Analysis and Machine Intelligence 25, 2 (Feb 2003), 218–233.
Sean Bell, Kavita Bala, and Noah Snavely. 2014. Intrinsic images in the wild. ACM
Trans. Graph. 33, 4 (2014), 159:1–159:12.
Volker Blanz and Thomas Vetter. 1999. A Morphable Model for the Synthesis of 3D Faces.
In Proceedings of the 26th Annual Conference on Computer Graphics and Interactive
Techniques (SIGGRAPH 1999). 187–194.
Nicolas Bonneel, Balazs Kovacs, Sylvain Paris, and Kavita Bala. 2017. Intrinsic Decompositions for Image Editing. Comput. Graph. Forum 36, 2 (2017), 593–609.
Menglei Chai, Linjie Luo, Kalyan Sunkavalli, Nathan Carr, Sunil Hadap, and Kun Zhou.
2015. High-quality hair modeling from a single portrait photo. ACM Trans. Graph.
34, 6 (2015), 204:1–204:10.
Manmohan Krishna Chandraker and Ravi Ramamoorthi. 2011. What an image reveals
about material reflectance. In IEEE International Conference on Computer Vision(ICCV 2011). 1076–1083.
R. Danerek, Endri Dibra, A. Cengiz Öztireli, Remo Ziegler, and Markus H. Gross. 2017.
DeepGarment: 3D Garment Shape Estimation from a Single Image. Comput. Graph.
Forum 36, 2 (2017), 269–280.
Paul Debevec. 2004.
Light Probe Image Gallery.
(2004).
http://www.pauldebevec.com/Probes/.
Paul E. Debevec, Tim Hawkins, Chris Tchou, Haarm-Pieter Duiker, Westley Sarokin,
and Mark Sagar. 2000. Acquiring the reflectance field of a human face. In Proceedingsof the 27th Annual Conference on Computer Graphics and Interactive Techniques(SIGGRAPH 2000). 145–156.
Yuki Endo, Yoshihiro Kanamori, and Jun Mitani. 2017. Deep reverse tone mapping.
ACM Trans. Graph. 36, 6 (2017), 177:1–177:10.
Marc-André Gardner, Kalyan Sunkavalli, Ersin Yumer, Xiaohui Shen, Emiliano Gambaretto, Christian Gagné, and Jean-François Lalonde. 2017. Learning to predict
indoor illumination from a single image. ACM Trans. Graph. 36, 6 (2017), 176:1–
176:14.
Pablo Garrido, Levi Valgaerts, Chenglei Wu, and Christian Theobalt. 2013. Reconstructing detailed dynamic face geometry from monocular video. ACM Trans. Graph. 32,
6 (2013), 158:1–158:10.
Stamatios Georgoulis, Konstantinos Rematas, Tobias Ritschel, Efstratios Gavves, Mario
Fritz, Luc Van Gool, and Tinne Tuytelaars. 2018. Reflectance and Natural Illuminationfrom Single-Material Specular Objects Using Deep Learning. IEEE Trans. PatternAnal. Mach. Intell. 40, 8 (2018), 1932–1947.
Peng Guan, Alexander Weiss, Alexandru O. Balan, and Michael J. Black. 2009. Estimating
human shape and pose from a single image. In IEEE 12th International Conference
on Computer Vision (ICCV 2009). 1381–1388.
Yannick Hold-Geoffroy, Kalyan Sunkavalli, Sunil Hadap, Emiliano Gambaretto, andJean-François Lalonde. 2017. Deep Outdoor Illumination Estimation. In 2017 IEEE
Conference on Computer Vision and Pattern Recognition (CVPR 2017). 2373–2382.
Berthold K. P. Horn. 1989. Shape from Shading. MIT Press, Chapter Obtaining Shapefrom Shading Information, 123–171.

270:11

Carlo Innamorati, Tobias Ritschel, Tim Weyrich, and Niloy J. Mitra. 2017. Decomposing
Single Images for Layered Photo Retouching. Comput. Graph. Forum 36, 4 (2017),
15–25.
Micah K. Johnson and Edward H. Adelson. 2011. Shape estimation in natural illumination. In The 24th IEEE Conference on Computer Vision and Pattern Recognition (CVPR
2011). 2553–2560.
Ira Kemelmacher-Shlizerman and Ronen Basri. 2011. 3D Face Reconstruction from aSingle Image Using a Single Reference Face Shape. IEEE Trans. Pattern Anal. Mach.
Intell. 33, 2 (2011), 394–405.
Natasha Kholgade, Tomas Simon, Alexei A. Efros, and Yaser Sheikh. 2014. 3D object
manipulation in a single photograph using stock 3D models. ACM Trans. Graph. 33,
4 (2014), 127:1–127:12.
Edwin H. Land and John J. McCann. 1971. Lightness and Retinex Theory. J. Opt. Soc.
Am. 61, 1 (Jan 1971), 1–11.
Guannan Li, Chenglei Wu, Carsten Stoll, Yebin Liu, Kiran Varanasi, Qionghai Dai,and Christian Theobalt. 2013. Capturing Relightable Human Performances under
General Uncontrolled Illumination. Comput. Graph. Forum 32, 2 (2013), 275–284.
Xiao Li, Yue Dong, Pieter Peers, and Xin Tong. 2017. Modeling surface appearance froma single photograph using self-augmented convolutional neural networks. ACM
Trans. Graph. 36, 4 (2017), 45:1–45:11.
Jorge Lopez-Moreno, Elena Garces, Sunil Hadap, Erik Reinhard, and Diego Gutierrez.
2013. Multiple Light Source Estimation in a Single Image. Comput. Graph. Forum32, 8 (2013), 170–182.
Zhaoliang Lun, Matheus Gadelha, Evangelos Kalogerakis, Subhransu Maji, and Rui
Wang. 2017. 3D Shape Reconstruction from Sketches via Multi-view Convolutional
Networks. In 2017 International Conference on 3D Vision (3DV 2017).
Takuya Narihira, Michael Maire, and Stella X. Yu. 2015. Direct Intrinsics: Learning
Albedo-Shading Decomposition by Convolutional Regression. In 2015 IEEE International Conference on Computer Vision (ICCV 2015). 2992.
Geoffrey Oxholm and Ko Nishino. 2012. Shape and Reflectance from Natural Illumination. In 12th European Conference on Computer Vision (ECCV 2012), Proceedings, Part
I. 528–541.
Pascal Paysan, Reinhard Knothe, Brian Amberg, Sami Romdhani, and Thomas Vetter.
2009. A 3D Face Model for Pose and Illumination Invariant Face Recognition. In
Sixth IEEE International Conference on Advanced Video and Signal Based Surveillance(AVSS 2009). 296–301.
Ravi Ramamoorthi and Pat Hanrahan. 2001. An efficient representation for irradiance environment maps. In Proceedings of the 28th Annual Conference on Computer
Graphics and Interactive Techniques, SIGGRAPH 2001. 497–500.
Andreas Schneider, Sandro Schönborn, Bernhard Egger, Lavrenti Frobeen, and Thomas
Vetter. 2017. Efficient Global Illumination for Morphable Models. In IEEE International Conference on Computer Vision (ICCV 2017). 3885–3893.
Soumyadip Sengupta, Angjoo Kanazawa, Carlos D. Castillo, and David W. Jacobs. 2018.
SfSNet: Learning Shape, Reflectance and Illuminance of Faces ‘in the Wild’. In
Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR
2018).
Jian Shi, Yue Dong, Hao Su, and Stella X. Yu. 2017. Learning Non-Lambertian Object
Intrinsics Across ShapeNet Categories. In 2017 IEEE Conference on Computer Vision
and Pattern Recognition (CVPR 2017). 5844–5853.
Zhixin Shu, Sunil Hadap, Eli Shechtman, Kalyan Sunkavalli, Sylvain Paris, and Dimitris
Samaras. 2017a. Portrait Lighting Transfer Using a Mass Transport Approach. ACM
Trans. Graph. 36, 4, Article 145a (Oct. 2017).
Zhixin Shu, Ersin Yumer, Sunil Hadap, Kalyan Sunkavalli, Eli Shechtman, and Dimitris
Samaras. 2017b. Neural Face Editing with Intrinsic Image Disentangling. In 2017
IEEE Conference on Computer Vision and Pattern Recognition (CVPR 2017). 5444–5453.
Peter-Pike J. Sloan, Jan Kautz, and John Snyder. 2002. Precomputed radiance transfer
for real-time rendering in dynamic, low-frequency lighting environments. ACM
Trans. Graph. 21, 3 (2002), 527–536.
Ayush Tewari, Michael Zollhöfer, Hyeongwoo Kim, Pablo Garrido, Florian Bernard,
Patrick Pérez, and Christian Theobalt. 2017. MoFA: Model-Based Deep Convolutional Face Autoencoder for Unsupervised Monocular Reconstruction. In IEEE
International Conference on Computer Vision (ICCV 2017). 3735–3744.
Su Xue, Aseem Agarwala, Julie Dorsey, and Holly E. Rushmeier. 2012. Understanding
and improving the realism of image composites. ACM Trans. Graph. 31, 4 (2012),
84:1–84:10.
Shugo Yamaguchi, Shunsuke Saito, Koki Nagano, Yajie Zhao, Weikai Chen, Kyle Olszewski, Shigeo Morishima, and Hao Li. 2018. High-fidelity Facial Reflectance andGeometry Inference from an Unconstrained Image. ACM Trans. Graph. 37, 4, Article
162 (July 2018), 162:1–162:14 pages.
Chao Zhang, Sergi Pujades, Michael J. Black, and Gerard Pons-Moll. 2017. Detailed,
Accurate, Human Shape Estimation from Clothed 3D Scan Sequences. In 2017 IEEE
Conference on Computer Vision and Pattern Recognition (CVPR 2017). 5484–5493.
Bin Zhou, Xiaowu Chen, Qiang Fu, Kan Guo, and Ping Tan. 2013. Garment Modelingfrom a Single Image. Comput. Graph. Forum 32, 7 (2013), 85–91.
Sergey Zhukov, Andrei Iones, and Grigorij Kronin. 1998. An Ambient Light Illumination
Model. In Rendering Techniques ’98, Proceedings of the Eurographics Workshop. 45–56.

ACM Transactions on Graphics, Vol. 37, No. 6, Article 270. Publication date: November 2018.