Generating Digital Painting Lighting Eﬀects via RGB-space Geometry
LVMIN ZHANG, Soochow University / Style2Paints, China
EDGAR SIMO-SERRA, Waseda University / JST PRESTO, JapanYI JI, Soochow University, China
CHUNPING LIU, Soochow University, China
Stroke Density

Stroke history & Canvas image

MeasuredLighting Effects

Estimated

00:06

29:17

278:25

Ours

Artist (conditioned)

Artist (unconditioned)

Fig. 1. Generating digital painting lighting eﬀects. An artist is invited to draw a digital painting on the left. We then estimate a stroke density map using
the canvas image, which is compared to the measured stroke density computed with the artist’s real stroke history. The estimated stroke density is used togenerate our lighting eﬀects in real-time. We also invite artists to draw lighting eﬀects for comparisons. One artist, namely conditioned artist, is invited topaint eﬀects with similar style to ours, and another artist, namely unconditioned artist, is asked to try best to relight the original image according to artistic
vision. The time taken for each approach is shown at the bottom of each image (in minutes), and the same with other figures in this paper.
We present an algorithm to generate digital painting lighting eﬀects from asingle image. Our algorithm is based on a key observation: artists use many
overlapping strokes to paint lighting eﬀects, i.e., pixels with dense stroke
history tend to gather more illumination strokes. Based on this observation,
we design an algorithm to both estimate the density of strokes in a digital
painting using color geometry, and then generate novel lighting eﬀects bymimicking artists’ coarse-to-ﬁne workﬂow. Coarse lighting eﬀects are ﬁrst
generated using a wave transform, and then retouched according to thestroke density of the original illustrations into usable lighting eﬀects.
Our algorithm is content-aware, with generated lighting eﬀects naturally
adapting to image structures, and can be used as an interactive tool tosimplify current labor-intensive workﬂows for generating lighting eﬀects
for digital and matte paintings. In addition, our algorithm can also produce
usable lighting eﬀects for photographs or 3D rendered images. We evaluate
our approach with both an in-depth qualitative and a quantitative analysis
which includes a perceptual user study. Results show that our proposed
approach is not only able to produce favorable lighting eﬀects with respect
to existing approaches, but also that it is able to signiﬁcantly reduce theneeded interaction time.
CCS Concepts: • Applied computing → Arts and humanities; Fine arts;
Media arts.
Authors’ addresses: Lvmin Zhang, lmzhang9@stu.suda.edu.cn, Soochow University/ Style2Paints, China; Edgar Simo-Serra, ess@waseda.jp, Waseda University / JST
PRESTO, Japan; Yi Ji, jiyi@suda.edu.cn, Soochow University, China; Chunping Liu,
cpliu@suda.edu.cn, Soochow University, China.
© 2020 Association for Computing Machinery.
This is the author’s version of the work. It is posted here for your personal use. Not forredistribution. The deﬁnitive Version of Record was published in ACM Transactions on
Graphics, https://doi.org/10.1145/3372176.

Additional Key Words and Phrases: relighting, color, convex hull
ACM Reference Format:
Lvmin Zhang, Edgar Simo-Serra, Yi Ji, and Chunping Liu. 2020. Generating
Digital Painting Lighting Eﬀects via RGB-space Geometry. ACM Trans. Graph.
1, 1, Article 1 (January 2020), 15 pages. https://doi.org/10.1145/3372176

1

INTRODUCTION

Lighting plays an important role in digital and matte painting. Unlike physical illumination in real world or rendered scenes, thepainted lighting eﬀects in digital paintings are created by artists
using heterogeneous strokes. The objective is artistic expression,and thus, more often than not, the illumination is not subject toany physical constraints. In current artistic workﬂows, artists manually paint these lighting eﬀects and tediously modify them to ﬁnd
the best composition. In order to create usable lighting eﬀect products, artists usually ﬁrst draw some global or coarse illumination
layers, and then retouch the details of these layers to naturally ﬁt
the original image content in a very time-consuming process. Some
examples of lighting eﬀects are shown in Fig. 1 and 2.
The problem of creating or manipulating digital painting lighting
eﬀects in illustrations or matte paintings has important diﬀerenceswith 3D relighting like [Debevec et al. 2000], which usually hasemphasis on exploiting the 3D object structure or physical lightgeometry, whereas in this work we focus on the artistic stroke structure in digital paintings. We aim at simplifying the labor-intensive
lighting eﬀect painting workﬂow, where many objects are not evenpainted to have 3D appearance, making it hardly possible to solve
ACM Trans. Graph., Vol. 1, No. 1, Article 1. Publication date: January 2020.

1:2

•

Lvmin Zhang, Edgar Simo-Serra, Yi Ji, and Chunping Liu

175:14

195:07200:01

191:31

154:07

145:12

18:21

12:04

55:24

41:51

31:42

45:31

Fig. 2. Examples of manually painted lighting eﬀects. We show several examples of lighting eﬀects created in diﬀerent styles by multiple artists. The
reference lighting direction is shown in the top-left while the recorded painting time is shown in the top-right (in minutes).

the accurate physical geometry of these objects. Moreover, in reallife lighting eﬀect creation, it is very important to ensure that details of artists’ carefully painted illustration are not signiﬁcantly
modiﬁed when applying eﬀects. Even small distortions can severely
limit the applicability to real-world professional digital painting use
cases. This makes it very diﬃcult for normal estimation or proxy
construction algorithms [Hudon et al. 2018; Xu et al. 2015] to meet
artists’ needs.
This paper presents an algorithm to synthesize digital painting
lighting eﬀects based on a key assumption: artists’ newly painted
strokes are related to their previous stroke history. In particular,when artists are asked to paint a novel lighting eﬀect layer, theirstrokes share similarity to their previously painted strokes. To be
speciﬁc, pixels with dense corresponding stroke history tend to
capture more diversiﬁed lighting or shadow eﬀects, whereas pixels
with relatively sparse stroke history are likely to correspond toambient or smooth color. Therefore, the density of stroke history is
an important clue when generating digital painting lighting eﬀects.
For most existing digital paintings, the stroke history is not available and thus the stroke density cannot be directly measured. Furthermore, recording and storing all strokes in a non-destructive
manner is impractical due to the high resolution and sheer numberof strokes used in most digital paintings. In our approach, we bypass this issue by designing an algorithm to directly estimate thedensity of stroke history from a single ﬁnished illustration. Our algorithm ﬁrst extracts a virtual palette from the input image, andthen makes use of the relationship between pixel colors and palettecolors to estimate the stroke density. This estimation does not actually decompose images or extract strokes, whereas it is particularly
eﬀective to guide the synthesis of novel lighting eﬀects.

ACM Trans. Graph., Vol. 1, No. 1, Article 1. Publication date: January 2020.

We then generate the lighting eﬀects by mimicking artists’ workﬂow in a coarse-to-ﬁne approach. Firstly, we generate coarse eﬀect
maps as an initial composition, and then proceed to reﬁne the map
according to the pixel-wise stroke density. The coarse eﬀect map
provides a rough and low-frequency eﬀect of the highlights, shadows, and the color variations caused by the inﬂuence of nearby objects. This coarse eﬀect map is then reﬁned to ﬁt the original image
structure, allowing the generation of aesthetically pleasing lighting
results. Our algorithm supports multiple types of light sources andcan directly serve as a plug-in for professional digital painting tools.
We perform both an in-depth quantitative and qualitative evaluation of our approach, and validate our algorithm in a perceptualuser study and compare with manually painted lighting eﬀects. We
ﬁnd that the lighting eﬀects generated by our approach are signiﬁcantly better than existing approaches. Furthermore, our approach
is found to be usable by artists and can be integrated in the mattepainting workﬂow, which we ﬁnd can speed up lighting eﬀect composition by roughly 50%. Finally, results show our approach also
has applicability to non-painted images such as photographs and
3D rendered ones.
In summary, our contributions are (a) an algorithm to generate
and manipulate digital painting lighting eﬀects, (b) an algorithm toestimate the density of artists’ stroke history for digital paintings, (c)experiments on the connection between colors, strokes, and painted
lighting eﬀects based on real data collected from professional artists,and (d) an in-depth evaluation including a perceptual user study.

Generating Digital Painting Lighting Eﬀects via RGB-space Geometry

2 RELATED WORK
2.1 Sample-based Relighting
In order to manipulate the lighting condition in images, one classic
approach is to use multiple samples with diﬀerent lighting directions on a same scene, and then formulate the relighting objective
as a light transport or interpolation problem. Some typical models
are reﬂectance ﬁeld [Debevec et al. 2000], progressive optimization [Matusik et al. 2004], wavelet [Peers and Dutre 2005], compressive transport [Peers et al. 2009], and deep neural networks [Chenet al. 2010; Peers et al. 2007]. Nevertheless, in digital painting use
cases, samples with diﬀerent lighting directions are labor-intensive
to obtain, making it a chicken-and-egg problem to generate eﬀects
using manually created ones. This limits the applicability of theserelighting approaches. Therefore, instead of requiring samples withmultiple lighting directions, our approach generates digital painting lighting eﬀects directly using the density of the artists’ stroke
history, which makes our tool well-suited for artistic creation scenarios.

2.2 Intrinsic Images
Many image relighting algorithms rely on decomposing images
into reﬂectance (albedo) maps and illumination (irradiance) maps,
known as intrinsic image decomposition [Barrow and Tenenbaum
1978]. Related techniques are usually based on optimization [Bellet al. 2014, 2013, 2015; Shen et al. 2011] or learning [Aksoy et al.
2018; Barron and Malik 2012; Gehler et al. 2011; Serra et al. 2012].
Although intrinsic decomposition is beneﬁcial and eﬀective to diversify relighting applications, in the case of digital paintings there areeven more challenges. Digital illustrations are carefully painted byartists, and thus it is important to preserve as much of the original
details as possible. Nevertheless, the perfect image reconstruction
after layer editing is still a non-trivial open problem to intrinsic
methods. For these reasons, instead of decomposing the illumination beyond input images, we base our pipeline on the structure ofartists’ painting strokes and try to conserve as much of the original
details as possible.

• 1:3

from a lack of generalization caused by dependency on training
data, which contains many implicit and explicit biases. Instead of
relying on the estimation of surface normal and albedo, we makeuse of artists’ stroke density to generate eﬀects that are visuallysimilar to those created manually by artists. Our approach exhibits
stable generalization to a wide variety of images and can be directly
used in digital illustration workﬂows.

2.4 Color Geometry
Besides the albedo-shading decomposition, images can also be decomposed using color geometry. Such color-based decomposition isoften used to analyze the layers or strokes in images. The mixture
or variance of image color has been extensively studied [Aksoy et al.
2017; Finlayson and amd Robert B. Fisher 2017; Koyama and Goto
2018; Lin et al. 2017]. It is also possible for images to be recoloredusing decomposed color layers [Aharoni-Mack et al. 2017; Changet al. 2015; Zhang et al. 2017]. Tan et al. [2016] is the seminal work
to use convex-hull based color palettes for image recoloring. Shih etal. [2013] generate night-time landscapes using a data-driven colortransform. Although our approach is based on analyzing the densityof stroke history, our algorithm does not explicitly decompose theimage nor extract strokes, unlike image layer or stroke decomposition methods [Aksoy et al. 2017; Lin et al. 2017; Tan et al. 2018, 2016;
Zhang et al. 2017]. On the contrary, our stroke density is directly
estimated from pixel colors and image palettes, which reduces theinaccuracy caused by the hyper-parameters in image decomposition processing.

3

APPROACH

Our algorithm is based on a key assumption: when painting lighting
eﬀects, the artists’ newly painted strokes are related to the stroke
history of the painting. Pixels with relatively denser stroke history
tend to gather more newly painted lighting or shadow strokes,
whereas pixels with relatively sparse stroke history are often shadedwith less newly painted eﬀect strokes. Based on this principle, ouralgorithm ﬁrst estimates the density of stroke history and thengenerates the lighting eﬀects.

2.3 Normal Estimation and Proxy Construction

3.1 Estimating Stroke Density

Another common relighting approach is to estimate the albedo
and surface normals, or proxy surfaces. This allows to eﬃciently
compute the illumination for diﬀerent directions [Kender and Smith
1984; Wu et al. 2008]. Fitting surfaces to artistic creations has also
been extensively studied [Sykora et al. 2014; Wu et al. 2007; Xu
et al. 2015]. These approaches typically depend on the consistencyof the image albedo, as well as use strong assumptions such as
Lambertian scenes. Recently, deep learning methods [Hudon et al.
2018; Su et al. 2018] have been also used to estimate surface normals.
Sengupta et al. [2018] propose to estimate the normal map of faces.
Kanamori and Endo [2018] extend this technique to full-body shotsof individuals. Yu and Smith [2019] design an unsupervised training
strategy to estimate normal maps for buildings and sculptures. Philip
et al. [2019] build up a geometry relighting proxy to help neuralnetworks, and Sun et al. [2019] propose to relight facial images
using neural networks from a small but high-quality omnidirectional
dataset. However, in general, learning based approaches may suﬀer

First of all, we consider the simple problem of blending two strokes:
𝒄 = 𝛼1 𝒄1 + 𝛼2 𝒄2s.t. 𝛼 1, 𝛼 2 > 0, 𝛼 1 + 𝛼 2 = 1(1)
3
where {𝒄 1, 𝒄 2 } ∈ R are the stroke colors, {𝛼 1, 𝛼 2 } ⊂ R are theblending weights, and 𝒄 ∈ R3 is the generated color. In this case, wedeﬁne the stroke density 𝑘 as:
𝑘=

1−

q
𝛼 12 + 𝛼 22

1 − √1(2)

2where we can see that if the two strokes are equally blended (𝛼 1 =𝛼 2 = 0.5), then 𝑘 becomes 1. On the contrary, if only one strokecolor is visible (e.g., 𝛼 1 = 0, 𝛼 2 = 1), then 𝑘 becomes 0. In this way,
the stroke density measures to what extent the stroke colors arevisible. The stroke density increases when multiple stroke colors
are visible, and it decreases to zero when only one stroke color canbe observed.
ACM Trans. Graph., Vol. 1, No. 1, Article 1. Publication date: January 2020.

1:4

•

Lvmin Zhang, Edgar Simo-Serra, Yi Ji, and Chunping Liu

Algorithm 1 Stroke density map computation.

front

back

Convex HullUV unfolding

Fig. 3. Palette extraction. The RGB-convex hull of the input image isregarded as the palette. We visualize all colors in the front side (looking
from the white position) and the back side (looking from the black position)of the mesh using RizomUV unfolding tool.

Similarly, we consider a set C including 𝑛 stroke colors and a set
A including 𝑛 blending weights:
C = {𝒄 1, . . . , 𝒄𝑛 }and A = {𝛼 1, . . . , 𝛼𝑛 }(3)3where each 𝒄𝑖 ∈ R is a stroke color and each 𝛼𝑖 ∈ R is a blending

weight. The blending of all these stroke colors can be formulated as:
𝑛
𝑛
Õ
Õ
𝒄=
𝛼𝑖 𝒄𝑖 , s.t.
𝛼𝑖 = 1, 𝛼𝑖 > 0, 𝑖 = 1, . . . , 𝑛(4)
𝑖=1

Input: An image consist of RGB vectors {𝒄 1,...,𝐻 ×𝑊 }, where all
pixel RGB color 𝒄𝑥 = (𝑟 𝑥 , 𝑔𝑥 , 𝑏𝑥 );Output: A map 𝐾 of the pixel-wise stroke density {𝑘 1,...,𝐻 ×𝑊 };1: {V, F } ← 𝐶𝑜𝑛𝑣𝑒𝑥𝐻𝑢𝑙𝑙 ({𝒄 1,...,𝐻 ×𝑊 }); // Calculate the RGB
convex hull, where V is a set of the vertices on the hull and F
is a set of the faces on the hull.
𝑠 𝑖 Í31Í
2: 𝒈 ← 𝑠 𝑛
𝑖=1 3 𝑡 =1 F𝑖,𝑡 ; // Calculate the barycentre 𝒈, whereF𝑖,𝑡 means the 𝑡-th vertex in the 𝑖-th triangle in F , 𝑠 means thetotal area of the convex hull, and 𝑠𝑖 means the area of the 𝑖-th
triangle.
3: M ← 𝑀𝑒𝑠ℎ(𝑣𝑒𝑟𝑡𝑖𝑐𝑒𝑠 = V, 𝑓 𝑎𝑐𝑒𝑠 = F ); // Build up the mesh
M in RGB space.
𝒄 𝐻 ×𝑊 −𝒈𝒄 1 −𝒈4: R ← 𝑅𝑎𝑦 (𝑝𝑜𝑠𝑖𝑡𝑖𝑜𝑛 = 𝒈, 𝑑𝑖𝑟𝑒𝑐𝑡𝑖𝑜𝑛𝑠 = {|𝒄 1 −𝒈 | , . . . , |𝒄 𝐻 ×𝑊 −𝒈 | });// Cast rays in RGB space.
5: H ← 𝐼𝑛𝑡𝑒𝑟𝑠𝑒𝑐𝑡𝑖𝑜𝑛(R, M); // Calculate the hit points between
the ray set R and the mesh M, where H = {𝒉 1,...,𝐻 ×𝑊 }.
|𝒄−𝒉𝐻 ×𝑊 ||𝒄−𝒉 1 |6: 𝐾 ← {|𝒈−𝒉 1 | , . . . , |𝒈−𝒉𝐻 ×𝑊 | };7: return 𝐾 = {𝑘 1,...,𝐻 ×𝑊 };𝑖=1

where 𝒄 is the generated color. In this case, the stroke density 𝑘 canbe written with Eq. (2) as:
qÍ
𝑛 𝛼2
1−
𝑖=1 𝑖
𝑘=
(5)
1 − √1𝑛

When all strokes are equally blended (𝛼𝑖 = 𝑛1 ), then 𝑘 becomes 1.
On the contrary, if no strokes are blended and only one color isvisible (e.g., 𝛼 1 = 1), then 𝑘 becomes 0. The stroke density increases
when more stroke colors can be observed, and decreases when lessstroke colors are visible.
We then solve the problem of stroke density map estimation
given a palette. In this paper, a palette refers to a speciﬁc set ofcolors used to paint a target image. In other words, all pixel colorsof a given image should be reproducible by mixing or sampling
the colors from its palette. Here we use a convex-hull-based palettebecause its dense and continuous mesh geometry can signiﬁcantly
meliorate and simplify our stroke density computation, which will
be discussed later.
To identify such mesh-like palette, we compute the RGB-spaceconvex hull of all observed pixel colors using the quick hull algorithm [Bradford et al. 1996; Tan et al. 2016] as
M = QH3 ( { 𝒄𝑖 | 𝑖 = 1, . . . ,𝑊 × 𝐻 } )(6)3
where 𝒄𝑖 ∈ R is an observed pixel color, QH3 (·) is the function

to compute the 3D convex hull, and M is the mesh of the convex
hull, including all faces and all vertices. The convex hull is a tightwrapping of the colors, and, as shown in Fig. 3, the colors on thesurfaces of the convex hull are regarded as the palette. The palette
is a mesh in RGB space, and the colors on it can be visualized by
ﬂattening the mesh into a 2D surface using UV unfolding.
In order to estimate the stroke density, our main idea is to discoverpossible color combinations in the palette that could reproduce eachpixel color, and then compute the stroke density using the blending
ACM Trans. Graph., Vol. 1, No. 1, Article 1. Publication date: January 2020.

weights of the discovered combinations. we ﬁrst randomly sample 𝑛
diﬀerent points {𝒄 1, . . . , 𝒄𝑛 } in the palette M. The more points are
sampled, the more accurate the estimation will be. We denote thequantity of these points as 𝑛. We will later approximate 𝑛 → +∞for optimal accuracy. First of all, the barycentre 𝒈 of these randomly
sampled points can be formulated as:
𝑛

𝒈=

1Õ
𝒄𝑖
𝑛 𝑖=1

(7)

For each pixel position 𝑝, the input of our estimation algorithm isa pixel color 𝒄𝑝 ∈ R3 , and the output is the corresponding stroke
density 𝑘𝑝 ∈ R. For each color 𝒄𝑝 , we cast a ray from 𝒈 ∈ R3 to
𝒄𝑝 ∈ R3 in the RGB-space. Such a ray will intersect with the surfaceof M at one point, namely 𝒉𝑝 ∈ R3 . Given {𝒈, 𝒄𝑝 , 𝒉𝑝 } are collinear,
the color of 𝒄𝑝 can be formulated as:|𝒄𝑝 − 𝒉𝑝 ||𝒄𝑝 − 𝒉𝑝 |𝒉𝑝 +𝒈
𝒄𝑝 = 1 −|𝒈 − 𝒉𝑝 ||𝒈 − 𝒉𝑝 |𝑛|𝒄𝑝 − 𝒉𝑝 |1 Õ |𝒄𝑝 − 𝒉𝑝 |
𝒉𝑝 +
= 1−
𝒄𝑖
|𝒈 − 𝒉𝑝 |𝑛 𝑖=1 |𝒈 − 𝒉𝑝 |(8)

where | · | is the Euclidean distance. Then, the given color 𝒄𝑝 canthen be written as color blending with Eq. (3)C = {𝒄 0 = 𝒉𝑝 } ∪ {𝒄𝑖 | 𝑖 = 1 . . . 𝑛}(9)
|𝒄𝑝 − 𝒉𝑝 ||𝒄𝑝 − 𝒉𝑝 |} ∪ {𝛼𝑖 =|𝑖 = 1 . . . 𝑛} (10)A = {𝛼 0 = 1 −|𝒈 − 𝒉𝑝 |𝑛|𝒈 − 𝒉𝑝 |

Generating Digital Painting Lighting Eﬀects via RGB-space Geometry

barycenter ( g )pixel color ( c )convex hull

hit point ( h )gr
OriginColor Space

Fig. 4. Visualization of the RGB-space geometry. A color point 𝑐 can be
described in function of the convex hull barycentre 𝑔 and the intersectionof a ray cast from 𝑔 to 𝑐 and the convex hull surface which we denote as 𝑝.

The stroke density 𝑘𝑝 can then be computed from the blending
weights A at the limit where 𝑛 → +∞ as 𝒈 approaches the barycentre of M at:r |𝒄 −𝒉 |  2
|𝒄𝑝 −𝒉𝑝 | 2 Í𝑛𝑝𝑝
1 − |𝒈−𝒉+1−𝑖=1 𝑛 |𝒈−𝒉𝑝 |𝑝|
𝑘𝑝 = lim
𝑛→+∞
1− √1
𝑛+1
v
t
2
2
𝑛 
|𝒄𝑝 − 𝒉𝑝 |1 Õ |𝒄𝑝 − 𝒉𝑝 |=1−
+ lim 2
1−𝑛→+∞ 𝑛|𝒈 − 𝒉𝑝 ||𝒈 − 𝒉𝑝 |𝑖=1
|𝒄𝑝 − 𝒉𝑝 ||𝒄𝑝 − 𝒉𝑝 |==1− 1−(11)|𝒈 − 𝒉𝑝 ||𝒈 − 𝒉𝑝 |We can see this allows the stroke density to be estimated easily
given the pixel color 𝒄𝑝 at position 𝑝, barycentre of the convex hull𝒈, and the intersection with the ray cast from 𝒈 to 𝒄𝑝 denoted as 𝒉𝑝 .
As shown in Fig. 4, this theoretical result is consistent to our previous claims: when all the colors in the palette are equally mixed, the𝑘 becomes 1 because the resulting color 𝒄𝑝 is exactly the barycentreof the palette mesh M and |𝒄𝑝 − 𝒉𝑝 | = |𝒈 − 𝒉𝑝 |. On the contrary,when the resulting color is exactly on M, the 𝑘 decreases to zero
because |𝒄𝑝 − 𝒉𝑝 | = 0. Finally, we provide a summarized version of
the stroke density estimation algorithm in Alg. 1.

3.2 Generating Lighting Eﬀect
Our lighting eﬀect generating algorithm is designed to mimic theartists’ lighting eﬀect workﬂow. As shown in Fig. 5, our proposed
algorithm shares many similarities with the standard artist workﬂow.
Screenshots of the artists’ original illustration stroke history are
shown in Fig. 5-(a) and the measured stroke density is presented inFig. 5-(b). To obtain the eﬀects, artists ﬁrst paint the coarse eﬀect
layer and then retouch it into a reﬁned eﬀect layer. Then, the eﬀect
is applied to the original illustration to achieve Fig. 5-(f).
In the workﬂow from Fig. 5-(a) to Fig. 5-(f), the artist is conditioned to only paint routine soft lighting eﬀects without any otherenhancements such as tone adjustment or adding additional details.
Our algorithm is designed to simulate such eﬀect. We would liketo also point out that if the artists are asked to try best to ﬁnish all
possible shading, it may take them hours to achieve the eﬀects with
their personal styles as shown in Fig. 5-(g).
Before we describe the detailed algorithm, we present several
observations based on the artist’s workﬂow. Firstly, we randomly
sample patches in the painted eﬀect in Fig. 5-(e). We divide thesepatches into two groups: low stroke density patches and high stroke• 1:5

density patches. We observe that areas with relative low stroke
density generally correspond to ambient and smooth colors, whereas
areas with relatively high stroke density are likely to be painted withhighlights or shadows. We will present this experiment in detaillater in §4.2.
Based on this observation, we generate a lighting eﬀect map by
mimicking artists’ coarse-to-ﬁne workﬂow. Given a single image
in Fig. 5-(h) without available stroke history, we directly estimate
the stroke density in Fig. 5-(j). Afterwards, using the normalized
channel intensity map, we generate a coarse eﬀect map, which
can then be reﬁned to achieve the ﬁnal lighting eﬀect. Finally, thelighting eﬀects are applied to the illustration as shown in Fig. 5-(n).
In particular, we here discuss the case of a point light source. Our
objective is to compute a lighting eﬀect map 𝑺 ∈ R𝑊 ×𝐻 ×3 fromthe input image 𝑹 ∈ R𝑊 ×𝐻 ×3 , stroke density map 𝑲 ∈ R𝑊 ×𝐻 ,light source 𝒍 ∈ R3 , light intensity 𝛾 ∈ R, and ambient intensity
𝑂 ∈ R. We compute the lighting eﬀects for each channel of the image
separately, and denote a channel 𝑐 as of the original image andlighting eﬀect map as 𝑹𝑐 ∈ R𝑊 ×𝐻 and 𝑺𝑐 ∈ R𝑊 ×𝐻 , respectively.
3.2.1 Normalization. To avoid the high-frequency noise, we calculate the low-frequency normalized channel intensity 𝑵𝑐 of theoriginal image channel 𝑹𝑐 with𝑵𝑐 =𝜌 (𝑹𝑐 ) − min(𝜌 (𝑹𝑐 ))max(𝜌 (𝑹𝑐 )) − min(𝜌 (𝑹𝑐 ))(12)

where 𝜌 (·) is a Gaussian ﬁlter. For the Gaussian ﬁlter we use a 64×64
pixel window and 𝜎 = 16 for 512px images.
3.2.2 Coarse Eﬀect Generation. We then generate the coarse lighting eﬀect 𝑬 shown in Fig. 5-(l). The motivation of this step is to create an initial draft of the lighting eﬀects, which is similar to artists’coarse eﬀect composition in Fig. 5-(c). Because our coarse eﬀect isonly used to provide low-frequency features for further reﬁnements,
it is not necessary to create eﬀects that are consistent with artists’coarse eﬀect. This step also produces the color inﬂuence of nearby
objects, e.g., the red hair in Fig. 5-(d) can inﬂuence the face color
with red or blue color oﬀsets on the face boundary.
We compute the channels 𝑬𝑖 in the coarse lighting eﬀect map 𝑬 ina pixel-wise fashion. We build up a xyz coordinate system using thex-axis and y-axis of original image and a new z-axis perpendicular
to the image panel. We denote the position of the light source as𝒍 = [𝑙𝑥 , 𝑙 𝑦 , 𝑙𝑧 ] ⊺ . For any pixel location 𝒑 = [𝑝𝑥 , 𝑝 𝑦 ] ⊺ , we deﬁne adistance 𝑝𝑑 and angle 𝜃 asq
(13)
𝑝𝑑 = (𝑝 𝑦 − 𝑙 𝑦 ) 2 + (𝑝𝑥 − 𝑙𝑥 ) 2
andsin 𝜃 =

𝑝𝑦 − 𝑙𝑦,𝑝𝑑

cos 𝜃 =

𝑝 𝑥 − 𝑙𝑥𝑝𝑑

(14)

We then deﬁne the following wave functions𝑒𝑐∗ (𝒍, 𝜃, 𝑝𝑑 ) = 𝑒𝑐 (𝑙𝑥 + 𝑝𝑑 cos 𝜃, 𝑙 𝑦 + 𝑝𝑑 sin 𝜃 )𝑛𝑐∗ (𝒍, 𝜃, 𝑝𝑑 ) = 𝑛𝑐 (𝑙𝑥 + 𝑝𝑑 cos 𝜃, 𝑙 𝑦 + 𝑝𝑑 sin 𝜃 )𝑘 ∗ (𝒍, 𝜃, 𝑝𝑑 ) = 𝑘 (𝑙𝑥 + 𝑝𝑑 cos 𝜃, 𝑙 𝑦 + 𝑝𝑑 sin 𝜃 )

(15)
(16)
(17)

ACM Trans. Graph., Vol. 1, No. 1, Article 1. Publication date: January 2020.

1:6

•

Lvmin Zhang, Edgar Simo-Serra, Yi Ji, and Chunping Liu

Artists’ workflow
Low Density PatchesHigh Density Patches(a) Artist’s realstroke history(b) Measured
stroke density(c) Artist’s coarse
effect layer(d) Artist’s refined
effect layer

33:24

241:59(e) Visualization ofpainted patches(f) Artist’s final
lighting effect(g) Effect createdwith another style(l) coarse
lighting effect (E)(m) Refined lighting
effect (S)(n) Output(I)

Proposed algorithm(h) Original image(R)(i) Extracted palette(M)(j) Estimated stroke
density (K)(k) Normalized
channel intensity (N)Fig. 5. Comparison of the artist workflow with the proposed approach. In the first row is components in the artist’ manual workflow, and in the second
row is components in the pipeline approach of our algorithm.

zln*c (l,Θ,pd)

py

y
px

pd
x

d
Θ

z
y

py
px

k*(l,Θ,pd)

map 𝑬𝑐 is estimated by calculating the dot production of the wave
direction unit vector and the light direction unit vector.

pd

3.2.3 Refinement. When artists are painting lighting eﬀects, pixels
with relatively low stroke density tend to be shaded with ambient
gray, whereas pixels with relatively high stroke density are likely togather diversiﬁed highlight or shadow colors. Based on this principle,
we reﬁne the coarse lighting eﬀect map using the stroke density
map with
𝑺𝑖 = 𝛾 𝑬𝑖 ⊙ 𝑲 + 𝑂(19)where ⊙ is the Hadamard product, and we recommend to set default
ambient intensity to 𝑂 = 0.55. This reﬁnement step can also be
calculated pixel-wise as

d
x Θ

K

Nc

where 𝑒𝑐 (𝑥, 𝑦), 𝑛𝑐 (𝑥, 𝑦), and 𝑘 (𝑥, 𝑦) are the pixel values at thelocation [𝑥, 𝑦] ⊺ for the coarse lighting eﬀect map 𝑬𝑐 , normalized
input image channel 𝑵𝑐 , and stroke density map 𝑲 , respectively.
We observe that if we ﬁx 𝒍 and 𝜃 , these functions can be viewed aswaves that only depend on 𝑝𝑑 . We use bilinear interpolation when
the coordinates 𝑥 and 𝑦 are not integer values.
Each peak of the wave function 𝑛𝑖∗ (·) has a side facing the lightsource and a side facing away from the light source. We can increase
the intensity of the front side and reduce the intensity of the backside to simulate a light source. Similar to the shape-from-shadow
algorithm [Kender and Smith 1984], all values in 𝑬𝑐 can be written
in pixel wise as:𝑒𝑖 (𝑝𝑥 , 𝑝 𝑦 ) = 𝑒𝑖 (𝑙𝑥 + 𝑝𝑑 cos 𝜃, 𝑙 𝑦 + 𝑝𝑑 sin 𝜃 ) = 𝑒𝑖∗ (𝒍, 𝜃, 𝑝𝑑 )[𝑛𝑖∗ (𝒍, 𝜃, Δ𝑝𝑑 + 𝑝𝑑 ) − 𝑛𝑖∗ (𝒍, 𝜃, 𝑝𝑑 ), Δ𝑝𝑑 ][𝑙𝑧 , 𝑝𝑑 ] ⊺·=|[𝑛𝑖∗ (𝒍, 𝜃, Δ𝑝𝑑 + 𝑝𝑑 ) − 𝑛𝑖∗ (𝒍, 𝜃, 𝑝𝑑 ), Δ𝑝𝑑 ]| |[𝑙𝑧 , 𝑝𝑑 ]||{z} | {z }
wave

(18)

light

where Δ𝑝𝑑 is a scaling scalar, which we set to Δ𝑝𝑑 = 1 in all our
experiments. Every unknown pixel value in the coarse lighting eﬀect
ACM Trans. Graph., Vol. 1, No. 1, Article 1. Publication date: January 2020.

𝑠𝑖 (𝑝𝑥 , 𝑝 𝑦 ) = 𝑠𝑖 (𝑙𝑥 + 𝑝𝑑 cos 𝜃, 𝑙 𝑦 + 𝑝𝑑 sin 𝜃 )= 𝛾 𝑒𝑖∗ (𝒍, 𝜃, 𝑝𝑑 ) 𝑘 ∗ (𝒍, 𝜃, 𝑝𝑑 ) + 𝑶(20)

where 𝑠𝑐 (𝑝𝑥 , 𝑝 𝑦 ) is the value in 𝑺𝑖 at 𝒑 = [𝑝𝑥 , 𝑝 𝑦 ] ⊺ . This reﬁnement encourages the shading values to stay away from the ambient
intensity when the pixel stroke density is high. On the contrary, ifsome pixels are of low stroke density, this reﬁnement discourages
the shading, and these pixels will be shaded with the ambient intensity. Finally, the lighting eﬀect can be multiplied to the original
image with
𝑰 =𝑹⊙𝑺(21)
where 𝑰 ∈ R𝑊 ×𝐻 ×3 is the rendered result. In cases where the lightsource has colors, the channels in 𝑺 can be multiplied with the lightcolor. Additionally, this algorithm also supports other types of lightGenerating Digital Painting Lighting Eﬀects via RGB-space Geometry

sources, e.g., lights at inﬁnity or spot lights. Please refer to our
supplementary material for other types of the light sources.

4

EXPERIMENTS

In this section, we validate the assumptions about the colors, strokes,
and painted lighting eﬀects. Additionally, we provide quantitativeand qualitative analysis which includes perceptual user studies.

4.1 Qualitative Results
We compare our stroke density maps estimated on single images tothe real stroke density maps measured with artists’ stroke history,
and, simultaneously, we compare our generated lighting eﬀects to
those created manually by artists, as shown in Fig. 6. In particular,
we conduct two user studies by dividing the participated artists into
two groups: unconditioned group and conditioned group.
In the unconditioned group, we do not show our tool to the artists
and directly ask them to try best to relight illustrations according tosome simple lighting directions. We use results from this group to
analyze to what extent our synthesized eﬀects are similar to thosecreated manually by artists.
In the conditioned group, we allow artists to test our tool with
arbitrary images to familiarize themselves with our eﬀect style, andwe then ask them to paint eﬀects with similar styles. We use resultsin this group to exploit how long time does it take for the artists tocreate eﬀects that have similar style to our generated ones.
It is notable that the artists, especially those in the conditioned
group, are prohibited from using our tools when painting, in order toensure that they are drawing with their own artistic visions insteadof simply reproducing our generated results. Furthermore, we would
like to point out that the lighting eﬀect creation is a labor-intensive
task, and we report the recorded painting time for each manuallycreated eﬀect in corresponding ﬁgures. We also include comparison
with multiple lighting direction in Fig. 7. Although the current
algorithm cannot replace the manually created eﬀects which takes
artists hours to create, i.e., the artist results in the unconditioned
group are generally better than the algorithm results, the majorityof our generated eﬀects are of satisfactory visual quality and are
acknowledged by those professional artists as usable. Finally, resultsin the conditioned group show that it take artists about 30 minutes
to create eﬀects with the style similar to ours.

4.2 Connection between Colors, Strokes, and Manually
Painted Artistic Lighting Eﬀects
As shown in Fig. 8, artists are invited to paint the example illustrations, and we record their strokes to compute the ground truth
stroke density, and, simultaneously, estimate the stroke density using the single canvas images as each stroke is drawn. Finally, we
visualize the error between the ground truth and the estimation,and report the Peak Signal to Noise Ratio (PSNR). The raw data of
this experiment is provided in the supplementary material.
During the digital painting process, our color-geometry-based
stroke density estimation algorithm becomes increasingly accurate
because more and more colors in the canvas are evenly distributedin the RGB color space. Although this estimation is not a perfectmeasurement, it yields visually similar stoke density maps and can• 1:7

achieve PSNR scores at roughly 26, which demonstrates that thisalgorithm is a reliable source of stoke density estimation.
Moreover, as shown in Fig. 9, we study the connection between
artists’ stroke history and their painting eﬀects. We invite artists tomanually paint artistic lighting eﬀects, and then randomly sample
patches over their created eﬀects in two groups: low stroke density
group and high stroke density group. These patches are sampled bysimply using a binary threshold to divide the stroke density map
into low density part and high density part. Random pixel positions
are then sampled and used as the central position of each patch. Inorder to avoid the bias caused by the background and the stroke
density oﬀsets between large objects, we ask artists to manuallycreate background masks to exclude background patches, and thenuse adaptive threshold1 when we binarize the stroke density maps.
In the visualization, we can see that patches with low stroke
density are likely to be painted with relatively ambient, smooth, anduniform eﬀect color, whereas pixels with high stroke density tends
to gather sharp and intensive eﬀect details and diversiﬁed lighting
and shadow. In order to highlight this observation, we also calculate
the mean standard deviation of these two patch groups, and weﬁnd that areas with high stroke density have signiﬁcantly larger
standard deviation than that of low stroke density. This evidence
further demonstrates that areas with relatively high stroke density
tends to gather diversiﬁed lighting and shadow eﬀects.

4.3 Significance of Using Stroke Density
We conduct experiments to study possible alternatives to the stroke
density, as shown in Fig. 10. Firstly, we present results from a line
drawing normal map estimation method DeepNormal [Hudon et al.
2018], and then use a sprite rendering tool Unity SpriteLamp to
produce the baseline results. This method requires a line drawing
and a mask for each illustration, and we invite artists to manuallycreate the inking lines and the masks. Although this method canperform relighting, it is not well-suited for digital painting because
many distortions of the original illustration details can be observed.
We also replace the stroke density map 𝑲 in our algorithm withthe alternative maps, e.g., the luminance map and the backgroundmap as shown in Fig. 10, where the lighting eﬀects becomes unusable.
Finally, we invite artists to manually create some simple gradientmaps to replace the lighting eﬀect 𝑺 in our algorithm. We observethat these simple gradients are only able to generate very simple andnon-natural looking illumination. Compared to these alternatives,
our algorithm is the best choice to achieve eﬀects that are visuallysimilar to the artists’ manually painted ones. We also include morealternative map formulations as well as animated ablative study inthe supplementary material.

4.4 Comparison with Non-digital-painting Relighting
As our approach does not rely on the real digital painting stroke
history, it is applicable to images even not created using strokes.
We show several results comparing with the surface optimization
approach of [Wu et al. 2008] and the CNN-based approach of [Yu andSmith 2019] in Fig. 11, where we can see how the optimization-based
1 THRESH_GAUSSIAN_C in OpenCV.

ACM Trans. Graph., Vol. 1, No. 1, Article 1. Publication date: January 2020.

1:8

•

Lvmin Zhang, Edgar Simo-Serra, Yi Ji, and Chunping Liu

Stroke Density

Original Image & Stroke History

Measured

Estimated

Lighting Effects

Ours

35:24

200:01

29:11

187:54

44:13

241:37

Artist (Conditioned)

Artist (Unconditioned)

Fig. 6. Comparison on stroke density and lighting eﬀects. We compare our estimated stroke density to the real stroke density measured on the stroke
history, and, at the same time, compare our generated lighting eﬀects to artists’ manually created ones.

approach suﬀers from overly simple model ﬁtting, while the CNNbased approach has trouble handling diversiﬁed image contents. We
also show a visual comparison against the classic height-to-normal
approach of [Kender and Smith 1984] using a Sobel ﬁlter, as shownin Fig. 12, where the height-based relighting results are not wellsuited for lighting eﬀects. Although our generated lighting eﬀects
may not strictly match the accurate physical structure of the 3D
objects, these results are usable from human perception, and canhelp artists in the matte painting or image light editing workﬂow.
Please also refer to the supplementary material for more animatedresults and interactive results.
In our generated eﬀects, we can see that the low frequency domain
plays the dominant role in directionality. We would like to pointout that the high frequency domain is also important to make thegenerated eﬀects naturally adjust to the image structure. In Fig. 13,
we present results only using the low frequency coarse eﬀect 𝑬
without reﬁnement and then compare it to the eﬀect 𝑺 with highfrequency domain reﬁned using our algorithm, where structural
inconsistency artifacts can be found if the high frequency patterns
are not retouched using our proposed method.

ACM Trans. Graph., Vol. 1, No. 1, Article 1. Publication date: January 2020.

4.5 Perceptual User Study
In order to perceptually evaluate and compare our approach withexisting methods, we perform a perceptual user study focusing on
usability aspects of diﬀerent relighting algorithms. In particular, 6
users participated and were asked to relight 10 images with various
approaches. For each image, we randomly select one user to givenew lighting condition instructions to the other users, and haveeach of the remaining ﬁve users relight the image with the newconditions, each using a diﬀerent approach. Finally, the user thatgave the new lighting condition ranks the results. Besides providing
the ranking results, we measure the time it takes to process an image
with each approach on average. Each approach is used in two stages
which are evaluated separately. The ﬁrst stage consists of using theapproach as is, while the second stage consists of using professional
photo editing software as post-processing. The two stages are ranked
separately. Full details are provided in the supplemental material.
We compare our approach with a shape-from-shadow-based algorithm [Kender and Smith 1984], a surface optimization algorithm [Wu et al. 2008], and a recent deep learning-based approach [Yuand Smith 2019]. As professional photo editing software, we use
Adobe PhotoShop Lighting Eﬀects. Results are shown in Table 1.
We ﬁnd that users prefer our approach over all other approaches

Generating Digital Painting Lighting Eﬀects via RGB-space Geometry• 1:9

Original images and our results

Artist (Conditioned)
22:28

Artist (Unconditioned)
25:01

195:36

154:39

16.58
PSNR
30

12.03

14.98

12.25

9.27

12.54

13.87

16.55

21.47

26.65

25

20
15
10
step: 1

24:57

31:03

197:01

177:28

30:01

24:22

215:44

155:03Fig. 7. Comparison on lighting eﬀects with multiple directions. We
compare our generated eﬀects against the manually created lighting eﬀects
painted by the artists. Besides, we present artists’ recorded painting time ineach manually created examples.

23

12.09
PSNR
30

46

69

92

9.84

115 138 161 184 207 230 253 276 299 322 345 368 391 414 437 460 483 506 529 552 575

8.72

10.74

10.24

13.39

14.02

16.77

19.41

26.58

25

20
15
10

in both the ﬁrst stage, using a single tool, and the second stage,
post-processing with professional tools. Of the other approaches weﬁnd users prefer the professional tools, followed [Wu et al. 2008].
The low performance of [Yu and Smith 2019] is likely explained by
the low generalization to general images caused by the learning
procedure. We additionally ﬁnd that our approach also provides asigniﬁcant decrease in the interaction time. When including postprocessing, our approach takes only roughly 50% of the time ofusing the professional photo editing software. We note that theother approaches take longer and obtain worse results than using
only the professional photo editing software. This highlights thestrengths of our approach, which is able to quickly and realisticallyrelight a diversity of images.

4.6 Color Geometry Model Variations
We present experiments with diﬀerent color geometry models andconﬁgurations. In particular, we replace our convex hull palette withthe simpliﬁed hull of [Tan et al. 2018], and replace the estimatedstroke density map with the calculated stroke density based on

step: 1

19

38

57

76

95

114 133 152 171 190 209 228 247 266 285 304 323 342 361 380 399 418 437 456 475

Fig. 8. Stroke density estimation accuracy. We validate our stroke density estimation algorithm by comparing the estimated stroke density to
the real stroke density measured from recorded stroked history. For eachexample, in the first row we show the artists’ canvases, in the second row
we show the measured stroke density computed on the stroke history, inthe third row we show the estimated stroke density calculated on single
canvas images, and in the fourth row we show the error map between the
ground truth and the estimation. We also illustrate the diagrams of thiserror in the last row of each example, where the x-axis is the painting stroke
steps, and the y-axis is the PSNR accuracy.

the actual layer decomposition from [Tan et al. 2016] in Fig. 14,
where the results may suﬀer from foggy artifacts. We also replace
the stroke density map with the image luminance map as shown in
Fig. 15. Finally, we can pre-process noisy input images, e.g., images
with JPEG artifacts, with a CNN model, e.g., SRCNN [Dong et al.
2015] to improve the quality of the estimated stroke density map,
ACM Trans. Graph., Vol. 1, No. 1, Article 1. Publication date: January 2020.

1:10

•

Lvmin Zhang, Edgar Simo-Serra, Yi Ji, and Chunping Liu
Low Stroke Density Patches

15.63

High Stroke Density Patches

24.58

Low Stroke Density Patches

17.19

High Stroke Density Patches

35.53

Low Stroke Density Patches

17.25

High Stroke Density Patches

37.71

Illustration

Stroke History

Stroke Density

Manually Painted Effect

Artist’s Canvas

Manually Created Mask

Patch Visualization

Fig. 9. Relationship between stroke density and painted lighting eﬀects. We study the relationship between the density of stroke history and themanually painted lighting eﬀects. From left to right is the original image, screenshots of the stroke history, stroke density measured on the real stroke history,
lighting eﬀects created by artists, artists’ final canvases, background masks manually created by artists, and the patches sampled from the painted eﬀect. We
also present the mean standard deviation of all pixel colors in each patch group at the left-top corner of each group in the last col.
Table 1. Average Interaction Time and User Ranking. We report therecorded interaction time and user ranking (1 to 5 indicates worst to best)of diﬀerent algorithm and tool combinations. For the interaction time using
the combination of professional tools and a single approach, we show thetime corresponding to the professional tools and single approach separately.
Best results are shown in bold.

Single Tool / Combinations[Kender and Smith 1984][Wu et al. 2008][Yu and Smith 2019]
Ours
Professional Tool (PT) only
PT + [Kender and Smith 1984]
PT + [Wu et al. 2008]
PT + [Yu and Smith 2019]
PT + Ours

Time (s)Preference Rank

14.4
18.1
24.6
7.2

1.8 ± 0.4
3.0 ± 0.0
1.2 ± 0.4
4.0 ± 0.0

46.6
66.8 + 14.4
58.5 + 18.1
73.7 + 24.6
17.9 + 7.2

4.1 ± 0.3
1.8 ± 0.4
3.0 ± 0.0
1.2 ± 0.4
4.9 ± 0.3

4.7 Masked Lighting Eﬀect Generation
In speciﬁc use cases where the input image contains complicatedbackground structure, or when separated lighting eﬀects for theforeground objects are required, we can use third-party image segmentation methods, e.g., Mask-RCNN [He et al. 2018], to mat independent object components, and then generate lighting eﬀects
for the separated segments. In particular, we can directly multiply
masks to the coarse eﬀect map 𝑬. Because the stroke density maps
show strong capability to reﬁne coarse lighting eﬀects, even simple
and aliased masks predicted by Mask-RCNN may yield satisfactory
results, as shown in Fig. 18.

4.8 Additional Types of Light Sources
Our algorithm supports multiple types of light sources, which canbe implemented as a plug-in for professional photo editing software, e.g., Adobe Photoshop Lighting Eﬀect Filter, allowing users
to relight the image in a content-aware manner. Please refer to the
supplementary for detailed formulations.

4.9 Limitations

as shown in Fig. 16. Please refer to the supplemental material for
additional ablative results.
ACM Trans. Graph., Vol. 1, No. 1, Article 1. Publication date: January 2020.

As shown in Fig. 17, our generated lighting eﬀects may contain halolike artifacts, but these artifacts are not visible in the ﬁnal result.
Besides, as shown in Fig. 19, our approach is unable to modify hardshadows. Complicated background object structures or backgrounds

Generating Digital Painting Lighting Eﬀects via RGB-space Geometry

original image

background map

inked line

measured stroke density

DeepNormal

SpriteLamp

1:11

luminance map

result with K replaced

33:54

127:33

result with K replaced simple gradient map result with S replaced stroke density map our proposed method artist (conditioned) artist (unconditioned)

original image

background map

stroke history•

inked line

stroke history

measured stroke density

DeepNormal

SpriteLamp

luminance map

result with K replaced

45:05

198:54

result with K replaced simple gradient map result with S replaced stroke density map our proposed method artist (conditioned) artist (unconditioned)

Fig. 10. Ablation study. We present ablation results from using other alternative algorithms without using stroke density maps, as well as results obtained
from our algorithm by replacing stroke density maps with other alternative maps.

that are signiﬁcantly brighter than the foreground objects may
also interfered the wave transform. Nevertheless, in many cases,
this limitation can be addressed by combining third-party image
segmentation approaches as we mentioned before. Moreover, thedirectionality of this algorithm is limited to the front side of theimage canvas, and it is unable to put a light source behind the image
objects as this would require 3D scene understanding.

5

CONCLUSION

We have presented an algorithm to generate digital painting lighting
eﬀects to help simplify the current labor-intensive eﬀect composition workﬂow in illustrations or matte paintings. We have studied
the connection between the colors, strokes, and painted lighting
eﬀects, and developed an eﬃcient coarse-to-ﬁne approach based on
estimating the stroke density of the illustration. Qualitative results,
including a perceptual user study, corroborate the capability of ouralgorithm to help artists in their daily lighting eﬀect workﬂows. We
hope that our research will stimulate more researches related toartistic lighting eﬀect creation.

ACKNOWLEDGMENTS
We would like to thank Chengze Li for the helpful discussions. We
would like to thank all our reviewers for their insightful, constructive, and high-quality comments. This work is supported by National Natural Science Foundation of China (61972059, 61773272),
The Natural Science Foundation of the Jiangsu Higher Education
Institutions of China (19KJA230001), Suzhou Technology Development Plan (Key Industry Technology Innovation-Prospective Application Research Project SYG201807), Key Laboratory of Symbolic
Computation and Knowledge Engineering of Ministry of Education,
Jilin University (93K172016K08), the Priority Academic Program
Development of Jiangsu Higher Education Institutions. This work
is also supported by JST PRESTO (Simo-Serra, Grant Number: JPMJPR1756).

REFERENCES
Elad Aharoni-Mack, Yakov Shambik, and Dani Lischinski. 2017. Pigment-Based Recoloring of Watercolor Paintings. NPAR (2017).
Yagiz Aksoy, Tung Ozan Aydin, Aljosa Smolic, and Marc Pollefeys. 2017. Unmixingbased soft color segmentation for image manipulation. ACM Transactions on Graphics(2017).
ACM Trans. Graph., Vol. 1, No. 1, Article 1. Publication date: January 2020.

1:12

•

Lvmin Zhang, Edgar Simo-Serra, Yi Ji, and Chunping Liu

Original Image

Surface Optimizing (Wu et al. 2008)

CNN Normal Estimation (Yu & Smith 2019)

Generated Digital Painting Effects (ours)

Fig. 11. Comparison with non-digital-painting relighting approaches. For each input image we provide two results with diﬀerent lighting directions,which is illustrated in each example.
ACM Trans. Graph., Vol. 1, No. 1, Article 1. Publication date: January 2020.

Generating Digital Painting Lighting Eﬀects via RGB-space Geometry

Original

Kender&Smith 1984

Ours

Ours

Fig. 12. Comparison with height-to-normal-based relighting. We
compare our algorithm to the shape-from-shadow transform, where theper-pixel intensity is used as the height map.

Original Image

w/o Refinement

Proposed Full Method

Fig. 13. Comparison to results without high frequency refinement.
We compare our results with those without high frequency refinement
using stroke density maps. We directly use the coarse lighting eﬀect 𝑬 toshade the original image for the results in the second column, whereas theresults in the third column is obtained using our proposed approach.

Yagiz Aksoy, Changil Kim, Petr Kellnhofer, Sylvain Paris, Mohamed Elgharib, Marc
Pollefeys, and Wojciech Matusik. 2018. A Dataset of Flash and Ambient Illumination
Pairs from the Crowd. ECCV (2018).
J.T. Barron and J. Malik. 2012. Color constancy and intrinsic images and shape estimation.
ECCV (2012).
H. G. Barrow and J. M. Tenenbaum. 1978. Recovering intrinsic scene characteristics fromimages. In Computer Vision Systems, A. Hanson and E. Riseman (Eds.). Academic
Press, 3–26.
Sean Bell, Kavita Bala, and Noah Snavely. 2014. Intrinsic Images in the Wild. ACM
Transactions on Graphics 33, 4 (2014).
Sean Bell, Paul Upchurch, Noah Snavely, and Kavita Bala. 2013. OpenSurfaces: A Richly
Annotated Catalog of Surface Appearance. ACM Transactions on Graphics 32, 4(2013).
Sean Bell, Paul Upchurch, Noah Snavely, and Kavita Bala. 2015. Material Recognitionin the Wild with the Materials in Context Database. CVPR (2015).
Barber C. Bradford, Dobkin David P., and Huhdanpaa Hannu. 1996. The quickhull
algorithm for convex hulls. ACM Trans. Math. Software (1996).
Huiwen Chang, Ohad Fried, Yiming Liu, Stephen DiVerdi, and Adam Finkelstein. 2015.
Palette-based Photo Recoloring. ACM Transactions on Graphics (2015).

•

1:13

Jiansheng Chen, Guangda Su, Jinping He, and Shenglan Ben. 2010. Face Image Relighting using Locally Constrained Global Optimization. ECCV (2010).
Paul Debevec, Tim Hawkins, Chris Tchou, Haarm-Pieter Duiker, Westley Sarokin, andMark Sagar. 2000. Acquiring the reﬂectance ﬁeld of a human face. the 27th annual
conference on Computer graphics and interactive techniques (2000).
Chao Dong, Chen Change Loy, Kaiming He, and Xiaoou Tang. 2015. Image SuperResolution Using Deep Convolutional Networks. TPAMI (2015).
Graham Finlayson and Han Gong amd Robert B. Fisher. 2017. Color Homography:
Theory and Applications. TPAMI (2017).
P.V. Gehler, C. Rother, M. Kiefel, L. Zhang, and B. Scholkopf. 2011. Recovering intrinsic
images with a global sparsity prior on reﬂectance. NIPS (2011).
Kaiming He, Georgia Gkioxari, Piotr Dollar, and Ross Girshick. 2018. Mask R-CNN.
CVPR (2018).
Matis Hudon, Rafael Pages, Mairead Grogan, and Aljosa Smolic. 2018. Deep Normal
Estimation for Automatic Shading of Hand-Drawn Characters. ECCV (2018).
Yoshihiro Kanamori and Yuki Endo. 2018. Relighting humans: occlusion-aware inverse
rendering for full-body human images. In ACM Transactions on Graphics.
John R. Kender and Earl Smith. 1984. Shape from Darkness: Deriving Surface Information from Dynamic Shadows. National Conference on Artiﬁcial Intelligence (1984).
Yuki Koyama and Masataka Goto. 2018. Decomposing Images into Layers with Advanced Color Blending. In Computer Graphics Forum, Vol. 37. Wiley Online Library,397–407.
Sharon Lin, Matthew Fisher, Angela Dai, and Pat Hanrahan. 2017. LayerBuilder: Layer
Decomposition for Interactive Image and Video Color Editing. Arxiv (2017).
Wojciech Matusik, Matthew Loper, and Hanspeter Pﬁster. 2004. Progressively-Reﬁned
Reﬂectance Functions from Natural Illumination. Eurographics Workshop on Rendering (2004).
Pieter Peers and Philip Dutre. 2005. Inferring reﬂectance functions from wavelet noise.
the Sixteenth Eurographics conference on Rendering Techniques (2005).
Pieter Peers, Dhruv K Mahajan, Bruce Lamond, Abhijeet Ghosh, Wojciech Matusik,
Ravi Ramamoorthi, and Paul Debevec. 2009. Compressive light transport sensing.
ACM Transactions on Graphics (2009).
Pieter Peers, Naoki Tamura, Wojciech Matusik, and Paul Debevec. 2007. Post-production
Facial Performance Relighting using Reﬂectance Transfer. ACM Transactions on
Graphics (2007).
Julien Philip, Michael Gharbi, Tinghui Zhou, Alexei, Efros, and George Drettakis. 2019.
Multi-view Relighting using a Geometry-Aware Network. SIGGRAPH (2019).
Soumyadip Sengupta, Angjoo Kanazawa, Carlos D. Castillo, and David W. Jacobs. 2018.
SfSNet: Learning Shape, Refectance and Illuminance of Faces in the Wild. In CVPR.
M. Serra, O. Penacchio, R. Benavente, and M. Vanrell. 2012. Names and shades of color
for intrinsic image estimation. CVPR (2012).
Jianbing Shen, Xiaoshan Yang, Yunde Jia, and Xuelong Li. 2011. Intrinsic Images Using
Optimization. CVPR (2011).
YiChang Shih, Sylvain Paris, Fredo Durand, and William T. Freeman. 2013. Datadriven Hallucination for Diﬀerent Times of Day from a Single Outdoor Photo. ACM
Transactions on Graphics (2013).
Wanchao Su, Dong Du, Xin Yang, Shizhe Zhou, and Hongbo Fu. 2018. Interactive SketchBased Normal Map Generation with Deep Neural Networks. ACM on Computer
Graphics and Interactive Techniques 1, 1 (jul 2018), 1–17.
Tiancheng Sun, Jonathan T. Barron, Yun-Ta Tsai, Zexiang Xu, Xueming Yu, Graham
Fyﬀe, Christoph Rhemann, Jay Busch, Paul Debevec, and Ravi Ramamoorthi. 2019.
Single Image Portrait Relighting. SIGGRAPH (2019).
Daniel Sykora, Ladislav Kavan, Martin Cadik, Ondrej Jamriska, Alec Jacobson, BrianWhited, Maryann Simmons, and Olga Sorkine-Hornung. 2014. Ink-and-Ray: BasRelief Meshes for Adding Global Illumination Eﬀects to Hand-Drawn Characters.
ACM Transactions on Graphics 33, 2 (apr 2014), 1–15.
Jianchao Tan, Jose Echevarria, and Yotam Gingold. 2018. Eﬃcient palette-based decomposition and recoloring of images via RGBXY-space geometry. ACM Transactions
on Graphics (2018).
Jianchao Tan, Jyh-Ming Lien, and Yotam Gingold. 2016. Decomposing Images into
Layers via RGB-space Geometry. ACM Transactions on Graphics (2016).
Tai-Pang Wu, Jian Sun, Chi-Keung Tang, and Heung-Yeung Shum. 2008. Interactive
normal reconstruction from a single image. ACM Transactions on Graphics 27, 5(dec 2008), 1.
Tai-Pang Wu, Chi-Keng Tang, Michael S. Brown, and Heung-Yeung Shum. 2007.
ShapePalettes: Interactive Normal Transfer via Sketching. ACM Transactions on
Graphics (2007).
Qiuying Xu, Yotam Gingold, and Karan Singh. 2015. Inverse Toon Shading: Interactive
Normal Field Modeling with Isophotes. Sketch-Based Interfaces and Modeling (2015).
Ye Yu and William A. P. Smith. 2019. InverseRenderNet: Learning single image inverse
rendering. CVPR (2019).
Qing Zhang, Chunxia Xiao, Hanqiu Sun, and Feng Tang. 2017. Palette-Based Image
Recoloring Using Color Decomposition Optimization. IEEE Transactions on Image
Processing (2017).

ACM Trans. Graph., Vol. 1, No. 1, Article 1. Publication date: January 2020.

1:14

•

Lvmin Zhang, Edgar Simo-Serra, Yi Ji, and Chunping Liu

Original Image

Simplified Convex Hull

Decomposition-based Stroke Density

Proposed Approach

Fig. 14. Comparison to results with other color geometry algorithm configurations. We analyze the eﬀect of replacing our convex-hull-based palettewith the simplified hull proposed in [Tan et al. 2016], and also replacing our proposed stroke density estimation algorithm with the actual stroke decompositionand density calculation. In each case we visualize the lighting eﬀect map and the final rendered result. The light position used is indicated in the original
figure. Plane © Anonymous.

Original image

Lighting effects

Ours

Fig. 19. Limitations. Our algorithm is easily interfered by hard shadows
and is unable to modify them. Additionally, our algorithm may fail when the
background is significantly brighter than the foreground. Nevertheless, thisshortcoming can be meliorated when masks are applied to the proposed
algorithm.

Original images

Mask-RCNN

Ours

Ours

Fig. 18. Additional application: masked lighting eﬀect generation.
Our algorithm can be combined with simple foreground segmentation
methods like Mask-RCNN to generate object-level lighting eﬀects.

Original Image

Guided by Stroke Density

Fig. 15. Replacing stroke density maps with naive luminance maps.
We compare an image processed using our approach guided by luminancemaps and stroke density maps.

(a)

ACM Trans. Graph., Vol. 1, No. 1, Article 1. Publication date: January 2020.

Guided by Luminance

(b)

(c)

(d)

Fig. 16. Influence of the pre-processing. (a) The original image compressed with 0.85 JPEG quality. (b,c) The estimated stroke density map withpre-processing. (d) The result without pre-processing.

Generating Digital Painting Lighting Eﬀects via RGB-space Geometry

Original Image

Lighting Effect

•

1:15

OutputFig. 17. Invisible halo-like artifacts. Some halo-like Artifacts, as marked
with the red arrow, can be found surrounding the leaves in the lighting
eﬀects in the second column. However, in the final result of the third column,
these artifacts become invisible.

ACM Trans. Graph., Vol. 1, No. 1, Article 1. Publication date: January 2020.