Decomposing Images into Layers with
Advanced Color Blending
Yuki Koyama & Masataka Goto

Quick Summary

Usage Scenario

Import the resulting layers
to Photoshop etc.
Input images except for the rightmost one courtesy of David Revoy

Perform non-trivial edits(e.g., lighting-aware hue change)

Background
What is Advanced Color Blending?

Color Blending & Blend Mode
A color-blend mode
defines the mapping fromthe colors of the layers to thecolor that will be rendered
Examples: multiply, overlay,
color-dodge, normal, etc.
Available in…

Yuki Koyama and Masataka Goto | Decomposing Images into Layers with Advanced Color Blending | Comput. Graph. Forum (Pacific Graphics 2018)7

Digital Drawing & Effects with Color Blending
Advanced (non-linear) color
blending is used for creatinginteresting color eﬀects

David Revoy:“Painting with Blending-modes”
https://youtu.be/AybFWViT-3Q

Example:digital drawing using
the color-dodge mode

Yuki Koyama and Masataka Goto | Decomposing Images into Layers with Advanced Color Blending | Comput. Graph. Forum (Pacific Graphics 2018)8

Digital Drawing & Effects with Color Blending
It is important to combine various blend modes

Yuki Koyama and Masataka Goto | Decomposing Images into Layers with Advanced Color Blending | Comput. Graph. Forum (Pacific Graphics 2018)9

Digital Drawing & Effects with Color Blending
It is important to combine various blend modes(N/A)

normalnormalmultiply
multiply hard-light
hard-lightscreenscreenYuki Koyama and Masataka Goto | Decomposing Images into Layers with Advanced Color Blending | Comput. Graph. Forum (Pacific Graphics 2018)10

Digital Drawing & Effects with Color Blending
It is important to combine various blend modes(N/A)

normalnormalmultiply
multiply hard-light
hard-lightscreenscreenNote: every digital artist has his/her own way
to combine blend modes
Yuki Koyama and Masataka Goto | Decomposing Images into Layers with Advanced Color Blending | Comput. Graph. Forum (Pacific Graphics 2018)11

Our Motivation

Motivation: Color Unblending
Arbitrarycombination ofblend modes
Unblending(N/A)

It should be useful if an existing image
can be decomposed into layers with anarbitrary combination of advanced
color-blend modes
normalnormalmultiply
multiply hard-light
hard-light

Yuki Koyama and Masataka Goto | Decomposing Images into Layers with Advanced Color Blending | Comput. Graph. Forum (Pacific Graphics 2018)

screenscreen13

Previous Work
Layer Decomposition
by Color Unblending

RGB-Space Geometry [Tan+, TOG 16]•

Support only a specific linear
blend mode (i.e., “normal”)•

Heavily rely on the linearity of theblend mode•

Cannot be (easily) extended for
non-linear color-blend modes

Yuki Koyama and Masataka Goto | Decomposing Images into Layers with Advanced Color Blending | Comput. Graph. Forum (Pacific Graphics 2018)15

Per-Pixel Optimization [Aksoy+, TOG 16; TOG 17]•

Support only aspecific linear
blend mode (and alinear alpha addition)

Yuki Koyama and Masataka Goto | Decomposing Images into Layers with Advanced Color Blending | Comput. Graph. Forum (Pacific Graphics 2018)16

Per-Pixel Optimization [Aksoy+, TOG 16; TOG 17]•

Support only aspecific linear
blend mode (and alinear alpha addition)

We generalize this method to support
advanced (non-linear) color-blend modes
Yuki Koyama and Masataka Goto | Decomposing Images into Layers with Advanced Color Blending | Comput. Graph. Forum (Pacific Graphics 2018)17

Details of Our Method [1/2]
Per-Pixel Unblending Optimization(Concept-Level Explanation)

Problem Setting: Per-Pixel Unblending
Target image

Yuki Koyama and Masataka Goto | Decomposing Images into Layers with Advanced Color Blending | Comput. Graph. Forum (Pacific Graphics 2018)19

Problem Setting: Per-Pixel Unblending
Target image

Targetpixel color⇥r, g, b, ↵⇤The pixel that we aregoing to handle now

Yuki Koyama and Masataka Goto | Decomposing Images into Layers with Advanced Color Blending | Comput. Graph. Forum (Pacific Graphics 2018)20

Problem Setting: Per-Pixel Unblending
Targetpixel color⇥r, g, b, ↵⇤Yuki Koyama and Masataka Goto | Decomposing Images into Layers with Advanced Color Blending | Comput. Graph. Forum (Pacific Graphics 2018)21

Problem Setting: Per-Pixel Unblending
Targetpixel colorr, g, b, ↵⇤Unblending…

⇥

multiply

screennormalYuki Koyama and Masataka Goto | Decomposing Images into Layers with Advanced Color Blending | Comput. Graph. Forum (Pacific Graphics 2018)22

Problem Setting: Per-Pixel Unblending
Blend mode
Targetpixel colorr, g, b, ↵⇤Unblending…

⇥

multiply

Desirable color distribution:
the distribution of RGB values thatthe decomposed layer contains

screennormalYuki Koyama and Masataka Goto | Decomposing Images into Layers with Advanced Color Blending | Comput. Graph. Forum (Pacific Graphics 2018)23

Problem Setting: Per-Pixel Unblending
Input

Targetpixel colorr, g, b, ↵⇤Unblending…

⇥

multiply

screennormalYuki Koyama and Masataka Goto | Decomposing Images into Layers with Advanced Color Blending | Comput. Graph. Forum (Pacific Graphics 2018)24

Problem Setting: Per-Pixel Unblending
Input

Targetpixel color
Unblending

screennormal⇥⇥⇥rn , gn , bn , ↵n…

r, g, b, ↵⇤multiply…

⇥

Outputr2 , g2 , b2 , ↵2
r1 , g1 , b1 , ↵1⇤⇤⇤

Yuki Koyama and Masataka Goto | Decomposing Images into Layers with Advanced Color Blending | Comput. Graph. Forum (Pacific Graphics 2018)25

Problem Setting: Per-Pixel Unblending
Input

Targetpixel color
Unblending

screennormal⇥⇥⇥rn , gn , bn , ↵n…

r, g, b, ↵⇤multiply…

⇥

Outputr2 , g2 , b2 , ↵2
r1 , g1 , b1 , ↵1

Composited
pixel color⇤⇤

Blending⇤

Yuki Koyama and Masataka Goto | Decomposing Images into Layers with Advanced Color Blending | Comput. Graph. Forum (Pacific Graphics 2018)⇥r̂, ĝ, b̂, ↵
ˆ

⇤

26

Problem Setting: Per-Pixel Unblending
Input

Targetpixel color
Unblending

screennormal⇥⇥⇥rn , gn , bn , ↵n…

r, g, b, ↵⇤multiply…

⇥

Outputr2 , g2 , b2 , ↵2
r1 , g1 , b1 , ↵1

Should equal

Composited
pixel color⇤⇤

Blending⇤

Yuki Koyama and Masataka Goto | Decomposing Images into Layers with Advanced Color Blending | Comput. Graph. Forum (Pacific Graphics 2018)⇥r̂, ĝ, b̂, ↵
ˆ

⇤

27

Problem Setting: Per-Pixel Unblending
Input

Targetpixel color
Unblending⇥⇥

rn , gn , bn , ↵n…

r, g, b, ↵⇤multiply…

⇥

OutputscreenShould be r2 , g2 , b2 , ↵2

as consistent as possible⇥
normalr1 , g1 , b1 , ↵1

Should equal

Composited
pixel color⇤⇤

Blending⇤

Yuki Koyama and Masataka Goto | Decomposing Images into Layers with Advanced Color Blending | Comput. Graph. Forum (Pacific Graphics 2018)⇥r̂, ĝ, b̂, ↵
ˆ

⇤

28

Example User Interaction

Input image courtesy of David Revoy

Basics (Not Our Method)
Math of Advanced Color BlendingDescriptor of a Pixel Color
Target pixel
Layer

Layer

Descriptor of the i-th layer’s pixel⇥⇤T
x i = ri g i bi ↵ i⇥ T⇤T4= ci ↵ i 2 R

Layer

Yuki Koyama and Masataka Goto | Decomposing Images into Layers with Advanced Color Blending | Comput. Graph. Forum (Pacific Graphics 2018)32

Descriptor of a Pixel Color
Target pixel
Layer

Layer

Layer

Descriptor of the i-th layer’s pixel⇥⇤T
x i = ri g i bi ↵ i⇥ T⇤T4= ci ↵ i 2 R

Descriptor of all the layers’ pixels
⇥ T
⇤
T
4n
T
x = x1 · · · xn 2 R
Yuki Koyama and Masataka Goto | Decomposing Images into Layers with Advanced Color Blending | Comput. Graph. Forum (Pacific Graphics 2018)33

General Composition Operator () for Compositing Two Layers

General composition operator

xs

?

xd
Yuki Koyama and Masataka Goto | Decomposing Images into Layers with Advanced Color Blending | Comput. Graph. Forum (Pacific Graphics 2018)34

General Composition Operator () for Compositing Two Layers

f
(c
,
c
)↵
↵
+
Y
↵
(1
↵
)c
+
Z↵
(1
s
d
s
d
s
d
s
d
rgb
xs
xd =
↵
xs
xd
↵
xs
xd = X↵s ↵d + Y ↵s (1 ↵d ) + Z↵d (1 ↵s )xs↵s )cd?

xd“SVG Compositing Specification” by W3C: https://www.w3.org/TR/2011/WD-SVGCompositing-20110315/
Yuki Koyama and Masataka Goto | Decomposing Images into Layers with Advanced Color Blending | Comput. Graph. Forum (Pacific Graphics 2018)35

General Composition Operator () for Compositing Two Layers

f
(c
,
c
)↵
↵
+
Y
↵
(1
↵
)c
+
Z↵
(1
s
d
s
d
s
d
s
d
rgb
xs
xd =
↵
xs
xd
↵
xs
xd = X↵s ↵d + Y ↵s (1 ↵d ) + Z↵d (1 ↵s )↵s )cdBlend function: Mode-specific function (can be non-linear)33f : [0, 1] ⇥ [0, 1] ! [0, 1]

3For example: fnormal , fmultiply , fcolor-dodge , . . .
“SVG Compositing Specification” by W3C: https://www.w3.org/TR/2011/WD-SVGCompositing-20110315/
Yuki Koyama and Masataka Goto | Decomposing Images into Layers with Advanced Color Blending | Comput. Graph. Forum (Pacific Graphics 2018)36

More-Than-Two-Layers Case: Recursive Layering
Recursive rule:

x̂k =

⇢

x1
xk

k x̂k 1

x4

x̂4
x̂3

x3
x2
x1

x̂2

(k = 1)(otherwise)

Final RGBA

overlaycolor-dodge

multiply

Yuki Koyama and Masataka Goto | Decomposing Images into Layers with Advanced Color Blending | Comput. Graph. Forum (Pacific Graphics 2018)37

Details of Our Method [2/2]
Per-Pixel Unblending Optimization(Equation-Level Explanation)

Unblending Optimization…

Unblending
More consistentLess consistent

Color consistency:
Each pixel color should be as
consistent to the specified color
distribution as possible

Blending

Equal

Color reproducibility:
The target pixel color should be
reproduced by compositing theresulting layer colors

Yuki Koyama and Masataka Goto | Decomposing Images into Layers with Advanced Color Blending | Comput. Graph. Forum (Pacific Graphics 2018)39

Unblending Optimization
Equality-constrained optimization problem
min4n E(x) s.t. C(x) = 0

x2R

Objective:
Each pixel color should be as
consistent to the specified color
distribution as possible

Equality constraint:
The target pixel color should be
reproduced by compositing theresulting layer colors

Yuki Koyama and Masataka Goto | Decomposing Images into Layers with Advanced Color Blending | Comput. Graph. Forum (Pacific Graphics 2018)40

Unblending Optimization
Equality-constrained optimization problem
min4n E(x) s.t. C(x) = 0

x2R

Objective:
Each pixel color should be as
consistent to the specified color
distribution as possible

Equality constraint:
The target pixel color should be
reproduced by compositing theresulting layer colors

Yuki Koyama and Masataka Goto | Decomposing Images into Layers with Advanced Color Blending | Comput. Graph. Forum (Pacific Graphics 2018)41

Constraint Function Definition
C(x) = (xnn (xn 1

E.g., multiply

n 1 (xn 2

n 2 (. . .))))

color-dodge

overlay …

ctargetYuki Koyama and Masataka Goto | Decomposing Images into Layers with Advanced Color Blending | Comput. Graph. Forum (Pacific Graphics 2018)=042

Constraint Function Definition
C(x) = (xnn (xn 1

E.g., multiply

n 1 (xn 2

n 2 (. . .))))

color-dodge

overlay …

ctarget=0C.f., [Aksoy+16; 17]: Assumed a special case of a linear modelC(x) =nX↵ i cictarget=0i

Yuki Koyama and Masataka Goto | Decomposing Images into Layers with Advanced Color Blending | Comput. Graph. Forum (Pacific Graphics 2018)43

Constraint Function Definition
C(x) = (xnn (xn 1

E.g., multiply

n 1 (xn 2

n 2 (. . .))))

color-dodge

overlay …

ctarget=0C.f., [Aksoy+16; 17]: Assumed a special case of a linear modelC(x) =nXi↵ i cictarget=0

Our method is a
generalization of[Aksoy+16; 17]

Yuki Koyama and Masataka Goto | Decomposing Images into Layers with Advanced Color Blending | Comput. Graph. Forum (Pacific Graphics 2018)

44

Solving Equality-Constrained Optimization [1/2]
Algorithm: The augmented Lagrangian method•

Iteratively solve (unconstrained) optimizations

while (not converged) {
n
o
⇢
T
2
solve min E(x)C(x) + kC(x)kx2
updateand ⇢}Yuki Koyama and Masataka Goto | Decomposing Images into Layers with Advanced Color Blending | Comput. Graph. Forum (Pacific Graphics 2018)

45

Solving Equality-Constrained Optimization [1/2]
Algorithm: The augmented Lagrangian method•

Iteratively solve (unconstrained) optimizations

while (not converged) {
n
o
⇢
T
2
solve min E(x)C(x) + kC(x)kx2
updateand ⇢}Need thederivative of C(x)to eﬃciently solve
this optimization

Yuki Koyama and Masataka Goto | Decomposing Images into Layers with Advanced Color Blending | Comput. Graph. Forum (Pacific Graphics 2018)

46

Solving Equality-Constrained Optimization [2/2]
It is possible to algorithmically calculate
the derivative of C(x) by recursively applying the chain rule
8
>
I
<

@C(x)
@
(x
x̂
)
k
k
k
1
=
@xk
>
@xi
: @ x̂k 1 · @ (xk@xi@ x̂k 1k x̂k 1 ),(i = k = 1)
(i = k 6= 1)(otherwise)

Yuki Koyama and Masataka Goto | Decomposing Images into Layers with Advanced Color Blending | Comput. Graph. Forum (Pacific Graphics 2018)47

Solving Equality-Constrained Optimization [2/2]
It is possible to algorithmically calculate
the derivative of C(x) by recursively applying the chain rule
8
>
I
<

@C(x)
@
(x
x̂
)
k
k
k
1
=
@xk
>
@xi
: @ x̂k 1 · @ (xk@xi@ x̂k 1k x̂k 1 ),(i = k = 1)
(i = k 6= 1)(otherwise)

Our unblending optimization problem can be eﬃciently
solved by gradient-based algorithms (e.g., L-BFGS)
Yuki Koyama and Masataka Goto | Decomposing Images into Layers with Advanced Color Blending | Comput. Graph. Forum (Pacific Graphics 2018)48

Unblending using linear blend mode
➡ Identical to the results by [Aksoy+17]

Unblending using various blend modes

color-dodge
Input image courtesy of David Revoymultiply

overlayPost-Processing
Refinement for SmoothnessRefinement Step for Smoothness■ Problem:
Resulting layers may not be smoothbecause the unblending optimization is“per-pixel” (no inter-pixel consideration)

Not smooth■ Solution:
Performing a procedure called “matterefinement” [Aksoy+17] as a postprocessing

SmoothInput image courtesy of David Revoy
Yuki Koyama and Masataka Goto | Decomposing Images into Layers with Advanced Color Blending | Comput. Graph. Forum (Pacific Graphics 2018)

51

Refinement Step for Smoothness■ Problem:
Resulting layers may not be smoothbecause the unblending optimization is“per-pixel” (no inter-pixel consideration)

Not smooth■ Solution:
Performing a procedure called “matterefinement” [Aksoy+17] as a postprocessing to enforce smoothness

SmoothInput image courtesy of David Revoy
Yuki Koyama and Masataka Goto | Decomposing Images into Layers with Advanced Color Blending | Comput. Graph. Forum (Pacific Graphics 2018)52

Usage Scenario
Example: Photoshop

Input image courtesy of David RevoyApplications #1
Lighting-Aware Editing of
an Existing Illustration

Input image courtesy of David RevoyApplications #2
Bringing a Physical Painting
into Digital Workflow

Other Applications

Intrinsic image
decomposition:
decomposing an image intoreflectance and shading layers(N/A)

multiply(N/A)

multiply

Input images are provided by [Innamorati+17]

screenAdding gradation to an
existing conference logo:
extracting a “fill” layer from thelogo and re-fill with gradation

Remixing existing illustrations:Harmonize colors of images from diﬀerent scenes to put them
together in a derivative work

Input image courtesy of David RevoySummarySummary■ Solution for a new problem•

Decomposing an image intolayers with advanced (nonlinear) color blending■ Techniques•

Generalizing Aksoy+’s method [16; 17] for handling advanced (nonlinear) color blend modes■ Future work•

Automatic determination of optimal color distributions etc.
Yuki Koyama and Masataka Goto | Decomposing Images into Layers with Advanced Color Blending | Comput. Graph. Forum (Pacific Graphics 2018)63In preparation…
We are preparing for releasing our source codes at GitHub
so please check our project page later

Decomposing Images into Layers with Advanced Color Blending
Yuki Koyama & Masataka Goto

Decomposing Images into Layers with Advanced Color Blending
Yuki Koyama & Masataka Goto{{

Two Main Usages of Layers

For segmenting regions(not our focus)

For blending colors(our focus)

Yuki Koyama and Masataka Goto | Decomposing Images into Layers with Advanced Color Blending | Comput. Graph. Forum (Pacific Graphics 2018)66

Specialization of the Composite Operatorf (cs , cd ) = cs ,f (cs , cd ) = cs + cd ,X=Y =Z=1

X = 2, Y = Z = 1

↵
c
+
(1
↵
)↵
c
s
s
s
d
d
rgb
xs
xd =
,
↵s ↵ ↵d↵↵s↵d = ↵s + (1 ↵s )↵dAlpha blending model
C.f., [Tan+16]xs↵s↵ s cs + ↵ d cd
xd =
,
↵
↵s
↵d
↵
↵d = ↵s + ↵drgb

Linear additive model
C.f., [Aksoy+16; 18]

Yuki Koyama and Masataka Goto | Decomposing Images into Layers with Advanced Color Blending | Comput. Graph. Forum (Pacific Graphics 2018)67

Computational Cost
1,400 s1,400 s 1,400 s

500 s 500 s

500 s

160 s 160 s

160 s

1,050 s1,050 s 1,050 s

375 s 375 s

375 s

120 s 120 s

120 s

700 s 700 s

700 s

250 s 250 s

250 s

80 s

80 s

80 s

350 s 350 s

350 s

125 s 125 s

125 s

40 s

40 s

40 s

0s

0s
1

0s
112

12
172

72
12

0s

0s
0s
0s
0s
72
0k
0 k 500 k0 k500 k1,000
500
k1,000
k k 1,000 2k0s
3 2 4 3 52 4 63 5

46

5

6

Numberof Numberlayersof layersof layers
NumberNumberof NumberCPUof CPUcores
ofcores
CPU cores
NumberNumberof Numberpixelsof pixelsof pixelsNumber•

Parallelization iseffective•

Increase cost alongwith the number ofpixels•Increase cost alongwith the number oflayers (due to therecursive calculation)

Yuki Koyama and Masataka Goto | Decomposing Images into Layers with Advanced Color Blending | Comput. Graph. Forum (Pacific Graphics 2018)

68Fixing Deep Colorization via Layer Decomposition(N/A)

Sketch& scribbles

overlay

overlay(N/A)

multiply

Deep-colorized
image

Compositedimage

Final image
with effects

Our method enables a new digital painting workflow by effectively using such a deep-colorized image as a good starting
point. Instead of directly editing the generated single-layered image, the user can decompose it into layers with familiar
advanced color blending then individually edit each unsatisfactory layer by adding strokes, changing hue, etc.

Yuki Koyama and Masataka Goto | Decomposing Images into Layers with Advanced Color Blending | Comput. Graph. Forum (Pacific Graphics 2018)69

Lighting-Aware Replacement of BackgroundnormalNaïve composition witha different background

Original image

Manual segmentation

Composition witha different backgroundRe-composition
after layer editing(No edit)

normal(No edit)overlay

Hue change

Decomposed layers and edits on them

Example use of our method, in which the blue sky background is replaced with a sunset one. With our method, thecharacter region is decomposed into three layers: the bottom two layers roughly correspond to albedo, and the top layerroughly corresponds to lighting by the blue sky represented as an overlay layer. To match the lighting in sunset, the top
layer’s hue is modified. Finally, the layers are re-composited with the sunset background. Note that naïvely compositing thecharacter region with the sunset background (the leftmost image) does not look good because lighting is mismatched.
Input image courtesy of David Revoy.
Yuki Koyama and Masataka Goto | Decomposing Images into Layers with Advanced Color Blending | Comput. Graph. Forum (Pacific Graphics 2018)70