BLENDING TEXTURE FEATURES FROMMULTIPLE REFERENCE IMAGES FOR STYLE TRANSFER
Hikaru Ikuta*1,*2, Keisuke Ogaki*2,Yuri Odagiri*2*1The University of Tokyo *2Dwango, Co. ltd.

+

OUR SYSTEM“Watercolor”+Content Image

Texture Image Set

Output Image

Objective: Obtain an image painted in the texture of a given set of images
Input: Content Image and Texture Image Set
Tested with a novel dataset “nico-illust” containing 500,000 digital paintings

Original Video

Our Results
The input style image set (50 images)

MOTIVATIONGatys, et al. [1]+Content Image

Texture Image Set

Output Image[1] Gatys, L. A., et al. 2016. Image style transfer using convolutional neural networks. CVPRMOTIVATION
Gatys, et al. [1]+Content Image

Texture Image Set

Output Image

Uses only one image, so…
The color depends heavily on the input texture image
The color correspondence of the original image is often lost[1] Gatys, L. A., et al. 2016. Image style transfer using convolutional neural networks. CVPRMOTIVATION
Gatys, et al. [1]+Content Image

Texture Image Set

Output Image

Uses only one image, so…
The color depends heavily on the input texture image
The color correspondence of the original image is often lostIs this really a “style” transfer?
[1] Gatys, L. A., et al. 2016. Image style transfer using convolutional neural networks. CVPRMOTIVATION
Gatys, et al. [1]The color of one artwork+Content Image

6= The “style”
Texture Image SetOutput Image

Uses only one image, so…
The color depends heavily on the input texture image
The color correspondence of the original image is often lostIs this really a “style” transfer?
[1] Gatys, L. A., et al. 2016. Image style transfer using convolutional neural networks. CVPRMOTIVATION
Gatys, et al. [1]+Content Image

Texture Image Set

Output Image

Uses only one image, so…
The color depends heavily on the input texture image
The color correspondence of the original image is often lostIs this really a “style” transfer?
[1] Gatys, L. A., et al. 2016. Image style transfer using convolutional neural networks. CVPRMOTIVATION
Gatys, et al. [1]+Content Image

Texture Image Set

Output Image

Texture Image Set

Output Image

Our Method+
Content Image[1] Gatys, L. A., et al. 2016. Image style transfer using convolutional neural networks. CVPRMOTIVATIONGatys, et al. [1]Our method…
Preserves the original+ color, while extracting the texture of thetexture image set
Therefore, it can transfer the same style onto different images
Content Image

Texture Image Set

Output Image

Texture Image Set

Output Image

Our Method+Content Image[1] Gatys, L. A., et al. 2016. Image style transfer using convolutional neural networks. CVPR

THE METHODSYSTEM OVERVIEW (FORMER)Neural Style Transfer is a joint distanceminimization problem
ContentFeatureF
Content ImageGTexture Image

TextureFeature

233.1
64.17
6 7
65.97
6 7
62.67
4 5..
.

23
2.7
61.87
6 7
64.27
6 7
68.47
4 5..
.

JointDistance
Minimization
Output ImageSYSTEM OVERVIEW (FORMER)Neural Style Transfer is a joint distanceThe
output
image
tries
tobe
close
tothe:
minimization problem- Content image in the “Content Feature” space-ContentTexture image
in the “Texture Feature” space
2 3Feature

FContent Image

GTexture Image

TextureFeature

3.1
64.17
6 7
65.97
6 7
62.67
4 5..
.

23
2.7
61.87
6 7
64.27
6 768.47
4 5..
.

JointDistance
Minimization
Output ImageSYSTEM OVERVIEW (FORMER)Neural Style Transfer is a joint distanceminimization problem
ContentFeature

F
Content Image

GTexture Image

TextureFeature

23
3.1
64.17
6 7
65.97
6 7
62.67
4 5..
.

23
2.7
61.87
6 7
64.27
6 7
68.47
4 5..
.

JointDistance
Minimization
Output ImageG:
SYSTEMOVERVIEW(FORMER)The Texture
Feature function
Rany⇥any(Image of arbitrary size)

Ntexture

R(Fixed size vector)

Neural Style Transfer is a joint distance
2 33.1
64.17
minimization problem
6 7ContentFeature

ContentFeature

F F

Content Image

FGG

Texture Image

TextureFeature

GG

23
3.1
64.17
6 7
65.97
6 7
62.67
4 5..
.

32.7
61.87
6 7
64.27
6 7
68.47
4 5..
.

G2

65.97
6 7
62.67
4 5..
.
2 3
2.7
61.87
6 7
64.27
6 7
7
Joint 648.45
Distance ..
.

Minimization

G230.5
67.77
6 7
62.17
6 7
65.67
4 5..
.

Output Image

GRany⇥any…

Middle layer channels ofthe CNN[ ]correlation
matrixDimension:
Number of filtersin the CNN
squared
Vectorize

R233.1
64.17
6 7
65.97
6 7
62.67
4 5..
.

NtextureSYSTEM OVERVIEW (FORMER)Neural Style Transfer is a joint distanceminimization problem
ContentFeature

F
Content Image

GTexture Image

TextureFeature

23
3.1
64.17
6 7
65.97
6 7
62.67
4 5..
.
23
2.7
61.87
6 7
64.27
6 7
68.47
4 5..
.

JointDistance
Minimization
Output ImageSYSTEM OVERVIEW (PROPOSED)
Neural Style Transfer is a joint distanceminimization problem
ContentFeature

F
Content Image

OptimalBlending OptimalTexture
FeatureNovelty of our system

23
3.1
64.17
6 765.97
6 762.67
4 5..
.
232.7
61.87
6 7
64.27
6 768.47
4 5..
.

JointDistance
Minimization
Output ImageSYSTEM OVERVIEW (PROPOSED)
Neural Style Transfer is a joint distanceminimization problem
Content
2 3
3.1a better texture feature that
GenerateFeature
64.17
6 7the colors of the content- Preserves
65.97
6 7
62.67 the author/genre’s “style”- Represents
4 5..
.
Joint

F

Content Image

OptimalBlending Optimal
TextureFeatureNovelty of our system

232.7
61.87
6 7
64.27
6 768.47
4 5..
.

Distance
Minimization

Output ImageSYSTEM OVERVIEW (PROPOSED)
Neural Style Transfer
isa
joint
distance
How? We use one key observation…
minimization problem
Content
2 3
3.1a better texture feature that
GenerateFeature
64.17
6 7the colors of the content- Preserves
65.97
6 7
62.67 the author/genre’s “style”- Represents
4 5..
.
JointF

Content Image

OptimalBlending Optimal
TextureFeatureNovelty of our system

23
2.7
61.87
6 7
64.27
6 7
68.47
4 5..
.

Distance
Minimization

Output ImageKEY OBSERVATION
Assume that: a concatenated image of some style belongs to the same styleKEY OBSERVATION
Assume that: a concatenated image of some style belongs to the same styleG(( ))G( )KEY OBSERVATION
Assume that: a concatenated image of some style belongs to the same styleG(( ))
G( )

1⇡ G
31+ G
31+ G
3

G( ) G( ) G( )It is the linear combination of the texture features of each image
Approximation is due to padding at boundaries in the CNN
Derived from the properties of G :(Proof) shown in paper

KEY OBSERVATION
Assume that: a concatenated image of some style belongs to the same styleG(( ))
G( )

1⇡ G
31+ G
31+ G
3

G( ) G( ) G( )It is the linear combination of the texture features of each image
Approximation is due to padding at boundaries in the CNN
Derived from the properties of G :(Proof) shown in paper
Linear combinations of texture features from a certain style,is again a texture feature of that styleKEY OBSERVATION
Assume that: a concatenated image of some style belongs to the same styleG(( ))
G( )

1⇡ G
31+ G
31+ G
3

G( ) G( ) G( )It is the linear combination of the texture features of each image
Approximation is due to padding
boundaries
in ittheCNN
Letatusunderstandgeometrically…
Derived from the properties of G :(Proof) shown in paper
Linear combinations of texture features from a certain style,is again a texture feature of that styleBLENDING IN THE “STYLE SPACE”

BLENDING IN THE “STYLE SPACE”

Dimensions
in theTexture
Feature G.

BLENDING IN THE “STYLE SPACE”㻌㻌㻌BLENDING IN THE “STYLE SPACE”㻌㻌㻌- Weights are positive and sum up to one- Number of images < Number of feature dimensionsRepresents a simplex

BLENDING IN THE “STYLE SPACE”㻌㻌㻌BLENDING IN THE “STYLE SPACE”㻌㻌㻌
The Watercolor “Style Space”

BLENDING IN THE “STYLE SPACE”
Content Image (not watercolor)㻌㻌㻌BLENDING IN THE “STYLE SPACE”
Our method: find the closest point
within the watercolor style space㻌㻌㻌BLENDING IN THE “STYLE SPACE”

arg minrs.t.

Xl,i,jKX
kGlij (Icontent )KX
krk Glij (Ik )!2

rk = 1, 0  rk  1 (k = 1, · · · , K)

Find the optimal weights rk㻌㻌㻌BLENDING IN THE “STYLE SPACE”

arg minrs.t.

X

Glij (Icontent )l,i,jKX
kKX

rk Glij (Ik )k!2

rk = 1, 0  rk  1 (k = 1, · · · , K)

Find the optimal weights rkG̃lij =K
X

rk Glij (Ik )㻌kOptimal Texture Feature

㻌㻌PIPELINE
ContentFeature

F
Content Image

OptimalBlending Optimal
TextureFeature

23
3.1
64.17
6 7
65.97
6 7
62.67
4 5..
.
23
2.7
61.87
6 7
64.27
6 7
68.47
4 5..
.

JointDistanceMinimization
Output Image

Now that we have chosen a texture feature,
for the rest of the style transfer, we use Gatys, et al.[1]PIPELINE
ContentFeature

F
Content Image

OptimalBlending Optimal
TextureFeature

23
3.1
64.17
6 7
65.97
6 7
62.67
4 5..
.
2

3
2.7
61.87
6 7
64.27
6 7
68.47
4 5..
.

JointDistance
Minimization
Output Image

Now that we have chosen a texture feature,
for the rest of the style transfer, we use Gatys, et al.[1]PIPELINE
ContentFeature

F
Content Image

OptimalBlending Optimal
TextureFeature

23
3.1
64.17
6 7
65.97
6 7
62.67
4 5..
.
23
2.7
61.87
6 7
64.27
6 7
68.47
4 5..
.

JointDistance
Minimization

We need a large image dataset with…
A large number of images

Output Image

Now that we have chosen a texture feature,
Annotations
oftheimage’sstyle(“watercolor,”etc…)[1]
for the rest of the style transfer, we use Gatys, et al.

EXPERIMENTS

[4]Novel dataset: “nico-illust”

500,000 images, largely consisting of digital paintings,
with rich annotations: Tags, user comments, Number of favorites, etc.
(Tags include motif name, style name, etc.)[4] https://nico-opendata.jp/en/Author: Kariwo(ID:33341043)(50 images)

Author: Last Hunter(ID:23607472)(50 images)

Author: Morin(ID:10195867)(4 images)

Tag: Watercolor(50 images)

Author: Niichi(ID:13762409)(50 images)

Tag: Pixel Art(10 images)

Author: Kariwo(ID:33341043)(50 images)

Author: Last Hunter(ID:23607472)(50 images)

Author: Morin(ID:10195867)(4 images)

Author: Niichi(ID:13762409)(50 images)

Tag: Watercolor(50 images)

Tag: Pixel Art(10 images)

Automatically captures the use of
perpendicular lines in the style of pixel art
Tag: Watercolor(50 images)

Tag: Pixel Art(10 images)

OUR SYSTEM

Weights of each texture image

Original Video“Watercolor” Style Transfer

Texture image weights for each frame

CONCLUSION

CONCLUSION
We proved that the Texture Features proposed in Gatys, et al.[1]can be linearly combined to construct a valid texture feature
We proposed a novel dataset “nico-illust” that contains digital
paintings
We proposed a neural style transfer method that preserves thecolor of the original input image, by taking a collection ofimages to represent a given style