Blending Texture Features from Multiple Reference Images for Style Transfer
Hikaru Ikuta1, 2∗
1

Keisuke Ogaki2†
2

The University of Tokyo

Yuri Odagiri2‡

DWANGO Co., Ltd.

Query by Authors
33341043 10195867

....

..

Query by Tags
“Watercolor” “Pixel Art” “Deformation”

23607472 13762409

....

..

InputImages

....

..

....

..

....

..

....

..

....

..

Synthesized Results

11799762778232

404415Figure 1: Image synthesis results.

Abstract
We present an algorithm that learns a desired style of artwork froma collection of images and transfers this style to an arbitrary image.
Our method is based on the observation that the style of artwork
is not characterized by the features of one work, but rather by thefeatures that commonly appear within a collection of works. To
learn such a representation of style, a sufficiently large dataset ofimages created in the same style is necessary. We present a novelillustration dataset that contains 500,000 images mainly consistingof digital paintings, annotated with rich information such as tags,comments, etc. We utilize a feature space constructed from statistical properties of CNN feature responses, and represent the style asa closed region within the feature space. We present experimental
results that show the closed region is capable of synthesizing an appropriate texture that belongs to the desired style, and is capable oftransferring the synthesized texture to a given input image.

1Introduction

Re-creating the style of an image is a basic process in the field ofart and design. For example, in animation production, backgrounds
are often based on photographs of an actual location, redrawn in theanimation’s style. The goal of our research is to create a system that
∗ e-mail:hikaru ikuta@ipc.i.u-tokyo.ac.jp† e-mail:keisuke ogaki@dwango.co.jp‡ e-mail:yuri odagiri@dwango.co.jp

captures a certain style of artwork from a set of reference images,
and re-creates an arbitrary input image in the desired style, as shownin Figure 1.
There are approaches that share the motivation of transferring acertain style. Image analogies, proposed by A. Hertzmann etal. [2001], is a method based on patch matching. The major restriction of this system is that it requires a pair of images with strict
region correspondences. L. A. Gatys et al. [2015] uses the image
features of a pre-learned convolutional neural network (CNN) forstyle transfer. Their method takes an arbitrary pair of an input image and a reference texture image and transfers the texture of thereference texture image to the input image, eliminating the need ofregion correspondences. The major drawback of this method is thatthe color distribution of the output image is restricted by the input
texture image. For example, if an input texture image is mainly
drawn by a blue color such as the sky, the output image tends to be
painted in blue. This often causes unnatural results as shown in Figure 3, where some regions in the original input image (Figure 3(b))
is unnaturally drawn in a blue color (Figure 3(c)).
Such a problem occurs in the method by L. A. Gatys et al., since
their algorithm does not distinguish the style of an artwork fromthe texture of a specific artwork. For example, Vincent Van Gogh’s
most prominent style of art is the use of small repetitive strokes,
which is captured by the frequency of a certain edge pattern, interms of features. However, since the color used in Gogh’s seriesof artwork depends on each piece of art, features representing color
may vary, therefore making it not a common style. From this exam-

Figure 2: Illustration of our style transfer pipeline. The system receives an input image and a style tag as the input. The system then queries
the tag within the nico-illust dataset to collect 50 images annotated with the given tag. The collected images are then used to learn the desired
style as a representation in a feature space constructed from CNN filter responses. The learned style representation is then used to estimate
the appropriate texture for drawing the input image in the desired style. The estimated texture feature is finally passed on to the style transfer
to obtain the final results.
ple, we learn that an artwork contains two types of visual features:(1) the features that are common in all of the images that share thesame style, and (2) features that are special in the certain instanceof the artwork. Gogh’s repetitive strokes are classified as (1), thecommon feature, and the different colors used in each of the artworks are classified as (2). From then on, we will refer to (2) as thestyle, and will refer to both (1) and (2) in general as textures.
In this paper, we propose a style transfer algorithm that transfers thecommon features of a certain style, by generating a texture featurethat optimally fits the input image from reference texture images.
Our algorithm takes a set of images created in a certain style, andlearns the style as a representation in a feature space constructedby statistical amounts of CNN filter responses. We then show thatthe extracted style could be transferred to an arbitrary input image,using the learned representation of the style.
For evaluation, we have applied our algorithm to a novel image
dataset, the nico-illust dataset, which we present in this paper. The
nico-illust dataset contains 500,000 images, mainly consisting ofdigital paintings, each annotated with rich text information. To theextent of our knowledge, our dataset is the first dataset with such a
volume of images mainly consisting of digital paintings, a popular
style of artwork.
Our contributions are summarized as follows: (1) we propose amethod of learning a given style from multiple reference images,
and then transferring the artist’s style to a given input image, bygenerating an appropriate texture from the artist’s style to transfer to the image, and (2) we present a novel dataset that containing
500,000 images largely consisting of modern-style illustrations, annotated with rich text information. All code and data required for
reproducing the experiments are available online 1 .

2The Method

2.1

Overview

Figure 2 illustrates the overview of our system. The input to oursystem is (1) the input image, for the style to be transferred, and(2) a text query. The novelty of the system is the style pipeline,
where the appropriate texture for rendering the output image with
1 https://nico-opendata.jp/en/index.html

Figure 3: Evaluation of color correspondence by clustering.

the given style, is estimated. This system is constructed from two
modules: (1) creating the model of the desired style, and (2) estimating the choice of texture to render the input image in the desiredstyle. In (1), the system first constructs a representation of the desired style from a collection of images taken from the nico-illust
dataset. The collection of images could be simply be derived froma single text query. The style is represented by a closed region in afeature space constructed from CNN filter responses.
Once the style representation is learned, it is used in (2), where thestyle model is used to estimate an appropriate texture to render theinput image. We use [Gatys et al. 2016] to perform style transfer tothe input image.

2.2

The Texture Feature

In our method, we learn the style model using a texture space that isa normalized version of the style feature G, derived from filter responses of comvolutional neural networks (CNNs), used in [Gatyset al. 2016]. [Gatys et al. 2016] indicates that the Grammian G
could be used to represent the “style” of an input image, and used
the style representation of G to transfer the style. In this paper,
we first show that the G of a concatenated image is approximately
equal to the linear combinations of the G of each of the concatenated images. We then exploit this property to show that the G ofdifferent images in the same style could be linearly blended to represent the texture of a well-defined image, belonging to a class ofimages created in the desired style as well.

㻌Figure 4: The sum of G of each image I is linear to G of concatinated image I1 ◦ I2 ◦ · · · ◦ IN .
To construct G, we first prepare a pre-learned CNN used for image
recognition. Given an image I, let Fijl (I) be the jth element of theresponse of the ith filter in the lth layer of the CNN. The texture
feature G is then represented using Fijl asMlGlij (I) =1 X llFik (I)Fjk(I).
Ml(1)

kHere, Ml is the dimentionality of the responses of each of the filtersin the lth layer of the CNN.
The difference of this definition from [Gatys et al. 2016] is thenormalization factor 1/Ml . In the previous definition of the texture feature, because the normalization factor 1/Ml was absent, themagnitude of the texture feature was dependent on the image size,
as Glij (I) is a summation of the filter responses along the pixels kin the CNN filter response. Our definition of Glij normalizes thefeature with the image size 1/Ml , and thus we are able to compare
Glij of different images I. Each element of Glij (I) then describes astatistical amount that could be viewed as the frequency of a certain
local feature among the entire image, or the co-occurence of certain
pairs of local features among the entire image.

2.3

Linear Combinations of the Texture Feature

Now, let I = {In }N
n=1 be a set of reference images, and · ◦ · be
the concatenation of two images. Then, the texture feature of theconcatenated image I1 ◦ I2 ◦ · · · ◦ IN is written as
Glij (I1 ◦ I2 ◦ · · · ◦ IN )style in the texture feature space. Since arbitrary concatenations ofimages In drawn in a given style also belong to the same style aswell, any linear combinations of the texture feature of these images
are also a valid texture features that belong to the given style. In
other words, any point in the following closed region)(NXl
rn = 1, 0 ≤ rn ≤ 1 (n = 1, · · · , N ) (3)rn Gij (In )n=1

could be chosen as a valid texture feature that belongs to the desiredstyle. We define this closed region in the feature space as the “stylespace”.
As mentioned in the introduction, each instance of an artwork created in a certain style contains two types of features: (1) the featurescommon in all of the images that share the style, and (2) featuresthat are special in the certain instance of the artwork. The restrictionof the choice of texture features within the style space corresponds
to (1), and the degrees of freedom of the textures within the stylespace corresponds to (2).
If a sufficient number of texture reference images is collected, themethod could synthesize a new texture that absorbs the differencesof each sample, appropriate for the input image.
Since we haveobtained a set of texture features that represent an instance of anartwork created in a certain style, the next task is to find the appropriate texture to render a given input image. Let Icontent be
the input content image. Since Icontent is not drawn in the target
style, G(Icontent ) does not necessarily lie within the representationof the desired style. We estimate the appropriate texture within thedesired style representation by finding the closest point in the style
representation from G(Icontent ):Estimating the Appropriate Texture to Transfer

arg minr=≈=1Ml,allMl,allXll
Fik(I1 ◦ I2 ◦ · · · ◦ IN )Fjk
(I1 ◦ I2 ◦ · · · ◦ IN )l,nN M
XX

NX
Ml,nMl,all
n=1

s.t.

NX

NX

rn Glij (In )n=1!2(4)rn = 1, 0 ≤ rn ≤ 1 (n = 1, · · · , N ).

n=1ll
Fik(In )Fjk
(In )k=1Glij (In ).

(2)

Here, Ml,n is the numberPof pixels for each filter response for thenth image, and Ml,all = n Ml,n . The approximation on the second line is due to the image padding that appears in the calculationof the convolution. When calculating Glij (Iall ), the window of theconvolution could lie across different images, which in Glij (In )corresponds to the image padding. Thus, the approximation becomes strictly equal in the limit where the padding size becomes
negligible against the total image size.
Therefore, by considering only large images as the input, we canuse Eq. (2) to say that G is linear against image concatenation(Figure 4). Using this fact, for every weighted linear combinationof G, such that the weight is rational number, there exists an image
that realizes its value of the texture feature.

2.4

Glij (Icontent ) −l,i,jk=1

1Ml,all n=1

X

The Style ModelIn the proposed method, we use
linear combinations of G to construct a representation of the desired
Preparing the Texture SpaceHere, r = [r1 , . . . , rn ] is the blending ratio of the texture images. This optimization problem could be easily transformed into aquadratic programming problem of r. We solved the optimization
problem using CVXOPT [Andersen et al. 2012], a Python package
for convex optimization.
Using the optimized weight r, we construct a
linearly weighted texture feature,

Texture TransferG̃lij =KX

rk Glij (Ik ),(5)k=1where K is the number of images in the reference texture images.
We then apply the method of Gatys et al. [Gatys et al. 2016] fortexture transfer. We incorporate the estimated texture feature into
the texture loss term, i.e., the second term in the loss function L,and modify it asL=+2
X 1  lFij (I) − Fijl (Icontent )2Nli,j2
XX  lβwl Gij (I) − G̃lij .
αli,j(6)3The Dataset

As mentioned in the introduction, learning a specific style of artwork requires a large collection of annotated image data. Although Wikipaintings [Karayev et al. 2014] is a dataset that contains 100,000 high-art images, to our knowledge, there is no datasetthat contains such a number of digital paintings.
We provide a new dataset, the nico-illust dataset, for analyzing thestyle of modern-style illustrations. The dataset includes 500,000
images largely consisting of modern-style illustrations. All images
in this dataset are in illustration-sharing service “Niconico Seiga” 2 .
We obtained permission from the authors to use them for academic
research purposes. Each image is associated with annotations suchas the author id, tags.
Among these annotations, our proposed method mainly uses the author ids and tags. The tags that are annotated include informationsuch as the author, the motif, and the style of the image (watercolor,pencil, acrylic, etc.). It is also noteworthy that there are many examples where the same motif is drawn by various artists.

4Result and Discussion

Application to Images Figure 1 shows an example of ourmethod applied to various pairs of input images and style queries.
The choice of style queries consists of 4 authors with IDs
33341043, 10195867, 23607472, and 13762409, and two styles,
watercolor and pixel art. The input images were chosen from thenico-illust dataset as well. To construct the feature space, we chose
the VGG model proposed by Simonyan et al. [Simonyan and Zisserman 2015]. In the figure, we ans see that the algorithm (1) does
not corrupt the color of the original image, and (2) effectively captures and transfers the queried style. For example, for the user
23607472, the algorithm effectively captures both the pointillismlike style and the colors of the artist.

We have also applied our method to eachframe of a video using “watercolor” as the style query. Figure 5
shows the plot of the series of weights of the 50 texture images inthe texture image set, and the two images with the largest weights.
The results show that the algorithm effectively chooses texture images that describes the input frame. The results are available in thesupplemental material.
Application to Videos

5Conclusion

We have presented a system that learns a desired style of artworkfrom a collection of images, and transfers the style to an arbitrary
image. Our novel illustration dataset contains 500,000 images, consisting mainly of digital paintings annotated with rich text information. We constructed our system using this presented dataset.
Experimental results show that our system is capable of effectivelytransferring various styles, and our system is capable of smoothlytransferring styles to videos as well.
2 http://seiga.nicovideo.jp/!"#$

Here, wl is the scalar weight for each CNN layer used to construct
the texture feature. The difference here is that the synthesized texture feature is used in place of the texture feature of the input texture
image. We then use probabilistic gradient descent to find an image
I that gives the local optimum of L. The obtained image I becomesthe final output image, where the estimated texture is transferred tothe input image.

!"*##!")##!"(##!"'##!"&##!"%##!"$##!"!##+,%%)!$&%#+,-&.(**#+,$!$$)$&#+,&$.---'#+,$)-..)(#+,%-.&()'#+,$-$!(*&#+,&)%!$!$#+,.(*$%$#+,$)&%'&&#+,$'%('-*#+,)'!!&(#+,&-.)'.)#+,%'%&)'$#+,%$$*$()#+,$.*-'--#+,%*$(-'-#%&'()*+"'(,im1701858

im857121

im926441

im1701858im640035+,'-$)(&!#+,)($$-)#+,$!(%)(&#+,-'$'-$#+,%-)*$..#+,&%%&&$)#+,%$$-$'%#+,%$*)*.-#+,$.%&!--#+,-.()$$#+,&*&%)-$#+,-%)''$#+,$*&'())#+,$%)*&$.#+,'-(*(%-#+,$$*%).%#+,$&-&!''#+,&'*&-$$#+,'))$-%#+,&$&)*)$#+,$!-(&&*#+,%--*'!!#+,%'%.$*)#+,$*!$.(.#+,&--$'%%#+,-(&*'&#+,-(*.(-#+,&%(.-$)#+,%''$)(.#+,')-&(!)#+,-'!)%%#+,&!$!&*-#+,$(.(&'.#im4916530

Figure 5: Series of texture blending weights across the Big Buck
Bunny c Blender Foundation, 2008.

Acknowledgements
We would like to thank all of the illustrators for their inspiring artwork, which has motivated us for this research, and was essential
for creating the dataset. We would also like to thank Yusuke Matsui
for the valuable suggestions and fruitful discussions.

References
A NDERSEN , M. S., DAHL , J., AND VANDENBERGHE , L.,2012. Cvxopt: A python package for convex optimization.
http://cvxopt.org/.
G ATYS , L. A., E CKER , A. S., AND B ETHGE , M. 2015. Texture synthesis using convolutional neural networks. Advances in
Neural Information Processing Systems, 262–270.
G ATYS , L. A., E CKER , A. S., AND B ETHGE , M. 2016. Image style transfer using convolutional neural networks. In IEEE
Computer Vision and Pattern Recognition.
H ERTZMANN , A., JACOBS , C. E., O LIVER , N., C URLESS , B.,
AND S ALESIN , D. H. 2001. Image analogies. In ACM SIGGRAPH, 327–340.
K ARAYEV, S., T RENTACOSTE , M., H AN , H., AGARWALA , A.,
DARRELL , T., H ERTZMANN , A., AND W INNEMOELLER , H.
2014. Recognizing image style. In British Machine Vision Conference.
S IMONYAN , K., AND Z ISSERMAN , A. 2015. Very deep convolutional networks for large-scale image recognition. In International Conference on Learning Representations.
YAMAGUCHI , S., F URUSAWA , C., K ATO , T., F UKUSATO , T.,
AND M ORISHIMA , S. 2015. Bgmaker: Example-based anime
background image creation from a photograph. In ACM SIGGRAPH Posters.