The 20th Meeting on Image Recognition and Understanding

CONFIDENTIAL EXTENDED ABSTRACT.
DO NOT DISTRIBUTE ANYWHERE.

Unseen Style Transfer Network
Ryosuke TANNO1,a)

1. Introduction

Keiji YANAI1,b)

Instance Normalization ([11]) instead of Batch Normalization ([6]) for normalization of activation signals, and theyIn this paper, we propose a feed-forward neural style

proposed to replace scale and bias parameters of instancetransfer network which can transfer unseen arbitrarynormalization layers depending on the styles. They callstyles. To do that, first, we extend the fast neural stylethis as “conditional instance normalization”. Althoughtransfer network proposed by Johnson et al. [7] so thatthey showed that the fast style network where all thethe network can learn multiple styles at the same time bybatch normalization layers were replaced with the con-adding a conditional input. We call this as “a conditional

ditional instance normalization layers had ability to learn

style transfer network”. Next, we add a style condition

32 artistic styles at the same time, the transferable styles

network which generates a conditional signal from a style

are limited to trained styles and their mixtures.

image directly, and train “a conditional style transfer net-

More recently, a fast arbitrary style transfer method

work with a style condition network” in an end-to-end

which can transfer even untrained styles has been pro-manner. The proposed network can generate a stylized

posed by [1]. They obtained feature map activations of aimage from a content image and a style image in one-time

given content image and a given style image by VGG16,

feed-forward computation instantly.

modify the feature maps by swapping each content activa-Gatys et al. proposed an algorithm on neural artistiction patch with its closet-matching style patch, and gener-style transfer [3], [4] which synthesizes an image which

ate a stylized image using the pre-trained inverse network

has the style of a given style image and the content of

which reconstructs a stylized image from the feature maps

a given content image using a CNN. However, since theirof the swapped activations. Their method is much fastermethod requires both forward and backward computation

than the original method by [3]. However, it takes moreiteratively to synthesize a stylized image, the processing

than one second to generate a stylized image, since style

time tends to be longer (several tens of seconds) even us-swapping is a little bit complicated processing.

ing a GPU. Then, to accelerate it, several works using

In this paper, we propose a feed-forward network forfeed-forward style transfer networks which require only

arbitrary style transfer. Our objective is the same as [1].

one-time feed-forward computation to realize style trans-

However, our approach is diﬀerent from theirs. First wefer have been published so far ([7], [10]). Johnson et al. [7]extend the fast neural style transfer network proposed byproposed a perceptual loss to train a ConvDeconv-style[7] by adding a conditional input so that it can learn multi-

network as a feed-forward style transfer network. Theirple styles. We call this as “a conditional style transfer net-

network can generate a stylized image for a given contentwork” which is similar to “a network with conditional in-image regarding a fixed pre-trained style in real-time.

stance normalization” ([2]) in terms of its objective. How-Although Johnson et al.’s feed-forward network canever, our network is simpler than theirs. While they havetreat only one fixed style, recently Dumoulin et al. [2]to prepare scale and shift parameters for all the instanceproposed a method to learn multiple styles with anormalization layers and for all the trained styles in the

ConvDeconv-style fast style transfer network. They used

style transfer network, we add and connect a conditional

1a)b)

The University of Electro-Communications, Tokyo
tanno-r@mm.inf.uec.ac.jp
yanai@cs.uec.ac.jpinput to the intermediate layer of the ConvDeconv-style
network.

1The 20th Meeting on Image Recognition and Understanding

In both method, mixing of trained multiple styles is pos-mini-batch contains s combinations for one content im-sible by providing mixed conditional weights of the diﬀer-age. This can be regarded as multiple style version ofent styles. From these characteristics of the conditional

Instance Normalization ([11]). In [2], they used diﬀerent

style transfer, we came up with the idea that training ofBN parameters for each of the training styles, while wemany styles and mixing of them might bring arbitrary

use common BN parameters. We guess that by putting

style transfer. To do that, a conditional network is suit-

all the styles in the same mini-batch, BN parameters areable, since it can accept a real-value conditional input the

averaged over all the styles, and the conditional network

dimension of which can be fixed regardless of the number

is expected to be trained so that residuals from the aver-of training styles.

aged BN parameters are compensated depending on theAfter many trials, we found that it was possible by con-selected style.

necting a network which generates a conditional signal

To train the network, we provide a content image and s-

from a style image directly to the conditional input of adimensional one-hot conditional vectors. As shown in Fig-

conditional style transfer network. We trained this con-ure 1, for example, a conditional vector is set as (1, 0, 0, ...)

ditional style transfer network with a style condition gen-for the style no.1, while it is set as (0, 1, 0, ...) for the styleerator network in an end-to-end manner, and obtainedno.2. As a loss function, we use a perceptual loss proposedpromising results.

by [7]. In the same way, we use VGG-16 ([9]) as a loss

The basic idea for arbitrary style transfer is diﬀerent

network, and optimize the weights of the network so thatfrom [1]. The proposed architecture is simpler than theirs,

the content feature of the output image extracted from thesince the network is trained in an end-to-end manner and

COIN3 3 layer of VGG-16 becomes closer to the one of angenerates a stylized image from a content image and ainput content image and the style features (i.e. Gram ma-style image directly by one-time feed-forward computa-trix) of the output image extracted from the CONV1 2,tion without surgery of activation signals such as style

CONV2 2, CONV3 3 and CONV4 3 layers become closer

swapping.

to the ones of the corresponding style image.

In this paper, we report a conditional fast neural style

The proposed network can mix multiple styles like [2],transfer network, and its extension for unseen style trans-

although the network is trained with each of the train-fer.

ing styles independently. The proposed network can also

2. Conditional fast style transfer networktransfer diﬀerent styles to the diﬀerent parts of a given image at the same time, which we call “spatial style transfer”
as shown in Figure 3. In the experiments, we confirmed

We modified the ConvDeconv-style network used in [7]that no quality degradation occurred in the multi-styleby adding a style condition input and an additional 1 × 1

network compared to the single network when the num-convolutional layer for fusing of feature map activations

ber of styles is not large (around 15), and linear-weighted

and a conditional signal as shown in Figure 1. This net-multi-style fusion enabled us to generate various kinds

work is inspired by Iizuka et al’s CNN-based colorizationof new styles which are diﬀerent from the trained sin-work [5]. They proposed adding a scene contextual stream

gle styles. This observation inspired the idea of the next

to a ConvDeconv-style colorization network. They trans-network.

formed a scene vector, which is an output of the scene

In the experiments, we used one-hot vectors for condi-recognition network, to a feature map by making the sametions. However, a conditional signal is not limited to asize of copies as the activation feature map of the mainbinary vector. It can be a real-value vector, if it reflects

network, and concatenate it with the intermediate feature“style embedding”. In fact, a real-value vector which is amap. We followed this for adding a conditional input.

PCA-compressed style feature derived from a Gram ma-

For training of the network, we prepare s style images

trix of VGG-16 feature maps worked. By using a real-and fifty thousands of content sample images (we usedvalue style condition, we can train unlimited number of

MS-COCO images in the same way as [7]), and make astyle images. This property is superior to the conditional

mini-batch with the combinations of one content train-instance normalization method ([2]), which can treat onlying image and all the style training images. That is, one2

The 20th Meeting on Image Recognition and Understanding

Fig. 1 A conditional fast style transfer network.

Fig. 2 A unseen style fast transfer network.

styles. To realize unseen style transfer, we expect that
this CNN has ability to generate a conditional style signal
for a unseen style by combining the conditional signals of
the trained styles. In the experiments, we set the dimension of a conditional style vector as 128 to obtain enoughrepresentation power for arbitrary image styles.
Note that we found that it brought better results whenFig. 3 Three diﬀerent ways to create a conditional input: (1)the last layer of the style condition network was a global

single style (2) mixed style by linear-weighting (3) spa-average pooling (GAP). This is consistent with the facttially mixed style.

indicated by [8], which is that per-channel mean and vari-finite number of styles.

ance also represent the statistics of image styles.
We train this network in the similar way to training of

3. Unseen Style Transfer Network

the normal conditional style network by providing style

In the previous network, conditional signals are as-images to a style signal generator CNN instead of using

sumed to be given by a user. Instead, in this section, weone-hot style conditional vectors. We train the network in

prepare a style condition generator network which takes aan end-to-end manner with fifty thousand content images

style image as an input and outputs a conditional signal,and fifty thousand style images (we used WikiArt images).

and connect it to the conditional input of the conditional

One mini-batch contains the combination of one content

style transfer network as shown in Figure 2. By adding

image and several randomly-selected style images. The

this style condition generator network to the previous net-

number of style images in one mini-batch depends on thework, the whole network can learn unlimited number ofamount of GPU memory.

3

The 20th Meeting on Image Recognition and Understanding

Fig. 4 Results of a conditional transfer network. A mixed styleand a spatial mixed style in the rightmost. All the styles
are trained.

Fig. 6 Qualitative comparison of the results by Fast Style
Transfer Network (FSTN) [7], Conditional Style Trans-

Fig. 5 Results of unseen style transfer by the second network.

fer Network (CSTN), and Unseen Style Transfer Net-All the styles shown above are not trained.

work (USTN).

4. Experimental Results[3]

Figure 4 shows the results of a conditional style transfer
network trained with 14 styles. Figure 5 shows the results[4]of unseen style transfer by the proposed arbitrary styletransfer network trained with 50,000 styles excluding all[5]the styles shown in the figure.
In Figure 6, we show the results including ones by [7]for qualitative comparison of the results by Johnson’s
Fast Style Transfer Network (FSTN) [7], Conditional Style
Transfer Network (CST), and Unseen Style Transfer Net-[6]work (USTN). Note that the style images shown above
were used for training in FSTN and CSTN, while the[7]above style images were NOT used for training in USTN.
The quality of FSTN and CSTN are almost the same,
while the results by USTN is slightly diﬀerent from them.

[8]

However, we think the results by USTN are a good ap[9]proximation of the results by FSTN and CSTN.
Additional

results,

demo

videosandtheiOSapp of real-time multi-style transfer is available at[10]

http://foodcam.mobi/deepstylecam/.
References[1][2]

Chen, T. Q. and Schmidr, M.: Fast Patch-based Style
Transfer of Arbitary Style, arXiv:1612.04337 (2016).
Dumoulin, V., Shlens, J. and Kudlur, M.: A Learned
Representation For Artistic Style, arXiv:1610.07629[11](2016).
Gatys, L. A., Ecker, A. S. and Bethge, M.: A Neural
Algorithm of Artistic Style, arXiv:1508.06576 (2015).
Gatys, L. A., Ecker, A. S. and Bethge, M.: Image Style
Transfer Using Convolutional Neural Networks, Proc.
of IEEE Computer Vision and Pattern Recognition(2016).
Iizuka, S., Simo-Serra, E. and Ishikawa, H.: Let there
be Color!: Joint End-to-end Learning of Global andLocal Image Priors for Automatic Image Colorization
with Simultaneous Classification, ACM Transactions onGraphics (Proc. of SIGGRAPH 2016), Vol. 35, No. 4,
pp. 110:1–110:11 (2016).
Ioﬀe, S. and Szegedy, C.: Batch normalization: Accelerating deep network training by reducing internal covariate shift, Proc. of International Conference on Machine
Learning, pp. 448–456 (2015).
Johnson, J., Alahi, A. and Fei, L. F.: Perceptual Losses
for Real-Time Style Transfer and Super-Resolution,
Proc. of European Conference on Computer Vision(2016).
Li, Y., Wang, N., Liu, J. and Hou, X.: Demystifying
Neural Style Transfer, arXiv:1701.01036 (2017).
Simonyan, K., Vedaldi, A. and Zisserman, A.: Very Deep
Convolutional Networks for Large-Scale Image Recognition, International Conference on Learning Representations (2015).
Ulyanov, D., Lebedev, V., Vedaldi, A. and Lempitsky,
V.: Texture networks: Feed-forward synthesis of textures and stylized images, Internation Conference onMachine Learning (2016).
Ulyanov, D., Vedaldi, A. and Lempitsky, V.: Instance
Normalization: The Missing Ingredient for Fast Stylization, arXiv:1607.08022 (2016).

4