Unseen Style Transfer
Based on a Conditional Fast Style Transfer Network

Project HP with online demos:
http:/foodcam.mobi/deepstylecam/

Keiji Yanai The University of Electro-Communications, Tokyo, Japan

1. Introduction

2. Conditional Fast Style Transfer
・ Add a style condition input to ConvDeconv network.

• Neural artistic style transfer by Gatys et al. (2015) [A]

con5_1

con4_2

Content 𝐹

con4_1

con3_1

content image 𝐼𝑐

con2_1

input

con1_1Pre-trained VGG(𝐹𝑥𝑙 ,𝑖,𝑗 − 𝐹𝐼𝑙 ,𝑖,𝑗 )2𝑐

𝐿𝑜𝑠𝑠content =𝑖,𝑗

(𝐺𝑥𝑙 ,𝑖,𝑗 − 𝐺𝐼𝑙 ,𝑖,𝑗 )2

𝐿𝑜𝑠𝑠𝑠𝑡𝑦𝑙𝑒 =Stylizedimage 𝑥Correlation of the activation𝑠𝑙𝑇

𝐺 𝑙 = 𝐹 𝑙 (𝐹 𝑙 )style image 𝐼𝑠− Add a fusion layer and a style input− Style input: one-hot conditional vector

𝑙𝑐e.g. style1 [1,0,0,0,…], style2: [0,1,0,0,…], style3: [0,0,1,0,…]….

− Base network: Johnson’s ConvDeconv network [B]Each layer has BN and ReLU except last one.

𝑖,𝑗Style {𝐺 𝑙 |𝑙𝜖𝑙𝑠 }x2
xbit.ly/deepstylecam

3. Unseen Style Transfer Network

4. Concurrent Works (mini survey)

・ Add a style condition network to the cond. FST network.

(A) Multiple Styles− Confirmed that a real-value condition vector is possible.
Style cond. vec. can be generated by a CNN (not only hand)− Propose a style condition network which generates
a style condition vectos from a given style image directly.
− As a result, “Unseen Style Transfer Net” got possible !!

[1] V. Dumoulin et al.: A Learned Representation of Artistic
Style, ICLR conf. tack, 2017.

+ConvDeconv + Conditional Instance Normalization

Fusion layer(concat + 1x1 conv.)

𝐺 𝑙 : Gram matrix offeature map 𝐹 𝑙DeepStyleCam(Train style-specific scale and shift parameters of all the IN layers)− Optimization-based method

time consuming (1 min or more)• Feed-forward fast style transfer network by Johnson et al.

[2] D. Chen et al.: StyleBank: An Explicit Representation for
Neural Image Style Transfer, arXiv: 1703.09210, 2017.

(2016) [B]− train a feed-forward
Conv-Deconv networkwith perceptual loss
− train only one fixed styleneed to train an individual model for each style imagee.g. ten styles ten models

Conditional Fast Style Transfer Network・Training− Perceptual loss with VGG16 (same as Johnson’s work [B])content: conv3_3, style: conv1_2, c2_2, c3_3, c4_3
− Each mini batch : one content image + all the style images(= multi-style version of Instance Normalization)

Multiple style feed-forward network is desirable !

・Generating stylized images in three ways

[idea 1] Conditional fast style transfer network

Input: content image + style condition vector− Single style: one-hot vector [1,0,0,0,..] , [0,1,0,0,..]− Mixed style: multiple-style-weighting [1,1,1,…], [0.2, 0.1, 0.8,..]− Spatial mixed style :However, only trained styles can be transferred.

Unseen style transfer feed-forward network is better !

Conditional Fast Style Transfer Networkwith a Style Condition Network

Unseen Style Transfer Network・Training− End-to-end training with perceptual loss
− Each mini batch : one content image+ randomly selected style images from 50,000 style images• Generating stylized images in three ways
− the same way as a Conditional Fast Style Transfer Network・Results of unseen style transfer with NOT-trained styles[idea 2] Unseen style transfer network(an extension of a conditional fast style transfer network)
Qualitativecomparison

ConvDeconv + StyleBank Layer(Train stylespecific conv.
layers, andswitch them )(B) Arbitrary Styles[3] T. Q. Chen et al.: Fast Patch-based Style Transfer of Arbitrary
Style, arXiv: 1612.04337, 2017.
VGG16 (Enc.)+ Style Swap+ Inverse VGG16 (Dec.)(modifying f-maps)[4] X. Huang et al.: Arbitrary Style Transfer in Real-time with
Adaptive Instance Normalization, ICLR WS, 2017.
(inspired by Y. Li et al.: Demystifying Neural Style Transfer, arXiv: 1701.01036, 2017.)

VGG16 (Enc.)+ Adaptive IN+ Inverse VGG16 (Dec.)[5] H. Zhang et al.: Multi-style Generative Network for Real-time
Transfer, arXiv: 1703.06953, 2017.

・Results of a cond. style transfer with trained styles

ConvDeconv+ Style feature net (VGG16)+ Integration (inspirarion) Layer

References[A] L. A. Gatys et al.: Image style transfer using convolutional neural networks, CVPR,
2016. (A Neural algorithm of artistic style, arXiv: 1508.06576, 2015)[B] J. Johnson et al.: Perceptual Losses for Real-Time Style Transfer and SuperResolution, ECCV, 2016.

Online Demo[6] H. Wang et al.: ZM-Net: Real-time Zero-shot Image
Manipulation Network, arXiv: 1703.07255, 2017.
Conditional style transfer
http://bit.ly/mixstyle
Unseen style transfer
http://bit.ly/unseenstylenatural extension of [1]ConvDeconv+ Parameter net+ Dynamic IN