Conditional Fast Style Transfer NetworkKeiji Yanai

Project HP with online demos:
http:/foodcam.mobi/deepstylecam/

The University of Electro-Communications, Tokyo, Japan

3. Conditional Fast Style Transfer• Neural artistic style transfer by Gatys et al. (2015) [A]・ Add a style condition input to ConvDeconv net.
− Add a fusion layer and a style input− Style input: one-hot conditional vector

style image 𝐼𝑠

con5_1

Content𝐹𝑙𝑐
con4_2con4_1

con3_1

con2_1

input

con1_1Pre-trained VGG

Stylizedimage 𝑥

bit.ly/deepstylecam

Ryosuke Tanno

1. Introduction

content image 𝐼𝑐DeepStyleCam(𝐹𝑥𝑙 ,𝑖,𝑗 − 𝐹𝐼𝑙𝑐 ,𝑖,𝑗 )2
𝑖,𝑗(𝐺𝑥𝑙 ,𝑖,𝑗 − 𝐺𝐼𝑙𝑠 ,𝑖,𝑗 )2𝐿𝑜𝑠𝑠𝑠𝑡𝑦𝑙𝑒 =𝑙・ Add a style condition network

to the conditional FST network.

x2

x

𝐿𝑜𝑠𝑠content =Correlation of the activation

4. Unseen Style Transfer𝑖,𝑗− Confirmed that a real-value cond. is OK.

e.g. style1 [1,0,0,0,…], style2: [0,1,0,0,…], style3: [0,0,1,0,…]….

− Base network: Johnson’s ConvDeconv net [B]Each layer has BN and ReLU except last one.

𝐺 𝑙 = 𝐹 𝑙 (𝐹 )𝑙 𝑇

Style {𝐺 𝑙 |𝑙𝜖𝑙𝑠 }Fusion layer(concat + 1x1 conv.)

𝐺 𝑙 : Gram matrix offeature map 𝐹 𝑙− Optimization-based method time consuming (1 min-)• Feed-forward fast style transfer network by Johnson et al.

Style condition vector can begenerated by a CNN (not by hand)− Propose a style condition network which
generates a style condition vector
from a given style image directly.

Conditional Style Transfer + Style Network“Unseen style transfer network”(2016) [B]− train a feed-forward

Conditional Fast Style Transfer Network

Conv-Deconv networkwith perceptual loss
− train only one fixed styleneed to train an individual model for each style imagee.g. ten styles
ten models

Multiple style feed-forward network is desirable !

・Training− Perceptual loss with VGG16(the same way as Johnson’s work [B])content: conv3_3
style: conv1_2, c2_2, c3_3, c4_3

[idea 1] Conditional fast style transfer network
However, only trained styles can be transferred.

Unseen style transfer feed-forward network is better !

[idea 2] Unseen style transfer network(Extension of a conditional fast style transfer net.)

2. Concurrent Works (mini survey)− Each mini batch : one content image+ all the style images(= multi-style version of Instance Normalization)
・Generating stylized images in three ways
Input: content image + style condition vector− Single style: one-hot vector [1,0,0,0,..] , [0,1,0,0,..]− Mixed style: multiple-style-weighting[1,1,1,…], [0.2, 0.1, 0.8,..](A) Multiple Styles[1] V. Dumoulin et al.: A Learned Representation of Artistic
Style, ICLR conf. tack, 2017/03.

− Spatial mixed style :Unseen Style Transfer Network= Conditional Fast Style Transfer Network
・Training + Style Condition Network- End-to-end training with perceptual loss
- Each mini batch : one content image+ randomly selected style images
from 50,000 style images (WikiArt)•Generating stylized images in threeways− the same way as a Conditional Fast
Style Transfer Network
・Results of unseen style transfer
with NOT-trained styles+ConvDeconv + Conditional Instance Normalization(Train style-specific scale and shift parameters of all the IN layers)[2] D. Chen et al.: StyleBank: An Explicit Representation for
Neural Image Style Transfer, arXiv: 1703.09210, 2017/03.

・Results of cond. style transfer with trained styles
Qualitative comparison (single/multi/unseen)(Train stylespecific conv.
layers, andswitch them )ConvDeconv + StyleBank Layer(B) Arbitrary Styles[3] T. Q. Chen et al.: Fast Patch-based Style Transfer of Arbitrary
VGG16 (Enc.)
Style, arXiv: 1612.04337, 2016/12.
+ Style Swap
+ Inverse VGG16 (Dec.)(modifying f-maps)

5. Mobile Implementation[4] X. Huang et al.: Arbitrary Style Transfer in Real-time with
Adaptive Instance Normalization, ICLR WS, 2017/03.

(inspired by Y. Li et al.: Demystifying Neural Style Transfer, arXiv: 1701.01036, 2017.)

Gradual Mixing Weight Change

Spatial Mix Transfer

Online Demo

VGG16 (Enc.)+ Adaptive IN+ Inverse VGG16 (Dec.)[5] H. Zhang et al.: Multi-style Generative Network for Real-time
Transfer, arXiv: 1703.06953, 2017/03.
ConvDeconv+ Style feature net (VGG16)+ Integration (inspirarion) Layer[6] H. Wang et al.: ZM-Net: Real-time Zero-shot Image
Manipulation Network, arXiv: 1703.07255, 2017/03.

180ms (250x250), 250ms (350x350)

image[7] G. Ghiasi et al.: Exploring the structure of a real-time, arbitraryneural artistic stylization network, arXiv: 1705.06830, 2017/05.

ConvDeconv Same ideaas [6]+ Style predict net by authors+ Dynamic IN of [1](Google Brain team)Conv
4x4x64
Conv
Conv 4x4x64 stride=2
Conv
5x5x32 4x4x64 stride=2
concatstride=1 stride=2[1, 0, 0, 0, 0, 0]

conditional
vectornatural extension of [1]ConvDeconv+ Parameter net+ Dynamic IN・Shrink the network for mobile devices.
- Add one down-conv. and one up-conv.
- Reduce the num of ResBlock from 5 to 3

Conditional style transfer
http://bit.ly/mixstyle

Unseen style transfer
http://bit.ly/unseenstyle

duplicate

Res Res Res
block block block
Conv
1x1x64
stride=2ResBloc
k3x3x64
stride=1x2x3Conv Conv
Conv
Conv
4x4x2564x4x128
5x5x3
4x4x128
stride=1
Unpool Unpool
Unpoolstride=2stride=2
stride=2+++Conv Conv
Conv
4x4x64 4x4x32
4x4x32styledimage

Conditional Style Transfer Network for Mobile

References[A] L. A. Gatys et al.: Image style transfer using
convolutional neural networks, CVPR, 2016.
(A Neural algorithm of artistic style, arXiv: 1508.06576, 2015)[B] J. Johnson et al.: Perceptual Losses for Real-Time Style
Transfer and Super-Resolution, ECCV, 2016.

For iPhone 7/7+/SE
iPadPro