DeepStyleCam:
A Real-time Style Transfer App on iOS

Ryosuke Tanno, Shin Matsuo, Wataru Shimoda, Keiji Yanai
The University of the Electro-Communications, Tokyo, Japan
3. Example

1. Introduction
・In 2015, Gatys et al.[2] proposed an algorithm on neural artistic styletransfer− synthesizes an image which has the style of a given style image
and the contents of a given content image using CNN
・However, since the method proposed by Gatys et al.[2] required
forward and backward computation many times− the processing time tends to become longer (several seconds)
even using GPUx2・ Then, several methods using only feed-forward computation of CNN torealize style transfer have been proposed so far.
− Johnson et al.[1] proposed perceptual loss functions to train theConvDeconvNetwork as a feed-forward style transfer network
・ However, the ConvDeconvNetwork trained by their method can treat only
one fixed style.
− If transferring ten kinds of styles, we have to train ten different
ConvDeconvNetwork independently.
− This is not good for mobile implementation in terms of required
memory size.

StylexContent・We modified [1] can train multiple styles at the same time
− adding a fusion layer and a style input stream(inspired by [3])・Training− We input sample images to the content stream
and style images to the style stream.
(The training method is the same as [1])・ We shrunk the ConvDeconvNetwork compared to [1]− added one down-sampling layer and up-sampling layer− replaced 9x9 kernels with smaller 5x5 kernels
in the first and last convolutional layers− reduced five Residual Elements into threeContentimageFusion
conv_4 layer
conv_3 4x4x128
4x4x64 stride=2conv_2 stride=2

conv_1
4x4x325x5x16 stride=2
stride=1

Training
Phase

Duplicate1x1x128 ⇒NxNx128

Style
image

conv2_4 Global
conv2_3 4x4x128 Average
4x4x64 stride=2 Poolingconv2_2(1x1x128)
stride=2
4x4x64conv2_1 stride=2
4x4x32
stride=2

Style
image
2

short-cut

Content

Partial Style
TransferContent

short-cut

conv_5,6 conv_6,7 conv_8,9
3x3x128 3x3x128 3x3x128 conv_7
4x4x64
stride=1/2 conv_8
4x4x32
conv_10
stride=1/2
5x5x3
conv_94x4x16 stride=1
stride=1/2
PartialMixedStyleweighted sum

Mixed Style

Style

2. Proposed Systemshort-cut

Mixed Style

StyleThen, we modified Johnson et al.’s method so that one
ConvDeconvNetwork can train multiple styles at the same time

Mixed StyleStyle

4. Demo Video

Partial Style Transfer

Output Images

SingleStyle

MixedStyle

PartialStyle

We create the style image
by horizontallyconcatenating the 4 images
on the left side.

ConvDeconvNet
with Style Input

5. Multiple Style Transfer App
Unpooling= stride 1/2

・Multi Style Transfer App (only iOS)

Nomal mode
DownLoadColor Preserving
mode[5]
Ex. Image Size: 250x250 ，Computation: 1,303,800,800 times(13billion)
Parameter num: 1,250,835

https://goo.gl/vmkg2j
175ms (iPhone7+)
180ms (iPad Pro)
200ms (iPhone SE)

・Recommend only for iPhone 7/6s/SE
and iPad ProReference[1] J. Johnson et al.: Perceptual Losses for Real-Time Style Transfer and Super-Resolution, ECCV, 2016.
[2] L. A. Gatys et al.: Image style transfer using convolutional neural networks, CVPR, 2016.
[3] S. Iizuka et al.: Let there be Color!: Joint End-to-end Learning of Global and Local Image Priors for Automatic Image Colorization with Simultaneous Classification, SIGGRAPH, 2016.
[4] K. Yanai et al.: Efficient mobile implementation of a cnn-based object recognition system, ACM MM, 2016.
[5]L. A. Gatys et al.: Preserving color in neural artistic style transfer, ArXiv:1606.05897, 2016.