D-3A-48

DeepXCam: Very Fast CNN-based Mobile Applications:
Multiple Style Transfer and Object Recognition

Ryosuke Tanno, Wataru Shimoda, Keiji Yanai The University of the Electro-Communications, Tokyo

1. Objective

4. Accuracy and Recognition Timex2x

Common Features of All the Apps

UEC-FOOD100 class recognition performance

・Standalone CNN mobile applications (no external server required)・Speeding up by multi-threading and fast framework
・Recognizing any size of images by multi-scale Fully Convolutional Network
・Significant reduction in memory requirements
・Being applicable to various kinds of mobile devices

Top-595.2%
Top-1
78.8%

Example: 100-class food recognition

・recognition time: 26.2ms(iPhone7Plus), top-5 accuracy: 91.5%

2. Proposal Contents
Anyone can build very fast CNN-based mobile apps
including object recognition apps and style transfer apps.
〜flow of making mobile app〜

Developed in our lab

Prepare a training image data

・Caffe2C / Chainer2C− convert parameter files toC source codes that run on
mobile devices
・Very fast CNN-based mobile

Train a CNN model byCaffe (or DIGITS)

recognition/transfer engine− speeding up by multithreading and fast framework

Generate a C source code byCaffe2C automatically・Adopting NIN architecture

for a recognition engine

Prepare a GUI code of mobile app

Generate CNN-based image
recognition app by compiling thegenerated C code and the GUI code

− any size of input images− the trade-off between
accuracy and processing timeby changing input image sizes

If you prepare training data,
you can create mobile
recognition apps in a day !!

3. DeepXCam for recognition ( X = Food, Dog, Bird, Flower )・ Training DCNN
− Use Network-In-Network(NIN)[3] considering mobile implementation− Save the size of the network parameters
Network In Network [3]・only conv layers
・no FC layers
・relatively smaller than the otherarchitectures
・any image size correspondence

ParamMemory

Top-5

AlexNet62Million

248MB

95.1%

NIN(4L+BN)

5.5Million

22MB

95.2%

NIN(5L+BN)

15.8Million

63MB

96.2%

・ Pre-trained CNNs with ImageNet 2000 category images

Trade-Off between Accuracy and Recognition Time
Input Image Size 227x227 200x200 180x180 160x160
iPhone 7 Plus 55.7[ms] 42.1[ms] 35.5[ms] 26.2[ms]iPad Pro

real realtime !!

iPhone SE
77.6[ms] 56.0[ms] 50.2[ms] 37.2[ms]Accuracy (top-5) 95.2%
95.1%
94.1%
91.5%

4. DeepStyleCam (Image Style Transfer）
・ConvDeconvNetwork[2] can treat only one fixed style.
− If transferring ten kinds of styles, we have to train ten different
ConvDeconvNetwork independently.
− This isn’t good for mobile implementation(required memory size)
・We modified [2] can train multiple styles at the same time
− adding a fusion layer and a style input stream(inspired by [1])・Training− We input sample images to the content stream and style images to thestyle stream.(The training method is the same as [2])・ We shrunk the ConvDeconvNetwork compared to [2]− added one down-sampling layer and up-sampling layer− replaced 9x9 kernels with smaller 5x5 kernels in the first and lastconvolutional layers− reduced five Residual Elements into three
short-cut

Contentimage

Fusionconv_4 layer
conv_3 4x4x128
4x4x64 stride=2
conv_2 stride=2

conv_1
4x4x325x5x16
stride=2
stride=1

short-cut

Styleimage

・ Speeding up Conv layers ⇒ Speeding up GEMM

short-cut

conv_5,6 conv_6,7 conv_8,93x3x128 3x3x128 3x3x128 conv_74x4x64
stride=1/2 conv_8
4x4x32
conv_10
stride=1/2
5x5x3
conv_94x4x16 stride=1
stride=1/2

Duplicate1x1x128 ⇒ NxNx128(totally 2.1 million images)− computation of conv layers is decomposed into “im2col” operationand matrix multiplications− BLAS(iOS: Accelerate Framework, Android: OpenBLAS)−we use the NEON instruction set which can execute four multiplications
and accumulating calculations at once.
−iOS: 2Core*4 = 8 calculation, Android: 4Core*4 = 16 calculation

66.6[ms] 49.7[ms] 44.0[ms] 32.6[ms]We achieve

conv2_4 Global
conv2_3 4x4x128 Average
4x4x64 stride=2 Poolingconv2_2 stride=2(1x1x128)
4x4x64conv2_1 stride=2
4x4x32
stride=2

Unpooling= stride 1/2

ConvDeconvNet with Style Input

Nomal mode
Color Preserving
mode

Ex. Image Size: 250x250 ，Computation: 1,303,800,800 times(13billion)
Parameter num: 1,250,835

175ms (iPhone7+)
180ms (iPad Pro)
200ms (iPhone SE)

Multiple Style Transfer and Object Recognition App・Food Rec App (both iOS/Android)・Multi Style Transfer (only iOS)

Our Project page

GEMM: generic

http://foodcam.mobi
Please search ‘’DeepFoodCam’’Please search‘’RealTimeMultiStyleTransfer’’

Reference[1] S. Iizuka et al.: Let there be Color!: Joint End-to-end Learning of Global and Local Image Priors for Automatic Image Colorization with Simultaneous Classification, SIGGRAPH, 2016.
[2] J. Johnson et al.: Perceptual Losses for Real-Time Style Transfer and Super-Resolution, ECCV, 2016.
[3] M. Lin et al. Network In Network, ICLR, 2014.