Caffe2C: A Framework for Easy Implementationof CNN-based Mobile Applications
Ryosuke Tanno and Keiji Yanai
Department of Informatics,
The University of Electro-Communications, Tokyoⓒ 2016 UEC Tokyo.

1. INTRODUCTIONⓒ 2016 UEC Tokyo.

Deep Learning(DNN,DCNN,CNN)• Deep Learning achieved remarkable progress– E.g. Audio Recognition, Natural Language Processing,• Especially, in Image Recognition, Deep Learning gave
the best performance– Outperform even humans such as recognition of 1000
object(He+, Delving deep into rectifier, 2015)
93.3% 96.4% 94.9%
88.3%
85%

10080

72%

75%

6040SIFT+BOF
Deeeeeeeep

200Outperform
Human !

Deep Learning2010ⓒ 2016 UEC Tokyo.
2011 20122013 2014Trained2015 HumanDeep Learning Framework• Many Deep Learning Framework have emerged– E.g. Caffe, TensorFlow, Chainerⓒ 2016 UEC Tokyo.

What is Caffe?
Convolution Architecture For Feature Extraction(CAFFE)

Open Framework, models and examples for Deep Learning• Focus on Compuer Vision• Pure C++/CUDA architecture for deep learning• Command line, Python MATLAB interface• Fastest processing speed• Caffe is the most popular framework in the worldⓒ 2016 UEC Tokyo.

Bring to CNN to Mobile• There are many attempts to archive CNN on themobile– Require a high computational power and memory

High Computational Power and Memory are Bottleneck!!

ⓒ 2016 UEC Tokyo.

How to train a model by caffe?
• 3 files are required for Training -> Output: Model– 3 files: Network definition, Mean, Label

DatasetTraining

Output･Caffemodel

Files

Use these 4 files
on mobile･Network3 files･Meanⓒ 2016 UEC Tokyo.
･LabelUse the 4 Filesby Caffe on the Mobile• We currently need to use OpenCV DNN module– not optimized for the mobile devices– their execution speed is relatively slow

･Network･Mean
4 files･Label･Modelⓒ 2016 UEC Tokyo.

Objective• We create a Caffe2C which converts the CNN model
definition files and the parameter files trained byCaffe to a single C language code that can run on
mobile devices･NetworkCaffe2C･Mean
4 filesSingle C code
･Label･Model• Caffe2C makes it easy to use deep learning on the C
language operating environment• Caffe2C achieves faster runtime in comparison tothe existing OpenCV DNN moduleⓒ 2016 UEC Tokyo.

Objective• In order to demonstrate the utilization of the Caffe2C,
we have implemented 4 kinds of mobile CNN-based
image recognition apps on iOS.

ⓒ 2016 UEC Tokyo.

Contributions
1. We create a Caffe2C which converts the model
definition files and the parameter files of Caffe intoa single C code that can run on mobile devices
2. We explain the flow of construction of recognition
app using Caffe2C
3. We have implemented 4 kinds of mobile CNN-based
image recognition apps on iOS.

ⓒ 2016 UEC Tokyo.

2. CONSTRUCTION OF CNN-BASEDMOBILE RECOGNITION SYSTEMⓒ 2016 UEC Tokyo.

Caffe2C• In order to use the learned parameters by Caffe onmobile devices, it is necessary to currently use theOpenCV DNN module
not optimized, relatively slow
• We create a Caffe2C which converts the CNN model
definition files and the parameter files trained by Caffe
to a single C language code
– We can use parameter files trained by Caffe on mobile devicesⓒ 2016 UEC Tokyo.

Caffe2C• Caffe2C achieves faster execution speed in comparison
to the existing OpenCV DNN module

Runtime[ms] Caffe2C vs. OpenCV DNN(Input size: 227x227)
Caffe2C OpenCV DNN
AlexNetiPhone 7 Plus 106.9
1663.8
iPad ProiPhone SE

141.5
141.5

1900.12239.8ⓒ 2016 UEC Tokyo.

Speedup Rate:
About 15X～

Reasons for Fast Execution
1. Caffe2C directly converts the Deep Neural Network toa C source code
･Network･Mean･Label･ModelCaffe2C

Caffe2C
Single C code

Execution
like Compiler
Execution
like Interpreter

OpenCV DNN ⓒ 2016 UEC Tokyo.

Reasons for Fast Execution
2. Caffe2C performs the pre-processing of the CNN asmuch as possible to reduce the amount of online
computation–Compute batch normalization in advance for conv weight.

･Network
Caffe2C･Mean4 files
Single C code･Label･Model

3. Caffe2C effectively uses NEON/BLAS by multi-threadingⓒ 2016 UEC Tokyo.

Deployment Procedure
1. Train Deep CNN model by Caffe
2. Prepare model files
3. Generate a C source code by Caffe2C automatically
4. Implement C code on mobile with GUI code
4 Implement
12 ModelTrain Phase

Deep CNN

Preparation

ConvertTrained Deep
CNN Model ・Caffemodel C code
・Network・Mean
3・LabelCaffe2Cⓒ 2016 UEC Tokyo.

on Mobile3. IMAGE RECOGNITION SYSTEMFOR EVALUATIONⓒ 2016 UEC Tokyo.

Image Recognition Systemfor evaluation• In order to demonstrate the utilization of the Caffe2C，we have implemented four kinds of mobile CNNbased image recognition apps on iOS• We explain image recognition engine used in the iOSapplicationⓒ 2016 UEC Tokyo.

CNN Architecture• A representative architectures are AlexNet VGG-16 GoogleNet
or NINAlexNet

VGG-16
Network-In-Networkⓒ 2016 UEC Tokyo.

CNN Architecture• The number of weights in AlexNet and VGG-16 istoo much for mobile.
• GoogLeNet is too complicatedfor efficient parallel implemen-tation. (It has many branches.)ⓒ 2016 UEC Tokyo.

CNN Architecture• We adopt Network-in-Network (NIN).
– No fully-connected layers (which bring less parameters)– Straight flow and consisting of many conv layers– relatively smaller than the other architectures⇒ It’s easy for parallel implementation.
Efficient computation for conv layers is needed !

Network-In-Network(NIN)ⓒ 2016 UEC Tokyo.

Fast computation of conv layers- efficient GEMM with 4 cores and BLAS/NEON • Conv = im2col + GEMM (Generic Matrix Multiplication)321

kernel 1
kernel 2
kernel 3
kernel 4

patch1✕patch 1
patch 2
patch 3
patch 4
patch 54

im2colconv. kernels

patch2patch
3

patch4

patch5

input feature maps
matrix multiplication (=conv. layer computation)

Core1✕Core2kernel 3ⓒ 2016 UEC Tokyo.

✕Core3kernel 4✕patch 1
patch 2
patch 3
patch 4
patch 5kernel 2

patch 1
patch 2
patch 3
patch 4
patch 5✕patch 1
patch 2
patch 3
patch 4
patch 5kernel 1

patch 1
patch 2
patch 3
patch 4
patch 5

Parallel computation over multiple cores
Inside each core NEON or BLAS is used.

Core4

Fast Implementation on Mobile• Speeding up Conv layers →Speeding up GEMM
– computation of conv layer is decomposed into “im2col”
operation and generic matric multiplications(GEMM)– Multi-threading: Use 2cores in iOS , 4 cores in Android inparallel– SIMD instruction(NEON in ARM-based processor)• Total: iOS: 2Core*4 = 8calculation, Android: 4Core*4 = 16 calculation– BLAS library(highly optimized for iOS ⇔ not optimized forAndroid)• BLAS(iOS: BLAS in iOS Accelerate Framework, Android: OpenBLAS)ⓒ 2016 UEC Tokyo.

Evaluation: Processing time
• iOS: BLAS >> NEON, Android: BLAS << NEON
– For iOS, using BLAS in the iOS Accelerate Framework is thebest choice.
– For Android, using NEON (SIMD instruction) is better than
OpenBLAS.

Recognition Time[ms] BLAS vs. NEON
NEON
181.0

BLAS
55.7

DevicesiPhone 7 Plus

BLASAccelerate

iOSiOS
222.4 66.0
iPad ProAccelerateiOS251.8 79.9
iPhone SEAccelerate
Android 251.0 1652.0 GALAXY Note 3 OpenBLASⓒ 2016 UEC Tokyo.

Highlyoptimized

Comparison to FV-based Previous Method
Deep Learning with UEC-FOOD100 dataset• Much improved ( 65.3% ⇒ 81.5% (top-1) )• Even for 160x160 improved ( 65.3% ⇒ 71.5% )Top-NClassification
100.0%
Accuracy
95.0%

Top5:
96.2%

Top1:
81.5%90.0%

85.0%
80.0%

Top5:
86.7%

75.0%

AlexNet
Kept almost
NIN 5layer [104ms]
the same
NIN 4layer [67ms]
NIN 4layer (160x160) [33ms]
FV (Color+HOG) [65ms]

70.0%

Top1:
65.3%

65.0%

60.0%

1Top-N

2345ⓒ 2016 UEC Tokyo.

6789104. MOBILE APPLICATIONSⓒ 2016 UEC Tokyo.

4 iOS Applications• We have implemented 4 kinds of mobile CNN-based
image recognition apps on iOS– Food recognition app: “DeepFoodCam”– Bird recognition app: “DeepBirdCam”– Dog recognition app: “DeepDogCam”– Flower recognition app: “DeepFlowerCam”ⓒ 2016 UEC Tokyo.

DeepFoodCam• Recognize 101 classes including 100 food classes andone nonfood class

Training Phase• fine-tuned the CNN with 101 class images– totally 20,000 images– UECFOOD-100 and non-food collected from Twitter

Accuracy
TargetTop-1 Top-5
Food 101 class 74.5% 93.5%ⓒ 2016 UEC Tokyo.

DeepBirdCam• Recognize 200 bird class

Training Phase• fine-tuning CNN with 6033 images of Caltech-UCSD
Birds 200 Dataset

Accuracy
TargetBird 200 classⓒ 2016 UEC Tokyo.

Top-1 Top-5
55.8% 80.2%DeepDogCam• Recognize 100 dog class

Training Phase• fine-tuning CNN with 150 and over images per classof Stanford Dogs Dataset Dataset

Accuracy
TargetDog 100 classTop-1 Top-5
69.0% 91.6%ⓒ 2016 UEC Tokyo.

DeepFlowerCam• Recognize 102 flower class

Training Phase• fine-tuning CNN with 80 and over images per class of102 Category Flower Dataset

Accuracy
TargetTop-1Top-5

Flower 102 class 64.1% 85.8%ⓒ 2016 UEC Tokyo.

4 iOS Applications• We have implemented 4 kinds of mobile CNN-based
image recognition apps on iOS– Food recognition app: “DeepFoodCam”
If you prepare training
data, you can create– Bird recognition app: “DeepBirdCam”
mobile recognition– Dog recognition app: “DeepDogCam”
apps in a day !!
– Flower recognition app: “DeepFlowerCam”ⓒ 2016 UEC Tokyo.

Conclusions
1. We create a Caffe2C which converts the model
definition files and the parameter files of Caffe intoa single C code that can run on mobile devices
2. We explain the flow of construction of recognition
app using Caffe2C
3. We have implemented 4 kinds of mobile CNN-based
image recognition apps on iOS.

ⓒ 2016 UEC Tokyo.

Additional work
• We implemented apply our mobile framework into
real-time CNN-based mobile image processing– such as Neural Style Transferⓒ 2016 UEC Tokyo.

Object Recognition

iOS App is Available !
“DeepFoodCam“

Thank you for listening
Neural Style Transfer

iOS App is Available !
“RealTimeMultiStyleTransfer”ⓒ 2016 UEC Tokyo.

Extension of NIN
adding BN, 5layers, multiple image size
• Modified models (BN, 5layer, multi-scale)– adding BN layers just after all the conv/cccp layers– replaced 5x5 conv with two 3x3 conv layers– reduced the number of kernels in conv 4 from 1024 to 768– replaced fixed average pooling with Global Average Pooling4layers
5layers+BN• Multiple image size
227x227 180x180 160x160

Trade-off: Accuracy vs speedⓒ 2016 UEC Tokyo.Global Average Pooling (GAP)

227x22755.7ms 78.8%
180x180
35.5ms 76.0%
160x160
26.3ms 71.5%