Food Image Recognition Using
Deep Convolutional Networkwith Pre-training and Fine-tuning
ICME Workshop on Multimedia
for Cooking and Eating Activities (CEA)
July 3th 2015

Keiji Yanai and Yoshiyuki Kawano
The Univ. of Electro-Communications, Tokyo, Japan

Introduction: Food & Deep
•Food image recognition :– One of important topics in CEA community– Helpful for food habit recording.
•Deep Convolutional Neural Network:– Best image classification method at present•ILSVRC, Pascal VOC, MIT-SUN, Caltech-101/256,..

How about food datasets such as
UEC-Food101/256 and ETH Food-101 ?

http://foodcam.mobi/

FoodCam : [Kawano et al. MTA13]•Real-time mobile food recognition

Android application

Objectives of this work•[Experiments]

Introduce deep convolutional neuralnetworks (DCNN) into food image
classification task
•Examine its effectiveness•[Application]•Apply DCNN-based food classifier toTwitter food photo mining .

Deep Convolutional Neural
Network (DCNN)• The most common network architecture forimage classification: AlexNET proposed byAlex Krizhevsky in ILSVRC2012.
Alex Krizhevsky, I.Sutskever, J. Hinton : ImageNet Classification with Deep
Convolutional Neural Networks, NIPS 2012.

We use this in this work.

6

Deep Convolutional Neural Network•Consists of convolutional layers and fullconnection layers.

Convolutional layers

Full-connection layers⇒ Feature extraction part⇒ classification part

Most activated location on 3rdconv. layer

[Exisiting works]
DCNN with Food Dataset•[Kawano et al. CEA2014]– DCNN activation features(pre-trained with ILSVRC data)– UEC-FOOD100: FV: 65.32%•DCNN 57.87%, DCNN+FV: 72.26% (best so far)•Other work:– [Kagaya et al. MM2014] 10 kinds of foods
with DCNN trained from scratch– [Bossard et al. ECCV2014] ETH-Food101
with AlexNet from scratch

４０９６-dim vector

Three ways to use DCNN

(L2 normailized)

Kagaya et al. , Bossard et al.

1. Train DCNN from scratch– Need large number of training images (1000～)– Take long time to train. (～1 week) Kawano et al.

2. Use activation features of pre-trained DCNN
– Extract activation signals from the previous layerof the last one, and use them as visual features– Easiest among three. (by using Caffe or Overfeat)3. Fine-tune a pre-trained DCNN

using non-large-scale dataset

Not explored yet– Even small data can improve performance over(2)Introducing pre-training with
non-ILSVRC and Fine-tuning•Existing works: training from scratch orusing pre-trained model with ILSVRC
Two kinds of extensions•Pre-training with food-related

ImageNet categories•Fine-tuningDCNN pre-training Datasets•ILSVRC2012– Large Scale Visual Recognition Challenge– one thousand training images per category– Generic 1000 categories– few food categories

DCNN pre-trained with the dataset containing
more food-related categories is desirable.

DCNN pre-training Datasets
containing more foods•Select 1000 food-related categories

from ImageNet– List up all the word under “food” in theImageNet hierarchy– 1526 synsets related to “food”– Exclude synsets included in ILSVRC dataset– Select the top 1000 synsets in terms of #of images in ImageNet* ImageNet 2011 Fall release

DCNN pre-training Datasets•ImageNet2000 categories– 1000 food-related categories from ImageNet– ImageNet1000 (ILSVRC) categories= 2000 categories

Pre-training with ImageNet2000•DCNN Features with ImageNet 2000 categories– Using Caffe– The dimension of full connection layers is modifiedfrom 4096 to 6144, since the output dimension israised from 1000 to 2000.
•Training time
– About one week (training from scratch)– GPU, Nvidia Geforce TITAN BLACK, 6GB

Fine-tuning with small dataset•Modify the size of the last layer (2000⇒100)•Re-train only weight parameters of fullconnection layers (L6,7,8)•Weights from L1-5 are fixed

100100Fine-tuning•It enables DCNN to be trained with smalldata.
•Feature extraction parts are trained withlarge-scale data such as ImageNet.
•Classification parts are trained with smallscale target data.

Feature extraction part

Classification part

Experiments: Food dataset•UEC-FOOD100/256 dataset– 100 / 256 food categories (Japanese and Asian)– More than 100 images for each category– Bounding box information for all the images•ETH Food-101 [Bossard et al. ECCV2014]– 101 categories, 1000 images for each category(mainly western and partly Japanese)– Collecting from http://www.foodspotting.com/– 20 categories are overlapped with UEC-FOOD100

UEC-FOOD 100

UEC-FOOD 100

UEC-FOOD 100

UEC-FOOD 100

UEC-FOOD 100

FoodRec: foodrec app with UECFOOD100
by Hamlyn Centre-Imperial College(UK)UEC-FOOD as a Fine-Grained
Image Classification Datasetⓒ 2014 UEC Tokyo.

UEC-FOOD 256

UEC-FOOD 256

UEC-FOOD 256

UEC-FOOD 256

ETH-Food101•Sushi•Takoyaki•RamenBaseline features & classifiers•Conventional baseline features– Root HOG patch and Color patch– Fisher Vector (FV)– SPM level2 (1x1+3x1+2x2)– 32768-dim RootHOG-FV– 24576-dim Color-FV•Classifiers– 1-vs-rest multiclass linear SVM•Evaluation: 5fold cross-validation

Results: UEC-FOOD100

Classification rate

+6.5%
+5.5%
+14%

# of candidates

Results: UEC-FOOD256

Classification rate+5%+9%+15%# of candidates

Results on ETH-Food101•Fine-tuned DCNN withImageNet+food1000 pretraining
achived the best results.

Summary•Pre-training with ImageNet + Food1000

is effective.
•Fine-tuning can improve performance overPre-training DCNN + FV.
•Fine-tuning the DCNN which was pre-trainedwith ImageNet1000 + Food1000 is the best.

Comments:•For further improvements– Using deeper network (VGG16 or
GoogLeNet) instead of Alexnet– Fusing DCNN-FOOD(ft) with FV•Both are not examined yet.
– (Because the second author graduated…)

Apply DCNN-FOOD(ft)to Twitter Real-time
Food Photo Mining
Keiji Yanai and Yoshiyuki Kawano: Twitter Food Image Mining and Analysis for
One Hundred Kinds of Foods , Pacifit-Rim Conference on Multimedia (PCM),(2014).
Yoshiyuki Kawano and Keiji Yanai, FoodCam: A Real-time Food Recognition
System on a Smartphone, Multimedia Tools and Applications (2014). (inpress) (http://dx.doi.org/10.1007/s11042-014-2000-8)

Twitter Real-time Food Photo
Mining System (mm.cs.uec.ac.jp/tw/)• What kinds of foods are being eaten in Japan ?

Objective•Twitter Photo Mining for Food Photos– As a case study of Twitter Photo Mining
on specific kinds of photos– Food is one of frequent topics of Twitter Photos.

– Real-time Photo Collection from the stream•To collect more food photos for training– Twitter is a good source of food photos.
– Unlike “FoodLog”, we have no users who upload
their food photos regularly. Twitter is alternative.

Preparation•Add “non-food” category to the bestclassifier (FOOD-DCNN(ft)).
•Prepare 10000 non-food samples
gathered from Twitter by 100 food names•Fine-tuning with 101 category.

•Food/non-food classification: 98.86%

Approach for food photo mining•[old] Three-step food photo selection
Keywordbased
selection

Food/nonfoodclassification

100-foods
classification

Image-based analysis[new] Two-step food photo selection
Keywordbased
selection

101-foods
classification

Experiments•Collect photo tweets via Twitter Streaming

API– From 2011/5 to 2013/8– About one billion tweets•Search for the tweets including any of 100-

food names (in Japanese)– 1.7 million ⇐ Apply food image classifier– 0.03 image/classification w/GPU(4 hours by 4 GPU machines)

Twitter food photo ranking
rankfoods#photos1

Ramen noodle80,021

2

Curry

59,264

3

Sushi

25,8984

Dipping noodle (tsukemen)

22,1585

Omelet17,5206

Pizza

16,921

7
Jiaozi16,014
Ramen noodle is the most popular food in Japan.
8 I have
Okonomiyaki
solved “ramen vs curry” problem !!!15,234

Precision of the top 5 foods
Foodraw

ramen

275,652
72.0%
224,68575.0%
86,50969.0%
33,165
88.7%
34,125
90.0%

curry

sushi
tsukemenomeletFV-based
80,021
99.7%
59,26499.3%
25,89892.7%
22,15899.0%
17,52099.0%

DCNN132,095
99.5%
68,091100.0%
224,908
99.8%
22,004100.0%
20,03999.9%

Only keyword search(Ramen noodle) (72.0%)

After applying 100-class foodclassifier (final)(99.5%)

Only keyword search (curry)(75.0%)

Final results (curry) (100%)

Final results (omelet) (99.9%)

Misclassified photos

Geographical-Temporal analysis
on ramen vs curry
12.6% of the obtained food photos have geotag.

Whole year

Dec. (winter)Aug. (summer)

Ramen is popular. Curry gets more popularthan ramen in many areas.
RamenCurryUtilization of large number offood photos: Omerice analysis•Omerice-style classification– Classification rate: 84.184％

letters

drawing texture source plain認識結果：
各1000枚で学習．1500枚を分類
分類先⇒ 文字絵模様

ソース プレーン

文字

85.6

10.0

3.1

1.1

0.1

絵

9.9

84.2

3.5

1.9

0.5

模様

3.1

5.7

80.7

8.8

1.6

ソース

1.2

1.9

6.9

86.5

3.5

プレーン

0.0

2.9

0.0

15.8

81.3

Real-time Food Collection•Monitor the Twitter stream– Photo Tweet– Text including any of 100 food names•13 candidate photo tweets / minute on avg.
•Download: 2～3sec. , recognition: ～1sec.
•Single machine is enough !

•Recognize 20,000 photos andfind 5,000 food photos from the TW
stream everyday in our lab

Demo visualization system•Map each food photo on an online mapwith online clustering [Yanai ICMR2012]– Geotagged Tweets– Non-geotagged Tweets for which GeoNLP can
assign locations based on text msg.
•Overlay a food photo on the Streetview– Finding “ramen noodle shop” game !

http:/mm.cs.uec.ac.jp/tw/Twitter Food Image Bots Bot who recognize foodphotos and return results
 @foodimg_bot

 Bot who re-tweets foodphoto tweets automaticallyDeepFoodCamRelease soon
at http://foodcam.mobi/•DCNN-based food rec. app– Network-in-network(NIN) instead of AlexNet– PQ-based weight compression (7MB ⇔ 256MB)

Conclusions•Food recognition with DCNN features– Pre-trained DCNN with ImageNet2000 categories– Fine-tuned the pre-trained DCNN with UEC-FOOD•Achieved the best performance so far– UEC FOOD-100: 78.48%– UEC FOOD-256: 94.85%•We showed the effectiveness for theapplication, Twitter food photo mining

Future works•CNN-based food region segmentation

Thank you
for your attention !

99

2.1 group and some food categories1012.1 group and some food categories102

2.1 group and some food categories

1032.1 group and some food categories

1042.1 group and some food categories

105

【復習】

クラウドソーシング体験課題

106大規模画像認識のための標準ネットワーク構成 : Alex Net• ILSVRC 2012 で，Alex Krizhevskyらが用いたネットワーク構成．
– Alex Krizhevsky, I.Sutskever, J. Hinton : ImageNet
Classification with Deep Convolutional NeuralNetworks, NIPS 2012.
• 2013，2014はどのチームもこれをベースに改良，拡張．
⇒ 事実上の標準ネットワーク．

108

Convolutional network•前半が畳み込み層(convolutional layer),後半が昔と同じ全結合層(fullconnection)

Convolutional layer
109

Full-connection layer