Twitter Food Photo Miningand Analysis for One
Hundred Kinds of Foods
Pacific-Rim Conf. on Multimedia (PCM)
Dec. 2nd 2014
Keiji Yanai and Yoshiyuki Kawano
The University of Electro-Communications,Tokyo, Japan

Twitter Realtime Food Photo
Mining System (mm.cs.uec.ac.jp/tw/)• What kinds of foods are being eaten in Japan ?

Background•Various kinds of photos are posted tomicroblogs (Twitter) every minutes•Twitter photos are uploaded with text
messages (Tweet messages)•Microblogs such as Twitter can beregarded as another tagged photosource than Flickr.

Twitter(TW) vs Flickr(FL)•Same points:– Photos with texts– Easy to collect data via WebAPI•Different points:– On-the-spot ⇔ at home (from mobile vs from PC)– TW:30million/day vs FL:3million/day– TW msg is not tag. Free msg.
•FL: Tags describes contents of photos in general•TW: Attached msgs do not always describe thecontent of the photos
needs for image analysis

Twitter Food Photo Mining•Twitter Photos represent thecurrent state of the world !
•Mining food photos from Twitter isthe best way to get to know whatpeople are eating now !!!
– What kinds of ramen is being eaten at
this moment in Japan / over the world?

•We propose

Twitter Food Photo MiningObjective•Twitter Photo Mining for Food Photos– As a case study of Twitter Photo MiningVSon specific kinds of photos– Food is one of frequent topics of Twitter Photos.

– Real-time Photo Collection from the stream

RamenCurry•Which food is the most popular in Japan?
– “Ramen vs Curry”problem ⇒ very controversial– I would like to put a period to thiscontroversy by Twitter food photo mining !!!

Related work on Twitter photo•Event photo mining from geotaggedTweet photos [ICME WS 2013]– Text-based event detection + photo sele.

The results of detected event photos in 2012

Related works on TW photos(2)•Classifying “visual”/”non-visual” tweets

by generic methods [Chen et al. MM13]
Visual70.5 %

Non-visual•Brand image mining [Gao et al. ICMR 2014]– Supervised logo detector

Approach for food photo mining•Two-step food photo selection– [1] Keyword-based tweet selection– [2] Image-based photo selection•Generic food/non-food classification•Specific food classifiers (100 kinds)
Keywordbasedselection

Food/nonfoodclassification

100-foods
classification

Image-based analysis

Targets: 100 kinds of foodsin the UEC-Food100 data set•Includes common foods in Japan•Has more than 100 images/category

RamenCurry

[1] Keyword-based selection•Select the photo tweets themessages of which include any of
100 kinds of food names– In the experiments, we used Japanesefood names.
– We tried query expansion as well.
e.g.) I came to eat ramen noodle.
Very delicious ramen !!!
Ramen is my life.

[2] Two-step
image-based selection[2-1] Food/non-food classification– Remove non-food photos andselect only food photos[2-2] Specific food classifiers– Extended version of FoodCam recognitionengine. 1-vs-rest 100-class classification– Select the food photo if the corresponding
food category is ranked within the top five.
I ate sushi !

100-classfood classification[top-5] Pizza, ramen,curry, sushi, tempra

Sushi
Photo

Food recognition method•Local patch + Fisher Vector + linear SVM– Color patch, HOG patch•Color: 24 dim HOG: 32 dim•dense sampling– GMM: K=64
– Spatial Pyramid: 1x1 + 2x2– Improved Fisher Vector [Perronnin et al.2010]•Color: 15360 dim, HOG: 20480 dim– Classifier•Linear SVM

[2-1]Food/non-food Classifier•Train 13 linear SVMs
– Pos.： UEC-FOOD 100– Neg.: typical irrelevant photos•Inside/outside restaurant•Menu, people eating, …

13SVM

test

train

train

SVM

test

SVM

test
•Classify of food/non-food– The maximum value of output of max
13 classifiers with pre-defined
Foodness scorethreshold values

groupstrain

[2-2]100-class specific foodcategory recognition• 100-class food classification engine• Extended version of FoodCam[Kawano et al. MTA14]– Very fast (0.025 sec. / image)•Multi-threaded implementation optimized fo
quad core CPU
Suitable for big data recognition– HOG-FV + Color-FV + 1-vs-rest linear
SVMhttp://foodcam.mobi/

FoodCam : [Kawano et al. MTA13]•Real-time mobile food recognition

Android application

Classification rate

100-Classclassification
rate
10.9
0.8
0.7

Rec. time: 0.024 sec.

0.60.5パラメタ変更後
Extended ver.

0.4

FoodCam(mobile)
モバイル

0.30.2Top-1 rate: ６０％，Top-5 rate:８３％

0.101234567# candidates

8910

Experiments•Collect photo tweets via Twitter Streaming

API– From 2011/5 to 2013/8– About one billion tweets•Search for the tweets including any of 100-food names (in Japanese)– 1.7 million ⇐ Apply food image analysis•Food/non-food classifier+ 100-food classifier– 470,335 food photos

Evaluations on five kinds ofrepresentative food•Num. of obtained food images•Precision (random sampling of 300imgs)(1) Only keyword search(2) Keyword + food/non-food classifier(3) Keyword + specific food classifier(4) All (kw+food/non-food+specific) proposed•Geographic analysis with geotagged photos– Ramen vs CurryTwitter food photo ranking
rankfoods#photos1

Ramen noodle

80,021

2

Curry

59,264

3

Sushi

25,8984

Dipping noodle (tsukemen)

22,1585

Omelet17,5206

Pizza

16,921

7
Jiaozi16,014
Ramen noodle is the most popular food in Japan.
8 I have
Okonomiyaki
solved “ramen vs curry” problem !!!15,234

Precision of the top 5 foods
Food

(1)KW

(2) f/nramen

275,652
72.0%
224,68575.0%
86,50969.0%
33,165
88.7%
34,12590.0%

200,17392.7%
163,047
95.0%
43,53686.0%
24,89696.3%
28,88796.3%

currysushi
tsukemenomelet(3)spec. (4) ALL
84,18995.0%
62,82497.0%
48,01972.3%
28,846
93.7%
18,37098.0%

80,021
99.7%
59,26499.3%
25,89892.7%
22,15899.0%
17,52099.0%

Only keyword search(Ramen noodle) (72.0%)

After applying food/nonfood classifier (92.7%)

After applying 100-classfood classifier (final)(99.7%)

Only keyword search (curry)(75.0%)

Final results (curry) (99.3%)

Some interesting findings•Letters or drawings are sometimes drawn

on omelets with ketchup•Fast-foods such as humberger(rank 30th)and beef bowl(rank 27th) are ranked lower,
since their appearance is always the same.
Not worth posting fastfood photos to Twitter

Geographical analysis on
ramen vs curry
12.6% of the obtained food photos have geotag.

Whole year

Dec. (winter)Aug. (summer)

Ramen is popular. Curry gets more popular
Ramen
Curry than ramen in many areas.

Real-time Food Collection•Monitor the Twitter stream– Photo Tweet– Text including any of 100 food names•13 candidate photo tweets / minute on avg.
•Download: 2～3sec. , recognition: ～1sec.
•Single machine is enough !

•Recognize 20,000 photos andfind 5,000 food photos from the TW
stream everyday in our lab

Demo visualization system•Map each food photo on an online mapwith online clustering [Yanai ICMR2012]– Geotagged Tweets– Non-geotagged Tweets for which GeoNLP
can assign locations based on text msg.
•Overlay a food photo on the Streetview– Finding “ramen noodle shop” game !

Additional work for more RamenPhotos: Finding “Koike-san”
Koike-san is a
Fujiko-Fujio comic’s
character who loves“ramen noodle”.
He is always eating“ramen noodle” when
he appears in thecomic. (Wikipedia)

Finding “Koike-san” on Twitter-query expansion based on user, loc & word•Pick up the top-k users who

frequently post ramen photos.
– (k=30 in the experiments)•Apply food classifiers toraumenbot,657,0
toyamamenrui (map),128,128
oishii_bot2,107,0Ramen___Bot,90,0
daikoubutsu_bot,88,0
shibumen,65,0
vientM535i (map),50,36oishii_bot,45,0
kido_maru,42,0
ishikawamenrui (map),42,42
shomax96 (map),36,33
rAsAmAya (map),36,28

all the photos “Koike-sans” posted.

•Other methods: Finding “Koike-san”

places , Finding co-occurence words

More “ramen/curry/sushi”•Precision is not as good as the resultsby keyword-based candidate selection.
Frequent user
“Koike-sans”

Frequent
Frequent
co-occurence places
word

Ramen

6050
58.0 %

5851
68.5 %

594
44.0 %

Curry

3163
23.5 %

2806
49.0 %

313
25.5 %

Sushi

2474
13.5 %1591
41.5 %

991
17.0 %

State-of-the-art (DCNN-based)(presented at MDBA WS)

Improved by
17 points

Top-1 rate 77.4％，Top-5 rate 94.8％

Conclusions•Food Photo Mining from Twitter

Photo data / the Twitter stream.
•Have completely solved the “ramenvs. curry” problem.
– Note that only in summer searson, Curry
becomes more popular than Ramen.

•Real-time system (demo)Future work
•One million “ramen noodle photodataset”– For all the “Ramen” fans over the world.

•Methods for collecting more Ramen !
– Use DCNN-based classifier– Improve “Koike-san” methods•Extension to World-wide foods

Thank you
for your attention !

2.1 group and some food categories

52

2.1 group and some food categories

53

2.1 group and some food categories

542.1 group and some food categories

55

2.1 group and some food categories

56