Real-time Photo Mining
from the Twitter Stream:
Event Photo Discovery
and Food Photo Detection
International Symposium on Multimedia (ISM)
Dec. 9th 2014
Keiji Yanai and Yoshiyuki Kawano
The Univ. of Electro-Communications, Tokyo, Japan

Background•Various kinds of photos are posted tomicroblogs every minutes.
•The photos on the microblogs areuploaded with text messages.
•Microblogs such as Twitter can be

regarded as another tagged photo source
than Flickr.

Twitter vs. Flickr as photo sources
Photosharing

Microblog• Travel, special event• Everyday life

• w/ keywords, description– describing the contents• w/ tweet messages– Not describing• offline uploading• instant (online)• manyphotos
at once isThebiggestdifference
online•offline
3 millionor/dayuploading –few photos• 30 million/day

Twitter photos are more helpful to understandthe current state/trend of the worlds .

So many works withFlickr tags are reliable, which can be regarded as “concept labels”.

•Flickr Distance [ACM MM 2008]– Measuring concept distances•Concept difference [CIVR 2009]– Detection of regional concepts•IM2GPS [CVPR 2008]– Estimating photo locations
with million-scale geotagged photos•Travel Analysis [ACM MM 2010]– Travel trajectory analysis with geo-photos

Many text mining works with
•Text analysis– Event detection– Trend mining– Positive/Negative reputation Typhoon trajectory estimatedby tweets [WWW 2010]•Photo analysis ⇒ very limited– Evaluation of relatedness between msg. and img.
– Brand image mining– Our works (event/food photo mining)Related works on TW photos•Classifying “visual”/”non-visual” tweets

by generic methods [Chen et al. MM13]
ｄｄVisual70.5 %Non-visual•Brand image mining [Gao et al. ICMR 2014]– Supervised logo detectorｄｄ

Our works on Twitter photos•Real-time TW geo-photo mapping system[ICMR 2012]•Visual topic analysis on TW photos[new results]•Event photo mining [ICME WS 2013]•Food photo mining [PCM 2014]

World Seer: A Real-time
Geo-Tweet Photo Mapping
SystemICMR 2012

World Seer: Real-time Twitter
Photo Mapping System [ICMR 2012]

Build geo-photo tweet database for researchMonitoring the TW stream &Recording Geo-Photo Tweets

demo•http://mm.cs.uec.ac.jp/tw/Tweet photo database•Since Feb. 2011
– 1 billion photo tweets– 200 million geo-photo tweets•We are doing several researcheswith this data
– Event Photo Mining– Food Photo Mining– Visual Topic Analysis

Characteristic of the Twitter photos•Normal condition : everday life
– Food– Scene– People•Something special: event photos– Artificial public events sport games– Natural phenomena
earthquake, typhoon– Personal events•go hiking, travel, birthdayTwitter photos: special event

Special big event photos on March 11th 2011
around Tokyo area

Twitter photos: special event

Everyday-life photos on March 11th 2011
in the western part of Japan

Visual Topic Analysis of
Twitter Photo Analysis
Unpublished.

demo•http://mm.cs.uec.ac.jp/twimg/•http://mm.cs.uec.ac.jp/twimg/dcnn.cgi

Food is one of the major topics ofTwitter photos•Visual topic analysis with half-million

Twitter photos employing DCNN fea.
Food-related topics

Food-related topics

Mining two types of photos•Event photo : special•Food photo: everyday-lifeTwitter Event Photo Mining
Takamu Kaneko and Keiji Yanai: Visual Event Mining from Geotweet Photos, IEEE ICME Workshop on Social Multimedia Research(SMMR), (2013).

Demo•http://mm.cs.uec.ac.jp/kaneko-t/tw/jp/index.html•http://mm.cs.uec.ac.jp/kaneko-t/tw/us/index.html

Twitter Event Photo Mining• Mine the photos related to the events

happened in the specific areas and times

The results of detected event photos in 2012

Twitter Event Photo Mining•ある日のある場所で，急に増えたキーワードを抽出し，関連画像を自動的に選択

２０１２年のアメリカでのイベント検出結果

Objective•Detect events from Twitter stream-Weather, natural events-Festivals, sport games•Understand events visually-Select representative photos-Map in a map
Mapping events with the photoProcessing flow
1. Event keyword detection
2. Keyword unification

3. Event photo clustering
4. Mapping event with photos

Event Keyword Burst Detection•Examine change of daily frequency

Keyword Unification andComplement•Unification keyword-more than half of the same tweets-Usethe “Endeavor”
highest score keyword“shuttle”,“shuttle”•Complement keyword-more than 80% of the same word
-beforeand after“Musicthe keyword“Festival”Festival”

Event Photo Clustering•Image features-Bag-of-Features with SURF
-Color histograms•Ward method-a hierarchical clustering method-threshold is 300(both)
𝐸 𝐶 =2𝑤
2𝑤(𝑥−𝑥+𝑥−𝑥
𝐵𝑜𝐹
𝐵𝑜𝐹
𝑅𝐺𝐵𝑅𝐺𝐵𝑅𝐺𝐵 )𝐵𝑜𝐹𝑥∈𝐶Event Photo Selection•Select a representative cluster-evaluate cluster score onrepresentativeness#𝑝ℎ𝑜𝑡𝑜𝑠𝐶 2𝑉𝐶 =𝑊𝑎𝑟𝑒𝑎
𝐸(𝐶)•Select a representative photo-from the maximum score cluster•Eliminate lower score cluster-less than 5(JPN) , 20(USA)Experiments•Japan Dataset 1 in Japan-Feb 10th, 2011 to Sep 30th, 2012
-about 3 million geo-tweet photos•US Dataset-Jan 1st, 2012 to Dec 31st, 2012
-about 17 million geo-tweet photos

Results of Keyword Detection
Keyword

DateKeyword

Datesnow

11/02/2011

snow

09/01/2012

earthquake

11/03/2011

sunset13/01/2012

fireworks

06/08/2011

Grammy12/02/2012

typhoon21/09/2011

Valentines

14/02/2012

Mt. Fuji

24/09/2011

SXSW

09/03/2012

Apple

06/10/2011

Easter

08/04/2012

eclipse10/12/2011

shuttle17/04/2012

illumination

10/12/2011

WWDC

10/06/2012

Christmas

24/12/2011

hurricane

26/08/2012

New yearseve

31/12/2011

rainbow

05/09/2012

sunrise01/01/2012

49ers

18/10/2012

firefly

06/05/2012

NYE

31/12/2012

JapanUSA“fireworks” photo clusters

“cherry blossoms” photo clusters
daytimenightMapping Results•Map event in a map-Calculate coordinates of event-Correspond information and the photo•Summary of results
JapanUSA# events

258

1676

accuracy

65.5% 72.5%

Result“cherry“Fire“fireworks”
of Japan
Festival”
blossoms”
dataset“River’sResult“BalloonEdge“rainbow”
of Music
USFiesta”
datasetFestival”

“sunset”

Twitter Real-time
Food Photo Mining
Keiji Yanai and Yoshiyuki Kawano: Twitter Food Image Mining and Analysis for
One Hundred Kinds of Foods , Pacifit-Rim Conference on Multimedia (PCM),(2014).
Yoshiyuki Kawano and Keiji Yanai, FoodCam: A Real-time Food Recognition
System on a Smartphone, Multimedia Tools and Applications (2014). (inpress) (http://dx.doi.org/10.1007/s11042-014-2000-8)

Twitter Realtime Food Photo
Mining System (mm.cs.uec.ac.jp/tw/)• What kinds of foods are being eaten in Japan ?

Objective•Twitter Photo Mining for Food Photos– As a case study of Twitter Photo Mining
VSon specific kinds of photos– Food is one of frequent topics of Twitter Photos.

– Real-time Photo Collection from the stream

RamenCurry•Which food is the most popular in Japan?
– “Ramen vs Curry”problem ⇒ very controversial– I would like to put a period to this controversy by
Twitter food photo mining !!!

Approach for food photo mining•Two-step food photo selection– [1] Keyword-based tweet selection– [2] Image-based photo selection•Generic food/non-food classification•Specific food classifiers (100 kinds)
Keywordbasedselection

Food/nonfoodclassification

100-foods
classification

Image-based analysis

Targets: 100 kinds of foodsin the UEC-Food100 data set•Includes common foods in Japan•Has more than 100 images/category

RamenCurry

[1] Keyword-based selection•Select the photo tweets the messagesof which include any of 100 kinds offood names– In the experiments, we used Japanese foodnames.
– We tried query expansion as well.
e.g.) I came to eat ramen noodle.
Very delicious ramen !!!
Ramen is my life.

[2] Two-step
image-based selection[2-1] Food/non-food classification– Remove non-food photos andselect only food photos[2-2] Specific food classifiers– Extended version of FoodCam recognitionengine. 1-vs-rest 100-class classification– Select the food photo if the corresponding foodcategory is ranked within the top five.
I ate sushi !

100-classfood classification[top-5] Pizza, ramen,curry, sushi, tempra

SushiPhotoFood recognition method•Local patch + Fisher Vector + linear SVM– Color patch, HOG patch•Color: 24 dim HOG: 32 dim•dense sampling– GMM: K=64
– Spatial Pyramid: 1x1 + 2x2– Improved Fisher Vector [Perronnin et al.2010]•Color: 15360 dim, HOG: 20480 dim– Classifier•Linear SVM

[2-1]Food/non-food Classifier•Train 13 linear SVMs
– Pos.： UEC-FOOD 100– Neg.: typical irrelevant photos•Inside/outside restaurant•Menu, people eating, …

13SVM

test

train

train

SVM

test

SVM

test
•Classify of food/non-foodmax– The maximum value of output of13 classifiers with pre-defined
Foodness scorethreshold values

groupstrain

[2-2]100-class specific foodcategory recognition• 100-class food classification engine• Extended version of FoodCam[Kawano et al. MTA14]– Very fast (0.025 sec. / image)•Multi-threaded implementation optimized forquad core CPU
Suitable for big data recognition– HOG-FV + Color-FV + 1-vs-rest linear SVM

http://foodcam.mobi/

FoodCam : [Kawano et al. MTA13]•Real-time mobile food recognition

Android application

Classification rate

100-Classclassificationrate
10.9
0.8
0.7

Rec. time: 0.024 sec.

0.60.5パラメタ変更後
Extended ver.

0.4

FoodCam(mobile)
モバイル

0.3

0.2

Top-1 rate: ６０％，Top-5 rate:８３％

0.1
012

34

567

# candidates

8910

Experiments•Collect photo tweets via Twitter Streaming API– From 2011/5 to 2013/8– About one billion tweets•Search for the tweets including any of 100-foodnames (in Japanese)– 1.7 million ⇐ Apply food image analysis•Food/non-food classifier+ 100-food classifier– 470,335 food photos

Evaluations on five kinds ofrepresentative food•Num. of obtained food images•Precision (random sampling of 300imgs)(1) Only keyword search(2) Keyword + food/non-food classifier(3) Keyword + specific food classifier(4) All (kw+food/non-food+specific) proposed•Geographic analysis with geotagged photos– Ramen vs CurryTwitter food photo ranking
rankfoods#photos

1Ramen noodle

80,021

2

Curry

59,264

3Sushi25,898

4Dipping noodle (tsukemen)22,158

5Omelet17,520

6Pizza16,921

7
Jiaozi16,014
Ramen noodle is the most popular food in Japan.
8 I have
Okonomiyaki
solved “ramen vs curry” problem !!!15,234

Precision of the top 5 foods
Food

(1)KW

(2) f/nramen

275,652
72.0%
224,68575.0%
86,50969.0%
33,165
88.7%
34,12590.0%

200,17392.7%
163,047
95.0%
43,53686.0%
24,89696.3%
28,887
96.3%

curry
sushi
tsukemenomelet(3)spec. (4) ALL
84,18995.0%
62,82497.0%
48,01972.3%
28,846
93.7%
18,37098.0%

80,021
99.7%
59,26499.3%
25,89892.7%
22,15899.0%
17,52099.0%

Only keyword search(Ramen noodle) (72.0%)

After applying food/non-foodclassifier (92.7%)

After applying 100-class foodclassifier (final)(99.7%)

Only keyword search (curry)(75.0%)

Final results (curry) (99.3%)

Some interesting findings•Letters or drawings are sometimes drawn onomelets with ketchup•Fast-foods such as humberger(rank 30th) andbeef bowl(rank 27th) are ranked lower,
since their appearance is always the same.
Not worth posting fastfood photos to Twitter

Geographical-Temporal analysis
on ramen vs curry
12.6% of the obtained food photos have geotag.

Whole year

Dec. (winter)Aug. (summer)

Ramen is popular. Curry gets more popularthan ramen in many areas.
RamenCurryReal-time Food Collection•Monitor the Twitter stream– Photo Tweet– Text including any of 100 food names•13 candidate photo tweets / minute on avg.
•Download: 2～3sec. , recognition: ～1sec.
•Single machine is enough !

•Recognize 20,000 photos andfind 5,000 food photos from the TW
stream everyday in our lab

Demo visualization system•Map each food photo on an online mapwith online clustering [Yanai ICMR2012]– Geotagged Tweets– Non-geotagged Tweets for which GeoNLP canassign locations based on text msg.
•Overlay a food photo on the Streetview– Finding “ramen noodle shop” game !

http:/mm.cs.uec.ac.jp/tw/Twitter Food Image Bots Bot who recognize foodphotos and return results
 @foodimg_bot

 Bot who re-tweets foodphoto tweets automaticallyAdditional work for more Ramen
Photos: Finding “Koike-san”
Koike-san is a
Fujiko-Fujio comic’s
character who loves“ramen noodle”.
He is always eating“ramen noodle” when
he appears in thecomic. (Wikipedia)

Finding “Koike-san” on Twitter-query expansion based on user, loc & word•Pick up the top-k users who frequently

post ramen photos.
– (k=30 in the experiments)•Apply food classifiers toraumenbot,657,0
toyamamenrui (map),128,128
oishii_bot2,107,0Ramen___Bot,90,0
daikoubutsu_bot,88,0
shibumen,65,0
vientM535i (map),50,36
oishii_bot,45,0
kido_maru,42,0
ishikawamenrui (map),42,42
shomax96 (map),36,33
rAsAmAya (map),36,28

all the photos “Koike-sans” posted.

•Other methods: Finding “Koike-san”

places , Finding co-occurence words

More “ramen/curry/sushi”•Precision is not as good as the results by

keyword-based candidate selection.
Frequent user
“Koike-sans”

Frequent
Frequent
co-occurence places
word

Ramen

6050
58.0 %

5851
68.5 %

594
44.0 %

Curry

3163
23.5 %

2806
49.0 %

313
25.5 %

Sushi

2474
13.5 %

1591
41.5 %

991
17.0 %

State-of-the-art (DCNN-based)(presented at MDBA WS)

Improved by17 points

Top-1 rate 77.4％，Top-5 rate 94.8％

Conclusions

Conclusions•Food Photo Mining from Twitter Photodata / the Twitter stream.
•Have completely solved the “ramen vs.

curry” problem.
– Note that only in summer searson, Curry
becomes more popular than Ramen.

•Real-time system (demo)Future work
•One million “ramen noodle photodataset”– For all the “Ramen” fans over the world.

•Methods for collecting more Ramen !
– Use DCNN-based classifier– Improve “Koike-san” methods•Extension to World-wide foods

Thank you
for your attention !

87

2.1 group and some food categories

892.1 group and some food categories

902.1 group and some food categories

912.1 group and some food categories

922.1 group and some food categories

93