Large-Scale Twitter Food Photo
Mining and Its Applications
The Fifth IEEE International Conference
on Multimedia Big Data (BIGMM)
Sep. 11th 2019

Keiji Yanai, Kaimu Okamoto,Tetsuya Nagano and Daichi Horita
Artificial Intelligence eXploration Research Center

The Univ. of Electro-Communications, Tokyo, Japan

Overview of this talk• Introducing our Twitter photo mining works
since Feb. 2011 (one month before the big earthquake)– Geotagged tweet photo analysis• Real-time geo-tweet photo mapping system [2012]• Event photo mining from geo-tweet photos [2012-2016]• Finding regional tendency on Twitter photos [2019]– Twitter food photo mining• Statistics on food image collection for 8 years [2012-]• Regional tendency on Twitter food photos [2019]• Applications of a large-scale Twitter food photoDB [2018-]– Food image translation by GAN, mobile app., food VR

Background• Various kinds of photos are posted to SNSssuch asevery seconds.
• Photos on SNSs are posted with text
messages and meta data such as geotags.
• SNSs can be regarded as usefuldata sources for multimedia research.
ⓒ 2016 UEC Tokyo.

Why
•

?

provides the API to watch the Tweet
stream in the real time way.
– Twitter API statuses/filter(formerly TW Streaming API(～2018/8))•

do not provide real-time API.
– Most of the Facebook msg. are not public.
– Instagram msg. are public, but its API is highly
restricted (mainly designed for mobile apps.)ⓒ 2016 UEC Tokyo.

vs
Image
Search eng.

• Typical, very relevant• Distributed over the Web– Various purpose

Microblog• Everyday life
• w/ tweet messages– Not describing

biggest differenceis(online)• Minimum The
time unit• instantinstant or not.
uploading –few photos
for searching is 24h.
• Appropriate for training CNN• 30 million/day
Twitter photos are more helpful to understandthe current state/trend of the worlds .
ⓒ 2016 UEC Tokyo.

Many text mining
works using• Text analysis ⇒ so many
– Event detection– Trend mining– Positive/Negative reputation
Typhoon trajectory estimatedby tweets [WWW 2010]• Photo analysis ⇒ limited before, but
recently increasing due to CNN– Evaluation of relatedness between msg. and img.
– Brand image mining, event photo detection– Fake News detectionⓒ 2016 UEC Tokyo.

Characteristic of the Twitter photos• Normal condition : everday life
– Food– Scene– People• Something special: event photos– Artificial public events sport games– Natural phenomena
earthquake, typhoon– Personal events• go hiking, travel, birthdayⓒ 2016 UEC Tokyo.

Mining two types ofphotos• Event photo : special• Food photo: everyday-lifeⓒ 2016 UEC Tokyo.

Twitter photos: special event

Special big event photos on March 11th 2011
around Tokyo areaⓒ 2016 UEC Tokyo.

Twitter photos: normal event

Everyday-life photos on March 11th 2011
in the western part of Japanⓒ 2016 UEC Tokyo.

World Seer: A Real-time
Geo-Tweet Photo Mapping
SystemICMR 2012

World Seer: Real-time Twitter
Photo Mapping System [ICMR 2012]

Build geo-photo tweet database for researchⓒ 2016 UEC Tokyo.

Monitoring the TW stream &Recording Geo-Photo Tweetsⓒ 2016 UEC Tokyo.

demo• http://mm.cs.uec.ac.jp/tw/ⓒ 2016 UEC Tokyo.

News: TW St API discon.
• Twitter Streaming API was discontinued
officially Aug. 2018.
But it was still available
until the end of July. 2019.
(When we wrote this review paper,
It was still available.)• Aug 1st 2019, a new connection to theTwitter Streaming API was not accepted
anymore.

Still OK : realtime tweet API• An alternative method is provided.
– Twitter API statuses/filter
Unfortunately in our system (in fact MY system)
API is not update. So currently it does not work.
I will update it soonafter completing reviewing five MMM papers.

Tweet photo database
2011/2～2019/7• Since Feb. 2011, we have collected– several billion photo tweets– 321million geo-photo tweets(5M geo-tweets/month before May 2015,
0.5M geo-tweets/month after May 2015)• We used this data for– Event Photo Mining– Food Photo Mining– Visual Topic Tendency Analysis– Training of GANs for food image translationⓒ 2016 UEC Tokyo.

Twitter Event Photo Mining
Yusuke Nakaji and Keiji Yanai: Visualization of Real World Eventswith Geotagged Tweet Photos, IEEE ICME Workshop on Social
Media Computing (SMC), (2012).
Takamu Kaneko and Keiji Yanai: Visual Event Mining from Geotweet Photos, IEEE ICME Workshop on Social Multimedia
Research (SMMR), (2013).
Takamu Kaneko and Keiji Yanai: Event Photo Mining from Twitter
Using Keyword Bursts and Image Clustering, Neurocomputing,
Elsevier, Vol.172, pp.143–158 (2016).

Demo• http://mm.cs.uec.ac.jp/kanekot/tw/jp/index.html• http://mm.cs.uec.ac.jp/kanekot/tw/us/index.htmlⓒ 2016 UEC Tokyo.

Twitter Event Photo
Mining• Mine the photos related to the events
happened in the specific areas and times

The results of detected
event photos in 2012
ⓒ 2016 UEC Tokyo.

Twitter Event Photo
Mining• ある日のある場所で，急に増えたキーワードを抽出し，関連画像を自動的に選択

２０１２年のアメリカでのイベント検出結果ⓒ 2016 UEC Tokyo.

Objective• Detect events from Twitter stream-Weather, natural events-Festivals, sport games• Understand events visually-Select representative photos-Place event photos on a map
Mapping events with the photoⓒ 2016 UEC Tokyo.

Processing flow
1. Event keyword detection
Textualanalysis
Visualfeatureanalysis

2. Keyword unification

3. Event photo clustering
4. Mapping event with photosⓒ 2016 UEC Tokyo.

Event Keyword Burst
Detection• Examine change of daily frequencyⓒ 2016 UEC Tokyo.

Event Photo Clustering• Hand-crafted image features (not CNN !)- Bag-of-Features with SURF
- Color histograms• Ward clustering method-a hierarchical clustering method-threshold is 300(both)
𝐸 𝐶 =

2𝑤
2𝑤
(
𝑥
−
𝑥
+
𝑥
−
𝑥
𝐵𝑜𝐹
𝐵𝑜𝐹
𝑅𝐺𝐵
𝑅𝐺𝐵
𝑅𝐺𝐵 )
𝐵𝑜𝐹
𝑥∈𝐶

ⓒ 2016 UEC Tokyo.

Experiments• Japan Dataset-Feb 10th, 2011 to Sep 30th, 2012
-about 3 million geo-tweet photos• US Dataset-Jan 1st, 2012 to Dec 31st, 2012
-about 17 million geo-tweet photosⓒ 2016 UEC Tokyo.

Results of Keyword
Detection
Japan

USA

KeywordDate

Keyword

Date

snow

11/02/2011

snow

09/01/2012

earthquake

11/03/2011

sunset13/01/2012

fireworks

06/08/2011

Grammy12/02/2012

typhoon21/09/2011

Valentines

14/02/2012

Mt. Fuji

24/09/2011

SXSW

09/03/2012

Apple

06/10/2011

Easter

08/04/2012

eclipse10/12/2011

shuttle17/04/2012

illumination

10/12/2011

WWDC

10/06/2012

Christmas

24/12/2011

hurricane

26/08/2012

New years eve

31/12/2011

rainbow

05/09/2012

sunrise01/01/2012

49ers

18/10/2012

firefly

06/05/2012ⓒ 2016 UEC Tokyo.

NYE

31/12/2012

“fireworks” photoclustersⓒ 2016 UEC Tokyo.

“cherry blossoms” photoclusters
daytimenightⓒ 2016 UEC Tokyo.

Mapping Results• Map event in a map-Calculate coordinates of event-Correspond information and the photo• Summary of results# events
accuracy

Japan

USA

258
65.5%

1676
72.5%

ⓒ 2016 UEC Tokyo.

Result“cherry“Fire“fireworks”
of Japan
Festival”
blossoms”
datasetⓒ 2016 UEC Tokyo.

“River’s
Result“BalloonEdge“rainbow”
of US
Music
Fiesta”
datasetFestival”ⓒ 2016 UEC Tokyo.

“sunset”ⓒ 2016 UEC Tokyo.

Visual Topic Analysis of
Twitter Photo Analysis
Unpublished.

2million photo clustering using
only visual features (no text)• http://mm.cs.uec.ac.jp/twimg/(BOF features)• http://mm.cs.uec.ac.jp/twimg/dcnn.cgi(CNN features)
Most of the tweet texts do not explain the attached images
directly. So text-based analysis might restrict target images
too much. ⇒ Twitter images with only visual analysisⓒ 2016 UEC Tokyo.

Food is one of the major
topics of Twitter photos• Visual topic analysis with half-million
Twitter photos employing DCNN feature.
Food-related topics

Food-related topicsⓒ 2016 UEC Tokyo.

Finding regional tendency
on Twitter photos
using only image features
Tetsuya Nagano, Takumi Ege, Wataru Shimoda and Keiji Yanai: A Largescale Analysis of Regional Tendency of Twitter Photos Using Only Image
Features, Proc. of IEEE International Conference on Multimedia
Information Processing and Retrieval (MIPR), (2019).

Regional tendency analysis
on Twitter geotagged photos• Apply visual clustering based photo topic
analysis on each of the regions over the world.

ⓒ 2017 UEC Tokyo.

Experiments• Dataset– Collected from January to June in 2016
– 2,161,000 geotagged Twitter images• Feature– CNN (128-d compressed by PCA)

• K-means– K-means with one-tenth images• Assigned rest of images into the nearest clusters– K=100ⓒ 2017 UEC Tokyo.

Regions• East Asia, North America, South America,
Europe, Africa, Middle East, South Asia andSouth-East Asia, Oceaniaⓒ 2017 UEC Tokyo.

Clustering results (CNN features)ⓒ 2017 UEC Tokyo.

Select five representative topics
from observation of clusters• pre-selected photo genres.
– “people”– “building”– “document”– “scene”– “food”ⓒ 2017 UEC Tokyo.

Ratio of five topics on each regionⓒ 2017 UEC Tokyo.

• East Asia
– No people photos– Many building and food photos– The total ratio of building and food photos weremore than 70%
PeopleBuilding
Document
LandscapeFoodⓒ 2017 UEC Tokyo.

• North America– The ratio of people and building were high morethan 60%.
PeopleBuilding
Document

LandscapeFoodⓒ 2017 UEC Tokyo.

• South America– People photos are the most popular genre (67%)
PeopleBuilding
Document

LandscapeFoodⓒ 2017 UEC Tokyo.

Analysis of Regional Tendency of
Photo Genres• Europe– The number of posted photos was the most large– The genres were well balanced.
PeopleDocument

Food• Africa– Almost no building, scene and food photos wereposted– People photos occupied 70%.
Building

Landscapeⓒ 2017 UEC Tokyo.

• Middle East
– Although the number of posts were fewer than
Europe, all the five genres were balanced as well
PeopleBuilding
Document

LandscapeFoodⓒ 2017 UEC Tokyo.

Analysis of Regional Tendency
regarding Photo Topics• South Asia

– More than half of the photos were document photos.
– This tendency was notobserved in other regions• SouthEast Asia
– People photos are the most and in addition foodphotos was the second mostPeopleDocument

Building

LandscapeFoodⓒ 2017 UEC Tokyo.

Sourth America：Peopleⓒ 2017 UEC Tokyo.

Africa：Peopleⓒ 2017 UEC Tokyo.

East Asia：Foodⓒ 2017 UEC Tokyo.

Sourth East Asia：Foodⓒ 2017 UEC Tokyo.

Findings• Tendency– East Asia and East-South Asia,• Food photos are relatively high– South America, South Asia and East-South Asia
• people photos are exceptionally high.

– Europe and MiddleEast• well balanced.

• East Asia enjoys posting food photos• South America, South Asia and EastSouth Asia liketo post people photos without caring privacy issue.
ⓒ 2017 UEC Tokyo.

Food Twitter Photo miningWhy food ?

Food is one of the major topics inTwitter photos.

Especially in East Asia
PeopleBuilding
Document
LandscapeFoodⓒ 2016 UEC Tokyo.

We are actively working on food images.

FoodCam : [Kawano et al. MTA13]• Real-time mobile food recognition
Android application

http://foodcam.mobi/ⓒ 2017 UEC Tokyo.

UEC-FOOD 100ⓒ 2016 UEC Tokyo.

UEC-FOOD 256ⓒ 2016 UEC Tokyo.

UEC-FOOD 256ⓒ 2016 UEC Tokyo.

FoodRec: foodrec app with UECFOOD100
by Hamlyn Centre-Imperial College(UK)ⓒ 2016 UEC Tokyo.

UEC-FOOD as a Fine-Grained
Image Classification Datasetⓒ 2016 UEC Tokyo.

UECFOOD-256 is usedin NVIDIA ICCV2019 paper• Ming-Yu Liu et al: Few-Shot Unsupervised
Image-to-Image Translation, ICCV2019.
x → style(y1,y2) → x’food image translation

X
X’ⓒ 2016 UEC Tokyo.

We are organizing food WS.
ACM MM Workshop related
to “food multimedia”

October 21thⓒ 2015 UEC Tokyo.

Twitter Real-time
Food Photo Mining
Keiji Yanai and Yoshiyuki Kawano: Twitter Food Image Mining andAnalysis for One Hundred Kinds of Foods , Pacifit-Rim Conference on
Multimedia (PCM), (2014).
Keiji Yanai, Kaimu Okamoto, Tetsuya Nagano and Daichi Horita:
Large-Scale Twitter Food Photo Mining and Its Applications,
IEEE International Conf. on Big Multimedia (BIGMM), (2019)

Twitter Real-time Food Photo
Mining System (mm.cs.uec.ac.jp/tw/)• What kinds of foods are being eaten in Japan ?

ⓒ 2016 UEC Tokyo.

Objective• Twitter Photo Mining for Food Photos– As a case study of Twitter Photo MiningVSon specific kinds of photos– Food is one of frequent topics of Twitter Photos.

– Real-time Photo Collection from the stream

Ramen

Curry

• Which food is the most popular in Japan?
– “Ramen vs Curry”problem ⇒ very controversial– I would like to put a period to this controversy
by Twitter food photo mining !!!
ⓒ 2016 UEC Tokyo.

Approach for food photomining• Two-step food photo selection– [1] Keyword-based tweet selection– [2] Image-based photo selection• 101-class (100 foods + non-food)classification (fine-tuned AlexNet)

100 Food/nonfoodclassification

Keyword-based
selectionⓒ 2016 UEC Tokyo.

Experiments• Collect photo tweets via Twitter Streaming API– From 2011/5 to 2019/07/16 (8year 2month)– About several billion photo tweets• Search for the tweets including any of 100-foodnames (in Japanese) and apply a food CNN– 16,044,090 images ⇐ Apply the 101-food CNN•

2,308,988 food photos(14.4%)
13,735,102 non-food photos (85.6%)ⓒ 2016 UEC Tokyo.

Twitter food photo ranking
rank

foods

#photos1

Ramen noodle

500,2102

Curry

209,3913

Sushi

130,501

4

Omelet103,199

5

Dipping noodle (tsukemen)

96,482

6
Pizza89,568Ramen noodle is the most popular food in Japan.
7 I have
Jiaozisolved(Gyoza)“ramen vs curry” problem !!!67,196
I got a “ramen vs curry” expert !!) 61,919
8 (AndOkonomiyakiⓒ 2016 UEC Tokyo.

Precision of the top 5 foods(May 2011-Aug. 2013)
Food
ramen

curry
sushi
tsukemenomelet(1)KW275,652
72.0%
224,68575.0%
86,50969.0%
33,165
88.7%
34,12590.0%(2) f/n200,17392.7%
163,047
95.0%
43,53686.0%
24,89696.3%
28,88796.3%ⓒ 2016 UEC Tokyo.

(3)spec. (4) ALL
84,18995.0%
62,82497.0%
48,01972.3%
28,846
93.7%
18,37098.0%

80,021
99.7%
59,26499.3%
25,89892.7%
22,15899.0%
17,52099.0%

Only keyword search(Ramen noodle) (72.0%)ⓒ 2016 UEC Tokyo.

After applying 100-class foodclassifier (final)(99.7%)ⓒ 2016 UEC Tokyo.

Only keyword search (curry)(75.0%)ⓒ 2016 UEC Tokyo.

Final results (curry)(99.3%)ⓒ 2016 UEC Tokyo.

Some interestingfindings• Letters or drawings are sometimes drawn onomelets with ketchup• Fast-foods such as humberger(rank 30th) andbeef bowl(rank 27th) are ranked lower,
since their appearance is always the same.
Not worth posting fastfood photos to Twitterⓒ 2016 UEC Tokyo.

Omlet wall paperⓒ 2016 UEC Tokyo.

Geographical-Temporal
analysis on ramen vs curry
12.6% of the obtained food photos have geotag.

Whole year

Dec. (winter)
Ramen is popular.

Ramen

Curry
ⓒ 2016 UEC Tokyo.

Aug. (summer)
Curry gets more popularthan ramen only in summer.

I appeared TV programin Japan (2018/11/1)ⓒ 2019 UEC Tokyo.

I commented on“ramen vs curry” problem
as a “ramen vs curry” expert.

ⓒ 2019 UEC Tokyo.

Ramen won !!!

ⓒ 2019 UEC Tokyo.

Curry lost !
But the rank is second.

ⓒ 2019 UEC Tokyo.

Regional Tendency
Analysis on TwitterFood PhotoK. Okamoto and K. Yanai, “Analyzing regional foodtrends with geo-tagged twitter food photos,” in Proc. ofInternational Conference on Content-Based Multimedia
Indexing (CBMI), 2019.

Method overview
The method is almost the same as the workon regional tendency analysis on genericimages. The difference is using a food/nonfood classifier at first.
[1] Classifying food and non-food photos[2] Extracting food CNN features[3] Clustering and analyzing of regionaltendencyⓒ 2016 UEC Tokyo.

Experiments• Selecting food images from Twitter images
in 2016 for whole a year with the
food/non-food classifier•

190,000 food images from 3.78
million raw Twitter imagesⓒ 2016 UEC Tokyo.

Resultsⓒ 2016 UEC Tokyo.

Tendency analysis‒ East Asia：“Noodles” and “Rice”, which are relatively
rare in the other regions, are included at the top.
‒ South-East Asia：“Soup” is ranked at the top, andmany dishes are made of vegetables and meats insoup.
‒ North America, South America, Europe：It turned
out that 4 items of the top 5 items of the categories
are the same
‒ Middle East：The top five food categories are thesame as Europe. However there are many browncoffee photos which are a unique type to Middle East.
ⓒ 2016 UEC Tokyo.

Applications ofLarge-Scal TwitterFood Photo DB[food image translation (food GAN)]
R. Tanno, D. Horita, W. Shimoda, and K. Yanai, “Magical rice bowl: Real-time
food category changer,” ACM Multimedia, demo, 2018.
D. Horita, R. Tanno, W. Shimoda, and K. Yanai, “Food category transfer withconditional cycle gan and a largescale food image dataset,” in Proc. ofInternational Workshop on Multimedia Assisted Dietary Management(MADIMA), 2018.

[Food VR]
S. Naritmo, R. Tanno, T. Ege, and A. K. Yanai, “Cnnbased food transformation
on hololens,” in Proc. of International Workshop on Interface and Experience
Design with AI for VR/AR (DAIVAR), 2018.

Food Image
Translation(FoodGAN)Real-time Food Translation“MagicalRiceBowl”ⓒ 2019 UEC Tokyo.

Original StarGAN
Cycle Consistency Loss

𝐺𝐶
Real Image
In domain A

C[1, 0, 0, 0, 0, ...]
Domain Selector

𝐺𝐶
Fake Image
In domain B

C[0, 1, 0, 0, 0, …] Reconstructed
Domain Selector

Image

Adversarial LossD

ⓒ 2019 UEC Tokyo.

fake or real

Auxiliary Classifier
LossExperiments• We use foods, which have similar dish plates
as target food category for simplification.
Selected 10 kinds of food category.

カレー
Curry

炒飯
Fried
rice

牛丼
Beef
bowl

Ramen
ラーメン

Rice
白飯

Buckwheat
蕎麦 noodle

2019/9/12

Chillednoodle
冷やし中華

UEC yanailab 2017

Eel bowl
うなぎ

Meat spaghetti
ミートスパ

Fried noodle
焼きそば

13

Experimanetal Data
• Total amount:• 230k images• Traing：0.9• Testing：0.1

Twitter Stream から収集

Target category

Image number

Chilled noodle

13,499Meat spaghetti

7,138

Buckwheat noodle

3,530

Ramen

74,007

Fried noodle

24,760Rice

21,324Curry rice

34,216

Beef bowl

18,396Eel bowl

5,329

Fried rice

27,854

total

230,05313
2019/9/12

UEC yanailab 2017

The number of training images affects quality
1000 10000 ALL

1000 10000 ALL

13
2019/9/12

UEC yanailab 2017

Experimental results• In case of one food included in an image
Input

Curry

Fried rice

Beef bowl

ChillednoodleMeatspaghetti

13
2019/9/12

UEC yanailab 2017

Experimental results• In case of one food included in an image
Input

Ramen

Rice

Buckwheat
noodleEel bowl

Fried noodle

14
2019/9/12

UEC yanailab 2017

Experimental results• In case of multiple foods included in an image
Input

Curry

Fried rice

Beef bowl Chilled noodle

Meat spaghet

14
2019/9/12

UEC yanailab 2017

Experimental results• In case of multiple foods included in an image
input

ramen

rice

Buckwheat
noodleEel bowl

Fried noodle

14
2019/9/12

UEC yanailab 2017

[extention] Attentional StarGAN(network of “MagicalRiceBowl”)
Reconstructed
Image

Cycle Consistency Loss

𝐺𝐶
Real Image
In domain A

× ○
+
○

𝐺𝐶

C[1, 0, 0, 0, 0, ...]
Domain Selector

Fake Image
In domain B

C[0, 1, 0, 0, 0, …]
Domain Selector

×
○

×
○
(1.0×
○

)

(1.0-

+
○

D

Real Image
In domain B)

Adversarial Lossfake or real

Auxiliary ClassifierLossⓒ 2019 UEC Tokyo.

Attentional StarGAN• w/o attention• w/ attention

Background is almost
unchanged.

Background is alsotranslated.
ⓒ 2019 UEC Tokyo.

MagicalRiceBowl with the modelwith attention

2019/9/12

UEC Yanai Lab. Tokyo.

145

MagicalRiceBowl
on iOS App Store
Only working
on iPhone 8 or more
(8/X/Xs/XR/11/11pro)
Using Neural Engine.

ⓒ 2016 UEC Tokyo.

Food VR

Integration of GAN-based
image translation and VRⓒ 2019 UEC Tokyo.

https://www.youtube.com/watch?v=
BJaQ5IF6iEI

2019/9/12

UEC Yanai Lab. Tokyo.

149食事変換＋HoloLens(VR Restaurant)

UEC Yanai Lab. Tokyo.

newszero 2019/05/24ⓒ 2016 UEC Tokyo.

Conclusions

Conclusions• Introduced our Twitter photo mining works
since Feb. 2011 (one month before the big earthquake)– Geotagged tweet photo analysis• Real-time geo-tweet photo mapping system [2012]• Event photo mining from geo-tweet photos [2012-2016]• Finding regional tendency on Twitter photos [2019]– Twitter food photo mining• Statistics on food image collection for 8 years [2012-]• Regional tendency on Twitter food photos [2019]• Applications of a large-scale Twitter food photoDB [2018-]– Food image translation by GAN, mobile app., food VRⓒ 2016 UEC Tokyo.

“Thank you” by fake ramen images

2019/9/12

UEC Yanai Lab. Tokyo.

180

ⓒ 2019 UEC Tokyo.