Real-Time Mobile Recipe
Recommendation System Using
Food Ingredient Recognition
ACM Multimedia 2012 Workshop
Interactive Multimedia on Mobile and Portable Devices(IMMPD'12)
Takuma Maruyama, Yoshiyuki Kawano, Keiji Yanai(The Univ. of Electro-Communications, Tokyo)1[Demo] Recipe search by object
recognition at a grocery store
Point a camera to carrots
Object recognition

Recipe lists
Select one from the list
Get to know how to cook

mobilerecipe search

object recognition

Background
OpenCV
Smart phone

Easy to implement
an object recognition
system on smartphones
3

Proposed System
mobileRecipe searchObject Recognition

Easy recipe search
while shopping
Recognition performed
on a smartphone

The first system for recipe
Impossible to use both hands search with object recognition
for operation of mobile devices
on smartphones
while shopping at a grocery stores
4

Related work －mobile image recognition－
 Google Goggles• Landmark• book• text• logs• Sudoku• Specific object recognition• Similar image search• OCR

Object recognition for generic object is impossible
Implement useful app with generic object recognition
6

The flow of the proposed system
The system recognizesphoto stream repeatedly
and continuously

8Screen of the System

9Obtain recipe list from WebAPI
Name of ingredient

Punch Fork /CoodPad

Smart Phone
Recipe list (XML)

Punch fork
User-generated recipe site

Services in USA

Currently, the order of the recipeon the sreen is the same as
the results from WebAPI.

Need to be improved

WebAPI is provided
11

Method for Object Recognition
Image Features
Bag-of-Local Color Histogram, Bag-of-SURF(with KD-tree, 1000 dim codebook)Multi-frame recognition
Classification method
Linear SVM for saving memory and speed-up

Can recognize a food ingredient in 0.17 second
12Time for recognition
Galaxy S2• June 3, 2011
• 1GHz Dual Core
• 1GB RAMHTC Desire HD• Nov. 12, 2010
• 1GHz• 756MB RAMPhone

Avg. (msec.)

GAXALYS2167.8

HTC
DesireHD394.0

単位はmsec13

Feature(1): Local Color Histogram

・・・

Divide color space
Into 64 bins (4x4x4)

12 x 12 grids

144 local color histogram
14

Feture(2): SURF
4dim * 4x4 grids = 64 dims.

 dx,  dy, | dx |, | dy |Tested in two ways of keypoints• Default way (Hessian-based)• 12x12, 24x24,48x48,96x96 multi-scale grid15

Bag-of-Features
Input image

Local features

BoF vectors

Vote each local feature to corresponding visual word
Codebook size: N=1000 (for both local color hist. and SURF)
Use KD-tree to search for visual words: O(logN)

16

Multi-frame
Aggregate local features extracted
from multi frames and build a BoF○ Single Framefeatures○ Multi Frame

BoF vector

in experiments n=1, 2, 3, 4, 5featuresfeatures

BoF vectorfeatures

17

Linear SVM for speed-up andsaving memory
Linear SVM classiferInner productNy ( x)   w x x  bi 1

Ti iO(DN )N

xi：support vector x  wi xi  b

wi：support vec. weighti 1

N：# support vectors

Can be computed off-line

 x wbT

y： output
x： inputO(D)D：dimension of Bof (1000)

Training: offline (on PC), Classifying: on-line (on smartphone)Linear SVM for speed-up
Estimation of required memory

Codebook (1000-dim)
1000  64  4byte  256Kbyte

Weight W (for 30 class)
Fast & small memory

1000  30  4byte  30Kbyte

Limitation: Android app -> 16MByte
19

Experiments
Dataset30 kinds of food ingredient: 10 5-sec videos for each
Evaluation

Recorded at 10 grocery storesin TokyoClassification rate (features, food, top k)
Evaluate by 10-cross validation

User study
食材3種類を対象に目的のレシピを検索、時間システムの評価、ユーザのコメント

20

30 kinds of food ingredientmeatchicken

beef

pork

ham

octopusshrimpsausageminced meat

fish

tuna

squid

vegetable

potatomushroom

radishtomatoShitake

fruit

apple

cucumberonion
bananastrawberrycarroteggplant

lettuce

Chinese cabbage

Green onion
pineapple

salmoncabbageorange

grapefruit

21

Examples

Recorded at 10 grocery stores

Grapefruit

Recognized correctly

recipeSalmonTouch 3rd candidates

recipe22

Difficulty in the Dataset

Packed or wrapped
Light reflected
Looks similar

23

Classification rate

Classification rate for features
0.5
0.45
0.4
0.35
0.3
0.25
0.2
0.15
0.1
0.05
042.5%

44.9%

24

Classification rate for each
Orange 81.1%

For 30 kinds

shrimp 5.14%

25

成功した食材、失敗した食材[best] orange： Orange color is specific to oranges.

[worst] shrimp： color is too various. Freezed, boiled, raw

26Confusing food ingredientApple: 2types (green and red)Tomato, grapefruit

27

Classification rate within top kWithin top6: 84.1%

1st6thTop k

28

User study
Method3 ingredient x 3recipes = 9 patterns
Measure times when using obj. rec. and manually•••

Recognition accuracy (5 step)
Usablity (5 step)
Which is better, obj. rec. or manually ? (5 step)

coments# Subjects

5 students
29

User evaluation (1)Usability

Recognition Accuracy

31

User evaluation (2)Which is better to use,by object recognition or manually？
CommentsWhen items was recognizedcorrectly, obj. rec.-based
was better.
If accuracy is improved,
it must be more practical.

32

Comments from subjects
comment
It is convenient, because easy to search for recipe at a grocery.
If acuracy is improved, I’d like to use it regularly.
Accuracy seems to depend on kinds of ingredient greatly.
Taking account of the prices would be more helpful.

33

Conclusions
Propose system By only pointing food ingredient, we can search forthe cooking recipe at a grocery store.
 It takes 0.17[sec] for one-time recognition of 30
kinds of food ingredient.

User study

The users pointed out improvement
on recognition accuracy is needed.

Evaluation

Classification rate: 44.9% (1st)84.1% (～6th)34

Future work
Toward being more practical
SystemSearch recipe DB by combination ofseveral ingredients (AND-search)
Taking account of budget and price for low-cost cooking
Taking account of ingredient left in the fridge

Recognition

Improve rate and increase kinds

Find better features and make DB bigger
35

You can try it !
The Android app can be downloaded !
(We do not provide iOS version. )Download site

http://mm.cs.uec.ac.jp/mobile_recipe/

36