Recognition of Multiple-Food Images
by Detecting Candidate Regions
Yuji Matsuda, Hajime Hoashi, and Keiji Yanai
The University of Electro-Communications, TokyoJAPANⓒ 2012 UEC Tokyo.

Background• Recording of food habits has become
popular.
– Users can become aware of own diet, andevaluate nutrition.

• To record food items in every meal isa quite troublesome task.
• It is desired to make recording of fooditems more easier and quickly.
ⓒ 2012 UEC Tokyo.

Objective• Recognition of “multiple food images”– Contain two or more food items
1. rice
2. miso soup
3. egg sunny-side up
4. sirloin cutlet
5. salmon meuniere

recognition

6. fried fish
7. boiled fish
8. sausage
9. sandwiches

food image

10. roll breadⓒ 2012 UEC Tokyo.

Related Work• [S. Yang et al. CVPR 2010]– specialized for American fast food– defined 8 basic food materials– classify 61 food categories using
detected materials and their relative position• [Z. Zong et al. ISM 2010]– the same fast food dataset– SIFT detector + LBPⓒ 2012 UEC Tokyo.

Related Work• Our previous work [Hoashi et al. ISM 2010]– 85 kinds of food categories– fusing various kinds of image features using MKL

Existing methods

Proposed method• Recognize only
single food-itemin one image.

• Recognize
multiple fooditems in one imageat the same time.
ⓒ 2012 UEC Tokyo.

Recognition Flow
Input image

1. rice

Candidate Region Detection

Whole DPM Circle JSEG
Coding Image Feature Vector
Color SIFT CSIFT HoG GaborClassification
Output resultsⓒ 2012 UEC Tokyo.

2. miso soup

3. egg sunny-side up
4. sirloin cutlet
5. salmon meuniere
6. fried fish
7. boiled fish
8. sausage
9. sandwiches
10. roll breadCandidate Region Detection• Whole image• Sliding window search• Circle detector• Region segmentationⓒ 2012 UEC Tokyo.

Whole Image• Advantage: suitable for larger dish
• Disadvantage: unsuitable for small dishⓒ 2012 UEC Tokyo.

Sliding Window Search(Deformable Part Model)• Advantage: can obtain region with a highevaluation value• Disadvantage: is based on only gradientbased features# region is 100 in total

P.F. Felzenszwalb, R.B. Girshick, D. McAllester, and D. Ramanan,
Object detection with discriminatively trained part-based models, PAMI 2010
ⓒ 2012 UEC Tokyo.

Circle Detector• Advantage: can detect food by circular plate• Disadvantage: dishes are not always circular# region is 4 on averageⓒ 2012 UEC Tokyo.

Region Segmentation (JSEG)• Advantage: detect dishes by segmentation• Disadvantage: does not always success# region is 10Y. Deng and B. S. Manjunath: Unsupervised segmentation of color texture
regions in images and video, PAMI 2001
ⓒ 2012 UEC Tokyo.

Region Segmentation (cont.)• One food item is sometimes divided into
several regions.
– combine regions based on circularity# combined region is 4
on averageⓒ 2012 UEC Tokyo.

Irrelevant Region Removal• Irrelevant region:– less than 60 pixels (shorter side)– apart from the average aspect ratioⓒ 2012 UEC Tokyo.

Classification• Image Features:– Color histogram– SIFT, CSIFT (Spatial-pyramid bag-of-features)• Frequency of the pattern of local patch– HOG• Rough shape of the object– Gabor• texture patternsⓒ 2012 UEC Tokyo.

Classification• Classifier:– Multiple Kernel Learning (MKL-SVM)– 1-vs-rest• Results:– A list of top N food categoriesⓒ 2012 UEC Tokyo.

Experiments• Dataset:– includes 100 kinds of food categories– has about 100 images for each category• For test:– 500 multiple food-item images (contain 1200 items)ⓒ 2012 UEC Tokyo.

100 food category databaseⓒ 2012 UEC Tokyo.

Experiments• Performance:• Comparison:
1. Proposed Method
2. Only single detector (includes previous method)
3. Ground truth bounding-box regionⓒ 2012 UEC Tokyo.

Multiple Food-Item Images
ProposedWholeCircle

JSEG

DPM

GTBB

Classification Rate

0.7
0.6

0.5
0.4

0.3proposed method
55.8%

0.2

whole image
15.4%

0.1
6E-16-0.1

1234567# of candidatesⓒ 2012 UEC Tokyo.

8910

Conclusions• Proposed two-step food recognition method
1. candidate region detection
2. classification for candidate region• Achieved 55.8% classification rate (top 10)– improved by 40.4 points• Future Work:– introduce co-occurrence probability– estimate caloriesⓒ 2012 UEC Tokyo.

Thank you !
Questions?

ⓒ 2012 UEC Tokyo.

DPM only vs. DPM+MKL(multiple food-item images)
Classification Rate

0.7

DPM MKLDPM only

0.60.5

DPM+MKL
52.8%

0.40.30.2

DPM only
41.5%

0.1
01234567# of candidateⓒ 2012 UEC Tokyo.

8910

Processing Time
• Total processing time is about 2 minutes.
– Candidate Region Detection : about 15 sec– Feature encoding : about 90 sec– Classification : about 20 secⓒ 2012 UEC Tokyo.

Easy categories
grilled salmon (100%) sausage (100%)rice (92.3%)

egg sunny-side up(91.7%)

miso soup (89.2%) hamburger (86.7%)

toast (85%)

sandwiches (81.8%)ⓒ 2012 UEC Tokyo.

Difficult categories
macaroni salad(0%)

pork miso soup
(6.3%)

Chinese soup (6.7%)

French fries (10%)

omelet (10%)

Hamburg steak(11.1%)

jiaozi (11.1%)

udon noodle(11.1%)ⓒ 2012 UEC Tokyo.