Real-time Mobile Food
Recognition System
Yoshiyuki Kawano and Keiji Yanai
Department of Informatics,
The University of Electro-Communications,Tokyo, JAPANⓒ 2013 UEC Tokyo.

Background• Recording of food habits has become popular.
– by using PC or mobile phone– quite troublesome task.

• Rapid progress of smartphone– obtain enough computational power for real-time
image recognitionⓒ 2013 UEC Tokyo.

Objective
Mobile Food Recognition System• recognize food in real-time• require no connection with serve– Recognition on a smartphone

Recognition Result• interactive system– bounding box is given by user

Selected Food Volumeⓒ 2013 UEC Tokyo.

Flow of proposed system
Draw bounding box
Point a smartphone
toward food items

Real-time recognition

Check balance
Nutrition information
Register food recordⓒ 2013 UEC Tokyo.

Select food items

Demonstrationⓒ 2013 UEC Tokyo.

Related Work (1)• [Matsuda et al. ICME’12]– 100 kinds of food categories– carry out on server side
Result1.
Rice
2. miso soup
3. Egg sunny-side up

4. Sirloin cutlet
5. Salmom meuniere

6. Fried fish
7. Boiled fish
8. Sausage
9. Sandwiches
10.Roll bread

Food Imageⓒ 2013 UEC Tokyo.Related Work (2)• Google Goggles– Specific object recognition– Similar image search– OCR• Leaf snap(Kumar et al,ECCV’12)– Identifying plant species
All the systems adopt server-side recognitionⓒ 2013 UEC Tokyo.

3 key features of the system• Bounding box adjustment• Real-time image recognition• Estimation of the more reliable directionⓒ 2013 UEC Tokyo.

Bounding Box Adjustment• Restriction : A user draws bounding boxes
covering food items by touching• Performed only once after bounding box was drawn

GrabCutⓒ 2013 UEC Tokyo.

Food Recognition• Image Features:– Color Histogram• 64×(3×3) = 576dim– SURF-BoF• code book size : 500• soft assignmentⓒ 2013 UEC Tokyo.

Food Recognition• Classifier:– Linear SVM
𝑀

𝑓 𝑥 =

𝑦𝑖 𝛼𝑖 𝐾 𝘅, 𝘅𝑖 + 𝑏

Independent of thenumber of samples

𝑦𝑖 𝛼𝑖 〈𝘅, 𝘅𝑖 ⟩ + 𝑏

computation： 𝑂 𝑁

𝑖=1
𝑀

=𝑖=1
𝑀

=〈memory ： 𝑂 𝑁
𝑦𝑖 𝛼𝑖 𝘅𝑖 , 𝘅⟩ + 𝑏

𝑖=1= 𝘄, 𝘅 + 𝑏ⓒ 2013 UEC Tokyo.

Food Recognition• fast 𝜒 2 feature map(Vedaldi et al, PAMI’12)– approximate the implicit feature mapping– the same accuracy as non linear SVM– speed comparable with linear SVM

0.8∅ 𝑥 = 𝑥 0.6cos(0.6 log 𝑥)
0.6sin(0.6 log 𝑥)ⓒ 2013 UEC Tokyo.

Estimation of the more reliable direction• Effective window search• No transposed SURF-BoF + linear SVM

result– Linear SVM
𝘄 = 𝘄+ + 𝘄 −– SVM score• Create integral images
about 𝘄+ 𝑎𝑛𝑑 𝘄−• 𝑂 1 operation ( refer to ESS (Lampert et al, CVPR ‘08))• Show the result– The window with the maximum scoreⓒ 2013 UEC Tokyo.

Demonstration of estimation themore reliable directionⓒ 2013 UEC Tokyo.Experiment• Experiment description– Recognition accuracy– Processing time
– User studyⓒ 2013 UEC Tokyo.

Set up• Recognition accuracy– Dataset• includes 50 kinds of food categories• has more than 100 images for each category• processing time
– device• Galaxy Note2 (1.6 GHz,4 core, 4 thread)– multi thread for Quad coreⓒ 2013 UEC Tokyo.

Classification Rate

Evaluation of Recognition Accuracy
1
0.9
0.8
0.7
0.6
0.5
0.4
0.3
0.2
0.1
0

top5
81.6%

top1
53.5%

Color(fast chi2)
SURF(fast chi2)
Color+SURF(fast chi2)
Color+SURF(linear)
Color+SURF(chi2-RBF,MKL)
12

34

567

Top n candidatesⓒ 2013 UEC Tokyo.

8910

Classification Rate

Evaluation of Bounding box adjustment
10.9
0.8
0.7
0.6
0.5
0.4
0.3
0.2
0.1
0

top5-4.4%

top5-10.1%

0%25%
adjustment
25%, BBGrabCut
1234567

Top n candidatesⓒ 2013 UEC Tokyo.

8910

Experiment of Processing Timefood region(bounding box)
Only once

Extract SURF

Adjust bounding box

Classify SURF
Extract Color+ classify
0.091sec

Estimate direction

Core2Core3

Recognition +Estimate
direction

0.26 sec

Get evaluation valueCore 10.70sec

Core4ⓒ 2013 UEC Tokyo.

0.34sec

User Study• Subject– Five students• Evaluation– 3 food items in a meal, 2trial, 3 or4 meal
goodbad– Outcome Measure (5step)135• Recognition Accuracy• Usability• Estimation direction• Which is better, proposed or manual system?
ⓒ 2013 UEC Tokyo.

User Evaluation• Result(average value)

Recognition Accuracy
Usability
Estimation direction
Which is better proposed or manualⓒ 2013 UEC Tokyo.

3.44.22.4
3.8

Comments from subjects• comment– If accuracy is improved, I want to try it– exclude food items of incorrect result from list
– increase the number of food categories• summary– unable to recognize some food items– Proposed system better than manual systemⓒ 2013 UEC Tokyo.

Conclusions• recognize 50 kinds food categories• run on common smartphone• adjust bounding box in the background• estimate of the more reliable directionⓒ 2013 UEC Tokyo.

Future Work• draw bounding box– Make less complex such as touch just a point• use additional information– User’s meal history– GPS and time informationⓒ 2013 UEC Tokyo.

You can try it• http://foodcam.mobi/
Requirements : Android 4.0 and up: Quad core CPU(No iPhone version)ⓒ 2013 UEC Tokyo.

ⓒ 2013 UEC Tokyo.

Meal Records

on the smartphoneⓒ 2013 UEC Tokyo.

on the Web

4-Gunⓒ 2013 UEC Tokyo.

Data Set
• 50 kinds food itemsⓒ 2013 UEC Tokyo.

Demonstrationⓒ 2013 UEC Tokyo.

Demonstration of estimatonⓒ 2013 UEC Tokyo.

Demonstrationⓒ 2013 UEC Tokyo.

Demonstrationⓒ 2013 UEC Tokyo.

Classification Rate

0.9
0.8
0.7
0.6
0.50.40.3
0.2
0.1
0

top5
top1

Feature Typeⓒ 2013 UEC Tokyo.

Set up for Experiment• Dataset (test)– Shifting the ground-truth bounding boxes – x%
– To each of eight directions• Evaluation– Error in the direction estimation– Cumulative classification rate
𝑛𝑢𝑚𝑏𝑒𝑟 𝑜𝑓 𝑙𝑒𝑠𝑠 𝑡ℎ𝑎𝑛 𝑦⁰
𝑛𝑢𝑚𝑏𝑒𝑟 𝑜𝑓 𝑎𝑙𝑙 𝑡ℎ𝑒 𝑡𝑒𝑠𝑡 𝑖𝑚𝑎𝑔𝑒𝑠ⓒ 2013 UEC Tokyo.

Classification Rate

Experiment of estimation direction
1
0.9
0.8
0.7
0.6
0.5
0.4
0.3
0.2
0.1
054.2%

Shifting Rate

50.3%

10%
15%
20%
25%

±20° ±40° ±60° ±80° ±100°±120°±140°±160°±180°

Allowable angular difference errorⓒ 2013 UEC Tokyo.

～2.0
2.0～3.0
3.0～4.0
4.0～5.0
5.0～6.0
6.0～7.0
7.0～8.0
8.0～9.0
9.0～10.0
10.0～11.0
11.0～12.0
12.0～13.0
13.0～14.0
14.0～15.0
15～

Count

25

20
proposed system

15
Manual system
hand

median1050

Spend Time[sec]ⓒ 2013 UEC Tokyo.

Set up for Experiment• Dataset– Includes 50 kinds of food categories– Has more 100 images for each category• Performance:– Classification Rate =𝑛𝑢𝑚𝑏𝑒𝑟 𝑜𝑓 𝑐𝑜𝑟𝑟𝑒𝑐𝑡𝑙𝑦 𝑑𝑒𝑡𝑒𝑐𝑡𝑒𝑑 𝑖𝑚𝑎𝑔𝑒𝑠 𝑖𝑛 𝑇𝑜𝑝 𝑛𝑛𝑢𝑚𝑏𝑒𝑟 𝑜𝑓 𝑎𝑙𝑙 𝑡ℎ𝑒 𝑡𝑒𝑠𝑡 𝑖𝑚𝑎𝑔𝑒𝑠ⓒ 2013 UEC Tokyo.

Objective• To record food items in every meal
– quite troublesome task.
More Quickly and Easier• Proposed SystemRecognition Result– Recognize food in real-time– require no connection with server

Selected Foodⓒ 2013 UEC Tokyo.

VolumeFlow of recognition and record
Start recognition
Point a smartphone
toward food items

Draw bounding box

Register food record

Select food itemsⓒ 2013 UEC Tokyo.

Bounding Box Adjustment• Why use Bounding Box?
– recognizes repeatedly in real-time– a smartphone is not fixed.

GrabCutⓒ 2013 UEC Tokyo.

Set up for Experiment• Evaluation of processing time
– device• Galaxy Note2(1.6GHz,4core,4thread, Android 4.1)

– multi thread for Quad core

• Evaluation– processing time
– averageⓒ 2013 UEC Tokyo.

Estimation of the more reliable direction• Effective window search• No transposed SURF-BoF + linear SVM– Linear SVM

result𝘄 = 𝘄+ + 𝘄−

– SVM score• Create integral images
about 𝘄+ 𝑎𝑛𝑑 𝘄−• 𝑂 1 operation ( ref ESS )• Show the result– The window with the maximum scoreⓒ 2013 UEC Tokyo.

Future Work• Estimate direction– Consider other information such as shape– Move the bounding box automatically• Draw bounding box– Make less complex such as touch just a point• Use additional information– User’s meal history– GPS and time informationⓒ 2013 UEC Tokyo.

You can try it• The Android application can bedownloaded.
– We do not provide iOS version.

• Download site
– http://foodcam.mobi/– Android 3.1 and up– Quad cores and upⓒ 2013 UEC Tokyo.

Objective• To record food items in every meal
– quite troublesome task.
Recognition ResultMore Quickly and Easier• Proposed system– recognize food in real-time– require no connection with server Selected Food Volume– return results with 81.6% in the top5.
ⓒ 2013 UEC Tokyo.

ⓒ 2013 UEC Tokyo.