Koichi Okamoto, Keiji Yanai
The Univ. of Electro-Communications
Tokyo, Japan• Food recording habit is helpful for diet

• However to record

foods, people
have to－take photos－input food names－estimate calories

Timeconsuming !

Give up recording soon !!

 Food recording app by image recognition◦ It requires taking meal photos before eating
inappropriate for sharing large platter,hot pot, or BBQ A novel food recording system

 Is applicable even for food sharing situation
 Recognize eating action during a meal
 Record all eaten foods, calories and amounts

Eating meat !

smartphone

Monitoring eating action

Mouth detection

Chopsticks detection

If they come close
Eating region detection

Food item
classification

Update total calories

Detect a face
2. Detect a mouth from the face region1.

(We used face and mouse detectors in OpenCV.)

Without face detection

With face detection

Detect moving areas
by background subtraction
2. Detect lines from the moving areas
by Hough transform
1.

The entire image

Only moving area If mouth and chopstick regions comeclose to each other“eating”

Eating region
containing a food item Fusion oftwo Image features

1. Bag-of-features with ORB
2. HSV color histogram Classifier : Linear SVM→fast χ2 feature map (Vedaldi al et. PAMI12) Target meal: ‘Yakiniku Grill’◦ Japanese-style BBQ: grill thin-sliced meat & vege.

Sharing a grill

outdoorYakiniku alone Target food items for classifiction:

5 kinds of typical items in Yakiniku
Meat,Rice, Pumpkin, Bell pepper, Carrot

1.

2.

Evaluation offood classiﬁcation accuracy
User study on system usability

#Training images : 450 #Test images : 50
Classification accuracy : 74.8%

 Comparison of two systems:◦ Baseline system:
manual recording system
by touching food item buttons◦ Proposed systemeating action recognition
 5-step evaluation◦ 5 (better) .... 3 (soso) .... 1 (bad) Two questions for five subjects

Screen of thebaseline system Two questions for five subjecs:

1.How easy to take eating record ?
Baseline system: 2.0
Proposed system: 4.8

2.How easy to see calorie intake during meal ?
Baseline system: 3.0
Proposed system: 3.4

Better than the manual baseline system A novel food recording systemRecognize eating action during a meal
Record all eaten foods, calories andamounts
Is appropriate for food sharing
situation such as “Yakiniku”

 Accuracy: 74.8%

 User study: Effective,but need to improve UI• Add other types of meals:• Improve classification accuracy:→ Fisher Vector• Estimate the food volume

Yakiniku alone with!

 Mapo doufu◦ Tofu, Minced meat…
 Shrimps in chili sauce◦ Shrimps, Onion…
 sweet‐and‐sour pork

◦ Pork, Carrot, Pineapple…
 Eat some food items in one bite→Pre-defined fixed calories is average calories Mouth detection accuracy is high
 Chopsticks detection accuracy is not high

◦ There are lines in background◦ Detected Chopsticks are longer than the actual

False Eating region Fork and knife are straight◦ Can be same method as chopsticks detection
 If eat sandwiches or onigiri◦ We have to detect handLeft : real foodRight : food sample Mixture of real foods and foodsamples for training image set◦ food samples are not good for training
image set

Food sample

Real Food

Koichi Okamoto and Keiji Yanai
The University of Electro-Communications, Tokyo• Recording food habit is helpfulfor dietary• However,－take photos－input food names－estimate calorie－etc…

Timeconsuming !

Quit recording soon !!

 Food recording app by image recognition◦ It requires taking meal photos before eating
inappropriate for sharing large platter,hot pot, and BBQ

Very new type of mobile food recording system Recognize eating action during meal

 Automatic recording all the eaten food

food item category, calorie, amounts…

Eating meat !

smartphone

Monitoring eating action

Capture frameMouth detection

Chopstick detection
segmentation
Classification offood item

Record eating actionAndroid App :(In the current implemention, the target is a “Yakiniku” meal.) Two kinds of evaluations

Classification accuracy (5-fold CV)→ 74.4 %2. Simple user study (5 subjects)→Comparison with the baseline system
which has no recognition in 5-steps
1.

usability

baseline
2.0

GrillCam
4.8

Much better than the no-recognition baseline

You can experience a new type of meal
with!