An Automatic Calorie Estimation Systemof Food Images on a Smartphone
Koichi Okamoto
Keiji Yanai
The University of Electro-Communications,Tokyo, Japan

How many calories ?

Proposed System: CalorieCam

CalorieCam (proposed system (Android app))

Pork cutlet, rice, miso soup,…

Introduction
New system: CalorieCam

Many recording meal apps

⇒ Semi-automatic calorie estimation• Automatic calorie estimation• Standalone system- need to teach food amounts manually- no need of a recognition server• Android app.

System Overview

Extract a regionof a ref. object

pre-registeredsize: 46.8𝑐𝑚2
pixels : 43149[foo.log inc.]Recognize a category ofeach detected foodDetect a bounding box ofa dish region based on edges②

Detect a bounding box ofa food region by k-means

pixels : 68066③
Detect an accurate foodregion by GrubCut
The same way fora reference object

Experiments

ExamplesSpring rolls 428kcal

Sweet pork 292kcal

Successful cases

Use for training of calorie estimation parameters (𝑎𝑖 , 𝑏𝑖 , 𝑐𝑖 )
 Prepare all the dishes and take photos in our lab.
Yakitori 165kcal

Boiled chikuzen 85kcal

Mean avg err.

Mean SD

Mean relative err.

Mean relative SD52.2 kcal

±40.4 kcal

21.3%

±0.823

Input
imageregions

GTPork cutlet

586
kcal

Beef bowl

kcal

559
kcal

1417
kcal

27 0.05kcal

95 0.07

kcal

Dish

Ask five subjects to estimate food calories
with FoodCam and CalorieCam

Beef bowl

Usability (5steps)

Croquette

FoodCam

2.83 ± 0.80

CalorieCam

4.25 ± 0.72

Conclusions
Implement an image-based food calorie estimation app.
 average error: ±52kcal, relative error: ±20%
1322EstimRel.
Error
atederr

Comparison with FoodCam

Gyoza 246kcal

Size-based estimation- Compare #pixels of a food region with #pixels
of a reference object and estimate a real size
𝑠𝑖𝑧𝑒𝑓𝑜𝑜𝑑 = #𝑝𝑖𝑥𝑓𝑜𝑜𝑑 ∗ #𝑝𝑖𝑥𝑟𝑒𝑓 /𝑠𝑖𝑧𝑒𝑟𝑒𝑓- Use quadratic curve based estimationfrom 𝑠𝑖𝑧𝑒𝑓𝑜𝑜𝑑 to a food calorie, 𝑐𝑎𝑙 .
2𝑐𝑎𝑙 = 𝑎𝑖 ∗ 𝑠𝑖𝑧𝑒𝑓𝑜𝑜𝑑+ 𝑏𝑖 ∗ 𝑠𝑖𝑧𝑒𝑓𝑜𝑜𝑑 + 𝑐𝑖
Beefwhere 𝑎𝑖 , 𝑏𝑖 , 𝑐𝑖 are trained
Kinpirabowl
params for 𝑖-th categories.

Evaluation of automatic calorie estimation

Datasets: calorie-annotated food photos(20 kinds of dishes with 3 different sizes)

Takoyaki 241kcal③ Size-based calorie estimation
Food category
Final output :and Calorie value Kinpira burdock 74.4 kcal

French fries 454kcal

CNN-based mobile food recognition engines- “DeepFoodCam” engine- Standalone, no need of a recognition server- Network-in-network (NIN) based CNN- “Real” real-time recognition (27ms in iPhone7)(Please see “DeepFoodCam” demo for detail !)size vs calorie

Chill shrimp 268kcal

Provide a meal photoExtract regionsof food itemsCalculate food calories based on theirrelative size and food categories② Food classification

GrubCut-based region extraction
with edge-based seeds①Compareboth region sizes

Need to teach the amountof foods by selecting items① Extraction of regions

Take a photo witha reference object①’FoodLog

Need to teach the amountof foods by the slider

Failure cases

FoodCam [Kawano et al. 2014]

SaladInput
imageGT962
552
14

regions

GT

Estim
Rel.
Error
atederr

Niku
jaga

170

122
kcal

kcal

Yakisoba

425

519

94

kcal
kcal

kcal

48kcal0.22

FoodCam
Proposed app
Avg. err
Avg.SD Avg. err Avg.SD-53.25 ±209.79-242±55.10-242 ±91.26-47.08±52.5254.83 ±36.28
4.86±11.87

Future works
CNN-based segmentation and calorie estimation
 Use calorie-annotated food images on the Web

0.28