Image-Based Estimation of Real
Food Size for Accurate FoodCalorie Estimation
Takumi Ege, Yoshikazu Ando, Ryosuke Tanno, Wataru
Shimoda and Keiji Yanai Department of Informatics, The
University of Electro-Communications, Tokyoⓒ 2017 UEC Tokyo.

Introduction• Food calorie estimation• Existing works– Associated with the estimated food categories– Require users to enter information such as size orvolume• Our works– Fully-automatic food calorie estimationⓒ 2017 UEC Tokyo.

Review of our works• (1) “CalorieCam”– reference-object-based food calorie estimation system.

• (2) Weakly-supervised segmentation based food calorieestimation• (3) “AR DeepCalorieCam V2”– A real food size and calorie estimation system based on iOS ARKit.

• (4) “DepthCalorieCam” (new system)– food calorie estimation system exploiting iPhone stereo cameras.

• (5) Rice grain based size estimation (new method)– uses rice grained the size of which are usually almost the same asa reference object.
ⓒ 2017 UEC Tokyo.

CalorieCam• Image-based calorie estimation system• User needs to register a size-known referenceobject– Wallet– Creditcard-size card

ⓒ 2017 UEC Tokyo.

CalorieCam• Segmentation– Food items– Pre-registered reference object.

• Real size of each detected food items– Comparing the number of pixels• Process– Estimate rough position based on edge detection– Apply color-pixel-based k-means clustering– Apply GrabCut with the detected bounding boxⓒ 2017 UEC Tokyo.

Weakly-supervised Segmentation
Based Calorie Estimation• Calorie Cam– One meal photo contains only one dish• This work
– Estimate calories from segmentation results.
– Without multiple-view photos– Without specific reference objects such as wallets
and cards.

ⓒ 2017 UEC Tokyo.

Weakly-supervised Segmentation
Based Calorie Estimationⓒ 2017 UEC Tokyo.

Weakly-supervised Segmentation
Based Calorie Estimation• Choose base food region• We decide priorities based on a tendency ofunchanging food volumes.
• Some food volumes change frequently, whilesome foods volume rarely change.
• In “Teishoku”– Japanese traditional food combo menu
– we can often change the volume of “rice”– we cannot change “miso-soup” volume in general.
ⓒ 2017 UEC Tokyo.

ⓒ 2017 UEC Tokyo.

AR DeepCalorieCam• Uses an inertial sensor built in a standard
smartphone in addition to photos• Recognize a category of each food item.
• Directly calculate the size using AR• Calculate food calories based on theirⓒ 2017 UEC Tokyo.

AR DeepCalorieCamⓒ 2017 UEC Tokyo.

AR
DeepCalorieCam
with iOS ARKit
ARKit estimates
real size of thebounding box.
ⓒ 2019 UEC Tokyo.

DepthCalorieCamⓒ 2017 UEC Tokyo.

Croquette calorie
estimation demo:

1 croquette=160kcal2 croquette=320kcalDepthCalorieCam• Food regions are extracted by U-NET trainedwith UECFood-100 and segmentation masksets.
• Estimate the depth of each of the pixels andtheir actual volumes.
• Calculate food calories from the actual volume
and the regression curvesⓒ 2017 UEC Tokyo.

DepthCalorieCam• CNN-based food region segmentation– Network• U-Net– Training• 5,301 masks• UECFOOD-100ⓒ 2017 UEC Tokyo.

Comparative Experiments with
CalorieCam and AR DeepCalorieCam V2• Targetⓒ 2017 UEC Tokyo.

Comparative Experiments with
CalorieCam and AR DeepCalorieCam V2ⓒ 2017 UEC Tokyo.

Rice grain based size estimation• Proposed network• Examples of datasetⓒ 2017 UEC Tokyo.

Dataset• 360 imagesⓒ 2017 UEC Tokyo.

Experiments• Training• Evaluationⓒ 2017 UEC Tokyo.

Experimentsⓒ 2017 UEC Tokyo.

Conclusion• DepthCalorieCam is the most promising
approach.
• Large-scale calorieannotated 3D food volumedata is needed to extend the system– very costly and time-consuming.

• The rice grain based method is also promising– Appropriate for Japanese foodsⓒ 2017 UEC Tokyo.