DepthCalorieCam: A Mobile Application forVolume-Based Food Calorie Estimation using Depth Cameras
Yoshikazu Ando, Takumi Ege, Jaehyeong Cho, Keiji Yanai
The University of Electro-Communications, Tokyo

Proposed method

Objective
The increased demand for food calorie estimation
using smartphones
But they…
• Ask users to provide the size of dishes• No care about the volumes of foods

Food calorie estimation just by
taking a picture !!

• RGB-D image

Take a picture

taken by
“iPhone DualFood segmentation
Volume estimationCameras”

Categoryestimation

Scores[0.7, …, 0.2]

Calorie estimation• Xception[*] trained withUECFOOD100• Top-1 Acc. = 77.8%
1. Camera〜Distance of

Estimated food calories
based on food volumes

RGB-D image taken by“iPhone Dual Cameras”

reference plane

U-Net[*] trained with UECFOOD100’s seg. mask
• mIoU = 80.0%
Existing methods
2. Area of reference planeFried chicken

CalorieCam[*]AR DeepCalorieCam V2[*]→GrabCut→Draw lines of he actual foodboundaries in the AR space

3. Volume

DepthCalorieCam※Food calorie estimation is
implemented only followingdevices

Demo app is published↓

Reduce time and effort
when taking a picture•iPhone 7/8 Plus
•iPhone X/XS/XS Max

Food calorie 𝑦𝑐𝑎𝑙 regression fromthe volume 𝑥𝑣𝑜𝑙

Experiments・Evaluation↓Food calorie 𝑦𝑐𝑎𝑙

•iPhone 11 Pro/Pro MAX

Comparative evaluation of food calorie estimation
CalorieCam(Baseline1) / AR DeepCalorieCam V2(Baseline2) / Proposal method

Highly accurate regression

Estimated volume 𝑥𝑣𝑜𝑙

※The mean and standard deviation of the errors in the calorie estimations

Conclusion・Future work
• We estimated food volumes and food calories based on
the RGB-D image taken by “iPhone Dual Cameras”.
1000400• In the calorie estimation, our system achieved higher
accuracy than existing methods

208000200600
400-20-4002000• Increase the number of food categories the system canhandle with. Currently supports 3 food categories.

-60-80-200-200-100-400-400

Baseline1

Baseline2

ProposedmethodSweet and sour pork-120

Baseline1

Baseline2

Proposed

Baseline1

Baseline2

methodFried chicken

ProposedmethodCroquette

Easier and more accurate estimation
than Existing methods→ Effect of regression from volume• Take RGB-D images of real foods the calorie amountof which are known by using iPhones having twobackside cameras.
[1] Y. Matsuda, H. Hoashi, and K. Yanai. Recognition of multiple-food images by detecting candidate regions. In Proc. of IEEE
International Conference on Multimedia and Expo (ICME), 2012.
[2] O Ronneberger, P Fischer, and T Brox. U-Net: Convolutional Networks for Biomedical Image Segmentation. In Proc. of Medical Image
Computing and Computer-Assisted Intervention (MICCAI), pp. 234–241, 2015.
[3] K. Okamoto and K. Yanai. An Automatic Calorie Estimation System of Food Images on a Smartphone. In Proc. Of ACM MM Workshop
on Multimedia Assisted Dietary Management (MADiMa), 2016.
[4] R. Tanno, T. Ege, and K. Yanai. AR DeepCalorieCam V2: Food Calorie Estimation with CNN and AR-based Actual Size Estimation. In
Proc. of ACM Symposium on Virtual Reality Software and Technology (VRST), 2018.
[5] F. Chollet. Xception: Deep Learning with Depthwise Separable Convolutions. In Proc. of IEEE International Conference on Computer
Vision and Pattern Recognition (CVPR), pp. 1800–1807, 2017.