A New Large-scale Food Image Segmentation Dataset
and Its Application to Food Calorie Estimation
Based on Grains of Rice
Takumi Ege Wataru Shimoda Keiji Yanai The University of Electro-Communications, Tokyo
Some meal recording app can estimate food calories.
But they …
• Need user’s manual input of food categories and volumes.
• Estimate food calories for each dish one by one.
• Are paid service to hire nutritionists who estimate food calories.

Purpose : Image-based food calorie estimation(5) Calorie estimation considering considering
food area size

Food calorie (kcal)Background

Use quadratic curve based estimation[1]Beef bowl

2𝑐𝑎𝑙𝑜𝑟𝑖𝑒 = 𝑎𝑖 ∗ 𝑠𝑖𝑧𝑒𝑓𝑜𝑜𝑑 + 𝑏𝑖 ∗ 𝑠𝑖𝑧𝑒𝑓𝑜𝑜𝑑 + 𝑐𝑖

Where 𝑎𝑖 , 𝑏𝑖 , 𝑐𝑖 are trained params for each i-th categories.
Food area (𝑐𝑚2 )Experiments
1. Real scale estimation from rice grain image

New Dataset : UECFoodPix• Annotated new instance-based bounding boxes to 10,000 images
included in UECFood100 [2] manually.
• Annotated segmentation masks to 9000 images automaticallyby GrabCut, and 1000 images manually for the evaluation.
※ We will publish the dataset as soon as all images are corrected by hand.
UECFood100[2] UECFoodPix(Ours) UECFood100[2] UECFoodPix(Ours)

Benchmarks
YOLOv2
DeeplabV3+

mAP : 60.4 %mIoU : 41.9 %DeeplabV3+

mIoU : 43.6 %(trained with 9,000 grabcut masks)(trained with 7,000 grabcut masks
and 2,000 manually masks)

Effects of hand correction

Absolute err. (cm/224pixel)
Relative err. (%)
Correlation

0.145
5.548
0.946

R=0.946

The correlation coefficient
between the estimation and ground-truth is 0.946
2. Real area estimation from rice grain image
Examples
Input

Bbox

Method : Food calorie estimation considering food area
The procedure of food calorie estimation(1) Take a food photo with rice.
(2) CNN-based Food detection and foodsegmentation for each dish .
(3) Estimate real scale from rice grain images.
(4) Calculate real size of food areafrom both estimated real scale andsegmentation masks.
(5) Estimate food calories based on
estimated food area size andcategory-dependent calorie density.

(1) Take a picture with rice(2) Food detection andsegmentation(3) Real scaleestimation(4) Calculate real size offood area
(5) Food calorie estimationconsidering food area

Seg. mask

Real scale
Rice image

Real area size

Absolute err. (𝑐𝑚2 )Relative err. (%)
Correlation

GrabCutU-Net

14.2
15.0
0.469

13.3
13.9
0.474

3. Food calorie estimation considering food area
ExamplesMiso soup 𝟏𝟐𝟕𝒄𝒎𝟐
Salad 𝟏𝟎𝟖𝒄𝒎𝟐 𝟖 𝒌𝒄𝒂𝒍

Rice 𝟖𝟑𝒄𝒎𝟐 𝟒𝟏 𝒌𝒄𝒂𝒍

Sauteed spinach 𝟕𝟓𝒄𝒎

𝟐

Bibimbap 𝟔𝟏𝒄𝒎

Cold tofu 𝟔𝟓𝒄𝒎𝟐
𝟐Goya chanpuru 𝟏𝟕𝟔𝒄𝒎𝟐
Sirloin cutlet 𝟏𝟏𝟒𝒄𝒎𝟐 𝟓𝟓𝟐 𝒌𝒄𝒂𝒍

𝟐

Rice 𝟏𝟕𝟏𝒄𝒎 𝟐𝟑𝟐 𝒌𝒄𝒂𝒍

(2) Food detection and segmentation

Ginger pork
saute 𝟐𝟕𝒄𝒎𝟐
Other foods 𝟐𝟗𝒄𝒎𝟐

Fermentedsoybeans 𝟏𝟗𝒄𝒎𝟐

Rice 𝟏𝟏𝟔𝒄𝒎𝟐 𝟏𝟏𝟕 𝒌𝒄𝒂𝒍
Cold tofu 𝟐𝟔𝒄𝒎𝟐
Other foods 𝟐𝟕𝒄𝒎𝟐

𝐎𝐭𝐡𝐞𝐫 𝐟𝐨𝐨𝐝𝐬 𝟐𝟑𝒄𝒎𝟐

ここに数式を入力します。

YOLOv2 [3] trained with UECFOODPix
U-Net [4] trained with UECFOODPix’s binary masks(mIoU : 84.1%)
UECFOODPix enables highly accurate
food detction and segmentation.

Conclusion
Real scale estimation

We proposedSegmentatoin, 3D

In the future work, considering food volumes
even when there is no rice, we are considering(3) Real scale estimation from rice grain images
Rice image dataset

Annotated with real scale and segmentation masks.
A total of 360 rice images were taken in our laboratory.

Real scale estimation

Takes a rice grain image as an input and outputs thereal size of the length of one side of the input image.

CNN enables real scale estimation
robust to the size and orientationof rice grains

・ New Large-scale Food Image Segmentation Dataset
・ Food calorie estimation considering food area.

・ Combining the method which employed
segmentation and a reference object
・ Using the depth information obtained from thecamera of iPhone[1] K. Okamoto and K. Yanai. An Automatic Calorie Estimation System of Food Images on a Smartphone, MADiMa, 2016 .
[2] Y. Matsuda, H. Hajime, and K. Yanai. Recognition of multiple-food images by detecting candidate regions. ICME , 2012 .
[3] J. Redmon, A. Farhadi , YOLO9000: Better, Faster, Stronger, CVPR 2017.
[4] O. Ronneberger, P. Fischer, and T. Brox. U-net: Convolutional networks for biomedical image segmentation. MICCAI, pp.234–241,2015 .