A New Large-scale Food Image Segmentation Dataset
and Its Application to Food Calorie Estimation
Based on Grains of Rice
Takumi Ege

Wataru Shimoda

Keiji Yanai

The University of Electro-Communications, Tokyo{ege-t,shimoda-k,yanai}@mm.inf.uec.ac.jp

ABSTRACT

1

To estimate food calorie accurately from food images, accurate foodimage segmentation is needed. So far no large-scale food image
segmentation datasets which have pixel-wise labels exists. In thispaper, we added segmentation masks to the food images in theexisting dataset, UEC-Food100, semi-automatically. To estimate
segmentation masks, we revised bounding boxes included in the
UEC-Food100 dataset so that bounding boxes bounds not dish regions but only food regions. As results, by applying GrubCut, we
obtained good segmentation masks, and we checked and corrected1000 images of them if needed by hand for the testing masks.
We trained segmentation networks with the newly-created foodimage masks. As results, segmentation accuracy was much improved, which is expected to bring more accurate food calorie
estimation. In addition, we propose a new method on food calorie estimation using grains of steamed rice which are typicallycontained in Japanese foods instead of a reference card. By theexperiments, we show real food size can be estimated from riceimages, which helps accurate food calorie estimation.

The food calorie is considered to be strongly dependent on the foodcategories and volumes, and it is useful in terms of food management if it becomes possible to estimate food categories and volumes
automatically from food images. In food classification from food images, the method using CNN has already achieved highly accurateclassification. Recently, there are some applications that estimate
food categories automatically from food images.
However, food volume estimation that is essential for food calorie
estimation still remains as a difficult problem. In most of the cases,
the estimated calories are just associated with the estimated foodcategories, or the relative size compared to the standard size of eachfood category which is usually indicated by a user manually.
Therefore, in this paper, we propose estimating food calorie froma food photo considering an area of food regions. The area of foodregion is calculated from a real scale and a food region.
The real scale of a pixel on an image is estimated with CNN
from rice grains of food images containing rice. Because of thesize of rice grains is almost constant, it works as a reference object
for estimating the real scale. In this paper, we construct a CNN
that takes patch images of rice grains as inputs and outputs real
size of the patch images. The food region is estimated by foodsegmentation with CNN. Note that in this method, it is assumed
that the food photo is taken vertically from right above of the tablesurface.
In the previous works using a reference object such as a card,a user always needs to take food photos with the pre-registeredreference object. In contrast to that, rice is usually included in a
Japanese meal, so that the method of using rice grains as a reference
object is not required to prepare that. So it is enabled to apply tothe food photos taken in the past uploaded on Web.
To summarize our contributions, we propose the method to estimate food calories from a food photo considering the area of foodregions. The area of food regions is calculated from an estimatedreal scale and food region with CNN. Firstly, for estimating the real
scale of the pixel, we construct a CNN which can estimate real sizeof the patch images of rice grains. Secondly, the food regions are
obtained by food segmentation based on CNN. Finally, the foodcalories considering the food area are estimated.

CCS CONCEPTS• Computing methodologies → Scene understanding; Object recognition.

KEYWORDS
food image segmentation, food calorie estimation, food image recognition
ACM Reference Format:
Takumi Ege Wataru Shimoda Keiji Yanai. 2019. A New Large-scale Food
Image Segmentation Dataset and Its Application to Food Calorie Estimation Based on Grains of Rice. In 5th International Workshop on Multimedia
Assisted Dietary Management (MADiMa ’19), October 21, 2019, Nice, France.
ACM, New York, NY, USA, 6 pages. https://doi.org/10.1145/3347448.3357162

Permission to make digital or hard copies of all or part of this work for personal orclassroom use is granted without fee provided that copies are not made or distributedfor profit or commercial advantage and that copies bear this notice and the full citationon the first page. Copyrights for components of this work owned by others than theauthor(s) must be honored. Abstracting with credit is permitted. To copy otherwise, or
republish, to post on servers or to redistribute to lists, requires prior specific permission
and/or a fee. Request permissions from permissions@acm.org.
MADiMa ’19, October 21, 2019, Nice, France© 2019 Copyright held by the owner/author(s). Publication rights licensed to ACM.
ACM ISBN 978-1-4503-6916-9/19/10. . . $15.00
https://doi.org/10.1145/3347448.3357162

2

INTRODUCTION

RELATED WORK

Various approaches has been proposed so far and the main approachis to estimate calories based on estimated food categories and its
size or volume using the value of food calorie per unit area orvolume.

MADiMa ’19, October 21, 2019, Nice, France
Chen et al. [3] proposed an image-based food calorie estimation method that estimates food categories and volumes by depth
cameras such as Kinect. Depth cameras such as Kinect are special
devices, so it is thought that ordinary people are difficult to use
usually.
Kong et al. [7] proposed a mobile application to estimate foodcalories from images multiple images,“DietCam". They carried out
segmentation and food item recognition, and in addition reconstructed 3D volumes of food items and calculate food calories based
estimated volumes. 3D reconstruction was performed with SIFTbased keypoint matching and homography estimation which werea standard method of 3D stereo vision.
Dehais et al. [5] proposed the other method for food dish segmentation. In the work of Dehais et al. [5], firstly, the Border Map,
which represents rough boundary lines of a food region is obtainedby CNN. Then the boundary lines of Border Map are refined by
the region growing/merging algorithm. The system which requires
some photos taken from multiple viewpoints needs to calibrate
devices, which limits the situation where the system can be used.
Im2Calories by Myers et al. [9] estimates food categories, ingredients, and the regions of each of the dishes included in a given
food photo and finally outputs food calories by calculation basedon the estimated volumes and the calorie density corresponding
to the estimated food category. In their experiment, they faced
the problem that the calorie-annotated dataset is insufficient andevaluation is not sufficiently performed.
Pouladzadhe et al. [11] proposed a food calorie estimation system
which needed two dish images taken from the top and the side andused a thumb of a user as a reference object. Their method toestimate volumes were calculated by multiplying the size of fooditems estimated from the top-view image by the height estimatedfrom the side-view image, which was relatively a straight-forward
way. In our method, the real size estimation is performed using ricegrains with the fixed size as reference objects.
Okamoto et al. [10] proposed an image-based calorie estimation
system which estimates food calories automatically by simply taking a meal photo from the top with a pre-registered reference object.
Firstly, both regions of food and reference object are estimated frommeal photo by GrabCut [15], Secondly, the real area of food is calculated by comparing regions of food and that of the reference object,
Finally, the calorie of the food is estimated based on the real areaof the food. Contrastly, our method treats rice grains as reference
objects and estimate real size from rice grains with CNN.

3DATASET: UECFOODPIX

A lot of food image datasets have been published so far. In foodcategorization tasks, some of them such as Food-101 [1], UECFood100 [8] and UECFood-256 [6] are used for standard benchmarksof food image recognition tasks. Among them, UECFood-100/256
are the only datasets which have bounding boxes of food areasof all the images in the dataset. However, the bounding boxes in
UECFood-100/256 are originally annotated for food classification ofmultiple dishes, so that the food category is limited to 100 kinds offoods included in UECFood-100/256 categories, and the other foods
than 100 categories are ignored. For example, if an image contains
bread toast and butter, a bounding box is annotated with only the

Takumi Ege

Wataru Shimoda

Keiji Yanai

area of the bread toast and no bounding box is provided to the areaof the butter, since butter is not contained in the categories of the
UECFood dataset. In addition, bounding boxes in the dataset do notconsider instances. If an image contains many pieces of sushi, only
one large bounding box is usually given to a set of sushi.
Currently, there are no large-scale food image dataset with segmentation masks. Only the UNIMIB2016 dataset provides food region information as polygons [4] which are equivalent to segmentation masks. However, its scale is not so large (1027 multiple-dish images with 73 food categories), and the food images in UNIMIB2016
are biased and not generic since all the food images were taken at
the same canteen.

3.1

Construction

Therefore, we have decided to construct a food image dataset annotated with instance-based bounding boxes and segmentation masks
by expanding the conventional UECFood-100. Firstly, we annotated
new instance-based bounding boxes to 10,000 images included in
UECFood-100 manually 1 Secondly, we annotated segmentation
masks to 9000 images automatically by GrabCut [15], and 1000
images manually for the evaluation. As shown in the Figure 1, thenew bounding boxes are applied to all of the food categories andinstances so that they enclose foods rather than dishes.
The number of bounding boxes in the new dataset becomes morethan twice that of UECFood-100. To save annotation cost, we annotated only bounding boxes on each of the food instances and did
not assign food categories manually. Instead, the food category ofeach new bounding box is annotated automatically based on theoverlap ratio ao between a new bounding box Buec f oodseд andone Buec f ood100 contained in the original UECFood-100 dataset
by Eq.(1). The new bounding boxes have the food category of conventional one with overlap ratio a 0 which exceed 0.5, and thathas no overlapped conventional one are assigned to “Other foods”
category.
overlap =area(Buec f oodseд ∩ Buec f ood100 )area(min{Buec f oodseд , Buec f ood100 })(1)

Since it costs too much to create all segmentation masks byhand, we annotate new segmentation masks automatically by GrabCut [15] using new bounding boxes. The segmentation mask foreach bounding box was generated one by one by using GrabCut.
Figure 2 shows some segmentation masks generated automaticallyfrom the newly annotated instance-based bounding boxes.

3.2

Benchmarks

For the benchmark of our dataset, we perform food detection andfood segmentation with our new dataset, UECFoodPix, of 10,000
food images. In this experiments, 9000 and 1000 images are used fortraining and evaluation, respectively. Note that the bounding boxes
and segmentation masks of the evaluation dataset are annotatedmanually.
For food detection, we use YOLOv2 proposed by Redmon etal. [13] to detect dishes. YOLOv2 improves YOLO [12] based on
CNN previously proposed, it enables high-speed and high-accuracy
1 In fact, UECFood-100 has 12,740 food images. The annotation work is still on the way.
We will release it after all annotation is done.

A New Large-scale Food Image Segmentation Dataset
and Its Application to Food Calorie Estimation
Based on Grains of Rice

MADiMa ’19, October 21, 2019, Nice, France

Figure 2: Segmentation masks with GrabCut [15], left isbased on bounding boxes of UECFood-100 [8], right is basedon that of UECFoodPix (Ours).

For food segmentation, we use DeeplabV3+ proposed by Chen etal. [2] as a baseline method. DeeplabV3+ has a deep Xception model
as a backbone network and a decoder network for refinement ofoutline of segmentation outputs. The model is known as a current
state-of-the-art semantic segmentation architecture. We obtained
41.9 % by Deeplabv3+ on mean Intersection over Union (IoU) for
the 101 class food segmentation, which includes “Other foods” anda background class.

4

METHOD

Our proposed method performs food detection, food segmentation
and real scale estimation for rice grain images. Finally, estimates
food calories considering food area, according to the following
processing.

Figure 1: Bounding boxes, left is UECFood-100 [8], right is
UECFoodPix (Ours).

object detection. As a result, mean Average Prediction (mAP) of
food 101 classes is 60.4 %.

(1) Take a food photo with rice.
(2) Food detection.
(3) Food segmentation for each of the estimated bounding boxes.
(4) Estimate real size from rice images.
(5) Calculate real size of food area from both estimated real scaleand segmentation masks.
(6) Estimate food calories based on estimated food area andcategory-dependent calorie density.
In the proposed method, it is assumed that the food photo is takenfrom directly above the dish vertically to the table surface. The
detail of each procesing step is described below.

MADiMa ’19, October 21, 2019, Nice, France

Takumi Ege

Wataru Shimoda

Keiji Yanai

the real scale estimation is a regression problem, we use the meansquare error as a loss function.

4.3

Calorie estimation considering food area

Our method estimates food calories based on the real size of foodregions calculated from the real scale and the food regions. The real
size of food regions Fr can be obtained by the following equation:
F r = S r ∗ Fp

Figure 3: The results of food detection and segmentation.

where Fp represents the number of pixels of the region of the target
food, and Sr represents the real area size of region of a pixel.
After calculating the real size of the regions of target food, we
estimate its food calorie value following the method proposed by
Okamoto et al. [10]. They convert the 2D top-view size of the foodto the calorie value according to the category-dependent quadratic
curves estimated based on the training data annotated with real
food calories. In this work, we use the quadratic curve of each foodcategory trained by Okamoto et al. [10] as they are.

5
4.1

Food detection and segmentation

In food detection, we use YOLOv2 [13] trained with our new bounding boxes of 9000 food images of our dataset.
In food segmentation, we use U-Net proposed by Ronneberger
et al. [14] For training of U-Net, we use a binary mask which represents food foreground or background ignoring the food categories,
since the detection network, YOLOv2, can estimate food categories
with bounding boxes and a segmentation network does not needto estimate food categories. As a result of the evaluation with 1000
food images annotated segmentation masks manually, mean IoU is
84.1 %.
The results of food detection and estimated segmentation masks
from each of the estimated bounding boxes are shown in Figure 3.

4.2

Real scale estimation from rice grain
imagesOur proposed method estimates real scale from rice grains images
with CNN to estimate food area. As shown in Figure 4, our modelof real scale estimation is based on VGG16 [17], which takes a ricegrain image as an input and outputs the real size of the length of
one side of the input image. The input image is a patch image ofrice images that length of one side is 224×224 in this work. Since

Figure 4: Real scale estimation network.

(2)

EXPERIMENTS

We perform two experiments of the real scale estimation from ricegrains images and the food calorie estimation base on food area. In
the real scale estimation from rice grain image, we use real scale
annotated rice image dataset. In the food calorie estimation basedon food area, 9000 and 1000 images of our UECFoodPix dataset areused for training and testing, respectively. Currently, since there
are no dataset annotated food calories considering food volume,the evaluation of the estimated food calories are not performed.

5.1

Real scale estimation from rice grain image

In this experiment, we estimate the real scale from rice grain images.
To do that, we construct a rice image dataset consisting of 360
images annotated with real size.
5.1.1 Rice image dataset. Because these rice images need to beannotated real scale, they were taken in our laboratory and the real
scale was given based on the diameter of the bowl of rice.
Rice of Japonica commercially available was used for the dataset
and we prepared three patterns of rice cooked with different amountsof water such as 180 ml, 200 ml and 220 ml for 150 g of rice. Then
we took 60 photos for each pattern of rice with two kinds of camera respectively such as COOLPIX AW120 and iPhone8 Plus. For
collecting various rice images with various real scales, the typesof dishes and the distance between the rice and the camera werechanged when photos were taken. Finally, a total of 360 rice images
with the real scale were collected.
After taking rice photos, the real scale per pixel was given basedon the diameter of the bowl of rice for each image. Furthermore, inorder to remove background information, rice segmentation masks
were created manually.
5.1.2 Real scale estimation. In this experiment, we estimate thereal scale from rice grain image using our rice image dataset. We
divide rice image dataset into six sets based on the camera type andthe amount of water for steaming rice. We use five sets for training
and one remaining set for evaluation. Thus, the training images are
300 images and test images are 60 images.

A New Large-scale Food Image Segmentation Dataset
and Its Application to Food Calorie Estimation
Based on Grains of RiceIn training of a network, 224×224 patch images are cropped fromrandom positions of the rice images. In addition, the augmentationsuch as flipping, resizing and rotating of rice images are performed.
In the case of resizing a rice image with the ratio of n, the real scaleof the rice image is multiplied by 1/n.
In the evaluation, input patch images are cropped by 5 × 5 grid
sampling from a rice image. The final output is an average value of
outputs of the network for 25 input patch images. Furthermore, inboth of the training and evaluation, input patch images which has
a background area of 1 % or more are removed based on annotatedrice segmentation masks.
Our network architecture of real scale estimation is based on
VGG16 [17] and the initial weights of the network are the pretrained model of ImageNet classification tasks except for the output
layer. For optimization of the CNN, we used SGD with the momentum value 0.9 and the size of mini-batch was 16. We used 10−5 ofthe learning rate for 3,000 iterations.
We show the average of the relative error representing the ratio
between the estimated values and the ground-truth, and the absolute error representing the differences between both. In addition,
we show the correlation coefficient between the estimated value
and ground-truth and the ratio of the estimated value within therelative error of 5 %, 10 %, and 20 %. Note that in the evaluation,
the estimated real scale for 224 pixels is used.
Table 1 shows the result of real scale estimation. Figure 5 shows
the relation between the estimated real scale values and groundtruth values. The average of the relative error and the absolute
error for the estimated real scale of 224 pixels were 0.145 cm and5.548%, respectively. In addition, the average correlation coefficient
between the estimated value and ground-truth was 0.946 whichshowed a very high correlation.
In all combinations of training data and evaluation data, the relative error was less than 10%, the correlation coefficient was higher
than 0.9, and most of the estimated values were included within therelative error of 20 %. From these results, we have confirmed thatthe effectiveness of the proposed model to estimates the real scaledirectly from the rice grain images without taking into account thevariations in size and orientation of rice grains.

5.2

Figure 5: The correlation between the estimated real scalevalues and ground-truth values in the real scale estimation.

Food calorie estimation considering foodarea

In this experiment, the food calories considering food area is estimated. Firstly, the food bounding boxes are obtained by the fooddetection with YOLOv2 [13], then the food regions are estimatedby food segmentation with U-Net [14] from each bounding boxes.
Secondly, the real scale is estimated from obtained rice images. Finally, the food area is calculated by the combination of estimatedreal scales and food regions, and food calories are estimated basedon the food area with a method proposed by Okamoto et al. [10].
Figure 6 shows an example of the result of estimated food areas
and food calories.

6

MADiMa ’19, October 21, 2019, Nice, France

CONCLUSIONSIn this work, we estimate food calories considering food area from afood photo. To do that, the food area is calculated by a combinationof the estimated and food regions and real scales.

Figure 6: Example of the food calorie estimation considering the food area for a multiple-dish food photo. The tags
on each red frame shows the estimated food areas and foodcalories.

In the real scale estimation, the average of the relative error
and the absolute error for the estimated real scale of 224 pixels
were 0.145 cm and 5.548 %, respectively. In addition, the averagecorrelation coefficient between the estimated value and groundtruth was 0.946 that shows a very high correlation.
Furthermore, for food detection and food segmentation, we construct newly food image dataset with bounding boxes and segmentation masks by expanding the conventional UECFood-100 [8].

MADiMa ’19, October 21, 2019, Nice, France

Takumi Ege

Wataru Shimoda

Keiji Yanai

Table 1: The real scale estimation from rice grains images(the estimated values of 224 pixels is used for the evaluation).
evaluation data
COOLPIX, 180ml/150g
COOLPIX, 200ml/150g
COOLPIX, 220ml/150g
iPhone8 Plus, 180ml/150g
iPhone8 Plus, 200ml/150g
iPnone8 Plus, 220ml/150g
Averageabs.err.(cm)
0.152
0.169
0.194
0.123
0.145
0.086
0.145

rel.err.(%)
4.822
5.513
5.906
5.706
7.305
4.037
5.548

correlation
0.963
0.959
0.920
0.949
0.910
0.976
0.946

As future work, we plan to evaluate the food calorie estimationconsidering food areas. For the evaluation, we prepare multiple-dishfood photos annotated food calories considering food volumes.
In addition, in order to construct a system that can estimate thefood calories considering food volumes even when there is no rice,
we are considering combining the method [10, 16] which employed
segmentation and a reference object, or the depth information
obtained from the camera of iPhone.
Note that we plan to release the new pixel-wise annotated foodimage segmentation dataset, “UECFoodPix”, at http://foodcam.mobi/
dataset/.

ACKNOWLEDGMENTS
This work was supported by JSPS KAKENHI Grant Number 17J10261,
15H05915, 17H01745, 17H06100 and 19H04929.

REFERENCES[1] L. Bossard, M. Guillaumin, and L. Van Gool. 2014. Food-101 – Mining Discriminative Components with Random Forests. In Proc. of European Conference on
Computer Vision.
[2] L-C. Chen, Y. Zhu, G. Papandreou, F. Schroff, and H. Adam. 2018. Encoderdecoder with atrous separable convolution for semantic image segmentation. In
Proc. of European Conference on Computer Vision.
[3] M. Chen, Y. Yang, C. Ho, S. Wang, E. Liu, E. Chang, C. Yeh, and M. Ouhyoung.
2012. Automatic Chinese Food Identification and Quantity Estimation. In Proc.
of SIGGRAPH Asia Technical Briefs. 1–4.
[4] G. Ciocca, P. Napoletano, and R. Schettini. 2017. Food recognition: a new dataset,experiments and results. IEEE Journal of Biomedical and Health Informatics 21, 3(2017), 588–598.
[5] J. Dehais, M. Anthimopoulos, and S. Mougiakakou. 2016. Food Image Segmentation for Dietary Assessment. In Proc. of ACM MM Workshop on Multimedia
Assisted Dietary Management.
[6] Y. Kawano and K. Yanai. 2014. Automatic Expansion of a Food Image Dataset
Leveraging Existing Categories with Domain Adaptation. In Proc. of ECCV Workshop on Transferring and Adapting Source Knowledge in Computer Vision (TASKCV).
[7] F. Kong and J. Tan. 2012. DietCam: Automatic dietary assessment with mobile
camera phones. Pervasive Mob. Comput. 8, 1 (2012), 147–163.
[8] Y. Matsuda, H. Hajime, and K. Yanai. 2012. Recognition of Multiple-Food Images
by Detecting Candidate Regions. In Proc. of IEEE International Conference on
Multimedia and Expo. 25–30.
[9] A. Myers, N. Johnston, V. Rathod, A. Korattikara, A. Gorban, N. Silberman, S.
Guadarrama, G. Papandreou, J. Huang, and P. K. Murphy. 2015. Im2Calories:
towards an automated mobile vision food diary. In Proc. of IEEE International
Conference on Computer Vision. 1233–1241.
[10] K. Okamoto and K. Yanai. 2016. An Automatic Calorie Estimation System ofFood Images on a Smartphone. In Proc. of ACM MM Workshop on Multimedia
Assisted Dietary Management.
[11] P. Pouladzadeh, S. Shirmohammadi, and R. Almaghrabi. 2014. Measuring calorie
and nutrition from food image. In IEEE Transactions on Instrumentation andMeasurement. 1947–1956.
[12] J. Redmon, S. Divvala, R. Girshick, and A. Farhadi. 2016. You Only Look Once:
Unified, Real-Time Object Detection. In Proc. of IEEE Computer Vision and Pattern
Recognition.

≤ 5 % rel.err.(%)
61.66755.00055.00051.66756.66771.66758.611≤ 10 % rel.err.(%)
88.33385.00086.66785.00066.66795.00084.444≤ 20 % rel.err.(%)
98.333100.000
96.667100.000
91.667100.000
97.778[13] J. Redmon and A. Farhadi. 2017. YOLO9000: Better, Faster, Stronger. In Proc. ofIEEE Computer Vision and Pattern Recognition.
[14] O. Ronneberger, P. Fischer, and T. Brox. 2015. U-net: Convolutional networks forbiomedical image segmentation. Springer (2015), 234–241.
[15] C. Rother, V. Kolmogorov, and A. Blake. 2004. "GrabCut": Interactive Foreground
Extraction Using Iterated Graph Cuts. ACM Trans. Graph. 23, 3 (2004), 309–314.
[16] W. Shimoda and K. Yanai. 2015. CNN-Based Food Image Segmentation Without
Pixel-Wise Annotation. In Proc. of IAPR International Conference on Image Analysis
and Processing.
[17] K. Simonyan and A Zisserman. 2014. Very deep convolutional networks forlarge-scale image recognition. In arXiv preprint arXiv:1409.1556.