AR DeepCalorieCam V2: Food Calorie Estimation
with CNN and AR-based Actual Size Estimation
Ryosuke Tanno∗

Takumi Ege, Keiji Yanai

Department of Informatics, The University ofElectro-Communications, Tokyo
tanno-r@mm.inf.uec.ac.jp

Department of Informatics, The University ofElectro-Communications, Tokyo{ege-t,yanai}@mm.inf.uec.ac.jp

ABSTRACTIn most of the cases, the estimated calories are just associated withthe estimated food categories, or the relative size compared to thestandard size of each food category which are usually provided
by a user manually. In addition, in the case of calorie estimation
based on the amount of meal, a user conventionally needs to register a size-known reference object in advance and to take a foodphoto with the registered reference object. In this demo, we propose a new approach for food calorie estimation with CNN and
Augmented Reality (AR)-based actual size estimation. By using Apple ARKit framework, we can measure the actual size of the meal
area by acquiring the coordinates on the real world as a threedimensional vector, we implemented this demo app. As a result,
it is possible to calculate the size more accurately than in the previous method by measuring the meal area directly, the calorie estimation accuracy has improved.

Figure 1: Example of the use of the proposed iOS App "AR
DeepCalorieCam V2"

KEYWORDS
Food Calorie Estimation, Augmented Reality, Deep Learning, iOS,Application
ACM Reference Format:
Ryosuke Tanno and Takumi Ege, Keiji Yanai. 2018. AR DeepCalorieCam V2:
Food Calorie Estimation with CNN and AR-based Actual Size Estimation.
In VRST 2018: 24th ACM Symposium on Virtual Reality Software and Technology (VRST ’18), November 28-December 1, 2018, Tokyo, Japan. ACM, New
York, NY, USA, Article 4, 2 pages. https://doi.org/10.1145/3281505.3281580

1INTRODUCTION

In recent years, due to a rise in healthy thinking on eating, manypeople take care of their eating habits, and some people record
daily diet regularly. To assist them, many mobile applications forrecording everyday meals have been released so far. Some of them
employ food image recognition which can estimate not only foodnames but also food calories. However, in most of the cases, theestimated calories are just associated with the estimated food categories, or the relative size compared to the standard size of eachfood category which are usually provided by a user manually. Mostof the applications does not estimate calories based on the amountof foods.
On the other hand, there are some studies on calorie estimationconsidering the amount of meals. For example, in [3], to estimate
food calorie from a single image, a user needs to register a sizeknown reference object in advance and to take a food photo withthe registered reference object. As a reference object, they assume∗ Currently he is working for NTT Communications.

Permission to make digital or hard copies of part or all of this work for personal orclassroom use is granted without fee provided that copies are not made or distributedfor proﬁt or commercial advantage and that copies bear this notice and the full citationon the ﬁrst page. Copyrights for third-party components of this work must be honored.
For all other uses, contact the owner/author(s).
VRST ’18, November 28-December 1, 2018, Tokyo, Japan© 2018 Copyright held by the owner/author(s).
ACM ISBN 978-1-4503-6086-9/18/11.
https://doi.org/10.1145/3281505.3281580

a personal belonging which they are always carrying such as acredit-card-size card. After taking a meal photo with a referenceobject, the system carries out segmentation of food items and thepre-registered reference object. Hence, the real size of the reference object is known (e.g. In case of a credit-card-size object, thesize is 85.6mm x 54mm.), the system can estimate the real size ofeach detected food items. By using the estimated real size and theequations to calculate food calorie from their size, the system ﬁnally estimates the calorie of the food items in the real photo. However, with the above method, it is absolutely necessary to have a
pre-registered reference object to estimate the real size of meal.
In addition, the dispersion of calorie estimation accuracy is large,
since indirectly obtaining the meal area from the reference object
depends on the segmentation accuracy of the meal area.
In this demo, we propose a new approach for food calorie estimation with CNN and AR-based actual size estimation. By using
Apple ARKit framework, it is possible to measure the actual sizeof the meal area by acquiring the coordinates on the real world asa three-dimensional vector. Figure 1 shows an example of usage ofthe proposed "AR DeepCalorieCam V2".
Our contributions are as follows.
• The reference object, which was conventionally necessary
to obtain the actual meal area, became unnecessary by using
AR technology on mobile devices such as iPhone.
• By measuring the meal area directly, it is possible to calculate the size more accurately than in the previous method.
(Please refer to the Table 1)

2 TECHNICAL DESCRIPTION
The demo system outline is as shown in the Figure 2. The processing of the proposed system consists of the following two steps:• Recognize a category of each food item.
• Directly calculate the size of the region of the food item offood items using AR and then calculate food calories basedon their actual size and food categories.

VRST ’18, November 28-December 1, 2018, Tokyo, Japan

Ryosuke Tanno and Takumi Ege, Keiji Yanai

Figure 2: An overview of the demo system(Left: Proposed Method, Right: Previous Method)
Table 1: Estimation errors by the baseline [1] and the baseline [3] and the proposed system.

food namebeef rice bowl
croquette
salad

2.1

real value(kcal)
962
552
14

baseline[1]avg. err.
SD-53.25
209.79
-242
91.26
54.83
36.28

Food Category Recognition

For mobile implementation, the memory capacity and processing
time of the device is an important factor in implementing deeplearning. For this time, we decided to use Inception-v3 [4] which
has a light memory, faster inference and high classiﬁcation accuracy. We ﬁne-tune a pre-trained ImageNet model in Keras Deep
Learning framework with UEC-FOOD100 dataset [2].

2.2

Calorie Estimation

We assume that the height of food portion correlates with the sizeof foods and food categories, and we estimate calories of food items
directly from the food sizes estimated from the top-view image. To
do that, we use not simple linear estimation but quadratic curve
estimation from the 2D size of foods to their calories. The quadratic curve of each food category is estimated based on the training data annotated with real food calories independently. We use aquadratic curve estimation from the 2D size of foods to their calories.
In the case of the proposed method, we can calculate the real sizeof foods area directly using AR technology on mobile devices suchas iPhone. Therefore, the reference object, which was conventionally necessary to obtain the actual meal area, became unnecessary.
By using Apple ARKit framework, we can measure the actual sizeof the meal area by acquiring the coordinates on the real world asa three-dimensional vector and then calculate food calories basedon their actual size and food categories.

baseline[3]avg. err.
SD-24255.1-47.08
52.52
4.86
11.87

proposed
avg. err. SD-67.14
18.8-127.0
9.0
-0.95
0.16

3 DEMO OVERVIEWThis demo is performed by the step on the left side of Figure 2.
First of all, we recognize meal categories by taking food samples
on iPhone. Next, we directly calculate regions of food items using
AR. Finally, we calculate food calories based on their actual sizeand food categories.

4 CONCLUSION
In this demo, we propose a new approach for food calorie estimation with CNN and AR-based actual size estimation. In the future,
we plan to implement an automatic calorie estimation system thatcombines the proposed method and segmentation, taking into account the size of the meal area.
ACKNOWLEDGMENTS: This work was supported by JSPS KAKENHI Grant Number 15H05915, 17H01745, 17H05972, 17H06026
and 17H06100.

REFERENCES[1] Y. Kawano and K. Yanai. 2013. Real-time Mobile Food Recognition System. In
Proc. of CVPR International Workshop on Mobile Vision (IWMV).
[2] Y. Matsuda, H. Hoashi, and K. Yanai. 2012. Recognition of Multiple-Food Images
by Detecting Candidate Regions. In Proc. of IEEE International Conference on Multimedia and Expo (ICME).
[3] K. Okamoto and K. Yanai. 2016. An Automatic Calorie Estimation System of FoodImages on a Smartphone. In Proc. of ACM MM Workshop on Multimedia Assisted
Dietary Management (MADiMa).
[4] Christian Szegedy, Vincent Vanhoucke, Sergey Ioﬀe, Jonathon Shlens, and Zbigniew Wojna. 2015. Rethinking the Inception Architecture for Computer Vision.
In Proc. of arXiv:1512.00567.