Comparison of Two Approaches
for Direct Food Calorie Estimation
Takumi Ege and Keiji Yanai
The University of Electro-Communications, Tokyo

Direct food calorie estimation
CaloNaviFoodlog

Volume selection required

Crop a dishFood category recognition
Select volume manually, etc…

Dietary advices bynutrition professional.
Human cost
Pay service

Fully-automatic food calorie estimation from a food photo
has still remained as an unsolved problem.
ⓒ 2017 UEC Tokyo.

2

Previous work: Regression-based• Regression-based direct calorie estimation.
– Multi-task CNN of food categories and calories.
– Create a new calorie-annotated food photo dataset.

550 kcal
CNN

Food calorie estimation

Pilaf
Food category recognition

Ege and Yanai. Simultaneous estimation of food categories and calories with multi-task cnn. In Proc. of IAPR
International Conference on Machine Vision Applications(MVA), 2017.
ⓒ 2017 UEC Tokyo.

3

Previous work: Search-based• Search-based food calorieestimation.
– Similar image search based on
hand-crafted features such as
SURF-based BoF and color
histograms.

– Create a new calorie-annotatedfood photo dataset.
Miyazaki et al. Image‐based Calorie Content Estimation for Dietary
Assessment, Workshop on Multimedia for Cooking and Eating Activities,
2011.
ⓒ 2017 UEC Tokyo.

4Comparison of two methods
Search-based

Regression-based

550 kcal

CNNPilafⓒ 2017 UEC Tokyo.

5

Regression-based• Multi-task learning of food categories and calories.
• The architecture is based on VGG16.

Calorie estimation

335 kcal
fc7c

fc6

fc8c

Food category recognition

Potato salad
fc7f

Conv layers ( VGG16 )fc8f

Fc layers

Ege and Yanai. Simultaneous estimation of food categories and calories with multi-task cnn. In Proc. of IAPR
International Conference on Machine Vision Applications(MVA), 2017.
ⓒ 2017 UEC Tokyo.

6

Regression-based• Loss function is a linear combination of food calorieestimation loss 𝐿𝑐𝑎𝑙 and classification loss 𝐿𝑐𝑎𝑡 .
– Food calorie estimation loss 𝐿𝑐𝑎𝑙 is a linear combinationof a relative error 𝐿𝑟𝑒 and an absolute error 𝐿𝑎𝑏 .

Let 𝑦𝑖 as the estimated value of an image 𝑥𝑖 and𝑔𝑖 as the ground-truth.
– Classification loss 𝐿𝑐𝑎𝑡 is softmax cross entropy.
ⓒ 2017 UEC Tokyo.

7

Search-based<clscls(Fc6activation)(Fc6activation)(Fc6activation)

Negative

For searching similarimages, we use fc6
activation of VGG16
trained with triplet loss 𝐿 𝑇and classification loss
𝐿𝐶+ , 𝐿𝐶 and 𝐿𝐶− .

∙ 2Querycls∙ 2ⓒ 2017 UEC Tokyo.

samedifferent

8

Shared parameters•

Search similar images
from the database anduse the food calorie ofthe images.

𝐿𝐶−𝐿𝐶 𝐿 𝑇

Positive•

𝐿𝐶+Search-based• Loss function is a linear combination of
𝐿 𝑇 , 𝐿𝐶+ , 𝐿𝐶 and 𝐿𝐶− .
– Triplet loss 𝐿 𝑇𝐿𝑇
𝑓+ − 𝑓 2 : Euclidean distance between fc6 activationsof the same category images.
𝑓− − 𝑓 2 : Euclidean distance between fc6 activationsof the different category images.

– Classification loss 𝐿𝐶+ , 𝐿𝐶 and 𝐿𝐶− are softmax crossentropy.
ⓒ 2017 UEC Tokyo.

9

Calorie-annotatedfood image dataset
We use calorie-annotated 15 food category dataset. [Ege and Yanai, 2017]
Recipe info. cites

A food photo

Food Calorie

About 83,000 recipes were collected.
ⓒ 2017 UEC Tokyo.

10

Calorie-annotatedfood image dataset
Labeling on 100 meals ofUEC food-100 [1].
Noise removal.

PilafCurryFried rice

GratinMiso soupStewCold tofu

Sushi bowl

Fried noodle

Spaghetti

Remove categories withfewer than 100 samples.
Simmered meat Hamburg steakand potatoes

15 categories.
A total of 4877images.
Food[1] Matsuda et al. Recognition of multiple-food images by detecting candidate regions.
In Proc. of IEEE International Conference on Multimedia and Expo, 2012.

ⓒ 2017 UEC Tokyo.

Omelet with Potato salad
fried rice

Mixed rice11

Experiment : Regression-based• We train CNN with relative error and absolute error
on calorie-annotated food image dataset.
– Training: 70%, Testing: 30%Food calorie forone person.

Food images containing
only one dish.

Calorie estimation

335 kcal
fc7c

fc6

Food category recognition
fc7f

Conv layers ( VGG16 )ⓒ 2017 UEC Tokyo.

fc8c

fc8f

Potato salad

Fc layers
12Experiment : Search-based<(Fc6activation)(Fc6activation)(Fc6activation)

Negativecls∙ 2clscls∙ 2Query• We create CNN feature
database on calorieannotated food image
dataset.

𝐿𝐶−𝐿𝐶 𝐿 𝑇

Positive• We use an activation signals
of fully connected layers of
the trained network as CNNfeatures.

𝐿𝐶+same– Database : 70%, Testing : 30%ⓒ 2017 UEC Tokyo.

different

Food images containing
only one dish.

13Shared parameters• We train VGG16 with triplet
loss and classification loss
on UECFOOD100.

Result : Comparison of two methods• Baseline (Classification)– Use of fixed calorie value of each category.

Rel.
err.
(kcal)

Abs.
err.
(%)

Corre < 20 %lation rel. err.
(%)

Baseline

32.4

93.6

0.784

50.0

VGG16 single-task

29.2

100.4

0.783

46.0

VGG16 multi-task

28.0

96.5

0.805

47.2

VGG16(uecfood100) fc6

38.0

101.0

0.764

47.2

VGG16(uecfood100) fc6 triplet

38.4

101.5

0.754

45.9

VGG16(uecfood100) fc6 triplet + cls

36.7

98.6

0.777

48.1

modelReg.

Sea.

ⓒ 2017 UEC Tokyo.

14

Result : Comparison of two methods• Correlation between estimated value and ground-truth

Baseline(Classification)

Regression-based(VGG16 multi-task)ⓒ 2017 UEC Tokyo.

Search-based(VGG16(UECFOOD100)
fc6 triplet + cls)
15

Conclusion• We compared two methods for direct foodcalorie estimation.
• Regression-based method is better thanothers, however it is not reliable results.
• It is necessary to consider food ingredients
as well as food categories.
ⓒ 2017 UEC Tokyo.

Future work
• We use food ingredients information onrecipe data.
• In order to realize highly accurate foodcalorie estimation, we create new datasets.
550 kcal

CNN

Food calorie estimation

Pilaf
Food category recognition

Food ingredientsⓒ 2017 UEC Tokyo.

18