Image-Based Food Calorie Estimation Using Knowledge on
Food Categories, Ingredients and Cooking Directions
Takumi Ege and Keiji Yanai ( The University of Electro-Communications, Tokyo )Experiment : Comparison of single & multi-task

Background & Objective
Meal management apps enable us to record food calories.
Some of them need human help for calorie estimation.

Image-based food calorie estimation
using recipe information

American recipe dataset

Two kinds of calorie-annotatedfood photo datasets.
American recipe datasets
Japanese recipe datasets

550 kcal

CNN

550 kcal
Pilaf
Ingredient vector
Directions vector

Within 20% error

Single
0.78
46 %

Multi
0.82
50 %

Top-1 accuracy (categorization)

81 %

84 %

Related work ① : Miyazaki et al.[1] 2011

Japanese Recipe

Search-based food calorie estimation with conventional features.
Similar image retrieval with SURF and color histograms and so on.
Calculate food calories from retrieved images’ calories.

Correlation coefficient

We propose regression-based method using CNN.

Within 20% error

Multi
0.37
34 %

Top-1 accuracy (categorization)

54 %

61 %

Correlation coefficientMulti-task estimation of food categories and food ingredients.
Multi-task CNN[3] of food categories and food ingredients
Multi-task learning improve both task’s performance.

Correlation coefficient

fc7b

0.35
0.25

0.79

0.2

0.78

0.15
0.1

0.75

47
46

American Recipe
Japanese Recipe

Miso soup

44
4383
82
81

American Recipe

80

Japanese Recipe

79

Categorization failure

432 kcal

677 kcal

35 kcal

184 kcalSpaghetti

Curry

Miso soupMiso soup

429 kcal

730 kcal

32 kcal

214 kcalSpaghetti

Curry

Miso soupMiso soup+3 kcal-53 kcal+3 kcal-30 kcalEstimatedvalue
GroundtruthError

243 kcal

520 kcal

243 kcal

486 kcalMiso soupSpaghetti

Miso soupCurry

575 kcal

228 kcal

58 kcal

944 kcal

Stew

Potato salad

Miso soupCurry-332 kcal+292 kcal+185 kcal-458 kcal

Extension : Combining with Food Detector
FasterR-CNN

𝑦 is an estimated food calorie.
𝑔 is ground-truth.

Food detection(2) Food Category loss: 𝑛Calorie-annotatedCurry : 761 kcal
Potato salad : 169 kcal

(softmax cross entropy)

60 kcalMultitask
CNN

𝑘=1(3) Food Ingredient loss, 𝑛(4) Cooking Direction loss: (mean squared error)

1𝑳𝒊𝒏𝒈,𝒅𝒊𝒓 = ෍ 𝑦𝑘 − 𝑔𝑘 2
284

BadGood

Estimatedvalue
Groundtruth
Error

4585

+++ Direction

𝐿𝑎𝑏 = 𝑦 − 𝑔𝑳𝒄𝒂𝒕 = − ෍ 𝑔𝑘 log 𝑦𝑘0.76

Bad86++ Ingredientabsolute err. loss𝑦−𝑔
𝐿𝑟𝑒 =𝑔Mixed rice

Food photos of calorie-annotated dish cards.

𝑳𝒄𝒂𝒍 = λ𝑟𝑒 𝐿𝑟𝑒 + λ𝑎𝑏 𝐿𝑎𝑏
relative err. loss

Potato salad+ Calorie(1) Calorie estimation loss:

Omelet withfried rice

Category

Calorie estimation loss
Ingredients estimation loss
Category recognition lossDirection estimation loss

Japanese Recipe

0.77

48

+++ Direction( λ is the weight on the loss. )Sushi bowl

61.5
61
60.5
60
59.5
59
58.5
58
57.5
57

49

++ Ingredient𝑁

Cold tofu

Hamburg steakTop-1 accuracy
50+ Category

Auxiliary
info.
(recipe info.)

Total loss (combining four tasks) :𝑛=0

American Recipe0

It is expected to improve the accuracy of each task.

1
𝐿 = ෍ 𝑳𝒄𝒂𝒍 + λ𝑐𝑎𝑡 𝑳𝒄𝒂𝒕 + λ𝑖𝑛𝑔 𝑳𝒊𝒏𝒈 + λ𝑑𝑖𝑟 𝑳𝒅𝒊𝒓
𝑁

0.8

0.3

0.05

[ Loss function ]NikujagaGround-truth / kcal

51

Calorie

fc7dConv layers ( VGG16 [4])

0.81

35.5
3534.53433.53332.53231.53130.5+++ Directions

fc8d

Cooking directions vector
Directions estimation

0.4++ Ingredientfc7c0.82+ Category

fc8cFood ingredients vector
Ingredients estimation

fc6Stew(%)

Within 20% error

0.45

Calorie

fc8b

Potato salad
Category recognition

Miso soup

Multi-task learning improves the performance of both tasks.

We use multi-task CNN with recipe info. for calorie estimation.

fc8aGratinSpaghetti

R=0.82

MSingle0.29
32 %American Recipe

Related work② : Chen and Ngo[2] 2016

fc7a

Fried noodle

Ground-truth / kcal

Estimated value / kcal

CNN

Method: Multi-task CNN with recipe info.

335 kcalCalorie estimation

Fried rice

R=0.78[Multi-task CNN (proposal method)]

Estimation
target(calories)

Curry[Single-task CNN (baseline)]

Miso soup 60kcal

Food calorie for one person.

Pilaf

Estimated value / kcal

Salad 35kcal

Tofu 42kcal[ Our network ]Japanese recipe dataset

We collected recipe data fromthe online cooking recipe sites.

Grilled fish 221kcal

Rice 168kcal

DATASET

Calorie estimation

Miso soup
Ingredient vec.
Direction vec.

( 𝑔 is a ingredient/cooking direction vector. )𝑘=1

We collected recipe data from the online cooking recipe sites.
We converted each word of food ingredient names andcooking directions into a real-value vector by Word2Vec,
and averaged them.
rotini or fusili pasta
slices bacon[0.053423 -0.087293 … 0.007234]

Word2vec [0.037983 -0.009285 … 0.034096]

onions, chopped[0.002938 -0.048793 … 0.001240]

・・・・・・pasta[0.008439 -0.046298 … 0.023428]

deep skillet

Word2vec[0.023982 -0.004298 … 0.073249]

crisp[0.003428 -0.092392 … 0.006345]

・・・・・・[0.078389 -0.015354 … 0.002023]

Food ingredients vector[0.005998 -0.035861 … 0.019575]

Cooking directions vector

Ground-truth
Stew : 382 kcal
Spaghetti : 518 kcal

Potato salad : 169 kcal
Miso soup : 74 kcal
Nikujaga : 352 kcal

Conclusions & Future work•••••

We proposed food calorie estimation with a multi-task CNN.
Multi-task learning improved both food calorie and category estimation.
We constructed a new calorie-annotated food photo dataset.
Food calorie estimation considering the volume of foods.
Construction of large-scale calorie-annotated food photo dataset.

[1] T. Miyazaki, G. Chaminda, D. Silva, and K. Aizawa. Image ‐ based calorie content estimation for dietary assessment. In Proc. of IEEE ISM Workshop on Multimedia for Cooking and Eating Activities, 2011.
[2] J. J. Chen and C. W. Ngo. Deep-based ingredient recognition for cooking recipe retrival. In Proc. of ACM International Conference Multimedia, 2016.
[3] H. A. Abrar, W. Gang, L. Jiwen, and J. Kui. Multi-task CNN model for attribute prediction. IEEE Transactions on Multimedia, Vol. 17, No. 11, pp. 1949–1959, 2015.
[4] K. Simonyan and A Zisserman. Very deep convolutional networks for large-scale image recognition. In arXiv preprint arXiv:1409.1556, 2014.
[5] Y. Matsuda, H. Hajime, and K. Yanai. Recognition of multiple-food images by detecting candidate regions. In Proc. of IEEE International Conference on Multimedia and Expo, 2012.