Simultaneous Estimation ofFood Categories and Calories
with Multi-task CNN
Takumi Ege and Keiji Yanai
The University of Electro-Communications, Tokyoⓒ 2017 UEC Tokyo.

1Introduction (1)Spread of meal management applications.
Recording food calorie.

e.g. Foodlog http://www.foodlog.jp/introduction/log.jaⓒ 2017 UEC Tokyo.

2

Introduction (2)Example of applications to record food calorie.
CaloNavi

Foodlog

Volume selection required

Crop a dishFood category recognition
Select volume manually, etc…

Dietary advices bynutrition professional.
Human cost
Pay serviceⓒ 2017 UEC Tokyo.

3

ObjectiveImage-based food calorie estimation
Grilled fish 221kcal

Rice 168kcal

Salad 35kcal

Miso soup 60kcal

Tofu 42kcalⓒ 2017 UEC Tokyo.

Related works (1)• Im2Calories [Myers et al. 2015]– CNN-based categorization– CNN-based 3D sized estimation etc…

Myers et al. Im2calories, In Proc. ofIEEE International Conference on
Computer Vision, 2015.
ⓒ 2017 UEC Tokyo.

5

Related works (2)Take a meal photowith a reference object• CalorieCam [Okamoto et al. 2016]– CNN-based categorization.
– 2D sized estimation.
Size-knownreference object.

Extract a region ofa reference object

Extract regions offood items

Compare both region sizes

Recognize a categoryof each detected foodCalculate food calories based on
their size and food categories

Okamoto et al. An Automatic Calorie Estimation
System of Food Images on a Smartphone,
MADiMa, 2016.
ⓒ 2017 UEC Tokyo.

6

Related works (3)• Direct calorie estimation
by regression based.
[Miyazaki et al. 2011]

Estimated value / kcal

• Without estimating foodcategories and volumes.
R = 0.32

Ground-truth / kcal

Miyazaki et al. Image-based Calorie Content
Estimation for Dietary Assessment, CEA, 2011
ⓒ 2017 UEC Tokyo.

7

Our approach• Image-based direct calorie estimation.
• CNN-based method.
• Multi-task estimationof food categories and calories.
ⓒ 2017 UEC Tokyo.

8

Single-task estimation•••

Direct calorie estimation by regression based.
CNN-based method.
Single-task estimation of food calories.

CNN

550 kcalFood calorie estimationⓒ 2017 UEC Tokyo.

9

Multi-task estimation (proposal method)•••

Direct calorie estimation by regression based.
CNN-based method.
Simultaneous estimationof food categories and calories.

550 kcal
CNN

Food calorie estimation

Pilaf
Food category recognition

Improve accuracy of each task
ⓒ 2017 UEC Tokyo.

10

Network architectureof Multi-task CNN• VGG16 are extended.
• Simultaneous estimationof food categories and calories.
Calorie estimation
4096
4096

fc6

fc7c

ⓒ 2017 UEC Tokyo.

fc8c

Food category recognition
4096
Potato salad
fc7f

Conv layers ( VGG16 )335 kcal

fc8f

Fc layers11Construction of calorie-annotatedfood photo dataset
Cooking recipe sitesA food photo

Food Calorie

About 83,000 recipes were collected.
ⓒ 2017 UEC Tokyo.

12

Construction of calorie-annotatedfood photo dataset
Labeling on 100 meals ofUEC food-100 [1].
Noise removal.

PilafCurryFried rice

GratinMiso soup

Stew

Cold tofu

Sushi bowl

Fried noodle

Spaghetti

Remove categories withfewer than 100 samples.
Simmered meat Hamburg steak
and potatoes

15 categories.
A total of 4877images.
Food[1] Matsuda et al. Recognition of multiple-food images by detecting candidate regions.
In Proc. of IEEE International Conference on Multimedia and Expo, 2012.

ⓒ 2017 UEC Tokyo.

Omelet with Potato salad
fried rice

Mixed rice13

Training of our network• Training: 70%, Validation during training: 10%, Testing: 20%• Frame work : Chainer [1].
Food calorie forone person.

Food images containing
only one dish.

Calorie estimation

4096

fc6

4096

1

fc7c

fc8c

Food category recognition

4096 15
fc7f

Conv layers ( VGG16 )335 kcal

fc8f

Potato salad

Fc layers[1] Tokui et al. Chainer: a nextgeneration open source framework for deep learning. NIPS, 2015.
ⓒ 2017 UEC Tokyo.

14

Experiments
Train and compare both networks.

550 kcal
550 kcal
Pilaf

Single-task CNN(baseline)

Multi-task CNN(proposal method)ⓒ 2017 UEC Tokyo.

15

Comparison of single and multi-task
Within 20% error

Calorieestimation

43%

48%

R=0.81

Estimated value / kcal

Estimated value / kcal

R=0.78Within 40% error

76%

80%

Top-1 accuracy

Category
recognition
80%
Ground-truth / kcal

82%

Single-task CNN(baseline)ⓒ 2017 UEC Tokyo.

Ground-truth / kcal

Multi-task CNN(proposal method)
16

Good results
Miso soupiⓒ 2017 UEC Tokyo.

17

Bad resultsCategorization failureiⓒ 2017 UEC Tokyo.

18

Comparison to a existing work
[Miyazaki et al. 2011] (No CNN)
Correlation coefficient

0.32

R=0.81

Estimated value / kcal

Estimated value / kcal

R=0.32

0.81

Within 20% error

35%

48%

Ground-truth / kcal

Direct estimation withconventional hand-craftedfeatures [Miyazaki et al.ⓒ2011]2017 UEC Tokyo.

Ground-truth / kcal

Multi-task CNN [Ours]
19

Conclusions• We proposed simultaneous estimation offood categories and calories for food photos.
• We achieved great improvementwith multi-task CNN.
• We constructed Calorie-annotatedfood photo dataset.
ⓒ 2017 UEC Tokyo.

20

Future work

• Use of ingredients and cooking directions.
• CNN-based region segmentation (+3D volumeestimation) for complicated background.

ⓒ 2017 UEC Tokyo.

21

ⓒ 2017 UEC Tokyo.

22

Construction of calorie-annotatedfood photo dataset• The data collected from the recipe sites has …
– food images containing only one dish.
– food calories for one person.

• We assume to …
– input a food image containing only one dish.
– estimate food calorie for one person.
(without considering volumes.)ⓒ 2017 UEC Tokyo.

23

Multiple dishes estimation
Combine food dish detection.
FasterR-CNN[1]Use food photos ofcalorie-annotateddish cards.

Object detection

Calorie estimation

Misosoup[1] S. Ren et al. Faster R-CNN: Towards realtime object
detection with region proposal networks. In Advances in
Neural Information Processing Systems, 2015.
ⓒ 2017 UEC Tokyo.

Calorie-annotatedCurry : 761 kcal
Potato salad : 169 kcal
24

Results of multiple dishes estimation

Calorie-annotatedCurry : 761 kcal
Potato salad : 169 kcal

Ground-truth
Stew : 382 kcal
Spaghetti: 518 kcal

results
Potato salad: 169 kcal
Hamburg steak : 405 kcal
Fried rice: 695 kcal
ⓒ 2017 UEC Tokyo.

25Comparison to a simple baseline
Correlation coefficient

Estimated value / kcal

0.78

0.81
Estimated value / kcal

R=0.78R=0.81

Within 20% error

50%

48%

Within 40% error
Ground-truth / kcal

77%

Ground-truth / kcal

80%

The calorie estimation basedon food category recognition.

Multi-task CNN. [Ours][Simple baseline]ⓒ 2017 UEC Tokyo.

26

Estimated value / kcal

Representative 3 categories
R=0.19R=0.45R=0.07Ground-truth / kcal

Ground-truth / kcal

Ground-truth / kcal

Spaghetti

Miso soup

Simmered meat andpotatoesⓒ 2017 UEC Tokyo.

27

Construction of calorie-annotatedfood photo dataset
The distributions of the food calories of the collected recipes.

“Spaghetti”, “Miso soup”,“Simmered meat and potatoes”.

All the 15 foods.
ⓒ 2017 UEC Tokyo.

28