Multi-task Learning of Dish
Detection and Calorie Estimation
Takumi Ege and Keiji Yanai
The University of Electro-Communications, TokyoIntroduction
CaloNavi

Foodlog

Volume selection required

Crop a dishFood category recognition
Select volume manually, etc…

Dietary advices by
nutrition professional.
Human cost
Pay service

Fully-automatic food calorie estimation from a food photo
has still remained as an unsolved problem.
ⓒ 2018 UEC Tokyo.

2

Introduction
Food image recognition for multiple-dish photos.

Detection of single-dish from multiple-dish photos.
It is possible to record meal easily in a short time.
ⓒ 2018 UEC Tokyo.

Objective
Image-based food calorie estimation
Grilled fish 221kcal

Rice 168kcal

Salad 35kcal

Miso soup 60kcal

Tofu 42kcalⓒ 2018 UEC Tokyo.

Related works : Calorie estimation• Im2Calories [Myers et al. 2015]– CNN-based categorization– CNN-based 3D sized estimation etc…

Myers et al. Im2calories: towards anautomated mobile vision food diary. In Proc. ofIEEE International Conference on Computer
Vision, 2015.

ⓒ 2018 UEC Tokyo.

5

Related works : Dish detection• CNN-based Food Image Segmentation [Shimoda et.al. 2015]– Region proposals are generated by selectie search.
– The saliency map for each region are unified.

W. Shimoda and K. Yanai. CNN-based food image segmentation without pixel-wise annotation.
In Proc. of IAPR International Conference on Image Analysis and Processing, 2015.
ⓒ 2018 UEC Tokyo.

6

Method : Overview of our network
Dish detection

Networkoutput

Conv layers

Non-Maximum suppression+Threshold processing

Input

Final output
Potato salad

General CNN-based detection networkⓒ 2018 UEC Tokyo.

7

Method : Overview of our network
Dish detection + Calorie estimationNetworkoutput

Conv layers

Non-Maximum suppression+Threshold processing

Input

Final output
Potato salad+150 kcal

Overview of our networkⓒ 2018 UEC Tokyo.

8

NetworkoutputDetection Network

NMS + Thresh

Method : Overview of our network
Potato salad+150 kcal

Output feature map( Ours )
1,2,3

C-1,C

(x,y,w,h) (class) (calorie)

Bbox 1ⓒ 2018 UEC Tokyo.

Bbox B

9Method : Dish detection
Introduction of SSD (Single Shot MultiBox Detector)••

High-speed and highly accurate CNN-based detection system.
End-to-end learning of the whole system is possible.

The architecture of network of SSD (quoted from [1]).
[1] W. Liu, D. Anguelov, D. Erhan, C. Szegedy, and S. E. Reed. SSD: single shot multibox detector. CoRR, abs/1512.02325, 2015.
ⓒ 2018 UEC Tokyo.

10

Method : food calorie estimation
Regression based method••

Direct calorie estimation by regression based.
CNN-based method for single-dish food photos.

CNN

550 kcalFood calorie estimation

Ege and Yanai. Simultaneous estimation of food categories and calories with multi-task cnn. In Proc. of IAPR
International Conference on Machine Vision Applications(MVA), 2017.
ⓒ 2018 UEC Tokyo.

11

Method : Multi-task learning
Training of network with two types of datasets• Dataset for dish detection– Bounding boxes– Classes• Dataset for calorie estimation– Food calories– Classes

Multi-task learning of Dish detection andCalorie estimation with both datasetⓒ 2018 UEC Tokyo.

12Dataset : two types of datasets
Training of network with two types of datasets• UECFood-100[1]– With bounding boxes and classes.
– Include multiple-dish food photos.

• Calorie50– With food calories and classes.
– Only single-dish food photos.

[1] Matsuda et al. Recognition of multiple-food images by detecting candidate regions.
In Proc. of IEEE International Conference on Multimedia and Expo, 2012.

ⓒ 2018 UEC Tokyo.

13

Dataset : UECFood-100• 100 kinds of foods.
• Train: 11566 single-dish food photos.
• Test: 1174 multiple-dish food photos.
• Annotation:Bounding boxes,
class labels.
Some of examples of test images[1] Matsuda et al. Recognition of multiple-food images by detecting candidate regions. In Proc. of IEEE International Conference on Multimedia andExpo, 2012.
ⓒ 2018 UEC Tokyo.

Dataset : Calorie50• 50 kinds of foods included in UECFood-100.
• Train : 5370 single-dish photos.
• Test : 2317 single-dish photos.

• Annotation:
Food calories,class labels.

[1] Ege and Yanai. Simultaneous estimation of food categories and calories with multi-task cnn. In Proc. of IAPR International
Conference on Machine Vision Applications(MVA), 2017.ⓒ 2018 UEC Tokyo.

Experiment : Our network
Detection network : SSD[1] +Calorie estimation : Regression method[2]Potato salad+150 kcal

NMS+
ThresholdNetworkoutputSSD network + Calorie[1] W. Liu, D. Anguelov, D. Erhan, C. Szegedy, and S. E. Reed. SSD: single shot multibox detector. CoRR, abs/1512.02325, 2015.
[2] Ege and Yanai. Simultaneous estimation of food categories and calories with multi-task cnn. In Proc. of IAPR International
Conference on Machine Vision Applications(MVA), 2017.ⓒ 2018 UEC Tokyo.

16

Experiment : Multi-task learning
Training of network with two types of datasets• UECFood-100 – Detection loss function[1]– With bounding boxes and classes.
– Dish detection• Calorie50 – Detection loss[1] + Calorie loss[2]– With pseudo-bounding boxes, classes and calories.
– Food calorie estimation[1] W. Liu, D. Anguelov, D. Erhan, C. Szegedy, and S. E. Reed. SSD: single shot multibox detector. CoRR, abs/1512.02325, 2015.
[2] Ege and Yanai. Simultaneous estimation of food categories and calories with multi-task cnn. In Proc. of IAPR International
Conference on Machine Vision Applications(MVA), 2017.ⓒ 2018 UEC Tokyo.

17

Experiment : Pseudo-bounding boxes
Bounding box and calorie correspondence required.
pseudo-bounding-boxes to Calorie50.
• Use Train images.
1. Prepare one background image.

Pseudo-bounding boxes
for Calorie50 dataset.

2. Paste single-dish image at arandom position,and make the image area thecorrect bounding box.
ⓒ 2018 UEC Tokyo.

18

Experiment : Training settings• Datasets– UECFood-100• Train : 11566 images (single-dish)• Test : 1174 images (multiple-dish)– Calorie50• Train : 5370 images (single-dish)+ 10000 images with pseudo-bboxes (multiple-dish)• Test : 2317 images (single-dish)• Training of networks• Input size : 300 x 300• SGD (momentum of 0.9)
• Mini-batch of 32.
• 𝟏𝟎−𝟑 of learning rate for 40,000 iterations and thenused 𝟏𝟎−𝟒 for 10,000 iterations.
ⓒ 2018 UEC Tokyo.

19

ExperimentComparison of single-task and multi-task learning• Baseline (Single-task learning)– Single-task learning of dish detection task (a)– Single-task learning of calorie estimation task (b)– Sequential model (a)→(b)• Our method (Multi-task learning)– Multi-task learning of dish detection andcalorie estimation with both datasets.
ⓒ 2018 UEC Tokyo.

20

Results : Dish detection＋Calorie estimation• Evaluation values
Detection

Calorie– mAP : mean Average Precision– Absolute error (kcal)： 𝑦𝑖 − 𝑔𝑖–𝑦 −𝑔Relative error (%)： 𝑖𝑔 𝑖𝑖

Let 𝑦𝑖 as the estimated value of an image 𝑥𝑖 and𝑔𝑖 as the ground-truth.

– Correlation value between estimated calorie value and ground-truth.
ModelmAP(%)

Rel.
err.

Abs.
err.

Corr.

Detection (uecfood100)

34.1

---

---

---

Detection (uecfood100+calorie50) (a)

37.8

---

---

---

Calorie estimation (calorie50) (b)

---

27.1

91.8

80.7

Sequential model ((a)→(b))

---

27.3

92.5

80.5

Detection+Calorie estimation (Ours)

37.7

26.6

89.4

81.0

ⓒ 2018 UEC Tokyo.

Results : Dish detection＋Calorie estimation
Potato salad 144kcal(ES)Fried noodle 601 kcal(ES)Pilaf 495kcal(ES)Miso soup 102kcal(ES)Nikujaga 417kcal(ES)Potato salad 169kcal(GT)Miso soup74kcal(GT)Nikujaga
352kcal(GT)Fried noodle 539kcal(GT)Pilaf475kcal(GT)Gratin 456kcal(ES)Curry 483kcal(ES)Spaghetti 536kcal(ES)Potato salad 170kcal(ES)Gratin450kcal(GT)Spaghetti 518kcal(GT)Curry761kcal(GT)Potato salad 169kcal(GT)The results of dish detection from multiple-dish food photos.(Our papers model)ⓒ 2018 UEC Tokyo.

Conclusion• We estimate food calories from multiple-dishfood photos.
• Multi-task learning of dish detection and foodcalorie estimation.

ⓒ 2018 UEC Tokyo.

Future work
• Construction of large-scale calorieannotated food photos dataset.
• Comparative experiments with additional
learning of dish detection and calorieestimation.

ⓒ 2018 UEC Tokyo.

25