Multi-task Learning ofDish Detection and Calorie Estimation
Takumi Ege and Keiji Yanai
The University of Electro-Communications, Tokyo

BackgroundExperiment

Some meal recording app can estimate food calories.
But they …

 Baseline (Single-task learning)• Need user’s manual input of food categories and volumes.
• Estimate food calories for each dish one by one.
• Are paid service to hire nutritionists who estimate food calories.

Purpose : Image-based food calorie estimation
Grilled fish221kcal• Single-task learning of dish detection task with UECFood-100 dataset [3]. (a)• Single-task learning of calorie estimation task with Calorie-50 dataset.
(b)• Sequential model ((a)→(b)) that detects each dish in a food image by (a) andthe food calorie of each dish image based on detected results are estimated by (b).

 Our method (Multi-task learning)• Multi-task learning of dish detection and calorie estimation with both datasets.
In training of UECFood-100 dataset, we use the detection loss.
In training of Calorie-50 dataset, we use the calorie loss and detection loss.

P

Salad 35kcalRice
168kcalMiso soup 60kcal

Tofu 42kcal

Calorie50 datasetwith food calories.

MethodMulti-task learning of dish detection and food calorie estimation• Simultaneous estimation of bounding boxes of food dishes and their calories.
• The output channels of estimated calories are added to the output features.

1,2,3

ResultComparison of single-task and multi-task learning
ModelmAP
(%)

Rel.
err.

Abs.
err.

Corr.

≤20%err.

Detection (uecfood100)

34.1

---

---

---

---

Detection (uecfood100+calorie50) (a)

37.8

---

---------

Calorie estimation (calorie50) (b)

---

27.1

91.8

80.7

50.9

Sequential model ((a)→(b))

---

27.3

92.5

80.5

50.6

Detection + Calorie estimation (Ours)

37.7

26.6

89.4

81.0

50.7

C-1,C

(x,y,w,h) (class) (calorie)

Bbox 1

Conv layers ( Detection Network )Bbox B

Output feature map ( Ours )1. SSD : Wei Liu et. al. [1] 2016 ( Detection Net )• High-speed and highly accurate CNN-based detection system.
• End-to-end learning of the whole system is possible.

Our method achieves the calorie estimation frommultiple-dish photos without degrading dish detection accuracy.
Our method achieves high speed and save memory bysimultaneous estimation in a single network.
Potato salad 144kcal(ES)Fried noodle 601 kcal(ES)P

Pilaf 495kcal(ES)The architecture of network of SSD ([1]).

We use the network for single-dish detection from multiple-dish photos.

Fried noodle 539kcal(GT)Pilaf475kcal(GT)• Image-based food calorie estimation with CNN.
• Regression based-method.
• Output food calories directly from single-dish food photos.
Food calorie for one person.

Food calorie estimation
fc6

fc8c335 kcal

Conv layers ( VGG16[4] )• We denote 𝑳𝒓𝒆 as an relative error and 𝑳𝒂𝒃 as a absolute error, 𝑳𝒄𝒂𝒍 isdefined as follows:

𝑳𝒄𝒂𝒍 = λ𝑟𝑒 𝐿𝑟𝑒 + λ𝑎𝑏 𝐿𝑎𝑏
𝑦−𝑔
𝐿𝑎𝑏 = 𝑦 − 𝑔
𝑳𝒓𝒆 =𝑔𝑦 is an estimated food calorie.
𝑔 is ground-truth.

We use the network for food calorie estimation.

Miso soup 102kcal(ES)Nikujaga 417kcal(ES)Potato salad 169kcal(GT)Miso soup74kcal(GT)Nikujaga352kcal(GT)Gratin 456kcal(ES)Curry 483kcal(ES)Potato salad 175 kcal(ES)Cold tofu 76kcal(ES)Gratin 482kcal(ES)Stew 270kcal(ES)Nikujaga 387kcal(ES)Potato salad 169kcal(GT)Cold tofu
95kcal(GT)Nikujaga
352kcal(GT)Gratin 450kcal(GT)Stew 382kcal(GT)Potato salad 147kcal(ES)Cold tofu 89kcal(ES)Spaghetti 536kcal(ES)2. Single-task CNN : Ege and Yanai [2] 2017 ( Calorie estimation Net )Single-dish Food Photo.

Pseudo-bounding boxes
for Calorie50 dataset.

•UEC Food-100 dataset [3]with bounding boxes.

Gratin 496kcal(ES)Potato salad 170kcal(ES)Nikujaga 450kcal(ES)Miso soup 106kcal(ES)Gratin450kcal(GT)Spaghetti 518kcal(GT)Curry
761kcal(GT)Potato salad 169kcal(GT)Cold tofu 95kcal(GT)Miso soup 74kcal(GT)Potato salad 169kcal(GT)Gratin450kcal(GT)Nikujaga
352kcal(GT)Examples of dish detection and food calorie estimationfrom multiple-dish food photos. (based on YOLOv2)

Conclusion
 We estimate food calories from multiple-dishes food photos.
 Multi-task learning of food detection and food calorie estimation.

Future work
 Image-based food calorie estimation based on amount of food.
 Construction of large-scale food photos dataset.

[1] W. Liu, D. Anguelov, D. Erhan, C. Szegedy, and S. E. Reed. SSD: single shot multibox detector. CoRR, abs/1512.02325, 2015.
[2] T. Ege and K. Yanai. Simultaneous estimation of food categories and calories with multi-task cnn. In Proc. of IAPR International Conference on Machine Vision Applications(MVA), 2017.
[3] Y. Matsuda, H. Hajime, and K. Yanai. Recognition of multiple-food images by detecting candidate regions. In Proc. of IEEE International Conference on Multimedia and Expo, 2012.
[4] K. Simonyan and A Zisserman. Very deep convolutional networks for large-scale image recognition. In arXiv preprint arXiv:1409.1556, 2014.