ICMR

Region-Based Food Calorie Estimation
for Multiple-Dish Meals
電気通信大学岡本 開夢<okamoto-ka@mm.inf.uec.ac.jp>

足立 賢人<adachi-k@mm.inf.uec.ac.jp>

柳井 啓司<yanai@cs.uec.ac.jp>

Out of 34

1

1.Background：Relationship between food area estimation and calorie

There is a big relationship between food area and calories.

Food area

Calorie

59.4𝑐𝑚289kcal

115.1𝑐𝑚2

177kcal160.0𝑐𝑚2354kcal
Out of 34

2

1.Background：Relationship between food area estimation and calorie

There is a big relationship between food area and calories.

Food area

Calorie

59.4𝑐𝑚289kcal

115.1𝑐𝑚2

177kcal160.0𝑐𝑚2

354kcalSegmentation is essential
for calorie estimation
Out of 34

3

1.Background：Relationship between food area estimation and calorie

To estimate calorie:⇒actual size and calorie value are needed

Calorie-Size relation
shrimp with chili sauce400

ref-object

calorie(kcal)

350

food area

300
250
200
150
10050

Food category

Food area

shrimp with chili sauce

115𝑐𝑚2

Calorie
177kcal002040

60

80

100

120

140160Food area(cm^2)

Out of 34

180

1.Background：Relationship between food area estimation and calorie

In the case of meals containing multiple item, each category need arelationship of food area and calorie content.
RiceFried chicken
1000

450calorie(kcal)

calorie(kcal)

400

800

350
300

600

250
200

400

150
100

200

500
-50 0

20

40

60

80

Food area(cm^2)

1000

Food area(cm^2)
0100

200

Out of 34

300

5

1.Background：Relationship between food area estimation and calorie

In the real situation, nutrition labeling shows only total calorie content

Calorie for each food item
is not provided

http://www.pref.kyoto.jp/shoku-kawaraban/hyouji02-8.html

Out of 34

6

2.Purpose for research

Purpose for fesearch
Estimate the calorie contents of each food category from single image
training with known total calories.

Long, Jonathan, Evan Shelhamer, and Trevor Darrell. Fully convolutional networks for semantic segmentation. In Proc. of the IEEE conference on computer vision and pattern
Out of 34
recognition. 2015 から図を引用

7

3.Literature review：Estimating calorie content of multiple items

Ege et al.[1]⇒ estimate calorie from each food item using detection model.

T. Ege and K. Yanai. Estimating food calories for multiple-dish food photos. InProc. of Asian Conference on Pattern Recognition (ACPR) 2017図を引用

Out of 34

8

3.Literature review：Estimating calorie content of multiple items

Ege et al[1]⇒ estimate calorie from each food item using detection model.

Each food item need individualcalorie data
T. Ege and K. Yanai. Estimating food calories for multiple-dish food photos. InProc. of Asian Conference on Pattern Recognition (ACPR) 2017図を引用

Out of 34

9

4.Methodology
1.Dataset
2.Model• Segmentation part• Calorie estimation partOut of 34

10

5. Methodology ：Dataset

Requires total calorie content dataset with mask image⇒Collect data from the Inzai city’s Lunch Center website.

http://inzai.ed.jp/kyusyoku/?page_id=32から引用
Out of 34

11

5. Methodology ：Dataset

Create a dataset by adding a mask image using the annotation tool•

474 learning images and 119 evaluation images•

60 different meal categories

Out of 34

12

5. Methodology ：Dataset

Number of images for each category in the dataset
Top 5 categories

Food category
MilkMiso-soupJapanese salad

RiceGreen saladImages593321273265
225

Bottom 5 categories
Food category
Spring rollFried noodles
Fried smelt fishOden
SausageImages
55542Out of 34

13

5. Methodology ：Model

DeepLab V3+ as calorie estimation model⇒Adding calorie estimation branch

Adding calorie estimation
branch here

Out of 34
Encoder-decoderwith atrous separable convolution for semantic image segmentation. In Proc. of European Conference on Computer Vision, 2018から図を引用、改変

14

5. Methodology ：Model

Consists of Segmentation part and Calorie estimation part

Calorie Branch
3x3 Conv

One-hot
Multi calorie vector
AveragepoolingSum
pooling

1：136kcal
2：200kcal
3：220kcal：

Out of 34
Encoder-decoderwith atrous separable convolution for semantic image segmentation. In Proc. of European Conference on Computer Vision, 2018から図を引用、改変

15

5. Methodology ：Model Overview

Consists of Segmentation part and Calorie estimation part

Calorie Branch
3x3 Conv

Two parts aretrained separately

One-hot
Multi calorie vector
AveragepoolingSum
pooling

1：136kcal
2：200kcal
3：220kcal：

Out of 34
Encoder-decoderwith atrous separable convolution for semantic image segmentation. In Proc. of European Conference on Computer Vision, 2018から図を引用、改変

165. Methodology : Segmentation partTrain Deeplab v3+ using our dataset⇒mIoU 0.48 Acc 0.61

Out of 34

17

5. Methodology : Segmentation partTrain Deeplab v3+ using our dataset⇒mIoU 0.48 Acc 0.61

Freeze params in order to train
calorie estimation partOut of 3418

5. Methodology : Calorie estimation part

Apply conv. to convert image feature into calorie feature map 𝑇𝑐𝑎𝑙𝑜𝑟𝑖𝑒⇒corresponding to calorie amount on each pixel
カロリー量特徴マップ𝑻𝒄𝒂𝒍𝒐𝒓𝒊𝒆[c(60) , w(191), h(191)]

画像特徴量

3x3 Conv

𝐶(60 categories)×W(191)×H(191)Same size as segmentation map

Out of 34

19

5. Methodology : Calorie estimation part

Convert segmentation map into one-hot tensor 𝑇𝑜𝑛𝑒−ℎ𝑜𝑡⇒corresponding class : 1 , other : 03x3 Conv

UpSampleby4One-hot

One-hot label tensor
𝑻𝒐𝒏𝒆−𝒉𝒐𝒕[c(60) , w(191), h(191)]
Out of 34

20

5. Methodology : Calorie estimation part

Convert segmentation map into one-hot tensor 𝑇𝑜𝑛𝑒−ℎ𝑜𝑡⇒corresponding class : 1 , other : 0

𝑻𝒐𝒏𝒆−𝒉𝒐𝒕
3x3 Conv

UpSampleby4One-hot

One-hot label tensor
𝑻𝒐𝒏𝒆−𝒉𝒐𝒕[c(60) , w(191), h(191)]
Out of 34215. Methodology : Calorie estimation part

Hadamard product 𝑇𝑐𝑎𝑙𝑜𝑟𝑖𝑒 and 𝑇𝑜𝑛𝑒−ℎ𝑜𝑡 ⇒Average pooling ⇒Sum pooling⇒Caliculate calorie vector 𝑣𝑐 corresponding to calorie for each category
Multi calorie 𝒗𝒄⨂Calorie feature map
𝑻𝒄𝒂𝒍𝒐𝒓𝒊𝒆[60 ,191, 191]

Averagepooling

One-hot label tensor
Input size𝑻𝒐𝒏𝒆−𝒉𝒐𝒕[60 , 191, 191]
[60 , 191, 191]

Sumpooling

Input size
[60 , 25, 25]
Out of 3422

5. Methodology : Loss function

Weighted absolute error and relative error|𝑣 − 𝑔|𝜆 = 1.0(𝑣 − 𝑔 ≥ 0)
𝐿𝑎𝑏 = 𝜆 𝑣 − 𝑔 ቊ𝐿𝑟𝑒 =𝜆 = 1.2(𝑜𝑡ℎ𝑒𝑟𝑤𝑖𝑠𝑒)
𝑔Total loss function

𝑳𝒄𝒂𝒍 = 𝝀𝒂𝒃 𝑳𝒂𝒃 + 𝝀𝒓𝒆 𝑳𝒓𝒆•𝑣 is the estimated total calorie of each food category•

𝑔 is ground truth total calorie•

𝜆𝑎𝑏 (=0.01) 𝜆𝑟𝑒 (=0.1) are hyper parameter weighting each loss.
Out of 34

23

6. experiment

Estimate …
• total amount of calories• calories of each food items
in school lunch photos.

Out of 34

24

6. experimentComparative experiments in three models• Model A：directly estimate total calorie• Model B：Model A + category label•

Train multi-label classification model (VGG16) ⇒ add calorie estimate branch (FC) after conv.

• Model C (ours)：segmentation + calorie estimation• Ege et al[1]：ditection ⇒ calorie regression
for each food item

Nash, Will, Tom Drummond and N. Birbilis. A review of deep learning in the study
Out of 34of materials degradation. npj Materials Degradation 2 (2018): 1-12.から図を引用

25

6. experiment

Absolute error on total calorie amounts (kcal)

Error in each category
has been accumulated
Out of 3426

6. experiment

Average estimated calorie amounts of common 13 category food items in 60
categories (kcal).

Out of 34

27

6. experiment

Average estimated calorie amounts of common 13 category food items in 60
categories (kcal).

Out of 34

28

6. experiment

Average estimated calorie amounts of common 13 category food items in 60
categories (kcal).

Our Model C is the most reasonable for
estimating calorie in each food category
training from total calorie.
Out of 34

29

6.実験結果：モデルCを用いた推定結果画像

Out of 34

31

6.実験結果：モデルCを用いた推定結果画像

We can estimate individual food calorie
training from total calorie
Out of 34

32

7.Future works

Experiment is still small scale, not generic.
• Dataset expansion• Cost reduction for annotating segmentation dataset

Out of 34

33

8.Conclusion

Conclusion• Proposed method that estimate calorie for each food category training
from total calorie.
• Adding a calorie estimation branch to semantic segmentation model.
• In experiment, proposed method can estimate the calories of multiple-dishschool lunch dataset in high accuracy.

Out of 34

34