AR DeepCalorieCam V2: Food Calorie Estimation
with CNN and AR-based Actual Size Estimation

D41-1

Ryosuke Tanno, Takumi Ege and Keiji Yanai( The University of Electro-Communications, Tokyo )Introduction
Many recording meal apps
⇒ Semi-automatic calorie estimation- need to teach food amounts manually [ex. FoodCam]- need to register a size-known reference object
in advance [ex. CalorieCam ]FoodCam [Kawano et al. 2014]

Need to teach the amountof foods by the slider

Previous Method
Ex. CalorieCam [Okamoto et al. 2016]⇒ Reference object is absolutely necessary
Take a photo witha reference object

’Extract a regionof a ref. object

pre-registered
size: 46.8 08 2pixels : 43149

Extract regionsof food items

Food category and Calorie value

No reference object required

Recognize a categoryof each food item

Directly calculate regionsof food items using AR Food classificationa dish region based on edges
Detect a bounding box ofa food region by k-means

Detect an accurate foodregion by GrubCutsize vs calorie

The same way fora reference object

 Size-based calorie estimation
Size-based estimation- Compare #pixels of a food region with #pixels
of a reference object and estimate a real size"#$%&''( = #+#,&''( ∗ #+#,.%& /"#$%.%&- Use quadratic curve based estimationfrom "#$%&''( to a food calorie, 012 .
2
012 = 1# ∗ "#$%&''(+ 6# ∗ "#$%&''( + 0#
where 1# , 6# , 0# are trained
params for #-th categories. Beef
Kinpirabowl

 Food Category Recognition- Convolutional Neural Networks (CNN)
based food recognition engine employs “Inception-v3[4]”- light memory, faster inference and highclassification accuracy- fine-tune a pre-trained ImageNet model in Keras DeepLearning framework with UEC-FOOD100 dataset
 AR-based Actual Size Estimation
Top-1: Pizza- Measure the actual size of the meal area using Apple ARKit framework- acquiring the coordinates on the real world as a three-dimensional vector- calculate food calories based on their actual size and food categories
Evaluation of automatic calorie estimation

Demo 1Take a photoFood Calorie Estimation with CNNand AR-based Actual Size Estimation⇒ No need a reference object
 Measuring the meal area directly⇒ Calculate the size more accurately
than in the previous method

GrubCut-based region extraction CNN-based mobile food recognition engines- “DeepFoodCam” engine
with edge-based seeds- Standalone, no need of a recognition server- Network-in-network (NIN) based CNN
Provide a meal photo- “Real” real-time recognition (27ms iniPhone7)
Detect a bounding box of(Please see “DeepFoodCam” demo for detail !)Final output :Kinpira burdock 74.4 kcal

Proposed Method

New system: AR DeepCalorieCam V2

Need to preparea reference object

 Extraction of regions

Recognize a category of each detected food
Food region
Comparesize: ??? 08 9
both region sizes
pixels : 68066

Calculate food calories based on theirrelative size and food categories

CalorieCam[Okamoto et al. 2016]

CNN

Pizza region
Size: 84.6 08 9

Calculate food calories based on theiractual size and food categories

Food category and Calorie value

DishBeef bowl
Croquette
SaladConclusions- Propose a new approach for food calorie estimation with CNN andAR-based actual size estimation

FoodCamCalorieCam
Proposed
GT Avg. err
Avg.SDAvg. err
Avg.SDAvg. err
Avg.SD
962-53.25 ±209.79-242 ±55.10-67.14±18.8
552-242±91.26-47.08 ±52.52-127.0
±9.0
14
54.83
±36.284.86 ±11.87
-0.95
±0.16

Future Works- Combines the proposed method and segmentation,
taking into account the size of the meal area