AR DeepCalorieCam:An iOS App for Food Calorie Estimation
with Augmented Reality-(-Background & Objective
Meal management apps enable us to record food calories.
Some of them need human help for calorie estimation.

Image-based food calorie estimation
using recipe information with Augmented Reality-,-)My Pet Project Introduction:
Food Image Transfer using
Background: Generative Adversarial Networks(GANs)

GANs are a kind generative models designed by Goodfellow et all in 2014.
In a GAN setup, two differentiable functions, represented by neural networks, are locked in a game.
InputThe two players, the Generator and the Discriminator,
fakeimagenoise
have different roles in this framework.
vector
Generator Player 1
fake100dimorThe Generator tries to produce datarealthat come from some probability distribution.
TrainingGD

set

Discriminator Player 2
The Discriminator, acts like a judge. It gets to decide
if its input comes from the G or from the true training set.
In summary, the game follows with:The Generator trying to maximize the probability of making
the discriminator mistakes its inputs as real.
The Discriminator guiding the generator to produce more realistic images.
In the perfect equilibrium, as a result,
The Generator would capture the general training data distribution.
The Discriminator is always unsure of whether its inputs are real or not.

[AR DeepCalorieCam][DeepCalorieCam]

Method: Multi-task CNN for calorie estimation
Related work

Objective:: Miyazaki et al.[1] 2011

Related work

Multi-task estimation of food categories and food ingredients.
Multi-task CNN[3] of food categories and food ingredients
Multi-task learning improve both task’s performance.

[ Our network ]Food calorie for one person.

fc8a

fc8b

Potato salad
Category recognition

fc6
fc7b

Estimation
target(calories)(2) Food Category loss:

Auxiliary Classifier Loss(ACL):

relative err. loss

absolute err. loss

$10: = − < => log B>[ Our network ][ DATASET ]For training our network,
we collected calorie-annotated food photos fromthe online cooking recipe sites.
Food 15 categories.
A total of 4877 images.

[ Implementation ]CurryGratin

Miso soup

Cold tofu

Sushi bowl

Fried riceStew

Fried noodle

NikujagaOmelet with friedPotato saladrice

Spaghetti[ Calorie Estimation with AR ]896074!5 = $&'()'*+ ,-./ $012'+,../ $112
AdversarialLoss

C[0, 1] Reconstructed
Fake Image
In domain B Domain Select
Image

Food 10 categories.
A total of 230,053 images.

Hamburg steak

Mixed rice

Real Image
In domain B

For training our network,
we collected 10 category food photos
from the Twitter.

Result:Input(1) Train with Keras(backend TensorFlow) framework(2) Convert Keras model to CoreML model for iOS deployment(3) Display calorie estimation result using Apple ARKit framework

Application Demo:3402!" = $&'()'*+,$-./ 012%

fake or real

Auxiliary Classifier
Lossclass[1, 0][ DATASET ]Pilafclass[0, 1, …, 0]

Total loss :D

B is an estimated food calorie.
= is ground-truth.

>DE

Auxiliary Classifier
Loss[ Loss function ]!-I = B − =(softmax cross entropy)

DEvery generated sample has a corresponding class label

C[1, 0]Real Image
In domain A Domain Select( λ is the weight on the loss. )CJC = 1realimage

89$102 = λGH !GH + λ-I !-I
B−=!GH ==JC

C [0, 0, 1, …, 0] JCNE = 1Domain SelectOne-hot Verctor signal

Cycle Consistency Loss

It is expected to improve the accuracy of each task.

(1) Calorie estimation loss:

JK JE JL

Capture the intuition that if they translate from one domain to the otherand back again they should arrive at where they started.
Related work : ACGAN[4] ICML2017
Adversarial
New methods for the improved training of GANs for image synthesis.
LossfakeThey introduce a Auxiliary Classifier Lossfake or real
imageto make high quality image .

Conv layers ( VGG16 [4])[ Loss function ]JL = 1

Cycle Consistency Loss(CCL):

We use multi-task CNN for calorie estimation.

fc7a

JE = 1

Related work : CycleGAN[3] ICCV2017
An approach for learning to translate an image
from a source domain X to a target domain Y.
They introduce a Cycle Consistency Lossto push F(G(X))≈X (and vice versa).

: Chen and Ngo[2] 2016

335 kcalCalorie estimation

Output89

Method: conditional CycleGAN

We propose regression-based method using CNN.

JK = 1Input

Transfer food images to multiple domains
with high quality using the GAN methodfor dietary images

Search-based food calorie estimation with conventional features.
Similar image retrieval with SURF and color histograms and so on.
Calculate food calories from retrieved images’ calories.

realimage-CurryFried Rice Gyudon

CurryFried Rice

GyudonHiyashiChuka

34,216
Ramen

27,854
Rice

18,396
Soba

13,499Unagi

7,138
Yakisoba

74,007

24,760

3,350

5,329

24,760

Rice

Soba

Hiyashi
Chuka Meet Supa Ramen

Meet SupaUnagi

Yakisoba[Food Image Transfer using GANs]

Conclusions[ Calorie Estimation with AR ]••We proposed food calorie estimation app with a multi-task CNN using Augmented Reality.
Multi-task learning improved both food calorie and category estimation.

[Food Image Transfer using GANs]••

We proposed food image transfer using conditional CycleGAN.
Conditional CycleGan can convert multiple domains while keeping the shape of the food.

[1] T. Miyazaki, G. Chaminda, D. Silva, and K. Aizawa. Image - based calorie content estimation for dietary assessment. In Proc. of IEEE ISM Workshop on Multimedia for Cooking and Eating Activities, 2011.
[2] J. J. Chen and C. W. Ngo. Deep-based ingredient recognition for cooking recipe retrival. In Proc. of ACM International Conference Multimedia, 2016.
[3] J. Y. Zhu, T.Park, P. Isola, A.A. Efros, Unpaired Image-to-Image Translation using Cycle-Consistent Adversarial Networks. In Proc. of IEEE International Conference on Computer Vision, 2017.
[4] A. Odena, C. Olah, and J. Shlens. Conditional Image Synthesis With Auxiliary Classifier GANs. In Proc. of the 34th International Conference on Machine Learning, 2017.
[5] I. Gulrajani, F. Ahmed, M. Arjovsky, V. Dumoulin, and A. Courville, Improved Training of Wasserstein GANs. Advances in Neural Information Processing Systems, 2017.