Food Category Transfer withConditional Cycle GAN anda Large-scale Food Image Dataset

Food Image Generation using A Large
Amount of Food Images with Conditional
GAN: RamenGAN and RecipeGAN

Daichi Horita Ryosuke Tanno Wataru Shimoda Keiji Yanai
The University of Electro-Communications, Tokyo, Japan

Yoshifumi Ito Wataru Shimoda Keiji Yanai
The University of Electro-Communications, Tokyo, Japan

Method Overview

Method Overview

Multi-Domain Image-to-Image Translation only to food area
-Based on a Conditional Cycle GAN (cCycle GAN) with a large-scale foodimage data collected from the Twitter Stream.

Our network

𝐺𝐶𝐶0 = 1Datasets

𝐶1 = 1-We use Two hundred and thirty
thousand food images of ten kindsof typical Japanese foods, but we
set restrictions on rice bowl.

𝐶2 = 1(1) Collecting labeled image.
(2) Conditional Generative Adversarial Network (cGAN).
(3) Improvement of cGAN with a dish plate discriminator and WGAN-GP.
(4) Training and image generation.

Improved Conditional GAN
Coniditonal GAN-We used WGAN GP[2] as method of GAN

min max 𝑉(𝐷, 𝐺) =𝐺

E𝑥 ~𝑝𝑑𝑎𝑡𝑎 𝑥 𝑙𝑜𝑔𝐷 𝑥 𝑐+E𝑥 ~𝑝𝑧 𝑥 log(1 − 𝐷 𝑥 𝑐 )C [0, 0, 1, …, 0]
Domain SelectOne-hot Vector signal

𝐶𝑛 = 1・・・𝐶𝑛

𝐶0 𝐶1 𝐶2𝐷Discriminator of dish plate outline.
-We prepared an additional discriminator which is trained with oval figure-Single class classifier with complete oval images as positive samples andthe other figure images as negative samples.

Conditional CycleGAN
Our architecture

Experiments
Ramen GAN
-We trained Conditional GAN (cGAN) model with/without an additional discriminatorof dish plates.

-We used StarGAN[1] architecture. Combining loss functions of [2] and [3].
𝑓𝑎𝑘𝑒𝑟𝑒𝑎𝑙′𝐿𝑎𝑐𝑙 = E𝑥,𝑐 −𝑙𝑜𝑔𝐷𝑎𝑐𝑙 (𝑐|𝐺(𝑥, 𝑐))
𝐿𝑎𝑐𝑙 = E −𝑙𝑜𝑔𝐷𝑎𝑐𝑙 (𝑐 |𝑥)𝑟𝑒𝑎𝑙
𝐿𝐷 = 𝐿𝑎𝑑𝑣 + 𝜆𝑎𝑐𝑙 𝐿𝑎𝑐𝑙

Conditional GAN

Improved
Conditional GAN

The 6 kindramen category.

𝑟𝑒𝑎𝑙
𝐿𝐺 = 𝐿𝑎𝑑𝑣 + 𝜆𝑎𝑐𝑙 𝐿𝑎𝑐𝑙 + 𝜆𝑐𝑐𝑙 𝐿𝑐𝑐𝑙

𝐿𝑐𝑐𝑙 = E𝑥,𝑐,𝑐′ | 𝑥 − 𝐺 𝐺 𝑥, 𝑐 , 𝑐 ′ |1Experiments

Recipe GAN We can translate only to food area.
Input

CurryFried
Rice

BeefBowl

white buckwheateel
Chilled meat
Ramen rice
noodle bowl
spaghetti
noodlesfriednoodles-For Recipe GAN, we generated dish images from cooking ingredients.
-Differ from the specific ramen image generation, image generation using recipe
data is difficult due to the diversity of the dataset.

Conditional GAN

Changes in quality due to differences of the number of
total images.

++Disciriminator of dish plate+WGAN-GP

The 10 kind ingredient category.

 The number of images and the quality are proportional

Input 10k / 230k

100k

FULL

10k100k

FULL

Recipes retrieval results with generated image-We show image-based recipe search results
which are retrieved by the generated images.

Future worksConditional GAN+Tomato+Tohu

Improved Conditional GAN+Tohu+poak-Translation with datasets which excluded constraint of food on rice bowl.
-Translation to categories not included in the datasets.
-Generate higher resolution images.

References[1] C.Yunjey et al. StarGAN:Unified Generative Adversarial Networks for Multi-Domain Image-to-Image Translation. CVPR,
2018.
[2] J. Zhu et al. Unpaired Image-to-Image Translation using Cycle-Consistent Adversarial Networks. ICCV, 2017.
[3] A. Odena et al. Conditional Image Synthesis With Auxiliary Classifier GANs. ICML, 2017

References[1] F. Ahmed et al. Improved Training of Wasserstein GANs. NIPS, 2017.
[2] J. Zhu et al. Unpaired Image-to-Image Translation using Cycle-Consistent Adversarial Networks. ICCV, 2017.