Food Category Transfer with
Conditional Cycle GAN anda Large-scale Food Image Dataset
Daichi Horita Ryosuke Tanno Wataru Shimoda Keiji Yanai
The University of Electro-Communications,Tokyo, Japan

Image generation mehtod：GAN• GAN (Generative Adversarial Network)[Goodfellow et al. NIPS 2014]• DCGAN (Deep Convolutional GAN) [Radford etal. ICLR 2016 ]2018/7/14

UEC yanailab 20172

Objective• Keep shapes of food, which is before conversion
Chilled noodles
Like beef bowl

Food imageconverterbeef bowl

soba
Like beef bowl

1. Convert with high quality
2. Convert to multiple foodcategory by one network

cary
Like beef bowl

• TaskDomain transfer(CycleGAN)
2018/7/14

UEC yanailab 20173

CycleGAN• CycleGAN([Zhu+ ICCV-17])
・Cycle GAN is trained with unpaired image set

2018/7/13

UEC yanailab 20174

CycleGAN• Convert input image domain form X to Y by network G.
• Reverse the domain form Y to X by network F.
• minimize the difference between
the input image 𝑥 and reversed image 𝑥ො

Conversion can be learned without pair images
Cycle Consistency Loss2018/7/14

UEC yanailab 20175Limitation of CycleGAN• Conversion is limited to 1 to 1.
→Extend 1 to n

Beef bowl to
Buckwheat
noodleFood image
converter

Beef bowl toCurry

2018/7/13

UEC yanailab 2017

Correspondingfeature“Food”

・・・

Food image
converter

・・・

Beef bowl to
Chilled noodle

・・・

Food image
converter6

Conditional CycleGAN• StarGAN: Unified Generative Adversarial Networks
for Multi-Domain Image-to-Image Translation[Choi+ arXiv-17]• Cycle GAN + AC-GAN

2018/7/13

UEC yanailab 20177

Discriminator for Conditional Cycle GAN• AC-GAN• Conditional Image Synthesis With Auxiliary
Classifier GANs AC-GAN[Odena+ ICML-17]

Food image
converter
Beef bowl
Category selection
vector2018/7/13

UEC yanailab 20178

Network Overview• Network otimizes three types of loss
Consistency Loss

Food image
converter

Food image
converter

Adversarialloss

2018/7/13

UEC yanailab 2017

Auxiliaryloss

9Experiments• We use foods, which have similar dish plates
as target food category for simplification.
Selected 10 kinds of food category.

カレー
Curry

炒飯
Fried
rice

牛丼
Beefbowl

Ramen
ラーメン

Rice
白飯

Buckwheat
蕎麦 noodle

2018/7/13

Chillednoodle冷やし中華

UEC yanailab 2017

Eel bowl
うなぎ

Meat spaghetti
ミートスパ

Fried noodle
焼きそば

10

Experimanetal Data• Total amount:• 230k images• Traing：0.9• Testing：0.1

2018/7/13

Target category

Image number

Chilled noodle

13,499Meat spaghetti

7,138

Buckwheat noodle

3,530

Ramen

74,007

Fried noodle

24,760Rice

21,324Curry rice

34,216

Beef bowl

18,396Eel bowl

5,329

Fried rice

27,854

total

230,053

UEC yanailab 201711

Experimental results• In case of one food included in an image
Input

2018/7/14

CurryFried rice

Beef bowl

UEC yanailab 2017

Chillednoodle

Meat spaghetti

12Experimental results• In case of one food included in an image
Input

2018/7/13

Ramen

Rice

Buckwheat
noodle

UEC yanailab 2017

Eel bowl

Fried noodle

13Experimental results• In case of multiple foods included in an image
Input

2018/7/13

CurryFried rice

Beef bowl Chilled noodle

UEC yanailab 2017

Meat spaghet

14

Experimental results• In case of multiple foods included in an image
input

2018/7/13

ramen

rice

Buckwheat
noodle

UEC yanailab 2017

Eel bowl

Fried noodle

15Demo videio

2018/7/13UEC yanailab 201716

Evaluation using user study

2018/7/14

UEC yanailab 201717

Evaluation

Imagegeneration

Imageconversion

Error rate for generated images

12%

30%

Error rate for generated images

16%

32%

Mean error rate

14%

31%

2018/7/14

UEC yanailab 201718

Conclusion and future work
• Conclusion• We transform a food image to another category of afood image automatically• We adapted conditional CycleGAN which is anextended version of CycleGAN• Future work
• Extend target food categories for conversion.
• Arbitrary category.

2018/7/14

UEC yanailab 201719