Ketchup GAN: A New Dataset forRealistic Synthesis of Letters on Food
Gibran Benitez-Garcia, and Keiji Yanai
The University of Electro-Communications
MMArt-ACM ‘21
1UEC Yanai Lab. Tokyo.

2021.11.16ⓒ 2021 UEC Tokyo.

Motivation– Food image generation and translation have been attracted particularattention due to the outstanding performance of GAN networks– To train GAN models, a large amount of real-world images is needed– There is no GAN-based model proposed for food image decoration

PizzaGAN** "How to make a pizza: Learning a compositional layer-based GAN model," CVPR 2019
UEC Yanai Lab. Tokyo.

ⓒ 2021 UEC
Tokyo. 2019
MADiMa2 ** "Unseen Food Creation by Mixing Existing Food Images with Conditional StyleGAN,"FoodGAN**

Objective• Objective:– Introduce a new dataset for realistic synthesis of letters on food• Approach:– Gathering real-world images of omelettes decorated with ketchup lettersand generate egg and ketchup masks, and synthetic ketchup-free images

Original photos
3Egg & Ketchup masks
UEC Yanai Lab. Tokyo.

ⓒ 2021 UEC Tokyo.

Synthetic ketchup-freeThe Ketchup GAN Dataset– We gathered omelette rice images with and without ketchup lettersfrom Twitter* in Japan (part of UECFOOD-100 dataset)

Images gathered from Twitter

Geotagged Twitter photo mining from UECFOOD-100
UEC Yanai Lab. Tokyo.

4 * "Large-scale Twitter Food Photo Mining and Its Applications," BigMM 2019ⓒ 2021 UEC Tokyo.

The Ketchup GAN Dataset– We train a CNN classifier with meaningful omelette dishes, five classes
are obtained with 2000 images per class– We train a weakly supervised segmentation* algorithm for obtainingketchup and omelette masks

Early results from weakly supervised semantic segmentationUEC Yanai Lab. Tokyo.

ⓒ 2021
UEC Tokyo.
supervised
semantic segmentation," CVPR 2018
5 * "Learning pixel-level semantic affinity with image-level supervision for weakly

The Ketchup GAN Dataset– Four sets with 1,150 images each conform the complete dataset

Omelette with ketchup
6Segmentation masks

Synthetic ketchup-free
UEC Yanai Lab. Tokyo.

ⓒ 2021 UEC Tokyo.

Original plain omelettePhotorealistic Ketchup Letter Synthesis– We test two GAN models with our dataset
Original– Pix2Pix* trained with ketchup-free
images and segmentation masks.

GeneratedInput
UEC Yanai Lab. Tokyo.

2021 UEC Tokyo.
2017
7 * "Image-to-image translation with conditional adversarial networks," CVPR ⓒPhotorealistic Ketchup Letter Synthesis– SPADE* trained only with segmentation masks.
Original

Masks

Input

SPADE Generator
UEC Yanai Lab. Tokyo.

ⓒ 2021 UEC Tokyo.
8 * "Semantic image synthesis with spatially-adaptive normalization," CVPR 2019

Generated

Photorealistic Ketchup Letter Synthesis– Results from test set of our dataset.

Input
9Pix2pix results
UEC Yanai Lab. Tokyo.

ⓒ 2021 UEC Tokyo.

Original

SPADE resultsPhotorealistic Ketchup Letter Synthesis– Results with customized ketchup patterns.

Pattern10Input

Pix2pix results
UEC Yanai Lab. Tokyo.

ⓒ 2021 UEC Tokyo.

SPADE resultsOmelette Decoration (Application)*UEC Yanai Lab. Tokyo.

ⓒ 2021
UEC Tokyo.ICPR 2020
Convolutions,"
11 * "Fast and Accurate Real-Time Semantic Segmentation with Dilated AsymmetricQualitative results– Comparison with manually edited photo.

12UEC Yanai Lab. Tokyo.

ⓒ 2021 UEC Tokyo.

13

UEC Yanai Lab. Tokyo.

Conclusion• Conclusions:– We introduced a new dataset for the realistic synthesis of ketchup.
– We evaluated two GAN-based methods for ketchup synthesis using ourdataset, and proposed a simple omelette decoration application.
– Our results show that the proposed dataset is suitable for training GANbased methods.

• Future work:– Improve the application by proposing a GAN model capable of aligning theinput texts and synthesizing omelettes with ketchup patterns simultaneously.
14

UEC Yanai Lab. Tokyo.

ⓒ 2021 UEC Tokyo.

Thank You For Your Attention

http://mm.cs.uec.ac.jp/omrice/
15

UEC Yanai Lab. Tokyo.

ⓒ 2021 UEC Tokyo.