Unseen Food Creation by Mixing Existing
Food Images with Conditional StyleGAN
MADiMa on ACMMM workshop
Oct 21st, 2019
Nice, FranceDaichi Horita, Wataru Shimoda and Keiji Yanai
UEC Tokyo, Japan.

1Unseen Food Creation by Mixing Existing
Food Images with Conditional StyleGAN
MADiMa on ACMMM workshop
Oct 21st, 2019
Nice, France
Daichi Horita, Wataru Shimoda and Keiji Yanai
UEC Tokyo, Japan.
21. Motivation● GANs show remarkable success in various tasks such asimage generation and image translation and have been
applied to food image generation.

RamenGAN[Ito+ MADiMa18]

R2GAN[Zhu+ CVPR19]!3

1. Motivation● However, both GANs fails to generate realistic food images.
● The quality is still low and it is difficult to generateappetizing and delicious-looking images.

RamenGAN[Ito+ MADiMa18]

R2GAN[Zhu+ CVPR19]!4

1. Motivation● StyleGAN[Karras+ CVPR19] is one of the state-of-the-art
GANs in an unsupervised manner.
● However, it is difficult to control the latent space…
● We extend the generator to manipulate the latent space!5

1. Motivation● Thanks to introduce conditional vectors, our proposed model
can generate the specific food and create the unseen foodimages by mixed multiple kinds of foods!

Images generated by one condition.

Images generated by mixing conditions.

!62. Approach - Conditional style-based generator

Conditional mapping function!7

2. Approach - Conditional style-based generator!8

2. Approach - Conditional style-based generator!92. Approach - Conditional style-based generator!10

2. Approach - Conditional style-based generator!11

2. Approach - Conditional style-based generator!12

2. Approach - Conditional style-based generator!13

2. Approach - Conditional style-based generator!14

2. Approach - Conditional discriminator● We adopt Projection Discriminator[Miyato+ ICLR18]● cGAN and AC-GANs didn’t progress learning.

Discriminator architecture (This figure is borrowed from [Miyato+ ICLR18])!15

3.1 Experiments - FOOD13 Dataset● The dataset consists of 13 categories and 220k images.
● Test set consists of 1000 images of each category.
Bibimbap

Fried riceBeef bowl

Pork cutlet bowl

Ramen

Eel bowl!16

3.2 Experiments - Manipulation of latent space● Each row of images are generated from the fixed input noise,styles and one condition vectors.
● Our model can generate an arbitrary class of images thanks
to the condition vectors.
Fixed nose, styles

Fixed nose, styles

Difference ofnoise, styles

Difference ofnoise, styles

Unsuccessfully generated samples

Successfully generated samples!17

3.2 Experiments - Manipulation of latent space● Thanks to the same input noises and styles, our model outputs
a plate with the same shape.
Fixed nose, styles

Difference ofnoise, styles

Successfully generated samples!18

3.2 Experiments - Manipulation of latent space● However, although the noises and styles are fixed, someimages have distorted plate because they have few roundsamples.

Unsuccessfully generated samples!19

3.3 Experiments - Manipulation of random noise● Ramen images generated from a fixed style, a condition
vector, and randomly sampled noise.
● Random noise plays a role in expressing differences such as
food topping.
Randomly noise

One-hotvector

LatentcodeDifferenceof styles

Latentstyle!20

Noise3.4 Experiments - Creation of Unseen Food Image● Mixing ramen and other foods.
Ramen+ conditions=!21

4. Ablation Study - Comparison of generator architecture● We show the proposed structure(a) is more effective thanbaseline model(b).
● (b) takes the condition to only the first layer.

!22

4. Ablation Study - Comparison of generator architecture● Mapping function (b) can not control the class without
multiple condition vector inputs.
● In other words, (b) is affected only by noise.

(a)(b)!23(b)4. Ablation Study - Comparison of generator architecture● FIDs is the metric of the distance between test set andgenerated samples.

● FIDs is the lowest when the proposed conditional mapping
network consists of eight FC layers!

!24

5. Conclusion● We propose conditional version of StyleGAN to control
latent space.
● Our model can create the unseen food images by
mixing multiple conditions.
● In the future, we want to produce the healthy andunhealthy food image generation by using theconditions as calories and carbohydrates.

Thank you!!
!25

Q&A!26

How to input the conditions● Sample latent noise[N, 512], (N: Batchsize).
● Add the one-hot condition to the noiseand get the feature [N, 512+C], (C: Numof conditions).
● Input the features [N, 512+C] to FC layer
and get the features [N, 512].
● Repeat.

!27

3.4 Experiments - Creation of Unseen Food Image● There are many red samples of seafood bowl, so the ramen
and seafood bowl are red.
● However, when many conditions are put, the effect faded.

1. Motivation● In addition, thanks to introduce conditional vectors, ourproposed model can create the unseen food images by
mixing multiple kinds of foods!

Images generated by our proposed Conditional StyleGAN.

4. Ablation Study - Comparison of generator architecture● FIDs is the metric of the distance between test set andgenerated samples.
● FIDs is the lowest when the proposed conditional mapping
network consists of eight FC layers!
● FIDs (a) is lower than (b)FIDs of proposed model(a) !30

FIDs of baseline model (b)