Food Category Transfer with Conditional CycleGAN
and a Large-scale Food Image Dataset
Daichi Horita Ryosuke Tanno Wataru Shimoda Keiji Yanai
The University of Electro-Communications, Tokyo

ABSTRACTThis paper describes “food image transformation” based ona conditional cycleGAN[A3] (cCycleGAN) with a large-scale
food image data collected from the Twitter stream. A cCycleGAN is an extension of CycleGAN, which enables “foodcategory transfer” among 10 types of foods and retain the
shape of a given food. We experimentally show that 200 and
30,000 food images with the cCycleGAN enable a very natural food category transfer among 10 types of typical Japanese
foods: ramen noodle, curry rice, fried rice, beef rice bowl,
chilled noodle, spaghetti with meat source, white rice, eelbowl, and fried noodle.

KEYWORDS
Food Image Transformation, Food Category Transfer, CycleGAN, Food Image Generation

1

INTRODUCTION

In recent years, generative adversarial networks (GANs) haveattracted particular attention. A GAN can generate an image that resembles a real image. A human face image dataset
such as CelebA and a numeric character image dataset suchas MNIST have been used for training the GAN as the target domains. In addition, recently [3] proposed a new task ofstyle transfer for clothes using GAN. Meanwhile, none havereported food image generation or transformation using the
GAN thus far. In this work, we propose a novel paradigm onfood image transformation to convert a given food image toanother category of food image automatically.
Our objective is to create a system that takes a food image and a food category to be transferred as inputs, andsubsequently outputs a new food image that corresponds to
the given food category. We propose a food category transfer
method by extending the CycleGAN, which converts an image into another domain image. To generate realistic images,
the number of training images is key. We gathered 230,000
food images consisting of 10 types of food categories fromTwitter streams for the food image transformation. We continuously gathered images from the Twitter stream for morethan eight years, and we mined the images corresponding
to any of the 10 food categories to create a large-scale food
Permission to make digital or hard copies of all or part of this workfor personal or classroom use is granted without fee provided that
copies are not made or distributed for profit or commercial advantage and that copies bear this notice and the full citation on the first
page. Copyrights for components of this work owned by others than
the author(s) must be honored. Abstracting with credit is permitted.
To copy otherwise, or republish, to post on servers or to redistributeto lists, requires prior specific permission and/or a fee. Request permissions from permissions@acm.org.
CEA/MADiMa’18, July 15, 2018, Mässvägen, Stockholm, Sweden© 2018 Copyright held by the owner/author(s). Publication rights
licensed to ACM.
ACM ISBN 978-1-4503-6537-6/18/07. . . $15.00
https://doi.org/10.1145/3230519.3230597

photo dataset for the food category transfer. We show thatit enables high-quality mutual transformation on a food domain with a conditional CycleGAN (cCycleGAN). In addition, we show that the number of training images is important to obtain more realistic images.

2

RELATED WORKWith the standard GAN, we cannot control the categoryof the generated images explicitly because the GAN uses
only a noise vector v sampled from a uniform distribution or
normal distribution as a seed. Meanwhile, with conditional
GAN (cGAN), which is an extension of GAN by adding cognitional inputs, we can control the category of generated
images by providing a conditional vector in addition to arandom noise vector. On the contrary, the cGAN cannot
transfer an image to another image because the model does
not have an encoder that converts an input image to a hidden representation. Pix2Pix [2] is an extension of the cGAN,
which uses an image as a conditional vector. In Pix2Pix, an
encoder–decoder network is used instead of a generator thatgenerates an image from a seed vector. Because an encoder–decoder network uses an image as an input and outputs atransformed image, image transformation can be performed.
To train the Pix2Pix network, many paired samples of raw
images and the corresponding transformed images are required for training.
Zhu et al. proposed a method to train an image transformation network using unpaired training samples consisting
two domains of image samples such as color images and thecorresponding grayscale images [9]. They introduced a cycle consistency loss for training, and successfully trained animage transformation network that transforms an originaldomain image to the other-domain image while retaining therough shape structure. We herein use a cycle consistency loss
to train a model. With this loss function, in the food domain,
we can transfer a food image to the other food image category while retaining the original food image structure.

3

IMAGE TRANSFORMATION USING
CONDITIONAL CYCLEGAN

In this section, we review two recent image transformation
methods, Pix2Pix [2] and Cycle GAN [9]. Subsequently, we
describe the cCycle GAN that we used in this study.

3.1

Pix2Pix

Isola et al. proposed Pix2Pix [2], which is an extension of
the conditional GAN. Prior to this paper, only the L2 meansquare loss was used for training an encoder–decoder-based
image transformation network, which was unable to transform images between diﬀerent domains clearly. In Pix2Pix,
they proposed to use an adversarial loss in addition to theconventional L1 loss function for training an encoder–decoder

Daichi Horita et al.

CEA/MADiMa’18, July 15, 2018, Mässvägen, Stockholm, Sweden
network. This can be regarded as an image-conditioned version of the cGAN. This enabled a between-domain imagetransformation by the CNN. For example, it can transformedge images into color drawings.
In Pix2Pix, Eq.1 is used as an adversarial loss, and Eq.2
as an L1 normalization term. Eq.3 shows the loss function of
Pix2Pix that is minimized for training a generator and maximized for training a discriminator. The Pix2Pix requires
paired samples that consist of an original image and thecorresponding image that is transformed from domain A todomain B. Although this setting is possible for a pair of edge
images and color drawings where the image of one domain
can be easily converted into the corresponding image of theother domain, it is impossible for category conversion, suchas for a pair of horse images and zebra images.
LcGAN (G, D)=

LL1 (G)
G∗

=
=

3.2

Lccl

=

′

Ex,c,c′ [||x − G(G(x, c), c )||1 ]
′

(7)

=

E[− log Dacl (c |x)](8)=

Ex,c [− log Dacl (c|G(x, c))]

(9)

LD

=

LG

=

Ladv + λacl Lrealacl
ake
Ladv + λacl Lfacl+ λccl Lccl

(11)

Lreal
acl
ake
Lfacl

(10)

where λccl and λacl are the weight biases for the auxiliary
classifier loss.

Ex,y [log D(x, y)] +(1)
Ex,z [log(1 − D(x, G(x, z)))]Ex,y,z [||y − G(x, z)||1 ](2)
arg min max LGAN (G, D) + λLL1 (G)G

D

CycleGAN

While Pix2Pix[2] requires many pairs of sample images of
two domains, which are sometimes diﬃcult to be prepared,
the Cycle GAN [9] has solved this problem. It requires only
unpaired samples.
We denote two types of domain space as X and Y . We represent the mapping X → Y as G and the inverse mapping as
F . The discriminator for domain Y is DY , and the discriminator for domain X is DX . The loss is defined as Eq.6 using
Eq.3 and Eq.4. Eq.3 is the general loss for the adversarial
network. Eq.4 is called the cycle consistency loss. Here, wedenote ŷ(x) as the generated image from x and x̂(ŷ) as thegenerated image. The cycle consistency loss constrains thevalue of x to be x̂(ŷ). If we minimize this loss, the converted
results by G(F (x)) retains the information for reconstruction. Hence, we can obtain a map that converts images belonging to domain X to images belonging to domain Y , andthe converted images retain their original image structure.
LGAN (G, DY , X, Y )=Lcyc (G, F )=+L(G, F, DX , DY )3.3

to the approach of [1]. The discriminator of the cCycleGAN
classifies not only a real or fake but also the category of theimages. By the discriminator, a multiclass generator can betrained. Finally, the loss of cCycleGAN is represented by thefollowing equation:

Ey∼pdata (y) [log DY (y)] +(3)
Ex∼pdata (x) [log(1 − DY (G(x))]
Ex∼pdata (x) [||F (G(x)) − x||1 ] (4)Ey∼pdata (y) [||G(F (y)) − y||1 ](5)=

LGAN (G, DY , X, Y ) +LGAN (F, DX , Y, X) +λLcyc (G, F )(6)

Conditional CycleGAN

We show the network of the cCycleGAN in Fig.1, which isa conditioned extension of the CycleGAN. The cCycleGAN
can convert an image to another image that belongs to theselected category by adding a conditional input to the image transformation network of the CycleGAN [9]. To use aconditional vector eﬀectively, in the cCycleGAN, we added
an auxiliary classifier Loss [6] to the discriminator, similar

Figure 1: Architecture of the Conditional CycleGAN.

4 EXPERIMENTS
4.1 Dataset
By adding the cycle consistency loss, we can generate an image that retains the original image structure. Therefore, inthis experiment, we imposed a constraint that uses images
that have only the structure “bowl,” such that the corresponding structure prompts the training of the cycle consistency loss. In fact, we selected 10 types of categories related
to “bowl” foods from UECFOOD-100 [5]. We gathered theimages from the large-scale food image dataset [8], which
was created by mining food images from the Twitter stream
for more than eight years continuously. We sorted the images in the dataset [8] using the confidence scores obtained
from a food classifier model, which was trained with the
UECFOOD-100 dataset [5]. We show the 10 bowl food categories and the number of selected images from the re-ranked
images in Table 1. In particular, for the “ramen” class, a
specific ramen class that is a very big ramen exists, called“Jiro ramen” in Japanese. The ramen is far from the typicalramen in terms of appearance. Mixing significantly diﬀerent appearances in the same category may negatively aﬀect
the GAN training. Therefore, we removed this class by clustering using a model pretrained with ImageNet. In fact, weused the VGG16 model of the compressed feature on the fc6
layer. It is noteworthy that we used k-means as a clustering
method and the cluster number is eight. We removed theimages belonging to the big “Jiro” ramen cluster. We separated all the selected images into a train set and a test set.
The ratios of the train set and test set are 90% and 10%,

Food Category Transfer with Conditional CycleGAN
and a Large-scale Food Image Dataset
respectively, retarding the total amount of the 10 types ofbowl food images.
Table 1: training data
category
chilled noodles
meat spaghetti
buckwheat noodle
ramen
fried noodles
white rice
curry rice
beef bowl
eel bowl
fried rice
TOTAL

4.2

image number
13,4997,138
3,530
74,007
24,76021,32434,216
18,396
5,329
27,854230,053

Network and Training Setting

We followed [9] for the network of the cCycleGAN. The generator is the same as FastStyleNet [4], which is added toseveral residual blocks of the conv–deconv network. The input image size is 256 × 256. As a conditional vector, we useda one-hot vector. After broadcasting the conditional vector
to input the image size, we concatenated it with an input
image in the middle of the encoder part. As a discriminator, we used PatchGAN [7]. After updating the discriminatorfive times, we updated the generator once. We used NVIDIA
Quadro P6000 for the training, in which the bath size is 32,
the optimization method is Adam, and the iteration epoch
is 20. During testing, we generated images with 512 × 512
resolution.

4.3

CEA/MADiMa’18, July 15, 2018, Mässvägen, Stockholm, Sweden(2) 10000 image per category: 100,000 images.
(3) All the images shown in Table1: 230,053 images.
In Fig.3 and Fig.4, we show the results obtained from themodel trained with diﬀerent number of images. The leftmost images are the input images, and the remaining six
images are generated images. The transformed images are
separated into two blocks by the food categories used for theconditional vector. In each block, we used 10,000 images forthe first column, 100,000 images in the second column, and230,000 images in the third column, for training. The generated image quality becomes better as the number of training
images increases. Although we obtained acceptable resultsby the model trained with a small training set, the details
are not reconstructed. In Fig.3, the results that are transformed to the “chilled noodle” category are shown in the second block. As shown in Table1, a small margin on training
image number exists between the subset of the second row
and third row in the “chilled noodle” category. However, thethird column results of the “chilled noodle” category showshigher quality than the second column results. We suspectthat the generator learned additional information from theother category domain; consequently, the generated image
quality becomes better with a small number of training images.

Results of Food Image Transformation

We show the results by the proposed method in Fig.2. The
leftmost image is the input image and the other 10 images
are the transformed images of each of the 10 categories, respectively. Our proposed method can clearly transform one
certain category of an input to any of the other 10 food categories. We transformed the given food images to the otherfood categories of images while retaining the shape structureof the cycle consistency loss. This means that the generator
trained the concept of “bowl.” In addition, the generator
Figure 3: The leftmost images are the input images.
generated an image that did not only fool the discriminator
The remaining six images are separated into two
but also minimized the classification error of the discrimblocks. The left blocks show the results of “white
inator by the auxiliary classifier loss. We consider that therice” and the right blocks show the results of “chilledauxiliary classifier loss is also beneficial for generating higher
noodle.” In each block, from left to right, we show
quality images than the typical GAN. The images generated
the generated images trained with 10,000 images,
using the auxiliary classifier loss do not contain blurs that
100,000 images, and 230,000 images, respectively.
frequently appear in a simple GAN model. Additional results
are shown at https://negi111111.github.io/FoodTransferProjectHP/.

4.4

Relation Between Image Quality and
the Number of Training Images

We show the food image transformation results, when weused a smaller dataset for training the model. Here, we prepare the following three types of subsets of the dataset.
(1) 1000 image per category: 10,000 images.

5

CONCLUSIONS

Herein, we proposed a novel paradigm to transform a foodimage to another category of food image automatically using
the convolutional neural network. We achieved the following
results by adapting the cCycleGAN, which is an extended
version of the CycleGAN.

Daichi Horita et al.

CEA/MADiMa’18, July 15, 2018, Mässvägen, Stockholm, Sweden

Figure 2: The leftmost images are the input images, and the remaining images are generated images witheach of the 10 categories.
As future work, we plan to evaluate our method quantitatively as an extension to our qualitative results herein.
Objective experiments are required that involve evaluation
scores using the classification accuracy of the generated images such as the inception score.
Acknowledgements: This work was supported by JSPS
KAKENHI Grant Number 15H05915, 17H01745, 17H05972,
17H06026 and 17H06100.

REFERENCES

Figure 4: The leftmost images are the input images. The remaining six images are separated into
two blocks. The left blocks show the results of “beefbowl” and the right blocks show the results of “curryrice.” In each block, from left to right, we show thegenerated images trained with with 10,000 images,
100,000 images, and 230,000 images, respectively.

(1) A food category transfer that retains the shape structure.
(2) Improvement in the quality of food image transformation by mining a large number of training samples of
the corresponding categories from the Twitter stream.

[1] Y. Choi, M. Choi, M. Kim, J. Ha, S. Kim, and J. Choo. 2017.
StarGAN: Unified Generative Adversarial Networks for MultiDomain Image-to-Image Translation. arXiv:1711.09020.
[2] P. Isola, J. Zhu, T Zhou, and A. A. Efros. 2017.
Image-to-Image Translation with Conditional Adversarial Networks.
In Proc. of IEEE Computer Vision and Pattern Recognition.
[3] S. Jiang and Y. Fu. 2017. Fashion Style Generator. In Proc. of
the Twenty-Sixth International Joint Conference on Artificial
Intelligence.
[4] J.
Johnson,A.
Alahi,
and
L.F.
Fei.
2016.
Perceptual Losses for Real-Time Style Transfer and Super-Resolution.
In Proc. of European Conference on Computer Vision.
[5] Y.
Matsuda,H.
Hoashi,andK.
Yanai.
2012.
Recognition of Multiple-Food Images by Detecting Candidate Regions.
In Proc. of IEEE International Conference on Multimedia andExpo.
[6] A. Odena, C. Olah, and J. Shlens. 2017. Conditional Image Synthesis With Auxiliary Classifier GANs. In Proc. of the 34th International Conference on Machine Learning.
[7] D. Pathak, P. Krähenbühl, J. Donahue, T. Darrell, and A.
Efros. 2016. Context Encoders: Feature Learning by Inpainting.
In Proc. of IEEE Conference on Computer Vision and Pattern
Recognition.
[8] K. Yanai and Y. Kawano. 2014. Twitter Food Image Mining andAnalysis for One Hundred Kinds of Foods. In Proc. of Pacifit-Rim
Conference on Multimedia (PCM).
[9] J. Y. Zhu, T. Park, P. Isola, and A. A. Efros. 2017. Unpaired
Image-to-Image Translation using Cycle-Consistent Adversarial
Networks. In Proc. of IEEE International Conference on Computer Vision.