Text-based Image Editing for Food Images with CLIP
Kohei Yamamoto

Keiji Yanai

The University of Electro-Communications, Tokyo, Japan{yamamoto-k,yanai}@mm.inf.uec.ac.jp

Figure 1: Example results of food image manipulation by texts. The leftmost column showed the original input images: chahan(fried rice in Japanese) and steamed rice. The second to sixth columns from the left showed the manipulated images by
VQGAN-CLIP. The prompt used in each manipulation combined the food name and “with” a topping name. For example, thetwo images in the second column are generated with the prompts, “chahan with egg” and “rice with egg” respectively.

ABSTRACT
Recently, the large-scale language-image pre-trained model, suchas CLIP, has drawn much attention due to its remarkable ability
for various tasks, including classification and image synthesis. The
combination of CLIP and GAN can be used for text-based image
manipulation and text-based image synthesis.Several models of a
combination of CLIP and GAN have been proposed so far. However,their effectiveness in the food image domain has not been examined
comprehensively yet. In this paper, we reported the results of theexperiments on text-based food image manipulation using VQGANCLIP and discussed the possibility of food image manipulation by
texts.

CCS CONCEPTS• Computing methodologies → Image manipulation; Machine learning approaches.

KEYWORDS
text-based image manipulation, food image manipulation, languageimage pre-training model, CLIP
Permission to make digital or hard copies of all or part of this work for personal orclassroom use is granted without fee provided that copies are not made or distributedfor profit or commercial advantage and that copies bear this notice and the full citationon the first page. Copyrights for components of this work owned by others than theauthor(s) must be honored. Abstracting with credit is permitted. To copy otherwise, or
republish, to post on servers or to redistribute to lists, requires prior specific permission
and/or a fee. Request permissions from permissions@acm.org.
MADiMa ’22, October 10, 2022, Lisboa, Portugal© 2022 Copyright held by the owner/author(s). Publication rights licensed to ACM.
ACM ISBN 978-1-4503-9502-1/22/10. . . $15.00
https://doi.org/10.1145/3552484.3555751

ACM Reference Format:
Kohei Yamamoto Keiji Yanai. 2022. Text-based Image Editing for Food
Images with CLIP. In Proceedings of the 7th International Workshop on Multimedia Assisted Dietary Management (MADiMa ’22), October 10, 2022, Lisboa,
Portugal. ACM, New York, NY, USA, 9 pages. https://doi.org/10.1145/3552484.
3555751

1

INTRODUCTION

With the development of smartphones and social media, people
have posted various photos on the Internet. Among them, one of
the most frequently posted photos is meals. Beautiful, enormous,and eccentric meal photos become a topic easily on social media
throughout the year. Some restaurants post many good-looking
pictures to increase their sales by becoming a trend on social media.
Taking a picture that looks delicious or novel requires many trialand-error processes. After taking photos, they often edit them using
advanced image editing software. Such software requires a highlevel of manipulating skill or knowledge.
The technology for editing images with deep neural networks hasdeveloped remarkably in computer vision, starting with GAN [1]in 2014. In this evolution, image editing using natural languages,
such as ManiGAN [2] and StyleCLIP [3], has attracted attention asa new way to edit images because they do not require special skills
or knowledge for editing. However, most of these models have notbeen applied to food images, mainly to human faces and animal
images.
Moreover, the large-scale language-image pre-trained model,such as CLIP, has recently drawn much attention due to its remarkable zero-shot ability for various tasks, including classification andMADiMa ’22, October 10, 2022, Lisboa, Portugal

image synthesis. The combination of CLIP and GAN can be used
for text-based image manipulation and text-based image synthesis. Several models combining CLIP and GAN have been proposed,such as StyleCLIP and StyleGAN-NADA [4]. These methods canmanipulate images with texts without training in a manipulation
model. It is possible because CLIP was trained with four hundred
million pairs of texts and images, and it has any knowledge of therelation between language and vision.
However, their effectiveness in the food image domain has notbeen examined comprehensively yet. This paper examines thepossibility of text-based food image manipulation with many experiments. As an image manipulation method, we used VQGANCLIP [5]. As a result, we confirmed the effectiveness of text-based
image manipulation using CLIP in the food domain.

2

RELATED WORK

There are two main types of natural language image editing models.
One is a model which learns image-text pairs from scratch. The
other is a model which uses pre-trained visual language models.
ManiGAN [2], which learns image-text pairs, contains a newtext-image affine combination module and a detail correction module. This GAN generates converted images to specified colors or
textures by text. A detail correction module could enhance theperformance of maintaining irrelevant parts while editing details.
TediGAN [6] is the model that used a pre-trained StyleGAN [7]. It
has a similarity module, which learns the similarity between images and texts by mapping them to the same latent space. Using a
StyleGAN trained on face images, TediGAN does not require GAN
training time for the generator. However, this GAN restricts the facedomain for the generated images. Models learning image-text pairs
from scratch require a large amount of training time and images
with text, limiting the edited type of images and manipulation.
The recent editing models often use the text encoder and image encoder of pre-trained visual-language models. In particular,
CLIP [8] is the most common model as a pre-trained visual-language
model. CLIP is trained on 400 million image-text pair data collectedfrom the Internet. While models trained from scratch have limited
training and narrow linguistic-visual features, CLIP has a sizeable
amount of training and comprehensive linguistic-visual features.
Therefore, it has been applied to various computer vision tasks,
such as image classification, detection, segmentation, VQA, andimage synthesis.
StyleCLIP [3] proposed three methods of editing the latent spacefor manipulating images by combining StyleGAN, a typical GAN inimage generation, and CLIP. Several papers studied image editingby manipulating the latent space of StyleGAN. However, thosewere learned in semantic supervision or required human guidance.
In this paper, CLIP automates such guidance. Paint by Word [9]is a part image editing model that combines StyleGAN and CLIP
using masks. There are few studies to change a part of the image
and keep the background. This model can edit the specific partin the mask by editing the latent code 𝑤 of StyleGAN from thereal images. However, this study uses StyleGAN and BigGAN [10],which are specialized GANs for bedrooms and birds or generic
GANs, and not specialized for meals. In our study, we used 𝑧ˆ oftrained VQGAN [11] in the meal domain.

Kohei Yamamoto and Keiji Yanai

The conclusive purpose of our study is to create an image editing
model specialized for food by images and texts. Therefore, we used
VQGAN-CLIP [5] and trained them on a set of meal images andtexts. We also examined the mask function that manipulates only apart of the image for convenient editing as they thought.

3 METHODOLOGY
3.1 Image Manipulation Model
The model we used for food image editing is VQGAN-CLIP. By using
VQGAN for the image generation part, it can control the image bya grid. Moreover, It will learn the vocabulary of image components
using CNN [12] and their composition using Transformer [13]. Additionally, this model can generate high-quality images. CLIP could
compute any linguistic-visual features and similarities between
images and texts with high accuracy. In conventional image editing
models, model architecture often fixed the text to the grammaticalform for training. However, CLIP, trained data from the Internet,
allows for various grammatical forms and can deal with ambiguous
texts. This study used CLIP because natural text editing requires
understanding this ambiguous text.
We examined VQGAN-CLIP, which trained VQGAN and CLIP on
meal images instead of general datasets such as ImageNet, whether
that is robust to meal features or not.

3.2

Architecture

The architecture of VQGAN-CLIP is shown in Figure 2.
First, the input image 𝑥 ∈ R3×𝐻 ×𝑊 is resized to get a resized′′image 𝑥𝑅𝑒𝑠𝑖𝑧𝑒 ∈ R3×𝐻 ×𝑊 . Then, the resized image 𝑥𝑅𝑒𝑠𝑖𝑧𝑒 is input to the encoder of VQGAN to generate the initial latent vector𝑧ˆ ∈ R𝑛𝑧 ×ℎ×𝑤 . Note that 𝑛𝑧 is the number of dimensions of the VQGAN’s codebook. Next, the latent vector 𝑧ˆ is input to the decoder of′′
VQGAN, and the output image 𝑥ˆ ∈ R3×𝐻 ×𝑊 and the input prompt𝑡 are input to CLIP encoders. These give the image token 𝐼 and thetext token 𝑇 , and CLIP compute the loss. Then, the loss function
updates the latent vector 𝑧ˆ by the gradient descent method. Finally,
the latent vector 𝑧ˆ is clamped between the maximum and minimum
values in the VQGAN’s codebook, and the latent vector 𝑧ˆ is updated.
Thus, the updated latent vector 𝑧ˆ is input to the decoder of VQGAN
again, input to CLIP calculating the loss, and updated the latentvector 𝑧ˆ repeatedly.
Note that the latent vector 𝑧ˆ of VQGAN is a vector 𝑛𝑧 assigned toeach square grid ℎ × 𝑤. The model can restrict the editing range byusing this. The formula1 shows the gradient calculation of the latentvector 𝑧ˆ by using 𝑧𝑚𝑎𝑠𝑘 ∈ Rℎ×𝑤 . One in the mask 𝑧𝑚𝑎𝑠𝑘 means amanipulatable grid, and zero means not a manipulatable image. In
computing the gradient of the latent vector 𝑧ˆ𝑔𝑟𝑎𝑑 ∈ R𝑛𝑧 ×ℎ×𝑤 , thisstudy computes their element-wise product so that the gradient of
the unchanged grid becomes zero. If we have a mask image that is
the same size as the input size, the mask image is scaled down andtransformed to fit the grid mask 𝑧𝑚𝑎𝑠𝑘 .
𝑧ˆ𝑔𝑟𝑎𝑑 ← 𝑧ˆ𝑔𝑟𝑎𝑑 ⊙ 𝑧𝑚𝑎𝑠𝑘

3.3

(1)

Loss Function

The manipulating model calculated the loss of equation 2. The
whole loss is the sum of CLIP loss L𝐶𝐿𝐼 𝑃 and image loss L𝑖𝑚𝑔 . This

Text-based Image Editing for Food Images with CLIP

MADiMa ’22, October 10, 2022, Lisboa, Portugal

Figure 2: The VQGAN-CLIP architecture for food image manipulation. We trained VQGAN and CLIP with food datasets for
specialization. VQGAN encoder makes a latent vector from an input image. From the latent vector, the VQGAN decoder makes
an initial edited image. After that, CLIP calculates the losses along with the prompt. That loss optimizes the latent vector. Thismodel repeated iterations to get the edited image. Optionally, we can use a mask when it optimizes.
Table 1: The list of food datasets. Regarding Recipe1M, weused both training and validation sets for training VQGAN.

study set the 𝜆𝐶𝐿𝐼 𝑃 and 𝜆𝑖𝑚𝑔 to one, respectively.
L=

𝜆𝐶𝐿𝐼 𝑃 L𝐶𝐿𝐼 𝑃 + 𝜆𝑖𝑚𝑔 L𝑖𝑚𝑔

(2)

Following equation 3 and equation 4 shows CLIP loss and image
loss. CLIP loss and the image loss used the spherical distance loss
from VQGAN-CLIP implementation. The spherical distance loss
works as almost the cosine similarity between an image token anda text token. This loss can also calculate between an initial andgenerated image token. The image token 𝐼, 𝐼𝑖𝑚𝑔 is the output of the
CLIP image encoder for the generated image 𝑥ˆ and the resized image
𝑥𝑅𝑒𝑠𝑖𝑧𝑒 . The text token 𝑇 is the output of the CLIP text encoder for
input prompt 𝑡.


𝐼 −𝑇
(3)
L𝐶𝐿𝐼 𝑃 = 2 arcsin22𝐼 − 𝐼𝑖𝑚𝑔
L𝑖𝑚𝑔 = 2 arcsin2
(4)
2

3.4

Food Image Datasets

Table 1 shows the statistic on the food image dataset we used in
the experiments. To specialize in meals, VQGAN was fine-tuned to
extract food features by three meal datasets with a different numberof meal categories and images. On the other hand, CLIP was finetuned with Recipe1M [14], which includes pairwise text. This study
used only the training and evaluation datasets of Recipe1M for
training VQGAN and CLIP and the test dataset of Recipe1M formeasuring metrics.

3.5

Prompts for Training CLIP Model

In order to determine whether to use the pre-trained CLIP model,
we evaluated the no-pre-training model “title_NoPretrain” andthe pre-trained model “title”, which were trained with recipe title
prompts. Then, we examined the learning prompts for transfer
learning to use the pre-trained model. The Table 2 shows the list

datasets name
Magical Rice Bowl [15]Foodx251 [16]Food500 [17]Recipe1M [14](Train,Valid)

number of categories
10
251
500
-

number of images
80,408
158,846
399,726
753,251

Table 2: The list of prompts for training CLIP.
abbreviation
title_NoPretrain
title
ingredients
ingredients_title
APhotoOfA
APhotoOfA_ATOF

prompts for training
some_title
some_title
ingredients
ingredients + ’ are ingredients in ’ + some_title + ’ .’’A photo of a ’ + some_title + ’ .’’A photo of a ’ + some_title + ’ , a type of food .’pre-train
×
◦
◦
◦
◦
◦

of learning prompts with the recipe title as “some_title” and therecipe material information as “ingredients”.
These prompts are based on CoOp [18]; adding “a” before theclass token improves the classification accuracy by over 5%. Most
times, adding “a photo of a” before the text improves the classification accuracy. Furthermore, adding context related to the task makea significant improvement. For example, adding “a type of flower”
increased the classification accuracy of the flower image dataset.
For the training prompts of CLIP, we also added these prefixes andsuffixes to examine the difference in performance.
Recipe1M contains recipe information such as titles and ingredients. This study used the titles as the main prompt and the material
information as the additional prompt. The training prompt “ingredients” used only the ingredients information. The “ingredients_title”
which used a combination of the dish name and the ingredientsname used title as “ingredients” and ingredients as “some_title”.

MADiMa ’22, October 10, 2022, Lisboa, Portugal

Kohei Yamamoto and Keiji Yanai

4 EXPERIMENTS
4.1 Overview
First, we compared the prompts for image editing. The compared
items by prompt are these.
(1) Differences by the calling way(2) Differences between inside and outside of the learning domains(3) Differences by taste adjectives(4) Differences by toppings
Then, we showed the differences when VQGAN trained on different datasets, followed by the differences because of the prompts
during CLIP training. Finally, we present a quantitative evaluationof VQGAN using metrics.

4.2

Evaluation Metrics

Quantitative evaluation of GAN in this study used Inception score(IS) [19], Freshet initiation distance (FID) [20], and Kernel-Inception
distance (KID) [21].
IS is the Kullback-Leibler divergence (KL-divergence) between
the conditional label distribution 𝑝 (𝑦|𝑥𝑖 ) and the surrounding label
distribution 𝑝 (𝑦) for the generated data. This metric marks higher
the more significant the diversity of images and the more manageable the images are to identify. In general, higher IS is suitable forimage editing. The IS is derived by averaging the KL-divergenceused number of images𝑖. It describes it as the equation 5.
1 ∑︁
KL(𝑝 (𝑦|𝑥𝑖 )||𝑝 (𝑦)))(5)IS = exp(
𝑁 𝑖
FID is one of the famous metrics to measure the feature distance
between a real image and a generated image. This metric is also
used to evaluate the quality of GAN. Let 𝒎 𝒘 and 𝑪 𝒘 denote themean and covariance matrix of feature vectors of the real image,and 𝒎 and 𝑪 denote the mean and covariance matrix of feature
vectors of the generated image, respectively, FID is defined by theequation 6. Let Tr be the trace of the matrix. The lower the FID
value is, the higher the image quality is.
√︁
FID = ∥𝒎 − 𝒎 𝒘 ∥ 2 + Tr 𝑪 + 𝑪 𝒘 − 2 𝑪 𝑪 𝒘(6)
KID is the dissimilarity calculated using Maximum Mean Discrepancy (MMD) with (𝑓𝑟𝑒𝑎𝑙 , 𝑓 𝑓 𝑎𝑘𝑒 ) samples drawn independently
from different distributions. KID is defined as in equation 7.
KID = MMD(𝑓𝑟𝑒𝑎𝑙 , 𝑓 𝑓 𝑎𝑘𝑒 ) 2(7)

We calculated these metrics between 50,000 real images on thetest dataset from Recipe1M and 50,000 reconstructed images fromtrained VQGAN. Those images were resized 256×256. We calculated
these metrics between 50,000 real images on the test dataset from
Recipe1M and 50,000 reconstructed images from trained VQGAN.
Those images were resized 256 × 256. We measured IS by using
torch-fidelity 1 . We used the clean-fid 2 to measure the FID. KID isalso measured by using torch-fidelity.
In addition, evaluation indiced for CLIP used median rank(medR),
Recall(R@1, R@5, R@10), and CLIPScore [22]. medR is the mediansearch rank. Recall is the percentage of the search results within

the first, fifth, and tenth ranks. CLIPScore evaluates the quality of
the generated caption candidates. The loss may be high using CLIP
as a loss function even though the caption is a match. Similarly, theloss may be low even if the prompts are inappropriate. CLIPScore
is used to evaluate how much the text match. With 𝑤 = 2.5, 𝑐 as thecaption token, and 𝑣 as the image token, it is calculated as follows.
CLIPScore = 𝑤 ∗ 𝑚𝑎𝑥 (𝑐𝑜𝑠 (𝑐, 𝑣), 0)(8)

We used random 10k Recipe1M test data for evaluating CLIP.
OpenCLIP 3 was used for CLIP training and calculating medR and
Recall. The authors’ implementation 4 used for measuring CLIPScore.

4.3

Implementation Details

We fine-tuned the following five VQGAN in this study.
(1) trained ImageNet-1024 model(2) trained ImageNet-16384 model(3) trained Magical Rice Bowl model for 59 epochs(4) trained Foodx251 model for 62epochs(5) trained Food500 model for 12 epochs
Each image was resized to a square of 256 × 256, and the output resolution was set to the same size. The size of the codebook
was 256, divided into a grid of 16 × 16, and the latent vector was𝑧ˆ ∈ R256×16×16 . CLIP was trained using openCLIP 3 , an open source
version of CLIP. OpenCLIP was trained on the training and evaluation datasets of Recipe1M with training prompts of Table 2. CLIP
trained in 32 epochs with ResNet50 [23] as the backbone. The optimization function in the training part was AdamW [24], which
set the batch size to 64, learning rate to 0.001, and weight decay
factor to 0.1. The optimization function in the image generation
part used Adam, which set the step size to 0.05. We iterated image
optimization 1000 times for image editing of one image.

5

EXPERIMENTAL RESULTS

We show the output result from Figure 3 to Figure 12. The required
time for image editing was about 4 to 6 minutes per image.

5.1

Prompt Differences

We compared the prompts for editing images using VQGAN-CLIP.
In this5.1 section, we used VQGAN trained on the Magical Rice
Bowl dataset and ViT-B/32’s pre-trained CLIP. We used VQGAN
trained on the Magical Rice Bowl dataset to clarify whether there isa difference between the trained and untrained images in Figure 4.
Figure 3 compared how meals are called. Magical Rice Bowl
dataset has only ten categories of Japanese dishes. We compared“adjective + English meal name”, “adjective + Japanese meal name”,
and “adjectives only”. In all the output images in Figure 3, the colorof each prompt is visible in various places. With adjectives only,
the color changed in a large range, and its change was also flat likepainted, instead of the color change by the ingredients in adjective
only. Comparing the “adjective + English name” and the “adjective +Japanese meal name”, the “adjective + Japanese name” looked morenatural. The English meal name might be perceived as a cooking

1 https://github.com/toshas/torch-fidelity

3 https://github.com/mlfoundations/open_clip

2 https://github.com/GaParmar/clean-fid

4 https://github.com/jmhessel/clipscore

Text-based Image Editing for Food Images with CLIP

Figure 3: Differences in a calling way. We compared “adjective+ English meal name”, “adjective + Japanese meal name”,
and “adjectives only”. Adjectives (colors) used as prompts
are listed at the bottom and the meal name on the left. For
example, the leftmost column has the prompts “red friedrice”, “red chahan”, and “red” from the top.
process rather than a food name if it is separated into two words,
such as “fried rice”.
Next, we showed the difference between included and excluded
meals in the Magical Rice Bowl dataset. Here, we chose gyudon (beefbowl), kaisendon (seafood bowl), and yakisoba(stir-fried noodles)
as images included in the Magical Rice Bowl dataset, and steak,pizza, and pasta as images not included. Figure 4 showed its results.
When we input the prompt gyudon, objects in gyudon ingredients
appeared. When the prompt was yakisoba, noodle-like objects wereseen in both input images. In addition, we saw the color and shapeof steak and the color of pizza, which did not include the ten meals
in the Magical Rice Bowl dataset. Thus, there are some changes inall the output images. There is no significant difference between themeals included in the Magical Rice Bowl dataset and those not. The
inclusion or exclusion in the training domain of VQGAN did not
significantly affect the image edits. Therefore, these image changes
were not caused by the training of VQGAN but depended on CLIP.
In addition, to observe the difference in the taste adjectives,
we added typical taste adjectives to the meal names. The selected
tastes are hot, sweet, salty, sour, bitter, and oily. Figure 5 showed
the outputs when input prompts of taste adjectives. No significant
change in appearance is observed in all images except for oily. Fromthese outputs, we inferred that the input related to taste might notchange much because of the weak visual meaning. To make theseedits, we should add words such as spicy ingredients with anothervisual change.
Finally, we showed the output results using the prompts foradding toppings in Figure 1. Here, we considered the model tochange the images by adding words of five ingredients: egg, bacon,
lettuce, seafood, and ham, after adding the word “with”. As shownin Figure 1, outputs are visible five ingredients that do not exist
in the categories of the Magical Rice Bowl dataset. It happened bylinking with the large-scale pre-trained visual language model CLIP.
We also found that there are cases editing different parts, such as

MADiMa ’22, October 10, 2022, Lisboa, Portugal

Figure 4: Differences between inside and outside of the learning domains on VQGAN. The left side is the domain within
the learning, and the right side is the domain outside thelearning. The input prompts are listed below. VQGAN’s learning domain or not does not make a significant difference inimage editing.

Figure 5: Differences by taste adjectives. The prompts used
are listed below. Except for “oily”, the appearance did notchange significantly.
the dish of the upper right part in “rice with egg”. Therefore, we
examined a function to input masks.
The function of adding toppings is a feature of this image editing
model. Hence, the following comparison by the VQGAN and CLIPmodels used the prompts about toppings.

5.2

Differences of Trained VQGAN

Figure 6 showed the differences among the models of trained VQGAN with food datasets. We used the shown VQGAN in the figure
and the pre-trained ViT-B/32 CLIP.
There are differences in image editing in all outputs, but they
are edited as the prompts. There are no significant differences inquality among all training models, only minor differences.

5.3

Differences of Trained CLIP

Figures 7 and 8 compared the training prompts when training
CLIP with Recipe1M. We used the CLIP learned from the prompts
shown in Table 2, except for ViT-B/32 pre-trained CLIP for theoriginal CLIP. This section outputs used the ImageNet1024 pretrained VQGAN.

MADiMa ’22, October 10, 2022, Lisboa, Portugal

Kohei Yamamoto and Keiji Yanai

Figure 6: Differences in training datasets of VQGAN. The
prompts are listed below, and the dataset used for training
VQGAN is on the left. There are no significant differences inquality among all training models.

Comparing “title_NoPretrain”, which was learned with the title
from scratch CLIP, and “title”, which was learned with title from thepre-trained CLIP, there was no apparent difference. Therefore, weused the pre-trained model for the other learning prompts. The outputs of “ingredients” and “ingredients_title” prompts that include
ingredients have similar artifacts. In addition, those CLIP models
could not manipulate images according to the prompts, and lower
quality of image editing was observed. We can assume that learning
CLIP with food ingredients does not improve the image quality.
This conjecture implies that the material does not directly play a significant role in the image’s appearance. We also found that training
CLIP with meal images does not suppress GAN-specific artifacts inwhole result images. As for “APhotoOfA” and “APhotoOfA_ATOF”,
both models were less disturbed, but “APhotoOfA_ATOF” was lesscorrupted overall outputs. “APhotoOfA_ATOF” was also more stable than “APhotoOfA”. In Figure 7, we can see bacon, lettuce, andseafood on a different plate from the rice in the upper right of
the image in “APhotoOfA_ATOF”. Also, in Figure 8, we can see an
egg, bacon, and ham in “APhotoOfA_ATOF”. Finally, comparing theoriginal CLIP and the whole trained CLIP, the original CLIP showedsome changes along with the prompts, but the image quality was
rough.

5.4

Figure 7: Difference by CLIP learning prompt on rice. The
prompts are listed below, and the CLIP-learned prompts (seeTable 2) are on the left. There was no obvious difference
between “title_NoPretrain” and “title”. The quality of image
editing was lower for “ingredients” and “ingredients_title”,
higher for “APhotoOfA_ATOF” and “PhotoOfA”.
Table 3: Reconstructed metrics of 50,000 meal images in VQGAN
50k
ImageNet1024
ImageNet16384
Magical Rice Bowl
foodx251
food500

IS↑
7.09 ± 0.10
7.05 ± 0.06
5.97 ± 0.06
6.15 ± 0.06
6.62 ± 0.05

FID↓
6.73
4.53
7.15
4.59
4.07

KID↓(×10−3 )3.92 ± 0.46
2.39 ± 0.36
3.51 ± 0.48
1.85 ± 0.31
1.66 ± 0.29

Difference with and without Mask

In this section, we showed the results using the mask. Figure 9 to
12 showed the results. These images used VQGAN trained on the
Magical Rice Bowl dataset and pre-trained CLIP on ViT-B/32. In
addition, in Figure 9, we manually created the square mask to cover

the rice, while in other Figures 10 to 12, we used the mask from
UECFoodPixComplete [25].
In Figure 9, the toppings were placed on top of the rice, andthe surrounding background was unchanged with the mask. In

Text-based Image Editing for Food Images with CLIP

MADiMa ’22, October 10, 2022, Lisboa, Portugal

Table 4: Quantitative evaluation of CLIP

10k
title_NoPretrain
title
ingredients
ingredients_title
APhotoOf
APhotoOf_typeOfFood

MedR↓
21
17
1022
1066
16
17

image to text
R@1↑ R@5↑
0.105 0.278
0.113 0.307
0.008 0.030
0.016 0.058
0.117 0.310
0.107 0.299

R@10↑
0.381
0.419
0.050
0.095
0.424
0.415

MedR↓
22
17
827
808
17
17

text to image
R@1↑ R@5↑
0.109 0.278
0.120 0.306
0.006 0.024
0.011 0.040
0.121 0.308
0.108 0.302

CLIPscore↑
R@10↑
0.384
0.416
0.041
0.066
0.422
0.412

1.28661.22700.44180.4176
1.15171.2537

Figure 9: Differences of output by using the manually createdgrid mask on rice images. The small images at the top are theinput image and the input mask. The prompts are shown at
the bottom. The upper images are the output result without
the mask, and the lower image is the output result with themask. The results show that the background is preserved.

Figure 8: Difference by CLIP learning prompt on chahan. The
prompts are listed below, and the CLIP-learned prompts areon the left. The result is almost the same as Figure7.

Figure 10, we used a circular mask got from the mask image from
UECFoodPixComplete. We can confirm that model added the toppings to the curry. In Figure 11, we used a large rectangular mask
got from UECFoodPixComplete. Toppings are even sometimes visible without the mask, but all the toppings are visible with the mask.

Figure 10: Differences of output by using the mask on curry
images. The format is the same as in Figure9. Those results
used a circular mask got from the UECFoodPixComplete
mask image. The model was edited to add toppings to themask.

In Figure 12, we used a large uneven mask like a circle from UECFoodPixComplete. The toppings are present even without masks,
but with masks, toppings are generated obviously. Moreover, themodel uses a mask not to change the spoon’s shape.

MADiMa ’22, October 10, 2022, Lisboa, Portugal

Figure 11: Differences of output by using the mask on yakisoba images. The format is the same as in Figure9. Toppings
are even sometimes visible without the mask, but all the toppings are visible with the mask.

Kohei Yamamoto and Keiji Yanai

ImageNet16384 and foodx251 were comparable to those pre-trainedon food500, but ImageNet16384 KID was a little far from food500.
The IS comparison showed that ImageNet1024 was the best, whichindicated the most recognizable and diverse, but food500 is not far
behind, and overall, food500 is the highest quality model.
The differences among the training datasets of VQGAN were
evaluated quantitatively and qualitatively in Table 3 and Figure 6.
The differences among the training datasets in Figure 6 showedthat the output images are not significantly different. Then, when
we look at Table 3, we find that the food500 model has the highest
accuracy. The quantitative evaluation showed a difference in the
numerical values, but there is no significant difference in quality inimage editing. It showed that qualitative evaluation by quantitativeevaluation is complex. In image editing, such as this study, no
standard evaluation index and benchmark has been formed, andquantitative evaluation does not directly evaluate the superiority ofimage editing. The establishment of a new quantitative evaluationindex is required in image editing.
Table 4 showed the quantitative evaluation of CLIP. “title”, “APhotoOF”, and “APhotoOf_ATOF” are relatively dominant. “title_NoPretrain”
had the highest CLIPScore, but may not be mature about the similarity between text and images. Figures 7 and 8 also showed thedifferences among the training prompts for CLIP. When models geta good quantitative evaluation, the output images also tend to be
good. Therefore, unlike GAN quantitative evaluation, CLIP evaluation has the potential to provide qualitative evaluation throughquantitative evaluation.

6
Figure 12: Differences of output by using the mask on chahan
images. The format is the same as in Figure9. Those results
used a large uneven mask that got from UECFoodPixComplete. The spoon shape is maintained by using a mask.

Using the mask, the model did not edit the image at the specified location, and the manipulated images were not corrupted. In
addition, this model edited the non-masked area appropriately, andthere was less image corruption near the mask’s boundaries. Whenan edited range of the image is small, editing without a mask often
results in not being edited well because manipulation attention isdrawn to various locations. On the contrary, when the edited rangeof the image is extensive, editing is often performed even withouta mask. However, as shown in Figure 12, a thing’s shape, like thespoon, is maintained by a mask. Therefore, we can say that editing
quality is better with a mask in all cases. Although there is a need to
input a mask, the input of a mask image is a very effective method
when we want to specify the editing points or when we want tomake drastic editing.

5.5

Quantitative Evaluation of the Model

Table 3 showed the evaluation of VQGAN. VQGAN pre-trainedon food500 had the lowest FID and KID, while VQGAN trained on
the Magical Rice Bowl dataset had the highest FID. The FIDs of

CONCLUSIONSIn this study, we examined the effectiveness of VQGAN-CLIP inediting images related to meals. In order to further specialize themodel to meal images, we trained VQGAN on a meal image dataset
and CLIP on a recipe dataset with various training prompts.
We compared the differences by editing prompt, training dataset
for VQGAN, and training prompts of CLIP, using a mask or not. We
found that the training dataset for VQGAN showed no significant
differences in the output images. However, the evaluation index isbetter with the meal image dataset. “APhotoOfA_ATOF” has relatively fewer image corruptions and better scores in the quantitativeevaluation among the CLIP training prompts. The function of mask
images is a very effective tool for image editing.
We will consider adding a model that automatically inputs masks.
We will also consider image generation models other than the
VQGAN model, such as PPDM[26].
Acknowledgment This work was supported by JSPS KAKENHI
Grant Numbers, 21H05812, 22H00540, 22H00548, and 22K19808.

REFERENCES[1] Ian Goodfellow, Jean Pouget-Abadie, Mehdi Mirza, Bing Xu, David Warde-Farley,
Sherjil Ozair, Aaron Courville, and Yoshua Bengio. Generative adversarial nets.
In Proc.of Advances in Neural Information Processing Systems, 2014.
[2] Bowen Li, Xiaojuan Qi, Thomas Lukasiewicz, and Philip HS Torr. Manigan:
Text-guided image manipulation. In Proc.of IEEE Computer Vision and Pattern
Recognition, pages 7880–7889, 2020.
[3] Or Patashnik, Zongze Wu, Eli Shechtman, Daniel Cohen-Or, and Dani Lischinski. Styleclip: Text-driven manipulation of stylegan imagery. In arXiv preprint
arXiv:2103.17249, 2021.
[4] Rinon Gal, Or Patashnik, Haggai Maron, Gal Chechik, and Daniel Cohen-Or.
Stylegan-nada: Clip-guided domain adaptation of image generators. In arXivText-based Image Editing for Food Images with CLIP

preprint arXiv:2108.00946, 2021.
[5] Katherine Crowson, Stella Biderman, Daniel Kornis, Dashiell Stander, Eric Hallahan, Louis Castricato, and Edward Raff. Vqgan-clip: Open domain image generation and editing with natural language guidance. arXiv preprint arXiv:2204.08583,
2022.
[6] Weihao Xia, Yujiu Yang, Jing-Hao Xue, and Baoyuan Wu. Tedigan: Text-guided
diverse face image generation and manipulation. In Proc.of IEEE Computer Vision
and Pattern Recognition, pages 2256–2265, 2021.
[7] Tero Karras, Samuli Laine, and Timo Aila. A style-based generator architecture
for generative adversarial networks. In Proc.of IEEE Computer Vision and Pattern
Recognition, pages 4401–4410, 2019.
[8] Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh,
Sandhini Agarwal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark,
et al. Learning transferable visual models from natural language supervision. In
arXiv preprint arXiv:2103.00020, 2021.
[9] David Bau, Alex Andonian, Audrey Cui, YeonHwan Park, Ali Jahanian, Aude
Oliva, and Antonio Torralba. Paint by word. In arXiv preprint arXiv:2103.10951,
2021.
[10] Andrew Brock, Jeff Donahue, and Karen Simonyan. Large scale gan training forhigh fidelity natural image synthesis. In arXiv preprint arXiv:1809.11096, 2018.
[11] Patrick Esser, Robin Rombach, and Bjorn Ommer. Taming transformers forhigh-resolution image synthesis. In Proc.of IEEE Computer Vision and Pattern
Recognition, pages 12873–12883, 2021.
[12] Yann LeCun, Patrick Haffner, Léon Bottou, and Yoshua Bengio. Object recognition
with gradient-based learning. In Proc.of Shape, Contour and Grouping in Computer
Vision, pages 319–345. Springer, 1999.
[13] Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones,
Aidan N Gomez, Łukasz Kaiser, and Illia Polosukhin. Attention is all you need.
In Proc.of Advances in Neural Information Processing Systems, pages 5998–6008,
2017.
[14] Amaia Salvador, Nicholas Hynes, Yusuf Aytar, Javier Marin, Ferda Ofli, Ingmar
Weber, and Antonio Torralba. Learning cross-modal embeddings for cooking
recipes and food images. In Proc.of IEEE Computer Vision and Pattern Recognition,
pages 3020–3028, 2017.

MADiMa ’22, October 10, 2022, Lisboa, Portugal[15] Ryosuke Tanno, Daichi Horita, Wataru Shimoda, and Keiji Yanai. Magical rice
bowl: A real-time food category changer. In Proc.of ACM International Conference
Multimedia, pages 1244–1246, 2018.
[16] Parneet Kaur, Karan Sikka, Weijun Wang, Serge Belongie, and Ajay Divakaran.
Foodx-251: A dataset for fine-grained food classification. In arXiv preprint
arXiv:1907.06167, 2019.
[17] Weiqing Min, Linhu Liu, Zhiling Wang, Zhengdong Luo, Xiaoming Wei, Xiaolin
Wei, and Shuqiang Jiang. Isia food-500: A dataset for large-scale food recognition via stacked global-local attention network. In Proc.of ACM International
Conference Multimedia, pages 393–401, 2020.
[18] Kaiyang Zhou, Jingkang Yang, Chen Change Loy, and Ziwei Liu. Learning toprompt for vision-language models. In arXiv preprint arXiv:2109.01134, 2021.
[19] Tim Salimans, Ian Goodfellow, Wojciech Zaremba, Vicki Cheung, Alec Radford,
and Xi Chen. Improved techniques for training gans. Proc.of Advances in Neural
Information Processing Systems, 29, 2016.
[20] Martin Heusel, Hubert Ramsauer, Thomas Unterthiner, Bernhard Nessler, andSepp Hochreiter. Gans trained by a two time-scale update rule converge to a
local nash equilibrium. Proc.of Advances in Neural Information Processing Systems,
30, 2017.
[21] Mikołaj Bińkowski, Danica J Sutherland, Michael Arbel, and Arthur Gretton.
Demystifying mmd gans. arXiv preprint arXiv:1801.01401, 2018.
[22] Jack Hessel, Ari Holtzman, Maxwell Forbes, Ronan Le Bras, and Yejin Choi.
Clipscore: A reference-free evaluation metric for image captioning. arXiv preprint
arXiv:2104.08718, 2021.
[23] Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep residual learningfor image recognition. In Proc.of IEEE Computer Vision and Pattern Recognition,
pages 770–778, 2016.
[24] Ilya Loshchilov and Frank Hutter. Decoupled weight decay regularization. In
arXiv preprint arXiv:1711.05101, 2017.
[25] Kaimu Okamoto and Keiji Yanai. Uec-foodpix complete: A large-scale food image
segmentation dataset. In Proc.of International Conference on Pattern Recognition,
pages 647–659. Springer, 2021.
[26] Jonathan Ho, Ajay Jain, and Pieter Abbeel. Denoising diffusion probabilistic
models. Proc.of Advances in Neural Information Processing Systems, 33:6840–6851,
2020.