madima72

Text-based Image Editing
for Food Images with CLIP

Kohei Yamamoto, Keiji Yanai
The University of Electro-Communications, Tokyo, Japan

1.INTRODUCTION

4. RESULTS

There are some image synthesis models used CLIP[1] and GAN.

Prompts of adding topping→Toppings were added

However, their effectiveness in the food domain has notbeen examined comprehensively yet.
We reported the results of the experiments on text-based foodimage manipulation using VQGAN-CLIP[2].

fried rice-

2. ARCHITECTURE
Based on VQGAN-CLIP

Editing with a mask image → backgrounds were saved.

1.

VQGAN encoder generates the latent vector from input images.

2.

VQGAN decoder generates output images from latent vector.

3.

CLIP calculate similarities between images and prompts.
fried rice-The model calculates the loss by similarities.

5.

(Optional) The latent vector gradient 𝒛!!"#$ updates by based on 𝑧%#&' .

6.

The loss function updates the latent vector by gradient descent method.

𝜆!"#$ = 𝜆%&' = 1, ℒ!"#$ = 2 arcsin(𝐼 − 𝐼%&'𝐼−𝑇(, ℒ%&' = 2 arcsin
22(𝐼: output image token, 𝐼'() :input image token, 𝑇:text token)-with bacon-with lettuce-with seafood-with ham

Prompts with taste adjective→little change-fried rice4.

-with egg

3. EXPERIMENT
1. Compare the prompts for food image editing
2. Fine-tune VQGAN and CLIP on food datasets
The list of food datasets
datasets name number of categories
Magical Rice Bowl[3]10Foodx251[4]251
Food500[5]500
Recipe1M(Train, Valid)[6]-number of images
80,408
158,846
399,726
753,251

Image
differencesin trained
VQGAN→littlePrompt for CLIP training
abbreviation prompts for training
pre-train
title_NoPretrain some_title
×title some_title∘
ingredients ingredients∘
ingredients_title ingredients + ' are ingredients in ' + some_title + ' .'∘APhotoOf 'A photo of a ' + some_title + ' .'∘APhotoOf_ATypeOfFood 'A photo of a ' + some_title + ' , a type of food .'∘𝟑×𝟐𝟓𝟔×𝟐𝟓𝟔• 𝑥, 𝑥# ∈ ℝ

𝟐𝟓𝟔×𝟏𝟔×𝟏𝟔• 𝒛# ∈ ℝ• 1000 times iteration• 4-6 minutes per images for image editing

red fried rice

blue fried ricegreen fried rice

Reconstructed quantitative evaluation of VQGAN→More images and categories, better results.
50k
imagenet1024
Magical Rice Bowl
foodx251
food500

IS↑
7.09 ± 0.10
5.97 ± 0.06
6.15 ± 0.06
6.62 ± 0.05

FID↓
6.73
7.15
4.59
4.07

KID↓(×10() )3.92 ± 0.46
3.51 ± 0.48
1.85 ± 0.31
1.66 ± 0.29

[1] Radford, Alec, et al. "Learning transferable visual models from natural language supervision." International Conference on Machine Learning. PMLR, 2021.
[2] Crowson, Katherine, et al. "Vqgan-clip: Open domain image generation and editing with natural language guidance." arXiv preprint arXiv:2204.08583 (2022).
[3] Horita, Daichi, et al. "Food category transfer with conditional cyclegan and a large-scale food image dataset." Proceedings of the Joint Workshop on Multimedia for Cooking and Eating Activities and Multimedia Assisted Dietary Management. 2018.
[4] Kaur, Parneet, et al. "Foodx-251: a dataset for fine-grained food classification." arXiv preprint arXiv:1907.06167 (2019).
[5] Min, Weiqing, et al. "Isia food-500: A dataset for large-scale food recognition via stacked global-local attention network." Proceedings of the 28th ACM International Conference on Multimedia. 2020.
[6] Salvador, Amaia, et al. "Learning cross-modal embeddings for cooking recipes and food images." Proceedings of the IEEE conference on computer vision and pattern recognition. 2017.