SetMealAsYouLike:
Sketch-based Set Meal Image Synthesis
with Plate Annotations

Yuma Honbu, Keiji Yanai ( The University of Electro-Communications )Abstract : background and purpose
GAN can generate meal image with the shape and meal category
specified by the user. However, there are problems withdistortion and merging of plates, because only the food region isspecified and generated. Furthermore, there is no food datasetthat includes a plate region.

2. Training SEAN on the food datasetwith added plate regions

fAdding a Plate Region Mask with Few-shot Segmentation
Generate high quality meal images
SEAN pipeline

SEAN norm

About SEAN [3]

SEAN is a method that allows different styles to be adapted for each region
Multiple normalizations dependent on segment masks and style features forhigh-quality generation

Method :Extract Plate Region using HSNet.
SEAN trained on Food dataset with plate and generate.

Result

1. Extracting Plate regions by Few-shot Segmentation(FSS) [2]
2. SEAN[3] trained on food dataset with plate regions

Few-shotGrabCut [4]1. Plate regions extraction with FSS

Weakly [5]UEC-FoodPix Complete[1] dataset

IoU
78.5
66.1
50.2

IoU of the plateregion is over 70

Pixel-level annotations included

100 food class, over 10K large dataset
9000 images for train, 1000 for validation

FID↓ to eval generation quality

Manually annotate 5 images of each classin the food+plate region for Few-shot

Network for extracting plate region

fSPADESEAN

w/o plate
71.5
51.1

FSS Plate
68.4
50.9

Using the plate region improved
the quality of the generated.

ここに数式を⼊⼒します。
Plate style conversion result
The impression of the dish isconsidered to change depending on
the style of the plate area.

Easier to imagine
before serving food.

About HSNet[2]Similarity between queries and support features from thebackbone can be used as new features to enable classindependent inference.
Using pre-trained HSNet trained on Pascal-5" dataset.

Plate region extraction

5-shot segmentation in food+plateregion with 5 support sets.
Remove food regions fromsegmentad regions.

Few-shot Segmentation enables
robust plate region segmentation

Demo System : Web-based application operated by tablet or PCWeb App

1. Select the class and style
image you want to draw.
2. Draw a mask and generate
an image by SEAN.

You can generate set mealin any shape and style[1] K. Okamoto and K Yanai. UEC-FoodPix Complete: A Large-scale Food Image Segmentation Dataset. In Proc. of the ICPR Workshop on Multimedia Assisted Dietary Management,2021.
[2] J. Min, D. Kang, and M. Cho. Hypercorrelation Squeeze for Few-Shot Segmentation. In Proc. of the IEEE/CVF International Conference on Computer Vision, 2021.
[3] P. Zhu and R. Abdal and Y. Qin and P. Wonka. SEAN: Image Synthesis with Semantic Region-Adaptive Normalization. In Proc.of the IEEE Conference on Computer Vision and Pattern Recognition.
[4] C. Rother, V. Kolmogorov, and A. Blake. "GrabCut": interactive foreground extraction using iterated graph cuts. Proc.of ACM Trans. 2004.
[5] W. Shimoda and K. Yanai. Predicting Plate Regions for Weakly-supervised Food Image Segmentation. In Proc. of IEEE International Conference on Multimedia and Expo. 2020.