Cross-Modal Recipe Embeddings
by Disentangling Recipe Contentsand Dish Styles
Yu Sugiyama and Keiji Yanai
The University of Electro-Communications,Tokyo

2021/9/17© 2020 UEC Yanai Lab. Tokyo.

1Introduction• Cooking recipe sharing site are widely used
• Recipe data has Text and Images• We propose a RDE-GAN(Recipe Disentangled EmbeddingGAN)

https://www.allrecipes.com

https://cookpad.com
2021/9/17© 2020 UEC Yanai Lab. Tokyo.

2

Background• After Recipe 1M+, text and image cross-modal learning
becomes hot topic

Recipe1M+[Salvador, CVPR2017]
2021/9/17© 2020 UEC Yanai Lab. Tokyo.

3

Background: Multi-modal retrieval• Embed text and image into shared space• Retrieval by distance in shared space
Harriet's Bran Muffins- 2 12 cups flour, sifted- 2 12 teaspoons baking soda
- …
1. 'Preheat oven to 400F andlightly spray muffin pans withcooking spray or line withcupcake papers.
2. …

Text
Encoder

SharedSpaceImage
Encoder

2021/9/17© 2020 UEC Yanai Lab. Tokyo.

4

Related Work• ACME[Hao, CVPR2019]• Adversarial Cross-Modal Embedding• Image generator improved cross-modal retrieval

ACME[Hao, CVPR2019]2021/9/17© 2020 UEC Yanai Lab. Tokyo.

5

Related Work• ACME[Hao, CVPR2019]• Adversarial Cross-Modal Embedding• Image generator improved cross-modal retrieval

ACME[Hao, CVPR2019]2021/9/17© 2020 UEC Yanai Lab. Tokyo.

6

Proposed Method: Overview• Recipe images have…
• recipe information: useful for recipe retrieval• non-recipe information: non-useful for recipe retrieval

recipe information•••••

non-recipeinformation•••••chilisouptomato basestew…

2021/9/17© 2020 UEC Yanai Lab. Tokyo.

round platewhite plate
centered picture
photo in bright location…

7

Proposed Method: Overview• Improve ACME, disentangle image features

ACME[Hao, CVPR2019]2021/9/17© 2020 UEC Yanai Lab. Tokyo.

8Proposed Method: Overview• Improve ACME, disentangle image features

Improve image Encoder/Decoder

ACME[Hao, CVPR2019]2021/9/17© 2020 UEC Yanai Lab. Tokyo.

9

Proposed Method: RDE-GAN• RDE-GAN separates image information• recipe information for learning retrieval• non-recipe information for generate imageRetrieval Spaceℒ𝑇𝑟𝑖𝐸𝑅Shared FClayer

Input Text

ℒ𝑀𝐴𝐷𝑀𝐸𝑉𝑚𝐸𝑉𝑠 ’Input Image
2021/9/17

𝑅𝑉

Title
Classifier
IngredientClassifier

1048 class[0, 0, …, 0]ℒ𝐶𝑙𝑎𝑠𝑠4102 multi class[0, 0, …, 0]

𝐸𝑉𝑚

𝐺Generated Image 𝐷© 2020 UEC Yanai Lab. Tokyo.

ℒ𝐷𝑖𝑠𝑒𝑛𝑡𝑎𝑛𝑔𝑙𝑒ℒ𝐺 , ℒ𝐷
10

Proposed Method: RDE-GAN• RDE-GAN separates image information• recipe information for learning retrieval• non-recipe information for generate image
Retrieval Spaceℒ𝑇𝑟𝑖

𝐸𝑅Shared FC
layer

Input Text

ℒ𝑀𝐴

𝐷𝑀
𝐸𝑉𝑚
𝐸𝑉𝑠 ’Input Image
2021/9/17

𝑅𝑉

Title
Classifier
IngredientClassifier

1048 class[0, 0, …, 0]ℒ𝐶𝑙𝑎𝑠𝑠4102 multi class[0, 0, …, 0]𝐸𝑉𝑚

𝐺Generated Image 𝐷© 2020 UEC Yanai Lab. Tokyo.

ℒ𝐷𝑖𝑠𝑒𝑛𝑡𝑎𝑛𝑔𝑙𝑒ℒ𝐺 , ℒ𝐷
11

Proposed Method – Encode Images• Use different network for recipe feature and non-recipefeature• Shallow network for non-recipe feature• Deep network for recipe feature
𝑬𝒎𝑽

ResNet50

Input Image

𝑬𝒔𝑽 ’𝑉

MLP

1024

recipeFeature Vector

1024

8

AdaIN Parameter 𝐺

Residual
Block

UpSampling

8

non-recipe
Feature Map
2021/9/17© 2020 UEC Yanai Lab. Tokyo.

Generated Image
12

Proposed Method: RDE-GAN
We propose RDEGAN(Recipe Disentangled EmbeddingGAN)RDE-GAN disentangles recipe feature and non-recipefeature1. Use only recipe feature for retrieval

2. Use recipe and non-recipe feature for image generate

2021/9/17© 2020 UEC Yanai Lab. Tokyo.

13

Experiments – Recipe Retrieval Evaluation• We test with median rank(MedR) and recalls• Ours performed state-of-the-art with large test set
Num

1k

10k

2021/9/17

Image2Recipe

Recipe2Image

method MedR↓

R@1↑

R@10↑ MedR↓

R@1↑

R@10↑

JE

5.2

25.6

65.0

5.1

25.0

65.0

ACME

1.0

51.8

87.5

1.0

56.3

85.9

Ours

1.0

59.4

87.4

1.0

61.2

87.2

JE

41.9

-----ACME

6.7

22.9

57.9

6.0

24.4

59.0

Ours

3.5

36.0

64.4

3.0

38.2

65.8

© 2020 UEC Yanai Lab. Tokyo.

14

Experiments – Retrieval Example
Query Text

#1

#2

#3

Ainsley Harriott's baked mussels
recipe- 350 g (12.3oz) Cherry tomatoes- 1 Red onion, finely chopped- …
1. Pre-heat the oven 200 degrees
C/400 degrees F gas mark 6.
2. Place the cherry tomatoes in alarge roasting tin and scatter over
the onions.
3. …

I Cant Believe Its Cutty: Victory Is
Mine Cocktail- 2 ounces Cutty Sark Prohibition
Edition- 3/4 ounce Marie Brizard Apry
- …
1.

2.
3.

Combine liquid ingredients in one
small shaker tin, and only the egg
white (no ice) in another.
Shake vigorously to froth egg
whites and other ingredients.
…
2021/9/17© 2020 UEC Yanai Lab. Tokyo.

15

Experiments – Retrieval Example#1

Query Image

Chicken Mushroom Barley Soup
- 4 garlic cloves- 1 leek
- …
1.
2.

3.

Mince garlic; slice leek longways
then chop.
Place a standard dutch oven(orsoup pot if you don't have a dutch "'oven) on the stove top.
…

#2Tonjiru (Miso Soup with Veggiesand Pork)- 100 g pork
- 1/2 carrot
- …

1.
2.

3.

Ultimate Green Beans- 2 slices bacon, diced- 1/2 white onion, minced
- …
1.

2.

3.
2021/9/17

Cook the bacon in a large, deepskillet over medium-high heat until
crisp, about 10 minutes.
Remove the bacon with a slottedspoon and drain on a paper towellined plate…
© 2020 UEC Yanai Lab. Tokyo.

Chop all vegetables and pork into
small pieces.
(Since I have a small child, I cut
them into a child-sized pieces, ' 'but
you can cut as you wish.)…

Herbed Green Beans andMushrooms- 2 lbs fresh green beans, trimmed- 3 cups water- …
1.
2.
3.

Bring salted water to boil.
Add trimmed green beans cook
until tender but still crisp.
…

16Experiments – Retrievals• RDE-GAN performed state-of-the-art retrieval score• The retrievals are improved with disentangle imagefeatures• It is important separate image feature into recipe feature andnon-recipe feature• Recipe feature denotes feature only for retrieval• Improve the retrieval

2021/9/17© 2020 UEC Yanai Lab. Tokyo.

17

Experiments – Image Generation• We compare generated image• ACME• RDEGAN(ours)• Ours generates better images
ACME

RDEGAN

Input
Image

Image2Image

Text2Image

2021/9/17© 2020 UEC Yanai Lab. Tokyo.

18

Experiments – Image Generation• Evaluate generated images with FID• Lower score is better• Our method performed better score in FID

MethodFID↓@
Image2Image

FID↓@
Text2Image

ACME

183.8

182.9

RDEGAN(ours)

158.9

158.6

2021/9/17© 2020 UEC Yanai Lab. Tokyo.

20

Experiments – Gradual Change• 3 experiments for test encoded space• Test1: Change only non-recipe feature with
Image A to Image B• Test2: Change only recipe feature with
Image A to Image B• Test3: Change only recipe feature with
Text A to Text B

2021/9/17© 2020 UEC Yanai Lab. Tokyo.

21

Experiments – Gradual Change• Test1: Change only non-recipe feature with
Image A to Image B• Non-recipe feature represents dish shapes, table color,rough shape, etc.
Input
Image

2021/9/17

TargetImage

Non-recipe feature© 2020 UEC Yanai Lab. Tokyo.

22

Experiments – Gradual Change• Test1: Change only non-recipe feature with
Image A to Image B• Non-recipe feature represents dish shapes, table color,rough shape, etc.
Input
Image

2021/9/17

TargetImage

Non-recipe feature© 2020 UEC Yanai Lab. Tokyo.

23

Experiments – Gradual Change• Test2: Change only recipe feature with
Image A to Image B• Recipe feature represents ingredients color, texture,
number of ingredients, etc.

Input
Image

2021/9/17

TargetImage

recipe feature© 2020 UEC Yanai Lab. Tokyo.

25

Experiments – Gradual Change• Test2: Change only recipe feature with
Image A to Image B• Recipe feature represents ingredients color, texture,
number of ingredients, etc.

InputImage

2021/9/17

TargetImage

recipe feature© 2020 UEC Yanai Lab. Tokyo.

26

Experiments – Gradual Change• Test3: Change only recipe feature with
Text A to Text B• recipe feature with text represents ingredients, texture, etc.
Rob and Lisa's
Island-Escape
Salad

Harriet's Bran
Muffins- 2 12 cups flour,
sifted- …

- 1/2 of a medium
pineapple, cored- …

Rob and Lisa's
Island-EscapeSaladFour Alarm Chili- 2 lbs groundsirloin- …

- 1/2 of a medium
pineapple, cored- …

Udon-Beef
Noodle Bowl

Chili Blanco- 1 tablespoon
vegetable oil- …

2021/9/17- 8 ounces
uncooked udon
noodles- …
© 2020 UEC Yanai Lab. Tokyo.

28

Experiments – Gradual Change• Test3: Change only recipe feature withText A to Text B• recipe feature with text represents ingredients, texture, etc.
Rob and Lisa's
Island-Escape
Salad

Harriet's Bran
Muffins- 2 12 cups flour,sifted- …

- 1/2 of a medium
pineapple, cored- …

Rob and Lisa's
Island-Escape
SaladFour Alarm Chili- 2 lbs groundsirloin- …

- 1/2 of a medium
pineapple, cored- …

Udon-Beef
Noodle Bowl

Chili Blanco- 1 tablespoon
vegetable oil- …

2021/9/17- 8 ounces
uncooked udon
noodles- …
© 2020 UEC Yanai Lab. Tokyo.

29

Experiments – Gradual Change• Our method generate images better than ACME
• non-recipe feature improved generator• recipe feature represents color, texture, etc.
• Same in Image2Image and Text2Text• non-recipe feature represents rough shape, dish, etc.

2021/9/17© 2020 UEC Yanai Lab. Tokyo.

30

Conclusion[Conclusion]• RDE-GAN improved retrieval accuracy• RDE-GAN can control separately rough shape andingredients colors[Future work]• Improve quality and size of synthesized images• Apply this method into other domains

2021/9/17© 2020 UEC Yanai Lab. Tokyo.

31