Categorization of Cooking Actions
Based onTextual/Visual Similarity
Yixin Zhang1, Yoko Yamakata2 and Keishi Tajima1
1Kyoto University and 2The University of Tokyo, JapanMADiMa 2019

Research Background
Chinese recipe site: 好豆 (Haodou)• 5,482,309 recipes in Chinese

1

Add less baking soda powder tothe shrimp and a little salt and mix
well, then wash with water, drain,
then add the cooking wine andmarinate for 10 minutes, then add
starch and a little oil to grab.

• Every procedural step is associated withan image.
• The pairs of text and image in proceduralsteps are used in this research.
• 57,361,678 steps in totalsource: https://www.haodou.com/recipe/1190778

Research Purpose

2Word embedding of action verbs in recipes.
• To calculate similarities and differences between action verbs.
• “Cut” and “Cut into” have similar meanings but the latter one
indicates the shape of ingredients after being cut.

• To be used for recipe retrieval or recipe automatic translation.
• Existing method: word2vec• Train word-embedding model by recipe text data and transform eachverb into a vector.
• The similarity between two verbs is computed by the cosine similarityof their embedded vectors.

Basic observation
Wash the green
pepper and cut.

Cut eggplant intosticks.

Pour the bone soup into the pot, stir
fry over high heat, and mix in salt
and pour in the walnut oil.

3
The semantic of “cut” and “cutinto” are similar, also the imagestyles of them are same
(ingredients on the choppingboard)
On the other hand, the semanticof “cut” and “stir fry” are notsimilar, also the image styles ofthem are not similar (latter one:
ingredients in the fry pan)Proposed Method

4• Word embedding by associated images.

(1) Clustering images in our dataset into groups which consist of images ofsimilar style.
(2) For each verb, calculating the probability distribution of the associatedimages over clusters.

Cluster 13(２)(１)
Cluster 8

Cluster 16“Stir fry”
associatedProposed Method

5• Probability distribution of associated images over clusters
indicates the meaning of the verb.

Not similarSimilar

Dataset• #Recipes = 12,548• #Images = 48,164• Word segmentation and POS tagging by Jieba [1]• #Total Verbs = 269,993• #Distinct Verbs = 3,175• 10.7% appears 100 times or more.
• 39.7% appears 2 to 9 times.
• 21.9% appears only once.
• Table 1 shows the top 20 most frequent words.

[1] https://github.com/fxsjy/jieba

6Image clustering method

7Comparison of methods

8• Ground truth: Word2vec• To evaluate whether the proposed method achieves word embedding
which represents word semantics as word2vec does.

• We compared three ways to vectorize verbs.

• Method 1 and 2 use the image vectors calculated from VGG16 and PCA.
• Method 3 uses the clusters of recipe images (proposed method).

Experiment• Whether image styles contain semantics of associated image.
• Computing top-10 similar verbs by using our three vectorization

methods.
• Calculating how many of the top-n results given by our image-based
method are also included in the top-10 results given by the word2vec
method for n = 1, . . . , 10.
• Dimensional compression from 4096 to 20 using image clustering did
not cause performance deterioration.

9

Experiment10• Producing a 20-dimensional vector for each verb by computing the ratio•of each cluster within the set of all images associated with it.
The heat map represents the ratio of each cluster for 14 example verbs.

Experiment11• The top-10 results of the word2vec method and the method
using the 20-dimensional vectors for the 14 example verbs.
• Text-based (Word2vec) similarity and image-based (proposedmethod) similarity of verbs are complementary to each other.

Conclusion

12• Word embedding of action verbs using pairs of text and images inrecipes.
• Three methods of vectorizing verbs are compared with Word2vec.
• Image-based word embedding achieves as good performance as Word2vec.
• Dimensional compression from 4096 to 20 using image clustering did not cause

performance deterioration.
• Text-based similarity and image-based similarity of verbs are complementary toeach other.

• Future work
• To evaluate our proposed method by manually generated ground truth.
• To further research on the relevance between textual and visual data.
• Not only verbs, but also extend our method to other part-of-speech (e.g. adj.)• To deploy our method for recipe auto-translation or recipe multi-lingual retrieval.

Thank you