Cat egor izat ion of Cook in g Act ion sBased on Text u al/ Visu al Sim ilar it yYixin Zh an g1, Yoko Yam ak at a 2 an d Keish i Tajim a 1
1Kyot o Un iver sit y an d 2Th e Un iver sit y of Tok yo, JapanRESEARCH BACKGROU N DRESEARCH PU RPOSE
Word embedding of action verbs in recipes.
- To calculate similarities and differences between action verbs.
- ?Cut? and ?Cut into? have similar meanings but the later one indicates theshape of ingredients after being cut.
- To be used for recipe retrieval or recipe automatic translation in the future.
- Existing method: Word2vec- Train word-embedding model by recipe text data and transform eachverb into multi-dimensional vector.
- Given word embedding vectors of verbs, the similarity between two
verbs is computed by the cosine similarity of their embedded vectors.

- 5,482,309 recipes in Chinese- ( source: https://www.haodou.com/recipe/1190778 )- Every procedural step is associated with an image.
- The pairs of text and image in procedural steps areused in this research.
- 57,361,678 steps in total

IM AGE CLU STERIN G M ETH ODPROPOSED M ETH OD- Word embedding by associated images.
(1) Clustering images into groups which consist of images of similarstyle.
(2) For each verb, calculating the probability distribution of theassociated images over clusters.

- Image clustering- Cluster all 48164
images in ourdataset into 20clusters by using
k-means clustering
methodEXPERIM EN T
- Probability distribution of associated images indicates the meaningof the verb.

- Dataset- #Recipe: 12,548- #Image: 48,164- Word segmentation and POS tagging by Jieba [1]- Total Verbs: 26,9993- Distinct Verbs: 3,175- 341 verbs (10.7%) of which appeared 100 times or more.
- 1262 verbs (39.7%) appeared 2 to 9 times.
- 695 verbs (21.9%) appeared only once.
[1] https://github.com/fxsjy/jieba- Computing top-10 similar verbs by using our three vectorization
methods.
- Calculating how many of the top-n results given by our image-based method arealso included in the top-10 results given by the word2vec method for n = 1, . . . ,10.
- Image-clustring based probability distortion holds word semantics even under
hard dimension compression.
- Dimensional compression from 4096 to 20 using image clustering did not

COM PARISON OF M ETH ODS

cause performance deterioration.

- Ground truth: Word2vec- To evaluate whether the proposed method achieves word embedding which
represents word semantics as word2vec does.

- We compared t h r ee w ays to vectorize verbs.
- Method 1 and 2 use the image vectors calculated from VGG16 and PCA .
- Method 3 uses the clusters of recipe images (proposed method).

- The top-10 results of the Word2vec method
and the method using the 20-dimensional
vectors for the 14 example verbs.
- Text-based similarity and image-basedsimilarity of verbs are complementary toeach other.

CON CLU SION
- Future work
- Word embedding of action verbs using pairs of text and images in recipes.
- To evaluate our proposed method by manulally generated ground truth.
- Image-based word embedding achieves as good performance as Word2vec.
- To further research on the relevance between textual and visual data.
- Dimensional compression from 4096 to 20 using image clustering did not cause- Not only verbs, but also extend the method to other POS (e.g. adj).
performance deterioration.
- Text-based similarity and image-based similarity of verbs are complementary to each - To deploy our method for recipe auto-translation or recipe multi-lingual
retrieval.
other.