Su pplem en t in g Om it t ed Nam ed En t it ies in
Cook in g Pr ocedu r al Text w it h At t ach ed Im ages
Yixin Zh an g1, Yoko Yam ak at a 2 an d Keish i Tajim a 1
1

Kyot o Un iver sit y an d 2Th e Un iver sit y of Tok yo, JapanRESEARCH PROBLEMRESEARCH BACKGROUN D- In recent years,
user-submitted
recipe sites havebecome popular.

Exam ple: when people interacting with smart
speakers?

- Recipes with
text-image paired
instructions.

- Foodst u f f s ar e om it t ed in some instructional
steps.

- Recipe website:
Haodou- The appearance of the same ingredients(in the images) may differ.
pot at oPrepare two potatoes.

Cut them into pieces.

Boil for 5 minutes.

- Even if the images are similar, the food (in- difficult to understand.
the text) might be different in different- However, those omitted entities in text descriptions
situations.
are sometimes shown in the at t ach ed im ages.

If we want to su pplem en t f ood in t ext , we need tor ecogn ize f ood in instructional images.
PotatoSpring onion

Beans

PROPOSED M ETH OD - FOOD RECOGN I TI ON BASED ON SI M I LARI TI ES- From the sub-dataset, select the sentence/image pair that is closest tothe target.
- The score is calculated by integrating the similarity of both thesentence and image vectors- Food Recognition Based on
Sen t en ce Sim ilar it y an dIm age Sim ilar it y
- Text data:- Sentence2vec? 100-D vector- Image data:
VGG16fc-layer? 4096-D vector .
- Sub-database:- Select 300 representative
instructional sentences withimages from the recipe dataset.
- For each sentence:- select 10 sentences that aresimilar to it- obtain the ten pairs ofsentence and image.

- At the last, the omitted food entity in the target sentence is
supplemented with the ingredients from this intersect.

- From each representative
sentence and its similar tensentences, a set of ingredientscontained in those sentences isextracted.
- The purpose: collect the setof ingredientscan be a target of the cooking
steps expressed in similarsentences.

EXPERI M EN T AN D EVALUATI ON- The example results of the sentence embedding method
for calculating text similarity.

- Evaluation: compare the result with m an u ally labeled results andcompute the intersection.
- Our method:67.55%- Can supplement arbitrary food classes appearing in thedataset- (Baseline) Ordinary Inception V3:43.57%- Cannot supplement food that are not included in the 10classes over which the model is trained.

CON CLUSI ON- Con t r ibu t ion s:
1. We construct a dataset of Chinese recipes consisting of 12,548 recipes.
2. We develop a recipe Named Entity (r-NE) recognizer [14] in Chinese
3. To solve the difficulty of recognizing food in different cooking stages, we
propose a method of obtaining food entity candidates from other steps that are
similar to the target step, both in sentence similarity and image featuresimilarity.

- Future work
- Since food states change over time during cooking, we could use thisfeature to greater effect in order to improve the identification of food.
- We would like to use the relationship between text and images toenrich the information content and structure of recipes so as to be
more conducive to the application of recipe retrieval or automatictranslation.