MIAIS: A Multimedia Recipe Datasetwith Ingredient Annotation at Each
Instructional Step
Yixin Zhang1, Yoko Yamakata2 and Keishi Tajima1
1Kyoto University 2The University of Tokyo

1

Background• In recent years, more and more datasets onrecipes are constructed.

• Food-101 dataset, Recipe 1M+, UEC Food256:• mainly focus on completed dish images
after cooking rather than images in recipe
instructions.

• Cookpad dataset:• 1, 642, 450 images of completed dishes• 3, 105, 594 instructional images

source: https://cookpad.com/recipe/6100648

2

Problem and Purpose• Most of the datasets:• No su cient multimedia information (text and image) on the cooking
instructions of the recipe.

• However, the instructional information is one of the most important and uniquecharacteristics of recipe data.

• MIAIS Dataset• Goal: construct a recipe dataset with su cient multimedia data and theannotations to them for every cooking step (based on Cookpad dataset)• Applications: cooking ow graph generation, recipe text generation, and cooking
action recognition

ffi

flffi

3

MIAIS Dataset• Multimedia recipe data

• 12,000 recipes in Japanese• The text and image data are originated from Cookpad.
• Recipe selection rule:• Recipes that have at least ve cooking steps• Recipes that contain a text description and an image for every stepfi4

MIAIS Dataset• 109,547 text-image paired instructional steps in total

5

Cooking Flow Graph• A challenge in understanding the cooking process from recipes:• present the complicated operations of cooking activities in a succinct
work ow.

The original form sometimes
makes it di cult to understandrecipes directly and e ciently.

• A common solution to this issue is to generate a ow graph to help them
6
fl

ffi

ffi

fl

understand the cooking process.

Cooking Flow Graph• The ow graph is usually

generated by extracting
the relationships
between cooking actions
and ingredients.

• Most existing methods

only focus on theingredients that appear
in the instructional text.

fl7

Cooking Flow Graph• To generate ow graph from multimedia recipes , one way is to focus on the ow ofingredients in recipes.

• Su cient information is needed.
• However, the ingredient information in text tends to be insu cient or omitted
when written by users.

flffi

flffi

8

Our Annotation Method• Our new annotation rule:
Each instructional step isannotated withall ingredients thatphysically exist in theintermediate products
processed in the stepno matter whether they are
mentioned in the instructional
text or not.

9

Flow Graph Generation• Cooking ow graphs• One essential component of MIAIS, which is derived from our annotation data
• We track the paths of ingredients in order to generate the cooking ow graph.
• If a speci c ingredient• is contained in a previous step;• can also be detected in one of the subsequent steps• we can consider that there is continuity between the two steps.
• The path is created for each ingredient and the nal ow graph is a union set of allthe paths.

fl

flfi

flfi

10

Flow Graph Generation• “Potato”: 1-2-3-4-7-8• “Beef”: 3-5-6-7-8• The ow graph of this recipe

can be generated based on theabove tracking information.

• The order in which the actual
cooking is performed is notsequential, but with several
branches.

fl11

Conclusion• A multimedia recipe dataset with ingredient annotation for every instructional step
• Containing both text and image data for every cooking step• Supplementary information about text-image pair of every step
• Cooking ow graphs• Directly derived based on the annotation information• These ow graphs can become the ground truth for the method of recipe ow graphgeneration

fl

fl

fl

12

Thanks for listening!

13