MIRecipe: A Recipe Dataset forStage-Aware Recognition ofChanges in Appearance of Ingredients
Yixin Zhang1, Yoko Yamakata2 and Keishi Tajima1
1Kyoto University 2The University of Tokyo1

Research Background• In recent years, user-submitted recipe sites
have become popular.

• Recipes with one-to-one correspondence
between images and texts• rich and valuable multimedia information.

• Recipe website: Haodou• About 400,000 recipes• Every instructional step is associated with animage.

source: https://www.haodou.com/recipe/1190778

2

Research Problem• Ingredients are omitted in some instructional steps.

• This can make the recipe or a particular procedure di cult to understand.
• For example, when people interacting with smart speakers.
• However, those entities omitted in text descriptions are sometimes shown in the attachedimages.

Extract ingredient
information from images• Therefore, we need to recognize ingredient in instructional images.
ffi

3Dif iculty of the Problem• Basic image recognition method:• Good for entities like tools: shapes of tools are stable

Beginning Stage

Intermediate Stage

Finishing Stagef4

Dif iculty of the Problem• Basic image recognition method:• Good for entities like tools → applying to tool recognition• Di cult for ingredient (ingredient shapes are changing as being cooked) → needadjusting• Images located in di erent positions in the cooking procedure of the same ingredient
may be very di erent.

ff

ff

fffi

• Thus, we would like to use the features of relative position of steps in the whole recipe.

5Research Purpose• The existing image recognition method barely takes the problem of objects
changing in appearance into account.

• Main works• Construct a tex -image instructional recipe dataset: MIRecipe.
• Stage-aware image recognition method• Recognize appearanc changing ingredients in recipe instructional images.
• In order to improve the recognition accuracy.
e­t­

6

Dataset• Recipe Dataset Statistics (up to now)
• Recipe Num. : 398,597• Ingredient Class Num. : 35,319• Instruction Num.(text and image): 3,745,544

7

Dataset• #Ingredient classes used in thisexperiments: 20• high frequencies ofoccurrence

Potato, ginger, onion, pork,shrimp, chicken, corn, carrot,
eggplant, shallot, tofu, spinach,sauce, chili, bread, dough, sh,egg, cucumber, soybean• #Images of 20 classes: 35,401fi

8

Image Classi ication• We compute the relative position of the steps in the whole recipe• E.g., Step No.4 out of 15 steps: 0.267

StepNum .
RelativePosition =TotalStepNum .

• Images are divided into 3 subsets according to their relative positions in recipes.
Subset 1:
Beginning Stage
Subset 2:
Intermediate Stagef

Subset 3:
Finishing Stage

Recipe A:
Step 1
Step 2
Step 3
Step 4
Step 5
Step 6
Step 7Step 8
Step 9
Step 10
Step 11
Step 12

Recipe B:
Step 1
Step 2
Step 3
Step 4
Step 5
Step 6
Step 7
Step 8
Step 9Recipe C:
Step 1
Step 2
Step 3
Step 4
Step 5
Step 6
Step 7
Step 8
Step 9
Step 10
Step 11
Step 12
Step 139

Image Features Visualization• Visualized embedded feature for the ingredient class per stage.
• Images of the same stage have similar characteristics.

Beginning Stage&Finishing Stage:
distributed
separately

Beginning
Stage: denser,
Finishing Stage:
scattered

Tofu

Potato10

Stage-Aware Ingredient Recognition• Stage-Aware Recipe Image Recognition For Ingredients Changing in Appearance• Overview of the baseline method and proposed methods:

Pattern 1
Pattern 2
Pattern 3

11

Experiment Result• Comparison of methods

12Experiment Result• Comparison of methods

13Conclusion• A recipe dataset• contains both text and image data for every cooking step.
• A recognition method• for ingredients whose appearance changes with the cooking progress.
• We only focused on the single-label recognition in this work.
• Experiments with multi-label data is also an important remaining issue for future work.

14Thanks for listening!

15