ReciPic: A Tool for Generating Infographic from Recipe Procedure Text
Lechang ZHANG† Takayuki ITOH††Graduate School of Humanities and Sciences, Ochanomizu University
2-1-1 Otsuka, Bunkyo-ku, Tokyo 112-8610, Japan
E-mail: †{zhang.lechang, itot}@is.ocha.ac.jp,
Abstract Food is one of the basic human needs and an important part of human culture. With the development of Internet, it
has become increasingly common for people to share recipes on recipe websites. However, in most cases, these recipes contain
only text information about the cooking procedures. It is difficult for users to have a big picture about the food materials andcooking methods at a glance. In this paper, we propose a tool for generating infographic from recipe procedure text, to
supplement the pure text information on the recipe websites. This tool accepts one step text of the cooking procedures, andthen segments the text stream into appropriate units. After that, it extracts the food material, cooking tool, cooking action
segmentations and label them with F, T, Ac (F: Food material, T: Cooking tool, Ac: Cooking action) respectively. Then, it
analyzes the grammatical relationship between the extracted words. Finally, it matches the words with prepared pictures andplace them with appropriate positions in an infographic and represents it to users.
Keyword Recipe · Cooking，User support，Visualization

1. Introduction

2. Related Work

Food is one of the most important part of our life. Human has
been using simple processing methods like fire to process food
before eating for more than 250,000 years. Cooking, as one of thefood processing ways, can not only improve food safety, but also
increase the appetite of consumers. In recent years, more and more
people choose to use recipe websites to look for recipes, cooking
advices or to share their own cooking ideas. The most popular
recipe websites in Japan is Cookpad [1], which has over 50 million
monthly users in Japan, and an additional 42 million monthly users
across 74 countries, with 3 million recipes. In order to support sucha large user community and improve their using satisfaction, it isnecessary to analyze the recipes in recipe websites.
A recipe in recipe websites simply contains the recipe name,ingredients, and cooking procedures. Among these components,
cooking procedures is the key part that users will follow. However,they are often displayed by pure text in the websites. It is difficult
for users to grasp the valuable information and have a big picture
about the food materials and cooking methods at a glance. For
users who are new to cooking, it is especially hard to understandbecause they are not familiar with the process, food materials andthe usage of cooking tools. Even though there are some recipes
which also contain picture descriptions, those pictures usually show
only the finished dishes to attract users rather than the cooking
process. It is still not easy for users to comprehend the cooking
procedures.
In this paper, we propose ReciPic, an interactive visualization tool
to make the cooking process clearer, simpler and more
understandable. Our tool allows users to enter recipe procedure
step text, and then transform it into the corresponding infographic.
Infographics, just as its name implies, are graphics which convey
information. They are visual representation of any kind of
information or data. We apply infographics to represent text
information based on the fact that visual information is easier tounderstand than text information. Abundant research conducted inlearning area showed how efficient graphics are in improving thelearning comprehension and efficiency of students [2]. Reading
recipes and cooking are also a form of learning. With the help of
infographics, we suppose users can understand the procedure text
information effectively.

2.1 Procedure text analysis
In order to make computers understand human languages better,
there are plentiful research conducted in the field of natural
language processing to analyze text and extracting semantic
structure that computers can easily read and understand.
Maeta et al. [3] described a method which takes recipe procedure
text as research object and converting it into a flow graph. Thismethod first performs word segmentation on the procedure text,
then identify the recipe named entities such as food, tools, action
by the chef, etc. Finally, it builds the flow graph that uses recipenamed entities as vertices and relationships between them as edges.
Hamada et al. [4] also proposed a similar method but taketextbooks for cooking programs as their research object. Whereas
Terashima! et al. [5] analyzed the install instruction text andgenerated a flow graph based on their analysis.
These studies focus on procedure text analysis and flow graphgenerating. Our research also refers to their text analysis method
but ultimately generates the infographic to assist users.

2.2 Recipe corpus
Recognizing useful terms, also known as Named Entity
Recognition (NER), is a pretty important task in natural language
processing. There are many studies that analyze newspaper articles
and extract named entities out of it. However, little research has
been done in analyzing recipe procedure texts.
Sasada et al. [6] discussed the definition of recipe terms and built
a recipe corpus based on their definition. They also developed atool that can automatically recognize the named entities of recipe
procedure text.
Harashima et al. [7] introduced a new recipe dataset called
Cookpad Parsed Corpus, which contains linguistic annotations forcookpad recipe procedure texts. They randomly extract recipes
from cookpad recipe dataset and annotated recipe proceduresentences with morphemes, named entities, and dependency
relations.
These studies concentrate on extracting useful information out ofrecipe procedure texts and provide a recipe corpus. Our researchalso refers to their recipe terms definition and classify different

word segmentation parts into different groups by their definition.

2.3 Infographic
Infographics are graphical representation of text or data. They arebroadly used in research to display information efficiently. Iwata etal. [8] proposed a method which uses infographic to visualize thesource code to help coding beginners to learn faster. Wu et al. [9]discussed a system which takes recipe website information as input
and generates infographics of recipe procedure texts.
These studies use infographics to improve people’s understanding
in one particular material. Our research also makes use of theadvantages of infographics, and proposes an interactive tool totransform text into infographics.

First, ReciPic reads the input and parses it to a Node.js childprocess that runs mecab segmentation instruction in the terminal.
Then it will generate the segmentation result as you can see in
Figure 1.

3. Proposed Method
3.1 Text analysis
To understand the meaning of the input, ReciPic will perform text
analysis on it.
In text analysis, it will first segment the input into appropriate
units and extract the useful parts, then classify different parts into
three categories (namely food materials, cooking tools and cooking
actions), and finally analyze the dependency relation between thesegmented parts. In addition, ReciPic will also complement theomitted cooking tools before generating the infographic.

3.1.1 Word segmentation
In order to generate infographic from the input recipe procedure
text, we first analyze the input text and segment the text into usefulparts.
There are two popular Japanese text analysis tools. One is calledkytea [10] and the other is called mecab [11]. We chose mecab forthe reason that mecab has a dictionary called mecab-ipadicNEologd which is constantly updated, while kytea has not been
updated for many years. Here, we have randomly picked 5 recipes
from cookpad to compare the accuracy of kytea and mecab(withmecab-ipadic-NEologd dictionary). The accuracy will be evaluatedby Formula (1), and the result is shown in Table 1.
Accuracy of word segmentation =Number of correct food/tool/action
Total number of food/tool/action(1)

Table 1. Recipe terms definition
Accuracy

Kytea

Mecab

Accuracy of food67.31%

96.15%

Accuracy of tool

71.43%

91.43%

Accuracy of action

97.44%

91.03%

Total Accuracy

82.42%

92.73%

As we can see, the total accuracy of word segmentation of kytea
is about 10% lower than mecab. Kytea performs poorly especiallyin food and tool segmentations. Even though kytea does well inaction segmentations, mecab is good enough for dealing with thecooking actions in most recipe procedure texts.
To read and analyze user's input on the server side, we chose
server side language Node.js with its framework express to keep
the language consistency between client side and server side. We
will take a user input as an example to explain the word
segmentation process in ReciPic.
[Input]: 鍋に⽔、本だし、料理酒、みりん、濃⼜醤油、上⽩糖を⼊れて、⼿⽻元を⼊れて強⽕にかける. (Put water,
hondashi, cooking sake, sweet sake, dark soy sauce, caster sugar
and the chicken wings in the pot, and set the strong fire.)

Figure 1. Word segmentation result
It consists of segmented words, parts of speech, pronunciation,etc. EOS in the end means End of Sentence. We simply want thesegmented words such as cooking tools(noun), foodmaterials(noun) and cooking actions(verb) which are useful ingenerating infographics. Therefore, we delete the unnecessary partsand connect the remaining parts with spaces. The word
segmentation result is shown below:[Output]: 鍋(pot) ⽔(water) 本だし(hondashi) 料理酒(cookingsake) みりん(sweet sake) 濃⼜醤油(dark soy sauce) 上⽩糖(caster sugar) ⼊れ(put) ⼿⽻元(chicken wings) ⼊れ(put) 強⽕(strong fire) かける(set)

3.1.2 Word classification
After word segmentation, we successfully got the words needed
for generating the corresponding infographic. But ReciPic has no
idea about the meaning of each part. We need to classify the words
in the word segmentation result to let ReciPic understand theirgeneral meaning.
Sasada et al. [6] and Harashima et al. [7] defined recipe terms intheir work, I will refer to part of their recipe terms definition whichis shown in Table 2.
Table 2. Recipe terms definition
Recipe terms

Tag

Explanation

Food

F

F represents foodmaterials, e.g. egg,
rice, etc.

Tool

T

T represents
cooking tools, e.g.
fry pan, ricecooker, etc.

Action of chef

Ac

Ac represents theactions of chef,
e.g. put, cut, etc.

The task of word classification can also be regarded as the taskof Named-Entity Recognition, or NER for short. Here ReciPic used
an NER tool for Japanese recipe terms recognition called PWNER[12]. PWNER only accepts input that has been segmented andseparated by space. This is the reason that we separated the word
segmentation result with spaces.
There still are some words such as 本だし(hondashi) and 濃⼜醤油(dark soy sauce) that could not be recognized correctly by
PWNER. To solve this problem, we build a word classification
dictionary(Shown in Table 3.) which contains the common
vocabularies of food, cooking tool and cooking action tocomplement PWNER’s result.

grammatically related unit. It is worth mentioning that the "0" in"かける(set)/12-0" means "かける(set)" is the last semantic unit.
Finally, ReciPic will combine the word classification result withthe dependency parsing result, and check if there is any omitted
part in the input, and complement it by a prepared dictionary, thengenerate the text analysis result.
A text analysis result is shown below：[Output]: 鍋(pot)/T/1-8 ⽔(water)/F/2-3 本だし(hondashi)/F/3-4
料理酒(cooking sake)/F/4-5 みりん(sweet sake)/F/5-6 濃⼜醤油
(dark soy sauce)/F/6-7 上⽩糖(caster sugar)/F/7-8 ⼊れ(put)/Ac/
8-12 ⼿⽻元(chicken wings)/F/9-10 ⼊れ(put)/Ac/10-12 強⽕(strong fire)/T/11-12 かける(set)/Ac/12-0

3.2 Infographic generating
3.2.1 Rules for generating infographics

Category

Vocabularies

Food

本だし(hondashi), 濃⼜醤油
(dark soy sauce), etc.

Tool

ガスバーナー(gas burner),
電⼦レンジ(microwave),etc.

In order to generate the infographics, we prepared an image
database which contains the illustrations of food and tools. Eachdata is stored as an object in MongoDB, it has two properties —name and image address — to indicate the name and image of acertain food or tool. The images themselves are stored in the cloud.
It is worth mentioning here that in order to solve the problem ofspelling inconsistencies in Japanese, I have prepared an array of all
possible spellings for each food/tool name. In addition, becausesome actions need to be completed by the combination of two
tools, I also prepared images of the combination of tools. The
details are shown in Table 4.

Action of chef

切る(cut), 炙る(toast), etc.

Table 4. Image database

Table 3. Word classification dictionary

The output of word classification step is shown below:[Output]: 鍋(pot)/T ⽔(water)/F 本だし(hondashi)/F 料理酒(cooking sake)/F みりん(sweet sake)/F 濃⼜醤油(dark soy sauce)/
F 上⽩糖(caster sugar)/F ⼊れ(put)/Ac ⼿⽻元(chicken wings)/F
⼊れ(put)/Ac 強⽕(strong fire)/T かける(set)/Ac

3.1.3 Dependency parsing
Now, we have seen how ReciPic performs word segmentation andclassification on the input text. However, as we can find in the
results of last step, we have already lost the dependency relations
between the extracted words. For example, we do not know whichfood is operated by the action “⼊れる(put)”, nor which tool isrelated to this action. Therefore, we also need to performdependency parsing on the input text to determine the relations
between words.
Here, ReciPic will use a Japanese dependency parsing tool calledcabocha [13]. It is worth mentioning that cabocha is built based on
mecab. They can use the same dictionary for text analysis. This
will not cause inconsistent results between word segmentation anddependency parsing. In Wu et al.’s [9] research, such errors
eventually led to the generation of incorrect inforgraphic.
A dependency parsing result is shown below:[Output]: 鍋(pot)/1-8 ⽔(water)/2-3 本だし(hondashi)/3-4 料理
酒(cooking sake)/4-5 みりん(sweet sake)/5-6 濃⼜醤油(dark soy
sauce)/6-7 上⽩糖(caster sugar)/7-8 ⼊れ(put)/8-12 ⼿⽻元(chicken wings)/9-10 ⼊れ(put)/10-12 強⽕(strong fire)/11-12 かける(set)/12-0
Each semantic unit is marked with two numbers connected withhyphen. The number before the hyphen indicates the order of thisunit, and the number after the hyphen indicates its grammaticallyrelated unit. Take "鍋(pot)/1-8" as an example, "1" shows "鍋(pot)"is the first semantic unit, and "8" shows "⼊れ(put)" is its

Data examples{"name": ['ねぎ', 'ネギ', '葱'], "url": "negi.png"}{"name": ['ボウル箸', '箸ボウル', 'ボウルはし', 'はしボウ
ル'], "url": "bowl-chopstricks.png"}
After the illustrations of each food and tools are determined,
ReciPic will combine them together to generate the final
infographic. This research referred the position relationships
between food and tools defined by Wu et al. [9].
The details are shown in Figure 2.

Figure 2. Rules for generating infographics
Which rule to pick is determined by the cooking action relevantwith the food and tool. For example, cooking action ⼊れ (put)means putting the food into the tool, so rule(d) will be picked.
Action 切る (cut) means cutting the food on the chopping-board,so rule(b) will be picked. Action 洗う (wash) means washing thefood under the water-tap, so rule(e) will be picked.

3.2.2 Algorithm for processing text analysis result
ReciPic will generate infographic based on each combined unit offood, tool and action. We will introduce the algorithm forprocessing the text analysis result into combined unit.
The overview of the algorithm is shown in Figure 3. Thisalgorithm accepts the text analysis result as input, and first splits
the input string by space and stores the result in an array
textAnalysisArr. Then it traverses the array, finds each element
marked with Ac. For each Ac tagged element, find its related foodand tool, combine them as a F/T/Ac unit, and then combine each F/
T/Ac unit into an object containing a set of units. Next, it traverses
the units object, and for the unit that contains no food or tool, findits related unit according to the related cooking action of this unit’s
action — the related cooking action of one action is decided by the
result of dependency parsing — then use the prepared omittedword completion dictionary to complete the remaining units withno tool. Finally, return the processed result.

Figure 4. Infographic of input 1

Figure 3. Algorithm for processing text analysis result
A result of the processed infographic generating units is shown inbelow:{‘1’: {
F: [‘⽔(water)’, ‘本だし(hondashi)’, ‘料理酒(cooking sake)’,‘みりん(sweet sake)’, 濃⼜醤油(dark soy sauce), ‘上⽩糖(castersugar)’, ‘⼿⽻元(chicken wings)’],
T: [‘鍋(pot)’],Ac: [‘⼊れ(put)’],},‘2’: {F: [], T: [‘強⽕(strong fire)’], Ac: [‘かける(set)’]},}
Finally, ReciPic will loop through each unit, find all theillustrations of food and tools in the image database, and look up
for the infographic generating rule according to the action, then
combine the illustrations of food and tools based on thecorresponding rule. For the unit that contains a group of food,
ReciPic will provide a 'next' button that allows users to see the foodin sequence. And for the unit that contains word ‘⽕(fire)’ in its tool
list, ReciPic will display a fire illustration at the bottom.

As we can from the result, food ‘⽔(water)’, 本だし‘(hondashi)’,‘料理酒(cooking sake)’, ‘みりん(sweet sake)’, ‘濃⼜醤油(darksoy sauce)’, ‘上⽩糖(caster sugar)’ and ‘⼿⽻元(chicken wings)’
has been successfully extracted and present in the infographic. Tool
‘鍋(pot)’ and action ‘⼊れ(put)’, ‘かける(set)’ has also been
extracted and present in the result. Besides, the result showed theorder of food to be put, users could see them in sequence byclicking the next button. However, the infographic did not show thesequence of two actions ‘⼊れ(put)’ and ‘かける(set)’. This isbecause the infographic generating rules of ReciPic could only
combine food with tool, when there are two tools to be combined,
it will check if there is any special tool like ‘⽕(fire)’ and add it at
the bottom. We aim to solve this problem by expanding the
infographic generating rules of ReciPic.
[Input2]: ご飯、卵、⼩ネギ、ベーコンを全てボウルに⼊れよく混ぜる(Put rice, eggs, small leek and bacon into bowl andmix them well)[Infographic]: The result is shown in Figure 5.

4. Experimental result
We used two different inputs to show the result of our tool
ReciPic.
[Input1]: 鍋に⽔、本だし、料理酒、みりん、濃⼜醤油、上⽩糖を⼊れて、⼿⽻元を⼊れて強⽕にかける. (Put water,
hondashi, cooking sake, sweet sake, dark soy sauce, caster sugar
and the chicken wings in the pot, and set the strong fire.)[Infographic]: The result is shown in Figure 4.

Figure 5. Infographic of input 2
As we can from the result, food ‘ご飯(rice)’, 卵‘(egg)’, ‘⼩ネギ
(spring onion)’ and ‘ベーコン(bacon)’ has been successfully
extracted and present in the infographic. Tool ‘ボウル(bowl)’ andaction ‘⼊れ(put)’, ‘混ぜる(mix)’ has also been extracted andpresent in the result. Besides, the result showed the order of food tobe put, users could see them in sequence by clicking the next
button. However, ReciPic presents action ‘混ぜる(mix)’ by
combining its tool bowl and chopsticks with all the food materialsof last action instead of mixing the food materials that has alreadybeen put in. This is because it is difficult to present the state of all
the food materials that has been put in. We will discuss that in thissituation, whether to combine all the food pictures and put them inthe bowl or use other method in the future.

5. Conclusion
In this paper, we introduced ReciPic, which is an interactive toolthat allows users to enter recipe procedure text, and transforms it
into infographic then represents it to users. We explained the key
components of ReciPic in details, and experimented it with two
input examples.
From the experimental results we can see that there are still many
issues that can be improved in ReciPic. In the future, we plan tosolve the problems we mentioned in experimental result, test
ReciPic with a wider range of input and complement the text
analysis dictionaries and the illustration database, to make it more
applicable for more recipe texts. In addition, we will extract otherwords such as adjective and the quantity of ingredients, and discuss
how to present them in the infographic.

Referencesど[1] Cookpad company, https://cookpad.com/.
[2] ANGLIN, Gary J.VAEZ, Hossein; CUNNINGHAM, Kathryn
L. Visual representations and learning: The role of static andanimated graphics. Handbook of research on educational
communications and technology. Routledge, 2013. p. 854-905.
[3] MAETA, Hirokuni; SASADA, Tetsuro; MORI, Shinsuke. A
framework for recipe text interpretation. Proceedings of the 2014
ACM International Joint Conference on Pervasive and Ubiquitous
Computing: Adjunct Publication. 2014. p. 553-558.
[4] HAMADA, Reiko, et al. Structural analysis of cooking
preparation steps in Japanese. Proceedings of the fifth international
workshop on on Information retrieval with Asian languages. 2000.
p. 157-164.
[5] 寺島太平，苅⽶志帆乃，佐藤哲司，⼿順テキストの可視化を⽬的とした構造解析⼿法, 第10回データ⼯学と情報マネジメントに関するフォーラム(DEIM2018), P7-6(2018).
[6] 笹⽥鉄郎，森信介，⼭肩洋⼦，前⽥浩邦，河原達也，レ
シピ⽤語の定義とその⾃動認識のためのタグ付与コーパスの構築，⾃然⾔語処理，Vol. 22， No. 2 (2015).
[7] HARASHIMA, Jun; HIRAMATSU, Makoto. Cookpad parsed
corpus: Linguistic annotations of Japanese recipes. Proceedings ofthe 14th Linguistic Annotation Workshop. 2020. p. 87-92.
[8] 岩⽥ま か，隼⽥尚彦，向⽥茂，齋藤⼀，安⽥光孝，インフォグラフィックを⽤いたソースコードの可視化，教育シ
ステム情報学会第39回全国⼤会, F5-1, 449-450(2014).
[9] 呉⼦昕, ⽜尼剛聡, 料理レシピの理解⽀援を⽬的としたインフォグラフィックの⾃動⽣成, 第13回データ⼯学と情報マ
ネジメントに関するフォーラム, 2021.03.
[ 1 0 ] K y t e a , K y o t o Te x t A n a l y s i s To o l k i t , h t t p : / /www.phontron.com/kytea/.
[11] Mecab, Yet Another Part-of-Speech and Morphological
Analyzer, https://taku910.github.io/mecab/.
[12] PWNER, PointWise-based Named Entity Recognizer, http://
www.ar.media.kyoto-u.ac.jp/tool/PWNER/.
[13] CaboCha, Yet Another Japanese Dependency Structure
Analyzer, http://taku910.github.io/cabocha/.