Proceedings of the 12th Conference on Language Resources and Evaluation (LREC 2020), pages 5187–5194
Marseille, 11–16 May 2020c European Language Resources Association (ELRA), licensed under CC-BY-NC

English Recipe Flow Graph Corpus
Yoko Yamakata∗ , Shinsuke Mori† , John Carroll‡∗

The University of Tokyo, 7-3-1 Hongo, Bunkyo-ku, Tokyo, Japan
yamakata@mi.u-tokyo.ac.jp†

Kyoto University, Yoshida-Honmachi, Sakyo-ku, Kyoto, Japan
forest@i.kyoto-u.ac.jp‡

University of Sussex, Falmer, Brighton, UK
J.A.Carroll@sussex.ac.uk
Abstract

We present an annotated corpus of English cooking recipe procedures, and describe and evaluate computational methods for learningthese annotations. The corpus consists of 300 recipes written by members of the public, which we have annotated with domain-specificlinguistic and semantic structure. Each recipe is annotated with (1) ‘recipe named entities’ (r-NEs) specific to the recipe domain, and(2) a flow graph representing in detail the sequencing of steps, and interactions between cooking tools, food ingredients and the products
of intermediate steps. For these two kinds of annotations, inter-annotator agreement ranges from 82.3 to 90.5 F1 , indicating that ourannotation scheme is appropriate and consistent. We experiment with producing these annotations automatically. For r-NE tagging wetrain a deep neural network NER tool; to compute flow graphs we train a dependency-style parsing procedure which we apply to theentire sequence of r-NEs in a recipe. In evaluations, our systems achieve 71.1 to 87.5 F1 , demonstrating that our annotation scheme islearnable.
Keywords: cooking recipe corpus, English recipes, recipe named entity, recipe flow graph, procedural text annotation

1. Introduction
Procedural text describes in an objective way how to carryout some task. Although this kind of text is widespread, it
is found particularly in instruction manuals and cookbooks.
The information in procedural text could be used to support a wide range of computer-based applications involving intelligent search, reasoning, and non-textual forms ofhuman-computer interaction; but first the text would need
to be transformed into a more computationally tractable
representation. At minimum, this representation would represent domain entities, actions, and relationships between
them.
A flow graph is one kind of computationally tractable representation; it contains steps linking actions with entities,
and specifies the obligatory sequencing relationships between steps. A flow graph abstracts away from differencesin surface linguistic expression that are irrelevant to correctly performing a procedure. For example, when cooking
a dish, the instructions add diced carrots to the pot and dicecarrots and add to the pot are different textually, but would
correspond to the same flow graph since they involve thesame actions carried out in the same order.
Our research focuses on the cooking domain, and specifically the procedural part of cooking recipes. Our generalapproach and some of our computational techniques are inspired by recent research on recipes written in Japanese(Mori et al., 2014). We extend that work by adapting theannotation to English, annotating more data, investigating
the consistency of annotation, and improving accuracy ofprocessing.
In this paper, we describe how we annotated a corpus of
300 English recipes, which were contributed to a recipes
web site by members of the public. We first annotated each

recipe with ‘recipe named entities’ (or r-NEs). These r-NEs
are specific to the recipe domain, and account for cooking
actions, cooking implements (tools), ingredients, intermediate products, durations and quantities. The r-NE annotation formed the basis for a second level of annotation: aflow graph representing in detail the relationships between
the r-NEs (e.g. an action carried out with a certain tool andfood items) and the sequencing of the actions1 .
We used the annotated corpus to investigate approaches
for automatic r-NE tagging and flow graph computation.
For r-NE tagging, we train a deep neural network NERtool, treating the task as a sequence labelling problem. To
derive the flow graph representation, we consider the entire sequence of r-NEs in a recipe as a single input string,and extract a flow graph using a dependency-style maximal
spanning tree parser. In contrast to standard sentence-bysentence parsing, the parser creates flow graph edges thatspan different sentences in exactly the same way as withinsentence edges; this means we do not need any extra processing stages such as co-reference resolution. Our evaluation results are very promising.
The rest of the paper is structured as follows. Section 2 surveys approaches to representing procedural text, and cooking recipe text annotation and processing. Section 3 describes the r-NE tagset, annotation, and an evaluation of automatic r-NE tagging accuracy. Section 4 describes the flow
graph representation, annotation, the parsing algorithm forcomputing a flow graph, and an evaluation of its accuracy.
Section 5 discusses the main findings and proposes directions for future research.
1

The annotated corpus of English recipes is available
athttps://sites.google.com/view/yy-lab/resource/english-recipe-flowgraph5187

2. Related Work
There has been much research into representing the semantics of natural language sentences (Banarescu et al., 2013,
inter alia) and developing parsers that output such representations (Flanigan et al., 2014, for example). Semantic representations encode the meanings of linguistic units
within a sentence, but do not attempt to capture domainspecific constraints or real-world context. In some domains
and applications these latter factors may be very important. (For example, in cooking, a mixture can sometimes bebeaten and sometimes not, depending on which ingredients
have been added to it). For this reason, approaches to processing text describing procedures often do not attempt toconstruct general semantic representations, but instead construct more specialised domain- and genre-specific representations. For example, Momouchi (1980) proposed representing the meaning of procedural text as a flow graph,and described algorithms to convert text documents to flow
graphs. Hamada et al. (2000) adopted a similar graph representation for analysing recipes.
More recent studies have created annotated corpora ofprocedural text, in order to build machine learningbased systems that extract entity and action information.
In the domain of cooking recipes, such information has been shown to be useful for the socalled ‘smart kitchen’ (Hashimoto et al., 2008), cooking robots (Bollini et al., 2013), etc.
In one recent
study, Mori et al. (2014) produced flow graph annotations for 266 cooking recipes written in Japanese. That
corpus has been used for testing empirical methods
for natural language understanding (Maeta et al., 2015)
and intelligent search (Yamakata et al., 2013), Similarly,
Jermsurawong and Habash (2014) described a corpus ofingredient trees for recipes, and evaluated methods forconstructing these automatically.
There have beensome attempts at unsupervised processing of recipes(Kiddon et al., 2015, inter alia) but the outputs are muchless rich than those obtained by supervised methods.
This paper concerns a flow graph corpus for recipes
in English; the flow graphs are based on the proposals of Mori et al. (2014) in which the graph nodes are‘recipe named entities’ (r-NEs). The r-NE types coverdomain entities such as ingredients; in addition, Moriet al. argue that r-NE types should also cover actions by
the cook / chef (which are often expressed as verbs).
Unlike the set of NE types commonly used for general text (Sang and Meulder, 2003)—person name, location, organization, etc.—r-NEs are an example of adomain-specific NE definition. Domain-specific NE definitions are widely used in fields such as bioinformatics (Ben Abacha and Zweigenbaum, 2011). A domainspecific NE definition facilitates the development of intelligent applications that process documents in that particulardomain.
The task of identifying NEs is called ‘named entity recognition’ (NER). NER is usually formulated
as a sequence labeling problem, in which each input token is tagged as being inside or outside an NE(and in some approaches, beginning an NE). In thisframework many techniques and refinements have been

Tag Meaning
F
Food
T
Tool
D
DurationQ
Quantity
Ac Action by chef
Discontinuous
Ac2 Ac (English
only)
Af
Action by foodAction by toolAt
(English only)
Sf
Food state
St
Tool state

Explanation
Eatable; also intermediate products
Knife, container, etc.
Duration of cooking
Quantity of food
Verb representing a chef’s action
Second, non-contiguous partof a single action by chef
Verb representing action of a food
Verb representing a tool’s action
Food’s initial or intermediate stateTool’s initial or intermediate state

Table 1: Recipe named entity (r-NE) tags.
investigated (Borthwick, 1999; Sang and Meulder, 2003;
Ratinov and Roth, 2009), and many general-purpose NERtools have been developed. In this study, we use one suchtool, PWNER (Sasada et al., 2015a), which computes probabilities for all possible NE tags based on pointwise prediction, and searches for the best sequence of tags under the
tag sequence constraints. The tool is distinctive in being
trainable on data that has been only partially annotated.

3.

Recipe Named Entities

3.1. Recipe Named Entity Tags
In previous work (Yamakata et al., 2017), we created a corpus of 100 recipes written in English, sampled from theAllrecipes UK/Ireland web site2 and annotated for ‘recipenamed entities’ (r-NEs). The r-NE tag types were based ona set of eight tags originally devised for annotating recipes
written in Japanese (Mori et al., 2014), but with the addition of a further two tags. One is introduced in orderto account for linguistic phenomena that occur in English
and many other languages but not Japanese: discontinuous
multi-word expressions (Constant et al., 2017). The otheris to cover expressions for the actions by automatic cooking
tools (e.g. food processors), which are not yet common in
Japan, so the tag was not necessary for recipes in Japanese.
Table 1 lists the ten r-NE tags that we use for English recipe
annotation. Note that this tag set is compatible with theoriginal.
Figure 1 shows the annotation that would be given to therecipe step Preheat oven to 180 C / Gas mark 4. The tag
suffixes -B and -I (abbreviating Begin and Inside respectively) indicate NE spans according to the IOB2 text chunking representation (Sang and Veenstra, 1999). A tag is assigned to each word or word-sequence designating a single and indivisible object/action/phenomenon in the cooking domain. For example, Gas mark 6 in the example sentence designates the state of the dial of the oven being at 6,
so it is annotated as a single r-NE. Therefore, the first word
Gas is annotated as St-B and the subsequent words markand 6 are both annotated as St-I. The word to is annotatedO because it is Outside any named entity.
25188

http://allrecipes.co.uk/

Preheat oven to 180
Ac-B

C/ Gas mark 4

Dataset100-r
200-r
Overall.

T-B O St-B St-I O St-B St-I St-I O

Figure 1: Example of r-NE annotation.

3.2. Annotating Recipe Named Entities
Our corpus of English recipes consists of the 100 recipes
mentioned above, plus a further 200, which we have selected and annotated recently.
The first, 100-recipe subset (which we call ‘100-r’)
was sampled from Allrecipes, selecting the most popular recipes within each dish category (e.g. ‘Main course’,‘Dessert’) in proportion to the total number of recipes inthat category. These recipes were annotated by an English
native speaker, and all annotations were verified by the firstauthor, as described by Yamakata et al. (2017).
The second, 200-recipe subset (‘200-r’) was sampled fromAllrecipes completely randomly. These recipes were annotated by two Japanese homemakers who had previously annotated Japanese recipes in the study of Mori et al. (2014).
Both are good English readers; in preparation, they refamilarized themselves with the annotation guidelines.
They annotated the first half of 200-r completely manually; the average annotation time was 19 minutes per recipe(including initial preparation). To speed up the workflow,
we automatically tagged the remainder of 200-r using the
PWNER NE recognizer trained on 100-r; the annotatorsthen checked and corrected the recognition results. The
300-recipe corpus comprises 38,062 tokens in total.
To estimate inter-annotator agreement, we asked the second
pair of annotators to re-annotate 100-r independently. Unfortunately we cannot use Cohen’s kappa or related measures, since they assume that each annotation decision is
independent of any other; however, with IOB2 some combinations of tags are illegal, e.g. an Inside tag may only
occur after a Begin or Inside with the same tag prefix. Instead, we took one of the annotators as the gold standard
and evaluated the accuracy of the other with respect to this.
We measured the agreement as 89.9% precision, 92.2% recall, and 90.5 F1 . We consider this to be a high level ofagreement.

3.3. Automatic Recognition of r-NEs
Having created the 300-recipe r-NE annotated English corpus, we investigated the extent to which the annotation
supports NER, and whether the r-NE tagging accuracy iscomparable to previous results for Japanese recipes. Although PWNER works well for r-NE tagging, we decided instead to experiment with the more modern deeplearning named entity recognizer BERT-NER3 , a state-ofthe-art tool based on the BERT neural network architecture (Devlin et al., 2019). For this evaluation we used 10fold cross-validation, splitting the data into 80% training,
10% hyperparameter tuning, and 10% test. The overall F1
for r-NE recognition is 87.6. Accuracies by recipe subset
are shown in Table 2. Table 3 shows accuracy by tag; F1
ranges from a low of 17.1 for the rarest tag At to 92.7 forhttps://github.com/kyzhouhzau/BERT-NER

Recall
87.4%
87.9%
88.8%

F1
84.4
86.7
87.6

#NEs
5629
9728
15435

Table 2: r-NE recognition accuracy by corpus subset andoverall.
Tag
F
T
D
Q
Ac
Ac2
Af
At
Sf
St

Total

Precision
90.5%
87.7%
87.4%
70.9%
92.3%
43.8%
51.4%
60.0%
66.1%
81.6%
86.5%

Recall
93.9%
90.1%
90.2%
76.8%
93.1%
46.8%
50.4%
10.0%
71.4%
82.5%
88.8%

F1
92.1
88.9
88.8
73.7
92.7
45.3
50.9
17.1
68.6
82.0
87.6

#NEs
5007
1904
589
529
4977
178
255
15
1105
876
15435

Table 3: r-NE recognition accuracy by tag for the full 300recipe corpus.
the very common tag Ac.
In previous research, Sasada et al. (2015b) trained PWNER
on a corpus of 193 Japanese recipes of comparable complexity annotated with the same set of r-NE tags, and reported 84.4 F1 ; our F1 for 200-r is comparable. We also
note that F1 is 2.3 points higher for 200-r than 100-r; thisindicates that annotating more data is likely to lead to further worthwhile improvements in NE recognition accuracy.
It is likely that annotating more data would be particularly
beneficial for the tags Ac2, Af and At, which are relatively
rare.

4.

Recipe Flow Graphs

4.1. Flow Graph Representation
Mori et al. (2014), in a study of Japanese recipe text,
showed how the procedural aspects of a recipe can be represented as a flow graph. A flow graph is a directed acyclic
graph (DAG) with a single root. Graph nodes are r-NEs andlabelled edges represent the relationships between thesenodes. Table 4 lists the flow graph edge labels.
Figure 2 shows the procedural text in a recipe “Almost no
fat banana bread” on the Allrecipes web site4 . Figure 3 contains the flow graph for this recipe. The flow graph makes
explicit what actions are applied to which ingredients andusing what tools. The graph is read downwards from thetop; nodes without any incoming edges represent ingredients, and a node with no outgoing edges denotes an end
product. A path of directed edges from an action A to an
action B implies that A must be carried out before B. For
example, the flow graph explicitly represents the fact thatbatter in sentence 4 is the end product of stirring the contents of the large bowl; this relationship is only implicit
43Precision
81.6%
85.4%
86.5%

From http://allrecipes.co.uk/recipe/722/
almost-no-fat-banana-bread.aspx, February 2020.

5189

Label
Agent
Targ
Dest
t-compF-compF-eq
F-part-of
F-set
T-eq
T-part-of
A-eq
V-tmother-mod

Meaning
Explanation
Subject
Direct object
Relationship with actions (Ac or Af)
Indirect object(container)
tool complement
Tool used in an action
Food complement Food used as a tool
Food equality
Identical food
Food part-ofRefer to a part of a food
Food set
Refer to a set of foods
Tool equality
Identical tool
Tool parf-ofRefer to a part of a tool
Action equality
Identical action (Ac, Af)
Head verb for timing, etc.
Other relationships
Table 4: Flow graph edge labels.

(#Step=1)-(#Sentence=0): Preheat oven to 180 C /Gas mark 4.
(1)-(1): Lightly grease a 20x10cm (8x4 in) loaf tin.
(2)-(2): In a large bowl, stir together flour, sugar, baking powder, bicarbonate of soda and cinnamon.
(2)-(3): Add egg whites, bananas and apple purée; stir
just until combined.
(2)-(4): Pour batter into prepared tin.
(3)-(5): Bake in preheated oven for 50 to 55 minutes,
until a skewer inserted into centre of loaf comes out
clean.
(3)-(6): Turn out onto wire rack and allow to cool before slicing.

4.2. Automatic Computation of Flow Graphs
Maeta et al. (2015) describe a procedure for computing aflow graph from recipe text. The procedure consists of threesteps, which are applied in turn to the entire r-NE tagged
recipe, and output a set of directed edges between r-NEs:
1. Calculate weights of labelled edges between all pairsof r-NEs, using an SVM-based machine learningmodel.
2. Given the resulting weighted digraph, select a root
node and compute a spanning arborescence of minimum weight. A reliable heuristic is to select as root
node the Ac that occurs last in the input.
3. Since the previous step can only produce a tree, extend
this to a DAG by adding further edges whose weights
are below a threshold and which satisfy certain conditions to ensure consistency.

Figure 2: The recipe “Almost no fat banana bread”, comprising 7 sentences (numbered 0 to 6), grouped into 3 steps.
Sentences used as examples are in bold.

in the recipe text since batter is not previously mentioned.
Please note that a flow graph is an internal, symbolic representation designed to support any kind of computation overrecipes, and we do not intend that it be displayed verbatim
to a cook in a recipe visualization application, for instance.
Starting from the r-NE annotations, the same Japanese
homemakers annotated all 300 sentences. The average annotation time was 37 minutes per recipe (including initial preparation); at first it was longer but as the annotators gained experience they got faster. The average annotation time for the final ten recipes was 26 minutes perrecipe. To measure inter-annotator agreement, we arranged
for both annotators to annotate the 100-r subset. We compute agreement similarly to the previous section, for similarreasons. Taking one of the annotators as the gold standard,
the other’s annotations had 84.4% precision, 80.4% recall,and 82.3 F1 . Although lower than the r-NE agreement, thisis still a good result bearing in mind the very large space ofpossible edge annotations.

In our work, we apply a similar procedure but ignore labels
in step 1 and omit step 3; we output a spanning arborescence with r-NEs as nodes connected by (directed) unlabelled edges5 .
We estimate the weight of a directed edge from node u to v
with label l as follows:
exp{Θ · f (u, v, l)}(1)(x,r)∈(V \{u})×L exp{Θ · f (u, x, r)}s(u, v, l) = ∑where V is a set of nodes, L is the set of edge labels (seeTable 4), Θ is a weight vector, and f (u, v, l) is a function
returning the feature vector for the edge. Intuitively, theright hand side of the equation computes the total weight ofthe edge’s features as a proportion of the grand total of theweights of all edges starting at node u.
5

An alternative approach would be to parse recipes syntactically in a sentence-by-sentence manner, and then induce flow
graph edges and their labels based on paths between nodes in theparses; however, as Maeta et al. (2015) argue, in that case further
processing would be required to deal with inter-sentence phenomena such as coreference, anaphora and ellipsis—whereas these do
not require any special treatment in our framework.

5190

Figure 3: Flow graph for the recipe “Almost no fat banana bread”.

5191(2 7) flour [F]t-comp

targ

targ

targ

targ

dest

Directed edge label(#sentence #word) Word [rNE Tag]

Directed edge

Any other type

Action by chef (Ac)

Tool node (T)(#sentence #word) Word [rNE Tag](#sentence #word) Word [rNE Tag]

Food node (F)(#sentence #word) Word [rNE Tag]targ

(4 1) batter [F]f-eq(3 9) stir [Ac]v-tm

(2 18) cinnamon [F]

t-eq

targ

v-eq
(5 13) inserted [Ac]t-comp(5 12) skewer [T]targ(6 10) slicing [Ac]targ(6 8) cool [Af]v-eq(6 6) allow [Ac]targ

(6 0) Turn_out [Ac]

dest

targ

(5 0) Bake [Ac]other-mod(5 5) 50_to_55_minutes [D](6 3) wire_rack [T](4 0) Pour [Ac]

dest

targ
(1 1) grease [Ac](4 3) prepared [Ac]targ(4 4) tin [T]other-mod(1 8) loaf_tin [T]other-mod(1 5) 8x4_in [St](3 6) apple_purée [F](1 3) 20x10cm [St]targ

(3 12) combined [Ac]

targ

targ

(3 4) bananas [F]targ(2 14) bicarbonate_of_soda [F](3 0) Add [Ac]targ(3 1) egg_whites [F](2 11) baking_powder [F](2 5) stir_together [Ac](2 9) sugar [F]

1. Preheat oven to 180 C / Gas mark 4. Lightly grease a
20x10cm (8x4 in) loaf tin.
2. In a large bowl, stir together ﬂour, sugar, baking powder,bicarbonate of soda and cinnamon.
Add egg whites, bananas and apple purée; stir just until
combined. Pour batter into prepared tin.
3. Bake in preheated oven for 50 to 55 minutes, until a skewer
inserted into centre of loaf comes out clean.
Turn out onto wire rack and allow to cool before slicing.

Method:(2 3) bowl [T]other-mod(2 2) large [St]other-mod

v-tm

targ

(5 18) comes_out [At]targ

(5 20) clean [St]dest

(5 15) centre [F]f-part-of(5 17) loaf [F](5 2) preheated [Ac]targ(5 3) oven [T]t-eq(0 1) oven [T]other-modv-eq(0 0) Preheat [Ac]targ(0 3) 180_C [St]other-mod(0 6) Gas_mark_4 [St]

r-NERGold
BERT-NER

Precision
73.7%
51.1%

Recall
68.6%
37.7%

F1
71.1
43.3

Label
Agent
Targ
Dest
T-compF-compF-eq
F-part-of
F-set
T-eq
T-part-of
A-eq
V-tmother-mod

Table 5: Edge estimation accuracy with gold r-NEs andwith automatically recognized r-NEs.
The function f (u, v, l) returns a feature vector for an edge,
containing information about the nodes u and v that theedge links. The vector comprises the following features:• words in u and v, and their concatenation;• concatenation of the r-NE tags of u and v;• whether u is in the same, a previous or a subsequent
sentence as v;

Total

In initial experiments, we confirmed that each of these fivefeatures improves edge detection accuracy.
The weight vector Θ is estimated from the training databy a log-linear model (Berger et al., 1996). Given training
data consisting of T manually annotated edge-label pairs(ut , vt , lt ), we maximize the following value:T∑1s(ut , vt , lt ) − ∥Θ∥2
2
t=1

(2)

To evaluate the accuracy of flow graph construction, weused 10-fold cross-validation, splitting the data into 270
recipes for training and 30 recipes for test. When we started
from the gold standard r-NE tagging, the overall F1 for edge
detection was 71.1. When we started instead from the automatic r-NE tagging, the edge detection F1 reduced to 43.3;
the main reason for the large decrease is that an edge will
always be wrong if the r-NE at even just one of its two end
points is mis-recognized. The results are shown in Table 5.
Table 6 shows the recall of edge detection for each label.
Recall is low for several labels: F-eq, F-part-of, F-set, T-eq,
T-part-of and A-eq. Edges with these labels often form confluences in the recipe flow graph and therefore cannot be
detected in step 2 of the procedure (the step that extracts aspanning arborescence). We intend to address this issue in
future work.

6
To detect prepositional context, we use the RASP grammatical analysis system (Briscoe et al., 2006)—having adapted its lexicon to account for the non-standard writing style of cooking
recipes.

Recall
55.1%
82.3%
79.5%
83.0%
88.2%
19.9%
12.3%
8.7%
13.0%
37.5%
35.7%
74.1%
73.2%
68.6%

flow graph resulting from end-to-end processing (i.e. with
r-NE recognition and edge detection7 both done automatically).
In (b) and (c), our method successfully detected the action equality A-eq between #(0-0) Preheat and #(5-2) Preheated, even though they are several steps apart. In contrast, tool equality T-eq was not detected between #(0-1)
oven and #(5-3) oven; this exemplifies the low recall ofT-eq edges as reported in Table 6. In (b), the out-edge of#(5-17) loaf is linked to the wrong node, although all theother edges are correct. In (c), our system failed to detect
the r-NEs for skewer, centre and inserted. The destinationof the out-edge from loaf should be centre, but this node
does not exist and the edge wrongly connects to the word
Bake. As a result, the part of the flow graph corresponding
to the subordinate clause until a skewer inserted into centreof loaf comes out clean fails to correctly represent the semantic structure. However, the edges in the rest of the flow
graph are correct.

5.

Conclusion

Taking as a starting point research on recipes written in
Japanese (Mori et al., 2014), we have presented an annotated corpus of English cooking recipe procedures, and described and evaluated computational methods for learningthese annotations. In this paper we have demonstrated that:• the approach to annotating and analysing recipes
transfers well to a typologically different language;• the annotation scheme supports high inter-annotator
agreement; and4.3. Flow Graph Computation Examples
Figure 4 shows flow graph nodes and edges corresponding
to the two sentences labeled (0)-(0) and (3)-(5) in Figure 2.
Figure 4(a) shows the manually-annotated gold standard
flow graph, (b) shows the flow graph produced automatically starting from gold standard r-NEs, and (c) shows the

#Est
371
5109
1576
540
252
227
91
2
41
71
85
474
2033
10,883

Table 6: Edge estimation recall.

• number of words between u and v (positive if u appears before v, otherwise negative); and• whether u and v are in the same sentence and there isa preposition between them6 .

#Gold
674
6210
1983
650
286
1139
737
23
316
190
237
640
2776
15,861• recipe named entities can be identified with good accuracy using a deep neural network tagger.
The paper described how we annotated each recipe in the
corpus with recipe named entities (r-NEs); inter-annotator
agreement was high. A deep neural network NER tool7

In this setup, we are pipelining two modules; eventually wewould like to combine these into a single model by training on
both module representations jointly.

5192

(5 17) loaf [F]f-part-of(5 12) skewer [T]t-comp(5 15) centre [F]dest(5 13) inserted [Ac]t-eq

(5 20) clean [St]targ(5 5) 50_to_55_minutes [D](0 1) oven [T]targ(5 18) comes_out [At]other-mod

v-tm

targ

(5 3) oven [T]other-mod(0 3) 180_C [St]other-mod(0 6) Gas_mark_4 [St]other-mod(0 0) Preheat [Ac]v-eq(5 2) preheated [Ac]targ(5 0) Bake [Ac](a) Ground truth: manually annotated flow graph.

(05-012) skewer [T](05-013) inserted [Ac](05-005) 50_to_55_minutes [D](05-015) centre [F](05-017) loaf [F](00-001) oven [T](05-020) clean [St](00-003) 180_C [St](05-003) oven [T](05-018) comes_out [At](00-006) Gas_mark_4 [St](00-000) Preheat [Ac](05-002) preheated [Ac](05-000) Bake [Ac](b) Flow graph computed from manually annotated r-NEs.

(00-001) oven [T](05-005) 50_to_55_minutes [D](05-017) loaf [F](00-003) 180_C [St](05-020) clean [St](05-003) oven [T](05-018) comes_out [At](05-002) preheated [Ac](00-006) Gas_mark_4 [St](00-000) Preheat [Ac](05-000) Bake [Ac](c) End-to-end flow graph estimation.
Figure 4: Flow graph nodes and edges corresponding to the sentences Preheat oven to 180 C / Gas mark 4 and Bake inpreheated oven for 50 to 55 minutes, until a skewer inserted into centre of loaf comes out clean.

trained on this data obtained an overall F1 of 87.5. This accuracy is comparable to r-NE tagging of Japanese recipes.
Starting from the r-NE annotation, inter-annotator agreement for flow-graph annotation was 82.3 F1 . This levelof agreement is good, considering the very large space ofpossible edge annotations. Computing flow graphs useda dependency-style parsing procedure, which achieved an
F1 for edge detection of 71.1. Previous work on analysing
Japanese recipes has shown that this level of accuracy is
sufficient to support tasks including recipe information retrieval and symbol grounding for cross-modal cooking applications.
In future work, we intend to refine the procedure for extracting a flow graph from the weighted digraph of r-NE
nodes. We will experiment with training classifiers foradding labels to edges, and for inserting links that form
confluences in the flow graph. Although these are extra

processing steps, they would add structure only monotonically to the minimum weight spanning arborescence, soshould not have a negative impact on efficiency or on correct structure that is already in the flow graph.

6.

Acknowledgements

This work was supported by JSPS KAKENHI Grant Number 18K11425 and ROIS NII Open Collaborative Research2020 (No. 19S0404).

7.

Bibliographical References

Banarescu, L., Bonial, C., Cai, S., Georgescu, M., Griffitt, K., Hermjakob, U., Knight, K., Koehn, P., Palmer,
M., and Schneider, N. (2013). Abstract meaning representation for sembanking. In the 7th Linguistic Annotation Workshop and Interoperability with Discourse,
pages 178–186.

5193

Ben Abacha, A. and Zweigenbaum, P. (2011). Medical entity recognition: A comparison of semantic and statistical methods. In Proceedings of BioNLP 2011 Workshop,
pages 56–64, Portland, Oregon, USA, June. Association
for Computational Linguistics.
Berger, A. L., Pietra, S. A. D., and Pietra, V. J. D. (1996).
A maximum entropy approach to natural language processing. Computational Linguistics, 22(1).
Bollini, M., Tellex, S., Thompson, T., Roy, N., and Rus, D.
(2013). Interpreting and executing recipes with a cooking robot. In Experimental Robotics, pages 481–495.
Borthwick, A. (1999). A Maximum Entropy Approach to
Named Entity Recognition. Ph.D. thesis, New York University.
Briscoe, T., Carroll, J., and Watson, R. (2006). The second
release of the RASP system. In Proceedings of the COLING/ACL 2006 Interactive Presentation Sessions, pages
77–80. Association for Computational Linguistics.
Constant, M., Eryigit, G., Monti, J., van der Plas, L., Rosner, C. R. M., and Todirascu, A. (2017). Multiword expression processing: A survey. Computational Linguistics, 43(4):837–892.
Devlin, J., Chang, M. W., Lee, K., and Toutanova, K.
(2019). BERT: Pre-training of deep bidirectional transformers for language understanding. In Proceedings ofthe 2019 Conference of the North American Chapter ofthe Association for Computational Linguistics: Human
Language Technologies, pages 4171–4186.
Flanigan, J., Thomson, S., Carbonell, J., Dyer, C., andSmith, N. A. (2014). A discriminative graph-basedparser for the abstract meaning representation. In Proceedings of the 52nd Annual Meeting of the Association
for Computational Linguistics, pages 1426–1436.
Hamada, R., Ide, I., Sakai, S., and Tanaka, H. (2000).
Structural analysis of cooking preparation steps inJapanese. In Proceedings of the fifth International Workshop on Information Retrieval with Asian Languages,
pages 157–164.
Hashimoto, A., Mori, N., Funatomi, T., Yamakata, Y.,
Kakusho, K., and Minoh, M. (2008). Smart kitchen: A
user centric cooking support system. In Proceedings ofthe 12th Information Processing and Management of Uncertainty in Knowledge-Based Systems, pages 848–854.
Jermsurawong, J. and Habash, N. (2014). Predicting thestructure of cooking recipes. In Proceedings of the 2015
Conference on Empirical Methods in Natural Language
Processing, pages 2370–2377.
Kiddon, C., Ponnuraj, G. T., Zettlemoyer, L., and Choi, Y.
(2015). Mise en place: Unsupervised interpretation of
instructional recipes. In Proceedings of the 2015 Conference on Empirical Methods in Natural Language Processing, pages 982–992.
Maeta, H., Sasada, T., and Mori, S. (2015). A frameworkfor procedural text understanding. In Proceedings of the
14th International Conference on Parsing Technologies,
pages 50–60.
Momouchi, Y. (1980). Control structures for actions inprocedural texts and PT-Chart. In Proceedings of theEighth International Conference on Computational Linguistics, pages 108–114.
Mori, S., Maeta, H., Yamakata, Y., and Sasada, T. (2014).
Flow graph corpus from recipe texts. In Proceedings of
the Nineth International Conference on Language Resources and Evaluation, pages 2370–2377.
Ratinov, L. and Roth, D. (2009). Design challenges andmisconceptions in named entity recognition. In Proceedings of the Thirteenth Conference on Computational Natural Language Learning (CoNLL-2009), pages 147–155,
Boulder, Colorado, June. Association for Computational
Linguistics.
Sang, E. F. T. K. and Meulder, F. D. (2003). Introduction
to the CoNLL-2003 shared task: Language-independent
named entity recognition. In Proceedings of the Seventh
Conference on Computational Natural Language Learning, pages 142–147.
Sang, E. F. T. K. and Veenstra, J. (1999). Representing text
chunks. In Proceedings of EACL’99, pages 173–179.
Sasada, T., Mori, S., Kawahara, T., and Yamakata, Y.
(2015a). Named entity recognizer trainable from partially annotated data. In Proceedings of the Eleventh International Conference Pacific Association for Computational Linguistics.
Sasada, T., Mori, S., Yamakata, Y., Maeta, H., and Kawahara, T. (2015b). Definition of recipe terms and corpus
annotation for their automatic recognition (in Japanese).
Journal of Natural Language Processing, 22(2):107–
131.
Yamakata, Y., Imahori, S., Sugiyama, Y., Mori, S., andTanaka, K. (2013). Feature extraction and summarization of recipes using flow graph. In Proceedings of
the 5th International Conference on Social Informatics,
LNCS 8238, pages 241–254.
Yamakata, Y., Carroll, J., and Mori, S. (2017). A comparison of cooking recipe named entities between Japaneseand English. In Proceedings of the 9th Workshop on
Multimedia for Cooking and Eating Activities, in Conjunction with IJCAI 2017, pages 7–12.

5194