Towards Flow Graph Prediction of Open-Domain Procedural Texts
Keisuke Shirai1 Hirotaka Kameko2 Shinsuke Mori2
1
Graduate School of Informatics, Kyoto University
2
Academic Center for Computing and Media Studies, Kyoto University
shirai.keisuke.64x@st.kyoto-u.ac.jp {kameko,forest}@i.kyoto-u.ac.jp

Abstract
Machine comprehension of procedural texts
is essential for reasoning about the steps andautomating the procedures. However, this requires identifying entities within a text and resolving the relationships between the entities.
Previous work focused on the cooking domain
and proposed a framework to convert a recipe
text into a flow graph (FG) representation. In
this work, we propose a framework based on
the recipe FG for flow graph prediction of opendomain procedural texts. To investigate flow
graph prediction performance in non-cooking
domains, we introduce the wikiHow-FG corpus from articles on wikiHow, a website of
how-to instruction articles. In experiments,
we consider using the existing recipe corpus
and performing domain adaptation from thecooking to the target domain. Experimental
results show that the domain adaptation models
achieve higher performance than those trained
only on the cooking or target domain data.

1

Introduction

A procedural text guides a human to complete daily
activities like cooking and furniture assembly. Machine comprehension of these texts is essential forreasoning about the steps (Zhang et al., 2020b) andautomating the procedures (Bollini et al., 2013).
However, it needs to identify entities within a text
and resolve relationships between the entities. Converting the text into an actionable representation(e.g., flow graph (Momouchi, 1980)) is an approachfor solving these problems.
There are several works on converting a procedural text into an action graph (Mori et al., 2014;
Kulkarni et al., 2018; Kuniyoshi et al., 2020). In thecooking domain, various approaches (Mori et al.,2014; Kiddon et al., 2015; Pan et al., 2020; Papadopoulos et al., 2022) have been taken because
there are a rich amount of available resources on theweb. Among them, recipe flow graph (FG) (MoriFigure 1: Example of flow graph prediction with ourframework. The prediction is in two stages: node prediction (colored in blue) and edge prediction (coloredin red). In this work, we use this framework to predictflow graphs of open-domain procedural texts.

et al., 2014) has the advantage of capturing finegrained relationships at entity-level. While the original work (Mori et al., 2014) introduced a framework and corpus in Japanese, the recent work (Yamakata et al., 2020) proposed those in English. The
current FG framework has two issues. Since FG
is designed to represent the flow of actions in aprocedural text, it should be applicable to otherprocedural domains such as crafting. One is thatthe framework has only been applied to the cooking
domain. The other is that preparing a large numberof annotations (e.g., thousands of articles) is unrealistic due to its complex annotation procedures.
In this work, we propose a framework based on
the English recipe flow graph (English-FG) (Yamakata et al., 2020) for FG prediction of open-

Tag
C (F)
T
D
Q
Ae (Ac)Ae2 (Ac2)
Ac (Af)At
Sc (Sf)
St

Label
Agent
Targ
Dest
T-compC-comp (F-comp)C-eq (F-eq)
C-part-of (F-part-of)
C-set (F-set)
T-eq
T-part-of
A-eq
V-tmother-mod

Meaning
Component (Food)
Tool
Duration
Quantity
Action by expert (chef)
Discontinuous Ae (Ac)
Action by component (food)Action by tool
State of component (food)State of tool

Meaning
Action agent
Action target
Action destination
Tool complement
Component (Food) complement
Component (Food) equality
Component (Food) part-of
Component (Food) set
Tool equality
Tool part-of
Action equality
Head of clause for timing
Other relationships

Table 1: Tags and their meanings. The inside of theparenthesis represents a tag and its meaning in EnglishFG.

Table 2: Labels and their meanings. The inside of theparenthesis represents a label and its meaning in EnglishFG.

domain procedural texts. We show the overview ofour framework in Figure 1. Our motivation is toexpand the scope of the recipe FG to non-cooking
domains by treating food ingredients as final product components. Our framework predicts an FG in
two stages: node prediction and edge prediction,
following Maeta et al. (2015). Our framework is
compatible with the English-FG, and we can jointly
learn representations from data in the cooking andnon-cooking domains. To investigate FG prediction performance in non-cooking domains, we introduce the wikiHow-FG corpus from wikiHow
articles. The corpus was constructed by selecting
four domains from the wikiHow categories andannotating 30 articles for each domain.
In experiments, we assume a low-resource scenario in which we can access only a few training
examples in the target domain. This is a realistic scenario considering the huge annotation costfor FG. To tackle this issue, we consider domain
adaptation from the existing cooking domain to thetarget domain. Experimental results show that domain adaptation models obtain higher performance
than those trained only on the cooking or target
domain data. We also considered two data augmentation techniques to boost performance. From theresults, we found that they improve performance inparticular domains.
Our contributions are three-fold:based on four wikiHow domains and has 30
annotated articles for each domain.

• We propose a framework based on the EnglishFG for flow graph prediction of open-domain
procedural texts.
• We introduce the wikiHow-FG corpus, a new
corpus from wikiHow articles. This corpus is• We assume a low-resource scenario in the target domain and consider domain adaptationfrom the cooking to the target domain. Experimental results show that domain adaptation
models outperform those trained only on thecooking or target domain data.

2

Recipe flow graphIn this section, we provide a brief description of
the recipe flow graph (FG) (Mori et al., 2014).
A recipe FG is a directed acyclic graph G(V, E),
where V represents entities as nodes, while E represents the relationships between the nodes as labelededges. Currently, FG annotations are available inJapanese (Mori et al., 2014) and English (Yamakata
et al., 2020), and these corpora provide annotations of hundreds of recipes. Note that Japaneseand English frameworks for FG are not compatible since the English FG uses additional tags tohandle English-specific expressions. As we focus
on texts in English, we consider the English-FG
framework (Yamakata et al., 2020) in the following
sections.
2.1

Flow graph representation

A recipe FG representation is divided into two
types of annotations; node and edge annotations.
Nodes represent entities with tags in the IOBformat (Ramshaw and Marcus, 1995). As listed inTable 1, 10 types of tags are used in the English-FG.
Labeled edges represent the relationships between

the nodes. As listed in Table 2, 13 types of labels
are used in the English-FG.
2.2

Flow graph prediction

For the automatic prediction of the FG, previous
work (Maeta et al., 2015) proposed to divide theproblem into two subtasks: node prediction andedge prediction. In both subtasks, models aretrained in a supervised fashion.
Node prediction identifies nodes in an article
with the tags. Maeta et al. (2015); Yamakata et al.
(2020) formulated this problem as a sequence labeling problem and used NER model (Lample et al.,2016). While predicting tags at sentence-level iscommon in NER (Lample et al., 2016), previous
work (Yamakata et al., 2020) used an entire recipe
text as input.1
Edge prediction constructs a directed acyclic
graph by predicting labeled edges between thenodes. This is formulated as a problem of finding the maximum spanning tree as:Ĝ = argmax
G∈GXs(u, v, l),(1)

Figure 2: Example of swapping steps. The first andsecond steps are swappable without violating causality.

With this framework, we consider predicting
flow graphs of open-domain procedural texts. The
prediction is performed in two stages: node prediction and edge prediction as in Section 2. Models
are trained in a supervised way as the previous approach (Maeta et al., 2015). However, preparing alarge number of examples in a new domain is unrealistic, considering the huge FG annotation cost.
Thus, we assume that only a few training examples
are available in the target domain. To tackle thisissue, we consider domain adaptation from the existing cooking domain data to the target domain
data. In the rest of this section, we formulate thetask in Section 3.1, then consider data augmentation techniques that fit our setting in Section 3.2.

(u,v,l)3.1
where s(u, v, l) represents the score of a labelededge from u to v with label l. We can solve thisproblem by using the Chu-Liu-Edmonds algorithm.
The scores are calculated using a graph-based dependency parser (McDonald et al., 2005).

3

Flow graph prediction of open-domain
procedural texts

Our framework is based on the English-FG andapplies to non-cooking domains by treating foodsin recipe texts as final product components. Examples of the components include tomato and beef forcooking, cardboard and glue for crafting, and gearand tire for vehicle maintenance. The frameworkuses tags and labels defined in Table 1 and Table 2,
respectively. These tags and labels are slightly modified from the definitions in the English-FG to avoid
confusion, and we did not add or delete any tags
and labels. Therefore, our framework is compatible
with the English-FG, and we can learn representations jointly from the cooking and non-cooking
domains.
1

In our preliminary experiments, we found that predicting the tags at document-level improves accuracy by 10%
compared to the prediction at sentence-level.

Task definition

We are given N examples in the cooking domain
C ) and M examples in the(V1C , E1C ), · · · , (VNC , EN
T
T
T , E T ), wheretarget domain (V1 , E1 ), · · · , (VM
MCC
V and E are a set of vertices and edges in thecooking domain, while V T and E T are those in
the target domain. Our goal is to maximize theperformance of node and edge prediction models in
the target domain. In this work, we use the EnglishFG corpus (300 articles) as the cooking domain
examples and the wikiHow-FG corpus (Section 4)
as the target domain examples. Note that M is aminimal number (namely, M = 5) in our setting,and the task has an aspect of low-resource domain
adaptation (Xu et al., 2021). Note also that training
with only the cooking or target-domain examples
becomes zero-shot or few-shot learning scenarios,
respectively.
3.2

Data augmentation

For improving performance in a low-resource setting, data augmentation is one of possible solutions (Fadaee et al., 2017; Ding et al., 2020). In
this work, we consider the following two data augmentation techniques: step swapping and word
replacement.

DomainFood and Entertaining
Hobbies and Crafts
Home and Garden
Cars & Other Vehicles

Task examples
Cooking acorn squash, Making lavender tea, Baking a cherry pie
Making a bar soap, Making a duct tape bow, Making a paper box
Cleaning a mattress pad, Installing a microwave, Making a scented candle
Fixing a slipped bike chain, Cleaning car window, Cleaning tail lights
Table 3: Examples of article titles for each domain.

DomainFood and Entertaining
Hobbies and Crafts
Home and Garden
Cars & Other Vehicles# Characters
10,167
9,407
7,700
6,432

# Words
2,761
2,556
2,010
1,622

# Steps
224
247
205
173

# Tags
1,132
1,062
894
625

# Labels
1,124
1,076
886
622

Table 4: Statistics of the wikiHow-FG corpus.

Step swapping augments an example by replacing two arbitrary steps in an article as illustrated in
Figure 2. However, randomly choosing and swapping two steps might break their causal relationship.
For example, we cannot swap two steps “Cut the
potatoes.” and “Add the potatoes to the pan.” in Figure 2. In this work, we augment examples keeping
this constraint by using flow graph annotations.
Word replacement augments an example by replacing a word with an arbitrary one. For eachword in a step, we replace a word with one of its
synonyms from WordNet (Dai and Adel, 2020)
with a probability p.

4

wikiHow-FG corpus

The wikiHow-FG corpus is a new flow graph corpus from articles on wikiHow2 , a website withmore than 110K how-to articles. wikiHow articles
have been used as a language resource for procedural texts (Zhou et al., 2019; Zellers et al., 2019;
Zhang et al., 2020b,a; Zhou et al., 2022). In the following, we describe the data collection, annotation
procedure, and statistics of the annotation results.
4.1

Data collection

For the target domains, we selected four categories
from wikiHow: (i) Food and Entertaining, (ii) Hobbies and Crafts, (iii) Home and Garden, and (iv)
Cars & Other Vehicles. We decided on those domains because many articles target and interact
with substantial objects. We show examples ofarticle titles in Table 3. Food and Entertaining
is a domain close to the cooking domain as with
the English-FG. Hobbies and Crafts is far from2

https://www.wikihow.com

the cooking domain in the sense that they assemble non-edible materials (e.g., making a bar soap).
The remaining two domains are further from these
domains because they contain non-assembly tasks(e.g., cleaning a table and fixing a broken chain).
We collected 30 articles for each domain fromthe wikiHow corpus (Zhang et al., 2020b). To
exclude low-quality articles, we collected articles
with 25 or more words and more than 50% user ratings. We then manually excluded articles with too
abstract goals or not targeting substantial objects.
We used article headlines as steps for annotationand experiments. Note that each article has a title and visual information (e.g., images or videos)
describing the procedures. Exploiting them is interesting, but we leave that direction for future work.
4.2

Annotation procedure

Due to the dense, complex nature of the flow graph,constructing a high-quality corpus is a challenging
problem. In order to guarantee the annotation quality, we first trained an annotator with 10 recipes
sampled from the English-FG corpus (Yamakata
et al., 2020). The training continued until the interannotator agreements with the ground-truth annotations reached over 80%. We then asked the annotator to annotate wikiHow articles. For both thenode and edge annotations, we used the flow graphannotation tool (Shirai et al., 2022).3 The wholeannotation took 40 hours.
4.3

Statistics

We show statistics of the wikiHow corpus in Table 4. The articles comprise 280.9 characters,
3
Before the annotation, we tokenized steps into words withthe stanza toolkit (Qi et al., 2020).

Annotation type
Node annotation
Edge annotation

Agreement
90.17%
57.43%

Table 5: Inter-annotator agreements.

74.6 words, and 7.08 steps on average. The average number of tags and labels per article is 37.73(±16.48) and 37.47 (±17.21) on Food and Entertaining, 35.40 (±9.08) and 35.87 (±9.44) on Hobbies and Crafts, 29.80 (±7.83) and 29.53 (±8.29)
on Home and Garden, and 20.83 (±6.40) and20.73 (±7.14) on Cars & Other Vehicles. Since
articles on Home and Garden and Cars & Other
Vehicles have fewer words than the other two domains, the number of tags and labels also becomes
smaller.
4.4Inter-annotator agreements

To assess the consistency of the annotations, we
asked another annotator to re-annotate 10% of articles for each domain. We then measured F1 scores
between the two sets of annotations using the original annotations as the ground-truth ones. Table 5shows the results. The agreement for the node annotation (90.14%) was high, considering entities
corresponding to the tags greatly change depending on the domain. For the edge annotation, theagreement (57.43%) drops from the one for nodeannotation. However, this agreement is also highconsidering errors from the node annotation influence this step and that a large number of candidate
edges for the annotation.

5

Node prediction

5.1

Experimental settings

Model. We
adopteda
BiLSTM-CRF
model (Lample et al., 2016) for node prediction by replacing a BiLSTM encoder with apre-trained language model (LM) (Devlin et al.,2019; Liu et al., 2019; He et al., 2021). Whileprevious work (Yamakata et al., 2020) used a
pre-trained BERT (Devlin et al., 2019) as theencoder, we used a pre-trained DeBERTa (He et al.,2021).4 This model has 140M parameters in total.
Training. We trained a domain adaptation modelby first training on the cooking domain data4

In our preliminary experiments, we confirmed that DeBERTa improves the accuracy of BERT by 0.47% on the
English-FG corpus.

(English-FG) and then training again on the target domain data (wikiHow-FG). Note that we use
only target domain examples of the wikiHow-FG
corpus (e.g., when targeting Hobbies and Crafts,
we do not use examples in the other three domains.).
We also train models only on the cooking or target
domain data and report the results to compare withthe domain adaptation results.
For an optimization method, we used
AdamW (Loshchilov and Hutter, 2019) withan initial learning rate of 5.0 × 10−5 and a weightdecay of 1.0 × 10−5 . We tuned a learning ratewith a cosine-annealing (Loshchilov and Hutter,2019) (Sd steps) with a linear warm-up (Sw steps)
at every iteration. We created a mini-batch from B
articles. We set (B, Sw , Sd ) = (5, 500, 4500) and(B, Sw , Sd ) = (3, 100, 900) for training on the
English-FG corpus and the wikiHow-FG corpus,
respectively. We tuned these hyper-parameters on
the development set. We used the data augmentation techniques only for the target domain data.
For the step swapping, we created 5 augmented
examples at maximum from one example. For theword replacement, we set 0.5 to p and created 10
examples from one example.
Evaluation. We split the English-FG corpus into
80% for training, 10% for validation, and the restof 10% for testing. For the wikiHow-FG corpus,
we split 30 articles of each domain into 6 folds. For
more reliable results, we performed 6-fold crossvalidation by using 1 fold for training, 1 fold forvalidation, and the remaining 4 folds for testing.
We used precision, recall, and F1 for evaluation
metrics, following Yamakata et al. (2020). On theevaluation, we report the average scores of the models on the test set.
Model configurations. We refer to the domain
adaptation models as domain-adaptation models.
We also refer to the models trained only on the
English-FG or the wikiHow-FG corpus as cookingonly and target-only models, respectively.
5.2

Results

The results are shown in Table 6. We see that thetarget-only models achieve an F1 score of 66.9%
or more in all the target domains. This implies
that the node prediction model can predict nodes
to an extent with a few annotated articles. We also
see that the cooking-only models achieve competitive performance with the target-only ones andoutperform them in the two domains of Food andDomainFood and Entertaining

Hobbies and Crafts

Home and Garden

Cars & Other Vehicles

Augmentation
Step-swap Word-replace

Model
Target-only
Cooking-only
Domain-adaptation
Domain-adaptation
Domain-adaptation
Target-only
Cooking-only
Domain-adaptation
Domain-adaptation
Domain-adaptation
Target-only
Cooking-only
Domain-adaptation
Domain-adaptation
Domain-adaptation
Target-only
Cooking-only
Domain-adaptation
Domain-adaptation
Domain-adaptation

✓
✓

✓
✓

✓
✓

✓
✓

Prec.

Recall

F1

0.770
0.884
0.890
0.894
0.885
0.698
0.703
0.794
0.784
0.781
0.663
0.734
0.780
0.787
0.765
0.650
0.646
0.748
0.734
0.729

0.784
0.877
0.892
0.895
0.891
0.707
0.684
0.805
0.795
0.790
0.676
0.742
0.786
0.791
0.773
0.690
0.695
0.784
0.784
0.772

0.777
0.880
0.891
0.895
0.888
0.702
0.693
0.799
0.789
0.785
0.669
0.738
0.783
0.786
0.769
0.669
0.670
0.765
0.761
0.750

Table 6: Results of the node prediction experiments. The check mark symbol (✓) indicates the used training data(in cooking and target domains) and augmentation techniques (step-swap and word-replace).
Ae
DomainFood and Entertaining
Hobbies and Crafts
Home and Garden
Cars & Other Vehicles

Duplicates
92.06%
69.03%
65.19%
46.04%

C
F1

Src
0.941
0.943
0.954
0.905

Adpt
0.952
0.951
0.961
0.919

Duplicates
72.11%
10.33%
18.40%
6.88%

T
F1

Src
0.932
0.717
0.716
0.666

Adpt
0.933
0.833
0.795
0.805

Duplicates
77.94%
51.79%
43.55%
27.47%

F1
Src
0.896
0.398
0.567
0.459

Adpt
0.882
0.588
0.678
0.557

Table 7: F1 scores of Ae, C, and T tags with the percentage of entities that appeared in the English-FG corpus andalso in the wikiHow-FG corpus. Src and Adpt denote cooking-only and domain-adaptation models, respectively.

Entertaining and Home and Garden. Particularly
in Food and Entertaining, the cooking-only modelsurpasses the target-only one by 10.3% in F1. We
consider that this domain is close to the cooking
domain of the English-FG; thus, the cooking-onlymodel is more advantageous as it can access more
examples.
Next, the domain-adaptation models achieve
the best performance in all the domains comparedwith the cooking-only and target-only models(76.5% or more F1). The most significant improvements are obtained in the two domains of Hobbies
and Crafts and Cars & Other Vehicles (9.5% and10.5% improvements in F1). These results indicate that domain adaptation from the cooking to
the target domain is effective for training the nodeprediction model.
Third, we see that using the augmented data by
the step swapping slightly improves performancefrom the domain-adaptation models in Food andEntertaining and Home and Garden. On the other
hand, the word replacement does not contribute to
any improvement. One possible reason is that areplaced word does not necessarily match the corresponding tag, and this disrupts the improvement.
5.3

Tag-level prediction performance

Entities for each tag can greatly change dependingon the domain. In that case, the degree of improvement from the cooking-only to the domainadaptation model is expected to increase as the
duplicate entities between the domains decrease.

DomainFood and Entertaining

Hobbies and Crafts

Home and Garden

Cars & Other Vehicles

Model

Augmentation
Step-Swap Word-Replace

Target-only
Cooking-only
Domain-adaptation
Domain-adaptation
Domain-adaptation
Target-only
Cooking-only
Domain-adaptation
Domain-adaptation
Domain-adaptation
Target-only
Cooking-only
Domain-adaptation
Domain-adaptation
Domain-adaptation
Target-only
Cooking-only
Domain-adaptation
Domain-adaptation
Domain-adaptation

✓
✓

✓
✓

✓
✓

✓
✓

Prec.

Recall

F1

0.335
0.725
0.750
0.747
0.761
0.285
0.613
0.649
0.646
0.653
0.229
0.644
0.659
0.656
0.674
0.154
0.587
0.607
0.607
0.617

0.338
0.731
0.756
0.752
0.752
0.281
0.605
0.640
0.638
0.644
0.232
0.649
0.665
0.662
0.680
0.155
0.590
0.610
0.610
0.620

0.337
0.728
0.753
0.750
0.749
0.283
0.609
0.644
0.642
0.648
0.231
0.646
0.662
0.659
0.677
0.154
0.587
0.609
0.608
0.618

Table 8: Results of the edge prediction experiments. The check mark symbol (✓) indicates the used training data (incooking and target domains) and augmentation techniques (step-swap and word-replace).

To investigate this assumption, we measured taglevel prediction performance in F1 with the duplicate ratio of entities of the wikiHow-FG in the
English-FG. We targeted the three tags of Ae, C,
and T because these tags frequently appear in all
the domains.

diction.5 This model uses separate modules foredge prediction and label prediction. The resulting
loss l is defined as a weighted sum of losses fromthe two modules:

The results are shown in Table 7. For Ae, thedegree of improvement from the cooking-only tothe domain-adaptation model is small regardlessof the duplicate ratios, which is contrary to ourassumption. These results imply that recognizing
entities for Ae is easy irrespective of the domain.
For C and T, the domain-adaptation models significantly outperform the cooking-only ones in thethree domains other than Food and Entertaining.
These results imply that the domain adaptation iseffective for recognizing C and T tags when thedomain is further from the cooking domain.

where λ controls the strength of the two losses. We
empirically set 0.5 to λ. We used a pre-trained
DeBERTa (He et al., 2021) to obtain contextualized word representations. This model has 149M
parameters in total.

6

Edge prediction

6.1

Experimental settings

Evaluation. We used the same splits of the
English-FG and wikiHow-FG corpora as in Section 5 and performed 6-fold cross-validation forl = λl(edge) + (1 − λ)l(label) ,Training. Similarly to Section 5.1, we trained adomain adaptation model for edge prediction firston the English-FG corpus and then on the wikiHowFG corpus. For an optimization method, we used
AdamW (Loshchilov and Hutter, 2019) with a combination of a cosine-annealing and linear warm-up
learning rate scheduling method. We used the same
hyperparameters in Section 5.1.

5Model. We adopted a biaffine dependency
parser (Dozat and Manning, 2018) for edge pre-(2)Previous work (Maeta et al., 2015) used a linear model,
but we confirmed that our model achieves higher performanceon the English-FG corpus.

DomainFood and Entertaining
Hobbies and Crafts
Home and Garden
Cars & Other Vehicles

F1
0.679 (-9.8%)
0.501 (-22.2%)
0.494 (-25.4%)
0.449 (-26.3%)

Table 9: Results of the pipeline experiments. The insideof the parenthesis represents the performance drop fromthe domain-adaptation model with ground-truth tags.

more reliable results. We report the average scoresof the models on the test set. For evaluation metrics,
we used precision, recall, and F1 between predicted
and ground-truth labeled edges of (u, v, l).
Model configurations. We used the same model
notations of cooking-only, target-only, anddomain-adaptation models as in Section 5.
6.2

Results

The results are shown in Table 8. We used groundtruth tags to identify nodes. Contrary to the nodeprediction results, the target-only models achieve
poor performance in all the domains (33.8% or lessin all the metrics). On the other hand, the scores of
the cooking-only models are more than twice thoseof the target-only models. These results show thatthe edge prediction model requires more training
examples than the node prediction one. These also
show that with the English-FG corpus, predictingedges with 58.7% or more F1 is possible in noncooking domains.
Next, the domain-adaptation models outperform the target-only and cooking-only ones in all
the domains. This is consistent with the results inthe node prediction task. These results mean thatthe domain adaptation from the cooking to the target domain is also effective for the edge prediction
model. For the results with the data augmentation
techniques, the step swapping does not contributeto any improvement, contrary to Section 5.2. The
word replacement improves the performance of thedomain-adaptation models in the three domains
other than Food and Entertaining.
6.3

Pipeline experiments

So far, the model has used ground-truth tags to
identify nodes. However, in a realistic scenario,
the model must predict labeled edges with the predicted nodes. In this scenario, errors in the nodeprediction step would affect performance in theedge prediction step. To investigate edge prediction

performance in this setting, we conducted experiments of edge prediction with the predicted nodes.
We predicted nodes using the models in Section 5.2.
In order to evaluate the model with tag information,
we measured F1 of tuples of (u, v, l, nu , nv ) between ground-truth and predicted ones, where nu
and nv are the tags of the starting and ending nodes,
respectively.
Table 9 shows the results with performancedrops from those in Table 8. We see that 9.8%
drops in Food and Entertaining, and more significant drops occur in the other three domains (about
24.6%). In these three domains, F1 scores of thenode prediction are about 10% smaller than that ofFood and Entertaining, and this gap would causesuch large performance drops. We consider that
improving node prediction performance would alleviate these drops.

7

Related work

Mori et al. (2014) designed a flow graph (FG) representation in the cooking domain and introduceda corpus of recipe texts. Subsequent works introduced a corpus in English (Yamakata et al., 2020)and corpora with visual annotations (Nishimura
et al., 2020; Shirai et al., 2022). Maeta et al. (2015)
proposed a method for an automatic FG prediction.
Our work stems from this line of research and is thefirst attempt to apply the framework to non-cooking
domains. Ours is also the first work to use a neuralnetwork-based method for the edge prediction.
Other than the recipe FG, there are several works
that focus on obtaining an actionable representationfrom a procedural text. In cooking, Kiddon et al.
(2015) proposed an unsupervised EM algorithm,
while Pan et al. (2020); Papadopoulos et al. (2022)
proposed supervised approaches. In biochemistry,
Kulkarni et al. (2018); Tamari et al. (2021) introduced datasets for mapping wet lab protocols to an
action graph. In material science, Kuniyoshi et al.
(2020) represented the synthesis process with flow
graphs. The works (Pan et al., 2020; Papadopoulos
et al., 2022; Tamari et al., 2021; Kuniyoshi et al.,2020) are especially close to ours in the sense thatthey aim to obtain a document-level action graphin a supervised way.

8

Conclusion

We proposed a framework based on the EnglishFG and investigated flow graph prediction performance in non-cooking domains. We presented thewikiHow-FG corpus from wikiHow articles. We
considered domain adaptation from the cooking
to the target domain. Experimental results showthat domain adaptation models outperform thosetrained only on the cooking or target domain data.
In future work, we consider applying this framework to other domains, such as material science andbiochemistry. One can also try improving performance using more sophisticated data augmentation
techniques. We hope that our work will provide
new insights into procedural text understanding.

Acknowledgments
We would like to thank anonymous reviewers
for their insightful comments. This work was
supported by JSPS KAKENHI Grant Number
20H04210 and 21H04910.

References
Mario Bollini, Stefanie Tellex, Tyler Thompson,
Nicholas Roy, and Daniela Rus. 2013. Interpreting and executing recipes with a cooking robot. In
Experimental Robotics, pages 481–495. Springer.
Xiang Dai and Heike Adel. 2020. An analysis of simple
data augmentation for named entity recognition. In
Proceedings of the 28th International Conference on
Computational Linguistics, pages 3861–3867. International Committee on Computational Linguistics.
Jacob Devlin, Ming-Wei Chang, Kenton Lee, andKristina Toutanova. 2019. BERT: Pre-training ofdeep bidirectional transformers for language understanding. In Proceedings of the 2019 Conference of
the North American Chapter of the Association for
Computational Linguistics: Human Language Technologies (Volume 1: Long and Short Papers), pages
4171–4186. Association for Computational Linguistics.
Bosheng Ding, Linlin Liu, Lidong Bing, Canasai Kruengkrai, Thien Hai Nguyen, Shafiq Joty, Luo Si, andChunyan Miao. 2020. DAGA: Data augmentation
with a generation approach for low-resource tagging
tasks. In Proceedings of the 2020 Conference on
Empirical Methods in Natural Language Processing(EMNLP), pages 6045–6057. Association for Computational Linguistics.
Timothy Dozat and Christopher D. Manning. 2018.
Simpler but more accurate semantic dependency parsing. In Proceedings of the 56th Annual Meeting of
the Association for Computational Linguistics (Volume 2: Short Papers), pages 484–490. Association
for Computational Linguistics.
Marzieh Fadaee, Arianna Bisazza, and Christof Monz.
2017. Data augmentation for low-resource neuralmachine translation. In Proceedings of the 55th Annual Meeting of the Association for Computational
Linguistics (Volume 2: Short Papers), pages 567–573.
Association for Computational Linguistics.
Pengcheng He, Xiaodong Liu, Jianfeng Gao, andWeizhu Chen. 2021. Deberta: decoding-enhanced
bert with disentangled attention. In 9th International
Conference on Learning Representations.
Chloé Kiddon, Ganesa Thandavam Ponnuraj, Luke
Zettlemoyer, and Yejin Choi. 2015. Mise en place:
Unsupervised interpretation of instructional recipes.
In Proceedings of the 2015 Conference on Empirical Methods in Natural Language Processing, pages
982–992. Association for Computational Linguistics.
Chaitanya Kulkarni, Wei Xu, Alan Ritter, and Raghu
Machiraju. 2018. An annotated corpus for machinereading of instructions in wet lab protocols. In Proceedings of the 2018 Conference of the North American Chapter of the Association for Computational
Linguistics: Human Language Technologies, Volume
2 (Short Papers), pages 97–106. Association for Computational Linguistics.
Fusataka Kuniyoshi, Kohei Makino, Jun Ozawa, and
Makoto Miwa. 2020. Annotating and extracting synthesis process of all-solid-state batteries from scientific literature. In Proceedings of the Twelfth Language Resources and Evaluation Conference, pages
1941–1950. European Language Resources Association.
Guillaume Lample, Miguel Ballesteros, Sandeep Subramanian, Kazuya Kawakami, and Chris Dyer. 2016.
Neural architectures for named entity recognition.
In Proceedings of the 2016 Conference of the North
American Chapter of the Association for Computational Linguistics: Human Language Technologies,
pages 260–270. Association for Computational Linguistics.
Yinhan Liu, Myle Ott, Naman Goyal, Jingfei Du, Mandar Joshi, Danqi Chen, Omer Levy, Mike Lewis,
Luke Zettlemoyer, and Veselin Stoyanov. 2019.
Roberta: A robustly optimized bert pretraining approach. arXiv preprint arXiv:1907.11692.
Ilya Loshchilov and Frank Hutter. 2019. Decoupledweight decay regularization. In Proceedings of the
7th International Conference on Learning Representations.
Hirokuni Maeta, Tetsuro Sasada, and Shinsuke Mori.
2015. A framework for procedural text understanding. In Proceedings of the 14th International Conference on Parsing Technologies, pages 50–60. Association for Computational Linguistics.
Ryan McDonald, Fernando Pereira, Kiril Ribarov, andJan Hajič. 2005. Non-projective dependency parsing
using spanning tree algorithms. In Proceedings of
Human Language Technology Conference and Conference on Empirical Methods in Natural Language

Processing, pages 523–530. Association for Computational Linguistics.
Yoshio Momouchi. 1980. Control structures for actionsin procedural texts and PT-chart. In COLING 1980
Volume 1: The 8th International Conference on Computational Linguistics.

Haoran Xu, Seth Ebner, Mahsa Yarmohammadi,
Aaron Steven White, Benjamin Van Durme, andKenton Murray. 2021. Gradual fine-tuning for lowresource domain adaptation. In Proceedings of theSecond Workshop on Domain Adaptation for NLP,
pages 214–221. Association for Computational Linguistics.

Shinsuke Mori, Hirokuni Maeta, Yoko Yamakata, andTetsuro Sasada. 2014. Flow graph corpus from recipe
texts. In Proceedings of the Ninth International
Conference on Language Resources and Evaluation,
pages 2370–2377.

Yoko Yamakata, Shinsuke Mori, and John A Carroll.
2020. English recipe flow graph corpus. In Proceedings of the 12th Language Resources and Evaluation
Conference, pages 5187–5194. European Language
Resources Association.

Taichi Nishimura, Suzushi Tomori, Hayato Hashimoto,Atsushi Hashimoto, Yoko Yamakata, Jun Harashima,
Yoshitaka Ushiku, and Shinsuke Mori. 2020. Visual grounding annotation of recipe flow graph. In
Proceedings of the 12th Language Resources and
Evaluation Conference, pages 4275–4284. European
Language Resources Association.

Rowan Zellers, Ari Holtzman, Yonatan Bisk, Ali
Farhadi, and Yejin Choi. 2019. HellaSwag: Cana machine really finish your sentence? In Proceedings of the 57th Annual Meeting of the Association
for Computational Linguistics, pages 4791–4800. Association for Computational Linguistics.

Liang-Ming Pan, Jingjing Chen, Jianlong Wu, Shaoteng
Liu, Chong-Wah Ngo, Min-Yen Kan, Yugang Jiang,
and Tat-Seng Chua. 2020. Multi-modal cooking
workflow construction for food recipes. In Proceedings of the 28th ACM International Conference on
Multimedia, page 1132–1141. Association for Computing Machinery.
Dim P. Papadopoulos, Enrique Mora, Nadiia Chepurko,
Kuan Wei Huang, Ferda Ofli, and Antonio Torralba.
2022. Learning program representations for foodimages and cooking recipes. In Proceedings of theIEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pages 16559–16569.
Peng Qi, Yuhao Zhang, Yuhui Zhang, Jason Bolton, andChristopher D. Manning. 2020. Stanza: A python
natural language processing toolkit for many humanlanguages. In Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics:
System Demonstrations, pages 101–108. Association
for Computational Linguistics.
Lance Ramshaw and Mitch Marcus. 1995. Text chunking using transformation-based learning. In Third
Workshop on Very Large Corpora.
Keisuke Shirai, Atsushi Hashimoto, Taichi Nishimura,
Hirotaka Kameko, Shuhei Kurita, Yoshitaka Ushiku,
and Shinsuke Mori. 2022. Visual recipe flow: A
dataset for learning visual state changes of objects
with recipe flows. In Proceedings of the 29th International Conference on Computational Linguistics,
pages 3570–3577. International Committee on Computational Linguistics.
Ronen Tamari, Fan Bai, Alan Ritter, and Gabriel
Stanovsky. 2021. Process-level representation ofscientific protocols with interactive annotation. In
Proceedings of the 16th Conference of the European
Chapter of the Association for Computational Linguistics: Main Volume, pages 2190–2202. Association for Computational Linguistics.

Li Zhang, Qing Lyu, and Chris Callison-Burch. 2020a.
Intent detection with WikiHow. In Proceedings ofthe 1st Conference of the Asia-Pacific Chapter ofthe Association for Computational Linguistics andthe 10th International Joint Conference on Natural
Language Processing, pages 328–333. Association
for Computational Linguistics.
Li Zhang, Qing Lyu, and Chris Callison-Burch. 2020b.
Reasoning about goals, steps, and temporal orderingwith WikiHow. In Proceedings of the 2020 Conference on Empirical Methods in Natural Language
Processing, pages 4630–4639. Association for Computational Linguistics.
Shuyan Zhou, Li Zhang, Yue Yang, Qing Lyu,
Pengcheng Yin, Chris Callison-Burch, and Graham
Neubig. 2022. Show me more details: Discovering
hierarchies of procedures from semi-structured web
data. In Proceedings of the 60th Annual Meeting of
the Association for Computational Linguistics (Volume 1: Long Papers), pages 2998–3012. Association
for Computational Linguistics.
Yilun Zhou, Julie Shah, and Steven Schockaert. 2019.
Learning household task knowledge from WikiHow
descriptions. In Proceedings of the 5th Workshop on
Semantic Deep Learning (SemDeep-5), pages 50–56.
Association for Computational Linguistics.