Towards Flow Graph Prediction of Open-Domain Procedural Texts
Year: 2023

Abstract:
Machine comprehension of procedural texts is essential for reasoning about the steps and automating the procedures. However, this requires identifying entities within a text and resolving the relationships between the entities. Previous work focused on the cooking domain and proposed a framework to convert a recipe text into a flow graph (FG) representation. In this work, we propose a framework based on the recipe FG for flow graph prediction of opendomain procedural texts. To investigate flow graph prediction performance in non-cooking domains, we introduce the wikiHow-FG corpus from articles on wikiHow, a website of how-to instruction articles. In experiments, we consider using the existing recipe corpus and performing domain adaptation from the cooking to the target domain. Experimental results show that the domain adaptation models achieve higher performance than those trained only on the cooking or target domain data.

heading: 1 Introduction
text: A procedural text guides a human to complete daily activities like cooking and furniture assembly. Machine comprehension of these texts is essential for reasoning about the steps (Zhang et al., 2020b) and automating the procedures (Bollini et al., 2013). However, it needs to identify entities within a text and resolve relationships between the entities. Converting the text into an actionable representation (e.g., flow graph (Momouchi, 1980)) is an approach for solving these problems.
There are several works on converting a procedural text into an action graph (Mori et al., 2014; Kulkarni et al., 2018; Kuniyoshi et al., 2020). In the cooking domain, various approaches (Mori et al., 2014; Kiddon et al., 2015; Pan et al., 2020; Papadopoulos et al., 2022) have been taken because there are a rich amount of available resources on the web. Among them, recipe flow graph (FG) (Mori
et al., 2014) has the advantage of capturing finegrained relationships at entity-level. While the original work (Mori et al., 2014) introduced a framework and corpus in Japanese, the recent work (Yamakata et al., 2020) proposed those in English. The current FG framework has two issues. Since FG is designed to represent the flow of actions in a procedural text, it should be applicable to other procedural domains such as crafting. One is that the framework has only been applied to the cooking domain. The other is that preparing a large number of annotations (e.g., thousands of articles) is unrealistic due to its complex annotation procedures.
In this work, we propose a framework based on the English recipe flow graph (English-FG) (Yamakata et al., 2020) for FG prediction of open-
87
domain procedural texts. We show the overview of our framework in Figure 1. Our motivation is to expand the scope of the recipe FG to non-cooking domains by treating food ingredients as final product components. Our framework predicts an FG in two stages: node prediction and edge prediction, following Maeta et al. (2015). Our framework is compatible with the English-FG, and we can jointly learn representations from data in the cooking and non-cooking domains. To investigate FG prediction performance in non-cooking domains, we introduce the wikiHow-FG corpus from wikiHow articles. The corpus was constructed by selecting four domains from the wikiHow categories and annotating 30 articles for each domain.
In experiments, we assume a low-resource scenario in which we can access only a few training examples in the target domain. This is a realistic scenario considering the huge annotation cost for FG. To tackle this issue, we consider domain adaptation from the existing cooking domain to the target domain. Experimental results show that domain adaptation models obtain higher performance than those trained only on the cooking or target domain data. We also considered two data augmentation techniques to boost performance. From the results, we found that they improve performance in particular domains.
Our contributions are three-fold:
• We propose a framework based on the EnglishFG for flow graph prediction of open-domain procedural texts.
• We introduce the wikiHow-FG corpus, a new corpus from wikiHow articles. This corpus is
based on four wikiHow domains and has 30 annotated articles for each domain.
• We assume a low-resource scenario in the target domain and consider domain adaptation from the cooking to the target domain. Experimental results show that domain adaptation models outperform those trained only on the cooking or target domain data.

heading: 2 Recipe flow graph
text: In this section, we provide a brief description of the recipe flow graph (FG) (Mori et al., 2014). A recipe FG is a directed acyclic graph G(V,E), where V represents entities as nodes, while E represents the relationships between the nodes as labeled edges. Currently, FG annotations are available in Japanese (Mori et al., 2014) and English (Yamakata et al., 2020), and these corpora provide annotations of hundreds of recipes. Note that Japanese and English frameworks for FG are not compatible since the English FG uses additional tags to handle English-specific expressions. As we focus on texts in English, we consider the English-FG framework (Yamakata et al., 2020) in the following sections.

heading: 2.1 Flow graph representation
text: A recipe FG representation is divided into two types of annotations; node and edge annotations. Nodes represent entities with tags in the IOBformat (Ramshaw and Marcus, 1995). As listed in Table 1, 10 types of tags are used in the English-FG. Labeled edges represent the relationships between
the nodes. As listed in Table 2, 13 types of labels are used in the English-FG.

heading: 2.2 Flow graph prediction
text: For the automatic prediction of the FG, previous work (Maeta et al., 2015) proposed to divide the problem into two subtasks: node prediction and edge prediction. In both subtasks, models are trained in a supervised fashion.
Node prediction identifies nodes in an article with the tags. Maeta et al. (2015); Yamakata et al. (2020) formulated this problem as a sequence labeling problem and used NER model (Lample et al., 2016). While predicting tags at sentence-level is common in NER (Lample et al., 2016), previous work (Yamakata et al., 2020) used an entire recipe text as input.1
Edge prediction constructs a directed acyclic graph by predicting labeled edges between the nodes. This is formulated as a problem of finding the maximum spanning tree as:
Ĝ = argmax G∈G
∑
(u,v,l)
s(u, v, l), (1)
where s(u, v, l) represents the score of a labeled edge from u to v with label l. We can solve this problem by using the Chu-Liu-Edmonds algorithm. The scores are calculated using a graph-based dependency parser (McDonald et al., 2005).

heading: 3 Flow graph prediction of open-domain procedural texts
text: Our framework is based on the English-FG and applies to non-cooking domains by treating foods in recipe texts as final product components. Examples of the components include tomato and beef for cooking, cardboard and glue for crafting, and gear and tire for vehicle maintenance. The framework uses tags and labels defined in Table 1 and Table 2, respectively. These tags and labels are slightly modified from the definitions in the English-FG to avoid confusion, and we did not add or delete any tags and labels. Therefore, our framework is compatible with the English-FG, and we can learn representations jointly from the cooking and non-cooking domains.
1In our preliminary experiments, we found that predicting the tags at document-level improves accuracy by 10% compared to the prediction at sentence-level.
With this framework, we consider predicting flow graphs of open-domain procedural texts. The prediction is performed in two stages: node prediction and edge prediction as in Section 2. Models are trained in a supervised way as the previous approach (Maeta et al., 2015). However, preparing a large number of examples in a new domain is unrealistic, considering the huge FG annotation cost. Thus, we assume that only a few training examples are available in the target domain. To tackle this issue, we consider domain adaptation from the existing cooking domain data to the target domain data. In the rest of this section, we formulate the task in Section 3.1, then consider data augmentation techniques that fit our setting in Section 3.2.

heading: 3.1 Task definition
text: We are given N examples in the cooking domain (V C1 , E C 1 ), · · · , (V CN , ECN ) and M examples in the target domain (V T1 , E T 1 ), · · · , (V TM , ETM ), where V C and EC are a set of vertices and edges in the cooking domain, while V T and ET are those in the target domain. Our goal is to maximize the performance of node and edge prediction models in the target domain. In this work, we use the EnglishFG corpus (300 articles) as the cooking domain examples and the wikiHow-FG corpus (Section 4) as the target domain examples. Note that M is a minimal number (namely, M = 5) in our setting, and the task has an aspect of low-resource domain adaptation (Xu et al., 2021). Note also that training with only the cooking or target-domain examples becomes zero-shot or few-shot learning scenarios, respectively.

heading: 3.2 Data augmentation
text: For improving performance in a low-resource setting, data augmentation is one of possible solutions (Fadaee et al., 2017; Ding et al., 2020). In this work, we consider the following two data augmentation techniques: step swapping and word replacement.
Step swapping augments an example by replacing two arbitrary steps in an article as illustrated in Figure 2. However, randomly choosing and swapping two steps might break their causal relationship. For example, we cannot swap two steps “Cut the potatoes.” and “Add the potatoes to the pan.” in Figure 2. In this work, we augment examples keeping this constraint by using flow graph annotations.
Word replacement augments an example by replacing a word with an arbitrary one. For each word in a step, we replace a word with one of its synonyms from WordNet (Dai and Adel, 2020) with a probability p.

heading: 4 wikiHow-FG corpus
text: The wikiHow-FG corpus is a new flow graph corpus from articles on wikiHow2, a website with more than 110K how-to articles. wikiHow articles have been used as a language resource for procedural texts (Zhou et al., 2019; Zellers et al., 2019; Zhang et al., 2020b,a; Zhou et al., 2022). In the following, we describe the data collection, annotation procedure, and statistics of the annotation results.

heading: 4.1 Data collection
text: For the target domains, we selected four categories from wikiHow: (i) Food and Entertaining, (ii) Hobbies and Crafts, (iii) Home and Garden, and (iv) Cars & Other Vehicles. We decided on those domains because many articles target and interact with substantial objects. We show examples of article titles in Table 3. Food and Entertaining is a domain close to the cooking domain as with the English-FG. Hobbies and Crafts is far from
2https://www.wikihow.com
the cooking domain in the sense that they assemble non-edible materials (e.g., making a bar soap). The remaining two domains are further from these domains because they contain non-assembly tasks (e.g., cleaning a table and fixing a broken chain).
We collected 30 articles for each domain from the wikiHow corpus (Zhang et al., 2020b). To exclude low-quality articles, we collected articles with 25 or more words and more than 50% user ratings. We then manually excluded articles with too abstract goals or not targeting substantial objects. We used article headlines as steps for annotation and experiments. Note that each article has a title and visual information (e.g., images or videos) describing the procedures. Exploiting them is interesting, but we leave that direction for future work.

heading: 4.2 Annotation procedure
text: Due to the dense, complex nature of the flow graph, constructing a high-quality corpus is a challenging problem. In order to guarantee the annotation quality, we first trained an annotator with 10 recipes sampled from the English-FG corpus (Yamakata et al., 2020). The training continued until the interannotator agreements with the ground-truth annotations reached over 80%. We then asked the annotator to annotate wikiHow articles. For both the node and edge annotations, we used the flow graph annotation tool (Shirai et al., 2022).3 The whole annotation took 40 hours.

heading: 4.3 Statistics
text: We show statistics of the wikiHow corpus in Table 4. The articles comprise 280.9 characters,
3Before the annotation, we tokenized steps into words with the stanza toolkit (Qi et al., 2020).
74.6 words, and 7.08 steps on average. The average number of tags and labels per article is 37.73 (±16.48) and 37.47 (±17.21) on Food and Entertaining, 35.40 (±9.08) and 35.87 (±9.44) on Hobbies and Crafts, 29.80 (±7.83) and 29.53 (±8.29) on Home and Garden, and 20.83 (±6.40) and 20.73 (±7.14) on Cars & Other Vehicles. Since articles on Home and Garden and Cars & Other Vehicles have fewer words than the other two domains, the number of tags and labels also becomes smaller.

heading: 4.4 Inter-annotator agreements
text: To assess the consistency of the annotations, we asked another annotator to re-annotate 10% of articles for each domain. We then measured F1 scores between the two sets of annotations using the original annotations as the ground-truth ones. Table 5 shows the results. The agreement for the node annotation (90.14%) was high, considering entities corresponding to the tags greatly change depending on the domain. For the edge annotation, the agreement (57.43%) drops from the one for node annotation. However, this agreement is also high considering errors from the node annotation influence this step and that a large number of candidate edges for the annotation.

heading: 5 Node prediction

heading: 5.1 Experimental settings
text: Model. We adopted a BiLSTM-CRF model (Lample et al., 2016) for node prediction by replacing a BiLSTM encoder with a pre-trained language model (LM) (Devlin et al., 2019; Liu et al., 2019; He et al., 2021). While previous work (Yamakata et al., 2020) used a pre-trained BERT (Devlin et al., 2019) as the encoder, we used a pre-trained DeBERTa (He et al., 2021).4 This model has 140M parameters in total.
Training. We trained a domain adaptation model by first training on the cooking domain data
4In our preliminary experiments, we confirmed that DeBERTa improves the accuracy of BERT by 0.47% on the English-FG corpus.
(English-FG) and then training again on the target domain data (wikiHow-FG). Note that we use only target domain examples of the wikiHow-FG corpus (e.g., when targeting Hobbies and Crafts, we do not use examples in the other three domains.). We also train models only on the cooking or target domain data and report the results to compare with the domain adaptation results.
For an optimization method, we used AdamW (Loshchilov and Hutter, 2019) with an initial learning rate of 5.0× 10−5 and a weight decay of 1.0 × 10−5. We tuned a learning rate with a cosine-annealing (Loshchilov and Hutter, 2019) (Sd steps) with a linear warm-up (Sw steps) at every iteration. We created a mini-batch from B articles. We set (B,Sw, Sd) = (5, 500, 4500) and (B,Sw, Sd) = (3, 100, 900) for training on the English-FG corpus and the wikiHow-FG corpus, respectively. We tuned these hyper-parameters on the development set. We used the data augmentation techniques only for the target domain data. For the step swapping, we created 5 augmented examples at maximum from one example. For the word replacement, we set 0.5 to p and created 10 examples from one example.
Evaluation. We split the English-FG corpus into 80% for training, 10% for validation, and the rest of 10% for testing. For the wikiHow-FG corpus, we split 30 articles of each domain into 6 folds. For more reliable results, we performed 6-fold crossvalidation by using 1 fold for training, 1 fold for validation, and the remaining 4 folds for testing. We used precision, recall, and F1 for evaluation metrics, following Yamakata et al. (2020). On the evaluation, we report the average scores of the models on the test set.
Model configurations. We refer to the domain adaptation models as domain-adaptation models. We also refer to the models trained only on the English-FG or the wikiHow-FG corpus as cookingonly and target-only models, respectively.

heading: 5.2 Results
text: The results are shown in Table 6. We see that the target-only models achieve an F1 score of 66.9% or more in all the target domains. This implies that the node prediction model can predict nodes to an extent with a few annotated articles. We also see that the cooking-only models achieve competitive performance with the target-only ones and outperform them in the two domains of Food and
Entertaining and Home and Garden. Particularly in Food and Entertaining, the cooking-only model surpasses the target-only one by 10.3% in F1. We consider that this domain is close to the cooking domain of the English-FG; thus, the cooking-only model is more advantageous as it can access more examples.
Next, the domain-adaptation models achieve the best performance in all the domains compared with the cooking-only and target-only models (76.5% or more F1). The most significant improvements are obtained in the two domains of Hobbies and Crafts and Cars & Other Vehicles (9.5% and 10.5% improvements in F1). These results indicate that domain adaptation from the cooking to the target domain is effective for training the node
prediction model. Third, we see that using the augmented data by the step swapping slightly improves performance from the domain-adaptation models in Food and Entertaining and Home and Garden. On the other hand, the word replacement does not contribute to any improvement. One possible reason is that a replaced word does not necessarily match the corresponding tag, and this disrupts the improvement.

heading: 5.3 Tag-level prediction performance
text: Entities for each tag can greatly change depending on the domain. In that case, the degree of improvement from the cooking-only to the domainadaptation model is expected to increase as the duplicate entities between the domains decrease.
To investigate this assumption, we measured taglevel prediction performance in F1 with the duplicate ratio of entities of the wikiHow-FG in the English-FG. We targeted the three tags of Ae, C, and T because these tags frequently appear in all the domains.
The results are shown in Table 7. For Ae, the degree of improvement from the cooking-only to the domain-adaptation model is small regardless of the duplicate ratios, which is contrary to our assumption. These results imply that recognizing entities for Ae is easy irrespective of the domain. For C and T, the domain-adaptation models significantly outperform the cooking-only ones in the three domains other than Food and Entertaining. These results imply that the domain adaptation is effective for recognizing C and T tags when the domain is further from the cooking domain.

heading: 6 Edge prediction

heading: 6.1 Experimental settings
text: Model. We adopted a biaffine dependency parser (Dozat and Manning, 2018) for edge pre-
diction.5 This model uses separate modules for edge prediction and label prediction. The resulting loss l is defined as a weighted sum of losses from the two modules:
l = λl(edge) + (1− λ)l(label), (2) where λ controls the strength of the two losses. We empirically set 0.5 to λ. We used a pre-trained DeBERTa (He et al., 2021) to obtain contextualized word representations. This model has 149M parameters in total.
Training. Similarly to Section 5.1, we trained a domain adaptation model for edge prediction first on the English-FG corpus and then on the wikiHowFG corpus. For an optimization method, we used AdamW (Loshchilov and Hutter, 2019) with a combination of a cosine-annealing and linear warm-up learning rate scheduling method. We used the same hyperparameters in Section 5.1.
Evaluation. We used the same splits of the English-FG and wikiHow-FG corpora as in Section 5 and performed 6-fold cross-validation for
5Previous work (Maeta et al., 2015) used a linear model, but we confirmed that our model achieves higher performance on the English-FG corpus.
more reliable results. We report the average scores of the models on the test set. For evaluation metrics, we used precision, recall, and F1 between predicted and ground-truth labeled edges of (u, v, l).
Model configurations. We used the same model notations of cooking-only, target-only, and domain-adaptation models as in Section 5.

heading: 6.2 Results
text: The results are shown in Table 8. We used groundtruth tags to identify nodes. Contrary to the node prediction results, the target-only models achieve poor performance in all the domains (33.8% or less in all the metrics). On the other hand, the scores of the cooking-only models are more than twice those of the target-only models. These results show that the edge prediction model requires more training examples than the node prediction one. These also show that with the English-FG corpus, predicting edges with 58.7% or more F1 is possible in noncooking domains.
Next, the domain-adaptation models outperform the target-only and cooking-only ones in all the domains. This is consistent with the results in the node prediction task. These results mean that the domain adaptation from the cooking to the target domain is also effective for the edge prediction model. For the results with the data augmentation techniques, the step swapping does not contribute to any improvement, contrary to Section 5.2. The word replacement improves the performance of the domain-adaptation models in the three domains other than Food and Entertaining.

heading: 6.3 Pipeline experiments
text: So far, the model has used ground-truth tags to identify nodes. However, in a realistic scenario, the model must predict labeled edges with the predicted nodes. In this scenario, errors in the node prediction step would affect performance in the edge prediction step. To investigate edge prediction
performance in this setting, we conducted experiments of edge prediction with the predicted nodes. We predicted nodes using the models in Section 5.2. In order to evaluate the model with tag information, we measured F1 of tuples of (u, v, l, nu, nv) between ground-truth and predicted ones, where nu and nv are the tags of the starting and ending nodes, respectively.
Table 9 shows the results with performance drops from those in Table 8. We see that 9.8% drops in Food and Entertaining, and more significant drops occur in the other three domains (about 24.6%). In these three domains, F1 scores of the node prediction are about 10% smaller than that of Food and Entertaining, and this gap would cause such large performance drops. We consider that improving node prediction performance would alleviate these drops.

heading: 7 Related work
text: Mori et al. (2014) designed a flow graph (FG) representation in the cooking domain and introduced a corpus of recipe texts. Subsequent works introduced a corpus in English (Yamakata et al., 2020) and corpora with visual annotations (Nishimura et al., 2020; Shirai et al., 2022). Maeta et al. (2015) proposed a method for an automatic FG prediction. Our work stems from this line of research and is the first attempt to apply the framework to non-cooking domains. Ours is also the first work to use a neural network-based method for the edge prediction.
Other than the recipe FG, there are several works that focus on obtaining an actionable representation from a procedural text. In cooking, Kiddon et al. (2015) proposed an unsupervised EM algorithm, while Pan et al. (2020); Papadopoulos et al. (2022) proposed supervised approaches. In biochemistry, Kulkarni et al. (2018); Tamari et al. (2021) introduced datasets for mapping wet lab protocols to an action graph. In material science, Kuniyoshi et al. (2020) represented the synthesis process with flow graphs. The works (Pan et al., 2020; Papadopoulos et al., 2022; Tamari et al., 2021; Kuniyoshi et al., 2020) are especially close to ours in the sense that they aim to obtain a document-level action graph in a supervised way.

heading: 8 Conclusion
text: We proposed a framework based on the EnglishFG and investigated flow graph prediction performance in non-cooking domains. We presented the
wikiHow-FG corpus from wikiHow articles. We considered domain adaptation from the cooking to the target domain. Experimental results show that domain adaptation models outperform those trained only on the cooking or target domain data. In future work, we consider applying this framework to other domains, such as material science and biochemistry. One can also try improving performance using more sophisticated data augmentation techniques. We hope that our work will provide new insights into procedural text understanding.

heading: Acknowledgments
text: We would like to thank anonymous reviewers for their insightful comments. This work was supported by JSPS KAKENHI Grant Number 20H04210 and 21H04910.

heading: Future Work
text: In future work, we consider applying this framework to other domains, such as material science and biochemistry. One can also try improving performance using more sophisticated data augmentation techniques. We hope that our work will provide new insights into procedural text understanding