Constructing Procedural Graphs with Multiple Dependency Relations: A New Dataset and Baseline
Year: 2023

Abstract:
Current structured and semi-structured knowledge bases mainly focus on representing descriptive knowledge but ignore another commonsense knowledge (Procedural Knowledge). To structure the procedural knowledge, existing methods are proposed to automatically generate flow graphs from procedural documents. They focus on extracting sequential dependency between sentences but neglect another two important dependencies (i.e., inclusion dependency and constraint dependency) in procedural documents. In our paper, we explore a problem of automatically generating procedural graph with multiple dependency relations to extend the flow graph constructed by existing methods and propose a procedural graph construction method with syntactic information and discourse structures. A new dataset (WHPG) is built and extensive experiments are conducted to evaluate the effectiveness of our proposed model.

heading: 1 Introduction
text: Many well-known structured knowledge bases (e.g., Wikidata1) and semi-structured knowledge bases (e.g., Wikipedia2) have been built and assist many applications to achieve remarkable performance, such as the question-answering (QA) (Li and Moens, 2022), information retrieval (Zhou et al., 2022)) and recommendation systems (Cui and Lee, 2022). They focus on representing the descriptive knowledge i.e. the knowledge of attributes or features of things (Yang and Nyberg, 2015), but lack another kind of commonsense knowledge— Procedural Knowledge. Specifically, the knowledge which is in the form of procedures or sequences of actions to achieve particular goals is called as procedural knowledge, such as cooking recipes and maintenance manuals.
∗The authors contribute equally. †Corresponding author
1http://wikidata.org/ 2http://wikipedia.org/
Generally, most procedural knowledge is expressed in unstructured texts (e.g., websites or books of cooking recipes). To extract the structured procedural knowledge, existing methods (Honkisz et al., 2018; Qian et al., 2020; Pal et al., 2021) are designed to transform the unstructured procedural documents into flow graphs (or workflows) which can effectively present the main operations and their ordering relations expressed in procedural documents. However, they only focus on extracting the sequential dependency (i.e., the dependency relation “Next” in Figure 1) between steps (operational sentences) in a procedural document, which is insufficient in real-world scenarios. As shown in Figure 1, sentences S2 and S3 are the sub-actions of sentence S1, which provide more fine-grained operational statements to finish operation S1. There is another kind of dependency— inclusion dependency between sentences S1 and S2 (or between S1 and S3). Nevertheless, the flow graphs constructed by current methods (Qian et al., 2020; Pal et al., 2021) ignore the inclusion dependencies among sentences and wrongly connect sentences S1 and S2 as a “Next” relation, as shown in Figure 1.
Furthermore, declarative (or descriptive) sentences commonly appear in real-world procedural documents, which state the constraints (e.g, reasons, conditions and effects) of doing things. Current researches have shown that declarative sentences in procedural documents can provide important clues for the procedural semantic understanding and reasoning (Georgeff and Lansky, 1986) in many downstream tasks such as operation diagnosis (Luo et al., 2021) and technical maintenance (Hoffmann et al., 2022). However, current knowledge structure methods (Qian et al., 2020; Pal et al., 2021) simply transform the declarative sentences into an information flow in a flow graph (e.g., S7 → S8 in Figure 1), which neglects the constraint dependency between operational and
8474
declarative sentences. As shown in Figure 1, the declarative sentences S7 and S8 respectively describe the effect constraint and condition constraint for the execution of operational sentence S6.
Based on the above motivations, we explore a problem of automatically constructing a procedural graph with multiple dependency relations between sentences in a procedural document. According to our observation, the syntactic structures of sentences can provide obvious features for identifying sentence types and then assist to detect the dependency relations between sentences. As shown in Figure 2, the syntactic pattern “verb(VB) obj−→noun(NN)” is a strong indicator for classifying sentences S2 and S3 into operation types. Meanwhile, the sentence type prediction can further benefit the dependency relation detection between sentences. For example, the constraint dependency cannot exist between two sentences with an operation type. Moreover, inspired by researches in discourse parsing (Zhu et al., 2019; Wang et al., 2021), we observe that the contextual dependency structures (which is called as dis-
course structures) can provide features to recognize the dependency relations between sentences. As shown in Figure 1, the dependency relation between S1 and S3 can be inferred as Sub-Action according to their contextual dependency structure S1
Sub-Action←− S2 Next-Action−→ S3. In our paper, we design a procedural graph construction method to detect the multiple dependency relations between sentences in procedural documents by utilizing syntactic information and discourse structures. Specifically, a GCN-based syntactic structure encoder with multi-query attention is proposed to capture the syntactic features of sentences and improve the ability to distinguish between operational and declarative sentences. Moreover, a structure-aware edge encoder is designed to assist the inference of dependencies between sentences by infusing the contextual structure features in procedural documents. Furthermore, due to the lack of dependencies between sentences in existing procedural text datasets, a new dataset WHPG is built based on the wikiHow3 database.
To summarize, the main contributions of this paper are listed as follows:
• We explore a problem of automatically generating procedural graphs with multiple dependencies from unstructured procedural documents, aiming to extend the flow graphs in existing methods that ignore dependencies of sentences. To the best of our knowledge, our work is the first study focusing on generating procedural graphs with multiple dependencies from procedural documents.
3http://www.wikihow.com/
• We design a GCN-based syntactic structure encoder and a discourse-structure aware edge encoder to effectively identify node types and assist the detection of dependency relations in procedural documents.
• We create a new procedural text dataset WHPG which builds dependency relations between operational and declarative sentences. Extensive experiments are conducted on two public datasets from different domains and our created dataset to evaluate the effectiveness of our model in automatically generating the procedural graph.

heading: 2 Related Work
text: Many prominent knowledge bases such as WikiData (Vrandečić and Krötzsch, 2014), Wikipedia (Lehmann et al., 2015) and FreeBase (Bollacker et al., 2007) mainly focus on representing descriptive knowledge (i.e., the knowledge of attributes or features of things (Yang and Nyberg, 2015)). But they do not sufficiently cover procedural knowledge (i.e., the knowledge of procedures or sequences of actions for achieving the particular goals (Georgeff and Lansky, 1986)). Recently, to obtain the structured procedural knowledge, two categories of methods (i.e., entitylevel and sentence-level based methods) are proposed. Specifically, the entity-level based methods (Jermsurawong and Habash, 2015; Feng et al., 2018; Mysore et al., 2019; Qian et al., 2020; Xu et al., 2020; Yamakata et al., 2020; Jiang et al., 2020; Fang et al., 2022) aim to extract the predefined entities and their relations from unstructured procedural texts (e.g., cooking recipes). However, they require large-scale fine-grained annotated data for each domain and lack domain generalization ability.
To alleviate these issues, sentence-level based methods (Pal et al., 2021) are designed, aiming to construct the flow graphs at sentence-level for procedural documents. However, they only focus on extracting the action flows with sequential dependencies, which is limited in real-world scenarios. In practice, both the inclusion dependency and constraint dependency are common in procedural texts and benefit the procedural text understanding and reasoning (Georgeff and Lansky, 1986; Hoffmann et al., 2022). Thus, our paper explores a problem of automatically generating a procedural graph with dependency relations from a procedural document.
Up to the present, several public entity-level procedural text datasets (Yamakata et al., 2020; Qian et al., 2020; Mysore et al., 2019; Mishra et al., 2018) and a sentence-level dataset CTFW (which is not publicly available due to ethical considerations) (Pal et al., 2021) are built. Nevertheless, existing public datasets do not annotate the dependency relations between sentences in a procedural document. Thus, a new dataset WHPG based on the wikiHow knowledge base is built and will be publicly available for evaluation in future research.

heading: 3 Model

heading: 3.1 Problem Definition and Notations
text: The goal of our task is to extract the dependency relations between sentences and construct a procedural graph for each procedural document. Specifically, given a procedural document D = {s1, s2, . . . , sN}, a procedural graph GD = {D,Ψ, R} with nodes (i.e., sentences) si ∈ D and triplets (si, ri,j , sj) ∈ Ψ is constructed, where ri,j ∈ R denotes the dependency relation between sentences si and sj ; N is the number of sentences in a procedural document D. Note that the dependency relation set R contains four kinds of dependency relations: Next-Action, Sub-Action, Constraint and None, as shown in Figure 1.
To construct the procedural graphs, three subtasks (i.e., Node Type Classification, Edge Prediction and Dependency Relation Classification) are required. For Node Type Classification task, each node si ∈ D is classified into one of the node types (i.e., Operation, Declaration, Both and None). Then, the extraction of triplets (si, ri,j , sj) from the procedural document D can be divided into two tasks: Edge Prediction P (si → sj |(s1, s2, . . . , sN )) which predicts whether an edge exists for each sentence pair Si and Sj ; and Dependency Relation Classification P (ri,j |si → sj) which classifies each sentence pair (predicted as an edge in Edge Prediction task) into one of the dependency relations (i.e., Next-Action, Sub-Action and Constraint).

heading: 3.2 Syntactic Graph Construction
text: The part-of-speech and syntactic structure of sentences can provide the evident clues for facilitating the inference of node types and dependency relations. For each sentence, we use the Standford CoreNLP libraries (Manning et al., 2014) to recognize the part-of-speech of each token and
dependency relations among tokens, as shown in Figure 3. Thus, given the sentence si = {xi,1, xi,2, . . . , xi,n}, a syntactic relational graph is created as follows:
Gsyni = {Pi,Φi, R syn i } (1)
where Pi represents a set of the part-of-speech in sentence si; R syn i is a set of syntactic dependency relations and Φi denotes a set of triplets (pi,j , r syn j,k , pi,k) with the part-of-speech pi,j ∈ Pi of the token xi,j and syntactic dependency relation rsynj,k ∈ R syn i between part-of-speech pi,j and pi,k.

heading: 3.3 Sentence (Node) Feature Representation

heading: 3.3.1 Syntactic RGCN Encoder
text: Each type of part-of-speech p ∈ P and dependency relation r ∈ Rsyn are respectively initialized into a learnable vector p ∈ Rdr and a learnable weight matrix Wr ∈ Rdr×dr . Then, given a syntactic graph Gsyni for sentence si, each part-of-speech node pi,j is encoded by Relational Graph Convolutional Networks (RGCN) (Schlichtkrull et al., 2018) encoder as follows:
p (l+1) i,j = σ(
∑
r∈Rsyni
∑
k∈Nrj
1
|N rj | W (l)r p (l) i,k+W (l) 0 p (l) i,j)
(2) where N rj is the set of neighborhood node for the node pi,j with the relation r ∈ Rsyni ; W0 ∈ Rdr
denotes the learning parameters and l is the number of layers in RGCN encoder.

heading: 3.3.2 Multi-Query Syntactic-Aware Attention
text: Moreover, not all the syntactic features are equally important to identify the node types and the dependency relations between the nodes. For example shown in Figure 3, the syntactic pattern “VB obj−→NN” is a strong indicator for classifying nodes as “Operation” types, while the pattern “Determiner (DT) det−→Noun (NN)” does not provide explicit features for the node type classification. Meanwhile, different tasks also focus on different syntactic features. Motivated by this, a multi-query syntactic-aware attention module is designed to enable the model to pay attention to the relevant syntactic features for the target tasks. Specifically, given a sentence si, the syntactic feature representation is obtained as follows:
vsyni =
Nq∑
k=1
n∑
j=1
αki,jpi,j
αki,j = exp(qkpi,j)∑n
m=1 exp(qkpi,m)
(3)
where n is the number of tokens in sentence si; Nq is the number of query and qk ∈ Rdr denotes a learnable vector.

heading: 3.3.3 BERT-based Bi-GRU Encoder
text: For a sentence si, each token xi,j can be encoded into a numerical vector vi,j ∈ Rdbert by the pre-trained language model BERT (Kenton and Toutanova, 2019). Then, a hierarchical GRU encoder consisting of two bidirectional GRUs (BiGRU) is utilized to learn the contextual features. Specifically, given a sequence of embedding vectors Ei = {vi,1,vi,2, . . . ,vi,n} of sentence si, the first Bi-GRU is utilized to encode them as hi ∈ Rdgru by concatenating the last hidden states from the two directions. In this way, each sentence si is encoded into a vector hi. Then, the procedural document D can be encoded as a sequence of vectors Hsent = {h1,h2, . . . ,hN}, where N is the number of sentences in the procedural document. Moreover, to capture the global contextual features, the second Bi-GRU encoder is adopted to transform Hsent into Hdialog = {vgru1 ,vgru2 , . . . ,vgruN }, where vgrui ∈ Rdgru denotes the feature representation of the sentence si ∈ D.

heading: 3.3.4 Feature Fusion
text: Each sentence si ∈ D can be obtained by concatenating the syntactic feature representation vsyni and the semantic feature representation vgrui , as follows:
vi = [v syn i ;v gru i ] (4)
where [·; ·] denotes the concatenation operation for the given two vectors.

heading: 3.4 Structural-Aware Edge Feature Representation
text: The contextual dependency structures in a procedural document have been proven to be effective in discourse parsing (Shi and Huang, 2019; Wang et al., 2021). The structure-aware attention is designed to capture the contextual structure features for each target sentence pair in both edge prediction and relation classification. Specifically, given a node pair (si, sj), the edge representation riniti,j is initialized by concatenating the syntactic feature representation vsyni , v syn j and the distance embedding vdisti,j as follows:
riniti,j = σ([v syn i ;v dist i,j ;v syn j ]W C) (5)
where i < j, j − i < win and win is the longest distance length between the given two nodes in a procedural document. Then, we update the node
representation vi in Equation (4) with the contextual features as follows:
vatti = N∑
j=1
αi,j(vjW V + riniti,j W F )
αi,j = exp(ei,j)∑N k=1 exp(ei,k)
ei,j = (viW
Q)(vjW K + riniti,j W R)T √ dr + dgru
(6)
where WQ,W F ,WK , W V and WR are learnable parameters and dr + dgru is the dimension of the node representations. Finally, the edge representation ri,j is calculated by refusing the node features, as follows:
γi,j = σ([v att i ;v att j ]W r) zi,j = σ([v att i ;v att j ]W z)
r ′ i,j = tanh([γi,j riniti,j ;vatti ;vattj ]W h)
ri,j = (1− zi,j) riniti,j + zi,j r ′ i,j
(7)
where W r, W z and W h are the learnable parameters; denotes the dot-product operation.

heading: 3.5 Projection and Loss Function
text: The representation of node si and edge ri,j can be encoded by Equation (4) and Equation (7) as vi and ri,j . We adopt a projection layer with a softmax function to calculate the probability distribution of categories (i.e., {Operation, Declaration, Both, None} for node type classification task; {Existing, Non-Existing} for edge prediction task and {Next-Action, Sub-Action, Constraint, None} for dependency relation classification task).
Given the training dataset M , the model is trained with the following training objective: £(M, θ) = ∑
D∈M (£t(D; θ)+£e(D; θ)+£r(D; θ))
(8) where £t(D; θ), £e(D; θ) and £r(D; θ) are the cross-entropy loss functions for node type classification, edge prediction and relation classification tasks; and D is a procedural document from the training dataset M .

heading: 4 Experiment
text: We firstly introduce the construction of the new dataset WHPG and then analyze the experimental results in detail.

heading: 4.1 Dataset Collection & Annotation
text: We build the original corpus from the online wikiHow knowledge base (Anthonio et al., 2020) which provides a collection of how-to articles about various topics (e.g., entertainment and crafts). We exploit the wikiHow knowledge base to create WHPG, a dataset of procedural texts with dependency-relation annotations among operational and declarative sentences. The online wikiHow knowledge base provides an Export pages4 service which allows exporting the texts of wikiHow articles (Anthonio et al., 2020). We adopt the python library urllib5 to request the Export pages services and crawl procedural articles. With the candidate set of procedural documents, we filter out the unnecessary information (e.g., writing date, citations and URLs). The procedural documents containing only one step are also filtered out. Finally, three parts (i.e., titles, method names and steps of procedural documents) are kept to form a complete instance. Statistically, we obtain a candidate set of 330 procedural documents from the Crafts topic.
As shown in Figure 1, for each procedural document, we provide three kinds of annotations: sentence type (i.e., “Operation”, “Declaration”, “Both” or “None”), edge (i.e., the connections between two sentences) and dependency relation (i.e., “NextAction”, “Sub-Action” and “Constraint”). Three well-educated annotators are employed to make annotations by averaging the candidate procedural
4https://www.wikihow.com/index.php?title= Special:Export&action=submit
5https://docs.python.org/2/library/urllib2. html
documents using the BRAT tool6. To ensure the annotation quality, each annotator are required to give the confidence score for each annotated label. We weigh the confidence score of each annotator for the same label and the label with the highest score will be preserved. Moreover, the annotation samples with the lowest confidence scores will be brainstormed to determine the final annotation results. Moreover, the labeled instances that are difficult to reach a consensus will be discarded. Finally, two well-trained annotators are required to recheck all the annotation results to further ensure the annotation quality.
Finally, the final dataset contains 283 procedural documents with about 7341 edges. The statistical comparison of WHPG with existing sentence-level procedural text datasets is shown in Table 1. Moreover, the statistics of sentence (node) types and dependency relations of our created dataset WHPG is shown in Table 2. We also show the label distributions of the node types and dependency relations, as shown in Table 3 and Table 4.

heading: 4.2 Experiment Settings

heading: 4.2.1 Datasets & Experimental Settings
text: We conduct extensive experiments7 on our annotated dataset WHPG. Following Pal et al. (2021), we split WHPG dataset into train, validation and test sets with 7:1:2 ratio. Furthermore, two public datasets (i.e., COR (Yamakata et al., 2020) and MAM (Qian et al., 2020) which do not consider dependency relations among sentences) are also
6http://brat.nlplab.org/index.html 7The code and datasets are publicly available at https:
//github.com/betterAndTogether/WHPG
utilized to conduct the comparative experiments on the Edge Prediction task.
For the Node Type Classification task, we use the accuracy as the evaluation metric. Considering the label imbalance in Edge Prediction task, F1score of the positive class (i.e., the sentence pair existing an edge) is used as the evaluation metric for the Edge Prediction. The performance of Dependency Relation classification task is affected by the previous stage Edge Prediction. Thus, we combine them to make an evaluation with the F1-score metric (i.e., Edge&Rel in Table 5).
In the edge prediction and dependency relation classification tasks, each sentence in the procedural document needs to be respectively combined with all the following sentences to determine whether there is an edge and what types of dependency relations they have. To evaluate the generalization ability, four experimental settings (i.e., win = 5, win = 10, win = 20 and ALL) are used to evaluate the effectiveness of our proposed model. For example, for the win = 5 setting, given the first sentence s1 ∈ D, five candidate sentence pairs i.e. {(s1, s2), (s1, s3), (s1, s4), (s1, s5), (s1, s6)} should be examined by the model to predict whether there are edges and which type of dependency relation they belong to. In training stage, we use AdamW optimizer with 4 batch size, 2e-5 learning rate and 0.4 dropout rate.

heading: 4.3 Result Analysis
text: To evaluate the effectiveness of our proposed model on the three tasks (i.e., Node Type Classification, Edge Prediction and Dependency Relation Clas-
sification), we compare the performance of our proposed model with 7 recent related works (Pal et al., 2021; Zhou and Feng, 2022) which focus on constructing flow graphs from a procedural document, as shown in Table 5. To explore the problem of generating procedural graphs from procedural documents, a new dataset WHPG is built and utilized to perform comparative experiments on both edge prediction and dependency relation classification tasks. Moreover, another two public datasets (i.e., COR (Yamakata et al., 2020) and MAM (Qian et al., 2020)) from different domains are used to conduct the comparative experiments. Since these two public datasets ignore the dependencies between sentences, we can only perform experiments on Edge Prediction task.

heading: 4.3.1 Node Type Classification
text: As shown in Table 6, five baselines (Pal et al., 2021; Zhou and Feng, 2022) are used to perform the comparative experiments. Compared with them, syntactic structures are embedded into the node feature representations in our model. From the experimental results, our model achieves the highest accuracies than current related works on both validation and test datasets. It can evaluate that syntactic structure features can be effectively captured and further improve the ability to distinguish between operational and declarative sentences.

heading: 4.3.2 Edge & Relation Classification
text: Table 5 shows the comparative experimental results under four window size settings (i.e., 5, 10, 20 and ALL) on both edge prediction and dependency relation classification tasks. Our proposed
model achieves the highest F1 scores with a large margin (nearly 15% F1 score in Edge and 20% F1 score in Edge&Rel) than current related works on all experiment settings. Specifically, current existing methods mainly focus on constructing a flow graph with only sequential dependency for each procedural document. They ignore another two significant dependencies (i.e., inclusion dependency and constraint dependency) between sentences. By capturing syntactic structures and discourse structures, our model can effectively identify the dependencies between sentences. Moreover, we observe that the performance of all models degrades as the window size increases (e.g., BERT-NS and BERTGCN drop nearly 15% F1 score from 5 to 10 window size settings). Instead, our proposed model has the smallest performance drop (only nearly 3% F1 score) on both edge prediction and dependency relation classification when the window size is reduced. Comparing with current related works, the contextual dependency structure (discourse structure) features are utilized to assist the inference of detecting dependency relations between sentences. The experimental results can evaluate the advantages of our model with structure-aware attention module in handling long-range inter-sentence dependency recognition.
Furthermore, to evaluate the domain generalization ability of our model, another two public datasets (i.e., COR in recipe domain and MAM in the maintenance domain) are used to conduct the comparative experiments. As shown in Table 7, our model obtains the best performance with a large margin than all related works on both datasets. The experimental results evaluate that our proposed model can effectively identify sequential dependency between sentences and have a better domain generalization ability.

heading: 4.3.3 Analysis for Each Dependency Relation
text: Figure 4 shows the comparative experiments on extracting each type of dependency relations between sentences. Compared with the related works, our model can obtain the highest F1 scores on all dependency relation types. Note that due to the imbalance in the number of existing and non-existing edges between sentences in procedural documents, current existing methods are prone to recognize the inter-sentences as None dependency and have low performance in the three dependency relations (i.e., Next-Action, Sub-Action and Constraint).

heading: 4.3.4 Ablation Study
text: As shown in Table 5, the ablation experiments are conducted to evaluate the effectiveness of the designed modules (i.e., SynEncoder, MultiQAtt and SAtt). The ablation experimental results can evaluate that both the syntactic information and discourse structure benefit the dependency relation detection. Specifically, both the SynEncoder and MultiQAtt modules can effectively capture the syntactic features and assist the dependency relation detection. Moreover, the performance of our model can be improved effectively when the discoursestructure features are embedded by the structure aware attention module.

heading: 4.4 Visualization
text: As shown in Figure 5, we show the weight distributions of each word measured by the MultiQuery Syntactic Aware Attention Module. We can observe that the phrases with the syntactic pattern “VB obj−→NN” (e.g„ “cut→hole” in S1 and “use→cup” in S2) obtain the higher weight values than other words, which indicates the sentences as operational sentences. Moreover, the token “Then” in S3 of Figure 5 is measured as the highest weight value, which indicates the sentence has the “NextAction” dependency relation with previous sentences. The visualization analyse can evaluate the effectiveness of our proposed module Multi-Query Syntactic Aware Attention.

heading: 5 Conclusion
text: In this paper, we explore a problem of automatically generating procedural graphs with multiple dependency relations for procedural documents. Existing procedural knowledge structured methods mainly focus on constructing action flows with the sequential dependency from procedural texts but neglect another two important dependencies: inclusion dependency and constraint dependency which are helpful for the procedural text understanding
and reasoning. To solve this problem, we build a new procedural text dataset with multiple dependency relations and propose a procedural graph construction method by utilizing syntactic and discourse structure features. Extensive experiments are conducted and evaluate the effectiveness of our proposed model.

heading: 6 Limitations
text: In this section, we draw conclusions for the limitations of our proposed model in this paper. Our proposed model mainly focuses on the sentencelevel procedural graph construction. The scenario that two actions in the same sentence cannot be considered in our proposed model. It is challenging to handle multi-grained (i.e., entity-level and sentence-level) dependencies between actions. We will consider this limitation as our future work.

heading: Acknowledgement
text: This work was supported by the National Natural Science Foundation of China (62076100), Fundamental Research Funds for the Central Universities, SCUT (x2rjD2220050), the Science and Technology Planning Project of Guangdong Province (2020B0101100002), CAAI-Huawei MindSpore Open Fund, CCF-Zhipu AI Large Model Fund.

heading: A For every submission:
text: 3 A1. Did you describe the limitations of your work?
6
A2. Did you discuss any potential risks of your work? Not applicable. Left blank.
3 A3. Do the abstract and introduction summarize the paper’s main claims? 1
7 A4. Have you used AI writing assistants when working on this paper? Left blank.
B 7 Did you use or create scientific artifacts? Left blank.
B1. Did you cite the creators of artifacts you used? No response.
B2. Did you discuss the license or terms for use and / or distribution of any artifacts? No response.
B3. Did you discuss if your use of existing artifact(s) was consistent with their intended use, provided that it was specified? For the artifacts you create, do you specify intended use and whether that is compatible with the original access conditions (in particular, derivatives of data accessed for research purposes should not be used outside of research contexts)? No response.
B4. Did you discuss the steps taken to check whether the data that was collected / used contains any information that names or uniquely identifies individual people or offensive content, and the steps taken to protect / anonymize it? No response.
B5. Did you provide documentation of the artifacts, e.g., coverage of domains, languages, and linguistic phenomena, demographic groups represented, etc.? No response.
B6. Did you report relevant statistics like the number of examples, details of train / test / dev splits, etc. for the data that you used / created? Even for commonly-used benchmark datasets, include the number of examples in train / validation / test splits, as these provide necessary context for a reader to understand experimental results. For example, small differences in accuracy on large test sets may be significant, while on small test sets they may not be. No response.
C 3 Did you run computational experiments? Left blank.
3 C1. Did you report the number of parameters in the models used, the total computational budget (e.g., GPU hours), and computing infrastructure used? 4
The Responsible NLP Checklist used at ACL 2023 is adopted from NAACL 2022, with the addition of a question on AI writing assistance.
3 C2. Did you discuss the experimental setup, including hyperparameter search and best-found hyperparameter values? 4
3 C3. Did you report descriptive statistics about your results (e.g., error bars around results, summary statistics from sets of experiments), and is it transparent whether you are reporting the max, mean, etc. or just a single run? 4
3 C4. If you used existing packages (e.g., for preprocessing, for normalization, or for evaluation), did you report the implementation, model, and parameter settings used (e.g., NLTK, Spacy, ROUGE, etc.)? 4
D 3 Did you use human annotators (e.g., crowdworkers) or research with human participants? 4
3 D1. Did you report the full text of instructions given to participants, including e.g., screenshots, disclaimers of any risks to participants or annotators, etc.? 4
3 D2. Did you report information about how you recruited (e.g., crowdsourcing platform, students) and paid participants, and discuss if such payment is adequate given the participants’ demographic (e.g., country of residence)? 4
3 D3. Did you discuss whether and how consent was obtained from people whose data you’re using/curating? For example, if you collected data via crowdsourcing, did your instructions to crowdworkers explain how the data would be used? 4
D4. Was the data collection protocol approved (or determined exempt) by an ethics review board? Not applicable. 4
D5. Did you report the basic demographic and geographic characteristics of the annotator population that is the source of the data? Not applicable. Left blank.

heading: Future Work
text: We will consider this limitation as our future work