Venue: ACL 2023

# Learning Action Conditions from Instructional Manuals

for Instruction Understanding

Te-Lin Wu\({}^{1}\), Caiqi Zhang\({}^{2}\), Qingyuan Hu\({}^{1}\), Alex Spangher\({}^{3}\), Nanyun Peng\({}^{1}\)

\({}^{1}\)University of California, Los Angeles, \({}^{2}\)University of Cambridge,

\({}^{3}\)Information Sciences Institute, University of Southern California

{telinwu,violetpeng,hu528}@cs.ucla.edu, cz391@cam.ac.uk, spangher@isi.edu

###### Abstract

The ability to infer pre- and postconditions of an action is vital for comprehending complex instructions, and is essential for applications such as autonomous instruction-guided agents and assistive AI that supports humans to perform physical tasks. In this work, we propose a task dubbed action condition inference, which extracts mentions of preconditions and postconditions of actions in instructional manuals. We propose a weakly supervised approach utilizing automatically constructed large-scale training instances from online instructions, and curate a densely human-annotated and validated dataset to study how well the current NLP models do on the proposed task. We design two types of models differ by whether contextualized and global information is leveraged, as well as various combinations of heuristics to construct the weak supervisions. Our experiments show a >20% F1-score improvement with considering the entire instruction contexts and a > 6% F1-score benefit with the proposed heuristics. However, the best performing model is still well-behind human performance.1

Footnote 1: Dataset and codes will be released at: here.

+
Footnote †: July 9-14, 2023 ©2023 Association for Computational Linguistics

## 1 Introduction

When performing complex tasks _(making a gourmet dish)_, instructional manuals are often referred to as useful guidelines. To follow the instructed actions, it is crucial to understand the _preconditions_, prerequisites before taking a particular action, and the _postconditions_, the status supposed to be reached after performing the action. Knowledge of action-condition dependencies is prevalent and inferable in many instructional texts. For example, in Figure 1, before performing the action "_place oionions_" in step 3, both _preconditions_: "_heat the pan_" (in step 2) and "_slice oionions_" (in step 1) have to be successfully accomplished. Likewise, executing "_stit onions_" (in step 4), leads to its _postcondition_, "_caramelized_" (also in step 4).

For autonomous agents or assistant AI that aids humans to accomplish tasks, understanding the conditions provides a structured view of a task Linden (1994); Aeronautiques et al. (1998); Branavan et al. (2012a); Sharma and Kroemer (2020) and helps the agent correctly judge whether to _proceed_ to the next action and _evaluate_ the action completions. However, no prior work has systematically studied automatically extracting pre- and postconditions from prevalent data resources. To bridge this gap, we propose the _action condition inference task_ on **real-world instructional manuals**, where a _dense dependency graph_ is produced, as in Figure 1, to denote the pre- and postconditions of actions. Such a dependency graph provides a systematic task execution plan that agents can closely follow.

We consider two online instruction resources, _WikiHow_Hadley et al. (2017) and _Instructables.com_Insturctables), to study the current NLP models' capabilities of performing the proposed task. As there is no densely annotated dataset on the desired action-condition-dependencies from real-world instructions, and annotating a comprehensive depen

dency structure of actions for long instruction contexts can be extremely expensive and laborious, we collect human annotations on a subset of totally 650 samples and benchmark models in either a **zero-shot** setting where no annotated data is used for training, or a **low-resource/shot** setting with limited amount of annotated training data.

We also design the following heuristics and show that they can effectively construct large-scale _weak supervisions_: (1) **Key entity tracing:** Key repetitive entity mentions (including **co-references**) across different instruction descriptions likely suggest a dependency. (2) **Keywords:** Certain keywords (_e.g_. the before in "\(do~{}\chi\) before doing \(\gamma\)") can often imply the condition dependencies. (3) **Temporal reasoning:** We adopt a temporal relation module Han et al. (2021b) to alleviate the potential inconsistencies between the narrated orders of conditional events and their actual temporal orders to better utilize their temporally grounded nature (_e.g_. preconditions are _prior to_ an action).

We benchmark two strong baselines based on pretrained language models with or without instruction contexts on our annotated held-out test-set, where the models are asked to make predictions _exhaustively_ on **every possible dependency**. We observe that contextualized information is essential (> 20% F1-score gain over non-contextualized counterparts), and that our proposed heuristics are able to augment an effective weakly-supervised training data to further improve the performance (> 6% F1-score gain) on the low-resource setting. However, the best results are still well below human performance (> 20% F1-score difference).

Our key contributions are three-fold: (1) We propose an action-condition inference task and create a densely human-annotated _evaluation dataset_ to spur research on structural instruction comprehensions. (2) We design linguistic-centric heuristics utilizing entity tracing, keywords, and temporal reasoning to construct effective large-scale weak supervisions. (3) We benchmark models on the proposed task to shed lights on future research.

## 2 Terminologies and Problem Definition

Our goal is to learn to infer action-condition dependencies in real-world instructional manuals. We first describe essential terminologies in details:

**Actionable** refers to a phrase that a person can follow and execute _in the real world_ (yellow colored phrases in Figure 2). We also consider negated actions (_e.g_. _do not ..._) or actions warned to avoid (_e.g_. _if you purchase the wrong..._) as they likely also carry useful knowledge regarding the tasks.2

Footnote 2: We ask workers to single out the actual _actionable_ phrases, _e.g_. _purchase the wrong line \(\rightarrow\) trimmer will not work_.

**Precondition** concerns the _prerequisites_ to be met for an actionable to be executable, which can be a status, a condition, and/or another prior actionable (blue colored phrases in Figure 2). It is worth noting that humans can omit explicitly writing out certain condition statements because of their triviality as long as the actions inducing them are mentioned (_e.g_. _heat the pan \(\rightarrow\) pan is heated_, the latter can often be omitted). We thus generalize the conventional precondition formulation, _i.e_. sets of statements evaluated to true/false Fikes and Nilsson (1971), to a phrase that is either a passive condition statement or an _actionable that induces_ the prerequisite conditions, as inspired by Linden (1994).

**Postcondition** is defined as the outcome caused by the execution of an actionable, which often involves status changes of certain objects (or the actor itself) or certain effects emerged to the surroundings or world state (green colored phrases in Figure 2).

Figure 2: **Terminologies: (Left) shows a few exemplar actionables with their associated preconditions and postconditions. Notice that an actionable can have multiple pre- or postconditions and they can span across different instruction steps (for simplicity we do not show an exhausted set of text segments, and the actual instruction contexts are much longer). (Right) SRL is used to postulate the text segments (actionables and conditions). We show a sample SRL extraction corresponding to one of the dependency linkages on the left. The SRL ARG labels also provide useful information for designing our heuristics (Section 4).

 



**Text segment** in this paper refers to a textual segment of interest, which can be one of: {actionable, precondition, postcondition}, in an article.

In reality, a valid actionable should have both _pre-_ and _postcondition_ dependencies, however, we do not enforce this in this work as conditions can occasionally be omitted by human authors.

**Problem Formulation.** Given an input instructional manual and some text segments of interest extracted from it, a model is asked to predict the _directed_ relation between a pair of segments, where the relation should be one of the followings: NULL (no relation), _precondition_, or _postcondition_.

## 3 Datasets and Human Annotations

As the condition-dependency knowledge we are interested in is prevalent in real-world instructions, we consider two popular online resources, **WikiHow** and **Instructables.com**, both consist of detailed multi-step task instructions, to support our investigation. For WikiHow, we use the provided dataset from Wu et al. (2022); for Instructables, we scrape the contents directly from their website.

Since densely annotating large-scale instruction sources for the desired dependencies is extremely expensive and laborious, we mainly annotate a _test-set_ and propose to train the models via weakly or self-supervised methods. We hence provide a small subset of the human-annotated data to adapt models to the problem domain. To this end, we collect comprehensive human annotations on a selected subset in each dataset to serve as our **annotated-set**, and particularly the subsets used to evaluate the models as the **annotated-test-set**.3 In total, our densely annotated-set has 500 samples in WikiHow and 150 samples in Instructables, spanning 7,191 distinct actions (defined by main predicate-object phrases) for diversity. In Section 6.2, we will describe how the annotated-set is split to facilitate the low-resource training. We also collect the human performance on the annotated-test-set to gauge the human upper bound of our proposed task. More dataset details are in Append. Sec. A.

Footnote 3: Following Wu et al. (2022), we first choose from physical categories and then sample a manually inspected subset.

### Annotations and Task Specifications

**Dataset Structure.** The desired structure of the constructed data, as in Figure 2, features two main components: (1) **text segment** of interest (see Section 2), and (2) **condition linkage**, a _directed_ and _relational_ link connecting a pair of text segments.

**Annotation Process.** We conduct the annotated-set construction via Amazon Mechanical Turk (MTurk). Each worker is asked to carefully **read over thoroughly** a prompted complex multi-step instructional manual, where the annotation process consists of three main steps: **(1) Text segments highlighting:** To facilitate this step (and postulating the text segments for constructing weak-supervisions in Section 4), we _pre-highlight_ several text segments extracted by _semantic role labelling_ (SRL) for workers to choose from.4 They can also freely annotate (highlight by cursor) their more desirable segments. **(2) Linking:** We encourage the workers to annotate all the possible segments of interest, and then they are asked to connect certain pairs of segments that are likely to have dependencies with a directed edge. **(3) Labelling:** Finally, each directed edge drawn will need to be labelled as either a _pre-_ or _postcondition_ (NULL relations do not need to be explicitly annotated).

Footnote 4: SRL V and ARGs are connected alongside intermediate words to form contiguous segments (see Append. Sec. C.1.1).

In general, for each article a worker is required to consider on average \(>\)\(500\) pairwise relations with all associated article contexts (\(>\)\(300\) tokens), which is a **decently laborious task**. Comparisons on the linkage annotations from different workers are as well made on _every_ pair of _their respective annotated_ text segments with the _actual candidate-consideration_ from the **entire** rest of article.

Since the agreements among workers on both text segments and condition linkages are sufficiently high5 given the complexity of the annotation task, our final human annotated-set retains the _majority voted_ segments and linkages.

Footnote 5: The mean inter-annotator agreements (IAAs) per Fleiss Kappa for (segments, linkages) are (0.90, 0.57) and (0.88, 0.58) for WikiHow and Instructables. Note that the Kappa agreement measures the extent to which the observed amount of agreement among raters exceeds what would be expected if all raters made their ratings completely randomly, so the agreement is high. See Append. Sec. B.1 for more details.

**Variants of Tasks.** Although proper machine extraction of the text segments of interest as a span-based prediction can be a valid and interesting task, we find that our automatic SRL extraction is already sufficiently reliable.6 In this paper, we thus mainly focus on the more essential linkage prediction (and their labels) task assuming that these text segments

are given, and leave the possible end-to-end system with the (refined) text segment extraction, as the future work. Our proposed task and the associated annotated-set can be approached by a **zero-shot** or **low-resource** setting: the former involves no training on any of the annotated data and a heuristically constructed training set can be utilized (Section 4), while the latter allows models to be finetuned on a limited annotated-subset (Section 5.3). For the low-resource setting particularly, only 30% of the annotated data will be used for training (details of splits and considerations see Section 6.2).

## 4 Training With Weak Supervision

As mentioned in Section 3, our proposed task can be approached via a zero-shot setting, where the vast amount of **un-annotated instruction data** can be transformed into useful training resources (same dataset structure as described in Section 3.1). Moreover, it is proven that in many low-resource NLP tasks, constructing a much larger heuristic-based weakly supervised data can be beneficial [9, 8].

### Linking Heuristics

The goal of designing certain heuristics is to perform a rule-based determination of the linkage (its direction and the condition label). Our design intuition is to harness dependency knowledge by exploiting relations between actions and entities (_entity-level_), certain linguistic patterns (_phrase-level_), and _event-level_ information, which should be widely applicable to all kinds of instructional data. Concretely, we design four types of heuristics: (1) **Keywords:** certain keywords are hypothesized to show strong implication of conditions such as _if_, _before_, _after_; (2) **Key entity tracing:** text segments that share the same key entities are likely indicating dependencies; (3) **Co-reference** resolution is adopted to supplement (2); (4) **Event temporal relation resolution** technique is incorporated to handle the inconsistencies between narrative order and the _actual_ temporal order of the events.

**SRL Extraction.** Without access to human refinements (Section 3.1), we leverage SRL to postulate all the segments of interests to construct the weakly-supervised set. As SRL can detect multiple plausible ways to form the ARG frames with respect to the same _central_ verb, we need to additionally determine the most desirable parses _for each action verb_. In this work, we simply select the most desirable SRL parses by choosing ones that maximize both: (1) the number of plausible segments (each centered around an action verb) _within a sentence_, where they do not overlap above a certain threshold (set to be 60% in this work), and (2) the number of ARGs in each of such segment.

#### 4.1.1 Keywords

Table 2 lists the major keywords that are considered in this work. Denote a text segment as \(a_{i}\), keywords are utilized so as the text segments separated with respect to them, _i.e_. \(a_{1}\) and \(a_{2}\), can be properly linked. Different keywords and their positions within sentences can lead to different _directions_ of the linkages, _i.e_. \(a_{1}\rightleftarrows a_{2}\) (see second row of Table 1, note that here condition labels are not yet determined). For example, keywords before and after intuitively can lead to different directions if they are placed at non-beginning positions. We follow

\begin{table}
\begin{tabular}{c|p{71.1pt}|p{142.3pt}} \hline \multicolumn{2}{c}{standardone} \\ \hline \hline
**Heuristics** & **Examples** & **Descriptions** \\ \hline Entity-Tracing & ... Heat the pan with olive oil ... Size 500 grams of options ... & The shared entities are pan and onions (linked via co-references to them). \\  & Production ... Place them in the frying fan ... Protocolno 2 & Keywords are used to link the segments they separate. If the keyword is at the beginning (2nd example), the (1st) comma is used to segment the sentences. \\  & ... Make sure everything is dry before you fill your flowerd out diff ... _If you’re using a machine punch, stick the rivet through the hole ..._Presentation._ & Keywords are used to link the segments they separate. If the keyword is at the beginning (2nd example), the (1st) comma is used to segment the sentences. \\ \hline Postcondition & Warm a pan with oil over medium heat. ... Do not pour surface into your look.. ... the water will be frozen sold ... & Certain linguistic hints (_e.g_. SRL tags) are utilized to propose plausible (and likely) postcondition text segments. \\ \hline Temporal & ... Step down hard on the rubber part of the irq ... _Presentation._ & The action prying should occur prior to stepping, but these two segments are reversely narrated in the contexts. \\  & ... After ... jump off the back side of the fire first ... & 
\begin{tabular}{l} The atomic prying should occur prior to stepping, but these two segments are reversely narrated in the contexts. \\ \end{tabular} \\ \hline \hline \end{tabular}
\end{table}
Table 1: **Heuristics** used for determining condition linkages between text segments, with sample use-cases and descriptions.

 

the rules listed in Table 2 to decide the directions.

#### 4.1.2 Key Entity Tracing

It is intuitive to assume that if the two text segments mention the same entity, a dependency between them likely exists, and hence a _trace_ of the same mentioned entity can postulate potential linkages. As exemplified in the first row of Table 1, that _heating the pan_ being a necessary precondition to _placing onions in the pan_ can be inferred by the shared mention "pan". We adopt two ways to propose the candidate entities: (1) We extract all the _noum phrases_ within the SRL segments (mostly ARG-tags), (2) Inspired by Bosselut et al. (2018), a model is learned to predict potential entities involved that are not explicitly mentioned (_e.g.__fry the chicken_ may imply a _pan_ is involved) in the context (more details see Append. Sec. C.1.4).

**Co-References.** Humans often use pronouns to refer to the same entity to alternate the mentions in articles, as exemplified by the mentions onions and them, in the first row of Table 1. Therefore, a straightforward augmentation to the aforementioned entity tracing is incorporating co-references of certain entities. We utilize a co-reference resolution model Lee et al. (2018) to propose possible co-referred terms of extracted entities of each segment within the same step description (we do not consider cross-step co-references for simplicity).

### Linking Algorithm

After applying the aforementioned linking heuristics, each text segment \(a_{i}\), can have \(M\) linked segments: \(\{a^{l_{i}}_{1},...,a^{l_{i}}_{M}\}\). For linkages that are _traced_ by entity mentions (and co-references), their directions always start from priorly narrated segments to the later ones, while linkages determined by the keywords follow Table 2 for deciding their directions. However, the text segments that are narrated too much distant away from \(a_{i}\) are less likely to have direct dependencies. We therefore _truncate_ the linked segments by ensuring any \(a^{l_{i}}_{j}\) is narrated **no more than** "\(S\) step" ahead of \(a_{i}\), where \(S\) is empirically chosen to be \(2\) in this work.

Despite pruning the traces with the aforementioned design choice \(S\) can largely reduce _condition-irrelevant_ segments, such heuristic indeed cannot guarantee the included text segments are always dependent with respect to an actionable. Our goal here is to exploit the generalization ability of language models to _recognize_ segments that are most probable conditions by including as many heuristically proposed linkages as possible, where a better strategy on designing the maximum allowed step-wise distance is left as a future work.

#### 4.2.1 Incorporating Temporal Relations

As hinted in Section 2, the conditions with respect to an actionable imply their temporal relations. The direction of an entity-trace-induced linkage is naively determined by the narrated order of text segments within contexts, however, in some circumstances (_e.g._ fourth row in Table 1), the narrative order can be inconsistent with the actual temporal order of the events. To alleviate such inconsistency, we apply an event temporal relation prediction model Han et al. (2021) (trained on various temporal relation datasets such as _MATRES_ Ning et al. (2018)) to fix the linkage directions.7

Footnote 7: These do not include linkages decided by the _keywords_.

We train the model on three different random seeds and make them produce a _consensus_ prediction, unless all of the models jointly predict a specific relation (BEFORE or AFTER), otherwise the relation will be regarded as VAGUE. The model is then applied to predict temporal relations of each pair of event triggers (extracted by SRL, verbs/predicates), and then we invert the direction of an entity-trace-induced linkage, \(a^{l_{i}}_{j}\to a_{i}\), if their predicted temporal relation is opposite to their narrated order (VAGUE is of course ignored).

#### 4.2.2 Labelling The Linkages

It is rather straightforward to label precondition linkages as a simple heuristic can be used: for a given segment, _any segments that linked to the current one that are either narrated or temporally prior to it_ are plausible candidates for being preconditions. For determining postconditions, where they are mostly descriptions of status (changes), we therefore make use of certain linguistic cues that likely indicate human written status, _e.g.__the

\begin{table}
\begin{tabular}{c|c c} \hline \hline
**Keywords** & Begin. & Within Sent. \\ \hline before, until. in order to, so & \(a_{1}\longrightarrow a_{2}\) & \(a_{1}\longleftarrow a_{2}\) \\ \hline requires & — & \(a_{1}\longrightarrow a_{2}\) \\ \hline after, once. if & \(a_{1}\longleftrightarrow a_{2}\) & \(a_{1}\longrightarrow a_{2}\) \\ \hline \hline \end{tabular}
\end{table}
Table 2: **Keywords for deciding a potential linkage: If a keyword is at the beginning of a sentence, we use the (first) comma of that sentence to separate it to two segments and link them accordingly, while the keyword itself is used as the separator otherwise. The segments are then either refined with SRL or kept as they are if SRL does not detect a valid verb.

 

water will be frozen_ and _the oil is sizzling_. Specifically, we consider: (1) \(be\text{-}verbs\) followed by present-progressive tenses if the subject **is an entity**, and (2) segments whose SRL tags start with ARGM as exemplified in Table 1.

## 5 Models

Our proposed heuristics do not assume specific model architecture to be applicable, and to benchmark the proposed task, we mainly consider two types of **base models**: (1) **Non-contextualized** model takes only the _two text segments_ of interest at a time and make the _pairwise_ trinary (directed) relation predictions, _i.e_. NULL, _precondition_, and _postcondition_; (2) **Contextualized** model also makes the relation predictions for every pair of input segments, but the inputs include the whole instruction article so the contexts are preserved. The two models are both based off pretrained language models (the non-contextualized model is essentially a standard transformer-based language model finetuned for classification tasks), and the relation prediction modules are multi-layer perceptrons (MLPs) added on top of the language models' outputs. Cross-entropy loss is used for training.

### Non-Contextualized Model

The non-contextualized model takes two separately extracted text segments, \(a_{i}\) and \(a_{j}\), as inputs and is trained similarly to the next sentence prediction in BERT Devlin et al. (2019) (_i.e_. the order of the segments matters, which will be considered in determining their relations), as shown in Figure 3.

### Contextualized Model

The architecture of the contextualized model is as depicted in Figure 3. Denote the tokens of the instruction text as \(\{t_{i}\}\) and the tokens of \(i\)-th text segment of interest (either automatically extracted by SRL or annotated by humans) as \(\{a_{ij}\}\). A special start and end of segment token, <a> and </a>, is wrapped around each text segment and hence the input tokens become: "\(t_{1},...,t_{k}\), <a>\(a_{il},a_{iz},...,a_{ik}\) </a>,...". The contextualized segment representation is then obtained by applying a mean pooling over the language model output representations of each of its tokens, _i.e_. denote the output representation of \(a_{ij}\) as \(\textbf{o}(a_{ij})\), the segment representation of \(\textbf{o}(a_{i})\) is \(AvgPool(\sum_{j=1}^{K}\textbf{o}(a_{ij}))\). To determine the relation between segment \(i\) and \(j\), we feed their _ordered_ concatenated representation, \(concat(\textbf{o}(a_{i}),\textbf{o}(a_{j}))\), to an MLP for the relation prediction.

### Learning

**Multi-Staged Training.** For different variants of our task (Section 3.1), we can utilize different combinations of the heuristically constructed dataset and the annotated-train-set. For the low-resource setting, our models can thus be firstly trained on the constructed training set, and then finetuned on the annotated-set. Furthermore, following the **self-training** paradigm Xie et al. (2020); Du et al. (2021), the previously obtained model predictions can be utilized to either _augment_ (_i.e_. adding linkages) or _correct_ (_i.e_. revising linkages) the original heuristically constructed data. And hence a second-stage finetuning can be conducted on this model-self-annotated data for improved performance.

**Label Balancing.** It is obvious that most of the relations between randomly sampled text segment pairs will be NULL, and therefore the training labels are imbalanced. To alleviate this, we downsample the negative samples when training the models. Specifically, we fill each training mini-batch with equal amount of positive (relations are not NULL) and

Figure 3: **Model architectures: (a) Non-contextualized model: The model only considers a pair of given text segments. (b) Contextualized model: The model takes the whole instruction paragraphs (_i.e_. contexts) and wrap each text segment with our special tokens (<a>), where each segment representation is obtained by taking an average over its token representations. The _ordered_ concatenated segment representations will then be fed into an MLP to make the final predictions.**



negative pairs, where the negatives are constructed by either _inverting_ the positive pairs or _replacing_ one of the segment with another randomly sampled _unrelated_ segment within the same article.

## 6 Experiments and Analysis

Our experiments seek to answer these questions: (1) How well can the models and humans perform on the proposed task? (2) Is instructional context information useful? (3) Are the proposed heuristics and the second-stage self-training effective?

### Training and Implementation Details

For both non-contextualized and contextualized models, we adopt the pretrained RoBERTa (-large) language model Liu et al. (2019) as the base model. All the linguistic features, SRL Shi and Lin (2019), co-references, POS-tags, are extracted using models implemented by AllenNLP Gardner et al. (2017). We truncate the input texts at maximum length of 500 while ensuring all the text segments within this length is preserved completely.

All the models in this work (both pretraining and finetuning) are trained on a single Nvidia A100 (40G RAM) GPU. The hyperparameters are manually tuned against different datasets, and the checkpoints used for testing are selected by the best performing ones on the held-out development sets.

### Experimental Setups

**Data Splits.** The primary benchmark of WikiHow annotated-set is partitioned into **train (30%)**, **development (10%)**, and **test (60%)** set, respectively, resulting in 150, 50, and 300 data samples, for low-resource setting. We mainly consider the Instructables annotated-set in a **zero-shot setting** where we hypothesize the models trained on WikiHow can be well-transferred to it. For training conducted on the heuristically constructed data, including the second-stage self-training, we use respective held-out development sets to select the checkpoints around performance convergence for finetuning.

**Evaluation Metrics.** We ask the models to predict the relations on _every_ pair of text segments in a given instruction, and compute the average precision (Prec.), recall, and F-1 scores separately with respect to each (pre/post) condition labels.

**Baselines.** There is no immediate baseline we are aware of for the proposed action condition inference task. However, we note that Dalvi et al. (2019)'s dependency graph prediction on scientific procedures Mishra et al. (2018) shares high-level similarities to specifically our precondition inference task. Our non-contextualized model (without the second-stage self-training) with _only_ the nounphrase-based entity tracing heuristic resembles the KB-induced _prior dependency likelihood_, \(g_{kb}\), in their proposed XPAD framework.8

Footnote 8: With all entity-state-related components excluded (irrelevant to our task) and encoder replaced by RoBERTa model.

Beside this _adapted_-**XPAD**, we also evaluate our task with (1) **probabilistic random-guess baseline** (random guesses proportional to the training-set label ratio), and (2) **zero-shot GPT-3**Brown et al. (2020) where we prompt GPT-3 with exemplar data instances as the task definition (**contextualized**, see Append. Sec. C.2 for prompts used). These baselines help us to set up a benchmark and justify the challenges our task poses.

### Experimental Results

Table 3 left half summarizes both the human and model performance on our standard split (30% train, 60% test) of WikiHow annotated-set. Contextualized model obviously outperforms the non-contextualized counterpart greatly, and all learned models perform well-above random baseline. Significant improvements on both pre- and postcondition inferences can be noticed when heuristically constructed data is utilized, especially when no second-stage self-training is involved. The best performance is achieved by **applying all the heuristics** we design, where further improvements are made by augmenting with second-stage pseudo supervisions. Similar performance trends can be observed in Table 3 right half where a zero-shot transfer from models trained on WikiHow data to Instructables is conducted.

Notice that the zero-shot GPT-3 performs quite poorly compared to our _best low-resource training setting_, and generally worse than our zero-shot contextualized model utilizing only the heuristically constructed data. We hypothetically attribute the poor performance to both the requirement of exhaustive search of the conditions across the whole manual, and its lacking of complex commonsense reasoning; justifying the effectiveness of our proposed training paradigm and the difficulty of our task. Nevertheless, there are still **large rooms** for improvement as the best model falls well-behind human performance (>20% F1-score gap).

**Heuristics Ablations.** Table 4 features ablation 

studies on the designed heuristics. One can observe that keywords are mostly effective on inferring the postconditions, and co-references are significantly beneficial in the Instructables data, which can hypothetically be attributed to the writing style of the datasets (_i.e_. authors of Instructables might use co-referred terms more). Temporal relation resolution is consistently helpful across pre- and postconditions as well as datasets, suggesting only relying on narrated orders could degenerate the performance.

#### 6.3.1 Error Analysis.

While our (best) models perform well on linkages that exhibit similar concepts to the designed heuristics and generalize beyond their surface forms, we are interested in investigating under which situations they are more likely to err. We therefore subsample 10% of the annotated test-set for manual qualitative inspections and summarize our observations in Table 6. We find that our models can sometimes **overfit to certain heuristic** concepts as in Table 6 first row (within a food preparation context). Another improvement the models can enjoy is **better causal understanding**, which is currently not explicitly handled by our heuristics and can be an interesting future work (Table 6 second row, in a biking and cleaning contexts).

Humans, on the other hand, exhibit much superior performance than the models, tend to fail more often on two kinds of situations: (1) Missing preconditions (of an action) in those _much earlier paragraphs_, and (2) Sophisticated temporal ordering of the events (often not narrated sequentially in the texts). Especially, the first sentences of each task-step are often regarded as the starting actions,

\begin{table}
\begin{tabular}{c|c|c|c} \hline \hline
**Type** & **Example** & **Description** \\ \hline Heus. & ...use a sharp blade to cut ... & Overfits on entity trace \\ Overfit & ...look for a blade s.t. & heuristic. \\ \hline Lacking Causal & ...body start leaning. & Knowledge-enhanced \\ Reason & ...canal completely dry. & soinha can be helpful. \\ \hline \end{tabular}
\end{table}
Table 6: **Exemplar model errors. The second row are from distant segments not link-able even via the keyword heuristic.**

\begin{table}
\begin{tabular}{l c r|r r r|r r r|r r r|r r r} \hline \hline  & & \multicolumn{4}{c|}{
\begin{tabular}{c} **WikiHow Annotated-Test-Set** & \multicolumn{8}{c}{**Zero-Shot Transfer to Instrucables**} \\ \multirow{-5}{*}{**Heuristics.**} & \multicolumn{4}{c|}{**Precondition**} & \multicolumn{4}{c|}{**Postcondition**} & \multicolumn{4}{c}{**Precondition**} & \multicolumn{4}{c}{**Postcondition**} \\  & Prec. & Recall & F-1 & Prec. & Recall & F-1 & Prec. & Recall & F-1 & Prec. & Recall & F-1 \\ \hline Prob. Random & — & N/N & 3.55 & 4.42 & 3.54 & 0.61 & 0.86 & 0.68 & 2.94 & 3.88 & 3.04 & 0.46 & 0.46 & 0.42 \\ Prompt. GPT-3 & — & N/N & 3.87 & 73.46 & 7.35 & 4.90 & **77.08** & 9.21 & 3.14 & 64.25 & 5.99 & 1.37 & 34.33 & 2.65 \\ Adapt..XPAD & — & Y/N & 6.21 & 58.38 & 10.64 & 9.47 & 13.83 & 10.45 & 5.11 & 57.53 & 8.92 & 7.74 & 9.00 & 7.89 \\ \hline Non-Context. & Y & Y/N & 8.21 & 79.52 & 14.32 & 15.43 & 44.99 & 20.56 & 6.49 & 65.05 & 11.31 & 13.64 & 43.50 & 18.65 \\  & Y & Y/Y & 8.56 & **81.19** & 14.91 & 26.53 & 65.95 & 34.31 & 6.64 & **67.13** & 11.54 & 24.53 & **61.93** & 31.78 \\ \cline{2-13}  & N & Y/N & 34.01 & 58.33 & 39.27 & 34.44 & 44.15 & 36.70 & 26.93 & 53.43 & 32.92 & 32.16 & 41.39 & 34.42 \\  & N & Y/Y & 42.26 & 58.45 & 45.41 & 40.99 & 46.51 & 42.32 & 38.16 & 55.77 & 42.23 & 42.57 & 48.00 & 44.07 \\ \cline{2-13} Context. & Y & N/N & 10.69 & 34.79 & 15.05 & 10.34 & 11.88 & 10.49 & 10.34 & 16.17 & 11.42 & 4.52 & 4.15 & 4.15 \\  & Y & Y/N & 47.92 & 64.63 & 51.38 & **51.25** & 57.45 & 52.90 & 47.05 & 58.97 & 45.17 & 47.92 & 56.51 & 50.06 \\  & Y & Y/Y & **49.42** & 68.40 & **53.51** & **52.39** & 57.35 & **53.42** & **43.81** & 62.71 & **48.34** & **53.41** & 60.51 & **55.17** \\ \hline Human & — & — & — & 83.91 & **83.86** & **83.55** & 77.39 & **84.81** & 78.81 & 84.74 & 81.32 & **82.78** & **71.90** & **82.51** & **75.53** \\ \hline \hline \end{tabular}
\end{table}
Table 3: **Annotated-test-set performance: The best performance is achieved by applying all of the proposed heuristics (heus.) and undergoing the two-stage training: finetuned on the annotated train-set first and then perform the self-training. Note that for the Instructables, both Finetuned and Self are done on the WikiHlow training sets and a zero-shot transfer is performed.**

\begin{table}
\begin{tabular}{c|c c c|c c c|c c c|c c c} \hline \hline \multirow{2}{*}{**Train**} & \multicolumn{4}{c|}{\begin{tabular}{c} **Precondition** \\ Prec. \\ \end{tabular} } & \multicolumn{4}{c|}{\begin{tabular}{c} **Postcondition** \\ Recall \\ \end{tabular} } & \multicolumn{4}{c|}{
\begin{tabular}{c} **Postercondition** \\ \end{tabular} } \\  & Prec. & Recall & F-1 & Prec. & Recall & F-1 \\ \hline
10\% & 41.34 & 61.71 & 46.06 & 45.24 & 55.56 & 47.95 \\
20\% & 45.60 & 67.55 & 50.78 & 49.30 & 58.02 & 51.62 \\
30\% & 57.38 & 64.46 & 57.53 & 50.49 & 54.57 & 51.09 \\
40\% & 49.61 & 73.09 & 55.14 & 50.45 & 57.77 & 52.27 \\
50\% & 54.27 & 70.89 & 57.84 & 51.35 & 55.85 & 52.23 \\
60\% & 53.21 & 69.36 & 56.42 & 53.68 & 58.09 & 54.46 \\ \hline \hline \end{tabular}
\end{table}
Table 5: **Varying annotated-train-set size: On WikiHow (test-set size is fixed at 30%). We use the (best) model trained with all the proposed heuristics and the self-training paradigm.** 

while in reality, they can be postconditions of the followed-up detailed contexts. However, we think both aforementioned errors are rather remediable if the annotators are more careful and search more exhaustively for condition statements.

#### 6.3.2 The Effect of Training Set Size

Table 3 shows that with a little amount of data for training, our models can perform significantly better than the zero-shot setting. This arouses a question - how would the performance change with respect to the training set size, do we just need more data? To quantify the effect of training size on model performance, we conduct an experiment where we vary the sample size in the training set while fixing the development (10%) and test (30%) set for consistency consideration. We use the best settings in Table 3, with all the heuristics and self-training paradigm, for this study. We can observe, from Table 5, a plateau in performance when the training set size is approaching 60%, implying that simply keep adding more training samples does not necessarily yield significant improvements, and hypothesize that the discussed potential improvements are the keys to further effectively exploit the rich knowledge in large-scale instructional data.

## 7 Related Works

**Procedural Text Understanding.** Uncovering knowledge in texts that specifically features _procedural structure_ has drawn many attentions, including aspects of tracking entity state changes Branawan et al. (2012); Bosselut et al. (2018); Mishra et al. (2018); Tandon et al. (2020), incorporating common sense or constraints Tandon et al. (2018); Du et al. (2019), procedure-centric question answering (QA) Tandon et al. (2019), and structural parsing or generations Malmaud et al. (2014); Zellers et al. (2021); Zhou et al. (2023). Clark et al. (2018) leverages VerbNet Schuler (2005) with _if-then_ constructed rules, one of the keywords we also utilize, to determine object-state postconditions for answering state-related reading comprehension questions. In addition, some prior works also specifically formulate precondition understanding as multiple choice QA for event triggers (verbs) Kwon et al. (2020) and common sense phrases Qasemi et al. (2021). We hope our work on inferring action-condition dependencies, an essential knowledge especially for understanding task-procedures, from long instruction texts, can help advancing the goal of more comprehensive procedural text understanding.

Drawing dependencies among procedure steps has been explored in Dalvi et al. (2019); Sakaguchi et al. (2021); Pal et al. (2021), however, their procedures are manually synthesized short paragraphs. Our work, in contrast, aims at inferring diverse dependency knowledge directly from complex real-world and task-solving-oriented instructional manuals, enabling the condition dependencies to go beyond inter-step and narrative boundaries.

**Event Relation Extraction.** Our work is also inspired by document-level event relation extraction Han et al. (2019, 2021); Huang et al. (2021); Ma et al. (2021). Specifically, certain works also adopt weak supervisions to learn event temporal relations Zhou et al. (2020, 2021); Han et al. (2021), while other relevant works aim at extracting causality relations (mainly cause-effect) automatically from texts Cao et al. (2016); Altenberg (1984); Stasaki et al. (2021). Our work combines multiple commonsensical heuristics tailored to the nature of the dependencies exhibited in actions and their conditions, in real-world instruction sources.

## 8 Conclusions

In this work we propose a task on inferring action and (pre/post)condition dependencies on real-world online instructional manuals. We formulate the problem in both zero-shot and low-resource settings, where several heuristics are designed to construct an effective large-scale weakly supervised data. While the proposed heuristics and the two-staged training leads to significant performance improvements, the results still highlight significant gaps below human performance (> 20% F1-score).

We hope our studies and the collected resources can spur relevant research, and suggest two main future directions: (1) End-to-end propose (refined) actionables, conditions, and their dependencies, by fully exploiting our featured span-annotations of the text segments. (2) Inferred world states from the text descriptions as well as external knowledge of the entities and causal common sense can be factored into the heuristics for weak-supervisions 



## 9 Limitations

We hereby discuss the current limitations of our work: **(1)** As mentioned in Section 3.1, although our annotated dataset enables the possibility of learning an extractive model that can be trained to predict the span of the text segments of interest from scratch, we focus on the more essential action-condition dependency linkage inference task as we find that the SRL extraction heuristic currently applied sufficiently reliable. In the future, we look forward to actualizing such an extractive module and other relevant works that can either further refine the SRL-spans or directly propose the text segments we require. More specifically, the extractive module can be supervised and/or evaluated against with our human annotations on the text segment start-end positions of an article. **(2)** The current system is only trained on unimodal (text-only) and English instruction resources. Multilingual and multimodal versions of our work could be as well an interesting future endeavors to make. **(3)** In this work, we mostly consider instructions from physical works. While certain conditions and actions can still be defined within more social domain of data (_e.g._ a precondition to _being a good person_ might be _cultivating good habits_). As a result, we do not really guarantee the performance of our models when applied to data from these less physical-oriented domains.

## 10 Ethics and Broader Impacts

We hereby acknowledge that all of the co-authors of this work are aware of the provided _ACL Code of Ethics_ and honor the code of conduct. This work is mainly about inferring pre- and postconditions of a given action item in an instructional manual. The followings give the aspects of both our ethical considerations and our potential impacts to the community.

**Dataset.** We collect the human annotation of the ground truth condition-action dependencies via Amazon Mechanical Turk (MTurk) and ensure that all the personal information of the workers involved (e.g., usernames, emails, urls, demographic information, etc.) is discarded in our dataset. Although we aim at providing a test set that is agreed upon from various people examining the instructions, there might still be unintended biases within the judgements, we make efforts on reducing these biases by collecting diverse set of instructions in order to arrive at a better general consensus on our task.

This research has been reviewed by the **IRB board** and granted the status of an **IRB exempt**. The detailed annotation process (pay per amount of work, guidelines) is included in the appendix; and overall, we ensure our pay per task is above the the annotator's local minimum wage (approximately $15 USD / Hour). We primarily consider English speaking regions for our annotations as the task requires certain level of English proficiency.

**Techniques.** We benchmark the proposed condition-inferring task with the state-of-the-art large-scale pretrained language models and our proposed training paradigms. As commonsense and task procedure understanding are of our main focus, we do not anticipate production of harmful outputs, especially towards vulnerable populations, after training (and evaluating) models on our proposed task.

## Acknowledgments

Many thanks to Rujun Han for his implementation on the temporal relation resolution model. This material is based on research supported by the Machine Common Sense (MCS) program under Cooperative Agreement N66001-19-2-4032 with the US Defense Advanced Research Projects Agency (DARPA). The views and conclusions contained herein are those of the authors and should not be interpreted as necessarily representing DARPA, or the U.S. Government.

## References

* Aeronautiques et al. (1998) Constructions Aeronautiques, Adele Howe, Craig Knoblock, ISI Drew McDermott, Ashwin Ram, Manuela Veloso, Daniel Weld, David Wilkins SRI, Anthony Barrett, Dave Christianson, et al. 1998. Pddll the planning domain definition language. _Technical Report, Tech. Rep._.
* Altenberg (1984) Bengt Altenberg. 1984. Causal linking in spoken and written english. _Studia linguistica_, 38(1):20-69.
* Bosselut et al. (2018) Antoine Bosselut, Omer Levy, Ari Holtzman, Corin Ennis, Dieter Fox, and Yejin Choi. 2018. Simulating action dynamics with neural process networks. In _International Conference on Learning Representations (ICLR)_.
* Branavan et al. (2012a) SRK Branavan, Nate Kushman, Tao Lei, and Regina Barzilay. 2012a. Learning high-level planning from text. In _Association for Computational Linguistics (ACL)_

S.R.K. Branavan, Nate Kushman, Tao Lei, and Regina Barzilay (2012)Learning high-level planning from text. In Association for Computational Linguistics (ACL), Cited by: SS4.
* T. Brown, B. Mann, N. Ryder, M. Subbiah, J. D. Kaplan, P. Dhariwal, A. Neelakantan, P. Shyam, G. Sastry, A. Askell, et al. (2020)Language models are few-shot learners. In Neural Information Processing Systems (NeurIPS), Vol. 33, pp. 1877-1901. External Links: Link Cited by: SS5.
* M. Cao, X. Sun, and H. Zhuge (2016)The role of cause-effect link within scientific paper. In 2016 12th International Conference on Semantics, Knowledge and Grids (SKG), pp. 32-39. Cited by: SS1.
* P. Clark, B. Dalvi, and N. Tandon (2018)What happened? leveraging verbnet to predict the effects of actions in procedural text. arXiv preprint arXiv:1804.05435. Cited by: SS1.
* B. Dalvi, N. Tandon, C. Bosseltu, W. Yih, and P. Clark (2019)Everything happens for a reason: discovering the purpose of actions in procedural text. In Empirical Methods in Natural Language Processing (EMNLP), pp. 4496-4505. Cited by: SS1.
* J. Devlin, M. Chang, K. Lee, and K. Toutanova (2019)BERT: pre-training of deep bidirectional transformers for language understanding. In North American Chapter of the Association for Computational Linguistics (NAACL-HLT), pp. 4171-4186. Cited by: SS1.
* J. Du, E. Grave, B. Gunel, V. Chaudhary, O. Celebi, M. Auli, V. Stoyanov, and A. Conneau (2021)Self-training improves retraining for natural language understanding. In North American Chapter of the Association for Computational Linguistics (NAACL-HLT), Cited by: SS1.
* X. Du, B. Dalvi Mishra, N. Tandon, C. Bosseltut, W. Yih, P. Clark, and C. Cardie (2019)Be consistent! improving procedural text comprehension using label consistency. In North American Chapter of the Association for Computational Linguistics (NAACL-HLT), Cited by: SS1.
* R. E. Fikes and N. J. Nilsson (1971)Strips: a new approach to the application of theorem proving to problem solving. In Artificial intelligence, Vol. 2, pp. 189-208. Cited by: SS1.
* M. Gardner, J. Grus, M. Neumann, O. Tafjord, P. Dasigi, N. F. Liu, M. Peters, M. Schmitz, and L. S. Zettlemoyer (2017)Allennlp: a deep semantic natural language processing platform. Cited by: SS1.
* C. Hadley, K. Uyemura, K. Hall, K. Jan, S. Volavong, and N. Harrington (2021a)Wikiow. Cited by: SS1.
* R. Han, I. Hsu, J. Sun, J. Baylon, Q. Ning, D. Roth, and N. Peng (2021a)Ester: a machine reading comprehension dataset for event semantic relation reasoning. In The 2021 Conference on Empirical Methods in Natural Language Processing (EMNLP), Cited by: SS1.
* R. Han, Q. Ning, and N. Peng (2019)Joint event and temporal relation extraction with shared representations and structured prediction. In 2019 Conference on Empirical Methods in Natural Language Processing (EMNLP), Cited by: SS1.
* R. Han, X. Ren, and N. Peng (2021b)EConet: effective continual pretraining of language models for event temporal reasoning. In Empirical Methods in Natural Language Processing (EMNLP), Cited by: SS1.
* K. Huang, S. Tang, and N. Peng (2021)Document-level entity-based extraction as template generation. In The 2021 Conference on Empirical Methods in Natural Language Processing (EMNLP), Cited by: SS1.
* I. Lin, J. Li, and K. Toutanova (2021)AVAX: an open source database for event semantic relations. In Proceedings of the IEEE/CVF Workshop on Semantic Parsing, pp. 1-1. Cited by: SS1.
* D. P. Kingma and J. Ba (2015)Adam: a method for stochastic optimization. In International Conference on Learning Representations (ICLR), Cited by: SS1.
* H. Kwon, M. Koupaee, P. Singh, G. Sawhney, A. Shukla, K. Kumar Kallur, N. Chambers, and N. Balasubramanian (2020)Modeling preconditions in text with a crowd-sourced dataset. In Empirical Methods in Natural Language Processing (EMNLP), Cited by: SS1.
* K. Lee, L. He, and L. Zettlemoyer (2018)Higher-order coreference resolution with coarse-to-fine inference. In North American Chapter of the Association for Computational Linguistics (NAACL-HLT), Cited by: SS1.
* K. Vander Linden (1994)Generating precondition expressions in instructional text. In Association for Computational Linguistics (ACL), Cited by: SS1.
* Y. Liu, M. Ott, N. Goyal, J. Du, M. Joshi, D. Chen, O. Levy, M. Lewis, L. Zettlemoyer, and V. Stoyanov (2019)Roberta: a robustly optimized bert pretraining approach. arXiv preprint arXiv:1907.11692. Cited by: SS1.
* M. Derek Ma, J. Sun, M. Yang, K. Huang, N. Wen, S. Singh, R. Han, and N. Peng (2021)Eventplus: a temporal event understanding pipeline. In 2021 Annual Conference of the North American Chapter of the Association for Computational Linguistics (NAACL), Demonstrations Track, Cited by: SS1.
* J. Malmaud, E. Wagner, N. Chang, and K. Murphy (2014)Cooking with semantics. In Proceedings of the ACL 2014 Workshop on Semantic Parsing, pp. 33-38.

 

Bhavana Dalvi Mishra, Lifu Huang, Niket Tandon, Wen-tau Yih, and Peter Clark. 2018. Tracking state changes in procedural text: a challenge dataset and models for process paragraph comprehension. In _North American Chapter of the Association for Computational Linguistics (NAACL-HLT)_.
* Ma et al. [2018] Aldrian Obaja Muis Naoki Otani Nidhi, Vyas Ruochen Xu, and Yiming Yang Teruko Mitamura Eduard Hovy. 2018. Low-resource cross-lingual event type detection in documents via distant supervision with minimal effort. In _International Conference on Computational Linguistics (COLING)_.
* Ning et al. [2018] Qiang Ning, Hao Wu, and Dan Roth. 2018. A multi-axis annotation scheme for event temporal relations. In _Association for Computational Linguistics (ACL)_.
* Pal et al. [2021] Kuntal Kumar Pal, Kazuaki Kashihara, Pratyay Banerjee, Swaroop Mishra, Ruoyu Wang, and Chitta Baral. 2021. Constructing flow graphs from procedural cybersecurity texts. In _Findings of the Association for Computational Linguistics: ACL-IJCNLP 2021_.
* Plank and Agic [2018] Barbara Plank and Zeljko Agic. 2018. Distant supervision from disparate sources for low-resource part-of-speech tagging. In _Empirical Methods in Natural Language Processing (EMNLP)_.
* Qasemi et al. [2021] Ehsan Qasemi, Filip Ilievski, Muhao Chen, and Pedro Szekely. 2021. Corequisite: Circumstantial preconditions of common sense knowledge. In _West Coast NLP Summit (WeCNLP)_.
* Sakaguchi et al. [2021] Keisuke Sakaguchi, Chandra Bhagavatula, Ronan Le Bras, Niket Tandon, Peter Clark, and Yejin Choi. 2021. proScript: Partially ordered scripts generation. In _Findings of the Association for Computational Linguistics: EMNLP 2021_.
* Schuler [2005] Karin Kipper Schuler. 2005. _VerbNet: A broad-coverage, comprehensive verb lexicon_. University of Pennsylvania.
* Sharma and Kroemer [2020] Mohit Sharma and Oliver Kroemer. 2020. Relational learning for skill preconditions. In _Conference on Robot Learning (CoRL)_.
* Shi and Lin [2019] Peng Shi and Jimmy Lin. 2019. Simple bert models for relation extraction and semantic role labeling. _ArXiv_abs/1904.05255.
* Stasaki et al. [2021]Katherine Stasaki, Manav Rathod, Tony Tu, Yunfang Xiao, and Marti A Hearst. 2021. Automatically generating cause-and-effect questions from passages. In _Proceedings of the 16th Workshop on Innovative Use of NLP for Building Educational Applications_, pages 158-170.
* Tandon et al. [2018] Niket Tandon, Bhavana Dalvi Mishra, Joel Grus, Wentau Yih, Antoine Bosseltut, and Peter Clark. 2018. Reasoning about actions and state changes by injecting commonsense knowledge. In _Empirical Methods in Natural Language Processing (EMNLP)_.
* Tandon et al. [2019] Niket Tandon, Bhavana Dalvi Mishra, Keisuke Sakaguchi, Antoine Bosseltu, and Peter Clark. 2019. Wiqa: A dataset for" what if..." reasoning over procedural text. In _Empirical Methods in Natural Language Processing (EMNLP)_.
* Tandon et al. [2020] Niket Tandon, Keisuke Sakaguchi, Bhavana Dalvi, Dheeraj Rajagopal, Peter Clark, Michal Guerquin, Kyle Richardson, and Eduard Hovy. 2020. A dataset for tracking entities in open domain procedural text. In _Empirical Methods in Natural Language Processing (EMNLP)_, pages 6408-6417.
* Wolf et al. [2020] Thomas Wolf, Lysandre Debut, Victor Sanh, Julien Chaumond, Clement Delangue, Anthony Moi, Pierric Cistac, Tim Rault, Remi Lout, Morgan Funtowicz, Joe Davison, Sam Shleifer, Patrick von Platten, Clara Ma, Yacine Jernie, Julien Plu, Canwen Xu, Teven Le Scao, Sylvian Gugger, Mariama Drame, Quentin Lhoest, and Alexander M. Rush. 2020. Transformers: State-of-the-art natural language processing. In _Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing: System Demonstrations_, pages 38-45, Online. Association for Computational Linguistics.
* Wu et al. [2022] Te-Lin Wu, Alex Spangher, Pegah Alipoormolabashi, Marjorie Freedman, Ralph Weischedel, and Nanyun Peng. 2022. Understanding multimodal procedural knowledge by sequencing multimodal instructional manuals. In _Association for Computational Linguistics (ACL)_.
* Xie et al. [2020] Qizhe Xie, Minh-Thang Luong, Eduard Hovy, and Quoc V Le. 2020. Self-training with noisy student improves imagenet classification. In _Proceedings of the IEEE/CVF conference on computer vision and pattern recognition_, pages 10687-10698.
* Zellers et al. [2021] Rowan Zellers, Ari Holtzman, Matthew Peters, Roozbeh Mottaghi, Aniruddha Kembhavi, Ali Farhadi, and Yejin Choi. 2021. Piglet: Language grounding through neuro-symbolic interaction in a 3d world. In _Association for Computational Linguistics (ACL)_.
* Zhang et al. [2020] Li Zhang, Qing Lyu, and Chris Callison-Burch. 2020. Reasoning about goals, steps, and temporal ordering with WikiHHow. In _Empirical Methods in Natural Language Processing (EMNLP)_, pages 4630-4639.
* Zhou et al. [2020] Ben Zhou, Qiang Ning, Daniel Khashabi, and Dan Roth. 2020. Temporal common sense acquisition with minimal supervision. In _Association for Computational Linguistics (ACL)_.
* Zhou et al. [2021] Ben Zhou, Kyle Richardson, Qiang Ning, Tshar Khot, Ashish Sabharwal, and Dan Roth. 2021. Temporal reasoning on implicit events from distant supervision. In _North American Chapter of the Association for Computational Linguistics (NAACL-HLT)_.
* Zhou et al. [2019] Yilun Zhou, Julie Shah, and Steven Schockaert. 2019. Learning household task knowledge from WikiHow descriptions. In _Proceedings of the 5th Workshop 

on Semantic Deep Learning (SemDeep-5)_, pages 50-56, Macau, China. Association for Computational Linguistics.
* Zhou et al. [2023] Yu Zhou, Sha Li, Manling Li, Xudong Lin, Shih-Fu Chang, Mohit Bansal, and Heng Ji. 2023. Nonsequential graph script induction via multimedia grounding. In _Association for Computational Linguistics (ACL)_.

 

Details of The Datasets

Resource-wise our work utilizes online instructional manuals (_e.g_. WikiHow) following many existing works Zhou et al. (2019); Zhang et al. (2020); Wu et al. (2022), specifically, the large-scale WikiHow training data is provided by Wu et al. (2022), while we scrape the Instructables.com data on our own. Since Instructables.com dataset tend to have noisier and more free-formed texts, we thus manually sub-sample a smaller (as compared to the test-set of WikiHow) high quality subset.

We report the essential statistics of the annotated-sets in Table 7. Although our definition of actionable is **any** textual phrase that can be actually **acted** in the real world, every unique phrase in our dataset is basically a distinct actionable. We compute the number of distinct actions by extracting the main verb-noun phrases (with lemmatization applied) in a text segment as a _valid-action_, and report their counts in Table 7 as well. Each unique action in this way can lead to roughly only 1-to-3 pairwise relation instance in our annotated dataset. Both this and the aforementioned unique action count justifies the diversity of our collected annotated-set.

Each unique URL of WikiHow can have different multi-step _sections_, and we denote each unique section as a _unique article_ in our dataset; while for Instructables.com, each URL only maps to a single section. As a result, for WikiHow we firstly manually select a set of URLs that are judged featuring high quality (_i.e_. articles consisting clear instructed actions, and contain not so much non-meaningful or unhelpful monologues from the writer) instructions and then sample one or two sections from each of the URLs to construct our annotated-set. The statistics of the datasets used to construct the large-scale weakly supervised WikiHow training set can be found in Section 3 of Wu et al. (2022), where we use their provided WikiHow training samples that are mostly from physical categories.

\({}^{*}\)Our densely annotated datasets and relevant tools will be made public upon paper acceptance.

### Dataset Splits

The whole annotated Instructables.com data samples are used as an evaluating set so we do not need to explicitly split them. For WikiHow, we split mainly with respect to the URLs to ensure that no articles (_i.e_. sections) from the same URL are put into different data splits, so as to prevent model exploiting the writing style and knowledge from the same URL of articles on WikiHow. The splitting on the URL-level is as well a random split.

## Appendix B Details of Human Annotations

### Inter-Annotator Agreements (IAAs)

There are two types of inter-annotator agreements (IAAs) we compute: (1) **ICAA on text segments** and (2) ** IAAa on linkages**, and we describe the details of their computations in this section.

**IAA on Text Segments.** For each worker-highlighted text segment, either coming from directly clicking the pre-highlighted segments or their own creations, we compute the percentage of the overlapping of the tokens between segments annotated by different workers. If this percentage is > 60% of each segment in comparison, we denote these two segments are _aligned_. Concretely, for all the unique segments of the same article, annotated by different workers, we can postulate a segment dictionary where the _aligned_ segments from different worker annotations are combined into the same ones. And hence each worker's annotation can be viewed as a binary existence of each of the

\begin{table}
\begin{tabular}{l r r r} \hline \multicolumn{4}{c}{**Type**} & \multicolumn{2}{c}{**Counts**} \\ \hline Total Unique Articles & 500 & \\ Total Unique URLs & 326 & \\ Anonet.Train/ Amonet.Test & 200 / 300 & \\ Type-Token Ratio & 9799 / 173920 = 0.06 & \\ Pre-Postcondition Ratio & 16457 / 8239 = 5.80 & \\ Distinct Actions & 5205 & \\ Avg. Instance per Unique Action & 3.33 & \\ Avg. Possible Text Segment Pairs & 717.49 & \\ \hline \end{tabular} 
\begin{tabular}{l c c c} \hline \multicolumn{4}{c}{**Type**} & \multicolumn{2}{c}{**Mean**} & \multicolumn{2}{c}{**Std**} & \multicolumn{2}{c}{**Min**} & \multicolumn{1}{r}{**Max**} \\ \hline Tokens in a Step Text & 67.67 & 23.77 & 2 & 161 \\ Sentences in a Step Text & 4.20 & 1.00 & 1 & 6 \\ Tokens in an article & 319.12 & 91.71 & 96 & 631 \\ Sentences in an article & 19.81 & 4.03 & 11 & 28 \\ \hline \end{tabular}
\end{table}
Table 7: **General statistics of the two annotated-sets**: We provide the detailed component counts of the annotated-sets used in this work, including the statistics of tokens and sentences from the instruction steps (lower halves).

 

items in such a segment dictionary, where we can compute the Cohen's Kappa inter-annotator agreement scores on every pair of annotators to derive the averaged IAA scores.

**IAA on Linkages.** Similar to the construction of a segment dictionary, we also construct a _linkage dictionary_ where every link has a _head segment_ pointing to the _tail segment_, with both of the segments coming from an item in the segment dictionary. We thus can also treat the annotation of the linkages across different worker annotations as a binary existence and perform similar inter-annotator agreement computations.

The resulting IAAs for each dataset and annotation types are reported in Section 3.1.

**Majority Vote.** To obtain the final multi-annotator-judged refined data, with our collection budget allowance, we ensure that the number of annotators per data instance (instruction article) is at least 2 (mostly 3), where _consensus_ (strict agreement) is used for instances with 2 annotators, and _majority vote_ is adopted for 3 annotators.

### Annotation Process

We adopt Amazon Mechanical Turk (MTurk) to publish and collect our annotations, where each of the annotation in the MTurk is called a Human Intelligence Task (HIT). As shown in Figure 4a, on the top of each HIT we have a detailed description of the task's introduction, terminologies, and instructions. For the terms we define, such as actionables and pre-/postconditions, we also illustrate them with detailed examples. To make it easier for workers to quickly understand our tasks, we provide a video version explaining important concepts and the basic operations. We also set up a Frequently Asked Question (FAQ) section and constantly update such section with some questions gathered from the workers.

Figure 4b shows the layout of the annotation panel. A few statements are pre-highlighted in grey and each of them is clickable. These statements are automatically pre-selected using the SRL heuristics described in Section 3.1, which are supposed to cover as much potential actionables and pre-/postconditions as possible. Workers can either simply click the pre-highlighted statements or _redo_ the selection to get their more desired segments. The clicked or selected statements will pop up to the right panel as the text-blocks. For the convenience to manage the page layout, each text-block is _dragable_ and can be moved anywhere within the panel. The workers then should examine with their intelligence and common sense to connect text-blocks (two at a time) by right clicking one of them to _start_ a directed linkage (which ends at another text-block) and choose a proper dependency label for that particular drawn linkage.

Since our annotation task can be rather complicated, we would like our workers to fully understand the requirements before proceeding to the actual annotation. All annotators are expected to pass three qualification rounds, each consisting of 5 HITs, before being selected as an official annotator. 15 HITs are annotated internally in advance as the standard answers to be used to judge the qualification round qualities.

We calculate the IAAs of each annotator against our standard answers to measure their performance in our task. In each round, only the best performers move on to the next. At the end of each round, we email annotators to explain the questions they asked or some of the more commonly made mistakes shared across multiple workers. In total, over 60 workers participated in our task, and 10 of them passed the qualification rounds.

We estimate the time required to complete each of our HITs to be 10-15 minutes, and adjust our pay rate to $2.5 and $3 USD for the qualification and the actual production rounds, respectively. This roughly equates to a $15 to $18 USD per hour wage, which is above the local minimum wage for the workers. We also ensure that each of our data samples in the official rounds is annotated by at least two different _good workers_.

**Confidence Levels.** We compute the averaged percentage of confidence levels reported by the workers in Table 8. Note that majority of the workers indicate a _moderately_ or _fairly_ confidence levels, implying they are sufficiently confident about their

\begin{table}
\begin{tabular}{l|r r} \hline \hline
**Conflidence Level** & **WikiHow** & **Instructables.com** \\ \hline
5 (Very) & 27.27 & 16.33 \\
4 (Fairly) & 27.11 & 23.47 \\
3 (Moderately) & 28.25 & 22.95 \\
2 (Somewhat) & 16.23 & 29.10 \\
1 (Not-At-All) & 1.14 & 8.16 \\ \hline \hline \end{tabular}
\end{table}
Table 8: **Confidence-Level Statistics (%):** In WikiHow, majority (> 80%) of the annotators indicate at least ¿ 3 (Moderatetely) confidence level. As for Instructables.com, it has lower confidence level as the articles tend to be more freeformed and noisy, however, there are still more than 60% of the time workers report confidence levels at least moderately.

 

annotations. We also see feedback from workers that some of them rarely use strong words such as _very_ to indicate their confidence levels, and hence the resulted statistics of their confidences could be a bit biased towards the medium.

**Human Performance.** We randomly select 100 samples from the WikiHow annotated-test-set and 50 samples from the Instructables.com annotated-test-set for computing the human performance. The allowed inputs are exactly the same as what models take, _i.e_. given all the instruction paragraph as context and highlighted (postulated text segment boxes) text segments of interests, workers are asked to predict the relations among such segments so as to induce a complete dependency graph. For each sample, we collect inputs from two different workers, and ensure that the workers are not the ones that give the original annotations of the action-condition dependencies. The human performance is then computed by taking the averaged metrics similar to the models on the given samples.

## Appendix C Modelling Details

### More on Heuristics

#### c.1.1 SRL Extraction

As SRL can detect multiple plausible ways to form the ARG frames to the same _central_ verb, we need to determine which one is the most likely to be desirable. When such multiple argument patterns exist for the same central verb, we simply determine the most desirable formation of segments by maximizing both the number of plausible segments (where they do not overlap above certain threshold, which is set to be 60% in this work) _within a sentence_ and the number of ARGs in each segment.

#### c.1.2 Linking Algorithm

In Section 4.2 we mention that a maximum distance of \(2\) steps between linked segments is imposed to filter out possible non-dependent conditions. While this still can potentially include many not-so-much depended text segments, our goal is to exploit the generalization ability of large-scale pretrained language models to _recognize_ segments that are most probable conditions by including as much as heuristically proposed linkages as possible, which is empirically proven effective. A better strategy on making such a design choice of maximum allowed step-wise distance is left as a future work.

#### c.1.3 Keywords

About 3% of the entire un-annotated data have sentences containing the keywords we use in this work (Table 2). Despite the relatively small amount compared to other heuristics, they are quite effective judging from the results reported in Table 3.

#### c.1.4 Key Entity Tracing

For the key entity tracing heuristic described in Section 4.1.2, as long as two segments share at least one mentioned entity, they can be linked (_i.e_. _traced_ by the shared entity). We do not constraint the number of key entities within a segment, so there can be more than one being used to conduct the tracing.

**Constructing Entity Prediction Datasets.** As mentioned in Section 4.1.2, one way to postulate the key entities is via constructing a predictive model for outputting potentially involved entities. To do so, we firstly construct an _entity vocabulary_ by extracting all the noun phrases within each SRL extracted segments of the entire un-annotated-set articles. To prevent from obtaining a too much large vocabulary as well as improbable entities, we only retain entities (without lemmatization) that appear with > \(5\) occurrences in at least one article.

We then train a language model (based on RoBERTa-large as well) where the output is the multi-label multi-class classification results on the predicted entities. When predicting the key entities for a given segment, we further constraint the predictions to be within the local vocabulary (more than \(5\) occurrences) within the article such segment belongs to. This model is inspired by the entity selector module proposed in Bosselut et al. (2018) while we only consider single step statements. We verify the performance of the learned model on the dataset provided by Bosselut et al. (2018) (the entity selection task), where our model can achieve roughly 60% on F-1 metric, indicating the trained model is sufficiently reliable.

#### c.1.5 Temporal Relations

We use the temporal relation resolution model from Han et al. (2021) that is trained on various temporal relation datasets such as _MATRES_Ning et al. (2018). We train the model on three different random seeds and make them produce a _consensus_ prediction, _i.e_. unless all of the models jointly predict a specific relation (BEFORE or AFTER), otherwise the relation will be regarded as VAGUE 



### GPT-3 Baseline

We use the most powerful version of GPT-3 (Davinci)9 provided by the OpenAI GPT-3 API (zero-shot prompted version) with the following prompt:

Footnote 9: [https://openai.com/api/pricing/](https://openai.com/api/pricing/)

_Extract the preconditions and postconditions from this text:_

_Text: "Slice 500 grams of onion. Heat the pan with olive oil. Wait until the oil is sizzling. Place onions in the frying pan. Stir the onions. In a few minutes, they should be caramelized."_

_Segment 1: "Heat the pan with olive oil."_

_Segment 2: "oil is sizzling."_

_Label: post-condition_

_Text: "Slice 500 grams of onion. Heat the pan with olive oil. Wait until the oil is sizzling. Place onions in the frying pan. Stir the onions. In a few minutes, they should be caramelized."_

_Segment 1: "Slice 500 grams of onion."_

_Segment 2: "Place the onions in the frying pan."_

_Label: pre-condition_

_Text: "Slice 500 grams of onion. Heat the pan with olive oil. Wait until the oil is sizzling. Place onions in the frying pan. Stir the onions. In a few minutes, they should be caramelized."_

_Segment 1: "Slice 500 grams of onion."_

_Segment 2: "Heat the pan with olive oil."_

_Label: no relation_

_Text: "Fill-In an Article"_

_Segment 1: "Fill-In Text Segment 1"_

_Segment 2: "Fill-In Text Segment 2"_

_Label:_ GPT-3 Prediction

In other words, we provide an exemplar simplified instance to instruct what pre- and postconditions should be like to the model with the article context and a pair of text segments of interest. And then, the GPT-3 model should _generate_ the text description-based prediction label (non-case-sensitive). For preconditions we allow verbalized label to be within {_precondition, pre-condition_}, and postconditions within {_postcondition, postcondition_}. For the NULL relation, we allow {_no relation, unrelated, null, none_}.

### Development Set Performance

We select the model checkpoints to be evaluated using the held-out development split (annotated-dev-set). We also report the performance on this annotated-dev-set in Table 9.

### More Results on Train-Set Size Varying

Table 10 is a similar experiment as Table 5 but here we conduct the experiments with the models that do not utilize the weakly supervised data constructed with the proposed heuristics at all. One can observe that similar trends hold that a plateau can be noticed when the training set size is approaching 60%. Compared to Table 5, we can also observe that the smaller the train-set size is, the larger gaps shown between the models with and without utilizing the heuristically constructed data. This can further imply the effectiveness of our heuristics to construct meaningful data for the action-condition dependency inferring task. The models with heuristics, if compared at the same train-set size respectively, significantly outperforms every model counterparts that do not utilize the heuristics.

Table 11 reports similar experiments but in the Instructables.com annotated-test-set. Note that we perform a direct zero-shot transfer from the WikiHow annotated-train-set, so the test-set size is always 100% for the Instructables.

Finally, both Tables 12 and 13 report the same experiments, however, this time the second-stage self-training is not applied. It is worth noting that the self-training is indeed effective throughout all the train-set-size and across different datasets and model variants, however, the trends of model performance hitting a saturation point when the train-set size increases still hold.

### Training & Implementation Details

**Training Details.** The maximum of 500 token length described in Section 6.1 is sufficient for most of the data in the annotated-test-sets, as evident in Table 7. All the models in this work are trained on a single Nvidia A100 GPU10 on a Ubuntu 20.04.2 operating system. The hyperparameters for each model are manually tuned against different datasets, and the checkpoints used for testing are selected by the best performing ones on the held-out development sets in their respective datasets.

Footnote 10: [https://www.nvidia.com/en-us/data-center/a100/](https://www.nvidia.com/en-us/data-center/a100/)

**Implementation Details.** The implementations of the transformer-based models are extended from the HuggingFace11 code base Wolf et al. (2020), and our entire code-base is implemented in PyTorch.12



### Hyperparameters

We train our models until performance convergence is observed on the heuristically constructed dataset. The training time for the weakly supervised learning is roughly 6-8 hours. For all the finetuning that involves our annotated-sets, we train the models for roughly 10-15 epochs for all the model variants, where the training time varies from 1-2 hours. We list all the hyperparameters used in Table 14. The basic hyperparameters such as learning rate, batch size, and gradient accumulation steps are kept consistent for all kinds of training in this work, including training on the weakly supervised data, finetuning on the annotated-sets, as well as during the second-stage self-training. All of our models adopt the same search bounds and ranges of trials as in Table 15.

\begin{table}
\begin{tabular}{c|r r r|r r r} \hline \hline \multirow{3}{*}{**Train**} & \multicolumn{3}{c|}{**Precondition**} & \multicolumn{3}{c}{**Postcondition**} \\  & Prec. & Recall & F-1 & Prec. & Recall & F-1 \\ \hline
10\% & 33.44 & 56.41 & 38.69 & 42.37 & 53.86 & 45.25 \\
20\% & 35.05 & 60.97 & 40.86 & 40.76 & 51.35 & 43.19 \\
30\% & 44.57 & 60.19 & 47.68 & 43.00 & 47.26 & 43.83 \\
40\% & 39.38 & 72.23 & 46.63 & 45.51 & 54.27 & 47.57 \\
50\% & 40.97 & 69.70 & 47.24 & 49.15 & 59.04 & 51.76 \\
60\% & 46.99 & 71.14 & 52.27 & 48.80 & 56.51 & 50.74 \\ \hline \hline \end{tabular}
\end{table}
Table 10: **Varying annotated-train-set size without weakly supervised training: on WikiHow (test-set size size is fixed at 30%). The model used in this experiment is without training on any of the heuristically constructed dataset, but we apply the self-training paradigm.**

\begin{table}
\begin{tabular}{c|r r r|r r r} \hline \hline \multirow{3}{*}{**Train**} & \multicolumn{3}{c|}{**Precondition**} & \multicolumn{3}{c}{**Postcondition**} \\  & Prec. & Recall & F-1 & Prec. & Recall & F-1 \\ \hline
10\% & 32.25 & 50.50 & 36.36 & 41.37 & 51.37 & 44.03 \\
20\% & 35.95 & 56.99 & 40.89 & 48.77 & 60.10 & 51.86 \\
40\% & 39.62 & 64.19 & 45.77 & 48.83 & 60.30 & 52.08 \\
50\% & 57.38 & 64.46 & 57.53 & 50.49 & 54.57 & 51.09 \\
60\% & 45.62 & 61.02 & 49.06 & 55.00 & 65.04 & 57.54 \\ \hline
10\% & 27.50 & 50.32 & 32.74 & 34.99 & 47.66 & 38.18 \\
20\% & 26.86 & 51.73 & 32.34 & 40.31 & 52.89 & 43.43 \\
40\% & 30.58 & 64.38 & 38.16 & 44.78 & 60.86 & 49.28 \\
50\% & 39.65 & 63.28 & 45.41 & 50.96 & 59.98 & 53.54 \\
60\% & 39.90 & 65.68 & 45.95 & 49.64 & 58.83 & 51.97 \\ \hline \hline \end{tabular}
\end{table}
Table 11: **Varying annotated-train-set size: on Instruatches.com (test-set size is fixed at 100%). Note that here the train-set size is from WikiHow annotated-set, and the 30% is basically Table 3. The upper half is with models that utilize both the heuristically constructed dataset and the self-training paradigm, while the lower half is with models that do not use any weak supervisions.**

\begin{table}
\begin{tabular}{c c c c|c c c c} \hline \hline \multirow{3}{*}{**Train**} & \multicolumn{3}{c|}{**Precondition**} & \multicolumn{3}{c}{**Postcondition**} \\  & Prec. & Recall & F-1 & Prec. & Recall & F-1 \\ \hline
10\% & 33.45 & 56.41 & 38.69 & 42.37 & 53.86 & 45.25 \\
20\% & 35.05 & 60.97 & 40.86 & 40.76 & 51.35 & 43.19 \\
30\% & 44.57 & 60.19 & 47.68 & 43.00 & 47.26 & 43.83 \\
40\% & 39.38 & 72.23 & 46.63 & 45.51 & 54.27 & 47.57 \\
50\% & 40.97 & 69.70 & 47.24 & 49.15 & 59.04 & 51.76 \\
60\% & 46.99 & 71.14 & 52.27 & 48.80 & 56.51 & 50.74 \\ \hline \hline \end{tabular}
\end{table}
Table 9: **Annotated-dev-set performance on WikiHow: Similar to Table 3, we report the development set performance on the WikiHow dataset (Instructables.com does not have the development set as we are conducting a zero-shot transfer).**



\begin{table}
\begin{tabular}{c c c c c c} \hline \hline
**Models** & **Batch Size** & **Initial LR** & _\# Training Epochs_ & \multicolumn{2}{c}{**Gradient Accumulation Steps**} \\ \hline Non-contextualized & 8 & \(1\times 10^{-5}\) & 15 & 1 & 355M \\ Contextualized & 4 & \(1\times 10^{-5}\) & 15 & 1 & 372M \\ \hline \hline \end{tabular}
\end{table}
Table 14: **Hyperparameters in this work:**_Initial LR_ denotes the initial learning rate. All the models are trained with Adam optimizers [69]. We include number of learnable parameters of each model in the column of _# params_.

Figure 4: **MTurk Annotation User Interface:****(a)** We ask workers to follow the indicated instruction. All the blue-colored text bars on the top of the page are expandable. Workers can click to expand them for detailed instructions of the annotation task. **(b)** The annotation task is designed for an intuitive _click/select-then-link_ usage, followed by a few additional questions such as confidence level and feedback (this example is obtained from WikiHow dataset). The grey-color-highlighted text segments are postulated by the SRL, where the color of a segment will turn yellow if either being selected or cursor highlighted. Notice that for better illustration, the directions of the links in our paper are opposite to those in the annotation process.





**ACL 2023 Responsible NLP Checklist+**

Footnote †: The Responsible NLP Checklist used at ACL 2023 is adopted from NAACL 2022, with the addition of a question on AI writing assistance.

## Appendix A For every submission:

* A1. Did you describe the limitations of your work? _Left blank._
* A2. Did you discuss any potential risks of your work? _Left blank._
* A3. Do the abstract and introduction summarize the paper's main claims? _Left blank._
* A4. Have you used AI writing assistants when working on this paper? _Left blank._
* B\(\square\)****Did you use or create scientific artifacts?_

_Left blank._

* B1. Did you cite the creators of artifacts you used? _Left blank._
* B2. Did you discuss the license or terms for use and / or distribution of any artifacts? _Left blank._
* B3. Did you discuss if your use of existing artifact(s) was consistent with their intended use, provided that it was specified? For the artifacts you create, do you specify intended use and whether that is compatible with the original access conditions (in particular, derivatives of data accessed for research purposes should not be used outside of research contexts)? _Left blank._
* B4. Did you discuss the steps taken to check whether the data that was collected / used contains any information that names or uniquely identifies individual people or offensive content, and the steps taken to protect / anonymize it? _Left blank._
* B5. Did you provide documentation of the artifacts, e.g., coverage of domains, languages, and linguistic phenomena, demographic groups represented, etc.? _Left blank._
* B6. Did you report relevant statistics like the number of examples, details of train / test / dev splits, etc. for the data that you used / created? Even for commonly-used benchmark datasets, include the number of examples in train / validation / test splits, as these provide necessary context for a reader to understand experimental results. For example, small differences in accuracy on large test sets may be significant, while on small test sets they may not be. _Left blank._
* C\(\square\)****Did you run computational experiments?_

_Left blank._

* C1. Did you report the number of parameters in the models used, the total computational budget (e.g., GPU hours), and computing infrastructure used? _Left blank._

* C2. Did you discuss the experimental setup, including hyperparameter search and best-found hyperparameter values? _Left blank._
* C3. Did you report descriptive statistics about your results (e.g., error bars around results, summary statistics from sets of experiments), and is it transparent whether you are reporting the max, mean, etc. or just a single run? _Left blank._
* C4. If you used existing packages (e.g., for preprocessing, for normalization, or for evaluation), did you report the implementation, model, and parameter settings used (e.g., NLTK, Spacy, ROUGE, etc.)? _Left blank._
* D\(\square\): **Did you use human annotators (e.g., crowdworkers) or research with human participants?** _Left blank._
* D1. Did you report the full text of instructions given to participants, including e.g., screenshots, disclaimers of any risks to participants or annotators, etc.? _Left blank._
* D2. Did you report information about how you recruited (e.g., crowdsourcing platform, students) and paid participants, and discuss if such payment is adequate given the participants' demographic (e.g., country of residence)? _Left blank._
* D3. Did you discuss whether and how consent was obtained from people whose data you're using/curating? For example, if you collected data via crowdsourcing, did your instructions to crowdworkers explain how the data would be used? _Left blank._
* D4. Was the data collection protocol approved (or determined exempt) by an ethics review board? _Left blank._
* D5. Did you report the basic demographic and geographic characteristics of the annotator population that is the source of the data? _Left blank._