SlowGuess/NAACL_2021_OCR huggingface dataset:
Hossein Rajaby Faghihi  
Michigan State University  
rajabyfa@msu.edu
Parisa Kordjamshidi  
Michigan State University  
kordjams@msu.edu
Tracking entities throughout a procedure described in a text is challenging due to the dynamic nature of the world described in the process. Firstly, we propose to formulate this task as a question answering problem. This enables us to use pre-trained transformer-based language models on other QA benchmarks by adapting those to the procedural text understanding. Secondly, since the transformer-based language models cannot encode the flow of events by themselves, we propose a Time-Stamped Language Model (TSLM model) to encode event information in LMs architecture by introducing the timestamp encoding. Our model evaluated on the Propara dataset shows improvements on the published state-of-the-art results with a  $3.1\%$  increase in F1 score. Moreover, our model yields better results on the location prediction task on the NPN-Cooking dataset. This result indicates that our approach is effective for procedural text understanding in general.
A procedural text such as a recipe or an instruction usually describes the interaction between multiple entities and their attribute changes at each step of a process. For example, the photosynthesis procedure can contain steps such as 1. Roots absorb water from soil; 2. The water flows to the leaf; 3. Light from the sun and CO2 enter the leaf; 4. The water, light, and CO2 combine into a mixture; 5. Mixture forms sugar. Procedural text understanding is a machine reading comprehension task defined on procedural texts. Answering questions such as "what is the location of the mixture at step 4", in the above example, requires tracking entities' interactions to predict their attributes at each step (Dalvi et al., 2018; Bosselut et al., 2018). This is quite challenging due to the dynamic nature of the entities' attributes in the context.
Transformer-based language models have shown promising results on multi-hop or single-hop question answering benchmarks such as HotpotQA (Yang et al., 2018), SQuAD (Rajpurkar et al., 2016), and Drop (Dua et al., 2019). However, it is hard to expect LMs to understand the flow of events and pay attention to the time in the procedure (e.g., step 4) without extra modeling efforts.
In recent research, different approaches are taken to address procedural reasoning based on language models using QA formulations. Following the intuition that attributes of entities can be retrieved based on the current and previous steps, DynaPro (Amini et al., 2020) modifies the input to only contain those sentences in the input at each time. This will provide a different input to the model based on each question to help it detect changes after adding each step. KG-MRC (Das et al., 2018) also generates a dynamic knowledge graph at each step to answer the questions. However, this intuition is contradicted in some scenarios such as detecting inputs of the process. For instance, the answer to the question "Where is light as step 0?" is "Sun", even if it is not mentioned in the first sentence of the process. Inputs are entities that are not created in the process.
The architecture of the QA transformer-based LMs is very similar to the traditional attention mechanism. Other methods such as ProLocal (Dalvi et al., 2018) and ProGlobal (Dalvi et al., 2018) have structured this task by finding the attention of each entity to the text at each step. To be sensitive to the changes at each step, ProLocal manually changes the model's input by removing all steps except the one related to the question. ProGlobal computes attention to the whole context while adding a distance value. Distance value is computed for each token based on its distance to the direct mention of the entity at each step.
The current language models convey rich linguistic knowledge and can serve as a strong basis for
Proceedings of the 2021 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, pages 4560-4570 June 6-11, 2021. ©2021 Association for Computational Linguistics
solving various NLP tasks (Liu et al., 2019; Devlin et al., 2019; Yang et al., 2019). That is why most of the state-of-the-art models on procedural reasoning are also built based on current language models (Amini et al., 2020; Gupta and Durrett, 2019). Following the same idea, we investigate the challenges that current models are facing for dealing with procedural text and propose a new approach for feeding the procedural information into LMs in a way that the LM-based QA models are aware of the taken steps and can answer the questions related to each specific step in the procedure.
We propose the Time-Stamped Language model (TSLM model), which uses timestamp embedding to encode past, current, and future time of events as a part of the input to the model. TSLM utilizes timestamp embedding to answer differently to the same question and context based on different steps of the process. As we do not change the portion of the input manually, our approach enables us to benefit from the pre-trained LMs on other QA benchmarks by using their parameters to initialize our model and adapt their architecture by introducing a new embedding type. Here, we use RoBERTa (Liu et al., 2019) as our baseline language model.
We evaluate our model on two benchmarks, Propara (Dalvi et al., 2018) and NPN-Cooking (Bosselut et al., 2018). Propara contains procedural paragraphs describing a series of events with detailed annotations of the entities along with their status and location. NPN-Cooking contains cooking recipes annotated with their ingredients and their changes after each step in criteria such as location, cleanliness, and temperature.
TSLM differs from previous research as its primary focus is on using pre-trained QA models and integrating the flow of events in the global representation of the text rather than manually changing the part of the input fed to the model at each step. TSLM outperforms the state-of-the-art models in nearly all metrics of two different evaluations defined on the Propara dataset. Results show a  $3.1\%$  F1 score improvement and a  $10.4\%$  improvement in recall. TSLM also achieves the state-of-the-art result on the location accuracy on the NPN-Cooking location change prediction task by a margin of  $1.55\%$ . In summary, our contribution is as follows:
- We propose Time-Stamped Language Model (TSLM model) to encode the meaning
of past, present, and future steps in processing a procedural text in language models.
- Our proposal enables procedural text understanding models to benefit from pre-trained LM-based QA models on general-domain QA benchmarks.
- TSLM outperforms the state-of-the-art models on the Propara benchmark on both document-level and sentence-level evaluations. TSLM improves the performance state-of-the-art models on the location prediction task of the NPN-Cooking (Bosselut et al., 2018) benchmark.
- Improving over two different procedural text understanding benchmarks suggests that our approach is effective, in general, for solving the problems that require the integration of the flow of events in a process.
An example of a procedural text is shown in Table 1. The example is taken from the Propara (Dalvi et al., 2018) dataset and shows the photosynthesis procedure. At each row, the first column is list of the sentences, each of which forms one step of the procedure. The second column contains the number of the step in the process and the rest are the entities interacting in the process and their location at each step. The location of entities at step 0 is their initial location, which is not affected by this process. If an entity has a known or unknown location (specified by “?”) at step 0, we call it an input.
The procedural text understanding task is defined as follows. Given a procedure  $\mathfrak{p}$  containing a list of  $n$  sentences  $P = \{s_1, \ldots, s_n\}$ , an entity  $e$  and a time step  $t_i$ , we find  $L$ , the location of that entity and specify the status  $S$  of that entity. Status  $S$  is one value in the predefined set of {non-existence, unknown-location, known-location}. location  $L$  is a span of text in the procedure that is specified with its beginning and end token. We formulate the task as finding function  $F$  that maps each triplet of entity, procedure and time step to a pair of entity location and status:  $(S, L) = F(e, P, t_i)$
3 Proposed Procedural Reasoning Model
3.1 QA Setting
To predict the status and the location of entities at each step, we model  $F$  with a question answering setting. For each entity  $e$ , we form the input  $Q_{e}$  as
Participants
Paragraph	State number	Water	Light	CO2	Mixture	Sugar
(Before the process starts)	State 0	Soil	Sun	?	-	-
Roots absorb water from soil	State 1	Root	Sun	?	-	-
The water flows to the leaf	State 2	Leaf	Sun	?	-	-
Light from the sun and CO2 enter the leaf	State 3	Leaf	Leaf	Leaf	-	-
The water, light, and CO2 combine into a mixture	State 4	-	-	-	Leaf	-
Mixture forms sugar	State 5	-	-	-	-	Leaf
Table 1: An example of procedural text and its annotations from the Propara dataset (Dalvi et al., 2018). "-" means entity does not exist. "?" means the location of entity is unknown.
$$
Q _ {e} = \left[ \text {C L S} \right] \text {W h e r e i s} e? [ \text {S E P} ] \tag {1}
$$
$$
s _ {1} [ \mathrm {S E P} ] s _ {2} [ \mathrm {S E P} ] \dots , s _ {n} [ \mathrm {S E P} ]
$$
Although  $Q_{e}$  is not a step-dependent representation and does not incorporate any different information for each step, our mapping function needs to generate different answers for the question "Where is entity  $e$ ?" based on each step of the procedure. For instance, consider the example in Table 1 and the question "where is water?", our model should generate different answers at four different steps. The answer will be "root", "leaf", "leaf", "nonexistence" for steps 1 to 4, respectively.
To model this, we create pairs of  $(Q_{e},t_{i})$  for each  $i\in \{0,1,\dots,n\}$ . For each pair,  $Q_{e}$  is timestamped according to  $t_i$  using Timestamp(). function described in Sec. 3.2 and mapped to an updated step-dependent representation,  $Q_{e}^{t_{i}} = \text{Timestamp}(Q_{e},t_{i})$ .
The updated input representation is fed to a language model (here ROBERTA) to obtain the step-dependent entity representation,  $R_{e}^{t_{i}}$ , as shown in Equation 2. We discuss the special case of  $i = 0$  in more details in Sec. 3.2.
$$
R _ {e} ^ {t _ {i}} = R o B E R T a \left(Q _ {e} ^ {t _ {i}}\right) \tag {2}
$$
We use the step-dependent entity representation,  $R_{e}^{t_i}$ , and forward it to another mapping function  $g(.)$  to obtain the location and status of the entity  $e$  in the output. In particular the output includes the following three vectors, a vector representing the predictions of entity status  $S$ , another vector for each token's probability of being the start of the location span  $L$ , and a third vector carrying the probability of each word being the last token of the location span. The outputs of the model are computed according to the Equation 3.
$$
\left(\text {s t a t u s}, \text {S t a r t} _ {\text {p r o b}}, \text {E n d} _ {\text {p r o b}}\right) = g \left(R _ {e} ^ {t _ {i}}\right) \tag {3}
$$
where  $R_{e}$  is the tokens' representations output of RoBERTa (Liu et al., 2019), and  $g(. )$  is a function we apply on the token representations to get the final predictions. We will discuss each part of the model separately in the following sections.
3.2 Timestamp Embedding
The timestamp embedding adds the step information to the input  $Q_{e}$  to be considered in the attention mechanism. The step attention is designed to distinguish between current (what is happening now), past (what has happened before), and future (what has not yet happened) information.
We use the mapping function  $\text{Timestamp}(.)$  from the pair  $(Q_e, t_i)$  to add a number along with each token in  $Q_e$  and retrieve the step-dependent input  $Q_e^{t_i}$  as shown in Figure 1. The Mapping function  $\text{TESTAMP}(.)$  integrates past, current, and future representations to all of the tokens related to each part.  $\text{TESTAMP}(.)$  function assigns number 1 for past, 2 for current, and 3 for future tokens in the paragraph by considering one step of the process as the current event. These values are used to compute an embedding vector for each token, which will be added to its initial representation as shown in Figure 2. The special number 0 is assigned to the question tokens, which are not part of the process timeline. For predicting State 0 (The inputs of the process), we set all the paragraph information as the current step.
3.3 Status classification
To predict the entities' status, we apply a linear classification module on top of the [CLS] token representation in  $R_e$  as shown in Equation 4.
$$
A t t r i b u t e = S o f t m a x \left(W ^ {T} R e _ {[ C ]}\right) \tag {4}
$$
where  $Re_{[C]}$  is the representation of the [CLS] token which is the first token in  $R_e$ .
Where is Water? Roots absorb water from soil. The water flows to the leaf. The water, light and CO2 combine into a mixture.
Figure 1: An example of timestamp embedding in a procedural text. The question is always ignored with value "0". At each step  $i$ , the tokens from that step are paired with "current" value, tokens from steps 0 to  $i$  are paired with "past" value, and the tokens from step  $i$  to last step are paired with the "future" value.
3.4 Span prediction
We predict a location span for each entity for each step of the process as shown in Equation 5, we follow the popular approach of selecting start/end tokens to detect a span of the text as the final answer. We compute the probability of each token being the start or the end of the answer span. If the index with the highest probability to be the start token is  $\text{token}_{\text{start}}$  and for the end token is  $\text{token}_{\text{end}}$ , the answer location will be  $\text{Location} = P[\text{token}_{\text{start}} : \text{token}_{\text{end}}}$ .
$$
\text {S t a r t} _ {\text {p r o b}} = \operatorname {S o f t m a x} \left(W _ {\text {s t a r t}} ^ {T} R _ {e} ^ {t _ {i}}\right)
$$
$$
\operatorname {E n d} _ {-} \operatorname {p r o b} = \operatorname {S o f t m a x} \left(W _ {\text {e n d}} ^ {T} R _ {e} ^ {t _ {i}}\right)
$$
$$
\operatorname {t o k e n} _ {\text {s t a r t}} = \underset {i} {\arg \max } (\text {S t a r t} _ {-} \text {p r o b}) \tag {5}
$$
$$
\operatorname {t o k e n} _ {\text {e n d}} = \underset {i} {\arg \max } (\text {E n d} _ {-} \operatorname {p r o b})
$$
We use the cross-entropy loss function to train the model. At each prediction for entity  $e$  at times-tamp  $t_i$ , we compute one loss value  $loss_{attribute}$  regarding the status prediction and one loss value  $loss_{location}$  for the span selection. The variable  $loss_{location}$  is the summation of the losses of the start token and the end token prediction,  $loss_{location} = loss_{location\_start} + loss_{location\_end}$ . The final loss of entity  $e$  at time  $t_i$  is computed as in Equation 6.
$$
L o s s _ {i} ^ {e} = l o s s _ {(i, a t t r i b u t e)} ^ {e} + l o s s _ {(i, l o c a t i o n)} ^ {e} \tag {6}
$$
Figure 2: An overview of the proposed model. The "Timestamp Embedding" module is introduced in this work and the rest of the modules are taken from basic language model architecture.
3.6 Inference
At inference time, we apply two different post-processing rules on the outputs of the model. First, we impose that the final selected location answer should be a noun phrase in the original procedure. Considering that a location span is a noun phrase, we limit the model to do a softmax over tokens of noun phrases in the paragraph to select the start and end tokens. Second, we apply consistency rules to make sure that our predicted status of entities are consistent. We define the two following rules:
- An entity can not be created if it has been already destroyed: if  $S_{e}^{t_{i}}$  is "non-existence" and  $S_{e}^{t_{i+1}}$  is unknown or known location, then for every step  $j$ , if  $S_{e}^{t_{j}}$  is unknown or known location and  $S_{e}^{t_{j+1}}$  is "non-existence", then  $i < j$ .
- An entity cannot be created/destroyed twice in a process: if  $S_{e}^{t_{j}}$  and  $S_{e}^{t_{i}}$  are both "-" and  $S_{e}^{t_{i+1}}$  are both either known or unknown location, then  $i = j$ .
$S_{e}^{t_{i}}$  is the status of entity  $e$  at step  $t_i$  of the process.
We do not apply an optimization/search algorithm to find the best assignment over the predictions according to the defined constraints. The constraints are only applied based on the order of the steps to ensure that the later predictions are consistent with the ones made before.
Propara (Dalvi et al., 2018): This dataset was created as a benchmark for procedural text understanding to track entities at each step of a process. Propara contains 488 paragraphs and 3,300 sentences with annotations that are provided by crowd-workers. The annotations (81,000) are the location of entities at each step of the process. The location can be either the name of the location, unknown location, or specified as non-existence.
NPN-Cooking (Bosselut et al., 2018): This is a benchmark containing textual cooking instructions. Annotators have specified ingredients of these recipes and explained the recipe using different changes happening on each ingredient at each step of the instructions. These changes are reported in categories such as location, temperature, cleanliness, and shape. We evaluate our model on the location prediction task of this benchmark, which is the hardest task due to having more than 260 candidate answers. We do not use the candidates to find the locations in our setting; Instead, we find a span of the text as the final location answer. This is a relatively harder setting but more flexible and generalizable than the classification setting.
We use SGD optimizer implemented by Pytorch (Paszke et al., 2017) to update the model parameters. The learning rate for the Propara implementation is set to  $3 - e4$  and is updated by a scheduler with a 0.5 coefficient every 50 steps. We use  $1 - e6$  as the learning rate and a scheduler with 0.5 coefficient to update the parameters every ten steps on the NPN-Cooking implementation. The implementation code is publicly available at
Step	Entity	Action	Before	After
1	Water	Move	Root	Leaf
2	Water	Destroy	Leaf	-
1	Sugar	Create	-	Leaf
2	Sugar	None	Leaf	Leaf
Table 2: A sample table to evaluate the Propara document-level task.
GitHub<sup>1</sup>.
We use RoBERTa (Liu et al., 2019) question answering architecture provided by HuggingFace (Wolf et al., 2019). RoBERTa is pretrained with SQuAD (Rajpurkar et al., 2016) and used as our base language model to compute the token representations. Our model executes batches containing an entity at every step and makes updates based on the average loss of entities per procedure. The network parameters are updated after executing one whole example. The implementation code will be publicly available on GitHub after acceptance.
Sentence-level evaluation is introduced in (Dalvi et al., 2018) for Propara dataset. This evaluation focuses on the following three categories.
- Cat1 Is  $e$  created (destroyed/moved) during the process?
- Cat2 When is  $e$  created (destroyed/moved) during the process?
- Cat3 Where is  $e$  created (destroyed/moved from or to) during the process?
Document-level evaluation is a more comprehensive evaluation process and introduced later in (Tandon et al., 2018) for Propara benchmark. Currently, this is the default evaluation in the Propara leaderboard containing four criteria:
- What are the Inputs? Which entities existed before the process began and do not exist after the process ends.
- What are the Outputs? Which entities got created during the process?
- What are the Conversions? Which entities got converted to other entities?
- What are the Moves? Which entities moved from one location to another?
The document-level evaluation requires models to reformat their predictions in a tabular format as shown in Table 2. At each row of this table, for each entity at a specific step, we can see the action
$^{1}$ https://github.com/HLR/TSLM
Sentence-level	Document-level
Model	Cat1	Cat2	Cat3	MacroAvg	MicroAvg	P	R	F1
ProLocal (Dalvi et al., 2018)	62.7	30.5	10.4	34.5	34.0	77.4	22.9	35.3
ProGlobal (Dalvi et al., 2018)	63.0	36.4	35.9	45.1	45.4	46.7	52.4	49.4
EntNet (Henaff et al., 2017)	51.6	18.8	7.8	26.1	26.0	50.2	33.5	40.2
QRN (Seo et al., 2017)	52.4	15.5	10.9	26.3	26.5	55.5	31.3	40.0
KG-MRC (Das et al., 2018)	62.9	40.0	38.2	47.0	46.6	64.5	50.7	56.8
NCET (Gupta and Durrett, 2019)	73.7	47.1	41.0	53.9	54.0	67.1	58.5	62.5
XPAD (Dalvi et al., 2019)	-	-	-	-	-	70.5	45.3	55.2
ProStruct (Tandon et al., 2018)	-	-	-	-	-	74.3	43.0	54.5
DYNAPRO (Amini et al., 2020)	72.4	49.3	44.5	55.4	55.5	75.2	58.0	65.5
TSLM (Our Model)	78.81	56.8	40.9	58.83	58.37	68.4	68.9	68.6
Table 3: Results from sentence-level and document-level evaluation on Propara. Cati evaluations are defined in Section 4.3.
applied on that entity, the location of that entity before that step, and the location of the entity after that step. Action takes values from a predefined set including, "None", "Create", "Move", and "Destroy". The exact action can be specified based on the before and after locations.
We have to process our (Status  $S$ , Location  $L$ ) predictions at each step to generate a similar tabular format as in Table 2. We define  $r_e^i$  as a row in this table which stores the predictions related to entity  $e$  at step  $t_i$ . To fill this row, we first process the status predictions. If the status prediction  $S$  is either “-” or “?”, we fill those values directly in the after location column. The before location column value of  $r_e^i$  is always equal to the after location column value of  $r_e^{i - 1}$ . If the status is predicted to be a “Known Location”, we fill the predicted location span  $L$  into the after location column of  $r_e^i$ .
The action column is filled based on the data provided in before and after locations columns. If the before location is/isn't "-" and after location is not/is "-" then the action is "Create"/"Destroy". If the before and after locations are equal, then the action is "None" and if the before and after locations are both spans and are different from each other, the action is "Move".
NPN-Cooking location change: We evaluate our model on the NPN-Cooking benchmark by computing the accuracy of the predicted locations at steps where the locations of ingredients change. We use the portion of the data that has been annotated by the location changes to train and evaluate our model. In this evaluation, we do not use the status prediction part of our proposed TSLM model. Since training our model on the whole training set
takes a very long time (around 20 hours per iteration), we use a reduced number of samples for training. This is a practice that is also used in other prior work (Das et al., 2018).
The performance of our model on Propara dataset (Dalvi et al., 2018) is quantified in Table 3. Results show that our model improves the SOTA by a  $3.1\%$  margin in the F1 score and improves the Recall metric with  $10.4\%$  on the document-level evaluation. On the sentence-level evaluation, we outperform SOTA models with a  $5.11\%$  in Cat1, and  $7.49\%$  in Cat2 and by a  $3.4\%$  margin in the macro-average. We report Table 3 without considering the consistency rules and evaluate the effect of those in the ablation study in Sec. 4.5.