SlowGuess/NAACL_2021_OCR huggingface dataset:
In Table 5, we report a more detailed quantified analysis of TSLM model's performance based on each different criteria defined in the document-level evaluation. Table 5 shows that our model performs best on detecting the procedure's outputs and performs worst on detecting the moves. Detecting moves is essentially hard for TSLM as it is predicting outputs based on the whole paragraph at once. Outperforming SOTA results on the input and output detection suggests that TSLM model can understand the interactions between entities and detect the entities which exist before the process begins. The detection of input entities is one of the weak aspects of the previous research that we improve here.
A recent unpublished research (Zhang et al., 2021) reports better results than our model. However, their primary focus is on common-sense rea
Model	Accuracy	Training Samples	Prediction task
NPN-cooking (Bosselut et al., 2018)	51.3	~ 83,000 (all data)	Classification
KG-MRC (Das et al., 2018)	51.6	~ 10,000	Span Prediction
DynaPro (Amini et al., 2020)	62.9	~ 83,000 (all data)	Classification
TSLM (Our Model)	63.73	~ 10,000	Span Prediction
64.45	~ 15,000	Span Prediction
Table 4: Results on the NPN-Cooking benchmark. Both class prediction and span prediction tasks are the same but use two different settings, one selects among candidates, and the other chooses a span from the recipe. However, each model has used a different setting and a different portion of the training data. The information of the data splits was not available that makes a fair comparison hard.
Criteria	Precision	Recall	F1
Inputs	89.8	71.3	79.5
Outputs	85.6	91.4	88.4
Conversions	57.7	56.7	57.2
Moves	40.5	56	47
Table 5: Detailed analysis of TSLM performance on the Propara test set on four criteria defined in the document-level evaluation.
soning and their goal is orthogonal to our main focus in proposing TSLM model. Such approaches can be later integrated with TSLM to benefit from common-sense knowledge on solving the Propara dataset.
The reason that TSLM performs better at recall and worse at precision is that our model looks at the global context, which increases the recall and lowers the precision when local information is strongly important. The same phenomenon (better recall) is observed in ProGlobal, which also considers global information as we do, compared to ProLocal.
Table 4 shows our results on the NPN-Cooking benchmark for the location prediction task. Results are computed by only considering the steps that contain a location change and are reported by computing the accuracy of predicting those changes. Our results show that TSLM outperforms the SOTA models with a  $1.55\%$  margin on accuracy even after training on 15,000 training samples. To be comparable with the KG-MRC (Das et al., 2018) experiment on NPN-Cooking which is only trained on 10k samples, we report the performance of our model trained on the same number of samples, where TSLM gets a  $12.1\%$  improvement over the performance of KG-MRC (Das et al., 2018).
To evaluate the importance of each module one at a time, we report the performance of the TSLM
by removing the noun-phrase filtering at inference, the consistency rules, timestamp embedding, SQuAD (Rajpurkar et al., 2016) pre-training, and by replacing RoBERTa (Liu et al., 2019) with BERT (Devlin et al., 2019). These variations are evaluated on the development set of the Propara dataset and reported in Table 6. As stated before and shown in Table 6, it is impossible to remove the timestamp embedding as that is the only part of the model enabling changes in the answer at each step. Hence, by removing that, the model cannot converge and yields a  $25\%$  decrease on the F1 score. The simple consistency and span filtering rules are relatively easy to be learned by the model based on the available data, therefore adding those does not affect the final performance of the model.
$\mathrm{TSLM}_{\mathrm{BERT}}$  experiment is designed to ensure a fair comparison with previous research (Amini et al., 2020) which has used BERT as their base language model. The comparison of  $\mathrm{TSLM}_{\mathrm{BERT}}$  to -SQuAD Pre-training and - Timestamp Embedding in Table 6 indicates that using RoBERTa instead of BERT is not as much important as our main proposal (using Time-stamp encoding) in TSLM model. Also,  $\mathrm{TSLM}_{\mathrm{BERT}}$  achieves  $66.7\%$  F1 score on the Propara test set, which is  $1.2\%$  better than the current SOTA performance.
By removing the SQuAD pre-training phase, the model performance drops with a  $10.6\%$  in the F1 score. This indicates that despite the difference between the procedural text understanding and the general MRC tasks, it is quite beneficial to design methods that can transfer knowledge from other QA data sources to help with procedural reasoning. This is crucial as annotating procedural texts is relatively more expensive and time-consuming.
Model	P	R	F1
\( \text{TSLM}_{\text{RoBERTa}} \)	72.9	74.1	73.5
- constraints	73.8	73.3	73.5
- noun-phrase filtering	73.5	73.3	73.4
- SQuAD Pre-training	78.8	52.2	62.8
- Timestamp Embedding	94.6	32.6	48.5
\( \text{TSLM}_{\text{BERT}} \)	69.2	73.5	71.3
Table 6: Ablation study results on the development set of the Propara document-level task. “- constraints”, “-Span filtering”, and “- Timestamp Encoding” shows our model performance while removing those modules. -SQuAD Pre-training is when we do not pre-train our base language model on SQuAD.  $\mathrm{TSLM}_{\mathrm{BERT}}$  is when we use BERT as the base language model.
We provide more samples to support our hypothesis in solving the procedural reasoning task and answer some of the main questions about the ideas presented in TSLM model.
Why is the whole context important? The main intuition behind TSLM is that the whole context, not just previous information, matters in reasoning over a process. Here, we provide some samples from Propara to show why this intuition is correct. Consider this partial paragraph, "Step  $i$ : With enough time the pressure builds up greatly. Step  $i + 1$ : The resulting volcano may explode." Looking at the annotated status and location, the "volcano" is being created at Step  $i$  without even being mentioned in that step. This is only detectable if we look at the next step saying "The resulting Volcano...".
As another example, consider this partial paragraph: "Step  $i$ : Dead plants form layers called peat. ... Step  $i + 3$ : Pressure squeezes water out of the peat." The annotation indicates that the location of "water" is being changed to "peat" at step  $i$ , which is only possible to detect if the model is aware of the following steps indicating that the water comes out of the peat.
Positional Embedding VS Time-stamp encoding: As mentioned before the whole context (future and past events) is essential for procedural reasoning at a specific step. However, the reasoning should focus on one step at a time, given the whole context. While positional encoding encodes the order of information at the token-level for reasoning over the entire text, we need another level of encoding to specify the steps' positions (bound-
aries) and, more importantly, to indicate the step that the model should focus on when answering a question.
Advantages/Disadvantages of TSLM model: TSLM integrates higher-level information into the token representations. This higher-level information can come from event-sequence (time of events), sentence-level, or any other higher source than the token-level information. The first advantage of TSLM is that it enables designing a model which is aware of the whole context, while previous methods had to customize the input at each step to only contain the information of earlier steps. Furthermore, using TSLM enables us to use pretrained QA models on other datasets without requiring us to retrain them with the added time-stamped encoding. One main disadvantage of TSLM model, which is natural due to the larger context setting in this model, is not being sensitive to local changes, which is consistent with the observation in the comparison between ProGlobal and ProLocal models.
ScoNe (Long et al., 2016), NPN-Cooking (Bosselut et al., 2018), bAbI (Weston et al., 2015), ProcessBank (Berant et al., 2014), and Propara (Dalvi et al., 2018) are benchmarks proposed to evaluate models on procedural text understanding. Processbank (Berant et al., 2014) contains procedural paragraphs mainly concentrated on extracting arguments and relations for the events rather than tracking the states of entities. ScoNe (Long et al., 2016) aims to handle co-reference in a procedural text expressed about a simulated environment. bAbI (Weston et al., 2015) is a simpler machine-generated textual dataset containing multiple procedural tasks such as motion tracking, which has encouraged the community to develop neural network models supporting explicit modeling of memories (Sukhbaatar et al., 2015; Santoro et al., 2018) and gated recurrent models (Cho et al., 2014; Henaff et al., 2017). NPN-Cooking (Bosselut et al., 2018) contains recipes annotated with the state changes of ingredients on criteria such as location, temperature, and composition. Propara (Dalvi et al., 2018) provides procedural paragraphs and detailed annotations of entity locations and the status of their existence at each step of a process.
Inspired by Propara and NPN-Cooking benchmarks, recent research has focused on tracking entities in a procedural text. Query Reduction Net
works (QRN) (Seo et al., 2017) performs gated propagation of a hidden state vector at each step. Neural Process Network (NPN) (Bosselut et al., 2018) computes the state changes at each step by looking at the predicted actions and involved entities. Prolocal (Dalvi et al., 2018) predicts locations and status changes locally based on each sentence and then globally propagates the predictions using a persistence rule. Proglobal (Dalvi et al., 2018) predicts the status changes and locations over the whole paragraph using distance values at each step and predicts current status based on current representation and the predictions of the previous step. ProStruct (Tandon et al., 2018) aims to integrate manually extracted rules or knowledge-base information on VerbNet (Schuler, 2005) as constraints to inject common-sense into the model. KG-MRC (Das et al., 2018) uses a dynamic knowledge graph of entities over time and predicts locations with spans of the text by utilizing reading comprehension models. Ncut (Gupta and Durrett, 2019) updates entities representation based on each sentence and connects sentences together with an LSTM. To ensure the consistency of predictions, Ncut uses a neural CRF over the changing entity representations. XPAD (Dalvi et al., 2019) is also proposed to make dependency graphs on the Propara dataset to explain the dependencies of events over time. Most recently, DynaPro (Amini et al., 2020) feeds an incremental input to pretrained LMs' question answering architecture to predict entity status and transitions jointly.
TSLM differs from recent research, as we propose a simple, straightforward, and effective technique to make our model benefit from pre-trained LMs on general MRC tasks and yet enhance their ability to operate on procedural text understanding. We explicitly inject past, current, and future timestamps into the language models input and implicitly train the model to understand the events' flow rather than manually feeding different portions of the context at each step. Procedural reasoning has also been pursued within the multi-modality domain (Yagcioglu et al., 2018; Rajaby Faghihi et al., 2020; Amac et al., 2019) which has additional challenges of aligning the representation spaces of different modalities.
We proposed the Time-Stamped Language Model (TSLM model), a novel approach based
on a simple and effective idea, which enables pre-trained QA models to process procedural texts and produce different outputs based on each step to track entities and their changes. TSLM utilizes a timestamp function that causes the attention modules in the transformer-based LM architecture to incorporate past, current, and future information by computing a timestamp embedding for each input token. Our experiments show a  $3.1\%$  improvement on the F1 score and a  $10.4\%$  improvement over the Recall metric on Propara Dataset. Our model further outperforms the state-of-the-art models with a  $1.55\%$  margin in the NPN-Cooking dataset accuracy for the location prediction task.
As a future direction, it is worth investigating how common-sense knowledge can be integrated with the TSLM setting by augmenting the process context using external sources of related domain knowledge. We also intend to investigate the effectiveness of our approach on similar tasks on other domains and benchmarks. As another future direction, it can be effective to apply an inference algorithm to impose the global consistency constraints over joint predictions in procedural reasoning instead of using naive post-processing rules.
This project is partially funded by National Science Foundation (NSF) CAREER Award #2028626 and the Office of Naval Research (ONR) grant #N00014-20-1-2005.
Mustafa Sercan Amac, Semih Yagcioglu, Aykut Erdem, and Erkut Erdem. 2019. Procedural reasoning networks for understanding multimodal procedures. In Proceedings of the 23rd Conference on Computational Natural Language Learning (CoNLL), pages 441-451.
Aida Amini, Antoine Bosselut, Bhavana Dalvi Mishra, Yejin Choi, and Hannaneh Hajishirzi. 2020. Procedural reading comprehension with attribute-aware context flow. In Proceedings of the Conference on Automated Knowledge Base Construction (AKBC).
Jonathan Berant, Vivek Srikumar, Pei-Chun Chen, Abby Vander Linden, Brittany Harding, Brad Huang, Peter Clark, and Christopher D Manning. 2014. Modeling biological processes for reading comprehension. In Proceedings of the 2014 Conference on Empirical Methods in Natural Language Processing (EMNLP), pages 1499-1510.
Antoine Bosselut, Omer Levy, Ari Holtzman, Corin Ennis, Dieter Fox, and Yejin Choi. 2018. Simulating action dynamics with neural process networks. In Proceedings of the 6th International Conference for Learning Representations (ICLR).
Kyunghyun Cho, Bart van Merrienboer, Dzmitry Bah-danau, and Yoshua Bengio. 2014. On the properties of neural machine translation: Encoder-decoder approaches. In Proceedings of SSST-8, Eighth Workshop on Syntax, Semantics and Structure in Statistical Translation, pages 103-111, Doha, Qatar. Association for Computational Linguistics.
Bhavana Dalvi, Lifu Huang, Niket Tandon, Wen-tau Yih, and Peter Clark. 2018. Tracking state changes in procedural text: a challenge dataset and models for process paragraph comprehension. In Proceedings of the 2018 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long Papers), pages 1595-1604, New Orleans, Louisiana. Association for Computational Linguistics.
Bhavana Dalvi, Niket Tandon, Antoine Bosselut, Wentau Yih, and Peter Clark. 2019. Everything happens for a reason: Discovering the purpose of actions in procedural text. In Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP), pages 4496-4505, Hong Kong, China. Association for Computational Linguistics.
Rajarshi Das, Tsendsuren Munkhdalai, Xingdi Yuan, Adam Trischler, and Andrew McCallum. 2018. Building dynamic knowledge graphs from text using machine reading comprehension. In International Conference on Learning Representations.
Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. 2019. BERT: Pre-training of deep bidirectional transformers for language understanding. In Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long and Short Papers), pages 4171-4186, Minneapolis, Minnesota. Association for Computational Linguistics.
Dheeru Dua, Yizhong Wang, Pradeep Dasigi, Gabriel Stanovsky, Sameer Singh, and Matt Gardner. 2019. DROP: A reading comprehension benchmark requiring discrete reasoning over paragraphs. In Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long and Short Papers), pages 2368-2378, Minneapolis, Minnesota. Association for Computational Linguistics.
Aditya Gupta and Greg Durrett. 2019. Tracking discrete and continuous entity state for process understanding. In Proceedings of the Third Workshop
on Structured Prediction for NLP, pages 7-12, Minneapolis, Minnesota. Association for Computational Linguistics.
Mikael Henaff, Jason Weston, Arthur Szlam, Antoine Bordes, and Yann LeCun. 2017. Tracking the world state with recurrent entity networks. In 5th International Conference on Learning Representations, ICLR 2017.
Yinhan Liu, Myle Ott, Naman Goyal, Jingfei Du, Mandar Joshi, Danqi Chen, Omer Levy, Mike Lewis, Luke Zettlemoyer, and Veselin Stoyanov. 2019. Roberta: A robustly optimized BERT pretraining approach. CoRR, abs/1907.11692.
Reginald Long, Panupong Pasupat, and Percy Liang. 2016. Simpler context-dependent logical forms via model projections. In Proceedings of the 54th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 1456-1465, Berlin, Germany. Association for Computational Linguistics.
Adam Paszke, Sam Gross, Soumith Chintala, Gregory Chanan, Edward Yang, Zachary DeVito, Zeming Lin, Alban Desmaison, Luca Antiga, and Adam Lerer. 2017. Automatic differentiation in pytorch.
Hossein Rajaby Faghihi, Roshanak Mirzaee, Sudarshan Paliwal, and Parisa Kordjamshidi. 2020. Latent alignment of procedural concepts in multimodal recipes. In Proceedings of the First Workshop on Advances in Language and Vision Research, pages 26-31.
Pranav Rajpurkar, Jian Zhang, Konstantin Lopyrev, and Percy Liang. 2016. SQuAD: 100,000+ questions for machine comprehension of text. In Proceedings of the 2016 Conference on Empirical Methods in Natural Language Processing, pages 2383-2392, Austin, Texas. Association for Computational Linguistics.
Adam Santoro, Ryan Faulkner, David Raposo, Jack Rae, Mike Chrzanowski, Theophane Weber, Daan Wierstra, Oriol Vinyals, Razvan Pascanu, and Timothy Lillicrap. 2018. Relational recurrent neural networks. In Advances in neural information processing systems, pages 7299-7310.
Karin Kipper Schuler. 2005. Verbnet: A broad-coverage, comprehensive verb lexicon.
Min Joon Seo, Sewon Min, Ali Farhadi, and Hannaneh Hajishirzi. 2017. Query-reduction networks for question answering. In 5th International Conference on Learning Representations, ICLR 2017, Toulon, France, April 24-26, 2017, Conference Track Proceedings. OpenReview.net.
Sainbayar Sukhbaatar, arthur szlam, Jason Weston, and Rob Fergus. 2015. End-to-end memory networks. In Advances in Neural Information Processing Systems, volume 28, pages 2440-2448. Curran Associates, Inc.
Niket Tandon, Bhavana Dalvi, Joel Grus, Wen-tau Yih, Antoine Bosselut, and Peter Clark. 2018. Reasoning about actions and state changes by injecting commonsense knowledge. In Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing, pages 57-66, Brussels, Belgium. Association for Computational Linguistics.
Jason Weston, Antoine Bordes, Sumit Chopra, Alexander M. Rush, Bart van Merrienboer, Armand Joulin, and Tomas Mikolov. 2015. Towards ai-complete question answering: A set of prerequisite toy tasks. Cite arxiv:1502.05698.
Thomas Wolf, Lysandre Debut, Victor Sanh, Julien Chaumont, Clement Delangue, Anthony Moi, Pierric Cistac, Tim Rault, Rémi Louf, Morgan Funtowicz, Joe Davison, Sam Shleifer, Patrick von Platen, Clara Ma, Yacine Jernite, Julien Plu, Canwen Xu, Teven Le Scao, Sylvain Gugger, Mariama Drame, Quentin Lhoest, and Alexander M. Rush. 2019. Huggingface's transformers: State-of-the-art natural language processing. ArXiv, abs/1910.03771.
Semin Yagcioglu, Aykut Erdem, Erkut Erdem, and Nazli Ikizler-Cinbis. 2018. Recipeqa: A challenge dataset for multimodal comprehension of cooking recipes. In EMNLP.
Zhilin Yang, Zihang Dai, Yiming Yang, Jaime Carbonell, Russ R Salakhutdinov, and Quoc V Le. 2019. Xlnet: Generalized autoregressive pretraining for language understanding. In Advances in neural information processing systems, pages 5753-5763.
Zhilin Yang, Peng Qi, Saizheng Zhang, Yoshua Bengio, William Cohen, Ruslan Salakhutdinov, and Christopher D. Manning. 2018. HotpotQA: A dataset for diverse, explainable multi-hop question answering. In Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing, pages 2369-2380, Brussels, Belgium. Association for Computational Linguistics.
Zhihan Zhang, Xiubo Geng, Tao Qin, Yunfang Wu, and Daxin Jiang. 2021. Knowledge-aware procedural text understanding with multi-stage training. In Proceedings of the Web Conference 2021.