SlowGuess/NAACL_2021_OCR huggingface dataset:
$$
\begin{array}{l} r _ {e n d s} \left(e _ {1}, e _ {2}\right) = \mathbf {b e f o r e} \\ \Leftrightarrow \mathbf {e n d} _ {1} <   \mathbf {s t a r t} _ {2} \\ \Leftrightarrow \text {s t a r t} _ {1} + \text {d u r a t i o n} _ {1} <   \text {s t a r t} _ {2} \\ \Leftrightarrow \left(\mathbf {s t a r t} _ {1} - \mathbf {s t a r t} _ {2}\right) + \mathbf {d u r a t i o n} _ {1} <   0 \\ \Leftrightarrow \operatorname {d i s t} \left(e _ {1}, e _ {2}\right) + \operatorname {d u r} \left(e _ {1}\right) <   0 \\ \end{array}
$$
Figure 6: A schematic overview of SYMTIME to compare event  $A$  's end time with event  $B$  's start time via modular predictions about  $A$  's duration and distance from  $B$  and their symbolic combination (bottom).
Hence, we have reduced the computation of the relation ends before to a symbolic computation over two numeric intervals. Conversely, we have  $r_{\text{ends}}(e_1, e_2) = \text{after} \Leftrightarrow \text{dist}(e_1, e_2) + \text{dur}(e_1) > 0$ . For the starts comparator, we have  $r_{\text{starts}}(e_1, e_2) = \text{before} \Leftrightarrow \text{dist}(e_1, e_2) < 0$  and vice versa for the after relation.
In what follows, we describe how we approximate the values of the two functions via individual neural modules (see illustration in Fig. 6).
5.2 Duration Estimation
To obtain a model to estimate  $\mathrm{dur}(\cdot)$ , we pre-train a sequence-to-sequence model with the duration data from Zhou et al. (2020), which is similarly collected from pattern-based extraction. The data contains over 1 million events with their corresponding duration values. We map each instance to an input sequence event: [Event] story: [Story] and a corresponding output sequence answer: [Value], where [Event] represents the tokens of an event with the trigger verb marked by a special token to its left, [Story] represents down-sampled tokens from the context, and [Value] is one of the 7 unit labels as described in §4.1 (i.e., { ≤minutes, hours, days, weeks, months, years, ≥decades}).
5.3 Computation and Learning
We use the output from PTNTIME to approximate the function  $\mathrm{dist}(\cdot)$ . Following the sequence formulation of PTNTIME in §4, we replace [EventA] with the textual description of  $e_1$ , [EventB] with
In the Allen algebra, the values  $\mathbf{end}_x$  and  $\mathbf{start}_y$  correspond to the right and left end points  $x^{+}, y^{-}$  in the intervals  $(x^{-}, x^{+})$ ,  $(y^{-}, y^{+})$ . Likewise, our  $\mathbf{duration}_x$  corresponds to the value  $(x^{+} - x^{-})$ .
<sup>6</sup>We note that one drawback of this inference rule is that it does not predict causal relations and, therefore, cannot handle instances where  $\mathbf{end}_1 = \mathbf{start}_2$  as our label definitions describe in §3.1. We leave this problem for future research.
the textual description of  $e_2$ , and [Paragraph] with the context (premise), and fix [Relation] to be before. By taking the values of the vocabulary indices corresponding to "positive" and "negative" from the logits of [Label] and applying a softmax operation, we get  $P_{\mathrm{before}}$  and  $P_{\mathrm{after}}$ . These are the probability of  $e_1$  starting before and after  $e_2$ , respectively, and are used to define the vector  $\mathbf{p} = [P_{\mathrm{before}}, P_{\mathrm{after}}]$ . Similarly, we apply softmax to the logits of [Distance] over the 7 words representing the temporal units to obtain 7 values that approximate the probabilities of the distance between two events' start times being closest to each temporal unit. We place the 7 values in temporal units' increasing order in vector  $\mathbf{d}$ . To represent  $|\mathbf{start}_1 - \mathbf{start}_2|$  with a single value, we dot product the probabilities with an incremental constant vector  $\mathbf{c} = [0, 1, 2, 3, 4, 5, 6]$ . To get the direction, we apply the tanh function to the difference between the probabilities in  $\mathbf{p}$ . As a result, we have:
$$
\begin{array}{l} \operatorname {d i s t} (\cdot) = \mathbf {s t a r t} _ {1} - \mathbf {s t a r t} _ {2} \\ = \mathbf {c} ^ {T} \mathbf {d} * \tanh  \left(\mathrm {I N T} _ {\max } * \left(\mathbf {p} _ {2} - \mathbf {p} _ {1}\right)\right) \tag {1} \\ \end{array}
$$
We use the pre-trained model in §5.2 to approximate the function  $\mathrm{dur}(\cdot)$ . Because the model is pre-trained with markers to the left of trigger verbs, we run a part-of-speech tagger on input phrases and add a marker to the left of the first verb. We apply softmax to the logit values of [Value] over the 7 temporal unit words and get, as above, 7 values representing the probabilities of the input event's duration being closest to each unit. We form  $\mathbf{v}$  by placing these values at the temporal unit's increasing order. With the same constant vector, we have:
$$
\operatorname {d u r} (\cdot) = \mathbf {d u r a t i o n _ {1}} = \mathbf {c} ^ {T} \mathbf {v} \tag {2}
$$
For hypotheses with comparator starts, we use PTNTIME and its sequence-to-sequence objective to learn (i.e., we take the input hypothesis and context as is and use [Label] directly as the prediction). For hypotheses where the comparator is ends, we use the inference process in §5.1 and the computation process described above to construct  $\text{logits} = [\text{pred}, -\text{pred}]$ ,  $\text{pred} = \text{dist}(e_1, e_2) + \text{dur}(e_1)$  as detailed in Fig. 6. We find the gold-temporal-relation in each training instance and compute a two-class cross-entropy loss with  $\text{logits}$ . The PTNTIME that predicts starts
hypotheses shares weights with the one used in computing logits. The final model SYMTIME can also be used to predict TRACIE instances without any task-specific supervision as the two functions are initialized with distant supervision.
In this section, we detail our experimental setup (§6.1-6.2) and report our main results (§6.3-6.5).<sup>8</sup>
6.1 Baselines and Systems
We use T5-Large implemented by Wolf et al. (2019) as our base sequence-to-sequence model for both PTNTIME and the duration model in §5.2 as it provides for faster iterations. We use early stopping, batch size of 32 and other default parameters. PTNTIME converges after 45k steps ( $\sim$ 1.4M instances) and the duration model converges after 80k steps ( $\sim$ 2.6M instances). We use these pretrained weights in SYMTIME as well as SYMTIME-ZEROSHOT which uses no TRACIE supervision.
We compare with our proposed models with a host of baselines based on the same pre-trained language model, including BaseLM: T5-Large, and BaseLM-MATRES: T5-Large fine-tuned on 20k MATRES training data. We also compare with other architectures/models, including BiLSTM as used in Williams et al. (2017), Roberta-Large (Liu et al., 2019) and T5-3B. All models and baselines follow a standard TE setup and default parameters. We report a 3-run average and each model is run until convergence.
6.2 Metrics and Settings
We measure system performance on TRACIE separately for start-time hypotheses and end-time hypotheses. We also employ a story-wide exact match metric, which is the percentage of stories with all its related hypotheses answered correctly.
In addition to TRACIE's standard i.i.d. split, we propose a pruned version of the training set with balanced prior distributions. For example, in the i.i.d. training set,  $70\%$  of the examples with the comparator ends and relation after are positive. We randomly remove instances from the majority classes to produce a uniform-prior training set such that a model can no longer rely on such prior distributions. We believe this setting better evaluates a system's true understanding of the task.
To ensure that tanh returns a value close to 1 or -1, we multiply the distance by a big number denoted as  $\mathrm{INT}_{\text{max}}$ .
<sup>8</sup>We release the systems for reproduction at http://cogcomp.org/page/publication_view/937
System	Start	End	All	Story
Majority	57.3	69.8	64.1	18.1
BiLSTM	53.7	63.5	59.1	10.9
Roberta-Large	78.5	78.3	78.4	26.1
T5-3B	79.4	77.4	78.3	26.9
BaseLM (T5-large)	75.5	75.4	75.4	22.6
BaseLM-MATRES	76.7	76.3	76.5	25.3
PTNTime (ours)	81.4	77.5	79.3	31.0
SYNTAX (ours)	82.1	79.4	80.6	32.0
SYNTAX-ZEROSHOT	77.0	73.1	74.9	21.6
Table 1: Performance on TRACIE, best numbers in bold. BaseLM is T5-large; Story is the percentage of story-wide exact match; Majority is based on the comparator and temporal-relation distribution; Zeroshot uses no TRACIE instance as supervision.
System	Start	End	All	ΔAll
Random	50.0	50.0	50.0	-14.1
BiLSTM	50.5	51.2	50.9	-8.2
Roberta-Large	75.1	68.1	71.3	-7.1
T5-3B	72.8	68.6	70.5	-7.8
BaseLM (T5-large)	68.1	67.8	67.9	-7.5
BaseLM-MATRES	76.3	69.9	72.8	-3.7
PTNTime (ours)	80.6	73.2	76.6	-2.7
SYNTAX (ours)	81.2	77.0	78.9	-1.7
SYNTAX-ZEROSHOT	77.0	73.1	74.9	0.0
Table 1 shows system performance on TRACIE's i.i.d. setting. We observe that PTNTIME improves on all metrics over the base language model, with  $6\%$  on start-time comparisons and  $8\%$  on storywide exact match. It also outperforms BaseLM-MATRES, suggesting that distant supervision is more efficient than extensive human annotation.
With a symbolic end-time inference, SYMTIME further improves on all metrics, with  $7\%$ ,  $4\%$ , and  $9\%$  gains over the base language model on start time, end time and story-wide exact match, respectively. SYMTIME can further improve the performance on start-time hypotheses over PTNTIME even though they use the same model to predict start-time queries. This is because PTNTIME is not designed to understand end time from pre-training, and fine-tuning on such data hurts its representation in general. This illustrates the benefits of models using explicit and sensible reasoning processes.
Table 2 compares systems in the uniform-prior training setting. Compared to the setting in Table 1,
Table 2: Performance on TRACIE uniform-prior training setting.  $\Delta$  All compares the difference with Table 1; Majority is equivalent to random guessing.
System	OT-NS	OT	OT-MS	PT
Wang et al. (2020)	85.9	-	-	-
BaseLM	86.0	87.5	77.4	69.0
SYMTIME	87.3	89.6	86.1	75.1
Table 3: Performance on MATRES. Wang et al. (2020) is not strictly comparable with the rest.
a system cannot exploit prior knowledge about the label distribution when making predictions. Given this, we see that all baselines produce a much lower performance, e.g., the BiLSTM, which is a model that lacks much of the pre-requisite knowledge for reasoning, suddenly performs near random chance. Compared to the baseline models, PTNTIME only drops  $2.7\%$ , suggesting that it is more invariant to evaluation settings and better understands temporal common sense. SYMTIME has the smallest drop among all models  $(1.7\%)$  because of its explicit reasoning process on end-time hypotheses. SYMTIME-ZEROSHOT does not use any TRACIE training examples, so it has the same performance in the uniform-prior setting which outperforms all supervised baselines including T5-3B.
6.4 Extrinsic Evaluation
To show that our model is not limited to the TRACIE dataset and is general in temporal relation reasoning, we also evaluate on MATRES (Ning et al., 2018b), a temporal relation dataset focused on comparing explicit events' start times. We train and evaluate only the instances with a label of either "before" or "after", which accounts for about  $80\%$  of all instances. We compare the performance of SYMTime with BaseLM. We report four results - OT-NS (original test, no story): train and test with only the sentences containing the trigger verbs; OT: train and test with the entire document (down-sampled to be below the maximum sequence length) as an auxiliary input; OT-MS (original test, minimal supervision): train with  $1.2k$  ( $6\%$ ) training instances; PT (perturbed test): train with the complete training set and test on a perturbed test set from Gardner et al. (2020). In OT-NS, we also report a SOTA system from Wang et al. (2020) under the same two-label setting.
Table 3 shows the performance of our model and the baselines. We see that our model is consistently
9This is virtually the same as using PTNTIME as MATRES does not evaluate duration nor end times.
10 Wang et al. (2020) is trained with two additional labels. We constraint the output space to only "before" and "after" using argmax, but this process makes it not directly comparable.
Sys.	BaseLM	PTNTIME	SYMTIME	Human
Acc.	52.6	72.2	75.3	82.5
Table 4: Performance on no-story TRACIE under the uniform-prior training setting.
Sys.	PTNTIME	cross-sentence	within-sentence
Acc.	80.6	79.9	63.7
Table 5: Comparison of pre-training data sources on TRACIE's start time prediction accuracy, under the uniform-prior training setting.
better than BaseLM, and at the same time, comparable to Wang et al. (2020). Our model benefits more from input contexts, and only drops  $4\%$  in the OT-MS setting with minimal supervision (from 89.6 to 86.1), comparing to the  $10\%$  drop from T5-Large. This shows the effectiveness of our distant signals in  $\S 4.1$ , which are also designed to encourage contextual understandings.
6.5 Ablation Studies and Analysis
To better understand the improvements from our models, we conduct several ablation studies.
Table 4 shows the results on TRACIE where the story is not provided as part of the inputs to systems (a no-story setting). While such a setting bares some resemblance to the partial-input baselines often employed in TE (Poliak et al., 2018), in our setting, it is often possible to predict temporal relations in the absence of stories because of strong commonsense priors. Indeed, we estimate that  $65\%$  of the instances can be correctly predicted from the hypotheses alone, based on expert analysis in  $\S 3.2$ . This suggests a  $82.5\%$  human upper-bound $^{11}$  in this no-story setting. Hence, such a setting partly evaluates a model's ability to incorporate commonsense priors when making decisions.
We see that BaseLM is close to random chance, whereas PTNTIME and SYMTIME improve  $20\%$  and  $22\%$  respectively. This suggests that our models better understand temporal common sense through the distant supervision on both start times and duration. On the other hand, we observe much smaller drops in our model's performances in this no-story setting. This suggests that our models do not improve as much on the  $35\%$  instances that require multi-hop timeline constructions over more than two events, motivating future work.
Table 5 compares the two pre-training sources
described in §4.1 by individually pre-training two models with only within-sentence or cross-sentence extracted data. We see that the cross-sentence extraction brings the most performance gain on TRACIE's start-time binary metric under the uniform-prior training setting. This suggests that the global extraction rule is able to introduce new knowledge that is not seen in localized language model pretraining. Combining the within-sentence data further improves the performance.
Through analysis on the interval predictions made by SYMTIME, we notice a tendency for the model to predict "after" for end-time instances, possibly due to overly-estimated durations: a byproduct of natural biases in text. Given the weak signal used to learn such intervals and these potential biases, this is not altogether surprising. We leave the task of learning more robust and faithful interval representations for future work.
We introduce a challenging dataset TRACIE, to evaluate systems' temporal understanding of implicit events. We propose a distant supervision process that improves language models' understanding of start times of both explicit and implicit events. We further combine this process with a distantly supervised model that estimates events' duration to compare event end times, under the explicit rule that end times are start times plus durations. We show that our model improves over TRACIE and MATRES, suggesting the effectiveness of high-precision pre-training and symbolic temporal reasoning. Despite these advances, TRACIE continues to be a challenging task for future work on general temporal reasoning.
This research is based upon work supported in part by the office of the Director of National Intelligence (ODNI), Intelligence Advanced Research Projects Activity (IARPA), via IARPA Contract No. 2019-19051600006 under the BETTER Program, and by Contract FA8750-19-2-1004 with the US Defense Advanced Research Projects Agency (DARPA). The views expressed are those of the authors and do not reflect the official policy or position of the Department of Defense or the U.S. Government.
<sup>11</sup>We assume that the remaining  $35\%$  non-predictable instances are decided by random guessing.
James F Allen. 1983. Maintaining knowledge about temporal intervals. Communications of the ACM, 26(11):832-843.
Prithviraj Ammanabrolu, Wesley Cheung, William Broniec, and Mark O Riedl. 2021. Automated Storytelling via Causal, Commonsense Plot Ordering. In AAAI.
Jacob Andreas, Marcus Rohrbach, Trevor Darrell, and D. Klein. 2016. Neural module networks. CVPR.
Steven Bethard, James H. Martin, and Sara Klingenstein. 2007. Timelines from Text: Identification of Syntactic Temporal Relations. ICSC.
Chandra Bhagavatula, Ronan Le Bras, Chaitanya Malaviya, Keisuke Sakaguchi, Ari Holtzman, Hannah Rashkin, Doug Downey, Scott Wen-tau Yih, and Yejin Choi. 2020. Abductive Commonsense Reasoning. In ICLR.
Taylor Cassidy, Bill McDowell, Nathanel Chambers, and Steven Bethard. 2014. An Annotation Framework for Dense Event Ordering. In ACL.
Quang Do, Wei Lu, and D. Roth. 2012. Joint Inference for Event Timeline Construction. In EMNLP-CoNLL.
Matt Gardner, Yoav Artzi, Victoria Basmova, Jonathan Berant, Ben Bogin, Sihao Chen, Pradeep Dasigi, Dheeru Dua, Yanai Elazar, Ananth Gottumukkala, Nitish Gupta, Hanna Hajishirzi, Gabriel Ilharco, Daniel Khashabi, Kevin Lin, Jiangming Liu, Nelson F. Liu, Phoebe Mulcaire, Qiang Ning, Sameer Singh, Noah A. Smith, Sanjay Subramanian, R. Tsarfaty, Eric Wallace, A. Zhang, and Ben Zhou. 2020. Evaluating Models' Local Decision Boundaries via Contrast Sets. In Findings of EMNLP.
Matt Gardner, Joel Grus, Mark Neumann, Oyvind Tafjord, Pradeep Dasigi, Nelson F. Liu, Matthew E. Peters, M. Schmitz, and L. Zettlemoyer. 2018. AllenNLP: A Deep Semantic Natural Language Processing Platform. NLP-OSS, abs/1803.07640.
Alfonso Gerevini and Lenthart Schubert. 1995. Efficient algorithms for qualitative reasoning about time. Artificial intelligence, 74(2):207-248.
Nitish Gupta, Kevin Lin, Dan Roth, Sameer Singh, and Matt Gardner. 2019. Neural Module Networks for Reasoning over Text. In ICLR.
Andrey Gusev, Nathanael Chambers, Divye Raj Khilnani, Pranav Khaitan, Steven Bethard, and Dan Jurafsky. 2011. Using Query Patterns to Learn the Duration of Events. In IWCS.
Rujun Han, Qiang Ning, and Nanyun Peng. 2019. Joint Event and Temporal Relation Extraction with Shared Representations and Structured Prediction. In EMNLP.
Daniel Khashabi, Tushar Khot, Ashish Sabharwal, and Dan Roth. 2018. Question Answering as Global Reasoning over Semantic Abstractions. In AAAI.
Tushar Khot, Daniel Khashabi, Kyle Richardson, Peter Clark, and Ashish Sabharwal. 2021. Text Modular Networks: Learning to Decompose Tasks in the Language of Existing Models. In *NAACL*.
Shibamouli Lahiri. 2014. Complexity of Word Collocation Networks: A Preliminary Structural Analysis. In Proceedings of ACL-SRW.
Alice Lai, Yonatan Bisk, and Julia Hockenmaier. 2017. Natural Language Inference from Multiple Premises. IJCNLP.
A. Leeuwenberg and Marie-Francine Moens. 2018. Temporal information extraction by predicting relative time-lines. In EMNLP.
Tao Li, Vivek Gupta, Maitrey Mehta, and V. Srikumar. 2019. A logic-driven framework for consistency of neural models. In EMNLP.
Yinhan Liu, Myle Ott, Naman Goyal, Jingfei Du, Mandar Joshi, Danqi Chen, Omer Levy, Mike Lewis, Luke Zettlemoyer, and Veselin Stoyanov. 2019. RoBERTa: A Robustly Optimized BERT Pretraining Approach. ArXiv, abs/1907.11692.
Yuanliang Meng and Anna Rumshisky. 2018. Context-aware neural model for temporal information extraction. In ACL.
Marie-Francine Moens and A. Leeuwenberg. 2017. Structured learning for temporal relation extraction from clinical records. In EACL.
Nasrin Mostafazadeh, Nathanael Chambers, Xiaodong He, Devi Parikh, Dhruv Batra, Lucy Vanderwende, Pushmeet Kohli, and James Allen. 2016. A Corpus and Evaluation Framework for Deeper Understanding of Commonsense Stories. In NAACL.
Yixin Nie, Adina Williams, Emily Dinan, Mohit Bansal, Jason Weston, and Douwe Kiela. 2020. Adversarial NLI: A New Benchmark for Natural Language Understanding. In ACL.
Qiang Ning, Z. Feng, and D. Roth. 2017. A Structured Learning Approach to Temporal Relation Extraction. In EMNLP.
Qiang Ning, H. Wu, H. Peng, and D. Roth. 2018a. Improving Temporal Relation Extraction with a Globally Acquired Statistical Resource. In NAACL.
Qiang Ning, Hao Wu, Pradeep Dasigi, Dheeru Dua, Matt Gardner, Robert L. Logan IV, and Zhen Nie. 2020a. Easy, Reproducible and Quality-Controlled Data Collection with CROWDAQ. In EMNLP.
Qiang Ning, Hao Wu, Rujun Han, Nanyun Peng, Matt Gardner, and Dan Roth. 2020b. Torque: A reading comprehension dataset of temporal ordering questions. In EMNLP.
Qiang Ning, Hao Wu, and Dan Roth. 2018b. A multi-axis annotation scheme for event temporal relations. In ACL.
Qiang Ning, Ben Zhou, Z. Feng, H. Peng, and D. Roth. 2018c. CogCompTime: A Tool for Understanding Time in Natural Language. In EMNLP.
Tim O'Gorman, Kristin Wright-Bettner, and Martha Palmer. 2016. Richer event description: Integrating event coreference with temporal, causal and bridging annotation. In CNS.
Feng Pan, Ritu Mulkar-Mehta, and Jerry R Hobbs. 2006. Extending TimeML with Typical Durations of Events. In Proceedings of the Workshop on Annotating and Reasoning about Time and Events.
J. Pearl. 2009. Causal inference in statistics: An overview. Statistics Surveys, 3:96-146.
Adam Poliak, Jason Naradowsky, Aparajita Haldar, Rachel Rudinger, and Benjamin Van Durme. 2018. Hypothesis only Baselines in Natural Language Inference. Proceedings of *SEM.
James Pustejovsky, José M Castano, Robert Ingria, Roser Sauri, Robert J Gaizauskas, Andrea Setzer, Graham Katz, and Dragomir R Radev. 2003. TimeM: Robust Specification of Event and Temporal Expressions in Text. In New Directions in Question Answering.
Colin Raffel, Noam Shazeer, Adam Roberts, Katherine Lee, Sharan Narang, Michael Matena, Yanqi Zhou, Wei Li, and Peter J Liu. 2020. Exploring the Limits of Transfer Learning with a Unified Text-to-text Transformer. JMLR, 21(140):1-67.
Nils Reimers, Nazanin Dehghani, and Iryna Gurevych. 2016. Temporal Anchoring of Events for the Time-bank corpus. In ACL.
Alon Talmor and Jonathan Berant. 2018. The Web as a Knowledge-base for Answering Complex Questions. In NAACL.
Niket Tandon, Bhavana Dalvi, Joel Grus, Wen tau Yih, Antoine Bosselut, and Peter Clark. 2018. Reasoning about actions and state changes by injecting commonsense knowledge. ArXiv, abs/1808.10012.
Niket Tandon, Keisuke Sakaguchi, Bhavana Dalvi Mishra, Dheeraj Rajagopal, Peter Clark, Michal Guerquin, Kyle Richardson, and Eduard Hovy. 2020. A dataset for tracking entities in open domain procedural text. In EMNLP.
Siddharth Vashishtha, Adam Poliak, Yash Kumar Lal, Benjamin Van Durme, and Aaron Steven White. 2020. Temporal Reasoning in Natural Language Inference. In Finding of EMNLP.
Siddharth Vashishtha, Benjamin Van Durme, and Aaron Steven White. 2019. Fine-grained Temporal Relation Extraction. In ACL.