SlowGuess/NAACL_2021_OCR huggingface dataset:
Ben Zhou $^{1,2}$  Kyle Richardson $^{1}$  Qiang Ning $^{3}$  Tushar Khot $^{1}$  Ashish Sabharwal $^{1}$  Dan Roth $^{2}$
*1Allen Institute for AI
$^{2}$ University of Pennsylvania
3Amazon
{kyler,tushark,ashishs}@allenai.org
{xyzhou,danroth}@cis.upenn.edu qning@amazon.com
We propose TRACIE, a novel temporal reasoning dataset that evaluates the degree to which systems understand implicit events—events that are not mentioned explicitly in natural language text but can be inferred from it. This introduces a new challenge in temporal reasoning research, where prior work has focused on explicitly mentioned events. Human readers can infer implicit events via commonsense reasoning, resulting in a more comprehensive understanding of the situation and, consequently, better reasoning about time. We find, however, that state-of-the-art models struggle when predicting temporal relationships between implicit and explicit events. To address this, we propose a neuro-symbolic temporal reasoning model, SYMTIME, which exploits distant supervision signals from large-scale text and uses temporal rules to combine start times and durations to infer end times. SYMTIME outperforms strong baseline systems on TRACIE by  $5\%$ , and by  $11\%$  in a zero prior knowledge training setting. Our approach also generalizes to other temporal reasoning tasks, as evidenced by a gain of  $1\%-9\%$  on MATRES, an explicit event benchmark.
Understanding temporal relations between events in narrative text is a crucial part of text understanding. When reading a story, a human can construct a latent timeline about events' start and end times, similar to the one shown in Fig. 1 about an automobile accident. This timeline not only contains the placements of explicitly mentioned events (e.g., ride a bicycle), but also accounts for implicit events (e.g., Farrah was distracted so she looked away). Such a latent timeline explains the dynamics between events; for example, the possible chain of events between ride and recovered in this context
Figure 1: A story, its latent timeline, and example TRACIE instances from it. For simplicity, events are shortened to single verbs and the timeline is exaggerated.
contains get hit and injured. The ability to construct such a timeline is essential for understanding the causal dynamics of a situation. Without it, NLP systems cannot truly understand situations and reliably solve tasks such as temporal question-answering, causal inference, and scheduling assistance.
To better evaluate this ability, we introduce a new dataset called TRACIE (TempoRAL Closure InfErence) that focuses on temporal relations on implicit events in short stories. Our dataset contains high-quality annotations of both start and end time queries that test a system's understanding of the full temporal closure (i.e., both start and end time) of events. As a task that requires considerable commonsense knowledge, we follow Zhou et al. (2020) in minimizing the size of the training set, therefore making TRACIE mainly an evaluation set. The final TRACIE dataset contains a total of 5.4k human-curated instances, provided in a (multi-premise) textual entailment (TE) format, as illustrated at the bottom of Fig 1. A Pre-trained language model such as T5-Large (Raffel et al., 2020)
*Most of the work was done when the third author was employed at the Allen Institute for AI and the first author was an intern there.
Proceedings of the 2021 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, pages 1361-1371 June 6-11, 2021. ©2021 Association for Computational Linguistics
fine-tuned on our new dataset achieves a modest binary prediction accuracy of  $67.9\%$ . Consistent with other studies on temporal reasoning (Zhou et al., 2020), these results reveal serious limitations in existing pre-trained language models.
To build models better capable of understanding time with minimal direct training data, we propose a novel distant supervision technique that improves generalization by extracting temporal patterns in large-scale free text as part of an additional pretraining step. In contrast to other attempts at extracting temporal data through patterns at a sentence level (Gusev et al., 2011; Zhou et al., 2020), we extract over large windows of text such as paragraphs. This allows for capturing global information related to multiple events and extracting signals that do not appear in small-window local contexts. The resulting model, PTNTIME (PatternTime), achieves a  $76.6\%$  accuracy on TRACIE, a  $9\%$  gain over using standard T5-Large. We also show the applicability of PTNTIME on a standard temporal reasoning benchmark involving only explicit events, MATRES (Ning et al., 2018b), with a 9 point gain in a low-resource setting.
We achieve further improvements by coupling PTNTIME with a duration model from Zhou et al. (2020) to create a neural-symbolic reasoning model called SYMTIME. The key idea in SYMTIME is to decompose the computation of temporal relations to the predictions of relative distances between start times and those of durations. For example, in Fig 1, we can decide that distracted likely ends before try starts because the duration of distracted is likely to be shorter than the distance between the two start times. This allows for better prediction on the end time, which rarely appears in the natural text and has been previously shown to be difficult to annotate (Ning et al., 2018b). Such a symbolic computation involves a logical combination of the individual models in a way that formalizes part of the Allen interval algebra (Allen, 1983). This model, which supports a wider range of temporal computation and can be used with and without task-specific supervision, achieves a final accuracy of  $78.9\%$  on TRACIE's binary classification metric. We also show that SYMTIME is more robust to different distributions of the training data, demonstrating the benefits of using a temporal model with a transparent reasoning process.
In summary, we make the following 3 contributions: (1) a temporal relation dataset TRACIE focusing on implicit events ( $\S 3$ ); (2) a distant supervision process for temporal understanding of implicit events ( $\S 4$ ); and (3) a reasoning model that makes end-time comparisons using predictions of start-time distances and durations ( $\S 5$ ). Finally, we demonstrate the effectiveness of our models on TRACIE, as well as the applicability of our approach to an existing temporal benchmark ( $\S 6$ ).
Temporal reasoning has received much attention in the NLP community, and to date, there are many datasets that focus on temporal ordering (Pustejovsky et al., 2003; Bethard et al., 2007; Cassidy et al., 2014; Reimers et al., 2016; O'Gorman et al., 2016; Ning et al., 2018b, 2020b), and other temporal knowledge (Pan et al., 2006; Zhou et al., 2019). We focus here on modeling implicit events, which has received relatively little attention. Multiple systems have been proposed as part of research into temporal ordering (Do et al., 2012; Moens and Leeuwenberg, 2017; Leeuwenberg and Moens, 2018; Meng and Rumshisky, 2018; Ning et al., 2018c; Han et al., 2019), duration prediction (Vashishtha et al., 2019) and other tasks. Our decision to use a textual entailment style follows recent work on natural language inference (Williams et al., 2017; Nie et al., 2020; Bhagavatula et al., 2020), which tends to not focus on time (for recent work on temporal NLI, see Vashishtha et al. (2020)). Many have used distant supervision for temporal reasoning (Gusev et al., 2011; Ning et al., 2018a; Zhou et al., 2020). Comparatively, our work captures longer-range dependencies in narrative text (for related ideas, see Ammanabrolu et al. (2021)).
We are inspired by structural predictions and constraints that combat the sparsity of temporal knowledge (Ning et al., 2017; Do et al., 2012), as well as neural module networks (Andreas et al., 2016; Gupta et al., 2019) and other decomposition-based approaches (Talmor and Berant, 2018; Khashabi et al., 2018; Li et al., 2019; Wolfson et al., 2020; Khot et al., 2021). In particular, we build neural-symbolic transformer models that operationalize some of the classical interval-based computations used in earlier work on temporal reasoning (Allen, 1983; Gerevini and Schubert, 1995) (for related ideas, compare with Leeuwenberg and Moens (2018); Vashishtha et al. (2019)).
<sup>1</sup>The same model achieves 77.4% on MATRES (Ning et al., 2018b) with a similar amount of training instances. All TRACIE numbers reported in this section are from Table 2.
Context Story (Premise)	Hypothesis	Inference Label
Tom needed to get braces. He was afraid of them. The dentist assured him everything would be fine. Tom had them on for a while. Once removed he felt it was worth it.	Tom avoids foods he can’t eat with braces starts before the braces are removed.	entailment
We were all watching Spongebob as a family. It is a kid’s show but all really enjoyed it. This one episode was especially funny for the adults. It has humor in it that is funny for kids and adults. It is something we can all watch...	The adults laughed at the jokes ends before we watch Spongebob as a family	contradiction
I was throwing the baseball with my son. He threw one past me that landed in the lake. I reached in to get the ball. I lost my balance and fell in. I got the ball and a bath all in one shot!	The ball was in the boys hand starts after he reached for the ball	contradiction
Figure 2: Example TRACIE instances. The comparator  $l \in \{ \text{starts, ends} \}$  and relation  $r \in \{ \text{before, after} \}$  in each hypothesis are highlighted, in addition to the corresponding explicit event from the story.
This work is broadly related to works on causal dynamics (Pearl, 2009). The nature of combined temporal and causal focuses is also related to procedural text modeling (Tandon et al., 2018, 2020).
3 The TRACIE Dataset
In this section, we introduce the TRACIE dataset.2
3.1 Task Overview and Dataset Construction
The goal of TRACIE is to test a system's ability to compare start and end times of non-extractive implicit event phrases instead of extractive triggers from the context. Such tests in TRACIE take the form of multi-premise textual entailment (TE) (Lai et al., 2017). Each TRACIE instance contains 1) a context story (or premise) consisting of a sequence of explicit narrative events; 2) an implicit event in the form of a natural language phrase that is unmentioned but has some role in the story; 3) a comparator of either {starts, ends}; 4) an explicit event also in the form of a phrase, and 5) a temporal relation of either {before, after} that marks the relationship in the dimension defined by the comparator between the implicit-event and the explicit-event. With these 4 components, we are able to generate TE-style instances, using the context story as the premise and temporal queries about pair-wise relations between implicit and explicit events as hypotheses. For example, in the first positive instance shown in Fig. 1, "distracted" is the implicit-event, "starts" is the comparator, "try" is explicit-event and "before" is the temporal-relation. They form a positive hypothesis "distracted starts before try." We flip the temporal-relation (i.e., "before" to "after" and vice versa) to create negative
Illustration	Allen's Relation	Tracie's Relation
Precedes, Meets	Starts Before Ends Before
Overlaps, Finished-by, Contains, Starts, Equals, Started-by	Starts Before Ends After
During, Finishes, Overlapped-by, Met-by, Preceded-by	Starts After Ends After
Figure 3: TRACIE's label definition and its relation to Allen's interval algebra, with a graph illustration between an implicit event and an explicit event.
(contradiction) instances, as shown in the second example instance in Fig. 1.
Since the start times of explicit-events are more obvious to human annotators, we use them as reference points and compare the implicit-event's start or end time with them (depending on the comparator), according to the label definitions shown in Fig. 3. In rare cases where two time points are the same (e.g., hit and get hit start at the same time in Fig.1), we use the causal relation to decide the order, so that hit starts before get hit. Such instances are created through a multi-stage annotation process as detailed (in respective order) below. All steps are implemented with the CrowdAQ platform (Ning et al., 2020a) with qualification exams.
Implicit Event Generation We randomly sample short stories from the ROCStories dataset (Mostafazadeh et al., 2016). For each story, one annotator writes 5 implicit event phrases that are not explicitly mentioned by the given story, but are inferable and relevant. The annotator additionally rewrites two explicit events closest to the implicit event's start and end time, respectively. With these two events, we can build two TRACIE instances (minus the temporal-relation) per implicit event, which accounts for 10 instances in total per story.
2We release TRACIE and its leaderboard at https:// leaderboard.allenai.org/tracie 3All event phrases are shortened to triggers here for simplicity. See Fig. 2 for actual phrases.
Automatic Instance Generation We use AllenNLP (Gardner et al., 2018) to extract all verbs and relevant arguments with its semantic role labeling (SRL) model. With all the verbs and their arguments, we construct a pool of explicit events in the form of short phrases. For each implicit event, we randomly select two {explicit-event, comparator} pairs from the pool and build 10 additional instances (without temporal-relation).
Label Collection For each of the 20 instances per story, we annotate the temporal-relation with four different annotators. Annotators follow the label definition in §3.1 to produce four temporal-relations for each instance. We use the majority agreement as the final label and filter out unagreeable instances. Two authors additionally verify the instances with ambiguous verbs (e.g., "have") and corrected  $5\%$  of the end-time instances.
3.2 Splits and Analysis
We split the data under the independent and identically distributed (i.i.d.) assumption based on stories, with a 20/80 train/test ratio. We use a small training set, following Zhou et al. (2019), as we believe temporal relations involve much commonsense knowledge. As we later show in §6.3, it is infeasible to collect a large enough human-annotated training set to capture all the knowledge needed to tackle this problem completely, and a system must acquire knowledge from external resources. As a result, we use a small training set just to define the task, and at the same time, use an extensive testing set for more robust evaluation.
The authors conduct a human upper-bound analysis on 100 randomly sampled instances, following the procedure in Zhou et al. (2020). There is a  $94\%$  agreement and a  $98\%$  resolved accuracy, suggesting that TRACIE has a high annotation quality.
4 Pattern-Based Pre-Training
As argued in §3.2, we believe that it is more efficient to build a model that learns the prior knowledge needed for the task with distant signals and only subsequently learns the task definition through a small training set. This section describes how we collect the distant signals related to events' start-time comparisons and pre-train a novel temporally-aware transformer model called PTNTIME. While PTNTIME will be used for fine-tuning directly on
TRACIE, it will also form the basis of a more general temporal reasoning model called SYMTIME that we describe in §5.
4.1 Distant Supervision Collection
We describe the sources of distant supervision signals with the goal of understanding the relative order between two events' start times as well as the relative distance between them.
Figure 4: Extraction for start-time comparisons applied to an example paragraph.
Within-Sentence Extraction We collect start time comparisons between pairs of events heuristically from free-text using "before/after" keywords (following much prior work in temporal modeling and extraction (Do et al., 2012)). We use AllenNLP's SRL model to process each input sentence and find verbs with a temporal argument that starts with either "before" or "after", and contains at least another verb. If there are multiple verbs in the temporal argument, we take the one with the largest number of tokens as arguments. We match the two extracted verbs with the relation indicated by the first word of either "before" or "after". As the example in Fig. 4 shows, the extractor identifies that purchase food is before go to park as indicated by the "before" keyword mentioned in the text. We acquire 2.8 million instances from the May 2020 Wikipedia dump using this process.
Cross-Sentence Extraction The data collected from the within-sentence patterns does not reveal the relative distance between two start times. In addition, because writers often save trivial inferences for efficiency, certain event pairs rarely co-occur within a small textual window, making one event often implicit to the other one in these pairs. To better collect such signals, we employ a cross-sentence extraction that finds direct temporal expressions of hours and dates. Because these temporal expressions (e.g., 2021-01-01) are globally comparable,
4This is obtained after the authors discuss and resolve any disagreements before comparing with the annotated labels.
the compared events can be anywhere in a document. Therefore, this process collects more supervision signals about time-point comparisons and their relative distance on event pairs with trivial causal relations. We apply the SRL model and find all temporal arguments and their associated verbs. We find the exact temporal values by filling unmentioned elements of a temporal expression with the nearest previous mention (e.g., we add "January" to the expression of "the 10th" in Fig. 4.) These extractions have high precision, as the SRL model does well on identifying temporal arguments.
We then construct supervision instances under the assumption that the extracted temporal expressions describe the start times of the associated verbs (e.g., went started on January  $1^{st}$  in Fig. 4). Each instance comprises an event pair, a temporal relation, and an estimation on the temporal difference between the two start times. Each event is a phrase constructed by taking all relevant arguments of the predicate verb in the SRL parses. We represent the differences between the two start times as one of seven coarse temporal units:  $\{\leq$  minutes, hours, days, weeks, months, years,  $\geq$  decades\}. For example, we get go to park is weeks before write review as shown in Fig. 4. In addition to the event pairs, we randomly sample sentences within the paragraph to use as the context that better defines the events. We collect 700k instances from this cross-sentence extraction process from Wikipedia.
Language Model (LM) Pre-Training Data We couple the specialized temporal pre-training data described above with additional paragraphs that are used to perform conventional language model pretraining using the original denoising task proposed in Raffel et al. (2020). This is done to maintain part of the original language model's semantics and to avoid overfitting. We use the Gutenberg Dataset (Lahiri, 2014) as the source and collect 1 million paragraphs for this purpose.
Data Format We then format the within / cross-sentence extraction data to consistent instances that have input sequences of event: [EventA] starts [Relation] [EventB].story: [Paragraph] and output sequences of answer: [Label] [Distance]. Here [EventA] represents the tokens that describe the first event; [EventB] represents the ones that describe the second event; and [Paragraph] represents the tokens of the context, which is non-empty only for cross-sentence extractions. [Relation]
is either before or after, and [Label] is either positive or negative. When the label is positive, the relation will be the gold relation extracted from the text; when it is negative, the relation will be the inverse of the extracted relation. We randomly make  $50\%$  of the instances negative. [Distance] is one of the 7 coarse temporal units represented with a set of blank tokens [extra_id_N]. We leave it to be blank for the within-sentence extractions so that the objective function will not include it in loss computations. The LM pre-training data follows the original format in Raffel et al. (2020).
4.2 Pattern-Based Temporal Model (PTNTIME)
We use a pre-trained sequence-to-sequence model as our base model and additionally pre-train this model using the data collected in §4.1 (for modeling details, see §6.1). We call the resulting model PTNTIME. As a result of this additional pre-training step, PTNTIME serves as a new set of temporally-aware model weights that can be used in place of existing pre-trained models and finetuned on TRACIE. As we describe next, we also use PTNTIME to build a modular temporal reasoning model called SYMTIME that attempts to go beyond a standard language modeling approach and improve start and end point prediction.
5 Symbolic Temporal Reasoning Model (SYMTIME)
To address the challenge of predicting event end times for which it is difficult to obtain high-quality direct or distant supervision, we introduce a new reasoning model called SYMTIME in this section. This model makes end-time comparisons by symbolically combining start time distance and duration from separate predictions based on some of the components introduced in the previous section. Different from Leeuwenberg and Moens (2018) and Vashishtha et al. (2019), our model does not rely on explicit annotations on timepoints, but only relative comparisons between them.
5.1 Formulation
As described in §3.1, hypotheses in TRACIE make pair-wise comparisons between two events  $e_1$  and  $e_2$  using a comparator  $l$  from {starts, ends} and a query-relation  $r$  from {before, after} based on a provided story context. We associate each  $e_j$  with a latent start time  $\mathbf{start}_j$  and an end
Figure 5: Decomposition of the relation functions that solve TRACIE instances (equal timepoints ignored).
time  $\mathbf{end}_j$ , as well as, for convenience, a duration  $\text{duration}_j = \text{end}_j - \text{start}_j$ . Under this formulation, a symbolic approach to solving TRACIE involves computing the relation functions  $r_l$  shown in Figure 5. For example, given exact numeric values  $\text{end}_1$  and  $\text{start}_2$ , as one would assume in a classical interval-based approach to temporal reasoning (Allen, 1983) $^5$ , determining if the first event ends before the second involves simply computing whether  $\text{end}_1$  is less than  $\text{start}_2$ .
Given that the exact values of start and end times are latent, we use the intervals to do the same comparisons, as they are more context-invariant. For example, we do not need the exact date to know that lunch starts before dinner in the same day, because there is a typical distribution of the relative distance between the two start times. Based on this idea, we build a neural-symbolic model that learns approximations of these simple functions in Fig. 5 in a differentiable way. Specifically, we use individual neural modules that make predictions about event intervals via distance and duration functions  $\mathrm{dist}(e_i,e_j)$  and  $\mathrm{dur}(e_j)$ , respectively.
To understand this decomposition, we define the distance and duration functions computed by these two modules as  $\mathrm{dist}(e_i,e_j) = \mathbf{start}_i - \mathbf{start}_j$  and  $\mathrm{dur}(e_j) = \mathbf{duration}_j$ . By exploiting the rule that an end point  $\mathbf{end}_j$  can be computed as  $\mathbf{end}_j = \mathbf{start}_j + \mathbf{duration}_j$ , we can, for example, decompose the relation  $r_{ends}(e_1,e_2) = \mathbf{before}$  (i.e.,  $e_1$  ends before  $e_2$ ) in terms of our two modules as follows via simple algebraic manipulation: