Venue: NAACL 2021

# Temporal Reasoning on Implicit Events from Distant Supervision

 Ben Zhou\({}^{1,2}\) Kyle Richardson\({}^{1}\) Qiang Ning\({}^{3}\) Tushar Khot\({}^{1}\) Ashish Sabharwal\({}^{1}\) Dan Roth\({}^{2}\)

\({}^{*}\)Mosti of the work was done when the third author was employed at the Allen Institute for AI and the first author was an intern there. \({}^{*}\)Most of the work was done when the third author was employed at the Allen Institute for AI and the first author was an intern there. \({}^{*}\)Allen Institute for AI \({}^{2}\)University of Pennsylvania \({}^{3}\)Amazon {kyler,tushark,ashisks}@allenai.org {xyzhou,danroth}@cis.upenn.edu qning@amazon.com

###### Abstract

We propose Tracie, a novel temporal reasoning dataset that evaluates the degree to which systems understand _implicit_ events--events that are not mentioned explicitly in natural language text but can be inferred from it. This introduces a new challenge in temporal reasoning research, where prior work has focused on explicitly mentioned events. Human readers can infer implicit events via commonsense reasoning, resulting in a more comprehensive understanding of the situation and, consequently, better reasoning about time. We find, however, that state-of-the-art models struggle when predicting temporal relationships between implicit and explicit events. To address this, we propose a neuro-symbolic temporal reasoning model, SymTime, which exploits distant supervision signals from large-scale text and uses temporal rules to combine start times and durations to infer end times. SynTime outperforms strong baseline systems on Tracie by 5%, and by 11% in a zero prior knowledge training setting. Our approach also generalizes to other temporal reasoning tasks, as evidenced by a gain of 1%-9% on MATRES, an explicit event benchmark.

## 1 Introduction

Understanding temporal relations between events in narrative text is a crucial part of text understanding. When reading a story, a human can construct a latent timeline about events' start and end times, similar to the one shown in Fig. 1 about an automobile accident. This timeline not only contains the placements of explicitly mentioned events (e.g., _ride a bicycle_), but also accounts for implicit events (e.g., Farrah was _distracted_ so she looked away). Such a latent timeline explains the dynamics between events; for example, the possible chain of events between _ride_ and _recovered_ in this context contains _get hit_ and _injured_. The ability to construct such a timeline is essential for understanding the causal dynamics of a situation. Without it, NLP systems cannot truly understand situations and reliably solve tasks such as temporal question-answering, causal inference, and scheduling assistance.

To better evaluate this ability, we introduce a new dataset called Tracie (_TempoRAI Closure InfErence_) that focuses on temporal relations on implicit events in short stories. Our dataset contains high-quality annotations of both start and end time queries that test a system's understanding of the full temporal closure (i.e., both start and end time) of events. As a task that requires considerable commonsense knowledge, we follow Zhou et al. (2020) in minimizing the size of the training set, therefore making Tracie mainly an evaluation set. The final Tracie dataset contains a total of 5.4k human-curated instances, provided in a (multi-premise) textual entailment (TE) format, as illustrated at the bottom of Fig 1. A Pre-trained language model such as T5-Large Raffel et al. (2020)

Figure 1: A story, its latent timeline, and example Tracie instances from it. For simplicity, events are shortened to single verbs and the timeline is exaggerated.



fine-tuned on our new dataset achieves a modest binary prediction accuracy of \(67.9\)%.1 Consistent with other studies on temporal reasoning Zhou et al. (2020), these results reveal serious limitations in existing pre-trained language models.

Footnote 1: The same model achieves 77.4% on MATRES Ning et al. (2018b) with a similar amount of training instances. All Tracie numbers reported in this section are from Table 2.

To build models better capable of understanding time with minimal direct training data, we propose a novel distant supervision technique that improves generalization by extracting temporal patterns in large-scale free text as part of an additional pretraining step. In contrast to other attempts at extracting temporal data through patterns at a sentence level Gusev et al. (2011); Zhou et al. (2020), we extract over large windows of text such as paragraphs. This allows for capturing global information related to multiple events and extracting signals that do not appear in small-window local contexts. The resulting model, PtnTimePatternTime), achieves a \(76.6\)% accuracy on Tracie, a \(9\%\) gain over using standard T5-Large. We also show the applicability of PtnTime on a standard temporal reasoning benchmark involving only explicit events, MATRES Ning et al. (2018b), with a \(9\) point gain in a low-resource setting.

We achieve further improvements by coupling PtnTime with a duration model from Zhou et al. (2020) to create a neural-symbolic reasoning model called SymTime. The key idea in SymTime is to _decompose_ the computation of temporal relations to the predictions of relative distances between start times and those of durations. For example, in Fig 1, we can decide that _distracted_ likely ends before _try_ starts because the duration of _distracted_ is likely to be shorter than the distance between the two start times. This allows for better prediction on the end time, which rarely appears in the natural text and has been previously shown to be difficult to annotate Ning et al. (2018b). Such a symbolic computation involves a logical combination of the individual models in a way that formalizes part of the Allen interval algebra Allen (1983). This model, which supports a wider range of temporal computation and can be used with and without task-specific supervision, achieves a final accuracy of \(78.9\)% on Tracie's binary classification metric. We also show that SymTime is more robust to different distributions of the training data, demonstrating the benefits of using a temporal model with a transparent reasoning process.

In summary, we make the following 3 contributions: (1) a temporal relation dataset Tracie focusing on implicit events SS3); (2) a distant supervision process for temporal understanding of implicit events (SS4); and (3) a reasoning model that makes end-time comparisons using predictions of start-time distances and durations SS5). Finally, we demonstrate the effectiveness of our models on Tracie, as well as the applicability of our approach to an existing temporal benchmark (SS6).

## 2 Related Work

Temporal reasoning has received much attention in the NLP community, and to date, there are many datasets that focus on temporal ordering Pustejovsky et al. (2003); Bethard et al. (2007); Cassidy et al. (2014); Reimers et al. (2016); O'Gorman et al. (2016); Ning et al. (2018b, 2020), and other temporal knowledge Pan et al. (2006); Zhou et al. (2019). We focus here on modeling implicit events, which has received relatively little attention. Multiple systems have been proposed as part of research into temporal ordering Do et al. (2012); Moens and Leeuwenberg (2017); Leenuenberg and Moens (2018); Meng and Rumshisky (2018); Ning et al. (2018c); Han et al. (2019), duration prediction Vashishtha et al. (2019) and other tasks. Our decision to use a textual entailment style follows recent work on natural language inference Williams et al. (2017); Nie et al. (2020); Bhagavatula et al. (2020), which tends to not focus on time (for recent work on temporal NLI, see Vashishtha et al. (2020)). Many have used distant supervision for temporal reasoning Gusev et al. (2011); Ning et al. (2018a); Zhou et al. (2020). Comparatively, our work captures longer-range dependencies in narrative text (for related ideas, see Ammanabrolu et al. (2021)).

We are inspired by structural predictions and constraints that combat the sparsity of temporal knowledge Ning et al. (2017); Do et al. (2012), as well as neural module networks Andreas et al. (2016); Gupta et al. (2019) and other decomposition-based approaches Talmor and Berant (2018); Khashabi et al. (2018); Li et al. (2019); Wolfson et al. (2020); Khot et al. (2021). In particular, we build neural-symbolic transformer models that operationalize some of the classical interval-based computations used in earlier work on temporal reasoning Allen (1983); Gerevini and Schubert (1995) (for related ideas, compare with Leenuenberg and Moens (2018); Vashishtha et al. (2019)).

 

This work is broadly related to works on causal dynamics Pearl (2009). The nature of combined temporal and causal focuses is also related to procedural text modeling Tandon et al. (2018, 2020).

## 3 The TRACIE Dataset

In this section, we introduce the Tracie dataset.2

Footnote 2: We release Tracic and its leaderboard at [https://leaderboard.allenai.org/tracie](https://leaderboard.allenai.org/tracie)

### Task Overview and Dataset Construction

The goal of Tracie is to test a system's ability to compare start and end times of non-extractive implicit event phrases instead of extractive triggers from the context. Such tests in Tracie take the form of multi-premise textual entailment (TE) Lai et al. (2017). Each Tracie instance contains 1) a **context story** (or premise) consisting of a sequence of _explicit_ narrative events; 2) an **implicit event** in the form of a natural language phrase that is unmentioned but has some role in the story; 3) a **comparator** of either {starts,ends}; 4) an **explicit event** also in the form of a phrase, and 5) a **temporal relation** of either {before,after} that marks the relationship in the dimension defined by the _comparator_ between the _implicit-event_ and the _explicit-event_. With these 4 components, we are able to generate TE-style instances, using the context story as the premise and temporal queries about pair-wise relations between implicit and explicit events as hypotheses. For example, in the first positive instance shown in Fig. 1, "distracted" is the _implicit-event_, "starts" is the _comparator_, "try" is _explicit-event_ and "before" is the _temporal-relation_. They form a positive hypothesis "distracted starts before try."3 We flip the _temporal-relation_ (i.e., "before" to "after" and vice versa) to create negative (contradiction) instances, as shown in the second example instance in Fig. 1.

Footnote 3: All event phrases are shortened to triggers here for simplicity. See Fig. 2 for actual phrases.

Since the start times of _explicit-events_ are more obvious to human annotators, we use them as reference points and compare the _implicit-event_'s start or end time with them (depending on the _comparator_), according to the label definitions shown in Fig. 3. In rare cases where two time points are the same (e.g., _hit_ and _get hit_ start at the same time in Fig.1), we use the causal relation to decide the order, so that _hit_ starts before _get hit_. Such instances are created through a multi-stage annotation process as detailed (in respective order) below. All steps are implemented with the CrowdAQ platform Ning et al. (2020) with qualification exams.

Implicit Event GenerationWe randomly sample short stories from the ROCStories dataset Mostafazadeh et al. (2016). For each story, one annotator writes 5 implicit event phrases that are not explicitly mentioned by the given story, but are inferable and relevant. The annotator additionally rewrites two explicit events closest to the implicit event's start and end time, respectively. With these two events, we can build two Tracie instances (minus the _temporal-relation_) per implicit event, which accounts for 10 instances in total per story.

Figure 3: Trace’s label definition and its relation to Allen’s interval algebra, with a graph illustration between an implicit event and an explicit event.

Figure 2: Example Tracie instances. The \(\boldsymbol{\mathsf{comparator}}l\in\){starts,ends} and \(\boldsymbol{\mathsf{relation}}r\in\){before,after} in each hypothesis are highlighted, in addition to the corresponding explicit event from the story.

 

Automatic Instance GenerationWe use AllenNLP Gardner et al. (2018) to extract all verbs and relevant arguments with its semantic role labeling (SRL) model. With all the verbs and their arguments, we construct a pool of explicit events in the form of short phrases. For each implicit event, we randomly select two {_explicit-event_, _comparator_} pairs from the pool and build 10 additional instances (without _temporal-relation_).

Label CollectionFor each of the 20 instances per story, we annotate the _temporal-relation_ with four different annotators. Annotators follow the label definition in SS3.1 to produce four _temporal-relations_ for each instance. We use the majority agreement as the final label and filter out unagreeable instances. Two authors additionally verify the instances with ambiguous verbs (e.g., "have") and corrected \(5\%\) of the end-time instances.

### Splits and Analysis

We split the data under the independent and identically distributed (i.i.d.) assumption based on stories, with a 20/80 train/test ratio. We use a small training set, following Zhou et al. (2019), as we believe temporal relations involve much commonsense knowledge. As we later show in SS6.3, it is infeasible to collect a large enough human-annotated training set to capture all the knowledge needed to tackle this problem completely, and a system must acquire knowledge from external resources. As a result, we use a small training set just to define the task, and at the same time, use an extensive testing set for more robust evaluation.

The authors conduct a human upper-bound analysis on 100 randomly sampled instances, following the procedure in Zhou et al. (2020). There is a \(94\%\) agreement and a \(98\%\) resolved accuracy,4 suggesting that Tracie has a high annotation quality.

Footnote 4: This is obtained after the authors discuss and resolve any disagreements before comparing with the annotated labels.

## 4 Pattern-Based Pre-Training

As argued in SS3.2, we believe that it is more efficient to build a model that learns the prior knowledge needed for the task with distant signals and only subsequently learns the task definition through a small training set. This section describes how we collect the distant signals related to events' start-time comparisons and pre-train a novel _temporally-aware_ transformer model called PtnTime. While PtnTime will be used for fine-tuning directly on Tracie, it will also form the basis of a more general temporal reasoning model called SymTime that we describe in SS5.

### Distant Supervision Collection

We describe the sources of distant supervision signals with the goal of understanding the relative order between two events' start times as well as the relative distance between them.

Within-Sentence ExtractionWe collect start time comparisons between pairs of events heuristically from free-text using "before/after" keywords (following much prior work in temporal modeling and extraction Do et al. (2012)). We use AllenNLP's SRL model to process each input sentence and find verbs with a temporal argument that starts with either "before" or "after", and contains at least another verb. If there are multiple verbs in the temporal argument, we take the one with the largest number of tokens as arguments. We match the two extracted verbs with the relation indicated by the first word of either "before" or "after". As the example in Fig. 4 shows, the extractor identifies that _purchase food_ is before _go to park_ as indicated by the "before" keyword mentioned in the text. We acquire 2.8 million instances from the May 2020 Wikipedia dump using this process.

Cross-Sentence ExtractionThe data collected from the within-sentence patterns does not reveal the relative distance between two start times. In addition, because writers often save trivial inferences for efficiency, certain event pairs rarely co-occur within a small textual window, making one event often implicit to the other one in these pairs. To better collect such signals, we employ a cross-sentence extraction that finds direct temporal expressions of hours and dates. Because these temporal expressions (e.g., 2021-01-01) are globally comparable,

Figure 4: Extraction for start-time comparisons applied to an example paragraph.

 

the compared events can be anywhere in a document. Therefore, this process collects more supervision signals about _time-point comparisons_ and their _relative distance_ on event pairs with trivial causal relations. We apply the SRL model and find all temporal arguments and their associated verbs. We find the exact temporal values by filling unmentioned elements of a temporal expression with the nearest previous mention (e.g., we add "January" to the expression of "the 10th" in Fig. 4.) These extractions have high precision, as the SRL model does well on identifying temporal arguments.

We then construct supervision instances under the assumption that the extracted temporal expressions describe the start times of the associated verbs (e.g., _went_ started on _January_\(1^{st}\) in Fig. 4) . Each instance comprises an event pair, a temporal relation, and an estimation on the temporal difference between the two start times. Each event is a phrase constructed by taking all relevant arguments of the predicate verb in the SRL parses. We represent the differences between the two start times as one of seven coarse temporal units: {\(\leq\)minutes, hours, days, weeks, months, years, \(\geq\)decades}. For example, we get _go to park_ is **weeks** before _write review_ as shown in Fig. 4. In addition to the event pairs, we randomly sample sentences within the paragraph to use as the context that better defines the events. We collect 700k instances from this cross-sentence extraction process from Wikipedia.

Language Model (LM) Pre-Training DataWe couple the specialized temporal pre-training data described above with additional paragraphs that are used to perform conventional language model pre-training using the original denoising task proposed in Raffel et al. (2020). This is done to maintain part of the original language model's semantics and to avoid overfitting. We use the Gutenberg Dataset Lahiri (2014) as the source and collect 1 million paragraphs for this purpose.

Data FormatWe then format the within / cross-sentence extraction data to consistent instances that have input sequences of event:[EventA] starts [Relation][EventB].story:[Paragraph] and output sequences of answer:[Label][Distance]. Here [EventA] represents the tokens that describe the first event; [EventB] represents the ones that describe the second event; and [Paragraph] represents the tokens of the context, which is non-empty only for cross-sentence extractions. [Relation] is either before or after, and [Label] is either positive or negative. When the label is positive, the relation will be the gold relation extracted from the text; when it is negative, the relation will be the inverse of the extracted relation. We randomly make 50% of the instances negative. [Distance] is one of the 7 coarse temporal units represented with a set of blank tokens [extra_id_N]. We leave it to be blank for the within-sentence extractions so that the objective function will not include it in loss computations. The LM pre-training data follows the original format in Raffel et al. (2020).

### Pattern-Based Temporal Model (PtnTime)

We use a pre-trained sequence-to-sequence model as our base model and additionally pre-train this model using the data collected in SS4.1 (for modeling details, see SS6.1). We call the resulting model PtnTime. As a result of this additional pre-training step, PtnTime serves as new set of _temporally-aware_ model weights that can be used in place of existing pre-trained models and fine-tuned on Tracie. As we describe next, we also tune PtnTime to build a modular temporal reasoning model called SymTime that attempts to go beyond a standard language modeling approach and improve start and end point prediction.

## 5 Symbolic Temporal Reasoning Model (SymTime)

To address the challenge of predicting event end times for which it is difficult to obtain high-quality direct or distant supervision, we introduce a new reasoning model called SymTime in this section. This model makes end-time comparisons by symbolically combining start time distance and duration from separate predictions based on some of the components introduced in the previous section. Different from Leeuwenberg and Moens (2018) and Vashishtha et al. (2019), our model does not rely on explicit annotations on timepoints, but only relative comparisons between them.

### Formulation

As described in SS3.1, hypotheses in Tracie make pair-wise comparisons between two events \(e_{1}\) and \(e_{2}\) using a _comparator_\(l\) from {starts,ends} and a _query-relation_\(r\) from {before,after} based on a provided story context. We associate each \(e_{j}\) with a latent start time \(\mathbf{start}_{j}\) and an end 

time \({\bf end}_{j}\), as well as, for convenience, a duration \({\bf duration}_{j}={\bf end}_{j}-{\bf start}_{j}\). Under this formulation, a symbolic approach to solving Tracie involves computing the _relation functions_\(r_{l}\) shown in Figure 5. For example, given exact numeric values \({\bf end}_{1}\) and \({\bf start}_{2}\), as one would assume in a classical interval-based approach to temporal reasoning Allen (1983)5, determining if the first event _ends before_ the second involves simply computing whether \({\bf end}_{1}\) is less than \({\bf start}_{2}\).

Footnote 5: In the Allen algebra, the values \({\bf end}_{x}\) and \({\bf start}_{y}\) correspond to the right and left endpoints \(x^{+},y^{-}\) in the intervals \(\langle x^{-},x^{+}\rangle,(y^{-},y^{+})\). Likewise, our \({\bf duration}_{x}\) corresponds to the value \((x^{+}-x^{-})\).

Given that the exact values of start and end times are latent, we use the intervals to do the same comparisons, as they are more context-invariant. For example, we do not need the exact date to know that _lunch_ starts before _dinner_ in the same day, because there is a typical distribution of the relative distance between the two start times. Based on this idea, we build a neural-symbolic model that learns approximations of these simple functions in Fig. 5 in a differentiable way. Specifically, we use individual neural modules that make predictions about event intervals via distance and duration functions \(\mathrm{dist}(e_{i},e_{j})\) and \(\mathrm{dur}(e_{j})\), respectively.

To understand this decomposition, we define the distance and duration functions computed by these two modules as \(\mathrm{dist}(e_{i},e_{j})={\bf start}_{i}-{\bf start}_{j}\) and \(\mathrm{dur}(e_{j})={\bf duration}_{j}\). By exploiting the rule that an end point \({\bf end}_{j}\) can be computed as \({\bf end}_{j}={\bf start}_{j}+{\bfduration}_{j}\), we can, for example, decompose the relation \(r_{ends}(e_{1},e_{2})={\bf before}\) (i.e., \(e_{1}\)_ends before_\(e_{2}\)) in terms of our two modules as follows via simple algebraic manipulation:

\[r_{ends}(e_{1},e_{2})={\bf before}\] \[\Leftrightarrow {\bf end}_{1}<{\bf start}_{2}\] \[\Leftrightarrow {\bf start}_{1}+{\bf duration}_{1}<{\bf start}_{2}\] \[\Leftrightarrow \big{(}{\bf start}_{1}-{\bf start}_{2}\big{)}+{\bf duration}_{ 1}<0\] \[\Leftrightarrow \mathrm{dist}(e_{1},e_{2})+\mathrm{dur}(e_{1})<0\]

Hence, we have reduced the computation of the relation _ends before_ to a _symbolic computation_ over two numeric intervals. Conversely, we have \(r_{ends}(e_{1},e_{2})={\bf after}\Leftrightarrow~{}\mathrm{dist}(e_{1},e_{2}) +\mathrm{dur}(e_{1})>0\),6 For the **starts** comparator, we have \(r_{stats}(e_{1},e_{2})={\bf before}\Leftrightarrow~{}\mathrm{dist}(e_{1},e_{2})<0\) and vice versa for the **after** relation.

Footnote 6: We note that one drawback of this inference rule is that it does not predict causal relations and, therefore, cannot handle instances where \({\bf end}_{1}={\bf start}_{2}\) as our label definitions describe in §3.1. We leave this problem for future research.

In what follows, we describe how we approximate the values of the two functions via individual neural modules (see illustration in Fig. 6).

### Duration Estimation

To obtain a model to estimate \(\mathrm{dur}(\cdot)\), we pre-train a sequence-to-sequence model with the duration data from Zhou et al. (2020), which is similarly collected from pattern-based extraction. The data contains over 1 million events with their corresponding duration values. We map each instance to an input sequence event:[Event]story:[Story] and a corresponding output sequence answer:[Value], where [Event] represents the tokens of an event with the trigger verb marked by a special token to its left, [Story] represents down-sampled tokens from the context, and [Value] is one of the 7 unit labels as described in SS4.1 (i.e., { \(\leq\)minutes, hours, days, weeks, months, years, \(\geq\)decades }).

### Computation and Learning

We use the output from PtnTime to approximate the function \(\mathrm{dist}(\cdot)\). Following the sequence formulation of PtnTime in SS4, we replace [EventA] with the textual description of \(e_{1}\), [EventB] with

Figure 5: Decomposition of the relation functions that solve Tracie instances (equal timepoints ignored).

Figure 6: A schematic overview of SymTime to compare event \(A\)’s end time with event \(B\)’s start time via modular predictions about \(A\)’s duration and distance from \(B\) and their symbolic combination (bottom).

 

the textual description of \(e_{2}\), and [Paragraph] with the context (premise), and fix [Relation] to be _before_. By taking the values of the vocabulary indices corresponding to "positive" and "negative" from the logits of [Label] and applying a softmax operation, we get \(P_{\text{before}}\) and \(P_{\text{after}}\). These are the probability of \(e_{1}\) starting before and after \(e_{2}\), respectively, and are used to define the vector \(\mathbf{p}=[P_{\text{before}},P_{\text{after}}]\). Similarly, we apply \(\operatorname*{softmax}\) to the logits of [Distance] over the 7 words representing the temporal units to obtain 7 values that approximate the probabilities of the distance between two events' start times being closest to each temporal unit. We place the 7 values in temporal units' increasing order in vector \(\mathbf{d}\). To represent \(|\mathbf{start_{1}-start_{2}}|\) with a single value, we dot product the probabilities with an incremental constant vector \(\mathbf{c}=[0,1,2,3,4,5,6]\). To get the direction, we apply the \(\tanh\) function to the difference between the probabilities in \(\mathbf{p}\).7 As a result, we have:

Footnote 7: To ensure that \(\tanh\) returns a value close to 1 or -1, we multiply the distance by a big number denoted as INT\({}_{max}\).

\[\begin{split}\operatorname*{dist}(\cdot)&=\mathbf{ start_{1}-start_{2}}\\ &=\mathbf{c}^{T}\mathbf{d}*\tanh(\text{INT}_{max}*(\mathbf{p}_{2 }-\mathbf{p}_{1}))\end{split} \tag{1}\]

We use the pre-trained model in SS5.2 to approximate the function \(\mathrm{dur}(\cdot)\). Because the model is pre-trained with markers to the left of trigger verbs, we run a part-of-speech tagger on input phrases and add a marker to the left of the first verb. We apply \(\operatorname*{softmax}\) to the logit values of [Value] over the 7 temporal unit words and get, as above, 7 values representing the probabilities of the input event's duration being closest to each unit. We form **v** by placing these values at the temporal unit's increasing order. With the same constant vector, we have:

\[\mathrm{dur}(\cdot)=\mathbf{duration_{1}}=\mathbf{c}^{T}\textbf{v} \tag{2}\]

For hypotheses with comparator starts, we use PtntTime and its sequence-to-sequence objective to learn (i.e., we take the input hypothesis and context as is and use [Label] directly as the prediction). For hypotheses where the comparator is ends, we use the inference process in SS5.1 and the computation process described above to construct \(logits=[\mathbf{pred},-\mathbf{pred}],\mathbf{pred}=\operatorname*{dist}(e_{1}, e_{2})+\mathrm{dur}(e_{1})\) as detailed in Fig. 6. We find the _gold-temporal-relation_ in each training instance and compute a two-class cross-entropy loss with \(logits\). The PtnTime that predicts starts hypotheses shares weights with the one used in computing \(logits\). The final model SymTime can also be used to predict Tracie instances without any task-specific supervision as the two functions are initialized with distant supervision.

## 6 Experiments

In this section, we detail our experimental setup (SS6.1-6.2) and report our main results (SS6.3-6.5).8

Footnote 8: We release the systems for reproduction at [http://cogcomp.org/page/publication_view/937](http://cogcomp.org/page/publication_view/937)

### Baselines and Systems

We use T5-Large implemented by Wolf et al. (2019) as our base sequence-to-sequence model for both PtntTime and the duration model in SS5.2 as it provides for faster iterations. We use early stopping, batch size of 32 and other default parameters. PtntTime converges after 45k steps (\(\sim\)1.4M instances) and the duration model converges after 80k steps (\(\sim\)2.6M instances). We use these pretrained weights in SymTime as well as SymTimeZeroShot which uses no Tracie supervision.

We compare with our proposed models with a host of baselines based on the same pre-trained language model, including **BaseLM**: T5-Large, and **BaseLM-MATRES**: T5-Large fine-tuned on 20k MATRES training data. We also compare with other architectures/models, including **BiLSTM** as used in Williams et al. (2017), **Roberta-Large**Liu et al. (2019) and **T5-3B**. All models and baselines follow a standard TE setup and default parameters. We report a 3-run average and each model is run until convergence.

### Metrics and Settings

We measure system performance on Tracie separately for start-time hypotheses and end-time hypotheses. We also employ a story-wide exact match metric, which is the percentage of stories with all its related hypotheses answered correctly.

In addition to Tracie's standard i.i.d. split, we propose a pruned version of the training set with balanced prior distributions. For example, in the i.i.d. training set, 70% of the examples with the comparator ends and relation _after_ are positive. We randomly remove instances from the majority classes to produce a uniform-prior training set such that a model can no longer rely on such prior distributions. We believe this setting better evaluates a system's true understanding of the task.

 



### Main Results

Table 1 shows system performance on Tracie's i.i.d. setting. We observe that PttTime improves on all metrics over the base language model, with 6% on start-time comparisons and 8% on storywide exact match. It also outperforms BaseLMMATRES, suggesting that distant supervision is more efficient than extensive human annotation.

With a symbolic end-time inference, SymTime further improves on all metrics, with 7%, 4%, and 9% gains over the base language model on start time, end time and story-wide exact match, respectively. SymTime can further improve the performance on start-time hypotheses over PttTime even though they use the same model to predict start-time queries. This is because PttTime is not designed to understand end time from pre-training, and fine-tuning on such data hurts its representation in general. This illustrates the benefits of models using explicit and sensible reasoning processes.

Table 2 compares systems in the uniform-prior training setting. Compared to the setting in Table 1, a system cannot exploit prior knowledge about the label distribution when making predictions. Given this, we see that all baselines produce a much lower performance, e.g., the BiLSTM, which is a model that lacks much of the pre-requisite knowledge for reasoning, suddenly performs near random chance. Compared to the baseline models, PttTime only drops \(2.7\)%, suggesting that it is more invariant to evaluation settings and better understands temporal common sense. SymTime has the smallest drop among all models (1.7%) because of its explicit reasoning process on end-time hypotheses. SymTime-ZeroShot does not use any Tracie training examples, so it has the same performance in the uniform-prior setting which outperforms all supervised baselines including T5-3B.

### Extrinsic Evaluation

To show that our model is not limited to the Tracie dataset and is general in temporal relation reasoning, we also evaluate on MATRES Ning et al. (2018b), a temporal relation dataset focused on comparing explicit events' start times. We train and evaluate only the instances with a label of either "before" or "after", which accounts for about 80% of all instances. We compare the performance of SymTime9 with BaseLM. We report four results - **OT-NS (original test, no story)**: train and test with only the sentences containing the trigger verbs; **OT**: train and test with the entire document (down-sampled to be below the maximum sequence length) as an auxiliary input; **OT-MS (original test, minimal supervision)**: train with 1.2k (6%) training instances; **PT (perturbed test)**: train with the complete training set and test on a perturbed test set from Gardner et al. (2020). In OT-NS, we also report a SOTA system from Wang et al. (2020) under the same two-label10 setting.

Footnote 9: This is virtually the same as using PttTime as MATRES does not evaluate duration nor end times.

Footnote 10: Wang et al. (2020) is trained with two additional labels. We constraint the output space to only “before” and “after” using argmax, but this process makes it not directly comparable.

Table 3 shows the performance of our model and the baselines. We see that our model is consistently

\begin{table}
\begin{tabular}{l|c c c c} \hline System & Start & End & All & Story \\ \hline Majority & 57.3 & 69.8 & 64.1 & 18.1 \\ BiLSTM & 53.7 & 63.5 & 59.1 & 10.9 \\ \hline Roberta-Large & 78.5 & 78.3 & 78.4 & 26.1 \\ T5-3B & 79.4 & 77.4 & 78.3 & 26.9 \\ \hline BaseLM (T5-large) & 75.5 & 75.4 & 75.4 & 22.6 \\ BaselM-MATRES & 76.7 & 76.3 & 76.5 & 25.3 \\ PythTine (ours) & 81.4 & 77.5 & 79.3 & 31.0 \\ SymTime (ours) & **82.1** & **79.4** & **80.6** & **32.0** \\ \hline SymptTime-ZeroShot & 77.0 & 73.1 & 74.9 & 21.6 \\ \hline \end{tabular}
\end{table}
Table 1: Performance on Tracie, best numbers in **bold**. BaseLM is T5-large; Story is the percentage of story-wide exact match; Majority is based on the comparator and temporal-relation distribution; Zeroshot uses no Tracie instance as supervision.

\begin{table}
\begin{tabular}{l|c|c|c|c} \hline System & GT-NS & DT & OT-MS & PT \\ \hline Wang et al. (2020) & 85.9 & - & - & - \\ BaeLM & 86.0 & 87.5 & 77.4 & 69.0 \\ SymTime & 87.3 & 89.6 & 86.1 & 75.1 \\ \hline \end{tabular}
\end{table}
Table 3: Performance on MATRES. Wang et al. (2020) is not strictly comparable with the rest.



better than BaseLM, and at the same time, comparable to Wang et al. (2020). Our model benefits more from input contexts, and only drops 4% in the OT-MS setting with minimal supervision (from 89.6 to 86.1), comparing to the 10% drop from T5-Large. This shows the effectiveness of our distant signals in SS4.1, which are also designed to encourage contextual understandings.

### Ablation Studies and Analysis

To better understand the improvements from our models, we conduct several ablation studies.

Table 4 shows the results on Tracie where the story is not provided as part of the inputs to systems (a _no-story_ setting). While such a setting bares some resemblance to the _partial-input_ baselines often employed in TE Poliak et al. (2018), in our setting, it is often possible to predict temporal relations in the absence of stories because of strong commonsense priors. Indeed, we estimate that \(65\)% of the instances can be correctly predicted from the hypotheses alone, based on expert analysis in SS 3.2. This suggests a \(82.5\)% human upper-bound11 in this _no-story_ setting. Hence, such a setting partly evaluates a model's ability to incorporate common-sense priors when making decisions.

Footnote 11: We assume that the remaining \(35\)% non-predictable instances are decided by random guessing.

We see that BaseLM is close to random chance, whereas PtntTime and SymTime improve 20% and 22% respectively. This suggests that our models better understand temporal common sense through the distant supervision on both start times and duration. On the other hand, we observe much smaller drops in our model's performances in this _no-story_ setting. This suggests that our models do not improve as much on the 35% instances that require multi-hop timeline constructions over more than two events, motivating future work.

Table 5 compares the two pre-training sources described in SS4.1 by individually pre-training two models with only within-sentence or cross-sentence extracted data. We see that the cross-sentence extraction brings the most performance gain on Tracing's start-time binary metric under the uniform-prior training setting. This suggests that the global extraction rule is able to introduce new knowledge that is not seen in localized language model pre-training. Combining the within-sentence data further improves the performance.

Through analysis on the interval predictions made by SymTime, we notice a tendency for the model to predict "after" for end-time instances, possibly due to overly-estimated durations: a byproduct of natural biases in text. Given the weak signal used to learn such intervals and these potential biases, this is not altogether surprising. We leave the task of learning more robust and faithful interval representations for future work.

## 7 Conclusion

We introduce a challenging dataset Tracie, to evaluate systems' temporal understanding of implicit events. We propose a distant supervision process that improves language models' understanding of start times of both explicit and implicit events. We further combine this process with a distantly supervised model that estimates events' duration to compare event end times, under the explicit rule that end times are start times plus durations. We show that our model improves over Traceie and MATRES, suggesting the effectiveness of high-precision pre-training and symbolic temporal reasoning. Despite these advances, Tracie continues to be a challenging task for future work on general temporal reasoning.

#### Acknowledgments

This research is based upon work supported in part by the office of the Director of National Intelligence (ODNI), Intelligence Advanced Research Projects Activity (IARPA), via IARPA Contract No. 2019-19051600006 under the BETTER Program, and by Contract FA8750-19-2-1004 with the US Defense Advanced Research Projects Agency (DARPA). The views expressed are those of the authors and do not reflect the official policy or position of the Department of Defense or the U.S. Government.

\begin{table}
\begin{tabular}{c|c|c|c|c} \hline \hline Sys. & BaselineLM & PtnTime & SymTime & Human \\ Acc. & 52.6 & 72.2 & 75.3 & 82.5 \\ \hline \hline \end{tabular}
\end{table}
Table 4: Performance on _no-story_ Tracie under the uniform-prior training setting.

\begin{table}
\begin{tabular}{l|c|c} \hline \hline Sys. & PtntTime & cross-sentence & within-sentence \\ \cline{2-3} Acc. & 80.6 & 79.9 & 63.7 \\ \hline \hline \end{tabular}
\end{table}
Table 5: Comparison of pre-training data sources on Tracie’s start time prediction accuracy, under the uniform-prior training setting.

 



## References

* Allen (1983) James F Allen. 1983. Maintaining knowledge about temporal intervals. _Communications of the ACM_, 26(11):832-843.
* Ammanabrolu et al. (2021) Prithviraj Ammanabrolu, Wesley Cheung, William Broniec, and Mark O Riedl. 2021. Automated Storytelling via Causal, Commonsense Plot Ordering. In _AAAI_.
* Andreas et al. (2016) Jacob Andreas, Marcus Rohrbach, Trevor Darrell, and D. Klein. 2016. Neural module networks. _CVPR_.
* Bethard et al. (2007) Steven Bethard, James H. Martin, and Sara Klingenstein. 2007. Timelines from Text: Identification of Syntactic Temporal Relations. _ICSC_.
* Bhagavatula et al. (2020) Chandra Bhagavatula, Ronan Le Bras, Chaitanya Malaviya, Keisuke Sakaguchi, Ari Holtzman, Hannah Rashkin, Doug Downey, Scott Wen-tau Yih, and Yejin Choi. 2020. Abductive Commonsense Reasoning. _In ICLR_.
* Cassidy et al. (2014) Taylor Cassidy, Bill McDowell, Nathanael Chambers, and Steven Bethard. 2014. An Annotation Framework for Dense Event Ordering. In _ACL_.
* Do et al. (2012) Quang Do, Wei Lu, and D. Roth. 2012. Joint Inference for Event Timeline Construction. In _EMNLPCoNLL_.
* Gardner et al. (2020) Matt Gardner, Yoav Artzi, Victoria Basmova, Jonathan Berant, Ben Bogin, Sihao Chen, Pradeep Dasigi, Dheeru Dua, Yanai Elazar, Ananth Gottumukkala, Nitish Gupta, Hanna Hajishirzi, Gabriel Ilharco, Daniel Khashabi, Kevin Lin, Jiangming Liu, Nelson F. Liu, Phoebe Mulcaire, Qiang Ning, Sameer Singh, Noah A. Smith, Sanjay Subramanian, R. Tsarfaty, Eric Wallace, A. Zhang, and Ben Zhou. 2020. Evaluating Models' Local Decision Boundaries via Contrast Sets. In _Findings of EMNLP_.
* Gardner et al. (2018) Matt Gardner, Joel Grus, Mark Neumann, Oyvind Tafjord, Pradeep Dasigi, Nelson F. Liu, Matthew E. Peters, M. Schmitz, and L. Zettlemoyer. 2018. AllenNLP: A Deep Semantic Natural Language Processing Platform. _NLP-OSS_, abs/1803.07640.
* Gerevini and Schubert (1995) Alfonso Gerevini and Lenhart Schubert. 1995. Efficient algorithms for qualitative reasoning about time. _Artificial intelligence_, 74(2):207-248.
* Gupta et al. (2019) Nitish Gupta, Kevin Lin, Dan Roth, Sameer Singh, and Matt Gardner. 2019. Neural Module Networks for Reasoning over Text. _In ICLR_.
* Gusev et al. (2011) Andrey Gusev, Nathanael Chambers, Divye Raj Khilnani, Pranav Khaitan, Steven Bethard, and Dan Jurafsky. 2011. Using Query Patterns to Learn the Duration of Events. In _IWCS_.
* Han et al. (2019) Rujun Han, Qiang Ning, and Nanyun Peng. 2019. Joint Event and Temporal Relation Extraction with Shared Representations and Structured Prediction. In _EMNLP_.
* Khashabi et al. (2018) Daniel Khashabi, Tushar Khot, Ashish Sabharwal, and Dan Roth. 2018. Question Answering as Global Reasoning over Semantic Abstractions. In _AAAI_.
* Khot et al. (2021) Tushar Khot, Daniel Khashabi, Kyle Richardson, Peter Clark, and Ashish Sabharwal. 2021. Text Modular Networks: Learning to Decompose Tasks in the Language of Existing Models. In _NAACL_.
* Lahiri (2014) Shibamouli Lahiri. 2014. Complexity of Word Collocation Networks: A Preliminary Structural Analysis. In _Proceedings of ACL-SRW_.
* Lai et al. (2017) Alice Lai, Yonatan Bisk, and Julia Hockenmaier. 2017. Natural Language Inference from Multiple Premises. _IJCNLP_.
* Leeuwenberg and Moens (2018) A. Leeuwenberg and Marie-Francine Moens. 2018. Temporal information extraction by predicting relative time-lines. In _EMNLP_.
* Li et al. (2019) Tao Li, Vivek Gupta, Maitrey Mehta, and V. Srikumar. 2019. A logic-driven framework for consistency of neural models. In _EMNLP_.
* Liu et al. (2019) Yinhan Liu, Myle Ott, Naman Goyal, Jingfei Du, Mandar Joshi, Danqi Chen, Omer Levy, Mike Lewis, Luke Zettlemoyer, and Veselin Stoyanov. 2019. RoBERTa: A Robustly Optimized BERT Pretraining Approach. _ArXiv_, abs/1907.11692.
* Meng and Rumshisky (2018) Yuanliang Meng and Anna Rumshisky. 2018. Context-aware neural model for temporal information extraction. In _ACL_.
* Moens and Lecuwenberg (2017) Marie-Francine Moens and A. Lecuwenberg. 2017. Structured learning for temporal relation extraction from clinical records. In _EACL_.
* Mostafazadeh et al. (2016) Nasrin Mostafazadeh, Nathanael Chambers, Xiaodong He, Devi Parikh, Dhruv Batra, Lucy Vanderweronde, Pushmeet Kohli, and James Allen. 2016. A Corpus and Evaluation Framework for Deeper Understanding of Commonsense Stripes. In _NAACL_.
* Nie et al. (2020) Yixin Nie, Adina Williams, Emily Dinan, Mohit Bansal, Jason Weston, and Douwe Kiela. 2020. Adversarial NLI: A New Benchmark for Natural Language Understanding. _In ACL_.
* Ning et al. (2017) Qiang Ning, Z. Feng, and D. Roth. 2017. A Structured Learning Approach to Temporal Relation Extraction. In _EMNLP_.
* Ning et al. (2018a) Qiang Ning, H. Wu, H. Peng, and D. Roth. 2018a. Improving Temporal Relation Extraction with a Globally Acquired Statistical Resource. In _NAACL_.
* Ning et al. (2020a) Qiang Ning, Hao Wu, Pradeep Dasigi, Dheeru Dua, Matt Gardner, Robert L. Logan IV, and Zhen Nie. 2020a. Easy, Reproducible and Quality-Controlled Data Collection with CROWDAQ. In _EMNLP_.
* Ning et al. (2020b) Qiang Ning, Hao Wu, Rujun Han, Nanyun Peng, Matt Gardner, and Dan Roth. 2020b. Torque: A reading comprehension dataset of temporal ordering questions. In _EMNLP_.



* Ning et al. (2018) Qiang Ning, Hao Wu, and Dan Roth. 2018b. A multi-axis annotation scheme for event temporal relations. In _ACL_.
* Ning et al. (2018c) Qiang Ning, Ben Zhou, Z. Feng, H. Peng, and D. Roth. 2018c. CogCompTime: A Tool for Understanding Time in Natural Language. In _EMNLP_.
* O'Gorman et al. (2016) Tim O'Gorman, Kristin Wright-Bettner, and Martha Palmer. 2016. Richer event description: Integrating event coreference with temporal, causal and bridging annotation. In _CNS_.
* Pan et al. (2006) Feng Pan, Rutu Mulkar-Mehta, and Jerry R Hobbs. 2006. Extending TimeML with Typical Divations of Events. In _Proceedings of the Workshop on Annotating and Reasoning about Time and Events_.
* Pearl (2009) J. Pearl. 2009. Causal inference in statistics: An overview. _Statistics Surveys_, 3:96-146.
* Poliak et al. (2018) Adam Poliak, Jason Naradowsky, Aparajita Haldar, Rachel Rudinger, and Benjamin Van Durme. 2018. Hypothesis only Baselines in Natural Language Inference. _Proceedings of *SEM_.
* Pustejovsky et al. (2003) James Pustejovsky, Jose M Castano, Robert Ingria, Roser Sauri, Robert J Gaizauskas, Andrea Setzer, Graham Katz, and Dragomir R Radev. 2003. TimeM: Robust Specification of Event and Temporal Expressions in Text. In _New Directions in Question Answering_.
* Raffel et al. (2020) Colin Raffel, Noam Shazeer, Adam Roberts, Katherine Lee, Sharan Narang, Michael Matena, Yanqi Zhou, Wei Li, and Peter J Liu. 2020. Exploring the Limits of Transfer Learning with a Unified Text-to-text Transformer. _JMLR_, 21(140):1-67.
* Reimers et al. (2016) Nils Reimers, Nazanin Dehghani, and Iryna Gurevych. 2016. Temporal Anchoring of Events for the Timebank corpus. In _ACL_.
* Talmor and Berant (2018) Alon Talmor and Jonathan Berant. 2018. The Web as a Knowledge-base for Answering Complex Questions. _In NAACL_.
* Tandon et al. (2018) Niket Tandon, Bhavana Dalvi, Joel Grus, Wen tau Yih, Antoine Bosseltur, and Peter Clark. 2018. Reasoning about actions and state changes by injecting commonsense knowledge. _ArXiv_, abs/1808.10012.
* Tandon et al. (2020) Niket Tandon, Keisuke Sakaguchi, Bhavana Dalvi Mishra, Dieheraj Rajagopal, Peter Clark, Michal Guerquin, Kyle Richardson, and Eduard Hovy. 2020. A dataset for tracking entities in open domain procedural text. In _EMNLP_.
* Vashishtha et al. (2020) Siddharth Vasishitha, Adam Poliak, Yash Kumar Lal, Benjamin Van Durme, and Aaron Steven White. 2020. Temporal Reasoning in Natural Language Inference. In _Finding of EMNLP_.
* Vashishtha et al. (2019) Siddharth Vashishtha, Benjamin Van Durme, and Aaron Steven White. 2019. Fine-grained Temporal Relation Extraction. In _ACL_.
* Wang et al. (2020) Haoyu Wang, Muhao Chen, Hongming Zhang, and Dan Roth. 2020. Joint constrained learning for event-event relation extraction. In _EMNLP_.
* Williams et al. (2017) Adina Williams, Nikita Nangia, and Samuel R Bowman. 2017. A broad-coverage challenge corpus for sentence understanding through inference. _In NAACL_.
* Wolf et al. (2019) Thomas Wolf, Lysandre Debut, Victor Sanh, Julien Chaumond, Clement Delangue, Anthony Moi, Pierric Cistac, Tim Rault, Remi Louf, Morgan Funtowicz, et al. 2019. Transformers: State-of-the-art natural language processing. _arXiv preprint arXiv:1910.03771_.
* Wolfson et al. (2020) Tomer Wolfson, Mor Geva, Ankit Gupta, Matt Gardner, Yoav Goldberg, Daniel Deutch, and Jonathan Berant. 2020. Break it Down: A Question Understanding Benchmark. _TACL_, 8:183-198.
* Zhou et al. (2019) Ben Zhou, Daniel Khashabi, Qiang Ning, and D. Roth. 2019. "going on a vacation" takes longer than "going for a walk": A study of temporal commonsense understanding. In _EMNLP_.
* Zhou et al. (2020) Ben Zhou, Qiang Ning, Daniel Khashabi, and Dan Roth. 2020. Temporal Common Sense Acquisition with Minimal Supervision. In _ACL_.