Venue: NAACL 2018

# Improving Temporal Relation Extraction with a Globally Acquired

Statistical Resource

Qiang Ning,\({}^{1}\) Hao Wu,\({}^{2}\) Haoruo Peng,\({}^{1}\) Dan Roth\({}^{1,2}\)

Department of Computer Science

\({}^{1}\)University of Illinois at Urbana-Champaign, Urbana, IL 61801, USA

\({}^{2}\)University of Pennsylvania, Philadelphia, PA 19104, USA

{qning2,hpeng7}@illinois.edu, {haowu4,danroth}@seas.upenn.edu

###### Abstract

Extracting temporal relations (_before, after, overlapping_, etc.) is a key aspect of understanding events described in natural language. We argue that this task would gain from the availability of a resource that provides prior knowledge in the form of the temporal order that events _usually_ follow. This paper develops such a resource - a probabilistic knowledge base acquired in the news domain - by extracting temporal relations between events from the New York Times (NYT) articles over a 20-year span (1987-2007). We show that existing temporal extraction systems can be improved via this resource. As a byproduct, we also show that interesting statistics can be retrieved from this resource, which can potentially benefit other time-aware tasks. The proposed system and resource are both publicly available1.

Footnote 1: [http://cogcomp.org/page/publication_view/830](http://cogcomp.org/page/publication_view/830)

+
Footnote †: New Orleans, Louisiana, June 1 - 6, 2018. ©2018 Association for Computational Linguistics

## 1 Introduction

Time is an important dimension of knowledge representation. In natural language, temporal information is often expressed as relations between events. Reasoning over these relations can help figuring out when things happened, estimating how long things take, and summarizing the timeline of a series of events. Several recent SemEval workshops are a good showcase of the importance of this topic Verhagen et al. (2007); Verhagen et al. (2010); Uz-Zaman et al. (2013); Llorens et al. (2015); Minard et al. (2015); Bethard et al. (2015); Bethard et al. (2016, 2017).

One of the challenges in temporal relation extraction is that it requires high-level prior knowledge of the temporal order that events _usually_ follow. In Example 1, we have deleted events from several snippets from CNN, so that we cannot use our prior knowledge of those events. We are also told that _e1_ and _e2_ have the same tense, and _e3_ and _e4_ have the same tense, so we cannot resort to their tenses to tell which one happens earlier. As a result, it is very difficult even for humans to figure out the temporal relations (referred to as "TempRels" hereafter) between those events. This is because rich temporal information is encoded in the events' names, and this often plays an indispensable role in making our decisions. In the first paragraph of Example 1, it is difficult to understand what really happened without the actual event verbs; let alone the TempRels between them. In the second paragraph, things are even more interesting: if we had _e3:dislike_ and _e4:stop_, then we would know easily that "I dislike" occurs _after_ "they stop the column". However, if we had _e3:ask_ and _ed:help_, then the relation between _e3_ and _e4_ is now reversed and _e3_ is _before e4_. We are in need of the event names to determine the TempRels; however, we do not have them in Example 1. In Example 2, where we show the complete sentences, the task has become much easier for humans due to our prior knowledge, namely, that explosion _usually_ leads to casualties and that people _usually_ ask before they get help. Motivated by these examples (which are in fact very common), we believe in the importance of such a prior knowledge in determining TempRels between events.

```
Example 1: Difficulty in understanding TempRels when event content is missing. Note that _e1_ and _e2_ have the same tense, and _e3_ and _e4_ have the same tense. More than 10 people have (_e1:_ ), police said. A car (_e2:_ ) on Friday in the middle of a group of men playing volleyball. The first thing I (_e3:_ ) is that they (_e4:_ ) writing this column.
```

However, most existing systems only make use of rather local features of these events, which cannot represent the prior knowledge humans have

about these events and their "typical" order. As a result, existing systems almost always attempt to solve the situations shown in Example 1, even when they are actually presented with input as in Example 2. The **first contribution** of this work is thus the construction of such a resource in the form of a probabilistic knowledge base, constructed from a large New York Times (NYT) corpus. We hereafter name our resource _TEMporal relation PROabilistic knowledge Base_ (TemProb), which can potentially benefit many time-aware tasks. A few example entries of TemProb are shown in Table 1. **Second**, we show that existing TempRel extraction systems can be improved using TemProb, either in a local method or in a global method (explained later), by a significant margin in performance on the benchmark TimeBank-Dense dataset Cassidy et al. (2014).

\begin{tabular}{|l|c|} \hline  & \multicolumn{1}{c|}{**Example 2: The original sentences in Example 1.**} \\ \hline More than 10 people have (_e1:died_), police said. A car (_e2:exploded_) on Friday in the middle of a group of men playing volleyball. \\ \hline The first thing I (_e3:ask_) is that they (_e4:help_) writing this column. \\ \end{tabular}

The rest of the paper is organized as follows. Section 2 provides a literature review of TempRels extraction in NLP. Section 3 describes in detail the construction of TemProb. In Sec. 4, we show that TemProb can be used in existing TempRes extraction systems and lead to significant improvement. Finally, we conclude in Sec. 5.

## 2 Related Work

The TempRels between events can be represented by an edge-labeled graph, where the nodes are events, and the edges are labeled with TempRels Chambers and Jurafsky (2008); Do et al. (2012); Ning et al. (2017). Given all the nodes, we work on the TempRel extraction task, which is to assign labels to the edges in a temporal graph (a "vague" or "none" label is often included to account for the non-existence of an edge).

Early work includes Mani et al. (2006); Chambers et al. (2007); Bethard et al. (2007); Verhagen and Pustejovsky (2008), where the problem was formulated as learning a classification model for determining the label of every edge _locally_ without referring to other edges (i.e., local methods). The predicted temporal graphs by these methods may violate the transitive properties that a temporal graph should possess. For example, given three nodes, _e1_, _e2_, and _e3_, a local method can possibly classify (_e1,e2_)=before_, (_e2,e3_)=before_, and (_e1,e3_)=after_, which is obviously wrong since _before_ is a transitive relation and (_e1,e2_)=before_ and (_e2,e3_)=before_ dictate that (_e1,e3_)=before_. Recent state-of-the-art methods, Chambers et al. (2014); Mirza and Tonelli (2016), circumvented this issue by growing the predicted temporal graph in a multi-step manner, where transitive graph closure is performed on the graph every time a new edge is labeled. This is conceptually solving the structured prediction problem greedily. Another family of methods resorted to Integer Linear Programming (ILP) Roth and Yih (2004) to get exact inference to this problem (i.e., global methods), where the entire graph is solved simultaneously and the transitive properties are enforced naturally via ILP constraints Bramsen et al. (2006); Chambers and Jurafsky (2008); Denis and Muller (2011); Do et al. (2012). A most recent work brought this idea even further, by incorporating structural constraints into the learning phase as well Ning et al. (2017).

The TempRel extraction task has a strong dependency on prior knowledge, as shown in our earlier examples. However, very limited attention has been paid to generating such a resource and to make use of it; to our knowledge, the TemProb proposed in this work is completely new. We find that the _time-sensitive relations_ proposed in Jiang et al. (2016) is a close one in literature (although it is still very different). Jiang et al. (2016) worked on the knowledge graph completion task. Based on YAGO2 Hoffwart et al. (2013) and Freebase Bollacker et al. (2008), it manually selects a small number of relations that are time-sensitive (10 relations from YAGO2 and 87 relations from Freebase, respectively). Exemplar relations are wasBornIn\(\rightarrow\)diedln\(\rightarrow\) and graduateFrom\(\rightarrow\)workAt, where \(\rightarrow\) means temporally be

\begin{table}
\begin{tabular}{r c|c|c} \hline \multicolumn{2}{c|}{Example Pairs} & Before (\%) & After (\%) \\ \hline accept & determine & 42 & 26 \\ ask & help & 86 & 9 \\ attend & schedule & 1 & 82 \\ accept & propose & 10 & 77 \\ die & explode & 14 & 83 \\  & ... & & \\ \hline \end{tabular}
\end{table}
Table 1: **TemProb is a unique source of information of the temporal order that events _usually_ follow.** The probabilities below do not add up to 100% because less frequent relations are omitted. The word sense numbers are not shown here for convenience.

 

fore.

Our work significantly differs from the time-sensitive relations in Jiang et al. (2016) in the following aspects. First, scale difference: Jiang et al. (2016) can only extract a small number of relations (\(<\)100), but we work on general semantic frames (tens of thousands) and the relations between any two of them, which we think has broader applications. Second, granularity difference: the smallest granularity in Jiang et al. (2016) is one year2, i.e., only when two events happened in different years can they know the temporal order of them, but we can handle implicit temporal orders without having to refer to the physical time points of events (i.e., the granularity can be arbitrarily small). Third, domain difference: while Jiang et al. (2016) extracts time-sensitive relations from structured knowledge bases (where events are explicitly anchored to a time point), we extract relations from unstructured natural language text (where the physical time points may not even exist in text). Our task is more general and it allows us to extract much more relations, as reflected by the 1st difference above.

Footnote 2: We notice that the smallest granularity in Freebase itself is one day, but Jiang et al. (2016) only used years.

Another related work is the VerbOcean Chklovski and Pantel (2004), which extracts temporal relations between pairs of verbs using manually designed lexico-syntactic patterns (there are in total 12 such patterns), in contrast to the automatic extraction method proposed in this work. In addition, the only temporal relation considered in VerbOceans is _before_, while we also consider relations such as _after_, _includes_, _included_, _equal_, and _vague_. As expected, the total numbers of verbs and _before_ relations in VerbOcean is about 3K and 4K, respectively, both of which are much smaller than TemProb, which contains 51K verb frames (i.e., disambiguated verbs), 9.2M \((verb{1},verb{2},relation)\) entries, and up to 80M temporal relations altogether.

All these differences necessitate the construction of a new resource for TempRel extraction, which we explain below.

## 3 TemProb: A Probabilistic Resource for TempRels

In the TempRel extraction task, people have usually assumed that events are already given. However, to construct the desired resource, we need to extract events (Sec. 3.1) and extract TempRes (Sec. 3.2), from a large, unannotated3 corpus (Sec. 3.3). We also show some interesting statistics discovered in TemProb that may benefit other tasks (Sec. 3.4). In the next, we describe each of these elements.

Footnote 3: Unannotated with TempRels.

### Event Extraction

Extracting events and the relations between them (e.g., coreference, causality, entailment, and temporal) have long been an active area in the NLP community. Generally speaking, an event is considered to be an action associated with corresponding participants involved in this action. In this work, following Peng and Roth (2016); Peng et al. (2016); Spiliopoulou et al. (2017) we consider semantic-frame based events, which can be directly detected via off-the-shelf semantic role labeling (SRL) tools. This aligns well with previous works on event detection Hovy et al. (2013); Peng et al. (2016).

Depending on the events of interest, the SRL results are often a superset of events and need to be filtered afterwards Spiliopoulou et al. (2017). For example, in ERE Song et al. (2015) and Event Nugget Detection Mitamura et al. (2015), events are limited to a set of predefined types (such as "Business", "Conflict", and "Justice"); in the context of TempRels, existing datasets have focused more on predicate verbs rather than nominals4Pustejovsky et al. (2003); Graff (2002); UzZaman et al. (2013). Therefore, we only look at verb semantic frames in this work due to the difficulty of getting TempRel annotation for nominal events, and we will use "verb (semantic frames)" interchangeably with "events" hereafter in this paper.

Footnote 4: Some nominal events were indeed annotated in TimeBank Pustejovsky et al. (2003), but their annotation did not align well with modern nominal-SRL methods.

### TempRel Extraction

Given the events extracted in a given article (i.e., given the nodes in a graph), we next explain how the TempRels are extracted (that is, the edge labels in the graph).

#### 3.2.1 Features

We adopt the commonly used feature set in Tempel extraction Do et al. (2012); Ning et al. (2017) and here we simply list them for reproducibility. For each pair of nodes, the follow 

ing features are extracted. (i) The part-of-speech (POS) tags from each individual verb and from its neighboring three words. (ii) The distance between them in terms of the number of tokens. (iii) The modal verbs between the event mention (i.e., will, would, can, could, may and might). (iv) The temporal connectives between the event mentions (e.g., before, after and since). (v) Whether the two verbs have a common synonym from their synsets in WordNet [25]. (vi) Whether the input event mentions have a common derivational form derived from WordNet. (vii) The head word of the preposition phrase that covers each verb, respectively.

#### 3.2.2 Learning

With the features defined above, we need to train a system that can annotate the TempRels in each document. The TimeBank-Dense dataset (TBDense) [19] is known to have the best quality in terms of its high density of TempRels and is a benchmark dataset for the Tempel extraction task. It contains 36 documents from TimeBank [40] which were re-annotated using the dense event ordering framework proposed in [18]. We follow its label set (denoted by \(R\)) of _before_, _after_, _includes_, _included_, _equal_, and _vague_ in this study.

Due to the slight event annotation difference in TBDense, we collect our training data as follows. We first extract all the verb semantic frames from the raw text of TBDense. Then we only keep those semantic frames that are matched to an event in TBDense (about 85% semantic frames are kept in this stage). By doing so, we can simply use the Tempeln annotations provided in TBDense. Hereafter the TBDense dataset used in this paper refers to this version unless otherwise specified.

We group the TempRels by the sentence distance of the two events of each relation5. Then we use the averaged perceptron algorithm [7] implemented in the Illinois LBJava package [10] to learn from the training data described above. Since only relations that have sentence distance 0 or 1 are annotated in TBDense, we will have two classifiers, one for same sentence relations, and one for neighboring sentence relations, respectively.

Footnote 5: That is, the difference of the appearance order of the sentence(s) containing the two target events.

Note that TBDense was originally split into Train (22 docs), Dev (5 docs), and Test (9 docs). In all subsequent analysis, we combined Train and Dev and we performed 3-fold cross validation on the 27 documents (in total about 10K relations) to tune the parameters in any classifier.

#### 3.2.3 Inference

When generating TemProb, we need to process a large number of articles, so we adopt the greedy inference strategy described earlier due to its computational efficiency [1, 19]. Specifically, we apply the same-sentence relation classifier before the neighboring-sentence relation classifier; whenever a new relation is added in this article, a transitive graph closure is performed immediately. By doing this, if an edge is already labeled during the closure phase, it will not be labeled again, so conflicts are avoided.

### Corpus

As mentioned earlier, the source corpus on which we are going to construct TemProb is comprised of NYT articles from 20 years (1987-2007)6. It contains more than 1 million documents and we extract events and corresponding features from each document using the Illinois Curator package [19] on Amazon Web Services (AWS) Cloud. In total, we discovered 51K unique verb semantic frames and 80M relations among them in the NYT corpus (15K of the verb frames had more than 20 relations extracted and 9K had more than 100 relations).

Footnote 6: [https://catalog.ldc.upenn.edu/LDC200ST19](https://catalog.ldc.upenn.edu/LDC200ST19)

### Interesting Statistics

We first describe the notations that we are going to use. We denote the set of all verb semantic frames by \(V\). Let \(D_{i},i=1,\ldots,N\) be the \(i\)-th document in our corpus, where \(N\) is the total number of documents. Let \(G_{i}=(V_{i},E_{i})\) be the temporal graph inferred from \(D_{i}\) using the approach described above, where \(V_{i}\subseteq V\) is the set of verbs/events extracted in \(D_{i}\) and \(E_{i}=\{(v_{m},v_{n},r_{mn})\}_{m<n}\subseteq V_{i}\times V_{i}\times R\) is the edge set of \(D_{i}\), which is composed of TempRel triplets; specifically, a TempRel triplet \((v_{m},v_{n},r_{mn})\in E_{i}\) represents that in document \(D_{i}\), the TempRel between \(v_{m}\) and \(v_{n}\) is \(r_{mn}\). Due to the symmetry in TempRels, we only keep the triplets with \(m<n\) in \(E_{i}\). Assuming that the 

verbs in \(V_{i}\) are ordered by their appearance order in text, then \(m<n\) means that in the \(i\)-th document, \(v_{m}\) appears earlier in text than \(v_{n}\) does.

Given the usual confusion between that one event is _temporally before_ another and that one event is _physically appearing_ before another in text, we will refer to temporally before as **T-Before** and physically before as **P-Before**. Using this language, for example, \(E_{i}\) only keeps the triplets that \(v_{m}\) is P-Before \(v_{n}\) in \(D_{i}\).

#### 3.4.1 Extreme cases

We first show extreme cases that some events are _almost always_ labeled as T-Before or T-After in the corpus. Specifically, for each pair of verbs \(v_{i},v_{j}\in V\), we define the following ratios:

\[\eta_{b}=\frac{C(v_{i},v_{j},before)}{C(v_{i},v_{j},before)+C(v_{i},v_{j},after )},\eta_{a}=1-\eta_{b}, \tag{1}\]

where \(C(v_{i},v_{j},r)\) is the count of \(v_{i}\) P-Before \(v_{j}\) with TempRel \(r\in R\):

\[C(v_{i},v_{j},r)=\sum_{i=1}^{N}\sum_{(v_{m},v_{m},v_{mn})\in E_{i}}\mathcal{I }_{\{v_{m}=v_{i}kv_{n}=v_{j}kr_{mn}=r\}}, \tag{2}\]

where \(\mathcal{I}_{\{\cdot\}}\) is the indicator function. Add-one smoothing technique from language modeling is used to avoid divided-by-zero errors. In Table 2, we show some event pairs with either \(\eta_{b}>0.9\) (upper part) or \(\eta_{a}>0.9\) (lower part).

We think the examples from Table 2 are intuitively appealing: _chop_ happens before _taste_, _clean_ happens after _containmate_, etc. More interestingly, in the lower part of the table, we show pairs in which the physical order is different from the temporal order: for example, when _achieve_ is P-Before _desire_, it is still labeled as T-After in most cases (104 out of 111 times), which is correct intuitively. In practice, e.g., in the TBDense dataset Cassidy et al. (2014), roughly 30%-40% of the P-Before pairs are T-After. Therefore, it is important to be able to capture their temporal order rather than simply taking their physical order if one wants to understand the temporal implication of verbs.

#### 3.4.2 Distribution of Following Events

For each verb \(v\), we define the marginal count of \(v\) being P-Before to arbitrary verbs with TempRel \(r\in R\) as \(C(v,r)=\sum_{v_{i}\in V}C(v,v_{i},r)\). Then for every other verb \(v^{\prime}\), we define

\[P(v\text{ T-Before $v^{\prime}|v\text{ T-Before}$})\triangleq\frac{C(v,v^{ \prime},before)}{C(v,before)}, \tag{3}\]

which is the probability of \(v\) T-Before \(v^{\prime}\), conditioned on \(v\) T-Before anything. Similarly, we define

\[P(v\text{ T-After $v^{\prime}|v\text{ T-After}$})\triangleq\frac{C(v,v^{ \prime},after)}{C(v,after)}. \tag{4}\]

For a specific verb, e.g., \(v\textbf{=investigate}\), each verb \(v^{\prime}\in V\) is sorted by the two conditional probabilities above. Then the most probable verbs that temporally precede or follow \(v\) are shown in Fig. 1, where the y-axes are the corresponding conditional probabilities. We can see reasonable event sequences like \(\{\textit{involve}\), _kill_, _suspect_, _steal_\(\}\mathbf{\rightarrow}\texttt{investigate}\mathbf{\rightarrow}\{\textit{report}\), _prosecute_, _pay_, _punish_\(\}\), which indicates the possibility of using TemProb for event sequence predictions or story cloze tasks. There are also suspicious pairs like _know_ in the T-Before list of _investigate_ (Fig. 0(a)), _report_ in the T-Before list of _bmb_ (Fig. 0(b)), and _play_ in the T-After list of _mourn_ (Fig. 0(c)). Since the arguments of these verb frames are not considered here, whether these few seemingly counter-intuitive pairs come from system error or from a special context needs further investigation.

## 4 Experiments

In the above, we have explained the construction of TemProb and shown some interesting examples from it, which were meant to visualize its correctness. In this section, we first quantify the correctness of the prior obtained in TemProb, and

\begin{table}
\begin{tabular}{c c|c|c} \hline Example Pairs & \#T-Before & \#T-After \\ \hline chop.01 & taste.01 & 133 & 8 \\ concern.01 & protect.01 & 110 & 10 \\ conspire.01 & kill.01 & 113 & 6 \\ debate.01 & vote.01 & 48 & 5 \\ dedicate.01 & promote.02 & 67 & 7 \\ fight.01 & overthrow.01 & 98 & 8 \\ \hline achieve.01 & desire.01 & 7 & 104 \\ admire.01 & respect.01 & 7 & 121 \\ clean.02 & contaminate.01 & 3 & 82 \\ defend.01 & accuse.01 & 13 & 160 \\ die.01 & crash.01 & 8 & 223 \\ overfitting.01 & elect.01 & 3 & 100 \\ \hline \end{tabular}
\end{table}
Table 2: **Several extreme cases from TemProb**, where some event is almost always labeled to be T-Before or T-After throughout the NYT corpus. By “extreme”, we mean that either the probability of T-Before or T-After is larger than 90%. The upper part of the table shows the pairs that are both P-Before and T-Before, while the lower part shows the pairs that are P-Before but T-After. In TemProb, there are about 7K event pairs being extreme cases.

 

then show TemProb can be used to improve existing TempRel extraction systems.

### Quality Analysis of TemProb

In Table 2, we showed examples with either \(\eta_{b}\) or \(\eta_{a}>0.9\). We argued that they _seem_ correct. Here we quantify the "correctness" of \(\eta_{b}\) and \(\eta_{a}\) based on TBDense. Specifically, we collected all the gold T-Before and T-After pairs. Let \(\tau\in[0.5,1)\) be a constant threshold. Imagine a naive predictor such that for each pair of events \(v_{i}\) and \(v_{j}\), if \(\eta_{b}>\tau\), it predicts that \(v_{i}\) is T-Before \(v_{j}\); if \(\eta_{a}>\tau\), it predicts that \(v_{i}\) is T-After \(v_{j}\); otherwise, it predicts that \(v_{i}\) is T-Vague to \(v_{j}\). We expect that a higher \(\eta_{b}\) (or \(\eta_{a}\)) represents a higher confidence for an instance to be labeled T-Before (or T-After).

Table 3 shows the performance of this predictor, which meets our expectation and thus justifies the validity of TemProb. As we gradually increase the value of \(\tau\) in Table 3, the precision increases in roughly the same pace with \(\tau\), which indicates that the values of \(\eta_{b}\) and \({\eta_{a}}^{7}\) from TemProb indeed represent the confidence level. The decrease in recall is also expected because more examples are labeled as T-Vague when \(\tau\) is larger.

To further justify the quality, we also used another dataset that is not in the TempRel domain. Instead, we downloaded the EventCausality dataset8Do et al. (2011). For each causally related pair _eI_ and _e2_, if EventCausality annotates that _eI_ causes _e2_, we changed it to be T-Before; if EventCausality annotates that _eI_ is caused by _e2_, we changed it to be T-after. Therefore, based on the assumption that the cause event is T-Before the result event, we converted the EventCausality dataset to be a Tempel dataset and it thus could also be used to evaluate the quality of TemProb. We adopted the same predictor used in Table 3

Figure 1: **Top events that most frequently precede or follow “investigate”, ‘‘bmb’, “mourn’, or “sentence” in time**, sorted by their conditional probabilities in %c. Word senses have been disambiguated and the “bomb” and “sentence” here are their verb meanings. There are some possible errors (e.g., _report_ is T-Before _bomb_) and some unclear pairs (e.g., _know_ is T-Before _investigate_ and _play_ is T-After _mourn_), but overall the event sequences discovered here are reasonable. More examples can be found in the appendix.

 

with \(\tau=0.5\) and in Table 4, we compared it with two baselines: (i) always predicting T-Before and (ii) always predicting T-After. First, the accuracy (66.2%) in Table 4 is rather consistent with its counterpart in Table 3, confirming the stability of statistics from TemProb. Second, by directly using the prior statistics \(\eta_{b}\) and \(\eta_{a}\) from TemProb, we can improve the precision of both labels with a significant margin relative to the two baselines (17.0% for "T-Before" and 15.9% for "T-After"). Overall, the accuracy was improved by 11.5%.

### Improving TempRel Extraction

The original purpose of TemProb was to improve Tempel extraction. We show it from two perspectives: How effective the prior distributions obtained from TemProb are (i) as features in local methods and (ii) as regularization terms in global methods. The results below were evaluated on the test split of TB-Dense Cassidy et al. (2014).

#### 4.2.1 Improving Local Methods

We first test how well the prior distributions from TemProb can be used as features in improving local methods for TempRel extraction. In Table 5, we used the original feature set proposed in Sec. 3.2.1 as the baseline, and added the prior distribution obtained from TemProb on top of it. Specifically, we added \(\eta_{b}\) (see Eq. (1)) and \(\{f_{r}\}_{r\in R}\), respectively, where \(\{f_{r}\}_{r\in R}\) is the prior distributions of all labels, i.e.,

\[f_{r}(v_{i},v_{j})=\frac{C(v_{i},v_{j},r)}{\sum_{r^{\prime}\in R}C(v_{i},v_{ j},r^{\prime})},~{}r\in R. \tag{5}\]

Recall function \(C\) is defined in Eq. (2). All comparisons were decomposed to same sentence relations (Dist=0) and neighboring sentence relations (Dist=1) for a better understanding of the behavior. All classifiers were trained using the averaged perceptron algorithm Freund and Schapire (1998) and tuned by 3-fold cross validation.

From Table 5, we can see that simply adding \(\eta_{b}\) into the feature set could improve the original system F\({}_{1}\) by 1.8% (Dist=0) and 3.0% (Dist=1). If we further add as features the full set of prior distributions \(\{f_{r}\}_{r\in R}\), the improvement comes to 2.7% and 6.5%, respectively. Noticing that the feature is more helpful for Dist=1, we think that it is because distant pairs usually have less lexical dependency and thus need more prior information provided by our new feature. With Dist=0 and Dist=1 combined (numbers not shown in the Table), the 3rd line improved the "original" by 4.7% in F\({}_{1}\) and by 5.1% in the temporal awareness F-score (another metric used in the TempEval3 workshop).

#### 4.2.2 Improving Global Methods

As mentioned earlier in Sec. 2, many systems adopt a global inference method via integer linear programming (ILP) Roth and Yih (2004) to enforce transitivity constraints over an entire temporal graph Bramsen et al. (2006); Chambers and Jurafsky (2008); Denis and Muller (2011); Do et al. (2012); Ning et al. (2017). In addition to the usage shown in Sec. 4.2.1, the prior distributions from TemProb can also be used to regularize the conventional ILP formulation. Specifically, in each document, let \(\mathcal{I}_{r}(ij)\in\{0,1\}\) be the indicator function of relation \(r\) for event \(i\) and event \(j\); let \(x_{r}(ij)\in[0,1]\) be the corresponding softmax score obtained from the local classifiers (depending on the sentence distance between \(i\) and \(j\)). Then the ILP objective for global inference is

\begin{table}
\begin{tabular}{c|c c|c c|c} \hline \multirow{2}{*}{System} & \multicolumn{3}{c|}{T-Before} & \multicolumn{3}{c|}{T-After} \\ \cline{2-5}  & P & R & P & R & Acc. \\ \hline T-Before Only & 54.7 & 100.0 & 0 & 0 & 54.7 \\ T-After Only & 0 & 0 & 45.3 & 100 & 45.3 \\ \(\tau=0.5\) & 71.7 & 63.3 & 61.2 & 69.8 & 66.2 \\ \hline \end{tabular}
\end{table}
Table 4: **Further justification of \(\eta_{b}\) and \(\eta_{a}\) from TemProb on the EventCausality dataset**. The thresholding predictor from Table 3 with \(\tau=0.5\) is used here. Compared to always predicting the majority label (i.e., T-Before in this case), \(\tau=0.5\) significantly improved the performance for both labels, with the overall accuracy improved by 11.5%.

\begin{table}
\begin{tabular}{l|c|c|c|c|c|c} \hline Feature Set & \multicolumn{3}{c|}{Dist=0} & \multicolumn{3}{c}{Dist=1} \\ \cline{2-7}  & P & R & F\({}_{1}\) & P & R & F\({}_{1}\) \\ \hline Original & 44.5 & 57.1 & 50.0 & 49.0 & 36.9 & 42.1 \\ +\(\eta_{b}\) & 46.2 & 58.9 & 51.8 & 55.3 & 38.1 & 45.1 \\ +\(\{f_{r}\}_{r\in R}\) & 46.9 & 60.1 & 52.7 & 51.3 & 46.2 & 48.6 \\ \hline \end{tabular}
\end{table}
Table 5: **Using prior distributions derived from TemProb as features in an example local method. Incorporating \(\eta_{b}\) to the original feature set already yields better performance. By using the full set of prior distributions, \(\{f_{r}\}_{r\in R}\), the final system improves the original in almost all metrics, and the improvement is statistically significant with p<0.005 per the McNemar’s test.

 

formulated as follows.

\[\hat{\mathcal{I}}= \underset{\text{argmax}}{\max}\sum_{ij\in F}\sum_{r\in R}(x_{r}( ij)+\lambda f_{r}(ij))\mathcal{L}_{r}(ij)\] (6) s.t. \[\Sigma_{\begin{subarray}{c}\mathcal{I}_{r}(ij)=1,\\ (\text{uniqueness})\end{subarray}}\underbrace{\mathcal{I}_{r}(ij)=\mathcal{I}_{ r}(ji)}_{\text{(symmetry)}},\] \[\mathcal{I}_{r_{1}}(ij)+\mathcal{I}_{r_{2}}(jk)-\Sigma^{M}_{m=1 }\mathcal{I}_{r_{3}^{m}}(ik)\leq 1,\]

for all distinct events \(i\), \(j\), and \(k\), where \(\mathcal{E}=\{ij~{}|~{}\text{sentence dist}(i,j){\leq}~{}1\}\), \(\lambda\) adjusts the regularization term and was heuristically set to 0.5 in this work, \(\bar{r}\) is the reverse relation of \(r\), and \(M\) is the number of possible relations for \(r_{3}\) when \(r_{1}\) and \(r_{2}\) are true. Note our difference from the ILP in Ning et al. (2017) is the underlined regularization term \(f_{r}(ij)\) (which itself is defined in Eq. (5)) obtained from TemProb.

We present our results on the test split of TBDense in Table 6, which is an ablation study showing step-by-step improvements in two metrics. In addition to the straightforward precision, recall, and F\({}_{1}\) metric, we also compared the F\({}_{1}\) of the temporal awareness metric used in TempEval3 UzZaman et al. (2013). The awareness metric performs graph reduction and closure before evaluation so as to better capture how useful a temporal graph is. Details of this metric can be found in UzZaman and Allen (2011); UzZaman et al. (2013); Ning et al. (2017).

In Table 6, the baseline used the original feature set proposed in Sec. 3.2.1 and applied global ILP inference with transitivity constraints. Technically, it is to solve Eq. (6) with \(\lambda=0\) (i.e., unregularized) on top of the original system in Table 5. Apart from some implementation details, this baseline is also the same as many existing global methods as Chambers and Jurafsky (2008); Do et al. (2012). System 2, "+Feature: \(\{f_{r}\}_{r\in R}\)", is to add prior distributions as features when training the local classifiers. Technically, the scores \(x_{r}(ij)\)'s in Eq. (6) used by baseline were changed. We know from Table 5 that adding \(\{f_{r}\}_{r\in R}\) made the local decisions better. Here the performance of System 2 shows that this was also the case for the global decisions made via ILP: both precision and recall got improved, and F\({}_{1}\) and awareness were both improved by a large margin, with 5.1% in F\({}_{1}\) and 6.6% in awareness F\({}_{1}\). On top of this, System 3 sets \(\lambda=0.5\) in Eq. (6) to add regularizations to the conventional ILP formulation. The sum of these regularization terms represents a confidence score of how coherent the predicted temporal graph is to our TemProb, which we also want to maximize. Even though a considerable amount of information from TemProb had already been encoded as features (as shown by the large improvements by System 2), these regularizations were still able to further improve the precision, recall and awareness scores. To sum up, the total improvement over the baseline system brought by TemProb is 5.9% in F\({}_{1}\) and 7.1% in awareness F\({}_{1}\), both with a notable margin. Table 7 furthermore decomposes this improvement into each TempRel label.

To compare with state-of-the-art systems, which all used gold event properties (i.e., Tense, Aspect, Modality, and Polarity), we retrained System 3 in Table 6 with these gold properties and show the results in Table 8. We reproduced the results of CAEVO9 Chambers et al. (2014) and Ning et al. (2017)10 and evaluated them on the partianglTBDense test split11. Under both metrics, the

\begin{table}
\begin{tabular}{c|c|c|c|c|c} \hline No. & System & P & R & F\({}_{1}\) & F\({}_{aware}\) \\ \hline
1 & Baseline & 48.1 & 44.4 & 46.2 & 42.5 \\
2 & +Feature: \(\{f_{r}\}_{r\in R}\) & 50.6 & 52.0 & 51.3 & 49.1 \\
3 & +Regularization & 51.3 & 53.0 & 52.1 & 49.6 \\ \hline \end{tabular}
\end{table}
Table 6: **Regularizing global methods by the prior distribution derived from TemProb.** The “+” means adding a component on top of its preceding line. F\({}_{\text{aware}}\) is the temporal awareness F-score, another evaluation metric used in TempEval3. The baseline system is to use (unregularized) ILP on top of the original system in Table 5. System 3 is the proposed. Per the McNemar’s test, System 3 is significantly better than System 1 with p\(<\)0.0005.

\begin{table}
\begin{tabular}{c|c|c|c} \hline Label & P & R & F\({}_{1}\) \\ \hline before & +0.3 & +15 & +6 \\ after & +4 & +4 & +4 \\ equal & +11 & 0 & +2 \\ includes & +17 & 0 & +0.2 \\ included & +8 & 0 & +2 \\ vague & +3 & -4 & -1 \\ \hline \end{tabular}
\end{table}
Table 7: **Label-wise performance improvement of System 3 over System 1 in Table 6.** We can see that incorporating TemProb improves the recall of _before_ and _after_, and improves the precision of all labels, with a slight drop in the recall of _vague_.

 

proposed system achieved the best performance. An interesting fact is that even without these gold properties, our System 3 in Table 6 was already better than CAEVO (on Line 1) and Ning et al. (2017) (on Line 2) in both metrics. This is appealing because in practice, those gold properties may not exist, but our proposed system can still generate state-of-the-art performance without them.

For readers who are interested in the complete TBDense dataset, we also performed a naive augmentation as follows. Recall that System 3 only makes predictions to a subset of the complete TBDense dataset. We kept this subset of predictions, and filled the missing predictions by Ning et al. (2017). Performances of this naively augmented proposed system is compared with CAEVO and Ning et al. (2017) on the _complete_ TBDense dataset. We can see that by replacing with predictions from our proposed system, Ning et al. (2017) got a better precision, recall, F\({}_{1}\), and awareness F\({}_{1}\), which is the new state-of-the-art on all reported performances on this dataset. Note that the awareness F\({}_{1}\) scores on Lines 4-5 are consistent with reported values in Ning et al. (2017). To our knowledge, the results in Table 8 is the first in literature that reports performances in both metrics, and it is promising to see that the proposed method outperformed state-of-the-art methods in both metrics.

## 5 Conclusion

Temporal relation (TempRel) extraction is an important and challenging task in NLP, partly due to its strong dependence on prior knowledge. Motivated by practical examples, this paper argues that a resource of the temporal order that events _usually_ follow is helpful. To construct such a resource, we automatically processed a large corpus from NYT with more than 1 million documents using an existing TempRel extraction system and obtained the _TEMporal relation PRObabilistic knowledge Base_ (TemProb). The TemProb is a good showcase of the capability of such prior knowledge, and it has shown its power in improving existing TempRel extraction systems on a benchmark dataset, TBDense. The resource and the system reported in this paper are both publicly available12 and we hope that it can foster more investigations into time-related tasks.

Footnote 12: [http://cogcomp.org/page/publication_view/830](http://cogcomp.org/page/publication_view/830)

## Acknowledgements

We thank all the reviewers for providing useful comments. This research is supported in part by a grant from the Allen Institute for Artificial Intelligence (allenai.org); the IBM-ILLINOIS Center for Cognitive Computing Systems Research (C3SR) - a research collaboration as part of the IBM AI Horizons Network; by DARPA under agreement number FA8750-13-2-0008; and by the Army Research Laboratory (ARL) under agreement W911NF-09-2-0053.

The U.S. Government is authorized to reproduce and distribute reprints for Governmental purposes notwithstanding any copyright notation thereon. The views and conclusions contained herein are those of the authors and should not be interpreted as necessarily representing the official policies or endorsements, either expressed or implied, of DARPA or the U.S. Government. Any opinions, findings, conclusions or recommendations are those of the authors and do not necessarily reflect the view of the ARL.

## References

* Bethard et al. (2015) Steven Bethard, Leon Derczynski, Guergana Savova, James Pustejovsky, and Marc Verhagen. 2015. SemEval-2015 Task 6: Clinical TempEval. In _Proceedings of the 9th International Workshop on Semantic Evaluation (SemEval 2015)_. Association for

\begin{table}
\begin{tabular}{c|c|c|c|c|c} \hline No. & System & P & R & F\({}_{1}\) & F\({}_{\text{aware}}\) \\ \hline \multicolumn{6}{c}{_Partial TBDense*: Focus of this work._} \\ \hline
1 & CAEVO & 52.3 & 43.7 & 47.6 & 46.7 \\
2 & Ning et al. (2017) & 47.4 & 56.3 & 51.5 & 49.1 \\
3 & Proposed & 50.0 & 62.4 & 55.5 & 52.8 \\ \hline \multicolumn{6}{c}{_Complete TBDense: Naive Augmentation._} \\ \hline
4 & CAEVO & 51.8 & 32.6 & 40.0 & 45.7 \\
5 & Ning et al. (2017) & 46.2 & 40.6 & 43.2 & 48.5 \\
6 & Proposed** & 47.2 & 42.4 & 44.7 & 49.2 \\ \hline \end{tabular}

*Note that TemProb is only available for events extracted by SRL (See Sec. 3.2.2 for details).

***Augment the output of Line 3 with predictions from Ning et al. (2017).

\end{table}
Table 8: **Comparison of the proposed TempRel extraction method with two best-so-far systems using two metrics. Since TemProb is only on SRL verb events, _Partial_ TBDense is the focus of our work, where we can see significant improvement brought by simply using the prior knowledge from TemProb. Per the McNemar’s test, Line 3 is better than Line 2 with p\(<\)0.0005. For interested readers, we also naively augmented the proposed method to the _complete_ TBDense and show state-of-the-art performance on it.

 

Computational Linguistics, Colorado, 2013.
* [Feund and Schapire1985] Timel M. Feund and R. Schapire. 1985. Large margin classification using the Perceptron algorithm. In _Proceedings of the Annual ACM Workshop on Computational Learning Theory (COLT)_. pages 209-217.
* [Graff2002] David Graff. 2002. The AQUAINT corpus of english news text. _Linguistic Data Consortium, Philadelphia_ .
* [Hoffart et al.2013] Johannes Hoffart, Fabian M. Suchanek, Klaus Berberich, and Gerhard Weikum. 2013. Yago2: A spatially and temporally enhanced knowledge base from wikipedia (extended abstract). In _Proceedings of the International Joint Conference on Artificial Intelligence (IJCAI)_.
* [Hovy et al.2013] Eduard Hovy, Teruko Mitamura, Felisa Verdejo, Jun Araki, and Andrew Philpot. 2013. Events are not simple: Identity, non-identity, and quasi-identity. In _Workshop on Events_.
* [Jiang et al.2016] Tingsong Jiang, Tianyu Liu, Tao Ge, Lei Sha, Baobao Chang, Sujian Li, and Zhifang Sui. 2016. Towards time-aware knowledge graph completion. In _Proceedings of COLING 2016, the 26th International Conference on Computational Linguistics: Technical Papers_. The COLLNG 2016 Organizing Committee, Osaka, Japan, pages 1715-1724.
* evaluating temporal information understanding with question answering. In _Proceedings of the 9th International Workshop on Semantic Evaluation (SemEval 2015)_. pages 792-800.
* [Mani et al.2006] Inderjeet Mani, Marc Verhagen, Ben Wellner, Chong Min Lee, and James Pustejovsky. 2006.

 

Machine learning of temporal relations. In _Proceedings of the Annual Meeting of the Association for Computational Linguistics (ACL)_. pages 753-760.
* Minard et al. [2015] Anne-Lyse Minard, Manuela Speranza, Eneko Agirre, Itziar Aldabe, Marieke van Erp, Bernardo Magnini, German Rigau, Ruben Urizar, and Fondazione Bruno Kessler. 2015. SemEval-2015 Task 4: TimeLine: Cross-document event ordering. In _Proceedings of the 9th International Workshop on Semantic Evaluation (SemEval 2015)_. pages 778-786.
* Mirza and Tonelli [2016] Paramita Mirza and Sara Tonelli. 2016. CATENA: CAusal and TEmporal relation extraction from NATural language texts. In _The 26th International Conference on Computational Linguistics_. pages 64-75.
* Mitamura et al. [2015] T. Mitamura, Y. Yamakawa, S. Holm, Z. Song, A. Bies, S. Kulick, and S. Strassel. 2015. Event nugget annotation: Processes and issues. In _Proceedings of the Workshop on Events at NAACL-HLT_.
* Ning et al. [2017] Qiang Ning, Zhili Feng, and Dan Roth. 2017. A structured learning approach to temporal relation extraction. In _Proceedings of the Conference on Empirical Methods for Natural Language Processing (EMNLP)_. Copenhagen, Denmark.
* Peng and Roth [2016] Haoruo Peng and Dan Roth. 2016. Two discourse driven language models for semantics. In _Proc. of the Annual Meeting of the Association for Computational Linguistics (ACL)_.
* Peng et al. [2016] Haoruo Peng, Yangqiu Song, and Dan Roth. 2016. Event detection and co-reference with minimal supervision. In _Proc. of the Conference on Empirical Methods in Natural Language Processing (EMNLP)_.
* Pustejovsky et al. [2003] James Pustejovsky, Patrick Hanks, Roser Sauri, Andrew See, Robert Gaizauskas, Andrea Setzer, Dragomir Radev, Beth Sundheim, David Day, Lisa Ferro, et al. 2003. The TIMEBANK corpus. In _Corpus linguistics_. volume 2003, page 40.
* Rizzollo and Roth [2010] N. Rizzollo and D. Roth. 2010. Learning based java for rapid development of nlp systems. In _Proc. of the International Conference on Language Resources and Evaluation (LREC)_. Valletta, Malta.
* Roth and Yih [2004] D. Roth and W. Yih. 2004. A linear programming formulation for global inference in natural language tasks. In Hwee Tou Ng and Ellen Riloff, editors, _Proc. of the Conference on Computational Natural Language Learning (CoNLL)_. pages 1-8.
* Song et al. [2015] Zhiyi Song, Ann Bies, Stephanie Strassel, Tom Riese, Justin Mott, Joe Ellis, Jonathan Wright, Seth Kulick, Neville Ryant, and Xiaoyi Ma. 2015. From light to rich ere: Annotation of entities, relations, and events. In _Proceedings of the The 3rd Workshop on EVENTS: Definition, Detection, Coreference, and Representation_. Association for Computational Linguistics, Denver, Colorado, pages 89-98.
* Spiliopoulou et al. [2017] Evangelia Spiliopoulou, Eduard Hovy, and Teruko Mitamura. 2017. Event detection using frame-semantic parser. In _Proceedings of the Events and Stories in the News Workshop_. Association for Computational Linguistics, Vancouver, Canada, pages 15-20.
* UzZaman and Allen [2011] Naushad Uzzaman and James F Allen. 2011. Temporal evaluation. In _Proceedings of the Annual Meeting of the Association for Computational Linguistics (ACL)_. pages 351-356.
* UzZaman et al. [2013] Naushad Uzzaman, Hector Llorens, James Allen, Leon Derczynski, Marc Verhagen, and James Pustejovsky. 2013. SemEval-2013 Task 1: TempEval-3: Evaluating time expressions, events, and temporal relations. In _Second Joint Conference on Lexical and Computational Semantics_. volume 2, pages 1-9.
* Verhagen et al. [2007] Marc Verhagen, Robert Gaizauskas, Frank Schilder, Mark Hepple, Graham Katz, and James Pustejovsky. 2007. SemEval-2007 Task 15: TempEval temporal relation identification. In _SemEval_. pages 75-80.
* Verhagen and Pustejovsky [2008] Marc Verhagen and James Pustejovsky. 2008. Temporal processing with the TARSQI toolkit. In _22nd International Conference on on Computational Linguistics: Demonstration Papers_. pages 189-192.
* Verhagen et al. [2010] Marc Verhagen, Roser Sauri, Tommaso Caselli, and James Pustejovsky. 2010. SemEval-2010 Task 13: TempEval-2. In _SemEval_. pages 57-62.