SlowGuess/NAACL_2021_OCR huggingface dataset:
Haoyang Wen $^{1}$ , Yanru Qu $^{1}$ , Heng Ji $^{1}$ , Qiang Ning $^{2*}$ , Jiawei Han $^{1}$ , Avirup Sil $^{3}$ , Hanghang Tong $^{1}$ , Dan Roth $^{4}$
<sup>1</sup> University of Illinois at Urbana-Champaign <sup>2</sup>Amazon
$^{3}$  IBM Research AI  $^{4}$ University of Pennsylvania
{wen17, yanruqu2, hengji, hanj, htong}@illinois.edu
qning@amazon.comavi@us.ibm.com danroth@seas.upenn.edu
Grounding events into a precise timeline is important for natural language understanding but has received limited attention in recent work. This problem is challenging due to the inherent ambiguity of language and the requirement for information propagation over inter-related events. This paper first formulates this problem based on a 4-tuple temporal representation used in entity slot filling, which allows us to represent fuzzy time spans more conveniently. We then propose a graph attention network-based approach to propagate temporal information over document-level event graphs constructed by shared entity arguments and temporal relations. To better evaluate our approach, we present a challenging new benchmark on the ACE2005 corpus, where more than  $78\%$  of events do not have time spans mentioned explicitly in their local contexts. The proposed approach yields an absolute gain of  $7.0\%$  in match rate over contextualized embedding approaches, and  $16.3\%$  higher match rate compared to sentence-level manual event time argument annotation. $^{1}$
Understanding and reasoning about time is a crucial component for comprehensive understanding of evolving situations, events, trends and forecasting event abstractions for the long-term. Event time extraction is also useful for many downstream Natural Language Processing (NLP) applications such as event timeline generation (Huang and Huang, 2013; Wang et al., 2015; Ge et al., 2015; Steen and Markert, 2019), temporal event tracking and prediction (Ji et al., 2009; Minard et al., 2015), and temporal question answering (Llorens et al., 2015; Meng et al., 2017).
In order to ground events into a timeline we need to determine the start time and end time of each event as precisely as possible (Reimers et al., 2016). However, the start and end time of an event is often not explicitly expressed in a document. For example, among 5,271 annotated event mentions in the Automatic Content Extraction (ACE2005) corpus<sup>2</sup>, only 1,100 of them have explicit time argument annotations. To solve the temporal event grounding (TEG) problem, previous efforts focus on its subtasks such as temporal event ordering (Bramsen et al., 2006; Chambers and Jurafsky, 2008; Yoshikawa et al., 2009; Do et al., 2012; Meng et al., 2017; Meng and Rumshisky, 2018; Ning et al., 2017, 2018, 2019; Han et al., 2019) and duration prediction (Pan et al., 2006, 2011; Vempala et al., 2018; Gusev et al., 2011; Vashishtha et al., 2019; Zhou et al., 2019). In this paper we aim to solve TEG directly using the following novel approaches.
To capture fuzzy time spans expressed in text, we adopt a 4-tuple temporal representation proposed in the TAC-KBP temporal slot filling task (Ji et al., 2011, 2013) to predict an event's earliest possible start date, latest possible start date, earliest possible end date and latest possible end date, given the entire document. We choose to work at the day-level and leave time scales smaller than that for future work since, for example, only  $0.6\%$  of the time expressions in the newswire documents in ACE contain smaller granularities (e.g., hours or minutes).
Fortunately, the uncertain time boundaries of an event can often be inferred from its related events in the global context of a document. For example, in Table 1, there are no explicit time expressions or clear linguistic clues in the local context to infer the time of the appeal event. But the earliest possible date of the refuse event is explicitly expressed as 2003-04-18. Since the appeal event must happen before the refuse event, we can infer
<sup>1</sup>The resource for this paper is available at https://github.com/wenhycs/NAACL2021-Event-Time-Extraction-and-Propagation-via-Graph-Attention-Networks.
Proceedings of the 2021 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, pages 62-73 June 6-11, 2021. ©2021 Association for Computational Linguistics
Malaysia's Appeal Court Friday[2003-04-18] refused to overturn the conviction and nine-year jail sentence imposed on ex-deputy prime minister Anwar Ibrahim. Anwar now faces an earliest possible release date of April 14, 2009[2009-04-14]. The former heir says he was framed for political reasons, after his appeal was rejected ... Mahathir's sacking of Anwar in September 1998[1998-09] rocked Malaysian politics ... Within weeks he was arrested and charged with ... Anwar was told Monday[2003-04-14] that he had been granted a standard one-third remission of a six-year corruption sentence for good behavior, and immediately began to serve the nine-year sentence ...
Event	Earliest Start Date	Latest Start Date	Earliest End Date	Latest End Date	Evidence
Local Context	sentence appeal	2003-04-14 -inf	2003-04-14 +inf	-inf -inf	+inf +inf
+ Sharing Arguments	sentence appeal	2003-04-14 -inf	2003-04-14 +inf	2009-04-14 2003-04-18	+inf 2003-04-18	release→Anwar→sentence refuse→Anwar→appeal
+ Temporal Relation	sentence appeal	2003-04-14 1998-09-01	2003-04-14 +inf	2009-04-14 2003-04-18	+inf 2003-04-18	sack→arrest→appeal
Table 1: Examples of temporal propagation via related events for two target events, sentence and appeal. By leveraging related events with temporal relations and shared arguments, some infinite dates can be refined with temporal boundaries. Note: The event triggers that we are focusing are highlighted in orange, time expressions in blue, and normalized TIMEX dates in subscripts. Related events are underlined.
the earliest start and the latest end date of appeal as 2003-04-18. However, there are usually many other irrelevant events that are in the same document, which requires us to develop an effective approach to select related events and perform temporal information propagation. We first use event-event relations to construct a document-level event graph for each input document, as illustrated in Figure 1. We leverage two types of event-event relations: (1) if two events share the same entity as their arguments, then they are implicitly connected; (2) automatic event-event temporal relation extraction methods such as (Ning et al., 2019) provide important clues about which element in the 4-tuple of an event can be propagated to which 4-tuple element of another event. We propose a novel time-aware graph propagation framework based on graph attention networks (GAT, Velickovic et al., 2018) to propagate temporal information across events in the constructed event graphs.
Experimental results on a benchmark, newly created on top of ACE2005 annotations, show that our proposed cross-event time propagation framework significantly outperforms state-of-the-art event time extraction methods using contextualized embedding features.
- This is the first work taking advantage of the flexibility of 4-tuple representation to formulate absolute event timeline construction.
- We propose a GAT based approach for timeline construction which effectively propagates temporal information over document-level event graphs without solving large constrained optimization problems (e.g., Integer Linear Program
ming (ILP)) as previous work did. We propose two effective methods to construct the event graphs, based on shared arguments and temporal relations, which allow the time information to be propagated across the entire document.
- We build a new benchmark with over 6,000 human annotated non-infinite time elements, which implements the 4-tuple representation for the first time as a timeline dataset, and is intended to be used for future research on absolute timeline construction.
2 A New Benchmark
2.1 4-tuple Event Time Representation
Grounding events into a timeline necessitates the extraction of the start and end time of each event. However, the start and end time of most events is not explicitly expressed in a document. To capture such uncertainty, we adopt the 4-tuple representation introduced by the TAC-KBP2011 temporal slot filling task (Ji et al., 2011, 2013). We define 4-tuple event time as four time elements for an event  $e \rightarrow \langle \tau_{\mathrm{start}}^{-}, \tau_{\mathrm{start}}^{+}, \tau_{\mathrm{end}}^{-}, \tau_{\mathrm{end}}^{+} \rangle$ , which indicate earliest possible start date, latest possible start date, earliest possible end date and latest possible end date, respectively. These four dates follow hard constraints:
$$
\left\{ \begin{array}{l} \tau_ {\text {s t a r t}} ^ {-} \leq \tau_ {\text {s t a r t}} ^ {+} \\ \tau_ {\text {e n d}} ^ {-} \leq \tau_ {\text {e n d}} ^ {+} \end{array} , \quad \left\{ \begin{array}{l} \tau_ {\text {s t a r t}} ^ {-} \leq \tau_ {\text {e n d}} ^ {-} \\ \tau_ {\text {s t a r t}} ^ {+} \leq \tau_ {\text {e n d}} ^ {+} \end{array} . \right. \right. \tag {1}
$$
3We use subscripts "start" and "end" to denote start and end time, and superscripts "—" and "+" to represent earliest and latest possible values.
The enemy have now been flown out and we're treating them including a man who is almost dead with a gunshot wound to the chest after we (Royal Marines) sent in one of our companies of about 100 men in here (Umm Kiou) this morning.
Figure 1: The example event graph. The graph with solid lines is constructed from event arguments. The graph with dash lines is constructed from temporal relations. Entities in the text are underlined and events in the text are in boldface.
Category	#
# documents	182
usenet	1
broadcast conversations	5
broadcast news	63
webblogs	26
newswire	87
# train/dev/test	92/39/51
# event mentions	2,084
# average tokens/document	436
# non-infinite elements	6,058
# infinite elements	2,278
Table 2: Data Statistics
Symbol	Explanation
wi	the i-th word of document D
D	a document, D = [w1, ..., wn]
ei	an event trigger in D
E	the event mention set of D, E = {e1, ..., em}
τi	a time element of event i, can be {τi,start, τi,start, τi,end, τi,+end}
ti	a time expression in D
T	the time set of D, T = {t1, ..., tl}
ri	a relation, either event argument roles or event temporal relations
R	relation set, R = {r1, ..., rq}
Table 3: Notations
The above temporal representation was originally designed for entity slot filling, and we regard it as an expressive way for describing events too as: (1) it allows for flexible representation of fuzzy time spans and thus, for those events that we cannot determine the accurate dates, they can also be grounded into a timeline; and (2) it allows for a unified treatment of various types of temporal information and thus makes it convenient to propagate over multiple events.
We choose the Automatic Content Extraction (ACE) 2005 dataset because it includes rich annotations of event types, entity/time/value argument roles, time expressions and their normalization results. In our annotation interface, each document is highlighted with event triggers and time expressions. The annotators are required to read the whole document and provide as precise information as possible for each element of the 4-tuple of each event. If there is no possible information for a specific time, the annotators are asked to provide +/-infinite labels.
Overall, we have annotated 182 documents from this dataset. Most of the documents are from broadcast news or newswire genres. Detailed data statistics and data splits are shown in Table 2. We annotated all the documents with two independent passes. Two experts led the final adjudication based on independent annotations and discussions with annotators since single annotation pass is likely to miss important clues, especially when the event and its associated time expression appear in different paragraphs.
The input is a document  $D = [w_{1},\ldots ,w_{n}]$ , containing event triggers  $E = [e_1,\dots ,e_m]$  and time expressions  $T = [t_{1},\ldots ,t_{l}]$ , and we use gold-standard annotation for event triggers and time expressions. Our goal is to connect the event triggers  $E$  and time expressions  $T$  scattered in a document, and estimate their association scores to select the most possible values for the 4-tuple elements. At a
high-level, our approach is composed of: (1) a text encoder to capture semantic and narrative information in local context, (2) a document-level event graph to facilitate global knowledge, (3) a graph-based time propagation model to propagate time along event-event relations, and (4) an extraction algorithm to generate 4-tuple output. Among these four components, (1) and (4) build up the minimal requirements of an extractor, which serve as our baseline model and will be described in Section 3.2. We will detail how we utilize event arguments and temporal ordering to construct the document-level event graph, namely component (2), in Section 3.3. We will present our graph-based time propagation model in Section 3.4, and wrap up our model with training objective and other details in Section 3.5.
We list notations in Table 3, which will be explained when encountered.
3.2 Baseline Extraction Model
Our baseline extraction model is an event-time pair classifier based on a pre-trained language model (Devlin et al., 2019; Liu et al., 2019; Beltagy et al., 2020) encoder. The pre-trained language models allow us to have contextualized representation for every token in a given text. We directly derive the local representation for event triggers and time expressions from the contextualized representation. The representations are denoted as  $\pmb{h}_{e_i}$  for event trigger  $e_i$  and  $\pmb{h}_{t_j}$  for time expression  $t_j$ . For events or time expressions containing multiple tokens, we take the average of token representations. Thus, all  $\pmb{h}_{e_i}$  and  $\pmb{h}_{t_j}$  are of the same dimensions.
We pair each event and time in the document, i.e.,  $\{(e_i,t_j)\mid e_i\in E,t_j\in T\}$ , to form the training examples. After obtaining event and time representations, we concatenate them and feed them into a 2-layer feed-forward neural classifier. The classifier estimates the probability of filling  $t_j$  in  $e_i$ 's 4-tuple time elements, i.e.,  $\langle \tau_{i,\mathrm{start}}^{-},\tau_{i,\mathrm{start}}^{+},\tau_{i,\mathrm{end}}^{-},\tau_{i,\mathrm{end}}^{+}\rangle$ . The probabilities are:
$$
p _ {i, j, k} = \sigma \left(\boldsymbol {w} _ {2, k} \operatorname {R e L U} \left(\boldsymbol {W} _ {1} \left[ \boldsymbol {h} _ {e _ {i}}; \boldsymbol {h} _ {t _ {j}} \right] + \boldsymbol {b} _ {1}\right) + b _ {2, k}\right) \tag {2}
$$
where  $\sigma(\cdot)$  is sigmoid function, and  $W_{1,2}$  and  $b_{1,2}$  are learnable parameters. In short, we use  $\tau_{i,k}$  to represent the  $k^{th}$  element in  $\tau_{i}$  ( $k \in \{1,2,3,4\}$ ) and  $p_{i,j,k}$  represents a probability that  $t_j$  fills in the  $k^{th}$  element of 4-tuple  $\tau_{i}$ . The baseline model consists of 4 binary classifiers, one for each element of the 4-tuple.
When determining the 4-tuple for each event  $e_i$ , we estimate the probability of  $t_1$  through  $t_l$ . For each element, we take the time expression with the highest probability to fill in this element. A practical issue is that the same time is often expressed by different granularity levels, such as 2020-01-01 and 2020-W1, following the most common TIMEX format (Ferro et al., 2005). To uniformly represent all the time expressions and allow certain degree of uncertainty, we introduce the following 2-tuple normalized form for time expressions, which indicates the time range of  $t_j$  by two dates,
$$
t _ {i} \rightarrow \left\langle t _ {i} ^ {-}, t _ {i} ^ {+} \right\rangle \tag {3}
$$
where  $t_*^-$  represents the earliest possible dates and  $t_*^+$  represents the latest possible dates.
We also make a simplification that the earliest possible values can only fill in earliest possible dates, i.e.,  $T^{-} = \{t_{1}^{-},\ldots ,t_{l}^{-}\} \mapsto \tau_{\mathrm{start}}^{-},\tau_{\mathrm{end}}^{-}$ , similarly for the latest dates,  $T^{+} = \{t_{1}^{+},\dots ,t_{l}^{+}\} \mapsto \tau_{\mathrm{start}}^{+},\tau_{\mathrm{end}}^{+}$ . This constraint can be relaxed in future work. Here is an example of how we determine the binary labels for event-time pairs. If the 4-tuple time for an event is  $\langle 2020 - 01 - 01,2020 - 01 - 03,2020 - 01 - 01,2020 - 01 - 07\rangle$  and the 2-tuple for time expression 2020-w1 is  $\langle 2020 - 01 - 01,2020 - 01 - 07\rangle$ , then the classification labels of this event-time pair will be  $\langle \mathrm{True},\mathrm{False},\mathrm{True},\mathrm{True}\rangle$ .
3.3 Event Graph Construction
Before we conduct the global time propagation, we first construct document-level event graphs. In this paper, we focus on two types of event-event relations: (1) shared entity arguments, and (2) temporal relations.
Event Argument Graph. Event argument roles provide local information about events and two events can be connected via their shared arguments.
We denote the event-argument graph as  $G_{\mathrm{arg}} = \{(e_i, v_j, r_{i,j})\}$ , where  $e_i$  represents an event,  $v_j$  represents an entity or a time expression, and  $r_{i,j}$  represents the bi-directed edge between  $e_i$  and  $v_j$ , namely the argument role. For example, in Figure 1, there will be two edges between the "sent" event  $(e_1)$  and the entity "Royal Marines"  $(v_1)$ , namely  $(e_1, v_1, \text{AGENT})$  and  $(v_1, e_1, \text{AGENT})$ . In addition, we add a self-loop for each node in this graph. The graph can be constructed by Information Extraction (IE) techniques and we use gold-standard event
annotation from ACE 2005 dataset in our experiments.
Event Temporal Graph. Event-event temporal relations provide explicit directions to propagate time information. If we know that an attack event happened before an injury event, the lower-bound end date of the attack can possibly be the start date of the injury. We denote the event temporal graph as  $G_{\mathrm{temp}} = \{(e_i,e_j,\gamma_{i,j})\}$ , where  $e_i$  and  $e_j$  denote events, and  $\gamma_{i,j}$  denotes the temporal order between  $e_i$  and  $e_j$ . Similar to  $G_{\mathrm{arg}}$ , we also add a self-loop in  $G_{\mathrm{temp}}$  and edges for two directions. For example, for a BEFORE relation from  $e_1$  to  $e_2$ , we will add two edges,  $(e_1,e_2, \mathrm{BEFORE})$  and  $(e_2,e_1, \mathrm{AFTER})$ . We only consider BEFORE and AFTER relations when constructing the event temporal graph. To propagate time information, we also use local time arguments as in event argument graphs.
We apply the state-of-the-art event temporal relation extraction model (Ning et al., 2019) to extract temporal relations for event pairs that appear in the same sentence or two consecutive sentences, and we only keep the relations whose confidence score is over  $90\%$ .
3.4 Event Graph-based Time Propagation
After obtaining the document-level graphs  $G_{\mathrm{arg}}$  and  $G_{\mathrm{temp}}$ , we design a novel time-aware graph neural network to perform document-level 4-tuple propagation.
Graph neural networks (Dai et al., 2016; Kipf and Welling, 2017; Hamilton et al., 2017; Schlichtkrull et al., 2018; Velickovic et al., 2018) have shown effective for relational reasoning (Zhang et al., 2018; Marcheggiani et al., 2018). We adopt graph attention networks (GAT, Velickovic et al., 2018) to propagate time through event-argument or event-event relations. GAT are proposed to aggregate and update information for each node from its neighbors through attention mechanism. Compared to the original GAT, we further include relational embedding for edge labels when performing attention to capture various types of relations between each event and its neighboring events.
The graphs  $G_{\mathrm{arg}}$  and  $G_{\mathrm{temp}}$  together with the GAT model are placed in the intermediate layer of our baseline extraction model (Section 3.2), i.e., between the pre-trained language model encoder and the 2-layer feed-forward neural classifier (Eq. (2)). For clarity, we denote all events and entities as
nodes  $V = \{v_{1},\dots ,v_{n}\}$ , and we use  $r_{i,j}$  to denote their relation types. More specifically, we stack several layers of GAT on top of the contextualized representations of nodes  $h_{v_i}$ . And we follow Vaswani et al. (2017) to use multi-head attention for each layer. We use the simplified notation  $h_{v_i}$  to describe one of the attention heads for  $h_{v_i}^k$ .
$$
\alpha_ {i j} = \frac {\exp \left(a _ {i j}\right)}{\sum_ {k \in \mathcal {N} (i)} \exp \left(a _ {i k}\right)} \tag {4}
$$
$$
\boldsymbol {h} _ {v _ {i}} ^ {\prime} = \operatorname {E L U} \left(\sum_ {j \in \mathcal {N} (i)} \alpha_ {i j} \boldsymbol {W} _ {5} \boldsymbol {h} _ {v _ {j}}\right) \tag {5}
$$
where ELU is exponential linear unit (Clevert et al., 2016),  $a_{ij}$  is the attention coefficient of node  $v_{i}$  and  $v_{j}$ ,  $\alpha_{ij}$  is the attention weight after softmax, and  $\pmb{h}_{v_i}$  and  $\pmb{h}_{v_i}^{\prime}$  are the hidden states of node  $v_{i}$  before and after one GAT layer, respectively. We use  $\mathcal{N}(i)$  to denote the neighborhood of  $v_{i}$ . The attention coefficients are calculated through
$$
a _ {i j} = \sigma \left(\boldsymbol {w} _ {4} \left[ \boldsymbol {W} _ {3} \boldsymbol {h} _ {v _ {i}}; \boldsymbol {W} _ {3} \boldsymbol {h} _ {v _ {j}}; \phi_ {r _ {i, j}} \right]\right) \tag {6}
$$
where  $\sigma$  is LeakyReLU (Clevert et al., 2016) activation function.  $\phi_{r_{i,j}}$  is the learnable relational embedding for relation type of  $r_{i,j}$  that we further add compared to the original GAT.
We concatenate  $m$  different attention heads to compute the representation of  $v_{i}$  for the next layer after performing attention for each head,
$$
\boldsymbol {h} _ {v _ {i}} ^ {\prime} = \left| \right| \begin{array}{l} m \\ k = 1 \end{array} \boldsymbol {h} _ {v _ {i}} ^ {\prime k}. \tag {7}
$$
We stack  $n_l$  GAT layers to obtain the final representations for events and time. These representations are fed into the 2-layer feed-forward neural classifier in Eq. (2) to generate the corresponding probabilities.
Since we model the 4-tuple extraction task by four binary classifiers, we adopt the log loss as our model objective:
$$
\begin{array}{l} \mathcal {L} (\tau_ {i, k}, t _ {j}) = \mathbb {1} \left(\tau_ {i, k} = t _ {j}\right) \log p _ {i, j, k} \\ + \mathbb {1} \left(\tau_ {i, k} \neq t _ {j}\right) \log \left(1 - p _ {i, j, k}\right) \tag {8} \\ \end{array}
$$
Since the 4-tuple elements are extracted from time expressions, the model cannot generate  $+ / - \inf$  (infinite) output. To address this issue,
we adopt another hyperparameter, inf threshold, and convert those predicted time values with scores lower than the threshold into  $+ / - \inf$  values. That is, we regard the probability  $p_{i,j,k}$  also as a confidence score. A low score indicates the model cannot determine the results for some 4-tuple elements. Thus it is natural to set those elements as inf. When this case happens in  $\tau_{\mathrm{start}}^{-}$  or  $\tau_{\mathrm{end}}^{-}$ , we correct the value to be  $-\inf$ , and when it is  $\tau_{\mathrm{start}}^{+}$  or  $\tau_{\mathrm{end}}^{+}$ , we set the value to be  $+\inf$ . This threshold and its searching will be applied to both baseline extract and GAT-based extraction systems. The extraction model may generate 4-tuples that do not follow the constraints on Eq. (1) and we leave enforcing the constraints for future work.
4.1 Data and Experiment Setting
We conduct our experiments on previously introduced annotated data. Statistics of the dataset and splits are shown in Table 2.
Experiment Setup. We compare our proposed graph-based time propagation model with the following baselines: