Toward Consistent and Informative Event-Event Temporal Relation Extraction
Year: 2023

Abstract:
Event-event temporal relation extraction aims to extract the temporal order between a pair of event mentions, which is usually used to construct temporal event graphs. However, event graphs generated by existing methods are usually globally inconsistent (event graphs containing cycles), semantically irrelevant (two unrelated events having temporal links), and context unaware (neglecting neighborhood information of an event node). In this paper, we propose a novel event-event temporal relation extraction method to address these limitations. Our model combines a pretrained language model and a graph neural network to output event embeddings, which captures the contextual information of event graphs. Moreover, to achieve global consistency and semantic relevance, (1) event temporal order should be in accordance with the norm of their embeddings, and (2) two events have temporal relation only if their embeddings are close enough. Experimental results on a real-world event dataset demonstrate that our method achieves state-of-the-art performance and generates high-quality event graphs.

heading: 1 Introduction
text: Event-event temporal relation extraction aims to extract the temporal order between a pair of event mentions in natural language text (i.e., an event is BEFORE or AFTER another event), which is essential for constructing temporal event graphs. Event-event temporal relation extraction enables researchers to understand the dynamics of complex events, and benefits a variety of downstream tasks, including event graph construction (Li et al., 2018), future event prediction (Li et al., 2021; Du et al., 2022; Wang et al., 2022; Jin et al., 2022), question answering (Souza Costa et al., 2020; Wang et al., 2021), and summarization (Glavaš and Šnajder, 2014).
Researchers have proposed many methods (Dligach et al., 2017; Han et al., 2020; Wen and Ji,
2021) to tackle this challenging task. Previous work usually formulates the problem as a pairwise classification task (Dligach et al., 2017; Han et al., 2020; Wen and Ji, 2021). However, they have three major issues when applied to constructing the temporal event graph:
(1) Global inconsistency. Local pairwise classification is likely to introduce conflicting predictions when constructing temporal event graphs. Figure 1a shows an example of conflicting local predictions, where yellow links (e.g., DIE → INJURE) conflict with blue links (e.g., DETONATE → INJURE). Although previous work can address conflicts through inference methods such as Integer Linear Programming (Bramsen et al., 2006; Han et al., 2019), resolving this issue directly in temporal relation extraction models yet receives limited attention. (2) Semantic irrelevance. Existing methods output a predicted temporal relation for any two given atom events, regardless of their semantic relevance. For example, as shown in Figure 1b, given two events MEDICAL INTERVENTION and SENTENCE, existing models will predict that there is a temporal link from MEDICAL INTERVENTION to SENTENCE. Though it is very likely that MEDICAL INTERVENTION happens before SENTENCE in a real bombing event, those two events have no direct semantic relation, which makes the predicted temporal link semantically irrelevant.
(3) Context unawareness. Events with sharing arguments are usually closely related in a temporal event graph, which provides valuable information about the nature of a particular event (Vo and Bagheri, 2019). As shown in Figure 1c, CRIMINAL (rather than VICTIM) is shared by SENTENCE event and DIE event, so it is not likely that the MOURN event follows the DIE (yellow link). However, existing work considers information from candidate event pairs only, while ignoring those rich connections among other related events.
In this paper, we propose a new event-event tem-
23
poral relation extraction approach that addresses the above limitations of existing methods. The goal of our approach is to learn event representations that are globally consistent, semantically relevant, and context-aware. As shown in Figure 2, given an input document as well as the entity mentions, we first use off-the-shelf information extraction tools (Du et al., 2022) to extract arguments of events. We then use a pretrained language model (PLM, Devlin et al., 2018) to encode events/arguments and get their PLM-based embeddings. To allow events to be aware of their contextual information, we construct an initial event graph consisting of events/arguments as nodes and eventargument links as edges, then use a graph neural network (GNN, Scarselli et al., 2008) to aggregate neighborhood information for each node iteratively and get their GNN-based embeddings. The PLMbased and GNN-based embeddings are combined together as the final embeddings of events.
Moreover, to ensure that the learned event embeddings are globally consistent and semantically relevant, we hypothesize that the event embedding space should be geometrically meaningful, in which event embeddings satisfy the following two rules: (1) The temporal order of events is in accordance with the norm of event embeddings. Specifically, if event A happens before event B, then the embedding norm of event A should be smaller than that of event B. (2) There exists a temporal link between two events if and only if their embeddings are close enough to each other in the event embedding space. Specifically, if events A and B are connected by a temporal edge (either A happens before B or after B), then the distance between A’s and B’s embedding should be smaller than a predefined threshold, and vice versa. The first rule ensures that the constructed event graph is
globally consistent, and the second rule ensures that there will be a temporal link between two events only if they are semantically relevant. We implement these two rules in our model by minimizing a corresponding margin-based loss w.r.t the model parameters, thus the whole model can be trained in an end-to-end fashion.
We conduct experiments on the Event Story Line dataset (Caselli and Vossen, 2017). The experimental results demonstrate that our proposed method achieves state-of-the-art performance on event-event temporal relation extraction. We also show that compared with baseline methods, event graphs generated by our method are globally consistent and semantically relevant.
In summary, our contributions are as follows:
• We review the literature on event-event temporal relation extraction thoroughly, and observe that a well-behaved event temporal relation extraction method should be globally consistent, semantically relevant, and context aware.
• Methodologically, we use graph neural networks to process event graphs and learn event representations, which enables the model to learn event embeddings that are context aware. Moreover, we use the distance between event embeddings as the criterion for judging the existence of event temporal edges, and use the norm of event embeddings as the criterion for determining the direction of event temporal edges, which enables our model to be globally consistent and semantically relevant.
• We conduct extensive experiments on eventevent temporal relation extraction task, and the results demonstrate that our proposed method achieves substantial improvements over state-of-the-art baseline methods.

heading: 2 Problem Formulation
text: The event-event temporal relation extraction problem is formulated as follows. Given a document, we use {e1, e2, · · · } to denote the set of event mentions, and {a1, a2, · · · } to denote the set of argument mentions. An event node ei and an argument aj are connected by a link if aj is an argument of ei. The event mentions are obtained from the gold standard annotations for the dataset. The arguments and event-argument links can be obtained by applying off-the-shelf information extraction (IE) tools (Luan et al., 2019; Wen et al., 2021) to the input document. We also assume they are given as our input.
Our task is to predict the temporal relation between a pair of events (ei, ej), which can be BEFORE, AFTER, or NO RELATION. The predicted results can then be used to construct a temporal event graph, in which each node represents an event, and each directed edge ei → ej represents a temporal relation indicating that event ei happens before event ej (or equivalently, event ej happens after event ei). If the relation type between ei and ej is predicted as NO RELATION, then there is no edge between the two event nodes in the temporal event graph. Our goal is to propose a globally consistent and semantically relevant event-event temporal relation extraction method, so that the generated temporal event graph is valid (no conflict), concise (only related events can be connected), and meaningful (temporal links should be aware of the meaning of event nodes).

heading: 3 Our Approach
text: The overall framework of our proposed approach is shown in Figure 2. In this section, we will introduce each part of the framework in detail.

heading: 3.1 PLM Module
text: Pretrained language models (PLMs) are usually trained on a large corpus and thus is able to encode words as vector representations while preserving their semantics. Following most existing methods, we feed an input document into a PLM1 first to obtain an initial vector representation for each token in the document. Specifically, for a document with a sequence of tokens {w1, w2, · · · }, we first sum the token, segment, and positional embeddings for each token to compute its initial input representation {h01,h02, · · · }, and then compute an output representation for each PLM layer l:
{hl1, · · · } = PLM-Layer({hl−11 , · · · }) (1)
for l = 1, · · · , L, where PLM-Layer(·) is a single PLM encoder layer, whose parameters are initialized using a pretrained model, and L is the number of PLM layers. We suggest readers to refer to Devlin et al. (2018) for technical details of these layers.
The representation of event mention ei output by the last PLM layer is denoted as hLi . If an event mention consists of multiple tokens, we simply average the embeddings of all tokens in this event mention. Finally, we use an MLP with two hidden
1In our case, we use BERT (Devlin et al., 2018).
layers to compute the final PLM-based representation of event ei:
hPLMei = MLP ( hLi ) . (2)

heading: 3.2 GNN Module
text: Note that in a temporal event graph, an event is usually closely related to its contextual events, which share common argument entities with the given event. Contextual events provide valuable information about the nature of a particular event and help improve the performance of temporal link prediction. As shown in Figure 1c, The contextual events of the right DIE (i.e., SENTENCE, ARREST) and the contextual events of the left DIE (i.e., MOURN) indicate that they are associated with criminal and victim, respectively, so the right DIE event should not be followed by a MOURN event.
To let our model be aware of contextual event information, we first construct an initial event graph where nodes represent event mentions and argument mentions extracted from the given input document, and edges represent event-argument links. Then we use Graph Neural Networks (GNNs, Kipf and Welling, 2017) to perform message passing on the initial event graph and learn event representations. Specifically, for an initial event graph G, we use ski to denote the representation of node i ∈ G at iteration k (which can be either an event or an entity). Then the node representation is updated by aggregating its neighborhood information:
ski = σ ( Wk ∑ j∈N (i)∪{i} αijs k−1 j ) (3)
for k = 1, · · · ,K, where K is the number of GNN layers.
For each node i, N (·) denotes the set of its neighbors 2. αij = 1/ √ |N (i)| · |N (j)| is the weight coefficient. s0i is initialized as the embedding of event mention ei (i.e., s0i = h L i ), which is provided by PLM in Section 3.1. The output of the GNN’s last layer is taken as the final GNN-based representation of event ei:
sGNNei = s K i . (4)

heading: 3.3 Globally Consistent and Semantically Relevant Event Representation
text: After obtaining the PLM-based and GNN-based event representations, we concatenate these two
2To alleviate the sparsity issue of event graphs, all edges in event graphs are treated undirected when counting neighbors.
types of embeddings for each event and get the final event embedding:
ei = CONCAT ( hPLMei , s GNN ei ) . (5)
To predict the temporal relation between a pair of events, a straightforward way is to train a classifier on their embeddings, for example, an MLP that takes the concatenation of two event embeddings as input and outputs labels of BEFORE, AFTER, or NO RELATION. However, the trained classifier is not guaranteed to be globally consistent (no cycle in event graphs) and semantically relevant (temporal links only exist between events that are closely related), which makes the predicted temporal event links invalid and irrelevant.
To address these issues, we hypothesize that the event embedding space should be geometrically meaningful, and event embeddings should satisfy the following two constraints:
• The temporal order of events is in accordance with the norm 3 of event embeddings. Specifically, if there is a temporal link from event ei to event ej , then the length of event ei’s embedding should be smaller than the length of event ej’s, embedding:
ei → ej ⇒ ∥ei∥ < ∥ej∥. (6)
It is clear to see that event graphs will be cyclefree under the above constraint. Otherwise, assume that there is cycle ei → ej → · · · → ei, then according to Eq. (6), we have ∥ei∥ < ∥ej∥ < · · · < ∥ei∥, which is impossible.
• There exists a temporal relation between two events if and only if their embeddings are close enough in the event embedding space, since we assume that a temporal relation is meaningful only if the two events are semantically related. Specifically, if events ei and ej are connected by a temporal edge (ei happens either before or after ej), then the distance between ei’s and ej’s embeddings should be less than a threshold δ that is a pre-defined real positive number, and vice versa:
ei → ej ∨ ej → ei ⇔ ∥ei − ej∥ < δ. (7)
Under the constraint in Eq. (7), two events can be connected by a temporal link only if their
3We use L2 norm in this paper.
embeddings are close to each other, which discourages the model from predicting a temporal link for two events that are distant in the timeline. In this way, our model will learn to output a “minimal” temporal event graph that preserves its essential chronological structure.

heading: 3.4 Model Training and Inference
text: Training. Each training document consists of a set of event temporal links used for training the model. According to the ground-truth label of an event temporal link, the loss function is designed as follows:
• If r(ei, ej) = BEFORE, i.e., event ei happens before event ej , then the loss term for event pair (ei, ej) is
LBEFOREij = [ ∥ei∥−∥ej∥ ]+ + [ ∥ei−ej∥−δ ]+ ,
where the function [·]+ = max(·, 0). The first term encourages the embedding length of ei to be smaller than ej , and the second term encourages the distance between ei’s and ej’s embedding to be less than δ.
• Similarly, if r(ei, ej) = AFTER, i.e., event ei happens after event ej , then the loss term for event pair (ei, ej) is
LAFTERij = [ ∥ej∥−∥ei∥ ]+ + [ ∥ej−ei∥−δ ]+ .
• Otherwise, if r(ei, ej) = NO RELATION, i.e., there is no explicit temporal ordering between ei and ej , then the loss term for event pair (ei, ej) is
LNO-RELij = [ δ − ∥ei − ej∥ ]+ ,
which encourages the distance between ei’s and ej’s embedding to be larger than δ.
The total loss function of our model is therefore as follows: L = ∑
D∈D
∑
(ei,ej)∈D
( 1[r(ei, ej) = BEFORE]L BEFORE ij
+ 1[r(ei, ej) = AFTER]L AFTER ij
+ 1[r(ei, ej) = NO RELATION]L NO-REL ij
) ,
where D is the training dataset, and D ∈ D is a training document. The whole model can thus be trained by minimizing the above total loss using gradient-based optimization methods.
Inference. In the inference stage, to predict the temporal relation between two events ei and ej , we first calculate the event embeddings of ei and ej using the PLM module and GNN module in our model, then output the label of (ei, ej) according to the following criteria:
r(ei, ej) =   BEFORE, if ∥ei − ej∥ < δ ∧ ∥ei∥ < ∥ej∥, AFTER, if ∥ei − ej∥ < δ ∧ ∥ei∥ ≥ ∥ej∥, NO RELATION, if ∥ei − ej∥ ≥ δ.

heading: 4 Experiments

heading: 4.1 Datasets
text: We conduct experiments on Event StoryLine Corpus (Caselli and Vossen, 2017), which contains 258 documents on 22 calamity topics.4 It consists of human-annotated event temporal links: RISING_ACTION, which means the former event happens earlier than and implicitly enables the later event, or FALLING_ACTION, which means the former event happens later than and is the outcome/effect of the later event. We map RISING_ACTION to BEFORE and FALLING_ACTION to AFTER in our method.
The statistics of the dataset are summarized in Table 1. We split the documents into train, validation, and test sets. There are also entity annotations in each document including location and person. We use these entity mentions as argument nodes in the initial event graph construction.

heading: 4.2 Baseline Methods
text: We compare our method with the following eventevent temporal relation extraction methods:
• BERT+MLP. Given two events ei and ej , we use BERT base model to encode each event and get their embeddings hBERTei and h BERT ej . Then the
temporal relation between ei and ej is computed by r(ei, ej) = MLP ( CONCAT(hBERTei ,h BERT ej ) ) .
4We do not conduct experiments on another popular dataset MATRES (Ning et al., 2018) because a large portion of the annotated temporal edges in MATRES are redundant and semantically irrelevant.
• GNN+MLP. This is similar to BERT+MLP, except that we use GNN to encode each event and get their embeddings. Specifically, the temporal relation between ei and ej is computed by r(ei, ej) = MLP ( CONCAT(hGNNei ,h GNN ej ) ) . • Wen and Ji (2021) propose a joint model for event-event temporal relation classification. It is the state-of-the-art event-event temporal relation extraction approach, which adopts a stackpropagation framework to incorporate relative event time prediction for temporal relation classification.
• Liu et al. (2021) propose an event causality identification model. It is an event-event causal relation identification model that uses a mechanism called event mention masking generalization. Note that this model performs a causality existence prediction on Event StoryLine Corpus. To make a fair comparison with this baseline, we modify our model output to binary classification. Specifically, the relation between two events i and j is decided by the distance between two event embeddings ei and ej : If ∥ei − ej∥ < δ, then r(ei, ej) = HAVE RELATION, otherwise r(ei, ej) = NO RELATION. In addition, to examine the effectiveness of using GNN to learn contextual information, we conduct ablation study and design the following reduced version of our model:
• Our method without GNN module, which uses the PLM-based embedding as the event embedding. Instead of Eq. (5), the final embedding of event ei is ei = hPLMei .
• Our method without PLM module, which uses the GNN-based embedding as the event embedding, i.e., the final embedding of event ei is ei = hGNNei .

heading: 4.3 Experimental Setup
text: Dataset preprocessing. Our model predicts the relation between events, which is a classification task. The ground truth annotation only includes the relation type BEFORE and AFTER, without NO RELATION. To compare with baselines, we randomly select negative samples from all event pairs that are not in the annotation set, and label them as NO RELATION. The number of negative samples is one half of annotated event pairs for each document to ensure that labels are balanced. To compare with Liu’s method (Liu et al., 2021), we merge the BEFORE and AFTER labels to HAVE RELATION, and treat all negative pairs as NO RELATION.
To construct the initial input graph, we first include event nodes which represent event mentions in the ground truth annotations. In addition, there are also annotations of person and location spans in the ground truth annotations. We add the annotations as argument nodes in the initial input graph.
We connect an event node and an argument node as an event-argument relation if they belong to the same sentence or consecutive sentences.
Evaluation metrics. We use the following metrics to evaluate our model and baseline methods:
• Accuracy. We use Precision, Recall, and F1 score to evaluate performance of our model and baseline methods. We report our averaged test performance on 5 random seeds.
• Consistency. Note that whether a temporal relation extraction model satisfies global consistency greatly affects its practical reliability. To investigate the global consistency of our model as well as baselines, we exchange the two events in an input event pair, and feed the reversed event pair into the model and obtain the prediction. For a pair of events (ei, ej), the consistent prediction of its reversed pair should be
r(ej , ei) =    BEFORE, if r(ei, ej) = AFTER, AFTER, if r(ei, ej) = BEFORE, NO-REL, if r(ei, ej) = NO-REL.
We count the number of event pairs in the test set whose reversed pair has the consistent prediction with the original pair, and define the Simple Consistency Rate (SCR) as
SCR = # consistent event pairs
# all event pairs .
Note that SCR does not consider the model’s prediction accuracy. Therefore, we define the Correct Consistency Rate (CCR) as
CCR = # consistent and correct event pairs
# all event pairs .
Hyperparameter Settings. For the GNN module, we use a three-layer GCN as the encoder, whose dimensions of hidden layers are 256, 128, and 16, respectively. For the PLM module, we use BERT base model uncased (Devlin et al., 2018) and the dimensions of the MLP hidden layers in Eq. (2) are 128 and 16, respectively. The learning rate is 10−5, the number of training epochs is 200, and δ is set to 16.

heading: 4.4 Results and Analysis
text: Comparison with baseline methods. The results of ternary and binary classification are reported in Tables 2 and 3, respectively. It is clear that our method achieves substantial gains over all baseline methods in both classification tasks. Specifically, the F1 score of our method surpass the the best baseline method by 2.1% and 11.2% in ternary and binary classification, respectively. This demonstrates that utilizing contextual information of event graphs and preserving the global consistency as well as semantic relevance are essential to improving the performance of event temporal relation extraction.
It is also worth noticing that the Simple Consistency Rate of all baseline methods are significantly lower than our method. Moreover, the Correct Consistency Rate is much lower than the Simple Consistency Rate. This is because these models do not take into account the global consistency during training and thus causing conflicts in prediction results. In contrast, our method is theoretically guaranteed to be globally consistent.
Ablation study. The results of the ablation study are shown in Tables 2 and 3. We observe a sub-
stantial performance degradation after removing the GNN module or PLM module from our model. The result demonstrates that both GNN module and PLM module are essential to learning high-quality event representations, since PLM provides general sense of events while GNN explicitly utilizes contextual information in event graphs.
Case study. The example temporal event graphs generated by Wen’s model (Wen and Ji, 2021) and our model are drawn in Figure 3. The input document is shown on the left of Figure 3, where texts in blue are event mentions and texts in orange are the annotated named entities (arguments). The graph in the middle is the temporal event graph predicted by Wen’s model. We use blue links to denote consistent temporal edges and red links to denote inconsistent temporal edges according to the prediction of Wen’s model. We observe that the prediction of Wen’s model has inconsistency problem since more than one half of the predicted temporal links are inconsistent. Specifically, the prediction of r(SPRAY, KILLED) is BEFORE whereas the prediction of r(KILLED, SPRAY) is NO RELATION. This is because Wen’s model does not consider the consistency issue, thus causes conflicts in its generated temporal event graph. In addition, Wen’s model fails to identify the relation between STRUCK and KILLED.
The graph on the right is predicted by our model. The additional green rectangles are arguments. As opposed to the middle graph, all the predictions by our model are correct and consistent. An important reason is that our model takes the contextual information of event graphs into account. For example, there are three named entities connecting STRUCK and KILLED (i.e., JONATHAN, ANDREW, and ZURANA), which provides valuable information to identify the temporal relation between them.

heading: 5 Related Work
text: Event-event temporal relation extraction can be viewed as a classification task that predicts the relation type between two event mentions. In general, existing event-event temporal relation extraction methods can be classified into two categories: traditional rule-based methods and neural network based methods.
The traditional rule-based methods apply linguistic rules to the features extracted from documents to predict the relation between a given event pair. For example, Laokulrat et al. (2013) propose a sys-
tem that uses a rule-based approach as baseline to determine temporal links and a machine learning classifier to filter out baseline candidates. Chambers et al. (2014) design a sieve-based architecture CAEVO that applies a sequence of temporal relation classifiers to label event-event temporal relations. This supports a combination of both rule-based and machine learned classifiers. However, these rule-based methods require substantial manual design of rules, which greatly limits their usage in practice. Moreover, rules are usually not comprehensive enough to capture the complex event-event temporal relations.
Another line of related work focuses on the neural network based methods, which extracts eventevent temporal relations via deep neural networks and pre-trained language models. For example, Wang et al. (2020) introduce a joint constrained learning framework that incorporates contextual features encoded with pre-trained language models and external knowledge from commonsense knowledge bases. Wen and Ji (2021) adopt a stackpropagation framework to combine relative time prediction and event-event temporal relation classification. However, they do not consider global consistency and semantic relevance of the generated event graphs.

heading: 6 Conclusion and Future Work
text: In this paper, we propose a globally consistent, semantically relevant, and context aware event-event relation extraction framework, which addresses the limitations of existing methods. Our model uses a pretrained language model module and graph neural network module to jointly represent event graphs. In addition, we make the event embedding space geometrically meaningful by imposing two constraints on event embeddings: event temporal order should be in accordance with event embedding norm, and event temporal relations should only exist between events whose embeddings are close enough. Experiments demonstrate that our method significantly outperforms baselines by generating accurate and globally consistent temporal event graphs.
In the future, we aim to incorporate external background knowledge and commonsense knowledge into our framework. We also plan to make use of the generated temporal event graphs in downstream tasks, such as future event prediction and question answering.
Limitations
In the current design setting, our proposed model is only able to classify temporal relations between event pairs into one of three classes: BEFORE, AFTER, and NO RELATION. Our model should be more practically useful if we can extend it to predict more relation types in addition to temporal relations, such as PARENT-CHILD and CAUSECAUSED_BY relations. We believe that our model is able to make such extension without too much modification.
In addition, as mentioned in the previous section, our model does not make use of any external knowledge, e.g., commonsense knowledge of event temporal relations. Our framework should be more powerful to deal with domain-specific articles if utilizing such knowledge in the framework.
Ethical Considerations
We acknowledge that our work is aligned with the ACL Code of the Ethics (Gotterbarn et al., 2018) and will not raise ethical concerns. We do not use sensitive datasets/models that may cause any potential issues. The design, implementation, and evaluation of our proposed method are robust and secure.

heading: Future Work
text: In the future, we aim to incorporate external background knowledge and commonsense knowledge into our framework. We also plan to make use of the generated temporal event graphs in downstream tasks, such as future event prediction and question answering. Limitations
In the current design setting, our proposed model is only able to classify temporal relations between event pairs into one of three classes: BEFORE, AFTER, and NO RELATION. Our model should be more practically useful if we can extend it to predict more relation types in addition to temporal relations, such as PARENT-CHILD and CAUSECAUSED_BY relations. We believe that our model is able to make such extension without too much modification. In addition, as mentioned in the previous section, our model does not make use of any external knowledge, e. g. , commonsense knowledge of event temporal relations. Our framework should be more powerful to deal with domain-specific articles if utilizing such knowledge in the framework