SlowGuess/NAACL_2022_OCR huggingface dataset:
Shuaicheng Zhang*, Qiang Ning*, Lifu Huang*
\*Virginia Tech, \*Amazon
$\clubsuit$  {zshuai8,lfuh}@vt.edu,  $\clubsuit$  qning@amazon.com
Extracting temporal relations (e.g., before, after, and simultaneous) among events is crucial to natural language understanding. One of the key challenges of this problem is that when the events of interest are far away in text, the context in-between often becomes complicated, making it challenging to resolve the temporal relationship between them. This paper thus proposes a new Syntax-guided Graph Transformer network (SGT) to mitigate this issue, by (1) explicitly exploiting the connection between two events based on their dependency parsing trees, and (2) automatically locating temporal cues between two events via a novel syntax-guided attention mechanism. Experiments on two benchmark datasets, MATRES and TB-DENSE, show that our approach significantly outperforms previous state-of-the-art methods on both end-to-end temporal relation extraction and temporal relation classification; This improvement also proves to be robust on the contrast set of MATRES. The code is publicly available at https://github.com/VT-NLP/Syntax-Guided-Graph-Transformer.
Temporal relationship, e.g., Before, After, and Simultaneous, is important for understanding the process of complex events and reasoning over them. Extracting temporal relationship automatically from text is thus an important component in many downstream applications, such as summarization (Jiang et al., 2011; Ng et al., 2014), dialog understanding and generation (Ritter et al., 2010; Sun et al., 2021), reading comprehension (Harabagiu and Bejan, 2005; Sun et al., 2018; Ning et al., 2020; Huang et al., 2019) and future event prediction (Li et al., 2021; Lin et al., 2022). While event mentions can often be detected reasonably well (Lin et al., 2020; Huang and Ji, 2020; Wang et al., 2021, 2022), extracting event-event relationships, especially temporal relationship, still remains challenging (Chen et al., 2021).
S1: Now, Lockheed Martin which  $(e_1$ : bought) an early version of such a computer from the Canadian company D-Wave systems two years ago is confident enough in the technology to upgrade it to commercial scale, becoming the first company to  $(e_2$ : use) quantum computing as part of its business.
S2: Mr. Erdogan's office  $(e_{1}$  said) he had  $(e_2$  accepted) the apology, " In the name of the Turkish people".
Temporal Relation  $(\mathbf{e}_1\rightarrow \mathbf{e}_2)$  : AFTER
S3: "The desk thing really ( $e_I$ : stuck) with me", Ms. Ayotte ( $e_2$ : said).
Temporal Relation  $(\mathbf{e}_1\rightarrow \mathbf{e}_2)$  : Before
Figure 1: Examples of temporal relation annotations. Event mentions are boldfaced, the temporal relations between these events are listed below each sentence, and the temporal cues deciding those temporal relations are highlighted in red.
Recent studies (Han et al., 2019b; Ning et al., 2017; Vashishtha et al., 2019; Wang et al., 2020a) have shown improved performance in temporal relation extraction by leveraging the contextual representations learned from pre-trained language models (Devlin et al., 2018; Liu et al., 2019). However, one remaining challenge of this task is that it requires accurate characterization of the connection between two event mentions and the cues indicating their temporal relationship, especially when the context is wide and complicated. For instance, by manually examining 200 examples of human annotated temporal relations from the MATRES (Ning et al., 2018) dataset, we find that about  $52\%$  of the temporal cues<sup>1</sup> come from the connection between two event mentions (e.g., S1 in Fig. 1),  $39\%$  from their surrounding contexts (S2 in Fig. 1) and the remaining  $9\%$  from others, e.g., event co-reference or subordinate clause structures (S3 in Fig. 1).
<sup>1</sup>Temporal cues refer to the words of which the semantic meaning or related syntactic relations can determine the temporal relation of two event mentions.
Findings of the Association for Computational Linguistics: NAACL 2022, pages 379 - 390 July 10-15, 2022 ©2022 Association for Computational Linguistics
Figure 2: Architecture overview. The tokens highlighted with red and blue colors in the Input Sentence show the source and target events to be detected. The bold edges in the Input Graph Structure indicate the triples from the dependency path between the source and target event mentions as well as their surrounding context, and are considered by the syntax-guided attention.
Syntactic features, such as dependency parsing trees, have proved to be effective in characterizing the connection of two event mentions in pre-neural methods (Chambers, 2013; Chambers et al., 2014; Mirza and Tonelli, 2016). However, how to make use of these features has been under-explored since the adoption of neural methods in this field. This paper closes this gap with a novel Syntax-guided Graph Transformer (SGT) network – in addition to the attention heads in a typical Graph Transformer, we bring in a new attention mechanism that specifically looks at the path from a source node to a target node over dependency parsing trees. SGT thus not only learns event representations as in a typical Graph Transformer, but also provides a way to represent syntactic dependency information between a pair of events (for temporal relation extraction, this means attending to the aforementioned temporal cues). We conduct experiments on two benchmark datasets, MATRES (Ning et al., 2018) and TB-DENSE (Cassidy et al., 2014) on both end-to-end temporal relation extraction and classification, which demonstrate the effectiveness of SGT over previous state-of-the-art methods. Experiments on the contrast set (Gardner et al., 2020) of MATRES further proves the robustness of our approach.
Figure 2 shows the overview of our approach. Given an input sentence  $\tilde{s} = [w_1, w_2, \dots, w_n]$  with  $n$  tokens, we aim to detect a set of event mentions  $\{e_1, e_2, \dots\}$  where each event mention  $e_i$  may contain one or multiple tokens by leveraging the contextual representations learned from a pre-trained BERT (Devlin et al., 2018) encoder. Then, following previous studies (Ning et al., 2019, 2017; Han et al., 2019b; Wang et al., 2020a), we consider each pair of event mentions that are detected from one or two continuous sentences, and predict their temporal relationship.
To effectively capture the temporal cues between two event mentions, we build a dependency graph from one or two input sentences and design a new Syntax-guided Graph Transformer network to automatically learn a new contextual representation for each event mention by considering the triples that they are locally involved as well as the triples along the dependency path of the two event mentions within the dependency graph. Finally, the two event mention representations are concatenated to predict their temporal relationship.
2.1 Sequence Encoder
Given an input sentence  $\tilde{s} = [w_1, w_2, \dots, w_n]$ , we apply the same tokenizer as BERT (Devlin et al., 2018) to get all the subtokens. Then, we feed the sequence of subtokens as input to a pre-trained BERT model to get a contextual representation for each token  $w_i$ . If a token  $w_i$  is split into multiple subtokens, we use the contextual representation of the first token to represent  $w_i$ . To enrich the contextualized representations, for each token, we create a one-hot Part-of-Speech (POS) tag vector and concatenate it with BERT contextual embeddings. In this way, we obtain a final representation  $c_i^2$  for each  $w_i$ . These representations will be later used for event mention detection and also as the initial representations to our syntax-guided graph transformer network.
2.2 Event Detection
To detect event mentions from the sentence, we take the contextual representation of each word as input to a binary linear classifier to determine whether it is an event mention or not, which is optimized by minimizing the following binary cross-entropy loss:
$$
\begin{array}{l} \tilde {\boldsymbol {y}} _ {i} = \operatorname {s o f t m a x} \left(\boldsymbol {W} _ {e v e} \boldsymbol {c} _ {i} + \boldsymbol {b} _ {e v e}\right) \\ \mathcal {L} _ {e v e} = - \sum_ {\tilde {s} \in \mathcal {S}} \sum_ {i = 1} ^ {| \tilde {s} |} \sum_ {\pi \in \{0, 1 \}} \alpha_ {\pi} y _ {i, \pi} \log (\tilde {y} _ {i, \pi}) \\ \end{array}
$$
where  $\mathcal{L}_{\text {eve }}$  denotes the cross-entropy loss for event detection.  $S$  is the set of sentences in the training dataset.  $\alpha_{\pi}$  is a weight coefficient for each class (0 or 1) to mitigate the data imbalance problem and  $\alpha_0 + \alpha_1 = 1$ .  $y_{i,\pi}$  is a binary indicator to show whether  $\pi$  is the same as the groundtruth binary label  $(y_{i,\pi} = 1)$  or not  $(y_{i,\pi} = 0)$ .  $\tilde{y}_{i,\pi}$  denotes the probability of the  $i$ -th token in  $s$  being predicted with a binary class label  $\pi$ .  $\boldsymbol{W}_{\text {eve }}$  and  $\boldsymbol{b}_{\text {eve }}$  are learnable parameters.
2.3 Syntax-guided Graph Transformer
From the example sentences in Fig. 1, the temporal cues for characterizing the temporal relationship between two event mentions mainly come from their surrounding contexts as well as their connections from their syntactic dependency path. However, a sequence encoder usually fails to capture such information, especially when the context between
two event mentions is complicated, thus we further design a new Syntax-guided Graph Transformer (SGT) network.
Given a source event  $e_{s}$  and a target event  $e_{t}$  detected from one or two continuous sentences, we apply a public dependency parser to parse each sentence into a tree-graph and connect the graphs of two continuous sentences with an arbitrary cross-sentence edge (Peng et al., 2017; Cheng and Miyao, 2017) pointing from the root node of the preceding sentence to the root node of the following one, and obtain a graph  $G = (V,E)$ . For each node  $v_{i}$ , we use  $\mathcal{N}_{i}^{in} = \{(v_{k},r_{ki},v_{i})\in E|v_{k},v_{i}\in V\}$  and  $\mathcal{N}_i^{out} = \{(v_i,r_{ij},v_j)\in E|v_i,v_j\in V\}$  to denote all the neighbor triples of  $v_{i}$  with in-going and outgoing edges respectively,  $r\in \Upsilon$  where  $\Upsilon$  is the label set for syntactic dependency relation, and use  $\mathcal{P}_{ij} = \{(v_i,r_{ig},v_g),\dots,(v_h,r_{hj},v_j)\}$  as the triple set along the path from  $v_{i}$  to  $v_{j}$ .
Node Representation Initialization For each node  $v_{i}$  in graph  $G$ , we map it to a particular token  $w_{i'}$  from the original sentence and obtain a contextual representation  $c_{i'}$  from the BERT encoder. Then, we learn an initial node representation for each node  $v_{i}$  as:
$$
\pmb {h} _ {i} ^ {0} = \pmb {W} _ {e} \pmb {c} _ {i ^ {\prime}} + \pmb {b} _ {e}
$$
where  $W_{e}$  and  $\pmb{b}_{e}$  are learnable parameters.
Graph Multi-head Self-attention Following transformer model (Vaswani et al., 2017; Wang et al., 2020b), we adapt the multi-head self-attention to learn a contextual representation for each node in the graph  $G$ . Each node  $v_{i}$  in graph  $G$  is associated with a set of neighbor triples  $\mathcal{N}_i^{in} \cup \mathcal{N}_i^{out}$  and a node representation  $\pmb{h}_i^{l-1}$  where  $l$  is the index of a layer in our transformer architecture. To perform self-attention, we first apply a linear transformation to obtain a query vector based on each node  $v_{i}$ , and employ another two linear transformations to get the key and value vectors based on the node's neighbor triples:
$$
\begin{array}{l} \boldsymbol {Q} _ {i} ^ {l} = \boldsymbol {W} _ {q} ^ {m} \boldsymbol {h} _ {i} ^ {l - 1} \\ \pmb {K} _ {i j} ^ {l} = \pmb {W} _ {k} ^ {m} \pmb {R} _ {i j} ^ {l - 1} \\ \boldsymbol {U} _ {i j} ^ {l} = \boldsymbol {W} _ {u} ^ {m} \boldsymbol {R} _ {i j} ^ {l - 1} \\ \boldsymbol {R} _ {i j} ^ {l - 1} = \boldsymbol {W} _ {r} ^ {m} (\boldsymbol {h} _ {i} ^ {l - 1} \left| \left| \boldsymbol {r} _ {i j} \right| \right| \left. \boldsymbol {h} _ {j} ^ {l - 1}\right) + \boldsymbol {b} _ {r} ^ {m} \\ \end{array}
$$
where  $m$  is the index of a particular head.  $Q_{i}^{l}$  denotes a query vector corresponding to node  $v_{i}$
2We use bold lower case symbols to denote vectors.
$^{3}$ https://spacy.io/api/dependencyparser
$K_{ij}^{l}$  and  $U_{ij}^{l}$  is a key and value vector respectively, and both of them are learned from a triple  $(v_{i},r_{ij},v_{j})\in \mathcal{N}_{i}^{in}\cup \mathcal{N}_{i}^{out}$ , which is represented as  $R_{ij}$ .  $m$  is the index of a particular head. || denotes the concatenation operation.  $r_{ij}$  denotes the representation of a particular relation  $r_{ij}$  between  $v_{i}$  and  $v_{j}$ , which is randomly initialized and optimized by the model.  $W_{q}^{m}$ ,  $W_{k}^{m}$ ,  $W_{u}^{m}$ ,  $W_{r}^{m}$  and  $b_{r}^{m}$  are learnable parameters.
For each node  $v_{i}$ , we then perform self-attention over all the neighbor triples that it is involved, and compute a new context representation with multiple attention heads:
$$
\boldsymbol {g} _ {i} ^ {l} = \left(\underset {m} {\overset {M} {\big |}} \operatorname {H e a d} _ {i} ^ {m}\right) \boldsymbol {W} _ {o}
$$
$$
\operatorname {H e a d} _ {i} ^ {m} = \operatorname {s o f t m a x} \left(\frac {\boldsymbol {Q} _ {i} ^ {l} (\boldsymbol {K} ^ {l}) ^ {\top}}{\sqrt {d _ {k}}}\right) \boldsymbol {U} ^ {l}
$$
where  $\pmb{g}_i^l$  is the aggregated representation computed over all neighbor triples of node  $v_i$  with  $M$  attention heads at  $l$ -th layer.  $\pmb{g}_i^l$  will be later used to learn the updated representation of node  $v_i$ .  $\sqrt{d_k}$  is the scaling factor denoting the dimension size of each key vector.  $W_o$  is a learnable parameter.
Syntax-guided Attention To automatically find the indicative temporal cues for two event mentions from their connection as well as surrounding contexts, we design a new syntax-guided attention mechanism. For two event nodes  $v_{s}$  and  $v_{t}$ , we first extract the set of nodes from the dependency path between  $v_{s}$  and  $v_{t}$  (including  $v_{s}$  and  $v_{t}$ ), which is denoted as  $\Theta_{st}$ . We then get all the triples from the dependency path between  $v_{s}$  and  $v_{t}$  as well as the triples that any node from  $\Theta_{st}$  is involved, which are denoted as  $\Phi_{st} = \cup_{v_{i} \in \Theta_{st}} \{\mathcal{N}_{i}^{in} \cup \mathcal{N}_{i}^{out}\} \cup \mathcal{P}_{st}$ . To compute the syntax-guided attention over all the triples from  $\Phi_{st}$ , we apply three linear transformations to get the query, key and value vectors where the query vector is obtained from the representation of two event mentions, and key and value vectors are computed from the triples in  $\Phi_{st}$ :
$$
\tilde {\boldsymbol {Q}} _ {s t} ^ {l} = \tilde {\boldsymbol {W}} _ {q} ^ {m} \cdot \left(\boldsymbol {h} _ {s} ^ {l - 1} \mid \mid \boldsymbol {h} _ {t} ^ {l - 1}\right) ^ {x}
$$
$$
\tilde {\boldsymbol {K}} _ {i j} ^ {l} = \tilde {\boldsymbol {W}} _ {k} ^ {m} \tilde {\boldsymbol {R}} _ {i j} ^ {l - 1}
$$
$$
\tilde {\boldsymbol {U}} _ {i j} ^ {l} = \tilde {\boldsymbol {W}} _ {u} ^ {m} \tilde {\boldsymbol {R}} _ {i j} ^ {l - 1}
$$
$$
\tilde {\boldsymbol {R}} _ {i j} ^ {l - 1} = \tilde {\boldsymbol {W}} _ {r} ^ {m} (\boldsymbol {h} _ {i} ^ {l - 1} \left| \left| \boldsymbol {r} _ {i j} \right| \right| \left| \boldsymbol {h} _ {j} ^ {l - 1}\right) + \tilde {\boldsymbol {b}} _ {r}
$$
where  $m$  is the index of a particular head,  $\tilde{\pmb{Q}}_{st}^{l},\tilde{\pmb{K}}_{ij}^{l},\tilde{\pmb{U}}_{ij}^{l}$  denote the query, key and value vec
tors respectively.  $\tilde{R}_{ij}^{l-1}$  is the representation of a triple  $(v_i, r_{ij}, v_j) \in \Phi_{st}$ .  $\tilde{W}_q^m$ ,  $\tilde{W}_k^m$ ,  $\tilde{W}_v^m$  and  $\tilde{W}_r^m$  are learnable parameters.
Given the query vector, we then compute the attention distribution over all triples from  $\Phi_{st}$  and get an aggregated representation to denote the meaningful temporal features captured from the connection between two event mentions and their surrounding contexts.
$$
\tilde {\boldsymbol {g}} _ {s t} ^ {l} = \left(\left| \begin{array}{c} M \\ \hline m \end{array} \right. \mathrm {H e a d} _ {s t} ^ {m}\right) \cdot \tilde {\boldsymbol {W}} _ {p}
$$
$$
\tilde {\mathrm {H e a d}} _ {s t} ^ {m} = \operatorname {s o f t m a x} \left(\frac {\tilde {\boldsymbol {Q}} _ {s t} ^ {l} \left(\tilde {\boldsymbol {K}} ^ {l}\right) ^ {\top}}{\sqrt {d _ {k}}}\right) \cdot \tilde {\boldsymbol {U}} ^ {l}
$$
where  $\tilde{\pmb{g}}_{st}^{l}$  is the aggregated temporal related information from all the triples in  $\Phi_{st}$  based on the syntax-guided attention at  $l$ -th layer.  $\pmb{W}_{p}$  is a learnable parameter.
Node Representation Fusion Each event node in graph  $G$  will receive two representations learned from the multi-head self-attention and syntax-guided attention, thus we further fuse the two representations for both the source node  $v_{s}$  and the target node  $v_{t}$ :
$$
\hat {\pmb {h}} _ {s} ^ {l} = \tilde {\pmb {W}} _ {f} (\pmb {g} _ {s} ^ {l} \big | \big | \tilde {\pmb {g}} _ {s t} ^ {l}), \hat {\pmb {h}} _ {t} ^ {l} = \tilde {\pmb {W}} _ {f} (\tilde {\pmb {g}} _ {s t} ^ {l} \big | \big | \pmb {g} _ {t} ^ {l})
$$
where  $\pmb{g}_s^l$  and  $\pmb{g}_t^l$  denote the context representations learned from the multi-head self-attention for  $v_{s}$  and  $v_{t}$ .  $\tilde{\pmb{g}}_{st}^{l}$  denotes the representation learned from the triples from  $\Phi_{st}$  using syntax-guided attention.  $\hat{\pmb{h}}_s^l$  and  $\hat{\pmb{h}}_t^l$  are the fused representations of  $v_{s}$  and  $v_{t}$ , respectively.  $\tilde{\pmb{W}}_f$  is a learnable parameter.
For each non-event node  $v_{i}$ , which only receives a context representation  $\pmb{g}_{i}^{l}$  learned from the multi-head self-attention, we apply a linear projection and get a new node representation:
$$
\hat {\boldsymbol {h}} _ {i} ^ {l} = \boldsymbol {W} _ {t} \boldsymbol {g} _ {i} ^ {l}
$$
Our Syntax-guided Graph Transformer encoder is composed of a stack of multiple layers, while each layer consists of the two attention mechanisms and the fusion sub-layer. We use residual connection followed by LayerNorm for each layer to get the final representations of all the nodes:
$$
\boldsymbol {H} ^ {l} = \operatorname {L a y e r N o r m} \left(\hat {\boldsymbol {H}} ^ {l} + \boldsymbol {H} ^ {l - 1}\right)
$$
2.4 Temporal Relation Prediction
To predict the temporal relation between two event mentions  $e_{s}$  and  $e_{t}$ , we concatenate the final hidden states of  $v_{s}$  and  $v_{t}$  obtained from the Syntax-guided Graph Transformer network, and apply a Feedforward Neural Network (FNN) to predict their relationship
$$
\tilde {\boldsymbol {y}} _ {s t} = \operatorname {s o f t m a x} (\boldsymbol {W} _ {z} (\boldsymbol {h} _ {s} ^ {L} | | \boldsymbol {h} _ {t} ^ {L}) + \boldsymbol {b} _ {t})
$$
where  $\tilde{\pmb{y}}_{st}$  denotes the probabilities over all possible temporal relations between event mentions  $e_s$  and  $e_t$ .
The training objective is to minimize the following cross-entropy loss function:
$$
\mathcal {L} _ {r e l} = - \sum_ {s t \in \Delta} \sum_ {x \in X} \beta_ {x} y _ {s t, x} \log (\tilde {y} _ {s t, x}))
$$
where  $\Delta$  denotes the total set of event pairs for temporal relation prediction and  $X$  denotes the whole set of relation labels.  $y_{st,x}$  is a binary indicator (0 or 1) to show whether  $x$  is the same as the groundtruth label  $(y_{st,x} = 1)$  or not  $(y_{st,x} = 0)$ . We also assign a weight  $\beta_{x}$  to each class to mitigate the label imbalance issue.
We perform experiments on two public benchmark datasets for temporal relation extraction: (1) TB-DENSE (Cassidy et al., 2014), which is a densely annotated dataset with 6 types of relations: Before, After, Simultaneous, Includes, Is Included and Vague. (2) MATRES (Ning et al., 2018), which annotates verb event mentions along with 4 types of temporal relations: Before, After, Simultaneous and Vague. Additionally, we use POS tag information from MATRES provided by (Ning et al., 2019). For TB-DENSE, we use spacy annotation for predicting POS tag information which is based on Universal POS tag set<sup>4</sup>. For both benchmark datasets, we use the same train/dev/test splits as previous studies (Ning et al., 2019, 2017; Han et al., 2019a,b). Note that, for evaluation, similar as previous work, we disregard the Vague relation from both datasets (in the evaluation phase, we simply remove all ground truth Vague relation pairs). In addition, we will only consider event pairs from adjacent sentences due to the fact that it will require
an exponential number of annotations if we also consider event pairs from non-adjacent sentences, which is beyond the scope of this study. Table 1 shows statistics of the two datasets and Table 2 shows the label distribution.
Corpora	Train	Dev	Test
TB-DENSE	# Documents	22	5	9
# Relation Pairs	4,032	629	1,427
MATRES	# Documents	255	20	25
# Relation Pairs	13K	2.6K	837
Table 1: Data statistics for TB-DENSE and MATRES
Labels	TB-DENSE	MATRES
Before	384	26.9%	417	49.8%
After	274	19.2%	266	31.8%
Includes	56	3.9%	-	-
Is_Included	53	3.7%	-	-
Simultaneous	22	1.5%	31	3.7%
Vague	638	44.7%	133	15.9%
Table 2: Label distribution for TB-DENSE and MATRES. For each dataset, the first column shows the number of instances of each relation type while the second column shows the percentage.
Implementation Details For fair comparisons with previous baseline approaches, we use the pretrained bert-large-cased model for fine-tuning and optimize our model with BertAdam. We optimize the parameters with grid search: training epoch 10, learning rate  $\in \{3e - 6,1e - 5\}$ , training batch size  $\in \{16,32\}$ , encoder layer size  $\in \{4,12\}$ , number of heads  $\in \{1,8\}$ . During training, we first optimize the event extraction module for 5 epochs to warm up, and then jointly optimize both event extraction and temporal relation extraction modules using gold event pairs for another 5 epochs.
We evaluate SGT against two public benchmark datasets under two settings: (1) joint event and temporal relation extraction (Table 3); (2) temporal relation classification, where the gold event mentions are known beforehand (Table 4). Note in the "joint" setting, we adopt the same strategy proposed in (Han et al., 2019b): we first train the event extraction module, and then jointly optimize both event extraction and temporal relation extraction
$^{5}$ https://huggingface.co/transformers/pretrained_models. html
4https://spacy.io/api/data-formats
Dataset	Model	Pre-trained Model	Event Detection	Relation Extraction
TB-DENSE	HNP19 (Han et al., 2019b)	BERT Base	90.9	49.4
Our Approach	BERT Base	91.0	51.8
MATRES	CogCompTime2.0 (Ning et al., 2019)	BERT Base	85.2	52.8
HNP19 (Han et al., 2019b)	BERT Base	87.8	59.6
Our Approach	BERT Base	90.5	62.3
Table 3: Comparison of various approaches on joint event and relation extraction with F-score (\%). Note that HPN19 fixes BERT embeddings but relies on BiLSTM to capture the contextual features.
Dataset	Model	Pre-trained Model	Relation Classification (F-score %)
TB-DENSE	LSTM (Cheng and Miyao, 2017)	BERT Base	62.2
HNP19 (Han et al., 2019b)	BERT Base	64.5
Our Approach	BERT Base	66.7
PSL (Zhou et al., 2020)	RoBERTa Large	65.2
DEER (Han et al., 2021)	RoBERTa Large	66.8
Our Approach	BERT Large	67.1
MATRES	CogCompTime2.0 (Ning et al., 2019)	BERT Base	71.4
LSTM (Cheng and Miyao, 2017)	BERT Base	73.4
HNP19 (Han et al., 2019b)	BERT Base	75.5
Our Approach	BERT Base	79.3
HMHD20 (Wang et al., 2020a)	RoBERTa Large	78.8
DEER (Han et al., 2021)	RoBERTa Large	79.3
Our Approach	BERT Large	80.3
(using gold event pairs as input to ensure training quality) modules. Overall, we observe that our approach significantly outperforms baseline systems in both settings, with up to  $7.9\%$  absolute F-score gain on MATRES and  $2.4\%$  on TB-DENSE.