ConTempo: A Unified Temporally Contrastive Framework for Temporal Relation Extraction (2024)

Abstract: The task of temporal relation extraction (TRE) involves identifying and extracting temporal relations between events from narratives. We identify two primary issues with TRE systems. First, by formulating TRE as a simple text classification task where every temporal relation is independent, it is hard to enhance the TRE model’s representation of meaning of temporal relations, and its facility with the underlying temporal calculus. We solve the issue by proposing a novel Temporally Contrastive learning model (ConTempo) that increase the model’s awareness of the meaning of temporal relations by leveraging their symmetric or antisymmetric properties. Second, the reusability of innovations has been limited due to incompatibilities in model architectures. Therefore, we propose a unified framework and show that ConTempo is compatible with all three main branches of TRE research. Our results demonstrate that the performance gains of ConTempo are more pronounced, with the total combination achieving state-of-the-art performance on the widely used MATRES and TBD corpora. We furthermore identified and corrected a large number of annotation errors present in the test set of MATRES, after which the performance increase brought by ConTempo becomes more apparent.

heading: 1 Introduction
text: Temporal relation extraction (TRE) is the task of classifying the temporal relations between pairs of events conveyed in narratives (Pustejovsky et al., 2006, 2010). This task is important to the natural language processing (NLP) community because the conception of time is a key component of text comprehension and reasoning. Additionally, it carries practical importance, as TRE is a crucial component of various downstream applications in question answering, information retrieval, and information extraction.
TRE is difficult. Despite its importance and popularity, TRE systems still have relatively poor performance compared to other natural language understanding tasks. TRE is also salient because recent large language models (LLMs) and chat systems have substantially worse performance than smaller, fine-tuned TRE models (Yuan et al., 2023; Huang et al., 2023).
In this paper, we enhance current TRE systems by identifying and addressing two major issue that hinder their performance:

heading: 1. It is difficult to understand relation labels.
text: TRE is typically formulated as a multi-label text classification task. The representation of the relation is determined by a neural classifier, which outputs logits for different label classes. These logits are then converted into label probabilities using a softmax layer. As a result, in a TRE model, all relation types are considered independent. However, temporal relations are inherently structured, governed by an underlying temporal calculus (Allen, 1981, 1983). For instance, if event A occurred before event B, it can be inferred that B happened after A. Integrating the meanings of each temporal relation and the principles of the underlying temporal calculus into the neural classifier presents a complex challenge, as there is no straightforward method to do so.
2. The reusability of innovations has been limited. Many of the recent innovations in TRE have not built upon one another, primarily due to the lack of a unified framework (beyond annotation standards) to accommodate them. For example, despite being widely cited and used as a baseline by subsequent systems, the relative time prediction technique (Wen and Ji, 2021) has rarely been applied on top of newer models, due to incompatibilities in model architecture. The evaluation methods employed by earlier systems vary significantly, moreover, which complicates direct comparisons
1521
between models and further hinders the development of future models.
We propose a novel Temporally Contrastive learning method (ConTempo)1 to enhance the model’s representation of the meaning of temporal relations, and its facility with the underlying temporal calculus. The intuition behind ConTempo is very simple. Contrastive learning aims to pull similar (positive) samples closer and push dissimilar (negative) samples apart. We leverage the symmetric and antisymmetric properties of temporal relations to augment a base graph of positive samples for contrastive learning. For antisymmetric relations (A before B), we want to push the relation’s representation away from its inverse (B after A). For a symmetric relation (A equals B), we likewise want to pull its representation closer to its inverse (B equals A). Our experimental results indicate that ConTempo is effective at creating distinctive representations for different types of temporal relation pairs and yields a substantial improvement in performance.
Then, building upon ConTempo, we create a unified framework that is compatible with various previous modes of improvement in TRE systems. In particular, we have identified two major threads of innovation in the previous literature on this topic: (1) GNN encoding of event and context information, (2) incorporation of common-sense information, and (3) the soft enforcement of logical constraints. Our experiment shows that ConTempo is compatible with techniques from all three major threads of research and can achieve state-of-the-art performance on major evaluation corpora, including MATRES (Ning et al., 2018b) and TimeBankDense (TBD; Cassidy et al., 2014). We nevertheless perceive the need to introduce CleanMATRES, a refined version of MATRES that we created by fixing certain annotation errors.
Among all of these innovations, an ablation study nevertheless shows that ConTempo brings the most significant performance increase. This finding underscores our initial hypothesis: the primary challenge for these models has been their difficulty with making sense of relation labels.
During our development of the unified ComTempo framework, we discovered a large number of annotation errors within MATRES and TBD, two commonly utilized corpora. Because of
1ConTempo and CleanMATRES are publicly available online: https://github.com/frankniujc/contempo.
TBD’s dense annotation scheme, the appropriate level of underspecificity cannot be resolved without further investigation. But we are able to provide a thorough re-examination of the MATRES corpus. After correcting these errors, performance improves yet again, and the performance increase brought by ConTempo becomes more apparent.

heading: 2 ConTempo
text: We begin our discussion by first formulating the TRE task. Then, using a comparison of relational representations, we will demonstrate the difficulty that a neural model faces in temporal reasoning and understanding the meaning of temporal labels. In particular, there is a significant correlation between the mistakes a TRE model makes and its degree of inference over the available symmetric and antisymmetric properties. Next, we will present our ConTempo method in detail, which enhances a model’s awareness of the meaning of each temporal relation and each principles of temporal reasoning. Finally, we will empirically demonstrate the effectiveness of ConTempo and compare it with previous methods in increasing temporal relation awareness.

heading: 2.1 TRE: Task Formulation
text: TRE can be formulated as a classification task. Given an input sequence S = [t1, . . . , tn] and two events represented by two distinct tokens from the sequence, e1, e2 ∈ S, where e1 ̸= e2, the TRE model should classify the relation between the event pair (e1, e2) into one of several temporal relations, R. Different corpora have different sets of temporal relations. Most TRE benchmark datasets aspire to embody Allen’s (1981) 13 original temporal relations, but creating datasets with such fine-grained annotations is challenging. Attempts to utilize all 13 temporal relation types have failed due to low inter-annotator agreement. Therefore, recent TRE datasets typically simplify the relation set to a more coarse subset of the original set. As listed in Table 1, MATRES has 4 relation types, and TBD uses 6 relation types.

heading: 2.2 Temporal Relation Labels Awareness
text: After simplification, Allen’s (1983) full temporal interval calculus can no longer be maintained. But interval symmetry and antisymmetry (as shown in Equation 1) still apply within both MATRES and TBD.2 In our opinion, it is crucial for TRE models to reason with intervals in order to yield good evaluation results on benchmarks—if the models know A happened before B but do not know that B happened after A, can we really say that the model understands time in narratives?
A INCLUDES B ⇐⇒ B IS_INCLUDED A A BEFORE B ⇐⇒ B AFTER A A EQUAL B ⇐⇒ B EQUAL A (1)
Symmetry and antisymmetry also provide us with an opportunity to test a model’s understanding of relation labels. In particular, we can measure the similarity of the model’s representation of a relation h→r in comparison to its dual h ← r :
sim(h→r ,h ← r ) = sim(h(e1,e2),h(e2,e1)), (2)
It is relatively commonplace in the context of TRE to speak of a representation of a temporal relation, although this in fact refers to the representation of one of its instances over some event pair h(e1,e2). Therefore, we will speak of the two interchangeably. If the relation between e1 and e2 is symmetric, then we can interpret high similarity as a sign of awareness concerning the relation’s meaning because the relation and its inverse are nearly the same. On the other hand, if the relation is antisymmetric, we should expect the relations to be distinct and, therefore, we should interpret low similarity as a good sign. Here, we measure similarity with cosine similarity (sim(a,b) = a·b∥a∥∥b∥ ).

heading: 2.3 Temporal Relation Awareness of Fine-tuned BERT-based Models
text: In this section, we examine the extent to which typical, fine-tuned BERT-based models can be said to understand temporal relations and the temporal calculus. All contemporary TRE systems use this paradigm. Therefore, we experimented with a baseline model (described in Equation 3) to establish a foundation for the capacity of TRE models to summarize the temporal interval calculus.
2We ignore the VAGUE relation in this part of the analysis because whether symmetry applies to the VAGUE relation requires more careful consideration (Appendix D.3).
The baseline model has three components: a pretrained encoder (we use a RoBERTa-large model (Liu et al., 2019)), a 2-layer MLP module (f1) that encodes the relation of the event pair under a unified representation scheme, and another single layer MLP (f2) that decodes the relational representation into logits for each, different relational type:
E = encoder(S) = [h1, . . . ,hn], h(e1,e2) = f1([he1 ∥ he2 ]), PR(e1, e2) = softmax(f2(h(e1,e2))).
(3)
The implementation details and hyperparameter settings are described in Appendix A.
Baseline Model Temporal Awareness Figure 1 shows the similarity between the relational representations h→r and their symmetric duals h ← r as generated by the baseline model. We can see an obvious negative correlation between the correctness of the prediction and the similarity measure. This is evident from observing that the incorrect predictions (red points) are located in the upper region of the figure for BEFORE and AFTER. Furthermoe, the baseline model often assigns similar representations to relations and their symmetric duals, even when the two are diametrically opposite in meaning. As shown by the box plot, the first quantile of both the BEFORE and AFTER relations occupy
the upper part of the figure, which corresponds to high similarity. This shows that while fine-tuned models might have some basic perception of the significance of these temporal labels, it is by no means efficient, as all temporal labels are presented to the model as independent.

heading: 2.4 ConTempo: The Method
text: Contrastive learning builds on the idea of pulling similar (positive) data points (x, x+) closer together and pushing dissimilar (negative) data points (x, x−) apart by adding a contrastive loss term to the training objective (Robinson et al., 2021). We thus need to define positive and negative samples for temporal relation instances, i.e., event pairs. In this section, we will explain how positive and negative samples are constructed based on whether the relation is symmetric (EQUAL) or antisymmetric (BEFORE, AFTER, INCLUDES, and IS_INCLUDED), and define how we compute the temporally contrastive loss term ℓtc.
Our implementation of ConTempo is inspired by SimCSE’s (Gao et al., 2021) use of contrastive learning on natural language inference (NLI) data. We likewise use a normalized temperature-scaled cross-entropy loss (NT-Xent; Chen et al., 2020).
Antisymmetric Relations For an antisymmetric relation h→r = h(e1,e2), we regard the dual of the relation h←r = h(e2,e1) as a hard negative sample. However, we cannot construct further positive samples using temporal reasoning. Instead, we follow SimCSE’s unsupervised approach to obtain positive pairs through the use of independently sampled dropout masks. Standard transformers will apply dropout (default p = 0.1) at different locations in the model. Therefore, we simply feed the same input to our model twice to obtain two relational representations h→r and h + r that are slightly different; different dropout masks having been applied during the two encoding passes. Therefore, we can compute the ConTempo loss by:
ℓtc = − log esim(h
→ r ,h + r )/τ
esim(h→r ,h + r )/τ + esim(h→r ,h←r )/τ
, (4)
where τ is a temperature hyperparameter. Unlike SimCSE, however, we did not treat other relational representations in the same mini-batch as negatives. While we are certain that the relation yields the exact dual when we reverse the order of the two events ((e1, e2) vs. (e2, e1)), the compari-
son is less clear when it is made between different event pairs. Consider the following examples:
(1) Alan Turing studiede1 at Cambridge for his undergraduate degree and attendede2 Princeton for his Ph.D.
(2) I failede3 my calculus course and then criede4 all night in a McDonald’s.
While both temporal relations r1 = (e1, e2) and r2 = (e3, e4) share the same BEFORE label, it is difficult to say whether they can be regarded as a positive pair, because they are different in many respects. First, we know that e1 and e2 are years apart but e3 and e4 happen on the same day. Second, we infer BEFORE for different reasons. We know that e1 happened before e2 through common-sense: a person needs to complete their undergraduate study before working towards a Ph.D. But we conclude that e3 happened before e4 because of and then and a likely causal relation between failing the course and crying. We want to preserve these differences and therefore avoid pushing the representations of r1 and r2 closer together.
We also do not need to consider relations with different labels, such as r→1 = (e1, e2) and r ← 2 = (e4, e3), as negative samples. Intuitively, contrastive learning is more effective when the negative samples occur nearby but should be far apart (Robinson et al., 2021). Since event pairs from different sentences contain drastically different syntactic and semantic information, it is not a serious problem for the classifier to distinguish them.
Symmetric Relations For a symmetric relation h→r = h(e1,e2), we use the dual of the relation h←r = h(e1,e2) as an additional, positive sample. Unlike with antisymmetric relations, we follow SimCSE and use other relations in the same minibatch (B) as negative samples:
ℓtc = − log esim(h
→ r ,h ← r )/τ
∑ i∈B(e sim(h→r ,hi)/τ + esim(h←r ,hi)/τ ) .
(5) We choose a different design here than for antisymmetric relations because it is impossible to construct true hard negative samples for symmetric relations. Also, we do not check whether other samples in the mini-batch have the same relation label as h→r because EQUAL is the least common relation type. With only 3.8% in MATRES and 1.7% of the relations in TBD, it is very rare that there are multiple EQUAL relations in the same
mini-batch, so the effect of a colliding relation type is negligible.
Final Training Objective The ConTempo loss is added to the final training objective with a scaling hyperparameter γ (Equation 6). The classification loss ℓclf is the standard cross-entropy loss computed using the model’s output logits:
ℓ = ℓclf + γ · ℓtc. (6)

heading: 2.5 Experiment: ConTempo at Increasing Temporal Relation Awareness
text: Here we present the effectiveness of ConTempo at increasing temporal relation awareness. In this section, we trained an enhanced ConTempo baseline that is identical to the fine-tuned baseline but with ConTempo loss added to the model. We trained the model using the same hyperparemeters for the same amount of time.
In Figure 2, we compare ConTempo with baselines using the same similarity measures. We can observe that the similarity of the model’s representation of a relation in comparison to its dual drops significantly after ConTempo is applied. Furthermore, now we can observe that the vast majority of similarity measures occupy the lower section of the figure, suggesting that ConTempo is effective at making antisymmetric relations generate representations that are distinct from their duals.
Unfortunately, we also see a similar drop in similarity for the symmetric relation (EQUAL). We believe this is caused by a data imbalance as there are too few EQUAL instances to balance the overall effect of the antisymmetric relations.

heading: 2.6 Previous Attempts at Increasing Temporal Calculus Awareness
text: Does similarity reduction really lead to better temporal label awareness? Also, are there any other ways to increase temporal label awareness? Unfortunately, it is impossible to measure temporal label awareness directly, but we can infer it indirectly by observing the model’s performance.
In this section we compared ConTempo with three previous methods that attempted to address similar problems. In the following paragraphs, we elucidate these methods, as they could also increase temporal relation awareness. In Table 2, we compare the effectiveness of all three with ConTempo. We use the same model setting for each method for a fair comparison. Overall, ConTempo is the most effective at reducing similarity. It also has the best performance on MATRES. This suggests ConTempo could help the model create more effective representations for TRE prediction.
Inverse Data Expansion Huang et al. (2023) simply expand the training set by adding all the inverse relations. This straightforward approach, however, does not address the issue that temporal relation labels are presented independently.
Relative Time Prediction Wen and Ji (2021) employed a relative timestamp prediction component to regulate the training process by predicting a real number (“relative event time,” RET) that indicates the relative position of events in a timeline. This component is trained to maximize the distance between events in antisymmetric relations and to minimize the distance between events that are equal. This approach shares broad similarities with contrastive learning but faces two major challenges. First, relative event-time prediction in RET is ultimately computed for each single event individually, which is impossible. By contrast, ConTempo considers events in pairs and accounts only
for relative information. Second, this model cannot specify the absolute distance between events, meaning it does not differentiate between events that occur seconds apart and those that are days apart. It also makes predicting chains of events (e.g., e1 < e2 < . . . < e5 where < indicates AFTER) difficult, as the model needs to balance various data points across the entire dataset and precisely position each event within the relative time range. ConTempo does not face this issue as the time distance between any two events (e1, e2) remains consistent regardless of the direction.
Logical Constraints Wang et al. (2020) proposed to impose soft logical constraints by adding a loss term that is calculated based on the model’s output logits. In particular:
ℓS = ∑ | log r(e1, e2)− log r̄(e2, e1)|, (7)
where r(e1, e2) denotes the logits the model outputs for relation type r given (e1, e2) as the input, and r̄ denotes the dual of the relation. Other logical constraints such as transitivity are also enforced by Wang et al. (2020); we temporarily skip those as they are not relevant to the topic of the current section. We will return to a discussion of other types of constraint in Section 3.3.
Logical constraint loss has some surface-level resemblance to ConTempo loss, but their method focuses on the output logits whereas ours focuses on regulating the quality of the models’ representations. The major motivation for Wang et al. (2020) to add logical constraints was to avoid label conflicts; enhancing the model’s understanding of temporal relation labels was secondary.

heading: 3 The Unified Framework
text: Our survey of previous systems suggests that the reusability of innovations of TRE has been limited. For example, papers working on GNN-encoding of document-level information (Zhang et al., 2022; Mathur et al., 2021; Yao et al., 2022) rarely incorporate other published innovations, such as additional common-sense information, in their models. We call for more collaboration between different directions of TRE research.
In this section, we argue that ConTempo is compatible with various recent innovations across different research directions. In particular, ConTempo is complementary to: (1) GNN encodings of event and context information, (2) incorporat-
ing common-sense information, and (3) the soft enforcement of logical constraints.

heading: 3.1 Event and Context Encoder
text: All natural language understanding models need methods to encode their input sequence into vectorized representations, and to extract the relevant event and context representations. Currently, researchers typically use pre-trained language models such as BERT (Devlin et al., 2019) or RoBERTa to encode the input sequence because of the rich information these models gather during pre-training.
In TRE, an event is usually described by a complete phrase or even an entire clause. But since it is hard to annotate a discontinuous sequence, TRE corpora typically label the semantic head as the sole event trigger to represent the complete event. Extracting representations from only the event triggers may not obtain all the relevant information to make a correct prediction, however.
Therefore, TRE systems now utilize Graph Neural Networks (GNNs) to encode additional information about events and context. Zhang et al. (2022) proposed using dependency trees to help the model access different event arguments. TIMERS (Mathur et al., 2021) proposed using a GNN to encode three graphs: a sentence structure graph, a temporal structure graph, and a discourse structure graph. More recently, MulCo (Yao et al., 2022) reported better results using only the sentence (GS) and temporal (GT ) structure graphs. They achieved better results with fewer graphs by using multiscale contrastive learning between the fine-tuning of BERT and the training of the GNN to decrease GPU memory usage. Unlike ConTempo, their contrastive learning is not used to enhance the understanding of temporal labels.
We add MulCo to the unified framework by concatenating the GNNs’ outputs to the event representations as h̃e = [hGSe ∥ hGTe ∥ he].
Please refer to Yao et al. (2022) for the detailed description of their model. We provide only a quick overview in Appendix B.

heading: 3.2 Common-sense Information
text: Similar to event and context encoding, we can also add additional features from external sources to enhance the representation. The feasibility of our unified framework is demonstrated by admitting temporal common-sense information.
Earlier TRE methods (Ning et al., 2018a, 2019; Wang et al., 2020) used temporal information de-
rived from simple heuristics, or from corpora such as TempProb, which contains typical temporalrelation information between event triggers. Unfortunately, TempProb’s coverage and accuracy is limited. Alternatively, the MC-TACO corpus (Zhou et al., 2019) was developed for temporal commonsense knowledge, presenting typical duration information through answers to multiple-choice questions. This approach, however, requires the model to select a typical duration from a restricted set of pre-defined options, and is incapable of addressing typical granularity and underspecificity problems. Our system is the first to apply temporal commonsense information from LLMs for TRE purposes.
We used GPT-3 (text-davinci-003), employing the template illustrated in Figure 3, to obtain descriptions of the typical durations of events in MATRES. We then use our system’s underlying RoBERTa encoder to encode a description of the typical duration, and use the [CLS]-pooler output as the representation, denoted as hdure . 3
ĥe = [h dur e ∥ h̃e]
h(e1,e2) = f1([ĥe1 ∥ ĥe2 ]) (8)

heading: 3.3 Training Objective
text: The third type of innovation is the modification of the training objective. ConTempo itself is an innovation to the training objective.
In this section, we show that ConTempo is compatible with other techniques that have been applied to training objectives. We have incorporated the aforementioned soft logical constraints (Wang et al., 2020) into the unified framework.
We add both the symmetric loss (ℓS) and the conjunction loss (ℓC) to the loss function. There is a bit of overlap between Wang et al.’s (2020)
3The prompts and the generated responses from GPT-3 will be made publicly accessible online.
symmetric loss and ConTempo loss, but as previously discussed, we focus on regulating the representation whereas Wang et al.’s (2020) focus on avoiding label conflicts. ConTempo and the logical constraints complement each other, however, as ConTempo does not attempt to treat transitivity within contrastive learning. Instead, it is covered by conjunction restrictions. The loss terms are:
ℓS = ∑ | log r(e1, e2)− log r̄(e2, e1)|, ℓC = ∑ |Lt1 |+ ∑ |Lt2 | (9)
where Lt1 and Lt2 are defined as:
Lt1 = log r1(e1, e2) + log r2(e2, e3)− log r3(e1, e3), Lt2 = log r1(e1, e2) + log r2(e2, e3)− log(1− r4(e1, e3)).
(10)
See Wang et al. (2020) for details, including the induction table for the conjunctive constraints. The final training objective is:
ℓ = ℓclf + γ · ℓtc + λS · ℓS + λC · ℓC . (11)

heading: 4 Experiments

heading: 4.1 Experimental Setup
text: The hyperparameter settings and training details are explained in Appendix A.
Data We experimented with both MATRES (Ning et al., 2018b) and TBD (Cassidy et al., 2014), the two popular corpora that are most often used to benchmark TRE systems. Both are derived from the TimeBank Corpus (Pustejovsky et al., 2003), which is annotated using the TimeML standard (Pustejovsky et al., 2005). The VAGUE labels are computed into the F1s for both MATRES and TBD but are regarded as negative labels.
TBD is a dense re-annotation of the TimeBank corpus where every possible pair of events/times in a given window is forced to be annotated regardless of whether there exists a specified temporal relation. As a result, TBD has ample data samples (12715 relations from 39 articles) but a relatively low inter-annotator agreement, with Cohen’s (1960) κ ranging between 0.56 and 0.64.
MATRES aimed at reducing noise in TBD’s annotation. It introduced the concept of orthogonal temporal axes for hypothetical, negated, opined, and static events. In practice, these events are ignored during temporal relation annotation. They further reduced the number of temporal relations
down to only four types. They also simplified the annotation scheme by asking the annotators to compare only the start times of events, as the end time of a durative event can often be ambiguous and cause confusion. During our survey of TRE baselines, we found that there are actually several versions of MATRES being used by different research labs (see Appendix C). We use the version that is available on the Ning et al.’s (2018b) GitHub page4 and reproduced several key baseline systems’ performances using our unified evaluation standards.
During the development of ConTempo, we discovered a large number of annotation errors in both MATRES and TBD. To further reduce the noise in the annotations, we re-examined and corrected the test set annotations of MATRES, creating the CleanMATRES test set. We will detail the process and describe the data in Section D.
Baseline Systems We compared ConTempo’s performance with seven prior systems: LSTM (Cheng and Miyao, 2017), SCS-EERE (Man et al., 2022), Joint Constrained Learning (Wang et al., 2020), Relative Time (Wen and Ji, 2021), MulCo (Yao et al., 2022), TIMERS (Mathur et al., 2021) and Endpoint Comparisons (Huang et al., 2023). We reproduced the results of SCS-EERE and Relative Time using their publicly available code. Additionally, we reimplemented MulCo according to the specifications provided in the paper.
LLM Baselines We incorporated the performance of LLMs as an additional baseline alongside the aforementioned baseline systems. Specifically,
4https://github.com/qiangning/MATRES
we reproduced the results from Yuan et al.’s (2023) using both ChatGPT-3.5 and ChatGPT-4. Chainof-thought (CoT) was deployed in the experiments. Additionally, we fine-tuned GPT-3.5 utilizing OpenAI’s API using the suggested settings to perform the task.

heading: 4.2 Experimental Results
text: Table 3 reports the performance of our ConTempo model compared to other baseline models on CleanMATRES, MATRES, and TBD. The results of our models and our reproductions are the average over three different runs with different random seeds.
Overall, ConTempo brings a substantial performance increase to the TRE model. Even the simple ConTempo baseline’s performance is in the same ballpark as previous state-of-the-art systems. When enhanced by other innovations under the unified framework, our model outperformed the previous state-of-the-art models by 1.32%, 0.57%, and 0.46% for the three datasets, respectively.

heading: 4.3 Ablation Study
text: In order to evaluate the efficacy of the proposed components in the unified framework, we conducted an ablation study, wherein we removed one
component at a time and reran the experiment using the aforementioned hyperparameters. The results of this study are presented in Table 4. Among the four components, contrastive learning demonstrated the most substantial impact on performance, with its removal leading to a 1.36% decrease in the model’s effectiveness. The GNN module also makes a substantial contribution to overall performance. Finally, both GPT-3 typical duration information and logical constraints exhibit moderate effects on the system’s performance.
The ablation study result confirms our initial hypothesis: the primary challenge for these models has been making sense of relational labels.

heading: 5 Conclusion
text: In this paper, we propose a novel ConTempo method that utilizes contrastive learning on the symmetric and antisymmetric properties to enhance the model’s awareness of temporal relations. Using ConTempo, we observe an apparent change in the representation of relations generated by the model. Particularly, we see a significant reduction in the similarity between relation representations h→r in comparison to their dual h←r .
ConTempo and GNN encoding, temporal common-sense information, and soft logical constraints can work in synergy to create a unified system that has achieved state-of-the-art performance across various benchmark corpora. Our success demonstrates the flexibility of ConTempo, and we call for more collaboration between different directions of TRE research.
TRE is a rare and curious case where fine-tuned small LMs perform considerably better LLMs. Even with the help of groundbreaking techniques such as chain-of-thought (Wei et al., 2022), the best performance ChatGPT can achieve is 52.4% on MATRES (Yuan et al., 2023) and 41.0% on TBD (Huang et al., 2023) — remarkably lower than the fine-tuned small language models’ performance. This anomaly presents tremendous opportunity for both TRE researchers and LLM researchers. TRE can be used as a hard task for the LLMs’ to analyse their capabilities at reasoning and comprehension. In conclusion, exploring the integration of existing TRE innovations with LLMs and incorporating LLMs as a key component within TRE frameworks could significantly enhance performance and drive forward the advancement of both fields.

heading: Acknowledgement
text: This study is funded by the Canadian Safety and Security Program (CSSP) from Defence Research and Development Canada (DRDC) awarded to the Public Health Agency of Canada (CSSP-2018CP2334: Incorporating Advanced Data Analytics into a Health Intelligence Surveillance System). We thank the Global Public Health Intelligence Network (GPHIN) and Epidemic Intelligence from Open Sources (EIOS) teams for their support.
Limitations
We only evaluate ConTempo’s performance using RoBERTa. Other language models, such as DeBERTa (He et al., 2021), T5 (Raffel et al., 2020) and DistilBERT (Sanh et al., 2020) may provide even better performance. We leave further exploration for future work.
Due to restricted dataset resources, our unified model’s evaluation was conducted only using TBDense and MATRES. There are other TimeMLbased and even non-TimeML-based corpora related to TRE. We leave more evaluation to future work and encourage the community to reproduce our results on more datasets.

heading: A Training Details
text: We use roberta-large as the pre-trained encoder. The best model is selected with the highest development set performance among 20 epochs of training,
optimized using AdamW (Loshchilov and Hutter, 2019) with an initial learning rate of 1e-5 and a 0.01 weight decay. We use a linear scheduler with a warm-up ratio of 0.1 and a batch size of 64. The dropout rate is 0.1 across the system. For ConTempo, we determined τ = 0.05, γ = 1.0 using a grid search over {0.1, 0.05, 0.01} × {1, 0.1, 0.01}.
For the components in the unified systems, we follow the hyperparameter suggested by the corresponding papers (Yao et al., 2022; Wang et al., 2020).

heading: B MulCo
text: MulCon utilized a sentence structure graph GS and a temporal structure graph GT to encode documentlevel information. The sentence structure graph has three types of nodes: the document, the sentences and the tokens. All sentences are connected to the document node and each token node is connected to the sentence it located in. For an event, GS’s embedding of the event trigger token is used.
The temporal structure graph, on the other hand, also has three types of nodes. The document creation time (DCT) node, the time expressions, and the events. Each time expression is connected to the DCT, and each event is connected to a time expression if there is an E2T link in the corpus. Finally, each event node has an edge point to itself.

heading: C MATRES Versions
text: We found two versions of MATRES that has been used by different works. Ning et al. (2018b) released a version on GitHub 5 that contains 837 relations in the Platinum test set. Wang et al. (2020) released another version6 with 818 relations in the Platinum test set. The difference (19 relations) could constitute as 2.32% of the test set data. We use Ning et al.’s (2018b).

heading: D CleanMATRES
text: As we previously mentioned, we identified a large quantity of annotation errors within both MATRES and TBD. This is not surprising, however, as both corpora reported adequate inter-annotator agreements. TBD reported κ ranged from 0.56 to 0.64 and MATRES reported a good — but not perfect — κ of 0.84 (Table 6). In this section, we will first provide a description of the annotation errors
5https://github.com/qiangning/MATRES 6https://github.com/CogComp/
JointConstrainedLearning/tree/main/MATRES
we identified in MATRES. Then, we will describe our re-annotation process. Finally, we will present an overview of the result of the re-annotation — CleanMATRES.
D.1 A Qualitative Analysis of the Annotation Errors
(3) He wone1 the Gusher Marathon, finishinge2 in 3:07:35.
Original Annotation: Correct Annotation: VAGUE EQUAL
(4) The last surviving member of the team which first conquerede1 Everest in 1953 has diede2 in a Derbyshire nursing home.
Original Annotation: Correct Annotation: VAGUE BEFORE
(5) More than 16,000 dead pigs have been founde1 floatinge2 in rivers that provide drinking water to Shanghai.
Original Annotation: Correct Annotation: EQUAL AFTER
Examples (3-5) show three examples of incorrect annotations in MATRES. In example 3, the relation between won and finishing is labelled as VAGUE,
but, nonetheless, the two mentions refer to two aspects of the same punctual event. For example 4, it is clear that the member died a long time after they conquered Everest. For example 5, since MATRES annotators were only comparing the start-time of events, the pigs floating in rivers must took place before the event that they were found.
The two examples presented in this study illustrate the main sources of annotation errors commonly found in data sets. As posited by Ning et al. (2018b), the first example’s error likely stems from the annotators’ lack of comprehension regarding time granularity and event coreference. MATRES annotations are created by asking annotators two heuristic question: (Q1) Is it possible that e1start is before e 2 start? (Q2) Is it possible that e2start is before e 1 start?
The heuristic questions, Q1 and Q2, do not adequately address these concepts, necessitating the need to develop improved annotation guidelines. By effectively conveying the intricacies of time granularity and event coreference, the frequency of such errors may decrease, although perhaps with a lower κ. The second example’s mistake should perhaps be attributed to a misunderstanding of the text.
D.2 CleanMATRES Annotation
We undertook a re-annotation of the MATRES test set. The test set were labelled independently by two authors of this paper, who were then forced to reconcile where they disagreed. Our pass through the test set reveals a significant number of annotation inaccuracies, particularly in the cases of the EQUAL and VAGUE relations. As detailed in Table 5, we found a mere 32.26% of the EQUAL and 21.24% of the VAGUE edges to have been accurately annotated.
D.3 Discussion: Vague Relations We want to take the opportunity to discuss the issues related to the VAGUE relation. The VAGUE relation, which could be better called the underspecified relation, describes relations that are uncertain due to insufficient context. However, this certainty is highly subjective to the annotator and hard to rigorously quantify. TBD adopted an “80% rule” that instruct annotators to label a relation as vague if they are “80% confident that it was the writer’s intent that a reader infer that relation.” As a result, inter-annotator agreement on vague edges is the lowest among all relation types.
MATRES took a different approach and treats certain combinations of the start time comparisons as vague relations. The approach attenuates the issue but by no means solve the issue. The two heuristic questions are also subject and involves a degree of uncertainty. Furthermore, similar to EQUAL relations, a simple heuristic question mistake could result in incorrect VAGUE relation annotation.
With all the aforementioned issues, CleanMATRES excludes all VAGUE relations from the test set. We only consider BEFORE, AFTER, and EQUAL relations during the evaluation of CleanMATRES. We also did not include the analysis of VAGUE relations in Section 2, as it requires a more thorough examination of the application of temporal calculus when underspecificity is involved. Allen’s (1983) original temporal calculus assumes all events are intervals with well-defined endpoints. Yao et al. (2020) and Niu et al. (2023) provided a more indepth analysis to the VAGUE edges and underspecificity.

heading: Future Work
text: We leave further exploration for future work. Due to restricted dataset resources, our unified model’s evaluation was conducted only using TBDense and MATRES. There are other TimeMLbased and even non-TimeML-based corpora related to TRE. We leave more evaluation to future work and encourage the community to reproduce our results on more datasets