Zero-shot Temporal Relation Extraction with ChatGPT
Year: 2023

Abstract:
The goal of temporal relation extraction is to infer the temporal relation between two events in the document. Supervised models are dominant in this task. In this work, we investigate ChatGPT’s ability on zero-shot temporal relation extraction. We designed three different prompt techniques to break down the task and evaluate ChatGPT. Our experiments show that ChatGPT’s performance has a large gap with that of supervised methods and can heavily rely on the design of prompts. We further demonstrate that ChatGPT can infer more small relation classes correctly than supervised methods. The current shortcomings of ChatGPT on temporal relation extraction are also discussed in this paper. We found that ChatGPT cannot keep consistency during temporal inference and it fails in actively long-dependency temporal inference.

text: The 22nd Workshop on Biomedical Natural Language Processing and BioNLP Shared Tasks, pages 92–102 July 13, 2023 ©2023 Association for Computational Linguistics

heading: 1 Introduction
text: The temporal relation extraction task aims to extract the temporal relation between either two event triggers in the given document (Dligach et al., 2017a). In this way, a timeline of events in the document can be constructed. It is a crucial task for many downstream NLP tasks, such as natural language understanding (Mani et al., 2006; Paul et al., 2017), storyline construction (Do et al., 2012; Minard et al., 2015), and temporal question answering (Jia et al., 2018b,a), etc. Conventionally, recent temporal relation extraction (RE) models are fine-tuned based on pre-trained language models (PLMs), such as BERT and RoBERTa (Devlin et al., 2019; Liu et al., 2019). On top of the PLMs, complex neural networks are applied to classify the temporal relations, such as self-attention (Lin et al., 2019; Ning et al., 2018a), graph convolutional networks (GCNs) (Mathur et al., 2021), and policy network (Man et al., 2022). Most well-performed temporal relation extractors are supervised models, that is, they heavily rely on annotated training documents first before extracting temporal relations on
the testing set. However, annotating the temporal relations in training documents requires much domain experts’ efforts (Naik et al., 2019; Ning et al., 2018b), which is a high cost.
Different from supervised learning methods, zero-shot learning (ZSL) (Xian et al., 2017) aims to train the model that can be generalized to unseen data without annotated training data and has attracted much attention in recent years. Most recently, large language models (LLMs) (Brown et al., 2020; Bubeck et al., 2023) such as ChatGPT1 have exhibited remarkable ability in zeroshot learning on various natural language processing (NLP) and medical tasks (Bang et al., 2023), such as information extraction (Wei et al., 2023), machine translation (Jiao et al., 2023), summarization evaluation (Luo et al., 2023), and mental health detection (Yang et al., 2023). However, the performance of LLMs in detecting temporal relations between events are not explored yet. Therefore, it is an urgent and spontaneous question if the LLM can perform zero-shot temporal relation extraction tasks well given a proper prompt approach, and if LLM can be the new paradigm for temporal RE.
In this paper, we explore the performance of ChatGPT on the zero-shot temporal relation extraction, and propose three different prompt strategies to interact with ChatGPT. More specifically, we start with the simple zero-shot prompt that directly requires ChatGPT to infer the temporal relation given the document. Then, we design the event ranking prompt, where ChatGPT is asked to infer the events shown in the given document instead of inferring temporal relations. Finally, we propose the chain-of-thought (CoT) prompt (Wei et al., 2022) to break down the task into two-stage, which guides ChatGPT to make temporal relation reasoning step by step. Based on our experimental results and analysis, we have the following findings:
1https://openai.com/blog/chatgpt
92
• Overall Performance. ChatGPT significantly underperforms advanced supervised methods and even traditional neural network methods such as Bi-LSTM, indicating the challenge of temporal relation detection with ChatGPT without task-specific fine-tuning.
• Prompts. Similar to the finding of recent efforts, the CoT prompt can significantly improve ChatGPT’s performance compared with other prompts across all datasets, indicating the importance of proper prompt engineering.
• Limitations. Compared with supervised methods, ChatGPT has better performance in the temporal relations with small proportions in the dataset. However, it is also found to have limitations in detecting long-dependency temporal relation extraction and inconsistent temporal relation inference.

heading: 2 Related Work

heading: 2.1 Temporal Relation Extraction
text: Several studies have explored the use of temporal information in relation extraction. For example, Han et al. (Han et al., 2019b,a) incorporated tense information and timex temporal interactions into their models. Other researchers have proposed using graph neural networks (GNNs) to encode dependency structures, which are important for temporal relation extraction (Mathur et al., 2021; Schlichtkrull et al., 2018). Wang et al. (Wang et al., 2022b) added an attention layer to an R-GCNbased model to focus on document-creation-time (DCT). Man et al. (Man et al., 2022) used a reinforcement learning framework to select optimized sentences for input into neural models, improving performance.
In the clinical domain, Leeuwenberg et al. (Leeuwenberg and Moens, 2017) applied integer linear programming constraints to learn structured temporal relations. Dligach et al. (Dligach et al., 2017b) improved performance by using neural networks such as CNN and LSTM as the backbone model. Lin et al. (Lin et al., 2019) utilized pretrained language models like BERT to learn contextualized embeddings. However, no work has yet explored the feasibility of using LLMs in temporal relation extraction.

heading: 2.2 Zero-shot Learning with ChatGPT
text: Since it was launched, ChatGPT has drawn much attention to its strong ability for various NLP tasks. In the clinical domain, Tang et al. explored ChatGPT’s ability on zero-shot named entity recognition and relation extraction (Tang et al., 2023). The experiments on NCBI Disease and BC5CDR Chemical datasets showed that ChatGPT cannot recognize named entities correctly in the clinical domain as the F1 drops around 55.94%-91.58% compared to SOTA-supervised methods. ChatGPT’s poor clinical NER was also proved by testing in i2b2 dataset (Hu et al., 2023). However, it can achieve comparable performance in clinical relation extraction as the F1 score only decreased by 4.73%-10.93% (Tang et al., 2023; Agrawal et al., 2022).
In extraction-related tasks, some work evaluated ChatGPT’s ability in event extraction, general information extraction, and relation extraction (Tang et al., 2023; Gao et al., 2023; Wei et al., 2023). The evaluation process follows multi-stage interactions/conversations with ChatGPT and guides it to produce the desired answers. The evaluation results of these studies showed that with proper prompting, ChatGPT can achieve comparable performance with the supervised methods on zero-shot or fewshot settings of extraction tasks. However, there is also some work pointing out that ChatGPT’s ability is still limited in some specific extraction scenarios such as extracting clinical notes with privacy information masked (Tang et al., 2023). Some work also discussed that for non-digital-available texts, such as historical documents, the entity recognition was performed poorly by ChatGPT (González-Gallardo et al., 2023; Borji, 2023).

heading: 3 ChatGPT for Zero-shot Temporal Relation Extraction
text: Given the input document, the temporal relation extraction aims to identify the temporal relation between any two event triggers in the document, which is modeled as the multi-classification problem. We propose three different prompts methods to evaluate ChatGPT’s performance on temporal relation extraction as shown in Figure 1.

heading: 3.1 Zero-shot Prompt
text: In this prompt, given the document D = {x1, x2, x3, · · · , xn}, we first label the event triggers with <EVENT></EVENT>. That is
if xi is an event trigger, we label it as <EVENT>xi</EVENT>. Then the whole labeled document is sent to ChatGPT and we ask ChatGPT to find the temporal relations between either two events. Note that our goal is to test ChatGPT’s zeroshot ability in the temporal relation extraction task. Therefore, we do not provide any examples in our prompts. As shown in Figure. 1, we give ChatGPT the whole document, the list of all temporal relations, and the annotations of event triggers. In the end, the input question is designed as “what is the temporal relation between <EVENT>xi</EVENT> and <EVENT>xj</EVENT>?”. We let ChatGPT to answer the question by using the temporal relations provided in the list.

heading: 3.2 Event Ranking Prompt
text: Further, we design a new prompt to make the task easier to learn for ChatGPT. Specifically, given the document D and one event trigger ei with the form <EVENT>xi</EVENT> and temporal relation set R, we require ChatGPT to complete the query (ei, rj , ?) ∀rj ∈ R. In this way, instead of querying (ei, ?, ej) as in the previous prompt, ChatGPT is required to predict the missing event trigger. As event triggers are already shown in the given text/document, ChatGPT is more likely to infer the event triggers than unseen temporal relations as they are not explicitly provided in the context. In detail, as shown in Fig. 1, we achieve this by asking the question first, such as “Which events happened
before ei?”. Then based on the feedback of ChatGPT, we form the < ei, r, ej > triplets to perform the evaluation. Note that if the same event pair is detected in different temporal relation (>2), we denote that this event pair has a “vague” relation as ChatGPT cannot confidently determine which temporal relation the event pair is classified. We also asked ChatGPT to provide some concise prompts to perform temporal relation extraction tasks. As shown in Fig 2, the prompts provided by ChatGPT are in line with our event ranking prompt approach.

heading: 3.3 Chain-of-thought Prompt
text: We notice that if two event triggers refer to the same event but point to different timestamps of the duration of that event, ChatGPT cannot distinguish them. For example, in Figure. 1, “<EVENT e3> started” and “<EVENT e6> ended” referred to the beginning and the end of the season event. ChatGPT assumes that these two event triggers happened at the same time if we directly ask about the temporal relation following the zero-shot prompt.
Therefore, we propose a new chain-of-thought prompt with two steps, which firstly navigates ChatGPT to distinguish event triggers referring to the same event, and then guides ChatGPT to infer their temporal relation. Specifically, given the document D and two event triggers e1 and e2, we first ask ChatGPT to determine if e1 and e2 refer to the same event. If they are not, we further ask ChatGPT to determine the temporal relation between the two event triggers. If they point to the same event, we use a similar prompt but with the extra phrase “in that event” to ask ChatGPT. As shown in Fig. 1, we first ask ChatGPT if the two event triggers <EVENT>started</EVENT> and <EVENT>ended</EVENT> refer to the same event. Then based on ChatGPT’s feedback, we further iteratively go through the whole temporal relation list to determine which temporal relation exists between the two event triggers.

heading: 4 Experiments

heading: 4.1 Datasets
text: We use three datasets to evaluate the performance of ChatGPT on the zero-shot temporal relation extraction. The statistical details of these datasets are shown in the Table 1.
• TimeBank-Dense (Cassidy et al., 2014) labeled 36 news documents in total. The temporal relations between event triggers-event triggers, timex-event triggers, timex-timex, are labeled. Following previous work, we only test our model on the event trigger-event trigger relation in the testing set.
• MATRES (Ning et al., 2018b) is a dataset primarily focusing on temporal relations of event triggers with local sentences (1 or 2 sentences).
• TDDiscourse (Naik et al., 2019) was created to explicitly emphasize global discourse-level
temporal ordering. Based on the annotation accuracy, the dataset is split into TDDMan and TDDAuto, where TDDAuto introduces much more automatic labels and noise. In this paper, we only evaluate ChatGPT on TDDMan because of the budget limitation.
We only use the testing set of each dataset directly to test our approaches as we do not require training of ChatGPT, following the zero-shot setting. We report the F1 score on each dataset and each temporal relation.

heading: 4.2 Baseline Models
text: Since there is no zero-shot learning methods for temporal relation extraction before, we compare the performance of ChatGPT with the following advanced supervised methods:
• CAEVO (Chambers et al., 2014) a sieve-based architecture that includes multi-level classifiers for intra-sentence temporal relation learning.
• SP+ILP (Ning et al., 2017) a structured learning approach that captures the global temporal features when inferring the relation between two local events.
• Bi-LSTM (Cheng and Miyao, 2017) a BiLSTM-based model that encodes the dependency path between two events to classify temporal relation.
• Joint (Han et al., 2019b) and Deep (Han et al., 2019a) models that utilized SSVM as the scoring function to learn temporal constraints and context embeddings.
• UCGraph (Liu et al., 2021) a graph-based model that is trained with mask pre-training mechanism. The model’s uncertainty level is used to guide the inference during testing.
• TIMERS (Mathur et al., 2021) a graph-based model that leverages three graphs to learn temporal, rhetorical, and syntactic information, respectively.
• SCS-EERE (Man et al., 2022) a reinforcement learning-based selector is designed to select the optimized sentences for temporal inference between the given two events.
• RSGT (Zhou et al., 2022) a syntactic-andsemantic-based graph model pre-trained on a temporal neighbor prediction task.
• FaithTRE (Wang et al., 2022a) a model that applied Dirichlet prior to estimating the correctness likelihood. A temperature scaling is also used to recalibrate the model confidence measure after bias mitigation.
• DTRE (Wang et al., 2022b) a document creation time(DCT)-aware graph with a global consistency mechanism when inferring temporal relations.
• MulCo (Yao et al., 2022) a joint model using the BERT to learn contextualized features and GNN to capture syntactic structures. The two models are combined via a multi-level contrastive learning framework.

heading: 4.3 Results
text: In table 2, we can see that ChatGPT struggles to outperform supervised state-of-the-art models such as SCS-EERE (Man et al., 2022) and RSGT (Zhou et al., 2022), and even the traditional neural networks methods such as CAEVO and BI-LSTM, indicating its ineffective in the temporal relation extraction task in the zero-shot setting. Table 2 also shows the performance of ChatGPT under different prompts in three datasets. We noticed that ChatGPT_ER yields the worst performance on the MATRES and TDDMan datasets. ChatGPT’s performance with the event ranking prompt on TDD-Man dataset is poor as most event trigger pairs cannot be detected under this prompt. However, if the event trigger pairs are explicitly fed to ChatGPT, it can somehow partially infer the temporal relations correctly. For example, the zeroshot prompt and CoT prompt could improve the F1 score by 14.8% and 23.8%, respectively. While for the TB-Dense dataset, ChatGPT_ZS has the worst performance. ChatGPT_CoT achieves the best performance across all datasets, such as it significantly outperforms ChatGPT_ZS and ChatGPT_ER by 27.1% and 33.1%. This illustrates the effectiveness of the CoT prompt with step-by-step guidance in prompting ChatGPT.
Table 3, Table 4 and Table 5 further list the detail results of ChatGPT with different prompts on three datasets. We can see that the performance with the event ranking prompt is much better than that on other datasets. ChatGPT with the zero-shot
prompt cannot determine the temporal relation “is included” and therefore yields a 0.0 performance in this type of relation. The CoT prompt improves the overall performance by significantly detecting “before” and “after” temporal relations. As these two relations take a great portion of the whole dataset, the overall performance is also improved.

heading: 5 Discussion

heading: 5.1 ChatGPT is slightly better on small temporal relation classes
text: The imbalanced data is a severe long-existing problem in the temporal relation extraction task. Because of the events’ temporal order frequency in real life, some temporal relations, such as “simultaneous” and “equal”, are very limited in most temporal relation extraction datasets. And popular NLP data-augmented methods are difficult to be applied in the temporal domain. Therefore, most stateof-the-art supervised methods yield much worse performance on small relation classes.
In Deep (Han et al., 2019a), the authors reported detailed performance on each temporal class in MATRES dataset. We therefore especially compare the performance on small relation classes against Deep, i.e., “EQUAL” and “VAGUE”. As shown in Table 3, the supervised model Deep could achieve much better overall performance (81.7 F1 score), due to the contribution of two majority relations, “before” and “after”. Compared to Deep’s 0.0 performance on “EQUAL” and “VAGUE”, ChatGPT with event ranking, CoT and zero-shot prompts can correctly extract some small class relations.

heading: 5.2 ChatGPT failed in actively long-dependency temporal relation extraction
text: As shown in Table 2, ChatGPT’s performance drops a lot in the TDDMan dataset. We argue that this is mainly because the TDDMan focuses more on discourse-level temporal relations and ChatGPT failed to extract useful information from long documents. As shown in Table 4, the event ranking prompt yields almost 0.0 on the whole dataset. In practice, we initially input the whole document D to ChatGPT and ask ChatGPT which event triggers in the document D happened before the given event trigger e1. That is, ChatGPT should actively search all event triggers in the document and produce answers. However, in the TDDMan dataset, ChatGPT cannot produce a formatted answer and the outputs
sometimes are even some random words in the document instead of event triggers. We then test the limit of the size of the input document, i.e., the number of sentences. We finally found that if we limit the size of the input document to at most 8 context sentences around the event triggers, ChatGPT would be more stable to produce formatted answers instead of repeating part of the document randomly. However, in this way, most temporal relations extracted by ChatGPT do not match with the golden labels in the TDDMan dataset because the extracted temporal relations are only in short dependency while TDDMan emphasizes long-distance temporal dependency.
Nevertheless, surprisingly, if we explicitly ask ChatGPT what the temporal relations between two event triggers labeled in the document are, ChatGPT can answer some of them correctly. Note that we do not cut the size of the document in this case
and ChatGPT can still learn some of the temporal dependency in the document. Compared to the event ranking prompt, ChatGPT passively receives two event trigger information with the zero-shot and CoT prompts. This may reduce the inference difficulty as more information is given.

heading: 5.3 ChatGPT can be improved via multi-stage “yes” or “no” prompts
text: Intuitively, the most efficient way to query ChatGPT about temporal relations between two events in the given document should be the zero-shot prompt, which directly asks the ChatGPT to answer the temporal relation between any two event triggers. If only one event trigger is given, then the prompts provided by ChatGPT, i.e., event ranking, should be used to interact with ChatGPT. However, our extensive experiments show that these two prompt methods produce much worse perfor-
mance in most cases compared to the CoT prompt. Note that both zero-shot and CoT provide sufficient information about event triggers. We argue there is another difference resulting in the performance gap.
Comparing the two prompts, one significant difference is that the CoT prompt only accepts “yes” or “no” answers while the zero-shot prompt returns a specific temporal relation label. In the zero-shot prompt, ChatGPT is required to select a temporal relation from the given list, which is similar to a conventional multi-class classification problem. However, in the CoT prompt, ChatGPT only has to determine if one specific temporal relation exists (or not) between two event triggers. This simplified the problem into a binary classification. Further, with the previous question-answer pair as context, ChatGPT has a higher probability of making the correct selection. For example, in Fig 1, in the first round, ChatGPT already inferred that the relation is not “EQUAL”. Then in the second round, ChatGPT is more confident to predict the temporal relation from [BEFORE, AFTER, VAGUE] instead of “EQUAL”.

heading: 5.4 ChatGPT’s temporal inference is inconsistent even with sufficient context
text: During the testing of ChatGPT with the event ranking prompt, we noticed a fatal issue in ChatGPT’s temporal relation extraction, namely the inconsistent temporal relation inference. Given the same input document, ChatGPT may produce different temporal relations between two event triggers. As the top example shown in Figure 3, given the document D, if the prompt is “Which event triggers happened before e1?”, ChatGPT will give a list of event triggers, e.g, [e2, e5, e6]. Now given the same document D, if the prompt is “Which event triggers happened after e6”, ChatGPT is expected to at least include e1 in the list. However, during the experiments, we noticed that ChatGPT failed in this scenario multiple times and the failures can be categorized into two cases. The first failure case is that ChatGPT does not include e1 in any list associated with e6. The second case is that ChatGPT includes e1 in a wrong list, e.g., “EQUAL”, associated with e6 which violates the temporal consistency.
A similar problem also happened to the CoT prompt. As shown in Figure 4, during the experiments, ChatGPT would give various answers instead of “vague” as we specified, if it thinks there
is no clue to infer the temporal relation between the given two event triggers. These answers include “Cannot determine.”, “I cannot answer that question as it is unclear from the given information.”, “Unknown.”, etc. We treat all of these answers as “vague” when we evaluate ChatGPT. Intuitively, if ChatGPT cannot answer “yes” or “no” for the specific temporal relation between two event triggers, it should also not be able to answer questions about other temporal relations of the same event triggers. However, ChatGPT may violate its “unknown” statement in the multi-stage prompts. We test if ChatGPT’s inconsistency exists in multistage prompts by implementing the following experiments. For example, during the i-th round, if we ask “Did e1 happen before e2?” and ChatGPT’s answer includes “It is unclear from the given information.”. We then ask the next i+ 1 question “Did e1 happen after e2?”. And surprisingly, in most cases (84%), ChatGPT can classify the event trigger pair into other temporal relation classes even if it claims that the information is insufficient. Moreover, 96% of these classification results is incorrect.

heading: 6 Conclusion
text: In this work, we comprehensively test ChatGPT’s zero-shot ability on temporal relation extraction. We designed three different prompts to evaluate ChatGPT’s performance. Our experiments demonstrate that with proper prompting, ChatGPT’s performance on zero-shot temporal relation extraction can be significantly improved, highlighting the importance of prompt engineering to better trigger ChatGPT’s ability in future work. However, compared to supervised methods, ChatGPT is still far behind. We further discuss our findings from experimental results, including its better performance in small classes than supervised methods and its drawbacks such as failures in long-distance temporal dependency inference and inconsistent temporal relation inference. Our work only takes the initial step of exploring the LLMs for zero-shot temporal relation extraction. To fill the gap between the performance of LLMs in the zero-shot setting and that of advanced supervised methods, we believe more efforts should be explored in the future.

heading: 7 Acknowledgements
text: This work is supported by the project JPNP20006 from New Energy and Industrial Technology Development Organization (NEDO).

heading: Future Work
text: Our experiments demonstrate that with proper prompting, ChatGPT’s performance on zero-shot temporal relation extraction can be significantly improved, highlighting the importance of prompt engineering to better trigger ChatGPT’s ability in future work. However, compared to supervised methods, ChatGPT is still far behind. We further discuss our findings from experimental results, including its better performance in small classes than supervised methods and its drawbacks such as failures in long-distance temporal dependency inference and inconsistent temporal relation inference. Our work only takes the initial step of exploring the LLMs for zero-shot temporal relation extraction. To fill the gap between the performance of LLMs in the zero-shot setting and that of advanced supervised methods, we believe more efforts should be explored in the future