Introduction
Reasoning regarding the temporality of events detected in a text (e.g., understanding their duration, frequency, and order) is an essential part of natural language understanding Allen ( 1983 ); Wenzel and Jatowt ( 2023 ) . Event ordering can be approached as identifying temporal relations between two events, a task often referred to as temporal relation extraction (TempRE).
This task can also be applied to medical text (BioTempRE), e.g., clinical notes written by clinicians regarding a patient’s visit, and various medical events such as symptoms, treatments, tests, and other medical terms (see Figure 1 ).
BioTempRE has numerous useful applications in healthcare and can assist in medical diagnosis, including adverse drug event detection and medical history construction Sun et al. ( 2013 ); Gumiel et al. ( 2021 ); Haq et al. ( 2021 ); Tu et al. ( 2023 ) .
Current state-of-the-art methods perform supervised learning, which requires annotated datasets Wang et al. ( 2022 ); Yao et al. ( 2022 ); Knez and Žitnik ( 2024 ) . However, acquiring high-quality annotated data for TempRE poses significant challenges causing problems to existing datasets like missing relations and low inter-annotator agreement Ning et al. ( 2017 ) . In the biomedical domain, this challenge is aggravated by the need for expert knowledge and the sensitive nature of medical data.

In TempRE, there are important properties that emerge from the temporal nature of this task and determine the relations between events (Figure 1 ). Such properties are symmetry (e.g., A BEFORE B \Rightarrow B AFTER A) and transitivity (e.g., A BEFORE B and B BEFORE C \Rightarrow A BEFORE C). We also identify the property of uniqueness : each pair of events can have only one temporal relation since they are mutually exclusive. These properties can be utilized to enforce global temporal consistency on a model’s predictions: for example, on a unified output of different classifiers Chambers et al. ( 2014 ); Tang et al. ( 2013 ) , on a model that operates locally (i.e., with one pair of events as input, Ning et al. ( 2017 ) ), or on predicted relations between different types of events Wang et al. ( 2022 ) .

Recently, LLMs have shown remarkable performance in several tasks even in a zero-shot setting, which helps to tackle the need for training data Bubeck et al. ( 2023 ); Wei et al. ( 2022a ) .
Numerous works experiment with predictions of LLMs and study their reasoning abilities and the impact of various prompts in different tasks Wu et al. ( 2023b ); Jain et al. ( 2023 ); Tan et al. ( 2023 ) . Despite the success of LLMs, studies show that these models continue to face challenges in temporal reasoning, especially in TempRE Tan et al. ( 2023 ); Jain et al. ( 2023 ); Yuan et al. ( 2023 ) , as well as in biomedical tasks Wu et al. ( 2023b ) .
In zero-shot TempRE, Yuan et al. ( 2023 ) employed different prompts for ChatGPT and found that it has a considerably lower performance compared to standard supervised methods. They also report ChatGPT’s tendency to provide temporally inconsistent responses, but did not perform an evaluation of temporal consistency specifically. Moreover, none of the mentioned papers investigated the temporal reasoning capabilities of LLMs on medical data.

In this paper, we perform zero-shot BioTempRE on clinical notes (i.e., medical texts documenting patients visits) by using prompts consisting of a clinical note and questions regarding which temporal relation exists between a pair of events. 1 1 1 We do not perform event detection and instead consider the events in each text already known. We experiment with two different prompting strategies (BatchQA and CoT) and five widely-used LLMs (GPT-3.5, Mixtral 8x7B, Llama2 70B, Gemma 7B, and PMC-LLaMA 13B). Our findings reveal a poor performance of LLMs in this task with a difference of approximately 0.2 in F1 score compared to supervised models.
Furthermore, we calculate consistency scores for uniqueness and transitivity for each LLM in order to assess their temporal consistency and its impact on accuracy.
Consistency is later enforced on the predictions with an Integer Linear Programming (ILP) method, revealing that solving the inconsistencies does not improve the F1 score.

Overall, our contributions are:

To the best of our knowledge, this is the first study of zero-shot BioTempRE.

We provide extensive quantitative results of two types of prompts and five different LLMs.

We perform a novel temporal consistency analysis by calculating consistency scores of temporal properties.

We study how temporal consistency relates to accuracy and enforce it using an ILP method.

The code and data containing the prompts, the raw and the processed responses by the LLMs for around 600,000 pair instances, will be publicly shared for further analysis. 2 2 2 https://github.com/vasilikikou/consistent_bioTempRE