RiteeshGrandhi/sciclaimeval-shared-task huggingface dataset:
Temporal Relation Extraction
Multiple studies in TempRE have applied temporal properties to classifiers’ predictions, either during training or at inference time, aiming to improve their performance Tang et al. ( 2013 ); Chambers et al. ( 2014 ); Ning et al. ( 2017 , 2018 ); Wang et al. ( 2022 ) . Other works have also employed linguistic properties or properties based on causality Chambers et al. ( 2014 ); Ning et al. ( 2018 ) . Ning et al. ( 2018 ) formulated temporal, causal, and linguistic properties as constraints for an ILP method. Later, Liu et al. ( 2021 ) showed that ILP constraints can improve temporal consistency, although in certain cases, the F1 score may decrease.
Temporal Relation Extraction in the Medical Domain.
The 2012 Informatics for Integrating Biology and the Bedside (i2b2) challenge was the first to address the BioTempRE task Sun et al. ( 2013 ) .
The best-performing method involved merging predictions from different SVM and CRF classifiers with regard to temporal consistency Tang et al. ( 2013 ) .
Following challenges at SemEval, called Clinical TempEval, were organized from 2015 to 2017 Bethard et al. ( 2015 , 2016 , 2017 ) and utilized the THYME corpus Styler IV et al. ( 2014 ) . 3 3 3 The i2b2 dataset is publicly available. The THYME corpus is provided upon request, however our requests were not answered. In 2015 and 2016, the best-performing methods were CRF- and SVM- based Velupillai et al. ( 2015 ); Lee et al. ( 2016 ); Khalifa et al. ( 2016 ) , while in 2017 the winning approach employed an LSTM Tourille et al. ( 2017 ) . Following approaches have utilized BERT Lin et al. ( 2019 ); Haq et al. ( 2021 ); Tu et al. ( 2023 ) for relation classification given a text and an event pair. Recently, Knez and Žitnik ( 2024 ) introduced a multimodal method in which, they construct a graph with medical information and then, they combine textual representations (extracted by BERT) and graph representations (extracted by a GNN). Even though temporal consistency has been used in existing TempRE works, it has not been utilized for analyzing the performance of a model by calculating consistency scores.
Zero-Shot Temporal Relation Extraction
Zero-shot learning (Xian et al., 2019 ) enables models to execute tasks without explicit training, a capability demonstrated by scaling models since GPT-3 (Brown et al., 2020 ; Wei et al., 2022a ) .
Instruction tuning techniques Wei et al. ( 2022a ) further enhance zero-shot learning in LLMs.
Recent openly available LLMs like LLama Touvron et al. ( 2023 ) and Mixtral Jiang et al. ( 2024 ) narrow the gap with closed-source models, while chain-of-thought (CoT) prompting Wei et al. ( 2022b ) has enhanced their ability to handle complex tasks.
Research studies have shown that the temporal reasoning tasks remain challenging for LLMs Jain et al. ( 2023 ) , and specifically for TempRE, where Yuan et al. ( 2023 ) explored zero-shot TempRE with ChatGPT and found that it yields a large performance gap compared to supervised methods. However, previous research has not analyzed zero-shot TempRE in the medical domain or the temporal consistency and its impact on the performance of zero-shot TempRE - both gaps we aim to fill in our work. In this paper, we calculate consistency scores and study their connection to the F1 scores.
Problem Formulation
Given a text document D and a set of events E=\{e_{1},..,e_{|E|}\} mentioned in the text, we create pairs of events, which are represented by the set P=\{p_{1},..,p_{i},..,p_{|P|}\} , where p_{i} indicates the i^{th} pair, 1\leq i\leq|P| .
BioTempRE aims at assigning the appropriate temporal relation r to the corresponding pair of events. Each p_{i}\in P is described by two distinct events \{e_{j},e_{k}\} , where 1\leq j,k\leq|E| . Furthermore, each event e\in E is characterized by the points in time at which it began and finished.
These temporal points are denoted as b and f , respectively.

Following the work of Ning et al. ( 2018 ) , we employ the same relation scheme, which consists of 5 different types of temporal relations r : before , after , includes , is included , and simultaneously , represented by the label set R_{T}=\{r_{B},r_{A},r_{I},r_{II},r_{S}\} . We choose this set of relations based on the fact that they are fine-grained and well-defined, and hence, suitable for creating temporal rules for our analysis.
An r_{B} temporal relation indicates that b(e_{j})<b(e_{k}) and f(e_{j})<f(e_{k}) , while an r_{A} temporal relation signifies that b(e_{j})>b(e_{k}) and f(e_{j})>f(e_{k}) . Furthermore, r_{I} indicates that b(e_{j})\leq b(e_{k}) and that f(e_{j})\geq f(e_{k}) , and r_{II} signifies that b(e_{j})\geq b(e_{k}) and that f(e_{j})\leq f(e_{k}) . Finally, r_{S} signifies that b(e_{j})=b(e_{k}) and f(e_{j})=f(e_{k}) .
Zero-shot BioTempRE
We experimented with two different types of prompting: Batch-of-Questions (BatchQA) and Chain-of-Thought (CoT) Wei et al. ( 2022b ); Yuan et al. ( 2023 ) (see Figure 4 in Appendix A ). In both, we start with a preamble consisting of the document text ( D ) and an instruction. Then, we introduce questions regarding the temporal relations for a pair of events p_{i} consisting of events e_{j} and e_{k} . 4 4 4 The questions were ordered randomly. We formulated the question for each relation based on its temporal definition, as follows:

BEFORE : Did e_{j} start before e_{k} started and end before e_{k} ended?

AFTER : Did e_{j} start after e_{k} started and end after e_{k} ended?

INCLUDES : Did e_{k} start and end while e_{j} was happening?

IS INCLUDED : Did e_{j} start and end while e_{k} was happening?

SIMULTANEOUS : Did e_{j} and e_{k} start and end at the same time?

We also specify the desired output format by adding “Answer with Yes or No” to the end of each question.
For each event pair there is an independent interaction with the LLM, and depending on the type of prompt the questions mentioned above is sent to the LLM in one or multiple prompts.
Batch-of-Questions (BatchQA).
In BatchQA, a single prompt is sent to the LLM. In the preamble, after the document D , this instruction follows: “Given document D , answer the following questions ONLY with Yes or No.” . Next, all the questions regarding the temporal relations are added in the same prompt. The expected model response includes five Yes/No answers for each of the questions.
Chain-of-Thought (CoT).
The CoT approach is based on the chain-of-thought method proposed by Yuan et al. ( 2023 ) and follows a chat-like style consisting of multiple prompts. The first prompt is the preamble composed of the document D and the question “Given the document D , are e_{j} and e_{k} referring to the same event? Answer ONLY with Yes or No.” .
If the response is No , then the questions are sent, each one in a separate prompt as they are mentioned above.
If the response is Yes , the phrase “In that event,” is appended in the beginning of each question.
Data
In our experiments, we use the dataset created for the 2012 i2b2 challenge, which consisted of 310 discharge summaries, 190 for training and 120 for testing. The texts were initially annotated with 8 fine-grained relations but due to low inter-annotator agreement these relations were merged to the following three: before, after and overlap . Each discharge text contains 30.8 sentences on average, with each sentence having an average number of 17.7 tokens. The average number of tokens per discharge text is 514.

The i2b2 dataset contains three types of events: 1) medical events, 2) time expressions, and 3) the dates of admission and discharge. The average number of medical events per discharge summary is 86.7, while the average number of time expressions is 10.5.
The admission and discharge dates are included in each text; however, in a few cases, one of them might be missing. The annotators of i2b2 have assigned temporal relations to 27,540 pairs of events (gold pairs).

An important step in TempRE is to identify the pairs of events for which the models will decide if there is a relation expressed or not since it would not be feasible to check for every pair of events mentioned in a document. In order to generate candidate event pairs, we follow the approach of the best-performing method in the i2b2 challenge Tang et al. ( 2013 ) . This is a rule-based approach, which creates pairs consisting of every event and the admission and discharge dates, every two consecutive events within the same sentence, and events in the same as well as in different sentences based on linguistic rules. The generated candidate pairs are 60,840 in total, from which 28.16% appears also in the gold pairs.

The five relations we use in our experiments (see Section 3 ) are different from the gold ones existing in the dataset. In order to evaluate the prediction of our methods, we map the five relations to the three gold ones as follows: before \rightarrow before, after \rightarrow after, includes \rightarrow overlap, is included \rightarrow overlap and simultaneously \rightarrow overlap.
LLMs
We employed the following five (one closed-source and four open-weight) models of various sizes: GPT-3.5 (“ChatGPT”), 5 5 5 https://openai.com/index/gpt-3-5-turbo-fine-tuning-and-api-updates/ Gemma 7B Team et al. ( 2024 ) , 6 6 6 https://huggingface.co/google/gemma-1.1-7b-it Mixtral 8x7B Jiang et al. ( 2024 ) , 7 7 7 https://huggingface.co/mistralai/Mixtral-8x7B-Instruct-v0.1 Llama2 70B Touvron et al. ( 2023 ) , 8 8 8 https://huggingface.co/meta-llama/Llama-2-70b-chat-hf and PMC-LLaMA 13B, which was pre-trained on medical text Wu et al. ( 2023a ) . 9 9 9 https://huggingface.co/axiong/PMC_LLaMA_13B PMC-LLaMA was only instruction-tuned on QA data (respond to one question at a time) and struggled to follow the format of BatchQA prompts. Therefore, it was possible to use it only for CoT. The experiments were costly in terms of time (and money for GPT-3.5), especially for CoT where each question is sent separately. The running times range from three hours (Gemma BatchQA) to 7 days (Llama CoT) (see more details in Appendix A ).
Baselines
We implemented a rule-based baseline, called W-order, where only the before and after relations are predicted for each event pair based on the order in which the events are mentioned in the text. A combination of the predictions of each LLM with the W-order predictions was also implemented. In cases where the LLM gave an negative or uncertain prediction for all the relations, the prediction of W-order was used instead.
Zero-shot TempRE results
To evaluate the correctness of the predicted relations, we calculate the precision, recall and F1 scores.
For each pair of events, we check if the predicted relation matches the gold relation. Hence, we calculate the triple match between the predictions and the ground truth. In Table 1 , the results for the gold and candidate pairs are presented. In order to perform a fair comparison, considering that not every candidate pair of events has a gold annotation (and therefore it is unknown whether a prediction is correct or wrong), we only evaluate those generated candidate pairs that also are contained in the gold pairs. (If a gold pair does not exist in the generated candidate pairs, there is no prediction for it, and that would affect the recall score negatively.) In the Supervised setting we show scores reported by the corresponding papers. Knez and Žitnik ( 2024 ) do not mention event detection or candidate pair generation, hence we assume they used the gold pairs. On the other hand, we show the results from Haq et al. ( 2021 ) and Tu et al. ( 2023 ) in the Candidate column since they operate on events they have detected in the text.

Our experiments demonstrate that the best performing methods are the same for the gold and the candidate pairs. As expected, the F1 score of the methods when the candidate pairs are used is lower, mostly due to the decrease in recall. The best performing method is Llama CoT + W-order in terms of F1 score. On the other hand, Mixtral CoT achieves the best precision score and Gemma BatchQA + W-order the best recall.
Overall, the supervised methods consistently outperform the methods in the zero-shot setting, with an avarage difference of approximately 20% F1 score.
In general, most LLMs (except for Gemma ) exhibit improved performance when the CoT prompting approach is used.
However, in an LLM-based comparison, we observe that the performance varies depending on the type of prompt used. For example, Llama with CoT has the highest F1 score, but when BatchQA is used, the score drops almost in half.
Moreover, the combination of W-order predictions with the zero-shot methods yields improvements in recall and F1 score, but in most cases, it harms precision. Notably, PMC-LLaMA , the medical LLM we employed, has low results and is often outperformed by the general domain LLMs, showing no advantage from pre-training on biomedical text.
Temporal consistency analysis
Considering the temporal nature of the TempRE task, we investigate the impact of incorporating the following two temporal properties in the zero-shot setting: 1. uniqueness , requiring that each event pair has exactly one relation, and 2. transitivity . First, we evaluate the zero-shot methods based on their consistency, i.e., if their predictions follow the temporal properties or not. Then, we use ILP to enforce temporal consistency on the predictions. Specifically we examine the following three questions:

How consistent are the different zero-shot methods?

How is the temporal consistency of the predictions connected to their correctness?

Can the predictions be improved by a temporal constraint-based ILP method?
How consistent are the different zero-shot methods?
We calculate two consistency scores: one for uniqueness c_{U} and one for transitivity c_{T} , which show the percentage of cases where the corresponding temporal property was not violated. The consistency score for uniqueness is calculated based on the number of pairs as follows:

c_{U}=\frac{\sum_{i=1}^{P}p_{i,|r_{i}|=1}}{|P|}, (1)

where only the pairs p_{i} with a singular predicted relation r_{i} are considered.
In Table 2 , the consistency scores for uniqueness are reported. Furthermore, we present the number of pairs for which no relation was predicted (# 0) and the number of pairs with more than one predicted relation (# >1).
We observe that all the models struggle to keep consistency, especially because of assigning more than one relation to a pair. For the majority of the evaluated LLMs, this occurs for at least 50% of the pairs and can go up to 97% (Gemma BatchQA). In this evaluation, we also find that there is no clear winner among the LLMs or the prompt types, since the same LLM shows different levels of consistency with a different prompt type. The combination with the highest consistency for uniqueness is Llama with BatchQA.

The consistency score for transitivity is calculated based on triples of event pairs in the following form: ((e_{i},e_{j}),(e_{j},e_{k}),(e_{i},e_{k})) . We first find these triples in the dataset and then obtain the relations predicted for them. If r_{1} , r_{2} and r_{3} are the predictions for each respective pair in the triple, then for r_{3} , it should hold that r_{3}\in trans(r_{1},r_{2}) . 10 10 10 The transitive relations for the relation set we used can be found in Table 4 in Appendix A . If it does not hold, then we have a transitivity violation. Therefore c_{T} is calculated as:

c_{T}=\frac{\sum_{i=1}^{|Tr|}t_{i,r_{3}\in trans(r_{1},r_{2})}}{|Tr|}, (2)

where Tr is the set of transitivity triples and, for each triple t_{i} , the transitivity for the predicted relations holds. 11 11 11 Triples where at least one pair was not assigned a relation are excluded from this calculation. Table 2 showcases the c_{T} scores for each of the evaluated methods. Similar to uniqueness , Llama BatchQA demonstrates the highest consistency for transitivity . In general, for all LLMs, we observe that the BatchQA approach yields higher transitivity consistency scores than CoT.
How is the temporal consistency of the predictions connected to their correctness?
When comparing the consistency scores with F1, we observe a contradiction. Models that have high consistency have a lower F1 score. In particular, Llama BatchQA is the most consistent in terms of uniqueness and transitivity, but has one of the lowest F1 scores. Especially for the candidate pairs, the F1 score is even lower than the rule-based baseline, yet c_{U} is 70.58% and c_{T} is 80.05%. Moreover, Llama CoT, which is the best in terms of F1 score, has low consistency with around only 30% of predictions being unique and 60% correct transitivity triples.
These insights suggest that temporal consistency does not always mean correctness.
Can the predictions be improved by a temporal constraint-based ILP approach?
Following the approach of Ning et al. ( 2017 , 2018 ) , we implemented an ILP step that uses the temporal properties as constraints and changes inconsistent predictions so that the constraints are not violated. 12 12 12 For the ILP implementation we used the Gurobi optimizer ( https://www.gurobi.com/solutions/gurobi-optimizer/ ). With this study we examine whether consistency can improve the predictions. First, we assign a confidence score sc to each triple (e_{i},e_{j},r_{k}),\forall r_{k}\in R_{T} . The score sc for a pair of events p=(e_{i},e_{j}) equals 1, if the relation was predicted from the model, and 0.2 otherwise. Next, we create a binary vector, which is optimized with ILP. We refer to it as the indicator I(p_{i},r_{i})\in[0,1],\forall p\in P,r\in R_{T} .
We formulate the constraints as follows:

Uniqueness:

\sum_{p\in P,r\in R_{T}}I(p,r)=1 (3)

Symmetry:

I(p_{i},r_{i})=I(p^{s}_{i},\bar{r}_{i}) (4)

where p_{i}=(e_{i},e_{j}) and p^{s}_{i}=(e_{j},e_{i}) , and \bar{r}_{i} is the reverse relation of r_{i} . 13 13 13 The reverse of each relation can be found in Table 5 in Appendix A .

Transitivity:

I((e_{i},e_{j}),r_{1})+I((e_{j},e_{k}),r_{2})-\sum_{r_{3}\in tr(r_{1},r_{2})}\leq 1 (5)

where r_{1},r_{2},r_{3}\in R_{T} and trans(r_{1},r_{2}) is the set of relations that are the transitive of relations r_{1} and r_{2} .

The objective of the ILP method is to maximize the confidence score based on the indicator:

\overset{\wedge}{I}=\operatorname*{arg\,max}\sum_{p\in P}\sum_{r\in R_{T}}sc(p,r)I(p,r) (6)

As shown in Table 2 , when the ILP reasoning step is applied, the consistency scores for both uniqueness and transitivity reach 100%.
We applied this step to the predictions of Llama BatchQA and Llama CoT, which are the models with the highest contradiction between F1 and consistency.
In Table 3 , we show the results before and after applying the temporal constraints. Even though the consistency of the predictions after reasoning is 100%, the F1 score decreases slightly for BatchQA and by 0.066 for CoT. This means that the predictions are temporally consistent, but they are not accurate. To get a better understanding of this issue, Figure 2 demonstrates two examples of transitivity triples for which the predictions violate the transitivity constraint. This indicates that at least one of the three predictions is incorrect and needs to change. In the top example, the first two relations were correct, but these relations were changed by the ILP step, resulting in only one relation being correct in the consistent predictions. In the bottom example, only one relation was changed to enforce consistency. This resulted in two correctly predicted relations after the ILP, but still the first relation remained incorrect. This analysis highlights our previous observation regarding the relation between consistency and accuracy, and points out to the need of a better alignment of these two aspects in order for models to achieve a better temporal reasoning.
Pairs distance-based analysis
Since clinical notes contain long texts (see Section 4.1 ), we perform an analysis based on the distance of event pairs for the best-performing LLM (Llama CoT). First, we calculate the distance in terms of characters between the events for all the gold pairs. Then, we sort the pairs by their distances and split them to 10 bins, so that each bin contains roughly the same amount of pairs. Finally, the F1 score is calculated for the prediction of the pairs contained in each bin. Figure 3 depicts the barplot with the bars representing the pairs in the specific distance range and the their corresponding F1 scores. We observe that 37.5% of the pairs have a distance of 0 to 100 characters. Larger distances appear less frequently and hence the range of distance is smaller in the first bars, while the last bars have larger ranges.
There is no consistent decrease in F1 score as the distance increases, meaning that the model is not affected by the distance of events in the text.
Conclusion
BioTempRE is an important task that tackles medical event ordering, with beneficial applications in healthcare. In this paper, we performed BioTempRE on clinical notes in a zero-shot setting employing five different LLMs. Two types of prompts were used, namely BatchQA and CoT, in order to obtain LLMs’ responses. The zero-shot performance of all LLMs is lower compared to supervised learning methods. Moreover, we perform a temporal evaluation by calculating the consistency score of each LLM for the temporal properties of uniqueness and transivity. We find that, in general, LLMs’ predictions are temporally inconsistent and, interestingly, the model with the higher consistency scores (Llama BatchQA) has one of the lowest F1 scores. An ILP method utilized to enforce consistency on the models’ predictions fails to improve their accuracy. These findings indicate the importance of the relation between temporal consistency and correctness, emphasizing the need for further study in order to assist in temporal reasoning.
Acknowledgments
This research has been funded by the Vienna Science and Technology Fund (WWTF)[10.47379/VRG19008] ”Knowledge-infused Deep Learning for Natural Language Processing”.
Technical details
Getting responses from GPT-3.5 for all the pairs for both types of prompts costed around 800$ and lasted 27 hours.
For the open-source models we used a single H100 GPU, and for the rest two H100 GPUs.
The running time for each model was:

Mixtral 8x7B BatchQA: 6 hours

Mixtral 8x7B CoT: 48 hours

Llama2 70B BatchQA: 24 hours

Llama2 70B CoT: 7 days

Gemma 7B BatchQA: 3 hours

Gemma 7B CoT: 25 hours

PMC-Llama 13B CoT: 2.5 days
Figure 1: An example of three event pairs annotated with temporal relations. In the right part, the order of the events with respect to time (t) is shown and the consistency of uniqueness and transitivity.; An example of three event pairs annotated with temporal relations. In the right part, the order of the events with respect to time (t) is shown and the consistency of uniqueness and transitivity.
Figure 2: Examples of two transitive triples with inconsistent predictions. After the ILP the predictions are consistent but still different from the gold relations.; Examples of two transitive triples with inconsistent predictions. After the ILP the predictions are consistent but still different from the gold relations.
Figure 3: Barplot where each bar represents a range of distances between events in the gold pairs. The y axis shows the F1 score of the predictions for the pairs in each bar.; Barplot where each bar represents a range of distances between events in the gold pairs. The y axis shows the F1 score of the predictions for the pairs in each bar.
Figure 4: Examples of an interaction with the LLM using two different prompting strategies: BatchQA and Chain-of-Thought.; Examples of an interaction with the LLM using two different prompting strategies: BatchQA and Chain-of-Thought.
Precision (P), recall (R) and F1 scores of TempRE methods on the gold and candidate pairs. Methods with † use a different candidate pair generation than ours, their results are not directly comparable to ours.
Temporal consistency scores for uniqueness ( c_{U} ) and transitivity ( c_{T} ) for each model. The consistency scores show the percentage of pairs which are consistent for the corresponding property. # 0 and # >1 shows the number of pairs with none and more than one predictions respectively.