Event-independent temporal positioning: application to French clinical text
Year: 2023

Abstract:
Extracting temporal relations usually entails identifying and classifying the relation between two mentions. However, the definition of temporal mentions strongly depends on the text type and the application domain. Clinical text in particular is complex. It may describe events that occurred at different times, contain redundant information and a variety of domain-specific temporal expressions. In this paper, we propose a novel eventindependent representation of temporal relations that is task-independent and, therefore, domain-independent. We are interested in identifying homogeneous text portions from a temporal standpoint and classifying the relation between each text portion and the document creation time. Temporal relation extraction is cast as a sequence labeling task and evaluated on oncology notes. We further evaluate our temporal representation by the temporal positioning of toxicity events of chemotherapy administrated to colon and lung cancer patients described in French clinical reports. An overall macro F-measure of 0.86 is obtained for temporal relation extraction by a neural token classification model trained on clinical texts written in French. Our results suggest that the toxicity event extraction task can be performed successfully by automatically identifying toxicity events and placing them within the patient timeline (F-measure .62). The proposed system has the potential to assist clinicians in the preparation of tumor board meetings.

text: The 22nd Workshop on Biomedical Natural Language Processing and BioNLP Shared Tasks, pages 191–205 July 13, 2023 ©2023 Association for Computational Linguistics

heading: 1 Introduction
text: Temporal information extraction from text is a critical task in natural language processing (NLP) research, and it has been employed in a wide range of NLP applications, including narrative construction (Do et al., 2012; Ning et al., 2017; Han et al., 2019), temporal question answering (Llorens et al., 2015), and clinical text processing (Tourille et al., 2017c; Moharasan and Ho, 2019; Lin et al.,
2020). Temporal information extraction was first addressed in the news article domain (Bögel et al., 2014; Chambers et al., 2014; Ning et al., 2017; Vashishtha et al., 2019). At the same time, there has been a significant interest in temporal information extraction from clinical narratives through the i2b2-2012 challenge (Sun et al., 2013) and the Clinical TempEval shared tasks (Bethard et al., 2015, 2016, 2017).
Narrative texts embedded in Electronic Health Records (EHR) contain essential temporal information, which can help better understand the clinical healthcare pathway. Temporal information extraction involves detecting events (EVENT), identifying temporal expressions (TIMEX), and extracting temporal relations between them. Several challenges arise when representing clinical temporal information (Najafabadipour et al., 2020). Temporal expressions vary widely, including domainspecific, non-standard, and abbreviated date expressions. Moreover, clinical narrative text is often ungrammatical and goes back and forth through time, making it difficult to link the events to temporal expressions. Sometimes, the time related to the clinical event is not explicitly specified. Redundant information in clinical text is another major problem when determining the chronology of events.
Temporal relation extraction denotes temporal ordering between text mentions, indicating events or temporal expressions. The TimeML (Pustejovsky et al., 2003) annotation scheme was initially developed to model general-domain events, temporal expressions, and their temporal links (TLINKs). THYME-TimeML (Pustejovsky and Stubbs, 2011; Styler IV et al., 2014), a similar annotation scheme adapted to the clinical domain, has been proposed, introducing another form of temporal relations linking clinical events to the Document Creation Time (DCT), namely DocTimeRel. Temporal relation extraction models evolved from rule-based models (Chang et al., 2013; Wang et al., 2016;
191
Najafabadipour et al., 2020) to machine learningbased (Tourille et al., 2016b; Chikka, 2016; Tourille et al., 2017c; Viani et al., 2019b) and deep learningbased models (Tourille et al., 2017a; Liu et al., 2019; Lin et al., 2020; Alfattni et al., 2021; Chokwijitkul et al., 2018). However, the performance of the proposed models is still insufficient for practical applications (Gumiel et al., 2021). Indeed, event modeling is task-specific and heavily dependent on the nature of the text, and annotating temporal relations in clinical texts is more challenging since it requires medical expertise, which is costly and time-consuming. As a result, inter-annotator agreement (IAA) is poor in available datasets (Ning et al., 2018), and annotated corpora are limited for non-English languages. Only a few studies, in particular, use French corpora (Tourille et al., 2016a, 2017c).
In this work, we introduce a novel eventindependent representation of temporal relations in clinical texts that is task-independent and easily adaptable to different domains. Each narrative portion is assigned to a temporal category, describing its relation to the DCT. Unlike the DocTimeRel extraction task, we do not begin by identifying the clinical events and then extracting their relation to the DCT. Instead, we extract the temporal positioning of each text portion according to the DCT, regardless of events. The clinical events could then be identified depending on the task, and each event will have the same temporal positioning as the text portion that includes it. Our main contributions are the following:
• We propose a novel representation of temporal relations that allows us to identify homogeneous text portions from a temporal standpoint and to classify their temporal positioning, regardless of the domain and the extraction task. This leads to a task much faster and easier for human annotators, as well as more reproducible through different event types.
• To evaluate our temporal representation, we annotate a corpus of clinical reports written in French using the THYME-TimeML annotation scheme, and we define the temporal positioning extraction task as a sequence classification task. The classification model is compared to a rule-based baseline model.
• To validate the effectiveness of our temporal representation, we apply our classification
model to another clinical corpus, identify the chemotherapy toxicity events in this corpus, and then infer the temporal positioning of these events according to the DCT.

heading: 2 Related Work
text: Prior works on temporal relation extraction in clinical text are based on hand-crafted rules. To define these rules, (Gaizauskas et al., 2006; Hernández et al., 2016; Viani et al., 2019a; Najafabadipour et al., 2020) used lexical and grammatical features such as part-of-speech (POS) tags, tense and aspect of events. Wang et al. (2016) proposed a model to extract relations between events and time expressions that relies only on the properties of relation entities without the need for extra grammatical information to ensure a minimal dependency on the text quality. Rule-based models need human expertise to create domain-specific rules, and such models are difficult to adapt to other domains.
Later, a variety of supervised Machine Learning approaches has been used, such as Support Vector Machine (SVM) classifiers and Conditional Random Fields (CRFs), with different sets of features, including syntactic, lexical, and semantic features (Lin et al., 2016; Tourille et al., 2016b, 2017c; Viani et al., 2019b; Barros et al., 2016; Chikka, 2016). Tourille et al. (2016a) presented a model based on the Random Forest (RF) algorithm, and Cohan et al. (2016) used a Logistic Regression (LR) classifier to extract the temporal relations between clinical events and the DCT. Velupillai et al. (2015) addresses the DocTimeRel task by using a CRF model and the narrative container relation sub-task by building a hybrid approach based on CRF and a rule-based technique. (Chang et al., 2013) also proposed a hybrid model combining a rule-based method with a maximum entropy model.
Deep neural networks have been used for temporal relation extraction in recent years. Li and Huang (2016) utilized Convolution Neural Networks (CNNs) to identify the relation between events and DCT. Dligach et al. (2017) proposed models based on CNNs and Long Short-Term Memory (LSTM) for extracting the event-event and event-time contains relations from the THYME corpus. Tourille et al. (2017a,b) used LSTMs to create inter-sentence and intra-sentence relation classifiers. Galvan et al. (2018) obtained state-ofart performance on the 2016 Clinical TempEval challenge for temporal relation extraction using a
tree-based LSTM model relying on dependency information. Liu et al. (2019) proposed an attention mechanism to enhance the overall performance of LSTM and GRU neural models for containment relations. Alfattni et al. (2021) investigated the attention mechanism built into a Bi-LSTM model on a broader set of temporal relations in clinical discharge summaries, including intra-sentence, crosssentence, and DocTimeRel temporal relations. Lin et al. (2019, 2020) introduced BERT (Kenton and Toutanova, 2019) based models using the combination of global embeddings and multi-task learning to extract TLINKs and DocTimeRel relations jointly.
Supervised Machine Learning and Deep Learning models require large amounts of annotated data. However, few annotated corpora in the clinical domain are available in languages other than English. As a result, few research efforts addressed French corpora (Tourille et al., 2016a, 2017c).

heading: 3 Material and Methods

heading: 3.1 Overview of the temporal relation representation
text: As illustrated in Figure 1a, temporal relations in the text are often represented by DocTimeRel and TLINKs relations. The extraction of DocTimeRel refers to identifying events and classifying their temporal relations with the Document Creation Time (DCT). According to the THYMETimeML scheme, each event will be assigned to one of the following categories: Before (orange), Before_Overlap (green), Overlap (yellow), and After (blue). However, since the events vary according to the task they are devised for, the DocTimeRel extraction task varies from domain to domain, and no generalization is possible. Additional challenges are also encountered in the definition of clinical events due to the complexity and the variety of medical terminologies prevalent in clinical narratives. The extraction of TLINKs relations starts with extracting possible pairs of events and temporal expressions. The most common strategy is to select the pairs in the same sentence and extract the intra-sentence temporal relations. Nevertheless, the characteristic of clinical text, such as the use of punctuation marks and the omission of sentence start and finish marks, make identifying sentence boundaries challenging. Moreover, other strategies must be adopted to resolve long-distance dependencies if the event and the temporal expression are in
different sentences. Therefore, we introduce a novel eventindependent representation of temporal relations. As described in Figure 1b, homogeneous text portions from a temporal standpoint are identified and assigned to a category of the THYME-TimeML annotation scheme that reflects the relation with the DCT. Events will subsequently have the same temporal category as the text portion that includes them. Thus, we do not have to deal with sentence boundaries or long dependency issues. Although this representation is coarser than the traditional representation of temporal information, it is totally independent of the type of mentions to be extracted and, therefore, of the application domain. Figure 2 illustrates an example of our event-independent temporal positioning representation.

heading: 3.2 Corpora description
text: To develop and evaluate our temporal relation representation and our temporal positioning models, we use the following two clinical corpora1:

heading: 3.2.1 Temporal extraction corpus
text: This corpus is restricted and is built with randomly selected de-identified hospital, operative, and consultation reports of colon cancer patients from a French clinical data warehouse of the Georges Pompidou European Hospital (Jannot et al., 2017). We annotate 220 documents to train and validate our model and 57 documents for evaluation. We use the temporal categories of the THYME-TimeML annotation scheme and two more categories, namely TemporalReference and End_Scope. The TemporalReference category is used to identify the beginning of a clinical report associated with a new Document Creation Time (DCT), which is useful when multiple clinical reports are concatenated in the same document. The End_Scope category marks the end of a text portion if the following portion is a heading or signature. This only allows us to exclude these sections in the preprocessing step. Three annotators annotated a selection of 9 clinical documents. The inter-annotator agreements between annotator pairs in terms of macro F-measure are: 0.62, 0.73, and 0.69, which is higher than the agreement previously observed for temporal relations in clinical corpora in French and English (Tourille et al., 2017c). For our temporal positioning classification task, we will thus have these five categories:
1 The scientific and ethical committee of AP-HP approved access to the clinical data (CSE21-15_TALONCO).
TemporalReference, Before, Before_Overlap, Overlap, and After. The default temporal category for TemporalReference is Overlap. The detailed annotation guideline is provided in Appendix A.

heading: 3.2.2 Toxicity corpus
text: This corpus is restricted and is built with randomly selected de-identified hospital clinical reports containing toxicity information of chemotherapy administrated to colon and lung cancer patients from the same French clinical data warehouse as the temporal extraction corpus (Jannot et al., 2017). An expert manually validated the toxicity events annotations on 43 clinical documents. The annotation process is done using the BRAT annotation tool (Stenetorp et al., 2012). This corpus includes 16 documents regarding colon cancer and 27 about lung cancer and is used to validate the efficacy of our temporal positioning approach.
Table 1 presents descriptive statistics for each category in the temporal extraction training and test
corpora.

heading: 3.3 Temporal relation extraction
text: With this representation, we cast temporal relation extraction as a supervised sequence labeling task. The main goal is to identify homogeneous text portions from a temporal standpoint and to classify each text portion into a pre-defined temporal category, describing its relationship with the Document Creation Time (DCT). We train a token classification model using the French model CamemBERT (Martin et al., 2020) from the HuggingFace transformers library (Wolf et al., 2020). We classify each token based on the BIO (Beginning-InsideOutside) tagging scheme. Hence, the model can identify tokens that indicate a temporal shift in the clinical text. The model weights were optimized with Adam (Kingma and Ba, 2014) without weight decay for 20 epochs. The batch size was set to 32. All the models were trained using a GPU NVIDIA Quadro P5000.

heading: 3.4 Chemotherapy toxicity event extraction
text: For the first pre-annotation and extraction of chemotherapy toxicity events, we use a dictionarybased model consisting of a simple matching between the clinical corpus and a chemotherapy toxicity dictionary (Rogier et al., 2021), containing French toxicity terms from different terminologies. This model is built using the QuickUMLS (Soldaini, 2016) algorithm. The obtained pre-annotations, as previously stated, are manually verified and corrected by a domain expert.

heading: 3.5 Baseline model
text: We compared our model with a defined rule-based baseline model. We map entire sections to a temporal positioning, based on terms that are often used to denote medical sections, in particular in hospital and operative clinical reports such as "Antécédents" (Case history), "Indication" (Indication), "Gestes réalisés" (Operative actions), "Plan de traitement" (Treatment plan), etc. These keywords are typically useful for the temporal annotation process, even though they do not cover all types of clinical reports. This baseline model will be evaluated on the temporal extraction test corpus.

heading: 3.6 Evaluation metrics
text: In our work, we are interested in identifying temporal shifts between large text portions. In this case, segmentation into sentences and tokens is no longer needed. We evaluate the performance of our models at the character level by measuring the macro Precision, Recall, and F-measure.
True positives, false positives, and false negatives are denoted as TP, FP, and FN, respectively.
The three used evaluation metrics per category are defined below:
Precision = TP
TP + FP
Recall = TP
TP + FN
F −measure = 2× (Recall × Precision) Recall + Precision
Furthermore, using the empirical bootstrap method (Dekking et al., 2005, p.275), we compute the 95% confidence intervals of our classification results. For this, we sample our test corpus with replacement 1000 times. Evaluation metrics will be calculated for each sample.
To measure2 the carbon footprint of training and testing our temporal positioning models, we use the Carbon tracker tool (Anthony et al., 2020).

heading: 4 Results
text: Table 2 summarizes the overall results of the baseline model and our temporal positioning model. The best results are obtained with our model with an F-measure of 0.86, which is higher than the inter-annotator agreements. The results are much lower with the baseline model, with an F-measure of 0.35. The CO2 emissions from training and testing our temporal positioning model are estimated to be 199 g.
Table 3 presents the detailed performance of our temporal positioning model over all categories on the temporal extraction test corpus.
Figure 2 shows a clinical text sample with predicted results of temporal positioning of homogeneous text portions.
Table 4 illustrates the toxicity events extraction performance, the results of event-independent temporal positioning of text portions, and the temporal positioning of toxicity events on the toxicity corpus. An F-measure of 0.59 is obtained for extracting toxicity events using the QuickUMLS algorithm with chemotherapy toxicity events. Our model achieves 0.8 of F-measure on extracting and temporal positioning the text narrative portions of the toxicity corpus. Table 4 also provides further performance
2Note that these estimates are approximative and are computed by using the World-wide average carbon intensity of electricity production in 2019.
details based on the type of cancer described in the toxicity corpus documents. Our model yields better results on colon narrative portions than lung narrative portions (an F-measure of 0.81 vs. an Fmeasure of 0.79). For temporal positioning of the toxicity events, inferior results are obtained with an F-measure of 0.62.

heading: 5 Discussion

heading: 5.1 Performance of temporal positioning models
text: As reported in Table 2, our temporal positioning model outperforms the baseline model on the temporal extraction test corpus, with an exact macro F-measure of 0.86 vs. 0.35. Table 3 presents the results per category of our model. The most prevalent categories are the best predicted (see Table 3). Thus, an F-measure of 0.9 is obtained for the Overlap category, representing 35.1% of the training corpus, and 0.86 for the Before category, representing 22.9% of the training corpus. However, high F-measures are also reported for less represented categories such as TemporalReference (10.3% and an F-measure of 0.91), After (12.3% and an F-measure of 0.88). This may be due to the well-specified boundaries of these categories. The text portions with the Before_Overlap category are often sentences included in text portions with the Before category with a temporal indication that shows consistency in time, such as "depuis le" (since the) (see Figure 2). This temporal shift is not always predicted, and despite the coverage of the Before_Overlap category (19.4% in the training corpus) training corpus) the performance is
lower (0.77 of F-measure). Except for the second ’Follow-up’ text span in Figure 2, most homogeneous text portions are adequately retrieved and classified. In particular, the temporal shift between the Before and the Before_Overlap categories is well predicted. The text portion "patient de 56 ans dans le contexte de" has been correctly assigned to the Before_Overlap category. The two text portions beginning with "Suivi et évolution dans le service:" and "Suivi:" respectively, are on follow-up care. The first one depicts the follow-up during the hospital stay and is well classified into the Overlap category. However, the second text portion starting with ’Suivi:’ is wrongly assigned to the Overlap category when, in fact, it should be assigned to the After category since we are discussing future follow-up after discharge, including future treatments and medications. Other mistakes may occur when predicting temporal categories. For instance, text portions starting with ’Soins post-opératoires’ (Post-operative care) and ’Soins de support’ (Support care) are about patient care. The first span, usually described in operative reports, discusses post-operative care and should be assigned to the After category. In contrast, the second statement, usually in discharge summaries, examines whether or not there is supportive care and should be classified as Overlap.
As previously stated, an F-measure of 0.35 is observed for the baseline model. Note that we do not use the End_Scope category to avoid the heading and signature sections in this baseline model since there is no defined term to identify such sections. Therefore, the precision of this model remains low. The TemporalReference category has poor precision because it specifies the start of a clinical report and is usually in the heading section. Moreover, we use the terms "Paris" and "Compte-rendu" (report). The first keyword usually indicates the start of consultation reports, as healthcare professionals begin by writing the location and date of the report. However, such terms may occur in various parts of the clinical text. The second keyword denotes the start of hospital and operative reports, which begin with a title such as "Compte rendu opératoire" (Op-
erative report) or "Compte rendu d’hospitalisation" (Hospitalization report). Similar observations are obtained for the After category, which tends to be at the end of the clinical report and just before the signature part. The keywords used in the rulebased model do not cover the consultation reports, which contain narrative text describing the patient visit summary. As a result, the baseline model also suffers from a low recall rate.
We also test the performance of our model on the toxicity corpus as shown in Table 4. An overall Fmeasure of 0.8 is obtained, which is slightly lower than the performance on the temporal extraction corpus (an F-measure of 0.86). This might be due to differences in the cancer types described in the texts in each corpus. Indeed, our temporal positioning model was trained on the temporal extraction corpus, which only includes clinical reports of colon cancer patients, but the toxicity corpus contains clinical reports of both colon and lung cancer patients. As a result, the performance of temporal positioning clinical reports of colon cancer patients in the toxicity corpus is better than that of lung cancer patients in the same corpus as reported in Table 4 (an F-measure of 0.81 vs. an F-measure of 0.79). This good performance show that our model can adapt to other corpora, including other types of cancer.

heading: 5.2 Environmental impact
text: The carbon footprint of our event-independent temporal positioning model is reported in Table 2 in terms of CO2 equivalent measure in grams. A total of 199 g of CO2 emissions is estimated from training and testing our model, which is roughly equivalent to 1.85 km traveled by car based on CO2 performance of new passenger cars in Europe3. Note that Carbon tracker fails to fetch the IP address and, therefore, to determine the geo-
3https://www.eea.europa.eu/ims/co2-performance-ofnew-passenger
graphic location dynamically. As a result, it uses the World-wide average carbon intensity of electricity production in 2019 (475 gCO2/kWh) instead of the used value for France (around 58 gCO2/kWh in 2021), which yields to overestimated CO2 equivalent measures. Moreover, Carbon tracker does not take into consideration the execution environment or the technique of energy production. Thus, the obtained carbon footprint measures remain very approximative.

heading: 5.3 Performance of toxicity events extraction
text: As reported in Table 4, an F-measure of 0.59 is obtained for the toxicity event extraction using the quickUMLS algorithm. The toxicity events extraction model extracts all toxicity events in clinical text. However, we are solely interested in toxicity events related to chemotherapy treatments. As a result, the precision of this model remains low. For instance, if "HTA" (hypertension, high blood pressure) is included in the comorbidity medical section, we do not consider it as a toxicity event. However, if such event is mentioned while describing the toxicities of previous chemotherapy cures, it will be retained as a toxicity event.
It is also worth noting that we extract even the negated toxicity events. In fact, "anémie de grade 0" (anemia of grade 0) and "pas d’anémie" (no anemia) are synonyms for the absence of such toxicity event. However, extracting the toxicity event anemia is still important to propose better treatment strategies.

heading: 5.4 Temporal positioning of chemotherapy toxicity events
text: This experiment aims to determine how effectively we can recognize and characterize the temporal relation between toxicity events and the DCT. To address this question independently of how well event recognition can be achieved, we have used the gold standard toxicity event annotations, which
"Discharge summary. Admission date: 07/10/2008 Discharge date: 07/17/2008. Reason for admission: 56 y.o female presented with asthenia, weight loss and lack of appetite following the recent discovery of sigmoid adenocarcinoma. Past medical history: appendectomy Hypertension treatment was initiated in 2008. Physical examination on admission: Weight: 65 kg, Size 160 OMS 3 Abdomen was soft. Hospital course: Further medical exams: Tests on admission: ... Discharge instructions/Follow-up: - analgesics - patient should continue her usual care."
are, therefore, ’perfectly’ recognized. As reported in Table 4, an F-measure of 0.62 is obtained. Looking at the outcomes by category, the majority of toxicity events are temporarily well-positioned into the three categories Before, Before_Overlap, and Overlap. Nevertheless, in our toxicity corpus, just one toxicity event matches the After category. This event, mentioned in a hypothesis statement, is incorrectly positioned as a Overlap category. As a result, the performance in terms of macro F-measure is a bit low (vs. a micro F-measure of 0.82). The good performance of temporal positioning of chemotherapy toxicity events validates the efficacy of our event-independent temporal representation of temporal information.

heading: 6 Conclusion
text: In this paper, we introduced a novel eventindependent representation of temporal relations that is task-independent and, hence, domainindependent. The temporal relation classification problem is cast as a sequence token classification task using our representation. The main goal of this classification task is to identify homogeneous text portions and to classify them into temporal categories reflecting their relations with the document creation time. To develop and evaluate our model, we annotate a corpus of clinical reports written in French using the THYME-TimeML annotation scheme. Our temporal positioning model yields good results when recognizing and categorizing text portions. Moreover, experiments on the temporal positioning of chemotherapy toxicity events for patients with colon and lung cancers have also shown that good results could be achieved using our representation of temporal relations. This problem modeling might be the initial step toward constructing a patient timeline to order all its medical events.

heading: Limitations
text: In our work, we manually annotated small portions of corpora. Such limited size is justified by the time-consuming task of temporal annotations and the requirement of expertise for toxicity event annotations. Although our temporal representation seems to perform well with other clinical reports containing information about a different type of cancer from that on which it was trained (e.g. lung cancer vs. colon cancer), such results must be validated on clinical reports containing information about additional cancer types. Additional experiments are also needed to validate the generalizability of our event-independent representation, such as evaluating it on other hospital or data warehouse clinical reports with various structures and evaluating it on other extraction tasks with different event definitions.

heading: Ethics Statement
text: This study uses de-identified clinical data with the approval of the partner French hospital scientific and ethical committee (IRB equivalent). This work might be recommended to clinicians as a useful tool for assisting in the systematic analysis of large patient records. Unfortunately, the annotated cor-
pora developed in this work cannot be shared with the community due to confidentiality restrictions.

heading: Acknowledgements
text: We would like to thank the scientific and ethical council of the AP-HP health data warehouse and the Georges Pompidou European Hospital, who gave us access to the corpora used in this work. Nesrine Bannour received funding from the ITMO Cancer Aviesan. Bastien Rance is supported by the SIRIC CARPEM program. The authors also acknowledge the support of ANR under grant CODEINE ANR-20-CE23-0026-01.

heading: A Temporal Annotation scheme for our clinical corpus

heading: A.1 Definitions of temporal categories
text: To annotate the temporal information in a clinical report, we define a temporal annotation scheme based on the Document Creation Time (DCT) and the possible categories of the Document creation Time Relation (DocTimeRel). The DCT might be the current medical visit date, usually stated in the document heading. It might also be the length of time spent in the hospital. The DCT does not need to be annotated.

heading: A.1.1 Document creation Time Relation
text: Document creation Time Relation is the relation between events and Document Creation Time. We consider these four possible categories for this time relation: Before, Before_Overlap, Overlap, and After. We annotate only the first word of each temporal portion. We consider that the start of a temporal portion denotes the end of the previous one.

heading: A.1.2 Before
text: The Before category is used to annotate narrative portions referring to what occurred before the Document Creation Time.
Examples
• Antécédents, antécédents médicaux, antécédents chirurgicaux, Antécédents familiaux, Histoire de la maladie, Rappel clinique, Rappel sur la pathologie → All terms referring to the medical history section.
• Except: Maladie traitée depuis le → Before_Overlap since we have a temporal indication that the procedure/disease is still ongoing for the patient (cf. Figure 3).

heading: A.1.3 Before_Overlap
text: The Before_Overlap category is used to annotate narrative portions that started before the document creation time and are still ongoing at that time.
Examples
• Comorbidités, Mode de vie, Autonomie, traitement habituel, traitement à l’entrée, Allergies, Traitements concomitants, Facteurs de risque, Indication, Indication opératoire,
décision d’une intervention, Tolérance intercure
• Patient de 70 ans
• HTA traitée depuis, dans le cadre d’un suivi d’un cancer → The patient is still suffering from the disease.
• METASTASES HEPATIQUES D’UN ADENOCARCINOME → The disease’s name as a title in operative reports, which is generally capitalized (cf. Figure 4).

heading: A.1.4 Overlap
text: The Overlap category is used to annotate narrative portions that happen at the same time as the document creation time.
Examples
• Examen pratique, Au total, Conclusion, Gestes opératoires, Gestes réalisés, Motif d’hospitalisation, Biologie, Biologie de sortie, INTERVENTION, constantes à l’arrivée, Date d’hospitalisation, Date d’entrée, Date de l’intervention, Motif
• Examens complémentaires, Examens paracliniques → Sometimes, some complementary exams are conducted before the document creation time but because they are done for the purpose of the hospital stay, we annotate them as Overlap (cf. Figure 3).
• Je vois ce jour, Je revois en consultation

heading: A.1.5 After
text: The After category is used to annotate narrative portions referring to what occurs after the document creation time.
Examples
• Traitement de sortie, Prochains rendez-vous, Rendez-vous à venir, Prescription de médicaments, Date de la prochaine cure, Ordonnance de sortie, Prochains examens
• Je reverrai ce patient, je prévois une coloscopie
• La pièce est envoyée pour un examen histologique

heading: A.2 Other categories

heading: A.2.1 TemporalReference
text: Because several medical reports might be written in the same document, the TemporalReference category specifies the beginning of a new clinical report. Because several medical reports might be written in the same document, the TemporalReference category specifies the beginning of a new clinical report. Each clinical report will then have its own Document Creation Time, and the annotations will be based on this DCT. The TemporalReference category’s default Document Time Relation is assumed to be Overlap and does not need to be annotated.
Examples
• Compte-rendu opératoire, Compte-rendu d’hospitalisation, Paris, le 14 octobre 2018

heading: A.2.2 End_Scope
text: We do not consider heading and signature information in our annotation. Therefore, we use the category End_Scope to mark the ending of a narrative portion if the next narrative portion is a heading or a signature. This way, we avoid annotating the contact information for the health care unit, which may be repeated in several clinical reports. Despite the fact that the clinical documents are de-identified, we avoid annotating specific patient information. In cases other than headings or signatures, the end of a temporal portion is implicitly considered the start of a new temporal portion.
A.3 Examples of annotations made in accordance with the above scheme and guidelines
Annotations of the first example (cf. Figure 3)
• From Compte to d’hospitalisation as TemporalReference
• From Hospitalisé to 30/07/2013 as Overlap
• From Motif to d’HOSPITALISATION : as Overlap, note that we don’t annotate the temporal portion after the End_Scope containing contact information of doctors
• From HISTOIRE to ANTECEDENTS as Before
• From HTA to 2012, as Before_Overlap since we have a temporal indication that the disease is still ongoing for the patient
• From phlébite to 07/2012 as Before since it’s part of the medical patient history
• From ALLERGIES to Autonome as Before_Overlap
• From Examens to et as Overlap despite the fact that the medical exams are conducted before the date of hospital admission
• From sera to 10/09/2013 as After. The signature of the document after the End_Scope category is not annotated
Annotations of the second example (cf. Figure 4)
• From COMPTE to OPERATOIRE as Temporal Reference
• ADENOCARCINOME as Before_Overlap
• COLECTOMIE as Overlap
• From Rappel to clinique: as Before
• From Indication to opératoire. as Before_Overlap
• From Gestes to réalisés: as Overlap

heading: Future Work
text: However, the second text portion starting with ’Suivi:’ is wrongly assigned to the Overlap category when, in fact, it should be assigned to the After category since we are discussing future follow-up after discharge, including future treatments and medications. Other mistakes may occur when predicting temporal categories. For instance, text portions starting with ’Soins post-opératoires’ (Post-operative care) and ’Soins de support’ (Support care) are about patient care. The first span, usually described in operative reports, discusses post-operative care and should be assigned to the After category. In contrast, the second statement, usually in discharge summaries, examines whether or not there is supportive care and should be classified as Overlap. As previously stated, an F-measure of 0. 35 is observed for the baseline model. Note that we do not use the End_Scope category to avoid the heading and signature sections in this baseline model since there is no defined term to identify such sections. Therefore, the precision of this model remains low. The TemporalReference category has poor precision because it specifies the start of a clinical report and is usually in the heading section. Moreover, we use the terms "Paris" and "Compte-rendu" (report). The first keyword usually indicates the start of consultation reports, as healthcare professionals begin by writing the location and date of the report. However, such terms may occur in various parts of the clinical text. The second keyword denotes the start of hospital and operative reports, which begin with a title such as "Compte rendu opératoire" (Op-
erative report) or "Compte rendu d’hospitalisation" (Hospitalization report). Similar observations are obtained for the After category, which tends to be at the end of the clinical report and just before the signature part. The keywords used in the rulebased model do not cover the consultation reports, which contain narrative text describing the patient visit summary. As a result, the baseline model also suffers from a low recall rate. We also test the performance of our model on the toxicity corpus as shown in Table 4. An overall Fmeasure of 0. 8 is obtained, which is slightly lower than the performance on the temporal extraction corpus (an F-measure of 0. 86). This might be due to differences in the cancer types described in the texts in each corpus. Indeed, our temporal positioning model was trained on the temporal extraction corpus, which only includes clinical reports of colon cancer patients, but the toxicity corpus contains clinical reports of both colon and lung cancer patients. As a result, the performance of temporal positioning clinical reports of colon cancer patients in the toxicity corpus is better than that of lung cancer patients in the same corpus as reported in Table 4 (an F-measure of 0. 81 vs. an F-measure of 0. 79). This good performance show that our model can adapt to other corpora, including other types of cancer