# Efficient extraction of medication information from clinical notes: an evaluation in 2 languages

Thibault Fabacher, Erik-Andre Sauleau, Emmanuelle Arcay, Bineta Faye, Maxime Alter, Archia Chahard, Nathan Miraillet, Adrien Coulet, Aurélie

Névéol

► To cite this version:

Thibault Fabacher, Erik-Andre Sauleau, Emmanuelle Arcay, Bineta Faye, Maxime Alter, et al. Efficient extraction of medication information from clinical notes: an evaluation in 2 languages. Journal of the American Medical Informatics Association, 2025, pp.ocaf113. 10.1093/jamia/ocaf113. hal-05375038

HAL Id: hal-05375038

https://inria.hal.science/hal-05375038v1

Submitted on 20 Nov 2025

HAL is a multi-disciplinary open access
archive for the deposit and dissemination of sci-
entific research documents, whether they are pub-
lished or not. The documents may come from
teaching and research institutions in France or
abroad, or from public or private research centers.

L'archive ouverte pluridisciplinaire HAL, est
destinée au dépôt et à la diffusion de documents
scientifiques de niveau recherche, publiés ou non,
émanant des établissements d'enseignement et de
recherche français ou étrangers, des laboratoires
publics ou privés.

<figure><img src="image_1.png" /></figure>

Distributed under a Creative Commons Attribution 4.0 International License
EFFICIENT EXTRACTION OF MEDICATION INFORMATION FROM
CLINICAL NOTES: AN EVALUATION IN TWO LANGUAGES

Thibaut FABACHER, MD

University hospital of Strasbourg
ICube Laboratory, Strasbourg, France
Inria, Inserm, Université Paris Cité, U1346 HeKA, Paris, France

Erik-André SAULEAU, PhD

University hospital of Strasbourg
ICube Laboratory, Strasbourg, France

Emmanuelle ARCAY

University hospital of Strasbourg

Bineta FAYE

University hospital of Strasbourg

Maxime ALTER

University hospital of Strasbourg

Archia CHAHARD

University hospital of Strasbourg

Nathan MIRAILLET

University hospital of Strasbourg

Adrien COULET, PhD

Inria, Inserm, Université Paris Cité, U1346 HeKA, Paris, France

Aurélie NÉVÉOL, PhD

Université Paris-Saclay, CNRS, LISN, Orsay, France

February 6, 2025

ABSTRACT

**Objective:** To evaluate the accuracy, computational cost and portability of a new Natural Language Processing (NLP) method for extracting medication information from clinical narratives. **Materials and Methods:** We propose an original transformer-based architecture for the extraction of entities and their relations pertaining to patients’ medication regimen. First, we used this approach to train and evaluate a model on French clinical notes, using a newly annotated corpus from Hôpitaux Universitaires de Strasbourg. Second, the portability of the approach was as- sessed by conducting an evaluation on clinical documents in English from the 2018 n2c2 shared task. Information extraction accuracy and computational cost were assessed by comparison with an available method using transformers. **Results:** The proposed architecture achieves on the task of relation extraction itself performance that are competitive with the state-of-the-art on both French and English (F-measures 0.82 and 0.96 vs 0.81 and 0.95), but reduce the computational cost by 10. End-to-end (Named Entity recognition and Relation Extraction) F1 performance is 0.69 and 0.82 for French and English corpus. **Discussion:** While an existing system developed for English notes was deployed in a French hospital setting with reasonable effort, we found that an alternative architecture offered end-to-end drug information extraction with comparable extraction per- formance and lower computational impact for both French and English clinical text processing, respectively. **Conclusion:** The proposed architecture can be used to extract medication information from clinical text with high performance and low computational cost and consequently suits with usually limited hospital IT resources

Introduction

Medication information is crucial to provide patients with medical care tailored to their condition. Understanding
detailed therapeutic regimens is highly valuable, especially for patients with chronic diseases who may receive care
across multiple facilities, including hospitals. Accurate representation of these regimens supports improved clinical research and facilitates better management of complex, long-term treatments.

Electronic health records (EHRs) are a major source of clinical data, providing comprehensive information for patients treated at a hospital. EHRs include both structured and unstructured data sources. While structured data is often the focus of clinical research, unstructured data represents an untapped resource that can yield significant clinical insights [1,2]. Among the various aspects of unstructured clinical data, medication-related information is particularly valuable.

Medication information extraction from EHRs involves identifying specific drugs, with their attributes such as dosages, frequencies, routes, duration and their contextualization within a patient's clinical timeline. Elements of contextualization can be dosage adjustments, therapy modifications, negation, proposition of treatment, etc. The international biomedical Natural Language Processing (NLP) community has been addressing this issue over the past decade. Despite notable success for English, this task remains challenging due to the variability of language in clinical texts and the complexity to account for heterogenous contextual factors [3,4]. Existing approaches typically focus on either drugs and their attributes or the contextual factors associated with them. Few methods capture the full scope of prescriptions, which involve specific combinations of drugs and their attributes tied to specific points in a patient's timeline. This gap leads to imprecise or incomplete therapeutic schemes, limiting their utility in real-world clinical applications.

Recent advances in NLP, particularly transformer-based architectures, have significantly enhanced performance in tasks such as Named Entity Recognition (NER) and Relation Extraction (RE) [5,6]. Models like Bidirectional Encoder Representations from Transformers (BERT) [7] and Generative Pre-trained Transformers (GPT) [8], achieve this by leveraging contextual word embeddings.

One major issue is computational complexity. These models can be extremely large, requiring substantial memory and processing power for either training or fine-tuning on domain-specific datasets. Additionally, models like BERT and GPT are often used as input of classifiers for relationships between entity pairs, where each pair is processed individually. This leads to high computational costs. Moreover, models like GPT, are particularly resource-intensive, making them impractical for low-resource environments [9].

Another challenge is data privacy. Clinical data is highly sensitive and subject to strict regulations, which often prohibit the use of external APIs or cloud-based systems. This necessitates the local deployment of resource-efficient models for RE.

Adapting models to real-world clinical datasets and settings presents additional challenges. Clinical notes are often noisy, inconsistently formatted, and highly variable in linguistic structure, which complicates model generalization [10]. Furthermore, studies have shown that while models trained on synthetic or curated datasets perform well in controlled environments, their performance often degrades when applied to real-world corpora [11]. The disparity between training data and real-world text underscores the need for robust methods that can handle the variability and complexity of actual clinical data within a constrained computational environment.

## Objectives

The primary objective of this study is to propose a method for extracting drug-related information from French patient medical records. This method is designed to be computationally efficient, as it will be deployed in an environment with low computational resources. A secondary objective is to validate the proposed method on an existing English dataset from the n2c2 challenge, to assess portability across languages and provide a comparison with state-of-the-art approaches.

## Contributions

This study makes two key contributions. First, we introduce an innovative architecture for RE that significantly reduces computational cost, while achieving state-of-the-art performance. This approach addresses the limitations of current models, particularly in low-resource and high-privacy clinical environments. It was initially designed for French clinical documents but also achieves state-of-the-art performance on English clinical text. Transitioning between languages, requires only a change in the transformer model, highlighting the adaptability of our approach. Second, we redefine how relationships between drugs and associated entities should be annotated in free text, to offer a more precise and comprehensive representation of therapeutic regimens.

## Research Questions

Our study addresses the following questions:
<figure><img src="image_2.png" /><figcaption>Figure 1: Comparative illustration of two concurrent types of annotations for medication relations : drug-centered vs. frame relations. In the case of drug-centered relations (bottom annotations), each relation starts from the drug name. In the case of the frames, each frame captures a specific drug, with specific attributes such as strength, dose, route of administration, and prescription duration. Relationships are visualized as links between attributes within a frame, providing a structured view of drug-attribute relations. In this example, there are two frames corresponding to two distinct pairs of prescription frequency/period</figcaption></figure>

* Can we reduce the computational burden of the RE task without compromising accuracy?

* How effectively can we capture nuanced relationships between drugs and their attributes, particularly in scenarios with complex temporal dependencies ?

* Can we generalize to languages other than English, particularly French clinical texts, while retaining high performance?

# Materials and Methods

## Representation of drugs and their attributes

Current representations often treat drug names and drug attributes as independent entities, which fails to support the accurate description of therapeutic regimens over time. For example, consider the following prescription from the French corpus, translated into English for simplicity:

treatment with tocilizumab IV every
4 weeks from July to October, then every
2 weeks until December

The drug (tocilizumab) is prescribed with a different frequency (every 2 weeks, every 4 weeks) in two periods (July to October, October to December), so that the same drug will need to be linked two distinct pairs of prescription frequency/period. Drug representations need to account for therapy adjustments over time. Existing methods tend to link every attributes to the associated drug, making it difficult to capture such nuances, particularly common in chronic disease management.

To improve the representation of relationships between drugs and their attributes, we propose to use the concept of frames [12]. A frame corresponds to a specific drug with associated properties, including strength, dose, route of administration, and prescription duration. This richer representation is shown in Figure 1. This framing mechanism was integral to our annotation process and provided a structured approach for capturing complex drug-attribute interactions.
<figure><img src="image_3.png" /><figcaption>Figure 2: Representation of Treatment Regimen: This figure illustrates the representation of a treatment regimen, defined as a specific drug and its associated attributes. Some attributes, such as Dosage and Frequency, have non-specific relation types to the drug, while others, like dates, can trigger more specific relation types (starting or stopping a medication). Contextual information enables the representation of these specific relation types without requiring a distinct entity type. A frame is triggered by a drug and includes a specific set of attributes. Among these attributes, only the drug itself is mandatory to create a frame.</figcaption></figure>

## Clinical corpora

To evaluate the proposed model, we used both an English and a French corpora. The English corpus, n2c2 [13], was employed to benchmark our approach against state-of-the-art methods, while the French corpus was curated to test the model's performance on domain-specific clinical text and test our proposed data representation scheme.

## French Corpus: Corp-HUS

The French clinical corpus, named Corp-HUS, was collected from Strasbourg University Hospital. This study is listed on hospital study register and follows the hospital clinical study protocol. It consists of 715 documents extracted from EHRs of patients diagnosed with probable rheumatoid arthritis (RA), including discharge summaries, paramedical and admission notes. These documents were annotated following a detailed annotation guide as part of a larger project aimed at extracting therapeutic schemas for RA patients<fnref n="1" />. Those annotations include the following entities: Drug, Class of Drug, Date, Relative Date, Dosage, Frequency, Route, Duration, and Context; and relations: Refer_to, Start, Stop, Augmentation, Diminution, Contraindication, Negation, Hypothetical, Experiencer, Administration Duration, Prescription Duration, Coreference, and Discontinue Entity. The Refer_to relation establishes a generic link between a drug name and its attributes like Dosage, Frequency, Route, or Date (Figure 2).

The corpus is divided into two subsets: a train set of 615 documents and a test set of 100 documents. It contains 7,581 entities and 4,293 relations in the train set, and 1,726 entities and 904 relations in the test set (Table 1). Notably, one relation type, Refer_to, constitutes the majority of relations, accounting for over half of the annotated links. In training set, 2,427 entities are identified as drugs, of which 1,851 frames can be constructed, with 3.9% classified as multi-frame drugs (drugs involved in several frames, as in Figure 1).

Due to data privacy policies, the French corpus cannot currently be shared. However, we are committed to anonymizing our corpus to enable data sharing with other academics.

<footnote n="1">¹https://github.com/DrFabach/guide annotation medicaments</footnote>
<table><caption>Table 1: Distribution of relations in the two corpora, n(%). †Refer_to relation in Corp-HUS. ‡Duration-Drug relation in n2c2 corpus</caption><thead><tr><th></th><th colspan="2">Corp-HUS</th><th colspan="2">n2c2</th></tr><tr><th></th><th>Train (%)</th><th>Test (%)</th><th>Train (%)</th><th>Test (%)</th></tr></thead><tbody><tr><td>Total doc</td><td>615</td><td>100</td><td>303</td><td>202</td></tr><tr><td>Total entities</td><td>7,581</td><td>1,726</td><td>50,951</td><td>36,384</td></tr><tr><td>Total relations</td><td>4,293</td><td>904</td><td>32,918</td><td>23,462</td></tr><tr><td>Strength→Drug</td><td></td><td></td><td>6,702 (18.42)</td><td>4,244 (18.09)</td></tr><tr><td>Form→Drug</td><td>2515 (58.58)</td><td>488 (53.98)</td><td>6,654 (18.29)</td><td>4,374 (18.64)</td></tr><tr><td>Dosage→Drug</td><td>†</td><td>†</td><td>4,225 (11.61)</td><td>2,695 (11.49)</td></tr><tr><td>Frequency→Drug</td><td></td><td></td><td>6,310 (17.34)</td><td>4,034 (17.19)</td></tr><tr><td>Route→Drug</td><td></td><td></td><td>5,538 (15.22)</td><td>3,546 (15.11)</td></tr><tr><td>Start</td><td>531 (12.37)</td><td>121 (13.38)</td><td>-</td><td>-</td></tr><tr><td>Stop</td><td>307 (7.15)</td><td>70 (7.74)</td><td>-</td><td>-</td></tr><tr><td>Ongoing</td><td>420 (9.78)</td><td>123 (13.61)</td><td>-</td><td>-</td></tr><tr><td>Duration prescription</td><td>51 (1.9)</td><td>21 (2.32)</td><td>643 (1.77)</td><td>426 (1.82)</td></tr><tr><td>Administration time</td><td>12 (0.28)</td><td>-</td><td>‡</td><td>‡</td></tr><tr><td>Increase</td><td>44 (1.02)</td><td>10 (1.11)</td><td>-</td><td>-</td></tr><tr><td>Decrease</td><td>33 (0.77)</td><td>10 (1.11)</td><td>-</td><td>-</td></tr><tr><td>Negation</td><td>54 (1.26)</td><td>12 (1.33)</td><td>-</td><td>-</td></tr><tr><td>Contraindicated</td><td>46 (1.07)</td><td>5 (0.55)</td><td>-</td><td>-</td></tr><tr><td>Hypothetical</td><td>48 (1.12)</td><td>9 (1)</td><td>-</td><td>-</td></tr><tr><td>Experiencer</td><td>1 (0.02)</td><td>-</td><td>-</td><td>-</td></tr><tr><td>Coref</td><td>152 (3.54)</td><td>34 (3.76)</td><td>-</td><td>-</td></tr><tr><td>Discontinue</td><td>34 (0.79)</td><td>1 (0.11)</td><td>-</td><td>-</td></tr><tr><td>Reason→Drug</td><td>-</td><td>-</td><td>5,169 (14.21)</td><td>3,410 (14.53)</td></tr><tr><td>ADE→Drug</td><td>-</td><td>-</td><td>1,107 (3.04)</td><td>733 (3.12)</td></tr></tbody></table>

In our dataset, inter-annotator agreement was measured to evaluate the consistency of manual annotations using F1 score for relations. The mean micro-average F1-score between human annotators was 0.58 (sd: 0.28). When comparing annotations between human annotators and the gold standard, the F1-score increased to 0.68 (sd: 0.1). For automated pre-annotation methods, rule-based dictionary approaches yielded an F1-score of 0.37, while the use of a preliminary trained algorithm improved the F1-score to 0.52.

## English Corpus: the 2018 n2c2

The corpus used in the 2018 n2c2 Challenge on Adverse Drug Events and Medication Extraction [13] comprises tasks for NER and RE. The concept extraction task involves identifying the following entities: Drug, Strength, Form, Dosage, Frequency, Route, Duration, Reason, and ADE (Adverse Drug Event). The relation classification task defines links between entities, including Strength-Drug, Form-Drug, Dosage-Drug, Frequency-Drug, Route-Drug, Duration-Drug, Reason-Drug, and ADE-Drug.

The corpus is divided into two subsets: a train and a test set with 303 and 202 discharge notes, respectively. The n2c2 corpus [13] includes 50,951 entities and 36,384 relations in the train, and 32,918 entities and 23,462 relations in the test set (Table 1).
# Relation extraction with NLP

## Task description

The task of extracting drugs and their attributes from unstructured clinical text involves two key components: NER and RE [6]. Although some studies combine these two steps into a single one [14, 15], separation can optimize performance [16], particularly when using distinct token representations for each task, which can improve overall results. Given the maturity of NER solutions, the focus of this work is on improving the RE component.

## Comparison to a transformer-based relation classification method

For RE, a common strategy involves tagging the two entities within a sentence and using a transformer augmented with an additional classification layer to determine the existence and type of relationship [16–18]. However, these methods face a significant challenge in terms of computation time. Each pair of entities must be presented to the model individually, drastically increasing training time. We compared our proposed architecture with usual models, such as the transformer-based approaches described in [16, 19–21]. Those classify relationships by marking entities within sentences and performing classification on the transformer output representations. A notable comparison system selected, detailed in [21], duplicates sentences containing entities for classification purposes. This method was selected because it achieves good state-of-the-art results on n2c2 corpus, and implementation code was publicly available as a Python package <fnref n="2" />.

## Proposed Architecture

Our model depicted Figure 3 is designed to classify all relationships simultaneously, similar to Zhang et al. [22]. It employs a standard transformer architecture to tokenize input sentences and generate contextual embeddings for each token, embedding size depending of the pretrained transformer model used. Additionally, we add a dedicated embedding layer for token labels.

These embeddings are concatenated to form a unified token representation, which is in turn passed through a multi-head self-attention layer, leveraging the attention mechanism to capture dependencies. To extract relationships, token pairs are represented by concatenating their embeddings and incorporating relative position embeddings to encode their positional relationship. These token-pair representations are then processed through one dense layer followed by a classification layer.

During training, we optimize computation by applying a mask to the loss function, restricting its application to labeled token pairs, i.e. token associated with an entity label. The final classification layer is a multiclass classifier, trained using the Adam optimizer with a linear scheduler incorporating a warmup phase. The loss function employed is cross-entropy.

We leveraged pretrained transformer models fine-tuned on clinical datasets to produce domain-specific word representations. For the English dataset, we experimented with ClinicalBERT [23] and BioBERT [24], while for the French corpus we used CamBERT-BIO [25]. Default tokenizers were used for each transformer model.

## Preprocessing

The French corpus is directly extracted from EHRs where newlines are frequently used to indicate the end of sentences. To address this, we incorporated the token 
 into the tokenizer to handle newline characters effectively and preserve the structure of the text.

Unlike prior transformer-based models, our approach eliminates the need for extensive preprocessing. The only required inputs are the text and entity labels for RE tasks. To optimize computational efficiency, we constrained the input text length using a sliding window of *n* characters. Segments containing fewer than two labeled entities were excluded from train and test datasets. Beyond this, no additional preprocessing was required.

## Hyperparameter selection

We selected the best hyperparameters through a grid search cross-validation on the train set. The optimized parameters included the learning rate, the dimensionality of label embeddings, the number of heads in the multi-head attention layer, the dimensionality of positional embeddings, and the number of nodes in the hidden layer of the final classifier.

<footnote n="2">²https://github.com/uf-hobi-informatics-lab/ClinicalTransformerRelationExtraction</footnote>
<figure><img src="image_4.png" /><figcaption>Figure 3: Overview of Our Model Architecture: The model uses a Transformer-based architecture to classify all relationships simultaneously. Input sentences are tokenized, and contextual embeddings are generated via a Transformer layer, while a dedicated embedding layer is simultaneously applied to token labels. These two representations are concatenated to form a unified token representation, which is passed through a multi-head self-attention layer to capture token dependencies. Subsequently, pairwise combinations of tokens are combined with their relative position embeddings and processed through a deep neural network consisting of a dense layer followed by a task-specific classification layer. The classification layer's output nodes correspond to the number of relation types being predicted. The model is trained using a masked cross-entropy loss applied exclusively to label token pairs, ensuring efficient optimization for the complexity of relational tasks. The matrix represents the two different approaches to representing a drug and its attributes.</figcaption></figure>
## End-to-end

For end-to-end drug and attribute RE, we use the NLStruct library [26] for NER and apply our proposed method for RE sequentially.

## Evaluation Metrics

Performance comparisons between algorithms are conducted using precision, recall and F1 scores as evaluation metrics for each category. Among these, the micro-average F1 score serves as the principal value for assessing and comparing architectures, as it emphasizes performance across all instances equally. For the n2c2 dataset, we used the evaluation script provided by the 2018 n2c2 Challenge and compared performance using the micro-average F1-score, precision, and recall. For French corpus, we adapt the n2c2 evaluation script to our task.

To assess computational efficiency, we tracked the training times using Python's datetime library and TensorBoard [27].

## Sensitivity analysis

To compare performance between the Corp-HUS and n2c2 corpora, we randomly reduced the n2c2 training set to match our Corp-HUS size by selecting five separate subsets of 25 documents. This ensured a comparable number of entities between the n2c2 subsets and the French corpus.

## Work environment and implementation

Our architecture will be made available in the medkit library [28]. The experiments were conducted on a Windows system equipped with an NVIDIA RTX 3070 GPU, an Intel(R) Xeon(R) E3-1245 v5 CPU, 64 GB of RAM, and Python 3.9. The main Python libraries used in this study were medkit [28], PyTorch [29], Transformers [30], Pandas [31], NumPy [32] and SeqEval [33]. Ecological impact was assessed using the Green Algorithms [34].

# Results

## Performance

The grid search process yielded consistent optimal hyperparameters for both the French and English corpora. The best-performing configuration included a batch size of 10, 4 attention heads for the additional multi-head attention layer, an embedding size of 32 nodes for entity labels, 75 nodes for positional embeddings, 256 nodes for the hidden layer, and a learning rate of 0.0001.

### Performance on the English Corpus

On the n2c2 corpus, our proposed model achieved an F1 score of 0.955 (table: 2) which is comparable to the performance of Yang et al. 2021 [21] algorithm (0.949). Notably, our method demonstrated a significant advantage in computational efficiency, being approximately 23 times faster during training. The choice of Transformer models (e.g., ClinicalBERT or BioBERT) did not influence the F1 score, suggesting that the model performance is robust to the selection of pretrained Transformer variants within this domain.

### Performance on the French Corpus

On the French corpus, our model achieved an F1 score of 0.807 using a sliding window size of 300 characters, which is comparable to Yang et al.'s algorithm (0.809) [21]. However, our approach demonstrated improved precision (0.834 vs. 0.786) at the cost of reduced recall (0.782 vs. 0.834). Increasing the sliding window size from 200 characters to 300 enhanced the recall by three points, reflecting the model's improved ability to capture long-range dependencies. Additionally, the integration of frame information led to an improvement in both precision and the global F1 score (0.821 vs. 0.807), emphasizing the potential significance of this additional contextual information for accurate RE. Despite achieving similar overall performance metrics, our algorithm demonstrated a significant computational advantage, training approximately 10 times faster than the baseline Transformer architecture.

## Error Analysis

In the task of RE, our model exhibited lower recall compared to baseline models. This limitation was primarily attributed to the handling of modifiers that apply to multiple drugs. For instance, a date indicating the start of multiple
<table><caption>Table 2: Performance comparison on French and English corpora, micro-average scores (mean (sd) over 5 runs). sd was calculated only on French corpus, to limit computational time. Models used: CamemBERT-BIO (French) and ClinicalBERT (English). Mean training time in minutes. * indicates statistical difference with Yang et al. 2021. †Reference Model from Literature for Comparison ‡3 best teams n2c2 challenge [13]</caption><thead><tr><th>Corpus</th><th>Model (sliding windows of n characters)</th><th>Precision</th><th>Recall</th><th>F1</th><th>Time</th></tr></thead><tbody><tr><td rowspan="4">Corp-HUS (Fr)</td><td>Our model (200)</td><td>0.827 (0.004)*</td><td>0.756 (0.009)*</td><td>0.79 (0.006)*</td><td>8</td></tr><tr><td>Our model (300)</td><td>0.834 (0.016)*</td><td>0.782 (0.008)*</td><td>0.807 (0.005)</td><td>8</td></tr><tr><td>Our model (300) + Frame RE</td><td><strong>0.852 (0.02)*</strong></td><td>0.793 (0.012)*</td><td><strong>0.821 (0.015)</strong></td><td>12</td></tr><tr><td>Yang et al. 2021 [21]†</td><td>0.786 (0.02)</td><td><strong>0.834 (0.007)</strong></td><td>0.809 (0.013)</td><td>108</td></tr><tr><td rowspan="6">RE only</td><td>Our model (300) Reduced Corpus</td><td>0.842 (0.023)</td><td>0.782 (0.045)</td><td>0.81 (0.026)</td><td>-</td></tr><tr><td>Our model (200)</td><td>0.960</td><td>0.942</td><td>0.952</td><td>18</td></tr><tr><td>Our model (300)</td><td>0.955</td><td><strong>0.955</strong></td><td>0.955</td><td>18</td></tr><tr><td>Yang et al. 2021 [21] †</td><td>0.955</td><td>0.843</td><td>0.949</td><td>413</td></tr><tr><td>Xu J et al. 2017 [35]‡</td><td><strong>0.972</strong></td><td><strong>0.955</strong></td><td><strong>0.963</strong></td><td>-</td></tr><tr><td>Chapman AB et al. 2018 [36]‡</td><td>0.957</td><td>0.949</td><td>0.953</td><td>-</td></tr><tr><td rowspan="2">End-to-end</td><td>Corp-HUS (Fr)</td><td>Our model (300)</td><td>0.699</td><td>0.672</td><td>0.686</td><td>78</td></tr><tr><td>N2c2 (En)</td><td>Our model (300)</td><td>0.880</td><td>0.770</td><td>0.821</td><td>83</td></tr></tbody></table>

prescriptions often extends beyond the boundaries of the sliding window, leading to missed relationships. Adding same-frame relations between the different attributes of a drug was found to slightly improve results, particularly in cases involving multi-frame prescriptions. This approach enhances the model's ability to capture complex relationships and dependencies within drug attributes, resulting in more accurate extraction and representation. However, the limited presence of multi-frame prescriptions (less than 5%) in the test set, make it difficult to fully evaluate the impact of such representation. Future work could benefit from larger datasets containing more examples of multi-frame scenarios to better quantify these improvements.

## Carbon impact

The selection of optimal hyperparameters for the proposed model required 6,110 minutes of computation time, consuming an estimated 57.39 kWh of energy and resulting in a carbon footprint of approximately 2.94 kg CO₂, as calculated using the methodology outlined in [34]. This computation was performed using our specific infrastructure. It is important to note that the reported carbon impact only accounts for hyperparameter optimization. If the time spent on additional testing and implementation were included, the total computation time and carbon footprint would be approximately three times higher.

## Discussion

### Annotation of French Corpus

The F1 score for inter-annotator agreement underscores the inherent challenges associated with achieving reliable annotation for clinical text. This score shows the complexity of the automated RE task. The variability in human annotations highlights the importance of rigorous annotation guidelines and the benefit of machine-assisted pre-annotation, which offers room for improvements in consistency and efficiency [39].

### Interpretation of Results

We demonstrate that transformer-based architectures can be effectively used for RE on French corpora and, more broadly, on real-world healthcare data. Despite the complexity of the task for human annotators, transformer-based models can perform remarkably well.

One key strength of our work, lies in the diversity of the French EHRs text, which extends beyond hospital discharge reports, unlike the n2c2 data. Our method appears robust to both context and language, demonstrating good performance on clinical data and challenge datasets in both French and English. The model's lower recall, particularly in scenarios
involving modifiers that span multiple drugs, suggests room for improvement. Future research could explore how to better capture long-range dependencies [40].

The limited size of our dataset constrains its performance compared to the n2c2 dataset. This appears to be primarily due to the limited annotated data. Performance seems adequate when using reduced subset of the n2c2 dataset. The micro-average F1 score demonstrates this, with 0.807 for our dataset versus 0.810 for the restricted n2c2 dataset.

When examining a restricted n2c2 dataset (Table S3) subset, notable variations emerge across different relation types. The Duration-Drug relation showed an F1 score of 0.593 (0.11), in contrast to 0.781 on the French corpus (Table S1). The Frequency-Drug relation showed more pronounced differences, with an F1 score of 0.845 (0.05) on the n2c2 subset compared to 91.3 on the French corpus.

The Dosage/Strength relation comparison proved more nuanced due to differing annotation schemas. While the n2c2 dataset initially separates Dosage and Strength, our schema combines them into a single category.

For the Route-Drug relation, the results were more consistent, with an F1 score of 0.85 (0.045) on the n2c2 subset and 0.81 on the French corpus. However, the comparison of the Dosage/Strength relation was less straightforward. While the n2c2 dataset separates Dosage and Strength, our schema combines them into a single category. When aligning the n2c2 subset with our approach by merging these categories (Table S2), the F1 score reached 0.891 (0.024), compared to 0.898 on the French corpus. These findings highlight the interesting influence of annotation schema design on performance. The variations across different relation types underscore the complexity of medical text annotation and the importance of testing algorithms in languages other than English.

The end-to-end performance is limited by the capacity of the NER component, indicating room for improvement in this initial task (see Tables S4 and S5).

## Comparison With Previous Work

The top three methods in the n2c2 challenge used complex algorithms for RE to achieve high performance, with F1 scores of 0.963, 0.953, and 0.947. The first team [35] adopted a joint NER-RE method, enhanced by external resources (e.g. MedDRA dictionaries) and syntactic features like POS tagging. They classify potential entity pairs individually. This aligns conceptually with transformer-based tagging methods [16].

The second team [36] used a feature-rich ensemble of Random Forest classifiers, using entity types, word forms, n-grams, and syntactic data for RE across three-sentence windows. The third team [37, 38] combined models using multiple embeddings.

Our approach achieves comparable performance to these state-of-the-art techniques while being simpler, reproducible, and computationally efficient. Our model's F1 score of 0.955 on the n2c2 dataset matches or surpasses prior transformer-based methods [19, 20]. Additionally, our F1 score of 0.821 on the French corpus (using frame-based representation) sets a strong benchmark for underexplored French applications. For NER (see appendix), the results are comparable to those reported in previous studies on French clinical data [41].

Our approach successfully addressed several methodological challenges in drugs RE. By eliminating extensive preprocessing requirements and using a sliding window approach, we simplified the data preparation pipeline. The ability to generalize across different transformer models (ClinicalBERT, BioBERT, CamemBERT-BIO) further highlights the robustness of our method.

## Key Contributions

### Computational time

This study introduces an innovative framework for drugs RE from EHRs, offering both technical advancements and practical benefits. By deploying a computational setup within a hospital, we provided a real-world example of practical implementation. All experiments were run in a controlled environment to ensure consistency and reproducibility. The significant reduction in training time—10 times faster on the French corpus and 23 times faster on the n2c2 dataset—demonstrates the potential for more sustainable AI approaches in healthcare settings. This efficiency is particularly crucial in low-resource environments and for institutions with limited computational infrastructure. Generative transformer models were not tested as they could not be used in our environment due to their larger VRAM requirements.

The carbon impact analysis provides transparency about the environmental costs of model development. The hyperparameter optimization alone consumed approximately 57.39 kWh and generated 2.94 kg of CO2 emissions. While this
might seem modest, it underscores the need for continued research into computationally efficient machine learning techniques.

## Representation of treatment regimen as frames

Several challenges have been proposed for extracting drugs from EHRs, each focusing on a specific aspect of the task. In 2018, the MADE challenge [42] and the n2c2 (Natural Language Processing for Clinical Data) challenge [13] concentrated on extracting drugs and their attributes. In 2022 [43], n2c2 addressed medication modifications such as stopping, starting, increasing, or decreasing dosages and shifted its focus to the extraction of context. However, no challenge has yet addressed the complete task of extracting drugs, their attributes, and their temporal context. Our annotation schema differs from the n2c2 dataset in several key aspects, although certain relation types are shared between the two datasets. A particular improvement was the introduction of the "frame" concept, which provides a more nuanced representation of drugs relationships. This approach captures drug-specific attributes and their relationships within a contextualized framework, enabling a more nuanced representation of drugs relationships. The frame-based representation proved particularly effective, improving the F1 score from 0.807 to 0.821 on the French corpus. Augmenting annotated data by adding relations between drug attributes could facilitate the extraction of multi-frame information from free text and potentially improve performance. The limited presence of multi-frame prescriptions in the test set restricts a comprehensive evaluation of the frame-based approach. Further analysis is necessary to fully explore the potential of this methodology, which was used for cancer phenotype extraction [44]. Expanding the dataset to include more complex medication regimens, such as those of cancer patients, would provide a more robust validation. These patients often undergo multiple chemotherapy line changes, offering valuable examples of complex, multi-frame prescriptions.

## Cross-language evaluation

Despite the limitations imposed by the small amount of manually labeled data, which affects the performance of RE, especially in end-to-end. Our research made significant strides in clinical text in English and French, demonstrating the model's adaptability by achieving comparable performance on both English and French corpora. This is particularly important given the predominance of English-language research in clinical natural language processing. By validating the proposed architecture on a French corpus, we address the scarcity of research on clinical corpora in languages other than English and highlight its potential for broader applicability across diverse linguistic contexts.

## Implications for Clinical Informatics

This research has significant implications for clinical informatics, offering a more precise and computationally efficient method for extracting treatment regimens from EHRs. The improved relationship extraction could support more accurate medication tracking, potentially enhancing patient safety and therapeutic management.

## Future Research

Applying this algorithm to longitudinal data could enable the extraction of detailed medication timelines for individual patients. However, the development of reconstruction algorithms is still required to accurately assemble and represent these timelines.

## Conclusion

This study introduces a novel architecture for RE from clinical data, validated on the n2c2 dataset and applied to French clinical data, demonstrating its robustness across languages and contexts. It combines classical attribute extraction like frequency, dosage with contextual attribute extraction like starting date, negation and more. We define a new scheme of attribute annotation, enhancing the representation of drugs, their attributes, and their context.

By presenting a novel, computationally efficient approach to drugs RE, our study contributes to the advancement of natural language processing in clinical settings. The proposed framework not only improves performance but also addresses critical challenges of computational complexity and data representation.

## References

[1] Dina Demner-Fushman, Wendy W Chapman, and Clement J McDonald. What can natural language processing do for clinical decision support? *Journal of biomedical informatics*, 42(5):760–772, 2009.
[2] Jean-Baptiste Escudié, Bastien Rance, Georgia Malamut, Sherine Khater, Anita Burgun, Christophe Cellier, and Anne-Sophie Jannot. A novel data-driven workflow combining literature and electronic health records to estimate comorbidities burden for a specific disease: a case study on autoimmune comorbidities in patients with celiac disease. *BMC medical informatics and decision making*, 17:1–10, 2017.

[3] Özlem Uzuner, Imre Solti, and Eithon Cadag. Extracting medication information from clinical text. *Journal of the American Medical Informatics Association*, 17(5):514–518, 2010.

[4] Salisu Modi, Khairul Azhar Kasmiran, Nurfadhlina Mohd Sharef, and Mohd Yunus Sharum. Extracting adverse drug events from clinical notes: A systematic review of approaches used. *Journal of Biomedical Informatics*, 151:104603, 2024.

[5] David Fraile Navarro, Kiran Ijaz, Dana Rezazadegan, Hania Rahimi-Ardabili, Mark Dras, Enrico Coiera, and Shlomo Berkovsky. Clinical named entity recognition and relation extraction using natural language processing of medical free text: A systematic review. *International Journal of Medical Informatics*, 177:105122, 2023. Review des articles qui s’interesse au NER et RE. Avec les différetnes approches. Parle de l’importance de tester les algos en pratique, sur d’autres langues.

[6] Mohamed Yassine Landolsi, Lobna Hlaoua, and Lotfi Ben Romdhane. Information extraction from electronic medical documents: state of the art and future research directions. *Knowledge and Information Systems*, 65(2):463–516, Feb 2023.

[7] Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. BERT: Pre-training of deep bidirectional transformers for language understanding. In Jill Burstein, Christy Doran, and Thamar Solorio, editors, *Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long and Short Papers)*, pages 4171–4186, Minneapolis, Minnesota, June 2019. Association for Computational Linguistics.

[8] Alec Radford and Karthik Narasimhan. Improving language understanding by generative pre-training. 2018.

[9] Aryo Gema, Giwon Hong, Pasquale Minervini, Luke Daines, and Beatrice Alex. Edinburgh clinical NLP at SemEval-2024 task 2: Fine-tune your model unless you have access to GPT-4. In Atul Kr. Ojha, A. Seza Doğruöz, Harish Tayyar Madabushi, Giovanni Da San Martino, Sara Rosenthal, and Aiala Rosá, editors, *Proceedings of the 18th International Workshop on Semantic Evaluation (SemEval-2024)*, pages 1894–1904, Mexico City, Mexico, June 2024. Association for Computational Linguistics.

[10] David S Carrell, Robert E Schoen, Daniel A Leffler, Michele Morris, Sherri Rose, Andrew Baer, Seth D Crockett, Rebecca A Gourevitch, Katie M Dean, and Ateev Mehrotra. Challenges in adapting existing clinical natural language processing systems to multiple, diverse health care settings. *J Am Med Inform Assoc*, 24(5):986–991, September 2017.

[11] Timothy Miller, Egoitz Laparra, and Steven Bethard. Domain adaptation in practice: Lessons from a real-world information extraction pipeline. In Eyal Ben-David, Shay Cohen, Ryan McDonald, Barbara Plank, Roi Reichart, Guy Rotman, and Yftah Ziser, editors, *Proceedings of the Second Workshop on Domain Adaptation for NLP*, pages 105–110, Kyiv, Ukraine, April 2021. Association for Computational Linguistics.

[12] Collin F. Baker, Charles J. Fillmore, and John B. Lowe. The Berkeley FrameNet project. In *36th Annual Meeting of the Association for Computational Linguistics and 17th International Conference on Computational Linguistics, Volume 1*, pages 86–90, Montreal, Quebec, Canada, August 1998. Association for Computational Linguistics.

[13] Sam Henry, Kevin Buchan, Michele Filannino, Amber Stubbs, and Ozlem Uzuner. 2018 n2c2 shared task on adverse drug events and medication extraction in electronic health records. *J Am Med Inform Assoc*, 27(1):3–12, January 2020.

[14] Kartik Detroja, C.K. Bhensdadia, and Brijesh S. Bhatt. A survey on relation extraction. *Intelligent Systems with Applications*, 19:200244, 2023.

[15] Xiaoyan Zhao, Yang Deng, Min Yang, Lingzhi Wang, Rui Zhang, Hong Cheng, Wai Lam, Ying Shen, and Ruifeng Xu. A comprehensive survey on relation extraction: Recent advances and new frontiers. *ACM Comput. Surv.*, 56(11), July 2024.

[16] Zexuan Zhong and Danqi Chen. A frustratingly easy approach for entity and relation extraction. In Kristina Toutanova, Anna Rumshisky, Luke Zettlemoyer, Dilek Hakkani-Tur, Iz Beltagy, Steven Bethard, Ryan Cotterell, Tanmoy Chakraborty, and Yichao Zhou, editors, *Proceedings of the 2021 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies*, pages 50–61, Online, June 2021. Association for Computational Linguistics.

[17] Suncong Zheng, Feng Wang, Hongyun Bao, Yuexing Hao, Peng Zhou, and Bo Xu. Joint extraction of entities and relations based on a novel tagging scheme. In Regina Barzilay and Min-Yen Kan, editors, *Proceedings of the 55th International Joint Conference on Artificial Intelligence and Its Applications, November 2016*.
*Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers)*, pages 1227–1236, Vancouver, Canada, July 2017. Association for Computational Linguistics.

[18] Frank Martin Mtumbuka and Steven Schockaert. Entity or relation embeddings? an analysis of encoding strategies for relation extraction. In Yaser Al-Onaizan, Mohit Bansal, and Yun-Nung Chen, editors, *Findings of the Association for Computational Linguistics: EMNLP 2024*, pages 6003–6022, Miami, Florida, USA, November 2024. Association for Computational Linguistics.

[19] Darshini Mahendran and Bridget T McInnes. Extracting adverse drug events from clinical notes. *AMIA Jt Summits Transl Sci Proc*, 2021:420–429, May 2021.

[20] Qiang Wei, Zongcheng Ji, Yuqi Si, Jingcheng Du, Jinqi Wang, Firat Tiryaki, Stephen Wu, Cui Tao, Kirk Roberts, and Hua Xu. Relation extraction from clinical narratives using pre-trained language models. *AMIA Annu Symp Proc*, 2019:1236–1245, March 2020.

[21] Xi Yang, Zehao Yu, Yi Guo, Jiang Bian, and Yonghui Wu. Clinical relation extraction using transformer-based models, 2021.

[22] Zhenyu Zhang, Lin Shi, Yang Yuan, Huanyue Zhou, and Shoukun Xu. Multi-level attention with 2d table-filling for joint entity-relation extraction. *Information*, 15:407, 07 2024. un peu la même approche mais dans la matrice de prédiction rajoute des lignes et colonne pour la relation.

[23] Kexin Huang, Jaan Altosaar, and Rajesh Ranganath. Clinicalbert: Modeling clinical notes and predicting hospital readmission. *arXiv:1904.05342*, 2019.

[24] Jinhyuk Lee, Wonjin Yoon, Sungdong Kim, Donghyeon Kim, Sunkyu Kim, Chan Ho So, and Jaewoo Kang. Biobert: a pre-trained biomedical language representation model for biomedical text mining. *Bioinformatics*, 36(4):1234–1240, 09 2019.

[25] Rian Touchent and Éric de la Clergerie. CamemBERT-bio: Leveraging continual pre-training for cost-effective models on French biomedical data. In Nicoletta Calzolari, Min-Yen Kan, Veronique Hoste, Alessandro Lenci, Sakriani Sakti, and Nianwen Xue, editors, *Proceedings of the 2024 Joint International Conference on Computational Linguistics, Language Resources and Evaluation (LREC-COLING 2024)*, pages 2692–2701, Torino, Italia, May 2024. ELRA and ICCL.

[26] Perceval Wajsbürt. *Extraction and normalization of simple and structured entities in medical documents*. Theses, Sorbonne Université, December 2021.

[27] Martín Abadi, Ashish Agarwal, Paul Barham, Eugene Brevdo, Zhifeng Chen, Craig Citro, Greg S. Corrado, Andy Davis, Jeffrey Dean, Matthieu Devin, Sanjay Ghemawat, Ian Goodfellow, Andrew Harp, Geoffrey Irving, Michael Isard, Yangqing Jia, Rafal Jozefowicz, Lukasz Kaiser, Manjunath Kudlur, Josh Levenberg, Dandelion Mané, Rajat Monga, Sherry Moore, Derek Murray, Chris Olah, Mike Schuster, Jonathon Shlens, Benoit Steiner, Ilya Sutskever, Kunal Talwar, Paul Tucker, Vincent Vanhoucke, Vijay Vasudevan, Fernanda Viégas, Oriol Vinyals, Pete Warden, Martin Wattenberg, Martin Wicke, Yuan Yu, and Xiaoqiang Zheng. TensorFlow: Large-scale machine learning on heterogeneous systems, 2015. Software available from tensorflow.org.

[28] Antoine Neuraz, Ghislain Vaillant, Camila Arias, Olivier Birot, Kim-Tam Huynh, Thibaut Fabacher, Alice Rogier, Nicolas Garcelon, Ivan Lerner, Bastien Rance, and Adrien Coulet. Facilitating phenotyping from clinical texts: the medkit library. *Bioinformatics*, 40(12):btae681, 11 2024.

[29] Adam Paszke, Sam Gross, Francisco Massa, Adam Lerer, James Bradbury, Gregory Chanan, Trevor Killeen, Zeming Lin, Natalia Gimelshein, Luca Antiga, Alban Desmaison, Andreas Kopf, Edward Yang, Zachary DeVito, Martin Raison, Alykhan Tejani, Sasank Chilamkurthy, Benoit Steiner, Lu Fang, Junjie Bai, and Soumith Chintala. Pytorch: An imperative style, high-performance deep learning library. In *Advances in Neural Information Processing Systems 32*, pages 8024–8035. Curran Associates, Inc., 2019.

[30] Thomas Wolf, Lysandre Debut, Victor Sanh, Julien Chaumond, Clement Delangue, Anthony Moi, Pierric Cistac, Tim Rault, Rémi Louf, Morgan Funtowicz, Joe Davison, Sam Shleifer, Patrick von Platen, Clara Ma, Yacine Jernite, Julien Plu, Canwen Xu, Teven Le Scao, Sylvain Gugger, Mariama Drame, Quentin Lhoest, and Alexander M. Rush. Transformers: State-of-the-art natural language processing. In *Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing: System Demonstrations*, pages 38–45, Online, October 2020. Association for Computational Linguistics.

[31] Wes McKinney. Data Structures for Statistical Computing in Python. In Stéfan van der Walt and Jarrod Millman, editors, *Proceedings of the 9th Python in Science Conference*, pages 56–61, 2010.

[32] Charles R. Harris, K. Jarrod Millman, Stéfan J. van der Walt, Ralf Gommers, Pauli Virtanen, David Cournapeau, Eric Wieser, Julian Taylor, Sebastian Berg, Nathaniel J. Smith, Robert Kern, Matti Picus, Stephan Hoyer, Marten H. van Kerkwijk, Matthew Brett, Allan Haldane, Jaime Fernández del Río, Mark Wiebe, Pearu Peterson, Pierre
Gérard-Marchant, Kevin Sheppard, Tyler Reddy, Warren Weckesser, Hameer Abbasi, Christoph Gohlke, and Travis E. Oliphant. Array programming with NumPy. *Nature*, 585(7825):357-362, September 2020.

[33] Hiroki Nakayama. seqeval: A python framework for sequence labeling evaluation, 2018. Software available from https://github.com/chakki-works/seqeval.

[34] Loïc Lannelongue, Jason Grealey, and Michael Inouye. Green algorithms: Quantifying the carbon footprint of computation. *Adv Sci (Weinh)*, 8(12):2100707, May 2021.

[35] Jun Xu, Hee-Jin Lee, Zongcheng Ji, Jingqi Wang, Qiang Wei, and Hua Xu. UthCCB system for adverse drug reaction extraction from drug labels at tac-adr 2017. *Theory and Applications of Categories*, 2017.

[36] Alec B. Chapman, Kelly S. Peterson, Patrick R. Alba, Scott L. DuVall, and Olga V. Patterson. Hybrid system for adverse drug event detection. In Feifan Liu, Abhyuday Jagannatha, and Hong Yu, editors, *Proceedings of the 1st International Workshop on Medication and Adverse Drug Event Detection*, volume 90 of *Proceedings of Machine Learning Research*, pages 16–24. PMLR, 04 May 2018.

[37] Fenia Christopoulou, Thy Thy Tran, Sunil Kumar Sahu, Makoto Miwa, and Sophia Ananiadou. Adverse drug events and medication relation extraction in electronic health records with ensemble deep learning methods. *Journal of the American Medical Informatics Association*, 27(1):39–46, 2020.

[38] Meizhi Ju, Nhung TH Nguyen, Makoto Miwa, and Sophia Ananiadou. An ensemble of neural models for nested adverse drug events and medication extraction with subwords. *Journal of the American Medical Informatics Association*, 27(1):22–30, 2020.

[39] Aurélie Névéol, Rezarta Islamaj Doğan, and Zhiyong Lu. Semi-automatic semantic annotation of pubmed queries: A study on quality, efficiency, satisfaction. *Journal of Biomedical Informatics*, 44(2):310–318, 2011.

[40] Xiao Wei and Yongqi Chen. Joint extraction of long-distance entity relation by aggregating local- and semantic-dependent features. *Wireless Communications and Mobile Computing*, 2022(1):3763940, 2022.

[41] Jordan Jouffroy, Sarah F Feldman, Ivan Lerner, Bastien Rance, Anita Burgun, and Antoine Neuraz. Hybrid deep learning for medication-related information extraction from clinical texts in french: Medext algorithm development study. *JMIR Med Inform*, 9(3):e17934, Mar 2021.

[42] Abhyuday Jagannatha, Feifan Liu, Weisong Liu, and Hong Yu. Overview of the first natural language processing challenge for extracting medication, indication, and adverse drug events from electronic health record notes (MADE 1.0). *Drug Saf*, 42(1):99-111, January 2019.

[43] Diwakar Mahajan, Jennifer J. Liang, Ching-Huei Tsou, and Özlem Uzuner. Overview of the 2022 n2c2 shared task on contextualized medication event extraction in clinical notes. *Journal of Biomedical Informatics*, 144:104432, 2023.

[44] Guergana K Savova, Eugene Tseytlin, Sean Finan, Melissa Castine, Timothy Miller, Olga Medvedeva, David Harris, Harry Hochheiser, Chen Lin, Girish Chavan, et al. Deepphe: a natural language processing system for extracting cancer phenotypes from clinical records. *Cancer research*, 77(21):e115-e118, 2017.
Supplementary material

<table><caption>Table S1: Relation extraction on French Corpus, Corp-HUS</caption><thead><tr><th>Relation</th><th>Prec.</th><th>Rec.</th><th>F1</th><th>Support</th></tr></thead><tbody><tr><td>Start</td><td>0.759 (0.069)</td><td>0.657 (0.02)</td><td>0.703 (0.039)</td><td>121</td></tr><tr><td>Stop</td><td>0.716 (0.038)</td><td>0.700 (0.058)</td><td>0.706 (0.030)</td><td>70</td></tr><tr><td>Ongoing</td><td>0.829 (0.025)</td><td>0.767 (0.075)</td><td>0.796 (0.050)</td><td>123</td></tr><tr><td>Increase</td><td>0.441 (0.110)</td><td>0.560 (0.152)</td><td>0.487 (0.116)</td><td>10</td></tr><tr><td>Decrease</td><td>0.573 (0.292)</td><td>0.300 (0.235)</td><td>0.353 (0.214)</td><td>10</td></tr><tr><td>Negation</td><td>0.598 (0.048)</td><td>0.667 (0.059)</td><td>0.630 (0.044)</td><td>12</td></tr><tr><td>Contraindicated</td><td>0.413 (0.368)</td><td>0.200 (0.000)</td><td>0.229 (0.093)</td><td>5</td></tr><tr><td>Hypothetical</td><td>0.690 (0.223)</td><td>0.511 (0.099)</td><td>0.572 (0.095)</td><td>9</td></tr><tr><td>Coref</td><td>0.810 (0.041)</td><td>0.565 (0.038)</td><td>0.664 (0.019)</td><td>34</td></tr><tr><td>Discontinue</td><td>0 (0)</td><td>0 (0)</td><td>0 (0)</td><td>1</td></tr><tr><td><strong>Overall (micro)</strong></td><td><strong>0.852 (0.02)</strong></td><td><strong>0.793 (0.012)</strong></td><td><strong>0.821 (0.015)</strong></td><td><strong>395</strong></td></tr></tbody></table>

<table><caption>Table S2: Relation extraction on Corp-HUS, with categories align to n2c2 categories</caption><thead><tr><th>Relation</th><th>Prec.</th><th>Rec.</th><th>F1</th></tr></thead><tbody><tr><td>Dosage → Drug</td><td>0.971</td><td>0.971</td><td>0.971</td></tr><tr><td>Duration → Drug</td><td>0.800</td><td>0.762</td><td>0.781</td></tr><tr><td>Frequency → Drug</td><td>0.930</td><td>0.896</td><td>0.912</td></tr><tr><td>Route → Drug</td><td>0.814</td><td>0.800</td><td>0.807</td></tr></tbody></table>

<table><caption>Table S3: Relation extraction for English Corpus</caption><thead><tr><th rowspan="2">Relation</th><th colspan="3">Entire Dataset</th><th colspan="3">Restricted Dataset</th></tr><tr><th>Prec.</th><th>Rec.</th><th>F1</th><th>Prec.</th><th>Rec.</th><th>F1</th></tr></thead><tbody><tr><td>Strength → Drug</td><td>0.988</td><td>0.987</td><td>0.988</td><td>0.913 (0.016)</td><td>0.900 (0.047)</td><td>0.906 (0.027)</td></tr><tr><td>Dosage → Drug</td><td>0.983</td><td>0.979</td><td>0.981</td><td>0.888 (0.014)</td><td>0.868 (0.025)</td><td>0.877 (0.009)</td></tr><tr><td>Duration → Drug</td><td>0.928</td><td>0.880</td><td>0.904</td><td>0.798 (0.057)</td><td>0.493 (0.159)</td><td>0.593 (0.112)</td></tr><tr><td>Frequency → Drug</td><td>0.986</td><td>0.971</td><td>0.978</td><td>0.926 (0.028)</td><td>0.780 (0.082)</td><td>0.845 (0.050)</td></tr><tr><td>Form → Drug</td><td>0.993</td><td>0.988</td><td>0.991</td><td>0.973 (0.007)</td><td>0.906 (0.030)</td><td>0.938 (0.015)</td></tr><tr><td>Route → Drug</td><td>0.980</td><td>0.985</td><td>0.983</td><td>0.899 (0.018)</td><td>0.805 (0.085)</td><td>0.847 (0.045)</td></tr><tr><td>Reason → Drug</td><td>0.838</td><td>0.772</td><td>0.804</td><td>0.490 (0.021)</td><td>0.519 (0.078)</td><td>0.501 (0.034)</td></tr><tr><td>ADE → Drug</td><td>0.850</td><td>0.735</td><td>0.789</td><td>0.575 (0.050)</td><td>0.354 (0.065)</td><td>0.435 (0.059)</td></tr><tr><td><strong>Overall (micro)</strong></td><td><strong>0.962</strong></td><td><strong>0.942</strong></td><td><strong>0.952</strong></td><td><strong>0.842 (0.023)</strong></td><td><strong>0.782 (0.045)</strong></td><td><strong>0.810 (0.026)</strong></td></tr></tbody></table>
<table><caption>Table S4: Performance Metrics for Entities and Relations, End-to-end on French Corpus</caption><thead><tr><th rowspan="2"></th><th colspan="3">Strict</th><th colspan="3">Lenient</th></tr><tr><th>Prec.</th><th>Rec.</th><th>F1</th><th>Prec.</th><th>Rec.</th><th>F1</th></tr></thead><tbody><tr><td><strong>Entities</strong></td><td></td><td></td><td></td><td></td><td></td><td></td></tr><tr><td>Drug</td><td>0.947</td><td>0.961</td><td>0.954</td><td>0.964</td><td>0.976</td><td>0.970</td></tr><tr><td>Drug Classe</td><td>0.779</td><td>0.853</td><td>0.814</td><td>0.789</td><td>0.863</td><td>0.824</td></tr><tr><td>Condition</td><td>0.737</td><td>0.778</td><td>0.757</td><td>0.895</td><td>0.944</td><td>0.919</td></tr><tr><td>Context</td><td>0.682</td><td>0.674</td><td>0.678</td><td>0.774</td><td>0.756</td><td>0.765</td></tr><tr><td>Date</td><td>0.941</td><td>0.969</td><td>0.955</td><td>0.970</td><td>0.979</td><td>0.974</td></tr><tr><td>Relative Date</td><td>0.674</td><td>0.818</td><td>0.739</td><td>0.751</td><td>0.912</td><td>0.824</td></tr><tr><td>Dosage</td><td>0.954</td><td>0.971</td><td>0.963</td><td>0.960</td><td>0.977</td><td>0.968</td></tr><tr><td>Duration</td><td>0.692</td><td>0.857</td><td>0.766</td><td>0.731</td><td>0.905</td><td>0.809</td></tr><tr><td>Frequency</td><td>0.832</td><td>0.853</td><td>0.842</td><td>0.963</td><td>0.953</td><td>0.958</td></tr><tr><td>Route</td><td>0.836</td><td>0.889</td><td>0.862</td><td>0.881</td><td>0.937</td><td>0.908</td></tr><tr><td>Overall (micro)</td><td>0.860</td><td>0.898</td><td>0.878</td><td>0.909</td><td>0.931</td><td>0.920</td></tr><tr><td><strong>Relations</strong></td><td></td><td></td><td></td><td></td><td></td><td></td></tr><tr><td>Refer to</td><td>0.786</td><td>0.786</td><td>0.786</td><td>0.850</td><td>0.839</td><td>0.844</td></tr><tr><td>Start</td><td>0.493</td><td>0.567</td><td>0.527</td><td>0.537</td><td>0.608</td><td>0.570</td></tr><tr><td>Stop</td><td>0.600</td><td>0.581</td><td>0.590</td><td>0.650</td><td>0.629</td><td>0.639</td></tr><tr><td>Ongoing</td><td>0.346</td><td>0.293</td><td>0.317</td><td>0.346</td><td>0.293</td><td>0.317</td></tr><tr><td>Duration prescription</td><td>0.542</td><td>0.619</td><td>0.578</td><td>0.583</td><td>0.667</td><td>0.622</td></tr><tr><td>Increase</td><td>0.455</td><td>0.500</td><td>0.476</td><td>0.455</td><td>0.500</td><td>0.476</td></tr><tr><td>Decrease</td><td>0.444</td><td>0.400</td><td>0.421</td><td>0.444</td><td>0.400</td><td>0.421</td></tr><tr><td>Administration Time</td><td>-</td><td>-</td><td>-</td><td>-</td><td>-</td><td>-</td></tr><tr><td>Negation</td><td>0.250</td><td>0.167</td><td>0.200</td><td>0.500</td><td>0.333</td><td>0.400</td></tr><tr><td>Contraindicated</td><td>0.000</td><td>0.000</td><td>0.000</td><td>0.000</td><td>0.000</td><td>0.000</td></tr><tr><td>Hypothetical</td><td>1.000</td><td>1.000</td><td>1.000</td><td>1.000</td><td>1.000</td><td>1.000</td></tr><tr><td>Experiencer Time</td><td>-</td><td>-</td><td>-</td><td>-</td><td>-</td><td>-</td></tr><tr><td>Coref</td><td>0.708</td><td>0.500</td><td>0.586</td><td>0.750</td><td>0.529</td><td>0.621</td></tr><tr><td>Discontinues</td><td>0.000</td><td>0.000</td><td>0.000</td><td>0.000</td><td>0.000</td><td>0.000</td></tr><tr><td>Overall (micro)</td><td>0.649</td><td>0.630</td><td>0.639</td><td>0.699</td><td>0.673</td><td>0.686</td></tr></tbody></table>
<table><caption>Table S5: Performance Metrics for Entities and Relations, End-to-end on English Corpus</caption><thead><tr><th rowspan="2"></th><th colspan="3">Strict</th><th colspan="3">Lenient</th></tr><tr><th>Prec.</th><th>Rec.</th><th>F1</th><th>Prec.</th><th>Rec.</th><th>F1</th></tr></thead><tbody><tr><td><strong>Entities</strong></td><td></td><td></td><td></td><td></td><td></td><td></td></tr><tr><td>Drug</td><td>0.913</td><td>0.931</td><td>0.922</td><td>0.948</td><td>0.962</td><td>0.955</td></tr><tr><td>Strength</td><td>0.946</td><td>0.953</td><td>0.949</td><td>0.980</td><td>0.985</td><td>0.982</td></tr><tr><td>Duration</td><td>0.719</td><td>0.738</td><td>0.729</td><td>0.858</td><td>0.878</td><td>0.868</td></tr><tr><td>Route</td><td>0.929</td><td>0.923</td><td>0.926</td><td>0.950</td><td>0.941</td><td>0.945</td></tr><tr><td>Form</td><td>0.902</td><td>0.896</td><td>0.899</td><td>0.963</td><td>0.941</td><td>0.952</td></tr><tr><td>Ade</td><td>0.615</td><td>0.013</td><td>0.025</td><td>0.692</td><td>0.014</td><td>0.028</td></tr><tr><td>Dosage</td><td>0.883</td><td>0.921</td><td>0.902</td><td>0.920</td><td>0.958</td><td>0.939</td></tr><tr><td>Reason</td><td>0.594</td><td>0.661</td><td>0.626</td><td>0.670</td><td>0.739</td><td>0.703</td></tr><tr><td>Frequency</td><td>0.822</td><td>0.850</td><td>0.836</td><td>0.961</td><td>0.983</td><td>0.972</td></tr><tr><td>Overall (micro)</td><td>0.874</td><td>0.877</td><td>0.876</td><td>0.928</td><td>0.926</td><td>0.927</td></tr><tr><td><strong>Relations</strong></td><td></td><td></td><td></td><td></td><td></td><td></td></tr><tr><td>Strength → Drug</td><td>0.858</td><td>0.916</td><td>0.886</td><td>0.958</td><td>0.972</td><td>0.965</td></tr><tr><td>Dosage → Drug</td><td>0.783</td><td>0.879</td><td>0.828</td><td>0.867</td><td>0.937</td><td>0.901</td></tr><tr><td>Duration → Drug</td><td>0.540</td><td>0.681</td><td>0.602</td><td>0.735</td><td>0.838</td><td>0.783</td></tr><tr><td>Frequency → Drug</td><td>0.708</td><td>0.809</td><td>0.755</td><td>0.929</td><td>0.965</td><td>0.946</td></tr><tr><td>Form → Drug</td><td>0.822</td><td>0.869</td><td>0.845</td><td>0.943</td><td>0.932</td><td>0.937</td></tr><tr><td>Route → Drug</td><td>0.840</td><td>0.886</td><td>0.863</td><td>0.905</td><td>0.926</td><td>0.916</td></tr><tr><td>Reason → Drug</td><td>0.406</td><td>0.548</td><td>0.467</td><td>0.497</td><td>0.633</td><td>0.557</td></tr><tr><td>ADE → Drug</td><td>0.500</td><td>0.011</td><td>0.021</td><td>0.563</td><td>0.012</td><td>0.024</td></tr><tr><td>Overall (micro)</td><td>0.726</td><td>0.794</td><td>0.758</td><td>0.844</td><td>0.871</td><td>0.857</td></tr></tbody></table>