# Few-shot clinical entity recognition in English, French and Spanish: masked language models outperform generative model prompting

Marco Naguib, Xavier Tannier, Aurélie Névéol

► To cite this version:

Marco Naguib, Xavier Tannier, Aurélie Névéol. Few-shot clinical entity recognition in English, French and Spanish: masked language models outperform generative model prompting. The 2024 Conference on Empirical Methods in Natural Language Processing (EMNLP 2024), Nov 2024, Miami, United States. pp.6829-6852, 10.18653/v1/2024.findings-emnlp.400 . hal-05331970

HAL Id: hal-05331970

https://hal.science/hal-05331970v1

Submitted on 26 Oct 2025

HAL is a multi-disciplinary open access
archive for the deposit and dissemination of sci-
entific research documents, whether they are pub-
lished or not. The documents may come from
teaching and research institutions in France or
abroad, or from public or private research centers.

L'archive ouverte pluridisciplinaire HAL, est
destinée au dépôt et à la diffusion de documents
scientifiques de niveau recherche, publiés ou non,
émanant des établissements d'enseignement et de
recherche français ou étrangers, des laboratoires
publics ou privés.

<figure><img src="image_1.png" /></figure>
# Few-shot clinical entity recognition in English, French and Spanish: masked language models outperform generative model prompting

Marco Naguib, Xavier Tannier, Aurélie Névéol

Université Paris-Saclay, CNRS, LISN, Orsay, France

Sorbonne Université, Inserm, Université Sorbonne Paris-Nord, Limics, Paris, France

firstname.lastname@lisn.upsaclay.fr

xavier.tannier@sorbonne-universite.fr

## Abstract

Large language models (LLMs) have become the preferred solution for many natural language processing tasks. In low-resource environments such as specialized domains, their few-shot capabilities are expected to deliver high performance. Named Entity Recognition (NER) is a critical task in information extraction that is not covered in recent LLM benchmarks. There is a need for better understanding the performance of LLMs for NER in a variety of settings including languages other than English. This study aims to evaluate generative LLMs, employed through prompt engineering, for few-shot clinical NER. We compare 13 auto-regressive models using prompting and 16 masked models using fine-tuning on 14 NER datasets covering English, French and Spanish. While prompt-based auto-regressive models achieve competitive F1 for general NER, they are outperformed within the clinical domain by lighter biLSTM-CRF taggers based on masked models. Additionally, masked models exhibit lower environmental impact compared to auto-regressive models. Findings are consistent across the three languages studied, which suggests that LLM prompting is not yet suited for NER production in the clinical domain.

## 1 Introduction

Electronic Health Records (EHR) are rich sources of clinical information (Demner-Fushman et al., 2009), which often appear in unstructured text only (Escudié et al., 2017). Efficiently extracting information from EHRs into a more structured form can help advance clinical research, public health surveillance and clinical decision support (Wang et al., 2018).

Named Entity Recognition (NER) is a critical primary step in information extraction that aims to identify and categorize mentions of relevant entities in text. In the context of clinical information extraction, these can be mentions of clinical entities such as disorders or drugs. Extracting these entities can

significantly enhance concept normalization (Cho et al., 2017; Wajsbürt et al., 2021; Sung et al., 2022) as well as facilitate interpreting patient profiling and phenotyping (Gérardin et al., 2022). Clinical NER is widely regarded as a challenging problem : clinical entities are often jargon or ambiguous, and clinical texts have a nonstandard phrasal structure (Luo et al., 2020; Leaman et al., 2015). Additionally, the sensitive nature of EHRs results in a lack of publicly available clinical corpora, which are often restrictively licensed and predominantly available in English. Moreover, the annotation of clinical NER data demands substantial domain expertise, rendering such campaigns both costly and time-intensive (Luo et al., 2020; Névéol et al., 2014; Doğan et al., 2014; Báez et al., 2020). Additionally, due to the diversity of clinical cases, data annotated for one biomedical application might not necessarily be helpful for another. Consequently, there is a critical need for data-efficient clinical NER, specifically few-shot NER approaches.

Large Language Models (LLMs) - specifically, causal, generative models - have demonstrated significant promise in few-shot learning across a wide variety of tasks, including text classification, machine translation, and question answering (Brown et al., 2020; Radford et al., 2019). This supports our main research question: Can few-shot learning performance of LLMs transfer to the task of named entity recognition?

This study aims to evaluate generative LLMs, employed through prompt engineering, for few-shot clinical NER from the perspective of F1 performance and environmental impact.

**Challenges of few-shot NER** We identify methodological challenges with the evaluation of few-shot prompting for NER.

First, there is no standard, widely adopted manner of prompting LLMs for NER tasks (Li et al., 2022; Shen et al., 2023; Wang et al., 2023b; Ker-
aghel et al., 2024), resulting in significant challenges for reproducibility and variations in results that are difficult to interpret. Second, many efforts towards "few-shot learning" with LLMs design prompts based on their performance on large held-out validation datasets (Brown et al., 2020; Tam et al., 2021; Radford et al., 2021; Qin and Eisner, 2021), which is not consistent with a few-shot setup. In addition, in-context learning performance is shown to depend greatly on the prompt structure: a small change in task phrasing, the examples presented, the order of examples, or the tagging format can affect the performance (Zhao et al., 2021; Lu et al., 2022; Min et al., 2022). Therefore, making these choices assuming large annotated validation dataset leads to performances that are shown (Perez et al., 2021) to be over-optimistic and impossible to find in a real few-shot setting. Third, Zaghir et al. (2024) show that the majority of recent studies employing prompt engineering in medical applications lack a non-prompt-related baseline, such as fine-tuned BERT-like Masked Language Models (MLMs), which complicates the accurate assessment of the performance of these LLMs. Finally, most of these studies are mainly concentrated on English, and based on GPT, which is mainly trained on English, (Jimenez Gutierrez et al., 2022; Wang et al., 2023b; Ashok and Lipton, 2023; Hu et al., 2023b; Zaghir et al., 2024), limiting the generalizability of evaluations to other languages. The contributions of this work are as follows:

1. We describe a systematic algorithm for creating and optimizing prompts for NER, bringing a particular attention to tagging prompts (Wang et al., 2023b), a novel NER prompting technique that recently showed particular promise (García-Barragán et al., 2024; MAGRON et al., 2024).

2. We evaluate our algorithm in a true few-shot setting, by allowing prompt optimization only on the few annotated instances through cross-validation. 14 NER tasks were evaluated, spanning 6 general-domain datasets and 8 clinical datasets, with a focus on English, French and Spanish.

3. We compare this approach, applied to 13 generative LLMs, to the standard fine-tuning approach, applied to 16 MLMs, both in terms of performance and environmental impact. We provide our code at

github.com/marconaguib/autoregressive_ner

## 2 Few-shot & clinical NER

**Few-shot NER with pre-trained LLMs** Leveraging LLMs for NER usually involves using them as encoders, and training a linear projection to map vector representations into an NER tagging of the sentence, while jointly fine-tuning the parameters of the language model itself for the downstream task of NER (Devlin et al., 2019). This approach has been widely studied (Liu, 2019; Petroni et al., 2020; Joshi et al., 2020; Schweter and Akbik, 2021). Wajsbürt (2021) propose a similar architecture, enhanced with an entity decoder, to iteratively predict entity spans in the input, allowing the model to detect nested entities.

Few-shot NER can be performed by simply training such systems with the limited data available. Other approaches have been proposed to leverage LLMs specifically in few-shot setting. Namely, metric learning (Fritzler et al., 2019; Yang and Katiyar, 2020; Huang et al., 2021a) proposes to train systems to instead learn a metric over the output space. New instances can then be classified based on the distance separating them from other labeled instances. Label encoding (Aly et al., 2021; Ma et al., 2022a; Hou et al., 2020) suggests, instead, to leverage label names or textual label descriptions and encode them along with the instances in order to better tag them.

**Few-shot NER with generative LLMs** Recently, prompt construction has gained interest in the community (Brown et al., 2020; Liu et al., 2023). While most related work focused on studying prompt formulation and exploring better-performing prompt structures (Wei et al., 2022; Ashok and Lipton, 2023; Vilar et al., 2023; Wang et al., 2023b) also known as "prompt engineering", other work proposed continuous optimization of the prompt through prompt tuning (Ma et al., 2022b; Layegh et al., 2023; Hu et al., 2023a), usually reporting marginal improvements over baselines.

There is no standard, widely adopted manner of building NER prompts (Liu et al., 2023). In fact, NER associates to each instance a set of spans, each of which having a type. This structured nature of the prediction make it hard to find an intuitive but efficient manner to prompt a language model for NER, that adapts well to all contexts.

For instance, the main practice is to use separate prompts for different entity types (Li et al., 2020;
Liu et al., 2022; Chen et al., 2023a). This choice seems well-suited when the task is interested in a handful of types of entities (typically 5-10). When interested in less entity types, a single prompt can be used for detecting all entities (Ashok and Lipton, 2023). On the other hand, if there is more entity types, it could be interesting to enumerate every possible span in the input sentence and let the model predict the entity type of the span, if any (Cui et al., 2021). This method, on the inverse, is impractical for long inputs.

We identify three strategies for prompting LLMs. Constrained prompting attempts to better formulate the NER task by constraining the generation to fill in specific hand-crafted templates, usually adapted to MLMs (Cui et al., 2021; Shen et al., 2023; Ye et al., 2023; Schick and Schütze, 2021). Listing prompts consist in simply making the language model predict the entities in a list (Ashok and Lipton, 2023). Tagging prompts were studied more recently by Wang et al. (2023b). They make the language model surround entity mentions with special tags.

**Few-shot clinical NER** MLM-based few-shot NER has also been explored in the biomedical domain (Ge et al., 2023). Metric leaning (Yang and Katiyar, 2020) and label encoding (Aly et al., 2021; Ma et al., 2022a) have been explored, as well as other approaches such as active learning (Kormilitzin et al., 2021), supervised pretraining (Huang et al., 2021b) and prompt-based learning (Lee et al., 2022; Chen et al., 2023b; Cui et al., 2024).

Few studies have focused on LLM-prompting-based few-shot clinical NER. In Hu et al. (2024), GPT-3 and ChatGPT are evaluated on the 2010 i2b2/VA task (Uzuner et al., 2011) in a zero-shot context. In Jimenez Gutierrez et al. (2022), GPT-3 is evaluated on a set of biomedical information extractions tasks including the NCBI-Disease Doğan et al. (2014).

On languages other than English, Meoni et al. (2023) use InstructGPT-3 is used to build multilingual training corpora to train smaller models, and in Ateia and Kruschwitz (2023), ChatGPT is evaluated on an NER challenge focused on extracting medical procedures in Spanish.

Another interesting direction is partly fine-tuning (Liao et al., 2023) a general-domain LLM on clinical text (Han et al., 2023; Toma et al., 2023; Yang et al., 2024), and prompting the resulting LLM.

# 3 Named Entity Recognition Experiments

## 3.1 Evaluation tasks

We use 14 publicly-available NER datasets (described in the next section) to compare prompted causal models to fine-tuned masked language models in few shot settings. For each study language, we selected two out-domain datasets and two or three in-domain datasets, aiming to use comparable resources (same genre, tagset, annotation guidelines) across languages whenever possible. We use official training, validation, and test subsets when available; otherwise, we apply an 80%-10%-10% split for training, validation, and testing.

### 3.1.1 General-domain evaluation datasets

WikiNER (Nothman et al., 2013) is a multilingual silver-standard annotated NER dataset. It consists of a late-2010 snapshot of Wikipedia in nine languages. Hyperlinks referring to persons, locations or organizations were automatically annotated. We use the English, French and Spanish versions of this dataset.

CoNLL-2002 (Tjong Kim Sang, 2002) and CoNLL-2003 (Tjong Kim Sang and De Meulder, 2003) are two manually-annotated multilingual NER dataset released as a part of CoNLL shared tasks. Mentions of persons, locations, organizations and miscellaneous entities are annotated. We use the Spanish data of the 2002 version, which is a collection of news wire articles made available by the Spanish EFE News Agency, released in May 2000. We use the English data of the 2003 version, which consists of Reuters news stories between 1996 and 1997.

Quaero French Press (Grouin et al., 2011) is a manually annotated corpus of about 100 hours of speech transcribed from French speaking radio broadcast. This corpus was used in the 2011 Quaero named entity evaluation campaign. It comprises annotations for 5 entity types further divided into 32 subtypes. Our experiments relied on the five entity types: persons, locations, organizations, functions, and facilities.

### 3.1.2 Clinical evaluation datasets

E3C (Magnini et al., 2021) is a European multilingual corpus (Italian, English, French, Spanish, and Basque) of semantically annotated clinical narratives. The texts are collected from multiple publicly-available sources such as abstracts extracted from CC-licensed journals. We use the
gold standard material available from the English, French and Spanish versions of this dataset. The clinical narratives are annotated with 6 entity types : actors, body parts, events, RMLs (measurements and test results) and clinical entities.

The **n2c2-2019** (Luo et al., 2020) shared task focuses on medical concept normalization. It uses the MCN corpus developed by (Luo et al., 2019), often referred to as the n2c2-2019 dataset. It includes de-identified discharge summaries from the Partners HealthCare and Beth Israel Deaconess Medical Center. In order to convert the medical concept normalization task into an NER task, we use the annotated Concept Unique Identifiers (CUIs) to map each mention to the corresponding UMLS semantic group (Lindberg et al., 1993; McCray et al., 2001).

The **NCBI-Disease** (Doğan et al., 2014) corpus gathers 793 PubMed abstracts where mentions of diseases are annotated in four types depending on their syntax : Specific Diseases (e.g. *diastrophic dysplasia*), Disease Classes (e.g. *an autosomal recessive disease*), Composite Mentions (e.g. *colorectal, endometrial, and ovarian cancers*), and Modifiers (e.g. *C7-deficient*).

QuaeroFrenchMed (Névéol et al., 2014) consists of two text sources that we treat separately. The first part, **EMEA** is a collection of 13 patient information leaflets on marketed drugs from the European Medicines Agency (EMEA). The second part, **MEDLINE**, consist of 2,500 titles of research articles indexed in the MEDLINE database<fnref n="1" />. The two parts are annotated with 10 entity types corresponding to UMLS semantic groups.

The Chilean Waiting List (Báez et al., 2020) corpus consists of 900 de-identified referrals for several specialty consultations in Spanish from the waiting list in Chilean public hospitals, manually annotated with 10 entity types : abbreviations, body parts, clinical findings, diagnostic procedure, diseases, family members, laboratory or test results, laboratory procedures, medications, procedures, signs or symptoms and therapeutic procedures. It can be noted that these types can be redundant (e.g. all diagnostic procedures are also annotated as procedures).

### 3.1.3 Few-shot learning set-up

We simulate the few-shot context by only providing the models with a few annotated examples that can be used in training, prompting and validation.

No additional examples are made available. In this study, we choose to mainly focus on *k* = 100 sentences, which corresponds to one to two hours of annotation in the clinical domain (Névéol et al., 2014; Campillos et al., 2018). We use a fixed random seed *p* to choose *k* examples among all those available in the corpus. In Section A.1.1, we discuss the effect of the choice of *k* and of *p*.

Additionally, we train the best-performing language models with the entire training dataset to provide a skyline comparison, i.e. performance of the models outside the few-shot setting.

## 3.2 Language Models

Table 5 (appendix A.2) presents an overview of the language models used in our study. English is covered in all causal language models, which is not the case for Spanish and French. Except for mBERT and XLM-RoBERTa, masked language models cover only one of our study languages.

## 3.3 NER with MLMs

Although MLMs have been adapted to few-shot learning in architectures suited for low-resource contexts (see section 2), we compare LLM prompting to the simple and standard MLM usage without any further adaptation for few-shot learning, as an accessible and easily-reproducible baseline.

We use NLStruct (Wajsbürt, 2021), an open-source Python library <fnref n="2" /> that implements the standard fine-tuning approach. NLStruct uses the representations provided by the language model to encode the input, then employs a bidirectional LSTM decoder and a CRF to iteratively predict the entities present in the encoded input, as described by Gérardin et al. (2022). This approach allows NLStruct to effectively handle nested entities, which are prevalent in some of the study corpora.

We train the model for 20 epochs on 80% of the data and use the remaining held-out 20% for early stopping.

## 3.4 NER with generative LLM prompting

Our experiments prompt models to tag entities in the input sentence, instead of listing them. This choice is supported by further experiments reported in Section A.1.2. The upper part of Figure 1 shows a sample tagging prompt, highlighting sections in the prompt that guided the design of features for prompt phrasing.

<footnote n="1">¹http://pubmed.ncbi.nlm.nih.gov/</footnote>

<footnote n="2">²https://github.com/percevalw/nlstruct</footnote>
**Prompt features** We describe below the nine optional features that control the phrasing of the prompt, as well as the criteria for selecting the few-shot examples featured in the prompt.

1. **Prompt language:** By default, we prompt all language models in English, as it is the most ubiquitous language in all of their training corpora. This feature allows the model to be prompted in French or Spanish, to align the prompt language with that of the test sentence.

2. **Additional sentences:** By default, we present 5 annotated sentences in the prompts. This feature presents 5 additional sentences (i.e., 10 sentences in total). Section A.1.3 discusses adding more demonstrations to the prompt.

3. **Self verification:** By default, we select the 5 closest sentences to the test sentence in terms of TF-IDF distance, among the training set. The mentions tagged by the model are then considered to be the model's final predictions. This feature selects instead the 5 sentences featuring the most entities of the target type and features them in an initial prompt. Intuitively, this prompt results in higher recall and lower precision. A second "self-verification" prompt is then used over the model's initial predictions in order to filter out the false positives. A sample self-verification prompt is shown in the bottom part of Figure 1.

The number of demonstrations follows that of the main prompt.

4. **Taggers:** By default, we follow (Wang et al., 2023b) prompting the model to surround mentions with @@ and ##. This feature prompts it to surround mentions with quotes « and » instead.

5. **Address a specialist in the prompt:** By default, the first sentence is the task description shown in Figure 1. This feature starts the prompt with *You are an excellent <specialist>. You can identify all the mentions of <entity-type> in a sentence, by putting them in a specific format. Here are some examples you can handle: instead. The <specialist> is a linguist or a clinician, following the task domain.*

6. **Include label definitions in the prompt:** This feature adds a one-sentence description for each entity type. Full entity descriptions used can be found in appendix A.3.

7. **Introductory sentence for the test instance:** By default, the demonstrations are immediately followed by the test instance. This feature separates them with *Identify all the mentions of <entity-type> in the following sentence, by putting <begin-tag> in front and a <end-tag> behind each of them.*

8. **Require a long answer for the self-verification:** By default, the self-verification prompt demonstrates *Yes* (respectively *No*) as answers. This feature demonstrates *<mention> is a(n) <entity-type>, yes*. (respectively *<mention> is not a(n) <entity-type>, no.*) instead.

9. **Dialogue template:** This feature replaces the *Input:* and *Output:* in the prompt by dashes to imitate a dialog template.

**Identification of optimal prompt configuration**

In-context learning performance is shown to vary greatly depending on the exact phrasing of prompts (Lu et al., 2022; Min et al., 2022). In addition, the optimal choice for each of these features can vary depending on the model used. For instance, intuitively, models that are heavily pretrained on the English language tend to perform better with an English template than one in the language of the corpus.

While our system aims to search for the best combination of parameters for each model, a grid search over them would require $2^9 = 512$ experiments for each model, for each dataset. In order to build a lighter system, we opt for a greedy search. We iterate over the features and select options that perform better than the default. In Section A.1.4, we illustrate how the optimal parameter combination can be obtained with a greedy search using 5% of the computation required for grid search. Many efforts towards "few-shot learning" with LLMs design prompts on large held-out validation datasets (Brown et al., 2020; Tam et al., 2021; Radford et al., 2021; Qin and Eisner, 2021). This leads to results that are shown (Perez et al., 2021) to be over-optimistic. A true few-shot evaluation involves optimization with access to a small number of annotated instances, which corresponds to our $k = 100$. In this no-training context, we follow (Perez et al., 2021) optimizing these features
<figure><img src="image_2.png" /><figcaption>Figure 1: Example of a tagging prompt, used in the main experiment (top) and a self-verification prompt (bottom) for detecting DISO mentions in n2c2-2019</figcaption></figure>

through a leave-one-out cross-validation (LOOCV) over the validation set.

## 3.5 Performance metrics

**Micro-F1** For simplicity, we evaluate models over one global information extraction performance score. It is computed as the micro-average of F1-measures for each entity type.

**Carbon footprint** We use GreenAlgorithms v2.2 (Lannelongue et al., 2021) 3 to estimate the carbon footprint of each experiment, based on factors such as runtime, computing hardware and location where electricity used by our computer facility was produced.

# 4 Results

## 4.1 Environmental Impact

Figure 2 compares carbon emission incurred by resolving ConLL-2003 using LLM prompting (using

<figure><img src="image_3.png" /><figcaption>Figure 2: Carbon emission (g) incurred by resolving ConLL-2003 using three models</figcaption></figure>

Llama-2-70B and Mistral-7B) as well as with fine-tuning BERT-large. For comparison, the impact of using Llama-2-70B is comparable to that of driving a thermal car of average size for 1 kilometer.

Appendix A.4 details the carbon emission estimations for all of our experiments. In total, the experiments described in this paper are estimated to have generated around 31kg of CO2 equivalent (29kg for main experiments, and 2kg for ablation).

<footnote n="3">³http:// calculator.green-algorithms.org/</footnote>
<table><caption>Table 1: This table presents the micro-F1 obtained from few-shot experiments. Skyline results are obtained using all training data available instead of the few-shot setting.</caption><thead><tr><th rowspan="2">#</th><th rowspan="2">Model</th><th colspan="5">English</th><th colspan="4">French</th><th colspan="4">Spanish</th></tr><tr><th>WikiNER</th><th>CoNLL2003</th><th>E3C</th><th>n2c2</th><th>NCBI</th><th>WikiNER</th><th>QFP</th><th>E3C</th><th>EMEA</th><th>MEDLINE</th><th>WikiNER</th><th>CoNLL2002</th><th>E3C</th><th>CWL</th></tr></thead><tbody><tr><td colspan="17">Few-shot approaches</td></tr><tr><td rowspan="13">Causal</td><td>1</td><td>Llama-2-70B</td><td>0.728</td><td>0.721</td><td>0.312</td><td><strong>0.309</strong></td><td><strong>0.400</strong></td><td>0.740</td><td>0.400</td><td>0.483</td><td>0.201</td><td>0.312</td><td><strong>0.805</strong></td><td>0.616</td><td>0.021</td><td>0.339</td></tr><tr><td>2</td><td>Llama-3-8B-Instruct</td><td><strong>0.756</strong></td><td><strong>0.727</strong></td><td><strong>0.478</strong></td><td>0.252</td><td>0.390</td><td><strong>0.796</strong></td><td><strong>0.508</strong></td><td><strong>0.652</strong></td><td><strong>0.443</strong></td><td><strong>0.433</strong></td><td>0.784</td><td><strong>0.764</strong></td><td><strong>0.226</strong></td><td><strong>0.437</strong></td></tr><tr><td>3</td><td>Mistral-7B</td><td>0.754</td><td>0.646</td><td>0.488</td><td>0.291</td><td>0.395</td><td>0.727</td><td>0.428</td><td>0.590</td><td>0.229</td><td>0.333</td><td>0.720</td><td>0.707</td><td>0.083</td><td>0.374</td></tr><tr><td>4</td><td>Phi-3-medium-instruct</td><td>0.464</td><td>0.302</td><td>0.234</td><td>0.112</td><td>0.178</td><td>0.394</td><td>0.322</td><td>0.317</td><td>0.139</td><td>0.161</td><td>0.612</td><td>0.461</td><td>0.045</td><td>0.280</td></tr><tr><td>5</td><td>BLOOM-7B1</td><td>0.524</td><td>0.557</td><td>0.279</td><td>0.113</td><td>0.151</td><td>0.148</td><td>0.206</td><td>0.320</td><td>0.197</td><td>0.120</td><td>0.470</td><td>0.419</td><td>0.051</td><td>0.117</td></tr><tr><td>6</td><td>Falcon-40B</td><td>0.686</td><td>0.708</td><td>0.280</td><td>0.279</td><td>0.305</td><td>0.662</td><td>0.456</td><td>0.378</td><td>0.279</td><td>0.283</td><td>0.720</td><td>0.543</td><td>0.072</td><td>0.267</td></tr><tr><td>7</td><td>GPT-J-6B</td><td>0.521</td><td>0.493</td><td>0.167</td><td>0.179</td><td>0.238</td><td>0.423</td><td>0.244</td><td>0.334</td><td>0.080</td><td>0.177</td><td>0.005</td><td>0.142</td><td>0.021</td><td>0.162</td></tr><tr><td>8</td><td>OPT-66B</td><td>0.608</td><td>0.495</td><td>0.227</td><td>0.157</td><td>0.234</td><td>0.624</td><td>0.406</td><td>0.019</td><td>0.206</td><td>0.283</td><td>0.166</td><td>0.273</td><td>0.043</td><td>0.204</td></tr><tr><td>9</td><td>Vicuna-13B</td><td>0.657</td><td>0.708</td><td>0.355</td><td>0.236</td><td>0.300</td><td>0.677</td><td>0.350</td><td>0.399</td><td>0.207</td><td>0.326</td><td>0.744</td><td>0.250</td><td>0.040</td><td>0.213</td></tr><tr><td>10</td><td>Vicuna-7B</td><td>0.594</td><td>0.489</td><td>0.259</td><td>0.147</td><td>0.172</td><td>0.591</td><td>0.277</td><td>0.439</td><td>0.152</td><td>0.296</td><td>0.659</td><td>0.569</td><td>0.042</td><td>0.151</td></tr><tr><td>11</td><td>BioMistral-7B</td><td>0.414</td><td>0.354</td><td>0.175</td><td>0.086</td><td>0.257</td><td>0.547</td><td>0.299</td><td>0.350</td><td>0.236</td><td>0.186</td><td>0.578</td><td>0.540</td><td>0.014</td><td>0.226</td></tr><tr><td>12</td><td>Medalpaca-7B</td><td>0.537</td><td>0.586</td><td>0.272</td><td>0.138</td><td>0.132</td><td>0.529</td><td>0.142</td><td>0.259</td><td>0.162</td><td>0.252</td><td>0.581</td><td>0.490</td><td>0.088</td><td>0.220</td></tr><tr><td>13</td><td>Vigogne-13B</td><td>0.593</td><td>0.655</td><td>0.252</td><td>0.176</td><td>0.309</td><td>0.515</td><td>0.250</td><td>0.464</td><td>0.099</td><td>0.142</td><td>0.580</td><td>0.561</td><td>0.010</td><td>0.198</td></tr><tr><td rowspan="13">Masked</td><td>14</td><td>mBERT</td><td>0.768</td><td>0.804</td><td>0.624</td><td>0.378</td><td>0.401</td><td>0.801</td><td>0.728</td><td>0.741</td><td>0.588</td><td>0.428</td><td><strong>0.812</strong></td><td>0.760</td><td>0.324</td><td>0.432</td></tr><tr><td>15</td><td>XLM-R-large</td><td>0.786</td><td>0.826</td><td><strong>0.637</strong></td><td>0.462</td><td>0.471</td><td>0.811</td><td>0.781</td><td>0.762</td><td>0.629</td><td>0.531</td><td>0.797</td><td>0.781</td><td>0.325</td><td>0.528</td></tr><tr><td>16</td><td>BERT-large</td><td>0.776</td><td>0.814</td><td>0.626</td><td>0.435</td><td>0.422</td><td>-</td><td>-</td><td>-</td><td>-</td><td>-</td><td>-</td><td>-</td><td>-</td><td>-</td></tr><tr><td>17</td><td>RoBERTa-large</td><td><strong>0.790</strong></td><td><strong>0.829</strong></td><td>0.626</td><td>0.462</td><td><strong>0.552</strong></td><td>-</td><td>-</td><td>-</td><td>-</td><td>-</td><td>-</td><td>-</td><td>-</td><td>-</td></tr><tr><td>18</td><td>Bio_ClinicalBERT</td><td>0.528</td><td>0.542</td><td>0.621</td><td>0.469</td><td>0.420</td><td>-</td><td>-</td><td>-</td><td>-</td><td>-</td><td>-</td><td>-</td><td>-</td><td>-</td></tr><tr><td>19</td><td>ClinicalBERT</td><td>0.462</td><td>0.597</td><td>0.622</td><td><strong>0.480</strong></td><td>0.397</td><td>-</td><td>-</td><td>-</td><td>-</td><td>-</td><td>-</td><td>-</td><td>-</td><td>-</td></tr><tr><td>20</td><td>MedBERT</td><td>0.613</td><td>0.673</td><td>0.607</td><td>0.478</td><td>0.504</td><td>-</td><td>-</td><td>-</td><td>-</td><td>-</td><td>-</td><td>-</td><td>-</td><td>-</td></tr><tr><td>21</td><td>CamemBERT-large</td><td>-</td><td>-</td><td>-</td><td>-</td><td>-</td><td><strong>0.829</strong></td><td><strong>0.793</strong></td><td>0.768</td><td><strong>0.661</strong></td><td><strong>0.564</strong></td><td>-</td><td>-</td><td>-</td><td>-</td></tr><tr><td>22</td><td>FlauBERT-large</td><td>-</td><td>-</td><td>-</td><td>-</td><td>-</td><td>0.826</td><td>0.778</td><td>0.760</td><td>0.635</td><td>0.540</td><td>-</td><td>-</td><td>-</td><td>-</td></tr><tr><td>23</td><td>DrBERT-4GB</td><td>-</td><td>-</td><td>-</td><td>-</td><td>-</td><td>0.587</td><td>0.599</td><td>0.730</td><td>0.602</td><td>0.497</td><td>-</td><td>-</td><td>-</td><td>-</td></tr><tr><td>24</td><td>CamemBERT-bio</td><td>-</td><td>-</td><td>-</td><td>-</td><td>-</td><td>0.782</td><td>0.761</td><td><strong>0.779</strong></td><td>0.636</td><td>0.557</td><td>-</td><td>-</td><td>-</td><td>-</td></tr><tr><td>25</td><td>BETO</td><td>-</td><td>-</td><td>-</td><td>-</td><td>-</td><td>-</td><td>-</td><td>-</td><td>-</td><td>-</td><td>0.794</td><td>0.732</td><td>0.352</td><td>0.522</td></tr><tr><td>26</td><td>PatanaBERT</td><td>-</td><td>-</td><td>-</td><td>-</td><td>-</td><td>-</td><td>-</td><td>-</td><td>-</td><td>-</td><td>0.802</td><td>0.769</td><td>0.343</td><td>0.487</td></tr><tr><td>27</td><td>TulioBERT</td><td>-</td><td>-</td><td>-</td><td>-</td><td>-</td><td>-</td><td>-</td><td>-</td><td>-</td><td>-</td><td>0.804</td><td><strong>0.798</strong></td><td>0.340</td><td>0.482</td></tr><tr><td>28</td><td>BSC-BioEHR</td><td>-</td><td>-</td><td>-</td><td>-</td><td>-</td><td>-</td><td>-</td><td>-</td><td>-</td><td>-</td><td>0.804</td><td>0.758</td><td>0.354</td><td><strong>0.578</strong></td></tr><tr><td>29</td><td>BSC-Bio</td><td>-</td><td>-</td><td>-</td><td>-</td><td>-</td><td>-</td><td>-</td><td>-</td><td>-</td><td>-</td><td>0.804</td><td>0.775</td><td><strong>0.358</strong></td><td>0.552</td></tr><tr><td colspan="17">Masked fully-supervised (skyline)</td></tr><tr><td></td><td>RoBERTa-large</td><td>0.919</td><td>0.939</td><td>0.718</td><td>0.712</td><td>0.815</td><td>-</td><td>-</td><td>-</td><td>-</td><td>-</td><td>-</td><td>-</td><td>-</td><td>-</td></tr><tr><td></td><td>CamemBERT-large</td><td>-</td><td>-</td><td>-</td><td>-</td><td>-</td><td>0.928</td><td>0.834</td><td>0.828</td><td>0.748</td><td>0.713</td><td>-</td><td>-</td><td>-</td><td>-</td></tr><tr><td></td><td>BETO</td><td>-</td><td>-</td><td>-</td><td>-</td><td>-</td><td>-</td><td>-</td><td>-</td><td>-</td><td>-</td><td>0.918</td><td>0.881</td><td>0.411</td><td>0.736</td></tr></tbody></table>

## 4.2 Comparison of model performance

Table 1 presents the micro-F1 performance of the models on the test set of each dataset. Figures 3 to 5 present the results for each language with a vizualization of model type (circles for causal models vs. squares for masked models), model size in terms of parameters and domain (clinical in green vs. general in blue). Overall, results show that, despite being smaller and theoretically requiring a larger amount of training data, masked, "BERT-like" models consistently outperform generative LLM prompting in the context of few-shot NER commonly found in biomedical applications. Specifically, on English and Spanish, Figures 3 and 5, show that, while some generative LLMs are competitive in the general domain, they suffer a sharper performance drop in clinical NER, compared to fine-tuned MLMs. On French, Figure 4 suggests that generative LLM prompting does not seem competitive on either domain.

Moreover, this performance comes at a much lower environmental impact (CO2 emissions are 10-50 times lower for MLMs vs. prompted LLMs). Another important finding is that, in addition to high performance scores, MLMs achieve results

that are relatively close to each other. For example, on the WikiNER generalist task in English, the 4 general-domain models tested achieved F1-scores of between 0.768 and 0.79.

Besides, we show that domain-adapted MLMs (e.g., ClinicalBERT, CamemBERT-bio) exhibit a sharp performance drop in general domain tasks, illustrating the classical issue of "catastrophic forgetting". In addition, they do not contribute performance improvements in specialized tasks, with the exception of Spanish tasks. However, there is a notable difference in model size: specialized models only have 110 million parameters (vs. 340 million for other models).

Named entity recognition based on BERT-type representations has received a great deal of attention in recent years, and is undoubtedly more mature than the use of LLM prompting for this task. We have implemented the prompt-based NER techniques recently published in the literature, to the best of our knowledge. It is, of course, possible that new approaches will make it possible to increase performance in the future. However, this is arguably a difficult task for a generative model, as it is highly constrained in its syntax and its evaluation.
<figure><img src="image_4.png" /><figcaption>Figure 3: Performance of models on English. The general performance is the average of micro-F1 obtained on WikiNER-en and CoNLL-2003. The clinical performance is the average on E3C-en, n2c2 and NCBI-Disease. The red lines represent the *skyline* performance obtained with the entirety of each training dataset.</figcaption></figure>

<figure><img src="image_5.png" /><figcaption>Figure 5: Performance of models on Spanish. The general performance is the average of micro-F1 obtained on WikiNER-es and CoNLL-2002. The clinical performance is the average on E3C-es and CWL. The red lines represent the *skyline* performance obtained with the entirety of each training dataset.</figcaption></figure>

<figure><img src="image_6.png" /><figcaption>Figure 4: Performance of models on French. The general performance is the average of micro-F1 obtained on WikiNER-fr and QuaeroFrenchPress. The clinical performance is the average on E3C-fr, EMEA and MEDLINE. The red lines represent the *skyline* performance obtained with the entirety of each training dataset.</figcaption></figure>

These results are no indication of performance on other tasks such as classification.

## 4.3 Practical use of language models for low-resource NER

Overall, our experiments suggest that the performance of language models for clinical named entity recognition is currently sub-optimal. In particular, even MLM-based models, simply fine-tuned on the limited data available, fail to approach the performance of fully supervised models. The three large models trained with the entirety of each training dataset (*skylines* Table 1) systematically outperform the best few-shot results, by 5% to 16% for the general domain, and 8% to 48% for the biomedical domain. However, performance can be judged satisfactory enough for pre-annotation use, to complement or accelerate manual annotation, for example in an online or active learning context.

# 5 Conclusion

This study assessed the performance of two types of large language models, for few-shot entity recognition in three languages. Our experiments show that few-shot learning performance is significantly lower in the clinical vs. general domain. While masked language models perform better than causal language models (higher F1, lower CO2 emissions), LLM prompting is not yet suited for effective information extraction.
## Limitations of our study

**Random Noise and Significance** In MLM experiments, the parameters of the NER tagging layer added on top of the pretrained language model are initialized randomly. Likewise, in LLM prompting experiments, the demonstrations in the prompts are shuffled randomly, and the negative examples in the self verification prompts are selected randomly. These random decisions can introduce noise in our performance measurements. Replicating all the experiments would allow us to draw more solid conclusions (Reimers and Gurevych, 2017), but would also come at a considerable cost (29kg of CO2 equivalent, and around 64 hours of computation for each replication). The large number of models tested and tasks addressed can however support the main observations of this article. For instance, we use Almost Stochastic Order (ASO)<fnref n="4" /> (Dror et al., 2019) with a confidence level $\alpha = 0.05$ to measure the significance of the superiority of fine-tuned MLMs over prompted LLMs for each dataset separately. We do not always observe satisfying values of $\epsilon_{min}$ as to whether MLMs perform better than prompted LLMs on general-domain NER (0.426, 0.081 and 0.028 respectively on WikiNER-English, CoNLL2003 and WikiNER-French). Regarding clinical NER, MLMs perform significantly better than prompted LLMs: MLMs are stochastically dominant over prompted LLMs ($\epsilon_{min}=0$) for all clinical datasets.

**Data contamination** The size of the training corpora used for creating LLMs makes it increasingly difficult to control for data contamination, i.e. the presence of test corpora. Moreover, (Balloccu et al., 2024) describe the problem of "indirect contamination", encountered when models are iteratively improved by using data coming from users, including test dataset.

The community is calling for efforts towards better documentation of training datasets (Bender and Friedman, 2018). While some datasets are by construction incompatible with some models (e.g., there is no Spanish training corpus in GPT-J or Llama-2) we are unable to affirm full exclusion of

all datasets from all models studied.

**Fine-tuning generative models** Our study focuses on low-data scenarios prevalent in real-world biomedical applications, where computational resources, notably GPU availability, are constrained. Additionally, the use of models via API calls is challenging due to privacy and security concerns. Consequently, we selected methods requiring limited computing power: either fine-tuning "small", BERT-like models, or prompting larger, frozen, models. This pragmatic choice stems from the need to balance performance with computational efficiency, ensuring that the proposed methods remain feasible for implementation in resource-limited settings.

However, more computationally expensive solutions may exist and may be effectively deployed in other environments. Typically, one could use the limited available data to partially fine-tune a generative LLM for NER (Liao et al., 2023).

**Error analysis** To facilitate comparison between models, we evaluated models over one global information extraction performance score: the micro-F1 measure. Overall, precision and recall scores (not shown) are balanced across entity types. In clinical datasets, some entity types, such as chemicals and anatomy, yield higher performance than others such as devices and procedures. This disparity cannot be explained by the distribution in the training corpus, since prompts provide the same number of examples across entity types. We hypothesize that the regularity and prevalence in LLM pretaining corpora of some entities may explain the higher performance. In future work, we plan more in-depth analysis, to better understand the performance of the LLM prompting approach.

## Ethical considerations

All the datasets analyzed in our experiments are publicly available corpora, used consistently with the relevant data use agreements.

The use of LLMs can incur significant environmental impact. We have measured carbon emissions using GreenAlgorithms. Our experiments support the conclusion that MLMs yield higher information extraction performance with lower carbon emission, compared to LLM prompting for NER.

<footnote n="4">⁴Given the performance scores of two algorithms A and B, each of which run several times with different settings, ASO computes a test-specific value ($\epsilon_{min}$) that indicates how far algorithm A is from being significantly better than algorithm B. If distance $\epsilon_{min} = 0.0$, one can claim that A stochastically dominant over B with the predefined significance level. The literature commonly interprets $\epsilon_{min} < 0.5$ as an indicator of a significant superiority of A over B.</footnote>
# Acknowledgements

This work has received funding from the French "Agence Nationale pour la Recherche" under grant agreement CODEINE ANR-20-CE23-0026-01. This work was performed using HPC resources from GENCI-IDRIS (Grant 2023-AD011014533). The authors thank Dr. Juan Manual Coria for his help phrasing prompts in Spanish.

# References

* Marah Abdin, Sam Ade Jacobs, Ammar Ahmad Awan, et al. 2024. Phi-3 Technical Report: A Highly Capable Language Model Locally on Your Phone.

* Emily Alsentzer, John Murphy, William Boag, Wei-Hung Weng, Di Jindi, Tristan Naumann, and Matthew McDermott. 2019. Publicly available clinical BERT embeddings. In *Proceedings of the 2nd Clinical Natural Language Processing Workshop*, pages 72–78, Minneapolis, Minnesota, USA. Association for Computational Linguistics.

* Rami Aly, Andreas Vlachos, and Ryan McDonald. 2021. Leveraging Type Descriptions for Zero-shot Named Entity Recognition and Classification. In *Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th International Joint Conference on Natural Language Processing (Volume 1: Long Papers)*, pages 1516–1528, Online. Association for Computational Linguistics.

* Dhananjay Ashok and Zachary C. Lipton. 2023. PromptNER: Prompting For Named Entity Recognition.

* Samy Ateia and Udo Kruschwitz. 2023. Is ChatGPT a Biomedical Expert? – Exploring the Zero-Shot Performance of Current GPT Models in Biomedical Tasks.

* Pablo Báez, Fabián Villena, Matías Rojas, Manuel Durán, and Jocelyn Dunstan. 2020. The Chilean waiting list corpus: a new resource for clinical named entity recognition in Spanish. In *Proceedings of the 3rd Clinical Natural Language Processing Workshop*, pages 291–300, Online. Association for Computational Linguistics.

* Simone Balloccu, Patrícia Schmidtová, Mateusz Lango, and Ondrej Dusek. 2024. Leak, cheat, repeat: Data contamination and evaluation malpractices in closed-source LLMs. In *Proceedings of the 18th Conference of the European Chapter of the Association for Computational Linguistics (Volume 1: Long Papers)*, pages 67–93, St. Julian's, Malta. Association for Computational Linguistics.

* Emily M. Bender and Batya Friedman. 2018. Data Statements for Natural Language Processing: Toward Mitigating System Bias and Enabling Better Science. *Transactions of the Association for Computational Linguistics*, 6:587–604.

* Tom Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared D Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, Sandhini Agarwal, Ariel Herbert-Voss, Gretchen Krueger, Tom Henighan, Rewon Child, Aditya Ramesh, Daniel Ziegler, Jeffrey Wu, Clemens Winter, Chris Hesse, Mark Chen, Eric Sigler, Mateusz Litwin, Scott Gray, Benjamin Chess, Jack Clark, Christopher Berner, Sam McCandlish, Alec Radford, Ilya Sutskever, and Dario Amodei. 2020. Language models are few-shot learners. In *Advances in Neural Information Processing Systems*, volume 33, pages 1877–1901. Curran Associates, Inc.

* Leonardo Campillos, Louise Deléger, Cyril Grouin, Thierry Hamon, Anne-Laure Ligozat, and Aurélie Névéol. 2018. A French clinical corpus with comprehensive semantic annotations: development of the Medical Entity and Relation LIMSI annotated Text corpus (MERLOT). *Language Resources and Evaluation*, 52:571–601.

* Casimiro Pio Carrino, Joan Llop, Marc Pàmies, Asier Gutiérrez-Fandiño, Jordi Armengol-Estapé, Joaquín Silveira-Ocampo, Alfonso Valencia, Aitor Gonzalez-Agirre, and Marta Villegas. 2022. Pretrained biomedical language models for clinical NLP in Spanish. In *Proceedings of the 21st Workshop on Biomedical Language Processing*, pages 193–199, Dublin, Ireland. Association for Computational Linguistics.

* José Cañete, Gabriel Chaperon, Rodrigo Fuentes, Jou-Hui Ho, Hojin Kang, and Jorge Pérez. 2020. Spanish Pre-Trained BERT Model and Evaluation Data. In *PML4DC at ICLR 2020*.

* Jiawei Chen, Yaojie Lu, Hongyu Lin, Jie Lou, Wei Jia, Dai Dai, Hua Wu, Boxi Cao, Xianpei Han, and Le Sun. 2023a. Learning In-context Learning for Named Entity Recognition. In *Proceedings of the 61st Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers)*, pages 13661–13675, Toronto, Canada. Association for Computational Linguistics.

* Peng Chen, Jian Wang, Hongfei Lin, Di Zhao, and Zhihao Yang. 2023b. Few-shot biomedical named entity recognition via knowledge-guided instance generation and prompt contrastive learning. *Bioinformatics*, 39(8):btad496.

* Hyejin Cho, Wonjun Choi, and Hyunju Lee. 2017. A method for named entity normalization in biomedical articles: Application to diseases and plants. *BMC Bioinformatics*, 18.

* Alexis Conneau, Kartikay Khandelwal, Naman Goyal, Vishrav Chaudhary, Guillaume Wenzek, Francisco Guzmán, Edouard Grave, Myle Ott, Luke Zettlemoyer, and Veselin Stoyanov. 2020. Unsupervised Cross-lingual Representation Learning at Scale. In *Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics*, pages 8440–8451, Online. Association for Computational Linguistics.
Leyang Cui, Yu Wu, Jian Liu, Sen Yang, and Yue Zhang. 2021. Template-based named entity recognition using BART. In Findings of the Association for Computational Linguistics: ACL-IJCNLP 2021, pages 1835–1845. Online. Association for Computational Linguistics.

Zhaojian Cui, Kai Yu, Zhenming Yuan, Xiaofeng Dong, and Weibin Luo. 2024. Language inference-based learning for Low-Resource Chinese clinical named entity recognition using language model. Journal of Biomedical Informatics, 149:104559.

Dina Demner-Fushman, Wendy W. Chapman, and Clement J. McDonald. 2009. What can natural language processing do for clinical decision support? Journal of Biomedical Informatics, 42(5):760–772. Biomedical Natural Language Processing.

Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. 2019. BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding. In North American Chapter of the Association for Computational Linguistics.

Rezarta Islamaj Doğan, Robert Leaman, and Zhiyong Lu. 2014. NCBI disease corpus: A resource for disease name recognition and concept normalization. Journal of Biomedical Informatics, 47:1–10.

Rotem Dror, Segev Shlomov, and Roi Reichart. 2019. Deep Dominance - How to Properly Compare Deep Neural Models. In Proceedings of the 57th Conference of the Association for Computational Linguistics, ACL 2019, Florence, Italy, July 28- August 2, 2019, Volume 1: Long Papers, pages 2773–2785. Association for Computational Linguistics.

Jean-Baptiste Escudié, Bastien Rance, Georgia Malmut, Sherine Khater, Anita Burgun, Christophe Cellier, and Anne-Sophie Jannot. 2017. A novel data-driven workflow combining literature and electronic health records to estimate comorbidities burden for a specific disease: a case study on autoimmune comorbidities in patients with celiac disease. BMC medical informatics and decision making, 17(1):1–10.

Alexander Fritzler, Varvara Logacheva, and Maksim Kretov. 2019. Few-Shot Classification in Named Entity Recognition Task. In Proceedings of the 34th ACM/SIGAPP Symposium on Applied Computing, SAC ’19, page 993–1000, New York, NY, USA. Association for Computing Machinery.

Leo Gao, Stella Biderman, Sid Black, Laurence Golding, Travis Hoppe, Charles Foster, Jason Phang, Horace He, Anish Thite, Noa Nabeshima, et al. 2020. The pile: An 800gb dataset of diverse text for language modeling. arXiv preprint arXiv:2101.00027.

Álvaro García-Barragán, Alberto González Calatayud, Oswaldo Solarte-Pabón, Mariano Provencio, Ernestina Menasalvas, and Víctor Robles. 2024. GPT for medical entity recognition in Spanish. Multimedia Tools and Applications, pages 1–20.

Yao Ge, Yuting Guo, Sudeshna Das, Mohammed Ali Al-Garadi, and Abeed Sarker. 2023. Few-shot learning for medical text: A review of advances, trends, and opportunities. Journal of Biomedical Informatics, 144:104458.

Cyril Grouin, Sophie Rosset, Pierre Zweigenbaum, Karën Fort, Olivier Galibert, and Ludovic Quin-tard. 2011. Proposal for an Extension of Traditional Named Entities: From Guidelines to Evaluation, an Overview. In Proceedings of the 5th Linguistic Annotation Workshop, pages 92–100, Portland, Oregon, USA. Association for Computational Linguistics.

Shivanshu Gupta, Matt Gardner, and Sameer Singh. 2023. Coverage-based Example Selection for In-Context Learning. In Findings of the Association for Computational Linguistics: EMNLP 2023, pages 13924–13950, Singapore. Association for Computational Linguistics.

Christel Gérardin, Perceval Wajsbürt, Pascal Vaillant, Ali Bellamine, Fabrice Carrat, and Xavier Tannier. 2022. Multilabel classification of medical concepts for patient clinical profile identification. Artificial Intelligence in Medicine, 128:102311.

Tianyu Han, Lisa C Adams, Jens-Michalis Papaioannou, Paul Grundmann, Tom Oberhauser, Alexander Löser, Daniel Truhn, and Keno K Bressem. 2023. MedAlpaca–An Open-Source Collection of Medical Conversational AI Models and Training Data. arXiv preprint arXiv:2304.08247.

Yutai Hou, Wanxiang Che, Yongkui Lai, Zhihan Zhou, Yijia Liu, Han Liu, and Ting Liu. 2020. Few-shot Slot Tagging with Collapsed Dependency Transfer and Label-enhanced Task-adaptive Projection Network. In Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, pages 1381–1393, Online. Association for Computational Linguistics.

Niu Hu, Xuan Zhou, Bing Xu, Hanqing Liu, Xiangjin Xie, and Hai-Tao Zheng. 2023a. VPN: Variation on Prompt Tuning for Named-Entity Recognition. Applied Sciences, 13(14).

Yan Hu, Iqra Ameer, Xu Zuo, Xueqing Peng, Yujia Zhou, Zehan Li, Yiming Li, Jianfu Li, Xiaoqian Jiang, and Hua Xu. 2023b. Zero-shot clinical entity recognition using chatgpt. arXiv preprint arXiv:2303.16416.

Yan Hu, Qingyu Chen, Jingcheng Du, Xueqing Peng, Vipina Kuttichi Keloth, Xu Zuo, Yujia Zhou, Zehan Li, Xiaoqian Jiang, Zhiyong Lu, Kirk Roberts, and Hua Xu. 2024. Improving large language models for clinical named entity recognition via prompt engineering. Journal of the American Medical Informatics Association, page ocad259.

Jiaxin Huang, Chunyuan Li, Krishan Subudhi, Damien Jose, Shobana Balakrishnan, Weizhu Chen, Baolin Peng, Jianfeng Gao, and Jiawei Han. 2021a. Few-Shot Named Entity Recognition: An Empirical Baseline Study. In Proceedings of the 2021 Conference
on *Empirical Methods in Natural Language Processing*, pages 10408–10423, Online and Punta Cana, Dominican Republic. Association for Computational Linguistics.

Jiaxin Huang, Chunyuan Li, Krishan Subudhi, Damien Jose, Shobana Balakrishnan, Weizhu Chen, Baolin Peng, Jianfeng Gao, and Jiawei Han. 2021b. Few-Shot Named Entity Recognition: An Empirical Baseline Study. In *Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing*, pages 10408–10423, Online and Punta Cana, Dominican Republic. Association for Computational Linguistics.

Albert Q Jiang, Alexandre Sablayrolles, Arthur Mensch, Chris Bamford, Devendra Singh Chaplot, Diego de las Casas, Florian Bressand, Gianna Lengyel, Guillaume Lample, Lucile Saulnier, et al. 2023. Mistral 7B. *arXiv preprint arXiv:2310.06825*.

Bernal Jimenez Gutierrez, Nikolas McNeal, Clayton Washington, You Chen, Lang Li, Huan Sun, and Yu Su. 2022. Thinking about GPT-3 in-context learning for biomedical IE? think again. In *Findings of the Association for Computational Linguistics: EMNLP 2022*, pages 4497–4512, Abu Dhabi, United Arab Emirates. Association for Computational Linguistics.

Alistair EW Johnson, Tom J Pollard, Lu Shen, Li-wei H Lehman, Mengling Feng, Mohammad Ghassemi, Benjamin Moody, Peter Szolovits, Leo Anthony Celi, and Roger G Mark. 2016. MIMIC-III, a freely accessible critical care database. *Scientific data*, 3(1):1–9.

Mandar Joshi, Danqi Chen, Yinhan Liu, Daniel S. Weld, Luke Zettlemoyer, and Omer Levy. 2020. SpanBERT: Improving Pre-training by Representing and Predicting Spans.

Imed Keraghel, Stanislas Morbieu, and Mohamed Nadif. 2024. A survey on recent advances in named entity recognition.

Andrey Kormilitzin, Nemanja Vaci, Qiang Liu, and Alejo Nevado-Holgado. 2021. Med7: A transferable clinical natural language processing model for electronic health records. *Artificial Intelligence in Medicine*, 118:102086.

Yanis Labrak, Adrien Bazoge, Richard Dufour, Mickael Rouvier, Emmanuel Morin, Béatrice Daille, and Pierre-Antoine Gourraud. 2023. DrBERT: A robust pre-trained model in French for biomedical and clinical domains. In *Proceedings of the 61st Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers)*, pages 16207–16221, Toronto, Canada. Association for Computational Linguistics.

Yanis Labrak, Adrien Bazoge, Emmanuel Morin, Pierre-Antoine Gourraud, Mickael Rouvier, and Richard Dufour. 2024. BioMistral: A Collection of Open-Source Pretrained Large Language Models for Medical Domains.

Loïc Lannelongue, Jason Grealey, and Michael Inouye. 2021. Green algorithms: quantifying the carbon footprint of computation. *Advanced science*, 8(12):2100707.

Hugo Laurençon, Lucile Saulnier, Thomas Wang, Christopher Akiki, Albert Villanova del Moral, Teven Le Scao, Leandro Von Werra, Chenghao Mou, Eduardo González Ponferrada, Huu Nguyen, et al. 2022. The bigscience roots corpus: A 1.6 tb composite multilingual dataset. *Advances in Neural Information Processing Systems*, 35:31809–31826.

Amirhossein Layegh, Amir H Payberah, Ahmet Soylu, Dumitru Roman, and Mihhail Matskin. 2023. ContrastNER: Contrastive-based Prompt Tuning for Few-shot NER. *arXiv preprint arXiv:2305.17951*.

Hang Le, Loïc Vial, Jibril Frej, Vincent Segonne, Maximin Coavoux, Benjamin Lecouteux, Alexandre Alauzen, Benoit Crabbé, Laurent Besacier, and Didier Schwab. 2020. FlauBERT: Unsupervised language model pre-training for French. In *Proceedings of the Twelfth Language Resources and Evaluation Conference*, pages 2479–2490, Marseille, France. European Language Resources Association.

Robert Leaman, Ritu Khare, and Zhiyong Lu. 2015. Challenges in clinical natural language processing for automated disorder normalization. *Journal of Biomedical Informatics*, 57:28–37.

Dong-Ho Lee, Akshen Kadakia, Kangmin Tan, Mahak Agarwal, Xinyu Feng, Takashi Shibuya, Ryosuke Mitani, Toshiyuki Sekiya, Jay Pujara, and Xiang Ren. 2022. Good examples make a faster learner: Simple demonstration-based learning for low-resource NER. In *Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers)*, pages 2687–2700, Dublin, Ireland. Association for Computational Linguistics.

Jing Li, Aixin Sun, Jianglei Han, and Chenliang Li. 2022. A Survey on Deep Learning for Named Entity Recognition. *IEEE Trans. on Knowl. and Data Eng.*, 34(1):50–70.

Xiaoya Li, Jingrong Feng, Yuxian Meng, Qinghong Han, Fei Wu, and Jiwei Li. 2020. A unified MRC framework for named entity recognition. In *Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics*, pages 5849–5859, Online. Association for Computational Linguistics.

Baohao Liao, Yan Meng, and Christof Monz. 2023. Parameter-Efficient Fine-Tuning without Introducing New Latency. In *Proceedings of the 61st Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers)*, pages 4242–4260, Toronto, Canada. Association for Computational Linguistics.

Donald AB Lindberg, Betsy L Humphreys, and Alexa T McCray. 1993. The unified medical language system. *Yearbook of medical informatics*, 2(01):41–51.
Andy T. Liu, Wei Xiao, Henghui Zhu, Dejiao Zhang, Shang-Wen Li, and Andrew O. Arnold. 2022. QaNER: Prompting Question Answering Models for Few-shot Named Entity Recognition. ArXiv, abs/2203.01543.

Pengfei Liu, Weizhe Yuan, Jinlan Fu, Zhengbao Jiang, Hiroaki Hayashi, and Graham Neubig. 2023. Pre-train, prompt, and predict: A systematic survey of prompting methods in natural language processing. ACM Computing Surveys, 55(9):1-35.

Yang Liu. 2019. Fine-tune BERT for Extractive Summarization.

Yinhan Liu, Myle Ott, Naman Goyal, Jingfei Du, Mandar Joshi, Danqi Chen, Omer Levy, Mike Lewis, Luke Zettlemoyer, and Veselin Stoyanov. 2019. Roberta: A robustly optimized bert pretraining approach. arXiv preprint arXiv:1907.11692.

Yao Lu, Max Bartolo, Alastair Moore, Sebastian Riedel, and Pontus Stenetorp. 2022. Fantastically Ordered Prompts and Where to Find Them: Overcoming Few-Shot Prompt Order Sensitivity. In Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 8086-8098, Dublin, Ireland. Association for Computational Linguistics.

Yen-Fu Luo, Sam Henry, Yanshan Wang, Feichen Shen, Ozlem Uzuner, and Anna Rumshisky. 2020. The 2019 n2c2/UMass Lowell shared task on clinical concept normalization. Journal of the American Medical Informatics Association, 27(10):1529-e1.

Yen-Fu Luo, Weiyi Sun, and Anna Rumshisky. 2019. MCN: A comprehensive corpus for medical concept normalization. Journal of Biomedical Informatics, 92:103132.

Jie Ma, Miguel Ballesteros, Srikanth Doss, Rishita Anubhai, Sunil Mallya, Yaser Al-Onaizan, and Dan Roth. 2022a. Label Semantics for Few Shot Named Entity Recognition. In Findings of the Association for Computational Linguistics: ACL 2022, pages 1956-1971, Dublin, Ireland. Association for Computational Linguistics.

Ruotian Ma, Xin Zhou, Tao Gui, Yiding Tan, Linyang Li, Qi Zhang, and Xuanjing Huang. 2022b. Template-free prompt tuning for few-shot NER. In Proceedings of the 2022 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, pages 5721-5732, Seattle, United States. Association for Computational Linguistics.

Bernardo Magnini, Begona Altuna, Alberto Lavelli, Manuela Speranza, and Roberto Zanoli. 2021. The E3C Project: European Clinical Case Corpus. Language, 1(L2):L3.

Antoine Magron, Anna Dai, Mike Zhang, Syrielle Montariol, and Antoine Bosselut. 2024. JobSkape: A framework for generating synthetic job postings to

enhance skill matching. In Proceedings of the First Workshop on Natural Language Processing for Human Resources (NLP4HR 2024), pages 43-58, St. Julian's, Malta. Association for Computational Linguistics.

Louis Martin, Benjamin Muller, Pedro Javier Ortiz Suárez, Yoann Dupont, Laurent Romary, Éric de la Clergerie, Djamé Seddah, and Benoît Sagot. 2020. CamemBERT: a tasty French language model. In Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, pages 7203-7219, Online. Association for Computational Linguistics.

Alexa McCray, Anita Burgun, and Olivier Bodenreider. 2001. Aggregating UMLS Semantic Types for Reducing Conceptual Complexity. Studies in health technology and informatics, 84:216-20.

Simon Meoni, Eric De la Clergerie, and Theo Ryffel. 2023. Large Language Models as Instructors: A Study on Multilingual Clinical Entity Extraction. In The 22nd Workshop on Biomedical Natural Language Processing and BioNLP Shared Tasks, pages 178-190, Toronto, Canada. Association for Computational Linguistics.

Sewon Min, Xinxi Lyu, Ari Holtzman, Mikel Artetxe, Mike Lewis, Hannaneh Hajishirzi, and Luke Zettlemoyer. 2022. Rethinking the Role of Demonstrations: What Makes In-Context Learning Work? In Proceedings of the 2022 Conference on Empirical Methods in Natural Language Processing, pages 11048-11064, Abu Dhabi, United Arab Emirates. Association for Computational Linguistics.

Aurélie Névéol, Cyril Grouin, Jeremy Leixa, Sophie Rosset, and Pierre Zweigenbaum. 2014. The QUAERO French medical corpus: A resource for medical entity recognition and normalization. Proc of BioTextMining Work, pages 24-30.

Joel Nothman, Nicky Ringland, Will Radford, Tara Murphy, and James R. Curran. 2013. Learning multilingual named entity recognition from Wikipedia. Artificial Intelligence, 194:151-175. Artificial Intelligence, Wikipedia and Semi-Structured Resources.

Guilherme Penedo, Quentin Malartic, Daniel Hesslow, Ruxandra Cojocaru, Alessandro Cappelli, Hamza Alobeidli, Baptiste Pannier, Ebtesam Almazrouei, and Julien Launay. 2023. The RefinedWeb dataset for Falcon LLM: outperforming curated corpora with web data, and web data only. arXiv preprint arXiv:2306.01116.

Ethan Perez, Douwe Kiela, and Kyunghyun Cho. 2021. True Few-Shot Learning with Language Models. In Advances in Neural Information Processing Systems.

Fabio Petroni, Patrick Lewis, Aleksandra Piktus, Tim Rocktäschel, Yuxiang Wu, Alexander H. Miller, and Sebastian Riedel. 2020. How Context Affects Language Models' Factual Predictions.
Guanghui Qin and Jason Eisner. 2021. Learning how to ask: Querying LMs with mixtures of soft prompts. In *Proceedings of the 2021 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies*, pages 5203–5212, Online. Association for Computational Linguistics.

Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sasthy, Amanda Askell, Pamela Mishkin, Jack Clark, Gretchen Krueger, and Ilya Sutskever. 2021. Learning Transferable Visual Models From Natural Language Supervision. In *Proceedings of the 38th International Conference on Machine Learning*, volume 139 of *Proceedings of Machine Learning Research*, pages 8748–8763. PMLR.

Alec Radford, Jeffrey Wu, Rewon Child, David Luan, Dario Amodei, Ilya Sutskever, et al. 2019. Language models are unsupervised multitask learners. *OpenAI blog*, 1(8):9.

Nils Reimers and Iryna Gurevych. 2017. Reporting score distributions makes a difference: Performance study of LSTM-networks for sequence tagging. In *Proceedings of the 2017 Conference on Empirical Methods in Natural Language Processing*, pages 338–348, Copenhagen, Denmark. Association for Computational Linguistics.

Timo Schick and Hinrich Schütze. 2021. It's not just size that matters: Small language models are also few-shot learners. In *Proceedings of the 2021 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies*, pages 2339–2352, Online. Association for Computational Linguistics.

Stefan Schweter and Alan Akbik. 2021. FLERT: Document-Level Features for Named Entity Recognition.

Yongliang Shen, Zeqi Tan, Shuhui Wu, Wenqi Zhang, Rongsheng Zhang, Yadong Xi, Weiming Lu, and Yueting Zhuang. 2023. PromptNER: Prompt locating and typing for named entity recognition. In *Proceedings of the 61st Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers)*, pages 12492–12507, Toronto, Canada. Association for Computational Linguistics.

Pedro Javier Ortiz Suárez, Laurent Romary, and Benoît Sagot. 2020. A monolingual approach to contextualized word embeddings for mid-resource languages. *arXiv preprint arXiv:2006.06202*.

Mujeen Sung, Minbyul Jeong, Yonghwa Choi, Donghyeon Kim, Jinhyuk Lee, and Jaewoo Kang. 2022. BERN2: an advanced neural biomedical named entity recognition and normalization tool. *Bioinformatics*, 38(20):4837–4839.

Derek Tam, Rakesh R. Menon, Mohit Bansal, Shashank Srivastava, and Colin Raffel. 2021. Improving and Simplifying Pattern Exploiting Training. In *Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing*, pages 4980–4991, Online and Punta Cana, Dominican Republic. Association for Computational Linguistics.

Jörg Tiedemann. 2012. Parallel data, tools and interfaces in OPUS. In *Proceedings of the Eighth International Conference on Language Resources and Evaluation (LREC’12)*, pages 2214–2218, Istanbul, Turkey. European Language Resources Association (ELRA).

Erik F. Tjong Kim Sang. 2002. Introduction to the CoNLL-2002 shared task: Language-independent named entity recognition. In *COLING-02: The 6th Conference on Natural Language Learning 2002 (CoNLL-2002)*.

Erik F. Tjong Kim Sang and Fien De Meulder. 2003. Introduction to the CoNLL-2003 shared task: Language-independent named entity recognition. In *Proceedings of the Seventh Conference on Natural Language Learning at HLT-NAACL 2003*, pages 142–147.

Augustin Toma, Patrick R Lawler, Jimmy Ba, Rahul G Krishnan, Barry B Rubin, and Bo Wang. 2023. Clinical Camel: An Open-Source Expert-Level Medical Language Model with Dialogue-Based Knowledge Encoding. *arXiv preprint arXiv:2305.12031*.

Rian Touchent and Éric de la Clergerie. 2024. CamemBERT-bio: Leveraging continual pre-training for cost-effective models on French biomedical data. In *Proceedings of the 2024 Joint International Conference on Computational Linguistics, Language Resources and Evaluation (LREC-COLING 2024)*, pages 2692–2701, Torino, Italia. ELRA and ICCL.

Hugo Touvron, Louis Martin, Kevin Stone, Peter Albert, Amjad Almahairi, Yasmine Babaei, Nikolay Bashlykov, Soumya Batra, Prajjwal Bhargava, Shruti Bhosale, et al. 2023. Llama 2: Open foundation and fine-tuned chat models. *arXiv preprint arXiv:2307.09288*.

Özlem Uzuner, Brett R South, Shuying Shen, and Scott L DuVall. 2011. 2010 i2b2/VA challenge on concepts, assertions, and relations in clinical text. *Journal of the American Medical Informatics Association*, 18(5):552–556.

Charangan Vasantharajan, Kyaw Zin Tun, Ho Thi-Nga, Sparsh Jain, Tong Rong, and Chng Eng Siong. 2022. MedBERT: A Pre-trained Language Model for Biomedical Named Entity Recognition. In *2022 Asia-Pacific Signal and Information Processing Association Annual Summit and Conference (APSIPA ASC)*, pages 1482–1488.

David Vilar, Markus Freitag, Colin Cherry, Jiaming Luo, Viresh Ratnakar, and George Foster. 2023. Prompting PaLM for translation: Assessing strategies and performance. In *Proceedings of the 61st Annual Conference on Empirical Methods in Natural Language Processing*, pages 4980–4991, Online and Punta Cana, Dominican Republic. Association for Computational Linguistics.
*Meeting of the Association for Computational Linguistics (Volume 1: Long Papers)*, pages 15406–15427, Toronto, Canada. Association for Computational Linguistics.

Perceval Wajsbürt. 2021. *Extraction and normalization of simple and structured entities in medical documents*. Theses, Sorbonne Université.

Perceval Wajsbürt, Arnaud Sarfati, and Xavier Tannier. 2021. Medical concept normalization in French using multilingual terminologies and contextual embeddings. *Journal of Biomedical Informatics*, 114:103684.

Ben Wang and Aran Komatsuzaki. 2021. GPT-J-6B: A 6 Billion Parameter Autoregressive Language Model. https://github.com/kingoflolz/mesh-transformer-jax.

Guangyu Wang, Xiaohong Liu, Zhen Ying, Guoxing Yang, Zhiwei Chen, Zhiwen Liu, Min Zhang, Hongmei Yan, Yuxing Lu, Yuanxu Gao, et al. 2023a. Optimized glycemic control of type 2 diabetes with reinforcement learning: a proof-of-concept trial. *Nature Medicine*, pages 1–10.

Shuhe Wang, Xiaofei Sun, Xiaoya Li, Rongbin Ouyang, Fei Wu, Tianwei Zhang, Jiwei Li, and Guoyin Wang. 2023b. GPT-NER: Named Entity Recognition via Large Language Models.

Yanshan Wang, Liwei Wang, Majid Rastegar-Mojarad, Sungrim Moon, Feichen Shen, Naveed Afzal, Sijia Liu, Yuqun Zeng, Saeed Mehrabi, Sunghwan Sohn, and Hongfang Liu. 2018. Clinical information extraction applications: A literature review. *Journal of Biomedical Informatics*, 77:34–49.

Jason Wei, Xuezhi Wang, Dale Schuurmans, Maarten Bosma, Fei Xia, Ed Chi, Quoc V Le, Denny Zhou, et al. 2022. Chain-of-thought prompting elicits reasoning in large language models. *Advances in Neural Information Processing Systems*, 35:24824–24837.

BigScience Workshop, Teven Le Scao, Angela Fan, Christopher Akiki, Ellie Pavlick, Suzana Ilić, Daniel Hesslow, Roman Castagné, Alexandra Sasha Lucioni, François Yvon, et al. 2022. Bloom: A 176b-parameter open-access multilingual language model. *arXiv preprint arXiv:2211.05100*.

Jingye Yang, Cong Liu, Wendy Deng, Da Wu, Chunhua Weng, Yunyun Zhou, and Kai Wang. 2024. Enhancing phenotype recognition in clinical notes using large language models: PhenoBCBERT and PhenoGPT. *Patterns*, 5(1):100887.

Yi Yang and Arzoo Katiyar. 2020. Simple and Effective Few-Shot Named Entity Recognition with Structured Nearest Neighbor Learning. In *Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP)*, pages 6365–6375, Online. Association for Computational Linguistics.

Feiyang Ye, Liang Huang, Senjie Liang, and KaiKai Chi. 2023. Decomposed Two-Stage Prompt Learning for Few-Shot Named Entity Recognition. *Information*, 14(5).

Jamil Zaghir, Marco Naguib, Mina Bjelogrlic, Aurélie Névéol, Xavier Tannier, and Christian Lovis. 2024. Prompt engineering paradigms for medical applications: Scoping review. *J Med Internet Res*, 26:e60501.

Susan Zhang, Stephen Roller, Naman Goyal, Mikel Artetxe, Moya Chen, Shuohui Chen, Christopher Dewan, Mona Diab, Xian Li, Xi Victoria Lin, et al. 2022. Opt: Open pre-trained transformer language models. *arXiv preprint arXiv:2205.01068*.

Zihao Zhao, Eric Wallace, Shi Feng, Dan Klein, and Sameer Singh. 2021. Calibrate before use: Improving few-shot performance of language models. In *International Conference on Machine Learning*, pages 12697–12706. PMLR.

Lianmin Zheng, Wei-Lin Chiang, Ying Sheng, Siyuan Zhuang, Zhanghao Wu, Yonghao Zhuang, Zi Lin, Zhuohan Li, Dacheng Li, Eric Xing, et al. 2023. Judging LLM-as-a-judge with MT-Bench and Chatbot Arena. *arXiv preprint arXiv:2306.05685*.

Yukun Zhu, Ryan Kiros, Rich Zemel, Ruslan Salakhutdinov, Raquel Urtasun, Antonio Torralba, and Sanja Fidler. 2015. Aligning books and movies: Towards story-like visual explanations by watching movies and reading books. In *Proceedings of the IEEE international conference on computer vision*, pages 19–27.
# A Appendix

## A.1 Ablation

To better understand the contribution of each step of our approach, we carried out a series of complementary experiments.

### A.1.1 Sample and sample size

We tested our approach with different samples and different sample sizes for one MLM: XLM-RoBERTa-large, and one prompted LLM: Mistral-7B. The results are reported in table 2. It can be noted that, whereas the standard deviation with respect to *p* is rather high, a significant difference can still be consistently observed between the two models across samples of the same size. We also observe that, as the number of annotated instances decreases, the performance of the MLM drops faster than that of the prompted LLM.

<table><caption>Table 2: F1 scores obtained over experiments with different training samples and different training sample sizes.</caption><thead><tr><th></th><th colspan="3">CoNLL2003</th><th colspan="3">n2c2</th></tr><tr><th></th><th colspan="6">100 annotated instances</th></tr><tr><th></th><th>p=1</th><th>p=2</th><th>p=3</th><th>p=1</th><th>p=2</th><th>p=3</th></tr></thead><tbody><tr><th>Mistral-7B</th><td>0.646</td><td>0.626</td><td>0.714</td><td>0.291</td><td>0.178</td><td>0.215</td></tr><tr><th>XLM-R-large</th><td>0.826</td><td>0.814</td><td>0.786</td><td>0.462</td><td>0.478</td><td>0.526</td></tr><tr><th></th><th colspan="6">50 annotated instances</th></tr><tr><th></th><th>p=1</th><th>p=2</th><th>p=3</th><th>p=1</th><th>p=2</th><th>p=3</th></tr><tr><th>Mistral-7B</th><td>0.615</td><td>0.648</td><td>0.637</td><td>0.278</td><td>0.176</td><td>0.106</td></tr><tr><th>XLM-R-large</th><td>0.697</td><td>0.77</td><td>0.714</td><td>0.431</td><td>0.476</td><td>0.35</td></tr><tr><th></th><th colspan="6">25 annotated instances</th></tr><tr><th></th><th>p=1</th><th>p=2</th><th>p=3</th><th>p=1</th><th>p=2</th><th>p=3</th></tr><tr><th>Mistral-7B</th><td>0.509</td><td>0.599</td><td>0.52</td><td>0.152</td><td>0.252</td><td>0.116</td></tr><tr><th>XLM-R-large</th><td>0.487</td><td>0.588</td><td>0.637</td><td>0.393</td><td>0.361</td><td>0.283</td></tr></tbody></table>

### A.1.2 Listing prompts

In this section, we compare the adopted tagging prompts to listing prompts. In listing prompts, demonstrations simply list the tagged mentions. The list separator is optimized (in the same way as the taggers) between a comma and a newline character. Eventually, the introductory sentences asks to list entities. The results shown in table 3 further corroborate our choice of only focusing on tagging prompts.

### A.1.3 Number of demonstrations

The decision to limit the number of annotated examples presented to generative models in the prompt to a maximum of 10 is dictated by two constraints.

Firstly, models impose a limit on the number of input tokens over which attention is calculated. Most models used have a limit of 2048 tokens, but more permissive models, such as Mistral-7B, allow up to 8096 tokens. This constraint translates into a limit on the number of sentences that can be presented in the prompt, ranging from 40 to 50 for Mistral-7B and from 10 to 15 for less permissive models, depending on the corpora and tokenizers. For instance, consider the task of detecting body parts in the French section of E3C. Using Mistral-7B (and its tokenizer), the practical limit is around 40 examples, resulting in an average prompt of 7779.5 tokens. Using Vicuna-13B (and its tokenizer), the limit is around 11 examples, resulting in an average of 1851.5 tokens in the prompt.

Secondly, the improvement brought by adding more examples does not appear to be significant, as observed in Table 4, which shows the results obtained with Mistral-7B when the number of annotated examples is tripled. It is noteworthy that the marginal improvements achieved by tripling the number of examples come at a considerable cost, especially given the quadratic complexity with respect to the length of the prompt.

Therefore, we choose to limit the prompt to 5-10 examples, selected based on the chosen criteria (TF-IDF proximity to the reference sentence or the number of entities present). Instead of selecting examples independently, Gupta et al. (2023) propose selecting them interdependently to improve the representativeness of the prompt. This approach would be interesting to implement in our system in future work.

### A.1.4 Hyperparameter grid search

In order to assess the quality of our adopted search method used to find the best feature combination to incorporate in the prompt, we compare this method to a naïve grid search over these features. We test all 512 combinations of our identified 9 features, for Mistral-7B over ConLL2003. The scores found through LOOCV vary between 0.0 and 0.656 with a mean value of 0.387 and a median of 0.46. The best-preforming combination is: *Additional sentences, Self-verification, Introductory sentence for the test instance and Require a long answer for the self-verification*, which is exactly the same combination we found initially through a greedy, tree search, that is around 20 times faster and less consuming.
<table><caption>Table 3: F1 scores obtained with the listing and tagging prompts.</caption><thead><tr><th rowspan="2">Model</th><th colspan="5">English</th><th colspan="4">French</th><th colspan="4">Spanish</th></tr><tr><th>WikiNER</th><th>CoNLL2003</th><th>E3C</th><th>n2c2</th><th>NCBI</th><th>WikiNER</th><th>QFP</th><th>E3C</th><th>EMEA</th><th>MEDLINE</th><th>WikiNER</th><th>CoNLL2002</th><th>E3C</th><th>CWL</th></tr></thead><tbody><tr><td colspan="16"><em>Listing prompts</em></td></tr><tr><td>Mistral-7B</td><td>0.659</td><td>0.533</td><td>0.417</td><td>0.281</td><td>0.340</td><td>0.676</td><td>0.083</td><td>0.451</td><td>0.169</td><td>0.403</td><td>0.697</td><td>0.620</td><td>0.211</td><td>0.273</td></tr><tr><td colspan="16"><em>Tagging prompts</em></td></tr><tr><td>Mistral-7B</td><td>0.754</td><td>0.646</td><td>0.488</td><td>0.291</td><td>0.395</td><td>0.727</td><td>0.428</td><td>0.590</td><td>0.229</td><td>0.333</td><td>0.720</td><td>0.707</td><td>0.083</td><td>0.374</td></tr></tbody></table>

<table><caption>Table 4: F1 scores obtained with 5/10 demonstrations vs. with 15/30 demonstrations</caption><thead><tr><th rowspan="2">Model</th><th colspan="5">English</th><th colspan="4">French</th><th colspan="4">Spanish</th></tr><tr><th>WikiNER</th><th>CoNLL2003</th><th>E3C</th><th>n2c2</th><th>NCBI</th><th>WikiNER</th><th>QFP</th><th>E3C</th><th>EMEA</th><th>MEDLINE</th><th>WikiNER</th><th>CoNLL2002</th><th>E3C</th><th>CWL</th></tr></thead><tbody><tr><td colspan="16"><em>5/10 demonstrations</em></td></tr><tr><td>Mistral-7B</td><td>0.754</td><td>0.646</td><td>0.488</td><td>0.291</td><td>0.395</td><td>0.727</td><td>0.428</td><td>0.590</td><td>0.229</td><td>0.333</td><td>0.720</td><td>0.707</td><td>0.083</td><td>0.374</td></tr><tr><td colspan="16"><em>15/30 demonstrations</em></td></tr><tr><td>Mistral-7B</td><td>0.763</td><td>0.692</td><td>0.453</td><td>0.263</td><td>0.377</td><td>0.782</td><td>0.355</td><td>0.587</td><td>0.237</td><td>0.396</td><td>0.785</td><td>0.751</td><td>0.163</td><td>0.413</td></tr></tbody></table>

## A.2 Evaluated models

Table 5 specifies relevant information about the tested models. In the training corpus column, we point out the data contaminations we know of. For instance, MedBERT's documentation explicitly mentions N2C2 as part of its training data. This might lead to artificially high evaluation metrics, as the model is not generalizing to unseen data, but instead leveraging pre-learned information, which compromises the validity of the results in real-world applications.

## A.3 NER labels descriptions

Tables 6 to 10 definitions of all the labels present in the studied datasets. These definitions were drawn from available annotation schemes and further curated by the authors and native speakers. These definitions were included in the prompt when the "Include label definitions in the prompt" feature was activated.

## A.4 Carbon footprint

Tables 11 and 12 detail the carbon emission estimations for all of our experiments. These estimations were computed with GreenAlgorithms v2.2 (Langelongue et al., 2021)<fnref n="5" />, based on factors such as runtime, computing hardware and location where electricity used by our computer facility was produced. In total, the experiments described in this paper are estimated to have generated around 31kg of CO2 equivalent (29kg for the main experiments, and 2kg for ablation).

<footnote n="5">⁵http:// calculator.green-algorithms.org/</footnote>
<table><caption>Table 5: Characterization of the language models used in our experiments in terms of parameters and training corpus. Models marked with [en] (respectively [fr], [es]) are heavily trained on English (respectively French, Spanish). CLMs marked with * are fine-tuned versions of other CLMs.</caption><thead><tr><th>#</th><th>Model</th><th>Number of parameters</th><th>Training data size</th><th>Training corpus</th></tr></thead><tbody><tr><td>1</td><td>Llama-2-70B<sup>[en]</sup> (Touvron et al., 2023)</td><td>70B</td><td>2 trillion tokens</td><td>A mix of publicly available online data, mainly in English</td></tr><tr><td>2</td><td>Llama-3-8B-Instruct<sup>[en]</sup></td><td>8B</td><td>over 15 trillion tokens</td><td>"A new mix" of publicly available online data, mainly in English</td></tr><tr><td>3</td><td>Mistral-7B<sup>[?]</sup> (Jiang et al., 2023)</td><td>7B</td><td>Undisclosed</td><td>Undisclosed</td></tr><tr><td>4</td><td>Phi-3-medium-instruct<sup>[en]</sup> (Abdin et al., 2024)</td><td>14B</td><td>4.8 trillion tokens</td><td>A combination of publicly available corpora, synthetic data and chat format supervised data, mainly in English</td></tr><tr><td>5</td><td>BLOOM-7B<sup>[en][fr][es]</sup> (Workshop et al., 2022)</td><td>7B</td><td>1.6 TB</td><td>ROOTS (Laurençon et al., 2022), a mix of datasets and pseudo-crawled data 59 languages</td></tr><tr><td>6</td><td>Falcon-40B<sup>[en][fr][es]</sup></td><td>40B</td><td>1 trillion tokens</td><td>RefinedWeb (Penedo et al., 2023), a dataset of filtered and deduplicated web data</td></tr><tr><td>7</td><td>GPT-J-6B<sup>[en]</sup> (Wang and Komatsuzaki, 2021)</td><td>6B</td><td>825 GiB</td><td>The Pile (Gao et al., 2020), a mixture of public datasets and web data in English</td></tr><tr><td>8</td><td>OPT-66B<sup>[en]</sup> (Zhang et al., 2022)</td><td>66B</td><td>180 billion tokens</td><td>Crawled data from the web, mainly in English</td></tr><tr><td>9</td><td>Vicuna-13B<sup>[en]*</sup> (Zheng et al., 2023)</td><td>13B</td><td>125K conversations</td><td>Llama 2, fine-tuned on conversations collected from ShareGPT.com, mainly in English</td></tr><tr><td>10</td><td>Vicuna-7B<sup>[en]*</sup> (Zheng et al., 2023)</td><td>7B</td><td>125K conversations</td><td>Llama 2, fine-tuned on conversations collected from ShareGPT.com, mainly in English</td></tr><tr><td>11</td><td>BioMistral-7B<sup>[en]*</sup> (Labrak et al., 2024)</td><td>7B</td><td>3 billion tokens</td><td>Mistral, fine-tuned on the PMC Open Access Subset</td></tr><tr><td>12</td><td>Medalpaca-7B<sup>[en]*</sup> (Han et al., 2023)</td><td>7B</td><td>400K Q.A. pairs</td><td>Llama 2, fine-tuned on semi-generated medical question-answer pairs in English</td></tr><tr><td>13</td><td>Vigogne-13B<sup>[fr][en]*</sup></td><td>13B</td><td>52K instructions</td><td>Llama 2, fine-tuned on English instructions automatically translated to French</td></tr><tr><td>14</td><td>mBERT<sup>[en][fr][es]</sup> (Devlin et al., 2019)</td><td>110M</td><td>Undisclosed</td><td>A corpus featuring 104 languages built from undisclosed sources</td></tr><tr><td>15</td><td>XLM-R-large<sup>[en][fr][es]</sup> (Conneau et al., 2020)</td><td>355M</td><td>2.5 TB</td><td>Filtered CommonCrawl data containing 100 languages</td></tr><tr><td>16</td><td>BERT-large<sup>[en]</sup> (Devlin et al., 2019)</td><td>345M</td><td>3.3 billion words</td><td>BookCorpus (Zhu et al., 2015), a dataset consisting of unpublished books and English Wikipedia.</td></tr><tr><td>17</td><td>RoBERTa-large<sup>[en]</sup> (Liu et al., 2019)</td><td>355M</td><td>160 GiB</td><td>BooksCorpus (Zhu et al., 2015), English Wikipedia, and crawled web data</td></tr><tr><td>18</td><td>Bio_ClinicalBERT<sup>[en]</sup> (Alsentzer et al., 2019)</td><td>110M</td><td>2 million clinical notes</td><td>MIMIC-III (Johnson et al., 2016), a database containing electronic health records from hospitalized ICU patients</td></tr><tr><td>19</td><td>ClinicalBERT<sup>[en]</sup> (Wang et al., 2023a)</td><td>110M</td><td>1.2 billion words</td><td>A large multi-center dataset with a corpus built from undisclosed sources</td></tr><tr><td>20</td><td>MedBERT<sup>[en]</sup> (Vasantharajan et al., 2022)</td><td>110M</td><td>57 million words</td><td>Community datasets (including N2C2 (Luo et al., 2020)) and Crawled medical-related articles from Wikipedia</td></tr><tr><td>21</td><td>CamemBERT-large<sup>[fr]</sup> (Martin et al., 2020)</td><td>335M</td><td>64 billion tokens</td><td>OSCAR (Suárez et al., 2020), a corpus of web data in French</td></tr><tr><td>22</td><td>FlaubERT-large<sup>[fr]</sup> (Le et al., 2020)</td><td>335M</td><td>13 billion tokens</td><td>A mix of French Wikipedia, French books, and French web data</td></tr><tr><td>23</td><td>DrBERT-4GB<sup>[fr]</sup> (Labrak et al., 2023)</td><td>110M</td><td>1 billion words</td><td>A mix of publicly available biomedical corpora in French (including QuaeroFrenchMed (Névéol et al., 2014)).</td></tr><tr><td>24</td><td>CamemBERT-bio<sup>[fr]</sup> (Touchent and de la Clergerie, 2024)</td><td>110M</td><td>413 million words</td><td>A mix of publicly available biomedical corpora in French (including E3C (Magnini et al., 2021)).</td></tr><tr><td>25</td><td>BETO<sup>[es]</sup> (Cañete et al., 2020)</td><td>110M</td><td>3 billion words</td><td>Spanish Wikipedia and Spanish data from OPUS (Tiedemann, 2012)</td></tr><tr><td>26</td><td>PatanaBERT<sup>[es]</sup></td><td>110M</td><td>Undisclosed</td><td>Spanish</td></tr><tr><td>27</td><td>TulioBERT<sup>[es]</sup></td><td>110M</td><td>Undisclosed</td><td>Spanish</td></tr><tr><td>28</td><td>BSC-BioEHR<sup>[es]</sup> (Carrino et al., 2022)</td><td>110M</td><td>1.1 billion tokens</td><td>A mixture of biomedical community datasets including EHR documents and crawled data in Spanish</td></tr><tr><td>29</td><td>BSC-Bio<sup>[es]</sup> (Carrino et al., 2022)</td><td>110M</td><td>963 million tokens</td><td>A mixture of biomedical community datasets and crawled data in Spanish</td></tr></tbody></table>
<table><caption>Table 6: Description of the NER tags used in our experiments for English.</caption><thead><tr><td>Tag</td><td>Tag name (in singular)</td><td>Description</td></tr></thead><tbody><tr><td>PER</td><td>person names (a person's name)</td><td>These are names of persons such as real people or fictional characters.</td></tr><tr><td>FAC</td><td>facilities (a facility)</td><td>These are names of man-made structures such as infrastructure, buildings and monuments.</td></tr><tr><td>LOC</td><td>locations (a location)</td><td>These are names of geographical locations such as landmarks, cities, countries and regions.</td></tr><tr><td>ORG</td><td>organizations (an organization)</td><td>These are names of organizations such as companies, agencies and political parties.</td></tr><tr><td>FUNC</td><td>functions and jobs (a function or a job)</td><td>These are words that refer to a profession or a job.</td></tr><tr><td>ACTI</td><td>activities and behaviors (an activity or behavior)</td><td>These are words that refer to human activities, behaviors or events as well as governmental or regulatory activities.</td></tr><tr><td>ANAT</td><td>anatomy (an anatomy)</td><td>These are words that refer to the structure of the human body, its organs and their position, such as body parts or organs, systems, tissues, cells, body substances and embryonic structures.</td></tr><tr><td>CHEM</td><td>chemicals and drugs (a chemical or a drug)</td><td>These are words that refer to a substance or composition that has a chemical characteristic, especially a curative or preventive property with regard to human or animal diseases, such as drugs, antibiotics, proteins, hormones, enzymes and hazardous or poisonous substances.</td></tr><tr><td>CONC</td><td>concepts and ideas (a concept or an idea)</td><td>These are words that refer to a concept or an idea, such as a classification, an intellectual product, a language, a law or a regulation.</td></tr><tr><td>DEVI</td><td>medical devices (a device)</td><td>These are words that refer to a medical device used to administer care or perform medical research.</td></tr><tr><td>DISO</td><td>disorders (a disorder)</td><td>These are words that refer to an alteration of morphology, function or health of a living organism, animal or plant, such as congenital abnormalities, dysfunction, injuries, signs or symptoms or observations.</td></tr><tr><td>GENE</td><td>genes and molecular sequences (a gene or a molecular sequence)</td><td>These are words that refer to a gene, a genome or a molecular sequence.</td></tr><tr><td>GEOG</td><td>geographical areas (a geographical area)</td><td>These are words that refer to a country, a region or a city.</td></tr><tr><td>LIVB</td><td>living beings (a living being)</td><td>These are words that refer to a living being or a group of living beings, such as a person or a group of persons, a plant or a category of plants, an animal or a category of animals.</td></tr><tr><td>OBJC</td><td>objects (an object)</td><td>These are words that refer to anything animate or inanimate that affects the senses, such as physical manufactured objects.</td></tr><tr><td>OCCU</td><td>occupations (an occupation)</td><td>These are words that refer to a professional occupation or discipline.</td></tr><tr><td>ORGA</td><td>organizations (an organization)</td><td>These are words that refer to an organization such as healthcare related organizations.</td></tr><tr><td>PHEN</td><td>phenomena (a phenomenon)</td><td>These are words that refer to a phenomenon that occurs naturally or as a result of an activity, such as a biologic function.</td></tr><tr><td>PHYS</td><td>physiology (a physiology)</td><td>These are words that refer to any element that contributes to the mechanical, physical and biochemical functioning or organization of living organisms and their components.</td></tr><tr><td>PROC</td><td>procedures (a procedure)</td><td>These are words that refer to an activity or a procedure that contributes to the diagnosis or treatment of patients, the information of patients, the training of medical personnel or biomedical research.</td></tr><tr><td>EVENT</td><td>events (an event)</td><td>These are words that refer to actions, states, and circumstances that are relevant to the clinical history of a patient such as pathologies and symptoms, or more generally words like "enters", "reports" or "continue".</td></tr></tbody></table>
<table><caption>Table 7: Description of the NER tags used in our experiments for English, continued.</caption><thead><tr><td>Tag</td><td>Tag name (in singular)</td><td>Description</td></tr></thead><tbody><tr><td>TIMEX3</td><td>time expressions (a time expression)</td><td>These are time expressions such as dates, times, durations, frequencies, or intervals.</td></tr><tr><td>RML</td><td>results and measurements (a result or a measurement)</td><td>These are test results, results of laboratory analyses, formulaic measurements, and measure values.</td></tr><tr><td>ACTOR</td><td>actors (an actor)</td><td>These are words that refer patients, healthcare professionals, or other actors relevant to the clinical history of a patient.</td></tr><tr><td>Abbreviation</td><td>abbreviations (an abbreviation)</td><td>These are words that refer to abbreviations.</td></tr><tr><td>Body Part</td><td>body parts (a body part)</td><td>These are words that refer to organs and anatomical parts of persons.</td></tr><tr><td>Clinical Finding</td><td>clinical findings (a clinical finding)</td><td>These are words that refer to observations, judgments or evaluations made about patients.</td></tr><tr><td>Diagnostic Procedure</td><td>diagnostic procedures (a diagnostic procedure)</td><td>These are words that refer to tests that allow determining the condition of the individual.</td></tr><tr><td>Disease</td><td>diseases (a disease)</td><td>These are words that describe an alteration of the physiological state in one or several parts of the body, due to generally known causes, manifested by characteristic symptoms and signs, and whose evolution is more or less predictable.</td></tr><tr><td>Family Member</td><td>family members (a family member)</td><td>These are words that refer to family members.</td></tr><tr><td>Laboratory or Test Result</td><td>laboratory or test results (a laboratory or test result)</td><td>These are words that refer to any measurement or evaluation obtained from a diagnostic support examination.</td></tr><tr><td>Laboratory Procedure</td><td>laboratory procedures (a laboratory procedure)</td><td>These are words that refer to tests that are performed on various patient samples that allow diagnosing diseases by detecting biomarkers and other parameters. Blood, urine, and other fluids and tissues that use biochemical, microbiological and/or cytological methods are considered.</td></tr><tr><td>Medication</td><td>medications (a medication)</td><td>These are words that refer to medications or drugs used in the treatment and/or prevention of diseases, including brand names and generics, as well as names for groups of medications.</td></tr><tr><td>Procedure</td><td>procedures (a procedure)</td><td>These are words that refer to activities derived from the care and care of patients.</td></tr><tr><td>Sign or Symptom</td><td>signs or symptoms (a sign or symptom)</td><td>These are words that refer to manifestations of a disease, determined by medical examination or perceived and expressed by the patient.</td></tr><tr><td>Therapeutic Procedure</td><td>therapeutic procedures (a therapeutic procedure)</td><td>These are words that refer to activities or treatments that are used to prevent, repair, eliminate or cure the individual's disease.</td></tr><tr><td>Composite Mention</td><td>composite mentions of diseases (a composite mention of diseases)</td><td>These are words that refer to mentions of multiple diseases, such as "colorectal, endometrial, and ovarian cancers".</td></tr><tr><td>Disease Class</td><td>disease classes (a disease class)</td><td>These are words that refer to classes of diseases, such as "an autosomal recessive disease".</td></tr><tr><td>Modifier</td><td>modifiers (a modifier of diseases)</td><td>These are words that refer to modifiers of diseases, such as "primary" or "C7-deficient".</td></tr><tr><td>Specific Disease</td><td>diseases (a disease)</td><td>These are words that refer to specific diseases, such as "diastrophic dysplasia".</td></tr></tbody></table>
<table><caption>Table 8: Description of the NER tags used in our experiments for French.</caption><thead><tr><td>Tag</td><td>Tag name (in singular)</td><td>Description</td></tr></thead><tbody><tr><td>PER</td><td>de noms de personnes (un nom de personne)</td><td>Il s'agit des noms de personnes, qu'elles soient réelles ou fictives.</td></tr><tr><td>FAC</td><td>de productions humaines (une production humaine)</td><td>Il s'agit des noms de structures faites par les humains comme des infrastructures, des bâtiments ou des monuments.</td></tr><tr><td>LOC</td><td>de lieux (un lieu)</td><td>Il s'agit des noms de lieux comme des endroits, villes, pays ou régions.</td></tr><tr><td>ORG</td><td>d'organisations (une organisation)</td><td>Il s'agit des noms d'organisations comme des entreprises, des agences ou des partis politiques.</td></tr><tr><td>FUNC</td><td>de fonctions et métiers (une fonction ou un métier)</td><td>Il s'agit de mots qui se rapportent à une activité professionnelle.</td></tr><tr><td>ANAT</td><td>d'anatomie (une partie du corps)</td><td>Il s'agit d'une entité se rapportant à la structure du corps humain, ses organes et leur position. Il s'agit principalement des parties du corpus ou organes, des appareils, des tissus, des cellules, des substances corporelles et des organismes embryonaires.</td></tr><tr><td>CHEM</td><td>de médicaments et substances chimiques (un médicament ou une substance chimique)</td><td>Il s'agit d'une substance ou composition présentant des propriétés chimiques caractéristiques, en particulier des propriétés curatives ou préventives à l'égard des maladies humaines ou animales. Il s'agit principalement des médicaments disponibles en pharmacie, des antibiotiques, des proteines, des hormones, des substances dangereuses, des enzymes.</td></tr><tr><td>DEVI</td><td>de matériel (un matériel)</td><td>Il s'agit d'un matériel utilisé pour administrer des soins ou effectuer des recherches médicales.</td></tr><tr><td>DISO</td><td>de problèmes médicaux (un problème médical)</td><td>Il s'agit d'une altération de la morphologie, des fonctions, ou de la santé d'un organisme vivant, animal ou végétal. Il peut s'agir de malformations, de maladies, de blessure, de signe ou symptome ou d'une observation.</td></tr><tr><td>GEOG</td><td>de zones géographiques (une zone géographique)</td><td>Il s'agit d'un pays, une région, ou une ville.</td></tr><tr><td>LIVB</td><td>d'êtres vivants (un être vivant)</td><td>Il s'agit d'un être vivant ou groupe d'êtres vivants. Il peut s'agir d'une personne ou d'un groupe de personnes, d'une plante ou d'une catégorie de végétaux, d'un animal ou d'une catégorie d'animaux.</td></tr><tr><td>OBJC</td><td>d'objets (un objet)</td><td>Il s'agit de tout ce qui, animé ou inanimé, affecte les sens. Ici, il s'agit principalement d'objets physiques manufacturés.</td></tr><tr><td>PHEN</td><td>de phénomènes (un phénomène)</td><td>Il s'agit d'un phénomène qui se produit naturellement ou à la suite d'une activité. Il s'agit principalement de fonctions biologiques.</td></tr><tr><td>PHYS</td><td>de physiologie (une physiologie)</td><td>Il s'agit de tout élément contribuant au fonctionnement ou à l'organisation mécanique, physique et biochimique des organismes vivants et de leurs composants.</td></tr><tr><td>PROC</td><td>de procédures (une procédure)</td><td>Il s'agit d'une activité ou procédure contribuant au diagnostic ou au traitement des patients, à l'information des patients, la formation du personnel médical ou à la recherche biomédicale.</td></tr><tr><td>EVENT</td><td>d'événements (un événement)</td><td>Il s'agit d'une action, d'un état ou d'une circonstance qui est pertinent pour l'histoire clinique d'un patient. Il peut s'agir de pathologies et symptômes, ou plus généralement de mots comme "entre", "rapporte" ou "continue".</td></tr><tr><td>TIMEX3</td><td>d'expressions temporelles (une expression temporelle)</td><td>Il s'agit d'expressions temporelles comme des dates, heures, durées, fréquences, ou intervalles.</td></tr><tr><td>RML</td><td>de résultats et mesures (un résultat ou une mesure)</td><td>Il s'agit de résultats d'analyses de laboratoire, de mesures formelles, et de valeurs de mesure.</td></tr><tr><td>ACTOR</td><td>d'acteurs (un acteur)</td><td>Il s'agit de patients, de professionnels de santé, ou d'autres acteurs pertinents pour l'histoire clinique d'un patient.</td></tr></tbody></table>
<table><caption>Table 9: Description of the NER tags used in our experiments for Spanish.</caption><thead><tr><td>Tag</td><td>Tag name (in singular)</td><td>Description</td></tr></thead><tbody><tr><td>PER</td><td>nombres de personas (un nombre de persona)</td><td>Estos son nombres de personas, ya sean reales o personajes ficticios.</td></tr><tr><td>FAC</td><td>instalaciones (una instalación)</td><td>Estos son nombres de estructuras hechas por el hombre como infraestructura, edificios y monumentos.</td></tr><tr><td>LOC</td><td>lugares (un lugar)</td><td>Estos son nombres de ubicaciones geográficas como hitos, ciudades, países y regiones.</td></tr><tr><td>ORG</td><td>organizaciones (una organización)</td><td>Estos son nombres de organizaciones como empresas, agencias y partidos políticos.</td></tr><tr><td>ACTI</td><td>actividades y comportamientos (una actividad o comportamiento)</td><td>Estos son palabras que se refieren a actividades humanas, comportamientos o eventos, así como actividades gubernamentales o regulatorias.</td></tr><tr><td>ANAT</td><td>anatomía (una anatomía)</td><td>Estos son palabras que se refieren a la estructura del cuerpo humano, sus órganos y su posición, como partes del cuerpo u órganos, sistemas, tejidos, células, sustancias corporales y estructuras embrionarias.</td></tr><tr><td>CHEM</td><td>productos químicos y medicamentos (un producto químico o un medicamento)</td><td>Estos son palabras que se refieren a una sustancia o composición que tiene una característica química, especialmente una propiedad curativa o preventiva con respecto a las enfermedades humanas o animales, como medicamentos, antibióticos, proteínas, hormonas, enzimas y sustancias peligrosas o venenosas.</td></tr><tr><td>CONC</td><td>conceptos e ideas (un concepto o una idea)</td><td>Estos son palabras que se refieren a un concepto o una idea, como una clasificación, un producto intelectual, un idioma, una ley o un reglamento.</td></tr><tr><td>DEVI</td><td>dispositivos médicos (un dispositivo)</td><td>Estos son palabras que se refieren a un dispositivo médico utilizado para administrar atención o realizar investigaciones médicas.</td></tr><tr><td>DISO</td><td>trastornos (un trastorno)</td><td>Estos son palabras que se refieren a una alteración de la morfología, la función o la salud de un organismo vivo, animal o vegetal, como anomalías congénitas, disfunción, lesiones, signos o síntomas u observaciones.</td></tr><tr><td>GENE</td><td>genes y secuencias moleculares (un gen o una secuencia molecular)</td><td>Estos son palabras que se refieren a un gen, un genoma o una secuencia molecular.</td></tr><tr><td>GEOG</td><td>áreas geográficas (un área geográfica)</td><td>Estos son palabras que se refieren a un país, una región o una ciudad.</td></tr><tr><td>LIVB</td><td>seres vivos (un ser vivo)</td><td>Estos son palabras que se refieren a un ser vivo o un grupo de seres vivos, como una persona o un grupo de personas, una planta o una categoría de plantas, un animal o una categoría de animales.</td></tr><tr><td>OBJC</td><td>objetos (un objeto)</td><td>Estos son palabras que se refieren a cualquier cosa animada o inanimada que afecte los sentidos, como objetos físicos fabricados.</td></tr><tr><td>OCCU</td><td>ocupaciones (una ocupación)</td><td>Estos son palabras que se refieren a una ocupación o disciplina profesional.</td></tr><tr><td>ORGA</td><td>organizaciones (una organización)</td><td>Estos son palabras que se refieren a una organización, por ejemplo organizaciones relacionadas con la salud.</td></tr><tr><td>PHEN</td><td>fenómenos (un fenómeno)</td><td>Estos son palabras que se refieren a un fenómeno que ocurre naturalmente o como resultado de una actividad, por ejemplo una función biológica.</td></tr></tbody></table>
<table><caption>Table 10: Description of the NER tags used in our experiments for Spanish, continued.</caption><thead><tr><td>Tag</td><td>Tag name (in singular)</td><td>Description</td></tr></thead><tbody><tr><td>PHYS</td><td>fisiología (una fisiología)</td><td>Estas son palabras que se refieren a cualquier elemento que contribuya al funcionamiento mecánico, físico y bioquímico o la organización de los organismos vivos y sus componentes.</td></tr><tr><td>PROC</td><td>procedimientos (un procedimiento)</td><td>Estas son palabras que se refieren a una actividad o un procedimiento que contribuye al diagnóstico o tratamiento de pacientes, la información de pacientes, la capacitación del personal médico o la investigación biomédica.</td></tr><tr><td>EVENT</td><td>eventos (un evento)</td><td>Estas son palabras que se refieren a acciones, estados y circunstancias que son relevantes para la historia clínica de un paciente, como patologías y síntomas, o más generalmente palabras como "entra", "reporta" o "continúa".</td></tr><tr><td>TIMEX3</td><td>expresiones de tiempo (una expresión de tiempo)</td><td>Estas son expresiones de tiempo como fechas, horas, duraciones, frecuencias o intervalos.</td></tr><tr><td>RML</td><td>resultados y mediciones (un resultado o una medida)</td><td>Estos son resultados de análisis de laboratorio, mediciones formales y valores de medición.</td></tr><tr><td>ACTOR</td><td>actores (un actor)</td><td>Estas son palabras que se refieren a pacientes, profesionales de la salud u otros actores relevantes para la historia clínica de un paciente.</td></tr><tr><td>Abbreviation</td><td>abreviaciones (una abreviación)</td><td>Estas son los casos de siglas y acrónimos.</td></tr><tr><td>Body Part</td><td>partes del cuerpo (una parte del cuerpo)</td><td>Estas son palabras que se refieren a órganos y partes anatómicas de personas.</td></tr><tr><td>Clinical Finding</td><td>hallazgos clínicos (un hallazgo clínico)</td><td>Estas son palabras que se refieren a observaciones, juicios o evaluaciones que se hacen sobre los pacientes.</td></tr><tr><td>Diagnostic Procedure</td><td>procedimientos diagnósticos (un procedimiento diagnóstico)</td><td>Estas son palabras que se refieren a exámenes que permiten determinar la condición del individuo.</td></tr><tr><td>Disease</td><td>enfermedades (una enfermedad)</td><td>Estas son palabras que describen una alteración del estado fisiológico en una o varias partes del cuerpo, por causas en general conocidas, manifestada por síntomas y signos característicos, y cuya evolución es más o menos previsible.</td></tr><tr><td>Family Member</td><td>miembros de la familia (un miembro de la familia)</td><td>Estas son palabras que se refieren a miembros de la familia.</td></tr><tr><td>Laboratory or Test Result</td><td>resultados de exámenes de laboratorio u otras pruebas (un resultado de un exámen de laboratorio u otra prueba)</td><td>Estas son palabras que se refieren a cualquier medición o evaluación obtenida a partir de un exámen de apoyo diagnóstico.</td></tr><tr><td>Laboratory Procedure</td><td>procedimientos de laboratorio (un procedimiento de laboratorio)</td><td>Estas son palabras que se refieren a exámenes que se realizan en diversas muestras de pacientes que permiten diagnosticar enfermedades mediante la detección de biomarcadores y otros parámetros. Se consideran los análisis de sangre, orina, y otros fluidos y tejidos que emplean métodos bioquímicos, microbiológicos y/o citológicos.</td></tr><tr><td>Medication</td><td>medicamentos o drogas (un medicamento o una droga)</td><td>Estas son palabras que se refieren a medicamentos o drogas empleados en el tratamiento y/o prevención de enfermedades, incluyendo marcas comerciales y genéricos, así como también nombres para grupos de medicamentos.</td></tr><tr><td>Procedure</td><td>procedimientos (un procedimiento)</td><td>Estas son palabras que se refieren a actividades derivadas de la atención y el cuidado de los pacientes.</td></tr><tr><td>Sign or Symptom</td><td>signos o síntomas (un signo o un síntoma)</td><td>Estas son palabras que se refieren a manifestaciones de una enfermedad, determinadas mediante la exploración médica o percibidas y expresadas por el paciente.</td></tr><tr><td>Therapeutic Procedure</td><td>procedimientos terapéuticos (un procedimiento terapéutico)</td><td>Estas son palabras que se refieren a actividades o tratamientos que es empleado para prevenir, reparar, eliminar o curar la enfermedad del individuo.</td></tr></tbody></table>
<table><caption>Table 11: This table presents the carbon emissions (in g) of the optimization on the validation set of each model over each dataset. For CLMs, this corresponds to the tree search over the prompt features through cross-validation. For MLMs, this corresponds to the supervised fine-tuning and training of the model.</caption><thead><tr><th rowspan="2">#</th><th rowspan="2">Model</th><th colspan="5">English</th><th colspan="4">French</th><th colspan="4">Spanish</th></tr><tr><th>WikiNER</th><th>CoNLL2003</th><th>E3C</th><th>n2c2</th><th>NCBI</th><th>WikiNER</th><th>QFP</th><th>E3C</th><th>EMEA</th><th>MEDLINE</th><th>WikiNER</th><th>CoNLL2002</th><th>E3C</th><th>CWL</th></tr></thead><tbody><tr><td colspan="17">Few-shot approaches</td></tr><tr><td rowspan="13">Causal</td><td>1</td><td>LLAMA-2-70B</td><td>46</td><td>44</td><td>126</td><td>233</td><td>54</td><td>85</td><td>131</td><td>129</td><td>273</td><td>284</td><td>41</td><td>76</td><td>114</td><td>344</td></tr><tr><td>2</td><td>LLAMA-3-8B-Instruct</td><td>3</td><td>5</td><td>12</td><td>19</td><td>12</td><td>5</td><td>7</td><td>7</td><td>11</td><td>23</td><td>4</td><td>7</td><td>12</td><td>36</td></tr><tr><td>3</td><td>Mistral-7B</td><td>4</td><td>6</td><td>12</td><td>24</td><td>8</td><td>5</td><td>8</td><td>14</td><td>13</td><td>25</td><td>7</td><td>5</td><td>11</td><td>27</td></tr><tr><td>4</td><td>Phi-3-medium-instruct</td><td>5</td><td>8</td><td>14</td><td>24</td><td>6</td><td>12</td><td>9</td><td>14</td><td>17</td><td>25</td><td>9</td><td>15</td><td>19</td><td>28</td></tr><tr><td>5</td><td>BLOOM-7B1</td><td>4</td><td>6</td><td>10</td><td>26</td><td>9</td><td>8</td><td>13</td><td>9</td><td>26</td><td>20</td><td>4</td><td>8</td><td>8</td><td>18</td></tr><tr><td>6</td><td>Falcon-40B</td><td>49</td><td>45</td><td>56</td><td>176</td><td>45</td><td>31</td><td>58</td><td>75</td><td>162</td><td>129</td><td>33</td><td>25</td><td>82</td><td>99</td></tr><tr><td>7</td><td>GPT-J-6B</td><td>7</td><td>6</td><td>8</td><td>23</td><td>7</td><td>5</td><td>8</td><td>13</td><td>21</td><td>17</td><td>6</td><td>6</td><td>13</td><td>28</td></tr><tr><td>8</td><td>OPT-66B</td><td>73</td><td>50</td><td>120</td><td>253</td><td>96</td><td>38</td><td>64</td><td>138</td><td>273</td><td>240</td><td>57</td><td>52</td><td>106</td><td>247</td></tr><tr><td>9</td><td>Vicuna-13B</td><td>10</td><td>11</td><td>20</td><td>52</td><td>11</td><td>11</td><td>12</td><td>18</td><td>33</td><td>40</td><td>10</td><td>11</td><td>22</td><td>51</td></tr><tr><td>10</td><td>Vicuna-7B</td><td>6</td><td>8</td><td>14</td><td>17</td><td>6</td><td>5</td><td>10</td><td>10</td><td>24</td><td>14</td><td>8</td><td>6</td><td>13</td><td>27</td></tr><tr><td>11</td><td>BioMistral-7B</td><td>7</td><td>8</td><td>11</td><td>17</td><td>5</td><td>7</td><td>12</td><td>10</td><td>28</td><td>14</td><td>8</td><td>8</td><td>11</td><td>23</td></tr><tr><td>12</td><td>Medalpaca-7B</td><td>8</td><td>4</td><td>17</td><td>24</td><td>10</td><td>7</td><td>14</td><td>11</td><td>19</td><td>13</td><td>7</td><td>8</td><td>15</td><td>26</td></tr><tr><td>13</td><td>Vigogne-13B</td><td>14</td><td>14</td><td>29</td><td>37</td><td>11</td><td>13</td><td>20</td><td>26</td><td>36</td><td>39</td><td>11</td><td>14</td><td>32</td><td>44</td></tr><tr><td rowspan="16">Masked</td><td>11</td><td>mBERT</td><td>2</td><td>1</td><td>2</td><td>2</td><td>2</td><td>2</td><td>2</td><td>2</td><td>1</td><td>1</td><td>1</td><td>2</td><td>1</td><td>2</td></tr><tr><td>12</td><td>XLM-R-large</td><td>2</td><td>2</td><td>2</td><td>1</td><td>2</td><td>2</td><td>2</td><td>2</td><td>2</td><td>1</td><td>1</td><td>1</td><td>2</td><td>2</td></tr><tr><td>13</td><td>BERT-large</td><td>2</td><td>1</td><td>2</td><td>2</td><td>2</td><td>-</td><td>-</td><td>-</td><td>-</td><td>-</td><td>-</td><td>-</td><td>-</td><td>-</td></tr><tr><td>14</td><td>RoBERTa-large</td><td>1</td><td>2</td><td>2</td><td>2</td><td>2</td><td>-</td><td>-</td><td>-</td><td>-</td><td>-</td><td>-</td><td>-</td><td>-</td><td>-</td></tr><tr><td>15</td><td>Bio_ClinicalBERT</td><td>2</td><td>2</td><td>1</td><td>2</td><td>1</td><td>-</td><td>-</td><td>-</td><td>-</td><td>-</td><td>-</td><td>-</td><td>-</td><td>-</td></tr><tr><td>16</td><td>ClinicalBERT</td><td>1</td><td>1</td><td>2</td><td>2</td><td>1</td><td>-</td><td>-</td><td>-</td><td>-</td><td>-</td><td>-</td><td>-</td><td>-</td><td>-</td></tr><tr><td>17</td><td>MedBERT</td><td>2</td><td>2</td><td>1</td><td>1</td><td>1</td><td>-</td><td>-</td><td>-</td><td>-</td><td>-</td><td>-</td><td>-</td><td>-</td><td>-</td></tr><tr><td>18</td><td>CamemBERT-large</td><td>-</td><td>-</td><td>-</td><td>-</td><td>-</td><td>1</td><td>1</td><td>1</td><td>2</td><td>2</td><td>-</td><td>-</td><td>-</td><td>-</td></tr><tr><td>19</td><td>FlauBERT-large</td><td>-</td><td>-</td><td>-</td><td>-</td><td>-</td><td>2</td><td>2</td><td>2</td><td>2</td><td>2</td><td>-</td><td>-</td><td>-</td><td>-</td></tr><tr><td>20</td><td>DrBERT-4GB</td><td>-</td><td>-</td><td>-</td><td>-</td><td>-</td><td>2</td><td>2</td><td>2</td><td>2</td><td>2</td><td>-</td><td>-</td><td>-</td><td>-</td></tr><tr><td>21</td><td>CamemBERT-bio</td><td>-</td><td>-</td><td>-</td><td>-</td><td>-</td><td>1</td><td>2</td><td>2</td><td>2</td><td>2</td><td>-</td><td>-</td><td>-</td><td>-</td></tr><tr><td>23</td><td>BETO</td><td>-</td><td>-</td><td>-</td><td>-</td><td>-</td><td>-</td><td>-</td><td>-</td><td>-</td><td>-</td><td>2</td><td>1</td><td>1</td><td>1</td></tr><tr><td>23</td><td>PatanaBERT</td><td>-</td><td>-</td><td>-</td><td>-</td><td>-</td><td>-</td><td>-</td><td>-</td><td>-</td><td>-</td><td>2</td><td>2</td><td>2</td><td>2</td></tr><tr><td>24</td><td>TulioBERT</td><td>-</td><td>-</td><td>-</td><td>-</td><td>-</td><td>-</td><td>-</td><td>-</td><td>-</td><td>-</td><td>1</td><td>2</td><td>2</td><td>1</td></tr><tr><td>25</td><td>BSC-BioEHR</td><td>-</td><td>-</td><td>-</td><td>-</td><td>-</td><td>-</td><td>-</td><td>-</td><td>-</td><td>-</td><td>2</td><td>2</td><td>2</td><td>2</td></tr><tr><td>26</td><td>BSC-Bio</td><td>-</td><td>-</td><td>-</td><td>-</td><td>-</td><td>-</td><td>-</td><td>-</td><td>-</td><td>-</td><td>2</td><td>2</td><td>2</td><td>2</td></tr><tr><td colspan="17">Masked fully-supervised (skyline)</td></tr><tr><td></td><td>RoBERTa-large</td><td>647</td><td>68</td><td>5</td><td>12</td><td>24</td><td>-</td><td>-</td><td>-</td><td>-</td><td>-</td><td>-</td><td>-</td><td>-</td><td>-</td></tr><tr><td></td><td>CamemBERT-large</td><td>-</td><td>-</td><td>-</td><td>-</td><td>-</td><td>595</td><td>15</td><td>4</td><td>5</td><td>8</td><td>-</td><td>-</td><td>-</td><td>-</td></tr><tr><td></td><td>BETO</td><td>-</td><td>-</td><td>-</td><td>-</td><td>-</td><td>-</td><td>-</td><td>-</td><td>-</td><td>-</td><td>579</td><td>41</td><td>3</td><td>21</td></tr></tbody></table>

<table><caption>Table 12: This table presents the carbon emissions (in g) of the inference on the test set of each model over each dataset.</caption><thead><tr><th rowspan="2">#</th><th rowspan="2">Model</th><th colspan="5">English</th><th colspan="4">French</th><th colspan="4">Spanish</th></tr><tr><th>WikiNER</th><th>CoNLL2003</th><th>E3C</th><th>n2c2</th><th>NCBI</th><th>WikiNER</th><th>QFP</th><th>E3C</th><th>EMEA</th><th>MEDLINE</th><th>WikiNER</th><th>CoNLL2002</th><th>E3C</th><th>CWL</th></tr></thead><tbody><tr><td colspan="17">Few-shot approaches</td></tr><tr><td rowspan="13">Causal</td><td>1</td><td>LLAMA-2-70B</td><td>812</td><td>147</td><td>36</td><td>196</td><td>33</td><td>508</td><td>11</td><td>13</td><td>92</td><td>47</td><td>514</td><td>201</td><td>11</td><td>198</td></tr><tr><td>2</td><td>LLAMA-3-8B-Instruct</td><td>240</td><td>39</td><td>9</td><td>63</td><td>22</td><td>158</td><td>3</td><td>4</td><td>29</td><td>21</td><td>279</td><td>53</td><td>2</td><td>34</td></tr><tr><td>3</td><td>Mistral-7B</td><td>234</td><td>35</td><td>8</td><td>59</td><td>21</td><td>148</td><td>3</td><td>4</td><td>27</td><td>20</td><td>261</td><td>50</td><td>2</td><td>32</td></tr><tr><td>4</td><td>Phi-3-medium-instruct</td><td>326</td><td>48</td><td>12</td><td>64</td><td>25</td><td>380</td><td>5</td><td>9</td><td>62</td><td>48</td><td>529</td><td>70</td><td>4</td><td>76</td></tr><tr><td>5</td><td>BLOOM-7B1</td><td>220</td><td>33</td><td>8</td><td>44</td><td>16</td><td>255</td><td>3</td><td>5</td><td>38</td><td>29</td><td>261</td><td>47</td><td>2</td><td>46</td></tr><tr><td>6</td><td>Falcon-40B</td><td>600</td><td>109</td><td>26</td><td>144</td><td>46</td><td>722</td><td>9</td><td>19</td><td>155</td><td>70</td><td>752</td><td>154</td><td>9</td><td>157</td></tr><tr><td>7</td><td>GPT-J-6B</td><td>146</td><td>17</td><td>4</td><td>53</td><td>20</td><td>245</td><td>2</td><td>6</td><td>14</td><td>26</td><td>154</td><td>40</td><td>3</td><td>53</td></tr><tr><td>8</td><td>OPT-66B</td><td>765</td><td>139</td><td>33</td><td>185</td><td>63</td><td>971</td><td>12</td><td>27</td><td>179</td><td>93</td><td>993</td><td>196</td><td>12</td><td>217</td></tr><tr><td>9</td><td>Vicuna-13B</td><td>314</td><td>47</td><td>11</td><td>63</td><td>24</td><td>363</td><td>5</td><td>8</td><td>61</td><td>46</td><td>502</td><td>67</td><td>4</td><td>74</td></tr><tr><td>10</td><td>Vicuna-7B</td><td>146</td><td>17</td><td>4</td><td>53</td><td>20</td><td>246</td><td>2</td><td>6</td><td>14</td><td>26</td><td>155</td><td>65</td><td>3</td><td>53</td></tr><tr><td>11</td><td>BioMistral-7B</td><td>235</td><td>35</td><td>9</td><td>49</td><td>17</td><td>269</td><td>3</td><td>5</td><td>43</td><td>32</td><td>272</td><td>49</td><td>2</td><td>48</td></tr><tr><td>12</td><td>Medalpaca-7B</td><td>192</td><td>24</td><td>5</td><td>39</td><td>14</td><td>98</td><td>2</td><td>2</td><td>17</td><td>13</td><td>172</td><td>53</td><td>1</td><td>21</td></tr><tr><td>13</td><td>Vigogne-13B</td><td>322</td><td>49</td><td>11</td><td>65</td><td>24</td><td>245</td><td>5</td><td>6</td><td>44</td><td>33</td><td>361</td><td>68</td><td>3</td><td>66</td></tr><tr><td rowspan="16">Masked</td><td>11</td><td>mBERT</td><td>14</td><td>4</td><td>&lt;1</td><td>2</td><td>&lt;1</td><td>15</td><td>1</td><td>&lt;1</td><td>1</td><td>1</td><td>13</td><td>2</td><td>&lt;1</td><td>2</td></tr><tr><td>12</td><td>XLM-R-large</td><td>14</td><td>4</td><td>&lt;1</td><td>2</td><td>&lt;1</td><td>15</td><td>1</td><td>&lt;1</td><td>1</td><td>1</td><td>13</td><td>2</td><td>&lt;1</td><td>2</td></tr><tr><td>13</td><td>BERT-large</td><td>14</td><td>4</td><td>&lt;1</td><td>2</td><td>&lt;1</td><td>-</td><td>-</td><td>-</td><td>-</td><td>-</td><td>-</td><td>-</td><td>-</td><td>-</td></tr><tr><td>14</td><td>RoBERTa-large</td><td>14</td><td>4</td><td>&lt;1</td><td>2</td><td>&lt;1</td><td>-</td><td>-</td><td>-</td><td>-</td><td>-</td><td>-</td><td>-</td><td>-</td><td>-</td></tr><tr><td>15</td><td>Bio_ClinicalBERT</td><td>14</td><td>4</td><td>&lt;1</td><td>2</td><td>&lt;1</td><td>-</td><td>-</td><td>-</td><td>-</td><td>-</td><td>-</td><td>-</td><td>-</td><td>-</td></tr><tr><td>16</td><td>ClinicalBERT</td><td>14</td><td>4</td><td>&lt;1</td><td>2</td><td>&lt;1</td><td>-</td><td>-</td><td>-</td><td>-</td><td>-</td><td>-</td><td>-</td><td>-</td><td>-</td></tr><tr><td>17</td><td>MedBERT</td><td>14</td><td>4</td><td>&lt;1</td><td>2</td><td>&lt;1</td><td>-</td><td>-</td><td>-</td><td>-</td><td>-</td><td>-</td><td>-</td><td>-</td><td>-</td></tr><tr><td>18</td><td>CamemBERT-large</td><td>-</td><td>-</td><td>-</td><td>-</td><td>-</td><td>15</td><td>1</td><td>&lt;1</td><td>1</td><td>1</td><td>-</td><td>-</td><td>-</td><td>-</td></tr><tr><td>19</td><td>FlauBERT-large</td><td>-</td><td>-</td><td>-</td><td>-</td><td>-</td><td>15</td><td>1</td><td>&lt;1</td><td>1</td><td>1</td><td>-</td><td>-</td><td>-</td><td>-</td></tr><tr><td>20</td><td>DrBERT-4GB</td><td>-</td><td>-</td><td>-</td><td>-</td><td>-</td><td>17</td><td>1</td><td>&lt;1</td><td>1</td><td>1</td><td>-</td><td>-</td><td>-</td><td>-</td></tr><tr><td>21</td><td>CamemBERT-bio</td><td>-</td><td>-</td><td>-</td><td>-</td><td>-</td><td>15</td><td>1</td><td>&lt;1</td><td>1</td><td>1</td><td>-</td><td>-</td><td>-</td><td>-</td></tr><tr><td>22</td><td>BETO</td><td>-</td><td>-</td><td>-</td><td>-</td><td>-</td><td>-</td><td>-</td><td>-</td><td>-</td><td>-</td><td>13</td><td>2</td><td>&lt;1</td><td>2</td></tr><tr><td>23</td><td>PatanaBERT</td><td>-</td><td>-</td><td>-</td><td>-</td><td>-</td><td>-</td><td>-</td><td>-</td><td>-</td><td>-</td><td>13</td><td>2</td><td>&lt;1</td><td>2</td></tr><tr><td>24</td><td>TulioBERT</td><td>-</td><td>-</td><td>-</td><td>-</td><td>-</td><td>-</td><td>-</td><td>-</td><td>-</td><td>-</td><td>13</td><td>2</td><td>&lt;1</td><td>2</td></tr><tr><td>25</td><td>BSC-BioEHR</td><td>-</td><td>-</td><td>-</td><td>-</td><td>-</td><td>-</td><td>-</td><td>-</td><td>-</td><td>-</td><td>13</td><td>2</td><td>&lt;1</td><td>2</td></tr><tr><td>26</td><td>BSC-Bio</td><td>-</td><td>-</td><td>-</td><td>-</td><td>-</td><td>-</td><td>-</td><td>-</td><td>-</td><td>-</td><td>13</td><td>2</td><td>&lt;1</td><td>2</td></tr><tr><td colspan="17">Masked fully-supervised (skyline)</td></tr><tr><td></td><td>RoBERTa-large</td><td>14</td><td>4</td><td>&lt;1</td><td>2</td><td>&lt;1</td><td>-</td><td>-</td><td>-</td><td>-</td><td>-</td><td>-</td><td>-</td><td>-</td><td>-</td></tr><tr><td></td><td>CamemBERT-large</td><td>-</td><td>-</td><td>-</td><td>-</td><td>-</td><td>15</td><td>1</td><td>&lt;1</td><td>1</td><td>1</td><td>-</td><td>-</td><td>-</td><td>-</td></tr><tr><td></td><td>BETO</td><td>-</td><td>-</td><td>-</td><td>-</td><td>-</td><td>-</td><td>-</td><td>-</td><td>-</td><td>-</td><td>13</td><td>2</td><td>&lt;1</td><td>2</td></tr></tbody></table>