# DrBERT: A Robust Pre-trained Model in French for Biomedical and Clinical domains

Yanis Labrak, Adrien Bazoge, Richard Dufour, Mickael Rouvier, Emmanuel Morin, Béatrice Daille, Pierre-Antoine Gourraud

► To cite this version:

Yanis Labrak, Adrien Bazoge, Richard Dufour, Mickael Rouvier, Emmanuel Morin, et al. DrBERT: A Robust Pre-trained Model in French for Biomedical and Clinical domains. 61st Annual Meeting of the Association for Computational Linguistics (ACL'23), Jul 2023, Toronto, Canada. hal-04056658

HAL Id: hal-04056658

https://hal.science/hal-04056658

Submitted on 14 Jul 2023

HAL is a multi-disciplinary open access
archive for the deposit and dissemination of sci-
entific research documents, whether they are pub-
lished or not. The documents may come from
teaching and research institutions in France or
abroad, or from public or private research centers.

L'archive ouverte pluridisciplinaire HAL, est
destinée au dépôt et à la diffusion de documents
scientifiques de niveau recherche, publiés ou non,
émanant des établissements d'enseignement et de
recherche français ou étrangers, des laboratoires
publics ou privés.

<figure><img src="image_1.png" /></figure>

Distributed under a Creative Commons CC0 - Public Domain Dedication 4.0 International License
# DrBERT: A Robust Pre-trained Model in French for Biomedical and Clinical domains

Yanis Labrak*¹,⁴ Adrien Bazoge*²,³ Richard Dufour² Mickael Rouvier¹

Emmanuel Morin² Béatrice Daille² Pierre-Antoine Gourraud³

¹LIA, Avignon Université ²LS2N, UMR CNRS 6004, Nantes Université

³Clinique des données, CHU de Nantes, Nantes Université ⁴Zenidoc

`{firstname.lastname}@univ-avignon.fr`

`{firstname.lastname}@univ-nantes.fr`

## Abstract

In recent years, pre-trained language models (PLMs) achieve the best performance on a wide range of natural language processing (NLP) tasks. While the first models were trained on general domain data, specialized ones have emerged to more effectively treat specific domains. In this paper, we propose an original study of PLMs in the medical domain on French language. We compare, for the first time, the performance of PLMs trained on both public data from the web and private data from healthcare establishments. We also evaluate different learning strategies on a set of biomedical tasks. In particular, we show that we can take advantage of already existing biomedical PLMs in a foreign language by further pre-train it on our targeted data. Finally, we release the first specialized PLMs for the biomedical field in French, called DrBERT, as well as the largest corpus of medical data under free license on which these models are trained.

## 1 Introduction

During the last years, pre-trained language models (PLMs) have been shown to significantly improve performance on many Natural Language Processing (NLP) tasks. Recent models, such as BERT (Devlin et al., 2018) or RoBERTa (Liu et al., 2019), are more and more taking advantage of the huge quantities of unlabeled data thanks to recent unsupervised approaches like masked language models based on the Transformers architecture (Vaswani et al., 2017). Most of these PLMs are frequently pre-trained on general domain corpora, such as news articles, books or encyclopedia. An additional step of fine-tuning can also be applied to use these PLMs on a targeted task (Devlin et al., 2018).

Although these generic models are used in various contexts, recent works have shown that opti-

mal performance in specialized domains, such as finance (Yang et al., 2020), medical (Yang et al., 2022) or traveling (Zhu et al., 2021), can only be achieved using PLMs adapted to the targeted conditions.

The adaptation of language models to a domain generally follows two strategies. The first is the training *from scratch* of a new model using only textual data of the targeted specialty. The second approach, called *continual pre-training* (Howard and Ruder, 2018), pursues the training of already pre-trained models, allowing them to pass from a generic model to a specialized one. Even if studies have shown that the first strategy generally offers better performance (Lee et al., 2019), the second requires a much more constrained number of resources (Chalkidis et al., 2020; El Boukkouri et al., 2022) whether in terms of computing resources or of amount of data.

However, domain specific data are generally difficult to obtain, resulting in quite a few specialized PLMs available. This difficulty is even greater for languages other than English. For the medical domain, data produced during clinical care contain the finesse of medical reasoning and are the most prevalent in terms of quantity. However, they are rarely accessible due to patient privacy constraints.

In this paper, we describe and freely disseminate DrBERT, the first RoBERTa-based PLMs specialized in the biomedical field in French, and the corpus that had allowed their trainings. We also propose an original study concerning the evaluation of different language model pre-training strategies for the medical field, while comparing it with our model derived from clinical private data, called ChuBERT. In our experiments, PLMs with publicly available biomedical data can result in similar or better performance compared to highly specialized private data collected from hospital reports or to larger corpora having only generic data. Our contributions can be summarized as follows:

<footnote>*Equal contribution.</footnote>
* A new benchmark aggregating a set of NLP tasks in the medical field in French has been set up, making it possible to evaluate language models at the syntactic and semantic level (multi-label classification, part-of-speech tagging, named entity recognition, etc.).

* A large textual data collection, called NACHOS, crawled from multiple biomedical online sources.

* The construction and evaluation of the first open-source PLMs in French for the biomedical domain based on RoBERTa architecture, called DrBERT, including the analysis of different pre-training strategies.

* A set of models using both public and private data trained on comparable data sizes. These models were then compared by evaluating their performance on a wide range of tasks, both public and private.

* The free distribution of the public PLMs under the open-source MIT license as well as the NACHOS corpus under CC0 1.0 license<fnref n="1" />.

## 2 Related work

BERT (Devlin et al., 2018) is a contextualized word representation model based on the concept of masked language model and pre-trained using bidirectional Transformers (Vaswani et al., 2017). Since its release, it obtains state-of-the-art (SOTA) performance on almost every NLP tasks, while requiring minimal task-specific architectural modifications.

However, the training cost of such a model is very high in terms of computation due to the complexity of each training objective and the quantity of data needed. Consequently, new methods emerge and propose more effective ways of performing pre-training. One of them is RoBERTa (Liu et al., 2019). In order to improve the initial BERT model, the authors made some simple design changes in its training procedure. They modify the masked-language strategy to perform dynamic masking, remove the next sentence prediction task, increase dramatically the batch sizes and use significantly more data during a longer training period. Nowadays, RoBERTa is the standard

model for a lot of NLP tasks and languages, including French with CamBERT model (Martin et al., 2020).

Recently, multiple language models have been developed for biomedical and clinical fields through unsupervised pre-training of Transformer-based architectures, mainly for English language. One of the first models was BioBERT (Lee et al., 2019), which is based on the initially pre-trained BERT model and further pre-trained using biomedical-specific data through continual pre-training. Other models like BlueBERT (Peng et al., 2019) and ClinicalBERT (Huang et al., 2019) also used this approach on various data sources. An alternative method, when enough in-domain data is available, is to directly pre-train models from scratch (SciBERT (Beltagy et al., 2019), PubMed-BERT (Gu et al., 2021), etc.). Note that SciBERT was trained on mixed-domain data from biomedical and computer science domains, while PubMed-BERT on biomedical data only. Gu et al. (2021) disputed the benefits of mixed-domain data for pre-training, based on results obtained on tasks from BLURB benchmark.

In other languages than English, BERT-based models are much rarer and primarily rely on continual pre-training. Examples include German (Shrestha, 2021), Portuguese (Schneider et al., 2020), and Swedish (Vakili et al., 2022). Only the Spanish (Carrino et al., 2021) and Turkish (Türkmen et al., 2022) models were trained from scratch with biomedical and clinical data from various sources. For French, there is, to our knowledge, no publicly available model specifically built for the biomedical domain.

## 3 Pre-training datasets

In the biomedical domain, previous works (Gu et al., 2021) on PLMs highlighted the importance of matching the data sources used for its training to the targeted downstream tasks. Due to their sensitive nature (protection of user data, protected health information of patients, etc.), medical data are extremely difficult to obtain. Massive collection of web data related to this domain appears to be a solution that can overcome this lack. However, these web documents vary in terms of quality. No comparison has been made between PLMs based on specific domain data from the web and those on private documents from clinical data warehouses, whose quality can be controlled.

<footnote n="1">¹https://drbert.univ-avignon.fr/</footnote>
We extracted two different medical datasets for French. The first one gathers data crawled from a variety of free-of-use online sources, and the second one private hospital stays reports from the Nantes University Hospital.

Table 1 gives a general overview of the two collected corpora. The public web-based data, detailed in Section 3.1, allowed the constitution of a corpus, called NACHOS<sub>large</sub>, containing 7.4 GB of data. The private dataset, called NBDW<sub>small</sub> is described in Section 3.2 and contains 4 GB of data. In order to perform comparable experiments, we extracted a NACHOS sub-corpus (NACHOS<sub>small</sub>) of the same size as the private data. Finally, Section 3.3 describes the pre-processing applied to both datasets.

<table><caption>Table 1: Overview of the public (NACHOS) and private (NBDW) collected datasets.</caption><thead><tr><th>Corpus</th><th>Size</th><th>#words</th><th>#sentences</th></tr></thead><tbody><tr><td>NACHOS<sub>large</sub> (pub.)</td><td>7.4 GB</td><td>1.1 B</td><td>54.2 M</td></tr><tr><td>NACHOS<sub>small</sub> (pub.)</td><td>4 GB</td><td>646 M</td><td>25.3 M</td></tr><tr><td>NBDW<sub>small</sub> (private)</td><td>4 GB</td><td>655 M</td><td>43.1 M</td></tr><tr><td>NBDW<sub>mixed</sub> (both)</td><td>4+4 GB</td><td>1.3 B</td><td>68.4 M</td></tr></tbody></table>

## 3.1 Public corpus - NACHOS

We introduce the opeN crAwled frenCh Health-care cOrpuS (NACHOS), a French medical open-source dataset compiled by crawling a variety of textual sources around the medical topic. It consists of more than one billion words, drawn from 24 French speaking high-quality websites. The corpus includes a wide range of medical information: descriptions of diseases and conditions, information on treatments and medications, general health-related advice, official scientific meeting reports, anonymized clinical cases, scientific literature, thesis, French translation pairs, university health courses and a large range of data obtained from raw textual sources, web scrapping, and optical character recognition (OCR). Table 2 summarizes the different data sources of NACHOS.

We use heuristics to split the texts into sentences and aggressively filter out short or low-quality sentences like those obtained from OCR. Finally, we classified them into languages by using our own classifier trained on the multilingual Opus EMEA (Tiedemann and Nygaard, 2004) and MAS-SIVE (FitzGerald et al., 2022) corpora to keep only the sentences in French.

For the 4 GB version of NACHOS (NACHOS<sub>small</sub>), we shuffled the whole corpus and selected randomly 25.3M sentences to maximize data sources homogeneity. The full NACHOS corpus is now freely available online<fnref n="2" />.

<table><caption>Table 2: Sources of the NACHOS corpus.</caption><thead><tr><th>Resource name</th><th># words</th></tr></thead><tbody><tr><td>HAL</td><td>638,508,261</td></tr><tr><td>Haute Autorité de Santé (HAS)</td><td>113,394,539</td></tr><tr><td>Drug leaflets</td><td>74,770,229</td></tr><tr><td>Medical Websites Scrapping</td><td>60,561,495</td></tr><tr><td>ANSES SAISINE</td><td>51,372,932</td></tr><tr><td>Public Drug Database (BDPM)</td><td>48,302,695</td></tr><tr><td>ISTEX</td><td>44,124,422</td></tr><tr><td>CRTT</td><td>26,210,756</td></tr><tr><td>WMT-16</td><td>10,282,494</td></tr><tr><td>EMEA-V3</td><td>6,601,617</td></tr><tr><td>Wikipedia Life Science French</td><td>4,671,944</td></tr><tr><td>Cochrane summaries</td><td>3,932,526</td></tr><tr><td>ANSES RCP</td><td>2,953,045</td></tr><tr><td>Cerimes</td><td>1,717,552</td></tr><tr><td>Vidal</td><td>410,313</td></tr><tr><td>LiSSa</td><td>235,838</td></tr><tr><td>DEFT-2020</td><td>231,396</td></tr><tr><td>CLEAR</td><td>225,898</td></tr><tr><td>CNEDiMTS</td><td>175,416</td></tr><tr><td>QUAERO French Medical Corpus</td><td>72,031</td></tr><tr><td>ANSM Clinical Study Registry</td><td>47,678</td></tr><tr><td>ECDC</td><td>44,482</td></tr><tr><td>QualiScope</td><td>12,718</td></tr><tr><td>WMT-18-Medline</td><td>7,673</td></tr></tbody><tfoot><tr><td><strong>Total</strong></td><td><strong>1,088,867,950</strong></td></tr></tfoot></table>

## 3.2 Private corpus - NBDW

The private corpus, called Nantes Biomedical Data Warehouse (NBDW), was obtained using the data warehouse from Nantes University Hospital. This data warehouse includes different dimensions of patients' related data: socio-demographic, drug prescriptions and other information associated with consultation or hospital stays (diagnosis, biology, imagery, etc.). The authorization to implement and exploit the NBDW dataset was granted in 2018 by the CNIL (Commission National de l'Informatique et des Libertés), the French independent supervisory authority in charge of application of national and European data privacy protection laws; authorization N°2129203.

For this work, a sample of 1.7 million identified hospital stays reports was randomly selected and extracted from the data warehouse. As described in Table 3, the reports are from various hospital departments, emergency medicine, gynecology and ambulatory care being the most frequent.

Each report was split into tokens sequence with an average of 15.26 words per sequence. Then,

<footnote n="2">²https://drbert.univ-avignon.fr/</footnote>
all tokens sequences from all reports were shuffled to build the corpus. This corpus contains 655M words, from 43.1M sentences, for a total size of approximately 4 GB.

<table><caption>Table 3: Sources of the NBDW corpus.</caption><thead><tr><th>Medical Specialty</th><th># documents</th><th># words</th></tr></thead><tbody><tr><td>Other</td><td>474,588</td><td>192,832,792</td></tr><tr><td>Emergency Medicine</td><td>235,579</td><td>90,807,406</td></tr><tr><td>Ambulatory Care</td><td>119,149</td><td>50,975,472</td></tr><tr><td>Consultation</td><td>95,135</td><td>38,335,804</td></tr><tr><td>Gynecology</td><td>132,983</td><td>38,204,495</td></tr><tr><td>Cardiology</td><td>29,633</td><td>22,654,583</td></tr><tr><td>Medical Oncology</td><td>45,603</td><td>22,587,869</td></tr><tr><td>Gastroenterology</td><td>46,600</td><td>21,340,794</td></tr><tr><td>Orthopaedic Surgery</td><td>82,084</td><td>18,983,791</td></tr><tr><td>Hematology</td><td>41,776</td><td>18,285,983</td></tr><tr><td>Critical Care Medicine</td><td>20,819</td><td>16,472,785</td></tr><tr><td>Otolaryngology</td><td>69,343</td><td>16,131,214</td></tr><tr><td>Dermatology</td><td>51,804</td><td>15,035,412</td></tr><tr><td>Rheumatology</td><td>31,527</td><td>14,647,543</td></tr><tr><td>Urology</td><td>51,535</td><td>14,272,231</td></tr><tr><td>Colon and Rectal Surgery</td><td>45,987</td><td>13,334,550</td></tr><tr><td>Internal Medicine</td><td>23,904</td><td>13,282,253</td></tr><tr><td>Psychiatry</td><td>26,628</td><td>12,496,503</td></tr><tr><td>Neurosurgery</td><td>34,481</td><td>10,360,533</td></tr><tr><td>Nephrology</td><td>19,171</td><td>9,548,533</td></tr><tr><td>Ophthalmology</td><td>19,700</td><td>4,464,515</td></tr></tbody><tfoot><tr><td>Total</td><td>1,698,029</td><td>655,055,061</td></tr></tfoot></table>

## 3.3 Pre-processing step

The supplied text data has been split into sub-word units using SentencePiece (Kudo and Richardson, 2018), an extension of Byte-Pair encoding (BPE) (Sennrich et al., 2016) and WordPiece (Wu et al., 2016) that does not require pre-tokenization (at the word or token level), thereby avoiding the requirement for language-specific tokenizers. We employ a vocabulary size of 32k subword tokens. For each model pre-trained from scratch (see Section 4.2), tokenizers were built using all the sentences from the pre-training dataset.

# 4 Models pre-training

In this section, we describe the pre-training modalities of our studied models from two points of view: 1) the influence of the data used (size and nature), and 2) the pre-training strategies of the models. These two levels are respectively detailed in Sections 4.1 and 4.2. Section 4.3 finally presents the existing state-of-the-art pre-trained models that will be used for comparison purposes.

## 4.1 Influence of data

One issue is to identify the amount of data required to create a model that performs well and can com-

pete with models trained on general domains. Recent studies, such as those by Zhang et al. (2020) and Martin et al. (2020), discuss the impact of the size of pre-training data on model performance. According to these studies, some tasks are performing better with fewer data while others, such as commonsense knowledge and reasoning tasks, keep improving performance when pre-training data are added.

In the medical field, no study has been conducted to compare the impact of varying the amount of domain-specific data during pre-training, or to assess the impact of the supposedly variable quality of the data depending on their source of collection.

We thus propose to evaluate the pre-training of several language models on either NACHOS<sub>small</sub> or NBDW<sub>small</sub> corpus, as described in Section 3. Additionally, we propose a model pre-trained on NACHOS<sub>large</sub> to investigate if having almost twice as much data improves model performance. Finally, a combination of both public NACHOS<sub>small</sub> and NBDW<sub>small</sub> sources for a total of 8 GB (NBDW<sub>mixed</sub>) is explored, to demonstrate if combining private and public data is a viable approach in low-resource domains.

## 4.2 Pre-training strategies

In addition to the analysis on the size and the sources of data, we also seek to evaluate three training strategies of PLMs for the medical domain:

* Training a full model from scratch, including the subword tokenizer.

* Continuing the pre-training of the state-of-the-art language model for French, called CamBERT, on our medical-specific data while keeping the initial tokenizer.

* Continuing the pre-training of a state-of-the-art domain specific language model for medical but here in English, called PubMedBERT, on our French data while keeping the initial tokenizer.

Regarding the last strategy, our objective is to compare the performance of an English medical model further pre-trained on our French medical data, against another one based on a generic French model. Indeed, the medical domains share many terms across languages that make relevant the mixture of resources from two languages.
Table 4 summarizes all the configurations evaluated in this paper, integrating both the study of data size and pre-training strategies.

<table><caption>Table 4: List of studied pre-trained model configurations.</caption><thead><tr><th>Model name</th><th>Strategy</th><th>Corpus</th></tr></thead><tbody><tr><td>DrBERT</td><td>From scratch</td><td>NACHOS<sub>large</sub></td></tr><tr><td>DrBERT</td><td>From scratch</td><td>NACHOS<sub>small</sub></td></tr><tr><td>ChuBERT</td><td>From scratch</td><td>NBDW<sub>small</sub></td></tr><tr><td>ChuBERT</td><td>From scratch</td><td>NBDW<sub>mixed</sub></td></tr><tr><td>CamemBERT</td><td>continual pre-training</td><td>NACHOS<sub>small</sub></td></tr><tr><td>PubMedBERT</td><td>continual pre-training</td><td>NACHOS<sub>small</sub></td></tr><tr><td>CamemBERT</td><td>continual pre-training</td><td>NBDW<sub>small</sub></td></tr></tbody></table>

**Model architecture** All models pre-trained from-scratch use the CamemBERT base configuration, which is the same as RoBERTa base architecture (12 layers, 768 hidden dimensions, 12 attention heads, 110M parameters). We did not train the large version of our models due to resources limitations.

**Language modeling** We train the models on the Masked Language Modeling (MLM) task using HuggingFace library (Wolf et al., 2019). It consists of randomly replacing a subset of tokens from the sequence by a special token, and asking the model to predict them using cross-entropy loss. In BERT and RoBERTa models (including CamemBERT), 15% of the tokens are randomly selected. Of those selected tokens, 80% are replaced with the <mask> token, 10% remain unchanged and 10% are randomly replaced by a token from the vocabulary. We keep this masking probability of 15% for the training of our models.

**Optimization & Pre-training** We optimize the models for 80k steps with batch sizes of 4,096 sequences, each sequence filled with 512 tokens, allowing to process 2.1M tokens per step. The learning rate is warmed up linearly for 10k steps, going up from zero to the initial $5 \times 10^{-5}$ learning rate. Models are trained on 128 Nvidia V100 32 GB GPUs for 20 hours on Jean Zay supercomputer. We use mixed precision training (FP16) (Micikević et al., 2017) to reduce the memory footprint, allowing us to enlarge the batch size to 32 sequences on each GPU.

## 4.3 Baseline models

We describe some existing pre-trained models used as baselines in our comparative study.

CamemBERT (Martin et al., 2020) is a RoBERTa based model pre-trained totally from scratch on the French subset of OSCAR corpus (138 GB). In our case, this model is our main baseline to compare our results on, since it is the state-of-the-art model for French. We also use the 4 GB model's variants of CamemBERT to compare the impact of the nature and quantity of the data.

PubMedBERT (Gu et al., 2021) is a BERT based biomedical-specific model pre-trained totally from scratch on the 3.1 billions words of PubMed corpus (21 GB).

ClinicalBERT (Huang et al., 2019) is a clinical-specific model based on BERT tokenizer and weights, which has been further pre-trained on the 0.5 billion words of MIMIC corpus (3.7 GB).

BioBERT v1.1 (Lee et al., 2019) is a biomedical-specific model based on BERT tokenizer and weights which has been further pre-trained using the 4.5 billion words of PubMed corpus.

# 5 Downstream evaluation tasks

To evaluate the different pre-training configurations of our models, a set of tasks in the medical domain is necessary. While this NLP domain-specific benchmark exists in English (BLURB (Gu et al., 2021)), none exist for French. In this section, we describe an original benchmark, summarized in Table 5, integrating various NLP medical tasks for French. Among them, some are from publicly-available datasets (Section 5.1), allowing the replication of our experiments. Other tasks come from private datasets (Section 5.2) and cannot be shared. However, they are useful to evaluate our models more accurately.

## 5.1 Publicly-available tasks

**ESSAIS / CAS: French Corpus with Clinical Cases** The ESSAIS (Dalloux et al., 2021) and CAS (Grabar et al., 2018) corpora respectively contain 13,848 and 7,580 clinical cases in French. Some clinical cases are associated with discussions. A subset of the whole set of cases is enriched with morpho-syntactic (part-of-speech (POS) tagging, lemmatization) and semantic (UMLS concepts, negation, uncertainty) annotations. In our case, we focus only on the POS tagging task.

**FrenchMedMCQA** The FrenchMedMCQA corpus (Labrak et al., 2022) is a publicly available Multiple-Choice Question Answering (MCQA) dataset in French for medical domain. It contains 3,105 questions coming from real exams of
<table><caption>Table 5: Corpus, tasks and metrics synthesis for evaluating medical-specific models.</caption><thead><tr><th>Thematic / Corpus name</th><th>Task</th><th>Metric</th><th>Train</th><th>Dev</th><th>Test</th></tr></thead><tbody><tr><td colspan="6"><em>Public Corpus</em></td></tr><tr><td>ESSAIS (Dalloux et al., 2021)</td><td>POS Tagging</td><td>Macro F1</td><td>9,693</td><td>2,077</td><td>2,078</td></tr><tr><td>CAS: French Corpus with Clinical Cases (Grabar et al., 2018)</td><td>POS Tagging</td><td>Macro F1</td><td>5,306</td><td>1,137</td><td>1,137</td></tr><tr><td>MUSCA-DET - Social Determinants of Health extraction (Task 1)</td><td>Nested NER</td><td>Macro F1</td><td>19,861</td><td>2,207</td><td>5,518</td></tr><tr><td>MUSCA-DET - Social Determinants of Health extraction (Task 2)</td><td>Multi-label Classification</td><td>Macro F1</td><td>19,861</td><td>2,207</td><td>5,518</td></tr><tr><td>QUAERO French Medical Corpus - EMEA (Névédol et al., 2014)</td><td>Nested NER</td><td>Weighted F1</td><td>11</td><td>12</td><td>15</td></tr><tr><td>QUAERO French Medical Corpus - MEDLINE (Névédol et al., 2014)</td><td>Nested NER</td><td>Weighted F1</td><td>833</td><td>832</td><td>833</td></tr><tr><td>FrenchMedMCQA (Labrak et al., 2022)</td><td>MCQA</td><td>EMR / Hamming Score</td><td>2,171</td><td>312</td><td>622</td></tr><tr><td colspan="6"><em>Private Corpus</em></td></tr><tr><td>Medical report acute heart failure structuration</td><td>Named Entity Recognition</td><td>Macro F1</td><td>2,527</td><td>281</td><td>703</td></tr><tr><td>Acute heart failure (aHF) classification</td><td>Binary Classification</td><td>Macro F1</td><td>1,179</td><td>132</td><td>328</td></tr><tr><td>Technical Specialties Sorting</td><td>Classification Multi-class</td><td>Macro F1</td><td>4,413</td><td>1,470</td><td>1,473</td></tr><tr><td>Medical report structuration prescriptions</td><td>Named Entity Recognition</td><td>Macro F1</td><td>61</td><td>15</td><td>26</td></tr></tbody></table>

the French medical specialization diploma in pharmacy, integrating single and multiple answers.

**QUAERO French Medical Corpus** The QUAERO French Medical Corpus (Névédol et al., 2014) introduces an extensive corpus of biomedical documents annotated at the entity and concept levels to provide NER and classification tasks. Three text genres are covered, comprising a total of 103,056 words obtained either from EMEA or MEDLINE. Ten entity categories corresponding to UMLS (Bodenreider, 2004) Semantic Groups were annotated, using automatic pre-annotations validated by trained human annotators. Overall, a total of 26,409 entity annotations were mapped to 5,797 unique UMLS concepts. To simplify the evaluation process, we sort the nested labels by alphabetical order and concatenate them together into a single one to transform the task to a usable format for token classification with BERT based architectures.

**MUSCA-DET** MUSCA-DET is a French corpus of sentences extracted from the "Lifestyle" section in clinical notes from Nantes University Hospital biomedical data warehouse. The corpus contains 27,000 pseudonymized sentences annotated with 26 entities related to Social Determinants of Health (living, marital status, housing, descendants, employment, alcohol, smoking, drug abuse, physical activity). The corpus includes two tasks: nested name entity recognition (NER) and multi-label classification.

## 5.2 Private tasks

**Technical Specialties Sorting** This classification task has to assign the specialty of a medical of a medical report based on its transcription. The dataset consists of 7,356 French medical reports that have been manually annotated and equally sampled

across 6 specialties: Psychiatry, Urology, Endocrinology, Cardiology, Diabetology, and Infectiology.

**Medical report structuration prescriptions (NER)** The task seeks to identify named entities in a gold sample of 100 long medical reports obtained from French speech transcriptions. The named entities are annotated using the BIO format and fall into 12 classes: O, AGE, CITY, DATE, EMAIL, HOSPITAL, PHONE, DOSAGE, DURATION, FORM, MEDICATION and POSOLOGY.

**Medical report acute heart failure structuration (NER)** This corpus contains 350 hospital stay reports (divided into 3,511 sentences) from Nantes University Hospital. The reports are annotated with 46 entity types related to the following clinical information: cause of chronic heart failure, triggering factor for acute heart failure, diabetes, smoking status, heart rate, blood pressure, weight, height, medical treatment, hypertension and left ventricular ejection fraction. Overall, the corpus contains 6,116 clinical entities.

**Acute heart failure (aHF) classification** This task consists of the classification of hospital stays reports according to the presence or absence of a diagnostic of acute heart failure. This corpus consists of 1,639 hospital stays reports from Nantes university hospital, which are labeled as positive or negative to acute heart failure.

## 6 Results and Discussions

As previously described, we evaluate the performance of our pre-trained language models proposed for the biomedical domain on a set of public and private NLP downstream tasks related to the medical domain. We first propose to analyze the results according to the different pre-training strategies used (Section 6.1) then to focus on the impact
<table><caption>Table 6: Performance on our private biomedical downstream tasks. Best model in bold and second is underlined.</caption><thead><tr><th rowspan="2"></th><th colspan="3">aHF NER</th><th colspan="3">aHF classification</th><th colspan="3">NER Medical Report</th><th colspan="3">Specialties Classification</th></tr><tr><th>P</th><th>R</th><th>F1</th><th>P</th><th>R</th><th>F1</th><th>P</th><th>R</th><th>F1</th><th>P</th><th>R</th><th>F1</th></tr></thead><tbody><tr><td>CamemBERT OSCAR 138 GB</td><td>40.89</td><td>35.22</td><td>35.13</td><td>81.90</td><td>79.12</td><td>80.13</td><td>87.98</td><td>91.66</td><td>89.35</td><td>99.32</td><td>99.09</td><td>99.20</td></tr><tr><td>CamemBERT OSCAR 4 GB</td><td>46.32</td><td>43.17</td><td>42.66</td><td>81.49</td><td>81.42</td><td>81.41</td><td>87.79</td><td>90.74</td><td>88.78</td><td>99.53</td><td>99.69</td><td>99.61</td></tr><tr><td>CamemBERT CCNET 4 GB</td><td>47.25</td><td>42.2</td><td>43.11</td><td>82.02</td><td>79.30</td><td>79.98</td><td>87.61</td><td>92.28</td><td>89.34</td><td>99.54</td><td>99.55</td><td>99.55</td></tr><tr><td>PubMedBERT</td><td>52.61</td><td>46.30</td><td>47.22</td><td>78.17</td><td>76.18</td><td>76.86</td><td>87.07</td><td>92.61</td><td>89.20</td><td>99.25</td><td>99.51</td><td>99.37</td></tr><tr><td>ClinicalBERT</td><td>50.11</td><td>44.15</td><td>44.70</td><td>80.13</td><td>75.92</td><td>77.12</td><td>87.04</td><td>92.14</td><td>88.77</td><td>98.58</td><td>98.62</td><td>98.58</td></tr><tr><td>BioBERT v1.1</td><td>49.37</td><td>47.25</td><td>46.01</td><td>79.69</td><td>78.51</td><td>79.00</td><td><strong>88.17</strong></td><td>91.80</td><td>89.38</td><td>98.59</td><td>99.03</td><td>98.80</td></tr><tr><td>DrBERT NACHOS<sub>large</sub></td><td><strong>55.29</strong></td><td>46.66</td><td>48.22</td><td>81.33</td><td>81.25</td><td>81.25</td><td><strong>87.99</strong></td><td><strong>92.80</strong></td><td><strong>89.83</strong></td><td><strong>99.82</strong></td><td><strong>99.90</strong></td><td><strong>99.86</strong></td></tr><tr><td>DrBERT NACHOS<sub>small</sub></td><td>54.55</td><td>43.39</td><td>45.93</td><td>79.85</td><td>80.10</td><td>79.87</td><td>87.57</td><td><strong>92.76</strong></td><td>89.44</td><td><strong>99.85</strong></td><td><strong>99.85</strong></td><td><strong>99.85</strong></td></tr><tr><td>ChuBERT NBDW<sub>small</sub></td><td><strong>56.92</strong></td><td>47.46</td><td>49.01</td><td>81.03</td><td><strong>82.67</strong></td><td>81.56</td><td>87.76</td><td>92.63</td><td>89.58</td><td>99.76</td><td><strong>99.90</strong></td><td>99.83</td></tr><tr><td>ChuBERT NBDW<sub>mixed</sub></td><td>54.62</td><td>47.81</td><td><strong>49.14</strong></td><td>82.23</td><td>81.71</td><td><strong>81.98</strong></td><td>87.42</td><td>92.36</td><td>89.30</td><td>99.81</td><td>99.82</td><td>99.81</td></tr><tr><td>CamemBERT NACHOS<sub>small</sub></td><td>22.02</td><td>16.67</td><td>16.08</td><td>74.86</td><td>69.82</td><td>69.80</td><td>65.72</td><td>68.49</td><td>66.74</td><td>99.44</td><td>99.67</td><td>99.54</td></tr><tr><td>PubMedBERT NACHOS<sub>small</sub></td><td>53.44</td><td><strong>48.21</strong></td><td>48.72</td><td><strong>83.06</strong></td><td>80.39</td><td>81.40</td><td>87.35</td><td>92.69</td><td>89.36</td><td>99.52</td><td>99.58</td><td>99.55</td></tr><tr><td>CamemBERT NBDW<sub>small</sub></td><td>25.44</td><td>19.33</td><td>19.12</td><td>79.50</td><td>74.74</td><td>76.02</td><td>68.80</td><td>71.23</td><td>69.64</td><td>99.60</td><td>99.57</td><td>99.58</td></tr></tbody></table>

of the pre-training data, whether in terms of size or nature (Section 6.2). Finally, we are interested in the generalization capacities of our domain-specific models by applying and comparing them on general domain NLP tasks (Section 6.3).

Note that all the PLMs have been fine-tuned in the same way for all downstream tasks and all the reported results are obtained by averaging the scores from four runs. Performance on biomedical downstream tasks are reported in Tables 6 and 7 for respectively private and public tasks. For readability reasons, the first part of each table presents the existing baseline models results, the second part our specialized models trained from-scratch, and the last part our models using continual pre-training.

## 6.1 Impact of pre-training strategies

As observed both in Tables 6 and 7, models pre-trained completely from scratch (DrBERT NACHOS and ChuBERT NBDW) tend to produce the best results for both types of data sources and tasks (i.e. private and public). Indeed, considering the F1-score, they obtain the best results on all private tasks and on almost all public ones (5 tasks out of 7). The two public remaining tasks (MUSCA-DET T2 and QUAERO-MEDLINE) are then better handled using PubMedBERT NACHOS<sub>small</sub>, a model that has already been pre-trained on domain-specific data (biomedical English data) then further pre-trained with our French medical data (NACHOS<sub>small</sub>).

We also observed that continual pre-training from domain generic models (CamemBERT NACHOS<sub>small</sub> or CamemBERT NBDW<sub>small</sub>) does not allow reaching the performance of the other specific models, neither of these two models reaching the first or second place (in terms of performance) on any task.

Finally, the baseline models trained on generic data (CamemBERT OSCAR) and those trained on biomedical data in English (PubMedBERT, ClinicalBERT and BioBERT) remain competitive in few biomedical public tasks (CAS POS, FrenchMCQA or MUSCA-DET T2), while none of them are placed in first or second place on private tasks. This seems to highlight the difficulty of private tasks when non-matching data are used.

## 6.2 Effect of data

Regarding the amount of data used for pre-training models (small vs. large or mixed), results show that, the larger the data are, the better the model performs, no matter the pre-training strategy or the source of data (private or public). However, the difference is very low for most tasks, with small systems often being ranked second behind large models, even though they contain half as much data.

We notice a clear dominance of models that were pre-trained on web-based sources, specifically OSCAR and NACHOS, when applied to public tasks. Indeed, models relying on private NBDW data only achieve the best performance (in terms of F1-score) on the MUSCA-DET T1 task. This trend is not quite observed on private tasks, where NBDW-based models obtain more acceptable or even better performance when mixed with public biomedical data (ChuBERT NBDW<sub>mixed</sub>), as seen in Table 6. We believe this discrepancy is mainly due to the different nature of processed data.

Finally, we observe that English-based models perform closely to the French-based CamemBERT model. This shows the usefulness of pre-training on domain specific data. For example, better results are obtained with continual pre-training of the PubMedBERT model with our specialized data in French (PubMedBERT NACHOS<sub>small</sub>), corroborating
<table><caption>Table 7: Performance on public biomedical downstream tasks. Best model in bold and second is underlined.</caption><thead><tr><th rowspan="2"></th><th colspan="3">MUSCA-DET T1</th><th colspan="3">MUSCA-DET T2</th><th colspan="3">ESSAI POS</th><th colspan="3">CAS POS</th><th colspan="3">FrenchMedMCQA</th><th colspan="3">QUAERO-EMEA</th><th colspan="3">QUAERO-MEDLINE</th></tr><tr><th>P</th><th>R</th><th>F1</th><th>P</th><th>R</th><th>F1</th><th>P</th><th>R</th><th>F1</th><th>P</th><th>R</th><th>F1</th><th>Hamming</th><th>EMR</th><th>P</th><th>R</th><th>F1</th><th>P</th><th>R</th><th>F1</th></tr></thead><tbody><tr><td>CamemBERT OSCAR 138 GB</td><td>89.04</td><td>88.59</td><td>88.54</td><td>89.87</td><td>87.12</td><td>88.20</td><td>81.57</td><td>81.01</td><td>81.10</td><td>96.37</td><td>94.53</td><td>95.22</td><td>36.24</td><td><strong>16.55</strong></td><td>90.57</td><td>91.06</td><td>90.71</td><td>76.58</td><td>78.67</td><td>77.41</td></tr><tr><td>CamemBERT OSCAR 4 GB</td><td>86.09</td><td>85.45</td><td>85.43</td><td>92.68</td><td>90.34</td><td>91.27</td><td>84.01</td><td>83.51</td><td>83.69</td><td>98.15</td><td>95.34</td><td>96.42</td><td>35.75</td><td>15.37</td><td>90.75</td><td>91.16</td><td>90.83</td><td>78.55</td><td>79.33</td><td><u>78.76</u></td></tr><tr><td>CamemBERT CCNET 4 GB</td><td>91.12</td><td>89.91</td><td>90.33</td><td>93.10</td><td><u>90.42</u></td><td>91.38</td><td>85.60</td><td>85.63</td><td>85.42</td><td><strong>98.19</strong></td><td><strong>96.75</strong></td><td><strong>97.33</strong></td><td>34.71</td><td>14.41</td><td>90.31</td><td>90.59</td><td>90.33</td><td>78.06</td><td>78.11</td><td>77.61</td></tr><tr><td>PubMedBERT</td><td>93.04</td><td>91.45</td><td>91.99</td><td>84.41</td><td>80.60</td><td>81.97</td><td>88.43</td><td>87.93</td><td>87.78</td><td>97.40</td><td>94.86</td><td>95.90</td><td>33.98</td><td>14.14</td><td>86.89</td><td>87.33</td><td>86.79</td><td>77.33</td><td>77.28</td><td>77.09</td></tr><tr><td>ClinicalBERT</td><td>91.79</td><td>89.44</td><td>90.36</td><td>85.43</td><td>81.23</td><td>82.95</td><td>89.09</td><td>88.78</td><td>88.24</td><td>97.94</td><td>95.88</td><td>96.73</td><td>32.78</td><td>14.19</td><td>84.91</td><td>85.47</td><td>84.79</td><td>75.56</td><td>74.85</td><td>75.05</td></tr><tr><td>BioBERT 1.1</td><td>91.82</td><td>89.82</td><td>90.46</td><td>85.52</td><td>80.14</td><td>81.91</td><td>86.76</td><td>84.90</td><td>85.18</td><td>98.10</td><td><u>96.39</u></td><td><u>97.12</u></td><td>36.19</td><td><u>15.43</u></td><td>84.55</td><td>85.03</td><td>84.29</td><td>72.62</td><td>73.30</td><td>72.68</td></tr><tr><td>DrBERT NACHOS<sub>large</sub></td><td>92.10</td><td>90.27</td><td>91.04</td><td><strong>94.97</strong></td><td>90.41</td><td><u>92.24</u></td><td><strong>90.96</strong></td><td><strong>89.19</strong></td><td><strong>89.75</strong></td><td>97.37</td><td>94.49</td><td>95.65</td><td><u>36.66</u></td><td>15.32</td><td><strong>91.93</strong></td><td><strong>92.52</strong></td><td><strong>92.09</strong></td><td>77.85</td><td>78.54</td><td>77.88</td></tr><tr><td>DrBERT NACHOS<sub>small</sub></td><td>93.35</td><td>90.62</td><td>91.77</td><td>91.31</td><td>86.60</td><td>88.57</td><td>90.12</td><td>88.37</td><td>88.76</td><td>97.04</td><td>94.88</td><td>95.70</td><td><strong>37.37</strong></td><td>13.34</td><td>91.54</td><td>92.00</td><td>91.66</td><td>77.91</td><td>79.34</td><td>78.18</td></tr><tr><td>ChuBERT NBDW<sub>small</sub></td><td><strong>94.88</strong></td><td>90.79</td><td><u>92.23</u></td><td>94.77</td><td>90.27</td><td>92.17</td><td>88.53</td><td>87.73</td><td>87.71</td><td>97.00</td><td>94.65</td><td>95.61</td><td>35.16</td><td>14.79</td><td>88.11</td><td>88.78</td><td>88.15</td><td>75.05</td><td>76.57</td><td>74.94</td></tr><tr><td>ChuBERT NBDW<sub>mixed</sub></td><td><u>94.39</u></td><td><strong>91.93</strong></td><td><strong>92.73</strong></td><td>94.22</td><td>90.02</td><td>91.71</td><td>86.36</td><td>85.50</td><td>85.73</td><td>97.77</td><td>95.30</td><td>96.35</td><td>34.58</td><td>12.21</td><td>90.36</td><td>90.94</td><td>90.52</td><td><u>78.61</u></td><td><u>79.32</u></td><td>78.63</td></tr><tr><td>CamemBERT NACHOS<sub>small</sub></td><td>81.44</td><td>81.39</td><td>80.96</td><td>97.74</td><td>78.08</td><td>78.70</td><td>80.59</td><td>79.88</td><td>80.04</td><td>95.64</td><td>91.57</td><td>92.46</td><td>32.87</td><td>13.76</td><td>67.56</td><td>77.48</td><td>71.10</td><td>55.45</td><td>62.34</td><td>57.43</td></tr><tr><td>PubMedBERT NACHOS<sub>small</sub></td><td>92.51</td><td><u>91.49</u></td><td>91.53</td><td><u>94.95</u></td><td><strong>92.55</strong></td><td><strong>93.62</strong></td><td>84.73</td><td>83.80</td><td>83.85</td><td>97.82</td><td>96.12</td><td>96.81</td><td>35.88</td><td>15.21</td><td>90.97</td><td>91.27</td><td>91.03</td><td><strong>82.03</strong></td><td><strong>81.71</strong></td><td><strong>81.73</strong></td></tr><tr><td>CamemBERT NBDW<sub>small</sub></td><td>82.35</td><td>81.59</td><td>81.57</td><td>78.14</td><td>76.38</td><td>77.12</td><td>79.44</td><td>79.79</td><td>79.25</td><td>95.98</td><td>92.11</td><td>93.18</td><td>27.73</td><td>11.89</td><td>53.44</td><td>73.11</td><td>61.75</td><td>48.71</td><td>61.33</td><td>53.05</td></tr></tbody></table>

orating our hypothesis about the effectiveness of cross-language knowledge transfer.

## 6.3 Performance on general-domain tasks

Table 8 gives the results obtained by all PLMs on general domain downstream tasks. These tasks come from Martin et al. (2020) who used them to evaluate the CamemBERT model. The first four are POS tagging tasks (GSD, SEQUOIA, SPOKEN and PARTUT), the last being a natural language inference task (XNLI).

All results of our models decrease in performance on all tasks. The most important drop is for the natural language inference task, with a performance of ChuBERT NBDW<sub>small</sub> almost 13% lower than CamemBERT 138 GB. We also observe that the specialized models in English are as efficient as our biomedical models in French. It seems quite clear from the previous observations that specialized models are difficult to generalize to other tasks, but that specialized information captured in one language could transfer to another language.

## 7 Conclusion

In this work, we proposed the first biomedical and clinical Transformer-based language models, based

on RoBERTa architecture, for French language. An extensive evaluation study of these specific models has been performed on an aggregated collection of diverse private and public medical tasks. Our open-source DrBERT models improved the state of the art in all medical tasks against both French general model (CamemBERT) and English medical ones (BioBERT, PubMedBERT and Clinical-BERT). In addition, we showed that pre-training on constrained resources (4 GB) of web-crawled medical makes it possible to compete with, and even frequently surpass, models trained with specialized data from medical reports.

Results also highlighted that continual pre-training on an existing domain-specific English model, here PubMedBERT, is a more viable solution than on a French domain-generalist model while targeting French biomedical downstream tasks. It needs to further investigate the performance of this approach using more data, similar to what we have done with DrBERT NACHOS<sub>large</sub>.

The NACHOS corpus as well as the models pre-trained and all the pre-training scripts<fnref n="3" /> have been publicly released online under a CC0 1.0 open-

<table><caption>Table 8: Performance on public domain-general downstream tasks. Best model in bold and second is underlined.</caption><thead><tr><th></th><th>GSD</th><th>SEQUOIA</th><th>SPOKEN</th><th>PARTUT</th><th>XNLI</th></tr></thead><tbody><tr><td>CamemBERT OSCAR 138 GB</td><td><strong>98.28</strong></td><td>98.68</td><td><strong>97.26</strong></td><td>97.70</td><td><strong>81.94</strong></td></tr><tr><td>CamemBERT OSCAR 4 GB</td><td>98.14</td><td><strong>99.18</strong></td><td><strong>97.57</strong></td><td>97.86</td><td><strong>81.76</strong></td></tr><tr><td>CamemBERT CCNET 4 GB</td><td><strong>98.18</strong></td><td><strong>98.92</strong></td><td>97.20</td><td><strong>97.92</strong></td><td>81.26</td></tr><tr><td>PubMedBERT</td><td>96.48</td><td>96.49</td><td>90.00</td><td>93.97</td><td>73.79</td></tr><tr><td>ClinicalBERT</td><td>96.49</td><td>96.31</td><td>89.60</td><td>93.17</td><td>70.57</td></tr><tr><td>BioBERT v1.1</td><td>97.32</td><td>96.54</td><td>91.81</td><td>94.52</td><td>71.54</td></tr><tr><td>DrBERT NACHOS<sub>large</sub></td><td>96.94</td><td>98.05</td><td>95.92</td><td>96.54</td><td>72.18</td></tr><tr><td>DrBERT NACHOS<sub>small</sub></td><td>97.17</td><td>98.21</td><td>96.38</td><td>96.45</td><td>72.86</td></tr><tr><td>ChuBERT NBDW<sub>small</sub></td><td>96.45</td><td>97.38</td><td>94.90</td><td>95.83</td><td>69.00</td></tr><tr><td>ChuBERT NBDW<sub>mixed</sub></td><td>97.18</td><td>98.10</td><td>96.43</td><td>96.33</td><td>72.32</td></tr><tr><td>CamemBERT NACHOS<sub>small</sub></td><td>97.63</td><td>96.90</td><td>91.12</td><td>94.00</td><td>71.26</td></tr><tr><td>PubMedBERT NACHOS<sub>small</sub></td><td>97.41</td><td>98.71</td><td>95.54</td><td>97.01</td><td>77.35</td></tr><tr><td>CamemBERT NBDW<sub>small</sub></td><td>97.55</td><td>96.26</td><td>89.17</td><td>91.34</td><td>72.73</td></tr></tbody></table>

<footnote n="3">³https://drbert.univ-avignon.fr/</footnote>
## 8 Ethical considerations

Concerning the risks and biases, all the freely available models pre-trained on NACHOS can supposedly be exposed to some of the concerns presented by the work of Bender et al. (2021) and Sheng et al. (2021) since some of the NACHOS sub-corpora quality might be lower than expected, specifically for non-governmental sources. When using a BERT-based biomedical language model, potential biases can be encountered including fairness, gendered language, limited representation and temporal correctness.

## 9 Limitations

It is important to mention some limitations of our work. Firstly, it would be wise to evaluate the impact of the tokenizer on the performance of the models to ensure that this is not the main reason for the observed performance gains.

Furthermore, we can not affirm in this study whether the medical domain transfer observed from English to French using continual pre-training on PubMedBERT can be generalized to other languages or other domains.

Finally, it is possible that training a ChuBERT model with more diverse private clinical data and in a larger quantity could have brought notable performance gains on private tasks.

A considerable amount of computational resources was used to conduct this study, since approximately 18,000 hours of GPU computation were used to create the 7 models presented here, as well as about 7,500 hours of GPU for debugging due to technical issues related to model configurations and poor performance, for a total of 25,500 hours. The total environmental cost, according to the Jean Zay supercomputer documentation <fnref n="4" /> is equivalent to 6,604,500 Wh or 376.45 kg CO2eq based on the carbon intensity of the energy grid mention by BLOOM environmental cost study also made on Jean Zay (Luccioni et al., 2022). This makes the present study difficult to reproduce and to transpose to other languages when limited material resources are available.

## Acknowledgments

This work was performed using HPC resources from GENCI-IDRIS (Grant 2022-AD011013061R1 and 2022-AD011013715) and from CCIPL (Centre de Calcul Intensif des Pays de la Loire). This work was financially supported by ANR AIBy4 (ANR-20-THIA-0011) and Zenidoc.

## References

Iz Beltagy, Kyle Lo, and Arman Cohan. 2019. *Scibert: Pretrained language model for scientific text*. In *EMNLP*.

Emily M. Bender, Timnit Gebru, Angelina McMillan-Major, and Shmargaret Shmitchell. 2021. *On the dangers of stochastic parrots: Can language models be too big? In Proceedings of the 2021 ACM Conference on Fairness, Accountability, and Transparency*, FAccT '21, page 610–623, New York, NY, USA. Association for Computing Machinery.

Olivier Bodenreider. 2004. *The unified medical language system (umls): Integrating biomedical terminology*. *Nucleic acids research*, 32:D267–70.

Casimiro Pio Carrino, Jordi Armengol-Estapé, Asier Gutiérrez-Fandiño, Joan Llop-Palao, Marc Pàmies, Aitor Gonzalez-Agirre, and Marta Villegas. 2021. *Biomedical and clinical language models for Spanish: On the benefits of domain-specific pretraining in a mid-resource scenario*.

Ilias Chalkidis, Manos Fergadiotis, Prodromos Malakasiotis, Nikolaos Aletras, and Ion Androutsopoulos. 2020. *LEGAL-BERT: the muppets straight out of law school*. *CoRR*, abs/2010.02559.

Clément Dalloux, Vincent Claveau, Natalia Grabar, Lucas Emanuel Silva Oliveira, Claudia Maria Cabral Moro, Yohan Bonescki Gumiel, and Debrah Ribeiro Carvalho. 2021. *Supervised learning for the detection of negation and of its scope in French and Brazilian Portuguese biomedical corpora*. *Natural Language Engineering*, 27(2):181–201.

Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. 2018. *BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding*. *CoRR*, abs/1810.04805.

Hicham El Boukkouri, Olivier Ferret, Thomas Lavergne, and Pierre Zweigenbaum. 2022. *Re-train or Train from Scratch? Comparing Pre-training Strategies of BERT in the Medical Domain*. In *Proceedings of the 13th Language Resources and Evaluation Conference (LREC)*, pages 2626–2633, Marseille, France. European Language Resources Association.

Jack FitzGerald, Christopher Hench, Charith Peris, Scott Mackie, Kay Rottmann, Ana Sanchez, Aaron

<footnote n="4">⁴http://www.idris.fr/media/jean-zay/jean-zay-consonheure-calcul.pdf</footnote>
Nash, Liam Urbach, Vishesh Kakarala, Richa Singh, Swetha Ranganath, Laurie Crist, Misha Britan, Wouter Leeuwis, Gokhan Tur, and Prem Natarajan. 2022. Massive: A 1m-example multilingual natural language understanding dataset with 51 typologically-diverse languages.

Natalia Grabar, Vincent Claveau, and Clément Dalloux. 2018. CAS: French Corpus with Clinical Cases. In Proceedings of the 9th International Workshop on Health Text Mining and Information Analysis (LOUHI), pages 1–7, Brussels, Belgium.

Yu Gu, Robert Tinn, Hao Cheng, Michael Lucas, Naoto Usuyama, Xiaodong Liu, Tristan Naumann, Jianfeng Gao, and Hoifung Poon. 2021. Domain-Specific Language Model Pretraining for Biomedical Natural Language Processing. ACM Transactions on Computing for Healthcare, 3(1):1–23.

Jeremy Howard and Sebastian Ruder. 2018. Universal Language Model Fine-tuning for Text Classification. In Proceedings of the 56th Annual Meeting of the Association for Computational Linguistics (ACL), pages 328–339, Melbourne, Australia. Association for Computational Linguistics.

Kexin Huang, Jaan Altosaar, and Rajesh Ranganath. 2019. Clinicalbert: Modeling clinical notes and predicting hospital readmission.

Taku Kudo and John Richardson. 2018. Sentencepiece: A simple and language independent subword tokenizer and detokenizer for neural text processing. CoRR, abs/1808.06226.

Yanis Labrak, Adrien Bazoge, Richard Dufour, Béatrice Daille, Pierre-Antoine Gourraud, Emmanuel Morin, and Mickael Rouvier. 2022. FrenchMedMCQA: A French Multiple-Choice Question Answering Dataset for Medical domain. In Proceedings of the 13th International Workshop on Health Text Mining and Information Analysis (LOUHI), Abou Dhabi, United Arab Emirates.

Jinhyuk Lee, Wonjin Yoon, Sungdong Kim, Donghyeon Kim, Sunkyu Kim, Chan Ho So, and Jaewoo Kang. 2019. BioBERT: a pre-trained biomedical language representation model for biomedical text mining. Bioinformatics.

Yinhan Liu, Myle Ott, Naman Goyal, Jingfei Du, Mandar Joshi, Danqi Chen, Omer Levy, Mike Lewis, Luke Zettlemoyer, and Veselin Stoyanov. 2019. Roberta: A robustly optimized BERT pretraining approach. CoRR, abs/1907.11692.

Alexandra Sasha Luccioni, Sylvain Viguier, and Anne-Laure Ligozat. 2022. Estimating the carbon footprint of bloom, a 176b parameter language model.

Louis Martin, Benjamin Muller, Pedro Javier Ortiz Suárez, Yoann Dupont, Laurent Romary, Éric de la Clergerie, Djamé Seddah, and Benoît Sagot. 2020. CamemBERT: a Tasty French Language Model. In

Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics (ACL), pages 7203–7219. Association for Computational Linguistics.

Paulius Micikevicius, Sharan Narang, Jonah Alben, Gregory F. Diamos, Erich Elsen, David García, Boris Ginsburg, Michael Houston, Oleksii Kuchaiev, Ganesh Venkatesh, and Hao Wu. 2017. Mixed precision training. CoRR, abs/1710.03740.

Aurélie Névéol, Cyril Grouin, Jérémy Leixa, Sophie Rosset, and Pierre Zweigenbaum. 2014. The quaero french medical corpus : A ressource for medical entity recognition and normalization.

Yifan Peng, Shankai Yan, and Zhiyong Lu. 2019. Transfer learning in biomedical natural language processing: An evaluation of bert and elmo on ten benchmarking datasets. In Proceedings of the 2019 Workshop on Biomedical Natural Language Processing (BioNLP 2019), pages 58–65.

Elisa Terumi Rubel Schneider, João Vitor Andrioli de Souza, Julien Knafou, Lucas Emanuel Silva e Oliveira, Jenny Copara, Yohan Bonescki Gumiel, Lucas Ferro Antunes de Oliveira, Emerson Cabrera Paraiso, Douglas Teodoro, and Cláudia Maria Cabral Moro Barra. 2020. BioBERTpt - a Portuguese neural language model for clinical named entity recognition. In Proceedings of the 3rd Clinical Natural Language Processing Workshop, pages 65–72, Online. Association for Computational Linguistics.

Rico Sennrich, Barry Haddow, and Alexandra Birch. 2016. Neural machine translation of rare words with subword units. In Proceedings of the 54th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 1715–1725, Berlin, Germany. Association for Computational Linguistics.

Emily Sheng, Kai-Wei Chang, Premkumar Natarajan, and Nanyun Peng. 2021. Societal biases in language generation: Progress and challenges.

Manjil Shrestha. 2021. Development of a language model for medical domain. masterthesis, Hochschule Rhein-Waal.

Jörg Tiedemann and Lars Nygaard. 2004. The OPUS corpus - parallel and free: http://logos.uio.no/opus. In Proceedings of the Fourth International Conference on Language Resources and Evaluation (LREC'04), Lisbon, Portugal. European Language Resources Association (ELRA).

Hazal Türkmen, Oğuz Dikenelli, Cenk Eraslan, and Mehmet Calli. 2022. Bioberturk: Exploring turkish biomedical language model development strategies in low resource setting.

Thomas Vakili, Anastasios Lamproudis, Aron Henriksson, and Hercules Dalianis. 2022. Downstream
task performance of BERT models pre-trained using automatically de-identified clinical data. In *Proceedings of the Thirteenth Language Resources and Evaluation Conference*, pages 4245–4252, Marseille, France. European Language Resources Association.

Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N. Gomez, Lukasz Kaiser, and Illia Polosukhin. 2017. Attention is all you need.

Thomas Wolf, Lysandre Debut, Victor Sanh, Julien Chaumond, Clement Delangue, Anthony Moi, Pierre Cistac, Tim Rault, Rémi Louf, Morgan Funtowicz, Joe Davison, Sam Shleifer, Patrick von Platen, Clara Ma, Yacine Jernite, Julien Plu, Canwen Xu, Teven Le Scao, Sylvain Gugger, Mariama Drame, Quentin Lhoest, and Alexander M. Rush. 2019. Huggingface's transformers: State-of-the-art natural language processing.

Yonghui Wu, Mike Schuster, Zhifeng Chen, Quoc V. Le, Mohammad Norouzi, Wolfgang Macherey, Maxim Krikun, Yuan Cao, Qin Gao, Klaus Macherey, Jeff Klingner, Apurva Shah, Melvin Johnson, Xiaobing Liu, Łukasz Kaiser, Stephan Gouws, Yoshikiyo Kato, Taku Kudo, Hideto Kazawa, Keith Stevens, George Kurian, Nishant Patil, Wei Wang, Cliff Young, Jason Smith, Jason Riesa, Alex Rudnick, Oriol Vinyals, Greg Corrado, Macduff Hughes, and Jeffrey Dean. 2016. Google's neural machine translation system: Bridging the gap between human and machine translation.

Xi Yang, Aokun Chen, Nima PourNejatian, Hoo Chang Shin, Kaleb E. Smith, Christopher Parisien, Colin Compas, Cheryl Martin, Anthony B. Costa, Mona G. Flores, Ying Zhang, Tanja Magoc, Christopher A. Harle, Gloria Lipori, Duane A. Mitchell, William R. Hogan, Elizabeth A. Shenkman, Jiang Bian, and Yonghui Wu. 2022. A large language model for electronic health records. *npj Digital Medicine*, 5(1):194.

Yi Yang, Mark Christopher Siy UY, and Allen Huang. 2020. Finbert: A pretrained language model for financial communications.

Yian Zhang, Alex Warstadt, Haau-Sing Li, and Samuel R. Bowman. 2020. When do you need billions of words of pretraining data?

Hongyin Zhu, Hao Peng, Zhiheng Lyu, Lei Hou, Juanzi Li, and Jinghui Xiao. 2021. Travelbert: Pre-training language model incorporating domain-specific heterogeneous knowledge into a unified representation.

# A Appendix

## A.1 Vocabularies Inter-coverage

<figure><img src="image_2.png" /><figcaption>Figure 1: Vocabularies Inter-coverage Matrix.</figcaption></figure>

As we can see in Figure 3, despite having similar performances, some of the models do not share a lot of mutual vocabulary.

## A.2 Models Stability

We observe during the evaluation phase that most of the models based on continual pre-training strategy from CamemBERT OSCAR 138 GB are suffering from bad consistency and stability during fine-tuning, which translates into fluctuation in performance between runs.

<figure><img src="image_3.png" /><figcaption>Figure 2: Box plot for each model.</figcaption></figure>

We also notice during PubMedBERT NACHOS<sub>small</sub> pre-training that the model
loss is globally stable during almost all the duration of the pre-training, until reaching the step 71,000, where the loss fall down until touching down zero at step 72,500.

<figure><img src="image_4.png" /><figcaption>Figure 3: PubMedBERT NACHOS<sub>small</sub> loss.</figcaption></figure>