# How Important Is Tokenization in French Medical Masked Language Models?

Yanis Labrak<fnref n="1" />,², Adrien Bazoge³

Béatrice Daille³, Mickael Rouvier<fnref n="1" />, Richard Dufour<fnref n="1" />,³

¹ LIA, Avignon University ² Zenidoc

³ Nantes Université, École Centrale Nantes, CNRS, LS2N, UMR 6004, F-44000 Nantes, France
{first.last}@{univ-avignon.fr, univ-nantes.fr}

## Abstract

Subword tokenization has become the prevailing standard in the field of natural language processing (NLP) over recent years, primarily due to the widespread utilization of pre-trained language models. This shift began with Byte-Pair Encoding (BPE) and was later followed by the adoption of SentencePiece and WordPiece. While subword tokenization consistently outperforms character and word-level tokenization, the precise factors contributing to its success remain unclear. Key aspects such as the optimal segmentation granularity for diverse tasks and languages, the influence of data sources on tokenizers, and the role of morphological information in Indo-European languages remain insufficiently explored. This is particularly pertinent for biomedical terminology, characterized by specific rules governing morpheme combinations. Despite the agglutinative nature of biomedical terminology, existing language models do not explicitly incorporate this knowledge, leading to inconsistent tokenization strategies for common terms. In this paper, we seek to delve into the complexities of subword tokenization in French biomedical domain across a variety of NLP tasks and pinpoint areas where further enhancements can be made. We analyze classical tokenization algorithms, including BPE and SentencePiece, and introduce an original tokenization strategy that integrates morpheme-enriched word segmentation into existing tokenization methods.

**Keywords:** Tokenization, Morphemes, Language Model, Biomedical, SentencePiece, BPE, RoBERTa, Transformers

## 1. Introduction

Word tokenization into subword units is a longstanding challenge in the field of natural language processing (NLP), initially conceived to address out-of-vocabulary words in language modeling (Larson, 2001; Bazzi and Glass, 2002; Szoke et al., 2008). In recent years, this strategy of splitting words into smaller units has gained prominence, primarily driven by the widespread adoption of pre-trained language models (PLMs) such as BERT (Devlin et al., 2019) and GPT (Brown et al., 2020). This shift began with statistical tokenizers, particularly Byte-Pair Encoding (BPE) (Gage, 1994), as introduced in BERT (Sennrich et al., 2016). It was later extended by other data-driven variants, such as SentencePiece (SP) (Kudo and Richardson, 2018) and WordPiece (Devlin et al., 2019).

While empirical evidence consistently demonstrates that subword tokenization outperforms character and word-level tokenization (Wang et al., 2017; Wu et al., 2016), the precise reasons behind this success remain not fully understood. Some studies have explored the impact of segmentation granularity on subword performance (Samuel and Øvrelid, 2023; Novotný et al., 2021), suggesting that each task or language may have its own optimal granularity for maximizing performance. However, other factors, such as the influence of data sources used to construct tokenizers or the role of morphological information, require more comprehensive investigation.

In the context of Indo-European languages, particularly in French, words are composed of a series of morphemes<fnref n="1" /> (Touratier, 2012). These morphemes can be categorized as either lexical or grammatical and are analogous to the subword units idea previously mentioned, but here adhere to well-defined linguistic rules.

In specialized domains, such as medical one, meaningful morphemes follow construction rules from Greek and Latin languages. These rules help medical professionals deduce the meanings of unfamiliar terms and remember complex terminology effectively. Despite the agglutinative nature of biomedical terminology, existing PLMs do not explicitly integrate this knowledge into their tokenization processes since they only rely on statistical tokenizers (BPE, SentencePiece, etc.). As a result, common terms are inconsistently tokenized into arbitrary subwords in these models.

In this paper, we investigate the impact of word tokenization strategies in the French biomedical domain and their effectiveness on downstream NLP tasks. Our study delves into the nuances of different tokenization algorithms, aiming to understand why subword tokenization strategies, such as BPE and SentencePiece, outperform other methods (Kudo, 2018). We also identify areas for further

<footnote n="1">¹In linguistics, a morpheme is defined as the smallest unit of meaning within a word.</footnote>
optimization and provide a comprehensive analysis of their performance on a large set of 23 diverse French biomedical NLP tasks, such as named entity recognition (NER), multi-label classification (CLS), or semantic textual similarity (STS). Finally, we propose an original tokenization strategy that integrates morpheme-enriched word segmentation into existing tokenization algorithms. The latter is included in the comparison of tokenizers and makes it possible to study the contribution of subword units constructed from linguistic rules.

Our contributions are as follows:

* We introduce an original tokenization strategy that integrates manually defined morphemes into statistical tokenization algorithms.

* We analyze the ability of statistical tokenizers to segment words regarding their real linguistic segmentation.

* We provide both qualitative and quantitative analyses to assess how word tokenization approaches (statistical methods vs. morpheme-enriched variants) and the data source on which they are trained impact the performance of BERT-based language models.

* We explore the relationship between tokenization granularity and its impact on performance in various downstream NLP tasks.

The morpheme-enriched tokenization strategy, experiment reproduction scripts, and resulting BERT-based PLMs are freely available under the MIT license on GitHub and Hugging Face<fnref n="2" />

## 2. Related works

Recent research into domain-specific language models has shown that utilizing specialized data during pre-training significantly enhances model performance in that domain. Various strategies, with varying proportions of in-domain and out-of-domain training data, have been proposed across diverse fields, including biomedicine (Lee et al., 2019; Gu et al., 2021; El Boukkouri et al., 2022; Labrak et al., 2023), scientific research (Beltagy et al., 2019) and clinical (Alsentzer et al., 2019).

In the context of biomedical-specific language models, it is widely recognized that training models from scratch using in-domain corpora (Gu et al., 2021) yields noticeable performance improvements compared to other pre-training strategies. The authors also demonstrated the benefits of using

domain-specific tokenizers generated through conventional statistical tokenization construction techniques, such as WordPiece (Schuster and Nakajima, 2012), SentencePiece (Kudo and Richardson, 2018), and BPE (Sennrich et al., 2016), on an in-domain corpus, resulting in improved performance in downstream tasks.

Although statistical-based tokenization algorithms are the predominant method employed in recent biomedical language models, some studies have raised questions about the effectiveness of this approach and its suitability for specific downstream tasks or languages (Mielke et al., 2021; Novotný et al., 2021). As a result of these inquiries, various methods for improving tokenization have emerged, some involving training models from scratch (Kudo, 2018), while others do not (Hofmann et al., 2022; Fan and Sun, 2023). One such method involves incorporating linguistic knowledge during the tokenization process by utilizing morphemes (Fujii et al., 2023; Pan et al., 2020; Chen and Fazio, 2021; Toraman et al., 2023), with the aim of mimicking how humans learn and understand languages. However, there have been fewer contributions in the context of biomedical domains and Indo-European languages (Jimenez Gutierrez et al., 2023), despite these fields being highly dependent on an agglutinative terminology.

## 3. Tokenization Strategies

In this section, we provide a brief overview of the two studied statistical-based tokenization approaches (Section 3.1), followed by the description of our original approach that integrates linguistic knowledge through morphemes into existing tokenizers algorithms (Section 3.2).

### 3.1. Statistical Tokenization Algorithms

In this study, we compare two statistical-based tokenization methods, BPE and SentencePiece. BPE begins with individual characters and progressively combines them into subword pairs based on their frequency in the training data. In contrast, SentencePiece employs two subword segmentation algorithms, Unigrams and BPE, offering flexibility in terms of segmentation granularity. While SentencePiece is widely used in French biomedical models (Touchent et al., 2023; Labrak et al., 2023; Copara et al., 2020; Berhe et al., 2023), its appropriateness for a specific language and domain may vary, potentially leading to suboptimal subword segmentation.

### 3.2. Morpheme-enriched Tokenization

In our study, focusing on improving the modeling of specialized medical terminology in the medical

<footnote n="2">² Repositories are currently private to respect the double-blind review process, but an anonymized version is available: https://anonymous.4open.science/r/BioMedTok-D665/</footnote>
field and reducing the impact of unseen words during model pre-training, our primary emphasis is on lexical morphemes (Touratier, 2012). To achieve this, we created a manual list of around 600 frequently used lexical morphemes in the French medical domain, sourced from the book by Cottez (1980). Examples of these morphemes include terms like céphal-, clinico-, -thérapie, thoraco-, -ome and -gène.

We trained our morpheme-enriched tokenizers by modifying both the BPE and SentencePiece algorithms. During training, we introduced a predefined list of language-specific morphemes as tokens. These morphemes were enforced selections by the tokenizer when encountered, while the remaining text underwent the standard tokenization process of the chosen algorithm. This approach enabled us to combine traditional BPE and SentencePiece tokenizations with morpheme tokens, mitigating issues related to unseen words during training.

## 4. Experimental Protocol

In this section, we outline the experimental approach used to evaluate the impact of tokenization strategies on French biomedical PLMs. Firstly, in Section 4.1, we present the set of 23 selected biomedical NLP downstream tasks used in our study. Next, we describe the different training data sources employed to train the statistical tokenizers in Section 4.2. Following this, in Section 4.3, we explain the training procedure for the chosen BERT-based model architecture. Finally, in Section 4.4, we provide a comprehensive description of the evaluation methodology used to assess the performance of these models.

### 4.1. Downstream Tasks

We summarize the datasets of the 23 NLP biomedical downstream tasks from DrBenchmark (Labrak et al., 2024), including NER, part-of-speech (POS) tagging, STS and classification.

**DEFT-2020** (Cardon et al., 2020) is a dataset featured in the 2020 edition of the annual French Text Mining Challenge, known as DEFT. It encompasses clinical cases, encyclopedia, and drug labels, all of which have been annotated for two specific tasks: (i) assessing textual similarity and (ii) performing multi-class classification. The first task is geared towards determining the degree of similarity between pairs of sentences, with a scale ranging from 0 to 5 and involves 1,010 sample pairs. The second task involves identifying, for a given sentence, which among three provided sentences is the most similar. There are 1,102 samples included in this task.

**DEFT-2021** (Grouin et al., 2021) is a subset of 275 clinical cases taken from the 2019 edition of DEFT. This dataset is manually annotated in two tasks: (i) multi-label classification with 275 samples and (ii) NER. The multi-label classification task focuses on identifying the patient's clinical profile based on the diseases, signs, or symptoms mentioned in the clinical cases with 4,712 samples. The dataset is annotated with 23 axes derived from Chapter C of the Medical Subject Headings (MeSH). The second task involves fine-grained information extraction for 13 entities.

**E3C** (Magnini et al., 2020) is a multilingual collection of clinical cases annotated for Named Entity Recognition (NER). It encompasses two types of annotations: (i) clinical entities and (ii) temporal information and factuality. While this dataset spans five languages, our evaluation focuses on the French portion. Since the dataset does not come with predefined subsets for its 1,402 samples, we conducted random splits of 70% for training, 10% for validation, and 20% for testing, as outlined in Table 1.

<table><caption>Table 1: Description of the sources for E3C.</caption><thead><tr><td>Subset</td><td>Train</td><td>Validation</td><td>Test</td></tr></thead><tbody><tr><td>Clinical</td><td>87.38% of L2</td><td>12.62% of L2</td><td>100% of L1</td></tr><tr><td>Temporal</td><td>70% of L1</td><td>10% of L1</td><td>20% of L1</td></tr></tbody></table>

**The QUAERO French Medical Corpus** (Névéol et al., 2014), simply referred to as QUAERO in this paper, contains annotated entities and concepts for NER tasks. The dataset covers two text genres (drug leaflets and biomedical titles). 10 entity categories corresponding to the UMLS Semantic Groups (Lindberg et al., 1993) were annotated, for a total of 26,409 entities, which were mapped to 5,797 unique UMLS concepts. Due to the presence of nested entities, we opted to simplify the evaluation process by retaining only annotations at the higher granularity level, following a similar approach to the one described in Touchent et al. (2023), which translates into an average loss of 6.06% of the annotations on EMEA and 8.90% on MEDLINE. Additionally, considering that some documents from EMEA exceed the maximum input sequence length that most current language models can handle, we decided to split these documents into sentences.

**MorFITT** (Labrak et al., 2023b) is a multi-label dataset that has been annotated with medical specialties. It comprises 3,624 biomedical abstracts sourced from PMC Open Access. These abstracts have been annotated across 12 distinct medical specialties, resulting in a total of 5,116 annotations.
**Mantra-GSC** (Kors et al., 2015) is a multilingual dataset annotated in biomedical NER for five languages, however we focused only on the French subset. It covers three sources (EMEA, Medline and Patents) and use two distinct annotation schemes. These sources encompass diverse types of documents, including biomedical abstracts/titles, drug labels, and patents. To maintain evaluation uniformity, we randomly divided the dataset into three subsets: 70% for training, 10% for validation, and 20% for testing.

**CLISTER** (Hiebel et al., 2022) is a collection of French clinical case sentence pairs used for Semantic Textual Similarity (STS) evaluation. It consists of 1,000 sentence pairs, manually annotated by multiple annotators who assigned similarity scores ranging from 0 to 5 for each pair. These individual scores were then averaged to derive a floating-point number that represents the overall similarity of the two sentences.

**CAS** (Grabar et al., 2018) dataset comprises 3,790 clinical cases that underwent POS tagging with 31 different classes, using automatic tagging through the Tagex tool<fnref n="3" />, achieving a 98% precision rate in comparison to manual annotations. This dataset involves tasks like classifying clinical cases for negation and uncertainty, as well as named-entity recognition for identifying markers of negations and speculation within medical histories and patient care. To create subsets, a random split was applied, allocating 70% for training, 10% for validation, and 20% for testing since predefined subsets were not provided.

**ESSAI** (Dalloux et al., 2021) consists of 7,247 clinical trial protocols that have been annotated with 41 POS tags using the TreeTagger tool (Schmid, 1994). It does also contain a classification and two named-entity recognition tasks similar to those from CAS dataset. As the dataset was not initially separated into three distinct subsets, we opted to apply the same processing methodology as we did for CAS dataset.

**PxCorpus** (Kocabiyikoglu et al., 2022) is a dataset designed for spoken language understanding in the medical domain, specifically focusing on transcripts related to drug prescriptions. It comprises 4 hours of transcribed dialogues, amounting to 1,981 recordings. These dialogues have been meticulously transcribed and semantically annotated. The primary task involves categorizing the textual utterances into one of four intent classes (prescribe, replace, negate, none). The second

task pertains to NER, where each word in a sequence is classified into one of 38 classes, including categories such as drug, dose, or mode.

## 4.2. Tokenizers Data Sources

To ensure a fair and comprehensive comparison of training data sources used by the statistical tokenizers, we carefully curated a 1GB subset of raw, lowercase text data from a variety of sources, including NACHOS (Labrak et al., 2023), PubMed Central, CC100 (Wenzek et al., 2020), and the French Wikipedia. We then constructed tokenizers using both tokenization algorithms, resulting in a total of 16 tokenizers: 8 with the integration of morphemes and 8 without. These specific data sources were chosen for their diversity: NACHOS focuses on French biomedical content, PubMed Central on English biomedical content, Wikipedia on general French language, and CC100 on general multilingual content. Each tokenizer was configured with a vocabulary size of 32k tokens, consistent with the original hyperparameters used in other French biomedical models such as Camembert-BIO (Touchent et al., 2023) and DrBERT (Labrak et al., 2023).

## 4.3. Language Model Pre-Training

To assess the impact of introducing morphemes into tokenizers on the pre-training process of biomedical language models, we conducted pre-training from scratch using the 16 tokenizer combinations (see Section 4.2). Our choice of architecture was RoBERTa (Liu et al., 2020), which is based on the masked language modeling objective and configured with standard token masking percentages as introduced by the authors.

For the PLMs training data, we utilized the NACHOS corpus created by (Labrak et al., 2023). This corpus, already pre-processed and converted to lowercase, is consistent with the data sources used for training the tokenizers. It comprises 1.1 billion words, equivalent to 7.4GB of raw text data, sourced from a wide range of online resources focusing on the French biomedical and clinical domains.

The pre-training process was conducted uniformly across all models, employing the same hyperparameters and executed over a 20-hour period. We harnessed the computational power of 32 V100 32GB GPUs available on the Jean-Zay supercomputer for this purpose. By maintaining consistent procedures and employing a fixed seed to mitigate randomness during training, we ensured the reliability and reproducibility of our experiments.

<footnote n="3">³https://allgo.inria.fr/app/tagex</footnote>
<table><caption>Table 2: Performance of the tokenization algorithms and different data sources used to train tokenizers (top). Average performance per type of tasks is also reported (bottom). w/o and w/ denote models without and with morphemes. Best models are in bold, and the second-best are underlined. Statistical significance is determined using Student's t-test, where * indicates p < 0.05, and ** p < 0.01.</caption><thead><tr><th rowspan="3">Dataset</th><th rowspan="3">Task</th><th rowspan="3">Metric</th><th colspan="8">BPE</th><th colspan="8">SentencePiece</th></tr><tr><th colspan="2">NACHOS</th><th colspan="2">PubMed</th><th colspan="2">CC100</th><th colspan="2">Wiki</th><th colspan="2">NACHOS</th><th colspan="2">PubMed</th><th colspan="2">CC100</th><th colspan="2">Wiki</th></tr><tr><th>w/o</th><th>w/</th><th>w/o</th><th>w/</th><th>w/o</th><th>w/</th><th>w/o</th><th>w/</th><th>w/o</th><th>w/</th><th>w/o</th><th>w/</th><th>w/o</th><th>w/</th><th>w/o</th><th>w/</th></tr></thead><tbody><tr><td rowspan="4">CAS</td><td>CLS</td><td>F1</td><td>94.2*</td><td>94.9</td><td>94.7</td><td>94.2*</td><td>95.2</td><td><strong>95.3</strong></td><td>94.8</td><td>94.8</td><td>94.8</td><td>94.7*</td><td>93.4**</td><td>93.6**</td><td>94.4</td><td>94.1</td><td><strong>95.3*</strong></td><td>95.1**</td></tr><tr><td>NER Neg</td><td>SeqEval</td><td><strong>87.0</strong></td><td>83.3*</td><td>82.4**</td><td>81.3**</td><td>84.9</td><td>84.2</td><td>84.7</td><td>84.5*</td><td>86.1</td><td>86.4</td><td>83.6*</td><td>83.9</td><td>85.4</td><td>84.2**</td><td>85.6</td><td>83.2</td></tr><tr><td>NER Spec</td><td>SeqEval</td><td>30.3*</td><td>30.6</td><td><strong>35.0</strong></td><td>28.2*</td><td>34.6</td><td>32.0</td><td>34.4</td><td>34.0</td><td><strong>36.1</strong></td><td>29.8</td><td>28.4*</td><td>22.2**</td><td>31.9</td><td>28.7*</td><td>32.1</td><td>27.0*</td></tr><tr><td>POS</td><td>SeqEval</td><td>97.0**</td><td>96.9**</td><td>97.1</td><td>96.9**</td><td>97.1*</td><td>97.0**</td><td><strong>97.2</strong></td><td>96.9**</td><td>97.1*</td><td>97.0*</td><td>96.9**</td><td>96.9**</td><td>97.1</td><td>97.1</td><td>97.1</td><td>97.1</td></tr><tr><td rowspan="2">PxCorpus</td><td>CLS</td><td>F1</td><td><strong>94.8</strong></td><td>94.2</td><td>93.6</td><td>93.9</td><td>94.2</td><td>94.6</td><td>93.4</td><td>93.7</td><td><strong>94.9</strong></td><td>94.1</td><td>94.8</td><td>94.1</td><td>94.8</td><td>93.7</td><td>93.7</td><td>94.5</td></tr><tr><td>NER</td><td>SeqEval</td><td>95.9</td><td>95.9</td><td>95.9</td><td>95.9</td><td>96.1</td><td>96.0</td><td>96.2</td><td>95.9</td><td>96.1</td><td>96.1</td><td>96.0</td><td>96.1</td><td>95.9</td><td>96.1</td><td><strong>96.2</strong></td><td>96.1</td></tr><tr><td rowspan="2">DEFT2020</td><td>STS</td><td>MSE</td><td>0.71</td><td>0.71</td><td>0.64*</td><td><strong>0.75</strong></td><td>0.70</td><td>0.67</td><td>0.71</td><td>0.69</td><td>0.72</td><td>0.71</td><td>0.63**</td><td>0.63</td><td>0.70</td><td>0.67*</td><td>0.70</td><td>0.67*</td></tr><tr><td>CLS</td><td>F1</td><td><strong>91.0</strong></td><td>85.9</td><td>57.6**</td><td>73.7</td><td>79.5</td><td>76.3</td><td>77.1</td><td>66.0</td><td>83.0</td><td>85.3</td><td>80.9</td><td>66.7**</td><td>61.1*</td><td>66.3*</td><td>75.0*</td><td>77.4*</td></tr><tr><td rowspan="2">MORFITT</td><td>CLS</td><td>F1</td><td>68.6**</td><td>68.0**</td><td>66.5**</td><td>65.9**</td><td>68.4**</td><td>67.0**</td><td>68.7</td><td>67.3**</td><td><strong>69.6</strong></td><td>68.8*</td><td>66.8**</td><td>66.2**</td><td>68.2</td><td>67.5**</td><td>69.1**</td><td>67.7**</td></tr><tr><td>NER Clinical</td><td>SeqEval</td><td><strong>54.2</strong></td><td>53.1</td><td>52.4</td><td>48.6**</td><td>52.7</td><td>51.3**</td><td>51.1*</td><td>52.0*</td><td>54.2</td><td>52.4</td><td>52.1</td><td>51.1**</td><td>53.8</td><td>52.5*</td><td>53.2</td><td>51.7</td></tr><tr><td rowspan="2">E3C</td><td>NER Temporal</td><td>SeqEval</td><td>82.0</td><td>81.2</td><td>80.9**</td><td>80.0**</td><td>81.8</td><td>81.2</td><td><strong>82.3</strong></td><td>80.6**</td><td>82.1</td><td>81.6</td><td>80.3**</td><td>79.8**</td><td>80.6**</td><td>81.1**</td><td>81.6*</td><td>81.73*</td></tr><tr><td>CLS</td><td>MSE</td><td>0.63*</td><td>0.63</td><td>0.63</td><td>0.60**</td><td>0.65</td><td>0.63</td><td>0.62**</td><td><strong>0.66</strong></td><td>0.61*</td><td>0.64</td><td>0.61**</td><td>0.62**</td><td>0.62</td><td>0.60*</td><td>0.64**</td><td>0.63**</td></tr><tr><td rowspan="2">CLISTER</td><td>STS</td><td>MSE</td><td>60.3</td><td>59.0**</td><td>58.1**</td><td>56.2**</td><td>59.4**</td><td>59.2**</td><td>60.1**</td><td>59.1**</td><td><strong>61.3</strong></td><td>60.1*</td><td>57.0**</td><td>56.6**</td><td>59.2**</td><td>59.9**</td><td>59.3**</td><td>58.9**</td></tr><tr><td>CLS</td><td>F1</td><td>32.9</td><td>34.5*</td><td>33.4</td><td>32.3</td><td>34.5*</td><td>33.9</td><td>34.2</td><td>32.9</td><td>34.3</td><td>33.1</td><td>34.3</td><td>33.1</td><td>31.0</td><td>31.9*</td><td>34.2</td><td><strong>34.9</strong></td></tr><tr><td rowspan="4">ESSAI</td><td>NER Spec</td><td>SeqEval</td><td>60.5</td><td>60.9</td><td>56.4*</td><td>59.2</td><td>57.9</td><td>61.5</td><td>63.6</td><td>57.4</td><td>63.9</td><td>62.8</td><td>57.6</td><td>55.7*</td><td><strong>64.6</strong></td><td>62.0</td><td>61.4</td><td>63.1</td></tr><tr><td>POS</td><td>SeqEval</td><td><strong>98.4</strong></td><td>98.3</td><td>98.3</td><td>98.2**</td><td>98.4</td><td>98.4</td><td>98.3</td><td>98.3</td><td>98.4</td><td>98.4</td><td>98.3</td><td>98.2*</td><td><strong>98.4</strong></td><td>98.3</td><td>98.3*</td><td>98.3</td></tr><tr><td>NER Neg</td><td>SeqEval</td><td>83.0</td><td>83.4</td><td>79.3</td><td>76.4</td><td>82.2</td><td>83.2</td><td>81.8</td><td><strong>84.2</strong></td><td>81.3</td><td>84.0*</td><td>80.2</td><td>81.1</td><td>83.2</td><td><strong>84.2</strong></td><td>82.1</td><td>79.6*</td></tr><tr><td>CLS</td><td>F1</td><td>97.3</td><td>97.1*</td><td>97.4</td><td>96.6**</td><td>97.4</td><td>96.7**</td><td>97.4</td><td>97.0**</td><td>97.3</td><td>97.3</td><td><strong>97.5</strong></td><td>97.2*</td><td>97.0</td><td>97.0</td><td><strong>97.5</strong></td><td>97.0*</td></tr><tr><td rowspan="2">QUAERO</td><td>NER Medline</td><td>SeqEval</td><td>57.7</td><td>56.2**</td><td>55.4**</td><td>53.6**</td><td><strong>57.9</strong></td><td>55.0**</td><td>57.3</td><td>56.4**</td><td><strong>58.2</strong></td><td>55.5**</td><td>54.8**</td><td>52.9**</td><td>57.5*</td><td>55.8**</td><td>56.9</td><td>54.9**</td></tr><tr><td>NER EMEA</td><td>SeqEval</td><td>65.6</td><td>65.1</td><td>63.9</td><td>63.1**</td><td><strong>62.1**</strong></td><td>62.7*</td><td>63.1**</td><td>62.6*</td><td>65.5</td><td><strong>65.9</strong></td><td>62.6**</td><td>63.8*</td><td>62.8**</td><td>63.1*</td><td>62.7*</td><td>62.0**</td></tr><tr><td rowspan="3">MantraGSC</td><td>NER EMEA</td><td>SeqEval</td><td>60.9</td><td>63.9</td><td>58.2*</td><td>60.6*</td><td><strong>69.3</strong></td><td>63.0</td><td>61.9*</td><td>62.3**</td><td>66.9</td><td>62.5*</td><td>56.8**</td><td>60.3</td><td>60.8*</td><td>59.5</td><td>64.0*</td><td>63.9**</td></tr><tr><td>NER Medline</td><td>SeqEval</td><td>41.4*</td><td>42.9</td><td>39.3</td><td>36.2**</td><td>44.3</td><td>41.2</td><td>43.8</td><td>40.8*</td><td>41.9</td><td>39.5*</td><td>36.4**</td><td>37.8</td><td><strong>46.4*</strong></td><td>39.9</td><td><strong>47.1</strong></td><td>36.1*</td></tr><tr><td>NER Patents</td><td>SeqEval</td><td>52.1*</td><td>53.3*</td><td><strong>57.0</strong></td><td>50.2*</td><td>57.0</td><td>53.9</td><td>53.6</td><td>52.3*</td><td>52.0</td><td>49.6*</td><td>50.7**</td><td>49.4</td><td>52.8*</td><td>48.0</td><td>50.6*</td><td>47.8*</td></tr><tr><td colspan="2"></td><td colspan="16">Average performances per tasks</td></tr><tr><td>CLS</td><td>F1</td><td><strong>79.80</strong></td><td>79.10</td><td><strong>73.87</strong></td><td>76.10</td><td>78.20</td><td>77.30</td><td>77.60</td><td>75.28</td><td>78.98</td><td>78.88</td><td>77.95</td><td>75.15</td><td>74.42</td><td>75.08</td><td>77.47</td><td>77.77</td></tr><tr><td>NER</td><td>SeqEval</td><td>63.92</td><td>63.75</td><td>62.63</td><td>60.73</td><td>64.63</td><td>63.42</td><td>64.15</td><td>63.24</td><td><strong>65.05</strong></td><td>63.55</td><td>61.27</td><td>60.82</td><td>64.22</td><td>62.69</td><td>64.06</td><td>62.00</td></tr><tr><td>POS</td><td>SeqEval</td><td>97.70</td><td>97.60</td><td>97.70</td><td><strong>97.55</strong></td><td><strong>97.75</strong></td><td>97.70</td><td>97.75</td><td>97.60</td><td>97.75</td><td>97.70</td><td>97.60</td><td>97.55</td><td>97.75</td><td>97.70</td><td>97.70</td><td>97.70</td></tr><tr><td>STS</td><td>MSE</td><td>0.67</td><td>0.67</td><td>0.64</td><td>0.68</td><td><strong>0.68</strong></td><td>0.65</td><td>0.67</td><td>0.68</td><td>0.67</td><td>0.68</td><td>0.62</td><td>0.63</td><td>0.66</td><td>0.64</td><td>0.67</td><td>0.65</td></tr></tbody></table>

## 4.4. Evaluation

All models undergo fine-tuning following a standardized protocol with identical hyperparameters for each downstream task, enabling a focused evaluation of tokenizers. We ensure robustness and reliability by averaging the results across four independent runs and performing statistical significance assessments using Student's t-test.

For consistent comparisons, especially in sequence-to-sequence tasks like POS tagging and NER, we employ the SeqEval (Nakayama, 2018) metric in conjunction with the IOB2 format. To align with established practices (Touchent et al., 2023), our models are trained to predict only the label for the initial token of each word.

# 5. Results and Discussions

In this section, we present the results of our tokenization strategies on various biomedical NLP tasks, with a focus on key aspects. We investigate the impact of tokenization granularity (Section 5.1), the introduction of morphological information during tokenizer construction (Section 5.2), and the influence of data sources on tokenizers, including token sparsity, morpheme coverage, and the overall performance of different tokenization algorithms

(Section 5.3).

Table 2 summarizes the performance of the BPE and SentencePiece strategies, both with (w/) and without our morpheme-enriched approach (w/o), across various French biomedical downstream tasks. Average performance per task type is also provided for clarity. It's worth noting that, before delving into detailed analysis, there is no consistent tokenization strategy that consistently yields the best results in all tasks, whether it employs a purely statistical algorithm or a statistical approach coupled with morpheme enrichment.

## 5.1. Impact of tokenization granularity

To assess the impact of tokenization granularity, Table 3 presents the average number of sub-word units per word for each tokenization strategy and data source used in the studied tasks. While deriving overarching conclusions from these results can be challenging, we calculated Pearson correlation ($\rho$) between models performances on the downstream tasks from Table 2 and the corresponding average number of sub-word units per word. These correlation scores range from -1 to +1, where -1 indicates a complete negative linear correlation, 0 represents no correlation, and +1 signifies a strong positive correlation. In the context of tokenization,
<table><caption>Table 3: Average number of sub-word units per word for each tokenization strategy and data source training. Their Pearson correlation (ρ) with each task performance is reported (last column). Cells colored in red correspond to lower performing models, while those in green represent higher ones. The last row represents the relative difference in terms of average subwords per word compared to the NACHOS BPE without morpheme baseline. w/o and w/ denote models without and with morphemes.</caption><thead><tr><th rowspan="3">Corpus</th><th rowspan="3">Task</th><th colspan="7">BPE</th><th colspan="7">SentencePiece</th></tr><tr><th colspan="2">NACHOS</th><th colspan="2">PubMed</th><th colspan="2">CC100</th><th colspan="2">Wikipedia</th><th colspan="2">NACHOS</th><th colspan="2">PubMed</th><th colspan="2">CC100</th><th colspan="2">Wikipedia</th></tr><tr><th>w/o</th><th>w/</th><th>w/o</th><th>w/</th><th>w/o</th><th>w/</th><th>w/o</th><th>w/</th><th>w/o</th><th>w/</th><th>w/o</th><th>w/</th><th>w/o</th><th>w/</th><th>w/o</th><th>w/</th><th>ρ</th></tr></thead><tbody><tr><td rowspan="4">CAS</td><td>CLS</td><td>1.32</td><td>1.38</td><td>2.20</td><td>2.13</td><td>1.49</td><td>1.49</td><td>1.51</td><td>1.50</td><td>1.32</td><td>1.45</td><td>2.18</td><td>2.15</td><td>1.49</td><td>1.56</td><td>1.51</td><td>1.57</td><td>-0.62</td></tr><tr><td>NER Neg</td><td>1.32</td><td>1.38</td><td>2.20</td><td>2.13</td><td>1.49</td><td>1.49</td><td>1.51</td><td>1.50</td><td>1.32</td><td>1.45</td><td>2.18</td><td>2.15</td><td>1.49</td><td>1.56</td><td>1.51</td><td>1.57</td><td>-0.70</td></tr><tr><td>NER Spec</td><td>1.32</td><td>1.38</td><td>2.20</td><td>2.13</td><td>1.49</td><td>1.49</td><td>1.51</td><td>1.50</td><td>1.32</td><td>1.45</td><td>2.18</td><td>2.15</td><td>1.49</td><td>1.56</td><td>1.51</td><td>1.57</td><td>-0.42</td></tr><tr><td>POS</td><td>1.32</td><td>1.38</td><td>2.20</td><td>2.13</td><td>1.49</td><td>1.49</td><td>1.51</td><td>1.50</td><td>1.32</td><td>1.45</td><td>2.18</td><td>2.15</td><td>1.49</td><td>1.56</td><td>1.51</td><td>1.57</td><td>-0.36</td></tr><tr><td rowspan="2">PxCorpus</td><td>CLS</td><td>1.54</td><td>1.62</td><td>2.26</td><td>2.27</td><td>1.76</td><td>1.72</td><td>1.73</td><td>1.72</td><td>1.54</td><td>1.67</td><td>2.24</td><td>2.30</td><td>1.72</td><td>1.77</td><td>1.77</td><td>1.82</td><td>-0.22</td></tr><tr><td>NER</td><td>1.54</td><td>1.62</td><td>2.26</td><td>2.27</td><td>1.76</td><td>1.72</td><td>1.73</td><td>1.72</td><td>1.54</td><td>1.67</td><td>2.24</td><td>2.30</td><td>1.72</td><td>1.77</td><td>1.77</td><td>1.82</td><td>-0.22</td></tr><tr><td rowspan="2">DEFT2020</td><td>STS</td><td>1.41</td><td>1.45</td><td>2.27</td><td>2.24</td><td>1.42</td><td>1.45</td><td>1.43</td><td>1.45</td><td>1.41</td><td>1.49</td><td>2.24</td><td>2.23</td><td>1.41</td><td>1.48</td><td>1.42</td><td>1.49</td><td>-0.47</td></tr><tr><td>CLS</td><td>1.21</td><td>1.26</td><td>2.13</td><td>2.09</td><td>1.31</td><td>1.34</td><td>1.33</td><td>1.36</td><td>1.20</td><td>1.32</td><td>2.05</td><td>2.04</td><td>1.25</td><td>1.34</td><td>1.29</td><td>1.37</td><td>-0.41</td></tr><tr><td rowspan="2">MorFITT</td><td>CLS</td><td>1.38</td><td>1.44</td><td>2.45</td><td>2.40</td><td>1.48</td><td>1.50</td><td>1.49</td><td>1.51</td><td>1.37</td><td>1.50</td><td>2.35</td><td>2.33</td><td>1.46</td><td>1.55</td><td>1.48</td><td>1.57</td><td>-0.82</td></tr><tr><td>NER</td><td>1.30</td><td>1.35</td><td>2.23</td><td>2.17</td><td>1.48</td><td>1.48</td><td>1.50</td><td>1.49</td><td>1.29</td><td>1.43</td><td>2.22</td><td>2.18</td><td>1.48</td><td>1.55</td><td>1.49</td><td>1.56</td><td>-0.59</td></tr><tr><td rowspan="2">E3C</td><td>NER Clinical</td><td>1.30</td><td>1.35</td><td>2.23</td><td>2.17</td><td>1.48</td><td>1.48</td><td>1.50</td><td>1.49</td><td>1.29</td><td>1.43</td><td>2.22</td><td>2.18</td><td>1.47</td><td>1.54</td><td>1.48</td><td>1.55</td><td>-0.75</td></tr><tr><td>NER Temporal</td><td>1.29</td><td>1.35</td><td>2.22</td><td>2.16</td><td>1.48</td><td>1.48</td><td>1.48</td><td>1.49</td><td>1.29</td><td>1.43</td><td>2.22</td><td>2.18</td><td>1.47</td><td>1.54</td><td>1.48</td><td>1.55</td><td>-0.33</td></tr><tr><td rowspan="2">CLISTER</td><td>STS</td><td>1.52</td><td>1.59</td><td>2.65</td><td>2.57</td><td>1.73</td><td>1.72</td><td>1.74</td><td>1.72</td><td>1.51</td><td>1.65</td><td>2.56</td><td>2.49</td><td>1.71</td><td>1.77</td><td>1.71</td><td>1.77</td><td>-0.33</td></tr><tr><td>NER</td><td>1.31</td><td>1.37</td><td>2.26</td><td>2.19</td><td>1.48</td><td>1.49</td><td>1.50</td><td>1.50</td><td>1.31</td><td>1.44</td><td>2.19</td><td>2.15</td><td>1.48</td><td>1.55</td><td>1.49</td><td>1.56</td><td>-0.88</td></tr><tr><td rowspan="2">DEFT2021</td><td>CLS</td><td>1.31</td><td>1.57</td><td>2.63</td><td>2.56</td><td>1.69</td><td>1.70</td><td>1.71</td><td>1.71</td><td>1.46</td><td>1.61</td><td>2.50</td><td>2.46</td><td>1.64</td><td>1.72</td><td>1.66</td><td>1.74</td><td>-0.11</td></tr><tr><td>NER</td><td>1.29</td><td>1.34</td><td>2.20</td><td>2.14</td><td>1.42</td><td>1.43</td><td>1.45</td><td>1.45</td><td>1.29</td><td>1.41</td><td>2.21</td><td>2.16</td><td>1.41</td><td>1.49</td><td>1.46</td><td>1.52</td><td>-0.68</td></tr><tr><td rowspan="4">ESSAI</td><td>NER Spec</td><td>1.28</td><td>1.33</td><td>2.19</td><td>2.13</td><td>1.41</td><td>1.42</td><td>1.44</td><td>1.44</td><td>1.28</td><td>1.41</td><td>2.19</td><td>2.15</td><td>1.40</td><td>1.48</td><td>1.44</td><td>1.51</td><td>-0.61</td></tr><tr><td>POS</td><td>1.28</td><td>1.33</td><td>2.19</td><td>2.13</td><td>1.41</td><td>1.42</td><td>1.44</td><td>1.44</td><td>1.28</td><td>1.41</td><td>2.19</td><td>2.15</td><td>1.40</td><td>1.48</td><td>1.44</td><td>1.51</td><td>-0.69</td></tr><tr><td>NER Neg</td><td>1.28</td><td>1.34</td><td>2.20</td><td>2.14</td><td>1.42</td><td>1.43</td><td>1.45</td><td>1.46</td><td>1.28</td><td>1.41</td><td>2.20</td><td>2.16</td><td>1.41</td><td>1.49</td><td>1.45</td><td>1.52</td><td>-0.02</td></tr><tr><td>CLS</td><td>1.28</td><td>1.34</td><td>2.20</td><td>2.14</td><td>1.42</td><td>1.43</td><td>1.45</td><td>1.46</td><td>1.28</td><td>1.41</td><td>2.20</td><td>2.16</td><td>1.41</td><td>1.49</td><td>1.45</td><td>1.52</td><td>-0.02</td></tr><tr><td rowspan="2">QUAERO</td><td>NER Medline</td><td>1.53</td><td>1.63</td><td>2.35</td><td>2.26</td><td>1.78</td><td>1.78</td><td>1.77</td><td>1.78</td><td>1.52</td><td>1.76</td><td>2.36</td><td>2.35</td><td>1.77</td><td>1.89</td><td>1.76</td><td>1.89</td><td>-0.77</td></tr><tr><td>NER EMEA</td><td>1.30</td><td>1.34</td><td>2.14</td><td>2.12</td><td>1.44</td><td>1.46</td><td>1.49</td><td>1.51</td><td>1.30</td><td>1.39</td><td>2.06</td><td>2.04</td><td>1.45</td><td>1.51</td><td>1.50</td><td>1.56</td><td>-0.28</td></tr><tr><td rowspan="3">MANTRAGSC</td><td>NER EMEA</td><td>1.33</td><td>1.40</td><td>2.47</td><td>2.41</td><td>1.49</td><td>1.51</td><td>1.50</td><td>1.52</td><td>1.32</td><td>1.43</td><td>2.33</td><td>2.30</td><td>1.46</td><td>1.53</td><td>1.49</td><td>1.55</td><td>-0.63</td></tr><tr><td>NER Medline</td><td>1.89</td><td>2.01</td><td>2.84</td><td>2.70</td><td>2.06</td><td>2.13</td><td>2.14</td><td>2.14</td><td>1.89</td><td>2.09</td><td>2.84</td><td>2.78</td><td>2.06</td><td>2.22</td><td>2.10</td><td>2.22</td><td>-0.64</td></tr><tr><td>NER Patents</td><td>1.54</td><td>1.59</td><td>2.34</td><td>2.30</td><td>1.61</td><td>1.63</td><td>1.59</td><td>1.62</td><td>1.43</td><td>1.52</td><td>2.20</td><td>2.20</td><td>1.50</td><td>1.58</td><td>1.51</td><td>1.60</td><td>0.06</td></tr><tr><td colspan="2">Average per model</td><td>1.39</td><td>1.45</td><td>2.30</td><td>2.25</td><td>1.54</td><td>1.55</td><td>1.56</td><td>1.56</td><td>1.38</td><td>1.51</td><td>2.26</td><td>2.24</td><td>1.52</td><td>1.60</td><td>1.55</td><td>1.62</td><td>-0.48</td></tr><tr><td colspan="2">Relative Difference (%)</td><td>0.0</td><td>4.5</td><td>65.9</td><td>61.8</td><td>11.2</td><td>11.8</td><td>12.3</td><td>12.6</td><td>-0.7</td><td>8.9</td><td>62.8</td><td>61.1</td><td>9.9</td><td>15.5</td><td>11.7</td><td>16.9</td><td></td></tr></tbody></table>

a negative correlation implies that fewer subword units are associated with higher scores, while a positive correlation suggests that more subword units are linked to higher scores.

In overall, we observe in Table 3 an average ρ correlation of -0.48 between tasks and models, indicating that, in general, higher performance scores tend to be associated with fewer subword units. To our knowledge, this is the first time such a correlation has been experimentally demonstrated. However, it's important to note that this correlation varies across the targeted tasks. Tasks like CLS show correlation close to zero, suggesting that they are less affected by the granularity of tokenization. In contrast, STS and sequence-to-sequence tasks, particularly NER, appear to be more influenced by tokenization granularity, likely due to their heavy reliance on immediate context for making predictions.

While the RoBERTa model's embeddings capture semantic meaning and the encoder module captures contextual information (Rogers et al., 2020), we aimed to determine whether the observed correlations are attributed to a specific part of this architecture. To investigate this, we isolated and froze the embeddings and/or encoder of our BERT-based model, based on the NACHOS SentencePiece, during fine-tuning for various tasks. The experimental approach, as detailed in Table 4,

involved several stages. Initially, we established a baseline for each task with no frozen components. Subsequently, we conducted experiments by freezing only the embedding layer, only the encoder, and both the embeddings and encoders. Our findings indicate a stronger dependence on RoBERTa's encoder for tasks such as POS tagging and STS, in contrast to other tasks, which corroborate the context dependency as an explanation to the correlation scores between segmentation granularity and models performances for these tasks but not for NER.

<table><caption>Table 4: Performance and relative loss (in %) of the PLMs based on SentencePiece NACHOS without morpheme with parts of the models being frozen.</caption><thead><tr><th></th><th>CAS<br>POS</th><th>PxCorpus<br>NER</th><th>PxCorpus<br>CLS</th><th>CLISTER<br>STS</th></tr></thead><tbody><tr><td>Full Fine-tuning</td><td>97.10</td><td>96.10</td><td>94.82</td><td>0.61</td></tr><tr><td>Embedding</td><td>97.03 ↓ 0.07</td><td>96.10 ↑ 0.00</td><td>94.73 ↓ 0.09</td><td>0.62 ↑ 1.63</td></tr><tr><td>Encoder</td><td>65.97 ↓ 32.05</td><td>83.95 ↓ 12.64</td><td>84.78 ↓ 10.58</td><td>0.45 ↓ 26.22</td></tr><tr><td>Embedding + Encoder</td><td>60.04 ↓ 38.16</td><td>79.62 ↓ 17.14</td><td>84.78 ↓ 10.58</td><td>0.44 ↓ 27.86</td></tr></tbody></table>

As shown in Table 3, higher performance scores are associated with fewer subword units. To gain a linguistic perspective on how tokenization strategies behave, we analyzed the segmentation of 150 biomedical terms equally distributed across cardiology, dermatology, obstetric-gynecology, and
ophthalmology, as presented in Table 5. Most models, except for those using SentencePiece NACHOS, struggle to precisely align with the official morphological segmentation established by the Académie Française (French Academy). However, upon closer examination, it is evident that these models often come very close to the desired segmentation. While the segmentations may exhibit slight variations, such as the relocation of a letter from one token to another, they maintain the same number of tokens as the official morphological segmentation. This observation is further supported when we analyze actual tokenizer outputs (see Table 6) and assess the segmentation statistics in Table 5. For example, BPE NACHOS tokenizes the term "ophtalmoscope" into the units "ophtalmoscope," whereas the morphological segmentation should be "ophtalmo scope," a segmentation achieved by its morpheme-enriched counterpart.

<table><caption>Table 5: The average Exact Match (EM*) and portion of terms aligned with the official segmentation length (Exact # Tok.), both in %, are based on the gold segmentation from 150 biomedical terms. Both last columns are referring to the portion of terms suffering from under and over segmentation. w/o and w/ denote without and with morphemes respectively. SP stands for SentencePiece.</caption><thead><tr><th rowspan="2"></th><th rowspan="2"></th><th rowspan="2"></th><th rowspan="2"></th><th colspan="4">Type of errors</th></tr><tr><th>EM*</th><th>Exact # Tok.</th><th>Under Seg.</th><th>Over Seg.</th></tr></thead><tbody><tr><td rowspan="4">BPE</td><td rowspan="2">NACHOS</td><td>w/o</td><td>21.3</td><td>41.3</td><td>9.3</td><td>49.3</td></tr><tr><td>w/</td><td>34.6</td><td>50.0</td><td>6.0</td><td>44.0</td></tr><tr><td rowspan="2">PubMed</td><td>w/o</td><td>2.6</td><td>12.0</td><td>2.6</td><td>85.3</td></tr><tr><td>w/</td><td>17.3</td><td>28.6</td><td>2.6</td><td>68.6</td></tr><tr><td rowspan="2">CC100</td><td>w/o</td><td>8.0</td><td>28.0</td><td>2.6</td><td>69.3</td></tr><tr><td>w/</td><td>23.3</td><td>38.6</td><td>2.6</td><td>58.6</td></tr><tr><td rowspan="2">Wikipedia</td><td>w/o</td><td>8.6</td><td>24.6</td><td>3.3</td><td>72.0</td></tr><tr><td>w/</td><td>22.0</td><td>36.6</td><td>4.6</td><td>58.6</td></tr><tr><td rowspan="6">SP</td><td rowspan="2">NACHOS</td><td>w/o</td><td>56.6</td><td>74.6</td><td>7.3</td><td>18.0</td></tr><tr><td>w/</td><td>61.3</td><td>70.6</td><td>2.6</td><td>26.6</td></tr><tr><td rowspan="2">PubMed</td><td>w/o</td><td>14.6</td><td>26.6</td><td>2.6</td><td>70.6</td></tr><tr><td>w/</td><td>32.0</td><td>42.0</td><td>2.6</td><td>55.3</td></tr><tr><td rowspan="2">CC100</td><td>w/o</td><td>24.0</td><td>42.0</td><td>4.0</td><td>54.0</td></tr><tr><td>w/</td><td>36.6</td><td>49.3</td><td>2.6</td><td>48.0</td></tr><tr><td rowspan="2">Wikipedia</td><td>w/o</td><td>18.0</td><td>42.0</td><td>3.3</td><td>54.6</td></tr><tr><td>w/</td><td>34.0</td><td>54.0</td><td>4.6</td><td>41.3</td></tr></tbody></table>

In Table 5, we observed various types of errors in segmentation, with the most common issue being over-segmentation of units that are not present in our biomedical lexical morphemes list. This over-segmentation results in smaller, more numerous, and sparser tokens, which can impact the efficiency of pre-training. The reduced frequency of tokens and the faster filling of RoBERTa's 512-token context window with less meaningful tokens can be problematic.

Finally, Table 5 reveals an interesting distinction between BPE and SentencePiece using NACHOS training data. SentencePiece outperforms BPE in achieving segmentations that closely resemble

<table><caption>Table 6: Instances of tokenization juxtaposed with their correct segmentation.</caption><thead><tr><th>Base</th><th>cancérigène</th><th>ophtalmoscope</th><th>anglographie</th></tr></thead><tbody><tr><td>Correct</td><td>cancérigène</td><td>ophtalm scope</td><td>anglographie</td></tr><tr><td>BPE Wiki</td><td>cancérigène</td><td>ophtalm scope</td><td>anglographie</td></tr><tr><td>BPE PubMed</td><td>cancérigène</td><td>ophtalm scope</td><td>anglographie</td></tr><tr><td>BPE NACHOS</td><td>cancérigène</td><td>ophtalm scope</td><td>anglographie</td></tr><tr><td>SentencePiece NACHOS</td><td>cancérigène</td><td>ophtalm scope</td><td>anglographie</td></tr><tr><td>BPE NACHOS +Morpheme</td><td>cancérigène</td><td>ophtalm scope</td><td>anglographie</td></tr><tr><td>SentencePiece NACHOS +Morpheme</td><td>cancérigène</td><td>ophtalm scope</td><td>anglographie</td></tr></tbody></table>

correct ones, both in terms of the number of tokens and their semantic accuracy. SentencePiece excels at matching correct segmentations, particularly for medical terminology, in 56.6% of cases without morphemes and 61.3% when morphemes are used, while BPE NACHOS achieves only 34.6% accuracy.

## 5.2. Impact of morphemes

One of our primary objectives was to approximate the correct morphological segmentation of words in the French biomedical language. Our analysis reveals that tokenizers, such as BPE and SentencePiece trained on NACHOS, enriched with morphemes, can often achieve this goal. Notably, SentencePiece NACHOS enriched with morphemes achieved the best performance, with a 61.3% exact match. Our morpheme-enriched approach offers the advantage of obtaining a tokenization that closely resembles what could be achieved through a complex rule-based method. This approach is easily adaptable to other languages with a list of lexical morphemes and similar principles.

As shown in Table 2, the introduction of morphemes (w/) may lead to performance enhancements in approximately 25% of the studied downstream tasks. However, it is noteworthy that the best results are primarily achieved by classical statistical tokenizers, BPE and SentencePiece, when not using morphemes, and when trained on our biomedical-specific data, NACHOS. This observation is intriguing because NACHOS-based tokenizers inherently contain a higher proportion of morphemes, as shown in Table 7, which presents the portion of correct morphemes already present in the tokenizers without introducing additional morphological information based on their length ranges. This suggests that introducing morphemes and other forms of morphological knowledge, such as grammatical endings, may have a more substantial impact in contexts that do not align directly with the target domains and languages. However, we can note that the results of this method are inconsistent and do not ensure an overall performance boost across all models or tasks.

Furthermore, it is worth noting that morphemes are often already present in the tokenizers in their complete form, as illustrated in Table 7, or with
<table><caption>Table 7: Percentage of the morphemes already present in the tokenizers vocabularies per range of morphemes lengths. SP stands for SentencePiece.</caption><thead><tr><th rowspan="2">Tokenizer</th><th rowspan="2">Source</th><th colspan="4">Coverage of the morphemes (%)</th></tr><tr><th>1 - 3</th><th>4 - 6</th><th>7 - 10</th><th>Global</th></tr></thead><tbody><tr><td rowspan="4">BPE</td><td><b>NACHOS</b></td><td>83.33</td><td>45.38</td><td>31.00</td><td>47.23</td></tr><tr><td><b>PubMed</b></td><td>65.15</td><td>39.32</td><td>15.00</td><td>38.06</td></tr><tr><td><b>CC100</b></td><td>78.78</td><td>34.46</td><td>7.00</td><td>34.77</td></tr><tr><td><b>Wikipedia</b></td><td>87.87</td><td>34.95</td><td>10.00</td><td>36.67</td></tr><tr><td rowspan="4">SP</td><td><b>NACHOS</b></td><td>83.33</td><td>41.01</td><td>28.00</td><td>43.59</td></tr><tr><td><b>PubMed</b></td><td>60.60</td><td>37.13</td><td>14.00</td><td>35.81</td></tr><tr><td><b>CC100</b></td><td>83.33</td><td>34.70</td><td>8.00</td><td>35.64</td></tr><tr><td><b>Wikipedia</b></td><td>93.93</td><td>37.37</td><td>12.0</td><td>39.44</td></tr></tbody></table>

minor modifications based on token probabilities, as shown in Table 6. Notably, tokenizers based on NACHOS contain a significantly higher percentage of morphemes, with 47.23% for BPE and 43.59% for SentencePiece. Conversely, the source with the fewest morphemes is CC100, with percentages of 34.77% for BPE and 35.64% for SentencePiece. This observation aligns with the fact that CC100 has fewer connections to both the target language and domain.

In general, we observe that despite the significant improvement in segmentation quality (as shown in Table 5), tokenizers enriched with morphemes do not exhibit a strong correlation with the results achieved in downstream tasks, as evident in Table 2. The ability to deliver satisfactory results despite encountering suboptimal segmentations, as seen in the case of PubMed, which frequently over-segments words, underscores the robustness of RoBERTa's architecture in handling noise and its capacity to compensate for such challenges.

## 5.3. Impact of data sources

As indicated in Table 2, the average performance across tasks demonstrates a significant impact of the training data source on the results obtained by the models. It becomes apparent that using data that is more suitable for the target language, even if it originates from various domains such as Wikipedia and CC100, is more effective than utilizing data from the target domains but from a different language. This is particularly evident in the CLS, NER, and STS tasks, where BPE PubMed achieves an average of 70.16% for classification, 0.63 MSE for STS, and 62.62% for NER, whereas CC100 outperforms with 74.14%, 0.67 MSE, and 64.62%, respectively.

The decrease in performance from PubMed can be attributed to over-segmentation, as seen in Table 3. This over-segmentation is primarily due to the significant differences between the data used to build the tokenizer and the language of the model's pre-training. These differences stem from distinct lexicons, writing styles, and morphological structures

in French compared to English, particularly for specialized words like "Péricardite" (French) and "Pericarditis" (English), or "Orthophoniste" (French) and "Speech Therapist" (English). Furthermore, variations in alphabets, such as special French characters like "é" or "è," can lead to token sparsity when encountered in positions not seen during tokenizer construction on PubMed. This results in a lack of both language and domain-specific information for French, as only limited tokens can be used to form sentences.

Some data sources are surprisingly less affected by the introduction of morphemes. For instance, the CC100 source is not positively impacted by morphemes, despite having a lower proportion of morphemes in its original version, as shown in Table 7. This behavior may be explained by the increased granularity introduced by morphemes, which reduces the probabilities of other tokens appearing. This can lead to a poorer representation of words.

## 6. Conclusion

In this study, we conducted a comprehensive investigation into the influence of various word tokenization strategies within a BERT-based masked language model across diverse French biomedical NLP tasks. Notably, we observed that existing methods for tokenizing biomedical text often fall short of aligning with morphological rules and how humans learn these specialized terms. This suboptimal segmentation can impact the agglutinating nature of biomedical terminology. To assess the effects of this segmentation on downstream applications, we developed a set of novel biomedical tokenizers that adhere more closely to morphological rules. These tokenizers combine various automatic tokenization approaches and vocabularies to enrich segmentation with morphemes. We employed these enhanced tokenizers in the pre-training of multiple RoBERTa-based models, which we then evaluated across a wide array of 23 French biomedical tasks, including POS, NER, STS, and CLS.

Our findings show that integrating morphemes into automatic tokenization approaches can achieve parity or improve performance in certain tasks, such as NER and POS tagging. However, this enhancement is not consistent across all tasks. While there is a correlation between segmentation granularity and downstream task performance, we also observe that pre-training processes exhibit robustness to suboptimal tokenization, yielding surprisingly good results even with very short and sparse subword units. To conclude, our study reveals that achieving optimal tokenization involves a combination of factors, including minimizing word segmentation and having access to domain-specific data in the target language.
## 7. Acknowledgements

This work was performed using HPC resources from GENCI-IDRIS (Grant 2022-AD011013061R1 and 2022-AD011013715). This work was financially supported by ANR MALADES (ANR-23-IAS1-0005) and Zenidoc.

## 8. Bibliographical References

Emily Alsentzer, John Murphy, William Boag, Wei-Hung Weng, Di Jindi, Tristan Naumann, and Matthew McDermott. 2019. Publicly available clinical BERT embeddings. In *Proceedings of the 2nd Clinical Natural Language Processing Workshop*, pages 72–78, Minneapolis, Minnesota, USA. Association for Computational Linguistics.

Issam Bazzi and James R Glass. 2002. A multi-class approach for modelling out-of-vocabulary words. In *Interspeech*.

Iz Beltagy, Kyle Lo, and Arman Cohan. 2019. SciBERT: A pretrained language model for scientific text. In *Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP)*, pages 3615–3620, Hong Kong, China. Association for Computational Linguistics.

Aman Berhe, Guillaume Draznieks, Vincent Martenot, Valentin Masdeu, Lucas Davy, and Jean-Daniel Zucker. 2023. ALIBERT: A pretrained language model for French biomedical text. In *The 22nd Workshop on Biomedical Natural Language Processing and BioNLP Shared Tasks*, pages 223–236, Toronto, Canada. Association for Computational Linguistics.

Tom Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared D Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, Sandhini Agarwal, Ariel Herbert-Voss, Gretchen Krueger, Tom Henighan, Rewon Child, Aditya Ramesh, Daniel Ziegler, Jeffrey Wu, Clemens Winter, Chris Hesse, Mark Chen, Eric Sigler, Mateusz Litwin, Scott Gray, Benjamin Chess, Jack Clark, Christopher Berner, Sam McCandlish, Alec Radford, Ilya Sutskever, and Dario Amodei. 2020. Language models are few-shot learners. In *Advances in Neural Information Processing Systems*, volume 33, pages 1877–1901. Curran Associates, Inc.

William Chen and Brett Fazio. 2021. Morphologically-guided segmentation for

translation of agglutinative low-resource languages. In *Proceedings of the 4th Workshop on Technologies for MT of Low Resource Languages (LoResMT2021)*, pages 20–31, Virtual Association for Machine Translation in the Americas.

Jenny Copara, Julien Knafou, Nona Naderi, Claudiia Moro, Patrick Ruch, and Douglas Teodoro. 2020. Contextualized French language models for biomedical named entity recognition. In *Actes de la 6e conférence conjointe Journées d'Études sur la Parole (JEP, 33e édition)*, Traitement Automatique des Langues Naturelles (TALN, 27e édition), Rencontre des Étudiants Chercheurs en Informatique pour le Traitement Automatique des Langues (RÉCITAL, 22e édition). Atelier DÉfi Fouille de Textes, pages 36–48, Nancy, France. ATALA et AFCP.

H. Cottez. 1980. *Dictionnaire des structures du vocabulaire savant: éléments et modèles de formation*. Collection Les usuels du Robert. Le Robert.

Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. 2019. BERT: Pre-training of deep bidirectional transformers for language understanding. In *Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long and Short Papers)*, pages 4171–4186, Minneapolis, Minnesota. Association for Computational Linguistics.

Hicham El Boukkouri, Olivier Ferret, Thomas Lavergne, and Pierre Zweigenbaum. 2022. Retrain or train from scratch? comparing pre-training strategies of BERT in the medical domain. In *Proceedings of the Thirteenth Language Resources and Evaluation Conference*, pages 2626–2633, Marseille, France. European Language Resources Association.

Allison Fan and Weiwei Sun. 2023. Constructivist tokenization for English. In *Proceedings of the First International Workshop on Construction Grammars and NLP (CxGs+NLP, GURT/SyntaxFest 2023)*, pages 36–40, Washington, D.C. Association for Computational Linguistics.

Takuro Fujii, Koki Shibata, Atsuki Yamaguchi, Terufumi Morishita, and Yasuhiro Sogawa. 2023. How do different tokenizers perform on downstream tasks in scriptio continua languages?: A case study in Japanese. In *Proceedings of the 61st Annual Meeting of the Association for Computational Linguistics (Volume 4: Student Research*
*Workshop)*, pages 39–49, Toronto, Canada. Association for Computational Linguistics.

Philip Gage. 1994. A new algorithm for data compression. *C Users J.*, 12(2):23–38.

Yu Gu, Robert Tinn, Hao Cheng, Michael Lucas, Naoto Usuyama, Xiaodong Liu, Tristan Naumann, Jianfeng Gao, and Hoifung Poon. 2021. Domain-specific language model pretraining for biomedical natural language processing. *ACM Trans. Comput. Healthcare*, 3(1).

Valentin Hofmann, Hinrich Schuetze, and Janet Pierrehumbert. 2022. An embarrassingly simple method to mitigate undesirable properties of pretrained language model tokenizers. In *Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics (Volume 2: Short Papers)*, pages 385–393, Dublin, Ireland. Association for Computational Linguistics.

Bernal Jimenez Gutierrez, Huan Sun, and Yu Su. 2023. Biomedical language models are robust to sub-optimal tokenization. In *The 22nd Workshop on Biomedical Natural Language Processing and BioNLP Shared Tasks*, pages 350–362, Toronto, Canada. Association for Computational Linguistics.

Taku Kudo. 2018. Subword regularization: Improving neural network translation models with multiple subword candidates. In *Proceedings of the 56th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers)*, pages 66–75, Melbourne, Australia. Association for Computational Linguistics.

Taku Kudo and John Richardson. 2018. *SentencePiece: A simple and language independent subword tokenizer and detokenizer for neural text processing*. In *Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing: System Demonstrations*, pages 66–71, Brussels, Belgium. Association for Computational Linguistics.

Yanis Labrak, Adrien Bazoge, Richard Dufour, Mickael Rouvier, Emmanuel Morin, Béatrice Daille, and Pierre-Antoine Gourraud. 2023. *DrBERT: A robust pre-trained model in French for biomedical and clinical domains*. In *Proceedings of the 61st Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers)*, pages 16207–16221, Toronto, Canada. Association for Computational Linguistics.

Yanis Labrak, Adrien Bazoge, Oumaima El Khettari, Mickael Rouvier, Pacome Constant dit Beaufils, Natalia Grabar, Beatrice Daille, Solen Quiniou, Emmanuel Morin, Pierre-Antoine Gourraud, and

Richard Dufour. 2024. *DrBenchmark: A large language understanding evaluation benchmark for french biomedical domain*.

Martha Larson. 2001. Sub-word-based language models for speech recognition: implications for spoken document retrieval. *Workshop on Language Modeling and Information Retrieval*.

Jinhyuk Lee, Wonjin Yoon, Sungdong Kim, Donghyeon Kim, Sunkyu Kim, Chan Ho So, and Jaewoo Kang. 2019. *BioBERT: a pretrained biomedical language representation model for biomedical text mining*. *Bioinformatics*, 36(4):1234–1240.

Yinhan Liu, Myle Ott, Naman Goyal, Jingfei Du, Mandar Joshi, Danqi Chen, Omer Levy, Mike Lewis, Luke Zettlemoyer, and Veselin Stoyanov. 2020. *RoBERTa: A robustly optimized {bert} pre-training approach*.

Sabrina J. Mielke, Zaid Alyafeai, Elizabeth Salesky, Colin Raffel, Manan Dey, Matthias Gallé, Arun Raja, Chenglei Si, Wilson Y. Lee, Benoît Sagot, and Samson Tan. 2021. *Between words and characters: A brief history of open-vocabulary modeling and tokenization in nlp*.

Vít Novotný, Eniafe Festus Ayetiran, Dalibor Bačovský, Dávid Lupták, Michal Štefánik, and Petr Sojka. 2021. One size does not fit all: Finding the optimal subword sizes for FastText models across languages. In *Proceedings of the International Conference on Recent Advances in Natural Language Processing (RANLP 2021)*, pages 1068–1074, Held Online. INCOMA Ltd.

Yirong Pan, Xiao Li, Yating Yang, and Rui Dong. 2020. Morphological word segmentation on agglutinative languages for neural machine translation.

Anna Rogers, Olga Kovaleva, and Anna Rumshisky. 2020. *A primer in BERTology: What we know about how BERT works*. *Transactions of the Association for Computational Linguistics*, 8:842–866.

David Samuel and Lilja Øvrelid. 2023. *Tokenization with factorized subword encoding*. In *Findings of the Association for Computational Linguistics: ACL 2023*, pages 14143–14161, Toronto, Canada. Association for Computational Linguistics.

Mike Schuster and Kaisuke Nakajima. 2012. *Japanese and korean voice search*. In *2012 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP)*, pages 5149–5152.
Rico Sennrich, Barry Haddow, and Alexandra Birch.
2016. Neural machine translation of rare words
with subword units. In *Proceedings of the 54th
Annual Meeting of the Association for Compu-
tational Linguistics (Volume 1: Long Papers)*,
pages 1715–1725, Berlin, Germany. Association
for Computational Linguistics.

Igor Szoke, Lukás Burget, Jan Cernocky, and Michal Fapso. 2008. Sub-word modeling of out of vocabulary words in spoken term detection. In *2008 IEEE Spoken Language Technology Workshop*, pages 273–276. IEEE.

Cagri Toraman, Eyup Halit Yilmaz, Furkan Şahinuç, and Oguzhan Ozcelik. 2023. *Impact of tokenization on language models: An analysis for turkish*. ACM Trans. Asian Low-Resour. Lang. Inf. Process., 22(4).

Rian Touchent, Laurent Romary, and Eric de la Clergerie. 2023. Camembert-bio: a tasty french language model better for your health.

Christian Touratier. 2012. Chapitre V. Les classes de morphèmes. In *Morphologie et morphématique : Analyse en morphèmes*, Langues et langage, pages 78–114. Presses universitaires de Provence, Aix-en-Provence.

Yining Wang, Long Zhou, Jiajun Zhang, and Chengqing Zong. 2017. Word, subword or character? an empirical study of granularity in chinese-english nmt. In *Machine Translation*, pages 30–42, Singapore. Springer Singapore.

Yonghui Wu, Mike Schuster, Zhifeng Chen, Quoc V. Le, Mohammad Norouzi, Wolfgang Macherey, Maxim Krikun, Yuan Cao, Qin Gao, Klaus Macherey, Jeff Klingner, Apurva Shah, Melvin Johnson, Xiaobing Liu, Łukasz Kaiser, Stephan Gouws, Yoshikiyo Kato, Taku Kudo, Hideto Kazawa, Keith Stevens, George Kurian, Nishant Patil, Wei Wang, Cliff Young, Jason Smith, Jason Riesa, Alex Rudnick, Oriol Vinyals, Greg Corrado, Macduff Hughes, and Jeffrey Dean. 2016. Google's neural machine translation system: Bridging the gap between human and machine translation.

9. Language Resource References

Rémi Cardon, Natalia Grabar, Cyril Grouin, and Thierry Hamon. 2020. Présentation de la campagne d'évaluation DEFT 2020 : similarité textuelle en domaine ouvert et extraction d'information précise dans des cas cliniques (presentation of the DEFT 2020 challenge : open

domain textual similarity and precise informa-
tion extraction from clinical cases ). In Actes
de la 6e conférence conjointe Journées d’Études
sur la Parole (JEP, 33e édition), Traitement Au-
tomatique des Langues Naturelles (TALN, 27e
édition), Rencontre des Étudiants Chercheurs
en Informatique pour le Traitement Automatique
des Langues (RÉCITAL, 22e édition). Atelier DÉfi
Fouille de Textes, pages 1–13, Nancy, France.
ATALA et AFCP.

Clément Dalloux, Vincent Claveau, Natalia Grabar, Lucas Emanuel Silva Oliveira, Claudia Maria Cabral Moro, Yohan Bonescki Gumiel, and Deborah Ribeiro Carvalho. 2021. Supervised learning for the detection of negation and of its scope in French and Brazilian Portuguese biomedical corpora. *Natural Language Engineering*, 27(2):181–201.

Natalia Grabar, Vincent Claveau, and Clément Dalloux. 2018. CAS: French Corpus with Clinical Cases. In *Proceedings of the 9th International Workshop on Health Text Mining and Information Analysis (LOUHI)*, pages 1–7, Brussels, Belgium.

Cyril Grouin, Natalia Grabar, and Gabriel Illouz. 2021. Classification de cas cliniques et évaluation automatique de réponses d'étudiants : présentation de la campagne DEFT 2021 (clinical cases classification and automatic evaluation of student answers : Presentation of the DEFT 2021 challenge). In *Actes de la 28e Conférence sur le Traitement Automatique des Langues Naturelles. Atelier DÉfi Fouille de Textes (DEFT)*, pages 1–13, Lille, France. ATALA.

Nicolas Hiebel, Olivier Ferret, Karën Fort, and Aurélie Névéol. 2022. CLISTER : A corpus for semantic textual similarity in French clinical narratives. In *Proceedings of the Thirteenth Language Resources and Evaluation Conference*, pages 4306–4315, Marseille, France. European Language Resources Association.

Alican Kocabiyikoglu, François Portet, Prudence Gibert, Hervé Blanchon, Jean-Marc Babouchkine, and Gaëtan Gavazzi. 2022. A spoken drug prescription dataset in french for spoken language understanding. In *13th Language Resources and Evaluation Conference (LREC 2022)*.

Jan A Kors, Simon Clematide, Saber A Akhondi, Erik M van Mulligen, and Dietrich Rebholz-Schuhmann. 2015. A multilingual gold-standard corpus for biomedical concept recognition: the Mantra GSC. *Journal of the American Medical Informatics Association*, 22(5):948–956.
Yanis Labrak, Adrien Bazoge, Richard Dufour, Mickael Rouvier, Emmanuel Morin, Béatrice Daille, and Pierre-Antoine Gourraud. 2023a. DrBERT: A robust pre-trained model in French for biomedical and clinical domains. In *Proceedings of the 61st Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers)*, pages 16207–16221, Toronto, Canada. Association for Computational Linguistics.

Yanis Labrak, Mickaël Rouvier, and Richard Dufour. 2023b. MORFITT: A multi-label corpus of French scientific articles in the biomedical domain. In *30e Conférence sur le Traitement Automatique des Langues Naturelles (TALN) Atelier sur l'Analyse et la Recherche de Textes Scientifiques*, Paris, France. Florian Boudin.

DA Lindberg, BL Humphreys, and AT McCray. 1993. The Unified Medical Language System. *Methods Inf Med*, 32(4):281–291.

Bernardo Magnini, Begoña Altuna, Alberto Lavelli, Manuela Speranza, and Roberto Zanoli. 2020. The e3c project: Collection and annotation of a multilingual corpus of clinical cases. *Proceedings of the Seventh Italian Conference on Computational Linguistics CLiC-it 2020*.

Hiroki Nakayama. 2018. seqeval: A python framework for sequence labeling evaluation. Software available from https://github.com/chakki-works/seqeval.

Aurélie Névéol, Cyril Grouin, Jeremy Leixa, Sophie Rosset, and Pierre Zweigenbaum. 2014. The QUAERO French medical corpus: A resource for medical entity recognition and normalization. In *Proc of BioTextMining Work*, pages 24–30.

Helmut Schmid. 1994. Probabilistic part-of-speech tagging using decision trees. In *Proceedings of the International Conference on New Methods in Language Processing*, Manchester, UK.

Rian Touchent, Laurent Romary, and Eric de la Clergerie. 2023. Camembert-bio: a tasty french language model better for your health.

Guillaume Wenzek, Marie-Anne Lachaux, Alexis Conneau, Vishrav Chaudhary, Francisco Guzmán, Armand Joulin, and Edouard Grave. 2020. CCNet: Extracting high quality monolingual datasets from web crawl data. In *Proceedings of the Twelfth Language Resources and Evaluation Conference*, pages 4003–4012, Marseille, France. European Language Resources Association.