<figure><img src="image_1.png" /></figure>

HAL
open science

# Textual data augmentation using generative approaches
- Impact on named entity recognition tasks

Danrun Cao, Nicolas Béchet, Pierre-François Marteau, Oussama Ahmia

► To cite this version:

Danrun Cao, Nicolas Béchet, Pierre-François Marteau, Oussama Ahmia. Textual data augmentation using generative approaches - Impact on named entity recognition tasks. Data and Knowledge Engineering, 2025, 156, pp.102403. 10.1016/j.datak.2024.102403. hal-05318240

HAL Id: hal-05318240

https://hal.science/hal-05318240v1

Submitted on 20 Nov 2025

HAL is a multi-disciplinary open access
archive for the deposit and dissemination of sci-
entific research documents, whether they are pub-
lished or not. The documents may come from
teaching and research institutions in France or
abroad, or from public or private research centers.

L'archive ouverte pluridisciplinaire HAL, est
destinée au dépôt et à la diffusion de documents
scientifiques de niveau recherche, publiés ou non,
émanant des établissements d'enseignement et de
recherche français ou étrangers, des laboratoires
publics ou privés.

<figure><img src="image_2.png" /></figure>

Distributed under a Creative Commons Attribution 4.0 International License
<figure><img src="image_3.png" /></figure>

ELSEVIER

<figure><img src="image_4.png" /></figure>

<figure><img src="image_5.png" /></figure>

Check for
updates

# Textual data augmentation using generative approaches - Impact on
named entity recognition tasks

Danrun Cao<sup>a,b</sup> <sup>D,∗</sup>, Nicolas Béchet<sup>a</sup>, Pierre-François Marteau<sup>a</sup>, Oussama Ahmia<sup>b</sup>

## ARTICLE INFO

MSC:

68T50

Keywords:

Data augmentation

Named entity recognition

Feature engineering

Word embeddings

Model robustness

Generative model

Call for tenders

## ABSTRACT

Industrial applications of Named Entity Recognition (NER) are usually confronted with small and imbalanced corpora. This could harm the performance of trained and finetuned recognition models, especially when they encounter unknown data. In this study we develop three generation-based data enrichment approaches, in order to increase the number of examples of underrepresented entities. We compare the impact of enriched corpora on NER models, using both non-contextual (fastText) and contextual (Bert-like) embedding models to provide discriminant features to a biLSTM-CRF used as an entity classifier. The approach is evaluated on a contract renewal detection task applied to a corpus of calls for tenders. The results show that the proposed data enrichment procedure effectively improves the NER model's effectiveness when applied on both known and unknown data.

## 1. Introduction

In Natural Language Processing (NLP), a named entity is defined as a nominal phrase that refers to an object or a concept well-defined in the world. This object can be concrete or abstract, and its nature can vary depending on the application context. Named Entity Recognition (NER) refers to the task of identifying and classifying such entities in unstructured text. NER finds numerous applications in a wide range of socio-economic domains, in particular health [1,2], finance [3,4], economic intelligence [5,6], etc. However, such applications of NER are often confronted with imbalanced corpora. Some entities are typically under-represented, due to their rare nature or the lack of available data containing them. This usually leads to an overfitting of models that could limit drastically their generalization capacity.

In this article, we explore generation-based data augmentation techniques in order to improve the characterization of under-represented entities in a given training set. Our work focuses on a French corpus, but the method can be easily adapted to other languages. The rest of the article is organized as follows: We start by briefly presenting related previous works in Section 2. Then we introduce the context of our work in Section 3. In Section 4 we will detail the three proposed methods. Section 5 describes the experimental protocol and finally Section 6 presents and discusses the results.

<footnote><sup>a</sup> Univ. Bretagne Sud, CNRS, IRISA, Rue Yves Mainguy, Vannes, 56000, France</footnote>

<footnote><sup>b</sup> OctopusMind, 2 Pl. Saint-Pierre, Nantes, 44000, France</footnote>

<footnote>* Corresponding author at: Univ. Bretagne Sud, CNRS, IRISA, Rue Yves Mainguy, Vannes, 56000, France.
E-mail address: danrun19@gmail.com (D. Cao).</footnote>
## 2. Related works

### 2.1. Named entity recognition

Traditionally, NER tasks are tackled using rule-based systems. These rules are finite state patterns manually crafted by experts in the relevant domain. They are often based on morphological characteristics, regular expressions, gazetteers, syntactic and lexical patterns, and knowledge bases. For example, Appelt et al. [7] participated in the MUC-6 shared tasks using the FASTUS system. It is composed of several transducers performing text analysis on different levels, and then it uses this information to create rules for information extraction. Thielen [8] developed a proper name tagging for German based on lexicons, syntactic and morphological analysis.

More modern systems prefer to rely on statistical approaches. The central concept of said systems lies in token-level classification using machine learning and deep learning approaches. They take raw text as input, vectorize it using features or embedding algorithms, and produce token-level entity annotations using machine learning algorithms. Florian et al. [9] proposed an NER system based on features and HMMs. Mehrbod and Grilo [10] used Conditional Random Fields (CRFs, Lafferty et al. [11]) in a procurement product detection task. Paliouras et al. [12] combined features and decision trees to work on MUC-6 data.

More recently, deep neural networks have been introduced into the field, bringing even better results. Long Short-Term Memory (LSTM, Hochreiter and Schmidhuber [13]) is the most widely used algorithm. First introduced into NER applications by Chiu and Nichols [14], LSTM has found its place in many NER works, and is often combined with CRFs [15,16]. CNNs are also used to solve NER tasks, sometimes in combination with LSTMs. Strubell et al. [17] introduced iterated dilated CNNs (ID-CNNs, Yu and Koltun [18]) and achieved the same level of performance as LSTMs. Zhu and Wang [19] used Gated Recurrent Units (GRUs, Wang and Jiang [20]) instead of LSTMs.

Finally, the latest trends in the domain consist of applying Large Language Models (LLMs) on more classic tasks than text generation. Wei et al. [21] proposed a zero-shot prompt-based method using *chatGPT*. Ji [22] put forward a similar pipeline but instead with an open-source model, *Vicuna*, for security and reproducibility reasons. Such techniques are also introduced into more specific domains of application such as bio-medicine [23,24], financial intelligence [25], paleontology [26].

### 2.2. Data augmentation

Data augmentation has been a long-time subject of study in NLP. Two main kinds of methods are addressed in the context of NER. The first one relies on textual operations to replace either entities or surrounding context to generate additional samples. Wei and Zou [27] proposed EDA (Easy Data Augmentation), which uses 4 textual operations (synonym replacement, random synonym insertion, random word exchange and deletion) to produce semantically similar sentences. While the method was originally designed for text classification, Sabty et al. [28] have successfully adapted it for Arabic NER data augmentation. Kang et al. [29] also presented an extension of EDA by combining it with Unified Medical Language System (UMLS, Bodenreider [30]) to improve model performance in biomedical NER. Dai and Adel [31] also use simple text operations such as token, entity mention and phrase replacement. Following their work, Sutiono and Hahn-Powell [32] introduced syntactic information into this method so as to assure the coherence of generated data.

The second kind of method makes use of generative models to produce new data. Ng et al. [33] explored one of the training task of BERT-like language models [34] which is masked language prediction (MLM). SSMB, their proposed method, takes samples of input data, corrupts the sentences by masking certain parts, and reconstructs them using a generative model trained on the original data. Similarly, Zhou et al. [35] put forward the MELM pipeline, using a fine-tuned language model for NER data augmentation for low-resource NER. Yaseen and Langer [36] adopted a back-translation system to reformulate text around entities. Ye et al. [37] presented an LLM-based method that combines the two mentioned approaches. More specifically, the LLM is asked either to replace an entity with another one of the same type, or reformulate the sentence containing the same entity. Finally, instead of augmenting the data, Nie et al. [38] sought to enrich word embedding by including neighboring words, and this by using an attention [39]-based neural network.

## 3. Context

### 3.1. Task specification

Public procurement is the procedure by which a public entity, such as the government or a local authority, purchases supplies or services from a third-party supplier or service provider. This procedure can be extended for an additional period without having to establish a new contract, which is referred to as the renewal of the contract. A contract can be renewed multiple times, either under the same conditions as the initial contract, or with modifications. In the following example, the overall renewal duration is fixed at three years, including three periods of one year. For easier understanding, all examples in this paper are translated from French to English.

*The present contract will take effect on January 1st, 2024, for an initial duration of 2 years, renewable 3 times for one year each, until December 31st, 2028.*
<table><caption>**Table 1**
CRF criteria analysis.</caption><thead><tr><th>Category</th><th>Feature</th><th>Rank</th></tr></thead><tbody><tr><td rowspan="2">debutm</td><td>text : 2023</td><td>4/2942</td></tr><tr><td>text : 01/01/2023</td><td>9/2942</td></tr><tr><td rowspan="2">debutr</td><td>text : 1.6.2019</td><td>19/382</td></tr><tr><td>text : 01/01/2024</td><td>25/382</td></tr><tr><td rowspan="2">debutp</td><td>text:+2 : 31/12/2024</td><td>20/795</td></tr><tr><td>text : January2024</td><td>24/795</td></tr></tbody></table>

The present work is part of a project undertaken by the company OctopusMind, focused on developing a solution for detecting contract renewal specifications in French procurement documents. The expected output of the solution is an ensemble of eleven information fields:

* Start and end dates, and duration of the initial contract (debutm, finm, delaim)

* Start and end dates, and duration of the overall renewal (debutr, finr, delair)

* Start and end dates, and duration of each renewal period (debutp, finp, period)

* Number of renewal periods allowed (nb rec.)

* Overall duration of the contract, all periods being included (dureem)

In the existing solution, a Conditional Random Fields (CRF, Lafferty et al. [11]) model is charged with the detection of available information in the text. It takes as input paragraphs of a public procurement announcement containing renewal information and makes token<fnref n="1" />-level classification. It relies on long-range features [40] to determine the class of a token, i.e. each token is represented by features describing itself and those of nearby tokens. More precisely, the following features are used: verbatim (text), lemma (lemma), POS tag (pos), like_num indicating if the token resembles a number, reg_date indicating if the token resembles a date, and finally reg_timemark indicating if the token refers to a certain notion of time (month, semester, trimester, and so on). The context window radius is fixed at 5, i.e. a total of eleven tokens are taken into account.

## 3.2. Problematic

While using the tool, we discovered that the CRF model had some difficulty detecting dates, despite its promising results during the training. This applies especially to dates that did not appear in the training data, indicating the model has a robustness problem. Concerning the source of this problem, we hypothesize that the imbalance in the data causes the model to overfit. As mentioned before, all dates of a contract are not always explicitly specified in procurement documents, which can lead to a lack of training examples for certain date types. Subsequently, the model tends to rely on specific dates, rather than the context in which dates are found, to make predictions. As a consequence, it has difficulties to generalize well when confronted with new examples. Its robustness is then potentially affected.

To confirm this hypothesis, we analyzed the classification criteria of CRF by ranking the importance of each feature to determine the category of a token. The results are shown in Table 1. For example, to identify a “debutm” token, the fact that its text is “2023” ranks fourth most important among all 2942, and “01/01/2023” the ninth. We can conclude that the model indeed relies on specific dates to detect date-related entities, which is surely not a desired behavior.

To quantify the impact of data imbalance, we evaluated the model on a temporally distant corpus. Two corpora were involved in this step. The first corpus, which was originally used to train the model, contains 2707 paragraphs extracted from procurement announcements published between 2015 and 2022 (Octo-2015/22 in the following text). It has 759,572 tokens among which 32,945 are identified as entities. Table 2 shows in detail the proportion of each type of entity. We can observe a significant imbalance among the categories, especially the three highlighted in bold. The second corpus is made up of 350 paragraphs published in 2011 (named Octo-2011). It has 55,245 tokens among which 2733 occur in entities. The evaluation is carried out with a cross-validation procedure (k = 10). Octo-2015/22 corpus is used only for training whereas Octo-2011 is used only for testing. For each fold, we train a CRF model on 90% of Octo-2015/22 (80% train + 10% dev), and once converged, it is evaluated first on the test fold (10%) of Octo-2015/22 corpus, and then on the entire Octo-2011 corpus.

In order to reduce the impact of data imbalance on the cross validation itself, the folds are generated in a stratified manner. To facilitate the process, we transform temporarily the task into a multi-label classification problem. In other words, we consider the token-level entity annotation as a multi-label classification of a paragraph. Since we only need to make sure that the proportion of entities in each fold respects that of the original corpus, we do not yet need to worry about the order of entities. Two stratification strategies are possible:

* Token-level : stratification considering the proportion of each type of tokens.

<footnote n="1">¹ Token: the smallest unity of text processing, such as a word, a letter, or a punctuation.</footnote>
<table><caption>**Table 2**
Entity counts of the corpus Octo-2015/22.</caption><thead><tr><th>Category</th><th>debutm</th><th>finm</th><th>delaim</th><th>debutr</th><th>finr</th><th>delair</th></tr></thead><tbody><tr><td>Nb. tokens</td><td>3142</td><td>2811</td><td>10 367</td><td>31</td><td>666</td><td>298</td></tr><tr><td>Nb. entities</td><td>1479</td><td>1370</td><td>4049</td><td>12</td><td>297</td><td>131</td></tr><tr><td>Category</td><td>period</td><td>nb rec.</td><td>debutp</td><td>finp</td><td>dureem</td><td></td></tr><tr><td>Nb. tokens</td><td>5504</td><td>3634</td><td>171</td><td>125</td><td>6196</td><td></td></tr><tr><td>Nb. entities</td><td>2637</td><td>2741</td><td>99</td><td>79</td><td>2382</td><td></td></tr></tbody></table>

<table><caption>**Table 3**
Cross validation results.</caption><thead><tr><th rowspan="2"></th><th colspan="3">Octo-2015/22</th><th colspan="3">Octo-2011</th><th colspan="3">loss</th></tr><tr><th>P</th><th>R</th><th>F1</th><th>P</th><th>R</th><th>F1</th><th>P</th><th>R</th><th>F1</th></tr></thead><tbody><tr><td>debutm</td><td>93.6</td><td>93.0</td><td>93.3</td><td>88.5</td><td>50.1</td><td>63.7</td><td>-5.1</td><td>-42.9</td><td>-29.6</td></tr><tr><td><strong>debutp</strong></td><td><strong>86.9</strong></td><td><strong>66.2</strong></td><td><strong>71.9</strong></td><td><strong>100.0</strong></td><td><strong>27.8</strong></td><td><strong>43.3</strong></td><td><strong>13.1</strong></td><td><strong>-38.5</strong></td><td><strong>-28.7</strong></td></tr><tr><td><strong>debutr</strong></td><td><strong>40.0</strong></td><td><strong>31.1</strong></td><td><strong>34.0</strong></td><td>-</td><td>-</td><td>-</td><td>-</td><td>-</td><td>-</td></tr><tr><td>delaim</td><td>95.9</td><td>95.1</td><td>95.5</td><td>94.5</td><td>88.4</td><td>91.3</td><td>-1.5</td><td>-6.7</td><td>-4.2</td></tr><tr><td>delair</td><td>87.4</td><td>84.8</td><td>84.9</td><td>100.0</td><td>70.1</td><td>82.4</td><td>12.6</td><td>-14.7</td><td>-2.5</td></tr><tr><td>dureem</td><td>97.0</td><td>92.2</td><td>94.5</td><td>91.7</td><td>85.4</td><td>88.4</td><td>-5.3</td><td>-6.8</td><td>-6.1</td></tr><tr><td>finm</td><td>94.0</td><td>95.8</td><td>94.9</td><td>85.9</td><td>56.5</td><td>67.8</td><td>-8.2</td><td>-39.2</td><td>-27.0</td></tr><tr><td><strong>finp</strong></td><td><strong>78.0</strong></td><td><strong>65.6</strong></td><td><strong>69.9</strong></td><td><strong>87.8</strong></td><td><strong>18.8</strong></td><td><strong>30.7</strong></td><td><strong>9.8</strong></td><td><strong>-46.8</strong></td><td><strong>-39.2</strong></td></tr><tr><td>finr</td><td>91.0</td><td>82.2</td><td>85.7</td><td>74.9</td><td>69.7</td><td>71.7</td><td>-16.2</td><td>-12.5</td><td>-14.0</td></tr><tr><td>period</td><td>97.1</td><td>95.9</td><td>96.4</td><td>92.9</td><td>83.9</td><td>88.2</td><td>-4.2</td><td>-11.9</td><td>-8.3</td></tr><tr><td>nb rec.</td><td>98.7</td><td>98.1</td><td>98.4</td><td>99.1</td><td>94.6</td><td>96.8</td><td>0.3</td><td>-3.5</td><td>-1.6</td></tr><tr><td>macro avg.</td><td>87.3</td><td>81.8</td><td>83.6</td><td>83.2</td><td>58.7</td><td>65.9</td><td>-4.1</td><td>-23.1</td><td>-17.7</td></tr></tbody></table>

• Entity-level: stratification considering the proportion of each type of entities.

After comparison, the token-level folds seem more appropriate since the stratification is more fine-grained. This process is implemented using the scikit-multilearn library [41].

The results are shown in Table 3. There is no debutr in the Octo-2011 corpus, so no valid score is provided. We observe a general performance loss in all categories, but mostly in date-related ones. For debutp and finp, the precision goes up but the recall rapidly down, which is also a clear sign of overfitting.

In addition to the previous hypothesis, we speculate that maybe the semantic and contextual information was not sufficiently encoded. The features fed to the CRF were selected following an exhaustive evaluation of all possible combinations of a larger set of empirical features. Our second hypothesis states that the feature set, although found optimal during the training process, is not sufficient to correctly represent new examples. As for the context, currently, the window size is fixed to 11, which is also selected among other values during an exhaustive evaluation. Similarly, this window value was optimal for most of the training data. However, in case a further context is necessary for a new example, the model would not be able to perform correctly.

In this work, we propose solutions to cover the drawbacks (over-fitting and insufficient input encoding) of these two hypotheses. For the first hypothesis, we develop three generation-based data augmentation approaches, with the aim of improving the entity proportion in the training corpus. For the second hypothesis, we introduce a new classification architecture, using word embedding models to improve semantic encoding, and then a bLSTM layer for automatic context management. In addition, we extend the data augmentation experiences to the newly proposed models in order to test their ability of generalization.

## 4. Proposed data augmentation methods

The methods mentioned above, while effective in certain contexts, are unfortunately unsuitable for the renewal detection task. Specifically, one of the main challenges is the underrepresentation of key entity types, particularly dates, which we aim to augment. Synonym-based approaches and simple phrase replacement methods are inadequate for this purpose, as they lack the complexity needed to generate the specific temporal information required. Furthermore, the generated text must adhere to the formal structure and format of a procurement announcement, which these methods fail to achieve consistently. Generation-based methods, though promising, may result in incoherent or unstructured data, further limiting their applicability to this task.

We present hereinafter the three data augmentation methods we consider in this study. The first method is simply based on regular expressions. The second one is inspired by Ng et al. [33] using a French BERT-like embedding model, CamemBERT [42]. And the third method relies on a generative LLM. Intending to improve entity proportion, we focus on the augmentation of the most underrepresented three categories, i.e. debutr, period and finp (which we will refer to as “target dates” in the following text). We will now introduce the three methods.
<table><caption>**Table 4**
Candidates generated by CamemBERT.</caption><thead><tr><th>Token 1</th><th>Token 2</th><th>Token 3</th><th>Token 4</th><th>Token 5</th></tr></thead><tbody><tr><td>01</td><td>.</td><td>09</td><td>.</td><td>2018</td></tr><tr><td>05</td><td>-</td><td>07</td><td>-</td><td>,</td></tr><tr><td>31</td><td>/01</td><td>08</td><td>-2018</td><td>2017</td></tr><tr><td>30</td><td>/</td><td>01</td><td>2013</td><td>2014</td></tr><tr><td>09</td><td>-01</td><td>02</td><td>2014</td><td>2015</td></tr></tbody></table>

## 4.1. Data augmentation using regex

The regex approach that we have designed replaces occurrences of target dates in the corpus with other randomly generated ones. We start by creating a regex to detect target dates in the corpus. Then for each paragraph containing at least one target date, we create five copies and replace the occurrences with randomly generated ones. We make sure to diversify the format of generated dates and also to deliberately introduce errors that document creators might make (typos, excessive spaces, etc.). Texts generated with this method strictly respect the format of procurement documents, since they are effectively copies of such documents. However, the context around the dates remains intact, hence there is a lack of variability. This could lead to overfitting on stereotypical types of documents.

## 4.2. Data augmentation using CamemBERT

The second method is based on the language model CamemBERT. More precisely, we mask the dates in the corpus and let the model complete the missing parts. This requires predefining the number of tokens of generated dates which can affect the variability of data. Moreover, if we ask the model to generate multiple candidates for each mask, the problem of candidate combination comes up. Let us take the following paragraph as an example: "The framework contract is concluded from 01/01/2016 for an initial period of 12 (twelve) months". We can mask the date "01/01/2016" and tell the model to replace it with a date of five tokens, like "01/01/2017" (/ is considered here as a token). We impose the model to generate five candidates for each token. Table 4 shows the generated candidates, sorted by the confidence score. We can effectively retrieve some viable dates such as (01, -, 09, -, 2018), but not all combinations are valid. (30, /, 01, 2013, 2014) for instance does not correspond to a date. Hence it is necessary to design a post-processing step to filter out the invalid combinations. Due to the time and resource limits of this work, we chose not to follow up this path.

## 4.3. Data augmentation with generative AI

The third method generates new data by asking an LLM to extend given prompts into full paragraphs. The model we chose is text-davinci-003 (davinci later on), a generative model of OpenAI's GPT 3.5 family. First, we prepare the prompts. We locate target dates in each sentence, and we take the twenty words preceding the first word of the first occurrence as the prompt. If the preceding part does not contain twenty words, then we stop at the beginning of the sentence. In the following example, the bold words indicate a target date, and the italic part identifies the prompt. We obtain 132 initial prompts from this step.

- The contract is concluded from the day of signature of the contract to December 31st 2023. The contract is renewable for the following periods: - from January 1st 2024 to December 31st, 2024; - from January 1st 2025 to December 31st, 2025.

Some of these prompts can contain non-target dates, as in the above example. These dates can limit the "creativity" of davinci, who may simply complete the calendar instead of coming up with a new random one. Using the example prompt, we can see the same calendar being repeated several times in the model's responses:

* from January 1st 2024 to December 31st, 2024; - from January 1st 2025 to December 31st, 2025

* from January 1st 2024 to December 31st, 2024; - from January 1st 2025 to December 31st, 2025. The buyer may optionally commission contract amendments and/or increases based on the criteria in the tender documents to the successful bidder of this tender.

* from January 1st 2024 to December 31st, 2024; - from January 1st 2025 to December 31st, 2025. The contract will end automatically after these periods unless one of the parties expresses intention of renewal before the last day of said period.

To avoid such duplicates, we reuse the regex strategy. For each prompt containing a non-target date, we create five copies and replace these non-target dates with random ones. This augmentation step gives a total of 386 ready-to-use prompts. Then these prompts are fed to davinci, who generates ten paragraphs for each prompt, which makes 3860 expected paragraphs. Finally, a semi-automatic paragraph selection is applied to filter out non-conform paragraphs such as (1) duplicates (2) non-procurement-related paragraphs and (3) non-renewal-related paragraphs. We obtain 1159 final paragraphs of high quality.

Table 5 shows the token counts of each type of entities in subcorpus generated with regex and davinci. In absolute numbers, regex provides the most synthesized data with almost 500% gain. This method is simple and, most importantly, free to use. However, davinci provides a more targeted augmentation, where the proportion of target dates are largely augmentation compared to the original corpus. The latter also presents a better variability in generated data. The shortcoming though is the paying nature of the tool, forcing us to reduce the amount of generated data and thus obtain a much smaller corpus than regex.
<table><caption>**Table 5**

Token count of data generated with regex and davinci.</caption><thead><tr><th>Category</th><th>debutm</th><th>finm</th><th>delaim</th><th>debutr</th><th>finr</th><th>delair</th></tr></thead><tbody><tr><td>origin</td><td>3142</td><td>2811</td><td>10 367</td><td>31</td><td>666</td><td>298</td></tr><tr><td>regex</td><td>15 390</td><td>13 970</td><td>6105</td><td>150</td><td>3320</td><td>415</td></tr><tr><td>davinci</td><td>792</td><td>1613</td><td>239</td><td>355</td><td>1458</td><td>175</td></tr></tbody></table><table><thead><tr><th>Category</th><th>period</th><th>nb rec.</th><th>debutp</th><th>finp</th><th>dureem</th></tr></thead><tbody><tr><td>origin</td><td>5504</td><td>3634</td><td>171</td><td>125</td><td>6196</td></tr><tr><td>regex</td><td>8495</td><td>5655</td><td>745</td><td>625</td><td>6105</td></tr><tr><td>davinci</td><td>1531</td><td>540</td><td>2030</td><td>1999</td><td>371</td></tr></tbody></table>

<figure><img src="image_6.png" /><figcaption>**Fig. 1.** Architecture of proposed methods.</figcaption></figure>

## 5. Experiment protocol

The evaluated model architecture is shown in Fig. 1. We apply a standard 10-fold cross-validation procedure. We adopt the same fold segmentation mentioned in Section 3, to keep the results comparable to those of our baseline CRF. The difference is that this time we introduce synthesized corpus into the train set. The models are evaluated at token level, using precision, recall and F1 score as metrics. For each metric, we calculate the average of ten folds and this constitutes the final result. Each model is evaluated four times:

* once without synthesized data (Origin)

* once with regex subcorpus (Regex)

* once with davinci subcorpus (Davinci)

* once with both corpus (combine)

The corpus (including Octo-2015/22, Octo-2011, regex and davinci) is made available on GitHub.<fnref n="2" />

Concerning new embedding-based models (cf. ), five embedding models are compared:

* fastText [43] (300D), French model pre-trained on Common Crawls, provided by the development team of fastText.

* fastText (300D) “métier” pre-trained on a corpus of procurement documents, property of OctopusMind

* CamemBERT (768D)

* multilingual BERT (768D)

* XLM-R [44] (768D)

The choice of these embeddings are made in accordance with results published in [45] since they showed the best performance on French WikiNER among their respective category. All embeddings are then fed into a biLSTM-CRF as an entity tagger. This process is implemented with the library flair [46]. We use the default training parameters of flair: biLSTM of 512 neurons, 100 epochs max with early stopping, initial learning rate at 0.1 which is reduced over time.

Both fastText models are used with their ngram features, meaning that they compute a vector for out-of-vocabulary words based on their component ngrams. As transformer-based models, we refer to the approach proposed by the author of BERT [34] to produce embeddings. After comparing several embedding generation strategies on CoNLL 2003 [47], the best result was obtained

<footnote n="2">² The first two corpora have been renamed to OM-renewal-2015/22 and OM-renewal-2011 respectively.</footnote>
<table><caption>**Table 6**
Results of baseline CRF on Octo-2015/22.</caption><thead><tr><th rowspan="2">Entity type</th><th colspan="3">Regex</th><th colspan="2">Davinci</th><th colspan="2">Combine</th></tr><tr><th>F1</th><th>F1</th><th>Δ</th><th>F1</th><th>Δ</th><th>F1</th><th>Δ</th></tr></thead><tbody><tr><td>debutp</td><td>71.9</td><td>82.3</td><td>10.3</td><td>71.5</td><td>-0.4</td><td>78.7</td><td>6.8</td></tr><tr><td>debutr</td><td>34.0</td><td>40.6</td><td>6.5</td><td>52.5</td><td>18.5</td><td>73.9</td><td>17.2</td></tr><tr><td>finp</td><td>69.9</td><td>77.0</td><td>7.1</td><td>77.4</td><td>7.4</td><td>86.5</td><td>16.5</td></tr><tr><td>Macro avg.</td><td>83.6</td><td>86.0</td><td>2.4</td><td>84.1</td><td>0.5</td><td><strong>88.4</strong></td><td>4.8</td></tr></tbody></table>

<table><caption>**Table 7**
Results of fastText + biLSTM-CRF on Octo-2015/22.</caption><thead><tr><th rowspan="2">Entity type</th><th colspan="3">Regex</th><th colspan="2">Davinci</th><th colspan="2">Combine</th></tr><tr><th>F1</th><th>F1</th><th>Δ</th><th>F1</th><th>Δ</th><th>F1</th><th>Δ</th></tr></thead><tbody><tr><td>CC</td><td></td><td></td><td></td><td></td><td></td><td></td><td></td></tr><tr><td>debutp</td><td>63.4</td><td>85.5</td><td>22.1</td><td>79.7</td><td>16.3</td><td>86.8</td><td>23.5</td></tr><tr><td>debutr</td><td>33.3</td><td>95.6</td><td>62.2</td><td>57.5</td><td>24.1</td><td>98.9</td><td>65.6</td></tr><tr><td>finp</td><td>70.7</td><td>94.3</td><td>23.6</td><td>91.4</td><td>20.7</td><td>97.8</td><td>27.1</td></tr><tr><td>Macro avg.</td><td>80.1</td><td>89.8</td><td>9.7</td><td>84.5</td><td>4.5</td><td><strong>90.3</strong></td><td>10.3</td></tr><tr><td>métier</td><td></td><td></td><td></td><td></td><td></td><td></td><td></td></tr><tr><td>debutp</td><td>67.9</td><td>84.9</td><td>17.0</td><td>75.5</td><td>7.6</td><td>81.5</td><td>13.7</td></tr><tr><td>debutr</td><td>16.7</td><td>85.3</td><td>68.6</td><td>88.3</td><td>71.6</td><td>87.4</td><td>70.7</td></tr><tr><td>finp</td><td>75.4</td><td>96.2</td><td>20.8</td><td>85.4</td><td>10.0</td><td>96.2</td><td>20.8</td></tr><tr><td>Macro avg.</td><td>80.6</td><td><strong>89.0</strong></td><td>8.4</td><td>85.5</td><td>4.9</td><td><strong>89.0</strong></td><td>8.4</td></tr></tbody></table>

by concatenating the hidden states of the last four layers (F1 96.1), followed by their average (F1 95.9). Intending to limit the complexity of the models for easier production use, we adopt the second strategy. More precisely, the model takes as input the text and processes it with its layers. We take the output of the activation of the last four hidden layers, and we calculate the token-level average.

For transformer-based models, we also compare their other common usage: end-to-end classification (finetuning). We simply add a perceptron layer for classification and we finetune the hyperparameters of the models for a maximum of 8 epochs. For every 100 steps, we make a checkpoint, and we keep the best step as the final model.

# 6. Results and discussions

## 6.1. Evaluation on Octo-2015/22

Tables 6–9 present the evaluation results of each model following data augmentation. These results provide a direct view of the impact of data augmentation on target categories as well as the overall performance. For better clarity, we report here the F1 score of the target categories and the macro average of all categories. We highlight the best score of each model obtained with the corresponding data augmentation strategy.

In most cases, data augmentation improves the performance of the models, and most often with **Combine**. On the three target categories, we can observe up to 85% of F1 score gains (finetuned XLM-R with **Combine** on **debutr**). The models do not always react the same way following different data augmentation strategies. For example, fastText CC + biLSTM-CRF prefers **Regex** to **Davinci**, but finetuned XLM-R goes the other way around. Combining the two augmentation corpora generally brings more significant gain than using them individually. And mBERT does not show improvement when augmentation corpora are introduced separately but gains 6% once the combined corpora.

One potential factor that might affect the impact of the evaluated augmentation methods is the entity proportion of the generated corpora. As shown earlier in Table 5, compared to **Regex**, the overall size of **Davinci** is much smaller, but the relative proportion of the target entities has greatly augmented. This shift in proportion may bias the training results of the models because, as a consequence, the proportion of other entity types is reduced. Perhaps the models previously optimized their overall performance by focusing on the abundant categories in the original data. However, with the shifting in the corpus balance, this focus is now more oriented toward the target entities. As a result, the models' performance may decrease for the previously abundant categories, ultimately reducing the overall performance of those categories. For example, in the following utterance, "January 1, 2023" is the start date of the initial contract, and is one of the non-enriched entity categories. The baseline CRF model (without data augmentation) and the **Regex**-enhanced CRF both detected this entity correctly, whereas the **Davinci**-enhanced CRF failed to recognize it.

Estimated contract start date: January 1, 2023. The contract is concluded for a duration of 12 months, renewable 3 times for the same duration, totaling 48 months including all periods.
<table><caption>**Table 8**
Results of finetuned transformers on Octo-2015/22.</caption><thead><tr><th rowspan="2">Entity type</th><th>Origin</th><th colspan="2">Regex</th><th colspan="2">Davinci</th><th colspan="2">Combine</th></tr><tr><th>F1</th><th>F1</th><th>Δ</th><th>F1</th><th>Δ</th><th>F1</th><th>Δ</th></tr></thead><tbody><tr><td colspan="8"><strong>CamemBERT</strong></td></tr><tr><td>debutp</td><td>91.9</td><td>86.5</td><td>-5.4</td><td>89.0</td><td>-2.8</td><td>98.6</td><td>6.7</td></tr><tr><td>debutr</td><td>87.0</td><td>0.0</td><td>-87.0</td><td>78.5</td><td>-8.5</td><td>97.4</td><td>10.4</td></tr><tr><td>finp</td><td>94.0</td><td>94.1</td><td>0.1</td><td>92.1</td><td>-1.8</td><td>99.1</td><td>5.2</td></tr><tr><td>Macro avg.</td><td>88.1</td><td>83.2</td><td>-4.9</td><td>86.8</td><td>-1.3</td><td><strong>92</strong></td><td>3.9</td></tr><tr><td colspan="8"><strong>mBERT</strong></td></tr><tr><td>debutp</td><td>87.7</td><td>82.9</td><td>-4.7</td><td>90.7</td><td>3.1</td><td>95.9</td><td>8.2</td></tr><tr><td>debutr</td><td>73.4</td><td>60.6</td><td>-12.9</td><td>76.5</td><td>3.1</td><td>97.4</td><td>24</td></tr><tr><td>finp</td><td>78.4</td><td>84.4</td><td>5.9</td><td>90.2</td><td>11.7</td><td>98.7</td><td>20.2</td></tr><tr><td>Macro avg.</td><td>85.2</td><td>84.2</td><td>-1.0</td><td>85.7</td><td>0.5</td><td><strong>91.2</strong></td><td>6</td></tr><tr><td colspan="8"><strong>XLM-R</strong></td></tr><tr><td>debutp</td><td>83.0</td><td>85.7</td><td>2.7</td><td>91.6</td><td>8.6</td><td>94.7</td><td>11.7</td></tr><tr><td>debutr</td><td>13.9</td><td>16.7</td><td>2.8</td><td>72.5</td><td>58.7</td><td>98.9</td><td>85.0</td></tr><tr><td>finp</td><td>85.5</td><td>93.3</td><td>7.7</td><td>92.2</td><td>6.7</td><td>98.6</td><td>13.1</td></tr><tr><td>Macro avg.</td><td>82.3</td><td>82.8</td><td>0.6</td><td>86.3</td><td>4.1</td><td><strong>91.1</strong></td><td>8.9</td></tr></tbody></table>

<table><caption>**Table 9**
Results of transformers + biLSTM-CRF on Octo-2015/22.</caption><thead><tr><th rowspan="2">Entity type</th><th>Origin</th><th colspan="2">Regex</th><th colspan="2">Davinci</th><th colspan="2">Combine</th></tr><tr><th>F1</th><th>F1</th><th>Δ</th><th>F1</th><th>Δ</th><th>F1</th><th>Δ</th></tr></thead><tbody><tr><td colspan="8"><strong>CamemBERT</strong></td></tr><tr><td>debutp</td><td>78.9</td><td>89.7</td><td>10.8</td><td>89.8</td><td>11.0</td><td>87.0</td><td>8.2</td></tr><tr><td>debutr</td><td>33.3</td><td>91.7</td><td>58.4</td><td>97.2</td><td>63.9</td><td>89.6</td><td>56.3</td></tr><tr><td>finp</td><td>62.2</td><td>97.3</td><td>35.0</td><td>97.2</td><td>34.9</td><td>97.9</td><td>35.6</td></tr><tr><td>Macro avg.</td><td>82.9</td><td>90.4</td><td>7.6</td><td><strong>92.0</strong></td><td>9.1</td><td>88.5</td><td>5.7</td></tr><tr><td colspan="8"><strong>mBERT</strong></td></tr><tr><td>debutp</td><td>88.3</td><td>87.5</td><td>-0.8</td><td>95.3</td><td>7.0</td><td>87.3</td><td>-1.0</td></tr><tr><td>debutr</td><td>54.6</td><td>91.7</td><td>37.1</td><td>100.0</td><td>45.4</td><td>98.9</td><td>44.3</td></tr><tr><td>finp</td><td>88.1</td><td>98.0</td><td>9.9</td><td>97.2</td><td>9.1</td><td>98.2</td><td>10.1</td></tr><tr><td>Macro avg.</td><td>88.6</td><td>90.1</td><td>1.5</td><td><strong>92.7</strong></td><td>4.1</td><td>90.3</td><td>1.7</td></tr><tr><td colspan="8"><strong>XLM-R</strong></td></tr><tr><td>debutp</td><td>87.8</td><td>84.3</td><td>-3.6</td><td>89.2</td><td>1.3</td><td>88.2</td><td>0.4</td></tr><tr><td>debutr</td><td>66.7</td><td>70.7</td><td>4.1</td><td>47.0</td><td>-19.6</td><td>98.9</td><td>32.2</td></tr><tr><td>finp</td><td>89.4</td><td>96.3</td><td>7.0</td><td>92.9</td><td>3.5</td><td>97.6</td><td>8.2</td></tr><tr><td>Macro avg.</td><td><strong>89.4</strong></td><td>86.8</td><td>-2.6</td><td>87.9</td><td>-1.5</td><td><strong>89.4</strong></td><td>0.0</td></tr></tbody></table>

<table><caption>**Table 10**
Results of baseline CRF on Octo-2011.</caption><thead><tr><th rowspan="2">Entity type</th><th>Origin</th><th colspan="2">Regex</th><th colspan="2">Davinci</th><th colspan="2">Combine</th></tr><tr><th>F1</th><th>F1</th><th>Δ</th><th>F1</th><th>Δ</th><th>F1</th><th>Δ</th></tr></thead><tbody><tr><td>debutp</td><td>43.3</td><td>65.2</td><td>22.0</td><td>66.3</td><td>23.0</td><td>84.7</td><td>41.4</td></tr><tr><td>finp</td><td>30.7</td><td>62.0</td><td>31.3</td><td>76.2</td><td>45.5</td><td>84.3</td><td>53.6</td></tr><tr><td>Macro avg.</td><td>65.9</td><td>70.8</td><td>4.9</td><td>74.1</td><td>8.2</td><td><strong>76.0</strong></td><td>10.1</td></tr></tbody></table>

## 6.2. Evaluation on Octo-2011

In addition, we evaluated the models on Octo-2011 to verify their robustness following data augmentation. Tables 10–13 show the evaluation results. As mentioned earlier, there is no debutr in the Octo-2011 corpus, therefore we present here only the F1 scores of debutp, finp and the macro average of all categories.

All models obtain a lower score on Octo-2011 than Octo-2015/22. This is expected since technically the models have never seen these data, and there may be some differences in the text due to the time difference. Most models however still see their overall performance improve with data augmentation. The highest F1 score is 82.7, obtained by fastText métier + biLSTM-CRF with Regex or Combine, and by CamemBERT + biLSTM-CRF with Regex. The baseline CRF witnesses the most significant gain of 10.1 with Combine. On individual categories, we observe mostly improvements on both debutp and finp. The biggest loss at
<table><caption>**Table 11**
Results of fastText + biLSTM-CRF on Octo-2011.</caption><thead><tr><th rowspan="2">Entity type</th><th colspan="3">Regex</th><th colspan="2">Davinci</th><th colspan="2">Combine</th></tr><tr><th>F1</th><th>F1</th><th>Δ</th><th>F1</th><th>Δ</th><th>F1</th><th>Δ</th></tr></thead><tbody><tr><td colspan="8"><strong>CC</strong></td></tr><tr><td>debutp</td><td>82.3</td><td>81.1</td><td>-1.2</td><td>76.6</td><td>-5.7</td><td>77.6</td><td>-4.8</td></tr><tr><td>finp</td><td>55.2</td><td>72.3</td><td>17.1</td><td>71.3</td><td>16.1</td><td>74.1</td><td>18.9</td></tr><tr><td>Macro avg.</td><td>74.4</td><td><strong>78.2</strong></td><td>3.8</td><td>76.1</td><td>1.7</td><td>78.1</td><td>3.7</td></tr><tr><td colspan="8"><strong>métier</strong></td></tr><tr><td>debutp</td><td>79.9</td><td>94.4</td><td>14.5</td><td>78.3</td><td>-1.6</td><td>81.9</td><td>2.0</td></tr><tr><td>finp</td><td>65.5</td><td>89.9</td><td>24.4</td><td>78.6</td><td>13.1</td><td>89.9</td><td>24.4</td></tr><tr><td>Macro avg.</td><td>78.6</td><td><strong>82.7</strong></td><td>4.1</td><td>78.9</td><td>0.4</td><td><strong>82.7</strong></td><td>4.1</td></tr></tbody></table>

<table><caption>**Table 12**
Results of finetuned transformers on Octo-2011.</caption><thead><tr><th rowspan="2">Entity type</th><th colspan="3">Regex</th><th colspan="2">Davinci</th><th colspan="2">Combine</th></tr><tr><th>F1</th><th>F1</th><th>Δ</th><th>F1</th><th>Δ</th><th>F1</th><th>Δ</th></tr></thead><tbody><tr><td colspan="8"><strong>CamemBERT</strong></td></tr><tr><td>debutp</td><td>81.9</td><td>84.8</td><td>2.8</td><td>81.3</td><td>-0.6</td><td>75.2</td><td>-6.7</td></tr><tr><td>finp</td><td>77.1</td><td>74.2</td><td>-2.8</td><td>83.5</td><td>6.4</td><td>87.2</td><td>10.2</td></tr><tr><td>Macro avg.</td><td>79.3</td><td>79.4</td><td>0.1</td><td><strong>80.2</strong></td><td>0.9</td><td>79.7</td><td>0.3</td></tr><tr><td colspan="8"><strong>mBERT</strong></td></tr><tr><td>debutp</td><td>77.6</td><td>78.5</td><td>0.9</td><td>80.3</td><td>2.7</td><td>77.5</td><td>-0.1</td></tr><tr><td>finp</td><td>59.8</td><td>68.1</td><td>8.3</td><td>75.2</td><td>15.4</td><td>82.2</td><td>22.4</td></tr><tr><td>Macro avg.</td><td>76.5</td><td>77.5</td><td>1.0</td><td>77.7</td><td>1.2</td><td><strong>79.2</strong></td><td>2.7</td></tr><tr><td colspan="8"><strong>XLM-R</strong></td></tr><tr><td>debutp</td><td>78.6</td><td>76.3</td><td>-2.4</td><td>76.1</td><td>-2.6</td><td>79.6</td><td>1.0</td></tr><tr><td>finp</td><td>66.3</td><td>67.7</td><td>1.3</td><td>76.2</td><td>9.9</td><td>81.0</td><td>14.7</td></tr><tr><td>Macro avg.</td><td>77.5</td><td>77.1</td><td>-0.4</td><td>78.0</td><td>0.5</td><td><strong>79.9</strong></td><td>2.4</td></tr></tbody></table>

<table><caption>**Table 13**
Results of transformers + biLSTM-CRF on Octo-2011.</caption><thead><tr><th rowspan="2">Entity type</th><th colspan="3">Regex</th><th colspan="2">Davinci</th><th colspan="2">Combine</th></tr><tr><th>F1</th><th>F1</th><th>Δ</th><th>F1</th><th>Δ</th><th>F1</th><th>Δ</th></tr></thead><tbody><tr><td colspan="8"><strong>CamemBERT</strong></td></tr><tr><td>debutp</td><td>71.2</td><td>91.6</td><td>2.8</td><td>82.1</td><td>10.9</td><td>84.3</td><td>13.1</td></tr><tr><td>finp</td><td>51.0</td><td>89.6</td><td>38.6</td><td>75.5</td><td>24.4</td><td>81.3</td><td>30.3</td></tr><tr><td>Macro avg.</td><td>75.4</td><td><strong>82.7</strong></td><td>7.3</td><td>79.4</td><td>4.0</td><td>80.0</td><td>4.6</td></tr><tr><td colspan="8"><strong>mBERT</strong></td></tr><tr><td>debutp</td><td>90.2</td><td>90.2</td><td>0.0</td><td>80.9</td><td>-9.3</td><td>86.6</td><td>-3.6</td></tr><tr><td>finp</td><td>78.8</td><td>84.7</td><td>5.9</td><td>78.4</td><td>-0.4</td><td>80.7</td><td>1.9</td></tr><tr><td>Macro avg.</td><td>81.1</td><td><strong>82.5</strong></td><td>1.4</td><td>79.7</td><td>4.0</td><td>80.3</td><td>-0.8</td></tr><tr><td colspan="8"><strong>XLM-R</strong></td></tr><tr><td>debutp</td><td>83.5</td><td>91.1</td><td>7.6</td><td>76.1</td><td>-7.5</td><td>86.9</td><td>3.4</td></tr><tr><td>finp</td><td>67.9</td><td>86.5</td><td>18.6</td><td>71.2</td><td>3.3</td><td>79.7</td><td>11.8</td></tr><tr><td>Macro avg.</td><td>79.3</td><td><strong>81.8</strong></td><td>2.5</td><td>78.2</td><td>-1.0</td><td>81.2</td><td>1.9</td></tr></tbody></table>

–9.3% comes from mBERT on debutp. One possible explanation would be that, since the model already achieved an acceptable level of performance without data augmentation, perhaps new data introduced more noise and helpful examples.

Interestingly, models based on biLSTM-CRF seem to benefit more from Regex than the other methods, contrary to what we observed in Octo-2015/22. This may indicate that Regex helps better at improving the robustness of these models. This is also an interesting lead for the company, since Regex is much cheaper to deploy.

## 7. Conclusion and perspectives

In this paper, we presented three data augmentation methods to improve the proportion of underrepresented entities, among which two were put to evaluation. We evaluated the impact of the introduction of synthesized data in many different models,
including the baseline CRF currently in use by the company and embedding-based classifiers. Data augmentation in general helps improve the performance, as well as the robustness, of models.

In future works, we plan to test other open-source generation models such as LlaMa3, Mistral [48], Mixtral, and Claire. This would allow us to harmonize the size of the synthesized corpus at a much lower cost. Besides, we can finetune these models on company data so that generated data corresponds better to our needs. Another interesting lead would be to evaluate the models on a document level since this is the most important output for the company. This may require further work on model architecture and loss functions so that the models are optimized on the desired level of evaluation. Finally, we did a preliminary analysis of the annotations produced by the models. The results show that the errors do not always overlap, to which another possible solution would be a meta-model combining the annotations to correct some more errors maybe.

## CRediT authorship contribution statement

**Danrun Cao:** Writing – review & editing, Writing – original draft, Visualization, Validation, Supervision, Software, Formal analysis, Data curation, Conceptualization.
**Nicolas Béchet:** Writing – review & editing, Validation, Supervision, Methodology, Formal analysis, Conceptualization.
**Pierre-François Marteau:** Writing – review & editing, Validation, Supervision, Methodology, Formal analysis, Conceptualization.
**Oussama Ahmia:** Writing – review & editing, Visualization, Validation, Supervision, Methodology, Data curation, Conceptualization.

## Declaration of competing interest

The authors declare that they have no known competing financial interests or personal relationships that could have appeared to influence the work reported in this paper.

## Data availability

The corpora are made available on github and the link is indicated in the text.

## References

[1] M. Cho, J. Ha, C. Park, S. Park, Combinatorial feature embedding based on CNN and LSTM for biomedical named entity recognition, J. Biomed. Inform. (2020) 8.

[2] G. Murugesan, S. Abdulkadhar, B. Bhasuran, J. Natarajan, BCC-NER: bidirectional, contextual clues named entity tagger for gene/protein mention recognition, J. Bioinform. Syst. Biol. 2017 (1) (2017) 7, URL: https://bsb-eurasipjournals.springeropen.com/articles/10.1186/s13637-017-0060-6.

[3] J.C.S. Alvarado, K. Verspoor, T. Baldwin, Domain adaption of named entity recognition to support credit risk assessment, in: Proceedings of the Australasian Language Technology Association Workshop 2015, Parramatta, Australia, 2015, pp. 84–90, URL: https://aclanthology.org/U15-1010.

[4] H. Jayakumar, M.S. Krishnakumar, V.V.V. Peddagopu, R. Sridhar, RNN based question answer generation and ranking for financial documents using financial NER, Sādhanā 45 (1) (2020) 269, URL: http://link.springer.com/10.1007/s12046-020-01501-3.

[5] Ł. Borchmann, D. Wisniewski, A. Gretkowski, I. Kosmala, D. Jurkiewicz, Ł. Szalkiewicz, G. Pąka, K. Kaczmarek, A. Kaliska, F. Graliński, Contract discovery: Dataset and a few-shot semantic retrieval challenge with competitive baselines, in: Findings of the Association for Computational Linguistics: EMNLP 2020, Association for Computational Linguistics, Online, 2020, pp. 4254–4268, URL: https://www.aclweb.org/anthology/2020.findings-emnlp.380.

[6] L. Sun, K. Zhang, Y. Sun, F. Weng, J. Zhang, ETIP: a lengthy nested NER problem for Chinese insurance policy analysis, Pattern Anal. Appl. 23 (4) (2020) 1755–1765, URL: https://link.springer.com/10.1007/s10044-020-00885-6.

[7] D.E. Appelt, J.R. Hobbs, J. Bear, D. Israel, M. Kameyama, A. Kehler, D. Martin, K. Myers, M. Tyson, SRI international fastus systemmuc-6 test results AND analysis, in: E: Sixth Message Understanding Conference (MUC-6): Proceedings of a Conference Held in Columbia, Maryland, November 6-8, 1995, Columbia, Maryland, 1995.

[8] C. Thielen, An approach to proper name tagging for German, 1995, http://dx.doi.org/10.48550/arXiv.cmp-lg/9506024.

[9] R. Florian, A. Ittycheriah, H. Jing, T. Zhang, Named entity recognition through classifier combination, in: Proceedings of the Seventh Conference on Natural Language Learning At HLT-NAACL 2003 –, 4, Association for Computational Linguistics, Edmonton, Canada, 2003, pp. 168–171, URL: http://portal.acm.org/citation.cfm?doid=1119176.1119201.

[10] A. Mehrbod, A. Grilo, Tender calls search using a procurement product named entity recogniser, Adv. Eng. Inform. 36 (2018) 216–228, URL: https://linkinghub.elsevier.com/retrieve/pii/S1474034617302409.

[11] J. Lafferty, A. Mccallum, F. Pereira, Conditional random fields: Probabilistic models for segmenting and labeling sequence data, in: Proceedings of the Eighteenth International Conference on Machine Learning, ICML '01, Morgan Kaufmann Publishers Inc., San Francisco, CA, USA, 2001, pp. 282–289.

[12] G. Paliouras, V. Karkaletsis, G. Petasis, C.D. Spyropoulos, Learning decision trees for named-entity recognition and classification, 2000, URL: https://api.semanticscholar.org/CorpusID:14300937.

[13] S. Hochreiter, J. Schmidhuber, Long short-term memory, Neural Comput. 9 (8) (1997) 1735–1780.

[14] J.P.C. Chiu, E. Nichols, Named entity recognition with bidirectional LSTM-CNNs, TACL 4 (2016) 357–370, URL: https://aclanthology.org/Q16-1026.

[15] T. Watanabe, A. Tamura, T. Ninomiya, T. Makino, T. Iwakura, Multi-task learning for chemical named entity recognition with chemical compound paraphrasing, in: Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing, EMNLP-IJCNLP, Association for Computational Linguistics, Hong Kong, China, 2019, p. 6, URL: https://aclanthology.org/D19-1648.

[16] L. Luo, Z. Yang, P. Yang, Y. Zhang, L. Wang, H. Lin, J. Wang, An attention-based BiLSTM-CRF approach to document-level chemical named entity recognition, in: J. Wren (Ed.), Bioinformatics 34 (8) (2018) 1381–1388, URL: https://academic.oup.com/bioinformatics/article/34/8/1381/4657076.

[17] E. Strubell, P. Verga, D. Belanger, A. McCallum, Fast and accurate entity recognition with iterated dilated convolutions, in: Proceedings of the 2017 Conference on Empirical Methods in Natural Language Processing, Association for Computational Linguistics, Copenhagen, Denmark, 2017, pp. 2670–2680, URL: http://aclweb.org/anthology/D17-1283.
[18] F. Yu, V. Koltun, Multi-scale context aggregation by dilated convolutions, 2016, arXiv:1511.07122 [cs]. URL: http://arxiv.org/abs/1511.07122. arXiv: 1511.07122.

[19] Y. Zhu, G. Wang, CAN-NER: Convolutional attention network for Chinese named entity recognition, in: Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long and Short Papers), vol. 1, Association for Computational Linguistics, Minneapolis, Minnesota, 2019, p. 10, URL: https://aclanthology.org/N19-1342.

[20] C. Wang, H. Jiang, The lower the simpler: Simplifying hierarchical recurrent models, in: Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long and Short Papers), ACL, Minneapolis, Minnesota, 2019, pp. 4005–4009, URL: https://aclanthology.org/N19-1402.

[21] X. Wei, X. Cui, N. Cheng, X. Wang, X. Zhang, S. Huang, P. Xie, J. Xu, Y. Chen, M. Zhang, Y. Jiang, W. Han, Zero-shot information extraction via chatting with ChatGPT, 2023, URL: http://arxiv.org/abs/2302.10205. arXiv:2302.10205 [cs].

[22] B. Ji, Vicuna-NER: Zero/few-shot named entity recognition using vicuna, 2023, URL: http://arxiv.org/abs/2305.03253.

[23] S.J. Jung, H. Kim, K.S. Jang, LLM based biological named entity recognition from scientific literature, in: 2024 IEEE International Conference on Big Data and Smart Computing, BigComp, IEEE, Bangkok, Thailand, 2024, pp. 433–435, URL: https://ieeexplore.ieee.org/document/10488292/.

[24] J. Biana, W. Zhai, X. Huang, J. Zheng, S. Zhu, VANER: Leveraging large language model for versatile and adaptive biomedical named entity recognition, 2024, URL: http://arxiv.org/abs/2404.17835. arXiv:2404.17835 [cs].

[25] H.S. Adibhatla, P. Baswani, M. Shrivastava, Fine-grained contract NER using instruction based model, in: Proceedings of the 37th Pacific Asia Conference on Language, Information and Computation, QCL, Hong Kong, China, 2023, pp. 889–902, URL: https://aclanthology.org/2023.paclic-1.89.

[26] K. Tsitseklis, G. Stavropoulou, S. Papavassiliou, Custom named entity recognition VS ChatGPT prompting: A paleontology experiment, in: 2024 Panhellenic Conference on Electronics & Telecommunications, PACET, IEEE, Thessaloniki, Greece, 2024, pp. 1–5, URL: https://ieeexplore.ieee.org/document/10497008/.

[27] J. Wei, K. Zou, EDA: Easy data augmentation techniques for boosting performance on text classification tasks, in: Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing, EMNLP-IJCNLP, Association for Computational Linguistics, Hong Kong, China, 2019, pp. 6381–6387, URL: https://www.aclweb.org/anthology/D19-1670.

[28] C. Sabty, I. Omar, F. Wasfalla, M. Islam, S. Abdennadher, Data augmentation techniques on arabic data for named entity recognition, Procedia Comput. Sci. 189 (2021) 292–299, URL: https://linkinghub.elsevier.com/retrieve/pii/S1877050921012126.

[29] T. Kang, A. Perotte, Y. Tang, C. Ta, C. Weng, UMLS-based data augmentation for natural language processing of clinical research literature, J. Am. Med. Inform. Assoc. 28 (4) 812–823, URL: https://academic.oup.com/jamia/article/28/4/812/6046153.

[30] O. Bodenreider, The unified medical language system (UMLS): integrating biomedical terminology, Nucleic Acids Res. 32 (90001) (2004) 267D–270, URL: https://academic.oup.com/nar/article-lookup/doi/10.1093/nar/gkh061.

[31] X. Dai, H. Adel, An analysis of simple data augmentation for named entity recognition, in: Proceedings of the 28th COLING, COLING, Barcelona, Spain (Online), 2020, pp. 3861–3867, URL: https://aclanthology.org/2020.coling-main.343.

[32] A.P. Sutiono, G. Hahn-Powell, Syntax-driven data augmentation for named entity recognition, in: Proceedings of the First Workshop on Pattern-Based Approaches To NLP in the Age of Deep Learning, COLING, Gyeongju, Republic of Korea, 2022, pp. 56–60, URL: https://aclanthology.org/2022.pandl-1.7.

[33] N. Ng, K. Cho, M. Ghassemi, SSMBA: Self-supervised manifold based data augmentation for improving out-of-domain robustness, in: Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing, EMNLP, Association for Computational Linguistics, Online, 2020, pp. 1268–1283, URL: https://www.aclweb.org/anthology/2020.emnlp-main.97.

[34] J. Devlin, M.-W. Chang, K. Lee, K. Toutanova, BERT: Pre-training of deep bidirectional transformers for language understanding, in: Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, ACL, Minneapolis, Minnesota, 2019, pp. 4171–4186.

[35] R. Zhou, X. Li, R. He, L. Bing, E. Cambria, L. Si, C. Miao, MELM: Data augmentation with masked entity language modeling for low-resource NER, in: Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), Association for Computational Linguistics, Dublin, Ireland, 2022, pp. 2251–2262, URL: https://aclanthology.org/2022.acl-long.160.

[36] U. Yaseen, S. Langer, Data augmentation for low-resource named entity recognition using backtranslation, in: Proceedings of the 18th International Conference on Natural Language Processing, ICON, NLP Association of India (NLPAI), National Institute of Technology Silchar, Silchar, India, 2021, pp. 352–358, URL: https://aclanthology.org/2021 icon-main.43.

[37] J. Ye, N. Xu, Y. Wang, J. Zhou, Q. Zhang, T. Gui, X. Huang, LLM-DA: Data augmentation via large language models for few-shot named entity recognition, 2024, URL: http://arxiv.org/abs/2402.14568. arXiv:2402.14568 [cs].

[38] Y. Nie, Y. Tian, X. Wan, Y. Song, B. Dai, Named entity recognition for social media texts with semantic augmentation, in: Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing, EMNLP, Association for Computational Linguistics, Online, 2020, pp. 1383–1391, URL: https://www.aclweb.org/anthology/2020.emnlp-main.107.

[39] A. Vaswani, N. Shazeer, N. Parmar, J. Uszkoreit, L. Jones, A.N. Gomez, L. Kaiser, I. Polosukhin, Attention is all you need, in: Proceedings of the 31st International Conference on Neural Information Processing Systems, Curran Associates Inc., Long Beach, California, USA, 2017, pp. 6000–6010.

[40] Y. Li, J. Jiang, H.L. Chieu, K.M.A. Chai, Extracting relation descriptors with conditional random fields, in: Proceedings of 5th International Joint Conference on Natural Language Processing, Proceedings of 5th International Joint Conference on Natural Language Processing, Asian Federation of Natural Language Processing, Chiang Mai, Thailand, 2011.

[41] P. Szymański, T. Kajdanowicz, A scikit-based python environment for performing multi-label classification, 2018, URL: http://arxiv.org/abs/1702.01460. arXiv:1702.01460 [cs].

[42] L. Martin, B. Muller, P.J. Ortiz Suárez, Y. Dupont, L. Romary, É. de la Clergerie, D. Seddah, B. Sagot, Camembert: a tasty french language model, in: Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, Association for Computational Linguistics, 2020, pp. 7203–7219.

[43] P. Bojanowski, E. Grave, A. Joulin, T. Mikolov, Enriching word vectors with subword information, TACL 5 (2017) 135–146.

[44] A. Conneau, K. Khandelwal, N. Goyal, V. Chaudhary, G. Wenzek, F. Guzmán, E. Grave, M. Ott, L. Zettlemoyer, V. Stoyanov, Unsupervised cross-lingual representation learning at scale, in: Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, Association for Computational Linguistics, 2020, pp. 8440–8451.

[45] D. Cao, N. Béchet, P.-F. Marteau, Étude comparative des plongements lexicaux pour l'extraction d'entités nommées en français, in: 18e Conférence en Recherche DInformation et Applications – 16e Rencontres Jeunes Chercheurs en RI – 30e Conférence sur le Traitement Automatique des Langues Naturelles – 25e Rencontre des Étudiants Chercheurs en Informatique Pour le Traitement Automatique Des Langues, Paris, France, 2023, p. 11.

[46] A. Akbik, T. Bergmann, D. Blythe, K. Rasul, S. Schweter, R. Vollgraf, FLAIR: An easy-to-use framework for state-of-the-art NLP, in: NAACL 2019, 2019, pp. 54–59.

[47] E.F.T.K. Sang, F. De Meulder, Introduction to the CoNLL-2003 shared task: Language-independent named entity recognition, in: Proceedings of the Seventh Conference on Natural Language Learning At HLT-NAACL 2003, 2003, pp. 142–147.

[48] A.Q. Jiang, A. Sablayrolles, A. Mensch, C. Bamford, D.S. Chaplot, D.d.l. Casas, F. Bressand, G. Lengyel, G. Lample, L. Saulnier, L.R. Lavaud, M.-A. Lachaux, P. Stock, T.L. Scao, T. Lavril, T. Wang, T. Lacroix, W.E. Sayed, Mistral 7B, 2023, URL: http://arxiv.org/abs/2310.06825. arXiv:2310.06825 [cs].
**Danrun Cao** received her master's degree in natural language processing from the University Paris Nanterre, during which she did a 10-month apprenticeship at the R&D of Electricity of France (EDF R&D) on the subject of "Scientific watch of text mining tools and methods". She is currently in her third year of a PhD program, a collaboration between the Institut de Recherche en Informatique et Systèmes Aléatoires (IRISA) and OctopusMind. The subject of the thesis is "Applied multilingual information extraction on a calls for tenders database and its visualization". Her works focus mainly on the industrial applications of information extraction methods and technologies, covering traditional machine learning techniques as well as LLM assisted approaches.

**Nicolas Béchet** has his Ph.D. degree in computer science from the University of Montpellier 2 (obtained in 2009 on the subject: "Extraction and grouping of morpho-syntactic descriptors for text mining processes"). After completing a postdoctoral position at INRIA Rocquencourt and one at the University of Caen, he is now an associate professor at the University of Bretagne Sud and a member of the IRISA Computer Science Lab. His research work focuses on text mining and natural language processing (NLP). He is particularly interested in combinations of data mining and NLP techniques such as the development of new pattern extraction algorithms. The latter can be used for the extraction of relationships from textual data to improve the quality of learning taxonomy or ontology, or even for the modeling of linguistic phenomena such as a style or a language register. Furthermore, he also works in the field of Machine Learning and Deep Learning and is particularly interested in classification or regression methods using textual data. This research work has been highlighted in more than 60 publications.

**Pierre-François Marteau** received his master engineering degree in computer science from Ecole Nationale Supérieure d'Electronique et d'Informatique de Bordeaux in 1984, and his Ph.D. degree in computer science in 1988 from Institut National Polytechnique de Grenoble (Grenoble INP). He experienced a post doctorate position at University of Geneva in 1989, and at the Institute for Non Linear Sciences at University of California San Diego in 1990. He then worked for eight years as an IT consultant in Bertin Technologies, a private company in Paris, before joining in 1999 the Computer Science Lab. at Université de Bretagne Sud where he has a professor. He recently joins the Institut de Recherche en Informatique et Systèmes Aléatoires (IRISA) in 2012. His current research interests include pattern recognition and machine learning with application in sequential (symbolic and digital) data processing.

**Oussama Ahmia** is a researcher in the field of computer science, currently affiliated with OctopusMind. He earned his Ph.D. from Université de Bretagne Sud in 2020, with the thesis "Assisted strategic monitoring on call for tender databases using natural language processing, text mining and deep learning" under the supervision of Pierre-François Marteau and Nicolas Béchet. His research interests include natural language processing, machine learning, and text mining.