# Token and Span Classification for Entity Recognition in French Historical Encyclopedias

Ludovic Moncla, Hédi Zeghidi

<figure><img src="image_1.png" /></figure>

## To cite this version:

Ludovic Moncla, Hédi Zeghidi. Token and Span Classification for Entity Recognition in French Historical Encyclopedias. 2025. hal-05111169

HAL Id: hal-05111169

https://hal.science/hal-05111169v1

Preprint submitted on 13 Jun 2025

HAL is a multi-disciplinary open access
archive for the deposit and dissemination of sci-
entific research documents, whether they are pub-
lished or not. The documents may come from
teaching and research institutions in France or
abroad, or from public or private research centers.

L'archive ouverte pluridisciplinaire HAL, est
destinée au dépôt et à la diffusion de documents
scientifiques de niveau recherche, publiés ou non,
émanant des établissements d'enseignement et de
recherche français ou étrangers, des laboratoires
publics ou privés.

<figure><img src="image_2.png" /></figure>

Distributed under a Creative Commons Attribution 4.0 International License
# Token and Span Classification for Entity Recognition in French Historical Encyclopedias

Ludovic Moncla and Hédi Zeghidi

INSA Lyon, CNRS, Université Claude Bernard Lyon 1,
LIRIS, UMR5205, 69621 Villeurbanne, France

Correspondence: ludovic.moncla@insa-lyon.fr

## Abstract

Named Entity Recognition (NER) in historical texts presents unique challenges due to non-standardized language, archaic orthography, and nested or overlapping entities. This study benchmarks a diverse set of NER approaches, ranging from classical Conditional Random Fields (CRFs) and spaCy-based models to transformer-based architectures such as CamemBERT and sequence-labeling models like Flair. Experiments are conducted on the GeoEDdA dataset, a richly annotated corpus derived from 18th-century French encyclopedias. We propose framing NER as both token-level and span-level classification to accommodate complex nested entity structures typical of historical documents. Additionally, we evaluate the emerging potential of few-shot prompting with generative language models for low-resource scenarios. Our results demonstrate that while transformer-based models achieve state-of-the-art performance, especially on nested entities, generative models offer promising alternatives when labeled data are scarce. The study highlights ongoing challenges in historical NER and suggests avenues for hybrid approaches combining symbolic and neural methods to better capture the intricacies of early modern French text.

## 1 Introduction

The field of digital humanities increasingly depends on computational tools to extract, analyze, and structure information from vast historical corpora. Among the most significant artifacts of Enlightenment knowledge dissemination is the Encyclopédie, edited by Diderot, d'Alembert, and their collaborators. Published in the mid-18th century, this extensive compendium reflects the intellectual, geographical, and scientific worldview of the era. Extracting semantic data from such texts opens up novel opportunities for historical analysis and spatial humanities.

However, processing French historical texts poses unique challenges. Unlike modern corpora, these documents feature non-standardized language, historical orthography, and complex syntactic patterns, which hinder the straightforward application of standard Natural Language Processing (NLP) techniques (McDonough et al., 2019). Furthermore, geographical and named entity mentions are often embedded within hierarchical or nested contexts (Gaio and Moncla, 2017). These difficulties necessitate flexible and robust Named Entity Recognition (NER) methodologies tailored for historical data.

In this paper, we address these challenges by framing NER as both token-level and span-level classification tasks, capable of recognizing nested and overlapping entities, including proper nouns, common nouns, and spatial relations. We conduct a comprehensive benchmark of six distinct NER architectures, from classical CRF-based models and spaCy's CNN and span-based systems to cutting-edge transformer models like CamemBERT and generative large language models (LLMs) such as GPT variants, fine-tuned or prompted on a carefully annotated subset of the Encyclopédie. Our contributions include quantitative analyses of performance on flat and nested entities, a critical comparison of few-shot learning versus fully supervised methods, and a discussion of the implications for digital humanities applications.

## 2 Related works

Named Entity Recognition (NER) has been a longstanding task in Natural Language Processing (NLP), with a rich literature base spanning over two decades. Classical NER focused on identifying entities such as persons, organizations, and locations in newswire texts, leading to the development of early rule-based systems and statistical models. Over time, the field evolved with advances in ma-
chine learning and deep learning architectures.

Rule-based NER systems were among the ear-
liest approaches, relying on handcrafted linguistic
rules and gazetteers. Although limited in adaptabil-
ity, these systems offer interpretability and can be
fine-tuned with domain expertise. Perdido (Mon-
cla and Gaio, 2023) exemplifies a domain-specific
rule-based system for French geoparsing. It per-
forms two major tasks—geotagging (identifying
spatial entities) and geocoding (resolving coordi-
nates)—and has proven effective in French histori-
cal corpora.

Machine learning methods, particularly Condi-
tional Random Fields (CRFs), brought probabilistic
modeling to NER. CRFs consider both word-level
features and contextual dependencies, enabling
globally coherent label sequences. Unlike token-
level classifiers that make independent predictions,
CRFs model the conditional probability of an en-
tire label sequence given the input, which is crucial
for ensuring valid tag transitions. When combined
with contextualized representations from models
like BiLSTMs or transformers, CRFs provide a
mechanism for enforcing global consistency in la-
bel assignments, leading to improved performance
on benchmark datasets (Huang et al., 2015).

Building on the success of CRFs and neural ar-
chitectures, Flair (Akbik et al., 2019) integrates
character-level language model embeddings with
a BiLSTM-CRF sequence labeling architecture.
Flair embeddings capture subword information and
contextual dependencies more effectively than tra-
ditional word embeddings. The framework sup-
ports various embeddings—including contextual
string embeddings, BERT, and ELMo—which can
be stacked to enhance model robustness. The un-
derlying BiLSTM captures contextual information
bidirectionally, while the CRF layer models depen-
dencies between output labels to ensure globally
optimal tag sequences.

Transformer-based models, particularly BERT
(Bidirectional Encoder Representations from Trans-
formers) (Devlin et al., 2018), and its French
variant CamemBERT (Martin et al., 2020), have
significantly advanced the state of the art in
NER. BERT leverages self-attention mechanisms
(Vaswani et al., 2017) to capture deep bidirectional
context across entire input sequences. This enables
a more nuanced understanding of word meaning
based on context, which is especially beneficial for
disambiguating entity types. In NER tasks, BERT
is typically fine-tuned by adding a token-level clas-

sification head. This paradigm has consistently
outperformed traditional architectures, including
BiLSTM-CRF models, on datasets such as CoNLL-
2003 and OntoNotes 5.0.

Building upon BERT's architecture, SpanBERT (Joshi et al., 2020) improves pretraining by focusing on span-level representations. It replaces BERT's token-level masking with span-level masking and introduces a span-boundary objective that trains the model to predict masked spans using only the boundary token representations. This approach enables SpanBERT to learn richer contextual representations of text spans. While it has shown notable improvements on tasks like question answering and coreference resolution, its span-centric modeling is also well suited to NER, particularly for capturing multi-token entities and complex contextual patterns.

Recently, (Tafer and Gaio, 2025) proposed a
comparative study of three architectures (includ-
ing Bi-LSTM+CRF, BERT, and GliNER (Zaratiana
et al., 2024)) for extracting spatial information from
itinerary descriptions in French. The annotation
schema also includes nominal mentions, named
entities and nested entities but applied to contem-
porary texts focused on displacement descriptions.

Recent studies explore prompt engineering and
context scaling strategies in LLMs for NER, reveal-
ing trade-offs in performance depending on model
size, example formatting, and input length (Xie
et al., 2023; Li and Zhang, 2023; Agarwal et al.,
2024; González-Gallardo et al., 2023; Zeghidi and
Moncla, 2024).

Standard NER systems typically assume flat,
non-overlapping entity spans. However, texts, par-
ticularly in digital humanities, often contain over-
lapping and nested entities that violate this as-
umption. Some approaches, such as joint-labeling
strategies, attempt to handle nesting by converting
nested entities into combinatorial labels, reducing
the task to token classification. In contrast, mod-
els like spaCy's spans adopt a span-based for-
mulation, using span-entity compatibility scoring
to identify and classify candidate spans. These
innovations are particularly important for entity
recognition in rich historical corpora, where com-
plex entity structures are common. In this work,
we build on these insights by providing an empiri-
cal comparison tailored to the domain-specific task
of historical entity recognition, leveraging diverse
modeling strategies and detailed evaluation proto-
cols.
<figure><img src="image_3.png" /><figcaption>Figure 1: Excerpt of annotations from the ILLESCAS entry.</figcaption></figure>

# 3 Methodology

## 3.1 Dataset Description

The foundation of our experiments is the GeoEDdA dataset<fnref n="1" /> (Moncla et al., 2024), developed as part of the GEODE project<fnref n="2" />. GeoEDdA consists of 2,200 paragraphs sampled from more than 2,000 entries in the 18th-century Encyclopédie. Each paragraph is annotated with spans corresponding to twelve different entity types, including common and proper nouns for spatial, person, and miscellaneous entities. The annotations were manually refined using active learning, leveraging the Prodigy annotation tool integrated with spaCy models (Moncla et al., 2024).

Each annotation includes not only the start and end character offsets and the corresponding entity label but also contextual metadata such as paragraph source, volume number, and thematic classification (e.g., Geography, History, Medicine) (Brenon et al., 2022). The dataset is partitioned into three splits: 1,800 paragraphs for training, 200 for validation, and 200 for testing. It supports nested entities by assigning overlapping spans, making it suitable for both flat and hierarchical entity recognition tasks (see Figure 1).

<table><caption>Table 1: Distribution of entity across the different sets</caption><thead><tr><th></th><th>Train</th><th>Validation</th><th>Test</th></tr></thead><tbody><tr><td>NC-Spatial</td><td>3,252</td><td>358</td><td>355</td></tr><tr><td>NP-Spatial</td><td>4,707</td><td>464</td><td>519</td></tr><tr><td>ENE-Spatial</td><td>3,033</td><td>326</td><td>334</td></tr><tr><td>Relation</td><td>2,093</td><td>219</td><td>226</td></tr><tr><td>Latlong</td><td>553</td><td>66</td><td>72</td></tr><tr><td>NC-Person</td><td>1,378</td><td>132</td><td>133</td></tr><tr><td>NP-Person</td><td>1,599</td><td>170</td><td>150</td></tr><tr><td>ENE-Person</td><td>492</td><td>49</td><td>57</td></tr><tr><td>NP-Misc</td><td>948</td><td>108</td><td>96</td></tr><tr><td>ENE-Misc</td><td>255</td><td>31</td><td>22</td></tr><tr><td>Head</td><td>1,261</td><td>142</td><td>153</td></tr><tr><td>Domain-Mark</td><td>1,069</td><td>122</td><td>133</td></tr><tr><td>Paragraphs</td><td>1,800</td><td>200</td><td>200</td></tr><tr><td>Tokens</td><td>132,398</td><td>14,959</td><td>13,881</td></tr></tbody></table>

Table 1 shows the distribution of entity across the datasets. ‘NP’ prefix is used for named entities

(i.e., proper nouns), ‘NC’ prefix for nominal mentions (i.e., common nouns) and ‘ENE’ prefix for nested entities. The class distribution is highly imbalanced, with spatial entities being more frequent than person or miscellaneous entities. For example, ‘NP-Spatial’ occurs more than 4,700 times, while miscellaneous nested entities (i.e., ENE-Misc) have less than 300 occurrences.

## 3.2 Model Architectures and Task Formulation

In this study, we evaluate the performance of a diverse set of NER architectures, encompassing both traditional machine learning and modern deep learning approaches. We begin with CRFs, a classical probabilistic sequence labeling method well-suited for NER. We then explore Convolutional Neural Networks (CNNs), implemented via the spaCy library, which capture local context through sliding window filters. Bidirectional Long Short-Term Memory networks (Bi-LSTMs), trained using the Flair framework (Akbik et al., 2019), enable the modeling of sequential dependencies in both forward and backward directions, offering richer contextual representations. To leverage recent advances in language modeling, we also fine-tune Camembert (Martin et al., 2020), a French-adapted transformer architecture based on RoBERTa, which encodes deep bidirectional context through self-attention mechanisms. Finally, we assess the potential of generative LLMs, particularly OpenAI GPTs, for structured NER tasks in a few-shot learning setup.

The GeoEDdA annotation schema supports three levels of entity nesting: flat (level 0), shallow nested (level 1), and deep nested (level 2 and beyond). In this study, we limit our scope to the flat and shallow nested levels, which already present considerable challenges due to token overlap and label ambiguity.

An illustrative example is the phrase “ville d’Espagne,” where the tokens “ville” and “Espagne” are annotated respectively as ‘NC-Spatial’ and ‘NP-Spatial’, while the full phrase is labeled as

<footnote n="1">¹https://huggingface.co/datasets/GEODE/GeoEDdA</footnote>

<footnote n="2">²https://geode-project.github.io</footnote>
<table><caption>Table 2: Top 5 best and worst features of CRF Model (configuration 3)</caption><thead><tr><th></th><th>Domain mark</th><th>Head</th><th>Relation</th><th>NC-Person</th><th>NC-Spatial</th><th>NP-Misc</th><th>NP-Person</th><th>NP-Spatial</th></tr></thead><tbody><tr><td rowspan="5">Best Features</td><td>'Token:':</td><td>'prev_TOKEN:'</td><td>'lower:midi'</td><td>'lower:pape'</td><td>'lower:royaume'</td><td>'shape:dddd'</td><td>'prev_TOKEN:Hunauld'</td><td>'Token:italie'</td></tr><tr><td>'lower:.'</td><td>'prev_lower:'</td><td>'lower:roi'</td><td>'lower:roi'</td><td>'lower:fleuve'</td><td>'Token:persan'</td><td>'prev_lower:Hunauld'</td><td>'lower:italie'</td></tr><tr><td>'Token:terme'</td><td>'Token:'</td><td>'lower:dessous'</td><td>'Token:Mans'</td><td>'lower:comté'</td><td>'Token:persan'</td><td>'lower:juifs'</td><td>'prev_lower:palus'</td></tr><tr><td>'lower:terme'</td><td>'lower:'</td><td>'Token:Midi'</td><td>'lower:mans'</td><td>'Token:ile'</td><td>'prev_TOKEN:xiij'</td><td>'Token:bazanés'</td><td>'prev_TOKEN:palus'</td></tr><tr><td>'Token:Géograph'</td><td>'isupper'</td><td>'next_TOKEN:petits'</td><td>'lower:président'</td><td>'lower:ile'</td><td>'prev_lower:xiij'</td><td>'lower:bazanés'</td><td>'lower:indes'</td></tr><tr><td rowspan="5">Worst Features</td><td>'shape:Xxxxxxxxxx'</td><td>'prev_lower:'</td><td>'next_lower:se'</td><td>'shape:Xxxxxxxxxx'</td><td>'next Dep:obl:agent'</td><td>'prev_TOKEN:v'</td><td>'prev_shape:Xxxxxxxxxx'</td><td>'shape:xxx'</td></tr><tr><td>'dep:appos'</td><td>'prev_shape:X'</td><td>'next_TOKEN:sur'</td><td>'isstop'</td><td>'Token:du'</td><td>'next Shape:dddd'</td><td>'shape:xxxxxx'</td><td>'next Dep:flat:name'</td></tr><tr><td>'pos:DET'</td><td>'prev_pos:NOUN'</td><td>'next_lower:sur'</td><td>'pos:ADV'</td><td>'lower:du'</td><td>'pos:ADP'</td><td>'shape:xxxxx'</td><td>'shape:xxxxxxxxx'</td></tr><tr><td>'pos:PUNCT'</td><td>'prev_pos:PUNCT'</td><td>'shape:x'</td><td>'shape:Xxxxxx'</td><td>'next Dep:det'</td><td>'shape:dd'</td><td>'shape:xxxxxxxxxx'</td><td>'shape:xxxxxxxxx'</td></tr><tr><td>'isupper'</td><td>'prevShape:X'</td><td>'pos:PROPN'</td><td>'next Dep:xcomp'</td><td>'isstop'</td><td>'isstop'</td><td>'shape:xxxxxxxxx'</td><td>'shape:xxxxxxxxxxxx'</td></tr></tbody></table>

'ENE-Spatial' (see Figure 1. This results in overlapping annotations for the same span, requiring a multi-label classification strategy.

To address this, we simplify the task by employing a joint-label encoding strategy. Overlapping entities are merged into composite labels (e.g., “NP-Spatial+ENE-Spatial”), allowing the problem to be recast as a flat classification task. These joint labels are mapped to unique class identifiers, enabling compatibility with models that do not natively support nested span prediction.

# 4 Experiments and Results

## 4.1 CRF Model

The CRF model served as a baseline for traditional sequence labeling. We explored three feature configurations: (1) basic lexical features (token, lower- ercase, punctuation), (2) additional syntactic features (POS tags), and (3) full features including dependency parses. Grid search over regularization parameters (C1 and C2), learning rate and dropout was conducted to mitigate overfitting. The model performed well on frequently occurring entity types like ‘NP-Spatial’ and ‘Relation’ but struggled on ‘NC-Person’ and ‘NP-Misc’ due to feature spars- ity and lack of contextual depth. Results also shows that adding additional features such as part- of-speech in the feature configurations have only a very limited impact on the performances (see Table 3).

Moreover, adding dependency parsing as a feature has a negative impact. Table 2 shows the best and worst features for configuration 3 (i.e., including part-of-speech and dependency parsing). Dependency features such as dep:appos and pos:PUNCT were among the least informative, suggesting that shallow syntactic features may introduce noise rather than signal in historical text NER.

<table><caption>Table 3: CRF F1-scores on test set with (1) basic lexical features, (2) additional syntactic features, and (3) full features.</caption><thead><tr><th></th><th>(1)</th><th>(2)</th><th>(3)</th></tr></thead><tbody><tr><td>NC-Spatial</td><td>90.0</td><td>89.8</td><td>89.6</td></tr><tr><td>NP-Spatial</td><td>90.7</td><td>90.9</td><td>91.1</td></tr><tr><td>ENE-Spatial</td><td>89.6</td><td>89.9</td><td>89.0</td></tr><tr><td>Relation</td><td>93.1</td><td>92.7</td><td>91.3</td></tr><tr><td>Latlong</td><td>97.0</td><td>96.9</td><td>91.0</td></tr><tr><td>NC-Person</td><td>58.7</td><td>64.0</td><td>62.6</td></tr><tr><td>NP-Person</td><td>77.7</td><td>79.0</td><td>76.4</td></tr><tr><td>ENE-Person</td><td>70.3</td><td>73.3</td><td>72.6</td></tr><tr><td>NP-Misc</td><td>62.0</td><td>67.1</td><td>65.2</td></tr><tr><td>ENE-Misc</td><td>43.0</td><td>43.9</td><td>40.0</td></tr><tr><td>Head</td><td>86.8</td><td>88.1</td><td>85.3</td></tr><tr><td>Domain-mark</td><td>99.0</td><td>99.0</td><td>99.0</td></tr><tr><td>Micro Avg</td><td>88.1</td><td>88.6</td><td>87.7</td></tr><tr><td>Macro Avg</td><td>79.8</td><td>81.2</td><td>79.9</td></tr></tbody></table>

## 4.2 spaCy NER and Spancat fine-tuning

We compare two different approaches from the spaCy platform<fnref n="3" />: spaCy NER and spaCy Span- cat<fnref n="4" />. spaCy NER is a transition-based parser with a CNN-based model (token-based classification) while spaCy Spancat is a feed-forward neural net- work over span representations (span-based clas- sification). Table 4 shows the precision and recall scores of the two fine-tuned spaCy models for each class.

Despite rapid convergence during training, the CNN suffered from underperformance (below 60%) on rare and nested classes (see Table 4). The spaCy NER architecture was more robust in rec- ognizing contiguous (at least for precision), flat spans but lacked mechanisms for long-range de- pendencies and hierarchy. The overall scores also show that spaCy NER is above the CRF scores for every label (see Table 8). In average Spancat has a higher recall than spaCy NER, but spacy NER has a higher precision. Except for some classes such as nested entities (‘ENE-Spatial’ and ‘ENE- Person’) and also under-represented classes such as ‘NP-Misc’ (support from the test set is listed in

<footnote n="3">³https://spacy.io/</footnote>

<footnote n="4">⁴https://spacy.io/api/spancategorizer</footnote>
<table><caption>Table 4: Precision and recall scores evaluation for our fine-tuned spaCy models.</caption><thead><tr><th rowspan="2"></th><th colspan="2">spaCy NER</th><th colspan="2">Spancat</th></tr><tr><th>P</th><th>R</th><th>P</th><th>R</th></tr></thead><tbody><tr><td>NC-Spatial</td><td><strong>79.7</strong></td><td>34.5</td><td>40.9</td><td><strong>59.8</strong></td></tr><tr><td>NP-Spatial</td><td><strong>93.3</strong></td><td>50.3</td><td>92.0</td><td><strong>69.1</strong></td></tr><tr><td>ENE-Spatial</td><td><strong>59.1</strong></td><td><strong>36.1</strong></td><td><strong>86.7</strong></td><td>16.2</td></tr><tr><td>Relation</td><td><strong>62.5</strong></td><td><strong>92.7</strong></td><td>26.9</td><td>69.7</td></tr><tr><td>Latlong</td><td><strong>92.5</strong></td><td>40.8</td><td>92.3</td><td><strong>86.6</strong></td></tr><tr><td>NC-Person</td><td><strong>82.1</strong></td><td>14.2</td><td>36.4</td><td><strong>52.0</strong></td></tr><tr><td>NP-Person</td><td>74.8</td><td>40.9</td><td><strong>78.3</strong></td><td><strong>69.5</strong></td></tr><tr><td>ENE-Person</td><td>45.5</td><td><strong>2.5</strong></td><td><strong>66.7</strong></td><td>1.0</td></tr><tr><td>NP-Misc</td><td>18.8</td><td>10.9</td><td><strong>24.6</strong></td><td><strong>34.9</strong></td></tr><tr><td>ENE-Misc</td><td><strong>2.9</strong></td><td><strong>1.2</strong></td><td>0.00</td><td>0.00</td></tr><tr><td>Head</td><td><strong>92.3</strong></td><td>52.0</td><td>82.8</td><td><strong>60.6</strong></td></tr><tr><td>Domain-mark</td><td><strong>100</strong></td><td>41.1</td><td>92.6</td><td><strong>88.8</strong></td></tr><tr><td>Micro Avg</td><td><strong>77.5</strong></td><td>33.4</td><td>58.5</td><td><strong>57.4</strong></td></tr><tr><td>Macro Avg</td><td><strong>66.9</strong></td><td>28.6</td><td>60.0</td><td><strong>50.7</strong></td></tr></tbody></table>

<table><caption>Table 5: Precision and recall scores evaluation for our fine-tuned CamemBERT models.</caption><thead><tr><th rowspan="2"></th><th colspan="2">Joint-label</th><th colspan="2">Multi-label</th></tr><tr><th>P</th><th>R</th><th>P</th><th>R</th></tr></thead><tbody><tr><td>NC-Spatial</td><td>90.5</td><td>95.1</td><td><strong>91.6</strong></td><td><strong>95.3</strong></td></tr><tr><td>NP-Spatial</td><td>95.2</td><td><strong>95.7</strong></td><td><strong>95.6</strong></td><td>93.7</td></tr><tr><td>ENE-Spatial</td><td>94.0</td><td>89.7</td><td><strong>94.4</strong></td><td><strong>90.9</strong></td></tr><tr><td>Relation</td><td>87.5</td><td>93.1</td><td><strong>90.8</strong></td><td><strong>93.6</strong></td></tr><tr><td>Latlong</td><td><strong>98.1</strong></td><td>96.8</td><td>96.0</td><td><strong>98.0</strong></td></tr><tr><td>NC-Person</td><td><strong>67.2</strong></td><td>80.9</td><td>63.0</td><td><strong>83.1</strong></td></tr><tr><td>NP-Person</td><td><strong>87.9</strong></td><td>86.2</td><td>86.1</td><td><strong>88.7</strong></td></tr><tr><td>ENE-Person</td><td><strong>89.5</strong></td><td>77.4</td><td>88.7</td><td><strong>86.4</strong></td></tr><tr><td>NP-Misc</td><td>68.7</td><td><strong>76.6</strong></td><td><strong>69.6</strong></td><td>76.0</td></tr><tr><td>ENE-Misc</td><td>40.0</td><td>66.7</td><td><strong>40.7</strong></td><td><strong>72.8</strong></td></tr><tr><td>Head</td><td><strong>97.3</strong></td><td><strong>98.0</strong></td><td>97.2</td><td>96.5</td></tr><tr><td>Domain-mark</td><td><strong>99.7</strong></td><td><strong>99.0</strong></td><td>99.2</td><td><strong>99.0</strong></td></tr><tr><td>Micro Avg</td><td><strong>90.1</strong></td><td>92.0</td><td>89.8</td><td><strong>92.7</strong></td></tr><tr><td>Macro Avg</td><td><strong>84.6</strong></td><td>87.9</td><td>84.4</td><td><strong>89.5</strong></td></tr></tbody></table>

Table 1).

Our fine-tuned spaCy spancat model (Moncla et al., 2024) is available on the HuggingFace hub<fnref n="5" />.

## 4.3 CamemBERT fine-tuning

To evaluate the capabilities of transformer-based models on historical NER, we fine-tuned CamemBERT, a pretrained RoBERTa-like model for French (Martin et al., 2020), using the GeoEDDA corpus. We experimented with two output strategies for handling the overlapping and nested nature of annotations: joint-label classification and multi-label classification (span-based annotations).

As described in Section 3, in the joint-label classification setup, each token is assigned a single composite label representing the combination of all entity types to which it belongs. This transforms the multi-label problem into a single-label task with an expanded label set. The multi-label classification strategy, on the other hand, models each entity type independently using a sigmoid activation for each class, allowing tokens to be tagged with multiple labels simultaneously.

As shown in Table 5, both strategies achieved closely matched results, with complementary strengths. The joint-label model showed slightly better precision, producing more accurate predictions with fewer false positives. This suggests that the joint-labeling approach may help CamemBERT better model interdependencies between overlapping entity types and avoid overprediction. Conversely, the multi-label model yielded slightly higher recall, capturing a broader range of correct

entities, including some low-frequency or ambiguous cases that the joint-label model missed.

These tendencies were particularly evident in complex constructions involving nested or overlapping entities, such as *ville d'Espagne*, where the multi-label approach sometimes detected more entity spans but occasionally introduced inconsistencies between related labels.

Despite their differences, both strategies demonstrated strong overall performance. Our fine-tuned CamemBERT models are available on the Hugging-Face Hub<fnref n="6" />.

## 4.4 Flair fine-tuning

In addition to transformer-based approaches, we fine-tuned the Flair sequence labeling model (Akbik et al., 2019), which is based on a BiLSTM-CRF architecture with contextual string embeddings. Flair has demonstrated strong performance on standard NER benchmarks and offers a lightweight alternative to transformer-based models, making it a relevant baseline for resource-constrained settings or historical text processing.

For our experiments, we used pretrained French Flair embeddings trained on OSCAR and Common Crawl corpora. Overall, Flair achieved competitive performance, especially considering its smaller size and lighter computational requirements compared to transformer models. However, its results remained slightly behind those of CamemBERT in terms of F1 score, for complex or overlapping entities (e.g., 'ENE-Spatial', 'ENE-Person', 'ENE-Misc'). This is likely due to Flair's limited capacity to model long-range dependencies and inter-label

<footnote n="5">⁵https://huggingface.co/GEODE/fr_spacy_custom_spancat_edda</footnote>

<footnote n="6">⁶https://huggingface.co/GEODE/</footnote>
<table><caption>Table 6: Precision, recall and F1-scores evaluation for our fine-tuned Flair model.</caption><thead><tr><th></th><th>P</th><th>R</th><th>F</th></tr></thead><tbody><tr><td>NC-Spatial</td><td>90.7</td><td>95.8</td><td>93.2</td></tr><tr><td>NP-Spatial</td><td>97.1</td><td>92.9</td><td>94.9</td></tr><tr><td>ENE-Spatial</td><td>91.4</td><td>93.0</td><td>92.2</td></tr><tr><td>Relation</td><td>91.0</td><td>93.6</td><td>92.3</td></tr><tr><td>Latlong</td><td>98.4</td><td>98.6</td><td>98.5</td></tr><tr><td>NC-Person</td><td>73.2</td><td>81.3</td><td>77.1</td></tr><tr><td>NP-Person</td><td>88.3</td><td>89.2</td><td>88.7</td></tr><tr><td>ENE-Person</td><td>87.5</td><td>73.9</td><td>80.1</td></tr><tr><td>NP-Misc</td><td>74.5</td><td>80.0</td><td>77.1</td></tr><tr><td>ENE-Misc</td><td>48.7</td><td>45.7</td><td>47.1</td></tr><tr><td>Head</td><td>96.4</td><td>94.1</td><td>95.2</td></tr><tr><td>Domain-mark</td><td>98.2</td><td>98.5</td><td>98.3</td></tr><tr><td>Micro Avg</td><td>91.5</td><td>92.1</td><td>91.8</td></tr><tr><td>Macro Avg</td><td>86.3</td><td>86.4</td><td>86.2</td></tr></tbody></table>

interactions compared to transformer-based architectures.

## 4.5 Few-shot prompting with generative LLMs

We evaluated the potential of few-shot prompting with state-of-the-art generative language models for the entity recognition task. In contrast to encoder-based fine-tuning approaches, this setup does not involve any training, relying solely on in-context learning. We focused our experiments on OpenAI's models: GPT3.5, GPT-4o, GPT-4.1-mini, GPT-4.1-nano and o4-mini, using prompt templates with only 1 labeled example and a consistent JSON structured output format (Zeghidi and Moncla, 2024).

Figure 2 shows the f1-scores comparison for each GPT models and each label. Table 7 also reports precision and recall scores for the three best models: GPT-4o, GPT-4.1-mini and o4-mini. These models demonstrate surprisingly strong performance, particularly GPT-o4-mini, which achieves the highest recall and competitive precision across most categories. In contrast, GPT-4o attains the best precision on several tags, including 'NC-Spatial', 'NP-Spatial', and 'Relation', but at the cost of significantly lower recall. GPT-4.1-mini shows more balanced scores but trails slightly behind GPT-o4-mini in both micro and macro averages.

Performance varies considerably across entity types. For well-represented and structurally clearer categories such as 'Latlong', 'NP-Spatial', 'NP-Person', and 'Domain-mark', models perform very well, even surpassing our fine-tuned spaCy NER for some labels such as 'LatLong' and 'Relation'. However, for more ambiguous or sparse classes

<table><caption>Table 7: Precision and recall scores evaluation for OpenAI's GPT models.</caption><thead><tr><th rowspan="2"></th><th colspan="2">4o</th><th colspan="2">4.1-mini</th><th colspan="2">o4-mini</th></tr><tr><th>P</th><th>R</th><th>P</th><th>R</th><th>P</th><th>R</th></tr></thead><tbody><tr><td>NC-Spatial</td><td><strong>79.7</strong></td><td>34.5</td><td>40.9</td><td><strong>59.8</strong></td><td>68.0</td><td>58.3</td></tr><tr><td>NP-Spatial</td><td><strong>93.3</strong></td><td>50.3</td><td>92.0</td><td>69.1</td><td>92.1</td><td><strong>79.1</strong></td></tr><tr><td>ENE-Spatial</td><td>59.1</td><td>36.1</td><td><strong>86.7</strong></td><td>16.2</td><td>81.3</td><td><strong>61.2</strong></td></tr><tr><td>Relation</td><td><strong>62.5</strong></td><td><strong>92.7</strong></td><td>26.9</td><td>69.7</td><td>46.8</td><td>76.3</td></tr><tr><td>Latlong</td><td><strong>92.5</strong></td><td>40.8</td><td>92.3</td><td><strong>86.6</strong></td><td><strong>92.5</strong></td><td>85.8</td></tr><tr><td>NC-Person</td><td><strong>82.1</strong></td><td>14.2</td><td>36.4</td><td>52.0</td><td>56.2</td><td><strong>60.4</strong></td></tr><tr><td>NP-Person</td><td>74.8</td><td>40.9</td><td>78.3</td><td>69.5</td><td><strong>82.7</strong></td><td><strong>75.4</strong></td></tr><tr><td>ENE-Person</td><td>45.5</td><td>2.5</td><td>66.7</td><td>1.0</td><td><strong>83.3</strong></td><td><strong>57.8</strong></td></tr><tr><td>NP-Misc</td><td>18.8</td><td>10.9</td><td>24.6</td><td>34.9</td><td><strong>39.7</strong></td><td><strong>42.9</strong></td></tr><tr><td>ENE-Misc</td><td>2.9</td><td>1.2</td><td>0.00</td><td>0.00</td><td><strong>15.9</strong></td><td><strong>8.6</strong></td></tr><tr><td>Head</td><td><strong>92.3</strong></td><td>52.0</td><td>82.8</td><td>60.6</td><td>91.2</td><td><strong>69.3</strong></td></tr><tr><td>Domain-mark</td><td><strong>100</strong></td><td>41.1</td><td>92.6</td><td>88.8</td><td>96.5</td><td><strong>90.6</strong></td></tr><tr><td>Micro Avg</td><td><strong>77.5</strong></td><td>33.4</td><td>58.5</td><td>57.4</td><td>75.6</td><td><strong>70.5</strong></td></tr><tr><td>Macro Avg</td><td>66.9</td><td>28.6</td><td>60.0</td><td>50.7</td><td><strong>70.5</strong></td><td><strong>63.8</strong></td></tr></tbody></table>

like 'ENE-Person', 'ENE-Misc', and 'NP-Misc', results degrade substantially, indicating challenges in generalizing from few-shot context to rarer or noisier labels. Still, it is interesting to note that recall for 'ENE-Person' improves dramatically from 2.5 with GPT-4o and 1.0 with GPT-4.1-mini to 57.8 with GPT-o4-mini, highlighting the rapid progress in newer model variants.

Notably, open-weights LLMs such as Llama 3-70b and local LLMs such as Mistral 7B were also evaluated but yielded consistently very poor performance and are therefore excluded from this study. Their limitations likely stem from insufficient instruction tuning and difficulty in following structured token-level labeling prompts. Code and detailed results are available on Github<fnref n="7" />.

## 5 Discussion

The results of our experiments are summarized in Table 8 (only F1-scores are reported). Our evaluation of multiple NER architectures on the GeoEDdA corpus offers several insights into the strengths, limitations, and trade-offs inherent in applying modern NLP techniques to historical French texts. First, the performance gap between token-level and span-level models underscores the importance of selecting an appropriate task formulation for the complexity of the data. Flat models, while simpler to train and evaluate, often fail to capture overlapping and nested entity structures that are commonplace in early modern writing. Span-based models such as spaCy Spancat and multi-label transformer variants better handle such complexities but at the cost of higher computational

<footnote n="7">⁷https://github.com/GEODE-project/ner-lm</footnote>
<figure><img src="image_4.png" /><figcaption>Figure 2: F1-scores comparison for OpenAI's GPT models.</figcaption></figure>

<table><caption>Table 8: Token-based F1-scores</caption><thead><tr><th>Tag</th><th>CRF</th><th>Fine-tuned spaCy NER</th><th>Fine-tuned spaCy Spancat</th><th>Fine-tuned CamemBERT</th><th>Fine-tuned Flair</th><th>GPT 4o-mini</th></tr></thead><tbody><tr><td>NC-Spatial</td><td>89.8</td><td>86.2</td><td><strong>95.0</strong></td><td>93.4</td><td>93.2</td><td>62.8</td></tr><tr><td>NP-Spatial</td><td>90.9</td><td>83.0</td><td>93.5</td><td>94.7</td><td><strong>94.9</strong></td><td>85.1</td></tr><tr><td>ENE-Spatial</td><td>89.9</td><td>88.3</td><td><strong>93.4</strong></td><td>92.6</td><td>92.2</td><td>69.8</td></tr><tr><td>Relation</td><td>92.7</td><td>84.8</td><td>50.6</td><td>92.2</td><td><strong>92.3</strong></td><td>58.0</td></tr><tr><td>Latlong</td><td>96.9</td><td>95.9</td><td>0.00</td><td>97.0</td><td><strong>98.5</strong></td><td>89.0</td></tr><tr><td>NC-Person</td><td>64.0</td><td>61.9</td><td><strong>77.5</strong></td><td>71.6</td><td>77.1</td><td>58.2</td></tr><tr><td>NP-Person</td><td>79.0</td><td>67.3</td><td><strong>92.0</strong></td><td>87.4</td><td>88.7</td><td>58.2</td></tr><tr><td>ENE-Person</td><td>73.3</td><td>65.5</td><td>87.1</td><td><strong>87.5</strong></td><td>80.1</td><td>68.2</td></tr><tr><td>NP-Misc</td><td>67.1</td><td>61.2</td><td>69.8</td><td>72.7</td><td><strong>77.1</strong></td><td>41.2</td></tr><tr><td>ENE-Misc</td><td>43.9</td><td>32.5</td><td>0.00</td><td><strong>52.2</strong></td><td>47.1</td><td>11.2</td></tr><tr><td>Head</td><td>88.1</td><td>85.6</td><td>45.1</td><td><strong>96.8</strong></td><td>95.2</td><td>78.7</td></tr><tr><td>Domain-mark</td><td>99.0</td><td>96.2</td><td>95.8</td><td><strong>99.1</strong></td><td>98.3</td><td>93.4</td></tr><tr><td>Micro avg</td><td>88.6</td><td>83.2</td><td>76.8</td><td>91.2</td><td><strong>91.8</strong></td><td>73.0</td></tr><tr><td>Macro avg</td><td>81.2</td><td>75.7</td><td>66.6</td><td><strong>86.4</strong></td><td>86.2</td><td>66.2</td></tr></tbody></table>

overhead and sensitivity to class imbalance.

Transformer-based models like CamemBERT consistently outperformed traditional approaches across most entity types, particularly in the presence of overlapping spans. The joint-label strategy, despite simplifying the task, performed well for high-frequency entities but introduced combinatorial label sparsity for less common categories. Conversely, the multi-label formulation showed stronger recall for underrepresented and ambiguous classes, highlighting its ability to model overlapping annotations more flexibly.

The results from few-shot prompting with generative LLMs (e.g., GPT-4o and GPT-o4-mini) are particularly noteworthy. Despite receiving minimal contextual supervision, these models produced competitive results for frequent entity types and even demonstrated surprising generalization capabilities on some nested entities. However, performance remains highly variable across entity

categories, with notable weaknesses for rare and context-dependent labels such as 'ENE-Misc' and 'NP-Misc'. These findings point to both the potential and the current limitations of generative models for structured prediction tasks, especially when fine control over outputs and class consistency is required.

Our experiments also show that model performance is tightly linked to entity frequency and annotation structure. Entities such as 'Latlong' and 'Domain-mark', which have clearer boundaries and more consistent patterns, are reliably extracted even by lightweight models. In contrast, entities with broader semantic scope or ambiguous context (such as 'NC-Person' or 'ENE-Misc') remain challenging across all architectures. This suggests a need for more targeted modeling strategies, possibly involving entity-type-specific encoders or adaptive training techniques.

Finally, model interpretability and integration
Louis Martin, Benjamin Muller, Pedro Javier Ortiz Suárez, Yoann Dupont, Laurent Romary, Éric Villemonte de La Clergerie, Djamé Seddah, and Benoît Sagot. 2020. CamemBERT: a tasty French language model. In *Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics*, pages 7203–7219.

Katherine McDonough, Ludovic Moncla, and Matje Van de Camp. 2019. Named entity recognition goes to old regime: geographic text analysis for early modern french corpora. *International Journal of Geographical Information Science*, 33(12):2498–2522.

Ludovic Moncla and Mauro Gaio. 2023. Perdido: Python library for geoparsing and geocoding french texts. In *First International Workshop on Geographic Information Extraction from Texts (GeoExT)*.

Ludovic Moncla, Denis Vigier, and Katherine McDonough. 2024. GeoEDdA: A Gold Standard Dataset for Geo-semantic Annotation of Diderot & d'Alembert's Encyclopédie. In *Second International Workshop on Geographic Information Extraction from Texts (GeoExT)*, Glasgow, Scotland.

Abdelkrim Tafer and Mauro Gaio. 2025. Extracting spatial entities involved in the description of a movement action using deep learning methods: A comparative study of three models. In *Third International Workshop on Geographic Information Extraction from Texts (GeoExT)*.

Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Łukasz Kaiser, and Illia Polosukhin. 2017. Attention is all you need. *Advances in neural information processing systems*, 30.

Tingyu Xie, Qi Li, Jian Zhang, Yan Zhang, Zuozhu Liu, and Hongwei Wang. 2023. Empirical study of zero-shot ner with chatgpt. In *Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing*, pages 7935–7956, Singapore.

Urchade Zaratiana, Nadi Tomeh, Pierre Holat, and Thierry Charnois. 2024. GLiNER: Generalist model for named entity recognition using bidirectional transformer. In *Proceedings of the 2024 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies (Volume 1: Long Papers)*, pages 5364–5376, Mexico City, Mexico. Association for Computational Linguistics.

Hédi Zeghidi and Ludovic Moncla. 2024. Evaluating named entity recognition using few-shot prompting with large language models. *arXiv preprint arXiv:2408.15796*.