# Corpus and Models for Lemmatisation and POS-tagging of Old French

Jean-Baptiste Camps, Thibault Clérice, Naomi Kanaoka, Ariane Pinche, Frédéric Duval, Lucence Ing

<figure><img src="image_1.png" /></figure>

To cite this version:

Jean-Baptiste Camps, Thibault Clérice, Naomi Kanaoka, Ariane Pinche, Frédéric Duval, et al. Corpus and Models for Lemmatisation and POS-tagging of Old French. 2021. halshs-03353125

HAL Id: halshs-03353125

https://shs.hal.science/halshs-03353125

Preprint submitted on 23 Sep 2021

HAL is a multi-disciplinary open access
archive for the deposit and dissemination of sci-
entific research documents, whether they are pub-
lished or not. The documents may come from
teaching and research institutions in France or
abroad, or from public or private research centers.

L'archive ouverte pluridisciplinaire HAL, est
destinée au dépôt et à la diffusion de documents
scientifiques de niveau recherche, publiés ou non,
émanant des établissements d'enseignement et de
recherche français ou étrangers, des laboratoires
publics ou privés.

<figure><img src="image_2.png" /></figure>

Distributed under a Creative Commons Attribution 4.0 International License
# Corpus and Models for Lemmatisation and POS-tagging
of Old French

Jean-Baptiste Camps<fnref n="1" />, Thibault Clérice<fnref n="1" />, Frédéric Duval<fnref n="1" />, Lucence Ing<fnref n="1" />, Naomi Kanaoka<fnref n="1" />, Ariane Pinche<fnref n="1" />

¹Centre Jean-Mabillon, École nationale des chartes, Université Paris, Sciences & Lettres

Corresponding author: Jean-Baptiste.Camps@chartes.psl.eu

## Abstract

Old French is a typical example of an under-resourced historic languages, that furtherly displays an important amount of linguistic variation. In this paper, we present the current results of a long going project (2015...) and describe how we broached the difficult question of providing lemmatisation and POS models for Old French with the help of neural taggers and the progressive constitution of dedicated corpora.

## Keywords

Lemmatisation; POS tagging; Old French; Historic Languages.

## I INTRODUCTION

Today, linguistically annotated corpora<fnref n="1" /> are at the centre of crucial research issues, as they constitute long-term and reusable data, ensure the reproductibility of a study, and allow the interrogation and exploitation of large-scale textual sets [Mellet and Purnelle, 2002]. Since the 1960s, the annotation of corpora has been automated by linguists, particularly in the field of NLP (Natural Language Processing), for the quantitative exploitation of linguistic phenomena. The identification of the occurrences of a word is thus facilitated by annotation, which allows a more efficient harvesting. These corpora also constitute essential data for dialectometry or stylometry, in particular for author attribution studies [Mellet, 2002] or for the automatic classification of literary genre [Feldman et al., 2009]. Finally, lemmatisation can be integrated into a semi-automatic processing pipeline starting from the digitisation of a manuscript and leading to the production of a critical edition by comparing the texts of different witnesses [Camps et al., 2019]. The range of possibilities makes this data extremely valuable.

Lemmatisation as an academic task was first developed in the study of flectional ancient languages, such as Latin or Greek, because this computerisation of the task already corresponded to the ancient use of concordance tables or indexes [Rouse and Rouse, 1984]. Roberto Busa and his team working on the lemmatisation of the *Summa Theologica* of Thomas Aquinas starting in the 1950s [Busa, 1980] were pioneers in this area. Ten years later, the LASLA<fnref n="2" /> set up a system of lemmatisation of its corpora by collecting all information manually and recording this information in computer files. In the 1990's, the emergence of lemmatisers such as *TreeTagger* [Schmid, 1994] made it possible to automatically annotate corpora. These tools were at the time

<footnote n="1">¹The process of lemmatisation associates a canonical form with each term in a text, regardless of its inflection.</footnote>

<footnote n="2">²Laboratoire d'Analyse Statistique des Langues Anciennes, University of Liège.</footnote>
based on rules and a dictionary to predict the linguistic annotation of a word (token) in a given language.

In the 1980s, linguistic annotation was extended to Old French with the desire to systematically study the language of a document and its dialectal particularities with the constitution of two manually annotated corpora with a set of 225 numeric tags encoding part of speech and other morphological categorie, the first one of charters and the second one of literary texts, *Amsterdam Corpus*, by Anthonij Dees (Vrije Universiteit Amsterdam) and his collaborators, including Piet van Reenen, in order to produce the two atlas of the linguistic forms of Old French [Dees et al., 1980, 1987]. While the Charter corpus has been lost, in the 2000s, the literary corpus of the second atlas was taken over and fully lemmatised and annotated (addition of POS), with the help of the lemmatiser *TreeTagger* for Old French and using lemmas proposed in the *Tobler-Lommatzsch dictionary* [Tobler and Lommatzsch, 2002] by Pierre Kunstmann and Achim Stein to form the *Nouveau corpus d'Amsterdam* (NCA, Kunstmann and Stein [2007]). But, vernacular languages such as Old French bring other challenges related to spelling variations, making it difficult to use a fixed dictionary for lemmas. Indeed, the same word can have a large number of graphic realisations. For example, the term *cheval* can have more than thirty different forms (see Table 1, Camps et al. [2019]), which makes it pretty much inefficient to use a lemmatisation tool based on a pre-established dictionary.

<table><caption>Table 1 – spellings for “cheval” in the *Nouveau corpus d'Amsterdam*</caption><thead><tr><th>Form</th><th>Freq.</th><th>Form</th><th>Freq.</th><th>Form</th><th>Freq.</th><th>Form</th><th>Freq.</th></tr></thead><tbody><tr><td>cheval</td><td>785</td><td>chevaux</td><td>30</td><td>ceuax</td><td>10</td><td>cheuas</td><td>2</td></tr><tr><td>cheual</td><td>375</td><td>chivaus</td><td>27</td><td>cevax</td><td>10</td><td>keval</td><td>2</td></tr><tr><td>chevaus</td><td>248</td><td>cheuax</td><td>23</td><td>ceuaus</td><td>9</td><td>chaval</td><td>1</td></tr><tr><td>ceval</td><td>98</td><td>chiual</td><td>23</td><td>chiuau</td><td>9</td><td>chavaux</td><td>1</td></tr><tr><td>chevax</td><td>92</td><td>cevaus</td><td>19</td><td>cheuaux</td><td>8</td><td>cheua</td><td>1</td></tr><tr><td>chevals</td><td>84</td><td>chevas</td><td>19</td><td>kevaus</td><td>6</td><td>cheualx</td><td>1</td></tr><tr><td>ceual</td><td>66</td><td>cheuals</td><td>14</td><td>chevau</td><td>5</td><td>cheuau</td><td>1</td></tr><tr><td>cheuaus</td><td>65</td><td>cevals</td><td>12</td><td>cevaux</td><td>3</td><td>chevalx</td><td>1</td></tr><tr><td>chival</td><td>34</td><td>chiuaus</td><td>11</td><td>chivals</td><td>3</td><td>chiuals</td><td>1</td></tr></tbody></table>

Traditional lemmatisers such as *Treetagger* work with rule sets, a lexicon that only recognises lemmas that already exist in its training corpus, and a decision-tree based algorithm. Due to their fixed lexicon which mainly takes into account only flectional variations and the token environment, the results are not as optimal as those of classical Latin or Modern French due to the unstandardised nature of Old French. However, the most recent developments in the fields of lemmatisation and linguistic annotation of vernacular languages and their variety through the use of recurrent neural networks (RNN) make it possible to set up systems capable of handling the linguistic variations in historical languages<fnref n="3" /> thanks to their learning and prediction capacity. Thanks to RNNs, the lemmatiser no longer needs to compare the word in the text to a list of lemmas contained in a dictionary, but is able to predict its sequence by sequence (seq2seq), character by character considering word and sentence levels. This is why we are focusing on the use of these technologies and the implementation of a lemmatisation model for Old French using the lemmatiser and POS tagger *Pie* developed by E. Manjavacas [Manjavacas et al., 2019]),

<footnote n="3">³eg. Middle Dutch [Kestemont et al., 2016], Medieval Latin [Kestemont and De Gussem, 2017], Medieval Occitan [Camps and Couffignal, 2017], Early Irish [Dereza, 2019], Middle High German and a variety of other languages [Manjavacas et al., 2019, Schmid, 2019])</footnote>
which we have trained from manually annotated corpora that have been aggregated over the course of the projects.

The first experiments on lemmatisation for Old French were launched in 2015 by Jean-Baptiste Camps and Mike Kestemont, on the texts of the *Chanson d'Otinel* [Camps, 2016], using the newly developed Pandora tagger/lemmatiser that used convolutional and recurrent neural networks [Kestemont et al., 2016]. It soon benefited from insertion in the broader LAKME project (Thierry Poibeau, Daniel Stoekl, J.B. Camps et al.). Initially, the first samples of the *Chanson d'Otinel* have been annotated using the TreeTagger parameters for Old French provided by Achim Stein [Stein, s.d.], then manually corrected, and used to train the first Pandora models; then, phases of data prediction, post-correction and model training were iteratively performed in order to build the corpus. A corrected corpus of 50,000 words annotated with lemmas, morphosyntax and flexion was created for Old French and another equivalent corpus, annotated with lemmas, for Occitan (the latter in collaboration with the CORLIG project of Paris-Sorbonne) [Camps and Couffignal, 2017]. Meanwhile, the lemmatiser Pandora was developed.

Thanks to financial support from PSL (IRIS Scripta) and above all from the DIM STCN (Ile-de-France Region), the project continued after the end of the LAKME project, taking the name OMÉLIE (*Outils et méthodes pour l'édition linguistique enrichie*) from 2018. It then scaled up, carrying out two tasks simultaneously:

1. the development of a lemmatisation post-correction tool, allowing fast correction and batch processing of lemmatiser output;

2. a massive growth of the lemmatised corpus, in order to increase lemmatiser scores.

From a computational point of view, the new annotated corpora allow, by a circular process, the training of more efficient models for lemmatisation and annotation, and thus a subsequent faster growth of the corpus. Both the post-correction application and the lemmatiser are language-independent, and particularly suitable for non-standardised language states. The application has been continuously developed to allow collaborative work and, above all, to be used without advanced computer experience. By 2018, the post-correction application Pyrrha [Clérice and Pilla, 2021] was being developed and used, including for educational purposes. This was a major step forward, since this open-source service significantly improved the lemmatisation speed.

The increasing of the training corpus has made continuous progress thanks to important funding. Its cost is high and can be estimated at between 50 000 and 100 000€, not counting the development of the tools. This means that it is necessary to avoid duplicating data as much as possible and to allow interoperability of the data produced by the various lemmatisation projects in Old French or to promote data sharing or exchange. To this end, a workshop was held at the École des Chartes on 28 October 2018, dedicated to lexical lemma repositories in Old French, in order to standardise practices or at least make them interoperable. It is now necessary to develop equivalence tables between lemmas and to implement them in order to share corpora.

Methods implemented in LAKME and then OMÉLIE have contributed to the setting up of corpora and models for other languages, in particular for Old Occitan, but also for Classical French [Cafiero and Camps, 2019, Camps et al., 2020] (see also infra for the Franco-Italian). After a complete reworking and harmonisation of the corpora, a new version of the Old French models is available since the beginning of 2021 via Pyrrha [Clérice and Pilla, 2021] as well as in the form of an API via Deucalion and Pie-extended [Clérice, 2021]<fnref n="4" />.

<footnote n="4">⁴Available at: https://dh.chartes.psl.eu/ and https://tal.chartes.psl.eu/deucalion/.</footnote>
## II CORPORA

<table><caption>Table 2 – Description of the different corpora produced by the project and used in the experiments.</caption><thead><tr><th>Dataset</th><th>Source</th><th>Annotators</th><th>Morph</th><th>N. tokens</th><th>genre</th></tr></thead><tbody><tr><td>Chrestien Code</td><td>Kunstmann 2009<br>Duval and Pастore, in progress</td><td>PK, LI<br>FC, FD, LI, NK</td><td>partial</td><td>252774<br>160007</td><td>romance<br>law</td></tr><tr><td>DocLing Geste</td><td>Glessgen 2016<br>Camps 2016 (et varia)</td><td>NBP, NK<br>ACC, JBC, LI, NK</td><td>full<br>full</td><td>68317<br>195303</td><td>charters<br>epic</td></tr><tr><td>Lancelot WauchierSConf Varia</td><td>Ing (in progress)<br>Pinche 2021</td><td>LI<br>AP</td><td>-<br>full<br>full</td><td>286095<br>113694<br>56659</td><td>romance<br>hagiography<br>mixed</td></tr></tbody><tfoot><tr><td colspan="5">TOTAL</td><td>1 132 849</td></tr></tfoot></table>

### 2.1 Sources and history of the corpus

The annotated texts originated, in a first time, from individual editorial (PhD) projects. The first texts to be tagged were the three manuscripts of the *Chanson d'Otinel* [Camps, 2016], followed by other Old French Epics, now part of the *Geste* database [Camps, 2019]. The *Wauchier* [Pinche, 2021] and *Lancelot* [Ing, 2021] corpora also originate in PhD projects and are constituted of new data. The corpus of juridical texts results from an ongoing research project, concerning a set of vernacular translations of the *Corpus juris civilis*, written between c. 1225-1275. It was also created in the context of the production of complete or partial digital editions of certain translations, in order to characterise and compare translation choices.

Apart from fully new data, preexisting datasets have also been aligned with our reference lists: it is the case of the *Chrestien* corpus, originally produced under the supervision of Kunstmann [2009].

Progressively, new texts have been selected and annotated with the goal of expanding the coverage in terms of chronology, regional scriptae and genres. It resulted in the inclusion of a charters corpus, selected from the DocLing with a sampling by scripta/regional variants [Gleßgen, 2016]. New texts keep being added progressively with this focus, with currently a corpus of lyrical poetry constituted by digitising and annotating preexisting editions [Thibaud IV, 1925, Conon de Béthune, 1925, Brulé and Dyggve Petersen, 1951, Doss-Quinby et al., 2001] and a late allegoric verse text by Guillaume de Digulleville.

### 2.2 Annotation practice and workflow

The annotation workflow uses a set of tools developed as part of an initiative to establish a fully integrated environment for linguistic annotation and post-correction of historical languages.

Our annotation is characterised by its relative complexity, including the analysis of nominal and verbal flexion: pronouns are distinguished (personal, demonstrative, indefinite, interrogative, relative, adverbial, impersonal, cardinal, ordinal) and times, mode and person are specified for verbs. Once annotated with extent models, the texts and their annotation undergo a post-correction phase inside Pyrrha [Clérice and Pilla, 2021].

Tagging post-correction can be done either linearly (from start to finish), or massively, using concordance tables (accessible from the “Search tokens” link). Certain categories of words are
conducive to massive correction by the relative ease of identification of lemmas (NOMcom, NOMpro, Adverbe, Verbs); on the other hand, POS and morpho-syntaxic flexions always require careful analysis of syntax. For now, inside the application lemma+POS+flexion are not handled as a coordinated unit, and consistency has to be verified by the human.

### 2.2.1 Lemma

The texts were annotated in lemmas, according to the entries in Tobler-Lommatzsch's dictionary (henceforth, TL), with some adaptations. The choice of using TL instead of, for instance, the lemmas of the *Dictionnaire de Moyen Français* [ATILF, 2015, henceforth DMF] was done bearing in mind the linguistic nature of the corpus to annotate (Old French, not Middle French) and was in line with existing tools at the time [e.g. Stein].

The reviewer verifies and manually corrects pre-annotated forms on the Pyrrha platform (and can also identify and correct at the same time the possible wrong base forms generated by the OCR/HTR of the text). For the disambiguation of lemmas (e.g., *ver1*, masc. noun, ‘spring’ and *ver2*, masc. noun, ‘male pig’), it is still necessary to consult the digitised TL [Tobler and Lommatzsch, 2002] to clearly distinguish the homonyms with their definitions, because the lemmas of the TL are not intuitive and the application does not yet link to definitions. Indeed, there are homonyms even for terms that we do not think about.

For instance, in the sentence

*El mont n' a home de si grant hardement*

*mont* isn't 'mountain', but *monde1* (<MUNDUS), 'world'. Such examples are very numerous, due to the phonetic evolution of French that creates many homographs, e.g., *errer*, *errer1* < ERRARE, 'to make a mistake' or *errer2* < ITERARE, 'to go, to move'; *mes*, *mes1* < MA(N)SUM, 'house, garden', *mes2* < MESSIS, 'harvest, reaping', *mes3* < MISSUS, 'messenger', *mes4* < MISSUS, 'delicacies'.

The number of homograph forms is yet increased by morphological and graphetic variation, that can cause alternative spellings to collide between lemmas: e.g. *mes* can be *mes1*, *mes2*, *mes3* or *mes4*, but it can also be a form of *mais1* < MAGIS, 'more', or of *mais2*, 'bad', *mois*, 'month', *manoir*, vb. 'to stay', *metre2*, vb. 'to put, to set', *mon*, poss. 'my',...

In general, we validate or correct the lemmas proposed by the lemmatiser, but the application allows also to easily locate "unallowed lemmas" that do not exist in the reference list. When suitable lemmas do not exist in the TL, we create and add new lemmas. These are mostly cases of:

* Proper names (first name, surname, toponym).

* Words not listed in the TL's dictionary: e.g. *departement* 'departure'; *enterinement2* 'completely'.

* Lemmas listed in the TL, but reported at the end of the article: ex. adverbs (*principaument*, *mëismement*, *ancïenement*, *covenablement*); participles as nouns (*sëu*: NOMcom, TL, s. v. *savoir*), etc.

### 2.2.2 Tagging principles

POS Tagging is done following the rules given in the retained annotation scheme, Cattex2009 [Guillot et al., 2013a]. Yet, some new rules had to be established and some adaptations were
made, the latter especially since we are the first to use – to our knowledge – the full annotation scheme, including morphological tags (Cattex2009-full).

**Contraction.** In the case of contractions, we have introduced dual labels; e.g., “aux” or “auquel” will be treated under “a3+le” “a3+lequel”, eg., for *al*, *au*, *as* (‘at the’)

**Agglutination/deglutination.** The division of the lexical units often differs according to the date of the text and the choice of the editor, which does not necessarily match the spacing of the manuscript; one can suppose that the script of ancient manuscripts of Old French texts is roughly more analytic than those of Middle French manuscripts. Cases include, *lors que/lorsque*; *toutes voies/toutesvoies*; *par mi/parmi*; *a fin/affin*; *ja mais/jamais*, etc. Solutions should be sought that do not unduly complicate the processing of the texts. In case of agglutination or deglutination, we choose a solution adapted to the form of the text. For instance, in “Le dit jour”, if “le dit” is noted in 2 words, it will be treated as *DETdef+VERppe*, whereas if it is noted in one word, *ledit* will be tagged as a compound determinant (*DETcom*). However, in accordance with the lemmatisation of Tobler and Lommatzsch [2002], complex adverbial formations such as *parfois*, *portant*, *porce*, *derechief*, *maintefois*, *naguère*, *jamais* etc. and adverbial variants with intensive prefix *tres* (*ADVgen*) are always treated analytically.

**Named entities.** The processing of named entities is left to an ulterior stage of corpus annotation. Regarding lemmatisation, each word is tagged according to the part of the original speech, hence the fact that “saint”, part of church names or of a toponym is treated as an adjective, e.g. “dou dit priorei de *Saint Jaike on Mont*”,

<table><tr><td>Saint</td><td>saint</td><td>ADJqua</td><td>NOMB.=s|GENRE=m|CAS=r|DEGRE=p</td></tr><tr><td>Jaike</td><td>Jacques</td><td>NOMpro</td><td>NOMB.=s|GENRE=m|CAS=r</td></tr><tr><td>on</td><td>en1+le</td><td>PRE.DETdef</td><td>MORPH=empty+NOMB.=s|GENRE=m|CAS=r</td></tr><tr><td>Mont</td><td>mont</td><td>NOMcom</td><td>NOMB.=s|GENRE=m|CAS=r</td></tr></table>

**Homonyms.** some very frequent homonyms deserve special attention when correcting POS and morphological tagging:

a a3 (PRE), avoir1 (VERcjg), a2 (INJ);

le, la, les il (PROper), le (DETdef);

se, s' soil (PROper), ce1 (PROind), se (CONsub), si (ADVgen), son4 (DETpos), ce2 (DETdem);

ou, o, u o3 (CONcoo), où (PROrel), en1+le (PRE.DETdef);

en en1 (PRE), en2 (PROadv), on (PROind);

ne ne1 (ADVneg), ne2 (CONcoo).

To this, one can add other homographs whose treatments requires a careful morpho-syntactic analysis and is tributary to the entries of the chosen lemma reference list and POS tagset [Tobler and Lommatzsch, 2002, Guillot et al., 2013a]. In particular, *que* can be

que1 (< QUAM) CONsub (in a comparison) or ADVgen (in constructions, ne... que, ne mais que...;

que2 (< QUI, QUEM, QUAM, QUOD) PROrel;

que3 (< QUID) ADVint (interrogative, exclamative);

que4 (< QUIA) CONsub (eg. “il avint que”) or CONcoo (“Si sont pres de trente mile, que chevaliers que sergenz que borjois”).

Regarding flexion tags, there are 12 possible composite labels for all these *que*. Yet its complexity is far from matching verbal forms, with about 60 possibilities (in mode, tense, person, number).
**Irregularities, scribal or editorial mistakes, rare forms.** When a rare or apparently irregular or mistaken form is encountered, the tagging becomes more difficult. For instance, we encounter phenomena of mismatches (in number, gender, verbal time) or case confusion, not only in versified texts (poetic license) but also in prose texts or practical documents. In this case, by examining the context, we choose the tagging that seems most consistent with the meaning:

* *Par Mahomet merveilles* (fem. pl.) *ai oie* (-> fem. pl. despite the form).

* *c'est assavoir de prés, de terres que* (subj. case fem. ‘qui’) *movoiet* (impft pers. 6) *de moi et de mon fié, que li dis abbes avoit encloses et covert* (-> fem. pl.) en son estant dou dit priorei

Diatopic variation can also cause tagging difficulties, as regional forms create new homographs. For instance, the Picard form of the singular feminine definite article or personal pronoun, *le*, homograph to the masculine forms; the possessive *se* (e.g. “*Chançon legiere a entendre Ferai, / car bien m' est mestiers / Ke chascuns le puist aprender / Et c'on le chant volentiers*”). When we cannot decide the gender of a word, we use the value x especially for nouns that can be masculine or feminine, such as *ost*, *onor*, *amour*, etc.

Pyrrha's interface [Clérice and Pilla, 2021] also allows to check directly the “unallowed” values of POS tags and, more importantly, morph (because of the way the lemmatiser is trained – see below –) POS tags unseen in the training set can not be predicted, while morph tags are predicted independently (i.e., genre, numb., person, case...) and then concatenated, resulting occasionally in impossible values (e.g., comparative adjective 2nd person). In all cases, linear verification is essential for lemma disambiguation and morphosyntactic analysis. Finally, to control and homogenise the tagging result, we can revise the work by concordance on the platform itself.

The current state of the system allows a fairly easy exploitation of the corpus and the tagger begins to function properly. However, it is essential to annotate the maximum number of texts now so that the tagger learns to better discern the syntax of the texts and so that we can establish its final version in an optimal form.

### III TRAINING SETUP

In order to get training samples whose structure mimics that of data observed in the real world, we ensured that our data is segmented by sentence (finishing by a *PUNfrt*) or by line (in manuscripts without punctuation). To do so, we transform the dataset using Protogenie [Clérice, 2020] which handles some form of normalisation: we normalise Roman numerals into Arabic numerals to reduce the complexity of the data, as well as the number of allowed numbers and we split the complex morphology into several simple categories (Case, Tense, etc.). The full corpus is then split into 3 different parts, for training, development and testing, with a 80/10/10% ratio.

Each task is trained separately, with a fork of Pie[Manjavacas et al., 2019], PaPie, which provides a few additions including:

* Ability to focus on various metric for improvement tracking (Accuracy, F1, Precision, Recall);

* New optimisers (Ranger for example) and learning rate schedulers;

* “Noise introduction strategies”, such as randomised full capitalisation of sentences.

For each task (Lemma, POS and morphology tags), we conducted a parameter research, using prior knowledge from other languages, for which we will state the value below. Each training phase was run at least 5 times to account for random local minima and each result was logged. Once all models were trained, we implemented a ranking method that ranks each model on each
<table><caption>Table 3 – Best parameters found after the sweep. Bold are the highest value, italic the lowest.</caption><thead><tr><th>Task</th><th>Character Layers</th><th>Embedding Size</th><th>Hidden Size</th><th>Target</th></tr></thead><tbody><tr><td>Lemma</td><td>2</td><td><strong>300</strong></td><td><strong>150</strong></td><td>Precision</td></tr><tr><td>POS</td><td>2</td><td>200</td><td><strong>350</strong></td><td>Accuracy</td></tr><tr><td>CAS</td><td>2</td><td>150</td><td><strong>150</strong></td><td>Accuracy</td></tr><tr><td>DEGRE</td><td>2</td><td>200</td><td>250</td><td>Accuracy</td></tr><tr><td>GENRE</td><td>2</td><td>200</td><td>250</td><td>Accuracy</td></tr><tr><td>MODE</td><td>2</td><td><strong>150</strong></td><td>200</td><td>Accuracy</td></tr><tr><td>NOMB</td><td>1</td><td>200</td><td>250</td><td>Accuracy</td></tr><tr><td>PERS</td><td>2</td><td><strong>150</strong></td><td><strong>350</strong></td><td>Accuracy</td></tr><tr><td>TEMPS</td><td>2</td><td>200</td><td><strong>350</strong></td><td>Accuracy</td></tr></tbody></table>

available and meaningful metric and chose the one whose sums of ranks is the lowest. For this, we excluded metrics such as the one applied to “unknown targets” based on a very small sample of lemma which would rapidly skew the overall ranking of models.

All models shared most of the same parameters: they used a single linear layer, LSTM cells for the hidden network, a character embedding using RNN, a dropout of 0.32, learning rate of 0.0049, patience for the learning rate evolution of 2, a patience for early stopping of 5. We provide the configuration on our repository. We used the Ranger optimizer that has shown less variation in training with better scores<fnref n="5" />.

Four parameters were used to generate diverse combinations:

* The Character Embedding size, `cemb_size`, could be of 100, 150, 200 or 300

* The number of layers for the Character Embeddings Encoder, `cemb_layers`, could be 1 or 2.

* The size of the hidden layer, that encodes most of the context, was a value in the set 150, 200, 250, 300, 350. We added a variation only for the lemma task at 170.

In general, while the hidden size has a low impact on the lemma task, morphological tasks were more inclined to get better results with it. The number of layers always yield better except for POS, and Nomb, indicating that the task was simpler to solve for the network at the character level, most information coming from the context. Targeting precision was efficient only for the lemmatisation task. The final best parameters are shown in table 3.

# IV RESULTS

## 4.1 Scores

Scores are shown in Table 4. They reach a global level of accuracy that is similar regarding lemma and POS (c. 97.5%), and more heterogeneous for morphological tags, from 95.25% for case to 99.05% for verbal mode. The accuracies are relatively robust to ambiguous tokens, at least for lemma and POS.

Thanks to its character level modelling, the model also achieve a **69% accuracy for lemma prediction on tokens never seen during training.**

It is to be noted that, behind the composite figure of 97.66% accuracy for lemmas, variation are to be found between grammatical categories, with 100% accuracy being reached for unambiguous and easy to tag punctuation signs or possessive adjectives (*mien*, *nostre*, etc.) and relatively low

<footnote n="5">⁵On this topic, see the discussion between TC and Enrique Manjavacas at https://web.archive.org/web/20210914113014/https://github.com/emanjavacas/pie/issues/76.</footnote>
<table><thead><tr><th rowspan="2">task</th><th colspan="3">All</th><th colspan="3">Known tokens</th><th colspan="3">Unknown tokens</th><th colspan="3">Ambiguous tokens</th></tr><tr><th>acc</th><th>pre</th><th>rec</th><th>acc</th><th>pre</th><th>rec</th><th>acc</th><th>pre</th><th>rec</th><th>acc</th><th>pre</th><th>rec</th></tr></thead><tbody><tr><td>lemma</td><td>97.66</td><td>76.08</td><td>75.86</td><td>98.38</td><td>91.72</td><td>91.71</td><td>69.01</td><td>48.86</td><td>48.72</td><td>97.79</td><td>78.24</td><td>78.11</td></tr><tr><td>POS</td><td>97.55</td><td>83.00</td><td>79.90</td><td>97.88</td><td>84.81</td><td>82.11</td><td>84.56</td><td>54.19</td><td>55.92</td><td>97.37</td><td>83.70</td><td>81.20</td></tr><tr><td>CAS</td><td>95.25</td><td>91.75</td><td>92.26</td><td>95.74</td><td>92.18</td><td>92.74</td><td>84.63</td><td>62.60</td><td>62.55</td><td>94.01</td><td>91.77</td><td>92.33</td></tr><tr><td>DEGRE</td><td>98.47</td><td>86.45</td><td>82.77</td><td>98.74</td><td>87.75</td><td>84.98</td><td>92.55</td><td>63.81</td><td>44.48</td><td>95.46</td><td>86.91</td><td>85.80</td></tr><tr><td>GENRE</td><td>96.21</td><td>93.22</td><td>89.03</td><td>96.76</td><td>93.67</td><td>89.66</td><td>84.47</td><td>88.01</td><td>69.87</td><td>94.59</td><td>92.11</td><td>87.90</td></tr><tr><td>MODE</td><td>99.05</td><td>91.59</td><td>89.00</td><td>99.36</td><td>93.56</td><td>92.06</td><td>92.39</td><td>79.46</td><td>74.89</td><td>97.61</td><td>88.89</td><td>92.14</td></tr><tr><td>NOMB</td><td>97.05</td><td>96.50</td><td>96.20</td><td>97.47</td><td>96.92</td><td>96.65</td><td>88.20</td><td>87.85</td><td>82.20</td><td>96.06</td><td>95.67</td><td>95.36</td></tr><tr><td>PERS</td><td>98.79</td><td>91.25</td><td>85.28</td><td>98.95</td><td>91.73</td><td>85.73</td><td>95.34</td><td>90.17</td><td>90.12</td><td>97.72</td><td>91.31</td><td>85.12</td></tr><tr><td>TEMPS</td><td>99.13</td><td>96.51</td><td>96.20</td><td>99.38</td><td>97.21</td><td>97.58</td><td>93.94</td><td>88.92</td><td>85.39</td><td>97.80</td><td>95.02</td><td>96.78</td></tr></tbody></table>

Table 4 – Lemmatisation and tagging accuracies on the test set for the best model for each configuration. “Unknown tokens” are tokens never seen during training, while “ambiguous tokens” are forms that can correspond to different lemmas. Results for “Unknown targets” (lemmas never seen in training, but that the neural network can still accurately predict thanks to its character level modelling) are not given because the support (258 tokens) was too low for it to be significant.

scores for misspelled words (OUT), interjections, categories with a grammatical ambiguity or proper names (see in appendix B for detailed scores).

Some lower scores are affected by grammatical ambiguity and arbitrary lemmatisation choices. Apart from the different forms of *que*, that we will discuss in next subsection, the adjectival forms of the verbs and the qualifying adjectives have score that are driven down by this. For instance, for VERppa, the choice of tags for the forms in -ant (*vaillant*, *chantant*, *vivant*,...), that in Modern French, can be depending on the case, either adjectives or present participles, but are not so strictly classified in Old French, necessitate a more theoretical and arbitrary decision than some other categories. They are here tagged (according to Cattex2009 principles, Guillot et al. [2013b]) always as verbal forms (VERppa) with a human annotated verbal infinitive as lemma (*vaillant→valoir*, *chantant→chanter*, etc.), yet the tagger is on occasion inclined to treat them as adjectives (indeed close to their actual syntactic function).

## 4.2 Errors and Most frequent confusions

To go beyond the constructed numeric values of scores, it is possible to inspect the confusion matrix indicating the most frequent types of wrong predictions and a classical tool in machine learning (Tables 5 and 6).

Concerning lemmas, two major conclusions can be drawn:

1. the most frequent errors are massively related to homographs, and particularly homographs of function words (*que1-4*, *ne1-ne2*, *le* as pronoun or definite determiner, etc.), that also cause trouble to human annotators (see above, subsection 2.2). Particularly, confusions between *que1*, *que2*, *que3* and *que4* account for 15.25% of all errors.

2. the most frequent errors are on function words, which is easily explainable given that they form a large majority of the total number of words for a small number of lemmas,

3. meanwhile, **around 40% of errors are distributed between 679 different lemma with a single error** (e.g., *avoutire1*, *hustin*, *Mordred*,...), mostly proper names, content words and rare forms (which are interesting to lexicographers),

4. and a majority of errors themselves (e.g., *il* instead of *avoir*, *venir* instead of *aler*, etc.) are in single occurrence (53%), which complicates the task of batch correction.

Concerning part-of-speech tags (see detailed scores per category in appendix, Table A), a few categories achieve a perfect score due to their mostly unambiguous nature and very low diversity,
<table><caption>Table 5 – Confusion matrix for lemmas (only values > 10 were kept).</caption><thead><tr><th>GT</th><th>Errors</th><th>Preds</th><th>Freq</th></tr></thead><tbody><tr><td>que2</td><td>147</td><td>que4</td><td>137</td></tr><tr><td>que4</td><td>50</td><td>que2</td><td>41</td></tr><tr><td>que1</td><td>49</td><td>que4</td><td>46</td></tr><tr><td>il</td><td>42</td><td>le</td><td>36</td></tr><tr><td>le</td><td>26</td><td>il</td><td>24</td></tr><tr><td>a3</td><td>21</td><td>avoir</td><td>19</td></tr><tr><td>ne1</td><td>20</td><td>ne2</td><td>20</td></tr><tr><td>avoir</td><td>19</td><td>a3</td><td>13</td></tr><tr><td>que3</td><td>16</td><td>que4</td><td>10</td></tr><tr><td>ne2</td><td>15</td><td>ne1</td><td>15</td></tr><tr><td>si</td><td>14</td><td>se</td><td>9</td></tr><tr><td>se</td><td>13</td><td>soi1</td><td>6</td></tr><tr><td>on</td><td>13</td><td>en3</td><td>5</td></tr><tr><td>en2</td><td>12</td><td>en1</td><td>6</td></tr><tr><td>ce2</td><td>11</td><td>ce1</td><td>6</td></tr></tbody></table>

<table><caption>Table 6 – Confusion matrix for parts-of-speech (only values ≥ 10 were kept).</caption><thead><tr><th>GT</th><th>Errors</th><th>Preds</th><th>Freq</th><th>GT</th><th>Errors</th><th>Preds</th><th>Freq</th></tr></thead><tbody><tr><td rowspan="6">NOMcom</td><td rowspan="6">203</td><td>ADJqua</td><td>53</td><td>OUT</td><td>70</td><td>VERcjg</td><td>13</td></tr><tr><td>VERppe</td><td>38</td><td>PROper</td><td>69</td><td>DETdef</td><td>22</td></tr><tr><td>VERcjg</td><td>31</td><td></td><td></td><td>PROimp</td><td>17</td></tr><tr><td>ADVgen</td><td>25</td><td></td><td></td><td>DETpos</td><td>15</td></tr><tr><td>NOMpro</td><td>19</td><td>PROind</td><td>64</td><td>DETind</td><td>22</td></tr><tr><td>VERinf</td><td>15</td><td></td><td></td><td>ADVgen</td><td>18</td></tr><tr><td rowspan="6">ADVgen</td><td rowspan="6">173</td><td>NOMcom</td><td>35</td><td>NOMpro</td><td>48</td><td>NOMcom</td><td>20</td></tr><tr><td>CONsub</td><td>26</td><td>CONcoo</td><td>45</td><td>ADVneg</td><td>26</td></tr><tr><td>PRE</td><td>21</td><td>DETdef</td><td>36</td><td>PROper</td><td>31</td></tr><tr><td>PROind</td><td>16</td><td>DETcar</td><td>28</td><td>DETndf</td><td>12</td></tr><tr><td>ADJqua</td><td>15</td><td></td><td></td><td>ADJcar</td><td>10</td></tr><tr><td>PROper</td><td>13</td><td>ADVneg</td><td>25</td><td>CONcoo</td><td>14</td></tr><tr><td rowspan="4">VERcjg</td><td rowspan="4">155</td><td>VERppe</td><td>59</td><td>ADJpos</td><td>25</td><td>DETpos</td><td>18</td></tr><tr><td>NOMcom</td><td>44</td><td>DETind</td><td>23</td><td>PROind</td><td>8</td></tr><tr><td>PRE</td><td>21</td><td>VERinf</td><td>22</td><td>NOMcom</td><td>15</td></tr><tr><td></td><td></td><td></td><td></td><td></td><td></td></tr><tr><td rowspan="2">ADJqua</td><td rowspan="2">119</td><td>NOMcom</td><td>66</td><td>DETpos</td><td>22</td><td>PROper</td><td>5</td></tr><tr><td>VERppe</td><td>23</td><td>PROcar</td><td>21</td><td>DETcar</td><td>6</td></tr><tr><td rowspan="2">CONsub</td><td rowspan="2">119</td><td>PROrel</td><td>83</td><td>ADJcar</td><td>20</td><td>DETcar</td><td>10</td></tr><tr><td>ADVgen</td><td>24</td><td>ADJind</td><td>19</td><td>DETind</td><td>13</td></tr><tr><td>PROrel</td><td>98</td><td>CONsub</td><td>86</td><td>VERppa</td><td>17</td><td>ADJqua</td><td>6</td></tr><tr><td rowspan="2">PRE</td><td rowspan="2">78</td><td>ADVgen</td><td>37</td><td>ADVint</td><td>17</td><td>CONsub</td><td>9</td></tr><tr><td>VERcjg</td><td>15</td><td>PROadv</td><td>15</td><td>PRE</td><td>6</td></tr><tr><td></td><td></td><td>PROadv</td><td>10</td><td>PROint</td><td>14</td><td>PROrel</td><td>9</td></tr><tr><td rowspan="3">VERppe</td><td rowspan="3">78</td><td>VERcjg</td><td>38</td><td>PROdem</td><td>13</td><td>DETdem</td><td>9</td></tr><tr><td>NOMcom</td><td>16</td><td>PROord</td><td>12</td><td>ADJord</td><td>8</td></tr><tr><td>ADJqua</td><td>14</td><td>ADVsub</td><td>11</td><td>CONsub</td><td>4</td></tr><tr><td>PROimp</td><td>78</td><td>PROper</td><td>78</td><td>DETdem</td><td>10</td><td>PROdem</td><td>6</td></tr></tbody></table>

such as punctuation signs (PON...), while, at the other end of the spectrum the lowest scores are reached for rare and ambiguous categories like the homographic impersonal pronoun *il* (PROimp, vs *il* PROper), the homographic *que3*, *come1* and other *ADVint*, or the category *OUT*, that was left in the test set, but whose presence is arguable, because this category concerns words that are taken out of grammatical analysis (mostly scribal mistakes, for instance, such as words repeated a second time or left unfinished by the scribe). It is to be noted that these low F1
scores<fnref n="6" /> are mostly driven down by a very low recall. It can be interpreted as a large tendency of the tagger to give to occurrences of this rarer categories for a given form or lemma the tags from one more often encountered for them in the training material, as can also be seen from the confusion matrix (Table 6).

Content-words categories (such as NOM..., ADJ... or VER...) achieve relatively high scores, yet their high frequency makes them featuring prominently in the confusion matrix, with some of the most frequent confusions being between common nouns and adjectives, or between the different verbal subcategories.

## 4.3 Qualitative inspection of lemmatisation results and comparisons with other models

For a more direct inspection by the human, we evaluate the results of the lemmatiser on a test set, which contains 2 849 random sentences drawn from the corpora. The evaluation is made on the comparison between the prediction of the model and the manually corrected gold standard. We then compute a sentence-level word-based accuracy (number of words with at least one error divided by total number of words). The distribution of sentences scores is presented in Table 7.

<table><caption>Table 7 – Number of sentences for each score</caption><thead><tr><td>Sentence score</td><td>Number of sentences</td></tr></thead><tbody><tr><td>1</td><td>1 871</td></tr><tr><td>0.9 – 1</td><td>842</td></tr><tr><td>0.8 – 0.9</td><td>114</td></tr><tr><td>&lt; 0.8</td><td>22</td></tr></tbody></table>

The results show that most sentences are correct (around 95%). The sentence with the worst score (0.32) is actually not written in Old French but in Latin. Two other sentences have a very weak score, of 0.4 (Table 8; from now on, errors are shown in italics in the tables).

<table><caption>Table 8 – Sentences with worst scores in the test set; row gives the original tokens, the human annotated ground truth (correct), our model prediction (predicted), and, for comparison, the results of two sets of parameters for the TreeTagger lemmatiser.</caption><tbody><tr><td>tokens</td><td>Qui</td><td>montagu</td><td>auoit</td><td>a</td><td>iustisier</td></tr><tr><td>correct</td><td>qui</td><td>Montagu</td><td>aler</td><td>a3</td><td>justicier2</td></tr><tr><td>predicted</td><td>qui</td><td>montaignor</td><td>avoir</td><td>a3</td><td>vistoier</td></tr><tr><td>TT-TL</td><td>cuidier1</td><td>montagu</td><td>avoir</td><td>a3la</td><td>justicier2|justicier</td></tr><tr><td>TT-MF</td><td>qui</td><td>montagu</td><td>avoir</td><td>a</td><td>iustisier</td></tr><tr><td>tokens</td><td>A</td><td>Gironuille</td><td>uont</td><td>ludie</td><td>veir</td></tr><tr><td>correct</td><td>a3</td><td>Gironville</td><td>aler</td><td>Ludie</td><td>vëoir</td></tr><tr><td>predicted</td><td>a3</td><td>Gironle</td><td>avoir</td><td>Ludie</td><td>vair1</td></tr><tr><td>TT-TL</td><td>avoir</td><td>Gironuille</td><td>aler</td><td>ludie</td><td>vair1|voir</td></tr><tr><td>TT-DMF</td><td>a</td><td>Gironuille</td><td>aller</td><td>ludie</td><td>voir</td></tr></tbody></table>

Two parameters explain the weakness of the score: first, the sentences are short, so one error strongly affects the score; secondly, these sentences present peculiar spellings, contrary to the majority of the tokens in the corpus, either because of diatopic variation or editorial choices (e.g., distinction *i/j* and *u/v*). In this particular example, we note several difficulties. One concerns the proper nouns, with *montagu* interpreted as *montaignor* and *Gironuille* as *Gironle*. An other one is the problem of multiple spellings, as *veir* as an occurrence of *vëoir* is difficult to identify. We also note that the human corrected value can sometimes be the wrong one, as *auoit* was wrongly

<footnote n="6">⁶The F1 score is the harmonic mean of precision and recall, and as such a global measure of accuracy.</footnote>
identified by the human annotator as an occurrence of *aler*. The lemmatiser, here, is right. This kind of error can happen elsewhere (Table 9).

<table><caption>Table 9 – lemmatiser can perform better than the human annotator.</caption><tr><td>tokens</td><td>Et</td><td>bien</td><td>sachiez</td><td>qu'</td><td>en</td><td>l'</td><td>eglisse</td></tr><tr><td>correct</td><td>et</td><td>bien1</td><td>sachier2</td><td>que4</td><td>en1</td><td>le</td><td>eglise</td></tr><tr><td>predicted</td><td>et</td><td>bien1</td><td>savoir</td><td>que4</td><td>en1</td><td>le</td><td>eglise</td></tr></table>

This kind of error tends to show that the score that we present is at times lower than it should be. Moreover, other frequent errors which bring down the sentence scores happen because of annotators tag choice regarding capitalisation, numerals, or are due to the tags of the interjections, all that may be at times inconsistently tagged by human annotators (Table 10).

<table><caption>Table 10 – Problems with the choice of the tag</caption><tr><td>tokens</td><td>sarrasins</td><td>dex</td><td>.l.m.</td><td>O</td><td>Ha</td></tr><tr><td>correct</td><td>sarrasin</td><td>dieu</td><td>50000</td><td>o2</td><td>a2</td></tr><tr><td>predicted</td><td>Sarrasin</td><td>Dieu</td><td>50</td><td>ho!</td><td>ha!</td></tr></table>

All of these are marginal errors, that could be resolved with some additional work on the harmonisation of the corpora. Some errors are more important. They are caused by homography and often concern personal pronouns and possessive, as we can see in Table 11.

<table><caption>Table 11 – Problems with personal pronouns and possessives</caption><tr><td>tokens</td><td>Et</td><td>veez</td><td>la</td><td>la</td><td>.</td></tr><tr><td>correct</td><td>et</td><td>vêoir</td><td>il</td><td>là</td><td>.</td></tr><tr><td>predicted</td><td>et</td><td>vêoir</td><td>là</td><td>là</td><td>.</td></tr><tr><td>TT-TL</td><td>et</td><td>vêer|vêoir</td><td>le</td><td>là|il</td><td>.</td></tr><tr><td>TT-DMF</td><td>et</td><td>vêoir</td><td>là</td><td>là</td><td>.</td></tr><tr><td>tokens</td><td>et</td><td>nos</td><td>partimes</td><td></td><td></td></tr><tr><td>correct</td><td>et</td><td>nos1</td><td>partir</td><td></td><td></td></tr><tr><td>predicted</td><td>et</td><td>nostre</td><td>partir</td><td></td><td></td></tr><tr><td>TT-TL</td><td>et</td><td>nos|nos1|noz1</td><td>partir</td><td></td><td></td></tr><tr><td>TT-DMF</td><td>et</td><td>nos</td><td>partir</td><td></td><td></td></tr><tr><td>tokens</td><td>li</td><td>reis</td><td>garsie</td><td>est</td><td>mis</td><td>germeins</td><td>cusins</td><td>mis</td><td>uncle</td><td>fu</td><td>fernagu</td></tr><tr><td>correct</td><td>le</td><td>roi</td><td>Garsie</td><td>estrel</td><td>mon1</td><td>germain</td><td>cosin</td><td>mon1</td><td>oncle</td><td>estrel</td><td>Fernagu</td></tr><tr><td>predicted</td><td>le</td><td>roi</td><td>Garsie</td><td>estrel</td><td>metre2</td><td>germain</td><td>cosin</td><td>mon1</td><td>oncle</td><td>estrel</td><td>Fernagu</td></tr><tr><td>TT-TL</td><td>le</td><td>roi</td><td>garsie</td><td>esterl</td><td>manoirl</td><td>&lt;nolem&gt;</td><td>cosin</td><td>mon1</td><td>oncle</td><td>estrellestre</td><td>fernagu</td></tr><tr><td></td><td></td><td></td><td></td><td>estrel</td><td>metrel</td><td>mettre</td><td></td><td></td><td></td><td></td><td></td></tr><tr><td>TT-DMF</td><td>le</td><td>roi</td><td>Garsie</td><td>estrel</td><td>metre2</td><td>germeins</td><td>cosin</td><td>mon1</td><td>oncle</td><td>estrel</td><td>Fernagu</td></tr></table>

The errors are understandable: the form *la* can indeed refer to the adverb *là*, in the first sentence, as the form *nos*, in the second one, can be an occurrence of the personal pronoun or of a possessive. Interesting is the form *mis*, which can be an occurrence of *metre2*, which is once predicted wrong, and once right, in the same sentence.

Other errors which are caused by the existence of similar spellings can be more important (Table 12). Here, the verb *descirier*, “to tear”, is predicted as *desirrier2*, a substantive meaning “whish”. This error is important because it changes the semantics of the whole sentence.

We can compare the results of the lemmatiser with the ones of an other one, TreeTagger, for which at least two sets of parameters for Old French exist, using the lemmas from TL [Stein, s.d.],
<table><caption>Table 12 – Wrong prediction due to homography</caption><tr><td>tokens</td><td>Et tant mantel desrompre et dessirier</td></tr><tr><td>correct</td><td>et tant mantel desrompre et descirier</td></tr><tr><td>predicted</td><td>et tant mantel desrompre et <i>desirier2</i></td></tr><tr><td>TT-TL</td><td><i>avoir</i> tant mantel desrompre et descirier</td></tr><tr><td>TT-DMF</td><td>et tant manteau _ et déchirer</td></tr></table>

henceforth **TT-TL** or from the DMF [Base de français médiéval, s.d.], henceforth **TT-DMF** <fnref n="7" />. The results for the two sentences with low scores tagged with TreeTagger are shown in Table 8.

The lemmatiser has the same difficulties which were mentioned above: proper nouns and particular spellings (*veir*). If its proposition for the form *iustisier* is better than the one from our model, it makes mistakes for easier occurrences (*Qui*, relative pronoun, is identified as an occurrence of *cuidier1*, “to believe”; A, preposition, is identified as an occurrence of the verb *avoir*). It also encounters difficulties with personal pronouns and possessive and produces the same errors as we saw in table 11.

The TT-TL and TT-DMF parameters produce one error our model didn't produce: it cannot identify the form *germeins*. The TT-TL parameters, on the test set, provides for 1,561 occurrences a *<nolem>* tag, and the TT-DMF ones, a “_” tag, as in table 12, for 2,878 occurrences. The TT-TL parameters also propose alternative choices where our model proposes just one.

On occasions, our model is seen better handling spellings marked from a diachronic or diatopic perspective, or more generally, less familiar spellings. For instance, it handles better the Anglo-Norman forms seen in 13, where TT-TL makes six mistakes, TT-DMF makes five and our model none (but it is a sample drawn from an in-domain text).

<table><caption>Table 13 – Comparisons of the results on one sentence with Anglo-Norman forms</caption><tr><td>tokens</td><td>laisum clarel cest saracin aler</td><td>, kar bin vez nel pouum mener</td></tr><tr><td>correct</td><td>laissier Clarel cest Sarrasin aler</td><td>, car bien1 vëoir nel+il pöoir mener</td></tr><tr><td>predicted</td><td>laissier Clarel cest Sarrasin aler</td><td>, car bien1 vëoir nel+il pöoir mener</td></tr><tr><td>TT-TL</td><td>laissier clarel cest saracin aler|foraler</td><td>, car <nolem> aler|vëoir il|ne1 pouvoir mener</td></tr><tr><td>TT-DMF</td><td>laisser clarel cest saracin aller</td><td>, car bin fois ne.il pouum mener</td></tr></table>

It can happen that the results of TreeTagger are better than ours, for instance, in the sentence with the difficult form *dessirier* (Table 12). However, TT-TL makes a mistake for the identification of the easy form *Et*, which cannot be identified, perhaps due to capitalisation. Moreover, it on turns produces hesitations or mistakes on similar forms (Table 14).

<table><caption>Table 14 – Prediction of TreeTagger on the form *desirant*</caption><tr><td>tokens</td><td>As paiens sont venu</td><td>, de ferir desirant</td></tr><tr><td>correct</td><td>a3+le paien estrel venir</td><td>, de ferir desirrer</td></tr><tr><td>predicted</td><td>a3+le paien estrel venir</td><td>, de ferir desirrer</td></tr><tr><td>TT-TL</td><td>a+|le|le paien estrel|estre venir</td><td>, de ferir <i>descirier|desirer</i></td></tr><tr><td>TT-DMF</td><td>a.le païen être venir</td><td>, de férir <i>déchirer</i></td></tr></table>

<footnote n="7">⁷They are available at: https://sites.google.com/site/achimstein/research/resources and https://www.cis.lmu.de/~schmid/tools/TreeTagger/.</footnote>
# V DISCUSSION AND FURTHER RESEARCH

Current models display very satisfying results, in particular when accounting for the specific difficulties of an under-resourced variation-rich non-standardised historical language, in which variation also increases the number of homograph forms and the size of the “vocabulary” of forms. The use of a neural tagger and the creation of a significant gold corpus has allowed to obtain very usable results, especially when tested on in-domain material.

In some cases, it might be possible to achieve further gains in accuracy by crossing predictions for lemma and for part-of-speech (which are for now predicted independently by the neural network). This could be the case, for instance, for the various occurrences of *que<n>* (Table 15).

<table><caption>Table 15 – Lemma and POS accuracy of the prediction for the different kinds of *que*; better performance of the POS indicates that in many cases lemmatisation accuracy could be increased through post-treatment after the lemma prediction phase.</caption><thead><tr><th>GT</th><th>Freq</th><th>Acc. POS</th><th>Acc. Lemma</th><th>Combined Accs</th><th>Pred=que4</th><th>Pred=que2</th></tr></thead><tbody><tr><td>que1</td><td>97</td><td><strong>82.47</strong></td><td>48.45</td><td>39.18</td><td>48.45</td><td>3.09</td></tr><tr><td>que2</td><td>587</td><td><strong>84.67</strong></td><td>75.98</td><td>72.57</td><td>22.49</td><td>75.98</td></tr><tr><td>que3</td><td>37</td><td><strong>67.57</strong></td><td>59.46</td><td>56.76</td><td>24.32</td><td>16.22</td></tr><tr><td>que4</td><td>1474</td><td>92.13</td><td><strong>96.40</strong></td><td>90.84</td><td>96.40</td><td>2.92</td></tr></tbody></table>

Nonetheless, the generality of the models will need to be evaluated in a more systematic fashion, regarding the variety of regional scriptae and written genres of Old French. For this, we will need to build an out-of-domain corpus to get a more general evaluation of the model, and to test its performances on specific scriptae or genre (What are the performances on Picard or Lorrain texts? On theatre? etc.).

Another interesting lead is to explore transfer learning approaches on neighbouring languages, such as Occitan for which some early experiments have been conducted [Camps and Couffignal, 2017], on later varieties such as Middle and Pre-Classical French, and on Franco-Romance hybrids, in particular Franco-Italian, for which the creation of an annotated corpus is ongoing [Gambino, Ceresato, to be published].

Diachrony should also be explored. For the moment, For the moment the corpus includes only one text from the 14th century, an extract from the *Pèlerinage de l'âme* by Guillaume de Digulleville (1355-1358). Given the costs of building lemmatised corpora, one must be careful not to create duplicates. For this reason, an extension towards Middle French would need to be done in order to maintain interoperability of the corpora with the *Dictionnaire de Moyen Français* and the tools developed around it. Integrating this period would fill the current break between the models we make available for Old French and for Early Modern French [Camps et al., 2020] in order to allow for studies in long diachrony [ATILF, 2015].

More generally, further work needs to investigate the interoperability of annotation schemes. One possibility could be to offer a version of the corpora converted to the use of Universal POS tags (UD POS), that are of broad use today for many Contemporary languages. In any case, there is a need for the elaboration of a common canonical reference tag-set and lemma list, to which and from which designing conversions for all the formats that are currently being used by the projects working on Old French lemmatisation.
## AUTHORS CONTRIBUTIONS

JBC started and over-viewed the projects, trained some of the models, took part alone or with others in the annotation or correction of the corpora Geste, worked on the workflows and reference lists; he coordinated the team, with FD. TC worked mostly on the engineering and NLP research around this corpus, built the interfaces that are used (Pyrrha, Deucalion), organized the training and optimization of the models, expanded tools to ensure the applicability outside of tests (PaPie) and engineered the various tools that ensure reproducibility and quality control (Protogenie, PyrrhaCI). He provided some insights when necessary regarding some annotation choices and technological limitations. FD worked on the post-correction of juridical texts and coordinated the team with JBC. LI joined the project during its early phase and worked on the correction of the texts in the LAKME and OMÉLIE projects; she also provided data from her PhD (in progress). NK worked on the post-correction (lemma, POS and morphosyntactic labelling) of texts that were used to improve automated tagging performance. AP joined JBC in annotating data in parallel of the LAKME project with data from her PhD Thesis, provided most of the use-cases and early tests of Pyrrha by correcting her corpus (lemma and morpho-syntax).

The authors have no competing interests to declare.

## MATERIALS AND DATA AVAILABILITY

The most up-to-date version of the models can be easily obtained and used thanks to the pie-extended Python package, available on Pypi (https://pypi.org/project/pie-extended/), with the command `pie-extended download fr`, and can be queried at https://tal chartes.psl.eu/deucalion/.

## ACKNOWLEDGEMENTS

This work benefited from several projects funded by various institutions: LAKME – *Linguistically Annotated Corpora Using Machine Learning Techniques* (2016-2018, Université PSL); OMÉLIE – *Outils et méthodes pour l'édition linguistique enrichie* (2018-..., Université PSL/IRIS Scripta and Région Île-de-France, DIM *Sciences du texte et connaissances nouvelles*), as well as, for the legal corpus, of *Biblissima*. We thank the DIM *Science du texte et connaissances nouvelles* for funding the acquisition of a GPU server, as well as the École nationale des chartes for providing infrastructure and support for the server.

We thank Mike Kestemont for the collaboration regarding neural lemmatisers, as well as Enrique Manjavacas for his precious advice regarding lemmatisation and Pie configuration. We also thank Thierry Poibeau (LATTICE) and Daniel Stoekl (EPHE), with which the initial LAKME project was launched. Our gratitude for the collaboration also goes to the team working on Franco-Italian texts, led by Francesca Gambino. It is difficult to acknowledge the help of all colleagues that helped along the years, but we non exhaustively thank Floriana Ceresato, Simon Gabay, Sophie Prevost as well as all the participants in the workshop "Référentiels de lemmes du français médiéval", in particular, Alexei Lavrentiev, Martin Glessgen, Simon Gaunt, Maud Becker and Gilles Souvay, without forgetting the students of the École des chartes.

## References

* ATILF. *Dictionnaire du Moyen Français (DMF 2015)*. CNRS and Université de Lorraine, Nancy, 2015. URL http://www.atilf.fr/dmf.

* Base de français médiéval. TreeTagger: Parameters for Old French, s.d. URL https://www.cis.uni-muenchen.de/~schmid/tools/TreeTagger/data/old-french.par.gz.
Gace Brulé and Holger Niels Dyggve Petersen. *Gace Brulé, trouvère champenois: édition des chansons et étude historique*. Number 16 in Mémoires de la Société néophilologique de Helsinki. Société néophilologique, Helsinki, 1951.

R. Busa. The Annals of Humanities Computing: The Index Thomisticus. *Computers and the Humanities*, 14 (2):83–90, 1980. ISSN 0010-4817. URL https://www.jstor.org/stable/30207304. Publisher: Springer.

Florian Cafiero and Jean-Baptiste Camps. Why Molière most likely did write his plays. *Science Advances*, 5(11), 2019. doi: 10.1126/sciadv.aax5489. URL https://advances.sciencemag.org/content/5/11/eaax5489.

Jean-Baptiste Camps. *La 'Chanson d'Otinel': édition complète du corpus manuscrit et prolégomènes à l'édition critique*. thèse de doctorat, dir. Dominique Boutet, Paris-Sorbonne, Paris, December 2016. URL https://halshs.archives-ouvertes.fr/tel-01664932.

Jean-Baptiste Camps, editor. *Geste: un corpus de chansons de geste, 2016-... (Version 02)*. École nationale des chartes, Paris, April 2019. URL http://doi.org/10.5281/zenodo.2630574. textes du domaine public, développements CC-BY-SA.

Jean-Baptiste Camps and Gilles Guilhem Couffignal. La production de corpus d'occitan médiéval et prémoderne: problèmes et perspectives de travail. In *Fidelitats e dissidencias/Fidélités et Dissidences, Actes du XIVième{} Congrès International de l'Association Internationale d'Études Occitanes, Albi 2017, July 2017*. doi: 10.5281/zenodo.2605497. URL https://halshs.archives-ouvertes.fr/halshs-02050089/. à paraître.

Jean-Baptiste Camps, Lucence Ing, and Elena Spadini. Collating Medieval Vernacular Texts. Aligning Witnesses, Classifying Variants. In *DH2019 Digital Humanities Conference 2019*, Utrecht, Netherlands, July 2019. URL https://hal.archives-ouvertes.fr/hal-02268348.

Jean-Baptiste Camps, Simon Gabay, Paul Fièvre, Thibault Clérice, and Florian Cafiero. Corpus and Models for Lemmatisation and POS-tagging of Classical French Theatre. *arXiv preprint arXiv:2005.07505*, 2020.

Floriana Ceresato. L'analisi lessicale dell'entrée d'espagne: bilancio di una prima sperimentazione. *Francigena*, to be published.

Thibault Clérice. Protogenie, post-processing for nlp dataset, June 2020. URL https://doi.org/10.5281/zenodo.3883585.

Thibault Clérice. Pie Extended, an extension for Pie with pre-processing and post-processing, 6 2021. URL https://github.com/beerst-philology/nlp-pie-taggers.

Thibault Clérice and Julien Pilla. Pyrrha, 7 2021. URL https://github.com/beerst-philology/pyrrha.

Conon de Béthune. *Les chansons de Conon de Béthune*. CFMA. Paris, 1925. URL https://gallica.bnf.fr/ark:/12148/bpt6k977f.

Anthonij Dees, Pieter van Reenen, and Johan A De Vries. *Atlas des formes et des constructions des chartes françaises du XIIIᵉ siècle*. Number 178 in Beihefte zur Zeitschrift für romanische Philologie. M. Niemeyer Verlag, Tübingen, 1980. ISBN 3-484-52084-1. URL http://dx.doi.org/10.1515/9783111328980.

Anthonij Dees, Marcel Dekker, Onno Huber, and Karin Van Reenen-Stein. *Atlas des formes linguistiques des textes littéraires de l'ancien français*. De Gruyter, Berlin, Boston, reprint 2014 edition, 1987. ISBN 978-3-484-52212-1. URL https://www.degruyter.com/viewbooktoc/product/160190.

Oksana Dereza. Lemmatisation for under-resourced languages with sequence-to-sequence learning: A case of early irish. In *Proceedings of Third Workshop Computational linguistics and language science*, volume 4, pages 113–124, 2019. doi: 10.29007/cxtl.

Eglal Éditeur scientifique Doss-Quinby, Joan Tasker Éditeur scientifique Grimbert, Wendy Éditeur scientifique Pfeffer, and Elizabeth Éditeur scientifique Aubrey. * Songs of the women 'trouvères'*. New Haven, 2001. ISBN 978-0-300-08412-2.

S. Feldman, M. A. Marin, M. Ostendorf, and M. R. Gupta. Part-of-speech histograms for genre classification of text. In *2009 IEEE International Conference on Acoustics, Speech and Signal Processing*, pages 4781–4784, Taipei, Taiwan, April 2009. IEEE. ISBN 978-1-4244-2353-8. doi: 10.1109/ICASSP.2009.4960700. URL http://ieeexplore.ieee.org/document/4960700/.

Francesca Gambino, editor. *RIALFrI – Repertorio Informatizzato Antica Letteratura Franco-Italiana*. URL http://www.rialfri.eu/.

Martin Dietrich Gleßgen. *Les plus anciens documents linguistiques de la France*. 2016. URL http://www.rose.uzh.ch/docling/. 3ᵉ édition.

Céline Guillot, Sophie Prévost, and Alexei Lavrentiev. *Manuel de référence du jeu Cattex09*. École normale supérieure de Lyon, Lyon, 2013a. URL http://bfm.ens-lyon.fr/IMG/pdf/Cattex2009_manuel_2.0.pdf. Version 2.0 – 8 avril 2013.
Céline Guillot, Sophie Prévost, and Alexei Lavrentiev. Principes d'annotation cattex09. Technical report, École normale supérieure de Lyon, Lyon, 2013b. version 2.0. http://bfm.ens-lyon.fr/IMG/pdf/Cattex2009_principes_2.0.pdf.

Lucence Ing. *Disparitions lexicales en diachronie: traitements automatiques sur le Lancelot en prose*. Phd thesis, dir. f. duval, codir. j.b. camps, Université PSL, Paris, 2021.

Mike Kestemont and Jeroen De Gussem. Integrated Sequence Tagging for Medieval Latin Using Deep Representation Learning. *Journal of Data Mining & Digital Humanities*, August 2017. URL https://jdmdh.episciences.org/3835. arXiv: 1603.01597.

Mike Kestemont, Guy de Pauw, Renske van Nie, and Walter Daelemans. Lemmatization for variation-rich languages using deep learning. *Digital Scholarship in the Humanities*, 32:fqw034, August 2016. ISSN 2055-7671, 2055-768X. doi: 10.1093/llc/fqw034. URL http://dsh.oxfordjournals.org/content/early/2016/08/26/llc.fqw034.

Pierre Kunstmann, editor. *Chrétien de Troyes: Cligès, Erec, Lancelot, Perceval, Yvain – manuscrit P* (BnF fr. 794). 2009. URL http://www.atilf.fr/dect.

Pierre Kunstmann and Achim Stein. *Le nouveau corpus d'Amsterdam: actes de l'atelier de Lauterbad*, 23-26 février 2006. F. Steiner, Stuttgart, Allemagne, 2007. ISBN 978-3-515-08997-5. ISSN: 0341-0811.

Enrique Manjavacas, Thibault Clérice, and Mike Kestemont. emanjavacas/pie v0.2.3, April 2019. URL https://zenodo.org/record/2654987#.XP-nvC3M0fM.

Enrique Manjavacas, Ákos Kádár, and Mike Kestemont. Improving Lemmatization of Non-Standard Languages with Joint Learning. *arXiv e-prints*, art. arXiv:1903.06939, Mar 2019. URL https://www.aclweb.org/anthology/N19-1153/.

Enrique Manjavacas, Akos Kádár, and Mike Kestemont. Improving lemmatization of non-standard languages with joint learning. *arXiv preprint arXiv:1903.06939*, pages 1493–1503, June 2019. doi: 10.18653/v1/N19-1153. URL https://www.aclweb.org/anthology/N19-1153.

Sylvie Mellet. La lemmatisation et l'encodage grammatical permettent-ils de reconnaître l'auteur d'un texte? *Médiévales*, 21(42):13–26, 2002. doi: 10.3406/medi.2002.1536. URL https://www.persee.fr/doc/medi_0751-2708_2002_num_21_42_1536. Publisher: Persée - Portail des revues scientifiques en SHS.

Sylvie Mellet and Gérald Purnelle. Les atouts multiples de la lemmatisation: l'exemple du latin. In *JADT 2002, 6es Journées internationales d'Analyse statistique des Données Textuelles*, pages 529–538. INRIA, IRISA, 2002. URL https://hal.univ-cotedazur.fr/hal-01365515.

Ariane Pinche. Édition nativement numérique du recueil hagiographique ‘Li Seint Confessor’ de Wauchier de Denain d’après le manuscrit fr. 412 de la Bibliothèque nationale de France. Phd thesis, dir. c. pierreville and b. bureau, Université de Lyon, Lyon, 2021.

Mary A. Rouse and Richard H. Rouse. La concordance verbale des Écritures. In Pierre Riché, Guy Lobrichon, and Michel Zink, editors, *Le Moyen âge et la Bible*, pages 115–122. Beauchesne, Paris, France, 1984. ISBN 978-2-7010-1091-5. ISSN: 0767-0826.

Helmut Schmid. *Probabilistic Part-of-Speech Tagging Using Decision Trees*. 1994. URL https://www.cis.uni-muenchen.de/~schmid/tools/TreeTagger/data/tree-tagger1.pdf.

Helmut Schmid. Deep learning-based morphological taggers and lemmatizers for annotating historical texts. In *Proceedings of the 3rd International Conference on Digital Access to Textual Cultural Heritage*, pages 133–137, 2019.

Achim Stein. *Nouveau Corpus d'Amsterdam. Corpus informatique de textes littéraires d'ancien français (ca 1150-1350)*.

Achim Stein. TreeTagger: Parameters for Old French, s.d. URL https://sites.google.com/site/achimstein/research/resources.

Thibaud IV. *Les chansons de Thibaut de Champagne, roi de Navarre*. CFMA. 1925. URL https://gallica.bnf.fr/ark:/12148/bpt6k53236.

Adolf Tobler and Erhard Friedrich Lommatzsch. *Altfranzösisches Wörterbuch: édition électronique*. F. Steiner, Stuttgart, 2002. ISBN 978-3-515-08199-3.
A DETAILED SCORES PER PART-OF-SPEECH

<table><caption>Table 16 – Detailed scores for each part-of-speech in the test set, giving precision, recall, F1 and support. POS with total frequency < 10 were removed (PRE.DETrel, PRE.PROper, DETcom, DETord, RED).</caption><thead><tr><th>target</th><th>precision</th><th>recall</th><th>f1-score</th><th>support</th></tr></thead><tbody><tr><td>PONfbl</td><td>1.00</td><td>1.00</td><td>1.00</td><td>4746</td></tr><tr><td>PONfrt</td><td>1.00</td><td>1.00</td><td>1.00</td><td>2764</td></tr><tr><td>PONpdr</td><td>1.00</td><td>1.00</td><td>1.00</td><td>450</td></tr><tr><td>PONpga</td><td>1.00</td><td>1.00</td><td>1.00</td><td>766</td></tr><tr><td>PROper.PROper</td><td>1.00</td><td>1.00</td><td>1.00</td><td>13</td></tr><tr><td>ADVneg.PROper</td><td>1.00</td><td>0.98</td><td>0.99</td><td>54</td></tr><tr><td>VERcjg</td><td>0.99</td><td>0.98</td><td>0.99</td><td>9745</td></tr><tr><td>CONcoo</td><td>0.99</td><td>0.99</td><td>0.99</td><td>3982</td></tr><tr><td>DETdef</td><td>0.99</td><td>0.99</td><td>0.99</td><td>3662</td></tr><tr><td>PRE</td><td>0.99</td><td>0.99</td><td>0.99</td><td>5618</td></tr><tr><td>PRE.DETdef</td><td>0.99</td><td>0.99</td><td>0.99</td><td>979</td></tr><tr><td>PROdem</td><td>0.99</td><td>0.99</td><td>0.99</td><td>913</td></tr><tr><td>ADVneg</td><td>0.98</td><td>0.98</td><td>0.98</td><td>1543</td></tr><tr><td>NOMcom</td><td>0.98</td><td>0.98</td><td>0.98</td><td>10007</td></tr><tr><td>PROadv</td><td>0.98</td><td>0.98</td><td>0.98</td><td>811</td></tr><tr><td>PROper</td><td>0.98</td><td>0.99</td><td>0.98</td><td>6168</td></tr><tr><td>VERinf</td><td>0.98</td><td>0.99</td><td>0.98</td><td>1537</td></tr><tr><td>DETpos</td><td>0.97</td><td>0.98</td><td>0.98</td><td>1361</td></tr><tr><td>NOMpro</td><td>0.98</td><td>0.97</td><td>0.97</td><td>1715</td></tr><tr><td>ADVgen</td><td>0.97</td><td>0.97</td><td>0.97</td><td>4999</td></tr><tr><td>INJ</td><td>0.97</td><td>0.97</td><td>0.97</td><td>33</td></tr><tr><td>DETdem</td><td>0.96</td><td>0.98</td><td>0.97</td><td>413</td></tr><tr><td>DETndf</td><td>0.95</td><td>0.98</td><td>0.96</td><td>403</td></tr><tr><td>CONsub</td><td>0.95</td><td>0.95</td><td>0.95</td><td>2578</td></tr><tr><td>VERppe</td><td>0.94</td><td>0.96</td><td>0.95</td><td>2213</td></tr><tr><td>ADVgen.PROper</td><td>0.91</td><td>1.00</td><td>0.95</td><td>10</td></tr><tr><td>ADJqua</td><td>0.94</td><td>0.93</td><td>0.94</td><td>1800</td></tr><tr><td>PROrel</td><td>0.93</td><td>0.94</td><td>0.94</td><td>1630</td></tr><tr><td>DETind</td><td>0.92</td><td>0.96</td><td>0.94</td><td>591</td></tr><tr><td>PROind</td><td>0.93</td><td>0.91</td><td>0.92</td><td>681</td></tr><tr><td>PRE.PROdem</td><td>0.88</td><td>0.93</td><td>0.90</td><td>15</td></tr><tr><td>ADJind</td><td>0.89</td><td>0.89</td><td>0.89</td><td>171</td></tr><tr><td>VERppa</td><td>0.89</td><td>0.87</td><td>0.88</td><td>134</td></tr><tr><td>DETcar</td><td>0.87</td><td>0.84</td><td>0.85</td><td>171</td></tr><tr><td>PROint</td><td>0.91</td><td>0.78</td><td>0.84</td><td>65</td></tr><tr><td>PROpos</td><td>0.84</td><td>0.84</td><td>0.84</td><td>43</td></tr><tr><td>ADJord</td><td>0.76</td><td>0.92</td><td>0.83</td><td>37</td></tr><tr><td>ADVsub</td><td>0.85</td><td>0.79</td><td>0.82</td><td>52</td></tr><tr><td>PROcar</td><td>0.82</td><td>0.79</td><td>0.81</td><td>101</td></tr><tr><td>DETrel</td><td>0.83</td><td>0.68</td><td>0.75</td><td>22</td></tr><tr><td>ADJcar</td><td>0.77</td><td>0.73</td><td>0.75</td><td>75</td></tr><tr><td>DETint</td><td>0.71</td><td>0.77</td><td>0.74</td><td>13</td></tr><tr><td>ADJpos</td><td>0.84</td><td>0.62</td><td>0.71</td><td>66</td></tr><tr><td>PROord</td><td>1.00</td><td>0.52</td><td>0.68</td><td>25</td></tr><tr><td>ADVint</td><td>0.78</td><td>0.55</td><td>0.65</td><td>38</td></tr><tr><td>OUT</td><td>0.91</td><td>0.49</td><td>0.64</td><td>137</td></tr><tr><td>PROimp</td><td>0.80</td><td>0.46</td><td>0.59</td><td>145</td></tr></tbody></table>
B LEMMA SCORES FOR EACH PART-OF-SPEECH CATEGORY

<table><thead><tr><th>POS</th><th>Lemmas Accuracy</th><th>Freq</th><th>Lemmas SDI</th></tr></thead><tbody><tr><td>PONfrt</td><td>100.00</td><td>2764</td><td>0.34</td></tr><tr><td>PONpga</td><td>100.00</td><td>766</td><td>0.00</td></tr><tr><td>PROimp</td><td>100.00</td><td>145</td><td>0.00</td></tr><tr><td>ADJpos</td><td>100.00</td><td>66</td><td>1.87</td></tr><tr><td>ADVneg.PROper</td><td>100.00</td><td>54</td><td>0.00</td></tr><tr><td>DETrel</td><td>100.00</td><td>22</td><td>0.30</td></tr><tr><td>PRE.PROdem</td><td>100.00</td><td>15</td><td>0.00</td></tr><tr><td>DETint</td><td>100.00</td><td>13</td><td>0.00</td></tr><tr><td>PROper.PROper</td><td>100.00</td><td>13</td><td>0.00</td></tr><tr><td>ADVgen.PROper</td><td>100.00</td><td>10</td><td>0.00</td></tr><tr><td>PONfbl</td><td>99.96</td><td>4746</td><td>0.55</td></tr><tr><td>DETind</td><td>99.66</td><td>591</td><td>1.87</td></tr><tr><td>ADJind</td><td>99.42</td><td>171</td><td>1.28</td></tr><tr><td>DETdef</td><td>99.34</td><td>3662</td><td>0.01</td></tr><tr><td>PONpdr</td><td>99.33</td><td>450</td><td>0.00</td></tr><tr><td>CONcoo</td><td>99.27</td><td>3982</td><td>0.99</td></tr><tr><td>DETndf</td><td>99.26</td><td>403</td><td>0.00</td></tr><tr><td>PRE</td><td>99.07</td><td>5618</td><td>2.19</td></tr><tr><td>PROper</td><td>98.96</td><td>6168</td><td>1.30</td></tr><tr><td>DETpos</td><td>98.90</td><td>1361</td><td>1.48</td></tr><tr><td>PRE.DETdef</td><td>98.88</td><td>979</td><td>1.00</td></tr><tr><td>PROdem</td><td>98.58</td><td>913</td><td>0.81</td></tr><tr><td>ADVneg</td><td>98.38</td><td>1543</td><td>0.92</td></tr><tr><td>PROadv</td><td>98.27</td><td>811</td><td>0.65</td></tr><tr><td>VERcjg</td><td>97.76</td><td>9745</td><td>3.96</td></tr><tr><td>ADVgen</td><td>97.76</td><td>4999</td><td>3.83</td></tr><tr><td>VERinf</td><td>97.46</td><td>1537</td><td>4.99</td></tr><tr><td>ADJord</td><td>97.30</td><td>37</td><td>1.87</td></tr><tr><td>DETdem</td><td>97.09</td><td>413</td><td>1.16</td></tr><tr><td>PROind</td><td>97.06</td><td>681</td><td>2.35</td></tr><tr><td>NOMcom</td><td>96.80</td><td>10007</td><td>6.10</td></tr><tr><td>ADVsub</td><td>96.15</td><td>52</td><td>0.70</td></tr><tr><td>PROcar</td><td>96.04</td><td>101</td><td>2.82</td></tr><tr><td>ADJcar</td><td>96.00</td><td>75</td><td>2.03</td></tr><tr><td>CONsub</td><td>95.62</td><td>2578</td><td>1.26</td></tr><tr><td>VERppe</td><td>95.62</td><td>2213</td><td>5.42</td></tr><tr><td>ADJqua</td><td>95.56</td><td>1800</td><td>4.31</td></tr><tr><td>PROpos</td><td>93.02</td><td>43</td><td>1.48</td></tr><tr><td>PROord</td><td>92.00</td><td>25</td><td>1.91</td></tr><tr><td>NOMpro</td><td>91.49</td><td>1715</td><td>5.17</td></tr><tr><td>PROint</td><td>90.77</td><td>65</td><td>1.52</td></tr><tr><td>PROrel</td><td>90.18</td><td>1630</td><td>1.33</td></tr><tr><td>DETcar</td><td>90.06</td><td>171</td><td>3.11</td></tr><tr><td>VERppa</td><td>88.06</td><td>134</td><td>3.90</td></tr><tr><td>INJ</td><td>84.85</td><td>33</td><td>1.68</td></tr><tr><td>ADVint</td><td>78.95</td><td>38</td><td>1.15</td></tr><tr><td>OUT</td><td>62.04</td><td>137</td><td>1.40</td></tr></tbody></table>

Table 17 – Detailed scores of the lemmas for each part-of-speech, giving lemma accuracy, total frequency of the tag, and the Shannon Diversity Index (0 means no diversity). POS with total frequency < 10 were removed (PRE.DETrel, PRE.PROper, DETcom, DETord, RED).