# Lemmatization Experiments on Two Low-Resourced Languages: Low Saxon and Occitan

Aleksandra Miletić, Janine Siewert

► To cite this version:

Aleksandra Miletić, Janine Siewert. Lemmatization Experiments on Two Low-Resourced Languages: Low Saxon and Occitan. Tenth Workshop on NLP for Similar Languages, Varieties and Dialects (VarDial 2023), May 2023, Dubrovnik, Croatia. pp.163-173, 10.18653/v1/2023.vardial-1.17. hal-05311058

HAL Id: hal-05311058

https://hal.science/hal-05311058v1

Submitted on 13 Oct 2025

HAL is a multi-disciplinary open access
archive for the deposit and dissemination of sci-
entific research documents, whether they are pub-
lished or not. The documents may come from
teaching and research institutions in France or
abroad, or from public or private research centers.

L'archive ouverte pluridisciplinaire HAL, est
destinée au dépôt et à la diffusion de documents
scientifiques de niveau recherche, publiés ou non,
émanant des établissements d'enseignement et de
recherche français ou étrangers, des laboratoires
publics ou privés.

<figure><img src="image_1.png" /></figure>

Distributed under a Creative Commons Attribution 4.0 International License
# Lemmatization Experiments on Two Low-Resourced Languages:
Low Saxon and Occitan

**Aleksandra Miletić**

University of Helsinki
Department of Digital Humanities
aleksandra.miletic@helsinki.fi

**Janine Siewert**

University of Helsinki
Department of Digital Humanities
janine.siewert@helsinki.fi

## Abstract

We present lemmatization experiments on the unstandardized low-resourced languages Low Saxon and Occitan using two machine-learning-based approaches represented by MaChAmp and Stanza. We show different ways to increase training data by leveraging historical corpora, small amounts of gold data and dictionary information, and discuss the usefulness of this additional data. In the results, we find some differences in the performance of the models depending on the language. This variation is likely to be partly due to differences in the corpora we used, such as the amount of internal variation. However, we also observe common tendencies, for instance that sequential models trained only on gold-annotated data often yield the best overall performance and generalize better to unknown tokens.

## 1 Introduction

Lemmatization consists in finding the base form of a given inflected form. The definition of the base-form for a grammatical category can vary across languages. It can include, e.g., finding the masculine singular for an adjective (bèlas 'beautiful.F.PL' > bèu 'beautiful.M.SG), or finding the infinitive for a verb (atten 'eat.3PL.IND.PRES' > eaten 'eat.INF'). The main benefit of lemmatization lies in reducing data sparsity by grouping together all surface forms stemming from the same lemma. It is especially useful for morphologically rich languages, for which the high number of surface forms leads to lower token - type ratios. For such languages, lemmatization is systematically used as a preprocessing step for downstream tasks such as parsing, and it is essential for building efficient corpus querying systems.

We approach this task from the perspective of two low-resourced, non standardized minority languages: Occitan and Low Saxon. In the case of non standardized varieties, acquiring even minimal amounts of manually lemmatized data can

be difficult. One of the reasons is the definition of lemmatization itself: in the absence of a common standard, which approach to lemmatization should be adopted? Should lemmatization respect different levels of variation (lexical, morphological, orthographic) which are present in multi-dialect datasets? Or should one variety be chosen for lemmatization purposes and used across all dialects? The former solution allows for the preservation of dialectal differences, but limits the positive impact of lemmatization on data sparsity. The latter is more effective in this respect but it compounds lemmatization and normalization, arguably making the task more difficult. Furthermore, it can be deemed problematic by the speakers of the language in question.

In this paper, we explore both of these approaches: our Low Saxon dataset adopts an interdialectal lemmatization approach, whereas the Occitan dataset's lemmas are dialect-specific. We evaluate the effects of using small, manually annotated datasets for training lemmatization models vs relying on a larger, automatically preannotated corpus. We investigate the utility of developing one general model for all dialects vs training dialect-specific models. Since lemmatization typically relies on PoS information to aid the processing of ambiguous tokens, we look into different ways of using this annotation layer in our corpora by evaluating two learning paradigms: joint learning and classical, sequential learning for PoS-tagging and lemmatization.

## 2 Related Work

Lemmatization methods based on machine learning can be divided into edit tree-based approaches and string transduction methods. The edit tree-based algorithms (ges; Grzegorz Chrupala and van Genabith, 2008; Müller et al., 2015) derive the sequence of edit operations needed to transform the inflected form into the lemma. The edit tree is
used as a label for each wordform – lemma pair. The model learns to predict the edit tree and not the lemma itself, thus treating lemmatization as a classification task.

With the advent of neural methods, lemmatization has been recast as a string-transduction task (e.g. Bergmanis and Goldwater, 2018; Manjavacas et al., 2019). Currently, the main contribution of these approaches to the state of the art seems to be better generalization capacities, measured as the model’s ability to correctly lemmatize unseen wordforms. Bergmanis and Goldwater (2018) report an important improvement on unknown tokens over non-neural approaches, and similar observations are made by Manjavacas et al. (2019). However, both works remark that the neural networks do not seem to outperform edit tree-based approaches on ambiguous tokens. In general, the capacity to deal with ambiguous tokens is believed to depend on the availability of contextual information, which is supposed to facilitate disambiguation. Bergmanis and Goldwater (2018) use a sliding character window as context, and Manjavacas et al. (2019) condition the decoder on sentence-level embeddings. These efforts to include contextual information do not seem sufficient to beat the edit-based methods on this type of tokens.

The well-suitedness of one type of lemmatization algorithm over the other may also depend on the linguistic properties of a given language. Manjavacas et al. (2019) note that, when evaluating on modern languages, the edit tree-based method outperforms the neural model on both West European and Uralic languages, whereas for Slavic languages the neural model yields better results. These results would, however, need to be confirmed, since Ljubešić and Dobrovoljc (2019) find that the edit tree-based approach beats the neural model on South Slavic languages they investigate.

Lemmatization is often paired with PoS (Part of Speech) tagging. Since inflected forms can be ambiguous as to their lemma, relying on PoS-tags can help the disambiguation process. This information can be exploited as part of joint multi-task learning (Kondratyuk et al., 2018; Manjavacas et al., 2019; van der Goot et al., 2021) or, more traditionally, in a sequential approach, in which the models for two tasks are learned separately, but the lemmatizer relies on the morphological information during training and prediction (e.g. Qi et al., 2020). Vatri and McGillivray (2020) compare lemmatizers

for Ancient Greek based on dictionary lookup that exploit PoS information to distinguish ambiguous tokens. Alternatively, some approaches do not rely on this type of information at all (e.g. Bergmanis and Goldwater, 2018), which may simplify lemmatization for low-resource languages.

More generally, lemmatization in the low-resource setting has also received attention in recent work. Bergmanis and Goldwater (2018) evaluate their models both on the full amount of available data and on 10k samples. Saunack et al. (2021) explore the lower bound for training data size on Indian languages: they compare a standard setting with low-resource settings with only 500 and 100 training instances, in which they rely on data augmentation techniques. Saurav et al. (2020) investigate cross-lingual approaches for lemmatizing low-resourced Indian languages.

In this work, we are particularly interested in the low-resource setting, since the gold standard datasets available for Low Saxon and Occitan are limited in size. We also experiment with the cross-lingual and cross-lectual approach by using historical data and related languages. We opt for neural models since we expect a high proportion of unknown tokens in our datasets due to the fact that we are dealing with non standardized languages. We examine both the joint and sequential learning in an attempt to identify the optimal approach to exploit the PoS tagging information present in our datasets.

# 3 Languages

## 3.1 Low Saxon

Low Saxon is a West Germanic language spoken by approximately 4.8 million people primarily in the north-eastern Netherlands and northern Germany (Moseley, 2010). Despite official recognition in both countries, no interdialectal standard variety has been established so far.

Dialect classification of Low Saxon is normally more finegrained than the three-fold subdivision we use here. Dutch Low Saxon is traditionally divided into Gronings, Stellingwerfs, Drents, Sallands, Twents, Veluws, Achterhoeks and Urkers (Bloemhoff et al., 2019, 20), but due to scarcity of data we treat it as one group. The traditional classification of German Low Saxon (see for instance Schröder, 2004 and Stellmacher, 1983) assumes an East-West division based on, among others, the history of settlement and the plural suffix of verbs
in the present tense. However, we have not found
this traditional division to correspond to overall
dialect similarity in our previous dialectometric
experiments. Therefore, we instead adopt a north-
south division following Lameli (2016) and our
own observations. The northern group consists of
North Saxon and Mecklenburgish - West Pomeran-
nian, and the southern group of Westphalian and
Eastphalian. We excluded Brandenburgish, East
Pomeranian and Low Prussian due to data scarcity.

Compared with Middle Low Saxon, the number
of inflectional categories has decreased, and there
is dialectal variation in the number of categories
preserved. For instance, while nouns in Middle
Low Saxon inflected for four cases, nominative,
genitive, dative and accusative (Lasch, 1974), only
a few of the southern varieties in Westphalia and
Eastphalia still distinguish the dative and accusative
(Lindow et al., 1998). Most Low Saxon varieties
in Germany distinguish the nominative and the ac-
cusative, whereas Dutch Low Saxon typically does
not. Usage of the genitive is very restricted in all
Low Saxon varieties.

At the phonological level, we find noticeable
variation in the number of distinct vowel phonemes
preserved and in the ways vowel phonemes have
merged. A typical example is the merger of Proto-
Germanic *â and lengthened *a<fnref n="1" /> that has occurred
outside of Westphalia (Niebaum, 2008; Bloemhoff
et al., 2019). As a result, we find the same
phoneme in *Språak<fnref n="2" /> ‘language’ and *Water ‘water’
in the north-western dialects, while Westphalian,
here Münsterlandic, shows distinct phonemes in
*Språoke and *Water.

In addition to the dialectal variation, there is
considerable orthographic variation as most Low
Saxon writers follow regional writing traditions to
different degrees or might devise their own spelling
systems. These regional or personal spellings of-
ten draw some inspiration from the majority lan-
guage orthographies. This can be seen, e.g., in
the frequent capitalization of nouns by German
Low Saxon writers and in the representation of the
voiced sibilant /z/ with the grapheme <z> by Dutch
Low Saxon writers, while German Low Saxon writ-
ers commonly use 
 for the same phoneme.

Our corpus reflects this orthographic and dialec-

tal variation that poses significant challenges to
NLP.

3.2 Occitan

Occitan is Romance language which belongs to
the Gallo-Romance group. It is closest to Catalan,
with which it forms a subgroup called occitano-
roman (Bec, 1970). It is spoken in southern France
(except in the Basque and Catalan areas), in sev-
eral valleys of the Italian Piedmont and in the Val
d'Aran in Spain. When it comes to its linguistic
properties, Occitan is a null subject language with
tense, person and number inflection marks on fi-
nite verbs. Number and gender are marked on all
components of the noun phrase in many dialects.

The most widely accepted classification pro-
posed by Bec (1995) includes 6 major dialec-
tal groups: Auvernhat, Gascon, Lengadocian,
Lemosin, Provençau and Vivaroaupenc³, each of
them with areas of greater or lesser variation. Ge-
ographic variation affects all levels of linguistic
structure. In this paper we focus on Lengado-
cian, Gascon, Provençau and Lemosin, due to the
availability of annotated material for these dialect
groups. Geographical variation affects all levels
of linguistic structure. Different phonological pro-
cesses have resulted in series of wordforms specific
to each dialect group, e.g. the word *son* translates
to *hilh* in Gascon, *filh* in Lengadocian and Lemosin,
and *fiu* in Provençau. On the lexical level, the word
*potato* corresponds to *mandòrra*, whereas it is *tr-
ufa/trufet* or *patana/patanon* in Lengadocian. On
the morpho-syntactic level, verb inflection varies
from one dialect to another, and there is also an
important degree of intra-dialectal variation. To
illustrate, *we are* corresponds to *èm* in Gascon,
*sem* in Lemosin, *sèm* in Lengadocian and *siam* in
Provençau based on the most frequent paradigm
for each dialect group.

This situation is further complicated by the exis-
tence of several orthographic norms, out of which
two seem to dominate today: the so-called *Mis-
tralian* orthography, inspired by French writing
conventions, and the *classical* orthography, closer
to the medieval troubadours’ spelling (Sibille,
2002). The data used in our experiments is lim-
ited to the classical orthography.

<footnote n="1">¹This lengthening happened relatively regularly in open syllables.</footnote>

<footnote n="2">²Notice the apocope of final -e that has occurred in most northern dialects. Vowel length is often marked by doubling the letter in closed syllables.</footnote>
<table><caption>Table 1: SMALL and LARGE dataset information for Low Saxon and Occitan</caption><thead><tr><th></th><th>Dataset</th><th>Sent.</th><th>Tok.</th><th>Types</th><th>Sent. len.</th></tr></thead><tbody><tr><td rowspan="4">Low Saxon</td><td>SMALL</td><td>All dialects</td><td>904</td><td>19258</td><td>6000</td><td>21.30</td></tr><tr><td></td><td>Dutch LS</td><td>310</td><td>6716</td><td>2297</td><td>21.66</td></tr><tr><td></td><td>North Ger. LS</td><td>265</td><td>5415</td><td>1961</td><td>20.43</td></tr><tr><td></td><td>South Ger. LS</td><td>326</td><td>7127</td><td>2635</td><td>21.86</td></tr><tr><td></td><td>LARGE</td><td>All dialects</td><td>126359</td><td>2431944</td><td>166625</td><td>19.25</td></tr><tr><td rowspan="5">Occitan</td><td>SMALL</td><td>All dialects</td><td>1522</td><td>26122</td><td>6196</td><td>17.16</td></tr><tr><td></td><td>Gascon</td><td>255</td><td>4170</td><td>1429</td><td>16.35</td></tr><tr><td></td><td>Lengadocian</td><td>1113</td><td>19315</td><td>4499</td><td>17.35</td></tr><tr><td></td><td>Lemosin</td><td>77</td><td>1344</td><td>596</td><td>17.45</td></tr><tr><td></td><td>Provençau</td><td>77</td><td>1293</td><td>583</td><td>16.79</td></tr><tr><td></td><td>LARGE</td><td>-</td><td>100000</td><td>2037723</td><td>147070</td><td>20.38</td></tr></tbody></table>

## 4 Datasets

For both languages, we use two basic datasets: the SMALL dataset is manually annotated and it was available for both languages at the beginning of the experiments reported here. The LARGE datasets are an order of magnitude greater than their SMALL counterparts, but contain only automatic preannotation with PoS-tags and lemmas. In the experiments presented here, the SMALL datasets were used for initial training of our models, and we make use of their dev and test splits for training and for evaluation. The LARGE datasets were used as additional training material in various setups in an attempt to improve model accuracies. In the remainder of this section, we provide some quantitative details and descriptions of each dataset. Note that the LARGE datasets were not annotated at the beginning of our work. The strategies used to palliate this are described in Section 6.

In the case of Low Saxon, both the SMALL and the LARGE dataset stem from the same corpus, described in Siewert et al. (2020), and contain several genres, for instance fiction texts such as fairytales or novels, and non-fiction texts such as letters, announcements or political speeches. The Low Saxon dataset is roughly split into two time periods: 1800–1939 and 1980–2022. The distribution within the dialect groups is as follows: Dutch Low Saxon 20% and 80%, North German Low Saxon 87% and 13%, South German Low Saxon 44% and 56%.

For Occitan, the SMALL dataset is based on the treebank presented in Miletic et al. (2020) and contains predominantly literary texts. The LARGE Occitan corpus contains Occitan Wikipedia articles

from 2021, taken from the Leipzig Corpora Collection<fnref n="4" />.

For both languages, the gold dataset has also been stratified into dialect groups in order to examine the usefulness of dialect-specific training data and evaluate model performance for different dialects. The gold sets are split into train, test and development sets (except in the case of Occitan, for which two dialect groups do not have a dev set, the total amount of annotated data being too small)<fnref n="5" />. A quantitative overview of gold splits is given in Table 2. The unknown and ambiguous tokens are defined in relation to the gold annotated train set including all dialects.

## 5 Tools

We make use of two training paradigms: multi-task learning applied to PoS-tagging and lemmatization, in which both tasks are learned as part of the same model, and traditional sequential learning, in which a separate model is trained for each task. We explore the former with MaChAmp (van der Goot et al., 2021) and use the Stanza NLP pipeline (Qi et al., 2020) for the latter.

### 5.1 MaChAmp

MaChAmp is a toolkit that allows for easy fine-tuning and joint learning of a wide range of NLP tasks, including PoS-tagging, lemmatization, parsing, masked language modelling and text generation. MaChAmp takes a pretrained contextualized model as the initial encoder and fine-tunes it according to a given set of downstream tasks. Each task has its own decoder for task-specific predictions. The tool also allows an initial round of training on a specific task, and then fine-tune it in a second round of training. We put this functionality to test in our lemmatization experiments. As the default embeddings, MaChAmp uses mBERT (Devlin et al., 2019). For a detailed description of the tool and the model it is based on, the reader is referred to van der Goot et al. (2021).

### 5.2 Stanza

Stanza is a Python NLP pipeline currently supporting 66 languages (which do not include Occitan and Low Saxon). The tool supports tokenization,

<footnote n="4">⁴https://corpora.uni-leipzig.de/en?corpusId=oci_wikipedia_2021</footnote>

<footnote n="5">⁵Since the original corpus did not have *dev* splits, the corpus was re-split into *train*, *dev* and *test* for the needs of the experiments we describe.</footnote>

<footnote n="3">³Names of dialects are given in Occitan (each one in its dialect) as there is no standardized orthographic form for those names in English.</footnote>
<table><caption>Table 2: SMALL dataset split into train, dev and test</caption><thead><tr><th rowspan="2">Dataset</th><th colspan="3">train</th><th colspan="3">test</th><th colspan="3">dev</th></tr><tr><th>Sent.</th><th>Tok.</th><th>Types</th><th>Sent.</th><th>Tok.</th><th>Types</th><th>Unk. (%)</th><th>Amb. (%)</th><th>Sent.</th><th>Tok.</th><th>Types</th><th>Unk. (%)</th><th>Amb. (%)</th></tr></thead><tbody><tr><td rowspan="4">Low Saxon</td><td>All dialects</td><td>723</td><td>15346</td><td>5083</td><td>91</td><td>1972</td><td>1020</td><td>26.88</td><td>36.76</td><td>90</td><td>1940</td><td>930</td><td>26.29</td><td>32.84</td></tr><tr><td>Dutch LS</td><td>249</td><td>5072</td><td>1878</td><td>31</td><td>925</td><td>469</td><td>24.54</td><td>37.73</td><td>30</td><td>719</td><td>410</td><td>23.5</td><td>32.55</td></tr><tr><td>North German LS</td><td>213</td><td>4447</td><td>1683</td><td>26</td><td>391</td><td>241</td><td>24.3</td><td>37.6</td><td>26</td><td>577</td><td>352</td><td>25.12</td><td>33.62</td></tr><tr><td>South German LS</td><td>262</td><td>5827</td><td>2220</td><td>32</td><td>656</td><td>407</td><td>31.71</td><td>34.91</td><td>32</td><td>644</td><td>406</td><td>30.43</td><td>32.45</td></tr><tr><td rowspan="5">Occitan</td><td>All dialects</td><td>1196</td><td>20551</td><td>5292</td><td>202</td><td>3179</td><td>1054</td><td>22.11</td><td>28.18</td><td>124</td><td>2392</td><td>1009</td><td>16.39</td><td>31.77</td></tr><tr><td>Gascon</td><td>195</td><td>3258</td><td>1173</td><td>35</td><td>421</td><td>230</td><td>26.37</td><td>23.28</td><td>25</td><td>491</td><td>267</td><td>19.35</td><td>33.60</td></tr><tr><td>Lengadocian</td><td>884</td><td>15494</td><td>3937</td><td>130</td><td>1920</td><td>577</td><td>19.64</td><td>27.50</td><td>99</td><td>1901</td><td>814</td><td>15.62</td><td>31.30</td></tr><tr><td>Lemosin</td><td>56</td><td>919</td><td>434</td><td>16</td><td>413</td><td>211</td><td>27.76</td><td>31.76</td><td>-</td><td>-</td><td>-</td><td>-</td><td>-</td></tr><tr><td>Provençau</td><td>61</td><td>880</td><td>424</td><td>16</td><td>413</td><td>211</td><td>23.49</td><td>32.69</td><td>-</td><td>-</td><td>-</td><td>-</td><td>-</td></tr></tbody></table>

multi-word token expansion, lemmatization, PoS and morphological feature tagging, dependency parsing, and named entity recognition. In this work, we utilize its PoS-tagger, based on a biLSTM model, and its lemmatizer, a neural seq2seq model. For more details, please see Qi et al. (2020).

## 6 Strategies for Creating Large(r) Amounts of Annotated Data

One of the dimensions of lemmatization we explored in this work relates to the size and the nature of the training material. Specifically, we compared the performance of tools trained on small amounts of gold-annotated data with using larger corpora that were automatically preannotated. As mentioned in Section 1, the corpora we used as our LARGE datasets were not annotated at the outset of the experiments presented here. There were, to the best of our knowledge, no freely available models based on neural approaches for the PoS-tagging and the lemmatization of Low Saxon and Occitan. The first round of our experiments was therefore dedicated to creating initial models for both tasks which would allow us to produce reliable automatic preannotation. For Low Saxon, we leveraged an existing historical corpus of Middle Low Saxon to train models that were then transferred to Modern Low Saxon. This had the advantage of using a corpus that was larger than the available gold standard in Modern Low Saxon. For Occitan, no comparable historical corpus was available. We therefore relied on bootstrapping using the SMALL dataset.

### 6.1 Leveraging a Historical Corpus for the Preannotation of Modern Text

The initial preannotation of the Low Saxon lemmas was done with MaChAmp and the reference corpus Middle Low German<fnref n="6" /> / Low Rhenish (Peters,

2017). The reference corpus uses a tagset specifically designed for the needs of Middle Low Saxon. Therefore, we instead made use of the automatic PoS annotation provided by Siewert et al. (2022).

The reference corpus consists of two parts: an annotated part that comes with supradialectal lemmatization following primarily the *Mittelniederdeutsches Handwörterbuch* by Lasch et al. and the one by Lübben (1995 - 1888) and a transcribed part without annotation. We converted the annotated part to the ConLLU-format required by the tools we used. The MaChAmp lemmatization model achieved an accuracy of 89.9% on this data. This model was subsequently finetuned on a small set of manually annotated modern data in order to annotate the rest of the corpus.

Our modern Low Saxon gold annotated dataset does not employ the Middle Low Saxon dictionary spelling, but the *Nysassiske Skryvwyse*<fnref n="7" /> ‘New Saxon spelling’, an interregional spelling based on historical sound correspondences and used by, for instance, the Dutch Low Saxon Wikipedia. As this spelling does not reduce all dialectal variation, the lemma form is, as far as possible, chosen based on the Middle Low Saxon dictionary form, attested Old Saxon forms or Proto-Germanic reconstructions. For future comparisons with the historical corpus, it would be desirable to add a Middle Low Saxon lemmatization layer to the modern data.

The final pretrained MaChAmp model for modern Low Saxon achieved a lemma accuracy of 87%, and a PoS accuracy of 94% on the manually annotated development set. These relatively good results (compared with our later experiments) might be explained by some overfitting as we used the same development set in two consecutive training steps: Original lemmatization finetuning and later joint training of lemmatization and PoS tagging.

<footnote n="6">⁶Called “Middle Low German” in the official English name of the reference corpus; We otherwise refer to this language</footnote>

<footnote>as “Middle Low Saxon”.</footnote>

<footnote n="7">⁷https://skryvwyse.eu</footnote>
## 6.2 Bootstrapping Using a Small Gold Standard Corpus and a Lexicon

For Occitan, we used MaChAmp in order to train a PoS-tagger and a lemmatizer which would allow us to preprocess the LARGE dataset. Since this was a preliminary experiment with the tool on this language, we opted for training independent models for each of the tasks in order to evaluate the baseline performance for each task on gold data. In this scenario, we did a single training run on the full SMALL dataset, using the default embeddings.

The PoS-tagger achieved global accuracy of 92.26% on the test set comprised of all dialects, the highest being 92.97% on Lengadocian and the lowest 89.10% on Provençau.

The lemmatizer's global accuracy reached 89.30%, ranging from 88.6% on Gascon to 93.33 on Lengadocian and Lemosin (detailed results for the global evaluation are available in Table 6, and for the dialect-specific evaluation in Table 7).

These models were ensembled with the morphological lexicon Loflòc (Vergez-Couret, 2016; Bras et al., 2020). If a wordform was present in the lexicon and only had one entry, it was annotated with information found in the lexicon. Otherwise, the models' predictions were used. Around a third of the wordforms received lexicon-based annotations.

The preannotated corpus was used as the LARGE dataset in the experiments described in the sections below.

# 7 Large Preannotated Corpora and Small Gold Datasets

Following the creation of additional annotated material, we trained new models with both MaChAmp and Stanza.

With MaChAmp, we used the LARGE datasets for the initial round of training, and the SMALL dataset for the fine-tuning of the model. The fine-tuning was done both with the full SMALL dataset and using dialect-specific subsets of it. We trained for lemmatization and PoS-tagging jointly, resulting in one model capable of performing both tasks.

With Stanza, we trained lemmatizers both on the SMALL dataset on its own and on a combined dataset, concatenating SMALL and LARGE datasets. This approach was chosen because the current version of Stanza does not support retraining. We leveraged the available morphological information in the training process. We also trained the corresponding PoS-tagging models: they are used to ap-

proximate a pipeline setup and evaluate the Stanza lemmatizers on predicted PoS-tags.

Additionally, we trained a lemmatizer that does not rely on morphological annotation with both tools. These models were intended as a baseline, but they also correspond to a real-life usecase in which a lemmatized corpus for a given language is available, but contains no PoS tags.

The global lemmatization results are given in Table 6, whereas the dialect-specific results are available in Tables 7 and 8. We report mean accuracy and standard deviation over three training runs on the test set<fnref n="8" />. In addition to results on the full evaluation set, we also report performance on unknown and ambiguous tokens. We consider as unknown tokens those that do not appear in any of the training material. We define as ambiguous all tokens having more than one possible lemma in the training material. In the case of dialect-specific evaluations, we evaluate the dialect-specific model trained using MaChAmp along with the general models trained with both tools. Our goal is to assess if dialect-specific training is useful even if it entails using less training data than for the general model.

## 7.1 General results

As an overall tendency, Occitan seems to be easier to lemmatize than Low Saxon, with the former's accuracy ranging often around 10% higher than the latter's. In case of the unknown tokens, the difference is even bigger. Given the greater orthographic variation in our Low Saxon dataset, this does not come as a surprise.

The sequential approach of the Stanza pipeline most of the time yields the best results for both Low Saxon and Occitan. Surprisingly, we found the MaChAmp base model⁹ to perform best for Low Saxon, with an almost 5% advantage over the finetuned model. On Occitan, finetuning the MaChAmp model does bring an improvement, albeit a small one (around 1.5%)

Large automatically annotated corpora seem to bring some benefit for the overall accuracy but they do not generally outperform the smaller Stanza models which have access to the PoS information. In the case of unknown tokens in particular, we see that the Stanza model trained only on gold data with gold PoS performs best.

<footnote n="8">⁸The results on the dev set are available in Appendix A.
⁹Only trained on a large corpus of automatically annotated data, no finetuning on gold data.</footnote>
<table><caption>Table 3: Global Lemmatization Accuracy for Occitan and Low Saxon</caption><thead><tr><th></th><th>Tool</th><th>Training set</th><th>Task</th><th>Train cond.</th><th>Test cond.</th><th>ALL</th><th>UNK</th><th>AMB</th></tr></thead><tbody><tr><td rowspan="10">Occitan</td><td rowspan="5">MaChAmp<br>Stanza</td><td>SMALL</td><td>LEM</td><td>no POS, gold LEM</td><td>no POS</td><td>91.28<sup>±0.42</sup></td><td>72.22<sup>±1.55</sup></td><td>96.23<sup>±0.37</sup></td></tr><tr><td>LARGE</td><td>POS+LEM</td><td>pred. POS+LEM</td><td>no POS</td><td>91.77<sup>±0.23</sup></td><td>68.54<sup>±1.86</sup></td><td>92.19<sup>±0.14</sup></td></tr><tr><td>L+S</td><td>POS+LEM</td><td>pred. POS+LEM</td><td>no POS</td><td>92.16<sup>±0.25</sup></td><td>67.2<sup>±0.33</sup></td><td>93.05<sup>±0.45</sup></td></tr><tr><td>SMALL</td><td>LEM</td><td>no POS</td><td>no POS</td><td>90.35<sup>±0.42</sup></td><td>66.86<sup>±1.85</sup></td><td>95.78<sup>±0.0</sup></td></tr><tr><td>SMALL</td><td>LEM</td><td>gold POS+LEM</td><td>pred. POS</td><td><strong>93.21</strong><sup>±0.09</sup></td><td><strong>78.43</strong><sup>±0.41</sup></td><td><strong>96.69</strong><sup>±0.0</sup></td></tr><tr><td>COMB</td><td>LEM</td><td>pred. POS+LEM</td><td>pred. POS</td><td>92.49<sup>±0.08</sup></td><td>68.4<sup>±0.98</sup></td><td>92.63<sup>±0.0</sup></td></tr><tr><td rowspan="5">Low Saxon</td><td rowspan="5">MaChAmp<br>Stanza</td><td>SMALL</td><td>LEM</td><td>no POS, gold LEM</td><td>no POS</td><td>70.74<sup>±0.09</sup></td><td>17.47<sup>±0.48</sup></td><td>88.63<sup>±0.26</sup></td></tr><tr><td>LARGE</td><td>POS+LEM</td><td>pred. POS+LEM</td><td>no POS</td><td><strong>83.42</strong><sup>±0.21</sup></td><td>30.19<sup>±1.33</sup></td><td>85.19<sup>±0.47</sup></td></tr><tr><td>L+S</td><td>POS+LEM</td><td>pred. POS+LEM</td><td>no POS</td><td>78.14<sup>±0.31</sup></td><td>20.44<sup>±1.18</sup></td><td>81.2<sup>±0.22</sup></td></tr><tr><td>SMALL</td><td>LEM</td><td>no POS</td><td>no POS</td><td>75.33<sup>±0.11</sup></td><td>36.41<sup>±0.42</sup></td><td>82.03<sup>±0.0</sup></td></tr><tr><td>SMALL</td><td>LEM</td><td>gold POS+LEM</td><td>pred. POS</td><td>80.52<sup>±0.43</sup></td><td><strong>45.66</strong><sup>±1.59</sup></td><td><strong>89.42</strong><sup>±0.0</sup></td></tr><tr><td>COMB</td><td>LEM</td><td>pred. POS+LEM</td><td>pred. POS</td><td>81.31<sup>±0.05</sup></td><td>20.12<sup>±0.89</sup></td><td>82.16<sup>±0.0</sup></td></tr></tbody></table>

<table><caption>Table 4: Dialect-Specific Lemmatization Accuracy on Occitan</caption><thead><tr><th rowspan="2">Tool</th><th rowspan="2">Train</th><th colspan="3">Gascon</th><th colspan="3">Lemosin</th></tr><tr><th>ALL</th><th>UNK</th><th>AMB</th><th>ALL</th><th>UNK</th><th>AMB</th></tr></thead><tbody><tr><td>MaChAmp</td><td>L+S</td><td>89.66<sup>±0.52</sup></td><td>57.01<sup>±1.24</sup></td><td>90.28<sup>±0.57</sup></td><td>MaChAmp</td><td>L+S</td><td><strong>90.91</strong><sup>±0.2</sup></td><td><strong>74.42</strong><sup>±1.9</sup></td><td>94.35<sup>±0.46</sup></td></tr><tr><td>MaChAmp</td><td>L+GAS</td><td>88.86<sup>±0.41</sup></td><td>54.38<sup>±1.24</sup></td><td>89.58<sup>±0.98</sup></td><td>MaChAmp</td><td>L+LEM</td><td>87.64<sup>±0.57</sup></td><td>64.34<sup>±1.1</sup></td><td>92.66<sup>±0.8</sup></td></tr><tr><td>Stanza</td><td>SMALL</td><td><strong>90.71</strong><sup>±0.75</sup></td><td><strong>77.78</strong><sup>±2.79</sup></td><td><strong>91.49</strong><sup>±0.0</sup></td><td>Stanza</td><td>SMALL</td><td>90.59<sup>±0.41</sup></td><td>72.6<sup>±0.8</sup></td><td><strong>99.22</strong><sup>±0.0</sup></td></tr><tr><td>Stanza</td><td>COMB</td><td>90.06<sup>±0.11</sup></td><td>67.54<sup>±1.24</sup></td><td>89.58<sup>±0.0</sup></td><td>Stanza</td><td>COMB</td><td>89.79<sup>±0.23</sup></td><td>66.67<sup>±1.09</sup></td><td>92.66<sup>±0.0</sup></td></tr></tbody></table><table><thead><tr><th rowspan="2">Tool</th><th rowspan="2">Train</th><th colspan="3">Lengadocian</th><th colspan="3">Provençau</th></tr><tr><th>ALL</th><th>UNK</th><th>AMB</th><th>ALL</th><th>UNK</th><th>AMB</th></tr></thead><tbody><tr><td>MaChAmp</td><td>L+S</td><td>93.08<sup>±0.48</sup></td><td>69.91<sup>±0.33</sup></td><td>92.76<sup>±0.69</sup></td><td>MaChAmp</td><td>L+S</td><td>91.67<sup>±0.0</sup></td><td>54.67<sup>±1.89</sup></td><td>95.14<sup>±0.44</sup></td></tr><tr><td>MaChAmp</td><td>L+LEN</td><td>92.56<sup>±0.6</sup></td><td>68.29<sup>±0.33</sup></td><td>92.29<sup>±0.8</sup></td><td>MaChAmp</td><td>L+PRO</td><td>86.6<sup>±0.11</sup></td><td>52.0<sup>±0.0</sup></td><td>89.55<sup>±0.25</sup></td></tr><tr><td>Stanza</td><td>SMALL</td><td><strong>94.42</strong><sup>±0.13</sup></td><td><strong>81.35</strong><sup>±0.9</sup></td><td><strong>96.54</strong><sup>±0.0</sup></td><td>Stanza</td><td>SMALL</td><td><strong>92.81</strong><sup>±0.31</sup></td><td><strong>74.92</strong><sup>±1.28</sup></td><td><strong>98.51</strong><sup>±0.0</sup></td></tr><tr><td>Stanza</td><td>COMB</td><td>93.72<sup>±0.11</sup></td><td>71.53<sup>±1.5</sup></td><td>92.98<sup>±0.0</sup></td><td>Stanza</td><td>COMB</td><td>92.08<sup>±0.12</sup></td><td>54.67<sup>±1.89</sup></td><td>93.51<sup>±0.0</sup></td></tr></tbody></table>

## 7.2 Dialect-Specific Results

When testing on individual dialects, too, the sequential approach of the Stanza model most often yields a higher accuracy for both Low Saxon and Occitan. As in case of the general tests, we do not find the automatically annotated data to benefit the model performance on Occitan. However, for both North and South German Low Saxon, we observed an improvement of the overall accuracy. Furthermore, we find MaChAmp to generalise particularly well to Lemosin.

When comparing the performance of the general and the dialect-specific MaChAmp models, the finetuning on a small dialect-specific dataset does not bring any improvement except for unknown tokens in German North Low Saxon. The MaChAmp models in fact consistently show a better overall accuracy when finetuned on the general gold train data. Since the general gold train data combines all the dialect-specific train sets, it is reasonable to suppose that these results are driven by the size difference between the finetuning datasets.

For Stanza, a more focused approach – here exclusive training on gold data without adding automatically annotated data – leads to a higher accuracy for lemmatizing unknown tokens. This holds true for both Low Saxon and Occitan, with the exception of Lemosin.

## 8 Discussion and Conclusion

The overall accuracy results for Low Saxon are noticeably lower than for Occitan, around 10% on average. One possible explanation could be the greater orthographic variation that is likely the reason behind the higher percentages of unknown and ambiguous tokens in Low Saxon seen in Table 2. While our Occitan corpus makes use of the same spelling convention throughout, the Low Saxon corpus contains various writing systems even within the same dialect group. Furthermore, we trained the models for Occitan on major dialects, whereas we used groups of major dialects for Low Saxon. Another reason might be found in the different diachronic structure of the datasets: Whereas the Occitan data mostly comes from the 20th and 21st century, the Low Saxon dataset covers the period
<table><caption>Table 5: Dialect-Specific Lemmatization Accuracy for Low Saxon</caption><thead><tr><th colspan="5">Dutch Low Saxon</th><th colspan="5">(German) North Low Saxon</th></tr><tr><th>Tool</th><th>Train</th><th>All</th><th>Unk</th><th>Amb</th><th>Tool</th><th>Train</th><th>All</th><th>Unk</th><th>Amb</th></tr></thead><tbody><tr><td>MaChAmp</td><td>L+S</td><td>77.46<sup>±0.24</sup></td><td>11.11<sup>±1.13</sup></td><td>82.39<sup>±0.22</sup></td><td>MaChAmp</td><td>L+S</td><td>86.77<sup>±0.8</sup></td><td>30.55<sup>±3.93</sup></td><td><strong>90.35</strong><sup>±0.62</sup></td></tr><tr><td>MaChAmp</td><td>L+DLS</td><td>76.31<sup>±0.23</sup></td><td>10.65<sup>±0.66</sup></td><td>81.16<sup>±0.08</sup></td><td>MaChAmp</td><td>L+NLS</td><td>82.65<sup>±0.32</sup></td><td><strong>33.33</strong><sup>±6.81</sup></td><td>85.35<sup>±0.79</sup></td></tr><tr><td>Stanza</td><td>SMALL</td><td><strong>80.41</strong><sup>±0.81</sup></td><td><strong>21.3</strong><sup>±4.72</sup></td><td><strong>84.45</strong><sup>±0.66</sup></td><td>Stanza</td><td>SMALL</td><td>84.79<sup>±0.92</sup></td><td><strong>33.33</strong><sup>±6.81</sup></td><td>89.01<sup>±1.08</sup></td></tr><tr><td>Stanza</td><td>COMB</td><td>78.93<sup>±0.11</sup></td><td>14.35<sup>±1.31</sup></td><td>81.98<sup>±0.0</sup></td><td>Stanza</td><td>COMB</td><td><strong>89.6</strong><sup>±0.12</sup></td><td>30.55<sup>±3.93</sup></td><td>89.01<sup>±0.0</sup></td></tr></tbody></table><table><thead><tr><th colspan="5">(German) South Low Saxon</th></tr><tr><th>Tool</th><th>Train</th><th>All</th><th>Unk</th><th>Amb</th></tr></thead><tbody><tr><td>MaChAmp</td><td>L+S</td><td>73.97<sup>±0.22</sup></td><td>45.45<sup>±0.00</sup></td><td>74.49<sup>±0.26</sup></td></tr><tr><td>MaChAmp</td><td>L+SLS</td><td>72.74<sup>±0.54</sup></td><td>42.42<sup>±4.29</sup></td><td>73.57<sup>±0.62</sup></td></tr><tr><td>Stanza</td><td>SMALL</td><td>78.15<sup>±0.56</sup></td><td><strong>46.97</strong><sup>±2.14</sup></td><td><strong>79.42</strong><sup>±1.22</sup></td></tr><tr><td>Stanza</td><td>COMB</td><td><strong>79.68</strong><sup>±0.08</sup></td><td>33.33<sup>±2.14</sup></td><td>78.44<sup>±0.0</sup></td></tr></tbody></table>

from the 19th century to the 21st.

This greater variation might be the reason why a sequential approach proves particularly useful for Low Saxon. As a result of the dialectal and orthographic variation, there are many ambiguous tokens that need to be lemmatized differently depending on the writing system and dialect. For instance, the character string *doe* typically refers to the feminine or masculine definite article in eastern Westphalian, where it should be lemmatized as *de*, whereas it should be lemmatized as *du* in Gronings, where it represents the pronoun of the second person singular. In addition, this string can stand for the 1st person singular in the present tense of the verb *doon* ‘to do’ in many dialects throughout the language area. PoS-tagging effectively disambiguates these three usages.

When it comes to Occitan, we noted that the Stanza model trained only on the gold data performs better than its counterpart trained on both preannotated and gold data. This may be due to the genre mismatch between the gold corpus (which is predominantly literary) and the automatically annotated corpus (which is extracted from Wikipedia). MaChAmp’s finetuning approach seems to be more robust to this, since the model trained on both preannotated and gold data achieves better general results than the one limited to the gold dataset.

The different model behaviour we have observed in our two low-resourced languages also warrants a more general question: How faithfully can low-resource scenarios be simulated by using small amounts of data from standardized high-resource languages? As this seems to be a relatively common practice, it would be worth investigating how this approach actually compares to the task it is supposed to simulate.

In conclusion, we found that the sequential approach implemented by Stanza was a good fit for both languages. The amount of training data also seemed to have more of an impact than dialect-level specificity, given that the MaChAmp models finetuned on the full gold dataset systematically outperformed the dialect-specific models. Another common tendency for both languages is the positive effect of using only gold data for training on the performance of the Stanza model over unknown tokens. This is a particularly interesting finding because it could be expected that a larger amount of training would make the model generalize better. It seems that in our case the reliability of the training data was more important.

## Data Access

The new annotated corpora created as part of this work are distributed on Zenodo.

The datasets for Low Saxon are available here: https://doi.org/10.5281/zenodo.7777282.

The large dataset for Occitan is available here: https://doi.org/10.5281/zenodo.7777340.

## Limitations

The MaChAmp and Stanza results are not fully comparable as we did not present the performance of dialect-specific Stanza models here. Since Stanza does not allow finetuning, we do not expect the small individual dialect-specific train sets to have a strong effect compared with the much larger amount of automatically annotated data. We defer testing this hypothesis to future work.
Acknowledgements

This work has been supported by the Academy of
Finland through project No. 342859 "CorCoDial –
Corpus-based computational dialectology".

References

Pierre Bec. 1970. *Manuel pratique de philologie romane*, volume Vol. 1. Picard.

Pierre Bec. 1995. *La langue occitane*, 6th edition. PUF.

Toms Bergmanis and Sharon Goldwater. 2018. Context sensitive neural lemmatization with Lematus. In *Proceedings of the 2018 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long Papers)*, pages 1391–1400, New Orleans, Louisiana. Association for Computational Linguistics.

Henk Bloemhoff, Philomèle Bloemhoff de Bruijn,
Jan Nijen Twilhaar, Henk Nijkeuter, and Harrie
Scholtmeijer. 2019. *Nedersaksisch in een notendop
– Inleiding in de Nedersaksische taal en literatuur*.
Koninklijke Van Gorcum, Assen.

Myriam Bras, Marianne Vergez-Couret, Nabil Hathout, Jean Sibille, Aure Séguier, and Benazet Dazéas. 2020. *Loflòc : Lexic obèrt flechit occitan. In Fidélités et dissidences (Actes du XIIe congrès de l'Association Internationale d'Études Occitanes)*, Albi. Centre d'Etude de la Littérature Occitane.

Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. 2019. BERT: Pre-training of deep bidirectional transformers for language understanding. In *Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long and Short Papers)*, pages 4171–4186, Minneapolis, Minnesota. Association for Computational Linguistics.

Georgiana Dinu Grzegorz Chrupala and Josef van Genabith. 2008. Learning morphology with morfette. In *Proceedings of the Sixth International Conference on Language Resources and Evaluation (LREC'08)*, Marrakech, Morocco. European Language Resources Association (ELRA). Http://www.lrec-conf.org/proceedings/lrec2008/.

Daniel Kondratyuk, Tomáš Gavenčiak, Milan Straka, and Jan Hajič. 2018. LemmaTag: Jointly tagging and lemmatizing for morphologically rich languages with BRNNs. In *Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing*, pages 4921–4928, Brussels, Belgium. Association for Computational Linguistics.

Alfred Lameli. 2016. Raumstrukturen im Niederdeutschen. Eine Re-Analyse der Wenkerdaten. *Niederdeutsches Jahrbuch. Jahrbuch des Vereins für niederdeutsche Sprachforschung*, 139:131–152.

Agathe. Lasch. 1974. *Mittelniederdeutsche Grammatik*. Sammlung kurzer Grammatiken germanischer Di-alekte ; 9. Niemeyer, Halle a. S.

Agathe Lasch, Conrad Borchling, Gerhard Cordes, Di-
eter Möhn, Ingrid Schröder, Jürgen Meier, and Sabina
Tsapaeva. *Mittelniederdeutsches Handwörterbuch*.

Wolfgang L window, Dieter Möhn, Hermann Niebaum, Dieter Stellmacher, Hans Taubken, and Jan Wirrer. 1998. *Niederdeutsche Grammatik*. Schuster, Leer.

Nikola Ljubešić and Kaja Dobrovoljc. 2019. What does neural bring? analysing improvements in morphosyn-tactic annotation and lemmatisation of Slovenian, Croatian and Serbian. In *Proceedings of the 7th Workshop on Balto-Slavic Natural Language Pro-cessing*, pages 29–34, Florence, Italy. Association for Computational Linguistics.

August. Lübben. 1995 - 1888. *Mittelniederdeutsches Handwörterbuch*. Wissenschaftliche Buchgesellschaft, Darmstadt.

Enrique Manjavacas, Ákos Kádár, and Mike Kestemont. 2019. Improving lemmatization of non-standard languages with joint learning. In *Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long and Short Papers)*, pages 1493–1503, Minneapolis, Minnesota. Association for Computational Linguistics.

Aleksandra Miletic, Myriam Bras, Marianne Vergez-Couret, Louise Esher, Clamença Poujade, and Jean Sibille. 2020. A four-dialect treebank for Occitan: Building process and parsing experiments. In *Proceedings of the 7th Workshop on NLP for Similar Languages, Varieties and Dialects*, pages 140–149, Barcelona, Spain (Online). International Committee on Computational Linguistics (ICCL).

Christopher Moseley, editor. 2010. *Atlas of the World's Languages in Danger*, 3 edition. UNESCO Publishing, Paris. Online version: http://www.unesco.org/culture/en/endangeredlanguages/atlas.

Thomas Müller, Ryan Cotterell, Alexander Fraser, and Hinrich Schütze. 2015. Joint lemmatization and morphological tagging with lemming. In *Proceedings of the 2015 Conference on Empirical Methods in Natural Language Processing*, pages 2268–2274, Lisbon, Portugal. Association for Computational Linguistics.

Hermann Niebaum. 2008. Het Nederduits. In Henk Bloemhoff, Jurjen van der Kooi, Hermann Niebaum, and Siemon Reker, editors, *Handboek Nedersaksische Taal- en Letterkunde*, pages 430–447. Koninklijke Van Gorcum, Assen.
Robert Peters. 2017. Das referenzkorpus mittel-
niederdeutsch/ niederrheinisch (1200–1650).
*Niederdeutsches Jahrbuch. Jahrbuch des Vereins für
niederdeutsche Sprachforschung*, 140:35–42.

Peng Qi, Yuhao Zhang, Yuhui Zhang, Jason Bolton, and Christopher D. Manning. 2020. Stanza: A Python natural language processing toolkit for many human languages. In *Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics: System Demonstrations*.

Kumar Saunack, Kumar Saurav, and Pushpak Bhat-
tacharyya. 2021. How low is too low? a monolingual
take on lemmatisation in Indian languages. In *Proceed-
ings of the 2021 Conference of the North Amer-
ican Chapter of the Association for Computational
Linguistics: Human Language Technologies*, pages
4088–4094, Online. Association for Computational
Linguistics.

Kumar Saurav, Kumar Saunack, and Pushpak Bhat-
tacharyya. 2020. Analysing cross-lingual transfer
in lemmatisation for Indian languages. In *Proceed-
ings of the 28th International Conference on Com-
putational Linguistics*, pages 6070–6076, Barcelona,
Spain (Online). International Committee on Compu-
tational Linguistics.

Ingrid Schröder. 2004. Niederdeutsch in der Gegenwart: Sprachgebiet – Grammatisches – Binnen-differenzierung. In Dieter Stellmacher, editor, *Niederdeutsche Sprache und Literatur der Gegenwart*, pages 35–97. Georg Olms Verlag, Hildesheim, Zürich and New York.

Jean Sibille. 2002. *Ecrire l'occitan : essai de présen-
tation et de synthèse*. In *Les langues de France
et leur codification. Ecrits divers – Ecrits ouverts*,
Paris, France. Inalco / Association Universitaire des
Langues de France, L'Harmattan.

Janine Siewert, Yves Scherrer, and Martijn Wieling. 2022. Low Saxon dialect distances at the orthographic and syntactic level. In *Proceedings of the 3rd Workshop on Computational Approaches to Historical Language Change*, pages 119–124, Dublin, Ireland. Association for Computational Linguistics.

Janine Siewert, Yves Scherrer, Martijn Wieling, and Jörg Tiedemann. 2020. LSDC - a comprehensive dataset for low Saxon dialect classification. In *Proceedings of the 7th Workshop on NLP for Similar Languages, Varieties and Dialects*, pages 25–35, Barcelona, Spain (Online). International Committee on Computational Linguistics (ICCL).

Dieter Stellmacher. 1983. Neuniederdeutsche Gram-
matik – Phonologie und Morphologie. In Gerhard
Cordes and Dieter Möhn, editors, *Handbuch zur
niederdeutschen Sprach- und Literaturwissenschaft*,
pages 238–278. Erich Schmidt Verlag, Berlin, Ger-
many.

Rob van der Goot, Ahmet Üstün, Alan Ramponi,
Ibrahim Sharaf, and Barbara Plank. 2021. Massive
choice, ample tasks (MaChAmp): A toolkit for multi-
task learning in NLP. In *Proceedings of the 16th
Conference of the European Chapter of the Associa-
tion for Computational Linguistics: System Demon-
strations*, pages 176–197, Online. Association for
Computational Linguistics.

Alessandro Vatri and Barbara McGillivray. 2020. Lemmatization for ancient greek: An experimental assessment of the state of the art. *Journal of Greek linguistics*, 20(2):179–196.

Marianne Vergez-Couret. 2016. Description du lexique Loflòc. Research report, CLLE-ERSS.
A Complementary Evaluation Results

<table><caption>Table 6: Global Lemmatization Accuracy for Occitan and Low Saxon. *Dev* set.</caption><thead><tr><th>Tool</th><th>Training set</th><th>Task</th><th>Train cond.</th><th>Test cond.</th><th>ALL</th><th>UNK</th><th>AMB</th></tr></thead><tbody><tr><td rowspan="6">Occitan</td><td rowspan="6">MaChAmp<br>Stanza</td><td>SMALL</td><td>LEM</td><td>no POS, gold LEM</td><td>no POS</td><td>93.57<sup>±0.06</sup></td><td>78.74<sup>±1.14</sup></td><td>95.08<sup>±0.28</sup></td></tr><tr><td>LARGE</td><td>POS+LEM</td><td>pred. POS+LEM</td><td>no POS</td><td>93.32<sup>±0.09</sup></td><td>76.07<sup>±0.5</sup></td><td>91.94<sup>±0.15</sup></td></tr><tr><td>L+S</td><td>POS+LEM</td><td>pred. POS+LEM</td><td>no POS</td><td>94.24<sup>±0.17</sup></td><td>73.49<sup>±0.74</sup></td><td>93.47<sup>±0.29</sup></td></tr><tr><td>SMALL</td><td>LEM</td><td>no POS</td><td>no POS</td><td>92.84<sup>±0.14</sup></td><td>75.43<sup>±0.84</sup></td><td>93.1<sup>±0.0</sup></td></tr><tr><td>SMALL</td><td>LEM</td><td>gold POS+LEM</td><td>pred. POS</td><td>94.68<sup>±0.03</sup></td><td>83.16<sup>±0.21</sup></td><td>94.86<sup>±0.0</sup></td></tr><tr><td>COMB</td><td>LEM</td><td>pred. POS+LEM</td><td>pred. POS</td><td>93.53<sup>±0.06</sup></td><td>74.01<sup>±1.11</sup></td><td>91.19<sup>±0.0</sup></td></tr><tr><td rowspan="6">Low Saxon</td><td rowspan="6">MaChAmp<br>Stanza</td><td>SMALL</td><td>LEM</td><td>no POS, gold LEM</td><td>no POS</td><td>74.72<sup>±0.62</sup></td><td>25.48<sup>±1.67</sup></td><td>91.84<sup>±0.44</sup></td></tr><tr><td>LARGE</td><td>POS+LEM</td><td>pred. POS+LEM</td><td>no POS</td><td>86.69<sup>±0.31</sup></td><td>52.01<sup>±0.89</sup></td><td>89.14<sup>±0.46</sup></td></tr><tr><td>L+S</td><td>POS+LEM</td><td>pred. POS+LEM</td><td>no POS</td><td>81.64<sup>±0.43</sup></td><td>56.74<sup>±0.58</sup></td><td>83.41<sup>±0.38</sup></td></tr><tr><td>SMALL</td><td>LEM</td><td>no POS</td><td>no POS</td><td>78.7<sup>±0.23</sup></td><td>42.38<sup>±0.9</sup></td><td>85.09<sup>±0.0</sup></td></tr><tr><td>SMALL</td><td>LEM</td><td>gold POS+LEM</td><td>pred. POS</td><td>82.4<sup>±0.16</sup></td><td>47.64<sup>±0.61</sup></td><td>92.15<sup>±0.0</sup></td></tr><tr><td>COMB</td><td>LEM</td><td>pred. POS+LEM</td><td>pred. POS</td><td>83.54<sup>±0.17</sup></td><td>55.79<sup>±2.34</sup></td><td>83.95<sup>±0.0</sup></td></tr></tbody></table>

<table><caption>Table 7: Dialect-Specific Lemmatization Accuracy on Occitan. *Dev* set (there are no dialect-specific *dev* sets for Lemosin and Provençau.)</caption><thead><tr><th rowspan="2">Tool</th><th rowspan="2">Train</th><th colspan="3">Gascon</th><th colspan="3">Lengadocian</th></tr><tr><th>ALL</th><th>UNK</th><th>AMB</th><th>ALL</th><th>UNK</th><th>AMB</th></tr></thead><tbody><tr><td>MaChAmp</td><td>L+S</td><td>93.6<sup>±0.36</sup></td><td>69.1<sup>±1.15</sup></td><td>93.55<sup>±1.11</sup></td><td>MaChAmp</td><td>L+S</td><td>94.4<sup>±0.14</sup></td><td>75.58<sup>±0.95</sup></td><td>93.45<sup>±0.1</sup></td></tr><tr><td>MaChAmp</td><td>L+GAS</td><td>92.83<sup>±0.1</sup></td><td>69.1<sup>±2.3</sup></td><td>92.14<sup>±0.22</sup></td><td>MaChAmp</td><td>L+LEN</td><td>94.12<sup>±0.13</sup></td><td>74.03<sup>±1.09</sup></td><td>93.25<sup>±0.11</sup></td></tr><tr><td>Stanza</td><td>SMALL</td><td>94.29<sup>±0.2</sup></td><td>79.65<sup>±0.99</sup></td><td>96.15<sup>±0.0</sup></td><td>Stanza</td><td>SMALL</td><td>94.78<sup>±0.02</sup></td><td>84.29<sup>±0.16</sup></td><td>94.51<sup>±0.0</sup></td></tr><tr><td>Stanza</td><td>COMB</td><td>90.4<sup>±0.0</sup></td><td>68.29<sup>±0.0</sup></td><td>87.26<sup>±0.0</sup></td><td>Stanza</td><td>COMB</td><td>94.33<sup>±0.08</sup></td><td>76.75<sup>±1.65</sup></td><td>92.16<sup>±0.0</sup></td></tr></tbody></table>

<table><caption>Table 8: Dialect-Specific Lemmatization Accuracy for Low Saxon. *Dev* set.</caption><thead><tr><th rowspan="2">Tool</th><th rowspan="2">Train</th><th colspan="3">Dutch Low Saxon</th><th colspan="3">(German) North Low Saxon</th></tr><tr><th>All</th><th>Unk</th><th>Amb</th><th>All</th><th>Unk</th><th>Amb</th></tr></thead><tbody><tr><td>MaChAmp</td><td>L+S</td><td>83.64<sup>±0.75</sup></td><td>60.0<sup>±1.26</sup></td><td>86.57<sup>±0.61</sup></td><td>MaChAmp</td><td>L+S</td><td>84.49<sup>±0.46</sup></td><td>60.49<sup>±1.74</sup></td><td>86.7<sup>±0.49</sup></td></tr><tr><td>MaChAmp</td><td>L+DLS</td><td>81.35<sup>±0.13</sup></td><td>52.31<sup>±0.00</sup></td><td>84.77<sup>±0.35</sup></td><td>MaChAmp</td><td>L+NLS</td><td>80.96<sup>±0.22</sup></td><td>60.49<sup>±3.49</sup></td><td>83.42<sup>±0.37</sup></td></tr><tr><td>Stanza</td><td>SMALL</td><td>84.2<sup>±0.5</sup></td><td>50.26<sup>±2.62</sup></td><td>87.55<sup>±0.11</sup></td><td>Stanza</td><td>SMALL</td><td>83.97<sup>±0.46</sup></td><td>45.68<sup>±1.75</sup></td><td>87.39<sup>±0.95</sup></td></tr><tr><td>Stanza</td><td>COMB</td><td>84.11<sup>±0.35</sup></td><td>55.9<sup>±3.84</sup></td><td>84.52<sup>±0.0</sup></td><td>Stanza</td><td>COMB</td><td>87.96<sup>±0.08</sup></td><td>65.43<sup>±1.75</sup></td><td>88.08<sup>±0.0</sup></td></tr></tbody></table><table><thead><tr><th rowspan="2">Tool</th><th rowspan="2">Train</th><th colspan="3">(German) South Low Saxon</th></tr><tr><th>All</th><th>Unk</th><th>Amb</th></tr></thead><tbody><tr><td>MaChAmp</td><td>L+S</td><td>76.82<sup>±0.07</sup></td><td>50.34<sup>±0.96</sup></td><td>77.63<sup>±0.11</sup></td></tr><tr><td>MaChAmp</td><td>L+SLS</td><td>74.25<sup>±0.71</sup></td><td>48.3<sup>±1.92</sup></td><td>74.68<sup>±0.88</sup></td></tr><tr><td>Stanza</td><td>SMALL</td><td>78.96<sup>±0.34</sup></td><td>42.86<sup>±0.0</sup></td><td>81.41<sup>±0.49</sup></td></tr><tr><td>Stanza</td><td>COMB</td><td>78.91<sup>±0.15</sup></td><td>50.34<sup>±1.92</sup></td><td>79.82<sup>±0.0</sup></td></tr></tbody></table>