Latin Treebanks in Review: An Evaluation of Morphological Tagging Across Time (2024)

Abstract: Existing Latin treebanks draw from Latin’s long written tradition, spanning 17 centuries and a variety of cultures. Recent efforts have begun to harmonize these treebanks’ annotations to better train and evaluate morphological taggers. However, the heterogeneity of these treebanks must be carefully considered to build effective and reliable data. In this work, we review existing Latin treebanks to identify the texts they draw from, identify their overlap, and document their coverage across time and genre. We additionally design automated conversions of their morphological feature annotations into the conventions of standard Latin grammar. From this, we build new time-period data splits that draw from the existing treebanks which we use to perform a broad cross-time analysis for POS and morphological feature tagging. We find that BERT-based taggers outperform existing taggers while also being more robust to cross-domain shifts.

heading: 1 Introduction
text: Large-scale digitized Latin archives now document cultures across many centuries in wide a variety of genres from literature to legal documents. With increasingly powerful Latin natural language processing tools (e.g. Bamman and Burns, 2020; Burns, 2023), morphological feature tagging is a promising method for Latin-based computational humanities. The goal of morphological tagging is to identify a set of morphological feature-value pairs for each token of a given sentence. These features can help researchers analyze agency, power, and other morphosyntactically-signalled phenomena which have been fruitfully investigated in English (Sap et al., 2017; Greene and Resnik, 2009) and other languages (Rashkin et al., 2017). For example, Voice (active, passive verbs) and Case (e.g., nominative, accusative ablative nouns) are useful for studying power and agency.
Although Latin taggers have relatively good performance, in our experience they often perform poorly on rarer feature values—such as passive voice–that may prove crucial for downstream analyses. Toward this end, we hope to develop a Latin morphological tagger whose accuracy is robust across time and genre by leveraging the recent development of five separate Latin Universal Dependencies (UD; de Marneffe et al., 2021) treebanks and recent efforts to harmonize their morphological tags (Gamba and Zeman, 2023a). In this work we review these harmonized treebanks1 plus the non-UD treebank LASLA (Denooz, 2004), and conclude that more data curation is required to fully evaluate and improve morphological tagging’s cross-domain accuracy.
Our contributions include: 1) precisely documenting genre and historical context for the 544 texts within the UD treebanks as a machinereadable, cross-treebank resource that will enable
1Perseus (Bamman and Crane, 2011), PROIEL (Haug and Jøhndal, 2008), LLCT (Cecchini et al., 2020b), ITTB (Passarotti, 2019), and UDante (Cecchini et al., 2020a)
203
future work to examine morphosyntactic association against these variables; 2) harmonizing the UD and LASLA treebanks to reduce annotation differences that can affect training; 3) proposing edits to the UD tagset that better align with standard analyses of Latin grammar to facilitate work by researchers with standard Latin training; and 4) conducting a cross-time analysis with experimental results broken down by historical period that show the promise of our harmonization efforts and BERT-based morphological taggers.2

heading: 2 Latin Treebanks Revisited

heading: 2.1 Time and Genre Metadata
text: Detailed metadata on the texts included in the Latin UD treebanks is difficult to aggregate or lacking altogether. Information on the included works’ time period, genre, author, and relative size has not been compiled in one place. Our work takes major steps to fill this gap. For all 544 texts across the five UD treebanks, we manually collected the following metadata: the source treebank, time period, century, internal treebank identifiers, cumulative and split-level sentence counts, author, and exhaustive genre labels.
Genre. Figure 2 shows the genre coverage of the UD treebanks. Previous EvaLatin campaigns (Sprugnoli et al., 2020, 2022) have implicitly defined several genres (prose, poetry, epics, and histories), which were then used to analyze cross-genre tagging accuracy on Classical era, non-UD data. We expand upon these genres by including more fine-grained labels and by covering non-Classical texts.
We annotate nine exclusive genres: short poems, epics, letters, histories, satires, speeches, legal texts, treatises, and the Bible.6
Time. We define the (approximate) century of each text (Figure 1 and 2). For cross-time analysis, we define three very broad time periods:
2We have publicly released our new text-level metadata, standardized morphologically tagged text from described treebanks, and conversion software on Github.
3Although unreleased, we determined the feature-value set by examining LatinCy’s outputs.
4EvaLatin 2020 also has annotated data that is not directly sourced from LASLA but consists of a subset of LASLA’s texts. This data is not annotated with morphological features.
5EvaLatin 2022 is a near-subset of LASLA because it has one non-Classical text that is not in LASLA.
6We also annotate additional non-exclusive genres (§A.1).
• Classical is defined as 3rd century BCE through the 2nd century CE, in line with conventional definitions of the Classical Latin language and periods (Sala and Posner, 2024) and previous Latin NLP literature (Sprugnoli et al., 2020, 2022).
• Biblical is defined as its own genre and time period, consisting solely of Jerome’s Vulgata from the 4th century CE. It is significantly different from other texts given it is a translation (from much earlier material), and has relatively simpler grammar (Nunn, 1922).
• Post-Classical is defined as 4th century CE and later, excluding the Bible, thus including Late and Medieval Latin texts. For simplicity, we do not split it further. Prior work in cross-time tagging either used a smaller set of time periods (Sprugnoli et al., 2022) or considered each UD treebank its own time period (Gamba and Zeman, 2023a), which we argue is too approximate given our metadata findings (§2.2).

heading: 2.2 UD Treebanks
text: Currently, there are five UD treebanks for Latin.7 Four of these—Perseus, PROIEL, LLCT, and ITTB—were automatically converted to UD format, while the fifth, UDante, was annotated directly in UD. Collectively, this corresponds to about 58,000 annotated sentences and 979,280 annotated tokens. As Figure 1 shows, these treebanks cover a wide range of time but far from evenly.
7In May 2024, a sixth, CIRCSE, was added; it is a subset of LASLA.
We find that the three Post-Classical treebanks (LLCT, ITTB, and UDante) are quite distinct from each other in terms of genre and time period. LLCT consists entirely of medieval legal charters from the 8th and 9th centuries. ITTB consists of three philosophical and religious works by Thomas Aquinas from the 13th century. Finally, UDante is comprised of Dante Alighieri’s 14th century Latin works, including treatises, letters, and poems.
The two remaining treebanks, Perseus and PROIEL, are more diverse. Most texts in Perseus are Classical, although 154 sentences are from Jerome’s Vulgata (the Book of Revelation). While PROIEL also includes Classical texts, the majority (11785) of its 18411 sentences are also taken from Jerome’s Vulgata. There is overlap between Perseus and PROIEL, as both share at least 145 sentences from the Book of Revelation.8 Aside from Classical and Biblical texts, PROIEL also includes one 4th-5th century work, Opus Agriculturae by Palladius.

heading: 2.3 LASLA: Additional Classical-era treebank
text: LASLA is a large, non-UD treebank for Latin (Denooz, 2004). By our own count, LASLA has 134 unique texts with 95,974 sentences and about 1.8M tokens.9 All texts are Classical. All genres included in UD are covered, in addition to plays. Unlike the UD treebanks, LASLA does not have dependency relations.
8See Table 8 for a breakdown of annotation agreement between these duplicate sentences.
9A full list of authors, works, and tokens per text is available here.

heading: 3 Harmonizing UD and LASLA Annotations
text: In this section, we describe steps taken to reduce the annotation differences between the Harmonized UD treebanks (Table 1 row 4) and LASLA (1 row 5). Throughout this section, we sometimes use "UD" as a shorthand for Gamba and Zeman (2023a)’s Harmonized UD treebanks.
In §3.1, we outline the annotation agreement between Harmonized UD and LASLA before any intervention on our part. Then, we describe two types of changes: harmonization (§3.2) and standardization (§3.3). During harmonization, we enforce consistency of arbitrary values to have fair training and evaluation. Standardization is more involved, where we change the grammatical system to be more Latin-specific. Both of these steps are done automatically and simultaneously through conversion scripts.

heading: 3.1 Annotation Agreement Between UD and LASLA
text: 10Ovid’s Metamorphoses appears in both treebanks, but they cover different books of the text.
UD and LASLA happen to have re-annotated many of the same sentences, which gives a way to analyze annotation agreement between the projects. We detect sentences that appear in both datasets (§A.2), finding 2607 such duplicates across eight Classical texts (Table 2), which may be an underestimate since our duplicate detector will miss cases where sentence segmentation or tokenization differ.
We calculate annotation agreement before and after harmonization and standardization on our reduced set of features (Table 3). Some features, such as Degree, Tense, and VerbForm, have low agreement due to mismatches between their possible value sets in UD and in LASLA. Other features, such as Gender, Person, and UPOS have low agreement due to remaining annotation differences.11

heading: 3.2 Our Harmonization Efforts
text: Gamba and Zeman (2023a) have already performed the bulk of the harmonization necessary for the UD treebanks. However, we are additionally attempting to harmonize LASLA with the UD treebanks.
Remaining inconsistencies we’ve harmonized. We have found some remaining inconsistencies, both within the UD treebanks and between UD and LASLA. Usually, neither is incorrect in their annotation, but without normalization this will cause unfair evaluation. Thus, we enforce consistent, arbitrary values in these cases. See §A.4 for specifics.
Collapsing feature values. Another issue we encountered is that some UD treebanks lack certain feature values that are present in the others. Gamba and Zeman (2023a) were aware of this issue, and chose not to harmonize these values in order to preserve as much information as possible. This is understandable, as these features may be of interest to researchers. However, for our purposes, we have collapsed certain feature values together in order to have fairer evaluation of models trained on different treebanks.
For UPOS (universal part of speech), we have collapsed INTJ into PART across all treebanks, since two UD treebanks (ITTB and LLCT) do not use INTJ, instead using the value PART. Additionally, for Degree, we have collapsed Degree=Pos into Degree=None, since LASLA is the only treebank to use Pos. The distinction between
11See appendix for agreement rates across all features (Table 9) and a comprehensive overview of the feature inventories (Table 12).
Degree=Pos and Degree=None is debated.12 We note that Gamba and Zeman (2023a) also collapsed Degree=Pos and Degree=Dim into Degree=None, so this decision has precedent.

heading: 3.3 Conversion to Standard Latin Grammar
text: UD was developed with cross-linguistic goals in mind, offering a set of universal tags applicable to all languages. However, prior to the harmonization efforts by Gamba and Zeman (2023a), many Latin UD treebanks employed standard Latin values for certain features, reflecting a long-standing desire for a more Latin-specific tagset. Harmonization and conversion to UD has relegated these Latinspecific values to a secondary status. This poses a key challenge for evaluation, as these two annotation styles are not comparable.
Although UD provides a valuable crosslinguistic framework, we believe Latin is also useful to study on its own, within long-standing approaches to Latin linguistics (e.g. Greenough and Allen 1903). The UD treebanks remain the most complete, high-quality source of morphological annotations for Latin. To bridge the gap between UD and standard Latin linguistics, we offer an alternative version that uses more standard Latin grammar. In particular, we standardize the treebanks to follow Pre-UD Perseus’s (Table 1 row 1) features: UPOS, person, number, tense, mood, voice, gender, case, and degree. This set is nearly identical to Burns
12See the UD documentation for Degree in Latin here.
(2023)’s, except that LatinCy separately predicts Mood and VerbForm (which we combine). For most of these features, UD has a corresponding feature that we can easily extract. The exceptions are Tense and Mood, where we developed a more elaborate method of standardization (§A.5). For example, Latin tense traditionally has six possible values (present, imperfect, perfect, future, pluperfect, future perfect) which are standardized across pedagogical materials (Greenough and Allen, 1903; Wheelock and LeFleur, 2010). However, UD’s Tense feature only includes four of these values (present, past, future, pluperfect), which is why we must perform a conversion.
We choose to convert to Standard Latin Grammar before training, rather than perform postprocessing on the predictions of a model trained on the UD tagset, for two reasons: 1) preprocessing allows for more precise conversions based on known treebank sources, addressing inconsistencies between treebanks, and 2) model predictions may combine features from various annotation schemes and be grammatically inconsistent, making postprocessing complex and potentially unreliable.

heading: 3.4 Remaining Inconsistencies
text: After our harmonization and standardization, most features have high annotation agreement between LASLA and UD (Table 3). Degree and UPOS, two features that already had low agreement within UD (Table 8), saw improved but not high agreement after UD-LASLA harmonization. These are likely due to fundamental differences in the annotation process which may require reannotation to fix.
We modify how our models are trained to account for the two following differences (§5.1):
• In LASLA, the Gender feature can take multiple values to represent possible genders based only on the word form (disregarding the context of the sentence). In the UD treebanks, Gender is assigned one value that depends on the sentence. This causes the low annotation agreement for Gender in Table 3. If we use a looser criterion—counting the annotations as the same when the UD gender value matches one of LASLA’s gender values—we do see higher agreement ((loose) in Table 3).
• In LASLA, personal pronouns are annotated with Person=None, but in the UD treebanks personal pronouns have non-None values.13
We list additional differences in §A.6.

heading: 3.5 Our Custom Data Splits
text: We create new data splits to emulate EvaLatin’s cross-time sub-task which evaluates models on texts of a different time period than what they are trained on. When creating train/test splits for each time period, we keep the following constraints in mind: 1) Individual works should be within a single split. For example, Ovid’s Metamorphoses should only appear in either the train or test set, rather than having a random sample of sentences in the train set with the rest in the test set. 2) Make sure the test set is large enough for reasonable statistical power. We specifically choose to have a minimum of 1000 sentences in each test set. 3) Only evaluate on UD data and not LASLA. Due to some annotation differences (see §3.4), UD treebanks have more complete information than LASLA. This is in contrast to EvaLatin campaigns which evaluate on subsets of LASLA.
To make our dev sets, we randomly sample 3% of sentences from each work in the train sets, making sure that we never sample from LASLA or any UD sentences that also appear in LASLA.
13This will be simple to fix in future work, since there are limited personal pronoun lemmas in Latin.
Due to these constraints, we are unable to keep the original UD test sets. Since we want test sets for each time period, we must construct Classicalspecific splits. Perseus, despite being largely Classical, is too small for effective training. PROIEL contains some Classical texts but is mostly comprised of Biblical texts. We separate the Biblical content and combine the Classical texts from both treebanks to ensure a sufficiently large Classical train set. Due to the first constraint, we cannot use ITTB’s original train/test splits since Summa Contra Gentiles appears in both the train and test set. To help meet our second constraint, we do not use UDante’s and LLCT’s original splits.
To achieve our third constraint, our Classical train set must include all works that appear in both LASLA and UD, shown in Table 2. We want to test two scenarios: training with and without LASLA data. In order to have enough training sentences in the UD-only scenario, we treat the letters of Cicero’s Letters to Atticus as separate texts (i.e. that can be distributed across the Classical train and test set), even though this conflicts with our first constraint.
We include a detailed description of which works appear in our custom train and test sets in the Appendix (Table 13).

heading: 4 Related Work: Morphological Tagging
text: There is a long history of work analyzing POS and morphological tagging of Latin (Eger et al., 2015, 2016; Straka and Straková, 2020). Our work follows recent trends of using transformer-based contextual representations.
Several recent papers have explored morphological tagging for Latin. As part of the 2022 EvaLatin feature identification task (Sprugnoli et al., 2022), participants trained and tested on a subset of data from the LASLA corpus that had been automatically converted to UD format (Wróbel and Nowak, 2022; Mercelis and Keersmaekers, 2022). Only a subset of UD morphological features were retained, partly to limit the task to morphological features identifiable by the word form, and partly to avoid features affected by annotation differences. Participants were then able to train on combined UD and LASLA data if they wished, but models were only evaluated on EvaLatin test sets, not UD test sets.
Nehrdich and Hellwig (2022) used LatinBERT (Bamman and Burns, 2020) to train a morphological tagger predicting the case, gender, number,
tense and verbform features. Its outputs were then fed into the authors’ dependency parser, outperforming prior work using UDPipe and static word embeddings (Straka et al., 2019). Their training and test data came from three UD treebanks (ITTB, PROIEL, and Perseus).
Burns (2023) developed LatinCy, a full NLP pipeline for Latin which includes morphological feature classification.14 Notably, this pipeline was trained on all five UD treebanks with early attempts made at harmonization, using a smaller tagset than UD that is closer to standard analyses of Latin grammar (Table 1 row 3). Recently, Gamba and Zeman (2023a) performed more rigorous harmonization of morphological features across the five UD Latin treebanks (Table 1 row 4). They reported accuracy before and after harmonization, training and testing on each pair of treebanks using fasttext embeddings (Grave et al., 2018) with UDPipe (Straka et al., 2016) or Stanza (Qi et al., 2020). Harmonization was shown to improve accuracy when training and testing on two different treebanks.
Part-of-speech (POS) tagging is closely related to morphological tagging. In the 2020 EvaLatin campaign, participants trained and tested POS taggers on a subset of the LASLA corpus (Sprugnoli et al., 2020). More recently, Riemenschneider and Frank pretrained a trilingual RoBERTa (Liu et al., 2019) model on English, Ancient Greek, and Latin which surpassed the 2022 EvaLatin competitors (Table 1 row 6). Thus, the current SOTA models for Latin POS tagging are all transformer-based. Additionally, Riemenschneider and Frank’s trilingual model underperformed their monolingual Ancient Greek model, suggesting a monolingual Latin model could prove even stronger, given sufficient pretraining data.
Researchers have also experimented with using GPT3.5-Turbo and GPT4 for POS tagging of 16th century Latin texts (Stüssi and Ströbel, 2024). No POS-annotated data exists for 16th century Latin, so the authors experimented with zero-shot prompting and finetuning using data from the five UD treebanks. Although the UD testsets are not entirely comparable with EvaLatin’s, the accuracy of these GPT-based approaches seems low when compared to the results of EvaLatin’s POS tagging shared task.
Although substantial progress has been made in Latin morphological tagging, gaps still exist. Aside
14Using SpaCy (Honnibal and Montani, 2017)
from Gamba and Zeman (2023a) and Burns (2023), prior work has not leveraged all five UD treebanks for training and evaluation. While Gamba and Zeman (2023a) measure overall tagging accuracy, more detailed analysis of specific morphological features and diachronic trends has been left to future work. Moreover, to our knowledge no recent paper has evaluated the currently available taggers on UD test data.

heading: 5 Experiments
text: We use three metrics in our evaluations: whole string morphological accuracy, macro F1 for individual features, and F1 for particular feature-values. See A.8 for more detailed explanations.

heading: 5.1 Our LatinBERT-based Tagger
text: Following other recent working finding SOTA performance with transformer-based taggers (Sprugnoli et al., 2022; Riemenschneider and Frank, 2023), we finetune a tagger on top of LatinBERT (Bamman and Burns, 2020). Similar to Riemenschneider and Frank (2023), our tagger uses a separate classification head for every morphological feature, all trained simultaneously—a simple choice which could be improved upon in future work.
When training on LASLA, we sometimes do not train a particular feature head based on a token’s feature values. First, if Gender has multiple values we do not train the Gender prediction head. We want to keep our set of possible Gender values limited to the standard three (Masc, Fem, and Neut). Second, if the token is a personal pronoun and Person=None, we do not train the Person prediction head. Having a null value here is inconsistent with the rest of our data. Since we do not know the true value, we choose not to train in this instance. If either of these two cases apply to a particular token, then that token will not contribute to the loss for either the Gender or Person classifier head. Other heads are unaffected.

heading: 5.2 Comparison to Previous Taggers
text: In this section, we use the official UD train/test splits for comparison to previous work but converted to our harmonized and standardized tagset. We compare our BERT taggers to two sets of taggers previously evaluated on UD data: LatinCy (Table 1 row 3) and five Stanza models trained on the five Harmonized UD treebanks (Table 1 row 4). LatinCy uses a non-transformer neural architecture as part of the SpaCy pipeline, along with
static floret vectors (Boyd and Warmerdam, 2022). Stanza has a Bi-LSTM architecture for its POS and morphological taggers and uses either word2vec (Zeman et al., 2018) or fasttext (Bojanowski et al., 2017) embeddings, depending on the language. For a fair comparison, we must convert between the different tagsets used by each tagger. For LatinCy, rather than retraining the SpaCy pipeline ourselves, we convert its predictions on each official UD test set to our tagset. This required little modification as LatinCy predicts a near-identical set of features and values.15 For the Stanza models, we retrain them on our harmonized and standardized versions of each UD training set (Table 1 row 8), since Gamba and Zeman (2023b)’s models and their predictions are unreleased. Replicating Gamba and Zeman (2023b), we only train the Stanza models on each individual treebank, rather than all UD data. We also use the same Latin fasttext embeddings (Grave et al., 2018) and default training parameters.
We report whole string morphological accuracy for each UD test set in Table 5. Our BERT taggers consistently have the highest accuracy. The smallest treebanks, Perseus and UDante, see the most benefit from the BERT architecture and the out-of-domain training data.16
15LatinCy lacks two possible Tense values, Perf and FutP, which our tagset includes. In a more generous evaluation, where Fut and Imp are considered correct predictions for gold FutP and Perf, respectively, all morphological accuracy scores in Table 5 increase by ≤ 5%, with maximum accuracy on the LLCT test set at 0.826.
16We see similar trends for UPOS; see Table 10.
When comparing two models’ performance, we calculate statistical significance via randomized permutation testing (Wasserman 2004).17 When comparing our All-UD model to LatinCy and our in-domain models to Stanza, all comparisons were significant (p=0) for both UPOS Macro-F1 and morphological accuracy, except for LLCT UPOS Macro-F1 (p=0.13). So, in nearly all cases our BERT taggers performed significantly better at both UPOS and morphological tagging than previously released taggers, when trained on all or in-domain data.
We also find that our BERT taggers are more robust to out-of-domain data than the Stanza taggers. In Table 6, for each UD test set, we report the average difference between the in-domain test performance (training and testing on the same treebank) and out-of-domain test performance (training on a different treebank). This difference is always lower for our BERT models than for the Stanza models, suggesting that BERT has better cross-domain performance than Stanza.

heading: 5.3 Performance on Our Custom Splits
text: In total, we train six models including four trained on the sets described in Table 4. The other two models are all-ud-custom, trained on the Classical (UD Only), Bible, and Postclass train sets; and all-both-custom, trained on the Classical (UD+LASLA), Bible, and Postclass train sets. Since our LatinBERT taggers outperform the other taggers, we limit our focus to these BERT-based taggers. We find that it is generally unneces-
17As detailed in §A.9, we simply report p-values based on 10,000 null simulations; thus p=0 is possible and could be more conservatively interpreted as p < .0003 (“rule of three”: Eypasch et al. 1995).
sary to train a period-specific model. As Table 7 shows, models trained on all time periods have only slightly reduced UPOS Macro F1 and have slightly increased morphological accuracy compared to the models trained on a single domain.
Addition of LASLA data boosts performance for some rare feature values, but decreases it for other features. Although there is only a slight difference in overall morphological accuracy with the addition of LASLA data, F1 of particular feature-values improves. When evaluating the classical-ud and classical-all models on the Classical test set, F1 increases from 0.907 to 0.941 for Case=Dat (p=0.0028), and 0.800 to 0.909 for Mood=Ger (p=0.0). We also found that some features’ Macro F1 scores decreased with the inclusion of LASLA. This behavior is most prominent for Degree (0.96 to 0.91, p=0.0001) and UPOS (0.96 to 0.95, p=0.0). Since the duplicate sentences in LASLA and UD have low annotation agreement for Degree and UPOS (Table 3), the addition of LASLA data likely led to noisier training labels for these two features.
Most errors involve acontextual ambiguity. We randomly sample 100 tokens whose morphology was predicted incorrectly by our all-ud-custom model,18 and annotated them according to six error types: illegal, lexical, genuine acontextual ambiguity, annotation differences, gold wrong, other.
Illegal. We found four illegal errors in which the model combined morphology and/or UPOS in a way that breaks rules of grammar. Three of these involved the token quod. For example, when the gold annotation labeled quod as SCONJ, the model correctly predicted SCONJ but incorrectly predicted Gender=Neut and Number=Sing, when a SCONJ should have no value for those features. In the fourth case, when the gold was PRON, the model again correctly predicted PRON but incorrectly predicted Case=None and Number=None, even though a PRON should have values for those features.
Lexical. We found eight lexical errors where the predicted combination of UPOS and morphological features is legal in general, but is impossible given the particular token based on lexical information. For example, let’s consider the token ista whose gold annotation is a DET with Case=Nom, Gender=Fem, and Num=Sing. This word is a
1833 tokens from Classical texts, 33 from the bible, 12 from Aquinas’ works, 11 from LLCT, 11 from Dante’s works.
demonstrative adjective with 1st and 2nd declension endings, so out of context there are only a few combinations of morphological features possible: either Case=Nom,Abl|Gender=Fem|Num=Sing or Case=Nom,Acc|Gender=Neut|Num=Plur. Our tagger incorrectly predicted Case=Acc but correctly predicted Gender=Fem and Num=Sing. Even though its predictions for Gender and Number are correct, they do not form a valid combination of feature values for this token.
Genuine acontextual ambiguity. Most errors (67) were due to genuine acontextual ambiguity. This means that, out of context, the model’s prediction is legal and valid given the particular token’s lexical information, but in context it is incorrect. We would hope that BERT, as a contextual model, could still predict these cases correctly but it seems to struggle. Figure 4 shows an example of this error type. In other contexts, the token opera can be accusative plural, as the model predicted, but within this sentence it must be ablative singular. The verb perierimus (we have been ruined) does not take an object, so opera cannot be accusative. Additionally, the structure of quorum opera is repeated with quorum insidiis and quorum scelere. The nouns insidiis and scelere are clearly ablative, suggesting that opera should be the same case. This makes more sense contextually: perierimus (we have been ruined) quorum opera (by whose work).
Annotation differences. Nine errors were due to remaining annotation differences, discussed more thoroughly in §A.6.
Gold wrong. Nine errors were caused by incorrect gold annotations. These include missing Case value for nouns, and incorrect UPOS and morphological features.
Words segmented by the tokenizer have a higher error rate. Because of the presence of lexical errors in our model’s predictions, we investigated
whether the LatinBERT tokenizer segments words in a morphologically-aware manner. We find that the majority (81%) of words in our three custom test sets correspond to a single subtoken for the tokenizer. For these word tokens, our all-ud-custom model achieves 98.3% accuracy on UPOS and 97.2% accuracy on all morphological features. In the case that word tokens are split into multiple subtokens, performance degrades; Accuracy drops slightly for UPOS to 97.5% and more dramatically for morphological features to 93.6%. Since most words are not segmented and those that are have worse performance, we hypothesize that the model is not able to learn Latin’s inflections, which could hypothetically aid in the tagging of rarer words. The relationship between token frequency, word segmentation, and downstream performance is a promising direction for analysis in future work. This aligns with previous findings for English that transformer models with WordPiece tokenizers have lower generalization ability than those with morphologically-aware tokenization (Hofmann et al., 2021).

heading: 6 Conclusion and Future Work
text: In this work, we consider the diverse time periods represented in the Latin treebanks when training and evaluating morphological taggers. We hope the genre metadata we provide can be used for future cross-genre analysis of Latin, similar to the crosstime analysis we present in this paper.
We also believe further improvements can be made through the harmonization of remaining annotation differences (§3.4) and more informed modeling choices. Specifically, we hypothesize that (1) conditioning morphological feature prediction on UPOS, or vice versa; (2) enforcing grammatical constraints through modeling, rather than only through training data; and (3) constructing a morphologically-aware tokenizer may all lead to improved performance.

heading: Acknowledgments
text: We thank Gregory Crane, David Smith, Geoff Bakewell, the UMass NLP group, and anonymous reviewers for feedback. This work was supported by NSF CAREER 1845576. Any opinions, findings, and conclusions or recommendations expressed in this material are those of the authors and do not necessarily reflect the views of the National Science Foundation.

heading: A Appendix
text: A.1 UD Genres
We mark the following 12 genres: narrative, poem, short poem, letter, epic, history, satire, speech, treatise, Christian, Bible, legal.
These genres are not exclusive, so each text will have at least one, but possibly more genres marked.
In Figure 2, for simplicity we showed a subset of genres which are mutually exclusive. This also ensures the number of sentences shown in the figure exactly matches the number of sentences that exist in the UD treebanks. The additional genres that we left out of the figure are broader, covering multiple sub-genres. Specifically, narratives includes some (not all) texts from every genre except for legal texts and speeches. Poems includes epics and short poems, the two of which are mutually exclusive. Christian includes the Bible itself, as well as the religious treatises of Thomas Aquinas.
A.2 Finding Duplicate Sentences in LASLA and UD Treebanks
In order to detect duplicate sentences between the treebanks, we first normalize the orthographic variation across the UD treebanks and LASLA. We used CLTK’s (Johnson et al., 2021) JV replacer on the harmonized UD treebanks, since LASLA’s texts do not use the letters ‘j’ or ‘v’. We also remove any punctuation present in the UD treebanks, as LASLA does not have punctuation.
We search for duplicate sentences by finding sentence pairs with exact character or token overlap at the beginning or end of each sentence.
Within the duplicate sentences, we identify duplicate tokens by searching for the longest overlapping, contiguous subsequence of tokens of each sentence. We search for exact token matches. Our reported number of duplicate tokens is an underestimate, since there are sometimes token mismatches within sentences that are genuine duplicates. For example, one sentence may have a numeral where the other has the word form of the number.
A.3 Annotation Agreement
In Table 8, we show the annotation agreement between duplicate sentences in Perseus and PROIEL after our standardization and harmonization. Notably, these are both (Harmonized) UD treebanks (Table 1 row 4), and some annotation differences still remain, although agreement is generally still higher than between UD and LASLA.
In Table 9, we show the annotation agreement between Harmonized UD and LASLA, before our harmonization and standardization. This table also shows the union of UD and LASLA’s feature sets. There are many features we did not consider which could benefit from harmonization.
Finally, Table 12 is a venn diagram showing all possible features and values in UD and LASLA, before our harmonization and standardization.
A.4 Remaining Inconsistencies We’ve Harmonized
Here is a full list of arbitrary values we’ve enforced for certain grammatical constructions. For each of these items, there are arguably multiple correct ways to annotate—and the Latin treebanks were annotating these differently.
• Gerunds, Infinitives, and Supines should have Number=None
• Gerunds, Gerundives, and Supines should have Tense=None
• If UPOS is AUX, then Voice=Act. This almost entirely applies to forms of sum.
• Gerunds should have Voice=Act, and Gerundives should have Voice=Pass.
• Supines have Voice=Act, unless used in a construction with iri, in which case Voice=Pass.
• Gerunds, Infinitives, and Supines should have Gender=None.
A.5 Standardizing Tense and Mood
Tense We use the TraditionalTense field of the harmonized treebanks (Gamba and Zeman, 2023a), rather than the UD approach to tense. Altogether, the UD Latin treebanks include four tenses (present, past, future, pluperfect) and four aspects (imperfective, perfective, prospective, inchoative). When tense and aspect are considered together, they can represent the seven traditional Latin tenses. However, this is less intuitive for Classicists or those whose goal is to study only Latin. We chose to revert back to the traditional tenses. We were able to use the TraditionalTense field for most tags, but to differentiate between future and future perfect it is also necessary to look at Aspect. Additionally, we found that infinitives did not have a
TraditionalTense, so we looked to the Aspect feature value to determine the tense of infinitives.
For LASLA, since it does not have a TraditionalTense field, we look at both Tense and Aspect feature values to determine tense.
Our final set of tenses is: present, imperfect, perfect, pluperfect, future, and future perfect.
Mood Similar to tense, the non-finite moods are represented by a combination of the Mood and VerbForm fields in Gamba and Zeman (2023a)’s harmonized treebanks, with references to Latin-specific constructions being moved to the TraditionalMood field. Strictly speaking, nonfinite verbs do not have mood, but traditional Latin grammars still classify the different non-finite verbforms as "mood."19 Again, we opt to use the traditional terminology and follow the same tagset as the Perseus treebank. For finite verbs, this includes indicative, subjunctive, imperative; and for non-finite verbs, infinitive, participle, gerund, gerundive, and supine.
For LASLA, we are able to take the mood directly from the Mood feature for finite verbs, and from VerbForm feature for non-finite verbs. This is because LASLA uses the Latin-specific Ger,Gdv,Sup values for VerbForm, unlike the harmonized UD treebanks.
A.6 Remaining Inconsistencies We’re Unable to Harmonize
We are aware of the following differences, but leave their harmonization to future work:
• The pre-UD Perseus treebank (Table 1 row 1) has an additional Voice value for deponent verbs. After Gamba and Zeman (2023a)’s harmonization, deponent verbs in Perseus always have Voice=Act, but deponent verbs in every other UD treebank have Voice=Pass. We would like a system more similar to pre-UD Perseus with an additional Voice=Dep value.
• ITTB is the only treebank that sometimes marks esse, the infinitive of sum, as NOUN with Mood=None.
The following annotation differences were found to cause 9% of sampled errors in our BERT tagger’s morphological predictions:
19This is explained in the EvaLatin 2022 guidelines: https: //github.com/CIRCSE/LT4HALA/blob/master/2022/ data_and_doc/EvaLatin_2022_guidelines_v1.pdf
• Whether to have Case=None for undeclined nouns.
• Whether deponent verbs should be labeled as Voice=Act or Voice=Pass.
• Whether infinitives should have a value for Case.
• Whether infinitives can have their UPOS be NOUN, Mood=None, and Tense=None.
• Whether the pronoun sui should always have Number=None.
A.7 Finetuning Details
We use the same hyperparameters that Bamman and Burns (2020) used to finetune a POS tagger: Adam optimizer with learning rate 5× 10−5, early stopping patience of 10 epochs, batch size 32, dropout rate 0.25. We keep the model with the lowest validation loss across all epochs.
A.8 Metrics
Whole-String Morphological Accuracy Following the convention of Gamba and Zeman (2023a) and Sprugnoli et al. (2022), we consider the model’s prediction correct when every morphological feature is correctly predicted. We construct a morphological feature string from the predicted feature set, making sure to sort the features alphabetically. Then, we can test whether the predicted morphological string is an exact match to the gold string. Although this is a strict criteria, it indicates whether the model understands how all the morphological features fit together.
Macro F1 for Individual Features For UPOS and each individual morphological feature, we report Macro F1 in order to see how the model performs on rare feature values. If we define F as a particular feature and VF = {v1, ..., vn} as the set of possible values that F can take, then macro F1 is defined as 1n ∑n i=1 F1(vi). Note that v = None
is a possible value for every morphological feature, and is included in our calculation.
A.9 Randomized Permutation Testing Within a null simulation, for each test set sentence we shuffle the two models’ predictions, and store the absolute difference in the performance metric calculated from the entire shuffled test set. We finally report the p-value as the fraction of 10,000 simulated absolute differences that are larger than the observed absolute difference. p=0 simply means the observed difference is larger than in all simulations; it could be more conservatively interpreted as p < .0003 (Eypasch et al., 1995) due to Monte Carlo error.

heading: Future Work
text: We hope the genre metadata we provide can be used for future cross-genre analysis of Latin, similar to the crosstime analysis we present in this paper. We also believe further improvements can be made through the harmonization of remaining annotation differences (§3. 4) and more informed modeling choices. Specifically, we hypothesize that (1) conditioning morphological feature prediction on UPOS, or vice versa; (2) enforcing grammatical constraints through modeling, rather than only through training data; and (3) constructing a morphologically-aware tokenizer may all lead to improved performance